public inbox for openembedded-core@lists.openembedded.org
 help / color / mirror / Atom feed
From: Alexander Kanavin <alexander.kanavin@linux.intel.com>
To: openembedded-core@lists.openembedded.org
Subject: [PATCHv2 0/2] Multi-threaded RPM support
Date: Thu,  8 Jun 2017 17:42:08 +0300	[thread overview]
Message-ID: <cover.1496932793.git.alexander.kanavin@linux.intel.com> (raw)

Changes in version 2: use openmp for multi-processing instead of NSPR library, as
requested by upstream. Add a link to the upstream submission to patches.

These two patches add multi-threaded features to RPM to speed up the do_package_write_rpm task.

Specifically:

1) Multi-threaded xz compressor is used instead of default single-threaded gz. This has the most
dramatic effect when a recipe produces a smaller number of large-sized packages.

2) Packages creation is run in multiple threads via thread pools. This is most beneficial when
a recipe produces a large amount of small packages.

Some not very scientific benchmarks for time and .rpm sizes
(time is measured for do_package_write_rpm tasks only):

webkitgtk
before: 9m12s 1550M
after: 1m40s 858M

glibc-locale
before: 2m52s 125M
after: 30s 56M

glibc
before: 46s 54M
after: 13s 38M

perl
before: 1m09s 63M
after: 45s 42M

python3
before: 30s 38M
after: 18s 24M

The following changes since commit 576821ea0a7558b626ccc87e9ae0e9ee40864956:

  bitbake: bitbake-layers: check layer dependencies before adding (2017-06-06 19:52:51 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib akanavin/parallel-rpm
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=akanavin/parallel-rpm

Alexander Kanavin (2):
  package_rpm.bbclass: use multithreaded xz compression
  rpm: run binary package generation via thread pools

 meta/classes/package_rpm.bbclass                   |   2 +
 ...y-package-building-into-a-separate-functi.patch |  83 ++++++++
 ...-binary-package-creation-via-thread-pools.patch | 125 ++++++++++++
 ...c-make-operations-over-string-pools-threa.patch | 207 ++++++++++++++++++++
 ...c-remove-static-local-variables-from-buil.patch | 216 +++++++++++++++++++++
 meta/recipes-devtools/rpm/rpm_git.bb               |   4 +
 6 files changed, 637 insertions(+)
 create mode 100644 meta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch
 create mode 100644 meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch
 create mode 100644 meta/recipes-devtools/rpm/files/0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch
 create mode 100644 meta/recipes-devtools/rpm/files/0004-build-pack.c-remove-static-local-variables-from-buil.patch

-- 
2.11.0



             reply	other threads:[~2017-06-08 14:42 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-06-08 14:42 Alexander Kanavin [this message]
2017-06-08 14:42 ` [PATCHv2 1/2] package_rpm.bbclass: use multithreaded xz compression Alexander Kanavin
2017-06-08 14:42 ` [PATCHv2 2/2] rpm: run binary package generation via thread pools Alexander Kanavin
2017-06-08 16:55   ` Mark Hatle
2017-06-08 17:23     ` Alexander Kanavin
2017-06-08 20:29       ` Mark Hatle
2017-06-09  9:02         ` Alexander Kanavin
2017-06-09 11:18           ` Alexander Kanavin
2017-06-08 21:15 ` [PATCHv2 0/2] Multi-threaded RPM support Leonardo Sandoval
2017-06-09  9:13   ` Alexander Kanavin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=cover.1496932793.git.alexander.kanavin@linux.intel.com \
    --to=alexander.kanavin@linux.intel.com \
    --cc=openembedded-core@lists.openembedded.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox