Openembedded Core Discussions
 help / color / mirror / Atom feed
From: Richard Purdie <richard.purdie@linuxfoundation.org>
To: openembedded-core@lists.openembedded.org
Subject: [PATCH 10/10] sstate/bitbake.conf: Use pigz if available
Date: Fri, 20 Jul 2018 11:39:48 +0100	[thread overview]
Message-ID: <20180720103948.30044-10-richard.purdie@linuxfoundation.org> (raw)
In-Reply-To: <20180720103948.30044-1-richard.purdie@linuxfoundation.org>

Currently the compression of sstate objects is single threaded. In the case
of ltp, this takes around 33s. If we add pigz into the list of non-fatal
HOSTTOOLS and then use if it available when building the sstate object,
this time drops to around 6s.

Since pigz is now widely available this is an optimisation we should
utilise.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/sstate.bbclass | 11 +++++++++--
 meta/conf/bitbake.conf      |  2 +-
 2 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 5a0722567a5..350d3107f59 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -715,17 +715,24 @@ sstate_task_postfunc[dirs] = "${WORKDIR}"
 #
 sstate_create_package () {
 	TFILE=`mktemp ${SSTATE_PKG}.XXXXXXXX`
+
+        # Use pigz if available
+        OPT="-cz"
+        if [ -x "$(command -v pigz)" ]; then
+            OPT="-I pigz -c"
+        fi
+
 	# Need to handle empty directories
 	if [ "$(ls -A)" ]; then
 		set +e
-		tar -czf $TFILE *
+		tar $OPT -f $TFILE *
 		ret=$?
 		if [ $ret -ne 0 ] && [ $ret -ne 1 ]; then
 			exit 1
 		fi
 		set -e
 	else
-		tar -cz --file=$TFILE --files-from=/dev/null
+		tar $OPT --file=$TFILE --files-from=/dev/null
 	fi
 	chmod 0664 $TFILE
 	mv -f $TFILE ${SSTATE_PKG}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 8f738545995..e28f3c7311c 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -490,7 +490,7 @@ HOSTTOOLS += " \
 HOSTTOOLS += "${@'ip ping ps scp ssh stty' if (bb.utils.contains_any('IMAGE_CLASSES', 'testimage testsdk', True, False, d) or any(x in (d.getVar("BBINCLUDED") or "") for x in ["testimage.bbclass", "testsdk.bbclass"])) else ''}"
 
 # Link to these if present
-HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc sftp socat ssh sudo"
+HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo"
 
 # Temporary add few more detected in bitbake world
 HOSTTOOLS_NONFATAL += "join nl size yes zcat"
-- 
2.17.1



  parent reply	other threads:[~2018-07-20 10:40 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-07-20 10:39 [PATCH 01/10] utils: Add multiprocess_launch API and testcase Richard Purdie
2018-07-20 10:39 ` [PATCH 02/10] package: Rework PACKAGELOCK based upon sstate for do_packagedata Richard Purdie
2018-07-20 10:39 ` [PATCH 03/10] package_ipk|deb: Use oe.utils.multiprocess_launch Richard Purdie
2018-07-20 10:39 ` [PATCH 04/10] package: Drop subshell usage for dwarfsrcfile generation Richard Purdie
2018-07-20 10:39 ` [PATCH 05/10] package: Don't use subshell to execute file Richard Purdie
2018-07-20 10:39 ` [PATCH 06/10] package: Allow parallel processing of debug splitting Richard Purdie
2018-07-20 10:39 ` [PATCH 07/10] package: Allow parallel processing of shlib analysis Richard Purdie
2018-07-20 10:39 ` [PATCH 08/10] package: Refactor to remove isElf/is_elf function duplication Richard Purdie
2018-07-20 10:39 ` [PATCH 09/10] package: Call file to determine elf status in parallel Richard Purdie
2018-07-20 10:39 ` Richard Purdie [this message]
2018-07-20 11:02 ` ✗ patchtest: failure for "utils: Add multiprocess_launch..." and 9 more Patchwork

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180720103948.30044-10-richard.purdie@linuxfoundation.org \
    --to=richard.purdie@linuxfoundation.org \
    --cc=openembedded-core@lists.openembedded.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox