Openembedded Core Discussions
 help / color / mirror / Atom feed
From: Richard Purdie <richard.purdie@linuxfoundation.org>
To: openembedded-core <openembedded-core@lists.openembedded.org>
Cc: "Hart, Darren" <darren.hart@intel.com>
Subject: [PATCH] lib/oe/path: Fix performance issue got copyhardlinktree()
Date: Fri, 08 Nov 2013 15:19:05 +0000	[thread overview]
Message-ID: <1383923945.2345.5.camel@ted> (raw)

With the directory copy was added to avoid race issues, it wasn't noticed that
tar was recursing the directories and copying files too. This is completely
crazy when we hardlink those files in the next command.

Resolve the issue by telling tar not to recurse. This gives a significant
performance boost to various parts of the system (do_package for linux-yocto
256s -> 178s for example).

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
index 1310e38..d0588ba 100644
--- a/meta/lib/oe/path.py
+++ b/meta/lib/oe/path.py
@@ -93,7 +93,7 @@ def copyhardlinktree(src, dst):
     if (os.stat(src).st_dev ==  os.stat(dst).st_dev):
         # Need to copy directories only with tar first since cp will error if two 
         # writers try and create a directory at the same time
-        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --files-from - | tar -xf - -C %s' % (src, src, dst)
+        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --files-from - --no-recursion | tar -xf - -C %s' % (src, src, dst)
         check_output(cmd, shell=True, stderr=subprocess.STDOUT)
         if os.path.isdir(src):
             src = src + "/*"




             reply	other threads:[~2013-11-08 15:19 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2013-11-08 15:19 Richard Purdie [this message]
2013-11-08 15:42 ` [PATCH] lib/oe/path: Fix performance issue got copyhardlinktree() Bruce Ashfield
2013-11-09 20:43   ` Andrea Adami
2013-11-09 22:52     ` Richard Purdie
2013-11-09 23:58       ` Andrea Adami

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1383923945.2345.5.camel@ted \
    --to=richard.purdie@linuxfoundation.org \
    --cc=darren.hart@intel.com \
    --cc=openembedded-core@lists.openembedded.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox