[PATCH 0/1] oe/copy_buildsystem.py: dereference symlink

[PATCH 0/1] oe/copy_buildsystem.py: dereference symlink

[Robert Yang]

The following changes since commit 42b4fa2f923244bc047874752d2e0381ff6f0a25:

  boost: fix the SRC_URI to point to an actual release, and not a master snapshot (2016-10-31 14:23:04 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib rbt/cp
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/cp

Robert Yang (1):
  oe/copy_buildsystem.py: dereference symlink

 meta/lib/oe/copy_buildsystem.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

-- 
2.9.0

[PATCH 1/1] oe/copy_buildsystem.py: dereference symlink

[Robert Yang]

When there is a relative symlink in the layer, for example:
symA -> ../out/of/layer/file

symA will be invalid fater copied, it would be invalid from build time
if it points to a relative path, and would be invalid after extracted
the sdk if it points to a absolute py. Dereference symlink when copy
will fix the problem.

Use tar rather than shutil.copytree() to copy is because:
1) shutil.copytree(symlinks=Fasle) has bugs when dereference symlinks:
   https://bugs.python.org/issue21697
   And Ubunutu 1404 doesn't upgrade python3 to fix the problem.

2) shutil.copytree(symlinks=False) raises errors when there is a invalid
   symlink, and tar just prints a warning, tar is preferred here since
   the real world is unpredicatable

3) tar is faster than shutil.copytree() as said by oe.path.copytree()

So use tar to copy.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/lib/oe/copy_buildsystem.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py
index afaff68..29ac6d4 100644
--- a/meta/lib/oe/copy_buildsystem.py
+++ b/meta/lib/oe/copy_buildsystem.py
@@ -4,11 +4,15 @@ import stat
 import shutil
 
 def _smart_copy(src, dest):
+    import subprocess
     # smart_copy will choose the correct function depending on whether the
     # source is a file or a directory.
     mode = os.stat(src).st_mode
     if stat.S_ISDIR(mode):
-        shutil.copytree(src, dest, symlinks=True, ignore=shutil.ignore_patterns('.git'))
+        bb.utils.mkdirhier(dest)
+        cmd = "tar --exclude='.git' --xattrs --xattrs-include='*' -chf - -C %s -p . \
+        | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, dest)
+        subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
     else:
         shutil.copyfile(src, dest)
         shutil.copymode(src, dest)
-- 
2.9.0

Re: [PATCH 1/1] oe/copy_buildsystem.py: dereference symlink

[Richard Purdie]

On Mon, 2016-10-31 at 08:48 -0700, Robert Yang wrote:
> When there is a relative symlink in the layer, for example:
> symA -> ../out/of/layer/file
> 
> symA will be invalid fater copied, it would be invalid from build
> time
> if it points to a relative path, and would be invalid after extracted
> the sdk if it points to a absolute py. Dereference symlink when copy
> will fix the problem.
> 
> Use tar rather than shutil.copytree() to copy is because:
> 1) shutil.copytree(symlinks=Fasle) has bugs when dereference
> symlinks:
>    https://bugs.python.org/issue21697
>    And Ubunutu 1404 doesn't upgrade python3 to fix the problem.
> 
> 2) shutil.copytree(symlinks=False) raises errors when there is a
> invalid
>    symlink, and tar just prints a warning, tar is preferred here
> since
>    the real world is unpredicatable
> 
> 3) tar is faster than shutil.copytree() as said by oe.path.copytree()

Could we just use oe.path.copytree() here?

Cheers,

Richard

Re: [PATCH 1/1] oe/copy_buildsystem.py: dereference symlink

[Robert Yang]

On 11/01/2016 06:09 PM, Richard Purdie wrote:
> On Mon, 2016-10-31 at 08:48 -0700, Robert Yang wrote:
>> When there is a relative symlink in the layer, for example:
>> symA -> ../out/of/layer/file
>>
>> symA will be invalid fater copied, it would be invalid from build
>> time
>> if it points to a relative path, and would be invalid after extracted
>> the sdk if it points to a absolute py. Dereference symlink when copy
>> will fix the problem.
>>
>> Use tar rather than shutil.copytree() to copy is because:
>> 1) shutil.copytree(symlinks=Fasle) has bugs when dereference
>> symlinks:
>>    https://bugs.python.org/issue21697
>>    And Ubunutu 1404 doesn't upgrade python3 to fix the problem.
>>
>> 2) shutil.copytree(symlinks=False) raises errors when there is a
>> invalid
>>    symlink, and tar just prints a warning, tar is preferred here
>> since
>>    the real world is unpredicatable
>>
>> 3) tar is faster than shutil.copytree() as said by oe.path.copytree()
>
> Could we just use oe.path.copytree() here?

I should explain why not use oe.path.copytree(), but it was late last
night, so I forgot that. We can't use oe.path.copytree() since:
1) oe.path.copytree() doesn't dereference symlink
2) We need --exclude='.git'

I'd like to add two arguments to oe.path.copytree() to do this if it is worth.

// Robert

>
> Cheers,
>
> Richard
>

Re: [PATCH 1/1] oe/copy_buildsystem.py: dereference symlink

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 512 bytes --]

On 1 November 2016 at 10:18, Robert Yang <liezhi.yang@windriver.com> wrote:

> I should explain why not use oe.path.copytree(), but it was late last
> night, so I forgot that. We can't use oe.path.copytree() since:
> 1) oe.path.copytree() doesn't dereference symlink
> 2) We need --exclude='.git'
>
> I'd like to add two arguments to oe.path.copytree() to do this if it is
> worth.
>

Adding options to dereference and exclude a specified list of files sounds
like a good improvement to me.

Ross

[-- Attachment #2: Type: text/html, Size: 947 bytes --]

Re: [PATCH 1/1] oe/copy_buildsystem.py: dereference symlink

[Robert Yang]

On 11/01/2016 08:15 PM, Burton, Ross wrote:
>
> On 1 November 2016 at 10:18, Robert Yang <liezhi.yang@windriver.com
> <mailto:liezhi.yang@windriver.com>> wrote:
>
>     I should explain why not use oe.path.copytree(), but it was late last
>     night, so I forgot that. We can't use oe.path.copytree() since:
>     1) oe.path.copytree() doesn't dereference symlink
>     2) We need --exclude='.git'
>
>     I'd like to add two arguments to oe.path.copytree() to do this if it is worth.
>
>
> Adding options to dereference and exclude a specified list of files sounds like
> a good improvement to me.

Thanks, will send a V2.

// Robert

>
> Ross