* [PATCH 0/1] oe/copy_buildsystem.py: dereference symlink @ 2016-10-31 15:48 Robert Yang 2016-10-31 15:48 ` [PATCH 1/1] " Robert Yang 0 siblings, 1 reply; 6+ messages in thread From: Robert Yang @ 2016-10-31 15:48 UTC (permalink / raw) To: openembedded-core The following changes since commit 42b4fa2f923244bc047874752d2e0381ff6f0a25: boost: fix the SRC_URI to point to an actual release, and not a master snapshot (2016-10-31 14:23:04 +0000) are available in the git repository at: git://git.openembedded.org/openembedded-core-contrib rbt/cp http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/cp Robert Yang (1): oe/copy_buildsystem.py: dereference symlink meta/lib/oe/copy_buildsystem.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) -- 2.9.0 ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 1/1] oe/copy_buildsystem.py: dereference symlink 2016-10-31 15:48 [PATCH 0/1] oe/copy_buildsystem.py: dereference symlink Robert Yang @ 2016-10-31 15:48 ` Robert Yang 2016-11-01 10:09 ` Richard Purdie 0 siblings, 1 reply; 6+ messages in thread From: Robert Yang @ 2016-10-31 15:48 UTC (permalink / raw) To: openembedded-core When there is a relative symlink in the layer, for example: symA -> ../out/of/layer/file symA will be invalid fater copied, it would be invalid from build time if it points to a relative path, and would be invalid after extracted the sdk if it points to a absolute py. Dereference symlink when copy will fix the problem. Use tar rather than shutil.copytree() to copy is because: 1) shutil.copytree(symlinks=Fasle) has bugs when dereference symlinks: https://bugs.python.org/issue21697 And Ubunutu 1404 doesn't upgrade python3 to fix the problem. 2) shutil.copytree(symlinks=False) raises errors when there is a invalid symlink, and tar just prints a warning, tar is preferred here since the real world is unpredicatable 3) tar is faster than shutil.copytree() as said by oe.path.copytree() So use tar to copy. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> --- meta/lib/oe/copy_buildsystem.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py index afaff68..29ac6d4 100644 --- a/meta/lib/oe/copy_buildsystem.py +++ b/meta/lib/oe/copy_buildsystem.py @@ -4,11 +4,15 @@ import stat import shutil def _smart_copy(src, dest): + import subprocess # smart_copy will choose the correct function depending on whether the # source is a file or a directory. mode = os.stat(src).st_mode if stat.S_ISDIR(mode): - shutil.copytree(src, dest, symlinks=True, ignore=shutil.ignore_patterns('.git')) + bb.utils.mkdirhier(dest) + cmd = "tar --exclude='.git' --xattrs --xattrs-include='*' -chf - -C %s -p . \ + | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, dest) + subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) else: shutil.copyfile(src, dest) shutil.copymode(src, dest) -- 2.9.0 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] oe/copy_buildsystem.py: dereference symlink 2016-10-31 15:48 ` [PATCH 1/1] " Robert Yang @ 2016-11-01 10:09 ` Richard Purdie 2016-11-01 10:18 ` Robert Yang 0 siblings, 1 reply; 6+ messages in thread From: Richard Purdie @ 2016-11-01 10:09 UTC (permalink / raw) To: Robert Yang, openembedded-core On Mon, 2016-10-31 at 08:48 -0700, Robert Yang wrote: > When there is a relative symlink in the layer, for example: > symA -> ../out/of/layer/file > > symA will be invalid fater copied, it would be invalid from build > time > if it points to a relative path, and would be invalid after extracted > the sdk if it points to a absolute py. Dereference symlink when copy > will fix the problem. > > Use tar rather than shutil.copytree() to copy is because: > 1) shutil.copytree(symlinks=Fasle) has bugs when dereference > symlinks: > https://bugs.python.org/issue21697 > And Ubunutu 1404 doesn't upgrade python3 to fix the problem. > > 2) shutil.copytree(symlinks=False) raises errors when there is a > invalid > symlink, and tar just prints a warning, tar is preferred here > since > the real world is unpredicatable > > 3) tar is faster than shutil.copytree() as said by oe.path.copytree() Could we just use oe.path.copytree() here? Cheers, Richard ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] oe/copy_buildsystem.py: dereference symlink 2016-11-01 10:09 ` Richard Purdie @ 2016-11-01 10:18 ` Robert Yang 2016-11-01 12:15 ` Burton, Ross 0 siblings, 1 reply; 6+ messages in thread From: Robert Yang @ 2016-11-01 10:18 UTC (permalink / raw) To: Richard Purdie, openembedded-core On 11/01/2016 06:09 PM, Richard Purdie wrote: > On Mon, 2016-10-31 at 08:48 -0700, Robert Yang wrote: >> When there is a relative symlink in the layer, for example: >> symA -> ../out/of/layer/file >> >> symA will be invalid fater copied, it would be invalid from build >> time >> if it points to a relative path, and would be invalid after extracted >> the sdk if it points to a absolute py. Dereference symlink when copy >> will fix the problem. >> >> Use tar rather than shutil.copytree() to copy is because: >> 1) shutil.copytree(symlinks=Fasle) has bugs when dereference >> symlinks: >> https://bugs.python.org/issue21697 >> And Ubunutu 1404 doesn't upgrade python3 to fix the problem. >> >> 2) shutil.copytree(symlinks=False) raises errors when there is a >> invalid >> symlink, and tar just prints a warning, tar is preferred here >> since >> the real world is unpredicatable >> >> 3) tar is faster than shutil.copytree() as said by oe.path.copytree() > > Could we just use oe.path.copytree() here? I should explain why not use oe.path.copytree(), but it was late last night, so I forgot that. We can't use oe.path.copytree() since: 1) oe.path.copytree() doesn't dereference symlink 2) We need --exclude='.git' I'd like to add two arguments to oe.path.copytree() to do this if it is worth. // Robert > > Cheers, > > Richard > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] oe/copy_buildsystem.py: dereference symlink 2016-11-01 10:18 ` Robert Yang @ 2016-11-01 12:15 ` Burton, Ross 2016-11-01 13:17 ` Robert Yang 0 siblings, 1 reply; 6+ messages in thread From: Burton, Ross @ 2016-11-01 12:15 UTC (permalink / raw) To: Robert Yang; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 512 bytes --] On 1 November 2016 at 10:18, Robert Yang <liezhi.yang@windriver.com> wrote: > I should explain why not use oe.path.copytree(), but it was late last > night, so I forgot that. We can't use oe.path.copytree() since: > 1) oe.path.copytree() doesn't dereference symlink > 2) We need --exclude='.git' > > I'd like to add two arguments to oe.path.copytree() to do this if it is > worth. > Adding options to dereference and exclude a specified list of files sounds like a good improvement to me. Ross [-- Attachment #2: Type: text/html, Size: 947 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] oe/copy_buildsystem.py: dereference symlink 2016-11-01 12:15 ` Burton, Ross @ 2016-11-01 13:17 ` Robert Yang 0 siblings, 0 replies; 6+ messages in thread From: Robert Yang @ 2016-11-01 13:17 UTC (permalink / raw) To: Burton, Ross; +Cc: OE-core On 11/01/2016 08:15 PM, Burton, Ross wrote: > > On 1 November 2016 at 10:18, Robert Yang <liezhi.yang@windriver.com > <mailto:liezhi.yang@windriver.com>> wrote: > > I should explain why not use oe.path.copytree(), but it was late last > night, so I forgot that. We can't use oe.path.copytree() since: > 1) oe.path.copytree() doesn't dereference symlink > 2) We need --exclude='.git' > > I'd like to add two arguments to oe.path.copytree() to do this if it is worth. > > > Adding options to dereference and exclude a specified list of files sounds like > a good improvement to me. Thanks, will send a V2. // Robert > > Ross ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-11-01 13:17 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2016-10-31 15:48 [PATCH 0/1] oe/copy_buildsystem.py: dereference symlink Robert Yang 2016-10-31 15:48 ` [PATCH 1/1] " Robert Yang 2016-11-01 10:09 ` Richard Purdie 2016-11-01 10:18 ` Robert Yang 2016-11-01 12:15 ` Burton, Ross 2016-11-01 13:17 ` Robert Yang
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.