[PATCH 0/3] Allows to copy symlinks to DUTs with testimage

[PATCH 0/3] Allows to copy symlinks to DUTs with testimage

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

Currently when using the install functionality in the DUT without package
manager, if you copy a symlink the function will fail.

These series will change the behaviour and will allow to recreate the
symlinks in the DUT.

[YOCTO #9932]


The following changes since commit 039f47ad197a9a53109c9f3deadd9c35e62c056d:

  uclibc: remove meta-yocto-bsp append (2016-07-26 08:56:32 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib mariano/bug9932
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mariano/bug9932

Mariano Lopez (3):
  oeqa/utils/commands.py: Add oeqa_copy function
  oeqa/oetest.py: Allow to export packages using symlinks
  oeqa/utils/sshcontrol.py: Allows to copy symlinks to target

 meta/classes/testexport.bbclass   | 22 ++++------------------
 meta/lib/oeqa/oetest.py           | 16 ++++++++++++----
 meta/lib/oeqa/utils/commands.py   |  9 +++++++++
 meta/lib/oeqa/utils/sshcontrol.py |  9 +++++++--
 4 files changed, 32 insertions(+), 24 deletions(-)

-- 
2.6.6

[PATCH 1/3] oeqa/utils/commands.py: Add oeqa_copy function

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This new function is just a call to shutil.copy2() with
follow_symlinks set to False. This function is needed
when using shutil.copytree() to preserve the symlinks.

[YOCTO #9932]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/lib/oeqa/utils/commands.py | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/meta/lib/oeqa/utils/commands.py b/meta/lib/oeqa/utils/commands.py
index 4f79d15..eedc4d7 100644
--- a/meta/lib/oeqa/utils/commands.py
+++ b/meta/lib/oeqa/utils/commands.py
@@ -18,6 +18,7 @@ from oeqa.utils import CommandError
 from oeqa.utils import ftools
 import re
 import contextlib
+import shutil
 # Export test doesn't require bb
 try:
     import bb
@@ -273,3 +274,11 @@ def updateEnv(env_file):
     for line in result.output.split("\0"):
         (key, _, value) = line.partition("=")
         os.environ[key] = value
+
+def oeqa_copy(src, dst):
+    """
+    Copy files and symlinks trying to preserve metadata.
+    """
+
+    shutil.copy2(src, dst, follow_symlinks=False)
+
-- 
2.6.6

[PATCH 2/3] oeqa/oetest.py: Allow to export packages using symlinks

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

Currently packages that contains symlinks can't be extracted
and exported. This allows to export extracted such packages.

A nice side effect is improved readability.

[YOCTO #9932]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/classes/testexport.bbclass | 22 ++++------------------
 meta/lib/oeqa/oetest.py         | 16 ++++++++++++----
 2 files changed, 16 insertions(+), 22 deletions(-)

diff --git a/meta/classes/testexport.bbclass b/meta/classes/testexport.bbclass
index 15fa470..4d2641b 100644
--- a/meta/classes/testexport.bbclass
+++ b/meta/classes/testexport.bbclass
@@ -47,6 +47,7 @@ def exportTests(d,tc):
     import shutil
     import pkgutil
     import re
+    from oeqa.utils.commands import oeqa_copy
 
     exportpath = d.getVar("TEST_EXPORT_DIR", True)
 
@@ -132,27 +133,12 @@ def exportTests(d,tc):
     create_tarball(d, "testexport.tar.gz", d.getVar("TEST_EXPORT_DIR", True))
 
     # Copy packages needed for runtime testing
-    export_pkg_dir = os.path.join(d.getVar("TEST_EXPORT_DIR", True), "packages")
     test_pkg_dir = d.getVar("TEST_NEEDED_PACKAGES_DIR", True)
-    need_pkg_dir = False
-    for root, subdirs, files in os.walk(test_pkg_dir):
-        for subdir in subdirs:
-            tmp_dir = os.path.join(root.replace(test_pkg_dir, "").lstrip("/"), subdir)
-            new_dir = os.path.join(export_pkg_dir, tmp_dir)
-            bb.utils.mkdirhier(new_dir)
-
-        for f in files:
-            need_pkg_dir = True
-            src_f = os.path.join(root, f)
-            dst_f = os.path.join(export_pkg_dir, root.replace(test_pkg_dir, "").lstrip("/"), f)
-            shutil.copy2(src_f, dst_f)
-
-    if need_pkg_dir:
+    if os.listdir(test_pkg_dir):
+        export_pkg_dir = os.path.join(d.getVar("TEST_EXPORT_DIR", True), "packages")
+        shutil.copytree(test_pkg_dir, export_pkg_dir, copy_function=oeqa_copy)
         # Create tar file for packages needed by the DUT
         create_tarball(d, "testexport_packages_%s.tar.gz" % d.getVar("MACHINE", True), export_pkg_dir)
-    else:
-        # Remov packages dir from exported test
-        bb.utils.remove(export_pkg_dir, True)
 
     # Copy SDK
     if d.getVar("TEST_EXPORT_SDK_ENABLED", True) == "1":
diff --git a/meta/lib/oeqa/oetest.py b/meta/lib/oeqa/oetest.py
index e63ca56..bb6b908 100644
--- a/meta/lib/oeqa/oetest.py
+++ b/meta/lib/oeqa/oetest.py
@@ -26,6 +26,7 @@ try:
     import oeqa.sdkext
 except ImportError:
     pass
+from oeqa.utils.commands import oeqa_copy
 from oeqa.utils.decorators import LogResults, gettag, getResults
 from oeqa.utils import avoid_paths_in_environ
 
@@ -447,6 +448,8 @@ class RuntimeTestContext(TestContext):
         modules = self.getTestModules()
         bbpaths = self.d.getVar("BBPATH", True).split(":")
 
+        shutil.rmtree(self.d.getVar("TEST_EXTRACTED_DIR", True))
+        shutil.rmtree(self.d.getVar("TEST_PACKAGED_DIR", True))
         for module in modules:
             json_file = self._getJsonFile(module)
             if json_file:
@@ -481,13 +484,18 @@ class RuntimeTestContext(TestContext):
                     dst_dir = os.path.join(packaged_path)
 
                 # Extract package and copy it to TEST_EXTRACTED_DIR
-                if extract and not os.path.exists(dst_dir):
-                    pkg_dir = self._extract_in_tmpdir(pkg)
-                    shutil.copytree(pkg_dir, dst_dir)
+                pkg_dir = self._extract_in_tmpdir(pkg)
+                if extract:
+
+                    # Same package used for more than one test,
+                    # don't need to extract again.
+                    if os.path.exists(dst_dir):
+                        continue
+                    shutil.copytree(pkg_dir, dst_dir, copy_function=oeqa_copy)
                     shutil.rmtree(pkg_dir)
 
                 # Copy package to TEST_PACKAGED_DIR
-                elif not extract:
+                else:
                     self._copy_package(pkg)
 
     def _getJsonFile(self, module):
-- 
2.6.6

[PATCH 3/3] oeqa/utils/sshcontrol.py: Allows to copy symlinks to target

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

Currently when copying a symlink to the target it will fail
throwing an exception. This will recreate symlinks from
the system performing the tests to the device under tests.

[YOCTO #9932]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/lib/oeqa/utils/sshcontrol.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/utils/sshcontrol.py b/meta/lib/oeqa/utils/sshcontrol.py
index f5d46e0..da485ee 100644
--- a/meta/lib/oeqa/utils/sshcontrol.py
+++ b/meta/lib/oeqa/utils/sshcontrol.py
@@ -147,8 +147,13 @@ class SSHControl(object):
         return self._internal_run(command, timeout, self.ignore_status)
 
     def copy_to(self, localpath, remotepath):
-        command = self.scp + [localpath, '%s@%s:%s' % (self.user, self.ip, remotepath)]
-        return self._internal_run(command, ignore_status=False)
+        if os.path.islink(localpath):
+            link = os.readlink(localpath)
+            dst_dir, dst_base = os.path.split(remotepath)
+            return self.run("cd %s; ln -s %s %s" % (dst_dir, link, dst_base))
+        else:
+            command = self.scp + [localpath, '%s@%s:%s' % (self.user, self.ip, remotepath)]
+            return self._internal_run(command, ignore_status=False)
 
     def copy_from(self, remotepath, localpath):
         command = self.scp + ['%s@%s:%s' % (self.user, self.ip, remotepath), localpath]
-- 
2.6.6

Re: [PATCH 1/3] oeqa/utils/commands.py: Add oeqa_copy function

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 326 bytes --]

On 26 July 2016 at 08:39, <mariano.lopez@linux.intel.com> wrote:

> +def oeqa_copy(src, dst):
> +    """
> +    Copy files and symlinks trying to preserve metadata.
> +    """
> +
> +    shutil.copy2(src, dst, follow_symlinks=False)
>

Why not just use shutil directly instead of having a wrapper function?

Ross

[-- Attachment #2: Type: text/html, Size: 787 bytes --]

Re: [PATCH 0/3] Allows to copy symlinks to DUTs with testimage

[Joshua G Lock]

On Tue, 2016-07-26 at 07:39 +0000, mariano.lopez@linux.intel.com wrote:
> From: Mariano Lopez <mariano.lopez@linux.intel.com>
> 
> Currently when using the install functionality in the DUT without
> package
> manager, if you copy a symlink the function will fail.
> 
> These series will change the behaviour and will allow to recreate the
> symlinks in the DUT.
> 
> [YOCTO #9932]


Would oe.path.copytree() work instead of using shutil.copytree()?
copytree() was added because shutil was quite slow for our usage.

Regards,

Joshua

Re: [PATCH 1/3] oeqa/utils/commands.py: Add oeqa_copy function

[Mariano Lopez]

[-- Attachment #1: Type: text/plain, Size: 661 bytes --]



On 07/26/2016 11:13 AM, Burton, Ross wrote:
>
> On 26 July 2016 at 08:39, <mariano.lopez@linux.intel.com 
> <mailto:mariano.lopez@linux.intel.com>> wrote:
>
>     +def oeqa_copy(src, dst):
>     +    """
>     +    Copy files and symlinks trying to preserve metadata.
>     +    """
>     +
>     +    shutil.copy2(src, dst, follow_symlinks=False)
>
>
> Why not just use shutil directly instead of having a wrapper function?

shutil.copytree didn't play nice with the symlinks, this is why I added 
this function, nevertheless, I like Joshua's approach of use 
oe.path.copytree(), so this function won't be needed.

>
> Ross

Mariano

[-- Attachment #2: Type: text/html, Size: 2002 bytes --]

Re: [PATCH 0/3] Allows to copy symlinks to DUTs with testimage

[Mariano Lopez]

On 07/26/2016 11:27 AM, Joshua G Lock wrote:
> On Tue, 2016-07-26 at 07:39 +0000, mariano.lopez@linux.intel.com wrote:
>> From: Mariano Lopez <mariano.lopez@linux.intel.com>
>>
>> Currently when using the install functionality in the DUT without
>> package
>> manager, if you copy a symlink the function will fail.
>>
>> These series will change the behaviour and will allow to recreate the
>> symlinks in the DUT.
>>
>> [YOCTO #9932]
>
> Would oe.path.copytree() work instead of using shutil.copytree()?
> copytree() was added because shutil was quite slow for our usage.

I didn't know the existence of that function, I tried it and it can 
replace shutil.copytree(). I'll send a v2.

>
> Regards,
>
> Joshua

Mariano Lopez