[PATCH 0/6] fixes and improvements for layers cloning

[PATCH 0/6] fixes and improvements for layers cloning

[Ed Bartosh]

Hi,

This patchset includes changes for Toaster cloning functionality:
 - used more appropriate git commands: git reset --hard instead of
   git checkout && git rebase.
 - changed naming of clone directories, made them unique
 - moved clones to subdirectory

Please, review.

The following changes since commit 7b86c771c80d0759c2ca0e57c46c4c966f89c49e:

  bitbake: bitbake: bb.fetch2.git: Import errno module (2015-09-19 22:38:44 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/toaster/clones
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/clones

Ed Bartosh (6):
  toaster: don't use --single-branch when cloning
  toaster: use git reset --hard instead of rebase
  toaster: fix bug in resetting git repository
  toaster: fix reimporting module
  toaster: make clone directory name unique
  toaster: move clones into subdirectory

 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

--
Ed

[PATCH 1/6] toaster: don't use --single-branch when cloning

[Ed Bartosh]

git clone --single-branch works only with ref names. It fails if
commit sha1 is set in layer configuration with this error:
fatal: Remote branch <commit sha1> not found in upstream origin

Cloning repository without using --single-branch should work for
refs and commit sha1.

[YOCTO #7505]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index a9909b8..3d2f8b0 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -260,8 +260,8 @@ class LocalhostBEController(BuildEnvironmentController):
                     self._shellcmd("git remote remove origin", localdirname)
                     self._shellcmd("git remote add origin \"%s\"" % giturl, localdirname)
                 else:
-                    logger.debug("localhostbecontroller: cloning %s:%s in %s" % (giturl, commit, localdirname))
-                    self._shellcmd("git clone \"%s\" --single-branch --branch \"%s\" \"%s\"" % (giturl, commit, localdirname))
+                    logger.debug("localhostbecontroller: cloning %s in %s" % (giturl, localdirname))
+                    self._shellcmd('git clone "%s" "%s"' % (giturl, localdirname))
 
             # branch magic name "HEAD" will inhibit checkout
             if commit != "HEAD":
-- 
2.1.4

[PATCH 2/6] toaster: use git reset --hard instead of rebase

[Ed Bartosh]

Replaced 'git checkout <ref> && git rebase' with 'git reset --hard' as
with git checkout repository ends up with detached HEAD. Rebase makes
things even worse as it can cause conflicts. git reset --hard resets
repository to the required state in a most straightforward and
reliable way.

[YOCTO #7505]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 3d2f8b0..42be4af 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -266,7 +266,7 @@ class LocalhostBEController(BuildEnvironmentController):
             # branch magic name "HEAD" will inhibit checkout
             if commit != "HEAD":
                 logger.debug("localhostbecontroller: checking out commit %s to %s " % (commit, localdirname))
-                self._shellcmd("git fetch --all && git checkout \"%s\" && git rebase \"origin/%s\"" % (commit, commit) , localdirname)
+                self._shellcmd('git fetch --all && git reset --hard "origin/%s"' % commit, localdirname)
 
             # take the localdirname as poky dir if we can find the oe-init-build-env
             if self.pokydirname is None and os.path.exists(os.path.join(localdirname, "oe-init-build-env")):
-- 
2.1.4

[PATCH 3/6] toaster: fix bug in resetting git repository

[Ed Bartosh]

git reset --hard should be given either commit id or
origin/<ref name> to work properly. Without this fix git will
complain that origin/<commit id> does not exist.

[YOCTO #7505]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 42be4af..0c34957 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -266,7 +266,8 @@ class LocalhostBEController(BuildEnvironmentController):
             # branch magic name "HEAD" will inhibit checkout
             if commit != "HEAD":
                 logger.debug("localhostbecontroller: checking out commit %s to %s " % (commit, localdirname))
-                self._shellcmd('git fetch --all && git reset --hard "origin/%s"' % commit, localdirname)
+                ref = commit if re.match('^[a-fA-F0-9]+$', commit) else 'origin/%s' % commit
+                self._shellcmd('git fetch --all && git reset --hard "%s"' % ref, localdirname)
 
             # take the localdirname as poky dir if we can find the oe-init-build-env
             if self.pokydirname is None and os.path.exists(os.path.join(localdirname, "oe-init-build-env")):
-- 
2.1.4

[PATCH 4/6] toaster: fix reimporting module

[Ed Bartosh]

removed second 'import re' from localhostbecontroller.py

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 1 -
 1 file changed, 1 deletion(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 0c34957..c22d41b 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -181,7 +181,6 @@ class LocalhostBEController(BuildEnvironmentController):
     def getGitCloneDirectory(self, url, branch):
         """ Utility that returns the last component of a git path as directory
         """
-        import re
         components = re.split(r'[:\.\/]', url)
         base = components[-2] if components[-1] == "git" else components[-1]
 
-- 
2.1.4

[PATCH 5/6] toaster: make clone directory name unique

[Ed Bartosh]

Changed naming scheme for clone directory. Used full git url and branch to
make it unique. This should fix the issue with using the same git
repository, but different protocols, e.g. git://some.git.repo and
http://some.git.repo.

[YOCTO #8101]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index c22d41b..f826ad6 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -179,14 +179,9 @@ class LocalhostBEController(BuildEnvironmentController):
         logger.debug("localhostbecontroller: Stopped bitbake server")
 
     def getGitCloneDirectory(self, url, branch):
-        """ Utility that returns the last component of a git path as directory
-        """
-        components = re.split(r'[:\.\/]', url)
-        base = components[-2] if components[-1] == "git" else components[-1]
-
+        """Construct unique clone directory name out of url and branch."""
         if branch != "HEAD":
-            return "_%s_%s.toaster_cloned" % (base, branch)
-
+            return "_%s_%s.toaster_cloned" % (re.sub('[:/]', '_', url), branch)
 
         # word of attention; this is a localhost-specific issue; only on the localhost we expect to have "HEAD" releases
         # which _ALWAYS_ means the current poky checkout
-- 
2.1.4

[PATCH 6/6] toaster: move clones into subdirectory

[Ed Bartosh]

By default Toaster clones layers to the same level where
poky clone is. This can look messy if a lot of layers are
used for Toaster builds.

Moving them into _toaster_clones/ subdirectory should make
directory structure looking cleaner. It also safer to isolate
toaster clones from what user might create.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index f826ad6..d3528e8 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -181,7 +181,7 @@ class LocalhostBEController(BuildEnvironmentController):
     def getGitCloneDirectory(self, url, branch):
         """Construct unique clone directory name out of url and branch."""
         if branch != "HEAD":
-            return "_%s_%s.toaster_cloned" % (re.sub('[:/]', '_', url), branch)
+            return "_toaster_clones/_%s_%s" % (re.sub('[:/]', '_', url), branch)
 
         # word of attention; this is a localhost-specific issue; only on the localhost we expect to have "HEAD" releases
         # which _ALWAYS_ means the current poky checkout
-- 
2.1.4

Re: [PATCH 0/6] fixes and improvements for layers cloning

[Brian Avery]

upstreamed. better names, in 1 dir, better git commands, what's not to like :)

ty,
b

On Wed, Sep 23, 2015 at 9:52 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> Hi,
>
> This patchset includes changes for Toaster cloning functionality:
>  - used more appropriate git commands: git reset --hard instead of
>    git checkout && git rebase.
>  - changed naming of clone directories, made them unique
>  - moved clones to subdirectory
>
> Please, review.
>
> The following changes since commit 7b86c771c80d0759c2ca0e57c46c4c966f89c49e:
>
>   bitbake: bitbake: bb.fetch2.git: Import errno module (2015-09-19 22:38:44 +0100)
>
> are available in the git repository at:
>
>   git://git.yoctoproject.org/poky-contrib ed/toaster/clones
>   http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/clones
>
> Ed Bartosh (6):
>   toaster: don't use --single-branch when cloning
>   toaster: use git reset --hard instead of rebase
>   toaster: fix bug in resetting git repository
>   toaster: fix reimporting module
>   toaster: make clone directory name unique
>   toaster: move clones into subdirectory
>
>  bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 17 ++++++-----------
>  1 file changed, 6 insertions(+), 11 deletions(-)
>
> --
> Ed
>
> --
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster

[PATCH 0/6] fixes and improvements for layers cloning

[brian avery]

This patchset includes changes for Toaster cloning functionality:
 - used more appropriate git commands: git reset --hard instead of
   git checkout && git rebase.
 - changed naming of clone directories, made them unique
 - moved clones to subdirectory

The following changes since commit 524ddd86c305af1b029c1773affe5cebffc62b28:

  oeqa/utils/qemurunner.py: Remove duplicate message on LoggingThread start (2015-09-23 09:53:21 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib bavery/submit/ed/20150923-better-git-layers
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=bavery/submit/ed/20150923-better-git-layers

Ed Bartosh (6):
  toaster: don't use --single-branch when cloning
  toaster: use git reset --hard instead of rebase
  toaster: fix bug in resetting git repository
  toaster: fix reimporting module
  toaster: make clone directory name unique
  toaster: move clones into subdirectory

 lib/toaster/bldcontrol/localhostbecontroller.py | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

-- 
1.9.1