[PATCH 0/7] combo-layer enhancements

[PATCH 0/7] combo-layer enhancements

[Paul Eggleton]

A batch of improvements to the combo-layer tool.

The following changes since commit e3c003282afb93ec52882496400b042620ab00ef:

  rt-tests: update to 0.83 (2011-11-04 15:43:19 +0000)

are available in the git repository at:
  git://git.openembedded.org/openembedded-core-contrib paule/combo-layer-fixes2
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/combo-layer-fixes2

Paul Eggleton (7):
  scripts/combo-layer: handle merge commits
  scripts/combo-layer: make init set up initial component data
  scripts/combo-layer: tidy up config example
  scripts/combo-layer: make component repo branch configurable
  scripts/combo-layer: fix dirty repo check
  scripts/combo-layer: auto-commit updated config file
  scripts/combo-layer: improve some messages

 scripts/combo-layer              |  121 +++++++++++++++++---------------------
 scripts/combo-layer.conf.example |   28 +++++++--
 2 files changed, 75 insertions(+), 74 deletions(-)

-- 
1.7.5.4

[PATCH 1/7] scripts/combo-layer: handle merge commits

[Paul Eggleton]

The combo-layer tool had trouble handling merge commits as they were
included in the revision list but not the patches; these are now
excluded from the revision list.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/combo-layer |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/scripts/combo-layer b/scripts/combo-layer
index f8763ee..d6d238d 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -218,7 +218,7 @@ def action_update(conf, args):
         logger.debug("generated patch set:\n%s" % output)
         patchlist = output.splitlines()
 
-        rev_cmd = 'git log --pretty=format:"%H" ' + rev_cmd_range
+        rev_cmd = 'git rev-list --no-merges ' + rev_cmd_range
         revlist = runcmd(rev_cmd, ldir).splitlines()
 
         # Step 3: Call repo specific hook to adjust patch
-- 
1.7.5.4

[PATCH 2/7] scripts/combo-layer: make init set up initial component data

[Paul Eggleton]

The "init" subcommand will now copy the current state of each component
repository into the combo layer repository, set last_revision for each
component, and then use "git add" to set up the initial commit (but will
not actually make the initial commit - that is left up to the user).
Also take the opportunity to bump the version number.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/combo-layer              |   22 ++++++++++++++++++++--
 scripts/combo-layer.conf.example |    5 +++--
 2 files changed, 23 insertions(+), 4 deletions(-)

diff --git a/scripts/combo-layer b/scripts/combo-layer
index d6d238d..db38a95 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -26,7 +26,7 @@ import logging
 import subprocess
 import ConfigParser
 
-__version__ = "0.1.0"
+__version__ = "0.2.0"
 
 def logger_create():
     logger = logging.getLogger("")
@@ -146,7 +146,7 @@ def runcmd(cmd,destdir=None):
 def action_init(conf, args):
     """
         Clone component repositories
-        Check git initialised and working tree is clean
+        Check git is initialised; if not, copy initial data from component repos
     """
     for name in conf.repos:
         ldir = conf.repos[name]['local_repo_dir']
@@ -155,6 +155,24 @@ def action_init(conf, args):
             subprocess.check_call("git clone %s %s" % (conf.repos[name]['src_uri'], ldir), shell=True)
     if not os.path.exists(".git"):
         runcmd("git init")
+        for name in conf.repos:
+            ldir = conf.repos[name]['local_repo_dir']
+            logger.info("copying data from %s..." % name)
+            dest_dir = conf.repos[name]['dest_dir']
+            if dest_dir and dest_dir != ".":
+                extract_dir = os.path.join(os.getcwd(), dest_dir)
+                os.makedirs(extract_dir)
+            else:
+                extract_dir = os.getcwd()
+            file_filter = conf.repos[name].get('file_filter',"")
+            runcmd("git archive master | tar -x -C %s %s" % (extract_dir, file_filter), ldir)
+            lastrev = runcmd("git rev-parse HEAD", ldir).strip()
+            conf.update(name, "last_revision", lastrev)
+        runcmd("git add .")
+        logger.info("Initial combo layer repository data has been created; please make any changes if desired and then use 'git commit' to make the initial commit.")
+    else:
+        logger.info("Repository already initialised, nothing to do.")
+
 
 def check_repo_clean(repodir):
     """
diff --git a/scripts/combo-layer.conf.example b/scripts/combo-layer.conf.example
index 09b9415..8cee04f 100644
--- a/scripts/combo-layer.conf.example
+++ b/scripts/combo-layer.conf.example
@@ -11,8 +11,9 @@ local_repo_dir = /home/kyu3/src/test/bitbake
 # use "." if it is root dir
 dest_dir = bitbake
 
-# the updated revision last time.
-# If empty, the tool will start from the first commit
+# the last update revision.
+# "init" will set this automatically, however if it is empty when "update"
+# is run, the tool will start from the first commit.
 last_revision =
 
 # optional options
-- 
1.7.5.4

[PATCH 3/7] scripts/combo-layer: tidy up config example

[Paul Eggleton]

Move all example configuration to the example config file, tidy up a few
long lines and fix some grammar.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/combo-layer              |   47 +-------------------------------------
 scripts/combo-layer.conf.example |   23 ++++++++++++++----
 2 files changed, 19 insertions(+), 51 deletions(-)

diff --git a/scripts/combo-layer b/scripts/combo-layer
index db38a95..e39e4e0 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -42,52 +42,7 @@ class Configuration(object):
     """
     Manages the configuration
 
-    A valid conf looks like:
-
-# component name
-[bitbake]
-
-# mandatory options
-
-# git upstream uri
-src_uri = git://git.openembedded.org/bitbake
-
-# the directory to clone the component repo
-local_repo_dir = ~/src/bitbake
-
-# the relative dir to commit the repo patch
-# use "." if it is root dir
-dest_dir = bitbake
-
-# the updated revision last time.
-# leave it empty if no commit updated yet, and then the tool
-# will start from the first commit
-last_revision =
-
-# optional options
-
-# file_filter: only include the interested file
-# file_filter = [path] [path] ...
-# example:
-#   file_filter = src/  : only include the subdir src
-#   file_filter = src/*.c : only include the src *.c file
-#   file_filter = src/main.c src/Makefile.am : only include these two files
-
-[oe-core]
-src_uri = git://git.openembedded.org/openembedded-core
-local_repo_dir = ~/src/oecore
-dest_dir = .
-last_revision =
-
-# it's also possible to embed python code in the config values. Similar
-# to bitbake it considers every value starting with @ to be a python script. 
-# So local_repo could be easily configured using an environment variable as:
-#
-# [bitbake]
-# local_repo = @os.getenv("LOCAL_REPO_DIR") + "/bitbake"
-#
-
-# more components ...
+    For an example config file, see combo-layer.conf.example
 
     """
     def __init__(self, options):
diff --git a/scripts/combo-layer.conf.example b/scripts/combo-layer.conf.example
index 8cee04f..3865829 100644
--- a/scripts/combo-layer.conf.example
+++ b/scripts/combo-layer.conf.example
@@ -1,3 +1,5 @@
+# combo-layer example configuration file
+
 # component name
 [bitbake]
 # mandatory options
@@ -7,8 +9,8 @@ src_uri = git://git.openembedded.org/bitbake
 # the directory to clone the component repo
 local_repo_dir = /home/kyu3/src/test/bitbake
 
-# the relative dir to commit the repo patch
-# use "." if it is root dir
+# the relative dir within the combo repo to put the component files
+# use "." if the files should be in the root dir
 dest_dir = bitbake
 
 # the last update revision.
@@ -25,9 +27,10 @@ last_revision =
 #   file_filter = src/*.c : only include the src *.c file
 #   file_filter = src/main.c src/Makefile.am : only include these two files
 
-# hook: if provided, the tool will call the hook to process the generated patch from upstream,
-#       and then apply the modified patch to combo repo
-# the hook's parameter is: ./hook patchpath revision reponame
+# hook: if provided, the tool will call the hook to process the generated
+#     patch from upstream, and then apply the modified patch to the combo
+#     repo.
+# the hook script is called as follows: ./hook patchpath revision reponame
 # example:
 #     hook = combo-layer-hook-default.sh
 
@@ -36,3 +39,13 @@ src_uri = git://git.openembedded.org/openembedded-core
 local_repo_dir = /home/kyu3/src/test/oecore
 dest_dir = .
 last_revision =
+
+# It is also possible to embed python code in the config values. Similar
+# to bitbake it considers every value starting with @ to be a python
+# script.
+# e.g. local_repo_dir could easily be configured using an environment
+# variable:
+#
+# [bitbake]
+# local_repo_dir = @os.getenv("LOCAL_REPO_DIR") + "/bitbake"
+#
-- 
1.7.5.4

[PATCH 4/7] scripts/combo-layer: make component repo branch configurable

[Paul Eggleton]

Add an optional per-component branch setting to allow specifying the
branch instead of always using master.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/combo-layer |   22 ++++++++++++++--------
 1 files changed, 14 insertions(+), 8 deletions(-)

diff --git a/scripts/combo-layer b/scripts/combo-layer
index e39e4e0..4cb9ee0 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -108,19 +108,23 @@ def action_init(conf, args):
         if not os.path.exists(ldir):
             logger.info("cloning %s to %s" %(conf.repos[name]['src_uri'], ldir))
             subprocess.check_call("git clone %s %s" % (conf.repos[name]['src_uri'], ldir), shell=True)
+        branch = conf.repos[name].get('branch', "master")
+        runcmd("git checkout %s" % branch, ldir)
     if not os.path.exists(".git"):
         runcmd("git init")
         for name in conf.repos:
-            ldir = conf.repos[name]['local_repo_dir']
+            repo = conf.repos[name]
+            ldir = repo['local_repo_dir']
             logger.info("copying data from %s..." % name)
-            dest_dir = conf.repos[name]['dest_dir']
+            dest_dir = repo['dest_dir']
             if dest_dir and dest_dir != ".":
                 extract_dir = os.path.join(os.getcwd(), dest_dir)
                 os.makedirs(extract_dir)
             else:
                 extract_dir = os.getcwd()
-            file_filter = conf.repos[name].get('file_filter',"")
-            runcmd("git archive master | tar -x -C %s %s" % (extract_dir, file_filter), ldir)
+            branch = repo.get('branch', "master")
+            file_filter = repo.get('file_filter', "")
+            runcmd("git archive %s | tar -x -C %s %s" % (branch, extract_dir, file_filter), ldir)
             lastrev = runcmd("git rev-parse HEAD", ldir).strip()
             conf.update(name, "last_revision", lastrev)
         runcmd("git add .")
@@ -162,9 +166,11 @@ def action_update(conf, args):
         repo = conf.repos[name]
         ldir = repo['local_repo_dir']
         dest_dir = repo['dest_dir']
+        branch = repo.get('branch', "master")
         repo_patch_dir = os.path.join(os.getcwd(), patch_dir, name)
 
         # Step 1: update the component repo
+        runcmd("git checkout %s" % branch, ldir)
         logger.info("git pull for component repo %s in %s ..." % (name, ldir))
         output=runcmd("git pull", ldir)
         logger.info(output)
@@ -177,11 +183,11 @@ def action_update(conf, args):
             prefix = ""
         if repo['last_revision'] == "":
             logger.info("Warning: last_revision of component %s is not set, so start from the first commit" % name)
-            patch_cmd_range = "--root master"
-            rev_cmd_range = "master"
+            patch_cmd_range = "--root %s" % branch
+            rev_cmd_range = branch
         else:
-            patch_cmd_range = "%s..master" % repo['last_revision']
-            rev_cmd_range = "%s..master" % repo['last_revision']
+            patch_cmd_range = "%s..%s" % (repo['last_revision'], branch)
+            rev_cmd_range = patch_cmd_range
 
         file_filter = repo.get('file_filter',"")
 
-- 
1.7.5.4

[PATCH 5/7] scripts/combo-layer: fix dirty repo check

[Paul Eggleton]

Fix the dirty repository check to detect any changes, including
untracked files and uncommitted changes in the index.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/combo-layer |    8 ++------
 1 files changed, 2 insertions(+), 6 deletions(-)

diff --git a/scripts/combo-layer b/scripts/combo-layer
index 4cb9ee0..f3480dc 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -138,12 +138,8 @@ def check_repo_clean(repodir):
         check if the repo is clean
         exit if repo is dirty
     """
-    try:
-        runcmd("git diff --quiet", repodir)
-        #TODO: also check the index using "git diff --cached"
-        #      but this will fail in just initialized git repo
-        #      so need figure out a way
-    except:
+    output=runcmd("git status --porcelain", repodir)
+    if output:
         logger.error("git repo %s is dirty, please fix it first", repodir)
         sys.exit(1)
 
-- 
1.7.5.4

[PATCH 6/7] scripts/combo-layer: auto-commit updated config file

[Paul Eggleton]

If the config file is tracked within the combo-layer repository and it
is updated at the end of the "update" operation (because last_revision
has been changed), then automatically commit the file. This ensures that
multiple people can perform updates on different machines without the
last revision information going missing.

(If the file is outside the repository or is masked via .gitignore, this
will do nothing.)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/combo-layer |   12 ++++++++++++
 1 files changed, 12 insertions(+), 0 deletions(-)

diff --git a/scripts/combo-layer b/scripts/combo-layer
index f3480dc..64ea9a9 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -230,6 +230,18 @@ def action_update(conf, args):
     action_apply_patch(conf, args)
     runcmd("rm -rf %s" % patch_dir)
 
+    # Step 7: commit the updated config file if it's being tracked
+    relpath = os.path.relpath(conf.conffile)
+    try:
+        output = runcmd("git status --porcelain %s" % relpath)
+    except:
+        # Outside the repository
+        output = None
+    if output:
+        logger.info("Committing updated configuration file")
+        if output.lstrip().startswith("M"):
+            runcmd('git commit --author="combo-layer <combo-layer@localhost>" -m "Automatic commit to update last_revision" %s' % relpath)
+
 def action_apply_patch(conf, args):
     """
         apply the generated patch list to combo repo
-- 
1.7.5.4

[PATCH 7/7] scripts/combo-layer: improve some messages

[Paul Eggleton]

Fix some grammar in output messages.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/combo-layer |   16 ++++++++--------
 1 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/scripts/combo-layer b/scripts/combo-layer
index 64ea9a9..fbf77e6 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -71,7 +71,7 @@ class Configuration(object):
                 if option not in self.repos[name]:
                     msg = "%s\nOption %s is not defined for component %s" %(msg, option, name)
         if msg != "":
-            logger.error("configuration file %s has the following error:%s" % (self.conffile,msg))
+            logger.error("configuration file %s has the following error: %s" % (self.conffile,msg))
             sys.exit(1)
 
         # filterdiff is required by action_splitpatch, so check its availability
@@ -145,7 +145,7 @@ def check_repo_clean(repodir):
 
 def action_update(conf, args):
     """
-        update the component repo
+        update the component repos
         generate the patch list
         apply the generated patches
     """
@@ -178,7 +178,7 @@ def action_update(conf, args):
         else:
             prefix = ""
         if repo['last_revision'] == "":
-            logger.info("Warning: last_revision of component %s is not set, so start from the first commit" % name)
+            logger.info("Warning: last_revision of component %s is not set, starting from the first commit" % name)
             patch_cmd_range = "--root %s" % branch
             rev_cmd_range = branch
         else:
@@ -218,12 +218,12 @@ def action_update(conf, args):
     if conf.interactive:
         print   'Edit the patch and patch list in %s\n' \
                 'For example, remove the unwanted patch entry from patchlist-*, so that it will be not applied later\n' \
-                'After finish, press following command to continue\n' \
+                'When you are finished, run the following to continue:\n' \
                 '       exit 0  -- exit and continue to apply the patch\n' \
-                '       exit 1  -- abort and not apply patch\n' % patch_dir
+                '       exit 1  -- abort and do not apply the patch\n' % patch_dir
         ret = subprocess.call(["bash"], cwd=patch_dir)
         if ret != 0:
-            print "Abort without applying patch"
+            print "Aborting without applying the patch"
             sys.exit(0)
 
     # Step 6: apply the generated and revised patch
@@ -257,11 +257,11 @@ def action_apply_patch(conf, args):
             try:
                 runcmd(cmd)
             except subprocess.CalledProcessError:
-                logger.info('"git am --abort" is executed to cleanup repo')
+                logger.info('running "git am --abort" to cleanup repo')
                 runcmd("git am --abort")
                 logger.error('"%s" failed' % cmd)
                 logger.info("please manually apply patch %s" % patchfile)
-                logger.info("After applying, run this tool again to apply the rest patches")
+                logger.info("After applying, run this tool again to apply the remaining patches")
                 conf.update(name, "last_revision", lastrev)
                 sys.exit(0)
         conf.update(name, "last_revision", lastrev)
-- 
1.7.5.4

Re: [PATCH 0/7] combo-layer enhancements

[Otavio Salvador]

On Fri, Nov 4, 2011 at 15:55, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
...
>  scripts/combo-layer: auto-commit updated config file
...

As said on IRC I'd prefer to the auto-commit do not replace the author
as it gives the possibility to easily figure who did the update on the
repository.

-- 
Otavio Salvador                             O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

Re: [PATCH 0/7] combo-layer enhancements

[Paul Eggleton]

On Friday 04 November 2011 16:15:59 Otavio Salvador wrote:
> As said on IRC I'd prefer to the auto-commit do not replace the author
> as it gives the possibility to easily figure who did the update on the
> repository.

OK, I've removed the override of the author and re-pushed the branch.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/7] combo-layer enhancements

[Koen Kooi]

[-- Attachment #1: Type: text/plain, Size: 517 bytes --]


Op 7 nov. 2011, om 11:14 heeft Paul Eggleton het volgende geschreven:

> On Friday 04 November 2011 16:15:59 Otavio Salvador wrote:
>> As said on IRC I'd prefer to the auto-commit do not replace the author
>> as it gives the possibility to easily figure who did the update on the
>> repository.
> 
> OK, I've removed the override of the author and re-pushed the branch.

With all the fixes in your branch does this allow creating a combo layer from scratch with all the history present?

regards,

Koen

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 169 bytes --]

Re: [PATCH 0/7] combo-layer enhancements

[Paul Eggleton]

On Monday 07 November 2011 11:16:26 Koen Kooi wrote:
> With all the fixes in your branch does this allow creating a combo layer
> from scratch with all the history present?

If you expect that it will do that automatically with no conflict resolution 
necessary, then the answer is no I'm afraid. My focus at least for one of 
these patches was to get repo creation starting from the current revision 
working with no hassle. Testing an import of the entire history just now 
though did point out a trivial issue with empty commits which I have now fixed, 
however, "git format-patch" (which is the basis for combo-layer) simply cannot 
handle merges where a conflict was resolved, and AFAICT there is no way around 
this. Typically we don't do merges in the OE-core or bitbake repositories so 
this is not too serious an issue; however, there is at least one merge in the 
OE-core history where there are changes that conflict (an example would be 
76e51a83e1af6c95a8ba780d8d543ddda52e1597 and 
3937aefbefda15a65436238a6c6f1485419c914b). When it gets to these you will have 
to resolve the conflict by hand.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre