[PATCH 0/2] fixes for combo-layer update history mode

[PATCH 0/2] fixes for combo-layer update history mode

[Patrick Ohly]

The first fix is urgent: the default mode was accidentally changed
from the traditional "apply patches" to "import the history".

The second fix is less urgent but still useful for those who
actually use the new mode.

The following changes since commit fcaaabb401fffcda4db9a7d1f927a2a404e4776d:

  gcc-runtime, libgcc: Symlink c++ header and startup files in target_triplet for SDK use (2016-05-13 13:40:53 +0100)

are available in the git repository at:

  git://github.com/pohly/openembedded-core combo-layer-history-fixes
  https://github.com/pohly/openembedded-core/tree/combo-layer-history-fixes

Patrick Ohly (2):
  combo-layer: fix default "update" mode
  combo-layer: avoid too long command lines in update with history

 scripts/combo-layer | 34 ++++++++++++++++++++++++++++++----
 1 file changed, 30 insertions(+), 4 deletions(-)

-- 
2.1.4

[PATCH 1/2] combo-layer: fix default "update" mode

[Patrick Ohly]

When the "history" option is not set in the combo-layer.conf, the
intended default was to use the traditional method. Passing "True" as
default when querying the config was unintentional.

Also remove some left-over debugging code.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 scripts/combo-layer | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/scripts/combo-layer b/scripts/combo-layer
index 92525ca..a0a737d 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -641,8 +641,7 @@ def action_update(conf, args):
         history = None
         for name in repos:
             repo = conf.repos[name]
-            repo_history = repo.get('history', True)
-            logger.error('%s: %s' % (name, repo_history))
+            repo_history = repo.get('history', False)
             if history is None:
                 history = repo_history
             elif history != repo_history:
-- 
2.1.4

[PATCH 2/2] combo-layer: avoid too long command lines in update with history

[Patrick Ohly]

As suspected, invoking "git archive" with all intended files as
parameters can run into command line length limitations. Splitting up
the parameters into multiple invocations (xargs-style) works and was
tested after encountering the situation in practice.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 scripts/combo-layer | 31 +++++++++++++++++++++++++++++--
 1 file changed, 29 insertions(+), 2 deletions(-)

diff --git a/scripts/combo-layer b/scripts/combo-layer
index a0a737d..1ca2ce6 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -1266,8 +1266,35 @@ def apply_commit(parent, rev, largs, wargs, dest_dir, file_filter=None):
         target = os.path.join(wargs["destdir"], dest_dir)
         if not os.path.isdir(target):
             os.makedirs(target)
-        runcmd("git archive %s %s | tar -C %s -xf -" % (rev, ' '.join([pipes.quote(x) for x in update]), pipes.quote(target)), **largs)
-        runcmd("git add -f".split() + [os.path.join(dest_dir, x) for x in update], **wargs)
+        quoted_target = pipes.quote(target)
+        # os.sysconf('SC_ARG_MAX') is lying: running a command with
+        # string length 629343 already failed with "Argument list too
+        # long" although SC_ARG_MAX = 2097152. "man execve" explains
+        # the limitations, but those are pretty complicated. So here
+        # we just hard-code a fixed value which is more likely to work.
+        max_cmdsize = 64 * 1024
+        while update:
+            quoted_args = []
+            unquoted_args = []
+            cmdsize = 100 + len(quoted_target)
+            while update:
+                quoted_next = pipes.quote(update[0])
+                size_next = len(quoted_next) + len(dest_dir) + 1
+                logger.debug('cmdline length %d + %d < %d?' % (cmdsize, size_next, os.sysconf('SC_ARG_MAX')))
+                if cmdsize + size_next < max_cmdsize:
+                    quoted_args.append(quoted_next)
+                    unquoted_args.append(update.pop(0))
+                    cmdsize += size_next
+                else:
+                    logger.debug('Breaking the cmdline at length %d' % cmdsize)
+                    break
+            logger.debug('Final cmdline length %d / %d' % (cmdsize, os.sysconf('SC_ARG_MAX')))
+            cmd = "git archive %s %s | tar -C %s -xf -" % (rev, ' '.join(quoted_args), quoted_target)
+            logger.debug('First cmdline length %d' % len(cmd))
+            runcmd(cmd, **largs)
+            cmd = "git add -f".split() + [os.path.join(dest_dir, x) for x in unquoted_args]
+            logger.debug('Second cmdline length %d' % reduce(lambda x, y: x + len(y), cmd, 0))
+            runcmd(cmd, **wargs)
     if delete:
         for path in delete:
             if dest_dir:
-- 
2.1.4