[PATCH 00/11] Fixes/enhancements for devtool/recipetool/externalsrc

[PATCH 00/11] Fixes/enhancements for devtool/recipetool/externalsrc

[Paul Eggleton]

Note in particular patch 6/11 which changes the behaviour of devtool
modify without -x to extract by default which is a bit more logical (I'd
been thinking about this for a while, but Khem suggested it most
recently).


The following changes since commit 4bd658f7dae76ae8790fd1dfdd89fa58a456a0a4:

  pseudo: uprev to 1.7.5 (2016-02-10 17:37:36 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/devtool13
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool13

Paul Eggleton (11):
  classes/externalsrc: disable rm_work when active
  classes/externalsrc: create symlinks for workdir and logs
  devtool: commit for extra tasks that modify source when extracting
  recipetool: create: support cmake find_library directive
  recipetool: create: determine name/version from github/bitbucket URLs
  devtool: modify: make -x the default behaviour
  devtool: add: fix adding from a local source directory
  recipetool: create: ensure URL parameters don't make it into the name
  recipetool: create: convert http git URLs that don't end in .git but contain /git/
  recipetool: create: set S when we set SRC_URI from local git repo
  oe-selftest: devtool: add another devtool add test

 meta/classes/externalsrc.bbclass          | 23 +++++++++++
 meta/lib/oe/patch.py                      |  3 ++
 meta/lib/oeqa/selftest/devtool.py         | 40 ++++++++++++++++++-
 scripts/lib/devtool/standard.py           | 64 +++++++++++++++++++++----------
 scripts/lib/recipetool/create.py          | 55 +++++++++++++++++++-------
 scripts/lib/recipetool/create_buildsys.py | 10 +++++
 6 files changed, 161 insertions(+), 34 deletions(-)

-- 
2.5.0

[PATCH 01/11] classes/externalsrc: disable rm_work when active

[Paul Eggleton]

If you're using externalsrc, it's very likely that you're going to want
to examine the intermediate build results even if the recipe builds
successfully; therefore you won't want rm_work to delete those.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/externalsrc.bbclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 607861f..9d7ab00 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -86,6 +86,9 @@ python () {
         # Ensure compilation happens every time
         d.setVarFlag('do_compile', 'nostamp', '1')
 
+        # We don't want the workdir to go away
+        d.appendVar('RM_WORK_EXCLUDE', ' ' + d.getVar('PN', True))
+
         # If B=S the same builddir is used even for different architectures.
         # Thus, use a shared CONFIGURESTAMPFILE so that change of do_configure
         # task hash is correctly detected if e.g. MACHINE changes. In addition,
-- 
2.5.0

[PATCH 02/11] classes/externalsrc: create symlinks for workdir and logs

[Paul Eggleton]

Auto-create symlinks in the source directory to the work directory
(${WORKDIR}) and logs directory (${T}) so that they are easier for the
user to find. This is particularly useful within the extensible SDK
where the user is less likely to be familiar enough with the structure
of the build system to know where to find these things, but otherwise
they are a useful shortcut for anyone.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/externalsrc.bbclass | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 9d7ab00..b608bd0 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -25,6 +25,7 @@
 #
 
 SRCTREECOVEREDTASKS ?= "do_patch do_unpack do_fetch"
+EXTERNALSRC_SYMLINKS ?= "oe-workdir:${WORKDIR} oe-logs:${T}"
 
 python () {
     externalsrc = d.getVar('EXTERNALSRC', True)
@@ -82,6 +83,7 @@ python () {
             bb.build.deltask(task, d)
 
         d.prependVarFlag('do_compile', 'prefuncs', "externalsrc_compile_prefunc ")
+        d.prependVarFlag('do_configure', 'prefuncs', "externalsrc_configure_prefunc ")
 
         # Ensure compilation happens every time
         d.setVarFlag('do_compile', 'nostamp', '1')
@@ -101,6 +103,24 @@ python () {
             d.setVarFlag('do_configure', 'file-checksums', configstamp + ':True')
 }
 
+python externalsrc_configure_prefunc() {
+    # Create desired symlinks
+    symlinks = (d.getVar('EXTERNALSRC_SYMLINKS', True) or '').split()
+    for symlink in symlinks:
+        symsplit = symlink.split(':', 1)
+        lnkfile = os.path.join(d.getVar('S', True), symsplit[0])
+        target = d.expand(symsplit[1])
+        if len(symsplit) > 1:
+            if os.path.islink(lnkfile):
+                # Link already exists, leave it if it points to the right location already
+                if os.readlink(lnkfile) == target:
+                    continue
+            elif os.path.exists(lnkfile):
+                # File/dir exists with same name as link, just leave it alone
+                continue
+            os.symlink(target, lnkfile)
+}
+
 python externalsrc_compile_prefunc() {
     # Make it obvious that this is happening, since forgetting about it could lead to much confusion
     bb.plain('NOTE: %s: compiling from external source tree %s' % (d.getVar('PN', True), d.getVar('EXTERNALSRC', True)))
-- 
2.5.0

[PATCH 03/11] devtool: commit for extra tasks that modify source when extracting

[Paul Eggleton]

When extracting source for a recipe, if there are additional custom
tasks run that make changes to the source, create a commit in the
generated git branch so they are contained. This is particularly
useful for tasks that come before do_patch since otherwise the changes
might get incorporated in the first applied patch, but otherwise it
helps avoid the tree being dirty at any point.

Fixes [YOCTO #7626].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/patch.py            |  3 +++
 scripts/lib/devtool/standard.py | 48 ++++++++++++++++++++++++++++++-----------
 2 files changed, 38 insertions(+), 13 deletions(-)

diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
index 4e77168..2464efd 100644
--- a/meta/lib/oe/patch.py
+++ b/meta/lib/oe/patch.py
@@ -264,6 +264,7 @@ class PatchTree(PatchSet):
 
 class GitApplyTree(PatchTree):
     patch_line_prefix = '%% original patch'
+    ignore_commit_prefix = '%% ignore'
 
     def __init__(self, dir, d):
         PatchTree.__init__(self, dir, d)
@@ -384,6 +385,8 @@ class GitApplyTree(PatchTree):
                             if line.startswith(GitApplyTree.patch_line_prefix):
                                 outfile = line.split()[-1].strip()
                                 continue
+                            if line.startswith(GitApplyTree.ignore_commit_prefix):
+                                continue
                             patchlines.append(line)
                     if not outfile:
                         outfile = os.path.basename(srcfile)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 390d98f..262ba09 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -393,6 +393,38 @@ class BbTaskExecutor(object):
             self.executed.append(func)
 
 
+class PatchTaskExecutor(BbTaskExecutor):
+    def __init__(self, rdata):
+        self.check_git = False
+        super(PatchTaskExecutor, self).__init__(rdata)
+
+    def exec_func(self, func, report):
+        from oe.patch import GitApplyTree
+        srcsubdir = self.rdata.getVar('S', True)
+        haspatches = False
+        if func == 'do_patch':
+            patchdir = os.path.join(srcsubdir, 'patches')
+            if os.path.exists(patchdir):
+                if os.listdir(patchdir):
+                    haspatches = True
+                else:
+                    os.rmdir(patchdir)
+
+        super(PatchTaskExecutor, self).exec_func(func, report)
+        if self.check_git and os.path.exists(srcsubdir):
+            if func == 'do_patch':
+                if os.path.exists(patchdir):
+                    shutil.rmtree(patchdir)
+                    if haspatches:
+                        stdout, _ = bb.process.run('git status --porcelain patches', cwd=srcsubdir)
+                        if stdout:
+                            bb.process.run('git checkout patches', cwd=srcsubdir)
+
+            stdout, _ = bb.process.run('git status --porcelain', cwd=srcsubdir)
+            if stdout:
+                bb.process.run('git add .; git commit -a -m "Committing changes from %s\n\n%s"' % (func, GitApplyTree.ignore_commit_prefix + ' - from %s' % func), cwd=srcsubdir)
+
+
 def _prep_extract_operation(config, basepath, recipename):
     """HACK: Ugly workaround for making sure that requirements are met when
        trying to extract a package. Returns the tinfoil instance to be used."""
@@ -477,7 +509,7 @@ def _extract_source(srctree, keep_temp, devbranch, sync, d):
             # We don't want to move the source to STAGING_KERNEL_DIR here
             crd.setVar('STAGING_KERNEL_DIR', '${S}')
 
-        task_executor = BbTaskExecutor(crd)
+        task_executor = PatchTaskExecutor(crd)
 
         crd.setVar('EXTERNALSRC_forcevariable', '')
 
@@ -491,6 +523,8 @@ def _extract_source(srctree, keep_temp, devbranch, sync, d):
             task_executor.exec_func('do_kernel_checkout', False)
         srcsubdir = crd.getVar('S', True)
 
+        task_executor.check_git = True
+
         # Move local source files into separate subdir
         recipe_patches = [os.path.basename(patch) for patch in
                           oe.recipeutils.get_recipe_patches(crd)]
@@ -524,13 +558,6 @@ def _extract_source(srctree, keep_temp, devbranch, sync, d):
 
         scriptutils.git_convert_standalone_clone(srcsubdir)
 
-        patchdir = os.path.join(srcsubdir, 'patches')
-        haspatches = False
-        if os.path.exists(patchdir):
-            if os.listdir(patchdir):
-                haspatches = True
-            else:
-                os.rmdir(patchdir)
         # Make sure that srcsubdir exists
         bb.utils.mkdirhier(srcsubdir)
         if not os.path.exists(srcsubdir) or not os.listdir(srcsubdir):
@@ -550,11 +577,6 @@ def _extract_source(srctree, keep_temp, devbranch, sync, d):
 
         bb.process.run('git tag -f devtool-patched', cwd=srcsubdir)
 
-        if os.path.exists(patchdir):
-            shutil.rmtree(patchdir)
-            if haspatches:
-                bb.process.run('git checkout patches', cwd=srcsubdir)
-
         if bb.data.inherits_class('kernel-yocto', d):
             # Store generate and store kernel config
             logger.info('Generating kernel config')
-- 
2.5.0

[PATCH 04/11] recipetool: create: support cmake find_library directive

[Paul Eggleton]

CMake supports a find_library() directive to find named libraries, so
detect dependencies from this.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create_buildsys.py | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/scripts/lib/recipetool/create_buildsys.py b/scripts/lib/recipetool/create_buildsys.py
index ba393a8..4d11e04 100644
--- a/scripts/lib/recipetool/create_buildsys.py
+++ b/scripts/lib/recipetool/create_buildsys.py
@@ -137,6 +137,7 @@ class CmakeRecipeHandler(RecipeHandler):
         pkgcm_re = re.compile('pkg_check_modules\s*\(\s*[a-zA-Z0-9-_]+\s*(REQUIRED)?\s+([^)\s]+)\s*\)', re.IGNORECASE)
         pkgsm_re = re.compile('pkg_search_module\s*\(\s*[a-zA-Z0-9-_]+\s*(REQUIRED)?((\s+[^)\s]+)+)\s*\)', re.IGNORECASE)
         findpackage_re = re.compile('find_package\s*\(\s*([a-zA-Z0-9-_]+)\s*.*', re.IGNORECASE)
+        findlibrary_re = re.compile('find_library\s*\(\s*[a-zA-Z0-9-_]+\s*(NAMES\s+)?([a-zA-Z0-9-_ ]+)\s*.*')
         checklib_re = re.compile('check_library_exists\s*\(\s*([^\s)]+)\s*.*', re.IGNORECASE)
         include_re = re.compile('include\s*\(\s*([^)\s]*)\s*\)', re.IGNORECASE)
         subdir_re = re.compile('add_subdirectory\s*\(\s*([^)\s]*)\s*([^)\s]*)\s*\)', re.IGNORECASE)
@@ -215,6 +216,15 @@ class CmakeRecipeHandler(RecipeHandler):
                         lib = interpret_value(res.group(1))
                         if not lib.startswith('$'):
                             libdeps.append(lib)
+                    res = findlibrary_re.match(line)
+                    if res:
+                        libs = res.group(2).split()
+                        for lib in libs:
+                            if lib in ['HINTS', 'PATHS', 'PATH_SUFFIXES', 'DOC', 'NAMES_PER_DIR'] or lib.startswith(('NO_', 'CMAKE_', 'ONLY_CMAKE_')):
+                                break
+                            lib = interpret_value(lib)
+                            if not lib.startswith('$'):
+                                libdeps.append(lib)
                     if line.lower().startswith('useswig'):
                         deps.append('swig-native')
                         continue
-- 
2.5.0

[PATCH 05/11] recipetool: create: determine name/version from github/bitbucket URLs

[Paul Eggleton]

Both BitBucket and GitHub provide "release" downloads that unfortunately
don't include the actual project name. For recipetool this means we have
to have special handling for these URLs or else the current code assumes
the name of the project is something like "v0.9.1". Borrow and adapt
some code from autospec to do this, and put it in its own function for
clarity.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create.py | 44 ++++++++++++++++++++++++++++++++--------
 1 file changed, 35 insertions(+), 9 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 5f90b10..ee27f8d 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -247,6 +247,35 @@ def determine_from_filename(srcfile):
         pv = None
     return (pn, pv)
 
+def determine_from_url(srcuri):
+    """Determine name and version from a URL"""
+    pn = None
+    pv = None
+    parseres = urlparse.urlparse(srcuri.lower())
+    if parseres.path:
+        if 'github.com' in parseres.netloc:
+            res = re.search(r'.*/(.*?)/archive/(.*)-final\.(tar|zip)', parseres.path)
+            if res:
+                pn = res.group(1).strip().replace('_', '-')
+                pv = res.group(2).strip().replace('_', '.')
+            else:
+                res = re.search(r'.*/(.*?)/archive/v?(.*)\.(tar|zip)', parseres.path)
+                if res:
+                    pn = res.group(1).strip().replace('_', '-')
+                    pv = res.group(2).strip().replace('_', '.')
+        elif 'bitbucket.org' in parseres.netloc:
+            res = re.search(r'.*/(.*?)/get/[a-zA-Z_-]*([0-9][0-9a-zA-Z_.]*)\.(tar|zip)', parseres.path)
+            if res:
+                pn = res.group(1).strip().replace('_', '-')
+                pv = res.group(2).strip().replace('_', '.')
+
+        if not pn and not pv:
+            srcfile = os.path.basename(parseres.path.rstrip('/'))
+            pn, pv = determine_from_filename(srcfile)
+
+    logger.debug('Determined from source URL: name = "%s", version = "%s"' % (pn, pv))
+    return (pn, pv)
+
 def supports_srcrev(uri):
     localdata = bb.data.createCopy(tinfoil.config_data)
     # This is a bit sad, but if you don't have this set there can be some
@@ -430,15 +459,12 @@ def create_recipe(args):
         realpv = None
 
     if srcuri and not realpv or not pn:
-        parseres = urlparse.urlparse(srcuri)
-        if parseres.path:
-            srcfile = os.path.basename(parseres.path.rstrip('/'))
-            name_pn, name_pv = determine_from_filename(srcfile)
-            logger.debug('Determined from filename: name = "%s", version = "%s"' % (name_pn, name_pv))
-            if name_pn and not pn:
-                pn = name_pn
-            if name_pv and not realpv:
-                realpv = name_pv
+        name_pn, name_pv = determine_from_url(srcuri)
+        if name_pn and not pn:
+            pn = name_pn
+        if name_pv and not realpv:
+            realpv = name_pv
+
 
     if not srcuri:
         lines_before.append('# No information for SRC_URI yet (only an external source tree was specified)')
-- 
2.5.0

[PATCH 06/11] devtool: modify: make -x the default behaviour

[Paul Eggleton]

It's going to be more common for users not to have the prepared source
tree for a recipe already, so the default behaviour ought to be to
extract it for them from the recipe. Change the default to extract
(effectively making the -x option a no-op) and add a --no-extract/-n
option to disable it. Later we can look at trying to be smart and
reusing an existing source tree instead of erroring out if it exists;
for now this is just the default reversal.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oeqa/selftest/devtool.py |  3 ++-
 scripts/lib/devtool/standard.py   | 14 ++++++++------
 2 files changed, 10 insertions(+), 7 deletions(-)

diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index d95cb08..345cabb 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -570,7 +570,8 @@ class DevtoolTests(DevtoolBase):
         self.track_for_cleanup(self.workspacedir)
         self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
         # (don't bother with cleaning the recipe on teardown, we won't be building it)
-        result = runCmd('devtool modify %s -x %s' % (testrecipe, tempdir))
+        # We don't use -x here so that we test the behaviour of devtool modify without it
+        result = runCmd('devtool modify %s %s' % (testrecipe, tempdir))
         # Check git repo
         self._check_src_repo(tempdir)
         # Add a couple of commits
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 262ba09..f1b2e12 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -685,11 +685,11 @@ def modify(args, config, basepath, workspace):
     else:
         srctree = get_default_srctree(config, args.recipename)
 
-    if not args.extract and not os.path.isdir(srctree):
-        raise DevtoolError("directory %s does not exist or not a directory "
-                           "(specify -x to extract source from recipe)" %
+    if args.no_extract and not os.path.isdir(srctree):
+        raise DevtoolError("--no-extract specified and source path %s does "
+                           "not exist or is not a directory" %
                            srctree)
-    if args.extract:
+    if not args.no_extract:
         tinfoil = _prep_extract_operation(config, basepath, args.recipename)
         if not tinfoil:
             # Error already shown
@@ -720,7 +720,7 @@ def modify(args, config, basepath, workspace):
 
     initial_rev = None
     commits = []
-    if args.extract:
+    if not args.no_extract:
         initial_rev = _extract_source(srctree, False, args.branch, False, rd)
         if not initial_rev:
             return 1
@@ -1319,7 +1319,9 @@ def register_commands(subparsers, context):
     parser_modify.add_argument('recipename', help='Name of existing recipe to edit (just name - no version, path or extension)')
     parser_modify.add_argument('srctree', nargs='?', help='Path to external source tree. If not specified, a subdirectory of %s will be used.' % defsrctree)
     parser_modify.add_argument('--wildcard', '-w', action="store_true", help='Use wildcard for unversioned bbappend')
-    parser_modify.add_argument('--extract', '-x', action="store_true", help='Extract source as well')
+    group = parser_modify.add_mutually_exclusive_group()
+    group.add_argument('--extract', '-x', action="store_true", help='Extract source for recipe (default)')
+    group.add_argument('--no-extract', '-n', action="store_true", help='Do not extract source, expect it to exist')
     group = parser_modify.add_mutually_exclusive_group()
     group.add_argument('--same-dir', '-s', help='Build in same directory as source', action="store_true")
     group.add_argument('--no-same-dir', help='Force build in a separate build directory', action="store_true")
-- 
2.5.0

[PATCH 07/11] devtool: add: fix adding from a local source directory

[Paul Eggleton]

Fix a regression introduced in in OE-Core revision
aedfc5a5db1c4b2b80a36147c9a13b31764d91dd where specifying a local source
tree without specifying a name resulted in a traceback.

Fixes [YOCTO #9086].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index f1b2e12..d12cc2e 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -51,7 +51,7 @@ def add(args, config, basepath, workspace):
                 raise DevtoolError('URI specified as positional argument as well as -f/--fetch')
             args.fetchuri = args.recipename
             args.recipename = ''
-    elif '://' in args.srctree:
+    elif args.srctree and '://' in args.srctree:
         if not args.fetchuri:
             if args.fetch:
                 raise DevtoolError('URI specified as positional argument as well as -f/--fetch')
-- 
2.5.0

[PATCH 08/11] recipetool: create: ensure URL parameters don't make it into the name

[Paul Eggleton]

When auto-detecting the name for a recipe from the URL, strip off any
parameters (";name=value...") before parsing the URL, otherwise this
just ends up in the recipe name which is undesirable.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index ee27f8d..f6d7515 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -251,7 +251,7 @@ def determine_from_url(srcuri):
     """Determine name and version from a URL"""
     pn = None
     pv = None
-    parseres = urlparse.urlparse(srcuri.lower())
+    parseres = urlparse.urlparse(srcuri.lower().split(';', 1)[0])
     if parseres.path:
         if 'github.com' in parseres.netloc:
             res = re.search(r'.*/(.*?)/archive/(.*)-final\.(tar|zip)', parseres.path)
-- 
2.5.0

[PATCH 09/11] recipetool: create: convert http git URLs that don't end in .git but contain /git/

[Paul Eggleton]

When recipetool create is given a URL that starts with http(s):// and
contains /git/, such as the URLs at git.yoctoproject.org, it's fairly safe
to assume it's a git repository and not something that should be fetched
with wget, so rewrite the URL.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create.py | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index f6d7515..daf13fb 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -291,10 +291,12 @@ def supports_srcrev(uri):
 
 def reformat_git_uri(uri):
     '''Convert any http[s]://....git URI into git://...;protocol=http[s]'''
-    res = re.match('(https?)://([^;]+\.git)(;.*)?$', uri)
-    if res:
-        # Need to switch the URI around so that the git fetcher is used
-        return 'git://%s;protocol=%s%s' % (res.group(2), res.group(1), res.group(3) or '')
+    checkuri = uri.split(';', 1)[0]
+    if checkuri.endswith('.git') or '/git/' in checkuri:
+        res = re.match('(https?)://([^;]+(\.git)?)(;.*)?$', uri)
+        if res:
+            # Need to switch the URI around so that the git fetcher is used
+            return 'git://%s;protocol=%s%s' % (res.group(2), res.group(1), res.group(4) or '')
     return uri
 
 def create_recipe(args):
-- 
2.5.0

[PATCH 10/11] recipetool: create: set S when we set SRC_URI from local git repo

[Paul Eggleton]

If you specify a local directory which happens to be a git repository
with an origin remote (and it is in fact remote), we can use that for
SRC_URI as implemented by OE-Core revision
b143d414846854dc8b3e0a47358daf5646eded38, however we also need to set S
if the recipe is going to be of any use fetching from that SRC_URI
later.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index daf13fb..3e4bab8 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -369,6 +369,7 @@ def create_recipe(args):
                     if len(splitline) > 1:
                         if splitline[0] == 'origin' and '://' in splitline[1]:
                             srcuri = reformat_git_uri(splitline[1])
+                            srcsubdir = 'git'
                             break
 
     if args.src_subdir:
-- 
2.5.0

[PATCH 11/11] oe-selftest: devtool: add another devtool add test

[Paul Eggleton]

Add a test so we validate the following recently implemented/fixed
aspects of "devtool add":

* Adding from a local directory with no name specified
* Auto-detecting the name and version from autoconf files
* Setting SRC_URI and S from the local git repo
* Showing the recipe file path in "devtool status" if it has been
  created within the workspace

Incidentally also tests:

* LICENSE and LIC_FILES_CHKSUM detection (though just for this piece of
  software)
* Extracting a dependency from autoconf (though just for this narrow
  case since there's only one)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oeqa/selftest/devtool.py | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index 345cabb..e5a2134 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -201,6 +201,43 @@ class DevtoolTests(DevtoolBase):
             bindir = bindir[1:]
         self.assertTrue(os.path.isfile(os.path.join(installdir, bindir, 'pv')), 'pv binary not found in D')
 
+    def test_devtool_add_git_local(self):
+        # Fetch source from a remote URL, but do it outside of devtool
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        pn = 'dbus-wait'
+        # We choose an https:// git URL here to check rewriting the URL works
+        url = 'https://git.yoctoproject.org/git/dbus-wait'
+        # Force fetching to "noname" subdir so we verify we're picking up the name from autoconf
+        # instead of the directory name
+        result = runCmd('git clone %s noname' % url, cwd=tempdir)
+        srcdir = os.path.join(tempdir, 'noname')
+        self.assertTrue(os.path.isfile(os.path.join(srcdir, 'configure.ac')), 'Unable to find configure script in source directory')
+        # Test devtool add
+        self.track_for_cleanup(self.workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        # Don't specify a name since we should be able to auto-detect it
+        result = runCmd('devtool add %s' % srcdir)
+        self.assertTrue(os.path.exists(os.path.join(self.workspacedir, 'conf', 'layer.conf')), 'Workspace directory not created')
+        # Check the recipe name is correct
+        recipefile = get_bb_var('FILE', pn)
+        self.assertIn('%s_git.bb' % pn, recipefile, 'Recipe file incorrectly named')
+        self.assertIn(recipefile, result.output)
+        # Test devtool status
+        result = runCmd('devtool status')
+        self.assertIn(pn, result.output)
+        self.assertIn(srcdir, result.output)
+        self.assertIn(recipefile, result.output)
+        checkvars = {}
+        checkvars['LICENSE'] = 'GPLv2'
+        checkvars['LIC_FILES_CHKSUM'] = 'file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263'
+        checkvars['S'] = '${WORKDIR}/git'
+        checkvars['PV'] = '0.1+git${SRCPV}'
+        checkvars['SRC_URI'] = 'git://git.yoctoproject.org/git/dbus-wait;protocol=https'
+        checkvars['SRCREV'] = '${AUTOREV}'
+        checkvars['DEPENDS'] = set(['dbus'])
+        self._test_recipe_contents(recipefile, checkvars, [])
+
     @testcase(1162)
     def test_devtool_add_library(self):
         # We don't have the ability to pick up this dependency automatically yet...
-- 
2.5.0