[PATCH 0/6] devtool: improve handling of local source files

[PATCH 0/6] devtool: improve handling of local source files

[Markus Lehtonen]

This patchset tries to improve handling of local source files (i.e. file:// in
SRC_URI). First, it improves packages for which S=WORKDIR (that possibly only
have local sources. Second, it makes local sources available in the srctree for
all packages.

See yocto bug #7602

Markus Lehtonen (6):
  devtool: extract: remove patches when S=WORKDIR
  recipeutils: implement get_recipe_local_files()
  oe.patch.GitApplyTree: add paths argument to extractPatches
  devtool: update-recipe: update local files directly
  devtool: extract: always import local files to srctree
  devtool: modify: make bitbake use local files from srctree

 meta/lib/oe/patch.py            |   5 +-
 meta/lib/oe/recipeutils.py      |   8 +++
 scripts/lib/devtool/standard.py | 116 +++++++++++++++++++++++++++++++++++++---
 3 files changed, 122 insertions(+), 7 deletions(-)

-- 
2.1.4

[PATCH 1/6] devtool: extract: remove patches when S=WORKDIR

[Markus Lehtonen]

Before this change, all files from the recipe (SRC_URI), including
patches, were added to to srctree repository when S==WORKDIR. The patch
files are useless as they are automatically applied on top of the
srctree by devtool.

This change causes devtool extract to not commit these unnecessary (and
possibly confusing) patch file(s) into srctree repository.

[YOCTO #7602]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 27 ++++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 94b5e0b..4b4a0a0 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -145,6 +145,16 @@ def _parse_recipe(config, tinfoil, pn, appends):
     return oe.recipeutils.parse_recipe(recipefile, append_files,
                                        tinfoil.config_data)
 
+
+def _ls_tree(directory):
+    """Recursive listing of files in a directory"""
+    ret = []
+    for root, dirs, files in os.walk(directory):
+        ret.extend([os.path.relpath(os.path.join(root, fname), directory) for
+                    fname in files])
+    return ret
+
+
 def extract(args, config, basepath, workspace):
     import bb
 
@@ -164,6 +174,7 @@ def extract(args, config, basepath, workspace):
 
 def _extract_source(srctree, keep_temp, devbranch, d):
     import bb.event
+    import oe.recipeutils
 
     def eventfilter(name, handler, event, d):
         if name == 'base_eventhandler':
@@ -232,7 +243,21 @@ def _extract_source(srctree, keep_temp, devbranch, d):
         logger.info('Unpacking...')
         exec_task_func('do_unpack', False)
         srcsubdir = crd.getVar('S', True)
-        if srcsubdir != workdir and os.path.dirname(srcsubdir) != workdir:
+        if srcsubdir == workdir:
+            # Find non-patch sources that were "unpacked" to srctree directory
+            recipe_patches = [os.path.basename(patch) for patch in
+                              oe.recipeutils.get_recipe_patches(crd)]
+            src_files = [fname for fname in _ls_tree(workdir) if
+                         os.path.basename(fname) not in recipe_patches]
+            # Force separate S so that patch files can be left out from srctree
+            srcsubdir = tempfile.mkdtemp(dir=workdir)
+            crd.setVar('S', srcsubdir)
+            # Move source files to S
+            for path in src_files:
+                tgt_dir = os.path.join(srcsubdir, os.path.dirname(path))
+                bb.utils.mkdirhier(tgt_dir)
+                shutil.move(os.path.join(workdir, path), tgt_dir)
+        elif os.path.dirname(srcsubdir) != workdir:
             # Handle if S is set to a subdirectory of the source
             srcsubdir = os.path.join(workdir, os.path.relpath(srcsubdir, workdir).split(os.sep)[0])
 
-- 
2.1.4

[PATCH 2/6] recipeutils: implement get_recipe_local_files()

[Markus Lehtonen]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oe/recipeutils.py | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index 19d97b6..f3af717 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -254,6 +254,14 @@ def copy_recipe_files(d, tgt_dir, whole_dir=False, download=True):
     return remotes
 
 
+def get_recipe_local_files(d):
+    """Get a list of local files in SRC_URI within a recipe."""
+    uris = (d.getVar('SRC_URI', True) or "").split()
+    fetch = bb.fetch2.Fetch(uris, d)
+    return dict([(fetch.ud[uri].basepath, fetch.localpath(uri)) for uri in uris
+                 if uri.startswith('file://')])
+
+
 def get_recipe_patches(d):
     """Get a list of the patches included in SRC_URI within a recipe."""
     patchfiles = []
-- 
2.1.4

[PATCH 3/6] oe.patch.GitApplyTree: add paths argument to extractPatches

[Markus Lehtonen]

Makes it possible to define which paths are included in the patches.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oe/patch.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
index f68d40f..85cb2c1 100644
--- a/meta/lib/oe/patch.py
+++ b/meta/lib/oe/patch.py
@@ -273,12 +273,15 @@ class GitApplyTree(PatchTree):
         return (tmpfile, cmd)
 
     @staticmethod
-    def extractPatches(tree, startcommit, outdir):
+    def extractPatches(tree, startcommit, outdir, paths=None):
         import tempfile
         import shutil
         tempdir = tempfile.mkdtemp(prefix='oepatch')
         try:
             shellcmd = ["git", "format-patch", startcommit, "-o", tempdir]
+            if paths:
+                shellcmd.append('--')
+                shellcmd.extend(paths)
             out = runcmd(["sh", "-c", " ".join(shellcmd)], tree)
             if out:
                 for srcfile in out.split():
-- 
2.1.4

[PATCH 4/6] devtool: update-recipe: update local files directly

[Markus Lehtonen]

Source files that are directly referenced in the SRC_URI are imported
into srctree (with devtool extract) in the case S=WORKDIR. If these
files are local (i.e. they reside in the "recipe space" and not behind a
remote URL) we don't want create a patch against them, but, rather copy
our modified version over the original source.

NOTE: if new files are created, they are represented as patches, rather
than copied over the orignal source.

[YOCTO #7602]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 48 ++++++++++++++++++++++++++++++++++++++---
 1 file changed, 45 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 4b4a0a0..8a2783b 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -146,6 +146,16 @@ def _parse_recipe(config, tinfoil, pn, appends):
                                        tinfoil.config_data)
 
 
+def _git_ls_tree(repodir, treeish='HEAD', recursive=False):
+    """List contents of a git treeish"""
+    import bb
+    cmd = ['git', 'ls-tree', '-z', 'HEAD']
+    if recursive:
+        cmd.append('-r')
+    out, _ = bb.process.run(cmd, cwd=repodir)
+    return [line.split(None, 4)[3] for line in out.split('\0') if line]
+
+
 def _ls_tree(directory):
     """Recursive listing of files in a directory"""
     ret = []
@@ -514,6 +524,35 @@ def update_recipe(args, config, basepath, workspace):
         logger.error('Invalid hash returned by git: %s' % stdout)
         return 1
 
+    # Find out local files (SRC_URI files that exist in the "recipe space").
+    # Local files that reside in srctree are not included in patch generation.
+    # Instead they are directly copied over the original source files (in
+    # recipe space).
+    #
+    # NOTE: "Filtering out" of local files in this way is not entirely reliable
+    # - we don't catch files that are deleted, for example. A more reliable way
+    # to implement this would be to use "negative pathspecs" which were
+    # introduced in Git v1.9.0.  Revisit this when/if the required Git version
+    # becomes greater than that.
+    local_files = oe.recipeutils.get_recipe_local_files(rd)
+    tempdir = tempfile.mkdtemp(prefix='devtool')
+    try:
+        # Copy local files from srctree HEAD to "recipe space"
+        # Local files might be "all over the place", need recursive ls-tree
+        git_files = set(_git_ls_tree(srctree, recursive=True))
+        copy_files = git_files.intersection(set(local_files.keys()))
+        patch_include_paths = git_files.difference(set(local_files.keys()))
+        bb.process.run(['git', 'checkout', 'HEAD', '--'] + list(copy_files),
+                        cwd=srctree,
+                        env=dict(os.environ, GIT_WORK_TREE=tempdir))
+        for fname in _ls_tree(local_src_dir):
+            logger.info('Updating file %s' % fname)
+            shutil.copy2(os.path.join(local_src_dir, fname),
+                         local_files[fname])
+    finally:
+        shutil.rmtree(tempdir)
+
+    # Update recipe and patches
     if mode == 'srcrev':
         logger.info('Updating SRCREV in recipe %s' % os.path.basename(recipefile))
         patchfields = {}
@@ -526,7 +565,8 @@ def update_recipe(args, config, basepath, workspace):
             tempdir = tempfile.mkdtemp(prefix='devtool')
             removepatches = []
             try:
-                GitApplyTree.extractPatches(srctree, old_srcrev, tempdir)
+                GitApplyTree.extractPatches(srctree, old_srcrev, tempdir,
+                                            patch_include_paths)
                 newpatches = os.listdir(tempdir)
                 for patch in existing_patches:
                     patchfile = os.path.basename(patch)
@@ -581,7 +621,8 @@ def update_recipe(args, config, basepath, workspace):
             # Get all patches from source tree and check if any should be removed
             tempdir = tempfile.mkdtemp(prefix='devtool')
             try:
-                GitApplyTree.extractPatches(srctree, initial_rev, tempdir)
+                GitApplyTree.extractPatches(srctree, initial_rev, tempdir,
+                                            patch_include_paths)
                 newpatches = os.listdir(tempdir)
                 for patch in existing_patches:
                     patchfile = os.path.basename(patch)
@@ -593,7 +634,8 @@ def update_recipe(args, config, basepath, workspace):
         # Get updated patches from source tree
         tempdir = tempfile.mkdtemp(prefix='devtool')
         try:
-            GitApplyTree.extractPatches(srctree, update_rev, tempdir)
+            GitApplyTree.extractPatches(srctree, update_rev, tempdir,
+                                        patch_include_paths)
 
             # Match up and replace existing patches with corresponding new patches
             updatepatches = False
-- 
2.1.4

[PATCH 5/6] devtool: extract: always import local files to srctree

[Markus Lehtonen]

Import all non-compressed/non-arcived local files from the SRC_URI
(excluding patches) to the srctree repository. The files will be placed
in 'local-files' subdirectory. However, in case S=WORKDIR, the files are
imported into root ot srctree (and not under 'local-files', as before.

[YOCTO #7602]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 56 ++++++++++++++++++++++++++++++++---------
 1 file changed, 44 insertions(+), 12 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 8a2783b..1e81919 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -253,10 +253,11 @@ def _extract_source(srctree, keep_temp, devbranch, d):
         logger.info('Unpacking...')
         exec_task_func('do_unpack', False)
         srcsubdir = crd.getVar('S', True)
+
+        recipe_patches = [os.path.basename(patch) for patch in
+                          oe.recipeutils.get_recipe_patches(crd)]
         if srcsubdir == workdir:
             # Find non-patch sources that were "unpacked" to srctree directory
-            recipe_patches = [os.path.basename(patch) for patch in
-                              oe.recipeutils.get_recipe_patches(crd)]
             src_files = [fname for fname in _ls_tree(workdir) if
                          os.path.basename(fname) not in recipe_patches]
             # Force separate S so that patch files can be left out from srctree
@@ -312,6 +313,23 @@ def _extract_source(srctree, keep_temp, devbranch, d):
 
         bb.process.run('git tag -f devtool-patched', cwd=srcsubdir)
 
+        # Add unpacked local files (in case of S=WORKDIR these were already
+        # moved to S) into srctree
+        local_files = oe.recipeutils.get_recipe_local_files(crd)
+        local_files = [fname for fname in local_files if
+                        os.path.basename(fname) not in recipe_patches and
+                        os.path.exists(os.path.join(workdir, fname))]
+        if local_files:
+            logger.info('Adding local files...')
+            for fname in local_files:
+                localf_subdir = os.path.join(srcsubdir, 'local-files',
+                                             os.path.dirname(fname))
+                bb.utils.mkdirhier(localf_subdir)
+                shutil.move(os.path.join(workdir, fname), localf_subdir)
+            bb.process.run(['git', 'add', 'local-files'], cwd=srcsubdir)
+            bb.process.run(['git', 'commit', '-q', '-m', 'Add local files'],
+                            cwd=srcsubdir)
+
         if os.path.exists(patchdir):
             shutil.rmtree(patchdir)
             if haspatches:
@@ -538,17 +556,31 @@ def update_recipe(args, config, basepath, workspace):
     tempdir = tempfile.mkdtemp(prefix='devtool')
     try:
         # Copy local files from srctree HEAD to "recipe space"
-        # Local files might be "all over the place", need recursive ls-tree
-        git_files = set(_git_ls_tree(srctree, recursive=True))
-        copy_files = git_files.intersection(set(local_files.keys()))
-        patch_include_paths = git_files.difference(set(local_files.keys()))
-        bb.process.run(['git', 'checkout', 'HEAD', '--'] + list(copy_files),
-                        cwd=srctree,
-                        env=dict(os.environ, GIT_WORK_TREE=tempdir))
+        if os.path.isdir(os.path.join(srctree, 'local-files')):
+            # Local files are in 'local-files' only list root of git repo
+            git_files = set(_git_ls_tree(srctree))
+            patch_include_paths = git_files.difference(set(['local-files']))
+            bb.process.run(['git', 'checkout', 'HEAD', '--', 'local-files'],
+                            cwd=srctree,
+                            env=dict(os.environ, GIT_WORK_TREE=tempdir))
+            local_src_dir = os.path.join(tempdir, 'local-files')
+        else:
+            # Local files might be "all over the place", need recursive ls-tree
+            git_files = set(_git_ls_tree(srctree, recursive=True))
+            copy_files = git_files.intersection(set(local_files.keys()))
+            patch_include_paths = git_files.difference(set(local_files.keys()))
+            bb.process.run(['git', 'checkout', 'HEAD', '--'] + list(copy_files),
+                            cwd=srctree,
+                            env=dict(os.environ, GIT_WORK_TREE=tempdir))
+            local_src_dir = tempdir
+
         for fname in _ls_tree(local_src_dir):
-            logger.info('Updating file %s' % fname)
-            shutil.copy2(os.path.join(local_src_dir, fname),
-                         local_files[fname])
+            if fname in local_files:
+                logger.info('Updating file %s' % fname)
+                shutil.copy2(os.path.join(local_src_dir, fname),
+                             local_files[fname])
+            else:
+                logger.warning('File %s not in SRC_URI, skipping it' % fname)
     finally:
         shutil.rmtree(tempdir)
 
-- 
2.1.4

[PATCH 6/6] devtool: modify: make bitbake use local files from srctree

[Markus Lehtonen]

This change makes it possible to have local files (SRC_URI files in
"recipe space") under the srctree even if S!=WORKDIR. The files must be
placed under the 'local-files' subdirectory.

Complements the previous patch that imports local files into srctree.

[YOCTO #7602]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 1e81919..fb68976 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -450,8 +450,13 @@ def modify(args, config, basepath, workspace):
         appendname = re.sub(r'_.*', '_%', appendname)
     appendfile = os.path.join(appendpath, appendname + '.bbappend')
     with open(appendfile, 'w') as f:
-        f.write('FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"\n\n')
-        f.write('inherit externalsrc\n')
+        f.write('FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"\n')
+        # Local files can be modified/tracked in separate subdir under srctree
+        # Mostly useful for packages with S != WORKDIR
+        f.write('FILESPATH_prepend := "%s:"\n' %
+                os.path.join(srctree, 'local-files'))
+
+        f.write('\ninherit externalsrc\n')
         f.write('# NOTE: We use pn- overrides here to avoid affecting multiple variants in the case where the recipe uses BBCLASSEXTEND\n')
         f.write('EXTERNALSRC_pn-%s = "%s"\n' % (args.recipename, srctree))
 
-- 
2.1.4

Re: [PATCH 0/6] devtool: improve handling of local source files

[Paul Eggleton]

Hi Markus,

On Thursday 30 April 2015 12:16:06 Markus Lehtonen wrote:
> This patchset tries to improve handling of local source files (i.e. file://
> in SRC_URI). First, it improves packages for which S=WORKDIR (that possibly
> only have local sources. Second, it makes local sources available in the
> srctree for all packages.
> 
> See yocto bug #7602

I've finally looked at these, apologies for the delay. Some comments:

* I don't think we really want the local files to become part of the git 
repository by default - they shouldn't be committed. Once users have finished 
with devtool, we want them to be able to push the source tree to their own 
repo and point to that within the recipe, whilst keeping the local files next 
to the recipe. 

* This implies that new files added to the local files dir when we do devtool 
update-recipe should not be added as a patch, they should be copied next to 
the recipe and added to SRC_URI. I'm more than happy for us to implement this 
separately as a follow-up (i.e. we could start by not handling adding files to 
the local files directory at all.)

* The local-files directory needs to be named specific to OE - "oe-local-files" 
would be ideal. If we could have one place in the code where this was defined 
that would be ideal as well (maybe at some point we'd allow it to be 
configured).

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/6] devtool: improve handling of local source files

[Markus Lehtonen]

Hi,

On Tue, 2015-05-12 at 19:01 +0100, Paul Eggleton wrote:
> Hi Markus,
> 
> On Thursday 30 April 2015 12:16:06 Markus Lehtonen wrote:
> > This patchset tries to improve handling of local source files (i.e. file://
> > in SRC_URI). First, it improves packages for which S=WORKDIR (that possibly
> > only have local sources. Second, it makes local sources available in the
> > srctree for all packages.
> > 
> > See yocto bug #7602
> 
> I've finally looked at these, apologies for the delay. Some comments:
> 
> * I don't think we really want the local files to become part of the git 
> repository by default - they shouldn't be committed. Once users have finished 
> with devtool, we want them to be able to push the source tree to their own 
> repo and point to that within the recipe, whilst keeping the local files next 
> to the recipe.

So you suggest to add a new command line option to devtool extract and
modify (--local-files or smth)? What to do when there are only local
files (no source tarball / repo) - automatically enable --local-files in
this case?


> * This implies that new files added to the local files dir when we do devtool 
> update-recipe should not be added as a patch, they should be copied next to 
> the recipe and added to SRC_URI. I'm more than happy for us to implement this 
> separately as a follow-up (i.e. we could start by not handling adding files to 
> the local files directory at all.)

Yeah, I actually have this WIP. Currently (i.e. with the current
patchset), new files added to 'local-files' are just ignored. They are
not copied and no patches is generated out of these.


> * The local-files directory needs to be named specific to OE - "oe-local-files" 
> would be ideal. If we could have one place in the code where this was defined 
> that would be ideal as well (maybe at some point we'd allow it to be 
> configured).

This is not a big deal. Should it perhaps be "bb-local-files" instead?


Thanks,
  Markus

Re: [PATCH 0/6] devtool: improve handling of local source files

[Paul Eggleton]

On Thursday 04 June 2015 16:12:07 Markus Lehtonen wrote:
> On Tue, 2015-05-12 at 19:01 +0100, Paul Eggleton wrote:
> > On Thursday 30 April 2015 12:16:06 Markus Lehtonen wrote:
> > > This patchset tries to improve handling of local source files (i.e.
> > > file://
> > > in SRC_URI). First, it improves packages for which S=WORKDIR (that
> > > possibly
> > > only have local sources. Second, it makes local sources available in the
> > > srctree for all packages.
> > > 
> > > See yocto bug #7602
> > 
> > I've finally looked at these, apologies for the delay. Some comments:
> > 
> > * I don't think we really want the local files to become part of the git
> > repository by default - they shouldn't be committed. Once users have
> > finished with devtool, we want them to be able to push the source tree to
> > their own repo and point to that within the recipe, whilst keeping the
> > local files next to the recipe.
> 
> So you suggest to add a new command line option to devtool extract and
> modify (--local-files or smth)? What to do when there are only local
> files (no source tarball / repo) - automatically enable --local-files in
> this case?

Is another option really required? Unless I'm missing something, I would have 
thought the behaviour for local files ought to be the same regardless of 
whether they are in addition to the upstream source, or the only files in 
SRC_URI.

> > * This implies that new files added to the local files dir when we do
> > devtool update-recipe should not be added as a patch, they should be
> > copied next to the recipe and added to SRC_URI. I'm more than happy for
> > us to implement this separately as a follow-up (i.e. we could start by
> > not handling adding files to the local files directory at all.)
> 
> Yeah, I actually have this WIP. Currently (i.e. with the current
> patchset), new files added to 'local-files' are just ignored. They are
> not copied and no patches is generated out of these.

OK, then it sounds like the behaviour for added files is reasonable for the 
moment and we can extend it as a follow-up.

> > * The local-files directory needs to be named specific to OE -
> > "oe-local-files" would be ideal. If we could have one place in the code
> > where this was defined that would be ideal as well (maybe at some point
> > we'd allow it to be configured).
> 
> This is not a big deal. Should it perhaps be "bb-local-files" instead?

Well strictly speaking all of this is being defined in OE, not bitbake, hence 
my suggestion of "oe-local-files".

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/6] devtool: improve handling of local source files

[Markus Lehtonen]

Hi Paul,

On Thu, 2015-06-04 at 14:49 +0100, Paul Eggleton wrote:
> On Thursday 04 June 2015 16:12:07 Markus Lehtonen wrote:
> > On Tue, 2015-05-12 at 19:01 +0100, Paul Eggleton wrote:
> > > On Thursday 30 April 2015 12:16:06 Markus Lehtonen wrote:
> > > > This patchset tries to improve handling of local source files (i.e.
> > > > file://
> > > > in SRC_URI). First, it improves packages for which S=WORKDIR (that
> > > > possibly
> > > > only have local sources. Second, it makes local sources available in the
> > > > srctree for all packages.
> > > > 
> > > > See yocto bug #7602
> > > 
> > > I've finally looked at these, apologies for the delay. Some comments:
> > > 
> > > * I don't think we really want the local files to become part of the git
> > > repository by default - they shouldn't be committed. Once users have
> > > finished with devtool, we want them to be able to push the source tree to
> > > their own repo and point to that within the recipe, whilst keeping the
> > > local files next to the recipe.
> > 
> > So you suggest to add a new command line option to devtool extract and
> > modify (--local-files or smth)? What to do when there are only local
> > files (no source tarball / repo) - automatically enable --local-files in
> > this case?
> 
> Is another option really required? Unless I'm missing something, I would have 
> thought the behaviour for local files ought to be the same regardless of 
> whether they are in addition to the upstream source, or the only files in 
> SRC_URI.

Currently, the local files are already committed into Git in some cases
(i.e. basically when S==WORKDIR, e.g. with makedevs).

What would you suggest to do with the local files? Copy the files to git
repo worktree but add them to .git/info/exclude? Or not copy them at
all? In this case some packages would not be supported anymore (e.g. the
makedevs mentioned before).

I agree that the behavior should be consistent and the same regardless
of the existence of upstream sources.


> > > * This implies that new files added to the local files dir when we do
> > > devtool update-recipe should not be added as a patch, they should be
> > > copied next to the recipe and added to SRC_URI. I'm more than happy for
> > > us to implement this separately as a follow-up (i.e. we could start by
> > > not handling adding files to the local files directory at all.)
> > 
> > Yeah, I actually have this WIP. Currently (i.e. with the current
> > patchset), new files added to 'local-files' are just ignored. They are
> > not copied and no patches is generated out of these.
> 
> OK, then it sounds like the behaviour for added files is reasonable for the 
> moment and we can extend it as a follow-up.

OK, thanks.


> > > * The local-files directory needs to be named specific to OE -
> > > "oe-local-files" would be ideal. If we could have one place in the code
> > > where this was defined that would be ideal as well (maybe at some point
> > > we'd allow it to be configured).
> > 
> > This is not a big deal. Should it perhaps be "bb-local-files" instead?
> 
> Well strictly speaking all of this is being defined in OE, not bitbake, hence 
> my suggestion of "oe-local-files".

OK, "oe-local-files" is probably fine. I was just thinking about Poky or
some other derived project where the files might not be from OE.


Thanks,
  Markus