* [RFC PATCH 1/7] lib/oe/patch: fall back to patch if git apply fails
2014-11-25 17:28 [RFC PATCH 0/7] Developer workflow improvements Paul Eggleton
@ 2014-11-25 17:28 ` Paul Eggleton
2014-11-25 17:40 ` Paul Barker
2014-11-25 17:28 ` [RFC PATCH 2/7] lib/oe/patch: auto-commit when falling back from git am Paul Eggleton
` (10 subsequent siblings)
11 siblings, 1 reply; 40+ messages in thread
From: Paul Eggleton @ 2014-11-25 17:28 UTC (permalink / raw)
To: openembedded-core
When PATCHTOOL = "git", git apply doesn't support fuzzy application, so
if a patch requires that it's better to be able to apply it rather than
just failing.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/patch.py | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
index b085c9d..788f465 100644
--- a/meta/lib/oe/patch.py
+++ b/meta/lib/oe/patch.py
@@ -219,7 +219,11 @@ class GitApplyTree(PatchTree):
return _applypatchhelper(shellcmd, patch, force, reverse, run)
except CmdError:
shellcmd = ["git", "--git-dir=.", "apply", "-p%s" % patch['strippath']]
- return _applypatchhelper(shellcmd, patch, force, reverse, run)
+ try:
+ output = _applypatchhelper(shellcmd, patch, force, reverse, run)
+ except CmdError:
+ output = PatchTree._applypatch(self, patch, force, reverse, run)
+ return output
class QuiltTree(PatchSet):
--
1.9.3
^ permalink raw reply related [flat|nested] 40+ messages in thread* Re: [RFC PATCH 1/7] lib/oe/patch: fall back to patch if git apply fails
2014-11-25 17:28 ` [RFC PATCH 1/7] lib/oe/patch: fall back to patch if git apply fails Paul Eggleton
@ 2014-11-25 17:40 ` Paul Barker
2014-11-25 17:58 ` Paul Eggleton
0 siblings, 1 reply; 40+ messages in thread
From: Paul Barker @ 2014-11-25 17:40 UTC (permalink / raw)
To: Paul Eggleton; +Cc: OE Core
On 25 November 2014 at 17:28, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
> When PATCHTOOL = "git", git apply doesn't support fuzzy application, so
> if a patch requires that it's better to be able to apply it rather than
> just failing.
>
> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> ---
> meta/lib/oe/patch.py | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
> index b085c9d..788f465 100644
> --- a/meta/lib/oe/patch.py
> +++ b/meta/lib/oe/patch.py
> @@ -219,7 +219,11 @@ class GitApplyTree(PatchTree):
> return _applypatchhelper(shellcmd, patch, force, reverse, run)
> except CmdError:
> shellcmd = ["git", "--git-dir=.", "apply", "-p%s" % patch['strippath']]
> - return _applypatchhelper(shellcmd, patch, force, reverse, run)
> + try:
> + output = _applypatchhelper(shellcmd, patch, force, reverse, run)
> + except CmdError:
> + output = PatchTree._applypatch(self, patch, force, reverse, run)
> + return output
>
Would this give a warning or other notification if the fallback to
'patch' is used? When developing patches it'd probably be good to know
whether they're exactly correct or not.
I haven't looked at _applypatchhelper or _applypatch so I'm not sure
if there's some notification hidden in there.
Cheers,
--
Paul Barker
Email: paul@paulbarker.me.uk
http://www.paulbarker.me.uk
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC PATCH 1/7] lib/oe/patch: fall back to patch if git apply fails
2014-11-25 17:40 ` Paul Barker
@ 2014-11-25 17:58 ` Paul Eggleton
0 siblings, 0 replies; 40+ messages in thread
From: Paul Eggleton @ 2014-11-25 17:58 UTC (permalink / raw)
To: Paul Barker; +Cc: OE Core
Hi Paul,
On Tuesday 25 November 2014 17:40:24 Paul Barker wrote:
> On 25 November 2014 at 17:28, Paul Eggleton
>
> <paul.eggleton@linux.intel.com> wrote:
> > When PATCHTOOL = "git", git apply doesn't support fuzzy application, so
> > if a patch requires that it's better to be able to apply it rather than
> > just failing.
> >
> > Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> > ---
> >
> > meta/lib/oe/patch.py | 6 +++++-
> > 1 file changed, 5 insertions(+), 1 deletion(-)
> >
> > diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
> > index b085c9d..788f465 100644
> > --- a/meta/lib/oe/patch.py
> > +++ b/meta/lib/oe/patch.py
> >
> > @@ -219,7 +219,11 @@ class GitApplyTree(PatchTree):
> > return _applypatchhelper(shellcmd, patch, force, reverse,
> > run)
> >
> > except CmdError:
> > shellcmd = ["git", "--git-dir=.", "apply", "-p%s" %
> > patch['strippath']]
> >
> > - return _applypatchhelper(shellcmd, patch, force, reverse,
> > run)
> > + try:
> > + output = _applypatchhelper(shellcmd, patch, force,
> > reverse, run) + except CmdError:
> > + output = PatchTree._applypatch(self, patch, force,
> > reverse, run) + return output
>
> Would this give a warning or other notification if the fallback to
> 'patch' is used? When developing patches it'd probably be good to know
> whether they're exactly correct or not.
In the context I'm attempting to use it I'd want it not to warn in this case,
but I can see that in normal usage if you've explicitly elected to use
PATCHTOOL = "git" it could be that you do expect all patches for the recipe to
apply that way and want to see a warning if they don't.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 40+ messages in thread
* [RFC PATCH 2/7] lib/oe/patch: auto-commit when falling back from git am
2014-11-25 17:28 [RFC PATCH 0/7] Developer workflow improvements Paul Eggleton
2014-11-25 17:28 ` [RFC PATCH 1/7] lib/oe/patch: fall back to patch if git apply fails Paul Eggleton
@ 2014-11-25 17:28 ` Paul Eggleton
2014-11-25 17:28 ` [RFC PATCH 3/7] lib/oe/patch: use --keep-cr with " Paul Eggleton
` (9 subsequent siblings)
11 siblings, 0 replies; 40+ messages in thread
From: Paul Eggleton @ 2014-11-25 17:28 UTC (permalink / raw)
To: openembedded-core
When PATCHTOOL = "git", if we're not able to use "git am" to apply a
patch and fall back to "git apply" or "patch", it is desirable to
actually commit the changes, attempting to preserve (and interpret) the
patch header as part of the commit message if present. As a bonus, the
code for extracting the commit message is callable externally in case it
is useful elsewhere.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/patch.py | 86 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 86 insertions(+)
diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
index 788f465..2d56ba4 100644
--- a/meta/lib/oe/patch.py
+++ b/meta/lib/oe/patch.py
@@ -202,6 +202,78 @@ class GitApplyTree(PatchTree):
def __init__(self, dir, d):
PatchTree.__init__(self, dir, d)
+ @staticmethod
+ def extractPatchHeader(patchfile):
+ """
+ Extract just the header lines from the top of a patch file
+ """
+ lines = []
+ with open(patchfile, 'r') as f:
+ for line in f.readlines():
+ if line.startswith('Index: ') or line.startswith('diff -') or line.startswith('---'):
+ break
+ lines.append(line)
+ return lines
+
+ @staticmethod
+ def prepareCommit(patchfile):
+ """
+ Prepare a git commit command line based on the header from a patch file
+ (typically this is useful for patches that cannot be applied with "git am" due to formatting)
+ """
+ import tempfile
+ import re
+ author_re = re.compile('[\S ]+ <\S+@\S+\.\S+>')
+ # Process patch header and extract useful information
+ lines = GitApplyTree.extractPatchHeader(patchfile)
+ outlines = []
+ author = None
+ date = None
+ for line in lines:
+ if line.startswith('Subject: '):
+ subject = line.split(':', 1)[1]
+ # Remove any [PATCH][oe-core] etc.
+ subject = re.sub(r'\[.+?\]\s*', '', subject)
+ outlines.insert(0, '%s\n\n' % subject.strip())
+ continue
+ if line.startswith('From: ') or line.startswith('Author: '):
+ authorval = line.split(':', 1)[1].strip().replace('"', '')
+ # git is fussy about author formatting i.e. it must be Name <email@domain>
+ if author_re.match(authorval):
+ author = authorval
+ continue
+ if line.startswith('Date: '):
+ if date is None:
+ dateval = line.split(':', 1)[1].strip()
+ # Very crude check for date format, since git will blow up if it's not in the right
+ # format. Without e.g. a python-dateutils dependency we can't do a whole lot more
+ if len(dateval) > 12:
+ date = dateval
+ continue
+ if line.startswith('Signed-off-by: '):
+ authorval = line.split(':', 1)[1].strip().replace('"', '')
+ # git is fussy about author formatting i.e. it must be Name <email@domain>
+ if author_re.match(authorval):
+ author = authorval
+ outlines.append(line)
+ # Add a pointer to the original patch file name
+ if outlines and outlines[-1].strip():
+ outlines.append('\n')
+ outlines.append('(from original patch: %s)\n' % os.path.basename(patchfile))
+ # Write out commit message to a file
+ with tempfile.NamedTemporaryFile('w', delete=False) as tf:
+ tmpfile = tf.name
+ for line in outlines:
+ tf.write(line)
+ # Prepare git command
+ cmd = ["git", "commit", "-F", tmpfile]
+ # git doesn't like plain email addresses as authors
+ if author and '<' in author:
+ cmd.append('--author="%s"' % author)
+ if date:
+ cmd.append('--date="%s"' % date)
+ return (tmpfile, cmd)
+
def _applypatch(self, patch, force = False, reverse = False, run = True):
def _applypatchhelper(shellcmd, patch, force = False, reverse = False, run = True):
if reverse:
@@ -218,11 +290,25 @@ class GitApplyTree(PatchTree):
shellcmd = ["git", "--work-tree=.", "am", "-3", "-p%s" % patch['strippath']]
return _applypatchhelper(shellcmd, patch, force, reverse, run)
except CmdError:
+ # Fall back to git apply
shellcmd = ["git", "--git-dir=.", "apply", "-p%s" % patch['strippath']]
try:
output = _applypatchhelper(shellcmd, patch, force, reverse, run)
except CmdError:
+ # Fall back to patch
output = PatchTree._applypatch(self, patch, force, reverse, run)
+ # Add all files
+ shellcmd = ["git", "add", "-f", "."]
+ output += runcmd(["sh", "-c", " ".join(shellcmd)], self.dir)
+ # Exclude the patches directory
+ shellcmd = ["git", "reset", "HEAD", self.patchdir]
+ output += runcmd(["sh", "-c", " ".join(shellcmd)], self.dir)
+ # Commit the result
+ (tmpfile, shellcmd) = self.prepareCommit(patch['file'])
+ try:
+ output += runcmd(["sh", "-c", " ".join(shellcmd)], self.dir)
+ finally:
+ os.remove(tmpfile)
return output
--
1.9.3
^ permalink raw reply related [flat|nested] 40+ messages in thread* [RFC PATCH 3/7] lib/oe/patch: use --keep-cr with git am
2014-11-25 17:28 [RFC PATCH 0/7] Developer workflow improvements Paul Eggleton
2014-11-25 17:28 ` [RFC PATCH 1/7] lib/oe/patch: fall back to patch if git apply fails Paul Eggleton
2014-11-25 17:28 ` [RFC PATCH 2/7] lib/oe/patch: auto-commit when falling back from git am Paul Eggleton
@ 2014-11-25 17:28 ` Paul Eggleton
2014-11-25 17:28 ` [RFC PATCH 4/7] scripts/recipetool: Add a recipe auto-creation script Paul Eggleton
` (8 subsequent siblings)
11 siblings, 0 replies; 40+ messages in thread
From: Paul Eggleton @ 2014-11-25 17:28 UTC (permalink / raw)
To: openembedded-core
Preserving carriage returns is important where the patch contains them.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/patch.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
index 2d56ba4..2bf3065 100644
--- a/meta/lib/oe/patch.py
+++ b/meta/lib/oe/patch.py
@@ -287,7 +287,7 @@ class GitApplyTree(PatchTree):
return runcmd(["sh", "-c", " ".join(shellcmd)], self.dir)
try:
- shellcmd = ["git", "--work-tree=.", "am", "-3", "-p%s" % patch['strippath']]
+ shellcmd = ["git", "--work-tree=.", "am", "-3", "--keep-cr", "-p%s" % patch['strippath']]
return _applypatchhelper(shellcmd, patch, force, reverse, run)
except CmdError:
# Fall back to git apply
--
1.9.3
^ permalink raw reply related [flat|nested] 40+ messages in thread* [RFC PATCH 4/7] scripts/recipetool: Add a recipe auto-creation script
2014-11-25 17:28 [RFC PATCH 0/7] Developer workflow improvements Paul Eggleton
` (2 preceding siblings ...)
2014-11-25 17:28 ` [RFC PATCH 3/7] lib/oe/patch: use --keep-cr with " Paul Eggleton
@ 2014-11-25 17:28 ` Paul Eggleton
2014-11-25 17:28 ` [RFC PATCH 5/7] lib/oe: add recipeutils module Paul Eggleton
` (7 subsequent siblings)
11 siblings, 0 replies; 40+ messages in thread
From: Paul Eggleton @ 2014-11-25 17:28 UTC (permalink / raw)
To: openembedded-core
Add a more maintainable and flexible script for creating at least the
skeleton of a recipe based on an examination of the source tree.
Commands can be added and the creation process can be extended through
plugins.
[YOCTO #6406]
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/__init__.py | 0
scripts/lib/recipetool/create.py | 375 ++++++++++++++++++++++++++++++
scripts/lib/recipetool/create_buildsys.py | 234 +++++++++++++++++++
scripts/recipetool | 124 ++++++++++
4 files changed, 733 insertions(+)
create mode 100644 scripts/lib/recipetool/__init__.py
create mode 100644 scripts/lib/recipetool/create.py
create mode 100644 scripts/lib/recipetool/create_buildsys.py
create mode 100755 scripts/recipetool
diff --git a/scripts/lib/recipetool/__init__.py b/scripts/lib/recipetool/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
new file mode 100644
index 0000000..07b33a8
--- /dev/null
+++ b/scripts/lib/recipetool/create.py
@@ -0,0 +1,375 @@
+# Recipe creation tool - create command plugin
+#
+# Copyright (C) 2014 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import sys
+import os
+import argparse
+import glob
+import fnmatch
+import re
+import logging
+
+logger = logging.getLogger('recipetool')
+
+tinfoil = None
+plugins = None
+
+def plugin_init(pluginlist):
+ # Take a reference to the list so we can use it later
+ global plugins
+ plugins = pluginlist
+
+def tinfoil_init(instance):
+ global tinfoil
+ tinfoil = instance
+
+class RecipeHandler():
+ def checkfiles(self, path, speclist):
+ results = []
+ for spec in speclist:
+ results.extend(glob.glob(os.path.join(path, spec)))
+ return results
+
+ def genfunction(self, outlines, funcname, content):
+ outlines.append('%s () {' % funcname)
+ for line in content:
+ outlines.append('\t%s' % line)
+ outlines.append('}')
+ outlines.append('')
+
+ def process(self, srctree, classes, lines_before, lines_after, handled):
+ return False
+
+
+
+def fetch_source(uri, destdir):
+ import bb.data
+ bb.utils.mkdirhier(destdir)
+ localdata = bb.data.createCopy(tinfoil.config_data)
+ bb.data.update_data(localdata)
+ localdata.setVar('BB_STRICT_CHECKSUM', '')
+ localdata.setVar('SRCREV', '${AUTOREV}')
+ ret = (None, None)
+ olddir = os.getcwd()
+ try:
+ fetcher = bb.fetch2.Fetch([uri], localdata)
+ for u in fetcher.ud:
+ ud = fetcher.ud[u]
+ ud.ignore_checksums = True
+ fetcher.download()
+ fetcher.unpack(destdir)
+ for u in fetcher.ud:
+ ud = fetcher.ud[u]
+ if ud.method.recommends_checksum(ud):
+ md5value = bb.utils.md5_file(ud.localpath)
+ sha256value = bb.utils.sha256_file(ud.localpath)
+ ret = (md5value, sha256value)
+ except bb.fetch2.BBFetchException, e:
+ raise bb.build.FuncFailed(e)
+ finally:
+ os.chdir(olddir)
+ return ret
+
+def supports_srcrev(uri):
+ localdata = bb.data.createCopy(tinfoil.config_data)
+ bb.data.update_data(localdata)
+ fetcher = bb.fetch2.Fetch([uri], localdata)
+ urldata = fetcher.ud
+ for u in urldata:
+ if urldata[u].method.supports_srcrev():
+ return True
+ return False
+
+def create_recipe(args):
+ import bb.process
+ import tempfile
+ import shutil
+
+ pkgarch = ""
+ if args.machine:
+ pkgarch = "${MACHINE_ARCH}"
+
+ checksums = (None, None)
+ tempsrc = ''
+ srcsubdir = ''
+ if '://' in args.source:
+ # Fetch a URL
+ srcuri = args.source
+ if args.externalsrc:
+ srctree = args.externalsrc
+ else:
+ tempsrc = tempfile.mkdtemp(prefix='recipetool-')
+ srctree = tempsrc
+ logger.info('Fetching %s...' % srcuri)
+ checksums = fetch_source(args.source, srctree)
+ dirlist = os.listdir(srctree)
+ if 'git.indirectionsymlink' in dirlist:
+ dirlist.remove('git.indirectionsymlink')
+ if len(dirlist) == 1 and os.path.isdir(os.path.join(srctree, dirlist[0])):
+ # We unpacked a single directory, so we should use that
+ srcsubdir = dirlist[0]
+ srctree = os.path.join(srctree, srcsubdir)
+ else:
+ # Assume we're pointing to an existing source tree
+ if args.externalsrc:
+ logger.error('externalsrc cannot be specified if source is a directory')
+ sys.exit(1)
+ if not os.path.isdir(args.source):
+ logger.error('Invalid source directory %s' % args.source)
+ sys.exit(1)
+ srcuri = ''
+ srctree = args.source
+
+ outfile = args.outfile
+ if outfile and outfile != '-':
+ if os.path.exists(outfile):
+ logger.error('Output file %s already exists' % outfile)
+ sys.exit(1)
+
+ lines_before = []
+ lines_after = []
+
+ lines_before.append('# Recipe created by %s' % os.path.basename(sys.argv[0]))
+ lines_before.append('# This is the basis of a recipe and may need further editing in order to be fully functional.')
+ lines_before.append('# (Feel free to remove these comments when editing.)')
+ lines_before.append('#')
+
+ licvalues = guess_license(srctree)
+ licenses = []
+ lic_files_chksum = []
+ for licvalue in licvalues:
+ if not licvalue[0] in licenses:
+ licenses.append(licvalue[0])
+ lic_files_chksum.append('file://%s;md5=%s' % (licvalue[1], licvalue[2]))
+ if licvalues:
+ lines_before.append('# WARNING: the following LICENSE and LIC_FILES_CHKSUM values are best guesses - it is')
+ lines_before.append('# your responsibility to verify that the values are complete and correct.')
+ if len(licvalues) > 1:
+ lines_before.append('#')
+ lines_before.append('# NOTE: multiple licenses have been detected; if that is correct you should separate')
+ lines_before.append('# these in the LICENSE value using & if the multiple licenses all apply, or | if there')
+ lines_before.append('# is a choice between the multiple licenses. If in doubt, check the accompanying')
+ lines_before.append('# documentation to determine which situation is applicable.')
+ else:
+ lines_before.append('# Unable to find any files that looked like license statements. Check the accompanying')
+ lines_before.append('# documentation and source headers and set LICENSE and LIC_FILES_CHKSUM accordingly.')
+ lines_before.append('LICENSE = "%s"' % ' '.join(licenses))
+ lines_before.append('LIC_FILES_CHKSUM = "%s"' % ' \\\n '.join(lic_files_chksum))
+ lines_before.append('')
+
+ # FIXME This is kind of a hack, we probably ought to be using bitbake to do this
+ # we'd also want a way to automatically set outfile based upon auto-detecting these values from the source if possible
+ recipefn = os.path.splitext(os.path.basename(outfile))[0]
+ fnsplit = recipefn.split('_')
+ if len(fnsplit) > 1:
+ pn = fnsplit[0]
+ pv = fnsplit[1]
+ else:
+ pn = recipefn
+ pv = None
+
+ if srcuri:
+ if pv and pv not in 'git svn hg'.split():
+ srcuri = srcuri.replace(pv, '${PV}')
+ else:
+ lines_before.append('# No information for SRC_URI yet (only an external source tree was specified)')
+ lines_before.append('SRC_URI = "%s"' % srcuri)
+ (md5value, sha256value) = checksums
+ if md5value:
+ lines_before.append('SRC_URI[md5sum] = "%s"' % md5value)
+ if sha256value:
+ lines_before.append('SRC_URI[sha256sum] = "%s"' % sha256value)
+ if srcuri and supports_srcrev(srcuri):
+ lines_before.append('')
+ lines_before.append('# Modify these as desired')
+ lines_before.append('PV = "1.0+git${SRCPV}"')
+ lines_before.append('SRCREV = "${AUTOREV}"')
+ lines_before.append('')
+
+ if srcsubdir and pv:
+ if srcsubdir == "%s-%s" % (pn, pv):
+ # This would be the default, so we don't need to set S in the recipe
+ srcsubdir = ''
+ if srcsubdir:
+ lines_before.append('S = "${WORKDIR}/%s"' % srcsubdir)
+ lines_before.append('')
+
+ if pkgarch:
+ lines_after.append('PACKAGE_ARCH = "%s"' % pkgarch)
+ lines_after.append('')
+
+ # Find all plugins that want to register handlers
+ handlers = []
+ for plugin in plugins:
+ if hasattr(plugin, 'register_recipe_handlers'):
+ plugin.register_recipe_handlers(handlers)
+
+ # Apply the handlers
+ classes = []
+ handled = []
+ for handler in handlers:
+ handler.process(srctree, classes, lines_before, lines_after, handled)
+
+ outlines = []
+ outlines.extend(lines_before)
+ if classes:
+ outlines.append('inherit %s' % ' '.join(classes))
+ outlines.append('')
+ outlines.extend(lines_after)
+
+ if outfile == '-':
+ sys.stdout.write('\n'.join(outlines) + '\n')
+ else:
+ with open(outfile, 'w') as f:
+ f.write('\n'.join(outlines) + '\n')
+ logger.info('Recipe %s has been created; further editing may be required to make it fully functional' % outfile)
+
+ if tempsrc:
+ shutil.rmtree(tempsrc)
+
+ return 0
+
+
+def guess_license(srctree):
+ import bb
+ # Gather md5sums of license files in common license dir
+ #commonlicdir = os.path.abspath(os.path.join(os.path.basename(__file__), '../meta/files/common-licenses'))
+ commonlicdir = tinfoil.config_data.getVar('COMMON_LICENSE_DIR', True)
+ md5sums = {}
+ for fn in os.listdir(commonlicdir):
+ md5value = bb.utils.md5_file(os.path.join(commonlicdir, fn))
+ md5sums[md5value] = fn
+ # FIXME we probably want more common values of this as well
+ md5sums['94d55d512a9ba36caa9b7df079bae19f'] = 'GPLv2'
+ md5sums['b234ee4d69f5fce4486a80fdaf4a4263'] = 'GPLv2'
+ md5sums['59530bdf33659b29e73d4adb9f9f6552'] = 'GPLv2'
+ md5sums['bbb461211a33b134d42ed5ee802b37ff'] = 'LGPLv2.1'
+ md5sums['7fbc338309ac38fefcd64b04bb903e34'] = 'LGPLv2.1'
+ md5sums['d32239bcb673463ab874e80d47fae504'] = 'GPLv3'
+ md5sums['6a6a8e020838b23406c81b19c1d46df6'] = 'LGPLv3'
+ md5sums['9f604d8a4f8e74f4f5140845a21b6674'] = 'LGPLv2'
+
+ licenses = []
+ licspecs = ['LICENSE*', 'COPYING*', '*[Ll]icense*', 'LICENCE*', 'LEGAL*', '[Ll]egal*', '*GPL*', 'README.lic*', 'COPYRIGHT*', '[Cc]opyright*']
+ licfiles = []
+ for root, dirs, files in os.walk(srctree):
+ for fn in files:
+ for spec in licspecs:
+ if fnmatch.fnmatch(fn, spec):
+ licfiles.append(os.path.join(root, fn))
+ for licfile in licfiles:
+ md5value = bb.utils.md5_file(licfile)
+ license = md5sums.get(md5value, 'Unknown')
+ licenses.append((license, os.path.relpath(licfile, srctree), md5value))
+
+ # FIXME should we grab at least one source file with a license header and add that too?
+
+ return licenses
+
+def read_pkgconfig_provides(d):
+ pkgdatadir = d.getVar('PKGDATA_DIR', True)
+ pkgmap = {}
+ for fn in glob.glob(os.path.join(pkgdatadir, 'shlibs2', '*.pclist')):
+ with open(fn, 'r') as f:
+ for line in f:
+ pkgmap[os.path.basename(line.rstrip())] = os.path.splitext(os.path.basename(fn))[0]
+ recipemap = {}
+ for pc, pkg in pkgmap.iteritems():
+ pkgdatafile = os.path.join(pkgdatadir, 'runtime', pkg)
+ if os.path.exists(pkgdatafile):
+ with open(pkgdatafile, 'r') as f:
+ for line in f:
+ if line.startswith('PN: '):
+ recipemap[pc] = line.split(':', 1)[1].strip()
+ return recipemap
+
+def convert_pkginfo(pkginfofile):
+ values = {}
+ with open(pkginfofile, 'r') as f:
+ indesc = False
+ for line in f:
+ if indesc:
+ if line.strip():
+ values['DESCRIPTION'] += ' ' + line.strip()
+ else:
+ indesc = False
+ else:
+ splitline = line.split(': ', 1)
+ key = line[0]
+ value = line[1]
+ if key == 'LICENSE':
+ for dep in value.split(','):
+ dep = dep.split()[0]
+ mapped = depmap.get(dep, '')
+ if mapped:
+ depends.append(mapped)
+ elif key == 'License':
+ values['LICENSE'] = value
+ elif key == 'Summary':
+ values['SUMMARY'] = value
+ elif key == 'Description':
+ values['DESCRIPTION'] = value
+ indesc = True
+ return values
+
+def convert_debian(debpath):
+ # FIXME extend this mapping - perhaps use distro_alias.inc?
+ depmap = {'libz-dev': 'zlib'}
+
+ values = {}
+ depends = []
+ with open(os.path.join(debpath, 'control')) as f:
+ indesc = False
+ for line in f:
+ if indesc:
+ if line.strip():
+ if line.startswith(' This package contains'):
+ indesc = False
+ else:
+ values['DESCRIPTION'] += ' ' + line.strip()
+ else:
+ indesc = False
+ else:
+ splitline = line.split(':', 1)
+ key = line[0]
+ value = line[1]
+ if key == 'Build-Depends':
+ for dep in value.split(','):
+ dep = dep.split()[0]
+ mapped = depmap.get(dep, '')
+ if mapped:
+ depends.append(mapped)
+ elif key == 'Section':
+ values['SECTION'] = value
+ elif key == 'Description':
+ values['SUMMARY'] = value
+ indesc = True
+
+ if depends:
+ values['DEPENDS'] = ' '.join(depends)
+
+ return values
+
+
+def register_command(subparsers):
+ parser_create = subparsers.add_parser('create', help='Create a new recipe')
+ parser_create.add_argument('source', help='Path or URL to source')
+ parser_create.add_argument('-o', '--outfile', help='Full path and filename to recipe to add', required=True)
+ parser_create.add_argument('-m', '--machine', help='Make recipe machine-specific as opposed to architecture-specific', action='store_true')
+ parser_create.add_argument('-x', '--externalsrc', help='Assuming source is a URL, fetch it and extract it to the specified directory')
+ parser_create.set_defaults(func=create_recipe)
+
diff --git a/scripts/lib/recipetool/create_buildsys.py b/scripts/lib/recipetool/create_buildsys.py
new file mode 100644
index 0000000..98fd7f8
--- /dev/null
+++ b/scripts/lib/recipetool/create_buildsys.py
@@ -0,0 +1,234 @@
+# Recipe creation tool - create command build system handlers
+#
+# Copyright (C) 2014 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import re
+import logging
+from recipetool.create import RecipeHandler, read_pkgconfig_provides
+
+logger = logging.getLogger('recipetool')
+
+tinfoil = None
+
+def tinfoil_init(instance):
+ global tinfoil
+ tinfoil = instance
+
+class CmakeRecipeHandler(RecipeHandler):
+ def process(self, srctree, classes, lines_before, lines_after, handled):
+ if 'buildsystem' in handled:
+ return False
+
+ if self.checkfiles(srctree, ['CMakeLists.txt']):
+ classes.append('cmake')
+ lines_after.append('# Specify any options you want to pass to cmake using EXTRA_OECMAKE:')
+ lines_after.append('EXTRA_OECMAKE = ""')
+ lines_after.append('')
+ handled.append('buildsystem')
+ return True
+ return False
+
+class SconsRecipeHandler(RecipeHandler):
+ def process(self, srctree, classes, lines_before, lines_after, handled):
+ if 'buildsystem' in handled:
+ return False
+
+ if self.checkfiles(srctree, ['SConstruct', 'Sconstruct', 'sconstruct']):
+ classes.append('scons')
+ lines_after.append('# Specify any options you want to pass to scons using EXTRA_OESCONS:')
+ lines_after.append('EXTRA_OESCONS = ""')
+ lines_after.append('')
+ handled.append('buildsystem')
+ return True
+ return False
+
+class QmakeRecipeHandler(RecipeHandler):
+ def process(self, srctree, classes, lines_before, lines_after, handled):
+ if 'buildsystem' in handled:
+ return False
+
+ if self.checkfiles(srctree, ['*.pro']):
+ classes.append('qmake2')
+ handled.append('buildsystem')
+ return True
+ return False
+
+class AutotoolsRecipeHandler(RecipeHandler):
+ def process(self, srctree, classes, lines_before, lines_after, handled):
+ if 'buildsystem' in handled:
+ return False
+
+ autoconf = False
+ if self.checkfiles(srctree, ['configure.ac', 'configure.in']):
+ autoconf = True
+ values = self._extract_autotools_deps(lines_before, srctree)
+ classes.extend(values.pop('inherit', '').split())
+ for var, value in values.iteritems():
+ lines_before.append('%s = "%s"' % (var, value))
+ else:
+ conffile = self.checkfiles(srctree, ['configure'])
+ if conffile:
+ # Check if this is just a pre-generated autoconf configure script
+ with open(conffile[0], 'r') as f:
+ for i in range(1, 10):
+ if 'Generated by GNU Autoconf' in f.readline():
+ autoconf = True
+ break
+
+ if autoconf:
+ lines_before.append('# NOTE: if this software is not capable of being built in a separate build directory')
+ lines_before.append('# from the source, you should replace autotools with autotools-brokensep in the')
+ lines_before.append('# inherit line')
+ classes.append('autotools')
+ lines_after.append('# Specify any options you want to pass to the configure script using EXTRA_OECONF:')
+ lines_after.append('EXTRA_OECONF = ""')
+ lines_after.append('')
+ handled.append('buildsystem')
+ return True
+
+ return False
+
+ def _extract_autotools_deps(self, outlines, srctree):
+ import shlex
+
+ values = {}
+ inherits = []
+
+ # FIXME this mapping is very thin
+ progmap = {'flex': 'flex-native',
+ 'bison': 'bison-native',
+ 'm4': 'm4-native'}
+ progclassmap = {'gconftool-2': 'gconf',
+ 'pkg-config': 'pkgconfig'}
+
+ pkg_re = re.compile('PKG_CHECK_MODULES\(\[?[a-zA-Z0-9]*\]?, \[?([^,\]]*)[),].*')
+ progs_re = re.compile('_PROGS?\(\[?[a-zA-Z0-9]*\]?, \[?([^,\]]*)\]?[),].*')
+ dep_re = re.compile('([^ ><=]+)( [<>=]+ [^ ><=]+)?')
+
+ # Since a configure.ac file is essentially a program, this is only ever going to be
+ # a hack unfortunately; but it ought to be enough of an approximation
+ srcfiles = self.checkfiles(srctree, ['configure.ac', 'configure.in'])
+ pcdeps = []
+ deps = []
+ unmapped = []
+ with open(srcfiles[0], 'r') as f:
+ for line in f:
+ if 'PKG_CHECK_MODULES' in line:
+ res = pkg_re.search(line)
+ if res:
+ res = dep_re.findall(res.group(1))
+ if res:
+ pcdeps.extend([x[0] for x in res])
+ inherits.append('pkgconfig')
+ if line.lstrip().startswith('AM_GNU_GETTEXT'):
+ inherits.append('gettext')
+ elif 'AC_CHECK_PROG' in line or 'AC_PATH_PROG' in line:
+ res = progs_re.search(line)
+ if res:
+ for prog in shlex.split(res.group(1)):
+ prog = prog.split()[0]
+ progclass = progclassmap.get(prog, None)
+ if progclass:
+ inherits.append(progclass)
+ else:
+ progdep = progmap.get(prog, None)
+ if progdep:
+ deps.append(progdep)
+ else:
+ unmapped.append(prog)
+
+ if unmapped:
+ outlines.append('# NOTE: the following prog dependencies are unknown, ignoring: %s' % ' '.join(unmapped))
+
+ recipemap = read_pkgconfig_provides(tinfoil.config_data)
+ unmapped = []
+ for pcdep in pcdeps:
+ recipe = recipemap.get(pcdep, None)
+ if recipe:
+ deps.append(recipe)
+ else:
+ unmapped.append(pcdep)
+
+ if unmapped:
+ outlines.append('# WARNING: unable to map the following pkg-config dependencies: %s' % ' '.join(unmapped))
+ if deps:
+ values['DEPENDS'] = ' '.join(list(set(deps)))
+
+ if inherits:
+ values['inherit'] = ' '.join(list(set(inherits)))
+
+ return values
+
+
+class MakefileRecipeHandler(RecipeHandler):
+ def process(self, srctree, classes, lines_before, lines_after, handled):
+ if 'buildsystem' in handled:
+ return False
+
+ makefile = self.checkfiles(srctree, ['Makefile'])
+ if makefile:
+ lines_after.append('# NOTE: this is a Makefile-only piece of software, so we cannot generate much of the')
+ lines_after.append('# recipe automatically - you will need to examine the Makefile yourself and ensure')
+ lines_after.append('# that the appropriate arguments are passed in.')
+ lines_after.append('')
+
+ self.genfunction(lines_after, 'do_configure', ['# Specify any needed configure commands here'])
+
+ func = []
+ func.append('# You will almost certainly need to add additional arguments here')
+ func.append('oe_runmake')
+ self.genfunction(lines_after, 'do_compile', func)
+
+ installtarget = True
+ try:
+ stdout, stderr = bb.process.run('make -qn install', cwd=srctree, shell=True)
+ except bb.process.ExecutionError as e:
+ if e.exitcode != 1:
+ installtarget = False
+ func = []
+ if installtarget:
+ func.append('# This is a guess; additional arguments may be required')
+ makeargs = ''
+ with open(makefile[0], 'r') as f:
+ for i in range(1, 100):
+ if 'DESTDIR' in f.readline():
+ makeargs += " 'DESTDIR=${D}'"
+ break
+ func.append('oe_runmake install%s' % makeargs)
+ else:
+ func.append('# NOTE: unable to determine what to put here - there is a Makefile but no')
+ func.append('# target named "install", so you will need to define this yourself')
+ self.genfunction(lines_after, 'do_install', func)
+
+ handled.append('buildsystem')
+ else:
+ lines_after.append('# NOTE: no Makefile found, unable to determine what needs to be done')
+ lines_after.append('')
+ self.genfunction(lines_after, 'do_configure', ['# Specify any needed configure commands here'])
+ self.genfunction(lines_after, 'do_compile', ['# Specify compilation commands here'])
+ self.genfunction(lines_after, 'do_install', ['# Specify install commands here'])
+
+
+def plugin_init(pluginlist):
+ pass
+
+def register_recipe_handlers(handlers):
+ # These are in a specific order so that the right one is detected first
+ handlers.append(CmakeRecipeHandler())
+ handlers.append(AutotoolsRecipeHandler())
+ handlers.append(SconsRecipeHandler())
+ handlers.append(QmakeRecipeHandler())
+ handlers.append(MakefileRecipeHandler())
diff --git a/scripts/recipetool b/scripts/recipetool
new file mode 100755
index 0000000..0e2f527
--- /dev/null
+++ b/scripts/recipetool
@@ -0,0 +1,124 @@
+#!/usr/bin/env python
+
+# Recipe creation tool
+#
+# Copyright (C) 2014 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import sys
+import os
+import argparse
+import glob
+import logging
+
+def logger_create(name):
+ logger = logging.getLogger(name)
+ loggerhandler = logging.StreamHandler()
+ loggerhandler.setFormatter(logging.Formatter("%(levelname)s: %(message)s"))
+ logger.addHandler(loggerhandler)
+ logger.setLevel(logging.INFO)
+ return logger
+logger = logger_create('recipetool')
+
+plugins = []
+
+def load_plugins(pluginpath):
+ global plugins
+
+ def load_plugin(name):
+ logger.debug('Loading plugin %s' % name)
+ fp, pathname, description = imp.find_module(name, [pluginpath])
+ try:
+ return imp.load_module(name, fp, pathname, description)
+ finally:
+ if fp:
+ fp.close()
+
+ logger.debug('Loading plugins from %s...' % pluginpath)
+ import imp
+ for fn in glob.glob(os.path.join(pluginpath, '*.py')):
+ name = os.path.splitext(os.path.basename(fn))[0]
+ if name != '__init__':
+ plugin = load_plugin(name)
+ if hasattr(plugin, 'plugin_init'):
+ plugin.plugin_init(plugins)
+ plugins.append(plugin)
+
+def tinfoil_init():
+ import bb.tinfoil
+ import logging
+ tinfoil = bb.tinfoil.Tinfoil()
+ tinfoil.prepare(True)
+
+ for plugin in plugins:
+ if hasattr(plugin, 'tinfoil_init'):
+ plugin.tinfoil_init(tinfoil)
+ tinfoil.logger.setLevel(logging.WARNING)
+
+def main():
+
+ scripts_path = os.path.abspath(os.path.dirname(os.path.abspath(sys.argv[0])))
+ lib_path = scripts_path + '/lib'
+ sys.path = sys.path + [lib_path]
+
+ if not os.environ.get('BUILDDIR', ''):
+ logger.error("This script can only be run after initialising the build environment (e.g. by using oe-init-build-env)")
+ sys.exit(1)
+
+ parser = argparse.ArgumentParser(description="Recipe tool")
+ parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true')
+ parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true')
+ subparsers = parser.add_subparsers()
+
+ load_plugins(os.path.join(os.path.dirname(__file__), 'lib', 'recipetool'))
+ registered = False
+ for plugin in plugins:
+ if hasattr(plugin, 'register_command'):
+ registered = True
+ plugin.register_command(subparsers)
+
+ if not registered:
+ logger.error("No commands registered - missing plugins?")
+ sys.exit(1)
+
+ args = parser.parse_args()
+
+ if args.debug:
+ logger.setLevel(logging.DEBUG)
+ elif args.quiet:
+ logger.setLevel(logging.ERROR)
+
+ import scriptpath
+ bitbakepath = scriptpath.add_bitbake_lib_path()
+ if not bitbakepath:
+ logger.error("Unable to find bitbake by searching parent directory of this script or PATH")
+ sys.exit(1)
+ logger.debug('Found bitbake path: %s' % bitbakepath)
+
+ tinfoil_init()
+
+ ret = args.func(args)
+
+ return ret
+
+
+if __name__ == "__main__":
+ try:
+ ret = main()
+ except Exception:
+ ret = 1
+ import traceback
+ traceback.print_exc(5)
+ sys.exit(ret)
--
1.9.3
^ permalink raw reply related [flat|nested] 40+ messages in thread* [RFC PATCH 5/7] lib/oe: add recipeutils module
2014-11-25 17:28 [RFC PATCH 0/7] Developer workflow improvements Paul Eggleton
` (3 preceding siblings ...)
2014-11-25 17:28 ` [RFC PATCH 4/7] scripts/recipetool: Add a recipe auto-creation script Paul Eggleton
@ 2014-11-25 17:28 ` Paul Eggleton
2014-11-25 17:28 ` [RFC PATCH 6/7] scripts/devtool: add development helper tool Paul Eggleton
` (6 subsequent siblings)
11 siblings, 0 replies; 40+ messages in thread
From: Paul Eggleton @ 2014-11-25 17:28 UTC (permalink / raw)
To: openembedded-core
Add a module to help provide utility functions for dealing with recipes.
This would typically be used by external tools.
Substantial portions of this module were borrowed from the OE Layer
index code; other functions originally contributed by
Markus Lehtonen <markus.lehtonen@intel.com>.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/recipeutils.py | 264 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 264 insertions(+)
create mode 100644 meta/lib/oe/recipeutils.py
diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
new file mode 100644
index 0000000..2597765
--- /dev/null
+++ b/meta/lib/oe/recipeutils.py
@@ -0,0 +1,264 @@
+# Utility functions for reading and modifying recipes
+#
+# Some code borrowed from the OE layer index
+#
+# Copyright (C) 2013-2014 Intel Corporation
+#
+
+import sys
+import os
+import os.path
+import tempfile
+import textwrap
+import difflib
+import utils
+import shutil
+import re
+from collections import OrderedDict, defaultdict
+
+
+# Help us to find places to insert values
+recipe_progression = ['SUMMARY', 'DESCRIPTION', 'HOMEPAGE', 'BUGTRACKER', 'SECTION', 'LICENSE', 'LIC_FILES_CHKSUM', 'PROVIDES', 'DEPENDS', 'PR', 'PV', 'SRC_URI', 'do_fetch', 'do_unpack', 'do_patch', 'EXTRA_OECONF', 'do_configure', 'EXTRA_OEMAKE', 'do_compile', 'do_install', 'do_populate_sysroot', 'INITSCRIPT', 'USERADD', 'GROUPADD', 'PACKAGES', 'FILES', 'RDEPENDS', 'RRECOMMENDS', 'RSUGGESTS', 'RPROVIDES', 'RREPLACES', 'RCONFLICTS', 'ALLOW_EMPTY', 'do_package', 'do_deploy']
+# Variables that sometimes are a bit long but shouldn't be wrapped
+nowrap_vars = ['SUMMARY', 'HOMEPAGE', 'BUGTRACKER', 'LIC_FILES_CHKSUM']
+meta_vars = ['SUMMARY', 'DESCRIPTION', 'HOMEPAGE', 'BUGTRACKER', 'SECTION']
+
+
+def pn_to_recipe(cooker, pn):
+ """Convert a recipe name (PN) to the path to the recipe file"""
+ import bb.providers
+
+ if pn in cooker.recipecache.pkg_pn:
+ filenames = cooker.recipecache.pkg_pn[pn]
+ best = bb.providers.findBestProvider(pn, cooker.data, cooker.recipecache, cooker.recipecache.pkg_pn)
+ return best[3]
+ else:
+ return None
+
+
+def parse_recipe(fn, d):
+ """Parse an individual recipe"""
+ import bb.cache
+ envdata = bb.cache.Cache.loadDataFull(fn, [], d)
+ return envdata
+
+
+def get_var_files(fn, varlist, d):
+ """Find the file in which each of a list of variables is set.
+ Note: requires variable history to be enabled when parsing.
+ """
+ envdata = parse_recipe(fn, d)
+ varfiles = {}
+ for v in varlist:
+ history = envdata.varhistory.variable(v)
+ files = []
+ for event in history:
+ if 'file' in event and not 'flag' in event:
+ files.append(event['file'])
+ if files:
+ actualfile = files[-1]
+ else:
+ actualfile = None
+ varfiles[v] = actualfile
+
+ return varfiles
+
+
+def patch_recipe_file(fn, values, patch=False, relpath=''):
+ """Update or insert variable values into a recipe file (assuming you
+ have already identified the exact file you want to update.)
+ Note that some manual inspection/intervention may be required
+ since this cannot handle all situations.
+ """
+ remainingnames = {}
+ for k in values.keys():
+ remainingnames[k] = recipe_progression.index(k) if k in recipe_progression else -1
+ remainingnames = OrderedDict(sorted(remainingnames.iteritems(), key=lambda x: x[1]))
+
+ with tempfile.NamedTemporaryFile('w', delete=False) as tf:
+ def outputvalue(name):
+ rawtext = '%s = "%s"\n' % (name, values[name])
+ if name in nowrap_vars:
+ tf.write(rawtext)
+ else:
+ wrapped = textwrap.wrap(rawtext)
+ for wrapline in wrapped[:-1]:
+ tf.write('%s \\\n' % wrapline)
+ tf.write('%s\n' % wrapped[-1])
+
+ tfn = tf.name
+ with open(fn, 'r') as f:
+ # First runthrough - find existing names (so we know not to insert based on recipe_progression)
+ # Second runthrough - make the changes
+ existingnames = []
+ for runthrough in [1, 2]:
+ currname = None
+ for line in f:
+ if not currname:
+ insert = False
+ for k in remainingnames.keys():
+ for p in recipe_progression:
+ if re.match('^%s[ ?:=]' % p, line):
+ if remainingnames[k] > -1 and recipe_progression.index(p) > remainingnames[k] and runthrough > 1 and not k in existingnames:
+ outputvalue(k)
+ del remainingnames[k]
+ break
+ for k in remainingnames.keys():
+ if re.match('^%s[ ?:=]' % k, line):
+ currname = k
+ if runthrough == 1:
+ existingnames.append(k)
+ else:
+ del remainingnames[k]
+ break
+ if currname and runthrough > 1:
+ outputvalue(currname)
+
+ if currname:
+ sline = line.rstrip()
+ if not sline.endswith('\\'):
+ currname = None
+ continue
+ if runthrough > 1:
+ tf.write(line)
+ f.seek(0)
+ if remainingnames:
+ tf.write('\n')
+ for k in remainingnames.keys():
+ outputvalue(k)
+
+ with open(tfn, 'U') as f:
+ tolines = f.readlines()
+ if patch:
+ with open(fn, 'U') as f:
+ fromlines = f.readlines()
+ relfn = os.path.relpath(fn, relpath)
+ diff = difflib.unified_diff(fromlines, tolines, 'a/%s' % relfn, 'b/%s' % relfn)
+ os.remove(tfn)
+ return diff
+ else:
+ with open(fn, 'w') as f:
+ f.writelines(tolines)
+ os.remove(tfn)
+ return None
+
+def localise_file_vars(fn, varfiles, varlist):
+ """Given a list of variables and variable history (fetched with get_var_files())
+ find where each variable should be set/changed. This handles for example where a
+ recipe includes an inc file where variables might be changed - in most cases
+ we want to update the inc file when changing the variable value rather than adding
+ it to the recipe itself.
+ """
+ fndir = os.path.dirname(fn) + os.sep
+
+ first_meta_file = None
+ for v in meta_vars:
+ f = varfiles.get(v, None)
+ if f:
+ actualdir = os.path.dirname(f) + os.sep
+ if actualdir.startswith(fndir):
+ first_meta_file = f
+ break
+
+ filevars = defaultdict(list)
+ for v in varlist:
+ f = varfiles[v]
+ # Only return files that are in the same directory as the recipe or in some directory below there
+ # (this excludes bbclass files and common inc files that wouldn't be appropriate to set the variable
+ # in if we were going to set a value specific to this recipe)
+ if f:
+ actualfile = f
+ else:
+ # Variable isn't in a file, if it's one of the "meta" vars, use the first file with a meta var in it
+ if first_meta_file:
+ actualfile = first_meta_file
+ else:
+ actualfile = fn
+
+ actualdir = os.path.dirname(actualfile) + os.sep
+ if not actualdir.startswith(fndir):
+ actualfile = fn
+ filevars[actualfile].append(v)
+
+ return filevars
+
+def patch_recipe(d, fn, varvalues, patch=False, relpath=''):
+ """Modify a list of variable values in the specified recipe. Handles inc files if
+ used by the recipe.
+ """
+ varlist = varvalues.keys()
+ varfiles = get_var_files(fn, varlist, d)
+ locs = localise_file_vars(fn, varfiles, varlist)
+ patches = []
+ for f,v in locs.iteritems():
+ vals = {k: varvalues[k] for k in v}
+ patchdata = patch_recipe_file(f, vals, patch, relpath)
+ if patch:
+ patches.append(patchdata)
+
+ if patch:
+ return patches
+ else:
+ return None
+
+
+
+def copy_recipe_files(d, tgt_dir, whole_dir=False, download=True):
+ """Copy (local) recipe files, including both files included via include/require,
+ and files referred to in the SRC_URI variable."""
+ import bb.fetch2
+ import oe.path
+
+ # FIXME need a warning if the unexpanded SRC_URI value contains variable references
+
+ uris = (d.getVar('SRC_URI', True) or "").split()
+ fetch = bb.fetch2.Fetch(uris, d)
+ if download:
+ fetch.download()
+
+ # Copy local files to target directory and gather any remote files
+ bb_dir = os.path.dirname(d.getVar('FILE', True)) + os.sep
+ remotes = []
+ includes = [path for path in d.getVar('BBINCLUDED', True).split() if
+ path.startswith(bb_dir) and os.path.exists(path)]
+ for path in fetch.localpaths() + includes:
+ # Only import files that are under the meta directory
+ if path.startswith(bb_dir):
+ if not whole_dir:
+ relpath = os.path.relpath(path, bb_dir)
+ subdir = os.path.join(tgt_dir, os.path.dirname(relpath))
+ if not os.path.exists(subdir):
+ os.makedirs(subdir)
+ shutil.copy2(path, os.path.join(tgt_dir, relpath))
+ else:
+ remotes.append(path)
+ # Simply copy whole meta dir, if requested
+ if whole_dir:
+ shutil.copytree(bb_dir, tgt_dir)
+
+ return remotes
+
+
+def get_recipe_patches(d):
+ """Get a list of the patches included in SRC_URI within a recipe."""
+ patchfiles = []
+ # Execute src_patches() defined in patch.bbclass - this works since that class
+ # is inherited globally
+ patches = bb.utils.exec_flat_python_func('src_patches', d)
+ for patch in patches:
+ _, _, local, _, _, parm = bb.fetch.decodeurl(patch)
+ patchfiles.append(local)
+ return patchfiles
+
+
+def validate_pn(pn):
+ """Perform validation on a recipe name (PN) for a new recipe."""
+ reserved_names = ['forcevariable', 'append', 'prepend', 'remove']
+ if not re.match('[0-9a-z-]+', pn):
+ return 'Recipe name "%s" is invalid: only characters 0-9, a-z and - are allowed' % pn
+ elif pn in reserved_names:
+ return 'Recipe name "%s" is invalid: is a reserved keyword' % pn
+ elif pn.startswith('pn-'):
+ return 'Recipe name "%s" is invalid: names starting with "pn-" are reserved' % pn
+ return ''
+
--
1.9.3
^ permalink raw reply related [flat|nested] 40+ messages in thread* [RFC PATCH 6/7] scripts/devtool: add development helper tool
2014-11-25 17:28 [RFC PATCH 0/7] Developer workflow improvements Paul Eggleton
` (4 preceding siblings ...)
2014-11-25 17:28 ` [RFC PATCH 5/7] lib/oe: add recipeutils module Paul Eggleton
@ 2014-11-25 17:28 ` Paul Eggleton
2014-11-25 17:28 ` [RFC PATCH 7/7] scripts/devtool: Support deploy/undeploy function Paul Eggleton
` (5 subsequent siblings)
11 siblings, 0 replies; 40+ messages in thread
From: Paul Eggleton @ 2014-11-25 17:28 UTC (permalink / raw)
To: openembedded-core
Provides an easy means to work on developing applications and system
components with the build system.
You can start by creating a workspace layer (only needs to be done
once):
$ devtool create-workspace
Then you can use the other functions, for example to "modify" the source
for an existing recipe:
$ devtool modify -x pango /home/projects/pango
Parsing recipes..done.
INFO: Fetching pango...
INFO: Unpacking...
INFO: Patching...
INFO: Source tree extracted to /tmp/woot2/
INFO: Recipe pango now set up to build from /home/projects/pango
The pango source is now extracted to /home/projects/pango, managed in
git, with each patch as a commit, and a bbappend is created in the
workspace layer to use the source in /home/projects/pango when building.
Additionally, you can add a new piece of software:
$ devtool add pv /home/projects/pv
INFO: Recipe /path/to/workspace/recipes/pv/pv.bb has been
automatically created; further editing may be required to make it
fully functional
The latter uses recipetool to create a skeleton recipe and again sets up
a bbappend to use the source in /home/projects/pv when building.
[YOCTO #6561]
[YOCTO #6653]
[YOCTO #6656]
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/devtool | 223 +++++++++++++++++++++++
scripts/lib/devtool/__init__.py | 79 ++++++++
scripts/lib/devtool/standard.py | 390 ++++++++++++++++++++++++++++++++++++++++
3 files changed, 692 insertions(+)
create mode 100755 scripts/devtool
create mode 100644 scripts/lib/devtool/__init__.py
create mode 100644 scripts/lib/devtool/standard.py
diff --git a/scripts/devtool b/scripts/devtool
new file mode 100755
index 0000000..eea93e0
--- /dev/null
+++ b/scripts/devtool
@@ -0,0 +1,223 @@
+#!/usr/bin/env python
+
+# OpenEmbedded Development tool
+#
+# Copyright (C) 2014 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import sys
+import os
+import argparse
+import glob
+import ConfigParser
+import subprocess
+import logging
+
+basepath = ''
+workspace = {}
+config = None
+context = None
+
+def logger_create(name):
+ logger = logging.getLogger(name)
+ loggerhandler = logging.StreamHandler()
+ loggerhandler.setFormatter(logging.Formatter("%(levelname)s: %(message)s"))
+ logger.addHandler(loggerhandler)
+ logger.setLevel(logging.INFO)
+ return logger
+logger = logger_create('devtool')
+
+plugins = []
+
+def load_plugins(pluginpath):
+ global plugins
+
+ def load_plugin(name):
+ logger.debug('Loading plugin %s' % name)
+ fp, pathname, description = imp.find_module(name, [pluginpath])
+ try:
+ return imp.load_module(name, fp, pathname, description)
+ finally:
+ if fp:
+ fp.close()
+
+ logger.debug('Loading plugins from %s...' % pluginpath)
+ import imp
+ for fn in glob.glob(os.path.join(pluginpath, '*.py')):
+ name = os.path.splitext(os.path.basename(fn))[0]
+ if name != '__init__':
+ plugin = load_plugin(name)
+ if hasattr(plugin, 'plugin_init'):
+ plugin.plugin_init(plugins)
+ plugins.append(plugin)
+
+
+class ConfigHandler(object):
+ config_file = ''
+ config_obj = None
+ init_path = ''
+ workspace_path = ''
+
+ def __init__(self, filename):
+ self.config_file = filename
+ self.config_obj = ConfigParser.SafeConfigParser()
+
+ def get(self, section, option, default=None):
+ try:
+ ret = self.config_obj.get(section, option)
+ except (ConfigParser.NoOptionError, ConfigParser.NoSectionError):
+ if default != None:
+ ret = default
+ else:
+ raise
+ return ret
+
+ def read(self):
+ if os.path.exists(self.config_file):
+ self.config_obj.read(self.config_file)
+
+ if self.config_obj.has_option('General', 'init_path'):
+ pth = self.get('General', 'init_path')
+ self.init_path = os.path.join(basepath, pth)
+ if not os.path.exists(self.init_path):
+ logger.error('init_path %s specified in config file cannot be found' % pth)
+ return False
+
+ self.workspace_path = self.get('General', 'workspace_path', os.path.join(basepath, 'workspace'))
+
+ return True
+ else:
+ self.config_obj.add_section('General')
+ return True
+
+ def write(self):
+ self.config_obj.set('General', 'workspace_path', self.workspace_path)
+ print self.config_file
+ with open(self.config_file, 'w') as f:
+ self.config_obj.write(f)
+
+class Context:
+ def __init__(self, **kwargs):
+ self.__dict__.update(kwargs)
+
+
+def read_workspace():
+ global workspace
+ workspace = {}
+ if not os.path.exists(os.path.join(config.workspace_path, 'conf', 'layer.conf')):
+ if context.fixed_setup:
+ logger.error("workspace layer not set up")
+ else:
+ logger.error("workspace layer not set up - you can create one by running %s create-workspace" % os.path.basename(sys.argv[0]))
+ sys.exit(1)
+
+ logger.debug('Reading workspace in %s' % config.workspace_path)
+ for fn in glob.glob(os.path.join(config.workspace_path, 'appends', '*.bbappend')):
+ pn = os.path.splitext(os.path.basename(fn))[0].split('_')[0]
+ with open(fn, 'r') as f:
+ for line in f:
+ if line.startswith('EXTERNALSRC ='):
+ splitval = line.split('=', 2)
+ workspace[pn] = splitval[1].strip('" \n\r\t')
+ break
+
+def main():
+ global basepath
+ global config
+ global context
+
+ context = Context(fixed_setup=False)
+
+ # Default basepath
+ basepath = os.path.dirname(os.path.abspath(__file__))
+ pth = basepath
+ while pth != '' and pth != os.sep:
+ if os.path.exists(os.path.join(pth, '.devtoolbase')):
+ context.fixed_setup = True
+ basepath = pth
+ break
+ pth = os.path.dirname(pth)
+
+ scripts_path = os.path.dirname(os.path.realpath(__file__))
+ lib_path = scripts_path + '/lib'
+ sys.path = sys.path + [lib_path]
+
+ parser = argparse.ArgumentParser(description="OpenEmbedded development tool")
+ parser.add_argument('--basepath', help='Base directory of SDK / build directory')
+ parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true')
+ parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true')
+
+ subparsers = parser.add_subparsers(dest="subparser_name")
+
+ load_plugins(os.path.join(scripts_path, 'lib', 'devtool'))
+ for plugin in plugins:
+ if hasattr(plugin, 'register_commands'):
+ plugin.register_commands(subparsers, context)
+
+ args = parser.parse_args()
+
+ if args.debug:
+ logger.setLevel(logging.DEBUG)
+ elif args.quiet:
+ logger.setLevel(logging.ERROR)
+
+ if args.basepath:
+ # Override
+ basepath = args.basepath
+ elif not context.fixed_setup:
+ basepath = os.environ.get('BUILDDIR')
+ if not basepath:
+ logger.error("This script can only be run after initialising the build environment (e.g. by using oe-init-build-env)")
+ sys.exit(1)
+
+ logger.debug('Using basepath %s' % basepath)
+
+ config = ConfigHandler(os.path.join(basepath, 'devtool.conf'))
+ if not config.read():
+ return -1
+
+ bitbake_subdir = config.get('General', 'bitbake_subdir', '')
+ if bitbake_subdir:
+ # Normally set for use within the SDK
+ logger.debug('Using bitbake subdir %s' % bitbake_subdir)
+ sys.path.insert(0, os.path.join(basepath, bitbake_subdir, 'lib'))
+ core_meta_subdir = config.get('General', 'core_meta_subdir')
+ sys.path.insert(0, os.path.join(basepath, core_meta_subdir, 'lib'))
+ else:
+ # Standard location
+ import scriptpath
+ bitbakepath = scriptpath.add_bitbake_lib_path()
+ if not bitbakepath:
+ logger.error("Unable to find bitbake by searching parent directory of this script or PATH")
+ sys.exit(1)
+ logger.debug('Using standard bitbake path %s' % bitbakepath)
+ scriptpath.add_oe_lib_path()
+
+ if args.subparser_name != 'create-workspace':
+ read_workspace()
+
+ ret = args.func(args, config, basepath, workspace)
+
+ return ret
+
+
+if __name__ == "__main__":
+ try:
+ ret = main()
+ except Exception:
+ ret = 1
+ import traceback
+ traceback.print_exc(5)
+ sys.exit(ret)
diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
new file mode 100644
index 0000000..3cad708
--- /dev/null
+++ b/scripts/lib/devtool/__init__.py
@@ -0,0 +1,79 @@
+#!/usr/bin/env python
+
+# Development tool - utility functions for plugins
+#
+# Copyright (C) 2014 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+
+import os
+import sys
+import subprocess
+import logging
+
+logger = logging.getLogger('devtool')
+
+def exec_build_env_command(init_path, builddir, cmd, watch=False, **options):
+ import bb
+ if not 'cwd' in options:
+ options["cwd"] = builddir
+ if init_path:
+ logger.debug('Executing command: "%s" using init path %s' % (cmd, init_path))
+ init_prefix = '. %s %s > /dev/null && ' % (init_path, builddir)
+ else:
+ logger.debug('Executing command "%s"' % cmd)
+ init_prefix = ''
+ if watch:
+ return exec_watch('%s%s' % (init_prefix, cmd), **options)
+ else:
+ return bb.process.run('%s%s' % (init_prefix, cmd), **options)
+
+def exec_watch(cmd, **options):
+ if isinstance(cmd, basestring) and not "shell" in options:
+ options["shell"] = True
+
+ process = subprocess.Popen(
+ cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, **options
+ )
+
+ buf = ''
+ while True:
+ out = process.stdout.read(1)
+ if out:
+ sys.stdout.write(out)
+ sys.stdout.flush()
+ buf += out
+ elif out == '' and process.poll() != None:
+ break
+ return buf
+
+def setup_tinfoil():
+ scripts_path = os.path.abspath(os.path.dirname(os.path.abspath(sys.argv[0])))
+ lib_path = scripts_path + '/lib'
+ sys.path = sys.path + [lib_path]
+
+ import scriptpath
+ bitbakepath = scriptpath.add_bitbake_lib_path()
+ if not bitbakepath:
+ logger.error("Unable to find bitbake by searching parent directory of this script or PATH")
+ sys.exit(1)
+
+ import bb.tinfoil
+ import logging
+ tinfoil = bb.tinfoil.Tinfoil()
+ tinfoil.prepare(False)
+ tinfoil.logger.setLevel(logging.WARNING)
+ return tinfoil
+
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
new file mode 100644
index 0000000..82c5b53
--- /dev/null
+++ b/scripts/lib/devtool/standard.py
@@ -0,0 +1,390 @@
+# Development tool - standard commands plugin
+#
+# Copyright (C) 2014 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import os
+import sys
+import re
+import shutil
+import glob
+import tempfile
+import logging
+from devtool import exec_build_env_command, setup_tinfoil
+
+logger = logging.getLogger('devtool')
+
+def plugin_init(pluginlist):
+ pass
+
+
+def create_workspace(args, config, basepath, workspace):
+ import bb
+
+ if args.directory:
+ workspacedir = os.path.abspath(args.directory)
+ else:
+ workspacedir = os.path.abspath(os.path.join(basepath, 'workspace'))
+
+ confdir = os.path.join(workspacedir, 'conf')
+ if os.path.exists(os.path.join(confdir, 'layer.conf')):
+ logger.info('Specified workspace already set up, leaving as-is')
+ else:
+ bb.utils.mkdirhier(confdir)
+ with open(os.path.join(confdir, 'layer.conf'), 'w') as f:
+ f.write('# ### workspace layer auto-generated by devtool ###\n')
+ f.write('BBPATH =. "$' + '{LAYERDIR}:"\n')
+ f.write('BBFILES += "$' + '{LAYERDIR}/recipes/*/*.bb \\\n')
+ f.write(' $' + '{LAYERDIR}/appends/*.bbappend"\n')
+ f.write('BBFILE_COLLECTIONS += "workspacelayer"\n')
+ f.write('BBFILE_PATTERN_workspacelayer = "^$' + '{LAYERDIR}/"\n')
+ f.write('BBFILE_PRIORITY_workspacelayer = "99"\n')
+ if not args.create_only:
+ # Add the workspace layer to bblayers.conf
+ bblayers_conf = os.path.join(basepath, 'conf', 'bblayers.conf')
+ if not os.path.exists(bblayers_conf):
+ logger.error('Unable to find bblayers.conf')
+ return -1
+ newlines = []
+ with open(bblayers_conf, 'r') as f:
+ bblayers_found = False
+ updated = False
+ in_bblayers = False
+ bblayers = ''
+ for line in f:
+ if in_bblayers:
+ value = line.rstrip()
+ bblayers += value[:-1]
+ if value.endswith('"'):
+ in_bblayers = False
+ bblayers = bblayers.split()
+ if workspacedir not in bblayers:
+ updated = True
+ bblayers.append(workspacedir)
+ if workspacedir != config.workspace_path and config.workspace_path in bblayers:
+ bblayers.remove(config.workspace_path)
+ newlines.append('BBLAYERS ?= " \\\n')
+ for layer in bblayers:
+ newlines.append(' %s \\\n' % layer)
+ newlines.append(' "\n')
+ else:
+ if line.startswith('BBLAYERS '):
+ bblayers_found = True
+ in_bblayers = True
+ value = line.split('"', 1)[1].rstrip()
+ if value.endswith('\\'):
+ value = value[:-1]
+ bblayers = value
+ else:
+ newlines.append(line)
+ if not bblayers_found:
+ logger.error('Found bblayers.conf but could not find BBLAYERS value to update')
+ return -1
+ if updated:
+ # Write out the new bblayers.conf
+ with open(bblayers_conf, 'w') as f:
+ f.writelines(newlines)
+ if config.workspace_path != workspacedir:
+ # Update our config to point to the new location
+ config.workspace_path = workspacedir
+ config.write()
+
+
+def add(args, config, basepath, workspace):
+ import bb
+ import oe.recipeutils
+
+ if args.recipename in workspace:
+ logger.error("recipe %s is already in your workspace" % args.recipename)
+ return -1
+
+ reason = oe.recipeutils.validate_pn(args.recipename)
+ if reason:
+ logger.error(reason)
+ return -1
+
+ srctree = os.path.abspath(args.srctree)
+ appendpath = os.path.join(config.workspace_path, 'appends')
+ if not os.path.exists(appendpath):
+ os.makedirs(appendpath)
+
+ recipedir = os.path.join(config.workspace_path, 'recipes', args.recipename)
+ bb.utils.mkdirhier(recipedir)
+ if args.version:
+ if '_' in args.version or ' ' in args.version:
+ logger.error('Invalid version string "%s"' % args.version)
+ return -1
+ bp = "%s_%s" % (args.recipename, args.version)
+ else:
+ bp = args.recipename
+ recipefile = os.path.join(recipedir, "%s.bb" % bp)
+ stdout, stderr = exec_build_env_command(config.init_path, basepath, 'recipetool create -o %s %s' % (recipefile, srctree))
+ logger.info('Recipe %s has been automatically created; further editing may be required to make it fully functional' % recipefile)
+
+ _add_md5(config, args.recipename, recipefile)
+
+ appendfile = os.path.join(appendpath, '%s.bbappend' % args.recipename)
+ with open(appendfile, 'w') as f:
+ f.write('inherit externalsrc\n')
+ f.write('EXTERNALSRC = "%s"\n' % srctree)
+ f.write('do_compile[nostamp] = "1"\n')
+
+ _add_md5(config, args.recipename, appendfile)
+
+ return 0
+
+
+def extract(args, config, basepath, workspace):
+ import bb
+ import oe.recipeutils
+
+ tinfoil = setup_tinfoil()
+
+ recipefile = oe.recipeutils.pn_to_recipe(tinfoil.cooker, args.recipename)
+ if not recipefile:
+ logger.error("Unable to find any recipe file matching %s" % args.recipename)
+ return -1
+ rd = oe.recipeutils.parse_recipe(recipefile, tinfoil.config_data)
+
+ srctree = os.path.abspath(args.srctree)
+ return _extract_source(srctree, rd)
+
+
+def _extract_source(srctree, d):
+ pn = d.getVar('PN', True)
+
+ if os.path.exists(srctree):
+ if not os.path.isdir(srctree):
+ logger.error("output path %s exists and is not a directory" % srctree)
+ return -1
+ elif os.listdir(srctree):
+ logger.error("output path %s already exists and is non-empty" % srctree)
+ return -1
+
+ # Prepare for shutil.move later on
+ bb.utils.mkdirhier(srctree)
+ os.rmdir(srctree)
+
+ tempdir = tempfile.mkdtemp(prefix='devtool')
+ try:
+ crd = d.createCopy()
+ # Make a subdir so we guard against WORKDIR==S
+ workdir = os.path.join(tempdir, 'workdir')
+ crd.setVar('WORKDIR', workdir)
+ crd.setVar('T', os.path.join(tempdir, 'temp'))
+ logger.info('Fetching %s...' % pn)
+ bb.build.exec_func('do_fetch', crd)
+ logger.info('Unpacking...')
+ bb.build.exec_func('do_unpack', crd)
+ srcsubdir = crd.getVar('S', True)
+ if srcsubdir != workdir and os.path.dirname(srcsubdir) != workdir:
+ # Handle if S is set to a subdirectory of the source
+ srcsubdir = os.path.join(workdir, os.path.relpath(srcsubdir, workdir).split(os.sep)[0])
+
+ patchdir = os.path.join(srcsubdir, 'patches')
+ haspatches = False
+ if os.path.exists(patchdir):
+ if os.listdir(patchdir):
+ haspatches = True
+ else:
+ os.rmdir(patchdir)
+
+ if not os.listdir(srcsubdir):
+ logger.error("no source unpacked to S, perhaps the %s recipe doesn't use any source?" % pn)
+ return -1
+
+ if not os.path.exists(os.path.join(srcsubdir, '.git')):
+ bb.process.run('git init', cwd=srcsubdir)
+ bb.process.run('git add .', cwd=srcsubdir)
+ bb.process.run('git commit -q -m "Initial commit from upstream at version %s"' % crd.getVar('PV', True), cwd=srcsubdir)
+ logger.info('Patching...')
+ crd.setVar('PATCHTOOL', 'git')
+ bb.build.exec_func('do_patch', crd)
+
+ if os.path.exists(patchdir):
+ shutil.rmtree(patchdir)
+ if haspatches:
+ bb.process.run('git checkout patches', cwd=srcsubdir)
+
+ shutil.move(srcsubdir, srctree)
+ logger.info('Source tree extracted to %s' % srctree)
+ finally:
+ shutil.rmtree(tempdir)
+ return 0
+
+def _add_md5(config, recipename, filename):
+ import bb.utils
+ md5 = bb.utils.md5_file(filename)
+ with open(os.path.join(config.workspace_path, '.devtool_md5'), 'a') as f:
+ f.write('%s|%s|%s\n' % (recipename, os.path.relpath(filename, config.workspace_path), md5))
+
+def _check_preserve(config, recipename):
+ import bb.utils
+ origfile = os.path.join(config.workspace_path, '.devtool_md5')
+ newfile = os.path.join(config.workspace_path, '.devtool_md5_new')
+ preservepath = os.path.join(config.workspace_path, 'attic')
+ with open(origfile, 'r') as f:
+ with open(newfile, 'w') as tf:
+ for line in f.readlines():
+ splitline = line.rstrip().split('|')
+ if splitline[0] == recipename:
+ removefile = os.path.join(config.workspace_path, splitline[1])
+ md5 = bb.utils.md5_file(removefile)
+ if splitline[2] != md5:
+ bb.utils.mkdirhier(preservepath)
+ preservefile = os.path.basename(removefile)
+ logger.warn('File %s modified since it was written, preserving in %s' % (preservefile, preservepath))
+ shutil.move(removefile, os.path.join(preservepath, preservefile))
+ else:
+ os.remove(removefile)
+ else:
+ tf.write(line)
+ os.rename(newfile, origfile)
+
+ return False
+
+
+def modify(args, config, basepath, workspace):
+ import bb
+ import oe.recipeutils
+
+ if args.recipename in workspace:
+ logger.error("recipe %s is already in your workspace" % args.recipename)
+ return -1
+
+ if not args.extract:
+ if not os.path.isdir(args.srctree):
+ logger.error("directory %s does not exist or not a directory (specify -x to extract source from recipe)" % args.srctree)
+ return -1
+
+ tinfoil = setup_tinfoil()
+
+ recipefile = oe.recipeutils.pn_to_recipe(tinfoil.cooker, args.recipename)
+ if not recipefile:
+ logger.error("Unable to find any recipe file matching %s" % args.recipename)
+ return -1
+ rd = oe.recipeutils.parse_recipe(recipefile, tinfoil.config_data)
+
+ srctree = os.path.abspath(args.srctree)
+ if args.extract:
+ ret = _extract_source(args.srctree, rd)
+ if ret:
+ return ret
+
+ # Handle if S is set to a subdirectory of the source
+ s = rd.getVar('S', True)
+ workdir = rd.getVar('WORKDIR', True)
+ if s != workdir and os.path.dirname(s) != workdir:
+ srcsubdir = os.sep.join(os.path.relpath(s, workdir).split(os.sep)[1:])
+ srctree = os.path.join(srctree, srcsubdir)
+
+ appendpath = os.path.join(config.workspace_path, 'appends')
+ if not os.path.exists(appendpath):
+ os.makedirs(appendpath)
+
+ appendname = os.path.splitext(os.path.basename(recipefile))[0]
+ if args.wildcard:
+ appendname = re.sub(r'_.*', '_%', appendname)
+ appendfile = os.path.join(appendpath, appendname + '.bbappend')
+ with open(appendfile, 'w') as f:
+ f.write('FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"\n\n')
+ f.write('inherit externalsrc\n')
+ f.write('EXTERNALSRC = "%s"\n' % srctree)
+ if bb.data.inherits_class('autotools-brokensep', rd):
+ logger.info('using source tree as build directory since original recipe inherits autotools-brokensep')
+ f.write('EXTERNALSRC_BUILD = "%s"\n' % srctree)
+ f.write('do_compile[nostamp] = "1"\n')
+
+ _add_md5(config, args.recipename, appendfile)
+
+ logger.info('Recipe %s now set up to build from %s' % (args.recipename, srctree))
+
+ return 0
+
+
+def status(args, config, basepath, workspace):
+ for recipe, value in workspace.iteritems():
+ print("%s: %s" % (recipe, value))
+ return 0
+
+
+def reset(args, config, basepath, workspace):
+ import bb.utils
+ if not args.recipename in workspace:
+ logger.error("no recipe named %s in your workspace" % args.recipename)
+ return -1
+ _check_preserve(config, args.recipename)
+
+ preservepath = os.path.join(config.workspace_path, 'attic', args.recipename)
+ def preservedir(origdir):
+ if os.path.exists(origdir):
+ for fn in os.listdir(origdir):
+ logger.warn('Preserving %s in %s' % (fn, preservepath))
+ bb.utils.mkdirhier(preservepath)
+ shutil.move(os.path.join(origdir, fn), os.path.join(preservepath, fn))
+ os.rmdir(origdir)
+
+ preservedir(os.path.join(config.workspace_path, 'recipes', args.recipename))
+ # We don't automatically create this dir next to appends, but the user can
+ preservedir(os.path.join(config.workspace_path, 'appends', args.recipename))
+ return 0
+
+
+def build(args, config, basepath, workspace):
+ import bb
+ if not args.recipename in workspace:
+ logger.error("no recipe named %s in your workspace" % args.recipename)
+ return -1
+ exec_build_env_command(config.init_path, basepath, 'bitbake -c install %s' % args.recipename, watch=True)
+
+ return 0
+
+
+def register_commands(subparsers, context):
+ if not context.fixed_setup:
+ parser_create_workspace = subparsers.add_parser('create-workspace', help='Set up a workspace')
+ parser_create_workspace.add_argument('directory', nargs='?', help='Directory for the workspace')
+ parser_create_workspace.add_argument('--create-only', action="store_true", help='Only create the workspace, do not alter configuration')
+ parser_create_workspace.set_defaults(func=create_workspace)
+
+ parser_add = subparsers.add_parser('add', help='Add a new recipe')
+ parser_add.add_argument('recipename', help='Name for new recipe to add')
+ parser_add.add_argument('srctree', help='Path to external source tree')
+ parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)')
+ parser_add.set_defaults(func=add)
+
+ parser_add = subparsers.add_parser('modify', help='Modify the source for an existing recipe')
+ parser_add.add_argument('recipename', help='Name for recipe to edit')
+ parser_add.add_argument('srctree', help='Path to external source tree')
+ parser_add.add_argument('--wildcard', '-w', action="store_true", help='Use wildcard for unversioned bbappend')
+ parser_add.add_argument('--extract', '-x', action="store_true", help='Extract source as well')
+ parser_add.set_defaults(func=modify)
+
+ parser_add = subparsers.add_parser('extract', help='Extract the source for an existing recipe')
+ parser_add.add_argument('recipename', help='Name for recipe to extract the source for')
+ parser_add.add_argument('srctree', help='Path to where to extract the source tree')
+ parser_add.set_defaults(func=extract)
+
+ parser_status = subparsers.add_parser('status', help='Show status')
+ parser_status.set_defaults(func=status)
+
+ parser_build = subparsers.add_parser('build', help='Build recipe')
+ parser_build.add_argument('recipename', help='Recipe to build')
+ parser_build.set_defaults(func=build)
+
+ parser_reset = subparsers.add_parser('reset', help='Remove a recipe from your workspace')
+ parser_reset.add_argument('recipename', help='Recipe to reset')
+ parser_reset.set_defaults(func=reset)
+
--
1.9.3
^ permalink raw reply related [flat|nested] 40+ messages in thread* [RFC PATCH 7/7] scripts/devtool: Support deploy/undeploy function
2014-11-25 17:28 [RFC PATCH 0/7] Developer workflow improvements Paul Eggleton
` (5 preceding siblings ...)
2014-11-25 17:28 ` [RFC PATCH 6/7] scripts/devtool: add development helper tool Paul Eggleton
@ 2014-11-25 17:28 ` Paul Eggleton
2014-11-25 17:51 ` [RFC PATCH 0/7] Developer workflow improvements Paul Barker
` (4 subsequent siblings)
11 siblings, 0 replies; 40+ messages in thread
From: Paul Eggleton @ 2014-11-25 17:28 UTC (permalink / raw)
To: openembedded-core
From: Junchun Guan <junchunx.guan@intel.com>
Deploy recipe output files to live target machine using scp
Store the files list and target machine info in localhost if deployment
is done
Undeploy recipe output files in target machine using the previous
deployment info
[YOCTO #6654]
Signed-off-by: Junchun Guan <junchunx.guan@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/deploy.py | 100 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 100 insertions(+)
create mode 100644 scripts/lib/devtool/deploy.py
diff --git a/scripts/lib/devtool/deploy.py b/scripts/lib/devtool/deploy.py
new file mode 100644
index 0000000..965179c
--- /dev/null
+++ b/scripts/lib/devtool/deploy.py
@@ -0,0 +1,100 @@
+# Development tool - deploy/undeploy command plugin
+#
+# Copyright (C) 2014 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import os
+import subprocess
+import logging
+from devtool import exec_build_env_command
+
+logger = logging.getLogger('devtool')
+
+def plugin_init(pluginlist):
+ pass
+
+
+def deploy(args, config, basepath, workspace):
+ import re
+ from devtool import exec_build_env_command
+
+ if not args.recipename in workspace:
+ logger.error("no recipe named %s in your workspace" % args.recipename)
+ return -1
+ try:
+ host, destdir = args.target.split(':')
+ except ValueError:
+ destdir = '/'
+ else:
+ args.target = host
+
+ deploy_dir = os.path.join(basepath, 'target_deploy', args.target)
+ deploy_file = os.path.join(deploy_dir, args.recipename + '.list')
+
+ if os.path.exists(deploy_file):
+ undeploy(args)
+
+ stdout, stderr = exec_build_env_command(config.init_path, basepath, 'bitbake -e %s' % args.recipename, shell=True)
+ recipe_outdir = re.search(r'^D="(.*)"', stdout, re.MULTILINE).group(1)
+ ret = subprocess.call('scp -qr %s/* %s:%s' % (recipe_outdir, args.target, destdir), shell=True)
+ if ret != 0:
+ return ret
+
+ logger.info('Successfully deployed %s' % recipe_outdir)
+
+ if not os.path.exists(deploy_dir):
+ os.makedirs(deploy_dir)
+
+ files_list = []
+ for root, _, files in os.walk(recipe_outdir):
+ for filename in files:
+ filename = os.path.relpath(os.path.join(root, filename), recipe_outdir)
+ files_list.append(os.path.join(destdir, filename))
+
+ with open(deploy_file, 'w') as fobj:
+ fobj.write('\n'.join(files_list))
+
+ return 0
+
+def undeploy(args, config, basepath, workspace):
+
+ deploy_file = os.path.join(basepath, 'target_deploy', args.target, args.recipename + '.list')
+ if not os.path.exists(deploy_file):
+ logger.error('%s has not been deployed' % args.recipename)
+ return -1
+
+ ret = subprocess.call("scp -q %s %s:/tmp" % (deploy_file, args.target), shell=True)
+ if ret != 0:
+ logger.error('Failed to copy %s to %s' % (deploy, args.target))
+ return -1
+
+ ret = subprocess.call("ssh %s 'xargs -n1 rm -f </tmp/%s'" % (args.target, os.path.basename(deploy_file)), shell=True)
+ if ret == 0:
+ logger.info('Successfully undeployed %s' % args.recipename)
+ os.remove(deploy_file)
+
+ return ret
+
+
+def register_commands(subparsers, context):
+ parser_deploy = subparsers.add_parser('deploy', help='Deploy recipe output files to live target machine')
+ parser_deploy.add_argument('recipename', help='Recipe to deploy')
+ parser_deploy.add_argument('target', help='Live target machine running an ssh server: user@hostname[:destdir]')
+ parser_deploy.set_defaults(func=deploy)
+
+ parser_undeploy = subparsers.add_parser('undeploy', help='Undeploy recipe output files in live target machine')
+ parser_undeploy.add_argument('recipename', help='Recipe to undeploy')
+ parser_undeploy.add_argument('target', help='Live target machine running an ssh server: user@hostname')
+ parser_undeploy.set_defaults(func=undeploy)
--
1.9.3
^ permalink raw reply related [flat|nested] 40+ messages in thread* Re: [RFC PATCH 0/7] Developer workflow improvements
2014-11-25 17:28 [RFC PATCH 0/7] Developer workflow improvements Paul Eggleton
` (6 preceding siblings ...)
2014-11-25 17:28 ` [RFC PATCH 7/7] scripts/devtool: Support deploy/undeploy function Paul Eggleton
@ 2014-11-25 17:51 ` Paul Barker
2014-11-25 18:34 ` Paul Eggleton
2014-11-25 19:56 ` Trevor Woerner
` (3 subsequent siblings)
11 siblings, 1 reply; 40+ messages in thread
From: Paul Barker @ 2014-11-25 17:51 UTC (permalink / raw)
To: Paul Eggleton; +Cc: OE Core
On 25 November 2014 at 17:28, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
> [Note: this is an RFC series for the moment, and shouldn't yet be merged.]
>
> I've been looking at how to make it easier for application and system
> component developers to get their work done using the tools we provide,
> and I believe this patchset is a piece of the solution. There's still a
> number of other pieces to come, but this should be usable on its own.
>
> The first three patches extend the PATCHTOOL = "git" code to ensure that
> we're able to apply all patches even if "git am" and "git apply" can't
> handle them, and that we do a commit to the repository per patch. This
> is needed for devtool later on.
>
> I've then added a new recipe auto-creation script, recipetool, which can
> take a source tree or URL and create a skeleton recipe to build it.
> (In case anyone is wondering about the existing scripts/create-recipe,
> frankly I consider it a dead end - it's written in Perl, which makes it
> a bit difficult to integrate with the rest of our code; it's also
> GPLv3-only which makes any such integration pretty much impossible from
> another angle.)
>
> Then we add devtool. This allows you to:
>
> * Add a new piece of software (auto-create the skeleton of a recipe
> using the aforementioned recipetool and point the build to an external
> source tree)
> * Modify the source for an existing recipe (point the build to an
> external source tree, possibly creating that tree in the same step
> and managing it with git)
> * Deploy the installed files from a recipe from ${D} to a target using
> scp.
>
> There will obviously be extensions to these tools, but I hope they are
> already functional enough to be useful at the state they are in at the
> moment.
>
> Known issues:
>
> * "devtool modify" can't currently deal with gcc, linux-yocto or perf
> because they manage their workdir differently, this will need to be
> fixed
> * devtool's workspace layer triggers a bitbake warning when empty.
> I suspect a mechanism to disable this warning internally will need to
> be added.
> * "recipetool create" ideally needs to become smarter and fill in
> more details of the recipe. At some point we'll probably have to
> make the process interactive and possibly have it automate parts of
> the build process and examine the output; some things just can't be
> practically detected from the source tree without building.
>
>
> (Note that some pieces of the code which are not essential rely on a couple
> of changes to bitbake, you can find these in the paule/devtool-bb branch
> in poky-contrib, or if you are happy to use poky, grab the paule/devtool
> branch in poky-contrib which has everything).
>
> Comments/questions/suggestions welcome!
>
This sounds great and looks like it will make some regular tasks quite
a bit easier!
One thing which may be useful within recipetool or devtool is a single
command to create patch files (using `git format-patch`), put them
into the relevant directory and add the file names to SRC_URI.
Obviously this assumes the external source tree is a git repo and we
can trace the local modifications back to either an external master
branch or an initial state after extracting a source archive. I'm just
thinking out loud here though, depends if other people think that work
flow is common enough to be worth creating a shortcut for.
Thanks,
--
Paul Barker
Email: paul@paulbarker.me.uk
http://www.paulbarker.me.uk
^ permalink raw reply [flat|nested] 40+ messages in thread* Re: [RFC PATCH 0/7] Developer workflow improvements
2014-11-25 17:51 ` [RFC PATCH 0/7] Developer workflow improvements Paul Barker
@ 2014-11-25 18:34 ` Paul Eggleton
0 siblings, 0 replies; 40+ messages in thread
From: Paul Eggleton @ 2014-11-25 18:34 UTC (permalink / raw)
To: Paul Barker; +Cc: OE Core
On Tuesday 25 November 2014 17:51:05 Paul Barker wrote:
> On 25 November 2014 at 17:28, Paul Eggleton
> <paul.eggleton@linux.intel.com> wrote:
> > I've been looking at how to make it easier for application and system
> > component developers to get their work done using the tools we provide,
> > and I believe this patchset is a piece of the solution. There's still a
> > number of other pieces to come, but this should be usable on its own.
> > ...
> > Comments/questions/suggestions welcome!
>
> This sounds great and looks like it will make some regular tasks quite
> a bit easier!
>
> One thing which may be useful within recipetool or devtool is a single
> command to create patch files (using `git format-patch`), put them
> into the relevant directory and add the file names to SRC_URI.
> Obviously this assumes the external source tree is a git repo and we
> can trace the local modifications back to either an external master
> branch or an initial state after extracting a source archive. I'm just
> thinking out loud here though, depends if other people think that work
> flow is common enough to be worth creating a shortcut for.
We've been thinking in this direction and it seems like a natural extension to
me, and one that I do hope to add. The tricky part is as you say, reconciling
the commits you have in the git tree with the patches you have in the recipe,
particularly when the user might have modified one or more of those commits.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC PATCH 0/7] Developer workflow improvements
2014-11-25 17:28 [RFC PATCH 0/7] Developer workflow improvements Paul Eggleton
` (7 preceding siblings ...)
2014-11-25 17:51 ` [RFC PATCH 0/7] Developer workflow improvements Paul Barker
@ 2014-11-25 19:56 ` Trevor Woerner
2014-11-26 9:02 ` Paul Eggleton
2014-11-28 17:28 ` Trevor Woerner
` (2 subsequent siblings)
11 siblings, 1 reply; 40+ messages in thread
From: Trevor Woerner @ 2014-11-25 19:56 UTC (permalink / raw)
To: Paul Eggleton, openembedded-core
Wow, this is exciting!
Just thinking out loud here... is there a way to way to have the
devshell come up without having applied any patches but with the patches
already queued in, say, a quilt series?
It seems to happen often enough to me that: a recipe has 10 local
patches which applied fine yesterday but now patch 7 is failing because
the upstream developers changed something. I would be nice to be able to
devshell to the point where patch 6 has been applied and patch 7 is
waiting to go on. Then I could apply the patch myself to figure out why
it's failing. Or just having all the patches ready to apply (in a quilt
series, as mentioned above) would be great too.
^ permalink raw reply [flat|nested] 40+ messages in thread* Re: [RFC PATCH 0/7] Developer workflow improvements
2014-11-25 19:56 ` Trevor Woerner
@ 2014-11-26 9:02 ` Paul Eggleton
0 siblings, 0 replies; 40+ messages in thread
From: Paul Eggleton @ 2014-11-26 9:02 UTC (permalink / raw)
To: Trevor Woerner; +Cc: openembedded-core
Hi Trevor,
On Tuesday 25 November 2014 14:56:14 Trevor Woerner wrote:
> Just thinking out loud here... is there a way to way to have the
> devshell come up without having applied any patches but with the patches
> already queued in, say, a quilt series?
>
> It seems to happen often enough to me that: a recipe has 10 local
> patches which applied fine yesterday but now patch 7 is failing because
> the upstream developers changed something. I would be nice to be able to
> devshell to the point where patch 6 has been applied and patch 7 is
> waiting to go on. Then I could apply the patch myself to figure out why
> it's failing. Or just having all the patches ready to apply (in a quilt
> series, as mentioned above) would be great too.
We do already have PATCHRESOLVE = "user" that's supposed to handle exactly
this, although to be honest I've never really felt comfortable using it (that
might have something to do with my lack of familiarity with quilt, though).
devtool is using the same code that we use for applying patches, which
supports quilt, so in theory it should be pretty easy to provide an option to
use quilt rather than git when doing "devtool modify" or "devtool extract" for
those that prefer it.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC PATCH 0/7] Developer workflow improvements
2014-11-25 17:28 [RFC PATCH 0/7] Developer workflow improvements Paul Eggleton
` (8 preceding siblings ...)
2014-11-25 19:56 ` Trevor Woerner
@ 2014-11-28 17:28 ` Trevor Woerner
2014-12-01 10:11 ` Paul Eggleton
2014-12-02 4:54 ` Trevor Woerner
[not found] ` <54866CD7.1050102@linaro.org>
11 siblings, 1 reply; 40+ messages in thread
From: Trevor Woerner @ 2014-11-28 17:28 UTC (permalink / raw)
To: Paul Eggleton, openembedded-core
Hi Paul,
These tools are really nice! Some thoughts/comments:
Maybe the "devtool.conf" that gets created could be placed in the
"conf/" subdirectory (along with the other configuration files such as
local.conf and bblayers.conf)?
Perhaps any recipe you're working on could be automatically included via
an IMAGE_INSTALL_append in conf/local.conf (or maybe that's too intrusive?)?
Do you envision users creating multiple workspaces? I'm wondering why
"devtool create-workspace" is required. Is there any advantage to
requiring users to create the workspace explicitly instead of just
having it be created implicitly?
Some of the commands to "devtool" include things like
- extract
- build
- deploy
- undeploy
but when a workspace is created, devtool (very intelligently) adds the
workspace to the set of BBLAYERS. So one could then just use bitbake to
build the recipe. Are there any advantages to using "devtool build
<recipe>" instead of "bitbake <recipe>"?
Best regards,
Trevor
^ permalink raw reply [flat|nested] 40+ messages in thread* Re: [RFC PATCH 0/7] Developer workflow improvements
2014-11-28 17:28 ` Trevor Woerner
@ 2014-12-01 10:11 ` Paul Eggleton
2014-12-02 4:36 ` Trevor Woerner
0 siblings, 1 reply; 40+ messages in thread
From: Paul Eggleton @ 2014-12-01 10:11 UTC (permalink / raw)
To: Trevor Woerner; +Cc: openembedded-core
On Friday 28 November 2014 12:28:00 Trevor Woerner wrote:
> These tools are really nice! Some thoughts/comments:
>
> Maybe the "devtool.conf" that gets created could be placed in the
> "conf/" subdirectory (along with the other configuration files such as
> local.conf and bblayers.conf)?
Yes, that's a good idea, I'll change that.
> Perhaps any recipe you're working on could be automatically included via
> an IMAGE_INSTALL_append in conf/local.conf (or maybe that's too intrusive?)?
This is something I'd wanted to do - it's certainly something that should be
made easy, but I was concerned about causing a full reparse just because of
adding that to local.conf. (There might be a workaround through some sort of
packagegroup for containing the packages produced by the recipes in the
workspace that is added once when we create the workspace - maybe that's the
answer?)
> Do you envision users creating multiple workspaces? I'm wondering why
> "devtool create-workspace" is required. Is there any advantage to
> requiring users to create the workspace explicitly instead of just
> having it be created implicitly?
I wouldn't expect users to want to create multiple workspaces, but I did want
users to be able to (a) choose where their workspace would go and (b) know
that it has been created, so that the workspace layer showing up in the
configuration isn't a surprise.
> Some of the commands to "devtool" include things like
> - extract
> - build
> - deploy
> - undeploy
> but when a workspace is created, devtool (very intelligently) adds the
> workspace to the set of BBLAYERS. So one could then just use bitbake to
> build the recipe. Are there any advantages to using "devtool build
> <recipe>" instead of "bitbake <recipe>"?
Not at the moment, although it is a convenient shortcut for "bitbake -c
install <recipename>" (which is all you need to do for "devtool deploy" - note
that "deploy" is distinct from our do_deploy, it could perhaps benefit from a
better name). The other reason it's there is more for use as part of the SDK
where the intention is to do everything through the devtool command, although
that is a usage model that isn't enabled yet.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC PATCH 0/7] Developer workflow improvements
2014-12-01 10:11 ` Paul Eggleton
@ 2014-12-02 4:36 ` Trevor Woerner
2014-12-02 11:46 ` Paul Eggleton
0 siblings, 1 reply; 40+ messages in thread
From: Trevor Woerner @ 2014-12-02 4:36 UTC (permalink / raw)
To: Paul Eggleton; +Cc: openembedded-core
On 12/01/14 05:11, Paul Eggleton wrote:
> On Friday 28 November 2014 12:28:00 Trevor Woerner wrote:
>> Perhaps any recipe you're working on could be automatically included via
>> an IMAGE_INSTALL_append in conf/local.conf (or maybe that's too intrusive?)?
> This is something I'd wanted to do - it's certainly something that should be
> made easy, but I was concerned about causing a full reparse just because of
> adding that to local.conf. (There might be a workaround through some sort of
> packagegroup for containing the packages produced by the recipes in the
> workspace that is added once when we create the workspace - maybe that's the
> answer?)
Maybe even just printing a bit of text after a successful "add" to
inform the user that the just-added project isn't part of any image and
what they might want to do to include it?
The packagegroup sounds good too. But if the user doesn't want it
included, they might be confused about how it magically appeared, and
have some trouble finding how to remove it.
>> Do you envision users creating multiple workspaces? I'm wondering why
>> "devtool create-workspace" is required. Is there any advantage to
>> requiring users to create the workspace explicitly instead of just
>> having it be created implicitly?
> I wouldn't expect users to want to create multiple workspaces, but I did want
> users to be able to (a) choose where their workspace would go and (b) know
> that it has been created, so that the workspace layer showing up in the
> configuration isn't a surprise.
I can't help think that there's no harm in an unused workspace (is
there?). Maybe the empty workspace from "create-workspace" should just
be created by default by the "oe-init-build-env" tool (or whatever a
given project is using).
If I were writing the documentation for this workflow, or giving a
presentation on it... I'm just trying to figure out how to justify the
extra, empty (but necessary) step of creating the environment before
using it. Maybe if someone tries to add a project before creating the
workspace, instead of erroring out, just run the create-workspace for
them with the defaults. If they're a power user who wants to put the
workspace somewhere specific then they're free to explicitly call
create-workspace on their own, otherwise it'll get called with the
defaults for them (or created automatically as part of the
oe-init-build-env).
To borrow the analogy from git: create-workspace is part of the pluming;
it's there if people what to call it explicitly, otherwise it gets
called implicitly with defaults?
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC PATCH 0/7] Developer workflow improvements
2014-12-02 4:36 ` Trevor Woerner
@ 2014-12-02 11:46 ` Paul Eggleton
2014-12-04 14:03 ` Trevor Woerner
0 siblings, 1 reply; 40+ messages in thread
From: Paul Eggleton @ 2014-12-02 11:46 UTC (permalink / raw)
To: Trevor Woerner; +Cc: openembedded-core
On Monday 01 December 2014 23:36:23 Trevor Woerner wrote:
> On 12/01/14 05:11, Paul Eggleton wrote:
> > On Friday 28 November 2014 12:28:00 Trevor Woerner wrote:
> >> Perhaps any recipe you're working on could be automatically included via
> >> an IMAGE_INSTALL_append in conf/local.conf (or maybe that's too
> >> intrusive?)?>
> > This is something I'd wanted to do - it's certainly something that should
> > be made easy, but I was concerned about causing a full reparse just
> > because of adding that to local.conf. (There might be a workaround
> > through some sort of packagegroup for containing the packages produced by
> > the recipes in the workspace that is added once when we create the
> > workspace - maybe that's the answer?)
>
> Maybe even just printing a bit of text after a successful "add" to
> inform the user that the just-added project isn't part of any image and
> what they might want to do to include it?
>
> The packagegroup sounds good too. But if the user doesn't want it
> included, they might be confused about how it magically appeared, and
> have some trouble finding how to remove it.
Right. I might defer this until we get to the SDK part where we'll need
something to handle this (since the aim in that case is to avoid any editing
of conf files or interacting with bitbake directly).
> >> Do you envision users creating multiple workspaces? I'm wondering why
> >> "devtool create-workspace" is required. Is there any advantage to
> >> requiring users to create the workspace explicitly instead of just
> >> having it be created implicitly?
> >
> > I wouldn't expect users to want to create multiple workspaces, but I did
> > want users to be able to (a) choose where their workspace would go and
> > (b) know that it has been created, so that the workspace layer showing up
> > in the configuration isn't a surprise.
>
> I can't help think that there's no harm in an unused workspace (is
> there?). Maybe the empty workspace from "create-workspace" should just
> be created by default by the "oe-init-build-env" tool (or whatever a
> given project is using).
I guess my thought was that some people are fussy about their setup, and this
is a bit of a new thing.
> If I were writing the documentation for this workflow, or giving a
> presentation on it... I'm just trying to figure out how to justify the
> extra, empty (but necessary) step of creating the environment before
> using it. Maybe if someone tries to add a project before creating the
> workspace, instead of erroring out, just run the create-workspace for
> them with the defaults. If they're a power user who wants to put the
> workspace somewhere specific then they're free to explicitly call
> create-workspace on their own, otherwise it'll get called with the
> defaults for them (or created automatically as part of the
> oe-init-build-env).
>
> To borrow the analogy from git: create-workspace is part of the pluming;
> it's there if people what to call it explicitly, otherwise it gets
> called implicitly with defaults?
Actually you make a very good case; auto-creating it (and stating so) might be
better than failing. I'll make the change.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC PATCH 0/7] Developer workflow improvements
2014-12-02 11:46 ` Paul Eggleton
@ 2014-12-04 14:03 ` Trevor Woerner
2014-12-04 15:33 ` Paul Eggleton
0 siblings, 1 reply; 40+ messages in thread
From: Trevor Woerner @ 2014-12-04 14:03 UTC (permalink / raw)
To: Paul Eggleton; +Cc: openembedded-core
Hi Paul,
Are you making these changes on the existing contrib/paule/devtool or
somewhere else?
A small nit I stumbled on recently... many layers contain a
"recipes-devtools" directory, and I'm thinking that using the same word
for this feature might get confusing (?). Maybe it would be best to
replace "devtool(s)" with "workflow(s)"?
Best regards,
Trevor
^ permalink raw reply [flat|nested] 40+ messages in thread* Re: [RFC PATCH 0/7] Developer workflow improvements
2014-12-04 14:03 ` Trevor Woerner
@ 2014-12-04 15:33 ` Paul Eggleton
0 siblings, 0 replies; 40+ messages in thread
From: Paul Eggleton @ 2014-12-04 15:33 UTC (permalink / raw)
To: Trevor Woerner; +Cc: openembedded-core
Hi Trevor,
On Thursday 04 December 2014 09:03:37 Trevor Woerner wrote:
> Are you making these changes on the existing contrib/paule/devtool or
> somewhere else?
Actually I hadn't pushed them anywhere; I have just done that on the
paule/devtool branch in both contrib repos. I'm collecting commits on top of
the original ones, these will be squashed in when I send the first non-RFC
series.
> A small nit I stumbled on recently... many layers contain a
> "recipes-devtools" directory, and I'm thinking that using the same word
> for this feature might get confusing (?). Maybe it would be best to
> replace "devtool(s)" with "workflow(s)"?
Well, naming is always tricky. I don't think "workflow" really describes what
the tool is supposed to do however; the tool certainly doesn't force a
particular workflow, just provides tools to enable various different developer
workflows. I'm happy to take other suggestions on naming though.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC PATCH 0/7] Developer workflow improvements
2014-11-25 17:28 [RFC PATCH 0/7] Developer workflow improvements Paul Eggleton
` (9 preceding siblings ...)
2014-11-28 17:28 ` Trevor Woerner
@ 2014-12-02 4:54 ` Trevor Woerner
2014-12-02 14:01 ` Paul Eggleton
[not found] ` <54866CD7.1050102@linaro.org>
11 siblings, 1 reply; 40+ messages in thread
From: Trevor Woerner @ 2014-12-02 4:54 UTC (permalink / raw)
To: Paul Eggleton, openembedded-core
On 11/25/14 12:28, Paul Eggleton wrote:
> I've then added a new recipe auto-creation script, recipetool, which can
> take a source tree or URL and create a skeleton recipe to build it.
...
> * "recipetool create" ideally needs to become smarter and fill in
> more details of the recipe. At some point we'll probably have to
> make the process interactive and possibly have it automate parts of
> the build process and examine the output; some things just can't be
> practically detected from the source tree without building.
There is a tool "autoscan" which might be of some use here. If nothing
else, the resulting configure.scan file could be parsed for DEPENDS
information and perhaps PACKAGECONFIG items too.
^ permalink raw reply [flat|nested] 40+ messages in thread* Re: [RFC PATCH 0/7] Developer workflow improvements
2014-12-02 4:54 ` Trevor Woerner
@ 2014-12-02 14:01 ` Paul Eggleton
2014-12-09 15:47 ` Trevor Woerner
0 siblings, 1 reply; 40+ messages in thread
From: Paul Eggleton @ 2014-12-02 14:01 UTC (permalink / raw)
To: Trevor Woerner; +Cc: openembedded-core
On Monday 01 December 2014 23:54:03 Trevor Woerner wrote:
> On 11/25/14 12:28, Paul Eggleton wrote:
> > I've then added a new recipe auto-creation script, recipetool, which can
> > take a source tree or URL and create a skeleton recipe to build it.
>
> ...
>
> > * "recipetool create" ideally needs to become smarter and fill in
> >
> > more details of the recipe. At some point we'll probably have to
> > make the process interactive and possibly have it automate parts of
> > the build process and examine the output; some things just can't be
> > practically detected from the source tree without building.
>
> There is a tool "autoscan" which might be of some use here. If nothing
> else, the resulting configure.scan file could be parsed for DEPENDS
> information and perhaps PACKAGECONFIG items too.
Interesting - I was not aware of this tool. I've added support for running it
as well as picking converting AC_CHECK_LIB lines over to DEPENDS (with a note
about some of them possibly being optional). Thanks!
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC PATCH 0/7] Developer workflow improvements
2014-12-02 14:01 ` Paul Eggleton
@ 2014-12-09 15:47 ` Trevor Woerner
2014-12-11 23:10 ` Trevor Woerner
0 siblings, 1 reply; 40+ messages in thread
From: Trevor Woerner @ 2014-12-09 15:47 UTC (permalink / raw)
To: Paul Eggleton; +Cc: openembedded-core
On 12/02/14 09:01, Paul Eggleton wrote:
> On Monday 01 December 2014 23:54:03 Trevor Woerner wrote:
>> On 11/25/14 12:28, Paul Eggleton wrote:
>>> I've then added a new recipe auto-creation script, recipetool, which can
>>> take a source tree or URL and create a skeleton recipe to build it.
>> ...
>>
>>> * "recipetool create" ideally needs to become smarter and fill in
>>>
>>> more details of the recipe. At some point we'll probably have to
>>> make the process interactive and possibly have it automate parts of
>>> the build process and examine the output; some things just can't be
>>> practically detected from the source tree without building.
>> There is a tool "autoscan" which might be of some use here. If nothing
>> else, the resulting configure.scan file could be parsed for DEPENDS
>> information and perhaps PACKAGECONFIG items too.
> Interesting - I was not aware of this tool. I've added support for running it
> as well as picking converting AC_CHECK_LIB lines over to DEPENDS (with a note
> about some of them possibly being optional). Thanks!
>
Finding "AC_PATH_X" in "configure.scan" is probably a good indication
there's an RDEPENDS on "libx11". Maybe "recipetool" could check for that?
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC PATCH 0/7] Developer workflow improvements
2014-12-09 15:47 ` Trevor Woerner
@ 2014-12-11 23:10 ` Trevor Woerner
2014-12-12 12:39 ` Paul Eggleton
0 siblings, 1 reply; 40+ messages in thread
From: Trevor Woerner @ 2014-12-11 23:10 UTC (permalink / raw)
To: Paul Eggleton; +Cc: openembedded-core
I think *this* thread (and question) got lost in the shuffle...
On 12/09/14 10:47, Trevor Woerner wrote:
> On 12/02/14 09:01, Paul Eggleton wrote:
>> On Monday 01 December 2014 23:54:03 Trevor Woerner wrote:
>>> On 11/25/14 12:28, Paul Eggleton wrote:
>>>> I've then added a new recipe auto-creation script, recipetool, which can
>>>> take a source tree or URL and create a skeleton recipe to build it.
>>> ...
>>>
>>>> * "recipetool create" ideally needs to become smarter and fill in
>>>>
>>>> more details of the recipe. At some point we'll probably have to
>>>> make the process interactive and possibly have it automate parts of
>>>> the build process and examine the output; some things just can't be
>>>> practically detected from the source tree without building.
>>> There is a tool "autoscan" which might be of some use here. If nothing
>>> else, the resulting configure.scan file could be parsed for DEPENDS
>>> information and perhaps PACKAGECONFIG items too.
>> Interesting - I was not aware of this tool. I've added support for running it
>> as well as picking converting AC_CHECK_LIB lines over to DEPENDS (with a note
>> about some of them possibly being optional). Thanks!
>>
> Finding "AC_PATH_X" in "configure.scan" is probably a good indication
> there's an RDEPENDS on "libx11". Maybe "recipetool" could check for that?
*ping*!!
Oddly enough, the default recipe even says:
# NOTE: the following library dependencies are unknown, ignoring: X11
Why is it ignoring x11? Coincidentally the code I'm testing this with
has a dependency on X11 and when I try building, bitbake throws out an
RDEPENDS warning.
^ permalink raw reply [flat|nested] 40+ messages in thread* Re: [RFC PATCH 0/7] Developer workflow improvements
2014-12-11 23:10 ` Trevor Woerner
@ 2014-12-12 12:39 ` Paul Eggleton
0 siblings, 0 replies; 40+ messages in thread
From: Paul Eggleton @ 2014-12-12 12:39 UTC (permalink / raw)
To: Trevor Woerner; +Cc: openembedded-core
On Thursday 11 December 2014 18:10:57 Trevor Woerner wrote:
> I think *this* thread (and question) got lost in the shuffle...
>
> On 12/09/14 10:47, Trevor Woerner wrote:
> > On 12/02/14 09:01, Paul Eggleton wrote:
> >> On Monday 01 December 2014 23:54:03 Trevor Woerner wrote:
> >>> On 11/25/14 12:28, Paul Eggleton wrote:
> >>>> I've then added a new recipe auto-creation script, recipetool, which
> >>>> can
> >>>> take a source tree or URL and create a skeleton recipe to build it.
> >>>
> >>> ...
> >>>
> >>>> * "recipetool create" ideally needs to become smarter and fill in
> >>>>
> >>>> more details of the recipe. At some point we'll probably have to
> >>>> make the process interactive and possibly have it automate parts of
> >>>> the build process and examine the output; some things just can't be
> >>>> practically detected from the source tree without building.
> >>>
> >>> There is a tool "autoscan" which might be of some use here. If nothing
> >>> else, the resulting configure.scan file could be parsed for DEPENDS
> >>> information and perhaps PACKAGECONFIG items too.
> >>
> >> Interesting - I was not aware of this tool. I've added support for
> >> running it as well as picking converting AC_CHECK_LIB lines over to
> >> DEPENDS (with a note about some of them possibly being optional).
> >> Thanks!
> >
> > Finding "AC_PATH_X" in "configure.scan" is probably a good indication
> > there's an RDEPENDS on "libx11". Maybe "recipetool" could check for that?
>
> *ping*!!
I hadn't missed this, I just hadn't got around to looking at it ;)
> Oddly enough, the default recipe even says:
>
> # NOTE: the following library dependencies are unknown, ignoring: X11
>
>
> Why is it ignoring x11? Coincidentally the code I'm testing this with
> has a dependency on X11 and when I try building, bitbake throws out an
> RDEPENDS warning.
It's ignoring anything that the extremely naive and incomplete mapping that I
initially added doesn't currently handle. It'll need to be extended, and X11
would obviously be something we should add straight away. I've added a general
task to expand the mapping to the todo list. FYI we are tracking more fine-
grained tasks than are covered in bugzilla in the following wiki pages
(especially since this is currently being worked on in a separate branch):
https://wiki.yoctoproject.org/wiki/Developer_Workflow_Improvements
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 40+ messages in thread
[parent not found: <54866CD7.1050102@linaro.org>]