[PATCH v2 0/1]

[PATCH v2 0/1]

[Amanda Brindle]

For v2, I removed some dead code (recipe_bbvars(), collect_bbvars(), and 
code that handled the -m option). 

I also added the parameter variants=False to all_recipe_files() so that
the script would not scan through variant recipe files. 

I also updated the commit message to describe why the changes were made. 

The following changes since commit 65d23bd7986615fdfb0f1717b615534a2a14ab80:

  README.qemu: qemuppc64 is not supported (2017-10-16 23:54:31 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib abrindle/bbvars_tinfoil
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=abrindle/bbvars_tinfoil

Amanda Brindle (1):
  scripts/contrib/bbvars.py: Rewrite to use tinfoil

 scripts/contrib/bbvars.py | 140 ++++++++++++++++++++++------------------------
 1 file changed, 67 insertions(+), 73 deletions(-)

-- 
2.7.4

[PATCH v2 1/1] scripts/contrib/bbvars.py: Rewrite to use tinfoil

[Amanda Brindle]

Use tinfoil to collect all variable names globally and in each recipe.
This will capture more variable references than parsing manually.

With the use of tinfoil, this script can no longer distinguish unique
references to each variable, so it's no longer showing the count of
undocumented variables.

Removed the -m option since this script now searches through all recipes
in the configuration.

Fixes [YOCTO #2086]

Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
---
 scripts/contrib/bbvars.py | 140 ++++++++++++++++++++++------------------------
 1 file changed, 67 insertions(+), 73 deletions(-)

diff --git a/scripts/contrib/bbvars.py b/scripts/contrib/bbvars.py
index d8d0594..286b5a9 100755
--- a/scripts/contrib/bbvars.py
+++ b/scripts/contrib/bbvars.py
@@ -23,62 +23,38 @@ import os
 import os.path
 import re
 
+# Set up sys.path to let us import tinfoil
+scripts_path = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
+lib_path = scripts_path + '/lib'
+sys.path.insert(0, lib_path)
+import scriptpath
+scriptpath.add_bitbake_lib_path()
+import bb.tinfoil
+
 def usage():
-    print('Usage: %s -d FILENAME [-d FILENAME]* -m METADIR [-m MATADIR]*' % os.path.basename(sys.argv[0]))
+    print('Usage: %s -d FILENAME [-d FILENAME]*' % os.path.basename(sys.argv[0]))
     print('  -d FILENAME         documentation file to search')
     print('  -h, --help          display this help and exit')
-    print('  -m METADIR          meta directory to search for recipes')
     print('  -t FILENAME         documentation config file (for doc tags)')
     print('  -T                  Only display variables with doc tags (requires -t)')
 
-def recipe_bbvars(recipe):
-    ''' Return a unique set of every bbvar encountered in the recipe '''
-    prog = re.compile("[A-Z_]+")
-    vset = set()
-    try:
-        r = open(recipe)
-    except IOError as err:
-        print('WARNING: Failed to open recipe ', recipe)
-        print(err.args[1])
-
-    for line in r:
-        # Strip any comments from the line
-        line = line.rsplit('#')[0]
-        vset = vset.union(set(prog.findall(line)))
-    r.close()
-
-    bbvars = {}
-    for v in vset:
-        bbvars[v] = 1
-
-    return bbvars
-
-def collect_bbvars(metadir):
-    ''' Walk the metadir and collect the bbvars from each recipe found '''
-    bbvars = {}
-    for root,dirs,files in os.walk(metadir):
-        for name in files:
-            if name.find(".bb") >= 0:
-                for key in recipe_bbvars(os.path.join(root,name)).keys():
-                    if key in bbvars:
-                        bbvars[key] = bbvars[key] + 1
-                    else:
-                        bbvars[key] = 1
-    return bbvars
-
-def bbvar_is_documented(var, docfiles):
-    prog = re.compile(".*($|[^A-Z_])%s([^A-Z_]|$)" % (var))
-    for doc in docfiles:
-        try:
-            f = open(doc)
-        except IOError as err:
-            print('WARNING: Failed to open doc ', doc)
-            print(err.args[1])
-        for line in f:
-            if prog.match(line):
-                return True
-        f.close()
-    return False
+def bbvar_is_documented(var, documented_vars):
+    ''' Check if variable (var) is in the list of documented variables(documented_vars) '''
+    if var in documented_vars:
+        return True
+    else:
+        return False
+
+def collect_documented_vars(docfiles):
+    ''' Walk the docfiles and collect the documented variables '''
+    documented_vars = []
+    prog = re.compile(".*($|[^A-Z_])<glossentry id=\'var-")
+    var_prog = re.compile('<glossentry id=\'var-(.*)\'>')
+    for d in docfiles:
+        with open(d) as f:
+            documented_vars += var_prog.findall(f.read())
+
+    return documented_vars
 
 def bbvar_doctag(var, docconf):
     prog = re.compile('^%s\[doc\] *= *"(.*)"' % (var))
@@ -100,8 +76,7 @@ def bbvar_doctag(var, docconf):
 
 def main():
     docfiles = []
-    metadirs = []
-    bbvars = {}
+    bbvars = set()
     undocumented = []
     docconf = ""
     onlydoctags = False
@@ -124,12 +99,6 @@ def main():
             else:
                 print('ERROR: documentation file %s is not a regular file' % a)
                 sys.exit(3)
-        elif o == '-m':
-            if os.path.isdir(a):
-                metadirs.append(a)
-            else:
-                print('ERROR: meta directory %s is not a directory' % a)
-                sys.exit(4)
         elif o == "-t":
             if os.path.isfile(a):
                 docconf = a
@@ -143,43 +112,68 @@ def main():
         usage()
         sys.exit(5)
 
-    if len(metadirs) == 0:
-        print('ERROR: no metadir specified')
-        usage()
-        sys.exit(6)
-
     if onlydoctags and docconf == "":
         print('ERROR: no docconf specified')
         usage()
         sys.exit(7)
 
-    # Collect all the variable names from the recipes in the metadirs
-    for m in metadirs:
-        for key,cnt in collect_bbvars(m).items():
-            if key in bbvars:
-                bbvars[key] = bbvars[key] + cnt
+    prog = re.compile("^[^a-z]*$")
+    with bb.tinfoil.Tinfoil() as tinfoil:
+        tinfoil.prepare(config_only=False)
+        parser = bb.codeparser.PythonParser('parser', None)
+        datastore = tinfoil.config_data
+
+        def bbvars_update(data):
+            if prog.match(data):
+                bbvars.add(data)
+            if tinfoil.config_data.getVarFlag(data, 'python'):
+                try:
+                    parser.parse_python(tinfoil.config_data.getVar(data))
+                except bb.data_smart.ExpansionError:
+                    pass
+                for var in parser.references:
+                    if prog.match(var):
+                        bbvars.add(var)
             else:
-                bbvars[key] = cnt
+                try:
+                    expandedVar = datastore.expandWithRefs(datastore.getVar(data, False), data)
+                    for var in expandedVar.references:
+                        if prog.match(var):
+                            bbvars.add(var)
+                except bb.data_smart.ExpansionError:
+                    pass
+
+        # Use tinfoil to collect all the variable names globally
+        for data in datastore:
+            bbvars_update(data)
+
+        # Collect variables from all recipes
+        for recipe in tinfoil.all_recipe_files(variants=False):
+            print("Checking %s" % recipe)
+            for data in tinfoil.parse_recipe_file(recipe):
+                bbvars_update(data)
+
+    documented_vars = collect_documented_vars(docfiles)
 
     # Check each var for documentation
     varlen = 0
-    for v in bbvars.keys():
+    for v in bbvars:
         if len(v) > varlen:
             varlen = len(v)
-        if not bbvar_is_documented(v, docfiles):
+        if not bbvar_is_documented(v, documented_vars):
             undocumented.append(v)
     undocumented.sort()
     varlen = varlen + 1
 
     # Report all undocumented variables
     print('Found %d undocumented bb variables (out of %d):' % (len(undocumented), len(bbvars)))
-    header = '%s%s%s' % (str("VARIABLE").ljust(varlen), str("COUNT").ljust(6), str("DOCTAG").ljust(7))
+    header = '%s%s' % (str("VARIABLE").ljust(varlen), str("DOCTAG").ljust(7))
     print(header)
     print(str("").ljust(len(header), '='))
     for v in undocumented:
         doctag = bbvar_doctag(v, docconf)
         if not onlydoctags or not doctag == "":
-            print('%s%s%s' % (v.ljust(varlen), str(bbvars[v]).ljust(6), doctag))
+            print('%s%s' % (v.ljust(varlen), doctag))
 
 
 if __name__ == "__main__":
-- 
2.7.4

✗ patchtest: failure for [v2] scripts/contrib/bbvars.py: Rewrite to use tinfoil

[Patchwork]

== Series Details ==

Series: [v2] scripts/contrib/bbvars.py: Rewrite to use tinfoil
Revision: 1
URL   : https://patchwork.openembedded.org/series/9816/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at a17f3ec910)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

✗ patchtest: failure for [v2] scripts/contrib/bbvars.py: Rewrite to use tinfoil

[Patchwork]

== Series Details ==

Series: [v2] scripts/contrib/bbvars.py: Rewrite to use tinfoil
Revision: 1
URL   : https://patchwork.openembedded.org/series/9816/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at a17f3ec910)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

Re: [PATCH v2 1/1] scripts/contrib/bbvars.py: Rewrite to use tinfoil

[Paul Eggleton]

On Thursday, 16 November 2017 1:21:04 PM NZDT Amanda Brindle wrote:
> Use tinfoil to collect all variable names globally and in each recipe.
> This will capture more variable references than parsing manually.
> 
> With the use of tinfoil, this script can no longer distinguish unique
> references to each variable, so it's no longer showing the count of
> undocumented variables.
> 
> Removed the -m option since this script now searches through all recipes
> in the configuration.
> 
> Fixes [YOCTO #2086]
> 
> Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>

Acked-by: Paul Eggleton <paul.eggleton@linux.intel.com>

Thanks,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

[PATCH v2 0/1]

[Amanda Brindle]

Added a signed-off-by for v2.

The following changes since commit 4ed19ac8c19afd56d445d84e02b622cb056b8359:

  poky: Switch to post release name/version (2017-11-14 17:26:58 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib abrindle/bbvars_tinfoil2
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=abrindle/bbvars_tinfoil2

Amanda Brindle (1):
  scripts/contrib/bbvars.py: Remove dead code

 scripts/contrib/bbvars.py | 52 ++---------------------------------------------
 1 file changed, 2 insertions(+), 50 deletions(-)

-- 
2.7.4