shell parser: deep nesting

shell parser: deep nesting

[Patrick Ohly]

Hello!

I ran into a bug
(https://bugzilla.yoctoproject.org/show_bug.cgi?id=10668) where parsing
a shell fragment like "echo hello & wait $!" triggers this error:

Traceback (most recent call last):
  File "/work/poky/bitbake/lib/bb/codeparser.py", line 400, in ShellParser.process_tokens(tokens=[('if_clause', <bb.pysh.pyshyacc.IfCond object at 0x7fb480963390>)]):
                 if more_tokens:
    >                self.process_tokens(more_tokens)
     
  File "/work/poky/bitbake/lib/bb/codeparser.py", line 400, in ShellParser.process_tokens(tokens=<itertools.chain object at 0x7fb480949278>):
                 if more_tokens:
    >                self.process_tokens(more_tokens)
     
  File "/work/poky/bitbake/lib/bb/codeparser.py", line 400, in ShellParser.process_tokens(tokens=[('while_clause', <bb.pysh.pyshyacc.WhileLoop object at 0x7fb480949518>)]):
                 if more_tokens:
    >                self.process_tokens(more_tokens)
     
  File "/work/poky/bitbake/lib/bb/codeparser.py", line 400, in ShellParser.process_tokens(tokens=<itertools.chain object at 0x7fb480446668>):
                 if more_tokens:
    >                self.process_tokens(more_tokens)
     
  File "/work/poky/bitbake/lib/bb/codeparser.py", line 393, in ShellParser.process_tokens(tokens=[[('pipeline', <bb.pysh.pyshyacc.Pipeline object at 0x7fb480949c88>)]]):
             for token in tokens:
    >            name, value = token
                 try:
ValueError: need more than 1 value to unpack

After thinking about this a bit more I *think* I understand the problem
(expecting a certain level of nesting while the parser may also hand out
tokens with deeper nesting in the pipeline case) and I'll send a patch
fixing the issue for me.

But I'm not entirely sure whether that's really addressing the root
cause of the problem - perhaps such deep nesting shouldn't occur?

So I hope that someone familiar with this code can have a look at the
problem and the proposed solution.

Also, does it have performance implications that
ShellParser.process_tokens() defines the same helper functions
repeatedly? They don't depend on the current invocation context and thus
could also be defined as private helper functions in ShellParser.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

[PATCH] codeparser.py: support deeply nested tokens

[Patrick Ohly]

For shell constructs like
   echo hello & wait $!
the process_tokens() method ended up with a situation where "token"
in the "name, value = token" assignment was a list of tuples
and not the expected tuple, causing the assignment to fail.

There were already two for loops (one in _parse_shell(), one in
process_tokens()) which iterated over token lists. Apparently the
actual nesting can also be deeper.

Now there is just one such loop in process_token_list() which calls
itself recursively when it detects that a list entry is another list.

As a side effect (improvement?!) of the loop removal in
_parse_shell(), the local function definitions in process_tokens() get
executed less often.

Fixes: [YOCTO #10668]

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 lib/bb/codeparser.py | 29 +++++++++++++++++------------
 1 file changed, 17 insertions(+), 12 deletions(-)

diff --git a/lib/bb/codeparser.py b/lib/bb/codeparser.py
index 25938d6..5d2d440 100644
--- a/lib/bb/codeparser.py
+++ b/lib/bb/codeparser.py
@@ -342,8 +342,7 @@ class ShellParser():
         except pyshlex.NeedMore:
             raise sherrors.ShellSyntaxError("Unexpected EOF")
 
-        for token in tokens:
-            self.process_tokens(token)
+        self.process_tokens(tokens)
 
     def process_tokens(self, tokens):
         """Process a supplied portion of the syntax tree as returned by
@@ -389,18 +388,24 @@ class ShellParser():
             "case_clause": case_clause,
         }
 
-        for token in tokens:
-            name, value = token
-            try:
-                more_tokens, words = token_handlers[name](value)
-            except KeyError:
-                raise NotImplementedError("Unsupported token type " + name)
+        def process_token_list(tokens):
+            for token in tokens:
+                if isinstance(token, list):
+                    process_token_list(token)
+                    continue
+                name, value = token
+                try:
+                    more_tokens, words = token_handlers[name](value)
+                except KeyError:
+                    raise NotImplementedError("Unsupported token type " + name)
+
+                if more_tokens:
+                    self.process_tokens(more_tokens)
 
-            if more_tokens:
-                self.process_tokens(more_tokens)
+                if words:
+                    self.process_words(words)
 
-            if words:
-                self.process_words(words)
+        process_token_list(tokens)
 
     def process_words(self, words):
         """Process a set of 'words' in pyshyacc parlance, which includes
-- 
2.1.4