[yocto-autobuilder][PATCH V4 0/1] better matching for refkit ci

[yocto-autobuilder][PATCH V4 0/1] better matching for refkit ci

[Stephano Cetola]

changed since V3:
Remove line breaks as well.

Stephano Cetola (1):
  ScrapeTargets.py: improve target search algorithm

 .../site-packages/autobuilder/buildsteps/ScrapeTargets.py    | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

-- 
2.13.2

[yocto-autobuilder][PATCH V4 1/1] ScrapeTargets.py: improve target search algorithm

[Stephano Cetola]

When scraping the build targets from refkit-ci.inc, grep would only
return the first line of the search result. By replacing grep with
awk, we are now searching for multi-line variables, and should now
scrape build targets even if they have line breaks between them.

[YOCTO #11720]

Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com>
---
 .../site-packages/autobuilder/buildsteps/ScrapeTargets.py    | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/ScrapeTargets.py b/lib/python2.7/site-packages/autobuilder/buildsteps/ScrapeTargets.py
index 80153fe85..07e2fdcf4 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/ScrapeTargets.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/ScrapeTargets.py
@@ -41,7 +41,9 @@ class ScrapeTargets(ShellCommand):
         workerdir = os.path.join(os.path.join(YOCTO_ABBASE, "yocto-worker"))
         buildername = self.getProperty("buildername")
         src = os.path.join(workerdir, buildername, "build", self.source)
-        self.command = "cat " + src + " | grep " + self.targetsvar
+        # find targetsvar then return lines up to a quote
+        self.command = ["awk",
+                '/%s/{flag=1;print;next}/"/{flag=0}flag' % self.targetsvar, src]
         ShellCommand.start(self)
 
     def commandComplete(self, cmd):
@@ -49,10 +51,10 @@ class ScrapeTargets(ShellCommand):
             return
 
         result = cmd.logs['stdio'].getText()
-        targets = result.replace(self.targetsvar, "")
-        targets = targets.strip()
-        targets = targets.replace('=', '')
-        targets = targets.strip('"')
+        targets = result.strip()
+        targets = targets.replace('%s="' % self.targetsvar, '')
+        targets = targets.replace('\\', '')
+        targets = targets.replace('\n', '')
         self.setProperty("scraped_targets",
                          targets,
                          'Targets "%s" scraped from %s' % (targets,
-- 
2.13.2

Re: [yocto-autobuilder][PATCH V4 1/1] ScrapeTargets.py: improve target search algorithm

[Joshua Lock]

On Wed, 2017-07-05 at 14:27 -0700, Stephano Cetola wrote:
> When scraping the build targets from refkit-ci.inc, grep would only
> return the first line of the search result. By replacing grep with
> awk, we are now searching for multi-line variables, and should now
> scrape build targets even if they have line breaks between them.
> 
> [YOCTO #11720]

Thanks, I've merged and pushed this. Please see my comment below.

> 
> Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com>
> ---
>  .../site-packages/autobuilder/buildsteps/ScrapeTargets.py    | 12
> +++++++-----
>  1 file changed, 7 insertions(+), 5 deletions(-)
> 
> diff --git a/lib/python2.7/site-
> packages/autobuilder/buildsteps/ScrapeTargets.py
> b/lib/python2.7/site-packages/autobuilder/buildsteps/ScrapeTargets.py
> index 80153fe85..07e2fdcf4 100644
> --- a/lib/python2.7/site-
> packages/autobuilder/buildsteps/ScrapeTargets.py
> +++ b/lib/python2.7/site-
> packages/autobuilder/buildsteps/ScrapeTargets.py
> @@ -41,7 +41,9 @@ class ScrapeTargets(ShellCommand):
>          workerdir = os.path.join(os.path.join(YOCTO_ABBASE, "yocto-
> worker"))
>          buildername = self.getProperty("buildername")
>          src = os.path.join(workerdir, buildername, "build",
> self.source)
> -        self.command = "cat " + src + " | grep " + self.targetsvar
> +        # find targetsvar then return lines up to a quote
> +        self.command = ["awk",
> +                '/%s/{flag=1;print;next}/"/{flag=0}flag' %
> self.targetsvar, src]
>          ShellCommand.start(self)
>  
>      def commandComplete(self, cmd):
> @@ -49,10 +51,10 @@ class ScrapeTargets(ShellCommand):
>              return
>  
>          result = cmd.logs['stdio'].getText()
> -        targets = result.replace(self.targetsvar, "")
> -        targets = targets.strip()
> -        targets = targets.replace('=', '')
> -        targets = targets.strip('"')
> +        targets = result.strip()
> +        targets = targets.replace('%s="' % self.targetsvar, '')

This still leaves us a bit brittle. What if someone adds whitespace
around the assignment operator? Could you create a follow-on patch for
that?

Thanks!

> +        targets = targets.replace('\\', '')
> +        targets = targets.replace('\n', '')
>          self.setProperty("scraped_targets",
>                           targets,
>                           'Targets "%s" scraped from %s' % (targets,
> -- 
> 2.13.2
> 

Re: [yocto-autobuilder][PATCH V4 1/1] ScrapeTargets.py: improve target search algorithm

[Stephano Cetola]

On 07/06, Joshua Lock wrote:
> This still leaves us a bit brittle. What if someone adds whitespace
> around the assignment operator? Could you create a follow-on patch for
> that?
> 
> Thanks!
Actually, since the awk command just searches for self.targetsvar now
instead of ``self.targetsvar="``, it will match even if there are
spaces between the equal.

I did just realize that if the variable is NOT multi-line, e.g.
REFKIT_CI_POSTBUILD_SELFTESTS, this search will fail because of the
"next" command.

I'll puzzle over this a bit.

-S

Re: [yocto-autobuilder][PATCH V4 1/1] ScrapeTargets.py: improve target search algorithm

[Joshua Lock]

On Thu, 2017-07-06 at 08:09 -0700, Stephano Cetola wrote:
> On 07/06, Joshua Lock wrote:
> > This still leaves us a bit brittle. What if someone adds whitespace
> > around the assignment operator? Could you create a follow-on patch
> > for
> > that?
> > 
> > Thanks!
> 
> Actually, since the awk command just searches for self.targetsvar now
> instead of ``self.targetsvar="``, it will match even if there are
> spaces between the equal.

Context is important. It's not the awk command, it's the str.replace()

> +        targets = targets.replace('%s="' % self.targetsvar, '')

we're going replace 'SOMEVARIABLE="' with '', but what if we get
'SOMEVARIABLE = "' ?

> I did just realize that if the variable is NOT multi-line, e.g.
> REFKIT_CI_POSTBUILD_SELFTESTS, this search will fail because of the
> "next" command.

Ah, yes. That could cause some surprises.

> I'll puzzle over this a bit.

Thanks!

Joshua