* [AB PATCH 01/22] buildsets: add a new buildset to run builds on target
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
@ 2014-03-19 20:50 ` Elizabeth Flanagan
2014-03-19 20:51 ` [AB PATCH 02/22] yoctogit.py: Fix cp construction from source to build Elizabeth Flanagan
` (20 subsequent siblings)
21 siblings, 0 replies; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:50 UTC (permalink / raw)
To: yocto
From: Stefan Stanacar <stefanx.stanacar@intel.com>
Builds core-image-sato-sdk for each qemu arch and
runs tests building sudoku/cvs/iptables.
These do take a while, so they shouldn't be added to nightly!
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
.../nightly-qa-targetbuilds.conf | 26 ++++++++++++++++++++++
1 file changed, 26 insertions(+)
create mode 100644 buildset-config.master/nightly-qa-targetbuilds.conf
diff --git a/buildset-config.master/nightly-qa-targetbuilds.conf b/buildset-config.master/nightly-qa-targetbuilds.conf
new file mode 100644
index 0000000..a95db78
--- /dev/null
+++ b/buildset-config.master/nightly-qa-targetbuilds.conf
@@ -0,0 +1,26 @@
+[nightly-qa-targetbuilds]
+builders: 'builder1'
+repos: [{'poky':
+ {'repourl':'git://git.yoctoproject.org/poky',
+ 'layerversion':{'core':'meta', 'yoctobsp':'meta-yocto-bsp'},
+ 'branch':'master'}}]
+steps: [{'SetDest':{}},
+ {'CheckOutLayers': {}},
+ {'RunPreamble': {}},
+ {'GetDistroVersion' : {'distro': 'poky'}},
+ {'CreateAutoConf': {'machine': 'qemux86-64', 'SDKMACHINE' : 'x86_64', 'distro': 'poky'}},
+ {'CreateBBLayersConf': {'buildprovider' : 'yocto'}},
+ {'BuildImages': {'images': 'core-image-sato-sdk'}},
+ {'RunSanityTests': {'images': 'core-image-sato-sdk', 'suites' : 'ping ssh buildsudoku buildcvs buildiptables'}},
+ {'CreateAutoConf': {'machine': 'qemux86', 'SDKMACHINE' : 'x86_64', 'distro': 'poky'}},
+ {'BuildImages': {'images': 'core-image-sato-sdk'}},
+ {'RunSanityTests': {'images': 'core-image-sato-sdk', 'suites' : 'ping ssh buildsudoku buildcvs buildiptables'}},
+ {'CreateAutoConf': {'machine': 'qemuarm', 'SDKMACHINE' : 'x86_64', 'distro': 'poky'}},
+ {'BuildImages': {'images': 'core-image-sato-sdk'}},
+ {'RunSanityTests': {'images': 'core-image-sato-sdk', 'suites' : 'ping ssh buildsudoku buildcvs buildiptables'}},
+ {'CreateAutoConf': {'machine': 'qemuppc', 'SDKMACHINE' : 'x86_64', 'distro': 'poky'}},
+ {'BuildImages': {'images': 'core-image-sato-sdk'}},
+ {'RunSanityTests': {'images': 'core-image-sato-sdk', 'suites' : 'ping ssh buildsudoku buildcvs buildiptables'}},
+ {'CreateAutoConf': {'machine': 'qemumips', 'SDKMACHINE' : 'x86_64', 'distro': 'poky'}},
+ {'BuildImages': {'images': 'core-image-sato-sdk'}},
+ {'RunSanityTests': {'images': 'core-image-sato-sdk', 'suites' : 'ping ssh buildsudoku buildcvs buildiptables'}}]
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* [AB PATCH 02/22] yoctogit.py: Fix cp construction from source to build
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
2014-03-19 20:50 ` [AB PATCH 01/22] buildsets: add a new buildset to run builds on target Elizabeth Flanagan
@ 2014-03-19 20:51 ` Elizabeth Flanagan
2014-03-20 14:29 ` Burton, Ross
2014-03-19 20:51 ` [AB PATCH 03/22] BuildImages.py: core-image-basic name change support Elizabeth Flanagan
` (19 subsequent siblings)
21 siblings, 1 reply; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:51 UTC (permalink / raw)
To: yocto
This commit removes the mkdir from movecopy as it is not needed.
It also changes the workdir in barecopy and movecopy to
the source dir, as we can't alwas be sure that the destination
dir even exists.
This also ensures that we are in fact constructing our cp string
correctly, by striping any trailing "/" from self.repourl
As repourl is used to construct the cp from source to build, that
trailing slash, if it exists, causes the directory structure to
not be correct (e.g. cp -R ./foo/ ./bar and cp -R ./foo ./bar
don't do the same thing, obviously)
Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
---
.../buildbot/steps/source/yoctogit.py | 35 +++++++---------------
1 file changed, 11 insertions(+), 24 deletions(-)
diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 26599e7..487e286 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -121,7 +121,7 @@ class YoctoGit(Source):
self.branch = branch
self.method = method
self.prog = progress
- self.repourl = repourl
+ self.repourl = repourl.rstrip('/')
self.retryFetch = retryFetch
self.submodules = submodules
self.shallow = shallow
@@ -255,13 +255,11 @@ class YoctoGit(Source):
if "poky" in self.layername or \
"oecore" in self.layername or \
"eclipse" in self.layername:
- todir = "."
- self.workdir = "build"
+ todir = "build"
+ self.workdir = 'source/'+self.repourl
else:
- todir = "./" + self.layername
- self.workdir = "build/" + self.layername
-
-
+ todir = "build/" + self.layername
+ self.workdir = 'source/'+self.repourl
d.addCallback(lambda _: self.incremental())
def copy(_):
cmd = buildstep.RemoteCommand('cpdir',
@@ -278,7 +276,7 @@ class YoctoGit(Source):
d.addCallback(lambda _: evaluateCommand(cmd))
return d
d.addCallback(copy)
-
+
def resetWorkdir(_):
if "poky" in self.layername or \
"oecore" in self.layername or \
@@ -316,26 +314,15 @@ class YoctoGit(Source):
log.msg("Source step failed while running command %s" % cmd)
raise buildstep.BuildStepFailed()
d.addCallback(lambda _: evaluateCommand(cmd))
-
- if self.layername == "poky" or self.layername == "oecore":
+ if "poky" in self.layername or \
+ "oecore" in self.layername or \
+ "eclipse" in self.layername:
todir = "build"
+ self.workdir = 'source/'+self.repourl.rstrip('/')
else:
todir = "build/" + self.layername
+ self.workdir = 'source/'+self.repourl
- def mkdir(_):
- cmd = buildstep.RemoteCommand('mkdir', {'dir': 'build',
- 'logEnviron': self.logEnviron,
- 'timeout': self.timeout,})
- cmd.useLog(self.stdio_log, False)
- d = self.runCommand(cmd)
- def evaluateCommand(cmd):
- if cmd.didFail():
- log.msg("Source step failed while running command %s" % cmd)
- raise buildstep.BuildStepFailed()
- d.addCallback(lambda _: evaluateCommand(cmd))
- return d
- d.addCallback(mkdir)
- d.addCallback(lambda _: self.incremental())
def copy(_):
cmd = buildstep.RemoteCommand('cpdir',
{'fromdir': srcdir,
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* Re: [AB PATCH 02/22] yoctogit.py: Fix cp construction from source to build
2014-03-19 20:51 ` [AB PATCH 02/22] yoctogit.py: Fix cp construction from source to build Elizabeth Flanagan
@ 2014-03-20 14:29 ` Burton, Ross
2014-03-20 21:53 ` Flanagan, Elizabeth
0 siblings, 1 reply; 28+ messages in thread
From: Burton, Ross @ 2014-03-20 14:29 UTC (permalink / raw)
To: Elizabeth Flanagan; +Cc: yocto@yoctoproject.org
On 19 March 2014 20:51, Elizabeth Flanagan <elizabeth.flanagan@intel.com> wrote:
> + if "poky" in self.layername or \
> + "oecore" in self.layername or \
> + "eclipse" in self.layername:
Do you actually need to do substring comparisons (in vs is) there? I
suspect not, so I prefer this idiom as it involves less repetition:
if self.layername in ('poky', 'oecore', 'eclipse')
Ross
^ permalink raw reply [flat|nested] 28+ messages in thread* Re: [AB PATCH 02/22] yoctogit.py: Fix cp construction from source to build
2014-03-20 14:29 ` Burton, Ross
@ 2014-03-20 21:53 ` Flanagan, Elizabeth
0 siblings, 0 replies; 28+ messages in thread
From: Flanagan, Elizabeth @ 2014-03-20 21:53 UTC (permalink / raw)
To: Burton, Ross; +Cc: yocto@yoctoproject.org
On Thu, Mar 20, 2014 at 7:29 AM, Burton, Ross <ross.burton@intel.com> wrote:
> On 19 March 2014 20:51, Elizabeth Flanagan <elizabeth.flanagan@intel.com> wrote:
>> + if "poky" in self.layername or \
>> + "oecore" in self.layername or \
>> + "eclipse" in self.layername:
>
> Do you actually need to do substring comparisons (in vs is) there? I
> suspect not, so I prefer this idiom as it involves less repetition:
>
> if self.layername in ('poky', 'oecore', 'eclipse')
In this case, yeah, we need a substring comparision, as we need to
support the layer being named eclipse-poky-<branch-name> and
poky/poky-contrib. We need to eventually figure out a better, more
standardized way of indicating the base layer, but until we do and
propagate that change out (as it changes configs), doing it this way
both supports the way people have come to understand what "base" is.
-b
>
> Ross
> ---------------------------------------------------------------------
> Intel Corporation (UK) Limited
> Registered No. 1134945 (England)
> Registered Office: Pipers Way, Swindon SN3 1RJ
> VAT No: 860 2173 47
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
--
Elizabeth Flanagan
Yocto Project
Build and Release
^ permalink raw reply [flat|nested] 28+ messages in thread
* [AB PATCH 03/22] BuildImages.py: core-image-basic name change support
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
2014-03-19 20:50 ` [AB PATCH 01/22] buildsets: add a new buildset to run builds on target Elizabeth Flanagan
2014-03-19 20:51 ` [AB PATCH 02/22] yoctogit.py: Fix cp construction from source to build Elizabeth Flanagan
@ 2014-03-19 20:51 ` Elizabeth Flanagan
2014-03-20 14:23 ` Burton, Ross
2014-03-19 20:51 ` [AB PATCH 04/22] yoctogit.py: movecopy doesn't care if it's updateable Elizabeth Flanagan
` (18 subsequent siblings)
21 siblings, 1 reply; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:51 UTC (permalink / raw)
To: yocto
Supports http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=b7f1cca517bbd4191828c6bae32e0c5041f1ff19
Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
---
.../site-packages/autobuilder/buildsteps/BuildImages.py | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py b/lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py
index 2dc72dd..5f1b549 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py
@@ -34,7 +34,16 @@ class BuildImages(ShellCommand):
def start(self):
self.layerversion_yoctobsp = self.getProperty("layerversion_yoctobsp")
+ self.layerversion_core = self.getProperty("layerversion_core")
self.machine = self.getProperty("MACHINE")
+ # core-image-basic rename
+ # See: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=b7f1cca517bbd4191828c6bae32e0c5041f1ff19
+ # I hate making people change their configs, so support both.
+ if self.layerversion_core < "4":
+ self.images=self.images.replace("core-image-full-cmdline", "core-image-basic")
+ elif self.layerversion_core > "3":
+ self.images=self.images.replace("core-image-basic", "core-image-full-cmdline")
+
if self.layerversion_yoctobsp is not None and int(self.layerversion_yoctobsp) < 2 and self.machine is not None and self.machine == "genericx86-64":
self.command = "echo 'Skipping Step.'"
else:
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* [AB PATCH 04/22] yoctogit.py: movecopy doesn't care if it's updateable
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
` (2 preceding siblings ...)
2014-03-19 20:51 ` [AB PATCH 03/22] BuildImages.py: core-image-basic name change support Elizabeth Flanagan
@ 2014-03-19 20:51 ` Elizabeth Flanagan
2014-03-19 20:51 ` [AB PATCH 05/22] yoctogit.py: movecopy needs an incremental callback Elizabeth Flanagan
` (17 subsequent siblings)
21 siblings, 0 replies; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:51 UTC (permalink / raw)
To: yocto
movecopy really shouldn't care if it's updatable or not. It should
have an incremental callback in it to take care of that (it doesn't
though, which is part of the problem!)
Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
---
.../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 487e286..25e5308 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -184,7 +184,7 @@ class YoctoGit(Source):
elif self.method == 'barecopy':
yield self.barecopy()
return
- elif updatable and self.method == 'movecopy':
+ elif self.method == 'movecopy':
yield self.movecopy()
return
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* [AB PATCH 05/22] yoctogit.py: movecopy needs an incremental callback
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
` (3 preceding siblings ...)
2014-03-19 20:51 ` [AB PATCH 04/22] yoctogit.py: movecopy doesn't care if it's updateable Elizabeth Flanagan
@ 2014-03-19 20:51 ` Elizabeth Flanagan
2014-03-20 14:25 ` Burton, Ross
2014-03-19 20:51 ` [AB PATCH 06/22] CheckOutLayers.py: Remove unset Props Elizabeth Flanagan
` (16 subsequent siblings)
21 siblings, 1 reply; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:51 UTC (permalink / raw)
To: yocto
We need to do an incremental before we copy or else, we're
essentially copying nothing.
Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
---
.../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 25e5308..080f5b2 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -322,6 +322,7 @@ class YoctoGit(Source):
else:
todir = "build/" + self.layername
self.workdir = 'source/'+self.repourl
+ d.addCallback(lambda _: self.incremental())
def copy(_):
cmd = buildstep.RemoteCommand('cpdir',
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* Re: [AB PATCH 05/22] yoctogit.py: movecopy needs an incremental callback
2014-03-19 20:51 ` [AB PATCH 05/22] yoctogit.py: movecopy needs an incremental callback Elizabeth Flanagan
@ 2014-03-20 14:25 ` Burton, Ross
0 siblings, 0 replies; 28+ messages in thread
From: Burton, Ross @ 2014-03-20 14:25 UTC (permalink / raw)
To: Elizabeth Flanagan; +Cc: yocto@yoctoproject.org
On 19 March 2014 20:51, Elizabeth Flanagan <elizabeth.flanagan@intel.com> wrote:
> We need to do an incremental before we copy or else, we're
> essentially copying nothing.
>
> Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
> ---
> .../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
> index 25e5308..080f5b2 100644
> --- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
> +++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
> @@ -322,6 +322,7 @@ class YoctoGit(Source):
> else:
> todir = "build/" + self.layername
> self.workdir = 'source/'+self.repourl
> + d.addCallback(lambda _: self.incremental())
>
> def copy(_):
> cmd = buildstep.RemoteCommand('cpdir',
> --
> 1.8.1.2
This is fixing a bug in 2/22 so should be squashed into that commit.
Ross
^ permalink raw reply [flat|nested] 28+ messages in thread
* [AB PATCH 06/22] CheckOutLayers.py: Remove unset Props
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
` (4 preceding siblings ...)
2014-03-19 20:51 ` [AB PATCH 05/22] yoctogit.py: movecopy needs an incremental callback Elizabeth Flanagan
@ 2014-03-19 20:51 ` Elizabeth Flanagan
2014-03-19 20:51 ` [AB PATCH 07/22] yoctogit.py: pathExists returns a defered value Elizabeth Flanagan
` (15 subsequent siblings)
21 siblings, 0 replies; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:51 UTC (permalink / raw)
To: yocto
We no longer set these trigger* props, so let's remove them.
Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
---
lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py | 4 ----
1 file changed, 4 deletions(-)
diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py b/lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py
index 90b90c5..5e95189 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py
@@ -82,10 +82,6 @@ class CheckOutLayers(YoctoGit):
self.getProperty('commit_resolvedhead_'+self.layername) is not None:
log.msg("Using Resolved HEAD")
self.commit = self.getProperty('commit_resolvedhead_'+self.layername)
- elif self.getProperty('commit_trigger_'+self.layername+"_"+self.scheduler):
- self.commit = self.getProperty('commit_trigger_'+self.layername+"_"+self.scheduler)
- elif self.getProperty('commit_triggerer_'+self.layername+"_"+self.scheduler) :
- self.commit = self.getProperty('commit_triggerer_'+self.layername+"_"+self.scheduler)
elif self.getProperty('commit_'+self.layername):
self.commit = self.getProperty('commit_'+self.layername)
else:
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* [AB PATCH 07/22] yoctogit.py: pathExists returns a defered value.
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
` (5 preceding siblings ...)
2014-03-19 20:51 ` [AB PATCH 06/22] CheckOutLayers.py: Remove unset Props Elizabeth Flanagan
@ 2014-03-19 20:51 ` Elizabeth Flanagan
2014-03-19 20:51 ` [AB PATCH 08/22] CheckOutLayers.py: Remove srcdir Elizabeth Flanagan
` (14 subsequent siblings)
21 siblings, 0 replies; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:51 UTC (permalink / raw)
To: yocto
We can't use pathExists this way to figure out if the path actually
exists as it will always evalutate to true as it returns a generator
object and not T/F.
In order to actually figure out if the mirrordir + repourl exists,
we need to call a function that returns the deferred generator
object outside of any of fetch/movecopy/barecopy functions (which
also return defereds and you can't yield a defered in a defered)
So, yield the generator object at the beginning, and save everyone
the headache.
Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
---
.../buildbot/steps/source/yoctogit.py | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 080f5b2..4c89ceb 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -177,6 +177,7 @@ class YoctoGit(Source):
@defer.inlineCallbacks
def full(self):
updatable = yield self._sourcedirIsUpdatable()
+ self.mirrorexists = yield self._mirrorExists()
if self.method == 'clobber':
yield self.clobber()
@@ -248,9 +249,9 @@ class YoctoGit(Source):
d.addCallback(lambda _: evaluateCommand(cmd))
if self.mirrordir:
mirror = self.mirrordir + "/" + self.repourl.replace("git://", "") + "/"
- if self.pathExists(mirror):
+ if self.mirrorexists:
srcdir=mirror
- else:
+ else:
srcdir='source/'+self.repourl
if "poky" in self.layername or \
"oecore" in self.layername or \
@@ -493,8 +494,10 @@ class YoctoGit(Source):
mirror = None
if self.mirrordir:
mirror = self.mirrordir + "/" + self.repourl.replace("git://", "") + "/"
- if self.pathExists(mirror):
- command = ['clone'] + ["-s", "-n"] + [mirror, '.']
+ if self.mirrorexists:
+ command = ['clone'] + ["-s", "-n"] + [mirror, '.']
+ else:
+ command = ['clone'] + ["-s", "-n"] + [self.repourl, '.']
#Fix references
if self.prog:
@@ -560,6 +563,10 @@ class YoctoGit(Source):
def _sourcedirIsUpdatable(self):
return self.pathExists(self.build.path_module.join(self.workdir, '.git'))
+ def _mirrorExists(self):
+ mirror = self.mirrordir + "/" + self.repourl.replace("git://", "") + "/"
+ return self.pathExists(self.build.path_module.join(mirror, '.git'))
+
def _updateSubmodule(self, _):
if self.submodules:
return self._dovccmd(['submodule', 'update', '--recursive'])
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* [AB PATCH 08/22] CheckOutLayers.py: Remove srcdir
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
` (6 preceding siblings ...)
2014-03-19 20:51 ` [AB PATCH 07/22] yoctogit.py: pathExists returns a defered value Elizabeth Flanagan
@ 2014-03-19 20:51 ` Elizabeth Flanagan
2014-03-19 20:51 ` [AB PATCH 09/22] yoctogit.py: Stop passing in srcdir Elizabeth Flanagan
` (13 subsequent siblings)
21 siblings, 0 replies; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:51 UTC (permalink / raw)
To: yocto
We're removing srcdir from yoctogit.py. It causes way too many
issues.
Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
---
lib/python2.7/site-packages/autobuilder/BuildSet.py | 2 --
lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py | 5 ++---
2 files changed, 2 insertions(+), 5 deletions(-)
diff --git a/lib/python2.7/site-packages/autobuilder/BuildSet.py b/lib/python2.7/site-packages/autobuilder/BuildSet.py
index ab7b0bb..4bcda66 100644
--- a/lib/python2.7/site-packages/autobuilder/BuildSet.py
+++ b/lib/python2.7/site-packages/autobuilder/BuildSet.py
@@ -73,7 +73,6 @@ class BuildSet():
mirrordir=None
method = 'clobber'
mode='full'
- srcdir = 'source'
if str(os.environ.get('OPTIMIZED_GIT_CLONE')) == "True":
if 'poky' in layername or 'oecore' in layername or 'eclipse' in layername:
method='movecopy'
@@ -93,7 +92,6 @@ class BuildSet():
func = getattr(m, step)
factoryFN(func(locals()['f'+name],
layername=layername,
- srcdir=srcdir,
mode=mode,
scheduler=name,
method=method,
diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py b/lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py
index 5e95189..029b523 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py
@@ -35,12 +35,11 @@ from buildbot.process import buildstep
class CheckOutLayers(YoctoGit):
def __init__(self, factory, scheduler=None, layername=None, mode='full', storedir=None, mirrordir=None,
- method='clobber', submodules=False, shallow=False, srcdir='source', workdir='build',
+ method='clobber', submodules=False, shallow=False, workdir='build',
timeout=100000, progress=True, retryFetch=True, clobberOnFailure=False,
getDescription=True, argdict=None,
**kwargs):
self.scheduler = scheduler
- self.srcdir=srcdir
self.workdir=workdir
self.branch = ""
self.commit = ""
@@ -63,7 +62,7 @@ class CheckOutLayers(YoctoGit):
for k, v in argdict.iteritems():
setattr(self, k, v)
YoctoGit.__init__(self, repourl=self.repourl, branch=self.branch, mode=self.mode,
- method=self.method, storedir=self.storedir, mirrordir=mirrordir, submodules=self.submodules, srcdir=self.srcdir+"/"+self.repourl,
+ method=self.method, storedir=self.storedir, mirrordir=mirrordir, submodules=self.submodules,
workdir=self.workdir, shallow=self.shallow, progress=self.progress,
retryFetch=self.retryFetch, clobberOnFailure=self.clobberOnFailure,
getDescription=self.getDescription, layername=self.layername, **kwargs)
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* [AB PATCH 09/22] yoctogit.py: Stop passing in srcdir.
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
` (7 preceding siblings ...)
2014-03-19 20:51 ` [AB PATCH 08/22] CheckOutLayers.py: Remove srcdir Elizabeth Flanagan
@ 2014-03-19 20:51 ` Elizabeth Flanagan
2014-03-19 20:51 ` [AB PATCH 10/22] yoctogit.py: Check for mirror on incremental builds Elizabeth Flanagan
` (12 subsequent siblings)
21 siblings, 0 replies; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:51 UTC (permalink / raw)
To: yocto
Don't pass in srcdir. Set it depending on mirrordir/method as a
class level property.
Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
---
.../buildbot/steps/source/yoctogit.py | 20 +++++++++++++++-----
1 file changed, 15 insertions(+), 5 deletions(-)
diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 4c89ceb..62e3a02 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -63,7 +63,7 @@ class YoctoGit(Source):
def __init__(self, repourl=None, branch='HEAD', mode='incremental',
method=None, submodules=False, shallow=False, progress=False,
retryFetch=False, clobberOnFailure=False, getDescription=False,
- workdir='build', layername=None, srcdir='sources', storedir="/tmp/junk", mirrordir=None,
+ workdir='build', layername=None, storedir="/tmp/junk", mirrordir=None,
config=None, **kwargs):
"""
@type repourl: string
@@ -98,9 +98,6 @@ class YoctoGit(Source):
@type getDescription: boolean or dict
@param getDescription: Use 'git describe' to describe the fetched revision
- @type srcdir: string
- @param srcdir: Used to pass in where git clones should occur
-
@type workdir: string
@param workdir: Used to indicate the destination of the copy of srcdir
@@ -125,7 +122,6 @@ class YoctoGit(Source):
self.retryFetch = retryFetch
self.submodules = submodules
self.shallow = shallow
- self.srcdir = srcdir
self.workdir = workdir
self.storedir = storedir
self.mirrordir = mirrordir
@@ -135,6 +131,20 @@ class YoctoGit(Source):
self.mode = mode
self.getDescription = getDescription
self.config = config
+ # We no longer pass srcdir in. There are way too many ways to hang
+ # yourself by doing that.
+ if self.mirrordir:
+ if self.method in ['barecopy', 'movecopy']:
+ self.srcdir='source/'+self.repourl
+ else:
+ if "poky" in self.layername or \
+ "oecore" in self.layername or \
+ "eclipse" in self.layername:
+ self.srcdir = "build"
+ else:
+ self.srcdir = 'build/'+self.layername
+ else:
+ self.srcdir = 'source'
Source.__init__(self, **kwargs)
if self.mode not in ['incremental', 'full']:
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* [AB PATCH 10/22] yoctogit.py: Check for mirror on incremental builds
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
` (8 preceding siblings ...)
2014-03-19 20:51 ` [AB PATCH 09/22] yoctogit.py: Stop passing in srcdir Elizabeth Flanagan
@ 2014-03-19 20:51 ` Elizabeth Flanagan
2014-03-19 20:51 ` [AB PATCH 11/22] yoctogit.py: Use self.srcdir, not local srcdir construct Elizabeth Flanagan
` (11 subsequent siblings)
21 siblings, 0 replies; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:51 UTC (permalink / raw)
To: yocto
We're assuming people are getting to incremental from full.
This is a bad assumption. So, check for mirror in both.
Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
---
.../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 62e3a02..fcbc2cb 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -212,6 +212,7 @@ class YoctoGit(Source):
@defer.inlineCallbacks
def incremental(self):
updatable = yield self._sourcedirIsUpdatable()
+ self.mirrorexists = yield self._mirrorExists()
# if not updateable, do a full checkout
if not updatable:
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* [AB PATCH 11/22] yoctogit.py: Use self.srcdir, not local srcdir construct.
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
` (9 preceding siblings ...)
2014-03-19 20:51 ` [AB PATCH 10/22] yoctogit.py: Check for mirror on incremental builds Elizabeth Flanagan
@ 2014-03-19 20:51 ` Elizabeth Flanagan
2014-03-19 20:51 ` [AB PATCH 12/22] yoctogit.py: movecopy should only move 'build' Elizabeth Flanagan
` (10 subsequent siblings)
21 siblings, 0 replies; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:51 UTC (permalink / raw)
To: yocto
From: Beth Flanagan <elizabeth.flanagan@intel.com>
Since we're now constructing the objects srcdir property, there
is no need to use a local srcdir prop.
Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
---
.../buildbot/steps/source/yoctogit.py | 12 ++----------
1 file changed, 2 insertions(+), 10 deletions(-)
diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index fcbc2cb..1924612 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -258,12 +258,6 @@ class YoctoGit(Source):
log.msg("Source step failed while running command %s" % cmd)
raise buildstep.BuildStepFailed()
d.addCallback(lambda _: evaluateCommand(cmd))
- if self.mirrordir:
- mirror = self.mirrordir + "/" + self.repourl.replace("git://", "") + "/"
- if self.mirrorexists:
- srcdir=mirror
- else:
- srcdir='source/'+self.repourl
if "poky" in self.layername or \
"oecore" in self.layername or \
"eclipse" in self.layername:
@@ -275,7 +269,7 @@ class YoctoGit(Source):
d.addCallback(lambda _: self.incremental())
def copy(_):
cmd = buildstep.RemoteCommand('cpdir',
- {'fromdir': srcdir,
+ {'fromdir': self.srcdir,
'todir': todir,
'logEnviron': self.logEnviron,
'timeout': self.timeout,})
@@ -311,8 +305,6 @@ class YoctoGit(Source):
# method probably won't work correctly crossing filesystems, so utilize
# movecopy with caution.
- srcdir='source/'+self.repourl
-
cmd = buildstep.RemoteCommand('mv', {'fromdir': self.workdir,
'todir': self.storedir + '/' + self.getProperty("DEST") + '/' +
str(int(time.time())) + '-' +
@@ -338,7 +330,7 @@ class YoctoGit(Source):
def copy(_):
cmd = buildstep.RemoteCommand('cpdir',
- {'fromdir': srcdir,
+ {'fromdir': self.srcdir,
'todir': todir,
'logEnviron': self.logEnviron,
'timeout': self.timeout,})
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* [AB PATCH 12/22] yoctogit.py: movecopy should only move 'build'
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
` (10 preceding siblings ...)
2014-03-19 20:51 ` [AB PATCH 11/22] yoctogit.py: Use self.srcdir, not local srcdir construct Elizabeth Flanagan
@ 2014-03-19 20:51 ` Elizabeth Flanagan
2014-03-19 20:51 ` [AB PATCH 13/22] yoctogit.py: DRYing up some code Elizabeth Flanagan
` (9 subsequent siblings)
21 siblings, 0 replies; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:51 UTC (permalink / raw)
To: yocto
From: Beth Flanagan <elizabeth.flanagan@intel.com>
Pin what movecopy moves to be 'build' and not workdir. This keeps
people from doing stupid stuff like passing in a weird workdir.
Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
---
.../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 1924612..66bfb51 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -305,7 +305,7 @@ class YoctoGit(Source):
# method probably won't work correctly crossing filesystems, so utilize
# movecopy with caution.
- cmd = buildstep.RemoteCommand('mv', {'fromdir': self.workdir,
+ cmd = buildstep.RemoteCommand('mv', {'fromdir': 'build',
'todir': self.storedir + '/' + self.getProperty("DEST") + '/' +
str(int(time.time())) + '-' +
str(random.randrange(100, 100000, 2)) +
@@ -322,10 +322,8 @@ class YoctoGit(Source):
"oecore" in self.layername or \
"eclipse" in self.layername:
todir = "build"
- self.workdir = 'source/'+self.repourl.rstrip('/')
else:
todir = "build/" + self.layername
- self.workdir = 'source/'+self.repourl
d.addCallback(lambda _: self.incremental())
def copy(_):
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* [AB PATCH 13/22] yoctogit.py: DRYing up some code
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
` (11 preceding siblings ...)
2014-03-19 20:51 ` [AB PATCH 12/22] yoctogit.py: movecopy should only move 'build' Elizabeth Flanagan
@ 2014-03-19 20:51 ` Elizabeth Flanagan
2014-03-19 20:51 ` [AB PATCH 14/22] yoctogit.py: Fix timeouts Elizabeth Flanagan
` (8 subsequent siblings)
21 siblings, 0 replies; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:51 UTC (permalink / raw)
To: yocto
From: Beth Flanagan <elizabeth.flanagan@intel.com>
Remove duplicate code from movecopy and barecopy and add it to
a new callback _srccopy.
Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
---
.../buildbot/steps/source/yoctogit.py | 73 +++++++++-------------
1 file changed, 30 insertions(+), 43 deletions(-)
diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 66bfb51..c5dbf05 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -194,9 +194,11 @@ class YoctoGit(Source):
return
elif self.method == 'barecopy':
yield self.barecopy()
+ yield self._srccopy()
return
elif self.method == 'movecopy':
yield self.movecopy()
+ yield self._srccopy()
return
if not updatable:
@@ -267,32 +269,6 @@ class YoctoGit(Source):
todir = "build/" + self.layername
self.workdir = 'source/'+self.repourl
d.addCallback(lambda _: self.incremental())
- def copy(_):
- cmd = buildstep.RemoteCommand('cpdir',
- {'fromdir': self.srcdir,
- 'todir': todir,
- 'logEnviron': self.logEnviron,
- 'timeout': self.timeout,})
- cmd.useLog(self.stdio_log, False)
- d = self.runCommand(cmd)
- def evaluateCommand(cmd):
- if cmd.didFail():
- log.msg("Source step failed while running command %s" % cmd)
- raise buildstep.BuildStepFailed()
- d.addCallback(lambda _: evaluateCommand(cmd))
- return d
- d.addCallback(copy)
-
- def resetWorkdir(_):
- if "poky" in self.layername or \
- "oecore" in self.layername or \
- "eclipse" in self.layername:
- self.workdir = 'build'
- else:
- self.workdir = "build/" + self.layername
- return 0
-
- d.addCallback(resetWorkdir)
return d
def movecopy(self):
@@ -325,22 +301,35 @@ class YoctoGit(Source):
else:
todir = "build/" + self.layername
d.addCallback(lambda _: self.incremental())
+ return d
- def copy(_):
- cmd = buildstep.RemoteCommand('cpdir',
- {'fromdir': self.srcdir,
- 'todir': todir,
- 'logEnviron': self.logEnviron,
- 'timeout': self.timeout,})
- cmd.useLog(self.stdio_log, False)
- d = self.runCommand(cmd)
- def evaluateCommand(cmd):
- if cmd.didFail():
- log.msg("Source step failed while running command %s" % cmd)
- raise buildstep.BuildStepFailed()
- d.addCallback(lambda _: evaluateCommand(cmd))
- return d
- d.addCallback(copy)
+ # Buildbot's standard copy has been removed from yoctogit.
+ # We use _srccopy from either movecopy or barecopy as a callback
+ # this uses self.srcdir to copy to the build dir.
+
+ def _srccopy(self):
+ if "poky" in self.layername or \
+ "oecore" in self.layername or \
+ "eclipse" in self.layername:
+ todir = "build"
+ else:
+ todir = "build/" + self.layername
+
+ cmd = buildstep.RemoteCommand('cpdir',
+ {'fromdir': self.srcdir,
+ 'todir': todir,
+ 'logEnviron': self.logEnviron,
+ 'timeout': self.timeout,})
+ cmd.useLog(self.stdio_log, False)
+ d = self.runCommand(cmd)
+
+ def evaluateCommand(cmd):
+ if cmd.didFail():
+ log.msg("Source step failed while running command %s" % cmd)
+ raise buildstep.BuildStepFailed()
+
+ d.addCallback(lambda _: evaluateCommand(cmd))
+ return d
def resetWorkdir(_):
if "poky" in self.layername or \
@@ -353,8 +342,6 @@ class YoctoGit(Source):
d.addCallback(resetWorkdir)
return d
- # Buildbot's standard copy has been removed from yoctogit.
-
def finish(self, res):
d = defer.succeed(res)
def _gotResults(results):
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* [AB PATCH 14/22] yoctogit.py: Fix timeouts.
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
` (12 preceding siblings ...)
2014-03-19 20:51 ` [AB PATCH 13/22] yoctogit.py: DRYing up some code Elizabeth Flanagan
@ 2014-03-19 20:51 ` Elizabeth Flanagan
2014-03-19 20:51 ` [AB PATCH 15/22] yoctogit.py: Set workdir for the class Elizabeth Flanagan
` (7 subsequent siblings)
21 siblings, 0 replies; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:51 UTC (permalink / raw)
To: yocto
From: Beth Flanagan <elizabeth.flanagan@intel.com>
We should really never need this big of a timeout, but
poky-contrib (despite my continual kvetching) can be a
large clone, or an autobuilder can have a large build
directory and the mv across filesystems *which you should
never do!* can go slow.
So let's set this sufficently high enough that we don't
have to have failures from a timeout, but sufficently low
enough that if things go south we don't wait forever to
know about it.
Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
---
.../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index c5dbf05..98a2fba 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -131,6 +131,7 @@ class YoctoGit(Source):
self.mode = mode
self.getDescription = getDescription
self.config = config
+ self.timeout = 10*60
# We no longer pass srcdir in. There are way too many ways to hang
# yourself by doing that.
if self.mirrordir:
@@ -286,7 +287,8 @@ class YoctoGit(Source):
str(int(time.time())) + '-' +
str(random.randrange(100, 100000, 2)) +
self.layername,
- 'logEnviron':self.logEnviron,})
+ 'logEnviron':self.logEnviron,
+ 'timeout': self.timeout,})
cmd.useLog(self.stdio_log, False)
d = self.runCommand(cmd)
def evaluateCommand(cmd):
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* [AB PATCH 15/22] yoctogit.py: Set workdir for the class
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
` (13 preceding siblings ...)
2014-03-19 20:51 ` [AB PATCH 14/22] yoctogit.py: Fix timeouts Elizabeth Flanagan
@ 2014-03-19 20:51 ` Elizabeth Flanagan
2014-03-19 20:51 ` [AB PATCH 16/22] yoctogit.py: Set workdir within dovccmd Elizabeth Flanagan
` (6 subsequent siblings)
21 siblings, 0 replies; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:51 UTC (permalink / raw)
To: yocto
From: Beth Flanagan <elizabeth.flanagan@intel.com>
Stop utilizing resetWorkdir, it just confuses things and is really
not needed.
Just set self.workdir for the class. This ignores the self.workdir passed
in (which I intend on removing soon.)
Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
---
.../buildbot/steps/source/yoctogit.py | 31 ++++------------------
1 file changed, 5 insertions(+), 26 deletions(-)
diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 98a2fba..c739d7d 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -146,6 +146,11 @@ class YoctoGit(Source):
self.srcdir = 'build/'+self.layername
else:
self.srcdir = 'source'
+ if self.method in ['barecopy', 'movecopy']:
+ self.workdir='source/'+self.repourl
+ else:
+ self.workdir = 'build'
+
Source.__init__(self, **kwargs)
if self.mode not in ['incremental', 'full']:
@@ -261,14 +266,6 @@ class YoctoGit(Source):
log.msg("Source step failed while running command %s" % cmd)
raise buildstep.BuildStepFailed()
d.addCallback(lambda _: evaluateCommand(cmd))
- if "poky" in self.layername or \
- "oecore" in self.layername or \
- "eclipse" in self.layername:
- todir = "build"
- self.workdir = 'source/'+self.repourl
- else:
- todir = "build/" + self.layername
- self.workdir = 'source/'+self.repourl
d.addCallback(lambda _: self.incremental())
return d
@@ -296,12 +293,6 @@ class YoctoGit(Source):
log.msg("Source step failed while running command %s" % cmd)
raise buildstep.BuildStepFailed()
d.addCallback(lambda _: evaluateCommand(cmd))
- if "poky" in self.layername or \
- "oecore" in self.layername or \
- "eclipse" in self.layername:
- todir = "build"
- else:
- todir = "build/" + self.layername
d.addCallback(lambda _: self.incremental())
return d
@@ -329,21 +320,9 @@ class YoctoGit(Source):
if cmd.didFail():
log.msg("Source step failed while running command %s" % cmd)
raise buildstep.BuildStepFailed()
-
d.addCallback(lambda _: evaluateCommand(cmd))
return d
- def resetWorkdir(_):
- if "poky" in self.layername or \
- "oecore" in self.layername or \
- "eclipse" in self.layername:
- self.workdir = 'build.'
- else:
- self.workdir = "build/" + self.layername
- return 0
- d.addCallback(resetWorkdir)
- return d
-
def finish(self, res):
d = defer.succeed(res)
def _gotResults(results):
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* [AB PATCH 16/22] yoctogit.py: Set workdir within dovccmd
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
` (14 preceding siblings ...)
2014-03-19 20:51 ` [AB PATCH 15/22] yoctogit.py: Set workdir for the class Elizabeth Flanagan
@ 2014-03-19 20:51 ` Elizabeth Flanagan
2014-03-19 20:51 ` [AB PATCH 17/22] yoctogit.py: change _srccopy fromdir to it's own construction Elizabeth Flanagan
` (5 subsequent siblings)
21 siblings, 0 replies; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:51 UTC (permalink / raw)
To: yocto
From: Beth Flanagan <elizabeth.flanagan@intel.com>
Ensure we're in the correct place by setting workdir in dovccmd.
Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
---
.../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index c739d7d..6990abe 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -376,6 +376,10 @@ class YoctoGit(Source):
full_command.append('-c')
full_command.append('%s=%s' % (name, value))
full_command.extend(command)
+ if self.method in ['barecopy', 'movecopy']:
+ self.workdir='source/'+self.repourl
+ else:
+ self.workdir = 'build'
cmd = buildstep.RemoteShellCommand(self.workdir,
full_command,
env=self.env,
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* [AB PATCH 17/22] yoctogit.py: change _srccopy fromdir to it's own construction
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
` (15 preceding siblings ...)
2014-03-19 20:51 ` [AB PATCH 16/22] yoctogit.py: Set workdir within dovccmd Elizabeth Flanagan
@ 2014-03-19 20:51 ` Elizabeth Flanagan
2014-03-19 20:51 ` [AB PATCH 18/22] PublishArtifacts.py: Exclude genericx86-64 pre-1.5 Elizabeth Flanagan
` (4 subsequent siblings)
21 siblings, 0 replies; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:51 UTC (permalink / raw)
To: yocto
From: Beth Flanagan <elizabeth.flanagan@intel.com>
fromdir was using self.srcdir, but lets be a bit safe and use
a variable we construct in the method.
Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
---
.../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 6990abe..4e1a334 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -307,9 +307,10 @@ class YoctoGit(Source):
todir = "build"
else:
todir = "build/" + self.layername
+ fromdir='source/'+self.repourl
cmd = buildstep.RemoteCommand('cpdir',
- {'fromdir': self.srcdir,
+ {'fromdir': fromdir,
'todir': todir,
'logEnviron': self.logEnviron,
'timeout': self.timeout,})
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* [AB PATCH 18/22] PublishArtifacts.py: Exclude genericx86-64 pre-1.5
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
` (16 preceding siblings ...)
2014-03-19 20:51 ` [AB PATCH 17/22] yoctogit.py: change _srccopy fromdir to it's own construction Elizabeth Flanagan
@ 2014-03-19 20:51 ` Elizabeth Flanagan
2014-03-19 20:51 ` [AB PATCH 19/22] SetDest.py: Use new UI variables Elizabeth Flanagan
` (3 subsequent siblings)
21 siblings, 0 replies; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:51 UTC (permalink / raw)
To: yocto
From: Beth Flanagan <elizabeth.flanagan@intel.com>
We don't build it so we shouldn't publish it.
Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
---
.../autobuilder/buildsteps/PublishArtifacts.py | 15 ++++++++++-----
1 file changed, 10 insertions(+), 5 deletions(-)
diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/PublishArtifacts.py b/lib/python2.7/site-packages/autobuilder/buildsteps/PublishArtifacts.py
index e23ecbf..ef9ad81 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/PublishArtifacts.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/PublishArtifacts.py
@@ -169,11 +169,16 @@ class PublishArtifacts(ShellCommand):
command=command+"echo 'Skipping copy of " + artifact + ".'"
else:
artifact_name, deploy_image_dir = self.getDeployNames(artifact, buildername)
-
- command=command+"mkdir -p " + DEST + "/"+ MACHINE_PUBLISH_DIR +"/" + artifact_name + ";"
- command=command+"cp -R --no-dereference --preserve=links " + \
- deploy_image_dir + \
- "/*"+artifact+"* " + DEST + "/" + MACHINE_PUBLISH_DIR +"/" + artifact_name + ";"
+ self.layerversion_yoctobsp = self.getProperty("layerversion_yoctobsp")
+ if self.layerversion_yoctobsp is not None and \
+ int(self.layerversion_yoctobsp) < 2 and \
+ "genericx86-64" in artifacts:
+ command = command+"echo 'Skipping copy of genericx86-64.'; "
+ else:
+ command=command+"mkdir -p " + DEST + "/"+ MACHINE_PUBLISH_DIR +"/" + artifact_name + ";"
+ command=command+"cp -R --no-dereference --preserve=links " + \
+ deploy_image_dir + \
+ "/*"+artifact+"* " + DEST + "/" + MACHINE_PUBLISH_DIR +"/" + artifact_name + ";"
self.command = command
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* [AB PATCH 19/22] SetDest.py: Use new UI variables.
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
` (17 preceding siblings ...)
2014-03-19 20:51 ` [AB PATCH 18/22] PublishArtifacts.py: Exclude genericx86-64 pre-1.5 Elizabeth Flanagan
@ 2014-03-19 20:51 ` Elizabeth Flanagan
2014-03-19 20:51 ` [AB PATCH 20/22] yocto-start-autobuilder: Make sure janitor logging goes to a log file Elizabeth Flanagan
` (2 subsequent siblings)
21 siblings, 0 replies; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:51 UTC (permalink / raw)
To: yocto
From: Beth Flanagan <elizabeth.flanagan@intel.com>
How we name releases is a lot more complex than the old UI could
handle. This alligns us a bit more with what we actually do.
Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
---
lib/python2.7/site-packages/autobuilder/buildsteps/SetDest.py | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/SetDest.py b/lib/python2.7/site-packages/autobuilder/buildsteps/SetDest.py
index cf901cb..8c1edc5 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/SetDest.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/SetDest.py
@@ -49,8 +49,11 @@ class SetDest(ShellCommand):
else:
buildername=self.getProperty("buildername")
self.workdir=os.path.join(self.slaveworkdir, buildername)
- if str(self.getProperty("custom_release_me_" + buildername)) == "True":
- rel_name = self.getProperty("custom_release_name_" + buildername)
+ if str(self.getProperty("custom_release_me")) == "True":
+ poky_name = self.getProperty("custom_poky_name")
+ poky_number = self.getProperty("custom_poky_number")
+ yocto_number = self.getProperty("custom_yocto_number")
+ rel_name = 'yocto-'+ yocto_number
if not rel_name or rel_name == "":
rel_name == self.getProperty(commit-description)
DEST=os.path.normpath(os.path.join(
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* [AB PATCH 20/22] yocto-start-autobuilder: Make sure janitor logging goes to a log file
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
` (18 preceding siblings ...)
2014-03-19 20:51 ` [AB PATCH 19/22] SetDest.py: Use new UI variables Elizabeth Flanagan
@ 2014-03-19 20:51 ` Elizabeth Flanagan
2014-03-19 20:51 ` [AB PATCH 21/22] buildslave-janitor: Add in try/except handling to make calls robust against failure Elizabeth Flanagan
2014-03-19 20:51 ` [AB PATCH 22/22] yoctogit.py: sync on mv Elizabeth Flanagan
21 siblings, 0 replies; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:51 UTC (permalink / raw)
To: yocto
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently logging from the janitor gets lost. Ensure it gets to a log
file instead including anything to stderr.
---
yocto-start-autobuilder | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/yocto-start-autobuilder b/yocto-start-autobuilder
index b34a60d..74e3a13 100755
--- a/yocto-start-autobuilder
+++ b/yocto-start-autobuilder
@@ -79,7 +79,7 @@ if os.path.isfile(os.path.join(AB_BASE, ".setupdone")):
os.chdir(AB_BASE)
if os.environ["OPTIMIZED_GIT_CLONE"] == "True":
os.chdir(AB_BASE)
- subprocess.Popen("python bin/buildslave-janitor " + os.path.join(AB_BASE, "config/autobuilder.conf"),
+ subprocess.Popen("python bin/buildslave-janitor " + os.path.join(AB_BASE, "config/autobuilder.conf") + ">> yocto-slave/janitor.log 2>&1",
shell=True, stdin=None,
stdout=None, stderr=None,
close_fds=True)
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* [AB PATCH 21/22] buildslave-janitor: Add in try/except handling to make calls robust against failure
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
` (19 preceding siblings ...)
2014-03-19 20:51 ` [AB PATCH 20/22] yocto-start-autobuilder: Make sure janitor logging goes to a log file Elizabeth Flanagan
@ 2014-03-19 20:51 ` Elizabeth Flanagan
2014-03-19 20:51 ` [AB PATCH 22/22] yoctogit.py: sync on mv Elizabeth Flanagan
21 siblings, 0 replies; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:51 UTC (permalink / raw)
To: yocto
From: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a suspicion that the command can fail causing the janitor trash
removal process to exit. This adds in some exception handling to deal
with this. It include a timeout to stop it entering nasty loops.
This combined with the previous patch should let us figure out what
error conditions its hitting.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
bin/buildslave-janitor | 15 ++++++++++-----
1 file changed, 10 insertions(+), 5 deletions(-)
diff --git a/bin/buildslave-janitor b/bin/buildslave-janitor
index 153f18d..7495ce9 100755
--- a/bin/buildslave-janitor
+++ b/bin/buildslave-janitor
@@ -66,11 +66,16 @@ def trash_processor(trashdir):
print("Not prepared to use a trashdir of /")
return
while True:
- files = os.listdir(trashdir)
- if files:
- os.system("ionice -c 3 rm %s -rf" % trashdir)
- else:
- time.sleep(30*60) # 30 minutes
+ try:
+ files = os.listdir(trashdir)
+ if files:
+ os.system("ionice -c 3 rm %s -rf" % trashdir)
+ else:
+ time.sleep(30*60) # 30 minutes
+ except Exception as e:
+ print("Exception %s in trash cleaner" % str(e))
+ time.sleep(60) # 1 minute timeout to prevent crazy looping
+ pass
return
def mirror_processor(mirrordir):
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* [AB PATCH 22/22] yoctogit.py: sync on mv
2014-03-19 20:50 [AB PATCH 00/22] Consolidated patch set Elizabeth Flanagan
` (20 preceding siblings ...)
2014-03-19 20:51 ` [AB PATCH 21/22] buildslave-janitor: Add in try/except handling to make calls robust against failure Elizabeth Flanagan
@ 2014-03-19 20:51 ` Elizabeth Flanagan
2014-03-20 23:18 ` Burton, Ross
21 siblings, 1 reply; 28+ messages in thread
From: Elizabeth Flanagan @ 2014-03-19 20:51 UTC (permalink / raw)
To: yocto
From: Beth Flanagan <elizabeth.flanagan@intel.com>
On systems with very high load sometimes the system still thinks the
build dir is there. I've only seen this once, but a sync here ensures
that the system knows that build has been moved.
Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
---
.../buildbot/steps/source/yoctogit.py | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 4e1a334..c6b8258 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -204,6 +204,7 @@ class YoctoGit(Source):
return
elif self.method == 'movecopy':
yield self.movecopy()
+ yield self._sync()
yield self._srccopy()
return
@@ -269,6 +270,23 @@ class YoctoGit(Source):
d.addCallback(lambda _: self.incremental())
return d
+ def _sync(self):
+ cmd = buildstep.RemoteShellCommand(self.workdir,
+ 'sync',
+ env=self.env,
+ logEnviron=self.logEnviron,
+ timeout=self.timeout)
+ cmd.useLog(self.stdio_log, False)
+ log.msg("syncing filesystem")
+ d = self.runCommand(cmd)
+ def evaluateCommand(cmd):
+ if cmd.didFail():
+ log.msg("Source step failed while running command %s" % cmd)
+ raise buildstep.BuildStepFailed()
+ d.addCallback(lambda _: evaluateCommand(cmd))
+ return d
+
+
def movecopy(self):
# This relies on an external method of cleaning up the todir, like a cron job
# it can be rather disk space intensive.
--
1.8.1.2
^ permalink raw reply related [flat|nested] 28+ messages in thread* Re: [AB PATCH 22/22] yoctogit.py: sync on mv
2014-03-19 20:51 ` [AB PATCH 22/22] yoctogit.py: sync on mv Elizabeth Flanagan
@ 2014-03-20 23:18 ` Burton, Ross
0 siblings, 0 replies; 28+ messages in thread
From: Burton, Ross @ 2014-03-20 23:18 UTC (permalink / raw)
To: Elizabeth Flanagan; +Cc: yocto@yoctoproject.org
On 19 March 2014 20:51, Elizabeth Flanagan <elizabeth.flanagan@intel.com> wrote:
> On systems with very high load sometimes the system still thinks the
> build dir is there. I've only seen this once, but a sync here ensures
> that the system knows that build has been moved.
That smells like a race in the AB code instead of the kernel being
confused over the state of the file system, and the best a sync can do
is reduce the window of failure at the cost of a potentially very
heavy IO burst.
I'd say drop this and if the problem occurs again, attempt to diagnose
where the race is.
Ross
^ permalink raw reply [flat|nested] 28+ messages in thread