From: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
To: yocto@yoctoproject.org
Subject: [AB PATCH 00/27] Autobuilder patches
Date: Wed, 5 Mar 2014 10:22:51 -0800 [thread overview]
Message-ID: <cover.1394042358.git.elizabeth.flanagan@intel.com> (raw)
This is a fairly large patchset and I'd like at least a second set of eyes on it.
There are a few things this does:
1. Updates the Autobuilder parser.
The autobuilder parser was fairly junky. It didn't dive too far into what it was
parsing. This made it so people couldn't do triggers of triggers because the
props wouldn't filter all the way up to the master trigger. This should be
possible now.
2. Simplify the commit/repo/branch combos/remove mix and match.
A long time ago we needed the ability to mix and match repos/branches when we
were building out nightly. This made it so that we could build everything but
nightly-foo on master. That use case is no longer really needed.
It was also a major complaint that the UI was complicated. This trims the UI
down and makes it much sparser
3. Git fetcher mirror.
The git fetcher with buildbot doesn't do a lot of what we need it to. This set
of patches creates a mirror on each slave of the major repos. This allows us to
clone from those repos. The git fetcher also has a movecopy method that allows
us to instead of clobbering a build dir, move it off to a trash directory. This
works well when your trash and slaves are on the same partition but is slowed
when you move it across disk.
When the ab starts a service runs in the background that 1. Updates all of the
repos every XX minutes 2. takes out the trash every XX minutes.
The following changes since commit ba57054aea5627b557327632ad3d6c2a1faa14f2:
4. Simplifies wait/no_wait triggers.
This needs to be revisited, as I know of a better way to do this now, but this
allows us to more optimally utilize the ab resources by triggering all builds,
but putting a few second sleep in for the no_wait builds to queue.
Bugs fixed:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5454
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5433
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5027
PublishLayerTarballs: Don't hang is PUBLISH_BUILDS isn't set (2014-02-18 10:37:36 +0000)
are available in the git repository at:
git://git.yoctoproject.org/yocto-autobuilder eflanagan/yocto-autobuilder-dev-rebase
http://git.yoctoproject.org/cgit.cgi/yocto-autobuilder/log/?h=eflanagan/yocto-autobuilder-dev-rebase
Beth Flanagan (8):
PrepPkgIndex.py: ln instead of cp
Autobuilder.py/Buildset.py: More sane parser/UI
nightly.conf: Add additional parameters
CheckOutLayers.py/ResolveLayerHead.py: support simplified git
Autobuilder.py: remove DEBUG code
BuildSet.py: Remove more debug code.
yocto-start/stop-autobuilder: start buildslave-janitor
forms.html/layout.html: support simplified UI
Elizabeth Flanagan (12):
nightly-world-uclibc: uclibc target
SyncPersistDB.py: First run failure correction
BuildSet.py: Set workdir based on layer not order
BuildSet.py: remove trailing spaces
yoctogit.py: fix callbacks and instance attrs
PublishArtifacts.py: Fix poky-tiny publish
PublishArtifacts.py: genericx86-64 publish
BuildEclipsePlugin.py: Change location of build.
PublishArtifacts.py: Change where we find eclipse artifacts
yocto-autobuilder-setup: set the correct bb version
fs.py: Bump default timeout
autobuilder: whitespace removal
Richard Purdie (5):
Add mirror functionality to git fetcher
trigger.py: Allow wait and nowait triggers
yoctogit: Don't use branch args for referenced clone
bin/buildslave-janitor: For fast-git checkouts
yoctogit: Remove use of oe.path.exists()
Stefan Stanacar (2):
buildsets: add a new buildset to run builds on target
Add buildstep and buildset for oe-selftest script
bin/buildslave-janitor | 125 ++++++
buildset-config.master/nightly-oe-selftest.conf | 15 +
.../nightly-qa-targetbuilds.conf | 26 ++
buildset-config.master/nightly-world-uclibc.conf | 18 +
buildset-config.master/nightly.conf | 40 +-
config/autobuilder.conf.example | 1 +
.../site-packages/autobuilder/Autobuilder.py | 422 ++++++++++-----------
.../site-packages/autobuilder/BuildSet.py | 67 ++--
.../autobuilder/buildsteps/BuildEclipsePlugin.py | 8 +-
.../autobuilder/buildsteps/BuildToolchainImages.py | 6 +-
.../autobuilder/buildsteps/CheckOutLayers.py | 32 +-
.../autobuilder/buildsteps/CreateAutoConf.py | 17 +-
.../autobuilder/buildsteps/CreateBBLayersConf.py | 10 +-
.../autobuilder/buildsteps/CreateCurrentLink.py | 6 +-
.../buildsteps/CreateIntelBSPPackage.py | 8 +-
.../autobuilder/buildsteps/GetDistroVersion.py | 4 +-
.../autobuilder/buildsteps/GetLayerVersion.py | 10 +-
.../autobuilder/buildsteps/HelloWorld.py | 6 +-
.../autobuilder/buildsteps/MaintainPersistDB.py | 12 +-
.../autobuilder/buildsteps/PrepPkgIndex.py | 12 +-
.../autobuilder/buildsteps/PublishArtifacts.py | 28 +-
.../autobuilder/buildsteps/PublishLayerTarballs.py | 33 +-
.../autobuilder/buildsteps/ResolveLayerHead.py | 32 +-
.../autobuilder/buildsteps/RunOeSelftest.py | 26 ++
.../autobuilder/buildsteps/RunSanityTests.py | 6 +-
.../autobuilder/buildsteps/SetDest.py | 12 +-
.../site-packages/autobuilder/buildsteps/Sleep.py | 20 +-
.../autobuilder/buildsteps/SyncPersistDB.py | 11 +-
.../autobuilder/buildsteps/TestFailStep.py | 4 +-
.../autobuilder/buildsteps/YoctoBlocker.py | 2 +-
.../buildbot/status/web/templates/forms.html | 1 -
.../buildbot/status/web/templates/layout.html | 185 +--------
.../buildbot/steps/source/yoctogit.py | 99 +++--
.../buildbot/steps/trigger.py | 15 +-
.../buildslave/commands/fs.py | 4 +-
yocto-autobuilder-setup | 4 +-
yocto-start-autobuilder | 24 +-
yocto-stop-autobuilder | 41 +-
38 files changed, 756 insertions(+), 636 deletions(-)
create mode 100755 bin/buildslave-janitor
create mode 100644 buildset-config.master/nightly-oe-selftest.conf
create mode 100644 buildset-config.master/nightly-qa-targetbuilds.conf
create mode 100644 buildset-config.master/nightly-world-uclibc.conf
create mode 100644 lib/python2.7/site-packages/autobuilder/buildsteps/RunOeSelftest.py
--
1.8.1.2
next reply other threads:[~2014-03-05 18:26 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-03-05 18:22 Elizabeth Flanagan [this message]
2014-03-05 18:22 ` [AB PATCH 01/27] buildsets: add a new buildset to run builds on target Elizabeth Flanagan
2014-03-05 18:22 ` [AB PATCH 02/27] Add buildstep and buildset for oe-selftest script Elizabeth Flanagan
2014-03-05 18:22 ` [AB PATCH 03/27] nightly-world-uclibc: uclibc target Elizabeth Flanagan
2014-03-05 18:31 ` Khem Raj
2014-03-05 21:39 ` Saul Wold
2014-03-05 18:22 ` [AB PATCH 04/27] PrepPkgIndex.py: ln instead of cp Elizabeth Flanagan
2014-03-05 18:22 ` [AB PATCH 05/27] Add mirror functionality to git fetcher Elizabeth Flanagan
2014-03-05 18:22 ` [AB PATCH 06/27] Autobuilder.py/Buildset.py: More sane parser/UI Elizabeth Flanagan
2014-03-06 13:43 ` Bryan Evenson
2014-03-07 2:53 ` Flanagan, Elizabeth
2014-03-05 18:22 ` [AB PATCH 07/27] nightly.conf: Add additional parameters Elizabeth Flanagan
2014-03-05 18:22 ` [AB PATCH 08/27] CheckOutLayers.py/ResolveLayerHead.py: support simplified git Elizabeth Flanagan
2014-03-05 18:23 ` [AB PATCH 09/27] Autobuilder.py: remove DEBUG code Elizabeth Flanagan
2014-03-05 18:23 ` [AB PATCH 10/27] BuildSet.py: Remove more debug code Elizabeth Flanagan
2014-03-05 18:23 ` [AB PATCH 11/27] trigger.py: Allow wait and nowait triggers Elizabeth Flanagan
2014-03-05 18:23 ` [AB PATCH 12/27] yoctogit: Don't use branch args for referenced clone Elizabeth Flanagan
2014-03-05 18:23 ` [AB PATCH 13/27] yocto-start/stop-autobuilder: start buildslave-janitor Elizabeth Flanagan
2014-03-05 18:23 ` [AB PATCH 14/27] bin/buildslave-janitor: For fast-git checkouts Elizabeth Flanagan
2014-03-05 18:23 ` [AB PATCH 15/27] forms.html/layout.html: support simplified UI Elizabeth Flanagan
2014-03-05 18:23 ` [AB PATCH 16/27] SyncPersistDB.py: First run failure correction Elizabeth Flanagan
2014-03-05 18:23 ` [AB PATCH 17/27] BuildSet.py: Set workdir based on layer not order Elizabeth Flanagan
2014-03-05 18:23 ` [AB PATCH 18/27] yoctogit: Remove use of oe.path.exists() Elizabeth Flanagan
2014-03-05 18:23 ` [AB PATCH 19/27] BuildSet.py: remove trailing spaces Elizabeth Flanagan
2014-03-05 18:23 ` [AB PATCH 20/27] yoctogit.py: fix callbacks and instance attrs Elizabeth Flanagan
2014-03-05 18:23 ` [AB PATCH 21/27] PublishArtifacts.py: Fix poky-tiny publish Elizabeth Flanagan
2014-03-05 18:23 ` [AB PATCH 22/27] PublishArtifacts.py: genericx86-64 publish Elizabeth Flanagan
2014-03-05 18:23 ` [AB PATCH 23/27] BuildEclipsePlugin.py: Change location of build Elizabeth Flanagan
2014-03-05 18:23 ` [AB PATCH 24/27] PublishArtifacts.py: Change where we find eclipse artifacts Elizabeth Flanagan
2014-03-05 18:23 ` [AB PATCH 25/27] yocto-autobuilder-setup: set the correct bb version Elizabeth Flanagan
2014-03-05 18:23 ` [AB PATCH 26/27] fs.py: Bump default timeout Elizabeth Flanagan
2014-03-05 18:23 ` [AB PATCH 27/27] autobuilder: whitespace removal Elizabeth Flanagan
2014-03-11 16:49 ` [AB PATCH 00/27] Autobuilder patches Burton, Ross
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1394042358.git.elizabeth.flanagan@intel.com \
--to=elizabeth.flanagan@intel.com \
--cc=yocto@yoctoproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.