[yocto-autobuilder][RFC 0/6] New buildsets and options for 1.5

[yocto-autobuilder][RFC 0/6] New buildsets and options for 1.5

[Stefan Stanacar]

Hello Beth,

Would you mind taking a look on these changes? Obviously they are meant post-dylan release, 
but I sent them earlier so you can have time to review and request changes.
It would be great to have them in 1.5 and I plan on adding even more builds 
(especially for multilib images and ipk images which proved tricky with postinstalls and systemd).

Thanks,
Stefan


The following changes since commit 0178163efc6670c9acb982dd579671396cb24290:

  PublishLayerTarballs.py: bad redirect. (2013-04-22 22:56:58 -0700)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib stefans/yab0
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=stefans/yab0

Stefan Stanacar (6):
  nightly-qa-extras.conf: add new buildset config
  CreateAutoConf.py: add initmgr option
  buildset-config/nightly-qa-systemd.conf: add a new target for systemd
    images
  nightly.conf, yoctoAB.conf: enable the new buildsets
  buildsteps/RunSanityTests.py: add suport for custom steps
  CreateAutoConf.py: add option for package types

 buildset-config/nightly-qa-extras.conf               | 16 ++++++++++++++++
 buildset-config/nightly-qa-systemd.conf              | 20 ++++++++++++++++++++
 buildset-config/nightly.conf                         |  3 ++-
 buildset-config/yoctoAB.conf                         |  3 ++-
 .../autobuilder/buildsteps/CreateAutoConf.py         | 20 +++++++++++++++++++-
 .../autobuilder/buildsteps/RunSanityTests.py         |  3 +++
 6 files changed, 62 insertions(+), 3 deletions(-)
 create mode 100644 buildset-config/nightly-qa-extras.conf
 create mode 100644 buildset-config/nightly-qa-systemd.conf

-- 
1.8.1.4

[yocto-autobuilder][RFC 1/6] nightly-qa-extras.conf: add new buildset config

[Stefan Stanacar]

This buildset is for extra images which don't really fit in other buildsets.
For now it includes builds for two images: one with btrfs and diffrent root
home dir and a sato image with read-only rootfs.

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 buildset-config/nightly-qa-extras.conf | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
 create mode 100644 buildset-config/nightly-qa-extras.conf

diff --git a/buildset-config/nightly-qa-extras.conf b/buildset-config/nightly-qa-extras.conf
new file mode 100644
index 0000000..4098f53
--- /dev/null
+++ b/buildset-config/nightly-qa-extras.conf
@@ -0,0 +1,16 @@
+[nightly-qa-extras]
+builders: 'builder1'
+repos: [{'poky':
+            {'repourl':'git://git.yoctoproject.org/poky',
+             'bbpriority':'1',
+             'branch':'master'}}]
+steps: [{'SetDest':{}},
+        {'CheckOutLayers': {}},
+        {'RunPreamble': {}},
+        {'GetDistroVersion' : {'distro': 'poky'}},
+        {'CreateAutoConf': {'machine': 'qemux86-64', 'SDKMACHINE' : 'i686', 'distro': 'poky', 'atextappend' : '\nROOT_HOME = "/root"\nIMAGE_FSTYPES = "btrfs"\nKERNEL_FEATURES_append = " cfg/fs/btrfs"\n'}},
+        {'CreateBBLayersConf': {'buildprovider' : 'yocto'}},
+        {'BuildImages': {'images': 'core-image-sato'}},
+        {'RunSanityTests': {'images': 'core-image-sato'}},
+        {'CreateAutoConf': {'machine': 'qemux86-64', 'SDKMACHINE' : 'i686', 'distro': 'poky', 'atextappend' : '\nEXTRA_IMAGE_FEATURES = "debug-tweaks read-only-rootfs"\n'}},
+        {'BuildImages': {'images': 'core-image-sato'}}]
-- 
1.8.1.4

[yocto-autobuilder][RFC 2/6] CreateAutoConf.py: add initmgr option

[Stefan Stanacar]

Add an option to set the init manager used. This is used
for building images with systemd.
The order is important as it decides the primary one.

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 .../site-packages/autobuilder/buildsteps/CreateAutoConf.py  | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/CreateAutoConf.py b/lib/python2.7/site-packages/autobuilder/buildsteps/CreateAutoConf.py
index 3a4b9f9..4af9860 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/CreateAutoConf.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/CreateAutoConf.py
@@ -36,6 +36,7 @@ class CreateAutoConf(ShellCommand):
         self.adtdev=False
         self.factory = factory
         self.buildappsrcrev = "${AUTOREV}"
+        self.initmgr=None
         self.kwargs = kwargs
         for k, v in argdict.iteritems():
             if type(v) is bool:
@@ -119,6 +120,18 @@ class CreateAutoConf(ShellCommand):
                 else:
                     adtrepo_url=os.environ.get("ADTREPO_URL")
                     fout=fout+'ADTREPO = "' + adtrepo_url + '/' + self.getProperty("distroversion") + '"\n'
+            if self.getProperty("branch_poky") != "danny" and self.initmgr:
+                # we don't neet to test sysvinit only, because that's the default
+                if self.initmgr == "systemd":
+                    fout = fout + 'DISTRO_FEATURES_append = " systemd"\n'
+                    fout = fout + 'VIRTUAL-RUNTIME_init_manager = "systemd"\n'
+                    fout = fout + 'DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"\n'
+                if self.initmgr == "systemd sysvinit":
+                    fout = fout + 'DISTRO_FEATURES_append = " systemd"\n'
+                    fout = fout + 'VIRTUAL-RUNTIME_init_manager = "systemd"\n'
+                if self.initmgr == "sysvinit systemd":
+                    fout = fout + 'DISTRO_FEATURES_append = " systemd"\n'
+                    fout = fout + 'VIRTUAL-RUNTIME_init_manager = "sysvinit"\n'
         if self.atextappend:
             fout = fout + self.atextappend
 
-- 
1.8.1.4

Re: [yocto-autobuilder][RFC 2/6] CreateAutoConf.py: add initmgr option

[Flanagan, Elizabeth]

On Tue, Apr 23, 2013 at 11:52 AM, Stefan Stanacar
<stefanx.stanacar@intel.com> wrote:
> Add an option to set the init manager used. This is used
> for building images with systemd.
> The order is important as it decides the primary one.
>
> Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
> ---
>  .../site-packages/autobuilder/buildsteps/CreateAutoConf.py  | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
>
> diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/CreateAutoConf.py b/lib/python2.7/site-packages/autobuilder/buildsteps/CreateAutoConf.py
> index 3a4b9f9..4af9860 100644
> --- a/lib/python2.7/site-packages/autobuilder/buildsteps/CreateAutoConf.py
> +++ b/lib/python2.7/site-packages/autobuilder/buildsteps/CreateAutoConf.py
> @@ -36,6 +36,7 @@ class CreateAutoConf(ShellCommand):
>          self.adtdev=False
>          self.factory = factory
>          self.buildappsrcrev = "${AUTOREV}"
> +        self.initmgr=None
>          self.kwargs = kwargs
>          for k, v in argdict.iteritems():
>              if type(v) is bool:
> @@ -119,6 +120,18 @@ class CreateAutoConf(ShellCommand):
>                  else:
>                      adtrepo_url=os.environ.get("ADTREPO_URL")
>                      fout=fout+'ADTREPO = "' + adtrepo_url + '/' + self.getProperty("distroversion") + '"\n'
> +            if self.getProperty("branch_poky") != "danny" and self.initmgr:

Here we will probably want to do:

                if "danny" not in self.getProperty("branch_poky")....

That covers cases where we have "danny-next" or some other issue. I'm
sure I do what you do above, and if so, that should be fixed.

> +                # we don't neet to test sysvinit only, because that's the default
> +                if self.initmgr == "systemd":
> +                    fout = fout + 'DISTRO_FEATURES_append = " systemd"\n'
> +                    fout = fout + 'VIRTUAL-RUNTIME_init_manager = "systemd"\n'
> +                    fout = fout + 'DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"\n'
> +                if self.initmgr == "systemd sysvinit":
> +                    fout = fout + 'DISTRO_FEATURES_append = " systemd"\n'
> +                    fout = fout + 'VIRTUAL-RUNTIME_init_manager = "systemd"\n'
> +                if self.initmgr == "sysvinit systemd":
> +                    fout = fout + 'DISTRO_FEATURES_append = " systemd"\n'
> +                    fout = fout + 'VIRTUAL-RUNTIME_init_manager = "sysvinit"\n'
>          if self.atextappend:
>              fout = fout + self.atextappend
>
> --
> 1.8.1.4
>



--
Elizabeth Flanagan
Yocto Project
Build and Release

[yocto-autobuilder][RFC 3/6] buildset-config/nightly-qa-systemd.conf: add a new target for systemd images

[Stefan Stanacar]

Build systemd images in different configurations: systemd only,
systemd+sysvinit and sysvinit+systemd (the order means which is the primary init system).

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 buildset-config/nightly-qa-systemd.conf | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
 create mode 100644 buildset-config/nightly-qa-systemd.conf

diff --git a/buildset-config/nightly-qa-systemd.conf b/buildset-config/nightly-qa-systemd.conf
new file mode 100644
index 0000000..e35e6b1
--- /dev/null
+++ b/buildset-config/nightly-qa-systemd.conf
@@ -0,0 +1,20 @@
+[nightly-qa-systemd]
+builders: 'builder1'
+repos: [{'poky':
+            {'repourl':'git://git.yoctoproject.org/poky',
+             'bbpriority':'1',
+             'branch':'master'}}]
+steps: [{'SetDest':{}},
+        {'CheckOutLayers': {}},
+        {'RunPreamble': {}},
+        {'GetDistroVersion' : {'distro': 'poky'}},
+        {'CreateAutoConf': {'machine': 'qemux86-64', 'SDKMACHINE' : 'i686', 'distro': 'poky', 'initmgr' : 'systemd sysvinit'}},
+        {'CreateBBLayersConf': {'buildprovider' : 'yocto'}},
+        {'BuildImages': {'images': 'core-image-sato'}},
+        {'RunSanityTests': {'images': 'core-image-sato'}},
+        {'CreateAutoConf': {'machine': 'qemux86-64', 'SDKMACHINE' : 'i686', 'distro': 'poky', 'initmgr' : 'sysvinit systemd'}},
+        {'BuildImages': {'images': 'core-image-sato'}},
+        {'RunSanityTests': {'images': 'core-image-sato'}},
+        {'CreateAutoConf': {'machine': 'qemux86-64', 'SDKMACHINE' : 'i686', 'distro': 'poky', 'initmgr' : 'systemd'}},
+        {'BuildImages': {'images': 'core-image-sato'}},
+        {'RunSanityTests': {'images': 'core-image-sato'}}]
-- 
1.8.1.4

[yocto-autobuilder][RFC 4/6] nightly.conf, yoctoAB.conf: enable the new buildsets

[Stefan Stanacar]

Add the new nightly-qa-extras and nightly-qa-systemd
buildsets to the nightly target.

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 buildset-config/nightly.conf | 3 ++-
 buildset-config/yoctoAB.conf | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/buildset-config/nightly.conf b/buildset-config/nightly.conf
index 58a8531..ffbc271 100644
--- a/buildset-config/nightly.conf
+++ b/buildset-config/nightly.conf
@@ -67,7 +67,8 @@ steps: [{'SetDest':{}},
                               'nightly-x32': {},
                               'nightly-x86-64': {}, 'nightly-x86-64-lsb': {},
                               'nightly-x86': {}, 'nightly-x86-lsb': {},
-                              'poky-tiny': {}}}},
+                              'poky-tiny': {},
+                              'nightly-qa-extras': {}, 'nightly-qa-systemd': {}}}},
         {'PrepPkgIndex' : {}},
         {'BuildImages': {'images': 'package-index'}},
         {'CreateAutoConf': {'machine': 'atom-pc', 'SDKMACHINE' : 'x86_64', 
diff --git a/buildset-config/yoctoAB.conf b/buildset-config/yoctoAB.conf
index 5f29400..2f22767 100644
--- a/buildset-config/yoctoAB.conf
+++ b/buildset-config/yoctoAB.conf
@@ -4,4 +4,5 @@ order: ['nightly', 'eclipse-plugin', 'nightly-arm', 'nightly-arm-lsb', 'nightly-
         'nightly-x86-64', 'nightly-x86-64-lsb', 'nightly-x86',
         'nightly-x86-lsb', 'nightly-x32', 'nightly-multilib',
         'nightly-world', 'nightly-non-gpl3', 'nightly-oecore',
-        'nightly-intel-gpl', 'poky-tiny', 'build-appliance']
+        'nightly-intel-gpl', 'poky-tiny', 'build-appliance',
+        'nightly-qa-extras', 'nightly-qa-systemd']
-- 
1.8.1.4

[yocto-autobuilder][RFC 5/6] buildsteps/RunSanityTests.py: add suport for custom steps

[Stefan Stanacar]

Add an option so we can pass custom qemuimage tests,
by appending TEST_SCEN to local.conf.

This isn't used by any buildsets right now.

It could be used when we want to run only certain sanity tests (i.e don't run
all the tests in the image scenario but run only sanity:boot and sanity:dmesg
by appending TEST_SCENE = "sanity:boot sanity:dmesg")

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 lib/python2.7/site-packages/autobuilder/buildsteps/RunSanityTests.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/RunSanityTests.py b/lib/python2.7/site-packages/autobuilder/buildsteps/RunSanityTests.py
index a352de5..1a428be 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/RunSanityTests.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/RunSanityTests.py
@@ -23,6 +23,7 @@ class RunSanityTests(ShellCommand):
     def __init__(self, factory, argdict=None, **kwargs):
         self.factory = factory
         self.images=""
+        self.scene=None
         for k, v in argdict.iteritems():
             setattr(self, k, v)
         self.description = "Running Sanity Tests"
@@ -31,6 +32,8 @@ class RunSanityTests(ShellCommand):
         command = command + "(echo > /dev/tcp/localhost/5901) 2> /dev/null;"
         command = command + 'if [ $? -ne 0 ]; then echo "Starting a VNC server on :1"; vncserver :1; else echo "Will use a VNC server already running on :1"; fi;'
         command = command + "echo 'IMAGETEST = \"qemu\"' >> ./conf/auto.conf;"
+        if self.scene:
+            command = command + "echo 'TEST_SCEN = \"" + self.scene + "\"'" + " >> ./conf/auto.conf;"
         command = command + "DISPLAY=localhost:1 bitbake " + self.images + " -c qemuimagetest_standalone"
         # Timeout needs to be passed to LoggingBuildStep as a kwarg
         self.timeout = 100000
-- 
1.8.1.4

[yocto-autobuilder][RFC 6/6] CreateAutoConf.py: add option for package types

[Stefan Stanacar]

Using this option we'll be able to select only certain packages
to be built and because the order is important we determine the
image type too. E.g 'packages' : 'deb rpm' will build deb and rpm
packages but the image will use deb as package manager.

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 .../site-packages/autobuilder/buildsteps/CreateAutoConf.py         | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/CreateAutoConf.py b/lib/python2.7/site-packages/autobuilder/buildsteps/CreateAutoConf.py
index 4af9860..01abfa9 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/CreateAutoConf.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/CreateAutoConf.py
@@ -22,6 +22,7 @@ class CreateAutoConf(ShellCommand):
         self.machine=""
         self.buildapp=""
         self.distro="poky"
+        self.packages=None
         self.buildhistory=False
         self.gplv3=True
         self.multilib=False
@@ -59,7 +60,11 @@ class CreateAutoConf(ShellCommand):
         else:
             if self.distro != "oecore":
                 fout = fout + 'DISTRO = "' + self.distro + '"\n'
-            fout = fout + 'PACKAGE_CLASSES = "package_rpm package_deb package_ipk"\n'
+            if self.packages:
+                self.classes = [ "package_%s" % p for p in self.packages.split() ]
+                fout = fout + 'PACKAGE_CLASSES = "' + " ".join(self.classes) + '"\n'
+            else:
+                fout = fout + 'PACKAGE_CLASSES = "package_rpm package_deb package_ipk"\n'
             fout = fout + 'BB_NUMBER_THREADS = "' + os.environ.get('BB_NUMBER_THREADS') + '"\n'
             fout = fout + 'PARALLEL_MAKE = "-j ' + os.environ.get('PARALLEL_MAKE') + '"\n'
             fout = fout + 'SDKMACHINE ?= "' + self.SDKMACHINE + '"\n'
-- 
1.8.1.4

Re: [yocto-autobuilder][RFC 0/6] New buildsets and options for 1.5

[Flanagan, Elizabeth]

Yes to these with the one fixed I suggest and a rebase. Nice work.

-b

On Tue, Apr 23, 2013 at 11:52 AM, Stefan Stanacar
<stefanx.stanacar@intel.com> wrote:
> Hello Beth,
>
> Would you mind taking a look on these changes? Obviously they are meant post-dylan release,
> but I sent them earlier so you can have time to review and request changes.
> It would be great to have them in 1.5 and I plan on adding even more builds
> (especially for multilib images and ipk images which proved tricky with postinstalls and systemd).
>
> Thanks,
> Stefan
>
>
> The following changes since commit 0178163efc6670c9acb982dd579671396cb24290:
>
>   PublishLayerTarballs.py: bad redirect. (2013-04-22 22:56:58 -0700)
>
> are available in the git repository at:
>
>   git://git.yoctoproject.org/poky-contrib stefans/yab0
>   http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=stefans/yab0
>
> Stefan Stanacar (6):
>   nightly-qa-extras.conf: add new buildset config
>   CreateAutoConf.py: add initmgr option
>   buildset-config/nightly-qa-systemd.conf: add a new target for systemd
>     images
>   nightly.conf, yoctoAB.conf: enable the new buildsets
>   buildsteps/RunSanityTests.py: add suport for custom steps
>   CreateAutoConf.py: add option for package types
>
>  buildset-config/nightly-qa-extras.conf               | 16 ++++++++++++++++
>  buildset-config/nightly-qa-systemd.conf              | 20 ++++++++++++++++++++
>  buildset-config/nightly.conf                         |  3 ++-
>  buildset-config/yoctoAB.conf                         |  3 ++-
>  .../autobuilder/buildsteps/CreateAutoConf.py         | 20 +++++++++++++++++++-
>  .../autobuilder/buildsteps/RunSanityTests.py         |  3 +++
>  6 files changed, 62 insertions(+), 3 deletions(-)
>  create mode 100644 buildset-config/nightly-qa-extras.conf
>  create mode 100644 buildset-config/nightly-qa-systemd.conf
>
> --
> 1.8.1.4
>



-- 
Elizabeth Flanagan
Yocto Project
Build and Release