[PATCH 0/1] Autobuilder fixes and added functionality

[PATCH 0/1] Autobuilder fixes and added functionality

[Beth Flanagan]

Some minor fixes to the Autobuilder. I've reverted to Buildbot 0.8.2 for the time being,
as well as fixed some issues with poky-autobuild-generate-sources-tarball.

There is also a nasty issue when you have easy_install installed on a system. The BB 
installer will attempt to install via easy_install, however, if the install directory 
does not exist, it fails to install. I now create the install directories beforehand.

This is actually something that should be fixed on the BB side, and when I have some time
I'll look at it.

I've also added a --installbase option which does away with having to set all of the 
--(*)dir= options when you are trying to relocate the install from ~.

Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: eflanagan/bb-0.8.3
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=eflanagan/bb-0.8.3

Thanks,
    Beth Flanagan <elizabeth.flanagan@intel.com>
---


Beth Flanagan (1):
  Fix for easy_install, source tarball and 0.8.3

 scripts/poky-autobuild-generate-sources-tarball |    2 +-
 scripts/poky-setup-autobuilder                  |   27 +++++++++++++++-------
 2 files changed, 19 insertions(+), 10 deletions(-)

[PATCH 1/1] Fix for easy_install, source tarball and 0.8.3

[Beth Flanagan]

A few fixes here. First, reverting down to 0.8.2 for the time being since
there was issues with 0.8.3's git poller. Also, fixing an issue with how
bb setup.py works when setuptools is installed.

Fixing a known error in how generate sources tarball performs

Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
---
 scripts/poky-autobuild-generate-sources-tarball |    2 +-
 scripts/poky-setup-autobuilder                  |   27 +++++++++++++++-------
 2 files changed, 19 insertions(+), 10 deletions(-)

diff --git a/scripts/poky-autobuild-generate-sources-tarball b/scripts/poky-autobuild-generate-sources-tarball
index 11ebda1..c445f24 100755
--- a/scripts/poky-autobuild-generate-sources-tarball
+++ b/scripts/poky-autobuild-generate-sources-tarball
@@ -19,7 +19,7 @@ if [[ -z "$RELEASE" || -z "$VERSION" ]]; then
 fi
 
 if [[ -z "$BRANCH" ]]; then
-	$BRANCH = "master"
+	BRANCH = "master"
 fi
 
 BASEDIR=poky-tarball
diff --git a/scripts/poky-setup-autobuilder b/scripts/poky-setup-autobuilder
index 51e06eb..ad31fef 100755
--- a/scripts/poky-setup-autobuilder
+++ b/scripts/poky-setup-autobuilder
@@ -140,22 +140,23 @@ def configureBot(buildtype):
             bbInstallDir = bbMasterDir
         elif buildtype == "slave":
             bbInstallDir = bbSlaveDir
-        cmd = "cd " + bbSourceDir + "; export PYTHONPATH=" + bbInstallDir + "/lib/python2.6/site-packages/:$PYTHONPATH; python ./setup.py install --prefix=" + bbInstallDir
+        cmd = "cd " + bbSourceDir + "; export PYTHONPATH=" + bbInstallDir + "/lib/python2.6/site-packages/:$PYTHONPATH; python ./setup.py build; python ./setup.py install --prefix=" + bbInstallDir
         os.system (cmd) 
     except:
         print "Issues Configuring "
         sys.exit(1)
 
 
-#BuildBot download parameters
-bbVersion = "0.8.3"
+# BuildBot download parameters
+#bbVersion = "0.8.3"
+# Pushing this back down to 0.8.2 until 0.8.3p1 comes out.
+bbVersion = "0.8.2"
 bbMasterDownloadUrl = "http://buildbot.googlecode.com/files/buildbot-" + bbVersion + ".tar.gz"
 bbSlaveDownloadUrl = "http://buildbot.googlecode.com/files/buildbot-slave-" + bbVersion + ".tar.gz"
 bbConfigDownloadUrl = None
 bbInstallerHome = os.getcwd()
 bbInstallFile = ""
-# We need the expanded path for ~ later
-bbHome = os.path.expanduser('~')
+bbHome = ""
 bbType = None
 bbSlaveDesc="Poky Autobuilder Example"
 
@@ -188,7 +189,8 @@ parser.add_option( "--maxlogs", help = "The max number of logs you want saved. I
                        action = "store", dest = "bbMaxLogs", default = "10" )                        
 parser.add_option( "--adminmail", help = "The administrator email address. If left unset we set it to current user @ localhost",
                        action = "store", dest = "bbAdminMail", default = "root@localhost" )                        
-
+parser.add_option( "--installbase", help = "The base install directory. If left unset we set we'll use --masterdir/slavedir/sourcedir/pstagedir/controldir or their defaults",
+                       action = "store", dest = "bbHome", default = os.path.expanduser('~') )          
 options, args = parser.parse_args( sys.argv )
 
 # We need to decide if we're doing a master or slave install or both
@@ -236,7 +238,10 @@ if bbPStagingDir == "":
 
 if bbType == "master" or bbType == "both":
     try:
-        os.mkdir(bbMasterDir)
+        # We do this here, because if setuputils is installed BuildBot wants to use it the easy_install way
+        # however, their easy_install way will fail if the dir does not exist. It's a bug in the BB installer
+        # that I'm going to have to figure out and push upstream.
+        os.makedirs(bbMasterDir + "/lib/python2.6/site-packages/")
     except:
         print bbMasterDir + " already exists."
         pass
@@ -332,7 +337,10 @@ c['projectURL'] = pokyABConfig.poky_projurl
 
 if bbType == "slave" or bbType == "both":
     try:
-        os.mkdir(bbSlaveDir) 
+        # We do this here, because if setuputils is installed BuildBot wants to use it the easy_install way
+        # however, their easy_install way will fail if the dir does not exist. It's a bug in the BB installer
+        # that I'm going to have to figure out and push upstream.
+        os.makedirs(bbSlaveDir + "/lib/python2.6/site-packages/") 
         os.mkdir(bbOutputDir) 
         os.mkdir(bbPStagingDir) 
 
@@ -400,7 +408,7 @@ PSTAGEDIR=%s
 # We should correct this for slave only/master only builds
 
 print """
-nstallation complete. Please review the output above for any errors.
+Installation complete. Please review the output above for any errors.
 Then edit the master.cfg file in %s and start the build master and
 build slave by running 'make start' in %s and %s directories.
 ---------------------------------------------------------------------
@@ -418,3 +426,4 @@ cd <poky-slave>; make start
 """ % (bbMasterDir, bbMasterDir, bbSlaveDir, bbSlaveDir, bbSlaveDir, bbSlaveDir)
 
 
+
-- 
1.7.1

Re: [PATCH 1/1] Fix for easy_install, source tarball and 0.8.3

[Elizabeth Flanagan]

Sorry about this. My initial reply to scrap the previous pull went to
only me and for some reason my generation of the second pull request is
pulling the wrong branch.

-b
On 01/03/2011 04:39 PM, Flanagan, Elizabeth wrote:
> A few fixes here. First, reverting down to 0.8.2 for the time being since
> there was issues with 0.8.3's git poller. Also, fixing an issue with how
> bb setup.py works when setuptools is installed.
> 
> Fixing a known error in how generate sources tarball performs
> 
> Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
> ---
>  scripts/poky-autobuild-generate-sources-tarball |    2 +-
>  scripts/poky-setup-autobuilder                  |   27 +++++++++++++++-------
>  2 files changed, 19 insertions(+), 10 deletions(-)
> 
> diff --git a/scripts/poky-autobuild-generate-sources-tarball b/scripts/poky-autobuild-generate-sources-tarball
> index 11ebda1..c445f24 100755
> --- a/scripts/poky-autobuild-generate-sources-tarball
> +++ b/scripts/poky-autobuild-generate-sources-tarball
> @@ -19,7 +19,7 @@ if [[ -z "$RELEASE" || -z "$VERSION" ]]; then
>  fi
>  
>  if [[ -z "$BRANCH" ]]; then
> -	$BRANCH = "master"
> +	BRANCH = "master"
>  fi
>  
>  BASEDIR=poky-tarball
> diff --git a/scripts/poky-setup-autobuilder b/scripts/poky-setup-autobuilder
> index 51e06eb..ad31fef 100755
> --- a/scripts/poky-setup-autobuilder
> +++ b/scripts/poky-setup-autobuilder
> @@ -140,22 +140,23 @@ def configureBot(buildtype):
>              bbInstallDir = bbMasterDir
>          elif buildtype == "slave":
>              bbInstallDir = bbSlaveDir
> -        cmd = "cd " + bbSourceDir + "; export PYTHONPATH=" + bbInstallDir + "/lib/python2.6/site-packages/:$PYTHONPATH; python ./setup.py install --prefix=" + bbInstallDir
> +        cmd = "cd " + bbSourceDir + "; export PYTHONPATH=" + bbInstallDir + "/lib/python2.6/site-packages/:$PYTHONPATH; python ./setup.py build; python ./setup.py install --prefix=" + bbInstallDir
>          os.system (cmd) 
>      except:
>          print "Issues Configuring "
>          sys.exit(1)
>  
>  
> -#BuildBot download parameters
> -bbVersion = "0.8.3"
> +# BuildBot download parameters
> +#bbVersion = "0.8.3"
> +# Pushing this back down to 0.8.2 until 0.8.3p1 comes out.
> +bbVersion = "0.8.2"
>  bbMasterDownloadUrl = "http://buildbot.googlecode.com/files/buildbot-" + bbVersion + ".tar.gz"
>  bbSlaveDownloadUrl = "http://buildbot.googlecode.com/files/buildbot-slave-" + bbVersion + ".tar.gz"
>  bbConfigDownloadUrl = None
>  bbInstallerHome = os.getcwd()
>  bbInstallFile = ""
> -# We need the expanded path for ~ later
> -bbHome = os.path.expanduser('~')
> +bbHome = ""
>  bbType = None
>  bbSlaveDesc="Poky Autobuilder Example"
>  
> @@ -188,7 +189,8 @@ parser.add_option( "--maxlogs", help = "The max number of logs you want saved. I
>                         action = "store", dest = "bbMaxLogs", default = "10" )                        
>  parser.add_option( "--adminmail", help = "The administrator email address. If left unset we set it to current user @ localhost",
>                         action = "store", dest = "bbAdminMail", default = "root@localhost" )                        
> -
> +parser.add_option( "--installbase", help = "The base install directory. If left unset we set we'll use --masterdir/slavedir/sourcedir/pstagedir/controldir or their defaults",
> +                       action = "store", dest = "bbHome", default = os.path.expanduser('~') )          
>  options, args = parser.parse_args( sys.argv )
>  
>  # We need to decide if we're doing a master or slave install or both
> @@ -236,7 +238,10 @@ if bbPStagingDir == "":
>  
>  if bbType == "master" or bbType == "both":
>      try:
> -        os.mkdir(bbMasterDir)
> +        # We do this here, because if setuputils is installed BuildBot wants to use it the easy_install way
> +        # however, their easy_install way will fail if the dir does not exist. It's a bug in the BB installer
> +        # that I'm going to have to figure out and push upstream.
> +        os.makedirs(bbMasterDir + "/lib/python2.6/site-packages/")
>      except:
>          print bbMasterDir + " already exists."
>          pass
> @@ -332,7 +337,10 @@ c['projectURL'] = pokyABConfig.poky_projurl
>  
>  if bbType == "slave" or bbType == "both":
>      try:
> -        os.mkdir(bbSlaveDir) 
> +        # We do this here, because if setuputils is installed BuildBot wants to use it the easy_install way
> +        # however, their easy_install way will fail if the dir does not exist. It's a bug in the BB installer
> +        # that I'm going to have to figure out and push upstream.
> +        os.makedirs(bbSlaveDir + "/lib/python2.6/site-packages/") 
>          os.mkdir(bbOutputDir) 
>          os.mkdir(bbPStagingDir) 
>  
> @@ -400,7 +408,7 @@ PSTAGEDIR=%s
>  # We should correct this for slave only/master only builds
>  
>  print """
> -nstallation complete. Please review the output above for any errors.
> +Installation complete. Please review the output above for any errors.
>  Then edit the master.cfg file in %s and start the build master and
>  build slave by running 'make start' in %s and %s directories.
>  ---------------------------------------------------------------------
> @@ -418,3 +426,4 @@ cd <poky-slave>; make start
>  """ % (bbMasterDir, bbMasterDir, bbSlaveDir, bbSlaveDir, bbSlaveDir, bbSlaveDir)
>  
>  
> +

[PATCH 0/1] Autobuilder fixes and added functionality

[Beth Flanagan]

Some minor fixes to the Autobuilder. I've reverted to Buildbot 0.8.2 for the time being,
as well as fixed some issues with poky-autobuild-generate-sources-tarball.

There is also a nasty issue when you have easy_install installed on a system. The BB 
installer will attempt to install via easy_install, however, if the install directory 
does not exist, it fails to install. I now create the install directories beforehand.

This is actually something that should be fixed on the BB side, and when I have some time
I'll look at it.

I've also added a --installbase option which does away with having to set all of the 
--(*)dir= options when you are trying to relocate the install from ~.


Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: eflanagan/bb-0.8.3
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=eflanagan/bb-0.8.3

Thanks,
    Beth Flanagan <elizabeth.flanagan@intel.com>
---


Beth Flanagan (1):
  Fix for easy_install, source tarball and 0.8.3

 scripts/poky-autobuild-generate-sources-tarball |    2 +-
 scripts/poky-setup-autobuilder                  |   27 +++++++++++++++-------
 2 files changed, 19 insertions(+), 10 deletions(-)

Re: [PATCH 0/1] Autobuilder fixes and added functionality

[Paul Eggleton]

On Tuesday 04 January 2011 00:58:43 Beth Flanagan wrote:
> Some minor fixes to the Autobuilder. I've reverted to Buildbot 0.8.2 for the
> time being, as well as fixed some issues with poky-autobuild-generate-
> sources-tarball.
> 
> There is also a nasty issue when you have easy_install installed on a
> system. The BB installer will attempt to install via easy_install, however,
> if the install directory does not exist, it fails to install. I now create
> the install directories beforehand.
> 
> This is actually something that should be fixed on the BB side, and when I
> have some time I'll look at it.
> 
> I've also added a --installbase option which does away with having to set
> all of the --(*)dir= options when you are trying to relocate the install
> from ~.

FYI I've attempted to install the autobuilder on my Fedora build box, however I ran into a few issues:

1) If you git clone poky-autobuilder into your home directory the install script falls over because it tries to create that same named directory by default. Of course you can override this path or clone it somewhere else but it would be nice if it could work out of the box.

2) The install script doesn't handle Python 2.7, but only due to paths. I have a patch for this which I'll post when your latest patches are merged.

3) BuildBot 0.8.2 unfortunately suffers from buildbot bug #1697 on my system, making it not very useful (as you can't view any of the build logs through the web interface). It's fixed in 0.8.3. More details:

   http://trac.buildbot.net/ticket/1697

4) Looking through the logs locally I found some disturbing output from poky, I guess I may have missed some configuration option, or otherwise something else is going badly wrong:

----- snip -----
NOTE: package quilt-native-0.48-r0: task do_fetch: Started
ERROR: Error, lockfile path is not writable!: /
NOTE: package gnu-config-native-0.1+cvs20080123-r1: task do_fetch: Started
NOTE: package m4-native-1.4.15-r0: task do_fetch: Started
ERROR: Error, lockfile path is not writable!: /
NOTE: package autoconf-native-2.65-r2: task do_fetch: Started
ERROR: Error, lockfile path is not writable!: /
NOTE: package automake-native-1.11.1-r1: task do_fetch: Started
ERROR: Error, lockfile path is not writable!: /
NOTE: package libtool-native-2.4-r0: task do_fetch: Started
ERROR: Error, lockfile path is not writable!: /
NOTE: package zlib-native-1.2.5-r0: task do_fetch: Started
ERROR: Error, lockfile path is not writable!: /
NOTE: package unifdef-native-2.6.18+git-r0: task do_unpack: Started
NOTE: package ncurses-native-5.7-r0: task do_fetch: Started
ERROR: Error, lockfile path is not writable!: /
ERROR: Function 'Unknown fetch Error: 1' failed
NOTE: package quilt-native-0.48-r0: task Unknown fetch Error: 1: Failed
----- snip -----

(Interesting also that the function / task names are being replaced with the result message, is this expected behaviour from poky?)

5) I did try with 0.8.3 earlier (i.e. without your patch), but I suspect I was hit by the git poller problem that 0.8.3p1 is supposed to fix as there were a lot of strange bitbake parsing errors. I also found that I needed to have python-jinja2 and python-setuptools installed, whereas with your latest patch using 0.8.2 it seemed to be installing local copies of these (along with twisted).

Cheers,
Paul

Re: [PATCH 0/1] Autobuilder fixes and added functionality

[Flanagan, Elizabeth]

Paul,

> 1) If you git clone poky-autobuilder into your home directory the install script falls over because it tries to create that same named directory by default. 
> Of course you can override this path or clone it somewhere else but it would be nice if it could work out of the box.

Yes, that's an issue I know about. Not sure how to handle it other than renaming either the git directory or the default autobuilder scripts home. 

> 2) The install script doesn't handle Python 2.7, but only due to paths. I have a patch for this which I'll post when your latest patches are merged.

Excellent. Ty.

>3) BuildBot 0.8.2 unfortunately suffers from buildbot bug #1697 on my system, making it not very useful (as you can't view any of the build logs through the web interface). It's fixed in 0.8.3. More details:
>
>   http://trac.buildbot.net/ticket/1697

I hadn't run into this, so thanks for pointing it out. I'm tempted to wait for 0.8.3p1 which should have the fix for this, rather than trying to pull the fix out to patch 0.8.2

> 4) Looking through the logs locally I found some disturbing output from poky, I guess I may have missed some configuration option, or otherwise something else is going badly wrong:

I found that error yesterday and will have a fix for it today. 

>5) I did try with 0.8.3 earlier (i.e. without your patch), but I suspect I was hit by the git poller problem that 0.8.3p1 is supposed to fix as there were a lot of strange bitbake parsing errors. I also found that I needed to have python-jinja2 and 
> python-setuptools installed, whereas with your latest patch using 0.8.2 it seemed to be installing local copies of these (along with twisted).

I don't recall needed setuptools, but that may be just because I tested on my desktop. I'll look into this when I push it to 0.8.3p1

-b

Re: [PATCH 0/1] Autobuilder fixes and added functionality

[Scott Garman]

On 01/05/2011 08:30 AM, Flanagan, Elizabeth wrote:
>> 3) BuildBot 0.8.2 unfortunately suffers from buildbot bug #1697 on
>> my system, making it not very useful (as you can't view any of the
>> build logs through the web interface). It's fixed in 0.8.3. More
>> details:
>>
>> http://trac.buildbot.net/ticket/1697
>
> I hadn't run into this, so thanks for pointing it out. I'm tempted to
> wait for 0.8.3p1 which should have the fix for this, rather than
> trying to pull the fix out to patch 0.8.2

Hi Beth,

Do you want me to merge this patchset into master now, or should I wait 
for a respin based on Buildbot 0.8.3p1 (which should be out by the end 
of the week, I think)?

Scott

-- 
Scott Garman
Embedded Linux Distro Engineer - Yocto Project

Re: [PATCH 0/1] Autobuilder fixes and added functionality

[Flanagan, Elizabeth]

Let's wait on all this. I'm running into a few issues with the AB that I want to fix that are unrelated to the upgrade and would rather do one big patchset that did the upgrade and fixed some of the issues I'm trying to fix.

I'll keep everyone posted. That said, we should probably discuss tagging/versioning process for non-poky projects within yocto. If I remember correctly, there was discussion about this going on already, and I'm a little uneasy leaving the ab master broken without a tag to a known good version. Anyone have any thoughts on this? 

-b
________________________________________
From: yocto-bounces@yoctoproject.org [yocto-bounces@yoctoproject.org] On Behalf Of Scott Garman [scott.a.garman@intel.com]
Sent: Wednesday, January 05, 2011 9:22 AM
To: yocto@yoctoproject.org
Subject: Re: [yocto] [PATCH 0/1] Autobuilder fixes and added functionality

On 01/05/2011 08:30 AM, Flanagan, Elizabeth wrote:
>> 3) BuildBot 0.8.2 unfortunately suffers from buildbot bug #1697 on
>> my system, making it not very useful (as you can't view any of the
>> build logs through the web interface). It's fixed in 0.8.3. More
>> details:
>>
>> http://trac.buildbot.net/ticket/1697
>
> I hadn't run into this, so thanks for pointing it out. I'm tempted to
> wait for 0.8.3p1 which should have the fix for this, rather than
> trying to pull the fix out to patch 0.8.2

Hi Beth,

Do you want me to merge this patchset into master now, or should I wait
for a respin based on Buildbot 0.8.3p1 (which should be out by the end
of the week, I think)?

Scott

--
Scott Garman
Embedded Linux Distro Engineer - Yocto Project
_______________________________________________
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

[PATCH 0/1] Autobuilder fixes and added functionality

[Beth Flanagan]

Some minor fixes to the Autobuilder. I've reverted to Buildbot 0.8.2 for the time being,
as well as fixed some issues with poky-autobuild-generate-sources-tarball.

There is also a nasty issue when you have easy_install installed on a system. The BB 
installer will attempt to install via easy_install, however, if the install directory 
does not exist, it fails to install. I now create the install directories beforehand.

This is actually something that should be fixed on the BB side, and when I have some time
I'll look at it.

I've also added a --installbase option which does away with having to set all of the 
--(*)dir= options when you are trying to relocate the install from ~.

Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: eflanagan/poky-ab-fixes
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=eflanagan/poky-ab-fixes

Thanks,
    Beth Flanagan <elizabeth.flanagan@intel.com>
---


Beth Flanagan (1):
  Fix for easy_install, source tarball and 0.8.3

 scripts/poky-autobuild-generate-sources-tarball |    2 +-
 scripts/poky-setup-autobuilder                  |   59 +++++++++++++----------
 2 files changed, 35 insertions(+), 26 deletions(-)