[PATCH 0/1] Yocto 1561: enable cleanup of WORKDIR

[PATCH 0/1] Yocto 1561: enable cleanup of WORKDIR

[Kang Kai]

Hi Saul,

This is implementation of Yocto 1561: enable cleanup of WORKDIR.
The original design was create a new script to fullfill the feature, but it
doesn't work. So I add a command line option to do it.

Regards,

The following changes since commit 6b6aa170d5c618cfcd016dd1de926db647f7f932:

  busybox: Restore 'date' compatability (2012-03-01 23:16:43 +0000)

are available in the git repository at:
  git://git.pokylinux.org/poky-contrib kangkai/cleanup
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kangkai/cleanup

Kang Kai (1):
  bitbake: enable cleanup of WORKDIR

 bitbake/bin/bitbake       |    4 +++
 bitbake/lib/bb/command.py |    5 +++
 bitbake/lib/bb/cooker.py  |   63 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 72 insertions(+), 0 deletions(-)

-- 
1.7.5.4

[PATCH 1/1] bitbake: enable cleanup of WORKDIR

[Kang Kai]

[Yocto 1561]
Add a command line option for bitbake to enable cleanup of WORKDIR.
It checks every package build directories under WORKDIR then parse
the directory name to get package name and version. If the version
is not the package prefer version then delete the directory.

Signed-off-by: Kang Kai <kai.kang@windriver.com>
---
 bitbake/bin/bitbake       |    4 +++
 bitbake/lib/bb/command.py |    5 +++
 bitbake/lib/bb/cooker.py  |   63 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 72 insertions(+), 0 deletions(-)

diff --git a/bitbake/bin/bitbake b/bitbake/bin/bitbake
index c06d4e8..89582f7 100755
--- a/bitbake/bin/bitbake
+++ b/bitbake/bin/bitbake
@@ -170,6 +170,10 @@ Default BBFILES are the .bb files in the current directory.""")
 
     parser.add_option("-B", "--bind", help = "The name/address for the bitbake server to bind to",
                action = "store", dest = "bind", default = False)
+
+    parser.add_option("-C", "--clean-workdir", help = "Clean up the old version build directories in workdir",
+               action = "store_true", dest = "clean_workdir", default = False)
+
     options, args = parser.parse_args(sys.argv)
 
     configuration = BBConfiguration(options)
diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
index 1799f1c..a924e85 100644
--- a/bitbake/lib/bb/command.py
+++ b/bitbake/lib/bb/command.py
@@ -294,6 +294,11 @@ class CommandsAsync:
         command.finishAsyncCommand()
     findConfigFilePath.needcache = False
 
+    def cleanupWorkdir(self, command, params):
+        command.cooker.cleanupWorkdir()
+        command.finishAsyncCommand()
+    cleanupWorkdir.needcache = True
+
     def showVersions(self, command, params):
         """
         Show the currently selected versions
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 558eadd..0870488 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -30,6 +30,7 @@ import logging
 import multiprocessing
 import sre_constants
 import threading
+import shutil
 from cStringIO import StringIO
 from contextlib import closing
 from functools import wraps
@@ -247,6 +248,8 @@ class BBCooker:
                 self.commandlineAction['action'] = ["generateDotGraph", self.configuration.pkgs_to_build, self.configuration.cmd]
             else:
                 self.commandlineAction['msg'] = "Please specify a package name for dependency graph generation."
+        elif self.configuration.clean_workdir:
+            self.commandlineAction['action'] = ["cleanupWorkdir"]
         else:
             if self.configuration.pkgs_to_build:
                 self.commandlineAction['action'] = ["buildTargets", self.configuration.pkgs_to_build, self.configuration.cmd]
@@ -254,6 +257,66 @@ class BBCooker:
                 #self.commandlineAction['msg'] = "Nothing to do.  Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information."
                 self.commandlineAction = None
 
+    def cleanupWorkdir(self):
+        """
+        Clean up the package old version build directory in workdir
+        """
+
+        def parseDirName(match):
+            pkg_name = match.group(1)
+            pkg_version = match.group(2)
+            if pkg_name in pkg_pn:
+                pref = preferred_versions[pkg_name]
+                epoch = pref[0][0]
+                if len(epoch) > 0:
+                    epoch += '_'
+                prefstr = epoch + pref[0][1] + '-' +pref[0][2]
+                if pkg_version != prefstr:
+                    obsolete_dirs.append(pkgabsdir)
+                return True
+            return False
+
+        # Need files parsed
+        self.updateCache()
+
+        pkg_pn = self.status.pkg_pn
+        (latest_versions, preferred_versions) = bb.providers.findProviders(self.configuration.data, self.status, pkg_pn)
+
+        tmpdir = self.configuration.data.getVar('TMPDIR', True)
+        workdir = os.path.join(tmpdir, 'work')
+
+        obsolete_dirs = []
+
+        for archdir in os.listdir(workdir):
+            archdir = os.path.join(workdir, archdir)
+            if not os.path.isdir(archdir):
+                pass
+
+            for pkgdir in os.listdir(archdir):
+                pkgabsdir = os.path.join(archdir, pkgdir)
+                if not os.path.isdir(pkgabsdir):
+                    pass
+
+                # parse the package directory names
+                # parse native/nativesdk packages first
+                match = re.match('(.*?-native.*?)-(.*)', pkgdir)
+                if match and parseDirName(match):
+                    continue
+
+                # parse package names which ends with numbers such as 'glib-2.0'
+                match = re.match('(.*?-[\.\d]+)-(\d.*)', pkgdir)
+                if match and parseDirName(match):
+                    continue
+
+                # other packages
+                match = re.match('(.*?)-(\d.*)', pkgdir)
+                if match and parseDirName(match):
+                    continue
+
+        for d in obsolete_dirs:
+            logger.warn("Deleleting %s", d)
+            shutil.rmtree(d, True)
+
     def runCommands(self, server, data, abort):
         """
         Run any queued asynchronous command
-- 
1.7.5.4

Re: [PATCH 1/1] bitbake: enable cleanup of WORKDIR

[Chris Larson]

On Fri, Mar 2, 2012 at 12:54 AM, Kang Kai <kai.kang@windriver.com> wrote:
> [Yocto 1561]
> Add a command line option for bitbake to enable cleanup of WORKDIR.
> It checks every package build directories under WORKDIR then parse
> the directory name to get package name and version. If the version
> is not the package prefer version then delete the directory.
>
> Signed-off-by: Kang Kai <kai.kang@windriver.com>
> ---
>  bitbake/bin/bitbake       |    4 +++
>  bitbake/lib/bb/command.py |    5 +++
>  bitbake/lib/bb/cooker.py  |   63 +++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 72 insertions(+), 0 deletions(-)

This is wrong on multiple levels. bitbake (other than some hacky stuff
in the fetcher that needs to die) has *zero* knowledge of the use of
WORKDIR today, that's entirely metadata implementation.
-- 
Christopher Larson

Re: [PATCH 1/1] bitbake: enable cleanup of WORKDIR

[Richard Purdie]

On Fri, 2012-03-02 at 08:33 -0700, Chris Larson wrote:
> On Fri, Mar 2, 2012 at 12:54 AM, Kang Kai <kai.kang@windriver.com> wrote:
> > [Yocto 1561]
> > Add a command line option for bitbake to enable cleanup of WORKDIR.
> > It checks every package build directories under WORKDIR then parse
> > the directory name to get package name and version. If the version
> > is not the package prefer version then delete the directory.
> >
> > Signed-off-by: Kang Kai <kai.kang@windriver.com>
> > ---
> >  bitbake/bin/bitbake       |    4 +++
> >  bitbake/lib/bb/command.py |    5 +++
> >  bitbake/lib/bb/cooker.py  |   63 +++++++++++++++++++++++++++++++++++++++++++++
> >  3 files changed, 72 insertions(+), 0 deletions(-)
> 
> This is wrong on multiple levels. bitbake (other than some hacky stuff
> in the fetcher that needs to die) has *zero* knowledge of the use of
> WORKDIR today, that's entirely metadata implementation.

Totally agreed. I saw a comment earlier today in the bugzilla about this
and made the same comment even before I saw this patch...

Cheers,

Richard

Re: [PATCH 1/1] bitbake: enable cleanup of WORKDIR

[Kang Kai]

On 2012年03月03日 00:36, Richard Purdie wrote:
> On Fri, 2012-03-02 at 08:33 -0700, Chris Larson wrote:
>> On Fri, Mar 2, 2012 at 12:54 AM, Kang Kai<kai.kang@windriver.com>  wrote:
>>> [Yocto 1561]
>>> Add a command line option for bitbake to enable cleanup of WORKDIR.
>>> It checks every package build directories under WORKDIR then parse
>>> the directory name to get package name and version. If the version
>>> is not the package prefer version then delete the directory.
>>>
>>> Signed-off-by: Kang Kai<kai.kang@windriver.com>
>>> ---
>>>   bitbake/bin/bitbake       |    4 +++
>>>   bitbake/lib/bb/command.py |    5 +++
>>>   bitbake/lib/bb/cooker.py  |   63 +++++++++++++++++++++++++++++++++++++++++++++
>>>   3 files changed, 72 insertions(+), 0 deletions(-)
>> This is wrong on multiple levels. bitbake (other than some hacky stuff
>> in the fetcher that needs to die) has *zero* knowledge of the use of
>> WORKDIR today, that's entirely metadata implementation.
> Totally agreed. I saw a comment earlier today in the bugzilla about this
> and made the same comment even before I saw this patch...
OK. I'll do it in another way.

Regards,
Kai
>
> Cheers,
>
> Richard
>