[PATCH 0/2] V2 Disk space monitoring

[PATCH 0/2] V2 Disk space monitoring

[Robert Yang]

Changes of V2:
* Rebase the code to the up to date master branch

Here are the test info:
1) set the following variables in conf/local.conf for testing:
   BB_DISKMON_DIRS = "${TMPDIR} ${DL_DIR} ${SSTATE_DIR} /home/lyang1"
   BB_DISKMON_MINSPACE = "200G"
   BB_DISKMON_MININODES = "20M"
   BB_DISKMON_INTERVAL = "30s"

   The TMPDIR, DL_DIR and SSTATE_DIR are in the same disk(/dev/sdb4) in
   my build, so I added another direcotory /home/lyang1 to test the
   ability of monitoring multi mount points.

2) bitbake core-image-sato
   WARNING: The free space of /dev/sdb5 is running low (25.915GB left)
   WARNING: The free inode of /dev/sdb5 is running low (1788.202K left)
   WARNING: The free space of /dev/sdb4 is running low (189.309GB left)
   WARNING: The free inode of /dev/sdb4 is running low (18005.443K left)

3) Aalso have tested about the following items:
   a) Use ctrl-c to stop the monitor
   b) The monitor should stop when the build stops
   c) Test the space unit G, B, K(case in-sensitive) and the time unit
      h, m, s(case-sensitive)

// Robert

The following changes since commit 468998cddbe1a803096c9b357e1b5daa3b7e8c2e:

  command.py: add parseConfigurationFiles API (2012-01-06 16:01:44 +0000)

are available in the git repository at:
  git://git.pokylinux.org/poky-contrib robert/disk-space-monitor
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/disk-space-monitor

Robert Yang (2):
  Disk space monitoring
  Add config sample for disk space monitoring

 bitbake/bin/bitbake               |   13 +++
 bitbake/lib/bb/monitordisk.py     |  153 +++++++++++++++++++++++++++++++++++++
 meta-yocto/conf/local.conf.sample |   28 +++++++
 3 files changed, 194 insertions(+), 0 deletions(-)
 create mode 100644 bitbake/lib/bb/monitordisk.py

-- 
1.7.4.1

[PATCH 1/2] V2 Disk space monitoring

[Robert Yang]

Monitor disk availability and warn the user when the free disk space or
amount of free inode is running low, it is enabled when BB_DISKMON_DIRS
is set.

* Variable meanings(from meta-yocto/conf/local.conf.sample):

  # Set the directories to monitor for disk usage, if more than one
  # directories are in the same device, then only one directory would be
  # warned since the monitor is based on the device.
  #
  BB_DISKMON_DIRS = "${TMPDIR} ${DL_DIR} ${SSTATE_DIR}"
  #
  # Set the minimum amount of disk space to warn, the unit can be G,
  # M or K, but do not use GB, MB or KB (B is not needed).
  # The default vaule is 1G if not set.
  #
  #BB_DISKMON_MINSPACE = "1G"
  #
  # Set the minimum amount of inodes to warn, the unit can be G, M or K.
  # The default value is 1K if not set.
  #
  #BB_DISKMON_MININODES = "1K"
  #
  # Set the monitor frequency, the unit can be h(hour), m(minute) or
  # s(second), when no unit is specified, the default unit is s(second),
  # for example, "60" means 60 seconds.
  # The default value is 60s if not set.
  #
  #BB_DISKMON_INTERVAL = "60s"

* Problems:

  1) The disk space monitor is running in a sub process of
     bin/bitbake, so it can't use the same logger as bitbake(or I don't
     know how to use it, I've tried to use the logger.warn('some
     information'), but it would print nothing in the subprocess),
     another solution maybe use an independent logger in the subprocess,
     but it seems that it doesn't worth because of the two reasons:
     a) The logger of bin/bitbake can't control the monitor's
        independent logger
     b) The monitor doesn't need print many things

     It would be great if anyone could give me more instructions.

 2) The code is placed in the new file lib/bb/monitordisk.py, since it
    seems that it is more independent.

[YOCTO #1589]
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/bin/bitbake           |   13 ++++
 bitbake/lib/bb/monitordisk.py |  153 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 166 insertions(+), 0 deletions(-)
 create mode 100644 bitbake/lib/bb/monitordisk.py

diff --git a/bitbake/bin/bitbake b/bitbake/bin/bitbake
index c2e6822..6bc34a4 100755
--- a/bitbake/bin/bitbake
+++ b/bitbake/bin/bitbake
@@ -39,6 +39,8 @@ import bb.msg
 from bb import cooker
 from bb import ui
 from bb import server
+from bb import monitordisk
+from multiprocessing import Process
 
 __version__ = "1.15.0"
 logger = logging.getLogger("BitBake")
@@ -223,6 +225,13 @@ Default BBFILES are the .bb files in the current directory.""")
     server.saveConnectionDetails()
     server.detach(cooker_logfile)
 
+    # For disk space monitor
+    diskMonDirs = cooker.configuration.data.getVar("BB_DISKMON_DIRS", 1)
+    if diskMonDirs:
+        monitorProcess = Process(target=monitordisk.monitorDiskSpace, \
+            args=(diskMonDirs, cooker.configuration.data))
+        monitorProcess.start()
+
     # Should no longer need to ever reference cooker
     del cooker
 
@@ -237,9 +246,13 @@ Default BBFILES are the .bb files in the current directory.""")
         finally:
             bb.event.ui_queue = []
             server_connection.terminate()
+            # Stop the disk space monitor
+            if diskMonDirs:
+                monitorProcess.terminate()
     else:
         print("server address: %s, server port: %s" % (server.serverinfo.host, server.serverinfo.port))
 
+
     return 1
 
 if __name__ == "__main__":
diff --git a/bitbake/lib/bb/monitordisk.py b/bitbake/lib/bb/monitordisk.py
new file mode 100644
index 0000000..cb67bc2
--- /dev/null
+++ b/bitbake/lib/bb/monitordisk.py
@@ -0,0 +1,153 @@
+#!/usr/bin/env python
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# Copyright (C) 2011 Robert Yang
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import time, os, logging, re, sys
+import bb
+
+def errExit(info):
+    print("ERROR: %s" % info)
+    print("       Disk space monitor will NOT be enabled")
+    sys.exit(1)
+
+def convertGMK(unit):
+
+    """ Convert the space unit G, M, K, the unit is case-insensitive """
+
+    unitG = re.match('([1-9][0-9]*)[gG]\s?$', unit)
+    if unitG:
+        return int(unitG.group(1)) * (1024 ** 3)
+    unitM = re.match('([1-9][0-9]*)[mM]\s?$', unit)
+    if unitM:
+        return int(unitM.group(1)) * (1024 ** 2)
+    unitK = re.match('([1-9][0-9]*)[kK]\s?$', unit)
+    if unitK:
+        return int(unitK.group(1)) * 1024
+    unitN = re.match('([1-9][0-9]*)\s?$', unit)
+    if unitN:
+        return int(unitN.group(1))
+    else:
+        return None
+
+def convertHMS(unit):
+
+    """ Convert the time unit h, m, s, the unit is case-sensitive """
+
+    unitH = re.match('([1-9][0-9]*)h\s?$', unit)
+    if unitH:
+        return int(unitH.group(1)) * 60 * 60
+    unitM = re.match('([1-9][0-9]*)m\s?$', unit)
+    if unitM:
+        return int(unitM.group(1)) * 60
+    unitS = re.match('([1-9][0-9]*)s?\s?$', unit)
+    if unitS:
+        return int(unitS.group(1))
+    else:
+        return None
+
+def getMountedDevice(path):
+
+    """ Get the device mounted at the path, uses /proc/mounts """
+
+    # Get the mount point of the filesystem containing path
+    # st_dev is the ID of device containing file
+    parentDevice = os.stat(path).st_dev
+    currentDevice = parentDevice
+    # When the current directory's device is different from the
+    # parrent's, then the current directory is a mount point
+    while parentDevice == currentDevice:
+        mountPoint = path
+        # Use dirname to get the parrent's directory
+        path = os.path.dirname(path)
+        # Reach the "/"
+        if path == mountPoint:
+            break
+        parentDevice= os.stat(path).st_dev
+
+    try:
+        with open("/proc/mounts", "r") as ifp:
+            for line in ifp:
+                procLines = line.rstrip('\n').split()
+                if procLines[1] == mountPoint:
+                    return procLines[0]
+    except EnvironmentError:
+        pass
+    return None
+
+def monitorDiskSpace(BBDirs, configuration):
+
+    """Monitor disk space availability, warning when the disk space or
+       the amount of inode is running low"""
+
+    BBSpace = configuration.getVar("BB_DISKMON_MINSPACE", 1)
+    if BBSpace:
+        minSpace = convertGMK(BBSpace)
+        if not minSpace:
+            errExit("Invalid BB_DISKMON_MINSPACE: %s" % BBSpace)
+    else:
+        # The default value is 1GB
+        minSpace = 1024 ** 3
+
+    BBInodes = configuration.getVar("BB_DISKMON_MININODES", 1)
+    if BBInodes:
+        minInodes = convertGMK(BBInodes)
+        if not minInodes:
+            errExit("Invalid BB_DISKMON_MININODES : %s" % BBInodes)
+    else:
+        # The default value is 1K
+        minInodes = 1024
+
+    BBInterval = configuration.getVar("BB_DISKMON_INTERVAL", 1)
+    if BBInterval:
+        # The unit is second, the BB_DISKMON_INTERVAL could be ns or n
+        interval = convertHMS(BBInterval)
+        if not interval:
+            errExit("Invalid BB_DISKMON_INTERVAL: %s" % BBInterval)
+    else:
+        # The default value is 60s
+        interval = 60
+
+    # Save the device IDs, need the ID to be unique (the dictionary's key is
+    # unique), so that when more than one directories are located in the same
+    # device, we just monitor it once
+    deviceDict = {}
+    for path in BBDirs.split():
+        path = os.path.realpath(path)
+        # mkdir for the directory since it may not exist, for example the
+        # DL_DIR may not exist at the very beginning
+        if not os.path.exists(path):
+            bb.utils.mkdirhier(path)
+        mountedDevice = getMountedDevice(path)
+        deviceDict[mountedDevice] = path
+    while True:
+        for mountedDevice in deviceDict:
+            st = os.statvfs(deviceDict[mountedDevice])
+            # The free space, float point number
+            freeSpace = st.f_bavail * st.f_frsize
+            if freeSpace < minSpace:
+                print("WARNING: The free space of %s is running low (%.3fGB left)" % (mountedDevice, freeSpace / 1024 / 1024 / 1024.0))
+
+            # The free inodes, float point number
+            freeInodes = st.f_favail
+            if freeInodes < minInodes:
+                print("WARNING: The free inode of %s is running low (%.3fK left)" % (mountedDevice, freeInodes / 1024.0))
+        try:
+            time.sleep(interval)
+        except (KeyboardInterrupt, SystemExit):
+            sys.exit(1)
+
-- 
1.7.4.1

[PATCH 2/2] V2 Add config sample for disk space monitoring

[Robert Yang]

Add config sample for disk space monitoring to
meta-yocto/conf/local.conf.sample

[YOCTO #1589]
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta-yocto/conf/local.conf.sample |   28 ++++++++++++++++++++++++++++
 1 files changed, 28 insertions(+), 0 deletions(-)

diff --git a/meta-yocto/conf/local.conf.sample b/meta-yocto/conf/local.conf.sample
index 38507e3..c82cb7e 100644
--- a/meta-yocto/conf/local.conf.sample
+++ b/meta-yocto/conf/local.conf.sample
@@ -224,3 +224,31 @@ PATCHRESOLVE = "noop"
 # track the version of this file when it was generated. This can safely be ignored if
 # this doesn't mean anything to you.
 CONF_VERSION = "1"
+
+#
+# Disk space monitor, warning when the disk space or the amount of inode
+# is running low, it is enabled when BB_DISKMON_DIRS is set.
+#
+# Set the directories to monitor for disk usage, if more than one
+# directories are in the same device, then only one directory would be
+# warned since the monitor is based on the device.
+#
+BB_DISKMON_DIRS = "${TMPDIR} ${DL_DIR} ${SSTATE_DIR}"
+#
+# Set the minimum amount of disk space to warn, the unit can be G,
+# M or K, but do not use GB, MB or KB (B is not needed).
+# The default vaule is 1G if not set.
+#
+#BB_DISKMON_MINSPACE = "1G"
+#
+# Set the minimum amount of inodes to warn, the unit can be G, M or K.
+# The default value is 1K if not set.
+#
+#BB_DISKMON_MININODES = "1K"
+#
+# Set the monitor frequency, the unit can be h(hour), m(minute) or
+# s(second), when no unit is specified, the default unit is s(second),
+# for example, "60" means 60 seconds.
+# The default value is 60s if not set.
+#
+#BB_DISKMON_INTERVAL = "60s"
-- 
1.7.4.1

Re: [PATCH 0/2] V2 Disk space monitoring

[Chris Larson]

On Tue, Jan 10, 2012 at 5:45 AM, Robert Yang <liezhi.yang@windriver.com> wrote:
> Here are the test info:
> 1) set the following variables in conf/local.conf for testing:
>   BB_DISKMON_DIRS = "${TMPDIR} ${DL_DIR} ${SSTATE_DIR} /home/lyang1"
>   BB_DISKMON_MINSPACE = "200G"
>   BB_DISKMON_MININODES = "20M"
>   BB_DISKMON_INTERVAL = "30s"
>
>   The TMPDIR, DL_DIR and SSTATE_DIR are in the same disk(/dev/sdb4) in
>   my build, so I added another direcotory /home/lyang1 to test the
>   ability of monitoring multi mount points.
>
> 2) bitbake core-image-sato
>   WARNING: The free space of /dev/sdb5 is running low (25.915GB left)
>   WARNING: The free inode of /dev/sdb5 is running low (1788.202K left)
>   WARNING: The free space of /dev/sdb4 is running low (189.309GB left)
>   WARNING: The free inode of /dev/sdb4 is running low (18005.443K left)
>
> 3) Aalso have tested about the following items:
>   a) Use ctrl-c to stop the monitor
>   b) The monitor should stop when the build stops
>   c) Test the space unit G, B, K(case in-sensitive) and the time unit
>      h, m, s(case-sensitive)

I would think it'd be more useful to be able to specify the thresholds
on a per-mountpoint or per-path basis. The space needed by sstate,
downloads, and the rest of the build can be quite different. Further,
the space requirements for DL_DIR and sstate both are dependent upon
the current contents of the directories, as if we already have all our
downloads, the free space there isn't much of a concern..
-- 
Christopher Larson

Re: [PATCH 1/2] V2 Disk space monitoring

[Richard Purdie]

On Tue, 2012-01-10 at 20:45 +0800, Robert Yang wrote:
> * Problems:
> 
>   1) The disk space monitor is running in a sub process of
>      bin/bitbake, so it can't use the same logger as bitbake(or I don't
>      know how to use it, I've tried to use the logger.warn('some
>      information'), but it would print nothing in the subprocess),
>      another solution maybe use an independent logger in the subprocess,
>      but it seems that it doesn't worth because of the two reasons:
>      a) The logger of bin/bitbake can't control the monitor's
>         independent logger
>      b) The monitor doesn't need print many things
> 
>      It would be great if anyone could give me more instructions.

This is an issue unfortunately since its fine to have the warnings about
low space but its not much use unless bitbake can act on the warnings
somehow (stop executing more tasks for example).

I agree with Chris' comments about being able to specify the warning
threshold per directory being monitored too.

So I think before this can merge its going to need some further work. I
can't give pointers to the right way to send log messages from the
subprocess right at the moment. Perhaps you can insert yourself as an
"idle" task into bitbake's "main loop" instead of running as a
subprocess? This would also make it simpler to stop executing more
tasks. There probably should be a way for the user to choose
"immediately abort, no new tasks, show warning" as the options for
running out of space.

Cheers,

Richard

Re: [PATCH 1/2] V2 Disk space monitoring

[Chris Larson]

On Tue, Jan 10, 2012 at 10:51 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Tue, 2012-01-10 at 20:45 +0800, Robert Yang wrote:
>> * Problems:
>>
>>   1) The disk space monitor is running in a sub process of
>>      bin/bitbake, so it can't use the same logger as bitbake(or I don't
>>      know how to use it, I've tried to use the logger.warn('some
>>      information'), but it would print nothing in the subprocess),
>>      another solution maybe use an independent logger in the subprocess,
>>      but it seems that it doesn't worth because of the two reasons:
>>      a) The logger of bin/bitbake can't control the monitor's
>>         independent logger

If the monitor spawns from the bitbake server process, then it can
send log messages to the standard bitbake UI process, and it wouldn't
need its own log handling.
-- 
Christopher Larson