[PATCH 0/2] Disk space monitoring

[PATCH 0/2] Disk space monitoring

[Robert Yang]

Here are the test info:
1) set the following variables in conf/local.conf for testing:
   BB_DISKMON_DIRS = "${TMPDIR} ${DL_DIR} ${SSTATE_DIR} /home/lyang1"
   BB_DISKMON_MINSPACE = "200G"
   BB_DISKMON_MININODES = "20M"
   BB_DISKMON_INTERVAL = "30s"

   The TMPDIR, DL_DIR and SSTATE_DIR are in the same disk(/dev/sdb4) in
   my build, so I added another direcotory /home/lyang1 to test the
   ability of monitoring multi mount points.

2) bitbake core-image-sato
   WARNING: The free space of /dev/sdb5 is running low (25.915GB left)
   WARNING: The free inode of /dev/sdb5 is running low (1788.202K left)
   WARNING: The free space of /dev/sdb4 is running low (189.309GB left)
   WARNING: The free inode of /dev/sdb4 is running low (18005.443K left)

3) Aalso have tested about the following items:
   a) Use ctrl-c to stop the monitor
   b) The monitor should stop when the build stops
   c) Test the space unit G, B, K(case in-sensitive) and the time unit
      h, m, s(case-sensitive)

// Robert


The following changes since commit 9d136b2db8f906c562cbdb23a9b238f0e237074b:

  gdk-pixbuf: Ensure the binaries can be relocated (2011-12-13 18:00:25 +0000)

are available in the git repository at:
  git://git.pokylinux.org/poky-contrib robert/disk-space-monitor
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/disk-space-monitor

Robert Yang (2):
  Disk space monitoring
  Add config sample for disk space monitoring

 bitbake/bin/bitbake               |   12 +++
 bitbake/lib/bb/monitordisk.py     |  153 +++++++++++++++++++++++++++++++++++++
 meta-yocto/conf/local.conf.sample |   28 +++++++
 3 files changed, 193 insertions(+), 0 deletions(-)
 create mode 100644 bitbake/lib/bb/monitordisk.py

-- 
1.7.4.1

[PATCH 1/2] Disk space monitoring

[Robert Yang]

Monitor disk availability and warn the user when the free disk space or
amount of free inode is running low, it is enabled when BB_DISKMON_DIRS
is set.

* Variable meanings(from meta-yocto/conf/local.conf.sample):

  # Set the directories to monitor for disk usage, if more than one
  # directories are in the same device, then only one directory would be
  # warned since the monitor is based on the device.
  #
  BB_DISKMON_DIRS = "${TMPDIR} ${DL_DIR} ${SSTATE_DIR}"
  #
  # Set the minimum amount of disk space to warn, the unit can be G,
  # M or K, but do not use GB, MB or KB (B is not needed).
  # The default vaule is 1G if not set.
  #
  #BB_DISKMON_MINSPACE = "1G"
  #
  # Set the minimum amount of inodes to warn, the unit can be G, M or K.
  # The default value is 1K if not set.
  #
  #BB_DISKMON_MININODES = "1K"
  #
  # Set the monitor frequency, the unit can be h(hour), m(minute) or
  # s(second), when no unit is specified, the default unit is s(second),
  # for example, "60" means 60 seconds.
  # The default value is 60s if not set.
  #
  #BB_DISKMON_INTERVAL = "60s"

* Problems:

  1) The disk space monitor is running in a sub process of
     bin/bitbake, so it can't use the same logger as bitbake(or I don't
     know how to use it, I've tried to use the logger.warn('some
     information'), but it would print nothing in the subprocess),
     another solution maybe use an independent logger in the subprocess,
     but it seems that it doesn't worth because of the two reasons:
     a) The logger of bin/bitbake can't control the monitor's
        independent logger
     b) The monitor doesn't need print many things

     It would be great if anyone could give me more instructions.

 2) The code is placed in the new file lib/bb/monitordisk.py, since it
    seems that it is more independent.

[YOCTO #1589]
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/bin/bitbake           |   12 +++
 bitbake/lib/bb/monitordisk.py |  153 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 165 insertions(+), 0 deletions(-)
 create mode 100644 bitbake/lib/bb/monitordisk.py

diff --git a/bitbake/bin/bitbake b/bitbake/bin/bitbake
index 9eb558b..5761376 100755
--- a/bitbake/bin/bitbake
+++ b/bitbake/bin/bitbake
@@ -39,6 +39,8 @@ import bb.msg
 from bb import cooker
 from bb import ui
 from bb import server
+from bb import monitordisk
+from multiprocessing import Process
 
 __version__ = "1.15.0"
 logger = logging.getLogger("BitBake")
@@ -217,6 +219,13 @@ Default BBFILES are the .bb files in the current directory.""")
     server.saveConnectionDetails()
     server.detach(cooker_logfile)
 
+    # For disk space monitor
+    diskMonDirs = cooker.configuration.data.getVar("BB_DISKMON_DIRS", 1)
+    if diskMonDirs:
+        monitorProcess = Process(target=monitordisk.monitorDiskSpace, \
+            args=(diskMonDirs, cooker.configuration.data))
+        monitorProcess.start()
+
     # Should no longer need to ever reference cooker
     del cooker
 
@@ -230,6 +239,9 @@ Default BBFILES are the .bb files in the current directory.""")
     finally:
         bb.event.ui_queue = []
         server_connection.terminate()
+        # Stop the disk space monitor
+        if diskMonDirs:
+            monitorProcess.terminate()
 
     return 1
 
diff --git a/bitbake/lib/bb/monitordisk.py b/bitbake/lib/bb/monitordisk.py
new file mode 100644
index 0000000..cb67bc2
--- /dev/null
+++ b/bitbake/lib/bb/monitordisk.py
@@ -0,0 +1,153 @@
+#!/usr/bin/env python
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# Copyright (C) 2011 Robert Yang
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import time, os, logging, re, sys
+import bb
+
+def errExit(info):
+    print("ERROR: %s" % info)
+    print("       Disk space monitor will NOT be enabled")
+    sys.exit(1)
+
+def convertGMK(unit):
+
+    """ Convert the space unit G, M, K, the unit is case-insensitive """
+
+    unitG = re.match('([1-9][0-9]*)[gG]\s?$', unit)
+    if unitG:
+        return int(unitG.group(1)) * (1024 ** 3)
+    unitM = re.match('([1-9][0-9]*)[mM]\s?$', unit)
+    if unitM:
+        return int(unitM.group(1)) * (1024 ** 2)
+    unitK = re.match('([1-9][0-9]*)[kK]\s?$', unit)
+    if unitK:
+        return int(unitK.group(1)) * 1024
+    unitN = re.match('([1-9][0-9]*)\s?$', unit)
+    if unitN:
+        return int(unitN.group(1))
+    else:
+        return None
+
+def convertHMS(unit):
+
+    """ Convert the time unit h, m, s, the unit is case-sensitive """
+
+    unitH = re.match('([1-9][0-9]*)h\s?$', unit)
+    if unitH:
+        return int(unitH.group(1)) * 60 * 60
+    unitM = re.match('([1-9][0-9]*)m\s?$', unit)
+    if unitM:
+        return int(unitM.group(1)) * 60
+    unitS = re.match('([1-9][0-9]*)s?\s?$', unit)
+    if unitS:
+        return int(unitS.group(1))
+    else:
+        return None
+
+def getMountedDevice(path):
+
+    """ Get the device mounted at the path, uses /proc/mounts """
+
+    # Get the mount point of the filesystem containing path
+    # st_dev is the ID of device containing file
+    parentDevice = os.stat(path).st_dev
+    currentDevice = parentDevice
+    # When the current directory's device is different from the
+    # parrent's, then the current directory is a mount point
+    while parentDevice == currentDevice:
+        mountPoint = path
+        # Use dirname to get the parrent's directory
+        path = os.path.dirname(path)
+        # Reach the "/"
+        if path == mountPoint:
+            break
+        parentDevice= os.stat(path).st_dev
+
+    try:
+        with open("/proc/mounts", "r") as ifp:
+            for line in ifp:
+                procLines = line.rstrip('\n').split()
+                if procLines[1] == mountPoint:
+                    return procLines[0]
+    except EnvironmentError:
+        pass
+    return None
+
+def monitorDiskSpace(BBDirs, configuration):
+
+    """Monitor disk space availability, warning when the disk space or
+       the amount of inode is running low"""
+
+    BBSpace = configuration.getVar("BB_DISKMON_MINSPACE", 1)
+    if BBSpace:
+        minSpace = convertGMK(BBSpace)
+        if not minSpace:
+            errExit("Invalid BB_DISKMON_MINSPACE: %s" % BBSpace)
+    else:
+        # The default value is 1GB
+        minSpace = 1024 ** 3
+
+    BBInodes = configuration.getVar("BB_DISKMON_MININODES", 1)
+    if BBInodes:
+        minInodes = convertGMK(BBInodes)
+        if not minInodes:
+            errExit("Invalid BB_DISKMON_MININODES : %s" % BBInodes)
+    else:
+        # The default value is 1K
+        minInodes = 1024
+
+    BBInterval = configuration.getVar("BB_DISKMON_INTERVAL", 1)
+    if BBInterval:
+        # The unit is second, the BB_DISKMON_INTERVAL could be ns or n
+        interval = convertHMS(BBInterval)
+        if not interval:
+            errExit("Invalid BB_DISKMON_INTERVAL: %s" % BBInterval)
+    else:
+        # The default value is 60s
+        interval = 60
+
+    # Save the device IDs, need the ID to be unique (the dictionary's key is
+    # unique), so that when more than one directories are located in the same
+    # device, we just monitor it once
+    deviceDict = {}
+    for path in BBDirs.split():
+        path = os.path.realpath(path)
+        # mkdir for the directory since it may not exist, for example the
+        # DL_DIR may not exist at the very beginning
+        if not os.path.exists(path):
+            bb.utils.mkdirhier(path)
+        mountedDevice = getMountedDevice(path)
+        deviceDict[mountedDevice] = path
+    while True:
+        for mountedDevice in deviceDict:
+            st = os.statvfs(deviceDict[mountedDevice])
+            # The free space, float point number
+            freeSpace = st.f_bavail * st.f_frsize
+            if freeSpace < minSpace:
+                print("WARNING: The free space of %s is running low (%.3fGB left)" % (mountedDevice, freeSpace / 1024 / 1024 / 1024.0))
+
+            # The free inodes, float point number
+            freeInodes = st.f_favail
+            if freeInodes < minInodes:
+                print("WARNING: The free inode of %s is running low (%.3fK left)" % (mountedDevice, freeInodes / 1024.0))
+        try:
+            time.sleep(interval)
+        except (KeyboardInterrupt, SystemExit):
+            sys.exit(1)
+
-- 
1.7.4.1

[PATCH 2/2] Add config sample for disk space monitoring

[Robert Yang]

Add config sample for disk space monitoring to
meta-yocto/conf/local.conf.sample

[YOCTO #1589]
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta-yocto/conf/local.conf.sample |   28 ++++++++++++++++++++++++++++
 1 files changed, 28 insertions(+), 0 deletions(-)

diff --git a/meta-yocto/conf/local.conf.sample b/meta-yocto/conf/local.conf.sample
index ca8636c..d863774 100644
--- a/meta-yocto/conf/local.conf.sample
+++ b/meta-yocto/conf/local.conf.sample
@@ -222,3 +222,31 @@ PATCHRESOLVE = "noop"
 # track the version of this file when it was generated. This can safely be ignored if
 # this doesn't mean anything to you.
 CONF_VERSION = "1"
+
+#
+# Disk space monitor, warning when the disk space or the amount of inode
+# is running low, it is enabled when BB_DISKMON_DIRS is set.
+#
+# Set the directories to monitor for disk usage, if more than one
+# directories are in the same device, then only one directory would be
+# warned since the monitor is based on the device.
+#
+BB_DISKMON_DIRS = "${TMPDIR} ${DL_DIR} ${SSTATE_DIR}"
+#
+# Set the minimum amount of disk space to warn, the unit can be G,
+# M or K, but do not use GB, MB or KB (B is not needed).
+# The default vaule is 1G if not set.
+#
+#BB_DISKMON_MINSPACE = "1G"
+#
+# Set the minimum amount of inodes to warn, the unit can be G, M or K.
+# The default value is 1K if not set.
+#
+#BB_DISKMON_MININODES = "1K"
+#
+# Set the monitor frequency, the unit can be h(hour), m(minute) or
+# s(second), when no unit is specified, the default unit is s(second),
+# for example, "60" means 60 seconds.
+# The default value is 60s if not set.
+#
+#BB_DISKMON_INTERVAL = "60s"
-- 
1.7.4.1

Re: [PATCH 0/2] Disk space monitoring

[Robert Yang]

On 12/15/2011 06:44 PM, Robert Yang wrote:
> Here are the test info:
> 1) set the following variables in conf/local.conf for testing:
>     BB_DISKMON_DIRS = "${TMPDIR} ${DL_DIR} ${SSTATE_DIR} /home/lyang1"
>     BB_DISKMON_MINSPACE = "200G"
>     BB_DISKMON_MININODES = "20M"
>     BB_DISKMON_INTERVAL = "30s"
>
>     The TMPDIR, DL_DIR and SSTATE_DIR are in the same disk(/dev/sdb4) in
>     my build, so I added another direcotory /home/lyang1 to test the
>     ability of monitoring multi mount points.
>
> 2) bitbake core-image-sato
>     WARNING: The free space of /dev/sdb5 is running low (25.915GB left)
>     WARNING: The free inode of /dev/sdb5 is running low (1788.202K left)
>     WARNING: The free space of /dev/sdb4 is running low (189.309GB left)
>     WARNING: The free inode of /dev/sdb4 is running low (18005.443K left)
>

And the messages will print every 30 seconds(the BB_DISKMON_INTERVAL), until
the build stops.

// Robert

> 3) Aalso have tested about the following items:
>     a) Use ctrl-c to stop the monitor
>     b) The monitor should stop when the build stops
>     c) Test the space unit G, B, K(case in-sensitive) and the time unit
>        h, m, s(case-sensitive)
>
> // Robert
>
>
> The following changes since commit 9d136b2db8f906c562cbdb23a9b238f0e237074b:
>
>    gdk-pixbuf: Ensure the binaries can be relocated (2011-12-13 18:00:25 +0000)
>
> are available in the git repository at:
>    git://git.pokylinux.org/poky-contrib robert/disk-space-monitor
>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/disk-space-monitor
>
> Robert Yang (2):
>    Disk space monitoring
>    Add config sample for disk space monitoring
>
>   bitbake/bin/bitbake               |   12 +++
>   bitbake/lib/bb/monitordisk.py     |  153 +++++++++++++++++++++++++++++++++++++
>   meta-yocto/conf/local.conf.sample |   28 +++++++
>   3 files changed, 193 insertions(+), 0 deletions(-)
>   create mode 100644 bitbake/lib/bb/monitordisk.py
>

Re: [PATCH 0/2] Disk space monitoring

[Mark Hatle]

Is the disk space monitoring implemented in the correct place in the system?

I would have expected this to have been implemented in the meta/class files, 
both in the initial sanity checking and also in something that can watch the 
events and doing the repetition/rechecking as necessary.

(BTW the code looks fine to me as implemented, I'm just not sure if it belongs 
in bitbake itself.)

--Mark

On 12/15/11 4:44 AM, Robert Yang wrote:
> Here are the test info:
> 1) set the following variables in conf/local.conf for testing:
>     BB_DISKMON_DIRS = "${TMPDIR} ${DL_DIR} ${SSTATE_DIR} /home/lyang1"
>     BB_DISKMON_MINSPACE = "200G"
>     BB_DISKMON_MININODES = "20M"
>     BB_DISKMON_INTERVAL = "30s"
>
>     The TMPDIR, DL_DIR and SSTATE_DIR are in the same disk(/dev/sdb4) in
>     my build, so I added another direcotory /home/lyang1 to test the
>     ability of monitoring multi mount points.
>
> 2) bitbake core-image-sato
>     WARNING: The free space of /dev/sdb5 is running low (25.915GB left)
>     WARNING: The free inode of /dev/sdb5 is running low (1788.202K left)
>     WARNING: The free space of /dev/sdb4 is running low (189.309GB left)
>     WARNING: The free inode of /dev/sdb4 is running low (18005.443K left)
>
> 3) Aalso have tested about the following items:
>     a) Use ctrl-c to stop the monitor
>     b) The monitor should stop when the build stops
>     c) Test the space unit G, B, K(case in-sensitive) and the time unit
>        h, m, s(case-sensitive)
>
> // Robert
>
>
> The following changes since commit 9d136b2db8f906c562cbdb23a9b238f0e237074b:
>
>    gdk-pixbuf: Ensure the binaries can be relocated (2011-12-13 18:00:25 +0000)
>
> are available in the git repository at:
>    git://git.pokylinux.org/poky-contrib robert/disk-space-monitor
>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/disk-space-monitor
>
> Robert Yang (2):
>    Disk space monitoring
>    Add config sample for disk space monitoring
>
>   bitbake/bin/bitbake               |   12 +++
>   bitbake/lib/bb/monitordisk.py     |  153 +++++++++++++++++++++++++++++++++++++
>   meta-yocto/conf/local.conf.sample |   28 +++++++
>   3 files changed, 193 insertions(+), 0 deletions(-)
>   create mode 100644 bitbake/lib/bb/monitordisk.py
>

Re: [PATCH 0/2] Disk space monitoring

[Robert Yang]

Hi Mark,

Thanks for your replying, Richard suggested we add this to bitbake itself,
here is his reply in 8 months ago:

As for disk monitoring, I'd suggest we just add this to bitbake itself
as it doesn't really fit the task model. It can be controlled via some
variables like:

# Set the directories to monitor for disk usage
BB_DISKMON_DIRS = "${TMPDIR} ${DL_DIR}"
# Set the minimum amount of disk space to exit on
BB_DISKMON_MINSPACE = "1GB"
# Set the monitor frequency
BB_DISKMON_INTERVAL = "60s"

and if the variables are not set, it doesn't get activated.


// Robert


On 12/15/2011 11:59 PM, Mark Hatle wrote:
> Is the disk space monitoring implemented in the correct place in the system?
>
> I would have expected this to have been implemented in the meta/class files,
> both in the initial sanity checking and also in something that can watch the
> events and doing the repetition/rechecking as necessary.
>
> (BTW the code looks fine to me as implemented, I'm just not sure if it belongs
> in bitbake itself.)
>
> --Mark
>
> On 12/15/11 4:44 AM, Robert Yang wrote:
>> Here are the test info:
>> 1) set the following variables in conf/local.conf for testing:
>> BB_DISKMON_DIRS = "${TMPDIR} ${DL_DIR} ${SSTATE_DIR} /home/lyang1"
>> BB_DISKMON_MINSPACE = "200G"
>> BB_DISKMON_MININODES = "20M"
>> BB_DISKMON_INTERVAL = "30s"
>>
>> The TMPDIR, DL_DIR and SSTATE_DIR are in the same disk(/dev/sdb4) in
>> my build, so I added another direcotory /home/lyang1 to test the
>> ability of monitoring multi mount points.
>>
>> 2) bitbake core-image-sato
>> WARNING: The free space of /dev/sdb5 is running low (25.915GB left)
>> WARNING: The free inode of /dev/sdb5 is running low (1788.202K left)
>> WARNING: The free space of /dev/sdb4 is running low (189.309GB left)
>> WARNING: The free inode of /dev/sdb4 is running low (18005.443K left)
>>
>> 3) Aalso have tested about the following items:
>> a) Use ctrl-c to stop the monitor
>> b) The monitor should stop when the build stops
>> c) Test the space unit G, B, K(case in-sensitive) and the time unit
>> h, m, s(case-sensitive)
>>
>> // Robert
>>
>>
>> The following changes since commit 9d136b2db8f906c562cbdb23a9b238f0e237074b:
>>
>> gdk-pixbuf: Ensure the binaries can be relocated (2011-12-13 18:00:25 +0000)
>>
>> are available in the git repository at:
>> git://git.pokylinux.org/poky-contrib robert/disk-space-monitor
>> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/disk-space-monitor
>>
>> Robert Yang (2):
>> Disk space monitoring
>> Add config sample for disk space monitoring
>>
>> bitbake/bin/bitbake | 12 +++
>> bitbake/lib/bb/monitordisk.py | 153 +++++++++++++++++++++++++++++++++++++
>> meta-yocto/conf/local.conf.sample | 28 +++++++
>> 3 files changed, 193 insertions(+), 0 deletions(-)
>> create mode 100644 bitbake/lib/bb/monitordisk.py
>>
>
>
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
>