* [Webhob] [PATCH 1/3] Rename main WebHob project
@ 2013-06-12 9:30 Calin Dragomir
2013-06-12 9:30 ` [Webhob] [PATCH 2/3] Create Tasks Django model Calin Dragomir
2013-06-12 9:30 ` [Webhob] [PATCH 3/3] Create Data Store Interface (DSI) file Calin Dragomir
0 siblings, 2 replies; 6+ messages in thread
From: Calin Dragomir @ 2013-06-12 9:30 UTC (permalink / raw)
To: webhob
Renames main webhob project to whbmain in order to avoid import issues
---
bitbake/lib/webhob/__init__.py | 0
bitbake/lib/webhob/manage.py | 2 +-
bitbake/lib/webhob/webhob/__init__.py | 0
bitbake/lib/webhob/webhob/settings.py | 156 ---------------------------------
bitbake/lib/webhob/webhob/urls.py | 17 ----
bitbake/lib/webhob/webhob/wsgi.py | 32 -------
bitbake/lib/webhob/whbmain/__init__.py | 0
bitbake/lib/webhob/whbmain/settings.py | 156 +++++++++++++++++++++++++++++++++
bitbake/lib/webhob/whbmain/urls.py | 17 ++++
bitbake/lib/webhob/whbmain/wsgi.py | 32 +++++++
10 files changed, 206 insertions(+), 206 deletions(-)
create mode 100644 bitbake/lib/webhob/__init__.py
delete mode 100644 bitbake/lib/webhob/webhob/__init__.py
delete mode 100644 bitbake/lib/webhob/webhob/settings.py
delete mode 100644 bitbake/lib/webhob/webhob/urls.py
delete mode 100644 bitbake/lib/webhob/webhob/wsgi.py
create mode 100644 bitbake/lib/webhob/whbmain/__init__.py
create mode 100644 bitbake/lib/webhob/whbmain/settings.py
create mode 100644 bitbake/lib/webhob/whbmain/urls.py
create mode 100644 bitbake/lib/webhob/whbmain/wsgi.py
diff --git a/bitbake/lib/webhob/__init__.py b/bitbake/lib/webhob/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/bitbake/lib/webhob/manage.py b/bitbake/lib/webhob/manage.py
index 5a209bc..1b2c1a7 100755
--- a/bitbake/lib/webhob/manage.py
+++ b/bitbake/lib/webhob/manage.py
@@ -3,7 +3,7 @@ import os
import sys
if __name__ == "__main__":
- os.environ.setdefault("DJANGO_SETTINGS_MODULE", "webhob.settings")
+ os.environ.setdefault("DJANGO_SETTINGS_MODULE", "whbmain.settings")
from django.core.management import execute_from_command_line
diff --git a/bitbake/lib/webhob/webhob/__init__.py b/bitbake/lib/webhob/webhob/__init__.py
deleted file mode 100644
index e69de29..0000000
diff --git a/bitbake/lib/webhob/webhob/settings.py b/bitbake/lib/webhob/webhob/settings.py
deleted file mode 100644
index 1556d7b..0000000
--- a/bitbake/lib/webhob/webhob/settings.py
+++ /dev/null
@@ -1,156 +0,0 @@
-# Django settings for webhob project.
-
-DEBUG = True
-TEMPLATE_DEBUG = DEBUG
-
-ADMINS = (
- # ('Your Name', 'your_email@example.com'),
-)
-
-MANAGERS = ADMINS
-
-DATABASES = {
- 'default': {
- 'ENGINE': 'django.db.backends.', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
- 'NAME': '', # Or path to database file if using sqlite3.
- # The following settings are not used with sqlite3:
- 'USER': '',
- 'PASSWORD': '',
- 'HOST': '', # Empty for localhost through domain sockets or '127.0.0.1' for localhost through TCP.
- 'PORT': '', # Set to empty string for default.
- }
-}
-
-# Hosts/domain names that are valid for this site; required if DEBUG is False
-# See https://docs.djangoproject.com/en/1.5/ref/settings/#allowed-hosts
-ALLOWED_HOSTS = []
-
-# Local time zone for this installation. Choices can be found here:
-# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
-# although not all choices may be available on all operating systems.
-# In a Windows environment this must be set to your system time zone.
-TIME_ZONE = 'America/Chicago'
-
-# Language code for this installation. All choices can be found here:
-# http://www.i18nguy.com/unicode/language-identifiers.html
-LANGUAGE_CODE = 'en-us'
-
-SITE_ID = 1
-
-# If you set this to False, Django will make some optimizations so as not
-# to load the internationalization machinery.
-USE_I18N = True
-
-# If you set this to False, Django will not format dates, numbers and
-# calendars according to the current locale.
-USE_L10N = True
-
-# If you set this to False, Django will not use timezone-aware datetimes.
-USE_TZ = True
-
-# Absolute filesystem path to the directory that will hold user-uploaded files.
-# Example: "/var/www/example.com/media/"
-MEDIA_ROOT = ''
-
-# URL that handles the media served from MEDIA_ROOT. Make sure to use a
-# trailing slash.
-# Examples: "http://example.com/media/", "http://media.example.com/"
-MEDIA_URL = ''
-
-# Absolute path to the directory static files should be collected to.
-# Don't put anything in this directory yourself; store your static files
-# in apps' "static/" subdirectories and in STATICFILES_DIRS.
-# Example: "/var/www/example.com/static/"
-STATIC_ROOT = ''
-
-# URL prefix for static files.
-# Example: "http://example.com/static/", "http://static.example.com/"
-STATIC_URL = '/static/'
-
-# Additional locations of static files
-STATICFILES_DIRS = (
- # Put strings here, like "/home/html/static" or "C:/www/django/static".
- # Always use forward slashes, even on Windows.
- # Don't forget to use absolute paths, not relative paths.
-)
-
-# List of finder classes that know how to find static files in
-# various locations.
-STATICFILES_FINDERS = (
- 'django.contrib.staticfiles.finders.FileSystemFinder',
- 'django.contrib.staticfiles.finders.AppDirectoriesFinder',
-# 'django.contrib.staticfiles.finders.DefaultStorageFinder',
-)
-
-# Make this unique, and don't share it with anybody.
-SECRET_KEY = 'fy@5+5w9zn5wa0vsegn1c8p_6w&pa_ms-4t0cna1vle79nzlkn'
-
-# List of callables that know how to import templates from various sources.
-TEMPLATE_LOADERS = (
- 'django.template.loaders.filesystem.Loader',
- 'django.template.loaders.app_directories.Loader',
-# 'django.template.loaders.eggs.Loader',
-)
-
-MIDDLEWARE_CLASSES = (
- 'django.middleware.common.CommonMiddleware',
- 'django.contrib.sessions.middleware.SessionMiddleware',
- 'django.middleware.csrf.CsrfViewMiddleware',
- 'django.contrib.auth.middleware.AuthenticationMiddleware',
- 'django.contrib.messages.middleware.MessageMiddleware',
- # Uncomment the next line for simple clickjacking protection:
- # 'django.middleware.clickjacking.XFrameOptionsMiddleware',
-)
-
-ROOT_URLCONF = 'webhob.urls'
-
-# Python dotted path to the WSGI application used by Django's runserver.
-WSGI_APPLICATION = 'webhob.wsgi.application'
-
-TEMPLATE_DIRS = (
- # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
- # Always use forward slashes, even on Windows.
- # Don't forget to use absolute paths, not relative paths.
-)
-
-INSTALLED_APPS = (
- 'django.contrib.auth',
- 'django.contrib.contenttypes',
- 'django.contrib.sessions',
- 'django.contrib.sites',
- 'django.contrib.messages',
- 'django.contrib.staticfiles',
- # Uncomment the next line to enable the admin:
- # 'django.contrib.admin',
- # Uncomment the next line to enable admin documentation:
- # 'django.contrib.admindocs',
-)
-
-# A sample logging configuration. The only tangible logging
-# performed by this configuration is to send an email to
-# the site admins on every HTTP 500 error when DEBUG=False.
-# See http://docs.djangoproject.com/en/dev/topics/logging for
-# more details on how to customize your logging configuration.
-LOGGING = {
- 'version': 1,
- 'disable_existing_loggers': False,
- 'filters': {
- 'require_debug_false': {
- '()': 'django.utils.log.RequireDebugFalse'
- }
- },
- 'handlers': {
- 'mail_admins': {
- 'level': 'ERROR',
- 'filters': ['require_debug_false'],
- 'class': 'django.utils.log.AdminEmailHandler'
- }
- },
- 'loggers': {
- 'django.request': {
- 'handlers': ['mail_admins'],
- 'level': 'ERROR',
- 'propagate': True,
- },
- }
-}
diff --git a/bitbake/lib/webhob/webhob/urls.py b/bitbake/lib/webhob/webhob/urls.py
deleted file mode 100644
index 04e0352..0000000
--- a/bitbake/lib/webhob/webhob/urls.py
+++ /dev/null
@@ -1,17 +0,0 @@
-from django.conf.urls import patterns, include, url
-
-# Uncomment the next two lines to enable the admin:
-# from django.contrib import admin
-# admin.autodiscover()
-
-urlpatterns = patterns('',
- # Examples:
- # url(r'^$', 'webhob.views.home', name='home'),
- # url(r'^webhob/', include('webhob.foo.urls')),
-
- # Uncomment the admin/doc line below to enable admin documentation:
- # url(r'^admin/doc/', include('django.contrib.admindocs.urls')),
-
- # Uncomment the next line to enable the admin:
- # url(r'^admin/', include(admin.site.urls)),
-)
diff --git a/bitbake/lib/webhob/webhob/wsgi.py b/bitbake/lib/webhob/webhob/wsgi.py
deleted file mode 100644
index 1b64e1b..0000000
--- a/bitbake/lib/webhob/webhob/wsgi.py
+++ /dev/null
@@ -1,32 +0,0 @@
-"""
-WSGI config for webhob project.
-
-This module contains the WSGI application used by Django's development server
-and any production WSGI deployments. It should expose a module-level variable
-named ``application``. Django's ``runserver`` and ``runfcgi`` commands discover
-this application via the ``WSGI_APPLICATION`` setting.
-
-Usually you will have the standard Django WSGI application here, but it also
-might make sense to replace the whole Django WSGI application with a custom one
-that later delegates to the Django one. For example, you could introduce WSGI
-middleware here, or combine a Django application with an application of another
-framework.
-
-"""
-import os
-
-# We defer to a DJANGO_SETTINGS_MODULE already in the environment. This breaks
-# if running multiple sites in the same mod_wsgi process. To fix this, use
-# mod_wsgi daemon mode with each site in its own daemon process, or use
-# os.environ["DJANGO_SETTINGS_MODULE"] = "webhob.settings"
-os.environ.setdefault("DJANGO_SETTINGS_MODULE", "webhob.settings")
-
-# This application object is used by any WSGI server configured to use this
-# file. This includes Django's development server, if the WSGI_APPLICATION
-# setting points here.
-from django.core.wsgi import get_wsgi_application
-application = get_wsgi_application()
-
-# Apply WSGI middleware here.
-# from helloworld.wsgi import HelloWorldApplication
-# application = HelloWorldApplication(application)
diff --git a/bitbake/lib/webhob/whbmain/__init__.py b/bitbake/lib/webhob/whbmain/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/bitbake/lib/webhob/whbmain/settings.py b/bitbake/lib/webhob/whbmain/settings.py
new file mode 100644
index 0000000..bc59543
--- /dev/null
+++ b/bitbake/lib/webhob/whbmain/settings.py
@@ -0,0 +1,156 @@
+# Django settings for webhob project.
+
+DEBUG = True
+TEMPLATE_DEBUG = DEBUG
+
+ADMINS = (
+ # ('Your Name', 'your_email@example.com'),
+)
+
+MANAGERS = ADMINS
+
+DATABASES = {
+ 'default': {
+ 'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
+ 'NAME': 'wbhob', # Or path to database file if using sqlite3.
+ # The following settings are not used with sqlite3:
+ 'USER': 'root',
+ 'PASSWORD': '',
+ 'HOST': '127.0.0.1', # Empty for localhost through domain sockets or '127.0.0.1' for localhost through TCP.
+ 'PORT': '3306', # Set to empty string for default.
+ }
+}
+
+# Hosts/domain names that are valid for this site; required if DEBUG is False
+# See https://docs.djangoproject.com/en/1.5/ref/settings/#allowed-hosts
+ALLOWED_HOSTS = []
+
+# Local time zone for this installation. Choices can be found here:
+# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
+# although not all choices may be available on all operating systems.
+# In a Windows environment this must be set to your system time zone.
+TIME_ZONE = 'America/Chicago'
+
+# Language code for this installation. All choices can be found here:
+# http://www.i18nguy.com/unicode/language-identifiers.html
+LANGUAGE_CODE = 'en-us'
+
+SITE_ID = 1
+
+# If you set this to False, Django will make some optimizations so as not
+# to load the internationalization machinery.
+USE_I18N = True
+
+# If you set this to False, Django will not format dates, numbers and
+# calendars according to the current locale.
+USE_L10N = True
+
+# If you set this to False, Django will not use timezone-aware datetimes.
+USE_TZ = True
+
+# Absolute filesystem path to the directory that will hold user-uploaded files.
+# Example: "/var/www/example.com/media/"
+MEDIA_ROOT = ''
+
+# URL that handles the media served from MEDIA_ROOT. Make sure to use a
+# trailing slash.
+# Examples: "http://example.com/media/", "http://media.example.com/"
+MEDIA_URL = ''
+
+# Absolute path to the directory static files should be collected to.
+# Don't put anything in this directory yourself; store your static files
+# in apps' "static/" subdirectories and in STATICFILES_DIRS.
+# Example: "/var/www/example.com/static/"
+STATIC_ROOT = ''
+
+# URL prefix for static files.
+# Example: "http://example.com/static/", "http://static.example.com/"
+STATIC_URL = '/static/'
+
+# Additional locations of static files
+STATICFILES_DIRS = (
+ # Put strings here, like "/home/html/static" or "C:/www/django/static".
+ # Always use forward slashes, even on Windows.
+ # Don't forget to use absolute paths, not relative paths.
+)
+
+# List of finder classes that know how to find static files in
+# various locations.
+STATICFILES_FINDERS = (
+ 'django.contrib.staticfiles.finders.FileSystemFinder',
+ 'django.contrib.staticfiles.finders.AppDirectoriesFinder',
+# 'django.contrib.staticfiles.finders.DefaultStorageFinder',
+)
+
+# Make this unique, and don't share it with anybody.
+SECRET_KEY = 'fy@5+5w9zn5wa0vsegn1c8p_6w&pa_ms-4t0cna1vle79nzlkn'
+
+# List of callables that know how to import templates from various sources.
+TEMPLATE_LOADERS = (
+ 'django.template.loaders.filesystem.Loader',
+ 'django.template.loaders.app_directories.Loader',
+# 'django.template.loaders.eggs.Loader',
+)
+
+MIDDLEWARE_CLASSES = (
+ 'django.middleware.common.CommonMiddleware',
+ 'django.contrib.sessions.middleware.SessionMiddleware',
+ 'django.middleware.csrf.CsrfViewMiddleware',
+ 'django.contrib.auth.middleware.AuthenticationMiddleware',
+ 'django.contrib.messages.middleware.MessageMiddleware',
+ # Uncomment the next line for simple clickjacking protection:
+ # 'django.middleware.clickjacking.XFrameOptionsMiddleware',
+)
+
+ROOT_URLCONF = 'whbmain.urls'
+
+# Python dotted path to the WSGI application used by Django's runserver.
+WSGI_APPLICATION = 'whbmain.wsgi.application'
+
+TEMPLATE_DIRS = (
+ # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
+ # Always use forward slashes, even on Windows.
+ # Don't forget to use absolute paths, not relative paths.
+)
+
+INSTALLED_APPS = (
+ #'django.contrib.auth',
+ #'django.contrib.contenttypes',
+ #'django.contrib.sessions',
+ #'django.contrib.sites',
+ #'django.contrib.messages',
+ #'django.contrib.staticfiles',
+ # Uncomment the next line to enable the admin:
+ # 'django.contrib.admin',
+ # Uncomment the next line to enable admin documentation:
+ # 'django.contrib.admindocs',
+)
+
+# A sample logging configuration. The only tangible logging
+# performed by this configuration is to send an email to
+# the site admins on every HTTP 500 error when DEBUG=False.
+# See http://docs.djangoproject.com/en/dev/topics/logging for
+# more details on how to customize your logging configuration.
+LOGGING = {
+ 'version': 1,
+ 'disable_existing_loggers': False,
+ 'filters': {
+ 'require_debug_false': {
+ '()': 'django.utils.log.RequireDebugFalse'
+ }
+ },
+ 'handlers': {
+ 'mail_admins': {
+ 'level': 'ERROR',
+ 'filters': ['require_debug_false'],
+ 'class': 'django.utils.log.AdminEmailHandler'
+ }
+ },
+ 'loggers': {
+ 'django.request': {
+ 'handlers': ['mail_admins'],
+ 'level': 'ERROR',
+ 'propagate': True,
+ },
+ }
+}
diff --git a/bitbake/lib/webhob/whbmain/urls.py b/bitbake/lib/webhob/whbmain/urls.py
new file mode 100644
index 0000000..04e0352
--- /dev/null
+++ b/bitbake/lib/webhob/whbmain/urls.py
@@ -0,0 +1,17 @@
+from django.conf.urls import patterns, include, url
+
+# Uncomment the next two lines to enable the admin:
+# from django.contrib import admin
+# admin.autodiscover()
+
+urlpatterns = patterns('',
+ # Examples:
+ # url(r'^$', 'webhob.views.home', name='home'),
+ # url(r'^webhob/', include('webhob.foo.urls')),
+
+ # Uncomment the admin/doc line below to enable admin documentation:
+ # url(r'^admin/doc/', include('django.contrib.admindocs.urls')),
+
+ # Uncomment the next line to enable the admin:
+ # url(r'^admin/', include(admin.site.urls)),
+)
diff --git a/bitbake/lib/webhob/whbmain/wsgi.py b/bitbake/lib/webhob/whbmain/wsgi.py
new file mode 100644
index 0000000..9595eda
--- /dev/null
+++ b/bitbake/lib/webhob/whbmain/wsgi.py
@@ -0,0 +1,32 @@
+"""
+WSGI config for webhob project.
+
+This module contains the WSGI application used by Django's development server
+and any production WSGI deployments. It should expose a module-level variable
+named ``application``. Django's ``runserver`` and ``runfcgi`` commands discover
+this application via the ``WSGI_APPLICATION`` setting.
+
+Usually you will have the standard Django WSGI application here, but it also
+might make sense to replace the whole Django WSGI application with a custom one
+that later delegates to the Django one. For example, you could introduce WSGI
+middleware here, or combine a Django application with an application of another
+framework.
+
+"""
+import os
+
+# We defer to a DJANGO_SETTINGS_MODULE already in the environment. This breaks
+# if running multiple sites in the same mod_wsgi process. To fix this, use
+# mod_wsgi daemon mode with each site in its own daemon process, or use
+# os.environ["DJANGO_SETTINGS_MODULE"] = "webhob.settings"
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "wbhmain.settings")
+
+# This application object is used by any WSGI server configured to use this
+# file. This includes Django's development server, if the WSGI_APPLICATION
+# setting points here.
+from django.core.wsgi import get_wsgi_application
+application = get_wsgi_application()
+
+# Apply WSGI middleware here.
+# from helloworld.wsgi import HelloWorldApplication
+# application = HelloWorldApplication(application)
--
1.8.1.2
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [Webhob] [PATCH 2/3] Create Tasks Django model
2013-06-12 9:30 [Webhob] [PATCH 1/3] Rename main WebHob project Calin Dragomir
@ 2013-06-12 9:30 ` Calin Dragomir
2013-06-12 9:30 ` [Webhob] [PATCH 3/3] Create Data Store Interface (DSI) file Calin Dragomir
1 sibling, 0 replies; 6+ messages in thread
From: Calin Dragomir @ 2013-06-12 9:30 UTC (permalink / raw)
To: webhob
This will be used to store Tasks related information in the database
using Django ORM.
---
bitbake/lib/webhob/orm/__init__.py | 0
bitbake/lib/webhob/orm/models.py | 50 ++++++++++++++++++++++++++++++++++
bitbake/lib/webhob/orm/tests.py | 16 +++++++++++
bitbake/lib/webhob/orm/views.py | 1 +
bitbake/lib/webhob/whbmain/settings.py | 1 +
5 files changed, 68 insertions(+)
create mode 100644 bitbake/lib/webhob/orm/__init__.py
create mode 100644 bitbake/lib/webhob/orm/models.py
create mode 100644 bitbake/lib/webhob/orm/tests.py
create mode 100644 bitbake/lib/webhob/orm/views.py
diff --git a/bitbake/lib/webhob/orm/__init__.py b/bitbake/lib/webhob/orm/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/bitbake/lib/webhob/orm/models.py b/bitbake/lib/webhob/orm/models.py
new file mode 100644
index 0000000..13eadd4
--- /dev/null
+++ b/bitbake/lib/webhob/orm/models.py
@@ -0,0 +1,50 @@
+from django.db import models
+
+class Tasks(models.Model):
+
+ SSTATE_RESULT = (
+ ('not_applicable', 'Not Applicable'), # For rest of tasks, but they still need checking.
+ ('unavailable', 'Unavailable'), # it is a miss
+ ('failed', 'Failed'), # there was a pkg, but the script failed
+ ('restored', 'Restored'), # succesfully restored
+ )
+
+ TASK_CODING = (
+ ('python', 'Python'),
+ ('shell', 'Shell'),
+ )
+
+ TASK_OUTCOME = (
+ ('succeded', 'Succeded'),
+ ('failed', 'Failed'),
+ ('existing', 'Existing'),
+ ('sstate', 'Sstate'),
+ ('covered', 'Covered'),
+ )
+
+ uuid = models.CharField(max_length=100)
+ task_id = models.IntegerField()
+ order = models.IntegerField()
+ task_executed = models.BooleanField() # True means Executed, False means Prebuilt
+ outcome = models.CharField(max_length=50, choices=TASK_OUTCOME)
+ sstate_checksum = models.CharField(max_length=100)
+ path_to_sstate_obj = models.FilePathField(max_length=500, blank=True)
+ recipe = models.CharField(max_length=100)
+ task_name = models.CharField(max_length=100)
+ source_url = models.FilePathField(max_length=200)
+ log_file = models.FilePathField(max_length=200, blank=True)
+ work_directory = models.FilePathField(max_length=200)
+ script_type = models.CharField(max_length=10, choices=TASK_CODING)
+ file_path = models.FilePathField(max_length=200)
+ line_number = models.IntegerField()
+ py_stack_trace = models.TextField()
+ disk_io = models.DecimalField(max_digits=20, decimal_places=10)
+ cpu_usage = models.IntegerField()
+ elapsed_time = models.CharField(max_length=50)
+ dependent_tasks = models.TextField() ## do we need only to show text here or insert references? how do we connect to other tasks from the same table ?
+ errors_no = models.IntegerField()
+ warnings_no = models.IntegerField()
+ error = models.TextField()
+ warning = models.TextField()
+ sstate_result = models.CharField(max_length=50, choices=SSTATE_RESULT)
+
diff --git a/bitbake/lib/webhob/orm/tests.py b/bitbake/lib/webhob/orm/tests.py
new file mode 100644
index 0000000..501deb7
--- /dev/null
+++ b/bitbake/lib/webhob/orm/tests.py
@@ -0,0 +1,16 @@
+"""
+This file demonstrates writing tests using the unittest module. These will pass
+when you run "manage.py test".
+
+Replace this with more appropriate tests for your application.
+"""
+
+from django.test import TestCase
+
+
+class SimpleTest(TestCase):
+ def test_basic_addition(self):
+ """
+ Tests that 1 + 1 always equals 2.
+ """
+ self.assertEqual(1 + 1, 2)
diff --git a/bitbake/lib/webhob/orm/views.py b/bitbake/lib/webhob/orm/views.py
new file mode 100644
index 0000000..60f00ef
--- /dev/null
+++ b/bitbake/lib/webhob/orm/views.py
@@ -0,0 +1 @@
+# Create your views here.
diff --git a/bitbake/lib/webhob/whbmain/settings.py b/bitbake/lib/webhob/whbmain/settings.py
index bc59543..6a4e67a 100644
--- a/bitbake/lib/webhob/whbmain/settings.py
+++ b/bitbake/lib/webhob/whbmain/settings.py
@@ -124,6 +124,7 @@ INSTALLED_APPS = (
# 'django.contrib.admin',
# Uncomment the next line to enable admin documentation:
# 'django.contrib.admindocs',
+ 'orm',
)
# A sample logging configuration. The only tangible logging
--
1.8.1.2
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [Webhob] [PATCH 3/3] Create Data Store Interface (DSI) file
2013-06-12 9:30 [Webhob] [PATCH 1/3] Rename main WebHob project Calin Dragomir
2013-06-12 9:30 ` [Webhob] [PATCH 2/3] Create Tasks Django model Calin Dragomir
@ 2013-06-12 9:30 ` Calin Dragomir
2013-06-12 9:53 ` Iorga, Cristian
1 sibling, 1 reply; 6+ messages in thread
From: Calin Dragomir @ 2013-06-12 9:30 UTC (permalink / raw)
To: webhob
This adds the first version of the DSI file. It uses the Knotty
code for now, but inserts task related information into the database
using the Django ORM
---
bitbake/lib/bb/ui/dsi.py | 609 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 609 insertions(+)
create mode 100644 bitbake/lib/bb/ui/dsi.py
diff --git a/bitbake/lib/bb/ui/dsi.py b/bitbake/lib/bb/ui/dsi.py
new file mode 100644
index 0000000..4bcd509
--- /dev/null
+++ b/bitbake/lib/bb/ui/dsi.py
@@ -0,0 +1,609 @@
+#
+# BitBake (No)TTY UI Implementation
+#
+# Handling output to TTYs or files (no TTY)
+#
+# Copyright (C) 2006-2012 Richard Purdie
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+from __future__ import division
+
+import os
+import sys
+import xmlrpclib
+import logging
+import progressbar
+import signal
+import bb.msg
+import time
+import fcntl
+import struct
+import copy
+import datetime
+
+from bb.ui import uihelper
+
+logger = logging.getLogger("BitBake")
+interactive = sys.stdout.isatty()
+
+class WebHOBHelper(object):
+
+ def __init__(self):
+ self.configure_django()
+ self.task_order = 0
+ self.tasks_information = {}
+
+ def configure_django(self):
+ import webhob.whbmain.settings as whb_django_settings
+ from django.core.management import setup_environ
+ setup_environ(whb_django_settings)
+ # Add webhob to sys path for importing modules
+ sys.path.append(os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), 'webhob'))
+
+ def store_started_task(self, event, uuid):
+ self.task_order += 1
+ self.tasks_information[event.pid] = {
+ 'uuid': uuid,
+ 'task_id': 1,
+ 'task_executed': True,
+ 'order': self.task_order,
+ 'recipe': event._package,
+ 'task_name': event._task,
+ 'start_time': time.time(),
+ }
+
+ def update_stored_tasks(self, event, uuid):
+ self.tasks_information[event.pid].update({
+ 'outcome': event.getDisplayName().lower(),
+ 'end_time': time.time(),
+ })
+
+ def write_in_database(self, tasks_information):
+ # This needs to be imported after we have configured the Django settings file
+ from webhob.orm.models import Tasks
+ for pid in tasks_information.keys():
+ task = tasks_information[pid]
+
+ task_obj = Tasks.objects.create(uuid=task['uuid'],
+ order=task['order'],
+ outcome=task['outcome'],
+ recipe=task['recipe'],
+ task_name=task['task_name'],
+ elapsed_time=task['end_time'] - task['start_time'],
+ # The next lines are just for testing reasons
+ task_id=1,
+ path_to_sstate_obj='/home/calin',
+ source_url='/',
+ log_file='/',
+ work_directory='/',
+ script_type='python',
+ file_path='/',
+ line_number=55,
+ py_stack_trace='Testing traceback',
+ disk_io = 23.55633,
+ cpu_usage=43,
+ dependent_tasks='Task 1',
+ errors_no=0,
+ warnings_no=0,
+ error='None',
+ warning='None',
+ sstate_result='not_applicable')
+ task_obj.save()
+
+class BBProgress(progressbar.ProgressBar):
+ def __init__(self, msg, maxval):
+ self.msg = msg
+ widgets = [progressbar.Percentage(), ' ', progressbar.Bar(), ' ',
+ progressbar.ETA()]
+
+ try:
+ self._resize_default = signal.getsignal(signal.SIGWINCH)
+ except:
+ self._resize_default = None
+ progressbar.ProgressBar.__init__(self, maxval, [self.msg + ": "] + widgets)
+
+ def _handle_resize(self, signum, frame):
+ progressbar.ProgressBar._handle_resize(self, signum, frame)
+ if self._resize_default:
+ self._resize_default(signum, frame)
+ def finish(self):
+ progressbar.ProgressBar.finish(self)
+ if self._resize_default:
+ signal.signal(signal.SIGWINCH, self._resize_default)
+
+class NonInteractiveProgress(object):
+ fobj = sys.stdout
+
+ def __init__(self, msg, maxval):
+ self.msg = msg
+ self.maxval = maxval
+
+ def start(self):
+ self.fobj.write("%s..." % self.msg)
+ self.fobj.flush()
+ return self
+
+ def update(self, value):
+ pass
+
+ def finish(self):
+ self.fobj.write("done.\n")
+ self.fobj.flush()
+
+def new_progress(msg, maxval):
+ if interactive:
+ return BBProgress(msg, maxval)
+ else:
+ return NonInteractiveProgress(msg, maxval)
+
+def pluralise(singular, plural, qty):
+ if(qty == 1):
+ return singular % qty
+ else:
+ return plural % qty
+
+
+class InteractConsoleLogFilter(logging.Filter):
+ def __init__(self, tf, format):
+ self.tf = tf
+ self.format = format
+
+ def filter(self, record):
+ if record.levelno == self.format.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")):
+ return False
+ self.tf.clearFooter()
+ return True
+
+class TerminalFilter(object):
+ columns = 80
+
+ def sigwinch_handle(self, signum, frame):
+ self.columns = self.getTerminalColumns()
+ if self._sigwinch_default:
+ self._sigwinch_default(signum, frame)
+
+ def getTerminalColumns(self):
+ def ioctl_GWINSZ(fd):
+ try:
+ cr = struct.unpack('hh', fcntl.ioctl(fd, self.termios.TIOCGWINSZ, '1234'))
+ except:
+ return None
+ return cr
+ cr = ioctl_GWINSZ(sys.stdout.fileno())
+ if not cr:
+ try:
+ fd = os.open(os.ctermid(), os.O_RDONLY)
+ cr = ioctl_GWINSZ(fd)
+ os.close(fd)
+ except:
+ pass
+ if not cr:
+ try:
+ cr = (env['LINES'], env['COLUMNS'])
+ except:
+ cr = (25, 80)
+ return cr[1]
+
+ def __init__(self, main, helper, console, format):
+ self.main = main
+ self.helper = helper
+ self.cuu = None
+ self.stdinbackup = None
+ self.interactive = sys.stdout.isatty()
+ self.footer_present = False
+ self.lastpids = []
+
+ if not self.interactive:
+ return
+
+ try:
+ import curses
+ except ImportError:
+ sys.exit("FATAL: The knotty ui could not load the required curses python module.")
+
+ import termios
+ self.curses = curses
+ self.termios = termios
+ try:
+ fd = sys.stdin.fileno()
+ self.stdinbackup = termios.tcgetattr(fd)
+ new = copy.deepcopy(self.stdinbackup)
+ new[3] = new[3] & ~termios.ECHO
+ termios.tcsetattr(fd, termios.TCSADRAIN, new)
+ curses.setupterm()
+ if curses.tigetnum("colors") > 2:
+ format.enable_color()
+ self.ed = curses.tigetstr("ed")
+ if self.ed:
+ self.cuu = curses.tigetstr("cuu")
+ try:
+ self._sigwinch_default = signal.getsignal(signal.SIGWINCH)
+ signal.signal(signal.SIGWINCH, self.sigwinch_handle)
+ except:
+ pass
+ self.columns = self.getTerminalColumns()
+ except:
+ self.cuu = None
+ console.addFilter(InteractConsoleLogFilter(self, format))
+
+ def clearFooter(self):
+ if self.footer_present:
+ lines = self.footer_present
+ sys.stdout.write(self.curses.tparm(self.cuu, lines))
+ sys.stdout.write(self.curses.tparm(self.ed))
+ self.footer_present = False
+
+ def updateFooter(self):
+ if not self.cuu:
+ return
+ activetasks = self.helper.running_tasks
+ failedtasks = self.helper.failed_tasks
+ runningpids = self.helper.running_pids
+ if self.footer_present and (self.lastcount == self.helper.tasknumber_current) and (self.lastpids == runningpids):
+ return
+ if self.footer_present:
+ self.clearFooter()
+ if (not self.helper.tasknumber_total or self.helper.tasknumber_current == self.helper.tasknumber_total) and not len(activetasks):
+ return
+ tasks = []
+ for t in runningpids:
+ tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
+
+ if self.main.shutdown:
+ content = "Waiting for %s running tasks to finish:" % len(activetasks)
+ elif not len(activetasks):
+ content = "No currently running tasks (%s of %s)" % (self.helper.tasknumber_current, self.helper.tasknumber_total)
+ else:
+ content = "Currently %s running tasks (%s of %s):" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total)
+ print(content)
+ lines = 1 + int(len(content) / (self.columns + 1))
+ for tasknum, task in enumerate(tasks):
+ content = "%s: %s" % (tasknum, task)
+ print(content)
+ lines = lines + 1 + int(len(content) / (self.columns + 1))
+ self.footer_present = lines
+ self.lastpids = runningpids[:]
+ self.lastcount = self.helper.tasknumber_current
+
+ def finish(self):
+ if self.stdinbackup:
+ fd = sys.stdin.fileno()
+ self.termios.tcsetattr(fd, self.termios.TCSADRAIN, self.stdinbackup)
+
+def main(server, eventHandler, params, tf = TerminalFilter):
+ print "This is running in DSI mode"
+
+ # Generate an unique ID for this build
+ # TODO: for multiple build commands this might not work as expected
+ import uuid
+ uuid = sessionid = str(uuid.uuid4())
+
+ # Get values of variables which control our output
+ includelogs, error = server.runCommand(["getVariable", "BBINCLUDELOGS"])
+ if error:
+ logger.error("Unable to get the value of BBINCLUDELOGS variable: %s" % error)
+ return 1
+ loglines, error = server.runCommand(["getVariable", "BBINCLUDELOGS_LINES"])
+ if error:
+ logger.error("Unable to get the value of BBINCLUDELOGS_LINES variable: %s" % error)
+ return 1
+ consolelogfile, error = server.runCommand(["getVariable", "BB_CONSOLELOG"])
+ if error:
+ logger.error("Unable to get the value of BB_CONSOLELOG variable: %s" % error)
+ return 1
+
+ if sys.stdin.isatty() and sys.stdout.isatty():
+ log_exec_tty = True
+ else:
+ log_exec_tty = False
+
+ helper = uihelper.BBUIHelper()
+
+ console = logging.StreamHandler(sys.stdout)
+ format_str = "%(levelname)s: %(message)s"
+ format = bb.msg.BBLogFormatter(format_str)
+ bb.msg.addDefaultlogFilter(console)
+ console.setFormatter(format)
+ logger.addHandler(console)
+
+ if consolelogfile and not params.options.show_environment:
+ bb.utils.mkdirhier(os.path.dirname(consolelogfile))
+ conlogformat = bb.msg.BBLogFormatter(format_str)
+ consolelog = logging.FileHandler(consolelogfile)
+ bb.msg.addDefaultlogFilter(consolelog)
+ consolelog.setFormatter(conlogformat)
+ logger.addHandler(consolelog)
+
+ try:
+ params.updateFromServer(server)
+ cmdline = params.parseActions()
+ if not cmdline:
+ print("Nothing to do. Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information.")
+ return 1
+ if 'msg' in cmdline and cmdline['msg']:
+ logger.error(cmdline['msg'])
+ return 1
+
+ ret, error = server.runCommand(cmdline['action'])
+ if error:
+ logger.error("Command '%s' failed: %s" % (cmdline, error))
+ return 1
+ elif ret != True:
+ logger.error("Command '%s' failed: returned %s" % (cmdline, ret))
+ return 1
+ except xmlrpclib.Fault as x:
+ logger.error("XMLRPC Fault getting commandline:\n %s" % x)
+ return 1
+
+ parseprogress = None
+ cacheprogress = None
+ main.shutdown = 0
+ interrupted = False
+ return_value = 0
+ errors = 0
+ warnings = 0
+ taskfailures = []
+ wbhbhelper = WebHOBHelper()
+
+ termfilter = tf(main, helper, console, format)
+
+ while True:
+ try:
+ termfilter.updateFooter()
+ event = eventHandler.waitEvent(0.25)
+
+ if event is None:
+ if main.shutdown > 1:
+ break
+ continue
+
+ helper.eventHandler(event)
+
+ if isinstance(event, bb.runqueue.runQueueExitWait):
+ if not main.shutdown:
+ main.shutdown = 1
+
+ if isinstance(event, bb.build.TaskStarted):
+ wbhbhelper.store_started_task(event, uuid)
+
+ if isinstance(event, (bb.build.TaskSucceeded, bb.build.TaskFailedSilent, bb.build.TaskFailed)):
+ wbhbhelper.update_stored_tasks(event, uuid)
+
+ if isinstance(event, bb.event.LogExecTTY):
+ if log_exec_tty:
+ tries = event.retries
+ while tries:
+ print("Trying to run: %s" % event.prog)
+ if os.system(event.prog) == 0:
+ break
+ time.sleep(event.sleep_delay)
+ tries -= 1
+ if tries:
+ continue
+ logger.warn(event.msg)
+ continue
+
+ if isinstance(event, logging.LogRecord):
+ if event.levelno >= format.ERROR:
+ errors = errors + 1
+ return_value = 1
+ elif event.levelno == format.WARNING:
+ warnings = warnings + 1
+ # For "normal" logging conditions, don't show note logs from tasks
+ # but do show them if the user has changed the default log level to
+ # include verbose/debug messages
+ if event.taskpid != 0 and event.levelno <= format.NOTE:
+ continue
+ logger.handle(event)
+ continue
+
+ if isinstance(event, bb.build.TaskFailed):
+ return_value = 1
+ logfile = event.logfile
+ if logfile and os.path.exists(logfile):
+ termfilter.clearFooter()
+ bb.error("Logfile of failure stored in: %s" % logfile)
+ if includelogs and not event.errprinted:
+ print("Log data follows:")
+ f = open(logfile, "r")
+ lines = []
+ while True:
+ l = f.readline()
+ if l == '':
+ break
+ l = l.rstrip()
+ if loglines:
+ lines.append(' | %s' % l)
+ if len(lines) > int(loglines):
+ lines.pop(0)
+ else:
+ print('| %s' % l)
+ f.close()
+ if lines:
+ for line in lines:
+ print(line)
+ if isinstance(event, bb.build.TaskBase):
+ logger.info(event._message)
+ continue
+ if isinstance(event, bb.event.ParseStarted):
+ if event.total == 0:
+ continue
+ parseprogress = new_progress("Parsing recipes", event.total).start()
+ continue
+ if isinstance(event, bb.event.ParseProgress):
+ parseprogress.update(event.current)
+ continue
+ if isinstance(event, bb.event.ParseCompleted):
+ if not parseprogress:
+ continue
+
+ parseprogress.finish()
+ print(("Parsing of %d .bb files complete (%d cached, %d parsed). %d targets, %d skipped, %d masked, %d errors."
+ % ( event.total, event.cached, event.parsed, event.virtuals, event.skipped, event.masked, event.errors)))
+ continue
+
+ if isinstance(event, bb.event.CacheLoadStarted):
+ cacheprogress = new_progress("Loading cache", event.total).start()
+ continue
+ if isinstance(event, bb.event.CacheLoadProgress):
+ cacheprogress.update(event.current)
+ continue
+ if isinstance(event, bb.event.CacheLoadCompleted):
+ cacheprogress.finish()
+ print("Loaded %d entries from dependency cache." % event.num_entries)
+ continue
+
+ if isinstance(event, bb.command.CommandFailed):
+ return_value = event.exitcode
+ errors = errors + 1
+ logger.error("Command execution failed: %s", event.error)
+ main.shutdown = 2
+ continue
+ if isinstance(event, bb.command.CommandExit):
+ if not return_value:
+ return_value = event.exitcode
+ continue
+ if isinstance(event, (bb.command.CommandCompleted, bb.cooker.CookerExit)):
+ main.shutdown = 2
+ continue
+ if isinstance(event, bb.event.MultipleProviders):
+ logger.info("multiple providers are available for %s%s (%s)", event._is_runtime and "runtime " or "",
+ event._item,
+ ", ".join(event._candidates))
+ logger.info("consider defining a PREFERRED_PROVIDER entry to match %s", event._item)
+ continue
+ if isinstance(event, bb.event.NoProvider):
+ return_value = 1
+ errors = errors + 1
+ if event._runtime:
+ r = "R"
+ else:
+ r = ""
+
+ if event._dependees:
+ logger.error("Nothing %sPROVIDES '%s' (but %s %sDEPENDS on or otherwise requires it)", r, event._item, ", ".join(event._dependees), r)
+ else:
+ logger.error("Nothing %sPROVIDES '%s'", r, event._item)
+ if event._reasons:
+ for reason in event._reasons:
+ logger.error("%s", reason)
+ continue
+
+ if isinstance(event, bb.runqueue.sceneQueueTaskStarted):
+ logger.info("Running setscene task %d of %d (%s)" % (event.stats.completed + event.stats.active + event.stats.failed + 1, event.stats.total, event.taskstring))
+ continue
+
+ if isinstance(event, bb.runqueue.runQueueTaskStarted):
+ if event.noexec:
+ tasktype = 'noexec task'
+ else:
+ tasktype = 'task'
+ logger.info("Running %s %s of %s (ID: %s, %s)",
+ tasktype,
+ event.stats.completed + event.stats.active +
+ event.stats.failed + 1,
+ event.stats.total, event.taskid, event.taskstring)
+ continue
+
+ if isinstance(event, bb.runqueue.runQueueTaskFailed):
+ taskfailures.append(event.taskstring)
+ logger.error("Task %s (%s) failed with exit code '%s'",
+ event.taskid, event.taskstring, event.exitcode)
+ continue
+
+ if isinstance(event, bb.runqueue.sceneQueueTaskFailed):
+ logger.warn("Setscene task %s (%s) failed with exit code '%s' - real task will be run instead",
+ event.taskid, event.taskstring, event.exitcode)
+ continue
+
+ if isinstance(event, bb.event.ConfigParsed):
+ # timestamp should be added for this
+ continue
+
+ if isinstance(event, bb.event.RecipeParsed):
+ # timestamp should be added for this
+ continue
+
+ if isinstance(event, bb.event.OperationStarted):
+ # timestamp should be added for this
+ continue
+
+ if isinstance(event, bb.event.OperationCompleted):
+ # timestamp should be added for this
+ # signal a complete operation
+ # calculate timing
+ continue
+
+ if isinstance(event, bb.event.DiskFull):
+ # trigger an error
+ continue
+
+ # ignore
+ if isinstance(event, (bb.event.BuildBase,
+ bb.event.StampUpdate,
+ bb.event.RecipePreFinalise,
+ bb.runqueue.runQueueEvent,
+ bb.runqueue.runQueueExitWait,
+ bb.event.OperationProgress)):
+ continue
+
+ logger.error("Unknown event: %s", event)
+
+ except EnvironmentError as ioerror:
+ termfilter.clearFooter()
+ # ignore interrupted io
+ if ioerror.args[0] == 4:
+ pass
+ except KeyboardInterrupt:
+ termfilter.clearFooter()
+ if main.shutdown == 1:
+ print("\nSecond Keyboard Interrupt, stopping...\n")
+ _, error = server.runCommand(["stateStop"])
+ if error:
+ logger.error("Unable to cleanly stop: %s" % error)
+ if main.shutdown == 0:
+ print("\nKeyboard Interrupt, closing down...\n")
+ interrupted = True
+ _, error = server.runCommand(["stateShutdown"])
+ if error:
+ logger.error("Unable to cleanly shutdown: %s" % error)
+ main.shutdown = main.shutdown + 1
+ pass
+
+ summary = ""
+ if taskfailures:
+ summary += pluralise("\nSummary: %s task failed:",
+ "\nSummary: %s tasks failed:", len(taskfailures))
+ for failure in taskfailures:
+ summary += "\n %s" % failure
+ if warnings:
+ summary += pluralise("\nSummary: There was %s WARNING message shown.",
+ "\nSummary: There were %s WARNING messages shown.", warnings)
+ if return_value:
+ summary += pluralise("\nSummary: There was %s ERROR message shown, returning a non-zero exit code.",
+ "\nSummary: There were %s ERROR messages shown, returning a non-zero exit code.", errors)
+ if summary:
+ print(summary)
+
+ if interrupted:
+ print("Execution was interrupted, returning a non-zero exit code.")
+ if return_value == 0:
+ return_value = 1
+
+ termfilter.finish()
+ wbhbhelper.write_in_database(wbhbhelper.tasks_information)
+
+ return return_value
+
--
1.8.1.2
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [Webhob] [PATCH 3/3] Create Data Store Interface (DSI) file
2013-06-12 9:30 ` [Webhob] [PATCH 3/3] Create Data Store Interface (DSI) file Calin Dragomir
@ 2013-06-12 9:53 ` Iorga, Cristian
2013-06-12 10:46 ` Calin Dragomir
0 siblings, 1 reply; 6+ messages in thread
From: Iorga, Cristian @ 2013-06-12 9:53 UTC (permalink / raw)
To: Dragomir, CalinX L, webhob@yoctoproject.org
Hello Calin,
I might be wrong, but this:
path_to_sstate_obj='/home/calin',
does not seem right.
Regards,
Cristian
-----Original Message-----
From: webhob-bounces@yoctoproject.org [mailto:webhob-bounces@yoctoproject.org] On Behalf Of Calin Dragomir
Sent: Wednesday, June 12, 2013 12:30 PM
To: webhob@yoctoproject.org
Subject: [Webhob] [PATCH 3/3] Create Data Store Interface (DSI) file
This adds the first version of the DSI file. It uses the Knotty code for now, but inserts task related information into the database using the Django ORM
---
bitbake/lib/bb/ui/dsi.py | 609 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 609 insertions(+)
create mode 100644 bitbake/lib/bb/ui/dsi.py
diff --git a/bitbake/lib/bb/ui/dsi.py b/bitbake/lib/bb/ui/dsi.py new file mode 100644 index 0000000..4bcd509
--- /dev/null
+++ b/bitbake/lib/bb/ui/dsi.py
@@ -0,0 +1,609 @@
+#
+# BitBake (No)TTY UI Implementation
+#
+# Handling output to TTYs or files (no TTY) # # Copyright (C) 2006-2012
+Richard Purdie # # This program is free software; you can redistribute
+it and/or modify # it under the terms of the GNU General Public License
+version 2 as # published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, # but
+WITHOUT ANY WARRANTY; without even the implied warranty of #
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU
+General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+along # with this program; if not, write to the Free Software
+Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+from __future__ import division
+
+import os
+import sys
+import xmlrpclib
+import logging
+import progressbar
+import signal
+import bb.msg
+import time
+import fcntl
+import struct
+import copy
+import datetime
+
+from bb.ui import uihelper
+
+logger = logging.getLogger("BitBake")
+interactive = sys.stdout.isatty()
+
+class WebHOBHelper(object):
+
+ def __init__(self):
+ self.configure_django()
+ self.task_order = 0
+ self.tasks_information = {}
+
+ def configure_django(self):
+ import webhob.whbmain.settings as whb_django_settings
+ from django.core.management import setup_environ
+ setup_environ(whb_django_settings)
+ # Add webhob to sys path for importing modules
+
+ sys.path.append(os.path.join(os.path.dirname(os.path.dirname(os.path.d
+ irname(__file__))), 'webhob'))
+
+ def store_started_task(self, event, uuid):
+ self.task_order += 1
+ self.tasks_information[event.pid] = {
+ 'uuid': uuid,
+ 'task_id': 1,
+ 'task_executed': True,
+ 'order': self.task_order,
+ 'recipe': event._package,
+ 'task_name': event._task,
+ 'start_time': time.time(),
+ }
+
+ def update_stored_tasks(self, event, uuid):
+ self.tasks_information[event.pid].update({
+ 'outcome': event.getDisplayName().lower(),
+ 'end_time': time.time(),
+ })
+
+ def write_in_database(self, tasks_information):
+ # This needs to be imported after we have configured the Django settings file
+ from webhob.orm.models import Tasks
+ for pid in tasks_information.keys():
+ task = tasks_information[pid]
+
+ task_obj = Tasks.objects.create(uuid=task['uuid'],
+ order=task['order'],
+ outcome=task['outcome'],
+ recipe=task['recipe'],
+ task_name=task['task_name'],
+ elapsed_time=task['end_time'] - task['start_time'],
+ # The next lines are just for testing reasons
+ task_id=1,
+ path_to_sstate_obj='/home/calin',
+ source_url='/',
+ log_file='/',
+ work_directory='/',
+ script_type='python',
+ file_path='/',
+ line_number=55,
+ py_stack_trace='Testing traceback',
+ disk_io = 23.55633,
+ cpu_usage=43,
+ dependent_tasks='Task 1',
+ errors_no=0,
+ warnings_no=0,
+ error='None',
+ warning='None',
+ sstate_result='not_applicable')
+ task_obj.save()
+
+class BBProgress(progressbar.ProgressBar):
+ def __init__(self, msg, maxval):
+ self.msg = msg
+ widgets = [progressbar.Percentage(), ' ', progressbar.Bar(), ' ',
+ progressbar.ETA()]
+
+ try:
+ self._resize_default = signal.getsignal(signal.SIGWINCH)
+ except:
+ self._resize_default = None
+ progressbar.ProgressBar.__init__(self, maxval, [self.msg + ":
+ "] + widgets)
+
+ def _handle_resize(self, signum, frame):
+ progressbar.ProgressBar._handle_resize(self, signum, frame)
+ if self._resize_default:
+ self._resize_default(signum, frame)
+ def finish(self):
+ progressbar.ProgressBar.finish(self)
+ if self._resize_default:
+ signal.signal(signal.SIGWINCH, self._resize_default)
+
+class NonInteractiveProgress(object):
+ fobj = sys.stdout
+
+ def __init__(self, msg, maxval):
+ self.msg = msg
+ self.maxval = maxval
+
+ def start(self):
+ self.fobj.write("%s..." % self.msg)
+ self.fobj.flush()
+ return self
+
+ def update(self, value):
+ pass
+
+ def finish(self):
+ self.fobj.write("done.\n")
+ self.fobj.flush()
+
+def new_progress(msg, maxval):
+ if interactive:
+ return BBProgress(msg, maxval)
+ else:
+ return NonInteractiveProgress(msg, maxval)
+
+def pluralise(singular, plural, qty):
+ if(qty == 1):
+ return singular % qty
+ else:
+ return plural % qty
+
+
+class InteractConsoleLogFilter(logging.Filter):
+ def __init__(self, tf, format):
+ self.tf = tf
+ self.format = format
+
+ def filter(self, record):
+ if record.levelno == self.format.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")):
+ return False
+ self.tf.clearFooter()
+ return True
+
+class TerminalFilter(object):
+ columns = 80
+
+ def sigwinch_handle(self, signum, frame):
+ self.columns = self.getTerminalColumns()
+ if self._sigwinch_default:
+ self._sigwinch_default(signum, frame)
+
+ def getTerminalColumns(self):
+ def ioctl_GWINSZ(fd):
+ try:
+ cr = struct.unpack('hh', fcntl.ioctl(fd, self.termios.TIOCGWINSZ, '1234'))
+ except:
+ return None
+ return cr
+ cr = ioctl_GWINSZ(sys.stdout.fileno())
+ if not cr:
+ try:
+ fd = os.open(os.ctermid(), os.O_RDONLY)
+ cr = ioctl_GWINSZ(fd)
+ os.close(fd)
+ except:
+ pass
+ if not cr:
+ try:
+ cr = (env['LINES'], env['COLUMNS'])
+ except:
+ cr = (25, 80)
+ return cr[1]
+
+ def __init__(self, main, helper, console, format):
+ self.main = main
+ self.helper = helper
+ self.cuu = None
+ self.stdinbackup = None
+ self.interactive = sys.stdout.isatty()
+ self.footer_present = False
+ self.lastpids = []
+
+ if not self.interactive:
+ return
+
+ try:
+ import curses
+ except ImportError:
+ sys.exit("FATAL: The knotty ui could not load the required
+ curses python module.")
+
+ import termios
+ self.curses = curses
+ self.termios = termios
+ try:
+ fd = sys.stdin.fileno()
+ self.stdinbackup = termios.tcgetattr(fd)
+ new = copy.deepcopy(self.stdinbackup)
+ new[3] = new[3] & ~termios.ECHO
+ termios.tcsetattr(fd, termios.TCSADRAIN, new)
+ curses.setupterm()
+ if curses.tigetnum("colors") > 2:
+ format.enable_color()
+ self.ed = curses.tigetstr("ed")
+ if self.ed:
+ self.cuu = curses.tigetstr("cuu")
+ try:
+ self._sigwinch_default = signal.getsignal(signal.SIGWINCH)
+ signal.signal(signal.SIGWINCH, self.sigwinch_handle)
+ except:
+ pass
+ self.columns = self.getTerminalColumns()
+ except:
+ self.cuu = None
+ console.addFilter(InteractConsoleLogFilter(self, format))
+
+ def clearFooter(self):
+ if self.footer_present:
+ lines = self.footer_present
+ sys.stdout.write(self.curses.tparm(self.cuu, lines))
+ sys.stdout.write(self.curses.tparm(self.ed))
+ self.footer_present = False
+
+ def updateFooter(self):
+ if not self.cuu:
+ return
+ activetasks = self.helper.running_tasks
+ failedtasks = self.helper.failed_tasks
+ runningpids = self.helper.running_pids
+ if self.footer_present and (self.lastcount == self.helper.tasknumber_current) and (self.lastpids == runningpids):
+ return
+ if self.footer_present:
+ self.clearFooter()
+ if (not self.helper.tasknumber_total or self.helper.tasknumber_current == self.helper.tasknumber_total) and not len(activetasks):
+ return
+ tasks = []
+ for t in runningpids:
+ tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
+
+ if self.main.shutdown:
+ content = "Waiting for %s running tasks to finish:" % len(activetasks)
+ elif not len(activetasks):
+ content = "No currently running tasks (%s of %s)" % (self.helper.tasknumber_current, self.helper.tasknumber_total)
+ else:
+ content = "Currently %s running tasks (%s of %s):" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total)
+ print(content)
+ lines = 1 + int(len(content) / (self.columns + 1))
+ for tasknum, task in enumerate(tasks):
+ content = "%s: %s" % (tasknum, task)
+ print(content)
+ lines = lines + 1 + int(len(content) / (self.columns + 1))
+ self.footer_present = lines
+ self.lastpids = runningpids[:]
+ self.lastcount = self.helper.tasknumber_current
+
+ def finish(self):
+ if self.stdinbackup:
+ fd = sys.stdin.fileno()
+ self.termios.tcsetattr(fd, self.termios.TCSADRAIN,
+ self.stdinbackup)
+
+def main(server, eventHandler, params, tf = TerminalFilter):
+ print "This is running in DSI mode"
+
+ # Generate an unique ID for this build
+ # TODO: for multiple build commands this might not work as expected
+ import uuid
+ uuid = sessionid = str(uuid.uuid4())
+
+ # Get values of variables which control our output
+ includelogs, error = server.runCommand(["getVariable", "BBINCLUDELOGS"])
+ if error:
+ logger.error("Unable to get the value of BBINCLUDELOGS variable: %s" % error)
+ return 1
+ loglines, error = server.runCommand(["getVariable", "BBINCLUDELOGS_LINES"])
+ if error:
+ logger.error("Unable to get the value of BBINCLUDELOGS_LINES variable: %s" % error)
+ return 1
+ consolelogfile, error = server.runCommand(["getVariable", "BB_CONSOLELOG"])
+ if error:
+ logger.error("Unable to get the value of BB_CONSOLELOG variable: %s" % error)
+ return 1
+
+ if sys.stdin.isatty() and sys.stdout.isatty():
+ log_exec_tty = True
+ else:
+ log_exec_tty = False
+
+ helper = uihelper.BBUIHelper()
+
+ console = logging.StreamHandler(sys.stdout)
+ format_str = "%(levelname)s: %(message)s"
+ format = bb.msg.BBLogFormatter(format_str)
+ bb.msg.addDefaultlogFilter(console)
+ console.setFormatter(format)
+ logger.addHandler(console)
+
+ if consolelogfile and not params.options.show_environment:
+ bb.utils.mkdirhier(os.path.dirname(consolelogfile))
+ conlogformat = bb.msg.BBLogFormatter(format_str)
+ consolelog = logging.FileHandler(consolelogfile)
+ bb.msg.addDefaultlogFilter(consolelog)
+ consolelog.setFormatter(conlogformat)
+ logger.addHandler(consolelog)
+
+ try:
+ params.updateFromServer(server)
+ cmdline = params.parseActions()
+ if not cmdline:
+ print("Nothing to do. Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information.")
+ return 1
+ if 'msg' in cmdline and cmdline['msg']:
+ logger.error(cmdline['msg'])
+ return 1
+
+ ret, error = server.runCommand(cmdline['action'])
+ if error:
+ logger.error("Command '%s' failed: %s" % (cmdline, error))
+ return 1
+ elif ret != True:
+ logger.error("Command '%s' failed: returned %s" % (cmdline, ret))
+ return 1
+ except xmlrpclib.Fault as x:
+ logger.error("XMLRPC Fault getting commandline:\n %s" % x)
+ return 1
+
+ parseprogress = None
+ cacheprogress = None
+ main.shutdown = 0
+ interrupted = False
+ return_value = 0
+ errors = 0
+ warnings = 0
+ taskfailures = []
+ wbhbhelper = WebHOBHelper()
+
+ termfilter = tf(main, helper, console, format)
+
+ while True:
+ try:
+ termfilter.updateFooter()
+ event = eventHandler.waitEvent(0.25)
+
+ if event is None:
+ if main.shutdown > 1:
+ break
+ continue
+
+ helper.eventHandler(event)
+
+ if isinstance(event, bb.runqueue.runQueueExitWait):
+ if not main.shutdown:
+ main.shutdown = 1
+
+ if isinstance(event, bb.build.TaskStarted):
+ wbhbhelper.store_started_task(event, uuid)
+
+ if isinstance(event, (bb.build.TaskSucceeded, bb.build.TaskFailedSilent, bb.build.TaskFailed)):
+ wbhbhelper.update_stored_tasks(event, uuid)
+
+ if isinstance(event, bb.event.LogExecTTY):
+ if log_exec_tty:
+ tries = event.retries
+ while tries:
+ print("Trying to run: %s" % event.prog)
+ if os.system(event.prog) == 0:
+ break
+ time.sleep(event.sleep_delay)
+ tries -= 1
+ if tries:
+ continue
+ logger.warn(event.msg)
+ continue
+
+ if isinstance(event, logging.LogRecord):
+ if event.levelno >= format.ERROR:
+ errors = errors + 1
+ return_value = 1
+ elif event.levelno == format.WARNING:
+ warnings = warnings + 1
+ # For "normal" logging conditions, don't show note logs from tasks
+ # but do show them if the user has changed the default log level to
+ # include verbose/debug messages
+ if event.taskpid != 0 and event.levelno <= format.NOTE:
+ continue
+ logger.handle(event)
+ continue
+
+ if isinstance(event, bb.build.TaskFailed):
+ return_value = 1
+ logfile = event.logfile
+ if logfile and os.path.exists(logfile):
+ termfilter.clearFooter()
+ bb.error("Logfile of failure stored in: %s" % logfile)
+ if includelogs and not event.errprinted:
+ print("Log data follows:")
+ f = open(logfile, "r")
+ lines = []
+ while True:
+ l = f.readline()
+ if l == '':
+ break
+ l = l.rstrip()
+ if loglines:
+ lines.append(' | %s' % l)
+ if len(lines) > int(loglines):
+ lines.pop(0)
+ else:
+ print('| %s' % l)
+ f.close()
+ if lines:
+ for line in lines:
+ print(line)
+ if isinstance(event, bb.build.TaskBase):
+ logger.info(event._message)
+ continue
+ if isinstance(event, bb.event.ParseStarted):
+ if event.total == 0:
+ continue
+ parseprogress = new_progress("Parsing recipes", event.total).start()
+ continue
+ if isinstance(event, bb.event.ParseProgress):
+ parseprogress.update(event.current)
+ continue
+ if isinstance(event, bb.event.ParseCompleted):
+ if not parseprogress:
+ continue
+
+ parseprogress.finish()
+ print(("Parsing of %d .bb files complete (%d cached, %d parsed). %d targets, %d skipped, %d masked, %d errors."
+ % ( event.total, event.cached, event.parsed, event.virtuals, event.skipped, event.masked, event.errors)))
+ continue
+
+ if isinstance(event, bb.event.CacheLoadStarted):
+ cacheprogress = new_progress("Loading cache", event.total).start()
+ continue
+ if isinstance(event, bb.event.CacheLoadProgress):
+ cacheprogress.update(event.current)
+ continue
+ if isinstance(event, bb.event.CacheLoadCompleted):
+ cacheprogress.finish()
+ print("Loaded %d entries from dependency cache." % event.num_entries)
+ continue
+
+ if isinstance(event, bb.command.CommandFailed):
+ return_value = event.exitcode
+ errors = errors + 1
+ logger.error("Command execution failed: %s", event.error)
+ main.shutdown = 2
+ continue
+ if isinstance(event, bb.command.CommandExit):
+ if not return_value:
+ return_value = event.exitcode
+ continue
+ if isinstance(event, (bb.command.CommandCompleted, bb.cooker.CookerExit)):
+ main.shutdown = 2
+ continue
+ if isinstance(event, bb.event.MultipleProviders):
+ logger.info("multiple providers are available for %s%s (%s)", event._is_runtime and "runtime " or "",
+ event._item,
+ ", ".join(event._candidates))
+ logger.info("consider defining a PREFERRED_PROVIDER entry to match %s", event._item)
+ continue
+ if isinstance(event, bb.event.NoProvider):
+ return_value = 1
+ errors = errors + 1
+ if event._runtime:
+ r = "R"
+ else:
+ r = ""
+
+ if event._dependees:
+ logger.error("Nothing %sPROVIDES '%s' (but %s %sDEPENDS on or otherwise requires it)", r, event._item, ", ".join(event._dependees), r)
+ else:
+ logger.error("Nothing %sPROVIDES '%s'", r, event._item)
+ if event._reasons:
+ for reason in event._reasons:
+ logger.error("%s", reason)
+ continue
+
+ if isinstance(event, bb.runqueue.sceneQueueTaskStarted):
+ logger.info("Running setscene task %d of %d (%s)" % (event.stats.completed + event.stats.active + event.stats.failed + 1, event.stats.total, event.taskstring))
+ continue
+
+ if isinstance(event, bb.runqueue.runQueueTaskStarted):
+ if event.noexec:
+ tasktype = 'noexec task'
+ else:
+ tasktype = 'task'
+ logger.info("Running %s %s of %s (ID: %s, %s)",
+ tasktype,
+ event.stats.completed + event.stats.active +
+ event.stats.failed + 1,
+ event.stats.total, event.taskid, event.taskstring)
+ continue
+
+ if isinstance(event, bb.runqueue.runQueueTaskFailed):
+ taskfailures.append(event.taskstring)
+ logger.error("Task %s (%s) failed with exit code '%s'",
+ event.taskid, event.taskstring, event.exitcode)
+ continue
+
+ if isinstance(event, bb.runqueue.sceneQueueTaskFailed):
+ logger.warn("Setscene task %s (%s) failed with exit code '%s' - real task will be run instead",
+ event.taskid, event.taskstring, event.exitcode)
+ continue
+
+ if isinstance(event, bb.event.ConfigParsed):
+ # timestamp should be added for this
+ continue
+
+ if isinstance(event, bb.event.RecipeParsed):
+ # timestamp should be added for this
+ continue
+
+ if isinstance(event, bb.event.OperationStarted):
+ # timestamp should be added for this
+ continue
+
+ if isinstance(event, bb.event.OperationCompleted):
+ # timestamp should be added for this
+ # signal a complete operation
+ # calculate timing
+ continue
+
+ if isinstance(event, bb.event.DiskFull):
+ # trigger an error
+ continue
+
+ # ignore
+ if isinstance(event, (bb.event.BuildBase,
+ bb.event.StampUpdate,
+ bb.event.RecipePreFinalise,
+ bb.runqueue.runQueueEvent,
+ bb.runqueue.runQueueExitWait,
+ bb.event.OperationProgress)):
+ continue
+
+ logger.error("Unknown event: %s", event)
+
+ except EnvironmentError as ioerror:
+ termfilter.clearFooter()
+ # ignore interrupted io
+ if ioerror.args[0] == 4:
+ pass
+ except KeyboardInterrupt:
+ termfilter.clearFooter()
+ if main.shutdown == 1:
+ print("\nSecond Keyboard Interrupt, stopping...\n")
+ _, error = server.runCommand(["stateStop"])
+ if error:
+ logger.error("Unable to cleanly stop: %s" % error)
+ if main.shutdown == 0:
+ print("\nKeyboard Interrupt, closing down...\n")
+ interrupted = True
+ _, error = server.runCommand(["stateShutdown"])
+ if error:
+ logger.error("Unable to cleanly shutdown: %s" % error)
+ main.shutdown = main.shutdown + 1
+ pass
+
+ summary = ""
+ if taskfailures:
+ summary += pluralise("\nSummary: %s task failed:",
+ "\nSummary: %s tasks failed:", len(taskfailures))
+ for failure in taskfailures:
+ summary += "\n %s" % failure
+ if warnings:
+ summary += pluralise("\nSummary: There was %s WARNING message shown.",
+ "\nSummary: There were %s WARNING messages shown.", warnings)
+ if return_value:
+ summary += pluralise("\nSummary: There was %s ERROR message shown, returning a non-zero exit code.",
+ "\nSummary: There were %s ERROR messages shown, returning a non-zero exit code.", errors)
+ if summary:
+ print(summary)
+
+ if interrupted:
+ print("Execution was interrupted, returning a non-zero exit code.")
+ if return_value == 0:
+ return_value = 1
+
+ termfilter.finish()
+ wbhbhelper.write_in_database(wbhbhelper.tasks_information)
+
+ return return_value
+
--
1.8.1.2
_______________________________________________
webhob mailing list
webhob@yoctoproject.org
https://lists.yoctoproject.org/listinfo/webhob
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Webhob] [PATCH 3/3] Create Data Store Interface (DSI) file
2013-06-12 9:53 ` Iorga, Cristian
@ 2013-06-12 10:46 ` Calin Dragomir
0 siblings, 0 replies; 6+ messages in thread
From: Calin Dragomir @ 2013-06-12 10:46 UTC (permalink / raw)
To: Iorga, Cristian; +Cc: webhob@yoctoproject.org
Hello Cristian,
You are not wrong, but that set of data is not ready yet.
We are working on integrating that information into an event object
which will come from Bitbake.
Until that is ready, this whole chunk of information is put here for
testing reasons:
+ # The next lines are just for testing reasons
+ task_id=1,
+ path_to_sstate_obj='/home/calin',
+ source_url='/',
+ log_file='/',
+ work_directory='/',
+ script_type='python',
+ file_path='/',
+ line_number=55,
+ py_stack_trace='Testing traceback',
+ disk_io = 23.55633,
+ cpu_usage=43,
+ dependent_tasks='Task 1',
+ errors_no=0,
+ warnings_no=0,
+ error='None',
+ warning='None',
+ sstate_result='not_applicable')
Thank you for the feedback,
Calin
On 12.06.2013 12:53, Iorga, Cristian wrote:
> Hello Calin,
>
> I might be wrong, but this:
>
> path_to_sstate_obj='/home/calin',
>
> does not seem right.
>
> Regards,
> Cristian
>
> -----Original Message-----
> From: webhob-bounces@yoctoproject.org [mailto:webhob-bounces@yoctoproject.org] On Behalf Of Calin Dragomir
> Sent: Wednesday, June 12, 2013 12:30 PM
> To: webhob@yoctoproject.org
> Subject: [Webhob] [PATCH 3/3] Create Data Store Interface (DSI) file
>
> This adds the first version of the DSI file. It uses the Knotty code for now, but inserts task related information into the database using the Django ORM
> ---
> bitbake/lib/bb/ui/dsi.py | 609 +++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 609 insertions(+)
> create mode 100644 bitbake/lib/bb/ui/dsi.py
>
> diff --git a/bitbake/lib/bb/ui/dsi.py b/bitbake/lib/bb/ui/dsi.py new file mode 100644 index 0000000..4bcd509
> --- /dev/null
> +++ b/bitbake/lib/bb/ui/dsi.py
> @@ -0,0 +1,609 @@
> +#
> +# BitBake (No)TTY UI Implementation
> +#
> +# Handling output to TTYs or files (no TTY) # # Copyright (C) 2006-2012
> +Richard Purdie # # This program is free software; you can redistribute
> +it and/or modify # it under the terms of the GNU General Public License
> +version 2 as # published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it will be useful, # but
> +WITHOUT ANY WARRANTY; without even the implied warranty of #
> +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU
> +General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +along # with this program; if not, write to the Free Software
> +Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +
> +from __future__ import division
> +
> +import os
> +import sys
> +import xmlrpclib
> +import logging
> +import progressbar
> +import signal
> +import bb.msg
> +import time
> +import fcntl
> +import struct
> +import copy
> +import datetime
> +
> +from bb.ui import uihelper
> +
> +logger = logging.getLogger("BitBake")
> +interactive = sys.stdout.isatty()
> +
> +class WebHOBHelper(object):
> +
> + def __init__(self):
> + self.configure_django()
> + self.task_order = 0
> + self.tasks_information = {}
> +
> + def configure_django(self):
> + import webhob.whbmain.settings as whb_django_settings
> + from django.core.management import setup_environ
> + setup_environ(whb_django_settings)
> + # Add webhob to sys path for importing modules
> +
> + sys.path.append(os.path.join(os.path.dirname(os.path.dirname(os.path.d
> + irname(__file__))), 'webhob'))
> +
> + def store_started_task(self, event, uuid):
> + self.task_order += 1
> + self.tasks_information[event.pid] = {
> + 'uuid': uuid,
> + 'task_id': 1,
> + 'task_executed': True,
> + 'order': self.task_order,
> + 'recipe': event._package,
> + 'task_name': event._task,
> + 'start_time': time.time(),
> + }
> +
> + def update_stored_tasks(self, event, uuid):
> + self.tasks_information[event.pid].update({
> + 'outcome': event.getDisplayName().lower(),
> + 'end_time': time.time(),
> + })
> +
> + def write_in_database(self, tasks_information):
> + # This needs to be imported after we have configured the Django settings file
> + from webhob.orm.models import Tasks
> + for pid in tasks_information.keys():
> + task = tasks_information[pid]
> +
> + task_obj = Tasks.objects.create(uuid=task['uuid'],
> + order=task['order'],
> + outcome=task['outcome'],
> + recipe=task['recipe'],
> + task_name=task['task_name'],
> + elapsed_time=task['end_time'] - task['start_time'],
> + # The next lines are just for testing reasons
> + task_id=1,
> + path_to_sstate_obj='/home/calin',
> + source_url='/',
> + log_file='/',
> + work_directory='/',
> + script_type='python',
> + file_path='/',
> + line_number=55,
> + py_stack_trace='Testing traceback',
> + disk_io = 23.55633,
> + cpu_usage=43,
> + dependent_tasks='Task 1',
> + errors_no=0,
> + warnings_no=0,
> + error='None',
> + warning='None',
> + sstate_result='not_applicable')
> + task_obj.save()
> +
> +class BBProgress(progressbar.ProgressBar):
> + def __init__(self, msg, maxval):
> + self.msg = msg
> + widgets = [progressbar.Percentage(), ' ', progressbar.Bar(), ' ',
> + progressbar.ETA()]
> +
> + try:
> + self._resize_default = signal.getsignal(signal.SIGWINCH)
> + except:
> + self._resize_default = None
> + progressbar.ProgressBar.__init__(self, maxval, [self.msg + ":
> + "] + widgets)
> +
> + def _handle_resize(self, signum, frame):
> + progressbar.ProgressBar._handle_resize(self, signum, frame)
> + if self._resize_default:
> + self._resize_default(signum, frame)
> + def finish(self):
> + progressbar.ProgressBar.finish(self)
> + if self._resize_default:
> + signal.signal(signal.SIGWINCH, self._resize_default)
> +
> +class NonInteractiveProgress(object):
> + fobj = sys.stdout
> +
> + def __init__(self, msg, maxval):
> + self.msg = msg
> + self.maxval = maxval
> +
> + def start(self):
> + self.fobj.write("%s..." % self.msg)
> + self.fobj.flush()
> + return self
> +
> + def update(self, value):
> + pass
> +
> + def finish(self):
> + self.fobj.write("done.\n")
> + self.fobj.flush()
> +
> +def new_progress(msg, maxval):
> + if interactive:
> + return BBProgress(msg, maxval)
> + else:
> + return NonInteractiveProgress(msg, maxval)
> +
> +def pluralise(singular, plural, qty):
> + if(qty == 1):
> + return singular % qty
> + else:
> + return plural % qty
> +
> +
> +class InteractConsoleLogFilter(logging.Filter):
> + def __init__(self, tf, format):
> + self.tf = tf
> + self.format = format
> +
> + def filter(self, record):
> + if record.levelno == self.format.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")):
> + return False
> + self.tf.clearFooter()
> + return True
> +
> +class TerminalFilter(object):
> + columns = 80
> +
> + def sigwinch_handle(self, signum, frame):
> + self.columns = self.getTerminalColumns()
> + if self._sigwinch_default:
> + self._sigwinch_default(signum, frame)
> +
> + def getTerminalColumns(self):
> + def ioctl_GWINSZ(fd):
> + try:
> + cr = struct.unpack('hh', fcntl.ioctl(fd, self.termios.TIOCGWINSZ, '1234'))
> + except:
> + return None
> + return cr
> + cr = ioctl_GWINSZ(sys.stdout.fileno())
> + if not cr:
> + try:
> + fd = os.open(os.ctermid(), os.O_RDONLY)
> + cr = ioctl_GWINSZ(fd)
> + os.close(fd)
> + except:
> + pass
> + if not cr:
> + try:
> + cr = (env['LINES'], env['COLUMNS'])
> + except:
> + cr = (25, 80)
> + return cr[1]
> +
> + def __init__(self, main, helper, console, format):
> + self.main = main
> + self.helper = helper
> + self.cuu = None
> + self.stdinbackup = None
> + self.interactive = sys.stdout.isatty()
> + self.footer_present = False
> + self.lastpids = []
> +
> + if not self.interactive:
> + return
> +
> + try:
> + import curses
> + except ImportError:
> + sys.exit("FATAL: The knotty ui could not load the required
> + curses python module.")
> +
> + import termios
> + self.curses = curses
> + self.termios = termios
> + try:
> + fd = sys.stdin.fileno()
> + self.stdinbackup = termios.tcgetattr(fd)
> + new = copy.deepcopy(self.stdinbackup)
> + new[3] = new[3] & ~termios.ECHO
> + termios.tcsetattr(fd, termios.TCSADRAIN, new)
> + curses.setupterm()
> + if curses.tigetnum("colors") > 2:
> + format.enable_color()
> + self.ed = curses.tigetstr("ed")
> + if self.ed:
> + self.cuu = curses.tigetstr("cuu")
> + try:
> + self._sigwinch_default = signal.getsignal(signal.SIGWINCH)
> + signal.signal(signal.SIGWINCH, self.sigwinch_handle)
> + except:
> + pass
> + self.columns = self.getTerminalColumns()
> + except:
> + self.cuu = None
> + console.addFilter(InteractConsoleLogFilter(self, format))
> +
> + def clearFooter(self):
> + if self.footer_present:
> + lines = self.footer_present
> + sys.stdout.write(self.curses.tparm(self.cuu, lines))
> + sys.stdout.write(self.curses.tparm(self.ed))
> + self.footer_present = False
> +
> + def updateFooter(self):
> + if not self.cuu:
> + return
> + activetasks = self.helper.running_tasks
> + failedtasks = self.helper.failed_tasks
> + runningpids = self.helper.running_pids
> + if self.footer_present and (self.lastcount == self.helper.tasknumber_current) and (self.lastpids == runningpids):
> + return
> + if self.footer_present:
> + self.clearFooter()
> + if (not self.helper.tasknumber_total or self.helper.tasknumber_current == self.helper.tasknumber_total) and not len(activetasks):
> + return
> + tasks = []
> + for t in runningpids:
> + tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
> +
> + if self.main.shutdown:
> + content = "Waiting for %s running tasks to finish:" % len(activetasks)
> + elif not len(activetasks):
> + content = "No currently running tasks (%s of %s)" % (self.helper.tasknumber_current, self.helper.tasknumber_total)
> + else:
> + content = "Currently %s running tasks (%s of %s):" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total)
> + print(content)
> + lines = 1 + int(len(content) / (self.columns + 1))
> + for tasknum, task in enumerate(tasks):
> + content = "%s: %s" % (tasknum, task)
> + print(content)
> + lines = lines + 1 + int(len(content) / (self.columns + 1))
> + self.footer_present = lines
> + self.lastpids = runningpids[:]
> + self.lastcount = self.helper.tasknumber_current
> +
> + def finish(self):
> + if self.stdinbackup:
> + fd = sys.stdin.fileno()
> + self.termios.tcsetattr(fd, self.termios.TCSADRAIN,
> + self.stdinbackup)
> +
> +def main(server, eventHandler, params, tf = TerminalFilter):
> + print "This is running in DSI mode"
> +
> + # Generate an unique ID for this build
> + # TODO: for multiple build commands this might not work as expected
> + import uuid
> + uuid = sessionid = str(uuid.uuid4())
> +
> + # Get values of variables which control our output
> + includelogs, error = server.runCommand(["getVariable", "BBINCLUDELOGS"])
> + if error:
> + logger.error("Unable to get the value of BBINCLUDELOGS variable: %s" % error)
> + return 1
> + loglines, error = server.runCommand(["getVariable", "BBINCLUDELOGS_LINES"])
> + if error:
> + logger.error("Unable to get the value of BBINCLUDELOGS_LINES variable: %s" % error)
> + return 1
> + consolelogfile, error = server.runCommand(["getVariable", "BB_CONSOLELOG"])
> + if error:
> + logger.error("Unable to get the value of BB_CONSOLELOG variable: %s" % error)
> + return 1
> +
> + if sys.stdin.isatty() and sys.stdout.isatty():
> + log_exec_tty = True
> + else:
> + log_exec_tty = False
> +
> + helper = uihelper.BBUIHelper()
> +
> + console = logging.StreamHandler(sys.stdout)
> + format_str = "%(levelname)s: %(message)s"
> + format = bb.msg.BBLogFormatter(format_str)
> + bb.msg.addDefaultlogFilter(console)
> + console.setFormatter(format)
> + logger.addHandler(console)
> +
> + if consolelogfile and not params.options.show_environment:
> + bb.utils.mkdirhier(os.path.dirname(consolelogfile))
> + conlogformat = bb.msg.BBLogFormatter(format_str)
> + consolelog = logging.FileHandler(consolelogfile)
> + bb.msg.addDefaultlogFilter(consolelog)
> + consolelog.setFormatter(conlogformat)
> + logger.addHandler(consolelog)
> +
> + try:
> + params.updateFromServer(server)
> + cmdline = params.parseActions()
> + if not cmdline:
> + print("Nothing to do. Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information.")
> + return 1
> + if 'msg' in cmdline and cmdline['msg']:
> + logger.error(cmdline['msg'])
> + return 1
> +
> + ret, error = server.runCommand(cmdline['action'])
> + if error:
> + logger.error("Command '%s' failed: %s" % (cmdline, error))
> + return 1
> + elif ret != True:
> + logger.error("Command '%s' failed: returned %s" % (cmdline, ret))
> + return 1
> + except xmlrpclib.Fault as x:
> + logger.error("XMLRPC Fault getting commandline:\n %s" % x)
> + return 1
> +
> + parseprogress = None
> + cacheprogress = None
> + main.shutdown = 0
> + interrupted = False
> + return_value = 0
> + errors = 0
> + warnings = 0
> + taskfailures = []
> + wbhbhelper = WebHOBHelper()
> +
> + termfilter = tf(main, helper, console, format)
> +
> + while True:
> + try:
> + termfilter.updateFooter()
> + event = eventHandler.waitEvent(0.25)
> +
> + if event is None:
> + if main.shutdown > 1:
> + break
> + continue
> +
> + helper.eventHandler(event)
> +
> + if isinstance(event, bb.runqueue.runQueueExitWait):
> + if not main.shutdown:
> + main.shutdown = 1
> +
> + if isinstance(event, bb.build.TaskStarted):
> + wbhbhelper.store_started_task(event, uuid)
> +
> + if isinstance(event, (bb.build.TaskSucceeded, bb.build.TaskFailedSilent, bb.build.TaskFailed)):
> + wbhbhelper.update_stored_tasks(event, uuid)
> +
> + if isinstance(event, bb.event.LogExecTTY):
> + if log_exec_tty:
> + tries = event.retries
> + while tries:
> + print("Trying to run: %s" % event.prog)
> + if os.system(event.prog) == 0:
> + break
> + time.sleep(event.sleep_delay)
> + tries -= 1
> + if tries:
> + continue
> + logger.warn(event.msg)
> + continue
> +
> + if isinstance(event, logging.LogRecord):
> + if event.levelno >= format.ERROR:
> + errors = errors + 1
> + return_value = 1
> + elif event.levelno == format.WARNING:
> + warnings = warnings + 1
> + # For "normal" logging conditions, don't show note logs from tasks
> + # but do show them if the user has changed the default log level to
> + # include verbose/debug messages
> + if event.taskpid != 0 and event.levelno <= format.NOTE:
> + continue
> + logger.handle(event)
> + continue
> +
> + if isinstance(event, bb.build.TaskFailed):
> + return_value = 1
> + logfile = event.logfile
> + if logfile and os.path.exists(logfile):
> + termfilter.clearFooter()
> + bb.error("Logfile of failure stored in: %s" % logfile)
> + if includelogs and not event.errprinted:
> + print("Log data follows:")
> + f = open(logfile, "r")
> + lines = []
> + while True:
> + l = f.readline()
> + if l == '':
> + break
> + l = l.rstrip()
> + if loglines:
> + lines.append(' | %s' % l)
> + if len(lines) > int(loglines):
> + lines.pop(0)
> + else:
> + print('| %s' % l)
> + f.close()
> + if lines:
> + for line in lines:
> + print(line)
> + if isinstance(event, bb.build.TaskBase):
> + logger.info(event._message)
> + continue
> + if isinstance(event, bb.event.ParseStarted):
> + if event.total == 0:
> + continue
> + parseprogress = new_progress("Parsing recipes", event.total).start()
> + continue
> + if isinstance(event, bb.event.ParseProgress):
> + parseprogress.update(event.current)
> + continue
> + if isinstance(event, bb.event.ParseCompleted):
> + if not parseprogress:
> + continue
> +
> + parseprogress.finish()
> + print(("Parsing of %d .bb files complete (%d cached, %d parsed). %d targets, %d skipped, %d masked, %d errors."
> + % ( event.total, event.cached, event.parsed, event.virtuals, event.skipped, event.masked, event.errors)))
> + continue
> +
> + if isinstance(event, bb.event.CacheLoadStarted):
> + cacheprogress = new_progress("Loading cache", event.total).start()
> + continue
> + if isinstance(event, bb.event.CacheLoadProgress):
> + cacheprogress.update(event.current)
> + continue
> + if isinstance(event, bb.event.CacheLoadCompleted):
> + cacheprogress.finish()
> + print("Loaded %d entries from dependency cache." % event.num_entries)
> + continue
> +
> + if isinstance(event, bb.command.CommandFailed):
> + return_value = event.exitcode
> + errors = errors + 1
> + logger.error("Command execution failed: %s", event.error)
> + main.shutdown = 2
> + continue
> + if isinstance(event, bb.command.CommandExit):
> + if not return_value:
> + return_value = event.exitcode
> + continue
> + if isinstance(event, (bb.command.CommandCompleted, bb.cooker.CookerExit)):
> + main.shutdown = 2
> + continue
> + if isinstance(event, bb.event.MultipleProviders):
> + logger.info("multiple providers are available for %s%s (%s)", event._is_runtime and "runtime " or "",
> + event._item,
> + ", ".join(event._candidates))
> + logger.info("consider defining a PREFERRED_PROVIDER entry to match %s", event._item)
> + continue
> + if isinstance(event, bb.event.NoProvider):
> + return_value = 1
> + errors = errors + 1
> + if event._runtime:
> + r = "R"
> + else:
> + r = ""
> +
> + if event._dependees:
> + logger.error("Nothing %sPROVIDES '%s' (but %s %sDEPENDS on or otherwise requires it)", r, event._item, ", ".join(event._dependees), r)
> + else:
> + logger.error("Nothing %sPROVIDES '%s'", r, event._item)
> + if event._reasons:
> + for reason in event._reasons:
> + logger.error("%s", reason)
> + continue
> +
> + if isinstance(event, bb.runqueue.sceneQueueTaskStarted):
> + logger.info("Running setscene task %d of %d (%s)" % (event.stats.completed + event.stats.active + event.stats.failed + 1, event.stats.total, event.taskstring))
> + continue
> +
> + if isinstance(event, bb.runqueue.runQueueTaskStarted):
> + if event.noexec:
> + tasktype = 'noexec task'
> + else:
> + tasktype = 'task'
> + logger.info("Running %s %s of %s (ID: %s, %s)",
> + tasktype,
> + event.stats.completed + event.stats.active +
> + event.stats.failed + 1,
> + event.stats.total, event.taskid, event.taskstring)
> + continue
> +
> + if isinstance(event, bb.runqueue.runQueueTaskFailed):
> + taskfailures.append(event.taskstring)
> + logger.error("Task %s (%s) failed with exit code '%s'",
> + event.taskid, event.taskstring, event.exitcode)
> + continue
> +
> + if isinstance(event, bb.runqueue.sceneQueueTaskFailed):
> + logger.warn("Setscene task %s (%s) failed with exit code '%s' - real task will be run instead",
> + event.taskid, event.taskstring, event.exitcode)
> + continue
> +
> + if isinstance(event, bb.event.ConfigParsed):
> + # timestamp should be added for this
> + continue
> +
> + if isinstance(event, bb.event.RecipeParsed):
> + # timestamp should be added for this
> + continue
> +
> + if isinstance(event, bb.event.OperationStarted):
> + # timestamp should be added for this
> + continue
> +
> + if isinstance(event, bb.event.OperationCompleted):
> + # timestamp should be added for this
> + # signal a complete operation
> + # calculate timing
> + continue
> +
> + if isinstance(event, bb.event.DiskFull):
> + # trigger an error
> + continue
> +
> + # ignore
> + if isinstance(event, (bb.event.BuildBase,
> + bb.event.StampUpdate,
> + bb.event.RecipePreFinalise,
> + bb.runqueue.runQueueEvent,
> + bb.runqueue.runQueueExitWait,
> + bb.event.OperationProgress)):
> + continue
> +
> + logger.error("Unknown event: %s", event)
> +
> + except EnvironmentError as ioerror:
> + termfilter.clearFooter()
> + # ignore interrupted io
> + if ioerror.args[0] == 4:
> + pass
> + except KeyboardInterrupt:
> + termfilter.clearFooter()
> + if main.shutdown == 1:
> + print("\nSecond Keyboard Interrupt, stopping...\n")
> + _, error = server.runCommand(["stateStop"])
> + if error:
> + logger.error("Unable to cleanly stop: %s" % error)
> + if main.shutdown == 0:
> + print("\nKeyboard Interrupt, closing down...\n")
> + interrupted = True
> + _, error = server.runCommand(["stateShutdown"])
> + if error:
> + logger.error("Unable to cleanly shutdown: %s" % error)
> + main.shutdown = main.shutdown + 1
> + pass
> +
> + summary = ""
> + if taskfailures:
> + summary += pluralise("\nSummary: %s task failed:",
> + "\nSummary: %s tasks failed:", len(taskfailures))
> + for failure in taskfailures:
> + summary += "\n %s" % failure
> + if warnings:
> + summary += pluralise("\nSummary: There was %s WARNING message shown.",
> + "\nSummary: There were %s WARNING messages shown.", warnings)
> + if return_value:
> + summary += pluralise("\nSummary: There was %s ERROR message shown, returning a non-zero exit code.",
> + "\nSummary: There were %s ERROR messages shown, returning a non-zero exit code.", errors)
> + if summary:
> + print(summary)
> +
> + if interrupted:
> + print("Execution was interrupted, returning a non-zero exit code.")
> + if return_value == 0:
> + return_value = 1
> +
> + termfilter.finish()
> + wbhbhelper.write_in_database(wbhbhelper.tasks_information)
> +
> + return return_value
> +
> --
> 1.8.1.2
>
> _______________________________________________
> webhob mailing list
> webhob@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/webhob
^ permalink raw reply [flat|nested] 6+ messages in thread
* [Webhob] [PATCH 3/3] Create Data Store Interface (DSI) file
2013-06-12 15:11 [PATCH 1/3] Create main WEBHOB project Calin Dragomir
@ 2013-06-12 15:11 ` Calin Dragomir
0 siblings, 0 replies; 6+ messages in thread
From: Calin Dragomir @ 2013-06-12 15:11 UTC (permalink / raw)
To: webhob, bitbake-devel
This adds the first version of the DSI file. It uses the Knotty
code for now, but inserts task related information into the database
using the Django ORM
---
bitbake/lib/bb/ui/dsi.py | 609 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 609 insertions(+)
create mode 100644 bitbake/lib/bb/ui/dsi.py
diff --git a/bitbake/lib/bb/ui/dsi.py b/bitbake/lib/bb/ui/dsi.py
new file mode 100644
index 0000000..4bcd509
--- /dev/null
+++ b/bitbake/lib/bb/ui/dsi.py
@@ -0,0 +1,609 @@
+#
+# BitBake (No)TTY UI Implementation
+#
+# Handling output to TTYs or files (no TTY)
+#
+# Copyright (C) 2006-2012 Richard Purdie
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+from __future__ import division
+
+import os
+import sys
+import xmlrpclib
+import logging
+import progressbar
+import signal
+import bb.msg
+import time
+import fcntl
+import struct
+import copy
+import datetime
+
+from bb.ui import uihelper
+
+logger = logging.getLogger("BitBake")
+interactive = sys.stdout.isatty()
+
+class WebHOBHelper(object):
+
+ def __init__(self):
+ self.configure_django()
+ self.task_order = 0
+ self.tasks_information = {}
+
+ def configure_django(self):
+ import webhob.whbmain.settings as whb_django_settings
+ from django.core.management import setup_environ
+ setup_environ(whb_django_settings)
+ # Add webhob to sys path for importing modules
+ sys.path.append(os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), 'webhob'))
+
+ def store_started_task(self, event, uuid):
+ self.task_order += 1
+ self.tasks_information[event.pid] = {
+ 'uuid': uuid,
+ 'task_id': 1,
+ 'task_executed': True,
+ 'order': self.task_order,
+ 'recipe': event._package,
+ 'task_name': event._task,
+ 'start_time': time.time(),
+ }
+
+ def update_stored_tasks(self, event, uuid):
+ self.tasks_information[event.pid].update({
+ 'outcome': event.getDisplayName().lower(),
+ 'end_time': time.time(),
+ })
+
+ def write_in_database(self, tasks_information):
+ # This needs to be imported after we have configured the Django settings file
+ from webhob.orm.models import Tasks
+ for pid in tasks_information.keys():
+ task = tasks_information[pid]
+
+ task_obj = Tasks.objects.create(uuid=task['uuid'],
+ order=task['order'],
+ outcome=task['outcome'],
+ recipe=task['recipe'],
+ task_name=task['task_name'],
+ elapsed_time=task['end_time'] - task['start_time'],
+ # The next lines are just for testing reasons
+ task_id=1,
+ path_to_sstate_obj='/home/calin',
+ source_url='/',
+ log_file='/',
+ work_directory='/',
+ script_type='python',
+ file_path='/',
+ line_number=55,
+ py_stack_trace='Testing traceback',
+ disk_io = 23.55633,
+ cpu_usage=43,
+ dependent_tasks='Task 1',
+ errors_no=0,
+ warnings_no=0,
+ error='None',
+ warning='None',
+ sstate_result='not_applicable')
+ task_obj.save()
+
+class BBProgress(progressbar.ProgressBar):
+ def __init__(self, msg, maxval):
+ self.msg = msg
+ widgets = [progressbar.Percentage(), ' ', progressbar.Bar(), ' ',
+ progressbar.ETA()]
+
+ try:
+ self._resize_default = signal.getsignal(signal.SIGWINCH)
+ except:
+ self._resize_default = None
+ progressbar.ProgressBar.__init__(self, maxval, [self.msg + ": "] + widgets)
+
+ def _handle_resize(self, signum, frame):
+ progressbar.ProgressBar._handle_resize(self, signum, frame)
+ if self._resize_default:
+ self._resize_default(signum, frame)
+ def finish(self):
+ progressbar.ProgressBar.finish(self)
+ if self._resize_default:
+ signal.signal(signal.SIGWINCH, self._resize_default)
+
+class NonInteractiveProgress(object):
+ fobj = sys.stdout
+
+ def __init__(self, msg, maxval):
+ self.msg = msg
+ self.maxval = maxval
+
+ def start(self):
+ self.fobj.write("%s..." % self.msg)
+ self.fobj.flush()
+ return self
+
+ def update(self, value):
+ pass
+
+ def finish(self):
+ self.fobj.write("done.\n")
+ self.fobj.flush()
+
+def new_progress(msg, maxval):
+ if interactive:
+ return BBProgress(msg, maxval)
+ else:
+ return NonInteractiveProgress(msg, maxval)
+
+def pluralise(singular, plural, qty):
+ if(qty == 1):
+ return singular % qty
+ else:
+ return plural % qty
+
+
+class InteractConsoleLogFilter(logging.Filter):
+ def __init__(self, tf, format):
+ self.tf = tf
+ self.format = format
+
+ def filter(self, record):
+ if record.levelno == self.format.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")):
+ return False
+ self.tf.clearFooter()
+ return True
+
+class TerminalFilter(object):
+ columns = 80
+
+ def sigwinch_handle(self, signum, frame):
+ self.columns = self.getTerminalColumns()
+ if self._sigwinch_default:
+ self._sigwinch_default(signum, frame)
+
+ def getTerminalColumns(self):
+ def ioctl_GWINSZ(fd):
+ try:
+ cr = struct.unpack('hh', fcntl.ioctl(fd, self.termios.TIOCGWINSZ, '1234'))
+ except:
+ return None
+ return cr
+ cr = ioctl_GWINSZ(sys.stdout.fileno())
+ if not cr:
+ try:
+ fd = os.open(os.ctermid(), os.O_RDONLY)
+ cr = ioctl_GWINSZ(fd)
+ os.close(fd)
+ except:
+ pass
+ if not cr:
+ try:
+ cr = (env['LINES'], env['COLUMNS'])
+ except:
+ cr = (25, 80)
+ return cr[1]
+
+ def __init__(self, main, helper, console, format):
+ self.main = main
+ self.helper = helper
+ self.cuu = None
+ self.stdinbackup = None
+ self.interactive = sys.stdout.isatty()
+ self.footer_present = False
+ self.lastpids = []
+
+ if not self.interactive:
+ return
+
+ try:
+ import curses
+ except ImportError:
+ sys.exit("FATAL: The knotty ui could not load the required curses python module.")
+
+ import termios
+ self.curses = curses
+ self.termios = termios
+ try:
+ fd = sys.stdin.fileno()
+ self.stdinbackup = termios.tcgetattr(fd)
+ new = copy.deepcopy(self.stdinbackup)
+ new[3] = new[3] & ~termios.ECHO
+ termios.tcsetattr(fd, termios.TCSADRAIN, new)
+ curses.setupterm()
+ if curses.tigetnum("colors") > 2:
+ format.enable_color()
+ self.ed = curses.tigetstr("ed")
+ if self.ed:
+ self.cuu = curses.tigetstr("cuu")
+ try:
+ self._sigwinch_default = signal.getsignal(signal.SIGWINCH)
+ signal.signal(signal.SIGWINCH, self.sigwinch_handle)
+ except:
+ pass
+ self.columns = self.getTerminalColumns()
+ except:
+ self.cuu = None
+ console.addFilter(InteractConsoleLogFilter(self, format))
+
+ def clearFooter(self):
+ if self.footer_present:
+ lines = self.footer_present
+ sys.stdout.write(self.curses.tparm(self.cuu, lines))
+ sys.stdout.write(self.curses.tparm(self.ed))
+ self.footer_present = False
+
+ def updateFooter(self):
+ if not self.cuu:
+ return
+ activetasks = self.helper.running_tasks
+ failedtasks = self.helper.failed_tasks
+ runningpids = self.helper.running_pids
+ if self.footer_present and (self.lastcount == self.helper.tasknumber_current) and (self.lastpids == runningpids):
+ return
+ if self.footer_present:
+ self.clearFooter()
+ if (not self.helper.tasknumber_total or self.helper.tasknumber_current == self.helper.tasknumber_total) and not len(activetasks):
+ return
+ tasks = []
+ for t in runningpids:
+ tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
+
+ if self.main.shutdown:
+ content = "Waiting for %s running tasks to finish:" % len(activetasks)
+ elif not len(activetasks):
+ content = "No currently running tasks (%s of %s)" % (self.helper.tasknumber_current, self.helper.tasknumber_total)
+ else:
+ content = "Currently %s running tasks (%s of %s):" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total)
+ print(content)
+ lines = 1 + int(len(content) / (self.columns + 1))
+ for tasknum, task in enumerate(tasks):
+ content = "%s: %s" % (tasknum, task)
+ print(content)
+ lines = lines + 1 + int(len(content) / (self.columns + 1))
+ self.footer_present = lines
+ self.lastpids = runningpids[:]
+ self.lastcount = self.helper.tasknumber_current
+
+ def finish(self):
+ if self.stdinbackup:
+ fd = sys.stdin.fileno()
+ self.termios.tcsetattr(fd, self.termios.TCSADRAIN, self.stdinbackup)
+
+def main(server, eventHandler, params, tf = TerminalFilter):
+ print "This is running in DSI mode"
+
+ # Generate an unique ID for this build
+ # TODO: for multiple build commands this might not work as expected
+ import uuid
+ uuid = sessionid = str(uuid.uuid4())
+
+ # Get values of variables which control our output
+ includelogs, error = server.runCommand(["getVariable", "BBINCLUDELOGS"])
+ if error:
+ logger.error("Unable to get the value of BBINCLUDELOGS variable: %s" % error)
+ return 1
+ loglines, error = server.runCommand(["getVariable", "BBINCLUDELOGS_LINES"])
+ if error:
+ logger.error("Unable to get the value of BBINCLUDELOGS_LINES variable: %s" % error)
+ return 1
+ consolelogfile, error = server.runCommand(["getVariable", "BB_CONSOLELOG"])
+ if error:
+ logger.error("Unable to get the value of BB_CONSOLELOG variable: %s" % error)
+ return 1
+
+ if sys.stdin.isatty() and sys.stdout.isatty():
+ log_exec_tty = True
+ else:
+ log_exec_tty = False
+
+ helper = uihelper.BBUIHelper()
+
+ console = logging.StreamHandler(sys.stdout)
+ format_str = "%(levelname)s: %(message)s"
+ format = bb.msg.BBLogFormatter(format_str)
+ bb.msg.addDefaultlogFilter(console)
+ console.setFormatter(format)
+ logger.addHandler(console)
+
+ if consolelogfile and not params.options.show_environment:
+ bb.utils.mkdirhier(os.path.dirname(consolelogfile))
+ conlogformat = bb.msg.BBLogFormatter(format_str)
+ consolelog = logging.FileHandler(consolelogfile)
+ bb.msg.addDefaultlogFilter(consolelog)
+ consolelog.setFormatter(conlogformat)
+ logger.addHandler(consolelog)
+
+ try:
+ params.updateFromServer(server)
+ cmdline = params.parseActions()
+ if not cmdline:
+ print("Nothing to do. Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information.")
+ return 1
+ if 'msg' in cmdline and cmdline['msg']:
+ logger.error(cmdline['msg'])
+ return 1
+
+ ret, error = server.runCommand(cmdline['action'])
+ if error:
+ logger.error("Command '%s' failed: %s" % (cmdline, error))
+ return 1
+ elif ret != True:
+ logger.error("Command '%s' failed: returned %s" % (cmdline, ret))
+ return 1
+ except xmlrpclib.Fault as x:
+ logger.error("XMLRPC Fault getting commandline:\n %s" % x)
+ return 1
+
+ parseprogress = None
+ cacheprogress = None
+ main.shutdown = 0
+ interrupted = False
+ return_value = 0
+ errors = 0
+ warnings = 0
+ taskfailures = []
+ wbhbhelper = WebHOBHelper()
+
+ termfilter = tf(main, helper, console, format)
+
+ while True:
+ try:
+ termfilter.updateFooter()
+ event = eventHandler.waitEvent(0.25)
+
+ if event is None:
+ if main.shutdown > 1:
+ break
+ continue
+
+ helper.eventHandler(event)
+
+ if isinstance(event, bb.runqueue.runQueueExitWait):
+ if not main.shutdown:
+ main.shutdown = 1
+
+ if isinstance(event, bb.build.TaskStarted):
+ wbhbhelper.store_started_task(event, uuid)
+
+ if isinstance(event, (bb.build.TaskSucceeded, bb.build.TaskFailedSilent, bb.build.TaskFailed)):
+ wbhbhelper.update_stored_tasks(event, uuid)
+
+ if isinstance(event, bb.event.LogExecTTY):
+ if log_exec_tty:
+ tries = event.retries
+ while tries:
+ print("Trying to run: %s" % event.prog)
+ if os.system(event.prog) == 0:
+ break
+ time.sleep(event.sleep_delay)
+ tries -= 1
+ if tries:
+ continue
+ logger.warn(event.msg)
+ continue
+
+ if isinstance(event, logging.LogRecord):
+ if event.levelno >= format.ERROR:
+ errors = errors + 1
+ return_value = 1
+ elif event.levelno == format.WARNING:
+ warnings = warnings + 1
+ # For "normal" logging conditions, don't show note logs from tasks
+ # but do show them if the user has changed the default log level to
+ # include verbose/debug messages
+ if event.taskpid != 0 and event.levelno <= format.NOTE:
+ continue
+ logger.handle(event)
+ continue
+
+ if isinstance(event, bb.build.TaskFailed):
+ return_value = 1
+ logfile = event.logfile
+ if logfile and os.path.exists(logfile):
+ termfilter.clearFooter()
+ bb.error("Logfile of failure stored in: %s" % logfile)
+ if includelogs and not event.errprinted:
+ print("Log data follows:")
+ f = open(logfile, "r")
+ lines = []
+ while True:
+ l = f.readline()
+ if l == '':
+ break
+ l = l.rstrip()
+ if loglines:
+ lines.append(' | %s' % l)
+ if len(lines) > int(loglines):
+ lines.pop(0)
+ else:
+ print('| %s' % l)
+ f.close()
+ if lines:
+ for line in lines:
+ print(line)
+ if isinstance(event, bb.build.TaskBase):
+ logger.info(event._message)
+ continue
+ if isinstance(event, bb.event.ParseStarted):
+ if event.total == 0:
+ continue
+ parseprogress = new_progress("Parsing recipes", event.total).start()
+ continue
+ if isinstance(event, bb.event.ParseProgress):
+ parseprogress.update(event.current)
+ continue
+ if isinstance(event, bb.event.ParseCompleted):
+ if not parseprogress:
+ continue
+
+ parseprogress.finish()
+ print(("Parsing of %d .bb files complete (%d cached, %d parsed). %d targets, %d skipped, %d masked, %d errors."
+ % ( event.total, event.cached, event.parsed, event.virtuals, event.skipped, event.masked, event.errors)))
+ continue
+
+ if isinstance(event, bb.event.CacheLoadStarted):
+ cacheprogress = new_progress("Loading cache", event.total).start()
+ continue
+ if isinstance(event, bb.event.CacheLoadProgress):
+ cacheprogress.update(event.current)
+ continue
+ if isinstance(event, bb.event.CacheLoadCompleted):
+ cacheprogress.finish()
+ print("Loaded %d entries from dependency cache." % event.num_entries)
+ continue
+
+ if isinstance(event, bb.command.CommandFailed):
+ return_value = event.exitcode
+ errors = errors + 1
+ logger.error("Command execution failed: %s", event.error)
+ main.shutdown = 2
+ continue
+ if isinstance(event, bb.command.CommandExit):
+ if not return_value:
+ return_value = event.exitcode
+ continue
+ if isinstance(event, (bb.command.CommandCompleted, bb.cooker.CookerExit)):
+ main.shutdown = 2
+ continue
+ if isinstance(event, bb.event.MultipleProviders):
+ logger.info("multiple providers are available for %s%s (%s)", event._is_runtime and "runtime " or "",
+ event._item,
+ ", ".join(event._candidates))
+ logger.info("consider defining a PREFERRED_PROVIDER entry to match %s", event._item)
+ continue
+ if isinstance(event, bb.event.NoProvider):
+ return_value = 1
+ errors = errors + 1
+ if event._runtime:
+ r = "R"
+ else:
+ r = ""
+
+ if event._dependees:
+ logger.error("Nothing %sPROVIDES '%s' (but %s %sDEPENDS on or otherwise requires it)", r, event._item, ", ".join(event._dependees), r)
+ else:
+ logger.error("Nothing %sPROVIDES '%s'", r, event._item)
+ if event._reasons:
+ for reason in event._reasons:
+ logger.error("%s", reason)
+ continue
+
+ if isinstance(event, bb.runqueue.sceneQueueTaskStarted):
+ logger.info("Running setscene task %d of %d (%s)" % (event.stats.completed + event.stats.active + event.stats.failed + 1, event.stats.total, event.taskstring))
+ continue
+
+ if isinstance(event, bb.runqueue.runQueueTaskStarted):
+ if event.noexec:
+ tasktype = 'noexec task'
+ else:
+ tasktype = 'task'
+ logger.info("Running %s %s of %s (ID: %s, %s)",
+ tasktype,
+ event.stats.completed + event.stats.active +
+ event.stats.failed + 1,
+ event.stats.total, event.taskid, event.taskstring)
+ continue
+
+ if isinstance(event, bb.runqueue.runQueueTaskFailed):
+ taskfailures.append(event.taskstring)
+ logger.error("Task %s (%s) failed with exit code '%s'",
+ event.taskid, event.taskstring, event.exitcode)
+ continue
+
+ if isinstance(event, bb.runqueue.sceneQueueTaskFailed):
+ logger.warn("Setscene task %s (%s) failed with exit code '%s' - real task will be run instead",
+ event.taskid, event.taskstring, event.exitcode)
+ continue
+
+ if isinstance(event, bb.event.ConfigParsed):
+ # timestamp should be added for this
+ continue
+
+ if isinstance(event, bb.event.RecipeParsed):
+ # timestamp should be added for this
+ continue
+
+ if isinstance(event, bb.event.OperationStarted):
+ # timestamp should be added for this
+ continue
+
+ if isinstance(event, bb.event.OperationCompleted):
+ # timestamp should be added for this
+ # signal a complete operation
+ # calculate timing
+ continue
+
+ if isinstance(event, bb.event.DiskFull):
+ # trigger an error
+ continue
+
+ # ignore
+ if isinstance(event, (bb.event.BuildBase,
+ bb.event.StampUpdate,
+ bb.event.RecipePreFinalise,
+ bb.runqueue.runQueueEvent,
+ bb.runqueue.runQueueExitWait,
+ bb.event.OperationProgress)):
+ continue
+
+ logger.error("Unknown event: %s", event)
+
+ except EnvironmentError as ioerror:
+ termfilter.clearFooter()
+ # ignore interrupted io
+ if ioerror.args[0] == 4:
+ pass
+ except KeyboardInterrupt:
+ termfilter.clearFooter()
+ if main.shutdown == 1:
+ print("\nSecond Keyboard Interrupt, stopping...\n")
+ _, error = server.runCommand(["stateStop"])
+ if error:
+ logger.error("Unable to cleanly stop: %s" % error)
+ if main.shutdown == 0:
+ print("\nKeyboard Interrupt, closing down...\n")
+ interrupted = True
+ _, error = server.runCommand(["stateShutdown"])
+ if error:
+ logger.error("Unable to cleanly shutdown: %s" % error)
+ main.shutdown = main.shutdown + 1
+ pass
+
+ summary = ""
+ if taskfailures:
+ summary += pluralise("\nSummary: %s task failed:",
+ "\nSummary: %s tasks failed:", len(taskfailures))
+ for failure in taskfailures:
+ summary += "\n %s" % failure
+ if warnings:
+ summary += pluralise("\nSummary: There was %s WARNING message shown.",
+ "\nSummary: There were %s WARNING messages shown.", warnings)
+ if return_value:
+ summary += pluralise("\nSummary: There was %s ERROR message shown, returning a non-zero exit code.",
+ "\nSummary: There were %s ERROR messages shown, returning a non-zero exit code.", errors)
+ if summary:
+ print(summary)
+
+ if interrupted:
+ print("Execution was interrupted, returning a non-zero exit code.")
+ if return_value == 0:
+ return_value = 1
+
+ termfilter.finish()
+ wbhbhelper.write_in_database(wbhbhelper.tasks_information)
+
+ return return_value
+
--
1.8.1.2
^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-06-12 15:10 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-06-12 9:30 [Webhob] [PATCH 1/3] Rename main WebHob project Calin Dragomir
2013-06-12 9:30 ` [Webhob] [PATCH 2/3] Create Tasks Django model Calin Dragomir
2013-06-12 9:30 ` [Webhob] [PATCH 3/3] Create Data Store Interface (DSI) file Calin Dragomir
2013-06-12 9:53 ` Iorga, Cristian
2013-06-12 10:46 ` Calin Dragomir
-- strict thread matches above, loose matches on Subject: below --
2013-06-12 15:11 [PATCH 1/3] Create main WEBHOB project Calin Dragomir
2013-06-12 15:11 ` [Webhob] [PATCH 3/3] Create Data Store Interface (DSI) file Calin Dragomir
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.