* [PATCH 1/3] Create main WEBHOB project
@ 2013-06-12 15:11 ` Calin Dragomir
0 siblings, 0 replies; 7+ messages in thread
From: Calin Dragomir @ 2013-06-12 15:11 UTC (permalink / raw)
To: webhob, bitbake-devel
This creates a basic Django project where the WEBHOB code will live.
The main project had to be renamed from webhob to whbmain in order to
avoid any import issues that might occur.
---
bitbake/lib/webhob/__init__.py | 0
bitbake/lib/webhob/manage.py | 10 +++
bitbake/lib/webhob/whbmain/__init__.py | 0
bitbake/lib/webhob/whbmain/settings.py | 156 +++++++++++++++++++++++++++++++++
bitbake/lib/webhob/whbmain/urls.py | 17 ++++
bitbake/lib/webhob/whbmain/wsgi.py | 32 +++++++
6 files changed, 215 insertions(+)
create mode 100644 bitbake/lib/webhob/__init__.py
create mode 100755 bitbake/lib/webhob/manage.py
create mode 100644 bitbake/lib/webhob/whbmain/__init__.py
create mode 100644 bitbake/lib/webhob/whbmain/settings.py
create mode 100644 bitbake/lib/webhob/whbmain/urls.py
create mode 100644 bitbake/lib/webhob/whbmain/wsgi.py
diff --git a/bitbake/lib/webhob/__init__.py b/bitbake/lib/webhob/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/bitbake/lib/webhob/manage.py b/bitbake/lib/webhob/manage.py
new file mode 100755
index 0000000..1b2c1a7
--- /dev/null
+++ b/bitbake/lib/webhob/manage.py
@@ -0,0 +1,10 @@
+#!/usr/bin/env python
+import os
+import sys
+
+if __name__ == "__main__":
+ os.environ.setdefault("DJANGO_SETTINGS_MODULE", "whbmain.settings")
+
+ from django.core.management import execute_from_command_line
+
+ execute_from_command_line(sys.argv)
diff --git a/bitbake/lib/webhob/whbmain/__init__.py b/bitbake/lib/webhob/whbmain/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/bitbake/lib/webhob/whbmain/settings.py b/bitbake/lib/webhob/whbmain/settings.py
new file mode 100644
index 0000000..bc59543
--- /dev/null
+++ b/bitbake/lib/webhob/whbmain/settings.py
@@ -0,0 +1,156 @@
+# Django settings for webhob project.
+
+DEBUG = True
+TEMPLATE_DEBUG = DEBUG
+
+ADMINS = (
+ # ('Your Name', 'your_email@example.com'),
+)
+
+MANAGERS = ADMINS
+
+DATABASES = {
+ 'default': {
+ 'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
+ 'NAME': 'wbhob', # Or path to database file if using sqlite3.
+ # The following settings are not used with sqlite3:
+ 'USER': 'root',
+ 'PASSWORD': '',
+ 'HOST': '127.0.0.1', # Empty for localhost through domain sockets or '127.0.0.1' for localhost through TCP.
+ 'PORT': '3306', # Set to empty string for default.
+ }
+}
+
+# Hosts/domain names that are valid for this site; required if DEBUG is False
+# See https://docs.djangoproject.com/en/1.5/ref/settings/#allowed-hosts
+ALLOWED_HOSTS = []
+
+# Local time zone for this installation. Choices can be found here:
+# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
+# although not all choices may be available on all operating systems.
+# In a Windows environment this must be set to your system time zone.
+TIME_ZONE = 'America/Chicago'
+
+# Language code for this installation. All choices can be found here:
+# http://www.i18nguy.com/unicode/language-identifiers.html
+LANGUAGE_CODE = 'en-us'
+
+SITE_ID = 1
+
+# If you set this to False, Django will make some optimizations so as not
+# to load the internationalization machinery.
+USE_I18N = True
+
+# If you set this to False, Django will not format dates, numbers and
+# calendars according to the current locale.
+USE_L10N = True
+
+# If you set this to False, Django will not use timezone-aware datetimes.
+USE_TZ = True
+
+# Absolute filesystem path to the directory that will hold user-uploaded files.
+# Example: "/var/www/example.com/media/"
+MEDIA_ROOT = ''
+
+# URL that handles the media served from MEDIA_ROOT. Make sure to use a
+# trailing slash.
+# Examples: "http://example.com/media/", "http://media.example.com/"
+MEDIA_URL = ''
+
+# Absolute path to the directory static files should be collected to.
+# Don't put anything in this directory yourself; store your static files
+# in apps' "static/" subdirectories and in STATICFILES_DIRS.
+# Example: "/var/www/example.com/static/"
+STATIC_ROOT = ''
+
+# URL prefix for static files.
+# Example: "http://example.com/static/", "http://static.example.com/"
+STATIC_URL = '/static/'
+
+# Additional locations of static files
+STATICFILES_DIRS = (
+ # Put strings here, like "/home/html/static" or "C:/www/django/static".
+ # Always use forward slashes, even on Windows.
+ # Don't forget to use absolute paths, not relative paths.
+)
+
+# List of finder classes that know how to find static files in
+# various locations.
+STATICFILES_FINDERS = (
+ 'django.contrib.staticfiles.finders.FileSystemFinder',
+ 'django.contrib.staticfiles.finders.AppDirectoriesFinder',
+# 'django.contrib.staticfiles.finders.DefaultStorageFinder',
+)
+
+# Make this unique, and don't share it with anybody.
+SECRET_KEY = 'fy@5+5w9zn5wa0vsegn1c8p_6w&pa_ms-4t0cna1vle79nzlkn'
+
+# List of callables that know how to import templates from various sources.
+TEMPLATE_LOADERS = (
+ 'django.template.loaders.filesystem.Loader',
+ 'django.template.loaders.app_directories.Loader',
+# 'django.template.loaders.eggs.Loader',
+)
+
+MIDDLEWARE_CLASSES = (
+ 'django.middleware.common.CommonMiddleware',
+ 'django.contrib.sessions.middleware.SessionMiddleware',
+ 'django.middleware.csrf.CsrfViewMiddleware',
+ 'django.contrib.auth.middleware.AuthenticationMiddleware',
+ 'django.contrib.messages.middleware.MessageMiddleware',
+ # Uncomment the next line for simple clickjacking protection:
+ # 'django.middleware.clickjacking.XFrameOptionsMiddleware',
+)
+
+ROOT_URLCONF = 'whbmain.urls'
+
+# Python dotted path to the WSGI application used by Django's runserver.
+WSGI_APPLICATION = 'whbmain.wsgi.application'
+
+TEMPLATE_DIRS = (
+ # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
+ # Always use forward slashes, even on Windows.
+ # Don't forget to use absolute paths, not relative paths.
+)
+
+INSTALLED_APPS = (
+ #'django.contrib.auth',
+ #'django.contrib.contenttypes',
+ #'django.contrib.sessions',
+ #'django.contrib.sites',
+ #'django.contrib.messages',
+ #'django.contrib.staticfiles',
+ # Uncomment the next line to enable the admin:
+ # 'django.contrib.admin',
+ # Uncomment the next line to enable admin documentation:
+ # 'django.contrib.admindocs',
+)
+
+# A sample logging configuration. The only tangible logging
+# performed by this configuration is to send an email to
+# the site admins on every HTTP 500 error when DEBUG=False.
+# See http://docs.djangoproject.com/en/dev/topics/logging for
+# more details on how to customize your logging configuration.
+LOGGING = {
+ 'version': 1,
+ 'disable_existing_loggers': False,
+ 'filters': {
+ 'require_debug_false': {
+ '()': 'django.utils.log.RequireDebugFalse'
+ }
+ },
+ 'handlers': {
+ 'mail_admins': {
+ 'level': 'ERROR',
+ 'filters': ['require_debug_false'],
+ 'class': 'django.utils.log.AdminEmailHandler'
+ }
+ },
+ 'loggers': {
+ 'django.request': {
+ 'handlers': ['mail_admins'],
+ 'level': 'ERROR',
+ 'propagate': True,
+ },
+ }
+}
diff --git a/bitbake/lib/webhob/whbmain/urls.py b/bitbake/lib/webhob/whbmain/urls.py
new file mode 100644
index 0000000..04e0352
--- /dev/null
+++ b/bitbake/lib/webhob/whbmain/urls.py
@@ -0,0 +1,17 @@
+from django.conf.urls import patterns, include, url
+
+# Uncomment the next two lines to enable the admin:
+# from django.contrib import admin
+# admin.autodiscover()
+
+urlpatterns = patterns('',
+ # Examples:
+ # url(r'^$', 'webhob.views.home', name='home'),
+ # url(r'^webhob/', include('webhob.foo.urls')),
+
+ # Uncomment the admin/doc line below to enable admin documentation:
+ # url(r'^admin/doc/', include('django.contrib.admindocs.urls')),
+
+ # Uncomment the next line to enable the admin:
+ # url(r'^admin/', include(admin.site.urls)),
+)
diff --git a/bitbake/lib/webhob/whbmain/wsgi.py b/bitbake/lib/webhob/whbmain/wsgi.py
new file mode 100644
index 0000000..9595eda
--- /dev/null
+++ b/bitbake/lib/webhob/whbmain/wsgi.py
@@ -0,0 +1,32 @@
+"""
+WSGI config for webhob project.
+
+This module contains the WSGI application used by Django's development server
+and any production WSGI deployments. It should expose a module-level variable
+named ``application``. Django's ``runserver`` and ``runfcgi`` commands discover
+this application via the ``WSGI_APPLICATION`` setting.
+
+Usually you will have the standard Django WSGI application here, but it also
+might make sense to replace the whole Django WSGI application with a custom one
+that later delegates to the Django one. For example, you could introduce WSGI
+middleware here, or combine a Django application with an application of another
+framework.
+
+"""
+import os
+
+# We defer to a DJANGO_SETTINGS_MODULE already in the environment. This breaks
+# if running multiple sites in the same mod_wsgi process. To fix this, use
+# mod_wsgi daemon mode with each site in its own daemon process, or use
+# os.environ["DJANGO_SETTINGS_MODULE"] = "webhob.settings"
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "wbhmain.settings")
+
+# This application object is used by any WSGI server configured to use this
+# file. This includes Django's development server, if the WSGI_APPLICATION
+# setting points here.
+from django.core.wsgi import get_wsgi_application
+application = get_wsgi_application()
+
+# Apply WSGI middleware here.
+# from helloworld.wsgi import HelloWorldApplication
+# application = HelloWorldApplication(application)
--
1.8.1.2
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [Webhob] [PATCH 1/3] Create main WEBHOB project
@ 2013-06-12 15:11 ` Calin Dragomir
0 siblings, 0 replies; 7+ messages in thread
From: Calin Dragomir @ 2013-06-12 15:11 UTC (permalink / raw)
To: webhob, bitbake-devel
This creates a basic Django project where the WEBHOB code will live.
The main project had to be renamed from webhob to whbmain in order to
avoid any import issues that might occur.
---
bitbake/lib/webhob/__init__.py | 0
bitbake/lib/webhob/manage.py | 10 +++
bitbake/lib/webhob/whbmain/__init__.py | 0
bitbake/lib/webhob/whbmain/settings.py | 156 +++++++++++++++++++++++++++++++++
bitbake/lib/webhob/whbmain/urls.py | 17 ++++
bitbake/lib/webhob/whbmain/wsgi.py | 32 +++++++
6 files changed, 215 insertions(+)
create mode 100644 bitbake/lib/webhob/__init__.py
create mode 100755 bitbake/lib/webhob/manage.py
create mode 100644 bitbake/lib/webhob/whbmain/__init__.py
create mode 100644 bitbake/lib/webhob/whbmain/settings.py
create mode 100644 bitbake/lib/webhob/whbmain/urls.py
create mode 100644 bitbake/lib/webhob/whbmain/wsgi.py
diff --git a/bitbake/lib/webhob/__init__.py b/bitbake/lib/webhob/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/bitbake/lib/webhob/manage.py b/bitbake/lib/webhob/manage.py
new file mode 100755
index 0000000..1b2c1a7
--- /dev/null
+++ b/bitbake/lib/webhob/manage.py
@@ -0,0 +1,10 @@
+#!/usr/bin/env python
+import os
+import sys
+
+if __name__ == "__main__":
+ os.environ.setdefault("DJANGO_SETTINGS_MODULE", "whbmain.settings")
+
+ from django.core.management import execute_from_command_line
+
+ execute_from_command_line(sys.argv)
diff --git a/bitbake/lib/webhob/whbmain/__init__.py b/bitbake/lib/webhob/whbmain/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/bitbake/lib/webhob/whbmain/settings.py b/bitbake/lib/webhob/whbmain/settings.py
new file mode 100644
index 0000000..bc59543
--- /dev/null
+++ b/bitbake/lib/webhob/whbmain/settings.py
@@ -0,0 +1,156 @@
+# Django settings for webhob project.
+
+DEBUG = True
+TEMPLATE_DEBUG = DEBUG
+
+ADMINS = (
+ # ('Your Name', 'your_email@example.com'),
+)
+
+MANAGERS = ADMINS
+
+DATABASES = {
+ 'default': {
+ 'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
+ 'NAME': 'wbhob', # Or path to database file if using sqlite3.
+ # The following settings are not used with sqlite3:
+ 'USER': 'root',
+ 'PASSWORD': '',
+ 'HOST': '127.0.0.1', # Empty for localhost through domain sockets or '127.0.0.1' for localhost through TCP.
+ 'PORT': '3306', # Set to empty string for default.
+ }
+}
+
+# Hosts/domain names that are valid for this site; required if DEBUG is False
+# See https://docs.djangoproject.com/en/1.5/ref/settings/#allowed-hosts
+ALLOWED_HOSTS = []
+
+# Local time zone for this installation. Choices can be found here:
+# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
+# although not all choices may be available on all operating systems.
+# In a Windows environment this must be set to your system time zone.
+TIME_ZONE = 'America/Chicago'
+
+# Language code for this installation. All choices can be found here:
+# http://www.i18nguy.com/unicode/language-identifiers.html
+LANGUAGE_CODE = 'en-us'
+
+SITE_ID = 1
+
+# If you set this to False, Django will make some optimizations so as not
+# to load the internationalization machinery.
+USE_I18N = True
+
+# If you set this to False, Django will not format dates, numbers and
+# calendars according to the current locale.
+USE_L10N = True
+
+# If you set this to False, Django will not use timezone-aware datetimes.
+USE_TZ = True
+
+# Absolute filesystem path to the directory that will hold user-uploaded files.
+# Example: "/var/www/example.com/media/"
+MEDIA_ROOT = ''
+
+# URL that handles the media served from MEDIA_ROOT. Make sure to use a
+# trailing slash.
+# Examples: "http://example.com/media/", "http://media.example.com/"
+MEDIA_URL = ''
+
+# Absolute path to the directory static files should be collected to.
+# Don't put anything in this directory yourself; store your static files
+# in apps' "static/" subdirectories and in STATICFILES_DIRS.
+# Example: "/var/www/example.com/static/"
+STATIC_ROOT = ''
+
+# URL prefix for static files.
+# Example: "http://example.com/static/", "http://static.example.com/"
+STATIC_URL = '/static/'
+
+# Additional locations of static files
+STATICFILES_DIRS = (
+ # Put strings here, like "/home/html/static" or "C:/www/django/static".
+ # Always use forward slashes, even on Windows.
+ # Don't forget to use absolute paths, not relative paths.
+)
+
+# List of finder classes that know how to find static files in
+# various locations.
+STATICFILES_FINDERS = (
+ 'django.contrib.staticfiles.finders.FileSystemFinder',
+ 'django.contrib.staticfiles.finders.AppDirectoriesFinder',
+# 'django.contrib.staticfiles.finders.DefaultStorageFinder',
+)
+
+# Make this unique, and don't share it with anybody.
+SECRET_KEY = 'fy@5+5w9zn5wa0vsegn1c8p_6w&pa_ms-4t0cna1vle79nzlkn'
+
+# List of callables that know how to import templates from various sources.
+TEMPLATE_LOADERS = (
+ 'django.template.loaders.filesystem.Loader',
+ 'django.template.loaders.app_directories.Loader',
+# 'django.template.loaders.eggs.Loader',
+)
+
+MIDDLEWARE_CLASSES = (
+ 'django.middleware.common.CommonMiddleware',
+ 'django.contrib.sessions.middleware.SessionMiddleware',
+ 'django.middleware.csrf.CsrfViewMiddleware',
+ 'django.contrib.auth.middleware.AuthenticationMiddleware',
+ 'django.contrib.messages.middleware.MessageMiddleware',
+ # Uncomment the next line for simple clickjacking protection:
+ # 'django.middleware.clickjacking.XFrameOptionsMiddleware',
+)
+
+ROOT_URLCONF = 'whbmain.urls'
+
+# Python dotted path to the WSGI application used by Django's runserver.
+WSGI_APPLICATION = 'whbmain.wsgi.application'
+
+TEMPLATE_DIRS = (
+ # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
+ # Always use forward slashes, even on Windows.
+ # Don't forget to use absolute paths, not relative paths.
+)
+
+INSTALLED_APPS = (
+ #'django.contrib.auth',
+ #'django.contrib.contenttypes',
+ #'django.contrib.sessions',
+ #'django.contrib.sites',
+ #'django.contrib.messages',
+ #'django.contrib.staticfiles',
+ # Uncomment the next line to enable the admin:
+ # 'django.contrib.admin',
+ # Uncomment the next line to enable admin documentation:
+ # 'django.contrib.admindocs',
+)
+
+# A sample logging configuration. The only tangible logging
+# performed by this configuration is to send an email to
+# the site admins on every HTTP 500 error when DEBUG=False.
+# See http://docs.djangoproject.com/en/dev/topics/logging for
+# more details on how to customize your logging configuration.
+LOGGING = {
+ 'version': 1,
+ 'disable_existing_loggers': False,
+ 'filters': {
+ 'require_debug_false': {
+ '()': 'django.utils.log.RequireDebugFalse'
+ }
+ },
+ 'handlers': {
+ 'mail_admins': {
+ 'level': 'ERROR',
+ 'filters': ['require_debug_false'],
+ 'class': 'django.utils.log.AdminEmailHandler'
+ }
+ },
+ 'loggers': {
+ 'django.request': {
+ 'handlers': ['mail_admins'],
+ 'level': 'ERROR',
+ 'propagate': True,
+ },
+ }
+}
diff --git a/bitbake/lib/webhob/whbmain/urls.py b/bitbake/lib/webhob/whbmain/urls.py
new file mode 100644
index 0000000..04e0352
--- /dev/null
+++ b/bitbake/lib/webhob/whbmain/urls.py
@@ -0,0 +1,17 @@
+from django.conf.urls import patterns, include, url
+
+# Uncomment the next two lines to enable the admin:
+# from django.contrib import admin
+# admin.autodiscover()
+
+urlpatterns = patterns('',
+ # Examples:
+ # url(r'^$', 'webhob.views.home', name='home'),
+ # url(r'^webhob/', include('webhob.foo.urls')),
+
+ # Uncomment the admin/doc line below to enable admin documentation:
+ # url(r'^admin/doc/', include('django.contrib.admindocs.urls')),
+
+ # Uncomment the next line to enable the admin:
+ # url(r'^admin/', include(admin.site.urls)),
+)
diff --git a/bitbake/lib/webhob/whbmain/wsgi.py b/bitbake/lib/webhob/whbmain/wsgi.py
new file mode 100644
index 0000000..9595eda
--- /dev/null
+++ b/bitbake/lib/webhob/whbmain/wsgi.py
@@ -0,0 +1,32 @@
+"""
+WSGI config for webhob project.
+
+This module contains the WSGI application used by Django's development server
+and any production WSGI deployments. It should expose a module-level variable
+named ``application``. Django's ``runserver`` and ``runfcgi`` commands discover
+this application via the ``WSGI_APPLICATION`` setting.
+
+Usually you will have the standard Django WSGI application here, but it also
+might make sense to replace the whole Django WSGI application with a custom one
+that later delegates to the Django one. For example, you could introduce WSGI
+middleware here, or combine a Django application with an application of another
+framework.
+
+"""
+import os
+
+# We defer to a DJANGO_SETTINGS_MODULE already in the environment. This breaks
+# if running multiple sites in the same mod_wsgi process. To fix this, use
+# mod_wsgi daemon mode with each site in its own daemon process, or use
+# os.environ["DJANGO_SETTINGS_MODULE"] = "webhob.settings"
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "wbhmain.settings")
+
+# This application object is used by any WSGI server configured to use this
+# file. This includes Django's development server, if the WSGI_APPLICATION
+# setting points here.
+from django.core.wsgi import get_wsgi_application
+application = get_wsgi_application()
+
+# Apply WSGI middleware here.
+# from helloworld.wsgi import HelloWorldApplication
+# application = HelloWorldApplication(application)
--
1.8.1.2
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 2/3] Create Tasks Django model
2013-06-12 15:11 ` [Webhob] " Calin Dragomir
@ 2013-06-12 15:11 ` Calin Dragomir
-1 siblings, 0 replies; 7+ messages in thread
From: Calin Dragomir @ 2013-06-12 15:11 UTC (permalink / raw)
To: webhob, bitbake-devel
This will be used to store Tasks related information in the database
using Django ORM.
---
bitbake/lib/webhob/orm/__init__.py | 0
bitbake/lib/webhob/orm/models.py | 50 ++++++++++++++++++++++++++++++++++
bitbake/lib/webhob/orm/tests.py | 16 +++++++++++
bitbake/lib/webhob/orm/views.py | 1 +
bitbake/lib/webhob/whbmain/settings.py | 1 +
5 files changed, 68 insertions(+)
create mode 100644 bitbake/lib/webhob/orm/__init__.py
create mode 100644 bitbake/lib/webhob/orm/models.py
create mode 100644 bitbake/lib/webhob/orm/tests.py
create mode 100644 bitbake/lib/webhob/orm/views.py
diff --git a/bitbake/lib/webhob/orm/__init__.py b/bitbake/lib/webhob/orm/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/bitbake/lib/webhob/orm/models.py b/bitbake/lib/webhob/orm/models.py
new file mode 100644
index 0000000..13eadd4
--- /dev/null
+++ b/bitbake/lib/webhob/orm/models.py
@@ -0,0 +1,50 @@
+from django.db import models
+
+class Tasks(models.Model):
+
+ SSTATE_RESULT = (
+ ('not_applicable', 'Not Applicable'), # For rest of tasks, but they still need checking.
+ ('unavailable', 'Unavailable'), # it is a miss
+ ('failed', 'Failed'), # there was a pkg, but the script failed
+ ('restored', 'Restored'), # succesfully restored
+ )
+
+ TASK_CODING = (
+ ('python', 'Python'),
+ ('shell', 'Shell'),
+ )
+
+ TASK_OUTCOME = (
+ ('succeded', 'Succeded'),
+ ('failed', 'Failed'),
+ ('existing', 'Existing'),
+ ('sstate', 'Sstate'),
+ ('covered', 'Covered'),
+ )
+
+ uuid = models.CharField(max_length=100)
+ task_id = models.IntegerField()
+ order = models.IntegerField()
+ task_executed = models.BooleanField() # True means Executed, False means Prebuilt
+ outcome = models.CharField(max_length=50, choices=TASK_OUTCOME)
+ sstate_checksum = models.CharField(max_length=100)
+ path_to_sstate_obj = models.FilePathField(max_length=500, blank=True)
+ recipe = models.CharField(max_length=100)
+ task_name = models.CharField(max_length=100)
+ source_url = models.FilePathField(max_length=200)
+ log_file = models.FilePathField(max_length=200, blank=True)
+ work_directory = models.FilePathField(max_length=200)
+ script_type = models.CharField(max_length=10, choices=TASK_CODING)
+ file_path = models.FilePathField(max_length=200)
+ line_number = models.IntegerField()
+ py_stack_trace = models.TextField()
+ disk_io = models.DecimalField(max_digits=20, decimal_places=10)
+ cpu_usage = models.IntegerField()
+ elapsed_time = models.CharField(max_length=50)
+ dependent_tasks = models.TextField() ## do we need only to show text here or insert references? how do we connect to other tasks from the same table ?
+ errors_no = models.IntegerField()
+ warnings_no = models.IntegerField()
+ error = models.TextField()
+ warning = models.TextField()
+ sstate_result = models.CharField(max_length=50, choices=SSTATE_RESULT)
+
diff --git a/bitbake/lib/webhob/orm/tests.py b/bitbake/lib/webhob/orm/tests.py
new file mode 100644
index 0000000..501deb7
--- /dev/null
+++ b/bitbake/lib/webhob/orm/tests.py
@@ -0,0 +1,16 @@
+"""
+This file demonstrates writing tests using the unittest module. These will pass
+when you run "manage.py test".
+
+Replace this with more appropriate tests for your application.
+"""
+
+from django.test import TestCase
+
+
+class SimpleTest(TestCase):
+ def test_basic_addition(self):
+ """
+ Tests that 1 + 1 always equals 2.
+ """
+ self.assertEqual(1 + 1, 2)
diff --git a/bitbake/lib/webhob/orm/views.py b/bitbake/lib/webhob/orm/views.py
new file mode 100644
index 0000000..60f00ef
--- /dev/null
+++ b/bitbake/lib/webhob/orm/views.py
@@ -0,0 +1 @@
+# Create your views here.
diff --git a/bitbake/lib/webhob/whbmain/settings.py b/bitbake/lib/webhob/whbmain/settings.py
index bc59543..6a4e67a 100644
--- a/bitbake/lib/webhob/whbmain/settings.py
+++ b/bitbake/lib/webhob/whbmain/settings.py
@@ -124,6 +124,7 @@ INSTALLED_APPS = (
# 'django.contrib.admin',
# Uncomment the next line to enable admin documentation:
# 'django.contrib.admindocs',
+ 'orm',
)
# A sample logging configuration. The only tangible logging
--
1.8.1.2
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [Webhob] [PATCH 2/3] Create Tasks Django model
@ 2013-06-12 15:11 ` Calin Dragomir
0 siblings, 0 replies; 7+ messages in thread
From: Calin Dragomir @ 2013-06-12 15:11 UTC (permalink / raw)
To: webhob, bitbake-devel
This will be used to store Tasks related information in the database
using Django ORM.
---
bitbake/lib/webhob/orm/__init__.py | 0
bitbake/lib/webhob/orm/models.py | 50 ++++++++++++++++++++++++++++++++++
bitbake/lib/webhob/orm/tests.py | 16 +++++++++++
bitbake/lib/webhob/orm/views.py | 1 +
bitbake/lib/webhob/whbmain/settings.py | 1 +
5 files changed, 68 insertions(+)
create mode 100644 bitbake/lib/webhob/orm/__init__.py
create mode 100644 bitbake/lib/webhob/orm/models.py
create mode 100644 bitbake/lib/webhob/orm/tests.py
create mode 100644 bitbake/lib/webhob/orm/views.py
diff --git a/bitbake/lib/webhob/orm/__init__.py b/bitbake/lib/webhob/orm/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/bitbake/lib/webhob/orm/models.py b/bitbake/lib/webhob/orm/models.py
new file mode 100644
index 0000000..13eadd4
--- /dev/null
+++ b/bitbake/lib/webhob/orm/models.py
@@ -0,0 +1,50 @@
+from django.db import models
+
+class Tasks(models.Model):
+
+ SSTATE_RESULT = (
+ ('not_applicable', 'Not Applicable'), # For rest of tasks, but they still need checking.
+ ('unavailable', 'Unavailable'), # it is a miss
+ ('failed', 'Failed'), # there was a pkg, but the script failed
+ ('restored', 'Restored'), # succesfully restored
+ )
+
+ TASK_CODING = (
+ ('python', 'Python'),
+ ('shell', 'Shell'),
+ )
+
+ TASK_OUTCOME = (
+ ('succeded', 'Succeded'),
+ ('failed', 'Failed'),
+ ('existing', 'Existing'),
+ ('sstate', 'Sstate'),
+ ('covered', 'Covered'),
+ )
+
+ uuid = models.CharField(max_length=100)
+ task_id = models.IntegerField()
+ order = models.IntegerField()
+ task_executed = models.BooleanField() # True means Executed, False means Prebuilt
+ outcome = models.CharField(max_length=50, choices=TASK_OUTCOME)
+ sstate_checksum = models.CharField(max_length=100)
+ path_to_sstate_obj = models.FilePathField(max_length=500, blank=True)
+ recipe = models.CharField(max_length=100)
+ task_name = models.CharField(max_length=100)
+ source_url = models.FilePathField(max_length=200)
+ log_file = models.FilePathField(max_length=200, blank=True)
+ work_directory = models.FilePathField(max_length=200)
+ script_type = models.CharField(max_length=10, choices=TASK_CODING)
+ file_path = models.FilePathField(max_length=200)
+ line_number = models.IntegerField()
+ py_stack_trace = models.TextField()
+ disk_io = models.DecimalField(max_digits=20, decimal_places=10)
+ cpu_usage = models.IntegerField()
+ elapsed_time = models.CharField(max_length=50)
+ dependent_tasks = models.TextField() ## do we need only to show text here or insert references? how do we connect to other tasks from the same table ?
+ errors_no = models.IntegerField()
+ warnings_no = models.IntegerField()
+ error = models.TextField()
+ warning = models.TextField()
+ sstate_result = models.CharField(max_length=50, choices=SSTATE_RESULT)
+
diff --git a/bitbake/lib/webhob/orm/tests.py b/bitbake/lib/webhob/orm/tests.py
new file mode 100644
index 0000000..501deb7
--- /dev/null
+++ b/bitbake/lib/webhob/orm/tests.py
@@ -0,0 +1,16 @@
+"""
+This file demonstrates writing tests using the unittest module. These will pass
+when you run "manage.py test".
+
+Replace this with more appropriate tests for your application.
+"""
+
+from django.test import TestCase
+
+
+class SimpleTest(TestCase):
+ def test_basic_addition(self):
+ """
+ Tests that 1 + 1 always equals 2.
+ """
+ self.assertEqual(1 + 1, 2)
diff --git a/bitbake/lib/webhob/orm/views.py b/bitbake/lib/webhob/orm/views.py
new file mode 100644
index 0000000..60f00ef
--- /dev/null
+++ b/bitbake/lib/webhob/orm/views.py
@@ -0,0 +1 @@
+# Create your views here.
diff --git a/bitbake/lib/webhob/whbmain/settings.py b/bitbake/lib/webhob/whbmain/settings.py
index bc59543..6a4e67a 100644
--- a/bitbake/lib/webhob/whbmain/settings.py
+++ b/bitbake/lib/webhob/whbmain/settings.py
@@ -124,6 +124,7 @@ INSTALLED_APPS = (
# 'django.contrib.admin',
# Uncomment the next line to enable admin documentation:
# 'django.contrib.admindocs',
+ 'orm',
)
# A sample logging configuration. The only tangible logging
--
1.8.1.2
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 3/3] Create Data Store Interface (DSI) file
2013-06-12 15:11 ` [Webhob] " Calin Dragomir
@ 2013-06-12 15:11 ` Calin Dragomir
-1 siblings, 0 replies; 7+ messages in thread
From: Calin Dragomir @ 2013-06-12 15:11 UTC (permalink / raw)
To: webhob, bitbake-devel
This adds the first version of the DSI file. It uses the Knotty
code for now, but inserts task related information into the database
using the Django ORM
---
bitbake/lib/bb/ui/dsi.py | 609 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 609 insertions(+)
create mode 100644 bitbake/lib/bb/ui/dsi.py
diff --git a/bitbake/lib/bb/ui/dsi.py b/bitbake/lib/bb/ui/dsi.py
new file mode 100644
index 0000000..4bcd509
--- /dev/null
+++ b/bitbake/lib/bb/ui/dsi.py
@@ -0,0 +1,609 @@
+#
+# BitBake (No)TTY UI Implementation
+#
+# Handling output to TTYs or files (no TTY)
+#
+# Copyright (C) 2006-2012 Richard Purdie
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+from __future__ import division
+
+import os
+import sys
+import xmlrpclib
+import logging
+import progressbar
+import signal
+import bb.msg
+import time
+import fcntl
+import struct
+import copy
+import datetime
+
+from bb.ui import uihelper
+
+logger = logging.getLogger("BitBake")
+interactive = sys.stdout.isatty()
+
+class WebHOBHelper(object):
+
+ def __init__(self):
+ self.configure_django()
+ self.task_order = 0
+ self.tasks_information = {}
+
+ def configure_django(self):
+ import webhob.whbmain.settings as whb_django_settings
+ from django.core.management import setup_environ
+ setup_environ(whb_django_settings)
+ # Add webhob to sys path for importing modules
+ sys.path.append(os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), 'webhob'))
+
+ def store_started_task(self, event, uuid):
+ self.task_order += 1
+ self.tasks_information[event.pid] = {
+ 'uuid': uuid,
+ 'task_id': 1,
+ 'task_executed': True,
+ 'order': self.task_order,
+ 'recipe': event._package,
+ 'task_name': event._task,
+ 'start_time': time.time(),
+ }
+
+ def update_stored_tasks(self, event, uuid):
+ self.tasks_information[event.pid].update({
+ 'outcome': event.getDisplayName().lower(),
+ 'end_time': time.time(),
+ })
+
+ def write_in_database(self, tasks_information):
+ # This needs to be imported after we have configured the Django settings file
+ from webhob.orm.models import Tasks
+ for pid in tasks_information.keys():
+ task = tasks_information[pid]
+
+ task_obj = Tasks.objects.create(uuid=task['uuid'],
+ order=task['order'],
+ outcome=task['outcome'],
+ recipe=task['recipe'],
+ task_name=task['task_name'],
+ elapsed_time=task['end_time'] - task['start_time'],
+ # The next lines are just for testing reasons
+ task_id=1,
+ path_to_sstate_obj='/home/calin',
+ source_url='/',
+ log_file='/',
+ work_directory='/',
+ script_type='python',
+ file_path='/',
+ line_number=55,
+ py_stack_trace='Testing traceback',
+ disk_io = 23.55633,
+ cpu_usage=43,
+ dependent_tasks='Task 1',
+ errors_no=0,
+ warnings_no=0,
+ error='None',
+ warning='None',
+ sstate_result='not_applicable')
+ task_obj.save()
+
+class BBProgress(progressbar.ProgressBar):
+ def __init__(self, msg, maxval):
+ self.msg = msg
+ widgets = [progressbar.Percentage(), ' ', progressbar.Bar(), ' ',
+ progressbar.ETA()]
+
+ try:
+ self._resize_default = signal.getsignal(signal.SIGWINCH)
+ except:
+ self._resize_default = None
+ progressbar.ProgressBar.__init__(self, maxval, [self.msg + ": "] + widgets)
+
+ def _handle_resize(self, signum, frame):
+ progressbar.ProgressBar._handle_resize(self, signum, frame)
+ if self._resize_default:
+ self._resize_default(signum, frame)
+ def finish(self):
+ progressbar.ProgressBar.finish(self)
+ if self._resize_default:
+ signal.signal(signal.SIGWINCH, self._resize_default)
+
+class NonInteractiveProgress(object):
+ fobj = sys.stdout
+
+ def __init__(self, msg, maxval):
+ self.msg = msg
+ self.maxval = maxval
+
+ def start(self):
+ self.fobj.write("%s..." % self.msg)
+ self.fobj.flush()
+ return self
+
+ def update(self, value):
+ pass
+
+ def finish(self):
+ self.fobj.write("done.\n")
+ self.fobj.flush()
+
+def new_progress(msg, maxval):
+ if interactive:
+ return BBProgress(msg, maxval)
+ else:
+ return NonInteractiveProgress(msg, maxval)
+
+def pluralise(singular, plural, qty):
+ if(qty == 1):
+ return singular % qty
+ else:
+ return plural % qty
+
+
+class InteractConsoleLogFilter(logging.Filter):
+ def __init__(self, tf, format):
+ self.tf = tf
+ self.format = format
+
+ def filter(self, record):
+ if record.levelno == self.format.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")):
+ return False
+ self.tf.clearFooter()
+ return True
+
+class TerminalFilter(object):
+ columns = 80
+
+ def sigwinch_handle(self, signum, frame):
+ self.columns = self.getTerminalColumns()
+ if self._sigwinch_default:
+ self._sigwinch_default(signum, frame)
+
+ def getTerminalColumns(self):
+ def ioctl_GWINSZ(fd):
+ try:
+ cr = struct.unpack('hh', fcntl.ioctl(fd, self.termios.TIOCGWINSZ, '1234'))
+ except:
+ return None
+ return cr
+ cr = ioctl_GWINSZ(sys.stdout.fileno())
+ if not cr:
+ try:
+ fd = os.open(os.ctermid(), os.O_RDONLY)
+ cr = ioctl_GWINSZ(fd)
+ os.close(fd)
+ except:
+ pass
+ if not cr:
+ try:
+ cr = (env['LINES'], env['COLUMNS'])
+ except:
+ cr = (25, 80)
+ return cr[1]
+
+ def __init__(self, main, helper, console, format):
+ self.main = main
+ self.helper = helper
+ self.cuu = None
+ self.stdinbackup = None
+ self.interactive = sys.stdout.isatty()
+ self.footer_present = False
+ self.lastpids = []
+
+ if not self.interactive:
+ return
+
+ try:
+ import curses
+ except ImportError:
+ sys.exit("FATAL: The knotty ui could not load the required curses python module.")
+
+ import termios
+ self.curses = curses
+ self.termios = termios
+ try:
+ fd = sys.stdin.fileno()
+ self.stdinbackup = termios.tcgetattr(fd)
+ new = copy.deepcopy(self.stdinbackup)
+ new[3] = new[3] & ~termios.ECHO
+ termios.tcsetattr(fd, termios.TCSADRAIN, new)
+ curses.setupterm()
+ if curses.tigetnum("colors") > 2:
+ format.enable_color()
+ self.ed = curses.tigetstr("ed")
+ if self.ed:
+ self.cuu = curses.tigetstr("cuu")
+ try:
+ self._sigwinch_default = signal.getsignal(signal.SIGWINCH)
+ signal.signal(signal.SIGWINCH, self.sigwinch_handle)
+ except:
+ pass
+ self.columns = self.getTerminalColumns()
+ except:
+ self.cuu = None
+ console.addFilter(InteractConsoleLogFilter(self, format))
+
+ def clearFooter(self):
+ if self.footer_present:
+ lines = self.footer_present
+ sys.stdout.write(self.curses.tparm(self.cuu, lines))
+ sys.stdout.write(self.curses.tparm(self.ed))
+ self.footer_present = False
+
+ def updateFooter(self):
+ if not self.cuu:
+ return
+ activetasks = self.helper.running_tasks
+ failedtasks = self.helper.failed_tasks
+ runningpids = self.helper.running_pids
+ if self.footer_present and (self.lastcount == self.helper.tasknumber_current) and (self.lastpids == runningpids):
+ return
+ if self.footer_present:
+ self.clearFooter()
+ if (not self.helper.tasknumber_total or self.helper.tasknumber_current == self.helper.tasknumber_total) and not len(activetasks):
+ return
+ tasks = []
+ for t in runningpids:
+ tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
+
+ if self.main.shutdown:
+ content = "Waiting for %s running tasks to finish:" % len(activetasks)
+ elif not len(activetasks):
+ content = "No currently running tasks (%s of %s)" % (self.helper.tasknumber_current, self.helper.tasknumber_total)
+ else:
+ content = "Currently %s running tasks (%s of %s):" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total)
+ print(content)
+ lines = 1 + int(len(content) / (self.columns + 1))
+ for tasknum, task in enumerate(tasks):
+ content = "%s: %s" % (tasknum, task)
+ print(content)
+ lines = lines + 1 + int(len(content) / (self.columns + 1))
+ self.footer_present = lines
+ self.lastpids = runningpids[:]
+ self.lastcount = self.helper.tasknumber_current
+
+ def finish(self):
+ if self.stdinbackup:
+ fd = sys.stdin.fileno()
+ self.termios.tcsetattr(fd, self.termios.TCSADRAIN, self.stdinbackup)
+
+def main(server, eventHandler, params, tf = TerminalFilter):
+ print "This is running in DSI mode"
+
+ # Generate an unique ID for this build
+ # TODO: for multiple build commands this might not work as expected
+ import uuid
+ uuid = sessionid = str(uuid.uuid4())
+
+ # Get values of variables which control our output
+ includelogs, error = server.runCommand(["getVariable", "BBINCLUDELOGS"])
+ if error:
+ logger.error("Unable to get the value of BBINCLUDELOGS variable: %s" % error)
+ return 1
+ loglines, error = server.runCommand(["getVariable", "BBINCLUDELOGS_LINES"])
+ if error:
+ logger.error("Unable to get the value of BBINCLUDELOGS_LINES variable: %s" % error)
+ return 1
+ consolelogfile, error = server.runCommand(["getVariable", "BB_CONSOLELOG"])
+ if error:
+ logger.error("Unable to get the value of BB_CONSOLELOG variable: %s" % error)
+ return 1
+
+ if sys.stdin.isatty() and sys.stdout.isatty():
+ log_exec_tty = True
+ else:
+ log_exec_tty = False
+
+ helper = uihelper.BBUIHelper()
+
+ console = logging.StreamHandler(sys.stdout)
+ format_str = "%(levelname)s: %(message)s"
+ format = bb.msg.BBLogFormatter(format_str)
+ bb.msg.addDefaultlogFilter(console)
+ console.setFormatter(format)
+ logger.addHandler(console)
+
+ if consolelogfile and not params.options.show_environment:
+ bb.utils.mkdirhier(os.path.dirname(consolelogfile))
+ conlogformat = bb.msg.BBLogFormatter(format_str)
+ consolelog = logging.FileHandler(consolelogfile)
+ bb.msg.addDefaultlogFilter(consolelog)
+ consolelog.setFormatter(conlogformat)
+ logger.addHandler(consolelog)
+
+ try:
+ params.updateFromServer(server)
+ cmdline = params.parseActions()
+ if not cmdline:
+ print("Nothing to do. Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information.")
+ return 1
+ if 'msg' in cmdline and cmdline['msg']:
+ logger.error(cmdline['msg'])
+ return 1
+
+ ret, error = server.runCommand(cmdline['action'])
+ if error:
+ logger.error("Command '%s' failed: %s" % (cmdline, error))
+ return 1
+ elif ret != True:
+ logger.error("Command '%s' failed: returned %s" % (cmdline, ret))
+ return 1
+ except xmlrpclib.Fault as x:
+ logger.error("XMLRPC Fault getting commandline:\n %s" % x)
+ return 1
+
+ parseprogress = None
+ cacheprogress = None
+ main.shutdown = 0
+ interrupted = False
+ return_value = 0
+ errors = 0
+ warnings = 0
+ taskfailures = []
+ wbhbhelper = WebHOBHelper()
+
+ termfilter = tf(main, helper, console, format)
+
+ while True:
+ try:
+ termfilter.updateFooter()
+ event = eventHandler.waitEvent(0.25)
+
+ if event is None:
+ if main.shutdown > 1:
+ break
+ continue
+
+ helper.eventHandler(event)
+
+ if isinstance(event, bb.runqueue.runQueueExitWait):
+ if not main.shutdown:
+ main.shutdown = 1
+
+ if isinstance(event, bb.build.TaskStarted):
+ wbhbhelper.store_started_task(event, uuid)
+
+ if isinstance(event, (bb.build.TaskSucceeded, bb.build.TaskFailedSilent, bb.build.TaskFailed)):
+ wbhbhelper.update_stored_tasks(event, uuid)
+
+ if isinstance(event, bb.event.LogExecTTY):
+ if log_exec_tty:
+ tries = event.retries
+ while tries:
+ print("Trying to run: %s" % event.prog)
+ if os.system(event.prog) == 0:
+ break
+ time.sleep(event.sleep_delay)
+ tries -= 1
+ if tries:
+ continue
+ logger.warn(event.msg)
+ continue
+
+ if isinstance(event, logging.LogRecord):
+ if event.levelno >= format.ERROR:
+ errors = errors + 1
+ return_value = 1
+ elif event.levelno == format.WARNING:
+ warnings = warnings + 1
+ # For "normal" logging conditions, don't show note logs from tasks
+ # but do show them if the user has changed the default log level to
+ # include verbose/debug messages
+ if event.taskpid != 0 and event.levelno <= format.NOTE:
+ continue
+ logger.handle(event)
+ continue
+
+ if isinstance(event, bb.build.TaskFailed):
+ return_value = 1
+ logfile = event.logfile
+ if logfile and os.path.exists(logfile):
+ termfilter.clearFooter()
+ bb.error("Logfile of failure stored in: %s" % logfile)
+ if includelogs and not event.errprinted:
+ print("Log data follows:")
+ f = open(logfile, "r")
+ lines = []
+ while True:
+ l = f.readline()
+ if l == '':
+ break
+ l = l.rstrip()
+ if loglines:
+ lines.append(' | %s' % l)
+ if len(lines) > int(loglines):
+ lines.pop(0)
+ else:
+ print('| %s' % l)
+ f.close()
+ if lines:
+ for line in lines:
+ print(line)
+ if isinstance(event, bb.build.TaskBase):
+ logger.info(event._message)
+ continue
+ if isinstance(event, bb.event.ParseStarted):
+ if event.total == 0:
+ continue
+ parseprogress = new_progress("Parsing recipes", event.total).start()
+ continue
+ if isinstance(event, bb.event.ParseProgress):
+ parseprogress.update(event.current)
+ continue
+ if isinstance(event, bb.event.ParseCompleted):
+ if not parseprogress:
+ continue
+
+ parseprogress.finish()
+ print(("Parsing of %d .bb files complete (%d cached, %d parsed). %d targets, %d skipped, %d masked, %d errors."
+ % ( event.total, event.cached, event.parsed, event.virtuals, event.skipped, event.masked, event.errors)))
+ continue
+
+ if isinstance(event, bb.event.CacheLoadStarted):
+ cacheprogress = new_progress("Loading cache", event.total).start()
+ continue
+ if isinstance(event, bb.event.CacheLoadProgress):
+ cacheprogress.update(event.current)
+ continue
+ if isinstance(event, bb.event.CacheLoadCompleted):
+ cacheprogress.finish()
+ print("Loaded %d entries from dependency cache." % event.num_entries)
+ continue
+
+ if isinstance(event, bb.command.CommandFailed):
+ return_value = event.exitcode
+ errors = errors + 1
+ logger.error("Command execution failed: %s", event.error)
+ main.shutdown = 2
+ continue
+ if isinstance(event, bb.command.CommandExit):
+ if not return_value:
+ return_value = event.exitcode
+ continue
+ if isinstance(event, (bb.command.CommandCompleted, bb.cooker.CookerExit)):
+ main.shutdown = 2
+ continue
+ if isinstance(event, bb.event.MultipleProviders):
+ logger.info("multiple providers are available for %s%s (%s)", event._is_runtime and "runtime " or "",
+ event._item,
+ ", ".join(event._candidates))
+ logger.info("consider defining a PREFERRED_PROVIDER entry to match %s", event._item)
+ continue
+ if isinstance(event, bb.event.NoProvider):
+ return_value = 1
+ errors = errors + 1
+ if event._runtime:
+ r = "R"
+ else:
+ r = ""
+
+ if event._dependees:
+ logger.error("Nothing %sPROVIDES '%s' (but %s %sDEPENDS on or otherwise requires it)", r, event._item, ", ".join(event._dependees), r)
+ else:
+ logger.error("Nothing %sPROVIDES '%s'", r, event._item)
+ if event._reasons:
+ for reason in event._reasons:
+ logger.error("%s", reason)
+ continue
+
+ if isinstance(event, bb.runqueue.sceneQueueTaskStarted):
+ logger.info("Running setscene task %d of %d (%s)" % (event.stats.completed + event.stats.active + event.stats.failed + 1, event.stats.total, event.taskstring))
+ continue
+
+ if isinstance(event, bb.runqueue.runQueueTaskStarted):
+ if event.noexec:
+ tasktype = 'noexec task'
+ else:
+ tasktype = 'task'
+ logger.info("Running %s %s of %s (ID: %s, %s)",
+ tasktype,
+ event.stats.completed + event.stats.active +
+ event.stats.failed + 1,
+ event.stats.total, event.taskid, event.taskstring)
+ continue
+
+ if isinstance(event, bb.runqueue.runQueueTaskFailed):
+ taskfailures.append(event.taskstring)
+ logger.error("Task %s (%s) failed with exit code '%s'",
+ event.taskid, event.taskstring, event.exitcode)
+ continue
+
+ if isinstance(event, bb.runqueue.sceneQueueTaskFailed):
+ logger.warn("Setscene task %s (%s) failed with exit code '%s' - real task will be run instead",
+ event.taskid, event.taskstring, event.exitcode)
+ continue
+
+ if isinstance(event, bb.event.ConfigParsed):
+ # timestamp should be added for this
+ continue
+
+ if isinstance(event, bb.event.RecipeParsed):
+ # timestamp should be added for this
+ continue
+
+ if isinstance(event, bb.event.OperationStarted):
+ # timestamp should be added for this
+ continue
+
+ if isinstance(event, bb.event.OperationCompleted):
+ # timestamp should be added for this
+ # signal a complete operation
+ # calculate timing
+ continue
+
+ if isinstance(event, bb.event.DiskFull):
+ # trigger an error
+ continue
+
+ # ignore
+ if isinstance(event, (bb.event.BuildBase,
+ bb.event.StampUpdate,
+ bb.event.RecipePreFinalise,
+ bb.runqueue.runQueueEvent,
+ bb.runqueue.runQueueExitWait,
+ bb.event.OperationProgress)):
+ continue
+
+ logger.error("Unknown event: %s", event)
+
+ except EnvironmentError as ioerror:
+ termfilter.clearFooter()
+ # ignore interrupted io
+ if ioerror.args[0] == 4:
+ pass
+ except KeyboardInterrupt:
+ termfilter.clearFooter()
+ if main.shutdown == 1:
+ print("\nSecond Keyboard Interrupt, stopping...\n")
+ _, error = server.runCommand(["stateStop"])
+ if error:
+ logger.error("Unable to cleanly stop: %s" % error)
+ if main.shutdown == 0:
+ print("\nKeyboard Interrupt, closing down...\n")
+ interrupted = True
+ _, error = server.runCommand(["stateShutdown"])
+ if error:
+ logger.error("Unable to cleanly shutdown: %s" % error)
+ main.shutdown = main.shutdown + 1
+ pass
+
+ summary = ""
+ if taskfailures:
+ summary += pluralise("\nSummary: %s task failed:",
+ "\nSummary: %s tasks failed:", len(taskfailures))
+ for failure in taskfailures:
+ summary += "\n %s" % failure
+ if warnings:
+ summary += pluralise("\nSummary: There was %s WARNING message shown.",
+ "\nSummary: There were %s WARNING messages shown.", warnings)
+ if return_value:
+ summary += pluralise("\nSummary: There was %s ERROR message shown, returning a non-zero exit code.",
+ "\nSummary: There were %s ERROR messages shown, returning a non-zero exit code.", errors)
+ if summary:
+ print(summary)
+
+ if interrupted:
+ print("Execution was interrupted, returning a non-zero exit code.")
+ if return_value == 0:
+ return_value = 1
+
+ termfilter.finish()
+ wbhbhelper.write_in_database(wbhbhelper.tasks_information)
+
+ return return_value
+
--
1.8.1.2
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [Webhob] [PATCH 3/3] Create Data Store Interface (DSI) file
@ 2013-06-12 15:11 ` Calin Dragomir
0 siblings, 0 replies; 7+ messages in thread
From: Calin Dragomir @ 2013-06-12 15:11 UTC (permalink / raw)
To: webhob, bitbake-devel
This adds the first version of the DSI file. It uses the Knotty
code for now, but inserts task related information into the database
using the Django ORM
---
bitbake/lib/bb/ui/dsi.py | 609 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 609 insertions(+)
create mode 100644 bitbake/lib/bb/ui/dsi.py
diff --git a/bitbake/lib/bb/ui/dsi.py b/bitbake/lib/bb/ui/dsi.py
new file mode 100644
index 0000000..4bcd509
--- /dev/null
+++ b/bitbake/lib/bb/ui/dsi.py
@@ -0,0 +1,609 @@
+#
+# BitBake (No)TTY UI Implementation
+#
+# Handling output to TTYs or files (no TTY)
+#
+# Copyright (C) 2006-2012 Richard Purdie
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+from __future__ import division
+
+import os
+import sys
+import xmlrpclib
+import logging
+import progressbar
+import signal
+import bb.msg
+import time
+import fcntl
+import struct
+import copy
+import datetime
+
+from bb.ui import uihelper
+
+logger = logging.getLogger("BitBake")
+interactive = sys.stdout.isatty()
+
+class WebHOBHelper(object):
+
+ def __init__(self):
+ self.configure_django()
+ self.task_order = 0
+ self.tasks_information = {}
+
+ def configure_django(self):
+ import webhob.whbmain.settings as whb_django_settings
+ from django.core.management import setup_environ
+ setup_environ(whb_django_settings)
+ # Add webhob to sys path for importing modules
+ sys.path.append(os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), 'webhob'))
+
+ def store_started_task(self, event, uuid):
+ self.task_order += 1
+ self.tasks_information[event.pid] = {
+ 'uuid': uuid,
+ 'task_id': 1,
+ 'task_executed': True,
+ 'order': self.task_order,
+ 'recipe': event._package,
+ 'task_name': event._task,
+ 'start_time': time.time(),
+ }
+
+ def update_stored_tasks(self, event, uuid):
+ self.tasks_information[event.pid].update({
+ 'outcome': event.getDisplayName().lower(),
+ 'end_time': time.time(),
+ })
+
+ def write_in_database(self, tasks_information):
+ # This needs to be imported after we have configured the Django settings file
+ from webhob.orm.models import Tasks
+ for pid in tasks_information.keys():
+ task = tasks_information[pid]
+
+ task_obj = Tasks.objects.create(uuid=task['uuid'],
+ order=task['order'],
+ outcome=task['outcome'],
+ recipe=task['recipe'],
+ task_name=task['task_name'],
+ elapsed_time=task['end_time'] - task['start_time'],
+ # The next lines are just for testing reasons
+ task_id=1,
+ path_to_sstate_obj='/home/calin',
+ source_url='/',
+ log_file='/',
+ work_directory='/',
+ script_type='python',
+ file_path='/',
+ line_number=55,
+ py_stack_trace='Testing traceback',
+ disk_io = 23.55633,
+ cpu_usage=43,
+ dependent_tasks='Task 1',
+ errors_no=0,
+ warnings_no=0,
+ error='None',
+ warning='None',
+ sstate_result='not_applicable')
+ task_obj.save()
+
+class BBProgress(progressbar.ProgressBar):
+ def __init__(self, msg, maxval):
+ self.msg = msg
+ widgets = [progressbar.Percentage(), ' ', progressbar.Bar(), ' ',
+ progressbar.ETA()]
+
+ try:
+ self._resize_default = signal.getsignal(signal.SIGWINCH)
+ except:
+ self._resize_default = None
+ progressbar.ProgressBar.__init__(self, maxval, [self.msg + ": "] + widgets)
+
+ def _handle_resize(self, signum, frame):
+ progressbar.ProgressBar._handle_resize(self, signum, frame)
+ if self._resize_default:
+ self._resize_default(signum, frame)
+ def finish(self):
+ progressbar.ProgressBar.finish(self)
+ if self._resize_default:
+ signal.signal(signal.SIGWINCH, self._resize_default)
+
+class NonInteractiveProgress(object):
+ fobj = sys.stdout
+
+ def __init__(self, msg, maxval):
+ self.msg = msg
+ self.maxval = maxval
+
+ def start(self):
+ self.fobj.write("%s..." % self.msg)
+ self.fobj.flush()
+ return self
+
+ def update(self, value):
+ pass
+
+ def finish(self):
+ self.fobj.write("done.\n")
+ self.fobj.flush()
+
+def new_progress(msg, maxval):
+ if interactive:
+ return BBProgress(msg, maxval)
+ else:
+ return NonInteractiveProgress(msg, maxval)
+
+def pluralise(singular, plural, qty):
+ if(qty == 1):
+ return singular % qty
+ else:
+ return plural % qty
+
+
+class InteractConsoleLogFilter(logging.Filter):
+ def __init__(self, tf, format):
+ self.tf = tf
+ self.format = format
+
+ def filter(self, record):
+ if record.levelno == self.format.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")):
+ return False
+ self.tf.clearFooter()
+ return True
+
+class TerminalFilter(object):
+ columns = 80
+
+ def sigwinch_handle(self, signum, frame):
+ self.columns = self.getTerminalColumns()
+ if self._sigwinch_default:
+ self._sigwinch_default(signum, frame)
+
+ def getTerminalColumns(self):
+ def ioctl_GWINSZ(fd):
+ try:
+ cr = struct.unpack('hh', fcntl.ioctl(fd, self.termios.TIOCGWINSZ, '1234'))
+ except:
+ return None
+ return cr
+ cr = ioctl_GWINSZ(sys.stdout.fileno())
+ if not cr:
+ try:
+ fd = os.open(os.ctermid(), os.O_RDONLY)
+ cr = ioctl_GWINSZ(fd)
+ os.close(fd)
+ except:
+ pass
+ if not cr:
+ try:
+ cr = (env['LINES'], env['COLUMNS'])
+ except:
+ cr = (25, 80)
+ return cr[1]
+
+ def __init__(self, main, helper, console, format):
+ self.main = main
+ self.helper = helper
+ self.cuu = None
+ self.stdinbackup = None
+ self.interactive = sys.stdout.isatty()
+ self.footer_present = False
+ self.lastpids = []
+
+ if not self.interactive:
+ return
+
+ try:
+ import curses
+ except ImportError:
+ sys.exit("FATAL: The knotty ui could not load the required curses python module.")
+
+ import termios
+ self.curses = curses
+ self.termios = termios
+ try:
+ fd = sys.stdin.fileno()
+ self.stdinbackup = termios.tcgetattr(fd)
+ new = copy.deepcopy(self.stdinbackup)
+ new[3] = new[3] & ~termios.ECHO
+ termios.tcsetattr(fd, termios.TCSADRAIN, new)
+ curses.setupterm()
+ if curses.tigetnum("colors") > 2:
+ format.enable_color()
+ self.ed = curses.tigetstr("ed")
+ if self.ed:
+ self.cuu = curses.tigetstr("cuu")
+ try:
+ self._sigwinch_default = signal.getsignal(signal.SIGWINCH)
+ signal.signal(signal.SIGWINCH, self.sigwinch_handle)
+ except:
+ pass
+ self.columns = self.getTerminalColumns()
+ except:
+ self.cuu = None
+ console.addFilter(InteractConsoleLogFilter(self, format))
+
+ def clearFooter(self):
+ if self.footer_present:
+ lines = self.footer_present
+ sys.stdout.write(self.curses.tparm(self.cuu, lines))
+ sys.stdout.write(self.curses.tparm(self.ed))
+ self.footer_present = False
+
+ def updateFooter(self):
+ if not self.cuu:
+ return
+ activetasks = self.helper.running_tasks
+ failedtasks = self.helper.failed_tasks
+ runningpids = self.helper.running_pids
+ if self.footer_present and (self.lastcount == self.helper.tasknumber_current) and (self.lastpids == runningpids):
+ return
+ if self.footer_present:
+ self.clearFooter()
+ if (not self.helper.tasknumber_total or self.helper.tasknumber_current == self.helper.tasknumber_total) and not len(activetasks):
+ return
+ tasks = []
+ for t in runningpids:
+ tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
+
+ if self.main.shutdown:
+ content = "Waiting for %s running tasks to finish:" % len(activetasks)
+ elif not len(activetasks):
+ content = "No currently running tasks (%s of %s)" % (self.helper.tasknumber_current, self.helper.tasknumber_total)
+ else:
+ content = "Currently %s running tasks (%s of %s):" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total)
+ print(content)
+ lines = 1 + int(len(content) / (self.columns + 1))
+ for tasknum, task in enumerate(tasks):
+ content = "%s: %s" % (tasknum, task)
+ print(content)
+ lines = lines + 1 + int(len(content) / (self.columns + 1))
+ self.footer_present = lines
+ self.lastpids = runningpids[:]
+ self.lastcount = self.helper.tasknumber_current
+
+ def finish(self):
+ if self.stdinbackup:
+ fd = sys.stdin.fileno()
+ self.termios.tcsetattr(fd, self.termios.TCSADRAIN, self.stdinbackup)
+
+def main(server, eventHandler, params, tf = TerminalFilter):
+ print "This is running in DSI mode"
+
+ # Generate an unique ID for this build
+ # TODO: for multiple build commands this might not work as expected
+ import uuid
+ uuid = sessionid = str(uuid.uuid4())
+
+ # Get values of variables which control our output
+ includelogs, error = server.runCommand(["getVariable", "BBINCLUDELOGS"])
+ if error:
+ logger.error("Unable to get the value of BBINCLUDELOGS variable: %s" % error)
+ return 1
+ loglines, error = server.runCommand(["getVariable", "BBINCLUDELOGS_LINES"])
+ if error:
+ logger.error("Unable to get the value of BBINCLUDELOGS_LINES variable: %s" % error)
+ return 1
+ consolelogfile, error = server.runCommand(["getVariable", "BB_CONSOLELOG"])
+ if error:
+ logger.error("Unable to get the value of BB_CONSOLELOG variable: %s" % error)
+ return 1
+
+ if sys.stdin.isatty() and sys.stdout.isatty():
+ log_exec_tty = True
+ else:
+ log_exec_tty = False
+
+ helper = uihelper.BBUIHelper()
+
+ console = logging.StreamHandler(sys.stdout)
+ format_str = "%(levelname)s: %(message)s"
+ format = bb.msg.BBLogFormatter(format_str)
+ bb.msg.addDefaultlogFilter(console)
+ console.setFormatter(format)
+ logger.addHandler(console)
+
+ if consolelogfile and not params.options.show_environment:
+ bb.utils.mkdirhier(os.path.dirname(consolelogfile))
+ conlogformat = bb.msg.BBLogFormatter(format_str)
+ consolelog = logging.FileHandler(consolelogfile)
+ bb.msg.addDefaultlogFilter(consolelog)
+ consolelog.setFormatter(conlogformat)
+ logger.addHandler(consolelog)
+
+ try:
+ params.updateFromServer(server)
+ cmdline = params.parseActions()
+ if not cmdline:
+ print("Nothing to do. Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information.")
+ return 1
+ if 'msg' in cmdline and cmdline['msg']:
+ logger.error(cmdline['msg'])
+ return 1
+
+ ret, error = server.runCommand(cmdline['action'])
+ if error:
+ logger.error("Command '%s' failed: %s" % (cmdline, error))
+ return 1
+ elif ret != True:
+ logger.error("Command '%s' failed: returned %s" % (cmdline, ret))
+ return 1
+ except xmlrpclib.Fault as x:
+ logger.error("XMLRPC Fault getting commandline:\n %s" % x)
+ return 1
+
+ parseprogress = None
+ cacheprogress = None
+ main.shutdown = 0
+ interrupted = False
+ return_value = 0
+ errors = 0
+ warnings = 0
+ taskfailures = []
+ wbhbhelper = WebHOBHelper()
+
+ termfilter = tf(main, helper, console, format)
+
+ while True:
+ try:
+ termfilter.updateFooter()
+ event = eventHandler.waitEvent(0.25)
+
+ if event is None:
+ if main.shutdown > 1:
+ break
+ continue
+
+ helper.eventHandler(event)
+
+ if isinstance(event, bb.runqueue.runQueueExitWait):
+ if not main.shutdown:
+ main.shutdown = 1
+
+ if isinstance(event, bb.build.TaskStarted):
+ wbhbhelper.store_started_task(event, uuid)
+
+ if isinstance(event, (bb.build.TaskSucceeded, bb.build.TaskFailedSilent, bb.build.TaskFailed)):
+ wbhbhelper.update_stored_tasks(event, uuid)
+
+ if isinstance(event, bb.event.LogExecTTY):
+ if log_exec_tty:
+ tries = event.retries
+ while tries:
+ print("Trying to run: %s" % event.prog)
+ if os.system(event.prog) == 0:
+ break
+ time.sleep(event.sleep_delay)
+ tries -= 1
+ if tries:
+ continue
+ logger.warn(event.msg)
+ continue
+
+ if isinstance(event, logging.LogRecord):
+ if event.levelno >= format.ERROR:
+ errors = errors + 1
+ return_value = 1
+ elif event.levelno == format.WARNING:
+ warnings = warnings + 1
+ # For "normal" logging conditions, don't show note logs from tasks
+ # but do show them if the user has changed the default log level to
+ # include verbose/debug messages
+ if event.taskpid != 0 and event.levelno <= format.NOTE:
+ continue
+ logger.handle(event)
+ continue
+
+ if isinstance(event, bb.build.TaskFailed):
+ return_value = 1
+ logfile = event.logfile
+ if logfile and os.path.exists(logfile):
+ termfilter.clearFooter()
+ bb.error("Logfile of failure stored in: %s" % logfile)
+ if includelogs and not event.errprinted:
+ print("Log data follows:")
+ f = open(logfile, "r")
+ lines = []
+ while True:
+ l = f.readline()
+ if l == '':
+ break
+ l = l.rstrip()
+ if loglines:
+ lines.append(' | %s' % l)
+ if len(lines) > int(loglines):
+ lines.pop(0)
+ else:
+ print('| %s' % l)
+ f.close()
+ if lines:
+ for line in lines:
+ print(line)
+ if isinstance(event, bb.build.TaskBase):
+ logger.info(event._message)
+ continue
+ if isinstance(event, bb.event.ParseStarted):
+ if event.total == 0:
+ continue
+ parseprogress = new_progress("Parsing recipes", event.total).start()
+ continue
+ if isinstance(event, bb.event.ParseProgress):
+ parseprogress.update(event.current)
+ continue
+ if isinstance(event, bb.event.ParseCompleted):
+ if not parseprogress:
+ continue
+
+ parseprogress.finish()
+ print(("Parsing of %d .bb files complete (%d cached, %d parsed). %d targets, %d skipped, %d masked, %d errors."
+ % ( event.total, event.cached, event.parsed, event.virtuals, event.skipped, event.masked, event.errors)))
+ continue
+
+ if isinstance(event, bb.event.CacheLoadStarted):
+ cacheprogress = new_progress("Loading cache", event.total).start()
+ continue
+ if isinstance(event, bb.event.CacheLoadProgress):
+ cacheprogress.update(event.current)
+ continue
+ if isinstance(event, bb.event.CacheLoadCompleted):
+ cacheprogress.finish()
+ print("Loaded %d entries from dependency cache." % event.num_entries)
+ continue
+
+ if isinstance(event, bb.command.CommandFailed):
+ return_value = event.exitcode
+ errors = errors + 1
+ logger.error("Command execution failed: %s", event.error)
+ main.shutdown = 2
+ continue
+ if isinstance(event, bb.command.CommandExit):
+ if not return_value:
+ return_value = event.exitcode
+ continue
+ if isinstance(event, (bb.command.CommandCompleted, bb.cooker.CookerExit)):
+ main.shutdown = 2
+ continue
+ if isinstance(event, bb.event.MultipleProviders):
+ logger.info("multiple providers are available for %s%s (%s)", event._is_runtime and "runtime " or "",
+ event._item,
+ ", ".join(event._candidates))
+ logger.info("consider defining a PREFERRED_PROVIDER entry to match %s", event._item)
+ continue
+ if isinstance(event, bb.event.NoProvider):
+ return_value = 1
+ errors = errors + 1
+ if event._runtime:
+ r = "R"
+ else:
+ r = ""
+
+ if event._dependees:
+ logger.error("Nothing %sPROVIDES '%s' (but %s %sDEPENDS on or otherwise requires it)", r, event._item, ", ".join(event._dependees), r)
+ else:
+ logger.error("Nothing %sPROVIDES '%s'", r, event._item)
+ if event._reasons:
+ for reason in event._reasons:
+ logger.error("%s", reason)
+ continue
+
+ if isinstance(event, bb.runqueue.sceneQueueTaskStarted):
+ logger.info("Running setscene task %d of %d (%s)" % (event.stats.completed + event.stats.active + event.stats.failed + 1, event.stats.total, event.taskstring))
+ continue
+
+ if isinstance(event, bb.runqueue.runQueueTaskStarted):
+ if event.noexec:
+ tasktype = 'noexec task'
+ else:
+ tasktype = 'task'
+ logger.info("Running %s %s of %s (ID: %s, %s)",
+ tasktype,
+ event.stats.completed + event.stats.active +
+ event.stats.failed + 1,
+ event.stats.total, event.taskid, event.taskstring)
+ continue
+
+ if isinstance(event, bb.runqueue.runQueueTaskFailed):
+ taskfailures.append(event.taskstring)
+ logger.error("Task %s (%s) failed with exit code '%s'",
+ event.taskid, event.taskstring, event.exitcode)
+ continue
+
+ if isinstance(event, bb.runqueue.sceneQueueTaskFailed):
+ logger.warn("Setscene task %s (%s) failed with exit code '%s' - real task will be run instead",
+ event.taskid, event.taskstring, event.exitcode)
+ continue
+
+ if isinstance(event, bb.event.ConfigParsed):
+ # timestamp should be added for this
+ continue
+
+ if isinstance(event, bb.event.RecipeParsed):
+ # timestamp should be added for this
+ continue
+
+ if isinstance(event, bb.event.OperationStarted):
+ # timestamp should be added for this
+ continue
+
+ if isinstance(event, bb.event.OperationCompleted):
+ # timestamp should be added for this
+ # signal a complete operation
+ # calculate timing
+ continue
+
+ if isinstance(event, bb.event.DiskFull):
+ # trigger an error
+ continue
+
+ # ignore
+ if isinstance(event, (bb.event.BuildBase,
+ bb.event.StampUpdate,
+ bb.event.RecipePreFinalise,
+ bb.runqueue.runQueueEvent,
+ bb.runqueue.runQueueExitWait,
+ bb.event.OperationProgress)):
+ continue
+
+ logger.error("Unknown event: %s", event)
+
+ except EnvironmentError as ioerror:
+ termfilter.clearFooter()
+ # ignore interrupted io
+ if ioerror.args[0] == 4:
+ pass
+ except KeyboardInterrupt:
+ termfilter.clearFooter()
+ if main.shutdown == 1:
+ print("\nSecond Keyboard Interrupt, stopping...\n")
+ _, error = server.runCommand(["stateStop"])
+ if error:
+ logger.error("Unable to cleanly stop: %s" % error)
+ if main.shutdown == 0:
+ print("\nKeyboard Interrupt, closing down...\n")
+ interrupted = True
+ _, error = server.runCommand(["stateShutdown"])
+ if error:
+ logger.error("Unable to cleanly shutdown: %s" % error)
+ main.shutdown = main.shutdown + 1
+ pass
+
+ summary = ""
+ if taskfailures:
+ summary += pluralise("\nSummary: %s task failed:",
+ "\nSummary: %s tasks failed:", len(taskfailures))
+ for failure in taskfailures:
+ summary += "\n %s" % failure
+ if warnings:
+ summary += pluralise("\nSummary: There was %s WARNING message shown.",
+ "\nSummary: There were %s WARNING messages shown.", warnings)
+ if return_value:
+ summary += pluralise("\nSummary: There was %s ERROR message shown, returning a non-zero exit code.",
+ "\nSummary: There were %s ERROR messages shown, returning a non-zero exit code.", errors)
+ if summary:
+ print(summary)
+
+ if interrupted:
+ print("Execution was interrupted, returning a non-zero exit code.")
+ if return_value == 0:
+ return_value = 1
+
+ termfilter.finish()
+ wbhbhelper.write_in_database(wbhbhelper.tasks_information)
+
+ return return_value
+
--
1.8.1.2
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH 3/3] Create Data Store Interface (DSI) file
2013-06-12 15:11 ` [Webhob] " Calin Dragomir
(?)
@ 2013-06-13 15:48 ` Alex Damian
-1 siblings, 0 replies; 7+ messages in thread
From: Alex Damian @ 2013-06-13 15:48 UTC (permalink / raw)
To: bitbake-devel
Comments below.
Cheers,
Alex
On 06/12/2013 04:11 PM, Calin Dragomir wrote:
> This adds the first version of the DSI file. It uses the Knotty
> code for now, but inserts task related information into the database
> using the Django ORM
> ---
> bitbake/lib/bb/ui/dsi.py | 609 +++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 609 insertions(+)
> create mode 100644 bitbake/lib/bb/ui/dsi.py
>
> diff --git a/bitbake/lib/bb/ui/dsi.py b/bitbake/lib/bb/ui/dsi.py
> new file mode 100644
> index 0000000..4bcd509
> --- /dev/null
> +++ b/bitbake/lib/bb/ui/dsi.py
> @@ -0,0 +1,609 @@
> +#
> +# BitBake (No)TTY UI Implementation
> +#
> +# Handling output to TTYs or files (no TTY)
> +#
Please update file description and copyright.
> +# Copyright (C) 2006-2012 Richard Purdie
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License version 2 as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with this program; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +
> +from __future__ import division
> +
> +import os
> +import sys
> +import xmlrpclib
> +import logging
> +import progressbar
> +import signal
> +import bb.msg
> +import time
> +import fcntl
> +import struct
> +import copy
> +import datetime
> +
> +from bb.ui import uihelper
> +
> +logger = logging.getLogger("BitBake")
> +interactive = sys.stdout.isatty()
> +
> +class WebHOBHelper(object):
> +
> + def __init__(self):
> + self.configure_django()
> + self.task_order = 0
> + self.tasks_information = {}
> +
> + def configure_django(self):
> + import webhob.whbmain.settings as whb_django_settings
> + from django.core.management import setup_environ
> + setup_environ(whb_django_settings)
> + # Add webhob to sys path for importing modules
> + sys.path.append(os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), 'webhob'))
> +
> + def store_started_task(self, event, uuid):
> + self.task_order += 1
> + self.tasks_information[event.pid] = {
> + 'uuid': uuid,
> + 'task_id': 1,
> + 'task_executed': True,
> + 'order': self.task_order,
> + 'recipe': event._package,
> + 'task_name': event._task,
> + 'start_time': time.time(),
> + }
> +
> + def update_stored_tasks(self, event, uuid):
> + self.tasks_information[event.pid].update({
> + 'outcome': event.getDisplayName().lower(),
> + 'end_time': time.time(),
> + })
> +
> + def write_in_database(self, tasks_information):
> + # This needs to be imported after we have configured the Django settings file
> + from webhob.orm.models import Tasks
> + for pid in tasks_information.keys():
> + task = tasks_information[pid]
I'm not sure that there is any guarantee that any PID refers to a single
task.
Primary lookup key for tasks should be task-name & task-recipe combination.
> +
> + task_obj = Tasks.objects.create(uuid=task['uuid'],
> + order=task['order'],
> + outcome=task['outcome'],
> + recipe=task['recipe'],
> + task_name=task['task_name'],
> + elapsed_time=task['end_time'] - task['start_time'],
> + # The next lines are just for testing reasons
> + task_id=1,
> + path_to_sstate_obj='/home/calin',
> + source_url='/',
> + log_file='/',
> + work_directory='/',
> + script_type='python',
> + file_path='/',
> + line_number=55,
> + py_stack_trace='Testing traceback',
> + disk_io = 23.55633,
> + cpu_usage=43,
> + dependent_tasks='Task 1',
> + errors_no=0,
> + warnings_no=0,
> + error='None',
> + warning='None',
> + sstate_result='not_applicable')
> + task_obj.save()
> +
> +class BBProgress(progressbar.ProgressBar):
> + def __init__(self, msg, maxval):
> + self.msg = msg
> + widgets = [progressbar.Percentage(), ' ', progressbar.Bar(), ' ',
> + progressbar.ETA()]
> +
> + try:
> + self._resize_default = signal.getsignal(signal.SIGWINCH)
> + except:
> + self._resize_default = None
> + progressbar.ProgressBar.__init__(self, maxval, [self.msg + ": "] + widgets)
> +
> + def _handle_resize(self, signum, frame):
> + progressbar.ProgressBar._handle_resize(self, signum, frame)
> + if self._resize_default:
> + self._resize_default(signum, frame)
> + def finish(self):
> + progressbar.ProgressBar.finish(self)
> + if self._resize_default:
> + signal.signal(signal.SIGWINCH, self._resize_default)
> +
> +class NonInteractiveProgress(object):
> + fobj = sys.stdout
> +
> + def __init__(self, msg, maxval):
> + self.msg = msg
> + self.maxval = maxval
> +
> + def start(self):
> + self.fobj.write("%s..." % self.msg)
> + self.fobj.flush()
> + return self
> +
> + def update(self, value):
> + pass
> +
> + def finish(self):
> + self.fobj.write("done.\n")
> + self.fobj.flush()
> +
> +def new_progress(msg, maxval):
> + if interactive:
> + return BBProgress(msg, maxval)
> + else:
> + return NonInteractiveProgress(msg, maxval)
> +
> +def pluralise(singular, plural, qty):
> + if(qty == 1):
> + return singular % qty
> + else:
> + return plural % qty
> +
> +
> +class InteractConsoleLogFilter(logging.Filter):
> + def __init__(self, tf, format):
> + self.tf = tf
> + self.format = format
> +
> + def filter(self, record):
> + if record.levelno == self.format.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")):
> + return False
> + self.tf.clearFooter()
> + return True
> +
> +class TerminalFilter(object):
> + columns = 80
> +
> + def sigwinch_handle(self, signum, frame):
> + self.columns = self.getTerminalColumns()
> + if self._sigwinch_default:
> + self._sigwinch_default(signum, frame)
> +
> + def getTerminalColumns(self):
> + def ioctl_GWINSZ(fd):
> + try:
> + cr = struct.unpack('hh', fcntl.ioctl(fd, self.termios.TIOCGWINSZ, '1234'))
> + except:
> + return None
> + return cr
> + cr = ioctl_GWINSZ(sys.stdout.fileno())
> + if not cr:
> + try:
> + fd = os.open(os.ctermid(), os.O_RDONLY)
> + cr = ioctl_GWINSZ(fd)
> + os.close(fd)
> + except:
> + pass
> + if not cr:
> + try:
> + cr = (env['LINES'], env['COLUMNS'])
> + except:
> + cr = (25, 80)
> + return cr[1]
> +
> + def __init__(self, main, helper, console, format):
> + self.main = main
> + self.helper = helper
> + self.cuu = None
> + self.stdinbackup = None
> + self.interactive = sys.stdout.isatty()
> + self.footer_present = False
> + self.lastpids = []
> +
> + if not self.interactive:
> + return
> +
> + try:
> + import curses
> + except ImportError:
> + sys.exit("FATAL: The knotty ui could not load the required curses python module.")
> +
> + import termios
> + self.curses = curses
> + self.termios = termios
> + try:
> + fd = sys.stdin.fileno()
> + self.stdinbackup = termios.tcgetattr(fd)
> + new = copy.deepcopy(self.stdinbackup)
> + new[3] = new[3] & ~termios.ECHO
> + termios.tcsetattr(fd, termios.TCSADRAIN, new)
> + curses.setupterm()
> + if curses.tigetnum("colors") > 2:
> + format.enable_color()
> + self.ed = curses.tigetstr("ed")
> + if self.ed:
> + self.cuu = curses.tigetstr("cuu")
> + try:
> + self._sigwinch_default = signal.getsignal(signal.SIGWINCH)
> + signal.signal(signal.SIGWINCH, self.sigwinch_handle)
> + except:
> + pass
> + self.columns = self.getTerminalColumns()
> + except:
> + self.cuu = None
> + console.addFilter(InteractConsoleLogFilter(self, format))
> +
> + def clearFooter(self):
> + if self.footer_present:
> + lines = self.footer_present
> + sys.stdout.write(self.curses.tparm(self.cuu, lines))
> + sys.stdout.write(self.curses.tparm(self.ed))
> + self.footer_present = False
> +
> + def updateFooter(self):
> + if not self.cuu:
> + return
> + activetasks = self.helper.running_tasks
> + failedtasks = self.helper.failed_tasks
> + runningpids = self.helper.running_pids
> + if self.footer_present and (self.lastcount == self.helper.tasknumber_current) and (self.lastpids == runningpids):
> + return
> + if self.footer_present:
> + self.clearFooter()
> + if (not self.helper.tasknumber_total or self.helper.tasknumber_current == self.helper.tasknumber_total) and not len(activetasks):
> + return
> + tasks = []
> + for t in runningpids:
> + tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
> +
> + if self.main.shutdown:
> + content = "Waiting for %s running tasks to finish:" % len(activetasks)
> + elif not len(activetasks):
> + content = "No currently running tasks (%s of %s)" % (self.helper.tasknumber_current, self.helper.tasknumber_total)
> + else:
> + content = "Currently %s running tasks (%s of %s):" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total)
> + print(content)
> + lines = 1 + int(len(content) / (self.columns + 1))
> + for tasknum, task in enumerate(tasks):
> + content = "%s: %s" % (tasknum, task)
> + print(content)
> + lines = lines + 1 + int(len(content) / (self.columns + 1))
> + self.footer_present = lines
> + self.lastpids = runningpids[:]
> + self.lastcount = self.helper.tasknumber_current
> +
> + def finish(self):
> + if self.stdinbackup:
> + fd = sys.stdin.fileno()
> + self.termios.tcsetattr(fd, self.termios.TCSADRAIN, self.stdinbackup)
> +
> +def main(server, eventHandler, params, tf = TerminalFilter):
> + print "This is running in DSI mode"
> +
> + # Generate an unique ID for this build
> + # TODO: for multiple build commands this might not work as expected
> + import uuid
> + uuid = sessionid = str(uuid.uuid4())
> +
> + # Get values of variables which control our output
> + includelogs, error = server.runCommand(["getVariable", "BBINCLUDELOGS"])
> + if error:
> + logger.error("Unable to get the value of BBINCLUDELOGS variable: %s" % error)
> + return 1
> + loglines, error = server.runCommand(["getVariable", "BBINCLUDELOGS_LINES"])
> + if error:
> + logger.error("Unable to get the value of BBINCLUDELOGS_LINES variable: %s" % error)
> + return 1
> + consolelogfile, error = server.runCommand(["getVariable", "BB_CONSOLELOG"])
> + if error:
> + logger.error("Unable to get the value of BB_CONSOLELOG variable: %s" % error)
> + return 1
> +
> + if sys.stdin.isatty() and sys.stdout.isatty():
> + log_exec_tty = True
> + else:
> + log_exec_tty = False
> +
> + helper = uihelper.BBUIHelper()
> +
> + console = logging.StreamHandler(sys.stdout)
> + format_str = "%(levelname)s: %(message)s"
> + format = bb.msg.BBLogFormatter(format_str)
> + bb.msg.addDefaultlogFilter(console)
> + console.setFormatter(format)
> + logger.addHandler(console)
> +
> + if consolelogfile and not params.options.show_environment:
> + bb.utils.mkdirhier(os.path.dirname(consolelogfile))
> + conlogformat = bb.msg.BBLogFormatter(format_str)
> + consolelog = logging.FileHandler(consolelogfile)
> + bb.msg.addDefaultlogFilter(consolelog)
> + consolelog.setFormatter(conlogformat)
> + logger.addHandler(consolelog)
> +
> + try:
> + params.updateFromServer(server)
> + cmdline = params.parseActions()
> + if not cmdline:
> + print("Nothing to do. Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information.")
> + return 1
> + if 'msg' in cmdline and cmdline['msg']:
> + logger.error(cmdline['msg'])
> + return 1
> +
> + ret, error = server.runCommand(cmdline['action'])
> + if error:
> + logger.error("Command '%s' failed: %s" % (cmdline, error))
> + return 1
> + elif ret != True:
> + logger.error("Command '%s' failed: returned %s" % (cmdline, ret))
> + return 1
> + except xmlrpclib.Fault as x:
> + logger.error("XMLRPC Fault getting commandline:\n %s" % x)
> + return 1
> +
> + parseprogress = None
> + cacheprogress = None
> + main.shutdown = 0
> + interrupted = False
> + return_value = 0
> + errors = 0
> + warnings = 0
> + taskfailures = []
> + wbhbhelper = WebHOBHelper()
> +
> + termfilter = tf(main, helper, console, format)
> +
> + while True:
> + try:
> + termfilter.updateFooter()
> + event = eventHandler.waitEvent(0.25)
> +
> + if event is None:
> + if main.shutdown > 1:
> + break
> + continue
> +
> + helper.eventHandler(event)
> +
> + if isinstance(event, bb.runqueue.runQueueExitWait):
> + if not main.shutdown:
> + main.shutdown = 1
> +
> + if isinstance(event, bb.build.TaskStarted):
> + wbhbhelper.store_started_task(event, uuid)
> +
> + if isinstance(event, (bb.build.TaskSucceeded, bb.build.TaskFailedSilent, bb.build.TaskFailed)):
> + wbhbhelper.update_stored_tasks(event, uuid)
> +
> + if isinstance(event, bb.event.LogExecTTY):
> + if log_exec_tty:
> + tries = event.retries
> + while tries:
> + print("Trying to run: %s" % event.prog)
> + if os.system(event.prog) == 0:
> + break
> + time.sleep(event.sleep_delay)
> + tries -= 1
> + if tries:
> + continue
> + logger.warn(event.msg)
> + continue
> +
> + if isinstance(event, logging.LogRecord):
> + if event.levelno >= format.ERROR:
> + errors = errors + 1
> + return_value = 1
> + elif event.levelno == format.WARNING:
> + warnings = warnings + 1
> + # For "normal" logging conditions, don't show note logs from tasks
> + # but do show them if the user has changed the default log level to
> + # include verbose/debug messages
> + if event.taskpid != 0 and event.levelno <= format.NOTE:
> + continue
> + logger.handle(event)
> + continue
> +
> + if isinstance(event, bb.build.TaskFailed):
> + return_value = 1
> + logfile = event.logfile
> + if logfile and os.path.exists(logfile):
> + termfilter.clearFooter()
> + bb.error("Logfile of failure stored in: %s" % logfile)
> + if includelogs and not event.errprinted:
> + print("Log data follows:")
> + f = open(logfile, "r")
> + lines = []
> + while True:
> + l = f.readline()
> + if l == '':
> + break
> + l = l.rstrip()
> + if loglines:
> + lines.append(' | %s' % l)
> + if len(lines) > int(loglines):
> + lines.pop(0)
> + else:
> + print('| %s' % l)
> + f.close()
> + if lines:
> + for line in lines:
> + print(line)
> + if isinstance(event, bb.build.TaskBase):
> + logger.info(event._message)
> + continue
> + if isinstance(event, bb.event.ParseStarted):
> + if event.total == 0:
> + continue
> + parseprogress = new_progress("Parsing recipes", event.total).start()
> + continue
> + if isinstance(event, bb.event.ParseProgress):
> + parseprogress.update(event.current)
> + continue
> + if isinstance(event, bb.event.ParseCompleted):
> + if not parseprogress:
> + continue
> +
> + parseprogress.finish()
> + print(("Parsing of %d .bb files complete (%d cached, %d parsed). %d targets, %d skipped, %d masked, %d errors."
> + % ( event.total, event.cached, event.parsed, event.virtuals, event.skipped, event.masked, event.errors)))
> + continue
> +
> + if isinstance(event, bb.event.CacheLoadStarted):
> + cacheprogress = new_progress("Loading cache", event.total).start()
> + continue
> + if isinstance(event, bb.event.CacheLoadProgress):
> + cacheprogress.update(event.current)
> + continue
> + if isinstance(event, bb.event.CacheLoadCompleted):
> + cacheprogress.finish()
> + print("Loaded %d entries from dependency cache." % event.num_entries)
> + continue
> +
> + if isinstance(event, bb.command.CommandFailed):
> + return_value = event.exitcode
> + errors = errors + 1
> + logger.error("Command execution failed: %s", event.error)
> + main.shutdown = 2
DSI should never exit.
> + continue
> + if isinstance(event, bb.command.CommandExit):
> + if not return_value:
> + return_value = event.exitcode
> + continue
> + if isinstance(event, (bb.command.CommandCompleted, bb.cooker.CookerExit)):
> + main.shutdown = 2
DSI should never exit.
> + continue
> + if isinstance(event, bb.event.MultipleProviders):
> + logger.info("multiple providers are available for %s%s (%s)", event._is_runtime and "runtime " or "",
> + event._item,
> + ", ".join(event._candidates))
> + logger.info("consider defining a PREFERRED_PROVIDER entry to match %s", event._item)
> + continue
> + if isinstance(event, bb.event.NoProvider):
> + return_value = 1
> + errors = errors + 1
> + if event._runtime:
> + r = "R"
> + else:
> + r = ""
> +
> + if event._dependees:
> + logger.error("Nothing %sPROVIDES '%s' (but %s %sDEPENDS on or otherwise requires it)", r, event._item, ", ".join(event._dependees), r)
> + else:
> + logger.error("Nothing %sPROVIDES '%s'", r, event._item)
> + if event._reasons:
> + for reason in event._reasons:
> + logger.error("%s", reason)
> + continue
> +
> + if isinstance(event, bb.runqueue.sceneQueueTaskStarted):
> + logger.info("Running setscene task %d of %d (%s)" % (event.stats.completed + event.stats.active + event.stats.failed + 1, event.stats.total, event.taskstring))
> + continue
> +
> + if isinstance(event, bb.runqueue.runQueueTaskStarted):
> + if event.noexec:
> + tasktype = 'noexec task'
> + else:
> + tasktype = 'task'
> + logger.info("Running %s %s of %s (ID: %s, %s)",
> + tasktype,
> + event.stats.completed + event.stats.active +
> + event.stats.failed + 1,
> + event.stats.total, event.taskid, event.taskstring)
> + continue
> +
> + if isinstance(event, bb.runqueue.runQueueTaskFailed):
> + taskfailures.append(event.taskstring)
> + logger.error("Task %s (%s) failed with exit code '%s'",
> + event.taskid, event.taskstring, event.exitcode)
> + continue
> +
> + if isinstance(event, bb.runqueue.sceneQueueTaskFailed):
> + logger.warn("Setscene task %s (%s) failed with exit code '%s' - real task will be run instead",
> + event.taskid, event.taskstring, event.exitcode)
> + continue
> +
> + if isinstance(event, bb.event.ConfigParsed):
> + # timestamp should be added for this
> + continue
> +
> + if isinstance(event, bb.event.RecipeParsed):
> + # timestamp should be added for this
> + continue
> +
> + if isinstance(event, bb.event.OperationStarted):
> + # timestamp should be added for this
> + continue
> +
> + if isinstance(event, bb.event.OperationCompleted):
> + # timestamp should be added for this
> + # signal a complete operation
> + # calculate timing
> + continue
> +
> + if isinstance(event, bb.event.DiskFull):
> + # trigger an error
> + continue
> +
> + # ignore
> + if isinstance(event, (bb.event.BuildBase,
> + bb.event.StampUpdate,
> + bb.event.RecipePreFinalise,
> + bb.runqueue.runQueueEvent,
> + bb.runqueue.runQueueExitWait,
> + bb.event.OperationProgress)):
> + continue
> +
> + logger.error("Unknown event: %s", event)
> +
> + except EnvironmentError as ioerror:
> + termfilter.clearFooter()
> + # ignore interrupted io
> + if ioerror.args[0] == 4:
> + pass
> + except KeyboardInterrupt:
> + termfilter.clearFooter()
> + if main.shutdown == 1:
> + print("\nSecond Keyboard Interrupt, stopping...\n")
> + _, error = server.runCommand(["stateStop"])
> + if error:
> + logger.error("Unable to cleanly stop: %s" % error)
> + if main.shutdown == 0:
> + print("\nKeyboard Interrupt, closing down...\n")
> + interrupted = True
> + _, error = server.runCommand(["stateShutdown"])
> + if error:
> + logger.error("Unable to cleanly shutdown: %s" % error)
> + main.shutdown = main.shutdown + 1
> + pass
> +
> + summary = ""
> + if taskfailures:
> + summary += pluralise("\nSummary: %s task failed:",
> + "\nSummary: %s tasks failed:", len(taskfailures))
> + for failure in taskfailures:
> + summary += "\n %s" % failure
> + if warnings:
> + summary += pluralise("\nSummary: There was %s WARNING message shown.",
> + "\nSummary: There were %s WARNING messages shown.", warnings)
> + if return_value:
> + summary += pluralise("\nSummary: There was %s ERROR message shown, returning a non-zero exit code.",
> + "\nSummary: There were %s ERROR messages shown, returning a non-zero exit code.", errors)
> + if summary:
> + print(summary)
> +
> + if interrupted:
> + print("Execution was interrupted, returning a non-zero exit code.")
> + if return_value == 0:
> + return_value = 1
> +
> + termfilter.finish()
We don't expect to ever get here - DSI should never exit.
> + wbhbhelper.write_in_database(wbhbhelper.tasks_information)
Probably we need to write tasks as they become complete (i.e.
TaskFailed or TaskSucceeded or TaskInvalid comes through)
and then clear the memory. This will prevent memory overusage.
> +
> + return return_value
> +
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2013-06-13 15:48 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-06-12 15:11 [PATCH 1/3] Create main WEBHOB project Calin Dragomir
2013-06-12 15:11 ` [Webhob] " Calin Dragomir
2013-06-12 15:11 ` [PATCH 2/3] Create Tasks Django model Calin Dragomir
2013-06-12 15:11 ` [Webhob] " Calin Dragomir
2013-06-12 15:11 ` [PATCH 3/3] Create Data Store Interface (DSI) file Calin Dragomir
2013-06-12 15:11 ` [Webhob] " Calin Dragomir
2013-06-13 15:48 ` Alex Damian
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.