* [PATCH 1/3] Create main WEBHOB project @ 2013-06-12 15:11 ` Calin Dragomir 0 siblings, 0 replies; 7+ messages in thread From: Calin Dragomir @ 2013-06-12 15:11 UTC (permalink / raw) To: webhob, bitbake-devel This creates a basic Django project where the WEBHOB code will live. The main project had to be renamed from webhob to whbmain in order to avoid any import issues that might occur. --- bitbake/lib/webhob/__init__.py | 0 bitbake/lib/webhob/manage.py | 10 +++ bitbake/lib/webhob/whbmain/__init__.py | 0 bitbake/lib/webhob/whbmain/settings.py | 156 +++++++++++++++++++++++++++++++++ bitbake/lib/webhob/whbmain/urls.py | 17 ++++ bitbake/lib/webhob/whbmain/wsgi.py | 32 +++++++ 6 files changed, 215 insertions(+) create mode 100644 bitbake/lib/webhob/__init__.py create mode 100755 bitbake/lib/webhob/manage.py create mode 100644 bitbake/lib/webhob/whbmain/__init__.py create mode 100644 bitbake/lib/webhob/whbmain/settings.py create mode 100644 bitbake/lib/webhob/whbmain/urls.py create mode 100644 bitbake/lib/webhob/whbmain/wsgi.py diff --git a/bitbake/lib/webhob/__init__.py b/bitbake/lib/webhob/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/bitbake/lib/webhob/manage.py b/bitbake/lib/webhob/manage.py new file mode 100755 index 0000000..1b2c1a7 --- /dev/null +++ b/bitbake/lib/webhob/manage.py @@ -0,0 +1,10 @@ +#!/usr/bin/env python +import os +import sys + +if __name__ == "__main__": + os.environ.setdefault("DJANGO_SETTINGS_MODULE", "whbmain.settings") + + from django.core.management import execute_from_command_line + + execute_from_command_line(sys.argv) diff --git a/bitbake/lib/webhob/whbmain/__init__.py b/bitbake/lib/webhob/whbmain/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/bitbake/lib/webhob/whbmain/settings.py b/bitbake/lib/webhob/whbmain/settings.py new file mode 100644 index 0000000..bc59543 --- /dev/null +++ b/bitbake/lib/webhob/whbmain/settings.py @@ -0,0 +1,156 @@ +# Django settings for webhob project. + +DEBUG = True +TEMPLATE_DEBUG = DEBUG + +ADMINS = ( + # ('Your Name', 'your_email@example.com'), +) + +MANAGERS = ADMINS + +DATABASES = { + 'default': { + 'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'. + 'NAME': 'wbhob', # Or path to database file if using sqlite3. + # The following settings are not used with sqlite3: + 'USER': 'root', + 'PASSWORD': '', + 'HOST': '127.0.0.1', # Empty for localhost through domain sockets or '127.0.0.1' for localhost through TCP. + 'PORT': '3306', # Set to empty string for default. + } +} + +# Hosts/domain names that are valid for this site; required if DEBUG is False +# See https://docs.djangoproject.com/en/1.5/ref/settings/#allowed-hosts +ALLOWED_HOSTS = [] + +# Local time zone for this installation. Choices can be found here: +# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name +# although not all choices may be available on all operating systems. +# In a Windows environment this must be set to your system time zone. +TIME_ZONE = 'America/Chicago' + +# Language code for this installation. All choices can be found here: +# http://www.i18nguy.com/unicode/language-identifiers.html +LANGUAGE_CODE = 'en-us' + +SITE_ID = 1 + +# If you set this to False, Django will make some optimizations so as not +# to load the internationalization machinery. +USE_I18N = True + +# If you set this to False, Django will not format dates, numbers and +# calendars according to the current locale. +USE_L10N = True + +# If you set this to False, Django will not use timezone-aware datetimes. +USE_TZ = True + +# Absolute filesystem path to the directory that will hold user-uploaded files. +# Example: "/var/www/example.com/media/" +MEDIA_ROOT = '' + +# URL that handles the media served from MEDIA_ROOT. Make sure to use a +# trailing slash. +# Examples: "http://example.com/media/", "http://media.example.com/" +MEDIA_URL = '' + +# Absolute path to the directory static files should be collected to. +# Don't put anything in this directory yourself; store your static files +# in apps' "static/" subdirectories and in STATICFILES_DIRS. +# Example: "/var/www/example.com/static/" +STATIC_ROOT = '' + +# URL prefix for static files. +# Example: "http://example.com/static/", "http://static.example.com/" +STATIC_URL = '/static/' + +# Additional locations of static files +STATICFILES_DIRS = ( + # Put strings here, like "/home/html/static" or "C:/www/django/static". + # Always use forward slashes, even on Windows. + # Don't forget to use absolute paths, not relative paths. +) + +# List of finder classes that know how to find static files in +# various locations. +STATICFILES_FINDERS = ( + 'django.contrib.staticfiles.finders.FileSystemFinder', + 'django.contrib.staticfiles.finders.AppDirectoriesFinder', +# 'django.contrib.staticfiles.finders.DefaultStorageFinder', +) + +# Make this unique, and don't share it with anybody. +SECRET_KEY = 'fy@5+5w9zn5wa0vsegn1c8p_6w&pa_ms-4t0cna1vle79nzlkn' + +# List of callables that know how to import templates from various sources. +TEMPLATE_LOADERS = ( + 'django.template.loaders.filesystem.Loader', + 'django.template.loaders.app_directories.Loader', +# 'django.template.loaders.eggs.Loader', +) + +MIDDLEWARE_CLASSES = ( + 'django.middleware.common.CommonMiddleware', + 'django.contrib.sessions.middleware.SessionMiddleware', + 'django.middleware.csrf.CsrfViewMiddleware', + 'django.contrib.auth.middleware.AuthenticationMiddleware', + 'django.contrib.messages.middleware.MessageMiddleware', + # Uncomment the next line for simple clickjacking protection: + # 'django.middleware.clickjacking.XFrameOptionsMiddleware', +) + +ROOT_URLCONF = 'whbmain.urls' + +# Python dotted path to the WSGI application used by Django's runserver. +WSGI_APPLICATION = 'whbmain.wsgi.application' + +TEMPLATE_DIRS = ( + # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates". + # Always use forward slashes, even on Windows. + # Don't forget to use absolute paths, not relative paths. +) + +INSTALLED_APPS = ( + #'django.contrib.auth', + #'django.contrib.contenttypes', + #'django.contrib.sessions', + #'django.contrib.sites', + #'django.contrib.messages', + #'django.contrib.staticfiles', + # Uncomment the next line to enable the admin: + # 'django.contrib.admin', + # Uncomment the next line to enable admin documentation: + # 'django.contrib.admindocs', +) + +# A sample logging configuration. The only tangible logging +# performed by this configuration is to send an email to +# the site admins on every HTTP 500 error when DEBUG=False. +# See http://docs.djangoproject.com/en/dev/topics/logging for +# more details on how to customize your logging configuration. +LOGGING = { + 'version': 1, + 'disable_existing_loggers': False, + 'filters': { + 'require_debug_false': { + '()': 'django.utils.log.RequireDebugFalse' + } + }, + 'handlers': { + 'mail_admins': { + 'level': 'ERROR', + 'filters': ['require_debug_false'], + 'class': 'django.utils.log.AdminEmailHandler' + } + }, + 'loggers': { + 'django.request': { + 'handlers': ['mail_admins'], + 'level': 'ERROR', + 'propagate': True, + }, + } +} diff --git a/bitbake/lib/webhob/whbmain/urls.py b/bitbake/lib/webhob/whbmain/urls.py new file mode 100644 index 0000000..04e0352 --- /dev/null +++ b/bitbake/lib/webhob/whbmain/urls.py @@ -0,0 +1,17 @@ +from django.conf.urls import patterns, include, url + +# Uncomment the next two lines to enable the admin: +# from django.contrib import admin +# admin.autodiscover() + +urlpatterns = patterns('', + # Examples: + # url(r'^$', 'webhob.views.home', name='home'), + # url(r'^webhob/', include('webhob.foo.urls')), + + # Uncomment the admin/doc line below to enable admin documentation: + # url(r'^admin/doc/', include('django.contrib.admindocs.urls')), + + # Uncomment the next line to enable the admin: + # url(r'^admin/', include(admin.site.urls)), +) diff --git a/bitbake/lib/webhob/whbmain/wsgi.py b/bitbake/lib/webhob/whbmain/wsgi.py new file mode 100644 index 0000000..9595eda --- /dev/null +++ b/bitbake/lib/webhob/whbmain/wsgi.py @@ -0,0 +1,32 @@ +""" +WSGI config for webhob project. + +This module contains the WSGI application used by Django's development server +and any production WSGI deployments. It should expose a module-level variable +named ``application``. Django's ``runserver`` and ``runfcgi`` commands discover +this application via the ``WSGI_APPLICATION`` setting. + +Usually you will have the standard Django WSGI application here, but it also +might make sense to replace the whole Django WSGI application with a custom one +that later delegates to the Django one. For example, you could introduce WSGI +middleware here, or combine a Django application with an application of another +framework. + +""" +import os + +# We defer to a DJANGO_SETTINGS_MODULE already in the environment. This breaks +# if running multiple sites in the same mod_wsgi process. To fix this, use +# mod_wsgi daemon mode with each site in its own daemon process, or use +# os.environ["DJANGO_SETTINGS_MODULE"] = "webhob.settings" +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "wbhmain.settings") + +# This application object is used by any WSGI server configured to use this +# file. This includes Django's development server, if the WSGI_APPLICATION +# setting points here. +from django.core.wsgi import get_wsgi_application +application = get_wsgi_application() + +# Apply WSGI middleware here. +# from helloworld.wsgi import HelloWorldApplication +# application = HelloWorldApplication(application) -- 1.8.1.2 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* [Webhob] [PATCH 1/3] Create main WEBHOB project @ 2013-06-12 15:11 ` Calin Dragomir 0 siblings, 0 replies; 7+ messages in thread From: Calin Dragomir @ 2013-06-12 15:11 UTC (permalink / raw) To: webhob, bitbake-devel This creates a basic Django project where the WEBHOB code will live. The main project had to be renamed from webhob to whbmain in order to avoid any import issues that might occur. --- bitbake/lib/webhob/__init__.py | 0 bitbake/lib/webhob/manage.py | 10 +++ bitbake/lib/webhob/whbmain/__init__.py | 0 bitbake/lib/webhob/whbmain/settings.py | 156 +++++++++++++++++++++++++++++++++ bitbake/lib/webhob/whbmain/urls.py | 17 ++++ bitbake/lib/webhob/whbmain/wsgi.py | 32 +++++++ 6 files changed, 215 insertions(+) create mode 100644 bitbake/lib/webhob/__init__.py create mode 100755 bitbake/lib/webhob/manage.py create mode 100644 bitbake/lib/webhob/whbmain/__init__.py create mode 100644 bitbake/lib/webhob/whbmain/settings.py create mode 100644 bitbake/lib/webhob/whbmain/urls.py create mode 100644 bitbake/lib/webhob/whbmain/wsgi.py diff --git a/bitbake/lib/webhob/__init__.py b/bitbake/lib/webhob/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/bitbake/lib/webhob/manage.py b/bitbake/lib/webhob/manage.py new file mode 100755 index 0000000..1b2c1a7 --- /dev/null +++ b/bitbake/lib/webhob/manage.py @@ -0,0 +1,10 @@ +#!/usr/bin/env python +import os +import sys + +if __name__ == "__main__": + os.environ.setdefault("DJANGO_SETTINGS_MODULE", "whbmain.settings") + + from django.core.management import execute_from_command_line + + execute_from_command_line(sys.argv) diff --git a/bitbake/lib/webhob/whbmain/__init__.py b/bitbake/lib/webhob/whbmain/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/bitbake/lib/webhob/whbmain/settings.py b/bitbake/lib/webhob/whbmain/settings.py new file mode 100644 index 0000000..bc59543 --- /dev/null +++ b/bitbake/lib/webhob/whbmain/settings.py @@ -0,0 +1,156 @@ +# Django settings for webhob project. + +DEBUG = True +TEMPLATE_DEBUG = DEBUG + +ADMINS = ( + # ('Your Name', 'your_email@example.com'), +) + +MANAGERS = ADMINS + +DATABASES = { + 'default': { + 'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'. + 'NAME': 'wbhob', # Or path to database file if using sqlite3. + # The following settings are not used with sqlite3: + 'USER': 'root', + 'PASSWORD': '', + 'HOST': '127.0.0.1', # Empty for localhost through domain sockets or '127.0.0.1' for localhost through TCP. + 'PORT': '3306', # Set to empty string for default. + } +} + +# Hosts/domain names that are valid for this site; required if DEBUG is False +# See https://docs.djangoproject.com/en/1.5/ref/settings/#allowed-hosts +ALLOWED_HOSTS = [] + +# Local time zone for this installation. Choices can be found here: +# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name +# although not all choices may be available on all operating systems. +# In a Windows environment this must be set to your system time zone. +TIME_ZONE = 'America/Chicago' + +# Language code for this installation. All choices can be found here: +# http://www.i18nguy.com/unicode/language-identifiers.html +LANGUAGE_CODE = 'en-us' + +SITE_ID = 1 + +# If you set this to False, Django will make some optimizations so as not +# to load the internationalization machinery. +USE_I18N = True + +# If you set this to False, Django will not format dates, numbers and +# calendars according to the current locale. +USE_L10N = True + +# If you set this to False, Django will not use timezone-aware datetimes. +USE_TZ = True + +# Absolute filesystem path to the directory that will hold user-uploaded files. +# Example: "/var/www/example.com/media/" +MEDIA_ROOT = '' + +# URL that handles the media served from MEDIA_ROOT. Make sure to use a +# trailing slash. +# Examples: "http://example.com/media/", "http://media.example.com/" +MEDIA_URL = '' + +# Absolute path to the directory static files should be collected to. +# Don't put anything in this directory yourself; store your static files +# in apps' "static/" subdirectories and in STATICFILES_DIRS. +# Example: "/var/www/example.com/static/" +STATIC_ROOT = '' + +# URL prefix for static files. +# Example: "http://example.com/static/", "http://static.example.com/" +STATIC_URL = '/static/' + +# Additional locations of static files +STATICFILES_DIRS = ( + # Put strings here, like "/home/html/static" or "C:/www/django/static". + # Always use forward slashes, even on Windows. + # Don't forget to use absolute paths, not relative paths. +) + +# List of finder classes that know how to find static files in +# various locations. +STATICFILES_FINDERS = ( + 'django.contrib.staticfiles.finders.FileSystemFinder', + 'django.contrib.staticfiles.finders.AppDirectoriesFinder', +# 'django.contrib.staticfiles.finders.DefaultStorageFinder', +) + +# Make this unique, and don't share it with anybody. +SECRET_KEY = 'fy@5+5w9zn5wa0vsegn1c8p_6w&pa_ms-4t0cna1vle79nzlkn' + +# List of callables that know how to import templates from various sources. +TEMPLATE_LOADERS = ( + 'django.template.loaders.filesystem.Loader', + 'django.template.loaders.app_directories.Loader', +# 'django.template.loaders.eggs.Loader', +) + +MIDDLEWARE_CLASSES = ( + 'django.middleware.common.CommonMiddleware', + 'django.contrib.sessions.middleware.SessionMiddleware', + 'django.middleware.csrf.CsrfViewMiddleware', + 'django.contrib.auth.middleware.AuthenticationMiddleware', + 'django.contrib.messages.middleware.MessageMiddleware', + # Uncomment the next line for simple clickjacking protection: + # 'django.middleware.clickjacking.XFrameOptionsMiddleware', +) + +ROOT_URLCONF = 'whbmain.urls' + +# Python dotted path to the WSGI application used by Django's runserver. +WSGI_APPLICATION = 'whbmain.wsgi.application' + +TEMPLATE_DIRS = ( + # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates". + # Always use forward slashes, even on Windows. + # Don't forget to use absolute paths, not relative paths. +) + +INSTALLED_APPS = ( + #'django.contrib.auth', + #'django.contrib.contenttypes', + #'django.contrib.sessions', + #'django.contrib.sites', + #'django.contrib.messages', + #'django.contrib.staticfiles', + # Uncomment the next line to enable the admin: + # 'django.contrib.admin', + # Uncomment the next line to enable admin documentation: + # 'django.contrib.admindocs', +) + +# A sample logging configuration. The only tangible logging +# performed by this configuration is to send an email to +# the site admins on every HTTP 500 error when DEBUG=False. +# See http://docs.djangoproject.com/en/dev/topics/logging for +# more details on how to customize your logging configuration. +LOGGING = { + 'version': 1, + 'disable_existing_loggers': False, + 'filters': { + 'require_debug_false': { + '()': 'django.utils.log.RequireDebugFalse' + } + }, + 'handlers': { + 'mail_admins': { + 'level': 'ERROR', + 'filters': ['require_debug_false'], + 'class': 'django.utils.log.AdminEmailHandler' + } + }, + 'loggers': { + 'django.request': { + 'handlers': ['mail_admins'], + 'level': 'ERROR', + 'propagate': True, + }, + } +} diff --git a/bitbake/lib/webhob/whbmain/urls.py b/bitbake/lib/webhob/whbmain/urls.py new file mode 100644 index 0000000..04e0352 --- /dev/null +++ b/bitbake/lib/webhob/whbmain/urls.py @@ -0,0 +1,17 @@ +from django.conf.urls import patterns, include, url + +# Uncomment the next two lines to enable the admin: +# from django.contrib import admin +# admin.autodiscover() + +urlpatterns = patterns('', + # Examples: + # url(r'^$', 'webhob.views.home', name='home'), + # url(r'^webhob/', include('webhob.foo.urls')), + + # Uncomment the admin/doc line below to enable admin documentation: + # url(r'^admin/doc/', include('django.contrib.admindocs.urls')), + + # Uncomment the next line to enable the admin: + # url(r'^admin/', include(admin.site.urls)), +) diff --git a/bitbake/lib/webhob/whbmain/wsgi.py b/bitbake/lib/webhob/whbmain/wsgi.py new file mode 100644 index 0000000..9595eda --- /dev/null +++ b/bitbake/lib/webhob/whbmain/wsgi.py @@ -0,0 +1,32 @@ +""" +WSGI config for webhob project. + +This module contains the WSGI application used by Django's development server +and any production WSGI deployments. It should expose a module-level variable +named ``application``. Django's ``runserver`` and ``runfcgi`` commands discover +this application via the ``WSGI_APPLICATION`` setting. + +Usually you will have the standard Django WSGI application here, but it also +might make sense to replace the whole Django WSGI application with a custom one +that later delegates to the Django one. For example, you could introduce WSGI +middleware here, or combine a Django application with an application of another +framework. + +""" +import os + +# We defer to a DJANGO_SETTINGS_MODULE already in the environment. This breaks +# if running multiple sites in the same mod_wsgi process. To fix this, use +# mod_wsgi daemon mode with each site in its own daemon process, or use +# os.environ["DJANGO_SETTINGS_MODULE"] = "webhob.settings" +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "wbhmain.settings") + +# This application object is used by any WSGI server configured to use this +# file. This includes Django's development server, if the WSGI_APPLICATION +# setting points here. +from django.core.wsgi import get_wsgi_application +application = get_wsgi_application() + +# Apply WSGI middleware here. +# from helloworld.wsgi import HelloWorldApplication +# application = HelloWorldApplication(application) -- 1.8.1.2 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 2/3] Create Tasks Django model 2013-06-12 15:11 ` [Webhob] " Calin Dragomir @ 2013-06-12 15:11 ` Calin Dragomir -1 siblings, 0 replies; 7+ messages in thread From: Calin Dragomir @ 2013-06-12 15:11 UTC (permalink / raw) To: webhob, bitbake-devel This will be used to store Tasks related information in the database using Django ORM. --- bitbake/lib/webhob/orm/__init__.py | 0 bitbake/lib/webhob/orm/models.py | 50 ++++++++++++++++++++++++++++++++++ bitbake/lib/webhob/orm/tests.py | 16 +++++++++++ bitbake/lib/webhob/orm/views.py | 1 + bitbake/lib/webhob/whbmain/settings.py | 1 + 5 files changed, 68 insertions(+) create mode 100644 bitbake/lib/webhob/orm/__init__.py create mode 100644 bitbake/lib/webhob/orm/models.py create mode 100644 bitbake/lib/webhob/orm/tests.py create mode 100644 bitbake/lib/webhob/orm/views.py diff --git a/bitbake/lib/webhob/orm/__init__.py b/bitbake/lib/webhob/orm/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/bitbake/lib/webhob/orm/models.py b/bitbake/lib/webhob/orm/models.py new file mode 100644 index 0000000..13eadd4 --- /dev/null +++ b/bitbake/lib/webhob/orm/models.py @@ -0,0 +1,50 @@ +from django.db import models + +class Tasks(models.Model): + + SSTATE_RESULT = ( + ('not_applicable', 'Not Applicable'), # For rest of tasks, but they still need checking. + ('unavailable', 'Unavailable'), # it is a miss + ('failed', 'Failed'), # there was a pkg, but the script failed + ('restored', 'Restored'), # succesfully restored + ) + + TASK_CODING = ( + ('python', 'Python'), + ('shell', 'Shell'), + ) + + TASK_OUTCOME = ( + ('succeded', 'Succeded'), + ('failed', 'Failed'), + ('existing', 'Existing'), + ('sstate', 'Sstate'), + ('covered', 'Covered'), + ) + + uuid = models.CharField(max_length=100) + task_id = models.IntegerField() + order = models.IntegerField() + task_executed = models.BooleanField() # True means Executed, False means Prebuilt + outcome = models.CharField(max_length=50, choices=TASK_OUTCOME) + sstate_checksum = models.CharField(max_length=100) + path_to_sstate_obj = models.FilePathField(max_length=500, blank=True) + recipe = models.CharField(max_length=100) + task_name = models.CharField(max_length=100) + source_url = models.FilePathField(max_length=200) + log_file = models.FilePathField(max_length=200, blank=True) + work_directory = models.FilePathField(max_length=200) + script_type = models.CharField(max_length=10, choices=TASK_CODING) + file_path = models.FilePathField(max_length=200) + line_number = models.IntegerField() + py_stack_trace = models.TextField() + disk_io = models.DecimalField(max_digits=20, decimal_places=10) + cpu_usage = models.IntegerField() + elapsed_time = models.CharField(max_length=50) + dependent_tasks = models.TextField() ## do we need only to show text here or insert references? how do we connect to other tasks from the same table ? + errors_no = models.IntegerField() + warnings_no = models.IntegerField() + error = models.TextField() + warning = models.TextField() + sstate_result = models.CharField(max_length=50, choices=SSTATE_RESULT) + diff --git a/bitbake/lib/webhob/orm/tests.py b/bitbake/lib/webhob/orm/tests.py new file mode 100644 index 0000000..501deb7 --- /dev/null +++ b/bitbake/lib/webhob/orm/tests.py @@ -0,0 +1,16 @@ +""" +This file demonstrates writing tests using the unittest module. These will pass +when you run "manage.py test". + +Replace this with more appropriate tests for your application. +""" + +from django.test import TestCase + + +class SimpleTest(TestCase): + def test_basic_addition(self): + """ + Tests that 1 + 1 always equals 2. + """ + self.assertEqual(1 + 1, 2) diff --git a/bitbake/lib/webhob/orm/views.py b/bitbake/lib/webhob/orm/views.py new file mode 100644 index 0000000..60f00ef --- /dev/null +++ b/bitbake/lib/webhob/orm/views.py @@ -0,0 +1 @@ +# Create your views here. diff --git a/bitbake/lib/webhob/whbmain/settings.py b/bitbake/lib/webhob/whbmain/settings.py index bc59543..6a4e67a 100644 --- a/bitbake/lib/webhob/whbmain/settings.py +++ b/bitbake/lib/webhob/whbmain/settings.py @@ -124,6 +124,7 @@ INSTALLED_APPS = ( # 'django.contrib.admin', # Uncomment the next line to enable admin documentation: # 'django.contrib.admindocs', + 'orm', ) # A sample logging configuration. The only tangible logging -- 1.8.1.2 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* [Webhob] [PATCH 2/3] Create Tasks Django model @ 2013-06-12 15:11 ` Calin Dragomir 0 siblings, 0 replies; 7+ messages in thread From: Calin Dragomir @ 2013-06-12 15:11 UTC (permalink / raw) To: webhob, bitbake-devel This will be used to store Tasks related information in the database using Django ORM. --- bitbake/lib/webhob/orm/__init__.py | 0 bitbake/lib/webhob/orm/models.py | 50 ++++++++++++++++++++++++++++++++++ bitbake/lib/webhob/orm/tests.py | 16 +++++++++++ bitbake/lib/webhob/orm/views.py | 1 + bitbake/lib/webhob/whbmain/settings.py | 1 + 5 files changed, 68 insertions(+) create mode 100644 bitbake/lib/webhob/orm/__init__.py create mode 100644 bitbake/lib/webhob/orm/models.py create mode 100644 bitbake/lib/webhob/orm/tests.py create mode 100644 bitbake/lib/webhob/orm/views.py diff --git a/bitbake/lib/webhob/orm/__init__.py b/bitbake/lib/webhob/orm/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/bitbake/lib/webhob/orm/models.py b/bitbake/lib/webhob/orm/models.py new file mode 100644 index 0000000..13eadd4 --- /dev/null +++ b/bitbake/lib/webhob/orm/models.py @@ -0,0 +1,50 @@ +from django.db import models + +class Tasks(models.Model): + + SSTATE_RESULT = ( + ('not_applicable', 'Not Applicable'), # For rest of tasks, but they still need checking. + ('unavailable', 'Unavailable'), # it is a miss + ('failed', 'Failed'), # there was a pkg, but the script failed + ('restored', 'Restored'), # succesfully restored + ) + + TASK_CODING = ( + ('python', 'Python'), + ('shell', 'Shell'), + ) + + TASK_OUTCOME = ( + ('succeded', 'Succeded'), + ('failed', 'Failed'), + ('existing', 'Existing'), + ('sstate', 'Sstate'), + ('covered', 'Covered'), + ) + + uuid = models.CharField(max_length=100) + task_id = models.IntegerField() + order = models.IntegerField() + task_executed = models.BooleanField() # True means Executed, False means Prebuilt + outcome = models.CharField(max_length=50, choices=TASK_OUTCOME) + sstate_checksum = models.CharField(max_length=100) + path_to_sstate_obj = models.FilePathField(max_length=500, blank=True) + recipe = models.CharField(max_length=100) + task_name = models.CharField(max_length=100) + source_url = models.FilePathField(max_length=200) + log_file = models.FilePathField(max_length=200, blank=True) + work_directory = models.FilePathField(max_length=200) + script_type = models.CharField(max_length=10, choices=TASK_CODING) + file_path = models.FilePathField(max_length=200) + line_number = models.IntegerField() + py_stack_trace = models.TextField() + disk_io = models.DecimalField(max_digits=20, decimal_places=10) + cpu_usage = models.IntegerField() + elapsed_time = models.CharField(max_length=50) + dependent_tasks = models.TextField() ## do we need only to show text here or insert references? how do we connect to other tasks from the same table ? + errors_no = models.IntegerField() + warnings_no = models.IntegerField() + error = models.TextField() + warning = models.TextField() + sstate_result = models.CharField(max_length=50, choices=SSTATE_RESULT) + diff --git a/bitbake/lib/webhob/orm/tests.py b/bitbake/lib/webhob/orm/tests.py new file mode 100644 index 0000000..501deb7 --- /dev/null +++ b/bitbake/lib/webhob/orm/tests.py @@ -0,0 +1,16 @@ +""" +This file demonstrates writing tests using the unittest module. These will pass +when you run "manage.py test". + +Replace this with more appropriate tests for your application. +""" + +from django.test import TestCase + + +class SimpleTest(TestCase): + def test_basic_addition(self): + """ + Tests that 1 + 1 always equals 2. + """ + self.assertEqual(1 + 1, 2) diff --git a/bitbake/lib/webhob/orm/views.py b/bitbake/lib/webhob/orm/views.py new file mode 100644 index 0000000..60f00ef --- /dev/null +++ b/bitbake/lib/webhob/orm/views.py @@ -0,0 +1 @@ +# Create your views here. diff --git a/bitbake/lib/webhob/whbmain/settings.py b/bitbake/lib/webhob/whbmain/settings.py index bc59543..6a4e67a 100644 --- a/bitbake/lib/webhob/whbmain/settings.py +++ b/bitbake/lib/webhob/whbmain/settings.py @@ -124,6 +124,7 @@ INSTALLED_APPS = ( # 'django.contrib.admin', # Uncomment the next line to enable admin documentation: # 'django.contrib.admindocs', + 'orm', ) # A sample logging configuration. The only tangible logging -- 1.8.1.2 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 3/3] Create Data Store Interface (DSI) file 2013-06-12 15:11 ` [Webhob] " Calin Dragomir @ 2013-06-12 15:11 ` Calin Dragomir -1 siblings, 0 replies; 7+ messages in thread From: Calin Dragomir @ 2013-06-12 15:11 UTC (permalink / raw) To: webhob, bitbake-devel This adds the first version of the DSI file. It uses the Knotty code for now, but inserts task related information into the database using the Django ORM --- bitbake/lib/bb/ui/dsi.py | 609 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 609 insertions(+) create mode 100644 bitbake/lib/bb/ui/dsi.py diff --git a/bitbake/lib/bb/ui/dsi.py b/bitbake/lib/bb/ui/dsi.py new file mode 100644 index 0000000..4bcd509 --- /dev/null +++ b/bitbake/lib/bb/ui/dsi.py @@ -0,0 +1,609 @@ +# +# BitBake (No)TTY UI Implementation +# +# Handling output to TTYs or files (no TTY) +# +# Copyright (C) 2006-2012 Richard Purdie +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +from __future__ import division + +import os +import sys +import xmlrpclib +import logging +import progressbar +import signal +import bb.msg +import time +import fcntl +import struct +import copy +import datetime + +from bb.ui import uihelper + +logger = logging.getLogger("BitBake") +interactive = sys.stdout.isatty() + +class WebHOBHelper(object): + + def __init__(self): + self.configure_django() + self.task_order = 0 + self.tasks_information = {} + + def configure_django(self): + import webhob.whbmain.settings as whb_django_settings + from django.core.management import setup_environ + setup_environ(whb_django_settings) + # Add webhob to sys path for importing modules + sys.path.append(os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), 'webhob')) + + def store_started_task(self, event, uuid): + self.task_order += 1 + self.tasks_information[event.pid] = { + 'uuid': uuid, + 'task_id': 1, + 'task_executed': True, + 'order': self.task_order, + 'recipe': event._package, + 'task_name': event._task, + 'start_time': time.time(), + } + + def update_stored_tasks(self, event, uuid): + self.tasks_information[event.pid].update({ + 'outcome': event.getDisplayName().lower(), + 'end_time': time.time(), + }) + + def write_in_database(self, tasks_information): + # This needs to be imported after we have configured the Django settings file + from webhob.orm.models import Tasks + for pid in tasks_information.keys(): + task = tasks_information[pid] + + task_obj = Tasks.objects.create(uuid=task['uuid'], + order=task['order'], + outcome=task['outcome'], + recipe=task['recipe'], + task_name=task['task_name'], + elapsed_time=task['end_time'] - task['start_time'], + # The next lines are just for testing reasons + task_id=1, + path_to_sstate_obj='/home/calin', + source_url='/', + log_file='/', + work_directory='/', + script_type='python', + file_path='/', + line_number=55, + py_stack_trace='Testing traceback', + disk_io = 23.55633, + cpu_usage=43, + dependent_tasks='Task 1', + errors_no=0, + warnings_no=0, + error='None', + warning='None', + sstate_result='not_applicable') + task_obj.save() + +class BBProgress(progressbar.ProgressBar): + def __init__(self, msg, maxval): + self.msg = msg + widgets = [progressbar.Percentage(), ' ', progressbar.Bar(), ' ', + progressbar.ETA()] + + try: + self._resize_default = signal.getsignal(signal.SIGWINCH) + except: + self._resize_default = None + progressbar.ProgressBar.__init__(self, maxval, [self.msg + ": "] + widgets) + + def _handle_resize(self, signum, frame): + progressbar.ProgressBar._handle_resize(self, signum, frame) + if self._resize_default: + self._resize_default(signum, frame) + def finish(self): + progressbar.ProgressBar.finish(self) + if self._resize_default: + signal.signal(signal.SIGWINCH, self._resize_default) + +class NonInteractiveProgress(object): + fobj = sys.stdout + + def __init__(self, msg, maxval): + self.msg = msg + self.maxval = maxval + + def start(self): + self.fobj.write("%s..." % self.msg) + self.fobj.flush() + return self + + def update(self, value): + pass + + def finish(self): + self.fobj.write("done.\n") + self.fobj.flush() + +def new_progress(msg, maxval): + if interactive: + return BBProgress(msg, maxval) + else: + return NonInteractiveProgress(msg, maxval) + +def pluralise(singular, plural, qty): + if(qty == 1): + return singular % qty + else: + return plural % qty + + +class InteractConsoleLogFilter(logging.Filter): + def __init__(self, tf, format): + self.tf = tf + self.format = format + + def filter(self, record): + if record.levelno == self.format.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")): + return False + self.tf.clearFooter() + return True + +class TerminalFilter(object): + columns = 80 + + def sigwinch_handle(self, signum, frame): + self.columns = self.getTerminalColumns() + if self._sigwinch_default: + self._sigwinch_default(signum, frame) + + def getTerminalColumns(self): + def ioctl_GWINSZ(fd): + try: + cr = struct.unpack('hh', fcntl.ioctl(fd, self.termios.TIOCGWINSZ, '1234')) + except: + return None + return cr + cr = ioctl_GWINSZ(sys.stdout.fileno()) + if not cr: + try: + fd = os.open(os.ctermid(), os.O_RDONLY) + cr = ioctl_GWINSZ(fd) + os.close(fd) + except: + pass + if not cr: + try: + cr = (env['LINES'], env['COLUMNS']) + except: + cr = (25, 80) + return cr[1] + + def __init__(self, main, helper, console, format): + self.main = main + self.helper = helper + self.cuu = None + self.stdinbackup = None + self.interactive = sys.stdout.isatty() + self.footer_present = False + self.lastpids = [] + + if not self.interactive: + return + + try: + import curses + except ImportError: + sys.exit("FATAL: The knotty ui could not load the required curses python module.") + + import termios + self.curses = curses + self.termios = termios + try: + fd = sys.stdin.fileno() + self.stdinbackup = termios.tcgetattr(fd) + new = copy.deepcopy(self.stdinbackup) + new[3] = new[3] & ~termios.ECHO + termios.tcsetattr(fd, termios.TCSADRAIN, new) + curses.setupterm() + if curses.tigetnum("colors") > 2: + format.enable_color() + self.ed = curses.tigetstr("ed") + if self.ed: + self.cuu = curses.tigetstr("cuu") + try: + self._sigwinch_default = signal.getsignal(signal.SIGWINCH) + signal.signal(signal.SIGWINCH, self.sigwinch_handle) + except: + pass + self.columns = self.getTerminalColumns() + except: + self.cuu = None + console.addFilter(InteractConsoleLogFilter(self, format)) + + def clearFooter(self): + if self.footer_present: + lines = self.footer_present + sys.stdout.write(self.curses.tparm(self.cuu, lines)) + sys.stdout.write(self.curses.tparm(self.ed)) + self.footer_present = False + + def updateFooter(self): + if not self.cuu: + return + activetasks = self.helper.running_tasks + failedtasks = self.helper.failed_tasks + runningpids = self.helper.running_pids + if self.footer_present and (self.lastcount == self.helper.tasknumber_current) and (self.lastpids == runningpids): + return + if self.footer_present: + self.clearFooter() + if (not self.helper.tasknumber_total or self.helper.tasknumber_current == self.helper.tasknumber_total) and not len(activetasks): + return + tasks = [] + for t in runningpids: + tasks.append("%s (pid %s)" % (activetasks[t]["title"], t)) + + if self.main.shutdown: + content = "Waiting for %s running tasks to finish:" % len(activetasks) + elif not len(activetasks): + content = "No currently running tasks (%s of %s)" % (self.helper.tasknumber_current, self.helper.tasknumber_total) + else: + content = "Currently %s running tasks (%s of %s):" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total) + print(content) + lines = 1 + int(len(content) / (self.columns + 1)) + for tasknum, task in enumerate(tasks): + content = "%s: %s" % (tasknum, task) + print(content) + lines = lines + 1 + int(len(content) / (self.columns + 1)) + self.footer_present = lines + self.lastpids = runningpids[:] + self.lastcount = self.helper.tasknumber_current + + def finish(self): + if self.stdinbackup: + fd = sys.stdin.fileno() + self.termios.tcsetattr(fd, self.termios.TCSADRAIN, self.stdinbackup) + +def main(server, eventHandler, params, tf = TerminalFilter): + print "This is running in DSI mode" + + # Generate an unique ID for this build + # TODO: for multiple build commands this might not work as expected + import uuid + uuid = sessionid = str(uuid.uuid4()) + + # Get values of variables which control our output + includelogs, error = server.runCommand(["getVariable", "BBINCLUDELOGS"]) + if error: + logger.error("Unable to get the value of BBINCLUDELOGS variable: %s" % error) + return 1 + loglines, error = server.runCommand(["getVariable", "BBINCLUDELOGS_LINES"]) + if error: + logger.error("Unable to get the value of BBINCLUDELOGS_LINES variable: %s" % error) + return 1 + consolelogfile, error = server.runCommand(["getVariable", "BB_CONSOLELOG"]) + if error: + logger.error("Unable to get the value of BB_CONSOLELOG variable: %s" % error) + return 1 + + if sys.stdin.isatty() and sys.stdout.isatty(): + log_exec_tty = True + else: + log_exec_tty = False + + helper = uihelper.BBUIHelper() + + console = logging.StreamHandler(sys.stdout) + format_str = "%(levelname)s: %(message)s" + format = bb.msg.BBLogFormatter(format_str) + bb.msg.addDefaultlogFilter(console) + console.setFormatter(format) + logger.addHandler(console) + + if consolelogfile and not params.options.show_environment: + bb.utils.mkdirhier(os.path.dirname(consolelogfile)) + conlogformat = bb.msg.BBLogFormatter(format_str) + consolelog = logging.FileHandler(consolelogfile) + bb.msg.addDefaultlogFilter(consolelog) + consolelog.setFormatter(conlogformat) + logger.addHandler(consolelog) + + try: + params.updateFromServer(server) + cmdline = params.parseActions() + if not cmdline: + print("Nothing to do. Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information.") + return 1 + if 'msg' in cmdline and cmdline['msg']: + logger.error(cmdline['msg']) + return 1 + + ret, error = server.runCommand(cmdline['action']) + if error: + logger.error("Command '%s' failed: %s" % (cmdline, error)) + return 1 + elif ret != True: + logger.error("Command '%s' failed: returned %s" % (cmdline, ret)) + return 1 + except xmlrpclib.Fault as x: + logger.error("XMLRPC Fault getting commandline:\n %s" % x) + return 1 + + parseprogress = None + cacheprogress = None + main.shutdown = 0 + interrupted = False + return_value = 0 + errors = 0 + warnings = 0 + taskfailures = [] + wbhbhelper = WebHOBHelper() + + termfilter = tf(main, helper, console, format) + + while True: + try: + termfilter.updateFooter() + event = eventHandler.waitEvent(0.25) + + if event is None: + if main.shutdown > 1: + break + continue + + helper.eventHandler(event) + + if isinstance(event, bb.runqueue.runQueueExitWait): + if not main.shutdown: + main.shutdown = 1 + + if isinstance(event, bb.build.TaskStarted): + wbhbhelper.store_started_task(event, uuid) + + if isinstance(event, (bb.build.TaskSucceeded, bb.build.TaskFailedSilent, bb.build.TaskFailed)): + wbhbhelper.update_stored_tasks(event, uuid) + + if isinstance(event, bb.event.LogExecTTY): + if log_exec_tty: + tries = event.retries + while tries: + print("Trying to run: %s" % event.prog) + if os.system(event.prog) == 0: + break + time.sleep(event.sleep_delay) + tries -= 1 + if tries: + continue + logger.warn(event.msg) + continue + + if isinstance(event, logging.LogRecord): + if event.levelno >= format.ERROR: + errors = errors + 1 + return_value = 1 + elif event.levelno == format.WARNING: + warnings = warnings + 1 + # For "normal" logging conditions, don't show note logs from tasks + # but do show them if the user has changed the default log level to + # include verbose/debug messages + if event.taskpid != 0 and event.levelno <= format.NOTE: + continue + logger.handle(event) + continue + + if isinstance(event, bb.build.TaskFailed): + return_value = 1 + logfile = event.logfile + if logfile and os.path.exists(logfile): + termfilter.clearFooter() + bb.error("Logfile of failure stored in: %s" % logfile) + if includelogs and not event.errprinted: + print("Log data follows:") + f = open(logfile, "r") + lines = [] + while True: + l = f.readline() + if l == '': + break + l = l.rstrip() + if loglines: + lines.append(' | %s' % l) + if len(lines) > int(loglines): + lines.pop(0) + else: + print('| %s' % l) + f.close() + if lines: + for line in lines: + print(line) + if isinstance(event, bb.build.TaskBase): + logger.info(event._message) + continue + if isinstance(event, bb.event.ParseStarted): + if event.total == 0: + continue + parseprogress = new_progress("Parsing recipes", event.total).start() + continue + if isinstance(event, bb.event.ParseProgress): + parseprogress.update(event.current) + continue + if isinstance(event, bb.event.ParseCompleted): + if not parseprogress: + continue + + parseprogress.finish() + print(("Parsing of %d .bb files complete (%d cached, %d parsed). %d targets, %d skipped, %d masked, %d errors." + % ( event.total, event.cached, event.parsed, event.virtuals, event.skipped, event.masked, event.errors))) + continue + + if isinstance(event, bb.event.CacheLoadStarted): + cacheprogress = new_progress("Loading cache", event.total).start() + continue + if isinstance(event, bb.event.CacheLoadProgress): + cacheprogress.update(event.current) + continue + if isinstance(event, bb.event.CacheLoadCompleted): + cacheprogress.finish() + print("Loaded %d entries from dependency cache." % event.num_entries) + continue + + if isinstance(event, bb.command.CommandFailed): + return_value = event.exitcode + errors = errors + 1 + logger.error("Command execution failed: %s", event.error) + main.shutdown = 2 + continue + if isinstance(event, bb.command.CommandExit): + if not return_value: + return_value = event.exitcode + continue + if isinstance(event, (bb.command.CommandCompleted, bb.cooker.CookerExit)): + main.shutdown = 2 + continue + if isinstance(event, bb.event.MultipleProviders): + logger.info("multiple providers are available for %s%s (%s)", event._is_runtime and "runtime " or "", + event._item, + ", ".join(event._candidates)) + logger.info("consider defining a PREFERRED_PROVIDER entry to match %s", event._item) + continue + if isinstance(event, bb.event.NoProvider): + return_value = 1 + errors = errors + 1 + if event._runtime: + r = "R" + else: + r = "" + + if event._dependees: + logger.error("Nothing %sPROVIDES '%s' (but %s %sDEPENDS on or otherwise requires it)", r, event._item, ", ".join(event._dependees), r) + else: + logger.error("Nothing %sPROVIDES '%s'", r, event._item) + if event._reasons: + for reason in event._reasons: + logger.error("%s", reason) + continue + + if isinstance(event, bb.runqueue.sceneQueueTaskStarted): + logger.info("Running setscene task %d of %d (%s)" % (event.stats.completed + event.stats.active + event.stats.failed + 1, event.stats.total, event.taskstring)) + continue + + if isinstance(event, bb.runqueue.runQueueTaskStarted): + if event.noexec: + tasktype = 'noexec task' + else: + tasktype = 'task' + logger.info("Running %s %s of %s (ID: %s, %s)", + tasktype, + event.stats.completed + event.stats.active + + event.stats.failed + 1, + event.stats.total, event.taskid, event.taskstring) + continue + + if isinstance(event, bb.runqueue.runQueueTaskFailed): + taskfailures.append(event.taskstring) + logger.error("Task %s (%s) failed with exit code '%s'", + event.taskid, event.taskstring, event.exitcode) + continue + + if isinstance(event, bb.runqueue.sceneQueueTaskFailed): + logger.warn("Setscene task %s (%s) failed with exit code '%s' - real task will be run instead", + event.taskid, event.taskstring, event.exitcode) + continue + + if isinstance(event, bb.event.ConfigParsed): + # timestamp should be added for this + continue + + if isinstance(event, bb.event.RecipeParsed): + # timestamp should be added for this + continue + + if isinstance(event, bb.event.OperationStarted): + # timestamp should be added for this + continue + + if isinstance(event, bb.event.OperationCompleted): + # timestamp should be added for this + # signal a complete operation + # calculate timing + continue + + if isinstance(event, bb.event.DiskFull): + # trigger an error + continue + + # ignore + if isinstance(event, (bb.event.BuildBase, + bb.event.StampUpdate, + bb.event.RecipePreFinalise, + bb.runqueue.runQueueEvent, + bb.runqueue.runQueueExitWait, + bb.event.OperationProgress)): + continue + + logger.error("Unknown event: %s", event) + + except EnvironmentError as ioerror: + termfilter.clearFooter() + # ignore interrupted io + if ioerror.args[0] == 4: + pass + except KeyboardInterrupt: + termfilter.clearFooter() + if main.shutdown == 1: + print("\nSecond Keyboard Interrupt, stopping...\n") + _, error = server.runCommand(["stateStop"]) + if error: + logger.error("Unable to cleanly stop: %s" % error) + if main.shutdown == 0: + print("\nKeyboard Interrupt, closing down...\n") + interrupted = True + _, error = server.runCommand(["stateShutdown"]) + if error: + logger.error("Unable to cleanly shutdown: %s" % error) + main.shutdown = main.shutdown + 1 + pass + + summary = "" + if taskfailures: + summary += pluralise("\nSummary: %s task failed:", + "\nSummary: %s tasks failed:", len(taskfailures)) + for failure in taskfailures: + summary += "\n %s" % failure + if warnings: + summary += pluralise("\nSummary: There was %s WARNING message shown.", + "\nSummary: There were %s WARNING messages shown.", warnings) + if return_value: + summary += pluralise("\nSummary: There was %s ERROR message shown, returning a non-zero exit code.", + "\nSummary: There were %s ERROR messages shown, returning a non-zero exit code.", errors) + if summary: + print(summary) + + if interrupted: + print("Execution was interrupted, returning a non-zero exit code.") + if return_value == 0: + return_value = 1 + + termfilter.finish() + wbhbhelper.write_in_database(wbhbhelper.tasks_information) + + return return_value + -- 1.8.1.2 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* [Webhob] [PATCH 3/3] Create Data Store Interface (DSI) file @ 2013-06-12 15:11 ` Calin Dragomir 0 siblings, 0 replies; 7+ messages in thread From: Calin Dragomir @ 2013-06-12 15:11 UTC (permalink / raw) To: webhob, bitbake-devel This adds the first version of the DSI file. It uses the Knotty code for now, but inserts task related information into the database using the Django ORM --- bitbake/lib/bb/ui/dsi.py | 609 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 609 insertions(+) create mode 100644 bitbake/lib/bb/ui/dsi.py diff --git a/bitbake/lib/bb/ui/dsi.py b/bitbake/lib/bb/ui/dsi.py new file mode 100644 index 0000000..4bcd509 --- /dev/null +++ b/bitbake/lib/bb/ui/dsi.py @@ -0,0 +1,609 @@ +# +# BitBake (No)TTY UI Implementation +# +# Handling output to TTYs or files (no TTY) +# +# Copyright (C) 2006-2012 Richard Purdie +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +from __future__ import division + +import os +import sys +import xmlrpclib +import logging +import progressbar +import signal +import bb.msg +import time +import fcntl +import struct +import copy +import datetime + +from bb.ui import uihelper + +logger = logging.getLogger("BitBake") +interactive = sys.stdout.isatty() + +class WebHOBHelper(object): + + def __init__(self): + self.configure_django() + self.task_order = 0 + self.tasks_information = {} + + def configure_django(self): + import webhob.whbmain.settings as whb_django_settings + from django.core.management import setup_environ + setup_environ(whb_django_settings) + # Add webhob to sys path for importing modules + sys.path.append(os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), 'webhob')) + + def store_started_task(self, event, uuid): + self.task_order += 1 + self.tasks_information[event.pid] = { + 'uuid': uuid, + 'task_id': 1, + 'task_executed': True, + 'order': self.task_order, + 'recipe': event._package, + 'task_name': event._task, + 'start_time': time.time(), + } + + def update_stored_tasks(self, event, uuid): + self.tasks_information[event.pid].update({ + 'outcome': event.getDisplayName().lower(), + 'end_time': time.time(), + }) + + def write_in_database(self, tasks_information): + # This needs to be imported after we have configured the Django settings file + from webhob.orm.models import Tasks + for pid in tasks_information.keys(): + task = tasks_information[pid] + + task_obj = Tasks.objects.create(uuid=task['uuid'], + order=task['order'], + outcome=task['outcome'], + recipe=task['recipe'], + task_name=task['task_name'], + elapsed_time=task['end_time'] - task['start_time'], + # The next lines are just for testing reasons + task_id=1, + path_to_sstate_obj='/home/calin', + source_url='/', + log_file='/', + work_directory='/', + script_type='python', + file_path='/', + line_number=55, + py_stack_trace='Testing traceback', + disk_io = 23.55633, + cpu_usage=43, + dependent_tasks='Task 1', + errors_no=0, + warnings_no=0, + error='None', + warning='None', + sstate_result='not_applicable') + task_obj.save() + +class BBProgress(progressbar.ProgressBar): + def __init__(self, msg, maxval): + self.msg = msg + widgets = [progressbar.Percentage(), ' ', progressbar.Bar(), ' ', + progressbar.ETA()] + + try: + self._resize_default = signal.getsignal(signal.SIGWINCH) + except: + self._resize_default = None + progressbar.ProgressBar.__init__(self, maxval, [self.msg + ": "] + widgets) + + def _handle_resize(self, signum, frame): + progressbar.ProgressBar._handle_resize(self, signum, frame) + if self._resize_default: + self._resize_default(signum, frame) + def finish(self): + progressbar.ProgressBar.finish(self) + if self._resize_default: + signal.signal(signal.SIGWINCH, self._resize_default) + +class NonInteractiveProgress(object): + fobj = sys.stdout + + def __init__(self, msg, maxval): + self.msg = msg + self.maxval = maxval + + def start(self): + self.fobj.write("%s..." % self.msg) + self.fobj.flush() + return self + + def update(self, value): + pass + + def finish(self): + self.fobj.write("done.\n") + self.fobj.flush() + +def new_progress(msg, maxval): + if interactive: + return BBProgress(msg, maxval) + else: + return NonInteractiveProgress(msg, maxval) + +def pluralise(singular, plural, qty): + if(qty == 1): + return singular % qty + else: + return plural % qty + + +class InteractConsoleLogFilter(logging.Filter): + def __init__(self, tf, format): + self.tf = tf + self.format = format + + def filter(self, record): + if record.levelno == self.format.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")): + return False + self.tf.clearFooter() + return True + +class TerminalFilter(object): + columns = 80 + + def sigwinch_handle(self, signum, frame): + self.columns = self.getTerminalColumns() + if self._sigwinch_default: + self._sigwinch_default(signum, frame) + + def getTerminalColumns(self): + def ioctl_GWINSZ(fd): + try: + cr = struct.unpack('hh', fcntl.ioctl(fd, self.termios.TIOCGWINSZ, '1234')) + except: + return None + return cr + cr = ioctl_GWINSZ(sys.stdout.fileno()) + if not cr: + try: + fd = os.open(os.ctermid(), os.O_RDONLY) + cr = ioctl_GWINSZ(fd) + os.close(fd) + except: + pass + if not cr: + try: + cr = (env['LINES'], env['COLUMNS']) + except: + cr = (25, 80) + return cr[1] + + def __init__(self, main, helper, console, format): + self.main = main + self.helper = helper + self.cuu = None + self.stdinbackup = None + self.interactive = sys.stdout.isatty() + self.footer_present = False + self.lastpids = [] + + if not self.interactive: + return + + try: + import curses + except ImportError: + sys.exit("FATAL: The knotty ui could not load the required curses python module.") + + import termios + self.curses = curses + self.termios = termios + try: + fd = sys.stdin.fileno() + self.stdinbackup = termios.tcgetattr(fd) + new = copy.deepcopy(self.stdinbackup) + new[3] = new[3] & ~termios.ECHO + termios.tcsetattr(fd, termios.TCSADRAIN, new) + curses.setupterm() + if curses.tigetnum("colors") > 2: + format.enable_color() + self.ed = curses.tigetstr("ed") + if self.ed: + self.cuu = curses.tigetstr("cuu") + try: + self._sigwinch_default = signal.getsignal(signal.SIGWINCH) + signal.signal(signal.SIGWINCH, self.sigwinch_handle) + except: + pass + self.columns = self.getTerminalColumns() + except: + self.cuu = None + console.addFilter(InteractConsoleLogFilter(self, format)) + + def clearFooter(self): + if self.footer_present: + lines = self.footer_present + sys.stdout.write(self.curses.tparm(self.cuu, lines)) + sys.stdout.write(self.curses.tparm(self.ed)) + self.footer_present = False + + def updateFooter(self): + if not self.cuu: + return + activetasks = self.helper.running_tasks + failedtasks = self.helper.failed_tasks + runningpids = self.helper.running_pids + if self.footer_present and (self.lastcount == self.helper.tasknumber_current) and (self.lastpids == runningpids): + return + if self.footer_present: + self.clearFooter() + if (not self.helper.tasknumber_total or self.helper.tasknumber_current == self.helper.tasknumber_total) and not len(activetasks): + return + tasks = [] + for t in runningpids: + tasks.append("%s (pid %s)" % (activetasks[t]["title"], t)) + + if self.main.shutdown: + content = "Waiting for %s running tasks to finish:" % len(activetasks) + elif not len(activetasks): + content = "No currently running tasks (%s of %s)" % (self.helper.tasknumber_current, self.helper.tasknumber_total) + else: + content = "Currently %s running tasks (%s of %s):" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total) + print(content) + lines = 1 + int(len(content) / (self.columns + 1)) + for tasknum, task in enumerate(tasks): + content = "%s: %s" % (tasknum, task) + print(content) + lines = lines + 1 + int(len(content) / (self.columns + 1)) + self.footer_present = lines + self.lastpids = runningpids[:] + self.lastcount = self.helper.tasknumber_current + + def finish(self): + if self.stdinbackup: + fd = sys.stdin.fileno() + self.termios.tcsetattr(fd, self.termios.TCSADRAIN, self.stdinbackup) + +def main(server, eventHandler, params, tf = TerminalFilter): + print "This is running in DSI mode" + + # Generate an unique ID for this build + # TODO: for multiple build commands this might not work as expected + import uuid + uuid = sessionid = str(uuid.uuid4()) + + # Get values of variables which control our output + includelogs, error = server.runCommand(["getVariable", "BBINCLUDELOGS"]) + if error: + logger.error("Unable to get the value of BBINCLUDELOGS variable: %s" % error) + return 1 + loglines, error = server.runCommand(["getVariable", "BBINCLUDELOGS_LINES"]) + if error: + logger.error("Unable to get the value of BBINCLUDELOGS_LINES variable: %s" % error) + return 1 + consolelogfile, error = server.runCommand(["getVariable", "BB_CONSOLELOG"]) + if error: + logger.error("Unable to get the value of BB_CONSOLELOG variable: %s" % error) + return 1 + + if sys.stdin.isatty() and sys.stdout.isatty(): + log_exec_tty = True + else: + log_exec_tty = False + + helper = uihelper.BBUIHelper() + + console = logging.StreamHandler(sys.stdout) + format_str = "%(levelname)s: %(message)s" + format = bb.msg.BBLogFormatter(format_str) + bb.msg.addDefaultlogFilter(console) + console.setFormatter(format) + logger.addHandler(console) + + if consolelogfile and not params.options.show_environment: + bb.utils.mkdirhier(os.path.dirname(consolelogfile)) + conlogformat = bb.msg.BBLogFormatter(format_str) + consolelog = logging.FileHandler(consolelogfile) + bb.msg.addDefaultlogFilter(consolelog) + consolelog.setFormatter(conlogformat) + logger.addHandler(consolelog) + + try: + params.updateFromServer(server) + cmdline = params.parseActions() + if not cmdline: + print("Nothing to do. Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information.") + return 1 + if 'msg' in cmdline and cmdline['msg']: + logger.error(cmdline['msg']) + return 1 + + ret, error = server.runCommand(cmdline['action']) + if error: + logger.error("Command '%s' failed: %s" % (cmdline, error)) + return 1 + elif ret != True: + logger.error("Command '%s' failed: returned %s" % (cmdline, ret)) + return 1 + except xmlrpclib.Fault as x: + logger.error("XMLRPC Fault getting commandline:\n %s" % x) + return 1 + + parseprogress = None + cacheprogress = None + main.shutdown = 0 + interrupted = False + return_value = 0 + errors = 0 + warnings = 0 + taskfailures = [] + wbhbhelper = WebHOBHelper() + + termfilter = tf(main, helper, console, format) + + while True: + try: + termfilter.updateFooter() + event = eventHandler.waitEvent(0.25) + + if event is None: + if main.shutdown > 1: + break + continue + + helper.eventHandler(event) + + if isinstance(event, bb.runqueue.runQueueExitWait): + if not main.shutdown: + main.shutdown = 1 + + if isinstance(event, bb.build.TaskStarted): + wbhbhelper.store_started_task(event, uuid) + + if isinstance(event, (bb.build.TaskSucceeded, bb.build.TaskFailedSilent, bb.build.TaskFailed)): + wbhbhelper.update_stored_tasks(event, uuid) + + if isinstance(event, bb.event.LogExecTTY): + if log_exec_tty: + tries = event.retries + while tries: + print("Trying to run: %s" % event.prog) + if os.system(event.prog) == 0: + break + time.sleep(event.sleep_delay) + tries -= 1 + if tries: + continue + logger.warn(event.msg) + continue + + if isinstance(event, logging.LogRecord): + if event.levelno >= format.ERROR: + errors = errors + 1 + return_value = 1 + elif event.levelno == format.WARNING: + warnings = warnings + 1 + # For "normal" logging conditions, don't show note logs from tasks + # but do show them if the user has changed the default log level to + # include verbose/debug messages + if event.taskpid != 0 and event.levelno <= format.NOTE: + continue + logger.handle(event) + continue + + if isinstance(event, bb.build.TaskFailed): + return_value = 1 + logfile = event.logfile + if logfile and os.path.exists(logfile): + termfilter.clearFooter() + bb.error("Logfile of failure stored in: %s" % logfile) + if includelogs and not event.errprinted: + print("Log data follows:") + f = open(logfile, "r") + lines = [] + while True: + l = f.readline() + if l == '': + break + l = l.rstrip() + if loglines: + lines.append(' | %s' % l) + if len(lines) > int(loglines): + lines.pop(0) + else: + print('| %s' % l) + f.close() + if lines: + for line in lines: + print(line) + if isinstance(event, bb.build.TaskBase): + logger.info(event._message) + continue + if isinstance(event, bb.event.ParseStarted): + if event.total == 0: + continue + parseprogress = new_progress("Parsing recipes", event.total).start() + continue + if isinstance(event, bb.event.ParseProgress): + parseprogress.update(event.current) + continue + if isinstance(event, bb.event.ParseCompleted): + if not parseprogress: + continue + + parseprogress.finish() + print(("Parsing of %d .bb files complete (%d cached, %d parsed). %d targets, %d skipped, %d masked, %d errors." + % ( event.total, event.cached, event.parsed, event.virtuals, event.skipped, event.masked, event.errors))) + continue + + if isinstance(event, bb.event.CacheLoadStarted): + cacheprogress = new_progress("Loading cache", event.total).start() + continue + if isinstance(event, bb.event.CacheLoadProgress): + cacheprogress.update(event.current) + continue + if isinstance(event, bb.event.CacheLoadCompleted): + cacheprogress.finish() + print("Loaded %d entries from dependency cache." % event.num_entries) + continue + + if isinstance(event, bb.command.CommandFailed): + return_value = event.exitcode + errors = errors + 1 + logger.error("Command execution failed: %s", event.error) + main.shutdown = 2 + continue + if isinstance(event, bb.command.CommandExit): + if not return_value: + return_value = event.exitcode + continue + if isinstance(event, (bb.command.CommandCompleted, bb.cooker.CookerExit)): + main.shutdown = 2 + continue + if isinstance(event, bb.event.MultipleProviders): + logger.info("multiple providers are available for %s%s (%s)", event._is_runtime and "runtime " or "", + event._item, + ", ".join(event._candidates)) + logger.info("consider defining a PREFERRED_PROVIDER entry to match %s", event._item) + continue + if isinstance(event, bb.event.NoProvider): + return_value = 1 + errors = errors + 1 + if event._runtime: + r = "R" + else: + r = "" + + if event._dependees: + logger.error("Nothing %sPROVIDES '%s' (but %s %sDEPENDS on or otherwise requires it)", r, event._item, ", ".join(event._dependees), r) + else: + logger.error("Nothing %sPROVIDES '%s'", r, event._item) + if event._reasons: + for reason in event._reasons: + logger.error("%s", reason) + continue + + if isinstance(event, bb.runqueue.sceneQueueTaskStarted): + logger.info("Running setscene task %d of %d (%s)" % (event.stats.completed + event.stats.active + event.stats.failed + 1, event.stats.total, event.taskstring)) + continue + + if isinstance(event, bb.runqueue.runQueueTaskStarted): + if event.noexec: + tasktype = 'noexec task' + else: + tasktype = 'task' + logger.info("Running %s %s of %s (ID: %s, %s)", + tasktype, + event.stats.completed + event.stats.active + + event.stats.failed + 1, + event.stats.total, event.taskid, event.taskstring) + continue + + if isinstance(event, bb.runqueue.runQueueTaskFailed): + taskfailures.append(event.taskstring) + logger.error("Task %s (%s) failed with exit code '%s'", + event.taskid, event.taskstring, event.exitcode) + continue + + if isinstance(event, bb.runqueue.sceneQueueTaskFailed): + logger.warn("Setscene task %s (%s) failed with exit code '%s' - real task will be run instead", + event.taskid, event.taskstring, event.exitcode) + continue + + if isinstance(event, bb.event.ConfigParsed): + # timestamp should be added for this + continue + + if isinstance(event, bb.event.RecipeParsed): + # timestamp should be added for this + continue + + if isinstance(event, bb.event.OperationStarted): + # timestamp should be added for this + continue + + if isinstance(event, bb.event.OperationCompleted): + # timestamp should be added for this + # signal a complete operation + # calculate timing + continue + + if isinstance(event, bb.event.DiskFull): + # trigger an error + continue + + # ignore + if isinstance(event, (bb.event.BuildBase, + bb.event.StampUpdate, + bb.event.RecipePreFinalise, + bb.runqueue.runQueueEvent, + bb.runqueue.runQueueExitWait, + bb.event.OperationProgress)): + continue + + logger.error("Unknown event: %s", event) + + except EnvironmentError as ioerror: + termfilter.clearFooter() + # ignore interrupted io + if ioerror.args[0] == 4: + pass + except KeyboardInterrupt: + termfilter.clearFooter() + if main.shutdown == 1: + print("\nSecond Keyboard Interrupt, stopping...\n") + _, error = server.runCommand(["stateStop"]) + if error: + logger.error("Unable to cleanly stop: %s" % error) + if main.shutdown == 0: + print("\nKeyboard Interrupt, closing down...\n") + interrupted = True + _, error = server.runCommand(["stateShutdown"]) + if error: + logger.error("Unable to cleanly shutdown: %s" % error) + main.shutdown = main.shutdown + 1 + pass + + summary = "" + if taskfailures: + summary += pluralise("\nSummary: %s task failed:", + "\nSummary: %s tasks failed:", len(taskfailures)) + for failure in taskfailures: + summary += "\n %s" % failure + if warnings: + summary += pluralise("\nSummary: There was %s WARNING message shown.", + "\nSummary: There were %s WARNING messages shown.", warnings) + if return_value: + summary += pluralise("\nSummary: There was %s ERROR message shown, returning a non-zero exit code.", + "\nSummary: There were %s ERROR messages shown, returning a non-zero exit code.", errors) + if summary: + print(summary) + + if interrupted: + print("Execution was interrupted, returning a non-zero exit code.") + if return_value == 0: + return_value = 1 + + termfilter.finish() + wbhbhelper.write_in_database(wbhbhelper.tasks_information) + + return return_value + -- 1.8.1.2 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH 3/3] Create Data Store Interface (DSI) file 2013-06-12 15:11 ` [Webhob] " Calin Dragomir (?) @ 2013-06-13 15:48 ` Alex Damian -1 siblings, 0 replies; 7+ messages in thread From: Alex Damian @ 2013-06-13 15:48 UTC (permalink / raw) To: bitbake-devel Comments below. Cheers, Alex On 06/12/2013 04:11 PM, Calin Dragomir wrote: > This adds the first version of the DSI file. It uses the Knotty > code for now, but inserts task related information into the database > using the Django ORM > --- > bitbake/lib/bb/ui/dsi.py | 609 +++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 609 insertions(+) > create mode 100644 bitbake/lib/bb/ui/dsi.py > > diff --git a/bitbake/lib/bb/ui/dsi.py b/bitbake/lib/bb/ui/dsi.py > new file mode 100644 > index 0000000..4bcd509 > --- /dev/null > +++ b/bitbake/lib/bb/ui/dsi.py > @@ -0,0 +1,609 @@ > +# > +# BitBake (No)TTY UI Implementation > +# > +# Handling output to TTYs or files (no TTY) > +# Please update file description and copyright. > +# Copyright (C) 2006-2012 Richard Purdie > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License version 2 as > +# published by the Free Software Foundation. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License along > +# with this program; if not, write to the Free Software Foundation, Inc., > +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > + > +from __future__ import division > + > +import os > +import sys > +import xmlrpclib > +import logging > +import progressbar > +import signal > +import bb.msg > +import time > +import fcntl > +import struct > +import copy > +import datetime > + > +from bb.ui import uihelper > + > +logger = logging.getLogger("BitBake") > +interactive = sys.stdout.isatty() > + > +class WebHOBHelper(object): > + > + def __init__(self): > + self.configure_django() > + self.task_order = 0 > + self.tasks_information = {} > + > + def configure_django(self): > + import webhob.whbmain.settings as whb_django_settings > + from django.core.management import setup_environ > + setup_environ(whb_django_settings) > + # Add webhob to sys path for importing modules > + sys.path.append(os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), 'webhob')) > + > + def store_started_task(self, event, uuid): > + self.task_order += 1 > + self.tasks_information[event.pid] = { > + 'uuid': uuid, > + 'task_id': 1, > + 'task_executed': True, > + 'order': self.task_order, > + 'recipe': event._package, > + 'task_name': event._task, > + 'start_time': time.time(), > + } > + > + def update_stored_tasks(self, event, uuid): > + self.tasks_information[event.pid].update({ > + 'outcome': event.getDisplayName().lower(), > + 'end_time': time.time(), > + }) > + > + def write_in_database(self, tasks_information): > + # This needs to be imported after we have configured the Django settings file > + from webhob.orm.models import Tasks > + for pid in tasks_information.keys(): > + task = tasks_information[pid] I'm not sure that there is any guarantee that any PID refers to a single task. Primary lookup key for tasks should be task-name & task-recipe combination. > + > + task_obj = Tasks.objects.create(uuid=task['uuid'], > + order=task['order'], > + outcome=task['outcome'], > + recipe=task['recipe'], > + task_name=task['task_name'], > + elapsed_time=task['end_time'] - task['start_time'], > + # The next lines are just for testing reasons > + task_id=1, > + path_to_sstate_obj='/home/calin', > + source_url='/', > + log_file='/', > + work_directory='/', > + script_type='python', > + file_path='/', > + line_number=55, > + py_stack_trace='Testing traceback', > + disk_io = 23.55633, > + cpu_usage=43, > + dependent_tasks='Task 1', > + errors_no=0, > + warnings_no=0, > + error='None', > + warning='None', > + sstate_result='not_applicable') > + task_obj.save() > + > +class BBProgress(progressbar.ProgressBar): > + def __init__(self, msg, maxval): > + self.msg = msg > + widgets = [progressbar.Percentage(), ' ', progressbar.Bar(), ' ', > + progressbar.ETA()] > + > + try: > + self._resize_default = signal.getsignal(signal.SIGWINCH) > + except: > + self._resize_default = None > + progressbar.ProgressBar.__init__(self, maxval, [self.msg + ": "] + widgets) > + > + def _handle_resize(self, signum, frame): > + progressbar.ProgressBar._handle_resize(self, signum, frame) > + if self._resize_default: > + self._resize_default(signum, frame) > + def finish(self): > + progressbar.ProgressBar.finish(self) > + if self._resize_default: > + signal.signal(signal.SIGWINCH, self._resize_default) > + > +class NonInteractiveProgress(object): > + fobj = sys.stdout > + > + def __init__(self, msg, maxval): > + self.msg = msg > + self.maxval = maxval > + > + def start(self): > + self.fobj.write("%s..." % self.msg) > + self.fobj.flush() > + return self > + > + def update(self, value): > + pass > + > + def finish(self): > + self.fobj.write("done.\n") > + self.fobj.flush() > + > +def new_progress(msg, maxval): > + if interactive: > + return BBProgress(msg, maxval) > + else: > + return NonInteractiveProgress(msg, maxval) > + > +def pluralise(singular, plural, qty): > + if(qty == 1): > + return singular % qty > + else: > + return plural % qty > + > + > +class InteractConsoleLogFilter(logging.Filter): > + def __init__(self, tf, format): > + self.tf = tf > + self.format = format > + > + def filter(self, record): > + if record.levelno == self.format.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")): > + return False > + self.tf.clearFooter() > + return True > + > +class TerminalFilter(object): > + columns = 80 > + > + def sigwinch_handle(self, signum, frame): > + self.columns = self.getTerminalColumns() > + if self._sigwinch_default: > + self._sigwinch_default(signum, frame) > + > + def getTerminalColumns(self): > + def ioctl_GWINSZ(fd): > + try: > + cr = struct.unpack('hh', fcntl.ioctl(fd, self.termios.TIOCGWINSZ, '1234')) > + except: > + return None > + return cr > + cr = ioctl_GWINSZ(sys.stdout.fileno()) > + if not cr: > + try: > + fd = os.open(os.ctermid(), os.O_RDONLY) > + cr = ioctl_GWINSZ(fd) > + os.close(fd) > + except: > + pass > + if not cr: > + try: > + cr = (env['LINES'], env['COLUMNS']) > + except: > + cr = (25, 80) > + return cr[1] > + > + def __init__(self, main, helper, console, format): > + self.main = main > + self.helper = helper > + self.cuu = None > + self.stdinbackup = None > + self.interactive = sys.stdout.isatty() > + self.footer_present = False > + self.lastpids = [] > + > + if not self.interactive: > + return > + > + try: > + import curses > + except ImportError: > + sys.exit("FATAL: The knotty ui could not load the required curses python module.") > + > + import termios > + self.curses = curses > + self.termios = termios > + try: > + fd = sys.stdin.fileno() > + self.stdinbackup = termios.tcgetattr(fd) > + new = copy.deepcopy(self.stdinbackup) > + new[3] = new[3] & ~termios.ECHO > + termios.tcsetattr(fd, termios.TCSADRAIN, new) > + curses.setupterm() > + if curses.tigetnum("colors") > 2: > + format.enable_color() > + self.ed = curses.tigetstr("ed") > + if self.ed: > + self.cuu = curses.tigetstr("cuu") > + try: > + self._sigwinch_default = signal.getsignal(signal.SIGWINCH) > + signal.signal(signal.SIGWINCH, self.sigwinch_handle) > + except: > + pass > + self.columns = self.getTerminalColumns() > + except: > + self.cuu = None > + console.addFilter(InteractConsoleLogFilter(self, format)) > + > + def clearFooter(self): > + if self.footer_present: > + lines = self.footer_present > + sys.stdout.write(self.curses.tparm(self.cuu, lines)) > + sys.stdout.write(self.curses.tparm(self.ed)) > + self.footer_present = False > + > + def updateFooter(self): > + if not self.cuu: > + return > + activetasks = self.helper.running_tasks > + failedtasks = self.helper.failed_tasks > + runningpids = self.helper.running_pids > + if self.footer_present and (self.lastcount == self.helper.tasknumber_current) and (self.lastpids == runningpids): > + return > + if self.footer_present: > + self.clearFooter() > + if (not self.helper.tasknumber_total or self.helper.tasknumber_current == self.helper.tasknumber_total) and not len(activetasks): > + return > + tasks = [] > + for t in runningpids: > + tasks.append("%s (pid %s)" % (activetasks[t]["title"], t)) > + > + if self.main.shutdown: > + content = "Waiting for %s running tasks to finish:" % len(activetasks) > + elif not len(activetasks): > + content = "No currently running tasks (%s of %s)" % (self.helper.tasknumber_current, self.helper.tasknumber_total) > + else: > + content = "Currently %s running tasks (%s of %s):" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total) > + print(content) > + lines = 1 + int(len(content) / (self.columns + 1)) > + for tasknum, task in enumerate(tasks): > + content = "%s: %s" % (tasknum, task) > + print(content) > + lines = lines + 1 + int(len(content) / (self.columns + 1)) > + self.footer_present = lines > + self.lastpids = runningpids[:] > + self.lastcount = self.helper.tasknumber_current > + > + def finish(self): > + if self.stdinbackup: > + fd = sys.stdin.fileno() > + self.termios.tcsetattr(fd, self.termios.TCSADRAIN, self.stdinbackup) > + > +def main(server, eventHandler, params, tf = TerminalFilter): > + print "This is running in DSI mode" > + > + # Generate an unique ID for this build > + # TODO: for multiple build commands this might not work as expected > + import uuid > + uuid = sessionid = str(uuid.uuid4()) > + > + # Get values of variables which control our output > + includelogs, error = server.runCommand(["getVariable", "BBINCLUDELOGS"]) > + if error: > + logger.error("Unable to get the value of BBINCLUDELOGS variable: %s" % error) > + return 1 > + loglines, error = server.runCommand(["getVariable", "BBINCLUDELOGS_LINES"]) > + if error: > + logger.error("Unable to get the value of BBINCLUDELOGS_LINES variable: %s" % error) > + return 1 > + consolelogfile, error = server.runCommand(["getVariable", "BB_CONSOLELOG"]) > + if error: > + logger.error("Unable to get the value of BB_CONSOLELOG variable: %s" % error) > + return 1 > + > + if sys.stdin.isatty() and sys.stdout.isatty(): > + log_exec_tty = True > + else: > + log_exec_tty = False > + > + helper = uihelper.BBUIHelper() > + > + console = logging.StreamHandler(sys.stdout) > + format_str = "%(levelname)s: %(message)s" > + format = bb.msg.BBLogFormatter(format_str) > + bb.msg.addDefaultlogFilter(console) > + console.setFormatter(format) > + logger.addHandler(console) > + > + if consolelogfile and not params.options.show_environment: > + bb.utils.mkdirhier(os.path.dirname(consolelogfile)) > + conlogformat = bb.msg.BBLogFormatter(format_str) > + consolelog = logging.FileHandler(consolelogfile) > + bb.msg.addDefaultlogFilter(consolelog) > + consolelog.setFormatter(conlogformat) > + logger.addHandler(consolelog) > + > + try: > + params.updateFromServer(server) > + cmdline = params.parseActions() > + if not cmdline: > + print("Nothing to do. Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information.") > + return 1 > + if 'msg' in cmdline and cmdline['msg']: > + logger.error(cmdline['msg']) > + return 1 > + > + ret, error = server.runCommand(cmdline['action']) > + if error: > + logger.error("Command '%s' failed: %s" % (cmdline, error)) > + return 1 > + elif ret != True: > + logger.error("Command '%s' failed: returned %s" % (cmdline, ret)) > + return 1 > + except xmlrpclib.Fault as x: > + logger.error("XMLRPC Fault getting commandline:\n %s" % x) > + return 1 > + > + parseprogress = None > + cacheprogress = None > + main.shutdown = 0 > + interrupted = False > + return_value = 0 > + errors = 0 > + warnings = 0 > + taskfailures = [] > + wbhbhelper = WebHOBHelper() > + > + termfilter = tf(main, helper, console, format) > + > + while True: > + try: > + termfilter.updateFooter() > + event = eventHandler.waitEvent(0.25) > + > + if event is None: > + if main.shutdown > 1: > + break > + continue > + > + helper.eventHandler(event) > + > + if isinstance(event, bb.runqueue.runQueueExitWait): > + if not main.shutdown: > + main.shutdown = 1 > + > + if isinstance(event, bb.build.TaskStarted): > + wbhbhelper.store_started_task(event, uuid) > + > + if isinstance(event, (bb.build.TaskSucceeded, bb.build.TaskFailedSilent, bb.build.TaskFailed)): > + wbhbhelper.update_stored_tasks(event, uuid) > + > + if isinstance(event, bb.event.LogExecTTY): > + if log_exec_tty: > + tries = event.retries > + while tries: > + print("Trying to run: %s" % event.prog) > + if os.system(event.prog) == 0: > + break > + time.sleep(event.sleep_delay) > + tries -= 1 > + if tries: > + continue > + logger.warn(event.msg) > + continue > + > + if isinstance(event, logging.LogRecord): > + if event.levelno >= format.ERROR: > + errors = errors + 1 > + return_value = 1 > + elif event.levelno == format.WARNING: > + warnings = warnings + 1 > + # For "normal" logging conditions, don't show note logs from tasks > + # but do show them if the user has changed the default log level to > + # include verbose/debug messages > + if event.taskpid != 0 and event.levelno <= format.NOTE: > + continue > + logger.handle(event) > + continue > + > + if isinstance(event, bb.build.TaskFailed): > + return_value = 1 > + logfile = event.logfile > + if logfile and os.path.exists(logfile): > + termfilter.clearFooter() > + bb.error("Logfile of failure stored in: %s" % logfile) > + if includelogs and not event.errprinted: > + print("Log data follows:") > + f = open(logfile, "r") > + lines = [] > + while True: > + l = f.readline() > + if l == '': > + break > + l = l.rstrip() > + if loglines: > + lines.append(' | %s' % l) > + if len(lines) > int(loglines): > + lines.pop(0) > + else: > + print('| %s' % l) > + f.close() > + if lines: > + for line in lines: > + print(line) > + if isinstance(event, bb.build.TaskBase): > + logger.info(event._message) > + continue > + if isinstance(event, bb.event.ParseStarted): > + if event.total == 0: > + continue > + parseprogress = new_progress("Parsing recipes", event.total).start() > + continue > + if isinstance(event, bb.event.ParseProgress): > + parseprogress.update(event.current) > + continue > + if isinstance(event, bb.event.ParseCompleted): > + if not parseprogress: > + continue > + > + parseprogress.finish() > + print(("Parsing of %d .bb files complete (%d cached, %d parsed). %d targets, %d skipped, %d masked, %d errors." > + % ( event.total, event.cached, event.parsed, event.virtuals, event.skipped, event.masked, event.errors))) > + continue > + > + if isinstance(event, bb.event.CacheLoadStarted): > + cacheprogress = new_progress("Loading cache", event.total).start() > + continue > + if isinstance(event, bb.event.CacheLoadProgress): > + cacheprogress.update(event.current) > + continue > + if isinstance(event, bb.event.CacheLoadCompleted): > + cacheprogress.finish() > + print("Loaded %d entries from dependency cache." % event.num_entries) > + continue > + > + if isinstance(event, bb.command.CommandFailed): > + return_value = event.exitcode > + errors = errors + 1 > + logger.error("Command execution failed: %s", event.error) > + main.shutdown = 2 DSI should never exit. > + continue > + if isinstance(event, bb.command.CommandExit): > + if not return_value: > + return_value = event.exitcode > + continue > + if isinstance(event, (bb.command.CommandCompleted, bb.cooker.CookerExit)): > + main.shutdown = 2 DSI should never exit. > + continue > + if isinstance(event, bb.event.MultipleProviders): > + logger.info("multiple providers are available for %s%s (%s)", event._is_runtime and "runtime " or "", > + event._item, > + ", ".join(event._candidates)) > + logger.info("consider defining a PREFERRED_PROVIDER entry to match %s", event._item) > + continue > + if isinstance(event, bb.event.NoProvider): > + return_value = 1 > + errors = errors + 1 > + if event._runtime: > + r = "R" > + else: > + r = "" > + > + if event._dependees: > + logger.error("Nothing %sPROVIDES '%s' (but %s %sDEPENDS on or otherwise requires it)", r, event._item, ", ".join(event._dependees), r) > + else: > + logger.error("Nothing %sPROVIDES '%s'", r, event._item) > + if event._reasons: > + for reason in event._reasons: > + logger.error("%s", reason) > + continue > + > + if isinstance(event, bb.runqueue.sceneQueueTaskStarted): > + logger.info("Running setscene task %d of %d (%s)" % (event.stats.completed + event.stats.active + event.stats.failed + 1, event.stats.total, event.taskstring)) > + continue > + > + if isinstance(event, bb.runqueue.runQueueTaskStarted): > + if event.noexec: > + tasktype = 'noexec task' > + else: > + tasktype = 'task' > + logger.info("Running %s %s of %s (ID: %s, %s)", > + tasktype, > + event.stats.completed + event.stats.active + > + event.stats.failed + 1, > + event.stats.total, event.taskid, event.taskstring) > + continue > + > + if isinstance(event, bb.runqueue.runQueueTaskFailed): > + taskfailures.append(event.taskstring) > + logger.error("Task %s (%s) failed with exit code '%s'", > + event.taskid, event.taskstring, event.exitcode) > + continue > + > + if isinstance(event, bb.runqueue.sceneQueueTaskFailed): > + logger.warn("Setscene task %s (%s) failed with exit code '%s' - real task will be run instead", > + event.taskid, event.taskstring, event.exitcode) > + continue > + > + if isinstance(event, bb.event.ConfigParsed): > + # timestamp should be added for this > + continue > + > + if isinstance(event, bb.event.RecipeParsed): > + # timestamp should be added for this > + continue > + > + if isinstance(event, bb.event.OperationStarted): > + # timestamp should be added for this > + continue > + > + if isinstance(event, bb.event.OperationCompleted): > + # timestamp should be added for this > + # signal a complete operation > + # calculate timing > + continue > + > + if isinstance(event, bb.event.DiskFull): > + # trigger an error > + continue > + > + # ignore > + if isinstance(event, (bb.event.BuildBase, > + bb.event.StampUpdate, > + bb.event.RecipePreFinalise, > + bb.runqueue.runQueueEvent, > + bb.runqueue.runQueueExitWait, > + bb.event.OperationProgress)): > + continue > + > + logger.error("Unknown event: %s", event) > + > + except EnvironmentError as ioerror: > + termfilter.clearFooter() > + # ignore interrupted io > + if ioerror.args[0] == 4: > + pass > + except KeyboardInterrupt: > + termfilter.clearFooter() > + if main.shutdown == 1: > + print("\nSecond Keyboard Interrupt, stopping...\n") > + _, error = server.runCommand(["stateStop"]) > + if error: > + logger.error("Unable to cleanly stop: %s" % error) > + if main.shutdown == 0: > + print("\nKeyboard Interrupt, closing down...\n") > + interrupted = True > + _, error = server.runCommand(["stateShutdown"]) > + if error: > + logger.error("Unable to cleanly shutdown: %s" % error) > + main.shutdown = main.shutdown + 1 > + pass > + > + summary = "" > + if taskfailures: > + summary += pluralise("\nSummary: %s task failed:", > + "\nSummary: %s tasks failed:", len(taskfailures)) > + for failure in taskfailures: > + summary += "\n %s" % failure > + if warnings: > + summary += pluralise("\nSummary: There was %s WARNING message shown.", > + "\nSummary: There were %s WARNING messages shown.", warnings) > + if return_value: > + summary += pluralise("\nSummary: There was %s ERROR message shown, returning a non-zero exit code.", > + "\nSummary: There were %s ERROR messages shown, returning a non-zero exit code.", errors) > + if summary: > + print(summary) > + > + if interrupted: > + print("Execution was interrupted, returning a non-zero exit code.") > + if return_value == 0: > + return_value = 1 > + > + termfilter.finish() We don't expect to ever get here - DSI should never exit. > + wbhbhelper.write_in_database(wbhbhelper.tasks_information) Probably we need to write tasks as they become complete (i.e. TaskFailed or TaskSucceeded or TaskInvalid comes through) and then clear the memory. This will prevent memory overusage. > + > + return return_value > + ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2013-06-13 15:48 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-06-12 15:11 [PATCH 1/3] Create main WEBHOB project Calin Dragomir 2013-06-12 15:11 ` [Webhob] " Calin Dragomir 2013-06-12 15:11 ` [PATCH 2/3] Create Tasks Django model Calin Dragomir 2013-06-12 15:11 ` [Webhob] " Calin Dragomir 2013-06-12 15:11 ` [PATCH 3/3] Create Data Store Interface (DSI) file Calin Dragomir 2013-06-12 15:11 ` [Webhob] " Calin Dragomir 2013-06-13 15:48 ` Alex Damian
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.