[PATCH 0/9] python 3 fixes

[PATCH 0/9] python 3 fixes

[Ed Bartosh]

Hi,

With this set of fixes I was able to run Toaster and build
core-image-minimal for 'Local Yocto Project'.

The following changes since commit 14d075921ef5c5577a0a0acc394399b10f4fcbe8:

  toaster: Rework displaying package dependencies across Toaster (2016-06-06 15:25:44 +0300)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/toaster/python3
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/python3

Ed Bartosh (6):
  bitbake: toaster: use python3 explicitly
  toaster: replace viewkeys() -> keys()
  toaster: fix migrations
  toaster: decode response content
  toaster: fix incorrect file mode
  toaster: fix test_toaster_tables

Elliot Smith (3):
  toaster: fix broken reference to urllib
  toaster-tests: fix tests for latest Selenium version
  toaster: open image files in binary mode when sending in response

 bitbake/bin/toaster                                |  4 +-
 bitbake/lib/bb/ui/buildinfohelper.py               |  2 +-
 .../migrations/0004_auto_20160523_1446.py          | 34 +++++++++
 .../orm/migrations/0007_auto_20160523_1446.py      | 89 ++++++++++++++++++++++
 bitbake/lib/toaster/tests/browser/README           |  7 +-
 .../toaster/tests/browser/test_js_unit_tests.py    |  8 +-
 bitbake/lib/toaster/toastergui/tests.py            | 46 +++++------
 bitbake/lib/toaster/toastergui/views.py            |  2 +-
 bitbake/lib/toaster/toastergui/widgets.py          |  2 +-
 9 files changed, 156 insertions(+), 38 deletions(-)
 create mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0004_auto_20160523_1446.py
 create mode 100644 bitbake/lib/toaster/orm/migrations/0007_auto_20160523_1446.py

--
Regards,
Ed

[PATCH 1/9] bitbake: toaster: use python3 explicitly

[Ed Bartosh]

Explicitly used python3 as default python for oe builds
will continue to be python2.

[YOCTO #9584]

(Bitbake rev: fde5c962cb69a11b072d1f238c2371a5137d030d)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 bitbake/bin/toaster | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
index be1bb9c..ad25c43 100755
--- a/bitbake/bin/toaster
+++ b/bitbake/bin/toaster
@@ -115,10 +115,10 @@ verify_prereq() {
     exp='s/Django\([><=]\+\)\([^,]\+\),\([><=]\+\)\(.\+\)/'
     exp=$exp'import sys,django;version=django.get_version().split(".");'
     exp=$exp'sys.exit(not (version \1 "\2".split(".") and version \3 "\4".split(".")))/p'
-    if ! sed -n "$exp" $reqfile | python - ; then
+    if ! sed -n "$exp" $reqfile | python3 - ; then
         req=`grep ^Django $reqfile`
         echo "This program needs $req"
-        echo "Please install with pip install -r $reqfile"
+        echo "Please install with pip3 install --user -r $reqfile"
         return 2
     fi
 
-- 
2.1.4

[PATCH 2/9] toaster: replace viewkeys() -> keys()

[Ed Bartosh]

Python 3 doesn't have dict.viewkeys method, renaming
to keys().

[YOCTO #9584]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index d59d6a5..d5ba629 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -1067,7 +1067,7 @@ class BuildInfoHelper(object):
 
         for t in self.internal_state['targets']:
             if t.is_image == True:
-                output_files = list(evdata.viewkeys())
+                output_files = list(evdata.keys())
                 for output in output_files:
                     if t.target in output and 'rootfs' in output and not output.endswith(".manifest"):
                         self.orm_wrapper.save_target_image_file_information(t, output, evdata[output])
-- 
2.1.4

[PATCH 3/9] toaster: fix migrations

[Ed Bartosh]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 .../migrations/0004_auto_20160523_1446.py          | 34 +++++++++
 .../orm/migrations/0007_auto_20160523_1446.py      | 89 ++++++++++++++++++++++
 2 files changed, 123 insertions(+)
 create mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0004_auto_20160523_1446.py
 create mode 100644 bitbake/lib/toaster/orm/migrations/0007_auto_20160523_1446.py

diff --git a/bitbake/lib/toaster/bldcontrol/migrations/0004_auto_20160523_1446.py b/bitbake/lib/toaster/bldcontrol/migrations/0004_auto_20160523_1446.py
new file mode 100644
index 0000000..3d90629
--- /dev/null
+++ b/bitbake/lib/toaster/bldcontrol/migrations/0004_auto_20160523_1446.py
@@ -0,0 +1,34 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('bldcontrol', '0003_add_cancelling_state'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='buildenvironment',
+            name='bbstate',
+            field=models.IntegerField(default=0, choices=[(0, 'stopped'), (1, 'started')]),
+        ),
+        migrations.AlterField(
+            model_name='buildenvironment',
+            name='betype',
+            field=models.IntegerField(choices=[(0, 'local')]),
+        ),
+        migrations.AlterField(
+            model_name='buildenvironment',
+            name='lock',
+            field=models.IntegerField(default=0, choices=[(0, 'free'), (1, 'lock'), (2, 'running')]),
+        ),
+        migrations.AlterField(
+            model_name='buildrequest',
+            name='state',
+            field=models.IntegerField(default=0, choices=[(0, 'created'), (1, 'queued'), (2, 'in progress'), (3, 'completed'), (4, 'failed'), (5, 'deleted'), (6, 'cancelling'), (7, 'archive')]),
+        ),
+    ]
diff --git a/bitbake/lib/toaster/orm/migrations/0007_auto_20160523_1446.py b/bitbake/lib/toaster/orm/migrations/0007_auto_20160523_1446.py
new file mode 100644
index 0000000..b472e7c
--- /dev/null
+++ b/bitbake/lib/toaster/orm/migrations/0007_auto_20160523_1446.py
@@ -0,0 +1,89 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('orm', '0006_add_cancelled_state'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='build',
+            name='outcome',
+            field=models.IntegerField(default=2, choices=[(0, 'Succeeded'), (1, 'Failed'), (2, 'In Progress'), (3, 'Cancelled')]),
+        ),
+        migrations.AlterField(
+            model_name='helptext',
+            name='area',
+            field=models.IntegerField(choices=[(0, 'variable')]),
+        ),
+        migrations.AlterField(
+            model_name='layer',
+            name='summary',
+            field=models.TextField(default=None, null=True, help_text='One-line description of the layer'),
+        ),
+        migrations.AlterField(
+            model_name='layer_version',
+            name='local_path',
+            field=models.FilePathField(default='/', max_length=1024),
+        ),
+        migrations.AlterField(
+            model_name='layersource',
+            name='sourcetype',
+            field=models.IntegerField(choices=[(0, 'local'), (1, 'layerindex'), (2, 'imported')]),
+        ),
+        migrations.AlterField(
+            model_name='logmessage',
+            name='level',
+            field=models.IntegerField(default=0, choices=[(0, 'info'), (1, 'warn'), (2, 'error'), (3, 'critical'), (-1, 'toaster exception')]),
+        ),
+        migrations.AlterField(
+            model_name='package',
+            name='installed_name',
+            field=models.CharField(default='', max_length=100),
+        ),
+        migrations.AlterField(
+            model_name='package_dependency',
+            name='dep_type',
+            field=models.IntegerField(choices=[(0, 'depends'), (1, 'depends'), (3, 'recommends'), (2, 'recommends'), (4, 'suggests'), (5, 'provides'), (6, 'replaces'), (7, 'conflicts')]),
+        ),
+        migrations.AlterField(
+            model_name='recipe_dependency',
+            name='dep_type',
+            field=models.IntegerField(choices=[(0, 'depends'), (1, 'rdepends')]),
+        ),
+        migrations.AlterField(
+            model_name='release',
+            name='branch_name',
+            field=models.CharField(default='', max_length=50),
+        ),
+        migrations.AlterField(
+            model_name='releasedefaultlayer',
+            name='layer_name',
+            field=models.CharField(default='', max_length=100),
+        ),
+        migrations.AlterField(
+            model_name='target_file',
+            name='inodetype',
+            field=models.IntegerField(choices=[(1, 'regular'), (2, 'directory'), (3, 'symlink'), (4, 'socket'), (5, 'fifo'), (6, 'character'), (7, 'block')]),
+        ),
+        migrations.AlterField(
+            model_name='task',
+            name='outcome',
+            field=models.IntegerField(default=-1, choices=[(-1, 'Not Available'), (0, 'Succeeded'), (1, 'Covered'), (2, 'Cached'), (3, 'Prebuilt'), (4, 'Failed'), (5, 'Empty')]),
+        ),
+        migrations.AlterField(
+            model_name='task',
+            name='script_type',
+            field=models.IntegerField(default=0, choices=[(0, 'N/A'), (2, 'Python'), (3, 'Shell')]),
+        ),
+        migrations.AlterField(
+            model_name='task',
+            name='sstate_result',
+            field=models.IntegerField(default=0, choices=[(0, 'Not Applicable'), (1, 'File not in cache'), (2, 'Failed'), (3, 'Succeeded')]),
+        ),
+    ]
-- 
2.1.4

[PATCH 4/9] toaster: decode response content

[Ed Bartosh]

Decoded response content to str to pass it to json.load as it breaks
in Python 3 with this error:
   TypeError: expected bytes, bytearray or buffer compatible object

[YOCTO #9584]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/toastergui/tests.py | 38 ++++++++++++++++-----------------
 1 file changed, 19 insertions(+), 19 deletions(-)

diff --git a/bitbake/lib/toaster/toastergui/tests.py b/bitbake/lib/toaster/toastergui/tests.py
index 9af1018..722b383 100644
--- a/bitbake/lib/toaster/toastergui/tests.py
+++ b/bitbake/lib/toaster/toastergui/tests.py
@@ -77,7 +77,7 @@ class ViewTests(TestCase):
         self.assertEqual(response.status_code, 200)
         self.assertTrue(response['Content-Type'].startswith('application/json'))
 
-        data = json.loads(response.content)
+        data = json.loads(response.content.decode('utf-8'))
 
         self.assertTrue("error" in data)
         self.assertEqual(data["error"], "ok")
@@ -106,7 +106,7 @@ class ViewTests(TestCase):
             self.assertEqual(response.status_code, 200)
             self.assertTrue(response['Content-Type'].startswith('application/json'))
 
-            data = json.loads(response.content)
+            data = json.loads(response.content.decode('utf-8'))
 
             self.assertTrue("error" in data)
             self.assertEqual(data["error"], "ok")
@@ -157,26 +157,26 @@ class ViewTests(TestCase):
                 'project_id': self.project.id,
                 'dir_path' : "/path/in/repository"}
         response = self.client.post(reverse('xhr_importlayer'), args)
-        data = json.loads(response.content)
+        data = json.loads(response.content.decode('utf-8'))
         self.assertEqual(response.status_code, 200)
         self.assertEqual(data["error"], "ok")
 
         #Test to verify import of a layer successful
         args['name'] = "meta-oe"
         response = self.client.post(reverse('xhr_importlayer'), args)
-        data = json.loads(response.content)
+        data = json.loads(response.content.decode('utf-8'))
         self.assertTrue(data["error"], "ok")
 
         #Test for html tag in the data
         args['<'] = "testing html tag"
         response = self.client.post(reverse('xhr_importlayer'), args)
-        data = json.loads(response.content)
+        data = json.loads(response.content.decode('utf-8'))
         self.assertNotEqual(data["error"], "ok")
 
         #Empty data passed
         args = {}
         response = self.client.post(reverse('xhr_importlayer'), args)
-        data = json.loads(response.content)
+        data = json.loads(response.content.decode('utf-8'))
         self.assertNotEqual(data["error"], "ok")
 
     def test_custom_ok(self):
@@ -186,7 +186,7 @@ class ViewTests(TestCase):
                   'base': self.recipe1.id}
         response = self.client.post(url, params)
         self.assertEqual(response.status_code, 200)
-        data = json.loads(response.content)
+        data = json.loads(response.content.decode('utf-8'))
         self.assertEqual(data['error'], 'ok')
         self.assertTrue('url' in data)
         # get recipe from the database
@@ -202,7 +202,7 @@ class ViewTests(TestCase):
                        {'name': 'custom', 'project': self.project.id}]:
             response = self.client.post(url, params)
             self.assertEqual(response.status_code, 200)
-            data = json.loads(response.content)
+            data = json.loads(response.content.decode('utf-8'))
             self.assertNotEqual(data["error"], "ok")
 
     def test_xhr_custom_wrong_project(self):
@@ -211,7 +211,7 @@ class ViewTests(TestCase):
         params = {'name': 'custom', 'project': 0, "base": self.recipe1.id}
         response = self.client.post(url, params)
         self.assertEqual(response.status_code, 200)
-        data = json.loads(response.content)
+        data = json.loads(response.content.decode('utf-8'))
         self.assertNotEqual(data["error"], "ok")
 
     def test_xhr_custom_wrong_base(self):
@@ -220,7 +220,7 @@ class ViewTests(TestCase):
         params = {'name': 'custom', 'project': self.project.id, "base": 0}
         response = self.client.post(url, params)
         self.assertEqual(response.status_code, 200)
-        data = json.loads(response.content)
+        data = json.loads(response.content.decode('utf-8'))
         self.assertNotEqual(data["error"], "ok")
 
     def test_xhr_custom_details(self):
@@ -235,7 +235,7 @@ class ViewTests(TestCase):
                              'project_id': self.project.id,
                             }
                    }
-        self.assertEqual(json.loads(response.content), expected)
+        self.assertEqual(json.loads(response.content.decode('utf-8')), expected)
 
     def test_xhr_custom_del(self):
         """Test deleting custom recipe"""
@@ -248,12 +248,12 @@ class ViewTests(TestCase):
         url = reverse('xhr_customrecipe_id', args=(recipe.id,))
         response = self.client.delete(url)
         self.assertEqual(response.status_code, 200)
-        self.assertEqual(json.loads(response.content), {"error": "ok"})
+        self.assertEqual(json.loads(response.content.decode('utf-8')), {"error": "ok"})
         # try to delete not-existent recipe
         url = reverse('xhr_customrecipe_id', args=(recipe.id,))
         response = self.client.delete(url)
         self.assertEqual(response.status_code, 200)
-        self.assertNotEqual(json.loads(response.content)["error"], "ok")
+        self.assertNotEqual(json.loads(response.content.decode('utf-8'))["error"], "ok")
 
     def test_xhr_custom_packages(self):
         """Test adding and deleting package to a custom recipe"""
@@ -263,7 +263,7 @@ class ViewTests(TestCase):
                                                  self.cust_package.id)))
 
         self.assertEqual(response.status_code, 200)
-        self.assertEqual(json.loads(response.content),
+        self.assertEqual(json.loads(response.content.decode('utf-8')),
                          {"error": "ok"})
         self.assertEqual(self.customr.appends_set.first().name,
                          self.cust_package.name)
@@ -274,7 +274,7 @@ class ViewTests(TestCase):
 
         response = self.client.delete(del_url)
         self.assertEqual(response.status_code, 200)
-        self.assertEqual(json.loads(response.content), {"error": "ok"})
+        self.assertEqual(json.loads(response.content.decode('utf-8')), {"error": "ok"})
         all_packages = self.customr.get_all_packages().values_list('pk',
                                                                    flat=True)
 
@@ -286,7 +286,7 @@ class ViewTests(TestCase):
 
         response = self.client.delete(del_url)
         self.assertEqual(response.status_code, 200)
-        self.assertNotEqual(json.loads(response.content)["error"], "ok")
+        self.assertNotEqual(json.loads(response.content.decode('utf-8'))["error"], "ok")
 
     def test_xhr_custom_packages_err(self):
         """Test error conditions of xhr_customrecipe_packages"""
@@ -297,7 +297,7 @@ class ViewTests(TestCase):
             for method in (self.client.put, self.client.delete):
                 response = method(url)
                 self.assertEqual(response.status_code, 200)
-                self.assertNotEqual(json.loads(response.content),
+                self.assertNotEqual(json.loads(response.content.decode('utf-8')),
                                     {"error": "ok"})
 
     def test_download_custom_recipe(self):
@@ -316,7 +316,7 @@ class ViewTests(TestCase):
         table = SoftwareRecipesTable()
         request = RequestFactory().get('/foo/', {'format': 'json'})
         response = table.get(request, pid=self.project.id)
-        data = json.loads(response.content)
+        data = json.loads(response.content.decode('utf-8'))
 
         recipes = Recipe.objects.filter(Q(is_image=False))
         self.assertTrue(len(recipes) > 1,
@@ -372,7 +372,7 @@ class ViewTests(TestCase):
                     'target_id': 1}
 
             response = table.get(request, **args)
-            return json.loads(response.content)
+            return json.loads(response.content.decode('utf-8'))
 
         def get_text_from_td(td):
             """If we have html in the td then extract the text portion"""
-- 
2.1.4

[PATCH 5/9] toaster: fix incorrect file mode

[Ed Bartosh]

Python 3 complains about 'wa' mode this way:
 ValueError: must have exactly one of create/read/write/append mode

Fixed by using 'a' mode.

[YOCTO #9584]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/toastergui/tests.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bitbake/lib/toaster/toastergui/tests.py b/bitbake/lib/toaster/toastergui/tests.py
index 722b383..31f1431 100644
--- a/bitbake/lib/toaster/toastergui/tests.py
+++ b/bitbake/lib/toaster/toastergui/tests.py
@@ -304,7 +304,7 @@ class ViewTests(TestCase):
         """Download the recipe file generated for the custom image"""
 
         # Create a dummy recipe file for the custom image generation to read
-        open("/tmp/a_recipe.bb", 'wa').close()
+        open("/tmp/a_recipe.bb", 'a').close()
         response = self.client.get(reverse('customrecipedownload',
                                            args=(self.project.id,
                                                  self.customr.id)))
-- 
2.1.4

[PATCH 6/9] toaster: fix test_toaster_tables

[Ed Bartosh]

Removed unneeded code as it causes the following error in Python 3:
    TypeError: expected bytes, bytearray or buffer compatible object

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/toastergui/tests.py | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/bitbake/lib/toaster/toastergui/tests.py b/bitbake/lib/toaster/toastergui/tests.py
index 31f1431..2cd2c7d 100644
--- a/bitbake/lib/toaster/toastergui/tests.py
+++ b/bitbake/lib/toaster/toastergui/tests.py
@@ -382,12 +382,6 @@ class ViewTests(TestCase):
             else:
                 ret = BeautifulSoup(td).text
 
-            # We change the td into ascii as a way to remove characters
-            # such as \xa0 (non break space) which mess with the ordering
-            # comparisons
-            ret.strip().encode('ascii',
-                               errors='replace').replace('?', ' ')
-            # Case where the td is empty
             if len(ret):
                 return "0"
             else:
-- 
2.1.4

[PATCH 7/9] toaster: fix broken reference to urllib

[Ed Bartosh]

From: Elliot Smith <elliot.smith@intel.com>

The code previously imported urllib to make use of querystring
quoting, but was modified to support Python 3. During this
process, the reference to urllib was not fixed, which resulted
in table filters breaking.

Remove the reference to urllib (which is no longer imported)
and instead reference the imported unquote_plus() function.

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/toastergui/widgets.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bitbake/lib/toaster/toastergui/widgets.py b/bitbake/lib/toaster/toastergui/widgets.py
index 4276c2a..005a562 100644
--- a/bitbake/lib/toaster/toastergui/widgets.py
+++ b/bitbake/lib/toaster/toastergui/widgets.py
@@ -219,7 +219,7 @@ class ToasterTable(TemplateView):
 
         try:
             filter_name, action_name = filters.split(':')
-            action_params = urllib.unquote_plus(filter_value)
+            action_params = unquote_plus(filter_value)
         except ValueError:
             return
 
-- 
2.1.4

[PATCH 8/9] toaster-tests: fix tests for latest Selenium version

[Ed Bartosh]

From: Elliot Smith <elliot.smith@intel.com>

Previously, we didn't specify a specific version of Selenium.
When upgrading to Python 3 and installing Selenium to work with it,
the JS unit test broke, as the report format produced by Selenium
had changed.

Modify the test so that it works with the latest Selenium report
format.

Add a note to the README that the given Selenium version should
be used to prevent unexpected test failures.

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/tests/browser/README                | 7 ++++---
 bitbake/lib/toaster/tests/browser/test_js_unit_tests.py | 8 ++++----
 2 files changed, 8 insertions(+), 7 deletions(-)

diff --git a/bitbake/lib/toaster/tests/browser/README b/bitbake/lib/toaster/tests/browser/README
index 63e8169..f57154e 100644
--- a/bitbake/lib/toaster/tests/browser/README
+++ b/bitbake/lib/toaster/tests/browser/README
@@ -4,10 +4,11 @@ These tests require Selenium to be installed in your Python environment.
 
 The simplest way to install this is via pip:
 
-  pip install selenium
+  pip install selenium==2.53.2
 
-Alternatively, if you used pip to install the libraries required by Toaster,
-selenium will already be installed.
+Note that if you use other versions of Selenium, some of the tests (such as
+tests.browser.test_js_unit_tests.TestJsUnitTests) may fail, as these rely on
+a Selenium test report with a version-specific format.
 
 To run tests against Chrome:
 
diff --git a/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py b/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py
index e63da8e..3c0b962 100644
--- a/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py
+++ b/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py
@@ -38,11 +38,11 @@ class TestJsUnitTests(SeleniumTestCase):
     def test_that_js_unit_tests_pass(self):
         url = reverse('js-unit-tests')
         self.get(url)
-        self.wait_until_present('#tests-failed')
+        self.wait_until_present('#qunit-testresult .failed')
 
-        failed = self.find("#tests-failed").text
-        passed = self.find("#tests-passed").text
-        total = self.find("#tests-total").text
+        failed = self.find("#qunit-testresult .failed").text
+        passed = self.find("#qunit-testresult .passed").text
+        total = self.find("#qunit-testresult .total").text
 
         logger.info("Js unit tests completed %s out of %s passed, %s failed",
                     passed,
-- 
2.1.4

[PATCH 9/9] toaster: open image files in binary mode when sending in response

[Ed Bartosh]

From: Elliot Smith <elliot.smith@intel.com>

The view code for downloading image files used the "r" flag
to read the file, then used the open file object to form the
HTTP response.

While this worked in Python 2, Python 3 appears to be more strict
about this sort of thing, and Django throws a UnicodeDecodeError
when a file opened this way is used in a response.

Open the file with the "b" flag (binary mode) so that Django can
correctly convert the binary file handle to an HTTP response.

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/toastergui/views.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bitbake/lib/toaster/toastergui/views.py b/bitbake/lib/toaster/toastergui/views.py
index 1f908ea..16f98ee 100755
--- a/bitbake/lib/toaster/toastergui/views.py
+++ b/bitbake/lib/toaster/toastergui/views.py
@@ -2339,7 +2339,7 @@ if True:
             )
 
             if file_name and response_file_name:
-                fsock = open(file_name, "r")
+                fsock = open(file_name, "rb")
                 content_type = MimeTypeFinder.get_mimetype(file_name)
 
                 response = HttpResponse(fsock, content_type = content_type)
-- 
2.1.4

Re: [PATCH 0/9] python 3 fixes

[Smith, Elliot]

[-- Attachment #1: Type: text/plain, Size: 2392 bytes --]

Thanks Ed. I've been through and reviewed your patches from this series,
and they look good to me.

Michael has said he will review my patches in this series.

Once they're all reviewed, I'll pull them into toaster-next.

Elliot

On 6 June 2016 at 17:01, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:

> Hi,
>
> With this set of fixes I was able to run Toaster and build
> core-image-minimal for 'Local Yocto Project'.
>
> The following changes since commit
> 14d075921ef5c5577a0a0acc394399b10f4fcbe8:
>
>   toaster: Rework displaying package dependencies across Toaster
> (2016-06-06 15:25:44 +0300)
>
> are available in the git repository at:
>
>   git://git.yoctoproject.org/poky-contrib ed/toaster/python3
>
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/python3
>
> Ed Bartosh (6):
>   bitbake: toaster: use python3 explicitly
>   toaster: replace viewkeys() -> keys()
>   toaster: fix migrations
>   toaster: decode response content
>   toaster: fix incorrect file mode
>   toaster: fix test_toaster_tables
>
> Elliot Smith (3):
>   toaster: fix broken reference to urllib
>   toaster-tests: fix tests for latest Selenium version
>   toaster: open image files in binary mode when sending in response
>
>  bitbake/bin/toaster                                |  4 +-
>  bitbake/lib/bb/ui/buildinfohelper.py               |  2 +-
>  .../migrations/0004_auto_20160523_1446.py          | 34 +++++++++
>  .../orm/migrations/0007_auto_20160523_1446.py      | 89
> ++++++++++++++++++++++
>  bitbake/lib/toaster/tests/browser/README           |  7 +-
>  .../toaster/tests/browser/test_js_unit_tests.py    |  8 +-
>  bitbake/lib/toaster/toastergui/tests.py            | 46 +++++------
>  bitbake/lib/toaster/toastergui/views.py            |  2 +-
>  bitbake/lib/toaster/toastergui/widgets.py          |  2 +-
>  9 files changed, 156 insertions(+), 38 deletions(-)
>  create mode 100644
> bitbake/lib/toaster/bldcontrol/migrations/0004_auto_20160523_1446.py
>  create mode 100644
> bitbake/lib/toaster/orm/migrations/0007_auto_20160523_1446.py
>
> --
> Regards,
> Ed
>
> --
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster
>



-- 
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 3493 bytes --]

Re: [PATCH 0/9] python 3 fixes

[Smith, Elliot]

[-- Attachment #1: Type: text/plain, Size: 2857 bytes --]

Hello Ed.

I've just run the Django unit tests, and there are some problems with
those: I get quite a lot of failures. Do you have plans to fix these, or
are you finished with Python 3 patches now?

Thanks.
Elliot

On 7 June 2016 at 10:42, Smith, Elliot <elliot.smith@intel.com> wrote:

> Thanks Ed. I've been through and reviewed your patches from this series,
> and they look good to me.
>
> Michael has said he will review my patches in this series.
>
> Once they're all reviewed, I'll pull them into toaster-next.
>
> Elliot
>
> On 6 June 2016 at 17:01, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
>
>> Hi,
>>
>> With this set of fixes I was able to run Toaster and build
>> core-image-minimal for 'Local Yocto Project'.
>>
>> The following changes since commit
>> 14d075921ef5c5577a0a0acc394399b10f4fcbe8:
>>
>>   toaster: Rework displaying package dependencies across Toaster
>> (2016-06-06 15:25:44 +0300)
>>
>> are available in the git repository at:
>>
>>   git://git.yoctoproject.org/poky-contrib ed/toaster/python3
>>
>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/python3
>>
>> Ed Bartosh (6):
>>   bitbake: toaster: use python3 explicitly
>>   toaster: replace viewkeys() -> keys()
>>   toaster: fix migrations
>>   toaster: decode response content
>>   toaster: fix incorrect file mode
>>   toaster: fix test_toaster_tables
>>
>> Elliot Smith (3):
>>   toaster: fix broken reference to urllib
>>   toaster-tests: fix tests for latest Selenium version
>>   toaster: open image files in binary mode when sending in response
>>
>>  bitbake/bin/toaster                                |  4 +-
>>  bitbake/lib/bb/ui/buildinfohelper.py               |  2 +-
>>  .../migrations/0004_auto_20160523_1446.py          | 34 +++++++++
>>  .../orm/migrations/0007_auto_20160523_1446.py      | 89
>> ++++++++++++++++++++++
>>  bitbake/lib/toaster/tests/browser/README           |  7 +-
>>  .../toaster/tests/browser/test_js_unit_tests.py    |  8 +-
>>  bitbake/lib/toaster/toastergui/tests.py            | 46 +++++------
>>  bitbake/lib/toaster/toastergui/views.py            |  2 +-
>>  bitbake/lib/toaster/toastergui/widgets.py          |  2 +-
>>  9 files changed, 156 insertions(+), 38 deletions(-)
>>  create mode 100644
>> bitbake/lib/toaster/bldcontrol/migrations/0004_auto_20160523_1446.py
>>  create mode 100644
>> bitbake/lib/toaster/orm/migrations/0007_auto_20160523_1446.py
>>
>> --
>> Regards,
>> Ed
>>
>> --
>> _______________________________________________
>> toaster mailing list
>> toaster@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/toaster
>>
>
>
>
> --
> Elliot Smith
> Software Engineer
> Intel Open Source Technology Centre
>



-- 
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 4418 bytes --]

Re: [PATCH 8/9] toaster-tests: fix tests for latest Selenium version

[Michael Wood]

On 06/06/16 17:01, Ed Bartosh wrote:
> From: Elliot Smith <elliot.smith@intel.com>
>
> Previously, we didn't specify a specific version of Selenium.
> When upgrading to Python 3 and installing Selenium to work with it,
> the JS unit test broke, as the report format produced by Selenium
> had changed.
>
> Modify the test so that it works with the latest Selenium report
> format.
>
> Add a note to the README that the given Selenium version should
> be used to prevent unexpected test failures.
>
> Signed-off-by: Elliot Smith <elliot.smith@intel.com>
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> ---
>   bitbake/lib/toaster/tests/browser/README                | 7 ++++---
>   bitbake/lib/toaster/tests/browser/test_js_unit_tests.py | 8 ++++----
>   2 files changed, 8 insertions(+), 7 deletions(-)
>
> diff --git a/bitbake/lib/toaster/tests/browser/README b/bitbake/lib/toaster/tests/browser/README
> index 63e8169..f57154e 100644
> --- a/bitbake/lib/toaster/tests/browser/README
> +++ b/bitbake/lib/toaster/tests/browser/README
> @@ -4,10 +4,11 @@ These tests require Selenium to be installed in your Python environment.
>   
>   The simplest way to install this is via pip:
>   
> -  pip install selenium
> +  pip install selenium==2.53.2
>   
> -Alternatively, if you used pip to install the libraries required by Toaster,
> -selenium will already be installed.
> +Note that if you use other versions of Selenium, some of the tests (such as
> +tests.browser.test_js_unit_tests.TestJsUnitTests) may fail, as these rely on
> +a Selenium test report with a version-specific format.
>   
>   To run tests against Chrome:
>   
> diff --git a/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py b/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py
> index e63da8e..3c0b962 100644
> --- a/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py
> +++ b/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py
> @@ -38,11 +38,11 @@ class TestJsUnitTests(SeleniumTestCase):
>       def test_that_js_unit_tests_pass(self):
>           url = reverse('js-unit-tests')
>           self.get(url)
> -        self.wait_until_present('#tests-failed')
> +        self.wait_until_present('#qunit-testresult .failed')
>   
> -        failed = self.find("#tests-failed").text
> -        passed = self.find("#tests-passed").text
> -        total = self.find("#tests-total").text

If these elements are no longer needed, can you also remove them from 
the jsunit test page (static/js/tests/tests.js) thanks

> +        failed = self.find("#qunit-testresult .failed").text
> +        passed = self.find("#qunit-testresult .passed").text
> +        total = self.find("#qunit-testresult .total").text
>   
>           logger.info("Js unit tests completed %s out of %s passed, %s failed",
>                       passed,

Re: [PATCH 0/9] python 3 fixes

[Michael Wood]

I've looked at Elliot's patches, apart from that one little thing it 
looks good to me.

Thanks,

Michael

On 07/06/16 10:42, Smith, Elliot wrote:
> Thanks Ed. I've been through and reviewed your patches from this 
> series, and they look good to me.
>
> Michael has said he will review my patches in this series.
>
> Once they're all reviewed, I'll pull them into toaster-next.
>
> Elliot
>
> On 6 June 2016 at 17:01, Ed Bartosh <ed.bartosh@linux.intel.com 
> <mailto:ed.bartosh@linux.intel.com>> wrote:
>
>     Hi,
>
>     With this set of fixes I was able to run Toaster and build
>     core-image-minimal for 'Local Yocto Project'.
>
>     The following changes since commit
>     14d075921ef5c5577a0a0acc394399b10f4fcbe8:
>
>       toaster: Rework displaying package dependencies across Toaster
>     (2016-06-06 15:25:44 +0300)
>
>     are available in the git repository at:
>
>       git://git.yoctoproject.org/poky-contrib
>     <http://git.yoctoproject.org/poky-contrib> ed/toaster/python3
>     http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/python3
>
>     Ed Bartosh (6):
>       bitbake: toaster: use python3 explicitly
>       toaster: replace viewkeys() -> keys()
>       toaster: fix migrations
>       toaster: decode response content
>       toaster: fix incorrect file mode
>       toaster: fix test_toaster_tables
>
>     Elliot Smith (3):
>       toaster: fix broken reference to urllib
>       toaster-tests: fix tests for latest Selenium version
>       toaster: open image files in binary mode when sending in response
>
>      bitbake/bin/toaster                                |  4 +-
>      bitbake/lib/bb/ui/buildinfohelper.py               |  2 +-
>      .../migrations/0004_auto_20160523_1446.py          | 34 +++++++++
>      .../orm/migrations/0007_auto_20160523_1446.py      | 89
>     ++++++++++++++++++++++
>      bitbake/lib/toaster/tests/browser/README           |  7 +-
>      .../toaster/tests/browser/test_js_unit_tests.py    |  8 +-
>      bitbake/lib/toaster/toastergui/tests.py            | 46 +++++------
>      bitbake/lib/toaster/toastergui/views.py            |  2 +-
>      bitbake/lib/toaster/toastergui/widgets.py          |  2 +-
>      9 files changed, 156 insertions(+), 38 deletions(-)
>      create mode 100644
>     bitbake/lib/toaster/bldcontrol/migrations/0004_auto_20160523_1446.py
>      create mode 100644
>     bitbake/lib/toaster/orm/migrations/0007_auto_20160523_1446.py
>
>     --
>     Regards,
>     Ed
>
>     --
>     _______________________________________________
>     toaster mailing list
>     toaster@yoctoproject.org <mailto:toaster@yoctoproject.org>
>     https://lists.yoctoproject.org/listinfo/toaster
>
>
>
>
> -- 
> Elliot Smith
> Software Engineer
> Intel Open Source Technology Centre
>
>

Re: [PATCH 0/9] python 3 fixes

[Smith, Elliot]

[-- Attachment #1: Type: text/plain, Size: 3844 bytes --]

I resubmitted this patch series with the change Michael requested, and also
removed one of the patches from the original submission (1/9 toaster: use
python3 explicitly), which is already in master.

Elliot

On 7 June 2016 at 14:35, Michael Wood <michael.g.wood@intel.com> wrote:

> I've looked at Elliot's patches, apart from that one little thing it looks
> good to me.
>
> Thanks,
>
> Michael
>
> On 07/06/16 10:42, Smith, Elliot wrote:
>
>> Thanks Ed. I've been through and reviewed your patches from this series,
>> and they look good to me.
>>
>> Michael has said he will review my patches in this series.
>>
>> Once they're all reviewed, I'll pull them into toaster-next.
>>
>> Elliot
>>
>> On 6 June 2016 at 17:01, Ed Bartosh <ed.bartosh@linux.intel.com <mailto:
>> ed.bartosh@linux.intel.com>> wrote:
>>
>>     Hi,
>>
>>     With this set of fixes I was able to run Toaster and build
>>     core-image-minimal for 'Local Yocto Project'.
>>
>>     The following changes since commit
>>     14d075921ef5c5577a0a0acc394399b10f4fcbe8:
>>
>>       toaster: Rework displaying package dependencies across Toaster
>>     (2016-06-06 15:25:44 +0300)
>>
>>     are available in the git repository at:
>>
>>       git://git.yoctoproject.org/poky-contrib
>>     <http://git.yoctoproject.org/poky-contrib> ed/toaster/python3
>>
>>
>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/python3
>>
>>     Ed Bartosh (6):
>>       bitbake: toaster: use python3 explicitly
>>       toaster: replace viewkeys() -> keys()
>>       toaster: fix migrations
>>       toaster: decode response content
>>       toaster: fix incorrect file mode
>>       toaster: fix test_toaster_tables
>>
>>     Elliot Smith (3):
>>       toaster: fix broken reference to urllib
>>       toaster-tests: fix tests for latest Selenium version
>>       toaster: open image files in binary mode when sending in response
>>
>>      bitbake/bin/toaster                                |  4 +-
>>      bitbake/lib/bb/ui/buildinfohelper.py               |  2 +-
>>      .../migrations/0004_auto_20160523_1446.py          | 34 +++++++++
>>      .../orm/migrations/0007_auto_20160523_1446.py      | 89
>>     ++++++++++++++++++++++
>>      bitbake/lib/toaster/tests/browser/README           |  7 +-
>>      .../toaster/tests/browser/test_js_unit_tests.py    |  8 +-
>>      bitbake/lib/toaster/toastergui/tests.py            | 46 +++++------
>>      bitbake/lib/toaster/toastergui/views.py            |  2 +-
>>      bitbake/lib/toaster/toastergui/widgets.py          |  2 +-
>>      9 files changed, 156 insertions(+), 38 deletions(-)
>>      create mode 100644
>>     bitbake/lib/toaster/bldcontrol/migrations/0004_auto_20160523_1446.py
>>      create mode 100644
>>     bitbake/lib/toaster/orm/migrations/0007_auto_20160523_1446.py
>>
>>     --
>>     Regards,
>>     Ed
>>
>>     --
>>     _______________________________________________
>>     toaster mailing list
>>     toaster@yoctoproject.org <mailto:toaster@yoctoproject.org>
>>     https://lists.yoctoproject.org/listinfo/toaster
>>
>>
>>
>>
>> --
>> Elliot Smith
>> Software Engineer
>> Intel Open Source Technology Centre
>>
>>
>>
> ---------------------------------------------------------------------
> Intel Corporation (UK) Limited
> Registered No. 1134945 (England)
> Registered Office: Pipers Way, Swindon SN3 1RJ
> VAT No: 860 2173 47
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
>
>


-- 
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 5528 bytes --]