[PATCH 0/2] rationalise mime type guessing in toaster

[PATCH 0/2] rationalise mime type guessing in toaster

[brian avery]

This eliminates the filemagic external requirement and fixes the
image download links.

-b

The following changes since commit 19e5f3378e7f6dad9696ef97c074c738dbcef157:

  bitbake: toaster: display warnings for bad "IMAGE_FSTYPES" values (2015-10-06 19:50:29 -0700)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib bavery/submit/elliot/2015-09-24_rationalise-mime-type-8369_t2
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=bavery/submit/elliot/2015-09-24_rationalise-mime-type-8369_t2

Elliot Smith (2):
  toaster: Use Python's mimetypes module
  toaster: Rationalise mimetype guessing to fix artifact downloads

 lib/toaster/bldcontrol/models.py | 34 ----------------------------------
 lib/toaster/toastergui/views.py  | 17 ++++++++++++-----
 toaster-requirements.txt         |  1 -
 3 files changed, 12 insertions(+), 40 deletions(-)

--
1.9.1

[PATCH 1/2] toaster: Use Python's mimetypes module

[brian avery]

From: Elliot Smith <elliot.smith@intel.com>

filemagic is used to guess the mimetype of files when a user
requests a download. However, this adds a dependency on an
external library.

Python does have a mimetypes module, though this guesses the
mimetype rather than doing anything clever with the actual
file content. But for our purposes, it's more than adequate.
(NB Django also uses this module when serving static files.)

Use this instead of relying on any external code, and remove
the filemagic dependency.

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
---
 lib/toaster/toastergui/views.py | 15 +++++++++++----
 toaster-requirements.txt        |  1 -
 2 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/lib/toaster/toastergui/views.py b/lib/toaster/toastergui/views.py
index e07f0e2..9e4cf55 100755
--- a/lib/toaster/toastergui/views.py
+++ b/lib/toaster/toastergui/views.py
@@ -47,19 +47,26 @@ import json
 from os.path import dirname
 from functools import wraps
 import itertools
+import mimetypes
 
-import magic
 import logging
 
 logger = logging.getLogger("toaster")
 
 class MimeTypeFinder(object):
-    _magic = magic.Magic(flags = magic.MAGIC_MIME_TYPE)
+    # setting this to False enables additional non-standard mimetypes
+    # to be included in the guess
+    _strict = False
 
-    # returns the mimetype for a file path
+    # returns the mimetype for a file path as a string,
+    # or 'application/octet-stream' if the type couldn't be guessed
     @classmethod
     def get_mimetype(self, path):
-        return self._magic.id_filename(path)
+        guess = mimetypes.guess_type(path, self._strict)
+        guessed_type = guess[0]
+        if guessed_type == None:
+            guessed_type = 'application/octet-stream'
+        return guessed_type
 
 # all new sessions should come through the landing page;
 # determine in which mode we are running in, and redirect appropriately
diff --git a/toaster-requirements.txt b/toaster-requirements.txt
index c4a2221..1d7d21b 100644
--- a/toaster-requirements.txt
+++ b/toaster-requirements.txt
@@ -2,5 +2,4 @@ Django==1.6
 South==0.8.4
 argparse==1.2.1
 wsgiref==0.1.2
-filemagic==1.6
 beautifulsoup4>=4.4.0
-- 
1.9.1

[PATCH 2/2] toaster: Rationalise mimetype guessing to fix artifact downloads

[brian avery]

From: Elliot Smith <elliot.smith@intel.com>

Artifact download links were broken because the function to
get the mimetype for the artifact was incorrectly using the
underlying mimetype library. The function was also attached
to the build environment controller, which was unnecessary, as
we only support local controllers anyway.

Remove the mimetype getter on the build environment and
use the one in the view code instead. This works correctly
and prevents the download error from occurring.

[YOCTO #8369]

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
---
 lib/toaster/bldcontrol/models.py | 34 ----------------------------------
 lib/toaster/toastergui/views.py  |  2 +-
 2 files changed, 1 insertion(+), 35 deletions(-)

diff --git a/lib/toaster/bldcontrol/models.py b/lib/toaster/bldcontrol/models.py
index f2493a8..ab41105 100644
--- a/lib/toaster/bldcontrol/models.py
+++ b/lib/toaster/bldcontrol/models.py
@@ -39,40 +39,6 @@ class BuildEnvironment(models.Model):
     created     = models.DateTimeField(auto_now_add = True)
     updated     = models.DateTimeField(auto_now = True)
 
-
-    def get_artifact_type(self, path):
-        if self.betype == BuildEnvironment.TYPE_LOCAL:
-            try:
-                import magic
-
-                # fair warning: this is a mess; there are multiple competeing and incompatible
-                # magic modules floating around, so we try some of the most common combinations
-
-                try:    # we try ubuntu's python-magic 5.4
-                    m = magic.open(magic.MAGIC_MIME_TYPE)
-                    m.load()
-                    return m.file(path)
-                except AttributeError:
-                    pass
-
-                try:    # we try python-magic 0.4.6
-                    m = magic.Magic(magic.MAGIC_MIME)
-                    return m.from_file(path)
-                except AttributeError:
-                    pass
-
-                try:    # we try pip filemagic 1.6
-                    m = magic.Magic(flags=magic.MAGIC_MIME_TYPE)
-                    return m.id_filename(path)
-                except AttributeError:
-                    pass
-
-                return "binary/octet-stream"
-            except ImportError:
-                return "binary/octet-stream"
-        raise Exception("FIXME: artifact type not implemented for build environment type %s" % self.get_betype_display())
-
-
     def get_artifact(self, path):
         if self.betype == BuildEnvironment.TYPE_LOCAL:
             return open(path, "r")
diff --git a/lib/toaster/toastergui/views.py b/lib/toaster/toastergui/views.py
index 9e4cf55..468cce3 100755
--- a/lib/toaster/toastergui/views.py
+++ b/lib/toaster/toastergui/views.py
@@ -2981,7 +2981,7 @@ if True:
             if file_name is None:
                 raise Exception("Could not handle artifact %s id %s" % (artifact_type, artifact_id))
             else:
-                content_type = b.buildrequest.environment.get_artifact_type(file_name)
+                content_type = MimeTypeFinder.get_mimetype(file_name)
                 fsock = b.buildrequest.environment.get_artifact(file_name)
                 file_name = os.path.basename(file_name) # we assume that the build environment system has the same path conventions as host
 
-- 
1.9.1