* [Buildroot] [PATCH 0/3] [RFC] python-package-generator
@ 2015-06-01 14:56 Denis THULIN
2015-06-01 14:56 ` [Buildroot] [PATCH 1/3] [RFC] python-package-generator: new utility Denis THULIN
` (2 more replies)
0 siblings, 3 replies; 8+ messages in thread
From: Denis THULIN @ 2015-06-01 14:56 UTC (permalink / raw)
To: buildroot
This patch series:
* Add utility python-package-generator.py
* Add package python-robotframework
* Add package python-magic
python-package-generator is an utility intended to automatically generate
python packages for buildroot using metadata provided by the python package
index (https://pypi.python.org).
python-robotframework and python magic are packages generated using
python-package-generator.py
Denis THULIN (3):
python-package-generator: new utility
python-robotframework: New package
python-magic: new package
docs/manual/adding-packages-python.txt | 36 ++
package/Config.in | 2 +
package/python-magic/Config.in | 7 +
package/python-magic/python-magic.hash | 2 +
package/python-magic/python-magic.mk | 14 +
package/python-robotframework/Config.in | 9 +
.../python-robotframework.hash | 2 +
.../python-robotframework/python-robotframework.mk | 14 +
support/scripts/python-package-generator.py | 435 +++++++++++++++++++++
9 files changed, 521 insertions(+)
create mode 100644 package/python-magic/Config.in
create mode 100644 package/python-magic/python-magic.hash
create mode 100644 package/python-magic/python-magic.mk
create mode 100644 package/python-robotframework/Config.in
create mode 100644 package/python-robotframework/python-robotframework.hash
create mode 100644 package/python-robotframework/python-robotframework.mk
create mode 100755 support/scripts/python-package-generator.py
--
2.4.2
^ permalink raw reply [flat|nested] 8+ messages in thread
* [Buildroot] [PATCH 1/3] [RFC] python-package-generator: new utility
2015-06-01 14:56 [Buildroot] [PATCH 0/3] [RFC] python-package-generator Denis THULIN
@ 2015-06-01 14:56 ` Denis THULIN
2015-06-01 22:37 ` Arnout Vandecappelle
2015-06-10 7:56 ` Thomas Petazzoni
2015-06-01 14:56 ` [Buildroot] [PATCH 2/3] python-robotframework: New package Denis THULIN
2015-06-01 14:56 ` [Buildroot] [PATCH 3/3] python-magic: new package Denis THULIN
2 siblings, 2 replies; 8+ messages in thread
From: Denis THULIN @ 2015-06-01 14:56 UTC (permalink / raw)
To: buildroot
This patch adds package python-package-generator.py
---
v0: initial commit
- python-pacakage-generator.py is an utility for automatically generating
python packages using metadata from the python package index:
https://pypi.python.org
I did not know where to put the script so I put it in support/scripts.
I have updated the python-package section of the manual as well.
Please bear in mind that python-package-generator.py does not add the packages
to your buildroot project and you need to do it manually.
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
docs/manual/adding-packages-python.txt | 36 +++
support/scripts/python-package-generator.py | 435 ++++++++++++++++++++++++++++
2 files changed, 471 insertions(+)
create mode 100755 support/scripts/python-package-generator.py
diff --git a/docs/manual/adding-packages-python.txt b/docs/manual/adding-packages-python.txt
index f81d625..6f608ed 100644
--- a/docs/manual/adding-packages-python.txt
+++ b/docs/manual/adding-packages-python.txt
@@ -7,6 +7,42 @@ This infrastructure applies to Python packages that use the standard
Python setuptools mechanism as their build system, generally
recognizable by the usage of a +setup.py+ script.
+[[python-package-generator]]
+
+==== generating a +python-package+ from a pypi repository
+
+You may want to use the +python-package-generator.py+ located in
++support/script+ to generate a package from an existing pypi(pip) package.
+
+you can find the list of existing pypi package here: (https://pypi.python.org).
+
+Please keep in mind that you most likely need
+to manually check the package for any mistakes
+as there are things that cannot be guessed by the generator (e.g.
+dependencies on any of the python core modules
+such as BR2_PACKAGE_PYTHON_ZLIB).
+
+When at the root of your buildroot directory just do :
+
+-----------------------
+./support/script/python-package-generator.py foo bar -o package
+-----------------------
+
+This will generate packages +python-foo+ and +python-bar+ in the package
+folder if they exist on https://pypi.python.org.
+
+You will need to manually write the path to the package inside
+the +package/Config.in+ file:
+
+Find the +external python modules+ menu and insert your package inside.
+Keep in mind that the items inside a menu should be in alphabetical order.
+
+Option +-h+ wil list the options available
+
+-----------------------
+./support/script/python-package-generator.py -h
+-----------------------
+
[[python-package-tutorial]]
==== +python-package+ tutorial
diff --git a/support/scripts/python-package-generator.py b/support/scripts/python-package-generator.py
new file mode 100755
index 0000000..4f5e884
--- /dev/null
+++ b/support/scripts/python-package-generator.py
@@ -0,0 +1,435 @@
+#!/usr/bin/python2
+"""
+ Utility for building buildroot packages for existing pypi packages
+
+ Any package built by brpy-generator should be manually checked for errors.
+"""
+from __future__ import print_function
+import argparse
+import json
+import urllib2
+import sys
+import os
+import shutil
+import StringIO
+import tarfile
+import errno
+import hashlib
+import re
+import magic
+import tempfile
+from functools import wraps
+
+# TODO: Create a real module instead of a 320 line script
+
+# private global
+_calls = {}
+
+
+def setup_info(pkg_name):
+ """Get a package info from _calls
+
+ Keyword arguments:
+ pkg_name -- the name of the package
+ """
+ return _calls[pkg_name]
+
+
+def setup_decorator(func, method):
+ """
+ Decorator for distutils.core.setup and setuptools.setup.
+ Puts the args of setup as a dict inside global private dict _calls.
+ Add key 'method' which should be either 'setuptools' or 'distutils'.
+
+ Keyword arguments:
+ func -- either setuptools.setup or distutils.core.setup
+ method -- either 'setuptools' or 'distutils'
+ """
+
+ @wraps(func)
+ def closure(*args, **kwargs):
+ _calls[kwargs['name']] = kwargs
+ _calls[kwargs['name']]['method'] = method
+
+ return closure
+
+
+def find_file_upper_case(filenames, path='./'):
+ """
+ List generator:
+ Recursively find files that matches one of the specified filenames.
+ Returns absolute path
+
+ Keyword arguments:
+ filenames -- List of filenames to be found
+ path -- Path to the directory to search
+ """
+ for root, dirs, files in os.walk(path):
+ for file in files:
+ if file.upper() in filenames:
+ yield (os.path.join(root, file))
+
+
+def pkg_new_name(pkg_name):
+ """
+ Returns name to avoid troublesome characters.
+ Remove all non alphanumeric characters except -
+ Also lowers the name
+
+ Keyword arguments:
+ pkg_name -- String to rename
+ """
+ name = re.sub('[^\w-]', '', pkg_name.lower())
+ name = name.lstrip('python-')
+ return name
+
+
+def find_setup(package_name, version, archive):
+ """
+ Search for setup.py file in an archive and returns True if found
+ Used for finding the correct path to the setup.py
+
+ Keyword arguments:
+ package_name -- base name of the package to search (e.g. Flask)
+ version -- version of the package to search (e.g. 0.8.1)
+ archive -- tar archive to search in
+ """
+ try:
+ archive.getmember('{name}-{version}/setup.py'.format(
+ name=package_name,
+ version=version))
+ except KeyError:
+ return False
+ else:
+ return True
+
+
+# monkey patch
+import setuptools
+setuptools.setup = setup_decorator(setuptools.setup, 'setuptools')
+import distutils
+distutils.core.setup = setup_decorator(setuptools.setup, 'distutils')
+
+if __name__ == "__main__":
+
+ # Building the parser
+ parser = argparse.ArgumentParser(
+ description=("Creates buildroot packages from the metadata of "
+ "an existing pypi(pip) packages and include it "
+ "in menuconfig"))
+ parser.add_argument("packages",
+ help="list of packages to be made",
+ nargs='+')
+ parser.add_argument("-o", "--output",
+ help="""
+ Output directory for packages
+ """,
+ default='.')
+
+ args = parser.parse_args()
+ packages = list(set(args.packages))
+
+ # tmp_path is where we'll extract the files later
+ tmp_prefix = '-python-package-generator'
+ # dl_dir is supposed to be your buildroot dl dir
+ pkg_folder = args.output
+ tmp_path = tempfile.mkdtemp(prefix=tmp_prefix)
+
+ packages_local_names = map(pkg_new_name, packages)
+ print(
+ 'Character . is forbidden.',
+ 'Generator will use only alphanumeric characters (including _ and -)',
+ sep='\n')
+ for index, real_pkg_name in enumerate(packages):
+ # First we download the package
+ # Most of the info we need can only be found inside the package
+ pkg_name = packages_local_names[index]
+ print('Package:', pkg_name)
+ print('Fetching package', real_pkg_name)
+ url = 'https://pypi.python.org/pypi/{pkg}/json'.format(
+ pkg=real_pkg_name)
+ print('URL:', url)
+ try:
+ pkg_json = urllib2.urlopen(url).read().decode()
+ except (urllib2.HTTPError, urllib2.URLError) as error:
+ print('ERROR:', error.getcode(), error.msg, file=sys.stderr)
+ print('ERROR: Could not find package {pkg}.\n'
+ 'Check syntax inside the python package index:\n'
+ 'https://pypi.python.org/pypi/ '.format(pkg=real_pkg_name))
+ continue
+
+ pkg_dir = ''.join([pkg_folder, '/python-', pkg_name])
+
+ package = json.loads(pkg_json)
+ used_url = ''
+ try:
+ targz = package['urls'][0]['filename']
+ except IndexError:
+ print(
+ 'Non conventional package, ',
+ 'please check manually after creation')
+ download_url = package['info']['download_url']
+ try:
+ download = urllib2.urlopen(download_url)
+ except urllib2.HTTPError:
+ pass
+ else:
+ used_url = {'url': download_url}
+ as_file = StringIO.StringIO(download.read())
+ md5_sum = hashlib.md5(as_file.read()).hexdigest()
+ used_url['md5_digest'] = md5_sum
+ as_file.seek(0)
+ print(magic.from_buffer(as_file.read()))
+ as_file.seek(0)
+ extension = 'tar.gz'
+ if 'gzip' not in magic.from_buffer(as_file.read()):
+ extension = 'tar.bz2'
+ targz = '{name}-{version}.{extension}'.format(
+ package['info']['name'], package['info']['version'],
+ extension)
+ as_file.seek(0)
+ used_url['filename'] = targz
+
+ print(
+ 'Downloading package {pkg}...'.format(pkg=package['info']['name']))
+ for download_url in package['urls']:
+ try:
+ download = urllib2.urlopen(download_url['url'])
+ except urllib2.HTTPError:
+ pass
+ else:
+ used_url = download_url
+ as_file = StringIO.StringIO(download.read())
+ md5_sum = hashlib.md5(as_file.read()).hexdigest()
+ if md5_sum == download_url['md5_digest']:
+ break
+ targz = used_url['filename']
+
+ if not download:
+ print('Error downloading package :', pkg_name)
+ continue
+
+ # extract the tarball
+ as_file.seek(0)
+ as_tarfile = tarfile.open(fileobj=as_file)
+ tmp_pkg = '/'.join([tmp_path, pkg_name])
+ try:
+ os.makedirs(tmp_pkg)
+ except OSError as exception:
+ if exception.errno != errno.EEXIST:
+ print("ERROR: ", exception.message, file=sys.stderr)
+ continue
+ print('WARNING:', exception.message, file=sys.stderr)
+ print('Removing {pkg}...'.format(pkg=tmp_pkg))
+ shutil.rmtree(tmp_pkg)
+ os.makedirs(tmp_pkg)
+ tar_folder_names = [real_pkg_name.capitalize(),
+ real_pkg_name.lower(),
+ package['info']['name']]
+ version = package['info']['version']
+ try:
+ tar_folder = next(folder for folder in tar_folder_names
+ if find_setup(folder, version, as_tarfile))
+ except StopIteration:
+ print('ERROR: Could not extract package %s' %
+ real_pkg_name,
+ file=sys.stderr)
+ continue
+ as_tarfile.extractall(tmp_pkg)
+ as_tarfile.close()
+ as_file.close()
+ tmp_extract = '{folder}/{name}-{version}'.format(
+ folder=tmp_pkg,
+ name=tar_folder,
+ version=package['info']['version'])
+
+ # Loading the package install info from the package
+ sys.path.append(tmp_extract)
+ import setup
+ setup = reload(setup)
+ sys.path.remove(tmp_extract)
+
+ pkg_req = None
+ # Package requierement are an argument of the setup function
+ if 'install_requires' in setup_info(tar_folder):
+ pkg_req = setup_info(tar_folder)['install_requires']
+ pkg_req = [re.sub('([\w-]+)[><=]*.*', r'\1', req).lower()
+ for req in pkg_req]
+ pkg_req = map(pkg_new_name, pkg_req)
+ req_not_found = [
+ pkg for pkg in pkg_req
+ if 'python-{name}'.format(name=pkg)
+ not in os.listdir(pkg_folder)
+ ]
+ req_not_found = [pkg for pkg in req_not_found
+ if pkg not in packages]
+ if (req_not_found) != 0:
+ print(
+ 'Error: could not find packages \'{packages}\'',
+ 'required by {current_package}'.format(
+ packages=", ".join(req_not_found),
+ current_package=pkg_name))
+ # We could stop here
+ # or ask the user if he still wants to continue
+
+ # Buildroot python packages require 3 files
+ # The first is the mk file
+ # See:
+ # http://buildroot.uclibc.org/downloads/manual/manual.html
+ pkg_mk = 'python-{name}.mk'.format(name=pkg_name)
+ path_to_mk = '/'.join([pkg_dir, pkg_mk])
+ print('Creating {file}...'.format(file=path_to_mk))
+ print('Checking if package {name} already exists...'.format(
+ name=pkg_dir))
+ try:
+ os.makedirs(pkg_dir)
+ except OSError as exception:
+ if exception.errno != errno.EEXIST:
+ print("ERROR: ", exception.message, file=sys.stderr)
+ continue
+ print('Error: Package {name} already exists'.format(name=pkg_dir))
+ del_pkg = raw_input(
+ 'Do you want to delete existing package ? [y/N]')
+ if del_pkg.lower() == 'y':
+ shutil.rmtree(pkg_dir)
+ os.makedirs(pkg_dir)
+ else:
+ continue
+ with open(path_to_mk, 'w') as mk_file:
+ # header
+ header = ['#' * 80 + '\n']
+ header.append('#\n')
+ header.append('# {name}\n'.format(name=pkg_dir))
+ header.append('#\n')
+ header.append('#' * 80 + '\n')
+ header.append('\n')
+ mk_file.writelines(header)
+
+ version_line = 'PYTHON_{name}_VERSION = {version}\n'.format(
+ name=pkg_name.upper(),
+ version=package['info']['version'])
+ mk_file.write(version_line)
+ targz = targz.replace(
+ package['info']['version'],
+ '$(PYTHON_{name}_VERSION)'.format(name=pkg_name.upper()))
+ targz_line = 'PYTHON_{name}_SOURCE = {filename}\n'.format(
+ name=pkg_name.upper(),
+ filename=targz)
+ mk_file.write(targz_line)
+
+ site_line = ('PYTHON_{name}_SITE = {url}\n'.format(
+ name=pkg_name.upper(),
+ url=used_url['url'].replace(used_url['filename'], '')))
+ if 'sourceforge' in site_line:
+ site_line = ('PYTHON_{name}_SITE = {url}\n'.format(
+ name=pkg_name.upper(),
+ url=used_url['url']))
+
+ mk_file.write(site_line)
+
+ # There are two things you can use to make an installer
+ # for a python package: distutils or setuptools
+ # distutils comes with python but does not support dependancies.
+ # distutils is mostly still there for backward support.
+ # setuptools is what smart people use,
+ # but it is not shipped with python :(
+
+ # setuptools.setup calls distutils.core.setup
+ # We use the monkey patch with a tag to know which one is used.
+ setup_type_line = 'PYTHON_{name}_SETUP_TYPE = {method}\n'.format(
+ name=pkg_name.upper(),
+ method=setup_info(tar_folder)['method'])
+ mk_file.write(setup_type_line)
+
+ license_line = 'PYTHON_{name}_LICENSE = {license}\n'.format(
+ name=pkg_name.upper(),
+ license=package['info']['license'])
+ mk_file.write(license_line)
+ print('WARNING: License has been set to "{license}",'
+ ' please change it manually if necessary'.format(
+ license=package['info']['license']))
+ filenames = ['LICENSE', 'LICENSE.TXT']
+ license_files = list(find_file_upper_case(filenames, tmp_extract))
+ license_files = [license.replace(tmp_extract, '')[1:]
+ for license in license_files]
+ if len(license_files) > 1:
+ print('More than one file found for license: ')
+ for index, item in enumerate(license_files):
+ print('\t{index})'.format(index), item)
+ license_choices = raw_input(
+ 'specify file numbers separated by spaces(default 0): ')
+ license_choices = [int(choice)
+ for choice in license_choices.split(' ')
+ if choice.isdigit() and int(choice) in
+ range(len(license_files))]
+ if len(license_choices) == 0:
+ license_choices = [0]
+ license_files = [file
+ for index, file in enumerate(license_files)
+ if index in license_choices]
+ elif len(license_files) == 0:
+ print('WARNING: No license file found,'
+ ' please specify it manually afterward')
+
+ license_file_line = ('PYTHON_{name}_LICEiNSE_FILES ='
+ ' {files}\n'.format(
+ name=pkg_name.upper(),
+ files=' '.join(license_files)))
+ mk_file.write(license_file_line)
+
+ if pkg_req:
+ python_pkg_req = ['python-{name}'.format(name=pkg)
+ for pkg in pkg_req]
+ dependencies_line = ('PYTHON_{name}_DEPENDENCIES ='
+ ' {reqs}\n'.format(
+ name=pkg_name.upper(),
+ reqs=' '.join(python_pkg_req)))
+ mk_file.write(dependencies_line)
+
+ mk_file.write('\n')
+ mk_file.write('$(eval $(python-package))')
+
+ # The second file we make is the hash file
+ # It consists of hashes of the package tarball
+ # http://buildroot.uclibc.org/downloads/manual/manual.html#adding-packages-hash
+ pkg_hash = 'python-{name}.hash'.format(name=pkg_name)
+ path_to_hash = '/'.join([pkg_dir, pkg_hash])
+ print('Creating {filename}...'.format(filename=path_to_hash))
+ with open(path_to_hash, 'w') as hash_file:
+ commented_line = '# md5 from {url}\n'.format(url=url)
+ hash_file.write(commented_line)
+
+ hash_line = 'md5\t{digest} {filename}\n'.format(
+ digest=used_url['md5_digest'],
+ filename=used_url['filename'])
+ hash_file.write(hash_line)
+
+ # The Config.in is the last file we create
+ # It is used by buildroot's menuconfig, gconfig, xconfig or nconfig
+ # it is used to displayspackage info and to select requirements
+ # http://buildroot.uclibc.org/downloads/manual/manual.html#_literal_config_in_literal_file
+ path_to_config = '/'.join([pkg_dir, 'Config.in'])
+ print('Creating {file}...'.format(file=path_to_config))
+ with open(path_to_config, 'w') as config_file:
+ config_line = 'config BR2_PACKAGE_PYTHON_{name}\n'.format(
+ name=pkg_name.upper())
+ config_file.write(config_line)
+ python_line = '\tdepends on BR2_PACKAGE_PYTHON\n'
+ config_file.write(python_line)
+
+ bool_line = '\tbool "python-{name}"\n'.format(name=pkg_name)
+ config_file.write(bool_line)
+ if pkg_req:
+ for dep in pkg_req:
+ dep_line = '\tselect BR2_PACKAGE_PYTHON_{req}\n'.format(
+ req=dep.upper())
+ config_file.write(dep_line)
+
+ config_file.write('\thelp\n')
+
+ help_lines = package['info']['summary'].split('\n')
+ help_lines.append('')
+ help_lines.append(package['info']['home_page'])
+ help_lines = ['\t {line}\n'.format(line=line)
+ for line in help_lines]
+ config_file.writelines(help_lines)
--
2.4.2
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [Buildroot] [PATCH 2/3] python-robotframework: New package
2015-06-01 14:56 [Buildroot] [PATCH 0/3] [RFC] python-package-generator Denis THULIN
2015-06-01 14:56 ` [Buildroot] [PATCH 1/3] [RFC] python-package-generator: new utility Denis THULIN
@ 2015-06-01 14:56 ` Denis THULIN
2015-06-12 16:09 ` Vincent Stehlé
2015-06-01 14:56 ` [Buildroot] [PATCH 3/3] python-magic: new package Denis THULIN
2 siblings, 1 reply; 8+ messages in thread
From: Denis THULIN @ 2015-06-01 14:56 UTC (permalink / raw)
To: buildroot
A generic test automation framework written in python.
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
v0:
- Generated using python-package-generator.py
- Dependencies on python-zlib and python-pyexpat were added manually
- Modification of packages/Config.in done manually.
---
package/Config.in | 1 +
package/python-robotframework/Config.in | 9 +++++++++
package/python-robotframework/python-robotframework.hash | 2 ++
package/python-robotframework/python-robotframework.mk | 14 ++++++++++++++
4 files changed, 26 insertions(+)
create mode 100644 package/python-robotframework/Config.in
create mode 100644 package/python-robotframework/python-robotframework.hash
create mode 100644 package/python-robotframework/python-robotframework.mk
diff --git a/package/Config.in b/package/Config.in
index e0c2e2a..7b015a3 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -633,6 +633,7 @@ menu "external python modules"
source "package/python-pyxml/Config.in"
source "package/python-pyzmq/Config.in"
source "package/python-requests/Config.in"
+ source "package/python-robotframework/Config.in"
source "package/python-rtslib-fb/Config.in"
source "package/python-serial/Config.in"
source "package/python-setuptools/Config.in"
diff --git a/package/python-robotframework/Config.in b/package/python-robotframework/Config.in
new file mode 100644
index 0000000..a3d5ea8
--- /dev/null
+++ b/package/python-robotframework/Config.in
@@ -0,0 +1,9 @@
+config BR2_PACKAGE_PYTHON_ROBOTFRAMEWORK
+ depends on BR2_PACKAGE_PYTHON
+ select BR2_PACKAGE_PYTHON_ZLIB
+ select BR2_PACKAGE_PYTHON_PYEXPAT
+ bool "python-robotframework"
+ help
+ A generic test automation framework
+
+ http://robotframework.org
diff --git a/package/python-robotframework/python-robotframework.hash b/package/python-robotframework/python-robotframework.hash
new file mode 100644
index 0000000..8aac9f4
--- /dev/null
+++ b/package/python-robotframework/python-robotframework.hash
@@ -0,0 +1,2 @@
+# md5 from https://pypi.python.org/pypi/robotframework/json
+md5 018ccc926223131325bc413ba02b2ffb robotframework-2.9a1.tar.gz
diff --git a/package/python-robotframework/python-robotframework.mk b/package/python-robotframework/python-robotframework.mk
new file mode 100644
index 0000000..6fc7619
--- /dev/null
+++ b/package/python-robotframework/python-robotframework.mk
@@ -0,0 +1,14 @@
+################################################################################
+#
+# package/python-robotframework
+#
+################################################################################
+
+PYTHON_ROBOTFRAMEWORK_VERSION = 2.9a1
+PYTHON_ROBOTFRAMEWORK_SOURCE = robotframework-$(PYTHON_ROBOTFRAMEWORK_VERSION).tar.gz
+PYTHON_ROBOTFRAMEWORK_SITE = https://pypi.python.org/packages/source/r/robotframework/
+PYTHON_ROBOTFRAMEWORK_SETUP_TYPE = distutils
+PYTHON_ROBOTFRAMEWORK_LICENSE = Apache License 2.0
+PYTHON_ROBOTFRAMEWORK_LICEiNSE_FILES = LICENSE.txt
+
+$(eval $(python-package))
\ No newline at end of file
--
2.4.2
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [Buildroot] [PATCH 3/3] python-magic: new package
2015-06-01 14:56 [Buildroot] [PATCH 0/3] [RFC] python-package-generator Denis THULIN
2015-06-01 14:56 ` [Buildroot] [PATCH 1/3] [RFC] python-package-generator: new utility Denis THULIN
2015-06-01 14:56 ` [Buildroot] [PATCH 2/3] python-robotframework: New package Denis THULIN
@ 2015-06-01 14:56 ` Denis THULIN
2 siblings, 0 replies; 8+ messages in thread
From: Denis THULIN @ 2015-06-01 14:56 UTC (permalink / raw)
To: buildroot
A file type identification for python using libmagic
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
v0:
- Generated using python-package-generator.py
- Modification on package/Config.in was done manually
---
package/Config.in | 1 +
package/python-magic/Config.in | 7 +++++++
package/python-magic/python-magic.hash | 2 ++
package/python-magic/python-magic.mk | 14 ++++++++++++++
4 files changed, 24 insertions(+)
create mode 100644 package/python-magic/Config.in
create mode 100644 package/python-magic/python-magic.hash
create mode 100644 package/python-magic/python-magic.mk
diff --git a/package/Config.in b/package/Config.in
index 7b015a3..c70ca4a 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -601,6 +601,7 @@ menu "external python modules"
source "package/python-keyring/Config.in"
source "package/python-libconfig/Config.in"
source "package/python-lxml/Config.in"
+ source "package/python-magic/Config.in"
source "package/python-mako/Config.in"
source "package/python-mad/Config.in"
source "package/python-markdown/Config.in"
diff --git a/package/python-magic/Config.in b/package/python-magic/Config.in
new file mode 100644
index 0000000..e6c06b9
--- /dev/null
+++ b/package/python-magic/Config.in
@@ -0,0 +1,7 @@
+config BR2_PACKAGE_PYTHON_MAGIC
+ depends on BR2_PACKAGE_PYTHON
+ bool "python-magic"
+ help
+ File type identification using libmagic
+
+ http://github.com/ahupp/python-magic
diff --git a/package/python-magic/python-magic.hash b/package/python-magic/python-magic.hash
new file mode 100644
index 0000000..65d0ad0
--- /dev/null
+++ b/package/python-magic/python-magic.hash
@@ -0,0 +1,2 @@
+# md5 from https://pypi.python.org/pypi/python-magic/json
+md5 07e7a0fea78dd81ed609414c3484df58 python-magic-0.4.6.tar.gz
diff --git a/package/python-magic/python-magic.mk b/package/python-magic/python-magic.mk
new file mode 100644
index 0000000..91d1707
--- /dev/null
+++ b/package/python-magic/python-magic.mk
@@ -0,0 +1,14 @@
+################################################################################
+#
+# package/python-magic
+#
+################################################################################
+
+PYTHON_MAGIC_VERSION = 0.4.6
+PYTHON_MAGIC_SOURCE = python-magic-$(PYTHON_MAGIC_VERSION).tar.gz
+PYTHON_MAGIC_SITE = https://pypi.python.org/packages/source/p/python-magic/
+PYTHON_MAGIC_SETUP_TYPE = setuptools
+PYTHON_MAGIC_LICENSE = PSF
+PYTHON_MAGIC_LICEiNSE_FILES =
+
+$(eval $(python-package))
\ No newline at end of file
--
2.4.2
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [Buildroot] [PATCH 1/3] [RFC] python-package-generator: new utility
2015-06-01 14:56 ` [Buildroot] [PATCH 1/3] [RFC] python-package-generator: new utility Denis THULIN
@ 2015-06-01 22:37 ` Arnout Vandecappelle
2015-06-02 16:23 ` Denis Thulin
2015-06-10 7:56 ` Thomas Petazzoni
1 sibling, 1 reply; 8+ messages in thread
From: Arnout Vandecappelle @ 2015-06-01 22:37 UTC (permalink / raw)
To: buildroot
Hi Denis,
Thanks for this contribution! A big patch like this you can expect some
comments of course...
I didn't have time to look at all details, but here is some initial input.
On 06/01/15 16:56, Denis THULIN wrote:
> This patch adds package python-package-generator.py
> ---
> v0: initial commit
> - python-pacakage-generator.py is an utility for automatically generating
> python packages using metadata from the python package index:
> https://pypi.python.org
>
> I did not know where to put the script so I put it in support/scripts.
> I have updated the python-package section of the manual as well.
>
> Please bear in mind that python-package-generator.py does not add the packages
> to your buildroot project and you need to do it manually.
>
> Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
SoB should go before the ---
> ---
> docs/manual/adding-packages-python.txt | 36 +++
> support/scripts/python-package-generator.py | 435 ++++++++++++++++++++++++++++
> 2 files changed, 471 insertions(+)
> create mode 100755 support/scripts/python-package-generator.py
>
> diff --git a/docs/manual/adding-packages-python.txt b/docs/manual/adding-packages-python.txt
> index f81d625..6f608ed 100644
> --- a/docs/manual/adding-packages-python.txt
> +++ b/docs/manual/adding-packages-python.txt
> @@ -7,6 +7,42 @@ This infrastructure applies to Python packages that use the standard
> Python setuptools mechanism as their build system, generally
> recognizable by the usage of a +setup.py+ script.
>
> +[[python-package-generator]]
> +
> +==== generating a +python-package+ from a pypi repository
> +
> +You may want to use the +python-package-generator.py+ located in
> ++support/script+ to generate a package from an existing pypi(pip) package.
Drop the (pip) - pip is a package installer, we don't use it.
> +
> +you can find the list of existing pypi package here: (https://pypi.python.org).
You can find a list of available packages in https://pypi.python.org[the Python
package index].
> +
> +Please keep in mind that you most likely need
> +to manually check the package for any mistakes
> +as there are things that cannot be guessed by the generator (e.g.
> +dependencies on any of the python core modules
> +such as BR2_PACKAGE_PYTHON_ZLIB).
Please re-wrap this paragraph.
> +
> +When at the root of your buildroot directory just do :
> +
> +-----------------------
> +./support/script/python-package-generator.py foo bar -o package
> +-----------------------
> +
> +This will generate packages +python-foo+ and +python-bar+ in the package
> +folder if they exist on https://pypi.python.org.
> +
> +You will need to manually write the path to the package inside
> +the +package/Config.in+ file:
> +
> +Find the +external python modules+ menu and insert your package inside.
> +Keep in mind that the items inside a menu should be in alphabetical order.
Just a single paragraph:
You will need to manually add the package to +package/Config.in+. Find the ...
> +
> +Option +-h+ wil list the options available
> +
> +-----------------------
> +./support/script/python-package-generator.py -h
> +-----------------------
> +
> [[python-package-tutorial]]
>
> ==== +python-package+ tutorial
> diff --git a/support/scripts/python-package-generator.py b/support/scripts/python-package-generator.py
> new file mode 100755
> index 0000000..4f5e884
> --- /dev/null
> +++ b/support/scripts/python-package-generator.py
> @@ -0,0 +1,435 @@
> +#!/usr/bin/python2
Any chance of making the script 2+3 compatible?
> +"""
> + Utility for building buildroot packages for existing pypi packages
> +
> + Any package built by brpy-generator should be manually checked for errors.
python-package-generator
> +"""
> +from __future__ import print_function
> +import argparse
> +import json
> +import urllib2
> +import sys
> +import os
> +import shutil
> +import StringIO
> +import tarfile
> +import errno
> +import hashlib
> +import re
> +import magic
> +import tempfile
> +from functools import wraps
> +
> +# TODO: Create a real module instead of a 320 line script
No need for that.
> +
> +# private global
> +_calls = {}
> +
> +
> +def setup_info(pkg_name):
> + """Get a package info from _calls
> +
> + Keyword arguments:
> + pkg_name -- the name of the package
> + """
> + return _calls[pkg_name]
> +
> +
> +def setup_decorator(func, method):
> + """
> + Decorator for distutils.core.setup and setuptools.setup.
> + Puts the args of setup as a dict inside global private dict _calls.
> + Add key 'method' which should be either 'setuptools' or 'distutils'.
> +
> + Keyword arguments:
> + func -- either setuptools.setup or distutils.core.setup
> + method -- either 'setuptools' or 'distutils'
> + """
> +
> + @wraps(func)
> + def closure(*args, **kwargs):
> + _calls[kwargs['name']] = kwargs
> + _calls[kwargs['name']]['method'] = method
> +
> + return closure
> +
> +
> +def find_file_upper_case(filenames, path='./'):
> + """
> + List generator:
> + Recursively find files that matches one of the specified filenames.
> + Returns absolute path
> +
> + Keyword arguments:
> + filenames -- List of filenames to be found
> + path -- Path to the directory to search
> + """
> + for root, dirs, files in os.walk(path):
> + for file in files:
> + if file.upper() in filenames:
> + yield (os.path.join(root, file))
> +
> +
> +def pkg_new_name(pkg_name):
Bit a weird name for the function. Perhaps pkg_buildroot_name?
> + """
> + Returns name to avoid troublesome characters.
> + Remove all non alphanumeric characters except -
> + Also lowers the name
> +
> + Keyword arguments:
> + pkg_name -- String to rename
> + """
> + name = re.sub('[^\w-]', '', pkg_name.lower())
> + name = name.lstrip('python-')
> + return name
> +
> +
> +def find_setup(package_name, version, archive):
> + """
> + Search for setup.py file in an archive and returns True if found
> + Used for finding the correct path to the setup.py
> +
> + Keyword arguments:
> + package_name -- base name of the package to search (e.g. Flask)
> + version -- version of the package to search (e.g. 0.8.1)
> + archive -- tar archive to search in
> + """
> + try:
> + archive.getmember('{name}-{version}/setup.py'.format(
> + name=package_name,
> + version=version))
> + except KeyError:
> + return False
> + else:
> + return True
> +
> +
> +# monkey patch
> +import setuptools
> +setuptools.setup = setup_decorator(setuptools.setup, 'setuptools')
> +import distutils
> +distutils.core.setup = setup_decorator(setuptools.setup, 'distutils')
> +
> +if __name__ == "__main__":
> +
> + # Building the parser
> + parser = argparse.ArgumentParser(
> + description=("Creates buildroot packages from the metadata of "
> + "an existing pypi(pip) packages and include it "
> + "in menuconfig"))
Spurious () around the string.
> + parser.add_argument("packages",
> + help="list of packages to be made",
> + nargs='+')
> + parser.add_argument("-o", "--output",
> + help="""
> + Output directory for packages
> + """,
> + default='.')
> +
> + args = parser.parse_args()
> + packages = list(set(args.packages))
> +
> + # tmp_path is where we'll extract the files later
> + tmp_prefix = '-python-package-generator'
> + # dl_dir is supposed to be your buildroot dl dir
What does this comment mean?
> + pkg_folder = args.output
> + tmp_path = tempfile.mkdtemp(prefix=tmp_prefix)
> +
> + packages_local_names = map(pkg_new_name, packages)
Remove this.
> + print(
> + 'Character . is forbidden.',
> + 'Generator will use only alphanumeric characters (including _ and -)',
> + sep='\n')
Why print this?
> + for index, real_pkg_name in enumerate(packages):
> + # First we download the package
> + # Most of the info we need can only be found inside the package
> + pkg_name = packages_local_names[index]
pkg_name = pkg_new_name(real_pkg_name)
and you no longer need enumerate and index.
> + print('Package:', pkg_name)
> + print('Fetching package', real_pkg_name)
> + url = 'https://pypi.python.org/pypi/{pkg}/json'.format(
> + pkg=real_pkg_name)
> + print('URL:', url)
> + try:
> + pkg_json = urllib2.urlopen(url).read().decode()
> + except (urllib2.HTTPError, urllib2.URLError) as error:
> + print('ERROR:', error.getcode(), error.msg, file=sys.stderr)
> + print('ERROR: Could not find package {pkg}.\n'
> + 'Check syntax inside the python package index:\n'
> + 'https://pypi.python.org/pypi/ '.format(pkg=real_pkg_name))
> + continue
> +
> + pkg_dir = ''.join([pkg_folder, '/python-', pkg_name])
IMHO using + is clearer.
> +
> + package = json.loads(pkg_json)
> + used_url = ''
> + try:
> + targz = package['urls'][0]['filename']
> + except IndexError:
> + print(
> + 'Non conventional package, ',
> + 'please check manually after creation')
> + download_url = package['info']['download_url']
Is that guaranteed to exist?
> + try:
> + download = urllib2.urlopen(download_url)
> + except urllib2.HTTPError:
> + pass
> + else:
> + used_url = {'url': download_url}
> + as_file = StringIO.StringIO(download.read())
> + md5_sum = hashlib.md5(as_file.read()).hexdigest()
Use sha256 instead of md5.
> + used_url['md5_digest'] = md5_sum
> + as_file.seek(0)
> + print(magic.from_buffer(as_file.read()))
> + as_file.seek(0)
> + extension = 'tar.gz'
> + if 'gzip' not in magic.from_buffer(as_file.read()):
> + extension = 'tar.bz2'
> + targz = '{name}-{version}.{extension}'.format(
> + package['info']['name'], package['info']['version'],
> + extension)
> + as_file.seek(0)
> + used_url['filename'] = targz
> +
> + print(
> + 'Downloading package {pkg}...'.format(pkg=package['info']['name']))
> + for download_url in package['urls']:
> + try:
> + download = urllib2.urlopen(download_url['url'])
> + except urllib2.HTTPError:
> + pass
> + else:
> + used_url = download_url
> + as_file = StringIO.StringIO(download.read())
> + md5_sum = hashlib.md5(as_file.read()).hexdigest()
> + if md5_sum == download_url['md5_digest']:
> + break
> + targz = used_url['filename']
> +
> + if not download:
> + print('Error downloading package :', pkg_name)
> + continue
> +
> + # extract the tarball
> + as_file.seek(0)
> + as_tarfile = tarfile.open(fileobj=as_file)
> + tmp_pkg = '/'.join([tmp_path, pkg_name])
> + try:
> + os.makedirs(tmp_pkg)
> + except OSError as exception:
> + if exception.errno != errno.EEXIST:
> + print("ERROR: ", exception.message, file=sys.stderr)
> + continue
> + print('WARNING:', exception.message, file=sys.stderr)
> + print('Removing {pkg}...'.format(pkg=tmp_pkg))
> + shutil.rmtree(tmp_pkg)
> + os.makedirs(tmp_pkg)
> + tar_folder_names = [real_pkg_name.capitalize(),
> + real_pkg_name.lower(),
> + package['info']['name']]
> + version = package['info']['version']
> + try:
> + tar_folder = next(folder for folder in tar_folder_names
> + if find_setup(folder, version, as_tarfile))
> + except StopIteration:
> + print('ERROR: Could not extract package %s' %
> + real_pkg_name,
> + file=sys.stderr)
> + continue
This looks really opaque to me. Can't it be expressed by a simple loop?
> + as_tarfile.extractall(tmp_pkg)
> + as_tarfile.close()
> + as_file.close()
> + tmp_extract = '{folder}/{name}-{version}'.format(
> + folder=tmp_pkg,
> + name=tar_folder,
> + version=package['info']['version'])
> +
> + # Loading the package install info from the package
> + sys.path.append(tmp_extract)
> + import setup
> + setup = reload(setup)
> + sys.path.remove(tmp_extract)
> +
> + pkg_req = None
> + # Package requierement are an argument of the setup function
> + if 'install_requires' in setup_info(tar_folder):
> + pkg_req = setup_info(tar_folder)['install_requires']
> + pkg_req = [re.sub('([\w-]+)[><=]*.*', r'\1', req).lower()
> + for req in pkg_req]
> + pkg_req = map(pkg_new_name, pkg_req)
> + req_not_found = [
> + pkg for pkg in pkg_req
> + if 'python-{name}'.format(name=pkg)
> + not in os.listdir(pkg_folder)
> + ]
> + req_not_found = [pkg for pkg in req_not_found
> + if pkg not in packages]
> + if (req_not_found) != 0:
> + print(
> + 'Error: could not find packages \'{packages}\'',
> + 'required by {current_package}'.format(
> + packages=", ".join(req_not_found),
> + current_package=pkg_name))
> + # We could stop here
> + # or ask the user if he still wants to continue
> +
> + # Buildroot python packages require 3 files
> + # The first is the mk file
> + # See:
> + # http://buildroot.uclibc.org/downloads/manual/manual.html
> + pkg_mk = 'python-{name}.mk'.format(name=pkg_name)
> + path_to_mk = '/'.join([pkg_dir, pkg_mk])
> + print('Creating {file}...'.format(file=path_to_mk))
> + print('Checking if package {name} already exists...'.format(
> + name=pkg_dir))
> + try:
> + os.makedirs(pkg_dir)
> + except OSError as exception:
> + if exception.errno != errno.EEXIST:
> + print("ERROR: ", exception.message, file=sys.stderr)
> + continue
> + print('Error: Package {name} already exists'.format(name=pkg_dir))
> + del_pkg = raw_input(
> + 'Do you want to delete existing package ? [y/N]')
> + if del_pkg.lower() == 'y':
> + shutil.rmtree(pkg_dir)
> + os.makedirs(pkg_dir)
> + else:
> + continue
> + with open(path_to_mk, 'w') as mk_file:
> + # header
> + header = ['#' * 80 + '\n']
> + header.append('#\n')
> + header.append('# {name}\n'.format(name=pkg_dir))
> + header.append('#\n')
> + header.append('#' * 80 + '\n')
> + header.append('\n')
> + mk_file.writelines(header)
> +
> + version_line = 'PYTHON_{name}_VERSION = {version}\n'.format(
> + name=pkg_name.upper(),
> + version=package['info']['version'])
> + mk_file.write(version_line)
> + targz = targz.replace(
> + package['info']['version'],
> + '$(PYTHON_{name}_VERSION)'.format(name=pkg_name.upper()))
> + targz_line = 'PYTHON_{name}_SOURCE = {filename}\n'.format(
> + name=pkg_name.upper(),
> + filename=targz)
> + mk_file.write(targz_line)
> +
> + site_line = ('PYTHON_{name}_SITE = {url}\n'.format(
> + name=pkg_name.upper(),
> + url=used_url['url'].replace(used_url['filename'], '')))
> + if 'sourceforge' in site_line:
> + site_line = ('PYTHON_{name}_SITE = {url}\n'.format(
> + name=pkg_name.upper(),
> + url=used_url['url']))
> +
> + mk_file.write(site_line)
> +
> + # There are two things you can use to make an installer
> + # for a python package: distutils or setuptools
> + # distutils comes with python but does not support dependancies.
> + # distutils is mostly still there for backward support.
> + # setuptools is what smart people use,
> + # but it is not shipped with python :(
> +
> + # setuptools.setup calls distutils.core.setup
> + # We use the monkey patch with a tag to know which one is used.
> + setup_type_line = 'PYTHON_{name}_SETUP_TYPE = {method}\n'.format(
> + name=pkg_name.upper(),
> + method=setup_info(tar_folder)['method'])
> + mk_file.write(setup_type_line)
> +
> + license_line = 'PYTHON_{name}_LICENSE = {license}\n'.format(
> + name=pkg_name.upper(),
> + license=package['info']['license'])
> + mk_file.write(license_line)
> + print('WARNING: License has been set to "{license}",'
> + ' please change it manually if necessary'.format(
> + license=package['info']['license']))
> + filenames = ['LICENSE', 'LICENSE.TXT']
> + license_files = list(find_file_upper_case(filenames, tmp_extract))
> + license_files = [license.replace(tmp_extract, '')[1:]
> + for license in license_files]
> + if len(license_files) > 1:
It's OK to have more than one license file, so I'd keep both of them.
> + print('More than one file found for license: ')
> + for index, item in enumerate(license_files):
> + print('\t{index})'.format(index), item)
> + license_choices = raw_input(
> + 'specify file numbers separated by spaces(default 0): ')
> + license_choices = [int(choice)
> + for choice in license_choices.split(' ')
> + if choice.isdigit() and int(choice) in
> + range(len(license_files))]
> + if len(license_choices) == 0:
> + license_choices = [0]
> + license_files = [file
> + for index, file in enumerate(license_files)
> + if index in license_choices]
> + elif len(license_files) == 0:
> + print('WARNING: No license file found,'
> + ' please specify it manually afterward')
> +
> + license_file_line = ('PYTHON_{name}_LICEiNSE_FILES ='
LICENSE (spurious i)
> + ' {files}\n'.format(
> + name=pkg_name.upper(),
> + files=' '.join(license_files)))
> + mk_file.write(license_file_line)
> +
> + if pkg_req:
> + python_pkg_req = ['python-{name}'.format(name=pkg)
> + for pkg in pkg_req]
> + dependencies_line = ('PYTHON_{name}_DEPENDENCIES ='
> + ' {reqs}\n'.format(
> + name=pkg_name.upper(),
> + reqs=' '.join(python_pkg_req)))
> + mk_file.write(dependencies_line)
> +
> + mk_file.write('\n')
> + mk_file.write('$(eval $(python-package))')
Missing newline at the end.
That's as far as I got :-)
Regards,
Arnout
> +
> + # The second file we make is the hash file
> + # It consists of hashes of the package tarball
> + # http://buildroot.uclibc.org/downloads/manual/manual.html#adding-packages-hash
> + pkg_hash = 'python-{name}.hash'.format(name=pkg_name)
> + path_to_hash = '/'.join([pkg_dir, pkg_hash])
> + print('Creating {filename}...'.format(filename=path_to_hash))
> + with open(path_to_hash, 'w') as hash_file:
> + commented_line = '# md5 from {url}\n'.format(url=url)
> + hash_file.write(commented_line)
> +
> + hash_line = 'md5\t{digest} {filename}\n'.format(
> + digest=used_url['md5_digest'],
> + filename=used_url['filename'])
> + hash_file.write(hash_line)
> +
> + # The Config.in is the last file we create
> + # It is used by buildroot's menuconfig, gconfig, xconfig or nconfig
> + # it is used to displayspackage info and to select requirements
> + # http://buildroot.uclibc.org/downloads/manual/manual.html#_literal_config_in_literal_file
> + path_to_config = '/'.join([pkg_dir, 'Config.in'])
> + print('Creating {file}...'.format(file=path_to_config))
> + with open(path_to_config, 'w') as config_file:
> + config_line = 'config BR2_PACKAGE_PYTHON_{name}\n'.format(
> + name=pkg_name.upper())
> + config_file.write(config_line)
> + python_line = '\tdepends on BR2_PACKAGE_PYTHON\n'
> + config_file.write(python_line)
> +
> + bool_line = '\tbool "python-{name}"\n'.format(name=pkg_name)
> + config_file.write(bool_line)
> + if pkg_req:
> + for dep in pkg_req:
> + dep_line = '\tselect BR2_PACKAGE_PYTHON_{req}\n'.format(
> + req=dep.upper())
> + config_file.write(dep_line)
> +
> + config_file.write('\thelp\n')
> +
> + help_lines = package['info']['summary'].split('\n')
> + help_lines.append('')
> + help_lines.append(package['info']['home_page'])
> + help_lines = ['\t {line}\n'.format(line=line)
> + for line in help_lines]
> + config_file.writelines(help_lines)
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
^ permalink raw reply [flat|nested] 8+ messages in thread
* [Buildroot] [PATCH 1/3] [RFC] python-package-generator: new utility
2015-06-01 22:37 ` Arnout Vandecappelle
@ 2015-06-02 16:23 ` Denis Thulin
0 siblings, 0 replies; 8+ messages in thread
From: Denis Thulin @ 2015-06-02 16:23 UTC (permalink / raw)
To: buildroot
Hi Arnout,
Thanks for the comments.
I will wait for other comments and submit a new version of the patch in a few
days.
> On 06/02/15 16:56, Arnout Vandecappelle wrote:
> Hi Denis,
>
> Thanks for this contribution! A big patch like this you can expect
> some
> comments of course...
>
> I didn't have time to look at all details, but here is some initial
> input.
>
>
> On 06/01/15 16:56, Denis THULIN wrote:
> > This patch adds package python-package-generator.py
> > ---
> > v0: initial commit
> > - python-pacakage-generator.py is an utility for automatically
> > generating
> > python packages using metadata from the python package index:
> > https://pypi.python.org
> >
> > I did not know where to put the script so I put it in
> > support/scripts.
> > I have updated the python-package section of the manual as well.
> >
> > Please bear in mind that python-package-generator.py does not add
> > the packages
> > to your buildroot project and you need to do it manually.
> >
> > Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
>
> SoB should go before the ---
My bad, I will change that.
>
> > ---
> > docs/manual/adding-packages-python.txt | 36 +++
> > support/scripts/python-package-generator.py | 435
> > ++++++++++++++++++++++++++++
> > 2 files changed, 471 insertions(+)
> > create mode 100755 support/scripts/python-package-generator.py
> >
> > diff --git a/docs/manual/adding-packages-python.txt
> > b/docs/manual/adding-packages-python.txt
> > index f81d625..6f608ed 100644
> > --- a/docs/manual/adding-packages-python.txt
> > +++ b/docs/manual/adding-packages-python.txt
> > @@ -7,6 +7,42 @@ This infrastructure applies to Python packages
> > that use the standard
> > Python setuptools mechanism as their build system, generally
> > recognizable by the usage of a +setup.py+ script.
> >
> > +[[python-package-generator]]
> > +
> > +==== generating a +python-package+ from a pypi repository
> > +
> > +You may want to use the +python-package-generator.py+ located in
> > ++support/script+ to generate a package from an existing pypi(pip)
> > package.
>
> Drop the (pip) - pip is a package installer, we don't use it.
Ok, I will change that
>
> > +
> > +you can find the list of existing pypi package here:
> > (https://pypi.python.org).
>
> You can find a list of available packages in
> https://pypi.python.org[the Python
> package index].
>
>
> > +
> > +Please keep in mind that you most likely need
> > +to manually check the package for any mistakes
> > +as there are things that cannot be guessed by the generator (e.g.
> > +dependencies on any of the python core modules
> > +such as BR2_PACKAGE_PYTHON_ZLIB).
>
> Please re-wrap this paragraph.
I'll wrap it to 80 chars
>
> > +
> > +When at the root of your buildroot directory just do :
> > +
> > +-----------------------
> > +./support/script/python-package-generator.py foo bar -o package
> > +-----------------------
> > +
> > +This will generate packages +python-foo+ and +python-bar+ in the
> > package
> > +folder if they exist on https://pypi.python.org.
> > +
> > +You will need to manually write the path to the package inside
> > +the +package/Config.in+ file:
> > +
> > +Find the +external python modules+ menu and insert your package
> > inside.
> > +Keep in mind that the items inside a menu should be in
> > alphabetical order.
>
> Just a single paragraph:
>
> You will need to manually add the package to +package/Config.in+.
> Find the ...
>
I will do that as well
>
> > +
> > +Option +-h+ wil list the options available
> > +
> > +-----------------------
> > +./support/script/python-package-generator.py -h
> > +-----------------------
> > +
> > [[python-package-tutorial]]
> >
> > ==== +python-package+ tutorial
> > diff --git a/support/scripts/python-package-generator.py
> > b/support/scripts/python-package-generator.py
> > new file mode 100755
> > index 0000000..4f5e884
> > --- /dev/null
> > +++ b/support/scripts/python-package-generator.py
> > @@ -0,0 +1,435 @@
> > +#!/usr/bin/python2
>
> Any chance of making the script 2+3 compatible?
I will look into it, probably not in the next version of the patch.
>
> > +"""
> > + Utility for building buildroot packages for existing pypi
> > packages
> > +
> > + Any package built by brpy-generator should be manually checked
> > for errors.
> python-package-generator
>
Correct, I forgot to change the doc here
> > +"""
> > +from __future__ import print_function
> > +import argparse
> > +import json
> > +import urllib2
> > +import sys
> > +import os
> > +import shutil
> > +import StringIO
> > +import tarfile
> > +import errno
> > +import hashlib
> > +import re
> > +import magic
> > +import tempfile
> > +from functools import wraps
> > +
> > +# TODO: Create a real module instead of a 320 line script
>
> No need for that.
I agree, I just forgot to remove that comment
>
> > +
> > +# private global
> > +_calls = {}
> > +
> > +
> > +def setup_info(pkg_name):
> > + """Get a package info from _calls
> > +
> > + Keyword arguments:
> > + pkg_name -- the name of the package
> > + """
> > + return _calls[pkg_name]
> > +
> > +
> > +def setup_decorator(func, method):
> > + """
> > + Decorator for distutils.core.setup and setuptools.setup.
> > + Puts the args of setup as a dict inside global private dict
> > _calls.
> > + Add key 'method' which should be either 'setuptools' or
> > 'distutils'.
> > +
> > + Keyword arguments:
> > + func -- either setuptools.setup or distutils.core.setup
> > + method -- either 'setuptools' or 'distutils'
> > + """
> > +
> > + @wraps(func)
> > + def closure(*args, **kwargs):
> > + _calls[kwargs['name']] = kwargs
> > + _calls[kwargs['name']]['method'] = method
> > +
> > + return closure
> > +
> > +
> > +def find_file_upper_case(filenames, path='./'):
> > + """
> > + List generator:
> > + Recursively find files that matches one of the specified
> > filenames.
> > + Returns absolute path
> > +
> > + Keyword arguments:
> > + filenames -- List of filenames to be found
> > + path -- Path to the directory to search
> > + """
> > + for root, dirs, files in os.walk(path):
> > + for file in files:
> > + if file.upper() in filenames:
> > + yield (os.path.join(root, file))
> > +
> > +
> > +def pkg_new_name(pkg_name):
>
> Bit a weird name for the function. Perhaps pkg_buildroot_name?
I will use that or find something better
>
> > + """
> > + Returns name to avoid troublesome characters.
> > + Remove all non alphanumeric characters except -
> > + Also lowers the name
> > +
> > + Keyword arguments:
> > + pkg_name -- String to rename
> > + """
> > + name = re.sub('[^\w-]', '', pkg_name.lower())
> > + name = name.lstrip('python-')
> > + return name
> > +
> > +
> > +def find_setup(package_name, version, archive):
> > + """
> > + Search for setup.py file in an archive and returns True if
> > found
> > + Used for finding the correct path to the setup.py
> > +
> > + Keyword arguments:
> > + package_name -- base name of the package to search (e.g.
> > Flask)
> > + version -- version of the package to search (e.g. 0.8.1)
> > + archive -- tar archive to search in
> > + """
> > + try:
> > + archive.getmember('{name}-{version}/setup.py'.format(
> > + name=package_name,
> > + version=version))
> > + except KeyError:
> > + return False
> > + else:
> > + return True
> > +
> > +
> > +# monkey patch
> > +import setuptools
> > +setuptools.setup = setup_decorator(setuptools.setup, 'setuptools')
> > +import distutils
> > +distutils.core.setup = setup_decorator(setuptools.setup,
> > 'distutils')
> > +
> > +if __name__ == "__main__":
> > +
> > + # Building the parser
> > + parser = argparse.ArgumentParser(
> > + description=("Creates buildroot packages from the metadata
> > of "
> > + "an existing pypi(pip) packages and include
> > it "
> > + "in menuconfig"))
>
> Spurious () around the string.
I will remove that
>
> > + parser.add_argument("packages",
> > + help="list of packages to be made",
> > + nargs='+')
> > + parser.add_argument("-o", "--output",
> > + help="""
> > + Output directory for packages
> > + """,
> > + default='.')
> > +
> > + args = parser.parse_args()
> > + packages = list(set(args.packages))
> > +
> > + # tmp_path is where we'll extract the files later
> > + tmp_prefix = '-python-package-generator'
> > + # dl_dir is supposed to be your buildroot dl dir
>
> What does this comment mean?
This comments refers to old code that I have already removed.
I will remove it.
>
> > + pkg_folder = args.output
> > + tmp_path = tempfile.mkdtemp(prefix=tmp_prefix)
> > +
> > + packages_local_names = map(pkg_new_name, packages)
>
> Remove this.
I will do as you say
>
> > + print(
> > + 'Character . is forbidden.',
> > + 'Generator will use only alphanumeric characters
> > (including _ and -)',
> > + sep='\n')
>
> Why print this?
I thought I had replaced that with a print of the buildroot package name.
I will replace it.
>
> > + for index, real_pkg_name in enumerate(packages):
> > + # First we download the package
> > + # Most of the info we need can only be found inside the
> > package
> > + pkg_name = packages_local_names[index]
>
> pkg_name = pkg_new_name(real_pkg_name)
> and you no longer need enumerate and index.
I will change that as well
>
> > + print('Package:', pkg_name)
> > + print('Fetching package', real_pkg_name)
> > + url = 'https://pypi.python.org/pypi/{pkg}/json'.format(
> > + pkg=real_pkg_name)
> > + print('URL:', url)
> > + try:
> > + pkg_json = urllib2.urlopen(url).read().decode()
> > + except (urllib2.HTTPError, urllib2.URLError) as error:
> > + print('ERROR:', error.getcode(), error.msg,
> > file=sys.stderr)
> > + print('ERROR: Could not find package {pkg}.\n'
> > + 'Check syntax inside the python package
> > index:\n'
> > + 'https://pypi.python.org/pypi/
> > '.format(pkg=real_pkg_name))
> > + continue
> > +
> > + pkg_dir = ''.join([pkg_folder, '/python-', pkg_name])
>
> IMHO using + is clearer.
I dislike using + in python, but if you think it is better, I will change it.
>
> > +
> > + package = json.loads(pkg_json)
> > + used_url = ''
> > + try:
> > + targz = package['urls'][0]['filename']
> > + except IndexError:
> > + print(
> > + 'Non conventional package, ',
> > + 'please check manually after creation')
> > + download_url = package['info']['download_url']
>
> Is that guaranteed to exist?
I think it is mandatory for all pypi packages. I should not pass the exception
though. Il will change that and stop building the package.
>
> > + try:
> > + download = urllib2.urlopen(download_url)
> > + except urllib2.HTTPError:
> > + pass
> > + else:
> > + used_url = {'url': download_url}
> > + as_file = StringIO.StringIO(download.read())
> > + md5_sum = hashlib.md5(as_file.read()).hexdigest()
>
> Use sha256 instead of md5.
Ok. I will do that.
>
> > + used_url['md5_digest'] = md5_sum
> > + as_file.seek(0)
> > + print(magic.from_buffer(as_file.read()))
> > + as_file.seek(0)
> > + extension = 'tar.gz'
> > + if 'gzip' not in
> > magic.from_buffer(as_file.read()):
> > + extension = 'tar.bz2'
> > + targz = '{name}-{version}.{extension}'.format(
> > + package['info']['name'],
> > package['info']['version'],
> > + extension)
> > + as_file.seek(0)
> > + used_url['filename'] = targz
> > +
> > + print(
> > + 'Downloading package
> > {pkg}...'.format(pkg=package['info']['name']))
> > + for download_url in package['urls']:
> > + try:
> > + download = urllib2.urlopen(download_url['url'])
> > + except urllib2.HTTPError:
> > + pass
> > + else:
> > + used_url = download_url
> > + as_file = StringIO.StringIO(download.read())
> > + md5_sum = hashlib.md5(as_file.read()).hexdigest()
> > + if md5_sum == download_url['md5_digest']:
> > + break
> > + targz = used_url['filename']
> > +
> > + if not download:
> > + print('Error downloading package :', pkg_name)
> > + continue
> > +
> > + # extract the tarball
> > + as_file.seek(0)
> > + as_tarfile = tarfile.open(fileobj=as_file)
> > + tmp_pkg = '/'.join([tmp_path, pkg_name])
> > + try:
> > + os.makedirs(tmp_pkg)
> > + except OSError as exception:
> > + if exception.errno != errno.EEXIST:
> > + print("ERROR: ", exception.message,
> > file=sys.stderr)
> > + continue
> > + print('WARNING:', exception.message, file=sys.stderr)
> > + print('Removing {pkg}...'.format(pkg=tmp_pkg))
> > + shutil.rmtree(tmp_pkg)
> > + os.makedirs(tmp_pkg)
> > + tar_folder_names = [real_pkg_name.capitalize(),
> > + real_pkg_name.lower(),
> > + package['info']['name']]
> > + version = package['info']['version']
> > + try:
> > + tar_folder = next(folder for folder in
> > tar_folder_names
> > + if find_setup(folder, version,
> > as_tarfile))
> > + except StopIteration:
> > + print('ERROR: Could not extract package %s' %
> > + real_pkg_name,
> > + file=sys.stderr)
> > + continue
>
> This looks really opaque to me. Can't it be expressed by a simple
> loop?
It can. I will change it
>
> > + as_tarfile.extractall(tmp_pkg)
> > + as_tarfile.close()
> > + as_file.close()
> > + tmp_extract = '{folder}/{name}-{version}'.format(
> > + folder=tmp_pkg,
> > + name=tar_folder,
> > + version=package['info']['version'])
> > +
> > + # Loading the package install info from the package
> > + sys.path.append(tmp_extract)
> > + import setup
> > + setup = reload(setup)
> > + sys.path.remove(tmp_extract)
> > +
> > + pkg_req = None
> > + # Package requierement are an argument of the setup
> > function
> > + if 'install_requires' in setup_info(tar_folder):
> > + pkg_req = setup_info(tar_folder)['install_requires']
> > + pkg_req = [re.sub('([\w-]+)[><=]*.*', r'\1',
> > req).lower()
> > + for req in pkg_req]
> > + pkg_req = map(pkg_new_name, pkg_req)
> > + req_not_found = [
> > + pkg for pkg in pkg_req
> > + if 'python-{name}'.format(name=pkg)
> > + not in os.listdir(pkg_folder)
> > + ]
> > + req_not_found = [pkg for pkg in req_not_found
> > + if pkg not in packages]
> > + if (req_not_found) != 0:
> > + print(
> > + 'Error: could not find packages
> > \'{packages}\'',
> > + 'required by {current_package}'.format(
> > + packages=", ".join(req_not_found),
> > + current_package=pkg_name))
> > + # We could stop here
> > + # or ask the user if he still wants to continue
> > +
> > + # Buildroot python packages require 3 files
> > + # The first is the mk file
> > + # See:
> > + #
> > http://buildroot.uclibc.org/downloads/manual/manual.html
> > + pkg_mk = 'python-{name}.mk'.format(name=pkg_name)
> > + path_to_mk = '/'.join([pkg_dir, pkg_mk])
> > + print('Creating {file}...'.format(file=path_to_mk))
> > + print('Checking if package {name} already
> > exists...'.format(
> > + name=pkg_dir))
> > + try:
> > + os.makedirs(pkg_dir)
> > + except OSError as exception:
> > + if exception.errno != errno.EEXIST:
> > + print("ERROR: ", exception.message,
> > file=sys.stderr)
> > + continue
> > + print('Error: Package {name} already
> > exists'.format(name=pkg_dir))
> > + del_pkg = raw_input(
> > + 'Do you want to delete existing package ? [y/N]')
> > + if del_pkg.lower() == 'y':
> > + shutil.rmtree(pkg_dir)
> > + os.makedirs(pkg_dir)
> > + else:
> > + continue
> > + with open(path_to_mk, 'w') as mk_file:
> > + # header
> > + header = ['#' * 80 + '\n']
> > + header.append('#\n')
> > + header.append('# {name}\n'.format(name=pkg_dir))
> > + header.append('#\n')
> > + header.append('#' * 80 + '\n')
> > + header.append('\n')
> > + mk_file.writelines(header)
> > +
> > + version_line = 'PYTHON_{name}_VERSION =
> > {version}\n'.format(
> > + name=pkg_name.upper(),
> > + version=package['info']['version'])
> > + mk_file.write(version_line)
> > + targz = targz.replace(
> > + package['info']['version'],
> > +
> > '$(PYTHON_{name}_VERSION)'.format(name=pkg_name.upper()))
> > + targz_line = 'PYTHON_{name}_SOURCE =
> > {filename}\n'.format(
> > + name=pkg_name.upper(),
> > + filename=targz)
> > + mk_file.write(targz_line)
> > +
> > + site_line = ('PYTHON_{name}_SITE = {url}\n'.format(
> > + name=pkg_name.upper(),
> > + url=used_url['url'].replace(used_url['filename'],
> > '')))
> > + if 'sourceforge' in site_line:
> > + site_line = ('PYTHON_{name}_SITE =
> > {url}\n'.format(
> > + name=pkg_name.upper(),
> > + url=used_url['url']))
> > +
> > + mk_file.write(site_line)
> > +
> > + # There are two things you can use to make an
> > installer
> > + # for a python package: distutils or setuptools
> > + # distutils comes with python but does not support
> > dependancies.
> > + # distutils is mostly still there for backward
> > support.
> > + # setuptools is what smart people use,
> > + # but it is not shipped with python :(
> > +
> > + # setuptools.setup calls distutils.core.setup
> > + # We use the monkey patch with a tag to know which one
> > is used.
> > + setup_type_line = 'PYTHON_{name}_SETUP_TYPE =
> > {method}\n'.format(
> > + name=pkg_name.upper(),
> > + method=setup_info(tar_folder)['method'])
> > + mk_file.write(setup_type_line)
> > +
> > + license_line = 'PYTHON_{name}_LICENSE =
> > {license}\n'.format(
> > + name=pkg_name.upper(),
> > + license=package['info']['license'])
> > + mk_file.write(license_line)
> > + print('WARNING: License has been set to "{license}",'
> > + ' please change it manually if
> > necessary'.format(
> > + license=package['info']['license']))
> > + filenames = ['LICENSE', 'LICENSE.TXT']
> > + license_files = list(find_file_upper_case(filenames,
> > tmp_extract))
> > + license_files = [license.replace(tmp_extract, '')[1:]
> > + for license in license_files]
> > + if len(license_files) > 1:
>
> It's OK to have more than one license file, so I'd keep both of
> them.
Sometimes there are License for images included in the package,
As I have no way to know what the license, I thought the user should chose
which ones to pick.
>
> > + print('More than one file found for license: ')
> > + for index, item in enumerate(license_files):
> > + print('\t{index})'.format(index), item)
> > + license_choices = raw_input(
> > + 'specify file numbers separated by
> > spaces(default 0): ')
> > + license_choices = [int(choice)
> > + for choice in
> > license_choices.split(' ')
> > + if choice.isdigit() and
> > int(choice) in
> > + range(len(license_files))]
> > + if len(license_choices) == 0:
> > + license_choices = [0]
> > + license_files = [file
> > + for index, file in
> > enumerate(license_files)
> > + if index in license_choices]
> > + elif len(license_files) == 0:
> > + print('WARNING: No license file found,'
> > + ' please specify it manually afterward')
> > +
> > + license_file_line = ('PYTHON_{name}_LICEiNSE_FILES ='
>
> LICENSE (spurious i)
True
>
> > + ' {files}\n'.format(
> > + name=pkg_name.upper(),
> > + files='
> > '.join(license_files)))
> > + mk_file.write(license_file_line)
> > +
> > + if pkg_req:
> > + python_pkg_req = ['python-{name}'.format(name=pkg)
> > + for pkg in pkg_req]
> > + dependencies_line = ('PYTHON_{name}_DEPENDENCIES
> > ='
> > + ' {reqs}\n'.format(
> > + name=pkg_name.upper(),
> > + reqs='
> > '.join(python_pkg_req)))
> > + mk_file.write(dependencies_line)
> > +
> > + mk_file.write('\n')
> > + mk_file.write('$(eval $(python-package))')
>
> Missing newline at the end.
>
>
I Will add one
>
> That's as far as I got :-)
>
> Regards,
> Arnout
Thank you for the comments :)
Regards,
Denis
>
> > +
> > + # The second file we make is the hash file
> > + # It consists of hashes of the package tarball
> > + #
> > http://buildroot.uclibc.org/downloads/manual/manual.html#adding-packages-hash
> > + pkg_hash = 'python-{name}.hash'.format(name=pkg_name)
> > + path_to_hash = '/'.join([pkg_dir, pkg_hash])
> > + print('Creating
> > {filename}...'.format(filename=path_to_hash))
> > + with open(path_to_hash, 'w') as hash_file:
> > + commented_line = '# md5 from {url}\n'.format(url=url)
> > + hash_file.write(commented_line)
> > +
> > + hash_line = 'md5\t{digest} {filename}\n'.format(
> > + digest=used_url['md5_digest'],
> > + filename=used_url['filename'])
> > + hash_file.write(hash_line)
> > +
> > + # The Config.in is the last file we create
> > + # It is used by buildroot's menuconfig, gconfig, xconfig
> > or nconfig
> > + # it is used to displayspackage info and to select
> > requirements
> > + #
> > http://buildroot.uclibc.org/downloads/manual/manual.html#_literal_config_in_literal_file
> > + path_to_config = '/'.join([pkg_dir, 'Config.in'])
> > + print('Creating {file}...'.format(file=path_to_config))
> > + with open(path_to_config, 'w') as config_file:
> > + config_line = 'config
> > BR2_PACKAGE_PYTHON_{name}\n'.format(
> > + name=pkg_name.upper())
> > + config_file.write(config_line)
> > + python_line = '\tdepends on BR2_PACKAGE_PYTHON\n'
> > + config_file.write(python_line)
> > +
> > + bool_line = '\tbool
> > "python-{name}"\n'.format(name=pkg_name)
> > + config_file.write(bool_line)
> > + if pkg_req:
> > + for dep in pkg_req:
> > + dep_line = '\tselect
> > BR2_PACKAGE_PYTHON_{req}\n'.format(
> > + req=dep.upper())
> > + config_file.write(dep_line)
> > +
> > + config_file.write('\thelp\n')
> > +
> > + help_lines = package['info']['summary'].split('\n')
> > + help_lines.append('')
> > + help_lines.append(package['info']['home_page'])
> > + help_lines = ['\t {line}\n'.format(line=line)
> > + for line in help_lines]
> > + config_file.writelines(help_lines)
> >
>
>
> --
> Arnout Vandecappelle arnout at mind be
> Senior Embedded Software Architect +32-16-286500
> Essensium/Mind http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR
> Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* [Buildroot] [PATCH 1/3] [RFC] python-package-generator: new utility
2015-06-01 14:56 ` [Buildroot] [PATCH 1/3] [RFC] python-package-generator: new utility Denis THULIN
2015-06-01 22:37 ` Arnout Vandecappelle
@ 2015-06-10 7:56 ` Thomas Petazzoni
1 sibling, 0 replies; 8+ messages in thread
From: Thomas Petazzoni @ 2015-06-10 7:56 UTC (permalink / raw)
To: buildroot
Denis,
Thanks for this contribution!
On Mon, 1 Jun 2015 16:56:07 +0200, Denis THULIN wrote:
> docs/manual/adding-packages-python.txt | 36 +++
> support/scripts/python-package-generator.py | 435 ++++++++++++++++++++++++++++
The location of the script is good, however to be consistent with the
existing support/scripts/scancpan script that we have for Perl
packages, I'd name your script support/scripts/scanpipy.
> +# monkey patch
> +import setuptools
> +setuptools.setup = setup_decorator(setuptools.setup, 'setuptools')
> +import distutils
> +distutils.core.setup = setup_decorator(setuptools.setup, 'distutils')
For non-Python experts, it's a bit hard to understand what is happening
here. Could you improve the comment to be more descriptive than just
"monkey patch" ?
> +if __name__ == "__main__":
> +
> + # Building the parser
> + parser = argparse.ArgumentParser(
> + description=("Creates buildroot packages from the metadata of "
> + "an existing pypi(pip) packages and include it "
> + "in menuconfig"))
> + parser.add_argument("packages",
> + help="list of packages to be made",
> + nargs='+')
> + parser.add_argument("-o", "--output",
> + help="""
> + Output directory for packages
> + """,
> + default='.')
> +
> + args = parser.parse_args()
> + packages = list(set(args.packages))
> +
> + # tmp_path is where we'll extract the files later
> + tmp_prefix = '-python-package-generator'
> + # dl_dir is supposed to be your buildroot dl dir
> + pkg_folder = args.output
> + tmp_path = tempfile.mkdtemp(prefix=tmp_prefix)
> +
> + packages_local_names = map(pkg_new_name, packages)
> + print(
> + 'Character . is forbidden.',
> + 'Generator will use only alphanumeric characters (including _ and -)',
> + sep='\n')
> + for index, real_pkg_name in enumerate(packages):
> + # First we download the package
> + # Most of the info we need can only be found inside the package
> + pkg_name = packages_local_names[index]
> + print('Package:', pkg_name)
> + print('Fetching package', real_pkg_name)
> + url = 'https://pypi.python.org/pypi/{pkg}/json'.format(
> + pkg=real_pkg_name)
> + print('URL:', url)
> + try:
> + pkg_json = urllib2.urlopen(url).read().decode()
> + except (urllib2.HTTPError, urllib2.URLError) as error:
> + print('ERROR:', error.getcode(), error.msg, file=sys.stderr)
> + print('ERROR: Could not find package {pkg}.\n'
> + 'Check syntax inside the python package index:\n'
> + 'https://pypi.python.org/pypi/ '.format(pkg=real_pkg_name))
> + continue
> +
> + pkg_dir = ''.join([pkg_folder, '/python-', pkg_name])
> +
> + package = json.loads(pkg_json)
> + used_url = ''
> + try:
> + targz = package['urls'][0]['filename']
> + except IndexError:
> + print(
> + 'Non conventional package, ',
> + 'please check manually after creation')
> + download_url = package['info']['download_url']
> + try:
> + download = urllib2.urlopen(download_url)
> + except urllib2.HTTPError:
> + pass
> + else:
> + used_url = {'url': download_url}
> + as_file = StringIO.StringIO(download.read())
> + md5_sum = hashlib.md5(as_file.read()).hexdigest()
> + used_url['md5_digest'] = md5_sum
> + as_file.seek(0)
> + print(magic.from_buffer(as_file.read()))
> + as_file.seek(0)
> + extension = 'tar.gz'
> + if 'gzip' not in magic.from_buffer(as_file.read()):
> + extension = 'tar.bz2'
> + targz = '{name}-{version}.{extension}'.format(
> + package['info']['name'], package['info']['version'],
> + extension)
> + as_file.seek(0)
> + used_url['filename'] = targz
> +
> + print(
> + 'Downloading package {pkg}...'.format(pkg=package['info']['name']))
> + for download_url in package['urls']:
> + try:
> + download = urllib2.urlopen(download_url['url'])
> + except urllib2.HTTPError:
> + pass
> + else:
> + used_url = download_url
> + as_file = StringIO.StringIO(download.read())
> + md5_sum = hashlib.md5(as_file.read()).hexdigest()
> + if md5_sum == download_url['md5_digest']:
> + break
> + targz = used_url['filename']
> +
> + if not download:
> + print('Error downloading package :', pkg_name)
> + continue
> +
> + # extract the tarball
> + as_file.seek(0)
> + as_tarfile = tarfile.open(fileobj=as_file)
> + tmp_pkg = '/'.join([tmp_path, pkg_name])
> + try:
> + os.makedirs(tmp_pkg)
> + except OSError as exception:
> + if exception.errno != errno.EEXIST:
> + print("ERROR: ", exception.message, file=sys.stderr)
> + continue
> + print('WARNING:', exception.message, file=sys.stderr)
> + print('Removing {pkg}...'.format(pkg=tmp_pkg))
> + shutil.rmtree(tmp_pkg)
> + os.makedirs(tmp_pkg)
> + tar_folder_names = [real_pkg_name.capitalize(),
> + real_pkg_name.lower(),
> + package['info']['name']]
> + version = package['info']['version']
> + try:
> + tar_folder = next(folder for folder in tar_folder_names
> + if find_setup(folder, version, as_tarfile))
> + except StopIteration:
> + print('ERROR: Could not extract package %s' %
> + real_pkg_name,
> + file=sys.stderr)
> + continue
> + as_tarfile.extractall(tmp_pkg)
> + as_tarfile.close()
> + as_file.close()
> + tmp_extract = '{folder}/{name}-{version}'.format(
> + folder=tmp_pkg,
> + name=tar_folder,
> + version=package['info']['version'])
> +
> + # Loading the package install info from the package
> + sys.path.append(tmp_extract)
> + import setup
> + setup = reload(setup)
> + sys.path.remove(tmp_extract)
> +
> + pkg_req = None
> + # Package requierement are an argument of the setup function
> + if 'install_requires' in setup_info(tar_folder):
> + pkg_req = setup_info(tar_folder)['install_requires']
> + pkg_req = [re.sub('([\w-]+)[><=]*.*', r'\1', req).lower()
> + for req in pkg_req]
> + pkg_req = map(pkg_new_name, pkg_req)
> + req_not_found = [
> + pkg for pkg in pkg_req
> + if 'python-{name}'.format(name=pkg)
> + not in os.listdir(pkg_folder)
> + ]
> + req_not_found = [pkg for pkg in req_not_found
> + if pkg not in packages]
> + if (req_not_found) != 0:
> + print(
> + 'Error: could not find packages \'{packages}\'',
> + 'required by {current_package}'.format(
> + packages=", ".join(req_not_found),
> + current_package=pkg_name))
> + # We could stop here
> + # or ask the user if he still wants to continue
> +
> + # Buildroot python packages require 3 files
> + # The first is the mk file
> + # See:
> + # http://buildroot.uclibc.org/downloads/manual/manual.html
> + pkg_mk = 'python-{name}.mk'.format(name=pkg_name)
> + path_to_mk = '/'.join([pkg_dir, pkg_mk])
> + print('Creating {file}...'.format(file=path_to_mk))
> + print('Checking if package {name} already exists...'.format(
> + name=pkg_dir))
> + try:
> + os.makedirs(pkg_dir)
> + except OSError as exception:
> + if exception.errno != errno.EEXIST:
> + print("ERROR: ", exception.message, file=sys.stderr)
> + continue
> + print('Error: Package {name} already exists'.format(name=pkg_dir))
> + del_pkg = raw_input(
> + 'Do you want to delete existing package ? [y/N]')
> + if del_pkg.lower() == 'y':
> + shutil.rmtree(pkg_dir)
> + os.makedirs(pkg_dir)
> + else:
> + continue
> + with open(path_to_mk, 'w') as mk_file:
> + # header
> + header = ['#' * 80 + '\n']
> + header.append('#\n')
> + header.append('# {name}\n'.format(name=pkg_dir))
> + header.append('#\n')
> + header.append('#' * 80 + '\n')
> + header.append('\n')
> + mk_file.writelines(header)
> +
> + version_line = 'PYTHON_{name}_VERSION = {version}\n'.format(
> + name=pkg_name.upper(),
> + version=package['info']['version'])
> + mk_file.write(version_line)
> + targz = targz.replace(
> + package['info']['version'],
> + '$(PYTHON_{name}_VERSION)'.format(name=pkg_name.upper()))
> + targz_line = 'PYTHON_{name}_SOURCE = {filename}\n'.format(
> + name=pkg_name.upper(),
> + filename=targz)
> + mk_file.write(targz_line)
> +
> + site_line = ('PYTHON_{name}_SITE = {url}\n'.format(
> + name=pkg_name.upper(),
> + url=used_url['url'].replace(used_url['filename'], '')))
> + if 'sourceforge' in site_line:
> + site_line = ('PYTHON_{name}_SITE = {url}\n'.format(
> + name=pkg_name.upper(),
> + url=used_url['url']))
> +
> + mk_file.write(site_line)
> +
> + # There are two things you can use to make an installer
> + # for a python package: distutils or setuptools
> + # distutils comes with python but does not support dependancies.
> + # distutils is mostly still there for backward support.
> + # setuptools is what smart people use,
> + # but it is not shipped with python :(
> +
> + # setuptools.setup calls distutils.core.setup
> + # We use the monkey patch with a tag to know which one is used.
> + setup_type_line = 'PYTHON_{name}_SETUP_TYPE = {method}\n'.format(
> + name=pkg_name.upper(),
> + method=setup_info(tar_folder)['method'])
> + mk_file.write(setup_type_line)
> +
> + license_line = 'PYTHON_{name}_LICENSE = {license}\n'.format(
> + name=pkg_name.upper(),
> + license=package['info']['license'])
> + mk_file.write(license_line)
> + print('WARNING: License has been set to "{license}",'
> + ' please change it manually if necessary'.format(
> + license=package['info']['license']))
> + filenames = ['LICENSE', 'LICENSE.TXT']
> + license_files = list(find_file_upper_case(filenames, tmp_extract))
> + license_files = [license.replace(tmp_extract, '')[1:]
> + for license in license_files]
> + if len(license_files) > 1:
> + print('More than one file found for license: ')
> + for index, item in enumerate(license_files):
> + print('\t{index})'.format(index), item)
> + license_choices = raw_input(
> + 'specify file numbers separated by spaces(default 0): ')
> + license_choices = [int(choice)
> + for choice in license_choices.split(' ')
> + if choice.isdigit() and int(choice) in
> + range(len(license_files))]
> + if len(license_choices) == 0:
> + license_choices = [0]
> + license_files = [file
> + for index, file in enumerate(license_files)
> + if index in license_choices]
> + elif len(license_files) == 0:
> + print('WARNING: No license file found,'
> + ' please specify it manually afterward')
> +
> + license_file_line = ('PYTHON_{name}_LICEiNSE_FILES ='
> + ' {files}\n'.format(
> + name=pkg_name.upper(),
> + files=' '.join(license_files)))
> + mk_file.write(license_file_line)
> +
> + if pkg_req:
> + python_pkg_req = ['python-{name}'.format(name=pkg)
> + for pkg in pkg_req]
> + dependencies_line = ('PYTHON_{name}_DEPENDENCIES ='
> + ' {reqs}\n'.format(
> + name=pkg_name.upper(),
> + reqs=' '.join(python_pkg_req)))
> + mk_file.write(dependencies_line)
> +
> + mk_file.write('\n')
> + mk_file.write('$(eval $(python-package))')
> +
> + # The second file we make is the hash file
> + # It consists of hashes of the package tarball
> + # http://buildroot.uclibc.org/downloads/manual/manual.html#adding-packages-hash
> + pkg_hash = 'python-{name}.hash'.format(name=pkg_name)
> + path_to_hash = '/'.join([pkg_dir, pkg_hash])
> + print('Creating {filename}...'.format(filename=path_to_hash))
> + with open(path_to_hash, 'w') as hash_file:
> + commented_line = '# md5 from {url}\n'.format(url=url)
> + hash_file.write(commented_line)
> +
> + hash_line = 'md5\t{digest} {filename}\n'.format(
> + digest=used_url['md5_digest'],
> + filename=used_url['filename'])
> + hash_file.write(hash_line)
> +
> + # The Config.in is the last file we create
> + # It is used by buildroot's menuconfig, gconfig, xconfig or nconfig
> + # it is used to displayspackage info and to select requirements
> + # http://buildroot.uclibc.org/downloads/manual/manual.html#_literal_config_in_literal_file
> + path_to_config = '/'.join([pkg_dir, 'Config.in'])
> + print('Creating {file}...'.format(file=path_to_config))
> + with open(path_to_config, 'w') as config_file:
> + config_line = 'config BR2_PACKAGE_PYTHON_{name}\n'.format(
> + name=pkg_name.upper())
> + config_file.write(config_line)
> + python_line = '\tdepends on BR2_PACKAGE_PYTHON\n'
> + config_file.write(python_line)
> +
> + bool_line = '\tbool "python-{name}"\n'.format(name=pkg_name)
> + config_file.write(bool_line)
> + if pkg_req:
> + for dep in pkg_req:
> + dep_line = '\tselect BR2_PACKAGE_PYTHON_{req}\n'.format(
> + req=dep.upper())
> + config_file.write(dep_line)
> +
> + config_file.write('\thelp\n')
> +
> + help_lines = package['info']['summary'].split('\n')
> + help_lines.append('')
> + help_lines.append(package['info']['home_page'])
> + help_lines = ['\t {line}\n'.format(line=line)
> + for line in help_lines]
> + config_file.writelines(help_lines)
This is really a huge body of code within a single function. Can you
split that up into multiple functions? At least one to generate
the .mk, one to generate the .hash, one to generate the Config.in.
The one generate the .mk would also benefit from having many
sub-functions: one sub-function to calculate the license related info,
one to calculate the setup type, etc.
Thanks,
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 8+ messages in thread
* [Buildroot] [PATCH 2/3] python-robotframework: New package
2015-06-01 14:56 ` [Buildroot] [PATCH 2/3] python-robotframework: New package Denis THULIN
@ 2015-06-12 16:09 ` Vincent Stehlé
0 siblings, 0 replies; 8+ messages in thread
From: Vincent Stehlé @ 2015-06-12 16:09 UTC (permalink / raw)
To: buildroot
Denis THULIN wrote:
> A generic test automation framework written in python.
Hi Denis,
Thank you for this robotframework package, here are a few remarks.
..
> diff --git a/package/python-robotframework/Config.in
> b/package/python-robotframework/Config.in
> new file mode 100644
> index 0000000..a3d5ea8
> --- /dev/null
> +++ b/package/python-robotframework/Config.in
> <at> <at> -0,0 +1,9 <at> <at>
> +config BR2_PACKAGE_PYTHON_ROBOTFRAMEWORK
> + depends on BR2_PACKAGE_PYTHON
> + select BR2_PACKAGE_PYTHON_ZLIB
> + select BR2_PACKAGE_PYTHON_PYEXPAT
^^^
I think the previous two lines have an indentation issue.
> + bool "python-robotframework"
> + help
> + A generic test automation framework
> +
> + http://robotframework.org
..
> diff --git a/package/python-robotframework/python-robotframework.mk
> b/package/python-robotframework/python-robotframework.mk
> new file mode 100644
> index 0000000..6fc7619
> --- /dev/null
> +++ b/package/python-robotframework/python-robotframework.mk
> <at> <at> -0,0 +1,14 <at> <at>
> +################################################################################
> +#
> +# package/python-robotframework
Shouldn't that rather be just 'python-robotframework', with no
'package/'?
> +#
> +################################################################################
> +
> +PYTHON_ROBOTFRAMEWORK_VERSION = 2.9a1
> +PYTHON_ROBOTFRAMEWORK_SOURCE =
> robotframework-$(PYTHON_ROBOTFRAMEWORK_VERSION).tar.gz
> +PYTHON_ROBOTFRAMEWORK_SITE = https://pypi.python.org/packages/source/r/robotframework/
> +PYTHON_ROBOTFRAMEWORK_SETUP_TYPE = distutils
> +PYTHON_ROBOTFRAMEWORK_LICENSE = Apache License 2.0
> +PYTHON_ROBOTFRAMEWORK_LICEiNSE_FILES = LICENSE.txt
^
The 'i' there is a typo, I think.
> +
> +$(eval $(python-package))
> \ No newline at end of file
Maybe add a newline?
Best regards,
V.
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2015-06-12 16:09 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-06-01 14:56 [Buildroot] [PATCH 0/3] [RFC] python-package-generator Denis THULIN
2015-06-01 14:56 ` [Buildroot] [PATCH 1/3] [RFC] python-package-generator: new utility Denis THULIN
2015-06-01 22:37 ` Arnout Vandecappelle
2015-06-02 16:23 ` Denis Thulin
2015-06-10 7:56 ` Thomas Petazzoni
2015-06-01 14:56 ` [Buildroot] [PATCH 2/3] python-robotframework: New package Denis THULIN
2015-06-12 16:09 ` Vincent Stehlé
2015-06-01 14:56 ` [Buildroot] [PATCH 3/3] python-magic: new package Denis THULIN
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox