From mboxrd@z Thu Jan 1 00:00:00 1970 From: Denis Thulin Date: Tue, 2 Jun 2015 18:23:34 +0200 (CEST) Subject: [Buildroot] [PATCH 1/3] [RFC] python-package-generator: new utility In-Reply-To: <556CDE9B.60700@mind.be> Message-ID: <1919051235.911559.1433262214393.JavaMail.root@openwide.fr> List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: buildroot@busybox.net Hi Arnout, Thanks for the comments. I will wait for other comments and submit a new version of the patch in a few days. > On 06/02/15 16:56, Arnout Vandecappelle wrote: > Hi Denis, > > Thanks for this contribution! A big patch like this you can expect > some > comments of course... > > I didn't have time to look at all details, but here is some initial > input. > > > On 06/01/15 16:56, Denis THULIN wrote: > > This patch adds package python-package-generator.py > > --- > > v0: initial commit > > - python-pacakage-generator.py is an utility for automatically > > generating > > python packages using metadata from the python package index: > > https://pypi.python.org > > > > I did not know where to put the script so I put it in > > support/scripts. > > I have updated the python-package section of the manual as well. > > > > Please bear in mind that python-package-generator.py does not add > > the packages > > to your buildroot project and you need to do it manually. > > > > Signed-off-by: Denis THULIN > > SoB should go before the --- My bad, I will change that. > > > --- > > docs/manual/adding-packages-python.txt | 36 +++ > > support/scripts/python-package-generator.py | 435 > > ++++++++++++++++++++++++++++ > > 2 files changed, 471 insertions(+) > > create mode 100755 support/scripts/python-package-generator.py > > > > diff --git a/docs/manual/adding-packages-python.txt > > b/docs/manual/adding-packages-python.txt > > index f81d625..6f608ed 100644 > > --- a/docs/manual/adding-packages-python.txt > > +++ b/docs/manual/adding-packages-python.txt > > @@ -7,6 +7,42 @@ This infrastructure applies to Python packages > > that use the standard > > Python setuptools mechanism as their build system, generally > > recognizable by the usage of a +setup.py+ script. > > > > +[[python-package-generator]] > > + > > +==== generating a +python-package+ from a pypi repository > > + > > +You may want to use the +python-package-generator.py+ located in > > ++support/script+ to generate a package from an existing pypi(pip) > > package. > > Drop the (pip) - pip is a package installer, we don't use it. Ok, I will change that > > > + > > +you can find the list of existing pypi package here: > > (https://pypi.python.org). > > You can find a list of available packages in > https://pypi.python.org[the Python > package index]. > > > > + > > +Please keep in mind that you most likely need > > +to manually check the package for any mistakes > > +as there are things that cannot be guessed by the generator (e.g. > > +dependencies on any of the python core modules > > +such as BR2_PACKAGE_PYTHON_ZLIB). > > Please re-wrap this paragraph. I'll wrap it to 80 chars > > > + > > +When at the root of your buildroot directory just do : > > + > > +----------------------- > > +./support/script/python-package-generator.py foo bar -o package > > +----------------------- > > + > > +This will generate packages +python-foo+ and +python-bar+ in the > > package > > +folder if they exist on https://pypi.python.org. > > + > > +You will need to manually write the path to the package inside > > +the +package/Config.in+ file: > > + > > +Find the +external python modules+ menu and insert your package > > inside. > > +Keep in mind that the items inside a menu should be in > > alphabetical order. > > Just a single paragraph: > > You will need to manually add the package to +package/Config.in+. > Find the ... > I will do that as well > > > + > > +Option +-h+ wil list the options available > > + > > +----------------------- > > +./support/script/python-package-generator.py -h > > +----------------------- > > + > > [[python-package-tutorial]] > > > > ==== +python-package+ tutorial > > diff --git a/support/scripts/python-package-generator.py > > b/support/scripts/python-package-generator.py > > new file mode 100755 > > index 0000000..4f5e884 > > --- /dev/null > > +++ b/support/scripts/python-package-generator.py > > @@ -0,0 +1,435 @@ > > +#!/usr/bin/python2 > > Any chance of making the script 2+3 compatible? I will look into it, probably not in the next version of the patch. > > > +""" > > + Utility for building buildroot packages for existing pypi > > packages > > + > > + Any package built by brpy-generator should be manually checked > > for errors. > python-package-generator > Correct, I forgot to change the doc here > > +""" > > +from __future__ import print_function > > +import argparse > > +import json > > +import urllib2 > > +import sys > > +import os > > +import shutil > > +import StringIO > > +import tarfile > > +import errno > > +import hashlib > > +import re > > +import magic > > +import tempfile > > +from functools import wraps > > + > > +# TODO: Create a real module instead of a 320 line script > > No need for that. I agree, I just forgot to remove that comment > > > + > > +# private global > > +_calls = {} > > + > > + > > +def setup_info(pkg_name): > > + """Get a package info from _calls > > + > > + Keyword arguments: > > + pkg_name -- the name of the package > > + """ > > + return _calls[pkg_name] > > + > > + > > +def setup_decorator(func, method): > > + """ > > + Decorator for distutils.core.setup and setuptools.setup. > > + Puts the args of setup as a dict inside global private dict > > _calls. > > + Add key 'method' which should be either 'setuptools' or > > 'distutils'. > > + > > + Keyword arguments: > > + func -- either setuptools.setup or distutils.core.setup > > + method -- either 'setuptools' or 'distutils' > > + """ > > + > > + @wraps(func) > > + def closure(*args, **kwargs): > > + _calls[kwargs['name']] = kwargs > > + _calls[kwargs['name']]['method'] = method > > + > > + return closure > > + > > + > > +def find_file_upper_case(filenames, path='./'): > > + """ > > + List generator: > > + Recursively find files that matches one of the specified > > filenames. > > + Returns absolute path > > + > > + Keyword arguments: > > + filenames -- List of filenames to be found > > + path -- Path to the directory to search > > + """ > > + for root, dirs, files in os.walk(path): > > + for file in files: > > + if file.upper() in filenames: > > + yield (os.path.join(root, file)) > > + > > + > > +def pkg_new_name(pkg_name): > > Bit a weird name for the function. Perhaps pkg_buildroot_name? I will use that or find something better > > > + """ > > + Returns name to avoid troublesome characters. > > + Remove all non alphanumeric characters except - > > + Also lowers the name > > + > > + Keyword arguments: > > + pkg_name -- String to rename > > + """ > > + name = re.sub('[^\w-]', '', pkg_name.lower()) > > + name = name.lstrip('python-') > > + return name > > + > > + > > +def find_setup(package_name, version, archive): > > + """ > > + Search for setup.py file in an archive and returns True if > > found > > + Used for finding the correct path to the setup.py > > + > > + Keyword arguments: > > + package_name -- base name of the package to search (e.g. > > Flask) > > + version -- version of the package to search (e.g. 0.8.1) > > + archive -- tar archive to search in > > + """ > > + try: > > + archive.getmember('{name}-{version}/setup.py'.format( > > + name=package_name, > > + version=version)) > > + except KeyError: > > + return False > > + else: > > + return True > > + > > + > > +# monkey patch > > +import setuptools > > +setuptools.setup = setup_decorator(setuptools.setup, 'setuptools') > > +import distutils > > +distutils.core.setup = setup_decorator(setuptools.setup, > > 'distutils') > > + > > +if __name__ == "__main__": > > + > > + # Building the parser > > + parser = argparse.ArgumentParser( > > + description=("Creates buildroot packages from the metadata > > of " > > + "an existing pypi(pip) packages and include > > it " > > + "in menuconfig")) > > Spurious () around the string. I will remove that > > > + parser.add_argument("packages", > > + help="list of packages to be made", > > + nargs='+') > > + parser.add_argument("-o", "--output", > > + help=""" > > + Output directory for packages > > + """, > > + default='.') > > + > > + args = parser.parse_args() > > + packages = list(set(args.packages)) > > + > > + # tmp_path is where we'll extract the files later > > + tmp_prefix = '-python-package-generator' > > + # dl_dir is supposed to be your buildroot dl dir > > What does this comment mean? This comments refers to old code that I have already removed. I will remove it. > > > + pkg_folder = args.output > > + tmp_path = tempfile.mkdtemp(prefix=tmp_prefix) > > + > > + packages_local_names = map(pkg_new_name, packages) > > Remove this. I will do as you say > > > + print( > > + 'Character . is forbidden.', > > + 'Generator will use only alphanumeric characters > > (including _ and -)', > > + sep='\n') > > Why print this? I thought I had replaced that with a print of the buildroot package name. I will replace it. > > > + for index, real_pkg_name in enumerate(packages): > > + # First we download the package > > + # Most of the info we need can only be found inside the > > package > > + pkg_name = packages_local_names[index] > > pkg_name = pkg_new_name(real_pkg_name) > and you no longer need enumerate and index. I will change that as well > > > + print('Package:', pkg_name) > > + print('Fetching package', real_pkg_name) > > + url = 'https://pypi.python.org/pypi/{pkg}/json'.format( > > + pkg=real_pkg_name) > > + print('URL:', url) > > + try: > > + pkg_json = urllib2.urlopen(url).read().decode() > > + except (urllib2.HTTPError, urllib2.URLError) as error: > > + print('ERROR:', error.getcode(), error.msg, > > file=sys.stderr) > > + print('ERROR: Could not find package {pkg}.\n' > > + 'Check syntax inside the python package > > index:\n' > > + 'https://pypi.python.org/pypi/ > > '.format(pkg=real_pkg_name)) > > + continue > > + > > + pkg_dir = ''.join([pkg_folder, '/python-', pkg_name]) > > IMHO using + is clearer. I dislike using + in python, but if you think it is better, I will change it. > > > + > > + package = json.loads(pkg_json) > > + used_url = '' > > + try: > > + targz = package['urls'][0]['filename'] > > + except IndexError: > > + print( > > + 'Non conventional package, ', > > + 'please check manually after creation') > > + download_url = package['info']['download_url'] > > Is that guaranteed to exist? I think it is mandatory for all pypi packages. I should not pass the exception though. Il will change that and stop building the package. > > > + try: > > + download = urllib2.urlopen(download_url) > > + except urllib2.HTTPError: > > + pass > > + else: > > + used_url = {'url': download_url} > > + as_file = StringIO.StringIO(download.read()) > > + md5_sum = hashlib.md5(as_file.read()).hexdigest() > > Use sha256 instead of md5. Ok. I will do that. > > > + used_url['md5_digest'] = md5_sum > > + as_file.seek(0) > > + print(magic.from_buffer(as_file.read())) > > + as_file.seek(0) > > + extension = 'tar.gz' > > + if 'gzip' not in > > magic.from_buffer(as_file.read()): > > + extension = 'tar.bz2' > > + targz = '{name}-{version}.{extension}'.format( > > + package['info']['name'], > > package['info']['version'], > > + extension) > > + as_file.seek(0) > > + used_url['filename'] = targz > > + > > + print( > > + 'Downloading package > > {pkg}...'.format(pkg=package['info']['name'])) > > + for download_url in package['urls']: > > + try: > > + download = urllib2.urlopen(download_url['url']) > > + except urllib2.HTTPError: > > + pass > > + else: > > + used_url = download_url > > + as_file = StringIO.StringIO(download.read()) > > + md5_sum = hashlib.md5(as_file.read()).hexdigest() > > + if md5_sum == download_url['md5_digest']: > > + break > > + targz = used_url['filename'] > > + > > + if not download: > > + print('Error downloading package :', pkg_name) > > + continue > > + > > + # extract the tarball > > + as_file.seek(0) > > + as_tarfile = tarfile.open(fileobj=as_file) > > + tmp_pkg = '/'.join([tmp_path, pkg_name]) > > + try: > > + os.makedirs(tmp_pkg) > > + except OSError as exception: > > + if exception.errno != errno.EEXIST: > > + print("ERROR: ", exception.message, > > file=sys.stderr) > > + continue > > + print('WARNING:', exception.message, file=sys.stderr) > > + print('Removing {pkg}...'.format(pkg=tmp_pkg)) > > + shutil.rmtree(tmp_pkg) > > + os.makedirs(tmp_pkg) > > + tar_folder_names = [real_pkg_name.capitalize(), > > + real_pkg_name.lower(), > > + package['info']['name']] > > + version = package['info']['version'] > > + try: > > + tar_folder = next(folder for folder in > > tar_folder_names > > + if find_setup(folder, version, > > as_tarfile)) > > + except StopIteration: > > + print('ERROR: Could not extract package %s' % > > + real_pkg_name, > > + file=sys.stderr) > > + continue > > This looks really opaque to me. Can't it be expressed by a simple > loop? It can. I will change it > > > + as_tarfile.extractall(tmp_pkg) > > + as_tarfile.close() > > + as_file.close() > > + tmp_extract = '{folder}/{name}-{version}'.format( > > + folder=tmp_pkg, > > + name=tar_folder, > > + version=package['info']['version']) > > + > > + # Loading the package install info from the package > > + sys.path.append(tmp_extract) > > + import setup > > + setup = reload(setup) > > + sys.path.remove(tmp_extract) > > + > > + pkg_req = None > > + # Package requierement are an argument of the setup > > function > > + if 'install_requires' in setup_info(tar_folder): > > + pkg_req = setup_info(tar_folder)['install_requires'] > > + pkg_req = [re.sub('([\w-]+)[><=]*.*', r'\1', > > req).lower() > > + for req in pkg_req] > > + pkg_req = map(pkg_new_name, pkg_req) > > + req_not_found = [ > > + pkg for pkg in pkg_req > > + if 'python-{name}'.format(name=pkg) > > + not in os.listdir(pkg_folder) > > + ] > > + req_not_found = [pkg for pkg in req_not_found > > + if pkg not in packages] > > + if (req_not_found) != 0: > > + print( > > + 'Error: could not find packages > > \'{packages}\'', > > + 'required by {current_package}'.format( > > + packages=", ".join(req_not_found), > > + current_package=pkg_name)) > > + # We could stop here > > + # or ask the user if he still wants to continue > > + > > + # Buildroot python packages require 3 files > > + # The first is the mk file > > + # See: > > + # > > http://buildroot.uclibc.org/downloads/manual/manual.html > > + pkg_mk = 'python-{name}.mk'.format(name=pkg_name) > > + path_to_mk = '/'.join([pkg_dir, pkg_mk]) > > + print('Creating {file}...'.format(file=path_to_mk)) > > + print('Checking if package {name} already > > exists...'.format( > > + name=pkg_dir)) > > + try: > > + os.makedirs(pkg_dir) > > + except OSError as exception: > > + if exception.errno != errno.EEXIST: > > + print("ERROR: ", exception.message, > > file=sys.stderr) > > + continue > > + print('Error: Package {name} already > > exists'.format(name=pkg_dir)) > > + del_pkg = raw_input( > > + 'Do you want to delete existing package ? [y/N]') > > + if del_pkg.lower() == 'y': > > + shutil.rmtree(pkg_dir) > > + os.makedirs(pkg_dir) > > + else: > > + continue > > + with open(path_to_mk, 'w') as mk_file: > > + # header > > + header = ['#' * 80 + '\n'] > > + header.append('#\n') > > + header.append('# {name}\n'.format(name=pkg_dir)) > > + header.append('#\n') > > + header.append('#' * 80 + '\n') > > + header.append('\n') > > + mk_file.writelines(header) > > + > > + version_line = 'PYTHON_{name}_VERSION = > > {version}\n'.format( > > + name=pkg_name.upper(), > > + version=package['info']['version']) > > + mk_file.write(version_line) > > + targz = targz.replace( > > + package['info']['version'], > > + > > '$(PYTHON_{name}_VERSION)'.format(name=pkg_name.upper())) > > + targz_line = 'PYTHON_{name}_SOURCE = > > {filename}\n'.format( > > + name=pkg_name.upper(), > > + filename=targz) > > + mk_file.write(targz_line) > > + > > + site_line = ('PYTHON_{name}_SITE = {url}\n'.format( > > + name=pkg_name.upper(), > > + url=used_url['url'].replace(used_url['filename'], > > ''))) > > + if 'sourceforge' in site_line: > > + site_line = ('PYTHON_{name}_SITE = > > {url}\n'.format( > > + name=pkg_name.upper(), > > + url=used_url['url'])) > > + > > + mk_file.write(site_line) > > + > > + # There are two things you can use to make an > > installer > > + # for a python package: distutils or setuptools > > + # distutils comes with python but does not support > > dependancies. > > + # distutils is mostly still there for backward > > support. > > + # setuptools is what smart people use, > > + # but it is not shipped with python :( > > + > > + # setuptools.setup calls distutils.core.setup > > + # We use the monkey patch with a tag to know which one > > is used. > > + setup_type_line = 'PYTHON_{name}_SETUP_TYPE = > > {method}\n'.format( > > + name=pkg_name.upper(), > > + method=setup_info(tar_folder)['method']) > > + mk_file.write(setup_type_line) > > + > > + license_line = 'PYTHON_{name}_LICENSE = > > {license}\n'.format( > > + name=pkg_name.upper(), > > + license=package['info']['license']) > > + mk_file.write(license_line) > > + print('WARNING: License has been set to "{license}",' > > + ' please change it manually if > > necessary'.format( > > + license=package['info']['license'])) > > + filenames = ['LICENSE', 'LICENSE.TXT'] > > + license_files = list(find_file_upper_case(filenames, > > tmp_extract)) > > + license_files = [license.replace(tmp_extract, '')[1:] > > + for license in license_files] > > + if len(license_files) > 1: > > It's OK to have more than one license file, so I'd keep both of > them. Sometimes there are License for images included in the package, As I have no way to know what the license, I thought the user should chose which ones to pick. > > > + print('More than one file found for license: ') > > + for index, item in enumerate(license_files): > > + print('\t{index})'.format(index), item) > > + license_choices = raw_input( > > + 'specify file numbers separated by > > spaces(default 0): ') > > + license_choices = [int(choice) > > + for choice in > > license_choices.split(' ') > > + if choice.isdigit() and > > int(choice) in > > + range(len(license_files))] > > + if len(license_choices) == 0: > > + license_choices = [0] > > + license_files = [file > > + for index, file in > > enumerate(license_files) > > + if index in license_choices] > > + elif len(license_files) == 0: > > + print('WARNING: No license file found,' > > + ' please specify it manually afterward') > > + > > + license_file_line = ('PYTHON_{name}_LICEiNSE_FILES =' > > LICENSE (spurious i) True > > > + ' {files}\n'.format( > > + name=pkg_name.upper(), > > + files=' > > '.join(license_files))) > > + mk_file.write(license_file_line) > > + > > + if pkg_req: > > + python_pkg_req = ['python-{name}'.format(name=pkg) > > + for pkg in pkg_req] > > + dependencies_line = ('PYTHON_{name}_DEPENDENCIES > > =' > > + ' {reqs}\n'.format( > > + name=pkg_name.upper(), > > + reqs=' > > '.join(python_pkg_req))) > > + mk_file.write(dependencies_line) > > + > > + mk_file.write('\n') > > + mk_file.write('$(eval $(python-package))') > > Missing newline at the end. > > I Will add one > > That's as far as I got :-) > > Regards, > Arnout Thank you for the comments :) Regards, Denis > > > + > > + # The second file we make is the hash file > > + # It consists of hashes of the package tarball > > + # > > http://buildroot.uclibc.org/downloads/manual/manual.html#adding-packages-hash > > + pkg_hash = 'python-{name}.hash'.format(name=pkg_name) > > + path_to_hash = '/'.join([pkg_dir, pkg_hash]) > > + print('Creating > > {filename}...'.format(filename=path_to_hash)) > > + with open(path_to_hash, 'w') as hash_file: > > + commented_line = '# md5 from {url}\n'.format(url=url) > > + hash_file.write(commented_line) > > + > > + hash_line = 'md5\t{digest} {filename}\n'.format( > > + digest=used_url['md5_digest'], > > + filename=used_url['filename']) > > + hash_file.write(hash_line) > > + > > + # The Config.in is the last file we create > > + # It is used by buildroot's menuconfig, gconfig, xconfig > > or nconfig > > + # it is used to displayspackage info and to select > > requirements > > + # > > http://buildroot.uclibc.org/downloads/manual/manual.html#_literal_config_in_literal_file > > + path_to_config = '/'.join([pkg_dir, 'Config.in']) > > + print('Creating {file}...'.format(file=path_to_config)) > > + with open(path_to_config, 'w') as config_file: > > + config_line = 'config > > BR2_PACKAGE_PYTHON_{name}\n'.format( > > + name=pkg_name.upper()) > > + config_file.write(config_line) > > + python_line = '\tdepends on BR2_PACKAGE_PYTHON\n' > > + config_file.write(python_line) > > + > > + bool_line = '\tbool > > "python-{name}"\n'.format(name=pkg_name) > > + config_file.write(bool_line) > > + if pkg_req: > > + for dep in pkg_req: > > + dep_line = '\tselect > > BR2_PACKAGE_PYTHON_{req}\n'.format( > > + req=dep.upper()) > > + config_file.write(dep_line) > > + > > + config_file.write('\thelp\n') > > + > > + help_lines = package['info']['summary'].split('\n') > > + help_lines.append('') > > + help_lines.append(package['info']['home_page']) > > + help_lines = ['\t {line}\n'.format(line=line) > > + for line in help_lines] > > + config_file.writelines(help_lines) > > > > > -- > Arnout Vandecappelle arnout at mind be > Senior Embedded Software Architect +32-16-286500 > Essensium/Mind http://www.mind.be > G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR > Leuven > LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle > GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F >