[Buildroot] Tool to check packages updates

[Buildroot] Tool to check packages updates

[arnaud aujon]

Hi,

As we talk earlier in the mailing list, I started working on a python
script to automatically check if a package can be updated.

For now, the result of the script is : 837 packages (115 up to date, 154
old and 568 unknown)
So the script is able to check that 115 packages are up to dates and that
154 have newer version or that the script is wrong.
There is still a lot of work to have something usefull, I will post the
script as soon as it's ready to be reviewed.

Cheers,

Arnaud
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131022/85f2bfc4/attachment-0001.html>

[Buildroot] Tool to check packages updates

[Thomas Petazzoni]

Arnaud,

On Tue, 22 Oct 2013 19:06:41 +0200, arnaud aujon wrote:

> As we talk earlier in the mailing list, I started working on a python
> script to automatically check if a package can be updated.
> 
> For now, the result of the script is : 837 packages (115 up to date, 154
> old and 568 unknown)
> So the script is able to check that 115 packages are up to dates and that
> 154 have newer version or that the script is wrong.
> There is still a lot of work to have something usefull, I will post the
> script as soon as it's ready to be reviewed.

Thanks, definitely looks interesting. Please post your script when you
think it's ok, but don't wait to much: the earlier the better, so that
you can get early feedback, and maybe help/testing from others.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Tool to check packages updates

[Thomas De Schampheleire]

Hi Arnaud,

On Tue, Oct 22, 2013 at 7:06 PM, arnaud aujon <arnaud.aujon@gmail.com> wrote:
> Hi,
>
> As we talk earlier in the mailing list, I started working on a python script
> to automatically check if a package can be updated.
>
> For now, the result of the script is : 837 packages (115 up to date, 154 old
> and 568 unknown)

The autobuild server seems to have a different package count:
http://autobuild.buildroot.org/stats/

Packages using the generic infrastructure 306
Packages using the cmake infrastructure 19
Packages using the autotools infrastructure 807
Packages not using any infrastructure 0

Since your script only sees 837 packages, I think you may be missing
some, unless the difference is only due to the duplicate counting of
host and target packages...

The above data is generated using support/scripts/pkg-stats

Best regards,
Thomas

[Buildroot] Tool to check packages updates

[arnaud aujon]

Both Thomas,

2013/10/23 Thomas De Schampheleire <patrickdepinguin@gmail.com>

>
>
> The autobuild server seems to have a different package count:
> http://autobuild.buildroot.org/stats/
>
> Packages using the generic infrastructure 306
> Packages using the cmake infrastructure 19
> Packages using the autotools infrastructure 807
> Packages not using any infrastructure 0
>
> Since your script only sees 837 packages, I think you may be missing
> some, unless the difference is only due to the duplicate counting of
> host and target packages...
>
>
>
You're right, my count isn't accurate, I intentionally skip some packages
that are not well handled by the script. The other difference is as you
said, I only count target packages.

Please post your script when you
> think it's ok, but don't wait to much: the earlier the better, so that
> you can get early feedback, and maybe help/testing from others.


I will do it as soon as possible, but I want to make it easy to read first.
As I'm far to be a python expert, it will take me some time to get used to
the right way to code in python.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131023/f36d10cf/attachment.html>

[Buildroot] Tool to check packages updates

[Arnout Vandecappelle]

On 23/10/13 17:43, arnaud aujon wrote:
>
>     Please post your script when you
>     think it's ok, but don't wait to much: the earlier the better, so that
>     you can get early feedback, and maybe help/testing from others.
>
>
> I will do it as soon as possible, but I want to make it easy to read
> first. As I'm far to be a python expert, it will take me some time to get
> used to the right way to code in python.

  Even so you can get feedback from the list already. In particular, it 
is useful to post something while it is still small - the larger the 
script, the less likely you get useful feedback.

  Regards,
  Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] Tool to check packages updates

[arnaud aujon]

Well, you're right, see the script bellow.
What need to be done is evaluate variable PACKAGE_SITE and PACKAGE_VERSION,
for the moment it doesn't evalute things like
PACKAGE_VERSION = $(VERSION_MAJOR).xx so I need to write a small recursive
function to retrieve the real value.
I also need to work on package source hosted with a CVS (mainly git and
svn).

Regards,

Arnaud

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import sys
import urllib2
from ftplib import FTP
import re
from os import listdir
from os.path import exists, isfile
import shutil
from distutils.version import LooseVersion

# mysql_client takes too long (why?ftp pb?)
exception_list = ('fan-ctrl', 'mysql_client')

def get_url_and_version(package_mk, package_name):
    version = ''
    site_url = ''
    package_name_mk = package_name.upper().replace('-','_')
    for line in open(package_mk):
        if 'include' in line:
            print 'TODO include in package name'
            site_url = ''
            version = '0'
        if package_name_mk + '_SITE ' in line:
            if '$' in line:
                print 'to be evaluated'
            else:
                site_url = line.split('= ')[1][:-1]
        if package_name_mk + '_VERSION ' in line:
            version = get_current_version(line)

    return (site_url, version)


# return version, evaluate it when using variables
def get_current_version(version_string):

    # if we need to evaluate the string
    if '$' in version_string:
        version = 'to be evaluated' + version_string
    else:
        try:
            version = version_string.split('= ')[1][:-1]
        except IndexError:
            version = 'unknown'

    return version

## Helpers functions

def helper_get_longest_sequence(search_str):
    longest_match = ''
    for match in re.finditer(r'((\d+\.)*(\d)+)', search_str):
        if len(match.group(0)) > len(longest_match):
            longest_match = match.group(0)
    return longest_match

def helper_get_last_version(search_str):
    last_version  = LooseVersion('0')
    for version in search_str:
        if (LooseVersion(version)) > last_version:
            last_version = version

    return last_version


## functions to retrieve last version ##

# * Specific website
# * try to find last version by listing available downloads
def get_last_version(package_name, site_url):
    print >> sys.stderr,'specific website : ' + site_url
    try:
        result = urllib2.urlopen(site_url).read()
    except:
        return 'website error'

    last_version = re.findall(r'%s-(.*)(.zip|.tar.gz|.tar.bz2)' %
package_name, result)
    last_version = [i[0] for i in last_version]
#    print last_version
    try:
        last_version = helper_get_last_version(last_version).split('-')[-1]
    except:
        print 'can\'t determine site version'
        return 'unknown'
    print last_version

    return last_version

# * FTP
# * try to find last version by listing FTP dir
def get_last_version_ftp(package_name, site_url):
    base_url = site_url.split('/')[2]
    directory = site_url.split(base_url)[1]

    print >> sys.stderr,'ftp : ' + base_url + ' ' + directory

    try:
        ftp = FTP(base_url)
        ftp.login()
        ftp.cwd(directory)
        files = []
        ftp.retrlines('LIST', files.append)
        ftp.quit()
    except:
        return 'ftp error'
    versions = [i.split(' ')[-1] for i in files]
#    print versions

    try:
        last_version = helper_get_last_version(versions).split('-')[-1]
    except:
        print 'can\'t determine last ftp version'
        return 'unknown'
    print last_version

    return last_version

# * Sourceforge.net
# * The following url give you the url of the last version available
# * http://sourceforge.net/projects/PROJECT_NAME/files/latest/download
def get_last_version_sf(site_url):
    print >> sys.stderr,'sourceforge website'
    if '/project/' in site_url:
        package_sf_name = site_url.split('project/')[1].split('/')[0]
    elif '/sourceforge/' in site_url:
        package_sf_name = site_url.split('/sourceforge/')[1].split('/')[0]
    elif '.cvs.' in site_url:
        package_sf_name = site_url.split('http://')[1].split('.')[0]
    elif '.sourceforge.net' in site_url:
        package_sf_name = site_url.split('.sourceforge.net
')[0].split('//')[1]
    else:
        print >> sys.stderr, 'Can\'t find sf package name'
#    print >> sys.stderr,package_sf_name
    project_url = 'http://sourceforge.net/projects/%s/files/latest/download'
% package_sf_name
#    print >> sys.stderr,project_url
    try:
        result = urllib2.urlopen(project_url).geturl()
    except urllib2.HTTPError:
        return 'website error'
    result = urllib2.unquote(result) # replace all hexa caracters like %20
#    print >> sys.stderr,result
    last_version = helper_get_longest_sequence(result)
    if (last_version == None):
        return 'unknown'
    print >> sys.stderr,last_version

    return last_version

# * googlecode.com
# * The following url give you the url of the last version available
# * https://code.google.com/p/PROJECT_NAME/downloads/list
def get_last_version_gc(site_url):
    print >> sys.stderr,'googlecode website'
    if '.googlecode.com' in site_url:
        package_gc_name = site_url.split('.googlecode.com
')[0].split('//')[1]
    else:
        print >> sys.stderr, 'Can\'t find gc package name'
#    print >> sys.stderr,package_gc_name

    try:
        result = urllib2.urlopen('
https://code.google.com/p/%s/downloads/list'% package_gc_name).read()
    except urllib2.HTTPError:
        print 'website error'
        return 'website error'

    #result = urllib2.unquote(result) # replace all hexa caracters like %20
    last_version = re.findall(r'name=%s-(.*)(.zip|.tar.gz|.tar.bz2)' %
package_gc_name, result)
    last_version = [i[0] for i in last_version]

    try:
        last_version = helper_get_last_version(last_version).split('-')[-1]
    except:
        print 'can\'t determine googlecode version'
        return 'unknown'
    print last_version

    return last_version


def check(package_name):
    print >> sys.stderr, 'Checking package: ' + package_name
    if package_name in exception_list:
        print >> sys.stderr, 'Exception package!TODO later'
        return('unknown', 'unknown', 'unknown')
    package_mk = '../../package/' + package_name + '/' + package_name +
'.mk'
    # Check if package exists
    if exists(package_mk) == False:
        print >> sys.stderr,'Error: package doesn\'t exist'
        return('unknown', 'unknown', 'unknown')

    # Get package url and version
    print >> sys.stderr,'package exists'
    site_url = ""
    provider = ""
    version = ""
    last_version = 'unknown'

    (site_url, version) = get_url_and_version(package_mk, package_name)

    if site_url == '':
        print >> sys.stderr,'Error: package url not found'
        return(version, last_version, 'unknown')

    print >> sys.stderr,'site :' + site_url
    print >> sys.stderr,'version: ' + version

    # Check if there is a new version
    if ('sf.net' in site_url or 'sourceforge.net' in site_url):
        provider = 'sourceforge.net'
        last_version = get_last_version_sf(site_url)
    elif 'googlecode.com' in site_url:
        provider = 'googlecode.com'
        last_version = get_last_version_gc(site_url)
    elif 'ftp://' in site_url:
        provider = 'specific ftp'
        last_version = get_last_version_ftp(package_name, site_url)
    elif 'git://' in site_url:
        provider = 'git'
        last_version = 'unknown'
    else:
        provider = 'specific website'
        last_version = get_last_version(package_name, site_url)

    return (version,last_version,provider)

if __name__ == '__main__':
    nb_package = 0
    nb_package_uptodate = 0
    nb_package_toupdate = 0
    nb_package_unknown = 0
    shutil.copyfile('template.html', 'out.html')
    f = open('out.html', 'a')

    for package in listdir('../../package'):
        if isfile('../../package/' + package):
            continue
        nb_package += 1
        (current_version, last_version, provider) = check(package)
        if last_version == 'unknown' or last_version == 'website error':
            style = 'centered'
            nb_package_unknown += 1
        elif current_version == last_version:
            style = 'centered uptodate'
            nb_package_uptodate += 1
        else:
            style = 'centered old'
            nb_package_toupdate += 1
        f.write('<tr class="%s"><td>%s</td><td >%s</td>' % (style, package,
current_version))
        f.write('<td>%s</td><td>%s</td></tr>' % (last_version, provider))

    f.write('Stats: %d packages (%d up to date, %d old and %d unknown)' %
(nb_package, nb_package_uptodate, nb_package_toupdate, nb_package_unknown))
    print 'done!'




2013/10/23 Arnout Vandecappelle <arnout@mind.be>

> On 23/10/13 17:43, arnaud aujon wrote:
>
>>
>>     Please post your script when you
>>     think it's ok, but don't wait to much: the earlier the better, so that
>>     you can get early feedback, and maybe help/testing from others.
>>
>>
>> I will do it as soon as possible, but I want to make it easy to read
>> first. As I'm far to be a python expert, it will take me some time to get
>> used to the right way to code in python.
>>
>
>  Even so you can get feedback from the list already. In particular, it is
> useful to post something while it is still small - the larger the script,
> the less likely you get useful feedback.
>
>  Regards,
>  Arnout
>
> --
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/**arnoutvandecappelle<http://www.linkedin.com/in/arnoutvandecappelle>
> GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131023/d721cc48/attachment-0001.html>

[Buildroot] Tool to check packages updates

[Thomas De Schampheleire]

Hi,

On Wed, Oct 23, 2013 at 8:58 PM, arnaud aujon <arnaud.aujon@gmail.com> wrote:
> Well, you're right, see the script bellow.
> What need to be done is evaluate variable PACKAGE_SITE and PACKAGE_VERSION,
> for the moment it doesn't evalute things like
> PACKAGE_VERSION = $(VERSION_MAJOR).xx so I need to write a small recursive
> function to retrieve the real value.
> I also need to work on package source hosted with a CVS (mainly git and
> svn).
>

make knows exactly how to evaluate such variables, so instead of
trying to do this from a script, isn't it possible to use the output
of 'make printvars' ? This prints _all_ the relevant variables but you
could filter on _SOURCE or _SITE to get what you need and base your
script on that.

Best regards,
Thomas

[Buildroot] Tool to check packages updates

[arnaud aujon]

2013/10/23 Thomas De Schampheleire <patrickdepinguin@gmail.com>

> make knows exactly how to evaluate such variables, so instead of
> trying to do this from a script, isn't it possible to use the output
> of 'make printvars' ? This prints _all_ the relevant variables but you
> could filter on _SOURCE or _SITE to get what you need and base your
> script on that.
>
>
Great ! I looked for something like that but I probably missed it, thanks
for the tip.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131023/dcf0a243/attachment.html>

[Buildroot] Tool to check packages updates

[arnaud aujon]

Hi,

Some news about the update checking script.
The results are now : 839 packages (this is not the real package number
because I don't check some of them), 294 are up-to-date, 323 can be bumped
and 222 unknown.
You can find the script (check-updates.py) and the result (out.html) here :
http://intelibre.fr/buildroot/

What's next :
  - check git packages
  - speed up the script
  - improve detection (there is still some false detection)

Cheers,

Arnaud Aujon


2013/10/23 arnaud aujon <arnaud.aujon@gmail.com>

> 2013/10/23 Thomas De Schampheleire <patrickdepinguin@gmail.com>
>
>> make knows exactly how to evaluate such variables, so instead of
>> trying to do this from a script, isn't it possible to use the output
>> of 'make printvars' ? This prints _all_ the relevant variables but you
>> could filter on _SOURCE or _SITE to get what you need and base your
>> script on that.
>>
>>
> Great ! I looked for something like that but I probably missed it, thanks
> for the tip.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131108/4208bdf6/attachment-0001.html>

[Buildroot] Tool to check packages updates

[Thomas Petazzoni]

Arnaud,

On Fri, 8 Nov 2013 15:29:27 +0100, arnaud aujon wrote:

> Some news about the update checking script.
> The results are now : 839 packages (this is not the real package
> number because I don't check some of them), 294 are up-to-date, 323
> can be bumped and 222 unknown.
> You can find the script (check-updates.py) and the result (out.html)
> here : http://intelibre.fr/buildroot/

Thanks! Starts looking really interesting. I haven't had a look at the
script for now, only the results.

A few comments:

 * For util-linux, current version is 2.23.2, and your script believes
   2.23 is more recent.

 * For bash, current version is 4.2, and your script believes version
   3.2 is more recent.

 * There seems to be an issue with libjpeg version detection.

 * The latest version detected for aumix is 2, while it should be 2.9.1.

 * One funny case: the latest version detected for tstools is
   "WindowsXP" :-)

 * There are several more problematic cases, but I guess many of them
   have the same cause.

Also, it would be nice if you could order the list of packages
alphabetically.

Keep up the good work!

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Tool to check packages updates

[arnaud aujon]

Hi everyone,

I updated the script to check package updates. It improved a lot :)
Current result are 841 packages checked : 372 up to date, 315 can be
updated and 154 are unknown.
It's still in developpement, there is a lot of thinks that can be improved :
  - handle svn
  - add a 'status' column
  - improve checking of sourceforge project, currently I retrieve last
version of the project using a /latest url, but it's not working when the
project provides several tools or libraries.
  - for projects using git, if the version is a hash, I retrieve HEAD hash
and compare it to the version, but I'm not checking if the hash represent a
tag (ie, most of the time a stable release)
  - code refactoring to avoid code duplication

2013/11/8 Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

> Also, it would be nice if you could order the list of packages
> alphabetically.
>

Done ! You can click on the column title to sort packages

Regards,

Arnaud
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131120/8fe45b76/attachment.html>

[Buildroot] Tool to check packages updates

[Thomas De Schampheleire]

Hi,

On Wed, Nov 20, 2013 at 4:34 PM, arnaud aujon <arnaud.aujon@gmail.com> wrote:
> Hi everyone,
>
> I updated the script to check package updates. It improved a lot :)
> Current result are 841 packages checked : 372 up to date, 315 can be updated
> and 154 are unknown.
> It's still in developpement, there is a lot of thinks that can be improved :
>   - handle svn
>   - add a 'status' column
>   - improve checking of sourceforge project, currently I retrieve last
> version of the project using a /latest url, but it's not working when the
> project provides several tools or libraries.
>   - for projects using git, if the version is a hash, I retrieve HEAD hash
> and compare it to the version, but I'm not checking if the hash represent a
> tag (ie, most of the time a stable release)
>   - code refactoring to avoid code duplication
>

Just a quick read-through of the output, here are some packages that
seem to be incorrectly marked as 'to be updated'

(package current new)
grantlee 0.2.0 v0.1.1
icu 51.2 2-Win64-msvc10    (win64?)
audiofile 0.3.6 68k.org/audiofile-0.3.6
bcusdk 0.0.5 examples
bonnie 1.03e .03c
celt051 0.5.1.3 060_tools_win32
(there are more)

Best regards,
Thomas