* [PATCH 00/12] Usage bitbake fetcher latest_versionstring in distrodata_class
@ 2014-11-28 1:10 ` Aníbal Limón
0 siblings, 0 replies; 16+ messages in thread
From: Aníbal Limón @ 2014-11-28 1:09 UTC (permalink / raw)
To: openembedded-core
Since Bitbake fetcher has latest_versionstring method that support search last version
of package in upstream sites, distrodata class should use it instead of do itself,
Summary of changes,
distrodata_class: Re-implement upstream version detection using Bitbake fetcher methods.
package_regex.inc: Update regex because Bitbake fetcher already provides support for search
in HTML href's tags instead of RAW lines.
bitbake: Improvements in fetcher wget latest_versionstring method based on [1] and clean code.
Compatibility testing was made running distrodata class and the result files can be found at [1].
The code has DEPENDENCIES the Bitbake changes should be MERGE first then package_regex.inc
and finally distrodata_class.
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=1813
The following changes since commit 0bc03af7ee6112fa0af0608b02f715ec8495e4ff:
maintainers.inc: update maintainers (2014-11-26 17:06:10 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib alimon/fetcher-distrodata
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=alimon/fetcher-distrodata
Aníbal Limón (12):
distrodata_class: checkpkg make usage of latest_versionstring methods
in bitbake fetcher
package_regex.inc: Update REGEX'es in order to find upstream versions
correctly
fetch2/wget.py: latest_versionstring create _init_regex method for
have one place when regex'es are defined
fetch2/wget.py: _init_regexes rename variables to be more consistent
and move dirver_regex into it
fetch/wget.php: latest version string only try to find latest
directory when REGEX_URI isn't specified to avoid unnecessary
processing and makes code easier
fetch2/wget.py: latest_versionstring add package_custom_regex_comp
fetch2/wget.py: latest_versionstring improvments in searching
fetch/wget.py: latest_versionstring remove unnecessary usage for name
in version comparision
fetch2/wget: latest_versionstring add support for search in RAW html
lines
fetch/wget.py: latest_versionstring remove newuri set because is
previous set
tests/fetch.py: Update wget latest_versionstring cups case
fetch2/wget.py: latest_versionstring clean improvments minor
bitbake/lib/bb/fetch2/wget.py | 146 ++++---
bitbake/lib/bb/tests/fetch.py | 2 +-
meta-yocto/conf/distro/include/package_regex.inc | 400 +++++++++---------
meta/classes/distrodata.bbclass | 492 ++---------------------
4 files changed, 337 insertions(+), 703 deletions(-)
--
1.9.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH 01/12] distrodata_class: checkpkg make usage of latest_versionstring methods in bitbake fetcher
2014-11-28 1:10 ` Aníbal Limón
(?)
(?)
@ 2014-11-28 1:09 ` Aníbal Limón
-1 siblings, 0 replies; 16+ messages in thread
From: Aníbal Limón @ 2014-11-28 1:09 UTC (permalink / raw)
To: openembedded-core
Because methods for get latest version of upstream package are now available
into bitbake removes duplicated code and make use of it.
Compatibility testing was made running distrodata class and the result files
can be found at:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=1813
[YOCTO #1813]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
meta/classes/distrodata.bbclass | 492 ++++------------------------------------
1 file changed, 43 insertions(+), 449 deletions(-)
diff --git a/meta/classes/distrodata.bbclass b/meta/classes/distrodata.bbclass
index a890de7..83aa381 100644
--- a/meta/classes/distrodata.bbclass
+++ b/meta/classes/distrodata.bbclass
@@ -268,240 +268,6 @@ python do_checkpkg() {
import tempfile
import subprocess
- """
- sanity check to ensure same name and type. Match as many patterns as possible
- such as:
- gnome-common-2.20.0.tar.gz (most common format)
- gtk+-2.90.1.tar.gz
- xf86-input-synaptics-12.6.9.tar.gz
- dri2proto-2.3.tar.gz
- blktool_4.orig.tar.gz
- libid3tag-0.15.1b.tar.gz
- unzip552.tar.gz
- icu4c-3_6-src.tgz
- genext2fs_1.3.orig.tar.gz
- gst-fluendo-mp3
- """
- prefix1 = "[a-zA-Z][a-zA-Z0-9]*([\-_][a-zA-Z]\w+)*\+?[\-_]" # match most patterns which uses "-" as separator to version digits
- prefix2 = "[a-zA-Z]+" # a loose pattern such as for unzip552.tar.gz
- prefix3 = "[0-9]+[\-]?[a-zA-Z]+" # a loose pattern such as for 80325-quicky-0.4.tar.gz
- prefix = "(%s|%s|%s)" % (prefix1, prefix2, prefix3)
- ver_regex = "(([A-Z]*\d+[a-zA-Z]*[\.\-_]*)+)"#"((\d+[\.\-_[a-z]])+)"
- # src.rpm extension was added only for rpm package. Can be removed if the rpm
- # packaged will always be considered as having to be manually upgraded
- suffix = "(tar\.gz|tgz|tar\.bz2|tar\.lz4|zip|xz|rpm|bz2|lz4|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)"
-
- suffixtuple = ("tar.gz", "tgz", "zip", "tar.bz2", "tar.xz", "tar.lz4", "bz2", "lz4", "orig.tar.gz", "src.tar.gz", "src.rpm", "src.tgz", "svnr\d+.tar.bz2", "stable.tar.gz", "src.rpm")
- sinterstr = "(?P<name>%s?)v?(?P<ver>%s)(\-source)?" % (prefix, ver_regex)
- sdirstr = "(?P<name>%s)\.?v?(?P<ver>%s)(\-source)?[\.\-](?P<type>%s$)" % (prefix, ver_regex, suffix)
-
- def parse_inter(s):
- m = re.search(sinterstr, s)
- if not m:
- return None
- else:
- return (m.group('name'), m.group('ver'), "")
-
- def parse_dir(s):
- m = re.search(sdirstr, s)
- if not m:
- return None
- else:
- return (m.group('name'), m.group('ver'), m.group('type'))
-
- def modelate_version(version):
- if version[0] in ['.', '-']:
- if version[1].isdigit():
- version = version[1] + version[0] + version[2:len(version)]
- else:
- version = version[1:len(version)]
-
- version = re.sub('\-', '.', version)
- version = re.sub('_', '.', version)
- version = re.sub('(rc)+', '.-1.', version)
- version = re.sub('(alpha)+', '.-3.', version)
- version = re.sub('(beta)+', '.-2.', version)
- if version[0] == 'v':
- version = version[1:len(version)]
- return version
-
- """
- Check whether 'new' is newer than 'old' version. We use existing vercmp() for the
- purpose. PE is cleared in comparison as it's not for build, and PV is cleared too
- for simplicity as it's somehow difficult to get from various upstream format
- """
- def __vercmp(old, new):
- (on, ov, ot) = old
- (en, ev, et) = new
- if on != en or (et and et not in suffixtuple):
- return False
- ov = modelate_version(ov)
- ev = modelate_version(ev)
-
- result = bb.utils.vercmp(("0", ov, ""), ("0", ev, ""))
- if result < 0:
- return True
- else:
- return False
-
- """
- wrapper for fetch upstream directory info
- 'url' - upstream link customized by regular expression
- 'd' - database
- 'tmpf' - tmpfile for fetcher output
- We don't want to exit whole build due to one recipe error. So handle all exceptions
- gracefully w/o leaking to outer.
- """
- def internal_fetch_wget(url, ud, d, tmpf):
- status = "ErrFetchUnknown"
-
- agent = "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101027 Ubuntu/9.10 (karmic) Firefox/3.6.12"
- fetchcmd = "/usr/bin/env wget -t 1 --passive-ftp -O %s --user-agent=\"%s\" '%s'" % (tmpf.name, agent, url)
- try:
- fetcher = bb.fetch2.wget.Wget(d)
- fetcher._runwget(ud, d, fetchcmd, True)
- status = "SUCC"
- except bb.fetch2.BBFetchException, e:
- status = "ErrFetch"
-
- return status
-
- """
- Check on middle version directory such as "2.4/" in "http://xxx/2.4/pkg-2.4.1.tar.gz",
- 'url' - upstream link customized by regular expression
- 'd' - database
- 'curver' - current version
- Return new version if success, or else error in "Errxxxx" style
- """
- def check_new_dir(url, curver, ud, d):
- pn = d.getVar('PN', True)
- f = tempfile.NamedTemporaryFile(delete=False, prefix="%s-1-" % pn)
- status = internal_fetch_wget(url, ud, d, f)
- fhtml = f.read()
- if status == "SUCC" and len(fhtml):
- newver = parse_inter(curver)
-
- """
- match "*4.1/">*4.1/ where '*' matches chars
- N.B. add package name, only match for digits
- """
- regex = d.getVar('REGEX', True)
- if regex == '':
- regex = "^%s" %prefix
- m = re.search("^%s" % regex, curver)
- if m:
- s = "%s[^\d\"]*?(\d+[\.\-_])+\d+/?" % m.group()
- else:
- s = "(\d+[\.\-_])+\d+/?"
-
- searchstr = "[hH][rR][eE][fF]=\"%s\">" % s
-
- reg = re.compile(searchstr)
- valid = 0
- for line in fhtml.split("\n"):
- if line.find(curver) >= 0:
- valid = 1
- m = reg.search(line)
- if m:
- ver = m.group().split("\"")[1]
- ver = ver.strip("/")
- ver = parse_inter(ver)
- if ver and __vercmp(newver, ver) == True:
- newver = ver
-
- """Expect a match for curver in directory list, or else it indicates unknown format"""
- if not valid:
- status = "ErrParseInterDir"
- else:
- """rejoin the path name"""
- status = newver[0] + newver[1]
- elif not len(fhtml):
- status = "ErrHostNoDir"
-
- f.close()
- if status != "ErrHostNoDir" and re.match("Err", status):
- logpath = d.getVar('LOG_DIR', True)
- subprocess.call("cp %s %s/" % (f.name, logpath), shell=True)
- os.unlink(f.name)
- return status
-
- """
- Check on the last directory to search '2.4.1' in "http://xxx/2.4/pkg-2.4.1.tar.gz",
- 'url' - upstream link customized by regular expression
- 'd' - database
- 'curname' - current package name
- Return new version if success, or else error in "Errxxxx" style
- """
- def check_new_version(url, curname, ud, d):
- """possible to have no version in pkg name, such as spectrum-fw"""
- if not re.search("\d+", curname):
- return pcurver
- pn = d.getVar('PN', True)
- newver_regex = d.getVar('REGEX', True)
- f = tempfile.NamedTemporaryFile(delete=False, prefix="%s-2-" % pn)
- status = internal_fetch_wget(url, ud, d, f)
- fhtml = f.read()
-
- if status == "SUCC" and len(fhtml):
- newver = parse_dir(curname)
-
- if not newver_regex:
- """this is the default matching pattern, if recipe does not """
- """provide a regex expression """
- """match "{PN}-5.21.1.tar.gz">{PN}-5.21.1.tar.gz """
- pn1 = re.search("^%s" % prefix, curname).group()
- s = "[^\"]*%s[^\d\"]*?(\d+[\.\-_])+[^\"]*" % pn1
- searchstr = "[hH][rR][eE][fF]=\"%s\".*[>\"]" % s
- reg = searchstr
- else:
- reg = newver_regex
- valid = 0
- count = 0
- for line in fhtml.split("\n"):
- if pn == 'kconfig-frontends':
- m = re.findall(reg, line)
- if m:
- valid = 1
- for match in m:
- (on, ov, oe) = newver
- ver = (on, match[0], oe)
- if ver and __vercmp(newver, ver) == True:
- newver = ver
- continue
- count += 1
- m = re.search(reg, line)
- if m:
- valid = 1
- if not newver_regex:
- ver = m.group().split("\"")[1].split("/")[-1]
- if ver == "download":
- ver = m.group().split("\"")[1].split("/")[-2]
- ver = parse_dir(ver)
- else:
- """ we cheat a little here, but we assume that the
- regular expression in the recipe will extract exacly
- the version """
- (on, ov, oe) = newver
- ver = (on, m.group('pver'), oe)
- if ver and __vercmp(newver, ver) == True:
- newver = ver
- """Expect a match for curver in directory list, or else it indicates unknown format"""
- if not valid:
- status = "ErrParseDir"
- else:
- """newver still contains a full package name string"""
- status = re.sub('_', '.', newver[1])
- elif not len(fhtml):
- status = "ErrHostNoDir"
-
- f.close()
- """if host hasn't directory information, no need to save tmp file"""
- if status != "ErrHostNoDir" and re.match("Err", status):
- logpath = d.getVar('LOG_DIR', True)
- subprocess.call("cp %s %s/" % (f.name, logpath), shell=True)
- os.unlink(f.name)
- return status
-
"""first check whether a uri is provided"""
src_uri = d.getVar('SRC_URI', True)
if not src_uri:
@@ -543,9 +309,6 @@ python do_checkpkg() {
localdata.setVar('OVERRIDES', "pn-" + pnstripped[0] + ":" + d.getVar('OVERRIDES', True))
bb.data.update_data(localdata)
- chk_uri = d.getVar('REGEX_URI', True)
- if not chk_uri:
- chk_uri = src_uri
pdesc = localdata.getVar('DESCRIPTION', True)
pgrp = localdata.getVar('SECTION', True)
if localdata.getVar('PRSPV', True):
@@ -562,232 +325,63 @@ python do_checkpkg() {
psrcuri = localdata.getVar('SRC_URI', True)
maintainer = localdata.getVar('RECIPE_MAINTAINER', True)
+ """ Get upstream version version """
+ pupver = None
+ pstatus = "ErrUnknown"
found = 0
+
for uri in src_uri.split():
- m = re.compile('(?P<type>[^:]*)').match(uri)
- if not m:
- raise MalformedUrl(uri)
- elif m.group('type') in ('http', 'https', 'ftp', 'cvs', 'svn', 'git'):
- found = 1
- pproto = m.group('type')
- break
+ m = re.compile('(?P<type>[^:]*)').match(uri)
+ if not m:
+ raise MalformedUrl(uri)
+ elif m.group('type') in ('http', 'https', 'ftp', 'cvs', 'svn', 'git'):
+ found = 1
+ psrcuri = uri
+ pproto = m.group('type')
+ break
if not found:
pproto = "file"
- pupver = "N/A"
- pstatus = "ErrUnknown"
-
- (type, host, path, user, pswd, parm) = bb.fetch.decodeurl(uri)
- if type in ['http', 'https', 'ftp']:
- if d.getVar('PRSPV', True):
- pcurver = d.getVar('PRSPV', True)
- else:
- pcurver = d.getVar('PV', True)
- else:
- if d.getVar('PRSPV', True):
- pcurver = d.getVar('PRSPV', True)
- else:
- pcurver = d.getVar("SRCREV", True)
-
-
- if type in ['http', 'https', 'ftp']:
- ud = bb.fetch2.FetchData(uri, d)
- newver = pcurver
- altpath = path
- dirver = "-"
- curname = "-"
-
- """
- match version number amid the path, such as "5.7" in:
- http://download.gnome.org/sources/${PN}/5.7/${PN}-${PV}.tar.gz
- N.B. how about sth. like "../5.7/5.8/..."? Not find such example so far :-P
- """
- m = re.search(r"[^/]*(\d+\.)+\d+([\-_]r\d+)*/", path)
- if m:
- altpath = path.split(m.group())[0]
- dirver = m.group().strip("/")
-
- """use new path and remove param. for wget only param is md5sum"""
- alturi = bb.fetch.encodeurl([type, host, altpath, user, pswd, {}])
- my_uri = d.getVar('REGEX_URI', True)
- if my_uri:
- if d.getVar('PRSPV', True):
- newver = d.getVar('PRSPV', True)
- else:
- newver = d.getVar('PV', True)
- else:
- newver = check_new_dir(alturi, dirver, ud, d)
- altpath = path
- if not re.match("Err", newver) and dirver != newver:
- altpath = altpath.replace(dirver, newver, True)
- # For folder in folder cases - try to enter the folder again and then try parsing
- """Now try to acquire all remote files in current directory"""
- if not re.match("Err", newver):
- curname = altpath.split("/")[-1]
-
- """get remote name by skipping pacakge name"""
- m = re.search(r"/.*/", altpath)
- if not m:
- altpath = "/"
- else:
- altpath = m.group()
-
- chk_uri = d.getVar('REGEX_URI', True)
- if not chk_uri:
- alturi = bb.fetch.encodeurl([type, host, altpath, user, pswd, {}])
- else:
- alturi = chk_uri
- newver = check_new_version(alturi, curname, ud, d)
- while(newver == "ErrHostNoDir"):
- if alturi == "/download":
- break
- else:
- alturi = "/".join(alturi.split("/")[0:-2]) + "/download"
- newver = check_new_version(alturi, curname, ud, d)
- if not re.match("Err", newver):
- pupver = newver
- if pupver != pcurver:
- pstatus = "UPDATE"
- else:
- pstatus = "MATCH"
-
- if re.match("Err", newver):
- pstatus = newver + ":" + altpath + ":" + dirver + ":" + curname
- elif type == 'git':
- if user:
- gituser = user + '@'
- else:
- gituser = ""
-
- if 'protocol' in parm:
- gitproto = parm['protocol']
- else:
- gitproto = "git"
- # Get all tags and HEAD
- if d.getVar('GIT_REGEX', True):
- gitcmd = "git ls-remote %s://%s%s%s %s 2>&1" % (gitproto, gituser, host, path, d.getVar('GIT_REGEX', True))
- else:
- gitcmd = "git ls-remote %s://%s%s%s *tag* 2>&1" % (gitproto, gituser, host, path)
- gitcmd2 = "git ls-remote %s://%s%s%s HEAD 2>&1" % (gitproto, gituser, host, path)
-
- tmp = os.popen(gitcmd).read()
- if 'unable to connect' in tmp:
- tmp = None
- tmp2 = os.popen(gitcmd2).read()
- if 'unable to connect' in tmp2:
- tmp2 = None
- #This is for those repos have tag like: refs/tags/1.2.2
- phash = pversion.rsplit("+")[-1]
- if tmp:
- tmpline = tmp.split("\n")
- verflag = 0
- pupver = pversion
- for line in tmpline:
- if len(line)==0:
- break;
- puptag = line.split("/")[-1]
- upstr_regex = d.getVar('REGEX', True)
- if upstr_regex:
- puptag = re.search(upstr_regex, puptag)
- else:
- puptag = re.search("(?P<pver>([0-9][\.|_]?)+)", puptag)
- if puptag == None:
- continue
- puptag = puptag.group('pver')
- puptag = re.sub("_",".",puptag)
- plocaltag = pupver.split("+git")[0]
- if "git" in plocaltag:
- plocaltag = plocaltag.split("-")[0]
- result = bb.utils.vercmp(("0", puptag, ""), ("0", plocaltag, ""))
-
- if result > 0:
- verflag = 1
- pupver = puptag
- elif verflag == 0 :
- pupver = plocaltag
- #This is for those no tag repo
- elif tmp2:
+ if pproto in ['http', 'https', 'ftp', 'git']:
+ try:
+ ud = bb.fetch2.FetchData(psrcuri, d)
+ pupver = ud.method.latest_versionstring(ud, d)
+ if pproto == 'git':
+ if pupver == "":
pupver = pversion.rsplit("+")[0]
- phash = pupver
- else:
- pstatus = "ErrGitAccess"
- if not ('ErrGitAccess' in pstatus):
-
- latest_head = tmp2.rsplit("\t")[0][:7]
- tmp3 = re.search('(?P<git_ver>(\d+[\.-]?)+)(?P<git_prefix>(\+git[r|\-|]?)AUTOINC\+)(?P<head_md5>([\w|_]+))', pversion)
- tmp4 = re.search('(?P<git_ver>(\d+[\.-]?)+)(?P<git_prefix>(\+git[r|\-|]?)AUTOINC\+)(?P<head_md5>([\w|_]+))', pupver)
- if not tmp4:
- tmp4 = re.search('(?P<git_ver>(\d+[\.-]?)+)', pupver)
-
- if tmp3:
- # Get status of the package - MATCH/UPDATE
- result = bb.utils.vercmp(("0", tmp3.group('git_ver'), ""), ("0",tmp3.group('git_ver') , ""))
- # Get the latest tag
- pstatus = 'MATCH'
- if result < 0:
- latest_pv = tmp3.group('git_ver')
- else:
- latest_pv = pupver
- if not(tmp3.group('head_md5')[:7] in latest_head) or not(latest_head in tmp3.group('head_md5')[:7]):
- pstatus = 'UPDATE'
-
- git_prefix = tmp3.group('git_prefix')
- pupver = latest_pv + tmp3.group('git_prefix') + latest_head
- else:
- if not tmp3:
- bb.plain("#DEBUG# Package %s: current version (%s) doesn't match the usual pattern" %(pname, pversion))
- elif type == 'svn':
- ud = bb.fetch2.FetchData(uri, d)
-
- svnFetcher = bb.fetch2.svn.Svn(d)
- svnFetcher.urldata_init(ud, d)
- try:
- pupver = svnFetcher.latest_revision(ud, d, ud.names[0])
- except bb.fetch2.FetchError:
- pstatus = "ErrSvnAccess"
-
- if pupver:
- if pupver in pversion:
- pstatus = "MATCH"
- else:
- pstatus = "UPDATE"
- else:
- pstatus = "ErrSvnAccess"
-
- if 'rev' in ud.parm:
- pcurver = ud.parm['rev']
-
- if pstatus != "ErrSvnAccess":
- tag = pversion.rsplit("+svn")[0]
- svn_prefix = re.search('(\+svn[r|\-]?)', pversion)
- if tag and svn_prefix:
- pupver = tag + svn_prefix.group() + pupver
-
- elif type == 'cvs':
- pupver = "HEAD"
- pstatus = "UPDATE"
- elif type == 'file':
- """local file is always up-to-date"""
- pupver = pcurver
- pstatus = "MATCH"
+ if re.search(pversion, "gitrAUTOINC"):
+ pupver += "+gitrAUTOINC+"
+ else:
+ pupver += "+gitAUTOINC+"
+ latest_revision = ud.method.latest_revision(ud, d, ud.names[0])
+ pupver += latest_revision[:10]
+ except Exception as inst:
+ bb.warn("%s: unexpected error: %s" % (pname, repr(inst)))
+ pstatus = "ErrAccess"
+ elif pproto == "file":
+ """Local files are always updated"""
+ pupver = pversion
else:
- pstatus = "ErrUnsupportedProto"
+ pstatus = "ErrUnsupportedProto"
+ bb.note("do_checkpkg, protocol %s isn't implemented" % pproto)
- if re.match("Err", pstatus):
- pstatus += ":%s%s" % (host, path)
+ if not pupver:
+ pupver = "N/A"
+ elif pupver == pversion:
+ pstatus = "MATCH"
+ else:
+ pstatus = "UPDATE"
"""Read from manual distro tracking fields as alternative"""
pmver = d.getVar("RECIPE_UPSTREAM_VERSION", True)
if not pmver:
- pmver = "N/A"
- pmstatus = "ErrNoRecipeData"
+ pmver = "N/A"
+ pmstatus = "ErrNoRecipeData"
+ elif pmver == pupver:
+ pmstatus = "MATCH"
else:
- if pmver == pcurver:
- pmstatus = "MATCH"
- else:
- pmstatus = "UPDATE"
+ pmstatus = "UPDATE"
- psrcuri = psrcuri.split()[0]
pdepends = "".join(pdepends.split("\t"))
pdesc = "".join(pdesc.split("\t"))
no_upgr_reason = d.getVar('RECIPE_NO_UPDATE_REASON', True)
--
1.9.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 00/12] Usage bitbake fetcher latest_versionstring in distrodata_class
@ 2014-11-28 1:10 ` Aníbal Limón
0 siblings, 0 replies; 16+ messages in thread
From: Aníbal Limón @ 2014-11-28 1:10 UTC (permalink / raw)
To: poky
Since Bitbake fetcher has latest_versionstring method that support search last version
of package in upstream sites, distrodata class should use it instead of do itself,
Summary of changes,
distrodata_class: Re-implement upstream version detection using Bitbake fetcher methods.
package_regex.inc: Update regex because Bitbake fetcher already provides support for search
in HTML href's tags instead of RAW lines.
bitbake: Improvements in fetcher wget latest_versionstring method based on [1] and clean code.
Compatibility testing was made running distrodata class and the result files can be found at [1].
The code has DEPENDENCIES the Bitbake changes should be MERGE first then package_regex.inc
and finally distrodata_class.
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=1813
The following changes since commit 0bc03af7ee6112fa0af0608b02f715ec8495e4ff:
maintainers.inc: update maintainers (2014-11-26 17:06:10 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib alimon/fetcher-distrodata
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=alimon/fetcher-distrodata
Aníbal Limón (12):
distrodata_class: checkpkg make usage of latest_versionstring methods
in bitbake fetcher
package_regex.inc: Update REGEX'es in order to find upstream versions
correctly
fetch2/wget.py: latest_versionstring create _init_regex method for
have one place when regex'es are defined
fetch2/wget.py: _init_regexes rename variables to be more consistent
and move dirver_regex into it
fetch/wget.php: latest version string only try to find latest
directory when REGEX_URI isn't specified to avoid unnecessary
processing and makes code easier
fetch2/wget.py: latest_versionstring add package_custom_regex_comp
fetch2/wget.py: latest_versionstring improvments in searching
fetch/wget.py: latest_versionstring remove unnecessary usage for name
in version comparision
fetch2/wget: latest_versionstring add support for search in RAW html
lines
fetch/wget.py: latest_versionstring remove newuri set because is
previous set
tests/fetch.py: Update wget latest_versionstring cups case
fetch2/wget.py: latest_versionstring clean improvments minor
bitbake/lib/bb/fetch2/wget.py | 146 ++++---
bitbake/lib/bb/tests/fetch.py | 2 +-
meta-yocto/conf/distro/include/package_regex.inc | 400 +++++++++---------
meta/classes/distrodata.bbclass | 492 ++---------------------
4 files changed, 337 insertions(+), 703 deletions(-)
--
1.9.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH 02/12] package_regex.inc: Update REGEX'es in order to find upstream versions correctly
2014-11-28 1:10 ` Aníbal Limón
` (2 preceding siblings ...)
(?)
@ 2014-11-28 1:10 ` Aníbal Limón
-1 siblings, 0 replies; 16+ messages in thread
From: Aníbal Limón @ 2014-11-28 1:10 UTC (permalink / raw)
To: poky
Since Bitbake fetcher implements latest version string methods using bs4 to parse
HTML data some regexes need update because bs4 gives functionality to get href
entries and isn't necessary put pattern [hH][rR][eE][fF] into it, this make
maintenance easier.
Compatibility testing was made running distrodata class and the result files
can be found at:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=1813
[YOCTO #1813]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
meta-yocto/conf/distro/include/package_regex.inc | 400 ++++++++++++-----------
1 file changed, 204 insertions(+), 196 deletions(-)
diff --git a/meta-yocto/conf/distro/include/package_regex.inc b/meta-yocto/conf/distro/include/package_regex.inc
index 92202fd..9bf77e7 100644
--- a/meta-yocto/conf/distro/include/package_regex.inc
+++ b/meta-yocto/conf/distro/include/package_regex.inc
@@ -7,358 +7,366 @@
#"
# The format is as a bitbake variable override for each recipe
#"
-# REGEX_URI_pn-<recipe name> = ""recipe_url"
+# REGEX_URI_pn-<recipe name> = "recipe_url"
# - This is the url used by the package checking system to
# get the latest version of the package
-# REGEX_pn-<recipe name> = ""package_regex"
+# REGEX_pn-<recipe name> = "package_regex"
# - This is the regex the package checking system uses to
# parse the page found at REGEX_URI_pn-<recipe name>
#
COMMON_REGEX = "((\d+[\.\-_]*)+)"
#This will need to be changed a little bit when versions are bigger than 9.
-REGEX_URI_pn-acpid = "http://sourceforge.net/projects/acpid/files/acpid/"
-REGEX_pn-acpid = "[hH][rR][eE][fF]=\"/projects/acpid/files/acpid/(acpid\-)?(?P<pver>(\d\.(\d+[\.\-_]*)+))/\""
+REGEX_URI_pn-acpid = "http://sourceforge.net/projects/acpid2/files/"
+REGEX_pn-acpid = "/projects/acpid2/files/(acpid\-)?(?P<pver>(\d\.(\d+[\.\-_]*)+)).tar.xz/"
REGEX_URI_pn-adt-installer = "http://code.google.com/p/opkg/downloads/list"
REGEX_URI_pn-arptables = "http://sourceforge.net/projects/ebtables/files/arptables/"
-REGEX_pn-arptables = "[hH][rR][eE][fF]=\"/projects/ebtables/files/arptables/arptables-v(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-arptables = "/projects/ebtables/files/arptables/arptables-v(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-beecrypt = "http://sourceforge.net/projects/beecrypt/files/beecrypt/"
-REGEX_pn-beecrypt = "[hH][rR][eE][fF]=\"/projects/beecrypt/files/beecrypt/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-beecrypt = "/projects/beecrypt/files/beecrypt/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-beecrypt-native = "http://sourceforge.net/projects/beecrypt/files/beecrypt/"
-REGEX_pn-beecrypt-native = "[hH][rR][eE][fF]=\"/projects/beecrypt/files/beecrypt/(?P<pver>((\d+[\.\-_]*)+))/\""
-REGEX_pn-bdwgc = "[hH][rR][eE][fF]=\"gc\-(?P<pver>((\d+[a-z]?[\.\-_]*)+))\.tar\.gz\""
+REGEX_pn-beecrypt-native = "/projects/beecrypt/files/beecrypt/(?P<pver>((\d+[\.\-_]*)+))/"
+REGEX_URI_pn-bdwgc = "http://www.hboehm.info/gc/gc_source/"
+REGEX_pn-bdwgc = "gc\-(?P<pver>((\d+[a-z]?[\.\-_]*)+))\.tar\.gz"
REGEX_URI_pn-bind = "ftp://ftp.isc.org/isc/bind9/"
-REGEX_pn-bind = "[hH][rR][eE][fF]=\"(?P<pver>((P?\d+[\.\-_]*)+))/\""
+REGEX_pn-bind = "(?P<pver>((P?\d+[\.\-_]*)+))/"
REGEX_URI_pn-bjam-native = "http://sourceforge.net/projects/boost/files/boost/"
-REGEX_pn-bjam-native = "[hH][rR][eE][fF]=\"/projects/boost/files/boost/(?P<pver>((\d+[\.\-_]*)+))/\""
-REGEX_pn-blktool = "[hH][rR][eE][fF]=\"blktool_(?P<pver>((\d+[\.\-_]*)+))\.(diff|orig\.tar)\.gz\""
+REGEX_pn-bjam-native = "/projects/boost/files/boost/(?P<pver>((\d+[\.\-_]*)+))/"
+REGEX_pn-blktool = "blktool_(?P<pver>((\d+[\.\-_]*)+))\.(diff|orig\.tar)\.gz"
REGEX_URI_pn-boost = "http://sourceforge.net/projects/boost/files/boost/"
-REGEX_pn-boost = "[hH][rR][eE][fF]=\"/projects/boost/files/boost/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-boost = "/projects/boost/files/boost/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_pn-btrfs-tools = "(?P<pver>(\d+(\.\d+)*(\-rc\d+)*))"
REGEX_pn-build-appliance-image = "(?P<pver>([0-9][\.|_]?)+)$"
REGEX_URI_pn-bzip2 = "http://www.bzip.org/downloads.html"
REGEX_pn-chkconfig-alternatives-native = "chkconfig\-(?P<pver>((\d+[\.\-_]*)+))"
REGEX_URI_pn-chrpath = "http://alioth.debian.org/frs/?group_id=31052"
-REGEX_pn-chrpath = "[hH][rR][eE][fF]=\"/frs/download.php/file/\d+/chrpath-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
-REGEX_pn-cmake = "[hH][rR][eE][fF]=\"cmake\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
-REGEX_pn-cmake-native = "[hH][rR][eE][fF]=\"cmake\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
-REGEX_pn-nativeesdk-cmake = "[hH][rR][eE][fF]=\"cmake\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
-REGEX_URI_pn-console-tools = "http://sourceforge.net/projects/lct/files/console-tools/"
-REGEX_pn-console-tools = "[hH][rR][eE][fF]=\"/projects/lct/files/console-tools/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-chrpath = "/frs/download.php/file/\d+/chrpath-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
+REGEX_pn-cmake = "cmake\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
+REGEX_pn-cmake-native = "cmake\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
+REGEX_pn-nativeesdk-cmake = "cmake\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
+REGEX_URI_pn-console-tools = "http://sourceforge.net/projects/lct/files/console-tools-devel/"
+REGEX_pn-console-tools = "/projects/lct/files/console-tools-devel/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-cracklib = "http://sourceforge.net/projects/cracklib/files/cracklib/"
-REGEX_pn-cracklib = "[hH][rR][eE][fF]=\"/projects/cracklib/files/cracklib/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-cracklib = "/projects/cracklib/files/cracklib/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-createrepo = "http://createrepo.baseurl.org/download/"
-REGEX_pn-createrepo = "[hH][rR][eE][fF]=\"createrepo\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
+REGEX_pn-createrepo = "createrepo\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
REGEX_URI_pn-createrepo-native = "http://createrepo.baseurl.org/download/"
-REGEX_pn-createrepo-native = "[hH][rR][eE][fF]=\"createrepo\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
+REGEX_pn-createrepo-native = "createrepo\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
REGEX_URI_pn-cups = "http://www.cups.org/software.php"
-REGEX_pn-cups = "<tt>cups\-(?P<pver>((\d+[\.\-_]*)+))\-source\.tar\.gz<\/tt>"
+REGEX_pn-cups = "cups\-(?P<pver>((\d+[\.\-_]*)+))\-source\.tar\.gz"
REGEX_URI_pn-curlpp = "http://code.google.com/p/curlpp/downloads/list"
REGEX_URI_pn-cwautomacros = "http://sourceforge.net/projects/cwautomacros.berlios/files/"
-REGEX_pn-cwautomacros = "[hH][rR][eE][fF]=\"http://sourceforge.net/projects/cwautomacros.berlios/files/cwautomacros\-(?P<pver>(\d+))\.tar\.bz2\/download""
+REGEX_pn-cwautomacros = "http://sourceforge.net/projects/cwautomacros.berlios/files/cwautomacros\-(?P<pver>(\d+))\.tar\.bz2\/download"
REGEX_URI_pn-cwautomacros-native = "http://sourceforge.net/projects/cwautomacros.berlios/files/"
-REGEX_pn-cwautomacros-native = "[hH][rR][eE][fF]=\"http://sourceforge.net/projects/cwautomacros.berlios/files/cwautomacros\-(?P<pver>(\d+))\.tar\.bz2\/download\""
+REGEX_pn-cwautomacros-native = "http://sourceforge.net/projects/cwautomacros.berlios/files/cwautomacros\-(?P<pver>(\d+))\.tar\.bz2\/download"
REGEX_URI_pn-db = "http://www.oracle.com/technetwork/products/berkeleydb/downloads/index-082944.html"
-REGEX_pn-db = "[hH][rR][eE][fF]=\"http://download.oracle.com/otn/berkeley-db/db-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
+REGEX_pn-db = "http://download.oracle.com/otn/berkeley-db/db-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
REGEX_URI_pn-db-native = "http://www.oracle.com/technetwork/products/berkeleydb/downloads/index-082944.html"
-REGEX_pn-db-native = "[hH][rR][eE][fF]=\"http://download.oracle.com/otn/berkeley-db/db-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
+REGEX_pn-db-native = "http://download.oracle.com/otn/berkeley-db/db-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
REGEX_URI_pn-nativesdk-db = "http://www.oracle.com/technetwork/products/berkeleydb/downloads/index-082944.html"
-REGEX_pn-nativesdk-db = "[hH][rR][eE][fF]=\"http://download.oracle.com/otn/berkeley-db/db-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
+REGEX_pn-nativesdk-db = "http://download.oracle.com/otn/berkeley-db/db-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
REGEX_URI_pn-distcc = "http://code.google.com/p/distcc/downloads/list"
-REGEX_pn-distcc = "[hH][rR][eE][fF]=\"//distcc.googlecode.com/files/distcc-(?P<pver>((\d+[\.\-_]*)+))\.tar\.bz2\""
+REGEX_pn-distcc = "//distcc.googlecode.com/files/distcc-(?P<pver>((\d+[\.\-_]*)+))\.tar\.bz2"
REGEX_URI_pn-docbook-dsssl-stylesheets-native = "http://sourceforge.net/projects/docbook/files/docbook-dsssl/"
-REGEX_pn-docbook-dsssl-stylesheets-native = "[hH][rR][eE][fF]=\"/projects/docbook/files/docbook-dsssl/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-docbook-dsssl-stylesheets-native = "/projects/docbook/files/docbook-dsssl/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-docbook-sgml-dtd-3.1-native = "http://docbook.org/sgml/3.1/"
-PRSPV_pn-docbook-sgml-dtd-3.1-native = "${@d.getVar('PV',1).replace('.','')}"
REGEX_URI_pn-docbook-sgml-dtd-4.1-native = "http://docbook.org/sgml/4.1/"
-PRSPV_pn-docbook-sgml-dtd-4.1-native = "${@d.getVar('PV',1).replace('.','')}"
REGEX_URI_pn-docbook-sgml-dtd-4.5-native = "http://docbook.org/sgml/4.5/"
REGEX_URI_pn-e2fsprogs = "http://sourceforge.net/projects/e2fsprogs/files/e2fsprogs/"
-REGEX_pn-e2fsprogs = "[hH][rR][eE][fF]=\"/projects/e2fsprogs/files/e2fsprogs/v(?P<pver>((\d+[\.\-_]*)+))/\""
-REGEX_pn-elfutils = "[hH][rR][eE][fF]=\"(elfutils\-)?(?P<pver>((\d+[\.\-_]*)+))(/|\.tar\.bz2)\""
+REGEX_pn-e2fsprogs = "/projects/e2fsprogs/files/e2fsprogs/v(?P<pver>((\d+[\.\-_]*)+))/"
+REGEX_pn-elfutils = "(elfutils\-)?(?P<pver>((\d+[\.\-_]*)+))(/|\.tar\.bz2)"
REGEX_URI_pn-expat = "http://sourceforge.net/projects/expat/files/expat/"
-REGEX_pn-expat-native = "[hH][rR][eE][fF]=\"/projects/expat/files/expat/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-expat-native = "/projects/expat/files/expat/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-expat-native = "http://sourceforge.net/projects/expat/files/expat/"
-REGEX_pn-expat = "[hH][rR][eE][fF]=\"/projects/expat/files/expat/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-expat = "/projects/expat/files/expat/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-nativesdk-expat = "http://sourceforge.net/projects/expat/files/expat/"
-REGEX_pn-nativesdk-expat = "[hH][rR][eE][fF]=\"/projects/expat/files/expat/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-nativesdk-expat = "/projects/expat/files/expat/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-fetchmail = "http://sourceforge.net/projects/fetchmail.berlios/files/"
-REGEX_URI_pn-flac = "http://sourceforge.net/projects/flac/files/flac-linux-i386/"
-REGEX_pn-flac = "[hH][rR][eE][fF]=\"/projects/flac/files/flac-linux-i386/flac\-(?P<pver>((\d+[\.\-_]*)+))\-linux\-i386/\""
REGEX_URI_pn-flex = "http://sourceforge.net/projects/flex/files/"
-REGEX_pn-flex = "[hH][rR][eE][fF]=\"http://sourceforge.net/projects/flex/files/flex-(?P<pver>((\d+[\.\-_]*)+)).tar.gz/download\""
-REGEX_pn-foomatic-filters = "[hH][rR][eE][fF]=\"foomatic-filters-(?P<pver>((\d|\d\d)\.*)+)\.tar\.gz\""
+REGEX_pn-flex = "http://sourceforge.net/projects/flex/files/flex-(?P<pver>((\d+[\.\-_]*)+)).tar.gz/download"
+REGEX_pn-foomatic-filters = "foomatic-filters-(?P<pver>((\d|\d\d)\.*)+)\.tar\.gz"
REGEX_URI_pn-fotowall = "https://code.google.com/p/fotowall/downloads/list"
-REGEX_pn-fotowall = "[hH][rR][eE][fF]=\"//fotowall.googlecode.com/files/[F|f]otowall\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.bz2\""
+REGEX_pn-fotowall = "//fotowall.googlecode.com/files/[F|f]otowall\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.bz2"
REGEX_URI_pn-freetype = "http://sourceforge.net/projects/freetype/files/freetype2"
-REGEX_pn-freetype = "[hH][rR][eE][fF]=\"/projects/freetype/files/freetype\d/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-freetype = "/projects/freetype/files/freetype\d/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-freetype-native = "http://sourceforge.net/projects/freetype/files/freetype2"
-REGEX_pn-freetype-native = "[hH][rR][eE][fF]=\"/projects/freetype/files/freetype\d/(?P<pver>((\d+[\.\-_]*)+))/\""
-REGEX_URI_pn-git = "http://code.google.com/p/git-core/downloads/list"
-REGEX_pn-git = "[hH][rR][eE][fF]=\"//git-core.googlecode.com/files/git-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
+REGEX_pn-freetype-native = "/projects/freetype/files/freetype\d/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-glew = "http://sourceforge.net/projects/glew/files/glew"
-REGEX_pn-glew = " [hH][rR][eE][fF]=\"/projects/glew/files/glew/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-glew = "/projects/glew/files/glew/(?P<pver>((\d+[\.\-_]*)+))/"
#REGEX_pn-gnutls = "ftp://ftp.gnutls.org/gcrypt/gnutls/"
-REGEX_pn-gstreamer = "[hH][rR][eE][fF]=\"gstreamer\-(?P<pver>(0\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)\""
-REGEX_pn-gstreamer1.0 = "[hH][rR][eE][fF]=\"gstreamer\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)\""
-REGEX_pn-gst-ffmpeg = "[hH][rR][eE][fF]=\"gst-ffmpeg\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)\""
-REGEX_pn-gst-fluendo-mpegdemux = "[hH][rR][eE][fF]=\"gst-fluendo-mpegdemux\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)\""
-REGEX_pn-gst-fluendo-mp3 = "[hH][rR][eE][fF]=\"gst-fluendo-mp3\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)\""
-REGEX_pn-gstreamer1.0-plugins-base = "[hH][rR][eE][fF]=\"gst\-plugins\-base\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)\""
-REGEX_pn-gst-plugins-base = "[hH][rR][eE][fF]=\"gst\-plugins\-base\-(?P<pver>(0\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)\""
-REGEX_pn-gstreamer1.0-plugins-bad = "[hH][rR][eE][fF]=\"gst\-plugins\-bad\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)\""
-REGEX_pn-gst-plugins-bad = "[hH][rR][eE][fF]=\"gst\-plugins\-bad\-(?P<pver>(0\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)\""
-REGEX_pn-gst-plugins-gl = "[hH][rR][eE][fF]=\"gst\-plugins\-gl\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)\""
-REGEX_pn-gstreamer1.0-plugins-good = "[hH][rR][eE][fF]=\"gst\-plugins\-good\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)\""
-REGEX_pn-gst-plugins-good = "[hH][rR][eE][fF]=\"gst\-plugins\-good\-(?P<pver>(0\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)\""
-REGEX_pn-gstreamer1.0-libav = "[hH][rR][eE][fF]=\"gst\-libav\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)\""
-REGEX_pn-gstreamer1.0-omx = "[hH][rR][eE][fF]=\"gst\-omx\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)\""
-REGEX_pn-gst-openmax = "[hH][rR][eE][fF]=\"gst\-openmax\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)\""
-REGEX_pn-gstreamer1.0-plugins-ugly = "[hH][rR][eE][fF]=\"gst\-plugins\-ugly\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)\""
-REGEX_pn-gst-plugins-ugly = "[hH][rR][eE][fF]=\"gst\-plugins\-ugly\-(?P<pver>(0\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)\""
-REGEX_pn-gtk+ = "[hH][rR][eE][fF]=\"gtk\+\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.xz\""
-REGEX_pn-gtk+3 = "[hH][rR][eE][fF]=\"gtk\+\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.xz\""
-REGEX_pn-gtk-update-icon-cache-native = "[hH][rR][eE][fF]=\"gtk\+\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.xz\""
-REGEX_pn-webkit-gtk = "[hH][rR][eE][fF]=\"webkitgtk\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.xz\""
+REGEX_pn-gstreamer = "gstreamer\-(?P<pver>(0\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)"
+REGEX_pn-gstreamer1.0 = "gstreamer\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)"
+REGEX_pn-gst-ffmpeg = "gst-ffmpeg\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)"
+REGEX_pn-gst-fluendo-mpegdemux = "gst-fluendo-mpegdemux\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)"
+REGEX_pn-gst-fluendo-mp3 = "gst-fluendo-mp3\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)"
+REGEX_pn-gstreamer1.0-plugins-base = "gst\-plugins\-base\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)"
+REGEX_pn-gst-plugins-base = "gst\-plugins\-base\-(?P<pver>(0\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)"
+REGEX_pn-gstreamer1.0-plugins-bad = "gst\-plugins\-bad\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)"
+REGEX_pn-gst-plugins-bad = "gst\-plugins\-bad\-(?P<pver>(0\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)"
+REGEX_pn-gst-plugins-gl = "gst\-plugins\-gl\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)"
+REGEX_pn-gstreamer1.0-plugins-good = "gst\-plugins\-good\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)"
+REGEX_pn-gst-plugins-good = "gst\-plugins\-good\-(?P<pver>(0\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)"
+REGEX_pn-gstreamer1.0-libav = "gst\-libav\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)"
+REGEX_pn-gstreamer1.0-omx = "gst\-omx\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)"
+REGEX_pn-gst-openmax = "gst\-openmax\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)"
+REGEX_pn-gstreamer1.0-plugins-ugly = "gst\-plugins\-ugly\-(?P<pver>(\d+\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)"
+REGEX_pn-gst-plugins-ugly = "gst\-plugins\-ugly\-(?P<pver>(0\.\d*[0|2|4|6|8]\.\d+))\.tar\.(gz|xz|bz2)"
+REGEX_pn-gtk+ = "gtk\+\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.xz"
+REGEX_pn-gtk+3 = "gtk\+\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.xz"
+REGEX_pn-gtk-update-icon-cache-native = "gtk\+\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.xz"
+REGEX_pn-webkit-gtk = "webkitgtk\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.xz"
REGEX_URI_pn-hdparm = "http://sourceforge.net/projects/hdparm/files/hdparm/"
-REGEX_pn-hdparm = "[hH][rR][eE][fF]=\"http://sourceforge.net/projects/hdparm/files/hdparm/hdparm-(?P<pver>((\d+[\.\-_]*)+)).tar.gz/download\""
-REGEX_URI_pn-icu = "http://site.icu-project.org/download"
-REGEX_pn-icu = "[hH][rR][eE][fF]=\"http://site.icu-project.org/download/((\d+[\.\-_]*)+)#(TOC\-)?ICU4J\-Download\" rel=\"nofollow\">(<b>)?(?P<pver>((\d+[\.\-_]*)+))(</b>)?"
+REGEX_pn-hdparm = "http://sourceforge.net/projects/hdparm/files/hdparm/hdparm-(?P<pver>((\d+[\.\-_]*)+)).tar.gz/download"
REGEX_URI_pn-intltool = "https://launchpad.net/intltool/+download"
-REGEX_pn-intltool = "[hH][rR][eE][fF]=\"https://launchpad.net/intltool/trunk/(?P<pver>((\d+[\.\-_]*)+))/\+download/intltool\-(\d+[\.\-_]*)+\.tar\.gz\""
+REGEX_pn-intltool = "https://launchpad.net/intltool/trunk/(?P<pver>((\d+[\.\-_]*)+))/\+download/intltool\-(\d+[\.\-_]*)+\.tar\.gz"
REGEX_URI_pn-intltool-native = "https://launchpad.net/intltool/+download"
-REGEX_pn-intltool-native = "[hH][rR][eE][fF]=\"https://launchpad.net/intltool/trunk/(?P<pver>((\d+[\.\-_]*)+))/\+download/intltool\-(\d+[\.\-_]*)+\.tar\.gz\""
+REGEX_pn-intltool-native = "https://launchpad.net/intltool/trunk/(?P<pver>((\d+[\.\-_]*)+))/\+download/intltool\-(\d+[\.\-_]*)+\.tar\.gz"
REGEX_URI_pn-irda-utils = "http://sourceforge.net/projects/irda/files/irda-utils/"
-REGEX_pn-irda-utils = "[hH][rR][eE][fF]=\"/projects/irda/files/irda-utils/(?P<pver>((\d+[\.\-_]*)+))/\""
-REGEX_pn-iputils = "[Hh][Rr][Ee][Ff]=\"iputils\-(?P<pver>(s\d+))\.tar\.bz2\""
+REGEX_pn-irda-utils = "/projects/irda/files/irda-utils/(?P<pver>((\d+[\.\-_]*)+))/"
+REGEX_pn-iputils = "iputils\-(?P<pver>(s\d+))\.tar\.bz2"
REGEX_URI_pn-jpeg = "http://www.ijg.org/files/"
-REGEX_pn-jpeg = "[hH][rR][eE][fF]=\"jpegsrc.v(?P<pver>((\d+[a-z]*\d*)+))\.tar\.gz""
+REGEX_pn-jpeg = "jpegsrc.v(?P<pver>((\d+[a-z]*\d*)+))\.tar\.gz"
REGEX_URI_pn-js = "http://ftp.mozilla.org/pub/mozilla.org/js/"
-REGEX_pn-js = "[hH][rR][eE][fF]=\"js-?(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
-REGEX_URI_pn-json-c = "http://s3.amazonaws.com/json-c_releases/"
-REGEX_pn-json-c = "releases/json-c-(?P<pver>(\d+[\.\-_]*)+).tar.gz"
-REGEX_URI_pn-kconfig-frontends = "http://ymorin.is-a-geek.org/download/kconfig-frontends/"
-REGEX_pn-kconfig-frontends = "[hH][rR][eE][fF]=\'kconfig\-frontends\-(?P<pver>((\d+[\.\-]*)+))\.tar\.xz\'"
-GIT_REGEX_pn-kern-tools-native = "HEEEAD"
+REGEX_pn-js = "js-?(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
+REGEX_URI_pn-json-c = "https://github.com/json-c/json-c/releases"
+REGEX_pn-json-c = "json-c-(?P<pver>(\d+[\.\-_]*)+)-\d\d\d\d\d\d\d\d\.tar\.gz"
REGEX_URI_pn-lame = "http://sourceforge.net/projects/lame/files/lame/"
-REGEX_pn-lame = "[hH][rR][eE][fF]=\"/projects/lame/files/lame/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-lame = "/projects/lame/files/lame/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-less = "http://www.greenwoodsoftware.com/less/download.html"
-REGEX_pn-less = "[hH][rR][eE][fF]=\"less\-(?P<pver>(\d+))\.tar\.gz\""
+REGEX_pn-less = "less\-(?P<pver>(\d+))\.tar\.gz"
REGEX_URI_pn-liba52 = "http://liba52.sourceforge.net/downloads.html"
-REGEX_pn-liba52 = "[hH][rR][eE][fF]=\"/files/a52\w*\-(?P<pver>((\d+[a-z]?[\.\-_]*)+))\.tar\.gz\""
+REGEX_pn-liba52 = "/files/a52\w*\-(?P<pver>((\d+[a-z]?[\.\-_]*)+))\.tar\.gz"
REGEX_URI_pn-libacpi = "http://www.ngolde.de/libacpi.html"
REGEX_URI_pn-libaio = "http://ftp.debian.org/debian/pool/main/liba/libaio/"
-REGEX_pn-libaio = "[hH][rR][eE][fF]=\"libaio_(?P<pver>((\d+[\.\-_]*)+)).orig.tar.gz\""
+REGEX_pn-libaio = "libaio_(?P<pver>((\d+[\.\-_]*)+)).orig.tar.gz"
REGEX_URI_pn-libarchive = "http://www.libarchive.org/"
-REGEX_pn-libarchive = "[hH][rR][eE][fF]=\"downloads/libarchive\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
+REGEX_pn-libarchive = "downloads/libarchive\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
REGEX_URI_pn-libcgroup = "http://sourceforge.net/projects/libcg/files/libcgroup/"
-#REGEX_pn-libcgroup = "[hH][rR][eE][fF]=\"/projects/libcg/files/libcgroup/v?(?P<pver>((\.?\d+[\.\-_]*)+(rc\d?)*)+)/\""
-REGEX_pn-libcgroup = "[hH][rR][eE][fF]=\"/projects/libcg/files/libcgroup/v?(?P<pver>((\.?\d+[\.\-_]*))+)/\""
+#REGEX_pn-libcgroup = "/projects/libcg/files/libcgroup/v?(?P<pver>((\.?\d+[\.\-_]*)+(rc\d?)*)+)/"
+REGEX_pn-libcgroup = "/projects/libcg/files/libcgroup/v?(?P<pver>((\.?\d+[\.\-_]*))+)/"
REGEX_URI_pn-libcheck = "http://sourceforge.net/projects/check/files/check/"
-REGEX_pn-libcheck = "[hH][rR][eE][fF]=\"/projects/check/files/check/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-libcheck = "/projects/check/files/check/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-libdnet = "http://code.google.com/p/libdnet/downloads/list"
#REGEX_URI_pn-libevent = "http://sourceforge.net/projects/levent/files/libevent/libevent-2.0/"
REGEX_URI_pn-libevent = "http://libevent.org/"
-#REGEX_pn-libevent = "[hH][rR][eE][fF]=\"http://sourceforge.net/projects/levent/files/libevent/libevent-2.0/libevent-(?P<pver>((\d+[\.\-_]*)+))\-stable\.tar\.gz/download\""
-REGEX_pn-libevent = "[hH][rR][eE][fF]=\"https://github.com/downloads/libevent/libevent/libevent-(?P<pver>((\d+[\.\-_]*)+))\-stable\.tar\.gz\""
+#REGEX_pn-libevent = "http://sourceforge.net/projects/levent/files/libevent/libevent-2.0/libevent-(?P<pver>((\d+[\.\-_]*)+))\-stable\.tar\.gz/download"
+REGEX_pn-libevent = "https://github.com/downloads/libevent/libevent/libevent-(?P<pver>((\d+[\.\-_]*)+))\-stable\.tar\.gz"
REGEX_URI_pn-libexif = "http://sourceforge.net/projects/libexif/files/libexif/"
-REGEX_pn-libexif = "[hH][rR][eE][fF]=\"/projects/libexif/files/libexif/(?P<pver>((\d+[\.\-_]*)+))/\""
-REGEX_pn-libffi = "[hH][rR][eE][fF]=\"libffi\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
-REGEX_pn-libffi-native = "[hH][rR][eE][fF]=\"ftp://sourceware.org:21/pub/libffi/libffi\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
-REGEX_pn-nativesdk-libffi = "[hH][rR][eE][fF]=\"ftp://sourceware.org:21/pub/libffi/libffi\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
+REGEX_pn-libexif = "/projects/libexif/files/libexif/(?P<pver>((\d+[\.\-_]*)+))/"
+REGEX_pn-libffi = "libffi\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
+REGEX_pn-libffi-native = "ftp://sourceware.org:21/pub/libffi/libffi\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
+REGEX_pn-nativesdk-libffi = "ftp://sourceware.org:21/pub/libffi/libffi\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
REGEX_URI_pn-libfm = "http://sourceforge.net/projects/pcmanfm/files/PCManFM%20%2B%20Libfm%20%28tarball%20release%29/PCManFM/"
-REGEX_pn-libfm = "[hH][rR][eE][fF]=\"http://sourceforge.net/projects/pcmanfm/files/PCManFM%20%2B%20Libfm%20%28tarball%20release%29/PCManFM/pcmanfm-(?P<pver>((\d+[\.\-_]*)+)).tar.gz/download\""
-REGEX_URI_pn-libical = "http://sourceforge.net/projects/libical/files/libical/"
-REGEX_pn-libical = "[hH][rR][eE][fF]=\"/projects/libical/files/libical/libical-(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-libfm = "pcmanfm-(?P<pver>((\d+[\.\-_]*)+)).tar.xz/download$"
+REGEX_URI_pn-libical = "https://github.com/libical/libical/releases"
+REGEX_pn-libical = "/releases/tag/v(?P<pver>((\d+[\.\-_]*)+))"
REGEX_URI_pn-libid3tag = "http://sourceforge.net/projects/mad/files/libid3tag/"
-REGEX_pn-libid3tag = "[hH][rR][eE][fF]=\"/projects/mad/files/libid3tag/(?P<pver>((\d+[\.\-_]*)+([a-z]?)))/\""
+REGEX_pn-libid3tag = "/projects/mad/files/libid3tag/(?P<pver>((\d+[\.\-_]*)+([a-z]?)))/"
REGEX_URI_pn-libmad = "http://sourceforge.net/projects/mad/files/libmad/"
-REGEX_pn-libmad = "[hH][rR][eE][fF]=\"/projects/mad/files/libmad/(?P<pver>((\d+[\.\-_]*)+([a-z]?)))/\""
+REGEX_pn-libmad = "/projects/mad/files/libmad/(?P<pver>((\d+[\.\-_]*)+([a-z]?)))/"
REGEX_URI_pn-libogg = "http://downloads.xiph.org/releases/ogg/"
-REGEX_pn-libogg = "[hH][rR][eE][fF]=\"libogg\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
+REGEX_pn-libogg = "libogg\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
REGEX_URI_pn-libmpc = "http://www.multiprecision.org/index.php?prog=mpc&page=download"
REGEX_URI_pn-libomxil = "http://sourceforge.net/projects/omxil/files/omxil/"
-REGEX_pn-libomxil = "[hH][rR][eE][fF]=\"/projects/omxil/files/omxil/Bellagio.20(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-libomxil = "/projects/omxil/files/omxil/Bellagio.20(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-libpcap = "http://www.tcpdump.org/release/"
-REGEX_pn-libpcap = "[hH][rR][eE][fF]=\"libpcap-(?P<pver>((\d+[\.\-_]*)+)).tar.gz\""
+REGEX_pn-libpcap = "libpcap-(?P<pver>((\d+[\.\-_]*)+)).tar.gz"
REGEX_URI_pn-libpcre = "http://sourceforge.net/projects/pcre/files/pcre/"
-REGEX_pn-libpcre = "[hH][rR][eE][fF]=\"/projects/pcre/files/pcre/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-libpcre = "/projects/pcre/files/pcre/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-libpcre-native = "http://sourceforge.net/projects/pcre/files/pcre/"
-REGEX_pn-libpcre-native = "[hH][rR][eE][fF]=\"/projects/pcre/files/pcre/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-libpcre-native = "/projects/pcre/files/pcre/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-libpfm4 = "http://sourceforge.net/projects/perfmon2/files/libpfm4/"
-REGEX_URI_pn-libpng = "http://sourceforge.net/projects/libpng/files/"
-REGEX_pn-libpng = "[hH][rR][eE][fF]=\"/projects/libpng/files/latest/download\?source=files\" title=\"/libpng\d+(/(\d+\.?)+/)?libpng-(?P<pver>((\d+[\.\-_]*)+)).tar.gz"
+REGEX_URI_pn-libpng = "http://sourceforge.net/projects/libpng/files/libpng16/"
+REGEX_pn-libpng = "/projects/libpng/files/libpng16/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-libpng-native = "ftp://ftp.simplesystems.org/pub/libpng/png/src/"
-REGEX_pn-libpng-native = "[hH][rR][eE][fF]=\"ftp://ftp.simplesystems.org:21/pub/libpng/png/src/libpng-(?P<pver>((\d+[\.\-_]*)+)).tar.gz""
+REGEX_pn-libpng-native = "ftp://ftp.simplesystems.org:21/pub/libpng/png/src/libpng-(?P<pver>((\d+[\.\-_]*)+)).tar.gz"
REGEX_URI_pn-libpng12 = "http://sourceforge.net/projects/libpng/files/libpng12/"
-REGEX_pn-libpng12 = "[hH][rR][eE][fF]=\"/projects/libpng/files/libpng12/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-libpng12 = "/projects/libpng/files/libpng12/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-libproxy = "http://code.google.com/p/libproxy/downloads/list"
-REGEX_pn-libtheora = "[Hh][Rr][Ee][Ff]=\"libtheora\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
+REGEX_pn-libtheora = "libtheora\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
REGEX_URI_pn-libsamplerate0 = "http://www.mega-nerd.com/SRC/download.html"
-REGEX_pn-libsamplerate0 = "[hH][rR][eE][fF]=\"libsamplerate\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
+REGEX_pn-libsamplerate0 = "libsamplerate\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
REGEX_URI_pn-libtirpc = "http://sourceforge.net/projects/libtirpc/files/libtirpc/"
-REGEX_pn-libtirpc = "[hH][rR][eE][fF]=\"/projects/libtirpc/files/libtirpc/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-libtirpc = "/projects/libtirpc/files/libtirpc/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-libusb-compat = "http://sourceforge.net/projects/libusb/files/libusb-compat-0.1/"
-REGEX_pn-libusb-compat = "[hH][rR][eE][fF]=\"/projects/libusb/files/libusb-compat-0.1/libusb-compat-(?P<pver>((\d+[\.\-_]*)+))/""
+REGEX_pn-libusb-compat = "/projects/libusb/files/libusb-compat-0.1/libusb-compat-(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-libusb1 = "http://sourceforge.net/projects/libusb/files/libusb-1.0/"
-REGEX_pn-libusb1 = "[hH][rR][eE][fF]=\"/projects/libusb/files/libusb-1.0/libusb-(?P<pver>((\d+[\.\-_]*)+))/""
+REGEX_pn-libusb1 = "/projects/libusb/files/libusb-1.0/libusb-(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-libvorbis = "http://downloads.xiph.org/releases/vorbis/"
-REGEX_pn-libvorbis = "[hH][rR][eE][fF]=\"libvorbis-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
-REGEX_pn-libxslt = "[Hh][Rr][Ee][Ff]=\"libxslt\-(?P<pver>((\d+\.*)+))\.tar\.gz\""
+REGEX_pn-libvorbis = "libvorbis-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
+REGEX_pn-libxslt = "libxslt\-(?P<pver>((\d+\.*)+))\.tar\.gz"
REGEX_pn-linux-libc-headers-yocto = "v((?P<pver>((\d+[\.\-_]*)+)))"
REGEX_URI_pn-lrzsz = "http://ohse.de/uwe/software/lrzsz.html"
REGEX_URI_pn-lsb = "http://sourceforge.net/projects/lsb/files/lsb_release/"
-REGEX_pn-lsb = "[hH][rR][eE][fF]=\"/projects/lsb/files/lsb_release/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-lsb = "/projects/lsb/files/lsb_release/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-lsbinitscripts = "http://pkgs.fedoraproject.org/repo/pkgs/initscripts/"
-REGEX_pn-lsbinitscripts = "[hH][rR][eE][fF]=\"initscripts\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.bz2/\""
+REGEX_pn-lsbinitscripts = "initscripts\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.bz2/"
REGEX_URI_pn-ltp = "http://sourceforge.net/projects/ltp/files/LTP%20Source"
-REGEX_pn-ltp = "[hH][rR][eE][fF]=\"/projects/ltp/files/LTP%20Source/ltp\-(?P<pver>(\d+))/\""
+REGEX_pn-ltp = "/projects/ltp/files/LTP%20Source/ltp\-(?P<pver>(\d+))/"
REGEX_URI_pn-lzop = "http://www.lzop.org/download/"
-REGEX_pn-lzop = "[hH][rR][eE][fF]=\"lzop\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
+REGEX_pn-lzop = "lzop\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
REGEX_URI_pn-lzop-native = "http://www.lzop.org/download/"
-REGEX_pn-lzop-native = "[hH][rR][eE][fF]=\"lzop\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
+REGEX_pn-lzop-native = "lzop\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
REGEX_URI_pn-nativesdk-lzop = "http://www.lzop.org/download/"
-REGEX_pn-nativesdk-lzop = "[hH][rR][eE][fF]=\"lzop\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
-REGEX_URI_pn-mailx = "http://sourceforge.net/projects/heirloom/files/heirloom-mailx/"
-REGEX_pn-mailx = "[hH][rR][eE][fF]=\"/projects/heirloom/files/heirloom-mailx/(?P<pver>((\d+[\.]*)+))/\""
+REGEX_pn-nativesdk-lzop = "lzop\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
REGEX_URI_pn-memcached = "http://code.google.com/p/memcached/downloads/list"
REGEX_URI_pn-menu-cache = "http://sourceforge.net/projects/lxde/files/menu-cache/"
-REGEX_pn-menu-cache = "[hH][rR][eE][fF]=\"/projects/lxde/files/menu\-cache/menu-cache%20(?P<pver>((\d+[\.\-_]*)+))/\""
-REGEX_pn-mesa = "[hH][rR][eE][fF]=\"MesaLib-(?P<pver>((\d+[\.\-_]*)+))(\-rc\d+)*.tar.gz\""
+REGEX_pn-menu-cache = "/projects/lxde/files/menu\-cache/menu-cache%20(?P<pver>((\d+[\.\-_]*)+))/"
+REGEX_pn-mesa = "MesaLib-(?P<pver>((\d+[\.\-_]*)+))(\-rc\d+)*.tar.gz"
REGEX_URI_pn-mesa-glsl-native = "ftp://ftp.freedesktop.org/pub/mesa/"
-REGEX_pn-mesa-glsl-native = "[hH][rR][eE][fF]=\"ftp://ftp.freedesktop.org:21/pub/mesa/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-mesa-glsl-native = "ftp://ftp.freedesktop.org:21/pub/mesa/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-minicom = "https://alioth.debian.org/frs/?group_id=30018"
-REGEX_pn-minicom = "[hH][rR][eE][fF]=\"/frs/download.php/file/\d+/minicom\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
+REGEX_pn-minicom = "/frs/download.php/file/\d+/minicom\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
REGEX_URI_pn-mingetty = "http://sourceforge.net/projects/mingetty/files/mingetty"
-REGEX_pn-mingetty = "[hH][rR][eE][fF]=\"/projects/mingetty/files/mingetty/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-mingetty = "/projects/mingetty/files/mingetty/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-mpeg2dec = "http://libmpeg2.sourceforge.net/downloads.html"
-REGEX_pn-mpeg2dec = "[hH][rR][eE][fF]=\"/files/mpeg2dec-(?P<pver>((\d+[\.\-_]*)+)).tar.gz\""
+REGEX_pn-mpeg2dec = "/files/mpeg2dec-(?P<pver>((\d+[\.\-_]*)+)).tar.gz"
REGEX_URI_pn-mpfr = "http://ftp.gnu.org/gnu/mpfr/"
-REGEX_pn-mpfr = "[hH][rR][eE][fF]=\"mpfr-(?P<pver>((\d+[\.\-_]*)+)).tar.gz""
+REGEX_pn-mpfr = "mpfr-(?P<pver>((\d+[\.\-_]*)+)).tar.gz"
REGEX_URI_pn-mpfr-native = "http://ftp.gnu.org/gnu/mpfr/"
-REGEX_pn-mpfr-native = "[hH][rR][eE][fF]=\"mpfr-(?P<pver>((\d+[\.\-_]*)+)).tar.gz\""
+REGEX_pn-mpfr-native = "mpfr-(?P<pver>((\d+[\.\-_]*)+)).tar.gz"
REGEX_URI_pn-msmtp = "http://sourceforge.net/projects/msmtp/files/msmtp/"
-REGEX_pn-msmtp = "[hH][rR][eE][fF]=\"/projects/msmtp/files/msmtp/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-msmtp = "/projects/msmtp/files/msmtp/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-nativesdk-mpfr = "http://ftp.gnu.org/gnu/mpfr/"
-REGEX_pn-nativesdk-mpfr = "[hH][rR][eE][fF]=\"mpfr-(?P<pver>((\d+[\.\-_]*)+)).tar.gz\""
+REGEX_pn-nativesdk-mpfr = "mpfr-(?P<pver>((\d+[\.\-_]*)+)).tar.gz"
REGEX_URI_pn-net-snmp = "http://sourceforge.net/projects/net-snmp/files/net-snmp/"
-REGEX_pn-net-snmp = "[hH][rR][eE][fF]=\"/projects/net-snmp/files/net-snmp/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-net-snmp = "/projects/net-snmp/files/net-snmp/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-netcat = "http://sourceforge.net/projects/netcat/files/netcat/"
-REGEX_pn-netcat = "[hH][rR][eE][fF]=\"/projects/netcat/files/netcat/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-netcat = "/projects/netcat/files/netcat/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-nfs-utils = "http://sourceforge.net/projects/nfs/files/nfs-utils/"
-REGEX_pn-nfs-utils = "[hH][rR][eE][fF]=\"/projects/nfs/files/nfs-utils/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-nfs-utils = "/projects/nfs/files/nfs-utils/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-ocf-linux = "http://sourceforge.net/projects/ocf-linux/files/ocf-linux/"
-REGEX_pn-ocf-linux = "[hH][rR][eE][fF]=\"/projects/ocf-linux/files/ocf-linux/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-ocf-linux = "/projects/ocf-linux/files/ocf-linux/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-openjade-native = "http://sourceforge.net/projects/openjade/files/openjade/"
-REGEX_pn-openjade-native = "[hH][rR][eE][fF]=\"/projects/openjade/files/openjade/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-openjade-native = "/projects/openjade/files/openjade/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-opensp = "http://sourceforge.net/projects/openjade/files/opensp/"
-REGEX_pn-opensp = "[hH][rR][eE][fF]=\"/projects/openjade/files/opensp/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-opensp = "/projects/openjade/files/opensp/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-opensp-native = "http://sourceforge.net/projects/openjade/files/opensp/"
-REGEX_pn-opensp-native = "[hH][rR][eE][fF]=\"/projects/openjade/files/opensp/(?P<pver>((\d+[\.\-_]*)+))/\""
-REGEX_URI_pn-opkg = "http://code.google.com/p/opkg/downloads/list"
+REGEX_pn-opensp-native = "/projects/openjade/files/opensp/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-oprofile = "http://sourceforge.net/projects/oprofile/files/oprofile/"
-REGEX_pn-oprofile = "[hH][rR][eE][fF]=\"/projects/oprofile/files/oprofile/oprofile-(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-oprofile = "/projects/oprofile/files/oprofile/oprofile-(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-ossp-uuid = "http://code.google.com/p/gnome-build-stage-1/downloads/list"
REGEX_URI_pn-pcmanfm = "http://sourceforge.net/projects/pcmanfm/files/PCManFM%20%2B%20Libfm%20%28tarball%20release%29/PCManFM/"
-REGEX_pn-pcmanfm = "[hH][rR][eE][fF]=\"http://sourceforge.net/projects/pcmanfm/files/PCManFM.20.2B.20Libfm.20.28tarball.20release.29/PCManFM/pcmanfm-(?P<pver>((\d+[\.\-_]*)+)).tar.gz/download\""
+REGEX_pn-pcmanfm = "http://sourceforge.net/projects/pcmanfm/files/PCManFM.20.2B.20Libfm.20.28tarball.20release.29/PCManFM/pcmanfm-(?P<pver>((\d+[\.\-_]*)+)).tar.gz/download"
REGEX_URI_pn-procps = "http://procps.sourceforge.net/download.html"
-REGEX_pn-procps = "[hH][rR][eE][fF]=\"procps\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
+REGEX_pn-procps = "procps\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
REGEX_URI_pn-powertop = "https://01.org/powertop/downloads"
REGEX_URI_pn-pptp-linux = "http://sourceforge.net/projects/pptpclient/files/pptp/"
-REGEX_pn-pptp-linux = "[hH][rR][eE][fF]=\"/projects/pptpclient/files/pptp/pptp-(?P<pver>((\d+[\.\-_]*)+))/\""
-GIT_REGEX_pn-prelink = "cross_prelink"
+REGEX_pn-pptp-linux = "/projects/pptpclient/files/pptp/pptp-(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_pn-prelink = "(?P<pver>cross_prelink)"
REGEX_URI_pn-psmisc = "http://sourceforge.net/projects/psmisc/files/psmisc/"
-REGEX_pn-psmisc = "[hH][rR][eE][fF]=\"http://sourceforge.net/projects/psmisc/files/psmisc/psmisc\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz/download\""
+REGEX_pn-psmisc = "http://sourceforge.net/projects/psmisc/files/psmisc/psmisc\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz/download"
REGEX_URI_pn-ptpd = "http://sourceforge.net/projects/ptpd/files/ptpd/"
-REGEX_pn-ptpd = "[hH][rR][eE][fF]=\"/projects/ptpd/files/ptpd/(?P<pver>((\d+[\.\-_]*)+))/stats/timeline\""
+REGEX_pn-ptpd = "/projects/ptpd/files/ptpd/(?P<pver>((\d+[\.\-_]*)+))/stats/timeline"
REGEX_URI_pn-python-argparse = "https://code.google.com/p/argparse/downloads/list"
REGEX_URI_pn-python-docutils = "http://sourceforge.net/projects/docutils/files/docutils/"
-REGEX_pn-python-docutils = "[hH][rR][eE][fF]=\"/projects/docutils/files/docutils/(?P<pver>((\d+[\.\-_]*)+)).*/\""
+REGEX_pn-python-docutils = "/projects/docutils/files/docutils/(?P<pver>((\d+[\.\-_]*)+)).*/"
REGEX_URI_pn-python-numpy = "http://sourceforge.net/projects/numpy/files/NumPy/"
-REGEX_pn-python-numpy = "[hH][rR][eE][fF]=\"/projects/numpy/files/latest/download\?source=files\" title=\"/NumPy/(\d+\.?)+/numpy-(?P<pver>((\d+[\.\-_]*)+)).tar.gz"
+REGEX_pn-python-numpy = "/projects/numpy/files/NumPy/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-python-pycurl = "http://pycurl.sourceforge.net/download/"
-REGEX_pn-python-pycurl = "[hH][rR][eE][fF]=\"pycurl-(?P<pver>((\d+[\.\-_]*)+)).tar.gz\""
+REGEX_pn-python-pycurl = "pycurl-(?P<pver>((\d+[\.\-_]*)+)).tar.gz"
REGEX_URI_pn-python-scons = "http://sourceforge.net/projects/scons/files/scons/"
-REGEX_pn-python-scons = "[hH][rR][eE][fF]=\"/projects/scons/files/scons/(?P<pver>((\d+[\.\-_]*)+)).*/\""
+REGEX_pn-python-scons = "/projects/scons/files/scons/(?P<pver>((\d+[\.\-_]*)+)).*/"
REGEX_URI_pn-python-scons-native = "http://sourceforge.net/projects/scons/files/scons/"
-REGEX_pn-python-scons-native = "[hH][rR][eE][fF]=\"/projects/scons/files/scons/(?P<pver>((\d+[\.\-_]*)+)).*/\""
-REGEX_pn-python-setuptools = "[hH][rR][eE][fF]=\"setuptools\-(?P<pver>((\d+([a-z]\d+)?[\.\-_]*)+))\.(tar\.gz|\.zip)\""
+REGEX_pn-python-scons-native = "/projects/scons/files/scons/(?P<pver>((\d+[\.\-_]*)+)).*/"
+REGEX_pn-python-setuptools = "setuptools\-(?P<pver>((\d+([a-z]\d+)?[\.\-_]*)+))\.(tar\.gz|\.zip)"
REGEX_URI_pn-quota = "http://sourceforge.net/projects/linuxquota/files/quota-tools/"
-REGEX_pn-quota = "[hH][rR][eE][fF]=\"/projects/linuxquota/files/quota-tools/(?P<pver>((\d+[\.\-_]*)+([-\w\d]+)))/\""
+REGEX_pn-quota = "/projects/linuxquota/files/quota-tools/(?P<pver>((\d+[\.\-_]*)+([-\w\d]+)))/"
REGEX_pn-remake = "(?P<pver>(\d+\.(\d+\.)*\d*(\+dbg\-\d+(\.\d+)*)*))"
REGEX_URI_pn-rpcbind = "http://sourceforge.net/projects/rpcbind/files/rpcbind/"
-REGEX_pn-rpcbind = "[hH][rR][eE][fF]=\"/projects/rpcbind/files/rpcbind/(?P<pver>((\d+[\.\-_]*)+))/\""
-REGEX_pn-rpm = "[hH][rR][eE][fF]=\"rpm\-(?P<pver>((\d+[\.\-_]*)+)\-(\d+[\.]*)+)\.src\.rpm\""
-REGEX_pn-rpm-native = "[hH][rR][eE][fF]=\"rpm\-(?P<pver>((\d+[\.\-_]*)+)\-(\d+[\.]*)+)\.src\.rpm\""
+REGEX_pn-rpcbind = "/projects/rpcbind/files/rpcbind/(?P<pver>((\d+[\.\-_]*)+))/"
+REGEX_pn-rpm = "rpm\-(?P<pver>((\d+[\.\-_]*)+)\-(\d+[\.]*)+)\.src\.rpm"
+REGEX_pn-rpm-native = "rpm\-(?P<pver>((\d+[\.\-_]*)+)\-(\d+[\.]*)+)\.src\.rpm"
REGEX_URI_pn-setserial = "http://sourceforge.net/projects/setserial/files/setserial/"
-REGEX_pn-setserial = "[hH][rR][eE][fF]=\"/projects/setserial/files/setserial/(?P<pver>((\d+[\.\-_]*)+))/\""
-REGEX_pn-shared-mime-info = "[hH][rR][eE][fF]=\"shared\-mime\-info\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.(bz2|gz|xz)\""
-REGEX_pn-shared-mime-info-native = "[hH][rR][eE][fF]=\"shared\-mime\-info\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.(bz2|gz|xz)\""
+REGEX_pn-setserial = "/projects/setserial/files/setserial/(?P<pver>((\d+[\.\-_]*)+))/"
+REGEX_pn-shared-mime-info = "shared\-mime\-info\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.(bz2|gz|xz)"
+REGEX_pn-shared-mime-info-native = "shared\-mime\-info\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.(bz2|gz|xz)"
REGEX_URI_pn-socat = "http://www.dest-unreach.org/socat/download/"
-REGEX_pn-socat = "[hH][rR][eE][fF]=\"socat\-(?P<pver>((\d+[\.\-]*)+))\.tar\.bz2\""
+REGEX_pn-socat = "socat\-(?P<pver>((\d+[\.\-]*)+))\.tar\.bz2"
REGEX_URI_pn-speex = "http://downloads.us.xiph.org/releases/speex/"
-REGEX_pn-speex = "[hH][rR][eE][fF]=\"speex\-(?P<pver>((\d+\.*)+))\.tar\.gz\""
+REGEX_pn-speex = "speex\-(?P<pver>((\d+\.*)+)(rc\d)?)\.tar\.gz"
REGEX_URI_pn-sqlite3 = "http://www.sqlite.org/download.html"
REGEX_pn-sqlite3 = "sqlite-autoconf-(?P<pver>(\d+)).tar.gz"
REGEX_URI_pn-sqlite3-native = "http://www.sqlite.org/download.html"
-REGEX_pn-sqlite3-native = "[hH][rR][eE][fF]=\"/sqlite-autoconf-(?P<pver>(\d+)).tar.gz\""
+REGEX_pn-sqlite3-native = "/sqlite-autoconf-(?P<pver>(\d+)).tar.gz"
REGEX_URI_pn-nativesdk-sqlite3 = "http://www.sqlite.org/download.html"
-REGEX_pn-nativesdk-sqlite3 = "[hH][rR][eE][fF]=\"/sqlite-autoconf-(?P<pver>(\d+)).tar.gz\""
+REGEX_pn-nativesdk-sqlite3 = "/sqlite-autoconf-(?P<pver>(\d+)).tar.gz"
REGEX_URI_pn-squashfs-tools = "http://sourceforge.net/projects/squashfs/files/squashfs/"
-REGEX_pn-squashfs-tools = "[hH][rR][eE][fF]=\"/projects/squashfs/files/squashfs/squashfs(?P<pver>((\d+[\.\-_]*)+)).*/\""
+REGEX_pn-squashfs-tools = "/projects/squashfs/files/squashfs/squashfs(?P<pver>((\d+[\.\-_]*)+)).*/"
REGEX_URI_pn-strace = "http://sourceforge.net/projects/strace/files/strace/"
-REGEX_pn-strace = "[hH][rR][eE][fF]=\"/projects/strace/files/strace/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-strace = "/projects/strace/files/strace/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-strace-native = "http://sourceforge.net/projects/strace/files/strace/"
-REGEX_pn-strace-native = "[hH][rR][eE][fF]=\"/projects/strace/files/strace/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-strace-native = "/projects/strace/files/strace/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-stunnel = "http://www.stunnel.org/downloads.html"
-REGEX_pn-stunnel = "[hH][rR][eE][fF]=\"downloads/stunnel-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
+REGEX_pn-stunnel = "downloads/stunnel-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
REGEX_URI_pn-sysfsutils = "http://sourceforge.net/projects/linux-diag/files/sysfsutils/"
-REGEX_pn-sysfsutils = "[hH][rR][eE][fF]=\"/projects/linux-diag/files/sysfsutils/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-sysfsutils = "/projects/linux-diag/files/sysfsutils/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-sysstat = "http://sebastien.godard.pagesperso-orange.fr/download.html"
-REGEX_pn-sysstat = "[hH][rR][eE][fF]=\"http://pagesperso-orange.fr/sebastien.godard/sysstat\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
+REGEX_pn-sysstat = "http://pagesperso-orange.fr/sebastien.godard/sysstat\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
REGEX_URI_pn-taglib = "http://github.com/taglib/taglib/releases/"
REGEX_URI_pn-tcl = "http://sourceforge.net/projects/tcl/files/Tcl/"
-REGEX_pn-tcl = "[hH][rR][eE][fF]=\"/projects/tcl/files/Tcl/(?P<pver>((\d+(b\d+)?[\.\-_]*)+))/\""
+REGEX_pn-tcl = "/projects/tcl/files/Tcl/(?P<pver>((\d+(b\d+)?[\.\-_]*)+))/"
REGEX_URI_pn-tcl-native = "http://sourceforge.net/projects/tcl/files/Tcl/"
-REGEX_pn-tcl-native = "[hH][rR][eE][fF]=\"/projects/tcl/files/Tcl/(?P<pver>((\d+(b\d+)?[\.\-_]*)+))/\""
+REGEX_pn-tcl-native = "/projects/tcl/files/Tcl/(?P<pver>((\d+(b\d+)?[\.\-_]*)+))/"
REGEX_URI_pn-tcpreplay = "http://sourceforge.net/projects/tcpreplay/files/tcpreplay/"
-REGEX_pn-tcpreplay = "[hH][rR][eE][fF]=\"/projects/tcpreplay/files/tcpreplay/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-tcpreplay = "/projects/tcpreplay/files/tcpreplay/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-tiff-native = "ftp://ftp.remotesensing.org/pub/libtiff/"
-REGEX_pn-tiff-native = "[hH][rR][eE][fF]=\"ftp://ftp.remotesensing.org:21/pub/libtiff/tiff-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
-REGEX_URI_pn-tslib = "http://sourceforge.net/projects/tslib.berlios/files/"
+REGEX_pn-tiff-native = "ftp://ftp.remotesensing.org:21/pub/libtiff/tiff-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
+REGEX_URI_pn-tslib = "https://github.com/kergoth/tslib/releases"
+REGEX_pn-tslib = "/releases/tag/(?P<pver>((\d+[\.\-_]*)+))"
REGEX_URI_pn-traceroute = "http://sourceforge.net/projects/traceroute/files/traceroute/"
-REGEX_pn-traceroute = "[hH][rR][eE][fF]=\"/projects/traceroute/files/traceroute/traceroute-(?P<pver>((\d+[\.\-_]*)+))/\""
-REGEX_pn-tslib = "[hH][rR][eE][fF]=\"http://sourceforge.net/projects/tslib.berlios/files/tslib\-(?P<pver>((\d+[\.\-_]*)+))\.tar\.bz2/download\""
+REGEX_pn-traceroute = "/projects/traceroute/files/traceroute/traceroute-(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-tunctl = "http://sourceforge.net/projects/tunctl/files/tunctl/"
-REGEX_pn-tunctl = "[hH][rR][eE][fF]=\"/projects/tunctl/files/tunctl/(?P<pver>((\d+[\.\-_]*)+))/\""
-REGEX_pn-tzdata = "[hH][rR][eE][fF]=\"tzdata(?P<pver>((\d+[a-z])+))\.tar\.gz\""
+REGEX_pn-tunctl = "/projects/tunctl/files/tunctl/(?P<pver>((\d+[\.\-_]*)+))/"
+REGEX_pn-tzdata = "tzdata(?P<pver>((\d+[a-z])+))\.tar\.gz"
REGEX_URI_pn-unzip = "http://sourceforge.net/projects/infozip/files/UnZip%206.x%20%28latest%29/UnZip%206.0/"
-REGEX_pn-unzip = "[hH][rR][eE][fF]=\"http://sourceforge.net/projects/infozip/files/UnZip%206.x%20%28latest%29/UnZip%206.0/unzip(?P<pver>(\d+))\.tar\.gz/download\""
-PRSPV_pn-unzip = "${@d.getVar('PV',1).replace('.','')}"
+REGEX_pn-unzip = "http://sourceforge.net/projects/infozip/files/UnZip%206.x%20%28latest%29/UnZip%206.0/unzip(?P<pver>(\d+))\.tar\.gz/download"
REGEX_URI_pn-unzip-native = "http://sourceforge.net/projects/infozip/files/UnZip%206.x%20%28latest%29/UnZip%206.0/"
-REGEX_pn-unzip-native = "[hH][rR][eE][fF]=\"http://sourceforge.net/projects/infozip/files/UnZip%206.x%20%28latest%29/UnZip%206.0/unzip(?P<pver>(\d+))\.tar\.gz/download\""
-PRSPV_pn-unzip-native = "${@d.getVar('PV',1).replace('.','')}"
+REGEX_pn-unzip-native = "http://sourceforge.net/projects/infozip/files/UnZip%206.x%20%28latest%29/UnZip%206.0/unzip(?P<pver>(\d+))\.tar\.gz/download"
#REGEX_URI_pn-v86d = "http://dev.gentoo.org/~spock/projects/uvesafb/archive/"
-REGEX_pn-v86d = "[hH][rR][eE][fF]=\"v86d\-(?P<pver>((\d+[\.]*)+))\.tar\.bz2\""
+REGEX_pn-v86d = "v86d\-(?P<pver>((\d+[\.]*)+))\.tar\.bz2"
REGEX_URI_pn-vblade = "http://sourceforge.net/projects/aoetools/files/vblade/"
-REGEX_pn-vblade = "[hH][rR][eE][fF]=\"/projects/aoetools/files/vblade/vblade-(?P<pver>((\d+[\.\-_]*)+))\.tgz/stats/timeline\""
+REGEX_pn-vblade = "/projects/aoetools/files/vblade/vblade-(?P<pver>((\d+[\.\-_]*)+))\.tgz/stats/timeline"
REGEX_URI_pn-vsftpd = "https://security.appspot.com/vsftpd.html"
-REGEX_pn-vsftpd = "[hH][rR][eE][fF]=\"https://security.appspot.com/downloads/vsftpd-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz\""
+REGEX_pn-vsftpd = "https://security.appspot.com/downloads/vsftpd-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
REGEX_URI_pn-watchdog = "http://sourceforge.net/projects/watchdog/files/watchdog/"
-REGEX_pn-watchdog = "[hH][rR][eE][fF]=\"/projects/watchdog/files/watchdog/(?P<pver>((\d+[\.\-_]*)+))/\""
+REGEX_pn-watchdog = "/projects/watchdog/files/watchdog/(?P<pver>((\d+[\.\-_]*)+))/"
REGEX_URI_pn-wireless-tools = "http://www.hpl.hp.com/personal/Jean_Tourrilhes/Linux/Tools.html"
-REGEX_pn-wireless-tools = "[hH][rR][eE][fF]=\"wireless_tools\.(?P<pver>(\d+))\.tar\.gz\""
+REGEX_pn-wireless-tools = "wireless_tools\.(?P<pver>(\d+))\.tar\.gz"
REGEX_URI_pn-x11vnc = "http://sourceforge.net/projects/libvncserver/files/x11vnc/"
-REGEX_pn-x11vnc = "[hH][rR][eE][fF]=\"/projects/libvncserver/files/x11vnc/(?P<pver>((\d+[\.\-_]*)+))/\""
-REGEX_pn-xdg-utils = "[hH][rR][eE][fF]=\"xdg\-utils\-(?P<pver>((\d+[\.\-_]*)+))\.(tar\.gz|tgz)\""
+REGEX_pn-x11vnc = "/projects/libvncserver/files/x11vnc/(?P<pver>((\d+[\.\-_]*)+))/"
+REGEX_pn-xdg-utils = "xdg\-utils\-(?P<pver>((\d+[\.\-_]*)+))\.(tar\.gz|tgz)"
REGEX_pn-xf86-video-omap = "(?P<pver>(\d+\.(\d\.?)*))"
REGEX_URI_pn-zip = "http://sourceforge.net/projects/infozip/files/Zip%203.x%20%28latest%29/3.0/"
-REGEX_pn-zip = "[hH][rR][eE][fF]=\"http://sourceforge.net/projects/infozip/files/Zip%203.x%20%28latest%29/3.0/zip(?P<pver>(\d+))\.tar\.gz/download\""
-PRSPV_pn-zip = "${@d.getVar('PV',1).replace('.','')}"
+REGEX_pn-zip = "http://sourceforge.net/projects/infozip/files/Zip%203.x%20%28latest%29/3.0/zip(?P<pver>(\d+))\.tar\.gz/download"
REGEX_URI_pn-zip-native = "http://sourceforge.net/projects/infozip/files/Zip%203.x%20%28latest%29/3.0/"
-REGEX_pn-zip-native = "[hH][rR][eE][fF]=\"http://sourceforge.net/projects/infozip/files/Zip%203.x%20%28latest%29/3.0/zip(?P<pver>(\d+))\.tar\.gz/download\""
-PRSPV_pn-zip-native = "${@d.getVar('PV',1).replace('.','')}"
+REGEX_pn-zip-native = "http://sourceforge.net/projects/infozip/files/Zip%203.x%20%28latest%29/3.0/zip(?P<pver>(\d+))\.tar\.gz/download"
REGEX_URI_pn-zisofs-tools-native ="http://pkgs.fedoraproject.org/repo/pkgs/zisofs-tools/"
+REGEX_pn-qt-mobility-embedded="qt-mobility-opensource-src-(?P<pver>((\d+[\.\-_]*)+)).*\.tar\.xz"
+REGEX_URI_pn-qt-mobility-embedded="http://pkgs.fedoraproject.org/repo/pkgs/qt-mobility/"
+REGEX_pn-qt-mobility-x11="qt-mobility-opensource-src-(?P<pver>((\d+[\.\-_]*)+)).*\.tar\.xz"
+REGEX_URI_pn-qt-mobility-x11="http://pkgs.fedoraproject.org/repo/pkgs/qt-mobility/"
+REGEX_pn-expect="/projects/expect/files/Expect/(?P<pver>((\d+[\.\-_]*)+))/"
+REGEX_URI_pn-expect="http://sourceforge.net/projects/expect/files/Expect/"
+REGEX_pn-gnu-efi="/projects/gnu-efi/files/gnu-efi_(?P<pver>((\d+[\.\-_]*)+).)\.orig\.tar\.gz/"
+REGEX_URI_pn-gnu-efi="http://sourceforge.net/projects/gnu-efi/files/"
+REGEX_pn-python-smartpm="/smart/\+milestone/(?P<pver>((\d+[\.\-_]*)+))"
+REGEX_URI_pn-python-smartpm="https://launchpad.net/smart/trunk/"
+REGEX_URI_pn-libatomics-ops="http://www.hpl.hp.com/research/linux/atomic_ops/download.php4"
+REGEX_URI_pn-sudo="http://www.sudo.ws"
+REGEX_pn-sudo="sudo-(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz"
+REGEX_URI_pn-docbook-xsl-stylesheets="http://sourceforge.net/projects/docbook/files/docbook-xsl/"
+REGEX_pn-docbook-xsl-stylesheets="/projects/docbook/files/docbook-xsl/(?P<pver>((\d+[\.\-_]*)+))/"
+REGEX_URI_pn-waffle="http://www.waffle-gl.org/releases.html"
+REGEX_URI_pn-qt4-x11-free="http://download.qt-project.org/official_releases/qt/4.8/"
+REGEX_pn-qt4-x11-free="(?P<pver>((\d+[\.\-_]*)+))/"
+REGEX_URI_pn-qt4-embedded="http://download.qt-project.org/official_releases/qt/4.8/"
+REGEX_pn-qt4-embedded="(?P<pver>((\d+[\.\-_]*)+))/"
+REGEX_URI_pn-dosfstools="http://daniel-baumann.ch/software/dosfstools/"
+REGEX_pn-midori="midori_(?P<pver>((\d+[\.\-_]*)+))_all_\.tar\.bz2"
+REGEX_URI_pn-midori="http://midori-browser.org/download/source/"
+REGEX_pn-icu="(?P<pver>((\d+[\.\-_]*)+))"
+REGEX_URI_pn-icu="http://download.icu-project.org/files/icu4c/"
+REGEX_URI_pn-pcmciautils="http://mirror.linux.org.au/linux/utils/kernel/pcmcia/"
# Regex used to parse tags and extract version
GITTAGREGEX_pn-build-appliance-image = "(?P<pver>(([0-9][\.|_]?)+[0-9]))"
--
1.9.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 00/12] Usage bitbake fetcher latest_versionstring in distrodata_class
@ 2014-11-28 1:10 ` Aníbal Limón
0 siblings, 0 replies; 16+ messages in thread
From: Aníbal Limón @ 2014-11-28 1:12 UTC (permalink / raw)
To: bitbake-devel
Since Bitbake fetcher has latest_versionstring method that support search last version
of package in upstream sites, distrodata class should use it instead of do itself,
Summary of changes,
distrodata_class: Re-implement upstream version detection using Bitbake fetcher methods.
package_regex.inc: Update regex because Bitbake fetcher already provides support for search
in HTML href's tags instead of RAW lines.
bitbake: Improvements in fetcher wget latest_versionstring method based on [1] and clean code.
Compatibility testing was made running distrodata class and the result files can be found at [1].
The code has DEPENDENCIES the Bitbake changes should be MERGE first then package_regex.inc
and finally distrodata_class.
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=1813
The following changes since commit 0bc03af7ee6112fa0af0608b02f715ec8495e4ff:
maintainers.inc: update maintainers (2014-11-26 17:06:10 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib alimon/fetcher-distrodata
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=alimon/fetcher-distrodata
Aníbal Limón (12):
distrodata_class: checkpkg make usage of latest_versionstring methods
in bitbake fetcher
package_regex.inc: Update REGEX'es in order to find upstream versions
correctly
fetch2/wget.py: latest_versionstring create _init_regex method for
have one place when regex'es are defined
fetch2/wget.py: _init_regexes rename variables to be more consistent
and move dirver_regex into it
fetch/wget.php: latest version string only try to find latest
directory when REGEX_URI isn't specified to avoid unnecessary
processing and makes code easier
fetch2/wget.py: latest_versionstring add package_custom_regex_comp
fetch2/wget.py: latest_versionstring improvments in searching
fetch/wget.py: latest_versionstring remove unnecessary usage for name
in version comparision
fetch2/wget: latest_versionstring add support for search in RAW html
lines
fetch/wget.py: latest_versionstring remove newuri set because is
previous set
tests/fetch.py: Update wget latest_versionstring cups case
fetch2/wget.py: latest_versionstring clean improvments minor
bitbake/lib/bb/fetch2/wget.py | 146 ++++---
bitbake/lib/bb/tests/fetch.py | 2 +-
meta-yocto/conf/distro/include/package_regex.inc | 400 +++++++++---------
meta/classes/distrodata.bbclass | 492 ++---------------------
4 files changed, 337 insertions(+), 703 deletions(-)
--
1.9.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH 03/12] fetch2/wget.py: latest_versionstring create _init_regex method for have one place when regex'es are defined
2014-11-28 1:10 ` Aníbal Limón
` (3 preceding siblings ...)
(?)
@ 2014-11-28 1:12 ` Aníbal Limón
-1 siblings, 0 replies; 16+ messages in thread
From: Aníbal Limón @ 2014-11-28 1:12 UTC (permalink / raw)
To: bitbake-devel
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
bitbake/lib/bb/fetch2/wget.py | 15 ++++++++++-----
1 file changed, 10 insertions(+), 5 deletions(-)
diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
index db5f27b..687d494 100644
--- a/bitbake/lib/bb/fetch2/wget.py
+++ b/bitbake/lib/bb/fetch2/wget.py
@@ -273,12 +273,9 @@ class Wget(FetchMethod):
if valid and version:
return re.sub('_', '.', version[1])
- def latest_versionstring(self, ud, d):
+ def _init_regexes(self):
"""
- Manipulate the URL and try to obtain the latest package version
-
- sanity check to ensure same name and type. Match as many patterns as possible
- such as:
+ Match as many patterns as possible such as:
gnome-common-2.20.0.tar.gz (most common format)
gtk+-2.90.1.tar.gz
xf86-input-synaptics-12.6.9.tar.gz
@@ -310,10 +307,18 @@ class Wget(FetchMethod):
# match name, version and archive type of a package
self.name_version_type_regex = re.compile("(?P<name>%s?)\.?v?(?P<ver>%s)(\-source)?[\.\-](?P<type>%s$)" % (self.pn_regex, version_regex, suffixlist))
+ def latest_versionstring(self, ud, d):
+ """
+ Manipulate the URL and try to obtain the latest package version
+
+ sanity check to ensure same name and type.
+ """
regex_uri = d.getVar("REGEX_URI", True)
newpath = ud.path
pupver = ""
+ self._init_regexes()
+
# search for version matches on folders inside the path, like:
# "5.7" in http://download.gnome.org/sources/${PN}/5.7/${PN}-${PV}.tar.gz
m = re.search("(?P<dirver>[^/]*(\d+\.)*\d+([\-_]r\d+)*)/", ud.path)
--
1.9.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 04/12] fetch2/wget.py: _init_regexes rename variables to be more consistent and move dirver_regex into it
2014-11-28 1:10 ` Aníbal Limón
` (4 preceding siblings ...)
(?)
@ 2014-11-28 1:12 ` Aníbal Limón
-1 siblings, 0 replies; 16+ messages in thread
From: Aníbal Limón @ 2014-11-28 1:12 UTC (permalink / raw)
To: bitbake-devel
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
bitbake/lib/bb/fetch2/wget.py | 30 ++++++++++++++++++------------
1 file changed, 18 insertions(+), 12 deletions(-)
diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
index 687d494..b6b1339 100644
--- a/bitbake/lib/bb/fetch2/wget.py
+++ b/bitbake/lib/bb/fetch2/wget.py
@@ -151,7 +151,7 @@ class Wget(FetchMethod):
Check for a new suffix type that we have never heard of before
"""
if (newsuffix):
- m = self.suffixregex.search(newsuffix)
+ m = self.suffix_regex_comp.search(newsuffix)
if not m:
bb.warn("%s has a possible unknown suffix: %s" % (newpn, newsuffix))
return False
@@ -233,7 +233,7 @@ class Wget(FetchMethod):
If error or no version, return ""
"""
valid = 0
- version = self._parse_path(self.name_version_type_regex, packagename)
+ version = self._parse_path(self.package_regex_comp, packagename)
bb.debug(3, "VersionURL: %s" % (url))
soup = BeautifulSoup(self._fetch_index(url, ud, d))
@@ -258,7 +258,7 @@ class Wget(FetchMethod):
else:
continue
else:
- newver = self._parse_path(self.name_version_type_regex, line['href'])
+ newver = self._parse_path(self.package_regex_comp, line['href'])
valid = 1
if newver and self._vercmp(version, newver) == True:
version = newver
@@ -294,18 +294,26 @@ class Wget(FetchMethod):
# a loose pattern such as for 80325-quicky-0.4.tar.gz
pn_prefix3 = "[0-9]+[\-]?[a-zA-Z]+"
# Save the Package Name (pn) Regex for use later
- self.pn_regex = "(%s|%s|%s)" % (pn_prefix1, pn_prefix2, pn_prefix3)
+ pn_regex = "(%s|%s|%s)" % (pn_prefix1, pn_prefix2, pn_prefix3)
# match version
- version_regex = "(([A-Z]*\d+[a-zA-Z]*[\.\-_]*)+)"
+ pver_regex = "(([A-Z]*\d+[a-zA-Z]*[\.\-_]*)+)"
+
+ # match arch
+ parch_regex = "\-source|_all_"
# src.rpm extension was added only for rpm package. Can be removed if the rpm
# packaged will always be considered as having to be manually upgraded
- suffixlist = "(tar\.gz|tgz|tar\.bz2|zip|xz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)"
- self.suffixregex = re.compile(suffixlist)
+ psuffix_regex = "(tar\.gz|tgz|tar\.bz2|zip|xz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)"
# match name, version and archive type of a package
- self.name_version_type_regex = re.compile("(?P<name>%s?)\.?v?(?P<ver>%s)(\-source)?[\.\-](?P<type>%s$)" % (self.pn_regex, version_regex, suffixlist))
+ self.package_regex_comp = re.compile("(?P<name>%s?)\.?v?(?P<ver>%s)(?P<arch>%s)?[\.\-](?P<type>%s$)"
+ % (pn_regex, pver_regex, parch_regex, psuffix_regex))
+ self.suffix_regex_comp = re.compile(psuffix_regex)
+
+ # search for version matches on folders inside the path, like:
+ # "5.7" in http://download.gnome.org/sources/${PN}/5.7/${PN}-${PV}.tar.gz
+ self.dirver_regex_comp = re.compile("(?P<dirver>[^/]*(\d+\.)*\d+([\-_]r\d+)*)/")
def latest_versionstring(self, ud, d):
"""
@@ -319,11 +327,9 @@ class Wget(FetchMethod):
self._init_regexes()
- # search for version matches on folders inside the path, like:
- # "5.7" in http://download.gnome.org/sources/${PN}/5.7/${PN}-${PV}.tar.gz
- m = re.search("(?P<dirver>[^/]*(\d+\.)*\d+([\-_]r\d+)*)/", ud.path)
+ m = self.dirver_regex_comp.search(ud.path)
bb.debug(3, "path = %s" % (ud.path))
- bb.debug(3, "Regex: %s" % (self.name_version_type_regex.pattern))
+ bb.debug(3, "Regex: %s" % (self.package_regex_comp.pattern))
if m and not regex_uri:
dirver = m.group('dirver')
# generate the new uri after removing version directory name
--
1.9.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 05/12] fetch/wget.php: latest version string only try to find latest directory when REGEX_URI isn't specified to avoid unnecessary processing and makes code easier
2014-11-28 1:10 ` Aníbal Limón
` (5 preceding siblings ...)
(?)
@ 2014-11-28 1:12 ` Aníbal Limón
-1 siblings, 0 replies; 16+ messages in thread
From: Aníbal Limón @ 2014-11-28 1:12 UTC (permalink / raw)
To: bitbake-devel
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
bitbake/lib/bb/fetch2/wget.py | 55 +++++++++++++++++++++++--------------------
1 file changed, 29 insertions(+), 26 deletions(-)
diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
index b6b1339..85485bf 100644
--- a/bitbake/lib/bb/fetch2/wget.py
+++ b/bitbake/lib/bb/fetch2/wget.py
@@ -197,7 +197,7 @@ class Wget(FetchMethod):
bb.debug(3, "DirURL: %s, %s" % (url, versionstring))
soup = BeautifulSoup(self._fetch_index(url, ud, d))
if not soup:
- return ""
+ return None
valid = 0
prefix = ''
@@ -225,21 +225,21 @@ class Wget(FetchMethod):
return prefix+version[1]
else:
bb.debug(3, "Not Valid")
- return ""
+ return None
- def _check_latest_version(self, url, packagename, ud, d):
+ def _check_latest_version(self, url, package, ud, d):
"""
Return the latest version of a package inside a given directory path
If error or no version, return ""
"""
valid = 0
- version = self._parse_path(self.package_regex_comp, packagename)
+ version = self._parse_path(self.package_regex_comp, package)
bb.debug(3, "VersionURL: %s" % (url))
soup = BeautifulSoup(self._fetch_index(url, ud, d))
if not soup:
- bb.debug(3, "*** %s NO SOUP" % (packagename))
- return ""
+ bb.debug(3, "*** %s NO SOUP" % (package))
+ return None
pn_regex = d.getVar('REGEX', True)
if pn_regex:
@@ -269,10 +269,12 @@ class Wget(FetchMethod):
version = ('', '', '')
if not pn_regex:
testversion = ('', '', '')
- bb.debug(3, "*** %s -> %s (TestVersion = %s)" % (packagename, version[1], testversion[1]))
+ bb.debug(3, "*** %s -> %s (TestVersion = %s)" % (package, version[1], testversion[1]))
if valid and version:
return re.sub('_', '.', version[1])
+ return None
+
def _init_regexes(self):
"""
Match as many patterns as possible such as:
@@ -321,36 +323,37 @@ class Wget(FetchMethod):
sanity check to ensure same name and type.
"""
+ package = ud.path.split("/")[-1]
regex_uri = d.getVar("REGEX_URI", True)
- newpath = ud.path
+ newpath = regex_uri or ud.path
pupver = ""
self._init_regexes()
- m = self.dirver_regex_comp.search(ud.path)
- bb.debug(3, "path = %s" % (ud.path))
- bb.debug(3, "Regex: %s" % (self.package_regex_comp.pattern))
- if m and not regex_uri:
- dirver = m.group('dirver')
- # generate the new uri after removing version directory name
- newuri = bb.fetch.encodeurl([ud.type, ud.host, ud.path.split(dirver)[0], ud.user, ud.pswd, {}])
- newversion = self._check_latest_dir(newuri, dirver, ud, d)
- if newversion and dirver != newversion:
- newpath = ud.path.replace(dirver, newversion, True)
-
- # try to acquire all remote files in current directory
- packagename = newpath.split("/")[-1] # current package name
- newpath = newpath.split(packagename)[0] or "/" # path to directory
+ if not regex_uri:
+ # generate the new uri with the appropriate latest directory
+ m = self.dirver_regex_comp.search(ud.path)
+ if m:
+ dirver = m.group('dirver')
+ newuri = bb.fetch.encodeurl([ud.type, ud.host,
+ ud.path.split(dirver)[0], ud.user, ud.pswd, {}])
+ new_dirver = self._check_latest_dir(newuri, dirver, ud, d)
+ if new_dirver and dirver != new_dirver:
+ newpath = ud.path.replace(dirver, new_dirver, True)
+
+ newpath = newpath.split(package)[0] or "/" # path to directory
+ newuri = bb.fetch.encodeurl([ud.type, ud.host, newpath, ud.user, ud.pswd, {}])
+ else:
+ newuri = newpath
# generate the new uri with the appropriate latest directory
newuri = regex_uri or bb.fetch.encodeurl([ud.type, ud.host, newpath, ud.user, ud.pswd, {}])
- newversion = self._check_latest_version(newuri, packagename, ud, d)
+ newversion = self._check_latest_version(newuri, package, ud, d)
while not newversion:
# maybe it's hiding in a download directory so try there
newuri = "/".join(newuri.split("/")[0:-2]) + "/download"
if newuri == "/download" or newuri == "http://download":
break
- newversion = self._check_latest_version(newuri, packagename, ud, d)
-
- return newversion
+ newversion = self._check_latest_version(newuri, package, ud, d)
+ return newversion or ""
--
1.9.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 06/12] fetch2/wget.py: latest_versionstring add package_custom_regex_comp
2014-11-28 1:10 ` Aníbal Limón
` (6 preceding siblings ...)
(?)
@ 2014-11-28 1:12 ` Aníbal Limón
-1 siblings, 0 replies; 16+ messages in thread
From: Aníbal Limón @ 2014-11-28 1:12 UTC (permalink / raw)
To: bitbake-devel
package_custom_regex_comp is built with the current package name and
then used to search upstream version this reduces custom regex'es in
sites that have different packages in the same directory.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
bitbake/lib/bb/fetch2/wget.py | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
index 85485bf..1a585a5 100644
--- a/bitbake/lib/bb/fetch2/wget.py
+++ b/bitbake/lib/bb/fetch2/wget.py
@@ -233,7 +233,7 @@ class Wget(FetchMethod):
If error or no version, return ""
"""
valid = 0
- version = self._parse_path(self.package_regex_comp, package)
+ version = self._parse_path(self.package_custom_regex_comp, package)
bb.debug(3, "VersionURL: %s" % (url))
soup = BeautifulSoup(self._fetch_index(url, ud, d))
@@ -258,7 +258,7 @@ class Wget(FetchMethod):
else:
continue
else:
- newver = self._parse_path(self.package_regex_comp, line['href'])
+ newver = self._parse_path(self.package_custom_regex_comp, line['href'])
valid = 1
if newver and self._vercmp(version, newver) == True:
version = newver
@@ -275,7 +275,7 @@ class Wget(FetchMethod):
return None
- def _init_regexes(self):
+ def _init_regexes(self, package):
"""
Match as many patterns as possible such as:
gnome-common-2.20.0.tar.gz (most common format)
@@ -317,6 +317,13 @@ class Wget(FetchMethod):
# "5.7" in http://download.gnome.org/sources/${PN}/5.7/${PN}-${PV}.tar.gz
self.dirver_regex_comp = re.compile("(?P<dirver>[^/]*(\d+\.)*\d+([\-_]r\d+)*)/")
+ # get current version and make custom regex for search in uri's
+ version = self._parse_path(self.package_regex_comp, package)
+ if version:
+ self.package_custom_regex_comp = re.compile(
+ "(?P<name>%s)(?P<ver>%s)(?P<arch>%s)?[\.\-](?P<type>%s)$" %
+ (version[0], pver_regex, parch_regex, psuffix_regex))
+
def latest_versionstring(self, ud, d):
"""
Manipulate the URL and try to obtain the latest package version
@@ -328,7 +335,7 @@ class Wget(FetchMethod):
newpath = regex_uri or ud.path
pupver = ""
- self._init_regexes()
+ self._init_regexes(package)
if not regex_uri:
# generate the new uri with the appropriate latest directory
--
1.9.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 07/12] fetch2/wget.py: latest_versionstring improvments in searching
2014-11-28 1:10 ` Aníbal Limón
` (7 preceding siblings ...)
(?)
@ 2014-11-28 1:12 ` Aníbal Limón
-1 siblings, 0 replies; 16+ messages in thread
From: Aníbal Limón @ 2014-11-28 1:12 UTC (permalink / raw)
To: bitbake-devel
Validate if package contain version string if not return the current
version cases for spectrum-fw and corpus recipes.
_check_latest_version return the latest version available don't
take into account the current version previous this only return
the upstream version if it greater than the current version.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
bitbake/lib/bb/fetch2/wget.py | 37 +++++++++++++++++++++++--------------
1 file changed, 23 insertions(+), 14 deletions(-)
diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
index 1a585a5..7e4f432 100644
--- a/bitbake/lib/bb/fetch2/wget.py
+++ b/bitbake/lib/bb/fetch2/wget.py
@@ -227,18 +227,18 @@ class Wget(FetchMethod):
bb.debug(3, "Not Valid")
return None
- def _check_latest_version(self, url, package, ud, d):
+ def _check_latest_version(self, url, package, current_version, ud, d):
"""
Return the latest version of a package inside a given directory path
If error or no version, return ""
"""
valid = 0
- version = self._parse_path(self.package_custom_regex_comp, package)
+ version = ('', '', '')
bb.debug(3, "VersionURL: %s" % (url))
soup = BeautifulSoup(self._fetch_index(url, ud, d))
if not soup:
- bb.debug(3, "*** %s NO SOUP" % (package))
+ bb.debug(3, "*** %s NO SOUP" % (url))
return None
pn_regex = d.getVar('REGEX', True)
@@ -248,7 +248,7 @@ class Wget(FetchMethod):
bb.debug(3, "pn_regex = '%s'" % (pn_regex.pattern))
for line in soup.find_all('a', href=True):
- newver = ('', '', '')
+ newver = None
bb.debug(3, "line = '%s'" % (line['href']))
if pn_regex:
m = pn_regex.search(line['href'])
@@ -259,17 +259,19 @@ class Wget(FetchMethod):
continue
else:
newver = self._parse_path(self.package_custom_regex_comp, line['href'])
- valid = 1
- if newver and self._vercmp(version, newver) == True:
- version = newver
+
+ if newver:
+ bb.debug(3, "Upstream version found: %s" % newver[1])
+ if valid == 0:
+ version = newver
+ valid = 1
+ elif self._vercmp(version, newver) == True:
+ version = newver
# check whether a valid package and version were found
+ bb.debug(3, "*** %s -> UpstreamVersion = %s (CurrentVersion = %s)" %
+ (package, version[1] or "N/A", current_version[1]))
- if not valid:
- version = ('', '', '')
- if not pn_regex:
- testversion = ('', '', '')
- bb.debug(3, "*** %s -> %s (TestVersion = %s)" % (package, version[1], testversion[1]))
if valid and version:
return re.sub('_', '.', version[1])
@@ -336,6 +338,11 @@ class Wget(FetchMethod):
pupver = ""
self._init_regexes(package)
+ current_version = ('', d.getVar('PV', True), '')
+
+ """possible to have no version in pkg name, such as spectrum-fw"""
+ if not re.search("\d+", package):
+ return re.sub('_', '.', current_version[1])
if not regex_uri:
# generate the new uri with the appropriate latest directory
@@ -355,12 +362,14 @@ class Wget(FetchMethod):
# generate the new uri with the appropriate latest directory
newuri = regex_uri or bb.fetch.encodeurl([ud.type, ud.host, newpath, ud.user, ud.pswd, {}])
- newversion = self._check_latest_version(newuri, package, ud, d)
+ newversion = self._check_latest_version(newuri, package,
+ current_version, ud, d)
while not newversion:
# maybe it's hiding in a download directory so try there
newuri = "/".join(newuri.split("/")[0:-2]) + "/download"
if newuri == "/download" or newuri == "http://download":
break
- newversion = self._check_latest_version(newuri, package, ud, d)
+ newversion = self._check_latest_version(newuri, package,
+ current_version, ud, d)
return newversion or ""
--
1.9.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 08/12] fetch/wget.py: latest_versionstring remove unnecessary usage for name in version comparision
2014-11-28 1:10 ` Aníbal Limón
` (8 preceding siblings ...)
(?)
@ 2014-11-28 1:12 ` Aníbal Limón
-1 siblings, 0 replies; 16+ messages in thread
From: Aníbal Limón @ 2014-11-28 1:12 UTC (permalink / raw)
To: bitbake-devel
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
bitbake/lib/bb/fetch2/wget.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
index 7e4f432..20e8df1 100644
--- a/bitbake/lib/bb/fetch2/wget.py
+++ b/bitbake/lib/bb/fetch2/wget.py
@@ -253,8 +253,8 @@ class Wget(FetchMethod):
if pn_regex:
m = pn_regex.search(line['href'])
if m:
- bb.debug(3, "Name = '%s', Pver = '%s'" % (m.group('name'), m.group('pver')))
- newver = (m.group('name'), m.group('pver'), '')
+ bb.debug(3, "Pver = '%s'" % (m.group('pver')))
+ newver = ('', m.group('pver'), '')
else:
continue
else:
--
1.9.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 09/12] fetch2/wget: latest_versionstring add support for search in RAW html lines
2014-11-28 1:10 ` Aníbal Limón
` (9 preceding siblings ...)
(?)
@ 2014-11-28 1:12 ` Aníbal Limón
-1 siblings, 0 replies; 16+ messages in thread
From: Aníbal Limón @ 2014-11-28 1:12 UTC (permalink / raw)
To: bitbake-devel
Some upstream sites put the name of the package in the body of href tags,
i.e. <a href="#43">somepackage-v1.4.10.tar.gz </a>.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
bitbake/lib/bb/fetch2/wget.py | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
index 20e8df1..f5bdfc7 100644
--- a/bitbake/lib/bb/fetch2/wget.py
+++ b/bitbake/lib/bb/fetch2/wget.py
@@ -256,9 +256,14 @@ class Wget(FetchMethod):
bb.debug(3, "Pver = '%s'" % (m.group('pver')))
newver = ('', m.group('pver'), '')
else:
- continue
+ m = pn_regex.search(str(line))
+ if m:
+ bb.debug(3, "Pver = '%s'" % (m.group('pver')))
+ newver = ('', m.group('pver'), '')
else:
newver = self._parse_path(self.package_custom_regex_comp, line['href'])
+ if not newver:
+ newver = self._parse_path(self.package_custom_regex_comp, str(line))
if newver:
bb.debug(3, "Upstream version found: %s" % newver[1])
--
1.9.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 10/12] fetch/wget.py: latest_versionstring remove newuri set because is previous set
2014-11-28 1:10 ` Aníbal Limón
` (10 preceding siblings ...)
(?)
@ 2014-11-28 15:37 ` Aníbal Limón
-1 siblings, 0 replies; 16+ messages in thread
From: Aníbal Limón @ 2014-11-28 15:37 UTC (permalink / raw)
To: bitbake-devel
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
bitbake/lib/bb/fetch2/wget.py | 2 --
1 file changed, 2 deletions(-)
diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
index f5bdfc7..7841553 100644
--- a/bitbake/lib/bb/fetch2/wget.py
+++ b/bitbake/lib/bb/fetch2/wget.py
@@ -365,8 +365,6 @@ class Wget(FetchMethod):
else:
newuri = newpath
- # generate the new uri with the appropriate latest directory
- newuri = regex_uri or bb.fetch.encodeurl([ud.type, ud.host, newpath, ud.user, ud.pswd, {}])
newversion = self._check_latest_version(newuri, package,
current_version, ud, d)
while not newversion:
--
1.9.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 11/12] tests/fetch.py: Update wget latest_versionstring cups case
2014-11-28 1:10 ` Aníbal Limón
` (11 preceding siblings ...)
(?)
@ 2014-11-28 15:37 ` Aníbal Limón
-1 siblings, 0 replies; 16+ messages in thread
From: Aníbal Limón @ 2014-11-28 15:37 UTC (permalink / raw)
To: bitbake-devel
Update test case for cups is needed because match only 2.0.0
versions see VERSION=2\.0\.0 in the previous string.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
bitbake/lib/bb/tests/fetch.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py
index 9d45743..3f80c4a 100644
--- a/bitbake/lib/bb/tests/fetch.py
+++ b/bitbake/lib/bb/tests/fetch.py
@@ -619,7 +619,7 @@ class FetchMethodTest(FetcherTest):
("xserver-xorg", "http://xorg.freedesktop.org/releases/individual/xserver/xorg-server-1.15.1.tar.bz2", "", "")
: "1.15.1",
# packages with valid REGEX_URI and REGEX
- ("cups", "http://www.cups.org/software/1.7.2/cups-1.7.2-source.tar.bz2", "http://www.cups.org/software.php", "/software\.php\?VERSION=2\.0\.0&FILE=2\.0\.0/(?P<name>cups\-)(?P<pver>((\d+[\.\-_]*)+))\-source\.tar\.gz")
+ ("cups", "http://www.cups.org/software/1.7.2/cups-1.7.2-source.tar.bz2", "http://www.cups.org/software.php", "(?P<name>cups\-)(?P<pver>((\d+[\.\-_]*)+))\-source\.tar\.gz")
: "2.0.0",
("db", "http://download.oracle.com/berkeley-db/db-5.3.21.tar.gz", "http://www.oracle.com/technetwork/products/berkeleydb/downloads/index-082944.html", "http://download.oracle.com/otn/berkeley-db/(?P<name>db-)(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz")
: "6.1.19",
--
1.9.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 12/12] fetch2/wget.py: latest_versionstring clean improvments minor
2014-11-28 1:10 ` Aníbal Limón
` (12 preceding siblings ...)
(?)
@ 2014-11-28 15:37 ` Aníbal Limón
-1 siblings, 0 replies; 16+ messages in thread
From: Aníbal Limón @ 2014-11-28 15:37 UTC (permalink / raw)
To: bitbake-devel
Update documentation strings in _check_latest_dir and _check_latest_version
methods with the correct return types.
_check_latest_version method remove unused testversion variable.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
bitbake/lib/bb/fetch2/wget.py | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
index 7841553..d82e78c 100644
--- a/bitbake/lib/bb/fetch2/wget.py
+++ b/bitbake/lib/bb/fetch2/wget.py
@@ -192,7 +192,7 @@ class Wget(FetchMethod):
def _check_latest_dir(self, url, versionstring, ud, d):
"""
Return the name of the directory with the greatest package version
- If error or no version, return ""
+ If error or no version, return None
"""
bb.debug(3, "DirURL: %s, %s" % (url, versionstring))
soup = BeautifulSoup(self._fetch_index(url, ud, d))
@@ -230,7 +230,7 @@ class Wget(FetchMethod):
def _check_latest_version(self, url, package, current_version, ud, d):
"""
Return the latest version of a package inside a given directory path
- If error or no version, return ""
+ If error or no version, return None
"""
valid = 0
version = ('', '', '')
@@ -243,7 +243,6 @@ class Wget(FetchMethod):
pn_regex = d.getVar('REGEX', True)
if pn_regex:
- testversion = version
pn_regex = re.compile(pn_regex)
bb.debug(3, "pn_regex = '%s'" % (pn_regex.pattern))
--
1.9.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH 00/12] Usage bitbake fetcher latest_versionstring in distrodata_class
2014-11-28 1:10 ` Aníbal Limón
` (13 preceding siblings ...)
(?)
@ 2014-12-02 22:52 ` Saul Wold
-1 siblings, 0 replies; 16+ messages in thread
From: Saul Wold @ 2014-12-02 22:52 UTC (permalink / raw)
To: Aníbal Limón, openembedded-core
On 11/27/2014 05:09 PM, Aníbal Limón wrote:
> Since Bitbake fetcher has latest_versionstring method that support search last version
> of package in upstream sites, distrodata class should use it instead of do itself,
>
> Summary of changes,
>
> distrodata_class: Re-implement upstream version detection using Bitbake fetcher methods.
> package_regex.inc: Update regex because Bitbake fetcher already provides support for search
> in HTML href's tags instead of RAW lines.
> bitbake: Improvements in fetcher wget latest_versionstring method based on [1] and clean code.
>
> Compatibility testing was made running distrodata class and the result files can be found at [1].
>
> The code has DEPENDENCIES the Bitbake changes should be MERGE first then package_regex.inc
> and finally distrodata_class.
>
> [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=1813
>
> The following changes since commit 0bc03af7ee6112fa0af0608b02f715ec8495e4ff:
>
> maintainers.inc: update maintainers (2014-11-26 17:06:10 +0000)
>
> are available in the git repository at:
>
> git://git.yoctoproject.org/poky-contrib alimon/fetcher-distrodata
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=alimon/fetcher-distrodata
>
> Aníbal Limón (12):
> distrodata_class: checkpkg make usage of latest_versionstring methods
> in bitbake fetcher
> package_regex.inc: Update REGEX'es in order to find upstream versions
> correctly
> fetch2/wget.py: latest_versionstring create _init_regex method for
> have one place when regex'es are defined
> fetch2/wget.py: _init_regexes rename variables to be more consistent
> and move dirver_regex into it
> fetch/wget.php: latest version string only try to find latest
> directory when REGEX_URI isn't specified to avoid unnecessary
> processing and makes code easier
> fetch2/wget.py: latest_versionstring add package_custom_regex_comp
> fetch2/wget.py: latest_versionstring improvments in searching
> fetch/wget.py: latest_versionstring remove unnecessary usage for name
> in version comparision
> fetch2/wget: latest_versionstring add support for search in RAW html
> lines
> fetch/wget.py: latest_versionstring remove newuri set because is
> previous set
> tests/fetch.py: Update wget latest_versionstring cups case
> fetch2/wget.py: latest_versionstring clean improvments minor
>
> bitbake/lib/bb/fetch2/wget.py | 146 ++++---
> bitbake/lib/bb/tests/fetch.py | 2 +-
> meta-yocto/conf/distro/include/package_regex.inc | 400 +++++++++---------
> meta/classes/distrodata.bbclass | 492 ++---------------------
> 4 files changed, 337 insertions(+), 703 deletions(-)
>
I know you are using the create-pull-request script, but you should to
it once per repo that you are doing it with, for example here in bitbake
repo and a second one in the oe-core repo, so that does not people
looking for the other 11 patches in oe-core vs bitbake.
Sau!
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2014-12-02 22:52 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-11-28 1:09 [PATCH 00/12] Usage bitbake fetcher latest_versionstring in distrodata_class Aníbal Limón
2014-11-28 1:12 ` Aníbal Limón
2014-11-28 1:10 ` Aníbal Limón
2014-11-28 1:09 ` [PATCH 01/12] distrodata_class: checkpkg make usage of latest_versionstring methods in bitbake fetcher Aníbal Limón
2014-11-28 1:10 ` [PATCH 02/12] package_regex.inc: Update REGEX'es in order to find upstream versions correctly Aníbal Limón
2014-11-28 1:12 ` [PATCH 03/12] fetch2/wget.py: latest_versionstring create _init_regex method for have one place when regex'es are defined Aníbal Limón
2014-11-28 1:12 ` [PATCH 04/12] fetch2/wget.py: _init_regexes rename variables to be more consistent and move dirver_regex into it Aníbal Limón
2014-11-28 1:12 ` [PATCH 05/12] fetch/wget.php: latest version string only try to find latest directory when REGEX_URI isn't specified to avoid unnecessary processing and makes code easier Aníbal Limón
2014-11-28 1:12 ` [PATCH 06/12] fetch2/wget.py: latest_versionstring add package_custom_regex_comp Aníbal Limón
2014-11-28 1:12 ` [PATCH 07/12] fetch2/wget.py: latest_versionstring improvments in searching Aníbal Limón
2014-11-28 1:12 ` [PATCH 08/12] fetch/wget.py: latest_versionstring remove unnecessary usage for name in version comparision Aníbal Limón
2014-11-28 1:12 ` [PATCH 09/12] fetch2/wget: latest_versionstring add support for search in RAW html lines Aníbal Limón
2014-11-28 15:37 ` [PATCH 10/12] fetch/wget.py: latest_versionstring remove newuri set because is previous set Aníbal Limón
2014-11-28 15:37 ` [PATCH 11/12] tests/fetch.py: Update wget latest_versionstring cups case Aníbal Limón
2014-11-28 15:37 ` [PATCH 12/12] fetch2/wget.py: latest_versionstring clean improvments minor Aníbal Limón
2014-12-02 22:52 ` [PATCH 00/12] Usage bitbake fetcher latest_versionstring in distrodata_class Saul Wold
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.