* [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl
@ 2013-01-21 9:17 Olof Johansson
2013-01-21 9:17 ` [PATCH 1/6] fetch2: Remove unused code in wget fetcher Olof Johansson
` (13 more replies)
0 siblings, 14 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-21 9:17 UTC (permalink / raw)
To: bitbake-devel
Hi,
In our early work in migrating to poky we realized that bitbake's current SSH
fetcher doesn't really suite our needs. It suffers from some specification
conformance bugs --- some which may be easily worked around. However, this
patch series instead introduces a new "SFTP" fetcher, meant to in a way,
replace the SSH fetcher, with stricter conformance to the SECSH URI spec [1].
The SSH fetcher has its own URI parsing, and can not rely on
bb.fetch2.decodeurl. I found that decodeurl was hard to extend, with it
returning a tuple. This prompted me to write a generic URI class, with
attributes for the various URI components (and some general bitbake kludges,
like support relative file:// uris).
I realize that these changes are a bit bold and non-trivial, but I hope you'll
find time to review them and point out problems to fix. If my reasoning is
completely wrong, please let me know and I can try to solve our direct problems
with the SSH fetcher with minimal changes instead.
The changes come with unittests for the URI class; it passes all the tests in
bb.tests.fetch and all other tests under bb.tests, except
bb.tests.data.DataExpansions, but the same errors are present on master as
well. I've also tested running bitbake core-image-minimal in a new tree without
any issues.
Olof Johansson (6):
fetch2: Remove unused code in wget fetcher
fetch2: Add a class representing a generic URI
fetch2: unittests for bb.fetch2.URI class
fetch2: Adapt encode/decode url to use URI class
fetch2: Add editor modelines for bb.tests.*
fetch2: Add SFTP fetcher
lib/bb/fetch2/__init__.py | 262 +++++++++++++++++++++++++++++++++++---------
lib/bb/fetch2/sftp.py | 104 ++++++++++++++++++
lib/bb/fetch2/wget.py | 5 -
lib/bb/tests/codeparser.py | 2 +
lib/bb/tests/cow.py | 2 +
lib/bb/tests/data.py | 2 +
lib/bb/tests/fetch.py | 228 ++++++++++++++++++++++++++++++++++++++
lib/bb/tests/utils.py | 2 +
8 files changed, 553 insertions(+), 54 deletions(-)
create mode 100644 lib/bb/fetch2/sftp.py
References:
1: http://tools.ietf.org/html/draft-ietf-secsh-scp-sftp-ssh-uri-04
--
Olof Johansson
Tools engineer
(IRC: zibri on Freenode)
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH 1/6] fetch2: Remove unused code in wget fetcher
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
@ 2013-01-21 9:17 ` Olof Johansson
2013-01-21 9:17 ` [PATCH 2/6] fetch2: Add a class representing a generic URI Olof Johansson
` (12 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-21 9:17 UTC (permalink / raw)
To: bitbake-devel
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/fetch2/wget.py | 5 -----
1 file changed, 5 deletions(-)
diff --git lib/bb/fetch2/wget.py lib/bb/fetch2/wget.py
index 2808df6..1fafc4a 100644
--- lib/bb/fetch2/wget.py
+++ lib/bb/fetch2/wget.py
@@ -32,8 +32,6 @@ import urllib
from bb import data
from bb.fetch2 import FetchMethod
from bb.fetch2 import FetchError
-from bb.fetch2 import encodeurl
-from bb.fetch2 import decodeurl
from bb.fetch2 import logger
from bb.fetch2 import runfetchcmd
@@ -77,9 +75,6 @@ class Wget(FetchMethod):
fetchcmd = d.getVar("FETCHCOMMAND_wget", True) or d.expand(basecmd + " -P ${DL_DIR} '${URI}'")
uri = uri.split(";")[0]
- uri_decoded = list(decodeurl(uri))
- uri_type = uri_decoded[0]
- uri_host = uri_decoded[1]
fetchcmd = fetchcmd.replace("${URI}", uri.split(";")[0])
fetchcmd = fetchcmd.replace("${FILE}", ud.basename)
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 2/6] fetch2: Add a class representing a generic URI
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
2013-01-21 9:17 ` [PATCH 1/6] fetch2: Remove unused code in wget fetcher Olof Johansson
@ 2013-01-21 9:17 ` Olof Johansson
2013-01-21 9:18 ` [PATCH 3/6] fetch2: unittests for bb.fetch2.URI class Olof Johansson
` (11 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-21 9:17 UTC (permalink / raw)
To: bitbake-devel
A class representing a generic URI, with methods for accessing the URI
components, and stringifies to the URI. This class should be a bit more
flexible than the existing {encode,decode}_url functions in that it
supports more components (e.g. port) and that it does not rely on a
specific order on the return values. This makes it easy to add new
properties without affecting the API.
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/fetch2/__init__.py | 194 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 194 insertions(+)
diff --git lib/bb/fetch2/__init__.py lib/bb/fetch2/__init__.py
index 150dc3c..3b83c0c 100644
--- lib/bb/fetch2/__init__.py
+++ lib/bb/fetch2/__init__.py
@@ -30,6 +30,7 @@ from __future__ import print_function
import os, re
import logging
import urllib
+from urlparse import urlparse
import bb.persist_data, bb.utils
import bb.checksum
from bb import data
@@ -119,6 +120,199 @@ class NonLocalMethod(Exception):
def __init__(self):
Exception.__init__(self)
+
+class URI(object):
+ """
+ A class representing a generic URI, with methods for
+ accessing the URI components, and stringifies to the
+ URI.
+
+ It is constructed by calling it with a URI, or setting
+ the attributes manually:
+
+ uri = URI("http://example.com/")
+
+ uri = URI()
+ uri.scheme = 'http'
+ uri.hostname = 'example.com'
+ uri.path = '/'
+
+ It has the following attributes:
+
+ * scheme (read/write)
+ * userinfo (authentication information) (read/write)
+ * username (read/write)
+ * password (read/write)
+
+ Note, password is deprecated as of RFC 3986.
+
+ * hostname (read/write)
+ * port (read/write)
+ * hostport (read only)
+ "hostname:port", if both are set, otherwise just "hostname"
+ * path (read/write)
+ * path_quoted (read/write)
+ A URI quoted version of path
+ * params (dict) (read/write)
+ * relative (bool) (read only)
+ True if this is a "relative URI", (e.g. file:foo.diff)
+
+ It stringifies to the URI itself.
+
+ Some notes about relative URIs: while it's specified that
+ a URI beginning with <scheme>:// should either be directly
+ followed by a hostname or a /, the old URI handling of the
+ fetch2 library did not comform to this. Therefore, this URI
+ class has some kludges to make sure that URIs are parsed in
+ a way comforming to bitbake's current usage. This URI class
+ supports the following:
+
+ file:relative/path.diff (IETF compliant)
+ git:relative/path.git (IETF compliant)
+ git:///absolute/path.git (IETF compliant)
+ file:///absolute/path.diff (IETF compliant)
+
+ file://relative/path.diff (not IETF compliant)
+
+ But it does not support the following:
+
+ file://hostname/absolute/path.diff (would be IETF compliant)
+
+ Note that the last case only applies to a list of
+ "whitelisted" schemes (currently only file://), that requires
+ its URIs to not have a network location.
+ """
+
+ _relative_schemes = ['file', 'git']
+ _netloc_forbidden = ['file']
+
+ def __init__(self, uri=None):
+ self.scheme = ''
+ self.userinfo = ''
+ self.hostname = ''
+ self.port = None
+ self._path = ''
+ self.params = {}
+ self.relative = False
+
+ if not uri:
+ return
+
+ urlp = urlparse(uri)
+ self.scheme = urlp.scheme
+
+ # Convert URI to be relative
+ if urlp.scheme in self._netloc_forbidden:
+ uri = re.sub("(?<=:)//(?!/)", "", uri, 1)
+ urlp = urlparse(uri)
+
+ # Identify if the URI is relative or not
+ if urlp.scheme in self._relative_schemes and \
+ re.compile("^\w+:(?!//)").match(uri):
+ self.relative = True
+
+ if not self.relative:
+ self.hostname = urlp.hostname or ''
+ self.port = urlp.port
+
+ self.userinfo += urlp.username or ''
+
+ if urlp.password:
+ self.userinfo += ':%s' % urlp.password
+
+ # Do support params even for URI schemes that Python's
+ # urlparse doesn't support params for.
+ path = ''
+ param_str = ''
+ if not urlp.params:
+ path, param_str = (list(urlp.path.split(";", 1)) + [None])[:2]
+ else:
+ path = urlp.path
+ param_str = urlp.params
+
+ self.path = urllib.unquote(path)
+
+ if param_str:
+ self.params = self._param_dict(param_str)
+
+ def __str__(self):
+ userinfo = self.userinfo
+ if userinfo:
+ userinfo += '@'
+
+ return "%s:%s%s%s%s%s" % (
+ self.scheme,
+ '' if self.relative else '//',
+ userinfo,
+ self.hostport,
+ self.path_quoted,
+ self._param_str)
+
+ @property
+ def _param_str(self):
+ ret = ''
+ for key, val in self.params.items():
+ ret += ";%s=%s" % (key, val)
+ return ret
+
+ def _param_dict(self, param_str):
+ parm = {}
+
+ for keyval in param_str.split(";"):
+ key, val = keyval.split("=", 1)
+ parm[key] = val
+
+ return parm
+
+ @property
+ def hostport(self):
+ if not self.port:
+ return self.hostname
+ return "%s:%d" % (self.hostname, self.port)
+
+ @property
+ def path_quoted(self):
+ return urllib.quote(self.path)
+
+ @path_quoted.setter
+ def path_quoted(self, path):
+ self.path = urllib.unquote(path)
+
+ @property
+ def path(self):
+ return self._path
+
+ @path.setter
+ def path(self, path):
+ self._path = path
+
+ if re.compile("^/").match(path):
+ self.relative = False
+ else:
+ self.relative = True
+
+ @property
+ def username(self):
+ if self.userinfo:
+ return (self.userinfo.split(":", 1))[0]
+ return ''
+
+ @username.setter
+ def username(self, username):
+ self.userinfo = username
+ if self.password:
+ self.userinfo += ":%s" % self.password
+
+ @property
+ def password(self):
+ if self.userinfo and ":" in self.userinfo:
+ return (self.userinfo.split(":", 1))[1]
+ return ''
+
+ @password.setter
+ def password(self, password):
+ self.userinfo = "%s:%s" % (self.username, password)
+
def decodeurl(url):
"""Decodes an URL into the tokens (scheme, network location, path,
user, password, parameters).
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 3/6] fetch2: unittests for bb.fetch2.URI class
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
2013-01-21 9:17 ` [PATCH 1/6] fetch2: Remove unused code in wget fetcher Olof Johansson
2013-01-21 9:17 ` [PATCH 2/6] fetch2: Add a class representing a generic URI Olof Johansson
@ 2013-01-21 9:18 ` Olof Johansson
2013-01-21 9:18 ` [PATCH 4/6] fetch2: Adapt encode/decode url to use URI class Olof Johansson
` (10 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-21 9:18 UTC (permalink / raw)
To: bitbake-devel
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/tests/fetch.py | 226 +++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 226 insertions(+)
diff --git lib/bb/tests/fetch.py lib/bb/tests/fetch.py
index 67eda59..e765ca7 100644
--- lib/bb/tests/fetch.py
+++ lib/bb/tests/fetch.py
@@ -21,8 +21,234 @@ import unittest
import tempfile
import subprocess
import os
+from bb.fetch2 import URI
import bb
+class URITest(unittest.TestCase):
+ test_uris = {
+ "http://www.google.com/index.html" : {
+ 'uri': 'http://www.google.com/index.html',
+ 'scheme': 'http',
+ 'hostname': 'www.google.com',
+ 'port': None,
+ 'hostport': 'www.google.com',
+ 'path': '/index.html',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': False
+ },
+ "http://www.google.com/index.html;param1=value1" : {
+ 'uri': 'http://www.google.com/index.html;param1=value1',
+ 'scheme': 'http',
+ 'hostname': 'www.google.com',
+ 'port': None,
+ 'hostport': 'www.google.com',
+ 'path': '/index.html',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {
+ 'param1': 'value1'
+ },
+ 'relative': False
+ },
+ "http://www.example.com:8080/index.html" : {
+ 'uri': 'http://www.example.com:8080/index.html',
+ 'scheme': 'http',
+ 'hostname': 'www.example.com',
+ 'port': 8080,
+ 'hostport': 'www.example.com:8080',
+ 'path': '/index.html',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': False
+ },
+ "cvs://anoncvs@cvs.handhelds.org/cvs;module=familiar/dist/ipkg" : {
+ 'uri': 'cvs://anoncvs@cvs.handhelds.org/cvs;module=familiar/dist/ipkg',
+ 'scheme': 'cvs',
+ 'hostname': 'cvs.handhelds.org',
+ 'port': None,
+ 'hostport': 'cvs.handhelds.org',
+ 'path': '/cvs',
+ 'userinfo': 'anoncvs',
+ 'username': 'anoncvs',
+ 'password': '',
+ 'params': {
+ 'module': 'familiar/dist/ipkg'
+ },
+ 'relative': False
+ },
+ "cvs://anoncvs:anonymous@cvs.handhelds.org/cvs;tag=V0-99-81;module=familiar/dist/ipkg": {
+ 'uri': 'cvs://anoncvs:anonymous@cvs.handhelds.org/cvs;tag=V0-99-81;module=familiar/dist/ipkg',
+ 'scheme': 'cvs',
+ 'hostname': 'cvs.handhelds.org',
+ 'port': None,
+ 'hostport': 'cvs.handhelds.org',
+ 'path': '/cvs',
+ 'userinfo': 'anoncvs:anonymous',
+ 'username': 'anoncvs',
+ 'password': 'anonymous',
+ 'params': {
+ 'tag': 'V0-99-81',
+ 'module': 'familiar/dist/ipkg'
+ },
+ 'relative': False
+ },
+ "file://example.diff": { # NOTE: Not RFC compliant!
+ 'uri': 'file:example.diff',
+ 'scheme': 'file',
+ 'hostname': '',
+ 'port': None,
+ 'hostport': '',
+ 'path': 'example.diff',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': True
+ },
+ "file:example.diff": { # NOTE: RFC compliant version of the former
+ 'uri': 'file:example.diff',
+ 'scheme': 'file',
+ 'hostname': '',
+ 'port': None,
+ 'hostport': '',
+ 'path': 'example.diff',
+ 'userinfo': '',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': True
+ },
+ "file:///tmp/example.diff": {
+ 'uri': 'file:///tmp/example.diff',
+ 'scheme': 'file',
+ 'hostname': '',
+ 'port': None,
+ 'hostport': '',
+ 'path': '/tmp/example.diff',
+ 'userinfo': '',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': False
+ },
+ "git:///path/example.git": {
+ 'uri': 'git:///path/example.git',
+ 'scheme': 'git',
+ 'hostname': '',
+ 'port': None,
+ 'hostport': '',
+ 'path': '/path/example.git',
+ 'userinfo': '',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': False
+ },
+ "git:path/example.git": {
+ 'uri': 'git:path/example.git',
+ 'scheme': 'git',
+ 'hostname': '',
+ 'port': None,
+ 'hostport': '',
+ 'path': 'path/example.git',
+ 'userinfo': '',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': True
+ },
+ "git://example.net/path/example.git": {
+ 'uri': 'git://example.net/path/example.git',
+ 'scheme': 'git',
+ 'hostname': 'example.net',
+ 'port': None,
+ 'hostport': 'example.net',
+ 'path': '/path/example.git',
+ 'userinfo': '',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': False
+ }
+ }
+
+ def test_uri(self):
+ for test_uri, ref in self.test_uris.items():
+ uri = URI(test_uri)
+
+ self.assertEqual(str(uri), ref['uri'])
+
+ # expected attributes
+ self.assertEqual(uri.scheme, ref['scheme'])
+
+ self.assertEqual(uri.userinfo, ref['userinfo'])
+ self.assertEqual(uri.username, ref['username'])
+ self.assertEqual(uri.password, ref['password'])
+
+ self.assertEqual(uri.hostname, ref['hostname'])
+ self.assertEqual(uri.port, ref['port'])
+ self.assertEqual(uri.hostport, ref['hostport'])
+
+ self.assertEqual(uri.path, ref['path'])
+ self.assertEqual(uri.params, ref['params'])
+
+ self.assertEqual(uri.relative, ref['relative'])
+
+ def test_dict(self):
+ for test in self.test_uris.values():
+ uri = URI()
+
+ self.assertEqual(uri.scheme, '')
+ self.assertEqual(uri.userinfo, '')
+ self.assertEqual(uri.username, '')
+ self.assertEqual(uri.password, '')
+ self.assertEqual(uri.hostname, '')
+ self.assertEqual(uri.port, None)
+ self.assertEqual(uri.path, '')
+ self.assertEqual(uri.params, {})
+
+
+ uri.scheme = test['scheme']
+ self.assertEqual(uri.scheme, test['scheme'])
+
+ uri.userinfo = test['userinfo']
+ self.assertEqual(uri.userinfo, test['userinfo'])
+ self.assertEqual(uri.username, test['username'])
+ self.assertEqual(uri.password, test['password'])
+
+ uri.hostname = test['hostname']
+ self.assertEqual(uri.hostname, test['hostname'])
+ self.assertEqual(uri.hostport, test['hostname'])
+
+ uri.port = test['port']
+ self.assertEqual(uri.port, test['port'])
+ self.assertEqual(uri.hostport, test['hostport'])
+
+ uri.path = test['path']
+ self.assertEqual(uri.path, test['path'])
+
+ uri.params = test['params']
+ self.assertEqual(uri.params, test['params'])
+
+ self.assertEqual(str(uri)+str(uri.relative), str(test['uri'])+str(test['relative']))
+
+ self.assertEqual(str(uri), test['uri'])
+
+ uri.params = {}
+ self.assertEqual(uri.params, {})
+ self.assertEqual(str(uri), (str(uri).split(";"))[0])
+
class FetcherTest(unittest.TestCase):
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 4/6] fetch2: Adapt encode/decode url to use URI class
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (2 preceding siblings ...)
2013-01-21 9:18 ` [PATCH 3/6] fetch2: unittests for bb.fetch2.URI class Olof Johansson
@ 2013-01-21 9:18 ` Olof Johansson
2013-01-21 9:18 ` [PATCH 5/6] fetch2: Add editor modelines for bb.tests.* Olof Johansson
` (9 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-21 9:18 UTC (permalink / raw)
To: bitbake-devel
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/fetch2/__init__.py | 66 ++++++++++++---------------------------------
1 file changed, 17 insertions(+), 49 deletions(-)
diff --git lib/bb/fetch2/__init__.py lib/bb/fetch2/__init__.py
index 3b83c0c..549e261 100644
--- lib/bb/fetch2/__init__.py
+++ lib/bb/fetch2/__init__.py
@@ -318,40 +318,9 @@ def decodeurl(url):
user, password, parameters).
"""
- m = re.compile('(?P<type>[^:]*)://((?P<user>.+)@)?(?P<location>[^;]+)(;(?P<parm>.*))?').match(url)
- if not m:
- raise MalformedUrl(url)
-
- type = m.group('type')
- location = m.group('location')
- if not location:
- raise MalformedUrl(url)
- user = m.group('user')
- parm = m.group('parm')
-
- locidx = location.find('/')
- if locidx != -1 and type.lower() != 'file':
- host = location[:locidx]
- path = location[locidx:]
- else:
- host = ""
- path = location
- if user:
- m = re.compile('(?P<user>[^:]+)(:?(?P<pswd>.*))').match(user)
- if m:
- user = m.group('user')
- pswd = m.group('pswd')
- else:
- user = ''
- pswd = ''
-
- p = {}
- if parm:
- for s in parm.split(';'):
- s1, s2 = s.split('=')
- p[s1] = s2
-
- return type, host, urllib.unquote(path), user, pswd, p
+ urlo = URI(url)
+ return (urlo.scheme, urlo.hostport, urlo.path,
+ urlo.username, urlo.password, urlo.params)
def encodeurl(decoded):
"""Encodes a URL from tokens (scheme, network location, path,
@@ -360,27 +329,26 @@ def encodeurl(decoded):
type, host, path, user, pswd, p = decoded
+ urlo = URI()
+
if not path:
raise MissingParameterError('path', "encoded from the data %s" % str(decoded))
if not type:
raise MissingParameterError('type', "encoded from the data %s" % str(decoded))
- url = '%s://' % type
- if user and type != "file":
- url += "%s" % user
- if pswd:
- url += ":%s" % pswd
- url += "@"
- if host and type != "file":
- url += "%s" % host
- # Standardise path to ensure comparisons work
- while '//' in path:
- path = path.replace("//", "/")
- url += "%s" % urllib.quote(path)
+
+ urlo.scheme = type
+ urlo.path = path
+
+ if host:
+ urlo.hostname = host
+ if user:
+ urlo.username = user
+ if pswd:
+ urlo.password = pswd
if p:
- for parm in p:
- url += ";%s=%s" % (parm, p[parm])
+ urlo.params = p
- return url
+ return str(urlo)
def uri_replace(ud, uri_find, uri_replace, replacements, d):
if not ud.url or not uri_find or not uri_replace:
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 5/6] fetch2: Add editor modelines for bb.tests.*
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (3 preceding siblings ...)
2013-01-21 9:18 ` [PATCH 4/6] fetch2: Adapt encode/decode url to use URI class Olof Johansson
@ 2013-01-21 9:18 ` Olof Johansson
2013-01-21 9:18 ` [PATCH 6/6] fetch2: Add SFTP fetcher Olof Johansson
` (8 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-21 9:18 UTC (permalink / raw)
To: bitbake-devel
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/tests/codeparser.py | 2 ++
lib/bb/tests/cow.py | 2 ++
lib/bb/tests/data.py | 2 ++
lib/bb/tests/fetch.py | 2 ++
lib/bb/tests/utils.py | 2 ++
5 files changed, 10 insertions(+)
diff --git lib/bb/tests/codeparser.py lib/bb/tests/codeparser.py
index 9b2d588..d2bd4a2 100644
--- lib/bb/tests/codeparser.py
+++ lib/bb/tests/codeparser.py
@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Test for codeparser.py
#
diff --git lib/bb/tests/cow.py lib/bb/tests/cow.py
index 599b2aa..35c5841 100644
--- lib/bb/tests/cow.py
+++ lib/bb/tests/cow.py
@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for Copy-on-Write (cow.py)
#
diff --git lib/bb/tests/data.py lib/bb/tests/data.py
index b2ab8ee..5161586 100644
--- lib/bb/tests/data.py
+++ lib/bb/tests/data.py
@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for the Data Store (data.py/data_smart.py)
#
diff --git lib/bb/tests/fetch.py lib/bb/tests/fetch.py
index e765ca7..778d94f 100644
--- lib/bb/tests/fetch.py
+++ lib/bb/tests/fetch.py
@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for the Fetcher (fetch2/)
#
diff --git lib/bb/tests/utils.py lib/bb/tests/utils.py
index 9f60491..7c50b1d 100644
--- lib/bb/tests/utils.py
+++ lib/bb/tests/utils.py
@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for utils.py
#
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 6/6] fetch2: Add SFTP fetcher
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (4 preceding siblings ...)
2013-01-21 9:18 ` [PATCH 5/6] fetch2: Add editor modelines for bb.tests.* Olof Johansson
@ 2013-01-21 9:18 ` Olof Johansson
2013-01-28 14:13 ` Martin Jansa
2013-01-28 15:43 ` Martin Jansa
2013-01-28 14:05 ` [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Martin Jansa
` (7 subsequent siblings)
13 siblings, 2 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-21 9:18 UTC (permalink / raw)
To: bitbake-devel
This fetcher differs from the SSH fetcher in that it adheres more
strictly to the SECSH URI internet draft --- it uses the sftp://
instead of the ssh:// scheme, and it uses sftp instead of scp.
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/fetch2/__init__.py | 2 +
lib/bb/fetch2/sftp.py | 104 +++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 106 insertions(+)
create mode 100644 lib/bb/fetch2/sftp.py
diff --git lib/bb/fetch2/__init__.py lib/bb/fetch2/__init__.py
index 549e261..86302f7 100644
--- lib/bb/fetch2/__init__.py
+++ lib/bb/fetch2/__init__.py
@@ -1448,6 +1448,7 @@ from . import svn
from . import wget
from . import svk
from . import ssh
+from . import sftp
from . import perforce
from . import bzr
from . import hg
@@ -1461,6 +1462,7 @@ methods.append(git.Git())
methods.append(cvs.Cvs())
methods.append(svk.Svk())
methods.append(ssh.SSH())
+methods.append(sftp.SFTP())
methods.append(perforce.Perforce())
methods.append(bzr.Bzr())
methods.append(hg.Hg())
diff --git lib/bb/fetch2/sftp.py lib/bb/fetch2/sftp.py
new file mode 100644
index 0000000..0902ce7
--- /dev/null
+++ lib/bb/fetch2/sftp.py
@@ -0,0 +1,104 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake SFTP Fetch implementation
+
+Class for fetching files via SFTP. It tries to adhere to the (now
+expired) IETF Internet Draft for "Uniform Resource Identifier (URI)
+Scheme for Secure File Transfer Protocol (SFTP) and Secure Shell
+(SSH)" (SECSH URI).
+
+It uses SFTP (as to adhere to the SECSH URI specification). It only
+supports key based authentication, not password. This class, unlike
+the SSH fetcher, does not support fetching a directory tree from the
+remote.
+
+ http://tools.ietf.org/html/draft-ietf-secsh-scp-sftp-ssh-uri-04
+ https://www.iana.org/assignments/uri-schemes/prov/sftp
+ https://tools.ietf.org/html/draft-ietf-secsh-filexfer-13
+
+"""
+
+# Copyright (C) 2013, Olof Johansson <olof.johansson@axis.com>
+#
+# Based in part on bb.fetch2.wget:
+# Copyright (C) 2003, 2004 Chris Larson
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# Based on functions from the base bb module, Copyright 2003 Holger Schurig
+
+import os
+import bb
+import urllib
+import commands
+from bb import data
+from bb.fetch2 import URI
+from bb.fetch2 import FetchMethod
+from bb.fetch2 import runfetchcmd
+
+
+class SFTP(FetchMethod):
+ """Class to fetch urls via 'sftp'"""
+ def supports(self, url, ud, d):
+ """
+ Check to see if a given url can be fetched with sftp.
+ """
+ return ud.type in ['sftp']
+
+ def recommends_checksum(self, urldata):
+ return True
+
+ def urldata_init(self, ud, d):
+ if 'protocol' in ud.parm and ud.parm['protocol'] == 'git':
+ raise bb.fetch2.ParameterError(
+ "Invalid protocol - if you wish to fetch from a git " +
+ "repository using ssh, you need to instead use the " +
+ "git:// prefix with protocol=ssh", ud.url)
+
+ if 'downloadfilename' in ud.parm:
+ ud.basename = ud.parm['downloadfilename']
+ else:
+ ud.basename = os.path.basename(ud.path)
+
+ ud.localfile = data.expand(urllib.unquote(ud.basename), d)
+
+ def download(self, uri, ud, d):
+ """Fetch urls"""
+
+ urlo = URI(uri)
+ basecmd = 'sftp -oPasswordAuthentication=no'
+ port = ''
+ if urlo.port:
+ port = '-P %d' % urlo.port
+ urlo.port = None
+
+ dldir = data.getVar('DL_DIR', d, True)
+ lpath = os.path.join(dldir, ud.localfile)
+
+ user = ''
+ if urlo.userinfo:
+ user = urlo.userinfo + '@'
+
+ remote = '%s%s:%s' % (user, urlo.hostname, urlo.path)
+
+ cmd = '%s %s %s %s' % (basecmd, port, commands.mkarg(remote),
+ commands.mkarg(lpath))
+
+ bb.fetch2.check_network_access(d, cmd, uri)
+ runfetchcmd(cmd, d)
+ return True
+
+ def checkstatus(self, uri, ud, d):
+ return self.download(uri, ud, d, True)
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (5 preceding siblings ...)
2013-01-21 9:18 ` [PATCH 6/6] fetch2: Add SFTP fetcher Olof Johansson
@ 2013-01-28 14:05 ` Martin Jansa
2013-01-28 14:23 ` Olof Johansson
2013-01-29 7:50 ` [PATCH v2 0/6] fetch2: Updated SFTP fetcher Olof Johansson
` (6 subsequent siblings)
13 siblings, 1 reply; 23+ messages in thread
From: Martin Jansa @ 2013-01-28 14:05 UTC (permalink / raw)
To: Olof Johansson; +Cc: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 3181 bytes --]
On Mon, Jan 21, 2013 at 10:17:57AM +0100, Olof Johansson wrote:
> Hi,
>
> In our early work in migrating to poky we realized that bitbake's current SSH
> fetcher doesn't really suite our needs. It suffers from some specification
> conformance bugs --- some which may be easily worked around. However, this
> patch series instead introduces a new "SFTP" fetcher, meant to in a way,
> replace the SSH fetcher, with stricter conformance to the SECSH URI spec [1].
>
> The SSH fetcher has its own URI parsing, and can not rely on
> bb.fetch2.decodeurl. I found that decodeurl was hard to extend, with it
> returning a tuple. This prompted me to write a generic URI class, with
> attributes for the various URI components (and some general bitbake kludges,
> like support relative file:// uris).
>
> I realize that these changes are a bit bold and non-trivial, but I hope you'll
> find time to review them and point out problems to fix. If my reasoning is
> completely wrong, please let me know and I can try to solve our direct problems
> with the SSH fetcher with minimal changes instead.
>
> The changes come with unittests for the URI class; it passes all the tests in
> bb.tests.fetch and all other tests under bb.tests, except
> bb.tests.data.DataExpansions, but the same errors are present on master as
> well. I've also tested running bitbake core-image-minimal in a new tree without
> any issues.
>
Can you please rebase your patchset on latest master?
2nd patch conflicts with
commit 22bd19d208f0251f5a1f9b98f3cac66181f3fc07
Author: Tyler Hall <tylerwhall@gmail.com>
Date: Wed Dec 19 18:26:50 2012 -0500
fetch2: Sort file checksums by value, not path
Are those patches generated from bitbake repo?
They start with 'lib/' so patch -p0 is needed instead of default -p1 and
cannot be applied directly from pw-am.sh :/.
Or please provide URL to your bitbake branch.
Cheers,
> Olof Johansson (6):
> fetch2: Remove unused code in wget fetcher
> fetch2: Add a class representing a generic URI
> fetch2: unittests for bb.fetch2.URI class
> fetch2: Adapt encode/decode url to use URI class
> fetch2: Add editor modelines for bb.tests.*
> fetch2: Add SFTP fetcher
>
> lib/bb/fetch2/__init__.py | 262 +++++++++++++++++++++++++++++++++++---------
> lib/bb/fetch2/sftp.py | 104 ++++++++++++++++++
> lib/bb/fetch2/wget.py | 5 -
> lib/bb/tests/codeparser.py | 2 +
> lib/bb/tests/cow.py | 2 +
> lib/bb/tests/data.py | 2 +
> lib/bb/tests/fetch.py | 228 ++++++++++++++++++++++++++++++++++++++
> lib/bb/tests/utils.py | 2 +
> 8 files changed, 553 insertions(+), 54 deletions(-)
> create mode 100644 lib/bb/fetch2/sftp.py
>
>
> References:
> 1: http://tools.ietf.org/html/draft-ietf-secsh-scp-sftp-ssh-uri-04
>
> --
> Olof Johansson
> Tools engineer
> (IRC: zibri on Freenode)
>
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
--
Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 6/6] fetch2: Add SFTP fetcher
2013-01-21 9:18 ` [PATCH 6/6] fetch2: Add SFTP fetcher Olof Johansson
@ 2013-01-28 14:13 ` Martin Jansa
2013-01-28 14:20 ` Olof Johansson
2013-01-28 15:43 ` Martin Jansa
1 sibling, 1 reply; 23+ messages in thread
From: Martin Jansa @ 2013-01-28 14:13 UTC (permalink / raw)
To: Olof Johansson; +Cc: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 5632 bytes --]
On Mon, Jan 21, 2013 at 10:18:03AM +0100, Olof Johansson wrote:
> This fetcher differs from the SSH fetcher in that it adheres more
> strictly to the SECSH URI internet draft --- it uses the sftp://
> instead of the ssh:// scheme, and it uses sftp instead of scp.
>
> Signed-off-by: Olof Johansson <olof.johansson@axis.com>
> ---
> lib/bb/fetch2/__init__.py | 2 +
> lib/bb/fetch2/sftp.py | 104 +++++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 106 insertions(+)
> create mode 100644 lib/bb/fetch2/sftp.py
>
> diff --git lib/bb/fetch2/__init__.py lib/bb/fetch2/__init__.py
> index 549e261..86302f7 100644
> --- lib/bb/fetch2/__init__.py
> +++ lib/bb/fetch2/__init__.py
> @@ -1448,6 +1448,7 @@ from . import svn
> from . import wget
> from . import svk
> from . import ssh
> +from . import sftp
> from . import perforce
> from . import bzr
> from . import hg
> @@ -1461,6 +1462,7 @@ methods.append(git.Git())
> methods.append(cvs.Cvs())
> methods.append(svk.Svk())
> methods.append(ssh.SSH())
> +methods.append(sftp.SFTP())
> methods.append(perforce.Perforce())
> methods.append(bzr.Bzr())
> methods.append(hg.Hg())
> diff --git lib/bb/fetch2/sftp.py lib/bb/fetch2/sftp.py
> new file mode 100644
> index 0000000..0902ce7
> --- /dev/null
> +++ lib/bb/fetch2/sftp.py
> @@ -0,0 +1,104 @@
> +# ex:ts=4:sw=4:sts=4:et
> +# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +"""
> +BitBake SFTP Fetch implementation
> +
> +Class for fetching files via SFTP. It tries to adhere to the (now
> +expired) IETF Internet Draft for "Uniform Resource Identifier (URI)
> +Scheme for Secure File Transfer Protocol (SFTP) and Secure Shell
> +(SSH)" (SECSH URI).
> +
> +It uses SFTP (as to adhere to the SECSH URI specification). It only
> +supports key based authentication, not password. This class, unlike
> +the SSH fetcher, does not support fetching a directory tree from the
> +remote.
> +
> + http://tools.ietf.org/html/draft-ietf-secsh-scp-sftp-ssh-uri-04
> + https://www.iana.org/assignments/uri-schemes/prov/sftp
> + https://tools.ietf.org/html/draft-ietf-secsh-filexfer-13
> +
> +"""
> +
> +# Copyright (C) 2013, Olof Johansson <olof.johansson@axis.com>
> +#
> +# Based in part on bb.fetch2.wget:
> +# Copyright (C) 2003, 2004 Chris Larson
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License version 2 as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with this program; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +#
> +# Based on functions from the base bb module, Copyright 2003 Holger Schurig
> +
> +import os
> +import bb
> +import urllib
> +import commands
> +from bb import data
> +from bb.fetch2 import URI
> +from bb.fetch2 import FetchMethod
> +from bb.fetch2 import runfetchcmd
> +
> +
> +class SFTP(FetchMethod):
> + """Class to fetch urls via 'sftp'"""
> + def supports(self, url, ud, d):
> + """
> + Check to see if a given url can be fetched with sftp.
> + """
> + return ud.type in ['sftp']
> +
> + def recommends_checksum(self, urldata):
> + return True
> +
> + def urldata_init(self, ud, d):
> + if 'protocol' in ud.parm and ud.parm['protocol'] == 'git':
> + raise bb.fetch2.ParameterError(
> + "Invalid protocol - if you wish to fetch from a git " +
> + "repository using ssh, you need to instead use the " +
extra 'instead' ^ ?
But I like this error, will add the same to ssh.py fetcher (which I'm
now trying to fix).
> + "git:// prefix with protocol=ssh", ud.url)
> +
> + if 'downloadfilename' in ud.parm:
> + ud.basename = ud.parm['downloadfilename']
> + else:
> + ud.basename = os.path.basename(ud.path)
> +
> + ud.localfile = data.expand(urllib.unquote(ud.basename), d)
> +
> + def download(self, uri, ud, d):
> + """Fetch urls"""
> +
> + urlo = URI(uri)
> + basecmd = 'sftp -oPasswordAuthentication=no'
> + port = ''
> + if urlo.port:
> + port = '-P %d' % urlo.port
> + urlo.port = None
> +
> + dldir = data.getVar('DL_DIR', d, True)
> + lpath = os.path.join(dldir, ud.localfile)
> +
> + user = ''
> + if urlo.userinfo:
> + user = urlo.userinfo + '@'
> +
> + remote = '%s%s:%s' % (user, urlo.hostname, urlo.path)
> +
> + cmd = '%s %s %s %s' % (basecmd, port, commands.mkarg(remote),
> + commands.mkarg(lpath))
> +
> + bb.fetch2.check_network_access(d, cmd, uri)
> + runfetchcmd(cmd, d)
> + return True
> +
> + def checkstatus(self, uri, ud, d):
> + return self.download(uri, ud, d, True)
> --
> 1.7.10.4
>
>
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
--
Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 6/6] fetch2: Add SFTP fetcher
2013-01-28 14:13 ` Martin Jansa
@ 2013-01-28 14:20 ` Olof Johansson
0 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-28 14:20 UTC (permalink / raw)
To: Martin Jansa; +Cc: bitbake-devel@lists.openembedded.org
On 2013-01-28 15:13, Martin Jansa wrote:
> On Mon, Jan 21, 2013 at 10:18:03AM +0100, Olof Johansson wrote:
> > + raise bb.fetch2.ParameterError(
> > + "Invalid protocol - if you wish to fetch from a git " +
> > + "repository using ssh, you need to instead use the " +
>
> extra 'instead' ^ ?
>
> But I like this error, will add the same to ssh.py fetcher (which I'm
> now trying to fix).
That is shamelessly stolen from the wget fetcher :-). The
"instead" may be redundant, can remove it.
--
Olof Johansson
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl
2013-01-28 14:05 ` [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Martin Jansa
@ 2013-01-28 14:23 ` Olof Johansson
0 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-28 14:23 UTC (permalink / raw)
To: Martin Jansa; +Cc: bitbake-devel@lists.openembedded.org
On 2013-01-28 15:05, Martin Jansa wrote:
> Can you please rebase your patchset on latest master?
>
> 2nd patch conflicts with
> commit 22bd19d208f0251f5a1f9b98f3cac66181f3fc07
> Author: Tyler Hall <tylerwhall@gmail.com>
> Date: Wed Dec 19 18:26:50 2012 -0500
> fetch2: Sort file checksums by value, not path
>
> Are those patches generated from bitbake repo?
> They start with 'lib/' so patch -p0 is needed instead of default -p1 and
> cannot be applied directly from pw-am.sh :/.
Sorry, will fix, but not until later this evening unfortunately.
(I've set `git config diff.noprefix=true` without realizing the
implications in cases like this. Will remove it.)
--
Olof Johansson
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 6/6] fetch2: Add SFTP fetcher
2013-01-21 9:18 ` [PATCH 6/6] fetch2: Add SFTP fetcher Olof Johansson
2013-01-28 14:13 ` Martin Jansa
@ 2013-01-28 15:43 ` Martin Jansa
1 sibling, 0 replies; 23+ messages in thread
From: Martin Jansa @ 2013-01-28 15:43 UTC (permalink / raw)
To: Olof Johansson; +Cc: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 5870 bytes --]
On Mon, Jan 21, 2013 at 10:18:03AM +0100, Olof Johansson wrote:
> This fetcher differs from the SSH fetcher in that it adheres more
> strictly to the SECSH URI internet draft --- it uses the sftp://
> instead of the ssh:// scheme, and it uses sftp instead of scp.
>
> Signed-off-by: Olof Johansson <olof.johansson@axis.com>
> ---
> lib/bb/fetch2/__init__.py | 2 +
> lib/bb/fetch2/sftp.py | 104 +++++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 106 insertions(+)
> create mode 100644 lib/bb/fetch2/sftp.py
>
> diff --git lib/bb/fetch2/__init__.py lib/bb/fetch2/__init__.py
> index 549e261..86302f7 100644
> --- lib/bb/fetch2/__init__.py
> +++ lib/bb/fetch2/__init__.py
> @@ -1448,6 +1448,7 @@ from . import svn
> from . import wget
> from . import svk
> from . import ssh
> +from . import sftp
> from . import perforce
> from . import bzr
> from . import hg
> @@ -1461,6 +1462,7 @@ methods.append(git.Git())
> methods.append(cvs.Cvs())
> methods.append(svk.Svk())
> methods.append(ssh.SSH())
> +methods.append(sftp.SFTP())
> methods.append(perforce.Perforce())
> methods.append(bzr.Bzr())
> methods.append(hg.Hg())
> diff --git lib/bb/fetch2/sftp.py lib/bb/fetch2/sftp.py
> new file mode 100644
> index 0000000..0902ce7
> --- /dev/null
> +++ lib/bb/fetch2/sftp.py
> @@ -0,0 +1,104 @@
> +# ex:ts=4:sw=4:sts=4:et
> +# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +"""
> +BitBake SFTP Fetch implementation
> +
> +Class for fetching files via SFTP. It tries to adhere to the (now
> +expired) IETF Internet Draft for "Uniform Resource Identifier (URI)
> +Scheme for Secure File Transfer Protocol (SFTP) and Secure Shell
> +(SSH)" (SECSH URI).
> +
> +It uses SFTP (as to adhere to the SECSH URI specification). It only
> +supports key based authentication, not password. This class, unlike
> +the SSH fetcher, does not support fetching a directory tree from the
> +remote.
> +
> + http://tools.ietf.org/html/draft-ietf-secsh-scp-sftp-ssh-uri-04
> + https://www.iana.org/assignments/uri-schemes/prov/sftp
> + https://tools.ietf.org/html/draft-ietf-secsh-filexfer-13
> +
> +"""
> +
> +# Copyright (C) 2013, Olof Johansson <olof.johansson@axis.com>
> +#
> +# Based in part on bb.fetch2.wget:
> +# Copyright (C) 2003, 2004 Chris Larson
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License version 2 as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with this program; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +#
> +# Based on functions from the base bb module, Copyright 2003 Holger Schurig
> +
> +import os
> +import bb
> +import urllib
> +import commands
> +from bb import data
> +from bb.fetch2 import URI
> +from bb.fetch2 import FetchMethod
> +from bb.fetch2 import runfetchcmd
> +
> +
> +class SFTP(FetchMethod):
> + """Class to fetch urls via 'sftp'"""
> + def supports(self, url, ud, d):
> + """
> + Check to see if a given url can be fetched with sftp.
> + """
> + return ud.type in ['sftp']
> +
> + def recommends_checksum(self, urldata):
> + return True
> +
> + def urldata_init(self, ud, d):
> + if 'protocol' in ud.parm and ud.parm['protocol'] == 'git':
> + raise bb.fetch2.ParameterError(
> + "Invalid protocol - if you wish to fetch from a git " +
> + "repository using ssh, you need to instead use the " +
> + "git:// prefix with protocol=ssh", ud.url)
> +
> + if 'downloadfilename' in ud.parm:
> + ud.basename = ud.parm['downloadfilename']
> + else:
> + ud.basename = os.path.basename(ud.path)
> +
> + ud.localfile = data.expand(urllib.unquote(ud.basename), d)
> +
> + def download(self, uri, ud, d):
> + """Fetch urls"""
> +
> + urlo = URI(uri)
> + basecmd = 'sftp -oPasswordAuthentication=no'
> + port = ''
> + if urlo.port:
> + port = '-P %d' % urlo.port
> + urlo.port = None
> +
> + dldir = data.getVar('DL_DIR', d, True)
> + lpath = os.path.join(dldir, ud.localfile)
> +
> + user = ''
> + if urlo.userinfo:
> + user = urlo.userinfo + '@'
> +
> + remote = '%s%s:%s' % (user, urlo.hostname, urlo.path)
Please add some SRC_URI examples, it always surprise me that there isn't
':' as host-path separator, but '/' and error as returned from bitbake
ia not so easy to read.
Also it looks like optional ~ syntax is not supported
sftp://user@host.example.com/~/file.txt
But it should be easy to implement if you remove '^/~/' from urlo.path.
Cheers,
> +
> + cmd = '%s %s %s %s' % (basecmd, port, commands.mkarg(remote),
> + commands.mkarg(lpath))
> +
> + bb.fetch2.check_network_access(d, cmd, uri)
> + runfetchcmd(cmd, d)
> + return True
> +
> + def checkstatus(self, uri, ud, d):
> + return self.download(uri, ud, d, True)
> --
> 1.7.10.4
>
>
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
--
Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v2 0/6] fetch2: Updated SFTP fetcher
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (6 preceding siblings ...)
2013-01-28 14:05 ` [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Martin Jansa
@ 2013-01-29 7:50 ` Olof Johansson
2013-01-30 7:55 ` Olof Johansson
2013-01-29 7:50 ` [PATCH v2 1/6] fetch2: Remove unused code in wget fetcher Olof Johansson
` (5 subsequent siblings)
13 siblings, 1 reply; 23+ messages in thread
From: Olof Johansson @ 2013-01-29 7:50 UTC (permalink / raw)
To: bitbake-devel
Changes since the first draft (thanks Martin Jansa for the feedback):
* Rebase against current master
* Regenerate diffs without git config diff.noprefix
* Support /~/ (home relative URIs) in SFTP fetcher
* Add an example section in the SFTP fetcher documentation
* Remove "checkstatus" method from SFTP fetcher (wasn't meant to be there)
Olof Johansson (6):
fetch2: Remove unused code in wget fetcher
fetch2: Add a class representing a generic URI
fetch2: unittests for bb.fetch2.URI class
fetch2: Adapt encode/decode url to use URI class
fetch2: Add editor modelines for bb.tests.*
fetch2: Add SFTP fetcher
lib/bb/fetch2/__init__.py | 262 +++++++++++++++++++++++++++++++++++---------
lib/bb/fetch2/sftp.py | 129 ++++++++++++++++++++++
lib/bb/fetch2/wget.py | 5 -
lib/bb/tests/codeparser.py | 2 +
lib/bb/tests/cow.py | 2 +
lib/bb/tests/data.py | 2 +
lib/bb/tests/fetch.py | 228 ++++++++++++++++++++++++++++++++++++++
lib/bb/tests/utils.py | 2 +
8 files changed, 578 insertions(+), 54 deletions(-)
create mode 100644 lib/bb/fetch2/sftp.py
--
1.7.10.4
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v2 1/6] fetch2: Remove unused code in wget fetcher
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (7 preceding siblings ...)
2013-01-29 7:50 ` [PATCH v2 0/6] fetch2: Updated SFTP fetcher Olof Johansson
@ 2013-01-29 7:50 ` Olof Johansson
2013-01-29 7:50 ` [PATCH v2 2/6] fetch2: Add a class representing a generic URI Olof Johansson
` (4 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-29 7:50 UTC (permalink / raw)
To: bitbake-devel
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/fetch2/wget.py | 5 -----
1 file changed, 5 deletions(-)
diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index 2808df6..1fafc4a 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -32,8 +32,6 @@ import urllib
from bb import data
from bb.fetch2 import FetchMethod
from bb.fetch2 import FetchError
-from bb.fetch2 import encodeurl
-from bb.fetch2 import decodeurl
from bb.fetch2 import logger
from bb.fetch2 import runfetchcmd
@@ -77,9 +75,6 @@ class Wget(FetchMethod):
fetchcmd = d.getVar("FETCHCOMMAND_wget", True) or d.expand(basecmd + " -P ${DL_DIR} '${URI}'")
uri = uri.split(";")[0]
- uri_decoded = list(decodeurl(uri))
- uri_type = uri_decoded[0]
- uri_host = uri_decoded[1]
fetchcmd = fetchcmd.replace("${URI}", uri.split(";")[0])
fetchcmd = fetchcmd.replace("${FILE}", ud.basename)
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v2 2/6] fetch2: Add a class representing a generic URI
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (8 preceding siblings ...)
2013-01-29 7:50 ` [PATCH v2 1/6] fetch2: Remove unused code in wget fetcher Olof Johansson
@ 2013-01-29 7:50 ` Olof Johansson
2013-01-29 7:50 ` [PATCH v2 3/6] fetch2: unittests for bb.fetch2.URI class Olof Johansson
` (3 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-29 7:50 UTC (permalink / raw)
To: bitbake-devel
A class representing a generic URI, with methods for accessing the URI
components, and stringifies to the URI. This class should be a bit more
flexible than the existing {encode,decode}_url functions in that it
supports more components (e.g. port) and that it does not rely on a
specific order on the return values. This makes it easy to add new
properties without affecting the API.
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/fetch2/__init__.py | 194 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 194 insertions(+)
diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 4746356..7f44976 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -30,6 +30,7 @@ from __future__ import print_function
import os, re
import logging
import urllib
+from urlparse import urlparse
import operator
import bb.persist_data, bb.utils
import bb.checksum
@@ -120,6 +121,199 @@ class NonLocalMethod(Exception):
def __init__(self):
Exception.__init__(self)
+
+class URI(object):
+ """
+ A class representing a generic URI, with methods for
+ accessing the URI components, and stringifies to the
+ URI.
+
+ It is constructed by calling it with a URI, or setting
+ the attributes manually:
+
+ uri = URI("http://example.com/")
+
+ uri = URI()
+ uri.scheme = 'http'
+ uri.hostname = 'example.com'
+ uri.path = '/'
+
+ It has the following attributes:
+
+ * scheme (read/write)
+ * userinfo (authentication information) (read/write)
+ * username (read/write)
+ * password (read/write)
+
+ Note, password is deprecated as of RFC 3986.
+
+ * hostname (read/write)
+ * port (read/write)
+ * hostport (read only)
+ "hostname:port", if both are set, otherwise just "hostname"
+ * path (read/write)
+ * path_quoted (read/write)
+ A URI quoted version of path
+ * params (dict) (read/write)
+ * relative (bool) (read only)
+ True if this is a "relative URI", (e.g. file:foo.diff)
+
+ It stringifies to the URI itself.
+
+ Some notes about relative URIs: while it's specified that
+ a URI beginning with <scheme>:// should either be directly
+ followed by a hostname or a /, the old URI handling of the
+ fetch2 library did not comform to this. Therefore, this URI
+ class has some kludges to make sure that URIs are parsed in
+ a way comforming to bitbake's current usage. This URI class
+ supports the following:
+
+ file:relative/path.diff (IETF compliant)
+ git:relative/path.git (IETF compliant)
+ git:///absolute/path.git (IETF compliant)
+ file:///absolute/path.diff (IETF compliant)
+
+ file://relative/path.diff (not IETF compliant)
+
+ But it does not support the following:
+
+ file://hostname/absolute/path.diff (would be IETF compliant)
+
+ Note that the last case only applies to a list of
+ "whitelisted" schemes (currently only file://), that requires
+ its URIs to not have a network location.
+ """
+
+ _relative_schemes = ['file', 'git']
+ _netloc_forbidden = ['file']
+
+ def __init__(self, uri=None):
+ self.scheme = ''
+ self.userinfo = ''
+ self.hostname = ''
+ self.port = None
+ self._path = ''
+ self.params = {}
+ self.relative = False
+
+ if not uri:
+ return
+
+ urlp = urlparse(uri)
+ self.scheme = urlp.scheme
+
+ # Convert URI to be relative
+ if urlp.scheme in self._netloc_forbidden:
+ uri = re.sub("(?<=:)//(?!/)", "", uri, 1)
+ urlp = urlparse(uri)
+
+ # Identify if the URI is relative or not
+ if urlp.scheme in self._relative_schemes and \
+ re.compile("^\w+:(?!//)").match(uri):
+ self.relative = True
+
+ if not self.relative:
+ self.hostname = urlp.hostname or ''
+ self.port = urlp.port
+
+ self.userinfo += urlp.username or ''
+
+ if urlp.password:
+ self.userinfo += ':%s' % urlp.password
+
+ # Do support params even for URI schemes that Python's
+ # urlparse doesn't support params for.
+ path = ''
+ param_str = ''
+ if not urlp.params:
+ path, param_str = (list(urlp.path.split(";", 1)) + [None])[:2]
+ else:
+ path = urlp.path
+ param_str = urlp.params
+
+ self.path = urllib.unquote(path)
+
+ if param_str:
+ self.params = self._param_dict(param_str)
+
+ def __str__(self):
+ userinfo = self.userinfo
+ if userinfo:
+ userinfo += '@'
+
+ return "%s:%s%s%s%s%s" % (
+ self.scheme,
+ '' if self.relative else '//',
+ userinfo,
+ self.hostport,
+ self.path_quoted,
+ self._param_str)
+
+ @property
+ def _param_str(self):
+ ret = ''
+ for key, val in self.params.items():
+ ret += ";%s=%s" % (key, val)
+ return ret
+
+ def _param_dict(self, param_str):
+ parm = {}
+
+ for keyval in param_str.split(";"):
+ key, val = keyval.split("=", 1)
+ parm[key] = val
+
+ return parm
+
+ @property
+ def hostport(self):
+ if not self.port:
+ return self.hostname
+ return "%s:%d" % (self.hostname, self.port)
+
+ @property
+ def path_quoted(self):
+ return urllib.quote(self.path)
+
+ @path_quoted.setter
+ def path_quoted(self, path):
+ self.path = urllib.unquote(path)
+
+ @property
+ def path(self):
+ return self._path
+
+ @path.setter
+ def path(self, path):
+ self._path = path
+
+ if re.compile("^/").match(path):
+ self.relative = False
+ else:
+ self.relative = True
+
+ @property
+ def username(self):
+ if self.userinfo:
+ return (self.userinfo.split(":", 1))[0]
+ return ''
+
+ @username.setter
+ def username(self, username):
+ self.userinfo = username
+ if self.password:
+ self.userinfo += ":%s" % self.password
+
+ @property
+ def password(self):
+ if self.userinfo and ":" in self.userinfo:
+ return (self.userinfo.split(":", 1))[1]
+ return ''
+
+ @password.setter
+ def password(self, password):
+ self.userinfo = "%s:%s" % (self.username, password)
+
def decodeurl(url):
"""Decodes an URL into the tokens (scheme, network location, path,
user, password, parameters).
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v2 3/6] fetch2: unittests for bb.fetch2.URI class
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (9 preceding siblings ...)
2013-01-29 7:50 ` [PATCH v2 2/6] fetch2: Add a class representing a generic URI Olof Johansson
@ 2013-01-29 7:50 ` Olof Johansson
2013-01-29 7:50 ` [PATCH v2 4/6] fetch2: Adapt encode/decode url to use URI class Olof Johansson
` (2 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-29 7:50 UTC (permalink / raw)
To: bitbake-devel
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/tests/fetch.py | 226 +++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 226 insertions(+)
diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 67eda59..e765ca7 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -21,8 +21,234 @@ import unittest
import tempfile
import subprocess
import os
+from bb.fetch2 import URI
import bb
+class URITest(unittest.TestCase):
+ test_uris = {
+ "http://www.google.com/index.html" : {
+ 'uri': 'http://www.google.com/index.html',
+ 'scheme': 'http',
+ 'hostname': 'www.google.com',
+ 'port': None,
+ 'hostport': 'www.google.com',
+ 'path': '/index.html',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': False
+ },
+ "http://www.google.com/index.html;param1=value1" : {
+ 'uri': 'http://www.google.com/index.html;param1=value1',
+ 'scheme': 'http',
+ 'hostname': 'www.google.com',
+ 'port': None,
+ 'hostport': 'www.google.com',
+ 'path': '/index.html',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {
+ 'param1': 'value1'
+ },
+ 'relative': False
+ },
+ "http://www.example.com:8080/index.html" : {
+ 'uri': 'http://www.example.com:8080/index.html',
+ 'scheme': 'http',
+ 'hostname': 'www.example.com',
+ 'port': 8080,
+ 'hostport': 'www.example.com:8080',
+ 'path': '/index.html',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': False
+ },
+ "cvs://anoncvs@cvs.handhelds.org/cvs;module=familiar/dist/ipkg" : {
+ 'uri': 'cvs://anoncvs@cvs.handhelds.org/cvs;module=familiar/dist/ipkg',
+ 'scheme': 'cvs',
+ 'hostname': 'cvs.handhelds.org',
+ 'port': None,
+ 'hostport': 'cvs.handhelds.org',
+ 'path': '/cvs',
+ 'userinfo': 'anoncvs',
+ 'username': 'anoncvs',
+ 'password': '',
+ 'params': {
+ 'module': 'familiar/dist/ipkg'
+ },
+ 'relative': False
+ },
+ "cvs://anoncvs:anonymous@cvs.handhelds.org/cvs;tag=V0-99-81;module=familiar/dist/ipkg": {
+ 'uri': 'cvs://anoncvs:anonymous@cvs.handhelds.org/cvs;tag=V0-99-81;module=familiar/dist/ipkg',
+ 'scheme': 'cvs',
+ 'hostname': 'cvs.handhelds.org',
+ 'port': None,
+ 'hostport': 'cvs.handhelds.org',
+ 'path': '/cvs',
+ 'userinfo': 'anoncvs:anonymous',
+ 'username': 'anoncvs',
+ 'password': 'anonymous',
+ 'params': {
+ 'tag': 'V0-99-81',
+ 'module': 'familiar/dist/ipkg'
+ },
+ 'relative': False
+ },
+ "file://example.diff": { # NOTE: Not RFC compliant!
+ 'uri': 'file:example.diff',
+ 'scheme': 'file',
+ 'hostname': '',
+ 'port': None,
+ 'hostport': '',
+ 'path': 'example.diff',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': True
+ },
+ "file:example.diff": { # NOTE: RFC compliant version of the former
+ 'uri': 'file:example.diff',
+ 'scheme': 'file',
+ 'hostname': '',
+ 'port': None,
+ 'hostport': '',
+ 'path': 'example.diff',
+ 'userinfo': '',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': True
+ },
+ "file:///tmp/example.diff": {
+ 'uri': 'file:///tmp/example.diff',
+ 'scheme': 'file',
+ 'hostname': '',
+ 'port': None,
+ 'hostport': '',
+ 'path': '/tmp/example.diff',
+ 'userinfo': '',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': False
+ },
+ "git:///path/example.git": {
+ 'uri': 'git:///path/example.git',
+ 'scheme': 'git',
+ 'hostname': '',
+ 'port': None,
+ 'hostport': '',
+ 'path': '/path/example.git',
+ 'userinfo': '',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': False
+ },
+ "git:path/example.git": {
+ 'uri': 'git:path/example.git',
+ 'scheme': 'git',
+ 'hostname': '',
+ 'port': None,
+ 'hostport': '',
+ 'path': 'path/example.git',
+ 'userinfo': '',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': True
+ },
+ "git://example.net/path/example.git": {
+ 'uri': 'git://example.net/path/example.git',
+ 'scheme': 'git',
+ 'hostname': 'example.net',
+ 'port': None,
+ 'hostport': 'example.net',
+ 'path': '/path/example.git',
+ 'userinfo': '',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': False
+ }
+ }
+
+ def test_uri(self):
+ for test_uri, ref in self.test_uris.items():
+ uri = URI(test_uri)
+
+ self.assertEqual(str(uri), ref['uri'])
+
+ # expected attributes
+ self.assertEqual(uri.scheme, ref['scheme'])
+
+ self.assertEqual(uri.userinfo, ref['userinfo'])
+ self.assertEqual(uri.username, ref['username'])
+ self.assertEqual(uri.password, ref['password'])
+
+ self.assertEqual(uri.hostname, ref['hostname'])
+ self.assertEqual(uri.port, ref['port'])
+ self.assertEqual(uri.hostport, ref['hostport'])
+
+ self.assertEqual(uri.path, ref['path'])
+ self.assertEqual(uri.params, ref['params'])
+
+ self.assertEqual(uri.relative, ref['relative'])
+
+ def test_dict(self):
+ for test in self.test_uris.values():
+ uri = URI()
+
+ self.assertEqual(uri.scheme, '')
+ self.assertEqual(uri.userinfo, '')
+ self.assertEqual(uri.username, '')
+ self.assertEqual(uri.password, '')
+ self.assertEqual(uri.hostname, '')
+ self.assertEqual(uri.port, None)
+ self.assertEqual(uri.path, '')
+ self.assertEqual(uri.params, {})
+
+
+ uri.scheme = test['scheme']
+ self.assertEqual(uri.scheme, test['scheme'])
+
+ uri.userinfo = test['userinfo']
+ self.assertEqual(uri.userinfo, test['userinfo'])
+ self.assertEqual(uri.username, test['username'])
+ self.assertEqual(uri.password, test['password'])
+
+ uri.hostname = test['hostname']
+ self.assertEqual(uri.hostname, test['hostname'])
+ self.assertEqual(uri.hostport, test['hostname'])
+
+ uri.port = test['port']
+ self.assertEqual(uri.port, test['port'])
+ self.assertEqual(uri.hostport, test['hostport'])
+
+ uri.path = test['path']
+ self.assertEqual(uri.path, test['path'])
+
+ uri.params = test['params']
+ self.assertEqual(uri.params, test['params'])
+
+ self.assertEqual(str(uri)+str(uri.relative), str(test['uri'])+str(test['relative']))
+
+ self.assertEqual(str(uri), test['uri'])
+
+ uri.params = {}
+ self.assertEqual(uri.params, {})
+ self.assertEqual(str(uri), (str(uri).split(";"))[0])
+
class FetcherTest(unittest.TestCase):
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v2 4/6] fetch2: Adapt encode/decode url to use URI class
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (10 preceding siblings ...)
2013-01-29 7:50 ` [PATCH v2 3/6] fetch2: unittests for bb.fetch2.URI class Olof Johansson
@ 2013-01-29 7:50 ` Olof Johansson
2013-02-20 7:12 ` Richard Purdie
2013-01-29 7:50 ` [PATCH v2 5/6] fetch2: Add editor modelines for bb.tests.* Olof Johansson
2013-01-29 7:50 ` [PATCH v2 6/6] fetch2: Add SFTP fetcher Olof Johansson
13 siblings, 1 reply; 23+ messages in thread
From: Olof Johansson @ 2013-01-29 7:50 UTC (permalink / raw)
To: bitbake-devel
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/fetch2/__init__.py | 66 ++++++++++++---------------------------------
1 file changed, 17 insertions(+), 49 deletions(-)
diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 7f44976..b6fc8f8 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -319,40 +319,9 @@ def decodeurl(url):
user, password, parameters).
"""
- m = re.compile('(?P<type>[^:]*)://((?P<user>.+)@)?(?P<location>[^;]+)(;(?P<parm>.*))?').match(url)
- if not m:
- raise MalformedUrl(url)
-
- type = m.group('type')
- location = m.group('location')
- if not location:
- raise MalformedUrl(url)
- user = m.group('user')
- parm = m.group('parm')
-
- locidx = location.find('/')
- if locidx != -1 and type.lower() != 'file':
- host = location[:locidx]
- path = location[locidx:]
- else:
- host = ""
- path = location
- if user:
- m = re.compile('(?P<user>[^:]+)(:?(?P<pswd>.*))').match(user)
- if m:
- user = m.group('user')
- pswd = m.group('pswd')
- else:
- user = ''
- pswd = ''
-
- p = {}
- if parm:
- for s in parm.split(';'):
- s1, s2 = s.split('=')
- p[s1] = s2
-
- return type, host, urllib.unquote(path), user, pswd, p
+ urlo = URI(url)
+ return (urlo.scheme, urlo.hostport, urlo.path,
+ urlo.username, urlo.password, urlo.params)
def encodeurl(decoded):
"""Encodes a URL from tokens (scheme, network location, path,
@@ -361,27 +330,26 @@ def encodeurl(decoded):
type, host, path, user, pswd, p = decoded
+ urlo = URI()
+
if not path:
raise MissingParameterError('path', "encoded from the data %s" % str(decoded))
if not type:
raise MissingParameterError('type', "encoded from the data %s" % str(decoded))
- url = '%s://' % type
- if user and type != "file":
- url += "%s" % user
- if pswd:
- url += ":%s" % pswd
- url += "@"
- if host and type != "file":
- url += "%s" % host
- # Standardise path to ensure comparisons work
- while '//' in path:
- path = path.replace("//", "/")
- url += "%s" % urllib.quote(path)
+
+ urlo.scheme = type
+ urlo.path = path
+
+ if host:
+ urlo.hostname = host
+ if user:
+ urlo.username = user
+ if pswd:
+ urlo.password = pswd
if p:
- for parm in p:
- url += ";%s=%s" % (parm, p[parm])
+ urlo.params = p
- return url
+ return str(urlo)
def uri_replace(ud, uri_find, uri_replace, replacements, d):
if not ud.url or not uri_find or not uri_replace:
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v2 5/6] fetch2: Add editor modelines for bb.tests.*
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (11 preceding siblings ...)
2013-01-29 7:50 ` [PATCH v2 4/6] fetch2: Adapt encode/decode url to use URI class Olof Johansson
@ 2013-01-29 7:50 ` Olof Johansson
2013-01-29 7:50 ` [PATCH v2 6/6] fetch2: Add SFTP fetcher Olof Johansson
13 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-29 7:50 UTC (permalink / raw)
To: bitbake-devel
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/tests/codeparser.py | 2 ++
lib/bb/tests/cow.py | 2 ++
lib/bb/tests/data.py | 2 ++
lib/bb/tests/fetch.py | 2 ++
lib/bb/tests/utils.py | 2 ++
5 files changed, 10 insertions(+)
diff --git a/lib/bb/tests/codeparser.py b/lib/bb/tests/codeparser.py
index 9b2d588..d2bd4a2 100644
--- a/lib/bb/tests/codeparser.py
+++ b/lib/bb/tests/codeparser.py
@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Test for codeparser.py
#
diff --git a/lib/bb/tests/cow.py b/lib/bb/tests/cow.py
index 599b2aa..35c5841 100644
--- a/lib/bb/tests/cow.py
+++ b/lib/bb/tests/cow.py
@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for Copy-on-Write (cow.py)
#
diff --git a/lib/bb/tests/data.py b/lib/bb/tests/data.py
index b2ab8ee..5161586 100644
--- a/lib/bb/tests/data.py
+++ b/lib/bb/tests/data.py
@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for the Data Store (data.py/data_smart.py)
#
diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index e765ca7..778d94f 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for the Fetcher (fetch2/)
#
diff --git a/lib/bb/tests/utils.py b/lib/bb/tests/utils.py
index 9f60491..7c50b1d 100644
--- a/lib/bb/tests/utils.py
+++ b/lib/bb/tests/utils.py
@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for utils.py
#
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v2 6/6] fetch2: Add SFTP fetcher
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (12 preceding siblings ...)
2013-01-29 7:50 ` [PATCH v2 5/6] fetch2: Add editor modelines for bb.tests.* Olof Johansson
@ 2013-01-29 7:50 ` Olof Johansson
13 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-29 7:50 UTC (permalink / raw)
To: bitbake-devel
This fetcher differs from the SSH fetcher in that it adheres more
strictly to the SECSH URI internet draft --- it uses the sftp://
instead of the ssh:// scheme, and it uses sftp instead of scp.
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/fetch2/__init__.py | 2 +
lib/bb/fetch2/sftp.py | 129 +++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 131 insertions(+)
create mode 100644 lib/bb/fetch2/sftp.py
diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index b6fc8f8..cc524c5 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -1449,6 +1449,7 @@ from . import svn
from . import wget
from . import svk
from . import ssh
+from . import sftp
from . import perforce
from . import bzr
from . import hg
@@ -1462,6 +1463,7 @@ methods.append(git.Git())
methods.append(cvs.Cvs())
methods.append(svk.Svk())
methods.append(ssh.SSH())
+methods.append(sftp.SFTP())
methods.append(perforce.Perforce())
methods.append(bzr.Bzr())
methods.append(hg.Hg())
diff --git a/lib/bb/fetch2/sftp.py b/lib/bb/fetch2/sftp.py
new file mode 100644
index 0000000..5fbbcfd
--- /dev/null
+++ b/lib/bb/fetch2/sftp.py
@@ -0,0 +1,129 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake SFTP Fetch implementation
+
+Class for fetching files via SFTP. It tries to adhere to the (now
+expired) IETF Internet Draft for "Uniform Resource Identifier (URI)
+Scheme for Secure File Transfer Protocol (SFTP) and Secure Shell
+(SSH)" (SECSH URI).
+
+It uses SFTP (as to adhere to the SECSH URI specification). It only
+supports key based authentication, not password. This class, unlike
+the SSH fetcher, does not support fetching a directory tree from the
+remote.
+
+ http://tools.ietf.org/html/draft-ietf-secsh-scp-sftp-ssh-uri-04
+ https://www.iana.org/assignments/uri-schemes/prov/sftp
+ https://tools.ietf.org/html/draft-ietf-secsh-filexfer-13
+
+Please note that '/' is used as host path seperator, and not ":"
+as you may be used to from the scp/sftp commands. You can use a
+~ (tilde) to specify a path relative to your home directory.
+(The /~user/ syntax, for specyfing a path relative to another
+user's home directory is not supported.) Note that the tilde must
+still follow the host path seperator ("/"). See exampels below.
+
+Example SRC_URIs:
+
+SRC_URI = "sftp://host.example.com/dir/path.file.txt"
+
+A path relative to your home directory.
+
+SRC_URI = "sftp://host.example.com/~/dir/path.file.txt"
+
+You can also specify a username (specyfing password in the
+URI is not supported, use SSH keys to authenticate):
+
+SRC_URI = "sftp://user@host.example.com/dir/path.file.txt"
+
+"""
+
+# Copyright (C) 2013, Olof Johansson <olof.johansson@axis.com>
+#
+# Based in part on bb.fetch2.wget:
+# Copyright (C) 2003, 2004 Chris Larson
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# Based on functions from the base bb module, Copyright 2003 Holger Schurig
+
+import os
+import bb
+import urllib
+import commands
+from bb import data
+from bb.fetch2 import URI
+from bb.fetch2 import FetchMethod
+from bb.fetch2 import runfetchcmd
+
+
+class SFTP(FetchMethod):
+ """Class to fetch urls via 'sftp'"""
+
+ def supports(self, url, ud, d):
+ """
+ Check to see if a given url can be fetched with sftp.
+ """
+ return ud.type in ['sftp']
+
+ def recommends_checksum(self, urldata):
+ return True
+
+ def urldata_init(self, ud, d):
+ if 'protocol' in ud.parm and ud.parm['protocol'] == 'git':
+ raise bb.fetch2.ParameterError(
+ "Invalid protocol - if you wish to fetch from a " +
+ "git repository using ssh, you need to use the " +
+ "git:// prefix with protocol=ssh", ud.url)
+
+ if 'downloadfilename' in ud.parm:
+ ud.basename = ud.parm['downloadfilename']
+ else:
+ ud.basename = os.path.basename(ud.path)
+
+ ud.localfile = data.expand(urllib.unquote(ud.basename), d)
+
+ def download(self, uri, ud, d):
+ """Fetch urls"""
+
+ urlo = URI(uri)
+ basecmd = 'sftp -oPasswordAuthentication=no'
+ port = ''
+ if urlo.port:
+ port = '-P %d' % urlo.port
+ urlo.port = None
+
+ dldir = data.getVar('DL_DIR', d, True)
+ lpath = os.path.join(dldir, ud.localfile)
+
+ user = ''
+ if urlo.userinfo:
+ user = urlo.userinfo + '@'
+
+ path = urlo.path
+
+ # Supoprt URIs relative to the user's home directory, with
+ # the tilde syntax. (E.g. <sftp://example.com/~/foo.diff>).
+ if path[:3] == '/~/':
+ path = path[3:]
+
+ remote = '%s%s:%s' % (user, urlo.hostname, path)
+
+ cmd = '%s %s %s %s' % (basecmd, port, commands.mkarg(remote),
+ commands.mkarg(lpath))
+
+ bb.fetch2.check_network_access(d, cmd, uri)
+ runfetchcmd(cmd, d)
+ return True
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [PATCH v2 0/6] fetch2: Updated SFTP fetcher
2013-01-29 7:50 ` [PATCH v2 0/6] fetch2: Updated SFTP fetcher Olof Johansson
@ 2013-01-30 7:55 ` Olof Johansson
2013-01-31 12:43 ` Richard Purdie
0 siblings, 1 reply; 23+ messages in thread
From: Olof Johansson @ 2013-01-30 7:55 UTC (permalink / raw)
To: bitbake-devel@lists.openembedded.org
Hi,
On 2013-01-29 08:50, Olof Johansson wrote:
> Olof Johansson (6):
> fetch2: Remove unused code in wget fetcher
> fetch2: Add a class representing a generic URI
> fetch2: unittests for bb.fetch2.URI class
> fetch2: Adapt encode/decode url to use URI class
> fetch2: Add editor modelines for bb.tests.*
> fetch2: Add SFTP fetcher
One thing I forgot to mention; after speaking with kergoth on IRC
last Friday, we found that there are some performance penelties
in using the URI class instead of encode/decodeurl. I tried to
profile and see if there were any obvious ways to get it down to
the level of the old encode/decodeurl, but nothing obvious.
I did the performance evaluation by running the unittests.
Forking and running the unittests in seperate processes shows no
change:
http://codepad.org/SYsVsckY
However, running them in a single process with the timeit module
shows a pretty big (proportional) regression:
http://codepad.org/Vzu7Pqqw
Is this significant? Running the tests 10000, each run creating
three URI objects, takes ~2 seconds.
--
Olof Johasnson
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 0/6] fetch2: Updated SFTP fetcher
2013-01-30 7:55 ` Olof Johansson
@ 2013-01-31 12:43 ` Richard Purdie
2013-02-01 8:58 ` Peter Kjellerstedt
0 siblings, 1 reply; 23+ messages in thread
From: Richard Purdie @ 2013-01-31 12:43 UTC (permalink / raw)
To: Olof Johansson; +Cc: bitbake-devel@lists.openembedded.org
On Wed, 2013-01-30 at 08:55 +0100, Olof Johansson wrote:
> Hi,
>
> On 2013-01-29 08:50, Olof Johansson wrote:
> > Olof Johansson (6):
> > fetch2: Remove unused code in wget fetcher
> > fetch2: Add a class representing a generic URI
> > fetch2: unittests for bb.fetch2.URI class
> > fetch2: Adapt encode/decode url to use URI class
> > fetch2: Add editor modelines for bb.tests.*
> > fetch2: Add SFTP fetcher
>
> One thing I forgot to mention; after speaking with kergoth on IRC
> last Friday, we found that there are some performance penelties
> in using the URI class instead of encode/decodeurl. I tried to
> profile and see if there were any obvious ways to get it down to
> the level of the old encode/decodeurl, but nothing obvious.
>
> I did the performance evaluation by running the unittests.
>
> Forking and running the unittests in seperate processes shows no
> change:
>
> http://codepad.org/SYsVsckY
>
> However, running them in a single process with the timeit module
> shows a pretty big (proportional) regression:
>
> http://codepad.org/Vzu7Pqqw
>
> Is this significant? Running the tests 10000, each run creating
> three URI objects, takes ~2 seconds.
In the scheme of things and other bitbake overheads, I'd suspect its not
significant although it is a good thing to check...
What were the other changes between v1 and v2? In cases like this a 0/6
summary email is helpful.
Cheers,
Richard
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 0/6] fetch2: Updated SFTP fetcher
2013-01-31 12:43 ` Richard Purdie
@ 2013-02-01 8:58 ` Peter Kjellerstedt
0 siblings, 0 replies; 23+ messages in thread
From: Peter Kjellerstedt @ 2013-02-01 8:58 UTC (permalink / raw)
To: Richard Purdie; +Cc: bitbake-devel@lists.openembedded.org, Olof Johansson
> -----Original Message-----
> From: bitbake-devel-bounces@lists.openembedded.org [mailto:bitbake-
> devel-bounces@lists.openembedded.org] On Behalf Of Richard Purdie
> Sent: den 31 januari 2013 13:43
> To: Olof Johansson
> Cc: bitbake-devel@lists.openembedded.org
> Subject: Re: [bitbake-devel] [PATCH v2 0/6] fetch2: Updated SFTP
> fetcher
>
> On Wed, 2013-01-30 at 08:55 +0100, Olof Johansson wrote:
> > Hi,
> >
> > On 2013-01-29 08:50, Olof Johansson wrote:
> > > Olof Johansson (6):
> > > fetch2: Remove unused code in wget fetcher
> > > fetch2: Add a class representing a generic URI
> > > fetch2: unittests for bb.fetch2.URI class
> > > fetch2: Adapt encode/decode url to use URI class
> > > fetch2: Add editor modelines for bb.tests.*
> > > fetch2: Add SFTP fetcher
> >
> > One thing I forgot to mention; after speaking with kergoth on IRC
> > last Friday, we found that there are some performance penelties
> > in using the URI class instead of encode/decodeurl. I tried to
> > profile and see if there were any obvious ways to get it down to
> > the level of the old encode/decodeurl, but nothing obvious.
> >
> > I did the performance evaluation by running the unittests.
> >
> > Forking and running the unittests in seperate processes shows no
> > change:
> >
> > http://codepad.org/SYsVsckY
> >
> > However, running them in a single process with the timeit module
> > shows a pretty big (proportional) regression:
> >
> > http://codepad.org/Vzu7Pqqw
> >
> > Is this significant? Running the tests 10000, each run creating
> > three URI objects, takes ~2 seconds.
>
> In the scheme of things and other bitbake overheads, I'd suspect its
> not significant although it is a good thing to check...
>
> What were the other changes between v1 and v2? In cases like this a 0/6
> summary email is helpful.
>
> Cheers,
>
> Richard
The list of changes between v1 and v2 were in the 0/6 mail that
Olof did send (the one he replied to above). I have included them
below:
> -----Original Message-----
> From: bitbake-devel-bounces@lists.openembedded.org [mailto:bitbake-
> devel-bounces@lists.openembedded.org] On Behalf Of Olof Johansson
> Sent: den 29 januari 2013 08:50
> To: bitbake-devel@lists.openembedded.org
> Subject: [bitbake-devel] [PATCH v2 0/6] fetch2: Updated SFTP fetcher
>
> Changes since the first draft (thanks Martin Jansa for the feedback):
>
> * Rebase against current master
> * Regenerate diffs without git config diff.noprefix
> * Support /~/ (home relative URIs) in SFTP fetcher
> * Add an example section in the SFTP fetcher documentation
> * Remove "checkstatus" method from SFTP fetcher (wasn't meant to be there)
>
> Olof Johansson (6):
> fetch2: Remove unused code in wget fetcher
> ...
//Peter
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 4/6] fetch2: Adapt encode/decode url to use URI class
2013-01-29 7:50 ` [PATCH v2 4/6] fetch2: Adapt encode/decode url to use URI class Olof Johansson
@ 2013-02-20 7:12 ` Richard Purdie
0 siblings, 0 replies; 23+ messages in thread
From: Richard Purdie @ 2013-02-20 7:12 UTC (permalink / raw)
To: Olof Johansson; +Cc: bitbake-devel
On Tue, 2013-01-29 at 08:50 +0100, Olof Johansson wrote:
> Signed-off-by: Olof Johansson <olof.johansson@axis.com>
> ---
> lib/bb/fetch2/__init__.py | 66 ++++++++++++---------------------------------
> 1 file changed, 17 insertions(+), 49 deletions(-)
I've had to revert this since it simply doesn't work. bitbake-selftest
fails all the tests with this applied, we also saw multiple failures in
other places where things should work :(
Cheers,
Richard
^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2013-02-20 7:29 UTC | newest]
Thread overview: 23+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
2013-01-21 9:17 ` [PATCH 1/6] fetch2: Remove unused code in wget fetcher Olof Johansson
2013-01-21 9:17 ` [PATCH 2/6] fetch2: Add a class representing a generic URI Olof Johansson
2013-01-21 9:18 ` [PATCH 3/6] fetch2: unittests for bb.fetch2.URI class Olof Johansson
2013-01-21 9:18 ` [PATCH 4/6] fetch2: Adapt encode/decode url to use URI class Olof Johansson
2013-01-21 9:18 ` [PATCH 5/6] fetch2: Add editor modelines for bb.tests.* Olof Johansson
2013-01-21 9:18 ` [PATCH 6/6] fetch2: Add SFTP fetcher Olof Johansson
2013-01-28 14:13 ` Martin Jansa
2013-01-28 14:20 ` Olof Johansson
2013-01-28 15:43 ` Martin Jansa
2013-01-28 14:05 ` [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Martin Jansa
2013-01-28 14:23 ` Olof Johansson
2013-01-29 7:50 ` [PATCH v2 0/6] fetch2: Updated SFTP fetcher Olof Johansson
2013-01-30 7:55 ` Olof Johansson
2013-01-31 12:43 ` Richard Purdie
2013-02-01 8:58 ` Peter Kjellerstedt
2013-01-29 7:50 ` [PATCH v2 1/6] fetch2: Remove unused code in wget fetcher Olof Johansson
2013-01-29 7:50 ` [PATCH v2 2/6] fetch2: Add a class representing a generic URI Olof Johansson
2013-01-29 7:50 ` [PATCH v2 3/6] fetch2: unittests for bb.fetch2.URI class Olof Johansson
2013-01-29 7:50 ` [PATCH v2 4/6] fetch2: Adapt encode/decode url to use URI class Olof Johansson
2013-02-20 7:12 ` Richard Purdie
2013-01-29 7:50 ` [PATCH v2 5/6] fetch2: Add editor modelines for bb.tests.* Olof Johansson
2013-01-29 7:50 ` [PATCH v2 6/6] fetch2: Add SFTP fetcher Olof Johansson
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.