* [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl
@ 2013-01-21 9:17 Olof Johansson
2013-01-21 9:17 ` [PATCH 1/6] fetch2: Remove unused code in wget fetcher Olof Johansson
` (13 more replies)
0 siblings, 14 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-21 9:17 UTC (permalink / raw)
To: bitbake-devel
Hi,
In our early work in migrating to poky we realized that bitbake's current SSH
fetcher doesn't really suite our needs. It suffers from some specification
conformance bugs --- some which may be easily worked around. However, this
patch series instead introduces a new "SFTP" fetcher, meant to in a way,
replace the SSH fetcher, with stricter conformance to the SECSH URI spec [1].
The SSH fetcher has its own URI parsing, and can not rely on
bb.fetch2.decodeurl. I found that decodeurl was hard to extend, with it
returning a tuple. This prompted me to write a generic URI class, with
attributes for the various URI components (and some general bitbake kludges,
like support relative file:// uris).
I realize that these changes are a bit bold and non-trivial, but I hope you'll
find time to review them and point out problems to fix. If my reasoning is
completely wrong, please let me know and I can try to solve our direct problems
with the SSH fetcher with minimal changes instead.
The changes come with unittests for the URI class; it passes all the tests in
bb.tests.fetch and all other tests under bb.tests, except
bb.tests.data.DataExpansions, but the same errors are present on master as
well. I've also tested running bitbake core-image-minimal in a new tree without
any issues.
Olof Johansson (6):
fetch2: Remove unused code in wget fetcher
fetch2: Add a class representing a generic URI
fetch2: unittests for bb.fetch2.URI class
fetch2: Adapt encode/decode url to use URI class
fetch2: Add editor modelines for bb.tests.*
fetch2: Add SFTP fetcher
lib/bb/fetch2/__init__.py | 262 +++++++++++++++++++++++++++++++++++---------
lib/bb/fetch2/sftp.py | 104 ++++++++++++++++++
lib/bb/fetch2/wget.py | 5 -
lib/bb/tests/codeparser.py | 2 +
lib/bb/tests/cow.py | 2 +
lib/bb/tests/data.py | 2 +
lib/bb/tests/fetch.py | 228 ++++++++++++++++++++++++++++++++++++++
lib/bb/tests/utils.py | 2 +
8 files changed, 553 insertions(+), 54 deletions(-)
create mode 100644 lib/bb/fetch2/sftp.py
References:
1: http://tools.ietf.org/html/draft-ietf-secsh-scp-sftp-ssh-uri-04
--
Olof Johansson
Tools engineer
(IRC: zibri on Freenode)
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH 1/6] fetch2: Remove unused code in wget fetcher
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
@ 2013-01-21 9:17 ` Olof Johansson
2013-01-21 9:17 ` [PATCH 2/6] fetch2: Add a class representing a generic URI Olof Johansson
` (12 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-21 9:17 UTC (permalink / raw)
To: bitbake-devel
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/fetch2/wget.py | 5 -----
1 file changed, 5 deletions(-)
diff --git lib/bb/fetch2/wget.py lib/bb/fetch2/wget.py
index 2808df6..1fafc4a 100644
--- lib/bb/fetch2/wget.py
+++ lib/bb/fetch2/wget.py
@@ -32,8 +32,6 @@ import urllib
from bb import data
from bb.fetch2 import FetchMethod
from bb.fetch2 import FetchError
-from bb.fetch2 import encodeurl
-from bb.fetch2 import decodeurl
from bb.fetch2 import logger
from bb.fetch2 import runfetchcmd
@@ -77,9 +75,6 @@ class Wget(FetchMethod):
fetchcmd = d.getVar("FETCHCOMMAND_wget", True) or d.expand(basecmd + " -P ${DL_DIR} '${URI}'")
uri = uri.split(";")[0]
- uri_decoded = list(decodeurl(uri))
- uri_type = uri_decoded[0]
- uri_host = uri_decoded[1]
fetchcmd = fetchcmd.replace("${URI}", uri.split(";")[0])
fetchcmd = fetchcmd.replace("${FILE}", ud.basename)
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 2/6] fetch2: Add a class representing a generic URI
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
2013-01-21 9:17 ` [PATCH 1/6] fetch2: Remove unused code in wget fetcher Olof Johansson
@ 2013-01-21 9:17 ` Olof Johansson
2013-01-21 9:18 ` [PATCH 3/6] fetch2: unittests for bb.fetch2.URI class Olof Johansson
` (11 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-21 9:17 UTC (permalink / raw)
To: bitbake-devel
A class representing a generic URI, with methods for accessing the URI
components, and stringifies to the URI. This class should be a bit more
flexible than the existing {encode,decode}_url functions in that it
supports more components (e.g. port) and that it does not rely on a
specific order on the return values. This makes it easy to add new
properties without affecting the API.
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/fetch2/__init__.py | 194 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 194 insertions(+)
diff --git lib/bb/fetch2/__init__.py lib/bb/fetch2/__init__.py
index 150dc3c..3b83c0c 100644
--- lib/bb/fetch2/__init__.py
+++ lib/bb/fetch2/__init__.py
@@ -30,6 +30,7 @@ from __future__ import print_function
import os, re
import logging
import urllib
+from urlparse import urlparse
import bb.persist_data, bb.utils
import bb.checksum
from bb import data
@@ -119,6 +120,199 @@ class NonLocalMethod(Exception):
def __init__(self):
Exception.__init__(self)
+
+class URI(object):
+ """
+ A class representing a generic URI, with methods for
+ accessing the URI components, and stringifies to the
+ URI.
+
+ It is constructed by calling it with a URI, or setting
+ the attributes manually:
+
+ uri = URI("http://example.com/")
+
+ uri = URI()
+ uri.scheme = 'http'
+ uri.hostname = 'example.com'
+ uri.path = '/'
+
+ It has the following attributes:
+
+ * scheme (read/write)
+ * userinfo (authentication information) (read/write)
+ * username (read/write)
+ * password (read/write)
+
+ Note, password is deprecated as of RFC 3986.
+
+ * hostname (read/write)
+ * port (read/write)
+ * hostport (read only)
+ "hostname:port", if both are set, otherwise just "hostname"
+ * path (read/write)
+ * path_quoted (read/write)
+ A URI quoted version of path
+ * params (dict) (read/write)
+ * relative (bool) (read only)
+ True if this is a "relative URI", (e.g. file:foo.diff)
+
+ It stringifies to the URI itself.
+
+ Some notes about relative URIs: while it's specified that
+ a URI beginning with <scheme>:// should either be directly
+ followed by a hostname or a /, the old URI handling of the
+ fetch2 library did not comform to this. Therefore, this URI
+ class has some kludges to make sure that URIs are parsed in
+ a way comforming to bitbake's current usage. This URI class
+ supports the following:
+
+ file:relative/path.diff (IETF compliant)
+ git:relative/path.git (IETF compliant)
+ git:///absolute/path.git (IETF compliant)
+ file:///absolute/path.diff (IETF compliant)
+
+ file://relative/path.diff (not IETF compliant)
+
+ But it does not support the following:
+
+ file://hostname/absolute/path.diff (would be IETF compliant)
+
+ Note that the last case only applies to a list of
+ "whitelisted" schemes (currently only file://), that requires
+ its URIs to not have a network location.
+ """
+
+ _relative_schemes = ['file', 'git']
+ _netloc_forbidden = ['file']
+
+ def __init__(self, uri=None):
+ self.scheme = ''
+ self.userinfo = ''
+ self.hostname = ''
+ self.port = None
+ self._path = ''
+ self.params = {}
+ self.relative = False
+
+ if not uri:
+ return
+
+ urlp = urlparse(uri)
+ self.scheme = urlp.scheme
+
+ # Convert URI to be relative
+ if urlp.scheme in self._netloc_forbidden:
+ uri = re.sub("(?<=:)//(?!/)", "", uri, 1)
+ urlp = urlparse(uri)
+
+ # Identify if the URI is relative or not
+ if urlp.scheme in self._relative_schemes and \
+ re.compile("^\w+:(?!//)").match(uri):
+ self.relative = True
+
+ if not self.relative:
+ self.hostname = urlp.hostname or ''
+ self.port = urlp.port
+
+ self.userinfo += urlp.username or ''
+
+ if urlp.password:
+ self.userinfo += ':%s' % urlp.password
+
+ # Do support params even for URI schemes that Python's
+ # urlparse doesn't support params for.
+ path = ''
+ param_str = ''
+ if not urlp.params:
+ path, param_str = (list(urlp.path.split(";", 1)) + [None])[:2]
+ else:
+ path = urlp.path
+ param_str = urlp.params
+
+ self.path = urllib.unquote(path)
+
+ if param_str:
+ self.params = self._param_dict(param_str)
+
+ def __str__(self):
+ userinfo = self.userinfo
+ if userinfo:
+ userinfo += '@'
+
+ return "%s:%s%s%s%s%s" % (
+ self.scheme,
+ '' if self.relative else '//',
+ userinfo,
+ self.hostport,
+ self.path_quoted,
+ self._param_str)
+
+ @property
+ def _param_str(self):
+ ret = ''
+ for key, val in self.params.items():
+ ret += ";%s=%s" % (key, val)
+ return ret
+
+ def _param_dict(self, param_str):
+ parm = {}
+
+ for keyval in param_str.split(";"):
+ key, val = keyval.split("=", 1)
+ parm[key] = val
+
+ return parm
+
+ @property
+ def hostport(self):
+ if not self.port:
+ return self.hostname
+ return "%s:%d" % (self.hostname, self.port)
+
+ @property
+ def path_quoted(self):
+ return urllib.quote(self.path)
+
+ @path_quoted.setter
+ def path_quoted(self, path):
+ self.path = urllib.unquote(path)
+
+ @property
+ def path(self):
+ return self._path
+
+ @path.setter
+ def path(self, path):
+ self._path = path
+
+ if re.compile("^/").match(path):
+ self.relative = False
+ else:
+ self.relative = True
+
+ @property
+ def username(self):
+ if self.userinfo:
+ return (self.userinfo.split(":", 1))[0]
+ return ''
+
+ @username.setter
+ def username(self, username):
+ self.userinfo = username
+ if self.password:
+ self.userinfo += ":%s" % self.password
+
+ @property
+ def password(self):
+ if self.userinfo and ":" in self.userinfo:
+ return (self.userinfo.split(":", 1))[1]
+ return ''
+
+ @password.setter
+ def password(self, password):
+ self.userinfo = "%s:%s" % (self.username, password)
+
def decodeurl(url):
"""Decodes an URL into the tokens (scheme, network location, path,
user, password, parameters).
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 3/6] fetch2: unittests for bb.fetch2.URI class
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
2013-01-21 9:17 ` [PATCH 1/6] fetch2: Remove unused code in wget fetcher Olof Johansson
2013-01-21 9:17 ` [PATCH 2/6] fetch2: Add a class representing a generic URI Olof Johansson
@ 2013-01-21 9:18 ` Olof Johansson
2013-01-21 9:18 ` [PATCH 4/6] fetch2: Adapt encode/decode url to use URI class Olof Johansson
` (10 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-21 9:18 UTC (permalink / raw)
To: bitbake-devel
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/tests/fetch.py | 226 +++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 226 insertions(+)
diff --git lib/bb/tests/fetch.py lib/bb/tests/fetch.py
index 67eda59..e765ca7 100644
--- lib/bb/tests/fetch.py
+++ lib/bb/tests/fetch.py
@@ -21,8 +21,234 @@ import unittest
import tempfile
import subprocess
import os
+from bb.fetch2 import URI
import bb
+class URITest(unittest.TestCase):
+ test_uris = {
+ "http://www.google.com/index.html" : {
+ 'uri': 'http://www.google.com/index.html',
+ 'scheme': 'http',
+ 'hostname': 'www.google.com',
+ 'port': None,
+ 'hostport': 'www.google.com',
+ 'path': '/index.html',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': False
+ },
+ "http://www.google.com/index.html;param1=value1" : {
+ 'uri': 'http://www.google.com/index.html;param1=value1',
+ 'scheme': 'http',
+ 'hostname': 'www.google.com',
+ 'port': None,
+ 'hostport': 'www.google.com',
+ 'path': '/index.html',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {
+ 'param1': 'value1'
+ },
+ 'relative': False
+ },
+ "http://www.example.com:8080/index.html" : {
+ 'uri': 'http://www.example.com:8080/index.html',
+ 'scheme': 'http',
+ 'hostname': 'www.example.com',
+ 'port': 8080,
+ 'hostport': 'www.example.com:8080',
+ 'path': '/index.html',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': False
+ },
+ "cvs://anoncvs@cvs.handhelds.org/cvs;module=familiar/dist/ipkg" : {
+ 'uri': 'cvs://anoncvs@cvs.handhelds.org/cvs;module=familiar/dist/ipkg',
+ 'scheme': 'cvs',
+ 'hostname': 'cvs.handhelds.org',
+ 'port': None,
+ 'hostport': 'cvs.handhelds.org',
+ 'path': '/cvs',
+ 'userinfo': 'anoncvs',
+ 'username': 'anoncvs',
+ 'password': '',
+ 'params': {
+ 'module': 'familiar/dist/ipkg'
+ },
+ 'relative': False
+ },
+ "cvs://anoncvs:anonymous@cvs.handhelds.org/cvs;tag=V0-99-81;module=familiar/dist/ipkg": {
+ 'uri': 'cvs://anoncvs:anonymous@cvs.handhelds.org/cvs;tag=V0-99-81;module=familiar/dist/ipkg',
+ 'scheme': 'cvs',
+ 'hostname': 'cvs.handhelds.org',
+ 'port': None,
+ 'hostport': 'cvs.handhelds.org',
+ 'path': '/cvs',
+ 'userinfo': 'anoncvs:anonymous',
+ 'username': 'anoncvs',
+ 'password': 'anonymous',
+ 'params': {
+ 'tag': 'V0-99-81',
+ 'module': 'familiar/dist/ipkg'
+ },
+ 'relative': False
+ },
+ "file://example.diff": { # NOTE: Not RFC compliant!
+ 'uri': 'file:example.diff',
+ 'scheme': 'file',
+ 'hostname': '',
+ 'port': None,
+ 'hostport': '',
+ 'path': 'example.diff',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': True
+ },
+ "file:example.diff": { # NOTE: RFC compliant version of the former
+ 'uri': 'file:example.diff',
+ 'scheme': 'file',
+ 'hostname': '',
+ 'port': None,
+ 'hostport': '',
+ 'path': 'example.diff',
+ 'userinfo': '',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': True
+ },
+ "file:///tmp/example.diff": {
+ 'uri': 'file:///tmp/example.diff',
+ 'scheme': 'file',
+ 'hostname': '',
+ 'port': None,
+ 'hostport': '',
+ 'path': '/tmp/example.diff',
+ 'userinfo': '',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': False
+ },
+ "git:///path/example.git": {
+ 'uri': 'git:///path/example.git',
+ 'scheme': 'git',
+ 'hostname': '',
+ 'port': None,
+ 'hostport': '',
+ 'path': '/path/example.git',
+ 'userinfo': '',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': False
+ },
+ "git:path/example.git": {
+ 'uri': 'git:path/example.git',
+ 'scheme': 'git',
+ 'hostname': '',
+ 'port': None,
+ 'hostport': '',
+ 'path': 'path/example.git',
+ 'userinfo': '',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': True
+ },
+ "git://example.net/path/example.git": {
+ 'uri': 'git://example.net/path/example.git',
+ 'scheme': 'git',
+ 'hostname': 'example.net',
+ 'port': None,
+ 'hostport': 'example.net',
+ 'path': '/path/example.git',
+ 'userinfo': '',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': False
+ }
+ }
+
+ def test_uri(self):
+ for test_uri, ref in self.test_uris.items():
+ uri = URI(test_uri)
+
+ self.assertEqual(str(uri), ref['uri'])
+
+ # expected attributes
+ self.assertEqual(uri.scheme, ref['scheme'])
+
+ self.assertEqual(uri.userinfo, ref['userinfo'])
+ self.assertEqual(uri.username, ref['username'])
+ self.assertEqual(uri.password, ref['password'])
+
+ self.assertEqual(uri.hostname, ref['hostname'])
+ self.assertEqual(uri.port, ref['port'])
+ self.assertEqual(uri.hostport, ref['hostport'])
+
+ self.assertEqual(uri.path, ref['path'])
+ self.assertEqual(uri.params, ref['params'])
+
+ self.assertEqual(uri.relative, ref['relative'])
+
+ def test_dict(self):
+ for test in self.test_uris.values():
+ uri = URI()
+
+ self.assertEqual(uri.scheme, '')
+ self.assertEqual(uri.userinfo, '')
+ self.assertEqual(uri.username, '')
+ self.assertEqual(uri.password, '')
+ self.assertEqual(uri.hostname, '')
+ self.assertEqual(uri.port, None)
+ self.assertEqual(uri.path, '')
+ self.assertEqual(uri.params, {})
+
+
+ uri.scheme = test['scheme']
+ self.assertEqual(uri.scheme, test['scheme'])
+
+ uri.userinfo = test['userinfo']
+ self.assertEqual(uri.userinfo, test['userinfo'])
+ self.assertEqual(uri.username, test['username'])
+ self.assertEqual(uri.password, test['password'])
+
+ uri.hostname = test['hostname']
+ self.assertEqual(uri.hostname, test['hostname'])
+ self.assertEqual(uri.hostport, test['hostname'])
+
+ uri.port = test['port']
+ self.assertEqual(uri.port, test['port'])
+ self.assertEqual(uri.hostport, test['hostport'])
+
+ uri.path = test['path']
+ self.assertEqual(uri.path, test['path'])
+
+ uri.params = test['params']
+ self.assertEqual(uri.params, test['params'])
+
+ self.assertEqual(str(uri)+str(uri.relative), str(test['uri'])+str(test['relative']))
+
+ self.assertEqual(str(uri), test['uri'])
+
+ uri.params = {}
+ self.assertEqual(uri.params, {})
+ self.assertEqual(str(uri), (str(uri).split(";"))[0])
+
class FetcherTest(unittest.TestCase):
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 4/6] fetch2: Adapt encode/decode url to use URI class
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (2 preceding siblings ...)
2013-01-21 9:18 ` [PATCH 3/6] fetch2: unittests for bb.fetch2.URI class Olof Johansson
@ 2013-01-21 9:18 ` Olof Johansson
2013-01-21 9:18 ` [PATCH 5/6] fetch2: Add editor modelines for bb.tests.* Olof Johansson
` (9 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-21 9:18 UTC (permalink / raw)
To: bitbake-devel
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/fetch2/__init__.py | 66 ++++++++++++---------------------------------
1 file changed, 17 insertions(+), 49 deletions(-)
diff --git lib/bb/fetch2/__init__.py lib/bb/fetch2/__init__.py
index 3b83c0c..549e261 100644
--- lib/bb/fetch2/__init__.py
+++ lib/bb/fetch2/__init__.py
@@ -318,40 +318,9 @@ def decodeurl(url):
user, password, parameters).
"""
- m = re.compile('(?P<type>[^:]*)://((?P<user>.+)@)?(?P<location>[^;]+)(;(?P<parm>.*))?').match(url)
- if not m:
- raise MalformedUrl(url)
-
- type = m.group('type')
- location = m.group('location')
- if not location:
- raise MalformedUrl(url)
- user = m.group('user')
- parm = m.group('parm')
-
- locidx = location.find('/')
- if locidx != -1 and type.lower() != 'file':
- host = location[:locidx]
- path = location[locidx:]
- else:
- host = ""
- path = location
- if user:
- m = re.compile('(?P<user>[^:]+)(:?(?P<pswd>.*))').match(user)
- if m:
- user = m.group('user')
- pswd = m.group('pswd')
- else:
- user = ''
- pswd = ''
-
- p = {}
- if parm:
- for s in parm.split(';'):
- s1, s2 = s.split('=')
- p[s1] = s2
-
- return type, host, urllib.unquote(path), user, pswd, p
+ urlo = URI(url)
+ return (urlo.scheme, urlo.hostport, urlo.path,
+ urlo.username, urlo.password, urlo.params)
def encodeurl(decoded):
"""Encodes a URL from tokens (scheme, network location, path,
@@ -360,27 +329,26 @@ def encodeurl(decoded):
type, host, path, user, pswd, p = decoded
+ urlo = URI()
+
if not path:
raise MissingParameterError('path', "encoded from the data %s" % str(decoded))
if not type:
raise MissingParameterError('type', "encoded from the data %s" % str(decoded))
- url = '%s://' % type
- if user and type != "file":
- url += "%s" % user
- if pswd:
- url += ":%s" % pswd
- url += "@"
- if host and type != "file":
- url += "%s" % host
- # Standardise path to ensure comparisons work
- while '//' in path:
- path = path.replace("//", "/")
- url += "%s" % urllib.quote(path)
+
+ urlo.scheme = type
+ urlo.path = path
+
+ if host:
+ urlo.hostname = host
+ if user:
+ urlo.username = user
+ if pswd:
+ urlo.password = pswd
if p:
- for parm in p:
- url += ";%s=%s" % (parm, p[parm])
+ urlo.params = p
- return url
+ return str(urlo)
def uri_replace(ud, uri_find, uri_replace, replacements, d):
if not ud.url or not uri_find or not uri_replace:
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 5/6] fetch2: Add editor modelines for bb.tests.*
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (3 preceding siblings ...)
2013-01-21 9:18 ` [PATCH 4/6] fetch2: Adapt encode/decode url to use URI class Olof Johansson
@ 2013-01-21 9:18 ` Olof Johansson
2013-01-21 9:18 ` [PATCH 6/6] fetch2: Add SFTP fetcher Olof Johansson
` (8 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-21 9:18 UTC (permalink / raw)
To: bitbake-devel
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/tests/codeparser.py | 2 ++
lib/bb/tests/cow.py | 2 ++
lib/bb/tests/data.py | 2 ++
lib/bb/tests/fetch.py | 2 ++
lib/bb/tests/utils.py | 2 ++
5 files changed, 10 insertions(+)
diff --git lib/bb/tests/codeparser.py lib/bb/tests/codeparser.py
index 9b2d588..d2bd4a2 100644
--- lib/bb/tests/codeparser.py
+++ lib/bb/tests/codeparser.py
@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Test for codeparser.py
#
diff --git lib/bb/tests/cow.py lib/bb/tests/cow.py
index 599b2aa..35c5841 100644
--- lib/bb/tests/cow.py
+++ lib/bb/tests/cow.py
@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for Copy-on-Write (cow.py)
#
diff --git lib/bb/tests/data.py lib/bb/tests/data.py
index b2ab8ee..5161586 100644
--- lib/bb/tests/data.py
+++ lib/bb/tests/data.py
@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for the Data Store (data.py/data_smart.py)
#
diff --git lib/bb/tests/fetch.py lib/bb/tests/fetch.py
index e765ca7..778d94f 100644
--- lib/bb/tests/fetch.py
+++ lib/bb/tests/fetch.py
@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for the Fetcher (fetch2/)
#
diff --git lib/bb/tests/utils.py lib/bb/tests/utils.py
index 9f60491..7c50b1d 100644
--- lib/bb/tests/utils.py
+++ lib/bb/tests/utils.py
@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for utils.py
#
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH 6/6] fetch2: Add SFTP fetcher
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (4 preceding siblings ...)
2013-01-21 9:18 ` [PATCH 5/6] fetch2: Add editor modelines for bb.tests.* Olof Johansson
@ 2013-01-21 9:18 ` Olof Johansson
2013-01-28 14:13 ` Martin Jansa
2013-01-28 15:43 ` Martin Jansa
2013-01-28 14:05 ` [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Martin Jansa
` (7 subsequent siblings)
13 siblings, 2 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-21 9:18 UTC (permalink / raw)
To: bitbake-devel
This fetcher differs from the SSH fetcher in that it adheres more
strictly to the SECSH URI internet draft --- it uses the sftp://
instead of the ssh:// scheme, and it uses sftp instead of scp.
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/fetch2/__init__.py | 2 +
lib/bb/fetch2/sftp.py | 104 +++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 106 insertions(+)
create mode 100644 lib/bb/fetch2/sftp.py
diff --git lib/bb/fetch2/__init__.py lib/bb/fetch2/__init__.py
index 549e261..86302f7 100644
--- lib/bb/fetch2/__init__.py
+++ lib/bb/fetch2/__init__.py
@@ -1448,6 +1448,7 @@ from . import svn
from . import wget
from . import svk
from . import ssh
+from . import sftp
from . import perforce
from . import bzr
from . import hg
@@ -1461,6 +1462,7 @@ methods.append(git.Git())
methods.append(cvs.Cvs())
methods.append(svk.Svk())
methods.append(ssh.SSH())
+methods.append(sftp.SFTP())
methods.append(perforce.Perforce())
methods.append(bzr.Bzr())
methods.append(hg.Hg())
diff --git lib/bb/fetch2/sftp.py lib/bb/fetch2/sftp.py
new file mode 100644
index 0000000..0902ce7
--- /dev/null
+++ lib/bb/fetch2/sftp.py
@@ -0,0 +1,104 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake SFTP Fetch implementation
+
+Class for fetching files via SFTP. It tries to adhere to the (now
+expired) IETF Internet Draft for "Uniform Resource Identifier (URI)
+Scheme for Secure File Transfer Protocol (SFTP) and Secure Shell
+(SSH)" (SECSH URI).
+
+It uses SFTP (as to adhere to the SECSH URI specification). It only
+supports key based authentication, not password. This class, unlike
+the SSH fetcher, does not support fetching a directory tree from the
+remote.
+
+ http://tools.ietf.org/html/draft-ietf-secsh-scp-sftp-ssh-uri-04
+ https://www.iana.org/assignments/uri-schemes/prov/sftp
+ https://tools.ietf.org/html/draft-ietf-secsh-filexfer-13
+
+"""
+
+# Copyright (C) 2013, Olof Johansson <olof.johansson@axis.com>
+#
+# Based in part on bb.fetch2.wget:
+# Copyright (C) 2003, 2004 Chris Larson
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# Based on functions from the base bb module, Copyright 2003 Holger Schurig
+
+import os
+import bb
+import urllib
+import commands
+from bb import data
+from bb.fetch2 import URI
+from bb.fetch2 import FetchMethod
+from bb.fetch2 import runfetchcmd
+
+
+class SFTP(FetchMethod):
+ """Class to fetch urls via 'sftp'"""
+ def supports(self, url, ud, d):
+ """
+ Check to see if a given url can be fetched with sftp.
+ """
+ return ud.type in ['sftp']
+
+ def recommends_checksum(self, urldata):
+ return True
+
+ def urldata_init(self, ud, d):
+ if 'protocol' in ud.parm and ud.parm['protocol'] == 'git':
+ raise bb.fetch2.ParameterError(
+ "Invalid protocol - if you wish to fetch from a git " +
+ "repository using ssh, you need to instead use the " +
+ "git:// prefix with protocol=ssh", ud.url)
+
+ if 'downloadfilename' in ud.parm:
+ ud.basename = ud.parm['downloadfilename']
+ else:
+ ud.basename = os.path.basename(ud.path)
+
+ ud.localfile = data.expand(urllib.unquote(ud.basename), d)
+
+ def download(self, uri, ud, d):
+ """Fetch urls"""
+
+ urlo = URI(uri)
+ basecmd = 'sftp -oPasswordAuthentication=no'
+ port = ''
+ if urlo.port:
+ port = '-P %d' % urlo.port
+ urlo.port = None
+
+ dldir = data.getVar('DL_DIR', d, True)
+ lpath = os.path.join(dldir, ud.localfile)
+
+ user = ''
+ if urlo.userinfo:
+ user = urlo.userinfo + '@'
+
+ remote = '%s%s:%s' % (user, urlo.hostname, urlo.path)
+
+ cmd = '%s %s %s %s' % (basecmd, port, commands.mkarg(remote),
+ commands.mkarg(lpath))
+
+ bb.fetch2.check_network_access(d, cmd, uri)
+ runfetchcmd(cmd, d)
+ return True
+
+ def checkstatus(self, uri, ud, d):
+ return self.download(uri, ud, d, True)
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (5 preceding siblings ...)
2013-01-21 9:18 ` [PATCH 6/6] fetch2: Add SFTP fetcher Olof Johansson
@ 2013-01-28 14:05 ` Martin Jansa
2013-01-28 14:23 ` Olof Johansson
2013-01-29 7:50 ` [PATCH v2 0/6] fetch2: Updated SFTP fetcher Olof Johansson
` (6 subsequent siblings)
13 siblings, 1 reply; 23+ messages in thread
From: Martin Jansa @ 2013-01-28 14:05 UTC (permalink / raw)
To: Olof Johansson; +Cc: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 3181 bytes --]
On Mon, Jan 21, 2013 at 10:17:57AM +0100, Olof Johansson wrote:
> Hi,
>
> In our early work in migrating to poky we realized that bitbake's current SSH
> fetcher doesn't really suite our needs. It suffers from some specification
> conformance bugs --- some which may be easily worked around. However, this
> patch series instead introduces a new "SFTP" fetcher, meant to in a way,
> replace the SSH fetcher, with stricter conformance to the SECSH URI spec [1].
>
> The SSH fetcher has its own URI parsing, and can not rely on
> bb.fetch2.decodeurl. I found that decodeurl was hard to extend, with it
> returning a tuple. This prompted me to write a generic URI class, with
> attributes for the various URI components (and some general bitbake kludges,
> like support relative file:// uris).
>
> I realize that these changes are a bit bold and non-trivial, but I hope you'll
> find time to review them and point out problems to fix. If my reasoning is
> completely wrong, please let me know and I can try to solve our direct problems
> with the SSH fetcher with minimal changes instead.
>
> The changes come with unittests for the URI class; it passes all the tests in
> bb.tests.fetch and all other tests under bb.tests, except
> bb.tests.data.DataExpansions, but the same errors are present on master as
> well. I've also tested running bitbake core-image-minimal in a new tree without
> any issues.
>
Can you please rebase your patchset on latest master?
2nd patch conflicts with
commit 22bd19d208f0251f5a1f9b98f3cac66181f3fc07
Author: Tyler Hall <tylerwhall@gmail.com>
Date: Wed Dec 19 18:26:50 2012 -0500
fetch2: Sort file checksums by value, not path
Are those patches generated from bitbake repo?
They start with 'lib/' so patch -p0 is needed instead of default -p1 and
cannot be applied directly from pw-am.sh :/.
Or please provide URL to your bitbake branch.
Cheers,
> Olof Johansson (6):
> fetch2: Remove unused code in wget fetcher
> fetch2: Add a class representing a generic URI
> fetch2: unittests for bb.fetch2.URI class
> fetch2: Adapt encode/decode url to use URI class
> fetch2: Add editor modelines for bb.tests.*
> fetch2: Add SFTP fetcher
>
> lib/bb/fetch2/__init__.py | 262 +++++++++++++++++++++++++++++++++++---------
> lib/bb/fetch2/sftp.py | 104 ++++++++++++++++++
> lib/bb/fetch2/wget.py | 5 -
> lib/bb/tests/codeparser.py | 2 +
> lib/bb/tests/cow.py | 2 +
> lib/bb/tests/data.py | 2 +
> lib/bb/tests/fetch.py | 228 ++++++++++++++++++++++++++++++++++++++
> lib/bb/tests/utils.py | 2 +
> 8 files changed, 553 insertions(+), 54 deletions(-)
> create mode 100644 lib/bb/fetch2/sftp.py
>
>
> References:
> 1: http://tools.ietf.org/html/draft-ietf-secsh-scp-sftp-ssh-uri-04
>
> --
> Olof Johansson
> Tools engineer
> (IRC: zibri on Freenode)
>
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
--
Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 6/6] fetch2: Add SFTP fetcher
2013-01-21 9:18 ` [PATCH 6/6] fetch2: Add SFTP fetcher Olof Johansson
@ 2013-01-28 14:13 ` Martin Jansa
2013-01-28 14:20 ` Olof Johansson
2013-01-28 15:43 ` Martin Jansa
1 sibling, 1 reply; 23+ messages in thread
From: Martin Jansa @ 2013-01-28 14:13 UTC (permalink / raw)
To: Olof Johansson; +Cc: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 5632 bytes --]
On Mon, Jan 21, 2013 at 10:18:03AM +0100, Olof Johansson wrote:
> This fetcher differs from the SSH fetcher in that it adheres more
> strictly to the SECSH URI internet draft --- it uses the sftp://
> instead of the ssh:// scheme, and it uses sftp instead of scp.
>
> Signed-off-by: Olof Johansson <olof.johansson@axis.com>
> ---
> lib/bb/fetch2/__init__.py | 2 +
> lib/bb/fetch2/sftp.py | 104 +++++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 106 insertions(+)
> create mode 100644 lib/bb/fetch2/sftp.py
>
> diff --git lib/bb/fetch2/__init__.py lib/bb/fetch2/__init__.py
> index 549e261..86302f7 100644
> --- lib/bb/fetch2/__init__.py
> +++ lib/bb/fetch2/__init__.py
> @@ -1448,6 +1448,7 @@ from . import svn
> from . import wget
> from . import svk
> from . import ssh
> +from . import sftp
> from . import perforce
> from . import bzr
> from . import hg
> @@ -1461,6 +1462,7 @@ methods.append(git.Git())
> methods.append(cvs.Cvs())
> methods.append(svk.Svk())
> methods.append(ssh.SSH())
> +methods.append(sftp.SFTP())
> methods.append(perforce.Perforce())
> methods.append(bzr.Bzr())
> methods.append(hg.Hg())
> diff --git lib/bb/fetch2/sftp.py lib/bb/fetch2/sftp.py
> new file mode 100644
> index 0000000..0902ce7
> --- /dev/null
> +++ lib/bb/fetch2/sftp.py
> @@ -0,0 +1,104 @@
> +# ex:ts=4:sw=4:sts=4:et
> +# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +"""
> +BitBake SFTP Fetch implementation
> +
> +Class for fetching files via SFTP. It tries to adhere to the (now
> +expired) IETF Internet Draft for "Uniform Resource Identifier (URI)
> +Scheme for Secure File Transfer Protocol (SFTP) and Secure Shell
> +(SSH)" (SECSH URI).
> +
> +It uses SFTP (as to adhere to the SECSH URI specification). It only
> +supports key based authentication, not password. This class, unlike
> +the SSH fetcher, does not support fetching a directory tree from the
> +remote.
> +
> + http://tools.ietf.org/html/draft-ietf-secsh-scp-sftp-ssh-uri-04
> + https://www.iana.org/assignments/uri-schemes/prov/sftp
> + https://tools.ietf.org/html/draft-ietf-secsh-filexfer-13
> +
> +"""
> +
> +# Copyright (C) 2013, Olof Johansson <olof.johansson@axis.com>
> +#
> +# Based in part on bb.fetch2.wget:
> +# Copyright (C) 2003, 2004 Chris Larson
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License version 2 as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with this program; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +#
> +# Based on functions from the base bb module, Copyright 2003 Holger Schurig
> +
> +import os
> +import bb
> +import urllib
> +import commands
> +from bb import data
> +from bb.fetch2 import URI
> +from bb.fetch2 import FetchMethod
> +from bb.fetch2 import runfetchcmd
> +
> +
> +class SFTP(FetchMethod):
> + """Class to fetch urls via 'sftp'"""
> + def supports(self, url, ud, d):
> + """
> + Check to see if a given url can be fetched with sftp.
> + """
> + return ud.type in ['sftp']
> +
> + def recommends_checksum(self, urldata):
> + return True
> +
> + def urldata_init(self, ud, d):
> + if 'protocol' in ud.parm and ud.parm['protocol'] == 'git':
> + raise bb.fetch2.ParameterError(
> + "Invalid protocol - if you wish to fetch from a git " +
> + "repository using ssh, you need to instead use the " +
extra 'instead' ^ ?
But I like this error, will add the same to ssh.py fetcher (which I'm
now trying to fix).
> + "git:// prefix with protocol=ssh", ud.url)
> +
> + if 'downloadfilename' in ud.parm:
> + ud.basename = ud.parm['downloadfilename']
> + else:
> + ud.basename = os.path.basename(ud.path)
> +
> + ud.localfile = data.expand(urllib.unquote(ud.basename), d)
> +
> + def download(self, uri, ud, d):
> + """Fetch urls"""
> +
> + urlo = URI(uri)
> + basecmd = 'sftp -oPasswordAuthentication=no'
> + port = ''
> + if urlo.port:
> + port = '-P %d' % urlo.port
> + urlo.port = None
> +
> + dldir = data.getVar('DL_DIR', d, True)
> + lpath = os.path.join(dldir, ud.localfile)
> +
> + user = ''
> + if urlo.userinfo:
> + user = urlo.userinfo + '@'
> +
> + remote = '%s%s:%s' % (user, urlo.hostname, urlo.path)
> +
> + cmd = '%s %s %s %s' % (basecmd, port, commands.mkarg(remote),
> + commands.mkarg(lpath))
> +
> + bb.fetch2.check_network_access(d, cmd, uri)
> + runfetchcmd(cmd, d)
> + return True
> +
> + def checkstatus(self, uri, ud, d):
> + return self.download(uri, ud, d, True)
> --
> 1.7.10.4
>
>
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
--
Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 6/6] fetch2: Add SFTP fetcher
2013-01-28 14:13 ` Martin Jansa
@ 2013-01-28 14:20 ` Olof Johansson
0 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-28 14:20 UTC (permalink / raw)
To: Martin Jansa; +Cc: bitbake-devel@lists.openembedded.org
On 2013-01-28 15:13, Martin Jansa wrote:
> On Mon, Jan 21, 2013 at 10:18:03AM +0100, Olof Johansson wrote:
> > + raise bb.fetch2.ParameterError(
> > + "Invalid protocol - if you wish to fetch from a git " +
> > + "repository using ssh, you need to instead use the " +
>
> extra 'instead' ^ ?
>
> But I like this error, will add the same to ssh.py fetcher (which I'm
> now trying to fix).
That is shamelessly stolen from the wget fetcher :-). The
"instead" may be redundant, can remove it.
--
Olof Johansson
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl
2013-01-28 14:05 ` [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Martin Jansa
@ 2013-01-28 14:23 ` Olof Johansson
0 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-28 14:23 UTC (permalink / raw)
To: Martin Jansa; +Cc: bitbake-devel@lists.openembedded.org
On 2013-01-28 15:05, Martin Jansa wrote:
> Can you please rebase your patchset on latest master?
>
> 2nd patch conflicts with
> commit 22bd19d208f0251f5a1f9b98f3cac66181f3fc07
> Author: Tyler Hall <tylerwhall@gmail.com>
> Date: Wed Dec 19 18:26:50 2012 -0500
> fetch2: Sort file checksums by value, not path
>
> Are those patches generated from bitbake repo?
> They start with 'lib/' so patch -p0 is needed instead of default -p1 and
> cannot be applied directly from pw-am.sh :/.
Sorry, will fix, but not until later this evening unfortunately.
(I've set `git config diff.noprefix=true` without realizing the
implications in cases like this. Will remove it.)
--
Olof Johansson
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 6/6] fetch2: Add SFTP fetcher
2013-01-21 9:18 ` [PATCH 6/6] fetch2: Add SFTP fetcher Olof Johansson
2013-01-28 14:13 ` Martin Jansa
@ 2013-01-28 15:43 ` Martin Jansa
1 sibling, 0 replies; 23+ messages in thread
From: Martin Jansa @ 2013-01-28 15:43 UTC (permalink / raw)
To: Olof Johansson; +Cc: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 5870 bytes --]
On Mon, Jan 21, 2013 at 10:18:03AM +0100, Olof Johansson wrote:
> This fetcher differs from the SSH fetcher in that it adheres more
> strictly to the SECSH URI internet draft --- it uses the sftp://
> instead of the ssh:// scheme, and it uses sftp instead of scp.
>
> Signed-off-by: Olof Johansson <olof.johansson@axis.com>
> ---
> lib/bb/fetch2/__init__.py | 2 +
> lib/bb/fetch2/sftp.py | 104 +++++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 106 insertions(+)
> create mode 100644 lib/bb/fetch2/sftp.py
>
> diff --git lib/bb/fetch2/__init__.py lib/bb/fetch2/__init__.py
> index 549e261..86302f7 100644
> --- lib/bb/fetch2/__init__.py
> +++ lib/bb/fetch2/__init__.py
> @@ -1448,6 +1448,7 @@ from . import svn
> from . import wget
> from . import svk
> from . import ssh
> +from . import sftp
> from . import perforce
> from . import bzr
> from . import hg
> @@ -1461,6 +1462,7 @@ methods.append(git.Git())
> methods.append(cvs.Cvs())
> methods.append(svk.Svk())
> methods.append(ssh.SSH())
> +methods.append(sftp.SFTP())
> methods.append(perforce.Perforce())
> methods.append(bzr.Bzr())
> methods.append(hg.Hg())
> diff --git lib/bb/fetch2/sftp.py lib/bb/fetch2/sftp.py
> new file mode 100644
> index 0000000..0902ce7
> --- /dev/null
> +++ lib/bb/fetch2/sftp.py
> @@ -0,0 +1,104 @@
> +# ex:ts=4:sw=4:sts=4:et
> +# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +"""
> +BitBake SFTP Fetch implementation
> +
> +Class for fetching files via SFTP. It tries to adhere to the (now
> +expired) IETF Internet Draft for "Uniform Resource Identifier (URI)
> +Scheme for Secure File Transfer Protocol (SFTP) and Secure Shell
> +(SSH)" (SECSH URI).
> +
> +It uses SFTP (as to adhere to the SECSH URI specification). It only
> +supports key based authentication, not password. This class, unlike
> +the SSH fetcher, does not support fetching a directory tree from the
> +remote.
> +
> + http://tools.ietf.org/html/draft-ietf-secsh-scp-sftp-ssh-uri-04
> + https://www.iana.org/assignments/uri-schemes/prov/sftp
> + https://tools.ietf.org/html/draft-ietf-secsh-filexfer-13
> +
> +"""
> +
> +# Copyright (C) 2013, Olof Johansson <olof.johansson@axis.com>
> +#
> +# Based in part on bb.fetch2.wget:
> +# Copyright (C) 2003, 2004 Chris Larson
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License version 2 as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with this program; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +#
> +# Based on functions from the base bb module, Copyright 2003 Holger Schurig
> +
> +import os
> +import bb
> +import urllib
> +import commands
> +from bb import data
> +from bb.fetch2 import URI
> +from bb.fetch2 import FetchMethod
> +from bb.fetch2 import runfetchcmd
> +
> +
> +class SFTP(FetchMethod):
> + """Class to fetch urls via 'sftp'"""
> + def supports(self, url, ud, d):
> + """
> + Check to see if a given url can be fetched with sftp.
> + """
> + return ud.type in ['sftp']
> +
> + def recommends_checksum(self, urldata):
> + return True
> +
> + def urldata_init(self, ud, d):
> + if 'protocol' in ud.parm and ud.parm['protocol'] == 'git':
> + raise bb.fetch2.ParameterError(
> + "Invalid protocol - if you wish to fetch from a git " +
> + "repository using ssh, you need to instead use the " +
> + "git:// prefix with protocol=ssh", ud.url)
> +
> + if 'downloadfilename' in ud.parm:
> + ud.basename = ud.parm['downloadfilename']
> + else:
> + ud.basename = os.path.basename(ud.path)
> +
> + ud.localfile = data.expand(urllib.unquote(ud.basename), d)
> +
> + def download(self, uri, ud, d):
> + """Fetch urls"""
> +
> + urlo = URI(uri)
> + basecmd = 'sftp -oPasswordAuthentication=no'
> + port = ''
> + if urlo.port:
> + port = '-P %d' % urlo.port
> + urlo.port = None
> +
> + dldir = data.getVar('DL_DIR', d, True)
> + lpath = os.path.join(dldir, ud.localfile)
> +
> + user = ''
> + if urlo.userinfo:
> + user = urlo.userinfo + '@'
> +
> + remote = '%s%s:%s' % (user, urlo.hostname, urlo.path)
Please add some SRC_URI examples, it always surprise me that there isn't
':' as host-path separator, but '/' and error as returned from bitbake
ia not so easy to read.
Also it looks like optional ~ syntax is not supported
sftp://user@host.example.com/~/file.txt
But it should be easy to implement if you remove '^/~/' from urlo.path.
Cheers,
> +
> + cmd = '%s %s %s %s' % (basecmd, port, commands.mkarg(remote),
> + commands.mkarg(lpath))
> +
> + bb.fetch2.check_network_access(d, cmd, uri)
> + runfetchcmd(cmd, d)
> + return True
> +
> + def checkstatus(self, uri, ud, d):
> + return self.download(uri, ud, d, True)
> --
> 1.7.10.4
>
>
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
--
Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v2 0/6] fetch2: Updated SFTP fetcher
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (6 preceding siblings ...)
2013-01-28 14:05 ` [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Martin Jansa
@ 2013-01-29 7:50 ` Olof Johansson
2013-01-30 7:55 ` Olof Johansson
2013-01-29 7:50 ` [PATCH v2 1/6] fetch2: Remove unused code in wget fetcher Olof Johansson
` (5 subsequent siblings)
13 siblings, 1 reply; 23+ messages in thread
From: Olof Johansson @ 2013-01-29 7:50 UTC (permalink / raw)
To: bitbake-devel
Changes since the first draft (thanks Martin Jansa for the feedback):
* Rebase against current master
* Regenerate diffs without git config diff.noprefix
* Support /~/ (home relative URIs) in SFTP fetcher
* Add an example section in the SFTP fetcher documentation
* Remove "checkstatus" method from SFTP fetcher (wasn't meant to be there)
Olof Johansson (6):
fetch2: Remove unused code in wget fetcher
fetch2: Add a class representing a generic URI
fetch2: unittests for bb.fetch2.URI class
fetch2: Adapt encode/decode url to use URI class
fetch2: Add editor modelines for bb.tests.*
fetch2: Add SFTP fetcher
lib/bb/fetch2/__init__.py | 262 +++++++++++++++++++++++++++++++++++---------
lib/bb/fetch2/sftp.py | 129 ++++++++++++++++++++++
lib/bb/fetch2/wget.py | 5 -
lib/bb/tests/codeparser.py | 2 +
lib/bb/tests/cow.py | 2 +
lib/bb/tests/data.py | 2 +
lib/bb/tests/fetch.py | 228 ++++++++++++++++++++++++++++++++++++++
lib/bb/tests/utils.py | 2 +
8 files changed, 578 insertions(+), 54 deletions(-)
create mode 100644 lib/bb/fetch2/sftp.py
--
1.7.10.4
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v2 1/6] fetch2: Remove unused code in wget fetcher
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (7 preceding siblings ...)
2013-01-29 7:50 ` [PATCH v2 0/6] fetch2: Updated SFTP fetcher Olof Johansson
@ 2013-01-29 7:50 ` Olof Johansson
2013-01-29 7:50 ` [PATCH v2 2/6] fetch2: Add a class representing a generic URI Olof Johansson
` (4 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-29 7:50 UTC (permalink / raw)
To: bitbake-devel
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/fetch2/wget.py | 5 -----
1 file changed, 5 deletions(-)
diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index 2808df6..1fafc4a 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -32,8 +32,6 @@ import urllib
from bb import data
from bb.fetch2 import FetchMethod
from bb.fetch2 import FetchError
-from bb.fetch2 import encodeurl
-from bb.fetch2 import decodeurl
from bb.fetch2 import logger
from bb.fetch2 import runfetchcmd
@@ -77,9 +75,6 @@ class Wget(FetchMethod):
fetchcmd = d.getVar("FETCHCOMMAND_wget", True) or d.expand(basecmd + " -P ${DL_DIR} '${URI}'")
uri = uri.split(";")[0]
- uri_decoded = list(decodeurl(uri))
- uri_type = uri_decoded[0]
- uri_host = uri_decoded[1]
fetchcmd = fetchcmd.replace("${URI}", uri.split(";")[0])
fetchcmd = fetchcmd.replace("${FILE}", ud.basename)
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v2 2/6] fetch2: Add a class representing a generic URI
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (8 preceding siblings ...)
2013-01-29 7:50 ` [PATCH v2 1/6] fetch2: Remove unused code in wget fetcher Olof Johansson
@ 2013-01-29 7:50 ` Olof Johansson
2013-01-29 7:50 ` [PATCH v2 3/6] fetch2: unittests for bb.fetch2.URI class Olof Johansson
` (3 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-29 7:50 UTC (permalink / raw)
To: bitbake-devel
A class representing a generic URI, with methods for accessing the URI
components, and stringifies to the URI. This class should be a bit more
flexible than the existing {encode,decode}_url functions in that it
supports more components (e.g. port) and that it does not rely on a
specific order on the return values. This makes it easy to add new
properties without affecting the API.
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/fetch2/__init__.py | 194 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 194 insertions(+)
diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 4746356..7f44976 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -30,6 +30,7 @@ from __future__ import print_function
import os, re
import logging
import urllib
+from urlparse import urlparse
import operator
import bb.persist_data, bb.utils
import bb.checksum
@@ -120,6 +121,199 @@ class NonLocalMethod(Exception):
def __init__(self):
Exception.__init__(self)
+
+class URI(object):
+ """
+ A class representing a generic URI, with methods for
+ accessing the URI components, and stringifies to the
+ URI.
+
+ It is constructed by calling it with a URI, or setting
+ the attributes manually:
+
+ uri = URI("http://example.com/")
+
+ uri = URI()
+ uri.scheme = 'http'
+ uri.hostname = 'example.com'
+ uri.path = '/'
+
+ It has the following attributes:
+
+ * scheme (read/write)
+ * userinfo (authentication information) (read/write)
+ * username (read/write)
+ * password (read/write)
+
+ Note, password is deprecated as of RFC 3986.
+
+ * hostname (read/write)
+ * port (read/write)
+ * hostport (read only)
+ "hostname:port", if both are set, otherwise just "hostname"
+ * path (read/write)
+ * path_quoted (read/write)
+ A URI quoted version of path
+ * params (dict) (read/write)
+ * relative (bool) (read only)
+ True if this is a "relative URI", (e.g. file:foo.diff)
+
+ It stringifies to the URI itself.
+
+ Some notes about relative URIs: while it's specified that
+ a URI beginning with <scheme>:// should either be directly
+ followed by a hostname or a /, the old URI handling of the
+ fetch2 library did not comform to this. Therefore, this URI
+ class has some kludges to make sure that URIs are parsed in
+ a way comforming to bitbake's current usage. This URI class
+ supports the following:
+
+ file:relative/path.diff (IETF compliant)
+ git:relative/path.git (IETF compliant)
+ git:///absolute/path.git (IETF compliant)
+ file:///absolute/path.diff (IETF compliant)
+
+ file://relative/path.diff (not IETF compliant)
+
+ But it does not support the following:
+
+ file://hostname/absolute/path.diff (would be IETF compliant)
+
+ Note that the last case only applies to a list of
+ "whitelisted" schemes (currently only file://), that requires
+ its URIs to not have a network location.
+ """
+
+ _relative_schemes = ['file', 'git']
+ _netloc_forbidden = ['file']
+
+ def __init__(self, uri=None):
+ self.scheme = ''
+ self.userinfo = ''
+ self.hostname = ''
+ self.port = None
+ self._path = ''
+ self.params = {}
+ self.relative = False
+
+ if not uri:
+ return
+
+ urlp = urlparse(uri)
+ self.scheme = urlp.scheme
+
+ # Convert URI to be relative
+ if urlp.scheme in self._netloc_forbidden:
+ uri = re.sub("(?<=:)//(?!/)", "", uri, 1)
+ urlp = urlparse(uri)
+
+ # Identify if the URI is relative or not
+ if urlp.scheme in self._relative_schemes and \
+ re.compile("^\w+:(?!//)").match(uri):
+ self.relative = True
+
+ if not self.relative:
+ self.hostname = urlp.hostname or ''
+ self.port = urlp.port
+
+ self.userinfo += urlp.username or ''
+
+ if urlp.password:
+ self.userinfo += ':%s' % urlp.password
+
+ # Do support params even for URI schemes that Python's
+ # urlparse doesn't support params for.
+ path = ''
+ param_str = ''
+ if not urlp.params:
+ path, param_str = (list(urlp.path.split(";", 1)) + [None])[:2]
+ else:
+ path = urlp.path
+ param_str = urlp.params
+
+ self.path = urllib.unquote(path)
+
+ if param_str:
+ self.params = self._param_dict(param_str)
+
+ def __str__(self):
+ userinfo = self.userinfo
+ if userinfo:
+ userinfo += '@'
+
+ return "%s:%s%s%s%s%s" % (
+ self.scheme,
+ '' if self.relative else '//',
+ userinfo,
+ self.hostport,
+ self.path_quoted,
+ self._param_str)
+
+ @property
+ def _param_str(self):
+ ret = ''
+ for key, val in self.params.items():
+ ret += ";%s=%s" % (key, val)
+ return ret
+
+ def _param_dict(self, param_str):
+ parm = {}
+
+ for keyval in param_str.split(";"):
+ key, val = keyval.split("=", 1)
+ parm[key] = val
+
+ return parm
+
+ @property
+ def hostport(self):
+ if not self.port:
+ return self.hostname
+ return "%s:%d" % (self.hostname, self.port)
+
+ @property
+ def path_quoted(self):
+ return urllib.quote(self.path)
+
+ @path_quoted.setter
+ def path_quoted(self, path):
+ self.path = urllib.unquote(path)
+
+ @property
+ def path(self):
+ return self._path
+
+ @path.setter
+ def path(self, path):
+ self._path = path
+
+ if re.compile("^/").match(path):
+ self.relative = False
+ else:
+ self.relative = True
+
+ @property
+ def username(self):
+ if self.userinfo:
+ return (self.userinfo.split(":", 1))[0]
+ return ''
+
+ @username.setter
+ def username(self, username):
+ self.userinfo = username
+ if self.password:
+ self.userinfo += ":%s" % self.password
+
+ @property
+ def password(self):
+ if self.userinfo and ":" in self.userinfo:
+ return (self.userinfo.split(":", 1))[1]
+ return ''
+
+ @password.setter
+ def password(self, password):
+ self.userinfo = "%s:%s" % (self.username, password)
+
def decodeurl(url):
"""Decodes an URL into the tokens (scheme, network location, path,
user, password, parameters).
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v2 3/6] fetch2: unittests for bb.fetch2.URI class
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (9 preceding siblings ...)
2013-01-29 7:50 ` [PATCH v2 2/6] fetch2: Add a class representing a generic URI Olof Johansson
@ 2013-01-29 7:50 ` Olof Johansson
2013-01-29 7:50 ` [PATCH v2 4/6] fetch2: Adapt encode/decode url to use URI class Olof Johansson
` (2 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-29 7:50 UTC (permalink / raw)
To: bitbake-devel
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/tests/fetch.py | 226 +++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 226 insertions(+)
diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 67eda59..e765ca7 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -21,8 +21,234 @@ import unittest
import tempfile
import subprocess
import os
+from bb.fetch2 import URI
import bb
+class URITest(unittest.TestCase):
+ test_uris = {
+ "http://www.google.com/index.html" : {
+ 'uri': 'http://www.google.com/index.html',
+ 'scheme': 'http',
+ 'hostname': 'www.google.com',
+ 'port': None,
+ 'hostport': 'www.google.com',
+ 'path': '/index.html',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': False
+ },
+ "http://www.google.com/index.html;param1=value1" : {
+ 'uri': 'http://www.google.com/index.html;param1=value1',
+ 'scheme': 'http',
+ 'hostname': 'www.google.com',
+ 'port': None,
+ 'hostport': 'www.google.com',
+ 'path': '/index.html',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {
+ 'param1': 'value1'
+ },
+ 'relative': False
+ },
+ "http://www.example.com:8080/index.html" : {
+ 'uri': 'http://www.example.com:8080/index.html',
+ 'scheme': 'http',
+ 'hostname': 'www.example.com',
+ 'port': 8080,
+ 'hostport': 'www.example.com:8080',
+ 'path': '/index.html',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': False
+ },
+ "cvs://anoncvs@cvs.handhelds.org/cvs;module=familiar/dist/ipkg" : {
+ 'uri': 'cvs://anoncvs@cvs.handhelds.org/cvs;module=familiar/dist/ipkg',
+ 'scheme': 'cvs',
+ 'hostname': 'cvs.handhelds.org',
+ 'port': None,
+ 'hostport': 'cvs.handhelds.org',
+ 'path': '/cvs',
+ 'userinfo': 'anoncvs',
+ 'username': 'anoncvs',
+ 'password': '',
+ 'params': {
+ 'module': 'familiar/dist/ipkg'
+ },
+ 'relative': False
+ },
+ "cvs://anoncvs:anonymous@cvs.handhelds.org/cvs;tag=V0-99-81;module=familiar/dist/ipkg": {
+ 'uri': 'cvs://anoncvs:anonymous@cvs.handhelds.org/cvs;tag=V0-99-81;module=familiar/dist/ipkg',
+ 'scheme': 'cvs',
+ 'hostname': 'cvs.handhelds.org',
+ 'port': None,
+ 'hostport': 'cvs.handhelds.org',
+ 'path': '/cvs',
+ 'userinfo': 'anoncvs:anonymous',
+ 'username': 'anoncvs',
+ 'password': 'anonymous',
+ 'params': {
+ 'tag': 'V0-99-81',
+ 'module': 'familiar/dist/ipkg'
+ },
+ 'relative': False
+ },
+ "file://example.diff": { # NOTE: Not RFC compliant!
+ 'uri': 'file:example.diff',
+ 'scheme': 'file',
+ 'hostname': '',
+ 'port': None,
+ 'hostport': '',
+ 'path': 'example.diff',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': True
+ },
+ "file:example.diff": { # NOTE: RFC compliant version of the former
+ 'uri': 'file:example.diff',
+ 'scheme': 'file',
+ 'hostname': '',
+ 'port': None,
+ 'hostport': '',
+ 'path': 'example.diff',
+ 'userinfo': '',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': True
+ },
+ "file:///tmp/example.diff": {
+ 'uri': 'file:///tmp/example.diff',
+ 'scheme': 'file',
+ 'hostname': '',
+ 'port': None,
+ 'hostport': '',
+ 'path': '/tmp/example.diff',
+ 'userinfo': '',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': False
+ },
+ "git:///path/example.git": {
+ 'uri': 'git:///path/example.git',
+ 'scheme': 'git',
+ 'hostname': '',
+ 'port': None,
+ 'hostport': '',
+ 'path': '/path/example.git',
+ 'userinfo': '',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': False
+ },
+ "git:path/example.git": {
+ 'uri': 'git:path/example.git',
+ 'scheme': 'git',
+ 'hostname': '',
+ 'port': None,
+ 'hostport': '',
+ 'path': 'path/example.git',
+ 'userinfo': '',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': True
+ },
+ "git://example.net/path/example.git": {
+ 'uri': 'git://example.net/path/example.git',
+ 'scheme': 'git',
+ 'hostname': 'example.net',
+ 'port': None,
+ 'hostport': 'example.net',
+ 'path': '/path/example.git',
+ 'userinfo': '',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'relative': False
+ }
+ }
+
+ def test_uri(self):
+ for test_uri, ref in self.test_uris.items():
+ uri = URI(test_uri)
+
+ self.assertEqual(str(uri), ref['uri'])
+
+ # expected attributes
+ self.assertEqual(uri.scheme, ref['scheme'])
+
+ self.assertEqual(uri.userinfo, ref['userinfo'])
+ self.assertEqual(uri.username, ref['username'])
+ self.assertEqual(uri.password, ref['password'])
+
+ self.assertEqual(uri.hostname, ref['hostname'])
+ self.assertEqual(uri.port, ref['port'])
+ self.assertEqual(uri.hostport, ref['hostport'])
+
+ self.assertEqual(uri.path, ref['path'])
+ self.assertEqual(uri.params, ref['params'])
+
+ self.assertEqual(uri.relative, ref['relative'])
+
+ def test_dict(self):
+ for test in self.test_uris.values():
+ uri = URI()
+
+ self.assertEqual(uri.scheme, '')
+ self.assertEqual(uri.userinfo, '')
+ self.assertEqual(uri.username, '')
+ self.assertEqual(uri.password, '')
+ self.assertEqual(uri.hostname, '')
+ self.assertEqual(uri.port, None)
+ self.assertEqual(uri.path, '')
+ self.assertEqual(uri.params, {})
+
+
+ uri.scheme = test['scheme']
+ self.assertEqual(uri.scheme, test['scheme'])
+
+ uri.userinfo = test['userinfo']
+ self.assertEqual(uri.userinfo, test['userinfo'])
+ self.assertEqual(uri.username, test['username'])
+ self.assertEqual(uri.password, test['password'])
+
+ uri.hostname = test['hostname']
+ self.assertEqual(uri.hostname, test['hostname'])
+ self.assertEqual(uri.hostport, test['hostname'])
+
+ uri.port = test['port']
+ self.assertEqual(uri.port, test['port'])
+ self.assertEqual(uri.hostport, test['hostport'])
+
+ uri.path = test['path']
+ self.assertEqual(uri.path, test['path'])
+
+ uri.params = test['params']
+ self.assertEqual(uri.params, test['params'])
+
+ self.assertEqual(str(uri)+str(uri.relative), str(test['uri'])+str(test['relative']))
+
+ self.assertEqual(str(uri), test['uri'])
+
+ uri.params = {}
+ self.assertEqual(uri.params, {})
+ self.assertEqual(str(uri), (str(uri).split(";"))[0])
+
class FetcherTest(unittest.TestCase):
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v2 4/6] fetch2: Adapt encode/decode url to use URI class
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (10 preceding siblings ...)
2013-01-29 7:50 ` [PATCH v2 3/6] fetch2: unittests for bb.fetch2.URI class Olof Johansson
@ 2013-01-29 7:50 ` Olof Johansson
2013-02-20 7:12 ` Richard Purdie
2013-01-29 7:50 ` [PATCH v2 5/6] fetch2: Add editor modelines for bb.tests.* Olof Johansson
2013-01-29 7:50 ` [PATCH v2 6/6] fetch2: Add SFTP fetcher Olof Johansson
13 siblings, 1 reply; 23+ messages in thread
From: Olof Johansson @ 2013-01-29 7:50 UTC (permalink / raw)
To: bitbake-devel
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/fetch2/__init__.py | 66 ++++++++++++---------------------------------
1 file changed, 17 insertions(+), 49 deletions(-)
diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 7f44976..b6fc8f8 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -319,40 +319,9 @@ def decodeurl(url):
user, password, parameters).
"""
- m = re.compile('(?P<type>[^:]*)://((?P<user>.+)@)?(?P<location>[^;]+)(;(?P<parm>.*))?').match(url)
- if not m:
- raise MalformedUrl(url)
-
- type = m.group('type')
- location = m.group('location')
- if not location:
- raise MalformedUrl(url)
- user = m.group('user')
- parm = m.group('parm')
-
- locidx = location.find('/')
- if locidx != -1 and type.lower() != 'file':
- host = location[:locidx]
- path = location[locidx:]
- else:
- host = ""
- path = location
- if user:
- m = re.compile('(?P<user>[^:]+)(:?(?P<pswd>.*))').match(user)
- if m:
- user = m.group('user')
- pswd = m.group('pswd')
- else:
- user = ''
- pswd = ''
-
- p = {}
- if parm:
- for s in parm.split(';'):
- s1, s2 = s.split('=')
- p[s1] = s2
-
- return type, host, urllib.unquote(path), user, pswd, p
+ urlo = URI(url)
+ return (urlo.scheme, urlo.hostport, urlo.path,
+ urlo.username, urlo.password, urlo.params)
def encodeurl(decoded):
"""Encodes a URL from tokens (scheme, network location, path,
@@ -361,27 +330,26 @@ def encodeurl(decoded):
type, host, path, user, pswd, p = decoded
+ urlo = URI()
+
if not path:
raise MissingParameterError('path', "encoded from the data %s" % str(decoded))
if not type:
raise MissingParameterError('type', "encoded from the data %s" % str(decoded))
- url = '%s://' % type
- if user and type != "file":
- url += "%s" % user
- if pswd:
- url += ":%s" % pswd
- url += "@"
- if host and type != "file":
- url += "%s" % host
- # Standardise path to ensure comparisons work
- while '//' in path:
- path = path.replace("//", "/")
- url += "%s" % urllib.quote(path)
+
+ urlo.scheme = type
+ urlo.path = path
+
+ if host:
+ urlo.hostname = host
+ if user:
+ urlo.username = user
+ if pswd:
+ urlo.password = pswd
if p:
- for parm in p:
- url += ";%s=%s" % (parm, p[parm])
+ urlo.params = p
- return url
+ return str(urlo)
def uri_replace(ud, uri_find, uri_replace, replacements, d):
if not ud.url or not uri_find or not uri_replace:
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v2 5/6] fetch2: Add editor modelines for bb.tests.*
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (11 preceding siblings ...)
2013-01-29 7:50 ` [PATCH v2 4/6] fetch2: Adapt encode/decode url to use URI class Olof Johansson
@ 2013-01-29 7:50 ` Olof Johansson
2013-01-29 7:50 ` [PATCH v2 6/6] fetch2: Add SFTP fetcher Olof Johansson
13 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-29 7:50 UTC (permalink / raw)
To: bitbake-devel
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/tests/codeparser.py | 2 ++
lib/bb/tests/cow.py | 2 ++
lib/bb/tests/data.py | 2 ++
lib/bb/tests/fetch.py | 2 ++
lib/bb/tests/utils.py | 2 ++
5 files changed, 10 insertions(+)
diff --git a/lib/bb/tests/codeparser.py b/lib/bb/tests/codeparser.py
index 9b2d588..d2bd4a2 100644
--- a/lib/bb/tests/codeparser.py
+++ b/lib/bb/tests/codeparser.py
@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Test for codeparser.py
#
diff --git a/lib/bb/tests/cow.py b/lib/bb/tests/cow.py
index 599b2aa..35c5841 100644
--- a/lib/bb/tests/cow.py
+++ b/lib/bb/tests/cow.py
@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for Copy-on-Write (cow.py)
#
diff --git a/lib/bb/tests/data.py b/lib/bb/tests/data.py
index b2ab8ee..5161586 100644
--- a/lib/bb/tests/data.py
+++ b/lib/bb/tests/data.py
@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for the Data Store (data.py/data_smart.py)
#
diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index e765ca7..778d94f 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for the Fetcher (fetch2/)
#
diff --git a/lib/bb/tests/utils.py b/lib/bb/tests/utils.py
index 9f60491..7c50b1d 100644
--- a/lib/bb/tests/utils.py
+++ b/lib/bb/tests/utils.py
@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for utils.py
#
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v2 6/6] fetch2: Add SFTP fetcher
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
` (12 preceding siblings ...)
2013-01-29 7:50 ` [PATCH v2 5/6] fetch2: Add editor modelines for bb.tests.* Olof Johansson
@ 2013-01-29 7:50 ` Olof Johansson
13 siblings, 0 replies; 23+ messages in thread
From: Olof Johansson @ 2013-01-29 7:50 UTC (permalink / raw)
To: bitbake-devel
This fetcher differs from the SSH fetcher in that it adheres more
strictly to the SECSH URI internet draft --- it uses the sftp://
instead of the ssh:// scheme, and it uses sftp instead of scp.
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
---
lib/bb/fetch2/__init__.py | 2 +
lib/bb/fetch2/sftp.py | 129 +++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 131 insertions(+)
create mode 100644 lib/bb/fetch2/sftp.py
diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index b6fc8f8..cc524c5 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -1449,6 +1449,7 @@ from . import svn
from . import wget
from . import svk
from . import ssh
+from . import sftp
from . import perforce
from . import bzr
from . import hg
@@ -1462,6 +1463,7 @@ methods.append(git.Git())
methods.append(cvs.Cvs())
methods.append(svk.Svk())
methods.append(ssh.SSH())
+methods.append(sftp.SFTP())
methods.append(perforce.Perforce())
methods.append(bzr.Bzr())
methods.append(hg.Hg())
diff --git a/lib/bb/fetch2/sftp.py b/lib/bb/fetch2/sftp.py
new file mode 100644
index 0000000..5fbbcfd
--- /dev/null
+++ b/lib/bb/fetch2/sftp.py
@@ -0,0 +1,129 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake SFTP Fetch implementation
+
+Class for fetching files via SFTP. It tries to adhere to the (now
+expired) IETF Internet Draft for "Uniform Resource Identifier (URI)
+Scheme for Secure File Transfer Protocol (SFTP) and Secure Shell
+(SSH)" (SECSH URI).
+
+It uses SFTP (as to adhere to the SECSH URI specification). It only
+supports key based authentication, not password. This class, unlike
+the SSH fetcher, does not support fetching a directory tree from the
+remote.
+
+ http://tools.ietf.org/html/draft-ietf-secsh-scp-sftp-ssh-uri-04
+ https://www.iana.org/assignments/uri-schemes/prov/sftp
+ https://tools.ietf.org/html/draft-ietf-secsh-filexfer-13
+
+Please note that '/' is used as host path seperator, and not ":"
+as you may be used to from the scp/sftp commands. You can use a
+~ (tilde) to specify a path relative to your home directory.
+(The /~user/ syntax, for specyfing a path relative to another
+user's home directory is not supported.) Note that the tilde must
+still follow the host path seperator ("/"). See exampels below.
+
+Example SRC_URIs:
+
+SRC_URI = "sftp://host.example.com/dir/path.file.txt"
+
+A path relative to your home directory.
+
+SRC_URI = "sftp://host.example.com/~/dir/path.file.txt"
+
+You can also specify a username (specyfing password in the
+URI is not supported, use SSH keys to authenticate):
+
+SRC_URI = "sftp://user@host.example.com/dir/path.file.txt"
+
+"""
+
+# Copyright (C) 2013, Olof Johansson <olof.johansson@axis.com>
+#
+# Based in part on bb.fetch2.wget:
+# Copyright (C) 2003, 2004 Chris Larson
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# Based on functions from the base bb module, Copyright 2003 Holger Schurig
+
+import os
+import bb
+import urllib
+import commands
+from bb import data
+from bb.fetch2 import URI
+from bb.fetch2 import FetchMethod
+from bb.fetch2 import runfetchcmd
+
+
+class SFTP(FetchMethod):
+ """Class to fetch urls via 'sftp'"""
+
+ def supports(self, url, ud, d):
+ """
+ Check to see if a given url can be fetched with sftp.
+ """
+ return ud.type in ['sftp']
+
+ def recommends_checksum(self, urldata):
+ return True
+
+ def urldata_init(self, ud, d):
+ if 'protocol' in ud.parm and ud.parm['protocol'] == 'git':
+ raise bb.fetch2.ParameterError(
+ "Invalid protocol - if you wish to fetch from a " +
+ "git repository using ssh, you need to use the " +
+ "git:// prefix with protocol=ssh", ud.url)
+
+ if 'downloadfilename' in ud.parm:
+ ud.basename = ud.parm['downloadfilename']
+ else:
+ ud.basename = os.path.basename(ud.path)
+
+ ud.localfile = data.expand(urllib.unquote(ud.basename), d)
+
+ def download(self, uri, ud, d):
+ """Fetch urls"""
+
+ urlo = URI(uri)
+ basecmd = 'sftp -oPasswordAuthentication=no'
+ port = ''
+ if urlo.port:
+ port = '-P %d' % urlo.port
+ urlo.port = None
+
+ dldir = data.getVar('DL_DIR', d, True)
+ lpath = os.path.join(dldir, ud.localfile)
+
+ user = ''
+ if urlo.userinfo:
+ user = urlo.userinfo + '@'
+
+ path = urlo.path
+
+ # Supoprt URIs relative to the user's home directory, with
+ # the tilde syntax. (E.g. <sftp://example.com/~/foo.diff>).
+ if path[:3] == '/~/':
+ path = path[3:]
+
+ remote = '%s%s:%s' % (user, urlo.hostname, path)
+
+ cmd = '%s %s %s %s' % (basecmd, port, commands.mkarg(remote),
+ commands.mkarg(lpath))
+
+ bb.fetch2.check_network_access(d, cmd, uri)
+ runfetchcmd(cmd, d)
+ return True
--
1.7.10.4
^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [PATCH v2 0/6] fetch2: Updated SFTP fetcher
2013-01-29 7:50 ` [PATCH v2 0/6] fetch2: Updated SFTP fetcher Olof Johansson
@ 2013-01-30 7:55 ` Olof Johansson
2013-01-31 12:43 ` Richard Purdie
0 siblings, 1 reply; 23+ messages in thread
From: Olof Johansson @ 2013-01-30 7:55 UTC (permalink / raw)
To: bitbake-devel@lists.openembedded.org
Hi,
On 2013-01-29 08:50, Olof Johansson wrote:
> Olof Johansson (6):
> fetch2: Remove unused code in wget fetcher
> fetch2: Add a class representing a generic URI
> fetch2: unittests for bb.fetch2.URI class
> fetch2: Adapt encode/decode url to use URI class
> fetch2: Add editor modelines for bb.tests.*
> fetch2: Add SFTP fetcher
One thing I forgot to mention; after speaking with kergoth on IRC
last Friday, we found that there are some performance penelties
in using the URI class instead of encode/decodeurl. I tried to
profile and see if there were any obvious ways to get it down to
the level of the old encode/decodeurl, but nothing obvious.
I did the performance evaluation by running the unittests.
Forking and running the unittests in seperate processes shows no
change:
http://codepad.org/SYsVsckY
However, running them in a single process with the timeit module
shows a pretty big (proportional) regression:
http://codepad.org/Vzu7Pqqw
Is this significant? Running the tests 10000, each run creating
three URI objects, takes ~2 seconds.
--
Olof Johasnson
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 0/6] fetch2: Updated SFTP fetcher
2013-01-30 7:55 ` Olof Johansson
@ 2013-01-31 12:43 ` Richard Purdie
2013-02-01 8:58 ` Peter Kjellerstedt
0 siblings, 1 reply; 23+ messages in thread
From: Richard Purdie @ 2013-01-31 12:43 UTC (permalink / raw)
To: Olof Johansson; +Cc: bitbake-devel@lists.openembedded.org
On Wed, 2013-01-30 at 08:55 +0100, Olof Johansson wrote:
> Hi,
>
> On 2013-01-29 08:50, Olof Johansson wrote:
> > Olof Johansson (6):
> > fetch2: Remove unused code in wget fetcher
> > fetch2: Add a class representing a generic URI
> > fetch2: unittests for bb.fetch2.URI class
> > fetch2: Adapt encode/decode url to use URI class
> > fetch2: Add editor modelines for bb.tests.*
> > fetch2: Add SFTP fetcher
>
> One thing I forgot to mention; after speaking with kergoth on IRC
> last Friday, we found that there are some performance penelties
> in using the URI class instead of encode/decodeurl. I tried to
> profile and see if there were any obvious ways to get it down to
> the level of the old encode/decodeurl, but nothing obvious.
>
> I did the performance evaluation by running the unittests.
>
> Forking and running the unittests in seperate processes shows no
> change:
>
> http://codepad.org/SYsVsckY
>
> However, running them in a single process with the timeit module
> shows a pretty big (proportional) regression:
>
> http://codepad.org/Vzu7Pqqw
>
> Is this significant? Running the tests 10000, each run creating
> three URI objects, takes ~2 seconds.
In the scheme of things and other bitbake overheads, I'd suspect its not
significant although it is a good thing to check...
What were the other changes between v1 and v2? In cases like this a 0/6
summary email is helpful.
Cheers,
Richard
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 0/6] fetch2: Updated SFTP fetcher
2013-01-31 12:43 ` Richard Purdie
@ 2013-02-01 8:58 ` Peter Kjellerstedt
0 siblings, 0 replies; 23+ messages in thread
From: Peter Kjellerstedt @ 2013-02-01 8:58 UTC (permalink / raw)
To: Richard Purdie; +Cc: bitbake-devel@lists.openembedded.org, Olof Johansson
> -----Original Message-----
> From: bitbake-devel-bounces@lists.openembedded.org [mailto:bitbake-
> devel-bounces@lists.openembedded.org] On Behalf Of Richard Purdie
> Sent: den 31 januari 2013 13:43
> To: Olof Johansson
> Cc: bitbake-devel@lists.openembedded.org
> Subject: Re: [bitbake-devel] [PATCH v2 0/6] fetch2: Updated SFTP
> fetcher
>
> On Wed, 2013-01-30 at 08:55 +0100, Olof Johansson wrote:
> > Hi,
> >
> > On 2013-01-29 08:50, Olof Johansson wrote:
> > > Olof Johansson (6):
> > > fetch2: Remove unused code in wget fetcher
> > > fetch2: Add a class representing a generic URI
> > > fetch2: unittests for bb.fetch2.URI class
> > > fetch2: Adapt encode/decode url to use URI class
> > > fetch2: Add editor modelines for bb.tests.*
> > > fetch2: Add SFTP fetcher
> >
> > One thing I forgot to mention; after speaking with kergoth on IRC
> > last Friday, we found that there are some performance penelties
> > in using the URI class instead of encode/decodeurl. I tried to
> > profile and see if there were any obvious ways to get it down to
> > the level of the old encode/decodeurl, but nothing obvious.
> >
> > I did the performance evaluation by running the unittests.
> >
> > Forking and running the unittests in seperate processes shows no
> > change:
> >
> > http://codepad.org/SYsVsckY
> >
> > However, running them in a single process with the timeit module
> > shows a pretty big (proportional) regression:
> >
> > http://codepad.org/Vzu7Pqqw
> >
> > Is this significant? Running the tests 10000, each run creating
> > three URI objects, takes ~2 seconds.
>
> In the scheme of things and other bitbake overheads, I'd suspect its
> not significant although it is a good thing to check...
>
> What were the other changes between v1 and v2? In cases like this a 0/6
> summary email is helpful.
>
> Cheers,
>
> Richard
The list of changes between v1 and v2 were in the 0/6 mail that
Olof did send (the one he replied to above). I have included them
below:
> -----Original Message-----
> From: bitbake-devel-bounces@lists.openembedded.org [mailto:bitbake-
> devel-bounces@lists.openembedded.org] On Behalf Of Olof Johansson
> Sent: den 29 januari 2013 08:50
> To: bitbake-devel@lists.openembedded.org
> Subject: [bitbake-devel] [PATCH v2 0/6] fetch2: Updated SFTP fetcher
>
> Changes since the first draft (thanks Martin Jansa for the feedback):
>
> * Rebase against current master
> * Regenerate diffs without git config diff.noprefix
> * Support /~/ (home relative URIs) in SFTP fetcher
> * Add an example section in the SFTP fetcher documentation
> * Remove "checkstatus" method from SFTP fetcher (wasn't meant to be there)
>
> Olof Johansson (6):
> fetch2: Remove unused code in wget fetcher
> ...
//Peter
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 4/6] fetch2: Adapt encode/decode url to use URI class
2013-01-29 7:50 ` [PATCH v2 4/6] fetch2: Adapt encode/decode url to use URI class Olof Johansson
@ 2013-02-20 7:12 ` Richard Purdie
0 siblings, 0 replies; 23+ messages in thread
From: Richard Purdie @ 2013-02-20 7:12 UTC (permalink / raw)
To: Olof Johansson; +Cc: bitbake-devel
On Tue, 2013-01-29 at 08:50 +0100, Olof Johansson wrote:
> Signed-off-by: Olof Johansson <olof.johansson@axis.com>
> ---
> lib/bb/fetch2/__init__.py | 66 ++++++++++++---------------------------------
> 1 file changed, 17 insertions(+), 49 deletions(-)
I've had to revert this since it simply doesn't work. bitbake-selftest
fails all the tests with this applied, we also saw multiple failures in
other places where things should work :(
Cheers,
Richard
^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2013-02-20 7:29 UTC | newest]
Thread overview: 23+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-01-21 9:17 [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Olof Johansson
2013-01-21 9:17 ` [PATCH 1/6] fetch2: Remove unused code in wget fetcher Olof Johansson
2013-01-21 9:17 ` [PATCH 2/6] fetch2: Add a class representing a generic URI Olof Johansson
2013-01-21 9:18 ` [PATCH 3/6] fetch2: unittests for bb.fetch2.URI class Olof Johansson
2013-01-21 9:18 ` [PATCH 4/6] fetch2: Adapt encode/decode url to use URI class Olof Johansson
2013-01-21 9:18 ` [PATCH 5/6] fetch2: Add editor modelines for bb.tests.* Olof Johansson
2013-01-21 9:18 ` [PATCH 6/6] fetch2: Add SFTP fetcher Olof Johansson
2013-01-28 14:13 ` Martin Jansa
2013-01-28 14:20 ` Olof Johansson
2013-01-28 15:43 ` Martin Jansa
2013-01-28 14:05 ` [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl Martin Jansa
2013-01-28 14:23 ` Olof Johansson
2013-01-29 7:50 ` [PATCH v2 0/6] fetch2: Updated SFTP fetcher Olof Johansson
2013-01-30 7:55 ` Olof Johansson
2013-01-31 12:43 ` Richard Purdie
2013-02-01 8:58 ` Peter Kjellerstedt
2013-01-29 7:50 ` [PATCH v2 1/6] fetch2: Remove unused code in wget fetcher Olof Johansson
2013-01-29 7:50 ` [PATCH v2 2/6] fetch2: Add a class representing a generic URI Olof Johansson
2013-01-29 7:50 ` [PATCH v2 3/6] fetch2: unittests for bb.fetch2.URI class Olof Johansson
2013-01-29 7:50 ` [PATCH v2 4/6] fetch2: Adapt encode/decode url to use URI class Olof Johansson
2013-02-20 7:12 ` Richard Purdie
2013-01-29 7:50 ` [PATCH v2 5/6] fetch2: Add editor modelines for bb.tests.* Olof Johansson
2013-01-29 7:50 ` [PATCH v2 6/6] fetch2: Add SFTP fetcher Olof Johansson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox