[PATCH 0/4] Add support for XML-RPC over SSH (take two)

[PATCH 0/4] Add support for XML-RPC over SSH (take two)

[Anthony Liguori]

This series is a repost of the previous series taking into account 
Ewan's feedback wrt error reporting.  Additionally, to make things a bit 
saner, I changed the ssh invocation from being always done in the ctor 
to being done on demand.  This means that xm commands like help won't 
actually require a password if the ssh transport is being used.

I've also included the HTTP/1.1 Keep-Alive patch as it's required for 
this patch set.

Regards,

Anthony Liguori

[PATCH 1/4] Fix Xend's HTTP/1.1 Keep-Alive support

[Anthony Liguori]

[-- Attachment #1: Type: text/plain, Size: 51 bytes --]

Repost of earlier patch

Regards,

Anthony Liguori

[-- Attachment #2: xend-keep-alive.diff --]
[-- Type: text/plain, Size: 3043 bytes --]

# HG changeset patch
# User anthony@rhesis.austin.ibm.com
# Node ID 9ad475c664bff5e668cebbf864861807583d657d
# Parent  4f1e39ec05d6ec711f9ba4a66a3653ed3e168311
Add support to Xend XML-RPC server for HTTP/1.1 Keep-Alive.

This patch fixes a few bugs in the Python SimpleXMLRPC server and enables
HTTP/1.1 by default.  This allows a client to use Keep-Alive.  Keep-Alive
improves performance by eliminating the overhead of connection setup and,
more importantly, avoids credential caching when executing multiple
commands over a secure connection.

Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>

diff -r 4f1e39ec05d6 -r 9ad475c664bf tools/python/xen/util/xmlrpclib2.py
--- a/tools/python/xen/util/xmlrpclib2.py	Thu Jun  8 15:51:39 2006
+++ b/tools/python/xen/util/xmlrpclib2.py	Fri Jun  9 23:16:19 2006
@@ -39,6 +39,31 @@
 # httpu:///absolute/path/to/socket.sock
 #
 # It assumes that the RPC handler is /RPC2.  This probably needs to be improved
+
+# We're forced to subclass the RequestHandler class so that we can work around
+# some bugs in Keep-Alive handling and also enabled it by default
+class XMLRPCRequestHandler(SimpleXMLRPCRequestHandler):
+    protocol_version = "HTTP/1.1"
+
+    # this is inspired by SimpleXMLRPCRequestHandler's do_POST but differs
+    # in a few non-trivial ways
+    # 1) we never generate internal server errors.  We let the exception
+    #    propagate so that it shows up in the Xend debug logs
+    # 2) we don't bother checking for a _dispatch function since we don't
+    #    use one
+    # 3) we never shutdown the connection.  This appears to be a bug in
+    #    SimpleXMLRPCServer.py as it breaks HTTP Keep-Alive
+    def do_POST(self):
+        data = self.rfile.read(int(self.headers["content-length"]))
+        rsp = self.server._marshaled_dispatch(data)
+
+        self.send_response(200)
+        self.send_header("Content-Type", "text/xml")
+        self.send_header("Content-Length", str(len(rsp)))
+        self.end_headers()
+
+        self.wfile.write(rsp)
+        self.wfile.flush()
 
 class HTTPUnixConnection(HTTPConnection):
     def connect(self):
@@ -94,6 +119,10 @@
 class TCPXMLRPCServer(SocketServer.ThreadingMixIn, SimpleXMLRPCServer):
     allow_reuse_address = True
 
+    def __init__(self, addr, requestHandler=XMLRPCRequestHandler,
+                 logRequests=1):
+        SimpleXMLRPCServer.__init__(self, addr, requestHandler, logRequests)
+
     def _marshaled_dispatch(self, data, dispatch_method = None):
         params, method = xmlrpclib.loads(data)
         try:
@@ -131,10 +160,10 @@
 # It implements proper support for allow_reuse_address by
 # unlink()'ing an existing socket.
 
-class UnixXMLRPCRequestHandler(SimpleXMLRPCRequestHandler):
+class UnixXMLRPCRequestHandler(XMLRPCRequestHandler):
     def address_string(self):
         try:
-            return SimpleXMLRPCRequestHandler.address_string(self)
+            return XMLRPCRequestHandler.address_string(self)
         except ValueError, e:
             return self.client_address[:2]
 

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

[PATCH 2/4] Add support for an xm serve command

[Anthony Liguori]

[-- Attachment #1: Type: text/plain, Size: 52 bytes --]

Repost of earlier patch.

Regards,

Anthony Liguori

[-- Attachment #2: xm-serve.diff --]
[-- Type: text/plain, Size: 2556 bytes --]

# HG changeset patch
# User anthony@rhesis.austin.ibm.com
# Node ID cf8e253723daf8b95b54b24f6988a8e74c6cc0aa
# Parent  a0212dab2954807a979058d91b246564a7bf2cc2
Add an xm serve command.

This command proxies the Xend XML-RPC over stdio.  This is similiar to
mercurial's hg serve --stdio command.

The most obvious use of this command is remote XML-RPC invocation over SSH.

Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>

diff -r a0212dab2954 -r cf8e253723da tools/python/xen/xm/main.py
--- a/tools/python/xen/xm/main.py	Sat Jun 10 19:07:26 2006
+++ b/tools/python/xen/xm/main.py	Sat Jun 10 19:11:55 2006
@@ -41,6 +41,7 @@
 import xen.xend.XendClient
 from xen.xend.XendClient import server
 from xen.util import security
+from select import select
 
 # getopt.gnu_getopt is better, but only exists in Python 2.3+.  Use
 # getopt.getopt if gnu_getopt is not available.  This will mean that options
@@ -124,6 +125,7 @@
 loadpolicy_help = "loadpolicy <policy>              Load binary policy into hypervisor"
 makepolicy_help = "makepolicy <policy>              Build policy and create .bin/.map files"
 labels_help     = "labels [policy] [type=DOM|..]    List <type> labels for (active) policy."
+serve_help      = "serve                            Proxy Xend XML-RPC over stdio"
 
 short_command_list = [
     "console",
@@ -171,7 +173,8 @@
 host_commands = [
     "dmesg",
     "info",
-    "log"
+    "log",
+    "serve",
     ]
 
 scheduler_commands = [
@@ -833,6 +836,32 @@
     arg_check(args, "log", 0)
     
     print server.xend.node.log()
+
+def xm_serve(args):
+    arg_check(args, "serve", 0)
+
+    from fcntl import fcntl, F_SETFL
+    
+    s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+    s.connect(xen.xend.XendClient.XML_RPC_SOCKET)
+    fcntl(sys.stdin, F_SETFL, os.O_NONBLOCK)
+
+    while True:
+        iwtd, owtd, ewtd = select([sys.stdin, s], [], [])
+        if s in iwtd:
+            data = s.recv(4096)
+            if len(data) > 0:
+                sys.stdout.write(data)
+                sys.stdout.flush()
+            else:
+                break
+        if sys.stdin in iwtd:
+            data = sys.stdin.read(4096)
+            if len(data) > 0:
+                s.sendall(data)
+            else:
+                break
+    s.close()
 
 def parse_dev_info(info):
     def get_info(n, t, d):
@@ -1072,6 +1101,7 @@
     "dmesg": xm_dmesg,
     "info": xm_info,
     "log": xm_log,
+    "serve": xm_serve,
     # scheduler
     "sched-bvt": xm_sched_bvt,
     "sched-bvt-ctxallow": xm_sched_bvt_ctxallow,

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

[PATCH 2/4] Client support for XML-RPC over SSH

[Anthony Liguori]

[-- Attachment #1: Type: text/plain, Size: 139 bytes --]

This incorporates Ewan's error reporting suggestion and uses lazy 
evaluation to initialize the ssh connection.

Regards,

Anthony Liguori

[-- Attachment #2: sshtransport-1.diff --]
[-- Type: text/plain, Size: 4369 bytes --]

# HG changeset patch
# User root@rhesis.austin.ibm.com
# Node ID ee7eb0713498591c50f43cabfcf48608b0b7aa32
# Parent  84596b81600e09c83e85eb8c7517747d57d1da98
This patch adds client side support for XML-RPC over ssh.  It differs from 
the previous submission in that it now uses lazy evaluation to connect
to the remote server and provides more user friendly error messages if ssh
is not available.

Signed-off-by: Anthony Liguori

diff -r 84596b81600e -r ee7eb0713498 tools/python/xen/util/xmlrpclib2.py
--- a/tools/python/xen/util/xmlrpclib2.py	Wed Jun 14 17:09:48 2006
+++ b/tools/python/xen/util/xmlrpclib2.py	Wed Jun 14 19:14:27 2006
@@ -24,13 +24,67 @@
 import types
 
 from httplib import HTTPConnection, HTTP
-from xmlrpclib import Transport
+from xmlrpclib import Transport, getparser, Fault
 from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler
-import xmlrpclib, socket, os, stat
+from subprocess import Popen, PIPE
+from getpass import getuser
+from fcntl import ioctl
+import xmlrpclib, socket, os, stat, termios, errno
 import SocketServer
 
-import xen.xend.XendClient
 from xen.xend.XendLogging import log
+
+class SSHTransport(object):
+    def __init__(self, host, user, askpass=None):
+        self.host = host
+        self.user = user
+        self.askpass = askpass
+        self.ssh = None
+
+    def getssh(self):
+        if self.ssh == None:
+            if self.askpass:
+                f = open('/dev/tty', 'w')
+                try:
+                    os.environ['SSH_ASKPASS'] = self.askpass
+                    ioctl(f.fileno(), termios.TIOCNOTTY)
+                finally:
+                    f.close()
+
+            cmd = ['ssh', '%s@%s' % (self.user, self.host), 'xm serve']
+            try:
+                self.ssh = Popen(cmd, bufsize=0, stdin=PIPE, stdout=PIPE)
+            except OSError, (err, msg):
+                if err == errno.ENOENT:
+                    raise Fault(0, "ssh executable not found!")
+                raise
+        return self.ssh
+
+    def request(self, host, handler, request_body, verbose=0):
+        p, u = getparser()
+        ssh = self.getssh()
+        ssh.stdin.write("""POST /%s HTTP/1.1
+User-Agent: Xen
+Host: %s
+Content-Type: text/xml
+Content-Length: %d
+
+%s""" % (handler, host, len(request_body), request_body))
+        ssh.stdin.flush()
+
+        content_length = 0
+        line = ssh.stdout.readline()
+        if line.split()[1] != '200':
+            raise Fault(0, 'Server returned %s' % (' '.join(line[1:])))
+        
+        while line not in ['', '\r\n', '\n']:
+            if line.lower().startswith('content-length:'):
+                content_length = int(line[15:].strip())
+            line = ssh.stdout.readline()
+        content = ssh.stdout.read(content_length)
+        p.feed(content)
+        p.close()
+        return u.close()
 
 
 # A new ServerProxy that also supports httpu urls.  An http URL comes in the
@@ -100,9 +154,24 @@
             if protocol == 'httpu':
                 uri = 'http:' + rest
                 transport = UnixTransport()
+            elif protocol == 'ssh':
+                if not rest.startswith('//'):
+                    raise ValueError("Invalid ssh URL '%s'" % uri)
+                rest = rest[2:]
+                user = getuser()
+                path = 'RPC2'
+                if rest.find('@') != -1:
+                    (user, rest) = rest.split('@', 1)
+                if rest.find('/') != -1:
+                    (host, rest) = rest.split('/', 1)
+                    if len(rest) > 0:
+                        path = rest
+                else:
+                    host = rest
+                transport = SSHTransport(host, user)
+                uri = 'http://%s/%s' % (host, path)
         xmlrpclib.ServerProxy.__init__(self, uri, transport, encoding,
                                        verbose, allow_none)
-
 
     def __request(self, methodname, params):
         response = xmlrpclib.ServerProxy.__request(self, methodname, params)
@@ -150,6 +219,7 @@
         except xmlrpclib.Fault, fault:
             response = xmlrpclib.dumps(fault)
         except Exception, exn:
+            import xen.xend.XendClient
             log.exception(exn)
             response = xmlrpclib.dumps(
                 xmlrpclib.Fault(xen.xend.XendClient.ERROR_INTERNAL, str(exn)))

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

[PATCH 4/4] Add support for XM_SERVER variable to choose XML-RPC URI

[Anthony Liguori]

[-- Attachment #1: Type: text/plain, Size: 52 bytes --]

Repost of earlier patch.

Regards,

Anthony Liguori

[-- Attachment #2: xm_server.diff --]
[-- Type: text/plain, Size: 1181 bytes --]

# HG changeset patch
# User anthony@rhesis.austin.ibm.com
# Node ID f8fcaee58fc1f7e13ae8443336a80de4c4e50538
# Parent  03bbc156d3e8fb01eee2af990176bb64e012c4e1
Use an environmental variable (XM_SERVER) to determine XML-RPC URI.

This allows users to invoke xm against a remote server.  This is much more
useful though for exercising XML-RPC transports with xm-test though since
xm-test will continue to (mostly) function if the URI is localhost.

Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>

diff -r 03bbc156d3e8 -r f8fcaee58fc1 tools/python/xen/xend/XendClient.py
--- a/tools/python/xen/xend/XendClient.py	Sat Jun 10 19:16:33 2006
+++ b/tools/python/xen/xend/XendClient.py	Sat Jun 10 19:19:02 2006
@@ -18,6 +18,7 @@
 #============================================================================
 
 from xen.util.xmlrpclib2 import ServerProxy
+import os
 
 XML_RPC_SOCKET = "/var/run/xend/xmlrpc.sock"
 
@@ -25,4 +26,8 @@
 ERROR_GENERIC = 2
 ERROR_INVALID_DOMAIN = 3
 
-server = ServerProxy('httpu:///var/run/xend/xmlrpc.sock')
+uri = 'httpu:///var/run/xend/xmlrpc.sock'
+if os.environ.has_key('XM_SERVER'):
+    uri = os.environ['XM_SERVER']
+
+server = ServerProxy(uri)

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH 0/4] Add support for XML-RPC over SSH (take two)

[Ewan Mellor]

On Wed, Jun 14, 2006 at 02:28:23PM -0500, Anthony Liguori wrote:

> This series is a repost of the previous series taking into account 
> Ewan's feedback wrt error reporting.  Additionally, to make things a bit 
> saner, I changed the ssh invocation from being always done in the ctor 
> to being done on demand.  This means that xm commands like help won't 
> actually require a password if the ssh transport is being used.
> 
> I've also included the HTTP/1.1 Keep-Alive patch as it's required for 
> this patch set.

I've applied this whole set.  Thanks Anthony.  Could we have some
documentation patches too?

Cheers,

Ewan.

Re: [PATCH 0/4] Add support for XML-RPC over SSH (take two)

[Ewan Mellor]

On Tue, Jun 20, 2006 at 10:26:39AM +0100, Ewan Mellor wrote:

> On Wed, Jun 14, 2006 at 02:28:23PM -0500, Anthony Liguori wrote:
> 
> > This series is a repost of the previous series taking into account 
> > Ewan's feedback wrt error reporting.  Additionally, to make things a bit 
> > saner, I changed the ssh invocation from being always done in the ctor 
> > to being done on demand.  This means that xm commands like help won't 
> > actually require a password if the ssh transport is being used.
> > 
> > I've also included the HTTP/1.1 Keep-Alive patch as it's required for 
> > this patch set.
> 
> I've applied this whole set.  Thanks Anthony.  Could we have some
> documentation patches too?

I've just noticed that these patches only work on Python 2.4, because they use
subprocess.  I've split SSHTransport out to its own file (which is reasonable
in it's own right anyway), and if the import fails, it now disables the
XML-RPC over SSH altogether.

If there's anyone out there who wants XML-RPC over SSH on Python 2.3, then a
patch that used popen2.popen2 or xen.util.xpopen.xPopen3 instead of
subprocess.Popen would be welcome.

Ewan.

Re: [PATCH 0/4] Add support for XML-RPC over SSH (take two)

[Anthony Liguori]

Ewan Mellor wrote:
> On Tue, Jun 20, 2006 at 10:26:39AM +0100, Ewan Mellor wrote:
>
>   
>> On Wed, Jun 14, 2006 at 02:28:23PM -0500, Anthony Liguori wrote:
>>
>>     
>>> This series is a repost of the previous series taking into account 
>>> Ewan's feedback wrt error reporting.  Additionally, to make things a bit 
>>> saner, I changed the ssh invocation from being always done in the ctor 
>>> to being done on demand.  This means that xm commands like help won't 
>>> actually require a password if the ssh transport is being used.
>>>
>>> I've also included the HTTP/1.1 Keep-Alive patch as it's required for 
>>> this patch set.
>>>       
>> I've applied this whole set.  Thanks Anthony.  Could we have some
>> documentation patches too?
>>     
>
> I've just noticed that these patches only work on Python 2.4, because they use
> subprocess.  I've split SSHTransport out to its own file (which is reasonable
> in it's own right anyway), and if the import fails, it now disables the
> XML-RPC over SSH altogether.
>   

Ah, sorry, didn't realize that.

> If there's anyone out there who wants XML-RPC over SSH on Python 2.3, then a
> patch that used popen2.popen2 or xen.util.xpopen.xPopen3 instead of
> subprocess.Popen would be welcome.
>   

I'll put this on my TODO.  For future reference, what are the official 
versions of Python that we support and do you happen to know which 
distros have the older versions?

Regards,

Anthony Liguori

> Ewan.
>   

Re: [PATCH 0/4] Add support for XML-RPC over SSH (take two)

[Ewan Mellor]

On Tue, Jun 20, 2006 at 04:07:54PM -0500, Anthony Liguori wrote:

> For future reference, what are the official 
> versions of Python that we support and do you happen to know which 
> distros have the older versions?

2.3 is a must -- that's Debian stable, SLES 9.2, RHEL 4, and things of that
age.  There's still support in the codebase for Python 2.2, and as far as I
know we still support that.  That's RHEL 3 and SLES 8.

Ewan.