From mboxrd@z Thu Jan 1 00:00:00 1970 From: Anthony Liguori Subject: Re: [PATCH] XML-RPC: Cope with large integers on x86-64 systems Date: Tue, 28 Mar 2006 09:15:34 -0600 Message-ID: <44295316.5090002@us.ibm.com> References: <44289F79.9000501@us.ibm.com> <20060328145510.GA16226@leeni.uk.xensource.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------090905020601010504050000" Return-path: In-Reply-To: <20060328145510.GA16226@leeni.uk.xensource.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Ewan Mellor Cc: xen-devel , "Puthiyaparambil, Aravindh" List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --------------090905020601010504050000 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Ewan Mellor wrote: > On Mon, Mar 27, 2006 at 08:29:13PM -0600, Anthony Liguori wrote: > > >> Attached patch fixes a problem in marshaling large integers. It seems >> to only occur on systems with very large memory. However, on those >> systems, it's a show-stopper. I see this as a temporary solution until >> we can sanitize the XML-RPC functions and get rid of all the >> S-Expression stuff. >> > > Anthony, > > This code is clearly plagiarised from Lib/xmlrpclib.py in the Python > distribution, without appropriate credit, copyright and licence statements in > the code, and without a patch to the documentation as per the licence > agreement. I can't accept this patch as it stands. > Sorry, I didn't realize I hadn't put that in there. Attached is another patch with the appropriate references. > Regardless, I don't see what is to be gained in marshalling ints >32bits as > strings, but those <=32 bits as integers; the receiving code is going to have > to convert the incoming argument unconditionally, so why not just send all > integers as strings? They're going to be stringified on the wire anyway, so > there's no efficiency loss, and I don't think that anyone fancies auditing > Xend to figure out which integers are strict 32 bit integers on all platforms > and which aren't, so it would be safer just to pass them all as strings. > This is a pretty big hack which based on Aravindh's note may not be as important as I thought it was. Aravindh bumped NR_CPUS which increased the size of vcpu_avail. I've audited the interface and the only place that large ints can be passed around are in the S-Expressions. The S-Expressions are untyped so the user always has to manually do type conversion. Passing the S-Expressions as strings is always safe. I think this is the least invasive and safest way to address this problem. Since it will only occur if you bump NR_CPUS, we could perhaps take the time to think of a more clever solution. Regards, Anthony Liguori > Ewan. > --------------090905020601010504050000 Content-Type: text/plain; name="xend-xmlrpc-large-ints2.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="xend-xmlrpc-large-ints2.diff" # HG changeset patch # User anthony@rhesis.austin.ibm.com # Node ID 16b3267df4ebbfeeab91aadb5a48ed89609e3622 # Parent 8b5a752167a17785b998a080a5c1ce1991b9379b In some cases, Xend can return a very large integer. This so far only happens on 64bit systems where the sizeof the Python integer is > 32bit. Presumably, this is occuring because of the fact that we return PFNs as part of the domain info so on a system with greater than 2G of memory we get into the invalid integer range. For now, the work-around is to overload the xmlrpclib Marshaller and instead of throwing an exception when we get out-of-range integers, we marshal them as strings. In the future, we can handle this in a more elegant way by introducing higher-level types. We also won't have to deal with things like PFNs once we clean up the XML-RPC interface. Signed-off-by: Anthony Liguori diff -r 8b5a752167a1 -r 16b3267df4eb tools/python/xen/util/xmlrpclib2.py --- a/tools/python/xen/util/xmlrpclib2.py Mon Mar 27 10:16:36 2006 +++ b/tools/python/xen/util/xmlrpclib2.py Tue Mar 28 15:02:39 2006 @@ -15,15 +15,45 @@ # Copyright (C) 2006 Anthony Liguori # Copyright (C) 2006 XenSource Ltd. #============================================================================ +# +# This file contains portions of the xmlrpclib.py that is distributed with +# Python. Those portions are: +# +# Copyright (c) 1999-2002 by Secret Labs AB +# Copyright (c) 1999-2002 by Fredrik Lundh +# +# By obtaining, using, and/or copying this software and/or its +# associated documentation, you agree that you have read, understood, +# and will comply with the following terms and conditions: +# +# Permission to use, copy, modify, and distribute this software and +# its associated documentation for any purpose and without fee is +# hereby granted, provided that the above copyright notice appears in +# all copies, and that both that copyright notice and this permission +# notice appear in supporting documentation, and that the name of +# Secret Labs AB or the author not be used in advertising or publicity +# pertaining to distribution of the software without specific, written +# prior permission. +# +# SECRET LABS AB AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD +# TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANT- +# ABILITY AND FITNESS. IN NO EVENT SHALL SECRET LABS AB OR THE AUTHOR +# BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY +# DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, +# WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS +# ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE +# OF THIS SOFTWARE. +# -------------------------------------------------------------------- """ An enhanced XML-RPC client/server interface for Python. """ from httplib import HTTPConnection, HTTP -from xmlrpclib import Transport +from xmlrpclib import Transport, Fault from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler -import xmlrpclib, socket, os, traceback +import xmlrpclib, socket, os, traceback, string +from types import * # A new ServerProxy that also supports httpu urls. An http URL comes in the # form: @@ -31,6 +61,9 @@ # httpu:///absolute/path/to/socket.sock # # It assumes that the RPC handler is /RPC2. This probably needs to be improved + +MAXINT = 2L**31-1 +MININT = -2L**31 class HTTPUnixConnection(HTTPConnection): def connect(self): @@ -58,6 +91,67 @@ xmlrpclib.ServerProxy.__init__(self, uri, transport, encoding, verbose, allow_none) +class Marshaller(xmlrpclib.Marshaller): + def __init__(self, encoding=None, allow_none=0): + xmlrpclib.Marshaller.__init__(self, encoding, allow_none) + self.dispatch[IntType] = Marshaller.dump_int + self.dispatch[LongType] = Marshaller.dump_long + + def dump_int(self, value, write): + if value > MAXINT or value < MININT: + self.dispatch[StringType](self, str(value), write) + else: + xmlrpclib.Marshaller.dump_int(self, value, write) + + def dump_long(self, value, write): + if value > MAXINT or value < MININT: + self.dispatch[StringType](self, str(value), write) + else: + xmlrpclib.Marshaller.dump_long(self, value, write) + +def dumps(params, methodname=None, methodresponse=None, encoding=None, + allow_none=0): + assert isinstance(params, TupleType) or isinstance(params, Fault),\ + "argument must be tuple or Fault instance" + + if isinstance(params, Fault): + methodresponse = 1 + elif methodresponse and isinstance(params, TupleType): + assert len(params) == 1, "response tuple must be a singleton" + + if not encoding: + encoding = "utf-8" + + m = Marshaller(encoding, allow_none) + + data = m.dumps(params) + + if encoding != "utf-8": + xmlheader = "\n" % str(encoding) + else: + xmlheader = "\n" # utf-8 is default + + if methodname: + if not isinstance(methodname, StringType): + methodname = methodname.encode(encoding) + data = ( + xmlheader, + "\n" + "", methodname, "\n", + data, + "\n" + ) + elif methodresponse: + data = ( + xmlheader, + "\n", + data, + "\n" + ) + else: + return data + return string.join(data, "") + # This is a base XML-RPC server for TCP. It sets allow_reuse_address to # true, and has an improved marshaller that serializes unknown exceptions # with full traceback information. @@ -74,13 +168,13 @@ response = self._dispatch(method, params) response = (response,) - response = xmlrpclib.dumps(response, - methodresponse=1, - allow_none=1) + response = dumps(response, + methodresponse=1, + allow_none=1) except xmlrpclib.Fault, fault: - response = xmlrpclib.dumps(fault) + response = dumps(fault) except: - response = xmlrpclib.dumps( + response = dumps( xmlrpclib.Fault(1, traceback.format_exc()) ) --------------090905020601010504050000 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --------------090905020601010504050000--