From mboxrd@z Thu Jan 1 00:00:00 1970 From: Anthony Liguori Subject: Re: Re: [PATCH] XML-RPC: Cope with large integers on x86-64 systems Date: Tue, 28 Mar 2006 09:45:35 -0600 Message-ID: <44295A1F.2070001@us.ibm.com> References: <44289F79.9000501@us.ibm.com> <20060328145510.GA16226@leeni.uk.xensource.com> <44295316.5090002@us.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <44295316.5090002@us.ibm.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Ewan Mellor Cc: xen-devel , "Puthiyaparambil, Aravindh" List-Id: xen-devel@lists.xenproject.org Hi Ewan, I'm not opposed to keeping this out of 3.0.2. This is a pretty skanky hack and since you have to manually modify the tree to even see this problem, having to manually apply a patch doesn't seem so bad. I can write a patch shortly after 3.0.2 that changes vcpu_avail to something a bit more sane and update xm accordingly. Regards, Anthony Liguori Anthony Liguori wrote: > Ewan Mellor wrote: >> On Mon, Mar 27, 2006 at 08:29:13PM -0600, Anthony Liguori wrote: >> >> >>> Attached patch fixes a problem in marshaling large integers. It >>> seems to only occur on systems with very large memory. However, on >>> those systems, it's a show-stopper. I see this as a temporary >>> solution until we can sanitize the XML-RPC functions and get rid of >>> all the S-Expression stuff. >>> >> >> Anthony, >> >> This code is clearly plagiarised from Lib/xmlrpclib.py in the Python >> distribution, without appropriate credit, copyright and licence >> statements in >> the code, and without a patch to the documentation as per the licence >> agreement. I can't accept this patch as it stands. >> > > Sorry, I didn't realize I hadn't put that in there. Attached is > another patch with the appropriate references. > >> Regardless, I don't see what is to be gained in marshalling ints >> >32bits as >> strings, but those <=32 bits as integers; the receiving code is going >> to have >> to convert the incoming argument unconditionally, so why not just >> send all >> integers as strings? They're going to be stringified on the wire >> anyway, so >> there's no efficiency loss, and I don't think that anyone fancies >> auditing >> Xend to figure out which integers are strict 32 bit integers on all >> platforms >> and which aren't, so it would be safer just to pass them all as strings. >> > > This is a pretty big hack which based on Aravindh's note may not be as > important as I thought it was. Aravindh bumped NR_CPUS which > increased the size of vcpu_avail. I've audited the interface and the > only place that large ints can be passed around are in the > S-Expressions. The S-Expressions are untyped so the user always has > to manually do type conversion. Passing the S-Expressions as strings > is always safe. > > I think this is the least invasive and safest way to address this > problem. Since it will only occur if you bump NR_CPUS, we could > perhaps take the time to think of a more clever solution. > > Regards, > > Anthony Liguori > >> Ewan. >> > > ------------------------------------------------------------------------ > > # HG changeset patch > # User anthony@rhesis.austin.ibm.com > # Node ID 16b3267df4ebbfeeab91aadb5a48ed89609e3622 > # Parent 8b5a752167a17785b998a080a5c1ce1991b9379b > In some cases, Xend can return a very large integer. This so far only happens > on 64bit systems where the sizeof the Python integer is > 32bit. Presumably, > this is occuring because of the fact that we return PFNs as part of the > domain info so on a system with greater than 2G of memory we get into the > invalid integer range. > > For now, the work-around is to overload the xmlrpclib Marshaller and instead > of throwing an exception when we get out-of-range integers, we marshal them > as strings. In the future, we can handle this in a more elegant way by > introducing higher-level types. We also won't have to deal with things like > PFNs once we clean up the XML-RPC interface. > > Signed-off-by: Anthony Liguori > > diff -r 8b5a752167a1 -r 16b3267df4eb tools/python/xen/util/xmlrpclib2.py > --- a/tools/python/xen/util/xmlrpclib2.py Mon Mar 27 10:16:36 2006 > +++ b/tools/python/xen/util/xmlrpclib2.py Tue Mar 28 15:02:39 2006 > @@ -15,15 +15,45 @@ > # Copyright (C) 2006 Anthony Liguori > # Copyright (C) 2006 XenSource Ltd. > #============================================================================ > +# > +# This file contains portions of the xmlrpclib.py that is distributed with > +# Python. Those portions are: > +# > +# Copyright (c) 1999-2002 by Secret Labs AB > +# Copyright (c) 1999-2002 by Fredrik Lundh > +# > +# By obtaining, using, and/or copying this software and/or its > +# associated documentation, you agree that you have read, understood, > +# and will comply with the following terms and conditions: > +# > +# Permission to use, copy, modify, and distribute this software and > +# its associated documentation for any purpose and without fee is > +# hereby granted, provided that the above copyright notice appears in > +# all copies, and that both that copyright notice and this permission > +# notice appear in supporting documentation, and that the name of > +# Secret Labs AB or the author not be used in advertising or publicity > +# pertaining to distribution of the software without specific, written > +# prior permission. > +# > +# SECRET LABS AB AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD > +# TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANT- > +# ABILITY AND FITNESS. IN NO EVENT SHALL SECRET LABS AB OR THE AUTHOR > +# BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY > +# DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, > +# WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS > +# ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE > +# OF THIS SOFTWARE. > +# -------------------------------------------------------------------- > > """ > An enhanced XML-RPC client/server interface for Python. > """ > > from httplib import HTTPConnection, HTTP > -from xmlrpclib import Transport > +from xmlrpclib import Transport, Fault > from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler > -import xmlrpclib, socket, os, traceback > +import xmlrpclib, socket, os, traceback, string > +from types import * > > # A new ServerProxy that also supports httpu urls. An http URL comes in the > # form: > @@ -31,6 +61,9 @@ > # httpu:///absolute/path/to/socket.sock > # > # It assumes that the RPC handler is /RPC2. This probably needs to be improved > + > +MAXINT = 2L**31-1 > +MININT = -2L**31 > > class HTTPUnixConnection(HTTPConnection): > def connect(self): > @@ -58,6 +91,67 @@ > xmlrpclib.ServerProxy.__init__(self, uri, transport, encoding, > verbose, allow_none) > > +class Marshaller(xmlrpclib.Marshaller): > + def __init__(self, encoding=None, allow_none=0): > + xmlrpclib.Marshaller.__init__(self, encoding, allow_none) > + self.dispatch[IntType] = Marshaller.dump_int > + self.dispatch[LongType] = Marshaller.dump_long > + > + def dump_int(self, value, write): > + if value > MAXINT or value < MININT: > + self.dispatch[StringType](self, str(value), write) > + else: > + xmlrpclib.Marshaller.dump_int(self, value, write) > + > + def dump_long(self, value, write): > + if value > MAXINT or value < MININT: > + self.dispatch[StringType](self, str(value), write) > + else: > + xmlrpclib.Marshaller.dump_long(self, value, write) > + > +def dumps(params, methodname=None, methodresponse=None, encoding=None, > + allow_none=0): > + assert isinstance(params, TupleType) or isinstance(params, Fault),\ > + "argument must be tuple or Fault instance" > + > + if isinstance(params, Fault): > + methodresponse = 1 > + elif methodresponse and isinstance(params, TupleType): > + assert len(params) == 1, "response tuple must be a singleton" > + > + if not encoding: > + encoding = "utf-8" > + > + m = Marshaller(encoding, allow_none) > + > + data = m.dumps(params) > + > + if encoding != "utf-8": > + xmlheader = "\n" % str(encoding) > + else: > + xmlheader = "\n" # utf-8 is default > + > + if methodname: > + if not isinstance(methodname, StringType): > + methodname = methodname.encode(encoding) > + data = ( > + xmlheader, > + "\n" > + "", methodname, "\n", > + data, > + "\n" > + ) > + elif methodresponse: > + data = ( > + xmlheader, > + "\n", > + data, > + "\n" > + ) > + else: > + return data > + return string.join(data, "") > + > # This is a base XML-RPC server for TCP. It sets allow_reuse_address to > # true, and has an improved marshaller that serializes unknown exceptions > # with full traceback information. > @@ -74,13 +168,13 @@ > response = self._dispatch(method, params) > > response = (response,) > - response = xmlrpclib.dumps(response, > - methodresponse=1, > - allow_none=1) > + response = dumps(response, > + methodresponse=1, > + allow_none=1) > except xmlrpclib.Fault, fault: > - response = xmlrpclib.dumps(fault) > + response = dumps(fault) > except: > - response = xmlrpclib.dumps( > + response = dumps( > xmlrpclib.Fault(1, traceback.format_exc()) > ) > > > ------------------------------------------------------------------------ > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >