* Re: RPC interface for xenmon
2006-04-02 18:47 ` Diwaker Gupta
@ 2006-04-04 5:06 ` Jayesh Salvi
2006-04-04 14:50 ` Rob Gardner
0 siblings, 1 reply; 4+ messages in thread
From: Jayesh Salvi @ 2006-04-04 5:06 UTC (permalink / raw)
To: Diwaker Gupta; +Cc: xen-devel, Kyle Mestery
[-- Attachment #1.1: Type: text/plain, Size: 2577 bytes --]
Hi,
Here is a patch that makes xenmon run in server-client mode doing XML-RPC. I
have used xmlrpclib (that comes in-built with python 2.2+) and
xmlrpcserver.py (attached).
I have added two options to the script in addition to the already available
ones.
-s, --server publish numbers on XML-RPC server
-rREMOTE, --remote=REMOTE
Display stats from remote server
When run in server mode the script will listen for XML-RPC requests. A
thread is forked that calls collect_stats function (which I copy-pasted from
show_livestats) that collects data from xenbaked and stores in some global
variables. A XenmonRequestHandler class publishes a remote function
'getstats' which is called by the RPC client to get stats.
When run in client mode, giving it the RPC server name, this script connects
to the remote server and collects the stats. The display_stats() function
(code borrowed from show_livestats) does this remote call and displays the
retrieved stat data.
Example run:
Server:
xenmon.py -s
Client:
xenmon.py -r<remote_server_name>
Problems:
There are some problems in this script mainly because of my lack of
understanding the original code and my being a python newbie.
1. The script running in server mode hogs around 99% CPU. I don't know why
that is happening, but I guess I am collecting stats too frequently. I could
not figure out the part of the code which decides how frequently this
polling should be done. Could someone have a look?
2. By adding two more options, I have done something wrong in OptionParser
code, causing the --live option not being reached. This should be a minor
mistake and someone knowing OptionParser code better and quickly tell the
mistake. Please let me know what is it.
I have used the xmlrpcserver script by Fredrik Lundh. As the script says it
is free for use in as is form. I am attaching it here, it could be found on
net as well. This script must be in pwd while running xenmon in server mode.
Please let me know the bugs you find in here.
Thanks,
Jayesh
------------------------------------------------------------------------
Everything you can imagine is real
On 4/2/06, Diwaker Gupta <diwaker.lists@gmail.com> wrote:
>
> > I read about RPC interface for xenmon in this presentation:
> > www.xensource.com/files/xs0106_xenmon_brief.pdf
> >
> > Has such interface been implemented yet by anyone?
>
> Not yet. Patches are always welcome! :-)
>
> Diwaker
> --
> Web/Blog/Gallery: http://floatingsun.net/blog
>
--
[-- Attachment #1.2: Type: text/html, Size: 3236 bytes --]
[-- Attachment #2: xenmon_xmlrpc.patch --]
[-- Type: text/x-patch, Size: 13897 bytes --]
--- ./xenmon.py 2006-04-03 23:36:58.000000000 -0500
+++ /root/xen/xenmon.py 2006-04-03 23:45:03.000000000 -0500
@@ -33,6 +33,11 @@
import math
import sys
+import thread
+import SocketServer
+import xmlrpcserver
+import xmlrpclib
+
# constants
NSAMPLES = 100
NDOMAINS = 32
@@ -66,6 +71,32 @@
# parsed options
options, args = None, None
+################################################################################
+# XML-RPC request handler class
+# This class is the Request handler class. The dump_response method is called
+# from the client to get stats. This method calls the local method getstats and
+# serializes its output using xmlrpclib.dumps method.
+################################################################################
+class XenmonRequestHandler(xmlrpcserver.RequestHandler):
+#Override method:
+ def call(self, method, params):
+ print "Dispatching: ", method, params
+ try:
+ server_method = getattr(self, method)
+ except:
+ raise AttributeError, "Server does not contain XML-RPC procedure %s" % method
+ return server_method(method, params)
+
+ def getstats(self, method, nr):
+ return (h1,l1,f1,h2,l2,f2,dom_in_use)
+
+ def dump_response(self, method, params):
+ response = self.call(params[0], tuple(params[1:]))
+ return xmlrpclib.dumps(response)
+
+
+
+
# the optparse module is quite smart
# to see help, just run xenmon -h
def setup_cmdline_parser():
@@ -74,6 +105,10 @@
default=True, help = "show the ncurses live monitoring frontend (default)")
parser.add_option("-n", "--notlive", dest="live", action="store_false",
default="True", help = "write to file instead of live monitoring")
+ parser.add_option("-s", "--server", dest="server", action="store_true",
+ default="False", help = "publish numbers on XML-RPC server")
+ parser.add_option("-r", "--remote", dest="remote", action="store",
+ default="localhost", help = "Display stats from remote server")
parser.add_option("-p", "--prefix", dest="prefix",
default = "log", help="prefix to use for output files")
parser.add_option("-t", "--time", dest="duration",
@@ -232,6 +267,7 @@
ncpu = 1 # number of cpu's on this platform
slen = 0 # size of shared data structure, incuding padding
global dom_in_use
+
# mmap the (the first chunk of the) file
shmf = open(SHM_FILE, "r+")
@@ -452,12 +488,6 @@
if c == ord('c'):
cpu = (cpu + 1) % ncpu
- # n/p = cycle to the next/previous CPU
- if c == ord('n'):
- cpu = (cpu + 1) % ncpu
- if c == ord('p'):
- cpu = (cpu - 1) % ncpu
-
stdscr.erase()
_c.nocbreak()
@@ -508,7 +538,6 @@
shm = mmap.mmap(shmf.fileno(), QOS_DATA_SIZE)
interval = 0
- curr = last = time.time()
outfiles = {}
for dom in range(0, NDOMAINS):
outfiles[dom] = Delayed("%s-dom%d.log" % (options.prefix, dom), 'w')
@@ -519,7 +548,6 @@
idx = cpuidx * slen # offset needed in mmap file
-
samples = []
doms = []
dom_in_use = []
@@ -568,10 +596,9 @@
h1[dom][4],
h1[dom][5][0], h1[dom][5][1]))
outfiles[dom].flush()
- curr = time.time()
- interval += (curr - last) * 1000
- last = curr
- time.sleep(options.interval / 1000.0)
+
+ interval += options.interval
+ time.sleep(1)
for dom in range(0, NDOMAINS):
outfiles[dom].close()
@@ -591,6 +618,268 @@
def stop_xenbaked():
os.system("killall -s INT xenbaked")
+##################################################################
+# This function displays the stats (second half of show_livestats())
+# It is called when xenmon is run in remote mode. This function collects
+# stats from remote xenmon process running XML-RPC server
+##################################################################
+def display_stats(server):
+ cpu = 0 # cpu of interest to display data for
+ ncpu = 1 # number of cpu's on this platform
+
+ # initialize curses
+ stdscr = _c.initscr()
+ _c.noecho()
+ _c.cbreak()
+
+ stdscr.keypad(1)
+ stdscr.timeout(1000)
+ [maxy, maxx] = stdscr.getmaxyx()
+
+
+ while True:
+
+ # Remote call to XML-RPC server
+ res_string = server.dump_response('getstats',0)
+ # unmarshalling of serialized response
+ data,dummy = xmlrpclib.loads(res_string)
+ # retrieving the stats from the reconstructed data
+ [h1,l1,f1,h2,l2,f2,dom_in_use] = data
+
+ # the actual display code
+ row = 0
+ display(stdscr, row, 1, "CPU = %d" % cpu, _c.A_STANDOUT)
+
+ display(stdscr, row, 10, "%sLast 10 seconds%sLast 1 second" % (6*' ', 30*' '), _c.A_BOLD)
+ row +=1
+ display(stdscr, row, 1, "%s" % ((maxx-2)*'='))
+
+ total_h1_cpu = 0
+ total_h2_cpu = 0
+
+ for dom in range(0, NDOMAINS):
+ if not dom_in_use[dom]:
+ continue
+
+ if h1[dom][0][1] > 0 or dom == NDOMAINS - 1:
+ # display gotten
+ row += 1
+ col = 2
+ display(stdscr, row, col, "%d" % dom)
+ col += 4
+ display(stdscr, row, col, "%s" % time_scale(h2[dom][0][0]))
+ col += 12
+ display(stdscr, row, col, "%3.2f%%" % h2[dom][0][1])
+ col += 12
+ display(stdscr, row, col, "%s/ex" % time_scale(h2[dom][0][2]))
+ col += 18
+ display(stdscr, row, col, "%s" % time_scale(h1[dom][0][0]))
+ col += 12
+ display(stdscr, row, col, "%3.2f%%" % h1[dom][0][1])
+ col += 12
+ display(stdscr, row, col, "%s/ex" % time_scale(h1[dom][0][2]))
+ col += 18
+ display(stdscr, row, col, "Gotten")
+
+ # display allocated
+ row += 1
+ col = 2
+ display(stdscr, row, col, "%d" % dom)
+ col += 28
+ display(stdscr, row, col, "%s/ex" % time_scale(h2[dom][1]))
+ col += 42
+ display(stdscr, row, col, "%s/ex" % time_scale(h1[dom][1]))
+ col += 18
+ display(stdscr, row, col, "Allocated")
+
+ # display blocked
+ row += 1
+ col = 2
+ display(stdscr, row, col, "%d" % dom)
+ col += 4
+ display(stdscr, row, col, "%s" % time_scale(h2[dom][2][0]))
+ col += 12
+ display(stdscr, row, col, "%3.2f%%" % h2[dom][2][1])
+ col += 12
+ display(stdscr, row, col, "%s/io" % time_scale(h2[dom][2][2]))
+ col += 18
+ display(stdscr, row, col, "%s" % time_scale(h1[dom][2][0]))
+ col += 12
+ display(stdscr, row, col, "%3.2f%%" % h1[dom][2][1])
+ col += 12
+ display(stdscr, row, col, "%s/io" % time_scale(h1[dom][2][2]))
+ col += 18
+ display(stdscr, row, col, "Blocked")
+
+ # display waited
+ row += 1
+ col = 2
+ display(stdscr, row, col, "%d" % dom)
+ col += 4
+ display(stdscr, row, col, "%s" % time_scale(h2[dom][3][0]))
+ col += 12
+ display(stdscr, row, col, "%3.2f%%" % h2[dom][3][1])
+ col += 12
+ display(stdscr, row, col, "%s/ex" % time_scale(h2[dom][3][2]))
+ col += 18
+ display(stdscr, row, col, "%s" % time_scale(h1[dom][3][0]))
+ col += 12
+ display(stdscr, row, col, "%3.2f%%" % h1[dom][3][1])
+ col += 12
+ display(stdscr, row, col, "%s/ex" % time_scale(h1[dom][3][2]))
+ col += 18
+ display(stdscr, row, col, "Waited")
+
+ # display ex count
+ row += 1
+ col = 2
+ display(stdscr, row, col, "%d" % dom)
+
+ col += 28
+ display(stdscr, row, col, "%d/s" % h2[dom][4])
+ col += 42
+ display(stdscr, row, col, "%d" % h1[dom][4])
+ col += 18
+ display(stdscr, row, col, "Execution count")
+
+ # display io count
+ row += 1
+ col = 2
+ display(stdscr, row, col, "%d" % dom)
+ col += 4
+ display(stdscr, row, col, "%d/s" % h2[dom][5][0])
+ col += 24
+ display(stdscr, row, col, "%d/ex" % h2[dom][5][1])
+ col += 18
+ display(stdscr, row, col, "%d" % h1[dom][5][0])
+ col += 24
+ display(stdscr, row, col, "%3.2f/ex" % h1[dom][5][1])
+ col += 18
+ display(stdscr, row, col, "I/O Count")
+
+ #row += 1
+ #stdscr.hline(row, 1, '-', maxx - 2)
+ total_h1_cpu += h1[dom][0][1]
+ total_h2_cpu += h2[dom][0][1]
+
+
+ row += 1
+ display(stdscr, row, 2, TOTALS % (total_h2_cpu, total_h1_cpu))
+ row += 1
+
+ if l1[1] > 1 :
+ row += 1
+ display(stdscr, row, 2,
+ "\tRecords lost: %d (Min: %d, Max: %d)\t\t\tRecords lost: %d (Min: %d, Max %d)" %
+ (math.ceil(l2[1]), l2[0], l2[2], math.ceil(l1[1]), l1[0], l1[2]), _c.A_BOLD)
+
+ # grab a char from tty input; exit if interrupt hit
+ try:
+ c = stdscr.getch()
+ except:
+ break
+
+ # q = quit
+ if c == ord('q'):
+ break
+
+ # c = cycle to a new cpu of interest
+ if c == ord('c'):
+ cpu = (cpu + 1) % ncpu
+
+ stdscr.erase()
+
+ _c.nocbreak()
+ stdscr.keypad(0)
+ _c.echo()
+ _c.endwin()
+ shm.close()
+ shmf.close()
+
+##################################################################
+# This function collects the stats (first half of show_livestats())
+# This function is called run in a separate thread when xenmon is
+# run in server mode.
+##################################################################
+def collect_stats(c):
+ cpu = 0 # cpu of interest to display data for
+ ncpu = 1 # number of cpu's on this platform
+ slen = 0 # size of shared data structure, incuding padding
+ global dom_in_use
+ global h1
+ global f1
+ global l1
+ global h2
+ global f2
+ global l2
+
+
+ # mmap the (the first chunk of the) file
+ shmf = open(SHM_FILE, "r+")
+ shm = mmap.mmap(shmf.fileno(), QOS_DATA_SIZE)
+
+ samples = []
+ doms = []
+ dom_in_use = []
+
+ # initialize curses
+ while True:
+
+ for cpuidx in range(0, ncpu):
+
+ # calculate offset in mmap file to start from
+ idx = cpuidx * slen
+
+ samples = []
+ doms = []
+
+ # read in data
+ for i in range(0, NSAMPLES):
+ len = struct.calcsize(ST_QDATA)
+ sample = struct.unpack(ST_QDATA, shm[idx:idx+len])
+ samples.append(sample)
+ idx += len
+
+ for i in range(0, NDOMAINS):
+ len = struct.calcsize(ST_DOM_INFO)
+ dom = struct.unpack(ST_DOM_INFO, shm[idx:idx+len])
+ doms.append(dom)
+# (last_update_time, start_time, runnable_start_time, blocked_start_time,
+# ns_since_boot, ns_oncpu_since_boot, runnable_at_last_update,
+# runnable, in_use, domid, name) = dom
+# dom_in_use.append(in_use)
+ dom_in_use.append(dom[8])
+ idx += len
+
+ len = struct.calcsize("4i")
+ oldncpu = ncpu
+ (next, ncpu, slen, freq) = struct.unpack("4i", shm[idx:idx+len])
+ idx += len
+
+ # xenbaked tells us how many cpu's it's got, so re-do
+ # the mmap if necessary to get multiple cpu data
+ if oldncpu != ncpu:
+ shm = mmap.mmap(shmf.fileno(), ncpu*slen)
+
+ # if we've just calculated data for the cpu of interest, then
+ # stop examining mmap data and start displaying stuff
+ if cpuidx == cpu:
+ break
+
+ # calculate starting and ending datapoints; never look at "next" since
+ # it represents live data that may be in transition.
+ startat = next - 1
+ if next + 10 < NSAMPLES:
+ endat = next + 10
+ else:
+ endat = 10
+
+
+ # get summary over desired interval
+ [h1, l1, f1] = summarize(startat, endat, 10**9, samples)
+ [h2, l2, f2] = summarize(startat, endat, 10 * 10**9, samples)
+
+
def main():
global options
global args
@@ -600,6 +889,22 @@
(options, args) = parser.parse_args()
start_xenbaked()
+
+ if options.remote != 'localhost':
+ server = xmlrpclib.Server("http://%s:8000"%options.remote)
+ display_stats(server)
+
+ if options.server:
+ try:
+ thread.start_new_thread(collect_stats,(0,))
+ print 'Starting XML-RPC server'
+ server = SocketServer.TCPServer(('', 8000), XenmonRequestHandler)
+ server.serve_forever()
+ except KeyboardInterrupt:
+ print 'Ctrl+C handled'
+ stop_xenbaked()
+ sys.exit(0)
+
if options.live:
show_livestats()
else:
[-- Attachment #3: xmlrpcserver.py --]
[-- Type: text/x-python, Size: 2100 bytes --]
#
# XML-RPC SERVER
# $Id: xmlrpcserver.py,v 1.1 2001/04/10 09:25:16 frank Exp $
#
# a simple XML-RPC server for Python
#
# History:
# 1999-02-01 fl added to xmlrpclib distribution
#
# written by Fredrik Lundh, January 1999.
#
# Copyright (c) 1999 by Secret Labs AB.
# Copyright (c) 1999 by Fredrik Lundh.
#
# fredrik@pythonware.com
# http://www.pythonware.com
#
# --------------------------------------------------------------------
# Permission to use, copy, modify, and distribute this software and
# its associated documentation for any purpose and without fee is
# hereby granted. This software is provided as is.
# --------------------------------------------------------------------
#
import SocketServer, BaseHTTPServer
import xmlrpclib
import sys
class RequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
def do_POST(self):
try:
# get arguments
data = self.rfile.read(int(self.headers["content-length"]))
params, method = xmlrpclib.loads(data)
# generate response
try:
response = self.call(method, params)
if type(response) != type(()):
response = (response,)
except:
# report exception back to server
response = xmlrpclib.dumps(
xmlrpclib.Fault(1, "%s:%s" % (sys.exc_type, sys.exc_value))
)
else:
response = xmlrpclib.dumps(
response,
methodresponse=1
)
except:
# internal error, report as HTTP server error
self.send_response(500)
self.end_headers()
else:
# got a valid XML RPC response
self.send_response(200)
self.send_header("Content-type", "text/xml")
self.send_header("Content-length", str(len(response)))
self.end_headers()
self.wfile.write(response)
# shut down the connection (from Skip Montanaro)
self.wfile.flush()
self.connection.shutdown(1)
def call(self, method, params):
# override this method to implement RPC methods
print "CALL", method, params
return params
if __name__ == '__main__':
server = SocketServer.TCPServer(('', 8888), RequestHandler)
server.serve_forever()
[-- Attachment #4: Type: text/plain, Size: 138 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
^ permalink raw reply [flat|nested] 4+ messages in thread