From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ben Catterall Subject: Re: [PATCH RFC v3 0/6] HVM x86 deprivileged mode summary Date: Fri, 11 Sep 2015 17:13:37 +0100 Message-ID: <55F2FDB1.5010301@citrix.com> References: <1441987717-31622-1-git-send-email-Ben.Catterall@citrix.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------050503050902010107060208" Return-path: In-Reply-To: <1441987717-31622-1-git-send-email-Ben.Catterall@citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: xen-devel@lists.xensource.com Cc: keir@xen.org, ian.campbell@citrix.com, george.dunlap@eu.citrix.com, andrew.cooper3@citrix.com, tim@xen.org, Aravind.Gopalakrishnan@amd.com, jbeulich@suse.com, boris.ostrovsky@oracle.com, suravee.suthikulpanit@amd.com List-Id: xen-devel@lists.xenproject.org --------------050503050902010107060208 Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit Hi all, Here are two Python scripts which I have used to collect performance benchmarks for this series. I am putting them here in case they are useful. Ben On 11/09/15 17:08, Ben Catterall wrote: > Hi all, > > I have now finished my internship at Citrix and am posting this final version of > my RFC series. I would like to express my thanks to all of those who have taken > the time to review, comment and discuss this series, as well as to my colleagues > who have provided excellent guidance and help. I have learned a great deal and > have greatly enjoyed working with all of you. Thank you. > > Hopefully the series will be beneficial. I believe that it has shown that a > deprivileged mode in Xen is a possible and viable option, as long as performance > impact vs security is carefully considered on a case-by-case basis. The end of > this series contains an example of moving some of the vpic into deprivileged > mode which has allowed me to test and verify that the feature works. There are > enhancements and some clean up which is needed but, after that, the feature > could be deployed to HVM devices currently found in Xen such as the VPIC. > > Patches one to four are (hopefully) now fairly stable. Patch 5 is the new > system call and deprivileged dispatch mode which is new to this series. Patch 6 > is also new and is a demonstration of using this for the vpic and hass mainly > been used to test and exercise this feature. > > As this patch series is in RFC, there are some debug printks which should be > removed when/if it leaves RFC but, they are useful in fixing the known issue so > I have left them in until that can be resolved. > > There are some efficiency savings that can be made and an instance of a general > issue (detailed later) which will need to be addressed. > > Many thanks once again, > Ben > > TODOs > ----- > There is a set of TODOs in this patch series, some issues in the later patches > which need addressing and some other considerations which I've summarised here. > > Patch 1: > - Consider hvm_deprivileged_map_* and an efficiency saving by mapping in larger > pages. See the TODO at the top of the L4 version of this method. > > Patch 2: > - We have a much more heavyweight version of the deprivileged mode context > switch after testing for AMD SVM found that this was necessary. However, > the FPU is currently also saved and this may not be necessary. Consideration > is needed to work out if we can cut this down even more. > > Patch 4: > - The watchdog timer is hooked currently to kill deprivileged mode operations > that run for too long and is hardcoded to be at least one watchdog tick and > at most two. This may want to be refined. > > Patch 5: > - Alias data for deprivileged mode. There is a large comment at the top of > deprivileged_syscall.c which outlines considerations. > - Check if we need to map_domain_page the pages when we do the copy in > hvm_deprivileged_copy_data{to/from} > - Check for unsigned integer wrapping on addition in > hvm_deprivileged_copy_data_{to/from} > - Move hvm_deprivileged_syscall into the syscall macro. It's a stub and > unless extra code is needed there it can be folded into the macro. > - Check maintainers' thoughts on the deprivileged mode function checks in > hvm_deprivileged_user_mode. See the TODO comment. > > Patches 5 & 6: > - Fix/work around the GCC switch statement issue. > > > KNOWN ISSUES > ------------ > - Page fault for vpic_ioport_write due to GCC switch statements placing the > jump table in .rodata which is in the privileged mode area. > > This has been traced to the first of the switch statements in the function. > Though other switches in that function may also be affected. > Compiled using GCC 4.9.2-10. > > You can get the offset into this function by doing: > (RIP - (depriv_vpic_ioport_write - __hvm_deprivileged_text_start)) > > It appears to be a built-in default of GCC to put switch jump tables in > .rodata or .text and there does not appear to be a way to change this > (except to patch the compiler, though hopefully there _is_ another > option I just haven't been able to find...). Note that GCC will not > necessarily allocate jump tables for each switch statment, it appears to > depends on a number of factors such as the optimiser, the number of cases, > the type of the case, compiler version etc. > > Thus, when we relocate a deprivileged method containing code using a switch > statement which GCC has created a jump table for, this leads to a page > fault. This is because we have not mapped in the rodata section > as we should not (depriv should not have access to it). > > A workaround would be to patch the generated assembly so that this table is > moved into hvm_deprivileged.rodata. This can be done by adding, > .section .hvm_deprivileged.rodata, around the generated table. We can then > relocate this. > > Note that GCC is using RIP-relative addressing for this, so the offset > of depriv .rodata to the depriv .text segment will need to be the same > when it is mapped in. > > > > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel > --------------050503050902010107060208 Content-Type: text/x-python; name="generator.py" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="generator.py" #!/usr/bin/python3 import numpy as np import matplotlib.pyplot as plt #Use for running rsync import subprocess #Use for command line args import sys, argparse import array import pickle from matplotlib.pyplot import savefig ''' Plot the data and output to a file ''' def plot(xList,filename): i = 0 colours = ["red", "blue"] labels = ["Enabled", "Disabled"] bins = array.array('f') #Get max and min max = np.amax(xList[0]) if np.amax(xList[1]) > max: max = np.amax(xList[1]) min = np.amin(xList[0]) if np.amin(xList[1]) < min: min = np.amin(xList[1]) intervalBins = np.linspace(min, max, num=100) for data in xList: n, bins, patches = plt.hist(data, intervalBins, facecolor=colours[i], alpha=0.75, label=labels[i]); print(str(n) + " " + str(bins) + " " + str(patches) + "\n") i = i + 1 print("Mean: " + str(np.mean(data)) + " seconds\n") print("Std: " + str(np.std(data,dtype=np.float64)) + " seconds\n") print("Max: " + str(np.amax(data)) + " Min: " + str(np.amin(data)) + "\n") plt.legend(loc='upper right') plt.ylabel("Frequency") plt.xlabel("Time/s") plt.title("Graph of Xen deprivileged user performance") plt.ticklabel_format(axis='x', style='sci', scilimits=(-2,2)) fg = plt.gcf() fg.savefig(filename) return def main(): #Setup parser parser = argparse.ArgumentParser(description="Test HVM depriv performance.") parser.add_argument("-p", "--plot", nargs=2, metavar=("[DATA FILE NAME]", "[GRAPH FILE NAME]"), help="Plot the data and output a graph at file name") args = parser.parse_args() #Process args if args.plot: print("Plotting...\n") #Use Pickle to load in the array timeData = pickle.load(open(args.plot[0], "rb")) #Parse it and plot plot(timeData, args.plot[1]) print("DONE") if __name__ == '__main__': main() --------------050503050902010107060208 Content-Type: text/x-python; name="hvm_test_depriv.py" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="hvm_test_depriv.py" #!/usr/bin/python3 ''' To make use of this script, you will need to run it as root and have installed the Python portio package. You need to have patched the portio operation in vmx.c and svm.c which can be found at xen/arch/x86/hvm/{vmx,svm}/{vmx.c,svm.c} as shown below so that it will run the deprivileged mode when we perform a particular port operation. My series has the basic hook in there for patch 2 but this is commented out/removed in later patches as there was a functional change (you can no longer just enter the mode, you need to supply an operation to jump to so this will need to be updated to work with that). This is in vmx_vmexit_handler and svm_vmexit_handler Here is the svm part of this copied here for completeness: uint16_t port = (vmcb->exitinfo1 >> 16) & 0xFFFF; int bytes = ((vmcb->exitinfo1 >> 4) & 0x07); int dir = (vmcb->exitinfo1 & 1) ? IOREQ_READ : IOREQ_WRITE; /* DEBUG: Run only for a specific port */ if(port == 0x1000) { if( guest_cpu_user_regs()->eax == 0x1) { hvm_deprivileged_user_mode(); } __update_guest_eip(regs, vmcb->exitinfo2 - vmcb->rip); break; } if ( handle_pio(port, bytes, dir) ) __update_guest_eip(regs, vmcb->exitinfo2 - vmcb->rip); } ''' import array #Use for command line args import sys import pickle import time from portio import ioperm, iopl, outb def prepare_portio(port): #Get access to all ports (needed on Linux pre 2.6.? kernel) if( iopl(3) != 0 ): print("ERROR: Elevating access to ports " + str(port) + "\n") return False #get access to port, only one port, and enable it if( ioperm(port, 1, 1) != 0 ): print("ERROR: Preparing port " + str(port) + "\n") return False return True """ Clean up after portio operations """ def release_portio(port): if( ioperm(port, 1, 0) != 0 ): print("ERROR: Releasing port " + str(port) + " \n") return False if( iopl(0) != 0 ): print("ERROR: Clearing access to ports " + str(port) + "\n") return False return True ''' Needs root to run ''' def send_data(port, data): outb(data, port) def profile(outputTimes, numIterations, port, data): i = 0 while (i < numIterations): before = time.time() send_data(port, data) after = time.time() timeDelta = after - before outputTimes.append( timeDelta) i = i + 1 def main(): outputTimesWith = array.array("f") outputTimesWithout = array.array("f") portWith = 0x1000 portWithout = 0x1000 dataWith = 0x1 dataWithout =0x2 numIterations = 100000 prepare_portio(portWith) #Profile the system #With the deprivileged op profile(outputTimesWith, numIterations, portWith, dataWith); release_portio(portWith) #Without the deprivileged op prepare_portio(portWithout) profile(outputTimesWithout, numIterations, portWithout, dataWithout); release_portio(portWithout) data =[outputTimesWith, outputTimesWithout] outputFile = open("data", "wb") pickle.dump(data, outputFile) outputFile.close() return if __name__ == '__main__': main() --------------050503050902010107060208 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --------------050503050902010107060208--