Re: [PATCH RFC v3 0/6] HVM x86 deprivileged mode summary

[Ben Catterall <Ben.Catterall@citrix.com>]

[-- Attachment #1: Type: text/plain, Size: 5346 bytes --]

Hi all,

Here are two Python scripts which I have used to collect performance 
benchmarks for this series. I am putting them here in case they are useful.

Ben

On 11/09/15 17:08, Ben Catterall wrote:
> Hi all,
>
> I have now finished my internship at Citrix and am posting this final version of
> my RFC series. I would like to express my thanks to all of those who have taken
> the time to review, comment and discuss this series, as well as to my colleagues
> who have provided excellent guidance and help. I have learned a great deal and
> have greatly enjoyed working with all of you. Thank you.
>
> Hopefully the series will be beneficial. I believe that it has shown that a
> deprivileged mode in Xen is a possible and viable option, as long as performance
> impact vs security is carefully considered on a case-by-case basis. The end of
> this series contains an example of moving some of the vpic into deprivileged
> mode which has allowed me to test and verify that the feature works. There are
> enhancements and some clean up which is needed but, after that, the feature
> could be deployed to HVM devices currently found in Xen such as the VPIC.
>
> Patches one to four are (hopefully) now fairly stable. Patch 5 is the new
> system call and deprivileged dispatch mode which is new to this series. Patch 6
> is also new and is a demonstration of using this for the vpic and hass mainly
> been used to test and exercise this feature.
>
> As this patch series is in RFC, there are some debug printks which should be
> removed when/if it leaves RFC but, they are useful in fixing the known issue so
> I have left them in until that can be resolved.
>
> There are some efficiency savings that can be made and an instance of a general
> issue (detailed later) which will need to be addressed.
>
> Many thanks once again,
> Ben
>
> TODOs
> -----
> There is a set of TODOs in this patch series, some issues in the later patches
> which need addressing and some other considerations which I've summarised here.
>
> Patch 1:
>   - Consider hvm_deprivileged_map_* and an efficiency saving by mapping in larger
>     pages. See the TODO at the top of the L4 version of this method.
>
> Patch 2:
>   - We have a much more heavyweight version of the deprivileged mode context
>     switch after testing for AMD SVM found that this was necessary. However,
>     the FPU is currently also saved and this may not be necessary. Consideration
>     is needed to work out if we can cut this down even more.
>
> Patch 4:
>   - The watchdog timer is hooked currently to kill deprivileged mode operations
>     that run for too long and is hardcoded to be at least one watchdog tick and
>     at most two. This may want to be refined.
>
> Patch 5:
>   - Alias data for deprivileged mode. There is a large comment at the top of
>     deprivileged_syscall.c which outlines considerations.
>   - Check if we need to map_domain_page the pages when we do the copy in
>     hvm_deprivileged_copy_data{to/from}
>   - Check for unsigned integer wrapping on addition in
>     hvm_deprivileged_copy_data_{to/from}
>   - Move hvm_deprivileged_syscall into the syscall macro. It's a stub and
>     unless extra code is needed there it can be folded into the macro.
>   - Check maintainers' thoughts on the deprivileged mode function checks in
>     hvm_deprivileged_user_mode. See the TODO comment.
>
> Patches 5 & 6:
>   - Fix/work around the GCC switch statement issue.
>
>
> KNOWN ISSUES
> ------------
>   - Page fault for vpic_ioport_write due to GCC switch statements placing the
>     jump table in .rodata which is in the privileged mode area.
>
>     This has been traced to the first of the switch statements in the function.
>     Though other switches in that function may also be affected.
>     Compiled using GCC 4.9.2-10.
>
>     You can get the offset into this function by doing:
>     (RIP - (depriv_vpic_ioport_write - __hvm_deprivileged_text_start))
>
>     It appears to be a built-in default of GCC to put switch jump tables in
>     .rodata or .text and there does not appear to be a way to change this
>     (except to patch the compiler, though hopefully there _is_ another
>     option I just haven't been able to find...). Note that GCC will not
>     necessarily allocate jump tables for each switch statment, it appears to
>     depends on a number of factors such as the optimiser, the number of cases,
>     the type of the case, compiler version etc.
>
>     Thus, when we relocate a deprivileged method containing code using a switch
>     statement which GCC has created a jump table for, this leads to a page
>     fault. This is because we have not mapped in the rodata section
>     as we should not (depriv should not have access to it).
>
>     A workaround would be to patch the generated assembly so that this table is
>     moved into hvm_deprivileged.rodata. This can be done by adding,
>     .section .hvm_deprivileged.rodata, around the generated table. We can then
>     relocate this.
>
>     Note that GCC is using RIP-relative addressing for this, so the offset
>     of depriv .rodata to the depriv .text segment will need to be the same
>     when it is mapped in.
>
>
>
>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>

[-- Attachment #2: generator.py --]
[-- Type: text/x-python, Size: 2023 bytes --]

#!/usr/bin/python3
import numpy as np
import matplotlib.pyplot as plt

#Use for running rsync
import subprocess

#Use for command line args
import sys, argparse

import array
import pickle
from matplotlib.pyplot import savefig

'''
Plot the data and output to a file
'''
def plot(xList,filename):
    i = 0
    colours = ["red", "blue"]
    labels = ["Enabled", "Disabled"]
    bins = array.array('f')

    #Get max and min
    max = np.amax(xList[0])
    if np.amax(xList[1]) > max:
        max = np.amax(xList[1])

    min = np.amin(xList[0])
    if np.amin(xList[1]) < min:
        min = np.amin(xList[1])

    intervalBins = np.linspace(min, max, num=100)

    for data in xList:
        n, bins, patches = plt.hist(data, intervalBins, facecolor=colours[i],
                                    alpha=0.75, label=labels[i]);

        print(str(n) + " " + str(bins) + " " + str(patches) + "\n")
        i = i + 1

        print("Mean: " + str(np.mean(data)) + " seconds\n")
        print("Std:  " + str(np.std(data,dtype=np.float64)) + " seconds\n")
        print("Max: "  + str(np.amax(data)) + " Min: " + str(np.amin(data)) + "\n")

    plt.legend(loc='upper right')
    plt.ylabel("Frequency")
    plt.xlabel("Time/s")
    plt.title("Graph of Xen deprivileged user performance")
    plt.ticklabel_format(axis='x', style='sci', scilimits=(-2,2))
    fg = plt.gcf()

    fg.savefig(filename)

    return

def main():
    #Setup parser
    parser = argparse.ArgumentParser(description="Test HVM depriv performance.")
    parser.add_argument("-p", "--plot", nargs=2, metavar=("[DATA FILE NAME]", "[GRAPH FILE NAME]"),
                        help="Plot the data and output a graph at file name")

    args = parser.parse_args()

    #Process args
    if args.plot:
        print("Plotting...\n")

        #Use Pickle to load in the array
        timeData = pickle.load(open(args.plot[0], "rb"))

        #Parse it and plot
        plot(timeData, args.plot[1])

    print("DONE")

if __name__ == '__main__':
        main()

[-- Attachment #3: hvm_test_depriv.py --]
[-- Type: text/x-python, Size: 3369 bytes --]

#!/usr/bin/python3
'''
To make use of this script, you will need to run it as root and have installed
the Python portio package.

You need to have patched the portio operation in vmx.c and svm.c which can be
found at xen/arch/x86/hvm/{vmx,svm}/{vmx.c,svm.c} as shown below so that it
will run the deprivileged mode when we perform a particular port operation.
My series has the basic hook in there for patch 2 but this is commented
 out/removed in later patches as there was a functional change (you can
no longer just enter the mode, you need to supply an operation to jump to so
this will need to be updated to work with that).

This is in vmx_vmexit_handler and svm_vmexit_handler

Here is the svm part of this copied here for completeness:

             uint16_t port = (vmcb->exitinfo1 >> 16) & 0xFFFF;
             int bytes = ((vmcb->exitinfo1 >> 4) & 0x07);

             int dir = (vmcb->exitinfo1 & 1) ? IOREQ_READ : IOREQ_WRITE;
             /* DEBUG: Run only for a specific port */
             if(port == 0x1000)
             {
                 if( guest_cpu_user_regs()->eax == 0x1)
                 {
                     hvm_deprivileged_user_mode();
                 }
                 __update_guest_eip(regs, vmcb->exitinfo2 - vmcb->rip);
                 break;
             }

             if ( handle_pio(port, bytes, dir) )
                 __update_guest_eip(regs, vmcb->exitinfo2 - vmcb->rip);
         }

'''
import array

#Use for command line args
import sys
import pickle
import time

from portio import ioperm, iopl, outb

def prepare_portio(port):
    #Get access to all ports (needed on Linux pre 2.6.? kernel)
    if( iopl(3) != 0 ):
        print("ERROR: Elevating access to ports " + str(port) + "\n")
        return False

    #get access to port, only one port, and enable it
    if( ioperm(port, 1, 1) != 0 ):
        print("ERROR: Preparing port " + str(port) + "\n")
        return False

    return True

"""
Clean up after portio operations
"""
def release_portio(port):

    if( ioperm(port, 1, 0) != 0 ):
        print("ERROR: Releasing port " + str(port) + " \n")
        return False

    if( iopl(0) != 0 ):
        print("ERROR: Clearing access to ports " + str(port) + "\n")
        return False

    return True

'''
Needs root to run
'''
def send_data(port, data):
    outb(data, port)

def profile(outputTimes, numIterations, port, data):
    i = 0
    while (i < numIterations):
        before = time.time()
        send_data(port, data)
        after = time.time()

        timeDelta = after - before

        outputTimes.append( timeDelta)

        i = i + 1

def main():
    outputTimesWith = array.array("f")
    outputTimesWithout = array.array("f")
    portWith = 0x1000
    portWithout = 0x1000
    dataWith = 0x1
    dataWithout =0x2
    numIterations = 100000

    prepare_portio(portWith)

    #Profile the system
    #With the deprivileged op
    profile(outputTimesWith, numIterations, portWith, dataWith);
    release_portio(portWith)

    #Without the deprivileged op
    prepare_portio(portWithout)
    profile(outputTimesWithout, numIterations, portWithout, dataWithout);
    release_portio(portWithout)

    data =[outputTimesWith, outputTimesWithout]

    outputFile = open("data", "wb")

    pickle.dump(data, outputFile)

    outputFile.close()
    return

if __name__ == '__main__':
    main()

[-- Attachment #4: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel