Re: LWP Interrupt Handler

[Wei Huang <wei.huang2@amd.com>]

[-- Attachment #1: Type: text/plain, Size: 2241 bytes --]

How about the approach in attached patches? A interrupt handler is 
registered at vector 0xf6 as a delegate for guest VM. Guest VCPU can 
register interrupt handler to receive notification. I will prepare 
formal ones after receiving your comments.

Thanks,
-Wei


On 03/25/2012 07:08 PM, Zhang, Xiantao wrote:
> Please make sure the per-cpu vector is considered in your case.  For CPU's built-in event, it always happens on all cpus,  but IRQ-based events may only happen on some special cpus which are determined by apic's mode.
> Xiantao
>
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xen.org [mailto:xen-devel-
>> bounces@lists.xen.org] On Behalf Of Keir Fraser
>> Sent: Saturday, March 24, 2012 6:50 AM
>> To: wei.huang2@amd.com; xen-devel@lists.xen.org; Jan Beulich
>> Subject: Re: [Xen-devel] LWP Interrupt Handler
>>
>> On 23/03/2012 22:03, "Wei Huang"<wei.huang2@amd.com>  wrote:
>>
>>> I am adding interrupt support for LWP, whose spec is available at
>>> http://support.amd.com/us/Processor_TechDocs/43724.pdf. Basically OS
>>> can specify an interrupt vector in LWP_CFG MSR; the interrupt will be
>>> triggered when event buffer overflows. For HVM guests, I want to
>>> re-inject this interrupt back into the guest VM. Here is one idea
>>> similar to virtualized PMU: It first registers a special interrupt
>>> handler (say on vector 0xf6) using set_intr_gate(). When triggered,
>>> this handler injects an IRQ (with vector copied from LWP_CFG) into
>>> guest VM via virtual local APIC. This worked from my test.
>>>
>>> But adding a interrupt handler seems to be an overkill. Is there any
>>> better way to create a dummy interrupt receiver on be-behalf of guest
>>> VMs? I also looked into IRQ and MSI solutions inside Xen. But most of
>>> them assume that interrupts are from physical device (but not in this
>>> LWP case, where interrupt is initiated from CPU itself); so they don't
>>> fit very well.
>> I think just allocating a vector is fine. If we get too many we could move to
>> dynamic allocation of them.
>>
>>   -- Keir
>>
>>> Thanks,
>>> -Wei
>>>
>>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel


[-- Attachment #2: RFC_amd_lwp_interrupt.txt --]
[-- Type: text/plain, Size: 2901 bytes --]

# HG changeset patch
# User Wei Huang <wei.huang2@amd.com>
# Date 1333061831 18000
# Node ID 07f17f70ffe4e733eff1965e011be5693622d129
# Parent  d7fdc948f6540427c4c20da216b304fc7a11cd76
AMD_LWP: add interrupt support for AMD LWP

This patch adds interrupt support for AMD lightweight profiling. It
registers interrupt handler using CPU affiliated interrupt delegate.
Hypervisor reinjects virtual interrupt into guest VM using local APIC.

Signed-off-by: Wei Huang <wei.huang2@amd.com>

diff -r d7fdc948f654 -r 07f17f70ffe4 xen/arch/x86/hvm/svm/svm.c
--- a/xen/arch/x86/hvm/svm/svm.c	Thu Mar 29 17:51:16 2012 -0500
+++ b/xen/arch/x86/hvm/svm/svm.c	Thu Mar 29 17:57:11 2012 -0500
@@ -759,7 +759,17 @@
 {
     /* Only LWP_CFG is reloaded. LWP_CBADDR will be reloaded via xrstor. */
    if ( v->arch.hvm_svm.guest_lwp_cfg ) 
-       wrmsrl(MSR_AMD64_LWP_CFG, v->arch.hvm_svm.guest_lwp_cfg);
+       wrmsrl(MSR_AMD64_LWP_CFG, v->arch.hvm_svm.cpu_lwp_cfg);
+}
+
+int svm_lwp_intr_handler(struct cpu_user_regs *regs)
+{
+    struct vcpu *v = current;
+
+    vlapic_set_irq(vcpu_vlapic(v), 
+                   (v->arch.hvm_svm.guest_lwp_cfg >> 40) && 0xff, 0);
+
+    return 1;
 }
 
 /* Update LWP_CFG MSR (0xc0000105). Return -1 if error; otherwise returns 0. */
@@ -776,12 +786,24 @@
         /* generate #GP if guest tries to turn on unsupported features. */
         if ( msr_low & ~edx)
             return -1;
-        
-        wrmsrl(MSR_AMD64_LWP_CFG, msr_content);
-        /* CPU might automatically correct reserved bits. So read it back. */
-        rdmsrl(MSR_AMD64_LWP_CFG, msr_content);
+
         v->arch.hvm_svm.guest_lwp_cfg = msr_content;
 
+        /* setup interrupt handler */
+        if ( (msr_content & 0x80000000) && ((msr_content >> 40) & 0xff) )
+        {
+            v->arch.hvm_svm.cpu_lwp_cfg = (msr_content & 0xffff00ffffffffffULL)
+                | ((uint64_t)CPU_AFFILIATE_VECTOR << 40);
+            v->arch.hvm_vcpu.affiliate_intr_handler = svm_lwp_intr_handler;
+        }
+        else
+        {
+            /* disable interrupt */
+            v->arch.hvm_svm.cpu_lwp_cfg = msr_content & 0xffff00ff7fffffffULL;
+            v->arch.hvm_vcpu.affiliate_intr_handler = NULL;
+        }
+        wrmsrl(MSR_AMD64_LWP_CFG, v->arch.hvm_svm.cpu_lwp_cfg);
+
         /* track nonalzy state if LWP_CFG is non-zero. */
         v->arch.nonlazy_xstate_used = !!(msr_content);
     }
diff -r d7fdc948f654 -r 07f17f70ffe4 xen/include/asm-x86/hvm/svm/vmcb.h
--- a/xen/include/asm-x86/hvm/svm/vmcb.h	Thu Mar 29 17:51:16 2012 -0500
+++ b/xen/include/asm-x86/hvm/svm/vmcb.h	Thu Mar 29 17:57:11 2012 -0500
@@ -515,7 +515,8 @@
     uint64_t guest_sysenter_eip;
     
     /* AMD lightweight profiling MSR */
-    uint64_t guest_lwp_cfg;
+    uint64_t guest_lwp_cfg;       /* guest version */
+    uint64_t cpu_lwp_cfg;          /* cpu version */
 
     /* OSVW MSRs */
     struct {

[-- Attachment #3: RFC_cpu_interrupt_delegate.txt --]
[-- Type: text/plain, Size: 5709 bytes --]

# HG changeset patch
# User Wei Huang <wei.huang2@amd.com>
# Date 1333061476 18000
# Node ID d7fdc948f6540427c4c20da216b304fc7a11cd76
# Parent  86fac5ca5d9234d293047fc7b310127f7eaa91f3
x86: add cpu affiliated interrupt delegate for guest VM

This patch adds a delegate on-behalf-of guest VM for interrupts which
are initiated by CPU. When this interrupt is triggered, hypervisor
calls registered interrupt handler to notify guest that the interrupt
has arrived. Guest VM can disable receiving this notification by
setting handler as NULL.

Signed-off-by: Wei Huang <wei.huang2@amd.com>

diff -r 86fac5ca5d92 -r d7fdc948f654 xen/arch/x86/apic.c
--- a/xen/arch/x86/apic.c	Thu Mar 29 17:46:18 2012 -0500
+++ b/xen/arch/x86/apic.c	Thu Mar 29 17:51:16 2012 -0500
@@ -111,6 +111,13 @@
 BUILD_SMP_INTERRUPT(thermal_interrupt,THERMAL_APIC_VECTOR)
 #endif
 
+/* The following vector is received on-behalf-of guest VM for interrupts which
+ * are initiated by CPU locally. One example is light-weight profiling (LWP),
+ * which can trigger an interrupt when event buffer overflows. Hypervisor will
+ * notify guest VM about the arrival of interrupts.
+ */
+BUILD_SMP_INTERRUPT(cpu_affiliate_interrupt, CPU_AFFILIATE_VECTOR)
+
 static int modern_apic(void)
 {
     unsigned int lvr, version;
diff -r 86fac5ca5d92 -r d7fdc948f654 xen/arch/x86/hvm/hvm.c
--- a/xen/arch/x86/hvm/hvm.c	Thu Mar 29 17:46:18 2012 -0500
+++ b/xen/arch/x86/hvm/hvm.c	Thu Mar 29 17:51:16 2012 -0500
@@ -1438,6 +1438,19 @@
     return -1;
 }
 
+int hvm_handle_affiliate_intr(struct cpu_user_regs *regs)
+{
+    struct vcpu *curr = current;
+
+    if ( !curr->arch.hvm_vcpu.affiliate_intr_handler )
+    {
+        printk("HVM: received an unregistered CPU affiliated interrupt!\n");
+        return 0;
+    }
+    else
+        return curr->arch.hvm_vcpu.affiliate_intr_handler(regs);
+}
+
 int hvm_set_efer(uint64_t value)
 {
     struct vcpu *v = current;
diff -r 86fac5ca5d92 -r d7fdc948f654 xen/arch/x86/irq.c
--- a/xen/arch/x86/irq.c	Thu Mar 29 17:46:18 2012 -0500
+++ b/xen/arch/x86/irq.c	Thu Mar 29 17:51:16 2012 -0500
@@ -608,6 +608,16 @@
     desc->handler->enable(desc);
 }
 
+fastcall void smp_cpu_affiliate_interrupt(struct cpu_user_regs *regs)
+{
+    struct cpu_user_regs *old_regs = set_irq_regs(regs);
+
+    ack_APIC_irq();
+    this_cpu(irq_count)++;
+    hvm_handle_affiliate_intr(regs);
+    set_irq_regs(old_regs);
+}
+
 static void dump_irqs(unsigned char key);
 
 fastcall void smp_irq_move_cleanup_interrupt(struct cpu_user_regs *regs)
diff -r 86fac5ca5d92 -r d7fdc948f654 xen/arch/x86/smpboot.c
--- a/xen/arch/x86/smpboot.c	Thu Mar 29 17:46:18 2012 -0500
+++ b/xen/arch/x86/smpboot.c	Thu Mar 29 17:51:16 2012 -0500
@@ -1037,4 +1037,7 @@
 
     /* IPI for generic function call */
     set_intr_gate(CALL_FUNCTION_VECTOR, call_function_interrupt);
+
+    /* CPU affiliate interrupt delegate */
+    set_intr_gate(CPU_AFFILIATE_VECTOR, cpu_affiliate_interrupt);
 }
diff -r 86fac5ca5d92 -r d7fdc948f654 xen/include/asm-x86/hvm/hvm.h
--- a/xen/include/asm-x86/hvm/hvm.h	Thu Mar 29 17:46:18 2012 -0500
+++ b/xen/include/asm-x86/hvm/hvm.h	Thu Mar 29 17:51:16 2012 -0500
@@ -309,6 +309,7 @@
 void hvm_migrate_pirqs(struct vcpu *v);
 
 void hvm_inject_exception(unsigned int trapnr, int errcode, unsigned long cr2);
+int hvm_handle_affiliate_intr(struct cpu_user_regs *regs);
 
 static inline int hvm_event_pending(struct vcpu *v)
 {
diff -r 86fac5ca5d92 -r d7fdc948f654 xen/include/asm-x86/hvm/vcpu.h
--- a/xen/include/asm-x86/hvm/vcpu.h	Thu Mar 29 17:46:18 2012 -0500
+++ b/xen/include/asm-x86/hvm/vcpu.h	Thu Mar 29 17:51:16 2012 -0500
@@ -170,6 +170,10 @@
     unsigned long inject_cr2;
 
     struct viridian_vcpu viridian;
+    
+    /* callback handler for CPU affiliated interrupt. Guest VM can setup this 
+     * handler if it wants to receive notification. NULL means no handler. */
+    int (*affiliate_intr_handler)(struct cpu_user_regs *regs);
 };
 
 #endif /* __ASM_X86_HVM_VCPU_H__ */
diff -r 86fac5ca5d92 -r d7fdc948f654 xen/include/asm-x86/irq.h
--- a/xen/include/asm-x86/irq.h	Thu Mar 29 17:46:18 2012 -0500
+++ b/xen/include/asm-x86/irq.h	Thu Mar 29 17:51:16 2012 -0500
@@ -89,6 +89,7 @@
 fastcall void thermal_interrupt(void);
 fastcall void cmci_interrupt(void);
 fastcall void irq_move_cleanup_interrupt(void);
+fastcall void cpu_affiliate_interrupt(void);
 
 fastcall void smp_event_check_interrupt(struct cpu_user_regs *regs);
 fastcall void smp_invalidate_interrupt(void);
@@ -100,6 +101,7 @@
 fastcall void smp_thermal_interrupt(struct cpu_user_regs *regs);
 fastcall void smp_cmci_interrupt(struct cpu_user_regs *regs);
 fastcall void smp_irq_move_cleanup_interrupt(struct cpu_user_regs *regs);
+fastcall void smp_cpu_affiliate_interrupt(struct cpu_user_regs *regs);
 
 void do_IRQ(struct cpu_user_regs *regs);
 
diff -r 86fac5ca5d92 -r d7fdc948f654 xen/include/asm-x86/mach-default/irq_vectors.h
--- a/xen/include/asm-x86/mach-default/irq_vectors.h	Thu Mar 29 17:46:18 2012 -0500
+++ b/xen/include/asm-x86/mach-default/irq_vectors.h	Thu Mar 29 17:51:16 2012 -0500
@@ -11,12 +11,18 @@
 #define LOCAL_TIMER_VECTOR	0xf9
 #define PMU_APIC_VECTOR 	0xf8
 #define CMCI_APIC_VECTOR	0xf7
+
+/* CPU affiliate interrupt vector, which is used for interrupts that are
+ * initiated by CPU locally. Hypervisor receives on-behalf-of guest VM.
+ */
+#define CPU_AFFILIATE_VECTOR    0xf6
+
 /*
  * High-priority dynamically-allocated vectors. For interrupts that
  * must be higher priority than any guest-bound interrupt.
  */
 #define FIRST_HIPRIORITY_VECTOR	0xf0
-#define LAST_HIPRIORITY_VECTOR  0xf6
+#define LAST_HIPRIORITY_VECTOR  0xf5
 
 /* Legacy PIC uses vectors 0xe0-0xef. */
 #define FIRST_LEGACY_VECTOR	0xe0

[-- Attachment #4: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel