* LWP Interrupt Handler @ 2012-03-23 22:03 Wei Huang 2012-03-23 22:49 ` Keir Fraser 0 siblings, 1 reply; 8+ messages in thread From: Wei Huang @ 2012-03-23 22:03 UTC (permalink / raw) To: xen-devel, Keir Fraser, Jan Beulich I am adding interrupt support for LWP, whose spec is available at http://support.amd.com/us/Processor_TechDocs/43724.pdf. Basically OS can specify an interrupt vector in LWP_CFG MSR; the interrupt will be triggered when event buffer overflows. For HVM guests, I want to re-inject this interrupt back into the guest VM. Here is one idea similar to virtualized PMU: It first registers a special interrupt handler (say on vector 0xf6) using set_intr_gate(). When triggered, this handler injects an IRQ (with vector copied from LWP_CFG) into guest VM via virtual local APIC. This worked from my test. But adding a interrupt handler seems to be an overkill. Is there any better way to create a dummy interrupt receiver on be-behalf of guest VMs? I also looked into IRQ and MSI solutions inside Xen. But most of them assume that interrupts are from physical device (but not in this LWP case, where interrupt is initiated from CPU itself); so they don't fit very well. Thanks, -Wei ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: LWP Interrupt Handler 2012-03-23 22:03 LWP Interrupt Handler Wei Huang @ 2012-03-23 22:49 ` Keir Fraser 2012-03-26 0:08 ` Zhang, Xiantao 0 siblings, 1 reply; 8+ messages in thread From: Keir Fraser @ 2012-03-23 22:49 UTC (permalink / raw) To: wei.huang2, xen-devel, Jan Beulich On 23/03/2012 22:03, "Wei Huang" <wei.huang2@amd.com> wrote: > I am adding interrupt support for LWP, whose spec is available at > http://support.amd.com/us/Processor_TechDocs/43724.pdf. Basically OS can > specify an interrupt vector in LWP_CFG MSR; the interrupt will be > triggered when event buffer overflows. For HVM guests, I want to > re-inject this interrupt back into the guest VM. Here is one idea > similar to virtualized PMU: It first registers a special interrupt > handler (say on vector 0xf6) using set_intr_gate(). When triggered, this > handler injects an IRQ (with vector copied from LWP_CFG) into guest VM > via virtual local APIC. This worked from my test. > > But adding a interrupt handler seems to be an overkill. Is there any > better way to create a dummy interrupt receiver on be-behalf of guest > VMs? I also looked into IRQ and MSI solutions inside Xen. But most of > them assume that interrupts are from physical device (but not in this > LWP case, where interrupt is initiated from CPU itself); so they don't > fit very well. I think just allocating a vector is fine. If we get too many we could move to dynamic allocation of them. -- Keir > Thanks, > -Wei > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: LWP Interrupt Handler 2012-03-23 22:49 ` Keir Fraser @ 2012-03-26 0:08 ` Zhang, Xiantao 2012-03-29 22:54 ` Wei Huang 0 siblings, 1 reply; 8+ messages in thread From: Zhang, Xiantao @ 2012-03-26 0:08 UTC (permalink / raw) To: Keir Fraser, wei.huang2@amd.com, xen-devel@lists.xen.org, Jan Beulich Cc: Zhang, Xiantao Please make sure the per-cpu vector is considered in your case. For CPU's built-in event, it always happens on all cpus, but IRQ-based events may only happen on some special cpus which are determined by apic's mode. Xiantao > -----Original Message----- > From: xen-devel-bounces@lists.xen.org [mailto:xen-devel- > bounces@lists.xen.org] On Behalf Of Keir Fraser > Sent: Saturday, March 24, 2012 6:50 AM > To: wei.huang2@amd.com; xen-devel@lists.xen.org; Jan Beulich > Subject: Re: [Xen-devel] LWP Interrupt Handler > > On 23/03/2012 22:03, "Wei Huang" <wei.huang2@amd.com> wrote: > > > I am adding interrupt support for LWP, whose spec is available at > > http://support.amd.com/us/Processor_TechDocs/43724.pdf. Basically OS > > can specify an interrupt vector in LWP_CFG MSR; the interrupt will be > > triggered when event buffer overflows. For HVM guests, I want to > > re-inject this interrupt back into the guest VM. Here is one idea > > similar to virtualized PMU: It first registers a special interrupt > > handler (say on vector 0xf6) using set_intr_gate(). When triggered, > > this handler injects an IRQ (with vector copied from LWP_CFG) into > > guest VM via virtual local APIC. This worked from my test. > > > > But adding a interrupt handler seems to be an overkill. Is there any > > better way to create a dummy interrupt receiver on be-behalf of guest > > VMs? I also looked into IRQ and MSI solutions inside Xen. But most of > > them assume that interrupts are from physical device (but not in this > > LWP case, where interrupt is initiated from CPU itself); so they don't > > fit very well. > > I think just allocating a vector is fine. If we get too many we could move to > dynamic allocation of them. > > -- Keir > > > Thanks, > > -Wei > > > > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: LWP Interrupt Handler 2012-03-26 0:08 ` Zhang, Xiantao @ 2012-03-29 22:54 ` Wei Huang 2012-03-30 8:21 ` Keir Fraser 0 siblings, 1 reply; 8+ messages in thread From: Wei Huang @ 2012-03-29 22:54 UTC (permalink / raw) To: Zhang, Xiantao; +Cc: Keir Fraser, Jan Beulich, xen-devel@lists.xen.org [-- Attachment #1: Type: text/plain, Size: 2241 bytes --] How about the approach in attached patches? A interrupt handler is registered at vector 0xf6 as a delegate for guest VM. Guest VCPU can register interrupt handler to receive notification. I will prepare formal ones after receiving your comments. Thanks, -Wei On 03/25/2012 07:08 PM, Zhang, Xiantao wrote: > Please make sure the per-cpu vector is considered in your case. For CPU's built-in event, it always happens on all cpus, but IRQ-based events may only happen on some special cpus which are determined by apic's mode. > Xiantao > >> -----Original Message----- >> From: xen-devel-bounces@lists.xen.org [mailto:xen-devel- >> bounces@lists.xen.org] On Behalf Of Keir Fraser >> Sent: Saturday, March 24, 2012 6:50 AM >> To: wei.huang2@amd.com; xen-devel@lists.xen.org; Jan Beulich >> Subject: Re: [Xen-devel] LWP Interrupt Handler >> >> On 23/03/2012 22:03, "Wei Huang"<wei.huang2@amd.com> wrote: >> >>> I am adding interrupt support for LWP, whose spec is available at >>> http://support.amd.com/us/Processor_TechDocs/43724.pdf. Basically OS >>> can specify an interrupt vector in LWP_CFG MSR; the interrupt will be >>> triggered when event buffer overflows. For HVM guests, I want to >>> re-inject this interrupt back into the guest VM. Here is one idea >>> similar to virtualized PMU: It first registers a special interrupt >>> handler (say on vector 0xf6) using set_intr_gate(). When triggered, >>> this handler injects an IRQ (with vector copied from LWP_CFG) into >>> guest VM via virtual local APIC. This worked from my test. >>> >>> But adding a interrupt handler seems to be an overkill. Is there any >>> better way to create a dummy interrupt receiver on be-behalf of guest >>> VMs? I also looked into IRQ and MSI solutions inside Xen. But most of >>> them assume that interrupts are from physical device (but not in this >>> LWP case, where interrupt is initiated from CPU itself); so they don't >>> fit very well. >> I think just allocating a vector is fine. If we get too many we could move to >> dynamic allocation of them. >> >> -- Keir >> >>> Thanks, >>> -Wei >>> >>> >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-devel [-- Attachment #2: RFC_amd_lwp_interrupt.txt --] [-- Type: text/plain, Size: 2901 bytes --] # HG changeset patch # User Wei Huang <wei.huang2@amd.com> # Date 1333061831 18000 # Node ID 07f17f70ffe4e733eff1965e011be5693622d129 # Parent d7fdc948f6540427c4c20da216b304fc7a11cd76 AMD_LWP: add interrupt support for AMD LWP This patch adds interrupt support for AMD lightweight profiling. It registers interrupt handler using CPU affiliated interrupt delegate. Hypervisor reinjects virtual interrupt into guest VM using local APIC. Signed-off-by: Wei Huang <wei.huang2@amd.com> diff -r d7fdc948f654 -r 07f17f70ffe4 xen/arch/x86/hvm/svm/svm.c --- a/xen/arch/x86/hvm/svm/svm.c Thu Mar 29 17:51:16 2012 -0500 +++ b/xen/arch/x86/hvm/svm/svm.c Thu Mar 29 17:57:11 2012 -0500 @@ -759,7 +759,17 @@ { /* Only LWP_CFG is reloaded. LWP_CBADDR will be reloaded via xrstor. */ if ( v->arch.hvm_svm.guest_lwp_cfg ) - wrmsrl(MSR_AMD64_LWP_CFG, v->arch.hvm_svm.guest_lwp_cfg); + wrmsrl(MSR_AMD64_LWP_CFG, v->arch.hvm_svm.cpu_lwp_cfg); +} + +int svm_lwp_intr_handler(struct cpu_user_regs *regs) +{ + struct vcpu *v = current; + + vlapic_set_irq(vcpu_vlapic(v), + (v->arch.hvm_svm.guest_lwp_cfg >> 40) && 0xff, 0); + + return 1; } /* Update LWP_CFG MSR (0xc0000105). Return -1 if error; otherwise returns 0. */ @@ -776,12 +786,24 @@ /* generate #GP if guest tries to turn on unsupported features. */ if ( msr_low & ~edx) return -1; - - wrmsrl(MSR_AMD64_LWP_CFG, msr_content); - /* CPU might automatically correct reserved bits. So read it back. */ - rdmsrl(MSR_AMD64_LWP_CFG, msr_content); + v->arch.hvm_svm.guest_lwp_cfg = msr_content; + /* setup interrupt handler */ + if ( (msr_content & 0x80000000) && ((msr_content >> 40) & 0xff) ) + { + v->arch.hvm_svm.cpu_lwp_cfg = (msr_content & 0xffff00ffffffffffULL) + | ((uint64_t)CPU_AFFILIATE_VECTOR << 40); + v->arch.hvm_vcpu.affiliate_intr_handler = svm_lwp_intr_handler; + } + else + { + /* disable interrupt */ + v->arch.hvm_svm.cpu_lwp_cfg = msr_content & 0xffff00ff7fffffffULL; + v->arch.hvm_vcpu.affiliate_intr_handler = NULL; + } + wrmsrl(MSR_AMD64_LWP_CFG, v->arch.hvm_svm.cpu_lwp_cfg); + /* track nonalzy state if LWP_CFG is non-zero. */ v->arch.nonlazy_xstate_used = !!(msr_content); } diff -r d7fdc948f654 -r 07f17f70ffe4 xen/include/asm-x86/hvm/svm/vmcb.h --- a/xen/include/asm-x86/hvm/svm/vmcb.h Thu Mar 29 17:51:16 2012 -0500 +++ b/xen/include/asm-x86/hvm/svm/vmcb.h Thu Mar 29 17:57:11 2012 -0500 @@ -515,7 +515,8 @@ uint64_t guest_sysenter_eip; /* AMD lightweight profiling MSR */ - uint64_t guest_lwp_cfg; + uint64_t guest_lwp_cfg; /* guest version */ + uint64_t cpu_lwp_cfg; /* cpu version */ /* OSVW MSRs */ struct { [-- Attachment #3: RFC_cpu_interrupt_delegate.txt --] [-- Type: text/plain, Size: 5709 bytes --] # HG changeset patch # User Wei Huang <wei.huang2@amd.com> # Date 1333061476 18000 # Node ID d7fdc948f6540427c4c20da216b304fc7a11cd76 # Parent 86fac5ca5d9234d293047fc7b310127f7eaa91f3 x86: add cpu affiliated interrupt delegate for guest VM This patch adds a delegate on-behalf-of guest VM for interrupts which are initiated by CPU. When this interrupt is triggered, hypervisor calls registered interrupt handler to notify guest that the interrupt has arrived. Guest VM can disable receiving this notification by setting handler as NULL. Signed-off-by: Wei Huang <wei.huang2@amd.com> diff -r 86fac5ca5d92 -r d7fdc948f654 xen/arch/x86/apic.c --- a/xen/arch/x86/apic.c Thu Mar 29 17:46:18 2012 -0500 +++ b/xen/arch/x86/apic.c Thu Mar 29 17:51:16 2012 -0500 @@ -111,6 +111,13 @@ BUILD_SMP_INTERRUPT(thermal_interrupt,THERMAL_APIC_VECTOR) #endif +/* The following vector is received on-behalf-of guest VM for interrupts which + * are initiated by CPU locally. One example is light-weight profiling (LWP), + * which can trigger an interrupt when event buffer overflows. Hypervisor will + * notify guest VM about the arrival of interrupts. + */ +BUILD_SMP_INTERRUPT(cpu_affiliate_interrupt, CPU_AFFILIATE_VECTOR) + static int modern_apic(void) { unsigned int lvr, version; diff -r 86fac5ca5d92 -r d7fdc948f654 xen/arch/x86/hvm/hvm.c --- a/xen/arch/x86/hvm/hvm.c Thu Mar 29 17:46:18 2012 -0500 +++ b/xen/arch/x86/hvm/hvm.c Thu Mar 29 17:51:16 2012 -0500 @@ -1438,6 +1438,19 @@ return -1; } +int hvm_handle_affiliate_intr(struct cpu_user_regs *regs) +{ + struct vcpu *curr = current; + + if ( !curr->arch.hvm_vcpu.affiliate_intr_handler ) + { + printk("HVM: received an unregistered CPU affiliated interrupt!\n"); + return 0; + } + else + return curr->arch.hvm_vcpu.affiliate_intr_handler(regs); +} + int hvm_set_efer(uint64_t value) { struct vcpu *v = current; diff -r 86fac5ca5d92 -r d7fdc948f654 xen/arch/x86/irq.c --- a/xen/arch/x86/irq.c Thu Mar 29 17:46:18 2012 -0500 +++ b/xen/arch/x86/irq.c Thu Mar 29 17:51:16 2012 -0500 @@ -608,6 +608,16 @@ desc->handler->enable(desc); } +fastcall void smp_cpu_affiliate_interrupt(struct cpu_user_regs *regs) +{ + struct cpu_user_regs *old_regs = set_irq_regs(regs); + + ack_APIC_irq(); + this_cpu(irq_count)++; + hvm_handle_affiliate_intr(regs); + set_irq_regs(old_regs); +} + static void dump_irqs(unsigned char key); fastcall void smp_irq_move_cleanup_interrupt(struct cpu_user_regs *regs) diff -r 86fac5ca5d92 -r d7fdc948f654 xen/arch/x86/smpboot.c --- a/xen/arch/x86/smpboot.c Thu Mar 29 17:46:18 2012 -0500 +++ b/xen/arch/x86/smpboot.c Thu Mar 29 17:51:16 2012 -0500 @@ -1037,4 +1037,7 @@ /* IPI for generic function call */ set_intr_gate(CALL_FUNCTION_VECTOR, call_function_interrupt); + + /* CPU affiliate interrupt delegate */ + set_intr_gate(CPU_AFFILIATE_VECTOR, cpu_affiliate_interrupt); } diff -r 86fac5ca5d92 -r d7fdc948f654 xen/include/asm-x86/hvm/hvm.h --- a/xen/include/asm-x86/hvm/hvm.h Thu Mar 29 17:46:18 2012 -0500 +++ b/xen/include/asm-x86/hvm/hvm.h Thu Mar 29 17:51:16 2012 -0500 @@ -309,6 +309,7 @@ void hvm_migrate_pirqs(struct vcpu *v); void hvm_inject_exception(unsigned int trapnr, int errcode, unsigned long cr2); +int hvm_handle_affiliate_intr(struct cpu_user_regs *regs); static inline int hvm_event_pending(struct vcpu *v) { diff -r 86fac5ca5d92 -r d7fdc948f654 xen/include/asm-x86/hvm/vcpu.h --- a/xen/include/asm-x86/hvm/vcpu.h Thu Mar 29 17:46:18 2012 -0500 +++ b/xen/include/asm-x86/hvm/vcpu.h Thu Mar 29 17:51:16 2012 -0500 @@ -170,6 +170,10 @@ unsigned long inject_cr2; struct viridian_vcpu viridian; + + /* callback handler for CPU affiliated interrupt. Guest VM can setup this + * handler if it wants to receive notification. NULL means no handler. */ + int (*affiliate_intr_handler)(struct cpu_user_regs *regs); }; #endif /* __ASM_X86_HVM_VCPU_H__ */ diff -r 86fac5ca5d92 -r d7fdc948f654 xen/include/asm-x86/irq.h --- a/xen/include/asm-x86/irq.h Thu Mar 29 17:46:18 2012 -0500 +++ b/xen/include/asm-x86/irq.h Thu Mar 29 17:51:16 2012 -0500 @@ -89,6 +89,7 @@ fastcall void thermal_interrupt(void); fastcall void cmci_interrupt(void); fastcall void irq_move_cleanup_interrupt(void); +fastcall void cpu_affiliate_interrupt(void); fastcall void smp_event_check_interrupt(struct cpu_user_regs *regs); fastcall void smp_invalidate_interrupt(void); @@ -100,6 +101,7 @@ fastcall void smp_thermal_interrupt(struct cpu_user_regs *regs); fastcall void smp_cmci_interrupt(struct cpu_user_regs *regs); fastcall void smp_irq_move_cleanup_interrupt(struct cpu_user_regs *regs); +fastcall void smp_cpu_affiliate_interrupt(struct cpu_user_regs *regs); void do_IRQ(struct cpu_user_regs *regs); diff -r 86fac5ca5d92 -r d7fdc948f654 xen/include/asm-x86/mach-default/irq_vectors.h --- a/xen/include/asm-x86/mach-default/irq_vectors.h Thu Mar 29 17:46:18 2012 -0500 +++ b/xen/include/asm-x86/mach-default/irq_vectors.h Thu Mar 29 17:51:16 2012 -0500 @@ -11,12 +11,18 @@ #define LOCAL_TIMER_VECTOR 0xf9 #define PMU_APIC_VECTOR 0xf8 #define CMCI_APIC_VECTOR 0xf7 + +/* CPU affiliate interrupt vector, which is used for interrupts that are + * initiated by CPU locally. Hypervisor receives on-behalf-of guest VM. + */ +#define CPU_AFFILIATE_VECTOR 0xf6 + /* * High-priority dynamically-allocated vectors. For interrupts that * must be higher priority than any guest-bound interrupt. */ #define FIRST_HIPRIORITY_VECTOR 0xf0 -#define LAST_HIPRIORITY_VECTOR 0xf6 +#define LAST_HIPRIORITY_VECTOR 0xf5 /* Legacy PIC uses vectors 0xe0-0xef. */ #define FIRST_LEGACY_VECTOR 0xe0 [-- Attachment #4: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: LWP Interrupt Handler 2012-03-29 22:54 ` Wei Huang @ 2012-03-30 8:21 ` Keir Fraser 2012-03-30 19:44 ` Wei Huang 0 siblings, 1 reply; 8+ messages in thread From: Keir Fraser @ 2012-03-30 8:21 UTC (permalink / raw) To: wei.huang2, Zhang, Xiantao; +Cc: Jan Beulich, xen-devel@lists.xen.org On 29/03/2012 23:54, "Wei Huang" <wei.huang2@amd.com> wrote: > How about the approach in attached patches? A interrupt handler is > registered at vector 0xf6 as a delegate for guest VM. Guest VCPU can > register interrupt handler to receive notification. I will prepare > formal ones after receiving your comments. Ugh, no, these ad-hoc dynamically-registered subhandlers are getting beyond the pale. Therefore, I've just spent half hour routing all our interrupts, even the direct APIC ones, through do_IRQ(), and now allow everything to be dynamically allocated. Please pull up to xen-unstable tip (>= 25113) and use alloc_direct_apic_vector(). You can find a few callers of it that I added myself, to give you an idea how to use it. In essence, you can call it from your SVM-specific code, to allocate a vector for your irq handler. And you can do it immediately before you poke the vector into your special MSR. You can call alloc_direct_apic_vector() unconditionally -- it will deal with ensuring the allocation happens only once. -- Keir > Thanks, > -Wei > > > On 03/25/2012 07:08 PM, Zhang, Xiantao wrote: >> Please make sure the per-cpu vector is considered in your case. For CPU's >> built-in event, it always happens on all cpus, but IRQ-based events may only >> happen on some special cpus which are determined by apic's mode. >> Xiantao >> >>> -----Original Message----- >>> From: xen-devel-bounces@lists.xen.org [mailto:xen-devel- >>> bounces@lists.xen.org] On Behalf Of Keir Fraser >>> Sent: Saturday, March 24, 2012 6:50 AM >>> To: wei.huang2@amd.com; xen-devel@lists.xen.org; Jan Beulich >>> Subject: Re: [Xen-devel] LWP Interrupt Handler >>> >>> On 23/03/2012 22:03, "Wei Huang"<wei.huang2@amd.com> wrote: >>> >>>> I am adding interrupt support for LWP, whose spec is available at >>>> http://support.amd.com/us/Processor_TechDocs/43724.pdf. Basically OS >>>> can specify an interrupt vector in LWP_CFG MSR; the interrupt will be >>>> triggered when event buffer overflows. For HVM guests, I want to >>>> re-inject this interrupt back into the guest VM. Here is one idea >>>> similar to virtualized PMU: It first registers a special interrupt >>>> handler (say on vector 0xf6) using set_intr_gate(). When triggered, >>>> this handler injects an IRQ (with vector copied from LWP_CFG) into >>>> guest VM via virtual local APIC. This worked from my test. >>>> >>>> But adding a interrupt handler seems to be an overkill. Is there any >>>> better way to create a dummy interrupt receiver on be-behalf of guest >>>> VMs? I also looked into IRQ and MSI solutions inside Xen. But most of >>>> them assume that interrupts are from physical device (but not in this >>>> LWP case, where interrupt is initiated from CPU itself); so they don't >>>> fit very well. >>> I think just allocating a vector is fine. If we get too many we could move >>> to >>> dynamic allocation of them. >>> >>> -- Keir >>> >>>> Thanks, >>>> -Wei >>>> >>>> >>> >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xen.org >>> http://lists.xen.org/xen-devel > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: LWP Interrupt Handler 2012-03-30 8:21 ` Keir Fraser @ 2012-03-30 19:44 ` Wei Huang 2012-03-30 20:06 ` Keir Fraser 0 siblings, 1 reply; 8+ messages in thread From: Wei Huang @ 2012-03-30 19:44 UTC (permalink / raw) To: Keir Fraser; +Cc: Jan Beulich, Zhang, Xiantao, xen-devel@lists.xen.org [-- Attachment #1: Type: text/plain, Size: 3802 bytes --] Thanks. I created the LWP interrupt patch based on your solution. It is attached in this email. This patch was tested using Hans Rosenfeld's LWP tree (http://www.amd64.org/gitweb/?p=linux/lwp.git;a=summary). ================= AMD_LWP: add interrupt support for AMD LWP This patch adds interrupt support for AMD lightweight profiling. It registers interrupt handler using alloc_direct_apic_vector(). When notified, SVM reinjects virtual interrupts into guest VM using guest's virtual local APIC. Signed-off-by: Wei Huang <wei.huang2@amd.com> ================= -Wei On 03/30/2012 03:21 AM, Keir Fraser wrote: > On 29/03/2012 23:54, "Wei Huang"<wei.huang2@amd.com> wrote: > >> How about the approach in attached patches? A interrupt handler is >> registered at vector 0xf6 as a delegate for guest VM. Guest VCPU can >> register interrupt handler to receive notification. I will prepare >> formal ones after receiving your comments. > Ugh, no, these ad-hoc dynamically-registered subhandlers are getting beyond > the pale. Therefore, I've just spent half hour routing all our interrupts, > even the direct APIC ones, through do_IRQ(), and now allow everything to be > dynamically allocated. > > Please pull up to xen-unstable tip (>= 25113) and use > alloc_direct_apic_vector(). You can find a few callers of it that I added > myself, to give you an idea how to use it. In essence, you can call it from > your SVM-specific code, to allocate a vector for your irq handler. And you > can do it immediately before you poke the vector into your special MSR. You > can call alloc_direct_apic_vector() unconditionally -- it will deal with > ensuring the allocation happens only once. > > -- Keir > >> Thanks, >> -Wei >> >> >> On 03/25/2012 07:08 PM, Zhang, Xiantao wrote: >>> Please make sure the per-cpu vector is considered in your case. For CPU's >>> built-in event, it always happens on all cpus, but IRQ-based events may only >>> happen on some special cpus which are determined by apic's mode. >>> Xiantao >>> >>>> -----Original Message----- >>>> From: xen-devel-bounces@lists.xen.org [mailto:xen-devel- >>>> bounces@lists.xen.org] On Behalf Of Keir Fraser >>>> Sent: Saturday, March 24, 2012 6:50 AM >>>> To: wei.huang2@amd.com; xen-devel@lists.xen.org; Jan Beulich >>>> Subject: Re: [Xen-devel] LWP Interrupt Handler >>>> >>>> On 23/03/2012 22:03, "Wei Huang"<wei.huang2@amd.com> wrote: >>>> >>>>> I am adding interrupt support for LWP, whose spec is available at >>>>> http://support.amd.com/us/Processor_TechDocs/43724.pdf. Basically OS >>>>> can specify an interrupt vector in LWP_CFG MSR; the interrupt will be >>>>> triggered when event buffer overflows. For HVM guests, I want to >>>>> re-inject this interrupt back into the guest VM. Here is one idea >>>>> similar to virtualized PMU: It first registers a special interrupt >>>>> handler (say on vector 0xf6) using set_intr_gate(). When triggered, >>>>> this handler injects an IRQ (with vector copied from LWP_CFG) into >>>>> guest VM via virtual local APIC. This worked from my test. >>>>> >>>>> But adding a interrupt handler seems to be an overkill. Is there any >>>>> better way to create a dummy interrupt receiver on be-behalf of guest >>>>> VMs? I also looked into IRQ and MSI solutions inside Xen. But most of >>>>> them assume that interrupts are from physical device (but not in this >>>>> LWP case, where interrupt is initiated from CPU itself); so they don't >>>>> fit very well. >>>> I think just allocating a vector is fine. If we get too many we could move >>>> to >>>> dynamic allocation of them. >>>> >>>> -- Keir >>>> >>>>> Thanks, >>>>> -Wei >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> Xen-devel mailing list >>>> Xen-devel@lists.xen.org >>>> http://lists.xen.org/xen-devel > > [-- Attachment #2: amd_lwp_interrupt.txt --] [-- Type: text/plain, Size: 3635 bytes --] # HG changeset patch # User Wei Huang <wei.huang2@amd.com> # Date 1333136756 18000 # Node ID b3d45e0ffd4b7e2ea409e6d376951f33ca7feadf # Parent 6765a2510ee7ad899dcb87eefdf206f8c8ae34ae AMD_LWP: add interrupt support for AMD LWP This patch adds interrupt support for AMD lightweight profiling. It registers interrupt handler using alloc_direct_apic_vector(). When notified, SVM reinjects virtual interrupts into guest VM using guest's virtual local APIC. Signed-off-by: Wei Huang <wei.huang2@amd.com> diff -r 6765a2510ee7 -r b3d45e0ffd4b xen/arch/x86/hvm/svm/svm.c --- a/xen/arch/x86/hvm/svm/svm.c Fri Mar 30 10:01:15 2012 +0100 +++ b/xen/arch/x86/hvm/svm/svm.c Fri Mar 30 14:45:56 2012 -0500 @@ -87,6 +87,9 @@ static uint64_t osvw_length, osvw_status; static DEFINE_SPINLOCK(osvw_lock); +/* LWP Vector */ +static uint8_t lwp_intr_vector; + void __update_guest_eip(struct cpu_user_regs *regs, unsigned int inst_len) { struct vcpu *curr = current; @@ -745,6 +748,26 @@ *(u16 *)(hypercall_page + (__HYPERVISOR_iret * 32)) = 0x0b0f; /* ud2 */ } +/* LWP interrupt handler */ +static void svm_lwp_interrupt(struct cpu_user_regs *regs) +{ + struct vcpu *v = current; + + ack_APIC_irq(); + vlapic_set_irq(vcpu_vlapic(v), + (v->arch.hvm_svm.guest_lwp_cfg >> 40) && 0xff, 0); +} + +/* Init LWP interrupt handler */ +static void svm_init_lwp_intr(void) +{ + /* return if already allocated */ + if ( lwp_intr_vector > 0 ) + return; + + alloc_direct_apic_vector(&lwp_intr_vector, svm_lwp_interrupt); +} + static inline void svm_lwp_save(struct vcpu *v) { /* Don't mess up with other guests. Disable LWP for next VCPU. */ @@ -759,7 +782,7 @@ { /* Only LWP_CFG is reloaded. LWP_CBADDR will be reloaded via xrstor. */ if ( v->arch.hvm_svm.guest_lwp_cfg ) - wrmsrl(MSR_AMD64_LWP_CFG, v->arch.hvm_svm.guest_lwp_cfg); + wrmsrl(MSR_AMD64_LWP_CFG, v->arch.hvm_svm.cpu_lwp_cfg); } /* Update LWP_CFG MSR (0xc0000105). Return -1 if error; otherwise returns 0. */ @@ -776,11 +799,23 @@ /* generate #GP if guest tries to turn on unsupported features. */ if ( msr_low & ~edx) return -1; + + v->arch.hvm_svm.guest_lwp_cfg = msr_content; + + /* setup interrupt handler if needed */ + if ( (msr_content & 0x80000000) && ((msr_content >> 40) & 0xff) ) + { + svm_init_lwp_intr(); + v->arch.hvm_svm.cpu_lwp_cfg = (msr_content & 0xffff00ffffffffffULL) + | ((uint64_t)lwp_intr_vector << 40); + } + else + { + /* otherwise disable it */ + v->arch.hvm_svm.cpu_lwp_cfg = msr_content & 0xffff00ff7fffffffULL; + } - wrmsrl(MSR_AMD64_LWP_CFG, msr_content); - /* CPU might automatically correct reserved bits. So read it back. */ - rdmsrl(MSR_AMD64_LWP_CFG, msr_content); - v->arch.hvm_svm.guest_lwp_cfg = msr_content; + wrmsrl(MSR_AMD64_LWP_CFG, v->arch.hvm_svm.cpu_lwp_cfg); /* track nonalzy state if LWP_CFG is non-zero. */ v->arch.nonlazy_xstate_used = !!(msr_content); diff -r 6765a2510ee7 -r b3d45e0ffd4b xen/include/asm-x86/hvm/svm/vmcb.h --- a/xen/include/asm-x86/hvm/svm/vmcb.h Fri Mar 30 10:01:15 2012 +0100 +++ b/xen/include/asm-x86/hvm/svm/vmcb.h Fri Mar 30 14:45:56 2012 -0500 @@ -515,7 +515,8 @@ uint64_t guest_sysenter_eip; /* AMD lightweight profiling MSR */ - uint64_t guest_lwp_cfg; + uint64_t guest_lwp_cfg; /* guest version */ + uint64_t cpu_lwp_cfg; /* CPU version */ /* OSVW MSRs */ struct { [-- Attachment #3: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: LWP Interrupt Handler 2012-03-30 19:44 ` Wei Huang @ 2012-03-30 20:06 ` Keir Fraser 2012-03-30 20:11 ` Wei Huang 0 siblings, 1 reply; 8+ messages in thread From: Keir Fraser @ 2012-03-30 20:06 UTC (permalink / raw) To: wei.huang2; +Cc: Jan Beulich, Zhang, Xiantao, xen-devel@lists.xen.org On 30/03/2012 20:44, "Wei Huang" <wei.huang2@amd.com> wrote: > Thanks. I created the LWP interrupt patch based on your solution. It is > attached in this email. This patch was tested using Hans Rosenfeld's LWP > tree (http://www.amd64.org/gitweb/?p=linux/lwp.git;a=summary). Thanks, I've simplified the patch a bit and applied it as xen-unstable:25115. Please check it out. K. > ================= > AMD_LWP: add interrupt support for AMD LWP > > This patch adds interrupt support for AMD lightweight profiling. It > registers interrupt handler using alloc_direct_apic_vector(). When > notified, SVM reinjects virtual interrupts into guest VM using > guest's virtual local APIC. > > Signed-off-by: Wei Huang <wei.huang2@amd.com> > ================= > > -Wei > > On 03/30/2012 03:21 AM, Keir Fraser wrote: >> On 29/03/2012 23:54, "Wei Huang"<wei.huang2@amd.com> wrote: >> >>> How about the approach in attached patches? A interrupt handler is >>> registered at vector 0xf6 as a delegate for guest VM. Guest VCPU can >>> register interrupt handler to receive notification. I will prepare >>> formal ones after receiving your comments. >> Ugh, no, these ad-hoc dynamically-registered subhandlers are getting beyond >> the pale. Therefore, I've just spent half hour routing all our interrupts, >> even the direct APIC ones, through do_IRQ(), and now allow everything to be >> dynamically allocated. >> >> Please pull up to xen-unstable tip (>= 25113) and use >> alloc_direct_apic_vector(). You can find a few callers of it that I added >> myself, to give you an idea how to use it. In essence, you can call it from >> your SVM-specific code, to allocate a vector for your irq handler. And you >> can do it immediately before you poke the vector into your special MSR. You >> can call alloc_direct_apic_vector() unconditionally -- it will deal with >> ensuring the allocation happens only once. >> >> -- Keir >> >>> Thanks, >>> -Wei >>> >>> >>> On 03/25/2012 07:08 PM, Zhang, Xiantao wrote: >>>> Please make sure the per-cpu vector is considered in your case. For CPU's >>>> built-in event, it always happens on all cpus, but IRQ-based events may >>>> only >>>> happen on some special cpus which are determined by apic's mode. >>>> Xiantao >>>> >>>>> -----Original Message----- >>>>> From: xen-devel-bounces@lists.xen.org [mailto:xen-devel- >>>>> bounces@lists.xen.org] On Behalf Of Keir Fraser >>>>> Sent: Saturday, March 24, 2012 6:50 AM >>>>> To: wei.huang2@amd.com; xen-devel@lists.xen.org; Jan Beulich >>>>> Subject: Re: [Xen-devel] LWP Interrupt Handler >>>>> >>>>> On 23/03/2012 22:03, "Wei Huang"<wei.huang2@amd.com> wrote: >>>>> >>>>>> I am adding interrupt support for LWP, whose spec is available at >>>>>> http://support.amd.com/us/Processor_TechDocs/43724.pdf. Basically OS >>>>>> can specify an interrupt vector in LWP_CFG MSR; the interrupt will be >>>>>> triggered when event buffer overflows. For HVM guests, I want to >>>>>> re-inject this interrupt back into the guest VM. Here is one idea >>>>>> similar to virtualized PMU: It first registers a special interrupt >>>>>> handler (say on vector 0xf6) using set_intr_gate(). When triggered, >>>>>> this handler injects an IRQ (with vector copied from LWP_CFG) into >>>>>> guest VM via virtual local APIC. This worked from my test. >>>>>> >>>>>> But adding a interrupt handler seems to be an overkill. Is there any >>>>>> better way to create a dummy interrupt receiver on be-behalf of guest >>>>>> VMs? I also looked into IRQ and MSI solutions inside Xen. But most of >>>>>> them assume that interrupts are from physical device (but not in this >>>>>> LWP case, where interrupt is initiated from CPU itself); so they don't >>>>>> fit very well. >>>>> I think just allocating a vector is fine. If we get too many we could move >>>>> to >>>>> dynamic allocation of them. >>>>> >>>>> -- Keir >>>>> >>>>>> Thanks, >>>>>> -Wei >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Xen-devel mailing list >>>>> Xen-devel@lists.xen.org >>>>> http://lists.xen.org/xen-devel >> >> > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: LWP Interrupt Handler 2012-03-30 20:06 ` Keir Fraser @ 2012-03-30 20:11 ` Wei Huang 0 siblings, 0 replies; 8+ messages in thread From: Wei Huang @ 2012-03-30 20:11 UTC (permalink / raw) To: Keir Fraser; +Cc: Jan Beulich, Zhang, Xiantao, xen-devel@lists.xen.org Tested and it worked. Thanks, -Wei On 03/30/2012 03:06 PM, Keir Fraser wrote: > On 30/03/2012 20:44, "Wei Huang"<wei.huang2@amd.com> wrote: > >> Thanks. I created the LWP interrupt patch based on your solution. It is >> attached in this email. This patch was tested using Hans Rosenfeld's LWP >> tree (http://www.amd64.org/gitweb/?p=linux/lwp.git;a=summary). > Thanks, I've simplified the patch a bit and applied it as > xen-unstable:25115. Please check it out. > > K. > >> ================= >> AMD_LWP: add interrupt support for AMD LWP >> >> This patch adds interrupt support for AMD lightweight profiling. It >> registers interrupt handler using alloc_direct_apic_vector(). When >> notified, SVM reinjects virtual interrupts into guest VM using >> guest's virtual local APIC. >> >> Signed-off-by: Wei Huang<wei.huang2@amd.com> >> ================= >> >> -Wei >> >> On 03/30/2012 03:21 AM, Keir Fraser wrote: >>> On 29/03/2012 23:54, "Wei Huang"<wei.huang2@amd.com> wrote: >>> >>>> How about the approach in attached patches? A interrupt handler is >>>> registered at vector 0xf6 as a delegate for guest VM. Guest VCPU can >>>> register interrupt handler to receive notification. I will prepare >>>> formal ones after receiving your comments. >>> Ugh, no, these ad-hoc dynamically-registered subhandlers are getting beyond >>> the pale. Therefore, I've just spent half hour routing all our interrupts, >>> even the direct APIC ones, through do_IRQ(), and now allow everything to be >>> dynamically allocated. >>> >>> Please pull up to xen-unstable tip (>= 25113) and use >>> alloc_direct_apic_vector(). You can find a few callers of it that I added >>> myself, to give you an idea how to use it. In essence, you can call it from >>> your SVM-specific code, to allocate a vector for your irq handler. And you >>> can do it immediately before you poke the vector into your special MSR. You >>> can call alloc_direct_apic_vector() unconditionally -- it will deal with >>> ensuring the allocation happens only once. >>> >>> -- Keir >>> >>>> Thanks, >>>> -Wei >>>> >>>> >>>> On 03/25/2012 07:08 PM, Zhang, Xiantao wrote: >>>>> Please make sure the per-cpu vector is considered in your case. For CPU's >>>>> built-in event, it always happens on all cpus, but IRQ-based events may >>>>> only >>>>> happen on some special cpus which are determined by apic's mode. >>>>> Xiantao >>>>> >>>>>> -----Original Message----- >>>>>> From: xen-devel-bounces@lists.xen.org [mailto:xen-devel- >>>>>> bounces@lists.xen.org] On Behalf Of Keir Fraser >>>>>> Sent: Saturday, March 24, 2012 6:50 AM >>>>>> To: wei.huang2@amd.com; xen-devel@lists.xen.org; Jan Beulich >>>>>> Subject: Re: [Xen-devel] LWP Interrupt Handler >>>>>> >>>>>> On 23/03/2012 22:03, "Wei Huang"<wei.huang2@amd.com> wrote: >>>>>> >>>>>>> I am adding interrupt support for LWP, whose spec is available at >>>>>>> http://support.amd.com/us/Processor_TechDocs/43724.pdf. Basically OS >>>>>>> can specify an interrupt vector in LWP_CFG MSR; the interrupt will be >>>>>>> triggered when event buffer overflows. For HVM guests, I want to >>>>>>> re-inject this interrupt back into the guest VM. Here is one idea >>>>>>> similar to virtualized PMU: It first registers a special interrupt >>>>>>> handler (say on vector 0xf6) using set_intr_gate(). When triggered, >>>>>>> this handler injects an IRQ (with vector copied from LWP_CFG) into >>>>>>> guest VM via virtual local APIC. This worked from my test. >>>>>>> >>>>>>> But adding a interrupt handler seems to be an overkill. Is there any >>>>>>> better way to create a dummy interrupt receiver on be-behalf of guest >>>>>>> VMs? I also looked into IRQ and MSI solutions inside Xen. But most of >>>>>>> them assume that interrupts are from physical device (but not in this >>>>>>> LWP case, where interrupt is initiated from CPU itself); so they don't >>>>>>> fit very well. >>>>>> I think just allocating a vector is fine. If we get too many we could move >>>>>> to >>>>>> dynamic allocation of them. >>>>>> >>>>>> -- Keir >>>>>> >>>>>>> Thanks, >>>>>>> -Wei >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> Xen-devel mailing list >>>>>> Xen-devel@lists.xen.org >>>>>> http://lists.xen.org/xen-devel >>> > > ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2012-03-30 20:11 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-03-23 22:03 LWP Interrupt Handler Wei Huang 2012-03-23 22:49 ` Keir Fraser 2012-03-26 0:08 ` Zhang, Xiantao 2012-03-29 22:54 ` Wei Huang 2012-03-30 8:21 ` Keir Fraser 2012-03-30 19:44 ` Wei Huang 2012-03-30 20:06 ` Keir Fraser 2012-03-30 20:11 ` Wei Huang
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).