[PATCH 4/5][RFC] lwp: adding support for AMD lightweight profiling

[PATCH 4/5][RFC] lwp: adding support for AMD lightweight profiling

[Wei Huang]

[-- Attachment #1: Type: text/plain, Size: 398 bytes --]

Add xsave/xrstor support for LWP

Because LWP is not tracked by TS, which is used in current xsave/xrstor 
implementation for FPU/AVX. As a result, we have to save and restore LWP 
in extended area whenever vcpu is re-scheduled. To avoid unnecessary lwp 
save/restore, this patch checks the LWP_CBADDR to determine whether LWP 
has been turned on.

Signed-off-by: Wei Huang <wei.huang2@amd.com>




[-- Attachment #2: lwp_patch_4.txt --]
[-- Type: text/plain, Size: 3428 bytes --]

exporting patch:
# HG changeset patch
# User Wei Huang <wei.huang2@amd.com>
# Date 1297378527 21600
# Node ID 1a1d593c16f36e9cf0153b2ca90dddae649a52bd
# Parent  705de1dacabeb899fae6e8121483b2d88d120e57
Add xsave/xrstor support for LWP

Because LWP is not tracked by TS, which is used in current xsave/xrstor implementation for FPU/AVX. As a result, we have to save and restore LWP in extended area whenever vcpu is re-scheduled. To avoid unnecessary lwp save/restore, this patch checks the LWP_CBADDR to determine whether LWP has been turned on.

diff -r 705de1dacabe -r 1a1d593c16f3 xen/arch/x86/domain.c
--- a/xen/arch/x86/domain.c	Thu Feb 10 16:25:09 2011 -0600
+++ b/xen/arch/x86/domain.c	Thu Feb 10 16:55:27 2011 -0600
@@ -1385,6 +1385,7 @@
                stack_regs,
                CTXT_SWITCH_STACK_BYTES);
         save_init_fpu(p);
+        xsave_lwp(p);
         p->arch.ctxt_switch_from(p);
     }
 
@@ -1404,6 +1405,7 @@
                CTXT_SWITCH_STACK_BYTES);
         if ( cpu_has_xsave && n->arch.xcr0 != get_xcr0() )
             set_xcr0(n->arch.xcr0);
+        xrstor_lwp(n);
         n->arch.ctxt_switch_to(n);
     }
 
diff -r 705de1dacabe -r 1a1d593c16f3 xen/arch/x86/i387.c
--- a/xen/arch/x86/i387.c	Thu Feb 10 16:25:09 2011 -0600
+++ b/xen/arch/x86/i387.c	Thu Feb 10 16:55:27 2011 -0600
@@ -65,6 +65,55 @@
 static void init_fpu(void);
 static void restore_fpu(struct vcpu *v);
 
+/* Save AMD LWP */
+void xsave_lwp(struct vcpu *v)
+{
+    uint64_t lwpcb;
+    bool_t ts;
+    struct xsave_struct *xsave_area = v->arch.xsave_area;
+
+    if ( cpu_has_lwp )
+    {
+        /* Has LWP been used? */
+        rdmsrl(MSR_AMD_LWP_CBADDR, lwpcb);
+        if ( !lwpcb ) {
+            /* Guest might have turned off LWP. So clean the bit here. */
+            xsave_area->xsave_hdr.xstate_bv &= ~XSTATE_LWP;
+            return;
+        }
+        
+        /* Disable TS temporarily to avoid recursion. */
+        ts = read_cr0() & X86_CR0_TS;
+        clts();
+        xsave(v, XSTATE_LWP);
+        if ( ts )
+            stts();
+
+        /* disable LWP for next VCPU */
+        wrmsrl(MSR_AMD_LWP_CBADDR, 0);
+    }
+}
+
+void xrstor_lwp(struct vcpu *v)
+{
+    bool_t ts;
+    struct xsave_struct *xsave_area = v->arch.xsave_area;
+
+    if ( cpu_has_lwp )
+    {
+        /* Don't do anything if there is no LWP state for this VCPU */
+        if ( !(xsave_area->xsave_hdr.xstate_bv & XSTATE_LWP) )
+            return;
+
+        /* Disable TS temporarily to avoid recursion. */
+        ts = read_cr0() & X86_CR0_TS;
+        clts();
+        xrstor(v, XSTATE_LWP);
+        if ( ts )
+            stts();
+    }
+}
+
 void setup_fpu(struct vcpu *v)
 {
     ASSERT(!is_idle_vcpu(v));
diff -r 705de1dacabe -r 1a1d593c16f3 xen/include/asm-x86/i387.h
--- a/xen/include/asm-x86/i387.h	Thu Feb 10 16:25:09 2011 -0600
+++ b/xen/include/asm-x86/i387.h	Thu Feb 10 16:55:27 2011 -0600
@@ -20,6 +20,8 @@
 void xsave_init(void);
 int xsave_alloc_save_area(struct vcpu *v);
 void xsave_free_save_area(struct vcpu *v);
+void xsave_lwp(struct vcpu *v);
+void xrstor_lwp(struct vcpu *v);
 
 #define XSAVE_AREA_MIN_SIZE (512 + 64) /* FP/SSE + XSAVE.HEADER */
 #define XSTATE_FP       (1ULL << 0)
@@ -33,7 +35,7 @@
 #define XSAVEOPT        (1 << 0)
 
 /* The features that the OS saves/restores by default. */
-#define XCNTXT_DEFAULT     (-1)
+#define XCNTXT_DEFAULT     (~XSTATE_LWP)
 
 struct xsave_struct
 {

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH 4/5][RFC] lwp: adding support for AMD lightweight profiling

[Jan Beulich]

>>> On 11.02.11 at 17:29, Wei Huang <wei.huang2@amd.com> wrote:
>--- a/xen/arch/x86/i387.c	Thu Feb 10 16:25:09 2011 -0600
>+++ b/xen/arch/x86/i387.c	Thu Feb 10 16:55:27 2011 -0600
>@@ -65,6 +65,55 @@
> static void init_fpu(void);
> static void restore_fpu(struct vcpu *v);
> 
>+/* Save AMD LWP */
>+void xsave_lwp(struct vcpu *v)
>+{
>+    uint64_t lwpcb;
>+    bool_t ts;
>+    struct xsave_struct *xsave_area = v->arch.xsave_area;
>+
>+    if ( cpu_has_lwp )
>+    {
>+        /* Has LWP been used? */
>+        rdmsrl(MSR_AMD_LWP_CBADDR, lwpcb);

There's no way to track LWP-using state for a vCPU, is there?
rdmsr seems pretty expensive for being used in the context
switch unconditionally (on CPUs supporting LWP)...

>+        if ( !lwpcb ) {
>+            /* Guest might have turned off LWP. So clean the bit here. */
>+            xsave_area->xsave_hdr.xstate_bv &= ~XSTATE_LWP;
>+            return;
>+        }
>+        
>+        /* Disable TS temporarily to avoid recursion. */
>+        ts = read_cr0() & X86_CR0_TS;
>+        clts();
>+        xsave(v, XSTATE_LWP);
>+        if ( ts )
>+            stts();

Together with the xrstor_lwp() ones, quite a few manipulations of
CR0, and hence making context switch between two LWP-using
vcpus pretty expensive. I'm sure some of this redundancy can be
eliminated.

Jan

RE: [PATCH 4/5][RFC] lwp: adding support for AMD lightweight profiling

[Huang2, Wei]

Hi Jan,

No there is no quick way (like TS bit) to keep track LWP state. Here is an excerpt from lwp spec:

"LWP does not support the "lazy" state save and restore that is possible for floating point and SSE state. It does not interact with the CR0.TS bit. Operating systems that support LWP must always do an XSAVE to preserve the old thread's LWP context and an XRSTOR to set up the new LWP context. The OS can continue to do a lazy switch of the FP and SSE state by ensuring that the corresponding bits in EDX:EAX are clear when it executes the XSAVE and XRSTOR to handle the LWP context."

Thanks,
-Wei

-----Original Message-----
From: Jan Beulich [mailto:JBeulich@novell.com] 
Sent: Monday, February 14, 2011 3:12 AM
To: Huang2, Wei
Cc: Gang Wei; xen-devel@lists.xensource.com; KeirFraser
Subject: Re: [Xen-devel] [PATCH 4/5][RFC] lwp: adding support for AMD lightweight profiling

>>> On 11.02.11 at 17:29, Wei Huang <wei.huang2@amd.com> wrote:
>--- a/xen/arch/x86/i387.c	Thu Feb 10 16:25:09 2011 -0600
>+++ b/xen/arch/x86/i387.c	Thu Feb 10 16:55:27 2011 -0600
>@@ -65,6 +65,55 @@
> static void init_fpu(void);
> static void restore_fpu(struct vcpu *v);
> 
>+/* Save AMD LWP */
>+void xsave_lwp(struct vcpu *v)
>+{
>+    uint64_t lwpcb;
>+    bool_t ts;
>+    struct xsave_struct *xsave_area = v->arch.xsave_area;
>+
>+    if ( cpu_has_lwp )
>+    {
>+        /* Has LWP been used? */
>+        rdmsrl(MSR_AMD_LWP_CBADDR, lwpcb);

There's no way to track LWP-using state for a vCPU, is there?
rdmsr seems pretty expensive for being used in the context
switch unconditionally (on CPUs supporting LWP)...

>+        if ( !lwpcb ) {
>+            /* Guest might have turned off LWP. So clean the bit here. */
>+            xsave_area->xsave_hdr.xstate_bv &= ~XSTATE_LWP;
>+            return;
>+        }
>+        
>+        /* Disable TS temporarily to avoid recursion. */
>+        ts = read_cr0() & X86_CR0_TS;
>+        clts();
>+        xsave(v, XSTATE_LWP);
>+        if ( ts )
>+            stts();

Together with the xrstor_lwp() ones, quite a few manipulations of
CR0, and hence making context switch between two LWP-using
vcpus pretty expensive. I'm sure some of this redundancy can be
eliminated.

Jan

RE: [PATCH 4/5][RFC] lwp: adding support for AMD lightweight profiling

[Jan Beulich]

>>> On 14.02.11 at 16:55, "Huang2, Wei" <Wei.Huang2@amd.com> wrote:
> No there is no quick way (like TS bit) to keep track LWP state. Here is an 
> excerpt from lwp spec:
> 
> "LWP does not support the "lazy" state save and restore that is possible for 
> floating point and SSE state. It does not interact with the CR0.TS bit. 
> Operating systems that support LWP must always do an XSAVE to preserve the 
> old thread's LWP context and an XRSTOR to set up the new LWP context. The OS 
> can continue to do a lazy switch of the FP and SSE state by ensuring that the 
> corresponding bits in EDX:EAX are clear when it executes the XSAVE and XRSTOR 
> to handle the LWP context."

I certainly wasn't thinking of CR0.TS, but after reading patch 5 my
question should have been re-phrased into whether the intercept
of the control MSR can't be used for tracking purposes.

Jan

RE: [PATCH 4/5][RFC] lwp: adding support for AMD lightweight profiling

[Huang2, Wei]

This might be possible. The frequency of MSR read/write inside guest VM will be less compared with vcpu context switch. I will try it in next spin.

-Wei

-----Original Message-----
From: Jan Beulich [mailto:JBeulich@novell.com] 
Sent: Monday, February 14, 2011 10:18 AM
To: Huang2, Wei
Cc: Gang Wei; xen-devel@lists.xensource.com; KeirFraser
Subject: RE: [Xen-devel] [PATCH 4/5][RFC] lwp: adding support for AMD lightweight profiling

>>> On 14.02.11 at 16:55, "Huang2, Wei" <Wei.Huang2@amd.com> wrote:
> No there is no quick way (like TS bit) to keep track LWP state. Here is an 
> excerpt from lwp spec:
> 
> "LWP does not support the "lazy" state save and restore that is possible for 
> floating point and SSE state. It does not interact with the CR0.TS bit. 
> Operating systems that support LWP must always do an XSAVE to preserve the 
> old thread's LWP context and an XRSTOR to set up the new LWP context. The OS 
> can continue to do a lazy switch of the FP and SSE state by ensuring that the 
> corresponding bits in EDX:EAX are clear when it executes the XSAVE and XRSTOR 
> to handle the LWP context."

I certainly wasn't thinking of CR0.TS, but after reading patch 5 my
question should have been re-phrased into whether the intercept
of the control MSR can't be used for tracking purposes.

Jan