[PATCH RFC 0/4] Use INVPCID to flush global mappings

[PATCH RFC 0/4] Use INVPCID to flush global mappings

[Wei Liu]

I wrote these patches sometime ago to explore PCID and INVPCID. I haven't
thought through whether how to use both in Xen yet. But seeing Juergen laid out
his thought on PCID and INVPCID I think some of the patches can be useful.

I had done some benchmark on the speed in one of my older branch by inserting
some trace points before and after the flush. It showed that twiddling CR4.PGE
is 3 to 5 times slower than invpcid.

This series is in incomplete -- obviously we have CR4.PGE twiddling in a few
other places. But if you think it is beneficial I can try to convert those
places as well.

Wei.

Wei Liu (4):
  x86: introduce cpu_has_invpcid
  x86: report if PCID and INVPCID are supported
  x86: add invpcid.h
  x86: use invpcid to do global flush

 xen/arch/x86/flushtlb.c          | 22 ++++++++++++---
 xen/arch/x86/setup.c             |  7 +++++
 xen/include/asm-x86/cpufeature.h |  1 +
 xen/include/asm-x86/invpcid.h    | 61 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 87 insertions(+), 4 deletions(-)
 create mode 100644 xen/include/asm-x86/invpcid.h

-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH RFC 1/4] x86: introduce cpu_has_invpcid

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 xen/include/asm-x86/cpufeature.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/xen/include/asm-x86/cpufeature.h b/xen/include/asm-x86/cpufeature.h
index 55b696ed07..db8072279d 100644
--- a/xen/include/asm-x86/cpufeature.h
+++ b/xen/include/asm-x86/cpufeature.h
@@ -93,6 +93,7 @@
 #define cpu_has_avx2            boot_cpu_has(X86_FEATURE_AVX2)
 #define cpu_has_smep            boot_cpu_has(X86_FEATURE_SMEP)
 #define cpu_has_bmi2            boot_cpu_has(X86_FEATURE_BMI2)
+#define cpu_has_invpcid         boot_cpu_has(X86_FEATURE_INVPCID)
 #define cpu_has_rtm             boot_cpu_has(X86_FEATURE_RTM)
 #define cpu_has_fpu_sel         (!boot_cpu_has(X86_FEATURE_NO_FPU_SEL))
 #define cpu_has_mpx             boot_cpu_has(X86_FEATURE_MPX)
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH RFC 2/4] x86: report if PCID and INVPCID are supported

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 xen/arch/x86/setup.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index ac530ece2c..89e42865a4 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -1701,6 +1701,13 @@ void __init noreturn __start_xen(unsigned long mbi_p)
            cpu_has_nx ? XENLOG_INFO : XENLOG_WARNING "Warning: ",
            cpu_has_nx ? "" : "not ");
 
+
+    printk(XENLOG_INFO
+           "PCID (Process-Context IDentifier) %ssupported\n",
+           cpu_has_pcid ? "" : "not ");
+
+    printk(XENLOG_INFO "INVPCID %ssupported\n", cpu_has_invpcid ? "" : "not ");
+
     /*
      * We're going to setup domain0 using the module(s) that we stashed safely
      * above our heap. The second module, if present, is an initrd ramdisk.
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH RFC 3/4] x86: add invpcid.h

[Wei Liu]

Provide the functions needed for different modes.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 xen/include/asm-x86/invpcid.h | 61 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 61 insertions(+)
 create mode 100644 xen/include/asm-x86/invpcid.h

diff --git a/xen/include/asm-x86/invpcid.h b/xen/include/asm-x86/invpcid.h
new file mode 100644
index 0000000000..7c307ecfc3
--- /dev/null
+++ b/xen/include/asm-x86/invpcid.h
@@ -0,0 +1,61 @@
+#ifndef _ASM_X86_INVPCID_H_
+#define _ASM_X86_INVPCID_H_
+
+#include <xen/types.h>
+
+#define INVPCID_TYPE_INDIV_ADDR      0
+#define INVPCID_TYPE_SINGLE_CTXT     1
+#define INVPCID_TYPE_ALL_INCL_GLOBAL 2
+#define INVPCID_TYPE_ALL_NON_GLOBAL  3
+
+struct invpcid_desc {
+    uint64_t pcid:12;
+    uint64_t reserved:52;
+    uint64_t addr;
+};
+
+static inline void invpcid(unsigned long pcid, unsigned long addr,
+                           unsigned long type)
+{
+    struct invpcid_desc desc = { .pcid = pcid, .addr = addr };
+
+    asm volatile ("invpcid (%0), %1"
+                  : : "r" (&desc), "r" (type) : "memory" );
+}
+
+/* Flush all mappings for a given PCID and addr, not including globals */
+static inline void invpcid_flush_one(unsigned long pcid,
+                                     unsigned long addr)
+{
+    invpcid(pcid, addr, INVPCID_TYPE_INDIV_ADDR);
+}
+
+/* Flush all mappings for a given PCID, not including globals */
+static inline void invpcid_flush_single_context(unsigned long pcid)
+{
+    invpcid(pcid, 0, INVPCID_TYPE_SINGLE_CTXT);
+}
+
+/* Flush all mappings, including globals, for all PCIDs */
+static inline void invpcid_flush_all(void)
+{
+    invpcid(0, 0, INVPCID_TYPE_ALL_INCL_GLOBAL);
+}
+
+/* Flush all mappings for all PCIDs, excluding globals */
+static inline void invpcid_flush_all_nonglobals(void)
+{
+    invpcid(0, 0, INVPCID_TYPE_ALL_NON_GLOBAL);
+}
+
+#endif	/* _ASM_X86_INVPCID_H_ */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH RFC 4/4] x86: use invpcid to do global flush

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 xen/arch/x86/flushtlb.c | 22 ++++++++++++++++++----
 1 file changed, 18 insertions(+), 4 deletions(-)

diff --git a/xen/arch/x86/flushtlb.c b/xen/arch/x86/flushtlb.c
index 8a7a76b8ff..e4ea4f3297 100644
--- a/xen/arch/x86/flushtlb.c
+++ b/xen/arch/x86/flushtlb.c
@@ -9,6 +9,7 @@
 
 #include <xen/sched.h>
 #include <xen/softirq.h>
+#include <asm/invpcid.h>
 #include <asm/flushtlb.h>
 #include <asm/page.h>
 
@@ -120,11 +121,24 @@ unsigned int flush_area_local(const void *va, unsigned int flags)
         else
         {
             u32 t = pre_flush();
-            unsigned long cr4 = read_cr4();
 
-            write_cr4(cr4 & ~X86_CR4_PGE);
-            barrier();
-            write_cr4(cr4);
+            if ( !cpu_has_invpcid )
+            {
+                unsigned long cr4 = read_cr4();
+
+                write_cr4(cr4 & ~X86_CR4_PGE);
+                barrier();
+                write_cr4(cr4);
+            }
+            else
+            {
+                /*
+                 * Using invpcid to flush all mappings works
+                 * regardless of whether PCID is enabled or not.
+                 * It is faster than read-modify-write CR4.
+                 */
+                invpcid_flush_all();
+            }
 
             post_flush(t);
         }
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH RFC 3/4] x86: add invpcid.h

[Andrew Cooper]

On 15/02/18 12:10, Wei Liu wrote:
> Provide the functions needed for different modes.
>
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> ---
>  xen/include/asm-x86/invpcid.h | 61 +++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 61 insertions(+)
>  create mode 100644 xen/include/asm-x86/invpcid.h
>
> diff --git a/xen/include/asm-x86/invpcid.h b/xen/include/asm-x86/invpcid.h
> new file mode 100644
> index 0000000000..7c307ecfc3
> --- /dev/null
> +++ b/xen/include/asm-x86/invpcid.h
> @@ -0,0 +1,61 @@
> +#ifndef _ASM_X86_INVPCID_H_
> +#define _ASM_X86_INVPCID_H_
> +
> +#include <xen/types.h>
> +
> +#define INVPCID_TYPE_INDIV_ADDR      0
> +#define INVPCID_TYPE_SINGLE_CTXT     1
> +#define INVPCID_TYPE_ALL_INCL_GLOBAL 2
> +#define INVPCID_TYPE_ALL_NON_GLOBAL  3
> +
> +struct invpcid_desc {
> +    uint64_t pcid:12;
> +    uint64_t reserved:52;
> +    uint64_t addr;
> +};
> +
> +static inline void invpcid(unsigned long pcid, unsigned long addr,
> +                           unsigned long type)
> +{
> +    struct invpcid_desc desc = { .pcid = pcid, .addr = addr };
> +
> +    asm volatile ("invpcid (%0), %1"
> +                  : : "r" (&desc), "r" (type) : "memory" );

invpcid %[desc], %[type]

And you can use [desc] "m" (desc) for the constraint.  The structure
will be built on the stack, meaning that an %rsp based memory reference
is more efficient than forcing the use of a register.

We probably also need a -DHAVE_GAS_INVPCID, as INVPCID is newer than
some of the instruction groups we already check for.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH RFC 3/4] x86: add invpcid.h

[Wei Liu]

On Thu, Feb 15, 2018 at 12:15:56PM +0000, Andrew Cooper wrote:
> On 15/02/18 12:10, Wei Liu wrote:
> > Provide the functions needed for different modes.
> >
> > Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> > ---
> >  xen/include/asm-x86/invpcid.h | 61 +++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 61 insertions(+)
> >  create mode 100644 xen/include/asm-x86/invpcid.h
> >
> > diff --git a/xen/include/asm-x86/invpcid.h b/xen/include/asm-x86/invpcid.h
> > new file mode 100644
> > index 0000000000..7c307ecfc3
> > --- /dev/null
> > +++ b/xen/include/asm-x86/invpcid.h
> > @@ -0,0 +1,61 @@
> > +#ifndef _ASM_X86_INVPCID_H_
> > +#define _ASM_X86_INVPCID_H_
> > +
> > +#include <xen/types.h>
> > +
> > +#define INVPCID_TYPE_INDIV_ADDR      0
> > +#define INVPCID_TYPE_SINGLE_CTXT     1
> > +#define INVPCID_TYPE_ALL_INCL_GLOBAL 2
> > +#define INVPCID_TYPE_ALL_NON_GLOBAL  3
> > +
> > +struct invpcid_desc {
> > +    uint64_t pcid:12;
> > +    uint64_t reserved:52;
> > +    uint64_t addr;
> > +};
> > +
> > +static inline void invpcid(unsigned long pcid, unsigned long addr,
> > +                           unsigned long type)
> > +{
> > +    struct invpcid_desc desc = { .pcid = pcid, .addr = addr };
> > +
> > +    asm volatile ("invpcid (%0), %1"
> > +                  : : "r" (&desc), "r" (type) : "memory" );
> 
> invpcid %[desc], %[type]
> 
> And you can use [desc] "m" (desc) for the constraint.  The structure
> will be built on the stack, meaning that an %rsp based memory reference
> is more efficient than forcing the use of a register.

NP.

> 
> We probably also need a -DHAVE_GAS_INVPCID, as INVPCID is newer than
> some of the instruction groups we already check for.
> 

Or we can just use the byte code directly -- it is the same for both 64
and 32 bit, then manually specify the ModRM byte. That's what Linux
does.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH RFC 3/4] x86: add invpcid.h

[Andrew Cooper]

On 15/02/18 12:24, Wei Liu wrote:
> On Thu, Feb 15, 2018 at 12:15:56PM +0000, Andrew Cooper wrote:
>> On 15/02/18 12:10, Wei Liu wrote:
>>> Provide the functions needed for different modes.
>>>
>>> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
>>> ---
>>>  xen/include/asm-x86/invpcid.h | 61 +++++++++++++++++++++++++++++++++++++++++++
>>>  1 file changed, 61 insertions(+)
>>>  create mode 100644 xen/include/asm-x86/invpcid.h
>>>
>>> diff --git a/xen/include/asm-x86/invpcid.h b/xen/include/asm-x86/invpcid.h
>>> new file mode 100644
>>> index 0000000000..7c307ecfc3
>>> --- /dev/null
>>> +++ b/xen/include/asm-x86/invpcid.h
>>> @@ -0,0 +1,61 @@
>>> +#ifndef _ASM_X86_INVPCID_H_
>>> +#define _ASM_X86_INVPCID_H_
>>> +
>>> +#include <xen/types.h>
>>> +
>>> +#define INVPCID_TYPE_INDIV_ADDR      0
>>> +#define INVPCID_TYPE_SINGLE_CTXT     1
>>> +#define INVPCID_TYPE_ALL_INCL_GLOBAL 2
>>> +#define INVPCID_TYPE_ALL_NON_GLOBAL  3
>>> +
>>> +struct invpcid_desc {
>>> +    uint64_t pcid:12;
>>> +    uint64_t reserved:52;
>>> +    uint64_t addr;
>>> +};
>>> +
>>> +static inline void invpcid(unsigned long pcid, unsigned long addr,
>>> +                           unsigned long type)
>>> +{
>>> +    struct invpcid_desc desc = { .pcid = pcid, .addr = addr };
>>> +
>>> +    asm volatile ("invpcid (%0), %1"
>>> +                  : : "r" (&desc), "r" (type) : "memory" );
>> invpcid %[desc], %[type]
>>
>> And you can use [desc] "m" (desc) for the constraint.  The structure
>> will be built on the stack, meaning that an %rsp based memory reference
>> is more efficient than forcing the use of a register.
> NP.
>
>> We probably also need a -DHAVE_GAS_INVPCID, as INVPCID is newer than
>> some of the instruction groups we already check for.
>>
> Or we can just use the byte code directly -- it is the same for both 64
> and 32 bit, then manually specify the ModRM byte. That's what Linux
> does.

See the vmx __invept() wrapper, which is very similar, but doesn't need
the BUG() handling.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH RFC 1/4] x86: introduce cpu_has_invpcid

[Jan Beulich]

>>> On 15.02.18 at 13:10, <wei.liu2@citrix.com> wrote:
> --- a/xen/include/asm-x86/cpufeature.h
> +++ b/xen/include/asm-x86/cpufeature.h
> @@ -93,6 +93,7 @@
>  #define cpu_has_avx2            boot_cpu_has(X86_FEATURE_AVX2)
>  #define cpu_has_smep            boot_cpu_has(X86_FEATURE_SMEP)
>  #define cpu_has_bmi2            boot_cpu_has(X86_FEATURE_BMI2)
> +#define cpu_has_invpcid         boot_cpu_has(X86_FEATURE_INVPCID)
>  #define cpu_has_rtm             boot_cpu_has(X86_FEATURE_RTM)
>  #define cpu_has_fpu_sel         (!boot_cpu_has(X86_FEATURE_NO_FPU_SEL))
>  #define cpu_has_mpx             boot_cpu_has(X86_FEATURE_MPX)

Please fold into whatever patch is first using the new macro.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH RFC 3/4] x86: add invpcid.h

[Jan Beulich]

>>> On 15.02.18 at 13:10, <wei.liu2@citrix.com> wrote:
> Provide the functions needed for different modes.

Do we really need all of these? Let's not have dead code sit around
and risk it becoming stale.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH RFC 3/4] x86: add invpcid.h

[Wei Liu]

On Thu, Feb 15, 2018 at 05:34:26AM -0700, Jan Beulich wrote:
> >>> On 15.02.18 at 13:10, <wei.liu2@citrix.com> wrote:
> > Provide the functions needed for different modes.
> 
> Do we really need all of these? Let's not have dead code sit around
> and risk it becoming stale.
> 

They will be needed when we want to use PCID and INVPCID at the same
time, which seems to be the plan?

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH RFC 3/4] x86: add invpcid.h

[Jan Beulich]

>>> On 15.02.18 at 13:35, <wei.liu2@citrix.com> wrote:
> On Thu, Feb 15, 2018 at 05:34:26AM -0700, Jan Beulich wrote:
>> >>> On 15.02.18 at 13:10, <wei.liu2@citrix.com> wrote:
>> > Provide the functions needed for different modes.
>> 
>> Do we really need all of these? Let's not have dead code sit around
>> and risk it becoming stale.
>> 
> 
> They will be needed when we want to use PCID and INVPCID at the same
> time, which seems to be the plan?

I know of INVPCID plans only, not any PCID ones.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH RFC 0/4] Use INVPCID to flush global mappings

[Juergen Gross]

On 15/02/18 13:10, Wei Liu wrote:
> I wrote these patches sometime ago to explore PCID and INVPCID. I haven't
> thought through whether how to use both in Xen yet. But seeing Juergen laid out
> his thought on PCID and INVPCID I think some of the patches can be useful.
> 
> I had done some benchmark on the speed in one of my older branch by inserting
> some trace points before and after the flush. It showed that twiddling CR4.PGE
> is 3 to 5 times slower than invpcid.
> 
> This series is in incomplete -- obviously we have CR4.PGE twiddling in a few
> other places. But if you think it is beneficial I can try to convert those
> places as well.

I just did a little experiment by replacing the %cr4 based TLB flush in
Jan's XPTI patches by invpcid, on top of my last XPTI speedup patch (no
fancy ALTERNATIVE, just a plain replacement).

Doing a parallel build of the hypervisor in dom0 showed following data:

                real      user      sys
xpti=no         61.2     167.7     71.9
xpti=yes       112.1     170.1    141.8
 + my speedup  103.0     171.2    131.2
  + invpcid     99.0     170.2    122.0

So system time reduction due to invpcid is quite nice.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel