[PATCH 7/8] x86/cpu/intel: enable X86_FEATURE_NT_GOOD on Intel Broadwellx

[Ankur Arora <ankur.a.arora@oracle.com>]

System:           Oracle X6-2
CPU:              2 nodes * 10 cores/node * 2 threads/core
		  Intel Xeon E5-2630 v4 (Broadwellx, 6:79:1)
Memory:           256 GB evenly split between nodes
Microcode:        0xb00002e
scaling_governor: performance
L3 size:          25MB
intel_pstate/no_turbo: 1

Performance comparison of 'perf bench mem memset -l 1' for x86-64-stosb
(X86_FEATURE_ERMS) and x86-64-movnt (X86_FEATURE_NT_GOOD):

              x86-64-stosb (5 runs)     x86-64-movnt (5 runs)       speedup
              -----------------------   -----------------------     -------
     size       BW        (   pstdev)          BW   (   pstdev)

     16MB      17.35 GB/s ( +- 9.27%)    11.83 GB/s ( +- 0.19%)     -31.81%
    128MB       5.31 GB/s ( +- 0.13%)    11.72 GB/s ( +- 0.44%)    +121.84%
   1024MB       5.42 GB/s ( +- 0.13%)    11.78 GB/s ( +- 0.03%)    +117.34%
   4096MB       5.41 GB/s ( +- 0.41%)    11.76 GB/s ( +- 0.07%)    +117.37%

The next workload exercises the page-clearing path directly by faulting over
an anonymous mmap region backed by 1GB pages. This workload is similar to the
creation phase of pinned guests in QEMU.

$ cat pf-test.c
 #include <stdlib.h>
 #include <sys/mman.h>
 #include <linux/mman.h>

 #define HPAGE_BITS 30
 int main(int argc, char **argv) {
	int i;
	unsigned long len = atoi(argv[1]); /* In GB */
	unsigned long offset = 0;
	unsigned long numpages;
	char *base;

	len *= 1UL << 30;
	numpages = len >> HPAGE_BITS;

	base = mmap(NULL, len, PROT_READ|PROT_WRITE,
	            MAP_PRIVATE | MAP_ANONYMOUS |
		    MAP_HUGETLB | MAP_HUGE_1GB, 0, 0);

	for (i = 0; i < numpages; i++) {
	        *((volatile char *)base + offset) = *(base + offset);
	        offset += 1UL << HPAGE_BITS;
	}

	return 0;
 }

The specific test is for a 128GB region but this is a single-threaded
O(n) workload so the exact region size is not material.

Page-clearing throughput for clear_page_erms(): 3.72 GBps
$ perf stat -r 5 --all-kernel -e ... bin/pf-test 128

 Performance counter stats for 'bin/pf-test 128' (5 runs):

    74,799,496,556      cpu-cycles                #    2.176 GHz                      ( +-  2.22% )  (29.41%)
     1,474,615,023      instructions              #    0.02  insn per cycle           ( +-  0.23% )  (35.29%)
     2,148,580,131      cache-references          #   62.502 M/sec                    ( +-  0.02% )  (35.29%)
        71,736,985      cache-misses              #    3.339 % of all cache refs      ( +-  0.94% )  (35.29%)
       433,713,165      branch-instructions       #   12.617 M/sec                    ( +-  0.15% )  (35.30%)
         1,008,251      branch-misses             #    0.23% of all branches          ( +-  1.88% )  (35.30%)
     3,406,821,966      bus-cycles                #   99.104 M/sec                    ( +-  2.22% )  (23.53%)
     2,156,059,110      L1-dcache-load-misses     #  445.35% of all L1-dcache accesses  ( +-  0.01% )  (23.53%)
       484,128,243      L1-dcache-loads           #   14.083 M/sec                    ( +-  0.22% )  (23.53%)
           944,216      LLC-loads                 #    0.027 M/sec                    ( +-  7.41% )  (23.53%)
           537,989      LLC-load-misses           #   56.98% of all LL-cache accesses  ( +- 13.64% )  (23.53%)
     2,150,138,476      LLC-stores                #   62.547 M/sec                    ( +-  0.01% )  (11.76%)
        69,598,760      LLC-store-misses          #    2.025 M/sec                    ( +-  0.47% )  (11.76%)
       483,923,875      dTLB-loads                #   14.077 M/sec                    ( +-  0.21% )  (17.64%)
             1,892      dTLB-load-misses          #    0.00% of all dTLB cache accesses  ( +- 30.63% )  (23.53%)
     4,799,154,980      dTLB-stores               #  139.606 M/sec                    ( +-  0.03% )  (23.53%)
                90      dTLB-store-misses         #    0.003 K/sec                    ( +- 35.92% )  (23.53%)

            34.377 +- 0.760 seconds time elapsed  ( +-  2.21% )

Page-clearing throughput with clear_page_nt(): 11.78GBps
$ perf stat -r 5 --all-kernel -e ... bin/pf-test 128

 Performance counter stats for 'bin/pf-test 128' (5 runs):

    23,699,446,603      cpu-cycles                #    2.182 GHz                      ( +-  0.01% )  (23.53%)
    24,794,548,512      instructions              #    1.05  insn per cycle           ( +-  0.00% )  (29.41%)
           432,775      cache-references          #    0.040 M/sec                    ( +-  3.96% )  (29.41%)
            75,580      cache-misses              #   17.464 % of all cache refs      ( +- 51.42% )  (29.41%)
     2,492,858,290      branch-instructions       #  229.475 M/sec                    ( +-  0.00% )  (29.42%)
        34,016,826      branch-misses             #    1.36% of all branches          ( +-  0.04% )  (29.42%)
     1,078,468,643      bus-cycles                #   99.276 M/sec                    ( +-  0.01% )  (23.53%)
           717,228      L1-dcache-load-misses     #    0.20% of all L1-dcache accesses  ( +-  3.77% )  (23.53%)
       351,999,535      L1-dcache-loads           #   32.403 M/sec                    ( +-  0.04% )  (23.53%)
            75,988      LLC-loads                 #    0.007 M/sec                    ( +-  4.20% )  (23.53%)
            24,503      LLC-load-misses           #   32.25% of all LL-cache accesses  ( +- 53.30% )  (23.53%)
            57,283      LLC-stores                #    0.005 M/sec                    ( +-  2.15% )  (11.76%)
            19,738      LLC-store-misses          #    0.002 M/sec                    ( +- 46.55% )  (11.76%)
       351,836,498      dTLB-loads                #   32.388 M/sec                    ( +-  0.04% )  (17.65%)
             1,171      dTLB-load-misses          #    0.00% of all dTLB cache accesses  ( +- 42.68% )  (23.53%)
    17,385,579,725      dTLB-stores               # 1600.392 M/sec                    ( +-  0.00% )  (23.53%)
               200      dTLB-store-misses         #    0.018 K/sec                    ( +- 10.63% )  (23.53%)

         10.863678 +- 0.000804 seconds time elapsed  ( +-  0.01% )

L1-dcache-load-misses (L1D.REPLACEMENT) is substantially lower which
suggests that, as expected, we aren't doing write-allocate or RFO.

Note that the IPC and instruction counts etc are quite different, but
that's just an artifact of switching from a single 'REP; STOSB' per
PAGE_SIZE region to a MOVNTI loop.

The page-clearing BW is substantially higher (~100% or more), so enable
X86_FEATURE_NT_GOOD for Intel Broadwellx.

Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 arch/x86/kernel/cpu/intel.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/x86/kernel/cpu/intel.c b/arch/x86/kernel/cpu/intel.c
index 59a1e3ce3f14..161028c1dee0 100644
--- a/arch/x86/kernel/cpu/intel.c
+++ b/arch/x86/kernel/cpu/intel.c
@@ -662,6 +662,8 @@ static void init_intel(struct cpuinfo_x86 *c)
 		c->x86_cache_alignment = c->x86_clflush_size * 2;
 	if (c->x86 == 6)
 		set_cpu_cap(c, X86_FEATURE_REP_GOOD);
+	if (c->x86 == 6 && c->x86_model == INTEL_FAM6_BROADWELL_X)
+		set_cpu_cap(c, X86_FEATURE_NT_GOOD);
 #else
 	/*
 	 * Names for the Pentium II/Celeron processors
-- 
2.9.3