[PATCH v4 1/2] powerpc/vcpu: Assume dedicated processors as non-preempt

[Srikar Dronamraju <srikar@linux.vnet.ibm.com>]

With commit 247f2f6f3c70 ("sched/core: Don't schedule threads on pre-empted
vCPUs"), scheduler avoids preempted vCPUs to schedule tasks on wakeup.
This leads to wrong choice of CPU, which in-turn leads to larger wakeup
latencies. Eventually, it leads to performance regression in latency
sensitive benchmarks like soltp, schbench etc.

On Powerpc, vcpu_is_preempted only looks at yield_count. If the
yield_count is odd, the vCPU is assumed to be preempted. However
yield_count is increased whenever LPAR enters CEDE state. So any CPU
that has entered CEDE state is assumed to be preempted.

Even if vCPU of dedicated LPAR is preempted/donated, it should have
right of first-use since they are suppose to own the vCPU.

On a Power9 System with 32 cores
 # lscpu
Architecture:        ppc64le
Byte Order:          Little Endian
CPU(s):              128
On-line CPU(s) list: 0-127
Thread(s) per core:  8
Core(s) per socket:  1
Socket(s):           16
NUMA node(s):        2
Model:               2.2 (pvr 004e 0202)
Model name:          POWER9 (architected), altivec supported
Hypervisor vendor:   pHyp
Virtualization type: para
L1d cache:           32K
L1i cache:           32K
L2 cache:            512K
L3 cache:            10240K
NUMA node0 CPU(s):   0-63
NUMA node1 CPU(s):   64-127

  # perf stat -a -r 5 ./schbench
v5.4                                     v5.4 + patch
Latency percentiles (usec)               Latency percentiles (usec)
	50.0000th: 45                    	50.0000th: 39
	75.0000th: 62                    	75.0000th: 53
	90.0000th: 71                    	90.0000th: 67
	95.0000th: 77                    	95.0000th: 76
	*99.0000th: 91                   	*99.0000th: 89
	99.5000th: 707                   	99.5000th: 93
	99.9000th: 6920                  	99.9000th: 118
	min=0, max=10048                 	min=0, max=211
Latency percentiles (usec)               Latency percentiles (usec)
	50.0000th: 45                    	50.0000th: 34
	75.0000th: 61                    	75.0000th: 45
	90.0000th: 72                    	90.0000th: 53
	95.0000th: 79                    	95.0000th: 56
	*99.0000th: 691                  	*99.0000th: 61
	99.5000th: 3972                  	99.5000th: 63
	99.9000th: 8368                  	99.9000th: 78
	min=0, max=16606                 	min=0, max=228
Latency percentiles (usec)               Latency percentiles (usec)
	50.0000th: 45                    	50.0000th: 34
	75.0000th: 61                    	75.0000th: 45
	90.0000th: 71                    	90.0000th: 53
	95.0000th: 77                    	95.0000th: 57
	*99.0000th: 106                  	*99.0000th: 63
	99.5000th: 2364                  	99.5000th: 68
	99.9000th: 7480                  	99.9000th: 100
	min=0, max=10001                 	min=0, max=134
Latency percentiles (usec)               Latency percentiles (usec)
	50.0000th: 45                    	50.0000th: 34
	75.0000th: 62                    	75.0000th: 46
	90.0000th: 72                    	90.0000th: 53
	95.0000th: 78                    	95.0000th: 56
	*99.0000th: 93                   	*99.0000th: 61
	99.5000th: 108                   	99.5000th: 64
	99.9000th: 6792                  	99.9000th: 85
	min=0, max=17681                 	min=0, max=121
Latency percentiles (usec)               Latency percentiles (usec)
	50.0000th: 46                    	50.0000th: 33
	75.0000th: 62                    	75.0000th: 44
	90.0000th: 73                    	90.0000th: 51
	95.0000th: 79                    	95.0000th: 54
	*99.0000th: 113                  	*99.0000th: 61
	99.5000th: 2724                  	99.5000th: 64
	99.9000th: 6184                  	99.9000th: 82
	min=0, max=9887                  	min=0, max=121

 Performance counter stats for 'system wide' (5 runs):

context-switches    43,373  ( +-  0.40% )   44,597 ( +-  0.55% )
cpu-migrations       1,211  ( +-  5.04% )      220 ( +-  6.23% )
page-faults         15,983  ( +-  5.21% )   15,360 ( +-  3.38% )

Waiman Long suggested using static_keys.

Fixes: 41946c86876e ("locking/core, powerpc: Implement vcpu_is_preempted(cpu)")

Cc: Parth Shah <parth@linux.ibm.com>
Cc: Ihor Pasichnyk <Ihor.Pasichnyk@ibm.com>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Phil Auld <pauld@redhat.com>
Cc: Waiman Long <longman@redhat.com>
Cc: Gautham R. Shenoy <ego@linux.vnet.ibm.com>
Cc: Vaidyanathan Srinivasan <svaidy@linux.ibm.com>
Reported-by: Parth Shah <parth@linux.ibm.com>
Reported-by: Ihor Pasichnyk <Ihor.Pasichnyk@ibm.com>
Tested-by: Juri Lelli <juri.lelli@redhat.com>
Tested-by: Parth Shah <parth@linux.ibm.com>
Acked-by: Waiman Long <longman@redhat.com>
Acked-by: Phil Auld <pauld@redhat.com>
Reviewed-by: Gautham R. Shenoy <ego@linux.vnet.ibm.com>
Reviewed-by: Vaidyanathan Srinivasan <svaidy@linux.ibm.com>
Signed-off-by: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
---
Changelog v1 (https://patchwork.ozlabs.org/patch/1204190/) ->v3:
Code is now under CONFIG_PPC_SPLPAR as it depends on CONFIG_PPC_PSERIES.
This was suggested by Waiman Long.

Changelog v3 (https://patchwork.ozlabs.org/patch/1204526) ->v4:
Fix a build issue in CONFIG_NUMA=n reported by Michael Ellerman
by moving the relevant code from mm/numa.c to kernel/smp.c

 arch/powerpc/include/asm/spinlock.h |  6 ++++--
 arch/powerpc/kernel/smp.c           | 19 ++++++++++++++-----
 arch/powerpc/mm/numa.c              |  8 +++-----
 3 files changed, 21 insertions(+), 12 deletions(-)

diff --git a/arch/powerpc/include/asm/spinlock.h b/arch/powerpc/include/asm/spinlock.h
index e9a960e28f3c..f318bfe3525f 100644
--- a/arch/powerpc/include/asm/spinlock.h
+++ b/arch/powerpc/include/asm/spinlock.h
@@ -17,6 +17,7 @@
  */
 #include <linux/irqflags.h>
 #ifdef CONFIG_PPC64
+#include <linux/jump_label.h>
 #include <asm/paca.h>
 #include <asm/hvcall.h>
 #endif
@@ -35,11 +36,12 @@
 #define LOCK_TOKEN	1
 #endif
 
-#ifdef CONFIG_PPC_PSERIES
+#ifdef CONFIG_PPC_SPLPAR
+DECLARE_STATIC_KEY_FALSE(shared_processor);
 #define vcpu_is_preempted vcpu_is_preempted
 static inline bool vcpu_is_preempted(int cpu)
 {
-	if (!firmware_has_feature(FW_FEATURE_SPLPAR))
+	if (!static_branch_unlikely(&shared_processor))
 		return false;
 	return !!(be32_to_cpu(lppaca_of(cpu).yield_count) & 1);
 }
diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
index ea6adbf6a221..96a44157b935 100644
--- a/arch/powerpc/kernel/smp.c
+++ b/arch/powerpc/kernel/smp.c
@@ -1348,6 +1348,9 @@ static struct sched_domain_topology_level power9_topology[] = {
 	{ NULL, },
 };
 
+DEFINE_STATIC_KEY_FALSE(shared_processor);
+EXPORT_SYMBOL_GPL(shared_processor);
+
 void __init smp_cpus_done(unsigned int max_cpus)
 {
 	/*
@@ -1359,11 +1362,17 @@ void __init smp_cpus_done(unsigned int max_cpus)
 	if (smp_ops && smp_ops->bringup_done)
 		smp_ops->bringup_done();
 
-	/*
-	 * On a shared LPAR, associativity needs to be requested.
-	 * Hence, get numa topology before dumping cpu topology
-	 */
-	shared_proc_topology_init();
+#ifdef CONFIG_PPC_SPLPAR
+	if (lppaca_shared_proc(get_lppaca())) {
+		static_branch_enable(&shared_processor);
+
+		/*
+		 * On a shared LPAR, associativity needs to be requested.
+		 * Hence, get numa topology before dumping cpu topology
+		 */
+		shared_proc_topology_init();
+	}
+#endif
 	dump_numa_cpu_topology();
 
 #ifdef CONFIG_SCHED_SMT
diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index 50d68d21ddcc..c352b1dfa99e 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c
@@ -1570,11 +1570,9 @@ int prrn_is_enabled(void)
 
 void __init shared_proc_topology_init(void)
 {
-	if (lppaca_shared_proc(get_lppaca())) {
-		bitmap_fill(cpumask_bits(&cpu_associativity_changes_mask),
-			    nr_cpumask_bits);
-		numa_update_cpu_topology(false);
-	}
+	bitmap_fill(cpumask_bits(&cpu_associativity_changes_mask),
+		    nr_cpumask_bits);
+	numa_update_cpu_topology(false);
 }
 
 static int topology_read(struct seq_file *file, void *v)
-- 
2.18.1