* [PATCH v7 0/2] powerpc: Detection and scheduler optimization for POWER9 bigcore
From: Gautham R. Shenoy @ 2018-08-20 5:41 UTC (permalink / raw)
To: Srikar Dronamraju, Michael Ellerman, Benjamin Herrenschmidt,
Michael Neuling, Vaidyanathan Srinivasan, Akshay Adiga,
Shilpasri G Bhat, Oliver O'Halloran, Nicholas Piggin,
Murilo Opsfelder Araujo, Anton Blanchard
Cc: linuxppc-dev, linux-kernel, Gautham R. Shenoy
From: "Gautham R. Shenoy" <ego@linux.vnet.ibm.com>
Hi,
This is the seventh iteration of the patchset to add support for
big-core on POWER9. This patch also optimizes the task placement on
such big-core systems.
The previous versions can be found here:
v6: https://lkml.org/lkml/2018/8/9/119
v5: https://lkml.org/lkml/2018/8/6/587
v4: https://lkml.org/lkml/2018/7/24/79
v3: https://lkml.org/lkml/2018/7/6/255
v2: https://lkml.org/lkml/2018/7/3/401
v1: https://lkml.org/lkml/2018/5/11/245
Changes :
v6 --> v7:
- Addressed the review comments from Srikar in Patch 1.
- For building the SMT level sched-domain with
small_core_sibling_mask, parse the "ibm,thread-groups" property
of the CPU node only once, i.e when the CPU is made online for
the first time.
Description:
~~~~~~~~~~~~~~~~~~~~
A pair of IBM POWER9 SMT4 cores can be fused together to form a
big-core with 8 SMT threads. This can be discovered via the
"ibm,thread-groups" CPU property in the device tree which will
indicate which group of threads that share the L1 cache, translation
cache and instruction data flow. If there are multiple such group of
threads, then the core is a big-core. Furthermore, on POWER9 the thread-ids of
such a big-core is obtained by interleaving the thread-ids of the
component SMT4 cores.
Eg: Threads in the pair of component SMT4 cores of an interleaved
big-core are numbered {0,2,4,6} and {1,3,5,7} respectively.
-------------------------
| L1 Cache |
----------------------------------
|L2| | | | |
| | 0 | 2 | 4 | 6 |Small Core0
|C | | | | |
Big |a --------------------------
Core |c | | | | |
|h | 1 | 3 | 5 | 7 | Small Core1
|e | | | | |
-----------------------------
| L1 Cache |
--------------------------
On such a big-core system, when multiple tasks are scheduled to run on
the big-core, we get the best performance when the tasks are spread
across the pair of SMT4 cores.
Eg: Suppose there 4 tasks {p1, p2, p3, p4} are run on a big core, then
An Example of Optimal Task placement:
--------------------------
| | | | |
| 0 | 2 | 4 | 6 | Small Core0
| (p1)| (p2)| | |
Big Core --------------------------
| | | | |
| 1 | 3 | 5 | 7 | Small Core1
| | (p3)| | (p4) |
--------------------------
An example of Suboptimal Task placement:
--------------------------
| | | | |
| 0 | 2 | 4 | 6 | Small Core0
| (p1)| (p2)| | (p4)|
Big Core --------------------------
| | | | |
| 1 | 3 | 5 | 7 | Small Core1
| | (p3)| | |
--------------------------
In order to achieve optimal task placement, on big-core systems, we
define the SMT level sched-domain to consist of the threads belonging
to the small cores. The CACHE level sched domain will consist of all
the threads belonging to the big-core. With this, the Linux Kernel
load-balancer will ensure that the tasks are spread across all the
component small cores in the system, thereby yielding optimum
performance.
Furthermore, this solution works correctly across all SMT modes
(8,4,2), as the interleaved thread-ids ensures that when we go to
lower SMT modes (4,2) the threads are offlined in a descending order,
thereby leaving equal number of threads from the component small cores
online as illustrated below.
With Patches: (ppc64_cpu --smt=on) : SMT domain
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CPU0 attaching sched-domain(s):
domain-0: span=0,2,4,6 level=SMT
groups: 0:{ span=0 cap=294 }, 2:{ span=2 cap=294 },
4:{ span=4 cap=294 }, 6:{ span=6 cap=294 }
CPU1 attaching sched-domain(s):
domain-0: span=1,3,5,7 level=SMT
groups: 1:{ span=1 cap=294 }, 3:{ span=3 cap=294 },
5:{ span=5 cap=294 }, 7:{ span=7 cap=294 }
Optimal Task placement (SMT 8)
--------------------------
| | | | |
| 0 | 2 | 4 | 6 | Small Core0
| (p1)| (p2)| | |
Big Core --------------------------
| | | | |
| 1 | 3 | 5 | 7 | Small Core1
| | (p3)| | (p4) |
--------------------------
With Patches : (ppc64_cpu --smt=4) : SMT domain
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CPU0 attaching sched-domain(s):
domain-0: span=0,2 level=SMT
groups: 0:{ span=0 cap=589 }, 2:{ span=2 cap=589 }
CPU1 attaching sched-domain(s):
domain-0: span=1,3 level=SMT
groups: 1:{ span=1 cap=589 }, 3:{ span=3 cap=589 }
Optimal Task placement (SMT 4)
--------------------------
| | | | |
| 0 | 2 | 4 | 6 | Small Core0
| (p1)| (p2)| Off | Off |
Big Core --------------------------
| | | | |
| 1 | 3 | 5 | 7 | Small Core1
| (p4)| (p3)| Off | Off |
--------------------------
With Patches : (ppc64_cpu --smt=2) : SMT domain ceases to exist.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Optimal Task placement (SMT 2)
--------------------------
| (p2)| | | |
| 0 | 2 | 4 | 6 | Small Core0
| (p1)| Off | Off | Off |
Big Core --------------------------
| (p3)| | | |
| 1 | 3 | 5 | 7 | Small Core1
| (p4)| Off | Off | Off |
--------------------------
Thus, as an added advantage in SMT=2 mode, we will only have 3 levels
in the sched-domain topology (CACHE, DIE and NUMA).
The SMT levels, without the patches are as follows.
Without Patches: (ppc64_cpu --smt=on) : SMT domain
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CPU0 attaching sched-domain(s):
domain-0: span=0-7 level=SMT
groups: 0:{ span=0 cap=147 }, 1:{ span=1 cap=147 },
2:{ span=2 cap=147 }, 3:{ span=3 cap=147 },
4:{ span=4 cap=147 }, 5:{ span=5 cap=147 },
6:{ span=6 cap=147 }, 7:{ span=7 cap=147 }
CPU1 attaching sched-domain(s):
domain-0: span=0-7 level=SMT
groups: 1:{ span=1 cap=147 }, 2:{ span=2 cap=147 },
3:{ span=3 cap=147 }, 4:{ span=4 cap=147 },
5:{ span=5 cap=147 }, 6:{ span=6 cap=147 },
7:{ span=7 cap=147 }, 0:{ span=0 cap=147 }
Without Patches: (ppc64_cpu --smt=4) : SMT domain
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CPU0 attaching sched-domain(s):
domain-0: span=0-3 level=SMT
groups: 0:{ span=0 cap=294 }, 1:{ span=1 cap=294 },
2:{ span=2 cap=294 }, 3:{ span=3 cap=294 },
CPU1 attaching sched-domain(s):
domain-0: span=0-3 level=SMT
groups: 1:{ span=1 cap=294 }, 2:{ span=2 cap=294 },
3:{ span=3 cap=294 }, 0:{ span=0 cap=294 }
Without Patches: (ppc64_cpu --smt=2) : SMT domain
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CPU0 attaching sched-domain(s):
domain-0: span=0-1 level=SMT
groups: 0:{ span=0 cap=589 }, 1:{ span=1 cap=589 },
CPU1 attaching sched-domain(s):
domain-0: span=0-1 level=SMT
groups: 1:{ span=1 cap=589 }, 0:{ span=0 cap=589 },
This patchset contains two patches which on detecting the presence of
big-cores, defines the SMT level sched domain to correspond to the
threads of the small cores.
Patch 1: adds support to detect the presence of
big-cores and reports the small-core siblings of each CPU X
via the sysfs file "/sys/devices/system/cpu/cpuX/small_core_siblings".
Patch 2: Defines the SMT level sched domain to correspond to the
threads of the small cores.
Results:
~~~~~~~~~~~~~~~~~
1) 2 thread ebizzy
~~~~~~~~~~~~~~~~~~~~~~
Experimental results for ebizzy with 2 threads, bound to a single big-core
show a marked improvement with this patchset over the 4.18.0 vanilla
kernel.
The result of 100 such runs for 4.18-rc7 kernel and the
4.18 + big-core-smt-patches are as follows
4.18.0 vanilla
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
records/s : # samples : Histogram
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[0 - 1000000] : 0 : #
[1000000 - 2000000] : 11 : ###
[2000000 - 3000000] : 9 : ##
[3000000 - 4000000] : 9 : ##
[4000000 - 5000000] : 0 : #
[5000000 - 6000000] : 71 : ###############
4.18.0 + big-core-smt-patches
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
records/s : # samples : Histogram
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[0 - 1000000] : 0 : #
[1000000 - 2000000] : 0 : #
[2000000 - 3000000] : 16 : ####
[3000000 - 4000000] : 0 : #
[4000000 - 5000000] : 1 : #
[5000000 - 6000000] : 83 : #################
2) Hackbench (perf bench sched pipe)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
500 iterations of the hackbench run both on 4.18.0 vanilla kernel
and v4.18.0 + big-core-smt-patches. More samples in the lower
numbered buckets is better. We can observe that for nearly 60% of the
samples are in the 4-5 seconds range when hackbench is run with this
patchset as opposed to < 20% of the time when the hackbench is
run on the vanilla kernel.
Similarly, nearly 80% of the samples are within the 4-6 seconds range
when hackbench is run with the patchset when compared with 50% of the
samples in the same range when hackbench is run on the vanilla kernel.
Though as a downside, we do see ~10% of the samples in the 7-9 seconds
range with the patchset as compared to 2% of the samples in the same
range without the patchset.
4.18.0 vanilla
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4 - 5 seconds : 74 samples
5 - 6 seconds : 169 samples
6 - 7 seconds : 248 samples
7 - 8 seconds : 6 samples
8 - 9 seconds : 3 samples
4.18.0 + big-core-smt-patches
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4 - 5 seconds : 289 samples
5 - 6 seconds : 99 samples
6 - 7 seconds : 58 samples
7 - 8 seconds : 28 samples
8 - 9 seconds : 26 samples
Gautham R. Shenoy (2):
powerpc: Detect the presence of big-cores via "ibm,thread-groups"
powerpc: Use cpu_smallcore_sibling_mask at SMT level on bigcores
Documentation/ABI/testing/sysfs-devices-system-cpu | 8 ++
arch/powerpc/include/asm/cputhreads.h | 25 ++++
arch/powerpc/kernel/setup-common.c | 151 +++++++++++++++++++++
arch/powerpc/kernel/smp.c | 136 ++++++++++++++++++-
arch/powerpc/kernel/sysfs.c | 38 ++++++
5 files changed, 356 insertions(+), 2 deletions(-)
--
1.9.4
^ permalink raw reply
* [PATCH v7 1/2] powerpc: Detect the presence of big-cores via "ibm, thread-groups"
From: Gautham R. Shenoy @ 2018-08-20 5:41 UTC (permalink / raw)
To: Srikar Dronamraju, Michael Ellerman, Benjamin Herrenschmidt,
Michael Neuling, Vaidyanathan Srinivasan, Akshay Adiga,
Shilpasri G Bhat, Oliver O'Halloran, Nicholas Piggin,
Murilo Opsfelder Araujo, Anton Blanchard
Cc: linuxppc-dev, linux-kernel, Gautham R. Shenoy
In-Reply-To: <1534743704-4760-1-git-send-email-ego@linux.vnet.ibm.com>
From: "Gautham R. Shenoy" <ego@linux.vnet.ibm.com>
On IBM POWER9, the device tree exposes a property array identifed by
"ibm,thread-groups" which will indicate which groups of threads share
a particular set of resources.
As of today we only have one form of grouping identifying the group of
threads in the core that share the L1 cache, translation cache and
instruction data flow.
This patch defines the helper function to parse the contents of
"ibm,thread-groups" and a new structure to contain the parsed output.
The patch also creates the sysfs file named "small_core_siblings" that
returns the physical ids of the threads in the core that share the L1
cache, translation cache and instruction data flow.
Signed-off-by: Gautham R. Shenoy <ego@linux.vnet.ibm.com>
---
Documentation/ABI/testing/sysfs-devices-system-cpu | 8 ++
arch/powerpc/include/asm/cputhreads.h | 25 ++++
arch/powerpc/kernel/setup-common.c | 151 +++++++++++++++++++++
arch/powerpc/kernel/sysfs.c | 38 ++++++
4 files changed, 222 insertions(+)
diff --git a/Documentation/ABI/testing/sysfs-devices-system-cpu b/Documentation/ABI/testing/sysfs-devices-system-cpu
index 9c5e7732..b09b051 100644
--- a/Documentation/ABI/testing/sysfs-devices-system-cpu
+++ b/Documentation/ABI/testing/sysfs-devices-system-cpu
@@ -487,3 +487,11 @@ Description: Information about CPU vulnerabilities
"Not affected" CPU is not affected by the vulnerability
"Vulnerable" CPU is affected and no mitigation in effect
"Mitigation: $M" CPU is affected and mitigation $M is in effect
+
+What: /sys/devices/system/cpu/cpu[0-9]+/small_core_siblings
+Date: Aug-2018
+KernelVersion: v4.19.0
+Contact: Linux for PowerPC mailing list <linuxppc-dev@ozlabs.org>
+Description: List of Physical ids of CPUs which share the L1 cache,
+ translation cache and instruction data-flow with this CPU.
+Values: Comma separated list of decimal integers.
diff --git a/arch/powerpc/include/asm/cputhreads.h b/arch/powerpc/include/asm/cputhreads.h
index d71a909..cb8b4a4 100644
--- a/arch/powerpc/include/asm/cputhreads.h
+++ b/arch/powerpc/include/asm/cputhreads.h
@@ -23,11 +23,13 @@
extern int threads_per_core;
extern int threads_per_subcore;
extern int threads_shift;
+extern bool has_big_cores;
extern cpumask_t threads_core_mask;
#else
#define threads_per_core 1
#define threads_per_subcore 1
#define threads_shift 0
+#define has_big_cores 0
#define threads_core_mask (*get_cpu_mask(0))
#endif
@@ -69,12 +71,35 @@ static inline cpumask_t cpu_online_cores_map(void)
return cpu_thread_mask_to_cores(cpu_online_mask);
}
+#define MAX_THREAD_LIST_SIZE 8
+#define THREAD_GROUP_SHARE_L1 1
+struct thread_groups {
+ unsigned int property;
+ unsigned int nr_groups;
+ unsigned int threads_per_group;
+ unsigned int thread_list[MAX_THREAD_LIST_SIZE];
+};
+
#ifdef CONFIG_SMP
int cpu_core_index_of_thread(int cpu);
int cpu_first_thread_of_core(int core);
+int parse_thread_groups(struct device_node *dn, struct thread_groups *tg,
+ unsigned property);
+int get_cpu_thread_group_start(int cpu, struct thread_groups *tg);
#else
static inline int cpu_core_index_of_thread(int cpu) { return cpu; }
static inline int cpu_first_thread_of_core(int core) { return core; }
+static inline int parse_thread_groups(struct device_node *dn,
+ struct thread_groups *tg,
+ unsigned property)
+{
+ return -ENODATA;
+}
+
+static inline int get_cpu_thread_group_start(int cpu, struct thread_groups *tg)
+{
+ return -1;
+}
#endif
static inline int cpu_thread_in_core(int cpu)
diff --git a/arch/powerpc/kernel/setup-common.c b/arch/powerpc/kernel/setup-common.c
index 40b44bb..272dbe5 100644
--- a/arch/powerpc/kernel/setup-common.c
+++ b/arch/powerpc/kernel/setup-common.c
@@ -402,10 +402,12 @@ void __init check_for_initrd(void)
#ifdef CONFIG_SMP
int threads_per_core, threads_per_subcore, threads_shift;
+bool has_big_cores;
cpumask_t threads_core_mask;
EXPORT_SYMBOL_GPL(threads_per_core);
EXPORT_SYMBOL_GPL(threads_per_subcore);
EXPORT_SYMBOL_GPL(threads_shift);
+EXPORT_SYMBOL_GPL(has_big_cores);
EXPORT_SYMBOL_GPL(threads_core_mask);
static void __init cpu_init_thread_core_maps(int tpc)
@@ -433,6 +435,150 @@ static void __init cpu_init_thread_core_maps(int tpc)
u32 *cpu_to_phys_id = NULL;
+/*
+ * parse_thread_groups: Parses the "ibm,thread-groups" device tree
+ * property for the CPU device node @dn and stores
+ * the parsed output in the thread_groups
+ * structure @tg if the ibm,thread-groups[0]
+ * matches @property.
+ *
+ * @dn: The device node of the CPU device.
+ * @tg: Pointer to a thread group structure into which the parsed
+ * output of "ibm,thread-groups" is stored.
+ * @property: The property of the thread-group that the caller is
+ * interested in.
+ *
+ * ibm,thread-groups[0..N-1] array defines which group of threads in
+ * the CPU-device node can be grouped together based on the property.
+ *
+ * ibm,thread-groups[0] tells us the property based on which the
+ * threads are being grouped together. If this value is 1, it implies
+ * that the threads in the same group share L1, translation cache.
+ *
+ * ibm,thread-groups[1] tells us how many such thread groups exist.
+ *
+ * ibm,thread-groups[2] tells us the number of threads in each such
+ * group.
+ *
+ * ibm,thread-groups[3..N-1] is the list of threads identified by
+ * "ibm,ppc-interrupt-server#s" arranged as per their membership in
+ * the grouping.
+ *
+ * Example: If ibm,thread-groups = [1,2,4,5,6,7,8,9,10,11,12] it
+ * implies that there are 2 groups of 4 threads each, where each group
+ * of threads share L1, translation cache.
+ *
+ * The "ibm,ppc-interrupt-server#s" of the first group is {5,6,7,8}
+ * and the "ibm,ppc-interrupt-server#s" of the second group is {9, 10,
+ * 11, 12} structure
+ *
+ * Returns 0 on success, -EINVAL if the property does not exist,
+ * -ENODATA if property does not have a value, and -EOVERFLOW if the
+ * property data isn't large enough.
+ */
+int parse_thread_groups(struct device_node *dn,
+ struct thread_groups *tg,
+ unsigned int property)
+{
+ int i;
+ u32 thread_group_array[3 + MAX_THREAD_LIST_SIZE];
+ u32 *thread_list;
+ size_t total_threads;
+ int ret;
+
+ ret = of_property_read_u32_array(dn, "ibm,thread-groups",
+ thread_group_array, 3);
+ if (ret)
+ goto out_err;
+
+ ret = -ENODATA;
+ tg->property = thread_group_array[0];
+ tg->nr_groups = thread_group_array[1];
+ tg->threads_per_group = thread_group_array[2];
+ if (tg->property != property ||
+ tg->nr_groups < 1 ||
+ tg->threads_per_group < 1)
+ goto out_err;
+
+ total_threads = tg->nr_groups * tg->threads_per_group;
+
+ ret = of_property_read_u32_array(dn, "ibm,thread-groups",
+ thread_group_array,
+ 3 + total_threads);
+ if (ret)
+ goto out_err;
+
+ thread_list = &thread_group_array[3];
+
+ for (i = 0 ; i < total_threads; i++)
+ tg->thread_list[i] = thread_list[i];
+
+ return 0;
+out_err:
+ tg->property = 0;
+ tg->nr_groups = 0;
+ tg->threads_per_group = 0;
+ return ret;
+}
+
+/*
+ * dt_has_big_core : Parses the device tree property
+ * "ibm,thread-groups" for device node pointed by @dn
+ * and stores the parsed output in the structure
+ * pointed to by @tg. Then checks if the output in
+ * @tg corresponds to a big-core.
+ *
+ * @dn: Device node pointer of the CPU node being checked for a
+ * big-core.
+ * @tg: Pointer to thread_groups struct in which parsed output of
+ * "ibm,thread-groups" is recorded.
+ *
+ * Returns true if the @dn points to a big-core.
+ * Returns false if there is an error in parsing "ibm,thread-groups"
+ * or the parsed output doesn't correspond to a big-core.
+ */
+static inline bool dt_has_big_core(struct device_node *dn,
+ struct thread_groups *tg)
+{
+ if (parse_thread_groups(dn, tg, THREAD_GROUP_SHARE_L1))
+ return false;
+
+ return true;
+}
+
+/*
+ * get_cpu_thread_group_start : Searches the thread group in tg->thread_list
+ * that @cpu belongs to.
+ *
+ * @cpu : The logical CPU whose thread group is being searched.
+ * @tg : The thread-group structure of the CPU node which @cpu belongs
+ * to.
+ *
+ * Returns the index to tg->thread_list that points to the the start
+ * of the thread_group that @cpu belongs to.
+ *
+ * Returns -1 if cpu doesn't belong to any of the groups pointed to by
+ * tg->thread_list.
+ */
+int get_cpu_thread_group_start(int cpu, struct thread_groups *tg)
+{
+ int hw_cpu_id = get_hard_smp_processor_id(cpu);
+ int i, j;
+
+ for (i = 0; i < tg->nr_groups; i++) {
+ int group_start = i * tg->threads_per_group;
+
+ for (j = 0; j < tg->threads_per_group; j++) {
+ int idx = group_start + j;
+
+ if (tg->thread_list[idx] == hw_cpu_id)
+ return group_start;
+ }
+ }
+
+ return -1;
+}
+
/**
* setup_cpu_maps - initialize the following cpu maps:
* cpu_possible_mask
@@ -457,6 +603,7 @@ void __init smp_setup_cpu_maps(void)
int cpu = 0;
int nthreads = 1;
+ has_big_cores = true;
DBG("smp_setup_cpu_maps()\n");
cpu_to_phys_id = __va(memblock_alloc(nr_cpu_ids * sizeof(u32),
@@ -467,6 +614,7 @@ void __init smp_setup_cpu_maps(void)
const __be32 *intserv;
__be32 cpu_be;
int j, len;
+ struct thread_groups tg;
DBG(" * %pOF...\n", dn);
@@ -505,6 +653,9 @@ void __init smp_setup_cpu_maps(void)
cpu++;
}
+ if (has_big_cores && !dt_has_big_core(dn, &tg))
+ has_big_cores = false;
+
if (cpu >= nr_cpu_ids) {
of_node_put(dn);
break;
diff --git a/arch/powerpc/kernel/sysfs.c b/arch/powerpc/kernel/sysfs.c
index 755dc98..83fd26b 100644
--- a/arch/powerpc/kernel/sysfs.c
+++ b/arch/powerpc/kernel/sysfs.c
@@ -18,6 +18,7 @@
#include <asm/smp.h>
#include <asm/pmc.h>
#include <asm/firmware.h>
+#include <asm/cputhreads.h>
#include "cacheinfo.h"
#include "setup.h"
@@ -1025,6 +1026,36 @@ static ssize_t show_physical_id(struct device *dev,
}
static DEVICE_ATTR(physical_id, 0444, show_physical_id, NULL);
+static ssize_t show_small_core_siblings(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct cpu *cpu = container_of(dev, struct cpu, dev);
+ struct device_node *dn = of_get_cpu_node(cpu->dev.id, NULL);
+ struct thread_groups tg;
+ int i, j, err;
+ ssize_t ret = 0;
+
+ err = parse_thread_groups(dn, &tg, THREAD_GROUP_SHARE_L1);
+ of_node_put(dn);
+
+ if (err)
+ return -ENODATA;
+
+ i = get_cpu_thread_group_start(cpu->dev.id, &tg);
+
+ if (i == -1)
+ return -ENODATA;
+
+ for (j = 0; j < tg.threads_per_group - 1; j++)
+ ret += sprintf(buf + ret, "%d,", tg.thread_list[i + j]);
+
+ ret += sprintf(buf + ret, "%d\n", tg.thread_list[i + j]);
+
+ return ret;
+}
+static DEVICE_ATTR(small_core_siblings, 0444, show_small_core_siblings, NULL);
+
static int __init topology_init(void)
{
int cpu, r;
@@ -1048,6 +1079,13 @@ static int __init topology_init(void)
register_cpu(c, cpu);
device_create_file(&c->dev, &dev_attr_physical_id);
+
+ if (has_big_cores) {
+ const struct device_attribute *attr =
+ &dev_attr_small_core_siblings;
+
+ device_create_file(&c->dev, attr);
+ }
}
}
r = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "powerpc/topology:online",
--
1.9.4
^ permalink raw reply related
* [PATCH v7 2/2] powerpc: Use cpu_smallcore_sibling_mask at SMT level on bigcores
From: Gautham R. Shenoy @ 2018-08-20 5:41 UTC (permalink / raw)
To: Srikar Dronamraju, Michael Ellerman, Benjamin Herrenschmidt,
Michael Neuling, Vaidyanathan Srinivasan, Akshay Adiga,
Shilpasri G Bhat, Oliver O'Halloran, Nicholas Piggin,
Murilo Opsfelder Araujo, Anton Blanchard
Cc: linuxppc-dev, linux-kernel, Gautham R. Shenoy
In-Reply-To: <1534743704-4760-1-git-send-email-ego@linux.vnet.ibm.com>
From: "Gautham R. Shenoy" <ego@linux.vnet.ibm.com>
Each of the SMT4 cores forming a big-core are more or less independent
units. Thus when multiple tasks are scheduled to run on the fused
core, we get the best performance when the tasks are spread across the
pair of SMT4 cores.
This patch achieves this by setting the SMT level mask to correspond
to the smallcore sibling mask on big-core systems. This patch also
ensures that while checked for shared-caches on big-core system, we
use the smallcore_sibling_mask to compare with the l2_cache_mask.
This ensure that the CACHE level sched-domain is created, whose groups
correspond to the threads of the big-core.
With this patch, the SMT sched-domain with SMT=8,4,2 on big-core
systems are as follows:
1) ppc64_cpu --smt=8
CPU0 attaching sched-domain(s):
domain-0: span=0,2,4,6 level=SMT
groups: 0:{ span=0 cap=294 }, 2:{ span=2 cap=294 },
4:{ span=4 cap=294 }, 6:{ span=6 cap=294 }
CPU1 attaching sched-domain(s):
domain-0: span=1,3,5,7 level=SMT
groups: 1:{ span=1 cap=294 }, 3:{ span=3 cap=294 },
5:{ span=5 cap=294 }, 7:{ span=7 cap=294 }
2) ppc64_cpu --smt=4
CPU0 attaching sched-domain(s):
domain-0: span=0,2 level=SMT
groups: 0:{ span=0 cap=589 }, 2:{ span=2 cap=589 }
CPU1 attaching sched-domain(s):
domain-0: span=1,3 level=SMT
groups: 1:{ span=1 cap=589 }, 3:{ span=3 cap=589 }
3) ppc64_cpu --smt=2
SMT domain is a trivial domain consisting of just
1 CPU. Hence this domain gets collapsed leaving only CACHE, DIE and
NUMA domains.
Signed-off-by: Gautham R. Shenoy <ego@linux.vnet.ibm.com>
---
arch/powerpc/kernel/smp.c | 136 +++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 134 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
index 4794d6b..00f60a8 100644
--- a/arch/powerpc/kernel/smp.c
+++ b/arch/powerpc/kernel/smp.c
@@ -76,6 +76,7 @@
struct thread_info *secondary_ti;
DEFINE_PER_CPU(cpumask_var_t, cpu_sibling_map);
+DEFINE_PER_CPU(cpumask_var_t, cpu_smallcore_map);
DEFINE_PER_CPU(cpumask_var_t, cpu_l2_cache_map);
DEFINE_PER_CPU(cpumask_var_t, cpu_core_map);
@@ -83,6 +84,23 @@
EXPORT_PER_CPU_SYMBOL(cpu_l2_cache_map);
EXPORT_PER_CPU_SYMBOL(cpu_core_map);
+/*
+ * On big-cores system, cpu_l1_cache_map for each CPU corresponds to
+ * the set its siblings that share the l1-cache. This map is
+ * initialized the first time the CPU comes online, and subsequently
+ * remains unchanged.
+ *
+ * parse_success records if there has been an error in parsing the
+ * "ibm,thread-groups" property which tells us which set of siblings
+ * share the l1-cache with the CPU.
+ */
+struct small_core_sibling {
+ cpumask_var_t cpu_l1_cache_map;
+ bool parse_success;
+};
+
+DEFINE_PER_CPU(struct small_core_sibling, small_core);
+
/* SMP operations for this machine */
struct smp_ops_t *smp_ops;
@@ -91,6 +109,11 @@
int smt_enabled_at_boot = 1;
+static inline struct cpumask *cpu_smallcore_mask(int cpu)
+{
+ return per_cpu(cpu_smallcore_map, cpu);
+}
+
/*
* Returns 1 if the specified cpu should be brought up during boot.
* Used to inhibit booting threads if they've been disabled or
@@ -670,6 +693,18 @@ static void set_cpus_unrelated(int i, int j,
}
#endif
+static inline void alloc_small_core_data(int cpu)
+{
+ struct small_core_sibling *this_small_core;
+
+ zalloc_cpumask_var_node(&per_cpu(cpu_smallcore_map, cpu),
+ GFP_KERNEL, cpu_to_node(cpu));
+
+ this_small_core = &per_cpu(small_core, cpu);
+ zalloc_cpumask_var_node(&this_small_core->cpu_l1_cache_map,
+ GFP_KERNEL, cpu_to_node(cpu));
+}
+
void __init smp_prepare_cpus(unsigned int max_cpus)
{
unsigned int cpu;
@@ -701,12 +736,19 @@ void __init smp_prepare_cpus(unsigned int max_cpus)
set_cpu_numa_mem(cpu,
local_memory_node(numa_cpu_lookup_table[cpu]));
}
+
+ if (has_big_cores)
+ alloc_small_core_data(cpu);
}
/* Init the cpumasks so the boot CPU is related to itself */
cpumask_set_cpu(boot_cpuid, cpu_sibling_mask(boot_cpuid));
cpumask_set_cpu(boot_cpuid, cpu_l2_cache_mask(boot_cpuid));
cpumask_set_cpu(boot_cpuid, cpu_core_mask(boot_cpuid));
+ if (has_big_cores) {
+ cpumask_set_cpu(boot_cpuid,
+ cpu_smallcore_mask(boot_cpuid));
+ }
if (smp_ops && smp_ops->probe)
smp_ops->probe();
@@ -991,10 +1033,83 @@ static void remove_cpu_from_masks(int cpu)
set_cpus_unrelated(cpu, i, cpu_core_mask);
set_cpus_unrelated(cpu, i, cpu_l2_cache_mask);
set_cpus_unrelated(cpu, i, cpu_sibling_mask);
+ if (has_big_cores)
+ set_cpus_unrelated(cpu, i, cpu_smallcore_mask);
}
}
#endif
+static inline void init_small_core_data(int cpu,
+ struct small_core_sibling *cpu_sc)
+{
+ struct device_node *dn;
+ int first_thread = cpu_first_thread_sibling(cpu);
+ int i, cpu_group_start = -1;
+ struct thread_groups tg;
+
+ cpumask_set_cpu(cpu, cpu_sc->cpu_l1_cache_map);
+
+ dn = of_get_cpu_node(cpu, NULL);
+ if (unlikely(!dn)) {
+ WARN_ON(1);
+ goto out;
+ }
+
+ if (unlikely(parse_thread_groups(dn, &tg,
+ THREAD_GROUP_SHARE_L1))) {
+ WARN_ON(1);
+ goto out;
+ }
+
+ cpu_group_start = get_cpu_thread_group_start(cpu, &tg);
+
+ if (unlikely(cpu_group_start == -1)) {
+ WARN_ON(1);
+ goto out;
+ }
+
+ for (i = first_thread; i < first_thread + threads_per_core; i++) {
+ int i_group_start = get_cpu_thread_group_start(i, &tg);
+
+ if (unlikely(i_group_start == -1)) {
+ WARN_ON(1);
+ goto out;
+ }
+
+ if (i_group_start == cpu_group_start)
+ cpumask_set_cpu(i, cpu_sc->cpu_l1_cache_map);
+ }
+
+ cpu_sc->parse_success = true;
+out:
+ of_node_put(dn);
+}
+
+static inline void add_cpu_to_smallcore_masks(int cpu)
+{
+ struct small_core_sibling *this_small_core = &per_cpu(small_core, cpu);
+ int i, first_thread = cpu_first_thread_sibling(cpu);
+
+ if (!has_big_cores)
+ return;
+
+ if (unlikely(cpumask_empty(this_small_core->cpu_l1_cache_map)))
+ init_small_core_data(cpu, this_small_core);
+
+ cpumask_set_cpu(cpu, cpu_smallcore_mask(cpu));
+
+ for (i = first_thread; i < first_thread + threads_per_core; i++) {
+ if (unlikely(!this_small_core->parse_success)) {
+ /* Fallback to siblings of the big-core */
+ set_cpus_related(i, cpu, cpu_smallcore_mask);
+ continue;
+ }
+
+ if (cpumask_test_cpu(i, this_small_core->cpu_l1_cache_map))
+ set_cpus_related(i, cpu, cpu_smallcore_mask);
+ }
+}
+
static void add_cpu_to_masks(int cpu)
{
int first_thread = cpu_first_thread_sibling(cpu);
@@ -1006,11 +1121,11 @@ static void add_cpu_to_masks(int cpu)
* add it to it's own thread sibling mask.
*/
cpumask_set_cpu(cpu, cpu_sibling_mask(cpu));
-
for (i = first_thread; i < first_thread + threads_per_core; i++)
if (cpu_online(i))
set_cpus_related(i, cpu, cpu_sibling_mask);
+ add_cpu_to_smallcore_masks(cpu);
/*
* Copy the thread sibling mask into the cache sibling mask
* and mark any CPUs that share an L2 with this CPU.
@@ -1040,6 +1155,7 @@ static void add_cpu_to_masks(int cpu)
void start_secondary(void *unused)
{
unsigned int cpu = smp_processor_id();
+ struct cpumask *(*sibling_mask)(int) = cpu_sibling_mask;
mmgrab(&init_mm);
current->active_mm = &init_mm;
@@ -1065,11 +1181,13 @@ void start_secondary(void *unused)
/* Update topology CPU masks */
add_cpu_to_masks(cpu);
+ if (has_big_cores)
+ sibling_mask = cpu_smallcore_mask;
/*
* Check for any shared caches. Note that this must be done on a
* per-core basis because one core in the pair might be disabled.
*/
- if (!cpumask_equal(cpu_l2_cache_mask(cpu), cpu_sibling_mask(cpu)))
+ if (!cpumask_equal(cpu_l2_cache_mask(cpu), sibling_mask(cpu)))
shared_caches = true;
set_numa_node(numa_cpu_lookup_table[cpu]);
@@ -1136,6 +1254,13 @@ static const struct cpumask *shared_cache_mask(int cpu)
return cpu_l2_cache_mask(cpu);
}
+#ifdef CONFIG_SCHED_SMT
+static const struct cpumask *smallcore_smt_mask(int cpu)
+{
+ return cpu_smallcore_mask(cpu);
+}
+#endif
+
static struct sched_domain_topology_level power9_topology[] = {
#ifdef CONFIG_SCHED_SMT
{ cpu_smt_mask, powerpc_smt_flags, SD_INIT_NAME(SMT) },
@@ -1158,6 +1283,13 @@ void __init smp_cpus_done(unsigned int max_cpus)
dump_numa_cpu_topology();
+#ifdef CONFIG_SCHED_SMT
+ if (has_big_cores) {
+ pr_info("Using small cores at SMT level\n");
+ power9_topology[0].mask = smallcore_smt_mask;
+ powerpc_topology[0].mask = smallcore_smt_mask;
+ }
+#endif
/*
* If any CPU detects that it's sharing a cache with another CPU then
* use the deeper topology that is aware of this sharing.
--
1.9.4
^ permalink raw reply related
* Re: [PATCH 3/3] powerpc/mm/ Add proper pte access check helper
From: Christophe LEROY @ 2018-08-20 6:12 UTC (permalink / raw)
To: Aneesh Kumar K.V, benh, paulus, mpe; +Cc: linuxppc-dev
In-Reply-To: <d4c28fbf-5663-b40a-c083-daf9f816fe91@c-s.fr>
Le 17/08/2018 à 17:12, Christophe LEROY a écrit :
>
>
> Le 04/12/2017 à 03:19, Aneesh Kumar K.V a écrit :
>> pte_access_premitted get called in get_user_pages_fast path. If we
>> have marked
>> the pte PROT_NONE, we should not allow a read access on the address. With
>> the current implementation we are not checking the READ and only check
>> for
>> WRITE. This is needed on archs like ppc64 that implement PROT_NONE using
>> _PAGE_USER access instead of _PAGE_PRESENT. Also add pte_user check
>> just to make sure
>> we are not accessing kernel mapping.
>>
>> Even though there is code duplication, keeping the low level pte
>> accessors
>> different for different platforms helps in code readability.
>>
>> Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
>> ---
>> arch/powerpc/include/asm/book3s/32/pgtable.h | 23
>> +++++++++++++++++++++++
>> arch/powerpc/include/asm/nohash/pgtable.h | 23
>> +++++++++++++++++++++++
>> 2 files changed, 46 insertions(+)
>>
>> diff --git a/arch/powerpc/include/asm/book3s/32/pgtable.h
>> b/arch/powerpc/include/asm/book3s/32/pgtable.h
>> index 016579ef16d3..30a155c0a6b0 100644
>> --- a/arch/powerpc/include/asm/book3s/32/pgtable.h
>> +++ b/arch/powerpc/include/asm/book3s/32/pgtable.h
>> @@ -311,6 +311,29 @@ static inline int pte_present(pte_t pte)
>> return pte_val(pte) & _PAGE_PRESENT;
>> }
>> +/*
>> + * We only find page table entry in the last level
>> + * Hence no need for other accessors
>> + */
>> +#define pte_access_permitted pte_access_permitted
>> +static inline bool pte_access_permitted(pte_t pte, bool write)
>> +{
>> + unsigned long pteval = pte_val(pte);
>> + /*
>> + * A read-only access is controlled by _PAGE_USER bit.
>> + * We have _PAGE_READ set for WRITE and EXECUTE
>> + */
>> + unsigned long need_pte_bits = _PAGE_PRESENT | _PAGE_USER;
>> +
>> + if (write)
>> + need_pte_bits |= _PAGE_WRITE;
>> +
>> + if ((pteval & need_pte_bits) != need_pte_bits)
>> + return false;
>> +
>> + return true;
>> +}
>> +
>> /* Conversion functions: convert a page and protection to a page entry,
>> * and a page entry and page directory to the page they refer to.
>> *
>> diff --git a/arch/powerpc/include/asm/nohash/pgtable.h
>> b/arch/powerpc/include/asm/nohash/pgtable.h
>> index 5c68f4a59f75..fc4376c8d444 100644
>> --- a/arch/powerpc/include/asm/nohash/pgtable.h
>> +++ b/arch/powerpc/include/asm/nohash/pgtable.h
>> @@ -45,6 +45,29 @@ static inline int pte_present(pte_t pte)
>> return pte_val(pte) & _PAGE_PRESENT;
>> }
>> +/*
>> + * We only find page table entry in the last level
>> + * Hence no need for other accessors
>> + */
>> +#define pte_access_permitted pte_access_permitted
>> +static inline bool pte_access_permitted(pte_t pte, bool write)
>> +{
>> + unsigned long pteval = pte_val(pte);
>> + /*
>> + * A read-only access is controlled by _PAGE_USER bit.
>> + * We have _PAGE_READ set for WRITE and EXECUTE
>> + */
>
> Not fully right. asm/pte-common.h defines:
>
> #define PAGE_NONE __pgprot(_PAGE_BASE | _PAGE_NA)
> #define PAGE_SHARED __pgprot(_PAGE_BASE | _PAGE_USER | _PAGE_RW)
> #define PAGE_SHARED_X __pgprot(_PAGE_BASE | _PAGE_USER | _PAGE_RW | \
> _PAGE_EXEC)
> #define PAGE_COPY __pgprot(_PAGE_BASE | _PAGE_USER | _PAGE_RO)
> #define PAGE_COPY_X __pgprot(_PAGE_BASE | _PAGE_USER | _PAGE_RO | \
> _PAGE_EXEC)
> #define PAGE_READONLY __pgprot(_PAGE_BASE | _PAGE_USER | _PAGE_RO)
> #define PAGE_READONLY_X __pgprot(_PAGE_BASE | _PAGE_USER | _PAGE_RO | \
> _PAGE_EXEC)
>
> On the 8xx, _PAGE_USER = 0
>
>> + unsigned long need_pte_bits = _PAGE_PRESENT | _PAGE_USER;
_PAGE_PRIVILEGED should be checked as well.
Indeed, it seems like our patches crossed each other. My patch for
adding _PAGE_PRIVILEGED is date January 16th while your's was merged on
the 17th.
I'm wondering if there are other places that are missing _PAGE_RO
handling and _PAGE_PRIVILEGED handling. Do you remember if you added any
recently ?
Christophe
>> +
>> + if (write)
>> + need_pte_bits |= _PAGE_WRITE;
>> +
>> + if ((pteval & need_pte_bits) != need_pte_bits)
>> + return false;
>
> This test is not fully correct:
> - To check access(read) permission, you also have to check that _PAGE_NA
> is not set.
> - To check write permission, you also have to check that neither
> _PAGE_NA nor _PAGE_RO are set.
>
> On the 8xx, you have:
> _PAGE_RW = _PAGE_WRITE = 0
> _PAGE_NA = 0x0200
> _PAGE_RO = 0x0600
>
> Christophe
>
>> +
>> + return true;
>> +}
>> +
>> /* Conversion functions: convert a page and protection to a page entry,
>> * and a page entry and page directory to the page they refer to.
>> *
>>
^ permalink raw reply
* Re: [PATCH 1/3] arch/powerpc/hugetlb: Use pte_access_permitted for hugetlb access check
From: Christophe LEROY @ 2018-08-20 6:12 UTC (permalink / raw)
To: Aneesh Kumar K.V, benh, paulus, mpe; +Cc: linuxppc-dev
In-Reply-To: <20171204021912.25974-1-aneesh.kumar@linux.vnet.ibm.com>
Le 04/12/2017 à 03:19, Aneesh Kumar K.V a écrit :
> No functional change in this patch. This update gup_hugepte to use the
> helper. This will help later when we add memory keys.
>
> Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
> ---
> arch/powerpc/mm/hugetlbpage.c | 4 +---
> 1 file changed, 1 insertion(+), 3 deletions(-)
>
> diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c
> index a9b9083c5e49..c7e5afe5e118 100644
> --- a/arch/powerpc/mm/hugetlbpage.c
> +++ b/arch/powerpc/mm/hugetlbpage.c
> @@ -855,9 +855,7 @@ int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr,
>
> pte = READ_ONCE(*ptep);
>
> - if (!pte_present(pte) || !pte_read(pte))
> - return 0;
> - if (write && !pte_write(pte))
> + if (!pte_access_permitted(pte, write))
Seems like pte_access_permitted() doesn't check _PAGE_RO whereas
pte_write() does.
Christophe
> return 0;
>
> /* hugepages are never "special" */
>
^ permalink raw reply
* Re: [PATCH v6 00/11] hugetlb: Factorize hugetlb architecture primitives
From: Alexandre Ghiti @ 2018-08-20 6:45 UTC (permalink / raw)
To: Michal Hocko
Cc: linux-mm, Mike Kravetz, linux, catalin.marinas, will.deacon,
tony.luck, fenghua.yu, ralf, paul.burton, jhogan, jejb, deller,
benh, paulus, mpe, ysato, dalias, davem, tglx, mingo, hpa, x86,
arnd, linux-arm-kernel, linux-kernel, linux-ia64, linux-mips,
linux-parisc, linuxppc-dev, linux-sh, sparclinux, linux-arch
In-Reply-To: <20180806175711.24438-1-alex@ghiti.fr>
Hi Michal,
This patchset got acked, tested and reviewed by quite a few people, and
it has been suggested
that it should be included in -mm tree: could you tell me if something
else needs to be done for
its inclusion ?
Thanks for your time,
Alex
On 08/06/2018 07:57 PM, Alexandre Ghiti wrote:
> [CC linux-mm for inclusion in -mm tree]
>
> In order to reduce copy/paste of functions across architectures and then
> make riscv hugetlb port (and future ports) simpler and smaller, this
> patchset intends to factorize the numerous hugetlb primitives that are
> defined across all the architectures.
>
> Except for prepare_hugepage_range, this patchset moves the versions that
> are just pass-through to standard pte primitives into
> asm-generic/hugetlb.h by using the same #ifdef semantic that can be
> found in asm-generic/pgtable.h, i.e. __HAVE_ARCH_***.
>
> s390 architecture has not been tackled in this serie since it does not
> use asm-generic/hugetlb.h at all.
>
> This patchset has been compiled on all addressed architectures with
> success (except for parisc, but the problem does not come from this
> series).
>
> v6:
> - Remove nohash/32 and book3s/32 powerpc specific implementations in
> order to use the generic ones.
> - Add all the Reviewed-by, Acked-by and Tested-by in the commits,
> thanks to everyone.
>
> v5:
> As suggested by Mike Kravetz, no need to move the #include
> <asm-generic/hugetlb.h> for arm and x86 architectures, let it live at
> the top of the file.
>
> v4:
> Fix powerpc build error due to misplacing of #include
> <asm-generic/hugetlb.h> outside of #ifdef CONFIG_HUGETLB_PAGE, as
> pointed by Christophe Leroy.
>
> v1, v2, v3:
> Same version, just problems with email provider and misuse of
> --batch-size option of git send-email
>
> Alexandre Ghiti (11):
> hugetlb: Harmonize hugetlb.h arch specific defines with pgtable.h
> hugetlb: Introduce generic version of hugetlb_free_pgd_range
> hugetlb: Introduce generic version of set_huge_pte_at
> hugetlb: Introduce generic version of huge_ptep_get_and_clear
> hugetlb: Introduce generic version of huge_ptep_clear_flush
> hugetlb: Introduce generic version of huge_pte_none
> hugetlb: Introduce generic version of huge_pte_wrprotect
> hugetlb: Introduce generic version of prepare_hugepage_range
> hugetlb: Introduce generic version of huge_ptep_set_wrprotect
> hugetlb: Introduce generic version of huge_ptep_set_access_flags
> hugetlb: Introduce generic version of huge_ptep_get
>
> arch/arm/include/asm/hugetlb-3level.h | 32 +---------
> arch/arm/include/asm/hugetlb.h | 30 ----------
> arch/arm64/include/asm/hugetlb.h | 39 +++---------
> arch/ia64/include/asm/hugetlb.h | 47 ++-------------
> arch/mips/include/asm/hugetlb.h | 40 +++----------
> arch/parisc/include/asm/hugetlb.h | 33 +++--------
> arch/powerpc/include/asm/book3s/32/pgtable.h | 6 --
> arch/powerpc/include/asm/book3s/64/pgtable.h | 1 +
> arch/powerpc/include/asm/hugetlb.h | 43 ++------------
> arch/powerpc/include/asm/nohash/32/pgtable.h | 6 --
> arch/powerpc/include/asm/nohash/64/pgtable.h | 1 +
> arch/sh/include/asm/hugetlb.h | 54 ++---------------
> arch/sparc/include/asm/hugetlb.h | 40 +++----------
> arch/x86/include/asm/hugetlb.h | 69 ----------------------
> include/asm-generic/hugetlb.h | 88 +++++++++++++++++++++++++++-
> 15 files changed, 135 insertions(+), 394 deletions(-)
>
^ permalink raw reply
* Re: [PATCH v6 00/11] hugetlb: Factorize hugetlb architecture primitives
From: Michal Hocko @ 2018-08-20 7:17 UTC (permalink / raw)
To: Alexandre Ghiti
Cc: linux-mm, Mike Kravetz, linux, catalin.marinas, will.deacon,
tony.luck, fenghua.yu, ralf, paul.burton, jhogan, jejb, deller,
benh, paulus, mpe, ysato, dalias, davem, tglx, mingo, hpa, x86,
arnd, linux-arm-kernel, linux-kernel, linux-ia64, linux-mips,
linux-parisc, linuxppc-dev, linux-sh, sparclinux, linux-arch
In-Reply-To: <81078a7f-09cf-7f19-f6bb-8a1f4968d6fb@ghiti.fr>
On Mon 20-08-18 08:45:10, Alexandre Ghiti wrote:
> Hi Michal,
>
> This patchset got acked, tested and reviewed by quite a few people, and it
> has been suggested
> that it should be included in -mm tree: could you tell me if something else
> needs to be done for
> its inclusion ?
>
> Thanks for your time,
I didn't really get to look at the series but seeing an Ack from Mike
and arch maintainers should be good enough for it to go. This email
doesn't have Andrew Morton in the CC list so you should add him if you
want the series to land into the mm tree.
--
Michal Hocko
SUSE Labs
^ permalink raw reply
* Re: [PATCH v6 00/11] hugetlb: Factorize hugetlb architecture primitives
From: Alexandre Ghiti @ 2018-08-20 7:36 UTC (permalink / raw)
To: Michal Hocko
Cc: linux-mm, Mike Kravetz, linux, catalin.marinas, will.deacon,
tony.luck, fenghua.yu, ralf, paul.burton, jhogan, jejb, deller,
benh, paulus, mpe, ysato, dalias, davem, tglx, mingo, hpa, x86,
arnd, linux-arm-kernel, linux-kernel, linux-ia64, linux-mips,
linux-parisc, linuxppc-dev, linux-sh, sparclinux, linux-arch,
Andrew Morton
In-Reply-To: <20180820071730.GC29735@dhcp22.suse.cz>
Ok, my bad, sorry about that, I have just added Andrew as CC then.
Thank you,
Alex
On 08/20/2018 09:17 AM, Michal Hocko wrote:
> On Mon 20-08-18 08:45:10, Alexandre Ghiti wrote:
>> Hi Michal,
>>
>> This patchset got acked, tested and reviewed by quite a few people, and it
>> has been suggested
>> that it should be included in -mm tree: could you tell me if something else
>> needs to be done for
>> its inclusion ?
>>
>> Thanks for your time,
> I didn't really get to look at the series but seeing an Ack from Mike
> and arch maintainers should be good enough for it to go. This email
> doesn't have Andrew Morton in the CC list so you should add him if you
> want the series to land into the mm tree.
^ permalink raw reply
* Re: [PATCH 6/9] PCI: hotplug: Constify hotplug_slot_ops
From: Rafael J. Wysocki @ 2018-08-20 8:10 UTC (permalink / raw)
To: Lukas Wunner
Cc: Bjorn Helgaas, Linux PCI, okaya, Mika Westerberg,
Rafael J. Wysocki, Len Brown, scott, Benjamin Herrenschmidt,
Paul Mackerras, Michael Ellerman, gwshan, sebott, gerald.schaefer,
Corentin Chary, Darren Hart, Andy Shevchenko, Greg Kroah-Hartman,
ACPI Devel Maling List, linux-s390, Platform Driver,
acpi4asus-user, linuxppc-dev
In-Reply-To: <40b17d029c4fbf099535f989b1d6dd013f708a73.1534686485.git.lukas@wunner.de>
On Sun, Aug 19, 2018 at 4:36 PM Lukas Wunner <lukas@wunner.de> wrote:
>
> Hotplug drivers cannot declare their hotplug_slot_ops const, making them
> attractive targets for attackers, because upon registration of a hotplug
> slot, __pci_hp_initialize() writes to the "owner" and "mod_name" members
> in that struct.
>
> Fix by moving these members to struct hotplug_slot and constify every
> driver's hotplug_slot_ops except for pciehp.
>
> pciehp constructs its hotplug_slot_ops at runtime based on the PCIe
> port's capabilities, hence cannot declare them const. It can be
> converted to __write_rarely once that's mainlined:
> http://www.openwall.com/lists/kernel-hardening/2016/11/16/3
>
> Signed-off-by: Lukas Wunner <lukas@wunner.de>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Cc: Len Brown <lenb@kernel.org>
> Cc: Scott Murray <scott@spiteful.org>
> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Cc: Paul Mackerras <paulus@samba.org>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Gavin Shan <gwshan@linux.vnet.ibm.com>
> Cc: Sebastian Ott <sebott@linux.vnet.ibm.com>
> Cc: Gerald Schaefer <gerald.schaefer@de.ibm.com>
> Cc: Corentin Chary <corentin.chary@gmail.com>
> Cc: Darren Hart <dvhart@infradead.org>
> Cc: Andy Shevchenko <andy@infradead.org>
> ---
> drivers/pci/hotplug/acpiphp_core.c | 2 +-
> drivers/pci/hotplug/cpci_hotplug_core.c | 2 +-
> drivers/pci/hotplug/cpqphp_core.c | 2 +-
> drivers/pci/hotplug/ibmphp.h | 2 +-
> drivers/pci/hotplug/ibmphp_core.c | 2 +-
> drivers/pci/hotplug/pci_hotplug_core.c | 27 +++++++++++++------------
> drivers/pci/hotplug/pnv_php.c | 2 +-
> drivers/pci/hotplug/rpaphp.h | 2 +-
> drivers/pci/hotplug/rpaphp_core.c | 2 +-
> drivers/pci/hotplug/s390_pci_hpc.c | 2 +-
> drivers/pci/hotplug/sgi_hotplug.c | 2 +-
> drivers/pci/hotplug/shpchp_core.c | 2 +-
> drivers/pci/pci.c | 4 ++--
> drivers/pci/slot.c | 2 +-
> drivers/platform/x86/asus-wmi.c | 3 +--
> drivers/platform/x86/eeepc-laptop.c | 3 +--
> include/linux/pci_hotplug.h | 10 ++++-----
> 17 files changed, 35 insertions(+), 36 deletions(-)
Nice!
Reviewed-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
^ permalink raw reply
* Re: [PATCH 7/9] PCI: hotplug: Drop hotplug_slot_info
From: Rafael J. Wysocki @ 2018-08-20 8:14 UTC (permalink / raw)
To: Lukas Wunner
Cc: Bjorn Helgaas, Linux PCI, okaya, Mika Westerberg,
Rafael J. Wysocki, Len Brown, scott, Benjamin Herrenschmidt,
Paul Mackerras, Michael Ellerman, gwshan, sebott, gerald.schaefer,
Corentin Chary, Darren Hart, Andy Shevchenko, Greg Kroah-Hartman,
ACPI Devel Maling List, linux-s390, Platform Driver,
acpi4asus-user, linuxppc-dev
In-Reply-To: <a364bcc959e01f2ebab27cefc5b87ff01633275c.1534686485.git.lukas@wunner.de>
On Sun, Aug 19, 2018 at 4:43 PM Lukas Wunner <lukas@wunner.de> wrote:
>
> Ever since the PCI hotplug core was introduced in 2002, drivers had to
> allocate and register a struct hotplug_slot_info for every slot:
> https://git.kernel.org/tglx/history/c/a8a2069f432c
>
> Apparently the idea was that drivers furnish the hotplug core with an
> up-to-date card presence status, power status, latch status and
> attention indicator status as well as notify the hotplug core of changes
> thereof. However only 4 out of 12 hotplug drivers bother to notify the
> hotplug core with pci_hp_change_slot_info() and the hotplug core never
> made any use of the information: There is just a single macro in
> pci_hotplug_core.c, GET_STATUS(), which uses the hotplug_slot_info if
> the driver lacks the corresponding callback in hotplug_slot_ops. The
> macro is called when the user reads the attribute via sysfs.
>
> Now, if the callback isn't defined, the attribute isn't exposed in sysfs
> in the first place (see e.g. has_power_file()). There are only two
> situations when the hotplug_slot_info would actually be accessed:
>
> * If the driver defines ->enable_slot or ->disable_slot but not
> ->get_power_status.
>
> * If the driver defines ->set_attention_status but not
> ->get_attention_status.
>
> There is no driver doing the former and just a single driver doing the
> latter, namely pnv_php.c. Amend it with a ->get_attention_status
> callback. With that, the hotplug_slot_info becomes completely unused by
> the PCI hotplug core. But a few drivers use it internally as a cache:
>
> cpcihp uses it to cache the latch_status and adapter_status.
> cpqhp uses it to cache the adapter_status.
> pnv_php and rpaphp use it to cache the attention_status.
> shpchp uses it to cache all four values.
>
> Amend these drivers to cache the information in their private slot
> struct. shpchp's slot struct already contains members to cache the
> power_status and adapter_status, so additional members are only needed
> for the other two values. In the case of cpqphp, the cached value is
> only accessed in a single place, so instead of caching it, read the
> current value from the hardware.
>
> Caution: acpiphp, cpci, cpqhp, shpchp, asus-wmi and eeepc-laptop
> populate the hotplug_slot_info with initial values on probe. That code
> is herewith removed. There is a theoretical chance that the code has
> side effects without which the driver fails to function, e.g. if the
> ACPI method to read the adapter status needs to be executed at least
> once on probe. That seems unlikely to me, still maintainers should
> review the changes carefully for this possibility.
I'm not aware of any case in which it will break anything, so
Reviewed-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
but if that happens, it may be necessary to add the execution of the
control methods in question directly to the initialization part.
^ permalink raw reply
* [PATCH v7 0/3] powerpc/fadump: Improvements for firmware-assisted dump.
From: Mahesh J Salgaonkar @ 2018-08-20 8:17 UTC (permalink / raw)
To: linuxppc-dev
Cc: Ananth N Mavinakayanahalli, Hari Bathini, Ananth Narayan,
Aneesh Kumar K.V, Hari Bathini, Nathan Fontenot, kernelfans
One of the primary issues with Firmware Assisted Dump (fadump) on Power
is that it needs a large amount of memory to be reserved. This reserved
memory is used for saving the contents of old crashed kernel's memory before
fadump capture kernel uses old kernel's memory area to boot. However, This
reserved memory area stays unused until system crash and isn't available
for production kernel to use.
Instead of setting aside a significant chunk of memory that nobody can use,
take advantage Linux kernel's Contiguous Memory Allocator (CMA) feature,
to reserve a significant chunk of memory that the kernel is prevented from
using, but applications are free to use it.
Patch 1 implements the usage of CMA region to allow production kernel to
use that memory for applications usage, making fadump reservationless.
We now initialize siginificant chunk of faump reserved memory for CMA.
Cahnges in V7:
- Revert back to use CMA for fadump reservation.
- Add fadump=nocma option to fall back default behaviour.
Changes in V6:
- Introduce an interface to mark reserved memory as ZONE_MOVABLE. Hence
sending this series as RFC again.
- Mark reserved area as ZONE_MOVABLE instead of CMA.
- Add fadump=nonmovable parameter for user who don't want to use ZONE_MOVABLE.
Changes in V5:
- Drop the patch that does metadata movement.
- Move the kexec fix patch to top (patch 1)
- Fold CMA documenation patch into patch 2
- Fix the compilation issues when CONFIG_CMA is not set reported by Hari.
- Use the approach of using boot memory size for CMA as suggested by Hari
except the movement of sections. Thanks to Hari.
Changes in V4:
- patch 1: Make fadump compatible irrespective of kernel versions.
- patch 4: moved out of the series and been posted seperatly at
http://patchwork.ozlabs.org/patch/896716/
- Documentation update about CMA reservation.
Changes in V3:
- patch 1 & 2: move metadata region and documentation update.
- patch 7: Un-register the faudmp on kexec path
---
Mahesh Salgaonkar (3):
powerpc/fadump: Reservationless firmware assisted dump
powerpc/fadump: throw proper error message on fadump registration failure.
powerpc/fadump: Do not allow hot-remove memory from fadump reserved area.
Documentation/powerpc/firmware-assisted-dump.txt | 17 +++
arch/powerpc/include/asm/fadump.h | 7 +
arch/powerpc/kernel/fadump.c | 140 ++++++++++++++++++++--
arch/powerpc/platforms/pseries/hotplug-memory.c | 7 +
4 files changed, 155 insertions(+), 16 deletions(-)
--
Signature
^ permalink raw reply
* [PATCH v7 1/3] powerpc/fadump: Reservationless firmware assisted dump
From: Mahesh J Salgaonkar @ 2018-08-20 8:17 UTC (permalink / raw)
To: linuxppc-dev
Cc: Ananth N Mavinakayanahalli, Hari Bathini, Ananth Narayan,
Aneesh Kumar K.V, Hari Bathini, Nathan Fontenot, kernelfans
In-Reply-To: <153475298147.22527.9680437074324546897.stgit@jupiter.in.ibm.com>
From: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>
One of the primary issues with Firmware Assisted Dump (fadump) on Power
is that it needs a large amount of memory to be reserved. On large
systems with TeraBytes of memory, this reservation can be quite
significant.
In some cases, fadump fails if the memory reserved is insufficient, or
if the reserved memory was DLPAR hot-removed.
In the normal case, post reboot, the preserved memory is filtered to
extract only relevant areas of interest using the makedumpfile tool.
While the tool provides flexibility to determine what needs to be part
of the dump and what memory to filter out, all supported distributions
default this to "Capture only kernel data and nothing else".
We take advantage of this default and the Linux kernel's Contiguous
Memory Allocator (CMA) to fundamentally change the memory reservation
model for fadump.
Instead of setting aside a significant chunk of memory nobody can use,
this patch uses CMA instead, to reserve a significant chunk of memory
that the kernel is prevented from using (due to MIGRATE_CMA), but
applications are free to use it. With this fadump will still be able
to capture all of the kernel memory and most of the user space memory
except the user pages that were present in CMA region.
Essentially, on a P9 LPAR with 2 cores, 8GB RAM and current upstream:
[root@zzxx-yy10 ~]# free -m
total used free shared buff/cache available
Mem: 7557 193 6822 12 541 6725
Swap: 4095 0 4095
With this patch:
[root@zzxx-yy10 ~]# free -m
total used free shared buff/cache available
Mem: 8133 194 7464 12 475 7338
Swap: 4095 0 4095
Changes made here are completely transparent to how fadump has
traditionally worked.
Thanks to Aneesh Kumar and Anshuman Khandual for helping us understand
CMA and its usage.
TODO:
- Handle case where CMA reservation spans nodes.
Signed-off-by: Ananth N Mavinakayanahalli <ananth@linux.vnet.ibm.com>
Signed-off-by: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
---
Documentation/powerpc/firmware-assisted-dump.txt | 17 ++++
arch/powerpc/include/asm/fadump.h | 5 +
arch/powerpc/kernel/fadump.c | 97 ++++++++++++++++++++--
3 files changed, 108 insertions(+), 11 deletions(-)
diff --git a/Documentation/powerpc/firmware-assisted-dump.txt b/Documentation/powerpc/firmware-assisted-dump.txt
index bdd344aa18d9..18c5feef2577 100644
--- a/Documentation/powerpc/firmware-assisted-dump.txt
+++ b/Documentation/powerpc/firmware-assisted-dump.txt
@@ -113,7 +113,15 @@ header, is usually reserved at an offset greater than boot memory
size (see Fig. 1). This area is *not* released: this region will
be kept permanently reserved, so that it can act as a receptacle
for a copy of the boot memory content in addition to CPU state
-and HPTE region, in the case a crash does occur.
+and HPTE region, in the case a crash does occur. Since this reserved
+memory area is used only after the system crash, there is no point in
+blocking this significant chunk of memory from production kernel.
+Hence, the implementation uses the Linux kernel's Contiguous Memory
+Allocator (CMA) for memory reservation if CMA is configured for kernel.
+With CMA reservation this memory will be available for applications to
+use it, while kernel is prevented from using it. With this fadump will
+still be able to capture all of the kernel memory and most of the user
+space memory except the user pages that were present in CMA region.
o Memory Reservation during first kernel
@@ -162,6 +170,9 @@ How to enable firmware-assisted dump (fadump):
1. Set config option CONFIG_FA_DUMP=y and build kernel.
2. Boot into linux kernel with 'fadump=on' kernel cmdline option.
+ By default, fadump reserved memory will be initialized as CMA area.
+ Alternatively, user can boot linux kernel with 'fadump=nocma' to
+ prevent fadump to use CMA.
3. Optionally, user can also set 'crashkernel=' kernel cmdline
to specify size of the memory to reserve for boot memory dump
preservation.
@@ -172,6 +183,10 @@ NOTE: 1. 'fadump_reserve_mem=' parameter has been deprecated. Instead
2. If firmware-assisted dump fails to reserve memory then it
will fallback to existing kdump mechanism if 'crashkernel='
option is set at kernel cmdline.
+ 3. if user wants to capture all of user space memory and ok with
+ reserved memory not available to production system, then
+ 'fadump=nocma' kernel parameter can be used to fallback to
+ old behaviour.
Sysfs/debugfs files:
------------
diff --git a/arch/powerpc/include/asm/fadump.h b/arch/powerpc/include/asm/fadump.h
index 5a23010af600..e9764b541927 100644
--- a/arch/powerpc/include/asm/fadump.h
+++ b/arch/powerpc/include/asm/fadump.h
@@ -48,6 +48,10 @@
#define memblock_num_regions(memblock_type) (memblock.memblock_type.cnt)
+/* Alignement per CMA requirement. */
+#define FADUMP_CMA_ALIGNMENT (PAGE_SIZE << \
+ max_t(unsigned long, MAX_ORDER - 1, pageblock_order))
+
/* Firmware provided dump sections */
#define FADUMP_CPU_STATE_DATA 0x0001
#define FADUMP_HPTE_REGION 0x0002
@@ -141,6 +145,7 @@ struct fw_dump {
unsigned long fadump_supported:1;
unsigned long dump_active:1;
unsigned long dump_registered:1;
+ unsigned long nocma:1;
};
/*
diff --git a/arch/powerpc/kernel/fadump.c b/arch/powerpc/kernel/fadump.c
index 07e8396d472b..166e71635921 100644
--- a/arch/powerpc/kernel/fadump.c
+++ b/arch/powerpc/kernel/fadump.c
@@ -34,6 +34,7 @@
#include <linux/crash_dump.h>
#include <linux/kobject.h>
#include <linux/sysfs.h>
+#include <linux/cma.h>
#include <asm/debugfs.h>
#include <asm/page.h>
@@ -45,11 +46,75 @@
static struct fw_dump fw_dump;
static struct fadump_mem_struct fdm;
static const struct fadump_mem_struct *fdm_active;
+#ifdef CONFIG_CMA
+static struct cma *fadump_cma;
+#endif
static DEFINE_MUTEX(fadump_mutex);
struct fad_crash_memory_ranges crash_memory_ranges[INIT_CRASHMEM_RANGES];
int crash_mem_ranges;
+#ifdef CONFIG_CMA
+/*
+ * fadump_cma_init() - Initialize CMA area from a fadump reserved memory
+ *
+ * This function initializes CMA area from fadump reserved memory.
+ * The total size of fadump reserved memory covers for boot memory size
+ * + cpu data size + hpte size and metadata.
+ * Initialize only the area equivalent to boot memory size for CMA use.
+ * The reamining portion of fadump reserved memory will be not given
+ * to CMA and pages for thoes will stay reserved. boot memory size is
+ * aligned per CMA requirement to satisy cma_init_reserved_mem() call.
+ * But for some reason even if it fails we still have the memory reservation
+ * with us and we can still continue doing fadump.
+ */
+int __init fadump_cma_init(void)
+{
+ unsigned long long base, size;
+ int rc;
+
+ if (!fw_dump.fadump_enabled)
+ return 0;
+
+ /*
+ * Do not use CMA if user has provided fadump=nocma kernel parameter.
+ * Return 1 to continue with fadump old behaviour.
+ */
+ if (fw_dump.nocma)
+ return 1;
+
+ base = fw_dump.reserve_dump_area_start;
+ size = fw_dump.boot_memory_size;
+
+ if (!size)
+ return 0;
+
+ rc = cma_init_reserved_mem(base, size, 0, "fadump_cma", &fadump_cma);
+ if (rc) {
+ pr_err("Failed to init cma area for firmware-assisted dump,%d\n", rc);
+ /*
+ * Though the CMA init has failed we still have memory
+ * reservation with us. The reserved memory will be
+ * blocked from production system usage. Hence return 1,
+ * so that we can continue with fadump.
+ */
+ return 1;
+ }
+
+ /*
+ * So we now have successfully initialized cma area for fadump.
+ */
+ pr_info("Initialized 0x%lx bytes cma area at %ldMB from 0x%lx "
+ "bytes of memory reserved for firmware-assisted dump\n",
+ cma_get_size(fadump_cma),
+ (unsigned long)cma_get_base(fadump_cma) >> 20,
+ fw_dump.reserve_dump_area_size);
+ return 1;
+}
+#else
+static int __init fadump_cma_init(void) { return 1; }
+#endif /* CONFIG_CMA */
+
/* Scan the Firmware Assisted dump configuration details. */
int __init early_init_dt_scan_fw_dump(unsigned long node,
const char *uname, int depth, void *data)
@@ -375,8 +440,15 @@ int __init fadump_reserve_mem(void)
*/
if (fdm_active)
fw_dump.boot_memory_size = be64_to_cpu(fdm_active->rmr_region.source_len);
- else
+ else {
fw_dump.boot_memory_size = fadump_calculate_reserve_size();
+#ifdef CONFIG_CMA
+ if (!fw_dump.nocma)
+ fw_dump.boot_memory_size =
+ ALIGN(fw_dump.boot_memory_size,
+ FADUMP_CMA_ALIGNMENT);
+#endif
+ }
/*
* Calculate the memory boundary.
@@ -423,8 +495,9 @@ int __init fadump_reserve_mem(void)
fw_dump.fadumphdr_addr =
be64_to_cpu(fdm_active->rmr_region.destination_address) +
be64_to_cpu(fdm_active->rmr_region.source_len);
- pr_debug("fadumphdr_addr = %p\n",
- (void *) fw_dump.fadumphdr_addr);
+ pr_debug("fadumphdr_addr = %pa\n", &fw_dump.fadumphdr_addr);
+ fw_dump.reserve_dump_area_start = base;
+ fw_dump.reserve_dump_area_size = size;
} else {
size = get_fadump_area_size();
@@ -452,10 +525,11 @@ int __init fadump_reserve_mem(void)
(unsigned long)(size >> 20),
(unsigned long)(base >> 20),
(unsigned long)(memblock_phys_mem_size() >> 20));
- }
- fw_dump.reserve_dump_area_start = base;
- fw_dump.reserve_dump_area_size = size;
+ fw_dump.reserve_dump_area_start = base;
+ fw_dump.reserve_dump_area_size = size;
+ return fadump_cma_init();
+ }
return 1;
}
@@ -474,6 +548,10 @@ static int __init early_fadump_param(char *p)
fw_dump.fadump_enabled = 1;
else if (strncmp(p, "off", 3) == 0)
fw_dump.fadump_enabled = 0;
+ else if (strncmp(p, "nocma", 5) == 0) {
+ fw_dump.fadump_enabled = 1;
+ fw_dump.nocma = 1;
+ }
return 0;
}
@@ -1146,7 +1224,7 @@ static int fadump_unregister_dump(struct fadump_mem_struct *fdm)
return 0;
}
-static int fadump_invalidate_dump(struct fadump_mem_struct *fdm)
+static int fadump_invalidate_dump(const struct fadump_mem_struct *fdm)
{
int rc = 0;
unsigned int wait_time;
@@ -1177,9 +1255,8 @@ void fadump_cleanup(void)
{
/* Invalidate the registration only if dump is active. */
if (fw_dump.dump_active) {
- init_fadump_mem_struct(&fdm,
- be64_to_cpu(fdm_active->cpu_state_data.destination_address));
- fadump_invalidate_dump(&fdm);
+ /* pass the same memory dump structure provided by platform */
+ fadump_invalidate_dump(fdm_active);
} else if (fw_dump.dump_registered) {
/* Un-register Firmware-assisted dump if it was registered. */
fadump_unregister_dump(&fdm);
^ permalink raw reply related
* Re: [PATCH 8/9] PCI: hotplug: Embed hotplug_slot
From: Rafael J. Wysocki @ 2018-08-20 8:17 UTC (permalink / raw)
To: Lukas Wunner
Cc: Bjorn Helgaas, Linux PCI, okaya, Mika Westerberg,
Rafael J. Wysocki, Len Brown, scott, Benjamin Herrenschmidt,
Paul Mackerras, Michael Ellerman, gwshan, sebott, gerald.schaefer,
Corentin Chary, Darren Hart, Andy Shevchenko, Greg Kroah-Hartman,
ACPI Devel Maling List, linux-s390, Platform Driver,
acpi4asus-user, linuxppc-dev
In-Reply-To: <9918fbca94bcfc03d1560b191d138d3e6664cc86.1534686485.git.lukas@wunner.de>
On Sun, Aug 19, 2018 at 4:46 PM Lukas Wunner <lukas@wunner.de> wrote:
>
> When the PCI hotplug core and its first user, cpqphp, were introduced in
> February 2002 with historic commit a8a2069f432c, cpqphp allocated a slot
> struct for its internal use plus a hotplug_slot struct to be registered
> with the hotplug core and linked the two with pointers:
> https://git.kernel.org/tglx/history/c/a8a2069f432c
>
> Nowadays, the predominant pattern in the tree is to embed ("subclass")
> such structures in one another and cast to the containing struct with
> container_of(). But it wasn't until July 2002 that container_of() was
> introduced with historic commit ec4f214232cf:
> https://git.kernel.org/tglx/history/c/ec4f214232cf
>
> pnv_php, introduced in 2016, did the right thing and embedded struct
> hotplug_slot in its internal struct pnv_php_slot, but all other drivers
> cargo-culted cpqphp's design and linked separate structs with pointers.
>
> Embedding structs is preferrable to linking them with pointers because
> it requires fewer allocations, thereby reducing overhead and simplifying
> error paths. Casting an embedded struct to the containing struct
> becomes a cheap subtraction rather than a dereference. And having fewer
> pointers reduces the risk of them pointing nowhere either accidentally
> or due to an attack.
>
> Convert all drivers to embed struct hotplug_slot in their internal slot
> struct. The "private" pointer in struct hotplug_slot thereby becomes
> unused, so drop it.
>
> Signed-off-by: Lukas Wunner <lukas@wunner.de>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Cc: Len Brown <lenb@kernel.org>
> Cc: Scott Murray <scott@spiteful.org>
> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Cc: Paul Mackerras <paulus@samba.org>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Gavin Shan <gwshan@linux.vnet.ibm.com>
> Cc: Sebastian Ott <sebott@linux.vnet.ibm.com>
> Cc: Gerald Schaefer <gerald.schaefer@de.ibm.com>
> Cc: Corentin Chary <corentin.chary@gmail.com>
> Cc: Darren Hart <dvhart@infradead.org>
> Cc: Andy Shevchenko <andy@infradead.org>
> ---
> drivers/pci/hotplug/acpiphp.h | 9 ++-
> drivers/pci/hotplug/acpiphp_core.c | 28 +++-----
> drivers/pci/hotplug/acpiphp_ibm.c | 2 +-
> drivers/pci/hotplug/cpci_hotplug.h | 9 ++-
> drivers/pci/hotplug/cpci_hotplug_core.c | 37 ++++------
> drivers/pci/hotplug/cpci_hotplug_pci.c | 6 +-
> drivers/pci/hotplug/cpqphp.h | 9 ++-
> drivers/pci/hotplug/cpqphp_core.c | 37 ++++------
> drivers/pci/hotplug/cpqphp_ctrl.c | 2 -
> drivers/pci/hotplug/ibmphp.h | 7 +-
> drivers/pci/hotplug/ibmphp_core.c | 92 +++++++++++--------------
> drivers/pci/hotplug/ibmphp_ebda.c | 37 +++-------
> drivers/pci/hotplug/pciehp.h | 11 ++-
> drivers/pci/hotplug/pciehp_core.c | 37 ++++------
> drivers/pci/hotplug/pciehp_ctrl.c | 4 +-
> drivers/pci/hotplug/pciehp_hpc.c | 8 +--
> drivers/pci/hotplug/pnv_php.c | 9 ++-
> drivers/pci/hotplug/rpaphp.h | 7 +-
> drivers/pci/hotplug/rpaphp_core.c | 14 ++--
> drivers/pci/hotplug/rpaphp_slot.c | 15 ++--
> drivers/pci/hotplug/s390_pci_hpc.c | 30 ++++----
> drivers/pci/hotplug/sgi_hotplug.c | 52 ++++++--------
> drivers/pci/hotplug/shpchp.h | 6 +-
> drivers/pci/hotplug/shpchp_core.c | 17 ++---
> drivers/platform/x86/asus-wmi.c | 26 +++----
> drivers/platform/x86/eeepc-laptop.c | 30 ++++----
> include/linux/pci_hotplug.h | 3 -
> 27 files changed, 223 insertions(+), 321 deletions(-)
Good cleanup.
Reviewed-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
^ permalink raw reply
* [PATCH v7 2/3] powerpc/fadump: throw proper error message on fadump registration failure.
From: Mahesh J Salgaonkar @ 2018-08-20 8:17 UTC (permalink / raw)
To: linuxppc-dev
Cc: Ananth Narayan, Aneesh Kumar K.V, Hari Bathini, Nathan Fontenot,
kernelfans
In-Reply-To: <153475298147.22527.9680437074324546897.stgit@jupiter.in.ibm.com>
From: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>
fadump fails to register when there are holes in reserved memory area.
This can happen if user has hot-removed a memory that falls in the fadump
reserved memory area. Throw a meaningful error message to the user in
such case.
Signed-off-by: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>
---
arch/powerpc/kernel/fadump.c | 33 +++++++++++++++++++++++++++++++++
1 file changed, 33 insertions(+)
diff --git a/arch/powerpc/kernel/fadump.c b/arch/powerpc/kernel/fadump.c
index 166e71635921..44a29e4d419c 100644
--- a/arch/powerpc/kernel/fadump.c
+++ b/arch/powerpc/kernel/fadump.c
@@ -234,6 +234,36 @@ static int is_boot_memory_area_contiguous(void)
return ret;
}
+/*
+ * Returns 1, if there are no holes in reserved memory area,
+ * 0 otherwise.
+ */
+static int is_reserved_memory_area_contiguous(void)
+{
+ struct memblock_region *reg;
+ unsigned long start, end;
+ unsigned long d_start = fw_dump.reserve_dump_area_start;
+ unsigned long d_end = d_start + fw_dump.reserve_dump_area_size;
+ int ret = 0;
+
+ for_each_memblock(memory, reg) {
+ start = max(d_start, (unsigned long)reg->base);
+ end = min(d_end, (unsigned long)(reg->base + reg->size));
+ if (d_start < end) {
+ /* Memory hole from d_start to start */
+ if (start > d_start)
+ break;
+
+ if (end == d_end) {
+ ret = 1;
+ break;
+ }
+ d_start = end + 1;
+ }
+ }
+ return ret;
+}
+
/* Print firmware assisted dump configurations for debugging purpose. */
static void fadump_show_config(void)
{
@@ -602,6 +632,9 @@ static int register_fw_dump(struct fadump_mem_struct *fdm)
if (!is_boot_memory_area_contiguous())
pr_err("Can't have holes in boot memory area while "
"registering fadump\n");
+ else if (!is_reserved_memory_area_contiguous())
+ pr_err("Can't have holes in reserved memory area while"
+ " registering fadump\n");
printk(KERN_ERR "Failed to register firmware-assisted kernel"
" dump. Parameter Error(%d).\n", rc);
^ permalink raw reply related
* [PATCH v7 3/3] powerpc/fadump: Do not allow hot-remove memory from fadump reserved area.
From: Mahesh J Salgaonkar @ 2018-08-20 8:17 UTC (permalink / raw)
To: linuxppc-dev
Cc: Ananth Narayan, Aneesh Kumar K.V, Hari Bathini, Nathan Fontenot,
kernelfans
In-Reply-To: <153475298147.22527.9680437074324546897.stgit@jupiter.in.ibm.com>
From: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>
For fadump to work successfully there should not be any holes in reserved
memory ranges where kernel has asked firmware to move the content of old
kernel memory in event of crash. Now that fadump uses CMA for reserved
area, this memory area is now not protected from hot-remove operations
unless it is cma allocated. Hence, fadump service can fail to re-register
after the hot-remove operation, if hot-removed memory belongs to fadump
reserved region. To avoid this make sure that memory from fadump reserved
area is not hot-removable if fadump is registered.
However, if user still wants to remove that memory, he can do so by
manually stopping fadump service before hot-remove operation.
Signed-off-by: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>
---
arch/powerpc/include/asm/fadump.h | 2 +-
arch/powerpc/kernel/fadump.c | 10 ++++++++--
arch/powerpc/platforms/pseries/hotplug-memory.c | 7 +++++--
3 files changed, 14 insertions(+), 5 deletions(-)
diff --git a/arch/powerpc/include/asm/fadump.h b/arch/powerpc/include/asm/fadump.h
index e9764b541927..43825111c479 100644
--- a/arch/powerpc/include/asm/fadump.h
+++ b/arch/powerpc/include/asm/fadump.h
@@ -208,7 +208,7 @@ struct fad_crash_memory_ranges {
unsigned long long size;
};
-extern int is_fadump_boot_memory_area(u64 addr, ulong size);
+extern int is_fadump_memory_area(u64 addr, ulong size);
extern int early_init_dt_scan_fw_dump(unsigned long node,
const char *uname, int depth, void *data);
extern int fadump_reserve_mem(void);
diff --git a/arch/powerpc/kernel/fadump.c b/arch/powerpc/kernel/fadump.c
index 44a29e4d419c..89bee4f4fe5c 100644
--- a/arch/powerpc/kernel/fadump.c
+++ b/arch/powerpc/kernel/fadump.c
@@ -180,13 +180,19 @@ int __init early_init_dt_scan_fw_dump(unsigned long node,
/*
* If fadump is registered, check if the memory provided
- * falls within boot memory area.
+ * falls within boot memory area and reserved memory area.
*/
-int is_fadump_boot_memory_area(u64 addr, ulong size)
+int is_fadump_memory_area(u64 addr, ulong size)
{
+ u64 d_start = fw_dump.reserve_dump_area_start;
+ u64 d_end = d_start + fw_dump.reserve_dump_area_size;
+
if (!fw_dump.dump_registered)
return 0;
+ if (((addr + size) > d_start) && (addr <= d_end))
+ return 1;
+
return (addr + size) > RMA_START && addr <= fw_dump.boot_memory_size;
}
diff --git a/arch/powerpc/platforms/pseries/hotplug-memory.c b/arch/powerpc/platforms/pseries/hotplug-memory.c
index c1578f54c626..e4c658cda3a7 100644
--- a/arch/powerpc/platforms/pseries/hotplug-memory.c
+++ b/arch/powerpc/platforms/pseries/hotplug-memory.c
@@ -389,8 +389,11 @@ static bool lmb_is_removable(struct drmem_lmb *lmb)
phys_addr = lmb->base_addr;
#ifdef CONFIG_FA_DUMP
- /* Don't hot-remove memory that falls in fadump boot memory area */
- if (is_fadump_boot_memory_area(phys_addr, block_sz))
+ /*
+ * Don't hot-remove memory that falls in fadump boot memory area
+ * and memory that is reserved for capturing old kernel memory.
+ */
+ if (is_fadump_memory_area(phys_addr, block_sz))
return false;
#endif
^ permalink raw reply related
* Re: [RFC 07/15] PCI/ACPI: clean up acpi_pci_root_create()
From: Rafael J. Wysocki @ 2018-08-20 8:23 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Linux PCI, Bjorn Helgaas, Linux Kernel Mailing List,
Christoph Hellwig, Lorenzo Pieralisi, Benjamin Herrenschmidt,
linuxppc-dev, ACPI Devel Maling List
In-Reply-To: <20180817102645.3839621-8-arnd@arndb.de>
On Fri, Aug 17, 2018 at 12:33 PM Arnd Bergmann <arnd@arndb.de> wrote:
>
> The acpi_pci_create_root_bus() can be fully integrated into
> acpi_pci_root_create(), improving a few things:
>
> * We can call pci_scan_root_bus_bridge(), which registers and
> scans the bridge in one step.
> * After a failure in pci_register_host_bridge(), we correctly
> clean up the resources.
> * The bridge settings (release function, flags, operations etc)
> can get set up before registering the bridge.
> * Further cleanup would be possible, removing duplication between
> pci_host_bridge and some ACPI structures.
>
> Signed-off-by: Arnd Bergmann <arnd@arndb.de>
> ---
> drivers/acpi/pci_root.c | 68 +++++++++++++++--------------------------
> 1 file changed, 24 insertions(+), 44 deletions(-)
>
> diff --git a/drivers/acpi/pci_root.c b/drivers/acpi/pci_root.c
> index 85dbcf47015b..5f73de3b67c8 100644
> --- a/drivers/acpi/pci_root.c
> +++ b/drivers/acpi/pci_root.c
> @@ -873,34 +873,6 @@ static void acpi_pci_root_release_info(struct pci_host_bridge *bridge)
> __acpi_pci_root_release_info(bridge->release_data);
> }
>
> -static struct pci_bus *acpi_pci_create_root_bus(struct device *parent, int bus,
> - struct pci_ops *ops, void *sysdata, struct list_head *resources)
> -{
> - int error;
> - struct pci_host_bridge *bridge;
> -
> - bridge = pci_alloc_host_bridge(0);
> - if (!bridge)
> - return NULL;
> -
> - bridge->dev.parent = parent;
> -
> - list_splice_init(resources, &bridge->windows);
> - bridge->sysdata = sysdata;
> - bridge->busnr = bus;
> - bridge->ops = ops;
> -
> - error = pci_register_host_bridge(bridge);
> - if (error < 0)
> - goto err_out;
> -
> - return bridge->bus;
> -
> -err_out:
> - kfree(bridge);
> - return NULL;
> -}
> -
> struct pci_bus *acpi_pci_root_create(struct acpi_pci_root *root,
> struct acpi_pci_root_ops *ops,
> struct acpi_pci_root_info *info,
> @@ -909,8 +881,7 @@ struct pci_bus *acpi_pci_root_create(struct acpi_pci_root *root,
> int ret, busnum = root->secondary.start;
> struct acpi_device *device = root->device;
> int node = acpi_get_node(device->handle);
> - struct pci_bus *bus;
> - struct pci_host_bridge *host_bridge;
> + struct pci_host_bridge *bridge;
Why "bridge" and not "host" or even something to stand for "root complex"?
Or maybe it can still be "host_bridge"?
>
> info->root = root;
> info->bridge = device;
> @@ -930,30 +901,39 @@ struct pci_bus *acpi_pci_root_create(struct acpi_pci_root *root,
>
> pci_acpi_root_add_resources(info);
> pci_add_resource(&info->resources, &root->secondary);
> - bus = acpi_pci_create_root_bus(NULL, busnum, ops->pci_ops,
> - sysdata, &info->resources);
> - if (!bus)
> +
> + bridge = pci_alloc_host_bridge(0);
> + if (!bridge)
> goto out_release_info;
>
> - host_bridge = to_pci_host_bridge(bus->bridge);
> + list_splice_init(&info->resources, &bridge->windows);
> + bridge->sysdata = sysdata;
> + bridge->busnr = busnum;
> + bridge->ops = ops->pci_ops;
> + pci_set_host_bridge_release(bridge, acpi_pci_root_release_info,
> + info);
> +
> if (!(root->osc_control_set & OSC_PCI_EXPRESS_NATIVE_HP_CONTROL))
> - host_bridge->native_pcie_hotplug = 0;
> + bridge->native_pcie_hotplug = 0;
> if (!(root->osc_control_set & OSC_PCI_SHPC_NATIVE_HP_CONTROL))
> - host_bridge->native_shpc_hotplug = 0;
> + bridge->native_shpc_hotplug = 0;
> if (!(root->osc_control_set & OSC_PCI_EXPRESS_AER_CONTROL))
> - host_bridge->native_aer = 0;
> + bridge->native_aer = 0;
> if (!(root->osc_control_set & OSC_PCI_EXPRESS_PME_CONTROL))
> - host_bridge->native_pme = 0;
> + bridge->native_pme = 0;
> if (!(root->osc_control_set & OSC_PCI_EXPRESS_LTR_CONTROL))
> - host_bridge->native_ltr = 0;
> + bridge->native_ltr = 0;
> +
> + ret = pci_scan_root_bus_bridge(bridge);
> + if (ret < 0)
> + goto out_release_bridge;
>
> - pci_scan_child_bus(bus);
> - pci_set_host_bridge_release(host_bridge, acpi_pci_root_release_info,
> - info);
> if (node != NUMA_NO_NODE)
> - dev_printk(KERN_DEBUG, &bus->dev, "on NUMA node %d\n", node);
> - return bus;
> + dev_printk(KERN_DEBUG, &bridge->bus->dev, "on NUMA node %d\n", node);
> + return bridge->bus;
>
> +out_release_bridge:
> + pci_free_host_bridge(bridge);
> out_release_info:
> __acpi_pci_root_release_info(info);
> return NULL;
> --
> 2.18.0
>
^ permalink raw reply
* Re: [RFC 08/15] x86: PCI: clean up pcibios_scan_root()
From: Rafael J. Wysocki @ 2018-08-20 8:31 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Linux PCI, Bjorn Helgaas, Linux Kernel Mailing List,
Christoph Hellwig, Lorenzo Pieralisi, Benjamin Herrenschmidt,
linuxppc-dev, ACPI Devel Maling List
In-Reply-To: <20180817102645.3839621-9-arnd@arndb.de>
On Fri, Aug 17, 2018 at 12:32 PM Arnd Bergmann <arnd@arndb.de> wrote:
>
> pcibios_scan_root() is now just a wrapper around pci_scan_root_bus(),
> and merging the two into one makes it shorter and more readable.
>
> We can also take advantage of pci_alloc_host_bridge() doing the
> allocation of the sysdata for us, which helps if we ever want to
> allow hot-unplugging the host bridge itself.
>
> We might be able to simplify it further using pci_host_probe(),
> but I wasn't sure about the resource registration there.
>
> Signed-off-by: Arnd Bergmann <arnd@arndb.de>
> ---
> arch/x86/pci/common.c | 53 ++++++++++++++-----------------------------
> 1 file changed, 17 insertions(+), 36 deletions(-)
>
> diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
> index e740d9aa4024..920d0885434c 100644
> --- a/arch/x86/pci/common.c
> +++ b/arch/x86/pci/common.c
> @@ -453,54 +453,35 @@ void __init dmi_check_pciprobe(void)
> dmi_check_system(pciprobe_dmi_table);
> }
>
> -static struct pci_bus *pci_scan_root_bus(struct device *parent, int bus,
> - struct pci_ops *ops, void *sysdata, struct list_head *resources)
> +void pcibios_scan_root(int busnum)
> {
> + struct pci_sysdata *sd;
> struct pci_host_bridge *bridge;
> int error;
>
> - bridge = pci_alloc_host_bridge(0);
> - if (!bridge)
> - return NULL;
> + bridge = pci_alloc_host_bridge(sizeof(sd));
> + if (!bridge) {
> + printk(KERN_ERR "PCI: OOM, skipping PCI bus %02x\n", busnum);
> + return;
> + }
> + sd = pci_host_bridge_priv(bridge);
This looks fishy, as bridge->private is not set at this point AFAICS,
unless one of the previous patches changes that.
>
> - list_splice_init(resources, &bridge->windows);
> - bridge->dev.parent = parent;
> - bridge->sysdata = sysdata;
> - bridge->busnr = bus;
> - bridge->ops = ops;
> + sd->node = x86_pci_root_bus_node(busnum);
> + x86_pci_root_bus_resources(busnum, &bridge->windows);
> + bridge->sysdata = sd;
> + bridge->busnr = busnum;
> + bridge->ops = &pci_root_ops;
>
> + printk(KERN_DEBUG "PCI: Probing PCI hardware (bus %02x)\n", busnum);
> error = pci_scan_root_bus_bridge(bridge);
> if (error < 0)
> goto err_out;
>
> - return bridge->bus;
> + pci_bus_add_devices(bridge->bus);
> + return;
>
> err_out:
> - kfree(bridge);
> - return NULL;
> -}
> -
> -void pcibios_scan_root(int busnum)
> -{
> - struct pci_bus *bus;
> - struct pci_sysdata *sd;
> - LIST_HEAD(resources);
> -
> - sd = kzalloc(sizeof(*sd), GFP_KERNEL);
> - if (!sd) {
> - printk(KERN_ERR "PCI: OOM, skipping PCI bus %02x\n", busnum);
> - return;
> - }
> - sd->node = x86_pci_root_bus_node(busnum);
> - x86_pci_root_bus_resources(busnum, &resources);
> - printk(KERN_DEBUG "PCI: Probing PCI hardware (bus %02x)\n", busnum);
> - bus = pci_scan_root_bus(NULL, busnum, &pci_root_ops, sd, &resources);
> - if (!bus) {
> - pci_free_resource_list(&resources);
> - kfree(sd);
> - return;
> - }
> - pci_bus_add_devices(bus);
> + pci_free_host_bridge(bridge);
> }
>
> void __init pcibios_set_cache_line_size(void)
> --
> 2.18.0
>
^ permalink raw reply
* Re: [RESEND PATCH] i2c/busses/pasemi: Remove hardcoded bus numbers on smbus
From: Wolfram Sang @ 2018-08-20 9:07 UTC (permalink / raw)
To: Darren Stevens; +Cc: linux-i2c, linux-kernel, linuxppc-dev
In-Reply-To: <4b3d0f2cad0.58555470@auth.smtp.1and1.co.uk>
[-- Attachment #1: Type: text/plain, Size: 815 bytes --]
On Sun, Dec 31, 2017 at 08:53:55PM +0000, Darren Stevens wrote:
> The pasemi smbus controller uses PCI_FUNC(dev->devfn) to define which
> number bus to attach to, however this fails when something else is
> probed first, for example an ATI Radeon graphics card will claim 9 or
> 10 busses, including the ones the pasemi wants.
> Patch the driver to call i2c_add_adapter rather than
> i2c_add_numbered_adapter.
>
> Signed-off-by: Darren Stevens <darren@stevens-zone.net>
>
Applied to for-next, thanks!
Disclaimer: I usually do not like to change the bus numbering because
some people may rely on that. But numbering based on PCI functions seems
really weak and all known users of pasemi seem to have issues here, so I
make an exception.
Thanks to Michael Ellerman for the additional info.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* [RFC PATCH 0/5] rewriting SLB miss handler in C
From: Nicholas Piggin @ 2018-08-20 9:41 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V
I'd like to rewrite the SLB miss handlers in C for maintainability
and ability to more easily extend the code.
I have not benchmarked it yet but obviously setting up the stack
and going to C code rather than carefully hand optimised assembly
is likely to slow down SLB misses by a reasonable amount. So I've
started looked at a few basic optimisations we can make to justify
this change. There is still more that can be done, but SLB misses
have been reduced significantly, and with more tuning and optimization
I think we could bring it down quite a bit more.
I'm trying to get the first patch solid, and it is the big change so
would really appreciate review and commets on that. Other patches are
not quite polished but comments would still be welcome on those (keep
in mind they are obviously not all polished).
Thanks,
Nick
Nicholas Piggin (5):
powerpc/64s/hash: convert SLB miss handlers to C
powerpc/64s/hash: remove user SLB data from the paca
powerpc/64s/hash: remove the first vmalloc segment from the bolted SLB
powerpc/64s/hash: Add SLB allocation bitmaps
powerpc/64s/hash: Add a SLB preload cache
arch/powerpc/include/asm/asm-prototypes.h | 2 +
arch/powerpc/include/asm/book3s/64/mmu-hash.h | 3 +-
arch/powerpc/include/asm/paca.h | 19 +-
arch/powerpc/include/asm/thread_info.h | 4 +
arch/powerpc/kernel/asm-offsets.c | 9 -
arch/powerpc/kernel/exceptions-64s.S | 200 ++-------
arch/powerpc/kernel/paca.c | 21 -
arch/powerpc/kernel/process.c | 6 +
arch/powerpc/mm/Makefile | 2 +-
arch/powerpc/mm/hash_utils_64.c | 46 +--
arch/powerpc/mm/mmu_context.c | 3 +-
arch/powerpc/mm/mmu_context_book3s64.c | 10 +-
arch/powerpc/mm/slb.c | 382 +++++++++++-------
arch/powerpc/mm/slb_low.S | 338 ----------------
arch/powerpc/mm/slice.c | 29 +-
arch/powerpc/xmon/xmon.c | 2 +-
16 files changed, 328 insertions(+), 748 deletions(-)
delete mode 100644 arch/powerpc/mm/slb_low.S
--
2.17.0
^ permalink raw reply
* [RFC PATCH 1/5] powerpc/64s/hash: convert SLB miss handlers to C
From: Nicholas Piggin @ 2018-08-20 9:41 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V
In-Reply-To: <20180820094200.13003-1-npiggin@gmail.com>
This patch moves SLB miss handlers completely to C, using the standard
exception handler macros to set up the stack and branch to C.
This can be done because the segment containing the kernel stack is
always bolted, so accessing it with relocation on will not cause an
SLB exception.
Arbitrary kernel memory may not be accessed when handling kernel space
SLB misses, so care should be taken there. However user SLB misses can
access any kernel memory, which can be used to move some fields out of
the paca (in later patches).
User SLB misses could quite easily reconcile IRQs and set up a first
class kernel environment and exit via ret_from_except, however that
doesn't seem to be necessary at the moment, so we only do that if a
bad fault is encountered.
[ Credit to Aneesh for bug fixes and improvements to bad address
handling ]
---
arch/powerpc/include/asm/asm-prototypes.h | 2 +
arch/powerpc/kernel/exceptions-64s.S | 200 +++----------
arch/powerpc/mm/Makefile | 2 +-
arch/powerpc/mm/slb.c | 237 +++++++--------
arch/powerpc/mm/slb_low.S | 338 ----------------------
5 files changed, 166 insertions(+), 613 deletions(-)
delete mode 100644 arch/powerpc/mm/slb_low.S
diff --git a/arch/powerpc/include/asm/asm-prototypes.h b/arch/powerpc/include/asm/asm-prototypes.h
index 1f4691ce4126..c330ed10074a 100644
--- a/arch/powerpc/include/asm/asm-prototypes.h
+++ b/arch/powerpc/include/asm/asm-prototypes.h
@@ -78,6 +78,8 @@ void kernel_bad_stack(struct pt_regs *regs);
void system_reset_exception(struct pt_regs *regs);
void machine_check_exception(struct pt_regs *regs);
void emulation_assist_interrupt(struct pt_regs *regs);
+long do_slb_fault(struct pt_regs *regs, unsigned long ea);
+void do_bad_slb_fault(struct pt_regs *regs, unsigned long ea, unsigned long err);
/* signals, syscalls and interrupts */
long sys_swapcontext(struct ucontext __user *old_ctx,
diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
index 9dad73722d1a..f22ddb301661 100644
--- a/arch/powerpc/kernel/exceptions-64s.S
+++ b/arch/powerpc/kernel/exceptions-64s.S
@@ -567,28 +567,35 @@ ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX)
EXC_REAL_BEGIN(data_access_slb, 0x380, 0x80)
- SET_SCRATCH0(r13)
- EXCEPTION_PROLOG_0(PACA_EXSLB)
- EXCEPTION_PROLOG_1(PACA_EXSLB, KVMTEST_PR, 0x380)
- mr r12,r3 /* save r3 */
- mfspr r3,SPRN_DAR
- mfspr r11,SPRN_SRR1
- crset 4*cr6+eq
- BRANCH_TO_COMMON(r10, slb_miss_common)
+EXCEPTION_PROLOG(PACA_EXSLB, data_access_slb_common, EXC_STD, KVMTEST_PR, 0x380);
EXC_REAL_END(data_access_slb, 0x380, 0x80)
EXC_VIRT_BEGIN(data_access_slb, 0x4380, 0x80)
- SET_SCRATCH0(r13)
- EXCEPTION_PROLOG_0(PACA_EXSLB)
- EXCEPTION_PROLOG_1(PACA_EXSLB, NOTEST, 0x380)
- mr r12,r3 /* save r3 */
- mfspr r3,SPRN_DAR
- mfspr r11,SPRN_SRR1
- crset 4*cr6+eq
- BRANCH_TO_COMMON(r10, slb_miss_common)
+EXCEPTION_RELON_PROLOG(PACA_EXSLB, data_access_slb_common, EXC_STD, NOTEST, 0x380);
EXC_VIRT_END(data_access_slb, 0x4380, 0x80)
+
TRAMP_KVM_SKIP(PACA_EXSLB, 0x380)
+EXC_COMMON_BEGIN(data_access_slb_common)
+ mfspr r10,SPRN_DAR
+ std r10,PACA_EXSLB+EX_DAR(r13)
+ EXCEPTION_PROLOG_COMMON(0x380, PACA_EXSLB)
+ ld r4,PACA_EXSLB+EX_DAR(r13)
+ std r4,_DAR(r1)
+ addi r3,r1,STACK_FRAME_OVERHEAD
+ bl do_slb_fault
+ cmpdi r3,0
+ bne- 1f
+ b fast_exception_return
+1: /* Error case */
+ bl save_nvgprs
+ RECONCILE_IRQ_STATE(r10, r11)
+ ld r4,_DAR(r1)
+ mr r5,r3
+ addi r3,r1,STACK_FRAME_OVERHEAD
+ bl do_bad_slb_fault
+ b ret_from_except
+
EXC_REAL(instruction_access, 0x400, 0x80)
EXC_VIRT(instruction_access, 0x4400, 0x80, 0x400)
@@ -611,160 +618,33 @@ ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX)
EXC_REAL_BEGIN(instruction_access_slb, 0x480, 0x80)
- SET_SCRATCH0(r13)
- EXCEPTION_PROLOG_0(PACA_EXSLB)
- EXCEPTION_PROLOG_1(PACA_EXSLB, KVMTEST_PR, 0x480)
- mr r12,r3 /* save r3 */
- mfspr r3,SPRN_SRR0 /* SRR0 is faulting address */
- mfspr r11,SPRN_SRR1
- crclr 4*cr6+eq
- BRANCH_TO_COMMON(r10, slb_miss_common)
+EXCEPTION_PROLOG(PACA_EXSLB, instruction_access_slb_common, EXC_STD, KVMTEST_PR, 0x480);
EXC_REAL_END(instruction_access_slb, 0x480, 0x80)
EXC_VIRT_BEGIN(instruction_access_slb, 0x4480, 0x80)
- SET_SCRATCH0(r13)
- EXCEPTION_PROLOG_0(PACA_EXSLB)
- EXCEPTION_PROLOG_1(PACA_EXSLB, NOTEST, 0x480)
- mr r12,r3 /* save r3 */
- mfspr r3,SPRN_SRR0 /* SRR0 is faulting address */
- mfspr r11,SPRN_SRR1
- crclr 4*cr6+eq
- BRANCH_TO_COMMON(r10, slb_miss_common)
+EXCEPTION_RELON_PROLOG(PACA_EXSLB, instruction_access_slb_common, EXC_STD, NOTEST, 0x480);
EXC_VIRT_END(instruction_access_slb, 0x4480, 0x80)
-TRAMP_KVM(PACA_EXSLB, 0x480)
-
-
-/*
- * This handler is used by the 0x380 and 0x480 SLB miss interrupts, as well as
- * the virtual mode 0x4380 and 0x4480 interrupts if AIL is enabled.
- */
-EXC_COMMON_BEGIN(slb_miss_common)
- /*
- * r13 points to the PACA, r9 contains the saved CR,
- * r12 contains the saved r3,
- * r11 contain the saved SRR1, SRR0 is still ready for return
- * r3 has the faulting address
- * r9 - r13 are saved in paca->exslb.
- * cr6.eq is set for a D-SLB miss, clear for a I-SLB miss
- * We assume we aren't going to take any exceptions during this
- * procedure.
- */
- mflr r10
- stw r9,PACA_EXSLB+EX_CCR(r13) /* save CR in exc. frame */
- std r10,PACA_EXSLB+EX_LR(r13) /* save LR */
-
- andi. r9,r11,MSR_PR // Check for exception from userspace
- cmpdi cr4,r9,MSR_PR // And save the result in CR4 for later
-
- /*
- * Test MSR_RI before calling slb_allocate_realmode, because the
- * MSR in r11 gets clobbered. However we still want to allocate
- * SLB in case MSR_RI=0, to minimise the risk of getting stuck in
- * recursive SLB faults. So use cr5 for this, which is preserved.
- */
- andi. r11,r11,MSR_RI /* check for unrecoverable exception */
- cmpdi cr5,r11,MSR_RI
-
- crset 4*cr0+eq
-#ifdef CONFIG_PPC_BOOK3S_64
-BEGIN_MMU_FTR_SECTION
- bl slb_allocate
-END_MMU_FTR_SECTION_IFCLR(MMU_FTR_TYPE_RADIX)
-#endif
-
- ld r10,PACA_EXSLB+EX_LR(r13)
- lwz r9,PACA_EXSLB+EX_CCR(r13) /* get saved CR */
- mtlr r10
-
- /*
- * Large address, check whether we have to allocate new contexts.
- */
- beq- 8f
-
- bne- cr5,2f /* if unrecoverable exception, oops */
-
- /* All done -- return from exception. */
-
- bne cr4,1f /* returning to kernel */
-
- mtcrf 0x80,r9
- mtcrf 0x08,r9 /* MSR[PR] indication is in cr4 */
- mtcrf 0x04,r9 /* MSR[RI] indication is in cr5 */
- mtcrf 0x02,r9 /* I/D indication is in cr6 */
- mtcrf 0x01,r9 /* slb_allocate uses cr0 and cr7 */
-
- RESTORE_CTR(r9, PACA_EXSLB)
- RESTORE_PPR_PACA(PACA_EXSLB, r9)
- mr r3,r12
- ld r9,PACA_EXSLB+EX_R9(r13)
- ld r10,PACA_EXSLB+EX_R10(r13)
- ld r11,PACA_EXSLB+EX_R11(r13)
- ld r12,PACA_EXSLB+EX_R12(r13)
- ld r13,PACA_EXSLB+EX_R13(r13)
- RFI_TO_USER
- b . /* prevent speculative execution */
-1:
- mtcrf 0x80,r9
- mtcrf 0x08,r9 /* MSR[PR] indication is in cr4 */
- mtcrf 0x04,r9 /* MSR[RI] indication is in cr5 */
- mtcrf 0x02,r9 /* I/D indication is in cr6 */
- mtcrf 0x01,r9 /* slb_allocate uses cr0 and cr7 */
-
- RESTORE_CTR(r9, PACA_EXSLB)
- RESTORE_PPR_PACA(PACA_EXSLB, r9)
- mr r3,r12
- ld r9,PACA_EXSLB+EX_R9(r13)
- ld r10,PACA_EXSLB+EX_R10(r13)
- ld r11,PACA_EXSLB+EX_R11(r13)
- ld r12,PACA_EXSLB+EX_R12(r13)
- ld r13,PACA_EXSLB+EX_R13(r13)
- RFI_TO_KERNEL
- b . /* prevent speculative execution */
-
-
-2: std r3,PACA_EXSLB+EX_DAR(r13)
- mr r3,r12
- mfspr r11,SPRN_SRR0
- mfspr r12,SPRN_SRR1
- LOAD_HANDLER(r10,unrecov_slb)
- mtspr SPRN_SRR0,r10
- ld r10,PACAKMSR(r13)
- mtspr SPRN_SRR1,r10
- RFI_TO_KERNEL
- b .
-8: std r3,PACA_EXSLB+EX_DAR(r13)
- mr r3,r12
- mfspr r11,SPRN_SRR0
- mfspr r12,SPRN_SRR1
- LOAD_HANDLER(r10, large_addr_slb)
- mtspr SPRN_SRR0,r10
- ld r10,PACAKMSR(r13)
- mtspr SPRN_SRR1,r10
- RFI_TO_KERNEL
- b .
+TRAMP_KVM(PACA_EXSLB, 0x480)
-EXC_COMMON_BEGIN(unrecov_slb)
- EXCEPTION_PROLOG_COMMON(0x4100, PACA_EXSLB)
- RECONCILE_IRQ_STATE(r10, r11)
+EXC_COMMON_BEGIN(instruction_access_slb_common)
+ EXCEPTION_PROLOG_COMMON(0x480, PACA_EXSLB)
+ ld r4,_NIP(r1)
+ addi r3,r1,STACK_FRAME_OVERHEAD
+ bl do_slb_fault
+ cmpdi r3,0
+ bne- 1f
+ b fast_exception_return
+1: /* Error case */
bl save_nvgprs
-1: addi r3,r1,STACK_FRAME_OVERHEAD
- bl unrecoverable_exception
- b 1b
-
-EXC_COMMON_BEGIN(large_addr_slb)
- EXCEPTION_PROLOG_COMMON(0x380, PACA_EXSLB)
RECONCILE_IRQ_STATE(r10, r11)
- ld r3, PACA_EXSLB+EX_DAR(r13)
- std r3, _DAR(r1)
- beq cr6, 2f
- li r10, 0x481 /* fix trap number for I-SLB miss */
- std r10, _TRAP(r1)
-2: bl save_nvgprs
- addi r3, r1, STACK_FRAME_OVERHEAD
- bl slb_miss_large_addr
+ ld r4,_NIP(r1)
+ mr r5,r3
+ addi r3,r1,STACK_FRAME_OVERHEAD
+ bl do_bad_slb_fault
b ret_from_except
+
EXC_REAL_BEGIN(hardware_interrupt, 0x500, 0x100)
.globl hardware_interrupt_hv;
hardware_interrupt_hv:
diff --git a/arch/powerpc/mm/Makefile b/arch/powerpc/mm/Makefile
index cdf6a9960046..892d4e061d62 100644
--- a/arch/powerpc/mm/Makefile
+++ b/arch/powerpc/mm/Makefile
@@ -15,7 +15,7 @@ obj-$(CONFIG_PPC_MMU_NOHASH) += mmu_context_nohash.o tlb_nohash.o \
obj-$(CONFIG_PPC_BOOK3E) += tlb_low_$(BITS)e.o
hash64-$(CONFIG_PPC_NATIVE) := hash_native_64.o
obj-$(CONFIG_PPC_BOOK3E_64) += pgtable-book3e.o
-obj-$(CONFIG_PPC_BOOK3S_64) += pgtable-hash64.o hash_utils_64.o slb_low.o slb.o $(hash64-y) mmu_context_book3s64.o pgtable-book3s64.o
+obj-$(CONFIG_PPC_BOOK3S_64) += pgtable-hash64.o hash_utils_64.o slb.o $(hash64-y) mmu_context_book3s64.o pgtable-book3s64.o
obj-$(CONFIG_PPC_RADIX_MMU) += pgtable-radix.o tlb-radix.o
obj-$(CONFIG_PPC_STD_MMU_32) += ppc_mmu_32.o hash_low_32.o mmu_context_hash32.o
obj-$(CONFIG_PPC_STD_MMU) += tlb_hash$(BITS).o
diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c
index a3a6e04296cd..5643879fa2b1 100644
--- a/arch/powerpc/mm/slb.c
+++ b/arch/powerpc/mm/slb.c
@@ -34,7 +34,7 @@ enum slb_index {
KSTACK_INDEX = 2, /* Kernel stack map */
};
-extern void slb_allocate(unsigned long ea);
+static long slb_allocate_user(struct mm_struct *mm, unsigned long ea);
#define slb_esid_mask(ssize) \
(((ssize) == MMU_SEGSIZE_256M)? ESID_MASK: ESID_MASK_1T)
@@ -287,49 +287,19 @@ void switch_slb(struct task_struct *tsk, struct mm_struct *mm)
is_kernel_addr(exec_base))
return;
- slb_allocate(pc);
+ slb_allocate_user(mm, pc);
if (!esids_match(pc, stack))
- slb_allocate(stack);
+ slb_allocate_user(mm, stack);
if (!esids_match(pc, exec_base) &&
!esids_match(stack, exec_base))
- slb_allocate(exec_base);
+ slb_allocate_user(mm, exec_base);
}
-static inline void patch_slb_encoding(unsigned int *insn_addr,
- unsigned int immed)
-{
-
- /*
- * This function patches either an li or a cmpldi instruction with
- * a new immediate value. This relies on the fact that both li
- * (which is actually addi) and cmpldi both take a 16-bit immediate
- * value, and it is situated in the same location in the instruction,
- * ie. bits 16-31 (Big endian bit order) or the lower 16 bits.
- * The signedness of the immediate operand differs between the two
- * instructions however this code is only ever patching a small value,
- * much less than 1 << 15, so we can get away with it.
- * To patch the value we read the existing instruction, clear the
- * immediate value, and or in our new value, then write the instruction
- * back.
- */
- unsigned int insn = (*insn_addr & 0xffff0000) | immed;
- patch_instruction(insn_addr, insn);
-}
-
-extern u32 slb_miss_kernel_load_linear[];
-extern u32 slb_miss_kernel_load_io[];
-extern u32 slb_compare_rr_to_size[];
-extern u32 slb_miss_kernel_load_vmemmap[];
-
void slb_set_size(u16 size)
{
- if (mmu_slb_size == size)
- return;
-
mmu_slb_size = size;
- patch_slb_encoding(slb_compare_rr_to_size, mmu_slb_size);
}
void slb_initialize(void)
@@ -351,19 +321,9 @@ void slb_initialize(void)
#endif
if (!slb_encoding_inited) {
slb_encoding_inited = 1;
- patch_slb_encoding(slb_miss_kernel_load_linear,
- SLB_VSID_KERNEL | linear_llp);
- patch_slb_encoding(slb_miss_kernel_load_io,
- SLB_VSID_KERNEL | io_llp);
- patch_slb_encoding(slb_compare_rr_to_size,
- mmu_slb_size);
-
pr_devel("SLB: linear LLP = %04lx\n", linear_llp);
pr_devel("SLB: io LLP = %04lx\n", io_llp);
-
#ifdef CONFIG_SPARSEMEM_VMEMMAP
- patch_slb_encoding(slb_miss_kernel_load_vmemmap,
- SLB_VSID_KERNEL | vmemmap_llp);
pr_devel("SLB: vmemmap LLP = %04lx\n", vmemmap_llp);
#endif
}
@@ -394,49 +354,10 @@ void slb_initialize(void)
asm volatile("isync":::"memory");
}
-static void insert_slb_entry(unsigned long vsid, unsigned long ea,
- int bpsize, int ssize)
+static void slb_cache_update(unsigned long esid_data)
{
- unsigned long flags, vsid_data, esid_data;
- enum slb_index index;
int slb_cache_index;
- /*
- * We are irq disabled, hence should be safe to access PACA.
- */
- VM_WARN_ON(!irqs_disabled());
-
- /*
- * We can't take a PMU exception in the following code, so hard
- * disable interrupts.
- */
- hard_irq_disable();
-
- index = get_paca()->stab_rr;
-
- /*
- * simple round-robin replacement of slb starting at SLB_NUM_BOLTED.
- */
- if (index < (mmu_slb_size - 1))
- index++;
- else
- index = SLB_NUM_BOLTED;
-
- get_paca()->stab_rr = index;
-
- flags = SLB_VSID_USER | mmu_psize_defs[bpsize].sllp;
- vsid_data = (vsid << slb_vsid_shift(ssize)) | flags |
- ((unsigned long) ssize << SLB_VSID_SSIZE_SHIFT);
- esid_data = mk_esid_data(ea, ssize, index);
-
- /*
- * No need for an isync before or after this slbmte. The exception
- * we enter with and the rfid we exit with are context synchronizing.
- * Also we only handle user segments here.
- */
- asm volatile("slbmte %0, %1" : : "r" (vsid_data), "r" (esid_data)
- : "memory");
-
if (cpu_has_feature(CPU_FTR_ARCH_300))
return; /* ISAv3.0B and later does not use slb_cache */
@@ -461,58 +382,146 @@ static void insert_slb_entry(unsigned long vsid, unsigned long ea,
}
}
-static void handle_multi_context_slb_miss(int context_id, unsigned long ea)
+static enum slb_index alloc_slb_index(void)
+{
+ enum slb_index index;
+
+ /* round-robin replacement of slb starting at SLB_NUM_BOLTED. */
+ index = get_paca()->stab_rr;
+ if (index < (mmu_slb_size - 1))
+ index++;
+ else
+ index = SLB_NUM_BOLTED;
+ get_paca()->stab_rr = index;
+
+ return index;
+}
+
+static void slb_insert_entry(unsigned long ea, unsigned long context,
+ unsigned long flags, int ssize)
{
- struct mm_struct *mm = current->mm;
unsigned long vsid;
- int bpsize;
+ unsigned long vsid_data, esid_data;
+ enum slb_index index = alloc_slb_index();
+
+ vsid = get_vsid(context, ea, ssize);
+ vsid_data = (vsid << slb_vsid_shift(ssize)) | flags |
+ ((unsigned long) ssize << SLB_VSID_SSIZE_SHIFT);
+ esid_data = mk_esid_data(ea, ssize, index);
/*
- * We are always above 1TB, hence use high user segment size.
+ * No need for an isync before or after this slbmte. The exception
+ * we enter with and the rfid we exit with are context synchronizing.
+ * Also we only handle user segments here.
*/
- vsid = get_vsid(context_id, ea, mmu_highuser_ssize);
- bpsize = get_slice_psize(mm, ea);
- insert_slb_entry(vsid, ea, bpsize, mmu_highuser_ssize);
+ asm volatile("slbmte %0, %1" : : "r" (vsid_data), "r" (esid_data));
+
+ slb_cache_update(esid_data);
}
-void slb_miss_large_addr(struct pt_regs *regs)
+static long slb_allocate_kernel(unsigned long ea, unsigned long id)
{
- enum ctx_state prev_state = exception_enter();
- unsigned long ea = regs->dar;
- int context;
+ unsigned long context;
+ unsigned long flags;
+ int ssize;
- if (REGION_ID(ea) != USER_REGION_ID)
- goto slb_bad_addr;
+ if ((ea & ~REGION_MASK) >= (1ULL << MAX_EA_BITS_PER_CONTEXT))
+ return -EFAULT;
- /*
- * Are we beyound what the page table layout supports ?
- */
- if ((ea & ~REGION_MASK) >= H_PGTABLE_RANGE)
- goto slb_bad_addr;
+ if (id == KERNEL_REGION_ID) {
+ flags = SLB_VSID_KERNEL | mmu_psize_defs[mmu_linear_psize].sllp;
+#ifdef CONFIG_SPARSEMEM_VMEMMAP
+ } else if (id == VMEMMAP_REGION_ID) {
+ flags = SLB_VSID_KERNEL | mmu_psize_defs[mmu_vmemmap_psize].sllp;
+#endif
+ } else {
+ if (ea < H_VMALLOC_END)
+ flags = get_paca()->vmalloc_sllp;
+ else
+ flags = SLB_VSID_KERNEL | mmu_psize_defs[mmu_io_psize].sllp;
+ }
+ ssize = MMU_SEGSIZE_1T;
+ if (!mmu_has_feature(MMU_FTR_1T_SEGMENT))
+ ssize = MMU_SEGSIZE_256M;
+
+ context = id - KERNEL_REGION_CONTEXT_OFFSET;
- /* Lower address should have been handled by asm code */
- if (ea < (1UL << MAX_EA_BITS_PER_CONTEXT))
- goto slb_bad_addr;
+ slb_insert_entry(ea, context, flags, ssize);
+
+ return 0;
+}
+
+static long slb_allocate_user(struct mm_struct *mm, unsigned long ea)
+{
+ unsigned long context;
+ unsigned long flags;
+ int bpsize;
+ int ssize;
/*
* consider this as bad access if we take a SLB miss
* on an address above addr limit.
*/
- if (ea >= current->mm->context.slb_addr_limit)
- goto slb_bad_addr;
+ if (ea >= mm->context.slb_addr_limit)
+ return -EFAULT;
- context = get_ea_context(¤t->mm->context, ea);
+ context = get_ea_context(&mm->context, ea);
if (!context)
- goto slb_bad_addr;
+ return -EFAULT;
+
+ if (unlikely(ea >= H_PGTABLE_RANGE)) {
+ WARN_ON(1);
+ return -EFAULT;
+ }
+
+ ssize = user_segment_size(ea);
+
+ bpsize = get_slice_psize(mm, ea);
+ flags = SLB_VSID_USER | mmu_psize_defs[bpsize].sllp;
+
+ slb_insert_entry(ea, context, flags, ssize);
+
+ return 0;
+}
+
+long do_slb_fault(struct pt_regs *regs, unsigned long ea)
+{
+ unsigned long id = REGION_ID(ea);
+
+ /* IRQs are not reconciled here, so can't check irqs_disabled */
+ VM_WARN_ON(mfmsr() & MSR_EE);
+
+ /*
+ * SLB kernel faults must be very careful not to touch anything
+ * that is not bolted. E.g., PACA and global variables are okay,
+ * mm->context stuff is not.
+ *
+ * SLB user faults can access all of kernel memory, but must be
+ * careful not to touch things like IRQ state because it is not
+ * "reconciled" here. The difficulty is that we must use
+ * fast_exception_return to return from kernel SLB faults without
+ * looking at possible non-bolted memory. We could test user vs
+ * kernel faults in the interrupt handler asm and do a full fault,
+ * reconcile, ret_from_except for user faults which would make them
+ * first class kernel code. But for performance it's probably nicer
+ * if they go via fast_exception_return too.
+ */
+ if (id >= KERNEL_REGION_ID) {
+ return slb_allocate_kernel(ea, id);
+ } else {
+ struct mm_struct *mm = current->mm;
+
+ if (unlikely(!mm))
+ return -EFAULT;
- handle_multi_context_slb_miss(context, ea);
- exception_exit(prev_state);
- return;
+ return slb_allocate_user(mm, ea);
+ }
+}
-slb_bad_addr:
+void do_bad_slb_fault(struct pt_regs *regs, unsigned long ea, long err)
+{
if (user_mode(regs))
_exception(SIGSEGV, regs, SEGV_BNDERR, ea);
else
bad_page_fault(regs, ea, SIGSEGV);
- exception_exit(prev_state);
}
diff --git a/arch/powerpc/mm/slb_low.S b/arch/powerpc/mm/slb_low.S
deleted file mode 100644
index dbbb7a59eaf8..000000000000
--- a/arch/powerpc/mm/slb_low.S
+++ /dev/null
@@ -1,338 +0,0 @@
-/*
- * Low-level SLB routines
- *
- * Copyright (C) 2004 David Gibson <dwg@au.ibm.com>, IBM
- *
- * Based on earlier C version:
- * Dave Engebretsen and Mike Corrigan {engebret|mikejc}@us.ibm.com
- * Copyright (c) 2001 Dave Engebretsen
- * Copyright (C) 2002 Anton Blanchard <anton@au.ibm.com>, IBM
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * as published by the Free Software Foundation; either version
- * 2 of the License, or (at your option) any later version.
- */
-
-#include <asm/processor.h>
-#include <asm/ppc_asm.h>
-#include <asm/asm-offsets.h>
-#include <asm/cputable.h>
-#include <asm/page.h>
-#include <asm/mmu.h>
-#include <asm/pgtable.h>
-#include <asm/firmware.h>
-#include <asm/feature-fixups.h>
-
-/*
- * This macro generates asm code to compute the VSID scramble
- * function. Used in slb_allocate() and do_stab_bolted. The function
- * computed is: (protovsid*VSID_MULTIPLIER) % VSID_MODULUS
- *
- * rt = register containing the proto-VSID and into which the
- * VSID will be stored
- * rx = scratch register (clobbered)
- * rf = flags
- *
- * - rt and rx must be different registers
- * - The answer will end up in the low VSID_BITS bits of rt. The higher
- * bits may contain other garbage, so you may need to mask the
- * result.
- */
-#define ASM_VSID_SCRAMBLE(rt, rx, rf, size) \
- lis rx,VSID_MULTIPLIER_##size@h; \
- ori rx,rx,VSID_MULTIPLIER_##size@l; \
- mulld rt,rt,rx; /* rt = rt * MULTIPLIER */ \
-/* \
- * powermac get slb fault before feature fixup, so make 65 bit part \
- * the default part of feature fixup \
- */ \
-BEGIN_MMU_FTR_SECTION \
- srdi rx,rt,VSID_BITS_65_##size; \
- clrldi rt,rt,(64-VSID_BITS_65_##size); \
- add rt,rt,rx; \
- addi rx,rt,1; \
- srdi rx,rx,VSID_BITS_65_##size; \
- add rt,rt,rx; \
- rldimi rf,rt,SLB_VSID_SHIFT_##size,(64 - (SLB_VSID_SHIFT_##size + VSID_BITS_65_##size)); \
-MMU_FTR_SECTION_ELSE \
- srdi rx,rt,VSID_BITS_##size; \
- clrldi rt,rt,(64-VSID_BITS_##size); \
- add rt,rt,rx; /* add high and low bits */ \
- addi rx,rt,1; \
- srdi rx,rx,VSID_BITS_##size; /* extract 2^VSID_BITS bit */ \
- add rt,rt,rx; \
- rldimi rf,rt,SLB_VSID_SHIFT_##size,(64 - (SLB_VSID_SHIFT_##size + VSID_BITS_##size)); \
-ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_68_BIT_VA)
-
-
-/* void slb_allocate(unsigned long ea);
- *
- * Create an SLB entry for the given EA (user or kernel).
- * r3 = faulting address, r13 = PACA
- * r9, r10, r11 are clobbered by this function
- * r3 is preserved.
- * No other registers are examined or changed.
- */
-_GLOBAL(slb_allocate)
- /*
- * Check if the address falls within the range of the first context, or
- * if we may need to handle multi context. For the first context we
- * allocate the slb entry via the fast path below. For large address we
- * branch out to C-code and see if additional contexts have been
- * allocated.
- * The test here is:
- * (ea & ~REGION_MASK) >= (1ull << MAX_EA_BITS_PER_CONTEXT)
- */
- rldicr. r9,r3,4,(63 - MAX_EA_BITS_PER_CONTEXT - 4)
- bne- 8f
-
- srdi r9,r3,60 /* get region */
- srdi r10,r3,SID_SHIFT /* get esid */
- cmpldi cr7,r9,0xc /* cmp PAGE_OFFSET for later use */
-
- /* r3 = address, r10 = esid, cr7 = <> PAGE_OFFSET */
- blt cr7,0f /* user or kernel? */
-
- /* Check if hitting the linear mapping or some other kernel space
- */
- bne cr7,1f
-
- /* Linear mapping encoding bits, the "li" instruction below will
- * be patched by the kernel at boot
- */
-.globl slb_miss_kernel_load_linear
-slb_miss_kernel_load_linear:
- li r11,0
- /*
- * context = (ea >> 60) - (0xc - 1)
- * r9 = region id.
- */
- subi r9,r9,KERNEL_REGION_CONTEXT_OFFSET
-
-BEGIN_FTR_SECTION
- b .Lslb_finish_load
-END_MMU_FTR_SECTION_IFCLR(MMU_FTR_1T_SEGMENT)
- b .Lslb_finish_load_1T
-
-1:
-#ifdef CONFIG_SPARSEMEM_VMEMMAP
- cmpldi cr0,r9,0xf
- bne 1f
-/* Check virtual memmap region. To be patched at kernel boot */
-.globl slb_miss_kernel_load_vmemmap
-slb_miss_kernel_load_vmemmap:
- li r11,0
- b 6f
-1:
-#endif /* CONFIG_SPARSEMEM_VMEMMAP */
-
- /*
- * r10 contains the ESID, which is the original faulting EA shifted
- * right by 28 bits. We need to compare that with (H_VMALLOC_END >> 28)
- * which is 0xd00038000. That can't be used as an immediate, even if we
- * ignored the 0xd, so we have to load it into a register, and we only
- * have one register free. So we must load all of (H_VMALLOC_END >> 28)
- * into a register and compare ESID against that.
- */
- lis r11,(H_VMALLOC_END >> 32)@h // r11 = 0xffffffffd0000000
- ori r11,r11,(H_VMALLOC_END >> 32)@l // r11 = 0xffffffffd0003800
- // Rotate left 4, then mask with 0xffffffff0
- rldic r11,r11,4,28 // r11 = 0xd00038000
- cmpld r10,r11 // if r10 >= r11
- bge 5f // goto io_mapping
-
- /*
- * vmalloc mapping gets the encoding from the PACA as the mapping
- * can be demoted from 64K -> 4K dynamically on some machines.
- */
- lhz r11,PACAVMALLOCSLLP(r13)
- b 6f
-5:
- /* IO mapping */
-.globl slb_miss_kernel_load_io
-slb_miss_kernel_load_io:
- li r11,0
-6:
- /*
- * context = (ea >> 60) - (0xc - 1)
- * r9 = region id.
- */
- subi r9,r9,KERNEL_REGION_CONTEXT_OFFSET
-
-BEGIN_FTR_SECTION
- b .Lslb_finish_load
-END_MMU_FTR_SECTION_IFCLR(MMU_FTR_1T_SEGMENT)
- b .Lslb_finish_load_1T
-
-0: /*
- * For userspace addresses, make sure this is region 0.
- */
- cmpdi r9, 0
- bne- 8f
- /*
- * user space make sure we are within the allowed limit
- */
- ld r11,PACA_SLB_ADDR_LIMIT(r13)
- cmpld r3,r11
- bge- 8f
-
- /* when using slices, we extract the psize off the slice bitmaps
- * and then we need to get the sllp encoding off the mmu_psize_defs
- * array.
- *
- * XXX This is a bit inefficient especially for the normal case,
- * so we should try to implement a fast path for the standard page
- * size using the old sllp value so we avoid the array. We cannot
- * really do dynamic patching unfortunately as processes might flip
- * between 4k and 64k standard page size
- */
-#ifdef CONFIG_PPC_MM_SLICES
- /* r10 have esid */
- cmpldi r10,16
- /* below SLICE_LOW_TOP */
- blt 5f
- /*
- * Handle hpsizes,
- * r9 is get_paca()->context.high_slices_psize[index], r11 is mask_index
- */
- srdi r11,r10,(SLICE_HIGH_SHIFT - SLICE_LOW_SHIFT + 1) /* index */
- addi r9,r11,PACAHIGHSLICEPSIZE
- lbzx r9,r13,r9 /* r9 is hpsizes[r11] */
- /* r11 = (r10 >> (SLICE_HIGH_SHIFT - SLICE_LOW_SHIFT)) & 0x1 */
- rldicl r11,r10,(64 - (SLICE_HIGH_SHIFT - SLICE_LOW_SHIFT)),63
- b 6f
-
-5:
- /*
- * Handle lpsizes
- * r9 is get_paca()->context.low_slices_psize[index], r11 is mask_index
- */
- srdi r11,r10,1 /* index */
- addi r9,r11,PACALOWSLICESPSIZE
- lbzx r9,r13,r9 /* r9 is lpsizes[r11] */
- rldicl r11,r10,0,63 /* r11 = r10 & 0x1 */
-6:
- sldi r11,r11,2 /* index * 4 */
- /* Extract the psize and multiply to get an array offset */
- srd r9,r9,r11
- andi. r9,r9,0xf
- mulli r9,r9,MMUPSIZEDEFSIZE
-
- /* Now get to the array and obtain the sllp
- */
- ld r11,PACATOC(r13)
- ld r11,mmu_psize_defs@got(r11)
- add r11,r11,r9
- ld r11,MMUPSIZESLLP(r11)
- ori r11,r11,SLB_VSID_USER
-#else
- /* paca context sllp already contains the SLB_VSID_USER bits */
- lhz r11,PACACONTEXTSLLP(r13)
-#endif /* CONFIG_PPC_MM_SLICES */
-
- ld r9,PACACONTEXTID(r13)
-BEGIN_FTR_SECTION
- cmpldi r10,0x1000
- bge .Lslb_finish_load_1T
-END_MMU_FTR_SECTION_IFSET(MMU_FTR_1T_SEGMENT)
- b .Lslb_finish_load
-
-8: /* invalid EA - return an error indication */
- crset 4*cr0+eq /* indicate failure */
- blr
-
-/*
- * Finish loading of an SLB entry and return
- *
- * r3 = EA, r9 = context, r10 = ESID, r11 = flags, clobbers r9, cr7 = <> PAGE_OFFSET
- */
-.Lslb_finish_load:
- rldimi r10,r9,ESID_BITS,0
- ASM_VSID_SCRAMBLE(r10,r9,r11,256M)
- /* r3 = EA, r11 = VSID data */
- /*
- * Find a slot, round robin. Previously we tried to find a
- * free slot first but that took too long. Unfortunately we
- * dont have any LRU information to help us choose a slot.
- */
-
- mr r9,r3
-
- /* slb_finish_load_1T continues here. r9=EA with non-ESID bits clear */
-7: ld r10,PACASTABRR(r13)
- addi r10,r10,1
- /* This gets soft patched on boot. */
-.globl slb_compare_rr_to_size
-slb_compare_rr_to_size:
- cmpldi r10,0
-
- blt+ 4f
- li r10,SLB_NUM_BOLTED
-
-4:
- std r10,PACASTABRR(r13)
-
-3:
- rldimi r9,r10,0,36 /* r9 = EA[0:35] | entry */
- oris r10,r9,SLB_ESID_V@h /* r10 = r9 | SLB_ESID_V */
-
- /* r9 = ESID data, r11 = VSID data */
-
- /*
- * No need for an isync before or after this slbmte. The exception
- * we enter with and the rfid we exit with are context synchronizing.
- */
- slbmte r11,r10
-
- crclr 4*cr0+eq /* set result to "success" */
-BEGIN_FTR_SECTION
- blr /* No SLB cache, see switch_slb */
-FTR_SECTION_ELSE
- bgelr cr7 /* we're done for kernel addresses */
-ALT_FTR_SECTION_END_IFSET(CPU_FTR_ARCH_300)
-
- /* Update the slb cache for user addresses */
- lhz r9,PACASLBCACHEPTR(r13) /* offset = paca->slb_cache_ptr */
- cmpldi r9,SLB_CACHE_ENTRIES
- bge 1f
-
- /* still room in the slb cache */
- sldi r11,r9,2 /* r11 = offset * sizeof(u32) */
- srdi r10,r10,28 /* get the 36 bits of the ESID */
- add r11,r11,r13 /* r11 = (u32 *)paca + offset */
- stw r10,PACASLBCACHE(r11) /* paca->slb_cache[offset] = esid */
- addi r9,r9,1 /* offset++ */
- b 2f
-1: /* offset >= SLB_CACHE_ENTRIES */
- li r9,SLB_CACHE_ENTRIES+1
-2:
- sth r9,PACASLBCACHEPTR(r13) /* paca->slb_cache_ptr = offset */
- crclr 4*cr0+eq /* set result to "success" */
- blr
-
-/*
- * Finish loading of a 1T SLB entry (for the kernel linear mapping) and return.
- *
- * r3 = EA, r9 = context, r10 = ESID(256MB), r11 = flags, clobbers r9
- */
-.Lslb_finish_load_1T:
- srdi r10,r10,(SID_SHIFT_1T - SID_SHIFT) /* get 1T ESID */
- rldimi r10,r9,ESID_BITS_1T,0
- ASM_VSID_SCRAMBLE(r10,r9,r11,1T)
-
- li r10,MMU_SEGSIZE_1T
- rldimi r11,r10,SLB_VSID_SSIZE_SHIFT,0 /* insert segment size */
-
- /* r3 = EA, r11 = VSID data */
- clrrdi r9,r3,SID_SHIFT_1T /* clear out non-ESID bits */
- b 7b
-
-
-_ASM_NOKPROBE_SYMBOL(slb_allocate)
-_ASM_NOKPROBE_SYMBOL(slb_miss_kernel_load_linear)
-_ASM_NOKPROBE_SYMBOL(slb_miss_kernel_load_io)
-_ASM_NOKPROBE_SYMBOL(slb_compare_rr_to_size)
-#ifdef CONFIG_SPARSEMEM_VMEMMAP
-_ASM_NOKPROBE_SYMBOL(slb_miss_kernel_load_vmemmap)
-#endif
--
2.17.0
^ permalink raw reply related
* [RFC PATCH 2/5] powerpc/64s/hash: remove user SLB data from the paca
From: Nicholas Piggin @ 2018-08-20 9:41 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V
In-Reply-To: <20180820094200.13003-1-npiggin@gmail.com>
User SLB mappig data is copied into the PACA from the mm->context
so it can be accessed by the SLB miss handlers.
After the previous patch, SLB miss handlers now run with relocation
on, and user SLB misses are able to take recursive kernel SLB misses,
so the user SLB mapping data can be removed from the paca and
accessed directly.
---
arch/powerpc/include/asm/book3s/64/mmu-hash.h | 1 +
arch/powerpc/include/asm/paca.h | 13 ------
arch/powerpc/kernel/asm-offsets.c | 9 ----
arch/powerpc/kernel/paca.c | 21 ---------
arch/powerpc/mm/hash_utils_64.c | 46 +++++--------------
arch/powerpc/mm/mmu_context.c | 3 +-
arch/powerpc/mm/slb.c | 20 +++++++-
arch/powerpc/mm/slice.c | 29 ++++--------
8 files changed, 40 insertions(+), 102 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/64/mmu-hash.h b/arch/powerpc/include/asm/book3s/64/mmu-hash.h
index b3520b549cba..39764214aef5 100644
--- a/arch/powerpc/include/asm/book3s/64/mmu-hash.h
+++ b/arch/powerpc/include/asm/book3s/64/mmu-hash.h
@@ -496,6 +496,7 @@ static inline void hpte_init_pseries(void) { }
extern void hpte_init_native(void);
extern void slb_initialize(void);
+extern void core_flush_all_slbs(struct mm_struct *mm);
extern void slb_flush_and_rebolt(void);
void slb_flush_all_realmode(void);
void __slb_restore_bolted_realmode(void);
diff --git a/arch/powerpc/include/asm/paca.h b/arch/powerpc/include/asm/paca.h
index 4331295db0f7..8c258a057207 100644
--- a/arch/powerpc/include/asm/paca.h
+++ b/arch/powerpc/include/asm/paca.h
@@ -143,18 +143,6 @@ struct paca_struct {
struct tlb_core_data tcd;
#endif /* CONFIG_PPC_BOOK3E */
-#ifdef CONFIG_PPC_BOOK3S
- mm_context_id_t mm_ctx_id;
-#ifdef CONFIG_PPC_MM_SLICES
- unsigned char mm_ctx_low_slices_psize[BITS_PER_LONG / BITS_PER_BYTE];
- unsigned char mm_ctx_high_slices_psize[SLICE_ARRAY_SIZE];
- unsigned long mm_ctx_slb_addr_limit;
-#else
- u16 mm_ctx_user_psize;
- u16 mm_ctx_sllp;
-#endif
-#endif
-
/*
* then miscellaneous read-write fields
*/
@@ -256,7 +244,6 @@ struct paca_struct {
#endif /* CONFIG_PPC_PSERIES */
} ____cacheline_aligned;
-extern void copy_mm_to_paca(struct mm_struct *mm);
extern struct paca_struct **paca_ptrs;
extern void initialise_paca(struct paca_struct *new_paca, int cpu);
extern void setup_paca(struct paca_struct *new_paca);
diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c
index 7834256585f1..43b67ead5b97 100644
--- a/arch/powerpc/kernel/asm-offsets.c
+++ b/arch/powerpc/kernel/asm-offsets.c
@@ -181,15 +181,6 @@ int main(void)
OFFSET(PACAIRQSOFTMASK, paca_struct, irq_soft_mask);
OFFSET(PACAIRQHAPPENED, paca_struct, irq_happened);
OFFSET(PACA_FTRACE_ENABLED, paca_struct, ftrace_enabled);
-#ifdef CONFIG_PPC_BOOK3S
- OFFSET(PACACONTEXTID, paca_struct, mm_ctx_id);
-#ifdef CONFIG_PPC_MM_SLICES
- OFFSET(PACALOWSLICESPSIZE, paca_struct, mm_ctx_low_slices_psize);
- OFFSET(PACAHIGHSLICEPSIZE, paca_struct, mm_ctx_high_slices_psize);
- OFFSET(PACA_SLB_ADDR_LIMIT, paca_struct, mm_ctx_slb_addr_limit);
- DEFINE(MMUPSIZEDEFSIZE, sizeof(struct mmu_psize_def));
-#endif /* CONFIG_PPC_MM_SLICES */
-#endif
#ifdef CONFIG_PPC_BOOK3E
OFFSET(PACAPGD, paca_struct, pgd);
diff --git a/arch/powerpc/kernel/paca.c b/arch/powerpc/kernel/paca.c
index 0ee3e6d50f28..6752e17f0281 100644
--- a/arch/powerpc/kernel/paca.c
+++ b/arch/powerpc/kernel/paca.c
@@ -259,24 +259,3 @@ void __init free_unused_pacas(void)
paca_ptrs_size + paca_struct_size, nr_cpu_ids);
}
-void copy_mm_to_paca(struct mm_struct *mm)
-{
-#ifdef CONFIG_PPC_BOOK3S
- mm_context_t *context = &mm->context;
-
- get_paca()->mm_ctx_id = context->id;
-#ifdef CONFIG_PPC_MM_SLICES
- VM_BUG_ON(!mm->context.slb_addr_limit);
- get_paca()->mm_ctx_slb_addr_limit = mm->context.slb_addr_limit;
- memcpy(&get_paca()->mm_ctx_low_slices_psize,
- &context->low_slices_psize, sizeof(context->low_slices_psize));
- memcpy(&get_paca()->mm_ctx_high_slices_psize,
- &context->high_slices_psize, TASK_SLICE_ARRAY_SZ(mm));
-#else /* CONFIG_PPC_MM_SLICES */
- get_paca()->mm_ctx_user_psize = context->user_psize;
- get_paca()->mm_ctx_sllp = context->sllp;
-#endif
-#else /* !CONFIG_PPC_BOOK3S */
- return;
-#endif
-}
diff --git a/arch/powerpc/mm/hash_utils_64.c b/arch/powerpc/mm/hash_utils_64.c
index f23a89d8e4ce..88c95dc8b141 100644
--- a/arch/powerpc/mm/hash_utils_64.c
+++ b/arch/powerpc/mm/hash_utils_64.c
@@ -1088,16 +1088,16 @@ unsigned int hash_page_do_lazy_icache(unsigned int pp, pte_t pte, int trap)
}
#ifdef CONFIG_PPC_MM_SLICES
-static unsigned int get_paca_psize(unsigned long addr)
+static unsigned int get_psize(struct mm_struct *mm, unsigned long addr)
{
unsigned char *psizes;
unsigned long index, mask_index;
if (addr < SLICE_LOW_TOP) {
- psizes = get_paca()->mm_ctx_low_slices_psize;
+ psizes = mm->context.low_slices_psize;
index = GET_LOW_SLICE_INDEX(addr);
} else {
- psizes = get_paca()->mm_ctx_high_slices_psize;
+ psizes = mm->context.high_slices_psize;
index = GET_HIGH_SLICE_INDEX(addr);
}
mask_index = index & 0x1;
@@ -1105,9 +1105,9 @@ static unsigned int get_paca_psize(unsigned long addr)
}
#else
-unsigned int get_paca_psize(unsigned long addr)
+unsigned int get_psize(struct mm_struct *mm, unsigned long addr)
{
- return get_paca()->mm_ctx_user_psize;
+ return mm->context.user_psize;
}
#endif
@@ -1118,15 +1118,11 @@ unsigned int get_paca_psize(unsigned long addr)
#ifdef CONFIG_PPC_64K_PAGES
void demote_segment_4k(struct mm_struct *mm, unsigned long addr)
{
- if (get_slice_psize(mm, addr) == MMU_PAGE_4K)
+ if (get_psize(mm, addr) == MMU_PAGE_4K)
return;
slice_set_range_psize(mm, addr, 1, MMU_PAGE_4K);
copro_flush_all_slbs(mm);
- if ((get_paca_psize(addr) != MMU_PAGE_4K) && (current->mm == mm)) {
-
- copy_mm_to_paca(mm);
- slb_flush_and_rebolt();
- }
+ core_flush_all_slbs(mm);
}
#endif /* CONFIG_PPC_64K_PAGES */
@@ -1191,22 +1187,6 @@ void hash_failure_debug(unsigned long ea, unsigned long access,
trap, vsid, ssize, psize, lpsize, pte);
}
-static void check_paca_psize(unsigned long ea, struct mm_struct *mm,
- int psize, bool user_region)
-{
- if (user_region) {
- if (psize != get_paca_psize(ea)) {
- copy_mm_to_paca(mm);
- slb_flush_and_rebolt();
- }
- } else if (get_paca()->vmalloc_sllp !=
- mmu_psize_defs[mmu_vmalloc_psize].sllp) {
- get_paca()->vmalloc_sllp =
- mmu_psize_defs[mmu_vmalloc_psize].sllp;
- slb_vmalloc_update();
- }
-}
-
/* Result code is:
* 0 - handled
* 1 - normal page fault
@@ -1239,7 +1219,7 @@ int hash_page_mm(struct mm_struct *mm, unsigned long ea,
rc = 1;
goto bail;
}
- psize = get_slice_psize(mm, ea);
+ psize = get_psize(mm, ea);
ssize = user_segment_size(ea);
vsid = get_user_vsid(&mm->context, ea, ssize);
break;
@@ -1327,9 +1307,6 @@ int hash_page_mm(struct mm_struct *mm, unsigned long ea,
WARN_ON(1);
}
#endif
- if (current->mm == mm)
- check_paca_psize(ea, mm, psize, user_region);
-
goto bail;
}
@@ -1364,15 +1341,14 @@ int hash_page_mm(struct mm_struct *mm, unsigned long ea,
"to 4kB pages because of "
"non-cacheable mapping\n");
psize = mmu_vmalloc_psize = MMU_PAGE_4K;
+ slb_vmalloc_update();
copro_flush_all_slbs(mm);
+ core_flush_all_slbs(mm);
}
}
#endif /* CONFIG_PPC_64K_PAGES */
- if (current->mm == mm)
- check_paca_psize(ea, mm, psize, user_region);
-
#ifdef CONFIG_PPC_64K_PAGES
if (psize == MMU_PAGE_64K)
rc = __hash_page_64K(ea, access, vsid, ptep, trap,
@@ -1460,7 +1436,7 @@ int __hash_page(unsigned long ea, unsigned long msr, unsigned long trap,
#ifdef CONFIG_PPC_MM_SLICES
static bool should_hash_preload(struct mm_struct *mm, unsigned long ea)
{
- int psize = get_slice_psize(mm, ea);
+ int psize = get_psize(mm, ea);
/* We only prefault standard pages for now */
if (unlikely(psize != mm->context.user_psize))
diff --git a/arch/powerpc/mm/mmu_context.c b/arch/powerpc/mm/mmu_context.c
index f84e14f23e50..28ae2835db3d 100644
--- a/arch/powerpc/mm/mmu_context.c
+++ b/arch/powerpc/mm/mmu_context.c
@@ -54,8 +54,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,
* MMU context id, which is then moved to SPRN_PID.
*
* For the hash MMU it is either the first load from slb_cache
- * in switch_slb(), and/or the store of paca->mm_ctx_id in
- * copy_mm_to_paca().
+ * in switch_slb(), and/or load of MMU context id.
*
* On the other side, the barrier is in mm/tlb-radix.c for
* radix which orders earlier stores to clear the PTEs vs
diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c
index 5643879fa2b1..221d94b4f9cf 100644
--- a/arch/powerpc/mm/slb.c
+++ b/arch/powerpc/mm/slb.c
@@ -274,8 +274,6 @@ void switch_slb(struct task_struct *tsk, struct mm_struct *mm)
get_paca()->slb_cache_ptr = 0;
}
- copy_mm_to_paca(mm);
-
/*
* preload some userspace segments into the SLB.
* Almost all 32 and 64bit PowerPC executables are linked at
@@ -302,6 +300,24 @@ void slb_set_size(u16 size)
mmu_slb_size = size;
}
+static void cpu_flush_slb(void *parm)
+{
+ struct mm_struct *mm = parm;
+ unsigned long flags;
+
+ if (mm != current->active_mm)
+ return;
+
+ local_irq_save(flags);
+ slb_flush_and_rebolt();
+ local_irq_restore(flags);
+}
+
+void core_flush_all_slbs(struct mm_struct *mm)
+{
+ on_each_cpu(cpu_flush_slb, mm, 1);
+}
+
void slb_initialize(void)
{
unsigned long linear_llp, vmalloc_llp, io_llp;
diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c
index 205fe557ca10..606f424aac47 100644
--- a/arch/powerpc/mm/slice.c
+++ b/arch/powerpc/mm/slice.c
@@ -207,23 +207,6 @@ static bool slice_check_range_fits(struct mm_struct *mm,
return true;
}
-static void slice_flush_segments(void *parm)
-{
-#ifdef CONFIG_PPC64
- struct mm_struct *mm = parm;
- unsigned long flags;
-
- if (mm != current->active_mm)
- return;
-
- copy_mm_to_paca(current->active_mm);
-
- local_irq_save(flags);
- slb_flush_and_rebolt();
- local_irq_restore(flags);
-#endif
-}
-
static void slice_convert(struct mm_struct *mm,
const struct slice_mask *mask, int psize)
{
@@ -289,6 +272,9 @@ static void slice_convert(struct mm_struct *mm,
spin_unlock_irqrestore(&slice_convert_lock, flags);
copro_flush_all_slbs(mm);
+#ifdef CONFIG_PPC64
+ core_flush_all_slbs(mm);
+#endif
}
/*
@@ -502,8 +488,9 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
* be already initialised beyond the old address limit.
*/
mm->context.slb_addr_limit = high_limit;
-
- on_each_cpu(slice_flush_segments, mm, 1);
+#ifdef CONFIG_PPC64
+ core_flush_all_slbs(mm);
+#endif
}
/* Sanity checks */
@@ -665,8 +652,10 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len,
(SLICE_NUM_HIGH &&
!bitmap_empty(potential_mask.high_slices, SLICE_NUM_HIGH))) {
slice_convert(mm, &potential_mask, psize);
+#ifdef CONFIG_PPC64
if (psize > MMU_PAGE_BASE)
- on_each_cpu(slice_flush_segments, mm, 1);
+ core_flush_all_slbs(mm);
+#endif
}
return newaddr;
--
2.17.0
^ permalink raw reply related
* [RFC PATCH 3/5] powerpc/64s/hash: remove the first vmalloc segment from the bolted SLB
From: Nicholas Piggin @ 2018-08-20 9:41 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V
In-Reply-To: <20180820094200.13003-1-npiggin@gmail.com>
Remove the first vmalloc segment from bolted SLBEs. This is not
required to be bolted, and seems like it was added to help pre-load
the SLB on context switch. However there are now other segments like
the vmemmap segment that often take misses after a context switch, so
it is better to solve this a different way and save a bolted entry.
---
arch/powerpc/include/asm/book3s/64/mmu-hash.h | 2 +-
arch/powerpc/mm/slb.c | 16 ++++------------
2 files changed, 5 insertions(+), 13 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/64/mmu-hash.h b/arch/powerpc/include/asm/book3s/64/mmu-hash.h
index 39764214aef5..4c8d413ce99a 100644
--- a/arch/powerpc/include/asm/book3s/64/mmu-hash.h
+++ b/arch/powerpc/include/asm/book3s/64/mmu-hash.h
@@ -30,7 +30,7 @@
* SLB
*/
-#define SLB_NUM_BOLTED 3
+#define SLB_NUM_BOLTED 2
#define SLB_CACHE_ENTRIES 8
#define SLB_MIN_SIZE 32
diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c
index 221d94b4f9cf..6e595d75d997 100644
--- a/arch/powerpc/mm/slb.c
+++ b/arch/powerpc/mm/slb.c
@@ -133,13 +133,11 @@ static void __slb_flush_and_rebolt(void)
{
/* If you change this make sure you change SLB_NUM_BOLTED
* and PR KVM appropriately too. */
- unsigned long linear_llp, vmalloc_llp, lflags, vflags;
+ unsigned long linear_llp, lflags;
unsigned long ksp_esid_data, ksp_vsid_data;
linear_llp = mmu_psize_defs[mmu_linear_psize].sllp;
- vmalloc_llp = mmu_psize_defs[mmu_vmalloc_psize].sllp;
lflags = SLB_VSID_KERNEL | linear_llp;
- vflags = SLB_VSID_KERNEL | vmalloc_llp;
ksp_esid_data = mk_esid_data(get_paca()->kstack, mmu_kernel_ssize, KSTACK_INDEX);
if ((ksp_esid_data & ~0xfffffffUL) <= PAGE_OFFSET) {
@@ -157,14 +155,10 @@ static void __slb_flush_and_rebolt(void)
* the stack between the slbia and rebolting it. */
asm volatile("isync\n"
"slbia\n"
- /* Slot 1 - first VMALLOC segment */
+ /* Slot 1 - kernel stack */
"slbmte %0,%1\n"
- /* Slot 2 - kernel stack */
- "slbmte %2,%3\n"
"isync"
- :: "r"(mk_vsid_data(VMALLOC_START, mmu_kernel_ssize, vflags)),
- "r"(mk_esid_data(VMALLOC_START, mmu_kernel_ssize, VMALLOC_INDEX)),
- "r"(ksp_vsid_data),
+ :: "r"(ksp_vsid_data),
"r"(ksp_esid_data)
: "memory");
}
@@ -321,7 +315,7 @@ void core_flush_all_slbs(struct mm_struct *mm)
void slb_initialize(void)
{
unsigned long linear_llp, vmalloc_llp, io_llp;
- unsigned long lflags, vflags;
+ unsigned long lflags;
static int slb_encoding_inited;
#ifdef CONFIG_SPARSEMEM_VMEMMAP
unsigned long vmemmap_llp;
@@ -347,14 +341,12 @@ void slb_initialize(void)
get_paca()->stab_rr = SLB_NUM_BOLTED - 1;
lflags = SLB_VSID_KERNEL | linear_llp;
- vflags = SLB_VSID_KERNEL | vmalloc_llp;
/* Invalidate the entire SLB (even entry 0) & all the ERATS */
asm volatile("isync":::"memory");
asm volatile("slbmte %0,%0"::"r" (0) : "memory");
asm volatile("isync; slbia; isync":::"memory");
create_shadowed_slbe(PAGE_OFFSET, mmu_kernel_ssize, lflags, LINEAR_INDEX);
- create_shadowed_slbe(VMALLOC_START, mmu_kernel_ssize, vflags, VMALLOC_INDEX);
/* For the boot cpu, we're running on the stack in init_thread_union,
* which is in the first segment of the linear mapping, and also
--
2.17.0
^ permalink raw reply related
* [RFC PATCH 4/5] powerpc/64s/hash: Add SLB allocation bitmaps
From: Nicholas Piggin @ 2018-08-20 9:41 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V
In-Reply-To: <20180820094200.13003-1-npiggin@gmail.com>
Add 32-entry bitmaps to track the allocation status of the first 32
SLB entries, and whether they are user or kernel entries. These are
used to prevent context switches rolling the SLB round robin allocator
and evicting important kernel SLBEs when there are obvious free
entries.
---
arch/powerpc/include/asm/paca.h | 6 +++--
arch/powerpc/mm/slb.c | 42 +++++++++++++++++++++++----------
arch/powerpc/xmon/xmon.c | 2 +-
3 files changed, 35 insertions(+), 15 deletions(-)
diff --git a/arch/powerpc/include/asm/paca.h b/arch/powerpc/include/asm/paca.h
index 8c258a057207..bf7ab59be3b8 100644
--- a/arch/powerpc/include/asm/paca.h
+++ b/arch/powerpc/include/asm/paca.h
@@ -113,7 +113,10 @@ struct paca_struct {
* on the linear mapping */
/* SLB related definitions */
u16 vmalloc_sllp;
- u16 slb_cache_ptr;
+ u8 slb_cache_ptr;
+ u8 stab_rr; /* stab/slb round-robin counter */
+ u32 slb_used_bitmap; /* Bitmaps for first 32 SLB entries. */
+ u32 slb_kern_bitmap;
u32 slb_cache[SLB_CACHE_ENTRIES];
#endif /* CONFIG_PPC_BOOK3S_64 */
@@ -148,7 +151,6 @@ struct paca_struct {
*/
struct task_struct *__current; /* Pointer to current */
u64 kstack; /* Saved Kernel stack addr */
- u64 stab_rr; /* stab/slb round-robin counter */
u64 saved_r1; /* r1 save for RTAS calls or PM or EE=0 */
u64 saved_msr; /* MSR saved here by enter_rtas */
u16 trap_save; /* Used when bad stack is encountered */
diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c
index 6e595d75d997..3de63598f7c4 100644
--- a/arch/powerpc/mm/slb.c
+++ b/arch/powerpc/mm/slb.c
@@ -267,6 +267,7 @@ void switch_slb(struct task_struct *tsk, struct mm_struct *mm)
get_paca()->slb_cache_ptr = 0;
}
+ get_paca()->slb_used_bitmap = get_paca()->slb_kern_bitmap;
/*
* preload some userspace segments into the SLB.
@@ -339,6 +340,8 @@ void slb_initialize(void)
}
get_paca()->stab_rr = SLB_NUM_BOLTED - 1;
+ get_paca()->slb_kern_bitmap |= (1U << SLB_NUM_BOLTED) - 1;
+ get_paca()->slb_used_bitmap = get_paca()->slb_kern_bitmap;
lflags = SLB_VSID_KERNEL | linear_llp;
@@ -390,27 +393,42 @@ static void slb_cache_update(unsigned long esid_data)
}
}
-static enum slb_index alloc_slb_index(void)
+static enum slb_index alloc_slb_index(bool kernel)
{
enum slb_index index;
- /* round-robin replacement of slb starting at SLB_NUM_BOLTED. */
- index = get_paca()->stab_rr;
- if (index < (mmu_slb_size - 1))
- index++;
- else
- index = SLB_NUM_BOLTED;
- get_paca()->stab_rr = index;
+ /*
+ * SLBs beyond 32 entries are allocated with stab_rr only
+ * POWER7/8/9 have 32 SLB entries, this could be expanded if a
+ * future CPU has more.
+ */
+ if (get_paca()->slb_used_bitmap != U32_MAX) {
+ index = ffz(get_paca()->slb_used_bitmap);
+ get_paca()->slb_used_bitmap |= 1U << index;
+ if (kernel)
+ get_paca()->slb_kern_bitmap |= 1U << index;
+ } else {
+ /* round-robin replacement of slb starting at SLB_NUM_BOLTED. */
+ index = get_paca()->stab_rr;
+ if (index < (mmu_slb_size - 1))
+ index++;
+ else
+ index = SLB_NUM_BOLTED;
+ get_paca()->stab_rr = index;
+ if (kernel && index < 32)
+ get_paca()->slb_kern_bitmap |= 1U << index;
+ }
+ BUG_ON(index < SLB_NUM_BOLTED);
return index;
}
static void slb_insert_entry(unsigned long ea, unsigned long context,
- unsigned long flags, int ssize)
+ unsigned long flags, int ssize, bool kernel)
{
unsigned long vsid;
unsigned long vsid_data, esid_data;
- enum slb_index index = alloc_slb_index();
+ enum slb_index index = alloc_slb_index(kernel);
vsid = get_vsid(context, ea, ssize);
vsid_data = (vsid << slb_vsid_shift(ssize)) | flags |
@@ -454,7 +472,7 @@ static long slb_allocate_kernel(unsigned long ea, unsigned long id)
context = id - KERNEL_REGION_CONTEXT_OFFSET;
- slb_insert_entry(ea, context, flags, ssize);
+ slb_insert_entry(ea, context, flags, ssize, true);
return 0;
}
@@ -487,7 +505,7 @@ static long slb_allocate_user(struct mm_struct *mm, unsigned long ea)
bpsize = get_slice_psize(mm, ea);
flags = SLB_VSID_USER | mmu_psize_defs[bpsize].sllp;
- slb_insert_entry(ea, context, flags, ssize);
+ slb_insert_entry(ea, context, flags, ssize, false);
return 0;
}
diff --git a/arch/powerpc/xmon/xmon.c b/arch/powerpc/xmon/xmon.c
index 8c1d8b4bf6d5..44e16675c7f9 100644
--- a/arch/powerpc/xmon/xmon.c
+++ b/arch/powerpc/xmon/xmon.c
@@ -2415,7 +2415,7 @@ static void dump_one_paca(int cpu)
DUMP(p, __current, "%-*px");
DUMP(p, kstack, "%#-*llx");
printf(" %-*s = 0x%016llx\n", 25, "kstack_base", p->kstack & ~(THREAD_SIZE - 1));
- DUMP(p, stab_rr, "%#-*llx");
+ DUMP(p, stab_rr, "%#-*x");
DUMP(p, saved_r1, "%#-*llx");
DUMP(p, trap_save, "%#-*x");
DUMP(p, irq_soft_mask, "%#-*x");
--
2.17.0
^ permalink raw reply related
* [RFC PATCH 5/5] powerpc/64s/hash: Add a SLB preload cache
From: Nicholas Piggin @ 2018-08-20 9:42 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin, Aneesh Kumar K . V
In-Reply-To: <20180820094200.13003-1-npiggin@gmail.com>
When switching processes, currently all user SLBEs are cleared, and
a few (exec_base, pc, and stack) are preloaded. In trivial testing
with small apps, this tends to miss the heap and low 256MB segments,
and it will also miss commonly accessed segments on large memory
workloads.
Add a simple round-robin preload cache that just inserts the last
SLB miss into the head of the cache and preloads those at context
switch time.
Much more could go into this, including into the SLB entry reclaim
side to track some LRU information etc, which would require a study
of large memory workloads. But this is a simple thing we can do now
that is an obvious win for common workloads.
This plus the previous patch reduces SLB misses of a bare bones boot
to busybox from 945 to 180 when using 256MB segments, and 900 to 100 when
using 1T segments. These could almost all be eliminated by preloading
a bit more carefully with ELF binary loading.
---
arch/powerpc/include/asm/thread_info.h | 4 +
arch/powerpc/kernel/process.c | 6 ++
arch/powerpc/mm/mmu_context_book3s64.c | 10 ++-
arch/powerpc/mm/slb.c | 107 ++++++++++++++++++++-----
4 files changed, 102 insertions(+), 25 deletions(-)
diff --git a/arch/powerpc/include/asm/thread_info.h b/arch/powerpc/include/asm/thread_info.h
index 3c0002044bc9..ee5e49ec12c7 100644
--- a/arch/powerpc/include/asm/thread_info.h
+++ b/arch/powerpc/include/asm/thread_info.h
@@ -29,6 +29,7 @@
#include <asm/page.h>
#include <asm/accounting.h>
+#define SLB_PRELOAD_NR 8U
/*
* low level task data.
*/
@@ -44,6 +45,9 @@ struct thread_info {
#if defined(CONFIG_VIRT_CPU_ACCOUNTING_NATIVE) && defined(CONFIG_PPC32)
struct cpu_accounting_data accounting;
#endif
+ unsigned int slb_preload_nr;
+ unsigned long slb_preload_ea[SLB_PRELOAD_NR];
+
/* low level flags - has atomic operations done on it */
unsigned long flags ____cacheline_aligned_in_smp;
};
diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index 913c5725cdb2..678a2c668270 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -1710,6 +1710,8 @@ int copy_thread(unsigned long clone_flags, unsigned long usp,
return 0;
}
+void preload_new_slb_context(unsigned long start, unsigned long sp);
+
/*
* Set up a thread for executing a new program
*/
@@ -1717,6 +1719,10 @@ void start_thread(struct pt_regs *regs, unsigned long start, unsigned long sp)
{
#ifdef CONFIG_PPC64
unsigned long load_addr = regs->gpr[2]; /* saved by ELF_PLAT_INIT */
+
+#ifdef CONFIG_PPC_BOOK3S_64
+ preload_new_slb_context(start, sp);
+#endif
#endif
/*
diff --git a/arch/powerpc/mm/mmu_context_book3s64.c b/arch/powerpc/mm/mmu_context_book3s64.c
index 4a892d894a0f..3671a32141e2 100644
--- a/arch/powerpc/mm/mmu_context_book3s64.c
+++ b/arch/powerpc/mm/mmu_context_book3s64.c
@@ -85,7 +85,9 @@ int hash__alloc_context_id(void)
}
EXPORT_SYMBOL_GPL(hash__alloc_context_id);
-static int hash__init_new_context(struct mm_struct *mm)
+void init_new_slb_context(struct task_struct *tsk, struct mm_struct *mm);
+
+static int hash__init_new_context(struct task_struct *tsk, struct mm_struct *mm)
{
int index;
@@ -107,8 +109,10 @@ static int hash__init_new_context(struct mm_struct *mm)
* We should not be calling init_new_context() on init_mm. Hence a
* check against 0 is OK.
*/
- if (mm->context.id == 0)
+ if (mm->context.id == 0) {
slice_init_new_context_exec(mm);
+ init_new_slb_context(tsk, mm);
+ }
subpage_prot_init_new_context(mm);
@@ -152,7 +156,7 @@ int init_new_context(struct task_struct *tsk, struct mm_struct *mm)
if (radix_enabled())
index = radix__init_new_context(mm);
else
- index = hash__init_new_context(mm);
+ index = hash__init_new_context(tsk, mm);
if (index < 0)
return index;
diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c
index 3de63598f7c4..e53846d4e474 100644
--- a/arch/powerpc/mm/slb.c
+++ b/arch/powerpc/mm/slb.c
@@ -216,14 +216,85 @@ static inline int esids_match(unsigned long addr1, unsigned long addr2)
return (GET_ESID_1T(addr1) == GET_ESID_1T(addr2));
}
+static bool preload_hit(struct thread_info *ti, unsigned long ea)
+{
+ int i;
+
+ for (i = 0; i < min(SLB_PRELOAD_NR, ti->slb_preload_nr); i++)
+ if (esids_match(ti->slb_preload_ea[i], ea))
+ return true;
+ return false;
+}
+
+static bool preload_add(struct thread_info *ti, unsigned long ea)
+{
+ if (preload_hit(ti, ea))
+ return false;
+
+ ti->slb_preload_ea[ti->slb_preload_nr % SLB_PRELOAD_NR] = ea;
+ ti->slb_preload_nr++;
+
+ return true;
+}
+
+void preload_new_slb_context(unsigned long start, unsigned long sp)
+{
+ struct thread_info *ti = current_thread_info();
+ struct mm_struct *mm = current->mm;
+ unsigned long heap = mm->start_brk;
+
+ // printk("preload new slb context tsk:%s pc:%lx heap:%lx stack:%lx\n", current->comm, start, heap, sp);
+
+ if (!is_kernel_addr(start)) {
+ if (preload_add(ti, start))
+ slb_allocate_user(mm, start);
+ }
+
+ if (!is_kernel_addr(sp)) {
+ if (preload_add(ti, sp))
+ slb_allocate_user(mm, sp);
+ }
+
+ if (heap && !is_kernel_addr(heap)) {
+ if (preload_add(ti, heap))
+ slb_allocate_user(mm, heap);
+ }
+
+ // preload mm->mmap_base is too late at this point
+}
+
+void init_new_slb_context(struct task_struct *tsk, struct mm_struct *mm)
+{
+ struct thread_info *ti = task_thread_info(tsk);
+
+ // printk("init new slb context tsk:%s\n", tsk->comm);
+
+ /*
+ * Clear out previous cache because new exec will put addresses
+ * at different places. Preloading is still generally a win here
+ * because we don't have all ELF information yet and take several
+ * misses in kernel space on the user addresss when loading binary
+ * and libraries. However we don't want to insert more slb entries
+ * than we need for small processes. Probably should look at aging
+ * out the preload cache slowly at context switch time.
+ */
+ ti->slb_preload_nr = 0;
+
+ /*
+ * preload some userspace segments into the SLB.
+ * Almost all 32 and 64bit PowerPC executables are linked at
+ * 0x10000000 so it makes sense to preload this segment.
+ */
+ preload_add(ti, 0x10000000);
+}
+
/* Flush all user entries from the segment table of the current processor. */
void switch_slb(struct task_struct *tsk, struct mm_struct *mm)
{
unsigned long offset;
unsigned long slbie_data = 0;
- unsigned long pc = KSTK_EIP(tsk);
- unsigned long stack = KSTK_ESP(tsk);
- unsigned long exec_base;
+ struct thread_info *ti = task_thread_info(tsk);
+ int i;
/*
* We need interrupts hard-disabled here, not just soft-disabled,
@@ -269,25 +340,12 @@ void switch_slb(struct task_struct *tsk, struct mm_struct *mm)
}
get_paca()->slb_used_bitmap = get_paca()->slb_kern_bitmap;
- /*
- * preload some userspace segments into the SLB.
- * Almost all 32 and 64bit PowerPC executables are linked at
- * 0x10000000 so it makes sense to preload this segment.
+ /* XXX: should we gradually age out SLBs after a number of context
+ * switches to reduce reload overhead of unused entries (like we do
+ * with FP/VEC reload)?
*/
- exec_base = 0x10000000;
-
- if (is_kernel_addr(pc) || is_kernel_addr(stack) ||
- is_kernel_addr(exec_base))
- return;
-
- slb_allocate_user(mm, pc);
-
- if (!esids_match(pc, stack))
- slb_allocate_user(mm, stack);
-
- if (!esids_match(pc, exec_base) &&
- !esids_match(stack, exec_base))
- slb_allocate_user(mm, exec_base);
+ for (i = 0; i < min(SLB_PRELOAD_NR, ti->slb_preload_nr); i++)
+ slb_allocate_user(mm, ti->slb_preload_ea[i]);
}
void slb_set_size(u16 size)
@@ -536,11 +594,16 @@ long do_slb_fault(struct pt_regs *regs, unsigned long ea)
return slb_allocate_kernel(ea, id);
} else {
struct mm_struct *mm = current->mm;
+ long err;
if (unlikely(!mm))
return -EFAULT;
- return slb_allocate_user(mm, ea);
+ err = slb_allocate_user(mm, ea);
+ if (!err)
+ preload_add(current_thread_info(), ea);
+
+ return err;
}
}
--
2.17.0
^ permalink raw reply related
* Re: [RFC PATCH 1/5] powerpc/64s/hash: convert SLB miss handlers to C
From: Nicholas Piggin @ 2018-08-20 10:08 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Aneesh Kumar K . V
In-Reply-To: <20180820094200.13003-2-npiggin@gmail.com>
On Mon, 20 Aug 2018 19:41:56 +1000
Nicholas Piggin <npiggin@gmail.com> wrote:
> +long do_slb_fault(struct pt_regs *regs, unsigned long ea)
> +{
> + unsigned long id = REGION_ID(ea);
> +
> + /* IRQs are not reconciled here, so can't check irqs_disabled */
> + VM_WARN_ON(mfmsr() & MSR_EE);
> +
> + /*
> + * SLB kernel faults must be very careful not to touch anything
> + * that is not bolted. E.g., PACA and global variables are okay,
> + * mm->context stuff is not.
> + *
> + * SLB user faults can access all of kernel memory, but must be
> + * careful not to touch things like IRQ state because it is not
> + * "reconciled" here. The difficulty is that we must use
> + * fast_exception_return to return from kernel SLB faults without
> + * looking at possible non-bolted memory. We could test user vs
> + * kernel faults in the interrupt handler asm and do a full fault,
> + * reconcile, ret_from_except for user faults which would make them
> + * first class kernel code. But for performance it's probably nicer
> + * if they go via fast_exception_return too.
> + */
> + if (id >= KERNEL_REGION_ID) {
> + return slb_allocate_kernel(ea, id);
> + } else {
> + struct mm_struct *mm = current->mm;
> +
> + if (unlikely(!mm))
> + return -EFAULT;
>
> - handle_multi_context_slb_miss(context, ea);
> - exception_exit(prev_state);
> - return;
> + return slb_allocate_user(mm, ea);
> + }
> +}
>
> -slb_bad_addr:
> +void do_bad_slb_fault(struct pt_regs *regs, unsigned long ea, long err)
> +{
> if (user_mode(regs))
> _exception(SIGSEGV, regs, SEGV_BNDERR, ea);
> else
> bad_page_fault(regs, ea, SIGSEGV);
> - exception_exit(prev_state);
> }
I knew I forgot something -- forgot to test MSR[RI] here. That can be
done just by returning a different error from do_slb_fault if RI is
clear, and do_bad_slb_fault will call unrecoverable_exception() if it
sees that code.
Thanks,
Nick
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox