* [PATCH 00/13] MIPS: c-r4k: SMP cache management fixes
2016-07-13 13:12 [PATCH 00/13] MIPS: c-r4k: SMP cache management fixes James Hogan
@ 2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` [PATCH 01/13] MIPS: SMP: Clear ASID without confusing has_valid_asid() James Hogan
` (12 subsequent siblings)
13 siblings, 0 replies; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle
Cc: James Hogan, Paul Burton, Leonid Yegoshin, Felix Fietkau,
David Daney, Florian Fainelli, Hongliang Tao, Huacai Chen,
Hua Yan, Jayachandran C., Kevin Cernekee, linux-mips
These patches collect together various SMP cache management fixes &
refactorings, particularly to fix the missing SMP calls for indexed
cache ops on MIPS Coherence Manager (CM) based SMP systems since commit
cccf34e9411c ("MIPS: c-r4k: Fix cache flushing for MT cores").
Patches 1-6 fix various issues in the SMP code and cache management code
(all in relation to cache management).
Patches 7-12 then prepare r4k_on_each_cpu() and its various
callers/callees to distinguish hit/index cache ops so SMP calls can be
avoided.
Finally patch 13 makes the change for CM so that indexed cache ops are
performed on each core via SMP calls.
Patch 9 should also work around the SMP 34Kc regression Felix saw
recently, but __flush_anon_page() and r4k_flush_data_cache_page()
definitely still need looking at too.
All feedback & testing gratefully received.
James Hogan (13):
MIPS: SMP: Clear ASID without confusing has_valid_asid()
MIPS: SMP: Update cpu_foreign_map on CPU disable
MIPS: SMP: Drop stop_this_cpu() cpu_foreign_map hack
MIPS: c-r4k: Fix protected_writeback_scache_line for EVA
MIPS: c-r4k: Fix sigtramp SMP call to use kmap
MIPS: c-r4k: Avoid dcache flush for sigtramps
MIPS: c-r4k: Add r4k_on_each_cpu cache op type arg
MIPS: c-r4k: Fix valid ASID optimisation
MIPS: c-r4k: Exclude sibling CPUs in SMP calls
MIPS: c-r4k: Split r4k_flush_kernel_vmap_range()
MIPS: c-r4k: Local flush_icache_range cache op override
MIPS: c-r4k: Avoid small flush_icache_range SMP calls
MIPS: c-r4k: Use SMP calls for CM indexed cache ops
arch/mips/cavium-octeon/smp.c | 1 +
arch/mips/include/asm/r4kcache.h | 4 +
arch/mips/include/asm/smp.h | 4 +-
arch/mips/kernel/smp-bmips.c | 1 +
arch/mips/kernel/smp-cps.c | 1 +
arch/mips/kernel/smp.c | 34 +++--
arch/mips/loongson64/loongson-3/smp.c | 1 +
arch/mips/mm/c-r4k.c | 274 ++++++++++++++++++++++++++++------
8 files changed, 257 insertions(+), 63 deletions(-)
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: Felix Fietkau <nbd@nbd.name>
Cc: David Daney <david.daney@cavium.com>
Cc: Florian Fainelli <f.fainelli@gmail.com>
Cc: Hongliang Tao <taohl@lemote.com>
Cc: Huacai Chen <chenhc@lemote.com>
Cc: Hua Yan <yanh@lemote.com>
Cc: Jayachandran C. <jchandra@broadcom.com>
Cc: Kevin Cernekee <cernekee@gmail.com>
Cc: linux-mips@linux-mips.org
--
2.4.10
^ permalink raw reply [flat|nested] 28+ messages in thread* [PATCH 01/13] MIPS: SMP: Clear ASID without confusing has_valid_asid()
2016-07-13 13:12 [PATCH 00/13] MIPS: c-r4k: SMP cache management fixes James Hogan
2016-07-13 13:12 ` James Hogan
@ 2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` [PATCH 02/13] MIPS: SMP: Update cpu_foreign_map on CPU disable James Hogan
` (11 subsequent siblings)
13 siblings, 1 reply; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, Paul Burton, Leonid Yegoshin, linux-mips
The SMP flush_tlb_*() functions may clear the memory map's ASIDs for
other CPUs if the mm has only a single user (the current CPU) in order
to avoid SMP calls. However this makes it appear to has_valid_asid(),
which is used by various cache flush functions, as if the CPUs have
never run in the mm, and therefore can't have cached any of its memory.
For flush_tlb_mm() this doesn't sound unreasonable.
flush_tlb_range() corresponds to flush_cache_range() which does do full
indexed cache flushes, but only on the icache if the specified mapping
is executable, otherwise it doesn't guarantee that there are no cache
contents left for the mm.
flush_tlb_page() corresponds to flush_cache_page(), which will perform
address based cache ops on the specified page only, and also only
touches the icache if the page is executable. It does not guarantee that
there are no cache contents left for the mm.
For example, this affects flush_cache_range() which uses the
has_valid_asid() optimisation. It is required to flush the icache when
mappings are made executable (e.g. using mprotect) so they are
immediately usable. If some code is changed to non executable in order
to be modified then it will not be flushed from the icache during that
time, but the ASID on other CPUs may still be cleared for TLB flushing.
When the code is changed back to executable, flush_cache_range() will
assume the code hasn't run on those other CPUs due to the zero ASID, and
won't invalidate the icache on them.
This is fixed by clearing the other CPUs ASIDs to 1 instead of 0 for the
above two flush_tlb_*() functions when the corresponding cache flushes
are likely to be incomplete (non executable range flush, or any page
flush). This ASID appears valid to has_valid_asid(), but still triggers
ASID regeneration due to the upper ASID version bits being 0, which is
less than the minimum ASID version of 1 and so always treated as stale.
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/kernel/smp.c | 17 +++++++++++++++--
arch/mips/mm/c-r4k.c | 4 ++++
2 files changed, 19 insertions(+), 2 deletions(-)
diff --git a/arch/mips/kernel/smp.c b/arch/mips/kernel/smp.c
index f9d01e953acb..0c98b4a313be 100644
--- a/arch/mips/kernel/smp.c
+++ b/arch/mips/kernel/smp.c
@@ -512,10 +512,17 @@ void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned l
smp_on_other_tlbs(flush_tlb_range_ipi, &fd);
} else {
unsigned int cpu;
+ int exec = vma->vm_flags & VM_EXEC;
for_each_online_cpu(cpu) {
+ /*
+ * flush_cache_range() will only fully flush icache if
+ * the VMA is executable, otherwise we must invalidate
+ * ASID without it appearing to has_valid_asid() as if
+ * mm has been completely unused by that CPU.
+ */
if (cpu != smp_processor_id() && cpu_context(cpu, mm))
- cpu_context(cpu, mm) = 0;
+ cpu_context(cpu, mm) = !exec;
}
}
local_flush_tlb_range(vma, start, end);
@@ -560,8 +567,14 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long page)
unsigned int cpu;
for_each_online_cpu(cpu) {
+ /*
+ * flush_cache_page() only does partial flushes, so
+ * invalidate ASID without it appearing to
+ * has_valid_asid() as if mm has been completely unused
+ * by that CPU.
+ */
if (cpu != smp_processor_id() && cpu_context(cpu, vma->vm_mm))
- cpu_context(cpu, vma->vm_mm) = 0;
+ cpu_context(cpu, vma->vm_mm) = 1;
}
}
local_flush_tlb_page(vma, page);
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 645c69c95c9c..9204d4e4f02f 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -484,6 +484,10 @@ static void r4k__flush_cache_vunmap(void)
r4k_blast_dcache();
}
+/*
+ * Note: flush_tlb_range() assumes flush_cache_range() sufficiently flushes
+ * whole caches when vma is executable.
+ */
static inline void local_r4k_flush_cache_range(void * args)
{
struct vm_area_struct *vma = args;
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH 01/13] MIPS: SMP: Clear ASID without confusing has_valid_asid()
2016-07-13 13:12 ` [PATCH 01/13] MIPS: SMP: Clear ASID without confusing has_valid_asid() James Hogan
@ 2016-07-13 13:12 ` James Hogan
0 siblings, 0 replies; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, Paul Burton, Leonid Yegoshin, linux-mips
The SMP flush_tlb_*() functions may clear the memory map's ASIDs for
other CPUs if the mm has only a single user (the current CPU) in order
to avoid SMP calls. However this makes it appear to has_valid_asid(),
which is used by various cache flush functions, as if the CPUs have
never run in the mm, and therefore can't have cached any of its memory.
For flush_tlb_mm() this doesn't sound unreasonable.
flush_tlb_range() corresponds to flush_cache_range() which does do full
indexed cache flushes, but only on the icache if the specified mapping
is executable, otherwise it doesn't guarantee that there are no cache
contents left for the mm.
flush_tlb_page() corresponds to flush_cache_page(), which will perform
address based cache ops on the specified page only, and also only
touches the icache if the page is executable. It does not guarantee that
there are no cache contents left for the mm.
For example, this affects flush_cache_range() which uses the
has_valid_asid() optimisation. It is required to flush the icache when
mappings are made executable (e.g. using mprotect) so they are
immediately usable. If some code is changed to non executable in order
to be modified then it will not be flushed from the icache during that
time, but the ASID on other CPUs may still be cleared for TLB flushing.
When the code is changed back to executable, flush_cache_range() will
assume the code hasn't run on those other CPUs due to the zero ASID, and
won't invalidate the icache on them.
This is fixed by clearing the other CPUs ASIDs to 1 instead of 0 for the
above two flush_tlb_*() functions when the corresponding cache flushes
are likely to be incomplete (non executable range flush, or any page
flush). This ASID appears valid to has_valid_asid(), but still triggers
ASID regeneration due to the upper ASID version bits being 0, which is
less than the minimum ASID version of 1 and so always treated as stale.
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/kernel/smp.c | 17 +++++++++++++++--
arch/mips/mm/c-r4k.c | 4 ++++
2 files changed, 19 insertions(+), 2 deletions(-)
diff --git a/arch/mips/kernel/smp.c b/arch/mips/kernel/smp.c
index f9d01e953acb..0c98b4a313be 100644
--- a/arch/mips/kernel/smp.c
+++ b/arch/mips/kernel/smp.c
@@ -512,10 +512,17 @@ void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned l
smp_on_other_tlbs(flush_tlb_range_ipi, &fd);
} else {
unsigned int cpu;
+ int exec = vma->vm_flags & VM_EXEC;
for_each_online_cpu(cpu) {
+ /*
+ * flush_cache_range() will only fully flush icache if
+ * the VMA is executable, otherwise we must invalidate
+ * ASID without it appearing to has_valid_asid() as if
+ * mm has been completely unused by that CPU.
+ */
if (cpu != smp_processor_id() && cpu_context(cpu, mm))
- cpu_context(cpu, mm) = 0;
+ cpu_context(cpu, mm) = !exec;
}
}
local_flush_tlb_range(vma, start, end);
@@ -560,8 +567,14 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long page)
unsigned int cpu;
for_each_online_cpu(cpu) {
+ /*
+ * flush_cache_page() only does partial flushes, so
+ * invalidate ASID without it appearing to
+ * has_valid_asid() as if mm has been completely unused
+ * by that CPU.
+ */
if (cpu != smp_processor_id() && cpu_context(cpu, vma->vm_mm))
- cpu_context(cpu, vma->vm_mm) = 0;
+ cpu_context(cpu, vma->vm_mm) = 1;
}
}
local_flush_tlb_page(vma, page);
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 645c69c95c9c..9204d4e4f02f 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -484,6 +484,10 @@ static void r4k__flush_cache_vunmap(void)
r4k_blast_dcache();
}
+/*
+ * Note: flush_tlb_range() assumes flush_cache_range() sufficiently flushes
+ * whole caches when vma is executable.
+ */
static inline void local_r4k_flush_cache_range(void * args)
{
struct vm_area_struct *vma = args;
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 02/13] MIPS: SMP: Update cpu_foreign_map on CPU disable
2016-07-13 13:12 [PATCH 00/13] MIPS: c-r4k: SMP cache management fixes James Hogan
2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` [PATCH 01/13] MIPS: SMP: Clear ASID without confusing has_valid_asid() James Hogan
@ 2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` [PATCH 03/13] MIPS: SMP: Drop stop_this_cpu() cpu_foreign_map hack James Hogan
` (10 subsequent siblings)
13 siblings, 1 reply; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle
Cc: James Hogan, Paul Burton, David Daney, Kevin Cernekee,
Florian Fainelli, Huacai Chen, Hongliang Tao, Hua Yan, linux-mips
When a CPU is disabled via CPU hotplug, cpu_foreign_map is not updated.
This could result in cache management SMP calls being sent to offline
CPUs instead of online siblings in the same core.
Add a call to calculate_cpu_foreign_map() in the various MIPS cpu
disable callbacks after set_cpu_online(). All cases are updated for
consistency and to keep cpu_foreign_map strictly up to date, not just
those which may support hardware multithreading.
Fixes: cccf34e9411c ("MIPS: c-r4k: Fix cache flushing for MT cores")
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: David Daney <david.daney@cavium.com>
Cc: Kevin Cernekee <cernekee@gmail.com>
Cc: Florian Fainelli <f.fainelli@gmail.com>
Cc: Huacai Chen <chenhc@lemote.com>
Cc: Hongliang Tao <taohl@lemote.com>
Cc: Hua Yan <yanh@lemote.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/cavium-octeon/smp.c | 1 +
arch/mips/include/asm/smp.h | 2 ++
arch/mips/kernel/smp-bmips.c | 1 +
arch/mips/kernel/smp-cps.c | 1 +
arch/mips/kernel/smp.c | 2 +-
arch/mips/loongson64/loongson-3/smp.c | 1 +
6 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/arch/mips/cavium-octeon/smp.c b/arch/mips/cavium-octeon/smp.c
index 33aab89259f3..4d457d602d3b 100644
--- a/arch/mips/cavium-octeon/smp.c
+++ b/arch/mips/cavium-octeon/smp.c
@@ -271,6 +271,7 @@ static int octeon_cpu_disable(void)
return -ENOTSUPP;
set_cpu_online(cpu, false);
+ calculate_cpu_foreign_map();
cpumask_clear_cpu(cpu, &cpu_callin_map);
octeon_fixup_irqs();
diff --git a/arch/mips/include/asm/smp.h b/arch/mips/include/asm/smp.h
index 03722d4326a1..0c534a03bb36 100644
--- a/arch/mips/include/asm/smp.h
+++ b/arch/mips/include/asm/smp.h
@@ -53,6 +53,8 @@ extern cpumask_t cpu_coherent_mask;
extern void asmlinkage smp_bootstrap(void);
+extern void calculate_cpu_foreign_map(void);
+
/*
* this function sends a 'reschedule' IPI to another CPU.
* it goes straight through and wastes no time serializing
diff --git a/arch/mips/kernel/smp-bmips.c b/arch/mips/kernel/smp-bmips.c
index e02addc0307f..6d0f1321e084 100644
--- a/arch/mips/kernel/smp-bmips.c
+++ b/arch/mips/kernel/smp-bmips.c
@@ -363,6 +363,7 @@ static int bmips_cpu_disable(void)
pr_info("SMP: CPU%d is offline\n", cpu);
set_cpu_online(cpu, false);
+ calculate_cpu_foreign_map();
cpumask_clear_cpu(cpu, &cpu_callin_map);
clear_c0_status(IE_IRQ5);
diff --git a/arch/mips/kernel/smp-cps.c b/arch/mips/kernel/smp-cps.c
index 4ed36f288d64..47ff5108bed5 100644
--- a/arch/mips/kernel/smp-cps.c
+++ b/arch/mips/kernel/smp-cps.c
@@ -397,6 +397,7 @@ static int cps_cpu_disable(void)
atomic_sub(1 << cpu_vpe_id(¤t_cpu_data), &core_cfg->vpe_mask);
smp_mb__after_atomic();
set_cpu_online(cpu, false);
+ calculate_cpu_foreign_map();
cpumask_clear_cpu(cpu, &cpu_callin_map);
return 0;
diff --git a/arch/mips/kernel/smp.c b/arch/mips/kernel/smp.c
index 0c98b4a313be..a4d4309ecff2 100644
--- a/arch/mips/kernel/smp.c
+++ b/arch/mips/kernel/smp.c
@@ -124,7 +124,7 @@ static inline void set_cpu_core_map(int cpu)
* Calculate a new cpu_foreign_map mask whenever a
* new cpu appears or disappears.
*/
-static inline void calculate_cpu_foreign_map(void)
+void calculate_cpu_foreign_map(void)
{
int i, k, core_present;
cpumask_t temp_foreign_map;
diff --git a/arch/mips/loongson64/loongson-3/smp.c b/arch/mips/loongson64/loongson-3/smp.c
index e59759af63d9..2fec6f753a35 100644
--- a/arch/mips/loongson64/loongson-3/smp.c
+++ b/arch/mips/loongson64/loongson-3/smp.c
@@ -417,6 +417,7 @@ static int loongson3_cpu_disable(void)
return -EBUSY;
set_cpu_online(cpu, false);
+ calculate_cpu_foreign_map();
cpumask_clear_cpu(cpu, &cpu_callin_map);
local_irq_save(flags);
fixup_irqs();
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH 02/13] MIPS: SMP: Update cpu_foreign_map on CPU disable
2016-07-13 13:12 ` [PATCH 02/13] MIPS: SMP: Update cpu_foreign_map on CPU disable James Hogan
@ 2016-07-13 13:12 ` James Hogan
0 siblings, 0 replies; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle
Cc: James Hogan, Paul Burton, David Daney, Kevin Cernekee,
Florian Fainelli, Huacai Chen, Hongliang Tao, Hua Yan, linux-mips
When a CPU is disabled via CPU hotplug, cpu_foreign_map is not updated.
This could result in cache management SMP calls being sent to offline
CPUs instead of online siblings in the same core.
Add a call to calculate_cpu_foreign_map() in the various MIPS cpu
disable callbacks after set_cpu_online(). All cases are updated for
consistency and to keep cpu_foreign_map strictly up to date, not just
those which may support hardware multithreading.
Fixes: cccf34e9411c ("MIPS: c-r4k: Fix cache flushing for MT cores")
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: David Daney <david.daney@cavium.com>
Cc: Kevin Cernekee <cernekee@gmail.com>
Cc: Florian Fainelli <f.fainelli@gmail.com>
Cc: Huacai Chen <chenhc@lemote.com>
Cc: Hongliang Tao <taohl@lemote.com>
Cc: Hua Yan <yanh@lemote.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/cavium-octeon/smp.c | 1 +
arch/mips/include/asm/smp.h | 2 ++
arch/mips/kernel/smp-bmips.c | 1 +
arch/mips/kernel/smp-cps.c | 1 +
arch/mips/kernel/smp.c | 2 +-
arch/mips/loongson64/loongson-3/smp.c | 1 +
6 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/arch/mips/cavium-octeon/smp.c b/arch/mips/cavium-octeon/smp.c
index 33aab89259f3..4d457d602d3b 100644
--- a/arch/mips/cavium-octeon/smp.c
+++ b/arch/mips/cavium-octeon/smp.c
@@ -271,6 +271,7 @@ static int octeon_cpu_disable(void)
return -ENOTSUPP;
set_cpu_online(cpu, false);
+ calculate_cpu_foreign_map();
cpumask_clear_cpu(cpu, &cpu_callin_map);
octeon_fixup_irqs();
diff --git a/arch/mips/include/asm/smp.h b/arch/mips/include/asm/smp.h
index 03722d4326a1..0c534a03bb36 100644
--- a/arch/mips/include/asm/smp.h
+++ b/arch/mips/include/asm/smp.h
@@ -53,6 +53,8 @@ extern cpumask_t cpu_coherent_mask;
extern void asmlinkage smp_bootstrap(void);
+extern void calculate_cpu_foreign_map(void);
+
/*
* this function sends a 'reschedule' IPI to another CPU.
* it goes straight through and wastes no time serializing
diff --git a/arch/mips/kernel/smp-bmips.c b/arch/mips/kernel/smp-bmips.c
index e02addc0307f..6d0f1321e084 100644
--- a/arch/mips/kernel/smp-bmips.c
+++ b/arch/mips/kernel/smp-bmips.c
@@ -363,6 +363,7 @@ static int bmips_cpu_disable(void)
pr_info("SMP: CPU%d is offline\n", cpu);
set_cpu_online(cpu, false);
+ calculate_cpu_foreign_map();
cpumask_clear_cpu(cpu, &cpu_callin_map);
clear_c0_status(IE_IRQ5);
diff --git a/arch/mips/kernel/smp-cps.c b/arch/mips/kernel/smp-cps.c
index 4ed36f288d64..47ff5108bed5 100644
--- a/arch/mips/kernel/smp-cps.c
+++ b/arch/mips/kernel/smp-cps.c
@@ -397,6 +397,7 @@ static int cps_cpu_disable(void)
atomic_sub(1 << cpu_vpe_id(¤t_cpu_data), &core_cfg->vpe_mask);
smp_mb__after_atomic();
set_cpu_online(cpu, false);
+ calculate_cpu_foreign_map();
cpumask_clear_cpu(cpu, &cpu_callin_map);
return 0;
diff --git a/arch/mips/kernel/smp.c b/arch/mips/kernel/smp.c
index 0c98b4a313be..a4d4309ecff2 100644
--- a/arch/mips/kernel/smp.c
+++ b/arch/mips/kernel/smp.c
@@ -124,7 +124,7 @@ static inline void set_cpu_core_map(int cpu)
* Calculate a new cpu_foreign_map mask whenever a
* new cpu appears or disappears.
*/
-static inline void calculate_cpu_foreign_map(void)
+void calculate_cpu_foreign_map(void)
{
int i, k, core_present;
cpumask_t temp_foreign_map;
diff --git a/arch/mips/loongson64/loongson-3/smp.c b/arch/mips/loongson64/loongson-3/smp.c
index e59759af63d9..2fec6f753a35 100644
--- a/arch/mips/loongson64/loongson-3/smp.c
+++ b/arch/mips/loongson64/loongson-3/smp.c
@@ -417,6 +417,7 @@ static int loongson3_cpu_disable(void)
return -EBUSY;
set_cpu_online(cpu, false);
+ calculate_cpu_foreign_map();
cpumask_clear_cpu(cpu, &cpu_callin_map);
local_irq_save(flags);
fixup_irqs();
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 03/13] MIPS: SMP: Drop stop_this_cpu() cpu_foreign_map hack
2016-07-13 13:12 [PATCH 00/13] MIPS: c-r4k: SMP cache management fixes James Hogan
` (2 preceding siblings ...)
2016-07-13 13:12 ` [PATCH 02/13] MIPS: SMP: Update cpu_foreign_map on CPU disable James Hogan
@ 2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` [PATCH 04/13] MIPS: c-r4k: Fix protected_writeback_scache_line for EVA James Hogan
` (9 subsequent siblings)
13 siblings, 1 reply; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, linux-mips, Paul Burton, Leonid Yegoshin
Commit cccf34e9411c ("MIPS: c-r4k: Fix cache flushing for MT cores")
added the cpu_foreign_map cpumask containing a single VPE from each
online core, and recalculated it when secondary CPUs are brought up.
stop_this_cpu() was also updated to recalculate cpu_foreign_map, but
with an additional hack before marking the CPU as offline to copy
cpu_online_mask into cpu_foreign_map and perform an SMP memory barrier.
This appears to have been intended to prevent cache management IPIs
being missed when the VPE representing the core in cpu_foreign_map is
taken offline while other VPEs remain online. Unfortunately there is
nothing in this hack to prevent r4k_on_each_cpu() from reading the old
cpu_foreign_map, and smp_call_function_many() from reading that new
cpu_online_mask with the core's representative VPE marked offline. It
then wouldn't send an IPI to any online VPEs of that core.
stop_this_cpu() is only actually called in panic and system shutdown /
halt / reboot situations, in which case all CPUs are going down and we
don't really need to care about cache management, so drop this hack.
Note that the __cpu_disable() case for CPU hotplug is handled in the
previous commit, and no synchronisation is needed there due to the use
of stop_machine() which prevents hotplug from taking place while any CPU
has disabled preemption (as r4k_on_each_cpu() does).
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: linux-mips@linux-mips.org
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
---
arch/mips/kernel/smp.c | 9 +--------
1 file changed, 1 insertion(+), 8 deletions(-)
diff --git a/arch/mips/kernel/smp.c b/arch/mips/kernel/smp.c
index a4d4309ecff2..783d5f50ab9d 100644
--- a/arch/mips/kernel/smp.c
+++ b/arch/mips/kernel/smp.c
@@ -344,16 +344,9 @@ asmlinkage void start_secondary(void)
static void stop_this_cpu(void *dummy)
{
/*
- * Remove this CPU. Be a bit slow here and
- * set the bits for every online CPU so we don't miss
- * any IPI whilst taking this VPE down.
+ * Remove this CPU:
*/
- cpumask_copy(&cpu_foreign_map, cpu_online_mask);
-
- /* Make it visible to every other CPU */
- smp_mb();
-
set_cpu_online(smp_processor_id(), false);
calculate_cpu_foreign_map();
local_irq_disable();
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH 03/13] MIPS: SMP: Drop stop_this_cpu() cpu_foreign_map hack
2016-07-13 13:12 ` [PATCH 03/13] MIPS: SMP: Drop stop_this_cpu() cpu_foreign_map hack James Hogan
@ 2016-07-13 13:12 ` James Hogan
0 siblings, 0 replies; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, linux-mips, Paul Burton, Leonid Yegoshin
Commit cccf34e9411c ("MIPS: c-r4k: Fix cache flushing for MT cores")
added the cpu_foreign_map cpumask containing a single VPE from each
online core, and recalculated it when secondary CPUs are brought up.
stop_this_cpu() was also updated to recalculate cpu_foreign_map, but
with an additional hack before marking the CPU as offline to copy
cpu_online_mask into cpu_foreign_map and perform an SMP memory barrier.
This appears to have been intended to prevent cache management IPIs
being missed when the VPE representing the core in cpu_foreign_map is
taken offline while other VPEs remain online. Unfortunately there is
nothing in this hack to prevent r4k_on_each_cpu() from reading the old
cpu_foreign_map, and smp_call_function_many() from reading that new
cpu_online_mask with the core's representative VPE marked offline. It
then wouldn't send an IPI to any online VPEs of that core.
stop_this_cpu() is only actually called in panic and system shutdown /
halt / reboot situations, in which case all CPUs are going down and we
don't really need to care about cache management, so drop this hack.
Note that the __cpu_disable() case for CPU hotplug is handled in the
previous commit, and no synchronisation is needed there due to the use
of stop_machine() which prevents hotplug from taking place while any CPU
has disabled preemption (as r4k_on_each_cpu() does).
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: linux-mips@linux-mips.org
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
---
arch/mips/kernel/smp.c | 9 +--------
1 file changed, 1 insertion(+), 8 deletions(-)
diff --git a/arch/mips/kernel/smp.c b/arch/mips/kernel/smp.c
index a4d4309ecff2..783d5f50ab9d 100644
--- a/arch/mips/kernel/smp.c
+++ b/arch/mips/kernel/smp.c
@@ -344,16 +344,9 @@ asmlinkage void start_secondary(void)
static void stop_this_cpu(void *dummy)
{
/*
- * Remove this CPU. Be a bit slow here and
- * set the bits for every online CPU so we don't miss
- * any IPI whilst taking this VPE down.
+ * Remove this CPU:
*/
- cpumask_copy(&cpu_foreign_map, cpu_online_mask);
-
- /* Make it visible to every other CPU */
- smp_mb();
-
set_cpu_online(smp_processor_id(), false);
calculate_cpu_foreign_map();
local_irq_disable();
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 04/13] MIPS: c-r4k: Fix protected_writeback_scache_line for EVA
2016-07-13 13:12 [PATCH 00/13] MIPS: c-r4k: SMP cache management fixes James Hogan
` (3 preceding siblings ...)
2016-07-13 13:12 ` [PATCH 03/13] MIPS: SMP: Drop stop_this_cpu() cpu_foreign_map hack James Hogan
@ 2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` [PATCH 05/13] MIPS: c-r4k: Fix sigtramp SMP call to use kmap James Hogan
` (8 subsequent siblings)
13 siblings, 1 reply; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, Leonid Yegoshin, linux-mips
The protected_writeback_scache_line() function is used by
local_r4k_flush_cache_sigtramp() to flush an FPU delay slot emulation
trampoline on the userland stack from the caches so it is visible to
subsequent instruction fetches.
Commit de8974e3f76c ("MIPS: asm: r4kcache: Add EVA cache flushing
functions") updated some protected_ cache flush functions to use EVA
CACHEE instructions via protected_cachee_op(), and commit 83fd43449baa
("MIPS: r4kcache: Add EVA case for protected_writeback_dcache_line") did
the same thing for protected_writeback_dcache_line(), but
protected_writeback_scache_line() never got updated. Lets fix that now
to flush the right user address from the secondary cache rather than
some arbitrary kernel unmapped address.
This issue was spotted through code inspection, and it seems unlikely to
be possible to hit this in practice. It theoretically affect EVA kernels
on EVA capable cores with an L2 cache, where the icache fetches straight
from RAM (cpu_icache_snoops_remote_store == 0), running a hard float
userland with FPU disabled (nofpu). That both Malta and Boston platforms
override cpu_icache_snoops_remote_store to 1 suggests that all MIPS
cores fetch instructions into icache straight from L2 rather than RAM.
Fixes: de8974e3f76c ("MIPS: asm: r4kcache: Add EVA cache flushing functions")
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/include/asm/r4kcache.h | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/arch/mips/include/asm/r4kcache.h b/arch/mips/include/asm/r4kcache.h
index 38902bf97adc..667ca3c467b7 100644
--- a/arch/mips/include/asm/r4kcache.h
+++ b/arch/mips/include/asm/r4kcache.h
@@ -210,7 +210,11 @@ static inline void protected_writeback_dcache_line(unsigned long addr)
static inline void protected_writeback_scache_line(unsigned long addr)
{
+#ifdef CONFIG_EVA
+ protected_cachee_op(Hit_Writeback_Inv_SD, addr);
+#else
protected_cache_op(Hit_Writeback_Inv_SD, addr);
+#endif
}
/*
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH 04/13] MIPS: c-r4k: Fix protected_writeback_scache_line for EVA
2016-07-13 13:12 ` [PATCH 04/13] MIPS: c-r4k: Fix protected_writeback_scache_line for EVA James Hogan
@ 2016-07-13 13:12 ` James Hogan
0 siblings, 0 replies; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, Leonid Yegoshin, linux-mips
The protected_writeback_scache_line() function is used by
local_r4k_flush_cache_sigtramp() to flush an FPU delay slot emulation
trampoline on the userland stack from the caches so it is visible to
subsequent instruction fetches.
Commit de8974e3f76c ("MIPS: asm: r4kcache: Add EVA cache flushing
functions") updated some protected_ cache flush functions to use EVA
CACHEE instructions via protected_cachee_op(), and commit 83fd43449baa
("MIPS: r4kcache: Add EVA case for protected_writeback_dcache_line") did
the same thing for protected_writeback_dcache_line(), but
protected_writeback_scache_line() never got updated. Lets fix that now
to flush the right user address from the secondary cache rather than
some arbitrary kernel unmapped address.
This issue was spotted through code inspection, and it seems unlikely to
be possible to hit this in practice. It theoretically affect EVA kernels
on EVA capable cores with an L2 cache, where the icache fetches straight
from RAM (cpu_icache_snoops_remote_store == 0), running a hard float
userland with FPU disabled (nofpu). That both Malta and Boston platforms
override cpu_icache_snoops_remote_store to 1 suggests that all MIPS
cores fetch instructions into icache straight from L2 rather than RAM.
Fixes: de8974e3f76c ("MIPS: asm: r4kcache: Add EVA cache flushing functions")
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/include/asm/r4kcache.h | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/arch/mips/include/asm/r4kcache.h b/arch/mips/include/asm/r4kcache.h
index 38902bf97adc..667ca3c467b7 100644
--- a/arch/mips/include/asm/r4kcache.h
+++ b/arch/mips/include/asm/r4kcache.h
@@ -210,7 +210,11 @@ static inline void protected_writeback_dcache_line(unsigned long addr)
static inline void protected_writeback_scache_line(unsigned long addr)
{
+#ifdef CONFIG_EVA
+ protected_cachee_op(Hit_Writeback_Inv_SD, addr);
+#else
protected_cache_op(Hit_Writeback_Inv_SD, addr);
+#endif
}
/*
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 05/13] MIPS: c-r4k: Fix sigtramp SMP call to use kmap
2016-07-13 13:12 [PATCH 00/13] MIPS: c-r4k: SMP cache management fixes James Hogan
` (4 preceding siblings ...)
2016-07-13 13:12 ` [PATCH 04/13] MIPS: c-r4k: Fix protected_writeback_scache_line for EVA James Hogan
@ 2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` [PATCH 06/13] MIPS: c-r4k: Avoid dcache flush for sigtramps James Hogan
` (7 subsequent siblings)
13 siblings, 1 reply; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, Paul Burton, Leonid Yegoshin, linux-mips
Fix r4k_flush_cache_sigtramp() and local_r4k_flush_cache_sigtramp() to
flush the delay slot emulation trampoline cacheline through a kmap
rather than directly when the active_mm doesn't match that of the task
initiating the flush, a bit like local_r4k_flush_cache_page() does.
This would fix a corner case on SMP systems without hardware globalized
hit cache ops, where a migration to another CPU after the flush, where
that CPU did not have the same mm active at the time of the flush, could
result in stale icache content being executed instead of the trampoline,
e.g. from a previous delay slot emulation with a similar stack pointer.
This case was artificially triggered by replacing the icache flush with
a full indexed flush (not globalized on CM systems) and forcing the SMP
call to take place, with a test program that alternated two FPU delay
slots with a parent process repeatedly changing scheduler affinity.
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/mm/c-r4k.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++-----
1 file changed, 69 insertions(+), 6 deletions(-)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 9204d4e4f02f..600b0ad48319 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -792,25 +792,72 @@ static void r4k_dma_cache_inv(unsigned long addr, unsigned long size)
}
#endif /* CONFIG_DMA_NONCOHERENT || CONFIG_DMA_MAYBE_COHERENT */
+struct flush_cache_sigtramp_args {
+ struct mm_struct *mm;
+ struct page *page;
+ unsigned long addr;
+};
+
/*
* While we're protected against bad userland addresses we don't care
* very much about what happens in that case. Usually a segmentation
* fault will dump the process later on anyway ...
*/
-static void local_r4k_flush_cache_sigtramp(void * arg)
+static void local_r4k_flush_cache_sigtramp(void *args)
{
+ struct flush_cache_sigtramp_args *fcs_args = args;
+ unsigned long addr = fcs_args->addr;
+ struct page *page = fcs_args->page;
+ struct mm_struct *mm = fcs_args->mm;
+ int map_coherent = 0;
+ void *vaddr;
+
unsigned long ic_lsize = cpu_icache_line_size();
unsigned long dc_lsize = cpu_dcache_line_size();
unsigned long sc_lsize = cpu_scache_line_size();
- unsigned long addr = (unsigned long) arg;
+
+ /*
+ * If owns no valid ASID yet, cannot possibly have gotten
+ * this page into the cache.
+ */
+ if (!has_valid_asid(mm))
+ return;
+
+ if (mm == current->active_mm) {
+ vaddr = NULL;
+ } else {
+ /*
+ * Use kmap_coherent or kmap_atomic to do flushes for
+ * another ASID than the current one.
+ */
+ map_coherent = (cpu_has_dc_aliases &&
+ page_mapcount(page) &&
+ !Page_dcache_dirty(page));
+ if (map_coherent)
+ vaddr = kmap_coherent(page, addr);
+ else
+ vaddr = kmap_atomic(page);
+ addr = (unsigned long)vaddr + (addr & ~PAGE_MASK);
+ }
R4600_HIT_CACHEOP_WAR_IMPL;
if (dc_lsize)
- protected_writeback_dcache_line(addr & ~(dc_lsize - 1));
+ vaddr ? flush_dcache_line(addr & ~(dc_lsize - 1))
+ : protected_writeback_dcache_line(addr & ~(dc_lsize - 1));
if (!cpu_icache_snoops_remote_store && scache_size)
- protected_writeback_scache_line(addr & ~(sc_lsize - 1));
+ vaddr ? flush_scache_line(addr & ~(sc_lsize - 1))
+ : protected_writeback_scache_line(addr & ~(sc_lsize - 1));
if (ic_lsize)
- protected_flush_icache_line(addr & ~(ic_lsize - 1));
+ vaddr ? flush_icache_line(addr & ~(ic_lsize - 1))
+ : protected_flush_icache_line(addr & ~(ic_lsize - 1));
+
+ if (vaddr) {
+ if (map_coherent)
+ kunmap_coherent();
+ else
+ kunmap_atomic(vaddr);
+ }
+
if (MIPS4K_ICACHE_REFILL_WAR) {
__asm__ __volatile__ (
".set push\n\t"
@@ -835,7 +882,23 @@ static void local_r4k_flush_cache_sigtramp(void * arg)
static void r4k_flush_cache_sigtramp(unsigned long addr)
{
- r4k_on_each_cpu(local_r4k_flush_cache_sigtramp, (void *) addr);
+ struct flush_cache_sigtramp_args args;
+ int npages;
+
+ down_read(¤t->mm->mmap_sem);
+
+ npages = get_user_pages_fast(addr, 1, 0, &args.page);
+ if (npages < 1)
+ goto out;
+
+ args.mm = current->mm;
+ args.addr = addr;
+
+ r4k_on_each_cpu(local_r4k_flush_cache_sigtramp, &args);
+
+ put_page(args.page);
+out:
+ up_read(¤t->mm->mmap_sem);
}
static void r4k_flush_icache_all(void)
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH 05/13] MIPS: c-r4k: Fix sigtramp SMP call to use kmap
2016-07-13 13:12 ` [PATCH 05/13] MIPS: c-r4k: Fix sigtramp SMP call to use kmap James Hogan
@ 2016-07-13 13:12 ` James Hogan
0 siblings, 0 replies; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, Paul Burton, Leonid Yegoshin, linux-mips
Fix r4k_flush_cache_sigtramp() and local_r4k_flush_cache_sigtramp() to
flush the delay slot emulation trampoline cacheline through a kmap
rather than directly when the active_mm doesn't match that of the task
initiating the flush, a bit like local_r4k_flush_cache_page() does.
This would fix a corner case on SMP systems without hardware globalized
hit cache ops, where a migration to another CPU after the flush, where
that CPU did not have the same mm active at the time of the flush, could
result in stale icache content being executed instead of the trampoline,
e.g. from a previous delay slot emulation with a similar stack pointer.
This case was artificially triggered by replacing the icache flush with
a full indexed flush (not globalized on CM systems) and forcing the SMP
call to take place, with a test program that alternated two FPU delay
slots with a parent process repeatedly changing scheduler affinity.
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/mm/c-r4k.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++-----
1 file changed, 69 insertions(+), 6 deletions(-)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 9204d4e4f02f..600b0ad48319 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -792,25 +792,72 @@ static void r4k_dma_cache_inv(unsigned long addr, unsigned long size)
}
#endif /* CONFIG_DMA_NONCOHERENT || CONFIG_DMA_MAYBE_COHERENT */
+struct flush_cache_sigtramp_args {
+ struct mm_struct *mm;
+ struct page *page;
+ unsigned long addr;
+};
+
/*
* While we're protected against bad userland addresses we don't care
* very much about what happens in that case. Usually a segmentation
* fault will dump the process later on anyway ...
*/
-static void local_r4k_flush_cache_sigtramp(void * arg)
+static void local_r4k_flush_cache_sigtramp(void *args)
{
+ struct flush_cache_sigtramp_args *fcs_args = args;
+ unsigned long addr = fcs_args->addr;
+ struct page *page = fcs_args->page;
+ struct mm_struct *mm = fcs_args->mm;
+ int map_coherent = 0;
+ void *vaddr;
+
unsigned long ic_lsize = cpu_icache_line_size();
unsigned long dc_lsize = cpu_dcache_line_size();
unsigned long sc_lsize = cpu_scache_line_size();
- unsigned long addr = (unsigned long) arg;
+
+ /*
+ * If owns no valid ASID yet, cannot possibly have gotten
+ * this page into the cache.
+ */
+ if (!has_valid_asid(mm))
+ return;
+
+ if (mm == current->active_mm) {
+ vaddr = NULL;
+ } else {
+ /*
+ * Use kmap_coherent or kmap_atomic to do flushes for
+ * another ASID than the current one.
+ */
+ map_coherent = (cpu_has_dc_aliases &&
+ page_mapcount(page) &&
+ !Page_dcache_dirty(page));
+ if (map_coherent)
+ vaddr = kmap_coherent(page, addr);
+ else
+ vaddr = kmap_atomic(page);
+ addr = (unsigned long)vaddr + (addr & ~PAGE_MASK);
+ }
R4600_HIT_CACHEOP_WAR_IMPL;
if (dc_lsize)
- protected_writeback_dcache_line(addr & ~(dc_lsize - 1));
+ vaddr ? flush_dcache_line(addr & ~(dc_lsize - 1))
+ : protected_writeback_dcache_line(addr & ~(dc_lsize - 1));
if (!cpu_icache_snoops_remote_store && scache_size)
- protected_writeback_scache_line(addr & ~(sc_lsize - 1));
+ vaddr ? flush_scache_line(addr & ~(sc_lsize - 1))
+ : protected_writeback_scache_line(addr & ~(sc_lsize - 1));
if (ic_lsize)
- protected_flush_icache_line(addr & ~(ic_lsize - 1));
+ vaddr ? flush_icache_line(addr & ~(ic_lsize - 1))
+ : protected_flush_icache_line(addr & ~(ic_lsize - 1));
+
+ if (vaddr) {
+ if (map_coherent)
+ kunmap_coherent();
+ else
+ kunmap_atomic(vaddr);
+ }
+
if (MIPS4K_ICACHE_REFILL_WAR) {
__asm__ __volatile__ (
".set push\n\t"
@@ -835,7 +882,23 @@ static void local_r4k_flush_cache_sigtramp(void * arg)
static void r4k_flush_cache_sigtramp(unsigned long addr)
{
- r4k_on_each_cpu(local_r4k_flush_cache_sigtramp, (void *) addr);
+ struct flush_cache_sigtramp_args args;
+ int npages;
+
+ down_read(¤t->mm->mmap_sem);
+
+ npages = get_user_pages_fast(addr, 1, 0, &args.page);
+ if (npages < 1)
+ goto out;
+
+ args.mm = current->mm;
+ args.addr = addr;
+
+ r4k_on_each_cpu(local_r4k_flush_cache_sigtramp, &args);
+
+ put_page(args.page);
+out:
+ up_read(¤t->mm->mmap_sem);
}
static void r4k_flush_icache_all(void)
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 06/13] MIPS: c-r4k: Avoid dcache flush for sigtramps
2016-07-13 13:12 [PATCH 00/13] MIPS: c-r4k: SMP cache management fixes James Hogan
` (5 preceding siblings ...)
2016-07-13 13:12 ` [PATCH 05/13] MIPS: c-r4k: Fix sigtramp SMP call to use kmap James Hogan
@ 2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` [PATCH 07/13] MIPS: c-r4k: Add r4k_on_each_cpu cache op type arg James Hogan
` (6 subsequent siblings)
13 siblings, 1 reply; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, Paul Burton, Leonid Yegoshin, linux-mips
Avoid the dcache and scache flush in local_r4k_flush_cache_sigtramp() if
the icache fills straight from the dcache.
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/mm/c-r4k.c | 16 ++++++++++------
1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 600b0ad48319..58b810e67bba 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -841,12 +841,16 @@ static void local_r4k_flush_cache_sigtramp(void *args)
}
R4600_HIT_CACHEOP_WAR_IMPL;
- if (dc_lsize)
- vaddr ? flush_dcache_line(addr & ~(dc_lsize - 1))
- : protected_writeback_dcache_line(addr & ~(dc_lsize - 1));
- if (!cpu_icache_snoops_remote_store && scache_size)
- vaddr ? flush_scache_line(addr & ~(sc_lsize - 1))
- : protected_writeback_scache_line(addr & ~(sc_lsize - 1));
+ if (!cpu_has_ic_fills_f_dc) {
+ if (dc_lsize)
+ vaddr ? flush_dcache_line(addr & ~(dc_lsize - 1))
+ : protected_writeback_dcache_line(
+ addr & ~(dc_lsize - 1));
+ if (!cpu_icache_snoops_remote_store && scache_size)
+ vaddr ? flush_scache_line(addr & ~(sc_lsize - 1))
+ : protected_writeback_scache_line(
+ addr & ~(sc_lsize - 1));
+ }
if (ic_lsize)
vaddr ? flush_icache_line(addr & ~(ic_lsize - 1))
: protected_flush_icache_line(addr & ~(ic_lsize - 1));
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH 06/13] MIPS: c-r4k: Avoid dcache flush for sigtramps
2016-07-13 13:12 ` [PATCH 06/13] MIPS: c-r4k: Avoid dcache flush for sigtramps James Hogan
@ 2016-07-13 13:12 ` James Hogan
0 siblings, 0 replies; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, Paul Burton, Leonid Yegoshin, linux-mips
Avoid the dcache and scache flush in local_r4k_flush_cache_sigtramp() if
the icache fills straight from the dcache.
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/mm/c-r4k.c | 16 ++++++++++------
1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 600b0ad48319..58b810e67bba 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -841,12 +841,16 @@ static void local_r4k_flush_cache_sigtramp(void *args)
}
R4600_HIT_CACHEOP_WAR_IMPL;
- if (dc_lsize)
- vaddr ? flush_dcache_line(addr & ~(dc_lsize - 1))
- : protected_writeback_dcache_line(addr & ~(dc_lsize - 1));
- if (!cpu_icache_snoops_remote_store && scache_size)
- vaddr ? flush_scache_line(addr & ~(sc_lsize - 1))
- : protected_writeback_scache_line(addr & ~(sc_lsize - 1));
+ if (!cpu_has_ic_fills_f_dc) {
+ if (dc_lsize)
+ vaddr ? flush_dcache_line(addr & ~(dc_lsize - 1))
+ : protected_writeback_dcache_line(
+ addr & ~(dc_lsize - 1));
+ if (!cpu_icache_snoops_remote_store && scache_size)
+ vaddr ? flush_scache_line(addr & ~(sc_lsize - 1))
+ : protected_writeback_scache_line(
+ addr & ~(sc_lsize - 1));
+ }
if (ic_lsize)
vaddr ? flush_icache_line(addr & ~(ic_lsize - 1))
: protected_flush_icache_line(addr & ~(ic_lsize - 1));
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 07/13] MIPS: c-r4k: Add r4k_on_each_cpu cache op type arg
2016-07-13 13:12 [PATCH 00/13] MIPS: c-r4k: SMP cache management fixes James Hogan
` (6 preceding siblings ...)
2016-07-13 13:12 ` [PATCH 06/13] MIPS: c-r4k: Avoid dcache flush for sigtramps James Hogan
@ 2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` [PATCH 08/13] MIPS: c-r4k: Fix valid ASID optimisation James Hogan
` (5 subsequent siblings)
13 siblings, 1 reply; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, Paul Burton, Leonid Yegoshin, linux-mips
The r4k_on_each_cpu() function calls the specified cache flush helper on
other CPUs if deemed necessary due to the cache ops not being
globalized by hardware. However this really depends on the cache op
addressing type, as the MIPS Coherence Manager (CM) if present will
globalize "hit" cache ops (addressed by virtual address), but not
"index" cache ops (addressed by cache index). This results in index
cache ops only being performed on a single CPU when CM is present.
Most (but not all) of the functions called by r4k_on_each_cpu() perform
cache operations exclusively with a single cache op type, so add a type
argument and modify the callers to pass in some combination of R4K_HIT
(global kernel virtual addressing or user virtual addressing
conditional upon matching active_mm) and R4K_INDEX (index into cache).
This will allow r4k_on_each_cpu() to later distinguish these cases and
decide whether to perform an SMP call based on it.
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/mm/c-r4k.c | 70 ++++++++++++++++++++++++++++++++++++++--------------
1 file changed, 51 insertions(+), 19 deletions(-)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 58b810e67bba..412052321472 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -40,6 +40,43 @@
#include <asm/mips-cm.h>
/*
+ * Bits describing what cache ops an SMP callback function may perform.
+ *
+ * R4K_HIT - Virtual user or kernel address based cache operations. The
+ * active_mm must be checked before using user addresses, falling
+ * back to kmap.
+ * R4K_INDEX - Index based cache operations.
+ */
+
+#define R4K_HIT BIT(0)
+#define R4K_INDEX BIT(1)
+
+/**
+ * r4k_op_needs_ipi() - Decide if a cache op needs to be done on every core.
+ * @type: Type of cache operations (R4K_HIT or R4K_INDEX).
+ *
+ * Decides whether a cache op needs to be performed on every core in the system.
+ * This may change depending on the @type of cache operation.
+ *
+ * Returns: 1 if the cache operation @type should be done on every core in
+ * the system.
+ * 0 if the cache operation @type is globalized and only needs to
+ * be performed on a simple CPU.
+ */
+static inline bool r4k_op_needs_ipi(unsigned int type)
+{
+ /* The MIPS Coherence Manager (CM) globalizes address-based cache ops */
+ if (mips_cm_present())
+ return false;
+
+ /*
+ * Hardware doesn't globalize the required cache ops, so SMP calls may
+ * be needed.
+ */
+ return true;
+}
+
+/*
* Special Variant of smp_call_function for use by cache functions:
*
* o No return value
@@ -48,19 +85,11 @@
* primary cache.
* o doesn't disable interrupts on the local CPU
*/
-static inline void r4k_on_each_cpu(void (*func) (void *info), void *info)
+static inline void r4k_on_each_cpu(unsigned int type,
+ void (*func)(void *info), void *info)
{
preempt_disable();
-
- /*
- * The Coherent Manager propagates address-based cache ops to other
- * cores but not index-based ops. However, r4k_on_each_cpu is used
- * in both cases so there is no easy way to tell what kind of op is
- * executed to the other cores. The best we can probably do is
- * to restrict that call when a CM is not present because both
- * CM-based SMP protocols (CMP & CPS) restrict index-based cache ops.
- */
- if (!mips_cm_present())
+ if (r4k_op_needs_ipi(type))
smp_call_function_many(&cpu_foreign_map, func, info, 1);
func(info);
preempt_enable();
@@ -456,7 +485,7 @@ static inline void local_r4k___flush_cache_all(void * args)
static void r4k___flush_cache_all(void)
{
- r4k_on_each_cpu(local_r4k___flush_cache_all, NULL);
+ r4k_on_each_cpu(R4K_INDEX, local_r4k___flush_cache_all, NULL);
}
static inline int has_valid_asid(const struct mm_struct *mm)
@@ -514,7 +543,7 @@ static void r4k_flush_cache_range(struct vm_area_struct *vma,
int exec = vma->vm_flags & VM_EXEC;
if (cpu_has_dc_aliases || exec)
- r4k_on_each_cpu(local_r4k_flush_cache_range, vma);
+ r4k_on_each_cpu(R4K_INDEX, local_r4k_flush_cache_range, vma);
}
static inline void local_r4k_flush_cache_mm(void * args)
@@ -546,7 +575,7 @@ static void r4k_flush_cache_mm(struct mm_struct *mm)
if (!cpu_has_dc_aliases)
return;
- r4k_on_each_cpu(local_r4k_flush_cache_mm, mm);
+ r4k_on_each_cpu(R4K_INDEX, local_r4k_flush_cache_mm, mm);
}
struct flush_cache_page_args {
@@ -641,7 +670,7 @@ static void r4k_flush_cache_page(struct vm_area_struct *vma,
args.addr = addr;
args.pfn = pfn;
- r4k_on_each_cpu(local_r4k_flush_cache_page, &args);
+ r4k_on_each_cpu(R4K_HIT, local_r4k_flush_cache_page, &args);
}
static inline void local_r4k_flush_data_cache_page(void * addr)
@@ -654,7 +683,8 @@ static void r4k_flush_data_cache_page(unsigned long addr)
if (in_atomic())
local_r4k_flush_data_cache_page((void *)addr);
else
- r4k_on_each_cpu(local_r4k_flush_data_cache_page, (void *) addr);
+ r4k_on_each_cpu(R4K_HIT, local_r4k_flush_data_cache_page,
+ (void *) addr);
}
struct flush_icache_range_args {
@@ -715,7 +745,8 @@ static void r4k_flush_icache_range(unsigned long start, unsigned long end)
args.start = start;
args.end = end;
- r4k_on_each_cpu(local_r4k_flush_icache_range_ipi, &args);
+ r4k_on_each_cpu(R4K_HIT | R4K_INDEX, local_r4k_flush_icache_range_ipi,
+ &args);
instruction_hazard();
}
@@ -898,7 +929,7 @@ static void r4k_flush_cache_sigtramp(unsigned long addr)
args.mm = current->mm;
args.addr = addr;
- r4k_on_each_cpu(local_r4k_flush_cache_sigtramp, &args);
+ r4k_on_each_cpu(R4K_HIT, local_r4k_flush_cache_sigtramp, &args);
put_page(args.page);
out:
@@ -941,7 +972,8 @@ static void r4k_flush_kernel_vmap_range(unsigned long vaddr, int size)
args.vaddr = (unsigned long) vaddr;
args.size = size;
- r4k_on_each_cpu(local_r4k_flush_kernel_vmap_range, &args);
+ r4k_on_each_cpu(R4K_HIT | R4K_INDEX, local_r4k_flush_kernel_vmap_range,
+ &args);
}
static inline void rm7k_erratum31(void)
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH 07/13] MIPS: c-r4k: Add r4k_on_each_cpu cache op type arg
2016-07-13 13:12 ` [PATCH 07/13] MIPS: c-r4k: Add r4k_on_each_cpu cache op type arg James Hogan
@ 2016-07-13 13:12 ` James Hogan
0 siblings, 0 replies; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, Paul Burton, Leonid Yegoshin, linux-mips
The r4k_on_each_cpu() function calls the specified cache flush helper on
other CPUs if deemed necessary due to the cache ops not being
globalized by hardware. However this really depends on the cache op
addressing type, as the MIPS Coherence Manager (CM) if present will
globalize "hit" cache ops (addressed by virtual address), but not
"index" cache ops (addressed by cache index). This results in index
cache ops only being performed on a single CPU when CM is present.
Most (but not all) of the functions called by r4k_on_each_cpu() perform
cache operations exclusively with a single cache op type, so add a type
argument and modify the callers to pass in some combination of R4K_HIT
(global kernel virtual addressing or user virtual addressing
conditional upon matching active_mm) and R4K_INDEX (index into cache).
This will allow r4k_on_each_cpu() to later distinguish these cases and
decide whether to perform an SMP call based on it.
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/mm/c-r4k.c | 70 ++++++++++++++++++++++++++++++++++++++--------------
1 file changed, 51 insertions(+), 19 deletions(-)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 58b810e67bba..412052321472 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -40,6 +40,43 @@
#include <asm/mips-cm.h>
/*
+ * Bits describing what cache ops an SMP callback function may perform.
+ *
+ * R4K_HIT - Virtual user or kernel address based cache operations. The
+ * active_mm must be checked before using user addresses, falling
+ * back to kmap.
+ * R4K_INDEX - Index based cache operations.
+ */
+
+#define R4K_HIT BIT(0)
+#define R4K_INDEX BIT(1)
+
+/**
+ * r4k_op_needs_ipi() - Decide if a cache op needs to be done on every core.
+ * @type: Type of cache operations (R4K_HIT or R4K_INDEX).
+ *
+ * Decides whether a cache op needs to be performed on every core in the system.
+ * This may change depending on the @type of cache operation.
+ *
+ * Returns: 1 if the cache operation @type should be done on every core in
+ * the system.
+ * 0 if the cache operation @type is globalized and only needs to
+ * be performed on a simple CPU.
+ */
+static inline bool r4k_op_needs_ipi(unsigned int type)
+{
+ /* The MIPS Coherence Manager (CM) globalizes address-based cache ops */
+ if (mips_cm_present())
+ return false;
+
+ /*
+ * Hardware doesn't globalize the required cache ops, so SMP calls may
+ * be needed.
+ */
+ return true;
+}
+
+/*
* Special Variant of smp_call_function for use by cache functions:
*
* o No return value
@@ -48,19 +85,11 @@
* primary cache.
* o doesn't disable interrupts on the local CPU
*/
-static inline void r4k_on_each_cpu(void (*func) (void *info), void *info)
+static inline void r4k_on_each_cpu(unsigned int type,
+ void (*func)(void *info), void *info)
{
preempt_disable();
-
- /*
- * The Coherent Manager propagates address-based cache ops to other
- * cores but not index-based ops. However, r4k_on_each_cpu is used
- * in both cases so there is no easy way to tell what kind of op is
- * executed to the other cores. The best we can probably do is
- * to restrict that call when a CM is not present because both
- * CM-based SMP protocols (CMP & CPS) restrict index-based cache ops.
- */
- if (!mips_cm_present())
+ if (r4k_op_needs_ipi(type))
smp_call_function_many(&cpu_foreign_map, func, info, 1);
func(info);
preempt_enable();
@@ -456,7 +485,7 @@ static inline void local_r4k___flush_cache_all(void * args)
static void r4k___flush_cache_all(void)
{
- r4k_on_each_cpu(local_r4k___flush_cache_all, NULL);
+ r4k_on_each_cpu(R4K_INDEX, local_r4k___flush_cache_all, NULL);
}
static inline int has_valid_asid(const struct mm_struct *mm)
@@ -514,7 +543,7 @@ static void r4k_flush_cache_range(struct vm_area_struct *vma,
int exec = vma->vm_flags & VM_EXEC;
if (cpu_has_dc_aliases || exec)
- r4k_on_each_cpu(local_r4k_flush_cache_range, vma);
+ r4k_on_each_cpu(R4K_INDEX, local_r4k_flush_cache_range, vma);
}
static inline void local_r4k_flush_cache_mm(void * args)
@@ -546,7 +575,7 @@ static void r4k_flush_cache_mm(struct mm_struct *mm)
if (!cpu_has_dc_aliases)
return;
- r4k_on_each_cpu(local_r4k_flush_cache_mm, mm);
+ r4k_on_each_cpu(R4K_INDEX, local_r4k_flush_cache_mm, mm);
}
struct flush_cache_page_args {
@@ -641,7 +670,7 @@ static void r4k_flush_cache_page(struct vm_area_struct *vma,
args.addr = addr;
args.pfn = pfn;
- r4k_on_each_cpu(local_r4k_flush_cache_page, &args);
+ r4k_on_each_cpu(R4K_HIT, local_r4k_flush_cache_page, &args);
}
static inline void local_r4k_flush_data_cache_page(void * addr)
@@ -654,7 +683,8 @@ static void r4k_flush_data_cache_page(unsigned long addr)
if (in_atomic())
local_r4k_flush_data_cache_page((void *)addr);
else
- r4k_on_each_cpu(local_r4k_flush_data_cache_page, (void *) addr);
+ r4k_on_each_cpu(R4K_HIT, local_r4k_flush_data_cache_page,
+ (void *) addr);
}
struct flush_icache_range_args {
@@ -715,7 +745,8 @@ static void r4k_flush_icache_range(unsigned long start, unsigned long end)
args.start = start;
args.end = end;
- r4k_on_each_cpu(local_r4k_flush_icache_range_ipi, &args);
+ r4k_on_each_cpu(R4K_HIT | R4K_INDEX, local_r4k_flush_icache_range_ipi,
+ &args);
instruction_hazard();
}
@@ -898,7 +929,7 @@ static void r4k_flush_cache_sigtramp(unsigned long addr)
args.mm = current->mm;
args.addr = addr;
- r4k_on_each_cpu(local_r4k_flush_cache_sigtramp, &args);
+ r4k_on_each_cpu(R4K_HIT, local_r4k_flush_cache_sigtramp, &args);
put_page(args.page);
out:
@@ -941,7 +972,8 @@ static void r4k_flush_kernel_vmap_range(unsigned long vaddr, int size)
args.vaddr = (unsigned long) vaddr;
args.size = size;
- r4k_on_each_cpu(local_r4k_flush_kernel_vmap_range, &args);
+ r4k_on_each_cpu(R4K_HIT | R4K_INDEX, local_r4k_flush_kernel_vmap_range,
+ &args);
}
static inline void rm7k_erratum31(void)
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 08/13] MIPS: c-r4k: Fix valid ASID optimisation
2016-07-13 13:12 [PATCH 00/13] MIPS: c-r4k: SMP cache management fixes James Hogan
` (7 preceding siblings ...)
2016-07-13 13:12 ` [PATCH 07/13] MIPS: c-r4k: Add r4k_on_each_cpu cache op type arg James Hogan
@ 2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` [PATCH 09/13] MIPS: c-r4k: Exclude sibling CPUs in SMP calls James Hogan
` (4 subsequent siblings)
13 siblings, 1 reply; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, Paul Burton, Leonid Yegoshin, linux-mips
Several cache operations are optimised to return early from the SMP call
handler if the memory map in question has no valid ASID on the current
CPU, or any online CPU in the case of MIPS_MT_SMP. The idea is that if a
memory map has never been used on a CPU it shouldn't have cache lines in
need of flushing.
However this doesn't cover all cases when ASIDs for other CPUs need to
be checked:
- Offline VPEs may have recently been online and brought lines into the
(shared) cache, so they should also be checked, rather than only
online CPUs.
- SMP systems with a Coherence Manager (CM), but with MT disabled still
have globalized hit cache ops, but don't use SMP calls, so all present
CPUs should be taken into account.
- R6 systems have a different multithreading implementation, so
MIPS_MT_SMP won't be set, but as above may still have a CM which
globalizes hit cache ops.
Additionally for non-globalized cache operations where an SMP call to a
single VPE in each foreign core is used, it is not necessary to check
every CPU in the system, only sibling CPUs sharing the same first level
cache.
Fix this by making has_valid_asid() take a cache op type argument like
r4k_on_each_cpu(), so it can determine whether r4k_on_each_cpu() will
have done SMP calls to other cores. It can then determine which set of
CPUs to check the ASIDs of based on that, excluding foreign CPUs if an
SMP call will have been performed.
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/mm/c-r4k.c | 48 +++++++++++++++++++++++++++++++++++-------------
1 file changed, 35 insertions(+), 13 deletions(-)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 412052321472..2a4bb5057ebc 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -488,19 +488,41 @@ static void r4k___flush_cache_all(void)
r4k_on_each_cpu(R4K_INDEX, local_r4k___flush_cache_all, NULL);
}
-static inline int has_valid_asid(const struct mm_struct *mm)
+/**
+ * has_valid_asid() - Determine if an mm already has an ASID.
+ * @mm: Memory map.
+ * @type: R4K_HIT or R4K_INDEX, type of cache op.
+ *
+ * Determines whether @mm already has an ASID on any of the CPUs which cache ops
+ * of type @type within an r4k_on_each_cpu() call will affect. If
+ * r4k_on_each_cpu() does an SMP call to a single VPE in each core, then the
+ * scope of the operation is confined to sibling CPUs, otherwise all online CPUs
+ * will need to be checked.
+ *
+ * Must be called in non-preemptive context.
+ *
+ * Returns: 1 if the CPUs affected by @type cache ops have an ASID for @mm.
+ * 0 otherwise.
+ */
+static inline int has_valid_asid(const struct mm_struct *mm, unsigned int type)
{
-#ifdef CONFIG_MIPS_MT_SMP
- int i;
+ unsigned int i;
+ const cpumask_t *mask = cpu_present_mask;
- for_each_online_cpu(i)
+ /* cpu_sibling_map[] undeclared when !CONFIG_SMP */
+#ifdef CONFIG_SMP
+ /*
+ * If r4k_on_each_cpu does SMP calls, it does them to a single VPE in
+ * each foreign core, so we only need to worry about siblings.
+ * Otherwise we need to worry about all present CPUs.
+ */
+ if (r4k_op_needs_ipi(type))
+ mask = &cpu_sibling_map[smp_processor_id()];
+#endif
+ for_each_cpu(i, mask)
if (cpu_context(i, mm))
return 1;
-
return 0;
-#else
- return cpu_context(smp_processor_id(), mm);
-#endif
}
static void r4k__flush_cache_vmap(void)
@@ -522,7 +544,7 @@ static inline void local_r4k_flush_cache_range(void * args)
struct vm_area_struct *vma = args;
int exec = vma->vm_flags & VM_EXEC;
- if (!(has_valid_asid(vma->vm_mm)))
+ if (!has_valid_asid(vma->vm_mm, R4K_INDEX))
return;
/*
@@ -550,7 +572,7 @@ static inline void local_r4k_flush_cache_mm(void * args)
{
struct mm_struct *mm = args;
- if (!has_valid_asid(mm))
+ if (!has_valid_asid(mm, R4K_INDEX))
return;
/*
@@ -600,10 +622,10 @@ static inline void local_r4k_flush_cache_page(void *args)
void *vaddr;
/*
- * If ownes no valid ASID yet, cannot possibly have gotten
+ * If owns no valid ASID yet, cannot possibly have gotten
* this page into the cache.
*/
- if (!has_valid_asid(mm))
+ if (!has_valid_asid(mm, R4K_HIT))
return;
addr &= PAGE_MASK;
@@ -851,7 +873,7 @@ static void local_r4k_flush_cache_sigtramp(void *args)
* If owns no valid ASID yet, cannot possibly have gotten
* this page into the cache.
*/
- if (!has_valid_asid(mm))
+ if (!has_valid_asid(mm, R4K_HIT))
return;
if (mm == current->active_mm) {
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH 08/13] MIPS: c-r4k: Fix valid ASID optimisation
2016-07-13 13:12 ` [PATCH 08/13] MIPS: c-r4k: Fix valid ASID optimisation James Hogan
@ 2016-07-13 13:12 ` James Hogan
0 siblings, 0 replies; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, Paul Burton, Leonid Yegoshin, linux-mips
Several cache operations are optimised to return early from the SMP call
handler if the memory map in question has no valid ASID on the current
CPU, or any online CPU in the case of MIPS_MT_SMP. The idea is that if a
memory map has never been used on a CPU it shouldn't have cache lines in
need of flushing.
However this doesn't cover all cases when ASIDs for other CPUs need to
be checked:
- Offline VPEs may have recently been online and brought lines into the
(shared) cache, so they should also be checked, rather than only
online CPUs.
- SMP systems with a Coherence Manager (CM), but with MT disabled still
have globalized hit cache ops, but don't use SMP calls, so all present
CPUs should be taken into account.
- R6 systems have a different multithreading implementation, so
MIPS_MT_SMP won't be set, but as above may still have a CM which
globalizes hit cache ops.
Additionally for non-globalized cache operations where an SMP call to a
single VPE in each foreign core is used, it is not necessary to check
every CPU in the system, only sibling CPUs sharing the same first level
cache.
Fix this by making has_valid_asid() take a cache op type argument like
r4k_on_each_cpu(), so it can determine whether r4k_on_each_cpu() will
have done SMP calls to other cores. It can then determine which set of
CPUs to check the ASIDs of based on that, excluding foreign CPUs if an
SMP call will have been performed.
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/mm/c-r4k.c | 48 +++++++++++++++++++++++++++++++++++-------------
1 file changed, 35 insertions(+), 13 deletions(-)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 412052321472..2a4bb5057ebc 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -488,19 +488,41 @@ static void r4k___flush_cache_all(void)
r4k_on_each_cpu(R4K_INDEX, local_r4k___flush_cache_all, NULL);
}
-static inline int has_valid_asid(const struct mm_struct *mm)
+/**
+ * has_valid_asid() - Determine if an mm already has an ASID.
+ * @mm: Memory map.
+ * @type: R4K_HIT or R4K_INDEX, type of cache op.
+ *
+ * Determines whether @mm already has an ASID on any of the CPUs which cache ops
+ * of type @type within an r4k_on_each_cpu() call will affect. If
+ * r4k_on_each_cpu() does an SMP call to a single VPE in each core, then the
+ * scope of the operation is confined to sibling CPUs, otherwise all online CPUs
+ * will need to be checked.
+ *
+ * Must be called in non-preemptive context.
+ *
+ * Returns: 1 if the CPUs affected by @type cache ops have an ASID for @mm.
+ * 0 otherwise.
+ */
+static inline int has_valid_asid(const struct mm_struct *mm, unsigned int type)
{
-#ifdef CONFIG_MIPS_MT_SMP
- int i;
+ unsigned int i;
+ const cpumask_t *mask = cpu_present_mask;
- for_each_online_cpu(i)
+ /* cpu_sibling_map[] undeclared when !CONFIG_SMP */
+#ifdef CONFIG_SMP
+ /*
+ * If r4k_on_each_cpu does SMP calls, it does them to a single VPE in
+ * each foreign core, so we only need to worry about siblings.
+ * Otherwise we need to worry about all present CPUs.
+ */
+ if (r4k_op_needs_ipi(type))
+ mask = &cpu_sibling_map[smp_processor_id()];
+#endif
+ for_each_cpu(i, mask)
if (cpu_context(i, mm))
return 1;
-
return 0;
-#else
- return cpu_context(smp_processor_id(), mm);
-#endif
}
static void r4k__flush_cache_vmap(void)
@@ -522,7 +544,7 @@ static inline void local_r4k_flush_cache_range(void * args)
struct vm_area_struct *vma = args;
int exec = vma->vm_flags & VM_EXEC;
- if (!(has_valid_asid(vma->vm_mm)))
+ if (!has_valid_asid(vma->vm_mm, R4K_INDEX))
return;
/*
@@ -550,7 +572,7 @@ static inline void local_r4k_flush_cache_mm(void * args)
{
struct mm_struct *mm = args;
- if (!has_valid_asid(mm))
+ if (!has_valid_asid(mm, R4K_INDEX))
return;
/*
@@ -600,10 +622,10 @@ static inline void local_r4k_flush_cache_page(void *args)
void *vaddr;
/*
- * If ownes no valid ASID yet, cannot possibly have gotten
+ * If owns no valid ASID yet, cannot possibly have gotten
* this page into the cache.
*/
- if (!has_valid_asid(mm))
+ if (!has_valid_asid(mm, R4K_HIT))
return;
addr &= PAGE_MASK;
@@ -851,7 +873,7 @@ static void local_r4k_flush_cache_sigtramp(void *args)
* If owns no valid ASID yet, cannot possibly have gotten
* this page into the cache.
*/
- if (!has_valid_asid(mm))
+ if (!has_valid_asid(mm, R4K_HIT))
return;
if (mm == current->active_mm) {
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 09/13] MIPS: c-r4k: Exclude sibling CPUs in SMP calls
2016-07-13 13:12 [PATCH 00/13] MIPS: c-r4k: SMP cache management fixes James Hogan
` (8 preceding siblings ...)
2016-07-13 13:12 ` [PATCH 08/13] MIPS: c-r4k: Fix valid ASID optimisation James Hogan
@ 2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` [PATCH 10/13] MIPS: c-r4k: Split r4k_flush_kernel_vmap_range() James Hogan
` (3 subsequent siblings)
13 siblings, 1 reply; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle
Cc: James Hogan, Paul Burton, Leonid Yegoshin, Felix Fietkau,
Jayachandran C., linux-mips
When performing SMP calls to foreign cores, exclude sibling CPUs from
the provided map, as we already handle the local core on the current
CPU. This prevents an SMP call from for example core 0, VPE 1 to VPE 0
on the same core.
In the process the cpu_foreign_map cpumask is turned into an array of
cpumasks, so that each CPU has its own version of it which excludes
sibling CPUs. r4k_op_needs_ipi() is also updated to reflect that cache
management SMP calls are not needed when all CPUs are siblings (i.e.
there are no foreign CPUs according to the new cpu_foreign_map[]
semantics which exclude siblings).
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: Felix Fietkau <nbd@nbd.name>
Cc: Jayachandran C. <jchandra@broadcom.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/include/asm/smp.h | 2 +-
arch/mips/kernel/smp.c | 6 ++++--
arch/mips/mm/c-r4k.c | 17 +++++++++++++----
3 files changed, 18 insertions(+), 7 deletions(-)
diff --git a/arch/mips/include/asm/smp.h b/arch/mips/include/asm/smp.h
index 0c534a03bb36..8bc6c70a4030 100644
--- a/arch/mips/include/asm/smp.h
+++ b/arch/mips/include/asm/smp.h
@@ -23,7 +23,7 @@
extern int smp_num_siblings;
extern cpumask_t cpu_sibling_map[];
extern cpumask_t cpu_core_map[];
-extern cpumask_t cpu_foreign_map;
+extern cpumask_t cpu_foreign_map[];
#define raw_smp_processor_id() (current_thread_info()->cpu)
diff --git a/arch/mips/kernel/smp.c b/arch/mips/kernel/smp.c
index 783d5f50ab9d..f95f094f36e4 100644
--- a/arch/mips/kernel/smp.c
+++ b/arch/mips/kernel/smp.c
@@ -72,7 +72,7 @@ EXPORT_SYMBOL(cpu_core_map);
* A logcal cpu mask containing only one VPE per core to
* reduce the number of IPIs on large MT systems.
*/
-cpumask_t cpu_foreign_map __read_mostly;
+cpumask_t cpu_foreign_map[NR_CPUS] __read_mostly;
EXPORT_SYMBOL(cpu_foreign_map);
/* representing cpus for which sibling maps can be computed */
@@ -141,7 +141,9 @@ void calculate_cpu_foreign_map(void)
cpumask_set_cpu(i, &temp_foreign_map);
}
- cpumask_copy(&cpu_foreign_map, &temp_foreign_map);
+ for_each_online_cpu(i)
+ cpumask_andnot(&cpu_foreign_map[i],
+ &temp_foreign_map, &cpu_sibling_map[i]);
}
struct plat_smp_ops *mp_ops;
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 2a4bb5057ebc..57374f0c33f2 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -56,7 +56,9 @@
* @type: Type of cache operations (R4K_HIT or R4K_INDEX).
*
* Decides whether a cache op needs to be performed on every core in the system.
- * This may change depending on the @type of cache operation.
+ * This may change depending on the @type of cache operation, as well as the set
+ * of online CPUs, so preemption should be disabled by the caller to prevent CPU
+ * hotplug from changing the result.
*
* Returns: 1 if the cache operation @type should be done on every core in
* the system.
@@ -71,9 +73,15 @@ static inline bool r4k_op_needs_ipi(unsigned int type)
/*
* Hardware doesn't globalize the required cache ops, so SMP calls may
- * be needed.
+ * be needed, but only if there are foreign CPUs (non-siblings with
+ * separate caches).
*/
- return true;
+ /* cpu_foreign_map[] undeclared when !CONFIG_SMP */
+#ifdef CONFIG_SMP
+ return !cpumask_empty(&cpu_foreign_map[0]);
+#else
+ return false;
+#endif
}
/*
@@ -90,7 +98,8 @@ static inline void r4k_on_each_cpu(unsigned int type,
{
preempt_disable();
if (r4k_op_needs_ipi(type))
- smp_call_function_many(&cpu_foreign_map, func, info, 1);
+ smp_call_function_many(&cpu_foreign_map[smp_processor_id()],
+ func, info, 1);
func(info);
preempt_enable();
}
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH 09/13] MIPS: c-r4k: Exclude sibling CPUs in SMP calls
2016-07-13 13:12 ` [PATCH 09/13] MIPS: c-r4k: Exclude sibling CPUs in SMP calls James Hogan
@ 2016-07-13 13:12 ` James Hogan
0 siblings, 0 replies; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle
Cc: James Hogan, Paul Burton, Leonid Yegoshin, Felix Fietkau,
Jayachandran C., linux-mips
When performing SMP calls to foreign cores, exclude sibling CPUs from
the provided map, as we already handle the local core on the current
CPU. This prevents an SMP call from for example core 0, VPE 1 to VPE 0
on the same core.
In the process the cpu_foreign_map cpumask is turned into an array of
cpumasks, so that each CPU has its own version of it which excludes
sibling CPUs. r4k_op_needs_ipi() is also updated to reflect that cache
management SMP calls are not needed when all CPUs are siblings (i.e.
there are no foreign CPUs according to the new cpu_foreign_map[]
semantics which exclude siblings).
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: Felix Fietkau <nbd@nbd.name>
Cc: Jayachandran C. <jchandra@broadcom.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/include/asm/smp.h | 2 +-
arch/mips/kernel/smp.c | 6 ++++--
arch/mips/mm/c-r4k.c | 17 +++++++++++++----
3 files changed, 18 insertions(+), 7 deletions(-)
diff --git a/arch/mips/include/asm/smp.h b/arch/mips/include/asm/smp.h
index 0c534a03bb36..8bc6c70a4030 100644
--- a/arch/mips/include/asm/smp.h
+++ b/arch/mips/include/asm/smp.h
@@ -23,7 +23,7 @@
extern int smp_num_siblings;
extern cpumask_t cpu_sibling_map[];
extern cpumask_t cpu_core_map[];
-extern cpumask_t cpu_foreign_map;
+extern cpumask_t cpu_foreign_map[];
#define raw_smp_processor_id() (current_thread_info()->cpu)
diff --git a/arch/mips/kernel/smp.c b/arch/mips/kernel/smp.c
index 783d5f50ab9d..f95f094f36e4 100644
--- a/arch/mips/kernel/smp.c
+++ b/arch/mips/kernel/smp.c
@@ -72,7 +72,7 @@ EXPORT_SYMBOL(cpu_core_map);
* A logcal cpu mask containing only one VPE per core to
* reduce the number of IPIs on large MT systems.
*/
-cpumask_t cpu_foreign_map __read_mostly;
+cpumask_t cpu_foreign_map[NR_CPUS] __read_mostly;
EXPORT_SYMBOL(cpu_foreign_map);
/* representing cpus for which sibling maps can be computed */
@@ -141,7 +141,9 @@ void calculate_cpu_foreign_map(void)
cpumask_set_cpu(i, &temp_foreign_map);
}
- cpumask_copy(&cpu_foreign_map, &temp_foreign_map);
+ for_each_online_cpu(i)
+ cpumask_andnot(&cpu_foreign_map[i],
+ &temp_foreign_map, &cpu_sibling_map[i]);
}
struct plat_smp_ops *mp_ops;
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 2a4bb5057ebc..57374f0c33f2 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -56,7 +56,9 @@
* @type: Type of cache operations (R4K_HIT or R4K_INDEX).
*
* Decides whether a cache op needs to be performed on every core in the system.
- * This may change depending on the @type of cache operation.
+ * This may change depending on the @type of cache operation, as well as the set
+ * of online CPUs, so preemption should be disabled by the caller to prevent CPU
+ * hotplug from changing the result.
*
* Returns: 1 if the cache operation @type should be done on every core in
* the system.
@@ -71,9 +73,15 @@ static inline bool r4k_op_needs_ipi(unsigned int type)
/*
* Hardware doesn't globalize the required cache ops, so SMP calls may
- * be needed.
+ * be needed, but only if there are foreign CPUs (non-siblings with
+ * separate caches).
*/
- return true;
+ /* cpu_foreign_map[] undeclared when !CONFIG_SMP */
+#ifdef CONFIG_SMP
+ return !cpumask_empty(&cpu_foreign_map[0]);
+#else
+ return false;
+#endif
}
/*
@@ -90,7 +98,8 @@ static inline void r4k_on_each_cpu(unsigned int type,
{
preempt_disable();
if (r4k_op_needs_ipi(type))
- smp_call_function_many(&cpu_foreign_map, func, info, 1);
+ smp_call_function_many(&cpu_foreign_map[smp_processor_id()],
+ func, info, 1);
func(info);
preempt_enable();
}
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 10/13] MIPS: c-r4k: Split r4k_flush_kernel_vmap_range()
2016-07-13 13:12 [PATCH 00/13] MIPS: c-r4k: SMP cache management fixes James Hogan
` (9 preceding siblings ...)
2016-07-13 13:12 ` [PATCH 09/13] MIPS: c-r4k: Exclude sibling CPUs in SMP calls James Hogan
@ 2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` [PATCH 11/13] MIPS: c-r4k: Local flush_icache_range cache op override James Hogan
` (2 subsequent siblings)
13 siblings, 1 reply; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, Paul Burton, Leonid Yegoshin, linux-mips
Split the operation of r4k_flush_kernel_vmap_range() into separate
SMP callbacks for the indexed cache flush and hit cache flush cases,
since the logic to determine which to use can be determined by the
initiating CPU prior to doing any SMP calls.
This will help when we change r4k_on_each_cpu() to distinguish indexed
and hit cache ops in a later patch, preventing globalized hit cache ops
being performed redundantly on multiple CPUs.
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/mm/c-r4k.c | 25 +++++++++++++++++--------
1 file changed, 17 insertions(+), 8 deletions(-)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 57374f0c33f2..004cf41dd717 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -978,6 +978,15 @@ struct flush_kernel_vmap_range_args {
int size;
};
+static inline void local_r4k_flush_kernel_vmap_range_index(void *args)
+{
+ /*
+ * Aliases only affect the primary caches so don't bother with
+ * S-caches or T-caches.
+ */
+ r4k_blast_dcache();
+}
+
static inline void local_r4k_flush_kernel_vmap_range(void *args)
{
struct flush_kernel_vmap_range_args *vmra = args;
@@ -988,12 +997,8 @@ static inline void local_r4k_flush_kernel_vmap_range(void *args)
* Aliases only affect the primary caches so don't bother with
* S-caches or T-caches.
*/
- if (size >= dcache_size)
- r4k_blast_dcache();
- else {
- R4600_HIT_CACHEOP_WAR_IMPL;
- blast_dcache_range(vaddr, vaddr + size);
- }
+ R4600_HIT_CACHEOP_WAR_IMPL;
+ blast_dcache_range(vaddr, vaddr + size);
}
static void r4k_flush_kernel_vmap_range(unsigned long vaddr, int size)
@@ -1003,8 +1008,12 @@ static void r4k_flush_kernel_vmap_range(unsigned long vaddr, int size)
args.vaddr = (unsigned long) vaddr;
args.size = size;
- r4k_on_each_cpu(R4K_HIT | R4K_INDEX, local_r4k_flush_kernel_vmap_range,
- &args);
+ if (size >= dcache_size)
+ r4k_on_each_cpu(R4K_INDEX,
+ local_r4k_flush_kernel_vmap_range_index, NULL);
+ else
+ r4k_on_each_cpu(R4K_HIT, local_r4k_flush_kernel_vmap_range,
+ &args);
}
static inline void rm7k_erratum31(void)
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH 10/13] MIPS: c-r4k: Split r4k_flush_kernel_vmap_range()
2016-07-13 13:12 ` [PATCH 10/13] MIPS: c-r4k: Split r4k_flush_kernel_vmap_range() James Hogan
@ 2016-07-13 13:12 ` James Hogan
0 siblings, 0 replies; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, Paul Burton, Leonid Yegoshin, linux-mips
Split the operation of r4k_flush_kernel_vmap_range() into separate
SMP callbacks for the indexed cache flush and hit cache flush cases,
since the logic to determine which to use can be determined by the
initiating CPU prior to doing any SMP calls.
This will help when we change r4k_on_each_cpu() to distinguish indexed
and hit cache ops in a later patch, preventing globalized hit cache ops
being performed redundantly on multiple CPUs.
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/mm/c-r4k.c | 25 +++++++++++++++++--------
1 file changed, 17 insertions(+), 8 deletions(-)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 57374f0c33f2..004cf41dd717 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -978,6 +978,15 @@ struct flush_kernel_vmap_range_args {
int size;
};
+static inline void local_r4k_flush_kernel_vmap_range_index(void *args)
+{
+ /*
+ * Aliases only affect the primary caches so don't bother with
+ * S-caches or T-caches.
+ */
+ r4k_blast_dcache();
+}
+
static inline void local_r4k_flush_kernel_vmap_range(void *args)
{
struct flush_kernel_vmap_range_args *vmra = args;
@@ -988,12 +997,8 @@ static inline void local_r4k_flush_kernel_vmap_range(void *args)
* Aliases only affect the primary caches so don't bother with
* S-caches or T-caches.
*/
- if (size >= dcache_size)
- r4k_blast_dcache();
- else {
- R4600_HIT_CACHEOP_WAR_IMPL;
- blast_dcache_range(vaddr, vaddr + size);
- }
+ R4600_HIT_CACHEOP_WAR_IMPL;
+ blast_dcache_range(vaddr, vaddr + size);
}
static void r4k_flush_kernel_vmap_range(unsigned long vaddr, int size)
@@ -1003,8 +1008,12 @@ static void r4k_flush_kernel_vmap_range(unsigned long vaddr, int size)
args.vaddr = (unsigned long) vaddr;
args.size = size;
- r4k_on_each_cpu(R4K_HIT | R4K_INDEX, local_r4k_flush_kernel_vmap_range,
- &args);
+ if (size >= dcache_size)
+ r4k_on_each_cpu(R4K_INDEX,
+ local_r4k_flush_kernel_vmap_range_index, NULL);
+ else
+ r4k_on_each_cpu(R4K_HIT, local_r4k_flush_kernel_vmap_range,
+ &args);
}
static inline void rm7k_erratum31(void)
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 11/13] MIPS: c-r4k: Local flush_icache_range cache op override
2016-07-13 13:12 [PATCH 00/13] MIPS: c-r4k: SMP cache management fixes James Hogan
` (10 preceding siblings ...)
2016-07-13 13:12 ` [PATCH 10/13] MIPS: c-r4k: Split r4k_flush_kernel_vmap_range() James Hogan
@ 2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` [PATCH 12/13] MIPS: c-r4k: Avoid small flush_icache_range SMP calls James Hogan
2016-07-13 13:12 ` [PATCH 13/13] MIPS: c-r4k: Use SMP calls for CM indexed cache ops James Hogan
13 siblings, 1 reply; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, Paul Burton, Leonid Yegoshin, linux-mips
Allow the permitted cache op types used by
local_r4k_flush_icache_range_ipi() to be overridden by the SMP caller.
This will allow SMP calls to be avoided under certain circumstances,
falling back to a single CPU performing globalized hit cache ops only.
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/mm/c-r4k.c | 24 ++++++++++++++++++------
1 file changed, 18 insertions(+), 6 deletions(-)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 004cf41dd717..cfcb336f57a0 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -721,12 +721,16 @@ static void r4k_flush_data_cache_page(unsigned long addr)
struct flush_icache_range_args {
unsigned long start;
unsigned long end;
+ unsigned int type;
};
-static inline void local_r4k_flush_icache_range(unsigned long start, unsigned long end)
+static inline void __local_r4k_flush_icache_range(unsigned long start,
+ unsigned long end,
+ unsigned int type)
{
if (!cpu_has_ic_fills_f_dc) {
- if (end - start >= dcache_size) {
+ if (type == R4K_INDEX ||
+ (type & R4K_INDEX && end - start >= dcache_size)) {
r4k_blast_dcache();
} else {
R4600_HIT_CACHEOP_WAR_IMPL;
@@ -734,7 +738,8 @@ static inline void local_r4k_flush_icache_range(unsigned long start, unsigned lo
}
}
- if (end - start > icache_size)
+ if (type == R4K_INDEX ||
+ (type & R4K_INDEX && end - start > icache_size))
r4k_blast_icache();
else {
switch (boot_cpu_type()) {
@@ -760,13 +765,20 @@ static inline void local_r4k_flush_icache_range(unsigned long start, unsigned lo
#endif
}
+static inline void local_r4k_flush_icache_range(unsigned long start,
+ unsigned long end)
+{
+ __local_r4k_flush_icache_range(start, end, R4K_HIT | R4K_INDEX);
+}
+
static inline void local_r4k_flush_icache_range_ipi(void *args)
{
struct flush_icache_range_args *fir_args = args;
unsigned long start = fir_args->start;
unsigned long end = fir_args->end;
+ unsigned int type = fir_args->type;
- local_r4k_flush_icache_range(start, end);
+ __local_r4k_flush_icache_range(start, end, type);
}
static void r4k_flush_icache_range(unsigned long start, unsigned long end)
@@ -775,9 +787,9 @@ static void r4k_flush_icache_range(unsigned long start, unsigned long end)
args.start = start;
args.end = end;
+ args.type = R4K_HIT | R4K_INDEX;
- r4k_on_each_cpu(R4K_HIT | R4K_INDEX, local_r4k_flush_icache_range_ipi,
- &args);
+ r4k_on_each_cpu(args.type, local_r4k_flush_icache_range_ipi, &args);
instruction_hazard();
}
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH 11/13] MIPS: c-r4k: Local flush_icache_range cache op override
2016-07-13 13:12 ` [PATCH 11/13] MIPS: c-r4k: Local flush_icache_range cache op override James Hogan
@ 2016-07-13 13:12 ` James Hogan
0 siblings, 0 replies; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, Paul Burton, Leonid Yegoshin, linux-mips
Allow the permitted cache op types used by
local_r4k_flush_icache_range_ipi() to be overridden by the SMP caller.
This will allow SMP calls to be avoided under certain circumstances,
falling back to a single CPU performing globalized hit cache ops only.
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/mm/c-r4k.c | 24 ++++++++++++++++++------
1 file changed, 18 insertions(+), 6 deletions(-)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 004cf41dd717..cfcb336f57a0 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -721,12 +721,16 @@ static void r4k_flush_data_cache_page(unsigned long addr)
struct flush_icache_range_args {
unsigned long start;
unsigned long end;
+ unsigned int type;
};
-static inline void local_r4k_flush_icache_range(unsigned long start, unsigned long end)
+static inline void __local_r4k_flush_icache_range(unsigned long start,
+ unsigned long end,
+ unsigned int type)
{
if (!cpu_has_ic_fills_f_dc) {
- if (end - start >= dcache_size) {
+ if (type == R4K_INDEX ||
+ (type & R4K_INDEX && end - start >= dcache_size)) {
r4k_blast_dcache();
} else {
R4600_HIT_CACHEOP_WAR_IMPL;
@@ -734,7 +738,8 @@ static inline void local_r4k_flush_icache_range(unsigned long start, unsigned lo
}
}
- if (end - start > icache_size)
+ if (type == R4K_INDEX ||
+ (type & R4K_INDEX && end - start > icache_size))
r4k_blast_icache();
else {
switch (boot_cpu_type()) {
@@ -760,13 +765,20 @@ static inline void local_r4k_flush_icache_range(unsigned long start, unsigned lo
#endif
}
+static inline void local_r4k_flush_icache_range(unsigned long start,
+ unsigned long end)
+{
+ __local_r4k_flush_icache_range(start, end, R4K_HIT | R4K_INDEX);
+}
+
static inline void local_r4k_flush_icache_range_ipi(void *args)
{
struct flush_icache_range_args *fir_args = args;
unsigned long start = fir_args->start;
unsigned long end = fir_args->end;
+ unsigned int type = fir_args->type;
- local_r4k_flush_icache_range(start, end);
+ __local_r4k_flush_icache_range(start, end, type);
}
static void r4k_flush_icache_range(unsigned long start, unsigned long end)
@@ -775,9 +787,9 @@ static void r4k_flush_icache_range(unsigned long start, unsigned long end)
args.start = start;
args.end = end;
+ args.type = R4K_HIT | R4K_INDEX;
- r4k_on_each_cpu(R4K_HIT | R4K_INDEX, local_r4k_flush_icache_range_ipi,
- &args);
+ r4k_on_each_cpu(args.type, local_r4k_flush_icache_range_ipi, &args);
instruction_hazard();
}
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 12/13] MIPS: c-r4k: Avoid small flush_icache_range SMP calls
2016-07-13 13:12 [PATCH 00/13] MIPS: c-r4k: SMP cache management fixes James Hogan
` (11 preceding siblings ...)
2016-07-13 13:12 ` [PATCH 11/13] MIPS: c-r4k: Local flush_icache_range cache op override James Hogan
@ 2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` [PATCH 13/13] MIPS: c-r4k: Use SMP calls for CM indexed cache ops James Hogan
13 siblings, 1 reply; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, Paul Burton, Leonid Yegoshin, linux-mips
Avoid SMP calls for flushing small icache ranges. On non-CM platforms,
and CM platforms too after we make r4k_on_each_cpu() take the cache op
type into account, it will be called on multiple CPUs due to the
possibility that local_r4k_flush_icache_range_ipi() could do
non-globalized indexed cache ops. This rougly copies the range size
check out into r4k_flush_icache_range(), which can disallow indexed
cache ops and allow r4k_on_each_cpu() to skip the SMP call.
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/mm/c-r4k.c | 21 +++++++++++++++++++++
1 file changed, 21 insertions(+)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index cfcb336f57a0..8016babe5c84 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -784,12 +784,33 @@ static inline void local_r4k_flush_icache_range_ipi(void *args)
static void r4k_flush_icache_range(unsigned long start, unsigned long end)
{
struct flush_icache_range_args args;
+ unsigned long size, cache_size;
args.start = start;
args.end = end;
args.type = R4K_HIT | R4K_INDEX;
+ /*
+ * Indexed cache ops require an SMP call.
+ * Consider if that can or should be avoided.
+ */
+ preempt_disable();
+ if (r4k_op_needs_ipi(R4K_INDEX) && !r4k_op_needs_ipi(R4K_HIT)) {
+ /*
+ * If address-based cache ops don't require an SMP call, then
+ * use them exclusively for small flushes.
+ */
+ size = start - end;
+ cache_size = icache_size;
+ if (!cpu_has_ic_fills_f_dc) {
+ size *= 2;
+ cache_size += dcache_size;
+ }
+ if (size <= cache_size)
+ args.type &= ~R4K_INDEX;
+ }
r4k_on_each_cpu(args.type, local_r4k_flush_icache_range_ipi, &args);
+ preempt_enable();
instruction_hazard();
}
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH 12/13] MIPS: c-r4k: Avoid small flush_icache_range SMP calls
2016-07-13 13:12 ` [PATCH 12/13] MIPS: c-r4k: Avoid small flush_icache_range SMP calls James Hogan
@ 2016-07-13 13:12 ` James Hogan
0 siblings, 0 replies; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, Paul Burton, Leonid Yegoshin, linux-mips
Avoid SMP calls for flushing small icache ranges. On non-CM platforms,
and CM platforms too after we make r4k_on_each_cpu() take the cache op
type into account, it will be called on multiple CPUs due to the
possibility that local_r4k_flush_icache_range_ipi() could do
non-globalized indexed cache ops. This rougly copies the range size
check out into r4k_flush_icache_range(), which can disallow indexed
cache ops and allow r4k_on_each_cpu() to skip the SMP call.
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/mm/c-r4k.c | 21 +++++++++++++++++++++
1 file changed, 21 insertions(+)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index cfcb336f57a0..8016babe5c84 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -784,12 +784,33 @@ static inline void local_r4k_flush_icache_range_ipi(void *args)
static void r4k_flush_icache_range(unsigned long start, unsigned long end)
{
struct flush_icache_range_args args;
+ unsigned long size, cache_size;
args.start = start;
args.end = end;
args.type = R4K_HIT | R4K_INDEX;
+ /*
+ * Indexed cache ops require an SMP call.
+ * Consider if that can or should be avoided.
+ */
+ preempt_disable();
+ if (r4k_op_needs_ipi(R4K_INDEX) && !r4k_op_needs_ipi(R4K_HIT)) {
+ /*
+ * If address-based cache ops don't require an SMP call, then
+ * use them exclusively for small flushes.
+ */
+ size = start - end;
+ cache_size = icache_size;
+ if (!cpu_has_ic_fills_f_dc) {
+ size *= 2;
+ cache_size += dcache_size;
+ }
+ if (size <= cache_size)
+ args.type &= ~R4K_INDEX;
+ }
r4k_on_each_cpu(args.type, local_r4k_flush_icache_range_ipi, &args);
+ preempt_enable();
instruction_hazard();
}
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 13/13] MIPS: c-r4k: Use SMP calls for CM indexed cache ops
2016-07-13 13:12 [PATCH 00/13] MIPS: c-r4k: SMP cache management fixes James Hogan
` (12 preceding siblings ...)
2016-07-13 13:12 ` [PATCH 12/13] MIPS: c-r4k: Avoid small flush_icache_range SMP calls James Hogan
@ 2016-07-13 13:12 ` James Hogan
2016-07-13 13:12 ` James Hogan
13 siblings, 1 reply; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, Paul Burton, Leonid Yegoshin, linux-mips
The MIPS Coherence Manager (CM) can propagate address-based ("hit")
cache operations to other cores in the coherent system, alleviating
software of the need to use SMP calls, however indexed cache operations
are not propagated by hardware since doing so makes no sense for
separate caches.
Update r4k_op_needs_ipi() to report that only hit cache operations are
globalized by the CM, requiring indexed cache operations to be
globalized by software via an SMP call.
r4k_on_each_cpu() previously had a special case for CONFIG_MIPS_MT_SMP,
intended to avoid the SMP calls when the only other CPUs in the system
were other VPEs in the same core, and hence sharing the same caches.
This was changed by commit cccf34e9411c ("MIPS: c-r4k: Fix cache
flushing for MT cores") to apparently handle multi-core multi-VPE
systems, but it focussed mainly on hit cache ops, so the SMP calls were
still disabled entirely for CM systems.
This doesn't normally cause problems, but tests can be written to hit
these corner cases by using multiple threads, or changing task
affinities to force the process to migrate cores. For example the
failure of mprotect RW->RX to globally sync icaches (via
flush_cache_range) can be detected by modifying and mprotecting a code
page on one core, and migrating to a different core to execute from it.
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/mm/c-r4k.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 8016babe5c84..3227a0a267f5 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -68,7 +68,7 @@
static inline bool r4k_op_needs_ipi(unsigned int type)
{
/* The MIPS Coherence Manager (CM) globalizes address-based cache ops */
- if (mips_cm_present())
+ if (type == R4K_HIT && mips_cm_present())
return false;
/*
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH 13/13] MIPS: c-r4k: Use SMP calls for CM indexed cache ops
2016-07-13 13:12 ` [PATCH 13/13] MIPS: c-r4k: Use SMP calls for CM indexed cache ops James Hogan
@ 2016-07-13 13:12 ` James Hogan
0 siblings, 0 replies; 28+ messages in thread
From: James Hogan @ 2016-07-13 13:12 UTC (permalink / raw)
To: Ralf Baechle; +Cc: James Hogan, Paul Burton, Leonid Yegoshin, linux-mips
The MIPS Coherence Manager (CM) can propagate address-based ("hit")
cache operations to other cores in the coherent system, alleviating
software of the need to use SMP calls, however indexed cache operations
are not propagated by hardware since doing so makes no sense for
separate caches.
Update r4k_op_needs_ipi() to report that only hit cache operations are
globalized by the CM, requiring indexed cache operations to be
globalized by software via an SMP call.
r4k_on_each_cpu() previously had a special case for CONFIG_MIPS_MT_SMP,
intended to avoid the SMP calls when the only other CPUs in the system
were other VPEs in the same core, and hence sharing the same caches.
This was changed by commit cccf34e9411c ("MIPS: c-r4k: Fix cache
flushing for MT cores") to apparently handle multi-core multi-VPE
systems, but it focussed mainly on hit cache ops, so the SMP calls were
still disabled entirely for CM systems.
This doesn't normally cause problems, but tests can be written to hit
these corner cases by using multiple threads, or changing task
affinities to force the process to migrate cores. For example the
failure of mprotect RW->RX to globally sync icaches (via
flush_cache_range) can be detected by modifying and mprotecting a code
page on one core, and migrating to a different core to execute from it.
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
---
arch/mips/mm/c-r4k.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 8016babe5c84..3227a0a267f5 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -68,7 +68,7 @@
static inline bool r4k_op_needs_ipi(unsigned int type)
{
/* The MIPS Coherence Manager (CM) globalizes address-based cache ops */
- if (mips_cm_present())
+ if (type == R4K_HIT && mips_cm_present())
return false;
/*
--
2.4.10
^ permalink raw reply related [flat|nested] 28+ messages in thread