* [PATCH 00/58] Add support for Enhanced Virtual Addressing
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Hi,
This patchset adds support for the Enhanced Virtual Addressing feature
of the MIPS 3.5 ISA currently present in the *Aptiv processors.
In Release 3, the memory architecture defines the partitioning of memory
space into fixed‐size segments, whose access properties and cacheability
attributes are programmable. The "mapped" and "unmapped" segment‐access
properties can now be modified so as to extend system memory space
(kseg0/kseg1) to include unmapped access to user space (xkseg), a
feature called Enhanced Virtual Addressing (EVA). Instructions that
facilitate system read/write access to xkseg have also been added to the
R3 ISA (lwe, swe, lbue, etc).
This patchset also adds EVA support for the Malta board. The unmapped
address space (kseg0/kseg1) has been extended to 2GB.
If you want to try it, you can use the 'eva-next' branch on
the following git repository
https://github.com/hwoarang/linux.git eva-next
The patchset applies on top of the upstream-sfr/mips-for-linux-next tree.
Leonid Yegoshin (18):
MIPS: Kconfig: Add Kconfig symbols for EVA support
MIPS: asm: Add prefetch instruction for EVA
MIPS: uapi: inst: Add new EVA opcodes
MIPS: uapi: inst: Add instruction format for SPECIAL3 instructions
MIPS: traps: Set correct address limit for breakpoints and traps
MIPS: asm: uaccess: Disable unaligned access macros for EVA
MIPS: kernel: unaligned: Handle unaligned accesses for EVA
MIPS: asm: checksum: Split kernel and user copy operations
MIPS: asm: checksum: Add MIPS specific csum_and_copy_from_user
function
MIPS: kernel: signal: Prevent save/restore FPU context in user memory
MIPS: asm: r4kcache: Build flushing code for instruction cache
MIPS: asm: r4kcache: Add protected cache operation for EVA
MIPS: asm: r4kcache: Add EVA cache flushing functions
MIPS: kernel: {ftrace,kgdb}: Set correct address limit for cache
flushes
MIPS: asm: page: Allow __pa_symbol overrides
MIPS: mm: c-r4k: Build EVA {d,i}cache flushing functions
MIPS: mm: c-r4k: Flush scache to avoid cache aliases
MIPS: malta: malta-init: Fix System Controller memory mapping for EVA
Markos Chandras (40):
MIPS: asm: Add wrappers for EVA/non-EVA instructions
MIPS: futex: Add EVA support for futex operations
MIPS: kernel: scall32-o32: Use EVA wrappers to fetch syscall arguments
MIPS: kernel: traps: Whitespace clean up
MIPS: lib: strnlen_user: Use macro to build the strnlen_user symbol
MIPS: lib: strnlen_user: Add EVA support
MIPS: lib: strlen_user: Use macro to build the strlen_user symbol
MIPS: lib: strlen_user: Add EVA support
MIPS: lib: strncpy_user: Use macro to build the strncpy_from_user
symbol
MIPS: lib: strncpy_user: Add EVA support
MIPS: lib: memcpy: Merge EXC and load/store macros
MIPS: lib: memcpy: Split source and destination prefetch macros
MIPS: lib: memcpy: Use macro to build the copy_user code
MIPS: lib: memcpy: Add EVA support
MIPS: lib: memset: Whitespace fixes
MIPS: lib: memset: Use macro to build the __bzero symbol
MIPS: lib: memset: Add EVA support for the __bzero function.
MIPS: asm: uaccess: Add instruction argument to __{put,get}_user_asm
MIPS: asm: uaccess: Move duplicated code to common function
MIPS: asm: uaccess: Use EVA instructions wrappers
MIPS: asm: uaccess: Rename {get,put}_user_asm macros
MIPS: asm: uaccess: Add EVA support to copy_{in, to,from}_user
MIPS: asm: uaccess: Add EVA support for str*_user operations
MIPS: kernel: unaligned: Add EVA instruction wrappers
MIPS: checksum: Split the 'copy_user' symbol
MIPS: lib: csum_partial: Merge EXC and load/store macros
MIPS: lib: csum_partial: Add macro to build csum_partial symbols
MIPS: lib: csum_partial: Add EVA support
MIPS: asm: cpu: Add cpu flag for Enhanced Virtual Addressing
MIPS: kernel: cpu-probe: Enable EVA option on supported cores
MIPS: kernel: proc: Add EVA to the list of CPU features
MIPS: mm: init: Add free_init_pages() callback for EVA
MIPS: mm: c-r4k: Add support for flushing user pages from cache
MIPS: malta: Configure Segment Control registers for EVA boot
MIPS: malta: spaces.h: Add spaces.h file for Malta (EVA)
MIPS: malta: malta-memory: Add support for the 'ememsize' variable
MIPS: malta: malta-memory: Use the PHYS_OFFSET to build the memory map
MIPS: malta: malta-memory: Add free_init_pages_eva() callback
MIPS: malta: Add support for SMP EVA
MIPS: Enable MIPS 3.5 features on Malta
arch/mips/Kconfig | 29 +-
arch/mips/include/asm/asm.h | 130 +++++
arch/mips/include/asm/bootinfo.h | 2 +
arch/mips/include/asm/checksum.h | 44 +-
arch/mips/include/asm/cpu-features.h | 4 +-
arch/mips/include/asm/cpu.h | 1 +
arch/mips/include/asm/futex.h | 9 +-
arch/mips/include/asm/fw/fw.h | 2 +-
.../include/asm/mach-malta/kernel-entry-init.h | 115 ++++-
arch/mips/include/asm/mach-malta/spaces.h | 46 ++
arch/mips/include/asm/page.h | 2 +
arch/mips/include/asm/r4kcache.h | 173 ++++++-
arch/mips/include/asm/uaccess.h | 559 ++++++++++++++++-----
arch/mips/include/uapi/asm/inst.h | 25 +-
arch/mips/kernel/cpu-probe.c | 3 +
arch/mips/kernel/ftrace.c | 4 +
arch/mips/kernel/head.S | 2 +-
arch/mips/kernel/kgdb.c | 17 +-
arch/mips/kernel/mips_ksyms.c | 24 +-
arch/mips/kernel/proc.c | 1 +
arch/mips/kernel/scall32-o32.S | 9 +-
arch/mips/kernel/signal.c | 26 +
arch/mips/kernel/traps.c | 29 +-
arch/mips/kernel/unaligned.c | 135 ++++-
arch/mips/lib/csum_partial.S | 282 +++++++----
arch/mips/lib/memcpy.S | 416 ++++++++++-----
arch/mips/lib/memset.S | 146 ++++--
arch/mips/lib/strlen_user.S | 36 +-
arch/mips/lib/strncpy_user.S | 40 +-
arch/mips/lib/strnlen_user.S | 36 +-
arch/mips/mm/c-r4k.c | 64 ++-
arch/mips/mm/init.c | 12 +-
arch/mips/mti-malta/malta-init.c | 13 +
arch/mips/mti-malta/malta-memory.c | 58 ++-
arch/mips/mti-malta/malta-setup.c | 4 +
35 files changed, 1974 insertions(+), 524 deletions(-)
create mode 100644 arch/mips/include/asm/mach-malta/spaces.h
--
1.8.5.3
^ permalink raw reply [flat|nested] 126+ messages in thread* [PATCH 01/58] MIPS: Kconfig: Add Kconfig symbols for EVA support
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Add basic Kconfig support for EVA. Not selectable by any platform
at this point.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/Kconfig | 28 +++++++++++++++++++++++++++-
1 file changed, 27 insertions(+), 1 deletion(-)
diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index f4c78c9..4230c7a 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -1446,6 +1446,26 @@ config CPU_XLP
Netlogic Microsystems XLP processors.
endchoice
+config CPU_MIPS32_3_5_FEATURES
+ bool "MIPS32 Release 3.5 Features"
+ depends on SYS_HAS_CPU_MIPS32_R3_5
+ depends on CPU_MIPS32_R2
+ help
+ Choose this option to build a kernel for release 2 or later of the
+ MIPS32 architecture including features from the 3.5 release such as
+ support for Enhanced Virtual Addressing (EVA).
+
+config CPU_MIPS32_3_5_EVA
+ bool "Enhanced Virtual Addressing (EVA)"
+ depends on CPU_MIPS32_3_5_FEATURES
+ select EVA
+ default y
+ help
+ Choose this option if you want to enable the Enhanced Virtual
+ Addressing (EVA) on your MIPS32 core (such as proAptiv).
+ One of its primary benefits is an increase in the maximum size
+ of lowmem (up to 3GB). If unsure, say 'N' here.
+
if CPU_LOONGSON2F
config CPU_NOP_WORKAROUNDS
bool
@@ -1539,6 +1559,9 @@ config SYS_HAS_CPU_MIPS32_R1
config SYS_HAS_CPU_MIPS32_R2
bool
+config SYS_HAS_CPU_MIPS32_R3_5
+ bool
+
config SYS_HAS_CPU_MIPS64_R1
bool
@@ -1658,6 +1681,9 @@ config CPU_MIPSR2
bool
default y if CPU_MIPS32_R2 || CPU_MIPS64_R2 || CPU_CAVIUM_OCTEON
+config EVA
+ bool
+
config SYS_SUPPORTS_32BIT_KERNEL
bool
config SYS_SUPPORTS_64BIT_KERNEL
@@ -2095,7 +2121,7 @@ config CPU_R4400_WORKAROUNDS
#
config HIGHMEM
bool "High Memory Support"
- depends on 32BIT && CPU_SUPPORTS_HIGHMEM && SYS_SUPPORTS_HIGHMEM
+ depends on 32BIT && CPU_SUPPORTS_HIGHMEM && SYS_SUPPORTS_HIGHMEM && !CPU_MIPS32_3_5_EVA
config CPU_SUPPORTS_HIGHMEM
bool
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 01/58] MIPS: Kconfig: Add Kconfig symbols for EVA support
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Add basic Kconfig support for EVA. Not selectable by any platform
at this point.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/Kconfig | 28 +++++++++++++++++++++++++++-
1 file changed, 27 insertions(+), 1 deletion(-)
diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index f4c78c9..4230c7a 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -1446,6 +1446,26 @@ config CPU_XLP
Netlogic Microsystems XLP processors.
endchoice
+config CPU_MIPS32_3_5_FEATURES
+ bool "MIPS32 Release 3.5 Features"
+ depends on SYS_HAS_CPU_MIPS32_R3_5
+ depends on CPU_MIPS32_R2
+ help
+ Choose this option to build a kernel for release 2 or later of the
+ MIPS32 architecture including features from the 3.5 release such as
+ support for Enhanced Virtual Addressing (EVA).
+
+config CPU_MIPS32_3_5_EVA
+ bool "Enhanced Virtual Addressing (EVA)"
+ depends on CPU_MIPS32_3_5_FEATURES
+ select EVA
+ default y
+ help
+ Choose this option if you want to enable the Enhanced Virtual
+ Addressing (EVA) on your MIPS32 core (such as proAptiv).
+ One of its primary benefits is an increase in the maximum size
+ of lowmem (up to 3GB). If unsure, say 'N' here.
+
if CPU_LOONGSON2F
config CPU_NOP_WORKAROUNDS
bool
@@ -1539,6 +1559,9 @@ config SYS_HAS_CPU_MIPS32_R1
config SYS_HAS_CPU_MIPS32_R2
bool
+config SYS_HAS_CPU_MIPS32_R3_5
+ bool
+
config SYS_HAS_CPU_MIPS64_R1
bool
@@ -1658,6 +1681,9 @@ config CPU_MIPSR2
bool
default y if CPU_MIPS32_R2 || CPU_MIPS64_R2 || CPU_CAVIUM_OCTEON
+config EVA
+ bool
+
config SYS_SUPPORTS_32BIT_KERNEL
bool
config SYS_SUPPORTS_64BIT_KERNEL
@@ -2095,7 +2121,7 @@ config CPU_R4400_WORKAROUNDS
#
config HIGHMEM
bool "High Memory Support"
- depends on 32BIT && CPU_SUPPORTS_HIGHMEM && SYS_SUPPORTS_HIGHMEM
+ depends on 32BIT && CPU_SUPPORTS_HIGHMEM && SYS_SUPPORTS_HIGHMEM && !CPU_MIPS32_3_5_EVA
config CPU_SUPPORTS_HIGHMEM
bool
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 02/58] MIPS: asm: Add prefetch instruction for EVA
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
EVA can use the PREFE instruction to perform the virtual address
translation using the user mapping of the address rather than the
kernel mapping.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/asm.h | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/arch/mips/include/asm/asm.h b/arch/mips/include/asm/asm.h
index 879691d..b79be18a 100644
--- a/arch/mips/include/asm/asm.h
+++ b/arch/mips/include/asm/asm.h
@@ -149,6 +149,13 @@ symbol = value
pref hint, addr; \
.set pop
+#define PREFE(hint, addr) \
+ .set push; \
+ .set mips0; \
+ .set eva; \
+ prefe hint, addr; \
+ .set pop
+
#define PREFX(hint,addr) \
.set push; \
.set mips4; \
@@ -158,6 +165,7 @@ symbol = value
#else /* !CONFIG_CPU_HAS_PREFETCH */
#define PREF(hint, addr)
+#define PREFE(hint, addr)
#define PREFX(hint, addr)
#endif /* !CONFIG_CPU_HAS_PREFETCH */
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 02/58] MIPS: asm: Add prefetch instruction for EVA
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
EVA can use the PREFE instruction to perform the virtual address
translation using the user mapping of the address rather than the
kernel mapping.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/asm.h | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/arch/mips/include/asm/asm.h b/arch/mips/include/asm/asm.h
index 879691d..b79be18a 100644
--- a/arch/mips/include/asm/asm.h
+++ b/arch/mips/include/asm/asm.h
@@ -149,6 +149,13 @@ symbol = value
pref hint, addr; \
.set pop
+#define PREFE(hint, addr) \
+ .set push; \
+ .set mips0; \
+ .set eva; \
+ prefe hint, addr; \
+ .set pop
+
#define PREFX(hint,addr) \
.set push; \
.set mips4; \
@@ -158,6 +165,7 @@ symbol = value
#else /* !CONFIG_CPU_HAS_PREFETCH */
#define PREF(hint, addr)
+#define PREFE(hint, addr)
#define PREFX(hint, addr)
#endif /* !CONFIG_CPU_HAS_PREFETCH */
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 03/58] MIPS: asm: Add wrappers for EVA/non-EVA instructions
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
EVA uses specific instructions for accessing user memory.
Instead of polluting the kernel with numerous #ifdef CONFIG_EVA
we add wrappers for all the instructions that need special
handling when EVA is enabled.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/asm.h | 122 ++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 122 insertions(+)
diff --git a/arch/mips/include/asm/asm.h b/arch/mips/include/asm/asm.h
index b79be18a..5b11598 100644
--- a/arch/mips/include/asm/asm.h
+++ b/arch/mips/include/asm/asm.h
@@ -28,6 +28,128 @@
#define CAT(str1, str2) __CAT(str1, str2)
#endif
+#ifndef __ASSEMBLY__
+#ifdef CONFIG_EVA
+
+#define __BUILD_EVA_INSN(insn, reg, addr) \
+ " .set push\n" \
+ " .set mips0\n" \
+ " .set eva\n" \
+ " "insn" "reg", "addr "\n" \
+ " .set pop\n"
+
+#define user_cache(op, base) __BUILD_EVA_INSN("cachee", op, base)
+#define user_ll(reg, addr) __BUILD_EVA_INSN("lle", reg, addr)
+#define user_sc(reg, addr) __BUILD_EVA_INSN("sce", reg, addr)
+#define user_lw(reg, addr) __BUILD_EVA_INSN("lwe", reg, addr)
+#define user_lwl(reg, addr) __BUILD_EVA_INSN("lwle", reg, addr)
+#define user_lwr(reg, addr) __BUILD_EVA_INSN("lwre", reg, addr)
+#define user_lh(reg, addr) __BUILD_EVA_INSN("lhe", reg, addr)
+#define user_lb(reg, addr) __BUILD_EVA_INSN("lbe", reg, addr)
+#define user_lbu(reg, addr) __BUILD_EVA_INSN("lbue", reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_ld(reg, addr) user_lw(reg, addr)
+#define user_sw(reg, addr) __BUILD_EVA_INSN("swe", reg, addr)
+#define user_swl(reg, addr) __BUILD_EVA_INSN("swle", reg, addr)
+#define user_swr(reg, addr) __BUILD_EVA_INSN("swre", reg, addr)
+#define user_sh(reg, addr) __BUILD_EVA_INSN("she", reg, addr)
+#define user_sb(reg, addr) __BUILD_EVA_INSN("sbe", reg, addr)
+/* No 64-bit EVA instruction for storing double words */
+#define user_sd(reg, addr) user_sw(reg, addr)
+
+#else
+
+#define user_cache(op, base) "cache " op ", " base "\n"
+#define user_ll(reg, addr) "ll " reg ", " addr "\n"
+#define user_sc(reg, addr) "sc " reg ", " addr "\n"
+#define user_lw(reg, addr) "lw " reg ", " addr "\n"
+#define user_lwl(reg, addr) "lwl " reg ", " addr "\n"
+#define user_lwr(reg, addr) "lwr " reg ", " addr "\n"
+#define user_lh(reg, addr) "lh " reg ", " addr "\n"
+#define user_lb(reg, addr) "lb " reg ", " addr "\n"
+#define user_lbu(reg, addr) "lbu " reg ", " addr "\n"
+#define user_sw(reg, addr) "sw " reg ", " addr "\n"
+#define user_swl(reg, addr) "swl " reg ", " addr "\n"
+#define user_swr(reg, addr) "swr " reg ", " addr "\n"
+#define user_sh(reg, addr) "sh " reg ", " addr "\n"
+#define user_sb(reg, addr) "sb " reg ", " addr "\n"
+
+#ifdef CONFIG_32BIT
+/*
+ * No 'sd' or 'ld' instructions in 32-bit but the code will
+ * do the correct thing
+ */
+#define user_sd(reg, addr) user_sw(reg, addr)
+#define user_ld(reg, addr) user_lw(reg, addr)
+#else
+#define user_sd(reg, addr) "sd " reg", " addr "\n"
+#define user_ld(reg, addr) "ld " reg", " addr "\n"
+#endif /* CONFIG_32BIT */
+
+#endif /* CONFIG_EVA */
+
+#else /* __ASSEMBLY__ */
+
+#ifdef CONFIG_EVA
+
+#define __BUILD_EVA_INSN(insn, reg, addr) \
+ .set push; \
+ .set mips0; \
+ .set eva; \
+ insn reg, addr; \
+ .set pop;
+
+#define user_cache(op, base) __BUILD_EVA_INSN(cachee, op, base)
+#define user_ll(reg, addr) __BUILD_EVA_INSN(lle, reg, addr)
+#define user_sc(reg, addr) __BUILD_EVA_INSN(sce, reg, addr)
+#define user_lw(reg, addr) __BUILD_EVA_INSN(lwe, reg, addr)
+#define user_lwl(reg, addr) __BUILD_EVA_INSN(lwle, reg, addr)
+#define user_lwr(reg, addr) __BUILD_EVA_INSN(lwre, reg, addr)
+#define user_lh(reg, addr) __BUILD_EVA_INSN(lhe, reg, addr)
+#define user_lb(reg, addr) __BUILD_EVA_INSN(lbe, reg, addr)
+#define user_lbu(reg, addr) __BUILD_EVA_INSN(lbue, reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_ld(reg, addr) user_lw(reg, addr)
+#define user_sw(reg, addr) __BUILD_EVA_INSN(swe, reg, addr)
+#define user_swl(reg, addr) __BUILD_EVA_INSN(swle, reg, addr)
+#define user_swr(reg, addr) __BUILD_EVA_INSN(swre, reg, addr)
+#define user_sh(reg, addr) __BUILD_EVA_INSN(she, reg, addr)
+#define user_sb(reg, addr) __BUILD_EVA_INSN(sbe, reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_sd(reg, addr) user_sw(reg, addr)
+#else
+
+#define user_cache(op, base) cache op, base
+#define user_ll(reg, addr) ll reg, addr
+#define user_sc(reg, addr) sc reg, addr
+#define user_lw(reg, addr) lw reg, addr
+#define user_lwl(reg, addr) lwl reg, addr
+#define user_lwr(reg, addr) lwr reg, addr
+#define user_lh(reg, addr) lh reg, addr
+#define user_lb(reg, addr) lb reg, addr
+#define user_lbu(reg, addr) lbu reg, addr
+#define user_sw(reg, addr) sw reg, addr
+#define user_swl(reg, addr) swl reg, addr
+#define user_swr(reg, addr) swr reg, addr
+#define user_sh(reg, addr) sh reg, addr
+#define user_sb(reg, addr) sb reg, addr
+
+#ifdef CONFIG_32BIT
+/*
+ * No 'sd' or 'ld' instructions in 32-bit but the code will
+ * do the correct thing
+ */
+#define user_sd(reg, addr) user_sw(reg, addr)
+#define user_ld(reg, addr) user_lw(reg, addr)
+#else
+#define user_sd(reg, addr) sd reg, addr
+#define user_ld(reg, addr) ld reg, addr
+#endif /* CONFIG_32BIT */
+
+#endif /* CONFIG_EVA */
+
+#endif /* __ASSEMBLY__ */
+
/*
* PIC specific declarations
* Not used for the kernel but here seems to be the right place.
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 03/58] MIPS: asm: Add wrappers for EVA/non-EVA instructions
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
EVA uses specific instructions for accessing user memory.
Instead of polluting the kernel with numerous #ifdef CONFIG_EVA
we add wrappers for all the instructions that need special
handling when EVA is enabled.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/asm.h | 122 ++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 122 insertions(+)
diff --git a/arch/mips/include/asm/asm.h b/arch/mips/include/asm/asm.h
index b79be18a..5b11598 100644
--- a/arch/mips/include/asm/asm.h
+++ b/arch/mips/include/asm/asm.h
@@ -28,6 +28,128 @@
#define CAT(str1, str2) __CAT(str1, str2)
#endif
+#ifndef __ASSEMBLY__
+#ifdef CONFIG_EVA
+
+#define __BUILD_EVA_INSN(insn, reg, addr) \
+ " .set push\n" \
+ " .set mips0\n" \
+ " .set eva\n" \
+ " "insn" "reg", "addr "\n" \
+ " .set pop\n"
+
+#define user_cache(op, base) __BUILD_EVA_INSN("cachee", op, base)
+#define user_ll(reg, addr) __BUILD_EVA_INSN("lle", reg, addr)
+#define user_sc(reg, addr) __BUILD_EVA_INSN("sce", reg, addr)
+#define user_lw(reg, addr) __BUILD_EVA_INSN("lwe", reg, addr)
+#define user_lwl(reg, addr) __BUILD_EVA_INSN("lwle", reg, addr)
+#define user_lwr(reg, addr) __BUILD_EVA_INSN("lwre", reg, addr)
+#define user_lh(reg, addr) __BUILD_EVA_INSN("lhe", reg, addr)
+#define user_lb(reg, addr) __BUILD_EVA_INSN("lbe", reg, addr)
+#define user_lbu(reg, addr) __BUILD_EVA_INSN("lbue", reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_ld(reg, addr) user_lw(reg, addr)
+#define user_sw(reg, addr) __BUILD_EVA_INSN("swe", reg, addr)
+#define user_swl(reg, addr) __BUILD_EVA_INSN("swle", reg, addr)
+#define user_swr(reg, addr) __BUILD_EVA_INSN("swre", reg, addr)
+#define user_sh(reg, addr) __BUILD_EVA_INSN("she", reg, addr)
+#define user_sb(reg, addr) __BUILD_EVA_INSN("sbe", reg, addr)
+/* No 64-bit EVA instruction for storing double words */
+#define user_sd(reg, addr) user_sw(reg, addr)
+
+#else
+
+#define user_cache(op, base) "cache " op ", " base "\n"
+#define user_ll(reg, addr) "ll " reg ", " addr "\n"
+#define user_sc(reg, addr) "sc " reg ", " addr "\n"
+#define user_lw(reg, addr) "lw " reg ", " addr "\n"
+#define user_lwl(reg, addr) "lwl " reg ", " addr "\n"
+#define user_lwr(reg, addr) "lwr " reg ", " addr "\n"
+#define user_lh(reg, addr) "lh " reg ", " addr "\n"
+#define user_lb(reg, addr) "lb " reg ", " addr "\n"
+#define user_lbu(reg, addr) "lbu " reg ", " addr "\n"
+#define user_sw(reg, addr) "sw " reg ", " addr "\n"
+#define user_swl(reg, addr) "swl " reg ", " addr "\n"
+#define user_swr(reg, addr) "swr " reg ", " addr "\n"
+#define user_sh(reg, addr) "sh " reg ", " addr "\n"
+#define user_sb(reg, addr) "sb " reg ", " addr "\n"
+
+#ifdef CONFIG_32BIT
+/*
+ * No 'sd' or 'ld' instructions in 32-bit but the code will
+ * do the correct thing
+ */
+#define user_sd(reg, addr) user_sw(reg, addr)
+#define user_ld(reg, addr) user_lw(reg, addr)
+#else
+#define user_sd(reg, addr) "sd " reg", " addr "\n"
+#define user_ld(reg, addr) "ld " reg", " addr "\n"
+#endif /* CONFIG_32BIT */
+
+#endif /* CONFIG_EVA */
+
+#else /* __ASSEMBLY__ */
+
+#ifdef CONFIG_EVA
+
+#define __BUILD_EVA_INSN(insn, reg, addr) \
+ .set push; \
+ .set mips0; \
+ .set eva; \
+ insn reg, addr; \
+ .set pop;
+
+#define user_cache(op, base) __BUILD_EVA_INSN(cachee, op, base)
+#define user_ll(reg, addr) __BUILD_EVA_INSN(lle, reg, addr)
+#define user_sc(reg, addr) __BUILD_EVA_INSN(sce, reg, addr)
+#define user_lw(reg, addr) __BUILD_EVA_INSN(lwe, reg, addr)
+#define user_lwl(reg, addr) __BUILD_EVA_INSN(lwle, reg, addr)
+#define user_lwr(reg, addr) __BUILD_EVA_INSN(lwre, reg, addr)
+#define user_lh(reg, addr) __BUILD_EVA_INSN(lhe, reg, addr)
+#define user_lb(reg, addr) __BUILD_EVA_INSN(lbe, reg, addr)
+#define user_lbu(reg, addr) __BUILD_EVA_INSN(lbue, reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_ld(reg, addr) user_lw(reg, addr)
+#define user_sw(reg, addr) __BUILD_EVA_INSN(swe, reg, addr)
+#define user_swl(reg, addr) __BUILD_EVA_INSN(swle, reg, addr)
+#define user_swr(reg, addr) __BUILD_EVA_INSN(swre, reg, addr)
+#define user_sh(reg, addr) __BUILD_EVA_INSN(she, reg, addr)
+#define user_sb(reg, addr) __BUILD_EVA_INSN(sbe, reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_sd(reg, addr) user_sw(reg, addr)
+#else
+
+#define user_cache(op, base) cache op, base
+#define user_ll(reg, addr) ll reg, addr
+#define user_sc(reg, addr) sc reg, addr
+#define user_lw(reg, addr) lw reg, addr
+#define user_lwl(reg, addr) lwl reg, addr
+#define user_lwr(reg, addr) lwr reg, addr
+#define user_lh(reg, addr) lh reg, addr
+#define user_lb(reg, addr) lb reg, addr
+#define user_lbu(reg, addr) lbu reg, addr
+#define user_sw(reg, addr) sw reg, addr
+#define user_swl(reg, addr) swl reg, addr
+#define user_swr(reg, addr) swr reg, addr
+#define user_sh(reg, addr) sh reg, addr
+#define user_sb(reg, addr) sb reg, addr
+
+#ifdef CONFIG_32BIT
+/*
+ * No 'sd' or 'ld' instructions in 32-bit but the code will
+ * do the correct thing
+ */
+#define user_sd(reg, addr) user_sw(reg, addr)
+#define user_ld(reg, addr) user_lw(reg, addr)
+#else
+#define user_sd(reg, addr) sd reg, addr
+#define user_ld(reg, addr) ld reg, addr
+#endif /* CONFIG_32BIT */
+
+#endif /* CONFIG_EVA */
+
+#endif /* __ASSEMBLY__ */
+
/*
* PIC specific declarations
* Not used for the kernel but here seems to be the right place.
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH v2 03/58] MIPS: asm: Add wrappers for EVA/non-EVA instructions
@ 2014-03-06 15:52 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-03-06 15:52 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
EVA uses specific instructions for accessing user memory.
Instead of polluting the kernel with numerous #ifdef CONFIG_EVA
we add wrappers for all the instructions that need special
handling when EVA is enabled.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
Changes since v1:
- Move EVA instruction wrappers to a new header file to avoid problematic
inclusions of the asm/asm.h header file.
This is already included in the upstream-sfr/mips-for-linux-next tree.
The files that have previously been modified to include <asm/asm.h> have been
updated to include <asm/asm-eva.h>. The fixes are also in the upstream-sfr/
mips-for-linux-next tree but I will not post new versions of these patches
because the change is rather trivial.
For this patch, no further action is required apart from updating the
patchwork URL for this commit in the mips-for-linux-next branch.
---
arch/mips/include/asm/asm-eva.h | 135 ++++++++++++++++++++++++++++++++++++++++
arch/mips/include/asm/asm.h | 1 +
2 files changed, 136 insertions(+)
create mode 100644 arch/mips/include/asm/asm-eva.h
diff --git a/arch/mips/include/asm/asm-eva.h b/arch/mips/include/asm/asm-eva.h
new file mode 100644
index 0000000..e41c56e
--- /dev/null
+++ b/arch/mips/include/asm/asm-eva.h
@@ -0,0 +1,135 @@
+/*
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License. See the file "COPYING" in the main directory of this archive
+ * for more details.
+ *
+ * Copyright (C) 2014 Imagination Technologies Ltd.
+ *
+ */
+
+#ifndef __ASM_ASM_EVA_H
+#define __ASM_ASM_EVA_H
+
+#ifndef __ASSEMBLY__
+#ifdef CONFIG_EVA
+
+#define __BUILD_EVA_INSN(insn, reg, addr) \
+ " .set push\n" \
+ " .set mips0\n" \
+ " .set eva\n" \
+ " "insn" "reg", "addr "\n" \
+ " .set pop\n"
+
+#define user_cache(op, base) __BUILD_EVA_INSN("cachee", op, base)
+#define user_ll(reg, addr) __BUILD_EVA_INSN("lle", reg, addr)
+#define user_sc(reg, addr) __BUILD_EVA_INSN("sce", reg, addr)
+#define user_lw(reg, addr) __BUILD_EVA_INSN("lwe", reg, addr)
+#define user_lwl(reg, addr) __BUILD_EVA_INSN("lwle", reg, addr)
+#define user_lwr(reg, addr) __BUILD_EVA_INSN("lwre", reg, addr)
+#define user_lh(reg, addr) __BUILD_EVA_INSN("lhe", reg, addr)
+#define user_lb(reg, addr) __BUILD_EVA_INSN("lbe", reg, addr)
+#define user_lbu(reg, addr) __BUILD_EVA_INSN("lbue", reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_ld(reg, addr) user_lw(reg, addr)
+#define user_sw(reg, addr) __BUILD_EVA_INSN("swe", reg, addr)
+#define user_swl(reg, addr) __BUILD_EVA_INSN("swle", reg, addr)
+#define user_swr(reg, addr) __BUILD_EVA_INSN("swre", reg, addr)
+#define user_sh(reg, addr) __BUILD_EVA_INSN("she", reg, addr)
+#define user_sb(reg, addr) __BUILD_EVA_INSN("sbe", reg, addr)
+/* No 64-bit EVA instruction for storing double words */
+#define user_sd(reg, addr) user_sw(reg, addr)
+
+#else
+
+#define user_cache(op, base) "cache " op ", " base "\n"
+#define user_ll(reg, addr) "ll " reg ", " addr "\n"
+#define user_sc(reg, addr) "sc " reg ", " addr "\n"
+#define user_lw(reg, addr) "lw " reg ", " addr "\n"
+#define user_lwl(reg, addr) "lwl " reg ", " addr "\n"
+#define user_lwr(reg, addr) "lwr " reg ", " addr "\n"
+#define user_lh(reg, addr) "lh " reg ", " addr "\n"
+#define user_lb(reg, addr) "lb " reg ", " addr "\n"
+#define user_lbu(reg, addr) "lbu " reg ", " addr "\n"
+#define user_sw(reg, addr) "sw " reg ", " addr "\n"
+#define user_swl(reg, addr) "swl " reg ", " addr "\n"
+#define user_swr(reg, addr) "swr " reg ", " addr "\n"
+#define user_sh(reg, addr) "sh " reg ", " addr "\n"
+#define user_sb(reg, addr) "sb " reg ", " addr "\n"
+
+#ifdef CONFIG_32BIT
+/*
+ * No 'sd' or 'ld' instructions in 32-bit but the code will
+ * do the correct thing
+ */
+#define user_sd(reg, addr) user_sw(reg, addr)
+#define user_ld(reg, addr) user_lw(reg, addr)
+#else
+#define user_sd(reg, addr) "sd " reg", " addr "\n"
+#define user_ld(reg, addr) "ld " reg", " addr "\n"
+#endif /* CONFIG_32BIT */
+
+#endif /* CONFIG_EVA */
+
+#else /* __ASSEMBLY__ */
+
+#ifdef CONFIG_EVA
+
+#define __BUILD_EVA_INSN(insn, reg, addr) \
+ .set push; \
+ .set mips0; \
+ .set eva; \
+ insn reg, addr; \
+ .set pop;
+
+#define user_cache(op, base) __BUILD_EVA_INSN(cachee, op, base)
+#define user_ll(reg, addr) __BUILD_EVA_INSN(lle, reg, addr)
+#define user_sc(reg, addr) __BUILD_EVA_INSN(sce, reg, addr)
+#define user_lw(reg, addr) __BUILD_EVA_INSN(lwe, reg, addr)
+#define user_lwl(reg, addr) __BUILD_EVA_INSN(lwle, reg, addr)
+#define user_lwr(reg, addr) __BUILD_EVA_INSN(lwre, reg, addr)
+#define user_lh(reg, addr) __BUILD_EVA_INSN(lhe, reg, addr)
+#define user_lb(reg, addr) __BUILD_EVA_INSN(lbe, reg, addr)
+#define user_lbu(reg, addr) __BUILD_EVA_INSN(lbue, reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_ld(reg, addr) user_lw(reg, addr)
+#define user_sw(reg, addr) __BUILD_EVA_INSN(swe, reg, addr)
+#define user_swl(reg, addr) __BUILD_EVA_INSN(swle, reg, addr)
+#define user_swr(reg, addr) __BUILD_EVA_INSN(swre, reg, addr)
+#define user_sh(reg, addr) __BUILD_EVA_INSN(she, reg, addr)
+#define user_sb(reg, addr) __BUILD_EVA_INSN(sbe, reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_sd(reg, addr) user_sw(reg, addr)
+#else
+
+#define user_cache(op, base) cache op, base
+#define user_ll(reg, addr) ll reg, addr
+#define user_sc(reg, addr) sc reg, addr
+#define user_lw(reg, addr) lw reg, addr
+#define user_lwl(reg, addr) lwl reg, addr
+#define user_lwr(reg, addr) lwr reg, addr
+#define user_lh(reg, addr) lh reg, addr
+#define user_lb(reg, addr) lb reg, addr
+#define user_lbu(reg, addr) lbu reg, addr
+#define user_sw(reg, addr) sw reg, addr
+#define user_swl(reg, addr) swl reg, addr
+#define user_swr(reg, addr) swr reg, addr
+#define user_sh(reg, addr) sh reg, addr
+#define user_sb(reg, addr) sb reg, addr
+
+#ifdef CONFIG_32BIT
+/*
+ * No 'sd' or 'ld' instructions in 32-bit but the code will
+ * do the correct thing
+ */
+#define user_sd(reg, addr) user_sw(reg, addr)
+#define user_ld(reg, addr) user_lw(reg, addr)
+#else
+#define user_sd(reg, addr) sd reg, addr
+#define user_ld(reg, addr) ld reg, addr
+#endif /* CONFIG_32BIT */
+
+#endif /* CONFIG_EVA */
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* __ASM_ASM_EVA_H */
diff --git a/arch/mips/include/asm/asm.h b/arch/mips/include/asm/asm.h
index b79be18a..b153e79 100644
--- a/arch/mips/include/asm/asm.h
+++ b/arch/mips/include/asm/asm.h
@@ -18,6 +18,7 @@
#define __ASM_ASM_H
#include <asm/sgidefs.h>
+#include <asm/asm-eva.h>
#ifndef CAT
#ifdef __STDC__
--
1.9.0
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH v2 03/58] MIPS: asm: Add wrappers for EVA/non-EVA instructions
@ 2014-03-06 15:52 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-03-06 15:52 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
EVA uses specific instructions for accessing user memory.
Instead of polluting the kernel with numerous #ifdef CONFIG_EVA
we add wrappers for all the instructions that need special
handling when EVA is enabled.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
Changes since v1:
- Move EVA instruction wrappers to a new header file to avoid problematic
inclusions of the asm/asm.h header file.
This is already included in the upstream-sfr/mips-for-linux-next tree.
The files that have previously been modified to include <asm/asm.h> have been
updated to include <asm/asm-eva.h>. The fixes are also in the upstream-sfr/
mips-for-linux-next tree but I will not post new versions of these patches
because the change is rather trivial.
For this patch, no further action is required apart from updating the
patchwork URL for this commit in the mips-for-linux-next branch.
---
arch/mips/include/asm/asm-eva.h | 135 ++++++++++++++++++++++++++++++++++++++++
arch/mips/include/asm/asm.h | 1 +
2 files changed, 136 insertions(+)
create mode 100644 arch/mips/include/asm/asm-eva.h
diff --git a/arch/mips/include/asm/asm-eva.h b/arch/mips/include/asm/asm-eva.h
new file mode 100644
index 0000000..e41c56e
--- /dev/null
+++ b/arch/mips/include/asm/asm-eva.h
@@ -0,0 +1,135 @@
+/*
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License. See the file "COPYING" in the main directory of this archive
+ * for more details.
+ *
+ * Copyright (C) 2014 Imagination Technologies Ltd.
+ *
+ */
+
+#ifndef __ASM_ASM_EVA_H
+#define __ASM_ASM_EVA_H
+
+#ifndef __ASSEMBLY__
+#ifdef CONFIG_EVA
+
+#define __BUILD_EVA_INSN(insn, reg, addr) \
+ " .set push\n" \
+ " .set mips0\n" \
+ " .set eva\n" \
+ " "insn" "reg", "addr "\n" \
+ " .set pop\n"
+
+#define user_cache(op, base) __BUILD_EVA_INSN("cachee", op, base)
+#define user_ll(reg, addr) __BUILD_EVA_INSN("lle", reg, addr)
+#define user_sc(reg, addr) __BUILD_EVA_INSN("sce", reg, addr)
+#define user_lw(reg, addr) __BUILD_EVA_INSN("lwe", reg, addr)
+#define user_lwl(reg, addr) __BUILD_EVA_INSN("lwle", reg, addr)
+#define user_lwr(reg, addr) __BUILD_EVA_INSN("lwre", reg, addr)
+#define user_lh(reg, addr) __BUILD_EVA_INSN("lhe", reg, addr)
+#define user_lb(reg, addr) __BUILD_EVA_INSN("lbe", reg, addr)
+#define user_lbu(reg, addr) __BUILD_EVA_INSN("lbue", reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_ld(reg, addr) user_lw(reg, addr)
+#define user_sw(reg, addr) __BUILD_EVA_INSN("swe", reg, addr)
+#define user_swl(reg, addr) __BUILD_EVA_INSN("swle", reg, addr)
+#define user_swr(reg, addr) __BUILD_EVA_INSN("swre", reg, addr)
+#define user_sh(reg, addr) __BUILD_EVA_INSN("she", reg, addr)
+#define user_sb(reg, addr) __BUILD_EVA_INSN("sbe", reg, addr)
+/* No 64-bit EVA instruction for storing double words */
+#define user_sd(reg, addr) user_sw(reg, addr)
+
+#else
+
+#define user_cache(op, base) "cache " op ", " base "\n"
+#define user_ll(reg, addr) "ll " reg ", " addr "\n"
+#define user_sc(reg, addr) "sc " reg ", " addr "\n"
+#define user_lw(reg, addr) "lw " reg ", " addr "\n"
+#define user_lwl(reg, addr) "lwl " reg ", " addr "\n"
+#define user_lwr(reg, addr) "lwr " reg ", " addr "\n"
+#define user_lh(reg, addr) "lh " reg ", " addr "\n"
+#define user_lb(reg, addr) "lb " reg ", " addr "\n"
+#define user_lbu(reg, addr) "lbu " reg ", " addr "\n"
+#define user_sw(reg, addr) "sw " reg ", " addr "\n"
+#define user_swl(reg, addr) "swl " reg ", " addr "\n"
+#define user_swr(reg, addr) "swr " reg ", " addr "\n"
+#define user_sh(reg, addr) "sh " reg ", " addr "\n"
+#define user_sb(reg, addr) "sb " reg ", " addr "\n"
+
+#ifdef CONFIG_32BIT
+/*
+ * No 'sd' or 'ld' instructions in 32-bit but the code will
+ * do the correct thing
+ */
+#define user_sd(reg, addr) user_sw(reg, addr)
+#define user_ld(reg, addr) user_lw(reg, addr)
+#else
+#define user_sd(reg, addr) "sd " reg", " addr "\n"
+#define user_ld(reg, addr) "ld " reg", " addr "\n"
+#endif /* CONFIG_32BIT */
+
+#endif /* CONFIG_EVA */
+
+#else /* __ASSEMBLY__ */
+
+#ifdef CONFIG_EVA
+
+#define __BUILD_EVA_INSN(insn, reg, addr) \
+ .set push; \
+ .set mips0; \
+ .set eva; \
+ insn reg, addr; \
+ .set pop;
+
+#define user_cache(op, base) __BUILD_EVA_INSN(cachee, op, base)
+#define user_ll(reg, addr) __BUILD_EVA_INSN(lle, reg, addr)
+#define user_sc(reg, addr) __BUILD_EVA_INSN(sce, reg, addr)
+#define user_lw(reg, addr) __BUILD_EVA_INSN(lwe, reg, addr)
+#define user_lwl(reg, addr) __BUILD_EVA_INSN(lwle, reg, addr)
+#define user_lwr(reg, addr) __BUILD_EVA_INSN(lwre, reg, addr)
+#define user_lh(reg, addr) __BUILD_EVA_INSN(lhe, reg, addr)
+#define user_lb(reg, addr) __BUILD_EVA_INSN(lbe, reg, addr)
+#define user_lbu(reg, addr) __BUILD_EVA_INSN(lbue, reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_ld(reg, addr) user_lw(reg, addr)
+#define user_sw(reg, addr) __BUILD_EVA_INSN(swe, reg, addr)
+#define user_swl(reg, addr) __BUILD_EVA_INSN(swle, reg, addr)
+#define user_swr(reg, addr) __BUILD_EVA_INSN(swre, reg, addr)
+#define user_sh(reg, addr) __BUILD_EVA_INSN(she, reg, addr)
+#define user_sb(reg, addr) __BUILD_EVA_INSN(sbe, reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_sd(reg, addr) user_sw(reg, addr)
+#else
+
+#define user_cache(op, base) cache op, base
+#define user_ll(reg, addr) ll reg, addr
+#define user_sc(reg, addr) sc reg, addr
+#define user_lw(reg, addr) lw reg, addr
+#define user_lwl(reg, addr) lwl reg, addr
+#define user_lwr(reg, addr) lwr reg, addr
+#define user_lh(reg, addr) lh reg, addr
+#define user_lb(reg, addr) lb reg, addr
+#define user_lbu(reg, addr) lbu reg, addr
+#define user_sw(reg, addr) sw reg, addr
+#define user_swl(reg, addr) swl reg, addr
+#define user_swr(reg, addr) swr reg, addr
+#define user_sh(reg, addr) sh reg, addr
+#define user_sb(reg, addr) sb reg, addr
+
+#ifdef CONFIG_32BIT
+/*
+ * No 'sd' or 'ld' instructions in 32-bit but the code will
+ * do the correct thing
+ */
+#define user_sd(reg, addr) user_sw(reg, addr)
+#define user_ld(reg, addr) user_lw(reg, addr)
+#else
+#define user_sd(reg, addr) sd reg, addr
+#define user_ld(reg, addr) ld reg, addr
+#endif /* CONFIG_32BIT */
+
+#endif /* CONFIG_EVA */
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* __ASM_ASM_EVA_H */
diff --git a/arch/mips/include/asm/asm.h b/arch/mips/include/asm/asm.h
index b79be18a..b153e79 100644
--- a/arch/mips/include/asm/asm.h
+++ b/arch/mips/include/asm/asm.h
@@ -18,6 +18,7 @@
#define __ASM_ASM_H
#include <asm/sgidefs.h>
+#include <asm/asm-eva.h>
#ifndef CAT
#ifdef __STDC__
--
1.9.0
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 04/58] MIPS: futex: Add EVA support for futex operations
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Use LLE/SCE instructions for performing an address translation for
userspace when EVA is enabled.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/futex.h | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/arch/mips/include/asm/futex.h b/arch/mips/include/asm/futex.h
index 6ea1581..2a572e1 100644
--- a/arch/mips/include/asm/futex.h
+++ b/arch/mips/include/asm/futex.h
@@ -12,6 +12,7 @@
#include <linux/futex.h>
#include <linux/uaccess.h>
+#include <asm/asm.h>
#include <asm/barrier.h>
#include <asm/errno.h>
#include <asm/war.h>
@@ -49,11 +50,11 @@
" .set push \n" \
" .set noat \n" \
" .set mips3 \n" \
- "1: ll %1, %4 # __futex_atomic_op \n" \
+ "1: "user_ll("%1", "%4")" # __futex_atomic_op\n" \
" .set mips0 \n" \
" " insn " \n" \
" .set mips3 \n" \
- "2: sc $1, %2 \n" \
+ "2: "user_sc("$1", "%2")" \n" \
" beqz $1, 1b \n" \
__WEAK_LLSC_MB \
"3: \n" \
@@ -174,12 +175,12 @@ futex_atomic_cmpxchg_inatomic(u32 *uval, u32 __user *uaddr,
" .set push \n"
" .set noat \n"
" .set mips3 \n"
- "1: ll %1, %3 \n"
+ "1: "user_ll("%1", "%3")" \n"
" bne %1, %z4, 3f \n"
" .set mips0 \n"
" move $1, %z5 \n"
" .set mips3 \n"
- "2: sc $1, %2 \n"
+ "2: "user_sc("$1", "%2")" \n"
" beqz $1, 1b \n"
__WEAK_LLSC_MB
"3: \n"
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 04/58] MIPS: futex: Add EVA support for futex operations
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Use LLE/SCE instructions for performing an address translation for
userspace when EVA is enabled.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/futex.h | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/arch/mips/include/asm/futex.h b/arch/mips/include/asm/futex.h
index 6ea1581..2a572e1 100644
--- a/arch/mips/include/asm/futex.h
+++ b/arch/mips/include/asm/futex.h
@@ -12,6 +12,7 @@
#include <linux/futex.h>
#include <linux/uaccess.h>
+#include <asm/asm.h>
#include <asm/barrier.h>
#include <asm/errno.h>
#include <asm/war.h>
@@ -49,11 +50,11 @@
" .set push \n" \
" .set noat \n" \
" .set mips3 \n" \
- "1: ll %1, %4 # __futex_atomic_op \n" \
+ "1: "user_ll("%1", "%4")" # __futex_atomic_op\n" \
" .set mips0 \n" \
" " insn " \n" \
" .set mips3 \n" \
- "2: sc $1, %2 \n" \
+ "2: "user_sc("$1", "%2")" \n" \
" beqz $1, 1b \n" \
__WEAK_LLSC_MB \
"3: \n" \
@@ -174,12 +175,12 @@ futex_atomic_cmpxchg_inatomic(u32 *uval, u32 __user *uaddr,
" .set push \n"
" .set noat \n"
" .set mips3 \n"
- "1: ll %1, %3 \n"
+ "1: "user_ll("%1", "%3")" \n"
" bne %1, %z4, 3f \n"
" .set mips0 \n"
" move $1, %z5 \n"
" .set mips3 \n"
- "2: sc $1, %2 \n"
+ "2: "user_sc("$1", "%2")" \n"
" beqz $1, 1b \n"
__WEAK_LLSC_MB
"3: \n"
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 05/58] MIPS: uapi: inst: Add new EVA opcodes
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/uapi/asm/inst.h | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/arch/mips/include/uapi/asm/inst.h b/arch/mips/include/uapi/asm/inst.h
index b39ba25..9ce652e 100644
--- a/arch/mips/include/uapi/asm/inst.h
+++ b/arch/mips/include/uapi/asm/inst.h
@@ -73,10 +73,16 @@ enum spec2_op {
enum spec3_op {
ext_op, dextm_op, dextu_op, dext_op,
ins_op, dinsm_op, dinsu_op, dins_op,
- lx_op = 0x0a,
- bshfl_op = 0x20,
- dbshfl_op = 0x24,
- rdhwr_op = 0x3b
+ lx_op = 0x0a, lwle_op = 0x19,
+ lwre_op = 0x1a, cachee_op = 0x1b,
+ sbe_op = 0x1c, she_op = 0x1d,
+ sce_op = 0x1e, swe_op = 0x1f,
+ bshfl_op = 0x20, swle_op = 0x21,
+ swre_op = 0x22, prefe_op = 0x23,
+ dbshfl_op = 0x24, lbue_op = 0x28,
+ lhue_op = 0x29, lbe_op = 0x2c,
+ lhe_op = 0x2d, lle_op = 0x2e,
+ lwe_op = 0x2f, rdhwr_op = 0x3b
};
/*
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 05/58] MIPS: uapi: inst: Add new EVA opcodes
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/uapi/asm/inst.h | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/arch/mips/include/uapi/asm/inst.h b/arch/mips/include/uapi/asm/inst.h
index b39ba25..9ce652e 100644
--- a/arch/mips/include/uapi/asm/inst.h
+++ b/arch/mips/include/uapi/asm/inst.h
@@ -73,10 +73,16 @@ enum spec2_op {
enum spec3_op {
ext_op, dextm_op, dextu_op, dext_op,
ins_op, dinsm_op, dinsu_op, dins_op,
- lx_op = 0x0a,
- bshfl_op = 0x20,
- dbshfl_op = 0x24,
- rdhwr_op = 0x3b
+ lx_op = 0x0a, lwle_op = 0x19,
+ lwre_op = 0x1a, cachee_op = 0x1b,
+ sbe_op = 0x1c, she_op = 0x1d,
+ sce_op = 0x1e, swe_op = 0x1f,
+ bshfl_op = 0x20, swle_op = 0x21,
+ swre_op = 0x22, prefe_op = 0x23,
+ dbshfl_op = 0x24, lbue_op = 0x28,
+ lhue_op = 0x29, lbe_op = 0x2c,
+ lhe_op = 0x2d, lle_op = 0x2e,
+ lwe_op = 0x2f, rdhwr_op = 0x3b
};
/*
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 06/58] MIPS: uapi: inst: Add instruction format for SPECIAL3 instructions
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/uapi/asm/inst.h | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/arch/mips/include/uapi/asm/inst.h b/arch/mips/include/uapi/asm/inst.h
index 9ce652e..1a44c5a 100644
--- a/arch/mips/include/uapi/asm/inst.h
+++ b/arch/mips/include/uapi/asm/inst.h
@@ -8,6 +8,7 @@
* Copyright (C) 1996, 2000 by Ralf Baechle
* Copyright (C) 2006 by Thiemo Seufer
* Copyright (C) 2012 MIPS Technologies, Inc. All rights reserved.
+ * Copyright (C) 2014 Imagination Technologies Ltd.
*/
#ifndef _UAPI_ASM_INST_H
#define _UAPI_ASM_INST_H
@@ -598,6 +599,15 @@ struct v_format { /* MDMX vector format */
;)))))))
};
+struct spec3_format { /* SPEC3 */
+ BITFIELD_FIELD(unsigned int opcode:6,
+ BITFIELD_FIELD(unsigned int rs:5,
+ BITFIELD_FIELD(unsigned int rt:5,
+ BITFIELD_FIELD(signed int simmediate:9,
+ BITFIELD_FIELD(unsigned int func:7,
+ ;)))))
+};
+
/*
* microMIPS instruction formats (32-bit length)
*
@@ -869,6 +879,7 @@ union mips_instruction {
struct b_format b_format;
struct ps_format ps_format;
struct v_format v_format;
+ struct spec3_format spec3_format;
struct fb_format fb_format;
struct fp0_format fp0_format;
struct mm_fp0_format mm_fp0_format;
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 06/58] MIPS: uapi: inst: Add instruction format for SPECIAL3 instructions
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/uapi/asm/inst.h | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/arch/mips/include/uapi/asm/inst.h b/arch/mips/include/uapi/asm/inst.h
index 9ce652e..1a44c5a 100644
--- a/arch/mips/include/uapi/asm/inst.h
+++ b/arch/mips/include/uapi/asm/inst.h
@@ -8,6 +8,7 @@
* Copyright (C) 1996, 2000 by Ralf Baechle
* Copyright (C) 2006 by Thiemo Seufer
* Copyright (C) 2012 MIPS Technologies, Inc. All rights reserved.
+ * Copyright (C) 2014 Imagination Technologies Ltd.
*/
#ifndef _UAPI_ASM_INST_H
#define _UAPI_ASM_INST_H
@@ -598,6 +599,15 @@ struct v_format { /* MDMX vector format */
;)))))))
};
+struct spec3_format { /* SPEC3 */
+ BITFIELD_FIELD(unsigned int opcode:6,
+ BITFIELD_FIELD(unsigned int rs:5,
+ BITFIELD_FIELD(unsigned int rt:5,
+ BITFIELD_FIELD(signed int simmediate:9,
+ BITFIELD_FIELD(unsigned int func:7,
+ ;)))))
+};
+
/*
* microMIPS instruction formats (32-bit length)
*
@@ -869,6 +879,7 @@ union mips_instruction {
struct b_format b_format;
struct ps_format ps_format;
struct v_format v_format;
+ struct spec3_format spec3_format;
struct fb_format fb_format;
struct fp0_format fp0_format;
struct mm_fp0_format mm_fp0_format;
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 07/58] MIPS: kernel: scall32-o32: Use EVA wrappers to fetch syscall arguments
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Arguments 4-8 are stored on user's stack, so use the EVA instructions
to fetch them if EVA is enabled.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/scall32-o32.S | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/arch/mips/kernel/scall32-o32.S b/arch/mips/kernel/scall32-o32.S
index e8e541b..2fae6eb 100644
--- a/arch/mips/kernel/scall32-o32.S
+++ b/arch/mips/kernel/scall32-o32.S
@@ -6,6 +6,7 @@
* Copyright (C) 1995-99, 2000- 02, 06 Ralf Baechle <ralf@linux-mips.org>
* Copyright (C) 2001 MIPS Technologies, Inc.
* Copyright (C) 2004 Thiemo Seufer
+ * Copyright (C) 2014 Imagination Technologies Ltd.
*/
#include <linux/errno.h>
#include <asm/asm.h>
@@ -74,10 +75,10 @@ NESTED(handle_sys, PT_SIZE, sp)
.set noreorder
.set nomacro
-1: lw t5, 16(t0) # argument #5 from usp
-4: lw t6, 20(t0) # argument #6 from usp
-3: lw t7, 24(t0) # argument #7 from usp
-2: lw t8, 28(t0) # argument #8 from usp
+1: user_lw(t5, 16(t0)) # argument #5 from usp
+4: user_lw(t6, 20(t0)) # argument #6 from usp
+3: user_lw(t7, 24(t0)) # argument #7 from usp
+2: user_lw(t8, 28(t0)) # argument #8 from usp
sw t5, 16(sp) # argument #5 to ksp
sw t6, 20(sp) # argument #6 to ksp
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 07/58] MIPS: kernel: scall32-o32: Use EVA wrappers to fetch syscall arguments
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Arguments 4-8 are stored on user's stack, so use the EVA instructions
to fetch them if EVA is enabled.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/scall32-o32.S | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/arch/mips/kernel/scall32-o32.S b/arch/mips/kernel/scall32-o32.S
index e8e541b..2fae6eb 100644
--- a/arch/mips/kernel/scall32-o32.S
+++ b/arch/mips/kernel/scall32-o32.S
@@ -6,6 +6,7 @@
* Copyright (C) 1995-99, 2000- 02, 06 Ralf Baechle <ralf@linux-mips.org>
* Copyright (C) 2001 MIPS Technologies, Inc.
* Copyright (C) 2004 Thiemo Seufer
+ * Copyright (C) 2014 Imagination Technologies Ltd.
*/
#include <linux/errno.h>
#include <asm/asm.h>
@@ -74,10 +75,10 @@ NESTED(handle_sys, PT_SIZE, sp)
.set noreorder
.set nomacro
-1: lw t5, 16(t0) # argument #5 from usp
-4: lw t6, 20(t0) # argument #6 from usp
-3: lw t7, 24(t0) # argument #7 from usp
-2: lw t8, 28(t0) # argument #8 from usp
+1: user_lw(t5, 16(t0)) # argument #5 from usp
+4: user_lw(t6, 20(t0)) # argument #6 from usp
+3: user_lw(t7, 24(t0)) # argument #7 from usp
+2: user_lw(t8, 28(t0)) # argument #8 from usp
sw t5, 16(sp) # argument #5 to ksp
sw t6, 20(sp) # argument #6 to ksp
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 08/58] MIPS: kernel: traps: Whitespace clean up
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/traps.c | 17 ++++++++++-------
1 file changed, 10 insertions(+), 7 deletions(-)
diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
index e0b4996..3e5e700 100644
--- a/arch/mips/kernel/traps.c
+++ b/arch/mips/kernel/traps.c
@@ -10,6 +10,7 @@
* Kevin D. Kissell, kevink@mips.com and Carsten Langgaard, carstenl@mips.com
* Copyright (C) 2002, 2003, 2004, 2005, 2007 Maciej W. Rozycki
* Copyright (C) 2000, 2001, 2012 MIPS Technologies, Inc. All rights reserved.
+ * Copyright (C) 2014, Imagination Technologies Ltd.
*/
#include <linux/bug.h>
#include <linux/compiler.h>
@@ -870,17 +871,19 @@ asmlinkage void do_bp(struct pt_regs *regs)
if ((__get_user(instr[0], (u16 __user *)msk_isa16_mode(epc)) ||
(__get_user(instr[1], (u16 __user *)msk_isa16_mode(epc + 2)))))
goto out_sigsegv;
- opcode = (instr[0] << 16) | instr[1];
+ opcode = (instr[0] << 16) | instr[1];
} else {
- /* MIPS16e mode */
- if (__get_user(instr[0], (u16 __user *)msk_isa16_mode(epc)))
+ /* MIPS16e mode */
+ if (__get_user(instr[0],
+ (u16 __user *)msk_isa16_mode(epc)))
goto out_sigsegv;
- bcode = (instr[0] >> 6) & 0x3f;
- do_trap_or_bp(regs, bcode, "Break");
- goto out;
+ bcode = (instr[0] >> 6) & 0x3f;
+ do_trap_or_bp(regs, bcode, "Break");
+ goto out;
}
} else {
- if (__get_user(opcode, (unsigned int __user *) exception_epc(regs)))
+ if (__get_user(opcode,
+ (unsigned int __user *) exception_epc(regs)))
goto out_sigsegv;
}
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 08/58] MIPS: kernel: traps: Whitespace clean up
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/traps.c | 17 ++++++++++-------
1 file changed, 10 insertions(+), 7 deletions(-)
diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
index e0b4996..3e5e700 100644
--- a/arch/mips/kernel/traps.c
+++ b/arch/mips/kernel/traps.c
@@ -10,6 +10,7 @@
* Kevin D. Kissell, kevink@mips.com and Carsten Langgaard, carstenl@mips.com
* Copyright (C) 2002, 2003, 2004, 2005, 2007 Maciej W. Rozycki
* Copyright (C) 2000, 2001, 2012 MIPS Technologies, Inc. All rights reserved.
+ * Copyright (C) 2014, Imagination Technologies Ltd.
*/
#include <linux/bug.h>
#include <linux/compiler.h>
@@ -870,17 +871,19 @@ asmlinkage void do_bp(struct pt_regs *regs)
if ((__get_user(instr[0], (u16 __user *)msk_isa16_mode(epc)) ||
(__get_user(instr[1], (u16 __user *)msk_isa16_mode(epc + 2)))))
goto out_sigsegv;
- opcode = (instr[0] << 16) | instr[1];
+ opcode = (instr[0] << 16) | instr[1];
} else {
- /* MIPS16e mode */
- if (__get_user(instr[0], (u16 __user *)msk_isa16_mode(epc)))
+ /* MIPS16e mode */
+ if (__get_user(instr[0],
+ (u16 __user *)msk_isa16_mode(epc)))
goto out_sigsegv;
- bcode = (instr[0] >> 6) & 0x3f;
- do_trap_or_bp(regs, bcode, "Break");
- goto out;
+ bcode = (instr[0] >> 6) & 0x3f;
+ do_trap_or_bp(regs, bcode, "Break");
+ goto out;
}
} else {
- if (__get_user(opcode, (unsigned int __user *) exception_epc(regs)))
+ if (__get_user(opcode,
+ (unsigned int __user *) exception_epc(regs)))
goto out_sigsegv;
}
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 09/58] MIPS: traps: Set correct address limit for breakpoints and traps
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
When a breakpoint or trap happens when operating in kernel mode but
on users behalf (eg syscall) it is necessary to change the address
limit to KERNEL_DS so any address checking can be bypassed and print
the correct stack trace.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/traps.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
index 3e5e700..a5a74d6 100644
--- a/arch/mips/kernel/traps.c
+++ b/arch/mips/kernel/traps.c
@@ -862,6 +862,11 @@ asmlinkage void do_bp(struct pt_regs *regs)
enum ctx_state prev_state;
unsigned long epc;
u16 instr[2];
+ mm_segment_t seg;
+
+ seg = get_fs();
+ if (!user_mode(regs))
+ set_fs(KERNEL_DS);
prev_state = exception_enter();
if (get_isa16_mode(regs->cp0_epc)) {
@@ -921,6 +926,7 @@ asmlinkage void do_bp(struct pt_regs *regs)
do_trap_or_bp(regs, bcode, "Break");
out:
+ set_fs(seg);
exception_exit(prev_state);
return;
@@ -934,8 +940,13 @@ asmlinkage void do_tr(struct pt_regs *regs)
u32 opcode, tcode = 0;
enum ctx_state prev_state;
u16 instr[2];
+ mm_segment_t seg;
unsigned long epc = msk_isa16_mode(exception_epc(regs));
+ seg = get_fs();
+ if (!user_mode(regs))
+ set_fs(get_ds());
+
prev_state = exception_enter();
if (get_isa16_mode(regs->cp0_epc)) {
if (__get_user(instr[0], (u16 __user *)(epc + 0)) ||
@@ -956,6 +967,7 @@ asmlinkage void do_tr(struct pt_regs *regs)
do_trap_or_bp(regs, tcode, "Trap");
out:
+ set_fs(seg);
exception_exit(prev_state);
return;
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 09/58] MIPS: traps: Set correct address limit for breakpoints and traps
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
When a breakpoint or trap happens when operating in kernel mode but
on users behalf (eg syscall) it is necessary to change the address
limit to KERNEL_DS so any address checking can be bypassed and print
the correct stack trace.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/traps.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
index 3e5e700..a5a74d6 100644
--- a/arch/mips/kernel/traps.c
+++ b/arch/mips/kernel/traps.c
@@ -862,6 +862,11 @@ asmlinkage void do_bp(struct pt_regs *regs)
enum ctx_state prev_state;
unsigned long epc;
u16 instr[2];
+ mm_segment_t seg;
+
+ seg = get_fs();
+ if (!user_mode(regs))
+ set_fs(KERNEL_DS);
prev_state = exception_enter();
if (get_isa16_mode(regs->cp0_epc)) {
@@ -921,6 +926,7 @@ asmlinkage void do_bp(struct pt_regs *regs)
do_trap_or_bp(regs, bcode, "Break");
out:
+ set_fs(seg);
exception_exit(prev_state);
return;
@@ -934,8 +940,13 @@ asmlinkage void do_tr(struct pt_regs *regs)
u32 opcode, tcode = 0;
enum ctx_state prev_state;
u16 instr[2];
+ mm_segment_t seg;
unsigned long epc = msk_isa16_mode(exception_epc(regs));
+ seg = get_fs();
+ if (!user_mode(regs))
+ set_fs(get_ds());
+
prev_state = exception_enter();
if (get_isa16_mode(regs->cp0_epc)) {
if (__get_user(instr[0], (u16 __user *)(epc + 0)) ||
@@ -956,6 +967,7 @@ asmlinkage void do_tr(struct pt_regs *regs)
do_trap_or_bp(regs, tcode, "Trap");
out:
+ set_fs(seg);
exception_exit(prev_state);
return;
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 10/58] MIPS: lib: strnlen_user: Use macro to build the strnlen_user symbol
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Build the __strnlen_user symbol using a macro. In EVA mode we will
need to use similar code to do the userspace load operations so
it is better if we use a macro to avoid code duplications.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/strnlen_user.S | 16 ++++++++++------
1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/arch/mips/lib/strnlen_user.S b/arch/mips/lib/strnlen_user.S
index fcacea5..4422160 100644
--- a/arch/mips/lib/strnlen_user.S
+++ b/arch/mips/lib/strnlen_user.S
@@ -25,22 +25,26 @@
* bytes. There's nothing secret there. On 64-bit accessing beyond
* the maximum is a tad hairier ...
*/
-LEAF(__strnlen_user_asm)
+ .macro __BUILD_STRNLEN_ASM func
+LEAF(__strnlen_\func\()_asm)
LONG_L v0, TI_ADDR_LIMIT($28) # pointer ok?
and v0, a0
- bnez v0, .Lfault
+ bnez v0, .Lfault\@
-FEXPORT(__strnlen_user_nocheck_asm)
+FEXPORT(__strnlen_\func\()_nocheck_asm)
move v0, a0
PTR_ADDU a1, a0 # stop pointer
1: beq v0, a1, 1f # limit reached?
- EX(lb, t0, (v0), .Lfault)
+ EX(lb, t0, (v0), .Lfault\@)
PTR_ADDIU v0, 1
bnez t0, 1b
1: PTR_SUBU v0, a0
jr ra
- END(__strnlen_user_asm)
+ END(__strnlen_\func\()_asm)
-.Lfault:
+.Lfault\@:
move v0, zero
jr ra
+ .endm
+
+__BUILD_STRNLEN_ASM user
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 10/58] MIPS: lib: strnlen_user: Use macro to build the strnlen_user symbol
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Build the __strnlen_user symbol using a macro. In EVA mode we will
need to use similar code to do the userspace load operations so
it is better if we use a macro to avoid code duplications.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/strnlen_user.S | 16 ++++++++++------
1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/arch/mips/lib/strnlen_user.S b/arch/mips/lib/strnlen_user.S
index fcacea5..4422160 100644
--- a/arch/mips/lib/strnlen_user.S
+++ b/arch/mips/lib/strnlen_user.S
@@ -25,22 +25,26 @@
* bytes. There's nothing secret there. On 64-bit accessing beyond
* the maximum is a tad hairier ...
*/
-LEAF(__strnlen_user_asm)
+ .macro __BUILD_STRNLEN_ASM func
+LEAF(__strnlen_\func\()_asm)
LONG_L v0, TI_ADDR_LIMIT($28) # pointer ok?
and v0, a0
- bnez v0, .Lfault
+ bnez v0, .Lfault\@
-FEXPORT(__strnlen_user_nocheck_asm)
+FEXPORT(__strnlen_\func\()_nocheck_asm)
move v0, a0
PTR_ADDU a1, a0 # stop pointer
1: beq v0, a1, 1f # limit reached?
- EX(lb, t0, (v0), .Lfault)
+ EX(lb, t0, (v0), .Lfault\@)
PTR_ADDIU v0, 1
bnez t0, 1b
1: PTR_SUBU v0, a0
jr ra
- END(__strnlen_user_asm)
+ END(__strnlen_\func\()_asm)
-.Lfault:
+.Lfault\@:
move v0, zero
jr ra
+ .endm
+
+__BUILD_STRNLEN_ASM user
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 11/58] MIPS: lib: strnlen_user: Add EVA support
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
In non-EVA mode, a strlen_user* alias is used for the
strlen_kernel* symbols since the code is identical. In EVA
mode, a new strlen_user* symbol is used which uses the EVA
specific instructions to load values from userspace.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/mips_ksyms.c | 4 ++++
arch/mips/lib/strnlen_user.S | 20 ++++++++++++++++++++
2 files changed, 24 insertions(+)
diff --git a/arch/mips/kernel/mips_ksyms.c b/arch/mips/kernel/mips_ksyms.c
index 6e58e97..a322db0 100644
--- a/arch/mips/kernel/mips_ksyms.c
+++ b/arch/mips/kernel/mips_ksyms.c
@@ -22,6 +22,8 @@ extern long __strncpy_from_user_asm(char *__to, const char *__from,
long __len);
extern long __strlen_user_nocheck_asm(const char *s);
extern long __strlen_user_asm(const char *s);
+extern long __strnlen_kernel_nocheck_asm(const char *s);
+extern long __strnlen_kernel_asm(const char *s);
extern long __strnlen_user_nocheck_asm(const char *s);
extern long __strnlen_user_asm(const char *s);
@@ -48,6 +50,8 @@ EXPORT_SYMBOL(__strncpy_from_user_nocheck_asm);
EXPORT_SYMBOL(__strncpy_from_user_asm);
EXPORT_SYMBOL(__strlen_user_nocheck_asm);
EXPORT_SYMBOL(__strlen_user_asm);
+EXPORT_SYMBOL(__strnlen_kernel_nocheck_asm);
+EXPORT_SYMBOL(__strnlen_kernel_asm);
EXPORT_SYMBOL(__strnlen_user_nocheck_asm);
EXPORT_SYMBOL(__strnlen_user_asm);
diff --git a/arch/mips/lib/strnlen_user.S b/arch/mips/lib/strnlen_user.S
index 4422160..f3af699 100644
--- a/arch/mips/lib/strnlen_user.S
+++ b/arch/mips/lib/strnlen_user.S
@@ -35,7 +35,11 @@ FEXPORT(__strnlen_\func\()_nocheck_asm)
move v0, a0
PTR_ADDU a1, a0 # stop pointer
1: beq v0, a1, 1f # limit reached?
+.ifeqs "\func", "kernel"
EX(lb, t0, (v0), .Lfault\@)
+.else
+ EX(lbe, t0, (v0), .Lfault\@)
+.endif
PTR_ADDIU v0, 1
bnez t0, 1b
1: PTR_SUBU v0, a0
@@ -47,4 +51,20 @@ FEXPORT(__strnlen_\func\()_nocheck_asm)
jr ra
.endm
+#ifndef CONFIG_EVA
+ /* Set aliases */
+ .global __strnlen_user_asm
+ .global __strnlen_user_nocheck_asm
+ .set __strnlen_user_asm, __strnlen_kernel_asm
+ .set __strnlen_user_nocheck_asm, __strnlen_kernel_nocheck_asm
+#endif
+
+__BUILD_STRNLEN_ASM kernel
+
+#ifdef CONFIG_EVA
+
+ .set push
+ .set eva
__BUILD_STRNLEN_ASM user
+ .set pop
+#endif
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 11/58] MIPS: lib: strnlen_user: Add EVA support
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
In non-EVA mode, a strlen_user* alias is used for the
strlen_kernel* symbols since the code is identical. In EVA
mode, a new strlen_user* symbol is used which uses the EVA
specific instructions to load values from userspace.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/mips_ksyms.c | 4 ++++
arch/mips/lib/strnlen_user.S | 20 ++++++++++++++++++++
2 files changed, 24 insertions(+)
diff --git a/arch/mips/kernel/mips_ksyms.c b/arch/mips/kernel/mips_ksyms.c
index 6e58e97..a322db0 100644
--- a/arch/mips/kernel/mips_ksyms.c
+++ b/arch/mips/kernel/mips_ksyms.c
@@ -22,6 +22,8 @@ extern long __strncpy_from_user_asm(char *__to, const char *__from,
long __len);
extern long __strlen_user_nocheck_asm(const char *s);
extern long __strlen_user_asm(const char *s);
+extern long __strnlen_kernel_nocheck_asm(const char *s);
+extern long __strnlen_kernel_asm(const char *s);
extern long __strnlen_user_nocheck_asm(const char *s);
extern long __strnlen_user_asm(const char *s);
@@ -48,6 +50,8 @@ EXPORT_SYMBOL(__strncpy_from_user_nocheck_asm);
EXPORT_SYMBOL(__strncpy_from_user_asm);
EXPORT_SYMBOL(__strlen_user_nocheck_asm);
EXPORT_SYMBOL(__strlen_user_asm);
+EXPORT_SYMBOL(__strnlen_kernel_nocheck_asm);
+EXPORT_SYMBOL(__strnlen_kernel_asm);
EXPORT_SYMBOL(__strnlen_user_nocheck_asm);
EXPORT_SYMBOL(__strnlen_user_asm);
diff --git a/arch/mips/lib/strnlen_user.S b/arch/mips/lib/strnlen_user.S
index 4422160..f3af699 100644
--- a/arch/mips/lib/strnlen_user.S
+++ b/arch/mips/lib/strnlen_user.S
@@ -35,7 +35,11 @@ FEXPORT(__strnlen_\func\()_nocheck_asm)
move v0, a0
PTR_ADDU a1, a0 # stop pointer
1: beq v0, a1, 1f # limit reached?
+.ifeqs "\func", "kernel"
EX(lb, t0, (v0), .Lfault\@)
+.else
+ EX(lbe, t0, (v0), .Lfault\@)
+.endif
PTR_ADDIU v0, 1
bnez t0, 1b
1: PTR_SUBU v0, a0
@@ -47,4 +51,20 @@ FEXPORT(__strnlen_\func\()_nocheck_asm)
jr ra
.endm
+#ifndef CONFIG_EVA
+ /* Set aliases */
+ .global __strnlen_user_asm
+ .global __strnlen_user_nocheck_asm
+ .set __strnlen_user_asm, __strnlen_kernel_asm
+ .set __strnlen_user_nocheck_asm, __strnlen_kernel_nocheck_asm
+#endif
+
+__BUILD_STRNLEN_ASM kernel
+
+#ifdef CONFIG_EVA
+
+ .set push
+ .set eva
__BUILD_STRNLEN_ASM user
+ .set pop
+#endif
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 12/58] MIPS: lib: strlen_user: Use macro to build the strlen_user symbol
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Build the __strlen_user symbol using a macro. In EVA mode we will
need to use similar code to do the userspace load operations so
it is better if we use a macro to avoid code duplications.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/strlen_user.S | 16 ++++++++++------
1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/arch/mips/lib/strlen_user.S b/arch/mips/lib/strlen_user.S
index e362dcd..6e8bdb3 100644
--- a/arch/mips/lib/strlen_user.S
+++ b/arch/mips/lib/strlen_user.S
@@ -22,19 +22,23 @@
*
* Return 0 for error
*/
-LEAF(__strlen_user_asm)
+ .macro __BUILD_STRLEN_ASM func
+LEAF(__strlen_\func\()_asm)
LONG_L v0, TI_ADDR_LIMIT($28) # pointer ok?
and v0, a0
- bnez v0, .Lfault
+ bnez v0, .Lfault\@
-FEXPORT(__strlen_user_nocheck_asm)
+FEXPORT(__strlen_\func\()_nocheck_asm)
move v0, a0
-1: EX(lbu, v1, (v0), .Lfault)
+1: EX(lbu, v1, (v0), .Lfault\@)
PTR_ADDIU v0, 1
bnez v1, 1b
PTR_SUBU v0, a0
jr ra
- END(__strlen_user_asm)
+ END(__strlen_\func\()_asm)
-.Lfault: move v0, zero
+.Lfault\@: move v0, zero
jr ra
+ .endm
+
+__BUILD_STRLEN_ASM user
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 12/58] MIPS: lib: strlen_user: Use macro to build the strlen_user symbol
@ 2014-01-27 20:18 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:18 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Build the __strlen_user symbol using a macro. In EVA mode we will
need to use similar code to do the userspace load operations so
it is better if we use a macro to avoid code duplications.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/strlen_user.S | 16 ++++++++++------
1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/arch/mips/lib/strlen_user.S b/arch/mips/lib/strlen_user.S
index e362dcd..6e8bdb3 100644
--- a/arch/mips/lib/strlen_user.S
+++ b/arch/mips/lib/strlen_user.S
@@ -22,19 +22,23 @@
*
* Return 0 for error
*/
-LEAF(__strlen_user_asm)
+ .macro __BUILD_STRLEN_ASM func
+LEAF(__strlen_\func\()_asm)
LONG_L v0, TI_ADDR_LIMIT($28) # pointer ok?
and v0, a0
- bnez v0, .Lfault
+ bnez v0, .Lfault\@
-FEXPORT(__strlen_user_nocheck_asm)
+FEXPORT(__strlen_\func\()_nocheck_asm)
move v0, a0
-1: EX(lbu, v1, (v0), .Lfault)
+1: EX(lbu, v1, (v0), .Lfault\@)
PTR_ADDIU v0, 1
bnez v1, 1b
PTR_SUBU v0, a0
jr ra
- END(__strlen_user_asm)
+ END(__strlen_\func\()_asm)
-.Lfault: move v0, zero
+.Lfault\@: move v0, zero
jr ra
+ .endm
+
+__BUILD_STRLEN_ASM user
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 13/58] MIPS: lib: strlen_user: Add EVA support
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
In non-EVA mode, strlen_user* aliases are used for the
strlen_kernel* symbols since the code is identical. In EVA
mode, new strlen_user* symbols are used which use the EVA
specific instructions to load values from userspace.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/mips_ksyms.c | 4 ++++
arch/mips/lib/strlen_user.S | 20 ++++++++++++++++++++
2 files changed, 24 insertions(+)
diff --git a/arch/mips/kernel/mips_ksyms.c b/arch/mips/kernel/mips_ksyms.c
index a322db0..742ed7d 100644
--- a/arch/mips/kernel/mips_ksyms.c
+++ b/arch/mips/kernel/mips_ksyms.c
@@ -20,6 +20,8 @@ extern long __strncpy_from_user_nocheck_asm(char *__to,
const char *__from, long __len);
extern long __strncpy_from_user_asm(char *__to, const char *__from,
long __len);
+extern long __strlen_kernel_nocheck_asm(const char *s);
+extern long __strlen_kernel_asm(const char *s);
extern long __strlen_user_nocheck_asm(const char *s);
extern long __strlen_user_asm(const char *s);
extern long __strnlen_kernel_nocheck_asm(const char *s);
@@ -48,6 +50,8 @@ EXPORT_SYMBOL(__copy_user_inatomic);
EXPORT_SYMBOL(__bzero);
EXPORT_SYMBOL(__strncpy_from_user_nocheck_asm);
EXPORT_SYMBOL(__strncpy_from_user_asm);
+EXPORT_SYMBOL(__strlen_kernel_nocheck_asm);
+EXPORT_SYMBOL(__strlen_kernel_asm);
EXPORT_SYMBOL(__strlen_user_nocheck_asm);
EXPORT_SYMBOL(__strlen_user_asm);
EXPORT_SYMBOL(__strnlen_kernel_nocheck_asm);
diff --git a/arch/mips/lib/strlen_user.S b/arch/mips/lib/strlen_user.S
index 6e8bdb3..bef65c9 100644
--- a/arch/mips/lib/strlen_user.S
+++ b/arch/mips/lib/strlen_user.S
@@ -30,7 +30,11 @@ LEAF(__strlen_\func\()_asm)
FEXPORT(__strlen_\func\()_nocheck_asm)
move v0, a0
+.ifeqs "\func", "kernel"
1: EX(lbu, v1, (v0), .Lfault\@)
+.else
+1: EX(lbue, v1, (v0), .Lfault\@)
+.endif
PTR_ADDIU v0, 1
bnez v1, 1b
PTR_SUBU v0, a0
@@ -41,4 +45,20 @@ FEXPORT(__strlen_\func\()_nocheck_asm)
jr ra
.endm
+#ifndef CONFIG_EVA
+ /* Set aliases */
+ .global __strlen_user_asm
+ .global __strlen_user_nocheck_asm
+ .set __strlen_user_asm, __strlen_kernel_asm
+ .set __strlen_user_nocheck_asm, __strlen_kernel_nocheck_asm
+#endif
+
+__BUILD_STRLEN_ASM kernel
+
+#ifdef CONFIG_EVA
+
+ .set push
+ .set eva
__BUILD_STRLEN_ASM user
+ .set pop
+#endif
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 13/58] MIPS: lib: strlen_user: Add EVA support
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
In non-EVA mode, strlen_user* aliases are used for the
strlen_kernel* symbols since the code is identical. In EVA
mode, new strlen_user* symbols are used which use the EVA
specific instructions to load values from userspace.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/mips_ksyms.c | 4 ++++
arch/mips/lib/strlen_user.S | 20 ++++++++++++++++++++
2 files changed, 24 insertions(+)
diff --git a/arch/mips/kernel/mips_ksyms.c b/arch/mips/kernel/mips_ksyms.c
index a322db0..742ed7d 100644
--- a/arch/mips/kernel/mips_ksyms.c
+++ b/arch/mips/kernel/mips_ksyms.c
@@ -20,6 +20,8 @@ extern long __strncpy_from_user_nocheck_asm(char *__to,
const char *__from, long __len);
extern long __strncpy_from_user_asm(char *__to, const char *__from,
long __len);
+extern long __strlen_kernel_nocheck_asm(const char *s);
+extern long __strlen_kernel_asm(const char *s);
extern long __strlen_user_nocheck_asm(const char *s);
extern long __strlen_user_asm(const char *s);
extern long __strnlen_kernel_nocheck_asm(const char *s);
@@ -48,6 +50,8 @@ EXPORT_SYMBOL(__copy_user_inatomic);
EXPORT_SYMBOL(__bzero);
EXPORT_SYMBOL(__strncpy_from_user_nocheck_asm);
EXPORT_SYMBOL(__strncpy_from_user_asm);
+EXPORT_SYMBOL(__strlen_kernel_nocheck_asm);
+EXPORT_SYMBOL(__strlen_kernel_asm);
EXPORT_SYMBOL(__strlen_user_nocheck_asm);
EXPORT_SYMBOL(__strlen_user_asm);
EXPORT_SYMBOL(__strnlen_kernel_nocheck_asm);
diff --git a/arch/mips/lib/strlen_user.S b/arch/mips/lib/strlen_user.S
index 6e8bdb3..bef65c9 100644
--- a/arch/mips/lib/strlen_user.S
+++ b/arch/mips/lib/strlen_user.S
@@ -30,7 +30,11 @@ LEAF(__strlen_\func\()_asm)
FEXPORT(__strlen_\func\()_nocheck_asm)
move v0, a0
+.ifeqs "\func", "kernel"
1: EX(lbu, v1, (v0), .Lfault\@)
+.else
+1: EX(lbue, v1, (v0), .Lfault\@)
+.endif
PTR_ADDIU v0, 1
bnez v1, 1b
PTR_SUBU v0, a0
@@ -41,4 +45,20 @@ FEXPORT(__strlen_\func\()_nocheck_asm)
jr ra
.endm
+#ifndef CONFIG_EVA
+ /* Set aliases */
+ .global __strlen_user_asm
+ .global __strlen_user_nocheck_asm
+ .set __strlen_user_asm, __strlen_kernel_asm
+ .set __strlen_user_nocheck_asm, __strlen_kernel_nocheck_asm
+#endif
+
+__BUILD_STRLEN_ASM kernel
+
+#ifdef CONFIG_EVA
+
+ .set push
+ .set eva
__BUILD_STRLEN_ASM user
+ .set pop
+#endif
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 14/58] MIPS: lib: strncpy_user: Use macro to build the strncpy_from_user symbol
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Build the __strncpy_from_user symbol using a macro. In EVA mode we will
need to use similar code to do the userspace load operations so
it is better if we use a macro to avoid code duplications.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/strncpy_user.S | 21 +++++++++++++--------
1 file changed, 13 insertions(+), 8 deletions(-)
diff --git a/arch/mips/lib/strncpy_user.S b/arch/mips/lib/strncpy_user.S
index 92870b6..51b38ab 100644
--- a/arch/mips/lib/strncpy_user.S
+++ b/arch/mips/lib/strncpy_user.S
@@ -28,16 +28,17 @@
* it happens at most some bytes of the exceptions handlers will be copied.
*/
-LEAF(__strncpy_from_user_asm)
+ .macro __BUILD_STRNCPY_ASM func
+LEAF(__strncpy_from_\func\()_asm)
LONG_L v0, TI_ADDR_LIMIT($28) # pointer ok?
and v0, a1
- bnez v0, .Lfault
+ bnez v0, .Lfault\@
-FEXPORT(__strncpy_from_user_nocheck_asm)
+FEXPORT(__strncpy_from_\func\()_nocheck_asm)
.set noreorder
move t0, zero
move v1, a1
-1: EX(lbu, v0, (v1), .Lfault)
+1: EX(lbu, v0, (v1), .Lfault\@)
PTR_ADDIU v1, 1
R10KCBARRIER(0(ra))
beqz v0, 2f
@@ -47,15 +48,19 @@ FEXPORT(__strncpy_from_user_nocheck_asm)
PTR_ADDIU a0, 1
2: PTR_ADDU v0, a1, t0
xor v0, a1
- bltz v0, .Lfault
+ bltz v0, .Lfault\@
nop
jr ra # return n
move v0, t0
- END(__strncpy_from_user_asm)
+ END(__strncpy_from_\func\()_asm)
-.Lfault: jr ra
+.Lfault\@: jr ra
li v0, -EFAULT
.section __ex_table,"a"
- PTR 1b, .Lfault
+ PTR 1b, .Lfault\@
.previous
+
+ .endm
+
+__BUILD_STRNCPY_ASM user
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 14/58] MIPS: lib: strncpy_user: Use macro to build the strncpy_from_user symbol
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Build the __strncpy_from_user symbol using a macro. In EVA mode we will
need to use similar code to do the userspace load operations so
it is better if we use a macro to avoid code duplications.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/strncpy_user.S | 21 +++++++++++++--------
1 file changed, 13 insertions(+), 8 deletions(-)
diff --git a/arch/mips/lib/strncpy_user.S b/arch/mips/lib/strncpy_user.S
index 92870b6..51b38ab 100644
--- a/arch/mips/lib/strncpy_user.S
+++ b/arch/mips/lib/strncpy_user.S
@@ -28,16 +28,17 @@
* it happens at most some bytes of the exceptions handlers will be copied.
*/
-LEAF(__strncpy_from_user_asm)
+ .macro __BUILD_STRNCPY_ASM func
+LEAF(__strncpy_from_\func\()_asm)
LONG_L v0, TI_ADDR_LIMIT($28) # pointer ok?
and v0, a1
- bnez v0, .Lfault
+ bnez v0, .Lfault\@
-FEXPORT(__strncpy_from_user_nocheck_asm)
+FEXPORT(__strncpy_from_\func\()_nocheck_asm)
.set noreorder
move t0, zero
move v1, a1
-1: EX(lbu, v0, (v1), .Lfault)
+1: EX(lbu, v0, (v1), .Lfault\@)
PTR_ADDIU v1, 1
R10KCBARRIER(0(ra))
beqz v0, 2f
@@ -47,15 +48,19 @@ FEXPORT(__strncpy_from_user_nocheck_asm)
PTR_ADDIU a0, 1
2: PTR_ADDU v0, a1, t0
xor v0, a1
- bltz v0, .Lfault
+ bltz v0, .Lfault\@
nop
jr ra # return n
move v0, t0
- END(__strncpy_from_user_asm)
+ END(__strncpy_from_\func\()_asm)
-.Lfault: jr ra
+.Lfault\@: jr ra
li v0, -EFAULT
.section __ex_table,"a"
- PTR 1b, .Lfault
+ PTR 1b, .Lfault\@
.previous
+
+ .endm
+
+__BUILD_STRNCPY_ASM user
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 15/58] MIPS: lib: strncpy_user: Add EVA support
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
In non-EVA mode, strncpy_from_user* aliases are used for the
strncpy_from_kernel* symbols since the code is identical. In EVA
mode, new strcpy_from_user* symbols are used which use the EVA
specific instructions to load values from userspace.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/mips_ksyms.c | 6 ++++++
arch/mips/lib/strncpy_user.S | 19 +++++++++++++++++++
2 files changed, 25 insertions(+)
diff --git a/arch/mips/kernel/mips_ksyms.c b/arch/mips/kernel/mips_ksyms.c
index 742ed7d..675bd05 100644
--- a/arch/mips/kernel/mips_ksyms.c
+++ b/arch/mips/kernel/mips_ksyms.c
@@ -16,6 +16,10 @@
#include <asm/ftrace.h>
extern void *__bzero(void *__s, size_t __count);
+extern long __strncpy_from_kernel_nocheck_asm(char *__to,
+ const char *__from, long __len);
+extern long __strncpy_from_kernel_asm(char *__to, const char *__from,
+ long __len);
extern long __strncpy_from_user_nocheck_asm(char *__to,
const char *__from, long __len);
extern long __strncpy_from_user_asm(char *__to, const char *__from,
@@ -48,6 +52,8 @@ EXPORT_SYMBOL(copy_page);
EXPORT_SYMBOL(__copy_user);
EXPORT_SYMBOL(__copy_user_inatomic);
EXPORT_SYMBOL(__bzero);
+EXPORT_SYMBOL(__strncpy_from_kernel_nocheck_asm);
+EXPORT_SYMBOL(__strncpy_from_kernel_asm);
EXPORT_SYMBOL(__strncpy_from_user_nocheck_asm);
EXPORT_SYMBOL(__strncpy_from_user_asm);
EXPORT_SYMBOL(__strlen_kernel_nocheck_asm);
diff --git a/arch/mips/lib/strncpy_user.S b/arch/mips/lib/strncpy_user.S
index 51b38ab..d3301cd 100644
--- a/arch/mips/lib/strncpy_user.S
+++ b/arch/mips/lib/strncpy_user.S
@@ -38,7 +38,11 @@ FEXPORT(__strncpy_from_\func\()_nocheck_asm)
.set noreorder
move t0, zero
move v1, a1
+.ifeqs "\func","kernel"
1: EX(lbu, v0, (v1), .Lfault\@)
+.else
+1: EX(lbue, v0, (v1), .Lfault\@)
+.endif
PTR_ADDIU v1, 1
R10KCBARRIER(0(ra))
beqz v0, 2f
@@ -63,4 +67,19 @@ FEXPORT(__strncpy_from_\func\()_nocheck_asm)
.endm
+#ifndef CONFIG_EVA
+ /* Set aliases */
+ .global __strncpy_from_user_asm
+ .global __strncpy_from_user_nocheck_asm
+ .set __strncpy_from_user_asm, __strncpy_from_kernel_asm
+ .set __strncpy_from_user_nocheck_asm, __strncpy_from_kernel_nocheck_asm
+#endif
+
+__BUILD_STRNCPY_ASM kernel
+
+#ifdef CONFIG_EVA
+ .set push
+ .set eva
__BUILD_STRNCPY_ASM user
+ .set pop
+#endif
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 15/58] MIPS: lib: strncpy_user: Add EVA support
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
In non-EVA mode, strncpy_from_user* aliases are used for the
strncpy_from_kernel* symbols since the code is identical. In EVA
mode, new strcpy_from_user* symbols are used which use the EVA
specific instructions to load values from userspace.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/mips_ksyms.c | 6 ++++++
arch/mips/lib/strncpy_user.S | 19 +++++++++++++++++++
2 files changed, 25 insertions(+)
diff --git a/arch/mips/kernel/mips_ksyms.c b/arch/mips/kernel/mips_ksyms.c
index 742ed7d..675bd05 100644
--- a/arch/mips/kernel/mips_ksyms.c
+++ b/arch/mips/kernel/mips_ksyms.c
@@ -16,6 +16,10 @@
#include <asm/ftrace.h>
extern void *__bzero(void *__s, size_t __count);
+extern long __strncpy_from_kernel_nocheck_asm(char *__to,
+ const char *__from, long __len);
+extern long __strncpy_from_kernel_asm(char *__to, const char *__from,
+ long __len);
extern long __strncpy_from_user_nocheck_asm(char *__to,
const char *__from, long __len);
extern long __strncpy_from_user_asm(char *__to, const char *__from,
@@ -48,6 +52,8 @@ EXPORT_SYMBOL(copy_page);
EXPORT_SYMBOL(__copy_user);
EXPORT_SYMBOL(__copy_user_inatomic);
EXPORT_SYMBOL(__bzero);
+EXPORT_SYMBOL(__strncpy_from_kernel_nocheck_asm);
+EXPORT_SYMBOL(__strncpy_from_kernel_asm);
EXPORT_SYMBOL(__strncpy_from_user_nocheck_asm);
EXPORT_SYMBOL(__strncpy_from_user_asm);
EXPORT_SYMBOL(__strlen_kernel_nocheck_asm);
diff --git a/arch/mips/lib/strncpy_user.S b/arch/mips/lib/strncpy_user.S
index 51b38ab..d3301cd 100644
--- a/arch/mips/lib/strncpy_user.S
+++ b/arch/mips/lib/strncpy_user.S
@@ -38,7 +38,11 @@ FEXPORT(__strncpy_from_\func\()_nocheck_asm)
.set noreorder
move t0, zero
move v1, a1
+.ifeqs "\func","kernel"
1: EX(lbu, v0, (v1), .Lfault\@)
+.else
+1: EX(lbue, v0, (v1), .Lfault\@)
+.endif
PTR_ADDIU v1, 1
R10KCBARRIER(0(ra))
beqz v0, 2f
@@ -63,4 +67,19 @@ FEXPORT(__strncpy_from_\func\()_nocheck_asm)
.endm
+#ifndef CONFIG_EVA
+ /* Set aliases */
+ .global __strncpy_from_user_asm
+ .global __strncpy_from_user_nocheck_asm
+ .set __strncpy_from_user_asm, __strncpy_from_kernel_asm
+ .set __strncpy_from_user_nocheck_asm, __strncpy_from_kernel_nocheck_asm
+#endif
+
+__BUILD_STRNCPY_ASM kernel
+
+#ifdef CONFIG_EVA
+ .set push
+ .set eva
__BUILD_STRNCPY_ASM user
+ .set pop
+#endif
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 16/58] MIPS: lib: memcpy: Merge EXC and load/store macros
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Each load/store macro always adds an entry to the __ex_table
using the EXC macro. Therefore, these load/store macros are now merged
with the EXC one. The argument list is also expanded in order to make
the macro more flexible. The extra 'type' argument is not used by this
commit, but it will be used when the EVA support is added to the memcpy.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/memcpy.S | 158 ++++++++++++++++++++++++++++---------------------
1 file changed, 89 insertions(+), 69 deletions(-)
diff --git a/arch/mips/lib/memcpy.S b/arch/mips/lib/memcpy.S
index c5c40da..ab9f046 100644
--- a/arch/mips/lib/memcpy.S
+++ b/arch/mips/lib/memcpy.S
@@ -10,6 +10,7 @@
* Copyright (C) 2002 Broadcom, Inc.
* memcpy/copy_user author: Mark Vandevoorde
* Copyright (C) 2007 Maciej W. Rozycki
+ * Copyright (C) 2014 Imagination Technologies Ltd.
*
* Mnemonic names for arguments to memcpy/__copy_user
*/
@@ -85,8 +86,22 @@
* they're not protected.
*/
-#define EXC(inst_reg,addr,handler) \
-9: inst_reg, addr; \
+/* Instruction type */
+#define LD_INSN 1
+#define ST_INSN 2
+
+/*
+ * Wrapper to add an entry in the exception table
+ * in case the insn causes a memory exception.
+ * Arguments:
+ * insn : Load/store instruction
+ * type : Instruction type
+ * reg : Register
+ * addr : Address
+ * handler : Exception handler
+ */
+#define EXC(insn, type, reg, addr, handler) \
+9: insn reg, addr; \
.section __ex_table,"a"; \
PTR 9b, handler; \
.previous
@@ -100,12 +115,13 @@
#ifdef USE_DOUBLE
-#define LOAD ld
-#define LOADL ldl
-#define LOADR ldr
-#define STOREL sdl
-#define STORER sdr
-#define STORE sd
+#define LOADK ld /* No exception */
+#define LOAD(reg, addr, handler) EXC(ld, LD_INSN, reg, addr, handler)
+#define LOADL(reg, addr, handler) EXC(ldl, LD_INSN, reg, addr, handler)
+#define LOADR(reg, addr, handler) EXC(ldr, LD_INSN, reg, addr, handler)
+#define STOREL(reg, addr, handler) EXC(sdl, ST_INSN, reg, addr, handler)
+#define STORER(reg, addr, handler) EXC(sdr, ST_INSN, reg, addr, handler)
+#define STORE(reg, addr, handler) EXC(sd, ST_INSN, reg, addr, handler)
#define ADD daddu
#define SUB dsubu
#define SRL dsrl
@@ -136,12 +152,13 @@
#else
-#define LOAD lw
-#define LOADL lwl
-#define LOADR lwr
-#define STOREL swl
-#define STORER swr
-#define STORE sw
+#define LOADK lw /* No exception */
+#define LOAD(reg, addr, handler) EXC(lw, LD_INSN, reg, addr, handler)
+#define LOADL(reg, addr, handler) EXC(lwl, LD_INSN, reg, addr, handler)
+#define LOADR(reg, addr, handler) EXC(lwr, LD_INSN, reg, addr, handler)
+#define STOREL(reg, addr, handler) EXC(swl, ST_INSN, reg, addr, handler)
+#define STORER(reg, addr, handler) EXC(swr, ST_INSN, reg, addr, handler)
+#define STORE(reg, addr, handler) EXC(sw, ST_INSN, reg, addr, handler)
#define ADD addu
#define SUB subu
#define SRL srl
@@ -154,6 +171,9 @@
#endif /* USE_DOUBLE */
+#define LOADB(reg, addr, handler) EXC(lb, LD_INSN, reg, addr, handler)
+#define STOREB(reg, addr, handler) EXC(sb, ST_INSN, reg, addr, handler)
+
#ifdef CONFIG_CPU_LITTLE_ENDIAN
#define LDFIRST LOADR
#define LDREST LOADL
@@ -243,25 +263,25 @@ __copy_user_common:
.align 4
1:
R10KCBARRIER(0(ra))
-EXC( LOAD t0, UNIT(0)(src), .Ll_exc)
-EXC( LOAD t1, UNIT(1)(src), .Ll_exc_copy)
-EXC( LOAD t2, UNIT(2)(src), .Ll_exc_copy)
-EXC( LOAD t3, UNIT(3)(src), .Ll_exc_copy)
+ LOAD(t0, UNIT(0)(src), .Ll_exc)
+ LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
+ LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
+ LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
SUB len, len, 8*NBYTES
-EXC( LOAD t4, UNIT(4)(src), .Ll_exc_copy)
-EXC( LOAD t7, UNIT(5)(src), .Ll_exc_copy)
-EXC( STORE t0, UNIT(0)(dst), .Ls_exc_p8u)
-EXC( STORE t1, UNIT(1)(dst), .Ls_exc_p7u)
-EXC( LOAD t0, UNIT(6)(src), .Ll_exc_copy)
-EXC( LOAD t1, UNIT(7)(src), .Ll_exc_copy)
+ LOAD(t4, UNIT(4)(src), .Ll_exc_copy)
+ LOAD(t7, UNIT(5)(src), .Ll_exc_copy)
+ STORE(t0, UNIT(0)(dst), .Ls_exc_p8u)
+ STORE(t1, UNIT(1)(dst), .Ls_exc_p7u)
+ LOAD(t0, UNIT(6)(src), .Ll_exc_copy)
+ LOAD(t1, UNIT(7)(src), .Ll_exc_copy)
ADD src, src, 8*NBYTES
ADD dst, dst, 8*NBYTES
-EXC( STORE t2, UNIT(-6)(dst), .Ls_exc_p6u)
-EXC( STORE t3, UNIT(-5)(dst), .Ls_exc_p5u)
-EXC( STORE t4, UNIT(-4)(dst), .Ls_exc_p4u)
-EXC( STORE t7, UNIT(-3)(dst), .Ls_exc_p3u)
-EXC( STORE t0, UNIT(-2)(dst), .Ls_exc_p2u)
-EXC( STORE t1, UNIT(-1)(dst), .Ls_exc_p1u)
+ STORE(t2, UNIT(-6)(dst), .Ls_exc_p6u)
+ STORE(t3, UNIT(-5)(dst), .Ls_exc_p5u)
+ STORE(t4, UNIT(-4)(dst), .Ls_exc_p4u)
+ STORE(t7, UNIT(-3)(dst), .Ls_exc_p3u)
+ STORE(t0, UNIT(-2)(dst), .Ls_exc_p2u)
+ STORE(t1, UNIT(-1)(dst), .Ls_exc_p1u)
PREF( 0, 8*32(src) )
PREF( 1, 8*32(dst) )
bne len, rem, 1b
@@ -278,17 +298,17 @@ EXC( STORE t1, UNIT(-1)(dst), .Ls_exc_p1u)
/*
* len >= 4*NBYTES
*/
-EXC( LOAD t0, UNIT(0)(src), .Ll_exc)
-EXC( LOAD t1, UNIT(1)(src), .Ll_exc_copy)
-EXC( LOAD t2, UNIT(2)(src), .Ll_exc_copy)
-EXC( LOAD t3, UNIT(3)(src), .Ll_exc_copy)
+ LOAD( t0, UNIT(0)(src), .Ll_exc)
+ LOAD( t1, UNIT(1)(src), .Ll_exc_copy)
+ LOAD( t2, UNIT(2)(src), .Ll_exc_copy)
+ LOAD( t3, UNIT(3)(src), .Ll_exc_copy)
SUB len, len, 4*NBYTES
ADD src, src, 4*NBYTES
R10KCBARRIER(0(ra))
-EXC( STORE t0, UNIT(0)(dst), .Ls_exc_p4u)
-EXC( STORE t1, UNIT(1)(dst), .Ls_exc_p3u)
-EXC( STORE t2, UNIT(2)(dst), .Ls_exc_p2u)
-EXC( STORE t3, UNIT(3)(dst), .Ls_exc_p1u)
+ STORE(t0, UNIT(0)(dst), .Ls_exc_p4u)
+ STORE(t1, UNIT(1)(dst), .Ls_exc_p3u)
+ STORE(t2, UNIT(2)(dst), .Ls_exc_p2u)
+ STORE(t3, UNIT(3)(dst), .Ls_exc_p1u)
.set reorder /* DADDI_WAR */
ADD dst, dst, 4*NBYTES
beqz len, .Ldone
@@ -301,10 +321,10 @@ EXC( STORE t3, UNIT(3)(dst), .Ls_exc_p1u)
nop
1:
R10KCBARRIER(0(ra))
-EXC( LOAD t0, 0(src), .Ll_exc)
+ LOAD(t0, 0(src), .Ll_exc)
ADD src, src, NBYTES
SUB len, len, NBYTES
-EXC( STORE t0, 0(dst), .Ls_exc_p1u)
+ STORE(t0, 0(dst), .Ls_exc_p1u)
.set reorder /* DADDI_WAR */
ADD dst, dst, NBYTES
bne rem, len, 1b
@@ -326,10 +346,10 @@ EXC( STORE t0, 0(dst), .Ls_exc_p1u)
ADD t1, dst, len # t1 is just past last byte of dst
li bits, 8*NBYTES
SLL rem, len, 3 # rem = number of bits to keep
-EXC( LOAD t0, 0(src), .Ll_exc)
+ LOAD(t0, 0(src), .Ll_exc)
SUB bits, bits, rem # bits = number of bits to discard
SHIFT_DISCARD t0, t0, bits
-EXC( STREST t0, -1(t1), .Ls_exc)
+ STREST(t0, -1(t1), .Ls_exc)
jr ra
move len, zero
.Ldst_unaligned:
@@ -343,13 +363,13 @@ EXC( STREST t0, -1(t1), .Ls_exc)
* Set match = (src and dst have same alignment)
*/
#define match rem
-EXC( LDFIRST t3, FIRST(0)(src), .Ll_exc)
+ LDFIRST(t3, FIRST(0)(src), .Ll_exc)
ADD t2, zero, NBYTES
-EXC( LDREST t3, REST(0)(src), .Ll_exc_copy)
+ LDREST(t3, REST(0)(src), .Ll_exc_copy)
SUB t2, t2, t1 # t2 = number of bytes copied
xor match, t0, t1
R10KCBARRIER(0(ra))
-EXC( STFIRST t3, FIRST(0)(dst), .Ls_exc)
+ STFIRST(t3, FIRST(0)(dst), .Ls_exc)
beq len, t2, .Ldone
SUB len, len, t2
ADD dst, dst, t2
@@ -370,24 +390,24 @@ EXC( STFIRST t3, FIRST(0)(dst), .Ls_exc)
* are to the same unit (unless src is aligned, but it's not).
*/
R10KCBARRIER(0(ra))
-EXC( LDFIRST t0, FIRST(0)(src), .Ll_exc)
-EXC( LDFIRST t1, FIRST(1)(src), .Ll_exc_copy)
+ LDFIRST(t0, FIRST(0)(src), .Ll_exc)
+ LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy)
SUB len, len, 4*NBYTES
-EXC( LDREST t0, REST(0)(src), .Ll_exc_copy)
-EXC( LDREST t1, REST(1)(src), .Ll_exc_copy)
-EXC( LDFIRST t2, FIRST(2)(src), .Ll_exc_copy)
-EXC( LDFIRST t3, FIRST(3)(src), .Ll_exc_copy)
-EXC( LDREST t2, REST(2)(src), .Ll_exc_copy)
-EXC( LDREST t3, REST(3)(src), .Ll_exc_copy)
+ LDREST(t0, REST(0)(src), .Ll_exc_copy)
+ LDREST(t1, REST(1)(src), .Ll_exc_copy)
+ LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy)
+ LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy)
+ LDREST(t2, REST(2)(src), .Ll_exc_copy)
+ LDREST(t3, REST(3)(src), .Ll_exc_copy)
PREF( 0, 9*32(src) ) # 0 is PREF_LOAD (not streamed)
ADD src, src, 4*NBYTES
#ifdef CONFIG_CPU_SB1
nop # improves slotting
#endif
-EXC( STORE t0, UNIT(0)(dst), .Ls_exc_p4u)
-EXC( STORE t1, UNIT(1)(dst), .Ls_exc_p3u)
-EXC( STORE t2, UNIT(2)(dst), .Ls_exc_p2u)
-EXC( STORE t3, UNIT(3)(dst), .Ls_exc_p1u)
+ STORE(t0, UNIT(0)(dst), .Ls_exc_p4u)
+ STORE(t1, UNIT(1)(dst), .Ls_exc_p3u)
+ STORE(t2, UNIT(2)(dst), .Ls_exc_p2u)
+ STORE(t3, UNIT(3)(dst), .Ls_exc_p1u)
PREF( 1, 9*32(dst) ) # 1 is PREF_STORE (not streamed)
.set reorder /* DADDI_WAR */
ADD dst, dst, 4*NBYTES
@@ -401,11 +421,11 @@ EXC( STORE t3, UNIT(3)(dst), .Ls_exc_p1u)
nop
1:
R10KCBARRIER(0(ra))
-EXC( LDFIRST t0, FIRST(0)(src), .Ll_exc)
-EXC( LDREST t0, REST(0)(src), .Ll_exc_copy)
+ LDFIRST(t0, FIRST(0)(src), .Ll_exc)
+ LDREST(t0, REST(0)(src), .Ll_exc_copy)
ADD src, src, NBYTES
SUB len, len, NBYTES
-EXC( STORE t0, 0(dst), .Ls_exc_p1u)
+ STORE(t0, 0(dst), .Ls_exc_p1u)
.set reorder /* DADDI_WAR */
ADD dst, dst, NBYTES
bne len, rem, 1b
@@ -418,10 +438,10 @@ EXC( STORE t0, 0(dst), .Ls_exc_p1u)
/* 0 < len < NBYTES */
R10KCBARRIER(0(ra))
#define COPY_BYTE(N) \
-EXC( lb t0, N(src), .Ll_exc); \
+ LOADB(t0, N(src), .Ll_exc); \
SUB len, len, 1; \
beqz len, .Ldone; \
-EXC( sb t0, N(dst), .Ls_exc_p1)
+ STOREB(t0, N(dst), .Ls_exc_p1)
COPY_BYTE(0)
COPY_BYTE(1)
@@ -431,10 +451,10 @@ EXC( sb t0, N(dst), .Ls_exc_p1)
COPY_BYTE(4)
COPY_BYTE(5)
#endif
-EXC( lb t0, NBYTES-2(src), .Ll_exc)
+ LOADB(t0, NBYTES-2(src), .Ll_exc)
SUB len, len, 1
jr ra
-EXC( sb t0, NBYTES-2(dst), .Ls_exc_p1)
+ STOREB(t0, NBYTES-2(dst), .Ls_exc_p1)
.Ldone:
jr ra
nop
@@ -451,11 +471,11 @@ EXC( sb t0, NBYTES-2(dst), .Ls_exc_p1)
*
* Assumes src < THREAD_BUADDR($28)
*/
- LOAD t0, TI_TASK($28)
+ LOADK t0, TI_TASK($28)
nop
- LOAD t0, THREAD_BUADDR(t0)
+ LOADK t0, THREAD_BUADDR(t0)
1:
-EXC( lb t1, 0(src), .Ll_exc)
+ LOADB(t1, 0(src), .Ll_exc)
ADD src, src, 1
sb t1, 0(dst) # can't fault -- we're copy_from_user
.set reorder /* DADDI_WAR */
@@ -463,9 +483,9 @@ EXC( lb t1, 0(src), .Ll_exc)
bne src, t0, 1b
.set noreorder
.Ll_exc:
- LOAD t0, TI_TASK($28)
+ LOADK t0, TI_TASK($28)
nop
- LOAD t0, THREAD_BUADDR(t0) # t0 is just past last good address
+ LOADK t0, THREAD_BUADDR(t0) # t0 is just past last good address
nop
SUB len, AT, t0 # len number of uncopied bytes
bnez t6, .Ldone /* Skip the zeroing part if inatomic */
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 16/58] MIPS: lib: memcpy: Merge EXC and load/store macros
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Each load/store macro always adds an entry to the __ex_table
using the EXC macro. Therefore, these load/store macros are now merged
with the EXC one. The argument list is also expanded in order to make
the macro more flexible. The extra 'type' argument is not used by this
commit, but it will be used when the EVA support is added to the memcpy.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/memcpy.S | 158 ++++++++++++++++++++++++++++---------------------
1 file changed, 89 insertions(+), 69 deletions(-)
diff --git a/arch/mips/lib/memcpy.S b/arch/mips/lib/memcpy.S
index c5c40da..ab9f046 100644
--- a/arch/mips/lib/memcpy.S
+++ b/arch/mips/lib/memcpy.S
@@ -10,6 +10,7 @@
* Copyright (C) 2002 Broadcom, Inc.
* memcpy/copy_user author: Mark Vandevoorde
* Copyright (C) 2007 Maciej W. Rozycki
+ * Copyright (C) 2014 Imagination Technologies Ltd.
*
* Mnemonic names for arguments to memcpy/__copy_user
*/
@@ -85,8 +86,22 @@
* they're not protected.
*/
-#define EXC(inst_reg,addr,handler) \
-9: inst_reg, addr; \
+/* Instruction type */
+#define LD_INSN 1
+#define ST_INSN 2
+
+/*
+ * Wrapper to add an entry in the exception table
+ * in case the insn causes a memory exception.
+ * Arguments:
+ * insn : Load/store instruction
+ * type : Instruction type
+ * reg : Register
+ * addr : Address
+ * handler : Exception handler
+ */
+#define EXC(insn, type, reg, addr, handler) \
+9: insn reg, addr; \
.section __ex_table,"a"; \
PTR 9b, handler; \
.previous
@@ -100,12 +115,13 @@
#ifdef USE_DOUBLE
-#define LOAD ld
-#define LOADL ldl
-#define LOADR ldr
-#define STOREL sdl
-#define STORER sdr
-#define STORE sd
+#define LOADK ld /* No exception */
+#define LOAD(reg, addr, handler) EXC(ld, LD_INSN, reg, addr, handler)
+#define LOADL(reg, addr, handler) EXC(ldl, LD_INSN, reg, addr, handler)
+#define LOADR(reg, addr, handler) EXC(ldr, LD_INSN, reg, addr, handler)
+#define STOREL(reg, addr, handler) EXC(sdl, ST_INSN, reg, addr, handler)
+#define STORER(reg, addr, handler) EXC(sdr, ST_INSN, reg, addr, handler)
+#define STORE(reg, addr, handler) EXC(sd, ST_INSN, reg, addr, handler)
#define ADD daddu
#define SUB dsubu
#define SRL dsrl
@@ -136,12 +152,13 @@
#else
-#define LOAD lw
-#define LOADL lwl
-#define LOADR lwr
-#define STOREL swl
-#define STORER swr
-#define STORE sw
+#define LOADK lw /* No exception */
+#define LOAD(reg, addr, handler) EXC(lw, LD_INSN, reg, addr, handler)
+#define LOADL(reg, addr, handler) EXC(lwl, LD_INSN, reg, addr, handler)
+#define LOADR(reg, addr, handler) EXC(lwr, LD_INSN, reg, addr, handler)
+#define STOREL(reg, addr, handler) EXC(swl, ST_INSN, reg, addr, handler)
+#define STORER(reg, addr, handler) EXC(swr, ST_INSN, reg, addr, handler)
+#define STORE(reg, addr, handler) EXC(sw, ST_INSN, reg, addr, handler)
#define ADD addu
#define SUB subu
#define SRL srl
@@ -154,6 +171,9 @@
#endif /* USE_DOUBLE */
+#define LOADB(reg, addr, handler) EXC(lb, LD_INSN, reg, addr, handler)
+#define STOREB(reg, addr, handler) EXC(sb, ST_INSN, reg, addr, handler)
+
#ifdef CONFIG_CPU_LITTLE_ENDIAN
#define LDFIRST LOADR
#define LDREST LOADL
@@ -243,25 +263,25 @@ __copy_user_common:
.align 4
1:
R10KCBARRIER(0(ra))
-EXC( LOAD t0, UNIT(0)(src), .Ll_exc)
-EXC( LOAD t1, UNIT(1)(src), .Ll_exc_copy)
-EXC( LOAD t2, UNIT(2)(src), .Ll_exc_copy)
-EXC( LOAD t3, UNIT(3)(src), .Ll_exc_copy)
+ LOAD(t0, UNIT(0)(src), .Ll_exc)
+ LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
+ LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
+ LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
SUB len, len, 8*NBYTES
-EXC( LOAD t4, UNIT(4)(src), .Ll_exc_copy)
-EXC( LOAD t7, UNIT(5)(src), .Ll_exc_copy)
-EXC( STORE t0, UNIT(0)(dst), .Ls_exc_p8u)
-EXC( STORE t1, UNIT(1)(dst), .Ls_exc_p7u)
-EXC( LOAD t0, UNIT(6)(src), .Ll_exc_copy)
-EXC( LOAD t1, UNIT(7)(src), .Ll_exc_copy)
+ LOAD(t4, UNIT(4)(src), .Ll_exc_copy)
+ LOAD(t7, UNIT(5)(src), .Ll_exc_copy)
+ STORE(t0, UNIT(0)(dst), .Ls_exc_p8u)
+ STORE(t1, UNIT(1)(dst), .Ls_exc_p7u)
+ LOAD(t0, UNIT(6)(src), .Ll_exc_copy)
+ LOAD(t1, UNIT(7)(src), .Ll_exc_copy)
ADD src, src, 8*NBYTES
ADD dst, dst, 8*NBYTES
-EXC( STORE t2, UNIT(-6)(dst), .Ls_exc_p6u)
-EXC( STORE t3, UNIT(-5)(dst), .Ls_exc_p5u)
-EXC( STORE t4, UNIT(-4)(dst), .Ls_exc_p4u)
-EXC( STORE t7, UNIT(-3)(dst), .Ls_exc_p3u)
-EXC( STORE t0, UNIT(-2)(dst), .Ls_exc_p2u)
-EXC( STORE t1, UNIT(-1)(dst), .Ls_exc_p1u)
+ STORE(t2, UNIT(-6)(dst), .Ls_exc_p6u)
+ STORE(t3, UNIT(-5)(dst), .Ls_exc_p5u)
+ STORE(t4, UNIT(-4)(dst), .Ls_exc_p4u)
+ STORE(t7, UNIT(-3)(dst), .Ls_exc_p3u)
+ STORE(t0, UNIT(-2)(dst), .Ls_exc_p2u)
+ STORE(t1, UNIT(-1)(dst), .Ls_exc_p1u)
PREF( 0, 8*32(src) )
PREF( 1, 8*32(dst) )
bne len, rem, 1b
@@ -278,17 +298,17 @@ EXC( STORE t1, UNIT(-1)(dst), .Ls_exc_p1u)
/*
* len >= 4*NBYTES
*/
-EXC( LOAD t0, UNIT(0)(src), .Ll_exc)
-EXC( LOAD t1, UNIT(1)(src), .Ll_exc_copy)
-EXC( LOAD t2, UNIT(2)(src), .Ll_exc_copy)
-EXC( LOAD t3, UNIT(3)(src), .Ll_exc_copy)
+ LOAD( t0, UNIT(0)(src), .Ll_exc)
+ LOAD( t1, UNIT(1)(src), .Ll_exc_copy)
+ LOAD( t2, UNIT(2)(src), .Ll_exc_copy)
+ LOAD( t3, UNIT(3)(src), .Ll_exc_copy)
SUB len, len, 4*NBYTES
ADD src, src, 4*NBYTES
R10KCBARRIER(0(ra))
-EXC( STORE t0, UNIT(0)(dst), .Ls_exc_p4u)
-EXC( STORE t1, UNIT(1)(dst), .Ls_exc_p3u)
-EXC( STORE t2, UNIT(2)(dst), .Ls_exc_p2u)
-EXC( STORE t3, UNIT(3)(dst), .Ls_exc_p1u)
+ STORE(t0, UNIT(0)(dst), .Ls_exc_p4u)
+ STORE(t1, UNIT(1)(dst), .Ls_exc_p3u)
+ STORE(t2, UNIT(2)(dst), .Ls_exc_p2u)
+ STORE(t3, UNIT(3)(dst), .Ls_exc_p1u)
.set reorder /* DADDI_WAR */
ADD dst, dst, 4*NBYTES
beqz len, .Ldone
@@ -301,10 +321,10 @@ EXC( STORE t3, UNIT(3)(dst), .Ls_exc_p1u)
nop
1:
R10KCBARRIER(0(ra))
-EXC( LOAD t0, 0(src), .Ll_exc)
+ LOAD(t0, 0(src), .Ll_exc)
ADD src, src, NBYTES
SUB len, len, NBYTES
-EXC( STORE t0, 0(dst), .Ls_exc_p1u)
+ STORE(t0, 0(dst), .Ls_exc_p1u)
.set reorder /* DADDI_WAR */
ADD dst, dst, NBYTES
bne rem, len, 1b
@@ -326,10 +346,10 @@ EXC( STORE t0, 0(dst), .Ls_exc_p1u)
ADD t1, dst, len # t1 is just past last byte of dst
li bits, 8*NBYTES
SLL rem, len, 3 # rem = number of bits to keep
-EXC( LOAD t0, 0(src), .Ll_exc)
+ LOAD(t0, 0(src), .Ll_exc)
SUB bits, bits, rem # bits = number of bits to discard
SHIFT_DISCARD t0, t0, bits
-EXC( STREST t0, -1(t1), .Ls_exc)
+ STREST(t0, -1(t1), .Ls_exc)
jr ra
move len, zero
.Ldst_unaligned:
@@ -343,13 +363,13 @@ EXC( STREST t0, -1(t1), .Ls_exc)
* Set match = (src and dst have same alignment)
*/
#define match rem
-EXC( LDFIRST t3, FIRST(0)(src), .Ll_exc)
+ LDFIRST(t3, FIRST(0)(src), .Ll_exc)
ADD t2, zero, NBYTES
-EXC( LDREST t3, REST(0)(src), .Ll_exc_copy)
+ LDREST(t3, REST(0)(src), .Ll_exc_copy)
SUB t2, t2, t1 # t2 = number of bytes copied
xor match, t0, t1
R10KCBARRIER(0(ra))
-EXC( STFIRST t3, FIRST(0)(dst), .Ls_exc)
+ STFIRST(t3, FIRST(0)(dst), .Ls_exc)
beq len, t2, .Ldone
SUB len, len, t2
ADD dst, dst, t2
@@ -370,24 +390,24 @@ EXC( STFIRST t3, FIRST(0)(dst), .Ls_exc)
* are to the same unit (unless src is aligned, but it's not).
*/
R10KCBARRIER(0(ra))
-EXC( LDFIRST t0, FIRST(0)(src), .Ll_exc)
-EXC( LDFIRST t1, FIRST(1)(src), .Ll_exc_copy)
+ LDFIRST(t0, FIRST(0)(src), .Ll_exc)
+ LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy)
SUB len, len, 4*NBYTES
-EXC( LDREST t0, REST(0)(src), .Ll_exc_copy)
-EXC( LDREST t1, REST(1)(src), .Ll_exc_copy)
-EXC( LDFIRST t2, FIRST(2)(src), .Ll_exc_copy)
-EXC( LDFIRST t3, FIRST(3)(src), .Ll_exc_copy)
-EXC( LDREST t2, REST(2)(src), .Ll_exc_copy)
-EXC( LDREST t3, REST(3)(src), .Ll_exc_copy)
+ LDREST(t0, REST(0)(src), .Ll_exc_copy)
+ LDREST(t1, REST(1)(src), .Ll_exc_copy)
+ LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy)
+ LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy)
+ LDREST(t2, REST(2)(src), .Ll_exc_copy)
+ LDREST(t3, REST(3)(src), .Ll_exc_copy)
PREF( 0, 9*32(src) ) # 0 is PREF_LOAD (not streamed)
ADD src, src, 4*NBYTES
#ifdef CONFIG_CPU_SB1
nop # improves slotting
#endif
-EXC( STORE t0, UNIT(0)(dst), .Ls_exc_p4u)
-EXC( STORE t1, UNIT(1)(dst), .Ls_exc_p3u)
-EXC( STORE t2, UNIT(2)(dst), .Ls_exc_p2u)
-EXC( STORE t3, UNIT(3)(dst), .Ls_exc_p1u)
+ STORE(t0, UNIT(0)(dst), .Ls_exc_p4u)
+ STORE(t1, UNIT(1)(dst), .Ls_exc_p3u)
+ STORE(t2, UNIT(2)(dst), .Ls_exc_p2u)
+ STORE(t3, UNIT(3)(dst), .Ls_exc_p1u)
PREF( 1, 9*32(dst) ) # 1 is PREF_STORE (not streamed)
.set reorder /* DADDI_WAR */
ADD dst, dst, 4*NBYTES
@@ -401,11 +421,11 @@ EXC( STORE t3, UNIT(3)(dst), .Ls_exc_p1u)
nop
1:
R10KCBARRIER(0(ra))
-EXC( LDFIRST t0, FIRST(0)(src), .Ll_exc)
-EXC( LDREST t0, REST(0)(src), .Ll_exc_copy)
+ LDFIRST(t0, FIRST(0)(src), .Ll_exc)
+ LDREST(t0, REST(0)(src), .Ll_exc_copy)
ADD src, src, NBYTES
SUB len, len, NBYTES
-EXC( STORE t0, 0(dst), .Ls_exc_p1u)
+ STORE(t0, 0(dst), .Ls_exc_p1u)
.set reorder /* DADDI_WAR */
ADD dst, dst, NBYTES
bne len, rem, 1b
@@ -418,10 +438,10 @@ EXC( STORE t0, 0(dst), .Ls_exc_p1u)
/* 0 < len < NBYTES */
R10KCBARRIER(0(ra))
#define COPY_BYTE(N) \
-EXC( lb t0, N(src), .Ll_exc); \
+ LOADB(t0, N(src), .Ll_exc); \
SUB len, len, 1; \
beqz len, .Ldone; \
-EXC( sb t0, N(dst), .Ls_exc_p1)
+ STOREB(t0, N(dst), .Ls_exc_p1)
COPY_BYTE(0)
COPY_BYTE(1)
@@ -431,10 +451,10 @@ EXC( sb t0, N(dst), .Ls_exc_p1)
COPY_BYTE(4)
COPY_BYTE(5)
#endif
-EXC( lb t0, NBYTES-2(src), .Ll_exc)
+ LOADB(t0, NBYTES-2(src), .Ll_exc)
SUB len, len, 1
jr ra
-EXC( sb t0, NBYTES-2(dst), .Ls_exc_p1)
+ STOREB(t0, NBYTES-2(dst), .Ls_exc_p1)
.Ldone:
jr ra
nop
@@ -451,11 +471,11 @@ EXC( sb t0, NBYTES-2(dst), .Ls_exc_p1)
*
* Assumes src < THREAD_BUADDR($28)
*/
- LOAD t0, TI_TASK($28)
+ LOADK t0, TI_TASK($28)
nop
- LOAD t0, THREAD_BUADDR(t0)
+ LOADK t0, THREAD_BUADDR(t0)
1:
-EXC( lb t1, 0(src), .Ll_exc)
+ LOADB(t1, 0(src), .Ll_exc)
ADD src, src, 1
sb t1, 0(dst) # can't fault -- we're copy_from_user
.set reorder /* DADDI_WAR */
@@ -463,9 +483,9 @@ EXC( lb t1, 0(src), .Ll_exc)
bne src, t0, 1b
.set noreorder
.Ll_exc:
- LOAD t0, TI_TASK($28)
+ LOADK t0, TI_TASK($28)
nop
- LOAD t0, THREAD_BUADDR(t0) # t0 is just past last good address
+ LOADK t0, THREAD_BUADDR(t0) # t0 is just past last good address
nop
SUB len, AT, t0 # len number of uncopied bytes
bnez t6, .Ldone /* Skip the zeroing part if inatomic */
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 17/58] MIPS: lib: memcpy: Split source and destination prefetch macros
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
In preparation for EVA support, the PREF macro is split into two
separate macros, PREFS and PREFD, for source and destination data
prefetching respectively.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/memcpy.S | 36 ++++++++++++++++++++++--------------
1 file changed, 22 insertions(+), 14 deletions(-)
diff --git a/arch/mips/lib/memcpy.S b/arch/mips/lib/memcpy.S
index ab9f046..eed6e07 100644
--- a/arch/mips/lib/memcpy.S
+++ b/arch/mips/lib/memcpy.S
@@ -89,6 +89,9 @@
/* Instruction type */
#define LD_INSN 1
#define ST_INSN 2
+/* Pretech type */
+#define SRC_PREFETCH 1
+#define DST_PREFETCH 2
/*
* Wrapper to add an entry in the exception table
@@ -174,6 +177,11 @@
#define LOADB(reg, addr, handler) EXC(lb, LD_INSN, reg, addr, handler)
#define STOREB(reg, addr, handler) EXC(sb, ST_INSN, reg, addr, handler)
+#define _PREF(hint, addr, type) PREF(hint, addr)
+
+#define PREFS(hint, addr) _PREF(hint, addr, SRC_PREFETCH)
+#define PREFD(hint, addr) _PREF(hint, addr, DST_PREFETCH)
+
#ifdef CONFIG_CPU_LITTLE_ENDIAN
#define LDFIRST LOADR
#define LDREST LOADL
@@ -237,16 +245,16 @@ __copy_user_common:
*
* If len < NBYTES use byte operations.
*/
- PREF( 0, 0(src) )
- PREF( 1, 0(dst) )
+ PREFS( 0, 0(src) )
+ PREFD( 1, 0(dst) )
sltu t2, len, NBYTES
and t1, dst, ADDRMASK
- PREF( 0, 1*32(src) )
- PREF( 1, 1*32(dst) )
+ PREFS( 0, 1*32(src) )
+ PREFD( 1, 1*32(dst) )
bnez t2, .Lcopy_bytes_checklen
and t0, src, ADDRMASK
- PREF( 0, 2*32(src) )
- PREF( 1, 2*32(dst) )
+ PREFS( 0, 2*32(src) )
+ PREFD( 1, 2*32(dst) )
bnez t1, .Ldst_unaligned
nop
bnez t0, .Lsrc_unaligned_dst_aligned
@@ -258,8 +266,8 @@ __copy_user_common:
SRL t0, len, LOG_NBYTES+3 # +3 for 8 units/iter
beqz t0, .Lcleanup_both_aligned # len < 8*NBYTES
and rem, len, (8*NBYTES-1) # rem = len % (8*NBYTES)
- PREF( 0, 3*32(src) )
- PREF( 1, 3*32(dst) )
+ PREFS( 0, 3*32(src) )
+ PREFD( 1, 3*32(dst) )
.align 4
1:
R10KCBARRIER(0(ra))
@@ -282,8 +290,8 @@ __copy_user_common:
STORE(t7, UNIT(-3)(dst), .Ls_exc_p3u)
STORE(t0, UNIT(-2)(dst), .Ls_exc_p2u)
STORE(t1, UNIT(-1)(dst), .Ls_exc_p1u)
- PREF( 0, 8*32(src) )
- PREF( 1, 8*32(dst) )
+ PREFS( 0, 8*32(src) )
+ PREFD( 1, 8*32(dst) )
bne len, rem, 1b
nop
@@ -378,10 +386,10 @@ __copy_user_common:
.Lsrc_unaligned_dst_aligned:
SRL t0, len, LOG_NBYTES+2 # +2 for 4 units/iter
- PREF( 0, 3*32(src) )
+ PREFS( 0, 3*32(src) )
beqz t0, .Lcleanup_src_unaligned
and rem, len, (4*NBYTES-1) # rem = len % 4*NBYTES
- PREF( 1, 3*32(dst) )
+ PREFD( 1, 3*32(dst) )
1:
/*
* Avoid consecutive LD*'s to the same register since some mips
@@ -399,7 +407,7 @@ __copy_user_common:
LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy)
LDREST(t2, REST(2)(src), .Ll_exc_copy)
LDREST(t3, REST(3)(src), .Ll_exc_copy)
- PREF( 0, 9*32(src) ) # 0 is PREF_LOAD (not streamed)
+ PREFS( 0, 9*32(src) ) # 0 is PREF_LOAD (not streamed)
ADD src, src, 4*NBYTES
#ifdef CONFIG_CPU_SB1
nop # improves slotting
@@ -408,7 +416,7 @@ __copy_user_common:
STORE(t1, UNIT(1)(dst), .Ls_exc_p3u)
STORE(t2, UNIT(2)(dst), .Ls_exc_p2u)
STORE(t3, UNIT(3)(dst), .Ls_exc_p1u)
- PREF( 1, 9*32(dst) ) # 1 is PREF_STORE (not streamed)
+ PREFD( 1, 9*32(dst) ) # 1 is PREF_STORE (not streamed)
.set reorder /* DADDI_WAR */
ADD dst, dst, 4*NBYTES
bne len, rem, 1b
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 17/58] MIPS: lib: memcpy: Split source and destination prefetch macros
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
In preparation for EVA support, the PREF macro is split into two
separate macros, PREFS and PREFD, for source and destination data
prefetching respectively.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/memcpy.S | 36 ++++++++++++++++++++++--------------
1 file changed, 22 insertions(+), 14 deletions(-)
diff --git a/arch/mips/lib/memcpy.S b/arch/mips/lib/memcpy.S
index ab9f046..eed6e07 100644
--- a/arch/mips/lib/memcpy.S
+++ b/arch/mips/lib/memcpy.S
@@ -89,6 +89,9 @@
/* Instruction type */
#define LD_INSN 1
#define ST_INSN 2
+/* Pretech type */
+#define SRC_PREFETCH 1
+#define DST_PREFETCH 2
/*
* Wrapper to add an entry in the exception table
@@ -174,6 +177,11 @@
#define LOADB(reg, addr, handler) EXC(lb, LD_INSN, reg, addr, handler)
#define STOREB(reg, addr, handler) EXC(sb, ST_INSN, reg, addr, handler)
+#define _PREF(hint, addr, type) PREF(hint, addr)
+
+#define PREFS(hint, addr) _PREF(hint, addr, SRC_PREFETCH)
+#define PREFD(hint, addr) _PREF(hint, addr, DST_PREFETCH)
+
#ifdef CONFIG_CPU_LITTLE_ENDIAN
#define LDFIRST LOADR
#define LDREST LOADL
@@ -237,16 +245,16 @@ __copy_user_common:
*
* If len < NBYTES use byte operations.
*/
- PREF( 0, 0(src) )
- PREF( 1, 0(dst) )
+ PREFS( 0, 0(src) )
+ PREFD( 1, 0(dst) )
sltu t2, len, NBYTES
and t1, dst, ADDRMASK
- PREF( 0, 1*32(src) )
- PREF( 1, 1*32(dst) )
+ PREFS( 0, 1*32(src) )
+ PREFD( 1, 1*32(dst) )
bnez t2, .Lcopy_bytes_checklen
and t0, src, ADDRMASK
- PREF( 0, 2*32(src) )
- PREF( 1, 2*32(dst) )
+ PREFS( 0, 2*32(src) )
+ PREFD( 1, 2*32(dst) )
bnez t1, .Ldst_unaligned
nop
bnez t0, .Lsrc_unaligned_dst_aligned
@@ -258,8 +266,8 @@ __copy_user_common:
SRL t0, len, LOG_NBYTES+3 # +3 for 8 units/iter
beqz t0, .Lcleanup_both_aligned # len < 8*NBYTES
and rem, len, (8*NBYTES-1) # rem = len % (8*NBYTES)
- PREF( 0, 3*32(src) )
- PREF( 1, 3*32(dst) )
+ PREFS( 0, 3*32(src) )
+ PREFD( 1, 3*32(dst) )
.align 4
1:
R10KCBARRIER(0(ra))
@@ -282,8 +290,8 @@ __copy_user_common:
STORE(t7, UNIT(-3)(dst), .Ls_exc_p3u)
STORE(t0, UNIT(-2)(dst), .Ls_exc_p2u)
STORE(t1, UNIT(-1)(dst), .Ls_exc_p1u)
- PREF( 0, 8*32(src) )
- PREF( 1, 8*32(dst) )
+ PREFS( 0, 8*32(src) )
+ PREFD( 1, 8*32(dst) )
bne len, rem, 1b
nop
@@ -378,10 +386,10 @@ __copy_user_common:
.Lsrc_unaligned_dst_aligned:
SRL t0, len, LOG_NBYTES+2 # +2 for 4 units/iter
- PREF( 0, 3*32(src) )
+ PREFS( 0, 3*32(src) )
beqz t0, .Lcleanup_src_unaligned
and rem, len, (4*NBYTES-1) # rem = len % 4*NBYTES
- PREF( 1, 3*32(dst) )
+ PREFD( 1, 3*32(dst) )
1:
/*
* Avoid consecutive LD*'s to the same register since some mips
@@ -399,7 +407,7 @@ __copy_user_common:
LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy)
LDREST(t2, REST(2)(src), .Ll_exc_copy)
LDREST(t3, REST(3)(src), .Ll_exc_copy)
- PREF( 0, 9*32(src) ) # 0 is PREF_LOAD (not streamed)
+ PREFS( 0, 9*32(src) ) # 0 is PREF_LOAD (not streamed)
ADD src, src, 4*NBYTES
#ifdef CONFIG_CPU_SB1
nop # improves slotting
@@ -408,7 +416,7 @@ __copy_user_common:
STORE(t1, UNIT(1)(dst), .Ls_exc_p3u)
STORE(t2, UNIT(2)(dst), .Ls_exc_p2u)
STORE(t3, UNIT(3)(dst), .Ls_exc_p1u)
- PREF( 1, 9*32(dst) ) # 1 is PREF_STORE (not streamed)
+ PREFD( 1, 9*32(dst) ) # 1 is PREF_STORE (not streamed)
.set reorder /* DADDI_WAR */
ADD dst, dst, 4*NBYTES
bne len, rem, 1b
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 18/58] MIPS: lib: memcpy: Use macro to build the copy_user code
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
The code can be shared between EVA and non-EVA configurations,
therefore use a macro to build it to avoid code duplications.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/memcpy.S | 253 ++++++++++++++++++++++++++++---------------------
1 file changed, 143 insertions(+), 110 deletions(-)
diff --git a/arch/mips/lib/memcpy.S b/arch/mips/lib/memcpy.S
index eed6e07..d630a28 100644
--- a/arch/mips/lib/memcpy.S
+++ b/arch/mips/lib/memcpy.S
@@ -92,6 +92,10 @@
/* Pretech type */
#define SRC_PREFETCH 1
#define DST_PREFETCH 2
+#define LEGACY_MODE 1
+#define EVA_MODE 2
+#define USEROP 1
+#define KERNELOP 2
/*
* Wrapper to add an entry in the exception table
@@ -103,12 +107,14 @@
* addr : Address
* handler : Exception handler
*/
-#define EXC(insn, type, reg, addr, handler) \
-9: insn reg, addr; \
- .section __ex_table,"a"; \
- PTR 9b, handler; \
- .previous
+#define EXC(insn, type, reg, addr, handler) \
+ .if \mode == LEGACY_MODE; \
+9: insn reg, addr; \
+ .section __ex_table,"a"; \
+ PTR 9b, handler; \
+ .previous; \
+ .endif
/*
* Only on the 64-bit kernel we can made use of 64-bit registers.
*/
@@ -177,7 +183,10 @@
#define LOADB(reg, addr, handler) EXC(lb, LD_INSN, reg, addr, handler)
#define STOREB(reg, addr, handler) EXC(sb, ST_INSN, reg, addr, handler)
-#define _PREF(hint, addr, type) PREF(hint, addr)
+#define _PREF(hint, addr, type) \
+ .if \mode == LEGACY_MODE; \
+ PREF(hint, addr); \
+ .endif
#define PREFS(hint, addr) _PREF(hint, addr, SRC_PREFETCH)
#define PREFD(hint, addr) _PREF(hint, addr, DST_PREFETCH)
@@ -210,27 +219,23 @@
.set at=v1
#endif
-/*
- * t6 is used as a flag to note inatomic mode.
- */
-LEAF(__copy_user_inatomic)
- b __copy_user_common
- li t6, 1
- END(__copy_user_inatomic)
-
-/*
- * A combined memcpy/__copy_user
- * __copy_user sets len to 0 for success; else to an upper bound of
- * the number of uncopied bytes.
- * memcpy sets v0 to dst.
- */
.align 5
-LEAF(memcpy) /* a0=dst a1=src a2=len */
- move v0, dst /* return value */
-.L__memcpy:
-FEXPORT(__copy_user)
- li t6, 0 /* not inatomic */
-__copy_user_common:
+
+ /*
+ * Macro to build the __copy_user common code
+ * Arguements:
+ * mode : LEGACY_MODE or EVA_MODE
+ * from : Source operand. USEROP or KERNELOP
+ * to : Destination operand. USEROP or KERNELOP
+ */
+ .macro __BUILD_COPY_USER mode, from, to
+
+ /* initialize __memcpy if this the first time we execute this macro */
+ .ifnotdef __memcpy
+ .set __memcpy, 1
+ .hidden __memcpy /* make sure it does not leak */
+ .endif
+
/*
* Note: dst & src may be unaligned, len may be 0
* Temps
@@ -251,45 +256,45 @@ __copy_user_common:
and t1, dst, ADDRMASK
PREFS( 0, 1*32(src) )
PREFD( 1, 1*32(dst) )
- bnez t2, .Lcopy_bytes_checklen
+ bnez t2, .Lcopy_bytes_checklen\@
and t0, src, ADDRMASK
PREFS( 0, 2*32(src) )
PREFD( 1, 2*32(dst) )
- bnez t1, .Ldst_unaligned
+ bnez t1, .Ldst_unaligned\@
nop
- bnez t0, .Lsrc_unaligned_dst_aligned
+ bnez t0, .Lsrc_unaligned_dst_aligned\@
/*
* use delay slot for fall-through
* src and dst are aligned; need to compute rem
*/
-.Lboth_aligned:
+.Lboth_aligned\@:
SRL t0, len, LOG_NBYTES+3 # +3 for 8 units/iter
- beqz t0, .Lcleanup_both_aligned # len < 8*NBYTES
+ beqz t0, .Lcleanup_both_aligned\@ # len < 8*NBYTES
and rem, len, (8*NBYTES-1) # rem = len % (8*NBYTES)
PREFS( 0, 3*32(src) )
PREFD( 1, 3*32(dst) )
.align 4
1:
R10KCBARRIER(0(ra))
- LOAD(t0, UNIT(0)(src), .Ll_exc)
- LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
- LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
- LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
+ LOAD(t0, UNIT(0)(src), .Ll_exc\@)
+ LOAD(t1, UNIT(1)(src), .Ll_exc_copy\@)
+ LOAD(t2, UNIT(2)(src), .Ll_exc_copy\@)
+ LOAD(t3, UNIT(3)(src), .Ll_exc_copy\@)
SUB len, len, 8*NBYTES
- LOAD(t4, UNIT(4)(src), .Ll_exc_copy)
- LOAD(t7, UNIT(5)(src), .Ll_exc_copy)
- STORE(t0, UNIT(0)(dst), .Ls_exc_p8u)
- STORE(t1, UNIT(1)(dst), .Ls_exc_p7u)
- LOAD(t0, UNIT(6)(src), .Ll_exc_copy)
- LOAD(t1, UNIT(7)(src), .Ll_exc_copy)
+ LOAD(t4, UNIT(4)(src), .Ll_exc_copy\@)
+ LOAD(t7, UNIT(5)(src), .Ll_exc_copy\@)
+ STORE(t0, UNIT(0)(dst), .Ls_exc_p8u\@)
+ STORE(t1, UNIT(1)(dst), .Ls_exc_p7u\@)
+ LOAD(t0, UNIT(6)(src), .Ll_exc_copy\@)
+ LOAD(t1, UNIT(7)(src), .Ll_exc_copy\@)
ADD src, src, 8*NBYTES
ADD dst, dst, 8*NBYTES
- STORE(t2, UNIT(-6)(dst), .Ls_exc_p6u)
- STORE(t3, UNIT(-5)(dst), .Ls_exc_p5u)
- STORE(t4, UNIT(-4)(dst), .Ls_exc_p4u)
- STORE(t7, UNIT(-3)(dst), .Ls_exc_p3u)
- STORE(t0, UNIT(-2)(dst), .Ls_exc_p2u)
- STORE(t1, UNIT(-1)(dst), .Ls_exc_p1u)
+ STORE(t2, UNIT(-6)(dst), .Ls_exc_p6u\@)
+ STORE(t3, UNIT(-5)(dst), .Ls_exc_p5u\@)
+ STORE(t4, UNIT(-4)(dst), .Ls_exc_p4u\@)
+ STORE(t7, UNIT(-3)(dst), .Ls_exc_p3u\@)
+ STORE(t0, UNIT(-2)(dst), .Ls_exc_p2u\@)
+ STORE(t1, UNIT(-1)(dst), .Ls_exc_p1u\@)
PREFS( 0, 8*32(src) )
PREFD( 1, 8*32(dst) )
bne len, rem, 1b
@@ -298,41 +303,41 @@ __copy_user_common:
/*
* len == rem == the number of bytes left to copy < 8*NBYTES
*/
-.Lcleanup_both_aligned:
- beqz len, .Ldone
+.Lcleanup_both_aligned\@:
+ beqz len, .Ldone\@
sltu t0, len, 4*NBYTES
- bnez t0, .Lless_than_4units
+ bnez t0, .Lless_than_4units\@
and rem, len, (NBYTES-1) # rem = len % NBYTES
/*
* len >= 4*NBYTES
*/
- LOAD( t0, UNIT(0)(src), .Ll_exc)
- LOAD( t1, UNIT(1)(src), .Ll_exc_copy)
- LOAD( t2, UNIT(2)(src), .Ll_exc_copy)
- LOAD( t3, UNIT(3)(src), .Ll_exc_copy)
+ LOAD( t0, UNIT(0)(src), .Ll_exc\@)
+ LOAD( t1, UNIT(1)(src), .Ll_exc_copy\@)
+ LOAD( t2, UNIT(2)(src), .Ll_exc_copy\@)
+ LOAD( t3, UNIT(3)(src), .Ll_exc_copy\@)
SUB len, len, 4*NBYTES
ADD src, src, 4*NBYTES
R10KCBARRIER(0(ra))
- STORE(t0, UNIT(0)(dst), .Ls_exc_p4u)
- STORE(t1, UNIT(1)(dst), .Ls_exc_p3u)
- STORE(t2, UNIT(2)(dst), .Ls_exc_p2u)
- STORE(t3, UNIT(3)(dst), .Ls_exc_p1u)
+ STORE(t0, UNIT(0)(dst), .Ls_exc_p4u\@)
+ STORE(t1, UNIT(1)(dst), .Ls_exc_p3u\@)
+ STORE(t2, UNIT(2)(dst), .Ls_exc_p2u\@)
+ STORE(t3, UNIT(3)(dst), .Ls_exc_p1u\@)
.set reorder /* DADDI_WAR */
ADD dst, dst, 4*NBYTES
- beqz len, .Ldone
+ beqz len, .Ldone\@
.set noreorder
-.Lless_than_4units:
+.Lless_than_4units\@:
/*
* rem = len % NBYTES
*/
- beq rem, len, .Lcopy_bytes
+ beq rem, len, .Lcopy_bytes\@
nop
1:
R10KCBARRIER(0(ra))
- LOAD(t0, 0(src), .Ll_exc)
+ LOAD(t0, 0(src), .Ll_exc\@)
ADD src, src, NBYTES
SUB len, len, NBYTES
- STORE(t0, 0(dst), .Ls_exc_p1u)
+ STORE(t0, 0(dst), .Ls_exc_p1u\@)
.set reorder /* DADDI_WAR */
ADD dst, dst, NBYTES
bne rem, len, 1b
@@ -350,17 +355,17 @@ __copy_user_common:
* more instruction-level parallelism.
*/
#define bits t2
- beqz len, .Ldone
+ beqz len, .Ldone\@
ADD t1, dst, len # t1 is just past last byte of dst
li bits, 8*NBYTES
SLL rem, len, 3 # rem = number of bits to keep
- LOAD(t0, 0(src), .Ll_exc)
+ LOAD(t0, 0(src), .Ll_exc\@)
SUB bits, bits, rem # bits = number of bits to discard
SHIFT_DISCARD t0, t0, bits
- STREST(t0, -1(t1), .Ls_exc)
+ STREST(t0, -1(t1), .Ls_exc\@)
jr ra
move len, zero
-.Ldst_unaligned:
+.Ldst_unaligned\@:
/*
* dst is unaligned
* t0 = src & ADDRMASK
@@ -371,23 +376,23 @@ __copy_user_common:
* Set match = (src and dst have same alignment)
*/
#define match rem
- LDFIRST(t3, FIRST(0)(src), .Ll_exc)
+ LDFIRST(t3, FIRST(0)(src), .Ll_exc\@)
ADD t2, zero, NBYTES
- LDREST(t3, REST(0)(src), .Ll_exc_copy)
+ LDREST(t3, REST(0)(src), .Ll_exc_copy\@)
SUB t2, t2, t1 # t2 = number of bytes copied
xor match, t0, t1
R10KCBARRIER(0(ra))
- STFIRST(t3, FIRST(0)(dst), .Ls_exc)
- beq len, t2, .Ldone
+ STFIRST(t3, FIRST(0)(dst), .Ls_exc\@)
+ beq len, t2, .Ldone\@
SUB len, len, t2
ADD dst, dst, t2
- beqz match, .Lboth_aligned
+ beqz match, .Lboth_aligned\@
ADD src, src, t2
-.Lsrc_unaligned_dst_aligned:
+.Lsrc_unaligned_dst_aligned\@:
SRL t0, len, LOG_NBYTES+2 # +2 for 4 units/iter
PREFS( 0, 3*32(src) )
- beqz t0, .Lcleanup_src_unaligned
+ beqz t0, .Lcleanup_src_unaligned\@
and rem, len, (4*NBYTES-1) # rem = len % 4*NBYTES
PREFD( 1, 3*32(dst) )
1:
@@ -398,58 +403,58 @@ __copy_user_common:
* are to the same unit (unless src is aligned, but it's not).
*/
R10KCBARRIER(0(ra))
- LDFIRST(t0, FIRST(0)(src), .Ll_exc)
- LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy)
+ LDFIRST(t0, FIRST(0)(src), .Ll_exc\@)
+ LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy\@)
SUB len, len, 4*NBYTES
- LDREST(t0, REST(0)(src), .Ll_exc_copy)
- LDREST(t1, REST(1)(src), .Ll_exc_copy)
- LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy)
- LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy)
- LDREST(t2, REST(2)(src), .Ll_exc_copy)
- LDREST(t3, REST(3)(src), .Ll_exc_copy)
+ LDREST(t0, REST(0)(src), .Ll_exc_copy\@)
+ LDREST(t1, REST(1)(src), .Ll_exc_copy\@)
+ LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy\@)
+ LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy\@)
+ LDREST(t2, REST(2)(src), .Ll_exc_copy\@)
+ LDREST(t3, REST(3)(src), .Ll_exc_copy\@)
PREFS( 0, 9*32(src) ) # 0 is PREF_LOAD (not streamed)
ADD src, src, 4*NBYTES
#ifdef CONFIG_CPU_SB1
nop # improves slotting
#endif
- STORE(t0, UNIT(0)(dst), .Ls_exc_p4u)
- STORE(t1, UNIT(1)(dst), .Ls_exc_p3u)
- STORE(t2, UNIT(2)(dst), .Ls_exc_p2u)
- STORE(t3, UNIT(3)(dst), .Ls_exc_p1u)
+ STORE(t0, UNIT(0)(dst), .Ls_exc_p4u\@)
+ STORE(t1, UNIT(1)(dst), .Ls_exc_p3u\@)
+ STORE(t2, UNIT(2)(dst), .Ls_exc_p2u\@)
+ STORE(t3, UNIT(3)(dst), .Ls_exc_p1u\@)
PREFD( 1, 9*32(dst) ) # 1 is PREF_STORE (not streamed)
.set reorder /* DADDI_WAR */
ADD dst, dst, 4*NBYTES
bne len, rem, 1b
.set noreorder
-.Lcleanup_src_unaligned:
- beqz len, .Ldone
+.Lcleanup_src_unaligned\@:
+ beqz len, .Ldone\@
and rem, len, NBYTES-1 # rem = len % NBYTES
- beq rem, len, .Lcopy_bytes
+ beq rem, len, .Lcopy_bytes\@
nop
1:
R10KCBARRIER(0(ra))
- LDFIRST(t0, FIRST(0)(src), .Ll_exc)
- LDREST(t0, REST(0)(src), .Ll_exc_copy)
+ LDFIRST(t0, FIRST(0)(src), .Ll_exc\@)
+ LDREST(t0, REST(0)(src), .Ll_exc_copy\@)
ADD src, src, NBYTES
SUB len, len, NBYTES
- STORE(t0, 0(dst), .Ls_exc_p1u)
+ STORE(t0, 0(dst), .Ls_exc_p1u\@)
.set reorder /* DADDI_WAR */
ADD dst, dst, NBYTES
bne len, rem, 1b
.set noreorder
-.Lcopy_bytes_checklen:
- beqz len, .Ldone
+.Lcopy_bytes_checklen\@:
+ beqz len, .Ldone\@
nop
-.Lcopy_bytes:
+.Lcopy_bytes\@:
/* 0 < len < NBYTES */
R10KCBARRIER(0(ra))
#define COPY_BYTE(N) \
- LOADB(t0, N(src), .Ll_exc); \
+ LOADB(t0, N(src), .Ll_exc\@); \
SUB len, len, 1; \
- beqz len, .Ldone; \
- STOREB(t0, N(dst), .Ls_exc_p1)
+ beqz len, .Ldone\@; \
+ STOREB(t0, N(dst), .Ls_exc_p1\@)
COPY_BYTE(0)
COPY_BYTE(1)
@@ -459,16 +464,19 @@ __copy_user_common:
COPY_BYTE(4)
COPY_BYTE(5)
#endif
- LOADB(t0, NBYTES-2(src), .Ll_exc)
+ LOADB(t0, NBYTES-2(src), .Ll_exc\@)
SUB len, len, 1
jr ra
- STOREB(t0, NBYTES-2(dst), .Ls_exc_p1)
-.Ldone:
+ STOREB(t0, NBYTES-2(dst), .Ls_exc_p1\@)
+.Ldone\@:
jr ra
- nop
+ .if __memcpy == 1
END(memcpy)
+ .set __memcpy, 0
+ .hidden __memcpy
+ .endif
-.Ll_exc_copy:
+.Ll_exc_copy\@:
/*
* Copy bytes from src until faulting load address (or until a
* lb faults)
@@ -483,20 +491,20 @@ __copy_user_common:
nop
LOADK t0, THREAD_BUADDR(t0)
1:
- LOADB(t1, 0(src), .Ll_exc)
+ LOADB(t1, 0(src), .Ll_exc\@)
ADD src, src, 1
sb t1, 0(dst) # can't fault -- we're copy_from_user
.set reorder /* DADDI_WAR */
ADD dst, dst, 1
bne src, t0, 1b
.set noreorder
-.Ll_exc:
+.Ll_exc\@:
LOADK t0, TI_TASK($28)
nop
LOADK t0, THREAD_BUADDR(t0) # t0 is just past last good address
nop
SUB len, AT, t0 # len number of uncopied bytes
- bnez t6, .Ldone /* Skip the zeroing part if inatomic */
+ bnez t6, .Ldone\@ /* Skip the zeroing part if inatomic */
/*
* Here's where we rely on src and dst being incremented in tandem,
* See (3) above.
@@ -510,7 +518,7 @@ __copy_user_common:
*/
.set reorder /* DADDI_WAR */
SUB src, len, 1
- beqz len, .Ldone
+ beqz len, .Ldone\@
.set noreorder
1: sb zero, 0(dst)
ADD dst, dst, 1
@@ -531,7 +539,7 @@ __copy_user_common:
#define SEXC(n) \
.set reorder; /* DADDI_WAR */ \
-.Ls_exc_p ## n ## u: \
+.Ls_exc_p ## n ## u\@: \
ADD len, len, n*NBYTES; \
jr ra; \
.set noreorder
@@ -545,14 +553,15 @@ SEXC(3)
SEXC(2)
SEXC(1)
-.Ls_exc_p1:
+.Ls_exc_p1\@:
.set reorder /* DADDI_WAR */
ADD len, len, 1
jr ra
.set noreorder
-.Ls_exc:
+.Ls_exc\@:
jr ra
nop
+ .endm
.align 5
LEAF(memmove)
@@ -603,3 +612,27 @@ LEAF(__rmemcpy) /* a0=dst a1=src a2=len */
jr ra
move a2, zero
END(__rmemcpy)
+
+/*
+ * t6 is used as a flag to note inatomic mode.
+ */
+LEAF(__copy_user_inatomic)
+ b __copy_user_common
+ li t6, 1
+ END(__copy_user_inatomic)
+
+/*
+ * A combined memcpy/__copy_user
+ * __copy_user sets len to 0 for success; else to an upper bound of
+ * the number of uncopied bytes.
+ * memcpy sets v0 to dst.
+ */
+ .align 5
+LEAF(memcpy) /* a0=dst a1=src a2=len */
+ move v0, dst /* return value */
+.L__memcpy:
+FEXPORT(__copy_user)
+ li t6, 0 /* not inatomic */
+__copy_user_common:
+ /* Legacy Mode, user <-> user */
+ __BUILD_COPY_USER LEGACY_MODE USEROP USEROP
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 18/58] MIPS: lib: memcpy: Use macro to build the copy_user code
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
The code can be shared between EVA and non-EVA configurations,
therefore use a macro to build it to avoid code duplications.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/memcpy.S | 253 ++++++++++++++++++++++++++++---------------------
1 file changed, 143 insertions(+), 110 deletions(-)
diff --git a/arch/mips/lib/memcpy.S b/arch/mips/lib/memcpy.S
index eed6e07..d630a28 100644
--- a/arch/mips/lib/memcpy.S
+++ b/arch/mips/lib/memcpy.S
@@ -92,6 +92,10 @@
/* Pretech type */
#define SRC_PREFETCH 1
#define DST_PREFETCH 2
+#define LEGACY_MODE 1
+#define EVA_MODE 2
+#define USEROP 1
+#define KERNELOP 2
/*
* Wrapper to add an entry in the exception table
@@ -103,12 +107,14 @@
* addr : Address
* handler : Exception handler
*/
-#define EXC(insn, type, reg, addr, handler) \
-9: insn reg, addr; \
- .section __ex_table,"a"; \
- PTR 9b, handler; \
- .previous
+#define EXC(insn, type, reg, addr, handler) \
+ .if \mode == LEGACY_MODE; \
+9: insn reg, addr; \
+ .section __ex_table,"a"; \
+ PTR 9b, handler; \
+ .previous; \
+ .endif
/*
* Only on the 64-bit kernel we can made use of 64-bit registers.
*/
@@ -177,7 +183,10 @@
#define LOADB(reg, addr, handler) EXC(lb, LD_INSN, reg, addr, handler)
#define STOREB(reg, addr, handler) EXC(sb, ST_INSN, reg, addr, handler)
-#define _PREF(hint, addr, type) PREF(hint, addr)
+#define _PREF(hint, addr, type) \
+ .if \mode == LEGACY_MODE; \
+ PREF(hint, addr); \
+ .endif
#define PREFS(hint, addr) _PREF(hint, addr, SRC_PREFETCH)
#define PREFD(hint, addr) _PREF(hint, addr, DST_PREFETCH)
@@ -210,27 +219,23 @@
.set at=v1
#endif
-/*
- * t6 is used as a flag to note inatomic mode.
- */
-LEAF(__copy_user_inatomic)
- b __copy_user_common
- li t6, 1
- END(__copy_user_inatomic)
-
-/*
- * A combined memcpy/__copy_user
- * __copy_user sets len to 0 for success; else to an upper bound of
- * the number of uncopied bytes.
- * memcpy sets v0 to dst.
- */
.align 5
-LEAF(memcpy) /* a0=dst a1=src a2=len */
- move v0, dst /* return value */
-.L__memcpy:
-FEXPORT(__copy_user)
- li t6, 0 /* not inatomic */
-__copy_user_common:
+
+ /*
+ * Macro to build the __copy_user common code
+ * Arguements:
+ * mode : LEGACY_MODE or EVA_MODE
+ * from : Source operand. USEROP or KERNELOP
+ * to : Destination operand. USEROP or KERNELOP
+ */
+ .macro __BUILD_COPY_USER mode, from, to
+
+ /* initialize __memcpy if this the first time we execute this macro */
+ .ifnotdef __memcpy
+ .set __memcpy, 1
+ .hidden __memcpy /* make sure it does not leak */
+ .endif
+
/*
* Note: dst & src may be unaligned, len may be 0
* Temps
@@ -251,45 +256,45 @@ __copy_user_common:
and t1, dst, ADDRMASK
PREFS( 0, 1*32(src) )
PREFD( 1, 1*32(dst) )
- bnez t2, .Lcopy_bytes_checklen
+ bnez t2, .Lcopy_bytes_checklen\@
and t0, src, ADDRMASK
PREFS( 0, 2*32(src) )
PREFD( 1, 2*32(dst) )
- bnez t1, .Ldst_unaligned
+ bnez t1, .Ldst_unaligned\@
nop
- bnez t0, .Lsrc_unaligned_dst_aligned
+ bnez t0, .Lsrc_unaligned_dst_aligned\@
/*
* use delay slot for fall-through
* src and dst are aligned; need to compute rem
*/
-.Lboth_aligned:
+.Lboth_aligned\@:
SRL t0, len, LOG_NBYTES+3 # +3 for 8 units/iter
- beqz t0, .Lcleanup_both_aligned # len < 8*NBYTES
+ beqz t0, .Lcleanup_both_aligned\@ # len < 8*NBYTES
and rem, len, (8*NBYTES-1) # rem = len % (8*NBYTES)
PREFS( 0, 3*32(src) )
PREFD( 1, 3*32(dst) )
.align 4
1:
R10KCBARRIER(0(ra))
- LOAD(t0, UNIT(0)(src), .Ll_exc)
- LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
- LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
- LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
+ LOAD(t0, UNIT(0)(src), .Ll_exc\@)
+ LOAD(t1, UNIT(1)(src), .Ll_exc_copy\@)
+ LOAD(t2, UNIT(2)(src), .Ll_exc_copy\@)
+ LOAD(t3, UNIT(3)(src), .Ll_exc_copy\@)
SUB len, len, 8*NBYTES
- LOAD(t4, UNIT(4)(src), .Ll_exc_copy)
- LOAD(t7, UNIT(5)(src), .Ll_exc_copy)
- STORE(t0, UNIT(0)(dst), .Ls_exc_p8u)
- STORE(t1, UNIT(1)(dst), .Ls_exc_p7u)
- LOAD(t0, UNIT(6)(src), .Ll_exc_copy)
- LOAD(t1, UNIT(7)(src), .Ll_exc_copy)
+ LOAD(t4, UNIT(4)(src), .Ll_exc_copy\@)
+ LOAD(t7, UNIT(5)(src), .Ll_exc_copy\@)
+ STORE(t0, UNIT(0)(dst), .Ls_exc_p8u\@)
+ STORE(t1, UNIT(1)(dst), .Ls_exc_p7u\@)
+ LOAD(t0, UNIT(6)(src), .Ll_exc_copy\@)
+ LOAD(t1, UNIT(7)(src), .Ll_exc_copy\@)
ADD src, src, 8*NBYTES
ADD dst, dst, 8*NBYTES
- STORE(t2, UNIT(-6)(dst), .Ls_exc_p6u)
- STORE(t3, UNIT(-5)(dst), .Ls_exc_p5u)
- STORE(t4, UNIT(-4)(dst), .Ls_exc_p4u)
- STORE(t7, UNIT(-3)(dst), .Ls_exc_p3u)
- STORE(t0, UNIT(-2)(dst), .Ls_exc_p2u)
- STORE(t1, UNIT(-1)(dst), .Ls_exc_p1u)
+ STORE(t2, UNIT(-6)(dst), .Ls_exc_p6u\@)
+ STORE(t3, UNIT(-5)(dst), .Ls_exc_p5u\@)
+ STORE(t4, UNIT(-4)(dst), .Ls_exc_p4u\@)
+ STORE(t7, UNIT(-3)(dst), .Ls_exc_p3u\@)
+ STORE(t0, UNIT(-2)(dst), .Ls_exc_p2u\@)
+ STORE(t1, UNIT(-1)(dst), .Ls_exc_p1u\@)
PREFS( 0, 8*32(src) )
PREFD( 1, 8*32(dst) )
bne len, rem, 1b
@@ -298,41 +303,41 @@ __copy_user_common:
/*
* len == rem == the number of bytes left to copy < 8*NBYTES
*/
-.Lcleanup_both_aligned:
- beqz len, .Ldone
+.Lcleanup_both_aligned\@:
+ beqz len, .Ldone\@
sltu t0, len, 4*NBYTES
- bnez t0, .Lless_than_4units
+ bnez t0, .Lless_than_4units\@
and rem, len, (NBYTES-1) # rem = len % NBYTES
/*
* len >= 4*NBYTES
*/
- LOAD( t0, UNIT(0)(src), .Ll_exc)
- LOAD( t1, UNIT(1)(src), .Ll_exc_copy)
- LOAD( t2, UNIT(2)(src), .Ll_exc_copy)
- LOAD( t3, UNIT(3)(src), .Ll_exc_copy)
+ LOAD( t0, UNIT(0)(src), .Ll_exc\@)
+ LOAD( t1, UNIT(1)(src), .Ll_exc_copy\@)
+ LOAD( t2, UNIT(2)(src), .Ll_exc_copy\@)
+ LOAD( t3, UNIT(3)(src), .Ll_exc_copy\@)
SUB len, len, 4*NBYTES
ADD src, src, 4*NBYTES
R10KCBARRIER(0(ra))
- STORE(t0, UNIT(0)(dst), .Ls_exc_p4u)
- STORE(t1, UNIT(1)(dst), .Ls_exc_p3u)
- STORE(t2, UNIT(2)(dst), .Ls_exc_p2u)
- STORE(t3, UNIT(3)(dst), .Ls_exc_p1u)
+ STORE(t0, UNIT(0)(dst), .Ls_exc_p4u\@)
+ STORE(t1, UNIT(1)(dst), .Ls_exc_p3u\@)
+ STORE(t2, UNIT(2)(dst), .Ls_exc_p2u\@)
+ STORE(t3, UNIT(3)(dst), .Ls_exc_p1u\@)
.set reorder /* DADDI_WAR */
ADD dst, dst, 4*NBYTES
- beqz len, .Ldone
+ beqz len, .Ldone\@
.set noreorder
-.Lless_than_4units:
+.Lless_than_4units\@:
/*
* rem = len % NBYTES
*/
- beq rem, len, .Lcopy_bytes
+ beq rem, len, .Lcopy_bytes\@
nop
1:
R10KCBARRIER(0(ra))
- LOAD(t0, 0(src), .Ll_exc)
+ LOAD(t0, 0(src), .Ll_exc\@)
ADD src, src, NBYTES
SUB len, len, NBYTES
- STORE(t0, 0(dst), .Ls_exc_p1u)
+ STORE(t0, 0(dst), .Ls_exc_p1u\@)
.set reorder /* DADDI_WAR */
ADD dst, dst, NBYTES
bne rem, len, 1b
@@ -350,17 +355,17 @@ __copy_user_common:
* more instruction-level parallelism.
*/
#define bits t2
- beqz len, .Ldone
+ beqz len, .Ldone\@
ADD t1, dst, len # t1 is just past last byte of dst
li bits, 8*NBYTES
SLL rem, len, 3 # rem = number of bits to keep
- LOAD(t0, 0(src), .Ll_exc)
+ LOAD(t0, 0(src), .Ll_exc\@)
SUB bits, bits, rem # bits = number of bits to discard
SHIFT_DISCARD t0, t0, bits
- STREST(t0, -1(t1), .Ls_exc)
+ STREST(t0, -1(t1), .Ls_exc\@)
jr ra
move len, zero
-.Ldst_unaligned:
+.Ldst_unaligned\@:
/*
* dst is unaligned
* t0 = src & ADDRMASK
@@ -371,23 +376,23 @@ __copy_user_common:
* Set match = (src and dst have same alignment)
*/
#define match rem
- LDFIRST(t3, FIRST(0)(src), .Ll_exc)
+ LDFIRST(t3, FIRST(0)(src), .Ll_exc\@)
ADD t2, zero, NBYTES
- LDREST(t3, REST(0)(src), .Ll_exc_copy)
+ LDREST(t3, REST(0)(src), .Ll_exc_copy\@)
SUB t2, t2, t1 # t2 = number of bytes copied
xor match, t0, t1
R10KCBARRIER(0(ra))
- STFIRST(t3, FIRST(0)(dst), .Ls_exc)
- beq len, t2, .Ldone
+ STFIRST(t3, FIRST(0)(dst), .Ls_exc\@)
+ beq len, t2, .Ldone\@
SUB len, len, t2
ADD dst, dst, t2
- beqz match, .Lboth_aligned
+ beqz match, .Lboth_aligned\@
ADD src, src, t2
-.Lsrc_unaligned_dst_aligned:
+.Lsrc_unaligned_dst_aligned\@:
SRL t0, len, LOG_NBYTES+2 # +2 for 4 units/iter
PREFS( 0, 3*32(src) )
- beqz t0, .Lcleanup_src_unaligned
+ beqz t0, .Lcleanup_src_unaligned\@
and rem, len, (4*NBYTES-1) # rem = len % 4*NBYTES
PREFD( 1, 3*32(dst) )
1:
@@ -398,58 +403,58 @@ __copy_user_common:
* are to the same unit (unless src is aligned, but it's not).
*/
R10KCBARRIER(0(ra))
- LDFIRST(t0, FIRST(0)(src), .Ll_exc)
- LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy)
+ LDFIRST(t0, FIRST(0)(src), .Ll_exc\@)
+ LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy\@)
SUB len, len, 4*NBYTES
- LDREST(t0, REST(0)(src), .Ll_exc_copy)
- LDREST(t1, REST(1)(src), .Ll_exc_copy)
- LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy)
- LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy)
- LDREST(t2, REST(2)(src), .Ll_exc_copy)
- LDREST(t3, REST(3)(src), .Ll_exc_copy)
+ LDREST(t0, REST(0)(src), .Ll_exc_copy\@)
+ LDREST(t1, REST(1)(src), .Ll_exc_copy\@)
+ LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy\@)
+ LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy\@)
+ LDREST(t2, REST(2)(src), .Ll_exc_copy\@)
+ LDREST(t3, REST(3)(src), .Ll_exc_copy\@)
PREFS( 0, 9*32(src) ) # 0 is PREF_LOAD (not streamed)
ADD src, src, 4*NBYTES
#ifdef CONFIG_CPU_SB1
nop # improves slotting
#endif
- STORE(t0, UNIT(0)(dst), .Ls_exc_p4u)
- STORE(t1, UNIT(1)(dst), .Ls_exc_p3u)
- STORE(t2, UNIT(2)(dst), .Ls_exc_p2u)
- STORE(t3, UNIT(3)(dst), .Ls_exc_p1u)
+ STORE(t0, UNIT(0)(dst), .Ls_exc_p4u\@)
+ STORE(t1, UNIT(1)(dst), .Ls_exc_p3u\@)
+ STORE(t2, UNIT(2)(dst), .Ls_exc_p2u\@)
+ STORE(t3, UNIT(3)(dst), .Ls_exc_p1u\@)
PREFD( 1, 9*32(dst) ) # 1 is PREF_STORE (not streamed)
.set reorder /* DADDI_WAR */
ADD dst, dst, 4*NBYTES
bne len, rem, 1b
.set noreorder
-.Lcleanup_src_unaligned:
- beqz len, .Ldone
+.Lcleanup_src_unaligned\@:
+ beqz len, .Ldone\@
and rem, len, NBYTES-1 # rem = len % NBYTES
- beq rem, len, .Lcopy_bytes
+ beq rem, len, .Lcopy_bytes\@
nop
1:
R10KCBARRIER(0(ra))
- LDFIRST(t0, FIRST(0)(src), .Ll_exc)
- LDREST(t0, REST(0)(src), .Ll_exc_copy)
+ LDFIRST(t0, FIRST(0)(src), .Ll_exc\@)
+ LDREST(t0, REST(0)(src), .Ll_exc_copy\@)
ADD src, src, NBYTES
SUB len, len, NBYTES
- STORE(t0, 0(dst), .Ls_exc_p1u)
+ STORE(t0, 0(dst), .Ls_exc_p1u\@)
.set reorder /* DADDI_WAR */
ADD dst, dst, NBYTES
bne len, rem, 1b
.set noreorder
-.Lcopy_bytes_checklen:
- beqz len, .Ldone
+.Lcopy_bytes_checklen\@:
+ beqz len, .Ldone\@
nop
-.Lcopy_bytes:
+.Lcopy_bytes\@:
/* 0 < len < NBYTES */
R10KCBARRIER(0(ra))
#define COPY_BYTE(N) \
- LOADB(t0, N(src), .Ll_exc); \
+ LOADB(t0, N(src), .Ll_exc\@); \
SUB len, len, 1; \
- beqz len, .Ldone; \
- STOREB(t0, N(dst), .Ls_exc_p1)
+ beqz len, .Ldone\@; \
+ STOREB(t0, N(dst), .Ls_exc_p1\@)
COPY_BYTE(0)
COPY_BYTE(1)
@@ -459,16 +464,19 @@ __copy_user_common:
COPY_BYTE(4)
COPY_BYTE(5)
#endif
- LOADB(t0, NBYTES-2(src), .Ll_exc)
+ LOADB(t0, NBYTES-2(src), .Ll_exc\@)
SUB len, len, 1
jr ra
- STOREB(t0, NBYTES-2(dst), .Ls_exc_p1)
-.Ldone:
+ STOREB(t0, NBYTES-2(dst), .Ls_exc_p1\@)
+.Ldone\@:
jr ra
- nop
+ .if __memcpy == 1
END(memcpy)
+ .set __memcpy, 0
+ .hidden __memcpy
+ .endif
-.Ll_exc_copy:
+.Ll_exc_copy\@:
/*
* Copy bytes from src until faulting load address (or until a
* lb faults)
@@ -483,20 +491,20 @@ __copy_user_common:
nop
LOADK t0, THREAD_BUADDR(t0)
1:
- LOADB(t1, 0(src), .Ll_exc)
+ LOADB(t1, 0(src), .Ll_exc\@)
ADD src, src, 1
sb t1, 0(dst) # can't fault -- we're copy_from_user
.set reorder /* DADDI_WAR */
ADD dst, dst, 1
bne src, t0, 1b
.set noreorder
-.Ll_exc:
+.Ll_exc\@:
LOADK t0, TI_TASK($28)
nop
LOADK t0, THREAD_BUADDR(t0) # t0 is just past last good address
nop
SUB len, AT, t0 # len number of uncopied bytes
- bnez t6, .Ldone /* Skip the zeroing part if inatomic */
+ bnez t6, .Ldone\@ /* Skip the zeroing part if inatomic */
/*
* Here's where we rely on src and dst being incremented in tandem,
* See (3) above.
@@ -510,7 +518,7 @@ __copy_user_common:
*/
.set reorder /* DADDI_WAR */
SUB src, len, 1
- beqz len, .Ldone
+ beqz len, .Ldone\@
.set noreorder
1: sb zero, 0(dst)
ADD dst, dst, 1
@@ -531,7 +539,7 @@ __copy_user_common:
#define SEXC(n) \
.set reorder; /* DADDI_WAR */ \
-.Ls_exc_p ## n ## u: \
+.Ls_exc_p ## n ## u\@: \
ADD len, len, n*NBYTES; \
jr ra; \
.set noreorder
@@ -545,14 +553,15 @@ SEXC(3)
SEXC(2)
SEXC(1)
-.Ls_exc_p1:
+.Ls_exc_p1\@:
.set reorder /* DADDI_WAR */
ADD len, len, 1
jr ra
.set noreorder
-.Ls_exc:
+.Ls_exc\@:
jr ra
nop
+ .endm
.align 5
LEAF(memmove)
@@ -603,3 +612,27 @@ LEAF(__rmemcpy) /* a0=dst a1=src a2=len */
jr ra
move a2, zero
END(__rmemcpy)
+
+/*
+ * t6 is used as a flag to note inatomic mode.
+ */
+LEAF(__copy_user_inatomic)
+ b __copy_user_common
+ li t6, 1
+ END(__copy_user_inatomic)
+
+/*
+ * A combined memcpy/__copy_user
+ * __copy_user sets len to 0 for success; else to an upper bound of
+ * the number of uncopied bytes.
+ * memcpy sets v0 to dst.
+ */
+ .align 5
+LEAF(memcpy) /* a0=dst a1=src a2=len */
+ move v0, dst /* return value */
+.L__memcpy:
+FEXPORT(__copy_user)
+ li t6, 0 /* not inatomic */
+__copy_user_common:
+ /* Legacy Mode, user <-> user */
+ __BUILD_COPY_USER LEGACY_MODE USEROP USEROP
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 19/58] MIPS: lib: memcpy: Add EVA support
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Add copy_{to,from,in}_user when the CPU operates in EVA mode.
This is necessary so the EVA specific instructions can be used
to perform the virtual to physical translation for user space
addresses. We will use the non-EVA functions to read from kernel
if needed.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/mips_ksyms.c | 6 ++++
arch/mips/lib/memcpy.S | 77 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 83 insertions(+)
diff --git a/arch/mips/kernel/mips_ksyms.c b/arch/mips/kernel/mips_ksyms.c
index 675bd05..13e60f6 100644
--- a/arch/mips/kernel/mips_ksyms.c
+++ b/arch/mips/kernel/mips_ksyms.c
@@ -51,6 +51,12 @@ EXPORT_SYMBOL(copy_page);
*/
EXPORT_SYMBOL(__copy_user);
EXPORT_SYMBOL(__copy_user_inatomic);
+#ifdef CONFIG_EVA
+EXPORT_SYMBOL(__copy_from_user_eva);
+EXPORT_SYMBOL(__copy_in_user_eva);
+EXPORT_SYMBOL(__copy_to_user_eva);
+EXPORT_SYMBOL(__copy_user_inatomic_eva);
+#endif
EXPORT_SYMBOL(__bzero);
EXPORT_SYMBOL(__strncpy_from_kernel_nocheck_asm);
EXPORT_SYMBOL(__strncpy_from_kernel_asm);
diff --git a/arch/mips/lib/memcpy.S b/arch/mips/lib/memcpy.S
index d630a28..c17ef80 100644
--- a/arch/mips/lib/memcpy.S
+++ b/arch/mips/lib/memcpy.S
@@ -114,7 +114,24 @@
.section __ex_table,"a"; \
PTR 9b, handler; \
.previous; \
+ /* This is assembled in EVA mode */ \
+ .else; \
+ /* If loading from user or storing to user */ \
+ .if ((\from == USEROP) && (type == LD_INSN)) || \
+ ((\to == USEROP) && (type == ST_INSN)); \
+9: __BUILD_EVA_INSN(insn##e, reg, addr); \
+ .section __ex_table,"a"; \
+ PTR 9b, handler; \
+ .previous; \
+ .else; \
+ /* \
+ * Still in EVA, but no need for \
+ * exception handler or EVA insn \
+ */ \
+ insn reg, addr; \
+ .endif; \
.endif
+
/*
* Only on the 64-bit kernel we can made use of 64-bit registers.
*/
@@ -186,6 +203,22 @@
#define _PREF(hint, addr, type) \
.if \mode == LEGACY_MODE; \
PREF(hint, addr); \
+ .else; \
+ .if ((\from == USEROP) && (type == SRC_PREFETCH)) || \
+ ((\to == USEROP) && (type == DST_PREFETCH)); \
+ /* \
+ * PREFE has only 9 bits for the offset \
+ * compared to PREF which has 16, so it may \
+ * need to use the $at register but this \
+ * register should remain intact because it's \
+ * used later on. Therefore use $v1. \
+ */ \
+ .set at=v1; \
+ PREFE(hint, addr); \
+ .set noat; \
+ .else; \
+ PREF(hint, addr); \
+ .endif; \
.endif
#define PREFS(hint, addr) _PREF(hint, addr, SRC_PREFETCH)
@@ -636,3 +669,47 @@ FEXPORT(__copy_user)
__copy_user_common:
/* Legacy Mode, user <-> user */
__BUILD_COPY_USER LEGACY_MODE USEROP USEROP
+
+#ifdef CONFIG_EVA
+
+/*
+ * For EVA we need distinct symbols for reading and writing to user space.
+ * This is because we need to use specific EVA instructions to perform the
+ * virtual <-> physical translation when a virtual address is actually in user
+ * space
+ */
+
+LEAF(__copy_user_inatomic_eva)
+ b __copy_from_user_common
+ li t6, 1
+ END(__copy_user_inatomic_eva)
+
+/*
+ * __copy_from_user (EVA)
+ */
+
+LEAF(__copy_from_user_eva)
+ li t6, 0 /* not inatomic */
+__copy_from_user_common:
+ __BUILD_COPY_USER EVA_MODE USEROP KERNELOP
+END(__copy_from_user_eva)
+
+
+
+/*
+ * __copy_to_user (EVA)
+ */
+
+LEAF(__copy_to_user_eva)
+__BUILD_COPY_USER EVA_MODE KERNELOP USEROP
+END(__copy_to_user_eva)
+
+/*
+ * __copy_in_user (EVA)
+ */
+
+LEAF(__copy_in_user_eva)
+__BUILD_COPY_USER EVA_MODE USEROP USEROP
+END(__copy_in_user_eva)
+
+#endif
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 19/58] MIPS: lib: memcpy: Add EVA support
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Add copy_{to,from,in}_user when the CPU operates in EVA mode.
This is necessary so the EVA specific instructions can be used
to perform the virtual to physical translation for user space
addresses. We will use the non-EVA functions to read from kernel
if needed.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/mips_ksyms.c | 6 ++++
arch/mips/lib/memcpy.S | 77 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 83 insertions(+)
diff --git a/arch/mips/kernel/mips_ksyms.c b/arch/mips/kernel/mips_ksyms.c
index 675bd05..13e60f6 100644
--- a/arch/mips/kernel/mips_ksyms.c
+++ b/arch/mips/kernel/mips_ksyms.c
@@ -51,6 +51,12 @@ EXPORT_SYMBOL(copy_page);
*/
EXPORT_SYMBOL(__copy_user);
EXPORT_SYMBOL(__copy_user_inatomic);
+#ifdef CONFIG_EVA
+EXPORT_SYMBOL(__copy_from_user_eva);
+EXPORT_SYMBOL(__copy_in_user_eva);
+EXPORT_SYMBOL(__copy_to_user_eva);
+EXPORT_SYMBOL(__copy_user_inatomic_eva);
+#endif
EXPORT_SYMBOL(__bzero);
EXPORT_SYMBOL(__strncpy_from_kernel_nocheck_asm);
EXPORT_SYMBOL(__strncpy_from_kernel_asm);
diff --git a/arch/mips/lib/memcpy.S b/arch/mips/lib/memcpy.S
index d630a28..c17ef80 100644
--- a/arch/mips/lib/memcpy.S
+++ b/arch/mips/lib/memcpy.S
@@ -114,7 +114,24 @@
.section __ex_table,"a"; \
PTR 9b, handler; \
.previous; \
+ /* This is assembled in EVA mode */ \
+ .else; \
+ /* If loading from user or storing to user */ \
+ .if ((\from == USEROP) && (type == LD_INSN)) || \
+ ((\to == USEROP) && (type == ST_INSN)); \
+9: __BUILD_EVA_INSN(insn##e, reg, addr); \
+ .section __ex_table,"a"; \
+ PTR 9b, handler; \
+ .previous; \
+ .else; \
+ /* \
+ * Still in EVA, but no need for \
+ * exception handler or EVA insn \
+ */ \
+ insn reg, addr; \
+ .endif; \
.endif
+
/*
* Only on the 64-bit kernel we can made use of 64-bit registers.
*/
@@ -186,6 +203,22 @@
#define _PREF(hint, addr, type) \
.if \mode == LEGACY_MODE; \
PREF(hint, addr); \
+ .else; \
+ .if ((\from == USEROP) && (type == SRC_PREFETCH)) || \
+ ((\to == USEROP) && (type == DST_PREFETCH)); \
+ /* \
+ * PREFE has only 9 bits for the offset \
+ * compared to PREF which has 16, so it may \
+ * need to use the $at register but this \
+ * register should remain intact because it's \
+ * used later on. Therefore use $v1. \
+ */ \
+ .set at=v1; \
+ PREFE(hint, addr); \
+ .set noat; \
+ .else; \
+ PREF(hint, addr); \
+ .endif; \
.endif
#define PREFS(hint, addr) _PREF(hint, addr, SRC_PREFETCH)
@@ -636,3 +669,47 @@ FEXPORT(__copy_user)
__copy_user_common:
/* Legacy Mode, user <-> user */
__BUILD_COPY_USER LEGACY_MODE USEROP USEROP
+
+#ifdef CONFIG_EVA
+
+/*
+ * For EVA we need distinct symbols for reading and writing to user space.
+ * This is because we need to use specific EVA instructions to perform the
+ * virtual <-> physical translation when a virtual address is actually in user
+ * space
+ */
+
+LEAF(__copy_user_inatomic_eva)
+ b __copy_from_user_common
+ li t6, 1
+ END(__copy_user_inatomic_eva)
+
+/*
+ * __copy_from_user (EVA)
+ */
+
+LEAF(__copy_from_user_eva)
+ li t6, 0 /* not inatomic */
+__copy_from_user_common:
+ __BUILD_COPY_USER EVA_MODE USEROP KERNELOP
+END(__copy_from_user_eva)
+
+
+
+/*
+ * __copy_to_user (EVA)
+ */
+
+LEAF(__copy_to_user_eva)
+__BUILD_COPY_USER EVA_MODE KERNELOP USEROP
+END(__copy_to_user_eva)
+
+/*
+ * __copy_in_user (EVA)
+ */
+
+LEAF(__copy_in_user_eva)
+__BUILD_COPY_USER EVA_MODE USEROP USEROP
+END(__copy_in_user_eva)
+
+#endif
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 20/58] MIPS: lib: memset: Whitespace fixes
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/memset.S | 30 +++++++++++++++---------------
1 file changed, 15 insertions(+), 15 deletions(-)
diff --git a/arch/mips/lib/memset.S b/arch/mips/lib/memset.S
index 0580194..d857985 100644
--- a/arch/mips/lib/memset.S
+++ b/arch/mips/lib/memset.S
@@ -74,7 +74,7 @@
.align 5
LEAF(memset)
beqz a1, 1f
- move v0, a0 /* result */
+ move v0, a0 /* result */
andi a1, 0xff /* spread fillword */
LONG_SLL t1, a1, 8
@@ -90,7 +90,7 @@ LEAF(memset)
FEXPORT(__bzero)
sltiu t0, a2, STORSIZE /* very small region? */
bnez t0, .Lsmall_memset
- andi t0, a0, STORMASK /* aligned? */
+ andi t0, a0, STORMASK /* aligned? */
#ifdef CONFIG_CPU_MICROMIPS
move t8, a1 /* used by 'swp' instruction */
@@ -98,12 +98,12 @@ FEXPORT(__bzero)
#endif
#ifndef CONFIG_CPU_DADDI_WORKAROUNDS
beqz t0, 1f
- PTR_SUBU t0, STORSIZE /* alignment in bytes */
+ PTR_SUBU t0, STORSIZE /* alignment in bytes */
#else
.set noat
li AT, STORSIZE
beqz t0, 1f
- PTR_SUBU t0, AT /* alignment in bytes */
+ PTR_SUBU t0, AT /* alignment in bytes */
.set at
#endif
@@ -120,7 +120,7 @@ FEXPORT(__bzero)
1: ori t1, a2, 0x3f /* # of full blocks */
xori t1, 0x3f
beqz t1, .Lmemset_partial /* no block to fill */
- andi t0, a2, 0x40-STORSIZE
+ andi t0, a2, 0x40-STORSIZE
PTR_ADDU t1, a0 /* end address */
.set reorder
@@ -145,7 +145,7 @@ FEXPORT(__bzero)
.set at
#endif
jr t1
- PTR_ADDU a0, t0 /* dest ptr */
+ PTR_ADDU a0, t0 /* dest ptr */
.set push
.set noreorder
@@ -155,7 +155,7 @@ FEXPORT(__bzero)
andi a2, STORMASK /* At most one long to go */
beqz a2, 1f
- PTR_ADDU a0, a2 /* What's left */
+ PTR_ADDU a0, a2 /* What's left */
R10KCBARRIER(0(ra))
#ifdef __MIPSEB__
EX(LONG_S_R, a1, -1(a0), .Llast_fixup)
@@ -164,24 +164,24 @@ FEXPORT(__bzero)
EX(LONG_S_L, a1, -1(a0), .Llast_fixup)
#endif
1: jr ra
- move a2, zero
+ move a2, zero
.Lsmall_memset:
beqz a2, 2f
- PTR_ADDU t1, a0, a2
+ PTR_ADDU t1, a0, a2
1: PTR_ADDIU a0, 1 /* fill bytewise */
R10KCBARRIER(0(ra))
bne t1, a0, 1b
- sb a1, -1(a0)
+ sb a1, -1(a0)
2: jr ra /* done */
- move a2, zero
+ move a2, zero
END(memset)
.Lfirst_fixup:
jr ra
- nop
+ nop
.Lfwd_fixup:
PTR_L t0, TI_TASK($28)
@@ -189,7 +189,7 @@ FEXPORT(__bzero)
LONG_L t0, THREAD_BUADDR(t0)
LONG_ADDU a2, t1
jr ra
- LONG_SUBU a2, t0
+ LONG_SUBU a2, t0
.Lpartial_fixup:
PTR_L t0, TI_TASK($28)
@@ -197,8 +197,8 @@ FEXPORT(__bzero)
LONG_L t0, THREAD_BUADDR(t0)
LONG_ADDU a2, t1
jr ra
- LONG_SUBU a2, t0
+ LONG_SUBU a2, t0
.Llast_fixup:
jr ra
- andi v1, a2, STORMASK
+ andi v1, a2, STORMASK
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 20/58] MIPS: lib: memset: Whitespace fixes
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/memset.S | 30 +++++++++++++++---------------
1 file changed, 15 insertions(+), 15 deletions(-)
diff --git a/arch/mips/lib/memset.S b/arch/mips/lib/memset.S
index 0580194..d857985 100644
--- a/arch/mips/lib/memset.S
+++ b/arch/mips/lib/memset.S
@@ -74,7 +74,7 @@
.align 5
LEAF(memset)
beqz a1, 1f
- move v0, a0 /* result */
+ move v0, a0 /* result */
andi a1, 0xff /* spread fillword */
LONG_SLL t1, a1, 8
@@ -90,7 +90,7 @@ LEAF(memset)
FEXPORT(__bzero)
sltiu t0, a2, STORSIZE /* very small region? */
bnez t0, .Lsmall_memset
- andi t0, a0, STORMASK /* aligned? */
+ andi t0, a0, STORMASK /* aligned? */
#ifdef CONFIG_CPU_MICROMIPS
move t8, a1 /* used by 'swp' instruction */
@@ -98,12 +98,12 @@ FEXPORT(__bzero)
#endif
#ifndef CONFIG_CPU_DADDI_WORKAROUNDS
beqz t0, 1f
- PTR_SUBU t0, STORSIZE /* alignment in bytes */
+ PTR_SUBU t0, STORSIZE /* alignment in bytes */
#else
.set noat
li AT, STORSIZE
beqz t0, 1f
- PTR_SUBU t0, AT /* alignment in bytes */
+ PTR_SUBU t0, AT /* alignment in bytes */
.set at
#endif
@@ -120,7 +120,7 @@ FEXPORT(__bzero)
1: ori t1, a2, 0x3f /* # of full blocks */
xori t1, 0x3f
beqz t1, .Lmemset_partial /* no block to fill */
- andi t0, a2, 0x40-STORSIZE
+ andi t0, a2, 0x40-STORSIZE
PTR_ADDU t1, a0 /* end address */
.set reorder
@@ -145,7 +145,7 @@ FEXPORT(__bzero)
.set at
#endif
jr t1
- PTR_ADDU a0, t0 /* dest ptr */
+ PTR_ADDU a0, t0 /* dest ptr */
.set push
.set noreorder
@@ -155,7 +155,7 @@ FEXPORT(__bzero)
andi a2, STORMASK /* At most one long to go */
beqz a2, 1f
- PTR_ADDU a0, a2 /* What's left */
+ PTR_ADDU a0, a2 /* What's left */
R10KCBARRIER(0(ra))
#ifdef __MIPSEB__
EX(LONG_S_R, a1, -1(a0), .Llast_fixup)
@@ -164,24 +164,24 @@ FEXPORT(__bzero)
EX(LONG_S_L, a1, -1(a0), .Llast_fixup)
#endif
1: jr ra
- move a2, zero
+ move a2, zero
.Lsmall_memset:
beqz a2, 2f
- PTR_ADDU t1, a0, a2
+ PTR_ADDU t1, a0, a2
1: PTR_ADDIU a0, 1 /* fill bytewise */
R10KCBARRIER(0(ra))
bne t1, a0, 1b
- sb a1, -1(a0)
+ sb a1, -1(a0)
2: jr ra /* done */
- move a2, zero
+ move a2, zero
END(memset)
.Lfirst_fixup:
jr ra
- nop
+ nop
.Lfwd_fixup:
PTR_L t0, TI_TASK($28)
@@ -189,7 +189,7 @@ FEXPORT(__bzero)
LONG_L t0, THREAD_BUADDR(t0)
LONG_ADDU a2, t1
jr ra
- LONG_SUBU a2, t0
+ LONG_SUBU a2, t0
.Lpartial_fixup:
PTR_L t0, TI_TASK($28)
@@ -197,8 +197,8 @@ FEXPORT(__bzero)
LONG_L t0, THREAD_BUADDR(t0)
LONG_ADDU a2, t1
jr ra
- LONG_SUBU a2, t0
+ LONG_SUBU a2, t0
.Llast_fixup:
jr ra
- andi v1, a2, STORMASK
+ andi v1, a2, STORMASK
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* Re: [PATCH 20/58] MIPS: lib: memset: Whitespace fixes
2014-01-27 20:19 ` Markos Chandras
(?)
@ 2014-01-27 21:43 ` Sergei Shtylyov
2014-01-27 20:55 ` Markos Chandras
-1 siblings, 1 reply; 126+ messages in thread
From: Sergei Shtylyov @ 2014-01-27 21:43 UTC (permalink / raw)
To: Markos Chandras, linux-mips
Hello.
On 01/27/2014 11:19 PM, Markos Chandras wrote:
> Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
You're spoiling intentionally marked delay slots, not fixing anything
here. This is a convention.
WBR, Sergei
^ permalink raw reply [flat|nested] 126+ messages in thread* Re: [PATCH 20/58] MIPS: lib: memset: Whitespace fixes
@ 2014-01-27 20:55 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:55 UTC (permalink / raw)
To: Sergei Shtylyov, linux-mips
On 01/27/2014 09:43 PM, Sergei Shtylyov wrote:
> ....
> Hello.
>
> On 01/27/2014 11:19 PM, Markos Chandras wrote:
>
>> Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
>
> You're spoiling intentionally marked delay slots, not fixing
> anything here. This is a convention.
>
> WBR, Sergei
>
>
Hi Sergei,
Apologies. I was not aware of this convention. I will drop this patch
and fix the next one. I will also double check if I have made the same
mistake in any of the other patches.
--
markos
^ permalink raw reply [flat|nested] 126+ messages in thread
* Re: [PATCH 20/58] MIPS: lib: memset: Whitespace fixes
@ 2014-01-27 20:55 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:55 UTC (permalink / raw)
To: Sergei Shtylyov, linux-mips
On 01/27/2014 09:43 PM, Sergei Shtylyov wrote:
> ....
> Hello.
>
> On 01/27/2014 11:19 PM, Markos Chandras wrote:
>
>> Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
>
> You're spoiling intentionally marked delay slots, not fixing
> anything here. This is a convention.
>
> WBR, Sergei
>
>
Hi Sergei,
Apologies. I was not aware of this convention. I will drop this patch
and fix the next one. I will also double check if I have made the same
mistake in any of the other patches.
--
markos
^ permalink raw reply [flat|nested] 126+ messages in thread
* [PATCH 21/58] MIPS: lib: memset: Use macro to build the __bzero symbol
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Build the __bzero symbol using a macor. In EVA mode we will
need to use similar code to do the userspace load operations so
it is better if we use a macro to avoid code duplications.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/memset.S | 95 +++++++++++++++++++++++++++++++-------------------
1 file changed, 60 insertions(+), 35 deletions(-)
diff --git a/arch/mips/lib/memset.S b/arch/mips/lib/memset.S
index d857985..05fac19 100644
--- a/arch/mips/lib/memset.S
+++ b/arch/mips/lib/memset.S
@@ -34,6 +34,9 @@
#define FILLPTRG t0
#endif
+#define LEGACY_MODE 1
+#define EVA_MODE 2
+
#define EX(insn,reg,addr,handler) \
9: insn reg, addr; \
.section __ex_table,"a"; \
@@ -63,33 +66,23 @@
#endif
.endm
-/*
- * memset(void *s, int c, size_t n)
- *
- * a0: start of area to clear
- * a1: char to fill with
- * a2: size of area to clear
- */
.set noreorder
.align 5
-LEAF(memset)
- beqz a1, 1f
- move v0, a0 /* result */
- andi a1, 0xff /* spread fillword */
- LONG_SLL t1, a1, 8
- or a1, t1
- LONG_SLL t1, a1, 16
-#if LONGSIZE == 8
- or a1, t1
- LONG_SLL t1, a1, 32
-#endif
- or a1, t1
-1:
+ /*
+ * Macro to generate the __bzero{,_user} symbol
+ * Arguments:
+ * mode: LEGACY_MODE or EVA_MODE
+ */
+ .macro __BUILD_BZERO mode
+ /* Initialize __memset if this is the first time we call this macro */
+ .ifnotdef __memset
+ .set __memset, 1
+ .hidden __memset /* Make sure it does not leak */
+ .endif
-FEXPORT(__bzero)
sltiu t0, a2, STORSIZE /* very small region? */
- bnez t0, .Lsmall_memset
+ bnez t0, .Lsmall_memset\@
andi t0, a0, STORMASK /* aligned? */
#ifdef CONFIG_CPU_MICROMIPS
@@ -109,28 +102,28 @@ FEXPORT(__bzero)
R10KCBARRIER(0(ra))
#ifdef __MIPSEB__
- EX(LONG_S_L, a1, (a0), .Lfirst_fixup) /* make word/dword aligned */
+ EX(LONG_S_L, a1, (a0), .Lfirst_fixup\@) /* make word/dword aligned */
#endif
#ifdef __MIPSEL__
- EX(LONG_S_R, a1, (a0), .Lfirst_fixup) /* make word/dword aligned */
+ EX(LONG_S_R, a1, (a0), .Lfirst_fixup\@) /* make word/dword aligned */
#endif
PTR_SUBU a0, t0 /* long align ptr */
PTR_ADDU a2, t0 /* correct size */
1: ori t1, a2, 0x3f /* # of full blocks */
xori t1, 0x3f
- beqz t1, .Lmemset_partial /* no block to fill */
+ beqz t1, .Lmemset_partial\@ /* no block to fill */
andi t0, a2, 0x40-STORSIZE
PTR_ADDU t1, a0 /* end address */
.set reorder
1: PTR_ADDIU a0, 64
R10KCBARRIER(0(ra))
- f_fill64 a0, -64, FILL64RG, .Lfwd_fixup
+ f_fill64 a0, -64, FILL64RG, .Lfwd_fixup\@
bne t1, a0, 1b
.set noreorder
-.Lmemset_partial:
+.Lmemset_partial\@:
R10KCBARRIER(0(ra))
PTR_LA t1, 2f /* where to start */
#ifdef CONFIG_CPU_MICROMIPS
@@ -150,7 +143,8 @@ FEXPORT(__bzero)
.set push
.set noreorder
.set nomacro
- f_fill64 a0, -64, FILL64RG, .Lpartial_fixup /* ... but first do longs ... */
+ /* ... but first do longs ... */
+ f_fill64 a0, -64, FILL64RG, .Lpartial_fixup\@
2: .set pop
andi a2, STORMASK /* At most one long to go */
@@ -158,15 +152,15 @@ FEXPORT(__bzero)
PTR_ADDU a0, a2 /* What's left */
R10KCBARRIER(0(ra))
#ifdef __MIPSEB__
- EX(LONG_S_R, a1, -1(a0), .Llast_fixup)
+ EX(LONG_S_R, a1, -1(a0), .Llast_fixup\@)
#endif
#ifdef __MIPSEL__
- EX(LONG_S_L, a1, -1(a0), .Llast_fixup)
+ EX(LONG_S_L, a1, -1(a0), .Llast_fixup\@)
#endif
1: jr ra
move a2, zero
-.Lsmall_memset:
+.Lsmall_memset\@:
beqz a2, 2f
PTR_ADDU t1, a0, a2
@@ -177,13 +171,17 @@ FEXPORT(__bzero)
2: jr ra /* done */
move a2, zero
+ .if __memset == 1
END(memset)
+ .set __memset, 0
+ .hidden __memset
+ .endif
-.Lfirst_fixup:
+.Lfirst_fixup\@:
jr ra
nop
-.Lfwd_fixup:
+.Lfwd_fixup\@:
PTR_L t0, TI_TASK($28)
andi a2, 0x3f
LONG_L t0, THREAD_BUADDR(t0)
@@ -191,7 +189,7 @@ FEXPORT(__bzero)
jr ra
LONG_SUBU a2, t0
-.Lpartial_fixup:
+.Lpartial_fixup\@:
PTR_L t0, TI_TASK($28)
andi a2, STORMASK
LONG_L t0, THREAD_BUADDR(t0)
@@ -199,6 +197,33 @@ FEXPORT(__bzero)
jr ra
LONG_SUBU a2, t0
-.Llast_fixup:
+.Llast_fixup\@:
jr ra
andi v1, a2, STORMASK
+
+ .endm
+
+/*
+ * memset(void *s, int c, size_t n)
+ *
+ * a0: start of area to clear
+ * a1: char to fill with
+ * a2: size of area to clear
+ */
+
+LEAF(memset)
+ beqz a1, 1f
+ move v0, a0 /* result */
+
+ andi a1, 0xff /* spread fillword */
+ LONG_SLL t1, a1, 8
+ or a1, t1
+ LONG_SLL t1, a1, 16
+#if LONGSIZE == 8
+ or a1, t1
+ LONG_SLL t1, a1, 32
+#endif
+ or a1, t1
+1:
+FEXPORT(__bzero)
+ __BUILD_BZERO LEGACY_MODE
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 21/58] MIPS: lib: memset: Use macro to build the __bzero symbol
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Build the __bzero symbol using a macor. In EVA mode we will
need to use similar code to do the userspace load operations so
it is better if we use a macro to avoid code duplications.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/memset.S | 95 +++++++++++++++++++++++++++++++-------------------
1 file changed, 60 insertions(+), 35 deletions(-)
diff --git a/arch/mips/lib/memset.S b/arch/mips/lib/memset.S
index d857985..05fac19 100644
--- a/arch/mips/lib/memset.S
+++ b/arch/mips/lib/memset.S
@@ -34,6 +34,9 @@
#define FILLPTRG t0
#endif
+#define LEGACY_MODE 1
+#define EVA_MODE 2
+
#define EX(insn,reg,addr,handler) \
9: insn reg, addr; \
.section __ex_table,"a"; \
@@ -63,33 +66,23 @@
#endif
.endm
-/*
- * memset(void *s, int c, size_t n)
- *
- * a0: start of area to clear
- * a1: char to fill with
- * a2: size of area to clear
- */
.set noreorder
.align 5
-LEAF(memset)
- beqz a1, 1f
- move v0, a0 /* result */
- andi a1, 0xff /* spread fillword */
- LONG_SLL t1, a1, 8
- or a1, t1
- LONG_SLL t1, a1, 16
-#if LONGSIZE == 8
- or a1, t1
- LONG_SLL t1, a1, 32
-#endif
- or a1, t1
-1:
+ /*
+ * Macro to generate the __bzero{,_user} symbol
+ * Arguments:
+ * mode: LEGACY_MODE or EVA_MODE
+ */
+ .macro __BUILD_BZERO mode
+ /* Initialize __memset if this is the first time we call this macro */
+ .ifnotdef __memset
+ .set __memset, 1
+ .hidden __memset /* Make sure it does not leak */
+ .endif
-FEXPORT(__bzero)
sltiu t0, a2, STORSIZE /* very small region? */
- bnez t0, .Lsmall_memset
+ bnez t0, .Lsmall_memset\@
andi t0, a0, STORMASK /* aligned? */
#ifdef CONFIG_CPU_MICROMIPS
@@ -109,28 +102,28 @@ FEXPORT(__bzero)
R10KCBARRIER(0(ra))
#ifdef __MIPSEB__
- EX(LONG_S_L, a1, (a0), .Lfirst_fixup) /* make word/dword aligned */
+ EX(LONG_S_L, a1, (a0), .Lfirst_fixup\@) /* make word/dword aligned */
#endif
#ifdef __MIPSEL__
- EX(LONG_S_R, a1, (a0), .Lfirst_fixup) /* make word/dword aligned */
+ EX(LONG_S_R, a1, (a0), .Lfirst_fixup\@) /* make word/dword aligned */
#endif
PTR_SUBU a0, t0 /* long align ptr */
PTR_ADDU a2, t0 /* correct size */
1: ori t1, a2, 0x3f /* # of full blocks */
xori t1, 0x3f
- beqz t1, .Lmemset_partial /* no block to fill */
+ beqz t1, .Lmemset_partial\@ /* no block to fill */
andi t0, a2, 0x40-STORSIZE
PTR_ADDU t1, a0 /* end address */
.set reorder
1: PTR_ADDIU a0, 64
R10KCBARRIER(0(ra))
- f_fill64 a0, -64, FILL64RG, .Lfwd_fixup
+ f_fill64 a0, -64, FILL64RG, .Lfwd_fixup\@
bne t1, a0, 1b
.set noreorder
-.Lmemset_partial:
+.Lmemset_partial\@:
R10KCBARRIER(0(ra))
PTR_LA t1, 2f /* where to start */
#ifdef CONFIG_CPU_MICROMIPS
@@ -150,7 +143,8 @@ FEXPORT(__bzero)
.set push
.set noreorder
.set nomacro
- f_fill64 a0, -64, FILL64RG, .Lpartial_fixup /* ... but first do longs ... */
+ /* ... but first do longs ... */
+ f_fill64 a0, -64, FILL64RG, .Lpartial_fixup\@
2: .set pop
andi a2, STORMASK /* At most one long to go */
@@ -158,15 +152,15 @@ FEXPORT(__bzero)
PTR_ADDU a0, a2 /* What's left */
R10KCBARRIER(0(ra))
#ifdef __MIPSEB__
- EX(LONG_S_R, a1, -1(a0), .Llast_fixup)
+ EX(LONG_S_R, a1, -1(a0), .Llast_fixup\@)
#endif
#ifdef __MIPSEL__
- EX(LONG_S_L, a1, -1(a0), .Llast_fixup)
+ EX(LONG_S_L, a1, -1(a0), .Llast_fixup\@)
#endif
1: jr ra
move a2, zero
-.Lsmall_memset:
+.Lsmall_memset\@:
beqz a2, 2f
PTR_ADDU t1, a0, a2
@@ -177,13 +171,17 @@ FEXPORT(__bzero)
2: jr ra /* done */
move a2, zero
+ .if __memset == 1
END(memset)
+ .set __memset, 0
+ .hidden __memset
+ .endif
-.Lfirst_fixup:
+.Lfirst_fixup\@:
jr ra
nop
-.Lfwd_fixup:
+.Lfwd_fixup\@:
PTR_L t0, TI_TASK($28)
andi a2, 0x3f
LONG_L t0, THREAD_BUADDR(t0)
@@ -191,7 +189,7 @@ FEXPORT(__bzero)
jr ra
LONG_SUBU a2, t0
-.Lpartial_fixup:
+.Lpartial_fixup\@:
PTR_L t0, TI_TASK($28)
andi a2, STORMASK
LONG_L t0, THREAD_BUADDR(t0)
@@ -199,6 +197,33 @@ FEXPORT(__bzero)
jr ra
LONG_SUBU a2, t0
-.Llast_fixup:
+.Llast_fixup\@:
jr ra
andi v1, a2, STORMASK
+
+ .endm
+
+/*
+ * memset(void *s, int c, size_t n)
+ *
+ * a0: start of area to clear
+ * a1: char to fill with
+ * a2: size of area to clear
+ */
+
+LEAF(memset)
+ beqz a1, 1f
+ move v0, a0 /* result */
+
+ andi a1, 0xff /* spread fillword */
+ LONG_SLL t1, a1, 8
+ or a1, t1
+ LONG_SLL t1, a1, 16
+#if LONGSIZE == 8
+ or a1, t1
+ LONG_SLL t1, a1, 32
+#endif
+ or a1, t1
+1:
+FEXPORT(__bzero)
+ __BUILD_BZERO LEGACY_MODE
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 22/58] MIPS: lib: memset: Add EVA support for the __bzero function.
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Build the __bzero function using the EVA load/store instructions
when operating in the EVA mode. This function is only used when
accessing user code so there is no need to build two distinct symbols
for user and kernel operations respectively.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/memset.S | 27 +++++++++++++++++++++++----
1 file changed, 23 insertions(+), 4 deletions(-)
diff --git a/arch/mips/lib/memset.S b/arch/mips/lib/memset.S
index 05fac19..7b0e546 100644
--- a/arch/mips/lib/memset.S
+++ b/arch/mips/lib/memset.S
@@ -37,13 +37,24 @@
#define LEGACY_MODE 1
#define EVA_MODE 2
+/*
+ * No need to protect it with EVA #ifdefery. The generated block of code
+ * will never be assembled if EVA is not enabled.
+ */
+#define __EVAFY(insn, reg, addr) __BUILD_EVA_INSN(insn##e, reg, addr)
+#define ___BUILD_EVA_INSN(insn, reg, addr) __EVAFY(insn, reg, addr)
+
#define EX(insn,reg,addr,handler) \
-9: insn reg, addr; \
+ .if \mode == LEGACY_MODE; \
+9: insn reg, addr; \
+ .else; \
+9: ___BUILD_EVA_INSN(insn, reg, addr); \
+ .endif; \
.section __ex_table,"a"; \
PTR 9b, handler; \
.previous
- .macro f_fill64 dst, offset, val, fixup
+ .macro f_fill64 dst, offset, val, fixup, mode
EX(LONG_S, \val, (\offset + 0 * STORSIZE)(\dst), \fixup)
EX(LONG_S, \val, (\offset + 1 * STORSIZE)(\dst), \fixup)
EX(LONG_S, \val, (\offset + 2 * STORSIZE)(\dst), \fixup)
@@ -119,7 +130,7 @@
.set reorder
1: PTR_ADDIU a0, 64
R10KCBARRIER(0(ra))
- f_fill64 a0, -64, FILL64RG, .Lfwd_fixup\@
+ f_fill64 a0, -64, FILL64RG, .Lfwd_fixup\@, \mode
bne t1, a0, 1b
.set noreorder
@@ -144,7 +155,7 @@
.set noreorder
.set nomacro
/* ... but first do longs ... */
- f_fill64 a0, -64, FILL64RG, .Lpartial_fixup\@
+ f_fill64 a0, -64, FILL64RG, .Lpartial_fixup\@, \mode
2: .set pop
andi a2, STORMASK /* At most one long to go */
@@ -225,5 +236,13 @@ LEAF(memset)
#endif
or a1, t1
1:
+#ifndef CONFIG_EVA
FEXPORT(__bzero)
+#endif
__BUILD_BZERO LEGACY_MODE
+
+#ifdef CONFIG_EVA
+LEAF(__bzero)
+ __BUILD_BZERO EVA_MODE
+END(__bzero)
+#endif
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 22/58] MIPS: lib: memset: Add EVA support for the __bzero function.
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Build the __bzero function using the EVA load/store instructions
when operating in the EVA mode. This function is only used when
accessing user code so there is no need to build two distinct symbols
for user and kernel operations respectively.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/memset.S | 27 +++++++++++++++++++++++----
1 file changed, 23 insertions(+), 4 deletions(-)
diff --git a/arch/mips/lib/memset.S b/arch/mips/lib/memset.S
index 05fac19..7b0e546 100644
--- a/arch/mips/lib/memset.S
+++ b/arch/mips/lib/memset.S
@@ -37,13 +37,24 @@
#define LEGACY_MODE 1
#define EVA_MODE 2
+/*
+ * No need to protect it with EVA #ifdefery. The generated block of code
+ * will never be assembled if EVA is not enabled.
+ */
+#define __EVAFY(insn, reg, addr) __BUILD_EVA_INSN(insn##e, reg, addr)
+#define ___BUILD_EVA_INSN(insn, reg, addr) __EVAFY(insn, reg, addr)
+
#define EX(insn,reg,addr,handler) \
-9: insn reg, addr; \
+ .if \mode == LEGACY_MODE; \
+9: insn reg, addr; \
+ .else; \
+9: ___BUILD_EVA_INSN(insn, reg, addr); \
+ .endif; \
.section __ex_table,"a"; \
PTR 9b, handler; \
.previous
- .macro f_fill64 dst, offset, val, fixup
+ .macro f_fill64 dst, offset, val, fixup, mode
EX(LONG_S, \val, (\offset + 0 * STORSIZE)(\dst), \fixup)
EX(LONG_S, \val, (\offset + 1 * STORSIZE)(\dst), \fixup)
EX(LONG_S, \val, (\offset + 2 * STORSIZE)(\dst), \fixup)
@@ -119,7 +130,7 @@
.set reorder
1: PTR_ADDIU a0, 64
R10KCBARRIER(0(ra))
- f_fill64 a0, -64, FILL64RG, .Lfwd_fixup\@
+ f_fill64 a0, -64, FILL64RG, .Lfwd_fixup\@, \mode
bne t1, a0, 1b
.set noreorder
@@ -144,7 +155,7 @@
.set noreorder
.set nomacro
/* ... but first do longs ... */
- f_fill64 a0, -64, FILL64RG, .Lpartial_fixup\@
+ f_fill64 a0, -64, FILL64RG, .Lpartial_fixup\@, \mode
2: .set pop
andi a2, STORMASK /* At most one long to go */
@@ -225,5 +236,13 @@ LEAF(memset)
#endif
or a1, t1
1:
+#ifndef CONFIG_EVA
FEXPORT(__bzero)
+#endif
__BUILD_BZERO LEGACY_MODE
+
+#ifdef CONFIG_EVA
+LEAF(__bzero)
+ __BUILD_BZERO EVA_MODE
+END(__bzero)
+#endif
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 23/58] MIPS: asm: uaccess: Add instruction argument to __{put,get}_user_asm
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
In preparation for EVA support, an instruction argument is needed
for the __get_user_asm{,_ll32} functions to allow instruction overrides in
EVA mode. Even though EVA only works for MIPS 32-bit, both codepaths are
changed (32-bit and 64-bit) for consistency reasons.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/uaccess.h | 26 +++++++++++++-------------
1 file changed, 13 insertions(+), 13 deletions(-)
diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index f3fa375..5ba3933 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -223,10 +223,10 @@ struct __large_struct { unsigned long buf[100]; };
* for 32 bit mode and old iron.
*/
#ifdef CONFIG_32BIT
-#define __GET_USER_DW(val, ptr) __get_user_asm_ll32(val, ptr)
+#define __GET_USER_DW(val, insn, ptr) __get_user_asm_ll32(val, insn, ptr)
#endif
#ifdef CONFIG_64BIT
-#define __GET_USER_DW(val, ptr) __get_user_asm(val, "ld", ptr)
+#define __GET_USER_DW(val, insn, ptr) __get_user_asm(val, "ld", ptr)
#endif
extern void __get_user_unknown(void);
@@ -237,7 +237,7 @@ do { \
case 1: __get_user_asm(val, "lb", ptr); break; \
case 2: __get_user_asm(val, "lh", ptr); break; \
case 4: __get_user_asm(val, "lw", ptr); break; \
- case 8: __GET_USER_DW(val, ptr); break; \
+ case 8: __GET_USER_DW(val, "lw", ptr); break; \
default: __get_user_unknown(); break; \
} \
} while (0)
@@ -287,7 +287,7 @@ do { \
/*
* Get a long long 64 using 32 bit registers.
*/
-#define __get_user_asm_ll32(val, addr) \
+#define __get_user_asm_ll32(val, insn, addr) \
{ \
union { \
unsigned long long l; \
@@ -295,8 +295,8 @@ do { \
} __gu_tmp; \
\
__asm__ __volatile__( \
- "1: lw %1, (%3) \n" \
- "2: lw %D1, 4(%3) \n" \
+ "1: " insn " %1, (%3) \n" \
+ "2: " insn " %D1, 4(%3) \n" \
"3: \n" \
" .insn \n" \
" .section .fixup,\"ax\" \n" \
@@ -320,10 +320,10 @@ do { \
* for 32 bit mode and old iron.
*/
#ifdef CONFIG_32BIT
-#define __PUT_USER_DW(ptr) __put_user_asm_ll32(ptr)
+#define __PUT_USER_DW(insn, ptr) __put_user_asm_ll32(insn, ptr)
#endif
#ifdef CONFIG_64BIT
-#define __PUT_USER_DW(ptr) __put_user_asm("sd", ptr)
+#define __PUT_USER_DW(insn, ptr) __put_user_asm("sd", ptr)
#endif
#define __put_user_nocheck(x, ptr, size) \
@@ -337,7 +337,7 @@ do { \
case 1: __put_user_asm("sb", ptr); break; \
case 2: __put_user_asm("sh", ptr); break; \
case 4: __put_user_asm("sw", ptr); break; \
- case 8: __PUT_USER_DW(ptr); break; \
+ case 8: __PUT_USER_DW("sw", ptr); break; \
default: __put_user_unknown(); break; \
} \
__pu_err; \
@@ -355,7 +355,7 @@ do { \
case 1: __put_user_asm("sb", __pu_addr); break; \
case 2: __put_user_asm("sh", __pu_addr); break; \
case 4: __put_user_asm("sw", __pu_addr); break; \
- case 8: __PUT_USER_DW(__pu_addr); break; \
+ case 8: __PUT_USER_DW("sw", __pu_addr); break; \
default: __put_user_unknown(); break; \
} \
} \
@@ -380,11 +380,11 @@ do { \
"i" (-EFAULT)); \
}
-#define __put_user_asm_ll32(ptr) \
+#define __put_user_asm_ll32(insn, ptr) \
{ \
__asm__ __volatile__( \
- "1: sw %2, (%3) # __put_user_asm_ll32 \n" \
- "2: sw %D2, 4(%3) \n" \
+ "1: " insn " %2, (%3)# __put_user_asm_ll32 \n" \
+ "2: " insn " %D2, 4(%3) \n" \
"3: \n" \
" .insn \n" \
" .section .fixup,\"ax\" \n" \
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 23/58] MIPS: asm: uaccess: Add instruction argument to __{put,get}_user_asm
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
In preparation for EVA support, an instruction argument is needed
for the __get_user_asm{,_ll32} functions to allow instruction overrides in
EVA mode. Even though EVA only works for MIPS 32-bit, both codepaths are
changed (32-bit and 64-bit) for consistency reasons.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/uaccess.h | 26 +++++++++++++-------------
1 file changed, 13 insertions(+), 13 deletions(-)
diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index f3fa375..5ba3933 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -223,10 +223,10 @@ struct __large_struct { unsigned long buf[100]; };
* for 32 bit mode and old iron.
*/
#ifdef CONFIG_32BIT
-#define __GET_USER_DW(val, ptr) __get_user_asm_ll32(val, ptr)
+#define __GET_USER_DW(val, insn, ptr) __get_user_asm_ll32(val, insn, ptr)
#endif
#ifdef CONFIG_64BIT
-#define __GET_USER_DW(val, ptr) __get_user_asm(val, "ld", ptr)
+#define __GET_USER_DW(val, insn, ptr) __get_user_asm(val, "ld", ptr)
#endif
extern void __get_user_unknown(void);
@@ -237,7 +237,7 @@ do { \
case 1: __get_user_asm(val, "lb", ptr); break; \
case 2: __get_user_asm(val, "lh", ptr); break; \
case 4: __get_user_asm(val, "lw", ptr); break; \
- case 8: __GET_USER_DW(val, ptr); break; \
+ case 8: __GET_USER_DW(val, "lw", ptr); break; \
default: __get_user_unknown(); break; \
} \
} while (0)
@@ -287,7 +287,7 @@ do { \
/*
* Get a long long 64 using 32 bit registers.
*/
-#define __get_user_asm_ll32(val, addr) \
+#define __get_user_asm_ll32(val, insn, addr) \
{ \
union { \
unsigned long long l; \
@@ -295,8 +295,8 @@ do { \
} __gu_tmp; \
\
__asm__ __volatile__( \
- "1: lw %1, (%3) \n" \
- "2: lw %D1, 4(%3) \n" \
+ "1: " insn " %1, (%3) \n" \
+ "2: " insn " %D1, 4(%3) \n" \
"3: \n" \
" .insn \n" \
" .section .fixup,\"ax\" \n" \
@@ -320,10 +320,10 @@ do { \
* for 32 bit mode and old iron.
*/
#ifdef CONFIG_32BIT
-#define __PUT_USER_DW(ptr) __put_user_asm_ll32(ptr)
+#define __PUT_USER_DW(insn, ptr) __put_user_asm_ll32(insn, ptr)
#endif
#ifdef CONFIG_64BIT
-#define __PUT_USER_DW(ptr) __put_user_asm("sd", ptr)
+#define __PUT_USER_DW(insn, ptr) __put_user_asm("sd", ptr)
#endif
#define __put_user_nocheck(x, ptr, size) \
@@ -337,7 +337,7 @@ do { \
case 1: __put_user_asm("sb", ptr); break; \
case 2: __put_user_asm("sh", ptr); break; \
case 4: __put_user_asm("sw", ptr); break; \
- case 8: __PUT_USER_DW(ptr); break; \
+ case 8: __PUT_USER_DW("sw", ptr); break; \
default: __put_user_unknown(); break; \
} \
__pu_err; \
@@ -355,7 +355,7 @@ do { \
case 1: __put_user_asm("sb", __pu_addr); break; \
case 2: __put_user_asm("sh", __pu_addr); break; \
case 4: __put_user_asm("sw", __pu_addr); break; \
- case 8: __PUT_USER_DW(__pu_addr); break; \
+ case 8: __PUT_USER_DW("sw", __pu_addr); break; \
default: __put_user_unknown(); break; \
} \
} \
@@ -380,11 +380,11 @@ do { \
"i" (-EFAULT)); \
}
-#define __put_user_asm_ll32(ptr) \
+#define __put_user_asm_ll32(insn, ptr) \
{ \
__asm__ __volatile__( \
- "1: sw %2, (%3) # __put_user_asm_ll32 \n" \
- "2: sw %D2, 4(%3) \n" \
+ "1: " insn " %2, (%3)# __put_user_asm_ll32 \n" \
+ "2: " insn " %D2, 4(%3) \n" \
"3: \n" \
" .insn \n" \
" .section .fixup,\"ax\" \n" \
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 24/58] MIPS: asm: uaccess: Move duplicated code to common function
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Similar to __get_user_* functions, move common code to
__put_user_*_common so it can be shared among similar users.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/uaccess.h | 61 ++++++++++++++++++++---------------------
1 file changed, 29 insertions(+), 32 deletions(-)
diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index 5ba3933..0df57e7 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -326,13 +326,8 @@ do { \
#define __PUT_USER_DW(insn, ptr) __put_user_asm("sd", ptr)
#endif
-#define __put_user_nocheck(x, ptr, size) \
-({ \
- __typeof__(*(ptr)) __pu_val; \
- int __pu_err = 0; \
- \
- __chk_user_ptr(ptr); \
- __pu_val = (x); \
+#define __put_user_common(ptr, size) \
+do { \
switch (size) { \
case 1: __put_user_asm("sb", ptr); break; \
case 2: __put_user_asm("sh", ptr); break; \
@@ -340,6 +335,16 @@ do { \
case 8: __PUT_USER_DW("sw", ptr); break; \
default: __put_user_unknown(); break; \
} \
+} while (0)
+
+#define __put_user_nocheck(x, ptr, size) \
+({ \
+ __typeof__(*(ptr)) __pu_val; \
+ int __pu_err = 0; \
+ \
+ __chk_user_ptr(ptr); \
+ __pu_val = (x); \
+ __put_user_common(ptr, size); \
__pu_err; \
})
@@ -350,15 +355,9 @@ do { \
int __pu_err = -EFAULT; \
\
might_fault(); \
- if (likely(access_ok(VERIFY_WRITE, __pu_addr, size))) { \
- switch (size) { \
- case 1: __put_user_asm("sb", __pu_addr); break; \
- case 2: __put_user_asm("sh", __pu_addr); break; \
- case 4: __put_user_asm("sw", __pu_addr); break; \
- case 8: __PUT_USER_DW("sw", __pu_addr); break; \
- default: __put_user_unknown(); break; \
- } \
- } \
+ if (likely(access_ok(VERIFY_WRITE, __pu_addr, size))) \
+ __put_user_common(__pu_addr, size); \
+ \
__pu_err; \
})
@@ -594,19 +593,23 @@ do { \
#define __PUT_USER_UNALIGNED_DW(ptr) __put_user_unaligned_asm("usd", ptr)
#endif
-#define __put_user_unaligned_nocheck(x,ptr,size) \
-({ \
- __typeof__(*(ptr)) __pu_val; \
- int __pu_err = 0; \
- \
- __pu_val = (x); \
+#define __put_user_unaligned_common(ptr, size) \
+do { \
switch (size) { \
case 1: __put_user_asm("sb", ptr); break; \
case 2: __put_user_unaligned_asm("ush", ptr); break; \
case 4: __put_user_unaligned_asm("usw", ptr); break; \
case 8: __PUT_USER_UNALIGNED_DW(ptr); break; \
default: __put_user_unaligned_unknown(); break; \
- } \
+} while (0)
+
+#define __put_user_unaligned_nocheck(x,ptr,size) \
+({ \
+ __typeof__(*(ptr)) __pu_val; \
+ int __pu_err = 0; \
+ \
+ __pu_val = (x); \
+ __put_user_unaligned_common(ptr, size); \
__pu_err; \
})
@@ -616,15 +619,9 @@ do { \
__typeof__(*(ptr)) __pu_val = (x); \
int __pu_err = -EFAULT; \
\
- if (likely(access_ok(VERIFY_WRITE, __pu_addr, size))) { \
- switch (size) { \
- case 1: __put_user_asm("sb", __pu_addr); break; \
- case 2: __put_user_unaligned_asm("ush", __pu_addr); break; \
- case 4: __put_user_unaligned_asm("usw", __pu_addr); break; \
- case 8: __PUT_USER_UNALGINED_DW(__pu_addr); break; \
- default: __put_user_unaligned_unknown(); break; \
- } \
- } \
+ if (likely(access_ok(VERIFY_WRITE, __pu_addr, size))) \
+ __put_user_unaligned_common(__pu_addr, size); \
+ \
__pu_err; \
})
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 24/58] MIPS: asm: uaccess: Move duplicated code to common function
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Similar to __get_user_* functions, move common code to
__put_user_*_common so it can be shared among similar users.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/uaccess.h | 61 ++++++++++++++++++++---------------------
1 file changed, 29 insertions(+), 32 deletions(-)
diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index 5ba3933..0df57e7 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -326,13 +326,8 @@ do { \
#define __PUT_USER_DW(insn, ptr) __put_user_asm("sd", ptr)
#endif
-#define __put_user_nocheck(x, ptr, size) \
-({ \
- __typeof__(*(ptr)) __pu_val; \
- int __pu_err = 0; \
- \
- __chk_user_ptr(ptr); \
- __pu_val = (x); \
+#define __put_user_common(ptr, size) \
+do { \
switch (size) { \
case 1: __put_user_asm("sb", ptr); break; \
case 2: __put_user_asm("sh", ptr); break; \
@@ -340,6 +335,16 @@ do { \
case 8: __PUT_USER_DW("sw", ptr); break; \
default: __put_user_unknown(); break; \
} \
+} while (0)
+
+#define __put_user_nocheck(x, ptr, size) \
+({ \
+ __typeof__(*(ptr)) __pu_val; \
+ int __pu_err = 0; \
+ \
+ __chk_user_ptr(ptr); \
+ __pu_val = (x); \
+ __put_user_common(ptr, size); \
__pu_err; \
})
@@ -350,15 +355,9 @@ do { \
int __pu_err = -EFAULT; \
\
might_fault(); \
- if (likely(access_ok(VERIFY_WRITE, __pu_addr, size))) { \
- switch (size) { \
- case 1: __put_user_asm("sb", __pu_addr); break; \
- case 2: __put_user_asm("sh", __pu_addr); break; \
- case 4: __put_user_asm("sw", __pu_addr); break; \
- case 8: __PUT_USER_DW("sw", __pu_addr); break; \
- default: __put_user_unknown(); break; \
- } \
- } \
+ if (likely(access_ok(VERIFY_WRITE, __pu_addr, size))) \
+ __put_user_common(__pu_addr, size); \
+ \
__pu_err; \
})
@@ -594,19 +593,23 @@ do { \
#define __PUT_USER_UNALIGNED_DW(ptr) __put_user_unaligned_asm("usd", ptr)
#endif
-#define __put_user_unaligned_nocheck(x,ptr,size) \
-({ \
- __typeof__(*(ptr)) __pu_val; \
- int __pu_err = 0; \
- \
- __pu_val = (x); \
+#define __put_user_unaligned_common(ptr, size) \
+do { \
switch (size) { \
case 1: __put_user_asm("sb", ptr); break; \
case 2: __put_user_unaligned_asm("ush", ptr); break; \
case 4: __put_user_unaligned_asm("usw", ptr); break; \
case 8: __PUT_USER_UNALIGNED_DW(ptr); break; \
default: __put_user_unaligned_unknown(); break; \
- } \
+} while (0)
+
+#define __put_user_unaligned_nocheck(x,ptr,size) \
+({ \
+ __typeof__(*(ptr)) __pu_val; \
+ int __pu_err = 0; \
+ \
+ __pu_val = (x); \
+ __put_user_unaligned_common(ptr, size); \
__pu_err; \
})
@@ -616,15 +619,9 @@ do { \
__typeof__(*(ptr)) __pu_val = (x); \
int __pu_err = -EFAULT; \
\
- if (likely(access_ok(VERIFY_WRITE, __pu_addr, size))) { \
- switch (size) { \
- case 1: __put_user_asm("sb", __pu_addr); break; \
- case 2: __put_user_unaligned_asm("ush", __pu_addr); break; \
- case 4: __put_user_unaligned_asm("usw", __pu_addr); break; \
- case 8: __PUT_USER_UNALGINED_DW(__pu_addr); break; \
- default: __put_user_unaligned_unknown(); break; \
- } \
- } \
+ if (likely(access_ok(VERIFY_WRITE, __pu_addr, size))) \
+ __put_user_unaligned_common(__pu_addr, size); \
+ \
__pu_err; \
})
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 25/58] MIPS: asm: uaccess: Disable unaligned access macros for EVA
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
ulb, ulh, ulw are macros which emulate unaligned access for MIPS.
However, no such macros exist for EVA mode, so the only way to do
EVA unaligned accesses is in the ADE exception handler. As a result
of which, disable these macros for EVA.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/uaccess.h | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index 0df57e7..9c6f5ce 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -402,6 +402,11 @@ do { \
extern void __put_user_unknown(void);
/*
+ * ul{b,h,w} are macros and there are no equivalent macros for EVA.
+ * EVA unaligned access is handled in the ADE exception handler.
+ */
+#ifndef CONFIG_EVA
+/*
* put_user_unaligned: - Write a simple value into user space.
* @x: Value to copy to user space.
* @ptr: Destination address, in user space.
@@ -666,6 +671,7 @@ do { \
}
extern void __put_user_unaligned_unknown(void);
+#endif
/*
* We're generating jump to subroutines which will be outside the range of
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 25/58] MIPS: asm: uaccess: Disable unaligned access macros for EVA
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
ulb, ulh, ulw are macros which emulate unaligned access for MIPS.
However, no such macros exist for EVA mode, so the only way to do
EVA unaligned accesses is in the ADE exception handler. As a result
of which, disable these macros for EVA.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/uaccess.h | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index 0df57e7..9c6f5ce 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -402,6 +402,11 @@ do { \
extern void __put_user_unknown(void);
/*
+ * ul{b,h,w} are macros and there are no equivalent macros for EVA.
+ * EVA unaligned access is handled in the ADE exception handler.
+ */
+#ifndef CONFIG_EVA
+/*
* put_user_unaligned: - Write a simple value into user space.
* @x: Value to copy to user space.
* @ptr: Destination address, in user space.
@@ -666,6 +671,7 @@ do { \
}
extern void __put_user_unaligned_unknown(void);
+#endif
/*
* We're generating jump to subroutines which will be outside the range of
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 26/58] MIPS: asm: uaccess: Use EVA instructions wrappers
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Use the EVA instruction wrappers from asm.h to perform
read/write operations from userland.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/uaccess.h | 133 ++++++++++++++++++++++++++++++++--------
1 file changed, 109 insertions(+), 24 deletions(-)
diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index 9c6f5ce..522fc41 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -6,6 +6,7 @@
* Copyright (C) 1996, 1997, 1998, 1999, 2000, 03, 04 by Ralf Baechle
* Copyright (C) 1999, 2000 Silicon Graphics, Inc.
* Copyright (C) 2007 Maciej W. Rozycki
+ * Copyright (C) 2014, Imagination Technologies Ltd.
*/
#ifndef _ASM_UACCESS_H
#define _ASM_UACCESS_H
@@ -13,6 +14,7 @@
#include <linux/kernel.h>
#include <linux/errno.h>
#include <linux/thread_info.h>
+#include <asm/asm.h>
/*
* The fs value determines whether argument validity checking should be
@@ -222,11 +224,44 @@ struct __large_struct { unsigned long buf[100]; };
* Yuck. We need two variants, one for 64bit operation and one
* for 32 bit mode and old iron.
*/
+#ifndef CONFIG_EVA
+#define __get_kernel_common(val, size, ptr) __get_user_common(val, size, ptr)
+#else
+/*
+ * Kernel specific functions for EVA. We need to use normal load instructions
+ * to read data from kernel when operating in EVA mode. We use these macros to
+ * avoid redefining __get_user_asm for EVA.
+ */
+#undef _loadd
+#undef _loadw
+#undef _loadh
+#undef _loadb
+#ifdef CONFIG_32BIT
+#define _loadd _loadw
+#else
+#define _loadd(reg, addr) "ld " reg ", " addr
+#endif
+#define _loadw(reg, addr) "lw " reg ", " addr
+#define _loadh(reg, addr) "lh " reg ", " addr
+#define _loadb(reg, addr) "lb " reg ", " addr
+
+#define __get_kernel_common(val, size, ptr) \
+do { \
+ switch (size) { \
+ case 1: __get_user_asm(val, _loadb, ptr); break; \
+ case 2: __get_user_asm(val, _loadh, ptr); break; \
+ case 4: __get_user_asm(val, _loadw, ptr); break; \
+ case 8: __GET_USER_DW(val, _loadd, ptr); break; \
+ default: __get_user_unknown(); break; \
+ } \
+} while (0)
+#endif
+
#ifdef CONFIG_32BIT
#define __GET_USER_DW(val, insn, ptr) __get_user_asm_ll32(val, insn, ptr)
#endif
#ifdef CONFIG_64BIT
-#define __GET_USER_DW(val, insn, ptr) __get_user_asm(val, "ld", ptr)
+#define __GET_USER_DW(val, insn, ptr) __get_user_asm(val, insn, ptr)
#endif
extern void __get_user_unknown(void);
@@ -234,10 +269,10 @@ extern void __get_user_unknown(void);
#define __get_user_common(val, size, ptr) \
do { \
switch (size) { \
- case 1: __get_user_asm(val, "lb", ptr); break; \
- case 2: __get_user_asm(val, "lh", ptr); break; \
- case 4: __get_user_asm(val, "lw", ptr); break; \
- case 8: __GET_USER_DW(val, "lw", ptr); break; \
+ case 1: __get_user_asm(val, user_lb, ptr); break; \
+ case 2: __get_user_asm(val, user_lh, ptr); break; \
+ case 4: __get_user_asm(val, user_lw, ptr); break; \
+ case 8: __GET_USER_DW(val, user_ld, ptr); break; \
default: __get_user_unknown(); break; \
} \
} while (0)
@@ -246,8 +281,12 @@ do { \
({ \
int __gu_err; \
\
- __chk_user_ptr(ptr); \
- __get_user_common((x), size, ptr); \
+ if (segment_eq(get_fs(), get_ds())) { \
+ __get_kernel_common((x), size, ptr); \
+ } else { \
+ __chk_user_ptr(ptr); \
+ __get_user_common((x), size, ptr); \
+ } \
__gu_err; \
})
@@ -257,8 +296,12 @@ do { \
const __typeof__(*(ptr)) __user * __gu_ptr = (ptr); \
\
might_fault(); \
- if (likely(access_ok(VERIFY_READ, __gu_ptr, size))) \
- __get_user_common((x), size, __gu_ptr); \
+ if (likely(access_ok(VERIFY_READ, __gu_ptr, size))) { \
+ if (segment_eq(get_fs(), get_ds())) \
+ __get_kernel_common((x), size, __gu_ptr); \
+ else \
+ __get_user_common((x), size, __gu_ptr); \
+ } \
\
__gu_err; \
})
@@ -268,7 +311,7 @@ do { \
long __gu_tmp; \
\
__asm__ __volatile__( \
- "1: " insn " %1, %3 \n" \
+ "1: "insn("%1", "%3")" \n" \
"2: \n" \
" .insn \n" \
" .section .fixup,\"ax\" \n" \
@@ -295,8 +338,8 @@ do { \
} __gu_tmp; \
\
__asm__ __volatile__( \
- "1: " insn " %1, (%3) \n" \
- "2: " insn " %D1, 4(%3) \n" \
+ "1: " insn("%1", "(%3)")" \n" \
+ "2: " insn("%D1", "4(%3)")" \n" \
"3: \n" \
" .insn \n" \
" .section .fixup,\"ax\" \n" \
@@ -315,6 +358,40 @@ do { \
(val) = __gu_tmp.t; \
}
+#ifndef CONFIG_EVA
+#define __put_kernel_common(ptr, size) __put_user_common(ptr, size)
+#else
+/*
+ * Kernel specific functions for EVA. We need to use normal load instructions
+ * to read data from kernel when operating in EVA mode. We use these macros to
+ * avoid redefining __get_user_asm for EVA.
+ */
+#undef _stored
+#undef _storew
+#undef _storeh
+#undef _storeb
+#ifdef CONFIG_32BIT
+#define _stored _storew
+#else
+#define _stored(reg, addr) "ld " reg ", " addr
+#endif
+
+#define _storew(reg, addr) "sw " reg ", " addr
+#define _storeh(reg, addr) "sh " reg ", " addr
+#define _storeb(reg, addr) "sb " reg ", " addr
+
+#define __put_kernel_common(ptr, size) \
+do { \
+ switch (size) { \
+ case 1: __put_user_asm(_storeb, ptr); break; \
+ case 2: __put_user_asm(_storeh, ptr); break; \
+ case 4: __put_user_asm(_storew, ptr); break; \
+ case 8: __PUT_USER_DW(_stored, ptr); break; \
+ default: __put_user_unknown(); break; \
+ } \
+} while(0)
+#endif
+
/*
* Yuck. We need two variants, one for 64bit operation and one
* for 32 bit mode and old iron.
@@ -323,16 +400,16 @@ do { \
#define __PUT_USER_DW(insn, ptr) __put_user_asm_ll32(insn, ptr)
#endif
#ifdef CONFIG_64BIT
-#define __PUT_USER_DW(insn, ptr) __put_user_asm("sd", ptr)
+#define __PUT_USER_DW(insn, ptr) __put_user_asm(insn, ptr)
#endif
#define __put_user_common(ptr, size) \
do { \
switch (size) { \
- case 1: __put_user_asm("sb", ptr); break; \
- case 2: __put_user_asm("sh", ptr); break; \
- case 4: __put_user_asm("sw", ptr); break; \
- case 8: __PUT_USER_DW("sw", ptr); break; \
+ case 1: __put_user_asm(user_sb, ptr); break; \
+ case 2: __put_user_asm(user_sh, ptr); break; \
+ case 4: __put_user_asm(user_sw, ptr); break; \
+ case 8: __PUT_USER_DW(user_sd, ptr); break; \
default: __put_user_unknown(); break; \
} \
} while (0)
@@ -342,9 +419,13 @@ do { \
__typeof__(*(ptr)) __pu_val; \
int __pu_err = 0; \
\
- __chk_user_ptr(ptr); \
__pu_val = (x); \
- __put_user_common(ptr, size); \
+ if (segment_eq(get_fs(), get_ds())) { \
+ __put_kernel_common(ptr, size); \
+ } else { \
+ __chk_user_ptr(ptr); \
+ __put_user_common(ptr, size); \
+ } \
__pu_err; \
})
@@ -355,8 +436,12 @@ do { \
int __pu_err = -EFAULT; \
\
might_fault(); \
- if (likely(access_ok(VERIFY_WRITE, __pu_addr, size))) \
- __put_user_common(__pu_addr, size); \
+ if (likely(access_ok(VERIFY_WRITE, __pu_addr, size))) { \
+ if (segment_eq(get_fs(), get_ds())) \
+ __put_kernel_common(__pu_addr, size); \
+ else \
+ __put_user_common(__pu_addr, size); \
+ } \
\
__pu_err; \
})
@@ -364,7 +449,7 @@ do { \
#define __put_user_asm(insn, ptr) \
{ \
__asm__ __volatile__( \
- "1: " insn " %z2, %3 # __put_user_asm\n" \
+ "1: "insn("%z2", "%3")" # __put_user_asm \n" \
"2: \n" \
" .insn \n" \
" .section .fixup,\"ax\" \n" \
@@ -382,8 +467,8 @@ do { \
#define __put_user_asm_ll32(insn, ptr) \
{ \
__asm__ __volatile__( \
- "1: " insn " %2, (%3)# __put_user_asm_ll32 \n" \
- "2: " insn " %D2, 4(%3) \n" \
+ "1: "insn("%2", "(%3)")" # __put_user_asm_ll32 \n" \
+ "2: "insn("%D2", "4(%3)")" \n" \
"3: \n" \
" .insn \n" \
" .section .fixup,\"ax\" \n" \
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 26/58] MIPS: asm: uaccess: Use EVA instructions wrappers
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Use the EVA instruction wrappers from asm.h to perform
read/write operations from userland.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/uaccess.h | 133 ++++++++++++++++++++++++++++++++--------
1 file changed, 109 insertions(+), 24 deletions(-)
diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index 9c6f5ce..522fc41 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -6,6 +6,7 @@
* Copyright (C) 1996, 1997, 1998, 1999, 2000, 03, 04 by Ralf Baechle
* Copyright (C) 1999, 2000 Silicon Graphics, Inc.
* Copyright (C) 2007 Maciej W. Rozycki
+ * Copyright (C) 2014, Imagination Technologies Ltd.
*/
#ifndef _ASM_UACCESS_H
#define _ASM_UACCESS_H
@@ -13,6 +14,7 @@
#include <linux/kernel.h>
#include <linux/errno.h>
#include <linux/thread_info.h>
+#include <asm/asm.h>
/*
* The fs value determines whether argument validity checking should be
@@ -222,11 +224,44 @@ struct __large_struct { unsigned long buf[100]; };
* Yuck. We need two variants, one for 64bit operation and one
* for 32 bit mode and old iron.
*/
+#ifndef CONFIG_EVA
+#define __get_kernel_common(val, size, ptr) __get_user_common(val, size, ptr)
+#else
+/*
+ * Kernel specific functions for EVA. We need to use normal load instructions
+ * to read data from kernel when operating in EVA mode. We use these macros to
+ * avoid redefining __get_user_asm for EVA.
+ */
+#undef _loadd
+#undef _loadw
+#undef _loadh
+#undef _loadb
+#ifdef CONFIG_32BIT
+#define _loadd _loadw
+#else
+#define _loadd(reg, addr) "ld " reg ", " addr
+#endif
+#define _loadw(reg, addr) "lw " reg ", " addr
+#define _loadh(reg, addr) "lh " reg ", " addr
+#define _loadb(reg, addr) "lb " reg ", " addr
+
+#define __get_kernel_common(val, size, ptr) \
+do { \
+ switch (size) { \
+ case 1: __get_user_asm(val, _loadb, ptr); break; \
+ case 2: __get_user_asm(val, _loadh, ptr); break; \
+ case 4: __get_user_asm(val, _loadw, ptr); break; \
+ case 8: __GET_USER_DW(val, _loadd, ptr); break; \
+ default: __get_user_unknown(); break; \
+ } \
+} while (0)
+#endif
+
#ifdef CONFIG_32BIT
#define __GET_USER_DW(val, insn, ptr) __get_user_asm_ll32(val, insn, ptr)
#endif
#ifdef CONFIG_64BIT
-#define __GET_USER_DW(val, insn, ptr) __get_user_asm(val, "ld", ptr)
+#define __GET_USER_DW(val, insn, ptr) __get_user_asm(val, insn, ptr)
#endif
extern void __get_user_unknown(void);
@@ -234,10 +269,10 @@ extern void __get_user_unknown(void);
#define __get_user_common(val, size, ptr) \
do { \
switch (size) { \
- case 1: __get_user_asm(val, "lb", ptr); break; \
- case 2: __get_user_asm(val, "lh", ptr); break; \
- case 4: __get_user_asm(val, "lw", ptr); break; \
- case 8: __GET_USER_DW(val, "lw", ptr); break; \
+ case 1: __get_user_asm(val, user_lb, ptr); break; \
+ case 2: __get_user_asm(val, user_lh, ptr); break; \
+ case 4: __get_user_asm(val, user_lw, ptr); break; \
+ case 8: __GET_USER_DW(val, user_ld, ptr); break; \
default: __get_user_unknown(); break; \
} \
} while (0)
@@ -246,8 +281,12 @@ do { \
({ \
int __gu_err; \
\
- __chk_user_ptr(ptr); \
- __get_user_common((x), size, ptr); \
+ if (segment_eq(get_fs(), get_ds())) { \
+ __get_kernel_common((x), size, ptr); \
+ } else { \
+ __chk_user_ptr(ptr); \
+ __get_user_common((x), size, ptr); \
+ } \
__gu_err; \
})
@@ -257,8 +296,12 @@ do { \
const __typeof__(*(ptr)) __user * __gu_ptr = (ptr); \
\
might_fault(); \
- if (likely(access_ok(VERIFY_READ, __gu_ptr, size))) \
- __get_user_common((x), size, __gu_ptr); \
+ if (likely(access_ok(VERIFY_READ, __gu_ptr, size))) { \
+ if (segment_eq(get_fs(), get_ds())) \
+ __get_kernel_common((x), size, __gu_ptr); \
+ else \
+ __get_user_common((x), size, __gu_ptr); \
+ } \
\
__gu_err; \
})
@@ -268,7 +311,7 @@ do { \
long __gu_tmp; \
\
__asm__ __volatile__( \
- "1: " insn " %1, %3 \n" \
+ "1: "insn("%1", "%3")" \n" \
"2: \n" \
" .insn \n" \
" .section .fixup,\"ax\" \n" \
@@ -295,8 +338,8 @@ do { \
} __gu_tmp; \
\
__asm__ __volatile__( \
- "1: " insn " %1, (%3) \n" \
- "2: " insn " %D1, 4(%3) \n" \
+ "1: " insn("%1", "(%3)")" \n" \
+ "2: " insn("%D1", "4(%3)")" \n" \
"3: \n" \
" .insn \n" \
" .section .fixup,\"ax\" \n" \
@@ -315,6 +358,40 @@ do { \
(val) = __gu_tmp.t; \
}
+#ifndef CONFIG_EVA
+#define __put_kernel_common(ptr, size) __put_user_common(ptr, size)
+#else
+/*
+ * Kernel specific functions for EVA. We need to use normal load instructions
+ * to read data from kernel when operating in EVA mode. We use these macros to
+ * avoid redefining __get_user_asm for EVA.
+ */
+#undef _stored
+#undef _storew
+#undef _storeh
+#undef _storeb
+#ifdef CONFIG_32BIT
+#define _stored _storew
+#else
+#define _stored(reg, addr) "ld " reg ", " addr
+#endif
+
+#define _storew(reg, addr) "sw " reg ", " addr
+#define _storeh(reg, addr) "sh " reg ", " addr
+#define _storeb(reg, addr) "sb " reg ", " addr
+
+#define __put_kernel_common(ptr, size) \
+do { \
+ switch (size) { \
+ case 1: __put_user_asm(_storeb, ptr); break; \
+ case 2: __put_user_asm(_storeh, ptr); break; \
+ case 4: __put_user_asm(_storew, ptr); break; \
+ case 8: __PUT_USER_DW(_stored, ptr); break; \
+ default: __put_user_unknown(); break; \
+ } \
+} while(0)
+#endif
+
/*
* Yuck. We need two variants, one for 64bit operation and one
* for 32 bit mode and old iron.
@@ -323,16 +400,16 @@ do { \
#define __PUT_USER_DW(insn, ptr) __put_user_asm_ll32(insn, ptr)
#endif
#ifdef CONFIG_64BIT
-#define __PUT_USER_DW(insn, ptr) __put_user_asm("sd", ptr)
+#define __PUT_USER_DW(insn, ptr) __put_user_asm(insn, ptr)
#endif
#define __put_user_common(ptr, size) \
do { \
switch (size) { \
- case 1: __put_user_asm("sb", ptr); break; \
- case 2: __put_user_asm("sh", ptr); break; \
- case 4: __put_user_asm("sw", ptr); break; \
- case 8: __PUT_USER_DW("sw", ptr); break; \
+ case 1: __put_user_asm(user_sb, ptr); break; \
+ case 2: __put_user_asm(user_sh, ptr); break; \
+ case 4: __put_user_asm(user_sw, ptr); break; \
+ case 8: __PUT_USER_DW(user_sd, ptr); break; \
default: __put_user_unknown(); break; \
} \
} while (0)
@@ -342,9 +419,13 @@ do { \
__typeof__(*(ptr)) __pu_val; \
int __pu_err = 0; \
\
- __chk_user_ptr(ptr); \
__pu_val = (x); \
- __put_user_common(ptr, size); \
+ if (segment_eq(get_fs(), get_ds())) { \
+ __put_kernel_common(ptr, size); \
+ } else { \
+ __chk_user_ptr(ptr); \
+ __put_user_common(ptr, size); \
+ } \
__pu_err; \
})
@@ -355,8 +436,12 @@ do { \
int __pu_err = -EFAULT; \
\
might_fault(); \
- if (likely(access_ok(VERIFY_WRITE, __pu_addr, size))) \
- __put_user_common(__pu_addr, size); \
+ if (likely(access_ok(VERIFY_WRITE, __pu_addr, size))) { \
+ if (segment_eq(get_fs(), get_ds())) \
+ __put_kernel_common(__pu_addr, size); \
+ else \
+ __put_user_common(__pu_addr, size); \
+ } \
\
__pu_err; \
})
@@ -364,7 +449,7 @@ do { \
#define __put_user_asm(insn, ptr) \
{ \
__asm__ __volatile__( \
- "1: " insn " %z2, %3 # __put_user_asm\n" \
+ "1: "insn("%z2", "%3")" # __put_user_asm \n" \
"2: \n" \
" .insn \n" \
" .section .fixup,\"ax\" \n" \
@@ -382,8 +467,8 @@ do { \
#define __put_user_asm_ll32(insn, ptr) \
{ \
__asm__ __volatile__( \
- "1: " insn " %2, (%3)# __put_user_asm_ll32 \n" \
- "2: " insn " %D2, 4(%3) \n" \
+ "1: "insn("%2", "(%3)")" # __put_user_asm_ll32 \n" \
+ "2: "insn("%D2", "4(%3)")" \n" \
"3: \n" \
" .insn \n" \
" .section .fixup,\"ax\" \n" \
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 27/58] MIPS: asm: uaccess: Rename {get,put}_user_asm macros
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
The {get,put}_user_asm functions can be used to load data from
kernel or the user address space so rename them to avoid
confusion.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/uaccess.h | 60 ++++++++++++++++++++---------------------
1 file changed, 30 insertions(+), 30 deletions(-)
diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index 522fc41..59bbebb 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -248,20 +248,20 @@ struct __large_struct { unsigned long buf[100]; };
#define __get_kernel_common(val, size, ptr) \
do { \
switch (size) { \
- case 1: __get_user_asm(val, _loadb, ptr); break; \
- case 2: __get_user_asm(val, _loadh, ptr); break; \
- case 4: __get_user_asm(val, _loadw, ptr); break; \
- case 8: __GET_USER_DW(val, _loadd, ptr); break; \
+ case 1: __get_data_asm(val, _loadb, ptr); break; \
+ case 2: __get_data_asm(val, _loadh, ptr); break; \
+ case 4: __get_data_asm(val, _loadw, ptr); break; \
+ case 8: __GET_DW(val, _loadd, ptr); break; \
default: __get_user_unknown(); break; \
} \
} while (0)
#endif
#ifdef CONFIG_32BIT
-#define __GET_USER_DW(val, insn, ptr) __get_user_asm_ll32(val, insn, ptr)
+#define __GET_DW(val, insn, ptr) __get_data_asm_ll32(val, insn, ptr)
#endif
#ifdef CONFIG_64BIT
-#define __GET_USER_DW(val, insn, ptr) __get_user_asm(val, insn, ptr)
+#define __GET_DW(val, insn, ptr) __get_data_asm(val, insn, ptr)
#endif
extern void __get_user_unknown(void);
@@ -269,10 +269,10 @@ extern void __get_user_unknown(void);
#define __get_user_common(val, size, ptr) \
do { \
switch (size) { \
- case 1: __get_user_asm(val, user_lb, ptr); break; \
- case 2: __get_user_asm(val, user_lh, ptr); break; \
- case 4: __get_user_asm(val, user_lw, ptr); break; \
- case 8: __GET_USER_DW(val, user_ld, ptr); break; \
+ case 1: __get_data_asm(val, user_lb, ptr); break; \
+ case 2: __get_data_asm(val, user_lh, ptr); break; \
+ case 4: __get_data_asm(val, user_lw, ptr); break; \
+ case 8: __GET_DW(val, user_ld, ptr); break; \
default: __get_user_unknown(); break; \
} \
} while (0)
@@ -306,7 +306,7 @@ do { \
__gu_err; \
})
-#define __get_user_asm(val, insn, addr) \
+#define __get_data_asm(val, insn, addr) \
{ \
long __gu_tmp; \
\
@@ -330,7 +330,7 @@ do { \
/*
* Get a long long 64 using 32 bit registers.
*/
-#define __get_user_asm_ll32(val, insn, addr) \
+#define __get_data_asm_ll32(val, insn, addr) \
{ \
union { \
unsigned long long l; \
@@ -364,7 +364,7 @@ do { \
/*
* Kernel specific functions for EVA. We need to use normal load instructions
* to read data from kernel when operating in EVA mode. We use these macros to
- * avoid redefining __get_user_asm for EVA.
+ * avoid redefining __get_data_asm for EVA.
*/
#undef _stored
#undef _storew
@@ -383,10 +383,10 @@ do { \
#define __put_kernel_common(ptr, size) \
do { \
switch (size) { \
- case 1: __put_user_asm(_storeb, ptr); break; \
- case 2: __put_user_asm(_storeh, ptr); break; \
- case 4: __put_user_asm(_storew, ptr); break; \
- case 8: __PUT_USER_DW(_stored, ptr); break; \
+ case 1: __put_data_asm(_storeb, ptr); break; \
+ case 2: __put_data_asm(_storeh, ptr); break; \
+ case 4: __put_data_asm(_storew, ptr); break; \
+ case 8: __PUT_DW(_stored, ptr); break; \
default: __put_user_unknown(); break; \
} \
} while(0)
@@ -397,19 +397,19 @@ do { \
* for 32 bit mode and old iron.
*/
#ifdef CONFIG_32BIT
-#define __PUT_USER_DW(insn, ptr) __put_user_asm_ll32(insn, ptr)
+#define __PUT_DW(insn, ptr) __put_data_asm_ll32(insn, ptr)
#endif
#ifdef CONFIG_64BIT
-#define __PUT_USER_DW(insn, ptr) __put_user_asm(insn, ptr)
+#define __PUT_DW(insn, ptr) __put_data_asm(insn, ptr)
#endif
#define __put_user_common(ptr, size) \
do { \
switch (size) { \
- case 1: __put_user_asm(user_sb, ptr); break; \
- case 2: __put_user_asm(user_sh, ptr); break; \
- case 4: __put_user_asm(user_sw, ptr); break; \
- case 8: __PUT_USER_DW(user_sd, ptr); break; \
+ case 1: __put_data_asm(user_sb, ptr); break; \
+ case 2: __put_data_asm(user_sh, ptr); break; \
+ case 4: __put_data_asm(user_sw, ptr); break; \
+ case 8: __PUT_DW(user_sd, ptr); break; \
default: __put_user_unknown(); break; \
} \
} while (0)
@@ -446,10 +446,10 @@ do { \
__pu_err; \
})
-#define __put_user_asm(insn, ptr) \
+#define __put_data_asm(insn, ptr) \
{ \
__asm__ __volatile__( \
- "1: "insn("%z2", "%3")" # __put_user_asm \n" \
+ "1: "insn("%z2", "%3")" # __put_data_asm \n" \
"2: \n" \
" .insn \n" \
" .section .fixup,\"ax\" \n" \
@@ -464,10 +464,10 @@ do { \
"i" (-EFAULT)); \
}
-#define __put_user_asm_ll32(insn, ptr) \
+#define __put_data_asm_ll32(insn, ptr) \
{ \
__asm__ __volatile__( \
- "1: "insn("%2", "(%3)")" # __put_user_asm_ll32 \n" \
+ "1: "insn("%2", "(%3)")" # __put_data_asm_ll32 \n" \
"2: "insn("%D2", "4(%3)")" \n" \
"3: \n" \
" .insn \n" \
@@ -593,7 +593,7 @@ extern void __get_user_unaligned_unknown(void);
#define __get_user_unaligned_common(val, size, ptr) \
do { \
switch (size) { \
- case 1: __get_user_asm(val, "lb", ptr); break; \
+ case 1: __get_data_asm(val, "lb", ptr); break; \
case 2: __get_user_unaligned_asm(val, "ulh", ptr); break; \
case 4: __get_user_unaligned_asm(val, "ulw", ptr); break; \
case 8: __GET_USER_UNALIGNED_DW(val, ptr); break; \
@@ -620,7 +620,7 @@ do { \
__gu_err; \
})
-#define __get_user_unaligned_asm(val, insn, addr) \
+#define __get_data_unaligned_asm(val, insn, addr) \
{ \
long __gu_tmp; \
\
@@ -686,7 +686,7 @@ do { \
#define __put_user_unaligned_common(ptr, size) \
do { \
switch (size) { \
- case 1: __put_user_asm("sb", ptr); break; \
+ case 1: __put_data_asm("sb", ptr); break; \
case 2: __put_user_unaligned_asm("ush", ptr); break; \
case 4: __put_user_unaligned_asm("usw", ptr); break; \
case 8: __PUT_USER_UNALIGNED_DW(ptr); break; \
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 27/58] MIPS: asm: uaccess: Rename {get,put}_user_asm macros
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
The {get,put}_user_asm functions can be used to load data from
kernel or the user address space so rename them to avoid
confusion.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/uaccess.h | 60 ++++++++++++++++++++---------------------
1 file changed, 30 insertions(+), 30 deletions(-)
diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index 522fc41..59bbebb 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -248,20 +248,20 @@ struct __large_struct { unsigned long buf[100]; };
#define __get_kernel_common(val, size, ptr) \
do { \
switch (size) { \
- case 1: __get_user_asm(val, _loadb, ptr); break; \
- case 2: __get_user_asm(val, _loadh, ptr); break; \
- case 4: __get_user_asm(val, _loadw, ptr); break; \
- case 8: __GET_USER_DW(val, _loadd, ptr); break; \
+ case 1: __get_data_asm(val, _loadb, ptr); break; \
+ case 2: __get_data_asm(val, _loadh, ptr); break; \
+ case 4: __get_data_asm(val, _loadw, ptr); break; \
+ case 8: __GET_DW(val, _loadd, ptr); break; \
default: __get_user_unknown(); break; \
} \
} while (0)
#endif
#ifdef CONFIG_32BIT
-#define __GET_USER_DW(val, insn, ptr) __get_user_asm_ll32(val, insn, ptr)
+#define __GET_DW(val, insn, ptr) __get_data_asm_ll32(val, insn, ptr)
#endif
#ifdef CONFIG_64BIT
-#define __GET_USER_DW(val, insn, ptr) __get_user_asm(val, insn, ptr)
+#define __GET_DW(val, insn, ptr) __get_data_asm(val, insn, ptr)
#endif
extern void __get_user_unknown(void);
@@ -269,10 +269,10 @@ extern void __get_user_unknown(void);
#define __get_user_common(val, size, ptr) \
do { \
switch (size) { \
- case 1: __get_user_asm(val, user_lb, ptr); break; \
- case 2: __get_user_asm(val, user_lh, ptr); break; \
- case 4: __get_user_asm(val, user_lw, ptr); break; \
- case 8: __GET_USER_DW(val, user_ld, ptr); break; \
+ case 1: __get_data_asm(val, user_lb, ptr); break; \
+ case 2: __get_data_asm(val, user_lh, ptr); break; \
+ case 4: __get_data_asm(val, user_lw, ptr); break; \
+ case 8: __GET_DW(val, user_ld, ptr); break; \
default: __get_user_unknown(); break; \
} \
} while (0)
@@ -306,7 +306,7 @@ do { \
__gu_err; \
})
-#define __get_user_asm(val, insn, addr) \
+#define __get_data_asm(val, insn, addr) \
{ \
long __gu_tmp; \
\
@@ -330,7 +330,7 @@ do { \
/*
* Get a long long 64 using 32 bit registers.
*/
-#define __get_user_asm_ll32(val, insn, addr) \
+#define __get_data_asm_ll32(val, insn, addr) \
{ \
union { \
unsigned long long l; \
@@ -364,7 +364,7 @@ do { \
/*
* Kernel specific functions for EVA. We need to use normal load instructions
* to read data from kernel when operating in EVA mode. We use these macros to
- * avoid redefining __get_user_asm for EVA.
+ * avoid redefining __get_data_asm for EVA.
*/
#undef _stored
#undef _storew
@@ -383,10 +383,10 @@ do { \
#define __put_kernel_common(ptr, size) \
do { \
switch (size) { \
- case 1: __put_user_asm(_storeb, ptr); break; \
- case 2: __put_user_asm(_storeh, ptr); break; \
- case 4: __put_user_asm(_storew, ptr); break; \
- case 8: __PUT_USER_DW(_stored, ptr); break; \
+ case 1: __put_data_asm(_storeb, ptr); break; \
+ case 2: __put_data_asm(_storeh, ptr); break; \
+ case 4: __put_data_asm(_storew, ptr); break; \
+ case 8: __PUT_DW(_stored, ptr); break; \
default: __put_user_unknown(); break; \
} \
} while(0)
@@ -397,19 +397,19 @@ do { \
* for 32 bit mode and old iron.
*/
#ifdef CONFIG_32BIT
-#define __PUT_USER_DW(insn, ptr) __put_user_asm_ll32(insn, ptr)
+#define __PUT_DW(insn, ptr) __put_data_asm_ll32(insn, ptr)
#endif
#ifdef CONFIG_64BIT
-#define __PUT_USER_DW(insn, ptr) __put_user_asm(insn, ptr)
+#define __PUT_DW(insn, ptr) __put_data_asm(insn, ptr)
#endif
#define __put_user_common(ptr, size) \
do { \
switch (size) { \
- case 1: __put_user_asm(user_sb, ptr); break; \
- case 2: __put_user_asm(user_sh, ptr); break; \
- case 4: __put_user_asm(user_sw, ptr); break; \
- case 8: __PUT_USER_DW(user_sd, ptr); break; \
+ case 1: __put_data_asm(user_sb, ptr); break; \
+ case 2: __put_data_asm(user_sh, ptr); break; \
+ case 4: __put_data_asm(user_sw, ptr); break; \
+ case 8: __PUT_DW(user_sd, ptr); break; \
default: __put_user_unknown(); break; \
} \
} while (0)
@@ -446,10 +446,10 @@ do { \
__pu_err; \
})
-#define __put_user_asm(insn, ptr) \
+#define __put_data_asm(insn, ptr) \
{ \
__asm__ __volatile__( \
- "1: "insn("%z2", "%3")" # __put_user_asm \n" \
+ "1: "insn("%z2", "%3")" # __put_data_asm \n" \
"2: \n" \
" .insn \n" \
" .section .fixup,\"ax\" \n" \
@@ -464,10 +464,10 @@ do { \
"i" (-EFAULT)); \
}
-#define __put_user_asm_ll32(insn, ptr) \
+#define __put_data_asm_ll32(insn, ptr) \
{ \
__asm__ __volatile__( \
- "1: "insn("%2", "(%3)")" # __put_user_asm_ll32 \n" \
+ "1: "insn("%2", "(%3)")" # __put_data_asm_ll32 \n" \
"2: "insn("%D2", "4(%3)")" \n" \
"3: \n" \
" .insn \n" \
@@ -593,7 +593,7 @@ extern void __get_user_unaligned_unknown(void);
#define __get_user_unaligned_common(val, size, ptr) \
do { \
switch (size) { \
- case 1: __get_user_asm(val, "lb", ptr); break; \
+ case 1: __get_data_asm(val, "lb", ptr); break; \
case 2: __get_user_unaligned_asm(val, "ulh", ptr); break; \
case 4: __get_user_unaligned_asm(val, "ulw", ptr); break; \
case 8: __GET_USER_UNALIGNED_DW(val, ptr); break; \
@@ -620,7 +620,7 @@ do { \
__gu_err; \
})
-#define __get_user_unaligned_asm(val, insn, addr) \
+#define __get_data_unaligned_asm(val, insn, addr) \
{ \
long __gu_tmp; \
\
@@ -686,7 +686,7 @@ do { \
#define __put_user_unaligned_common(ptr, size) \
do { \
switch (size) { \
- case 1: __put_user_asm("sb", ptr); break; \
+ case 1: __put_data_asm("sb", ptr); break; \
case 2: __put_user_unaligned_asm("ush", ptr); break; \
case 4: __put_user_unaligned_asm("usw", ptr); break; \
case 8: __PUT_USER_UNALIGNED_DW(ptr); break; \
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 28/58] MIPS: asm: uaccess: Add EVA support to copy_{in, to,from}_user
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Use the EVA specific functions from memcpy.S to perform
userspace operations. When get_fs() == get_ds() the usual load/store
instructions are used because the destination address is located in
the kernel address space region. Otherwise, the EVA specifc load/store
instructions are used which will go through th TLB to perform the virtual
to physical translation for the userspace address.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/uaccess.h | 191 +++++++++++++++++++++++++++++++++++-----
1 file changed, 171 insertions(+), 20 deletions(-)
diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index 59bbebb..fe72837 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -781,6 +781,7 @@ extern void __put_user_unaligned_unknown(void);
extern size_t __copy_user(void *__to, const void *__from, size_t __n);
+#ifndef CONFIG_EVA
#define __invoke_copy_to_user(to, from, n) \
({ \
register void __user *__cu_to_r __asm__("$4"); \
@@ -799,6 +800,11 @@ extern size_t __copy_user(void *__to, const void *__from, size_t __n);
__cu_len_r; \
})
+#define __invoke_copy_to_kernel(to, from, n) \
+ __invoke_copy_to_user(to, from, n)
+
+#endif
+
/*
* __copy_to_user: - Copy a block of data into user space, with less checking.
* @to: Destination address, in user space.
@@ -823,7 +829,12 @@ extern size_t __copy_user(void *__to, const void *__from, size_t __n);
__cu_from = (from); \
__cu_len = (n); \
might_fault(); \
- __cu_len = __invoke_copy_to_user(__cu_to, __cu_from, __cu_len); \
+ if (segment_eq(get_fs(), get_ds())) \
+ __cu_len = __invoke_copy_to_kernel(__cu_to, __cu_from, \
+ __cu_len); \
+ else \
+ __cu_len = __invoke_copy_to_user(__cu_to, __cu_from, \
+ __cu_len); \
__cu_len; \
})
@@ -838,7 +849,12 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
__cu_to = (to); \
__cu_from = (from); \
__cu_len = (n); \
- __cu_len = __invoke_copy_to_user(__cu_to, __cu_from, __cu_len); \
+ if (segment_eq(get_fs(), get_ds())) \
+ __cu_len = __invoke_copy_to_kernel(__cu_to, __cu_from, \
+ __cu_len); \
+ else \
+ __cu_len = __invoke_copy_to_user(__cu_to, __cu_from, \
+ __cu_len); \
__cu_len; \
})
@@ -851,8 +867,14 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
__cu_to = (to); \
__cu_from = (from); \
__cu_len = (n); \
- __cu_len = __invoke_copy_from_user_inatomic(__cu_to, __cu_from, \
- __cu_len); \
+ if (segment_eq(get_fs(), get_ds())) \
+ __cu_len = __invoke_copy_from_kernel_inatomic(__cu_to, \
+ __cu_from,\
+ __cu_len);\
+ else \
+ __cu_len = __invoke_copy_from_user_inatomic(__cu_to, \
+ __cu_from, \
+ __cu_len); \
__cu_len; \
})
@@ -878,14 +900,23 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
__cu_to = (to); \
__cu_from = (from); \
__cu_len = (n); \
- if (access_ok(VERIFY_WRITE, __cu_to, __cu_len)) { \
- might_fault(); \
- __cu_len = __invoke_copy_to_user(__cu_to, __cu_from, \
- __cu_len); \
+ if (segment_eq(get_fs(), get_ds())) { \
+ __cu_len = __invoke_copy_to_kernel(__cu_to, \
+ __cu_from, \
+ __cu_len); \
+ } else { \
+ if (access_ok(VERIFY_WRITE, __cu_to, __cu_len)) { \
+ might_fault(); \
+ __cu_len = __invoke_copy_to_user(__cu_to, \
+ __cu_from, \
+ __cu_len); \
+ } \
} \
__cu_len; \
})
+#ifndef CONFIG_EVA
+
#define __invoke_copy_from_user(to, from, n) \
({ \
register void *__cu_to_r __asm__("$4"); \
@@ -909,6 +940,17 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
__cu_len_r; \
})
+#define __invoke_copy_from_kernel(to, from, n) \
+ __invoke_copy_from_user(to, from, n)
+
+/* For userland <-> userland operations */
+#define ___invoke_copy_in_user(to, from, n) \
+ __invoke_copy_from_user(to, from, n)
+
+/* For kernel <-> kernel operations */
+#define ___invoke_copy_in_kernel(to, from, n) \
+ __invoke_copy_from_user(to, from, n)
+
#define __invoke_copy_from_user_inatomic(to, from, n) \
({ \
register void *__cu_to_r __asm__("$4"); \
@@ -932,6 +974,97 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
__cu_len_r; \
})
+#define __invoke_copy_from_kernel_inatomic(to, from, n) \
+ __invoke_copy_from_user_inatomic(to, from, n) \
+
+#else
+
+/* EVA specific functions */
+
+extern size_t __copy_user_inatomic_eva(void *__to, const void *__from,
+ size_t __n);
+extern size_t __copy_from_user_eva(void *__to, const void *__from,
+ size_t __n);
+extern size_t __copy_to_user_eva(void *__to, const void *__from,
+ size_t __n);
+extern size_t __copy_in_user_eva(void *__to, const void *__from, size_t __n);
+
+#define __invoke_copy_from_user_eva_generic(to, from, n, func_ptr) \
+({ \
+ register void *__cu_to_r __asm__("$4"); \
+ register const void __user *__cu_from_r __asm__("$5"); \
+ register long __cu_len_r __asm__("$6"); \
+ \
+ __cu_to_r = (to); \
+ __cu_from_r = (from); \
+ __cu_len_r = (n); \
+ __asm__ __volatile__( \
+ ".set\tnoreorder\n\t" \
+ __MODULE_JAL(func_ptr) \
+ ".set\tnoat\n\t" \
+ __UA_ADDU "\t$1, %1, %2\n\t" \
+ ".set\tat\n\t" \
+ ".set\treorder" \
+ : "+r" (__cu_to_r), "+r" (__cu_from_r), "+r" (__cu_len_r) \
+ : \
+ : "$8", "$9", "$10", "$11", "$12", "$14", "$15", "$24", "$31", \
+ DADDI_SCRATCH, "memory"); \
+ __cu_len_r; \
+})
+
+#define __invoke_copy_to_user_eva_generic(to, from, n, func_ptr) \
+({ \
+ register void *__cu_to_r __asm__("$4"); \
+ register const void __user *__cu_from_r __asm__("$5"); \
+ register long __cu_len_r __asm__("$6"); \
+ \
+ __cu_to_r = (to); \
+ __cu_from_r = (from); \
+ __cu_len_r = (n); \
+ __asm__ __volatile__( \
+ __MODULE_JAL(func_ptr) \
+ : "+r" (__cu_to_r), "+r" (__cu_from_r), "+r" (__cu_len_r) \
+ : \
+ : "$8", "$9", "$10", "$11", "$12", "$14", "$15", "$24", "$31", \
+ DADDI_SCRATCH, "memory"); \
+ __cu_len_r; \
+})
+
+/*
+ * Source or destination address is in userland. We need to go through
+ * the TLB
+ */
+#define __invoke_copy_from_user(to, from, n) \
+ __invoke_copy_from_user_eva_generic(to, from, n, __copy_from_user_eva)
+
+#define __invoke_copy_from_user_inatomic(to, from, n) \
+ __invoke_copy_from_user_eva_generic(to, from, n, \
+ __copy_user_inatomic_eva)
+
+#define __invoke_copy_to_user(to, from, n) \
+ __invoke_copy_to_user_eva_generic(to, from, n, __copy_to_user_eva)
+
+#define ___invoke_copy_in_user(to, from, n) \
+ __invoke_copy_from_user_eva_generic(to, from, n, __copy_in_user_eva)
+
+/*
+ * Source or destination address in the kernel. We are not going through
+ * the TLB
+ */
+#define __invoke_copy_from_kernel(to, from, n) \
+ __invoke_copy_from_user_eva_generic(to, from, n, __copy_user)
+
+#define __invoke_copy_from_kernel_inatomic(to, from, n) \
+ __invoke_copy_from_user_eva_generic(to, from, n, __copy_user_inatomic)
+
+#define __invoke_copy_to_kernel(to, from, n) \
+ __invoke_copy_to_user_eva_generic(to, from, n, __copy_user)
+
+#define ___invoke_copy_in_kernel(to, from, n) \
+ __invoke_copy_from_user_eva_generic(to, from, n, __copy_user)
+
+#endif /* CONFIG_EVA */
+
/*
* __copy_from_user: - Copy a block of data from user space, with less checking.
* @to: Destination address, in kernel space.
@@ -989,10 +1122,17 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
__cu_to = (to); \
__cu_from = (from); \
__cu_len = (n); \
- if (access_ok(VERIFY_READ, __cu_from, __cu_len)) { \
- might_fault(); \
- __cu_len = __invoke_copy_from_user(__cu_to, __cu_from, \
- __cu_len); \
+ if (segment_eq(get_fs(), get_ds())) { \
+ __cu_len = __invoke_copy_from_kernel(__cu_to, \
+ __cu_from, \
+ __cu_len); \
+ } else { \
+ if (access_ok(VERIFY_READ, __cu_from, __cu_len)) { \
+ might_fault(); \
+ __cu_len = __invoke_copy_from_user(__cu_to, \
+ __cu_from, \
+ __cu_len); \
+ } \
} \
__cu_len; \
})
@@ -1006,9 +1146,14 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
__cu_to = (to); \
__cu_from = (from); \
__cu_len = (n); \
- might_fault(); \
- __cu_len = __invoke_copy_from_user(__cu_to, __cu_from, \
- __cu_len); \
+ if (segment_eq(get_fs(), get_ds())) { \
+ __cu_len = ___invoke_copy_in_kernel(__cu_to, __cu_from, \
+ __cu_len); \
+ } else { \
+ might_fault(); \
+ __cu_len = ___invoke_copy_in_user(__cu_to, __cu_from, \
+ __cu_len); \
+ } \
__cu_len; \
})
@@ -1021,11 +1166,17 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
__cu_to = (to); \
__cu_from = (from); \
__cu_len = (n); \
- if (likely(access_ok(VERIFY_READ, __cu_from, __cu_len) && \
- access_ok(VERIFY_WRITE, __cu_to, __cu_len))) { \
- might_fault(); \
- __cu_len = __invoke_copy_from_user(__cu_to, __cu_from, \
- __cu_len); \
+ if (segment_eq(get_fs(), get_ds())) { \
+ __cu_len = ___invoke_copy_in_kernel(__cu_to,__cu_from, \
+ __cu_len); \
+ } else { \
+ if (likely(access_ok(VERIFY_READ, __cu_from, __cu_len) &&\
+ access_ok(VERIFY_WRITE, __cu_to, __cu_len))) {\
+ might_fault(); \
+ __cu_len = ___invoke_copy_in_user(__cu_to, \
+ __cu_from, \
+ __cu_len); \
+ } \
} \
__cu_len; \
})
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 28/58] MIPS: asm: uaccess: Add EVA support to copy_{in, to,from}_user
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Use the EVA specific functions from memcpy.S to perform
userspace operations. When get_fs() == get_ds() the usual load/store
instructions are used because the destination address is located in
the kernel address space region. Otherwise, the EVA specifc load/store
instructions are used which will go through th TLB to perform the virtual
to physical translation for the userspace address.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/uaccess.h | 191 +++++++++++++++++++++++++++++++++++-----
1 file changed, 171 insertions(+), 20 deletions(-)
diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index 59bbebb..fe72837 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -781,6 +781,7 @@ extern void __put_user_unaligned_unknown(void);
extern size_t __copy_user(void *__to, const void *__from, size_t __n);
+#ifndef CONFIG_EVA
#define __invoke_copy_to_user(to, from, n) \
({ \
register void __user *__cu_to_r __asm__("$4"); \
@@ -799,6 +800,11 @@ extern size_t __copy_user(void *__to, const void *__from, size_t __n);
__cu_len_r; \
})
+#define __invoke_copy_to_kernel(to, from, n) \
+ __invoke_copy_to_user(to, from, n)
+
+#endif
+
/*
* __copy_to_user: - Copy a block of data into user space, with less checking.
* @to: Destination address, in user space.
@@ -823,7 +829,12 @@ extern size_t __copy_user(void *__to, const void *__from, size_t __n);
__cu_from = (from); \
__cu_len = (n); \
might_fault(); \
- __cu_len = __invoke_copy_to_user(__cu_to, __cu_from, __cu_len); \
+ if (segment_eq(get_fs(), get_ds())) \
+ __cu_len = __invoke_copy_to_kernel(__cu_to, __cu_from, \
+ __cu_len); \
+ else \
+ __cu_len = __invoke_copy_to_user(__cu_to, __cu_from, \
+ __cu_len); \
__cu_len; \
})
@@ -838,7 +849,12 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
__cu_to = (to); \
__cu_from = (from); \
__cu_len = (n); \
- __cu_len = __invoke_copy_to_user(__cu_to, __cu_from, __cu_len); \
+ if (segment_eq(get_fs(), get_ds())) \
+ __cu_len = __invoke_copy_to_kernel(__cu_to, __cu_from, \
+ __cu_len); \
+ else \
+ __cu_len = __invoke_copy_to_user(__cu_to, __cu_from, \
+ __cu_len); \
__cu_len; \
})
@@ -851,8 +867,14 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
__cu_to = (to); \
__cu_from = (from); \
__cu_len = (n); \
- __cu_len = __invoke_copy_from_user_inatomic(__cu_to, __cu_from, \
- __cu_len); \
+ if (segment_eq(get_fs(), get_ds())) \
+ __cu_len = __invoke_copy_from_kernel_inatomic(__cu_to, \
+ __cu_from,\
+ __cu_len);\
+ else \
+ __cu_len = __invoke_copy_from_user_inatomic(__cu_to, \
+ __cu_from, \
+ __cu_len); \
__cu_len; \
})
@@ -878,14 +900,23 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
__cu_to = (to); \
__cu_from = (from); \
__cu_len = (n); \
- if (access_ok(VERIFY_WRITE, __cu_to, __cu_len)) { \
- might_fault(); \
- __cu_len = __invoke_copy_to_user(__cu_to, __cu_from, \
- __cu_len); \
+ if (segment_eq(get_fs(), get_ds())) { \
+ __cu_len = __invoke_copy_to_kernel(__cu_to, \
+ __cu_from, \
+ __cu_len); \
+ } else { \
+ if (access_ok(VERIFY_WRITE, __cu_to, __cu_len)) { \
+ might_fault(); \
+ __cu_len = __invoke_copy_to_user(__cu_to, \
+ __cu_from, \
+ __cu_len); \
+ } \
} \
__cu_len; \
})
+#ifndef CONFIG_EVA
+
#define __invoke_copy_from_user(to, from, n) \
({ \
register void *__cu_to_r __asm__("$4"); \
@@ -909,6 +940,17 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
__cu_len_r; \
})
+#define __invoke_copy_from_kernel(to, from, n) \
+ __invoke_copy_from_user(to, from, n)
+
+/* For userland <-> userland operations */
+#define ___invoke_copy_in_user(to, from, n) \
+ __invoke_copy_from_user(to, from, n)
+
+/* For kernel <-> kernel operations */
+#define ___invoke_copy_in_kernel(to, from, n) \
+ __invoke_copy_from_user(to, from, n)
+
#define __invoke_copy_from_user_inatomic(to, from, n) \
({ \
register void *__cu_to_r __asm__("$4"); \
@@ -932,6 +974,97 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
__cu_len_r; \
})
+#define __invoke_copy_from_kernel_inatomic(to, from, n) \
+ __invoke_copy_from_user_inatomic(to, from, n) \
+
+#else
+
+/* EVA specific functions */
+
+extern size_t __copy_user_inatomic_eva(void *__to, const void *__from,
+ size_t __n);
+extern size_t __copy_from_user_eva(void *__to, const void *__from,
+ size_t __n);
+extern size_t __copy_to_user_eva(void *__to, const void *__from,
+ size_t __n);
+extern size_t __copy_in_user_eva(void *__to, const void *__from, size_t __n);
+
+#define __invoke_copy_from_user_eva_generic(to, from, n, func_ptr) \
+({ \
+ register void *__cu_to_r __asm__("$4"); \
+ register const void __user *__cu_from_r __asm__("$5"); \
+ register long __cu_len_r __asm__("$6"); \
+ \
+ __cu_to_r = (to); \
+ __cu_from_r = (from); \
+ __cu_len_r = (n); \
+ __asm__ __volatile__( \
+ ".set\tnoreorder\n\t" \
+ __MODULE_JAL(func_ptr) \
+ ".set\tnoat\n\t" \
+ __UA_ADDU "\t$1, %1, %2\n\t" \
+ ".set\tat\n\t" \
+ ".set\treorder" \
+ : "+r" (__cu_to_r), "+r" (__cu_from_r), "+r" (__cu_len_r) \
+ : \
+ : "$8", "$9", "$10", "$11", "$12", "$14", "$15", "$24", "$31", \
+ DADDI_SCRATCH, "memory"); \
+ __cu_len_r; \
+})
+
+#define __invoke_copy_to_user_eva_generic(to, from, n, func_ptr) \
+({ \
+ register void *__cu_to_r __asm__("$4"); \
+ register const void __user *__cu_from_r __asm__("$5"); \
+ register long __cu_len_r __asm__("$6"); \
+ \
+ __cu_to_r = (to); \
+ __cu_from_r = (from); \
+ __cu_len_r = (n); \
+ __asm__ __volatile__( \
+ __MODULE_JAL(func_ptr) \
+ : "+r" (__cu_to_r), "+r" (__cu_from_r), "+r" (__cu_len_r) \
+ : \
+ : "$8", "$9", "$10", "$11", "$12", "$14", "$15", "$24", "$31", \
+ DADDI_SCRATCH, "memory"); \
+ __cu_len_r; \
+})
+
+/*
+ * Source or destination address is in userland. We need to go through
+ * the TLB
+ */
+#define __invoke_copy_from_user(to, from, n) \
+ __invoke_copy_from_user_eva_generic(to, from, n, __copy_from_user_eva)
+
+#define __invoke_copy_from_user_inatomic(to, from, n) \
+ __invoke_copy_from_user_eva_generic(to, from, n, \
+ __copy_user_inatomic_eva)
+
+#define __invoke_copy_to_user(to, from, n) \
+ __invoke_copy_to_user_eva_generic(to, from, n, __copy_to_user_eva)
+
+#define ___invoke_copy_in_user(to, from, n) \
+ __invoke_copy_from_user_eva_generic(to, from, n, __copy_in_user_eva)
+
+/*
+ * Source or destination address in the kernel. We are not going through
+ * the TLB
+ */
+#define __invoke_copy_from_kernel(to, from, n) \
+ __invoke_copy_from_user_eva_generic(to, from, n, __copy_user)
+
+#define __invoke_copy_from_kernel_inatomic(to, from, n) \
+ __invoke_copy_from_user_eva_generic(to, from, n, __copy_user_inatomic)
+
+#define __invoke_copy_to_kernel(to, from, n) \
+ __invoke_copy_to_user_eva_generic(to, from, n, __copy_user)
+
+#define ___invoke_copy_in_kernel(to, from, n) \
+ __invoke_copy_from_user_eva_generic(to, from, n, __copy_user)
+
+#endif /* CONFIG_EVA */
+
/*
* __copy_from_user: - Copy a block of data from user space, with less checking.
* @to: Destination address, in kernel space.
@@ -989,10 +1122,17 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
__cu_to = (to); \
__cu_from = (from); \
__cu_len = (n); \
- if (access_ok(VERIFY_READ, __cu_from, __cu_len)) { \
- might_fault(); \
- __cu_len = __invoke_copy_from_user(__cu_to, __cu_from, \
- __cu_len); \
+ if (segment_eq(get_fs(), get_ds())) { \
+ __cu_len = __invoke_copy_from_kernel(__cu_to, \
+ __cu_from, \
+ __cu_len); \
+ } else { \
+ if (access_ok(VERIFY_READ, __cu_from, __cu_len)) { \
+ might_fault(); \
+ __cu_len = __invoke_copy_from_user(__cu_to, \
+ __cu_from, \
+ __cu_len); \
+ } \
} \
__cu_len; \
})
@@ -1006,9 +1146,14 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
__cu_to = (to); \
__cu_from = (from); \
__cu_len = (n); \
- might_fault(); \
- __cu_len = __invoke_copy_from_user(__cu_to, __cu_from, \
- __cu_len); \
+ if (segment_eq(get_fs(), get_ds())) { \
+ __cu_len = ___invoke_copy_in_kernel(__cu_to, __cu_from, \
+ __cu_len); \
+ } else { \
+ might_fault(); \
+ __cu_len = ___invoke_copy_in_user(__cu_to, __cu_from, \
+ __cu_len); \
+ } \
__cu_len; \
})
@@ -1021,11 +1166,17 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
__cu_to = (to); \
__cu_from = (from); \
__cu_len = (n); \
- if (likely(access_ok(VERIFY_READ, __cu_from, __cu_len) && \
- access_ok(VERIFY_WRITE, __cu_to, __cu_len))) { \
- might_fault(); \
- __cu_len = __invoke_copy_from_user(__cu_to, __cu_from, \
- __cu_len); \
+ if (segment_eq(get_fs(), get_ds())) { \
+ __cu_len = ___invoke_copy_in_kernel(__cu_to,__cu_from, \
+ __cu_len); \
+ } else { \
+ if (likely(access_ok(VERIFY_READ, __cu_from, __cu_len) &&\
+ access_ok(VERIFY_WRITE, __cu_to, __cu_len))) {\
+ might_fault(); \
+ __cu_len = ___invoke_copy_in_user(__cu_to, \
+ __cu_from, \
+ __cu_len); \
+ } \
} \
__cu_len; \
})
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 29/58] MIPS: asm: uaccess: Add EVA support for str*_user operations
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
The str*_user functions are used to securely access NULL terminated
strings from userland. Therefore, it's necessary to use the appropriate
EVA function. However, if the string is in kernel space, then the normal
instructions are being used to access it. The __str*_kernel_asm and
__str*_user_asm symbols are the same for non-EVA mode so there is no
functional change for the non-EVA kernels.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/uaccess.h | 172 +++++++++++++++++++++++++++-------------
1 file changed, 119 insertions(+), 53 deletions(-)
diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index fe72837..da52765 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -1246,16 +1246,28 @@ __strncpy_from_user(char *__to, const char __user *__from, long __len)
{
long res;
- might_fault();
- __asm__ __volatile__(
- "move\t$4, %1\n\t"
- "move\t$5, %2\n\t"
- "move\t$6, %3\n\t"
- __MODULE_JAL(__strncpy_from_user_nocheck_asm)
- "move\t%0, $2"
- : "=r" (res)
- : "r" (__to), "r" (__from), "r" (__len)
- : "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+ if (segment_eq(get_fs(), get_ds())) {
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ "move\t$5, %2\n\t"
+ "move\t$6, %3\n\t"
+ __MODULE_JAL(__strncpy_from_kernel_nocheck_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (__to), "r" (__from), "r" (__len)
+ : "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+ } else {
+ might_fault();
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ "move\t$5, %2\n\t"
+ "move\t$6, %3\n\t"
+ __MODULE_JAL(__strncpy_from_user_nocheck_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (__to), "r" (__from), "r" (__len)
+ : "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+ }
return res;
}
@@ -1283,16 +1295,28 @@ strncpy_from_user(char *__to, const char __user *__from, long __len)
{
long res;
- might_fault();
- __asm__ __volatile__(
- "move\t$4, %1\n\t"
- "move\t$5, %2\n\t"
- "move\t$6, %3\n\t"
- __MODULE_JAL(__strncpy_from_user_asm)
- "move\t%0, $2"
- : "=r" (res)
- : "r" (__to), "r" (__from), "r" (__len)
- : "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+ if (segment_eq(get_fs(), get_ds())) {
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ "move\t$5, %2\n\t"
+ "move\t$6, %3\n\t"
+ __MODULE_JAL(__strncpy_from_kernel_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (__to), "r" (__from), "r" (__len)
+ : "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+ } else {
+ might_fault();
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ "move\t$5, %2\n\t"
+ "move\t$6, %3\n\t"
+ __MODULE_JAL(__strncpy_from_user_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (__to), "r" (__from), "r" (__len)
+ : "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+ }
return res;
}
@@ -1302,14 +1326,24 @@ static inline long __strlen_user(const char __user *s)
{
long res;
- might_fault();
- __asm__ __volatile__(
- "move\t$4, %1\n\t"
- __MODULE_JAL(__strlen_user_nocheck_asm)
- "move\t%0, $2"
- : "=r" (res)
- : "r" (s)
- : "$2", "$4", __UA_t0, "$31");
+ if (segment_eq(get_fs(), get_ds())) {
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ __MODULE_JAL(__strlen_kernel_nocheck_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (s)
+ : "$2", "$4", __UA_t0, "$31");
+ } else {
+ might_fault();
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ __MODULE_JAL(__strlen_user_nocheck_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (s)
+ : "$2", "$4", __UA_t0, "$31");
+ }
return res;
}
@@ -1332,14 +1366,24 @@ static inline long strlen_user(const char __user *s)
{
long res;
- might_fault();
- __asm__ __volatile__(
- "move\t$4, %1\n\t"
- __MODULE_JAL(__strlen_user_asm)
- "move\t%0, $2"
- : "=r" (res)
- : "r" (s)
- : "$2", "$4", __UA_t0, "$31");
+ if (segment_eq(get_fs(), get_ds())) {
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ __MODULE_JAL(__strlen_kernel_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (s)
+ : "$2", "$4", __UA_t0, "$31");
+ } else {
+ might_fault();
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ __MODULE_JAL(__strlen_kernel_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (s)
+ : "$2", "$4", __UA_t0, "$31");
+ }
return res;
}
@@ -1349,15 +1393,26 @@ static inline long __strnlen_user(const char __user *s, long n)
{
long res;
- might_fault();
- __asm__ __volatile__(
- "move\t$4, %1\n\t"
- "move\t$5, %2\n\t"
- __MODULE_JAL(__strnlen_user_nocheck_asm)
- "move\t%0, $2"
- : "=r" (res)
- : "r" (s), "r" (n)
- : "$2", "$4", "$5", __UA_t0, "$31");
+ if (segment_eq(get_fs(), get_ds())) {
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ "move\t$5, %2\n\t"
+ __MODULE_JAL(__strnlen_kernel_nocheck_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (s), "r" (n)
+ : "$2", "$4", "$5", __UA_t0, "$31");
+ } else {
+ might_fault();
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ "move\t$5, %2\n\t"
+ __MODULE_JAL(__strnlen_user_nocheck_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (s), "r" (n)
+ : "$2", "$4", "$5", __UA_t0, "$31");
+ }
return res;
}
@@ -1381,14 +1436,25 @@ static inline long strnlen_user(const char __user *s, long n)
long res;
might_fault();
- __asm__ __volatile__(
- "move\t$4, %1\n\t"
- "move\t$5, %2\n\t"
- __MODULE_JAL(__strnlen_user_asm)
- "move\t%0, $2"
- : "=r" (res)
- : "r" (s), "r" (n)
- : "$2", "$4", "$5", __UA_t0, "$31");
+ if (segment_eq(get_fs(), get_ds())) {
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ "move\t$5, %2\n\t"
+ __MODULE_JAL(__strnlen_kernel_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (s), "r" (n)
+ : "$2", "$4", "$5", __UA_t0, "$31");
+ } else {
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ "move\t$5, %2\n\t"
+ __MODULE_JAL(__strnlen_user_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (s), "r" (n)
+ : "$2", "$4", "$5", __UA_t0, "$31");
+ }
return res;
}
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 29/58] MIPS: asm: uaccess: Add EVA support for str*_user operations
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
The str*_user functions are used to securely access NULL terminated
strings from userland. Therefore, it's necessary to use the appropriate
EVA function. However, if the string is in kernel space, then the normal
instructions are being used to access it. The __str*_kernel_asm and
__str*_user_asm symbols are the same for non-EVA mode so there is no
functional change for the non-EVA kernels.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/uaccess.h | 172 +++++++++++++++++++++++++++-------------
1 file changed, 119 insertions(+), 53 deletions(-)
diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index fe72837..da52765 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -1246,16 +1246,28 @@ __strncpy_from_user(char *__to, const char __user *__from, long __len)
{
long res;
- might_fault();
- __asm__ __volatile__(
- "move\t$4, %1\n\t"
- "move\t$5, %2\n\t"
- "move\t$6, %3\n\t"
- __MODULE_JAL(__strncpy_from_user_nocheck_asm)
- "move\t%0, $2"
- : "=r" (res)
- : "r" (__to), "r" (__from), "r" (__len)
- : "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+ if (segment_eq(get_fs(), get_ds())) {
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ "move\t$5, %2\n\t"
+ "move\t$6, %3\n\t"
+ __MODULE_JAL(__strncpy_from_kernel_nocheck_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (__to), "r" (__from), "r" (__len)
+ : "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+ } else {
+ might_fault();
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ "move\t$5, %2\n\t"
+ "move\t$6, %3\n\t"
+ __MODULE_JAL(__strncpy_from_user_nocheck_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (__to), "r" (__from), "r" (__len)
+ : "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+ }
return res;
}
@@ -1283,16 +1295,28 @@ strncpy_from_user(char *__to, const char __user *__from, long __len)
{
long res;
- might_fault();
- __asm__ __volatile__(
- "move\t$4, %1\n\t"
- "move\t$5, %2\n\t"
- "move\t$6, %3\n\t"
- __MODULE_JAL(__strncpy_from_user_asm)
- "move\t%0, $2"
- : "=r" (res)
- : "r" (__to), "r" (__from), "r" (__len)
- : "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+ if (segment_eq(get_fs(), get_ds())) {
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ "move\t$5, %2\n\t"
+ "move\t$6, %3\n\t"
+ __MODULE_JAL(__strncpy_from_kernel_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (__to), "r" (__from), "r" (__len)
+ : "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+ } else {
+ might_fault();
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ "move\t$5, %2\n\t"
+ "move\t$6, %3\n\t"
+ __MODULE_JAL(__strncpy_from_user_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (__to), "r" (__from), "r" (__len)
+ : "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+ }
return res;
}
@@ -1302,14 +1326,24 @@ static inline long __strlen_user(const char __user *s)
{
long res;
- might_fault();
- __asm__ __volatile__(
- "move\t$4, %1\n\t"
- __MODULE_JAL(__strlen_user_nocheck_asm)
- "move\t%0, $2"
- : "=r" (res)
- : "r" (s)
- : "$2", "$4", __UA_t0, "$31");
+ if (segment_eq(get_fs(), get_ds())) {
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ __MODULE_JAL(__strlen_kernel_nocheck_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (s)
+ : "$2", "$4", __UA_t0, "$31");
+ } else {
+ might_fault();
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ __MODULE_JAL(__strlen_user_nocheck_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (s)
+ : "$2", "$4", __UA_t0, "$31");
+ }
return res;
}
@@ -1332,14 +1366,24 @@ static inline long strlen_user(const char __user *s)
{
long res;
- might_fault();
- __asm__ __volatile__(
- "move\t$4, %1\n\t"
- __MODULE_JAL(__strlen_user_asm)
- "move\t%0, $2"
- : "=r" (res)
- : "r" (s)
- : "$2", "$4", __UA_t0, "$31");
+ if (segment_eq(get_fs(), get_ds())) {
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ __MODULE_JAL(__strlen_kernel_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (s)
+ : "$2", "$4", __UA_t0, "$31");
+ } else {
+ might_fault();
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ __MODULE_JAL(__strlen_kernel_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (s)
+ : "$2", "$4", __UA_t0, "$31");
+ }
return res;
}
@@ -1349,15 +1393,26 @@ static inline long __strnlen_user(const char __user *s, long n)
{
long res;
- might_fault();
- __asm__ __volatile__(
- "move\t$4, %1\n\t"
- "move\t$5, %2\n\t"
- __MODULE_JAL(__strnlen_user_nocheck_asm)
- "move\t%0, $2"
- : "=r" (res)
- : "r" (s), "r" (n)
- : "$2", "$4", "$5", __UA_t0, "$31");
+ if (segment_eq(get_fs(), get_ds())) {
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ "move\t$5, %2\n\t"
+ __MODULE_JAL(__strnlen_kernel_nocheck_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (s), "r" (n)
+ : "$2", "$4", "$5", __UA_t0, "$31");
+ } else {
+ might_fault();
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ "move\t$5, %2\n\t"
+ __MODULE_JAL(__strnlen_user_nocheck_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (s), "r" (n)
+ : "$2", "$4", "$5", __UA_t0, "$31");
+ }
return res;
}
@@ -1381,14 +1436,25 @@ static inline long strnlen_user(const char __user *s, long n)
long res;
might_fault();
- __asm__ __volatile__(
- "move\t$4, %1\n\t"
- "move\t$5, %2\n\t"
- __MODULE_JAL(__strnlen_user_asm)
- "move\t%0, $2"
- : "=r" (res)
- : "r" (s), "r" (n)
- : "$2", "$4", "$5", __UA_t0, "$31");
+ if (segment_eq(get_fs(), get_ds())) {
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ "move\t$5, %2\n\t"
+ __MODULE_JAL(__strnlen_kernel_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (s), "r" (n)
+ : "$2", "$4", "$5", __UA_t0, "$31");
+ } else {
+ __asm__ __volatile__(
+ "move\t$4, %1\n\t"
+ "move\t$5, %2\n\t"
+ __MODULE_JAL(__strnlen_user_asm)
+ "move\t%0, $2"
+ : "=r" (res)
+ : "r" (s), "r" (n)
+ : "$2", "$4", "$5", __UA_t0, "$31");
+ }
return res;
}
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 30/58] MIPS: kernel: unaligned: Add EVA instruction wrappers
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Use the load/store instruction wrappers from asm/asm.h to
perform such operations when operating in EVA mode.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/unaligned.c | 49 ++++++++++++++++++++++----------------------
1 file changed, 25 insertions(+), 24 deletions(-)
diff --git a/arch/mips/kernel/unaligned.c b/arch/mips/kernel/unaligned.c
index c369a5d..5ec8f00 100644
--- a/arch/mips/kernel/unaligned.c
+++ b/arch/mips/kernel/unaligned.c
@@ -7,6 +7,7 @@
*
* Copyright (C) 1996, 1998, 1999, 2002 by Ralf Baechle
* Copyright (C) 1999 Silicon Graphics, Inc.
+ * Copyright (C) 2014 Imagination Technologies Ltd.
*
* This file contains exception handler for address error exception with the
* special capability to execute faulting instructions in software. The
@@ -110,8 +111,8 @@ extern void show_registers(struct pt_regs *regs);
#ifdef __BIG_ENDIAN
#define LoadHW(addr, value, res) \
__asm__ __volatile__ (".set\tnoat\n" \
- "1:\tlb\t%0, 0(%2)\n" \
- "2:\tlbu\t$1, 1(%2)\n\t" \
+ "1:\t"user_lb("%0", "0(%2)")"\n" \
+ "2:\t"user_lbu("$1", "1(%2)")"\n\t" \
"sll\t%0, 0x8\n\t" \
"or\t%0, $1\n\t" \
"li\t%1, 0\n" \
@@ -130,8 +131,8 @@ extern void show_registers(struct pt_regs *regs);
#define LoadW(addr, value, res) \
__asm__ __volatile__ ( \
- "1:\tlwl\t%0, (%2)\n" \
- "2:\tlwr\t%0, 3(%2)\n\t" \
+ "1:\t"user_lwl("%0", "(%2)")"\n" \
+ "2:\t"user_lwr("%0", "3(%2)")"\n\t" \
"li\t%1, 0\n" \
"3:\n\t" \
".insn\n\t" \
@@ -149,8 +150,8 @@ extern void show_registers(struct pt_regs *regs);
#define LoadHWU(addr, value, res) \
__asm__ __volatile__ ( \
".set\tnoat\n" \
- "1:\tlbu\t%0, 0(%2)\n" \
- "2:\tlbu\t$1, 1(%2)\n\t" \
+ "1:\t"user_lbu("%0", "0(%2)")"\n" \
+ "2:\t"user_lbu("$1", "1(%2)")"\n\t" \
"sll\t%0, 0x8\n\t" \
"or\t%0, $1\n\t" \
"li\t%1, 0\n" \
@@ -170,8 +171,8 @@ extern void show_registers(struct pt_regs *regs);
#define LoadWU(addr, value, res) \
__asm__ __volatile__ ( \
- "1:\tlwl\t%0, (%2)\n" \
- "2:\tlwr\t%0, 3(%2)\n\t" \
+ "1:\t"user_lwl("%0", "(%2)")"\n" \
+ "2:\t"user_lwr("%0", "3(%2)")"\n\t" \
"dsll\t%0, %0, 32\n\t" \
"dsrl\t%0, %0, 32\n\t" \
"li\t%1, 0\n" \
@@ -209,9 +210,9 @@ extern void show_registers(struct pt_regs *regs);
#define StoreHW(addr, value, res) \
__asm__ __volatile__ ( \
".set\tnoat\n" \
- "1:\tsb\t%1, 1(%2)\n\t" \
+ "1:\t"user_sb("%1", "1(%2)")"\n" \
"srl\t$1, %1, 0x8\n" \
- "2:\tsb\t$1, 0(%2)\n\t" \
+ "2:\t"user_sb("$1", "0(%2)")"\n" \
".set\tat\n\t" \
"li\t%0, 0\n" \
"3:\n\t" \
@@ -229,8 +230,8 @@ extern void show_registers(struct pt_regs *regs);
#define StoreW(addr, value, res) \
__asm__ __volatile__ ( \
- "1:\tswl\t%1,(%2)\n" \
- "2:\tswr\t%1, 3(%2)\n\t" \
+ "1:\t"user_swl("%1", "(%2)")"\n" \
+ "2:\t"user_swr("%1", "3(%2)")"\n\t" \
"li\t%0, 0\n" \
"3:\n\t" \
".insn\n\t" \
@@ -267,8 +268,8 @@ extern void show_registers(struct pt_regs *regs);
#ifdef __LITTLE_ENDIAN
#define LoadHW(addr, value, res) \
__asm__ __volatile__ (".set\tnoat\n" \
- "1:\tlb\t%0, 1(%2)\n" \
- "2:\tlbu\t$1, 0(%2)\n\t" \
+ "1:\t"user_lb("%0", "1(%2)")"\n" \
+ "2:\t"user_lbu("$1", "0(%2)")"\n\t" \
"sll\t%0, 0x8\n\t" \
"or\t%0, $1\n\t" \
"li\t%1, 0\n" \
@@ -287,8 +288,8 @@ extern void show_registers(struct pt_regs *regs);
#define LoadW(addr, value, res) \
__asm__ __volatile__ ( \
- "1:\tlwl\t%0, 3(%2)\n" \
- "2:\tlwr\t%0, (%2)\n\t" \
+ "1:\t"user_lwl("%0", "3(%2)")"\n" \
+ "2:\t"user_lwr("%0", "(%2)")"\n\t" \
"li\t%1, 0\n" \
"3:\n\t" \
".insn\n\t" \
@@ -306,8 +307,8 @@ extern void show_registers(struct pt_regs *regs);
#define LoadHWU(addr, value, res) \
__asm__ __volatile__ ( \
".set\tnoat\n" \
- "1:\tlbu\t%0, 1(%2)\n" \
- "2:\tlbu\t$1, 0(%2)\n\t" \
+ "1:\t"user_lbu("%0", "1(%2)")"\n" \
+ "2:\t"user_lbu("$1", "0(%2)")"\n\t" \
"sll\t%0, 0x8\n\t" \
"or\t%0, $1\n\t" \
"li\t%1, 0\n" \
@@ -327,8 +328,8 @@ extern void show_registers(struct pt_regs *regs);
#define LoadWU(addr, value, res) \
__asm__ __volatile__ ( \
- "1:\tlwl\t%0, 3(%2)\n" \
- "2:\tlwr\t%0, (%2)\n\t" \
+ "1:\t"user_lwl("%0", "3(%2)")"\n" \
+ "2:\t"user_lwr("%0", "(%2)")"\n\t" \
"dsll\t%0, %0, 32\n\t" \
"dsrl\t%0, %0, 32\n\t" \
"li\t%1, 0\n" \
@@ -366,9 +367,9 @@ extern void show_registers(struct pt_regs *regs);
#define StoreHW(addr, value, res) \
__asm__ __volatile__ ( \
".set\tnoat\n" \
- "1:\tsb\t%1, 0(%2)\n\t" \
+ "1:\t"user_sb("%1", "0(%2)")"\n" \
"srl\t$1,%1, 0x8\n" \
- "2:\tsb\t$1, 1(%2)\n\t" \
+ "2:\t"user_sb("$1", "1(%2)")"\n" \
".set\tat\n\t" \
"li\t%0, 0\n" \
"3:\n\t" \
@@ -386,8 +387,8 @@ extern void show_registers(struct pt_regs *regs);
#define StoreW(addr, value, res) \
__asm__ __volatile__ ( \
- "1:\tswl\t%1, 3(%2)\n" \
- "2:\tswr\t%1, (%2)\n\t" \
+ "1:\t"user_swl("%1", "3(%2)")"\n" \
+ "2:\t"user_swr("%1", "(%2)")"\n\t" \
"li\t%0, 0\n" \
"3:\n\t" \
".insn\n\t" \
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 30/58] MIPS: kernel: unaligned: Add EVA instruction wrappers
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Use the load/store instruction wrappers from asm/asm.h to
perform such operations when operating in EVA mode.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/unaligned.c | 49 ++++++++++++++++++++++----------------------
1 file changed, 25 insertions(+), 24 deletions(-)
diff --git a/arch/mips/kernel/unaligned.c b/arch/mips/kernel/unaligned.c
index c369a5d..5ec8f00 100644
--- a/arch/mips/kernel/unaligned.c
+++ b/arch/mips/kernel/unaligned.c
@@ -7,6 +7,7 @@
*
* Copyright (C) 1996, 1998, 1999, 2002 by Ralf Baechle
* Copyright (C) 1999 Silicon Graphics, Inc.
+ * Copyright (C) 2014 Imagination Technologies Ltd.
*
* This file contains exception handler for address error exception with the
* special capability to execute faulting instructions in software. The
@@ -110,8 +111,8 @@ extern void show_registers(struct pt_regs *regs);
#ifdef __BIG_ENDIAN
#define LoadHW(addr, value, res) \
__asm__ __volatile__ (".set\tnoat\n" \
- "1:\tlb\t%0, 0(%2)\n" \
- "2:\tlbu\t$1, 1(%2)\n\t" \
+ "1:\t"user_lb("%0", "0(%2)")"\n" \
+ "2:\t"user_lbu("$1", "1(%2)")"\n\t" \
"sll\t%0, 0x8\n\t" \
"or\t%0, $1\n\t" \
"li\t%1, 0\n" \
@@ -130,8 +131,8 @@ extern void show_registers(struct pt_regs *regs);
#define LoadW(addr, value, res) \
__asm__ __volatile__ ( \
- "1:\tlwl\t%0, (%2)\n" \
- "2:\tlwr\t%0, 3(%2)\n\t" \
+ "1:\t"user_lwl("%0", "(%2)")"\n" \
+ "2:\t"user_lwr("%0", "3(%2)")"\n\t" \
"li\t%1, 0\n" \
"3:\n\t" \
".insn\n\t" \
@@ -149,8 +150,8 @@ extern void show_registers(struct pt_regs *regs);
#define LoadHWU(addr, value, res) \
__asm__ __volatile__ ( \
".set\tnoat\n" \
- "1:\tlbu\t%0, 0(%2)\n" \
- "2:\tlbu\t$1, 1(%2)\n\t" \
+ "1:\t"user_lbu("%0", "0(%2)")"\n" \
+ "2:\t"user_lbu("$1", "1(%2)")"\n\t" \
"sll\t%0, 0x8\n\t" \
"or\t%0, $1\n\t" \
"li\t%1, 0\n" \
@@ -170,8 +171,8 @@ extern void show_registers(struct pt_regs *regs);
#define LoadWU(addr, value, res) \
__asm__ __volatile__ ( \
- "1:\tlwl\t%0, (%2)\n" \
- "2:\tlwr\t%0, 3(%2)\n\t" \
+ "1:\t"user_lwl("%0", "(%2)")"\n" \
+ "2:\t"user_lwr("%0", "3(%2)")"\n\t" \
"dsll\t%0, %0, 32\n\t" \
"dsrl\t%0, %0, 32\n\t" \
"li\t%1, 0\n" \
@@ -209,9 +210,9 @@ extern void show_registers(struct pt_regs *regs);
#define StoreHW(addr, value, res) \
__asm__ __volatile__ ( \
".set\tnoat\n" \
- "1:\tsb\t%1, 1(%2)\n\t" \
+ "1:\t"user_sb("%1", "1(%2)")"\n" \
"srl\t$1, %1, 0x8\n" \
- "2:\tsb\t$1, 0(%2)\n\t" \
+ "2:\t"user_sb("$1", "0(%2)")"\n" \
".set\tat\n\t" \
"li\t%0, 0\n" \
"3:\n\t" \
@@ -229,8 +230,8 @@ extern void show_registers(struct pt_regs *regs);
#define StoreW(addr, value, res) \
__asm__ __volatile__ ( \
- "1:\tswl\t%1,(%2)\n" \
- "2:\tswr\t%1, 3(%2)\n\t" \
+ "1:\t"user_swl("%1", "(%2)")"\n" \
+ "2:\t"user_swr("%1", "3(%2)")"\n\t" \
"li\t%0, 0\n" \
"3:\n\t" \
".insn\n\t" \
@@ -267,8 +268,8 @@ extern void show_registers(struct pt_regs *regs);
#ifdef __LITTLE_ENDIAN
#define LoadHW(addr, value, res) \
__asm__ __volatile__ (".set\tnoat\n" \
- "1:\tlb\t%0, 1(%2)\n" \
- "2:\tlbu\t$1, 0(%2)\n\t" \
+ "1:\t"user_lb("%0", "1(%2)")"\n" \
+ "2:\t"user_lbu("$1", "0(%2)")"\n\t" \
"sll\t%0, 0x8\n\t" \
"or\t%0, $1\n\t" \
"li\t%1, 0\n" \
@@ -287,8 +288,8 @@ extern void show_registers(struct pt_regs *regs);
#define LoadW(addr, value, res) \
__asm__ __volatile__ ( \
- "1:\tlwl\t%0, 3(%2)\n" \
- "2:\tlwr\t%0, (%2)\n\t" \
+ "1:\t"user_lwl("%0", "3(%2)")"\n" \
+ "2:\t"user_lwr("%0", "(%2)")"\n\t" \
"li\t%1, 0\n" \
"3:\n\t" \
".insn\n\t" \
@@ -306,8 +307,8 @@ extern void show_registers(struct pt_regs *regs);
#define LoadHWU(addr, value, res) \
__asm__ __volatile__ ( \
".set\tnoat\n" \
- "1:\tlbu\t%0, 1(%2)\n" \
- "2:\tlbu\t$1, 0(%2)\n\t" \
+ "1:\t"user_lbu("%0", "1(%2)")"\n" \
+ "2:\t"user_lbu("$1", "0(%2)")"\n\t" \
"sll\t%0, 0x8\n\t" \
"or\t%0, $1\n\t" \
"li\t%1, 0\n" \
@@ -327,8 +328,8 @@ extern void show_registers(struct pt_regs *regs);
#define LoadWU(addr, value, res) \
__asm__ __volatile__ ( \
- "1:\tlwl\t%0, 3(%2)\n" \
- "2:\tlwr\t%0, (%2)\n\t" \
+ "1:\t"user_lwl("%0", "3(%2)")"\n" \
+ "2:\t"user_lwr("%0", "(%2)")"\n\t" \
"dsll\t%0, %0, 32\n\t" \
"dsrl\t%0, %0, 32\n\t" \
"li\t%1, 0\n" \
@@ -366,9 +367,9 @@ extern void show_registers(struct pt_regs *regs);
#define StoreHW(addr, value, res) \
__asm__ __volatile__ ( \
".set\tnoat\n" \
- "1:\tsb\t%1, 0(%2)\n\t" \
+ "1:\t"user_sb("%1", "0(%2)")"\n" \
"srl\t$1,%1, 0x8\n" \
- "2:\tsb\t$1, 1(%2)\n\t" \
+ "2:\t"user_sb("$1", "1(%2)")"\n" \
".set\tat\n\t" \
"li\t%0, 0\n" \
"3:\n\t" \
@@ -386,8 +387,8 @@ extern void show_registers(struct pt_regs *regs);
#define StoreW(addr, value, res) \
__asm__ __volatile__ ( \
- "1:\tswl\t%1, 3(%2)\n" \
- "2:\tswr\t%1, (%2)\n\t" \
+ "1:\t"user_swl("%1", "3(%2)")"\n" \
+ "2:\t"user_swr("%1", "(%2)")"\n\t" \
"li\t%0, 0\n" \
"3:\n\t" \
".insn\n\t" \
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 31/58] MIPS: kernel: unaligned: Handle unaligned accesses for EVA
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Handle unaligned accesses when we access userspace memory
EVA mode.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/unaligned.c | 86 +++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 85 insertions(+), 1 deletion(-)
diff --git a/arch/mips/kernel/unaligned.c b/arch/mips/kernel/unaligned.c
index 5ec8f00..2b35172 100644
--- a/arch/mips/kernel/unaligned.c
+++ b/arch/mips/kernel/unaligned.c
@@ -431,7 +431,9 @@ static void emulate_load_store_insn(struct pt_regs *regs,
unsigned long origpc;
unsigned long orig31;
void __user *fault_addr = NULL;
-
+#ifdef CONFIG_EVA
+ mm_segment_t seg;
+#endif
origpc = (unsigned long)pc;
orig31 = regs->regs[31];
@@ -476,6 +478,88 @@ static void emulate_load_store_insn(struct pt_regs *regs,
* The remaining opcodes are the ones that are really of
* interest.
*/
+#ifdef CONFIG_EVA
+ case spec3_op:
+ /*
+ * we can land here only from kernel accessing user memory,
+ * so we need to "switch" the address limit to user space, so
+ * address check can work properly.
+ */
+ seg = get_fs();
+ set_fs(USER_DS);
+ switch (insn.spec3_format.func) {
+ case lhe_op:
+ if (!access_ok(VERIFY_READ, addr, 2)) {
+ set_fs(seg);
+ goto sigbus;
+ }
+ LoadHW(addr, value, res);
+ if (res) {
+ set_fs(seg);
+ goto fault;
+ }
+ compute_return_epc(regs);
+ regs->regs[insn.spec3_format.rt] = value;
+ break;
+ case lwe_op:
+ if (!access_ok(VERIFY_READ, addr, 4)) {
+ set_fs(seg);
+ goto sigbus;
+ }
+ LoadW(addr, value, res);
+ if (res) {
+ set_fs(seg);
+ goto fault;
+ }
+ compute_return_epc(regs);
+ regs->regs[insn.spec3_format.rt] = value;
+ break;
+ case lhue_op:
+ if (!access_ok(VERIFY_READ, addr, 2)) {
+ set_fs(seg);
+ goto sigbus;
+ }
+ LoadHWU(addr, value, res);
+ if (res) {
+ set_fs(seg);
+ goto fault;
+ }
+ compute_return_epc(regs);
+ regs->regs[insn.spec3_format.rt] = value;
+ break;
+ case she_op:
+ if (!access_ok(VERIFY_WRITE, addr, 2)) {
+ set_fs(seg);
+ goto sigbus;
+ }
+ compute_return_epc(regs);
+ value = regs->regs[insn.spec3_format.rt];
+ StoreHW(addr, value, res);
+ if (res) {
+ set_fs(seg);
+ goto fault;
+ }
+ break;
+ case swe_op:
+ if (!access_ok(VERIFY_WRITE, addr, 4)) {
+ set_fs(seg);
+ goto sigbus;
+ }
+ compute_return_epc(regs);
+ value = regs->regs[insn.spec3_format.rt];
+ StoreW(addr, value, res);
+ if (res) {
+ set_fs(seg);
+ goto fault;
+ }
+ break;
+ default:
+ set_fs(seg);
+ goto sigill;
+ }
+ set_fs(seg);
+ break;
+#endif
case lh_op:
if (!access_ok(VERIFY_READ, addr, 2))
goto sigbus;
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 31/58] MIPS: kernel: unaligned: Handle unaligned accesses for EVA
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Handle unaligned accesses when we access userspace memory
EVA mode.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/unaligned.c | 86 +++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 85 insertions(+), 1 deletion(-)
diff --git a/arch/mips/kernel/unaligned.c b/arch/mips/kernel/unaligned.c
index 5ec8f00..2b35172 100644
--- a/arch/mips/kernel/unaligned.c
+++ b/arch/mips/kernel/unaligned.c
@@ -431,7 +431,9 @@ static void emulate_load_store_insn(struct pt_regs *regs,
unsigned long origpc;
unsigned long orig31;
void __user *fault_addr = NULL;
-
+#ifdef CONFIG_EVA
+ mm_segment_t seg;
+#endif
origpc = (unsigned long)pc;
orig31 = regs->regs[31];
@@ -476,6 +478,88 @@ static void emulate_load_store_insn(struct pt_regs *regs,
* The remaining opcodes are the ones that are really of
* interest.
*/
+#ifdef CONFIG_EVA
+ case spec3_op:
+ /*
+ * we can land here only from kernel accessing user memory,
+ * so we need to "switch" the address limit to user space, so
+ * address check can work properly.
+ */
+ seg = get_fs();
+ set_fs(USER_DS);
+ switch (insn.spec3_format.func) {
+ case lhe_op:
+ if (!access_ok(VERIFY_READ, addr, 2)) {
+ set_fs(seg);
+ goto sigbus;
+ }
+ LoadHW(addr, value, res);
+ if (res) {
+ set_fs(seg);
+ goto fault;
+ }
+ compute_return_epc(regs);
+ regs->regs[insn.spec3_format.rt] = value;
+ break;
+ case lwe_op:
+ if (!access_ok(VERIFY_READ, addr, 4)) {
+ set_fs(seg);
+ goto sigbus;
+ }
+ LoadW(addr, value, res);
+ if (res) {
+ set_fs(seg);
+ goto fault;
+ }
+ compute_return_epc(regs);
+ regs->regs[insn.spec3_format.rt] = value;
+ break;
+ case lhue_op:
+ if (!access_ok(VERIFY_READ, addr, 2)) {
+ set_fs(seg);
+ goto sigbus;
+ }
+ LoadHWU(addr, value, res);
+ if (res) {
+ set_fs(seg);
+ goto fault;
+ }
+ compute_return_epc(regs);
+ regs->regs[insn.spec3_format.rt] = value;
+ break;
+ case she_op:
+ if (!access_ok(VERIFY_WRITE, addr, 2)) {
+ set_fs(seg);
+ goto sigbus;
+ }
+ compute_return_epc(regs);
+ value = regs->regs[insn.spec3_format.rt];
+ StoreHW(addr, value, res);
+ if (res) {
+ set_fs(seg);
+ goto fault;
+ }
+ break;
+ case swe_op:
+ if (!access_ok(VERIFY_WRITE, addr, 4)) {
+ set_fs(seg);
+ goto sigbus;
+ }
+ compute_return_epc(regs);
+ value = regs->regs[insn.spec3_format.rt];
+ StoreW(addr, value, res);
+ if (res) {
+ set_fs(seg);
+ goto fault;
+ }
+ break;
+ default:
+ set_fs(seg);
+ goto sigill;
+ }
+ set_fs(seg);
+ break;
+#endif
case lh_op:
if (!access_ok(VERIFY_READ, addr, 2))
goto sigbus;
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 32/58] MIPS: checksum: Split the 'copy_user' symbol
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
The 'copy_user' symbol can be used to copy from or to
userland so we will use two different symbols for these
operations. This makes no difference in the existing code,
but when the core is operating in EVA mode, different instructions
need to be used to read and write to userland address space.
The old function has also been renamed to 'copy_kernel' to denote
that it is suitable for copy data to and from kernel space.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/checksum.h | 18 ++++++++++++------
arch/mips/kernel/mips_ksyms.c | 4 +++-
arch/mips/lib/csum_partial.S | 9 ++++++---
3 files changed, 21 insertions(+), 10 deletions(-)
diff --git a/arch/mips/include/asm/checksum.h b/arch/mips/include/asm/checksum.h
index ac3d2b8..3c9aea5 100644
--- a/arch/mips/include/asm/checksum.h
+++ b/arch/mips/include/asm/checksum.h
@@ -7,6 +7,7 @@
* Copyright (C) 1999 Silicon Graphics, Inc.
* Copyright (C) 2001 Thiemo Seufer.
* Copyright (C) 2002 Maciej W. Rozycki
+ * Copyright (C) 2014 Imagination Technologies Ltd.
*/
#ifndef _ASM_CHECKSUM_H
#define _ASM_CHECKSUM_H
@@ -29,8 +30,13 @@
*/
__wsum csum_partial(const void *buff, int len, __wsum sum);
-__wsum __csum_partial_copy_user(const void *src, void *dst,
- int len, __wsum sum, int *err_ptr);
+__wsum __csum_partial_copy_kernel(const void *src, void *dst,
+ int len, __wsum sum, int *err_ptr);
+
+__wsum __csum_partial_copy_from_user(const void *src, void *dst,
+ int len, __wsum sum, int *err_ptr);
+__wsum __csum_partial_copy_to_user(const void *src, void *dst,
+ int len, __wsum sum, int *err_ptr);
/*
* this is a new version of the above that records errors it finds in *errp,
@@ -41,8 +47,8 @@ __wsum csum_partial_copy_from_user(const void __user *src, void *dst, int len,
__wsum sum, int *err_ptr)
{
might_fault();
- return __csum_partial_copy_user((__force void *)src, dst,
- len, sum, err_ptr);
+ return __csum_partial_copy_from_user((__force void *)src, dst,
+ len, sum, err_ptr);
}
/*
@@ -55,8 +61,8 @@ __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len,
{
might_fault();
if (access_ok(VERIFY_WRITE, dst, len))
- return __csum_partial_copy_user(src, (__force void *)dst,
- len, sum, err_ptr);
+ return __csum_partial_copy_to_user(src, (__force void *)dst,
+ len, sum, err_ptr);
if (len)
*err_ptr = -EFAULT;
diff --git a/arch/mips/kernel/mips_ksyms.c b/arch/mips/kernel/mips_ksyms.c
index 13e60f6..2607c3a 100644
--- a/arch/mips/kernel/mips_ksyms.c
+++ b/arch/mips/kernel/mips_ksyms.c
@@ -73,7 +73,9 @@ EXPORT_SYMBOL(__strnlen_user_asm);
EXPORT_SYMBOL(csum_partial);
EXPORT_SYMBOL(csum_partial_copy_nocheck);
-EXPORT_SYMBOL(__csum_partial_copy_user);
+EXPORT_SYMBOL(__csum_partial_copy_kernel);
+EXPORT_SYMBOL(__csum_partial_copy_to_user);
+EXPORT_SYMBOL(__csum_partial_copy_from_user);
EXPORT_SYMBOL(invalid_pte_table);
#ifdef CONFIG_FUNCTION_TRACER
diff --git a/arch/mips/lib/csum_partial.S b/arch/mips/lib/csum_partial.S
index a6adffb..5d73d0d 100644
--- a/arch/mips/lib/csum_partial.S
+++ b/arch/mips/lib/csum_partial.S
@@ -8,6 +8,7 @@
* Copyright (C) 1998, 1999 Ralf Baechle
* Copyright (C) 1999 Silicon Graphics, Inc.
* Copyright (C) 2007 Maciej W. Rozycki
+ * Copyright (C) 2014 Imagination Technologies Ltd.
*/
#include <linux/errno.h>
#include <asm/asm.h>
@@ -296,7 +297,7 @@ LEAF(csum_partial)
* checksum and copy routines based on memcpy.S
*
* csum_partial_copy_nocheck(src, dst, len, sum)
- * __csum_partial_copy_user(src, dst, len, sum, errp)
+ * __csum_partial_copy_kernel(src, dst, len, sum, errp)
*
* See "Spec" in memcpy.S for details. Unlike __copy_user, all
* function in this file use the standard calling convention.
@@ -396,7 +397,9 @@ LEAF(csum_partial)
.set at=v1
#endif
-LEAF(__csum_partial_copy_user)
+LEAF(__csum_partial_copy_kernel)
+FEXPORT(__csum_partial_copy_to_user)
+FEXPORT(__csum_partial_copy_from_user)
PTR_ADDU AT, src, len /* See (1) above. */
#ifdef CONFIG_64BIT
move errptr, a4
@@ -757,4 +760,4 @@ EXC( lbu t1, 0(src), .Ll_exc)
jr ra
sw v1, (errptr)
.set pop
- END(__csum_partial_copy_user)
+ END(__csum_partial_copy_kernel)
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 32/58] MIPS: checksum: Split the 'copy_user' symbol
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
The 'copy_user' symbol can be used to copy from or to
userland so we will use two different symbols for these
operations. This makes no difference in the existing code,
but when the core is operating in EVA mode, different instructions
need to be used to read and write to userland address space.
The old function has also been renamed to 'copy_kernel' to denote
that it is suitable for copy data to and from kernel space.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/checksum.h | 18 ++++++++++++------
arch/mips/kernel/mips_ksyms.c | 4 +++-
arch/mips/lib/csum_partial.S | 9 ++++++---
3 files changed, 21 insertions(+), 10 deletions(-)
diff --git a/arch/mips/include/asm/checksum.h b/arch/mips/include/asm/checksum.h
index ac3d2b8..3c9aea5 100644
--- a/arch/mips/include/asm/checksum.h
+++ b/arch/mips/include/asm/checksum.h
@@ -7,6 +7,7 @@
* Copyright (C) 1999 Silicon Graphics, Inc.
* Copyright (C) 2001 Thiemo Seufer.
* Copyright (C) 2002 Maciej W. Rozycki
+ * Copyright (C) 2014 Imagination Technologies Ltd.
*/
#ifndef _ASM_CHECKSUM_H
#define _ASM_CHECKSUM_H
@@ -29,8 +30,13 @@
*/
__wsum csum_partial(const void *buff, int len, __wsum sum);
-__wsum __csum_partial_copy_user(const void *src, void *dst,
- int len, __wsum sum, int *err_ptr);
+__wsum __csum_partial_copy_kernel(const void *src, void *dst,
+ int len, __wsum sum, int *err_ptr);
+
+__wsum __csum_partial_copy_from_user(const void *src, void *dst,
+ int len, __wsum sum, int *err_ptr);
+__wsum __csum_partial_copy_to_user(const void *src, void *dst,
+ int len, __wsum sum, int *err_ptr);
/*
* this is a new version of the above that records errors it finds in *errp,
@@ -41,8 +47,8 @@ __wsum csum_partial_copy_from_user(const void __user *src, void *dst, int len,
__wsum sum, int *err_ptr)
{
might_fault();
- return __csum_partial_copy_user((__force void *)src, dst,
- len, sum, err_ptr);
+ return __csum_partial_copy_from_user((__force void *)src, dst,
+ len, sum, err_ptr);
}
/*
@@ -55,8 +61,8 @@ __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len,
{
might_fault();
if (access_ok(VERIFY_WRITE, dst, len))
- return __csum_partial_copy_user(src, (__force void *)dst,
- len, sum, err_ptr);
+ return __csum_partial_copy_to_user(src, (__force void *)dst,
+ len, sum, err_ptr);
if (len)
*err_ptr = -EFAULT;
diff --git a/arch/mips/kernel/mips_ksyms.c b/arch/mips/kernel/mips_ksyms.c
index 13e60f6..2607c3a 100644
--- a/arch/mips/kernel/mips_ksyms.c
+++ b/arch/mips/kernel/mips_ksyms.c
@@ -73,7 +73,9 @@ EXPORT_SYMBOL(__strnlen_user_asm);
EXPORT_SYMBOL(csum_partial);
EXPORT_SYMBOL(csum_partial_copy_nocheck);
-EXPORT_SYMBOL(__csum_partial_copy_user);
+EXPORT_SYMBOL(__csum_partial_copy_kernel);
+EXPORT_SYMBOL(__csum_partial_copy_to_user);
+EXPORT_SYMBOL(__csum_partial_copy_from_user);
EXPORT_SYMBOL(invalid_pte_table);
#ifdef CONFIG_FUNCTION_TRACER
diff --git a/arch/mips/lib/csum_partial.S b/arch/mips/lib/csum_partial.S
index a6adffb..5d73d0d 100644
--- a/arch/mips/lib/csum_partial.S
+++ b/arch/mips/lib/csum_partial.S
@@ -8,6 +8,7 @@
* Copyright (C) 1998, 1999 Ralf Baechle
* Copyright (C) 1999 Silicon Graphics, Inc.
* Copyright (C) 2007 Maciej W. Rozycki
+ * Copyright (C) 2014 Imagination Technologies Ltd.
*/
#include <linux/errno.h>
#include <asm/asm.h>
@@ -296,7 +297,7 @@ LEAF(csum_partial)
* checksum and copy routines based on memcpy.S
*
* csum_partial_copy_nocheck(src, dst, len, sum)
- * __csum_partial_copy_user(src, dst, len, sum, errp)
+ * __csum_partial_copy_kernel(src, dst, len, sum, errp)
*
* See "Spec" in memcpy.S for details. Unlike __copy_user, all
* function in this file use the standard calling convention.
@@ -396,7 +397,9 @@ LEAF(csum_partial)
.set at=v1
#endif
-LEAF(__csum_partial_copy_user)
+LEAF(__csum_partial_copy_kernel)
+FEXPORT(__csum_partial_copy_to_user)
+FEXPORT(__csum_partial_copy_from_user)
PTR_ADDU AT, src, len /* See (1) above. */
#ifdef CONFIG_64BIT
move errptr, a4
@@ -757,4 +760,4 @@ EXC( lbu t1, 0(src), .Ll_exc)
jr ra
sw v1, (errptr)
.set pop
- END(__csum_partial_copy_user)
+ END(__csum_partial_copy_kernel)
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 33/58] MIPS: lib: csum_partial: Merge EXC and load/store macros
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Each load/store macro always adds an entry to the __ex_table
using the EXC macro. There are cases where a load instruction may
never fail such as when we are sure the load happens in the kernel
address space. Therefore, we merge these the EXC and LOADX/STOREX
macros into a single one. We also expand the argument list in the EXC
macro to make the macro more flexible. The extra 'type' argument is not
used by this commit, but it will be used when EVA support is added to
memcpy.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/csum_partial.S | 160 ++++++++++++++++++++++++-------------------
1 file changed, 91 insertions(+), 69 deletions(-)
diff --git a/arch/mips/lib/csum_partial.S b/arch/mips/lib/csum_partial.S
index 5d73d0d..bff5167 100644
--- a/arch/mips/lib/csum_partial.S
+++ b/arch/mips/lib/csum_partial.S
@@ -328,20 +328,39 @@ LEAF(csum_partial)
* These handlers do not need to overwrite any data.
*/
-#define EXC(inst_reg,addr,handler) \
-9: inst_reg, addr; \
+/* Instruction type */
+#define LD_INSN 1
+#define ST_INSN 2
+
+/*
+ * Wrapper to add an entry in the exception table
+ * in case the insn causes a memory exception.
+ * Arguments:
+ * insn : Load/store instruction
+ * type : Instruction type
+ * reg : Register
+ * addr : Address
+ * handler : Exception handler
+ */
+#define EXC(insn, type, reg, addr, handler) \
+9: insn reg, addr; \
.section __ex_table,"a"; \
PTR 9b, handler; \
.previous
+#undef LOAD
+
#ifdef USE_DOUBLE
-#define LOAD ld
-#define LOADL ldl
-#define LOADR ldr
-#define STOREL sdl
-#define STORER sdr
-#define STORE sd
+#define LOADK ld /* No exception */
+#define LOAD(reg, addr, handler) EXC(ld, LD_INSN, reg, addr, handler)
+#define LOADBU(reg, addr, handler) EXC(lbu, LD_INSN, reg, addr, handler)
+#define LOADL(reg, addr, handler) EXC(ldl, LD_INSN, reg, addr, handler)
+#define LOADR(reg, addr, handler) EXC(ldr, LD_INSN, reg, addr, handler)
+#define STOREB(reg, addr, handler) EXC(sb, ST_INSN, reg, addr, handler)
+#define STOREL(reg, addr, handler) EXC(sdl, ST_INSN, reg, addr, handler)
+#define STORER(reg, addr, handler) EXC(sdr, ST_INSN, reg, addr, handler)
+#define STORE(reg, addr, handler) EXC(sd, ST_INSN, reg, addr, handler)
#define ADD daddu
#define SUB dsubu
#define SRL dsrl
@@ -353,12 +372,15 @@ LEAF(csum_partial)
#else
-#define LOAD lw
-#define LOADL lwl
-#define LOADR lwr
-#define STOREL swl
-#define STORER swr
-#define STORE sw
+#define LOADK lw /* No exception */
+#define LOAD(reg, addr, handler) EXC(lw, LD_INSN, reg, addr, handler)
+#define LOADBU(reg, addr, handler) EXC(lbu, LD_INSN, reg, addr, handler)
+#define LOADL(reg, addr, handler) EXC(lwl, LD_INSN, reg, addr, handler)
+#define LOADR(reg, addr, handler) EXC(lwr, LD_INSN, reg, addr, handler)
+#define STOREB(reg, addr, handler) EXC(sb, ST_INSN, reg, addr, handler)
+#define STOREL(reg, addr, handler) EXC(swl, ST_INSN, reg, addr, handler)
+#define STORER(reg, addr, handler) EXC(swr, ST_INSN, reg, addr, handler)
+#define STORE(reg, addr, handler) EXC(sw, ST_INSN, reg, addr, handler)
#define ADD addu
#define SUB subu
#define SRL srl
@@ -439,31 +461,31 @@ FEXPORT(csum_partial_copy_nocheck)
SUB len, 8*NBYTES # subtract here for bgez loop
.align 4
1:
-EXC( LOAD t0, UNIT(0)(src), .Ll_exc)
-EXC( LOAD t1, UNIT(1)(src), .Ll_exc_copy)
-EXC( LOAD t2, UNIT(2)(src), .Ll_exc_copy)
-EXC( LOAD t3, UNIT(3)(src), .Ll_exc_copy)
-EXC( LOAD t4, UNIT(4)(src), .Ll_exc_copy)
-EXC( LOAD t5, UNIT(5)(src), .Ll_exc_copy)
-EXC( LOAD t6, UNIT(6)(src), .Ll_exc_copy)
-EXC( LOAD t7, UNIT(7)(src), .Ll_exc_copy)
+ LOAD(t0, UNIT(0)(src), .Ll_exc)
+ LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
+ LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
+ LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
+ LOAD(t4, UNIT(4)(src), .Ll_exc_copy)
+ LOAD(t5, UNIT(5)(src), .Ll_exc_copy)
+ LOAD(t6, UNIT(6)(src), .Ll_exc_copy)
+ LOAD(t7, UNIT(7)(src), .Ll_exc_copy)
SUB len, len, 8*NBYTES
ADD src, src, 8*NBYTES
-EXC( STORE t0, UNIT(0)(dst), .Ls_exc)
+ STORE(t0, UNIT(0)(dst), .Ls_exc)
ADDC(sum, t0)
-EXC( STORE t1, UNIT(1)(dst), .Ls_exc)
+ STORE(t1, UNIT(1)(dst), .Ls_exc)
ADDC(sum, t1)
-EXC( STORE t2, UNIT(2)(dst), .Ls_exc)
+ STORE(t2, UNIT(2)(dst), .Ls_exc)
ADDC(sum, t2)
-EXC( STORE t3, UNIT(3)(dst), .Ls_exc)
+ STORE(t3, UNIT(3)(dst), .Ls_exc)
ADDC(sum, t3)
-EXC( STORE t4, UNIT(4)(dst), .Ls_exc)
+ STORE(t4, UNIT(4)(dst), .Ls_exc)
ADDC(sum, t4)
-EXC( STORE t5, UNIT(5)(dst), .Ls_exc)
+ STORE(t5, UNIT(5)(dst), .Ls_exc)
ADDC(sum, t5)
-EXC( STORE t6, UNIT(6)(dst), .Ls_exc)
+ STORE(t6, UNIT(6)(dst), .Ls_exc)
ADDC(sum, t6)
-EXC( STORE t7, UNIT(7)(dst), .Ls_exc)
+ STORE(t7, UNIT(7)(dst), .Ls_exc)
ADDC(sum, t7)
.set reorder /* DADDI_WAR */
ADD dst, dst, 8*NBYTES
@@ -483,19 +505,19 @@ EXC( STORE t7, UNIT(7)(dst), .Ls_exc)
/*
* len >= 4*NBYTES
*/
-EXC( LOAD t0, UNIT(0)(src), .Ll_exc)
-EXC( LOAD t1, UNIT(1)(src), .Ll_exc_copy)
-EXC( LOAD t2, UNIT(2)(src), .Ll_exc_copy)
-EXC( LOAD t3, UNIT(3)(src), .Ll_exc_copy)
+ LOAD(t0, UNIT(0)(src), .Ll_exc)
+ LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
+ LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
+ LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
SUB len, len, 4*NBYTES
ADD src, src, 4*NBYTES
-EXC( STORE t0, UNIT(0)(dst), .Ls_exc)
+ STORE(t0, UNIT(0)(dst), .Ls_exc)
ADDC(sum, t0)
-EXC( STORE t1, UNIT(1)(dst), .Ls_exc)
+ STORE(t1, UNIT(1)(dst), .Ls_exc)
ADDC(sum, t1)
-EXC( STORE t2, UNIT(2)(dst), .Ls_exc)
+ STORE(t2, UNIT(2)(dst), .Ls_exc)
ADDC(sum, t2)
-EXC( STORE t3, UNIT(3)(dst), .Ls_exc)
+ STORE(t3, UNIT(3)(dst), .Ls_exc)
ADDC(sum, t3)
.set reorder /* DADDI_WAR */
ADD dst, dst, 4*NBYTES
@@ -508,10 +530,10 @@ EXC( STORE t3, UNIT(3)(dst), .Ls_exc)
beq rem, len, .Lcopy_bytes
nop
1:
-EXC( LOAD t0, 0(src), .Ll_exc)
+ LOAD(t0, 0(src), .Ll_exc)
ADD src, src, NBYTES
SUB len, len, NBYTES
-EXC( STORE t0, 0(dst), .Ls_exc)
+ STORE(t0, 0(dst), .Ls_exc)
ADDC(sum, t0)
.set reorder /* DADDI_WAR */
ADD dst, dst, NBYTES
@@ -534,10 +556,10 @@ EXC( STORE t0, 0(dst), .Ls_exc)
ADD t1, dst, len # t1 is just past last byte of dst
li bits, 8*NBYTES
SLL rem, len, 3 # rem = number of bits to keep
-EXC( LOAD t0, 0(src), .Ll_exc)
+ LOAD(t0, 0(src), .Ll_exc)
SUB bits, bits, rem # bits = number of bits to discard
SHIFT_DISCARD t0, t0, bits
-EXC( STREST t0, -1(t1), .Ls_exc)
+ STREST(t0, -1(t1), .Ls_exc)
SHIFT_DISCARD_REVERT t0, t0, bits
.set reorder
ADDC(sum, t0)
@@ -554,12 +576,12 @@ EXC( STREST t0, -1(t1), .Ls_exc)
* Set match = (src and dst have same alignment)
*/
#define match rem
-EXC( LDFIRST t3, FIRST(0)(src), .Ll_exc)
+ LDFIRST(t3, FIRST(0)(src), .Ll_exc)
ADD t2, zero, NBYTES
-EXC( LDREST t3, REST(0)(src), .Ll_exc_copy)
+ LDREST(t3, REST(0)(src), .Ll_exc_copy)
SUB t2, t2, t1 # t2 = number of bytes copied
xor match, t0, t1
-EXC( STFIRST t3, FIRST(0)(dst), .Ls_exc)
+ STFIRST(t3, FIRST(0)(dst), .Ls_exc)
SLL t4, t1, 3 # t4 = number of bits to discard
SHIFT_DISCARD t3, t3, t4
/* no SHIFT_DISCARD_REVERT to handle odd buffer properly */
@@ -581,26 +603,26 @@ EXC( STFIRST t3, FIRST(0)(dst), .Ls_exc)
* It's OK to load FIRST(N+1) before REST(N) because the two addresses
* are to the same unit (unless src is aligned, but it's not).
*/
-EXC( LDFIRST t0, FIRST(0)(src), .Ll_exc)
-EXC( LDFIRST t1, FIRST(1)(src), .Ll_exc_copy)
+ LDFIRST(t0, FIRST(0)(src), .Ll_exc)
+ LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy)
SUB len, len, 4*NBYTES
-EXC( LDREST t0, REST(0)(src), .Ll_exc_copy)
-EXC( LDREST t1, REST(1)(src), .Ll_exc_copy)
-EXC( LDFIRST t2, FIRST(2)(src), .Ll_exc_copy)
-EXC( LDFIRST t3, FIRST(3)(src), .Ll_exc_copy)
-EXC( LDREST t2, REST(2)(src), .Ll_exc_copy)
-EXC( LDREST t3, REST(3)(src), .Ll_exc_copy)
+ LDREST(t0, REST(0)(src), .Ll_exc_copy)
+ LDREST(t1, REST(1)(src), .Ll_exc_copy)
+ LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy)
+ LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy)
+ LDREST(t2, REST(2)(src), .Ll_exc_copy)
+ LDREST(t3, REST(3)(src), .Ll_exc_copy)
ADD src, src, 4*NBYTES
#ifdef CONFIG_CPU_SB1
nop # improves slotting
#endif
-EXC( STORE t0, UNIT(0)(dst), .Ls_exc)
+ STORE(t0, UNIT(0)(dst), .Ls_exc)
ADDC(sum, t0)
-EXC( STORE t1, UNIT(1)(dst), .Ls_exc)
+ STORE(t1, UNIT(1)(dst), .Ls_exc)
ADDC(sum, t1)
-EXC( STORE t2, UNIT(2)(dst), .Ls_exc)
+ STORE(t2, UNIT(2)(dst), .Ls_exc)
ADDC(sum, t2)
-EXC( STORE t3, UNIT(3)(dst), .Ls_exc)
+ STORE(t3, UNIT(3)(dst), .Ls_exc)
ADDC(sum, t3)
.set reorder /* DADDI_WAR */
ADD dst, dst, 4*NBYTES
@@ -613,11 +635,11 @@ EXC( STORE t3, UNIT(3)(dst), .Ls_exc)
beq rem, len, .Lcopy_bytes
nop
1:
-EXC( LDFIRST t0, FIRST(0)(src), .Ll_exc)
-EXC( LDREST t0, REST(0)(src), .Ll_exc_copy)
+ LDFIRST(t0, FIRST(0)(src), .Ll_exc)
+ LDREST(t0, REST(0)(src), .Ll_exc_copy)
ADD src, src, NBYTES
SUB len, len, NBYTES
-EXC( STORE t0, 0(dst), .Ls_exc)
+ STORE(t0, 0(dst), .Ls_exc)
ADDC(sum, t0)
.set reorder /* DADDI_WAR */
ADD dst, dst, NBYTES
@@ -640,9 +662,9 @@ EXC( STORE t0, 0(dst), .Ls_exc)
li t3, SHIFT_START # shift
/* use .Ll_exc_copy here to return correct sum on fault */
#define COPY_BYTE(N) \
-EXC( lbu t0, N(src), .Ll_exc_copy); \
+ LOADBU(t0, N(src), .Ll_exc_copy); \
SUB len, len, 1; \
-EXC( sb t0, N(dst), .Ls_exc); \
+ STOREB(t0, N(dst), .Ls_exc); \
SLLV t0, t0, t3; \
addu t3, SHIFT_INC; \
beqz len, .Lcopy_bytes_done; \
@@ -656,9 +678,9 @@ EXC( sb t0, N(dst), .Ls_exc); \
COPY_BYTE(4)
COPY_BYTE(5)
#endif
-EXC( lbu t0, NBYTES-2(src), .Ll_exc_copy)
+ LOADBU(t0, NBYTES-2(src), .Ll_exc_copy)
SUB len, len, 1
-EXC( sb t0, NBYTES-2(dst), .Ls_exc)
+ STOREB(t0, NBYTES-2(dst), .Ls_exc)
SLLV t0, t0, t3
or t2, t0
.Lcopy_bytes_done:
@@ -703,11 +725,11 @@ EXC( sb t0, NBYTES-2(dst), .Ls_exc)
*
* Assumes src < THREAD_BUADDR($28)
*/
- LOAD t0, TI_TASK($28)
+ LOADK t0, TI_TASK($28)
li t2, SHIFT_START
- LOAD t0, THREAD_BUADDR(t0)
+ LOADK t0, THREAD_BUADDR(t0)
1:
-EXC( lbu t1, 0(src), .Ll_exc)
+ LOADBU(t1, 0(src), .Ll_exc)
ADD src, src, 1
sb t1, 0(dst) # can't fault -- we're copy_from_user
SLLV t1, t1, t2
@@ -718,9 +740,9 @@ EXC( lbu t1, 0(src), .Ll_exc)
bne src, t0, 1b
.set noreorder
.Ll_exc:
- LOAD t0, TI_TASK($28)
+ LOADK t0, TI_TASK($28)
nop
- LOAD t0, THREAD_BUADDR(t0) # t0 is just past last good address
+ LOADK t0, THREAD_BUADDR(t0) # t0 is just past last good address
nop
SUB len, AT, t0 # len number of uncopied bytes
/*
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 33/58] MIPS: lib: csum_partial: Merge EXC and load/store macros
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Each load/store macro always adds an entry to the __ex_table
using the EXC macro. There are cases where a load instruction may
never fail such as when we are sure the load happens in the kernel
address space. Therefore, we merge these the EXC and LOADX/STOREX
macros into a single one. We also expand the argument list in the EXC
macro to make the macro more flexible. The extra 'type' argument is not
used by this commit, but it will be used when EVA support is added to
memcpy.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/csum_partial.S | 160 ++++++++++++++++++++++++-------------------
1 file changed, 91 insertions(+), 69 deletions(-)
diff --git a/arch/mips/lib/csum_partial.S b/arch/mips/lib/csum_partial.S
index 5d73d0d..bff5167 100644
--- a/arch/mips/lib/csum_partial.S
+++ b/arch/mips/lib/csum_partial.S
@@ -328,20 +328,39 @@ LEAF(csum_partial)
* These handlers do not need to overwrite any data.
*/
-#define EXC(inst_reg,addr,handler) \
-9: inst_reg, addr; \
+/* Instruction type */
+#define LD_INSN 1
+#define ST_INSN 2
+
+/*
+ * Wrapper to add an entry in the exception table
+ * in case the insn causes a memory exception.
+ * Arguments:
+ * insn : Load/store instruction
+ * type : Instruction type
+ * reg : Register
+ * addr : Address
+ * handler : Exception handler
+ */
+#define EXC(insn, type, reg, addr, handler) \
+9: insn reg, addr; \
.section __ex_table,"a"; \
PTR 9b, handler; \
.previous
+#undef LOAD
+
#ifdef USE_DOUBLE
-#define LOAD ld
-#define LOADL ldl
-#define LOADR ldr
-#define STOREL sdl
-#define STORER sdr
-#define STORE sd
+#define LOADK ld /* No exception */
+#define LOAD(reg, addr, handler) EXC(ld, LD_INSN, reg, addr, handler)
+#define LOADBU(reg, addr, handler) EXC(lbu, LD_INSN, reg, addr, handler)
+#define LOADL(reg, addr, handler) EXC(ldl, LD_INSN, reg, addr, handler)
+#define LOADR(reg, addr, handler) EXC(ldr, LD_INSN, reg, addr, handler)
+#define STOREB(reg, addr, handler) EXC(sb, ST_INSN, reg, addr, handler)
+#define STOREL(reg, addr, handler) EXC(sdl, ST_INSN, reg, addr, handler)
+#define STORER(reg, addr, handler) EXC(sdr, ST_INSN, reg, addr, handler)
+#define STORE(reg, addr, handler) EXC(sd, ST_INSN, reg, addr, handler)
#define ADD daddu
#define SUB dsubu
#define SRL dsrl
@@ -353,12 +372,15 @@ LEAF(csum_partial)
#else
-#define LOAD lw
-#define LOADL lwl
-#define LOADR lwr
-#define STOREL swl
-#define STORER swr
-#define STORE sw
+#define LOADK lw /* No exception */
+#define LOAD(reg, addr, handler) EXC(lw, LD_INSN, reg, addr, handler)
+#define LOADBU(reg, addr, handler) EXC(lbu, LD_INSN, reg, addr, handler)
+#define LOADL(reg, addr, handler) EXC(lwl, LD_INSN, reg, addr, handler)
+#define LOADR(reg, addr, handler) EXC(lwr, LD_INSN, reg, addr, handler)
+#define STOREB(reg, addr, handler) EXC(sb, ST_INSN, reg, addr, handler)
+#define STOREL(reg, addr, handler) EXC(swl, ST_INSN, reg, addr, handler)
+#define STORER(reg, addr, handler) EXC(swr, ST_INSN, reg, addr, handler)
+#define STORE(reg, addr, handler) EXC(sw, ST_INSN, reg, addr, handler)
#define ADD addu
#define SUB subu
#define SRL srl
@@ -439,31 +461,31 @@ FEXPORT(csum_partial_copy_nocheck)
SUB len, 8*NBYTES # subtract here for bgez loop
.align 4
1:
-EXC( LOAD t0, UNIT(0)(src), .Ll_exc)
-EXC( LOAD t1, UNIT(1)(src), .Ll_exc_copy)
-EXC( LOAD t2, UNIT(2)(src), .Ll_exc_copy)
-EXC( LOAD t3, UNIT(3)(src), .Ll_exc_copy)
-EXC( LOAD t4, UNIT(4)(src), .Ll_exc_copy)
-EXC( LOAD t5, UNIT(5)(src), .Ll_exc_copy)
-EXC( LOAD t6, UNIT(6)(src), .Ll_exc_copy)
-EXC( LOAD t7, UNIT(7)(src), .Ll_exc_copy)
+ LOAD(t0, UNIT(0)(src), .Ll_exc)
+ LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
+ LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
+ LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
+ LOAD(t4, UNIT(4)(src), .Ll_exc_copy)
+ LOAD(t5, UNIT(5)(src), .Ll_exc_copy)
+ LOAD(t6, UNIT(6)(src), .Ll_exc_copy)
+ LOAD(t7, UNIT(7)(src), .Ll_exc_copy)
SUB len, len, 8*NBYTES
ADD src, src, 8*NBYTES
-EXC( STORE t0, UNIT(0)(dst), .Ls_exc)
+ STORE(t0, UNIT(0)(dst), .Ls_exc)
ADDC(sum, t0)
-EXC( STORE t1, UNIT(1)(dst), .Ls_exc)
+ STORE(t1, UNIT(1)(dst), .Ls_exc)
ADDC(sum, t1)
-EXC( STORE t2, UNIT(2)(dst), .Ls_exc)
+ STORE(t2, UNIT(2)(dst), .Ls_exc)
ADDC(sum, t2)
-EXC( STORE t3, UNIT(3)(dst), .Ls_exc)
+ STORE(t3, UNIT(3)(dst), .Ls_exc)
ADDC(sum, t3)
-EXC( STORE t4, UNIT(4)(dst), .Ls_exc)
+ STORE(t4, UNIT(4)(dst), .Ls_exc)
ADDC(sum, t4)
-EXC( STORE t5, UNIT(5)(dst), .Ls_exc)
+ STORE(t5, UNIT(5)(dst), .Ls_exc)
ADDC(sum, t5)
-EXC( STORE t6, UNIT(6)(dst), .Ls_exc)
+ STORE(t6, UNIT(6)(dst), .Ls_exc)
ADDC(sum, t6)
-EXC( STORE t7, UNIT(7)(dst), .Ls_exc)
+ STORE(t7, UNIT(7)(dst), .Ls_exc)
ADDC(sum, t7)
.set reorder /* DADDI_WAR */
ADD dst, dst, 8*NBYTES
@@ -483,19 +505,19 @@ EXC( STORE t7, UNIT(7)(dst), .Ls_exc)
/*
* len >= 4*NBYTES
*/
-EXC( LOAD t0, UNIT(0)(src), .Ll_exc)
-EXC( LOAD t1, UNIT(1)(src), .Ll_exc_copy)
-EXC( LOAD t2, UNIT(2)(src), .Ll_exc_copy)
-EXC( LOAD t3, UNIT(3)(src), .Ll_exc_copy)
+ LOAD(t0, UNIT(0)(src), .Ll_exc)
+ LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
+ LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
+ LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
SUB len, len, 4*NBYTES
ADD src, src, 4*NBYTES
-EXC( STORE t0, UNIT(0)(dst), .Ls_exc)
+ STORE(t0, UNIT(0)(dst), .Ls_exc)
ADDC(sum, t0)
-EXC( STORE t1, UNIT(1)(dst), .Ls_exc)
+ STORE(t1, UNIT(1)(dst), .Ls_exc)
ADDC(sum, t1)
-EXC( STORE t2, UNIT(2)(dst), .Ls_exc)
+ STORE(t2, UNIT(2)(dst), .Ls_exc)
ADDC(sum, t2)
-EXC( STORE t3, UNIT(3)(dst), .Ls_exc)
+ STORE(t3, UNIT(3)(dst), .Ls_exc)
ADDC(sum, t3)
.set reorder /* DADDI_WAR */
ADD dst, dst, 4*NBYTES
@@ -508,10 +530,10 @@ EXC( STORE t3, UNIT(3)(dst), .Ls_exc)
beq rem, len, .Lcopy_bytes
nop
1:
-EXC( LOAD t0, 0(src), .Ll_exc)
+ LOAD(t0, 0(src), .Ll_exc)
ADD src, src, NBYTES
SUB len, len, NBYTES
-EXC( STORE t0, 0(dst), .Ls_exc)
+ STORE(t0, 0(dst), .Ls_exc)
ADDC(sum, t0)
.set reorder /* DADDI_WAR */
ADD dst, dst, NBYTES
@@ -534,10 +556,10 @@ EXC( STORE t0, 0(dst), .Ls_exc)
ADD t1, dst, len # t1 is just past last byte of dst
li bits, 8*NBYTES
SLL rem, len, 3 # rem = number of bits to keep
-EXC( LOAD t0, 0(src), .Ll_exc)
+ LOAD(t0, 0(src), .Ll_exc)
SUB bits, bits, rem # bits = number of bits to discard
SHIFT_DISCARD t0, t0, bits
-EXC( STREST t0, -1(t1), .Ls_exc)
+ STREST(t0, -1(t1), .Ls_exc)
SHIFT_DISCARD_REVERT t0, t0, bits
.set reorder
ADDC(sum, t0)
@@ -554,12 +576,12 @@ EXC( STREST t0, -1(t1), .Ls_exc)
* Set match = (src and dst have same alignment)
*/
#define match rem
-EXC( LDFIRST t3, FIRST(0)(src), .Ll_exc)
+ LDFIRST(t3, FIRST(0)(src), .Ll_exc)
ADD t2, zero, NBYTES
-EXC( LDREST t3, REST(0)(src), .Ll_exc_copy)
+ LDREST(t3, REST(0)(src), .Ll_exc_copy)
SUB t2, t2, t1 # t2 = number of bytes copied
xor match, t0, t1
-EXC( STFIRST t3, FIRST(0)(dst), .Ls_exc)
+ STFIRST(t3, FIRST(0)(dst), .Ls_exc)
SLL t4, t1, 3 # t4 = number of bits to discard
SHIFT_DISCARD t3, t3, t4
/* no SHIFT_DISCARD_REVERT to handle odd buffer properly */
@@ -581,26 +603,26 @@ EXC( STFIRST t3, FIRST(0)(dst), .Ls_exc)
* It's OK to load FIRST(N+1) before REST(N) because the two addresses
* are to the same unit (unless src is aligned, but it's not).
*/
-EXC( LDFIRST t0, FIRST(0)(src), .Ll_exc)
-EXC( LDFIRST t1, FIRST(1)(src), .Ll_exc_copy)
+ LDFIRST(t0, FIRST(0)(src), .Ll_exc)
+ LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy)
SUB len, len, 4*NBYTES
-EXC( LDREST t0, REST(0)(src), .Ll_exc_copy)
-EXC( LDREST t1, REST(1)(src), .Ll_exc_copy)
-EXC( LDFIRST t2, FIRST(2)(src), .Ll_exc_copy)
-EXC( LDFIRST t3, FIRST(3)(src), .Ll_exc_copy)
-EXC( LDREST t2, REST(2)(src), .Ll_exc_copy)
-EXC( LDREST t3, REST(3)(src), .Ll_exc_copy)
+ LDREST(t0, REST(0)(src), .Ll_exc_copy)
+ LDREST(t1, REST(1)(src), .Ll_exc_copy)
+ LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy)
+ LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy)
+ LDREST(t2, REST(2)(src), .Ll_exc_copy)
+ LDREST(t3, REST(3)(src), .Ll_exc_copy)
ADD src, src, 4*NBYTES
#ifdef CONFIG_CPU_SB1
nop # improves slotting
#endif
-EXC( STORE t0, UNIT(0)(dst), .Ls_exc)
+ STORE(t0, UNIT(0)(dst), .Ls_exc)
ADDC(sum, t0)
-EXC( STORE t1, UNIT(1)(dst), .Ls_exc)
+ STORE(t1, UNIT(1)(dst), .Ls_exc)
ADDC(sum, t1)
-EXC( STORE t2, UNIT(2)(dst), .Ls_exc)
+ STORE(t2, UNIT(2)(dst), .Ls_exc)
ADDC(sum, t2)
-EXC( STORE t3, UNIT(3)(dst), .Ls_exc)
+ STORE(t3, UNIT(3)(dst), .Ls_exc)
ADDC(sum, t3)
.set reorder /* DADDI_WAR */
ADD dst, dst, 4*NBYTES
@@ -613,11 +635,11 @@ EXC( STORE t3, UNIT(3)(dst), .Ls_exc)
beq rem, len, .Lcopy_bytes
nop
1:
-EXC( LDFIRST t0, FIRST(0)(src), .Ll_exc)
-EXC( LDREST t0, REST(0)(src), .Ll_exc_copy)
+ LDFIRST(t0, FIRST(0)(src), .Ll_exc)
+ LDREST(t0, REST(0)(src), .Ll_exc_copy)
ADD src, src, NBYTES
SUB len, len, NBYTES
-EXC( STORE t0, 0(dst), .Ls_exc)
+ STORE(t0, 0(dst), .Ls_exc)
ADDC(sum, t0)
.set reorder /* DADDI_WAR */
ADD dst, dst, NBYTES
@@ -640,9 +662,9 @@ EXC( STORE t0, 0(dst), .Ls_exc)
li t3, SHIFT_START # shift
/* use .Ll_exc_copy here to return correct sum on fault */
#define COPY_BYTE(N) \
-EXC( lbu t0, N(src), .Ll_exc_copy); \
+ LOADBU(t0, N(src), .Ll_exc_copy); \
SUB len, len, 1; \
-EXC( sb t0, N(dst), .Ls_exc); \
+ STOREB(t0, N(dst), .Ls_exc); \
SLLV t0, t0, t3; \
addu t3, SHIFT_INC; \
beqz len, .Lcopy_bytes_done; \
@@ -656,9 +678,9 @@ EXC( sb t0, N(dst), .Ls_exc); \
COPY_BYTE(4)
COPY_BYTE(5)
#endif
-EXC( lbu t0, NBYTES-2(src), .Ll_exc_copy)
+ LOADBU(t0, NBYTES-2(src), .Ll_exc_copy)
SUB len, len, 1
-EXC( sb t0, NBYTES-2(dst), .Ls_exc)
+ STOREB(t0, NBYTES-2(dst), .Ls_exc)
SLLV t0, t0, t3
or t2, t0
.Lcopy_bytes_done:
@@ -703,11 +725,11 @@ EXC( sb t0, NBYTES-2(dst), .Ls_exc)
*
* Assumes src < THREAD_BUADDR($28)
*/
- LOAD t0, TI_TASK($28)
+ LOADK t0, TI_TASK($28)
li t2, SHIFT_START
- LOAD t0, THREAD_BUADDR(t0)
+ LOADK t0, THREAD_BUADDR(t0)
1:
-EXC( lbu t1, 0(src), .Ll_exc)
+ LOADBU(t1, 0(src), .Ll_exc)
ADD src, src, 1
sb t1, 0(dst) # can't fault -- we're copy_from_user
SLLV t1, t1, t2
@@ -718,9 +740,9 @@ EXC( lbu t1, 0(src), .Ll_exc)
bne src, t0, 1b
.set noreorder
.Ll_exc:
- LOAD t0, TI_TASK($28)
+ LOADK t0, TI_TASK($28)
nop
- LOAD t0, THREAD_BUADDR(t0) # t0 is just past last good address
+ LOADK t0, THREAD_BUADDR(t0) # t0 is just past last good address
nop
SUB len, AT, t0 # len number of uncopied bytes
/*
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 34/58] MIPS: lib: csum_partial: Add macro to build csum_partial symbols
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
In preparation for EVA support, we use a macro to build the
__csum_partial_copy_user main code so it can be shared across
multiple implementations. EVA uses the same code but it replaces
the load/store/prefetch instructions with the EVA specific ones
therefore using a macro avoids unnecessary code duplications.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/csum_partial.S | 200 +++++++++++++++++++++++--------------------
1 file changed, 108 insertions(+), 92 deletions(-)
diff --git a/arch/mips/lib/csum_partial.S b/arch/mips/lib/csum_partial.S
index bff5167..62c8768 100644
--- a/arch/mips/lib/csum_partial.S
+++ b/arch/mips/lib/csum_partial.S
@@ -331,6 +331,10 @@ LEAF(csum_partial)
/* Instruction type */
#define LD_INSN 1
#define ST_INSN 2
+#define LEGACY_MODE 1
+#define EVA_MODE 2
+#define USEROP 1
+#define KERNELOP 2
/*
* Wrapper to add an entry in the exception table
@@ -343,10 +347,12 @@ LEAF(csum_partial)
* handler : Exception handler
*/
#define EXC(insn, type, reg, addr, handler) \
-9: insn reg, addr; \
- .section __ex_table,"a"; \
- PTR 9b, handler; \
- .previous
+ .if \mode == LEGACY_MODE; \
+9: insn reg, addr; \
+ .section __ex_table,"a"; \
+ PTR 9b, handler; \
+ .previous; \
+ .endif
#undef LOAD
@@ -419,16 +425,20 @@ LEAF(csum_partial)
.set at=v1
#endif
-LEAF(__csum_partial_copy_kernel)
-FEXPORT(__csum_partial_copy_to_user)
-FEXPORT(__csum_partial_copy_from_user)
+ .macro __BUILD_CSUM_PARTIAL_COPY_USER mode, from, to, __nocheck
+
PTR_ADDU AT, src, len /* See (1) above. */
+ /* initialize __nocheck if this the first time we execute this
+ * macro
+ */
#ifdef CONFIG_64BIT
move errptr, a4
#else
lw errptr, 16(sp)
#endif
-FEXPORT(csum_partial_copy_nocheck)
+ .if \__nocheck == 1
+ FEXPORT(csum_partial_copy_nocheck)
+ .endif
move sum, zero
move odd, zero
/*
@@ -444,48 +454,48 @@ FEXPORT(csum_partial_copy_nocheck)
*/
sltu t2, len, NBYTES
and t1, dst, ADDRMASK
- bnez t2, .Lcopy_bytes_checklen
+ bnez t2, .Lcopy_bytes_checklen\@
and t0, src, ADDRMASK
andi odd, dst, 0x1 /* odd buffer? */
- bnez t1, .Ldst_unaligned
+ bnez t1, .Ldst_unaligned\@
nop
- bnez t0, .Lsrc_unaligned_dst_aligned
+ bnez t0, .Lsrc_unaligned_dst_aligned\@
/*
* use delay slot for fall-through
* src and dst are aligned; need to compute rem
*/
-.Lboth_aligned:
+.Lboth_aligned\@:
SRL t0, len, LOG_NBYTES+3 # +3 for 8 units/iter
- beqz t0, .Lcleanup_both_aligned # len < 8*NBYTES
+ beqz t0, .Lcleanup_both_aligned\@ # len < 8*NBYTES
nop
SUB len, 8*NBYTES # subtract here for bgez loop
.align 4
1:
- LOAD(t0, UNIT(0)(src), .Ll_exc)
- LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
- LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
- LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
- LOAD(t4, UNIT(4)(src), .Ll_exc_copy)
- LOAD(t5, UNIT(5)(src), .Ll_exc_copy)
- LOAD(t6, UNIT(6)(src), .Ll_exc_copy)
- LOAD(t7, UNIT(7)(src), .Ll_exc_copy)
+ LOAD(t0, UNIT(0)(src), .Ll_exc\@)
+ LOAD(t1, UNIT(1)(src), .Ll_exc_copy\@)
+ LOAD(t2, UNIT(2)(src), .Ll_exc_copy\@)
+ LOAD(t3, UNIT(3)(src), .Ll_exc_copy\@)
+ LOAD(t4, UNIT(4)(src), .Ll_exc_copy\@)
+ LOAD(t5, UNIT(5)(src), .Ll_exc_copy\@)
+ LOAD(t6, UNIT(6)(src), .Ll_exc_copy\@)
+ LOAD(t7, UNIT(7)(src), .Ll_exc_copy\@)
SUB len, len, 8*NBYTES
ADD src, src, 8*NBYTES
- STORE(t0, UNIT(0)(dst), .Ls_exc)
+ STORE(t0, UNIT(0)(dst), .Ls_exc\@)
ADDC(sum, t0)
- STORE(t1, UNIT(1)(dst), .Ls_exc)
+ STORE(t1, UNIT(1)(dst), .Ls_exc\@)
ADDC(sum, t1)
- STORE(t2, UNIT(2)(dst), .Ls_exc)
+ STORE(t2, UNIT(2)(dst), .Ls_exc\@)
ADDC(sum, t2)
- STORE(t3, UNIT(3)(dst), .Ls_exc)
+ STORE(t3, UNIT(3)(dst), .Ls_exc\@)
ADDC(sum, t3)
- STORE(t4, UNIT(4)(dst), .Ls_exc)
+ STORE(t4, UNIT(4)(dst), .Ls_exc\@)
ADDC(sum, t4)
- STORE(t5, UNIT(5)(dst), .Ls_exc)
+ STORE(t5, UNIT(5)(dst), .Ls_exc\@)
ADDC(sum, t5)
- STORE(t6, UNIT(6)(dst), .Ls_exc)
+ STORE(t6, UNIT(6)(dst), .Ls_exc\@)
ADDC(sum, t6)
- STORE(t7, UNIT(7)(dst), .Ls_exc)
+ STORE(t7, UNIT(7)(dst), .Ls_exc\@)
ADDC(sum, t7)
.set reorder /* DADDI_WAR */
ADD dst, dst, 8*NBYTES
@@ -496,44 +506,44 @@ FEXPORT(csum_partial_copy_nocheck)
/*
* len == the number of bytes left to copy < 8*NBYTES
*/
-.Lcleanup_both_aligned:
+.Lcleanup_both_aligned\@:
#define rem t7
- beqz len, .Ldone
+ beqz len, .Ldone\@
sltu t0, len, 4*NBYTES
- bnez t0, .Lless_than_4units
+ bnez t0, .Lless_than_4units\@
and rem, len, (NBYTES-1) # rem = len % NBYTES
/*
* len >= 4*NBYTES
*/
- LOAD(t0, UNIT(0)(src), .Ll_exc)
- LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
- LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
- LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
+ LOAD(t0, UNIT(0)(src), .Ll_exc\@)
+ LOAD(t1, UNIT(1)(src), .Ll_exc_copy\@)
+ LOAD(t2, UNIT(2)(src), .Ll_exc_copy\@)
+ LOAD(t3, UNIT(3)(src), .Ll_exc_copy\@)
SUB len, len, 4*NBYTES
ADD src, src, 4*NBYTES
- STORE(t0, UNIT(0)(dst), .Ls_exc)
+ STORE(t0, UNIT(0)(dst), .Ls_exc\@)
ADDC(sum, t0)
- STORE(t1, UNIT(1)(dst), .Ls_exc)
+ STORE(t1, UNIT(1)(dst), .Ls_exc\@)
ADDC(sum, t1)
- STORE(t2, UNIT(2)(dst), .Ls_exc)
+ STORE(t2, UNIT(2)(dst), .Ls_exc\@)
ADDC(sum, t2)
- STORE(t3, UNIT(3)(dst), .Ls_exc)
+ STORE(t3, UNIT(3)(dst), .Ls_exc\@)
ADDC(sum, t3)
.set reorder /* DADDI_WAR */
ADD dst, dst, 4*NBYTES
- beqz len, .Ldone
+ beqz len, .Ldone\@
.set noreorder
-.Lless_than_4units:
+.Lless_than_4units\@:
/*
* rem = len % NBYTES
*/
- beq rem, len, .Lcopy_bytes
+ beq rem, len, .Lcopy_bytes\@
nop
1:
- LOAD(t0, 0(src), .Ll_exc)
+ LOAD(t0, 0(src), .Ll_exc\@)
ADD src, src, NBYTES
SUB len, len, NBYTES
- STORE(t0, 0(dst), .Ls_exc)
+ STORE(t0, 0(dst), .Ls_exc\@)
ADDC(sum, t0)
.set reorder /* DADDI_WAR */
ADD dst, dst, NBYTES
@@ -552,20 +562,20 @@ FEXPORT(csum_partial_copy_nocheck)
* more instruction-level parallelism.
*/
#define bits t2
- beqz len, .Ldone
+ beqz len, .Ldone\@
ADD t1, dst, len # t1 is just past last byte of dst
li bits, 8*NBYTES
SLL rem, len, 3 # rem = number of bits to keep
- LOAD(t0, 0(src), .Ll_exc)
+ LOAD(t0, 0(src), .Ll_exc\@)
SUB bits, bits, rem # bits = number of bits to discard
SHIFT_DISCARD t0, t0, bits
- STREST(t0, -1(t1), .Ls_exc)
+ STREST(t0, -1(t1), .Ls_exc\@)
SHIFT_DISCARD_REVERT t0, t0, bits
.set reorder
ADDC(sum, t0)
- b .Ldone
+ b .Ldone\@
.set noreorder
-.Ldst_unaligned:
+.Ldst_unaligned\@:
/*
* dst is unaligned
* t0 = src & ADDRMASK
@@ -576,25 +586,25 @@ FEXPORT(csum_partial_copy_nocheck)
* Set match = (src and dst have same alignment)
*/
#define match rem
- LDFIRST(t3, FIRST(0)(src), .Ll_exc)
+ LDFIRST(t3, FIRST(0)(src), .Ll_exc\@)
ADD t2, zero, NBYTES
- LDREST(t3, REST(0)(src), .Ll_exc_copy)
+ LDREST(t3, REST(0)(src), .Ll_exc_copy\@)
SUB t2, t2, t1 # t2 = number of bytes copied
xor match, t0, t1
- STFIRST(t3, FIRST(0)(dst), .Ls_exc)
+ STFIRST(t3, FIRST(0)(dst), .Ls_exc\@)
SLL t4, t1, 3 # t4 = number of bits to discard
SHIFT_DISCARD t3, t3, t4
/* no SHIFT_DISCARD_REVERT to handle odd buffer properly */
ADDC(sum, t3)
- beq len, t2, .Ldone
+ beq len, t2, .Ldone\@
SUB len, len, t2
ADD dst, dst, t2
- beqz match, .Lboth_aligned
+ beqz match, .Lboth_aligned\@
ADD src, src, t2
-.Lsrc_unaligned_dst_aligned:
+.Lsrc_unaligned_dst_aligned\@:
SRL t0, len, LOG_NBYTES+2 # +2 for 4 units/iter
- beqz t0, .Lcleanup_src_unaligned
+ beqz t0, .Lcleanup_src_unaligned\@
and rem, len, (4*NBYTES-1) # rem = len % 4*NBYTES
1:
/*
@@ -603,53 +613,53 @@ FEXPORT(csum_partial_copy_nocheck)
* It's OK to load FIRST(N+1) before REST(N) because the two addresses
* are to the same unit (unless src is aligned, but it's not).
*/
- LDFIRST(t0, FIRST(0)(src), .Ll_exc)
- LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy)
+ LDFIRST(t0, FIRST(0)(src), .Ll_exc\@)
+ LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy\@)
SUB len, len, 4*NBYTES
- LDREST(t0, REST(0)(src), .Ll_exc_copy)
- LDREST(t1, REST(1)(src), .Ll_exc_copy)
- LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy)
- LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy)
- LDREST(t2, REST(2)(src), .Ll_exc_copy)
- LDREST(t3, REST(3)(src), .Ll_exc_copy)
+ LDREST(t0, REST(0)(src), .Ll_exc_copy\@)
+ LDREST(t1, REST(1)(src), .Ll_exc_copy\@)
+ LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy\@)
+ LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy\@)
+ LDREST(t2, REST(2)(src), .Ll_exc_copy\@)
+ LDREST(t3, REST(3)(src), .Ll_exc_copy\@)
ADD src, src, 4*NBYTES
#ifdef CONFIG_CPU_SB1
nop # improves slotting
#endif
- STORE(t0, UNIT(0)(dst), .Ls_exc)
+ STORE(t0, UNIT(0)(dst), .Ls_exc\@)
ADDC(sum, t0)
- STORE(t1, UNIT(1)(dst), .Ls_exc)
+ STORE(t1, UNIT(1)(dst), .Ls_exc\@)
ADDC(sum, t1)
- STORE(t2, UNIT(2)(dst), .Ls_exc)
+ STORE(t2, UNIT(2)(dst), .Ls_exc\@)
ADDC(sum, t2)
- STORE(t3, UNIT(3)(dst), .Ls_exc)
+ STORE(t3, UNIT(3)(dst), .Ls_exc\@)
ADDC(sum, t3)
.set reorder /* DADDI_WAR */
ADD dst, dst, 4*NBYTES
bne len, rem, 1b
.set noreorder
-.Lcleanup_src_unaligned:
- beqz len, .Ldone
+.Lcleanup_src_unaligned\@:
+ beqz len, .Ldone\@
and rem, len, NBYTES-1 # rem = len % NBYTES
- beq rem, len, .Lcopy_bytes
+ beq rem, len, .Lcopy_bytes\@
nop
1:
- LDFIRST(t0, FIRST(0)(src), .Ll_exc)
- LDREST(t0, REST(0)(src), .Ll_exc_copy)
+ LDFIRST(t0, FIRST(0)(src), .Ll_exc\@)
+ LDREST(t0, REST(0)(src), .Ll_exc_copy\@)
ADD src, src, NBYTES
SUB len, len, NBYTES
- STORE(t0, 0(dst), .Ls_exc)
+ STORE(t0, 0(dst), .Ls_exc\@)
ADDC(sum, t0)
.set reorder /* DADDI_WAR */
ADD dst, dst, NBYTES
bne len, rem, 1b
.set noreorder
-.Lcopy_bytes_checklen:
- beqz len, .Ldone
+.Lcopy_bytes_checklen\@:
+ beqz len, .Ldone\@
nop
-.Lcopy_bytes:
+.Lcopy_bytes\@:
/* 0 < len < NBYTES */
#ifdef CONFIG_CPU_LITTLE_ENDIAN
#define SHIFT_START 0
@@ -662,12 +672,12 @@ FEXPORT(csum_partial_copy_nocheck)
li t3, SHIFT_START # shift
/* use .Ll_exc_copy here to return correct sum on fault */
#define COPY_BYTE(N) \
- LOADBU(t0, N(src), .Ll_exc_copy); \
+ LOADBU(t0, N(src), .Ll_exc_copy\@); \
SUB len, len, 1; \
- STOREB(t0, N(dst), .Ls_exc); \
+ STOREB(t0, N(dst), .Ls_exc\@); \
SLLV t0, t0, t3; \
addu t3, SHIFT_INC; \
- beqz len, .Lcopy_bytes_done; \
+ beqz len, .Lcopy_bytes_done\@; \
or t2, t0
COPY_BYTE(0)
@@ -678,14 +688,14 @@ FEXPORT(csum_partial_copy_nocheck)
COPY_BYTE(4)
COPY_BYTE(5)
#endif
- LOADBU(t0, NBYTES-2(src), .Ll_exc_copy)
+ LOADBU(t0, NBYTES-2(src), .Ll_exc_copy\@)
SUB len, len, 1
- STOREB(t0, NBYTES-2(dst), .Ls_exc)
+ STOREB(t0, NBYTES-2(dst), .Ls_exc\@)
SLLV t0, t0, t3
or t2, t0
-.Lcopy_bytes_done:
+.Lcopy_bytes_done\@:
ADDC(sum, t2)
-.Ldone:
+.Ldone\@:
/* fold checksum */
#ifdef USE_DOUBLE
dsll32 v1, sum, 0
@@ -714,7 +724,7 @@ FEXPORT(csum_partial_copy_nocheck)
jr ra
.set noreorder
-.Ll_exc_copy:
+.Ll_exc_copy\@:
/*
* Copy bytes from src until faulting load address (or until a
* lb faults)
@@ -729,7 +739,7 @@ FEXPORT(csum_partial_copy_nocheck)
li t2, SHIFT_START
LOADK t0, THREAD_BUADDR(t0)
1:
- LOADBU(t1, 0(src), .Ll_exc)
+ LOADBU(t1, 0(src), .Ll_exc\@)
ADD src, src, 1
sb t1, 0(dst) # can't fault -- we're copy_from_user
SLLV t1, t1, t2
@@ -739,7 +749,7 @@ FEXPORT(csum_partial_copy_nocheck)
ADD dst, dst, 1
bne src, t0, 1b
.set noreorder
-.Ll_exc:
+.Ll_exc\@:
LOADK t0, TI_TASK($28)
nop
LOADK t0, THREAD_BUADDR(t0) # t0 is just past last good address
@@ -758,7 +768,7 @@ FEXPORT(csum_partial_copy_nocheck)
*/
.set reorder /* DADDI_WAR */
SUB src, len, 1
- beqz len, .Ldone
+ beqz len, .Ldone\@
.set noreorder
1: sb zero, 0(dst)
ADD dst, dst, 1
@@ -773,13 +783,19 @@ FEXPORT(csum_partial_copy_nocheck)
SUB src, src, v1
#endif
li v1, -EFAULT
- b .Ldone
+ b .Ldone\@
sw v1, (errptr)
-.Ls_exc:
+.Ls_exc\@:
li v0, -1 /* invalid checksum */
li v1, -EFAULT
jr ra
sw v1, (errptr)
.set pop
- END(__csum_partial_copy_kernel)
+ .endm
+
+LEAF(__csum_partial_copy_kernel)
+FEXPORT(__csum_partial_copy_to_user)
+FEXPORT(__csum_partial_copy_from_user)
+__BUILD_CSUM_PARTIAL_COPY_USER LEGACY_MODE USEROP USEROP 1
+END(__csum_partial_copy_kernel)
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 34/58] MIPS: lib: csum_partial: Add macro to build csum_partial symbols
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
In preparation for EVA support, we use a macro to build the
__csum_partial_copy_user main code so it can be shared across
multiple implementations. EVA uses the same code but it replaces
the load/store/prefetch instructions with the EVA specific ones
therefore using a macro avoids unnecessary code duplications.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/csum_partial.S | 200 +++++++++++++++++++++++--------------------
1 file changed, 108 insertions(+), 92 deletions(-)
diff --git a/arch/mips/lib/csum_partial.S b/arch/mips/lib/csum_partial.S
index bff5167..62c8768 100644
--- a/arch/mips/lib/csum_partial.S
+++ b/arch/mips/lib/csum_partial.S
@@ -331,6 +331,10 @@ LEAF(csum_partial)
/* Instruction type */
#define LD_INSN 1
#define ST_INSN 2
+#define LEGACY_MODE 1
+#define EVA_MODE 2
+#define USEROP 1
+#define KERNELOP 2
/*
* Wrapper to add an entry in the exception table
@@ -343,10 +347,12 @@ LEAF(csum_partial)
* handler : Exception handler
*/
#define EXC(insn, type, reg, addr, handler) \
-9: insn reg, addr; \
- .section __ex_table,"a"; \
- PTR 9b, handler; \
- .previous
+ .if \mode == LEGACY_MODE; \
+9: insn reg, addr; \
+ .section __ex_table,"a"; \
+ PTR 9b, handler; \
+ .previous; \
+ .endif
#undef LOAD
@@ -419,16 +425,20 @@ LEAF(csum_partial)
.set at=v1
#endif
-LEAF(__csum_partial_copy_kernel)
-FEXPORT(__csum_partial_copy_to_user)
-FEXPORT(__csum_partial_copy_from_user)
+ .macro __BUILD_CSUM_PARTIAL_COPY_USER mode, from, to, __nocheck
+
PTR_ADDU AT, src, len /* See (1) above. */
+ /* initialize __nocheck if this the first time we execute this
+ * macro
+ */
#ifdef CONFIG_64BIT
move errptr, a4
#else
lw errptr, 16(sp)
#endif
-FEXPORT(csum_partial_copy_nocheck)
+ .if \__nocheck == 1
+ FEXPORT(csum_partial_copy_nocheck)
+ .endif
move sum, zero
move odd, zero
/*
@@ -444,48 +454,48 @@ FEXPORT(csum_partial_copy_nocheck)
*/
sltu t2, len, NBYTES
and t1, dst, ADDRMASK
- bnez t2, .Lcopy_bytes_checklen
+ bnez t2, .Lcopy_bytes_checklen\@
and t0, src, ADDRMASK
andi odd, dst, 0x1 /* odd buffer? */
- bnez t1, .Ldst_unaligned
+ bnez t1, .Ldst_unaligned\@
nop
- bnez t0, .Lsrc_unaligned_dst_aligned
+ bnez t0, .Lsrc_unaligned_dst_aligned\@
/*
* use delay slot for fall-through
* src and dst are aligned; need to compute rem
*/
-.Lboth_aligned:
+.Lboth_aligned\@:
SRL t0, len, LOG_NBYTES+3 # +3 for 8 units/iter
- beqz t0, .Lcleanup_both_aligned # len < 8*NBYTES
+ beqz t0, .Lcleanup_both_aligned\@ # len < 8*NBYTES
nop
SUB len, 8*NBYTES # subtract here for bgez loop
.align 4
1:
- LOAD(t0, UNIT(0)(src), .Ll_exc)
- LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
- LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
- LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
- LOAD(t4, UNIT(4)(src), .Ll_exc_copy)
- LOAD(t5, UNIT(5)(src), .Ll_exc_copy)
- LOAD(t6, UNIT(6)(src), .Ll_exc_copy)
- LOAD(t7, UNIT(7)(src), .Ll_exc_copy)
+ LOAD(t0, UNIT(0)(src), .Ll_exc\@)
+ LOAD(t1, UNIT(1)(src), .Ll_exc_copy\@)
+ LOAD(t2, UNIT(2)(src), .Ll_exc_copy\@)
+ LOAD(t3, UNIT(3)(src), .Ll_exc_copy\@)
+ LOAD(t4, UNIT(4)(src), .Ll_exc_copy\@)
+ LOAD(t5, UNIT(5)(src), .Ll_exc_copy\@)
+ LOAD(t6, UNIT(6)(src), .Ll_exc_copy\@)
+ LOAD(t7, UNIT(7)(src), .Ll_exc_copy\@)
SUB len, len, 8*NBYTES
ADD src, src, 8*NBYTES
- STORE(t0, UNIT(0)(dst), .Ls_exc)
+ STORE(t0, UNIT(0)(dst), .Ls_exc\@)
ADDC(sum, t0)
- STORE(t1, UNIT(1)(dst), .Ls_exc)
+ STORE(t1, UNIT(1)(dst), .Ls_exc\@)
ADDC(sum, t1)
- STORE(t2, UNIT(2)(dst), .Ls_exc)
+ STORE(t2, UNIT(2)(dst), .Ls_exc\@)
ADDC(sum, t2)
- STORE(t3, UNIT(3)(dst), .Ls_exc)
+ STORE(t3, UNIT(3)(dst), .Ls_exc\@)
ADDC(sum, t3)
- STORE(t4, UNIT(4)(dst), .Ls_exc)
+ STORE(t4, UNIT(4)(dst), .Ls_exc\@)
ADDC(sum, t4)
- STORE(t5, UNIT(5)(dst), .Ls_exc)
+ STORE(t5, UNIT(5)(dst), .Ls_exc\@)
ADDC(sum, t5)
- STORE(t6, UNIT(6)(dst), .Ls_exc)
+ STORE(t6, UNIT(6)(dst), .Ls_exc\@)
ADDC(sum, t6)
- STORE(t7, UNIT(7)(dst), .Ls_exc)
+ STORE(t7, UNIT(7)(dst), .Ls_exc\@)
ADDC(sum, t7)
.set reorder /* DADDI_WAR */
ADD dst, dst, 8*NBYTES
@@ -496,44 +506,44 @@ FEXPORT(csum_partial_copy_nocheck)
/*
* len == the number of bytes left to copy < 8*NBYTES
*/
-.Lcleanup_both_aligned:
+.Lcleanup_both_aligned\@:
#define rem t7
- beqz len, .Ldone
+ beqz len, .Ldone\@
sltu t0, len, 4*NBYTES
- bnez t0, .Lless_than_4units
+ bnez t0, .Lless_than_4units\@
and rem, len, (NBYTES-1) # rem = len % NBYTES
/*
* len >= 4*NBYTES
*/
- LOAD(t0, UNIT(0)(src), .Ll_exc)
- LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
- LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
- LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
+ LOAD(t0, UNIT(0)(src), .Ll_exc\@)
+ LOAD(t1, UNIT(1)(src), .Ll_exc_copy\@)
+ LOAD(t2, UNIT(2)(src), .Ll_exc_copy\@)
+ LOAD(t3, UNIT(3)(src), .Ll_exc_copy\@)
SUB len, len, 4*NBYTES
ADD src, src, 4*NBYTES
- STORE(t0, UNIT(0)(dst), .Ls_exc)
+ STORE(t0, UNIT(0)(dst), .Ls_exc\@)
ADDC(sum, t0)
- STORE(t1, UNIT(1)(dst), .Ls_exc)
+ STORE(t1, UNIT(1)(dst), .Ls_exc\@)
ADDC(sum, t1)
- STORE(t2, UNIT(2)(dst), .Ls_exc)
+ STORE(t2, UNIT(2)(dst), .Ls_exc\@)
ADDC(sum, t2)
- STORE(t3, UNIT(3)(dst), .Ls_exc)
+ STORE(t3, UNIT(3)(dst), .Ls_exc\@)
ADDC(sum, t3)
.set reorder /* DADDI_WAR */
ADD dst, dst, 4*NBYTES
- beqz len, .Ldone
+ beqz len, .Ldone\@
.set noreorder
-.Lless_than_4units:
+.Lless_than_4units\@:
/*
* rem = len % NBYTES
*/
- beq rem, len, .Lcopy_bytes
+ beq rem, len, .Lcopy_bytes\@
nop
1:
- LOAD(t0, 0(src), .Ll_exc)
+ LOAD(t0, 0(src), .Ll_exc\@)
ADD src, src, NBYTES
SUB len, len, NBYTES
- STORE(t0, 0(dst), .Ls_exc)
+ STORE(t0, 0(dst), .Ls_exc\@)
ADDC(sum, t0)
.set reorder /* DADDI_WAR */
ADD dst, dst, NBYTES
@@ -552,20 +562,20 @@ FEXPORT(csum_partial_copy_nocheck)
* more instruction-level parallelism.
*/
#define bits t2
- beqz len, .Ldone
+ beqz len, .Ldone\@
ADD t1, dst, len # t1 is just past last byte of dst
li bits, 8*NBYTES
SLL rem, len, 3 # rem = number of bits to keep
- LOAD(t0, 0(src), .Ll_exc)
+ LOAD(t0, 0(src), .Ll_exc\@)
SUB bits, bits, rem # bits = number of bits to discard
SHIFT_DISCARD t0, t0, bits
- STREST(t0, -1(t1), .Ls_exc)
+ STREST(t0, -1(t1), .Ls_exc\@)
SHIFT_DISCARD_REVERT t0, t0, bits
.set reorder
ADDC(sum, t0)
- b .Ldone
+ b .Ldone\@
.set noreorder
-.Ldst_unaligned:
+.Ldst_unaligned\@:
/*
* dst is unaligned
* t0 = src & ADDRMASK
@@ -576,25 +586,25 @@ FEXPORT(csum_partial_copy_nocheck)
* Set match = (src and dst have same alignment)
*/
#define match rem
- LDFIRST(t3, FIRST(0)(src), .Ll_exc)
+ LDFIRST(t3, FIRST(0)(src), .Ll_exc\@)
ADD t2, zero, NBYTES
- LDREST(t3, REST(0)(src), .Ll_exc_copy)
+ LDREST(t3, REST(0)(src), .Ll_exc_copy\@)
SUB t2, t2, t1 # t2 = number of bytes copied
xor match, t0, t1
- STFIRST(t3, FIRST(0)(dst), .Ls_exc)
+ STFIRST(t3, FIRST(0)(dst), .Ls_exc\@)
SLL t4, t1, 3 # t4 = number of bits to discard
SHIFT_DISCARD t3, t3, t4
/* no SHIFT_DISCARD_REVERT to handle odd buffer properly */
ADDC(sum, t3)
- beq len, t2, .Ldone
+ beq len, t2, .Ldone\@
SUB len, len, t2
ADD dst, dst, t2
- beqz match, .Lboth_aligned
+ beqz match, .Lboth_aligned\@
ADD src, src, t2
-.Lsrc_unaligned_dst_aligned:
+.Lsrc_unaligned_dst_aligned\@:
SRL t0, len, LOG_NBYTES+2 # +2 for 4 units/iter
- beqz t0, .Lcleanup_src_unaligned
+ beqz t0, .Lcleanup_src_unaligned\@
and rem, len, (4*NBYTES-1) # rem = len % 4*NBYTES
1:
/*
@@ -603,53 +613,53 @@ FEXPORT(csum_partial_copy_nocheck)
* It's OK to load FIRST(N+1) before REST(N) because the two addresses
* are to the same unit (unless src is aligned, but it's not).
*/
- LDFIRST(t0, FIRST(0)(src), .Ll_exc)
- LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy)
+ LDFIRST(t0, FIRST(0)(src), .Ll_exc\@)
+ LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy\@)
SUB len, len, 4*NBYTES
- LDREST(t0, REST(0)(src), .Ll_exc_copy)
- LDREST(t1, REST(1)(src), .Ll_exc_copy)
- LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy)
- LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy)
- LDREST(t2, REST(2)(src), .Ll_exc_copy)
- LDREST(t3, REST(3)(src), .Ll_exc_copy)
+ LDREST(t0, REST(0)(src), .Ll_exc_copy\@)
+ LDREST(t1, REST(1)(src), .Ll_exc_copy\@)
+ LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy\@)
+ LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy\@)
+ LDREST(t2, REST(2)(src), .Ll_exc_copy\@)
+ LDREST(t3, REST(3)(src), .Ll_exc_copy\@)
ADD src, src, 4*NBYTES
#ifdef CONFIG_CPU_SB1
nop # improves slotting
#endif
- STORE(t0, UNIT(0)(dst), .Ls_exc)
+ STORE(t0, UNIT(0)(dst), .Ls_exc\@)
ADDC(sum, t0)
- STORE(t1, UNIT(1)(dst), .Ls_exc)
+ STORE(t1, UNIT(1)(dst), .Ls_exc\@)
ADDC(sum, t1)
- STORE(t2, UNIT(2)(dst), .Ls_exc)
+ STORE(t2, UNIT(2)(dst), .Ls_exc\@)
ADDC(sum, t2)
- STORE(t3, UNIT(3)(dst), .Ls_exc)
+ STORE(t3, UNIT(3)(dst), .Ls_exc\@)
ADDC(sum, t3)
.set reorder /* DADDI_WAR */
ADD dst, dst, 4*NBYTES
bne len, rem, 1b
.set noreorder
-.Lcleanup_src_unaligned:
- beqz len, .Ldone
+.Lcleanup_src_unaligned\@:
+ beqz len, .Ldone\@
and rem, len, NBYTES-1 # rem = len % NBYTES
- beq rem, len, .Lcopy_bytes
+ beq rem, len, .Lcopy_bytes\@
nop
1:
- LDFIRST(t0, FIRST(0)(src), .Ll_exc)
- LDREST(t0, REST(0)(src), .Ll_exc_copy)
+ LDFIRST(t0, FIRST(0)(src), .Ll_exc\@)
+ LDREST(t0, REST(0)(src), .Ll_exc_copy\@)
ADD src, src, NBYTES
SUB len, len, NBYTES
- STORE(t0, 0(dst), .Ls_exc)
+ STORE(t0, 0(dst), .Ls_exc\@)
ADDC(sum, t0)
.set reorder /* DADDI_WAR */
ADD dst, dst, NBYTES
bne len, rem, 1b
.set noreorder
-.Lcopy_bytes_checklen:
- beqz len, .Ldone
+.Lcopy_bytes_checklen\@:
+ beqz len, .Ldone\@
nop
-.Lcopy_bytes:
+.Lcopy_bytes\@:
/* 0 < len < NBYTES */
#ifdef CONFIG_CPU_LITTLE_ENDIAN
#define SHIFT_START 0
@@ -662,12 +672,12 @@ FEXPORT(csum_partial_copy_nocheck)
li t3, SHIFT_START # shift
/* use .Ll_exc_copy here to return correct sum on fault */
#define COPY_BYTE(N) \
- LOADBU(t0, N(src), .Ll_exc_copy); \
+ LOADBU(t0, N(src), .Ll_exc_copy\@); \
SUB len, len, 1; \
- STOREB(t0, N(dst), .Ls_exc); \
+ STOREB(t0, N(dst), .Ls_exc\@); \
SLLV t0, t0, t3; \
addu t3, SHIFT_INC; \
- beqz len, .Lcopy_bytes_done; \
+ beqz len, .Lcopy_bytes_done\@; \
or t2, t0
COPY_BYTE(0)
@@ -678,14 +688,14 @@ FEXPORT(csum_partial_copy_nocheck)
COPY_BYTE(4)
COPY_BYTE(5)
#endif
- LOADBU(t0, NBYTES-2(src), .Ll_exc_copy)
+ LOADBU(t0, NBYTES-2(src), .Ll_exc_copy\@)
SUB len, len, 1
- STOREB(t0, NBYTES-2(dst), .Ls_exc)
+ STOREB(t0, NBYTES-2(dst), .Ls_exc\@)
SLLV t0, t0, t3
or t2, t0
-.Lcopy_bytes_done:
+.Lcopy_bytes_done\@:
ADDC(sum, t2)
-.Ldone:
+.Ldone\@:
/* fold checksum */
#ifdef USE_DOUBLE
dsll32 v1, sum, 0
@@ -714,7 +724,7 @@ FEXPORT(csum_partial_copy_nocheck)
jr ra
.set noreorder
-.Ll_exc_copy:
+.Ll_exc_copy\@:
/*
* Copy bytes from src until faulting load address (or until a
* lb faults)
@@ -729,7 +739,7 @@ FEXPORT(csum_partial_copy_nocheck)
li t2, SHIFT_START
LOADK t0, THREAD_BUADDR(t0)
1:
- LOADBU(t1, 0(src), .Ll_exc)
+ LOADBU(t1, 0(src), .Ll_exc\@)
ADD src, src, 1
sb t1, 0(dst) # can't fault -- we're copy_from_user
SLLV t1, t1, t2
@@ -739,7 +749,7 @@ FEXPORT(csum_partial_copy_nocheck)
ADD dst, dst, 1
bne src, t0, 1b
.set noreorder
-.Ll_exc:
+.Ll_exc\@:
LOADK t0, TI_TASK($28)
nop
LOADK t0, THREAD_BUADDR(t0) # t0 is just past last good address
@@ -758,7 +768,7 @@ FEXPORT(csum_partial_copy_nocheck)
*/
.set reorder /* DADDI_WAR */
SUB src, len, 1
- beqz len, .Ldone
+ beqz len, .Ldone\@
.set noreorder
1: sb zero, 0(dst)
ADD dst, dst, 1
@@ -773,13 +783,19 @@ FEXPORT(csum_partial_copy_nocheck)
SUB src, src, v1
#endif
li v1, -EFAULT
- b .Ldone
+ b .Ldone\@
sw v1, (errptr)
-.Ls_exc:
+.Ls_exc\@:
li v0, -1 /* invalid checksum */
li v1, -EFAULT
jr ra
sw v1, (errptr)
.set pop
- END(__csum_partial_copy_kernel)
+ .endm
+
+LEAF(__csum_partial_copy_kernel)
+FEXPORT(__csum_partial_copy_to_user)
+FEXPORT(__csum_partial_copy_from_user)
+__BUILD_CSUM_PARTIAL_COPY_USER LEGACY_MODE USEROP USEROP 1
+END(__csum_partial_copy_kernel)
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 35/58] MIPS: lib: csum_partial: Add EVA support
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Use EVA specific functions to read and write data to
user address space.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/csum_partial.S | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)
diff --git a/arch/mips/lib/csum_partial.S b/arch/mips/lib/csum_partial.S
index 62c8768..2e4825e 100644
--- a/arch/mips/lib/csum_partial.S
+++ b/arch/mips/lib/csum_partial.S
@@ -352,6 +352,19 @@ LEAF(csum_partial)
.section __ex_table,"a"; \
PTR 9b, handler; \
.previous; \
+ /* This is enabled in EVA mode */ \
+ .else; \
+ /* If loading from user or storing to user */ \
+ .if ((\from == USEROP) && (type == LD_INSN)) || \
+ ((\to == USEROP) && (type == ST_INSN)); \
+9: __BUILD_EVA_INSN(insn##e, reg, addr); \
+ .section __ex_table,"a"; \
+ PTR 9b, handler; \
+ .previous; \
+ .else; \
+ /* EVA without exception */ \
+ insn reg, addr; \
+ .endif; \
.endif
#undef LOAD
@@ -795,7 +808,19 @@ LEAF(csum_partial)
.endm
LEAF(__csum_partial_copy_kernel)
+#ifndef CONFIG_EVA
FEXPORT(__csum_partial_copy_to_user)
FEXPORT(__csum_partial_copy_from_user)
+#endif
__BUILD_CSUM_PARTIAL_COPY_USER LEGACY_MODE USEROP USEROP 1
END(__csum_partial_copy_kernel)
+
+#ifdef CONFIG_EVA
+LEAF(__csum_partial_copy_to_user)
+__BUILD_CSUM_PARTIAL_COPY_USER EVA_MODE KERNELOP USEROP 0
+END(__csum_partial_copy_to_user)
+
+LEAF(__csum_partial_copy_from_user)
+__BUILD_CSUM_PARTIAL_COPY_USER EVA_MODE USEROP KERNELOP 0
+END(__csum_partial_copy_from_user)
+#endif
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 35/58] MIPS: lib: csum_partial: Add EVA support
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Use EVA specific functions to read and write data to
user address space.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/lib/csum_partial.S | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)
diff --git a/arch/mips/lib/csum_partial.S b/arch/mips/lib/csum_partial.S
index 62c8768..2e4825e 100644
--- a/arch/mips/lib/csum_partial.S
+++ b/arch/mips/lib/csum_partial.S
@@ -352,6 +352,19 @@ LEAF(csum_partial)
.section __ex_table,"a"; \
PTR 9b, handler; \
.previous; \
+ /* This is enabled in EVA mode */ \
+ .else; \
+ /* If loading from user or storing to user */ \
+ .if ((\from == USEROP) && (type == LD_INSN)) || \
+ ((\to == USEROP) && (type == ST_INSN)); \
+9: __BUILD_EVA_INSN(insn##e, reg, addr); \
+ .section __ex_table,"a"; \
+ PTR 9b, handler; \
+ .previous; \
+ .else; \
+ /* EVA without exception */ \
+ insn reg, addr; \
+ .endif; \
.endif
#undef LOAD
@@ -795,7 +808,19 @@ LEAF(csum_partial)
.endm
LEAF(__csum_partial_copy_kernel)
+#ifndef CONFIG_EVA
FEXPORT(__csum_partial_copy_to_user)
FEXPORT(__csum_partial_copy_from_user)
+#endif
__BUILD_CSUM_PARTIAL_COPY_USER LEGACY_MODE USEROP USEROP 1
END(__csum_partial_copy_kernel)
+
+#ifdef CONFIG_EVA
+LEAF(__csum_partial_copy_to_user)
+__BUILD_CSUM_PARTIAL_COPY_USER EVA_MODE KERNELOP USEROP 0
+END(__csum_partial_copy_to_user)
+
+LEAF(__csum_partial_copy_from_user)
+__BUILD_CSUM_PARTIAL_COPY_USER EVA_MODE USEROP KERNELOP 0
+END(__csum_partial_copy_from_user)
+#endif
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 36/58] MIPS: asm: checksum: Split kernel and user copy operations
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
In EVA mode, different instructions need to be used to read/write
from kernel and userland. In non-EVA mode, there is no functional
difference. The current address limit is checked to decide the
type of operation that will be performed.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/checksum.h | 22 ++++++++++++++++------
1 file changed, 16 insertions(+), 6 deletions(-)
diff --git a/arch/mips/include/asm/checksum.h b/arch/mips/include/asm/checksum.h
index 3c9aea5..f5602c6 100644
--- a/arch/mips/include/asm/checksum.h
+++ b/arch/mips/include/asm/checksum.h
@@ -37,7 +37,6 @@ __wsum __csum_partial_copy_from_user(const void *src, void *dst,
int len, __wsum sum, int *err_ptr);
__wsum __csum_partial_copy_to_user(const void *src, void *dst,
int len, __wsum sum, int *err_ptr);
-
/*
* this is a new version of the above that records errors it finds in *errp,
* but continues and zeros the rest of the buffer.
@@ -47,8 +46,12 @@ __wsum csum_partial_copy_from_user(const void __user *src, void *dst, int len,
__wsum sum, int *err_ptr)
{
might_fault();
- return __csum_partial_copy_from_user((__force void *)src, dst,
- len, sum, err_ptr);
+ if (segment_eq(get_fs(), get_ds()))
+ return __csum_partial_copy_kernel((__force void *)src, dst,
+ len, sum, err_ptr);
+ else
+ return __csum_partial_copy_from_user((__force void *)src, dst,
+ len, sum, err_ptr);
}
/*
@@ -60,9 +63,16 @@ __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len,
__wsum sum, int *err_ptr)
{
might_fault();
- if (access_ok(VERIFY_WRITE, dst, len))
- return __csum_partial_copy_to_user(src, (__force void *)dst,
- len, sum, err_ptr);
+ if (access_ok(VERIFY_WRITE, dst, len)) {
+ if (segment_eq(get_fs(), get_ds()))
+ return __csum_partial_copy_kernel(src,
+ (__force void *)dst,
+ len, sum, err_ptr);
+ else
+ return __csum_partial_copy_to_user(src,
+ (__force void *)dst,
+ len, sum, err_ptr);
+ }
if (len)
*err_ptr = -EFAULT;
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 36/58] MIPS: asm: checksum: Split kernel and user copy operations
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
In EVA mode, different instructions need to be used to read/write
from kernel and userland. In non-EVA mode, there is no functional
difference. The current address limit is checked to decide the
type of operation that will be performed.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/checksum.h | 22 ++++++++++++++++------
1 file changed, 16 insertions(+), 6 deletions(-)
diff --git a/arch/mips/include/asm/checksum.h b/arch/mips/include/asm/checksum.h
index 3c9aea5..f5602c6 100644
--- a/arch/mips/include/asm/checksum.h
+++ b/arch/mips/include/asm/checksum.h
@@ -37,7 +37,6 @@ __wsum __csum_partial_copy_from_user(const void *src, void *dst,
int len, __wsum sum, int *err_ptr);
__wsum __csum_partial_copy_to_user(const void *src, void *dst,
int len, __wsum sum, int *err_ptr);
-
/*
* this is a new version of the above that records errors it finds in *errp,
* but continues and zeros the rest of the buffer.
@@ -47,8 +46,12 @@ __wsum csum_partial_copy_from_user(const void __user *src, void *dst, int len,
__wsum sum, int *err_ptr)
{
might_fault();
- return __csum_partial_copy_from_user((__force void *)src, dst,
- len, sum, err_ptr);
+ if (segment_eq(get_fs(), get_ds()))
+ return __csum_partial_copy_kernel((__force void *)src, dst,
+ len, sum, err_ptr);
+ else
+ return __csum_partial_copy_from_user((__force void *)src, dst,
+ len, sum, err_ptr);
}
/*
@@ -60,9 +63,16 @@ __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len,
__wsum sum, int *err_ptr)
{
might_fault();
- if (access_ok(VERIFY_WRITE, dst, len))
- return __csum_partial_copy_to_user(src, (__force void *)dst,
- len, sum, err_ptr);
+ if (access_ok(VERIFY_WRITE, dst, len)) {
+ if (segment_eq(get_fs(), get_ds()))
+ return __csum_partial_copy_kernel(src,
+ (__force void *)dst,
+ len, sum, err_ptr);
+ else
+ return __csum_partial_copy_to_user(src,
+ (__force void *)dst,
+ len, sum, err_ptr);
+ }
if (len)
*err_ptr = -EFAULT;
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 37/58] MIPS: asm: checksum: Add MIPS specific csum_and_copy_from_user function
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
A MIPS specific csum_and_copy_from_user function is necessary because
the generic one from include/net/checksum.h will not work for EVA.
This is because the generic one will link to symbols from lib/checksum.c
which are not EVA aware.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/checksum.h | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/arch/mips/include/asm/checksum.h b/arch/mips/include/asm/checksum.h
index f5602c6..3418c51 100644
--- a/arch/mips/include/asm/checksum.h
+++ b/arch/mips/include/asm/checksum.h
@@ -54,6 +54,20 @@ __wsum csum_partial_copy_from_user(const void __user *src, void *dst, int len,
len, sum, err_ptr);
}
+#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER
+static inline
+__wsum csum_and_copy_from_user(const void __user *src, void *dst,
+ int len, __wsum sum, int *err_ptr)
+{
+ if (access_ok(VERIFY_READ, src, len))
+ return csum_partial_copy_from_user(src, dst, len, sum,
+ err_ptr);
+ if (len)
+ *err_ptr = -EFAULT;
+
+ return sum;
+}
+
/*
* Copy and checksum to user
*/
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 37/58] MIPS: asm: checksum: Add MIPS specific csum_and_copy_from_user function
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
A MIPS specific csum_and_copy_from_user function is necessary because
the generic one from include/net/checksum.h will not work for EVA.
This is because the generic one will link to symbols from lib/checksum.c
which are not EVA aware.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/checksum.h | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/arch/mips/include/asm/checksum.h b/arch/mips/include/asm/checksum.h
index f5602c6..3418c51 100644
--- a/arch/mips/include/asm/checksum.h
+++ b/arch/mips/include/asm/checksum.h
@@ -54,6 +54,20 @@ __wsum csum_partial_copy_from_user(const void __user *src, void *dst, int len,
len, sum, err_ptr);
}
+#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER
+static inline
+__wsum csum_and_copy_from_user(const void __user *src, void *dst,
+ int len, __wsum sum, int *err_ptr)
+{
+ if (access_ok(VERIFY_READ, src, len))
+ return csum_partial_copy_from_user(src, dst, len, sum,
+ err_ptr);
+ if (len)
+ *err_ptr = -EFAULT;
+
+ return sum;
+}
+
/*
* Copy and checksum to user
*/
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 38/58] MIPS: kernel: signal: Prevent save/restore FPU context in user memory
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
EVA does not have FPU specific instructions for reading or writing
FPU registers from userspace memory.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/signal.c | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/arch/mips/kernel/signal.c b/arch/mips/kernel/signal.c
index 5199563..6f4937b 100644
--- a/arch/mips/kernel/signal.c
+++ b/arch/mips/kernel/signal.c
@@ -6,6 +6,7 @@
* Copyright (C) 1991, 1992 Linus Torvalds
* Copyright (C) 1994 - 2000 Ralf Baechle
* Copyright (C) 1999, 2000 Silicon Graphics, Inc.
+ * Copyright (C) 2014, Imagination Technologies Ltd.
*/
#include <linux/cache.h>
#include <linux/context_tracking.h>
@@ -69,6 +70,7 @@ struct rt_sigframe {
static int protected_save_fp_context(struct sigcontext __user *sc)
{
int err;
+#ifndef CONFIG_EVA
while (1) {
lock_fpu_owner();
err = own_fpu_inatomic(1);
@@ -84,12 +86,21 @@ static int protected_save_fp_context(struct sigcontext __user *sc)
if (err)
break; /* really bad sigcontext */
}
+#else
+ /*
+ * EVA does not have FPU EVA instructions so saving fpu context directly
+ * does not work.
+ */
+ lose_fpu(1);
+ err = save_fp_context(sc); /* this might fail */
+#endif
return err;
}
static int protected_restore_fp_context(struct sigcontext __user *sc)
{
int err, tmp __maybe_unused;
+#ifndef CONFIG_EVA
while (1) {
lock_fpu_owner();
err = own_fpu_inatomic(0);
@@ -105,6 +116,14 @@ static int protected_restore_fp_context(struct sigcontext __user *sc)
if (err)
break; /* really bad sigcontext */
}
+#else
+ /*
+ * EVA does not have FPU EVA instructions so restoring fpu context
+ * directly does not work.
+ */
+ lose_fpu(0);
+ err = restore_fp_context(sc); /* this might fail */
+#endif
return err;
}
@@ -591,6 +610,7 @@ asmlinkage void do_notify_resume(struct pt_regs *regs, void *unused,
}
#ifdef CONFIG_SMP
+#ifndef CONFIG_EVA
static int smp_save_fp_context(struct sigcontext __user *sc)
{
return raw_cpu_has_fpu
@@ -604,10 +624,12 @@ static int smp_restore_fp_context(struct sigcontext __user *sc)
? _restore_fp_context(sc)
: fpu_emulator_restore_context(sc);
}
+#endif /* CONFIG_EVA */
#endif
static int signal_setup(void)
{
+#ifndef CONFIG_EVA
#ifdef CONFIG_SMP
/* For now just do the cpu_has_fpu check when the functions are invoked */
save_fp_context = smp_save_fp_context;
@@ -620,6 +642,10 @@ static int signal_setup(void)
save_fp_context = fpu_emulator_save_context;
restore_fp_context = fpu_emulator_restore_context;
}
+#endif /* CONFIG_SMP */
+#else
+ save_fp_context = fpu_emulator_save_context;
+ restore_fp_context = fpu_emulator_restore_context;
#endif
return 0;
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 38/58] MIPS: kernel: signal: Prevent save/restore FPU context in user memory
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
EVA does not have FPU specific instructions for reading or writing
FPU registers from userspace memory.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/signal.c | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/arch/mips/kernel/signal.c b/arch/mips/kernel/signal.c
index 5199563..6f4937b 100644
--- a/arch/mips/kernel/signal.c
+++ b/arch/mips/kernel/signal.c
@@ -6,6 +6,7 @@
* Copyright (C) 1991, 1992 Linus Torvalds
* Copyright (C) 1994 - 2000 Ralf Baechle
* Copyright (C) 1999, 2000 Silicon Graphics, Inc.
+ * Copyright (C) 2014, Imagination Technologies Ltd.
*/
#include <linux/cache.h>
#include <linux/context_tracking.h>
@@ -69,6 +70,7 @@ struct rt_sigframe {
static int protected_save_fp_context(struct sigcontext __user *sc)
{
int err;
+#ifndef CONFIG_EVA
while (1) {
lock_fpu_owner();
err = own_fpu_inatomic(1);
@@ -84,12 +86,21 @@ static int protected_save_fp_context(struct sigcontext __user *sc)
if (err)
break; /* really bad sigcontext */
}
+#else
+ /*
+ * EVA does not have FPU EVA instructions so saving fpu context directly
+ * does not work.
+ */
+ lose_fpu(1);
+ err = save_fp_context(sc); /* this might fail */
+#endif
return err;
}
static int protected_restore_fp_context(struct sigcontext __user *sc)
{
int err, tmp __maybe_unused;
+#ifndef CONFIG_EVA
while (1) {
lock_fpu_owner();
err = own_fpu_inatomic(0);
@@ -105,6 +116,14 @@ static int protected_restore_fp_context(struct sigcontext __user *sc)
if (err)
break; /* really bad sigcontext */
}
+#else
+ /*
+ * EVA does not have FPU EVA instructions so restoring fpu context
+ * directly does not work.
+ */
+ lose_fpu(0);
+ err = restore_fp_context(sc); /* this might fail */
+#endif
return err;
}
@@ -591,6 +610,7 @@ asmlinkage void do_notify_resume(struct pt_regs *regs, void *unused,
}
#ifdef CONFIG_SMP
+#ifndef CONFIG_EVA
static int smp_save_fp_context(struct sigcontext __user *sc)
{
return raw_cpu_has_fpu
@@ -604,10 +624,12 @@ static int smp_restore_fp_context(struct sigcontext __user *sc)
? _restore_fp_context(sc)
: fpu_emulator_restore_context(sc);
}
+#endif /* CONFIG_EVA */
#endif
static int signal_setup(void)
{
+#ifndef CONFIG_EVA
#ifdef CONFIG_SMP
/* For now just do the cpu_has_fpu check when the functions are invoked */
save_fp_context = smp_save_fp_context;
@@ -620,6 +642,10 @@ static int signal_setup(void)
save_fp_context = fpu_emulator_save_context;
restore_fp_context = fpu_emulator_restore_context;
}
+#endif /* CONFIG_SMP */
+#else
+ save_fp_context = fpu_emulator_save_context;
+ restore_fp_context = fpu_emulator_restore_context;
#endif
return 0;
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 39/58] MIPS: asm: r4kcache: Build flushing code for instruction cache
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Build code to invalidate an address range in the instruction cache
using the Hit Invalidate cache operation.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/r4kcache.h | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/mips/include/asm/r4kcache.h b/arch/mips/include/asm/r4kcache.h
index 34d1a19..a89e86d 100644
--- a/arch/mips/include/asm/r4kcache.h
+++ b/arch/mips/include/asm/r4kcache.h
@@ -455,6 +455,7 @@ __BUILD_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I, protected_, )
__BUILD_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I_Loongson23, \
protected_, loongson23_)
__BUILD_BLAST_CACHE_RANGE(d, dcache, Hit_Writeback_Inv_D, , )
+__BUILD_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I, , )
__BUILD_BLAST_CACHE_RANGE(s, scache, Hit_Writeback_Inv_SD, , )
/* blast_inv_dcache_range */
__BUILD_BLAST_CACHE_RANGE(inv_d, dcache, Hit_Invalidate_D, , )
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 39/58] MIPS: asm: r4kcache: Build flushing code for instruction cache
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Build code to invalidate an address range in the instruction cache
using the Hit Invalidate cache operation.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/r4kcache.h | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/mips/include/asm/r4kcache.h b/arch/mips/include/asm/r4kcache.h
index 34d1a19..a89e86d 100644
--- a/arch/mips/include/asm/r4kcache.h
+++ b/arch/mips/include/asm/r4kcache.h
@@ -455,6 +455,7 @@ __BUILD_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I, protected_, )
__BUILD_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I_Loongson23, \
protected_, loongson23_)
__BUILD_BLAST_CACHE_RANGE(d, dcache, Hit_Writeback_Inv_D, , )
+__BUILD_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I, , )
__BUILD_BLAST_CACHE_RANGE(s, scache, Hit_Writeback_Inv_SD, , )
/* blast_inv_dcache_range */
__BUILD_BLAST_CACHE_RANGE(inv_d, dcache, Hit_Invalidate_D, , )
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 40/58] MIPS: asm: r4kcache: Add protected cache operation for EVA
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/r4kcache.h | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/arch/mips/include/asm/r4kcache.h b/arch/mips/include/asm/r4kcache.h
index a89e86d..ae026bf 100644
--- a/arch/mips/include/asm/r4kcache.h
+++ b/arch/mips/include/asm/r4kcache.h
@@ -212,6 +212,20 @@ static inline void flush_scache_line(unsigned long addr)
: \
: "i" (op), "r" (addr))
+#define protected_cachee_op(op,addr) \
+ __asm__ __volatile__( \
+ " .set push \n" \
+ " .set noreorder \n" \
+ " .set mips0 \n" \
+ " .set eva \n" \
+ "1: cachee %0, (%1) \n" \
+ "2: .set pop \n" \
+ " .section __ex_table,\"a\" \n" \
+ " "STR(PTR)" 1b, 2b \n" \
+ " .previous" \
+ : \
+ : "i" (op), "r" (addr))
+
/*
* The next two are for badland addresses like signal trampolines.
*/
@@ -223,7 +237,11 @@ static inline void protected_flush_icache_line(unsigned long addr)
break;
default:
+#ifdef CONFIG_EVA
+ protected_cachee_op(Hit_Invalidate_I, addr);
+#else
protected_cache_op(Hit_Invalidate_I, addr);
+#endif
break;
}
}
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 40/58] MIPS: asm: r4kcache: Add protected cache operation for EVA
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/r4kcache.h | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/arch/mips/include/asm/r4kcache.h b/arch/mips/include/asm/r4kcache.h
index a89e86d..ae026bf 100644
--- a/arch/mips/include/asm/r4kcache.h
+++ b/arch/mips/include/asm/r4kcache.h
@@ -212,6 +212,20 @@ static inline void flush_scache_line(unsigned long addr)
: \
: "i" (op), "r" (addr))
+#define protected_cachee_op(op,addr) \
+ __asm__ __volatile__( \
+ " .set push \n" \
+ " .set noreorder \n" \
+ " .set mips0 \n" \
+ " .set eva \n" \
+ "1: cachee %0, (%1) \n" \
+ "2: .set pop \n" \
+ " .section __ex_table,\"a\" \n" \
+ " "STR(PTR)" 1b, 2b \n" \
+ " .previous" \
+ : \
+ : "i" (op), "r" (addr))
+
/*
* The next two are for badland addresses like signal trampolines.
*/
@@ -223,7 +237,11 @@ static inline void protected_flush_icache_line(unsigned long addr)
break;
default:
+#ifdef CONFIG_EVA
+ protected_cachee_op(Hit_Invalidate_I, addr);
+#else
protected_cache_op(Hit_Invalidate_I, addr);
+#endif
break;
}
}
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 41/58] MIPS: asm: r4kcache: Add EVA cache flushing functions
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Add EVA cache flushing functions similar to non-EVA configurations.
Because the cache may or may not contain user virtual addresses, we
need to use the 'cache' or 'cachee' instruction based on whether we
flush the cache on behalf of kernel or user respectively.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/r4kcache.h | 154 ++++++++++++++++++++++++++++++++++++++-
1 file changed, 152 insertions(+), 2 deletions(-)
diff --git a/arch/mips/include/asm/r4kcache.h b/arch/mips/include/asm/r4kcache.h
index ae026bf..b973e02 100644
--- a/arch/mips/include/asm/r4kcache.h
+++ b/arch/mips/include/asm/r4kcache.h
@@ -17,6 +17,7 @@
#include <asm/cpu-features.h>
#include <asm/cpu-type.h>
#include <asm/mipsmtregs.h>
+#include <asm/uaccess.h> /* for segment_eq() */
/*
* This macro return a properly sign-extended address suitable as base address
@@ -374,6 +375,91 @@ static inline void invalidate_tcache_page(unsigned long addr)
: "r" (base), \
"i" (op));
+/*
+ * Perform the cache operation specified by op using a user mode virtual
+ * address while in kernel mode.
+ */
+#define cache16_unroll32_user(base,op) \
+ __asm__ __volatile__( \
+ " .set push \n" \
+ " .set noreorder \n" \
+ " .set mips0 \n" \
+ " .set eva \n" \
+ " cachee %1, 0x000(%0); cachee %1, 0x010(%0) \n" \
+ " cachee %1, 0x020(%0); cachee %1, 0x030(%0) \n" \
+ " cachee %1, 0x040(%0); cachee %1, 0x050(%0) \n" \
+ " cachee %1, 0x060(%0); cachee %1, 0x070(%0) \n" \
+ " cachee %1, 0x080(%0); cachee %1, 0x090(%0) \n" \
+ " cachee %1, 0x0a0(%0); cachee %1, 0x0b0(%0) \n" \
+ " cachee %1, 0x0c0(%0); cachee %1, 0x0d0(%0) \n" \
+ " cachee %1, 0x0e0(%0); cachee %1, 0x0f0(%0) \n" \
+ " cachee %1, 0x100(%0); cachee %1, 0x110(%0) \n" \
+ " cachee %1, 0x120(%0); cachee %1, 0x130(%0) \n" \
+ " cachee %1, 0x140(%0); cachee %1, 0x150(%0) \n" \
+ " cachee %1, 0x160(%0); cachee %1, 0x170(%0) \n" \
+ " cachee %1, 0x180(%0); cachee %1, 0x190(%0) \n" \
+ " cachee %1, 0x1a0(%0); cachee %1, 0x1b0(%0) \n" \
+ " cachee %1, 0x1c0(%0); cachee %1, 0x1d0(%0) \n" \
+ " cachee %1, 0x1e0(%0); cachee %1, 0x1f0(%0) \n" \
+ " .set pop \n" \
+ : \
+ : "r" (base), \
+ "i" (op));
+
+#define cache32_unroll32_user(base, op) \
+ __asm__ __volatile__( \
+ " .set push \n" \
+ " .set noreorder \n" \
+ " .set mips0 \n" \
+ " .set eva \n" \
+ " cachee %1, 0x000(%0); cachee %1, 0x020(%0) \n" \
+ " cachee %1, 0x040(%0); cachee %1, 0x060(%0) \n" \
+ " cachee %1, 0x080(%0); cachee %1, 0x0a0(%0) \n" \
+ " cachee %1, 0x0c0(%0); cachee %1, 0x0e0(%0) \n" \
+ " cachee %1, 0x100(%0); cachee %1, 0x120(%0) \n" \
+ " cachee %1, 0x140(%0); cachee %1, 0x160(%0) \n" \
+ " cachee %1, 0x180(%0); cachee %1, 0x1a0(%0) \n" \
+ " cachee %1, 0x1c0(%0); cachee %1, 0x1e0(%0) \n" \
+ " cachee %1, 0x200(%0); cachee %1, 0x220(%0) \n" \
+ " cachee %1, 0x240(%0); cachee %1, 0x260(%0) \n" \
+ " cachee %1, 0x280(%0); cachee %1, 0x2a0(%0) \n" \
+ " cachee %1, 0x2c0(%0); cachee %1, 0x2e0(%0) \n" \
+ " cachee %1, 0x300(%0); cachee %1, 0x320(%0) \n" \
+ " cachee %1, 0x340(%0); cachee %1, 0x360(%0) \n" \
+ " cachee %1, 0x380(%0); cachee %1, 0x3a0(%0) \n" \
+ " cachee %1, 0x3c0(%0); cachee %1, 0x3e0(%0) \n" \
+ " .set pop \n" \
+ : \
+ : "r" (base), \
+ "i" (op));
+
+#define cache64_unroll32_user(base, op) \
+ __asm__ __volatile__( \
+ " .set push \n" \
+ " .set noreorder \n" \
+ " .set mips0 \n" \
+ " .set eva \n" \
+ " cachee %1, 0x000(%0); cachee %1, 0x040(%0) \n" \
+ " cachee %1, 0x080(%0); cachee %1, 0x0c0(%0) \n" \
+ " cachee %1, 0x100(%0); cachee %1, 0x140(%0) \n" \
+ " cachee %1, 0x180(%0); cachee %1, 0x1c0(%0) \n" \
+ " cachee %1, 0x200(%0); cachee %1, 0x240(%0) \n" \
+ " cachee %1, 0x280(%0); cachee %1, 0x2c0(%0) \n" \
+ " cachee %1, 0x300(%0); cachee %1, 0x340(%0) \n" \
+ " cachee %1, 0x380(%0); cachee %1, 0x3c0(%0) \n" \
+ " cachee %1, 0x400(%0); cachee %1, 0x440(%0) \n" \
+ " cachee %1, 0x480(%0); cachee %1, 0x4c0(%0) \n" \
+ " cachee %1, 0x500(%0); cachee %1, 0x540(%0) \n" \
+ " cachee %1, 0x580(%0); cachee %1, 0x5c0(%0) \n" \
+ " cachee %1, 0x600(%0); cachee %1, 0x640(%0) \n" \
+ " cachee %1, 0x680(%0); cachee %1, 0x6c0(%0) \n" \
+ " cachee %1, 0x700(%0); cachee %1, 0x740(%0) \n" \
+ " cachee %1, 0x780(%0); cachee %1, 0x7c0(%0) \n" \
+ " .set pop \n" \
+ : \
+ : "r" (base), \
+ "i" (op));
+
/* build blast_xxx, blast_xxx_page, blast_xxx_page_indexed */
#define __BUILD_BLAST_CACHE(pfx, desc, indexop, hitop, lsize) \
static inline void blast_##pfx##cache##lsize(void) \
@@ -446,9 +532,35 @@ __BUILD_BLAST_CACHE(inv_s, scache, Index_Writeback_Inv_SD, Hit_Invalidate_SD, 32
__BUILD_BLAST_CACHE(inv_s, scache, Index_Writeback_Inv_SD, Hit_Invalidate_SD, 64)
__BUILD_BLAST_CACHE(inv_s, scache, Index_Writeback_Inv_SD, Hit_Invalidate_SD, 128)
+#define __BUILD_BLAST_USER_CACHE(pfx, desc, indexop, hitop, lsize) \
+static inline void blast_##pfx##cache##lsize##_user_page(unsigned long page) \
+{ \
+ unsigned long start = page; \
+ unsigned long end = page + PAGE_SIZE; \
+ \
+ __##pfx##flush_prologue \
+ \
+ do { \
+ cache##lsize##_unroll32_user(start, hitop); \
+ start += lsize * 32; \
+ } while (start < end); \
+ \
+ __##pfx##flush_epilogue \
+}
+
+__BUILD_BLAST_USER_CACHE(d, dcache, Index_Writeback_Inv_D, Hit_Writeback_Inv_D,
+ 16)
+__BUILD_BLAST_USER_CACHE(i, icache, Index_Invalidate_I, Hit_Invalidate_I, 16)
+__BUILD_BLAST_USER_CACHE(d, dcache, Index_Writeback_Inv_D, Hit_Writeback_Inv_D,
+ 32)
+__BUILD_BLAST_USER_CACHE(i, icache, Index_Invalidate_I, Hit_Invalidate_I, 32)
+__BUILD_BLAST_USER_CACHE(d, dcache, Index_Writeback_Inv_D, Hit_Writeback_Inv_D,
+ 64)
+__BUILD_BLAST_USER_CACHE(i, icache, Index_Invalidate_I, Hit_Invalidate_I, 64)
+
/* build blast_xxx_range, protected_blast_xxx_range */
#define __BUILD_BLAST_CACHE_RANGE(pfx, desc, hitop, prot, extra) \
-static inline void prot##extra##blast_##pfx##cache##_range(unsigned long start, \
+static inline void prot##extra##blast_##pfx##cache##_range(unsigned long start,\
unsigned long end) \
{ \
unsigned long lsize = cpu_##desc##_line_size(); \
@@ -467,9 +579,47 @@ static inline void prot##extra##blast_##pfx##cache##_range(unsigned long start,
__##pfx##flush_epilogue \
}
+#ifndef CONFIG_EVA
+
__BUILD_BLAST_CACHE_RANGE(d, dcache, Hit_Writeback_Inv_D, protected_, )
-__BUILD_BLAST_CACHE_RANGE(s, scache, Hit_Writeback_Inv_SD, protected_, )
__BUILD_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I, protected_, )
+
+#else
+
+#define __BUILD_PROT_BLAST_CACHE_RANGE(pfx, desc, hitop) \
+static inline void protected_blast_##pfx##cache##_range(unsigned long start,\
+ unsigned long end) \
+{ \
+ unsigned long lsize = cpu_##desc##_line_size(); \
+ unsigned long addr = start & ~(lsize - 1); \
+ unsigned long aend = (end - 1) & ~(lsize - 1); \
+ \
+ __##pfx##flush_prologue \
+ \
+ if (segment_eq(get_fs(), USER_DS)) { \
+ while (1) { \
+ protected_cachee_op(hitop, addr); \
+ if (addr == aend) \
+ break; \
+ addr += lsize; \
+ } \
+ } else { \
+ while (1) { \
+ protected_cache_op(hitop, addr); \
+ if (addr == aend) \
+ break; \
+ addr += lsize; \
+ } \
+ \
+ } \
+ __##pfx##flush_epilogue \
+}
+
+__BUILD_PROT_BLAST_CACHE_RANGE(d, dcache, Hit_Writeback_Inv_D)
+__BUILD_PROT_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I)
+
+#endif
+__BUILD_BLAST_CACHE_RANGE(s, scache, Hit_Writeback_Inv_SD, protected_, )
__BUILD_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I_Loongson23, \
protected_, loongson23_)
__BUILD_BLAST_CACHE_RANGE(d, dcache, Hit_Writeback_Inv_D, , )
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 41/58] MIPS: asm: r4kcache: Add EVA cache flushing functions
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Add EVA cache flushing functions similar to non-EVA configurations.
Because the cache may or may not contain user virtual addresses, we
need to use the 'cache' or 'cachee' instruction based on whether we
flush the cache on behalf of kernel or user respectively.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/r4kcache.h | 154 ++++++++++++++++++++++++++++++++++++++-
1 file changed, 152 insertions(+), 2 deletions(-)
diff --git a/arch/mips/include/asm/r4kcache.h b/arch/mips/include/asm/r4kcache.h
index ae026bf..b973e02 100644
--- a/arch/mips/include/asm/r4kcache.h
+++ b/arch/mips/include/asm/r4kcache.h
@@ -17,6 +17,7 @@
#include <asm/cpu-features.h>
#include <asm/cpu-type.h>
#include <asm/mipsmtregs.h>
+#include <asm/uaccess.h> /* for segment_eq() */
/*
* This macro return a properly sign-extended address suitable as base address
@@ -374,6 +375,91 @@ static inline void invalidate_tcache_page(unsigned long addr)
: "r" (base), \
"i" (op));
+/*
+ * Perform the cache operation specified by op using a user mode virtual
+ * address while in kernel mode.
+ */
+#define cache16_unroll32_user(base,op) \
+ __asm__ __volatile__( \
+ " .set push \n" \
+ " .set noreorder \n" \
+ " .set mips0 \n" \
+ " .set eva \n" \
+ " cachee %1, 0x000(%0); cachee %1, 0x010(%0) \n" \
+ " cachee %1, 0x020(%0); cachee %1, 0x030(%0) \n" \
+ " cachee %1, 0x040(%0); cachee %1, 0x050(%0) \n" \
+ " cachee %1, 0x060(%0); cachee %1, 0x070(%0) \n" \
+ " cachee %1, 0x080(%0); cachee %1, 0x090(%0) \n" \
+ " cachee %1, 0x0a0(%0); cachee %1, 0x0b0(%0) \n" \
+ " cachee %1, 0x0c0(%0); cachee %1, 0x0d0(%0) \n" \
+ " cachee %1, 0x0e0(%0); cachee %1, 0x0f0(%0) \n" \
+ " cachee %1, 0x100(%0); cachee %1, 0x110(%0) \n" \
+ " cachee %1, 0x120(%0); cachee %1, 0x130(%0) \n" \
+ " cachee %1, 0x140(%0); cachee %1, 0x150(%0) \n" \
+ " cachee %1, 0x160(%0); cachee %1, 0x170(%0) \n" \
+ " cachee %1, 0x180(%0); cachee %1, 0x190(%0) \n" \
+ " cachee %1, 0x1a0(%0); cachee %1, 0x1b0(%0) \n" \
+ " cachee %1, 0x1c0(%0); cachee %1, 0x1d0(%0) \n" \
+ " cachee %1, 0x1e0(%0); cachee %1, 0x1f0(%0) \n" \
+ " .set pop \n" \
+ : \
+ : "r" (base), \
+ "i" (op));
+
+#define cache32_unroll32_user(base, op) \
+ __asm__ __volatile__( \
+ " .set push \n" \
+ " .set noreorder \n" \
+ " .set mips0 \n" \
+ " .set eva \n" \
+ " cachee %1, 0x000(%0); cachee %1, 0x020(%0) \n" \
+ " cachee %1, 0x040(%0); cachee %1, 0x060(%0) \n" \
+ " cachee %1, 0x080(%0); cachee %1, 0x0a0(%0) \n" \
+ " cachee %1, 0x0c0(%0); cachee %1, 0x0e0(%0) \n" \
+ " cachee %1, 0x100(%0); cachee %1, 0x120(%0) \n" \
+ " cachee %1, 0x140(%0); cachee %1, 0x160(%0) \n" \
+ " cachee %1, 0x180(%0); cachee %1, 0x1a0(%0) \n" \
+ " cachee %1, 0x1c0(%0); cachee %1, 0x1e0(%0) \n" \
+ " cachee %1, 0x200(%0); cachee %1, 0x220(%0) \n" \
+ " cachee %1, 0x240(%0); cachee %1, 0x260(%0) \n" \
+ " cachee %1, 0x280(%0); cachee %1, 0x2a0(%0) \n" \
+ " cachee %1, 0x2c0(%0); cachee %1, 0x2e0(%0) \n" \
+ " cachee %1, 0x300(%0); cachee %1, 0x320(%0) \n" \
+ " cachee %1, 0x340(%0); cachee %1, 0x360(%0) \n" \
+ " cachee %1, 0x380(%0); cachee %1, 0x3a0(%0) \n" \
+ " cachee %1, 0x3c0(%0); cachee %1, 0x3e0(%0) \n" \
+ " .set pop \n" \
+ : \
+ : "r" (base), \
+ "i" (op));
+
+#define cache64_unroll32_user(base, op) \
+ __asm__ __volatile__( \
+ " .set push \n" \
+ " .set noreorder \n" \
+ " .set mips0 \n" \
+ " .set eva \n" \
+ " cachee %1, 0x000(%0); cachee %1, 0x040(%0) \n" \
+ " cachee %1, 0x080(%0); cachee %1, 0x0c0(%0) \n" \
+ " cachee %1, 0x100(%0); cachee %1, 0x140(%0) \n" \
+ " cachee %1, 0x180(%0); cachee %1, 0x1c0(%0) \n" \
+ " cachee %1, 0x200(%0); cachee %1, 0x240(%0) \n" \
+ " cachee %1, 0x280(%0); cachee %1, 0x2c0(%0) \n" \
+ " cachee %1, 0x300(%0); cachee %1, 0x340(%0) \n" \
+ " cachee %1, 0x380(%0); cachee %1, 0x3c0(%0) \n" \
+ " cachee %1, 0x400(%0); cachee %1, 0x440(%0) \n" \
+ " cachee %1, 0x480(%0); cachee %1, 0x4c0(%0) \n" \
+ " cachee %1, 0x500(%0); cachee %1, 0x540(%0) \n" \
+ " cachee %1, 0x580(%0); cachee %1, 0x5c0(%0) \n" \
+ " cachee %1, 0x600(%0); cachee %1, 0x640(%0) \n" \
+ " cachee %1, 0x680(%0); cachee %1, 0x6c0(%0) \n" \
+ " cachee %1, 0x700(%0); cachee %1, 0x740(%0) \n" \
+ " cachee %1, 0x780(%0); cachee %1, 0x7c0(%0) \n" \
+ " .set pop \n" \
+ : \
+ : "r" (base), \
+ "i" (op));
+
/* build blast_xxx, blast_xxx_page, blast_xxx_page_indexed */
#define __BUILD_BLAST_CACHE(pfx, desc, indexop, hitop, lsize) \
static inline void blast_##pfx##cache##lsize(void) \
@@ -446,9 +532,35 @@ __BUILD_BLAST_CACHE(inv_s, scache, Index_Writeback_Inv_SD, Hit_Invalidate_SD, 32
__BUILD_BLAST_CACHE(inv_s, scache, Index_Writeback_Inv_SD, Hit_Invalidate_SD, 64)
__BUILD_BLAST_CACHE(inv_s, scache, Index_Writeback_Inv_SD, Hit_Invalidate_SD, 128)
+#define __BUILD_BLAST_USER_CACHE(pfx, desc, indexop, hitop, lsize) \
+static inline void blast_##pfx##cache##lsize##_user_page(unsigned long page) \
+{ \
+ unsigned long start = page; \
+ unsigned long end = page + PAGE_SIZE; \
+ \
+ __##pfx##flush_prologue \
+ \
+ do { \
+ cache##lsize##_unroll32_user(start, hitop); \
+ start += lsize * 32; \
+ } while (start < end); \
+ \
+ __##pfx##flush_epilogue \
+}
+
+__BUILD_BLAST_USER_CACHE(d, dcache, Index_Writeback_Inv_D, Hit_Writeback_Inv_D,
+ 16)
+__BUILD_BLAST_USER_CACHE(i, icache, Index_Invalidate_I, Hit_Invalidate_I, 16)
+__BUILD_BLAST_USER_CACHE(d, dcache, Index_Writeback_Inv_D, Hit_Writeback_Inv_D,
+ 32)
+__BUILD_BLAST_USER_CACHE(i, icache, Index_Invalidate_I, Hit_Invalidate_I, 32)
+__BUILD_BLAST_USER_CACHE(d, dcache, Index_Writeback_Inv_D, Hit_Writeback_Inv_D,
+ 64)
+__BUILD_BLAST_USER_CACHE(i, icache, Index_Invalidate_I, Hit_Invalidate_I, 64)
+
/* build blast_xxx_range, protected_blast_xxx_range */
#define __BUILD_BLAST_CACHE_RANGE(pfx, desc, hitop, prot, extra) \
-static inline void prot##extra##blast_##pfx##cache##_range(unsigned long start, \
+static inline void prot##extra##blast_##pfx##cache##_range(unsigned long start,\
unsigned long end) \
{ \
unsigned long lsize = cpu_##desc##_line_size(); \
@@ -467,9 +579,47 @@ static inline void prot##extra##blast_##pfx##cache##_range(unsigned long start,
__##pfx##flush_epilogue \
}
+#ifndef CONFIG_EVA
+
__BUILD_BLAST_CACHE_RANGE(d, dcache, Hit_Writeback_Inv_D, protected_, )
-__BUILD_BLAST_CACHE_RANGE(s, scache, Hit_Writeback_Inv_SD, protected_, )
__BUILD_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I, protected_, )
+
+#else
+
+#define __BUILD_PROT_BLAST_CACHE_RANGE(pfx, desc, hitop) \
+static inline void protected_blast_##pfx##cache##_range(unsigned long start,\
+ unsigned long end) \
+{ \
+ unsigned long lsize = cpu_##desc##_line_size(); \
+ unsigned long addr = start & ~(lsize - 1); \
+ unsigned long aend = (end - 1) & ~(lsize - 1); \
+ \
+ __##pfx##flush_prologue \
+ \
+ if (segment_eq(get_fs(), USER_DS)) { \
+ while (1) { \
+ protected_cachee_op(hitop, addr); \
+ if (addr == aend) \
+ break; \
+ addr += lsize; \
+ } \
+ } else { \
+ while (1) { \
+ protected_cache_op(hitop, addr); \
+ if (addr == aend) \
+ break; \
+ addr += lsize; \
+ } \
+ \
+ } \
+ __##pfx##flush_epilogue \
+}
+
+__BUILD_PROT_BLAST_CACHE_RANGE(d, dcache, Hit_Writeback_Inv_D)
+__BUILD_PROT_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I)
+
+#endif
+__BUILD_BLAST_CACHE_RANGE(s, scache, Hit_Writeback_Inv_SD, protected_, )
__BUILD_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I_Loongson23, \
protected_, loongson23_)
__BUILD_BLAST_CACHE_RANGE(d, dcache, Hit_Writeback_Inv_D, , )
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 42/58] MIPS: kernel: {ftrace,kgdb}: Set correct address limit for cache flushes
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
When flushing the icache, make sure the address limit is correct
so the appropriate 'cache' instruction will be used. This has no
impact on cores operating in non-eva mode. However, when EVA is
enabled, we ensure that 'cache' will be used instead of 'cachee'.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/ftrace.c | 4 ++++
arch/mips/kernel/kgdb.c | 17 ++++++++++++++++-
2 files changed, 20 insertions(+), 1 deletion(-)
diff --git a/arch/mips/kernel/ftrace.c b/arch/mips/kernel/ftrace.c
index 185ba25..ddcc350 100644
--- a/arch/mips/kernel/ftrace.c
+++ b/arch/mips/kernel/ftrace.c
@@ -90,6 +90,7 @@ static inline void ftrace_dyn_arch_init_insns(void)
static int ftrace_modify_code(unsigned long ip, unsigned int new_code)
{
int faulted;
+ mm_segment_t old_fs;
/* *(unsigned int *)ip = new_code; */
safe_store_code(new_code, ip, faulted);
@@ -97,7 +98,10 @@ static int ftrace_modify_code(unsigned long ip, unsigned int new_code)
if (unlikely(faulted))
return -EFAULT;
+ old_fs = get_fs();
+ set_fs(get_ds());
flush_icache_range(ip, ip + 8);
+ set_fs(old_fs);
return 0;
}
diff --git a/arch/mips/kernel/kgdb.c b/arch/mips/kernel/kgdb.c
index fcaac2f..a70ffe4 100644
--- a/arch/mips/kernel/kgdb.c
+++ b/arch/mips/kernel/kgdb.c
@@ -208,7 +208,14 @@ void arch_kgdb_breakpoint(void)
static void kgdb_call_nmi_hook(void *ignored)
{
+ mm_segment_t old_fs;
+
+ old_fs = get_fs();
+ set_fs(get_ds());
+
kgdb_nmicallback(raw_smp_processor_id(), NULL);
+
+ set_fs(old_fs);
}
void kgdb_roundup_cpus(unsigned long flags)
@@ -282,6 +289,7 @@ static int kgdb_mips_notify(struct notifier_block *self, unsigned long cmd,
struct die_args *args = (struct die_args *)ptr;
struct pt_regs *regs = args->regs;
int trap = (regs->cp0_cause & 0x7c) >> 2;
+ mm_segment_t old_fs;
#ifdef CONFIG_KPROBES
/*
@@ -296,11 +304,17 @@ static int kgdb_mips_notify(struct notifier_block *self, unsigned long cmd,
if (user_mode(regs))
return NOTIFY_DONE;
+ /* Kernel mode. Set correct address limit */
+ old_fs = get_fs();
+ set_fs(get_ds());
+
if (atomic_read(&kgdb_active) != -1)
kgdb_nmicallback(smp_processor_id(), regs);
- if (kgdb_handle_exception(trap, compute_signal(trap), cmd, regs))
+ if (kgdb_handle_exception(trap, compute_signal(trap), cmd, regs)) {
+ set_fs(old_fs);
return NOTIFY_DONE;
+ }
if (atomic_read(&kgdb_setting_breakpoint))
if ((trap == 9) && (regs->cp0_epc == (unsigned long)breakinst))
@@ -310,6 +324,7 @@ static int kgdb_mips_notify(struct notifier_block *self, unsigned long cmd,
local_irq_enable();
__flush_cache_all();
+ set_fs(old_fs);
return NOTIFY_STOP;
}
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 42/58] MIPS: kernel: {ftrace,kgdb}: Set correct address limit for cache flushes
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
When flushing the icache, make sure the address limit is correct
so the appropriate 'cache' instruction will be used. This has no
impact on cores operating in non-eva mode. However, when EVA is
enabled, we ensure that 'cache' will be used instead of 'cachee'.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/ftrace.c | 4 ++++
arch/mips/kernel/kgdb.c | 17 ++++++++++++++++-
2 files changed, 20 insertions(+), 1 deletion(-)
diff --git a/arch/mips/kernel/ftrace.c b/arch/mips/kernel/ftrace.c
index 185ba25..ddcc350 100644
--- a/arch/mips/kernel/ftrace.c
+++ b/arch/mips/kernel/ftrace.c
@@ -90,6 +90,7 @@ static inline void ftrace_dyn_arch_init_insns(void)
static int ftrace_modify_code(unsigned long ip, unsigned int new_code)
{
int faulted;
+ mm_segment_t old_fs;
/* *(unsigned int *)ip = new_code; */
safe_store_code(new_code, ip, faulted);
@@ -97,7 +98,10 @@ static int ftrace_modify_code(unsigned long ip, unsigned int new_code)
if (unlikely(faulted))
return -EFAULT;
+ old_fs = get_fs();
+ set_fs(get_ds());
flush_icache_range(ip, ip + 8);
+ set_fs(old_fs);
return 0;
}
diff --git a/arch/mips/kernel/kgdb.c b/arch/mips/kernel/kgdb.c
index fcaac2f..a70ffe4 100644
--- a/arch/mips/kernel/kgdb.c
+++ b/arch/mips/kernel/kgdb.c
@@ -208,7 +208,14 @@ void arch_kgdb_breakpoint(void)
static void kgdb_call_nmi_hook(void *ignored)
{
+ mm_segment_t old_fs;
+
+ old_fs = get_fs();
+ set_fs(get_ds());
+
kgdb_nmicallback(raw_smp_processor_id(), NULL);
+
+ set_fs(old_fs);
}
void kgdb_roundup_cpus(unsigned long flags)
@@ -282,6 +289,7 @@ static int kgdb_mips_notify(struct notifier_block *self, unsigned long cmd,
struct die_args *args = (struct die_args *)ptr;
struct pt_regs *regs = args->regs;
int trap = (regs->cp0_cause & 0x7c) >> 2;
+ mm_segment_t old_fs;
#ifdef CONFIG_KPROBES
/*
@@ -296,11 +304,17 @@ static int kgdb_mips_notify(struct notifier_block *self, unsigned long cmd,
if (user_mode(regs))
return NOTIFY_DONE;
+ /* Kernel mode. Set correct address limit */
+ old_fs = get_fs();
+ set_fs(get_ds());
+
if (atomic_read(&kgdb_active) != -1)
kgdb_nmicallback(smp_processor_id(), regs);
- if (kgdb_handle_exception(trap, compute_signal(trap), cmd, regs))
+ if (kgdb_handle_exception(trap, compute_signal(trap), cmd, regs)) {
+ set_fs(old_fs);
return NOTIFY_DONE;
+ }
if (atomic_read(&kgdb_setting_breakpoint))
if ((trap == 9) && (regs->cp0_epc == (unsigned long)breakinst))
@@ -310,6 +324,7 @@ static int kgdb_mips_notify(struct notifier_block *self, unsigned long cmd,
local_irq_enable();
__flush_cache_all();
+ set_fs(old_fs);
return NOTIFY_STOP;
}
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 43/58] MIPS: asm: page: Allow __pa_symbol overrides
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
This will allow platforms to use an alternative way to get
the physical address of a symbol.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/page.h | 2 ++
1 file changed, 2 insertions(+)
diff --git a/arch/mips/include/asm/page.h b/arch/mips/include/asm/page.h
index 5e08bcc..5699ec3 100644
--- a/arch/mips/include/asm/page.h
+++ b/arch/mips/include/asm/page.h
@@ -190,7 +190,9 @@ typedef struct { unsigned long pgprot; } pgprot_t;
* https://patchwork.linux-mips.org/patch/1541/
*/
+#ifndef __pa_symbol
#define __pa_symbol(x) __pa(RELOC_HIDE((unsigned long)(x), 0))
+#endif
#define pfn_to_kaddr(pfn) __va((pfn) << PAGE_SHIFT)
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 43/58] MIPS: asm: page: Allow __pa_symbol overrides
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
This will allow platforms to use an alternative way to get
the physical address of a symbol.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/page.h | 2 ++
1 file changed, 2 insertions(+)
diff --git a/arch/mips/include/asm/page.h b/arch/mips/include/asm/page.h
index 5e08bcc..5699ec3 100644
--- a/arch/mips/include/asm/page.h
+++ b/arch/mips/include/asm/page.h
@@ -190,7 +190,9 @@ typedef struct { unsigned long pgprot; } pgprot_t;
* https://patchwork.linux-mips.org/patch/1541/
*/
+#ifndef __pa_symbol
#define __pa_symbol(x) __pa(RELOC_HIDE((unsigned long)(x), 0))
+#endif
#define pfn_to_kaddr(pfn) __va((pfn) << PAGE_SHIFT)
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 44/58] MIPS: asm: cpu: Add cpu flag for Enhanced Virtual Addressing
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
The MIPS *Aptiv family uses bit 28 in Config5 CP0 register to
indicate whether the core supports EVA or not.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/cpu-features.h | 4 +++-
arch/mips/include/asm/cpu.h | 1 +
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/arch/mips/include/asm/cpu-features.h b/arch/mips/include/asm/cpu-features.h
index 9052fb9..6b95c8e 100644
--- a/arch/mips/include/asm/cpu-features.h
+++ b/arch/mips/include/asm/cpu-features.h
@@ -26,7 +26,9 @@
#ifndef cpu_has_segments
#define cpu_has_segments (cpu_data[0].options & MIPS_CPU_SEGMENTS)
#endif
-
+#ifndef cpu_has_eva
+#define cpu_has_eva (cpu_data[0].options & MIPS_CPU_EVA)
+#endif
/*
* For the moment we don't consider R6000 and R8000 so we can assume that
diff --git a/arch/mips/include/asm/cpu.h b/arch/mips/include/asm/cpu.h
index 76411df..1474750 100644
--- a/arch/mips/include/asm/cpu.h
+++ b/arch/mips/include/asm/cpu.h
@@ -358,6 +358,7 @@ enum cpu_type_enum {
#define MIPS_CPU_MICROMIPS 0x01000000 /* CPU has microMIPS capability */
#define MIPS_CPU_TLBINV 0x02000000 /* CPU supports TLBINV/F */
#define MIPS_CPU_SEGMENTS 0x04000000 /* CPU supports Segmentation Control registers */
+#define MIPS_CPU_EVA 0x80000000 /* CPU supports Enhanced Virtual Addressing */
/*
* CPU ASE encodings
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 44/58] MIPS: asm: cpu: Add cpu flag for Enhanced Virtual Addressing
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
The MIPS *Aptiv family uses bit 28 in Config5 CP0 register to
indicate whether the core supports EVA or not.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/cpu-features.h | 4 +++-
arch/mips/include/asm/cpu.h | 1 +
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/arch/mips/include/asm/cpu-features.h b/arch/mips/include/asm/cpu-features.h
index 9052fb9..6b95c8e 100644
--- a/arch/mips/include/asm/cpu-features.h
+++ b/arch/mips/include/asm/cpu-features.h
@@ -26,7 +26,9 @@
#ifndef cpu_has_segments
#define cpu_has_segments (cpu_data[0].options & MIPS_CPU_SEGMENTS)
#endif
-
+#ifndef cpu_has_eva
+#define cpu_has_eva (cpu_data[0].options & MIPS_CPU_EVA)
+#endif
/*
* For the moment we don't consider R6000 and R8000 so we can assume that
diff --git a/arch/mips/include/asm/cpu.h b/arch/mips/include/asm/cpu.h
index 76411df..1474750 100644
--- a/arch/mips/include/asm/cpu.h
+++ b/arch/mips/include/asm/cpu.h
@@ -358,6 +358,7 @@ enum cpu_type_enum {
#define MIPS_CPU_MICROMIPS 0x01000000 /* CPU has microMIPS capability */
#define MIPS_CPU_TLBINV 0x02000000 /* CPU supports TLBINV/F */
#define MIPS_CPU_SEGMENTS 0x04000000 /* CPU supports Segmentation Control registers */
+#define MIPS_CPU_EVA 0x80000000 /* CPU supports Enhanced Virtual Addressing */
/*
* CPU ASE encodings
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 45/58] MIPS: kernel: cpu-probe: Enable EVA option on supported cores
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/cpu-probe.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/arch/mips/kernel/cpu-probe.c b/arch/mips/kernel/cpu-probe.c
index 530f832..c42731c 100644
--- a/arch/mips/kernel/cpu-probe.c
+++ b/arch/mips/kernel/cpu-probe.c
@@ -367,6 +367,9 @@ static inline unsigned int decode_config5(struct cpuinfo_mips *c)
config5 &= ~MIPS_CONF5_UFR;
write_c0_config5(config5);
+ if (config5 & MIPS_CONF5_EVA)
+ c->options |= MIPS_CPU_EVA;
+
return config5 & MIPS_CONF_M;
}
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 45/58] MIPS: kernel: cpu-probe: Enable EVA option on supported cores
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/cpu-probe.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/arch/mips/kernel/cpu-probe.c b/arch/mips/kernel/cpu-probe.c
index 530f832..c42731c 100644
--- a/arch/mips/kernel/cpu-probe.c
+++ b/arch/mips/kernel/cpu-probe.c
@@ -367,6 +367,9 @@ static inline unsigned int decode_config5(struct cpuinfo_mips *c)
config5 &= ~MIPS_CONF5_UFR;
write_c0_config5(config5);
+ if (config5 & MIPS_CONF5_EVA)
+ c->options |= MIPS_CPU_EVA;
+
return config5 & MIPS_CONF_M;
}
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 46/58] MIPS: kernel: proc: Add EVA to the list of CPU features
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/proc.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/mips/kernel/proc.c b/arch/mips/kernel/proc.c
index 00d2097..ead0e47 100644
--- a/arch/mips/kernel/proc.c
+++ b/arch/mips/kernel/proc.c
@@ -95,6 +95,7 @@ static int show_cpuinfo(struct seq_file *m, void *v)
if (cpu_has_mipsmt) seq_printf(m, "%s", " mt");
if (cpu_has_mmips) seq_printf(m, "%s", " micromips");
if (cpu_has_vz) seq_printf(m, "%s", " vz");
+ if (cpu_has_eva) seq_printf(m, "%s", " eva");
seq_printf(m, "\n");
if (cpu_has_mmips) {
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 46/58] MIPS: kernel: proc: Add EVA to the list of CPU features
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/kernel/proc.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/mips/kernel/proc.c b/arch/mips/kernel/proc.c
index 00d2097..ead0e47 100644
--- a/arch/mips/kernel/proc.c
+++ b/arch/mips/kernel/proc.c
@@ -95,6 +95,7 @@ static int show_cpuinfo(struct seq_file *m, void *v)
if (cpu_has_mipsmt) seq_printf(m, "%s", " mt");
if (cpu_has_mmips) seq_printf(m, "%s", " micromips");
if (cpu_has_vz) seq_printf(m, "%s", " vz");
+ if (cpu_has_eva) seq_printf(m, "%s", " eva");
seq_printf(m, "\n");
if (cpu_has_mmips) {
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 47/58] MIPS: mm: init: Add free_init_pages() callback for EVA
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
A core in EVA mode can have any possible segment mapping, so the
default free_initmem_default() function may not always work as expected.
Therefore, add a callback that platforms can use to free up the init section.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/bootinfo.h | 2 ++
arch/mips/mm/init.c | 12 +++++++++++-
2 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/arch/mips/include/asm/bootinfo.h b/arch/mips/include/asm/bootinfo.h
index 4d2cdea..83a4db4e 100644
--- a/arch/mips/include/asm/bootinfo.h
+++ b/arch/mips/include/asm/bootinfo.h
@@ -112,6 +112,8 @@ extern void prom_free_prom_memory(void);
extern void free_init_pages(const char *what,
unsigned long begin, unsigned long end);
+extern void (*free_init_pages_eva)(void *begin, void *end);
+
/*
* Initial kernel command line, usually setup by prom_init()
*/
diff --git a/arch/mips/mm/init.c b/arch/mips/mm/init.c
index 6b59617..4fc74c7 100644
--- a/arch/mips/mm/init.c
+++ b/arch/mips/mm/init.c
@@ -422,10 +422,20 @@ void free_initrd_mem(unsigned long start, unsigned long end)
}
#endif
+void (*free_init_pages_eva)(void *begin, void *end) = NULL;
+
void __init_refok free_initmem(void)
{
prom_free_prom_memory();
- free_initmem_default(POISON_FREE_INITMEM);
+ /*
+ * Let the platform define a specific function to free the
+ * init section since EVA may have used any possible mapping
+ * between virtual and physical addresses.
+ */
+ if (free_init_pages_eva)
+ free_init_pages_eva((void *)&__init_begin, (void *)&__init_end);
+ else
+ free_initmem_default(POISON_FREE_INITMEM);
}
#ifndef CONFIG_MIPS_PGD_C0_CONTEXT
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 47/58] MIPS: mm: init: Add free_init_pages() callback for EVA
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
A core in EVA mode can have any possible segment mapping, so the
default free_initmem_default() function may not always work as expected.
Therefore, add a callback that platforms can use to free up the init section.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/bootinfo.h | 2 ++
arch/mips/mm/init.c | 12 +++++++++++-
2 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/arch/mips/include/asm/bootinfo.h b/arch/mips/include/asm/bootinfo.h
index 4d2cdea..83a4db4e 100644
--- a/arch/mips/include/asm/bootinfo.h
+++ b/arch/mips/include/asm/bootinfo.h
@@ -112,6 +112,8 @@ extern void prom_free_prom_memory(void);
extern void free_init_pages(const char *what,
unsigned long begin, unsigned long end);
+extern void (*free_init_pages_eva)(void *begin, void *end);
+
/*
* Initial kernel command line, usually setup by prom_init()
*/
diff --git a/arch/mips/mm/init.c b/arch/mips/mm/init.c
index 6b59617..4fc74c7 100644
--- a/arch/mips/mm/init.c
+++ b/arch/mips/mm/init.c
@@ -422,10 +422,20 @@ void free_initrd_mem(unsigned long start, unsigned long end)
}
#endif
+void (*free_init_pages_eva)(void *begin, void *end) = NULL;
+
void __init_refok free_initmem(void)
{
prom_free_prom_memory();
- free_initmem_default(POISON_FREE_INITMEM);
+ /*
+ * Let the platform define a specific function to free the
+ * init section since EVA may have used any possible mapping
+ * between virtual and physical addresses.
+ */
+ if (free_init_pages_eva)
+ free_init_pages_eva((void *)&__init_begin, (void *)&__init_end);
+ else
+ free_initmem_default(POISON_FREE_INITMEM);
}
#ifndef CONFIG_MIPS_PGD_C0_CONTEXT
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 48/58] MIPS: mm: c-r4k: Build EVA {d,i}cache flushing functions
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Build EVA specific cache flushing functions (ie cachee).
They will be used by a subsequent patch.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/mm/c-r4k.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 47 insertions(+)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 13b549a..0c9c693 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -123,6 +123,28 @@ static void r4k_blast_dcache_page_setup(void)
r4k_blast_dcache_page = r4k_blast_dcache_page_dc64;
}
+#ifndef CONFIG_EVA
+#define r4k_blast_dcache_user_page r4k_blast_dcache_page
+#else
+
+static void (*r4k_blast_dcache_user_page)(unsigned long addr);
+
+static void r4k_blast_dcache_user_page_setup(void)
+{
+ unsigned long dc_lsize = cpu_dcache_line_size();
+
+ if (dc_lsize == 0)
+ r4k_blast_dcache_user_page = (void *)cache_noop;
+ else if (dc_lsize == 16)
+ r4k_blast_dcache_user_page = blast_dcache16_user_page;
+ else if (dc_lsize == 32)
+ r4k_blast_dcache_user_page = blast_dcache32_user_page;
+ else if (dc_lsize == 64)
+ r4k_blast_dcache_user_page = blast_dcache64_user_page;
+}
+
+#endif
+
static void (* r4k_blast_dcache_page_indexed)(unsigned long addr);
static void r4k_blast_dcache_page_indexed_setup(void)
@@ -243,6 +265,27 @@ static void r4k_blast_icache_page_setup(void)
r4k_blast_icache_page = blast_icache64_page;
}
+#ifndef CONFIG_EVA
+#define r4k_blast_icache_user_page r4k_blast_icache_page
+#else
+
+static void (*r4k_blast_icache_user_page)(unsigned long addr);
+
+static void __cpuinit r4k_blast_icache_user_page_setup(void)
+{
+ unsigned long ic_lsize = cpu_icache_line_size();
+
+ if (ic_lsize == 0)
+ r4k_blast_icache_user_page = (void *)cache_noop;
+ else if (ic_lsize == 16)
+ r4k_blast_icache_user_page = blast_icache16_user_page;
+ else if (ic_lsize == 32)
+ r4k_blast_icache_user_page = blast_icache32_user_page;
+ else if (ic_lsize == 64)
+ r4k_blast_icache_user_page = blast_icache64_user_page;
+}
+
+#endif
static void (* r4k_blast_icache_page_indexed)(unsigned long addr);
@@ -1454,6 +1497,10 @@ void r4k_cache_init(void)
r4k_blast_scache_page_setup();
r4k_blast_scache_page_indexed_setup();
r4k_blast_scache_setup();
+#ifdef CONFIG_EVA
+ r4k_blast_dcache_user_page_setup();
+ r4k_blast_icache_user_page_setup();
+#endif
/*
* Some MIPS32 and MIPS64 processors have physically indexed caches.
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 48/58] MIPS: mm: c-r4k: Build EVA {d,i}cache flushing functions
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Build EVA specific cache flushing functions (ie cachee).
They will be used by a subsequent patch.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/mm/c-r4k.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 47 insertions(+)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 13b549a..0c9c693 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -123,6 +123,28 @@ static void r4k_blast_dcache_page_setup(void)
r4k_blast_dcache_page = r4k_blast_dcache_page_dc64;
}
+#ifndef CONFIG_EVA
+#define r4k_blast_dcache_user_page r4k_blast_dcache_page
+#else
+
+static void (*r4k_blast_dcache_user_page)(unsigned long addr);
+
+static void r4k_blast_dcache_user_page_setup(void)
+{
+ unsigned long dc_lsize = cpu_dcache_line_size();
+
+ if (dc_lsize == 0)
+ r4k_blast_dcache_user_page = (void *)cache_noop;
+ else if (dc_lsize == 16)
+ r4k_blast_dcache_user_page = blast_dcache16_user_page;
+ else if (dc_lsize == 32)
+ r4k_blast_dcache_user_page = blast_dcache32_user_page;
+ else if (dc_lsize == 64)
+ r4k_blast_dcache_user_page = blast_dcache64_user_page;
+}
+
+#endif
+
static void (* r4k_blast_dcache_page_indexed)(unsigned long addr);
static void r4k_blast_dcache_page_indexed_setup(void)
@@ -243,6 +265,27 @@ static void r4k_blast_icache_page_setup(void)
r4k_blast_icache_page = blast_icache64_page;
}
+#ifndef CONFIG_EVA
+#define r4k_blast_icache_user_page r4k_blast_icache_page
+#else
+
+static void (*r4k_blast_icache_user_page)(unsigned long addr);
+
+static void __cpuinit r4k_blast_icache_user_page_setup(void)
+{
+ unsigned long ic_lsize = cpu_icache_line_size();
+
+ if (ic_lsize == 0)
+ r4k_blast_icache_user_page = (void *)cache_noop;
+ else if (ic_lsize == 16)
+ r4k_blast_icache_user_page = blast_icache16_user_page;
+ else if (ic_lsize == 32)
+ r4k_blast_icache_user_page = blast_icache32_user_page;
+ else if (ic_lsize == 64)
+ r4k_blast_icache_user_page = blast_icache64_user_page;
+}
+
+#endif
static void (* r4k_blast_icache_page_indexed)(unsigned long addr);
@@ -1454,6 +1497,10 @@ void r4k_cache_init(void)
r4k_blast_scache_page_setup();
r4k_blast_scache_page_indexed_setup();
r4k_blast_scache_setup();
+#ifdef CONFIG_EVA
+ r4k_blast_dcache_user_page_setup();
+ r4k_blast_icache_user_page_setup();
+#endif
/*
* Some MIPS32 and MIPS64 processors have physically indexed caches.
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 49/58] MIPS: mm: c-r4k: Add support for flushing user pages from cache
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Use the userspace cache flushing functions if the interrupted
process is a userspace one.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/mm/c-r4k.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 0c9c693..5f8da6a 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -555,7 +555,8 @@ static inline void local_r4k_flush_cache_page(void *args)
}
if (cpu_has_dc_aliases || (exec && !cpu_has_ic_fills_f_dc)) {
- r4k_blast_dcache_page(addr);
+ vaddr ? r4k_blast_dcache_page(addr) :
+ r4k_blast_dcache_user_page(addr);
if (exec && !cpu_icache_snoops_remote_store)
r4k_blast_scache_page(addr);
}
@@ -566,7 +567,8 @@ static inline void local_r4k_flush_cache_page(void *args)
if (cpu_context(cpu, mm) != 0)
drop_mmu_context(mm, cpu);
} else
- r4k_blast_icache_page(addr);
+ vaddr ? r4k_blast_icache_page(addr) :
+ r4k_blast_icache_user_page(addr);
}
if (vaddr) {
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 49/58] MIPS: mm: c-r4k: Add support for flushing user pages from cache
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Use the userspace cache flushing functions if the interrupted
process is a userspace one.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/mm/c-r4k.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 0c9c693..5f8da6a 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -555,7 +555,8 @@ static inline void local_r4k_flush_cache_page(void *args)
}
if (cpu_has_dc_aliases || (exec && !cpu_has_ic_fills_f_dc)) {
- r4k_blast_dcache_page(addr);
+ vaddr ? r4k_blast_dcache_page(addr) :
+ r4k_blast_dcache_user_page(addr);
if (exec && !cpu_icache_snoops_remote_store)
r4k_blast_scache_page(addr);
}
@@ -566,7 +567,8 @@ static inline void local_r4k_flush_cache_page(void *args)
if (cpu_context(cpu, mm) != 0)
drop_mmu_context(mm, cpu);
} else
- r4k_blast_icache_page(addr);
+ vaddr ? r4k_blast_icache_page(addr) :
+ r4k_blast_icache_user_page(addr);
}
if (vaddr) {
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 50/58] MIPS: mm: c-r4k: Flush scache to avoid cache aliases
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
There is a chance for the secondary cache to have memory
aliases. This can happen if the bootloader is in a non-EVA mode
(or even in EVA mode but with different mapping from the kernel)
and the kernel switching to EVA afterwards. It's best to flush
the icache to avoid having the secondary CPUs fetching stale
data from it.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/mm/c-r4k.c | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 5f8da6a..9761dfa 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -633,6 +633,17 @@ static inline void local_r4k_flush_icache_range(unsigned long start, unsigned lo
break;
}
}
+#ifdef CONFIG_EVA
+ /*
+ * Due to all possible segment mappings, there might cache aliases
+ * caused by the bootloader being in non-EVA mode, and the CPU switching
+ * to EVA during early kernel init. It's best to flush the scache
+ * to avoid having secondary cores fetching stale data and lead to
+ * kernel crashes.
+ */
+ bc_wback_inv(start, (end - start));
+ __sync();
+#endif
}
static inline void local_r4k_flush_icache_range_ipi(void *args)
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 50/58] MIPS: mm: c-r4k: Flush scache to avoid cache aliases
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
There is a chance for the secondary cache to have memory
aliases. This can happen if the bootloader is in a non-EVA mode
(or even in EVA mode but with different mapping from the kernel)
and the kernel switching to EVA afterwards. It's best to flush
the icache to avoid having the secondary CPUs fetching stale
data from it.
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/mm/c-r4k.c | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 5f8da6a..9761dfa 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -633,6 +633,17 @@ static inline void local_r4k_flush_icache_range(unsigned long start, unsigned lo
break;
}
}
+#ifdef CONFIG_EVA
+ /*
+ * Due to all possible segment mappings, there might cache aliases
+ * caused by the bootloader being in non-EVA mode, and the CPU switching
+ * to EVA during early kernel init. It's best to flush the scache
+ * to avoid having secondary cores fetching stale data and lead to
+ * kernel crashes.
+ */
+ bc_wback_inv(start, (end - start));
+ __sync();
+#endif
}
static inline void local_r4k_flush_icache_range_ipi(void *args)
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 51/58] MIPS: malta: Configure Segment Control registers for EVA boot
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
The Malta board aliases 0x80000000 - 0xffffffff to 0x00000000
- 0x7fffffff ignoring the 256 MB IO hole in 0x10000000.
The physical memory is shifted to 0x80000000 so up to 2GB
can be used. Kuseg is expanded to 3GB (due to board limitations
only 2GB can be accessed) and lowmem (kernel space) is expanded to 2GB.
The Segment Control registers are programmed as follows:
Virtual memory Physical memory Mapping
0x00000000 - 0x7fffffff 0x80000000 - 0xfffffffff MUSUK (kuseg)
0x80000000 - 0x9fffffff 0x00000000 - 0x1ffffffff MUSUK (kseg0)
0xa0000000 - 0xbf000000 0x00000000 - 0x1ffffffff MUSUK (kseg1)
0xc0000000 - 0xdfffffff - MK (kseg2)
0xe0000000 - 0xffffffff - MK (kseg3)
The location of exception vectors remain the same since 0xbfc00000
(traditional exception base) still maps to 0x1fc00000 physical.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
.../include/asm/mach-malta/kernel-entry-init.h | 109 ++++++++++++++++++++-
arch/mips/mti-malta/malta-setup.c | 4 +
2 files changed, 112 insertions(+), 1 deletion(-)
diff --git a/arch/mips/include/asm/mach-malta/kernel-entry-init.h b/arch/mips/include/asm/mach-malta/kernel-entry-init.h
index 0b793e7..9bace9c 100644
--- a/arch/mips/include/asm/mach-malta/kernel-entry-init.h
+++ b/arch/mips/include/asm/mach-malta/kernel-entry-init.h
@@ -5,10 +5,80 @@
*
* Chris Dearman (chris@mips.com)
* Copyright (C) 2007 Mips Technologies, Inc.
+ * Copyright (C) 2014 Imagination Technologies Ltd.
*/
#ifndef __ASM_MACH_MIPS_KERNEL_ENTRY_INIT_H
#define __ASM_MACH_MIPS_KERNEL_ENTRY_INIT_H
+ /*
+ * Prepare segments for EVA boot:
+ *
+ * This is in case the processor boots in legacy configuration
+ * (SI_EVAReset is de-asserted and CONFIG5.K == 0)
+ *
+ * On entry, t1 is loaded with CP0_CONFIG
+ *
+ * ========================= Mappings =============================
+ * Virtual memory Physical memory Mapping
+ * 0x00000000 - 0x7fffffff 0x80000000 - 0xfffffffff MUSUK (kuseg)
+ * Flat 2GB physical memory
+ *
+ * 0x80000000 - 0x9fffffff 0x00000000 - 0x1ffffffff MUSUK (kseg0)
+ * 0xa0000000 - 0xbf000000 0x00000000 - 0x1ffffffff MUSUK (kseg1)
+ * 0xc0000000 - 0xdfffffff - MK (kseg2)
+ * 0xe0000000 - 0xffffffff - MK (kseg3)
+ *
+ *
+ * Lowmem is expanded to 2GB
+ */
+ .macro eva_entry
+ /*
+ * Get Config.K0 value and use it to program
+ * the segmentation registers
+ */
+ andi t1, 0x7 /* CCA */
+ move t2, t1
+ ins t2, t1, 16, 3
+ /* SegCtl0 */
+ li t0, ((MIPS_SEGCFG_MK << MIPS_SEGCFG_AM_SHIFT) | \
+ (0 << MIPS_SEGCFG_PA_SHIFT) | \
+ (1 << MIPS_SEGCFG_EU_SHIFT)) | \
+ (((MIPS_SEGCFG_MK << MIPS_SEGCFG_AM_SHIFT) | \
+ (0 << MIPS_SEGCFG_PA_SHIFT) | \
+ (1 << MIPS_SEGCFG_EU_SHIFT)) << 16)
+ or t0, t2
+ mtc0 t0, $5, 2
+
+ /* SegCtl1 */
+ li t0, ((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) | \
+ (0 << MIPS_SEGCFG_PA_SHIFT) | \
+ (2 << MIPS_SEGCFG_C_SHIFT) | \
+ (1 << MIPS_SEGCFG_EU_SHIFT)) | \
+ (((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) | \
+ (0 << MIPS_SEGCFG_PA_SHIFT) | \
+ (1 << MIPS_SEGCFG_EU_SHIFT)) << 16)
+ ins t0, t1, 16, 3
+ mtc0 t0, $5, 3
+
+ /* SegCtl2 */
+ li t0, ((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) | \
+ (6 << MIPS_SEGCFG_PA_SHIFT) | \
+ (1 << MIPS_SEGCFG_EU_SHIFT)) | \
+ (((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) | \
+ (4 << MIPS_SEGCFG_PA_SHIFT) | \
+ (1 << MIPS_SEGCFG_EU_SHIFT)) << 16)
+ or t0, t2
+ mtc0 t0, $5, 4
+
+ jal mips_ihb
+ mfc0 t0, $16, 5
+ li t2, 0x40000000 /* K bit */
+ or t0, t0, t2
+ mtc0 t0, $16, 5
+ sync
+ jal mips_ihb
+ .endm
+
.macro kernel_entry_setup
#ifdef CONFIG_MIPS_MT_SMTC
mfc0 t0, CP0_CONFIG
@@ -39,8 +109,45 @@
nonmt_processor:
.asciz "SMTC kernel requires the MT ASE to run\n"
__FINIT
-0:
#endif
+
+#ifdef CONFIG_EVA
+ sync
+ ehb
+
+ mfc0 t1, CP0_CONFIG
+ bgez t1, 9f
+ mfc0 t0, CP0_CONFIG, 1
+ bgez t0, 9f
+ mfc0 t0, CP0_CONFIG, 2
+ bgez t0, 9f
+ mfc0 t0, CP0_CONFIG, 3
+ sll t0, t0, 6 /* SC bit */
+ bgez t0, 9f
+
+ eva_entry
+ b 0f
+9:
+ /* Assume we came from YAMON... */
+ PTR_LA v0, 0x9fc00534 /* YAMON print */
+ lw v0, (v0)
+ move a0, zero
+ PTR_LA a1, nonsc_processor
+ jal v0
+
+ PTR_LA v0, 0x9fc00520 /* YAMON exit */
+ lw v0, (v0)
+ li a0, 1
+ jal v0
+
+1: b 1b
+ nop
+ __INITDATA
+nonsc_processor:
+ .asciz "EVA kernel requires a MIPS core with Segment Control implemented\n"
+ __FINIT
+#endif /* CONFIG_EVA */
+0:
.endm
/*
diff --git a/arch/mips/mti-malta/malta-setup.c b/arch/mips/mti-malta/malta-setup.c
index c72a069..35057a8 100644
--- a/arch/mips/mti-malta/malta-setup.c
+++ b/arch/mips/mti-malta/malta-setup.c
@@ -247,6 +247,10 @@ void __init plat_mem_setup(void)
{
unsigned int i;
+ if (config_enabled(CONFIG_EVA))
+ /* EVA has already been configured in mach-malta/kernel-init.h */
+ pr_info("Enhanced Virtual Addressing (EVA) activated\n");
+
mips_pcibios_init();
/* Request I/O space for devices used on the Malta board. */
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 51/58] MIPS: malta: Configure Segment Control registers for EVA boot
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
The Malta board aliases 0x80000000 - 0xffffffff to 0x00000000
- 0x7fffffff ignoring the 256 MB IO hole in 0x10000000.
The physical memory is shifted to 0x80000000 so up to 2GB
can be used. Kuseg is expanded to 3GB (due to board limitations
only 2GB can be accessed) and lowmem (kernel space) is expanded to 2GB.
The Segment Control registers are programmed as follows:
Virtual memory Physical memory Mapping
0x00000000 - 0x7fffffff 0x80000000 - 0xfffffffff MUSUK (kuseg)
0x80000000 - 0x9fffffff 0x00000000 - 0x1ffffffff MUSUK (kseg0)
0xa0000000 - 0xbf000000 0x00000000 - 0x1ffffffff MUSUK (kseg1)
0xc0000000 - 0xdfffffff - MK (kseg2)
0xe0000000 - 0xffffffff - MK (kseg3)
The location of exception vectors remain the same since 0xbfc00000
(traditional exception base) still maps to 0x1fc00000 physical.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
.../include/asm/mach-malta/kernel-entry-init.h | 109 ++++++++++++++++++++-
arch/mips/mti-malta/malta-setup.c | 4 +
2 files changed, 112 insertions(+), 1 deletion(-)
diff --git a/arch/mips/include/asm/mach-malta/kernel-entry-init.h b/arch/mips/include/asm/mach-malta/kernel-entry-init.h
index 0b793e7..9bace9c 100644
--- a/arch/mips/include/asm/mach-malta/kernel-entry-init.h
+++ b/arch/mips/include/asm/mach-malta/kernel-entry-init.h
@@ -5,10 +5,80 @@
*
* Chris Dearman (chris@mips.com)
* Copyright (C) 2007 Mips Technologies, Inc.
+ * Copyright (C) 2014 Imagination Technologies Ltd.
*/
#ifndef __ASM_MACH_MIPS_KERNEL_ENTRY_INIT_H
#define __ASM_MACH_MIPS_KERNEL_ENTRY_INIT_H
+ /*
+ * Prepare segments for EVA boot:
+ *
+ * This is in case the processor boots in legacy configuration
+ * (SI_EVAReset is de-asserted and CONFIG5.K == 0)
+ *
+ * On entry, t1 is loaded with CP0_CONFIG
+ *
+ * ========================= Mappings =============================
+ * Virtual memory Physical memory Mapping
+ * 0x00000000 - 0x7fffffff 0x80000000 - 0xfffffffff MUSUK (kuseg)
+ * Flat 2GB physical memory
+ *
+ * 0x80000000 - 0x9fffffff 0x00000000 - 0x1ffffffff MUSUK (kseg0)
+ * 0xa0000000 - 0xbf000000 0x00000000 - 0x1ffffffff MUSUK (kseg1)
+ * 0xc0000000 - 0xdfffffff - MK (kseg2)
+ * 0xe0000000 - 0xffffffff - MK (kseg3)
+ *
+ *
+ * Lowmem is expanded to 2GB
+ */
+ .macro eva_entry
+ /*
+ * Get Config.K0 value and use it to program
+ * the segmentation registers
+ */
+ andi t1, 0x7 /* CCA */
+ move t2, t1
+ ins t2, t1, 16, 3
+ /* SegCtl0 */
+ li t0, ((MIPS_SEGCFG_MK << MIPS_SEGCFG_AM_SHIFT) | \
+ (0 << MIPS_SEGCFG_PA_SHIFT) | \
+ (1 << MIPS_SEGCFG_EU_SHIFT)) | \
+ (((MIPS_SEGCFG_MK << MIPS_SEGCFG_AM_SHIFT) | \
+ (0 << MIPS_SEGCFG_PA_SHIFT) | \
+ (1 << MIPS_SEGCFG_EU_SHIFT)) << 16)
+ or t0, t2
+ mtc0 t0, $5, 2
+
+ /* SegCtl1 */
+ li t0, ((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) | \
+ (0 << MIPS_SEGCFG_PA_SHIFT) | \
+ (2 << MIPS_SEGCFG_C_SHIFT) | \
+ (1 << MIPS_SEGCFG_EU_SHIFT)) | \
+ (((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) | \
+ (0 << MIPS_SEGCFG_PA_SHIFT) | \
+ (1 << MIPS_SEGCFG_EU_SHIFT)) << 16)
+ ins t0, t1, 16, 3
+ mtc0 t0, $5, 3
+
+ /* SegCtl2 */
+ li t0, ((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) | \
+ (6 << MIPS_SEGCFG_PA_SHIFT) | \
+ (1 << MIPS_SEGCFG_EU_SHIFT)) | \
+ (((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) | \
+ (4 << MIPS_SEGCFG_PA_SHIFT) | \
+ (1 << MIPS_SEGCFG_EU_SHIFT)) << 16)
+ or t0, t2
+ mtc0 t0, $5, 4
+
+ jal mips_ihb
+ mfc0 t0, $16, 5
+ li t2, 0x40000000 /* K bit */
+ or t0, t0, t2
+ mtc0 t0, $16, 5
+ sync
+ jal mips_ihb
+ .endm
+
.macro kernel_entry_setup
#ifdef CONFIG_MIPS_MT_SMTC
mfc0 t0, CP0_CONFIG
@@ -39,8 +109,45 @@
nonmt_processor:
.asciz "SMTC kernel requires the MT ASE to run\n"
__FINIT
-0:
#endif
+
+#ifdef CONFIG_EVA
+ sync
+ ehb
+
+ mfc0 t1, CP0_CONFIG
+ bgez t1, 9f
+ mfc0 t0, CP0_CONFIG, 1
+ bgez t0, 9f
+ mfc0 t0, CP0_CONFIG, 2
+ bgez t0, 9f
+ mfc0 t0, CP0_CONFIG, 3
+ sll t0, t0, 6 /* SC bit */
+ bgez t0, 9f
+
+ eva_entry
+ b 0f
+9:
+ /* Assume we came from YAMON... */
+ PTR_LA v0, 0x9fc00534 /* YAMON print */
+ lw v0, (v0)
+ move a0, zero
+ PTR_LA a1, nonsc_processor
+ jal v0
+
+ PTR_LA v0, 0x9fc00520 /* YAMON exit */
+ lw v0, (v0)
+ li a0, 1
+ jal v0
+
+1: b 1b
+ nop
+ __INITDATA
+nonsc_processor:
+ .asciz "EVA kernel requires a MIPS core with Segment Control implemented\n"
+ __FINIT
+#endif /* CONFIG_EVA */
+0:
.endm
/*
diff --git a/arch/mips/mti-malta/malta-setup.c b/arch/mips/mti-malta/malta-setup.c
index c72a069..35057a8 100644
--- a/arch/mips/mti-malta/malta-setup.c
+++ b/arch/mips/mti-malta/malta-setup.c
@@ -247,6 +247,10 @@ void __init plat_mem_setup(void)
{
unsigned int i;
+ if (config_enabled(CONFIG_EVA))
+ /* EVA has already been configured in mach-malta/kernel-init.h */
+ pr_info("Enhanced Virtual Addressing (EVA) activated\n");
+
mips_pcibios_init();
/* Request I/O space for devices used on the Malta board. */
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 52/58] MIPS: malta: spaces.h: Add spaces.h file for Malta (EVA)
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Add a spaces.h file for Malta to override certain memory macros
when operating in EVA mode.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/mach-malta/spaces.h | 46 +++++++++++++++++++++++++++++++
1 file changed, 46 insertions(+)
create mode 100644 arch/mips/include/asm/mach-malta/spaces.h
diff --git a/arch/mips/include/asm/mach-malta/spaces.h b/arch/mips/include/asm/mach-malta/spaces.h
new file mode 100644
index 0000000..d7e5497
--- /dev/null
+++ b/arch/mips/include/asm/mach-malta/spaces.h
@@ -0,0 +1,46 @@
+/*
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License. See the file "COPYING" in the main directory of this archive
+ * for more details.
+ *
+ * Copyright (C) 2014 Imagination Technologies Ltd.
+ */
+
+#ifndef _ASM_MALTA_SPACES_H
+#define _ASM_MALTA_SPACES_H
+
+#ifdef CONFIG_EVA
+
+/*
+ * Traditional Malta Board Memory Map for EVA
+ *
+ * 0x00000000 - 0x0fffffff: 1st RAM region, 256MB
+ * 0x10000000 - 0x1bffffff: GIC and CPC Control Registers
+ * 0x1c000000 - 0x1fffffff: I/O And Flash
+ * 0x20000000 - 0x7fffffff: 2nd RAM region, 1.5GB
+ * 0x80000000 - 0xffffffff: Physical memory aliases to 0x0 (2GB)
+ *
+ * The kernel is still located in 0x80000000(kseg0). However,
+ * the physical mask has been shifted to 0x80000000 which exploits the alias
+ * on the Malta board. As a result of which, we override the __pa_symbol
+ * to peform direct mapping from virtual to physical addresses. In other
+ * words, the 0x80000000 virtual address maps to 0x80000000 physical address
+ * which in turn aliases to 0x0. We do this in order to be able to use a flat
+ * 2GB of memory (0x80000000 - 0xffffffff) so we can avoid the I/O hole in
+ * 0x10000000 - 0x1fffffff.
+ * The last 64KB of physical memory are reserved for correct HIGHMEM
+ * macros arithmetics.
+ *
+ */
+
+#define PAGE_OFFSET _AC(0x0, UL)
+#define PHYS_OFFSET _AC(0x80000000, UL)
+#define HIGHMEM_START _AC(0xffff0000, UL)
+
+#define __pa_symbol(x) (RELOC_HIDE((unsigned long)(x), 0))
+
+#endif /* CONFIG_EVA */
+
+#include <asm/mach-generic/spaces.h>
+
+#endif /* _ASM_MALTA_SPACES_H */
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 52/58] MIPS: malta: spaces.h: Add spaces.h file for Malta (EVA)
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Add a spaces.h file for Malta to override certain memory macros
when operating in EVA mode.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/mach-malta/spaces.h | 46 +++++++++++++++++++++++++++++++
1 file changed, 46 insertions(+)
create mode 100644 arch/mips/include/asm/mach-malta/spaces.h
diff --git a/arch/mips/include/asm/mach-malta/spaces.h b/arch/mips/include/asm/mach-malta/spaces.h
new file mode 100644
index 0000000..d7e5497
--- /dev/null
+++ b/arch/mips/include/asm/mach-malta/spaces.h
@@ -0,0 +1,46 @@
+/*
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License. See the file "COPYING" in the main directory of this archive
+ * for more details.
+ *
+ * Copyright (C) 2014 Imagination Technologies Ltd.
+ */
+
+#ifndef _ASM_MALTA_SPACES_H
+#define _ASM_MALTA_SPACES_H
+
+#ifdef CONFIG_EVA
+
+/*
+ * Traditional Malta Board Memory Map for EVA
+ *
+ * 0x00000000 - 0x0fffffff: 1st RAM region, 256MB
+ * 0x10000000 - 0x1bffffff: GIC and CPC Control Registers
+ * 0x1c000000 - 0x1fffffff: I/O And Flash
+ * 0x20000000 - 0x7fffffff: 2nd RAM region, 1.5GB
+ * 0x80000000 - 0xffffffff: Physical memory aliases to 0x0 (2GB)
+ *
+ * The kernel is still located in 0x80000000(kseg0). However,
+ * the physical mask has been shifted to 0x80000000 which exploits the alias
+ * on the Malta board. As a result of which, we override the __pa_symbol
+ * to peform direct mapping from virtual to physical addresses. In other
+ * words, the 0x80000000 virtual address maps to 0x80000000 physical address
+ * which in turn aliases to 0x0. We do this in order to be able to use a flat
+ * 2GB of memory (0x80000000 - 0xffffffff) so we can avoid the I/O hole in
+ * 0x10000000 - 0x1fffffff.
+ * The last 64KB of physical memory are reserved for correct HIGHMEM
+ * macros arithmetics.
+ *
+ */
+
+#define PAGE_OFFSET _AC(0x0, UL)
+#define PHYS_OFFSET _AC(0x80000000, UL)
+#define HIGHMEM_START _AC(0xffff0000, UL)
+
+#define __pa_symbol(x) (RELOC_HIDE((unsigned long)(x), 0))
+
+#endif /* CONFIG_EVA */
+
+#include <asm/mach-generic/spaces.h>
+
+#endif /* _ASM_MALTA_SPACES_H */
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 53/58] MIPS: malta: malta-memory: Add support for the 'ememsize' variable
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
The 'ememsize' variable is used to denote the real RAM which is
present on the Malta board. This is different compared to 'memsize'
which is capped to 256MB. The 'ememsize' is used to get the actual
physical memory when setting up the Malta memory layout. This only
makes sense in case the core operates in the EVA mode, and it's
ignored otherwise.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/fw/fw.h | 2 +-
arch/mips/mti-malta/malta-memory.c | 36 +++++++++++++++++++++++++++---------
2 files changed, 28 insertions(+), 10 deletions(-)
diff --git a/arch/mips/include/asm/fw/fw.h b/arch/mips/include/asm/fw/fw.h
index d6c50a7..f3e6978 100644
--- a/arch/mips/include/asm/fw/fw.h
+++ b/arch/mips/include/asm/fw/fw.h
@@ -38,7 +38,7 @@ extern int *_fw_envp;
extern void fw_init_cmdline(void);
extern char *fw_getcmdline(void);
-extern fw_memblock_t *fw_getmdesc(void);
+extern fw_memblock_t *fw_getmdesc(int);
extern void fw_meminit(void);
extern char *fw_getenv(char *name);
extern unsigned long fw_getenvl(char *name);
diff --git a/arch/mips/mti-malta/malta-memory.c b/arch/mips/mti-malta/malta-memory.c
index 1f73d63..cd04008 100644
--- a/arch/mips/mti-malta/malta-memory.c
+++ b/arch/mips/mti-malta/malta-memory.c
@@ -24,22 +24,30 @@ static fw_memblock_t mdesc[FW_MAX_MEMBLOCKS];
/* determined physical memory size, not overridden by command line args */
unsigned long physical_memsize = 0L;
-fw_memblock_t * __init fw_getmdesc(void)
+fw_memblock_t * __init fw_getmdesc(int eva)
{
- char *memsize_str, *ptr;
- unsigned int memsize;
+ char *memsize_str, *ememsize_str __maybe_unused = NULL, *ptr;
+ unsigned long memsize, ememsize __maybe_unused = 0;
static char cmdline[COMMAND_LINE_SIZE] __initdata;
- long val;
int tmp;
/* otherwise look in the environment */
+
memsize_str = fw_getenv("memsize");
- if (!memsize_str) {
+ if (memsize_str)
+ tmp = kstrtol(memsize_str, 0, &memsize);
+ if (eva) {
+ /* Look for ememsize for EVA */
+ ememsize_str = fw_getenv("ememsize");
+ if (ememsize_str)
+ tmp = kstrtol(ememsize_str, 0, &ememsize);
+ }
+ if (!memsize && !ememsize) {
pr_warn("memsize not set in YAMON, set to default (32Mb)\n");
physical_memsize = 0x02000000;
} else {
- tmp = kstrtol(memsize_str, 0, &val);
- physical_memsize = (unsigned long)val;
+ /* If ememsize is set, then set physical_memsize to that */
+ physical_memsize = ememsize ? : memsize;
}
#ifdef CONFIG_CPU_BIG_ENDIAN
@@ -54,12 +62,22 @@ fw_memblock_t * __init fw_getmdesc(void)
ptr = strstr(cmdline, "memsize=");
if (ptr && (ptr != cmdline) && (*(ptr - 1) != ' '))
ptr = strstr(ptr, " memsize=");
+ /* And now look for ememsize */
+ if (eva) {
+ ptr = strstr(cmdline, "ememsize=");
+ if (ptr && (ptr != cmdline) && (*(ptr - 1) != ' '))
+ ptr = strstr(ptr, " ememsize=");
+ }
if (ptr)
- memsize = memparse(ptr + 8, &ptr);
+ memsize = memparse(ptr + 8 + (eva ? 1 : 0), &ptr);
else
memsize = physical_memsize;
+ /* Last 64K for HIGHMEM arithmetics */
+ if (memsize > 0x7fff0000)
+ memsize = 0x7fff0000;
+
memset(mdesc, 0, sizeof(mdesc));
mdesc[0].type = fw_dontuse;
@@ -109,7 +127,7 @@ void __init fw_meminit(void)
{
fw_memblock_t *p;
- p = fw_getmdesc();
+ p = fw_getmdesc(config_enabled(CONFIG_EVA));
while (p->size) {
long type;
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 53/58] MIPS: malta: malta-memory: Add support for the 'ememsize' variable
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
The 'ememsize' variable is used to denote the real RAM which is
present on the Malta board. This is different compared to 'memsize'
which is capped to 256MB. The 'ememsize' is used to get the actual
physical memory when setting up the Malta memory layout. This only
makes sense in case the core operates in the EVA mode, and it's
ignored otherwise.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/fw/fw.h | 2 +-
arch/mips/mti-malta/malta-memory.c | 36 +++++++++++++++++++++++++++---------
2 files changed, 28 insertions(+), 10 deletions(-)
diff --git a/arch/mips/include/asm/fw/fw.h b/arch/mips/include/asm/fw/fw.h
index d6c50a7..f3e6978 100644
--- a/arch/mips/include/asm/fw/fw.h
+++ b/arch/mips/include/asm/fw/fw.h
@@ -38,7 +38,7 @@ extern int *_fw_envp;
extern void fw_init_cmdline(void);
extern char *fw_getcmdline(void);
-extern fw_memblock_t *fw_getmdesc(void);
+extern fw_memblock_t *fw_getmdesc(int);
extern void fw_meminit(void);
extern char *fw_getenv(char *name);
extern unsigned long fw_getenvl(char *name);
diff --git a/arch/mips/mti-malta/malta-memory.c b/arch/mips/mti-malta/malta-memory.c
index 1f73d63..cd04008 100644
--- a/arch/mips/mti-malta/malta-memory.c
+++ b/arch/mips/mti-malta/malta-memory.c
@@ -24,22 +24,30 @@ static fw_memblock_t mdesc[FW_MAX_MEMBLOCKS];
/* determined physical memory size, not overridden by command line args */
unsigned long physical_memsize = 0L;
-fw_memblock_t * __init fw_getmdesc(void)
+fw_memblock_t * __init fw_getmdesc(int eva)
{
- char *memsize_str, *ptr;
- unsigned int memsize;
+ char *memsize_str, *ememsize_str __maybe_unused = NULL, *ptr;
+ unsigned long memsize, ememsize __maybe_unused = 0;
static char cmdline[COMMAND_LINE_SIZE] __initdata;
- long val;
int tmp;
/* otherwise look in the environment */
+
memsize_str = fw_getenv("memsize");
- if (!memsize_str) {
+ if (memsize_str)
+ tmp = kstrtol(memsize_str, 0, &memsize);
+ if (eva) {
+ /* Look for ememsize for EVA */
+ ememsize_str = fw_getenv("ememsize");
+ if (ememsize_str)
+ tmp = kstrtol(ememsize_str, 0, &ememsize);
+ }
+ if (!memsize && !ememsize) {
pr_warn("memsize not set in YAMON, set to default (32Mb)\n");
physical_memsize = 0x02000000;
} else {
- tmp = kstrtol(memsize_str, 0, &val);
- physical_memsize = (unsigned long)val;
+ /* If ememsize is set, then set physical_memsize to that */
+ physical_memsize = ememsize ? : memsize;
}
#ifdef CONFIG_CPU_BIG_ENDIAN
@@ -54,12 +62,22 @@ fw_memblock_t * __init fw_getmdesc(void)
ptr = strstr(cmdline, "memsize=");
if (ptr && (ptr != cmdline) && (*(ptr - 1) != ' '))
ptr = strstr(ptr, " memsize=");
+ /* And now look for ememsize */
+ if (eva) {
+ ptr = strstr(cmdline, "ememsize=");
+ if (ptr && (ptr != cmdline) && (*(ptr - 1) != ' '))
+ ptr = strstr(ptr, " ememsize=");
+ }
if (ptr)
- memsize = memparse(ptr + 8, &ptr);
+ memsize = memparse(ptr + 8 + (eva ? 1 : 0), &ptr);
else
memsize = physical_memsize;
+ /* Last 64K for HIGHMEM arithmetics */
+ if (memsize > 0x7fff0000)
+ memsize = 0x7fff0000;
+
memset(mdesc, 0, sizeof(mdesc));
mdesc[0].type = fw_dontuse;
@@ -109,7 +127,7 @@ void __init fw_meminit(void)
{
fw_memblock_t *p;
- p = fw_getmdesc();
+ p = fw_getmdesc(config_enabled(CONFIG_EVA));
while (p->size) {
long type;
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 54/58] MIPS: malta: malta-memory: Use the PHYS_OFFSET to build the memory map
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
PHYS_OFFSET is used to denote the physical start address of the
first bank of RAM. When the Malta board is in EVA mode, the physical
start address of RAM is shifted to 0x80000000 so it's necessary to use
this macro in order to make the code EVA agnostic.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/mti-malta/malta-memory.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/arch/mips/mti-malta/malta-memory.c b/arch/mips/mti-malta/malta-memory.c
index cd04008..9235aee8 100644
--- a/arch/mips/mti-malta/malta-memory.c
+++ b/arch/mips/mti-malta/malta-memory.c
@@ -81,11 +81,11 @@ fw_memblock_t * __init fw_getmdesc(int eva)
memset(mdesc, 0, sizeof(mdesc));
mdesc[0].type = fw_dontuse;
- mdesc[0].base = 0x00000000;
+ mdesc[0].base = PHYS_OFFSET;
mdesc[0].size = 0x00001000;
mdesc[1].type = fw_code;
- mdesc[1].base = 0x00001000;
+ mdesc[1].base = mdesc[0].base + 0x00001000UL;
mdesc[1].size = 0x000ef000;
/*
@@ -96,17 +96,17 @@ fw_memblock_t * __init fw_getmdesc(int eva)
* devices.
*/
mdesc[2].type = fw_dontuse;
- mdesc[2].base = 0x000f0000;
+ mdesc[2].base = mdesc[0].base + 0x000f0000UL;
mdesc[2].size = 0x00010000;
mdesc[3].type = fw_dontuse;
- mdesc[3].base = 0x00100000;
+ mdesc[3].base = mdesc[0].base + 0x00100000UL;
mdesc[3].size = CPHYSADDR(PFN_ALIGN((unsigned long)&_end)) -
- mdesc[3].base;
+ 0x00100000UL;
mdesc[4].type = fw_free;
- mdesc[4].base = CPHYSADDR(PFN_ALIGN(&_end));
- mdesc[4].size = memsize - mdesc[4].base;
+ mdesc[4].base = mdesc[0].base + CPHYSADDR(PFN_ALIGN(&_end));
+ mdesc[4].size = memsize - CPHYSADDR(mdesc[4].base);
return &mdesc[0];
}
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 54/58] MIPS: malta: malta-memory: Use the PHYS_OFFSET to build the memory map
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
PHYS_OFFSET is used to denote the physical start address of the
first bank of RAM. When the Malta board is in EVA mode, the physical
start address of RAM is shifted to 0x80000000 so it's necessary to use
this macro in order to make the code EVA agnostic.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/mti-malta/malta-memory.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/arch/mips/mti-malta/malta-memory.c b/arch/mips/mti-malta/malta-memory.c
index cd04008..9235aee8 100644
--- a/arch/mips/mti-malta/malta-memory.c
+++ b/arch/mips/mti-malta/malta-memory.c
@@ -81,11 +81,11 @@ fw_memblock_t * __init fw_getmdesc(int eva)
memset(mdesc, 0, sizeof(mdesc));
mdesc[0].type = fw_dontuse;
- mdesc[0].base = 0x00000000;
+ mdesc[0].base = PHYS_OFFSET;
mdesc[0].size = 0x00001000;
mdesc[1].type = fw_code;
- mdesc[1].base = 0x00001000;
+ mdesc[1].base = mdesc[0].base + 0x00001000UL;
mdesc[1].size = 0x000ef000;
/*
@@ -96,17 +96,17 @@ fw_memblock_t * __init fw_getmdesc(int eva)
* devices.
*/
mdesc[2].type = fw_dontuse;
- mdesc[2].base = 0x000f0000;
+ mdesc[2].base = mdesc[0].base + 0x000f0000UL;
mdesc[2].size = 0x00010000;
mdesc[3].type = fw_dontuse;
- mdesc[3].base = 0x00100000;
+ mdesc[3].base = mdesc[0].base + 0x00100000UL;
mdesc[3].size = CPHYSADDR(PFN_ALIGN((unsigned long)&_end)) -
- mdesc[3].base;
+ 0x00100000UL;
mdesc[4].type = fw_free;
- mdesc[4].base = CPHYSADDR(PFN_ALIGN(&_end));
- mdesc[4].size = memsize - mdesc[4].base;
+ mdesc[4].base = mdesc[0].base + CPHYSADDR(PFN_ALIGN(&_end));
+ mdesc[4].size = memsize - CPHYSADDR(mdesc[4].base);
return &mdesc[0];
}
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 55/58] MIPS: malta: malta-memory: Add free_init_pages_eva() callback
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Use a Malta specific function to free the init section once the
kernel has booted. When operating in EVA mode, the physical memory
is shifted to 0x80000000. Kernel is loaded into 0x80000000 (virtual)
so the offset between physical and virtual addresses is 0.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/mti-malta/malta-memory.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/arch/mips/mti-malta/malta-memory.c b/arch/mips/mti-malta/malta-memory.c
index 9235aee8..6d0f4ab 100644
--- a/arch/mips/mti-malta/malta-memory.c
+++ b/arch/mips/mti-malta/malta-memory.c
@@ -111,6 +111,12 @@ fw_memblock_t * __init fw_getmdesc(int eva)
return &mdesc[0];
}
+static void free_init_pages_eva_malta(void *begin, void *end)
+{
+ free_init_pages("unused kernel", __pa_symbol((unsigned long *)begin),
+ __pa_symbol((unsigned long *)end));
+}
+
static int __init fw_memtype_classify(unsigned int type)
{
switch (type) {
@@ -128,6 +134,8 @@ void __init fw_meminit(void)
fw_memblock_t *p;
p = fw_getmdesc(config_enabled(CONFIG_EVA));
+ free_init_pages_eva = (config_enabled(CONFIG_EVA) ?
+ free_init_pages_eva_malta : NULL);
while (p->size) {
long type;
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread* [PATCH 55/58] MIPS: malta: malta-memory: Add free_init_pages_eva() callback
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Use a Malta specific function to free the init section once the
kernel has booted. When operating in EVA mode, the physical memory
is shifted to 0x80000000. Kernel is loaded into 0x80000000 (virtual)
so the offset between physical and virtual addresses is 0.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/mti-malta/malta-memory.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/arch/mips/mti-malta/malta-memory.c b/arch/mips/mti-malta/malta-memory.c
index 9235aee8..6d0f4ab 100644
--- a/arch/mips/mti-malta/malta-memory.c
+++ b/arch/mips/mti-malta/malta-memory.c
@@ -111,6 +111,12 @@ fw_memblock_t * __init fw_getmdesc(int eva)
return &mdesc[0];
}
+static void free_init_pages_eva_malta(void *begin, void *end)
+{
+ free_init_pages("unused kernel", __pa_symbol((unsigned long *)begin),
+ __pa_symbol((unsigned long *)end));
+}
+
static int __init fw_memtype_classify(unsigned int type)
{
switch (type) {
@@ -128,6 +134,8 @@ void __init fw_meminit(void)
fw_memblock_t *p;
p = fw_getmdesc(config_enabled(CONFIG_EVA));
+ free_init_pages_eva = (config_enabled(CONFIG_EVA) ?
+ free_init_pages_eva_malta : NULL);
while (p->size) {
long type;
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 56/58] MIPS: malta: malta-init: Fix System Controller memory mapping for EVA
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Shift System Controller memory mapping to 0x80000000
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/mti-malta/malta-init.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/arch/mips/mti-malta/malta-init.c b/arch/mips/mti-malta/malta-init.c
index fcebfce..deb1d7b 100644
--- a/arch/mips/mti-malta/malta-init.c
+++ b/arch/mips/mti-malta/malta-init.c
@@ -238,10 +238,23 @@ mips_pci_controller:
MSC01_PCI_SWAP_BYTESWAP << MSC01_PCI_SWAP_MEM_SHF |
MSC01_PCI_SWAP_BYTESWAP << MSC01_PCI_SWAP_BAR0_SHF);
#endif
+#ifndef CONFIG_EVA
/* Fix up target memory mapping. */
MSC_READ(MSC01_PCI_BAR0, mask);
MSC_WRITE(MSC01_PCI_P2SCMSKL, mask & MSC01_PCI_BAR0_SIZE_MSK);
+#else
+ /*
+ * Setup the Malta max (2GB) memory for PCI DMA in host bridge
+ * in transparent addressing mode, starting from 0x80000000.
+ */
+ mask = PHYS_OFFSET | (1<<3);
+ MSC_WRITE(MSC01_PCI_BAR0, mask);
+ mask = PHYS_OFFSET;
+ MSC_WRITE(MSC01_PCI_HEAD4, mask);
+ MSC_WRITE(MSC01_PCI_P2SCMSKL, mask);
+ MSC_WRITE(MSC01_PCI_P2SCMAPL, mask);
+#endif
/* Don't handle target retries indefinitely. */
if ((data & MSC01_PCI_CFG_MAXRTRY_MSK) ==
MSC01_PCI_CFG_MAXRTRY_MSK)
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 56/58] MIPS: malta: malta-init: Fix System Controller memory mapping for EVA
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Leonid Yegoshin, Markos Chandras
From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Shift System Controller memory mapping to 0x80000000
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/mti-malta/malta-init.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/arch/mips/mti-malta/malta-init.c b/arch/mips/mti-malta/malta-init.c
index fcebfce..deb1d7b 100644
--- a/arch/mips/mti-malta/malta-init.c
+++ b/arch/mips/mti-malta/malta-init.c
@@ -238,10 +238,23 @@ mips_pci_controller:
MSC01_PCI_SWAP_BYTESWAP << MSC01_PCI_SWAP_MEM_SHF |
MSC01_PCI_SWAP_BYTESWAP << MSC01_PCI_SWAP_BAR0_SHF);
#endif
+#ifndef CONFIG_EVA
/* Fix up target memory mapping. */
MSC_READ(MSC01_PCI_BAR0, mask);
MSC_WRITE(MSC01_PCI_P2SCMSKL, mask & MSC01_PCI_BAR0_SIZE_MSK);
+#else
+ /*
+ * Setup the Malta max (2GB) memory for PCI DMA in host bridge
+ * in transparent addressing mode, starting from 0x80000000.
+ */
+ mask = PHYS_OFFSET | (1<<3);
+ MSC_WRITE(MSC01_PCI_BAR0, mask);
+ mask = PHYS_OFFSET;
+ MSC_WRITE(MSC01_PCI_HEAD4, mask);
+ MSC_WRITE(MSC01_PCI_P2SCMSKL, mask);
+ MSC_WRITE(MSC01_PCI_P2SCMAPL, mask);
+#endif
/* Don't handle target retries indefinitely. */
if ((data & MSC01_PCI_CFG_MAXRTRY_MSK) ==
MSC01_PCI_CFG_MAXRTRY_MSK)
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 57/58] MIPS: malta: Add support for SMP EVA
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Allow secondary cores to program their segment control registers
during smp bootstrap code. This enables EVA on Malta SMP
configurations
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/mach-malta/kernel-entry-init.h | 6 ++++++
arch/mips/kernel/head.S | 2 +-
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/arch/mips/include/asm/mach-malta/kernel-entry-init.h b/arch/mips/include/asm/mach-malta/kernel-entry-init.h
index 9bace9c..7c5e17a 100644
--- a/arch/mips/include/asm/mach-malta/kernel-entry-init.h
+++ b/arch/mips/include/asm/mach-malta/kernel-entry-init.h
@@ -154,6 +154,12 @@ nonsc_processor:
* Do SMP slave processor setup necessary before we can safely execute C code.
*/
.macro smp_slave_setup
+#ifdef CONFIG_EVA
+ sync
+ ehb
+ mfc0 t1, CP0_CONFIG
+ eva_entry
+#endif
.endm
#endif /* __ASM_MACH_MIPS_KERNEL_ENTRY_INIT_H */
diff --git a/arch/mips/kernel/head.S b/arch/mips/kernel/head.S
index 7b6a5b3..e712dcf 100644
--- a/arch/mips/kernel/head.S
+++ b/arch/mips/kernel/head.S
@@ -175,8 +175,8 @@ NESTED(smp_bootstrap, 16, sp)
DMT 10 # dmt t2 /* t0, t1 are used by CLI and setup_c0_status() */
jal mips_ihb
#endif /* CONFIG_MIPS_MT_SMTC */
- setup_c0_status_sec
smp_slave_setup
+ setup_c0_status_sec
#ifdef CONFIG_MIPS_MT_SMTC
andi t2, t2, VPECONTROL_TE
beqz t2, 2f
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 57/58] MIPS: malta: Add support for SMP EVA
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Allow secondary cores to program their segment control registers
during smp bootstrap code. This enables EVA on Malta SMP
configurations
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/include/asm/mach-malta/kernel-entry-init.h | 6 ++++++
arch/mips/kernel/head.S | 2 +-
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/arch/mips/include/asm/mach-malta/kernel-entry-init.h b/arch/mips/include/asm/mach-malta/kernel-entry-init.h
index 9bace9c..7c5e17a 100644
--- a/arch/mips/include/asm/mach-malta/kernel-entry-init.h
+++ b/arch/mips/include/asm/mach-malta/kernel-entry-init.h
@@ -154,6 +154,12 @@ nonsc_processor:
* Do SMP slave processor setup necessary before we can safely execute C code.
*/
.macro smp_slave_setup
+#ifdef CONFIG_EVA
+ sync
+ ehb
+ mfc0 t1, CP0_CONFIG
+ eva_entry
+#endif
.endm
#endif /* __ASM_MACH_MIPS_KERNEL_ENTRY_INIT_H */
diff --git a/arch/mips/kernel/head.S b/arch/mips/kernel/head.S
index 7b6a5b3..e712dcf 100644
--- a/arch/mips/kernel/head.S
+++ b/arch/mips/kernel/head.S
@@ -175,8 +175,8 @@ NESTED(smp_bootstrap, 16, sp)
DMT 10 # dmt t2 /* t0, t1 are used by CLI and setup_c0_status() */
jal mips_ihb
#endif /* CONFIG_MIPS_MT_SMTC */
- setup_c0_status_sec
smp_slave_setup
+ setup_c0_status_sec
#ifdef CONFIG_MIPS_MT_SMTC
andi t2, t2, VPECONTROL_TE
beqz t2, 2f
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 58/58] MIPS: Enable MIPS 3.5 features on Malta
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/Kconfig | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index 4230c7a..607792b 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -318,6 +318,7 @@ config MIPS_MALTA
select SWAP_IO_SPACE
select SYS_HAS_CPU_MIPS32_R1
select SYS_HAS_CPU_MIPS32_R2
+ select SYS_HAS_CPU_MIPS32_R3_5
select SYS_HAS_CPU_MIPS64_R1
select SYS_HAS_CPU_MIPS64_R2
select SYS_HAS_CPU_NEVADA
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* [PATCH 58/58] MIPS: Enable MIPS 3.5 features on Malta
@ 2014-01-27 20:19 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-01-27 20:19 UTC (permalink / raw)
To: linux-mips; +Cc: Markos Chandras
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
arch/mips/Kconfig | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index 4230c7a..607792b 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -318,6 +318,7 @@ config MIPS_MALTA
select SWAP_IO_SPACE
select SYS_HAS_CPU_MIPS32_R1
select SYS_HAS_CPU_MIPS32_R2
+ select SYS_HAS_CPU_MIPS32_R3_5
select SYS_HAS_CPU_MIPS64_R1
select SYS_HAS_CPU_MIPS64_R2
select SYS_HAS_CPU_NEVADA
--
1.8.5.3
^ permalink raw reply related [flat|nested] 126+ messages in thread
* Re: [PATCH 00/58] Add support for Enhanced Virtual Addressing
2014-01-27 20:18 ` Markos Chandras
` (58 preceding siblings ...)
(?)
@ 2014-01-29 18:37 ` David Daney
2014-02-04 9:00 ` Markos Chandras
-1 siblings, 1 reply; 126+ messages in thread
From: David Daney @ 2014-01-29 18:37 UTC (permalink / raw)
To: Markos Chandras; +Cc: linux-mips
This whole thing seems very messy. I see a couple of problems:
1) There are if(CONFIG_EVA) ... else ... endif all over the place. It
is very ugly.
2) You cannot have a signel kernel with both EVA and non-EVA support.
Have you considered just tagging all the instructions that touch the
user address space, and patching them at system boot with their EVA
equivalents if EVA support is desired?
You might even be able to use the existing exception tables and not have
to add a special table for EVA. The instructions that can trap are
exactly those where the EVA version should be used.
David Daney
On 01/27/2014 12:18 PM, Markos Chandras wrote:
> Hi,
>
> This patchset adds support for the Enhanced Virtual Addressing feature
> of the MIPS 3.5 ISA currently present in the *Aptiv processors.
> In Release 3, the memory architecture defines the partitioning of memory
> space into fixed‐size segments, whose access properties and cacheability
> attributes are programmable. The "mapped" and "unmapped" segment‐access
> properties can now be modified so as to extend system memory space
> (kseg0/kseg1) to include unmapped access to user space (xkseg), a
> feature called Enhanced Virtual Addressing (EVA). Instructions that
> facilitate system read/write access to xkseg have also been added to the
> R3 ISA (lwe, swe, lbue, etc).
>
> This patchset also adds EVA support for the Malta board. The unmapped
> address space (kseg0/kseg1) has been extended to 2GB.
>
> If you want to try it, you can use the 'eva-next' branch on
> the following git repository
>
> https://github.com/hwoarang/linux.git eva-next
>
> The patchset applies on top of the upstream-sfr/mips-for-linux-next tree.
>
> Leonid Yegoshin (18):
> MIPS: Kconfig: Add Kconfig symbols for EVA support
> MIPS: asm: Add prefetch instruction for EVA
> MIPS: uapi: inst: Add new EVA opcodes
> MIPS: uapi: inst: Add instruction format for SPECIAL3 instructions
> MIPS: traps: Set correct address limit for breakpoints and traps
> MIPS: asm: uaccess: Disable unaligned access macros for EVA
> MIPS: kernel: unaligned: Handle unaligned accesses for EVA
> MIPS: asm: checksum: Split kernel and user copy operations
> MIPS: asm: checksum: Add MIPS specific csum_and_copy_from_user
> function
> MIPS: kernel: signal: Prevent save/restore FPU context in user memory
> MIPS: asm: r4kcache: Build flushing code for instruction cache
> MIPS: asm: r4kcache: Add protected cache operation for EVA
> MIPS: asm: r4kcache: Add EVA cache flushing functions
> MIPS: kernel: {ftrace,kgdb}: Set correct address limit for cache
> flushes
> MIPS: asm: page: Allow __pa_symbol overrides
> MIPS: mm: c-r4k: Build EVA {d,i}cache flushing functions
> MIPS: mm: c-r4k: Flush scache to avoid cache aliases
> MIPS: malta: malta-init: Fix System Controller memory mapping for EVA
>
> Markos Chandras (40):
> MIPS: asm: Add wrappers for EVA/non-EVA instructions
> MIPS: futex: Add EVA support for futex operations
> MIPS: kernel: scall32-o32: Use EVA wrappers to fetch syscall arguments
> MIPS: kernel: traps: Whitespace clean up
> MIPS: lib: strnlen_user: Use macro to build the strnlen_user symbol
> MIPS: lib: strnlen_user: Add EVA support
> MIPS: lib: strlen_user: Use macro to build the strlen_user symbol
> MIPS: lib: strlen_user: Add EVA support
> MIPS: lib: strncpy_user: Use macro to build the strncpy_from_user
> symbol
> MIPS: lib: strncpy_user: Add EVA support
> MIPS: lib: memcpy: Merge EXC and load/store macros
> MIPS: lib: memcpy: Split source and destination prefetch macros
> MIPS: lib: memcpy: Use macro to build the copy_user code
> MIPS: lib: memcpy: Add EVA support
> MIPS: lib: memset: Whitespace fixes
> MIPS: lib: memset: Use macro to build the __bzero symbol
> MIPS: lib: memset: Add EVA support for the __bzero function.
> MIPS: asm: uaccess: Add instruction argument to __{put,get}_user_asm
> MIPS: asm: uaccess: Move duplicated code to common function
> MIPS: asm: uaccess: Use EVA instructions wrappers
> MIPS: asm: uaccess: Rename {get,put}_user_asm macros
> MIPS: asm: uaccess: Add EVA support to copy_{in, to,from}_user
> MIPS: asm: uaccess: Add EVA support for str*_user operations
> MIPS: kernel: unaligned: Add EVA instruction wrappers
> MIPS: checksum: Split the 'copy_user' symbol
> MIPS: lib: csum_partial: Merge EXC and load/store macros
> MIPS: lib: csum_partial: Add macro to build csum_partial symbols
> MIPS: lib: csum_partial: Add EVA support
> MIPS: asm: cpu: Add cpu flag for Enhanced Virtual Addressing
> MIPS: kernel: cpu-probe: Enable EVA option on supported cores
> MIPS: kernel: proc: Add EVA to the list of CPU features
> MIPS: mm: init: Add free_init_pages() callback for EVA
> MIPS: mm: c-r4k: Add support for flushing user pages from cache
> MIPS: malta: Configure Segment Control registers for EVA boot
> MIPS: malta: spaces.h: Add spaces.h file for Malta (EVA)
> MIPS: malta: malta-memory: Add support for the 'ememsize' variable
> MIPS: malta: malta-memory: Use the PHYS_OFFSET to build the memory map
> MIPS: malta: malta-memory: Add free_init_pages_eva() callback
> MIPS: malta: Add support for SMP EVA
> MIPS: Enable MIPS 3.5 features on Malta
>
> arch/mips/Kconfig | 29 +-
> arch/mips/include/asm/asm.h | 130 +++++
> arch/mips/include/asm/bootinfo.h | 2 +
> arch/mips/include/asm/checksum.h | 44 +-
> arch/mips/include/asm/cpu-features.h | 4 +-
> arch/mips/include/asm/cpu.h | 1 +
> arch/mips/include/asm/futex.h | 9 +-
> arch/mips/include/asm/fw/fw.h | 2 +-
> .../include/asm/mach-malta/kernel-entry-init.h | 115 ++++-
> arch/mips/include/asm/mach-malta/spaces.h | 46 ++
> arch/mips/include/asm/page.h | 2 +
> arch/mips/include/asm/r4kcache.h | 173 ++++++-
> arch/mips/include/asm/uaccess.h | 559 ++++++++++++++++-----
> arch/mips/include/uapi/asm/inst.h | 25 +-
> arch/mips/kernel/cpu-probe.c | 3 +
> arch/mips/kernel/ftrace.c | 4 +
> arch/mips/kernel/head.S | 2 +-
> arch/mips/kernel/kgdb.c | 17 +-
> arch/mips/kernel/mips_ksyms.c | 24 +-
> arch/mips/kernel/proc.c | 1 +
> arch/mips/kernel/scall32-o32.S | 9 +-
> arch/mips/kernel/signal.c | 26 +
> arch/mips/kernel/traps.c | 29 +-
> arch/mips/kernel/unaligned.c | 135 ++++-
> arch/mips/lib/csum_partial.S | 282 +++++++----
> arch/mips/lib/memcpy.S | 416 ++++++++++-----
> arch/mips/lib/memset.S | 146 ++++--
> arch/mips/lib/strlen_user.S | 36 +-
> arch/mips/lib/strncpy_user.S | 40 +-
> arch/mips/lib/strnlen_user.S | 36 +-
> arch/mips/mm/c-r4k.c | 64 ++-
> arch/mips/mm/init.c | 12 +-
> arch/mips/mti-malta/malta-init.c | 13 +
> arch/mips/mti-malta/malta-memory.c | 58 ++-
> arch/mips/mti-malta/malta-setup.c | 4 +
> 35 files changed, 1974 insertions(+), 524 deletions(-)
> create mode 100644 arch/mips/include/asm/mach-malta/spaces.h
>
^ permalink raw reply [flat|nested] 126+ messages in thread* Re: [PATCH 00/58] Add support for Enhanced Virtual Addressing
@ 2014-02-04 9:00 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-02-04 9:00 UTC (permalink / raw)
To: David Daney; +Cc: linux-mips
Hi David,
I have some comments
On 01/29/2014 06:37 PM, David Daney wrote:
> This whole thing seems very messy. I see a couple of problems:
>
> 1) There are if(CONFIG_EVA) ... else ... endif all over the place. It
> is very ugly.
>
> 2) You cannot have a signel kernel with both EVA and non-EVA support.
Unfortunately, I don't think using eva=1 or whatever in the kernel
command line would work. First of all, there is board specific init code
long before the command line is available. See patch #51. At that point,
the kernel command line is not 'exposed' yet so there is no way to tell
if you are booting in EVA or non-EVA mode.
>
> Have you considered just tagging all the instructions that touch the
> user address space, and patching them at system boot with their EVA
> equivalents if EVA support is desired?
I don't think this is possible (or maybe i don't understand your
proposal). Consider for example the copy_user function in memcpy.S. This
function does copy_{to,from,in} user/kernel depending on the supplied
virtual address. There is no way to tell in advance if a 'lb'
instruction will load from kernel or user space. This is only possible
during runtime by examining the get_fs() == get_ds() status. Therefore,
it's necessary to have 4 variants of this functions replacing only the
'store' or 'load' instructions that you really need to. For example
- load from user, store to kernel
- load from kernel, store to user
- load from user, store to user
- load from kernel, store to kernel
As you can see, static replacement of the instructions with the EVA ones
during boot will not work as expected.
This is the reason I converted these functions to macros, so all the
variants will be a simple 3 line assembler code that only replaces the
instructions you are really interested in.
--
markos
^ permalink raw reply [flat|nested] 126+ messages in thread* Re: [PATCH 00/58] Add support for Enhanced Virtual Addressing
@ 2014-02-04 9:00 ` Markos Chandras
0 siblings, 0 replies; 126+ messages in thread
From: Markos Chandras @ 2014-02-04 9:00 UTC (permalink / raw)
To: David Daney; +Cc: linux-mips
Hi David,
I have some comments
On 01/29/2014 06:37 PM, David Daney wrote:
> This whole thing seems very messy. I see a couple of problems:
>
> 1) There are if(CONFIG_EVA) ... else ... endif all over the place. It
> is very ugly.
>
> 2) You cannot have a signel kernel with both EVA and non-EVA support.
Unfortunately, I don't think using eva=1 or whatever in the kernel
command line would work. First of all, there is board specific init code
long before the command line is available. See patch #51. At that point,
the kernel command line is not 'exposed' yet so there is no way to tell
if you are booting in EVA or non-EVA mode.
>
> Have you considered just tagging all the instructions that touch the
> user address space, and patching them at system boot with their EVA
> equivalents if EVA support is desired?
I don't think this is possible (or maybe i don't understand your
proposal). Consider for example the copy_user function in memcpy.S. This
function does copy_{to,from,in} user/kernel depending on the supplied
virtual address. There is no way to tell in advance if a 'lb'
instruction will load from kernel or user space. This is only possible
during runtime by examining the get_fs() == get_ds() status. Therefore,
it's necessary to have 4 variants of this functions replacing only the
'store' or 'load' instructions that you really need to. For example
- load from user, store to kernel
- load from kernel, store to user
- load from user, store to user
- load from kernel, store to kernel
As you can see, static replacement of the instructions with the EVA ones
during boot will not work as expected.
This is the reason I converted these functions to macros, so all the
variants will be a simple 3 line assembler code that only replaces the
instructions you are really interested in.
--
markos
^ permalink raw reply [flat|nested] 126+ messages in thread