[PATCH 00/58] Add support for Enhanced Virtual Addressing

[PATCH 00/58] Add support for Enhanced Virtual Addressing

[Markos Chandras]

Hi,

This patchset adds support for the Enhanced Virtual Addressing feature
of the MIPS 3.5 ISA currently present in the *Aptiv processors.
In Release 3, the memory architecture defines the partitioning of memory
space into fixed‐size segments, whose access properties and cacheability
attributes are programmable. The "mapped" and "unmapped" segment‐access
properties can now be modified so as to extend system memory space
(kseg0/kseg1) to include unmapped access to user space (xkseg), a
feature called Enhanced Virtual Addressing (EVA). Instructions that
facilitate system read/write access to xkseg have also been added to the
R3 ISA (lwe, swe, lbue, etc).

This patchset also adds EVA support for the Malta board. The unmapped
address space (kseg0/kseg1) has been extended to 2GB.

If you want to try it, you can use the 'eva-next' branch on
the following git repository

https://github.com/hwoarang/linux.git eva-next

The patchset applies on top of the upstream-sfr/mips-for-linux-next tree.

Leonid Yegoshin (18):
  MIPS: Kconfig: Add Kconfig symbols for EVA support
  MIPS: asm: Add prefetch instruction for EVA
  MIPS: uapi: inst: Add new EVA opcodes
  MIPS: uapi: inst: Add instruction format for SPECIAL3 instructions
  MIPS: traps: Set correct address limit for breakpoints and traps
  MIPS: asm: uaccess: Disable unaligned access macros for EVA
  MIPS: kernel: unaligned: Handle unaligned accesses for EVA
  MIPS: asm: checksum: Split kernel and user copy operations
  MIPS: asm: checksum: Add MIPS specific csum_and_copy_from_user
    function
  MIPS: kernel: signal: Prevent save/restore FPU context in user memory
  MIPS: asm: r4kcache: Build flushing code for instruction cache
  MIPS: asm: r4kcache: Add protected cache operation for EVA
  MIPS: asm: r4kcache: Add EVA cache flushing functions
  MIPS: kernel: {ftrace,kgdb}: Set correct address limit for cache
    flushes
  MIPS: asm: page: Allow __pa_symbol overrides
  MIPS: mm: c-r4k: Build EVA {d,i}cache flushing functions
  MIPS: mm: c-r4k: Flush scache to avoid cache aliases
  MIPS: malta: malta-init: Fix System Controller memory mapping for EVA

Markos Chandras (40):
  MIPS: asm: Add wrappers for EVA/non-EVA instructions
  MIPS: futex: Add EVA support for futex operations
  MIPS: kernel: scall32-o32: Use EVA wrappers to fetch syscall arguments
  MIPS: kernel: traps: Whitespace clean up
  MIPS: lib: strnlen_user: Use macro to build the strnlen_user symbol
  MIPS: lib: strnlen_user: Add EVA support
  MIPS: lib: strlen_user: Use macro to build the strlen_user symbol
  MIPS: lib: strlen_user: Add EVA support
  MIPS: lib: strncpy_user: Use macro to build the strncpy_from_user
    symbol
  MIPS: lib: strncpy_user: Add EVA support
  MIPS: lib: memcpy: Merge EXC and load/store macros
  MIPS: lib: memcpy: Split source and destination prefetch macros
  MIPS: lib: memcpy: Use macro to build the copy_user code
  MIPS: lib: memcpy: Add EVA support
  MIPS: lib: memset: Whitespace fixes
  MIPS: lib: memset: Use macro to build the __bzero symbol
  MIPS: lib: memset: Add EVA support for the __bzero function.
  MIPS: asm: uaccess: Add instruction argument to __{put,get}_user_asm
  MIPS: asm: uaccess: Move duplicated code to common function
  MIPS: asm: uaccess: Use EVA instructions wrappers
  MIPS: asm: uaccess: Rename {get,put}_user_asm macros
  MIPS: asm: uaccess: Add EVA support to copy_{in, to,from}_user
  MIPS: asm: uaccess: Add EVA support for str*_user operations
  MIPS: kernel: unaligned: Add EVA instruction wrappers
  MIPS: checksum: Split the 'copy_user' symbol
  MIPS: lib: csum_partial: Merge EXC and load/store macros
  MIPS: lib: csum_partial: Add macro to build csum_partial symbols
  MIPS: lib: csum_partial: Add EVA support
  MIPS: asm: cpu: Add cpu flag for Enhanced Virtual Addressing
  MIPS: kernel: cpu-probe: Enable EVA option on supported cores
  MIPS: kernel: proc: Add EVA to the list of CPU features
  MIPS: mm: init: Add free_init_pages() callback for EVA
  MIPS: mm: c-r4k: Add support for flushing user pages from cache
  MIPS: malta: Configure Segment Control registers for EVA boot
  MIPS: malta: spaces.h: Add spaces.h file for Malta (EVA)
  MIPS: malta: malta-memory: Add support for the 'ememsize' variable
  MIPS: malta: malta-memory: Use the PHYS_OFFSET to build the memory map
  MIPS: malta: malta-memory: Add free_init_pages_eva() callback
  MIPS: malta: Add support for SMP EVA
  MIPS: Enable MIPS 3.5 features on Malta

 arch/mips/Kconfig                                  |  29 +-
 arch/mips/include/asm/asm.h                        | 130 +++++
 arch/mips/include/asm/bootinfo.h                   |   2 +
 arch/mips/include/asm/checksum.h                   |  44 +-
 arch/mips/include/asm/cpu-features.h               |   4 +-
 arch/mips/include/asm/cpu.h                        |   1 +
 arch/mips/include/asm/futex.h                      |   9 +-
 arch/mips/include/asm/fw/fw.h                      |   2 +-
 .../include/asm/mach-malta/kernel-entry-init.h     | 115 ++++-
 arch/mips/include/asm/mach-malta/spaces.h          |  46 ++
 arch/mips/include/asm/page.h                       |   2 +
 arch/mips/include/asm/r4kcache.h                   | 173 ++++++-
 arch/mips/include/asm/uaccess.h                    | 559 ++++++++++++++++-----
 arch/mips/include/uapi/asm/inst.h                  |  25 +-
 arch/mips/kernel/cpu-probe.c                       |   3 +
 arch/mips/kernel/ftrace.c                          |   4 +
 arch/mips/kernel/head.S                            |   2 +-
 arch/mips/kernel/kgdb.c                            |  17 +-
 arch/mips/kernel/mips_ksyms.c                      |  24 +-
 arch/mips/kernel/proc.c                            |   1 +
 arch/mips/kernel/scall32-o32.S                     |   9 +-
 arch/mips/kernel/signal.c                          |  26 +
 arch/mips/kernel/traps.c                           |  29 +-
 arch/mips/kernel/unaligned.c                       | 135 ++++-
 arch/mips/lib/csum_partial.S                       | 282 +++++++----
 arch/mips/lib/memcpy.S                             | 416 ++++++++++-----
 arch/mips/lib/memset.S                             | 146 ++++--
 arch/mips/lib/strlen_user.S                        |  36 +-
 arch/mips/lib/strncpy_user.S                       |  40 +-
 arch/mips/lib/strnlen_user.S                       |  36 +-
 arch/mips/mm/c-r4k.c                               |  64 ++-
 arch/mips/mm/init.c                                |  12 +-
 arch/mips/mti-malta/malta-init.c                   |  13 +
 arch/mips/mti-malta/malta-memory.c                 |  58 ++-
 arch/mips/mti-malta/malta-setup.c                  |   4 +
 35 files changed, 1974 insertions(+), 524 deletions(-)
 create mode 100644 arch/mips/include/asm/mach-malta/spaces.h

-- 
1.8.5.3

[PATCH 00/58] Add support for Enhanced Virtual Addressing

[Markos Chandras]

Hi,

This patchset adds support for the Enhanced Virtual Addressing feature
of the MIPS 3.5 ISA currently present in the *Aptiv processors.
In Release 3, the memory architecture defines the partitioning of memory
space into fixed‐size segments, whose access properties and cacheability
attributes are programmable. The "mapped" and "unmapped" segment‐access
properties can now be modified so as to extend system memory space
(kseg0/kseg1) to include unmapped access to user space (xkseg), a
feature called Enhanced Virtual Addressing (EVA). Instructions that
facilitate system read/write access to xkseg have also been added to the
R3 ISA (lwe, swe, lbue, etc).

This patchset also adds EVA support for the Malta board. The unmapped
address space (kseg0/kseg1) has been extended to 2GB.

If you want to try it, you can use the 'eva-next' branch on
the following git repository

https://github.com/hwoarang/linux.git eva-next

The patchset applies on top of the upstream-sfr/mips-for-linux-next tree.

Leonid Yegoshin (18):
  MIPS: Kconfig: Add Kconfig symbols for EVA support
  MIPS: asm: Add prefetch instruction for EVA
  MIPS: uapi: inst: Add new EVA opcodes
  MIPS: uapi: inst: Add instruction format for SPECIAL3 instructions
  MIPS: traps: Set correct address limit for breakpoints and traps
  MIPS: asm: uaccess: Disable unaligned access macros for EVA
  MIPS: kernel: unaligned: Handle unaligned accesses for EVA
  MIPS: asm: checksum: Split kernel and user copy operations
  MIPS: asm: checksum: Add MIPS specific csum_and_copy_from_user
    function
  MIPS: kernel: signal: Prevent save/restore FPU context in user memory
  MIPS: asm: r4kcache: Build flushing code for instruction cache
  MIPS: asm: r4kcache: Add protected cache operation for EVA
  MIPS: asm: r4kcache: Add EVA cache flushing functions
  MIPS: kernel: {ftrace,kgdb}: Set correct address limit for cache
    flushes
  MIPS: asm: page: Allow __pa_symbol overrides
  MIPS: mm: c-r4k: Build EVA {d,i}cache flushing functions
  MIPS: mm: c-r4k: Flush scache to avoid cache aliases
  MIPS: malta: malta-init: Fix System Controller memory mapping for EVA

Markos Chandras (40):
  MIPS: asm: Add wrappers for EVA/non-EVA instructions
  MIPS: futex: Add EVA support for futex operations
  MIPS: kernel: scall32-o32: Use EVA wrappers to fetch syscall arguments
  MIPS: kernel: traps: Whitespace clean up
  MIPS: lib: strnlen_user: Use macro to build the strnlen_user symbol
  MIPS: lib: strnlen_user: Add EVA support
  MIPS: lib: strlen_user: Use macro to build the strlen_user symbol
  MIPS: lib: strlen_user: Add EVA support
  MIPS: lib: strncpy_user: Use macro to build the strncpy_from_user
    symbol
  MIPS: lib: strncpy_user: Add EVA support
  MIPS: lib: memcpy: Merge EXC and load/store macros
  MIPS: lib: memcpy: Split source and destination prefetch macros
  MIPS: lib: memcpy: Use macro to build the copy_user code
  MIPS: lib: memcpy: Add EVA support
  MIPS: lib: memset: Whitespace fixes
  MIPS: lib: memset: Use macro to build the __bzero symbol
  MIPS: lib: memset: Add EVA support for the __bzero function.
  MIPS: asm: uaccess: Add instruction argument to __{put,get}_user_asm
  MIPS: asm: uaccess: Move duplicated code to common function
  MIPS: asm: uaccess: Use EVA instructions wrappers
  MIPS: asm: uaccess: Rename {get,put}_user_asm macros
  MIPS: asm: uaccess: Add EVA support to copy_{in, to,from}_user
  MIPS: asm: uaccess: Add EVA support for str*_user operations
  MIPS: kernel: unaligned: Add EVA instruction wrappers
  MIPS: checksum: Split the 'copy_user' symbol
  MIPS: lib: csum_partial: Merge EXC and load/store macros
  MIPS: lib: csum_partial: Add macro to build csum_partial symbols
  MIPS: lib: csum_partial: Add EVA support
  MIPS: asm: cpu: Add cpu flag for Enhanced Virtual Addressing
  MIPS: kernel: cpu-probe: Enable EVA option on supported cores
  MIPS: kernel: proc: Add EVA to the list of CPU features
  MIPS: mm: init: Add free_init_pages() callback for EVA
  MIPS: mm: c-r4k: Add support for flushing user pages from cache
  MIPS: malta: Configure Segment Control registers for EVA boot
  MIPS: malta: spaces.h: Add spaces.h file for Malta (EVA)
  MIPS: malta: malta-memory: Add support for the 'ememsize' variable
  MIPS: malta: malta-memory: Use the PHYS_OFFSET to build the memory map
  MIPS: malta: malta-memory: Add free_init_pages_eva() callback
  MIPS: malta: Add support for SMP EVA
  MIPS: Enable MIPS 3.5 features on Malta

 arch/mips/Kconfig                                  |  29 +-
 arch/mips/include/asm/asm.h                        | 130 +++++
 arch/mips/include/asm/bootinfo.h                   |   2 +
 arch/mips/include/asm/checksum.h                   |  44 +-
 arch/mips/include/asm/cpu-features.h               |   4 +-
 arch/mips/include/asm/cpu.h                        |   1 +
 arch/mips/include/asm/futex.h                      |   9 +-
 arch/mips/include/asm/fw/fw.h                      |   2 +-
 .../include/asm/mach-malta/kernel-entry-init.h     | 115 ++++-
 arch/mips/include/asm/mach-malta/spaces.h          |  46 ++
 arch/mips/include/asm/page.h                       |   2 +
 arch/mips/include/asm/r4kcache.h                   | 173 ++++++-
 arch/mips/include/asm/uaccess.h                    | 559 ++++++++++++++++-----
 arch/mips/include/uapi/asm/inst.h                  |  25 +-
 arch/mips/kernel/cpu-probe.c                       |   3 +
 arch/mips/kernel/ftrace.c                          |   4 +
 arch/mips/kernel/head.S                            |   2 +-
 arch/mips/kernel/kgdb.c                            |  17 +-
 arch/mips/kernel/mips_ksyms.c                      |  24 +-
 arch/mips/kernel/proc.c                            |   1 +
 arch/mips/kernel/scall32-o32.S                     |   9 +-
 arch/mips/kernel/signal.c                          |  26 +
 arch/mips/kernel/traps.c                           |  29 +-
 arch/mips/kernel/unaligned.c                       | 135 ++++-
 arch/mips/lib/csum_partial.S                       | 282 +++++++----
 arch/mips/lib/memcpy.S                             | 416 ++++++++++-----
 arch/mips/lib/memset.S                             | 146 ++++--
 arch/mips/lib/strlen_user.S                        |  36 +-
 arch/mips/lib/strncpy_user.S                       |  40 +-
 arch/mips/lib/strnlen_user.S                       |  36 +-
 arch/mips/mm/c-r4k.c                               |  64 ++-
 arch/mips/mm/init.c                                |  12 +-
 arch/mips/mti-malta/malta-init.c                   |  13 +
 arch/mips/mti-malta/malta-memory.c                 |  58 ++-
 arch/mips/mti-malta/malta-setup.c                  |   4 +
 35 files changed, 1974 insertions(+), 524 deletions(-)
 create mode 100644 arch/mips/include/asm/mach-malta/spaces.h

-- 
1.8.5.3

[PATCH 01/58] MIPS: Kconfig: Add Kconfig symbols for EVA support

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

Add basic Kconfig support for EVA. Not selectable by any platform
at this point.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/Kconfig | 28 +++++++++++++++++++++++++++-
 1 file changed, 27 insertions(+), 1 deletion(-)

diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index f4c78c9..4230c7a 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -1446,6 +1446,26 @@ config CPU_XLP
 	  Netlogic Microsystems XLP processors.
 endchoice
 
+config CPU_MIPS32_3_5_FEATURES
+	bool "MIPS32 Release 3.5 Features"
+	depends on SYS_HAS_CPU_MIPS32_R3_5
+	depends on CPU_MIPS32_R2
+	help
+	  Choose this option to build a kernel for release 2 or later of the
+	  MIPS32 architecture including features from the 3.5 release such as
+	  support for Enhanced Virtual Addressing (EVA).
+
+config CPU_MIPS32_3_5_EVA
+	bool "Enhanced Virtual Addressing (EVA)"
+	depends on CPU_MIPS32_3_5_FEATURES
+	select EVA
+	default y
+	help
+	  Choose this option if you want to enable the Enhanced Virtual
+	  Addressing (EVA) on your MIPS32 core (such as proAptiv).
+	  One of its primary benefits is an increase in the maximum size
+	  of lowmem (up to 3GB). If unsure, say 'N' here.
+
 if CPU_LOONGSON2F
 config CPU_NOP_WORKAROUNDS
 	bool
@@ -1539,6 +1559,9 @@ config SYS_HAS_CPU_MIPS32_R1
 config SYS_HAS_CPU_MIPS32_R2
 	bool
 
+config SYS_HAS_CPU_MIPS32_R3_5
+	bool
+
 config SYS_HAS_CPU_MIPS64_R1
 	bool
 
@@ -1658,6 +1681,9 @@ config CPU_MIPSR2
 	bool
 	default y if CPU_MIPS32_R2 || CPU_MIPS64_R2 || CPU_CAVIUM_OCTEON
 
+config EVA
+	bool
+
 config SYS_SUPPORTS_32BIT_KERNEL
 	bool
 config SYS_SUPPORTS_64BIT_KERNEL
@@ -2095,7 +2121,7 @@ config CPU_R4400_WORKAROUNDS
 #
 config HIGHMEM
 	bool "High Memory Support"
-	depends on 32BIT && CPU_SUPPORTS_HIGHMEM && SYS_SUPPORTS_HIGHMEM
+	depends on 32BIT && CPU_SUPPORTS_HIGHMEM && SYS_SUPPORTS_HIGHMEM && !CPU_MIPS32_3_5_EVA
 
 config CPU_SUPPORTS_HIGHMEM
 	bool
-- 
1.8.5.3

[PATCH 01/58] MIPS: Kconfig: Add Kconfig symbols for EVA support

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

Add basic Kconfig support for EVA. Not selectable by any platform
at this point.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/Kconfig | 28 +++++++++++++++++++++++++++-
 1 file changed, 27 insertions(+), 1 deletion(-)

diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index f4c78c9..4230c7a 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -1446,6 +1446,26 @@ config CPU_XLP
 	  Netlogic Microsystems XLP processors.
 endchoice
 
+config CPU_MIPS32_3_5_FEATURES
+	bool "MIPS32 Release 3.5 Features"
+	depends on SYS_HAS_CPU_MIPS32_R3_5
+	depends on CPU_MIPS32_R2
+	help
+	  Choose this option to build a kernel for release 2 or later of the
+	  MIPS32 architecture including features from the 3.5 release such as
+	  support for Enhanced Virtual Addressing (EVA).
+
+config CPU_MIPS32_3_5_EVA
+	bool "Enhanced Virtual Addressing (EVA)"
+	depends on CPU_MIPS32_3_5_FEATURES
+	select EVA
+	default y
+	help
+	  Choose this option if you want to enable the Enhanced Virtual
+	  Addressing (EVA) on your MIPS32 core (such as proAptiv).
+	  One of its primary benefits is an increase in the maximum size
+	  of lowmem (up to 3GB). If unsure, say 'N' here.
+
 if CPU_LOONGSON2F
 config CPU_NOP_WORKAROUNDS
 	bool
@@ -1539,6 +1559,9 @@ config SYS_HAS_CPU_MIPS32_R1
 config SYS_HAS_CPU_MIPS32_R2
 	bool
 
+config SYS_HAS_CPU_MIPS32_R3_5
+	bool
+
 config SYS_HAS_CPU_MIPS64_R1
 	bool
 
@@ -1658,6 +1681,9 @@ config CPU_MIPSR2
 	bool
 	default y if CPU_MIPS32_R2 || CPU_MIPS64_R2 || CPU_CAVIUM_OCTEON
 
+config EVA
+	bool
+
 config SYS_SUPPORTS_32BIT_KERNEL
 	bool
 config SYS_SUPPORTS_64BIT_KERNEL
@@ -2095,7 +2121,7 @@ config CPU_R4400_WORKAROUNDS
 #
 config HIGHMEM
 	bool "High Memory Support"
-	depends on 32BIT && CPU_SUPPORTS_HIGHMEM && SYS_SUPPORTS_HIGHMEM
+	depends on 32BIT && CPU_SUPPORTS_HIGHMEM && SYS_SUPPORTS_HIGHMEM && !CPU_MIPS32_3_5_EVA
 
 config CPU_SUPPORTS_HIGHMEM
 	bool
-- 
1.8.5.3

[PATCH 02/58] MIPS: asm: Add prefetch instruction for EVA

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

EVA can use the PREFE instruction to perform the virtual address
translation using the user mapping of the address rather than the
kernel mapping.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/asm.h | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/arch/mips/include/asm/asm.h b/arch/mips/include/asm/asm.h
index 879691d..b79be18a 100644
--- a/arch/mips/include/asm/asm.h
+++ b/arch/mips/include/asm/asm.h
@@ -149,6 +149,13 @@ symbol		=	value
 		pref	hint, addr;			\
 		.set	pop
 
+#define PREFE(hint, addr)				\
+		.set	push;				\
+		.set	mips0;				\
+		.set	eva;				\
+		prefe	hint, addr;			\
+		.set	pop
+
 #define PREFX(hint,addr)				\
 		.set	push;				\
 		.set	mips4;				\
@@ -158,6 +165,7 @@ symbol		=	value
 #else /* !CONFIG_CPU_HAS_PREFETCH */
 
 #define PREF(hint, addr)
+#define PREFE(hint, addr)
 #define PREFX(hint, addr)
 
 #endif /* !CONFIG_CPU_HAS_PREFETCH */
-- 
1.8.5.3

[PATCH 02/58] MIPS: asm: Add prefetch instruction for EVA

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

EVA can use the PREFE instruction to perform the virtual address
translation using the user mapping of the address rather than the
kernel mapping.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/asm.h | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/arch/mips/include/asm/asm.h b/arch/mips/include/asm/asm.h
index 879691d..b79be18a 100644
--- a/arch/mips/include/asm/asm.h
+++ b/arch/mips/include/asm/asm.h
@@ -149,6 +149,13 @@ symbol		=	value
 		pref	hint, addr;			\
 		.set	pop
 
+#define PREFE(hint, addr)				\
+		.set	push;				\
+		.set	mips0;				\
+		.set	eva;				\
+		prefe	hint, addr;			\
+		.set	pop
+
 #define PREFX(hint,addr)				\
 		.set	push;				\
 		.set	mips4;				\
@@ -158,6 +165,7 @@ symbol		=	value
 #else /* !CONFIG_CPU_HAS_PREFETCH */
 
 #define PREF(hint, addr)
+#define PREFE(hint, addr)
 #define PREFX(hint, addr)
 
 #endif /* !CONFIG_CPU_HAS_PREFETCH */
-- 
1.8.5.3

[PATCH 03/58] MIPS: asm: Add wrappers for EVA/non-EVA instructions

[Markos Chandras]

EVA uses specific instructions for accessing user memory.
Instead of polluting the kernel with numerous #ifdef CONFIG_EVA
we add wrappers for all the instructions that need special
handling when EVA is enabled.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/asm.h | 122 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 122 insertions(+)

diff --git a/arch/mips/include/asm/asm.h b/arch/mips/include/asm/asm.h
index b79be18a..5b11598 100644
--- a/arch/mips/include/asm/asm.h
+++ b/arch/mips/include/asm/asm.h
@@ -28,6 +28,128 @@
 #define CAT(str1, str2) __CAT(str1, str2)
 #endif
 
+#ifndef __ASSEMBLY__
+#ifdef CONFIG_EVA
+
+#define __BUILD_EVA_INSN(insn, reg, addr)				\
+				"	.set	push\n"			\
+				"	.set	mips0\n"		\
+				"	.set	eva\n"			\
+				"	"insn" "reg", "addr "\n"	\
+				"	.set	pop\n"
+
+#define user_cache(op, base)		__BUILD_EVA_INSN("cachee", op, base)
+#define user_ll(reg, addr)		__BUILD_EVA_INSN("lle", reg, addr)
+#define user_sc(reg, addr)		__BUILD_EVA_INSN("sce", reg, addr)
+#define user_lw(reg, addr)		__BUILD_EVA_INSN("lwe", reg, addr)
+#define user_lwl(reg, addr)		__BUILD_EVA_INSN("lwle", reg, addr)
+#define user_lwr(reg, addr)		__BUILD_EVA_INSN("lwre", reg, addr)
+#define user_lh(reg, addr)		__BUILD_EVA_INSN("lhe", reg, addr)
+#define user_lb(reg, addr)		__BUILD_EVA_INSN("lbe", reg, addr)
+#define user_lbu(reg, addr)		__BUILD_EVA_INSN("lbue", reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_ld(reg, addr)		user_lw(reg, addr)
+#define user_sw(reg, addr)		__BUILD_EVA_INSN("swe", reg, addr)
+#define user_swl(reg, addr)		__BUILD_EVA_INSN("swle", reg, addr)
+#define user_swr(reg, addr)		__BUILD_EVA_INSN("swre", reg, addr)
+#define user_sh(reg, addr)		__BUILD_EVA_INSN("she", reg, addr)
+#define user_sb(reg, addr)		__BUILD_EVA_INSN("sbe", reg, addr)
+/* No 64-bit EVA instruction for storing double words */
+#define user_sd(reg, addr)		user_sw(reg, addr)
+
+#else
+
+#define user_cache(op, base)		"cache " op ", " base "\n"
+#define user_ll(reg, addr)		"ll " reg ", " addr "\n"
+#define user_sc(reg, addr)		"sc " reg ", " addr "\n"
+#define user_lw(reg, addr)		"lw " reg ", " addr "\n"
+#define user_lwl(reg, addr)		"lwl " reg ", " addr "\n"
+#define user_lwr(reg, addr)		"lwr " reg ", " addr "\n"
+#define user_lh(reg, addr)		"lh " reg ", " addr "\n"
+#define user_lb(reg, addr)		"lb " reg ", " addr "\n"
+#define user_lbu(reg, addr)		"lbu " reg ", " addr "\n"
+#define user_sw(reg, addr)		"sw " reg ", " addr "\n"
+#define user_swl(reg, addr)		"swl " reg ", " addr "\n"
+#define user_swr(reg, addr)		"swr " reg ", " addr "\n"
+#define user_sh(reg, addr)		"sh " reg ", " addr "\n"
+#define user_sb(reg, addr)		"sb " reg ", " addr "\n"
+
+#ifdef CONFIG_32BIT
+/*
+ * No 'sd' or 'ld' instructions in 32-bit but the code will
+ * do the correct thing
+ */
+#define user_sd(reg, addr)		user_sw(reg, addr)
+#define user_ld(reg, addr)		user_lw(reg, addr)
+#else
+#define user_sd(reg, addr)		"sd " reg", " addr "\n"
+#define user_ld(reg, addr)		"ld " reg", " addr "\n"
+#endif /* CONFIG_32BIT */
+
+#endif /* CONFIG_EVA */
+
+#else /* __ASSEMBLY__ */
+
+#ifdef CONFIG_EVA
+
+#define __BUILD_EVA_INSN(insn, reg, addr)			\
+				.set	push;			\
+				.set	mips0;			\
+				.set	eva;			\
+				insn reg, addr;			\
+				.set	pop;
+
+#define user_cache(op, base)		__BUILD_EVA_INSN(cachee, op, base)
+#define user_ll(reg, addr)		__BUILD_EVA_INSN(lle, reg, addr)
+#define user_sc(reg, addr)		__BUILD_EVA_INSN(sce, reg, addr)
+#define user_lw(reg, addr)		__BUILD_EVA_INSN(lwe, reg, addr)
+#define user_lwl(reg, addr)		__BUILD_EVA_INSN(lwle, reg, addr)
+#define user_lwr(reg, addr)		__BUILD_EVA_INSN(lwre, reg, addr)
+#define user_lh(reg, addr)		__BUILD_EVA_INSN(lhe, reg, addr)
+#define user_lb(reg, addr)		__BUILD_EVA_INSN(lbe, reg, addr)
+#define user_lbu(reg, addr)		__BUILD_EVA_INSN(lbue, reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_ld(reg, addr)		user_lw(reg, addr)
+#define user_sw(reg, addr)		__BUILD_EVA_INSN(swe, reg, addr)
+#define user_swl(reg, addr)		__BUILD_EVA_INSN(swle, reg, addr)
+#define user_swr(reg, addr)		__BUILD_EVA_INSN(swre, reg, addr)
+#define user_sh(reg, addr)		__BUILD_EVA_INSN(she, reg, addr)
+#define user_sb(reg, addr)		__BUILD_EVA_INSN(sbe, reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_sd(reg, addr)		user_sw(reg, addr)
+#else
+
+#define user_cache(op, base)		cache op, base
+#define user_ll(reg, addr)		ll reg, addr
+#define user_sc(reg, addr)		sc reg, addr
+#define user_lw(reg, addr)		lw reg, addr
+#define user_lwl(reg, addr)		lwl reg, addr
+#define user_lwr(reg, addr)		lwr reg, addr
+#define user_lh(reg, addr)		lh reg, addr
+#define user_lb(reg, addr)		lb reg, addr
+#define user_lbu(reg, addr)		lbu reg, addr
+#define user_sw(reg, addr)		sw reg, addr
+#define user_swl(reg, addr)		swl reg, addr
+#define user_swr(reg, addr)		swr reg, addr
+#define user_sh(reg, addr)		sh reg, addr
+#define user_sb(reg, addr)		sb reg, addr
+
+#ifdef CONFIG_32BIT
+/*
+ * No 'sd' or 'ld' instructions in 32-bit but the code will
+ * do the correct thing
+ */
+#define user_sd(reg, addr)		user_sw(reg, addr)
+#define user_ld(reg, addr)		user_lw(reg, addr)
+#else
+#define user_sd(reg, addr)		sd reg, addr
+#define user_ld(reg, addr)		ld reg, addr
+#endif /* CONFIG_32BIT */
+
+#endif /* CONFIG_EVA */
+
+#endif /* __ASSEMBLY__ */
+
 /*
  * PIC specific declarations
  * Not used for the kernel but here seems to be the right place.
-- 
1.8.5.3

[PATCH 03/58] MIPS: asm: Add wrappers for EVA/non-EVA instructions

[Markos Chandras]

EVA uses specific instructions for accessing user memory.
Instead of polluting the kernel with numerous #ifdef CONFIG_EVA
we add wrappers for all the instructions that need special
handling when EVA is enabled.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/asm.h | 122 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 122 insertions(+)

diff --git a/arch/mips/include/asm/asm.h b/arch/mips/include/asm/asm.h
index b79be18a..5b11598 100644
--- a/arch/mips/include/asm/asm.h
+++ b/arch/mips/include/asm/asm.h
@@ -28,6 +28,128 @@
 #define CAT(str1, str2) __CAT(str1, str2)
 #endif
 
+#ifndef __ASSEMBLY__
+#ifdef CONFIG_EVA
+
+#define __BUILD_EVA_INSN(insn, reg, addr)				\
+				"	.set	push\n"			\
+				"	.set	mips0\n"		\
+				"	.set	eva\n"			\
+				"	"insn" "reg", "addr "\n"	\
+				"	.set	pop\n"
+
+#define user_cache(op, base)		__BUILD_EVA_INSN("cachee", op, base)
+#define user_ll(reg, addr)		__BUILD_EVA_INSN("lle", reg, addr)
+#define user_sc(reg, addr)		__BUILD_EVA_INSN("sce", reg, addr)
+#define user_lw(reg, addr)		__BUILD_EVA_INSN("lwe", reg, addr)
+#define user_lwl(reg, addr)		__BUILD_EVA_INSN("lwle", reg, addr)
+#define user_lwr(reg, addr)		__BUILD_EVA_INSN("lwre", reg, addr)
+#define user_lh(reg, addr)		__BUILD_EVA_INSN("lhe", reg, addr)
+#define user_lb(reg, addr)		__BUILD_EVA_INSN("lbe", reg, addr)
+#define user_lbu(reg, addr)		__BUILD_EVA_INSN("lbue", reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_ld(reg, addr)		user_lw(reg, addr)
+#define user_sw(reg, addr)		__BUILD_EVA_INSN("swe", reg, addr)
+#define user_swl(reg, addr)		__BUILD_EVA_INSN("swle", reg, addr)
+#define user_swr(reg, addr)		__BUILD_EVA_INSN("swre", reg, addr)
+#define user_sh(reg, addr)		__BUILD_EVA_INSN("she", reg, addr)
+#define user_sb(reg, addr)		__BUILD_EVA_INSN("sbe", reg, addr)
+/* No 64-bit EVA instruction for storing double words */
+#define user_sd(reg, addr)		user_sw(reg, addr)
+
+#else
+
+#define user_cache(op, base)		"cache " op ", " base "\n"
+#define user_ll(reg, addr)		"ll " reg ", " addr "\n"
+#define user_sc(reg, addr)		"sc " reg ", " addr "\n"
+#define user_lw(reg, addr)		"lw " reg ", " addr "\n"
+#define user_lwl(reg, addr)		"lwl " reg ", " addr "\n"
+#define user_lwr(reg, addr)		"lwr " reg ", " addr "\n"
+#define user_lh(reg, addr)		"lh " reg ", " addr "\n"
+#define user_lb(reg, addr)		"lb " reg ", " addr "\n"
+#define user_lbu(reg, addr)		"lbu " reg ", " addr "\n"
+#define user_sw(reg, addr)		"sw " reg ", " addr "\n"
+#define user_swl(reg, addr)		"swl " reg ", " addr "\n"
+#define user_swr(reg, addr)		"swr " reg ", " addr "\n"
+#define user_sh(reg, addr)		"sh " reg ", " addr "\n"
+#define user_sb(reg, addr)		"sb " reg ", " addr "\n"
+
+#ifdef CONFIG_32BIT
+/*
+ * No 'sd' or 'ld' instructions in 32-bit but the code will
+ * do the correct thing
+ */
+#define user_sd(reg, addr)		user_sw(reg, addr)
+#define user_ld(reg, addr)		user_lw(reg, addr)
+#else
+#define user_sd(reg, addr)		"sd " reg", " addr "\n"
+#define user_ld(reg, addr)		"ld " reg", " addr "\n"
+#endif /* CONFIG_32BIT */
+
+#endif /* CONFIG_EVA */
+
+#else /* __ASSEMBLY__ */
+
+#ifdef CONFIG_EVA
+
+#define __BUILD_EVA_INSN(insn, reg, addr)			\
+				.set	push;			\
+				.set	mips0;			\
+				.set	eva;			\
+				insn reg, addr;			\
+				.set	pop;
+
+#define user_cache(op, base)		__BUILD_EVA_INSN(cachee, op, base)
+#define user_ll(reg, addr)		__BUILD_EVA_INSN(lle, reg, addr)
+#define user_sc(reg, addr)		__BUILD_EVA_INSN(sce, reg, addr)
+#define user_lw(reg, addr)		__BUILD_EVA_INSN(lwe, reg, addr)
+#define user_lwl(reg, addr)		__BUILD_EVA_INSN(lwle, reg, addr)
+#define user_lwr(reg, addr)		__BUILD_EVA_INSN(lwre, reg, addr)
+#define user_lh(reg, addr)		__BUILD_EVA_INSN(lhe, reg, addr)
+#define user_lb(reg, addr)		__BUILD_EVA_INSN(lbe, reg, addr)
+#define user_lbu(reg, addr)		__BUILD_EVA_INSN(lbue, reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_ld(reg, addr)		user_lw(reg, addr)
+#define user_sw(reg, addr)		__BUILD_EVA_INSN(swe, reg, addr)
+#define user_swl(reg, addr)		__BUILD_EVA_INSN(swle, reg, addr)
+#define user_swr(reg, addr)		__BUILD_EVA_INSN(swre, reg, addr)
+#define user_sh(reg, addr)		__BUILD_EVA_INSN(she, reg, addr)
+#define user_sb(reg, addr)		__BUILD_EVA_INSN(sbe, reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_sd(reg, addr)		user_sw(reg, addr)
+#else
+
+#define user_cache(op, base)		cache op, base
+#define user_ll(reg, addr)		ll reg, addr
+#define user_sc(reg, addr)		sc reg, addr
+#define user_lw(reg, addr)		lw reg, addr
+#define user_lwl(reg, addr)		lwl reg, addr
+#define user_lwr(reg, addr)		lwr reg, addr
+#define user_lh(reg, addr)		lh reg, addr
+#define user_lb(reg, addr)		lb reg, addr
+#define user_lbu(reg, addr)		lbu reg, addr
+#define user_sw(reg, addr)		sw reg, addr
+#define user_swl(reg, addr)		swl reg, addr
+#define user_swr(reg, addr)		swr reg, addr
+#define user_sh(reg, addr)		sh reg, addr
+#define user_sb(reg, addr)		sb reg, addr
+
+#ifdef CONFIG_32BIT
+/*
+ * No 'sd' or 'ld' instructions in 32-bit but the code will
+ * do the correct thing
+ */
+#define user_sd(reg, addr)		user_sw(reg, addr)
+#define user_ld(reg, addr)		user_lw(reg, addr)
+#else
+#define user_sd(reg, addr)		sd reg, addr
+#define user_ld(reg, addr)		ld reg, addr
+#endif /* CONFIG_32BIT */
+
+#endif /* CONFIG_EVA */
+
+#endif /* __ASSEMBLY__ */
+
 /*
  * PIC specific declarations
  * Not used for the kernel but here seems to be the right place.
-- 
1.8.5.3

[PATCH 04/58] MIPS: futex: Add EVA support for futex operations

[Markos Chandras]

Use LLE/SCE instructions for performing an address translation for
userspace when EVA is enabled.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/futex.h | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/arch/mips/include/asm/futex.h b/arch/mips/include/asm/futex.h
index 6ea1581..2a572e1 100644
--- a/arch/mips/include/asm/futex.h
+++ b/arch/mips/include/asm/futex.h
@@ -12,6 +12,7 @@
 
 #include <linux/futex.h>
 #include <linux/uaccess.h>
+#include <asm/asm.h>
 #include <asm/barrier.h>
 #include <asm/errno.h>
 #include <asm/war.h>
@@ -49,11 +50,11 @@
 		"	.set	push				\n"	\
 		"	.set	noat				\n"	\
 		"	.set	mips3				\n"	\
-		"1:	ll	%1, %4	# __futex_atomic_op	\n"	\
+		"1:	"user_ll("%1", "%4")" # __futex_atomic_op\n"	\
 		"	.set	mips0				\n"	\
 		"	" insn	"				\n"	\
 		"	.set	mips3				\n"	\
-		"2:	sc	$1, %2				\n"	\
+		"2:	"user_sc("$1", "%2")"			\n"	\
 		"	beqz	$1, 1b				\n"	\
 		__WEAK_LLSC_MB						\
 		"3:						\n"	\
@@ -174,12 +175,12 @@ futex_atomic_cmpxchg_inatomic(u32 *uval, u32 __user *uaddr,
 		"	.set	push					\n"
 		"	.set	noat					\n"
 		"	.set	mips3					\n"
-		"1:	ll	%1, %3					\n"
+		"1:	"user_ll("%1", "%3")"				\n"
 		"	bne	%1, %z4, 3f				\n"
 		"	.set	mips0					\n"
 		"	move	$1, %z5					\n"
 		"	.set	mips3					\n"
-		"2:	sc	$1, %2					\n"
+		"2:	"user_sc("$1", "%2")"				\n"
 		"	beqz	$1, 1b					\n"
 		__WEAK_LLSC_MB
 		"3:							\n"
-- 
1.8.5.3

[PATCH 04/58] MIPS: futex: Add EVA support for futex operations

[Markos Chandras]

Use LLE/SCE instructions for performing an address translation for
userspace when EVA is enabled.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/futex.h | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/arch/mips/include/asm/futex.h b/arch/mips/include/asm/futex.h
index 6ea1581..2a572e1 100644
--- a/arch/mips/include/asm/futex.h
+++ b/arch/mips/include/asm/futex.h
@@ -12,6 +12,7 @@
 
 #include <linux/futex.h>
 #include <linux/uaccess.h>
+#include <asm/asm.h>
 #include <asm/barrier.h>
 #include <asm/errno.h>
 #include <asm/war.h>
@@ -49,11 +50,11 @@
 		"	.set	push				\n"	\
 		"	.set	noat				\n"	\
 		"	.set	mips3				\n"	\
-		"1:	ll	%1, %4	# __futex_atomic_op	\n"	\
+		"1:	"user_ll("%1", "%4")" # __futex_atomic_op\n"	\
 		"	.set	mips0				\n"	\
 		"	" insn	"				\n"	\
 		"	.set	mips3				\n"	\
-		"2:	sc	$1, %2				\n"	\
+		"2:	"user_sc("$1", "%2")"			\n"	\
 		"	beqz	$1, 1b				\n"	\
 		__WEAK_LLSC_MB						\
 		"3:						\n"	\
@@ -174,12 +175,12 @@ futex_atomic_cmpxchg_inatomic(u32 *uval, u32 __user *uaddr,
 		"	.set	push					\n"
 		"	.set	noat					\n"
 		"	.set	mips3					\n"
-		"1:	ll	%1, %3					\n"
+		"1:	"user_ll("%1", "%3")"				\n"
 		"	bne	%1, %z4, 3f				\n"
 		"	.set	mips0					\n"
 		"	move	$1, %z5					\n"
 		"	.set	mips3					\n"
-		"2:	sc	$1, %2					\n"
+		"2:	"user_sc("$1", "%2")"				\n"
 		"	beqz	$1, 1b					\n"
 		__WEAK_LLSC_MB
 		"3:							\n"
-- 
1.8.5.3

[PATCH 05/58] MIPS: uapi: inst: Add new EVA opcodes

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/uapi/asm/inst.h | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/arch/mips/include/uapi/asm/inst.h b/arch/mips/include/uapi/asm/inst.h
index b39ba25..9ce652e 100644
--- a/arch/mips/include/uapi/asm/inst.h
+++ b/arch/mips/include/uapi/asm/inst.h
@@ -73,10 +73,16 @@ enum spec2_op {
 enum spec3_op {
 	ext_op, dextm_op, dextu_op, dext_op,
 	ins_op, dinsm_op, dinsu_op, dins_op,
-	lx_op = 0x0a,
-	bshfl_op = 0x20,
-	dbshfl_op = 0x24,
-	rdhwr_op = 0x3b
+	lx_op     = 0x0a, lwle_op   = 0x19,
+	lwre_op   = 0x1a, cachee_op = 0x1b,
+	sbe_op    = 0x1c, she_op    = 0x1d,
+	sce_op    = 0x1e, swe_op    = 0x1f,
+	bshfl_op  = 0x20, swle_op   = 0x21,
+	swre_op   = 0x22, prefe_op  = 0x23,
+	dbshfl_op = 0x24, lbue_op   = 0x28,
+	lhue_op   = 0x29, lbe_op    = 0x2c,
+	lhe_op    = 0x2d, lle_op    = 0x2e,
+	lwe_op    = 0x2f, rdhwr_op  = 0x3b
 };
 
 /*
-- 
1.8.5.3

[PATCH 05/58] MIPS: uapi: inst: Add new EVA opcodes

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/uapi/asm/inst.h | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/arch/mips/include/uapi/asm/inst.h b/arch/mips/include/uapi/asm/inst.h
index b39ba25..9ce652e 100644
--- a/arch/mips/include/uapi/asm/inst.h
+++ b/arch/mips/include/uapi/asm/inst.h
@@ -73,10 +73,16 @@ enum spec2_op {
 enum spec3_op {
 	ext_op, dextm_op, dextu_op, dext_op,
 	ins_op, dinsm_op, dinsu_op, dins_op,
-	lx_op = 0x0a,
-	bshfl_op = 0x20,
-	dbshfl_op = 0x24,
-	rdhwr_op = 0x3b
+	lx_op     = 0x0a, lwle_op   = 0x19,
+	lwre_op   = 0x1a, cachee_op = 0x1b,
+	sbe_op    = 0x1c, she_op    = 0x1d,
+	sce_op    = 0x1e, swe_op    = 0x1f,
+	bshfl_op  = 0x20, swle_op   = 0x21,
+	swre_op   = 0x22, prefe_op  = 0x23,
+	dbshfl_op = 0x24, lbue_op   = 0x28,
+	lhue_op   = 0x29, lbe_op    = 0x2c,
+	lhe_op    = 0x2d, lle_op    = 0x2e,
+	lwe_op    = 0x2f, rdhwr_op  = 0x3b
 };
 
 /*
-- 
1.8.5.3

[PATCH 06/58] MIPS: uapi: inst: Add instruction format for SPECIAL3 instructions

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/uapi/asm/inst.h | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/arch/mips/include/uapi/asm/inst.h b/arch/mips/include/uapi/asm/inst.h
index 9ce652e..1a44c5a 100644
--- a/arch/mips/include/uapi/asm/inst.h
+++ b/arch/mips/include/uapi/asm/inst.h
@@ -8,6 +8,7 @@
  * Copyright (C) 1996, 2000 by Ralf Baechle
  * Copyright (C) 2006 by Thiemo Seufer
  * Copyright (C) 2012 MIPS Technologies, Inc.  All rights reserved.
+ * Copyright (C) 2014 Imagination Technologies Ltd.
  */
 #ifndef _UAPI_ASM_INST_H
 #define _UAPI_ASM_INST_H
@@ -598,6 +599,15 @@ struct v_format {				/* MDMX vector format */
 	;)))))))
 };
 
+struct spec3_format {   /* SPEC3 */
+	BITFIELD_FIELD(unsigned int opcode:6,
+	BITFIELD_FIELD(unsigned int rs:5,
+	BITFIELD_FIELD(unsigned int rt:5,
+	BITFIELD_FIELD(signed int simmediate:9,
+	BITFIELD_FIELD(unsigned int func:7,
+	;)))))
+};
+
 /*
  * microMIPS instruction formats (32-bit length)
  *
@@ -869,6 +879,7 @@ union mips_instruction {
 	struct b_format b_format;
 	struct ps_format ps_format;
 	struct v_format v_format;
+	struct spec3_format spec3_format;
 	struct fb_format fb_format;
 	struct fp0_format fp0_format;
 	struct mm_fp0_format mm_fp0_format;
-- 
1.8.5.3

[PATCH 06/58] MIPS: uapi: inst: Add instruction format for SPECIAL3 instructions

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/uapi/asm/inst.h | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/arch/mips/include/uapi/asm/inst.h b/arch/mips/include/uapi/asm/inst.h
index 9ce652e..1a44c5a 100644
--- a/arch/mips/include/uapi/asm/inst.h
+++ b/arch/mips/include/uapi/asm/inst.h
@@ -8,6 +8,7 @@
  * Copyright (C) 1996, 2000 by Ralf Baechle
  * Copyright (C) 2006 by Thiemo Seufer
  * Copyright (C) 2012 MIPS Technologies, Inc.  All rights reserved.
+ * Copyright (C) 2014 Imagination Technologies Ltd.
  */
 #ifndef _UAPI_ASM_INST_H
 #define _UAPI_ASM_INST_H
@@ -598,6 +599,15 @@ struct v_format {				/* MDMX vector format */
 	;)))))))
 };
 
+struct spec3_format {   /* SPEC3 */
+	BITFIELD_FIELD(unsigned int opcode:6,
+	BITFIELD_FIELD(unsigned int rs:5,
+	BITFIELD_FIELD(unsigned int rt:5,
+	BITFIELD_FIELD(signed int simmediate:9,
+	BITFIELD_FIELD(unsigned int func:7,
+	;)))))
+};
+
 /*
  * microMIPS instruction formats (32-bit length)
  *
@@ -869,6 +879,7 @@ union mips_instruction {
 	struct b_format b_format;
 	struct ps_format ps_format;
 	struct v_format v_format;
+	struct spec3_format spec3_format;
 	struct fb_format fb_format;
 	struct fp0_format fp0_format;
 	struct mm_fp0_format mm_fp0_format;
-- 
1.8.5.3

[PATCH 07/58] MIPS: kernel: scall32-o32: Use EVA wrappers to fetch syscall arguments

[Markos Chandras]

Arguments 4-8 are stored on user's stack, so use the EVA instructions
to fetch them if EVA is enabled.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/scall32-o32.S | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/arch/mips/kernel/scall32-o32.S b/arch/mips/kernel/scall32-o32.S
index e8e541b..2fae6eb 100644
--- a/arch/mips/kernel/scall32-o32.S
+++ b/arch/mips/kernel/scall32-o32.S
@@ -6,6 +6,7 @@
  * Copyright (C) 1995-99, 2000- 02, 06 Ralf Baechle <ralf@linux-mips.org>
  * Copyright (C) 2001 MIPS Technologies, Inc.
  * Copyright (C) 2004 Thiemo Seufer
+ * Copyright (C) 2014 Imagination Technologies Ltd.
  */
 #include <linux/errno.h>
 #include <asm/asm.h>
@@ -74,10 +75,10 @@ NESTED(handle_sys, PT_SIZE, sp)
 	.set    noreorder
 	.set	nomacro
 
-1:	lw	t5, 16(t0)		# argument #5 from usp
-4:	lw	t6, 20(t0)		# argument #6 from usp
-3:	lw	t7, 24(t0)		# argument #7 from usp
-2:	lw	t8, 28(t0)		# argument #8 from usp
+1:	user_lw(t5, 16(t0))		# argument #5 from usp
+4:	user_lw(t6, 20(t0))		# argument #6 from usp
+3:	user_lw(t7, 24(t0))		# argument #7 from usp
+2:	user_lw(t8, 28(t0))		# argument #8 from usp
 
 	sw	t5, 16(sp)		# argument #5 to ksp
 	sw	t6, 20(sp)		# argument #6 to ksp
-- 
1.8.5.3

[PATCH 07/58] MIPS: kernel: scall32-o32: Use EVA wrappers to fetch syscall arguments

[Markos Chandras]

Arguments 4-8 are stored on user's stack, so use the EVA instructions
to fetch them if EVA is enabled.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/scall32-o32.S | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/arch/mips/kernel/scall32-o32.S b/arch/mips/kernel/scall32-o32.S
index e8e541b..2fae6eb 100644
--- a/arch/mips/kernel/scall32-o32.S
+++ b/arch/mips/kernel/scall32-o32.S
@@ -6,6 +6,7 @@
  * Copyright (C) 1995-99, 2000- 02, 06 Ralf Baechle <ralf@linux-mips.org>
  * Copyright (C) 2001 MIPS Technologies, Inc.
  * Copyright (C) 2004 Thiemo Seufer
+ * Copyright (C) 2014 Imagination Technologies Ltd.
  */
 #include <linux/errno.h>
 #include <asm/asm.h>
@@ -74,10 +75,10 @@ NESTED(handle_sys, PT_SIZE, sp)
 	.set    noreorder
 	.set	nomacro
 
-1:	lw	t5, 16(t0)		# argument #5 from usp
-4:	lw	t6, 20(t0)		# argument #6 from usp
-3:	lw	t7, 24(t0)		# argument #7 from usp
-2:	lw	t8, 28(t0)		# argument #8 from usp
+1:	user_lw(t5, 16(t0))		# argument #5 from usp
+4:	user_lw(t6, 20(t0))		# argument #6 from usp
+3:	user_lw(t7, 24(t0))		# argument #7 from usp
+2:	user_lw(t8, 28(t0))		# argument #8 from usp
 
 	sw	t5, 16(sp)		# argument #5 to ksp
 	sw	t6, 20(sp)		# argument #6 to ksp
-- 
1.8.5.3

[PATCH 08/58] MIPS: kernel: traps: Whitespace clean up

[Markos Chandras]

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/traps.c | 17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)

diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
index e0b4996..3e5e700 100644
--- a/arch/mips/kernel/traps.c
+++ b/arch/mips/kernel/traps.c
@@ -10,6 +10,7 @@
  * Kevin D. Kissell, kevink@mips.com and Carsten Langgaard, carstenl@mips.com
  * Copyright (C) 2002, 2003, 2004, 2005, 2007  Maciej W. Rozycki
  * Copyright (C) 2000, 2001, 2012 MIPS Technologies, Inc.  All rights reserved.
+ * Copyright (C) 2014, Imagination Technologies Ltd.
  */
 #include <linux/bug.h>
 #include <linux/compiler.h>
@@ -870,17 +871,19 @@ asmlinkage void do_bp(struct pt_regs *regs)
 			if ((__get_user(instr[0], (u16 __user *)msk_isa16_mode(epc)) ||
 			    (__get_user(instr[1], (u16 __user *)msk_isa16_mode(epc + 2)))))
 				goto out_sigsegv;
-		    opcode = (instr[0] << 16) | instr[1];
+			opcode = (instr[0] << 16) | instr[1];
 		} else {
-		    /* MIPS16e mode */
-		    if (__get_user(instr[0], (u16 __user *)msk_isa16_mode(epc)))
+			/* MIPS16e mode */
+			if (__get_user(instr[0],
+				       (u16 __user *)msk_isa16_mode(epc)))
 				goto out_sigsegv;
-		    bcode = (instr[0] >> 6) & 0x3f;
-		    do_trap_or_bp(regs, bcode, "Break");
-		    goto out;
+			bcode = (instr[0] >> 6) & 0x3f;
+			do_trap_or_bp(regs, bcode, "Break");
+			goto out;
 		}
 	} else {
-		if (__get_user(opcode, (unsigned int __user *) exception_epc(regs)))
+		if (__get_user(opcode,
+			       (unsigned int __user *) exception_epc(regs)))
 			goto out_sigsegv;
 	}
 
-- 
1.8.5.3

[PATCH 08/58] MIPS: kernel: traps: Whitespace clean up

[Markos Chandras]

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/traps.c | 17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)

diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
index e0b4996..3e5e700 100644
--- a/arch/mips/kernel/traps.c
+++ b/arch/mips/kernel/traps.c
@@ -10,6 +10,7 @@
  * Kevin D. Kissell, kevink@mips.com and Carsten Langgaard, carstenl@mips.com
  * Copyright (C) 2002, 2003, 2004, 2005, 2007  Maciej W. Rozycki
  * Copyright (C) 2000, 2001, 2012 MIPS Technologies, Inc.  All rights reserved.
+ * Copyright (C) 2014, Imagination Technologies Ltd.
  */
 #include <linux/bug.h>
 #include <linux/compiler.h>
@@ -870,17 +871,19 @@ asmlinkage void do_bp(struct pt_regs *regs)
 			if ((__get_user(instr[0], (u16 __user *)msk_isa16_mode(epc)) ||
 			    (__get_user(instr[1], (u16 __user *)msk_isa16_mode(epc + 2)))))
 				goto out_sigsegv;
-		    opcode = (instr[0] << 16) | instr[1];
+			opcode = (instr[0] << 16) | instr[1];
 		} else {
-		    /* MIPS16e mode */
-		    if (__get_user(instr[0], (u16 __user *)msk_isa16_mode(epc)))
+			/* MIPS16e mode */
+			if (__get_user(instr[0],
+				       (u16 __user *)msk_isa16_mode(epc)))
 				goto out_sigsegv;
-		    bcode = (instr[0] >> 6) & 0x3f;
-		    do_trap_or_bp(regs, bcode, "Break");
-		    goto out;
+			bcode = (instr[0] >> 6) & 0x3f;
+			do_trap_or_bp(regs, bcode, "Break");
+			goto out;
 		}
 	} else {
-		if (__get_user(opcode, (unsigned int __user *) exception_epc(regs)))
+		if (__get_user(opcode,
+			       (unsigned int __user *) exception_epc(regs)))
 			goto out_sigsegv;
 	}
 
-- 
1.8.5.3

[PATCH 09/58] MIPS: traps: Set correct address limit for breakpoints and traps

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

When a breakpoint or trap happens when operating in kernel mode but
on users behalf (eg syscall) it is necessary to change the address
limit to KERNEL_DS so any address checking can be bypassed and print
the correct stack trace.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/traps.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
index 3e5e700..a5a74d6 100644
--- a/arch/mips/kernel/traps.c
+++ b/arch/mips/kernel/traps.c
@@ -862,6 +862,11 @@ asmlinkage void do_bp(struct pt_regs *regs)
 	enum ctx_state prev_state;
 	unsigned long epc;
 	u16 instr[2];
+	mm_segment_t seg;
+
+	seg = get_fs();
+	if (!user_mode(regs))
+		set_fs(KERNEL_DS);
 
 	prev_state = exception_enter();
 	if (get_isa16_mode(regs->cp0_epc)) {
@@ -921,6 +926,7 @@ asmlinkage void do_bp(struct pt_regs *regs)
 	do_trap_or_bp(regs, bcode, "Break");
 
 out:
+	set_fs(seg);
 	exception_exit(prev_state);
 	return;
 
@@ -934,8 +940,13 @@ asmlinkage void do_tr(struct pt_regs *regs)
 	u32 opcode, tcode = 0;
 	enum ctx_state prev_state;
 	u16 instr[2];
+	mm_segment_t seg;
 	unsigned long epc = msk_isa16_mode(exception_epc(regs));
 
+	seg = get_fs();
+	if (!user_mode(regs))
+		set_fs(get_ds());
+
 	prev_state = exception_enter();
 	if (get_isa16_mode(regs->cp0_epc)) {
 		if (__get_user(instr[0], (u16 __user *)(epc + 0)) ||
@@ -956,6 +967,7 @@ asmlinkage void do_tr(struct pt_regs *regs)
 	do_trap_or_bp(regs, tcode, "Trap");
 
 out:
+	set_fs(seg);
 	exception_exit(prev_state);
 	return;
 
-- 
1.8.5.3

[PATCH 09/58] MIPS: traps: Set correct address limit for breakpoints and traps

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

When a breakpoint or trap happens when operating in kernel mode but
on users behalf (eg syscall) it is necessary to change the address
limit to KERNEL_DS so any address checking can be bypassed and print
the correct stack trace.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/traps.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
index 3e5e700..a5a74d6 100644
--- a/arch/mips/kernel/traps.c
+++ b/arch/mips/kernel/traps.c
@@ -862,6 +862,11 @@ asmlinkage void do_bp(struct pt_regs *regs)
 	enum ctx_state prev_state;
 	unsigned long epc;
 	u16 instr[2];
+	mm_segment_t seg;
+
+	seg = get_fs();
+	if (!user_mode(regs))
+		set_fs(KERNEL_DS);
 
 	prev_state = exception_enter();
 	if (get_isa16_mode(regs->cp0_epc)) {
@@ -921,6 +926,7 @@ asmlinkage void do_bp(struct pt_regs *regs)
 	do_trap_or_bp(regs, bcode, "Break");
 
 out:
+	set_fs(seg);
 	exception_exit(prev_state);
 	return;
 
@@ -934,8 +940,13 @@ asmlinkage void do_tr(struct pt_regs *regs)
 	u32 opcode, tcode = 0;
 	enum ctx_state prev_state;
 	u16 instr[2];
+	mm_segment_t seg;
 	unsigned long epc = msk_isa16_mode(exception_epc(regs));
 
+	seg = get_fs();
+	if (!user_mode(regs))
+		set_fs(get_ds());
+
 	prev_state = exception_enter();
 	if (get_isa16_mode(regs->cp0_epc)) {
 		if (__get_user(instr[0], (u16 __user *)(epc + 0)) ||
@@ -956,6 +967,7 @@ asmlinkage void do_tr(struct pt_regs *regs)
 	do_trap_or_bp(regs, tcode, "Trap");
 
 out:
+	set_fs(seg);
 	exception_exit(prev_state);
 	return;
 
-- 
1.8.5.3

[PATCH 10/58] MIPS: lib: strnlen_user: Use macro to build the strnlen_user symbol

[Markos Chandras]

Build the __strnlen_user symbol using a macro. In EVA mode we will
need to use similar code to do the userspace load operations so
it is better if we use a macro to avoid code duplications.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/strnlen_user.S | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/arch/mips/lib/strnlen_user.S b/arch/mips/lib/strnlen_user.S
index fcacea5..4422160 100644
--- a/arch/mips/lib/strnlen_user.S
+++ b/arch/mips/lib/strnlen_user.S
@@ -25,22 +25,26 @@
  *	 bytes.	 There's nothing secret there.	On 64-bit accessing beyond
  *	 the maximum is a tad hairier ...
  */
-LEAF(__strnlen_user_asm)
+	.macro __BUILD_STRNLEN_ASM func
+LEAF(__strnlen_\func\()_asm)
 	LONG_L		v0, TI_ADDR_LIMIT($28)	# pointer ok?
 	and		v0, a0
-	bnez		v0, .Lfault
+	bnez		v0, .Lfault\@
 
-FEXPORT(__strnlen_user_nocheck_asm)
+FEXPORT(__strnlen_\func\()_nocheck_asm)
 	move		v0, a0
 	PTR_ADDU	a1, a0			# stop pointer
 1:	beq		v0, a1, 1f		# limit reached?
-	EX(lb, t0, (v0), .Lfault)
+	EX(lb, t0, (v0), .Lfault\@)
 	PTR_ADDIU	v0, 1
 	bnez		t0, 1b
 1:	PTR_SUBU	v0, a0
 	jr		ra
-	END(__strnlen_user_asm)
+	END(__strnlen_\func\()_asm)
 
-.Lfault:
+.Lfault\@:
 	move		v0, zero
 	jr		ra
+	.endm
+
+__BUILD_STRNLEN_ASM user
-- 
1.8.5.3

[PATCH 10/58] MIPS: lib: strnlen_user: Use macro to build the strnlen_user symbol

[Markos Chandras]

Build the __strnlen_user symbol using a macro. In EVA mode we will
need to use similar code to do the userspace load operations so
it is better if we use a macro to avoid code duplications.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/strnlen_user.S | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/arch/mips/lib/strnlen_user.S b/arch/mips/lib/strnlen_user.S
index fcacea5..4422160 100644
--- a/arch/mips/lib/strnlen_user.S
+++ b/arch/mips/lib/strnlen_user.S
@@ -25,22 +25,26 @@
  *	 bytes.	 There's nothing secret there.	On 64-bit accessing beyond
  *	 the maximum is a tad hairier ...
  */
-LEAF(__strnlen_user_asm)
+	.macro __BUILD_STRNLEN_ASM func
+LEAF(__strnlen_\func\()_asm)
 	LONG_L		v0, TI_ADDR_LIMIT($28)	# pointer ok?
 	and		v0, a0
-	bnez		v0, .Lfault
+	bnez		v0, .Lfault\@
 
-FEXPORT(__strnlen_user_nocheck_asm)
+FEXPORT(__strnlen_\func\()_nocheck_asm)
 	move		v0, a0
 	PTR_ADDU	a1, a0			# stop pointer
 1:	beq		v0, a1, 1f		# limit reached?
-	EX(lb, t0, (v0), .Lfault)
+	EX(lb, t0, (v0), .Lfault\@)
 	PTR_ADDIU	v0, 1
 	bnez		t0, 1b
 1:	PTR_SUBU	v0, a0
 	jr		ra
-	END(__strnlen_user_asm)
+	END(__strnlen_\func\()_asm)
 
-.Lfault:
+.Lfault\@:
 	move		v0, zero
 	jr		ra
+	.endm
+
+__BUILD_STRNLEN_ASM user
-- 
1.8.5.3

[PATCH 11/58] MIPS: lib: strnlen_user: Add EVA support

[Markos Chandras]

In non-EVA mode, a strlen_user* alias is used for the
strlen_kernel* symbols since the code is identical. In EVA
mode, a new strlen_user* symbol is used which uses the EVA
specific instructions to load values from userspace.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/mips_ksyms.c |  4 ++++
 arch/mips/lib/strnlen_user.S  | 20 ++++++++++++++++++++
 2 files changed, 24 insertions(+)

diff --git a/arch/mips/kernel/mips_ksyms.c b/arch/mips/kernel/mips_ksyms.c
index 6e58e97..a322db0 100644
--- a/arch/mips/kernel/mips_ksyms.c
+++ b/arch/mips/kernel/mips_ksyms.c
@@ -22,6 +22,8 @@ extern long __strncpy_from_user_asm(char *__to, const char *__from,
 				    long __len);
 extern long __strlen_user_nocheck_asm(const char *s);
 extern long __strlen_user_asm(const char *s);
+extern long __strnlen_kernel_nocheck_asm(const char *s);
+extern long __strnlen_kernel_asm(const char *s);
 extern long __strnlen_user_nocheck_asm(const char *s);
 extern long __strnlen_user_asm(const char *s);
 
@@ -48,6 +50,8 @@ EXPORT_SYMBOL(__strncpy_from_user_nocheck_asm);
 EXPORT_SYMBOL(__strncpy_from_user_asm);
 EXPORT_SYMBOL(__strlen_user_nocheck_asm);
 EXPORT_SYMBOL(__strlen_user_asm);
+EXPORT_SYMBOL(__strnlen_kernel_nocheck_asm);
+EXPORT_SYMBOL(__strnlen_kernel_asm);
 EXPORT_SYMBOL(__strnlen_user_nocheck_asm);
 EXPORT_SYMBOL(__strnlen_user_asm);
 
diff --git a/arch/mips/lib/strnlen_user.S b/arch/mips/lib/strnlen_user.S
index 4422160..f3af699 100644
--- a/arch/mips/lib/strnlen_user.S
+++ b/arch/mips/lib/strnlen_user.S
@@ -35,7 +35,11 @@ FEXPORT(__strnlen_\func\()_nocheck_asm)
 	move		v0, a0
 	PTR_ADDU	a1, a0			# stop pointer
 1:	beq		v0, a1, 1f		# limit reached?
+.ifeqs "\func", "kernel"
 	EX(lb, t0, (v0), .Lfault\@)
+.else
+	EX(lbe, t0, (v0), .Lfault\@)
+.endif
 	PTR_ADDIU	v0, 1
 	bnez		t0, 1b
 1:	PTR_SUBU	v0, a0
@@ -47,4 +51,20 @@ FEXPORT(__strnlen_\func\()_nocheck_asm)
 	jr		ra
 	.endm
 
+#ifndef CONFIG_EVA
+	/* Set aliases */
+	.global __strnlen_user_asm
+	.global __strnlen_user_nocheck_asm
+	.set __strnlen_user_asm, __strnlen_kernel_asm
+	.set __strnlen_user_nocheck_asm, __strnlen_kernel_nocheck_asm
+#endif
+
+__BUILD_STRNLEN_ASM kernel
+
+#ifdef CONFIG_EVA
+
+	.set push
+	.set eva
 __BUILD_STRNLEN_ASM user
+	.set pop
+#endif
-- 
1.8.5.3

[PATCH 11/58] MIPS: lib: strnlen_user: Add EVA support

[Markos Chandras]

In non-EVA mode, a strlen_user* alias is used for the
strlen_kernel* symbols since the code is identical. In EVA
mode, a new strlen_user* symbol is used which uses the EVA
specific instructions to load values from userspace.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/mips_ksyms.c |  4 ++++
 arch/mips/lib/strnlen_user.S  | 20 ++++++++++++++++++++
 2 files changed, 24 insertions(+)

diff --git a/arch/mips/kernel/mips_ksyms.c b/arch/mips/kernel/mips_ksyms.c
index 6e58e97..a322db0 100644
--- a/arch/mips/kernel/mips_ksyms.c
+++ b/arch/mips/kernel/mips_ksyms.c
@@ -22,6 +22,8 @@ extern long __strncpy_from_user_asm(char *__to, const char *__from,
 				    long __len);
 extern long __strlen_user_nocheck_asm(const char *s);
 extern long __strlen_user_asm(const char *s);
+extern long __strnlen_kernel_nocheck_asm(const char *s);
+extern long __strnlen_kernel_asm(const char *s);
 extern long __strnlen_user_nocheck_asm(const char *s);
 extern long __strnlen_user_asm(const char *s);
 
@@ -48,6 +50,8 @@ EXPORT_SYMBOL(__strncpy_from_user_nocheck_asm);
 EXPORT_SYMBOL(__strncpy_from_user_asm);
 EXPORT_SYMBOL(__strlen_user_nocheck_asm);
 EXPORT_SYMBOL(__strlen_user_asm);
+EXPORT_SYMBOL(__strnlen_kernel_nocheck_asm);
+EXPORT_SYMBOL(__strnlen_kernel_asm);
 EXPORT_SYMBOL(__strnlen_user_nocheck_asm);
 EXPORT_SYMBOL(__strnlen_user_asm);
 
diff --git a/arch/mips/lib/strnlen_user.S b/arch/mips/lib/strnlen_user.S
index 4422160..f3af699 100644
--- a/arch/mips/lib/strnlen_user.S
+++ b/arch/mips/lib/strnlen_user.S
@@ -35,7 +35,11 @@ FEXPORT(__strnlen_\func\()_nocheck_asm)
 	move		v0, a0
 	PTR_ADDU	a1, a0			# stop pointer
 1:	beq		v0, a1, 1f		# limit reached?
+.ifeqs "\func", "kernel"
 	EX(lb, t0, (v0), .Lfault\@)
+.else
+	EX(lbe, t0, (v0), .Lfault\@)
+.endif
 	PTR_ADDIU	v0, 1
 	bnez		t0, 1b
 1:	PTR_SUBU	v0, a0
@@ -47,4 +51,20 @@ FEXPORT(__strnlen_\func\()_nocheck_asm)
 	jr		ra
 	.endm
 
+#ifndef CONFIG_EVA
+	/* Set aliases */
+	.global __strnlen_user_asm
+	.global __strnlen_user_nocheck_asm
+	.set __strnlen_user_asm, __strnlen_kernel_asm
+	.set __strnlen_user_nocheck_asm, __strnlen_kernel_nocheck_asm
+#endif
+
+__BUILD_STRNLEN_ASM kernel
+
+#ifdef CONFIG_EVA
+
+	.set push
+	.set eva
 __BUILD_STRNLEN_ASM user
+	.set pop
+#endif
-- 
1.8.5.3

[PATCH 12/58] MIPS: lib: strlen_user: Use macro to build the strlen_user symbol

[Markos Chandras]

Build the __strlen_user symbol using a macro. In EVA mode we will
need to use similar code to do the userspace load operations so
it is better if we use a macro to avoid code duplications.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/strlen_user.S | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/arch/mips/lib/strlen_user.S b/arch/mips/lib/strlen_user.S
index e362dcd..6e8bdb3 100644
--- a/arch/mips/lib/strlen_user.S
+++ b/arch/mips/lib/strlen_user.S
@@ -22,19 +22,23 @@
  *
  * Return 0 for error
  */
-LEAF(__strlen_user_asm)
+	.macro __BUILD_STRLEN_ASM func
+LEAF(__strlen_\func\()_asm)
 	LONG_L		v0, TI_ADDR_LIMIT($28)	# pointer ok?
 	and		v0, a0
-	bnez		v0, .Lfault
+	bnez		v0, .Lfault\@
 
-FEXPORT(__strlen_user_nocheck_asm)
+FEXPORT(__strlen_\func\()_nocheck_asm)
 	move		v0, a0
-1:	EX(lbu, v1, (v0), .Lfault)
+1:	EX(lbu, v1, (v0), .Lfault\@)
 	PTR_ADDIU	v0, 1
 	bnez		v1, 1b
 	PTR_SUBU	v0, a0
 	jr		ra
-	END(__strlen_user_asm)
+	END(__strlen_\func\()_asm)
 
-.Lfault:	move		v0, zero
+.Lfault\@:	move		v0, zero
 	jr		ra
+	.endm
+
+__BUILD_STRLEN_ASM user
-- 
1.8.5.3

[PATCH 12/58] MIPS: lib: strlen_user: Use macro to build the strlen_user symbol

[Markos Chandras]

Build the __strlen_user symbol using a macro. In EVA mode we will
need to use similar code to do the userspace load operations so
it is better if we use a macro to avoid code duplications.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/strlen_user.S | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/arch/mips/lib/strlen_user.S b/arch/mips/lib/strlen_user.S
index e362dcd..6e8bdb3 100644
--- a/arch/mips/lib/strlen_user.S
+++ b/arch/mips/lib/strlen_user.S
@@ -22,19 +22,23 @@
  *
  * Return 0 for error
  */
-LEAF(__strlen_user_asm)
+	.macro __BUILD_STRLEN_ASM func
+LEAF(__strlen_\func\()_asm)
 	LONG_L		v0, TI_ADDR_LIMIT($28)	# pointer ok?
 	and		v0, a0
-	bnez		v0, .Lfault
+	bnez		v0, .Lfault\@
 
-FEXPORT(__strlen_user_nocheck_asm)
+FEXPORT(__strlen_\func\()_nocheck_asm)
 	move		v0, a0
-1:	EX(lbu, v1, (v0), .Lfault)
+1:	EX(lbu, v1, (v0), .Lfault\@)
 	PTR_ADDIU	v0, 1
 	bnez		v1, 1b
 	PTR_SUBU	v0, a0
 	jr		ra
-	END(__strlen_user_asm)
+	END(__strlen_\func\()_asm)
 
-.Lfault:	move		v0, zero
+.Lfault\@:	move		v0, zero
 	jr		ra
+	.endm
+
+__BUILD_STRLEN_ASM user
-- 
1.8.5.3

[PATCH 13/58] MIPS: lib: strlen_user: Add EVA support

[Markos Chandras]

In non-EVA mode, strlen_user* aliases are used for the
strlen_kernel* symbols since the code is identical. In EVA
mode, new strlen_user* symbols are used which use the EVA
specific instructions to load values from userspace.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/mips_ksyms.c |  4 ++++
 arch/mips/lib/strlen_user.S   | 20 ++++++++++++++++++++
 2 files changed, 24 insertions(+)

diff --git a/arch/mips/kernel/mips_ksyms.c b/arch/mips/kernel/mips_ksyms.c
index a322db0..742ed7d 100644
--- a/arch/mips/kernel/mips_ksyms.c
+++ b/arch/mips/kernel/mips_ksyms.c
@@ -20,6 +20,8 @@ extern long __strncpy_from_user_nocheck_asm(char *__to,
 					    const char *__from, long __len);
 extern long __strncpy_from_user_asm(char *__to, const char *__from,
 				    long __len);
+extern long __strlen_kernel_nocheck_asm(const char *s);
+extern long __strlen_kernel_asm(const char *s);
 extern long __strlen_user_nocheck_asm(const char *s);
 extern long __strlen_user_asm(const char *s);
 extern long __strnlen_kernel_nocheck_asm(const char *s);
@@ -48,6 +50,8 @@ EXPORT_SYMBOL(__copy_user_inatomic);
 EXPORT_SYMBOL(__bzero);
 EXPORT_SYMBOL(__strncpy_from_user_nocheck_asm);
 EXPORT_SYMBOL(__strncpy_from_user_asm);
+EXPORT_SYMBOL(__strlen_kernel_nocheck_asm);
+EXPORT_SYMBOL(__strlen_kernel_asm);
 EXPORT_SYMBOL(__strlen_user_nocheck_asm);
 EXPORT_SYMBOL(__strlen_user_asm);
 EXPORT_SYMBOL(__strnlen_kernel_nocheck_asm);
diff --git a/arch/mips/lib/strlen_user.S b/arch/mips/lib/strlen_user.S
index 6e8bdb3..bef65c9 100644
--- a/arch/mips/lib/strlen_user.S
+++ b/arch/mips/lib/strlen_user.S
@@ -30,7 +30,11 @@ LEAF(__strlen_\func\()_asm)
 
 FEXPORT(__strlen_\func\()_nocheck_asm)
 	move		v0, a0
+.ifeqs "\func", "kernel"
 1:	EX(lbu, v1, (v0), .Lfault\@)
+.else
+1:	EX(lbue, v1, (v0), .Lfault\@)
+.endif
 	PTR_ADDIU	v0, 1
 	bnez		v1, 1b
 	PTR_SUBU	v0, a0
@@ -41,4 +45,20 @@ FEXPORT(__strlen_\func\()_nocheck_asm)
 	jr		ra
 	.endm
 
+#ifndef CONFIG_EVA
+	/* Set aliases */
+	.global __strlen_user_asm
+	.global __strlen_user_nocheck_asm
+	.set __strlen_user_asm, __strlen_kernel_asm
+	.set __strlen_user_nocheck_asm, __strlen_kernel_nocheck_asm
+#endif
+
+__BUILD_STRLEN_ASM kernel
+
+#ifdef CONFIG_EVA
+
+	.set push
+	.set eva
 __BUILD_STRLEN_ASM user
+	.set pop
+#endif
-- 
1.8.5.3

[PATCH 13/58] MIPS: lib: strlen_user: Add EVA support

[Markos Chandras]

In non-EVA mode, strlen_user* aliases are used for the
strlen_kernel* symbols since the code is identical. In EVA
mode, new strlen_user* symbols are used which use the EVA
specific instructions to load values from userspace.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/mips_ksyms.c |  4 ++++
 arch/mips/lib/strlen_user.S   | 20 ++++++++++++++++++++
 2 files changed, 24 insertions(+)

diff --git a/arch/mips/kernel/mips_ksyms.c b/arch/mips/kernel/mips_ksyms.c
index a322db0..742ed7d 100644
--- a/arch/mips/kernel/mips_ksyms.c
+++ b/arch/mips/kernel/mips_ksyms.c
@@ -20,6 +20,8 @@ extern long __strncpy_from_user_nocheck_asm(char *__to,
 					    const char *__from, long __len);
 extern long __strncpy_from_user_asm(char *__to, const char *__from,
 				    long __len);
+extern long __strlen_kernel_nocheck_asm(const char *s);
+extern long __strlen_kernel_asm(const char *s);
 extern long __strlen_user_nocheck_asm(const char *s);
 extern long __strlen_user_asm(const char *s);
 extern long __strnlen_kernel_nocheck_asm(const char *s);
@@ -48,6 +50,8 @@ EXPORT_SYMBOL(__copy_user_inatomic);
 EXPORT_SYMBOL(__bzero);
 EXPORT_SYMBOL(__strncpy_from_user_nocheck_asm);
 EXPORT_SYMBOL(__strncpy_from_user_asm);
+EXPORT_SYMBOL(__strlen_kernel_nocheck_asm);
+EXPORT_SYMBOL(__strlen_kernel_asm);
 EXPORT_SYMBOL(__strlen_user_nocheck_asm);
 EXPORT_SYMBOL(__strlen_user_asm);
 EXPORT_SYMBOL(__strnlen_kernel_nocheck_asm);
diff --git a/arch/mips/lib/strlen_user.S b/arch/mips/lib/strlen_user.S
index 6e8bdb3..bef65c9 100644
--- a/arch/mips/lib/strlen_user.S
+++ b/arch/mips/lib/strlen_user.S
@@ -30,7 +30,11 @@ LEAF(__strlen_\func\()_asm)
 
 FEXPORT(__strlen_\func\()_nocheck_asm)
 	move		v0, a0
+.ifeqs "\func", "kernel"
 1:	EX(lbu, v1, (v0), .Lfault\@)
+.else
+1:	EX(lbue, v1, (v0), .Lfault\@)
+.endif
 	PTR_ADDIU	v0, 1
 	bnez		v1, 1b
 	PTR_SUBU	v0, a0
@@ -41,4 +45,20 @@ FEXPORT(__strlen_\func\()_nocheck_asm)
 	jr		ra
 	.endm
 
+#ifndef CONFIG_EVA
+	/* Set aliases */
+	.global __strlen_user_asm
+	.global __strlen_user_nocheck_asm
+	.set __strlen_user_asm, __strlen_kernel_asm
+	.set __strlen_user_nocheck_asm, __strlen_kernel_nocheck_asm
+#endif
+
+__BUILD_STRLEN_ASM kernel
+
+#ifdef CONFIG_EVA
+
+	.set push
+	.set eva
 __BUILD_STRLEN_ASM user
+	.set pop
+#endif
-- 
1.8.5.3

[PATCH 14/58] MIPS: lib: strncpy_user: Use macro to build the strncpy_from_user symbol

[Markos Chandras]

Build the __strncpy_from_user symbol using a macro. In EVA mode we will
need to use similar code to do the userspace load operations so
it is better if we use a macro to avoid code duplications.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/strncpy_user.S | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/arch/mips/lib/strncpy_user.S b/arch/mips/lib/strncpy_user.S
index 92870b6..51b38ab 100644
--- a/arch/mips/lib/strncpy_user.S
+++ b/arch/mips/lib/strncpy_user.S
@@ -28,16 +28,17 @@
  * it happens at most some bytes of the exceptions handlers will be copied.
  */
 
-LEAF(__strncpy_from_user_asm)
+	.macro __BUILD_STRNCPY_ASM func
+LEAF(__strncpy_from_\func\()_asm)
 	LONG_L		v0, TI_ADDR_LIMIT($28)	# pointer ok?
 	and		v0, a1
-	bnez		v0, .Lfault
+	bnez		v0, .Lfault\@
 
-FEXPORT(__strncpy_from_user_nocheck_asm)
+FEXPORT(__strncpy_from_\func\()_nocheck_asm)
 	.set		noreorder
 	move		t0, zero
 	move		v1, a1
-1:	EX(lbu, v0, (v1), .Lfault)
+1:	EX(lbu, v0, (v1), .Lfault\@)
 	PTR_ADDIU	v1, 1
 	R10KCBARRIER(0(ra))
 	beqz		v0, 2f
@@ -47,15 +48,19 @@ FEXPORT(__strncpy_from_user_nocheck_asm)
 	 PTR_ADDIU	a0, 1
 2:	PTR_ADDU	v0, a1, t0
 	xor		v0, a1
-	bltz		v0, .Lfault
+	bltz		v0, .Lfault\@
 	 nop
 	jr		ra			# return n
 	 move		v0, t0
-	END(__strncpy_from_user_asm)
+	END(__strncpy_from_\func\()_asm)
 
-.Lfault: jr		ra
+.Lfault\@: jr		ra
 	  li		v0, -EFAULT
 
 	.section	__ex_table,"a"
-	PTR		1b, .Lfault
+	PTR		1b, .Lfault\@
 	.previous
+
+	.endm
+
+__BUILD_STRNCPY_ASM user
-- 
1.8.5.3

[PATCH 14/58] MIPS: lib: strncpy_user: Use macro to build the strncpy_from_user symbol

[Markos Chandras]

Build the __strncpy_from_user symbol using a macro. In EVA mode we will
need to use similar code to do the userspace load operations so
it is better if we use a macro to avoid code duplications.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/strncpy_user.S | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/arch/mips/lib/strncpy_user.S b/arch/mips/lib/strncpy_user.S
index 92870b6..51b38ab 100644
--- a/arch/mips/lib/strncpy_user.S
+++ b/arch/mips/lib/strncpy_user.S
@@ -28,16 +28,17 @@
  * it happens at most some bytes of the exceptions handlers will be copied.
  */
 
-LEAF(__strncpy_from_user_asm)
+	.macro __BUILD_STRNCPY_ASM func
+LEAF(__strncpy_from_\func\()_asm)
 	LONG_L		v0, TI_ADDR_LIMIT($28)	# pointer ok?
 	and		v0, a1
-	bnez		v0, .Lfault
+	bnez		v0, .Lfault\@
 
-FEXPORT(__strncpy_from_user_nocheck_asm)
+FEXPORT(__strncpy_from_\func\()_nocheck_asm)
 	.set		noreorder
 	move		t0, zero
 	move		v1, a1
-1:	EX(lbu, v0, (v1), .Lfault)
+1:	EX(lbu, v0, (v1), .Lfault\@)
 	PTR_ADDIU	v1, 1
 	R10KCBARRIER(0(ra))
 	beqz		v0, 2f
@@ -47,15 +48,19 @@ FEXPORT(__strncpy_from_user_nocheck_asm)
 	 PTR_ADDIU	a0, 1
 2:	PTR_ADDU	v0, a1, t0
 	xor		v0, a1
-	bltz		v0, .Lfault
+	bltz		v0, .Lfault\@
 	 nop
 	jr		ra			# return n
 	 move		v0, t0
-	END(__strncpy_from_user_asm)
+	END(__strncpy_from_\func\()_asm)
 
-.Lfault: jr		ra
+.Lfault\@: jr		ra
 	  li		v0, -EFAULT
 
 	.section	__ex_table,"a"
-	PTR		1b, .Lfault
+	PTR		1b, .Lfault\@
 	.previous
+
+	.endm
+
+__BUILD_STRNCPY_ASM user
-- 
1.8.5.3

[PATCH 15/58] MIPS: lib: strncpy_user: Add EVA support

[Markos Chandras]

In non-EVA mode, strncpy_from_user* aliases are used for the
strncpy_from_kernel* symbols since the code is identical. In EVA
mode, new strcpy_from_user* symbols are used which use the EVA
specific instructions to load values from userspace.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/mips_ksyms.c |  6 ++++++
 arch/mips/lib/strncpy_user.S  | 19 +++++++++++++++++++
 2 files changed, 25 insertions(+)

diff --git a/arch/mips/kernel/mips_ksyms.c b/arch/mips/kernel/mips_ksyms.c
index 742ed7d..675bd05 100644
--- a/arch/mips/kernel/mips_ksyms.c
+++ b/arch/mips/kernel/mips_ksyms.c
@@ -16,6 +16,10 @@
 #include <asm/ftrace.h>
 
 extern void *__bzero(void *__s, size_t __count);
+extern long __strncpy_from_kernel_nocheck_asm(char *__to,
+					      const char *__from, long __len);
+extern long __strncpy_from_kernel_asm(char *__to, const char *__from,
+				      long __len);
 extern long __strncpy_from_user_nocheck_asm(char *__to,
 					    const char *__from, long __len);
 extern long __strncpy_from_user_asm(char *__to, const char *__from,
@@ -48,6 +52,8 @@ EXPORT_SYMBOL(copy_page);
 EXPORT_SYMBOL(__copy_user);
 EXPORT_SYMBOL(__copy_user_inatomic);
 EXPORT_SYMBOL(__bzero);
+EXPORT_SYMBOL(__strncpy_from_kernel_nocheck_asm);
+EXPORT_SYMBOL(__strncpy_from_kernel_asm);
 EXPORT_SYMBOL(__strncpy_from_user_nocheck_asm);
 EXPORT_SYMBOL(__strncpy_from_user_asm);
 EXPORT_SYMBOL(__strlen_kernel_nocheck_asm);
diff --git a/arch/mips/lib/strncpy_user.S b/arch/mips/lib/strncpy_user.S
index 51b38ab..d3301cd 100644
--- a/arch/mips/lib/strncpy_user.S
+++ b/arch/mips/lib/strncpy_user.S
@@ -38,7 +38,11 @@ FEXPORT(__strncpy_from_\func\()_nocheck_asm)
 	.set		noreorder
 	move		t0, zero
 	move		v1, a1
+.ifeqs "\func","kernel"
 1:	EX(lbu, v0, (v1), .Lfault\@)
+.else
+1:	EX(lbue, v0, (v1), .Lfault\@)
+.endif
 	PTR_ADDIU	v1, 1
 	R10KCBARRIER(0(ra))
 	beqz		v0, 2f
@@ -63,4 +67,19 @@ FEXPORT(__strncpy_from_\func\()_nocheck_asm)
 
 	.endm
 
+#ifndef CONFIG_EVA
+	/* Set aliases */
+	.global __strncpy_from_user_asm
+	.global __strncpy_from_user_nocheck_asm
+	.set __strncpy_from_user_asm, __strncpy_from_kernel_asm
+	.set __strncpy_from_user_nocheck_asm, __strncpy_from_kernel_nocheck_asm
+#endif
+
+__BUILD_STRNCPY_ASM kernel
+
+#ifdef CONFIG_EVA
+	.set push
+	.set eva
 __BUILD_STRNCPY_ASM user
+	.set pop
+#endif
-- 
1.8.5.3

[PATCH 15/58] MIPS: lib: strncpy_user: Add EVA support

[Markos Chandras]

In non-EVA mode, strncpy_from_user* aliases are used for the
strncpy_from_kernel* symbols since the code is identical. In EVA
mode, new strcpy_from_user* symbols are used which use the EVA
specific instructions to load values from userspace.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/mips_ksyms.c |  6 ++++++
 arch/mips/lib/strncpy_user.S  | 19 +++++++++++++++++++
 2 files changed, 25 insertions(+)

diff --git a/arch/mips/kernel/mips_ksyms.c b/arch/mips/kernel/mips_ksyms.c
index 742ed7d..675bd05 100644
--- a/arch/mips/kernel/mips_ksyms.c
+++ b/arch/mips/kernel/mips_ksyms.c
@@ -16,6 +16,10 @@
 #include <asm/ftrace.h>
 
 extern void *__bzero(void *__s, size_t __count);
+extern long __strncpy_from_kernel_nocheck_asm(char *__to,
+					      const char *__from, long __len);
+extern long __strncpy_from_kernel_asm(char *__to, const char *__from,
+				      long __len);
 extern long __strncpy_from_user_nocheck_asm(char *__to,
 					    const char *__from, long __len);
 extern long __strncpy_from_user_asm(char *__to, const char *__from,
@@ -48,6 +52,8 @@ EXPORT_SYMBOL(copy_page);
 EXPORT_SYMBOL(__copy_user);
 EXPORT_SYMBOL(__copy_user_inatomic);
 EXPORT_SYMBOL(__bzero);
+EXPORT_SYMBOL(__strncpy_from_kernel_nocheck_asm);
+EXPORT_SYMBOL(__strncpy_from_kernel_asm);
 EXPORT_SYMBOL(__strncpy_from_user_nocheck_asm);
 EXPORT_SYMBOL(__strncpy_from_user_asm);
 EXPORT_SYMBOL(__strlen_kernel_nocheck_asm);
diff --git a/arch/mips/lib/strncpy_user.S b/arch/mips/lib/strncpy_user.S
index 51b38ab..d3301cd 100644
--- a/arch/mips/lib/strncpy_user.S
+++ b/arch/mips/lib/strncpy_user.S
@@ -38,7 +38,11 @@ FEXPORT(__strncpy_from_\func\()_nocheck_asm)
 	.set		noreorder
 	move		t0, zero
 	move		v1, a1
+.ifeqs "\func","kernel"
 1:	EX(lbu, v0, (v1), .Lfault\@)
+.else
+1:	EX(lbue, v0, (v1), .Lfault\@)
+.endif
 	PTR_ADDIU	v1, 1
 	R10KCBARRIER(0(ra))
 	beqz		v0, 2f
@@ -63,4 +67,19 @@ FEXPORT(__strncpy_from_\func\()_nocheck_asm)
 
 	.endm
 
+#ifndef CONFIG_EVA
+	/* Set aliases */
+	.global __strncpy_from_user_asm
+	.global __strncpy_from_user_nocheck_asm
+	.set __strncpy_from_user_asm, __strncpy_from_kernel_asm
+	.set __strncpy_from_user_nocheck_asm, __strncpy_from_kernel_nocheck_asm
+#endif
+
+__BUILD_STRNCPY_ASM kernel
+
+#ifdef CONFIG_EVA
+	.set push
+	.set eva
 __BUILD_STRNCPY_ASM user
+	.set pop
+#endif
-- 
1.8.5.3

[PATCH 16/58] MIPS: lib: memcpy: Merge EXC and load/store macros

[Markos Chandras]

Each load/store macro always adds an entry to the __ex_table
using the EXC macro. Therefore, these load/store macros are now merged
with the EXC one. The argument list is also expanded in order to make
the macro more flexible. The extra 'type' argument is not used by this
commit, but it will be used when the EVA support is added to the memcpy.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/memcpy.S | 158 ++++++++++++++++++++++++++++---------------------
 1 file changed, 89 insertions(+), 69 deletions(-)

diff --git a/arch/mips/lib/memcpy.S b/arch/mips/lib/memcpy.S
index c5c40da..ab9f046 100644
--- a/arch/mips/lib/memcpy.S
+++ b/arch/mips/lib/memcpy.S
@@ -10,6 +10,7 @@
  * Copyright (C) 2002 Broadcom, Inc.
  *   memcpy/copy_user author: Mark Vandevoorde
  * Copyright (C) 2007  Maciej W. Rozycki
+ * Copyright (C) 2014 Imagination Technologies Ltd.
  *
  * Mnemonic names for arguments to memcpy/__copy_user
  */
@@ -85,8 +86,22 @@
  * they're not protected.
  */
 
-#define EXC(inst_reg,addr,handler)		\
-9:	inst_reg, addr;				\
+/* Instruction type */
+#define LD_INSN 1
+#define ST_INSN 2
+
+/*
+ * Wrapper to add an entry in the exception table
+ * in case the insn causes a memory exception.
+ * Arguments:
+ * insn    : Load/store instruction
+ * type    : Instruction type
+ * reg     : Register
+ * addr    : Address
+ * handler : Exception handler
+ */
+#define EXC(insn, type, reg, addr, handler)	\
+9:	insn reg, addr;				\
 	.section __ex_table,"a";		\
 	PTR	9b, handler;			\
 	.previous
@@ -100,12 +115,13 @@
 
 #ifdef USE_DOUBLE
 
-#define LOAD   ld
-#define LOADL  ldl
-#define LOADR  ldr
-#define STOREL sdl
-#define STORER sdr
-#define STORE  sd
+#define LOADK ld /* No exception */
+#define LOAD(reg, addr, handler)	EXC(ld, LD_INSN, reg, addr, handler)
+#define LOADL(reg, addr, handler)	EXC(ldl, LD_INSN, reg, addr, handler)
+#define LOADR(reg, addr, handler)	EXC(ldr, LD_INSN, reg, addr, handler)
+#define STOREL(reg, addr, handler)	EXC(sdl, ST_INSN, reg, addr, handler)
+#define STORER(reg, addr, handler)	EXC(sdr, ST_INSN, reg, addr, handler)
+#define STORE(reg, addr, handler)	EXC(sd, ST_INSN, reg, addr, handler)
 #define ADD    daddu
 #define SUB    dsubu
 #define SRL    dsrl
@@ -136,12 +152,13 @@
 
 #else
 
-#define LOAD   lw
-#define LOADL  lwl
-#define LOADR  lwr
-#define STOREL swl
-#define STORER swr
-#define STORE  sw
+#define LOADK lw /* No exception */
+#define LOAD(reg, addr, handler)	EXC(lw, LD_INSN, reg, addr, handler)
+#define LOADL(reg, addr, handler)	EXC(lwl, LD_INSN, reg, addr, handler)
+#define LOADR(reg, addr, handler)	EXC(lwr, LD_INSN, reg, addr, handler)
+#define STOREL(reg, addr, handler)	EXC(swl, ST_INSN, reg, addr, handler)
+#define STORER(reg, addr, handler)	EXC(swr, ST_INSN, reg, addr, handler)
+#define STORE(reg, addr, handler)	EXC(sw, ST_INSN, reg, addr, handler)
 #define ADD    addu
 #define SUB    subu
 #define SRL    srl
@@ -154,6 +171,9 @@
 
 #endif /* USE_DOUBLE */
 
+#define LOADB(reg, addr, handler)	EXC(lb, LD_INSN, reg, addr, handler)
+#define STOREB(reg, addr, handler)	EXC(sb, ST_INSN, reg, addr, handler)
+
 #ifdef CONFIG_CPU_LITTLE_ENDIAN
 #define LDFIRST LOADR
 #define LDREST	LOADL
@@ -243,25 +263,25 @@ __copy_user_common:
 	.align	4
 1:
 	R10KCBARRIER(0(ra))
-EXC(	LOAD	t0, UNIT(0)(src),	.Ll_exc)
-EXC(	LOAD	t1, UNIT(1)(src),	.Ll_exc_copy)
-EXC(	LOAD	t2, UNIT(2)(src),	.Ll_exc_copy)
-EXC(	LOAD	t3, UNIT(3)(src),	.Ll_exc_copy)
+	LOAD(t0, UNIT(0)(src), .Ll_exc)
+	LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
+	LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
+	LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
 	SUB	len, len, 8*NBYTES
-EXC(	LOAD	t4, UNIT(4)(src),	.Ll_exc_copy)
-EXC(	LOAD	t7, UNIT(5)(src),	.Ll_exc_copy)
-EXC(	STORE	t0, UNIT(0)(dst),	.Ls_exc_p8u)
-EXC(	STORE	t1, UNIT(1)(dst),	.Ls_exc_p7u)
-EXC(	LOAD	t0, UNIT(6)(src),	.Ll_exc_copy)
-EXC(	LOAD	t1, UNIT(7)(src),	.Ll_exc_copy)
+	LOAD(t4, UNIT(4)(src), .Ll_exc_copy)
+	LOAD(t7, UNIT(5)(src), .Ll_exc_copy)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc_p8u)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc_p7u)
+	LOAD(t0, UNIT(6)(src), .Ll_exc_copy)
+	LOAD(t1, UNIT(7)(src), .Ll_exc_copy)
 	ADD	src, src, 8*NBYTES
 	ADD	dst, dst, 8*NBYTES
-EXC(	STORE	t2, UNIT(-6)(dst),	.Ls_exc_p6u)
-EXC(	STORE	t3, UNIT(-5)(dst),	.Ls_exc_p5u)
-EXC(	STORE	t4, UNIT(-4)(dst),	.Ls_exc_p4u)
-EXC(	STORE	t7, UNIT(-3)(dst),	.Ls_exc_p3u)
-EXC(	STORE	t0, UNIT(-2)(dst),	.Ls_exc_p2u)
-EXC(	STORE	t1, UNIT(-1)(dst),	.Ls_exc_p1u)
+	STORE(t2, UNIT(-6)(dst), .Ls_exc_p6u)
+	STORE(t3, UNIT(-5)(dst), .Ls_exc_p5u)
+	STORE(t4, UNIT(-4)(dst), .Ls_exc_p4u)
+	STORE(t7, UNIT(-3)(dst), .Ls_exc_p3u)
+	STORE(t0, UNIT(-2)(dst), .Ls_exc_p2u)
+	STORE(t1, UNIT(-1)(dst), .Ls_exc_p1u)
 	PREF(	0, 8*32(src) )
 	PREF(	1, 8*32(dst) )
 	bne	len, rem, 1b
@@ -278,17 +298,17 @@ EXC(	STORE	t1, UNIT(-1)(dst),	.Ls_exc_p1u)
 	/*
 	 * len >= 4*NBYTES
 	 */
-EXC(	LOAD	t0, UNIT(0)(src),	.Ll_exc)
-EXC(	LOAD	t1, UNIT(1)(src),	.Ll_exc_copy)
-EXC(	LOAD	t2, UNIT(2)(src),	.Ll_exc_copy)
-EXC(	LOAD	t3, UNIT(3)(src),	.Ll_exc_copy)
+	LOAD( t0, UNIT(0)(src),	.Ll_exc)
+	LOAD( t1, UNIT(1)(src),	.Ll_exc_copy)
+	LOAD( t2, UNIT(2)(src),	.Ll_exc_copy)
+	LOAD( t3, UNIT(3)(src),	.Ll_exc_copy)
 	SUB	len, len, 4*NBYTES
 	ADD	src, src, 4*NBYTES
 	R10KCBARRIER(0(ra))
-EXC(	STORE	t0, UNIT(0)(dst),	.Ls_exc_p4u)
-EXC(	STORE	t1, UNIT(1)(dst),	.Ls_exc_p3u)
-EXC(	STORE	t2, UNIT(2)(dst),	.Ls_exc_p2u)
-EXC(	STORE	t3, UNIT(3)(dst),	.Ls_exc_p1u)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc_p4u)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc_p3u)
+	STORE(t2, UNIT(2)(dst),	.Ls_exc_p2u)
+	STORE(t3, UNIT(3)(dst),	.Ls_exc_p1u)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 4*NBYTES
 	beqz	len, .Ldone
@@ -301,10 +321,10 @@ EXC(	STORE	t3, UNIT(3)(dst),	.Ls_exc_p1u)
 	 nop
 1:
 	R10KCBARRIER(0(ra))
-EXC(	LOAD	t0, 0(src),		.Ll_exc)
+	LOAD(t0, 0(src), .Ll_exc)
 	ADD	src, src, NBYTES
 	SUB	len, len, NBYTES
-EXC(	STORE	t0, 0(dst),		.Ls_exc_p1u)
+	STORE(t0, 0(dst), .Ls_exc_p1u)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, NBYTES
 	bne	rem, len, 1b
@@ -326,10 +346,10 @@ EXC(	STORE	t0, 0(dst),		.Ls_exc_p1u)
 	 ADD	t1, dst, len	# t1 is just past last byte of dst
 	li	bits, 8*NBYTES
 	SLL	rem, len, 3	# rem = number of bits to keep
-EXC(	LOAD	t0, 0(src),		.Ll_exc)
+	LOAD(t0, 0(src), .Ll_exc)
 	SUB	bits, bits, rem # bits = number of bits to discard
 	SHIFT_DISCARD t0, t0, bits
-EXC(	STREST	t0, -1(t1),		.Ls_exc)
+	STREST(t0, -1(t1), .Ls_exc)
 	jr	ra
 	 move	len, zero
 .Ldst_unaligned:
@@ -343,13 +363,13 @@ EXC(	STREST	t0, -1(t1),		.Ls_exc)
 	 * Set match = (src and dst have same alignment)
 	 */
 #define match rem
-EXC(	LDFIRST t3, FIRST(0)(src),	.Ll_exc)
+	LDFIRST(t3, FIRST(0)(src), .Ll_exc)
 	ADD	t2, zero, NBYTES
-EXC(	LDREST	t3, REST(0)(src),	.Ll_exc_copy)
+	LDREST(t3, REST(0)(src), .Ll_exc_copy)
 	SUB	t2, t2, t1	# t2 = number of bytes copied
 	xor	match, t0, t1
 	R10KCBARRIER(0(ra))
-EXC(	STFIRST t3, FIRST(0)(dst),	.Ls_exc)
+	STFIRST(t3, FIRST(0)(dst), .Ls_exc)
 	beq	len, t2, .Ldone
 	 SUB	len, len, t2
 	ADD	dst, dst, t2
@@ -370,24 +390,24 @@ EXC(	STFIRST t3, FIRST(0)(dst),	.Ls_exc)
  * are to the same unit (unless src is aligned, but it's not).
  */
 	R10KCBARRIER(0(ra))
-EXC(	LDFIRST t0, FIRST(0)(src),	.Ll_exc)
-EXC(	LDFIRST t1, FIRST(1)(src),	.Ll_exc_copy)
+	LDFIRST(t0, FIRST(0)(src), .Ll_exc)
+	LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy)
 	SUB	len, len, 4*NBYTES
-EXC(	LDREST	t0, REST(0)(src),	.Ll_exc_copy)
-EXC(	LDREST	t1, REST(1)(src),	.Ll_exc_copy)
-EXC(	LDFIRST t2, FIRST(2)(src),	.Ll_exc_copy)
-EXC(	LDFIRST t3, FIRST(3)(src),	.Ll_exc_copy)
-EXC(	LDREST	t2, REST(2)(src),	.Ll_exc_copy)
-EXC(	LDREST	t3, REST(3)(src),	.Ll_exc_copy)
+	LDREST(t0, REST(0)(src), .Ll_exc_copy)
+	LDREST(t1, REST(1)(src), .Ll_exc_copy)
+	LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy)
+	LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy)
+	LDREST(t2, REST(2)(src), .Ll_exc_copy)
+	LDREST(t3, REST(3)(src), .Ll_exc_copy)
 	PREF(	0, 9*32(src) )		# 0 is PREF_LOAD  (not streamed)
 	ADD	src, src, 4*NBYTES
 #ifdef CONFIG_CPU_SB1
 	nop				# improves slotting
 #endif
-EXC(	STORE	t0, UNIT(0)(dst),	.Ls_exc_p4u)
-EXC(	STORE	t1, UNIT(1)(dst),	.Ls_exc_p3u)
-EXC(	STORE	t2, UNIT(2)(dst),	.Ls_exc_p2u)
-EXC(	STORE	t3, UNIT(3)(dst),	.Ls_exc_p1u)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc_p4u)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc_p3u)
+	STORE(t2, UNIT(2)(dst),	.Ls_exc_p2u)
+	STORE(t3, UNIT(3)(dst),	.Ls_exc_p1u)
 	PREF(	1, 9*32(dst) )		# 1 is PREF_STORE (not streamed)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 4*NBYTES
@@ -401,11 +421,11 @@ EXC(	STORE	t3, UNIT(3)(dst),	.Ls_exc_p1u)
 	 nop
 1:
 	R10KCBARRIER(0(ra))
-EXC(	LDFIRST t0, FIRST(0)(src),	.Ll_exc)
-EXC(	LDREST	t0, REST(0)(src),	.Ll_exc_copy)
+	LDFIRST(t0, FIRST(0)(src), .Ll_exc)
+	LDREST(t0, REST(0)(src), .Ll_exc_copy)
 	ADD	src, src, NBYTES
 	SUB	len, len, NBYTES
-EXC(	STORE	t0, 0(dst),		.Ls_exc_p1u)
+	STORE(t0, 0(dst), .Ls_exc_p1u)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, NBYTES
 	bne	len, rem, 1b
@@ -418,10 +438,10 @@ EXC(	STORE	t0, 0(dst),		.Ls_exc_p1u)
 	/* 0 < len < NBYTES  */
 	R10KCBARRIER(0(ra))
 #define COPY_BYTE(N)			\
-EXC(	lb	t0, N(src), .Ll_exc);	\
+	LOADB(t0, N(src), .Ll_exc);	\
 	SUB	len, len, 1;		\
 	beqz	len, .Ldone;		\
-EXC(	 sb	t0, N(dst), .Ls_exc_p1)
+	STOREB(t0, N(dst), .Ls_exc_p1)
 
 	COPY_BYTE(0)
 	COPY_BYTE(1)
@@ -431,10 +451,10 @@ EXC(	 sb	t0, N(dst), .Ls_exc_p1)
 	COPY_BYTE(4)
 	COPY_BYTE(5)
 #endif
-EXC(	lb	t0, NBYTES-2(src), .Ll_exc)
+	LOADB(t0, NBYTES-2(src), .Ll_exc)
 	SUB	len, len, 1
 	jr	ra
-EXC(	 sb	t0, NBYTES-2(dst), .Ls_exc_p1)
+	STOREB(t0, NBYTES-2(dst), .Ls_exc_p1)
 .Ldone:
 	jr	ra
 	 nop
@@ -451,11 +471,11 @@ EXC(	 sb	t0, NBYTES-2(dst), .Ls_exc_p1)
 	 *
 	 * Assumes src < THREAD_BUADDR($28)
 	 */
-	LOAD	t0, TI_TASK($28)
+	LOADK	t0, TI_TASK($28)
 	 nop
-	LOAD	t0, THREAD_BUADDR(t0)
+	LOADK	t0, THREAD_BUADDR(t0)
 1:
-EXC(	lb	t1, 0(src),	.Ll_exc)
+	LOADB(t1, 0(src), .Ll_exc)
 	ADD	src, src, 1
 	sb	t1, 0(dst)	# can't fault -- we're copy_from_user
 	.set	reorder				/* DADDI_WAR */
@@ -463,9 +483,9 @@ EXC(	lb	t1, 0(src),	.Ll_exc)
 	bne	src, t0, 1b
 	.set	noreorder
 .Ll_exc:
-	LOAD	t0, TI_TASK($28)
+	LOADK	t0, TI_TASK($28)
 	 nop
-	LOAD	t0, THREAD_BUADDR(t0)	# t0 is just past last good address
+	LOADK	t0, THREAD_BUADDR(t0)	# t0 is just past last good address
 	 nop
 	SUB	len, AT, t0		# len number of uncopied bytes
 	bnez	t6, .Ldone	/* Skip the zeroing part if inatomic */
-- 
1.8.5.3

[PATCH 16/58] MIPS: lib: memcpy: Merge EXC and load/store macros

[Markos Chandras]

Each load/store macro always adds an entry to the __ex_table
using the EXC macro. Therefore, these load/store macros are now merged
with the EXC one. The argument list is also expanded in order to make
the macro more flexible. The extra 'type' argument is not used by this
commit, but it will be used when the EVA support is added to the memcpy.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/memcpy.S | 158 ++++++++++++++++++++++++++++---------------------
 1 file changed, 89 insertions(+), 69 deletions(-)

diff --git a/arch/mips/lib/memcpy.S b/arch/mips/lib/memcpy.S
index c5c40da..ab9f046 100644
--- a/arch/mips/lib/memcpy.S
+++ b/arch/mips/lib/memcpy.S
@@ -10,6 +10,7 @@
  * Copyright (C) 2002 Broadcom, Inc.
  *   memcpy/copy_user author: Mark Vandevoorde
  * Copyright (C) 2007  Maciej W. Rozycki
+ * Copyright (C) 2014 Imagination Technologies Ltd.
  *
  * Mnemonic names for arguments to memcpy/__copy_user
  */
@@ -85,8 +86,22 @@
  * they're not protected.
  */
 
-#define EXC(inst_reg,addr,handler)		\
-9:	inst_reg, addr;				\
+/* Instruction type */
+#define LD_INSN 1
+#define ST_INSN 2
+
+/*
+ * Wrapper to add an entry in the exception table
+ * in case the insn causes a memory exception.
+ * Arguments:
+ * insn    : Load/store instruction
+ * type    : Instruction type
+ * reg     : Register
+ * addr    : Address
+ * handler : Exception handler
+ */
+#define EXC(insn, type, reg, addr, handler)	\
+9:	insn reg, addr;				\
 	.section __ex_table,"a";		\
 	PTR	9b, handler;			\
 	.previous
@@ -100,12 +115,13 @@
 
 #ifdef USE_DOUBLE
 
-#define LOAD   ld
-#define LOADL  ldl
-#define LOADR  ldr
-#define STOREL sdl
-#define STORER sdr
-#define STORE  sd
+#define LOADK ld /* No exception */
+#define LOAD(reg, addr, handler)	EXC(ld, LD_INSN, reg, addr, handler)
+#define LOADL(reg, addr, handler)	EXC(ldl, LD_INSN, reg, addr, handler)
+#define LOADR(reg, addr, handler)	EXC(ldr, LD_INSN, reg, addr, handler)
+#define STOREL(reg, addr, handler)	EXC(sdl, ST_INSN, reg, addr, handler)
+#define STORER(reg, addr, handler)	EXC(sdr, ST_INSN, reg, addr, handler)
+#define STORE(reg, addr, handler)	EXC(sd, ST_INSN, reg, addr, handler)
 #define ADD    daddu
 #define SUB    dsubu
 #define SRL    dsrl
@@ -136,12 +152,13 @@
 
 #else
 
-#define LOAD   lw
-#define LOADL  lwl
-#define LOADR  lwr
-#define STOREL swl
-#define STORER swr
-#define STORE  sw
+#define LOADK lw /* No exception */
+#define LOAD(reg, addr, handler)	EXC(lw, LD_INSN, reg, addr, handler)
+#define LOADL(reg, addr, handler)	EXC(lwl, LD_INSN, reg, addr, handler)
+#define LOADR(reg, addr, handler)	EXC(lwr, LD_INSN, reg, addr, handler)
+#define STOREL(reg, addr, handler)	EXC(swl, ST_INSN, reg, addr, handler)
+#define STORER(reg, addr, handler)	EXC(swr, ST_INSN, reg, addr, handler)
+#define STORE(reg, addr, handler)	EXC(sw, ST_INSN, reg, addr, handler)
 #define ADD    addu
 #define SUB    subu
 #define SRL    srl
@@ -154,6 +171,9 @@
 
 #endif /* USE_DOUBLE */
 
+#define LOADB(reg, addr, handler)	EXC(lb, LD_INSN, reg, addr, handler)
+#define STOREB(reg, addr, handler)	EXC(sb, ST_INSN, reg, addr, handler)
+
 #ifdef CONFIG_CPU_LITTLE_ENDIAN
 #define LDFIRST LOADR
 #define LDREST	LOADL
@@ -243,25 +263,25 @@ __copy_user_common:
 	.align	4
 1:
 	R10KCBARRIER(0(ra))
-EXC(	LOAD	t0, UNIT(0)(src),	.Ll_exc)
-EXC(	LOAD	t1, UNIT(1)(src),	.Ll_exc_copy)
-EXC(	LOAD	t2, UNIT(2)(src),	.Ll_exc_copy)
-EXC(	LOAD	t3, UNIT(3)(src),	.Ll_exc_copy)
+	LOAD(t0, UNIT(0)(src), .Ll_exc)
+	LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
+	LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
+	LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
 	SUB	len, len, 8*NBYTES
-EXC(	LOAD	t4, UNIT(4)(src),	.Ll_exc_copy)
-EXC(	LOAD	t7, UNIT(5)(src),	.Ll_exc_copy)
-EXC(	STORE	t0, UNIT(0)(dst),	.Ls_exc_p8u)
-EXC(	STORE	t1, UNIT(1)(dst),	.Ls_exc_p7u)
-EXC(	LOAD	t0, UNIT(6)(src),	.Ll_exc_copy)
-EXC(	LOAD	t1, UNIT(7)(src),	.Ll_exc_copy)
+	LOAD(t4, UNIT(4)(src), .Ll_exc_copy)
+	LOAD(t7, UNIT(5)(src), .Ll_exc_copy)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc_p8u)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc_p7u)
+	LOAD(t0, UNIT(6)(src), .Ll_exc_copy)
+	LOAD(t1, UNIT(7)(src), .Ll_exc_copy)
 	ADD	src, src, 8*NBYTES
 	ADD	dst, dst, 8*NBYTES
-EXC(	STORE	t2, UNIT(-6)(dst),	.Ls_exc_p6u)
-EXC(	STORE	t3, UNIT(-5)(dst),	.Ls_exc_p5u)
-EXC(	STORE	t4, UNIT(-4)(dst),	.Ls_exc_p4u)
-EXC(	STORE	t7, UNIT(-3)(dst),	.Ls_exc_p3u)
-EXC(	STORE	t0, UNIT(-2)(dst),	.Ls_exc_p2u)
-EXC(	STORE	t1, UNIT(-1)(dst),	.Ls_exc_p1u)
+	STORE(t2, UNIT(-6)(dst), .Ls_exc_p6u)
+	STORE(t3, UNIT(-5)(dst), .Ls_exc_p5u)
+	STORE(t4, UNIT(-4)(dst), .Ls_exc_p4u)
+	STORE(t7, UNIT(-3)(dst), .Ls_exc_p3u)
+	STORE(t0, UNIT(-2)(dst), .Ls_exc_p2u)
+	STORE(t1, UNIT(-1)(dst), .Ls_exc_p1u)
 	PREF(	0, 8*32(src) )
 	PREF(	1, 8*32(dst) )
 	bne	len, rem, 1b
@@ -278,17 +298,17 @@ EXC(	STORE	t1, UNIT(-1)(dst),	.Ls_exc_p1u)
 	/*
 	 * len >= 4*NBYTES
 	 */
-EXC(	LOAD	t0, UNIT(0)(src),	.Ll_exc)
-EXC(	LOAD	t1, UNIT(1)(src),	.Ll_exc_copy)
-EXC(	LOAD	t2, UNIT(2)(src),	.Ll_exc_copy)
-EXC(	LOAD	t3, UNIT(3)(src),	.Ll_exc_copy)
+	LOAD( t0, UNIT(0)(src),	.Ll_exc)
+	LOAD( t1, UNIT(1)(src),	.Ll_exc_copy)
+	LOAD( t2, UNIT(2)(src),	.Ll_exc_copy)
+	LOAD( t3, UNIT(3)(src),	.Ll_exc_copy)
 	SUB	len, len, 4*NBYTES
 	ADD	src, src, 4*NBYTES
 	R10KCBARRIER(0(ra))
-EXC(	STORE	t0, UNIT(0)(dst),	.Ls_exc_p4u)
-EXC(	STORE	t1, UNIT(1)(dst),	.Ls_exc_p3u)
-EXC(	STORE	t2, UNIT(2)(dst),	.Ls_exc_p2u)
-EXC(	STORE	t3, UNIT(3)(dst),	.Ls_exc_p1u)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc_p4u)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc_p3u)
+	STORE(t2, UNIT(2)(dst),	.Ls_exc_p2u)
+	STORE(t3, UNIT(3)(dst),	.Ls_exc_p1u)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 4*NBYTES
 	beqz	len, .Ldone
@@ -301,10 +321,10 @@ EXC(	STORE	t3, UNIT(3)(dst),	.Ls_exc_p1u)
 	 nop
 1:
 	R10KCBARRIER(0(ra))
-EXC(	LOAD	t0, 0(src),		.Ll_exc)
+	LOAD(t0, 0(src), .Ll_exc)
 	ADD	src, src, NBYTES
 	SUB	len, len, NBYTES
-EXC(	STORE	t0, 0(dst),		.Ls_exc_p1u)
+	STORE(t0, 0(dst), .Ls_exc_p1u)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, NBYTES
 	bne	rem, len, 1b
@@ -326,10 +346,10 @@ EXC(	STORE	t0, 0(dst),		.Ls_exc_p1u)
 	 ADD	t1, dst, len	# t1 is just past last byte of dst
 	li	bits, 8*NBYTES
 	SLL	rem, len, 3	# rem = number of bits to keep
-EXC(	LOAD	t0, 0(src),		.Ll_exc)
+	LOAD(t0, 0(src), .Ll_exc)
 	SUB	bits, bits, rem # bits = number of bits to discard
 	SHIFT_DISCARD t0, t0, bits
-EXC(	STREST	t0, -1(t1),		.Ls_exc)
+	STREST(t0, -1(t1), .Ls_exc)
 	jr	ra
 	 move	len, zero
 .Ldst_unaligned:
@@ -343,13 +363,13 @@ EXC(	STREST	t0, -1(t1),		.Ls_exc)
 	 * Set match = (src and dst have same alignment)
 	 */
 #define match rem
-EXC(	LDFIRST t3, FIRST(0)(src),	.Ll_exc)
+	LDFIRST(t3, FIRST(0)(src), .Ll_exc)
 	ADD	t2, zero, NBYTES
-EXC(	LDREST	t3, REST(0)(src),	.Ll_exc_copy)
+	LDREST(t3, REST(0)(src), .Ll_exc_copy)
 	SUB	t2, t2, t1	# t2 = number of bytes copied
 	xor	match, t0, t1
 	R10KCBARRIER(0(ra))
-EXC(	STFIRST t3, FIRST(0)(dst),	.Ls_exc)
+	STFIRST(t3, FIRST(0)(dst), .Ls_exc)
 	beq	len, t2, .Ldone
 	 SUB	len, len, t2
 	ADD	dst, dst, t2
@@ -370,24 +390,24 @@ EXC(	STFIRST t3, FIRST(0)(dst),	.Ls_exc)
  * are to the same unit (unless src is aligned, but it's not).
  */
 	R10KCBARRIER(0(ra))
-EXC(	LDFIRST t0, FIRST(0)(src),	.Ll_exc)
-EXC(	LDFIRST t1, FIRST(1)(src),	.Ll_exc_copy)
+	LDFIRST(t0, FIRST(0)(src), .Ll_exc)
+	LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy)
 	SUB	len, len, 4*NBYTES
-EXC(	LDREST	t0, REST(0)(src),	.Ll_exc_copy)
-EXC(	LDREST	t1, REST(1)(src),	.Ll_exc_copy)
-EXC(	LDFIRST t2, FIRST(2)(src),	.Ll_exc_copy)
-EXC(	LDFIRST t3, FIRST(3)(src),	.Ll_exc_copy)
-EXC(	LDREST	t2, REST(2)(src),	.Ll_exc_copy)
-EXC(	LDREST	t3, REST(3)(src),	.Ll_exc_copy)
+	LDREST(t0, REST(0)(src), .Ll_exc_copy)
+	LDREST(t1, REST(1)(src), .Ll_exc_copy)
+	LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy)
+	LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy)
+	LDREST(t2, REST(2)(src), .Ll_exc_copy)
+	LDREST(t3, REST(3)(src), .Ll_exc_copy)
 	PREF(	0, 9*32(src) )		# 0 is PREF_LOAD  (not streamed)
 	ADD	src, src, 4*NBYTES
 #ifdef CONFIG_CPU_SB1
 	nop				# improves slotting
 #endif
-EXC(	STORE	t0, UNIT(0)(dst),	.Ls_exc_p4u)
-EXC(	STORE	t1, UNIT(1)(dst),	.Ls_exc_p3u)
-EXC(	STORE	t2, UNIT(2)(dst),	.Ls_exc_p2u)
-EXC(	STORE	t3, UNIT(3)(dst),	.Ls_exc_p1u)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc_p4u)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc_p3u)
+	STORE(t2, UNIT(2)(dst),	.Ls_exc_p2u)
+	STORE(t3, UNIT(3)(dst),	.Ls_exc_p1u)
 	PREF(	1, 9*32(dst) )		# 1 is PREF_STORE (not streamed)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 4*NBYTES
@@ -401,11 +421,11 @@ EXC(	STORE	t3, UNIT(3)(dst),	.Ls_exc_p1u)
 	 nop
 1:
 	R10KCBARRIER(0(ra))
-EXC(	LDFIRST t0, FIRST(0)(src),	.Ll_exc)
-EXC(	LDREST	t0, REST(0)(src),	.Ll_exc_copy)
+	LDFIRST(t0, FIRST(0)(src), .Ll_exc)
+	LDREST(t0, REST(0)(src), .Ll_exc_copy)
 	ADD	src, src, NBYTES
 	SUB	len, len, NBYTES
-EXC(	STORE	t0, 0(dst),		.Ls_exc_p1u)
+	STORE(t0, 0(dst), .Ls_exc_p1u)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, NBYTES
 	bne	len, rem, 1b
@@ -418,10 +438,10 @@ EXC(	STORE	t0, 0(dst),		.Ls_exc_p1u)
 	/* 0 < len < NBYTES  */
 	R10KCBARRIER(0(ra))
 #define COPY_BYTE(N)			\
-EXC(	lb	t0, N(src), .Ll_exc);	\
+	LOADB(t0, N(src), .Ll_exc);	\
 	SUB	len, len, 1;		\
 	beqz	len, .Ldone;		\
-EXC(	 sb	t0, N(dst), .Ls_exc_p1)
+	STOREB(t0, N(dst), .Ls_exc_p1)
 
 	COPY_BYTE(0)
 	COPY_BYTE(1)
@@ -431,10 +451,10 @@ EXC(	 sb	t0, N(dst), .Ls_exc_p1)
 	COPY_BYTE(4)
 	COPY_BYTE(5)
 #endif
-EXC(	lb	t0, NBYTES-2(src), .Ll_exc)
+	LOADB(t0, NBYTES-2(src), .Ll_exc)
 	SUB	len, len, 1
 	jr	ra
-EXC(	 sb	t0, NBYTES-2(dst), .Ls_exc_p1)
+	STOREB(t0, NBYTES-2(dst), .Ls_exc_p1)
 .Ldone:
 	jr	ra
 	 nop
@@ -451,11 +471,11 @@ EXC(	 sb	t0, NBYTES-2(dst), .Ls_exc_p1)
 	 *
 	 * Assumes src < THREAD_BUADDR($28)
 	 */
-	LOAD	t0, TI_TASK($28)
+	LOADK	t0, TI_TASK($28)
 	 nop
-	LOAD	t0, THREAD_BUADDR(t0)
+	LOADK	t0, THREAD_BUADDR(t0)
 1:
-EXC(	lb	t1, 0(src),	.Ll_exc)
+	LOADB(t1, 0(src), .Ll_exc)
 	ADD	src, src, 1
 	sb	t1, 0(dst)	# can't fault -- we're copy_from_user
 	.set	reorder				/* DADDI_WAR */
@@ -463,9 +483,9 @@ EXC(	lb	t1, 0(src),	.Ll_exc)
 	bne	src, t0, 1b
 	.set	noreorder
 .Ll_exc:
-	LOAD	t0, TI_TASK($28)
+	LOADK	t0, TI_TASK($28)
 	 nop
-	LOAD	t0, THREAD_BUADDR(t0)	# t0 is just past last good address
+	LOADK	t0, THREAD_BUADDR(t0)	# t0 is just past last good address
 	 nop
 	SUB	len, AT, t0		# len number of uncopied bytes
 	bnez	t6, .Ldone	/* Skip the zeroing part if inatomic */
-- 
1.8.5.3

[PATCH 17/58] MIPS: lib: memcpy: Split source and destination prefetch macros

[Markos Chandras]

In preparation for EVA support, the PREF macro is split into two
separate macros, PREFS and PREFD, for source and destination data
prefetching respectively.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/memcpy.S | 36 ++++++++++++++++++++++--------------
 1 file changed, 22 insertions(+), 14 deletions(-)

diff --git a/arch/mips/lib/memcpy.S b/arch/mips/lib/memcpy.S
index ab9f046..eed6e07 100644
--- a/arch/mips/lib/memcpy.S
+++ b/arch/mips/lib/memcpy.S
@@ -89,6 +89,9 @@
 /* Instruction type */
 #define LD_INSN 1
 #define ST_INSN 2
+/* Pretech type */
+#define SRC_PREFETCH 1
+#define DST_PREFETCH 2
 
 /*
  * Wrapper to add an entry in the exception table
@@ -174,6 +177,11 @@
 #define LOADB(reg, addr, handler)	EXC(lb, LD_INSN, reg, addr, handler)
 #define STOREB(reg, addr, handler)	EXC(sb, ST_INSN, reg, addr, handler)
 
+#define _PREF(hint, addr, type)	        PREF(hint, addr)
+
+#define PREFS(hint, addr) _PREF(hint, addr, SRC_PREFETCH)
+#define PREFD(hint, addr) _PREF(hint, addr, DST_PREFETCH)
+
 #ifdef CONFIG_CPU_LITTLE_ENDIAN
 #define LDFIRST LOADR
 #define LDREST	LOADL
@@ -237,16 +245,16 @@ __copy_user_common:
 	 *
 	 * If len < NBYTES use byte operations.
 	 */
-	PREF(	0, 0(src) )
-	PREF(	1, 0(dst) )
+	PREFS(	0, 0(src) )
+	PREFD(	1, 0(dst) )
 	sltu	t2, len, NBYTES
 	and	t1, dst, ADDRMASK
-	PREF(	0, 1*32(src) )
-	PREF(	1, 1*32(dst) )
+	PREFS(	0, 1*32(src) )
+	PREFD(	1, 1*32(dst) )
 	bnez	t2, .Lcopy_bytes_checklen
 	 and	t0, src, ADDRMASK
-	PREF(	0, 2*32(src) )
-	PREF(	1, 2*32(dst) )
+	PREFS(	0, 2*32(src) )
+	PREFD(	1, 2*32(dst) )
 	bnez	t1, .Ldst_unaligned
 	 nop
 	bnez	t0, .Lsrc_unaligned_dst_aligned
@@ -258,8 +266,8 @@ __copy_user_common:
 	 SRL	t0, len, LOG_NBYTES+3	 # +3 for 8 units/iter
 	beqz	t0, .Lcleanup_both_aligned # len < 8*NBYTES
 	 and	rem, len, (8*NBYTES-1)	 # rem = len % (8*NBYTES)
-	PREF(	0, 3*32(src) )
-	PREF(	1, 3*32(dst) )
+	PREFS(	0, 3*32(src) )
+	PREFD(	1, 3*32(dst) )
 	.align	4
 1:
 	R10KCBARRIER(0(ra))
@@ -282,8 +290,8 @@ __copy_user_common:
 	STORE(t7, UNIT(-3)(dst), .Ls_exc_p3u)
 	STORE(t0, UNIT(-2)(dst), .Ls_exc_p2u)
 	STORE(t1, UNIT(-1)(dst), .Ls_exc_p1u)
-	PREF(	0, 8*32(src) )
-	PREF(	1, 8*32(dst) )
+	PREFS(	0, 8*32(src) )
+	PREFD(	1, 8*32(dst) )
 	bne	len, rem, 1b
 	 nop
 
@@ -378,10 +386,10 @@ __copy_user_common:
 
 .Lsrc_unaligned_dst_aligned:
 	SRL	t0, len, LOG_NBYTES+2	 # +2 for 4 units/iter
-	PREF(	0, 3*32(src) )
+	PREFS(	0, 3*32(src) )
 	beqz	t0, .Lcleanup_src_unaligned
 	 and	rem, len, (4*NBYTES-1)	 # rem = len % 4*NBYTES
-	PREF(	1, 3*32(dst) )
+	PREFD(	1, 3*32(dst) )
 1:
 /*
  * Avoid consecutive LD*'s to the same register since some mips
@@ -399,7 +407,7 @@ __copy_user_common:
 	LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy)
 	LDREST(t2, REST(2)(src), .Ll_exc_copy)
 	LDREST(t3, REST(3)(src), .Ll_exc_copy)
-	PREF(	0, 9*32(src) )		# 0 is PREF_LOAD  (not streamed)
+	PREFS(	0, 9*32(src) )		# 0 is PREF_LOAD  (not streamed)
 	ADD	src, src, 4*NBYTES
 #ifdef CONFIG_CPU_SB1
 	nop				# improves slotting
@@ -408,7 +416,7 @@ __copy_user_common:
 	STORE(t1, UNIT(1)(dst),	.Ls_exc_p3u)
 	STORE(t2, UNIT(2)(dst),	.Ls_exc_p2u)
 	STORE(t3, UNIT(3)(dst),	.Ls_exc_p1u)
-	PREF(	1, 9*32(dst) )		# 1 is PREF_STORE (not streamed)
+	PREFD(	1, 9*32(dst) )		# 1 is PREF_STORE (not streamed)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 4*NBYTES
 	bne	len, rem, 1b
-- 
1.8.5.3

[PATCH 17/58] MIPS: lib: memcpy: Split source and destination prefetch macros

[Markos Chandras]

In preparation for EVA support, the PREF macro is split into two
separate macros, PREFS and PREFD, for source and destination data
prefetching respectively.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/memcpy.S | 36 ++++++++++++++++++++++--------------
 1 file changed, 22 insertions(+), 14 deletions(-)

diff --git a/arch/mips/lib/memcpy.S b/arch/mips/lib/memcpy.S
index ab9f046..eed6e07 100644
--- a/arch/mips/lib/memcpy.S
+++ b/arch/mips/lib/memcpy.S
@@ -89,6 +89,9 @@
 /* Instruction type */
 #define LD_INSN 1
 #define ST_INSN 2
+/* Pretech type */
+#define SRC_PREFETCH 1
+#define DST_PREFETCH 2
 
 /*
  * Wrapper to add an entry in the exception table
@@ -174,6 +177,11 @@
 #define LOADB(reg, addr, handler)	EXC(lb, LD_INSN, reg, addr, handler)
 #define STOREB(reg, addr, handler)	EXC(sb, ST_INSN, reg, addr, handler)
 
+#define _PREF(hint, addr, type)	        PREF(hint, addr)
+
+#define PREFS(hint, addr) _PREF(hint, addr, SRC_PREFETCH)
+#define PREFD(hint, addr) _PREF(hint, addr, DST_PREFETCH)
+
 #ifdef CONFIG_CPU_LITTLE_ENDIAN
 #define LDFIRST LOADR
 #define LDREST	LOADL
@@ -237,16 +245,16 @@ __copy_user_common:
 	 *
 	 * If len < NBYTES use byte operations.
 	 */
-	PREF(	0, 0(src) )
-	PREF(	1, 0(dst) )
+	PREFS(	0, 0(src) )
+	PREFD(	1, 0(dst) )
 	sltu	t2, len, NBYTES
 	and	t1, dst, ADDRMASK
-	PREF(	0, 1*32(src) )
-	PREF(	1, 1*32(dst) )
+	PREFS(	0, 1*32(src) )
+	PREFD(	1, 1*32(dst) )
 	bnez	t2, .Lcopy_bytes_checklen
 	 and	t0, src, ADDRMASK
-	PREF(	0, 2*32(src) )
-	PREF(	1, 2*32(dst) )
+	PREFS(	0, 2*32(src) )
+	PREFD(	1, 2*32(dst) )
 	bnez	t1, .Ldst_unaligned
 	 nop
 	bnez	t0, .Lsrc_unaligned_dst_aligned
@@ -258,8 +266,8 @@ __copy_user_common:
 	 SRL	t0, len, LOG_NBYTES+3	 # +3 for 8 units/iter
 	beqz	t0, .Lcleanup_both_aligned # len < 8*NBYTES
 	 and	rem, len, (8*NBYTES-1)	 # rem = len % (8*NBYTES)
-	PREF(	0, 3*32(src) )
-	PREF(	1, 3*32(dst) )
+	PREFS(	0, 3*32(src) )
+	PREFD(	1, 3*32(dst) )
 	.align	4
 1:
 	R10KCBARRIER(0(ra))
@@ -282,8 +290,8 @@ __copy_user_common:
 	STORE(t7, UNIT(-3)(dst), .Ls_exc_p3u)
 	STORE(t0, UNIT(-2)(dst), .Ls_exc_p2u)
 	STORE(t1, UNIT(-1)(dst), .Ls_exc_p1u)
-	PREF(	0, 8*32(src) )
-	PREF(	1, 8*32(dst) )
+	PREFS(	0, 8*32(src) )
+	PREFD(	1, 8*32(dst) )
 	bne	len, rem, 1b
 	 nop
 
@@ -378,10 +386,10 @@ __copy_user_common:
 
 .Lsrc_unaligned_dst_aligned:
 	SRL	t0, len, LOG_NBYTES+2	 # +2 for 4 units/iter
-	PREF(	0, 3*32(src) )
+	PREFS(	0, 3*32(src) )
 	beqz	t0, .Lcleanup_src_unaligned
 	 and	rem, len, (4*NBYTES-1)	 # rem = len % 4*NBYTES
-	PREF(	1, 3*32(dst) )
+	PREFD(	1, 3*32(dst) )
 1:
 /*
  * Avoid consecutive LD*'s to the same register since some mips
@@ -399,7 +407,7 @@ __copy_user_common:
 	LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy)
 	LDREST(t2, REST(2)(src), .Ll_exc_copy)
 	LDREST(t3, REST(3)(src), .Ll_exc_copy)
-	PREF(	0, 9*32(src) )		# 0 is PREF_LOAD  (not streamed)
+	PREFS(	0, 9*32(src) )		# 0 is PREF_LOAD  (not streamed)
 	ADD	src, src, 4*NBYTES
 #ifdef CONFIG_CPU_SB1
 	nop				# improves slotting
@@ -408,7 +416,7 @@ __copy_user_common:
 	STORE(t1, UNIT(1)(dst),	.Ls_exc_p3u)
 	STORE(t2, UNIT(2)(dst),	.Ls_exc_p2u)
 	STORE(t3, UNIT(3)(dst),	.Ls_exc_p1u)
-	PREF(	1, 9*32(dst) )		# 1 is PREF_STORE (not streamed)
+	PREFD(	1, 9*32(dst) )		# 1 is PREF_STORE (not streamed)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 4*NBYTES
 	bne	len, rem, 1b
-- 
1.8.5.3

[PATCH 18/58] MIPS: lib: memcpy: Use macro to build the copy_user code

[Markos Chandras]

The code can be shared between EVA and non-EVA configurations,
therefore use a macro to build it to avoid code duplications.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/memcpy.S | 253 ++++++++++++++++++++++++++++---------------------
 1 file changed, 143 insertions(+), 110 deletions(-)

diff --git a/arch/mips/lib/memcpy.S b/arch/mips/lib/memcpy.S
index eed6e07..d630a28 100644
--- a/arch/mips/lib/memcpy.S
+++ b/arch/mips/lib/memcpy.S
@@ -92,6 +92,10 @@
 /* Pretech type */
 #define SRC_PREFETCH 1
 #define DST_PREFETCH 2
+#define LEGACY_MODE 1
+#define EVA_MODE    2
+#define USEROP   1
+#define KERNELOP 2
 
 /*
  * Wrapper to add an entry in the exception table
@@ -103,12 +107,14 @@
  * addr    : Address
  * handler : Exception handler
  */
-#define EXC(insn, type, reg, addr, handler)	\
-9:	insn reg, addr;				\
-	.section __ex_table,"a";		\
-	PTR	9b, handler;			\
-	.previous
 
+#define EXC(insn, type, reg, addr, handler)			\
+	.if \mode == LEGACY_MODE;				\
+9:		insn reg, addr;					\
+		.section __ex_table,"a";			\
+		PTR	9b, handler;				\
+		.previous;					\
+	.endif
 /*
  * Only on the 64-bit kernel we can made use of 64-bit registers.
  */
@@ -177,7 +183,10 @@
 #define LOADB(reg, addr, handler)	EXC(lb, LD_INSN, reg, addr, handler)
 #define STOREB(reg, addr, handler)	EXC(sb, ST_INSN, reg, addr, handler)
 
-#define _PREF(hint, addr, type)	        PREF(hint, addr)
+#define _PREF(hint, addr, type)						\
+	.if \mode == LEGACY_MODE;					\
+		PREF(hint, addr);					\
+	.endif
 
 #define PREFS(hint, addr) _PREF(hint, addr, SRC_PREFETCH)
 #define PREFD(hint, addr) _PREF(hint, addr, DST_PREFETCH)
@@ -210,27 +219,23 @@
 	.set	at=v1
 #endif
 
-/*
- * t6 is used as a flag to note inatomic mode.
- */
-LEAF(__copy_user_inatomic)
-	b	__copy_user_common
-	 li	t6, 1
-	END(__copy_user_inatomic)
-
-/*
- * A combined memcpy/__copy_user
- * __copy_user sets len to 0 for success; else to an upper bound of
- * the number of uncopied bytes.
- * memcpy sets v0 to dst.
- */
 	.align	5
-LEAF(memcpy)					/* a0=dst a1=src a2=len */
-	move	v0, dst				/* return value */
-.L__memcpy:
-FEXPORT(__copy_user)
-	li	t6, 0	/* not inatomic */
-__copy_user_common:
+
+	/*
+	 * Macro to build the __copy_user common code
+	 * Arguements:
+	 * mode : LEGACY_MODE or EVA_MODE
+	 * from : Source operand. USEROP or KERNELOP
+	 * to   : Destination operand. USEROP or KERNELOP
+	 */
+	.macro __BUILD_COPY_USER mode, from, to
+
+	/* initialize __memcpy if this the first time we execute this macro */
+	.ifnotdef __memcpy
+	.set __memcpy, 1
+	.hidden __memcpy /* make sure it does not leak */
+	.endif
+
 	/*
 	 * Note: dst & src may be unaligned, len may be 0
 	 * Temps
@@ -251,45 +256,45 @@ __copy_user_common:
 	and	t1, dst, ADDRMASK
 	PREFS(	0, 1*32(src) )
 	PREFD(	1, 1*32(dst) )
-	bnez	t2, .Lcopy_bytes_checklen
+	bnez	t2, .Lcopy_bytes_checklen\@
 	 and	t0, src, ADDRMASK
 	PREFS(	0, 2*32(src) )
 	PREFD(	1, 2*32(dst) )
-	bnez	t1, .Ldst_unaligned
+	bnez	t1, .Ldst_unaligned\@
 	 nop
-	bnez	t0, .Lsrc_unaligned_dst_aligned
+	bnez	t0, .Lsrc_unaligned_dst_aligned\@
 	/*
 	 * use delay slot for fall-through
 	 * src and dst are aligned; need to compute rem
 	 */
-.Lboth_aligned:
+.Lboth_aligned\@:
 	 SRL	t0, len, LOG_NBYTES+3	 # +3 for 8 units/iter
-	beqz	t0, .Lcleanup_both_aligned # len < 8*NBYTES
+	beqz	t0, .Lcleanup_both_aligned\@ # len < 8*NBYTES
 	 and	rem, len, (8*NBYTES-1)	 # rem = len % (8*NBYTES)
 	PREFS(	0, 3*32(src) )
 	PREFD(	1, 3*32(dst) )
 	.align	4
 1:
 	R10KCBARRIER(0(ra))
-	LOAD(t0, UNIT(0)(src), .Ll_exc)
-	LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
-	LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
-	LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
+	LOAD(t0, UNIT(0)(src), .Ll_exc\@)
+	LOAD(t1, UNIT(1)(src), .Ll_exc_copy\@)
+	LOAD(t2, UNIT(2)(src), .Ll_exc_copy\@)
+	LOAD(t3, UNIT(3)(src), .Ll_exc_copy\@)
 	SUB	len, len, 8*NBYTES
-	LOAD(t4, UNIT(4)(src), .Ll_exc_copy)
-	LOAD(t7, UNIT(5)(src), .Ll_exc_copy)
-	STORE(t0, UNIT(0)(dst),	.Ls_exc_p8u)
-	STORE(t1, UNIT(1)(dst),	.Ls_exc_p7u)
-	LOAD(t0, UNIT(6)(src), .Ll_exc_copy)
-	LOAD(t1, UNIT(7)(src), .Ll_exc_copy)
+	LOAD(t4, UNIT(4)(src), .Ll_exc_copy\@)
+	LOAD(t7, UNIT(5)(src), .Ll_exc_copy\@)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc_p8u\@)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc_p7u\@)
+	LOAD(t0, UNIT(6)(src), .Ll_exc_copy\@)
+	LOAD(t1, UNIT(7)(src), .Ll_exc_copy\@)
 	ADD	src, src, 8*NBYTES
 	ADD	dst, dst, 8*NBYTES
-	STORE(t2, UNIT(-6)(dst), .Ls_exc_p6u)
-	STORE(t3, UNIT(-5)(dst), .Ls_exc_p5u)
-	STORE(t4, UNIT(-4)(dst), .Ls_exc_p4u)
-	STORE(t7, UNIT(-3)(dst), .Ls_exc_p3u)
-	STORE(t0, UNIT(-2)(dst), .Ls_exc_p2u)
-	STORE(t1, UNIT(-1)(dst), .Ls_exc_p1u)
+	STORE(t2, UNIT(-6)(dst), .Ls_exc_p6u\@)
+	STORE(t3, UNIT(-5)(dst), .Ls_exc_p5u\@)
+	STORE(t4, UNIT(-4)(dst), .Ls_exc_p4u\@)
+	STORE(t7, UNIT(-3)(dst), .Ls_exc_p3u\@)
+	STORE(t0, UNIT(-2)(dst), .Ls_exc_p2u\@)
+	STORE(t1, UNIT(-1)(dst), .Ls_exc_p1u\@)
 	PREFS(	0, 8*32(src) )
 	PREFD(	1, 8*32(dst) )
 	bne	len, rem, 1b
@@ -298,41 +303,41 @@ __copy_user_common:
 	/*
 	 * len == rem == the number of bytes left to copy < 8*NBYTES
 	 */
-.Lcleanup_both_aligned:
-	beqz	len, .Ldone
+.Lcleanup_both_aligned\@:
+	beqz	len, .Ldone\@
 	 sltu	t0, len, 4*NBYTES
-	bnez	t0, .Lless_than_4units
+	bnez	t0, .Lless_than_4units\@
 	 and	rem, len, (NBYTES-1)	# rem = len % NBYTES
 	/*
 	 * len >= 4*NBYTES
 	 */
-	LOAD( t0, UNIT(0)(src),	.Ll_exc)
-	LOAD( t1, UNIT(1)(src),	.Ll_exc_copy)
-	LOAD( t2, UNIT(2)(src),	.Ll_exc_copy)
-	LOAD( t3, UNIT(3)(src),	.Ll_exc_copy)
+	LOAD( t0, UNIT(0)(src),	.Ll_exc\@)
+	LOAD( t1, UNIT(1)(src),	.Ll_exc_copy\@)
+	LOAD( t2, UNIT(2)(src),	.Ll_exc_copy\@)
+	LOAD( t3, UNIT(3)(src),	.Ll_exc_copy\@)
 	SUB	len, len, 4*NBYTES
 	ADD	src, src, 4*NBYTES
 	R10KCBARRIER(0(ra))
-	STORE(t0, UNIT(0)(dst),	.Ls_exc_p4u)
-	STORE(t1, UNIT(1)(dst),	.Ls_exc_p3u)
-	STORE(t2, UNIT(2)(dst),	.Ls_exc_p2u)
-	STORE(t3, UNIT(3)(dst),	.Ls_exc_p1u)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc_p4u\@)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc_p3u\@)
+	STORE(t2, UNIT(2)(dst),	.Ls_exc_p2u\@)
+	STORE(t3, UNIT(3)(dst),	.Ls_exc_p1u\@)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 4*NBYTES
-	beqz	len, .Ldone
+	beqz	len, .Ldone\@
 	.set	noreorder
-.Lless_than_4units:
+.Lless_than_4units\@:
 	/*
 	 * rem = len % NBYTES
 	 */
-	beq	rem, len, .Lcopy_bytes
+	beq	rem, len, .Lcopy_bytes\@
 	 nop
 1:
 	R10KCBARRIER(0(ra))
-	LOAD(t0, 0(src), .Ll_exc)
+	LOAD(t0, 0(src), .Ll_exc\@)
 	ADD	src, src, NBYTES
 	SUB	len, len, NBYTES
-	STORE(t0, 0(dst), .Ls_exc_p1u)
+	STORE(t0, 0(dst), .Ls_exc_p1u\@)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, NBYTES
 	bne	rem, len, 1b
@@ -350,17 +355,17 @@ __copy_user_common:
 	 * more instruction-level parallelism.
 	 */
 #define bits t2
-	beqz	len, .Ldone
+	beqz	len, .Ldone\@
 	 ADD	t1, dst, len	# t1 is just past last byte of dst
 	li	bits, 8*NBYTES
 	SLL	rem, len, 3	# rem = number of bits to keep
-	LOAD(t0, 0(src), .Ll_exc)
+	LOAD(t0, 0(src), .Ll_exc\@)
 	SUB	bits, bits, rem # bits = number of bits to discard
 	SHIFT_DISCARD t0, t0, bits
-	STREST(t0, -1(t1), .Ls_exc)
+	STREST(t0, -1(t1), .Ls_exc\@)
 	jr	ra
 	 move	len, zero
-.Ldst_unaligned:
+.Ldst_unaligned\@:
 	/*
 	 * dst is unaligned
 	 * t0 = src & ADDRMASK
@@ -371,23 +376,23 @@ __copy_user_common:
 	 * Set match = (src and dst have same alignment)
 	 */
 #define match rem
-	LDFIRST(t3, FIRST(0)(src), .Ll_exc)
+	LDFIRST(t3, FIRST(0)(src), .Ll_exc\@)
 	ADD	t2, zero, NBYTES
-	LDREST(t3, REST(0)(src), .Ll_exc_copy)
+	LDREST(t3, REST(0)(src), .Ll_exc_copy\@)
 	SUB	t2, t2, t1	# t2 = number of bytes copied
 	xor	match, t0, t1
 	R10KCBARRIER(0(ra))
-	STFIRST(t3, FIRST(0)(dst), .Ls_exc)
-	beq	len, t2, .Ldone
+	STFIRST(t3, FIRST(0)(dst), .Ls_exc\@)
+	beq	len, t2, .Ldone\@
 	 SUB	len, len, t2
 	ADD	dst, dst, t2
-	beqz	match, .Lboth_aligned
+	beqz	match, .Lboth_aligned\@
 	 ADD	src, src, t2
 
-.Lsrc_unaligned_dst_aligned:
+.Lsrc_unaligned_dst_aligned\@:
 	SRL	t0, len, LOG_NBYTES+2	 # +2 for 4 units/iter
 	PREFS(	0, 3*32(src) )
-	beqz	t0, .Lcleanup_src_unaligned
+	beqz	t0, .Lcleanup_src_unaligned\@
 	 and	rem, len, (4*NBYTES-1)	 # rem = len % 4*NBYTES
 	PREFD(	1, 3*32(dst) )
 1:
@@ -398,58 +403,58 @@ __copy_user_common:
  * are to the same unit (unless src is aligned, but it's not).
  */
 	R10KCBARRIER(0(ra))
-	LDFIRST(t0, FIRST(0)(src), .Ll_exc)
-	LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy)
+	LDFIRST(t0, FIRST(0)(src), .Ll_exc\@)
+	LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy\@)
 	SUB	len, len, 4*NBYTES
-	LDREST(t0, REST(0)(src), .Ll_exc_copy)
-	LDREST(t1, REST(1)(src), .Ll_exc_copy)
-	LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy)
-	LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy)
-	LDREST(t2, REST(2)(src), .Ll_exc_copy)
-	LDREST(t3, REST(3)(src), .Ll_exc_copy)
+	LDREST(t0, REST(0)(src), .Ll_exc_copy\@)
+	LDREST(t1, REST(1)(src), .Ll_exc_copy\@)
+	LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy\@)
+	LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy\@)
+	LDREST(t2, REST(2)(src), .Ll_exc_copy\@)
+	LDREST(t3, REST(3)(src), .Ll_exc_copy\@)
 	PREFS(	0, 9*32(src) )		# 0 is PREF_LOAD  (not streamed)
 	ADD	src, src, 4*NBYTES
 #ifdef CONFIG_CPU_SB1
 	nop				# improves slotting
 #endif
-	STORE(t0, UNIT(0)(dst),	.Ls_exc_p4u)
-	STORE(t1, UNIT(1)(dst),	.Ls_exc_p3u)
-	STORE(t2, UNIT(2)(dst),	.Ls_exc_p2u)
-	STORE(t3, UNIT(3)(dst),	.Ls_exc_p1u)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc_p4u\@)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc_p3u\@)
+	STORE(t2, UNIT(2)(dst),	.Ls_exc_p2u\@)
+	STORE(t3, UNIT(3)(dst),	.Ls_exc_p1u\@)
 	PREFD(	1, 9*32(dst) )		# 1 is PREF_STORE (not streamed)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 4*NBYTES
 	bne	len, rem, 1b
 	.set	noreorder
 
-.Lcleanup_src_unaligned:
-	beqz	len, .Ldone
+.Lcleanup_src_unaligned\@:
+	beqz	len, .Ldone\@
 	 and	rem, len, NBYTES-1  # rem = len % NBYTES
-	beq	rem, len, .Lcopy_bytes
+	beq	rem, len, .Lcopy_bytes\@
 	 nop
 1:
 	R10KCBARRIER(0(ra))
-	LDFIRST(t0, FIRST(0)(src), .Ll_exc)
-	LDREST(t0, REST(0)(src), .Ll_exc_copy)
+	LDFIRST(t0, FIRST(0)(src), .Ll_exc\@)
+	LDREST(t0, REST(0)(src), .Ll_exc_copy\@)
 	ADD	src, src, NBYTES
 	SUB	len, len, NBYTES
-	STORE(t0, 0(dst), .Ls_exc_p1u)
+	STORE(t0, 0(dst), .Ls_exc_p1u\@)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, NBYTES
 	bne	len, rem, 1b
 	.set	noreorder
 
-.Lcopy_bytes_checklen:
-	beqz	len, .Ldone
+.Lcopy_bytes_checklen\@:
+	beqz	len, .Ldone\@
 	 nop
-.Lcopy_bytes:
+.Lcopy_bytes\@:
 	/* 0 < len < NBYTES  */
 	R10KCBARRIER(0(ra))
 #define COPY_BYTE(N)			\
-	LOADB(t0, N(src), .Ll_exc);	\
+	LOADB(t0, N(src), .Ll_exc\@);	\
 	SUB	len, len, 1;		\
-	beqz	len, .Ldone;		\
-	STOREB(t0, N(dst), .Ls_exc_p1)
+	beqz	len, .Ldone\@;		\
+	STOREB(t0, N(dst), .Ls_exc_p1\@)
 
 	COPY_BYTE(0)
 	COPY_BYTE(1)
@@ -459,16 +464,19 @@ __copy_user_common:
 	COPY_BYTE(4)
 	COPY_BYTE(5)
 #endif
-	LOADB(t0, NBYTES-2(src), .Ll_exc)
+	LOADB(t0, NBYTES-2(src), .Ll_exc\@)
 	SUB	len, len, 1
 	jr	ra
-	STOREB(t0, NBYTES-2(dst), .Ls_exc_p1)
-.Ldone:
+	STOREB(t0, NBYTES-2(dst), .Ls_exc_p1\@)
+.Ldone\@:
 	jr	ra
-	 nop
+	.if __memcpy == 1
 	END(memcpy)
+	.set __memcpy, 0
+	.hidden __memcpy
+	.endif
 
-.Ll_exc_copy:
+.Ll_exc_copy\@:
 	/*
 	 * Copy bytes from src until faulting load address (or until a
 	 * lb faults)
@@ -483,20 +491,20 @@ __copy_user_common:
 	 nop
 	LOADK	t0, THREAD_BUADDR(t0)
 1:
-	LOADB(t1, 0(src), .Ll_exc)
+	LOADB(t1, 0(src), .Ll_exc\@)
 	ADD	src, src, 1
 	sb	t1, 0(dst)	# can't fault -- we're copy_from_user
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 1
 	bne	src, t0, 1b
 	.set	noreorder
-.Ll_exc:
+.Ll_exc\@:
 	LOADK	t0, TI_TASK($28)
 	 nop
 	LOADK	t0, THREAD_BUADDR(t0)	# t0 is just past last good address
 	 nop
 	SUB	len, AT, t0		# len number of uncopied bytes
-	bnez	t6, .Ldone	/* Skip the zeroing part if inatomic */
+	bnez	t6, .Ldone\@	/* Skip the zeroing part if inatomic */
 	/*
 	 * Here's where we rely on src and dst being incremented in tandem,
 	 *   See (3) above.
@@ -510,7 +518,7 @@ __copy_user_common:
 	 */
 	.set	reorder				/* DADDI_WAR */
 	SUB	src, len, 1
-	beqz	len, .Ldone
+	beqz	len, .Ldone\@
 	.set	noreorder
 1:	sb	zero, 0(dst)
 	ADD	dst, dst, 1
@@ -531,7 +539,7 @@ __copy_user_common:
 
 #define SEXC(n)							\
 	.set	reorder;			/* DADDI_WAR */ \
-.Ls_exc_p ## n ## u:						\
+.Ls_exc_p ## n ## u\@:						\
 	ADD	len, len, n*NBYTES;				\
 	jr	ra;						\
 	.set	noreorder
@@ -545,14 +553,15 @@ SEXC(3)
 SEXC(2)
 SEXC(1)
 
-.Ls_exc_p1:
+.Ls_exc_p1\@:
 	.set	reorder				/* DADDI_WAR */
 	ADD	len, len, 1
 	jr	ra
 	.set	noreorder
-.Ls_exc:
+.Ls_exc\@:
 	jr	ra
 	 nop
+	.endm
 
 	.align	5
 LEAF(memmove)
@@ -603,3 +612,27 @@ LEAF(__rmemcpy)					/* a0=dst a1=src a2=len */
 	jr	ra
 	 move	a2, zero
 	END(__rmemcpy)
+
+/*
+ * t6 is used as a flag to note inatomic mode.
+ */
+LEAF(__copy_user_inatomic)
+	b	__copy_user_common
+	li	t6, 1
+	END(__copy_user_inatomic)
+
+/*
+ * A combined memcpy/__copy_user
+ * __copy_user sets len to 0 for success; else to an upper bound of
+ * the number of uncopied bytes.
+ * memcpy sets v0 to dst.
+ */
+	.align	5
+LEAF(memcpy)					/* a0=dst a1=src a2=len */
+	move	v0, dst				/* return value */
+.L__memcpy:
+FEXPORT(__copy_user)
+	li	t6, 0	/* not inatomic */
+__copy_user_common:
+	/* Legacy Mode, user <-> user */
+	__BUILD_COPY_USER LEGACY_MODE USEROP USEROP
-- 
1.8.5.3

[PATCH 18/58] MIPS: lib: memcpy: Use macro to build the copy_user code

[Markos Chandras]

The code can be shared between EVA and non-EVA configurations,
therefore use a macro to build it to avoid code duplications.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/memcpy.S | 253 ++++++++++++++++++++++++++++---------------------
 1 file changed, 143 insertions(+), 110 deletions(-)

diff --git a/arch/mips/lib/memcpy.S b/arch/mips/lib/memcpy.S
index eed6e07..d630a28 100644
--- a/arch/mips/lib/memcpy.S
+++ b/arch/mips/lib/memcpy.S
@@ -92,6 +92,10 @@
 /* Pretech type */
 #define SRC_PREFETCH 1
 #define DST_PREFETCH 2
+#define LEGACY_MODE 1
+#define EVA_MODE    2
+#define USEROP   1
+#define KERNELOP 2
 
 /*
  * Wrapper to add an entry in the exception table
@@ -103,12 +107,14 @@
  * addr    : Address
  * handler : Exception handler
  */
-#define EXC(insn, type, reg, addr, handler)	\
-9:	insn reg, addr;				\
-	.section __ex_table,"a";		\
-	PTR	9b, handler;			\
-	.previous
 
+#define EXC(insn, type, reg, addr, handler)			\
+	.if \mode == LEGACY_MODE;				\
+9:		insn reg, addr;					\
+		.section __ex_table,"a";			\
+		PTR	9b, handler;				\
+		.previous;					\
+	.endif
 /*
  * Only on the 64-bit kernel we can made use of 64-bit registers.
  */
@@ -177,7 +183,10 @@
 #define LOADB(reg, addr, handler)	EXC(lb, LD_INSN, reg, addr, handler)
 #define STOREB(reg, addr, handler)	EXC(sb, ST_INSN, reg, addr, handler)
 
-#define _PREF(hint, addr, type)	        PREF(hint, addr)
+#define _PREF(hint, addr, type)						\
+	.if \mode == LEGACY_MODE;					\
+		PREF(hint, addr);					\
+	.endif
 
 #define PREFS(hint, addr) _PREF(hint, addr, SRC_PREFETCH)
 #define PREFD(hint, addr) _PREF(hint, addr, DST_PREFETCH)
@@ -210,27 +219,23 @@
 	.set	at=v1
 #endif
 
-/*
- * t6 is used as a flag to note inatomic mode.
- */
-LEAF(__copy_user_inatomic)
-	b	__copy_user_common
-	 li	t6, 1
-	END(__copy_user_inatomic)
-
-/*
- * A combined memcpy/__copy_user
- * __copy_user sets len to 0 for success; else to an upper bound of
- * the number of uncopied bytes.
- * memcpy sets v0 to dst.
- */
 	.align	5
-LEAF(memcpy)					/* a0=dst a1=src a2=len */
-	move	v0, dst				/* return value */
-.L__memcpy:
-FEXPORT(__copy_user)
-	li	t6, 0	/* not inatomic */
-__copy_user_common:
+
+	/*
+	 * Macro to build the __copy_user common code
+	 * Arguements:
+	 * mode : LEGACY_MODE or EVA_MODE
+	 * from : Source operand. USEROP or KERNELOP
+	 * to   : Destination operand. USEROP or KERNELOP
+	 */
+	.macro __BUILD_COPY_USER mode, from, to
+
+	/* initialize __memcpy if this the first time we execute this macro */
+	.ifnotdef __memcpy
+	.set __memcpy, 1
+	.hidden __memcpy /* make sure it does not leak */
+	.endif
+
 	/*
 	 * Note: dst & src may be unaligned, len may be 0
 	 * Temps
@@ -251,45 +256,45 @@ __copy_user_common:
 	and	t1, dst, ADDRMASK
 	PREFS(	0, 1*32(src) )
 	PREFD(	1, 1*32(dst) )
-	bnez	t2, .Lcopy_bytes_checklen
+	bnez	t2, .Lcopy_bytes_checklen\@
 	 and	t0, src, ADDRMASK
 	PREFS(	0, 2*32(src) )
 	PREFD(	1, 2*32(dst) )
-	bnez	t1, .Ldst_unaligned
+	bnez	t1, .Ldst_unaligned\@
 	 nop
-	bnez	t0, .Lsrc_unaligned_dst_aligned
+	bnez	t0, .Lsrc_unaligned_dst_aligned\@
 	/*
 	 * use delay slot for fall-through
 	 * src and dst are aligned; need to compute rem
 	 */
-.Lboth_aligned:
+.Lboth_aligned\@:
 	 SRL	t0, len, LOG_NBYTES+3	 # +3 for 8 units/iter
-	beqz	t0, .Lcleanup_both_aligned # len < 8*NBYTES
+	beqz	t0, .Lcleanup_both_aligned\@ # len < 8*NBYTES
 	 and	rem, len, (8*NBYTES-1)	 # rem = len % (8*NBYTES)
 	PREFS(	0, 3*32(src) )
 	PREFD(	1, 3*32(dst) )
 	.align	4
 1:
 	R10KCBARRIER(0(ra))
-	LOAD(t0, UNIT(0)(src), .Ll_exc)
-	LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
-	LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
-	LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
+	LOAD(t0, UNIT(0)(src), .Ll_exc\@)
+	LOAD(t1, UNIT(1)(src), .Ll_exc_copy\@)
+	LOAD(t2, UNIT(2)(src), .Ll_exc_copy\@)
+	LOAD(t3, UNIT(3)(src), .Ll_exc_copy\@)
 	SUB	len, len, 8*NBYTES
-	LOAD(t4, UNIT(4)(src), .Ll_exc_copy)
-	LOAD(t7, UNIT(5)(src), .Ll_exc_copy)
-	STORE(t0, UNIT(0)(dst),	.Ls_exc_p8u)
-	STORE(t1, UNIT(1)(dst),	.Ls_exc_p7u)
-	LOAD(t0, UNIT(6)(src), .Ll_exc_copy)
-	LOAD(t1, UNIT(7)(src), .Ll_exc_copy)
+	LOAD(t4, UNIT(4)(src), .Ll_exc_copy\@)
+	LOAD(t7, UNIT(5)(src), .Ll_exc_copy\@)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc_p8u\@)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc_p7u\@)
+	LOAD(t0, UNIT(6)(src), .Ll_exc_copy\@)
+	LOAD(t1, UNIT(7)(src), .Ll_exc_copy\@)
 	ADD	src, src, 8*NBYTES
 	ADD	dst, dst, 8*NBYTES
-	STORE(t2, UNIT(-6)(dst), .Ls_exc_p6u)
-	STORE(t3, UNIT(-5)(dst), .Ls_exc_p5u)
-	STORE(t4, UNIT(-4)(dst), .Ls_exc_p4u)
-	STORE(t7, UNIT(-3)(dst), .Ls_exc_p3u)
-	STORE(t0, UNIT(-2)(dst), .Ls_exc_p2u)
-	STORE(t1, UNIT(-1)(dst), .Ls_exc_p1u)
+	STORE(t2, UNIT(-6)(dst), .Ls_exc_p6u\@)
+	STORE(t3, UNIT(-5)(dst), .Ls_exc_p5u\@)
+	STORE(t4, UNIT(-4)(dst), .Ls_exc_p4u\@)
+	STORE(t7, UNIT(-3)(dst), .Ls_exc_p3u\@)
+	STORE(t0, UNIT(-2)(dst), .Ls_exc_p2u\@)
+	STORE(t1, UNIT(-1)(dst), .Ls_exc_p1u\@)
 	PREFS(	0, 8*32(src) )
 	PREFD(	1, 8*32(dst) )
 	bne	len, rem, 1b
@@ -298,41 +303,41 @@ __copy_user_common:
 	/*
 	 * len == rem == the number of bytes left to copy < 8*NBYTES
 	 */
-.Lcleanup_both_aligned:
-	beqz	len, .Ldone
+.Lcleanup_both_aligned\@:
+	beqz	len, .Ldone\@
 	 sltu	t0, len, 4*NBYTES
-	bnez	t0, .Lless_than_4units
+	bnez	t0, .Lless_than_4units\@
 	 and	rem, len, (NBYTES-1)	# rem = len % NBYTES
 	/*
 	 * len >= 4*NBYTES
 	 */
-	LOAD( t0, UNIT(0)(src),	.Ll_exc)
-	LOAD( t1, UNIT(1)(src),	.Ll_exc_copy)
-	LOAD( t2, UNIT(2)(src),	.Ll_exc_copy)
-	LOAD( t3, UNIT(3)(src),	.Ll_exc_copy)
+	LOAD( t0, UNIT(0)(src),	.Ll_exc\@)
+	LOAD( t1, UNIT(1)(src),	.Ll_exc_copy\@)
+	LOAD( t2, UNIT(2)(src),	.Ll_exc_copy\@)
+	LOAD( t3, UNIT(3)(src),	.Ll_exc_copy\@)
 	SUB	len, len, 4*NBYTES
 	ADD	src, src, 4*NBYTES
 	R10KCBARRIER(0(ra))
-	STORE(t0, UNIT(0)(dst),	.Ls_exc_p4u)
-	STORE(t1, UNIT(1)(dst),	.Ls_exc_p3u)
-	STORE(t2, UNIT(2)(dst),	.Ls_exc_p2u)
-	STORE(t3, UNIT(3)(dst),	.Ls_exc_p1u)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc_p4u\@)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc_p3u\@)
+	STORE(t2, UNIT(2)(dst),	.Ls_exc_p2u\@)
+	STORE(t3, UNIT(3)(dst),	.Ls_exc_p1u\@)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 4*NBYTES
-	beqz	len, .Ldone
+	beqz	len, .Ldone\@
 	.set	noreorder
-.Lless_than_4units:
+.Lless_than_4units\@:
 	/*
 	 * rem = len % NBYTES
 	 */
-	beq	rem, len, .Lcopy_bytes
+	beq	rem, len, .Lcopy_bytes\@
 	 nop
 1:
 	R10KCBARRIER(0(ra))
-	LOAD(t0, 0(src), .Ll_exc)
+	LOAD(t0, 0(src), .Ll_exc\@)
 	ADD	src, src, NBYTES
 	SUB	len, len, NBYTES
-	STORE(t0, 0(dst), .Ls_exc_p1u)
+	STORE(t0, 0(dst), .Ls_exc_p1u\@)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, NBYTES
 	bne	rem, len, 1b
@@ -350,17 +355,17 @@ __copy_user_common:
 	 * more instruction-level parallelism.
 	 */
 #define bits t2
-	beqz	len, .Ldone
+	beqz	len, .Ldone\@
 	 ADD	t1, dst, len	# t1 is just past last byte of dst
 	li	bits, 8*NBYTES
 	SLL	rem, len, 3	# rem = number of bits to keep
-	LOAD(t0, 0(src), .Ll_exc)
+	LOAD(t0, 0(src), .Ll_exc\@)
 	SUB	bits, bits, rem # bits = number of bits to discard
 	SHIFT_DISCARD t0, t0, bits
-	STREST(t0, -1(t1), .Ls_exc)
+	STREST(t0, -1(t1), .Ls_exc\@)
 	jr	ra
 	 move	len, zero
-.Ldst_unaligned:
+.Ldst_unaligned\@:
 	/*
 	 * dst is unaligned
 	 * t0 = src & ADDRMASK
@@ -371,23 +376,23 @@ __copy_user_common:
 	 * Set match = (src and dst have same alignment)
 	 */
 #define match rem
-	LDFIRST(t3, FIRST(0)(src), .Ll_exc)
+	LDFIRST(t3, FIRST(0)(src), .Ll_exc\@)
 	ADD	t2, zero, NBYTES
-	LDREST(t3, REST(0)(src), .Ll_exc_copy)
+	LDREST(t3, REST(0)(src), .Ll_exc_copy\@)
 	SUB	t2, t2, t1	# t2 = number of bytes copied
 	xor	match, t0, t1
 	R10KCBARRIER(0(ra))
-	STFIRST(t3, FIRST(0)(dst), .Ls_exc)
-	beq	len, t2, .Ldone
+	STFIRST(t3, FIRST(0)(dst), .Ls_exc\@)
+	beq	len, t2, .Ldone\@
 	 SUB	len, len, t2
 	ADD	dst, dst, t2
-	beqz	match, .Lboth_aligned
+	beqz	match, .Lboth_aligned\@
 	 ADD	src, src, t2
 
-.Lsrc_unaligned_dst_aligned:
+.Lsrc_unaligned_dst_aligned\@:
 	SRL	t0, len, LOG_NBYTES+2	 # +2 for 4 units/iter
 	PREFS(	0, 3*32(src) )
-	beqz	t0, .Lcleanup_src_unaligned
+	beqz	t0, .Lcleanup_src_unaligned\@
 	 and	rem, len, (4*NBYTES-1)	 # rem = len % 4*NBYTES
 	PREFD(	1, 3*32(dst) )
 1:
@@ -398,58 +403,58 @@ __copy_user_common:
  * are to the same unit (unless src is aligned, but it's not).
  */
 	R10KCBARRIER(0(ra))
-	LDFIRST(t0, FIRST(0)(src), .Ll_exc)
-	LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy)
+	LDFIRST(t0, FIRST(0)(src), .Ll_exc\@)
+	LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy\@)
 	SUB	len, len, 4*NBYTES
-	LDREST(t0, REST(0)(src), .Ll_exc_copy)
-	LDREST(t1, REST(1)(src), .Ll_exc_copy)
-	LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy)
-	LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy)
-	LDREST(t2, REST(2)(src), .Ll_exc_copy)
-	LDREST(t3, REST(3)(src), .Ll_exc_copy)
+	LDREST(t0, REST(0)(src), .Ll_exc_copy\@)
+	LDREST(t1, REST(1)(src), .Ll_exc_copy\@)
+	LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy\@)
+	LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy\@)
+	LDREST(t2, REST(2)(src), .Ll_exc_copy\@)
+	LDREST(t3, REST(3)(src), .Ll_exc_copy\@)
 	PREFS(	0, 9*32(src) )		# 0 is PREF_LOAD  (not streamed)
 	ADD	src, src, 4*NBYTES
 #ifdef CONFIG_CPU_SB1
 	nop				# improves slotting
 #endif
-	STORE(t0, UNIT(0)(dst),	.Ls_exc_p4u)
-	STORE(t1, UNIT(1)(dst),	.Ls_exc_p3u)
-	STORE(t2, UNIT(2)(dst),	.Ls_exc_p2u)
-	STORE(t3, UNIT(3)(dst),	.Ls_exc_p1u)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc_p4u\@)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc_p3u\@)
+	STORE(t2, UNIT(2)(dst),	.Ls_exc_p2u\@)
+	STORE(t3, UNIT(3)(dst),	.Ls_exc_p1u\@)
 	PREFD(	1, 9*32(dst) )		# 1 is PREF_STORE (not streamed)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 4*NBYTES
 	bne	len, rem, 1b
 	.set	noreorder
 
-.Lcleanup_src_unaligned:
-	beqz	len, .Ldone
+.Lcleanup_src_unaligned\@:
+	beqz	len, .Ldone\@
 	 and	rem, len, NBYTES-1  # rem = len % NBYTES
-	beq	rem, len, .Lcopy_bytes
+	beq	rem, len, .Lcopy_bytes\@
 	 nop
 1:
 	R10KCBARRIER(0(ra))
-	LDFIRST(t0, FIRST(0)(src), .Ll_exc)
-	LDREST(t0, REST(0)(src), .Ll_exc_copy)
+	LDFIRST(t0, FIRST(0)(src), .Ll_exc\@)
+	LDREST(t0, REST(0)(src), .Ll_exc_copy\@)
 	ADD	src, src, NBYTES
 	SUB	len, len, NBYTES
-	STORE(t0, 0(dst), .Ls_exc_p1u)
+	STORE(t0, 0(dst), .Ls_exc_p1u\@)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, NBYTES
 	bne	len, rem, 1b
 	.set	noreorder
 
-.Lcopy_bytes_checklen:
-	beqz	len, .Ldone
+.Lcopy_bytes_checklen\@:
+	beqz	len, .Ldone\@
 	 nop
-.Lcopy_bytes:
+.Lcopy_bytes\@:
 	/* 0 < len < NBYTES  */
 	R10KCBARRIER(0(ra))
 #define COPY_BYTE(N)			\
-	LOADB(t0, N(src), .Ll_exc);	\
+	LOADB(t0, N(src), .Ll_exc\@);	\
 	SUB	len, len, 1;		\
-	beqz	len, .Ldone;		\
-	STOREB(t0, N(dst), .Ls_exc_p1)
+	beqz	len, .Ldone\@;		\
+	STOREB(t0, N(dst), .Ls_exc_p1\@)
 
 	COPY_BYTE(0)
 	COPY_BYTE(1)
@@ -459,16 +464,19 @@ __copy_user_common:
 	COPY_BYTE(4)
 	COPY_BYTE(5)
 #endif
-	LOADB(t0, NBYTES-2(src), .Ll_exc)
+	LOADB(t0, NBYTES-2(src), .Ll_exc\@)
 	SUB	len, len, 1
 	jr	ra
-	STOREB(t0, NBYTES-2(dst), .Ls_exc_p1)
-.Ldone:
+	STOREB(t0, NBYTES-2(dst), .Ls_exc_p1\@)
+.Ldone\@:
 	jr	ra
-	 nop
+	.if __memcpy == 1
 	END(memcpy)
+	.set __memcpy, 0
+	.hidden __memcpy
+	.endif
 
-.Ll_exc_copy:
+.Ll_exc_copy\@:
 	/*
 	 * Copy bytes from src until faulting load address (or until a
 	 * lb faults)
@@ -483,20 +491,20 @@ __copy_user_common:
 	 nop
 	LOADK	t0, THREAD_BUADDR(t0)
 1:
-	LOADB(t1, 0(src), .Ll_exc)
+	LOADB(t1, 0(src), .Ll_exc\@)
 	ADD	src, src, 1
 	sb	t1, 0(dst)	# can't fault -- we're copy_from_user
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 1
 	bne	src, t0, 1b
 	.set	noreorder
-.Ll_exc:
+.Ll_exc\@:
 	LOADK	t0, TI_TASK($28)
 	 nop
 	LOADK	t0, THREAD_BUADDR(t0)	# t0 is just past last good address
 	 nop
 	SUB	len, AT, t0		# len number of uncopied bytes
-	bnez	t6, .Ldone	/* Skip the zeroing part if inatomic */
+	bnez	t6, .Ldone\@	/* Skip the zeroing part if inatomic */
 	/*
 	 * Here's where we rely on src and dst being incremented in tandem,
 	 *   See (3) above.
@@ -510,7 +518,7 @@ __copy_user_common:
 	 */
 	.set	reorder				/* DADDI_WAR */
 	SUB	src, len, 1
-	beqz	len, .Ldone
+	beqz	len, .Ldone\@
 	.set	noreorder
 1:	sb	zero, 0(dst)
 	ADD	dst, dst, 1
@@ -531,7 +539,7 @@ __copy_user_common:
 
 #define SEXC(n)							\
 	.set	reorder;			/* DADDI_WAR */ \
-.Ls_exc_p ## n ## u:						\
+.Ls_exc_p ## n ## u\@:						\
 	ADD	len, len, n*NBYTES;				\
 	jr	ra;						\
 	.set	noreorder
@@ -545,14 +553,15 @@ SEXC(3)
 SEXC(2)
 SEXC(1)
 
-.Ls_exc_p1:
+.Ls_exc_p1\@:
 	.set	reorder				/* DADDI_WAR */
 	ADD	len, len, 1
 	jr	ra
 	.set	noreorder
-.Ls_exc:
+.Ls_exc\@:
 	jr	ra
 	 nop
+	.endm
 
 	.align	5
 LEAF(memmove)
@@ -603,3 +612,27 @@ LEAF(__rmemcpy)					/* a0=dst a1=src a2=len */
 	jr	ra
 	 move	a2, zero
 	END(__rmemcpy)
+
+/*
+ * t6 is used as a flag to note inatomic mode.
+ */
+LEAF(__copy_user_inatomic)
+	b	__copy_user_common
+	li	t6, 1
+	END(__copy_user_inatomic)
+
+/*
+ * A combined memcpy/__copy_user
+ * __copy_user sets len to 0 for success; else to an upper bound of
+ * the number of uncopied bytes.
+ * memcpy sets v0 to dst.
+ */
+	.align	5
+LEAF(memcpy)					/* a0=dst a1=src a2=len */
+	move	v0, dst				/* return value */
+.L__memcpy:
+FEXPORT(__copy_user)
+	li	t6, 0	/* not inatomic */
+__copy_user_common:
+	/* Legacy Mode, user <-> user */
+	__BUILD_COPY_USER LEGACY_MODE USEROP USEROP
-- 
1.8.5.3

[PATCH 19/58] MIPS: lib: memcpy: Add EVA support

[Markos Chandras]

Add copy_{to,from,in}_user when the CPU operates in EVA mode.
This is necessary so the EVA specific instructions can be used
to perform the virtual to physical translation for user space
addresses. We will use the non-EVA functions to read from kernel
if needed.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/mips_ksyms.c |  6 ++++
 arch/mips/lib/memcpy.S        | 77 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 83 insertions(+)

diff --git a/arch/mips/kernel/mips_ksyms.c b/arch/mips/kernel/mips_ksyms.c
index 675bd05..13e60f6 100644
--- a/arch/mips/kernel/mips_ksyms.c
+++ b/arch/mips/kernel/mips_ksyms.c
@@ -51,6 +51,12 @@ EXPORT_SYMBOL(copy_page);
  */
 EXPORT_SYMBOL(__copy_user);
 EXPORT_SYMBOL(__copy_user_inatomic);
+#ifdef CONFIG_EVA
+EXPORT_SYMBOL(__copy_from_user_eva);
+EXPORT_SYMBOL(__copy_in_user_eva);
+EXPORT_SYMBOL(__copy_to_user_eva);
+EXPORT_SYMBOL(__copy_user_inatomic_eva);
+#endif
 EXPORT_SYMBOL(__bzero);
 EXPORT_SYMBOL(__strncpy_from_kernel_nocheck_asm);
 EXPORT_SYMBOL(__strncpy_from_kernel_asm);
diff --git a/arch/mips/lib/memcpy.S b/arch/mips/lib/memcpy.S
index d630a28..c17ef80 100644
--- a/arch/mips/lib/memcpy.S
+++ b/arch/mips/lib/memcpy.S
@@ -114,7 +114,24 @@
 		.section __ex_table,"a";			\
 		PTR	9b, handler;				\
 		.previous;					\
+	/* This is assembled in EVA mode */			\
+	.else;							\
+		/* If loading from user or storing to user */	\
+		.if ((\from == USEROP) && (type == LD_INSN)) || \
+		    ((\to == USEROP) && (type == ST_INSN));	\
+9:			__BUILD_EVA_INSN(insn##e, reg, addr);	\
+			.section __ex_table,"a";		\
+			PTR	9b, handler;			\
+			.previous;				\
+		.else;						\
+			/*					\
+			 *  Still in EVA, but no need for	\
+			 * exception handler or EVA insn	\
+			 */					\
+			insn reg, addr;				\
+		.endif;						\
 	.endif
+
 /*
  * Only on the 64-bit kernel we can made use of 64-bit registers.
  */
@@ -186,6 +203,22 @@
 #define _PREF(hint, addr, type)						\
 	.if \mode == LEGACY_MODE;					\
 		PREF(hint, addr);					\
+	.else;								\
+		.if ((\from == USEROP) && (type == SRC_PREFETCH)) ||	\
+		    ((\to == USEROP) && (type == DST_PREFETCH));	\
+			/*						\
+			 * PREFE has only 9 bits for the offset		\
+			 * compared to PREF which has 16, so it may	\
+			 * need to use the $at register but this	\
+			 * register should remain intact because it's	\
+			 * used later on. Therefore use $v1.		\
+			 */						\
+			.set at=v1;					\
+			PREFE(hint, addr);				\
+			.set noat;					\
+		.else;							\
+			PREF(hint, addr);				\
+		.endif;							\
 	.endif
 
 #define PREFS(hint, addr) _PREF(hint, addr, SRC_PREFETCH)
@@ -636,3 +669,47 @@ FEXPORT(__copy_user)
 __copy_user_common:
 	/* Legacy Mode, user <-> user */
 	__BUILD_COPY_USER LEGACY_MODE USEROP USEROP
+
+#ifdef CONFIG_EVA
+
+/*
+ * For EVA we need distinct symbols for reading and writing to user space.
+ * This is because we need to use specific EVA instructions to perform the
+ * virtual <-> physical translation when a virtual address is actually in user
+ * space
+ */
+
+LEAF(__copy_user_inatomic_eva)
+	b       __copy_from_user_common
+	li	t6, 1
+	END(__copy_user_inatomic_eva)
+
+/*
+ * __copy_from_user (EVA)
+ */
+
+LEAF(__copy_from_user_eva)
+	li	t6, 0	/* not inatomic */
+__copy_from_user_common:
+	__BUILD_COPY_USER EVA_MODE USEROP KERNELOP
+END(__copy_from_user_eva)
+
+
+
+/*
+ * __copy_to_user (EVA)
+ */
+
+LEAF(__copy_to_user_eva)
+__BUILD_COPY_USER EVA_MODE KERNELOP USEROP
+END(__copy_to_user_eva)
+
+/*
+ * __copy_in_user (EVA)
+ */
+
+LEAF(__copy_in_user_eva)
+__BUILD_COPY_USER EVA_MODE USEROP USEROP
+END(__copy_in_user_eva)
+
+#endif
-- 
1.8.5.3

[PATCH 19/58] MIPS: lib: memcpy: Add EVA support

[Markos Chandras]

Add copy_{to,from,in}_user when the CPU operates in EVA mode.
This is necessary so the EVA specific instructions can be used
to perform the virtual to physical translation for user space
addresses. We will use the non-EVA functions to read from kernel
if needed.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/mips_ksyms.c |  6 ++++
 arch/mips/lib/memcpy.S        | 77 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 83 insertions(+)

diff --git a/arch/mips/kernel/mips_ksyms.c b/arch/mips/kernel/mips_ksyms.c
index 675bd05..13e60f6 100644
--- a/arch/mips/kernel/mips_ksyms.c
+++ b/arch/mips/kernel/mips_ksyms.c
@@ -51,6 +51,12 @@ EXPORT_SYMBOL(copy_page);
  */
 EXPORT_SYMBOL(__copy_user);
 EXPORT_SYMBOL(__copy_user_inatomic);
+#ifdef CONFIG_EVA
+EXPORT_SYMBOL(__copy_from_user_eva);
+EXPORT_SYMBOL(__copy_in_user_eva);
+EXPORT_SYMBOL(__copy_to_user_eva);
+EXPORT_SYMBOL(__copy_user_inatomic_eva);
+#endif
 EXPORT_SYMBOL(__bzero);
 EXPORT_SYMBOL(__strncpy_from_kernel_nocheck_asm);
 EXPORT_SYMBOL(__strncpy_from_kernel_asm);
diff --git a/arch/mips/lib/memcpy.S b/arch/mips/lib/memcpy.S
index d630a28..c17ef80 100644
--- a/arch/mips/lib/memcpy.S
+++ b/arch/mips/lib/memcpy.S
@@ -114,7 +114,24 @@
 		.section __ex_table,"a";			\
 		PTR	9b, handler;				\
 		.previous;					\
+	/* This is assembled in EVA mode */			\
+	.else;							\
+		/* If loading from user or storing to user */	\
+		.if ((\from == USEROP) && (type == LD_INSN)) || \
+		    ((\to == USEROP) && (type == ST_INSN));	\
+9:			__BUILD_EVA_INSN(insn##e, reg, addr);	\
+			.section __ex_table,"a";		\
+			PTR	9b, handler;			\
+			.previous;				\
+		.else;						\
+			/*					\
+			 *  Still in EVA, but no need for	\
+			 * exception handler or EVA insn	\
+			 */					\
+			insn reg, addr;				\
+		.endif;						\
 	.endif
+
 /*
  * Only on the 64-bit kernel we can made use of 64-bit registers.
  */
@@ -186,6 +203,22 @@
 #define _PREF(hint, addr, type)						\
 	.if \mode == LEGACY_MODE;					\
 		PREF(hint, addr);					\
+	.else;								\
+		.if ((\from == USEROP) && (type == SRC_PREFETCH)) ||	\
+		    ((\to == USEROP) && (type == DST_PREFETCH));	\
+			/*						\
+			 * PREFE has only 9 bits for the offset		\
+			 * compared to PREF which has 16, so it may	\
+			 * need to use the $at register but this	\
+			 * register should remain intact because it's	\
+			 * used later on. Therefore use $v1.		\
+			 */						\
+			.set at=v1;					\
+			PREFE(hint, addr);				\
+			.set noat;					\
+		.else;							\
+			PREF(hint, addr);				\
+		.endif;							\
 	.endif
 
 #define PREFS(hint, addr) _PREF(hint, addr, SRC_PREFETCH)
@@ -636,3 +669,47 @@ FEXPORT(__copy_user)
 __copy_user_common:
 	/* Legacy Mode, user <-> user */
 	__BUILD_COPY_USER LEGACY_MODE USEROP USEROP
+
+#ifdef CONFIG_EVA
+
+/*
+ * For EVA we need distinct symbols for reading and writing to user space.
+ * This is because we need to use specific EVA instructions to perform the
+ * virtual <-> physical translation when a virtual address is actually in user
+ * space
+ */
+
+LEAF(__copy_user_inatomic_eva)
+	b       __copy_from_user_common
+	li	t6, 1
+	END(__copy_user_inatomic_eva)
+
+/*
+ * __copy_from_user (EVA)
+ */
+
+LEAF(__copy_from_user_eva)
+	li	t6, 0	/* not inatomic */
+__copy_from_user_common:
+	__BUILD_COPY_USER EVA_MODE USEROP KERNELOP
+END(__copy_from_user_eva)
+
+
+
+/*
+ * __copy_to_user (EVA)
+ */
+
+LEAF(__copy_to_user_eva)
+__BUILD_COPY_USER EVA_MODE KERNELOP USEROP
+END(__copy_to_user_eva)
+
+/*
+ * __copy_in_user (EVA)
+ */
+
+LEAF(__copy_in_user_eva)
+__BUILD_COPY_USER EVA_MODE USEROP USEROP
+END(__copy_in_user_eva)
+
+#endif
-- 
1.8.5.3

[PATCH 20/58] MIPS: lib: memset: Whitespace fixes

[Markos Chandras]

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/memset.S | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)

diff --git a/arch/mips/lib/memset.S b/arch/mips/lib/memset.S
index 0580194..d857985 100644
--- a/arch/mips/lib/memset.S
+++ b/arch/mips/lib/memset.S
@@ -74,7 +74,7 @@
 	.align	5
 LEAF(memset)
 	beqz		a1, 1f
-	 move		v0, a0			/* result */
+	move		v0, a0			/* result */
 
 	andi		a1, 0xff		/* spread fillword */
 	LONG_SLL		t1, a1, 8
@@ -90,7 +90,7 @@ LEAF(memset)
 FEXPORT(__bzero)
 	sltiu		t0, a2, STORSIZE	/* very small region? */
 	bnez		t0, .Lsmall_memset
-	 andi		t0, a0, STORMASK	/* aligned? */
+	andi		t0, a0, STORMASK	/* aligned? */
 
 #ifdef CONFIG_CPU_MICROMIPS
 	move		t8, a1			/* used by 'swp' instruction */
@@ -98,12 +98,12 @@ FEXPORT(__bzero)
 #endif
 #ifndef CONFIG_CPU_DADDI_WORKAROUNDS
 	beqz		t0, 1f
-	 PTR_SUBU	t0, STORSIZE		/* alignment in bytes */
+	PTR_SUBU	t0, STORSIZE		/* alignment in bytes */
 #else
 	.set		noat
 	li		AT, STORSIZE
 	beqz		t0, 1f
-	 PTR_SUBU	t0, AT			/* alignment in bytes */
+	PTR_SUBU	t0, AT			/* alignment in bytes */
 	.set		at
 #endif
 
@@ -120,7 +120,7 @@ FEXPORT(__bzero)
 1:	ori		t1, a2, 0x3f		/* # of full blocks */
 	xori		t1, 0x3f
 	beqz		t1, .Lmemset_partial	/* no block to fill */
-	 andi		t0, a2, 0x40-STORSIZE
+	andi		t0, a2, 0x40-STORSIZE
 
 	PTR_ADDU	t1, a0			/* end address */
 	.set		reorder
@@ -145,7 +145,7 @@ FEXPORT(__bzero)
 	.set		at
 #endif
 	jr		t1
-	 PTR_ADDU	a0, t0			/* dest ptr */
+	PTR_ADDU	a0, t0			/* dest ptr */
 
 	.set		push
 	.set		noreorder
@@ -155,7 +155,7 @@ FEXPORT(__bzero)
 	andi		a2, STORMASK		/* At most one long to go */
 
 	beqz		a2, 1f
-	 PTR_ADDU	a0, a2			/* What's left */
+	PTR_ADDU	a0, a2			/* What's left */
 	R10KCBARRIER(0(ra))
 #ifdef __MIPSEB__
 	EX(LONG_S_R, a1, -1(a0), .Llast_fixup)
@@ -164,24 +164,24 @@ FEXPORT(__bzero)
 	EX(LONG_S_L, a1, -1(a0), .Llast_fixup)
 #endif
 1:	jr		ra
-	 move		a2, zero
+	move		a2, zero
 
 .Lsmall_memset:
 	beqz		a2, 2f
-	 PTR_ADDU	t1, a0, a2
+	PTR_ADDU	t1, a0, a2
 
 1:	PTR_ADDIU	a0, 1			/* fill bytewise */
 	R10KCBARRIER(0(ra))
 	bne		t1, a0, 1b
-	 sb		a1, -1(a0)
+	sb		a1, -1(a0)
 
 2:	jr		ra			/* done */
-	 move		a2, zero
+	move		a2, zero
 	END(memset)
 
 .Lfirst_fixup:
 	jr	ra
-	 nop
+	nop
 
 .Lfwd_fixup:
 	PTR_L		t0, TI_TASK($28)
@@ -189,7 +189,7 @@ FEXPORT(__bzero)
 	LONG_L		t0, THREAD_BUADDR(t0)
 	LONG_ADDU	a2, t1
 	jr		ra
-	 LONG_SUBU	a2, t0
+	LONG_SUBU	a2, t0
 
 .Lpartial_fixup:
 	PTR_L		t0, TI_TASK($28)
@@ -197,8 +197,8 @@ FEXPORT(__bzero)
 	LONG_L		t0, THREAD_BUADDR(t0)
 	LONG_ADDU	a2, t1
 	jr		ra
-	 LONG_SUBU	a2, t0
+	LONG_SUBU	a2, t0
 
 .Llast_fixup:
 	jr		ra
-	 andi		v1, a2, STORMASK
+	andi		v1, a2, STORMASK
-- 
1.8.5.3

[PATCH 20/58] MIPS: lib: memset: Whitespace fixes

[Markos Chandras]

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/memset.S | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)

diff --git a/arch/mips/lib/memset.S b/arch/mips/lib/memset.S
index 0580194..d857985 100644
--- a/arch/mips/lib/memset.S
+++ b/arch/mips/lib/memset.S
@@ -74,7 +74,7 @@
 	.align	5
 LEAF(memset)
 	beqz		a1, 1f
-	 move		v0, a0			/* result */
+	move		v0, a0			/* result */
 
 	andi		a1, 0xff		/* spread fillword */
 	LONG_SLL		t1, a1, 8
@@ -90,7 +90,7 @@ LEAF(memset)
 FEXPORT(__bzero)
 	sltiu		t0, a2, STORSIZE	/* very small region? */
 	bnez		t0, .Lsmall_memset
-	 andi		t0, a0, STORMASK	/* aligned? */
+	andi		t0, a0, STORMASK	/* aligned? */
 
 #ifdef CONFIG_CPU_MICROMIPS
 	move		t8, a1			/* used by 'swp' instruction */
@@ -98,12 +98,12 @@ FEXPORT(__bzero)
 #endif
 #ifndef CONFIG_CPU_DADDI_WORKAROUNDS
 	beqz		t0, 1f
-	 PTR_SUBU	t0, STORSIZE		/* alignment in bytes */
+	PTR_SUBU	t0, STORSIZE		/* alignment in bytes */
 #else
 	.set		noat
 	li		AT, STORSIZE
 	beqz		t0, 1f
-	 PTR_SUBU	t0, AT			/* alignment in bytes */
+	PTR_SUBU	t0, AT			/* alignment in bytes */
 	.set		at
 #endif
 
@@ -120,7 +120,7 @@ FEXPORT(__bzero)
 1:	ori		t1, a2, 0x3f		/* # of full blocks */
 	xori		t1, 0x3f
 	beqz		t1, .Lmemset_partial	/* no block to fill */
-	 andi		t0, a2, 0x40-STORSIZE
+	andi		t0, a2, 0x40-STORSIZE
 
 	PTR_ADDU	t1, a0			/* end address */
 	.set		reorder
@@ -145,7 +145,7 @@ FEXPORT(__bzero)
 	.set		at
 #endif
 	jr		t1
-	 PTR_ADDU	a0, t0			/* dest ptr */
+	PTR_ADDU	a0, t0			/* dest ptr */
 
 	.set		push
 	.set		noreorder
@@ -155,7 +155,7 @@ FEXPORT(__bzero)
 	andi		a2, STORMASK		/* At most one long to go */
 
 	beqz		a2, 1f
-	 PTR_ADDU	a0, a2			/* What's left */
+	PTR_ADDU	a0, a2			/* What's left */
 	R10KCBARRIER(0(ra))
 #ifdef __MIPSEB__
 	EX(LONG_S_R, a1, -1(a0), .Llast_fixup)
@@ -164,24 +164,24 @@ FEXPORT(__bzero)
 	EX(LONG_S_L, a1, -1(a0), .Llast_fixup)
 #endif
 1:	jr		ra
-	 move		a2, zero
+	move		a2, zero
 
 .Lsmall_memset:
 	beqz		a2, 2f
-	 PTR_ADDU	t1, a0, a2
+	PTR_ADDU	t1, a0, a2
 
 1:	PTR_ADDIU	a0, 1			/* fill bytewise */
 	R10KCBARRIER(0(ra))
 	bne		t1, a0, 1b
-	 sb		a1, -1(a0)
+	sb		a1, -1(a0)
 
 2:	jr		ra			/* done */
-	 move		a2, zero
+	move		a2, zero
 	END(memset)
 
 .Lfirst_fixup:
 	jr	ra
-	 nop
+	nop
 
 .Lfwd_fixup:
 	PTR_L		t0, TI_TASK($28)
@@ -189,7 +189,7 @@ FEXPORT(__bzero)
 	LONG_L		t0, THREAD_BUADDR(t0)
 	LONG_ADDU	a2, t1
 	jr		ra
-	 LONG_SUBU	a2, t0
+	LONG_SUBU	a2, t0
 
 .Lpartial_fixup:
 	PTR_L		t0, TI_TASK($28)
@@ -197,8 +197,8 @@ FEXPORT(__bzero)
 	LONG_L		t0, THREAD_BUADDR(t0)
 	LONG_ADDU	a2, t1
 	jr		ra
-	 LONG_SUBU	a2, t0
+	LONG_SUBU	a2, t0
 
 .Llast_fixup:
 	jr		ra
-	 andi		v1, a2, STORMASK
+	andi		v1, a2, STORMASK
-- 
1.8.5.3

[PATCH 21/58] MIPS: lib: memset: Use macro to build the __bzero symbol

[Markos Chandras]

Build the __bzero symbol using a macor. In EVA mode we will
need to use similar code to do the userspace load operations so
it is better if we use a macro to avoid code duplications.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/memset.S | 95 +++++++++++++++++++++++++++++++-------------------
 1 file changed, 60 insertions(+), 35 deletions(-)

diff --git a/arch/mips/lib/memset.S b/arch/mips/lib/memset.S
index d857985..05fac19 100644
--- a/arch/mips/lib/memset.S
+++ b/arch/mips/lib/memset.S
@@ -34,6 +34,9 @@
 #define FILLPTRG t0
 #endif
 
+#define LEGACY_MODE 1
+#define EVA_MODE    2
+
 #define EX(insn,reg,addr,handler)			\
 9:	insn	reg, addr;				\
 	.section __ex_table,"a";			\
@@ -63,33 +66,23 @@
 #endif
 	.endm
 
-/*
- * memset(void *s, int c, size_t n)
- *
- * a0: start of area to clear
- * a1: char to fill with
- * a2: size of area to clear
- */
 	.set	noreorder
 	.align	5
-LEAF(memset)
-	beqz		a1, 1f
-	move		v0, a0			/* result */
 
-	andi		a1, 0xff		/* spread fillword */
-	LONG_SLL		t1, a1, 8
-	or		a1, t1
-	LONG_SLL		t1, a1, 16
-#if LONGSIZE == 8
-	or		a1, t1
-	LONG_SLL		t1, a1, 32
-#endif
-	or		a1, t1
-1:
+	/*
+	 * Macro to generate the __bzero{,_user} symbol
+	 * Arguments:
+	 * mode: LEGACY_MODE or EVA_MODE
+	 */
+	.macro __BUILD_BZERO mode
+	/* Initialize __memset if this is the first time we call this macro */
+	.ifnotdef __memset
+	.set __memset, 1
+	.hidden __memset /* Make sure it does not leak */
+	.endif
 
-FEXPORT(__bzero)
 	sltiu		t0, a2, STORSIZE	/* very small region? */
-	bnez		t0, .Lsmall_memset
+	bnez		t0, .Lsmall_memset\@
 	andi		t0, a0, STORMASK	/* aligned? */
 
 #ifdef CONFIG_CPU_MICROMIPS
@@ -109,28 +102,28 @@ FEXPORT(__bzero)
 
 	R10KCBARRIER(0(ra))
 #ifdef __MIPSEB__
-	EX(LONG_S_L, a1, (a0), .Lfirst_fixup)	/* make word/dword aligned */
+	EX(LONG_S_L, a1, (a0), .Lfirst_fixup\@)	/* make word/dword aligned */
 #endif
 #ifdef __MIPSEL__
-	EX(LONG_S_R, a1, (a0), .Lfirst_fixup)	/* make word/dword aligned */
+	EX(LONG_S_R, a1, (a0), .Lfirst_fixup\@)	/* make word/dword aligned */
 #endif
 	PTR_SUBU	a0, t0			/* long align ptr */
 	PTR_ADDU	a2, t0			/* correct size */
 
 1:	ori		t1, a2, 0x3f		/* # of full blocks */
 	xori		t1, 0x3f
-	beqz		t1, .Lmemset_partial	/* no block to fill */
+	beqz		t1, .Lmemset_partial\@	/* no block to fill */
 	andi		t0, a2, 0x40-STORSIZE
 
 	PTR_ADDU	t1, a0			/* end address */
 	.set		reorder
 1:	PTR_ADDIU	a0, 64
 	R10KCBARRIER(0(ra))
-	f_fill64 a0, -64, FILL64RG, .Lfwd_fixup
+	f_fill64 a0, -64, FILL64RG, .Lfwd_fixup\@
 	bne		t1, a0, 1b
 	.set		noreorder
 
-.Lmemset_partial:
+.Lmemset_partial\@:
 	R10KCBARRIER(0(ra))
 	PTR_LA		t1, 2f			/* where to start */
 #ifdef CONFIG_CPU_MICROMIPS
@@ -150,7 +143,8 @@ FEXPORT(__bzero)
 	.set		push
 	.set		noreorder
 	.set		nomacro
-	f_fill64 a0, -64, FILL64RG, .Lpartial_fixup	/* ... but first do longs ... */
+	/* ... but first do longs ... */
+	f_fill64 a0, -64, FILL64RG, .Lpartial_fixup\@
 2:	.set		pop
 	andi		a2, STORMASK		/* At most one long to go */
 
@@ -158,15 +152,15 @@ FEXPORT(__bzero)
 	PTR_ADDU	a0, a2			/* What's left */
 	R10KCBARRIER(0(ra))
 #ifdef __MIPSEB__
-	EX(LONG_S_R, a1, -1(a0), .Llast_fixup)
+	EX(LONG_S_R, a1, -1(a0), .Llast_fixup\@)
 #endif
 #ifdef __MIPSEL__
-	EX(LONG_S_L, a1, -1(a0), .Llast_fixup)
+	EX(LONG_S_L, a1, -1(a0), .Llast_fixup\@)
 #endif
 1:	jr		ra
 	move		a2, zero
 
-.Lsmall_memset:
+.Lsmall_memset\@:
 	beqz		a2, 2f
 	PTR_ADDU	t1, a0, a2
 
@@ -177,13 +171,17 @@ FEXPORT(__bzero)
 
 2:	jr		ra			/* done */
 	move		a2, zero
+	.if __memset == 1
 	END(memset)
+	.set __memset, 0
+	.hidden __memset
+	.endif
 
-.Lfirst_fixup:
+.Lfirst_fixup\@:
 	jr	ra
 	nop
 
-.Lfwd_fixup:
+.Lfwd_fixup\@:
 	PTR_L		t0, TI_TASK($28)
 	andi		a2, 0x3f
 	LONG_L		t0, THREAD_BUADDR(t0)
@@ -191,7 +189,7 @@ FEXPORT(__bzero)
 	jr		ra
 	LONG_SUBU	a2, t0
 
-.Lpartial_fixup:
+.Lpartial_fixup\@:
 	PTR_L		t0, TI_TASK($28)
 	andi		a2, STORMASK
 	LONG_L		t0, THREAD_BUADDR(t0)
@@ -199,6 +197,33 @@ FEXPORT(__bzero)
 	jr		ra
 	LONG_SUBU	a2, t0
 
-.Llast_fixup:
+.Llast_fixup\@:
 	jr		ra
 	andi		v1, a2, STORMASK
+
+	.endm
+
+/*
+ * memset(void *s, int c, size_t n)
+ *
+ * a0: start of area to clear
+ * a1: char to fill with
+ * a2: size of area to clear
+ */
+
+LEAF(memset)
+	beqz		a1, 1f
+	move		v0, a0			/* result */
+
+	andi		a1, 0xff		/* spread fillword */
+	LONG_SLL		t1, a1, 8
+	or		a1, t1
+	LONG_SLL		t1, a1, 16
+#if LONGSIZE == 8
+	or		a1, t1
+	LONG_SLL		t1, a1, 32
+#endif
+	or		a1, t1
+1:
+FEXPORT(__bzero)
+	__BUILD_BZERO LEGACY_MODE
-- 
1.8.5.3

[PATCH 21/58] MIPS: lib: memset: Use macro to build the __bzero symbol

[Markos Chandras]

Build the __bzero symbol using a macor. In EVA mode we will
need to use similar code to do the userspace load operations so
it is better if we use a macro to avoid code duplications.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/memset.S | 95 +++++++++++++++++++++++++++++++-------------------
 1 file changed, 60 insertions(+), 35 deletions(-)

diff --git a/arch/mips/lib/memset.S b/arch/mips/lib/memset.S
index d857985..05fac19 100644
--- a/arch/mips/lib/memset.S
+++ b/arch/mips/lib/memset.S
@@ -34,6 +34,9 @@
 #define FILLPTRG t0
 #endif
 
+#define LEGACY_MODE 1
+#define EVA_MODE    2
+
 #define EX(insn,reg,addr,handler)			\
 9:	insn	reg, addr;				\
 	.section __ex_table,"a";			\
@@ -63,33 +66,23 @@
 #endif
 	.endm
 
-/*
- * memset(void *s, int c, size_t n)
- *
- * a0: start of area to clear
- * a1: char to fill with
- * a2: size of area to clear
- */
 	.set	noreorder
 	.align	5
-LEAF(memset)
-	beqz		a1, 1f
-	move		v0, a0			/* result */
 
-	andi		a1, 0xff		/* spread fillword */
-	LONG_SLL		t1, a1, 8
-	or		a1, t1
-	LONG_SLL		t1, a1, 16
-#if LONGSIZE == 8
-	or		a1, t1
-	LONG_SLL		t1, a1, 32
-#endif
-	or		a1, t1
-1:
+	/*
+	 * Macro to generate the __bzero{,_user} symbol
+	 * Arguments:
+	 * mode: LEGACY_MODE or EVA_MODE
+	 */
+	.macro __BUILD_BZERO mode
+	/* Initialize __memset if this is the first time we call this macro */
+	.ifnotdef __memset
+	.set __memset, 1
+	.hidden __memset /* Make sure it does not leak */
+	.endif
 
-FEXPORT(__bzero)
 	sltiu		t0, a2, STORSIZE	/* very small region? */
-	bnez		t0, .Lsmall_memset
+	bnez		t0, .Lsmall_memset\@
 	andi		t0, a0, STORMASK	/* aligned? */
 
 #ifdef CONFIG_CPU_MICROMIPS
@@ -109,28 +102,28 @@ FEXPORT(__bzero)
 
 	R10KCBARRIER(0(ra))
 #ifdef __MIPSEB__
-	EX(LONG_S_L, a1, (a0), .Lfirst_fixup)	/* make word/dword aligned */
+	EX(LONG_S_L, a1, (a0), .Lfirst_fixup\@)	/* make word/dword aligned */
 #endif
 #ifdef __MIPSEL__
-	EX(LONG_S_R, a1, (a0), .Lfirst_fixup)	/* make word/dword aligned */
+	EX(LONG_S_R, a1, (a0), .Lfirst_fixup\@)	/* make word/dword aligned */
 #endif
 	PTR_SUBU	a0, t0			/* long align ptr */
 	PTR_ADDU	a2, t0			/* correct size */
 
 1:	ori		t1, a2, 0x3f		/* # of full blocks */
 	xori		t1, 0x3f
-	beqz		t1, .Lmemset_partial	/* no block to fill */
+	beqz		t1, .Lmemset_partial\@	/* no block to fill */
 	andi		t0, a2, 0x40-STORSIZE
 
 	PTR_ADDU	t1, a0			/* end address */
 	.set		reorder
 1:	PTR_ADDIU	a0, 64
 	R10KCBARRIER(0(ra))
-	f_fill64 a0, -64, FILL64RG, .Lfwd_fixup
+	f_fill64 a0, -64, FILL64RG, .Lfwd_fixup\@
 	bne		t1, a0, 1b
 	.set		noreorder
 
-.Lmemset_partial:
+.Lmemset_partial\@:
 	R10KCBARRIER(0(ra))
 	PTR_LA		t1, 2f			/* where to start */
 #ifdef CONFIG_CPU_MICROMIPS
@@ -150,7 +143,8 @@ FEXPORT(__bzero)
 	.set		push
 	.set		noreorder
 	.set		nomacro
-	f_fill64 a0, -64, FILL64RG, .Lpartial_fixup	/* ... but first do longs ... */
+	/* ... but first do longs ... */
+	f_fill64 a0, -64, FILL64RG, .Lpartial_fixup\@
 2:	.set		pop
 	andi		a2, STORMASK		/* At most one long to go */
 
@@ -158,15 +152,15 @@ FEXPORT(__bzero)
 	PTR_ADDU	a0, a2			/* What's left */
 	R10KCBARRIER(0(ra))
 #ifdef __MIPSEB__
-	EX(LONG_S_R, a1, -1(a0), .Llast_fixup)
+	EX(LONG_S_R, a1, -1(a0), .Llast_fixup\@)
 #endif
 #ifdef __MIPSEL__
-	EX(LONG_S_L, a1, -1(a0), .Llast_fixup)
+	EX(LONG_S_L, a1, -1(a0), .Llast_fixup\@)
 #endif
 1:	jr		ra
 	move		a2, zero
 
-.Lsmall_memset:
+.Lsmall_memset\@:
 	beqz		a2, 2f
 	PTR_ADDU	t1, a0, a2
 
@@ -177,13 +171,17 @@ FEXPORT(__bzero)
 
 2:	jr		ra			/* done */
 	move		a2, zero
+	.if __memset == 1
 	END(memset)
+	.set __memset, 0
+	.hidden __memset
+	.endif
 
-.Lfirst_fixup:
+.Lfirst_fixup\@:
 	jr	ra
 	nop
 
-.Lfwd_fixup:
+.Lfwd_fixup\@:
 	PTR_L		t0, TI_TASK($28)
 	andi		a2, 0x3f
 	LONG_L		t0, THREAD_BUADDR(t0)
@@ -191,7 +189,7 @@ FEXPORT(__bzero)
 	jr		ra
 	LONG_SUBU	a2, t0
 
-.Lpartial_fixup:
+.Lpartial_fixup\@:
 	PTR_L		t0, TI_TASK($28)
 	andi		a2, STORMASK
 	LONG_L		t0, THREAD_BUADDR(t0)
@@ -199,6 +197,33 @@ FEXPORT(__bzero)
 	jr		ra
 	LONG_SUBU	a2, t0
 
-.Llast_fixup:
+.Llast_fixup\@:
 	jr		ra
 	andi		v1, a2, STORMASK
+
+	.endm
+
+/*
+ * memset(void *s, int c, size_t n)
+ *
+ * a0: start of area to clear
+ * a1: char to fill with
+ * a2: size of area to clear
+ */
+
+LEAF(memset)
+	beqz		a1, 1f
+	move		v0, a0			/* result */
+
+	andi		a1, 0xff		/* spread fillword */
+	LONG_SLL		t1, a1, 8
+	or		a1, t1
+	LONG_SLL		t1, a1, 16
+#if LONGSIZE == 8
+	or		a1, t1
+	LONG_SLL		t1, a1, 32
+#endif
+	or		a1, t1
+1:
+FEXPORT(__bzero)
+	__BUILD_BZERO LEGACY_MODE
-- 
1.8.5.3

[PATCH 22/58] MIPS: lib: memset: Add EVA support for the __bzero function.

[Markos Chandras]

Build the __bzero function using the EVA load/store instructions
when operating in the EVA mode. This function is only used when
accessing user code so there is no need to build two distinct symbols
for user and kernel operations respectively.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/memset.S | 27 +++++++++++++++++++++++----
 1 file changed, 23 insertions(+), 4 deletions(-)

diff --git a/arch/mips/lib/memset.S b/arch/mips/lib/memset.S
index 05fac19..7b0e546 100644
--- a/arch/mips/lib/memset.S
+++ b/arch/mips/lib/memset.S
@@ -37,13 +37,24 @@
 #define LEGACY_MODE 1
 #define EVA_MODE    2
 
+/*
+ * No need to protect it with EVA #ifdefery. The generated block of code
+ * will never be assembled if EVA is not enabled.
+ */
+#define __EVAFY(insn, reg, addr) __BUILD_EVA_INSN(insn##e, reg, addr)
+#define ___BUILD_EVA_INSN(insn, reg, addr) __EVAFY(insn, reg, addr)
+
 #define EX(insn,reg,addr,handler)			\
-9:	insn	reg, addr;				\
+	.if \mode == LEGACY_MODE;			\
+9:		insn	reg, addr;			\
+	.else;						\
+9:		___BUILD_EVA_INSN(insn, reg, addr);	\
+	.endif;						\
 	.section __ex_table,"a";			\
 	PTR	9b, handler;				\
 	.previous
 
-	.macro	f_fill64 dst, offset, val, fixup
+	.macro	f_fill64 dst, offset, val, fixup, mode
 	EX(LONG_S, \val, (\offset +  0 * STORSIZE)(\dst), \fixup)
 	EX(LONG_S, \val, (\offset +  1 * STORSIZE)(\dst), \fixup)
 	EX(LONG_S, \val, (\offset +  2 * STORSIZE)(\dst), \fixup)
@@ -119,7 +130,7 @@
 	.set		reorder
 1:	PTR_ADDIU	a0, 64
 	R10KCBARRIER(0(ra))
-	f_fill64 a0, -64, FILL64RG, .Lfwd_fixup\@
+	f_fill64 a0, -64, FILL64RG, .Lfwd_fixup\@, \mode
 	bne		t1, a0, 1b
 	.set		noreorder
 
@@ -144,7 +155,7 @@
 	.set		noreorder
 	.set		nomacro
 	/* ... but first do longs ... */
-	f_fill64 a0, -64, FILL64RG, .Lpartial_fixup\@
+	f_fill64 a0, -64, FILL64RG, .Lpartial_fixup\@, \mode
 2:	.set		pop
 	andi		a2, STORMASK		/* At most one long to go */
 
@@ -225,5 +236,13 @@ LEAF(memset)
 #endif
 	or		a1, t1
 1:
+#ifndef CONFIG_EVA
 FEXPORT(__bzero)
+#endif
 	__BUILD_BZERO LEGACY_MODE
+
+#ifdef CONFIG_EVA
+LEAF(__bzero)
+	__BUILD_BZERO EVA_MODE
+END(__bzero)
+#endif
-- 
1.8.5.3

[PATCH 22/58] MIPS: lib: memset: Add EVA support for the __bzero function.

[Markos Chandras]

Build the __bzero function using the EVA load/store instructions
when operating in the EVA mode. This function is only used when
accessing user code so there is no need to build two distinct symbols
for user and kernel operations respectively.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/memset.S | 27 +++++++++++++++++++++++----
 1 file changed, 23 insertions(+), 4 deletions(-)

diff --git a/arch/mips/lib/memset.S b/arch/mips/lib/memset.S
index 05fac19..7b0e546 100644
--- a/arch/mips/lib/memset.S
+++ b/arch/mips/lib/memset.S
@@ -37,13 +37,24 @@
 #define LEGACY_MODE 1
 #define EVA_MODE    2
 
+/*
+ * No need to protect it with EVA #ifdefery. The generated block of code
+ * will never be assembled if EVA is not enabled.
+ */
+#define __EVAFY(insn, reg, addr) __BUILD_EVA_INSN(insn##e, reg, addr)
+#define ___BUILD_EVA_INSN(insn, reg, addr) __EVAFY(insn, reg, addr)
+
 #define EX(insn,reg,addr,handler)			\
-9:	insn	reg, addr;				\
+	.if \mode == LEGACY_MODE;			\
+9:		insn	reg, addr;			\
+	.else;						\
+9:		___BUILD_EVA_INSN(insn, reg, addr);	\
+	.endif;						\
 	.section __ex_table,"a";			\
 	PTR	9b, handler;				\
 	.previous
 
-	.macro	f_fill64 dst, offset, val, fixup
+	.macro	f_fill64 dst, offset, val, fixup, mode
 	EX(LONG_S, \val, (\offset +  0 * STORSIZE)(\dst), \fixup)
 	EX(LONG_S, \val, (\offset +  1 * STORSIZE)(\dst), \fixup)
 	EX(LONG_S, \val, (\offset +  2 * STORSIZE)(\dst), \fixup)
@@ -119,7 +130,7 @@
 	.set		reorder
 1:	PTR_ADDIU	a0, 64
 	R10KCBARRIER(0(ra))
-	f_fill64 a0, -64, FILL64RG, .Lfwd_fixup\@
+	f_fill64 a0, -64, FILL64RG, .Lfwd_fixup\@, \mode
 	bne		t1, a0, 1b
 	.set		noreorder
 
@@ -144,7 +155,7 @@
 	.set		noreorder
 	.set		nomacro
 	/* ... but first do longs ... */
-	f_fill64 a0, -64, FILL64RG, .Lpartial_fixup\@
+	f_fill64 a0, -64, FILL64RG, .Lpartial_fixup\@, \mode
 2:	.set		pop
 	andi		a2, STORMASK		/* At most one long to go */
 
@@ -225,5 +236,13 @@ LEAF(memset)
 #endif
 	or		a1, t1
 1:
+#ifndef CONFIG_EVA
 FEXPORT(__bzero)
+#endif
 	__BUILD_BZERO LEGACY_MODE
+
+#ifdef CONFIG_EVA
+LEAF(__bzero)
+	__BUILD_BZERO EVA_MODE
+END(__bzero)
+#endif
-- 
1.8.5.3

[PATCH 23/58] MIPS: asm: uaccess: Add instruction argument to __{put,get}_user_asm

[Markos Chandras]

In preparation for EVA support, an instruction argument is needed
for the __get_user_asm{,_ll32} functions to allow instruction overrides in
EVA mode. Even though EVA only works for MIPS 32-bit, both codepaths are
changed (32-bit and 64-bit) for consistency reasons.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/uaccess.h | 26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index f3fa375..5ba3933 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -223,10 +223,10 @@ struct __large_struct { unsigned long buf[100]; };
  * for 32 bit mode and old iron.
  */
 #ifdef CONFIG_32BIT
-#define __GET_USER_DW(val, ptr) __get_user_asm_ll32(val, ptr)
+#define __GET_USER_DW(val, insn, ptr) __get_user_asm_ll32(val, insn, ptr)
 #endif
 #ifdef CONFIG_64BIT
-#define __GET_USER_DW(val, ptr) __get_user_asm(val, "ld", ptr)
+#define __GET_USER_DW(val, insn, ptr) __get_user_asm(val, "ld", ptr)
 #endif
 
 extern void __get_user_unknown(void);
@@ -237,7 +237,7 @@ do {									\
 	case 1: __get_user_asm(val, "lb", ptr); break;			\
 	case 2: __get_user_asm(val, "lh", ptr); break;			\
 	case 4: __get_user_asm(val, "lw", ptr); break;			\
-	case 8: __GET_USER_DW(val, ptr); break;				\
+	case 8: __GET_USER_DW(val, "lw", ptr); break;			\
 	default: __get_user_unknown(); break;				\
 	}								\
 } while (0)
@@ -287,7 +287,7 @@ do {									\
 /*
  * Get a long long 64 using 32 bit registers.
  */
-#define __get_user_asm_ll32(val, addr)					\
+#define __get_user_asm_ll32(val, insn, addr)				\
 {									\
 	union {								\
 		unsigned long long	l;				\
@@ -295,8 +295,8 @@ do {									\
 	} __gu_tmp;							\
 									\
 	__asm__ __volatile__(						\
-	"1:	lw	%1, (%3)				\n"	\
-	"2:	lw	%D1, 4(%3)				\n"	\
+	"1:	" insn "	%1, (%3)			\n"	\
+	"2:	" insn "	%D1, 4(%3)			\n"	\
 	"3:							\n"	\
 	"	.insn						\n"	\
 	"	.section	.fixup,\"ax\"			\n"	\
@@ -320,10 +320,10 @@ do {									\
  * for 32 bit mode and old iron.
  */
 #ifdef CONFIG_32BIT
-#define __PUT_USER_DW(ptr) __put_user_asm_ll32(ptr)
+#define __PUT_USER_DW(insn, ptr) __put_user_asm_ll32(insn, ptr)
 #endif
 #ifdef CONFIG_64BIT
-#define __PUT_USER_DW(ptr) __put_user_asm("sd", ptr)
+#define __PUT_USER_DW(insn, ptr) __put_user_asm("sd", ptr)
 #endif
 
 #define __put_user_nocheck(x, ptr, size)				\
@@ -337,7 +337,7 @@ do {									\
 	case 1: __put_user_asm("sb", ptr); break;			\
 	case 2: __put_user_asm("sh", ptr); break;			\
 	case 4: __put_user_asm("sw", ptr); break;			\
-	case 8: __PUT_USER_DW(ptr); break;				\
+	case 8: __PUT_USER_DW("sw", ptr); break;			\
 	default: __put_user_unknown(); break;				\
 	}								\
 	__pu_err;							\
@@ -355,7 +355,7 @@ do {									\
 		case 1: __put_user_asm("sb", __pu_addr); break;		\
 		case 2: __put_user_asm("sh", __pu_addr); break;		\
 		case 4: __put_user_asm("sw", __pu_addr); break;		\
-		case 8: __PUT_USER_DW(__pu_addr); break;		\
+		case 8: __PUT_USER_DW("sw", __pu_addr); break;		\
 		default: __put_user_unknown(); break;			\
 		}							\
 	}								\
@@ -380,11 +380,11 @@ do {									\
 	  "i" (-EFAULT));						\
 }
 
-#define __put_user_asm_ll32(ptr)					\
+#define __put_user_asm_ll32(insn, ptr)					\
 {									\
 	__asm__ __volatile__(						\
-	"1:	sw	%2, (%3)	# __put_user_asm_ll32	\n"	\
-	"2:	sw	%D2, 4(%3)				\n"	\
+	"1:	" insn "	%2, (%3)# __put_user_asm_ll32	\n"	\
+	"2:	" insn "	%D2, 4(%3)			\n"	\
 	"3:							\n"	\
 	"	.insn						\n"	\
 	"	.section	.fixup,\"ax\"			\n"	\
-- 
1.8.5.3

[PATCH 23/58] MIPS: asm: uaccess: Add instruction argument to __{put,get}_user_asm

[Markos Chandras]

In preparation for EVA support, an instruction argument is needed
for the __get_user_asm{,_ll32} functions to allow instruction overrides in
EVA mode. Even though EVA only works for MIPS 32-bit, both codepaths are
changed (32-bit and 64-bit) for consistency reasons.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/uaccess.h | 26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index f3fa375..5ba3933 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -223,10 +223,10 @@ struct __large_struct { unsigned long buf[100]; };
  * for 32 bit mode and old iron.
  */
 #ifdef CONFIG_32BIT
-#define __GET_USER_DW(val, ptr) __get_user_asm_ll32(val, ptr)
+#define __GET_USER_DW(val, insn, ptr) __get_user_asm_ll32(val, insn, ptr)
 #endif
 #ifdef CONFIG_64BIT
-#define __GET_USER_DW(val, ptr) __get_user_asm(val, "ld", ptr)
+#define __GET_USER_DW(val, insn, ptr) __get_user_asm(val, "ld", ptr)
 #endif
 
 extern void __get_user_unknown(void);
@@ -237,7 +237,7 @@ do {									\
 	case 1: __get_user_asm(val, "lb", ptr); break;			\
 	case 2: __get_user_asm(val, "lh", ptr); break;			\
 	case 4: __get_user_asm(val, "lw", ptr); break;			\
-	case 8: __GET_USER_DW(val, ptr); break;				\
+	case 8: __GET_USER_DW(val, "lw", ptr); break;			\
 	default: __get_user_unknown(); break;				\
 	}								\
 } while (0)
@@ -287,7 +287,7 @@ do {									\
 /*
  * Get a long long 64 using 32 bit registers.
  */
-#define __get_user_asm_ll32(val, addr)					\
+#define __get_user_asm_ll32(val, insn, addr)				\
 {									\
 	union {								\
 		unsigned long long	l;				\
@@ -295,8 +295,8 @@ do {									\
 	} __gu_tmp;							\
 									\
 	__asm__ __volatile__(						\
-	"1:	lw	%1, (%3)				\n"	\
-	"2:	lw	%D1, 4(%3)				\n"	\
+	"1:	" insn "	%1, (%3)			\n"	\
+	"2:	" insn "	%D1, 4(%3)			\n"	\
 	"3:							\n"	\
 	"	.insn						\n"	\
 	"	.section	.fixup,\"ax\"			\n"	\
@@ -320,10 +320,10 @@ do {									\
  * for 32 bit mode and old iron.
  */
 #ifdef CONFIG_32BIT
-#define __PUT_USER_DW(ptr) __put_user_asm_ll32(ptr)
+#define __PUT_USER_DW(insn, ptr) __put_user_asm_ll32(insn, ptr)
 #endif
 #ifdef CONFIG_64BIT
-#define __PUT_USER_DW(ptr) __put_user_asm("sd", ptr)
+#define __PUT_USER_DW(insn, ptr) __put_user_asm("sd", ptr)
 #endif
 
 #define __put_user_nocheck(x, ptr, size)				\
@@ -337,7 +337,7 @@ do {									\
 	case 1: __put_user_asm("sb", ptr); break;			\
 	case 2: __put_user_asm("sh", ptr); break;			\
 	case 4: __put_user_asm("sw", ptr); break;			\
-	case 8: __PUT_USER_DW(ptr); break;				\
+	case 8: __PUT_USER_DW("sw", ptr); break;			\
 	default: __put_user_unknown(); break;				\
 	}								\
 	__pu_err;							\
@@ -355,7 +355,7 @@ do {									\
 		case 1: __put_user_asm("sb", __pu_addr); break;		\
 		case 2: __put_user_asm("sh", __pu_addr); break;		\
 		case 4: __put_user_asm("sw", __pu_addr); break;		\
-		case 8: __PUT_USER_DW(__pu_addr); break;		\
+		case 8: __PUT_USER_DW("sw", __pu_addr); break;		\
 		default: __put_user_unknown(); break;			\
 		}							\
 	}								\
@@ -380,11 +380,11 @@ do {									\
 	  "i" (-EFAULT));						\
 }
 
-#define __put_user_asm_ll32(ptr)					\
+#define __put_user_asm_ll32(insn, ptr)					\
 {									\
 	__asm__ __volatile__(						\
-	"1:	sw	%2, (%3)	# __put_user_asm_ll32	\n"	\
-	"2:	sw	%D2, 4(%3)				\n"	\
+	"1:	" insn "	%2, (%3)# __put_user_asm_ll32	\n"	\
+	"2:	" insn "	%D2, 4(%3)			\n"	\
 	"3:							\n"	\
 	"	.insn						\n"	\
 	"	.section	.fixup,\"ax\"			\n"	\
-- 
1.8.5.3

[PATCH 24/58] MIPS: asm: uaccess: Move duplicated code to common function

[Markos Chandras]

Similar to __get_user_* functions, move common code to
__put_user_*_common so it can be shared among similar users.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/uaccess.h | 61 ++++++++++++++++++++---------------------
 1 file changed, 29 insertions(+), 32 deletions(-)

diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index 5ba3933..0df57e7 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -326,13 +326,8 @@ do {									\
 #define __PUT_USER_DW(insn, ptr) __put_user_asm("sd", ptr)
 #endif
 
-#define __put_user_nocheck(x, ptr, size)				\
-({									\
-	__typeof__(*(ptr)) __pu_val;					\
-	int __pu_err = 0;						\
-									\
-	__chk_user_ptr(ptr);						\
-	__pu_val = (x);							\
+#define __put_user_common(ptr, size)					\
+do {									\
 	switch (size) {							\
 	case 1: __put_user_asm("sb", ptr); break;			\
 	case 2: __put_user_asm("sh", ptr); break;			\
@@ -340,6 +335,16 @@ do {									\
 	case 8: __PUT_USER_DW("sw", ptr); break;			\
 	default: __put_user_unknown(); break;				\
 	}								\
+} while (0)
+
+#define __put_user_nocheck(x, ptr, size)				\
+({									\
+	__typeof__(*(ptr)) __pu_val;					\
+	int __pu_err = 0;						\
+									\
+	__chk_user_ptr(ptr);						\
+	__pu_val = (x);							\
+	__put_user_common(ptr, size);					\
 	__pu_err;							\
 })
 
@@ -350,15 +355,9 @@ do {									\
 	int __pu_err = -EFAULT;						\
 									\
 	might_fault();							\
-	if (likely(access_ok(VERIFY_WRITE,  __pu_addr, size))) {	\
-		switch (size) {						\
-		case 1: __put_user_asm("sb", __pu_addr); break;		\
-		case 2: __put_user_asm("sh", __pu_addr); break;		\
-		case 4: __put_user_asm("sw", __pu_addr); break;		\
-		case 8: __PUT_USER_DW("sw", __pu_addr); break;		\
-		default: __put_user_unknown(); break;			\
-		}							\
-	}								\
+	if (likely(access_ok(VERIFY_WRITE,  __pu_addr, size)))		\
+		__put_user_common(__pu_addr, size);			\
+									\
 	__pu_err;							\
 })
 
@@ -594,19 +593,23 @@ do {									\
 #define __PUT_USER_UNALIGNED_DW(ptr) __put_user_unaligned_asm("usd", ptr)
 #endif
 
-#define __put_user_unaligned_nocheck(x,ptr,size)			\
-({									\
-	__typeof__(*(ptr)) __pu_val;					\
-	int __pu_err = 0;						\
-									\
-	__pu_val = (x);							\
+#define __put_user_unaligned_common(ptr, size)				\
+do {									\
 	switch (size) {							\
 	case 1: __put_user_asm("sb", ptr); break;			\
 	case 2: __put_user_unaligned_asm("ush", ptr); break;		\
 	case 4: __put_user_unaligned_asm("usw", ptr); break;		\
 	case 8: __PUT_USER_UNALIGNED_DW(ptr); break;			\
 	default: __put_user_unaligned_unknown(); break;			\
-	}								\
+} while (0)
+
+#define __put_user_unaligned_nocheck(x,ptr,size)			\
+({									\
+	__typeof__(*(ptr)) __pu_val;					\
+	int __pu_err = 0;						\
+									\
+	__pu_val = (x);							\
+	__put_user_unaligned_common(ptr, size);				\
 	__pu_err;							\
 })
 
@@ -616,15 +619,9 @@ do {									\
 	__typeof__(*(ptr)) __pu_val = (x);				\
 	int __pu_err = -EFAULT;						\
 									\
-	if (likely(access_ok(VERIFY_WRITE,  __pu_addr, size))) {	\
-		switch (size) {						\
-		case 1: __put_user_asm("sb", __pu_addr); break;		\
-		case 2: __put_user_unaligned_asm("ush", __pu_addr); break; \
-		case 4: __put_user_unaligned_asm("usw", __pu_addr); break; \
-		case 8: __PUT_USER_UNALGINED_DW(__pu_addr); break;	\
-		default: __put_user_unaligned_unknown(); break;		\
-		}							\
-	}								\
+	if (likely(access_ok(VERIFY_WRITE,  __pu_addr, size)))		\
+		__put_user_unaligned_common(__pu_addr, size);		\
+									\
 	__pu_err;							\
 })
 
-- 
1.8.5.3

[PATCH 24/58] MIPS: asm: uaccess: Move duplicated code to common function

[Markos Chandras]

Similar to __get_user_* functions, move common code to
__put_user_*_common so it can be shared among similar users.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/uaccess.h | 61 ++++++++++++++++++++---------------------
 1 file changed, 29 insertions(+), 32 deletions(-)

diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index 5ba3933..0df57e7 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -326,13 +326,8 @@ do {									\
 #define __PUT_USER_DW(insn, ptr) __put_user_asm("sd", ptr)
 #endif
 
-#define __put_user_nocheck(x, ptr, size)				\
-({									\
-	__typeof__(*(ptr)) __pu_val;					\
-	int __pu_err = 0;						\
-									\
-	__chk_user_ptr(ptr);						\
-	__pu_val = (x);							\
+#define __put_user_common(ptr, size)					\
+do {									\
 	switch (size) {							\
 	case 1: __put_user_asm("sb", ptr); break;			\
 	case 2: __put_user_asm("sh", ptr); break;			\
@@ -340,6 +335,16 @@ do {									\
 	case 8: __PUT_USER_DW("sw", ptr); break;			\
 	default: __put_user_unknown(); break;				\
 	}								\
+} while (0)
+
+#define __put_user_nocheck(x, ptr, size)				\
+({									\
+	__typeof__(*(ptr)) __pu_val;					\
+	int __pu_err = 0;						\
+									\
+	__chk_user_ptr(ptr);						\
+	__pu_val = (x);							\
+	__put_user_common(ptr, size);					\
 	__pu_err;							\
 })
 
@@ -350,15 +355,9 @@ do {									\
 	int __pu_err = -EFAULT;						\
 									\
 	might_fault();							\
-	if (likely(access_ok(VERIFY_WRITE,  __pu_addr, size))) {	\
-		switch (size) {						\
-		case 1: __put_user_asm("sb", __pu_addr); break;		\
-		case 2: __put_user_asm("sh", __pu_addr); break;		\
-		case 4: __put_user_asm("sw", __pu_addr); break;		\
-		case 8: __PUT_USER_DW("sw", __pu_addr); break;		\
-		default: __put_user_unknown(); break;			\
-		}							\
-	}								\
+	if (likely(access_ok(VERIFY_WRITE,  __pu_addr, size)))		\
+		__put_user_common(__pu_addr, size);			\
+									\
 	__pu_err;							\
 })
 
@@ -594,19 +593,23 @@ do {									\
 #define __PUT_USER_UNALIGNED_DW(ptr) __put_user_unaligned_asm("usd", ptr)
 #endif
 
-#define __put_user_unaligned_nocheck(x,ptr,size)			\
-({									\
-	__typeof__(*(ptr)) __pu_val;					\
-	int __pu_err = 0;						\
-									\
-	__pu_val = (x);							\
+#define __put_user_unaligned_common(ptr, size)				\
+do {									\
 	switch (size) {							\
 	case 1: __put_user_asm("sb", ptr); break;			\
 	case 2: __put_user_unaligned_asm("ush", ptr); break;		\
 	case 4: __put_user_unaligned_asm("usw", ptr); break;		\
 	case 8: __PUT_USER_UNALIGNED_DW(ptr); break;			\
 	default: __put_user_unaligned_unknown(); break;			\
-	}								\
+} while (0)
+
+#define __put_user_unaligned_nocheck(x,ptr,size)			\
+({									\
+	__typeof__(*(ptr)) __pu_val;					\
+	int __pu_err = 0;						\
+									\
+	__pu_val = (x);							\
+	__put_user_unaligned_common(ptr, size);				\
 	__pu_err;							\
 })
 
@@ -616,15 +619,9 @@ do {									\
 	__typeof__(*(ptr)) __pu_val = (x);				\
 	int __pu_err = -EFAULT;						\
 									\
-	if (likely(access_ok(VERIFY_WRITE,  __pu_addr, size))) {	\
-		switch (size) {						\
-		case 1: __put_user_asm("sb", __pu_addr); break;		\
-		case 2: __put_user_unaligned_asm("ush", __pu_addr); break; \
-		case 4: __put_user_unaligned_asm("usw", __pu_addr); break; \
-		case 8: __PUT_USER_UNALGINED_DW(__pu_addr); break;	\
-		default: __put_user_unaligned_unknown(); break;		\
-		}							\
-	}								\
+	if (likely(access_ok(VERIFY_WRITE,  __pu_addr, size)))		\
+		__put_user_unaligned_common(__pu_addr, size);		\
+									\
 	__pu_err;							\
 })
 
-- 
1.8.5.3

[PATCH 25/58] MIPS: asm: uaccess: Disable unaligned access macros for EVA

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

ulb, ulh, ulw are macros which emulate unaligned access for MIPS.
However, no such macros exist for EVA mode, so the only way to do
EVA unaligned accesses is in the ADE exception handler. As a result
of which, disable these macros for EVA.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/uaccess.h | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index 0df57e7..9c6f5ce 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -402,6 +402,11 @@ do {									\
 extern void __put_user_unknown(void);
 
 /*
+ * ul{b,h,w} are macros and there are no equivalent macros for EVA.
+ * EVA unaligned access is handled in the ADE exception handler.
+ */
+#ifndef CONFIG_EVA
+/*
  * put_user_unaligned: - Write a simple value into user space.
  * @x:	 Value to copy to user space.
  * @ptr: Destination address, in user space.
@@ -666,6 +671,7 @@ do {									\
 }
 
 extern void __put_user_unaligned_unknown(void);
+#endif
 
 /*
  * We're generating jump to subroutines which will be outside the range of
-- 
1.8.5.3

[PATCH 25/58] MIPS: asm: uaccess: Disable unaligned access macros for EVA

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

ulb, ulh, ulw are macros which emulate unaligned access for MIPS.
However, no such macros exist for EVA mode, so the only way to do
EVA unaligned accesses is in the ADE exception handler. As a result
of which, disable these macros for EVA.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/uaccess.h | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index 0df57e7..9c6f5ce 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -402,6 +402,11 @@ do {									\
 extern void __put_user_unknown(void);
 
 /*
+ * ul{b,h,w} are macros and there are no equivalent macros for EVA.
+ * EVA unaligned access is handled in the ADE exception handler.
+ */
+#ifndef CONFIG_EVA
+/*
  * put_user_unaligned: - Write a simple value into user space.
  * @x:	 Value to copy to user space.
  * @ptr: Destination address, in user space.
@@ -666,6 +671,7 @@ do {									\
 }
 
 extern void __put_user_unaligned_unknown(void);
+#endif
 
 /*
  * We're generating jump to subroutines which will be outside the range of
-- 
1.8.5.3

[PATCH 26/58] MIPS: asm: uaccess: Use EVA instructions wrappers

[Markos Chandras]

Use the EVA instruction wrappers from asm.h to perform
read/write operations from userland.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/uaccess.h | 133 ++++++++++++++++++++++++++++++++--------
 1 file changed, 109 insertions(+), 24 deletions(-)

diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index 9c6f5ce..522fc41 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -6,6 +6,7 @@
  * Copyright (C) 1996, 1997, 1998, 1999, 2000, 03, 04 by Ralf Baechle
  * Copyright (C) 1999, 2000 Silicon Graphics, Inc.
  * Copyright (C) 2007  Maciej W. Rozycki
+ * Copyright (C) 2014, Imagination Technologies Ltd.
  */
 #ifndef _ASM_UACCESS_H
 #define _ASM_UACCESS_H
@@ -13,6 +14,7 @@
 #include <linux/kernel.h>
 #include <linux/errno.h>
 #include <linux/thread_info.h>
+#include <asm/asm.h>
 
 /*
  * The fs value determines whether argument validity checking should be
@@ -222,11 +224,44 @@ struct __large_struct { unsigned long buf[100]; };
  * Yuck.  We need two variants, one for 64bit operation and one
  * for 32 bit mode and old iron.
  */
+#ifndef CONFIG_EVA
+#define __get_kernel_common(val, size, ptr) __get_user_common(val, size, ptr)
+#else
+/*
+ * Kernel specific functions for EVA. We need to use normal load instructions
+ * to read data from kernel when operating in EVA mode. We use these macros to
+ * avoid redefining __get_user_asm for EVA.
+ */
+#undef _loadd
+#undef _loadw
+#undef _loadh
+#undef _loadb
+#ifdef CONFIG_32BIT
+#define _loadd			_loadw
+#else
+#define _loadd(reg, addr)	"ld " reg ", " addr
+#endif
+#define _loadw(reg, addr)	"lw " reg ", " addr
+#define _loadh(reg, addr)	"lh " reg ", " addr
+#define _loadb(reg, addr)	"lb " reg ", " addr
+
+#define __get_kernel_common(val, size, ptr)				\
+do {									\
+	switch (size) {							\
+	case 1: __get_user_asm(val, _loadb, ptr); break;		\
+	case 2: __get_user_asm(val, _loadh, ptr); break;		\
+	case 4: __get_user_asm(val, _loadw, ptr); break;		\
+	case 8: __GET_USER_DW(val, _loadd, ptr); break;			\
+	default: __get_user_unknown(); break;				\
+	}								\
+} while (0)
+#endif
+
 #ifdef CONFIG_32BIT
 #define __GET_USER_DW(val, insn, ptr) __get_user_asm_ll32(val, insn, ptr)
 #endif
 #ifdef CONFIG_64BIT
-#define __GET_USER_DW(val, insn, ptr) __get_user_asm(val, "ld", ptr)
+#define __GET_USER_DW(val, insn, ptr) __get_user_asm(val, insn, ptr)
 #endif
 
 extern void __get_user_unknown(void);
@@ -234,10 +269,10 @@ extern void __get_user_unknown(void);
 #define __get_user_common(val, size, ptr)				\
 do {									\
 	switch (size) {							\
-	case 1: __get_user_asm(val, "lb", ptr); break;			\
-	case 2: __get_user_asm(val, "lh", ptr); break;			\
-	case 4: __get_user_asm(val, "lw", ptr); break;			\
-	case 8: __GET_USER_DW(val, "lw", ptr); break;			\
+	case 1: __get_user_asm(val, user_lb, ptr); break;		\
+	case 2: __get_user_asm(val, user_lh, ptr); break;		\
+	case 4: __get_user_asm(val, user_lw, ptr); break;		\
+	case 8: __GET_USER_DW(val, user_ld, ptr); break;		\
 	default: __get_user_unknown(); break;				\
 	}								\
 } while (0)
@@ -246,8 +281,12 @@ do {									\
 ({									\
 	int __gu_err;							\
 									\
-	__chk_user_ptr(ptr);						\
-	__get_user_common((x), size, ptr);				\
+	if (segment_eq(get_fs(), get_ds())) {				\
+		__get_kernel_common((x), size, ptr);			\
+	} else {							\
+		__chk_user_ptr(ptr);					\
+		__get_user_common((x), size, ptr);			\
+	}								\
 	__gu_err;							\
 })
 
@@ -257,8 +296,12 @@ do {									\
 	const __typeof__(*(ptr)) __user * __gu_ptr = (ptr);		\
 									\
 	might_fault();							\
-	if (likely(access_ok(VERIFY_READ,  __gu_ptr, size)))		\
-		__get_user_common((x), size, __gu_ptr);			\
+	if (likely(access_ok(VERIFY_READ,  __gu_ptr, size))) {		\
+		if (segment_eq(get_fs(), get_ds()))			\
+			__get_kernel_common((x), size, __gu_ptr);	\
+		else							\
+			__get_user_common((x), size, __gu_ptr);		\
+	}								\
 									\
 	__gu_err;							\
 })
@@ -268,7 +311,7 @@ do {									\
 	long __gu_tmp;							\
 									\
 	__asm__ __volatile__(						\
-	"1:	" insn "	%1, %3				\n"	\
+	"1:	"insn("%1", "%3")"				\n"	\
 	"2:							\n"	\
 	"	.insn						\n"	\
 	"	.section .fixup,\"ax\"				\n"	\
@@ -295,8 +338,8 @@ do {									\
 	} __gu_tmp;							\
 									\
 	__asm__ __volatile__(						\
-	"1:	" insn "	%1, (%3)			\n"	\
-	"2:	" insn "	%D1, 4(%3)			\n"	\
+	"1:	" insn("%1", "(%3)")"				\n"	\
+	"2:	" insn("%D1", "4(%3)")"				\n"	\
 	"3:							\n"	\
 	"	.insn						\n"	\
 	"	.section	.fixup,\"ax\"			\n"	\
@@ -315,6 +358,40 @@ do {									\
 	(val) = __gu_tmp.t;						\
 }
 
+#ifndef CONFIG_EVA
+#define __put_kernel_common(ptr, size) __put_user_common(ptr, size)
+#else
+/*
+ * Kernel specific functions for EVA. We need to use normal load instructions
+ * to read data from kernel when operating in EVA mode. We use these macros to
+ * avoid redefining __get_user_asm for EVA.
+ */
+#undef _stored
+#undef _storew
+#undef _storeh
+#undef _storeb
+#ifdef CONFIG_32BIT
+#define _stored			_storew
+#else
+#define _stored(reg, addr)	"ld " reg ", " addr
+#endif
+
+#define _storew(reg, addr)	"sw " reg ", " addr
+#define _storeh(reg, addr)	"sh " reg ", " addr
+#define _storeb(reg, addr)	"sb " reg ", " addr
+
+#define __put_kernel_common(ptr, size)					\
+do {									\
+	switch (size) {							\
+	case 1: __put_user_asm(_storeb, ptr); break;			\
+	case 2: __put_user_asm(_storeh, ptr); break;			\
+	case 4: __put_user_asm(_storew, ptr); break;			\
+	case 8: __PUT_USER_DW(_stored, ptr); break;			\
+	default: __put_user_unknown(); break;				\
+	}								\
+} while(0)
+#endif
+
 /*
  * Yuck.  We need two variants, one for 64bit operation and one
  * for 32 bit mode and old iron.
@@ -323,16 +400,16 @@ do {									\
 #define __PUT_USER_DW(insn, ptr) __put_user_asm_ll32(insn, ptr)
 #endif
 #ifdef CONFIG_64BIT
-#define __PUT_USER_DW(insn, ptr) __put_user_asm("sd", ptr)
+#define __PUT_USER_DW(insn, ptr) __put_user_asm(insn, ptr)
 #endif
 
 #define __put_user_common(ptr, size)					\
 do {									\
 	switch (size) {							\
-	case 1: __put_user_asm("sb", ptr); break;			\
-	case 2: __put_user_asm("sh", ptr); break;			\
-	case 4: __put_user_asm("sw", ptr); break;			\
-	case 8: __PUT_USER_DW("sw", ptr); break;			\
+	case 1: __put_user_asm(user_sb, ptr); break;			\
+	case 2: __put_user_asm(user_sh, ptr); break;			\
+	case 4: __put_user_asm(user_sw, ptr); break;			\
+	case 8: __PUT_USER_DW(user_sd, ptr); break;			\
 	default: __put_user_unknown(); break;				\
 	}								\
 } while (0)
@@ -342,9 +419,13 @@ do {									\
 	__typeof__(*(ptr)) __pu_val;					\
 	int __pu_err = 0;						\
 									\
-	__chk_user_ptr(ptr);						\
 	__pu_val = (x);							\
-	__put_user_common(ptr, size);					\
+	if (segment_eq(get_fs(), get_ds())) {				\
+		__put_kernel_common(ptr, size);				\
+	} else {							\
+		__chk_user_ptr(ptr);					\
+		__put_user_common(ptr, size);				\
+	}								\
 	__pu_err;							\
 })
 
@@ -355,8 +436,12 @@ do {									\
 	int __pu_err = -EFAULT;						\
 									\
 	might_fault();							\
-	if (likely(access_ok(VERIFY_WRITE,  __pu_addr, size)))		\
-		__put_user_common(__pu_addr, size);			\
+	if (likely(access_ok(VERIFY_WRITE,  __pu_addr, size))) {	\
+		if (segment_eq(get_fs(), get_ds()))			\
+			__put_kernel_common(__pu_addr, size);		\
+		else							\
+			__put_user_common(__pu_addr, size);		\
+	}								\
 									\
 	__pu_err;							\
 })
@@ -364,7 +449,7 @@ do {									\
 #define __put_user_asm(insn, ptr)					\
 {									\
 	__asm__ __volatile__(						\
-	"1:	" insn "	%z2, %3		# __put_user_asm\n"	\
+	"1:	"insn("%z2", "%3")"	# __put_user_asm	\n"	\
 	"2:							\n"	\
 	"	.insn						\n"	\
 	"	.section	.fixup,\"ax\"			\n"	\
@@ -382,8 +467,8 @@ do {									\
 #define __put_user_asm_ll32(insn, ptr)					\
 {									\
 	__asm__ __volatile__(						\
-	"1:	" insn "	%2, (%3)# __put_user_asm_ll32	\n"	\
-	"2:	" insn "	%D2, 4(%3)			\n"	\
+	"1:	"insn("%2", "(%3)")"	# __put_user_asm_ll32	\n"	\
+	"2:	"insn("%D2", "4(%3)")"				\n"	\
 	"3:							\n"	\
 	"	.insn						\n"	\
 	"	.section	.fixup,\"ax\"			\n"	\
-- 
1.8.5.3

[PATCH 26/58] MIPS: asm: uaccess: Use EVA instructions wrappers

[Markos Chandras]

Use the EVA instruction wrappers from asm.h to perform
read/write operations from userland.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/uaccess.h | 133 ++++++++++++++++++++++++++++++++--------
 1 file changed, 109 insertions(+), 24 deletions(-)

diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index 9c6f5ce..522fc41 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -6,6 +6,7 @@
  * Copyright (C) 1996, 1997, 1998, 1999, 2000, 03, 04 by Ralf Baechle
  * Copyright (C) 1999, 2000 Silicon Graphics, Inc.
  * Copyright (C) 2007  Maciej W. Rozycki
+ * Copyright (C) 2014, Imagination Technologies Ltd.
  */
 #ifndef _ASM_UACCESS_H
 #define _ASM_UACCESS_H
@@ -13,6 +14,7 @@
 #include <linux/kernel.h>
 #include <linux/errno.h>
 #include <linux/thread_info.h>
+#include <asm/asm.h>
 
 /*
  * The fs value determines whether argument validity checking should be
@@ -222,11 +224,44 @@ struct __large_struct { unsigned long buf[100]; };
  * Yuck.  We need two variants, one for 64bit operation and one
  * for 32 bit mode and old iron.
  */
+#ifndef CONFIG_EVA
+#define __get_kernel_common(val, size, ptr) __get_user_common(val, size, ptr)
+#else
+/*
+ * Kernel specific functions for EVA. We need to use normal load instructions
+ * to read data from kernel when operating in EVA mode. We use these macros to
+ * avoid redefining __get_user_asm for EVA.
+ */
+#undef _loadd
+#undef _loadw
+#undef _loadh
+#undef _loadb
+#ifdef CONFIG_32BIT
+#define _loadd			_loadw
+#else
+#define _loadd(reg, addr)	"ld " reg ", " addr
+#endif
+#define _loadw(reg, addr)	"lw " reg ", " addr
+#define _loadh(reg, addr)	"lh " reg ", " addr
+#define _loadb(reg, addr)	"lb " reg ", " addr
+
+#define __get_kernel_common(val, size, ptr)				\
+do {									\
+	switch (size) {							\
+	case 1: __get_user_asm(val, _loadb, ptr); break;		\
+	case 2: __get_user_asm(val, _loadh, ptr); break;		\
+	case 4: __get_user_asm(val, _loadw, ptr); break;		\
+	case 8: __GET_USER_DW(val, _loadd, ptr); break;			\
+	default: __get_user_unknown(); break;				\
+	}								\
+} while (0)
+#endif
+
 #ifdef CONFIG_32BIT
 #define __GET_USER_DW(val, insn, ptr) __get_user_asm_ll32(val, insn, ptr)
 #endif
 #ifdef CONFIG_64BIT
-#define __GET_USER_DW(val, insn, ptr) __get_user_asm(val, "ld", ptr)
+#define __GET_USER_DW(val, insn, ptr) __get_user_asm(val, insn, ptr)
 #endif
 
 extern void __get_user_unknown(void);
@@ -234,10 +269,10 @@ extern void __get_user_unknown(void);
 #define __get_user_common(val, size, ptr)				\
 do {									\
 	switch (size) {							\
-	case 1: __get_user_asm(val, "lb", ptr); break;			\
-	case 2: __get_user_asm(val, "lh", ptr); break;			\
-	case 4: __get_user_asm(val, "lw", ptr); break;			\
-	case 8: __GET_USER_DW(val, "lw", ptr); break;			\
+	case 1: __get_user_asm(val, user_lb, ptr); break;		\
+	case 2: __get_user_asm(val, user_lh, ptr); break;		\
+	case 4: __get_user_asm(val, user_lw, ptr); break;		\
+	case 8: __GET_USER_DW(val, user_ld, ptr); break;		\
 	default: __get_user_unknown(); break;				\
 	}								\
 } while (0)
@@ -246,8 +281,12 @@ do {									\
 ({									\
 	int __gu_err;							\
 									\
-	__chk_user_ptr(ptr);						\
-	__get_user_common((x), size, ptr);				\
+	if (segment_eq(get_fs(), get_ds())) {				\
+		__get_kernel_common((x), size, ptr);			\
+	} else {							\
+		__chk_user_ptr(ptr);					\
+		__get_user_common((x), size, ptr);			\
+	}								\
 	__gu_err;							\
 })
 
@@ -257,8 +296,12 @@ do {									\
 	const __typeof__(*(ptr)) __user * __gu_ptr = (ptr);		\
 									\
 	might_fault();							\
-	if (likely(access_ok(VERIFY_READ,  __gu_ptr, size)))		\
-		__get_user_common((x), size, __gu_ptr);			\
+	if (likely(access_ok(VERIFY_READ,  __gu_ptr, size))) {		\
+		if (segment_eq(get_fs(), get_ds()))			\
+			__get_kernel_common((x), size, __gu_ptr);	\
+		else							\
+			__get_user_common((x), size, __gu_ptr);		\
+	}								\
 									\
 	__gu_err;							\
 })
@@ -268,7 +311,7 @@ do {									\
 	long __gu_tmp;							\
 									\
 	__asm__ __volatile__(						\
-	"1:	" insn "	%1, %3				\n"	\
+	"1:	"insn("%1", "%3")"				\n"	\
 	"2:							\n"	\
 	"	.insn						\n"	\
 	"	.section .fixup,\"ax\"				\n"	\
@@ -295,8 +338,8 @@ do {									\
 	} __gu_tmp;							\
 									\
 	__asm__ __volatile__(						\
-	"1:	" insn "	%1, (%3)			\n"	\
-	"2:	" insn "	%D1, 4(%3)			\n"	\
+	"1:	" insn("%1", "(%3)")"				\n"	\
+	"2:	" insn("%D1", "4(%3)")"				\n"	\
 	"3:							\n"	\
 	"	.insn						\n"	\
 	"	.section	.fixup,\"ax\"			\n"	\
@@ -315,6 +358,40 @@ do {									\
 	(val) = __gu_tmp.t;						\
 }
 
+#ifndef CONFIG_EVA
+#define __put_kernel_common(ptr, size) __put_user_common(ptr, size)
+#else
+/*
+ * Kernel specific functions for EVA. We need to use normal load instructions
+ * to read data from kernel when operating in EVA mode. We use these macros to
+ * avoid redefining __get_user_asm for EVA.
+ */
+#undef _stored
+#undef _storew
+#undef _storeh
+#undef _storeb
+#ifdef CONFIG_32BIT
+#define _stored			_storew
+#else
+#define _stored(reg, addr)	"ld " reg ", " addr
+#endif
+
+#define _storew(reg, addr)	"sw " reg ", " addr
+#define _storeh(reg, addr)	"sh " reg ", " addr
+#define _storeb(reg, addr)	"sb " reg ", " addr
+
+#define __put_kernel_common(ptr, size)					\
+do {									\
+	switch (size) {							\
+	case 1: __put_user_asm(_storeb, ptr); break;			\
+	case 2: __put_user_asm(_storeh, ptr); break;			\
+	case 4: __put_user_asm(_storew, ptr); break;			\
+	case 8: __PUT_USER_DW(_stored, ptr); break;			\
+	default: __put_user_unknown(); break;				\
+	}								\
+} while(0)
+#endif
+
 /*
  * Yuck.  We need two variants, one for 64bit operation and one
  * for 32 bit mode and old iron.
@@ -323,16 +400,16 @@ do {									\
 #define __PUT_USER_DW(insn, ptr) __put_user_asm_ll32(insn, ptr)
 #endif
 #ifdef CONFIG_64BIT
-#define __PUT_USER_DW(insn, ptr) __put_user_asm("sd", ptr)
+#define __PUT_USER_DW(insn, ptr) __put_user_asm(insn, ptr)
 #endif
 
 #define __put_user_common(ptr, size)					\
 do {									\
 	switch (size) {							\
-	case 1: __put_user_asm("sb", ptr); break;			\
-	case 2: __put_user_asm("sh", ptr); break;			\
-	case 4: __put_user_asm("sw", ptr); break;			\
-	case 8: __PUT_USER_DW("sw", ptr); break;			\
+	case 1: __put_user_asm(user_sb, ptr); break;			\
+	case 2: __put_user_asm(user_sh, ptr); break;			\
+	case 4: __put_user_asm(user_sw, ptr); break;			\
+	case 8: __PUT_USER_DW(user_sd, ptr); break;			\
 	default: __put_user_unknown(); break;				\
 	}								\
 } while (0)
@@ -342,9 +419,13 @@ do {									\
 	__typeof__(*(ptr)) __pu_val;					\
 	int __pu_err = 0;						\
 									\
-	__chk_user_ptr(ptr);						\
 	__pu_val = (x);							\
-	__put_user_common(ptr, size);					\
+	if (segment_eq(get_fs(), get_ds())) {				\
+		__put_kernel_common(ptr, size);				\
+	} else {							\
+		__chk_user_ptr(ptr);					\
+		__put_user_common(ptr, size);				\
+	}								\
 	__pu_err;							\
 })
 
@@ -355,8 +436,12 @@ do {									\
 	int __pu_err = -EFAULT;						\
 									\
 	might_fault();							\
-	if (likely(access_ok(VERIFY_WRITE,  __pu_addr, size)))		\
-		__put_user_common(__pu_addr, size);			\
+	if (likely(access_ok(VERIFY_WRITE,  __pu_addr, size))) {	\
+		if (segment_eq(get_fs(), get_ds()))			\
+			__put_kernel_common(__pu_addr, size);		\
+		else							\
+			__put_user_common(__pu_addr, size);		\
+	}								\
 									\
 	__pu_err;							\
 })
@@ -364,7 +449,7 @@ do {									\
 #define __put_user_asm(insn, ptr)					\
 {									\
 	__asm__ __volatile__(						\
-	"1:	" insn "	%z2, %3		# __put_user_asm\n"	\
+	"1:	"insn("%z2", "%3")"	# __put_user_asm	\n"	\
 	"2:							\n"	\
 	"	.insn						\n"	\
 	"	.section	.fixup,\"ax\"			\n"	\
@@ -382,8 +467,8 @@ do {									\
 #define __put_user_asm_ll32(insn, ptr)					\
 {									\
 	__asm__ __volatile__(						\
-	"1:	" insn "	%2, (%3)# __put_user_asm_ll32	\n"	\
-	"2:	" insn "	%D2, 4(%3)			\n"	\
+	"1:	"insn("%2", "(%3)")"	# __put_user_asm_ll32	\n"	\
+	"2:	"insn("%D2", "4(%3)")"				\n"	\
 	"3:							\n"	\
 	"	.insn						\n"	\
 	"	.section	.fixup,\"ax\"			\n"	\
-- 
1.8.5.3

[PATCH 27/58] MIPS: asm: uaccess: Rename {get,put}_user_asm macros

[Markos Chandras]

The {get,put}_user_asm functions can be used to load data from
kernel or the user address space so rename them to avoid
confusion.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/uaccess.h | 60 ++++++++++++++++++++---------------------
 1 file changed, 30 insertions(+), 30 deletions(-)

diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index 522fc41..59bbebb 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -248,20 +248,20 @@ struct __large_struct { unsigned long buf[100]; };
 #define __get_kernel_common(val, size, ptr)				\
 do {									\
 	switch (size) {							\
-	case 1: __get_user_asm(val, _loadb, ptr); break;		\
-	case 2: __get_user_asm(val, _loadh, ptr); break;		\
-	case 4: __get_user_asm(val, _loadw, ptr); break;		\
-	case 8: __GET_USER_DW(val, _loadd, ptr); break;			\
+	case 1: __get_data_asm(val, _loadb, ptr); break;		\
+	case 2: __get_data_asm(val, _loadh, ptr); break;		\
+	case 4: __get_data_asm(val, _loadw, ptr); break;		\
+	case 8: __GET_DW(val, _loadd, ptr); break;			\
 	default: __get_user_unknown(); break;				\
 	}								\
 } while (0)
 #endif
 
 #ifdef CONFIG_32BIT
-#define __GET_USER_DW(val, insn, ptr) __get_user_asm_ll32(val, insn, ptr)
+#define __GET_DW(val, insn, ptr) __get_data_asm_ll32(val, insn, ptr)
 #endif
 #ifdef CONFIG_64BIT
-#define __GET_USER_DW(val, insn, ptr) __get_user_asm(val, insn, ptr)
+#define __GET_DW(val, insn, ptr) __get_data_asm(val, insn, ptr)
 #endif
 
 extern void __get_user_unknown(void);
@@ -269,10 +269,10 @@ extern void __get_user_unknown(void);
 #define __get_user_common(val, size, ptr)				\
 do {									\
 	switch (size) {							\
-	case 1: __get_user_asm(val, user_lb, ptr); break;		\
-	case 2: __get_user_asm(val, user_lh, ptr); break;		\
-	case 4: __get_user_asm(val, user_lw, ptr); break;		\
-	case 8: __GET_USER_DW(val, user_ld, ptr); break;		\
+	case 1: __get_data_asm(val, user_lb, ptr); break;		\
+	case 2: __get_data_asm(val, user_lh, ptr); break;		\
+	case 4: __get_data_asm(val, user_lw, ptr); break;		\
+	case 8: __GET_DW(val, user_ld, ptr); break;			\
 	default: __get_user_unknown(); break;				\
 	}								\
 } while (0)
@@ -306,7 +306,7 @@ do {									\
 	__gu_err;							\
 })
 
-#define __get_user_asm(val, insn, addr)					\
+#define __get_data_asm(val, insn, addr)					\
 {									\
 	long __gu_tmp;							\
 									\
@@ -330,7 +330,7 @@ do {									\
 /*
  * Get a long long 64 using 32 bit registers.
  */
-#define __get_user_asm_ll32(val, insn, addr)				\
+#define __get_data_asm_ll32(val, insn, addr)				\
 {									\
 	union {								\
 		unsigned long long	l;				\
@@ -364,7 +364,7 @@ do {									\
 /*
  * Kernel specific functions for EVA. We need to use normal load instructions
  * to read data from kernel when operating in EVA mode. We use these macros to
- * avoid redefining __get_user_asm for EVA.
+ * avoid redefining __get_data_asm for EVA.
  */
 #undef _stored
 #undef _storew
@@ -383,10 +383,10 @@ do {									\
 #define __put_kernel_common(ptr, size)					\
 do {									\
 	switch (size) {							\
-	case 1: __put_user_asm(_storeb, ptr); break;			\
-	case 2: __put_user_asm(_storeh, ptr); break;			\
-	case 4: __put_user_asm(_storew, ptr); break;			\
-	case 8: __PUT_USER_DW(_stored, ptr); break;			\
+	case 1: __put_data_asm(_storeb, ptr); break;			\
+	case 2: __put_data_asm(_storeh, ptr); break;			\
+	case 4: __put_data_asm(_storew, ptr); break;			\
+	case 8: __PUT_DW(_stored, ptr); break;				\
 	default: __put_user_unknown(); break;				\
 	}								\
 } while(0)
@@ -397,19 +397,19 @@ do {									\
  * for 32 bit mode and old iron.
  */
 #ifdef CONFIG_32BIT
-#define __PUT_USER_DW(insn, ptr) __put_user_asm_ll32(insn, ptr)
+#define __PUT_DW(insn, ptr) __put_data_asm_ll32(insn, ptr)
 #endif
 #ifdef CONFIG_64BIT
-#define __PUT_USER_DW(insn, ptr) __put_user_asm(insn, ptr)
+#define __PUT_DW(insn, ptr) __put_data_asm(insn, ptr)
 #endif
 
 #define __put_user_common(ptr, size)					\
 do {									\
 	switch (size) {							\
-	case 1: __put_user_asm(user_sb, ptr); break;			\
-	case 2: __put_user_asm(user_sh, ptr); break;			\
-	case 4: __put_user_asm(user_sw, ptr); break;			\
-	case 8: __PUT_USER_DW(user_sd, ptr); break;			\
+	case 1: __put_data_asm(user_sb, ptr); break;			\
+	case 2: __put_data_asm(user_sh, ptr); break;			\
+	case 4: __put_data_asm(user_sw, ptr); break;			\
+	case 8: __PUT_DW(user_sd, ptr); break;				\
 	default: __put_user_unknown(); break;				\
 	}								\
 } while (0)
@@ -446,10 +446,10 @@ do {									\
 	__pu_err;							\
 })
 
-#define __put_user_asm(insn, ptr)					\
+#define __put_data_asm(insn, ptr)					\
 {									\
 	__asm__ __volatile__(						\
-	"1:	"insn("%z2", "%3")"	# __put_user_asm	\n"	\
+	"1:	"insn("%z2", "%3")"	# __put_data_asm	\n"	\
 	"2:							\n"	\
 	"	.insn						\n"	\
 	"	.section	.fixup,\"ax\"			\n"	\
@@ -464,10 +464,10 @@ do {									\
 	  "i" (-EFAULT));						\
 }
 
-#define __put_user_asm_ll32(insn, ptr)					\
+#define __put_data_asm_ll32(insn, ptr)					\
 {									\
 	__asm__ __volatile__(						\
-	"1:	"insn("%2", "(%3)")"	# __put_user_asm_ll32	\n"	\
+	"1:	"insn("%2", "(%3)")"	# __put_data_asm_ll32	\n"	\
 	"2:	"insn("%D2", "4(%3)")"				\n"	\
 	"3:							\n"	\
 	"	.insn						\n"	\
@@ -593,7 +593,7 @@ extern void __get_user_unaligned_unknown(void);
 #define __get_user_unaligned_common(val, size, ptr)			\
 do {									\
 	switch (size) {							\
-	case 1: __get_user_asm(val, "lb", ptr); break;			\
+	case 1: __get_data_asm(val, "lb", ptr); break;			\
 	case 2: __get_user_unaligned_asm(val, "ulh", ptr); break;	\
 	case 4: __get_user_unaligned_asm(val, "ulw", ptr); break;	\
 	case 8: __GET_USER_UNALIGNED_DW(val, ptr); break;		\
@@ -620,7 +620,7 @@ do {									\
 	__gu_err;							\
 })
 
-#define __get_user_unaligned_asm(val, insn, addr)			\
+#define __get_data_unaligned_asm(val, insn, addr)			\
 {									\
 	long __gu_tmp;							\
 									\
@@ -686,7 +686,7 @@ do {									\
 #define __put_user_unaligned_common(ptr, size)				\
 do {									\
 	switch (size) {							\
-	case 1: __put_user_asm("sb", ptr); break;			\
+	case 1: __put_data_asm("sb", ptr); break;			\
 	case 2: __put_user_unaligned_asm("ush", ptr); break;		\
 	case 4: __put_user_unaligned_asm("usw", ptr); break;		\
 	case 8: __PUT_USER_UNALIGNED_DW(ptr); break;			\
-- 
1.8.5.3

[PATCH 27/58] MIPS: asm: uaccess: Rename {get,put}_user_asm macros

[Markos Chandras]

The {get,put}_user_asm functions can be used to load data from
kernel or the user address space so rename them to avoid
confusion.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/uaccess.h | 60 ++++++++++++++++++++---------------------
 1 file changed, 30 insertions(+), 30 deletions(-)

diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index 522fc41..59bbebb 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -248,20 +248,20 @@ struct __large_struct { unsigned long buf[100]; };
 #define __get_kernel_common(val, size, ptr)				\
 do {									\
 	switch (size) {							\
-	case 1: __get_user_asm(val, _loadb, ptr); break;		\
-	case 2: __get_user_asm(val, _loadh, ptr); break;		\
-	case 4: __get_user_asm(val, _loadw, ptr); break;		\
-	case 8: __GET_USER_DW(val, _loadd, ptr); break;			\
+	case 1: __get_data_asm(val, _loadb, ptr); break;		\
+	case 2: __get_data_asm(val, _loadh, ptr); break;		\
+	case 4: __get_data_asm(val, _loadw, ptr); break;		\
+	case 8: __GET_DW(val, _loadd, ptr); break;			\
 	default: __get_user_unknown(); break;				\
 	}								\
 } while (0)
 #endif
 
 #ifdef CONFIG_32BIT
-#define __GET_USER_DW(val, insn, ptr) __get_user_asm_ll32(val, insn, ptr)
+#define __GET_DW(val, insn, ptr) __get_data_asm_ll32(val, insn, ptr)
 #endif
 #ifdef CONFIG_64BIT
-#define __GET_USER_DW(val, insn, ptr) __get_user_asm(val, insn, ptr)
+#define __GET_DW(val, insn, ptr) __get_data_asm(val, insn, ptr)
 #endif
 
 extern void __get_user_unknown(void);
@@ -269,10 +269,10 @@ extern void __get_user_unknown(void);
 #define __get_user_common(val, size, ptr)				\
 do {									\
 	switch (size) {							\
-	case 1: __get_user_asm(val, user_lb, ptr); break;		\
-	case 2: __get_user_asm(val, user_lh, ptr); break;		\
-	case 4: __get_user_asm(val, user_lw, ptr); break;		\
-	case 8: __GET_USER_DW(val, user_ld, ptr); break;		\
+	case 1: __get_data_asm(val, user_lb, ptr); break;		\
+	case 2: __get_data_asm(val, user_lh, ptr); break;		\
+	case 4: __get_data_asm(val, user_lw, ptr); break;		\
+	case 8: __GET_DW(val, user_ld, ptr); break;			\
 	default: __get_user_unknown(); break;				\
 	}								\
 } while (0)
@@ -306,7 +306,7 @@ do {									\
 	__gu_err;							\
 })
 
-#define __get_user_asm(val, insn, addr)					\
+#define __get_data_asm(val, insn, addr)					\
 {									\
 	long __gu_tmp;							\
 									\
@@ -330,7 +330,7 @@ do {									\
 /*
  * Get a long long 64 using 32 bit registers.
  */
-#define __get_user_asm_ll32(val, insn, addr)				\
+#define __get_data_asm_ll32(val, insn, addr)				\
 {									\
 	union {								\
 		unsigned long long	l;				\
@@ -364,7 +364,7 @@ do {									\
 /*
  * Kernel specific functions for EVA. We need to use normal load instructions
  * to read data from kernel when operating in EVA mode. We use these macros to
- * avoid redefining __get_user_asm for EVA.
+ * avoid redefining __get_data_asm for EVA.
  */
 #undef _stored
 #undef _storew
@@ -383,10 +383,10 @@ do {									\
 #define __put_kernel_common(ptr, size)					\
 do {									\
 	switch (size) {							\
-	case 1: __put_user_asm(_storeb, ptr); break;			\
-	case 2: __put_user_asm(_storeh, ptr); break;			\
-	case 4: __put_user_asm(_storew, ptr); break;			\
-	case 8: __PUT_USER_DW(_stored, ptr); break;			\
+	case 1: __put_data_asm(_storeb, ptr); break;			\
+	case 2: __put_data_asm(_storeh, ptr); break;			\
+	case 4: __put_data_asm(_storew, ptr); break;			\
+	case 8: __PUT_DW(_stored, ptr); break;				\
 	default: __put_user_unknown(); break;				\
 	}								\
 } while(0)
@@ -397,19 +397,19 @@ do {									\
  * for 32 bit mode and old iron.
  */
 #ifdef CONFIG_32BIT
-#define __PUT_USER_DW(insn, ptr) __put_user_asm_ll32(insn, ptr)
+#define __PUT_DW(insn, ptr) __put_data_asm_ll32(insn, ptr)
 #endif
 #ifdef CONFIG_64BIT
-#define __PUT_USER_DW(insn, ptr) __put_user_asm(insn, ptr)
+#define __PUT_DW(insn, ptr) __put_data_asm(insn, ptr)
 #endif
 
 #define __put_user_common(ptr, size)					\
 do {									\
 	switch (size) {							\
-	case 1: __put_user_asm(user_sb, ptr); break;			\
-	case 2: __put_user_asm(user_sh, ptr); break;			\
-	case 4: __put_user_asm(user_sw, ptr); break;			\
-	case 8: __PUT_USER_DW(user_sd, ptr); break;			\
+	case 1: __put_data_asm(user_sb, ptr); break;			\
+	case 2: __put_data_asm(user_sh, ptr); break;			\
+	case 4: __put_data_asm(user_sw, ptr); break;			\
+	case 8: __PUT_DW(user_sd, ptr); break;				\
 	default: __put_user_unknown(); break;				\
 	}								\
 } while (0)
@@ -446,10 +446,10 @@ do {									\
 	__pu_err;							\
 })
 
-#define __put_user_asm(insn, ptr)					\
+#define __put_data_asm(insn, ptr)					\
 {									\
 	__asm__ __volatile__(						\
-	"1:	"insn("%z2", "%3")"	# __put_user_asm	\n"	\
+	"1:	"insn("%z2", "%3")"	# __put_data_asm	\n"	\
 	"2:							\n"	\
 	"	.insn						\n"	\
 	"	.section	.fixup,\"ax\"			\n"	\
@@ -464,10 +464,10 @@ do {									\
 	  "i" (-EFAULT));						\
 }
 
-#define __put_user_asm_ll32(insn, ptr)					\
+#define __put_data_asm_ll32(insn, ptr)					\
 {									\
 	__asm__ __volatile__(						\
-	"1:	"insn("%2", "(%3)")"	# __put_user_asm_ll32	\n"	\
+	"1:	"insn("%2", "(%3)")"	# __put_data_asm_ll32	\n"	\
 	"2:	"insn("%D2", "4(%3)")"				\n"	\
 	"3:							\n"	\
 	"	.insn						\n"	\
@@ -593,7 +593,7 @@ extern void __get_user_unaligned_unknown(void);
 #define __get_user_unaligned_common(val, size, ptr)			\
 do {									\
 	switch (size) {							\
-	case 1: __get_user_asm(val, "lb", ptr); break;			\
+	case 1: __get_data_asm(val, "lb", ptr); break;			\
 	case 2: __get_user_unaligned_asm(val, "ulh", ptr); break;	\
 	case 4: __get_user_unaligned_asm(val, "ulw", ptr); break;	\
 	case 8: __GET_USER_UNALIGNED_DW(val, ptr); break;		\
@@ -620,7 +620,7 @@ do {									\
 	__gu_err;							\
 })
 
-#define __get_user_unaligned_asm(val, insn, addr)			\
+#define __get_data_unaligned_asm(val, insn, addr)			\
 {									\
 	long __gu_tmp;							\
 									\
@@ -686,7 +686,7 @@ do {									\
 #define __put_user_unaligned_common(ptr, size)				\
 do {									\
 	switch (size) {							\
-	case 1: __put_user_asm("sb", ptr); break;			\
+	case 1: __put_data_asm("sb", ptr); break;			\
 	case 2: __put_user_unaligned_asm("ush", ptr); break;		\
 	case 4: __put_user_unaligned_asm("usw", ptr); break;		\
 	case 8: __PUT_USER_UNALIGNED_DW(ptr); break;			\
-- 
1.8.5.3

[PATCH 28/58] MIPS: asm: uaccess: Add EVA support to copy_{in, to,from}_user

[Markos Chandras]

Use the EVA specific functions from memcpy.S to perform
userspace operations. When get_fs() == get_ds() the usual load/store
instructions are used because the destination address is located in
the kernel address space region. Otherwise, the EVA specifc load/store
instructions are used which will go through th TLB to perform the virtual
to physical translation for the userspace address.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/uaccess.h | 191 +++++++++++++++++++++++++++++++++++-----
 1 file changed, 171 insertions(+), 20 deletions(-)

diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index 59bbebb..fe72837 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -781,6 +781,7 @@ extern void __put_user_unaligned_unknown(void);
 
 extern size_t __copy_user(void *__to, const void *__from, size_t __n);
 
+#ifndef CONFIG_EVA
 #define __invoke_copy_to_user(to, from, n)				\
 ({									\
 	register void __user *__cu_to_r __asm__("$4");			\
@@ -799,6 +800,11 @@ extern size_t __copy_user(void *__to, const void *__from, size_t __n);
 	__cu_len_r;							\
 })
 
+#define __invoke_copy_to_kernel(to, from, n)				\
+	__invoke_copy_to_user(to, from, n)
+
+#endif
+
 /*
  * __copy_to_user: - Copy a block of data into user space, with less checking.
  * @to:	  Destination address, in user space.
@@ -823,7 +829,12 @@ extern size_t __copy_user(void *__to, const void *__from, size_t __n);
 	__cu_from = (from);						\
 	__cu_len = (n);							\
 	might_fault();							\
-	__cu_len = __invoke_copy_to_user(__cu_to, __cu_from, __cu_len); \
+	if (segment_eq(get_fs(), get_ds()))				\
+		__cu_len = __invoke_copy_to_kernel(__cu_to, __cu_from,	\
+						   __cu_len);		\
+	else								\
+		__cu_len = __invoke_copy_to_user(__cu_to, __cu_from,	\
+						 __cu_len);		\
 	__cu_len;							\
 })
 
@@ -838,7 +849,12 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
 	__cu_to = (to);							\
 	__cu_from = (from);						\
 	__cu_len = (n);							\
-	__cu_len = __invoke_copy_to_user(__cu_to, __cu_from, __cu_len); \
+	if (segment_eq(get_fs(), get_ds()))				\
+		__cu_len = __invoke_copy_to_kernel(__cu_to, __cu_from,	\
+						   __cu_len);		\
+	else								\
+		__cu_len = __invoke_copy_to_user(__cu_to, __cu_from,	\
+						 __cu_len);		\
 	__cu_len;							\
 })
 
@@ -851,8 +867,14 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
 	__cu_to = (to);							\
 	__cu_from = (from);						\
 	__cu_len = (n);							\
-	__cu_len = __invoke_copy_from_user_inatomic(__cu_to, __cu_from, \
-						    __cu_len);		\
+	if (segment_eq(get_fs(), get_ds()))				\
+		__cu_len = __invoke_copy_from_kernel_inatomic(__cu_to,	\
+							      __cu_from,\
+							      __cu_len);\
+	else								\
+		__cu_len = __invoke_copy_from_user_inatomic(__cu_to,	\
+							    __cu_from,	\
+							    __cu_len);	\
 	__cu_len;							\
 })
 
@@ -878,14 +900,23 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
 	__cu_to = (to);							\
 	__cu_from = (from);						\
 	__cu_len = (n);							\
-	if (access_ok(VERIFY_WRITE, __cu_to, __cu_len)) {		\
-		might_fault();						\
-		__cu_len = __invoke_copy_to_user(__cu_to, __cu_from,	\
-						 __cu_len);		\
+	if (segment_eq(get_fs(), get_ds())) {				\
+		__cu_len = __invoke_copy_to_kernel(__cu_to,		\
+						   __cu_from,		\
+						   __cu_len);		\
+	} else {							\
+		if (access_ok(VERIFY_WRITE, __cu_to, __cu_len)) {       \
+			might_fault();                                  \
+			__cu_len = __invoke_copy_to_user(__cu_to,	\
+							 __cu_from,	\
+							 __cu_len);     \
+		}							\
 	}								\
 	__cu_len;							\
 })
 
+#ifndef CONFIG_EVA
+
 #define __invoke_copy_from_user(to, from, n)				\
 ({									\
 	register void *__cu_to_r __asm__("$4");				\
@@ -909,6 +940,17 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
 	__cu_len_r;							\
 })
 
+#define __invoke_copy_from_kernel(to, from, n)				\
+	__invoke_copy_from_user(to, from, n)
+
+/* For userland <-> userland operations */
+#define ___invoke_copy_in_user(to, from, n)				\
+	__invoke_copy_from_user(to, from, n)
+
+/* For kernel <-> kernel operations */
+#define ___invoke_copy_in_kernel(to, from, n)				\
+	__invoke_copy_from_user(to, from, n)
+
 #define __invoke_copy_from_user_inatomic(to, from, n)			\
 ({									\
 	register void *__cu_to_r __asm__("$4");				\
@@ -932,6 +974,97 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
 	__cu_len_r;							\
 })
 
+#define __invoke_copy_from_kernel_inatomic(to, from, n)			\
+	__invoke_copy_from_user_inatomic(to, from, n)			\
+
+#else
+
+/* EVA specific functions */
+
+extern size_t __copy_user_inatomic_eva(void *__to, const void *__from,
+				       size_t __n);
+extern size_t __copy_from_user_eva(void *__to, const void *__from,
+				   size_t __n);
+extern size_t __copy_to_user_eva(void *__to, const void *__from,
+				 size_t __n);
+extern size_t __copy_in_user_eva(void *__to, const void *__from, size_t __n);
+
+#define __invoke_copy_from_user_eva_generic(to, from, n, func_ptr)	\
+({									\
+	register void *__cu_to_r __asm__("$4");				\
+	register const void __user *__cu_from_r __asm__("$5");		\
+	register long __cu_len_r __asm__("$6");				\
+									\
+	__cu_to_r = (to);						\
+	__cu_from_r = (from);						\
+	__cu_len_r = (n);						\
+	__asm__ __volatile__(						\
+	".set\tnoreorder\n\t"						\
+	__MODULE_JAL(func_ptr)						\
+	".set\tnoat\n\t"						\
+	__UA_ADDU "\t$1, %1, %2\n\t"					\
+	".set\tat\n\t"							\
+	".set\treorder"							\
+	: "+r" (__cu_to_r), "+r" (__cu_from_r), "+r" (__cu_len_r)	\
+	:								\
+	: "$8", "$9", "$10", "$11", "$12", "$14", "$15", "$24", "$31",	\
+	  DADDI_SCRATCH, "memory");					\
+	__cu_len_r;							\
+})
+
+#define __invoke_copy_to_user_eva_generic(to, from, n, func_ptr)	\
+({									\
+	register void *__cu_to_r __asm__("$4");				\
+	register const void __user *__cu_from_r __asm__("$5");		\
+	register long __cu_len_r __asm__("$6");				\
+									\
+	__cu_to_r = (to);						\
+	__cu_from_r = (from);						\
+	__cu_len_r = (n);						\
+	__asm__ __volatile__(						\
+	__MODULE_JAL(func_ptr)						\
+	: "+r" (__cu_to_r), "+r" (__cu_from_r), "+r" (__cu_len_r)	\
+	:								\
+	: "$8", "$9", "$10", "$11", "$12", "$14", "$15", "$24", "$31",	\
+	  DADDI_SCRATCH, "memory");					\
+	__cu_len_r;							\
+})
+
+/*
+ * Source or destination address is in userland. We need to go through
+ * the TLB
+ */
+#define __invoke_copy_from_user(to, from, n)				\
+	__invoke_copy_from_user_eva_generic(to, from, n, __copy_from_user_eva)
+
+#define __invoke_copy_from_user_inatomic(to, from, n)			\
+	__invoke_copy_from_user_eva_generic(to, from, n,		\
+					    __copy_user_inatomic_eva)
+
+#define __invoke_copy_to_user(to, from, n)				\
+	__invoke_copy_to_user_eva_generic(to, from, n, __copy_to_user_eva)
+
+#define ___invoke_copy_in_user(to, from, n)				\
+	__invoke_copy_from_user_eva_generic(to, from, n, __copy_in_user_eva)
+
+/*
+ * Source or destination address in the kernel. We are not going through
+ * the TLB
+ */
+#define __invoke_copy_from_kernel(to, from, n)				\
+	__invoke_copy_from_user_eva_generic(to, from, n, __copy_user)
+
+#define __invoke_copy_from_kernel_inatomic(to, from, n)			\
+	__invoke_copy_from_user_eva_generic(to, from, n, __copy_user_inatomic)
+
+#define __invoke_copy_to_kernel(to, from, n)				\
+	__invoke_copy_to_user_eva_generic(to, from, n, __copy_user)
+
+#define ___invoke_copy_in_kernel(to, from, n)				\
+	__invoke_copy_from_user_eva_generic(to, from, n, __copy_user)
+
+#endif /* CONFIG_EVA */
+
 /*
  * __copy_from_user: - Copy a block of data from user space, with less checking.
  * @to:	  Destination address, in kernel space.
@@ -989,10 +1122,17 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
 	__cu_to = (to);							\
 	__cu_from = (from);						\
 	__cu_len = (n);							\
-	if (access_ok(VERIFY_READ, __cu_from, __cu_len)) {		\
-		might_fault();						\
-		__cu_len = __invoke_copy_from_user(__cu_to, __cu_from,	\
-						   __cu_len);		\
+	if (segment_eq(get_fs(), get_ds())) {				\
+		__cu_len = __invoke_copy_from_kernel(__cu_to,		\
+						     __cu_from,		\
+						     __cu_len);		\
+	} else {							\
+		if (access_ok(VERIFY_READ, __cu_from, __cu_len)) {	\
+			might_fault();                                  \
+			__cu_len = __invoke_copy_from_user(__cu_to,	\
+							   __cu_from,	\
+							   __cu_len);   \
+		}							\
 	}								\
 	__cu_len;							\
 })
@@ -1006,9 +1146,14 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
 	__cu_to = (to);							\
 	__cu_from = (from);						\
 	__cu_len = (n);							\
-	might_fault();							\
-	__cu_len = __invoke_copy_from_user(__cu_to, __cu_from,		\
-					   __cu_len);			\
+	if (segment_eq(get_fs(), get_ds())) {				\
+		__cu_len = ___invoke_copy_in_kernel(__cu_to, __cu_from,	\
+						    __cu_len);		\
+	} else {							\
+		might_fault();						\
+		__cu_len = ___invoke_copy_in_user(__cu_to, __cu_from,	\
+						  __cu_len);		\
+	}								\
 	__cu_len;							\
 })
 
@@ -1021,11 +1166,17 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
 	__cu_to = (to);							\
 	__cu_from = (from);						\
 	__cu_len = (n);							\
-	if (likely(access_ok(VERIFY_READ, __cu_from, __cu_len) &&	\
-		   access_ok(VERIFY_WRITE, __cu_to, __cu_len))) {	\
-		might_fault();						\
-		__cu_len = __invoke_copy_from_user(__cu_to, __cu_from,	\
-						   __cu_len);		\
+	if (segment_eq(get_fs(), get_ds())) {				\
+		__cu_len = ___invoke_copy_in_kernel(__cu_to,__cu_from,	\
+						    __cu_len);		\
+	} else {							\
+		if (likely(access_ok(VERIFY_READ, __cu_from, __cu_len) &&\
+			   access_ok(VERIFY_WRITE, __cu_to, __cu_len))) {\
+			might_fault();					\
+			__cu_len = ___invoke_copy_in_user(__cu_to,	\
+							  __cu_from,	\
+							  __cu_len);	\
+		}							\
 	}								\
 	__cu_len;							\
 })
-- 
1.8.5.3

[PATCH 28/58] MIPS: asm: uaccess: Add EVA support to copy_{in, to,from}_user

[Markos Chandras]

Use the EVA specific functions from memcpy.S to perform
userspace operations. When get_fs() == get_ds() the usual load/store
instructions are used because the destination address is located in
the kernel address space region. Otherwise, the EVA specifc load/store
instructions are used which will go through th TLB to perform the virtual
to physical translation for the userspace address.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/uaccess.h | 191 +++++++++++++++++++++++++++++++++++-----
 1 file changed, 171 insertions(+), 20 deletions(-)

diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index 59bbebb..fe72837 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -781,6 +781,7 @@ extern void __put_user_unaligned_unknown(void);
 
 extern size_t __copy_user(void *__to, const void *__from, size_t __n);
 
+#ifndef CONFIG_EVA
 #define __invoke_copy_to_user(to, from, n)				\
 ({									\
 	register void __user *__cu_to_r __asm__("$4");			\
@@ -799,6 +800,11 @@ extern size_t __copy_user(void *__to, const void *__from, size_t __n);
 	__cu_len_r;							\
 })
 
+#define __invoke_copy_to_kernel(to, from, n)				\
+	__invoke_copy_to_user(to, from, n)
+
+#endif
+
 /*
  * __copy_to_user: - Copy a block of data into user space, with less checking.
  * @to:	  Destination address, in user space.
@@ -823,7 +829,12 @@ extern size_t __copy_user(void *__to, const void *__from, size_t __n);
 	__cu_from = (from);						\
 	__cu_len = (n);							\
 	might_fault();							\
-	__cu_len = __invoke_copy_to_user(__cu_to, __cu_from, __cu_len); \
+	if (segment_eq(get_fs(), get_ds()))				\
+		__cu_len = __invoke_copy_to_kernel(__cu_to, __cu_from,	\
+						   __cu_len);		\
+	else								\
+		__cu_len = __invoke_copy_to_user(__cu_to, __cu_from,	\
+						 __cu_len);		\
 	__cu_len;							\
 })
 
@@ -838,7 +849,12 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
 	__cu_to = (to);							\
 	__cu_from = (from);						\
 	__cu_len = (n);							\
-	__cu_len = __invoke_copy_to_user(__cu_to, __cu_from, __cu_len); \
+	if (segment_eq(get_fs(), get_ds()))				\
+		__cu_len = __invoke_copy_to_kernel(__cu_to, __cu_from,	\
+						   __cu_len);		\
+	else								\
+		__cu_len = __invoke_copy_to_user(__cu_to, __cu_from,	\
+						 __cu_len);		\
 	__cu_len;							\
 })
 
@@ -851,8 +867,14 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
 	__cu_to = (to);							\
 	__cu_from = (from);						\
 	__cu_len = (n);							\
-	__cu_len = __invoke_copy_from_user_inatomic(__cu_to, __cu_from, \
-						    __cu_len);		\
+	if (segment_eq(get_fs(), get_ds()))				\
+		__cu_len = __invoke_copy_from_kernel_inatomic(__cu_to,	\
+							      __cu_from,\
+							      __cu_len);\
+	else								\
+		__cu_len = __invoke_copy_from_user_inatomic(__cu_to,	\
+							    __cu_from,	\
+							    __cu_len);	\
 	__cu_len;							\
 })
 
@@ -878,14 +900,23 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
 	__cu_to = (to);							\
 	__cu_from = (from);						\
 	__cu_len = (n);							\
-	if (access_ok(VERIFY_WRITE, __cu_to, __cu_len)) {		\
-		might_fault();						\
-		__cu_len = __invoke_copy_to_user(__cu_to, __cu_from,	\
-						 __cu_len);		\
+	if (segment_eq(get_fs(), get_ds())) {				\
+		__cu_len = __invoke_copy_to_kernel(__cu_to,		\
+						   __cu_from,		\
+						   __cu_len);		\
+	} else {							\
+		if (access_ok(VERIFY_WRITE, __cu_to, __cu_len)) {       \
+			might_fault();                                  \
+			__cu_len = __invoke_copy_to_user(__cu_to,	\
+							 __cu_from,	\
+							 __cu_len);     \
+		}							\
 	}								\
 	__cu_len;							\
 })
 
+#ifndef CONFIG_EVA
+
 #define __invoke_copy_from_user(to, from, n)				\
 ({									\
 	register void *__cu_to_r __asm__("$4");				\
@@ -909,6 +940,17 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
 	__cu_len_r;							\
 })
 
+#define __invoke_copy_from_kernel(to, from, n)				\
+	__invoke_copy_from_user(to, from, n)
+
+/* For userland <-> userland operations */
+#define ___invoke_copy_in_user(to, from, n)				\
+	__invoke_copy_from_user(to, from, n)
+
+/* For kernel <-> kernel operations */
+#define ___invoke_copy_in_kernel(to, from, n)				\
+	__invoke_copy_from_user(to, from, n)
+
 #define __invoke_copy_from_user_inatomic(to, from, n)			\
 ({									\
 	register void *__cu_to_r __asm__("$4");				\
@@ -932,6 +974,97 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
 	__cu_len_r;							\
 })
 
+#define __invoke_copy_from_kernel_inatomic(to, from, n)			\
+	__invoke_copy_from_user_inatomic(to, from, n)			\
+
+#else
+
+/* EVA specific functions */
+
+extern size_t __copy_user_inatomic_eva(void *__to, const void *__from,
+				       size_t __n);
+extern size_t __copy_from_user_eva(void *__to, const void *__from,
+				   size_t __n);
+extern size_t __copy_to_user_eva(void *__to, const void *__from,
+				 size_t __n);
+extern size_t __copy_in_user_eva(void *__to, const void *__from, size_t __n);
+
+#define __invoke_copy_from_user_eva_generic(to, from, n, func_ptr)	\
+({									\
+	register void *__cu_to_r __asm__("$4");				\
+	register const void __user *__cu_from_r __asm__("$5");		\
+	register long __cu_len_r __asm__("$6");				\
+									\
+	__cu_to_r = (to);						\
+	__cu_from_r = (from);						\
+	__cu_len_r = (n);						\
+	__asm__ __volatile__(						\
+	".set\tnoreorder\n\t"						\
+	__MODULE_JAL(func_ptr)						\
+	".set\tnoat\n\t"						\
+	__UA_ADDU "\t$1, %1, %2\n\t"					\
+	".set\tat\n\t"							\
+	".set\treorder"							\
+	: "+r" (__cu_to_r), "+r" (__cu_from_r), "+r" (__cu_len_r)	\
+	:								\
+	: "$8", "$9", "$10", "$11", "$12", "$14", "$15", "$24", "$31",	\
+	  DADDI_SCRATCH, "memory");					\
+	__cu_len_r;							\
+})
+
+#define __invoke_copy_to_user_eva_generic(to, from, n, func_ptr)	\
+({									\
+	register void *__cu_to_r __asm__("$4");				\
+	register const void __user *__cu_from_r __asm__("$5");		\
+	register long __cu_len_r __asm__("$6");				\
+									\
+	__cu_to_r = (to);						\
+	__cu_from_r = (from);						\
+	__cu_len_r = (n);						\
+	__asm__ __volatile__(						\
+	__MODULE_JAL(func_ptr)						\
+	: "+r" (__cu_to_r), "+r" (__cu_from_r), "+r" (__cu_len_r)	\
+	:								\
+	: "$8", "$9", "$10", "$11", "$12", "$14", "$15", "$24", "$31",	\
+	  DADDI_SCRATCH, "memory");					\
+	__cu_len_r;							\
+})
+
+/*
+ * Source or destination address is in userland. We need to go through
+ * the TLB
+ */
+#define __invoke_copy_from_user(to, from, n)				\
+	__invoke_copy_from_user_eva_generic(to, from, n, __copy_from_user_eva)
+
+#define __invoke_copy_from_user_inatomic(to, from, n)			\
+	__invoke_copy_from_user_eva_generic(to, from, n,		\
+					    __copy_user_inatomic_eva)
+
+#define __invoke_copy_to_user(to, from, n)				\
+	__invoke_copy_to_user_eva_generic(to, from, n, __copy_to_user_eva)
+
+#define ___invoke_copy_in_user(to, from, n)				\
+	__invoke_copy_from_user_eva_generic(to, from, n, __copy_in_user_eva)
+
+/*
+ * Source or destination address in the kernel. We are not going through
+ * the TLB
+ */
+#define __invoke_copy_from_kernel(to, from, n)				\
+	__invoke_copy_from_user_eva_generic(to, from, n, __copy_user)
+
+#define __invoke_copy_from_kernel_inatomic(to, from, n)			\
+	__invoke_copy_from_user_eva_generic(to, from, n, __copy_user_inatomic)
+
+#define __invoke_copy_to_kernel(to, from, n)				\
+	__invoke_copy_to_user_eva_generic(to, from, n, __copy_user)
+
+#define ___invoke_copy_in_kernel(to, from, n)				\
+	__invoke_copy_from_user_eva_generic(to, from, n, __copy_user)
+
+#endif /* CONFIG_EVA */
+
 /*
  * __copy_from_user: - Copy a block of data from user space, with less checking.
  * @to:	  Destination address, in kernel space.
@@ -989,10 +1122,17 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
 	__cu_to = (to);							\
 	__cu_from = (from);						\
 	__cu_len = (n);							\
-	if (access_ok(VERIFY_READ, __cu_from, __cu_len)) {		\
-		might_fault();						\
-		__cu_len = __invoke_copy_from_user(__cu_to, __cu_from,	\
-						   __cu_len);		\
+	if (segment_eq(get_fs(), get_ds())) {				\
+		__cu_len = __invoke_copy_from_kernel(__cu_to,		\
+						     __cu_from,		\
+						     __cu_len);		\
+	} else {							\
+		if (access_ok(VERIFY_READ, __cu_from, __cu_len)) {	\
+			might_fault();                                  \
+			__cu_len = __invoke_copy_from_user(__cu_to,	\
+							   __cu_from,	\
+							   __cu_len);   \
+		}							\
 	}								\
 	__cu_len;							\
 })
@@ -1006,9 +1146,14 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
 	__cu_to = (to);							\
 	__cu_from = (from);						\
 	__cu_len = (n);							\
-	might_fault();							\
-	__cu_len = __invoke_copy_from_user(__cu_to, __cu_from,		\
-					   __cu_len);			\
+	if (segment_eq(get_fs(), get_ds())) {				\
+		__cu_len = ___invoke_copy_in_kernel(__cu_to, __cu_from,	\
+						    __cu_len);		\
+	} else {							\
+		might_fault();						\
+		__cu_len = ___invoke_copy_in_user(__cu_to, __cu_from,	\
+						  __cu_len);		\
+	}								\
 	__cu_len;							\
 })
 
@@ -1021,11 +1166,17 @@ extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
 	__cu_to = (to);							\
 	__cu_from = (from);						\
 	__cu_len = (n);							\
-	if (likely(access_ok(VERIFY_READ, __cu_from, __cu_len) &&	\
-		   access_ok(VERIFY_WRITE, __cu_to, __cu_len))) {	\
-		might_fault();						\
-		__cu_len = __invoke_copy_from_user(__cu_to, __cu_from,	\
-						   __cu_len);		\
+	if (segment_eq(get_fs(), get_ds())) {				\
+		__cu_len = ___invoke_copy_in_kernel(__cu_to,__cu_from,	\
+						    __cu_len);		\
+	} else {							\
+		if (likely(access_ok(VERIFY_READ, __cu_from, __cu_len) &&\
+			   access_ok(VERIFY_WRITE, __cu_to, __cu_len))) {\
+			might_fault();					\
+			__cu_len = ___invoke_copy_in_user(__cu_to,	\
+							  __cu_from,	\
+							  __cu_len);	\
+		}							\
 	}								\
 	__cu_len;							\
 })
-- 
1.8.5.3

[PATCH 29/58] MIPS: asm: uaccess: Add EVA support for str*_user operations

[Markos Chandras]

The str*_user functions are used to securely access NULL terminated
strings from userland. Therefore, it's necessary to use the appropriate
EVA function. However, if the string is in kernel space, then the normal
instructions are being used to access it. The __str*_kernel_asm and
__str*_user_asm symbols are the same for non-EVA mode so there is no
functional change for the non-EVA kernels.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/uaccess.h | 172 +++++++++++++++++++++++++++-------------
 1 file changed, 119 insertions(+), 53 deletions(-)

diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index fe72837..da52765 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -1246,16 +1246,28 @@ __strncpy_from_user(char *__to, const char __user *__from, long __len)
 {
 	long res;
 
-	might_fault();
-	__asm__ __volatile__(
-		"move\t$4, %1\n\t"
-		"move\t$5, %2\n\t"
-		"move\t$6, %3\n\t"
-		__MODULE_JAL(__strncpy_from_user_nocheck_asm)
-		"move\t%0, $2"
-		: "=r" (res)
-		: "r" (__to), "r" (__from), "r" (__len)
-		: "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+	if (segment_eq(get_fs(), get_ds())) {
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			"move\t$5, %2\n\t"
+			"move\t$6, %3\n\t"
+			__MODULE_JAL(__strncpy_from_kernel_nocheck_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (__to), "r" (__from), "r" (__len)
+			: "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+	} else {
+		might_fault();
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			"move\t$5, %2\n\t"
+			"move\t$6, %3\n\t"
+			__MODULE_JAL(__strncpy_from_user_nocheck_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (__to), "r" (__from), "r" (__len)
+			: "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+	}
 
 	return res;
 }
@@ -1283,16 +1295,28 @@ strncpy_from_user(char *__to, const char __user *__from, long __len)
 {
 	long res;
 
-	might_fault();
-	__asm__ __volatile__(
-		"move\t$4, %1\n\t"
-		"move\t$5, %2\n\t"
-		"move\t$6, %3\n\t"
-		__MODULE_JAL(__strncpy_from_user_asm)
-		"move\t%0, $2"
-		: "=r" (res)
-		: "r" (__to), "r" (__from), "r" (__len)
-		: "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+	if (segment_eq(get_fs(), get_ds())) {
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			"move\t$5, %2\n\t"
+			"move\t$6, %3\n\t"
+			__MODULE_JAL(__strncpy_from_kernel_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (__to), "r" (__from), "r" (__len)
+			: "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+	} else {
+		might_fault();
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			"move\t$5, %2\n\t"
+			"move\t$6, %3\n\t"
+			__MODULE_JAL(__strncpy_from_user_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (__to), "r" (__from), "r" (__len)
+			: "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+	}
 
 	return res;
 }
@@ -1302,14 +1326,24 @@ static inline long __strlen_user(const char __user *s)
 {
 	long res;
 
-	might_fault();
-	__asm__ __volatile__(
-		"move\t$4, %1\n\t"
-		__MODULE_JAL(__strlen_user_nocheck_asm)
-		"move\t%0, $2"
-		: "=r" (res)
-		: "r" (s)
-		: "$2", "$4", __UA_t0, "$31");
+	if (segment_eq(get_fs(), get_ds())) {
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			__MODULE_JAL(__strlen_kernel_nocheck_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (s)
+			: "$2", "$4", __UA_t0, "$31");
+	} else {
+		might_fault();
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			__MODULE_JAL(__strlen_user_nocheck_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (s)
+			: "$2", "$4", __UA_t0, "$31");
+	}
 
 	return res;
 }
@@ -1332,14 +1366,24 @@ static inline long strlen_user(const char __user *s)
 {
 	long res;
 
-	might_fault();
-	__asm__ __volatile__(
-		"move\t$4, %1\n\t"
-		__MODULE_JAL(__strlen_user_asm)
-		"move\t%0, $2"
-		: "=r" (res)
-		: "r" (s)
-		: "$2", "$4", __UA_t0, "$31");
+	if (segment_eq(get_fs(), get_ds())) {
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			__MODULE_JAL(__strlen_kernel_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (s)
+			: "$2", "$4", __UA_t0, "$31");
+	} else {
+		might_fault();
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			__MODULE_JAL(__strlen_kernel_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (s)
+			: "$2", "$4", __UA_t0, "$31");
+	}
 
 	return res;
 }
@@ -1349,15 +1393,26 @@ static inline long __strnlen_user(const char __user *s, long n)
 {
 	long res;
 
-	might_fault();
-	__asm__ __volatile__(
-		"move\t$4, %1\n\t"
-		"move\t$5, %2\n\t"
-		__MODULE_JAL(__strnlen_user_nocheck_asm)
-		"move\t%0, $2"
-		: "=r" (res)
-		: "r" (s), "r" (n)
-		: "$2", "$4", "$5", __UA_t0, "$31");
+	if (segment_eq(get_fs(), get_ds())) {
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			"move\t$5, %2\n\t"
+			__MODULE_JAL(__strnlen_kernel_nocheck_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (s), "r" (n)
+			: "$2", "$4", "$5", __UA_t0, "$31");
+	} else {
+		might_fault();
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			"move\t$5, %2\n\t"
+			__MODULE_JAL(__strnlen_user_nocheck_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (s), "r" (n)
+			: "$2", "$4", "$5", __UA_t0, "$31");
+	}
 
 	return res;
 }
@@ -1381,14 +1436,25 @@ static inline long strnlen_user(const char __user *s, long n)
 	long res;
 
 	might_fault();
-	__asm__ __volatile__(
-		"move\t$4, %1\n\t"
-		"move\t$5, %2\n\t"
-		__MODULE_JAL(__strnlen_user_asm)
-		"move\t%0, $2"
-		: "=r" (res)
-		: "r" (s), "r" (n)
-		: "$2", "$4", "$5", __UA_t0, "$31");
+	if (segment_eq(get_fs(), get_ds())) {
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			"move\t$5, %2\n\t"
+			__MODULE_JAL(__strnlen_kernel_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (s), "r" (n)
+			: "$2", "$4", "$5", __UA_t0, "$31");
+	} else {
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			"move\t$5, %2\n\t"
+			__MODULE_JAL(__strnlen_user_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (s), "r" (n)
+			: "$2", "$4", "$5", __UA_t0, "$31");
+	}
 
 	return res;
 }
-- 
1.8.5.3

[PATCH 29/58] MIPS: asm: uaccess: Add EVA support for str*_user operations

[Markos Chandras]

The str*_user functions are used to securely access NULL terminated
strings from userland. Therefore, it's necessary to use the appropriate
EVA function. However, if the string is in kernel space, then the normal
instructions are being used to access it. The __str*_kernel_asm and
__str*_user_asm symbols are the same for non-EVA mode so there is no
functional change for the non-EVA kernels.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/uaccess.h | 172 +++++++++++++++++++++++++++-------------
 1 file changed, 119 insertions(+), 53 deletions(-)

diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index fe72837..da52765 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -1246,16 +1246,28 @@ __strncpy_from_user(char *__to, const char __user *__from, long __len)
 {
 	long res;
 
-	might_fault();
-	__asm__ __volatile__(
-		"move\t$4, %1\n\t"
-		"move\t$5, %2\n\t"
-		"move\t$6, %3\n\t"
-		__MODULE_JAL(__strncpy_from_user_nocheck_asm)
-		"move\t%0, $2"
-		: "=r" (res)
-		: "r" (__to), "r" (__from), "r" (__len)
-		: "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+	if (segment_eq(get_fs(), get_ds())) {
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			"move\t$5, %2\n\t"
+			"move\t$6, %3\n\t"
+			__MODULE_JAL(__strncpy_from_kernel_nocheck_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (__to), "r" (__from), "r" (__len)
+			: "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+	} else {
+		might_fault();
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			"move\t$5, %2\n\t"
+			"move\t$6, %3\n\t"
+			__MODULE_JAL(__strncpy_from_user_nocheck_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (__to), "r" (__from), "r" (__len)
+			: "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+	}
 
 	return res;
 }
@@ -1283,16 +1295,28 @@ strncpy_from_user(char *__to, const char __user *__from, long __len)
 {
 	long res;
 
-	might_fault();
-	__asm__ __volatile__(
-		"move\t$4, %1\n\t"
-		"move\t$5, %2\n\t"
-		"move\t$6, %3\n\t"
-		__MODULE_JAL(__strncpy_from_user_asm)
-		"move\t%0, $2"
-		: "=r" (res)
-		: "r" (__to), "r" (__from), "r" (__len)
-		: "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+	if (segment_eq(get_fs(), get_ds())) {
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			"move\t$5, %2\n\t"
+			"move\t$6, %3\n\t"
+			__MODULE_JAL(__strncpy_from_kernel_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (__to), "r" (__from), "r" (__len)
+			: "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+	} else {
+		might_fault();
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			"move\t$5, %2\n\t"
+			"move\t$6, %3\n\t"
+			__MODULE_JAL(__strncpy_from_user_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (__to), "r" (__from), "r" (__len)
+			: "$2", "$3", "$4", "$5", "$6", __UA_t0, "$31", "memory");
+	}
 
 	return res;
 }
@@ -1302,14 +1326,24 @@ static inline long __strlen_user(const char __user *s)
 {
 	long res;
 
-	might_fault();
-	__asm__ __volatile__(
-		"move\t$4, %1\n\t"
-		__MODULE_JAL(__strlen_user_nocheck_asm)
-		"move\t%0, $2"
-		: "=r" (res)
-		: "r" (s)
-		: "$2", "$4", __UA_t0, "$31");
+	if (segment_eq(get_fs(), get_ds())) {
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			__MODULE_JAL(__strlen_kernel_nocheck_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (s)
+			: "$2", "$4", __UA_t0, "$31");
+	} else {
+		might_fault();
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			__MODULE_JAL(__strlen_user_nocheck_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (s)
+			: "$2", "$4", __UA_t0, "$31");
+	}
 
 	return res;
 }
@@ -1332,14 +1366,24 @@ static inline long strlen_user(const char __user *s)
 {
 	long res;
 
-	might_fault();
-	__asm__ __volatile__(
-		"move\t$4, %1\n\t"
-		__MODULE_JAL(__strlen_user_asm)
-		"move\t%0, $2"
-		: "=r" (res)
-		: "r" (s)
-		: "$2", "$4", __UA_t0, "$31");
+	if (segment_eq(get_fs(), get_ds())) {
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			__MODULE_JAL(__strlen_kernel_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (s)
+			: "$2", "$4", __UA_t0, "$31");
+	} else {
+		might_fault();
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			__MODULE_JAL(__strlen_kernel_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (s)
+			: "$2", "$4", __UA_t0, "$31");
+	}
 
 	return res;
 }
@@ -1349,15 +1393,26 @@ static inline long __strnlen_user(const char __user *s, long n)
 {
 	long res;
 
-	might_fault();
-	__asm__ __volatile__(
-		"move\t$4, %1\n\t"
-		"move\t$5, %2\n\t"
-		__MODULE_JAL(__strnlen_user_nocheck_asm)
-		"move\t%0, $2"
-		: "=r" (res)
-		: "r" (s), "r" (n)
-		: "$2", "$4", "$5", __UA_t0, "$31");
+	if (segment_eq(get_fs(), get_ds())) {
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			"move\t$5, %2\n\t"
+			__MODULE_JAL(__strnlen_kernel_nocheck_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (s), "r" (n)
+			: "$2", "$4", "$5", __UA_t0, "$31");
+	} else {
+		might_fault();
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			"move\t$5, %2\n\t"
+			__MODULE_JAL(__strnlen_user_nocheck_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (s), "r" (n)
+			: "$2", "$4", "$5", __UA_t0, "$31");
+	}
 
 	return res;
 }
@@ -1381,14 +1436,25 @@ static inline long strnlen_user(const char __user *s, long n)
 	long res;
 
 	might_fault();
-	__asm__ __volatile__(
-		"move\t$4, %1\n\t"
-		"move\t$5, %2\n\t"
-		__MODULE_JAL(__strnlen_user_asm)
-		"move\t%0, $2"
-		: "=r" (res)
-		: "r" (s), "r" (n)
-		: "$2", "$4", "$5", __UA_t0, "$31");
+	if (segment_eq(get_fs(), get_ds())) {
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			"move\t$5, %2\n\t"
+			__MODULE_JAL(__strnlen_kernel_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (s), "r" (n)
+			: "$2", "$4", "$5", __UA_t0, "$31");
+	} else {
+		__asm__ __volatile__(
+			"move\t$4, %1\n\t"
+			"move\t$5, %2\n\t"
+			__MODULE_JAL(__strnlen_user_asm)
+			"move\t%0, $2"
+			: "=r" (res)
+			: "r" (s), "r" (n)
+			: "$2", "$4", "$5", __UA_t0, "$31");
+	}
 
 	return res;
 }
-- 
1.8.5.3

[PATCH 30/58] MIPS: kernel: unaligned: Add EVA instruction wrappers

[Markos Chandras]

Use the load/store instruction wrappers from asm/asm.h to
perform such operations when operating in EVA mode.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/unaligned.c | 49 ++++++++++++++++++++++----------------------
 1 file changed, 25 insertions(+), 24 deletions(-)

diff --git a/arch/mips/kernel/unaligned.c b/arch/mips/kernel/unaligned.c
index c369a5d..5ec8f00 100644
--- a/arch/mips/kernel/unaligned.c
+++ b/arch/mips/kernel/unaligned.c
@@ -7,6 +7,7 @@
  *
  * Copyright (C) 1996, 1998, 1999, 2002 by Ralf Baechle
  * Copyright (C) 1999 Silicon Graphics, Inc.
+ * Copyright (C) 2014 Imagination Technologies Ltd.
  *
  * This file contains exception handler for address error exception with the
  * special capability to execute faulting instructions in software.  The
@@ -110,8 +111,8 @@ extern void show_registers(struct pt_regs *regs);
 #ifdef __BIG_ENDIAN
 #define     LoadHW(addr, value, res)  \
 		__asm__ __volatile__ (".set\tnoat\n"        \
-			"1:\tlb\t%0, 0(%2)\n"               \
-			"2:\tlbu\t$1, 1(%2)\n\t"            \
+			"1:\t"user_lb("%0", "0(%2)")"\n"    \
+			"2:\t"user_lbu("$1", "1(%2)")"\n\t" \
 			"sll\t%0, 0x8\n\t"                  \
 			"or\t%0, $1\n\t"                    \
 			"li\t%1, 0\n"                       \
@@ -130,8 +131,8 @@ extern void show_registers(struct pt_regs *regs);
 
 #define     LoadW(addr, value, res)   \
 		__asm__ __volatile__ (                      \
-			"1:\tlwl\t%0, (%2)\n"               \
-			"2:\tlwr\t%0, 3(%2)\n\t"            \
+			"1:\t"user_lwl("%0", "(%2)")"\n"    \
+			"2:\t"user_lwr("%0", "3(%2)")"\n\t" \
 			"li\t%1, 0\n"                       \
 			"3:\n\t"                            \
 			".insn\n\t"                         \
@@ -149,8 +150,8 @@ extern void show_registers(struct pt_regs *regs);
 #define     LoadHWU(addr, value, res) \
 		__asm__ __volatile__ (                      \
 			".set\tnoat\n"                      \
-			"1:\tlbu\t%0, 0(%2)\n"              \
-			"2:\tlbu\t$1, 1(%2)\n\t"            \
+			"1:\t"user_lbu("%0", "0(%2)")"\n"   \
+			"2:\t"user_lbu("$1", "1(%2)")"\n\t" \
 			"sll\t%0, 0x8\n\t"                  \
 			"or\t%0, $1\n\t"                    \
 			"li\t%1, 0\n"                       \
@@ -170,8 +171,8 @@ extern void show_registers(struct pt_regs *regs);
 
 #define     LoadWU(addr, value, res)  \
 		__asm__ __volatile__ (                      \
-			"1:\tlwl\t%0, (%2)\n"               \
-			"2:\tlwr\t%0, 3(%2)\n\t"            \
+			"1:\t"user_lwl("%0", "(%2)")"\n"    \
+			"2:\t"user_lwr("%0", "3(%2)")"\n\t" \
 			"dsll\t%0, %0, 32\n\t"              \
 			"dsrl\t%0, %0, 32\n\t"              \
 			"li\t%1, 0\n"                       \
@@ -209,9 +210,9 @@ extern void show_registers(struct pt_regs *regs);
 #define     StoreHW(addr, value, res) \
 		__asm__ __volatile__ (                      \
 			".set\tnoat\n"                      \
-			"1:\tsb\t%1, 1(%2)\n\t"             \
+			"1:\t"user_sb("%1", "1(%2)")"\n"    \
 			"srl\t$1, %1, 0x8\n"                \
-			"2:\tsb\t$1, 0(%2)\n\t"             \
+			"2:\t"user_sb("$1", "0(%2)")"\n"    \
 			".set\tat\n\t"                      \
 			"li\t%0, 0\n"                       \
 			"3:\n\t"                            \
@@ -229,8 +230,8 @@ extern void show_registers(struct pt_regs *regs);
 
 #define     StoreW(addr, value, res)  \
 		__asm__ __volatile__ (                      \
-			"1:\tswl\t%1,(%2)\n"                \
-			"2:\tswr\t%1, 3(%2)\n\t"            \
+			"1:\t"user_swl("%1", "(%2)")"\n"    \
+			"2:\t"user_swr("%1", "3(%2)")"\n\t" \
 			"li\t%0, 0\n"                       \
 			"3:\n\t"                            \
 			".insn\n\t"                         \
@@ -267,8 +268,8 @@ extern void show_registers(struct pt_regs *regs);
 #ifdef __LITTLE_ENDIAN
 #define     LoadHW(addr, value, res)  \
 		__asm__ __volatile__ (".set\tnoat\n"        \
-			"1:\tlb\t%0, 1(%2)\n"               \
-			"2:\tlbu\t$1, 0(%2)\n\t"            \
+			"1:\t"user_lb("%0", "1(%2)")"\n"    \
+			"2:\t"user_lbu("$1", "0(%2)")"\n\t" \
 			"sll\t%0, 0x8\n\t"                  \
 			"or\t%0, $1\n\t"                    \
 			"li\t%1, 0\n"                       \
@@ -287,8 +288,8 @@ extern void show_registers(struct pt_regs *regs);
 
 #define     LoadW(addr, value, res)   \
 		__asm__ __volatile__ (                      \
-			"1:\tlwl\t%0, 3(%2)\n"              \
-			"2:\tlwr\t%0, (%2)\n\t"             \
+			"1:\t"user_lwl("%0", "3(%2)")"\n"   \
+			"2:\t"user_lwr("%0", "(%2)")"\n\t"  \
 			"li\t%1, 0\n"                       \
 			"3:\n\t"                            \
 			".insn\n\t"                         \
@@ -306,8 +307,8 @@ extern void show_registers(struct pt_regs *regs);
 #define     LoadHWU(addr, value, res) \
 		__asm__ __volatile__ (                      \
 			".set\tnoat\n"                      \
-			"1:\tlbu\t%0, 1(%2)\n"              \
-			"2:\tlbu\t$1, 0(%2)\n\t"            \
+			"1:\t"user_lbu("%0", "1(%2)")"\n"   \
+			"2:\t"user_lbu("$1", "0(%2)")"\n\t" \
 			"sll\t%0, 0x8\n\t"                  \
 			"or\t%0, $1\n\t"                    \
 			"li\t%1, 0\n"                       \
@@ -327,8 +328,8 @@ extern void show_registers(struct pt_regs *regs);
 
 #define     LoadWU(addr, value, res)  \
 		__asm__ __volatile__ (                      \
-			"1:\tlwl\t%0, 3(%2)\n"              \
-			"2:\tlwr\t%0, (%2)\n\t"             \
+			"1:\t"user_lwl("%0", "3(%2)")"\n"   \
+			"2:\t"user_lwr("%0", "(%2)")"\n\t"  \
 			"dsll\t%0, %0, 32\n\t"              \
 			"dsrl\t%0, %0, 32\n\t"              \
 			"li\t%1, 0\n"                       \
@@ -366,9 +367,9 @@ extern void show_registers(struct pt_regs *regs);
 #define     StoreHW(addr, value, res) \
 		__asm__ __volatile__ (                      \
 			".set\tnoat\n"                      \
-			"1:\tsb\t%1, 0(%2)\n\t"             \
+			"1:\t"user_sb("%1", "0(%2)")"\n"    \
 			"srl\t$1,%1, 0x8\n"                 \
-			"2:\tsb\t$1, 1(%2)\n\t"             \
+			"2:\t"user_sb("$1", "1(%2)")"\n"    \
 			".set\tat\n\t"                      \
 			"li\t%0, 0\n"                       \
 			"3:\n\t"                            \
@@ -386,8 +387,8 @@ extern void show_registers(struct pt_regs *regs);
 
 #define     StoreW(addr, value, res)  \
 		__asm__ __volatile__ (                      \
-			"1:\tswl\t%1, 3(%2)\n"              \
-			"2:\tswr\t%1, (%2)\n\t"             \
+			"1:\t"user_swl("%1", "3(%2)")"\n"   \
+			"2:\t"user_swr("%1", "(%2)")"\n\t"  \
 			"li\t%0, 0\n"                       \
 			"3:\n\t"                            \
 			".insn\n\t"                         \
-- 
1.8.5.3

[PATCH 30/58] MIPS: kernel: unaligned: Add EVA instruction wrappers

[Markos Chandras]

Use the load/store instruction wrappers from asm/asm.h to
perform such operations when operating in EVA mode.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/unaligned.c | 49 ++++++++++++++++++++++----------------------
 1 file changed, 25 insertions(+), 24 deletions(-)

diff --git a/arch/mips/kernel/unaligned.c b/arch/mips/kernel/unaligned.c
index c369a5d..5ec8f00 100644
--- a/arch/mips/kernel/unaligned.c
+++ b/arch/mips/kernel/unaligned.c
@@ -7,6 +7,7 @@
  *
  * Copyright (C) 1996, 1998, 1999, 2002 by Ralf Baechle
  * Copyright (C) 1999 Silicon Graphics, Inc.
+ * Copyright (C) 2014 Imagination Technologies Ltd.
  *
  * This file contains exception handler for address error exception with the
  * special capability to execute faulting instructions in software.  The
@@ -110,8 +111,8 @@ extern void show_registers(struct pt_regs *regs);
 #ifdef __BIG_ENDIAN
 #define     LoadHW(addr, value, res)  \
 		__asm__ __volatile__ (".set\tnoat\n"        \
-			"1:\tlb\t%0, 0(%2)\n"               \
-			"2:\tlbu\t$1, 1(%2)\n\t"            \
+			"1:\t"user_lb("%0", "0(%2)")"\n"    \
+			"2:\t"user_lbu("$1", "1(%2)")"\n\t" \
 			"sll\t%0, 0x8\n\t"                  \
 			"or\t%0, $1\n\t"                    \
 			"li\t%1, 0\n"                       \
@@ -130,8 +131,8 @@ extern void show_registers(struct pt_regs *regs);
 
 #define     LoadW(addr, value, res)   \
 		__asm__ __volatile__ (                      \
-			"1:\tlwl\t%0, (%2)\n"               \
-			"2:\tlwr\t%0, 3(%2)\n\t"            \
+			"1:\t"user_lwl("%0", "(%2)")"\n"    \
+			"2:\t"user_lwr("%0", "3(%2)")"\n\t" \
 			"li\t%1, 0\n"                       \
 			"3:\n\t"                            \
 			".insn\n\t"                         \
@@ -149,8 +150,8 @@ extern void show_registers(struct pt_regs *regs);
 #define     LoadHWU(addr, value, res) \
 		__asm__ __volatile__ (                      \
 			".set\tnoat\n"                      \
-			"1:\tlbu\t%0, 0(%2)\n"              \
-			"2:\tlbu\t$1, 1(%2)\n\t"            \
+			"1:\t"user_lbu("%0", "0(%2)")"\n"   \
+			"2:\t"user_lbu("$1", "1(%2)")"\n\t" \
 			"sll\t%0, 0x8\n\t"                  \
 			"or\t%0, $1\n\t"                    \
 			"li\t%1, 0\n"                       \
@@ -170,8 +171,8 @@ extern void show_registers(struct pt_regs *regs);
 
 #define     LoadWU(addr, value, res)  \
 		__asm__ __volatile__ (                      \
-			"1:\tlwl\t%0, (%2)\n"               \
-			"2:\tlwr\t%0, 3(%2)\n\t"            \
+			"1:\t"user_lwl("%0", "(%2)")"\n"    \
+			"2:\t"user_lwr("%0", "3(%2)")"\n\t" \
 			"dsll\t%0, %0, 32\n\t"              \
 			"dsrl\t%0, %0, 32\n\t"              \
 			"li\t%1, 0\n"                       \
@@ -209,9 +210,9 @@ extern void show_registers(struct pt_regs *regs);
 #define     StoreHW(addr, value, res) \
 		__asm__ __volatile__ (                      \
 			".set\tnoat\n"                      \
-			"1:\tsb\t%1, 1(%2)\n\t"             \
+			"1:\t"user_sb("%1", "1(%2)")"\n"    \
 			"srl\t$1, %1, 0x8\n"                \
-			"2:\tsb\t$1, 0(%2)\n\t"             \
+			"2:\t"user_sb("$1", "0(%2)")"\n"    \
 			".set\tat\n\t"                      \
 			"li\t%0, 0\n"                       \
 			"3:\n\t"                            \
@@ -229,8 +230,8 @@ extern void show_registers(struct pt_regs *regs);
 
 #define     StoreW(addr, value, res)  \
 		__asm__ __volatile__ (                      \
-			"1:\tswl\t%1,(%2)\n"                \
-			"2:\tswr\t%1, 3(%2)\n\t"            \
+			"1:\t"user_swl("%1", "(%2)")"\n"    \
+			"2:\t"user_swr("%1", "3(%2)")"\n\t" \
 			"li\t%0, 0\n"                       \
 			"3:\n\t"                            \
 			".insn\n\t"                         \
@@ -267,8 +268,8 @@ extern void show_registers(struct pt_regs *regs);
 #ifdef __LITTLE_ENDIAN
 #define     LoadHW(addr, value, res)  \
 		__asm__ __volatile__ (".set\tnoat\n"        \
-			"1:\tlb\t%0, 1(%2)\n"               \
-			"2:\tlbu\t$1, 0(%2)\n\t"            \
+			"1:\t"user_lb("%0", "1(%2)")"\n"    \
+			"2:\t"user_lbu("$1", "0(%2)")"\n\t" \
 			"sll\t%0, 0x8\n\t"                  \
 			"or\t%0, $1\n\t"                    \
 			"li\t%1, 0\n"                       \
@@ -287,8 +288,8 @@ extern void show_registers(struct pt_regs *regs);
 
 #define     LoadW(addr, value, res)   \
 		__asm__ __volatile__ (                      \
-			"1:\tlwl\t%0, 3(%2)\n"              \
-			"2:\tlwr\t%0, (%2)\n\t"             \
+			"1:\t"user_lwl("%0", "3(%2)")"\n"   \
+			"2:\t"user_lwr("%0", "(%2)")"\n\t"  \
 			"li\t%1, 0\n"                       \
 			"3:\n\t"                            \
 			".insn\n\t"                         \
@@ -306,8 +307,8 @@ extern void show_registers(struct pt_regs *regs);
 #define     LoadHWU(addr, value, res) \
 		__asm__ __volatile__ (                      \
 			".set\tnoat\n"                      \
-			"1:\tlbu\t%0, 1(%2)\n"              \
-			"2:\tlbu\t$1, 0(%2)\n\t"            \
+			"1:\t"user_lbu("%0", "1(%2)")"\n"   \
+			"2:\t"user_lbu("$1", "0(%2)")"\n\t" \
 			"sll\t%0, 0x8\n\t"                  \
 			"or\t%0, $1\n\t"                    \
 			"li\t%1, 0\n"                       \
@@ -327,8 +328,8 @@ extern void show_registers(struct pt_regs *regs);
 
 #define     LoadWU(addr, value, res)  \
 		__asm__ __volatile__ (                      \
-			"1:\tlwl\t%0, 3(%2)\n"              \
-			"2:\tlwr\t%0, (%2)\n\t"             \
+			"1:\t"user_lwl("%0", "3(%2)")"\n"   \
+			"2:\t"user_lwr("%0", "(%2)")"\n\t"  \
 			"dsll\t%0, %0, 32\n\t"              \
 			"dsrl\t%0, %0, 32\n\t"              \
 			"li\t%1, 0\n"                       \
@@ -366,9 +367,9 @@ extern void show_registers(struct pt_regs *regs);
 #define     StoreHW(addr, value, res) \
 		__asm__ __volatile__ (                      \
 			".set\tnoat\n"                      \
-			"1:\tsb\t%1, 0(%2)\n\t"             \
+			"1:\t"user_sb("%1", "0(%2)")"\n"    \
 			"srl\t$1,%1, 0x8\n"                 \
-			"2:\tsb\t$1, 1(%2)\n\t"             \
+			"2:\t"user_sb("$1", "1(%2)")"\n"    \
 			".set\tat\n\t"                      \
 			"li\t%0, 0\n"                       \
 			"3:\n\t"                            \
@@ -386,8 +387,8 @@ extern void show_registers(struct pt_regs *regs);
 
 #define     StoreW(addr, value, res)  \
 		__asm__ __volatile__ (                      \
-			"1:\tswl\t%1, 3(%2)\n"              \
-			"2:\tswr\t%1, (%2)\n\t"             \
+			"1:\t"user_swl("%1", "3(%2)")"\n"   \
+			"2:\t"user_swr("%1", "(%2)")"\n\t"  \
 			"li\t%0, 0\n"                       \
 			"3:\n\t"                            \
 			".insn\n\t"                         \
-- 
1.8.5.3

[PATCH 31/58] MIPS: kernel: unaligned: Handle unaligned accesses for EVA

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

Handle unaligned accesses when we access userspace memory
EVA mode.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/unaligned.c | 86 +++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 85 insertions(+), 1 deletion(-)

diff --git a/arch/mips/kernel/unaligned.c b/arch/mips/kernel/unaligned.c
index 5ec8f00..2b35172 100644
--- a/arch/mips/kernel/unaligned.c
+++ b/arch/mips/kernel/unaligned.c
@@ -431,7 +431,9 @@ static void emulate_load_store_insn(struct pt_regs *regs,
 	unsigned long origpc;
 	unsigned long orig31;
 	void __user *fault_addr = NULL;
-
+#ifdef	CONFIG_EVA
+	mm_segment_t seg;
+#endif
 	origpc = (unsigned long)pc;
 	orig31 = regs->regs[31];
 
@@ -476,6 +478,88 @@ static void emulate_load_store_insn(struct pt_regs *regs,
 		 * The remaining opcodes are the ones that are really of
 		 * interest.
 		 */
+#ifdef CONFIG_EVA
+	case spec3_op:
+		/*
+		 * we can land here only from kernel accessing user memory,
+		 * so we need to "switch" the address limit to user space, so
+		 * address check can work properly.
+		 */
+		seg = get_fs();
+		set_fs(USER_DS);
+		switch (insn.spec3_format.func) {
+		case lhe_op:
+			if (!access_ok(VERIFY_READ, addr, 2)) {
+				set_fs(seg);
+				goto sigbus;
+			}
+			LoadHW(addr, value, res);
+			if (res) {
+				set_fs(seg);
+				goto fault;
+			}
+			compute_return_epc(regs);
+			regs->regs[insn.spec3_format.rt] = value;
+			break;
+		case lwe_op:
+			if (!access_ok(VERIFY_READ, addr, 4)) {
+				set_fs(seg);
+				goto sigbus;
+			}
+				LoadW(addr, value, res);
+			if (res) {
+				set_fs(seg);
+				goto fault;
+			}
+			compute_return_epc(regs);
+			regs->regs[insn.spec3_format.rt] = value;
+			break;
+		case lhue_op:
+			if (!access_ok(VERIFY_READ, addr, 2)) {
+				set_fs(seg);
+				goto sigbus;
+			}
+			LoadHWU(addr, value, res);
+			if (res) {
+				set_fs(seg);
+				goto fault;
+			}
+			compute_return_epc(regs);
+			regs->regs[insn.spec3_format.rt] = value;
+			break;
+		case she_op:
+			if (!access_ok(VERIFY_WRITE, addr, 2)) {
+				set_fs(seg);
+				goto sigbus;
+			}
+			compute_return_epc(regs);
+			value = regs->regs[insn.spec3_format.rt];
+			StoreHW(addr, value, res);
+			if (res) {
+				set_fs(seg);
+				goto fault;
+			}
+			break;
+		case swe_op:
+			if (!access_ok(VERIFY_WRITE, addr, 4)) {
+				set_fs(seg);
+				goto sigbus;
+			}
+			compute_return_epc(regs);
+			value = regs->regs[insn.spec3_format.rt];
+			StoreW(addr, value, res);
+			if (res) {
+				set_fs(seg);
+				goto fault;
+			}
+			break;
+		default:
+			set_fs(seg);
+			goto sigill;
+		}
+		set_fs(seg);
+		break;
+#endif
 	case lh_op:
 		if (!access_ok(VERIFY_READ, addr, 2))
 			goto sigbus;
-- 
1.8.5.3

[PATCH 31/58] MIPS: kernel: unaligned: Handle unaligned accesses for EVA

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

Handle unaligned accesses when we access userspace memory
EVA mode.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/unaligned.c | 86 +++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 85 insertions(+), 1 deletion(-)

diff --git a/arch/mips/kernel/unaligned.c b/arch/mips/kernel/unaligned.c
index 5ec8f00..2b35172 100644
--- a/arch/mips/kernel/unaligned.c
+++ b/arch/mips/kernel/unaligned.c
@@ -431,7 +431,9 @@ static void emulate_load_store_insn(struct pt_regs *regs,
 	unsigned long origpc;
 	unsigned long orig31;
 	void __user *fault_addr = NULL;
-
+#ifdef	CONFIG_EVA
+	mm_segment_t seg;
+#endif
 	origpc = (unsigned long)pc;
 	orig31 = regs->regs[31];
 
@@ -476,6 +478,88 @@ static void emulate_load_store_insn(struct pt_regs *regs,
 		 * The remaining opcodes are the ones that are really of
 		 * interest.
 		 */
+#ifdef CONFIG_EVA
+	case spec3_op:
+		/*
+		 * we can land here only from kernel accessing user memory,
+		 * so we need to "switch" the address limit to user space, so
+		 * address check can work properly.
+		 */
+		seg = get_fs();
+		set_fs(USER_DS);
+		switch (insn.spec3_format.func) {
+		case lhe_op:
+			if (!access_ok(VERIFY_READ, addr, 2)) {
+				set_fs(seg);
+				goto sigbus;
+			}
+			LoadHW(addr, value, res);
+			if (res) {
+				set_fs(seg);
+				goto fault;
+			}
+			compute_return_epc(regs);
+			regs->regs[insn.spec3_format.rt] = value;
+			break;
+		case lwe_op:
+			if (!access_ok(VERIFY_READ, addr, 4)) {
+				set_fs(seg);
+				goto sigbus;
+			}
+				LoadW(addr, value, res);
+			if (res) {
+				set_fs(seg);
+				goto fault;
+			}
+			compute_return_epc(regs);
+			regs->regs[insn.spec3_format.rt] = value;
+			break;
+		case lhue_op:
+			if (!access_ok(VERIFY_READ, addr, 2)) {
+				set_fs(seg);
+				goto sigbus;
+			}
+			LoadHWU(addr, value, res);
+			if (res) {
+				set_fs(seg);
+				goto fault;
+			}
+			compute_return_epc(regs);
+			regs->regs[insn.spec3_format.rt] = value;
+			break;
+		case she_op:
+			if (!access_ok(VERIFY_WRITE, addr, 2)) {
+				set_fs(seg);
+				goto sigbus;
+			}
+			compute_return_epc(regs);
+			value = regs->regs[insn.spec3_format.rt];
+			StoreHW(addr, value, res);
+			if (res) {
+				set_fs(seg);
+				goto fault;
+			}
+			break;
+		case swe_op:
+			if (!access_ok(VERIFY_WRITE, addr, 4)) {
+				set_fs(seg);
+				goto sigbus;
+			}
+			compute_return_epc(regs);
+			value = regs->regs[insn.spec3_format.rt];
+			StoreW(addr, value, res);
+			if (res) {
+				set_fs(seg);
+				goto fault;
+			}
+			break;
+		default:
+			set_fs(seg);
+			goto sigill;
+		}
+		set_fs(seg);
+		break;
+#endif
 	case lh_op:
 		if (!access_ok(VERIFY_READ, addr, 2))
 			goto sigbus;
-- 
1.8.5.3

[PATCH 32/58] MIPS: checksum: Split the 'copy_user' symbol

[Markos Chandras]

The 'copy_user' symbol can be used to copy from or to
userland so we will use two different symbols for these
operations. This makes no difference in the existing code,
but when the core is operating in EVA mode, different instructions
need to be used to read and write to userland address space.
The old function has also been renamed to 'copy_kernel' to denote
that it is suitable for copy data to and from kernel space.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/checksum.h | 18 ++++++++++++------
 arch/mips/kernel/mips_ksyms.c    |  4 +++-
 arch/mips/lib/csum_partial.S     |  9 ++++++---
 3 files changed, 21 insertions(+), 10 deletions(-)

diff --git a/arch/mips/include/asm/checksum.h b/arch/mips/include/asm/checksum.h
index ac3d2b8..3c9aea5 100644
--- a/arch/mips/include/asm/checksum.h
+++ b/arch/mips/include/asm/checksum.h
@@ -7,6 +7,7 @@
  * Copyright (C) 1999 Silicon Graphics, Inc.
  * Copyright (C) 2001 Thiemo Seufer.
  * Copyright (C) 2002 Maciej W. Rozycki
+ * Copyright (C) 2014 Imagination Technologies Ltd.
  */
 #ifndef _ASM_CHECKSUM_H
 #define _ASM_CHECKSUM_H
@@ -29,8 +30,13 @@
  */
 __wsum csum_partial(const void *buff, int len, __wsum sum);
 
-__wsum __csum_partial_copy_user(const void *src, void *dst,
-				int len, __wsum sum, int *err_ptr);
+__wsum __csum_partial_copy_kernel(const void *src, void *dst,
+				  int len, __wsum sum, int *err_ptr);
+
+__wsum __csum_partial_copy_from_user(const void *src, void *dst,
+				     int len, __wsum sum, int *err_ptr);
+__wsum __csum_partial_copy_to_user(const void *src, void *dst,
+				   int len, __wsum sum, int *err_ptr);
 
 /*
  * this is a new version of the above that records errors it finds in *errp,
@@ -41,8 +47,8 @@ __wsum csum_partial_copy_from_user(const void __user *src, void *dst, int len,
 				   __wsum sum, int *err_ptr)
 {
 	might_fault();
-	return __csum_partial_copy_user((__force void *)src, dst,
-					len, sum, err_ptr);
+	return __csum_partial_copy_from_user((__force void *)src, dst,
+					     len, sum, err_ptr);
 }
 
 /*
@@ -55,8 +61,8 @@ __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len,
 {
 	might_fault();
 	if (access_ok(VERIFY_WRITE, dst, len))
-		return __csum_partial_copy_user(src, (__force void *)dst,
-						len, sum, err_ptr);
+		return __csum_partial_copy_to_user(src, (__force void *)dst,
+						   len, sum, err_ptr);
 	if (len)
 		*err_ptr = -EFAULT;
 
diff --git a/arch/mips/kernel/mips_ksyms.c b/arch/mips/kernel/mips_ksyms.c
index 13e60f6..2607c3a 100644
--- a/arch/mips/kernel/mips_ksyms.c
+++ b/arch/mips/kernel/mips_ksyms.c
@@ -73,7 +73,9 @@ EXPORT_SYMBOL(__strnlen_user_asm);
 
 EXPORT_SYMBOL(csum_partial);
 EXPORT_SYMBOL(csum_partial_copy_nocheck);
-EXPORT_SYMBOL(__csum_partial_copy_user);
+EXPORT_SYMBOL(__csum_partial_copy_kernel);
+EXPORT_SYMBOL(__csum_partial_copy_to_user);
+EXPORT_SYMBOL(__csum_partial_copy_from_user);
 
 EXPORT_SYMBOL(invalid_pte_table);
 #ifdef CONFIG_FUNCTION_TRACER
diff --git a/arch/mips/lib/csum_partial.S b/arch/mips/lib/csum_partial.S
index a6adffb..5d73d0d 100644
--- a/arch/mips/lib/csum_partial.S
+++ b/arch/mips/lib/csum_partial.S
@@ -8,6 +8,7 @@
  * Copyright (C) 1998, 1999 Ralf Baechle
  * Copyright (C) 1999 Silicon Graphics, Inc.
  * Copyright (C) 2007  Maciej W. Rozycki
+ * Copyright (C) 2014 Imagination Technologies Ltd.
  */
 #include <linux/errno.h>
 #include <asm/asm.h>
@@ -296,7 +297,7 @@ LEAF(csum_partial)
  * checksum and copy routines based on memcpy.S
  *
  *	csum_partial_copy_nocheck(src, dst, len, sum)
- *	__csum_partial_copy_user(src, dst, len, sum, errp)
+ *	__csum_partial_copy_kernel(src, dst, len, sum, errp)
  *
  * See "Spec" in memcpy.S for details.	Unlike __copy_user, all
  * function in this file use the standard calling convention.
@@ -396,7 +397,9 @@ LEAF(csum_partial)
 	.set	at=v1
 #endif
 
-LEAF(__csum_partial_copy_user)
+LEAF(__csum_partial_copy_kernel)
+FEXPORT(__csum_partial_copy_to_user)
+FEXPORT(__csum_partial_copy_from_user)
 	PTR_ADDU	AT, src, len	/* See (1) above. */
 #ifdef CONFIG_64BIT
 	move	errptr, a4
@@ -757,4 +760,4 @@ EXC(	lbu	t1, 0(src),	.Ll_exc)
 	jr	ra
 	 sw	v1, (errptr)
 	.set	pop
-	END(__csum_partial_copy_user)
+	END(__csum_partial_copy_kernel)
-- 
1.8.5.3

[PATCH 32/58] MIPS: checksum: Split the 'copy_user' symbol

[Markos Chandras]

The 'copy_user' symbol can be used to copy from or to
userland so we will use two different symbols for these
operations. This makes no difference in the existing code,
but when the core is operating in EVA mode, different instructions
need to be used to read and write to userland address space.
The old function has also been renamed to 'copy_kernel' to denote
that it is suitable for copy data to and from kernel space.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/checksum.h | 18 ++++++++++++------
 arch/mips/kernel/mips_ksyms.c    |  4 +++-
 arch/mips/lib/csum_partial.S     |  9 ++++++---
 3 files changed, 21 insertions(+), 10 deletions(-)

diff --git a/arch/mips/include/asm/checksum.h b/arch/mips/include/asm/checksum.h
index ac3d2b8..3c9aea5 100644
--- a/arch/mips/include/asm/checksum.h
+++ b/arch/mips/include/asm/checksum.h
@@ -7,6 +7,7 @@
  * Copyright (C) 1999 Silicon Graphics, Inc.
  * Copyright (C) 2001 Thiemo Seufer.
  * Copyright (C) 2002 Maciej W. Rozycki
+ * Copyright (C) 2014 Imagination Technologies Ltd.
  */
 #ifndef _ASM_CHECKSUM_H
 #define _ASM_CHECKSUM_H
@@ -29,8 +30,13 @@
  */
 __wsum csum_partial(const void *buff, int len, __wsum sum);
 
-__wsum __csum_partial_copy_user(const void *src, void *dst,
-				int len, __wsum sum, int *err_ptr);
+__wsum __csum_partial_copy_kernel(const void *src, void *dst,
+				  int len, __wsum sum, int *err_ptr);
+
+__wsum __csum_partial_copy_from_user(const void *src, void *dst,
+				     int len, __wsum sum, int *err_ptr);
+__wsum __csum_partial_copy_to_user(const void *src, void *dst,
+				   int len, __wsum sum, int *err_ptr);
 
 /*
  * this is a new version of the above that records errors it finds in *errp,
@@ -41,8 +47,8 @@ __wsum csum_partial_copy_from_user(const void __user *src, void *dst, int len,
 				   __wsum sum, int *err_ptr)
 {
 	might_fault();
-	return __csum_partial_copy_user((__force void *)src, dst,
-					len, sum, err_ptr);
+	return __csum_partial_copy_from_user((__force void *)src, dst,
+					     len, sum, err_ptr);
 }
 
 /*
@@ -55,8 +61,8 @@ __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len,
 {
 	might_fault();
 	if (access_ok(VERIFY_WRITE, dst, len))
-		return __csum_partial_copy_user(src, (__force void *)dst,
-						len, sum, err_ptr);
+		return __csum_partial_copy_to_user(src, (__force void *)dst,
+						   len, sum, err_ptr);
 	if (len)
 		*err_ptr = -EFAULT;
 
diff --git a/arch/mips/kernel/mips_ksyms.c b/arch/mips/kernel/mips_ksyms.c
index 13e60f6..2607c3a 100644
--- a/arch/mips/kernel/mips_ksyms.c
+++ b/arch/mips/kernel/mips_ksyms.c
@@ -73,7 +73,9 @@ EXPORT_SYMBOL(__strnlen_user_asm);
 
 EXPORT_SYMBOL(csum_partial);
 EXPORT_SYMBOL(csum_partial_copy_nocheck);
-EXPORT_SYMBOL(__csum_partial_copy_user);
+EXPORT_SYMBOL(__csum_partial_copy_kernel);
+EXPORT_SYMBOL(__csum_partial_copy_to_user);
+EXPORT_SYMBOL(__csum_partial_copy_from_user);
 
 EXPORT_SYMBOL(invalid_pte_table);
 #ifdef CONFIG_FUNCTION_TRACER
diff --git a/arch/mips/lib/csum_partial.S b/arch/mips/lib/csum_partial.S
index a6adffb..5d73d0d 100644
--- a/arch/mips/lib/csum_partial.S
+++ b/arch/mips/lib/csum_partial.S
@@ -8,6 +8,7 @@
  * Copyright (C) 1998, 1999 Ralf Baechle
  * Copyright (C) 1999 Silicon Graphics, Inc.
  * Copyright (C) 2007  Maciej W. Rozycki
+ * Copyright (C) 2014 Imagination Technologies Ltd.
  */
 #include <linux/errno.h>
 #include <asm/asm.h>
@@ -296,7 +297,7 @@ LEAF(csum_partial)
  * checksum and copy routines based on memcpy.S
  *
  *	csum_partial_copy_nocheck(src, dst, len, sum)
- *	__csum_partial_copy_user(src, dst, len, sum, errp)
+ *	__csum_partial_copy_kernel(src, dst, len, sum, errp)
  *
  * See "Spec" in memcpy.S for details.	Unlike __copy_user, all
  * function in this file use the standard calling convention.
@@ -396,7 +397,9 @@ LEAF(csum_partial)
 	.set	at=v1
 #endif
 
-LEAF(__csum_partial_copy_user)
+LEAF(__csum_partial_copy_kernel)
+FEXPORT(__csum_partial_copy_to_user)
+FEXPORT(__csum_partial_copy_from_user)
 	PTR_ADDU	AT, src, len	/* See (1) above. */
 #ifdef CONFIG_64BIT
 	move	errptr, a4
@@ -757,4 +760,4 @@ EXC(	lbu	t1, 0(src),	.Ll_exc)
 	jr	ra
 	 sw	v1, (errptr)
 	.set	pop
-	END(__csum_partial_copy_user)
+	END(__csum_partial_copy_kernel)
-- 
1.8.5.3

[PATCH 33/58] MIPS: lib: csum_partial: Merge EXC and load/store macros

[Markos Chandras]

Each load/store macro always adds an entry to the __ex_table
using the EXC macro. There are cases where a load instruction may
never fail such as when we are sure the load happens in the kernel
address space. Therefore, we merge these the EXC and LOADX/STOREX
macros into a single one. We also expand the argument list in the EXC
macro to make the macro more flexible. The extra 'type' argument is not
used by this commit, but it will be used when EVA support is added to
memcpy.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/csum_partial.S | 160 ++++++++++++++++++++++++-------------------
 1 file changed, 91 insertions(+), 69 deletions(-)

diff --git a/arch/mips/lib/csum_partial.S b/arch/mips/lib/csum_partial.S
index 5d73d0d..bff5167 100644
--- a/arch/mips/lib/csum_partial.S
+++ b/arch/mips/lib/csum_partial.S
@@ -328,20 +328,39 @@ LEAF(csum_partial)
  * These handlers do not need to overwrite any data.
  */
 
-#define EXC(inst_reg,addr,handler)		\
-9:	inst_reg, addr;				\
+/* Instruction type */
+#define LD_INSN 1
+#define ST_INSN 2
+
+/*
+ * Wrapper to add an entry in the exception table
+ * in case the insn causes a memory exception.
+ * Arguments:
+ * insn    : Load/store instruction
+ * type    : Instruction type
+ * reg     : Register
+ * addr    : Address
+ * handler : Exception handler
+ */
+#define EXC(insn, type, reg, addr, handler)	\
+9:	insn reg, addr;				\
 	.section __ex_table,"a";		\
 	PTR	9b, handler;			\
 	.previous
 
+#undef LOAD
+
 #ifdef USE_DOUBLE
 
-#define LOAD   ld
-#define LOADL  ldl
-#define LOADR  ldr
-#define STOREL sdl
-#define STORER sdr
-#define STORE  sd
+#define LOADK	ld /* No exception */
+#define LOAD(reg, addr, handler)	EXC(ld, LD_INSN, reg, addr, handler)
+#define LOADBU(reg, addr, handler)	EXC(lbu, LD_INSN, reg, addr, handler)
+#define LOADL(reg, addr, handler)	EXC(ldl, LD_INSN, reg, addr, handler)
+#define LOADR(reg, addr, handler)	EXC(ldr, LD_INSN, reg, addr, handler)
+#define STOREB(reg, addr, handler)	EXC(sb, ST_INSN, reg, addr, handler)
+#define STOREL(reg, addr, handler)	EXC(sdl, ST_INSN, reg, addr, handler)
+#define STORER(reg, addr, handler)	EXC(sdr, ST_INSN, reg, addr, handler)
+#define STORE(reg, addr, handler)	EXC(sd, ST_INSN, reg, addr, handler)
 #define ADD    daddu
 #define SUB    dsubu
 #define SRL    dsrl
@@ -353,12 +372,15 @@ LEAF(csum_partial)
 
 #else
 
-#define LOAD   lw
-#define LOADL  lwl
-#define LOADR  lwr
-#define STOREL swl
-#define STORER swr
-#define STORE  sw
+#define LOADK	lw /* No exception */
+#define LOAD(reg, addr, handler)	EXC(lw, LD_INSN, reg, addr, handler)
+#define LOADBU(reg, addr, handler)	EXC(lbu, LD_INSN, reg, addr, handler)
+#define LOADL(reg, addr, handler)	EXC(lwl, LD_INSN, reg, addr, handler)
+#define LOADR(reg, addr, handler)	EXC(lwr, LD_INSN, reg, addr, handler)
+#define STOREB(reg, addr, handler)	EXC(sb, ST_INSN, reg, addr, handler)
+#define STOREL(reg, addr, handler)	EXC(swl, ST_INSN, reg, addr, handler)
+#define STORER(reg, addr, handler)	EXC(swr, ST_INSN, reg, addr, handler)
+#define STORE(reg, addr, handler)	EXC(sw, ST_INSN, reg, addr, handler)
 #define ADD    addu
 #define SUB    subu
 #define SRL    srl
@@ -439,31 +461,31 @@ FEXPORT(csum_partial_copy_nocheck)
 	SUB	len, 8*NBYTES		# subtract here for bgez loop
 	.align	4
 1:
-EXC(	LOAD	t0, UNIT(0)(src),	.Ll_exc)
-EXC(	LOAD	t1, UNIT(1)(src),	.Ll_exc_copy)
-EXC(	LOAD	t2, UNIT(2)(src),	.Ll_exc_copy)
-EXC(	LOAD	t3, UNIT(3)(src),	.Ll_exc_copy)
-EXC(	LOAD	t4, UNIT(4)(src),	.Ll_exc_copy)
-EXC(	LOAD	t5, UNIT(5)(src),	.Ll_exc_copy)
-EXC(	LOAD	t6, UNIT(6)(src),	.Ll_exc_copy)
-EXC(	LOAD	t7, UNIT(7)(src),	.Ll_exc_copy)
+	LOAD(t0, UNIT(0)(src), .Ll_exc)
+	LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
+	LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
+	LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
+	LOAD(t4, UNIT(4)(src), .Ll_exc_copy)
+	LOAD(t5, UNIT(5)(src), .Ll_exc_copy)
+	LOAD(t6, UNIT(6)(src), .Ll_exc_copy)
+	LOAD(t7, UNIT(7)(src), .Ll_exc_copy)
 	SUB	len, len, 8*NBYTES
 	ADD	src, src, 8*NBYTES
-EXC(	STORE	t0, UNIT(0)(dst),	.Ls_exc)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc)
 	ADDC(sum, t0)
-EXC(	STORE	t1, UNIT(1)(dst),	.Ls_exc)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc)
 	ADDC(sum, t1)
-EXC(	STORE	t2, UNIT(2)(dst),	.Ls_exc)
+	STORE(t2, UNIT(2)(dst),	.Ls_exc)
 	ADDC(sum, t2)
-EXC(	STORE	t3, UNIT(3)(dst),	.Ls_exc)
+	STORE(t3, UNIT(3)(dst),	.Ls_exc)
 	ADDC(sum, t3)
-EXC(	STORE	t4, UNIT(4)(dst),	.Ls_exc)
+	STORE(t4, UNIT(4)(dst),	.Ls_exc)
 	ADDC(sum, t4)
-EXC(	STORE	t5, UNIT(5)(dst),	.Ls_exc)
+	STORE(t5, UNIT(5)(dst),	.Ls_exc)
 	ADDC(sum, t5)
-EXC(	STORE	t6, UNIT(6)(dst),	.Ls_exc)
+	STORE(t6, UNIT(6)(dst),	.Ls_exc)
 	ADDC(sum, t6)
-EXC(	STORE	t7, UNIT(7)(dst),	.Ls_exc)
+	STORE(t7, UNIT(7)(dst),	.Ls_exc)
 	ADDC(sum, t7)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 8*NBYTES
@@ -483,19 +505,19 @@ EXC(	STORE	t7, UNIT(7)(dst),	.Ls_exc)
 	/*
 	 * len >= 4*NBYTES
 	 */
-EXC(	LOAD	t0, UNIT(0)(src),	.Ll_exc)
-EXC(	LOAD	t1, UNIT(1)(src),	.Ll_exc_copy)
-EXC(	LOAD	t2, UNIT(2)(src),	.Ll_exc_copy)
-EXC(	LOAD	t3, UNIT(3)(src),	.Ll_exc_copy)
+	LOAD(t0, UNIT(0)(src), .Ll_exc)
+	LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
+	LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
+	LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
 	SUB	len, len, 4*NBYTES
 	ADD	src, src, 4*NBYTES
-EXC(	STORE	t0, UNIT(0)(dst),	.Ls_exc)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc)
 	ADDC(sum, t0)
-EXC(	STORE	t1, UNIT(1)(dst),	.Ls_exc)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc)
 	ADDC(sum, t1)
-EXC(	STORE	t2, UNIT(2)(dst),	.Ls_exc)
+	STORE(t2, UNIT(2)(dst),	.Ls_exc)
 	ADDC(sum, t2)
-EXC(	STORE	t3, UNIT(3)(dst),	.Ls_exc)
+	STORE(t3, UNIT(3)(dst),	.Ls_exc)
 	ADDC(sum, t3)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 4*NBYTES
@@ -508,10 +530,10 @@ EXC(	STORE	t3, UNIT(3)(dst),	.Ls_exc)
 	beq	rem, len, .Lcopy_bytes
 	 nop
 1:
-EXC(	LOAD	t0, 0(src),		.Ll_exc)
+	LOAD(t0, 0(src), .Ll_exc)
 	ADD	src, src, NBYTES
 	SUB	len, len, NBYTES
-EXC(	STORE	t0, 0(dst),		.Ls_exc)
+	STORE(t0, 0(dst), .Ls_exc)
 	ADDC(sum, t0)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, NBYTES
@@ -534,10 +556,10 @@ EXC(	STORE	t0, 0(dst),		.Ls_exc)
 	 ADD	t1, dst, len	# t1 is just past last byte of dst
 	li	bits, 8*NBYTES
 	SLL	rem, len, 3	# rem = number of bits to keep
-EXC(	LOAD	t0, 0(src),		.Ll_exc)
+	LOAD(t0, 0(src), .Ll_exc)
 	SUB	bits, bits, rem # bits = number of bits to discard
 	SHIFT_DISCARD t0, t0, bits
-EXC(	STREST	t0, -1(t1),		.Ls_exc)
+	STREST(t0, -1(t1), .Ls_exc)
 	SHIFT_DISCARD_REVERT t0, t0, bits
 	.set reorder
 	ADDC(sum, t0)
@@ -554,12 +576,12 @@ EXC(	STREST	t0, -1(t1),		.Ls_exc)
 	 * Set match = (src and dst have same alignment)
 	 */
 #define match rem
-EXC(	LDFIRST t3, FIRST(0)(src),	.Ll_exc)
+	LDFIRST(t3, FIRST(0)(src), .Ll_exc)
 	ADD	t2, zero, NBYTES
-EXC(	LDREST	t3, REST(0)(src),	.Ll_exc_copy)
+	LDREST(t3, REST(0)(src), .Ll_exc_copy)
 	SUB	t2, t2, t1	# t2 = number of bytes copied
 	xor	match, t0, t1
-EXC(	STFIRST t3, FIRST(0)(dst),	.Ls_exc)
+	STFIRST(t3, FIRST(0)(dst), .Ls_exc)
 	SLL	t4, t1, 3		# t4 = number of bits to discard
 	SHIFT_DISCARD t3, t3, t4
 	/* no SHIFT_DISCARD_REVERT to handle odd buffer properly */
@@ -581,26 +603,26 @@ EXC(	STFIRST t3, FIRST(0)(dst),	.Ls_exc)
  * It's OK to load FIRST(N+1) before REST(N) because the two addresses
  * are to the same unit (unless src is aligned, but it's not).
  */
-EXC(	LDFIRST t0, FIRST(0)(src),	.Ll_exc)
-EXC(	LDFIRST t1, FIRST(1)(src),	.Ll_exc_copy)
+	LDFIRST(t0, FIRST(0)(src), .Ll_exc)
+	LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy)
 	SUB	len, len, 4*NBYTES
-EXC(	LDREST	t0, REST(0)(src),	.Ll_exc_copy)
-EXC(	LDREST	t1, REST(1)(src),	.Ll_exc_copy)
-EXC(	LDFIRST t2, FIRST(2)(src),	.Ll_exc_copy)
-EXC(	LDFIRST t3, FIRST(3)(src),	.Ll_exc_copy)
-EXC(	LDREST	t2, REST(2)(src),	.Ll_exc_copy)
-EXC(	LDREST	t3, REST(3)(src),	.Ll_exc_copy)
+	LDREST(t0, REST(0)(src), .Ll_exc_copy)
+	LDREST(t1, REST(1)(src), .Ll_exc_copy)
+	LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy)
+	LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy)
+	LDREST(t2, REST(2)(src), .Ll_exc_copy)
+	LDREST(t3, REST(3)(src), .Ll_exc_copy)
 	ADD	src, src, 4*NBYTES
 #ifdef CONFIG_CPU_SB1
 	nop				# improves slotting
 #endif
-EXC(	STORE	t0, UNIT(0)(dst),	.Ls_exc)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc)
 	ADDC(sum, t0)
-EXC(	STORE	t1, UNIT(1)(dst),	.Ls_exc)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc)
 	ADDC(sum, t1)
-EXC(	STORE	t2, UNIT(2)(dst),	.Ls_exc)
+	STORE(t2, UNIT(2)(dst),	.Ls_exc)
 	ADDC(sum, t2)
-EXC(	STORE	t3, UNIT(3)(dst),	.Ls_exc)
+	STORE(t3, UNIT(3)(dst),	.Ls_exc)
 	ADDC(sum, t3)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 4*NBYTES
@@ -613,11 +635,11 @@ EXC(	STORE	t3, UNIT(3)(dst),	.Ls_exc)
 	beq	rem, len, .Lcopy_bytes
 	 nop
 1:
-EXC(	LDFIRST t0, FIRST(0)(src),	.Ll_exc)
-EXC(	LDREST	t0, REST(0)(src),	.Ll_exc_copy)
+	LDFIRST(t0, FIRST(0)(src), .Ll_exc)
+	LDREST(t0, REST(0)(src), .Ll_exc_copy)
 	ADD	src, src, NBYTES
 	SUB	len, len, NBYTES
-EXC(	STORE	t0, 0(dst),		.Ls_exc)
+	STORE(t0, 0(dst), .Ls_exc)
 	ADDC(sum, t0)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, NBYTES
@@ -640,9 +662,9 @@ EXC(	STORE	t0, 0(dst),		.Ls_exc)
 	li	t3, SHIFT_START # shift
 /* use .Ll_exc_copy here to return correct sum on fault */
 #define COPY_BYTE(N)			\
-EXC(	lbu	t0, N(src), .Ll_exc_copy);	\
+	LOADBU(t0, N(src), .Ll_exc_copy);	\
 	SUB	len, len, 1;		\
-EXC(	sb	t0, N(dst), .Ls_exc);	\
+	STOREB(t0, N(dst), .Ls_exc);	\
 	SLLV	t0, t0, t3;		\
 	addu	t3, SHIFT_INC;		\
 	beqz	len, .Lcopy_bytes_done; \
@@ -656,9 +678,9 @@ EXC(	sb	t0, N(dst), .Ls_exc);	\
 	COPY_BYTE(4)
 	COPY_BYTE(5)
 #endif
-EXC(	lbu	t0, NBYTES-2(src), .Ll_exc_copy)
+	LOADBU(t0, NBYTES-2(src), .Ll_exc_copy)
 	SUB	len, len, 1
-EXC(	sb	t0, NBYTES-2(dst), .Ls_exc)
+	STOREB(t0, NBYTES-2(dst), .Ls_exc)
 	SLLV	t0, t0, t3
 	or	t2, t0
 .Lcopy_bytes_done:
@@ -703,11 +725,11 @@ EXC(	sb	t0, NBYTES-2(dst), .Ls_exc)
 	 *
 	 * Assumes src < THREAD_BUADDR($28)
 	 */
-	LOAD	t0, TI_TASK($28)
+	LOADK	t0, TI_TASK($28)
 	 li	t2, SHIFT_START
-	LOAD	t0, THREAD_BUADDR(t0)
+	LOADK	t0, THREAD_BUADDR(t0)
 1:
-EXC(	lbu	t1, 0(src),	.Ll_exc)
+	LOADBU(t1, 0(src), .Ll_exc)
 	ADD	src, src, 1
 	sb	t1, 0(dst)	# can't fault -- we're copy_from_user
 	SLLV	t1, t1, t2
@@ -718,9 +740,9 @@ EXC(	lbu	t1, 0(src),	.Ll_exc)
 	bne	src, t0, 1b
 	.set	noreorder
 .Ll_exc:
-	LOAD	t0, TI_TASK($28)
+	LOADK	t0, TI_TASK($28)
 	 nop
-	LOAD	t0, THREAD_BUADDR(t0)	# t0 is just past last good address
+	LOADK	t0, THREAD_BUADDR(t0)	# t0 is just past last good address
 	 nop
 	SUB	len, AT, t0		# len number of uncopied bytes
 	/*
-- 
1.8.5.3

[PATCH 33/58] MIPS: lib: csum_partial: Merge EXC and load/store macros

[Markos Chandras]

Each load/store macro always adds an entry to the __ex_table
using the EXC macro. There are cases where a load instruction may
never fail such as when we are sure the load happens in the kernel
address space. Therefore, we merge these the EXC and LOADX/STOREX
macros into a single one. We also expand the argument list in the EXC
macro to make the macro more flexible. The extra 'type' argument is not
used by this commit, but it will be used when EVA support is added to
memcpy.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/csum_partial.S | 160 ++++++++++++++++++++++++-------------------
 1 file changed, 91 insertions(+), 69 deletions(-)

diff --git a/arch/mips/lib/csum_partial.S b/arch/mips/lib/csum_partial.S
index 5d73d0d..bff5167 100644
--- a/arch/mips/lib/csum_partial.S
+++ b/arch/mips/lib/csum_partial.S
@@ -328,20 +328,39 @@ LEAF(csum_partial)
  * These handlers do not need to overwrite any data.
  */
 
-#define EXC(inst_reg,addr,handler)		\
-9:	inst_reg, addr;				\
+/* Instruction type */
+#define LD_INSN 1
+#define ST_INSN 2
+
+/*
+ * Wrapper to add an entry in the exception table
+ * in case the insn causes a memory exception.
+ * Arguments:
+ * insn    : Load/store instruction
+ * type    : Instruction type
+ * reg     : Register
+ * addr    : Address
+ * handler : Exception handler
+ */
+#define EXC(insn, type, reg, addr, handler)	\
+9:	insn reg, addr;				\
 	.section __ex_table,"a";		\
 	PTR	9b, handler;			\
 	.previous
 
+#undef LOAD
+
 #ifdef USE_DOUBLE
 
-#define LOAD   ld
-#define LOADL  ldl
-#define LOADR  ldr
-#define STOREL sdl
-#define STORER sdr
-#define STORE  sd
+#define LOADK	ld /* No exception */
+#define LOAD(reg, addr, handler)	EXC(ld, LD_INSN, reg, addr, handler)
+#define LOADBU(reg, addr, handler)	EXC(lbu, LD_INSN, reg, addr, handler)
+#define LOADL(reg, addr, handler)	EXC(ldl, LD_INSN, reg, addr, handler)
+#define LOADR(reg, addr, handler)	EXC(ldr, LD_INSN, reg, addr, handler)
+#define STOREB(reg, addr, handler)	EXC(sb, ST_INSN, reg, addr, handler)
+#define STOREL(reg, addr, handler)	EXC(sdl, ST_INSN, reg, addr, handler)
+#define STORER(reg, addr, handler)	EXC(sdr, ST_INSN, reg, addr, handler)
+#define STORE(reg, addr, handler)	EXC(sd, ST_INSN, reg, addr, handler)
 #define ADD    daddu
 #define SUB    dsubu
 #define SRL    dsrl
@@ -353,12 +372,15 @@ LEAF(csum_partial)
 
 #else
 
-#define LOAD   lw
-#define LOADL  lwl
-#define LOADR  lwr
-#define STOREL swl
-#define STORER swr
-#define STORE  sw
+#define LOADK	lw /* No exception */
+#define LOAD(reg, addr, handler)	EXC(lw, LD_INSN, reg, addr, handler)
+#define LOADBU(reg, addr, handler)	EXC(lbu, LD_INSN, reg, addr, handler)
+#define LOADL(reg, addr, handler)	EXC(lwl, LD_INSN, reg, addr, handler)
+#define LOADR(reg, addr, handler)	EXC(lwr, LD_INSN, reg, addr, handler)
+#define STOREB(reg, addr, handler)	EXC(sb, ST_INSN, reg, addr, handler)
+#define STOREL(reg, addr, handler)	EXC(swl, ST_INSN, reg, addr, handler)
+#define STORER(reg, addr, handler)	EXC(swr, ST_INSN, reg, addr, handler)
+#define STORE(reg, addr, handler)	EXC(sw, ST_INSN, reg, addr, handler)
 #define ADD    addu
 #define SUB    subu
 #define SRL    srl
@@ -439,31 +461,31 @@ FEXPORT(csum_partial_copy_nocheck)
 	SUB	len, 8*NBYTES		# subtract here for bgez loop
 	.align	4
 1:
-EXC(	LOAD	t0, UNIT(0)(src),	.Ll_exc)
-EXC(	LOAD	t1, UNIT(1)(src),	.Ll_exc_copy)
-EXC(	LOAD	t2, UNIT(2)(src),	.Ll_exc_copy)
-EXC(	LOAD	t3, UNIT(3)(src),	.Ll_exc_copy)
-EXC(	LOAD	t4, UNIT(4)(src),	.Ll_exc_copy)
-EXC(	LOAD	t5, UNIT(5)(src),	.Ll_exc_copy)
-EXC(	LOAD	t6, UNIT(6)(src),	.Ll_exc_copy)
-EXC(	LOAD	t7, UNIT(7)(src),	.Ll_exc_copy)
+	LOAD(t0, UNIT(0)(src), .Ll_exc)
+	LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
+	LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
+	LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
+	LOAD(t4, UNIT(4)(src), .Ll_exc_copy)
+	LOAD(t5, UNIT(5)(src), .Ll_exc_copy)
+	LOAD(t6, UNIT(6)(src), .Ll_exc_copy)
+	LOAD(t7, UNIT(7)(src), .Ll_exc_copy)
 	SUB	len, len, 8*NBYTES
 	ADD	src, src, 8*NBYTES
-EXC(	STORE	t0, UNIT(0)(dst),	.Ls_exc)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc)
 	ADDC(sum, t0)
-EXC(	STORE	t1, UNIT(1)(dst),	.Ls_exc)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc)
 	ADDC(sum, t1)
-EXC(	STORE	t2, UNIT(2)(dst),	.Ls_exc)
+	STORE(t2, UNIT(2)(dst),	.Ls_exc)
 	ADDC(sum, t2)
-EXC(	STORE	t3, UNIT(3)(dst),	.Ls_exc)
+	STORE(t3, UNIT(3)(dst),	.Ls_exc)
 	ADDC(sum, t3)
-EXC(	STORE	t4, UNIT(4)(dst),	.Ls_exc)
+	STORE(t4, UNIT(4)(dst),	.Ls_exc)
 	ADDC(sum, t4)
-EXC(	STORE	t5, UNIT(5)(dst),	.Ls_exc)
+	STORE(t5, UNIT(5)(dst),	.Ls_exc)
 	ADDC(sum, t5)
-EXC(	STORE	t6, UNIT(6)(dst),	.Ls_exc)
+	STORE(t6, UNIT(6)(dst),	.Ls_exc)
 	ADDC(sum, t6)
-EXC(	STORE	t7, UNIT(7)(dst),	.Ls_exc)
+	STORE(t7, UNIT(7)(dst),	.Ls_exc)
 	ADDC(sum, t7)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 8*NBYTES
@@ -483,19 +505,19 @@ EXC(	STORE	t7, UNIT(7)(dst),	.Ls_exc)
 	/*
 	 * len >= 4*NBYTES
 	 */
-EXC(	LOAD	t0, UNIT(0)(src),	.Ll_exc)
-EXC(	LOAD	t1, UNIT(1)(src),	.Ll_exc_copy)
-EXC(	LOAD	t2, UNIT(2)(src),	.Ll_exc_copy)
-EXC(	LOAD	t3, UNIT(3)(src),	.Ll_exc_copy)
+	LOAD(t0, UNIT(0)(src), .Ll_exc)
+	LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
+	LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
+	LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
 	SUB	len, len, 4*NBYTES
 	ADD	src, src, 4*NBYTES
-EXC(	STORE	t0, UNIT(0)(dst),	.Ls_exc)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc)
 	ADDC(sum, t0)
-EXC(	STORE	t1, UNIT(1)(dst),	.Ls_exc)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc)
 	ADDC(sum, t1)
-EXC(	STORE	t2, UNIT(2)(dst),	.Ls_exc)
+	STORE(t2, UNIT(2)(dst),	.Ls_exc)
 	ADDC(sum, t2)
-EXC(	STORE	t3, UNIT(3)(dst),	.Ls_exc)
+	STORE(t3, UNIT(3)(dst),	.Ls_exc)
 	ADDC(sum, t3)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 4*NBYTES
@@ -508,10 +530,10 @@ EXC(	STORE	t3, UNIT(3)(dst),	.Ls_exc)
 	beq	rem, len, .Lcopy_bytes
 	 nop
 1:
-EXC(	LOAD	t0, 0(src),		.Ll_exc)
+	LOAD(t0, 0(src), .Ll_exc)
 	ADD	src, src, NBYTES
 	SUB	len, len, NBYTES
-EXC(	STORE	t0, 0(dst),		.Ls_exc)
+	STORE(t0, 0(dst), .Ls_exc)
 	ADDC(sum, t0)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, NBYTES
@@ -534,10 +556,10 @@ EXC(	STORE	t0, 0(dst),		.Ls_exc)
 	 ADD	t1, dst, len	# t1 is just past last byte of dst
 	li	bits, 8*NBYTES
 	SLL	rem, len, 3	# rem = number of bits to keep
-EXC(	LOAD	t0, 0(src),		.Ll_exc)
+	LOAD(t0, 0(src), .Ll_exc)
 	SUB	bits, bits, rem # bits = number of bits to discard
 	SHIFT_DISCARD t0, t0, bits
-EXC(	STREST	t0, -1(t1),		.Ls_exc)
+	STREST(t0, -1(t1), .Ls_exc)
 	SHIFT_DISCARD_REVERT t0, t0, bits
 	.set reorder
 	ADDC(sum, t0)
@@ -554,12 +576,12 @@ EXC(	STREST	t0, -1(t1),		.Ls_exc)
 	 * Set match = (src and dst have same alignment)
 	 */
 #define match rem
-EXC(	LDFIRST t3, FIRST(0)(src),	.Ll_exc)
+	LDFIRST(t3, FIRST(0)(src), .Ll_exc)
 	ADD	t2, zero, NBYTES
-EXC(	LDREST	t3, REST(0)(src),	.Ll_exc_copy)
+	LDREST(t3, REST(0)(src), .Ll_exc_copy)
 	SUB	t2, t2, t1	# t2 = number of bytes copied
 	xor	match, t0, t1
-EXC(	STFIRST t3, FIRST(0)(dst),	.Ls_exc)
+	STFIRST(t3, FIRST(0)(dst), .Ls_exc)
 	SLL	t4, t1, 3		# t4 = number of bits to discard
 	SHIFT_DISCARD t3, t3, t4
 	/* no SHIFT_DISCARD_REVERT to handle odd buffer properly */
@@ -581,26 +603,26 @@ EXC(	STFIRST t3, FIRST(0)(dst),	.Ls_exc)
  * It's OK to load FIRST(N+1) before REST(N) because the two addresses
  * are to the same unit (unless src is aligned, but it's not).
  */
-EXC(	LDFIRST t0, FIRST(0)(src),	.Ll_exc)
-EXC(	LDFIRST t1, FIRST(1)(src),	.Ll_exc_copy)
+	LDFIRST(t0, FIRST(0)(src), .Ll_exc)
+	LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy)
 	SUB	len, len, 4*NBYTES
-EXC(	LDREST	t0, REST(0)(src),	.Ll_exc_copy)
-EXC(	LDREST	t1, REST(1)(src),	.Ll_exc_copy)
-EXC(	LDFIRST t2, FIRST(2)(src),	.Ll_exc_copy)
-EXC(	LDFIRST t3, FIRST(3)(src),	.Ll_exc_copy)
-EXC(	LDREST	t2, REST(2)(src),	.Ll_exc_copy)
-EXC(	LDREST	t3, REST(3)(src),	.Ll_exc_copy)
+	LDREST(t0, REST(0)(src), .Ll_exc_copy)
+	LDREST(t1, REST(1)(src), .Ll_exc_copy)
+	LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy)
+	LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy)
+	LDREST(t2, REST(2)(src), .Ll_exc_copy)
+	LDREST(t3, REST(3)(src), .Ll_exc_copy)
 	ADD	src, src, 4*NBYTES
 #ifdef CONFIG_CPU_SB1
 	nop				# improves slotting
 #endif
-EXC(	STORE	t0, UNIT(0)(dst),	.Ls_exc)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc)
 	ADDC(sum, t0)
-EXC(	STORE	t1, UNIT(1)(dst),	.Ls_exc)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc)
 	ADDC(sum, t1)
-EXC(	STORE	t2, UNIT(2)(dst),	.Ls_exc)
+	STORE(t2, UNIT(2)(dst),	.Ls_exc)
 	ADDC(sum, t2)
-EXC(	STORE	t3, UNIT(3)(dst),	.Ls_exc)
+	STORE(t3, UNIT(3)(dst),	.Ls_exc)
 	ADDC(sum, t3)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 4*NBYTES
@@ -613,11 +635,11 @@ EXC(	STORE	t3, UNIT(3)(dst),	.Ls_exc)
 	beq	rem, len, .Lcopy_bytes
 	 nop
 1:
-EXC(	LDFIRST t0, FIRST(0)(src),	.Ll_exc)
-EXC(	LDREST	t0, REST(0)(src),	.Ll_exc_copy)
+	LDFIRST(t0, FIRST(0)(src), .Ll_exc)
+	LDREST(t0, REST(0)(src), .Ll_exc_copy)
 	ADD	src, src, NBYTES
 	SUB	len, len, NBYTES
-EXC(	STORE	t0, 0(dst),		.Ls_exc)
+	STORE(t0, 0(dst), .Ls_exc)
 	ADDC(sum, t0)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, NBYTES
@@ -640,9 +662,9 @@ EXC(	STORE	t0, 0(dst),		.Ls_exc)
 	li	t3, SHIFT_START # shift
 /* use .Ll_exc_copy here to return correct sum on fault */
 #define COPY_BYTE(N)			\
-EXC(	lbu	t0, N(src), .Ll_exc_copy);	\
+	LOADBU(t0, N(src), .Ll_exc_copy);	\
 	SUB	len, len, 1;		\
-EXC(	sb	t0, N(dst), .Ls_exc);	\
+	STOREB(t0, N(dst), .Ls_exc);	\
 	SLLV	t0, t0, t3;		\
 	addu	t3, SHIFT_INC;		\
 	beqz	len, .Lcopy_bytes_done; \
@@ -656,9 +678,9 @@ EXC(	sb	t0, N(dst), .Ls_exc);	\
 	COPY_BYTE(4)
 	COPY_BYTE(5)
 #endif
-EXC(	lbu	t0, NBYTES-2(src), .Ll_exc_copy)
+	LOADBU(t0, NBYTES-2(src), .Ll_exc_copy)
 	SUB	len, len, 1
-EXC(	sb	t0, NBYTES-2(dst), .Ls_exc)
+	STOREB(t0, NBYTES-2(dst), .Ls_exc)
 	SLLV	t0, t0, t3
 	or	t2, t0
 .Lcopy_bytes_done:
@@ -703,11 +725,11 @@ EXC(	sb	t0, NBYTES-2(dst), .Ls_exc)
 	 *
 	 * Assumes src < THREAD_BUADDR($28)
 	 */
-	LOAD	t0, TI_TASK($28)
+	LOADK	t0, TI_TASK($28)
 	 li	t2, SHIFT_START
-	LOAD	t0, THREAD_BUADDR(t0)
+	LOADK	t0, THREAD_BUADDR(t0)
 1:
-EXC(	lbu	t1, 0(src),	.Ll_exc)
+	LOADBU(t1, 0(src), .Ll_exc)
 	ADD	src, src, 1
 	sb	t1, 0(dst)	# can't fault -- we're copy_from_user
 	SLLV	t1, t1, t2
@@ -718,9 +740,9 @@ EXC(	lbu	t1, 0(src),	.Ll_exc)
 	bne	src, t0, 1b
 	.set	noreorder
 .Ll_exc:
-	LOAD	t0, TI_TASK($28)
+	LOADK	t0, TI_TASK($28)
 	 nop
-	LOAD	t0, THREAD_BUADDR(t0)	# t0 is just past last good address
+	LOADK	t0, THREAD_BUADDR(t0)	# t0 is just past last good address
 	 nop
 	SUB	len, AT, t0		# len number of uncopied bytes
 	/*
-- 
1.8.5.3

[PATCH 34/58] MIPS: lib: csum_partial: Add macro to build csum_partial symbols

[Markos Chandras]

In preparation for EVA support, we use a macro to build the
__csum_partial_copy_user main code so it can be shared across
multiple implementations. EVA uses the same code but it replaces
the load/store/prefetch instructions with the EVA specific ones
therefore using a macro avoids unnecessary code duplications.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/csum_partial.S | 200 +++++++++++++++++++++++--------------------
 1 file changed, 108 insertions(+), 92 deletions(-)

diff --git a/arch/mips/lib/csum_partial.S b/arch/mips/lib/csum_partial.S
index bff5167..62c8768 100644
--- a/arch/mips/lib/csum_partial.S
+++ b/arch/mips/lib/csum_partial.S
@@ -331,6 +331,10 @@ LEAF(csum_partial)
 /* Instruction type */
 #define LD_INSN 1
 #define ST_INSN 2
+#define LEGACY_MODE 1
+#define EVA_MODE    2
+#define USEROP   1
+#define KERNELOP 2
 
 /*
  * Wrapper to add an entry in the exception table
@@ -343,10 +347,12 @@ LEAF(csum_partial)
  * handler : Exception handler
  */
 #define EXC(insn, type, reg, addr, handler)	\
-9:	insn reg, addr;				\
-	.section __ex_table,"a";		\
-	PTR	9b, handler;			\
-	.previous
+	.if \mode == LEGACY_MODE;		\
+9:		insn reg, addr;			\
+		.section __ex_table,"a";	\
+		PTR	9b, handler;		\
+		.previous;			\
+	.endif
 
 #undef LOAD
 
@@ -419,16 +425,20 @@ LEAF(csum_partial)
 	.set	at=v1
 #endif
 
-LEAF(__csum_partial_copy_kernel)
-FEXPORT(__csum_partial_copy_to_user)
-FEXPORT(__csum_partial_copy_from_user)
+	.macro __BUILD_CSUM_PARTIAL_COPY_USER mode, from, to, __nocheck
+
 	PTR_ADDU	AT, src, len	/* See (1) above. */
+	/* initialize __nocheck if this the first time we execute this
+	 * macro
+	 */
 #ifdef CONFIG_64BIT
 	move	errptr, a4
 #else
 	lw	errptr, 16(sp)
 #endif
-FEXPORT(csum_partial_copy_nocheck)
+	.if \__nocheck == 1
+	FEXPORT(csum_partial_copy_nocheck)
+	.endif
 	move	sum, zero
 	move	odd, zero
 	/*
@@ -444,48 +454,48 @@ FEXPORT(csum_partial_copy_nocheck)
 	 */
 	sltu	t2, len, NBYTES
 	and	t1, dst, ADDRMASK
-	bnez	t2, .Lcopy_bytes_checklen
+	bnez	t2, .Lcopy_bytes_checklen\@
 	 and	t0, src, ADDRMASK
 	andi	odd, dst, 0x1			/* odd buffer? */
-	bnez	t1, .Ldst_unaligned
+	bnez	t1, .Ldst_unaligned\@
 	 nop
-	bnez	t0, .Lsrc_unaligned_dst_aligned
+	bnez	t0, .Lsrc_unaligned_dst_aligned\@
 	/*
 	 * use delay slot for fall-through
 	 * src and dst are aligned; need to compute rem
 	 */
-.Lboth_aligned:
+.Lboth_aligned\@:
 	 SRL	t0, len, LOG_NBYTES+3	 # +3 for 8 units/iter
-	beqz	t0, .Lcleanup_both_aligned # len < 8*NBYTES
+	beqz	t0, .Lcleanup_both_aligned\@ # len < 8*NBYTES
 	 nop
 	SUB	len, 8*NBYTES		# subtract here for bgez loop
 	.align	4
 1:
-	LOAD(t0, UNIT(0)(src), .Ll_exc)
-	LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
-	LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
-	LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
-	LOAD(t4, UNIT(4)(src), .Ll_exc_copy)
-	LOAD(t5, UNIT(5)(src), .Ll_exc_copy)
-	LOAD(t6, UNIT(6)(src), .Ll_exc_copy)
-	LOAD(t7, UNIT(7)(src), .Ll_exc_copy)
+	LOAD(t0, UNIT(0)(src), .Ll_exc\@)
+	LOAD(t1, UNIT(1)(src), .Ll_exc_copy\@)
+	LOAD(t2, UNIT(2)(src), .Ll_exc_copy\@)
+	LOAD(t3, UNIT(3)(src), .Ll_exc_copy\@)
+	LOAD(t4, UNIT(4)(src), .Ll_exc_copy\@)
+	LOAD(t5, UNIT(5)(src), .Ll_exc_copy\@)
+	LOAD(t6, UNIT(6)(src), .Ll_exc_copy\@)
+	LOAD(t7, UNIT(7)(src), .Ll_exc_copy\@)
 	SUB	len, len, 8*NBYTES
 	ADD	src, src, 8*NBYTES
-	STORE(t0, UNIT(0)(dst),	.Ls_exc)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc\@)
 	ADDC(sum, t0)
-	STORE(t1, UNIT(1)(dst),	.Ls_exc)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc\@)
 	ADDC(sum, t1)
-	STORE(t2, UNIT(2)(dst),	.Ls_exc)
+	STORE(t2, UNIT(2)(dst),	.Ls_exc\@)
 	ADDC(sum, t2)
-	STORE(t3, UNIT(3)(dst),	.Ls_exc)
+	STORE(t3, UNIT(3)(dst),	.Ls_exc\@)
 	ADDC(sum, t3)
-	STORE(t4, UNIT(4)(dst),	.Ls_exc)
+	STORE(t4, UNIT(4)(dst),	.Ls_exc\@)
 	ADDC(sum, t4)
-	STORE(t5, UNIT(5)(dst),	.Ls_exc)
+	STORE(t5, UNIT(5)(dst),	.Ls_exc\@)
 	ADDC(sum, t5)
-	STORE(t6, UNIT(6)(dst),	.Ls_exc)
+	STORE(t6, UNIT(6)(dst),	.Ls_exc\@)
 	ADDC(sum, t6)
-	STORE(t7, UNIT(7)(dst),	.Ls_exc)
+	STORE(t7, UNIT(7)(dst),	.Ls_exc\@)
 	ADDC(sum, t7)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 8*NBYTES
@@ -496,44 +506,44 @@ FEXPORT(csum_partial_copy_nocheck)
 	/*
 	 * len == the number of bytes left to copy < 8*NBYTES
 	 */
-.Lcleanup_both_aligned:
+.Lcleanup_both_aligned\@:
 #define rem t7
-	beqz	len, .Ldone
+	beqz	len, .Ldone\@
 	 sltu	t0, len, 4*NBYTES
-	bnez	t0, .Lless_than_4units
+	bnez	t0, .Lless_than_4units\@
 	 and	rem, len, (NBYTES-1)	# rem = len % NBYTES
 	/*
 	 * len >= 4*NBYTES
 	 */
-	LOAD(t0, UNIT(0)(src), .Ll_exc)
-	LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
-	LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
-	LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
+	LOAD(t0, UNIT(0)(src), .Ll_exc\@)
+	LOAD(t1, UNIT(1)(src), .Ll_exc_copy\@)
+	LOAD(t2, UNIT(2)(src), .Ll_exc_copy\@)
+	LOAD(t3, UNIT(3)(src), .Ll_exc_copy\@)
 	SUB	len, len, 4*NBYTES
 	ADD	src, src, 4*NBYTES
-	STORE(t0, UNIT(0)(dst),	.Ls_exc)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc\@)
 	ADDC(sum, t0)
-	STORE(t1, UNIT(1)(dst),	.Ls_exc)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc\@)
 	ADDC(sum, t1)
-	STORE(t2, UNIT(2)(dst),	.Ls_exc)
+	STORE(t2, UNIT(2)(dst),	.Ls_exc\@)
 	ADDC(sum, t2)
-	STORE(t3, UNIT(3)(dst),	.Ls_exc)
+	STORE(t3, UNIT(3)(dst),	.Ls_exc\@)
 	ADDC(sum, t3)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 4*NBYTES
-	beqz	len, .Ldone
+	beqz	len, .Ldone\@
 	.set	noreorder
-.Lless_than_4units:
+.Lless_than_4units\@:
 	/*
 	 * rem = len % NBYTES
 	 */
-	beq	rem, len, .Lcopy_bytes
+	beq	rem, len, .Lcopy_bytes\@
 	 nop
 1:
-	LOAD(t0, 0(src), .Ll_exc)
+	LOAD(t0, 0(src), .Ll_exc\@)
 	ADD	src, src, NBYTES
 	SUB	len, len, NBYTES
-	STORE(t0, 0(dst), .Ls_exc)
+	STORE(t0, 0(dst), .Ls_exc\@)
 	ADDC(sum, t0)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, NBYTES
@@ -552,20 +562,20 @@ FEXPORT(csum_partial_copy_nocheck)
 	 * more instruction-level parallelism.
 	 */
 #define bits t2
-	beqz	len, .Ldone
+	beqz	len, .Ldone\@
 	 ADD	t1, dst, len	# t1 is just past last byte of dst
 	li	bits, 8*NBYTES
 	SLL	rem, len, 3	# rem = number of bits to keep
-	LOAD(t0, 0(src), .Ll_exc)
+	LOAD(t0, 0(src), .Ll_exc\@)
 	SUB	bits, bits, rem # bits = number of bits to discard
 	SHIFT_DISCARD t0, t0, bits
-	STREST(t0, -1(t1), .Ls_exc)
+	STREST(t0, -1(t1), .Ls_exc\@)
 	SHIFT_DISCARD_REVERT t0, t0, bits
 	.set reorder
 	ADDC(sum, t0)
-	b	.Ldone
+	b	.Ldone\@
 	.set noreorder
-.Ldst_unaligned:
+.Ldst_unaligned\@:
 	/*
 	 * dst is unaligned
 	 * t0 = src & ADDRMASK
@@ -576,25 +586,25 @@ FEXPORT(csum_partial_copy_nocheck)
 	 * Set match = (src and dst have same alignment)
 	 */
 #define match rem
-	LDFIRST(t3, FIRST(0)(src), .Ll_exc)
+	LDFIRST(t3, FIRST(0)(src), .Ll_exc\@)
 	ADD	t2, zero, NBYTES
-	LDREST(t3, REST(0)(src), .Ll_exc_copy)
+	LDREST(t3, REST(0)(src), .Ll_exc_copy\@)
 	SUB	t2, t2, t1	# t2 = number of bytes copied
 	xor	match, t0, t1
-	STFIRST(t3, FIRST(0)(dst), .Ls_exc)
+	STFIRST(t3, FIRST(0)(dst), .Ls_exc\@)
 	SLL	t4, t1, 3		# t4 = number of bits to discard
 	SHIFT_DISCARD t3, t3, t4
 	/* no SHIFT_DISCARD_REVERT to handle odd buffer properly */
 	ADDC(sum, t3)
-	beq	len, t2, .Ldone
+	beq	len, t2, .Ldone\@
 	 SUB	len, len, t2
 	ADD	dst, dst, t2
-	beqz	match, .Lboth_aligned
+	beqz	match, .Lboth_aligned\@
 	 ADD	src, src, t2
 
-.Lsrc_unaligned_dst_aligned:
+.Lsrc_unaligned_dst_aligned\@:
 	SRL	t0, len, LOG_NBYTES+2	 # +2 for 4 units/iter
-	beqz	t0, .Lcleanup_src_unaligned
+	beqz	t0, .Lcleanup_src_unaligned\@
 	 and	rem, len, (4*NBYTES-1)	 # rem = len % 4*NBYTES
 1:
 /*
@@ -603,53 +613,53 @@ FEXPORT(csum_partial_copy_nocheck)
  * It's OK to load FIRST(N+1) before REST(N) because the two addresses
  * are to the same unit (unless src is aligned, but it's not).
  */
-	LDFIRST(t0, FIRST(0)(src), .Ll_exc)
-	LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy)
+	LDFIRST(t0, FIRST(0)(src), .Ll_exc\@)
+	LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy\@)
 	SUB	len, len, 4*NBYTES
-	LDREST(t0, REST(0)(src), .Ll_exc_copy)
-	LDREST(t1, REST(1)(src), .Ll_exc_copy)
-	LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy)
-	LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy)
-	LDREST(t2, REST(2)(src), .Ll_exc_copy)
-	LDREST(t3, REST(3)(src), .Ll_exc_copy)
+	LDREST(t0, REST(0)(src), .Ll_exc_copy\@)
+	LDREST(t1, REST(1)(src), .Ll_exc_copy\@)
+	LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy\@)
+	LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy\@)
+	LDREST(t2, REST(2)(src), .Ll_exc_copy\@)
+	LDREST(t3, REST(3)(src), .Ll_exc_copy\@)
 	ADD	src, src, 4*NBYTES
 #ifdef CONFIG_CPU_SB1
 	nop				# improves slotting
 #endif
-	STORE(t0, UNIT(0)(dst),	.Ls_exc)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc\@)
 	ADDC(sum, t0)
-	STORE(t1, UNIT(1)(dst),	.Ls_exc)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc\@)
 	ADDC(sum, t1)
-	STORE(t2, UNIT(2)(dst),	.Ls_exc)
+	STORE(t2, UNIT(2)(dst),	.Ls_exc\@)
 	ADDC(sum, t2)
-	STORE(t3, UNIT(3)(dst),	.Ls_exc)
+	STORE(t3, UNIT(3)(dst),	.Ls_exc\@)
 	ADDC(sum, t3)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 4*NBYTES
 	bne	len, rem, 1b
 	.set	noreorder
 
-.Lcleanup_src_unaligned:
-	beqz	len, .Ldone
+.Lcleanup_src_unaligned\@:
+	beqz	len, .Ldone\@
 	 and	rem, len, NBYTES-1  # rem = len % NBYTES
-	beq	rem, len, .Lcopy_bytes
+	beq	rem, len, .Lcopy_bytes\@
 	 nop
 1:
-	LDFIRST(t0, FIRST(0)(src), .Ll_exc)
-	LDREST(t0, REST(0)(src), .Ll_exc_copy)
+	LDFIRST(t0, FIRST(0)(src), .Ll_exc\@)
+	LDREST(t0, REST(0)(src), .Ll_exc_copy\@)
 	ADD	src, src, NBYTES
 	SUB	len, len, NBYTES
-	STORE(t0, 0(dst), .Ls_exc)
+	STORE(t0, 0(dst), .Ls_exc\@)
 	ADDC(sum, t0)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, NBYTES
 	bne	len, rem, 1b
 	.set	noreorder
 
-.Lcopy_bytes_checklen:
-	beqz	len, .Ldone
+.Lcopy_bytes_checklen\@:
+	beqz	len, .Ldone\@
 	 nop
-.Lcopy_bytes:
+.Lcopy_bytes\@:
 	/* 0 < len < NBYTES  */
 #ifdef CONFIG_CPU_LITTLE_ENDIAN
 #define SHIFT_START 0
@@ -662,12 +672,12 @@ FEXPORT(csum_partial_copy_nocheck)
 	li	t3, SHIFT_START # shift
 /* use .Ll_exc_copy here to return correct sum on fault */
 #define COPY_BYTE(N)			\
-	LOADBU(t0, N(src), .Ll_exc_copy);	\
+	LOADBU(t0, N(src), .Ll_exc_copy\@);	\
 	SUB	len, len, 1;		\
-	STOREB(t0, N(dst), .Ls_exc);	\
+	STOREB(t0, N(dst), .Ls_exc\@);	\
 	SLLV	t0, t0, t3;		\
 	addu	t3, SHIFT_INC;		\
-	beqz	len, .Lcopy_bytes_done; \
+	beqz	len, .Lcopy_bytes_done\@; \
 	 or	t2, t0
 
 	COPY_BYTE(0)
@@ -678,14 +688,14 @@ FEXPORT(csum_partial_copy_nocheck)
 	COPY_BYTE(4)
 	COPY_BYTE(5)
 #endif
-	LOADBU(t0, NBYTES-2(src), .Ll_exc_copy)
+	LOADBU(t0, NBYTES-2(src), .Ll_exc_copy\@)
 	SUB	len, len, 1
-	STOREB(t0, NBYTES-2(dst), .Ls_exc)
+	STOREB(t0, NBYTES-2(dst), .Ls_exc\@)
 	SLLV	t0, t0, t3
 	or	t2, t0
-.Lcopy_bytes_done:
+.Lcopy_bytes_done\@:
 	ADDC(sum, t2)
-.Ldone:
+.Ldone\@:
 	/* fold checksum */
 #ifdef USE_DOUBLE
 	dsll32	v1, sum, 0
@@ -714,7 +724,7 @@ FEXPORT(csum_partial_copy_nocheck)
 	jr	ra
 	.set noreorder
 
-.Ll_exc_copy:
+.Ll_exc_copy\@:
 	/*
 	 * Copy bytes from src until faulting load address (or until a
 	 * lb faults)
@@ -729,7 +739,7 @@ FEXPORT(csum_partial_copy_nocheck)
 	 li	t2, SHIFT_START
 	LOADK	t0, THREAD_BUADDR(t0)
 1:
-	LOADBU(t1, 0(src), .Ll_exc)
+	LOADBU(t1, 0(src), .Ll_exc\@)
 	ADD	src, src, 1
 	sb	t1, 0(dst)	# can't fault -- we're copy_from_user
 	SLLV	t1, t1, t2
@@ -739,7 +749,7 @@ FEXPORT(csum_partial_copy_nocheck)
 	ADD	dst, dst, 1
 	bne	src, t0, 1b
 	.set	noreorder
-.Ll_exc:
+.Ll_exc\@:
 	LOADK	t0, TI_TASK($28)
 	 nop
 	LOADK	t0, THREAD_BUADDR(t0)	# t0 is just past last good address
@@ -758,7 +768,7 @@ FEXPORT(csum_partial_copy_nocheck)
 	 */
 	.set	reorder				/* DADDI_WAR */
 	SUB	src, len, 1
-	beqz	len, .Ldone
+	beqz	len, .Ldone\@
 	.set	noreorder
 1:	sb	zero, 0(dst)
 	ADD	dst, dst, 1
@@ -773,13 +783,19 @@ FEXPORT(csum_partial_copy_nocheck)
 	 SUB	src, src, v1
 #endif
 	li	v1, -EFAULT
-	b	.Ldone
+	b	.Ldone\@
 	 sw	v1, (errptr)
 
-.Ls_exc:
+.Ls_exc\@:
 	li	v0, -1 /* invalid checksum */
 	li	v1, -EFAULT
 	jr	ra
 	 sw	v1, (errptr)
 	.set	pop
-	END(__csum_partial_copy_kernel)
+	.endm
+
+LEAF(__csum_partial_copy_kernel)
+FEXPORT(__csum_partial_copy_to_user)
+FEXPORT(__csum_partial_copy_from_user)
+__BUILD_CSUM_PARTIAL_COPY_USER LEGACY_MODE USEROP USEROP 1
+END(__csum_partial_copy_kernel)
-- 
1.8.5.3

[PATCH 34/58] MIPS: lib: csum_partial: Add macro to build csum_partial symbols

[Markos Chandras]

In preparation for EVA support, we use a macro to build the
__csum_partial_copy_user main code so it can be shared across
multiple implementations. EVA uses the same code but it replaces
the load/store/prefetch instructions with the EVA specific ones
therefore using a macro avoids unnecessary code duplications.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/csum_partial.S | 200 +++++++++++++++++++++++--------------------
 1 file changed, 108 insertions(+), 92 deletions(-)

diff --git a/arch/mips/lib/csum_partial.S b/arch/mips/lib/csum_partial.S
index bff5167..62c8768 100644
--- a/arch/mips/lib/csum_partial.S
+++ b/arch/mips/lib/csum_partial.S
@@ -331,6 +331,10 @@ LEAF(csum_partial)
 /* Instruction type */
 #define LD_INSN 1
 #define ST_INSN 2
+#define LEGACY_MODE 1
+#define EVA_MODE    2
+#define USEROP   1
+#define KERNELOP 2
 
 /*
  * Wrapper to add an entry in the exception table
@@ -343,10 +347,12 @@ LEAF(csum_partial)
  * handler : Exception handler
  */
 #define EXC(insn, type, reg, addr, handler)	\
-9:	insn reg, addr;				\
-	.section __ex_table,"a";		\
-	PTR	9b, handler;			\
-	.previous
+	.if \mode == LEGACY_MODE;		\
+9:		insn reg, addr;			\
+		.section __ex_table,"a";	\
+		PTR	9b, handler;		\
+		.previous;			\
+	.endif
 
 #undef LOAD
 
@@ -419,16 +425,20 @@ LEAF(csum_partial)
 	.set	at=v1
 #endif
 
-LEAF(__csum_partial_copy_kernel)
-FEXPORT(__csum_partial_copy_to_user)
-FEXPORT(__csum_partial_copy_from_user)
+	.macro __BUILD_CSUM_PARTIAL_COPY_USER mode, from, to, __nocheck
+
 	PTR_ADDU	AT, src, len	/* See (1) above. */
+	/* initialize __nocheck if this the first time we execute this
+	 * macro
+	 */
 #ifdef CONFIG_64BIT
 	move	errptr, a4
 #else
 	lw	errptr, 16(sp)
 #endif
-FEXPORT(csum_partial_copy_nocheck)
+	.if \__nocheck == 1
+	FEXPORT(csum_partial_copy_nocheck)
+	.endif
 	move	sum, zero
 	move	odd, zero
 	/*
@@ -444,48 +454,48 @@ FEXPORT(csum_partial_copy_nocheck)
 	 */
 	sltu	t2, len, NBYTES
 	and	t1, dst, ADDRMASK
-	bnez	t2, .Lcopy_bytes_checklen
+	bnez	t2, .Lcopy_bytes_checklen\@
 	 and	t0, src, ADDRMASK
 	andi	odd, dst, 0x1			/* odd buffer? */
-	bnez	t1, .Ldst_unaligned
+	bnez	t1, .Ldst_unaligned\@
 	 nop
-	bnez	t0, .Lsrc_unaligned_dst_aligned
+	bnez	t0, .Lsrc_unaligned_dst_aligned\@
 	/*
 	 * use delay slot for fall-through
 	 * src and dst are aligned; need to compute rem
 	 */
-.Lboth_aligned:
+.Lboth_aligned\@:
 	 SRL	t0, len, LOG_NBYTES+3	 # +3 for 8 units/iter
-	beqz	t0, .Lcleanup_both_aligned # len < 8*NBYTES
+	beqz	t0, .Lcleanup_both_aligned\@ # len < 8*NBYTES
 	 nop
 	SUB	len, 8*NBYTES		# subtract here for bgez loop
 	.align	4
 1:
-	LOAD(t0, UNIT(0)(src), .Ll_exc)
-	LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
-	LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
-	LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
-	LOAD(t4, UNIT(4)(src), .Ll_exc_copy)
-	LOAD(t5, UNIT(5)(src), .Ll_exc_copy)
-	LOAD(t6, UNIT(6)(src), .Ll_exc_copy)
-	LOAD(t7, UNIT(7)(src), .Ll_exc_copy)
+	LOAD(t0, UNIT(0)(src), .Ll_exc\@)
+	LOAD(t1, UNIT(1)(src), .Ll_exc_copy\@)
+	LOAD(t2, UNIT(2)(src), .Ll_exc_copy\@)
+	LOAD(t3, UNIT(3)(src), .Ll_exc_copy\@)
+	LOAD(t4, UNIT(4)(src), .Ll_exc_copy\@)
+	LOAD(t5, UNIT(5)(src), .Ll_exc_copy\@)
+	LOAD(t6, UNIT(6)(src), .Ll_exc_copy\@)
+	LOAD(t7, UNIT(7)(src), .Ll_exc_copy\@)
 	SUB	len, len, 8*NBYTES
 	ADD	src, src, 8*NBYTES
-	STORE(t0, UNIT(0)(dst),	.Ls_exc)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc\@)
 	ADDC(sum, t0)
-	STORE(t1, UNIT(1)(dst),	.Ls_exc)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc\@)
 	ADDC(sum, t1)
-	STORE(t2, UNIT(2)(dst),	.Ls_exc)
+	STORE(t2, UNIT(2)(dst),	.Ls_exc\@)
 	ADDC(sum, t2)
-	STORE(t3, UNIT(3)(dst),	.Ls_exc)
+	STORE(t3, UNIT(3)(dst),	.Ls_exc\@)
 	ADDC(sum, t3)
-	STORE(t4, UNIT(4)(dst),	.Ls_exc)
+	STORE(t4, UNIT(4)(dst),	.Ls_exc\@)
 	ADDC(sum, t4)
-	STORE(t5, UNIT(5)(dst),	.Ls_exc)
+	STORE(t5, UNIT(5)(dst),	.Ls_exc\@)
 	ADDC(sum, t5)
-	STORE(t6, UNIT(6)(dst),	.Ls_exc)
+	STORE(t6, UNIT(6)(dst),	.Ls_exc\@)
 	ADDC(sum, t6)
-	STORE(t7, UNIT(7)(dst),	.Ls_exc)
+	STORE(t7, UNIT(7)(dst),	.Ls_exc\@)
 	ADDC(sum, t7)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 8*NBYTES
@@ -496,44 +506,44 @@ FEXPORT(csum_partial_copy_nocheck)
 	/*
 	 * len == the number of bytes left to copy < 8*NBYTES
 	 */
-.Lcleanup_both_aligned:
+.Lcleanup_both_aligned\@:
 #define rem t7
-	beqz	len, .Ldone
+	beqz	len, .Ldone\@
 	 sltu	t0, len, 4*NBYTES
-	bnez	t0, .Lless_than_4units
+	bnez	t0, .Lless_than_4units\@
 	 and	rem, len, (NBYTES-1)	# rem = len % NBYTES
 	/*
 	 * len >= 4*NBYTES
 	 */
-	LOAD(t0, UNIT(0)(src), .Ll_exc)
-	LOAD(t1, UNIT(1)(src), .Ll_exc_copy)
-	LOAD(t2, UNIT(2)(src), .Ll_exc_copy)
-	LOAD(t3, UNIT(3)(src), .Ll_exc_copy)
+	LOAD(t0, UNIT(0)(src), .Ll_exc\@)
+	LOAD(t1, UNIT(1)(src), .Ll_exc_copy\@)
+	LOAD(t2, UNIT(2)(src), .Ll_exc_copy\@)
+	LOAD(t3, UNIT(3)(src), .Ll_exc_copy\@)
 	SUB	len, len, 4*NBYTES
 	ADD	src, src, 4*NBYTES
-	STORE(t0, UNIT(0)(dst),	.Ls_exc)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc\@)
 	ADDC(sum, t0)
-	STORE(t1, UNIT(1)(dst),	.Ls_exc)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc\@)
 	ADDC(sum, t1)
-	STORE(t2, UNIT(2)(dst),	.Ls_exc)
+	STORE(t2, UNIT(2)(dst),	.Ls_exc\@)
 	ADDC(sum, t2)
-	STORE(t3, UNIT(3)(dst),	.Ls_exc)
+	STORE(t3, UNIT(3)(dst),	.Ls_exc\@)
 	ADDC(sum, t3)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 4*NBYTES
-	beqz	len, .Ldone
+	beqz	len, .Ldone\@
 	.set	noreorder
-.Lless_than_4units:
+.Lless_than_4units\@:
 	/*
 	 * rem = len % NBYTES
 	 */
-	beq	rem, len, .Lcopy_bytes
+	beq	rem, len, .Lcopy_bytes\@
 	 nop
 1:
-	LOAD(t0, 0(src), .Ll_exc)
+	LOAD(t0, 0(src), .Ll_exc\@)
 	ADD	src, src, NBYTES
 	SUB	len, len, NBYTES
-	STORE(t0, 0(dst), .Ls_exc)
+	STORE(t0, 0(dst), .Ls_exc\@)
 	ADDC(sum, t0)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, NBYTES
@@ -552,20 +562,20 @@ FEXPORT(csum_partial_copy_nocheck)
 	 * more instruction-level parallelism.
 	 */
 #define bits t2
-	beqz	len, .Ldone
+	beqz	len, .Ldone\@
 	 ADD	t1, dst, len	# t1 is just past last byte of dst
 	li	bits, 8*NBYTES
 	SLL	rem, len, 3	# rem = number of bits to keep
-	LOAD(t0, 0(src), .Ll_exc)
+	LOAD(t0, 0(src), .Ll_exc\@)
 	SUB	bits, bits, rem # bits = number of bits to discard
 	SHIFT_DISCARD t0, t0, bits
-	STREST(t0, -1(t1), .Ls_exc)
+	STREST(t0, -1(t1), .Ls_exc\@)
 	SHIFT_DISCARD_REVERT t0, t0, bits
 	.set reorder
 	ADDC(sum, t0)
-	b	.Ldone
+	b	.Ldone\@
 	.set noreorder
-.Ldst_unaligned:
+.Ldst_unaligned\@:
 	/*
 	 * dst is unaligned
 	 * t0 = src & ADDRMASK
@@ -576,25 +586,25 @@ FEXPORT(csum_partial_copy_nocheck)
 	 * Set match = (src and dst have same alignment)
 	 */
 #define match rem
-	LDFIRST(t3, FIRST(0)(src), .Ll_exc)
+	LDFIRST(t3, FIRST(0)(src), .Ll_exc\@)
 	ADD	t2, zero, NBYTES
-	LDREST(t3, REST(0)(src), .Ll_exc_copy)
+	LDREST(t3, REST(0)(src), .Ll_exc_copy\@)
 	SUB	t2, t2, t1	# t2 = number of bytes copied
 	xor	match, t0, t1
-	STFIRST(t3, FIRST(0)(dst), .Ls_exc)
+	STFIRST(t3, FIRST(0)(dst), .Ls_exc\@)
 	SLL	t4, t1, 3		# t4 = number of bits to discard
 	SHIFT_DISCARD t3, t3, t4
 	/* no SHIFT_DISCARD_REVERT to handle odd buffer properly */
 	ADDC(sum, t3)
-	beq	len, t2, .Ldone
+	beq	len, t2, .Ldone\@
 	 SUB	len, len, t2
 	ADD	dst, dst, t2
-	beqz	match, .Lboth_aligned
+	beqz	match, .Lboth_aligned\@
 	 ADD	src, src, t2
 
-.Lsrc_unaligned_dst_aligned:
+.Lsrc_unaligned_dst_aligned\@:
 	SRL	t0, len, LOG_NBYTES+2	 # +2 for 4 units/iter
-	beqz	t0, .Lcleanup_src_unaligned
+	beqz	t0, .Lcleanup_src_unaligned\@
 	 and	rem, len, (4*NBYTES-1)	 # rem = len % 4*NBYTES
 1:
 /*
@@ -603,53 +613,53 @@ FEXPORT(csum_partial_copy_nocheck)
  * It's OK to load FIRST(N+1) before REST(N) because the two addresses
  * are to the same unit (unless src is aligned, but it's not).
  */
-	LDFIRST(t0, FIRST(0)(src), .Ll_exc)
-	LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy)
+	LDFIRST(t0, FIRST(0)(src), .Ll_exc\@)
+	LDFIRST(t1, FIRST(1)(src), .Ll_exc_copy\@)
 	SUB	len, len, 4*NBYTES
-	LDREST(t0, REST(0)(src), .Ll_exc_copy)
-	LDREST(t1, REST(1)(src), .Ll_exc_copy)
-	LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy)
-	LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy)
-	LDREST(t2, REST(2)(src), .Ll_exc_copy)
-	LDREST(t3, REST(3)(src), .Ll_exc_copy)
+	LDREST(t0, REST(0)(src), .Ll_exc_copy\@)
+	LDREST(t1, REST(1)(src), .Ll_exc_copy\@)
+	LDFIRST(t2, FIRST(2)(src), .Ll_exc_copy\@)
+	LDFIRST(t3, FIRST(3)(src), .Ll_exc_copy\@)
+	LDREST(t2, REST(2)(src), .Ll_exc_copy\@)
+	LDREST(t3, REST(3)(src), .Ll_exc_copy\@)
 	ADD	src, src, 4*NBYTES
 #ifdef CONFIG_CPU_SB1
 	nop				# improves slotting
 #endif
-	STORE(t0, UNIT(0)(dst),	.Ls_exc)
+	STORE(t0, UNIT(0)(dst),	.Ls_exc\@)
 	ADDC(sum, t0)
-	STORE(t1, UNIT(1)(dst),	.Ls_exc)
+	STORE(t1, UNIT(1)(dst),	.Ls_exc\@)
 	ADDC(sum, t1)
-	STORE(t2, UNIT(2)(dst),	.Ls_exc)
+	STORE(t2, UNIT(2)(dst),	.Ls_exc\@)
 	ADDC(sum, t2)
-	STORE(t3, UNIT(3)(dst),	.Ls_exc)
+	STORE(t3, UNIT(3)(dst),	.Ls_exc\@)
 	ADDC(sum, t3)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, 4*NBYTES
 	bne	len, rem, 1b
 	.set	noreorder
 
-.Lcleanup_src_unaligned:
-	beqz	len, .Ldone
+.Lcleanup_src_unaligned\@:
+	beqz	len, .Ldone\@
 	 and	rem, len, NBYTES-1  # rem = len % NBYTES
-	beq	rem, len, .Lcopy_bytes
+	beq	rem, len, .Lcopy_bytes\@
 	 nop
 1:
-	LDFIRST(t0, FIRST(0)(src), .Ll_exc)
-	LDREST(t0, REST(0)(src), .Ll_exc_copy)
+	LDFIRST(t0, FIRST(0)(src), .Ll_exc\@)
+	LDREST(t0, REST(0)(src), .Ll_exc_copy\@)
 	ADD	src, src, NBYTES
 	SUB	len, len, NBYTES
-	STORE(t0, 0(dst), .Ls_exc)
+	STORE(t0, 0(dst), .Ls_exc\@)
 	ADDC(sum, t0)
 	.set	reorder				/* DADDI_WAR */
 	ADD	dst, dst, NBYTES
 	bne	len, rem, 1b
 	.set	noreorder
 
-.Lcopy_bytes_checklen:
-	beqz	len, .Ldone
+.Lcopy_bytes_checklen\@:
+	beqz	len, .Ldone\@
 	 nop
-.Lcopy_bytes:
+.Lcopy_bytes\@:
 	/* 0 < len < NBYTES  */
 #ifdef CONFIG_CPU_LITTLE_ENDIAN
 #define SHIFT_START 0
@@ -662,12 +672,12 @@ FEXPORT(csum_partial_copy_nocheck)
 	li	t3, SHIFT_START # shift
 /* use .Ll_exc_copy here to return correct sum on fault */
 #define COPY_BYTE(N)			\
-	LOADBU(t0, N(src), .Ll_exc_copy);	\
+	LOADBU(t0, N(src), .Ll_exc_copy\@);	\
 	SUB	len, len, 1;		\
-	STOREB(t0, N(dst), .Ls_exc);	\
+	STOREB(t0, N(dst), .Ls_exc\@);	\
 	SLLV	t0, t0, t3;		\
 	addu	t3, SHIFT_INC;		\
-	beqz	len, .Lcopy_bytes_done; \
+	beqz	len, .Lcopy_bytes_done\@; \
 	 or	t2, t0
 
 	COPY_BYTE(0)
@@ -678,14 +688,14 @@ FEXPORT(csum_partial_copy_nocheck)
 	COPY_BYTE(4)
 	COPY_BYTE(5)
 #endif
-	LOADBU(t0, NBYTES-2(src), .Ll_exc_copy)
+	LOADBU(t0, NBYTES-2(src), .Ll_exc_copy\@)
 	SUB	len, len, 1
-	STOREB(t0, NBYTES-2(dst), .Ls_exc)
+	STOREB(t0, NBYTES-2(dst), .Ls_exc\@)
 	SLLV	t0, t0, t3
 	or	t2, t0
-.Lcopy_bytes_done:
+.Lcopy_bytes_done\@:
 	ADDC(sum, t2)
-.Ldone:
+.Ldone\@:
 	/* fold checksum */
 #ifdef USE_DOUBLE
 	dsll32	v1, sum, 0
@@ -714,7 +724,7 @@ FEXPORT(csum_partial_copy_nocheck)
 	jr	ra
 	.set noreorder
 
-.Ll_exc_copy:
+.Ll_exc_copy\@:
 	/*
 	 * Copy bytes from src until faulting load address (or until a
 	 * lb faults)
@@ -729,7 +739,7 @@ FEXPORT(csum_partial_copy_nocheck)
 	 li	t2, SHIFT_START
 	LOADK	t0, THREAD_BUADDR(t0)
 1:
-	LOADBU(t1, 0(src), .Ll_exc)
+	LOADBU(t1, 0(src), .Ll_exc\@)
 	ADD	src, src, 1
 	sb	t1, 0(dst)	# can't fault -- we're copy_from_user
 	SLLV	t1, t1, t2
@@ -739,7 +749,7 @@ FEXPORT(csum_partial_copy_nocheck)
 	ADD	dst, dst, 1
 	bne	src, t0, 1b
 	.set	noreorder
-.Ll_exc:
+.Ll_exc\@:
 	LOADK	t0, TI_TASK($28)
 	 nop
 	LOADK	t0, THREAD_BUADDR(t0)	# t0 is just past last good address
@@ -758,7 +768,7 @@ FEXPORT(csum_partial_copy_nocheck)
 	 */
 	.set	reorder				/* DADDI_WAR */
 	SUB	src, len, 1
-	beqz	len, .Ldone
+	beqz	len, .Ldone\@
 	.set	noreorder
 1:	sb	zero, 0(dst)
 	ADD	dst, dst, 1
@@ -773,13 +783,19 @@ FEXPORT(csum_partial_copy_nocheck)
 	 SUB	src, src, v1
 #endif
 	li	v1, -EFAULT
-	b	.Ldone
+	b	.Ldone\@
 	 sw	v1, (errptr)
 
-.Ls_exc:
+.Ls_exc\@:
 	li	v0, -1 /* invalid checksum */
 	li	v1, -EFAULT
 	jr	ra
 	 sw	v1, (errptr)
 	.set	pop
-	END(__csum_partial_copy_kernel)
+	.endm
+
+LEAF(__csum_partial_copy_kernel)
+FEXPORT(__csum_partial_copy_to_user)
+FEXPORT(__csum_partial_copy_from_user)
+__BUILD_CSUM_PARTIAL_COPY_USER LEGACY_MODE USEROP USEROP 1
+END(__csum_partial_copy_kernel)
-- 
1.8.5.3

[PATCH 35/58] MIPS: lib: csum_partial: Add EVA support

[Markos Chandras]

Use EVA specific functions to read and write data to
user address space.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/csum_partial.S | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/arch/mips/lib/csum_partial.S b/arch/mips/lib/csum_partial.S
index 62c8768..2e4825e 100644
--- a/arch/mips/lib/csum_partial.S
+++ b/arch/mips/lib/csum_partial.S
@@ -352,6 +352,19 @@ LEAF(csum_partial)
 		.section __ex_table,"a";	\
 		PTR	9b, handler;		\
 		.previous;			\
+	/* This is enabled in EVA mode */	\
+	.else;					\
+		/* If loading from user or storing to user */	\
+		.if ((\from == USEROP) && (type == LD_INSN)) || \
+		    ((\to == USEROP) && (type == ST_INSN));	\
+9:			__BUILD_EVA_INSN(insn##e, reg, addr);	\
+			.section __ex_table,"a";		\
+			PTR	9b, handler;			\
+			.previous;				\
+		.else;						\
+			/* EVA without exception */		\
+			insn reg, addr;				\
+		.endif;						\
 	.endif
 
 #undef LOAD
@@ -795,7 +808,19 @@ LEAF(csum_partial)
 	.endm
 
 LEAF(__csum_partial_copy_kernel)
+#ifndef CONFIG_EVA
 FEXPORT(__csum_partial_copy_to_user)
 FEXPORT(__csum_partial_copy_from_user)
+#endif
 __BUILD_CSUM_PARTIAL_COPY_USER LEGACY_MODE USEROP USEROP 1
 END(__csum_partial_copy_kernel)
+
+#ifdef CONFIG_EVA
+LEAF(__csum_partial_copy_to_user)
+__BUILD_CSUM_PARTIAL_COPY_USER EVA_MODE KERNELOP USEROP 0
+END(__csum_partial_copy_to_user)
+
+LEAF(__csum_partial_copy_from_user)
+__BUILD_CSUM_PARTIAL_COPY_USER EVA_MODE USEROP KERNELOP 0
+END(__csum_partial_copy_from_user)
+#endif
-- 
1.8.5.3

[PATCH 35/58] MIPS: lib: csum_partial: Add EVA support

[Markos Chandras]

Use EVA specific functions to read and write data to
user address space.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/lib/csum_partial.S | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/arch/mips/lib/csum_partial.S b/arch/mips/lib/csum_partial.S
index 62c8768..2e4825e 100644
--- a/arch/mips/lib/csum_partial.S
+++ b/arch/mips/lib/csum_partial.S
@@ -352,6 +352,19 @@ LEAF(csum_partial)
 		.section __ex_table,"a";	\
 		PTR	9b, handler;		\
 		.previous;			\
+	/* This is enabled in EVA mode */	\
+	.else;					\
+		/* If loading from user or storing to user */	\
+		.if ((\from == USEROP) && (type == LD_INSN)) || \
+		    ((\to == USEROP) && (type == ST_INSN));	\
+9:			__BUILD_EVA_INSN(insn##e, reg, addr);	\
+			.section __ex_table,"a";		\
+			PTR	9b, handler;			\
+			.previous;				\
+		.else;						\
+			/* EVA without exception */		\
+			insn reg, addr;				\
+		.endif;						\
 	.endif
 
 #undef LOAD
@@ -795,7 +808,19 @@ LEAF(csum_partial)
 	.endm
 
 LEAF(__csum_partial_copy_kernel)
+#ifndef CONFIG_EVA
 FEXPORT(__csum_partial_copy_to_user)
 FEXPORT(__csum_partial_copy_from_user)
+#endif
 __BUILD_CSUM_PARTIAL_COPY_USER LEGACY_MODE USEROP USEROP 1
 END(__csum_partial_copy_kernel)
+
+#ifdef CONFIG_EVA
+LEAF(__csum_partial_copy_to_user)
+__BUILD_CSUM_PARTIAL_COPY_USER EVA_MODE KERNELOP USEROP 0
+END(__csum_partial_copy_to_user)
+
+LEAF(__csum_partial_copy_from_user)
+__BUILD_CSUM_PARTIAL_COPY_USER EVA_MODE USEROP KERNELOP 0
+END(__csum_partial_copy_from_user)
+#endif
-- 
1.8.5.3

[PATCH 36/58] MIPS: asm: checksum: Split kernel and user copy operations

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

In EVA mode, different instructions need to be used to read/write
from kernel and userland. In non-EVA mode, there is no functional
difference. The current address limit is checked to decide the
type of operation that will be performed.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/checksum.h | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)

diff --git a/arch/mips/include/asm/checksum.h b/arch/mips/include/asm/checksum.h
index 3c9aea5..f5602c6 100644
--- a/arch/mips/include/asm/checksum.h
+++ b/arch/mips/include/asm/checksum.h
@@ -37,7 +37,6 @@ __wsum __csum_partial_copy_from_user(const void *src, void *dst,
 				     int len, __wsum sum, int *err_ptr);
 __wsum __csum_partial_copy_to_user(const void *src, void *dst,
 				   int len, __wsum sum, int *err_ptr);
-
 /*
  * this is a new version of the above that records errors it finds in *errp,
  * but continues and zeros the rest of the buffer.
@@ -47,8 +46,12 @@ __wsum csum_partial_copy_from_user(const void __user *src, void *dst, int len,
 				   __wsum sum, int *err_ptr)
 {
 	might_fault();
-	return __csum_partial_copy_from_user((__force void *)src, dst,
-					     len, sum, err_ptr);
+	if (segment_eq(get_fs(), get_ds()))
+		return __csum_partial_copy_kernel((__force void *)src, dst,
+						  len, sum, err_ptr);
+	else
+		return __csum_partial_copy_from_user((__force void *)src, dst,
+						     len, sum, err_ptr);
 }
 
 /*
@@ -60,9 +63,16 @@ __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len,
 			     __wsum sum, int *err_ptr)
 {
 	might_fault();
-	if (access_ok(VERIFY_WRITE, dst, len))
-		return __csum_partial_copy_to_user(src, (__force void *)dst,
-						   len, sum, err_ptr);
+	if (access_ok(VERIFY_WRITE, dst, len)) {
+		if (segment_eq(get_fs(), get_ds()))
+			return __csum_partial_copy_kernel(src,
+							  (__force void *)dst,
+							  len, sum, err_ptr);
+		else
+			return __csum_partial_copy_to_user(src,
+							   (__force void *)dst,
+							   len, sum, err_ptr);
+	}
 	if (len)
 		*err_ptr = -EFAULT;
 
-- 
1.8.5.3

[PATCH 36/58] MIPS: asm: checksum: Split kernel and user copy operations

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

In EVA mode, different instructions need to be used to read/write
from kernel and userland. In non-EVA mode, there is no functional
difference. The current address limit is checked to decide the
type of operation that will be performed.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/checksum.h | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)

diff --git a/arch/mips/include/asm/checksum.h b/arch/mips/include/asm/checksum.h
index 3c9aea5..f5602c6 100644
--- a/arch/mips/include/asm/checksum.h
+++ b/arch/mips/include/asm/checksum.h
@@ -37,7 +37,6 @@ __wsum __csum_partial_copy_from_user(const void *src, void *dst,
 				     int len, __wsum sum, int *err_ptr);
 __wsum __csum_partial_copy_to_user(const void *src, void *dst,
 				   int len, __wsum sum, int *err_ptr);
-
 /*
  * this is a new version of the above that records errors it finds in *errp,
  * but continues and zeros the rest of the buffer.
@@ -47,8 +46,12 @@ __wsum csum_partial_copy_from_user(const void __user *src, void *dst, int len,
 				   __wsum sum, int *err_ptr)
 {
 	might_fault();
-	return __csum_partial_copy_from_user((__force void *)src, dst,
-					     len, sum, err_ptr);
+	if (segment_eq(get_fs(), get_ds()))
+		return __csum_partial_copy_kernel((__force void *)src, dst,
+						  len, sum, err_ptr);
+	else
+		return __csum_partial_copy_from_user((__force void *)src, dst,
+						     len, sum, err_ptr);
 }
 
 /*
@@ -60,9 +63,16 @@ __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len,
 			     __wsum sum, int *err_ptr)
 {
 	might_fault();
-	if (access_ok(VERIFY_WRITE, dst, len))
-		return __csum_partial_copy_to_user(src, (__force void *)dst,
-						   len, sum, err_ptr);
+	if (access_ok(VERIFY_WRITE, dst, len)) {
+		if (segment_eq(get_fs(), get_ds()))
+			return __csum_partial_copy_kernel(src,
+							  (__force void *)dst,
+							  len, sum, err_ptr);
+		else
+			return __csum_partial_copy_to_user(src,
+							   (__force void *)dst,
+							   len, sum, err_ptr);
+	}
 	if (len)
 		*err_ptr = -EFAULT;
 
-- 
1.8.5.3

[PATCH 37/58] MIPS: asm: checksum: Add MIPS specific csum_and_copy_from_user function

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

A MIPS specific csum_and_copy_from_user function is necessary because
the generic one from include/net/checksum.h will not work for EVA.
This is because the generic one will link to symbols from lib/checksum.c
which are not EVA aware.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/checksum.h | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/arch/mips/include/asm/checksum.h b/arch/mips/include/asm/checksum.h
index f5602c6..3418c51 100644
--- a/arch/mips/include/asm/checksum.h
+++ b/arch/mips/include/asm/checksum.h
@@ -54,6 +54,20 @@ __wsum csum_partial_copy_from_user(const void __user *src, void *dst, int len,
 						     len, sum, err_ptr);
 }
 
+#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER
+static inline
+__wsum csum_and_copy_from_user(const void __user *src, void *dst,
+			       int len, __wsum sum, int *err_ptr)
+{
+	if (access_ok(VERIFY_READ, src, len))
+		return csum_partial_copy_from_user(src, dst, len, sum,
+						   err_ptr);
+	if (len)
+		*err_ptr = -EFAULT;
+
+	return sum;
+}
+
 /*
  * Copy and checksum to user
  */
-- 
1.8.5.3

[PATCH 37/58] MIPS: asm: checksum: Add MIPS specific csum_and_copy_from_user function

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

A MIPS specific csum_and_copy_from_user function is necessary because
the generic one from include/net/checksum.h will not work for EVA.
This is because the generic one will link to symbols from lib/checksum.c
which are not EVA aware.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/checksum.h | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/arch/mips/include/asm/checksum.h b/arch/mips/include/asm/checksum.h
index f5602c6..3418c51 100644
--- a/arch/mips/include/asm/checksum.h
+++ b/arch/mips/include/asm/checksum.h
@@ -54,6 +54,20 @@ __wsum csum_partial_copy_from_user(const void __user *src, void *dst, int len,
 						     len, sum, err_ptr);
 }
 
+#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER
+static inline
+__wsum csum_and_copy_from_user(const void __user *src, void *dst,
+			       int len, __wsum sum, int *err_ptr)
+{
+	if (access_ok(VERIFY_READ, src, len))
+		return csum_partial_copy_from_user(src, dst, len, sum,
+						   err_ptr);
+	if (len)
+		*err_ptr = -EFAULT;
+
+	return sum;
+}
+
 /*
  * Copy and checksum to user
  */
-- 
1.8.5.3

[PATCH 38/58] MIPS: kernel: signal: Prevent save/restore FPU context in user memory

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

EVA does not have FPU specific instructions for reading or writing
FPU registers from userspace memory.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/signal.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/arch/mips/kernel/signal.c b/arch/mips/kernel/signal.c
index 5199563..6f4937b 100644
--- a/arch/mips/kernel/signal.c
+++ b/arch/mips/kernel/signal.c
@@ -6,6 +6,7 @@
  * Copyright (C) 1991, 1992  Linus Torvalds
  * Copyright (C) 1994 - 2000  Ralf Baechle
  * Copyright (C) 1999, 2000 Silicon Graphics, Inc.
+ * Copyright (C) 2014, Imagination Technologies Ltd.
  */
 #include <linux/cache.h>
 #include <linux/context_tracking.h>
@@ -69,6 +70,7 @@ struct rt_sigframe {
 static int protected_save_fp_context(struct sigcontext __user *sc)
 {
 	int err;
+#ifndef CONFIG_EVA
 	while (1) {
 		lock_fpu_owner();
 		err = own_fpu_inatomic(1);
@@ -84,12 +86,21 @@ static int protected_save_fp_context(struct sigcontext __user *sc)
 		if (err)
 			break;	/* really bad sigcontext */
 	}
+#else
+	/*
+	 * EVA does not have FPU EVA instructions so saving fpu context directly
+	 * does not work.
+	 */
+	lose_fpu(1);
+	err = save_fp_context(sc); /* this might fail */
+#endif
 	return err;
 }
 
 static int protected_restore_fp_context(struct sigcontext __user *sc)
 {
 	int err, tmp __maybe_unused;
+#ifndef CONFIG_EVA
 	while (1) {
 		lock_fpu_owner();
 		err = own_fpu_inatomic(0);
@@ -105,6 +116,14 @@ static int protected_restore_fp_context(struct sigcontext __user *sc)
 		if (err)
 			break;	/* really bad sigcontext */
 	}
+#else
+	/*
+	 * EVA does not have FPU EVA instructions so restoring fpu context
+	 * directly does not work.
+	 */
+	lose_fpu(0);
+	err = restore_fp_context(sc); /* this might fail */
+#endif
 	return err;
 }
 
@@ -591,6 +610,7 @@ asmlinkage void do_notify_resume(struct pt_regs *regs, void *unused,
 }
 
 #ifdef CONFIG_SMP
+#ifndef CONFIG_EVA
 static int smp_save_fp_context(struct sigcontext __user *sc)
 {
 	return raw_cpu_has_fpu
@@ -604,10 +624,12 @@ static int smp_restore_fp_context(struct sigcontext __user *sc)
 	       ? _restore_fp_context(sc)
 	       : fpu_emulator_restore_context(sc);
 }
+#endif /* CONFIG_EVA */
 #endif
 
 static int signal_setup(void)
 {
+#ifndef CONFIG_EVA
 #ifdef CONFIG_SMP
 	/* For now just do the cpu_has_fpu check when the functions are invoked */
 	save_fp_context = smp_save_fp_context;
@@ -620,6 +642,10 @@ static int signal_setup(void)
 		save_fp_context = fpu_emulator_save_context;
 		restore_fp_context = fpu_emulator_restore_context;
 	}
+#endif /* CONFIG_SMP */
+#else
+	save_fp_context = fpu_emulator_save_context;
+	restore_fp_context = fpu_emulator_restore_context;
 #endif
 
 	return 0;
-- 
1.8.5.3

[PATCH 38/58] MIPS: kernel: signal: Prevent save/restore FPU context in user memory

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

EVA does not have FPU specific instructions for reading or writing
FPU registers from userspace memory.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/signal.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/arch/mips/kernel/signal.c b/arch/mips/kernel/signal.c
index 5199563..6f4937b 100644
--- a/arch/mips/kernel/signal.c
+++ b/arch/mips/kernel/signal.c
@@ -6,6 +6,7 @@
  * Copyright (C) 1991, 1992  Linus Torvalds
  * Copyright (C) 1994 - 2000  Ralf Baechle
  * Copyright (C) 1999, 2000 Silicon Graphics, Inc.
+ * Copyright (C) 2014, Imagination Technologies Ltd.
  */
 #include <linux/cache.h>
 #include <linux/context_tracking.h>
@@ -69,6 +70,7 @@ struct rt_sigframe {
 static int protected_save_fp_context(struct sigcontext __user *sc)
 {
 	int err;
+#ifndef CONFIG_EVA
 	while (1) {
 		lock_fpu_owner();
 		err = own_fpu_inatomic(1);
@@ -84,12 +86,21 @@ static int protected_save_fp_context(struct sigcontext __user *sc)
 		if (err)
 			break;	/* really bad sigcontext */
 	}
+#else
+	/*
+	 * EVA does not have FPU EVA instructions so saving fpu context directly
+	 * does not work.
+	 */
+	lose_fpu(1);
+	err = save_fp_context(sc); /* this might fail */
+#endif
 	return err;
 }
 
 static int protected_restore_fp_context(struct sigcontext __user *sc)
 {
 	int err, tmp __maybe_unused;
+#ifndef CONFIG_EVA
 	while (1) {
 		lock_fpu_owner();
 		err = own_fpu_inatomic(0);
@@ -105,6 +116,14 @@ static int protected_restore_fp_context(struct sigcontext __user *sc)
 		if (err)
 			break;	/* really bad sigcontext */
 	}
+#else
+	/*
+	 * EVA does not have FPU EVA instructions so restoring fpu context
+	 * directly does not work.
+	 */
+	lose_fpu(0);
+	err = restore_fp_context(sc); /* this might fail */
+#endif
 	return err;
 }
 
@@ -591,6 +610,7 @@ asmlinkage void do_notify_resume(struct pt_regs *regs, void *unused,
 }
 
 #ifdef CONFIG_SMP
+#ifndef CONFIG_EVA
 static int smp_save_fp_context(struct sigcontext __user *sc)
 {
 	return raw_cpu_has_fpu
@@ -604,10 +624,12 @@ static int smp_restore_fp_context(struct sigcontext __user *sc)
 	       ? _restore_fp_context(sc)
 	       : fpu_emulator_restore_context(sc);
 }
+#endif /* CONFIG_EVA */
 #endif
 
 static int signal_setup(void)
 {
+#ifndef CONFIG_EVA
 #ifdef CONFIG_SMP
 	/* For now just do the cpu_has_fpu check when the functions are invoked */
 	save_fp_context = smp_save_fp_context;
@@ -620,6 +642,10 @@ static int signal_setup(void)
 		save_fp_context = fpu_emulator_save_context;
 		restore_fp_context = fpu_emulator_restore_context;
 	}
+#endif /* CONFIG_SMP */
+#else
+	save_fp_context = fpu_emulator_save_context;
+	restore_fp_context = fpu_emulator_restore_context;
 #endif
 
 	return 0;
-- 
1.8.5.3

[PATCH 39/58] MIPS: asm: r4kcache: Build flushing code for instruction cache

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

Build code to invalidate an address range in the  instruction cache
using the Hit Invalidate cache operation.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/r4kcache.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/mips/include/asm/r4kcache.h b/arch/mips/include/asm/r4kcache.h
index 34d1a19..a89e86d 100644
--- a/arch/mips/include/asm/r4kcache.h
+++ b/arch/mips/include/asm/r4kcache.h
@@ -455,6 +455,7 @@ __BUILD_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I, protected_, )
 __BUILD_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I_Loongson23, \
 	protected_, loongson23_)
 __BUILD_BLAST_CACHE_RANGE(d, dcache, Hit_Writeback_Inv_D, , )
+__BUILD_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I, , )
 __BUILD_BLAST_CACHE_RANGE(s, scache, Hit_Writeback_Inv_SD, , )
 /* blast_inv_dcache_range */
 __BUILD_BLAST_CACHE_RANGE(inv_d, dcache, Hit_Invalidate_D, , )
-- 
1.8.5.3

[PATCH 39/58] MIPS: asm: r4kcache: Build flushing code for instruction cache

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

Build code to invalidate an address range in the  instruction cache
using the Hit Invalidate cache operation.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/r4kcache.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/mips/include/asm/r4kcache.h b/arch/mips/include/asm/r4kcache.h
index 34d1a19..a89e86d 100644
--- a/arch/mips/include/asm/r4kcache.h
+++ b/arch/mips/include/asm/r4kcache.h
@@ -455,6 +455,7 @@ __BUILD_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I, protected_, )
 __BUILD_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I_Loongson23, \
 	protected_, loongson23_)
 __BUILD_BLAST_CACHE_RANGE(d, dcache, Hit_Writeback_Inv_D, , )
+__BUILD_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I, , )
 __BUILD_BLAST_CACHE_RANGE(s, scache, Hit_Writeback_Inv_SD, , )
 /* blast_inv_dcache_range */
 __BUILD_BLAST_CACHE_RANGE(inv_d, dcache, Hit_Invalidate_D, , )
-- 
1.8.5.3

[PATCH 40/58] MIPS: asm: r4kcache: Add protected cache operation for EVA

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/r4kcache.h | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/arch/mips/include/asm/r4kcache.h b/arch/mips/include/asm/r4kcache.h
index a89e86d..ae026bf 100644
--- a/arch/mips/include/asm/r4kcache.h
+++ b/arch/mips/include/asm/r4kcache.h
@@ -212,6 +212,20 @@ static inline void flush_scache_line(unsigned long addr)
 	:							\
 	: "i" (op), "r" (addr))
 
+#define protected_cachee_op(op,addr)				\
+	__asm__ __volatile__(					\
+	"	.set	push			\n"		\
+	"	.set	noreorder		\n"		\
+	"	.set	mips0			\n"		\
+	"	.set	eva			\n"		\
+	"1:	cachee	%0, (%1)		\n"		\
+	"2:	.set	pop			\n"		\
+	"	.section __ex_table,\"a\"	\n"		\
+	"	"STR(PTR)" 1b, 2b		\n"		\
+	"	.previous"					\
+	:							\
+	: "i" (op), "r" (addr))
+
 /*
  * The next two are for badland addresses like signal trampolines.
  */
@@ -223,7 +237,11 @@ static inline void protected_flush_icache_line(unsigned long addr)
 		break;
 
 	default:
+#ifdef CONFIG_EVA
+		protected_cachee_op(Hit_Invalidate_I, addr);
+#else
 		protected_cache_op(Hit_Invalidate_I, addr);
+#endif
 		break;
 	}
 }
-- 
1.8.5.3

[PATCH 40/58] MIPS: asm: r4kcache: Add protected cache operation for EVA

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/r4kcache.h | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/arch/mips/include/asm/r4kcache.h b/arch/mips/include/asm/r4kcache.h
index a89e86d..ae026bf 100644
--- a/arch/mips/include/asm/r4kcache.h
+++ b/arch/mips/include/asm/r4kcache.h
@@ -212,6 +212,20 @@ static inline void flush_scache_line(unsigned long addr)
 	:							\
 	: "i" (op), "r" (addr))
 
+#define protected_cachee_op(op,addr)				\
+	__asm__ __volatile__(					\
+	"	.set	push			\n"		\
+	"	.set	noreorder		\n"		\
+	"	.set	mips0			\n"		\
+	"	.set	eva			\n"		\
+	"1:	cachee	%0, (%1)		\n"		\
+	"2:	.set	pop			\n"		\
+	"	.section __ex_table,\"a\"	\n"		\
+	"	"STR(PTR)" 1b, 2b		\n"		\
+	"	.previous"					\
+	:							\
+	: "i" (op), "r" (addr))
+
 /*
  * The next two are for badland addresses like signal trampolines.
  */
@@ -223,7 +237,11 @@ static inline void protected_flush_icache_line(unsigned long addr)
 		break;
 
 	default:
+#ifdef CONFIG_EVA
+		protected_cachee_op(Hit_Invalidate_I, addr);
+#else
 		protected_cache_op(Hit_Invalidate_I, addr);
+#endif
 		break;
 	}
 }
-- 
1.8.5.3

[PATCH 41/58] MIPS: asm: r4kcache: Add EVA cache flushing functions

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

Add EVA cache flushing functions similar to non-EVA configurations.
Because the cache may or may not contain user virtual addresses, we
need to use the 'cache' or 'cachee' instruction based on whether we
flush the cache on behalf of kernel or user respectively.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/r4kcache.h | 154 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 152 insertions(+), 2 deletions(-)

diff --git a/arch/mips/include/asm/r4kcache.h b/arch/mips/include/asm/r4kcache.h
index ae026bf..b973e02 100644
--- a/arch/mips/include/asm/r4kcache.h
+++ b/arch/mips/include/asm/r4kcache.h
@@ -17,6 +17,7 @@
 #include <asm/cpu-features.h>
 #include <asm/cpu-type.h>
 #include <asm/mipsmtregs.h>
+#include <asm/uaccess.h> /* for segment_eq() */
 
 /*
  * This macro return a properly sign-extended address suitable as base address
@@ -374,6 +375,91 @@ static inline void invalidate_tcache_page(unsigned long addr)
 		: "r" (base),						\
 		  "i" (op));
 
+/*
+ * Perform the cache operation specified by op using a user mode virtual
+ * address while in kernel mode.
+ */
+#define cache16_unroll32_user(base,op)					\
+	__asm__ __volatile__(						\
+	"	.set push					\n"	\
+	"	.set noreorder					\n"	\
+	"	.set mips0					\n"	\
+	"	.set eva					\n"	\
+	"	cachee %1, 0x000(%0); cachee %1, 0x010(%0)	\n"	\
+	"	cachee %1, 0x020(%0); cachee %1, 0x030(%0)	\n"	\
+	"	cachee %1, 0x040(%0); cachee %1, 0x050(%0)	\n"	\
+	"	cachee %1, 0x060(%0); cachee %1, 0x070(%0)	\n"	\
+	"	cachee %1, 0x080(%0); cachee %1, 0x090(%0)	\n"	\
+	"	cachee %1, 0x0a0(%0); cachee %1, 0x0b0(%0)	\n"	\
+	"	cachee %1, 0x0c0(%0); cachee %1, 0x0d0(%0)	\n"	\
+	"	cachee %1, 0x0e0(%0); cachee %1, 0x0f0(%0)	\n"	\
+	"	cachee %1, 0x100(%0); cachee %1, 0x110(%0)	\n"	\
+	"	cachee %1, 0x120(%0); cachee %1, 0x130(%0)	\n"	\
+	"	cachee %1, 0x140(%0); cachee %1, 0x150(%0)	\n"	\
+	"	cachee %1, 0x160(%0); cachee %1, 0x170(%0)	\n"	\
+	"	cachee %1, 0x180(%0); cachee %1, 0x190(%0)	\n"	\
+	"	cachee %1, 0x1a0(%0); cachee %1, 0x1b0(%0)	\n"	\
+	"	cachee %1, 0x1c0(%0); cachee %1, 0x1d0(%0)	\n"	\
+	"	cachee %1, 0x1e0(%0); cachee %1, 0x1f0(%0)	\n"	\
+	"	.set pop					\n"	\
+		:							\
+		: "r" (base),						\
+		  "i" (op));
+
+#define cache32_unroll32_user(base, op)					\
+	__asm__ __volatile__(						\
+	"	.set push					\n"	\
+	"	.set noreorder					\n"	\
+	"	.set mips0					\n"	\
+	"	.set eva					\n"	\
+	"	cachee %1, 0x000(%0); cachee %1, 0x020(%0)	\n"	\
+	"	cachee %1, 0x040(%0); cachee %1, 0x060(%0)	\n"	\
+	"	cachee %1, 0x080(%0); cachee %1, 0x0a0(%0)	\n"	\
+	"	cachee %1, 0x0c0(%0); cachee %1, 0x0e0(%0)	\n"	\
+	"	cachee %1, 0x100(%0); cachee %1, 0x120(%0)	\n"	\
+	"	cachee %1, 0x140(%0); cachee %1, 0x160(%0)	\n"	\
+	"	cachee %1, 0x180(%0); cachee %1, 0x1a0(%0)	\n"	\
+	"	cachee %1, 0x1c0(%0); cachee %1, 0x1e0(%0)	\n"	\
+	"	cachee %1, 0x200(%0); cachee %1, 0x220(%0)	\n"	\
+	"	cachee %1, 0x240(%0); cachee %1, 0x260(%0)	\n"	\
+	"	cachee %1, 0x280(%0); cachee %1, 0x2a0(%0)	\n"	\
+	"	cachee %1, 0x2c0(%0); cachee %1, 0x2e0(%0)	\n"	\
+	"	cachee %1, 0x300(%0); cachee %1, 0x320(%0)	\n"	\
+	"	cachee %1, 0x340(%0); cachee %1, 0x360(%0)	\n"	\
+	"	cachee %1, 0x380(%0); cachee %1, 0x3a0(%0)	\n"	\
+	"	cachee %1, 0x3c0(%0); cachee %1, 0x3e0(%0)	\n"	\
+	"	.set pop					\n"	\
+		:							\
+		: "r" (base),						\
+		  "i" (op));
+
+#define cache64_unroll32_user(base, op)					\
+	__asm__ __volatile__(						\
+	"	.set push					\n"	\
+	"	.set noreorder					\n"	\
+	"	.set mips0					\n"	\
+	"	.set eva					\n"	\
+	"	cachee %1, 0x000(%0); cachee %1, 0x040(%0)	\n"	\
+	"	cachee %1, 0x080(%0); cachee %1, 0x0c0(%0)	\n"	\
+	"	cachee %1, 0x100(%0); cachee %1, 0x140(%0)	\n"	\
+	"	cachee %1, 0x180(%0); cachee %1, 0x1c0(%0)	\n"	\
+	"	cachee %1, 0x200(%0); cachee %1, 0x240(%0)	\n"	\
+	"	cachee %1, 0x280(%0); cachee %1, 0x2c0(%0)	\n"	\
+	"	cachee %1, 0x300(%0); cachee %1, 0x340(%0)	\n"	\
+	"	cachee %1, 0x380(%0); cachee %1, 0x3c0(%0)	\n"	\
+	"	cachee %1, 0x400(%0); cachee %1, 0x440(%0)	\n"	\
+	"	cachee %1, 0x480(%0); cachee %1, 0x4c0(%0)	\n"	\
+	"	cachee %1, 0x500(%0); cachee %1, 0x540(%0)	\n"	\
+	"	cachee %1, 0x580(%0); cachee %1, 0x5c0(%0)	\n"	\
+	"	cachee %1, 0x600(%0); cachee %1, 0x640(%0)	\n"	\
+	"	cachee %1, 0x680(%0); cachee %1, 0x6c0(%0)	\n"	\
+	"	cachee %1, 0x700(%0); cachee %1, 0x740(%0)	\n"	\
+	"	cachee %1, 0x780(%0); cachee %1, 0x7c0(%0)	\n"	\
+	"	.set pop					\n"	\
+		:							\
+		: "r" (base),						\
+		  "i" (op));
+
 /* build blast_xxx, blast_xxx_page, blast_xxx_page_indexed */
 #define __BUILD_BLAST_CACHE(pfx, desc, indexop, hitop, lsize) \
 static inline void blast_##pfx##cache##lsize(void)			\
@@ -446,9 +532,35 @@ __BUILD_BLAST_CACHE(inv_s, scache, Index_Writeback_Inv_SD, Hit_Invalidate_SD, 32
 __BUILD_BLAST_CACHE(inv_s, scache, Index_Writeback_Inv_SD, Hit_Invalidate_SD, 64)
 __BUILD_BLAST_CACHE(inv_s, scache, Index_Writeback_Inv_SD, Hit_Invalidate_SD, 128)
 
+#define __BUILD_BLAST_USER_CACHE(pfx, desc, indexop, hitop, lsize) \
+static inline void blast_##pfx##cache##lsize##_user_page(unsigned long page) \
+{									\
+	unsigned long start = page;					\
+	unsigned long end = page + PAGE_SIZE;				\
+									\
+	__##pfx##flush_prologue						\
+									\
+	do {								\
+		cache##lsize##_unroll32_user(start, hitop);             \
+		start += lsize * 32;					\
+	} while (start < end);						\
+									\
+	__##pfx##flush_epilogue						\
+}
+
+__BUILD_BLAST_USER_CACHE(d, dcache, Index_Writeback_Inv_D, Hit_Writeback_Inv_D,
+			 16)
+__BUILD_BLAST_USER_CACHE(i, icache, Index_Invalidate_I, Hit_Invalidate_I, 16)
+__BUILD_BLAST_USER_CACHE(d, dcache, Index_Writeback_Inv_D, Hit_Writeback_Inv_D,
+			 32)
+__BUILD_BLAST_USER_CACHE(i, icache, Index_Invalidate_I, Hit_Invalidate_I, 32)
+__BUILD_BLAST_USER_CACHE(d, dcache, Index_Writeback_Inv_D, Hit_Writeback_Inv_D,
+			 64)
+__BUILD_BLAST_USER_CACHE(i, icache, Index_Invalidate_I, Hit_Invalidate_I, 64)
+
 /* build blast_xxx_range, protected_blast_xxx_range */
 #define __BUILD_BLAST_CACHE_RANGE(pfx, desc, hitop, prot, extra)	\
-static inline void prot##extra##blast_##pfx##cache##_range(unsigned long start, \
+static inline void prot##extra##blast_##pfx##cache##_range(unsigned long start,\
 						    unsigned long end)	\
 {									\
 	unsigned long lsize = cpu_##desc##_line_size();			\
@@ -467,9 +579,47 @@ static inline void prot##extra##blast_##pfx##cache##_range(unsigned long start,
 	__##pfx##flush_epilogue						\
 }
 
+#ifndef CONFIG_EVA
+
 __BUILD_BLAST_CACHE_RANGE(d, dcache, Hit_Writeback_Inv_D, protected_, )
-__BUILD_BLAST_CACHE_RANGE(s, scache, Hit_Writeback_Inv_SD, protected_, )
 __BUILD_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I, protected_, )
+
+#else
+
+#define __BUILD_PROT_BLAST_CACHE_RANGE(pfx, desc, hitop)		\
+static inline void protected_blast_##pfx##cache##_range(unsigned long start,\
+							unsigned long end) \
+{									\
+	unsigned long lsize = cpu_##desc##_line_size();			\
+	unsigned long addr = start & ~(lsize - 1);			\
+	unsigned long aend = (end - 1) & ~(lsize - 1);			\
+									\
+	__##pfx##flush_prologue						\
+									\
+	if (segment_eq(get_fs(), USER_DS)) {				\
+		while (1) {						\
+			protected_cachee_op(hitop, addr);		\
+			if (addr == aend)				\
+				break;					\
+			addr += lsize;					\
+		}							\
+	} else {							\
+		while (1) {						\
+			protected_cache_op(hitop, addr);		\
+			if (addr == aend)				\
+				break;					\
+			addr += lsize;					\
+		}                                                       \
+									\
+	}								\
+	__##pfx##flush_epilogue						\
+}
+
+__BUILD_PROT_BLAST_CACHE_RANGE(d, dcache, Hit_Writeback_Inv_D)
+__BUILD_PROT_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I)
+
+#endif
+__BUILD_BLAST_CACHE_RANGE(s, scache, Hit_Writeback_Inv_SD, protected_, )
 __BUILD_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I_Loongson23, \
 	protected_, loongson23_)
 __BUILD_BLAST_CACHE_RANGE(d, dcache, Hit_Writeback_Inv_D, , )
-- 
1.8.5.3

[PATCH 41/58] MIPS: asm: r4kcache: Add EVA cache flushing functions

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

Add EVA cache flushing functions similar to non-EVA configurations.
Because the cache may or may not contain user virtual addresses, we
need to use the 'cache' or 'cachee' instruction based on whether we
flush the cache on behalf of kernel or user respectively.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/r4kcache.h | 154 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 152 insertions(+), 2 deletions(-)

diff --git a/arch/mips/include/asm/r4kcache.h b/arch/mips/include/asm/r4kcache.h
index ae026bf..b973e02 100644
--- a/arch/mips/include/asm/r4kcache.h
+++ b/arch/mips/include/asm/r4kcache.h
@@ -17,6 +17,7 @@
 #include <asm/cpu-features.h>
 #include <asm/cpu-type.h>
 #include <asm/mipsmtregs.h>
+#include <asm/uaccess.h> /* for segment_eq() */
 
 /*
  * This macro return a properly sign-extended address suitable as base address
@@ -374,6 +375,91 @@ static inline void invalidate_tcache_page(unsigned long addr)
 		: "r" (base),						\
 		  "i" (op));
 
+/*
+ * Perform the cache operation specified by op using a user mode virtual
+ * address while in kernel mode.
+ */
+#define cache16_unroll32_user(base,op)					\
+	__asm__ __volatile__(						\
+	"	.set push					\n"	\
+	"	.set noreorder					\n"	\
+	"	.set mips0					\n"	\
+	"	.set eva					\n"	\
+	"	cachee %1, 0x000(%0); cachee %1, 0x010(%0)	\n"	\
+	"	cachee %1, 0x020(%0); cachee %1, 0x030(%0)	\n"	\
+	"	cachee %1, 0x040(%0); cachee %1, 0x050(%0)	\n"	\
+	"	cachee %1, 0x060(%0); cachee %1, 0x070(%0)	\n"	\
+	"	cachee %1, 0x080(%0); cachee %1, 0x090(%0)	\n"	\
+	"	cachee %1, 0x0a0(%0); cachee %1, 0x0b0(%0)	\n"	\
+	"	cachee %1, 0x0c0(%0); cachee %1, 0x0d0(%0)	\n"	\
+	"	cachee %1, 0x0e0(%0); cachee %1, 0x0f0(%0)	\n"	\
+	"	cachee %1, 0x100(%0); cachee %1, 0x110(%0)	\n"	\
+	"	cachee %1, 0x120(%0); cachee %1, 0x130(%0)	\n"	\
+	"	cachee %1, 0x140(%0); cachee %1, 0x150(%0)	\n"	\
+	"	cachee %1, 0x160(%0); cachee %1, 0x170(%0)	\n"	\
+	"	cachee %1, 0x180(%0); cachee %1, 0x190(%0)	\n"	\
+	"	cachee %1, 0x1a0(%0); cachee %1, 0x1b0(%0)	\n"	\
+	"	cachee %1, 0x1c0(%0); cachee %1, 0x1d0(%0)	\n"	\
+	"	cachee %1, 0x1e0(%0); cachee %1, 0x1f0(%0)	\n"	\
+	"	.set pop					\n"	\
+		:							\
+		: "r" (base),						\
+		  "i" (op));
+
+#define cache32_unroll32_user(base, op)					\
+	__asm__ __volatile__(						\
+	"	.set push					\n"	\
+	"	.set noreorder					\n"	\
+	"	.set mips0					\n"	\
+	"	.set eva					\n"	\
+	"	cachee %1, 0x000(%0); cachee %1, 0x020(%0)	\n"	\
+	"	cachee %1, 0x040(%0); cachee %1, 0x060(%0)	\n"	\
+	"	cachee %1, 0x080(%0); cachee %1, 0x0a0(%0)	\n"	\
+	"	cachee %1, 0x0c0(%0); cachee %1, 0x0e0(%0)	\n"	\
+	"	cachee %1, 0x100(%0); cachee %1, 0x120(%0)	\n"	\
+	"	cachee %1, 0x140(%0); cachee %1, 0x160(%0)	\n"	\
+	"	cachee %1, 0x180(%0); cachee %1, 0x1a0(%0)	\n"	\
+	"	cachee %1, 0x1c0(%0); cachee %1, 0x1e0(%0)	\n"	\
+	"	cachee %1, 0x200(%0); cachee %1, 0x220(%0)	\n"	\
+	"	cachee %1, 0x240(%0); cachee %1, 0x260(%0)	\n"	\
+	"	cachee %1, 0x280(%0); cachee %1, 0x2a0(%0)	\n"	\
+	"	cachee %1, 0x2c0(%0); cachee %1, 0x2e0(%0)	\n"	\
+	"	cachee %1, 0x300(%0); cachee %1, 0x320(%0)	\n"	\
+	"	cachee %1, 0x340(%0); cachee %1, 0x360(%0)	\n"	\
+	"	cachee %1, 0x380(%0); cachee %1, 0x3a0(%0)	\n"	\
+	"	cachee %1, 0x3c0(%0); cachee %1, 0x3e0(%0)	\n"	\
+	"	.set pop					\n"	\
+		:							\
+		: "r" (base),						\
+		  "i" (op));
+
+#define cache64_unroll32_user(base, op)					\
+	__asm__ __volatile__(						\
+	"	.set push					\n"	\
+	"	.set noreorder					\n"	\
+	"	.set mips0					\n"	\
+	"	.set eva					\n"	\
+	"	cachee %1, 0x000(%0); cachee %1, 0x040(%0)	\n"	\
+	"	cachee %1, 0x080(%0); cachee %1, 0x0c0(%0)	\n"	\
+	"	cachee %1, 0x100(%0); cachee %1, 0x140(%0)	\n"	\
+	"	cachee %1, 0x180(%0); cachee %1, 0x1c0(%0)	\n"	\
+	"	cachee %1, 0x200(%0); cachee %1, 0x240(%0)	\n"	\
+	"	cachee %1, 0x280(%0); cachee %1, 0x2c0(%0)	\n"	\
+	"	cachee %1, 0x300(%0); cachee %1, 0x340(%0)	\n"	\
+	"	cachee %1, 0x380(%0); cachee %1, 0x3c0(%0)	\n"	\
+	"	cachee %1, 0x400(%0); cachee %1, 0x440(%0)	\n"	\
+	"	cachee %1, 0x480(%0); cachee %1, 0x4c0(%0)	\n"	\
+	"	cachee %1, 0x500(%0); cachee %1, 0x540(%0)	\n"	\
+	"	cachee %1, 0x580(%0); cachee %1, 0x5c0(%0)	\n"	\
+	"	cachee %1, 0x600(%0); cachee %1, 0x640(%0)	\n"	\
+	"	cachee %1, 0x680(%0); cachee %1, 0x6c0(%0)	\n"	\
+	"	cachee %1, 0x700(%0); cachee %1, 0x740(%0)	\n"	\
+	"	cachee %1, 0x780(%0); cachee %1, 0x7c0(%0)	\n"	\
+	"	.set pop					\n"	\
+		:							\
+		: "r" (base),						\
+		  "i" (op));
+
 /* build blast_xxx, blast_xxx_page, blast_xxx_page_indexed */
 #define __BUILD_BLAST_CACHE(pfx, desc, indexop, hitop, lsize) \
 static inline void blast_##pfx##cache##lsize(void)			\
@@ -446,9 +532,35 @@ __BUILD_BLAST_CACHE(inv_s, scache, Index_Writeback_Inv_SD, Hit_Invalidate_SD, 32
 __BUILD_BLAST_CACHE(inv_s, scache, Index_Writeback_Inv_SD, Hit_Invalidate_SD, 64)
 __BUILD_BLAST_CACHE(inv_s, scache, Index_Writeback_Inv_SD, Hit_Invalidate_SD, 128)
 
+#define __BUILD_BLAST_USER_CACHE(pfx, desc, indexop, hitop, lsize) \
+static inline void blast_##pfx##cache##lsize##_user_page(unsigned long page) \
+{									\
+	unsigned long start = page;					\
+	unsigned long end = page + PAGE_SIZE;				\
+									\
+	__##pfx##flush_prologue						\
+									\
+	do {								\
+		cache##lsize##_unroll32_user(start, hitop);             \
+		start += lsize * 32;					\
+	} while (start < end);						\
+									\
+	__##pfx##flush_epilogue						\
+}
+
+__BUILD_BLAST_USER_CACHE(d, dcache, Index_Writeback_Inv_D, Hit_Writeback_Inv_D,
+			 16)
+__BUILD_BLAST_USER_CACHE(i, icache, Index_Invalidate_I, Hit_Invalidate_I, 16)
+__BUILD_BLAST_USER_CACHE(d, dcache, Index_Writeback_Inv_D, Hit_Writeback_Inv_D,
+			 32)
+__BUILD_BLAST_USER_CACHE(i, icache, Index_Invalidate_I, Hit_Invalidate_I, 32)
+__BUILD_BLAST_USER_CACHE(d, dcache, Index_Writeback_Inv_D, Hit_Writeback_Inv_D,
+			 64)
+__BUILD_BLAST_USER_CACHE(i, icache, Index_Invalidate_I, Hit_Invalidate_I, 64)
+
 /* build blast_xxx_range, protected_blast_xxx_range */
 #define __BUILD_BLAST_CACHE_RANGE(pfx, desc, hitop, prot, extra)	\
-static inline void prot##extra##blast_##pfx##cache##_range(unsigned long start, \
+static inline void prot##extra##blast_##pfx##cache##_range(unsigned long start,\
 						    unsigned long end)	\
 {									\
 	unsigned long lsize = cpu_##desc##_line_size();			\
@@ -467,9 +579,47 @@ static inline void prot##extra##blast_##pfx##cache##_range(unsigned long start,
 	__##pfx##flush_epilogue						\
 }
 
+#ifndef CONFIG_EVA
+
 __BUILD_BLAST_CACHE_RANGE(d, dcache, Hit_Writeback_Inv_D, protected_, )
-__BUILD_BLAST_CACHE_RANGE(s, scache, Hit_Writeback_Inv_SD, protected_, )
 __BUILD_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I, protected_, )
+
+#else
+
+#define __BUILD_PROT_BLAST_CACHE_RANGE(pfx, desc, hitop)		\
+static inline void protected_blast_##pfx##cache##_range(unsigned long start,\
+							unsigned long end) \
+{									\
+	unsigned long lsize = cpu_##desc##_line_size();			\
+	unsigned long addr = start & ~(lsize - 1);			\
+	unsigned long aend = (end - 1) & ~(lsize - 1);			\
+									\
+	__##pfx##flush_prologue						\
+									\
+	if (segment_eq(get_fs(), USER_DS)) {				\
+		while (1) {						\
+			protected_cachee_op(hitop, addr);		\
+			if (addr == aend)				\
+				break;					\
+			addr += lsize;					\
+		}							\
+	} else {							\
+		while (1) {						\
+			protected_cache_op(hitop, addr);		\
+			if (addr == aend)				\
+				break;					\
+			addr += lsize;					\
+		}                                                       \
+									\
+	}								\
+	__##pfx##flush_epilogue						\
+}
+
+__BUILD_PROT_BLAST_CACHE_RANGE(d, dcache, Hit_Writeback_Inv_D)
+__BUILD_PROT_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I)
+
+#endif
+__BUILD_BLAST_CACHE_RANGE(s, scache, Hit_Writeback_Inv_SD, protected_, )
 __BUILD_BLAST_CACHE_RANGE(i, icache, Hit_Invalidate_I_Loongson23, \
 	protected_, loongson23_)
 __BUILD_BLAST_CACHE_RANGE(d, dcache, Hit_Writeback_Inv_D, , )
-- 
1.8.5.3

[PATCH 42/58] MIPS: kernel: {ftrace,kgdb}: Set correct address limit for cache flushes

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

When flushing the icache, make sure the address limit is correct
so the appropriate 'cache' instruction will be used. This has no
impact on cores operating in non-eva mode. However, when EVA is
enabled, we ensure that 'cache' will be used instead of 'cachee'.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/ftrace.c |  4 ++++
 arch/mips/kernel/kgdb.c   | 17 ++++++++++++++++-
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/arch/mips/kernel/ftrace.c b/arch/mips/kernel/ftrace.c
index 185ba25..ddcc350 100644
--- a/arch/mips/kernel/ftrace.c
+++ b/arch/mips/kernel/ftrace.c
@@ -90,6 +90,7 @@ static inline void ftrace_dyn_arch_init_insns(void)
 static int ftrace_modify_code(unsigned long ip, unsigned int new_code)
 {
 	int faulted;
+	mm_segment_t old_fs;
 
 	/* *(unsigned int *)ip = new_code; */
 	safe_store_code(new_code, ip, faulted);
@@ -97,7 +98,10 @@ static int ftrace_modify_code(unsigned long ip, unsigned int new_code)
 	if (unlikely(faulted))
 		return -EFAULT;
 
+	old_fs = get_fs();
+	set_fs(get_ds());
 	flush_icache_range(ip, ip + 8);
+	set_fs(old_fs);
 
 	return 0;
 }
diff --git a/arch/mips/kernel/kgdb.c b/arch/mips/kernel/kgdb.c
index fcaac2f..a70ffe4 100644
--- a/arch/mips/kernel/kgdb.c
+++ b/arch/mips/kernel/kgdb.c
@@ -208,7 +208,14 @@ void arch_kgdb_breakpoint(void)
 
 static void kgdb_call_nmi_hook(void *ignored)
 {
+	mm_segment_t old_fs;
+
+	old_fs = get_fs();
+	set_fs(get_ds());
+
 	kgdb_nmicallback(raw_smp_processor_id(), NULL);
+
+	set_fs(old_fs);
 }
 
 void kgdb_roundup_cpus(unsigned long flags)
@@ -282,6 +289,7 @@ static int kgdb_mips_notify(struct notifier_block *self, unsigned long cmd,
 	struct die_args *args = (struct die_args *)ptr;
 	struct pt_regs *regs = args->regs;
 	int trap = (regs->cp0_cause & 0x7c) >> 2;
+	mm_segment_t old_fs;
 
 #ifdef CONFIG_KPROBES
 	/*
@@ -296,11 +304,17 @@ static int kgdb_mips_notify(struct notifier_block *self, unsigned long cmd,
 	if (user_mode(regs))
 		return NOTIFY_DONE;
 
+	/* Kernel mode. Set correct address limit */
+	old_fs = get_fs();
+	set_fs(get_ds());
+
 	if (atomic_read(&kgdb_active) != -1)
 		kgdb_nmicallback(smp_processor_id(), regs);
 
-	if (kgdb_handle_exception(trap, compute_signal(trap), cmd, regs))
+	if (kgdb_handle_exception(trap, compute_signal(trap), cmd, regs)) {
+		set_fs(old_fs);
 		return NOTIFY_DONE;
+	}
 
 	if (atomic_read(&kgdb_setting_breakpoint))
 		if ((trap == 9) && (regs->cp0_epc == (unsigned long)breakinst))
@@ -310,6 +324,7 @@ static int kgdb_mips_notify(struct notifier_block *self, unsigned long cmd,
 	local_irq_enable();
 	__flush_cache_all();
 
+	set_fs(old_fs);
 	return NOTIFY_STOP;
 }
 
-- 
1.8.5.3

[PATCH 42/58] MIPS: kernel: {ftrace,kgdb}: Set correct address limit for cache flushes

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

When flushing the icache, make sure the address limit is correct
so the appropriate 'cache' instruction will be used. This has no
impact on cores operating in non-eva mode. However, when EVA is
enabled, we ensure that 'cache' will be used instead of 'cachee'.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/ftrace.c |  4 ++++
 arch/mips/kernel/kgdb.c   | 17 ++++++++++++++++-
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/arch/mips/kernel/ftrace.c b/arch/mips/kernel/ftrace.c
index 185ba25..ddcc350 100644
--- a/arch/mips/kernel/ftrace.c
+++ b/arch/mips/kernel/ftrace.c
@@ -90,6 +90,7 @@ static inline void ftrace_dyn_arch_init_insns(void)
 static int ftrace_modify_code(unsigned long ip, unsigned int new_code)
 {
 	int faulted;
+	mm_segment_t old_fs;
 
 	/* *(unsigned int *)ip = new_code; */
 	safe_store_code(new_code, ip, faulted);
@@ -97,7 +98,10 @@ static int ftrace_modify_code(unsigned long ip, unsigned int new_code)
 	if (unlikely(faulted))
 		return -EFAULT;
 
+	old_fs = get_fs();
+	set_fs(get_ds());
 	flush_icache_range(ip, ip + 8);
+	set_fs(old_fs);
 
 	return 0;
 }
diff --git a/arch/mips/kernel/kgdb.c b/arch/mips/kernel/kgdb.c
index fcaac2f..a70ffe4 100644
--- a/arch/mips/kernel/kgdb.c
+++ b/arch/mips/kernel/kgdb.c
@@ -208,7 +208,14 @@ void arch_kgdb_breakpoint(void)
 
 static void kgdb_call_nmi_hook(void *ignored)
 {
+	mm_segment_t old_fs;
+
+	old_fs = get_fs();
+	set_fs(get_ds());
+
 	kgdb_nmicallback(raw_smp_processor_id(), NULL);
+
+	set_fs(old_fs);
 }
 
 void kgdb_roundup_cpus(unsigned long flags)
@@ -282,6 +289,7 @@ static int kgdb_mips_notify(struct notifier_block *self, unsigned long cmd,
 	struct die_args *args = (struct die_args *)ptr;
 	struct pt_regs *regs = args->regs;
 	int trap = (regs->cp0_cause & 0x7c) >> 2;
+	mm_segment_t old_fs;
 
 #ifdef CONFIG_KPROBES
 	/*
@@ -296,11 +304,17 @@ static int kgdb_mips_notify(struct notifier_block *self, unsigned long cmd,
 	if (user_mode(regs))
 		return NOTIFY_DONE;
 
+	/* Kernel mode. Set correct address limit */
+	old_fs = get_fs();
+	set_fs(get_ds());
+
 	if (atomic_read(&kgdb_active) != -1)
 		kgdb_nmicallback(smp_processor_id(), regs);
 
-	if (kgdb_handle_exception(trap, compute_signal(trap), cmd, regs))
+	if (kgdb_handle_exception(trap, compute_signal(trap), cmd, regs)) {
+		set_fs(old_fs);
 		return NOTIFY_DONE;
+	}
 
 	if (atomic_read(&kgdb_setting_breakpoint))
 		if ((trap == 9) && (regs->cp0_epc == (unsigned long)breakinst))
@@ -310,6 +324,7 @@ static int kgdb_mips_notify(struct notifier_block *self, unsigned long cmd,
 	local_irq_enable();
 	__flush_cache_all();
 
+	set_fs(old_fs);
 	return NOTIFY_STOP;
 }
 
-- 
1.8.5.3

[PATCH 43/58] MIPS: asm: page: Allow __pa_symbol overrides

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

This will allow platforms to use an alternative way to get
the physical address of a symbol.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/page.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/mips/include/asm/page.h b/arch/mips/include/asm/page.h
index 5e08bcc..5699ec3 100644
--- a/arch/mips/include/asm/page.h
+++ b/arch/mips/include/asm/page.h
@@ -190,7 +190,9 @@ typedef struct { unsigned long pgprot; } pgprot_t;
  * https://patchwork.linux-mips.org/patch/1541/
  */
 
+#ifndef __pa_symbol
 #define __pa_symbol(x)	__pa(RELOC_HIDE((unsigned long)(x), 0))
+#endif
 
 #define pfn_to_kaddr(pfn)	__va((pfn) << PAGE_SHIFT)
 
-- 
1.8.5.3

[PATCH 43/58] MIPS: asm: page: Allow __pa_symbol overrides

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

This will allow platforms to use an alternative way to get
the physical address of a symbol.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/page.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/mips/include/asm/page.h b/arch/mips/include/asm/page.h
index 5e08bcc..5699ec3 100644
--- a/arch/mips/include/asm/page.h
+++ b/arch/mips/include/asm/page.h
@@ -190,7 +190,9 @@ typedef struct { unsigned long pgprot; } pgprot_t;
  * https://patchwork.linux-mips.org/patch/1541/
  */
 
+#ifndef __pa_symbol
 #define __pa_symbol(x)	__pa(RELOC_HIDE((unsigned long)(x), 0))
+#endif
 
 #define pfn_to_kaddr(pfn)	__va((pfn) << PAGE_SHIFT)
 
-- 
1.8.5.3

[PATCH 44/58] MIPS: asm: cpu: Add cpu flag for Enhanced Virtual Addressing

[Markos Chandras]

The MIPS *Aptiv family uses bit 28 in Config5 CP0 register to
indicate whether the core supports EVA or not.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/cpu-features.h | 4 +++-
 arch/mips/include/asm/cpu.h          | 1 +
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/arch/mips/include/asm/cpu-features.h b/arch/mips/include/asm/cpu-features.h
index 9052fb9..6b95c8e 100644
--- a/arch/mips/include/asm/cpu-features.h
+++ b/arch/mips/include/asm/cpu-features.h
@@ -26,7 +26,9 @@
 #ifndef cpu_has_segments
 #define cpu_has_segments	(cpu_data[0].options & MIPS_CPU_SEGMENTS)
 #endif
-
+#ifndef cpu_has_eva
+#define cpu_has_eva		(cpu_data[0].options & MIPS_CPU_EVA)
+#endif
 
 /*
  * For the moment we don't consider R6000 and R8000 so we can assume that
diff --git a/arch/mips/include/asm/cpu.h b/arch/mips/include/asm/cpu.h
index 76411df..1474750 100644
--- a/arch/mips/include/asm/cpu.h
+++ b/arch/mips/include/asm/cpu.h
@@ -358,6 +358,7 @@ enum cpu_type_enum {
 #define MIPS_CPU_MICROMIPS	0x01000000 /* CPU has microMIPS capability */
 #define MIPS_CPU_TLBINV		0x02000000 /* CPU supports TLBINV/F */
 #define MIPS_CPU_SEGMENTS	0x04000000 /* CPU supports Segmentation Control registers */
+#define MIPS_CPU_EVA		0x80000000 /* CPU supports Enhanced Virtual Addressing */
 
 /*
  * CPU ASE encodings
-- 
1.8.5.3

[PATCH 44/58] MIPS: asm: cpu: Add cpu flag for Enhanced Virtual Addressing

[Markos Chandras]

The MIPS *Aptiv family uses bit 28 in Config5 CP0 register to
indicate whether the core supports EVA or not.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/cpu-features.h | 4 +++-
 arch/mips/include/asm/cpu.h          | 1 +
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/arch/mips/include/asm/cpu-features.h b/arch/mips/include/asm/cpu-features.h
index 9052fb9..6b95c8e 100644
--- a/arch/mips/include/asm/cpu-features.h
+++ b/arch/mips/include/asm/cpu-features.h
@@ -26,7 +26,9 @@
 #ifndef cpu_has_segments
 #define cpu_has_segments	(cpu_data[0].options & MIPS_CPU_SEGMENTS)
 #endif
-
+#ifndef cpu_has_eva
+#define cpu_has_eva		(cpu_data[0].options & MIPS_CPU_EVA)
+#endif
 
 /*
  * For the moment we don't consider R6000 and R8000 so we can assume that
diff --git a/arch/mips/include/asm/cpu.h b/arch/mips/include/asm/cpu.h
index 76411df..1474750 100644
--- a/arch/mips/include/asm/cpu.h
+++ b/arch/mips/include/asm/cpu.h
@@ -358,6 +358,7 @@ enum cpu_type_enum {
 #define MIPS_CPU_MICROMIPS	0x01000000 /* CPU has microMIPS capability */
 #define MIPS_CPU_TLBINV		0x02000000 /* CPU supports TLBINV/F */
 #define MIPS_CPU_SEGMENTS	0x04000000 /* CPU supports Segmentation Control registers */
+#define MIPS_CPU_EVA		0x80000000 /* CPU supports Enhanced Virtual Addressing */
 
 /*
  * CPU ASE encodings
-- 
1.8.5.3

[PATCH 45/58] MIPS: kernel: cpu-probe: Enable EVA option on supported cores

[Markos Chandras]

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/cpu-probe.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/mips/kernel/cpu-probe.c b/arch/mips/kernel/cpu-probe.c
index 530f832..c42731c 100644
--- a/arch/mips/kernel/cpu-probe.c
+++ b/arch/mips/kernel/cpu-probe.c
@@ -367,6 +367,9 @@ static inline unsigned int decode_config5(struct cpuinfo_mips *c)
 	config5 &= ~MIPS_CONF5_UFR;
 	write_c0_config5(config5);
 
+	if (config5 & MIPS_CONF5_EVA)
+		c->options |= MIPS_CPU_EVA;
+
 	return config5 & MIPS_CONF_M;
 }
 
-- 
1.8.5.3

[PATCH 45/58] MIPS: kernel: cpu-probe: Enable EVA option on supported cores

[Markos Chandras]

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/cpu-probe.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/mips/kernel/cpu-probe.c b/arch/mips/kernel/cpu-probe.c
index 530f832..c42731c 100644
--- a/arch/mips/kernel/cpu-probe.c
+++ b/arch/mips/kernel/cpu-probe.c
@@ -367,6 +367,9 @@ static inline unsigned int decode_config5(struct cpuinfo_mips *c)
 	config5 &= ~MIPS_CONF5_UFR;
 	write_c0_config5(config5);
 
+	if (config5 & MIPS_CONF5_EVA)
+		c->options |= MIPS_CPU_EVA;
+
 	return config5 & MIPS_CONF_M;
 }
 
-- 
1.8.5.3

[PATCH 46/58] MIPS: kernel: proc: Add EVA to the list of CPU features

[Markos Chandras]

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/proc.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/mips/kernel/proc.c b/arch/mips/kernel/proc.c
index 00d2097..ead0e47 100644
--- a/arch/mips/kernel/proc.c
+++ b/arch/mips/kernel/proc.c
@@ -95,6 +95,7 @@ static int show_cpuinfo(struct seq_file *m, void *v)
 	if (cpu_has_mipsmt)	seq_printf(m, "%s", " mt");
 	if (cpu_has_mmips)	seq_printf(m, "%s", " micromips");
 	if (cpu_has_vz)		seq_printf(m, "%s", " vz");
+	if (cpu_has_eva)	seq_printf(m, "%s", " eva");
 	seq_printf(m, "\n");
 
 	if (cpu_has_mmips) {
-- 
1.8.5.3

[PATCH 46/58] MIPS: kernel: proc: Add EVA to the list of CPU features

[Markos Chandras]

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/kernel/proc.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/mips/kernel/proc.c b/arch/mips/kernel/proc.c
index 00d2097..ead0e47 100644
--- a/arch/mips/kernel/proc.c
+++ b/arch/mips/kernel/proc.c
@@ -95,6 +95,7 @@ static int show_cpuinfo(struct seq_file *m, void *v)
 	if (cpu_has_mipsmt)	seq_printf(m, "%s", " mt");
 	if (cpu_has_mmips)	seq_printf(m, "%s", " micromips");
 	if (cpu_has_vz)		seq_printf(m, "%s", " vz");
+	if (cpu_has_eva)	seq_printf(m, "%s", " eva");
 	seq_printf(m, "\n");
 
 	if (cpu_has_mmips) {
-- 
1.8.5.3

[PATCH 47/58] MIPS: mm: init: Add free_init_pages() callback for EVA

[Markos Chandras]

A core in EVA mode can have any possible segment mapping, so the
default free_initmem_default() function may not always work as expected.
Therefore, add a callback that platforms can use to free up the init section.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/bootinfo.h |  2 ++
 arch/mips/mm/init.c              | 12 +++++++++++-
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/arch/mips/include/asm/bootinfo.h b/arch/mips/include/asm/bootinfo.h
index 4d2cdea..83a4db4e 100644
--- a/arch/mips/include/asm/bootinfo.h
+++ b/arch/mips/include/asm/bootinfo.h
@@ -112,6 +112,8 @@ extern void prom_free_prom_memory(void);
 extern void free_init_pages(const char *what,
 			    unsigned long begin, unsigned long end);
 
+extern void (*free_init_pages_eva)(void *begin, void *end);
+
 /*
  * Initial kernel command line, usually setup by prom_init()
  */
diff --git a/arch/mips/mm/init.c b/arch/mips/mm/init.c
index 6b59617..4fc74c7 100644
--- a/arch/mips/mm/init.c
+++ b/arch/mips/mm/init.c
@@ -422,10 +422,20 @@ void free_initrd_mem(unsigned long start, unsigned long end)
 }
 #endif
 
+void (*free_init_pages_eva)(void *begin, void *end) = NULL;
+
 void __init_refok free_initmem(void)
 {
 	prom_free_prom_memory();
-	free_initmem_default(POISON_FREE_INITMEM);
+	/*
+	 * Let the platform define a specific function to free the
+	 * init section since EVA may have used any possible mapping
+	 * between virtual and physical addresses.
+	 */
+	if (free_init_pages_eva)
+		free_init_pages_eva((void *)&__init_begin, (void *)&__init_end);
+	else
+		free_initmem_default(POISON_FREE_INITMEM);
 }
 
 #ifndef CONFIG_MIPS_PGD_C0_CONTEXT
-- 
1.8.5.3

[PATCH 47/58] MIPS: mm: init: Add free_init_pages() callback for EVA

[Markos Chandras]

A core in EVA mode can have any possible segment mapping, so the
default free_initmem_default() function may not always work as expected.
Therefore, add a callback that platforms can use to free up the init section.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/bootinfo.h |  2 ++
 arch/mips/mm/init.c              | 12 +++++++++++-
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/arch/mips/include/asm/bootinfo.h b/arch/mips/include/asm/bootinfo.h
index 4d2cdea..83a4db4e 100644
--- a/arch/mips/include/asm/bootinfo.h
+++ b/arch/mips/include/asm/bootinfo.h
@@ -112,6 +112,8 @@ extern void prom_free_prom_memory(void);
 extern void free_init_pages(const char *what,
 			    unsigned long begin, unsigned long end);
 
+extern void (*free_init_pages_eva)(void *begin, void *end);
+
 /*
  * Initial kernel command line, usually setup by prom_init()
  */
diff --git a/arch/mips/mm/init.c b/arch/mips/mm/init.c
index 6b59617..4fc74c7 100644
--- a/arch/mips/mm/init.c
+++ b/arch/mips/mm/init.c
@@ -422,10 +422,20 @@ void free_initrd_mem(unsigned long start, unsigned long end)
 }
 #endif
 
+void (*free_init_pages_eva)(void *begin, void *end) = NULL;
+
 void __init_refok free_initmem(void)
 {
 	prom_free_prom_memory();
-	free_initmem_default(POISON_FREE_INITMEM);
+	/*
+	 * Let the platform define a specific function to free the
+	 * init section since EVA may have used any possible mapping
+	 * between virtual and physical addresses.
+	 */
+	if (free_init_pages_eva)
+		free_init_pages_eva((void *)&__init_begin, (void *)&__init_end);
+	else
+		free_initmem_default(POISON_FREE_INITMEM);
 }
 
 #ifndef CONFIG_MIPS_PGD_C0_CONTEXT
-- 
1.8.5.3

[PATCH 48/58] MIPS: mm: c-r4k: Build EVA {d,i}cache flushing functions

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

Build EVA specific cache flushing functions (ie cachee).
They will be used by a subsequent patch.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/mm/c-r4k.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)

diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 13b549a..0c9c693 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -123,6 +123,28 @@ static void r4k_blast_dcache_page_setup(void)
 		r4k_blast_dcache_page = r4k_blast_dcache_page_dc64;
 }
 
+#ifndef CONFIG_EVA
+#define r4k_blast_dcache_user_page  r4k_blast_dcache_page
+#else
+
+static void (*r4k_blast_dcache_user_page)(unsigned long addr);
+
+static void r4k_blast_dcache_user_page_setup(void)
+{
+	unsigned long  dc_lsize = cpu_dcache_line_size();
+
+	if (dc_lsize == 0)
+		r4k_blast_dcache_user_page = (void *)cache_noop;
+	else if (dc_lsize == 16)
+		r4k_blast_dcache_user_page = blast_dcache16_user_page;
+	else if (dc_lsize == 32)
+		r4k_blast_dcache_user_page = blast_dcache32_user_page;
+	else if (dc_lsize == 64)
+		r4k_blast_dcache_user_page = blast_dcache64_user_page;
+}
+
+#endif
+
 static void (* r4k_blast_dcache_page_indexed)(unsigned long addr);
 
 static void r4k_blast_dcache_page_indexed_setup(void)
@@ -243,6 +265,27 @@ static void r4k_blast_icache_page_setup(void)
 		r4k_blast_icache_page = blast_icache64_page;
 }
 
+#ifndef CONFIG_EVA
+#define r4k_blast_icache_user_page  r4k_blast_icache_page
+#else
+
+static void (*r4k_blast_icache_user_page)(unsigned long addr);
+
+static void __cpuinit r4k_blast_icache_user_page_setup(void)
+{
+	unsigned long ic_lsize = cpu_icache_line_size();
+
+	if (ic_lsize == 0)
+		r4k_blast_icache_user_page = (void *)cache_noop;
+	else if (ic_lsize == 16)
+		r4k_blast_icache_user_page = blast_icache16_user_page;
+	else if (ic_lsize == 32)
+		r4k_blast_icache_user_page = blast_icache32_user_page;
+	else if (ic_lsize == 64)
+		r4k_blast_icache_user_page = blast_icache64_user_page;
+}
+
+#endif
 
 static void (* r4k_blast_icache_page_indexed)(unsigned long addr);
 
@@ -1454,6 +1497,10 @@ void r4k_cache_init(void)
 	r4k_blast_scache_page_setup();
 	r4k_blast_scache_page_indexed_setup();
 	r4k_blast_scache_setup();
+#ifdef CONFIG_EVA
+	r4k_blast_dcache_user_page_setup();
+	r4k_blast_icache_user_page_setup();
+#endif
 
 	/*
 	 * Some MIPS32 and MIPS64 processors have physically indexed caches.
-- 
1.8.5.3

[PATCH 48/58] MIPS: mm: c-r4k: Build EVA {d,i}cache flushing functions

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

Build EVA specific cache flushing functions (ie cachee).
They will be used by a subsequent patch.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/mm/c-r4k.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)

diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 13b549a..0c9c693 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -123,6 +123,28 @@ static void r4k_blast_dcache_page_setup(void)
 		r4k_blast_dcache_page = r4k_blast_dcache_page_dc64;
 }
 
+#ifndef CONFIG_EVA
+#define r4k_blast_dcache_user_page  r4k_blast_dcache_page
+#else
+
+static void (*r4k_blast_dcache_user_page)(unsigned long addr);
+
+static void r4k_blast_dcache_user_page_setup(void)
+{
+	unsigned long  dc_lsize = cpu_dcache_line_size();
+
+	if (dc_lsize == 0)
+		r4k_blast_dcache_user_page = (void *)cache_noop;
+	else if (dc_lsize == 16)
+		r4k_blast_dcache_user_page = blast_dcache16_user_page;
+	else if (dc_lsize == 32)
+		r4k_blast_dcache_user_page = blast_dcache32_user_page;
+	else if (dc_lsize == 64)
+		r4k_blast_dcache_user_page = blast_dcache64_user_page;
+}
+
+#endif
+
 static void (* r4k_blast_dcache_page_indexed)(unsigned long addr);
 
 static void r4k_blast_dcache_page_indexed_setup(void)
@@ -243,6 +265,27 @@ static void r4k_blast_icache_page_setup(void)
 		r4k_blast_icache_page = blast_icache64_page;
 }
 
+#ifndef CONFIG_EVA
+#define r4k_blast_icache_user_page  r4k_blast_icache_page
+#else
+
+static void (*r4k_blast_icache_user_page)(unsigned long addr);
+
+static void __cpuinit r4k_blast_icache_user_page_setup(void)
+{
+	unsigned long ic_lsize = cpu_icache_line_size();
+
+	if (ic_lsize == 0)
+		r4k_blast_icache_user_page = (void *)cache_noop;
+	else if (ic_lsize == 16)
+		r4k_blast_icache_user_page = blast_icache16_user_page;
+	else if (ic_lsize == 32)
+		r4k_blast_icache_user_page = blast_icache32_user_page;
+	else if (ic_lsize == 64)
+		r4k_blast_icache_user_page = blast_icache64_user_page;
+}
+
+#endif
 
 static void (* r4k_blast_icache_page_indexed)(unsigned long addr);
 
@@ -1454,6 +1497,10 @@ void r4k_cache_init(void)
 	r4k_blast_scache_page_setup();
 	r4k_blast_scache_page_indexed_setup();
 	r4k_blast_scache_setup();
+#ifdef CONFIG_EVA
+	r4k_blast_dcache_user_page_setup();
+	r4k_blast_icache_user_page_setup();
+#endif
 
 	/*
 	 * Some MIPS32 and MIPS64 processors have physically indexed caches.
-- 
1.8.5.3

[PATCH 49/58] MIPS: mm: c-r4k: Add support for flushing user pages from cache

[Markos Chandras]

Use the userspace cache flushing functions if the interrupted
process is a userspace one.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/mm/c-r4k.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 0c9c693..5f8da6a 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -555,7 +555,8 @@ static inline void local_r4k_flush_cache_page(void *args)
 	}
 
 	if (cpu_has_dc_aliases || (exec && !cpu_has_ic_fills_f_dc)) {
-		r4k_blast_dcache_page(addr);
+		vaddr ? r4k_blast_dcache_page(addr) :
+			r4k_blast_dcache_user_page(addr);
 		if (exec && !cpu_icache_snoops_remote_store)
 			r4k_blast_scache_page(addr);
 	}
@@ -566,7 +567,8 @@ static inline void local_r4k_flush_cache_page(void *args)
 			if (cpu_context(cpu, mm) != 0)
 				drop_mmu_context(mm, cpu);
 		} else
-			r4k_blast_icache_page(addr);
+			vaddr ? r4k_blast_icache_page(addr) :
+				r4k_blast_icache_user_page(addr);
 	}
 
 	if (vaddr) {
-- 
1.8.5.3

[PATCH 49/58] MIPS: mm: c-r4k: Add support for flushing user pages from cache

[Markos Chandras]

Use the userspace cache flushing functions if the interrupted
process is a userspace one.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/mm/c-r4k.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 0c9c693..5f8da6a 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -555,7 +555,8 @@ static inline void local_r4k_flush_cache_page(void *args)
 	}
 
 	if (cpu_has_dc_aliases || (exec && !cpu_has_ic_fills_f_dc)) {
-		r4k_blast_dcache_page(addr);
+		vaddr ? r4k_blast_dcache_page(addr) :
+			r4k_blast_dcache_user_page(addr);
 		if (exec && !cpu_icache_snoops_remote_store)
 			r4k_blast_scache_page(addr);
 	}
@@ -566,7 +567,8 @@ static inline void local_r4k_flush_cache_page(void *args)
 			if (cpu_context(cpu, mm) != 0)
 				drop_mmu_context(mm, cpu);
 		} else
-			r4k_blast_icache_page(addr);
+			vaddr ? r4k_blast_icache_page(addr) :
+				r4k_blast_icache_user_page(addr);
 	}
 
 	if (vaddr) {
-- 
1.8.5.3

[PATCH 50/58] MIPS: mm: c-r4k: Flush scache to avoid cache aliases

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

There is a chance for the secondary cache to have memory
aliases. This can happen if the bootloader is in a non-EVA mode
(or even in EVA mode but with different mapping from the kernel)
and the kernel switching to EVA afterwards. It's best to flush
the icache to avoid having the secondary CPUs fetching stale
data from it.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/mm/c-r4k.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 5f8da6a..9761dfa 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -633,6 +633,17 @@ static inline void local_r4k_flush_icache_range(unsigned long start, unsigned lo
 			break;
 		}
 	}
+#ifdef CONFIG_EVA
+	/*
+	 * Due to all possible segment mappings, there might cache aliases
+	 * caused by the bootloader being in non-EVA mode, and the CPU switching
+	 * to EVA during early kernel init. It's best to flush the scache
+	 * to avoid having secondary cores fetching stale data and lead to
+	 * kernel crashes.
+	 */
+	bc_wback_inv(start, (end - start));
+	__sync();
+#endif
 }
 
 static inline void local_r4k_flush_icache_range_ipi(void *args)
-- 
1.8.5.3

[PATCH 50/58] MIPS: mm: c-r4k: Flush scache to avoid cache aliases

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

There is a chance for the secondary cache to have memory
aliases. This can happen if the bootloader is in a non-EVA mode
(or even in EVA mode but with different mapping from the kernel)
and the kernel switching to EVA afterwards. It's best to flush
the icache to avoid having the secondary CPUs fetching stale
data from it.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/mm/c-r4k.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 5f8da6a..9761dfa 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -633,6 +633,17 @@ static inline void local_r4k_flush_icache_range(unsigned long start, unsigned lo
 			break;
 		}
 	}
+#ifdef CONFIG_EVA
+	/*
+	 * Due to all possible segment mappings, there might cache aliases
+	 * caused by the bootloader being in non-EVA mode, and the CPU switching
+	 * to EVA during early kernel init. It's best to flush the scache
+	 * to avoid having secondary cores fetching stale data and lead to
+	 * kernel crashes.
+	 */
+	bc_wback_inv(start, (end - start));
+	__sync();
+#endif
 }
 
 static inline void local_r4k_flush_icache_range_ipi(void *args)
-- 
1.8.5.3

[PATCH 51/58] MIPS: malta: Configure Segment Control registers for EVA boot

[Markos Chandras]

The Malta board aliases 0x80000000 - 0xffffffff to 0x00000000
- 0x7fffffff ignoring the 256 MB IO hole in 0x10000000.
The physical memory is shifted to 0x80000000 so up to 2GB
can be used. Kuseg is expanded to 3GB (due to board limitations
only 2GB can be accessed) and lowmem (kernel space) is expanded to 2GB.

The Segment Control registers are programmed as follows:

Virtual memory           Physical memory           Mapping
0x00000000 - 0x7fffffff  0x80000000 - 0xfffffffff   MUSUK (kuseg)
0x80000000 - 0x9fffffff  0x00000000 - 0x1ffffffff   MUSUK (kseg0)
0xa0000000 - 0xbf000000  0x00000000 - 0x1ffffffff   MUSUK (kseg1)
0xc0000000 - 0xdfffffff             -                 MK  (kseg2)
0xe0000000 - 0xffffffff             -                 MK  (kseg3)

The location of exception vectors remain the same since 0xbfc00000
(traditional exception base) still maps to 0x1fc00000 physical.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 .../include/asm/mach-malta/kernel-entry-init.h     | 109 ++++++++++++++++++++-
 arch/mips/mti-malta/malta-setup.c                  |   4 +
 2 files changed, 112 insertions(+), 1 deletion(-)

diff --git a/arch/mips/include/asm/mach-malta/kernel-entry-init.h b/arch/mips/include/asm/mach-malta/kernel-entry-init.h
index 0b793e7..9bace9c 100644
--- a/arch/mips/include/asm/mach-malta/kernel-entry-init.h
+++ b/arch/mips/include/asm/mach-malta/kernel-entry-init.h
@@ -5,10 +5,80 @@
  *
  * Chris Dearman (chris@mips.com)
  * Copyright (C) 2007 Mips Technologies, Inc.
+ * Copyright (C) 2014 Imagination Technologies Ltd.
  */
 #ifndef __ASM_MACH_MIPS_KERNEL_ENTRY_INIT_H
 #define __ASM_MACH_MIPS_KERNEL_ENTRY_INIT_H
 
+	/*
+	 * Prepare segments for EVA boot:
+	 *
+	 * This is in case the processor boots in legacy configuration
+	 * (SI_EVAReset is de-asserted and CONFIG5.K == 0)
+	 *
+	 * On entry, t1 is loaded with CP0_CONFIG
+	 *
+	 * ========================= Mappings =============================
+	 * Virtual memory           Physical memory           Mapping
+	 * 0x00000000 - 0x7fffffff  0x80000000 - 0xfffffffff   MUSUK (kuseg)
+	 *                          Flat 2GB physical memory
+	 *
+	 * 0x80000000 - 0x9fffffff  0x00000000 - 0x1ffffffff   MUSUK (kseg0)
+	 * 0xa0000000 - 0xbf000000  0x00000000 - 0x1ffffffff   MUSUK (kseg1)
+	 * 0xc0000000 - 0xdfffffff             -                 MK  (kseg2)
+	 * 0xe0000000 - 0xffffffff             -                 MK  (kseg3)
+	 *
+	 *
+	 * Lowmem is expanded to 2GB
+	 */
+	.macro	eva_entry
+	/*
+	 * Get Config.K0 value and use it to program
+	 * the segmentation registers
+	 */
+	andi	t1, 0x7 /* CCA */
+	move	t2, t1
+	ins	t2, t1, 16, 3
+	/* SegCtl0 */
+	li      t0, ((MIPS_SEGCFG_MK << MIPS_SEGCFG_AM_SHIFT) |		\
+		(0 << MIPS_SEGCFG_PA_SHIFT) |				\
+		(1 << MIPS_SEGCFG_EU_SHIFT)) |				\
+		(((MIPS_SEGCFG_MK << MIPS_SEGCFG_AM_SHIFT) |		\
+		(0 << MIPS_SEGCFG_PA_SHIFT) |				\
+		(1 << MIPS_SEGCFG_EU_SHIFT)) << 16)
+	or	t0, t2
+	mtc0	t0, $5, 2
+
+	/* SegCtl1 */
+	li      t0, ((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) |	\
+		(0 << MIPS_SEGCFG_PA_SHIFT) |				\
+		(2 << MIPS_SEGCFG_C_SHIFT) |				\
+		(1 << MIPS_SEGCFG_EU_SHIFT)) |				\
+		(((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) |		\
+		(0 << MIPS_SEGCFG_PA_SHIFT) |				\
+		(1 << MIPS_SEGCFG_EU_SHIFT)) << 16)
+	ins	t0, t1, 16, 3
+	mtc0	t0, $5, 3
+
+	/* SegCtl2 */
+	li	t0, ((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) |	\
+		(6 << MIPS_SEGCFG_PA_SHIFT) |				\
+		(1 << MIPS_SEGCFG_EU_SHIFT)) |				\
+		(((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) |		\
+		(4 << MIPS_SEGCFG_PA_SHIFT) |				\
+		(1 << MIPS_SEGCFG_EU_SHIFT)) << 16)
+	or	t0, t2
+	mtc0	t0, $5, 4
+
+	jal	mips_ihb
+	mfc0    t0, $16, 5
+	li      t2, 0x40000000      /* K bit */
+	or      t0, t0, t2
+	mtc0    t0, $16, 5
+	sync
+	jal	mips_ihb
+	.endm
+
 	.macro	kernel_entry_setup
 #ifdef CONFIG_MIPS_MT_SMTC
 	mfc0	t0, CP0_CONFIG
@@ -39,8 +109,45 @@
 nonmt_processor:
 	.asciz	"SMTC kernel requires the MT ASE to run\n"
 	__FINIT
-0:
 #endif
+
+#ifdef CONFIG_EVA
+	sync
+	ehb
+
+	mfc0    t1, CP0_CONFIG
+	bgez    t1, 9f
+	mfc0	t0, CP0_CONFIG, 1
+	bgez	t0, 9f
+	mfc0	t0, CP0_CONFIG, 2
+	bgez	t0, 9f
+	mfc0	t0, CP0_CONFIG, 3
+	sll     t0, t0, 6   /* SC bit */
+	bgez    t0, 9f
+
+	eva_entry
+	b       0f
+9:
+	/* Assume we came from YAMON... */
+	PTR_LA	v0, 0x9fc00534	/* YAMON print */
+	lw	v0, (v0)
+	move	a0, zero
+	PTR_LA  a1, nonsc_processor
+	jal	v0
+
+	PTR_LA	v0, 0x9fc00520	/* YAMON exit */
+	lw	v0, (v0)
+	li	a0, 1
+	jal	v0
+
+1:	b	1b
+	nop
+	__INITDATA
+nonsc_processor:
+	.asciz  "EVA kernel requires a MIPS core with Segment Control implemented\n"
+	__FINIT
+#endif /* CONFIG_EVA */
+0:
 	.endm
 
 /*
diff --git a/arch/mips/mti-malta/malta-setup.c b/arch/mips/mti-malta/malta-setup.c
index c72a069..35057a8 100644
--- a/arch/mips/mti-malta/malta-setup.c
+++ b/arch/mips/mti-malta/malta-setup.c
@@ -247,6 +247,10 @@ void __init plat_mem_setup(void)
 {
 	unsigned int i;
 
+	if (config_enabled(CONFIG_EVA))
+		/* EVA has already been configured in mach-malta/kernel-init.h */
+		pr_info("Enhanced Virtual Addressing (EVA) activated\n");
+
 	mips_pcibios_init();
 
 	/* Request I/O space for devices used on the Malta board. */
-- 
1.8.5.3

[PATCH 51/58] MIPS: malta: Configure Segment Control registers for EVA boot

[Markos Chandras]

The Malta board aliases 0x80000000 - 0xffffffff to 0x00000000
- 0x7fffffff ignoring the 256 MB IO hole in 0x10000000.
The physical memory is shifted to 0x80000000 so up to 2GB
can be used. Kuseg is expanded to 3GB (due to board limitations
only 2GB can be accessed) and lowmem (kernel space) is expanded to 2GB.

The Segment Control registers are programmed as follows:

Virtual memory           Physical memory           Mapping
0x00000000 - 0x7fffffff  0x80000000 - 0xfffffffff   MUSUK (kuseg)
0x80000000 - 0x9fffffff  0x00000000 - 0x1ffffffff   MUSUK (kseg0)
0xa0000000 - 0xbf000000  0x00000000 - 0x1ffffffff   MUSUK (kseg1)
0xc0000000 - 0xdfffffff             -                 MK  (kseg2)
0xe0000000 - 0xffffffff             -                 MK  (kseg3)

The location of exception vectors remain the same since 0xbfc00000
(traditional exception base) still maps to 0x1fc00000 physical.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 .../include/asm/mach-malta/kernel-entry-init.h     | 109 ++++++++++++++++++++-
 arch/mips/mti-malta/malta-setup.c                  |   4 +
 2 files changed, 112 insertions(+), 1 deletion(-)

diff --git a/arch/mips/include/asm/mach-malta/kernel-entry-init.h b/arch/mips/include/asm/mach-malta/kernel-entry-init.h
index 0b793e7..9bace9c 100644
--- a/arch/mips/include/asm/mach-malta/kernel-entry-init.h
+++ b/arch/mips/include/asm/mach-malta/kernel-entry-init.h
@@ -5,10 +5,80 @@
  *
  * Chris Dearman (chris@mips.com)
  * Copyright (C) 2007 Mips Technologies, Inc.
+ * Copyright (C) 2014 Imagination Technologies Ltd.
  */
 #ifndef __ASM_MACH_MIPS_KERNEL_ENTRY_INIT_H
 #define __ASM_MACH_MIPS_KERNEL_ENTRY_INIT_H
 
+	/*
+	 * Prepare segments for EVA boot:
+	 *
+	 * This is in case the processor boots in legacy configuration
+	 * (SI_EVAReset is de-asserted and CONFIG5.K == 0)
+	 *
+	 * On entry, t1 is loaded with CP0_CONFIG
+	 *
+	 * ========================= Mappings =============================
+	 * Virtual memory           Physical memory           Mapping
+	 * 0x00000000 - 0x7fffffff  0x80000000 - 0xfffffffff   MUSUK (kuseg)
+	 *                          Flat 2GB physical memory
+	 *
+	 * 0x80000000 - 0x9fffffff  0x00000000 - 0x1ffffffff   MUSUK (kseg0)
+	 * 0xa0000000 - 0xbf000000  0x00000000 - 0x1ffffffff   MUSUK (kseg1)
+	 * 0xc0000000 - 0xdfffffff             -                 MK  (kseg2)
+	 * 0xe0000000 - 0xffffffff             -                 MK  (kseg3)
+	 *
+	 *
+	 * Lowmem is expanded to 2GB
+	 */
+	.macro	eva_entry
+	/*
+	 * Get Config.K0 value and use it to program
+	 * the segmentation registers
+	 */
+	andi	t1, 0x7 /* CCA */
+	move	t2, t1
+	ins	t2, t1, 16, 3
+	/* SegCtl0 */
+	li      t0, ((MIPS_SEGCFG_MK << MIPS_SEGCFG_AM_SHIFT) |		\
+		(0 << MIPS_SEGCFG_PA_SHIFT) |				\
+		(1 << MIPS_SEGCFG_EU_SHIFT)) |				\
+		(((MIPS_SEGCFG_MK << MIPS_SEGCFG_AM_SHIFT) |		\
+		(0 << MIPS_SEGCFG_PA_SHIFT) |				\
+		(1 << MIPS_SEGCFG_EU_SHIFT)) << 16)
+	or	t0, t2
+	mtc0	t0, $5, 2
+
+	/* SegCtl1 */
+	li      t0, ((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) |	\
+		(0 << MIPS_SEGCFG_PA_SHIFT) |				\
+		(2 << MIPS_SEGCFG_C_SHIFT) |				\
+		(1 << MIPS_SEGCFG_EU_SHIFT)) |				\
+		(((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) |		\
+		(0 << MIPS_SEGCFG_PA_SHIFT) |				\
+		(1 << MIPS_SEGCFG_EU_SHIFT)) << 16)
+	ins	t0, t1, 16, 3
+	mtc0	t0, $5, 3
+
+	/* SegCtl2 */
+	li	t0, ((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) |	\
+		(6 << MIPS_SEGCFG_PA_SHIFT) |				\
+		(1 << MIPS_SEGCFG_EU_SHIFT)) |				\
+		(((MIPS_SEGCFG_MUSUK << MIPS_SEGCFG_AM_SHIFT) |		\
+		(4 << MIPS_SEGCFG_PA_SHIFT) |				\
+		(1 << MIPS_SEGCFG_EU_SHIFT)) << 16)
+	or	t0, t2
+	mtc0	t0, $5, 4
+
+	jal	mips_ihb
+	mfc0    t0, $16, 5
+	li      t2, 0x40000000      /* K bit */
+	or      t0, t0, t2
+	mtc0    t0, $16, 5
+	sync
+	jal	mips_ihb
+	.endm
+
 	.macro	kernel_entry_setup
 #ifdef CONFIG_MIPS_MT_SMTC
 	mfc0	t0, CP0_CONFIG
@@ -39,8 +109,45 @@
 nonmt_processor:
 	.asciz	"SMTC kernel requires the MT ASE to run\n"
 	__FINIT
-0:
 #endif
+
+#ifdef CONFIG_EVA
+	sync
+	ehb
+
+	mfc0    t1, CP0_CONFIG
+	bgez    t1, 9f
+	mfc0	t0, CP0_CONFIG, 1
+	bgez	t0, 9f
+	mfc0	t0, CP0_CONFIG, 2
+	bgez	t0, 9f
+	mfc0	t0, CP0_CONFIG, 3
+	sll     t0, t0, 6   /* SC bit */
+	bgez    t0, 9f
+
+	eva_entry
+	b       0f
+9:
+	/* Assume we came from YAMON... */
+	PTR_LA	v0, 0x9fc00534	/* YAMON print */
+	lw	v0, (v0)
+	move	a0, zero
+	PTR_LA  a1, nonsc_processor
+	jal	v0
+
+	PTR_LA	v0, 0x9fc00520	/* YAMON exit */
+	lw	v0, (v0)
+	li	a0, 1
+	jal	v0
+
+1:	b	1b
+	nop
+	__INITDATA
+nonsc_processor:
+	.asciz  "EVA kernel requires a MIPS core with Segment Control implemented\n"
+	__FINIT
+#endif /* CONFIG_EVA */
+0:
 	.endm
 
 /*
diff --git a/arch/mips/mti-malta/malta-setup.c b/arch/mips/mti-malta/malta-setup.c
index c72a069..35057a8 100644
--- a/arch/mips/mti-malta/malta-setup.c
+++ b/arch/mips/mti-malta/malta-setup.c
@@ -247,6 +247,10 @@ void __init plat_mem_setup(void)
 {
 	unsigned int i;
 
+	if (config_enabled(CONFIG_EVA))
+		/* EVA has already been configured in mach-malta/kernel-init.h */
+		pr_info("Enhanced Virtual Addressing (EVA) activated\n");
+
 	mips_pcibios_init();
 
 	/* Request I/O space for devices used on the Malta board. */
-- 
1.8.5.3

[PATCH 52/58] MIPS: malta: spaces.h: Add spaces.h file for Malta (EVA)

[Markos Chandras]

Add a spaces.h file for Malta to override certain memory macros
when operating in EVA mode.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/mach-malta/spaces.h | 46 +++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
 create mode 100644 arch/mips/include/asm/mach-malta/spaces.h

diff --git a/arch/mips/include/asm/mach-malta/spaces.h b/arch/mips/include/asm/mach-malta/spaces.h
new file mode 100644
index 0000000..d7e5497
--- /dev/null
+++ b/arch/mips/include/asm/mach-malta/spaces.h
@@ -0,0 +1,46 @@
+/*
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License.  See the file "COPYING" in the main directory of this archive
+ * for more details.
+ *
+ * Copyright (C) 2014 Imagination Technologies Ltd.
+ */
+
+#ifndef _ASM_MALTA_SPACES_H
+#define _ASM_MALTA_SPACES_H
+
+#ifdef CONFIG_EVA
+
+/*
+ * Traditional Malta Board Memory Map for EVA
+ *
+ * 0x00000000 - 0x0fffffff: 1st RAM region, 256MB
+ * 0x10000000 - 0x1bffffff: GIC and CPC Control Registers
+ * 0x1c000000 - 0x1fffffff: I/O And Flash
+ * 0x20000000 - 0x7fffffff: 2nd RAM region, 1.5GB
+ * 0x80000000 - 0xffffffff: Physical memory aliases to 0x0 (2GB)
+ *
+ * The kernel is still located in 0x80000000(kseg0). However,
+ * the physical mask has been shifted to 0x80000000 which exploits the alias
+ * on the Malta board. As a result of which, we override the __pa_symbol
+ * to peform direct mapping from virtual to physical addresses. In other
+ * words, the 0x80000000 virtual address maps to 0x80000000 physical address
+ * which in turn aliases to 0x0. We do this in order to be able to use a flat
+ * 2GB of memory (0x80000000 - 0xffffffff) so we can avoid the I/O hole in
+ * 0x10000000 - 0x1fffffff.
+ * The last 64KB of physical memory are reserved for correct HIGHMEM
+ * macros arithmetics.
+ *
+ */
+
+#define PAGE_OFFSET	_AC(0x0, UL)
+#define PHYS_OFFSET	_AC(0x80000000, UL)
+#define HIGHMEM_START	_AC(0xffff0000, UL)
+
+#define __pa_symbol(x)	(RELOC_HIDE((unsigned long)(x), 0))
+
+#endif /* CONFIG_EVA */
+
+#include <asm/mach-generic/spaces.h>
+
+#endif /* _ASM_MALTA_SPACES_H */
-- 
1.8.5.3

[PATCH 52/58] MIPS: malta: spaces.h: Add spaces.h file for Malta (EVA)

[Markos Chandras]

Add a spaces.h file for Malta to override certain memory macros
when operating in EVA mode.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/mach-malta/spaces.h | 46 +++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
 create mode 100644 arch/mips/include/asm/mach-malta/spaces.h

diff --git a/arch/mips/include/asm/mach-malta/spaces.h b/arch/mips/include/asm/mach-malta/spaces.h
new file mode 100644
index 0000000..d7e5497
--- /dev/null
+++ b/arch/mips/include/asm/mach-malta/spaces.h
@@ -0,0 +1,46 @@
+/*
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License.  See the file "COPYING" in the main directory of this archive
+ * for more details.
+ *
+ * Copyright (C) 2014 Imagination Technologies Ltd.
+ */
+
+#ifndef _ASM_MALTA_SPACES_H
+#define _ASM_MALTA_SPACES_H
+
+#ifdef CONFIG_EVA
+
+/*
+ * Traditional Malta Board Memory Map for EVA
+ *
+ * 0x00000000 - 0x0fffffff: 1st RAM region, 256MB
+ * 0x10000000 - 0x1bffffff: GIC and CPC Control Registers
+ * 0x1c000000 - 0x1fffffff: I/O And Flash
+ * 0x20000000 - 0x7fffffff: 2nd RAM region, 1.5GB
+ * 0x80000000 - 0xffffffff: Physical memory aliases to 0x0 (2GB)
+ *
+ * The kernel is still located in 0x80000000(kseg0). However,
+ * the physical mask has been shifted to 0x80000000 which exploits the alias
+ * on the Malta board. As a result of which, we override the __pa_symbol
+ * to peform direct mapping from virtual to physical addresses. In other
+ * words, the 0x80000000 virtual address maps to 0x80000000 physical address
+ * which in turn aliases to 0x0. We do this in order to be able to use a flat
+ * 2GB of memory (0x80000000 - 0xffffffff) so we can avoid the I/O hole in
+ * 0x10000000 - 0x1fffffff.
+ * The last 64KB of physical memory are reserved for correct HIGHMEM
+ * macros arithmetics.
+ *
+ */
+
+#define PAGE_OFFSET	_AC(0x0, UL)
+#define PHYS_OFFSET	_AC(0x80000000, UL)
+#define HIGHMEM_START	_AC(0xffff0000, UL)
+
+#define __pa_symbol(x)	(RELOC_HIDE((unsigned long)(x), 0))
+
+#endif /* CONFIG_EVA */
+
+#include <asm/mach-generic/spaces.h>
+
+#endif /* _ASM_MALTA_SPACES_H */
-- 
1.8.5.3

[PATCH 53/58] MIPS: malta: malta-memory: Add support for the 'ememsize' variable

[Markos Chandras]

The 'ememsize' variable is used to denote the real RAM which is
present on the Malta board. This is different compared to 'memsize'
which is capped to 256MB. The 'ememsize' is used to get the actual
physical memory when setting up the Malta memory layout. This only
makes sense in case the core operates in the EVA mode, and it's
ignored otherwise.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/fw/fw.h      |  2 +-
 arch/mips/mti-malta/malta-memory.c | 36 +++++++++++++++++++++++++++---------
 2 files changed, 28 insertions(+), 10 deletions(-)

diff --git a/arch/mips/include/asm/fw/fw.h b/arch/mips/include/asm/fw/fw.h
index d6c50a7..f3e6978 100644
--- a/arch/mips/include/asm/fw/fw.h
+++ b/arch/mips/include/asm/fw/fw.h
@@ -38,7 +38,7 @@ extern int *_fw_envp;
 
 extern void fw_init_cmdline(void);
 extern char *fw_getcmdline(void);
-extern fw_memblock_t *fw_getmdesc(void);
+extern fw_memblock_t *fw_getmdesc(int);
 extern void fw_meminit(void);
 extern char *fw_getenv(char *name);
 extern unsigned long fw_getenvl(char *name);
diff --git a/arch/mips/mti-malta/malta-memory.c b/arch/mips/mti-malta/malta-memory.c
index 1f73d63..cd04008 100644
--- a/arch/mips/mti-malta/malta-memory.c
+++ b/arch/mips/mti-malta/malta-memory.c
@@ -24,22 +24,30 @@ static fw_memblock_t mdesc[FW_MAX_MEMBLOCKS];
 /* determined physical memory size, not overridden by command line args	 */
 unsigned long physical_memsize = 0L;
 
-fw_memblock_t * __init fw_getmdesc(void)
+fw_memblock_t * __init fw_getmdesc(int eva)
 {
-	char *memsize_str, *ptr;
-	unsigned int memsize;
+	char *memsize_str, *ememsize_str __maybe_unused = NULL, *ptr;
+	unsigned long memsize, ememsize __maybe_unused = 0;
 	static char cmdline[COMMAND_LINE_SIZE] __initdata;
-	long val;
 	int tmp;
 
 	/* otherwise look in the environment */
+
 	memsize_str = fw_getenv("memsize");
-	if (!memsize_str) {
+	if (memsize_str)
+		tmp = kstrtol(memsize_str, 0, &memsize);
+	if (eva) {
+	/* Look for ememsize for EVA */
+		ememsize_str = fw_getenv("ememsize");
+		if (ememsize_str)
+			tmp = kstrtol(ememsize_str, 0, &ememsize);
+	}
+	if (!memsize && !ememsize) {
 		pr_warn("memsize not set in YAMON, set to default (32Mb)\n");
 		physical_memsize = 0x02000000;
 	} else {
-		tmp = kstrtol(memsize_str, 0, &val);
-		physical_memsize = (unsigned long)val;
+		/* If ememsize is set, then set physical_memsize to that */
+		physical_memsize = ememsize ? : memsize;
 	}
 
 #ifdef CONFIG_CPU_BIG_ENDIAN
@@ -54,12 +62,22 @@ fw_memblock_t * __init fw_getmdesc(void)
 	ptr = strstr(cmdline, "memsize=");
 	if (ptr && (ptr != cmdline) && (*(ptr - 1) != ' '))
 		ptr = strstr(ptr, " memsize=");
+	/* And now look for ememsize */
+	if (eva) {
+		ptr = strstr(cmdline, "ememsize=");
+		if (ptr && (ptr != cmdline) && (*(ptr - 1) != ' '))
+			ptr = strstr(ptr, " ememsize=");
+	}
 
 	if (ptr)
-		memsize = memparse(ptr + 8, &ptr);
+		memsize = memparse(ptr + 8 + (eva ? 1 : 0), &ptr);
 	else
 		memsize = physical_memsize;
 
+	/* Last 64K for HIGHMEM arithmetics */
+	if (memsize > 0x7fff0000)
+		memsize = 0x7fff0000;
+
 	memset(mdesc, 0, sizeof(mdesc));
 
 	mdesc[0].type = fw_dontuse;
@@ -109,7 +127,7 @@ void __init fw_meminit(void)
 {
 	fw_memblock_t *p;
 
-	p = fw_getmdesc();
+	p = fw_getmdesc(config_enabled(CONFIG_EVA));
 
 	while (p->size) {
 		long type;
-- 
1.8.5.3

[PATCH 53/58] MIPS: malta: malta-memory: Add support for the 'ememsize' variable

[Markos Chandras]

The 'ememsize' variable is used to denote the real RAM which is
present on the Malta board. This is different compared to 'memsize'
which is capped to 256MB. The 'ememsize' is used to get the actual
physical memory when setting up the Malta memory layout. This only
makes sense in case the core operates in the EVA mode, and it's
ignored otherwise.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/fw/fw.h      |  2 +-
 arch/mips/mti-malta/malta-memory.c | 36 +++++++++++++++++++++++++++---------
 2 files changed, 28 insertions(+), 10 deletions(-)

diff --git a/arch/mips/include/asm/fw/fw.h b/arch/mips/include/asm/fw/fw.h
index d6c50a7..f3e6978 100644
--- a/arch/mips/include/asm/fw/fw.h
+++ b/arch/mips/include/asm/fw/fw.h
@@ -38,7 +38,7 @@ extern int *_fw_envp;
 
 extern void fw_init_cmdline(void);
 extern char *fw_getcmdline(void);
-extern fw_memblock_t *fw_getmdesc(void);
+extern fw_memblock_t *fw_getmdesc(int);
 extern void fw_meminit(void);
 extern char *fw_getenv(char *name);
 extern unsigned long fw_getenvl(char *name);
diff --git a/arch/mips/mti-malta/malta-memory.c b/arch/mips/mti-malta/malta-memory.c
index 1f73d63..cd04008 100644
--- a/arch/mips/mti-malta/malta-memory.c
+++ b/arch/mips/mti-malta/malta-memory.c
@@ -24,22 +24,30 @@ static fw_memblock_t mdesc[FW_MAX_MEMBLOCKS];
 /* determined physical memory size, not overridden by command line args	 */
 unsigned long physical_memsize = 0L;
 
-fw_memblock_t * __init fw_getmdesc(void)
+fw_memblock_t * __init fw_getmdesc(int eva)
 {
-	char *memsize_str, *ptr;
-	unsigned int memsize;
+	char *memsize_str, *ememsize_str __maybe_unused = NULL, *ptr;
+	unsigned long memsize, ememsize __maybe_unused = 0;
 	static char cmdline[COMMAND_LINE_SIZE] __initdata;
-	long val;
 	int tmp;
 
 	/* otherwise look in the environment */
+
 	memsize_str = fw_getenv("memsize");
-	if (!memsize_str) {
+	if (memsize_str)
+		tmp = kstrtol(memsize_str, 0, &memsize);
+	if (eva) {
+	/* Look for ememsize for EVA */
+		ememsize_str = fw_getenv("ememsize");
+		if (ememsize_str)
+			tmp = kstrtol(ememsize_str, 0, &ememsize);
+	}
+	if (!memsize && !ememsize) {
 		pr_warn("memsize not set in YAMON, set to default (32Mb)\n");
 		physical_memsize = 0x02000000;
 	} else {
-		tmp = kstrtol(memsize_str, 0, &val);
-		physical_memsize = (unsigned long)val;
+		/* If ememsize is set, then set physical_memsize to that */
+		physical_memsize = ememsize ? : memsize;
 	}
 
 #ifdef CONFIG_CPU_BIG_ENDIAN
@@ -54,12 +62,22 @@ fw_memblock_t * __init fw_getmdesc(void)
 	ptr = strstr(cmdline, "memsize=");
 	if (ptr && (ptr != cmdline) && (*(ptr - 1) != ' '))
 		ptr = strstr(ptr, " memsize=");
+	/* And now look for ememsize */
+	if (eva) {
+		ptr = strstr(cmdline, "ememsize=");
+		if (ptr && (ptr != cmdline) && (*(ptr - 1) != ' '))
+			ptr = strstr(ptr, " ememsize=");
+	}
 
 	if (ptr)
-		memsize = memparse(ptr + 8, &ptr);
+		memsize = memparse(ptr + 8 + (eva ? 1 : 0), &ptr);
 	else
 		memsize = physical_memsize;
 
+	/* Last 64K for HIGHMEM arithmetics */
+	if (memsize > 0x7fff0000)
+		memsize = 0x7fff0000;
+
 	memset(mdesc, 0, sizeof(mdesc));
 
 	mdesc[0].type = fw_dontuse;
@@ -109,7 +127,7 @@ void __init fw_meminit(void)
 {
 	fw_memblock_t *p;
 
-	p = fw_getmdesc();
+	p = fw_getmdesc(config_enabled(CONFIG_EVA));
 
 	while (p->size) {
 		long type;
-- 
1.8.5.3

[PATCH 54/58] MIPS: malta: malta-memory: Use the PHYS_OFFSET to build the memory map

[Markos Chandras]

PHYS_OFFSET is used to denote the physical start address of the
first bank of RAM. When the Malta board is in EVA mode, the physical
start address of RAM is shifted to 0x80000000 so it's necessary to use
this macro in order to make the code EVA agnostic.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/mti-malta/malta-memory.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/arch/mips/mti-malta/malta-memory.c b/arch/mips/mti-malta/malta-memory.c
index cd04008..9235aee8 100644
--- a/arch/mips/mti-malta/malta-memory.c
+++ b/arch/mips/mti-malta/malta-memory.c
@@ -81,11 +81,11 @@ fw_memblock_t * __init fw_getmdesc(int eva)
 	memset(mdesc, 0, sizeof(mdesc));
 
 	mdesc[0].type = fw_dontuse;
-	mdesc[0].base = 0x00000000;
+	mdesc[0].base = PHYS_OFFSET;
 	mdesc[0].size = 0x00001000;
 
 	mdesc[1].type = fw_code;
-	mdesc[1].base = 0x00001000;
+	mdesc[1].base = mdesc[0].base + 0x00001000UL;
 	mdesc[1].size = 0x000ef000;
 
 	/*
@@ -96,17 +96,17 @@ fw_memblock_t * __init fw_getmdesc(int eva)
 	 * devices.
 	 */
 	mdesc[2].type = fw_dontuse;
-	mdesc[2].base = 0x000f0000;
+	mdesc[2].base = mdesc[0].base + 0x000f0000UL;
 	mdesc[2].size = 0x00010000;
 
 	mdesc[3].type = fw_dontuse;
-	mdesc[3].base = 0x00100000;
+	mdesc[3].base = mdesc[0].base + 0x00100000UL;
 	mdesc[3].size = CPHYSADDR(PFN_ALIGN((unsigned long)&_end)) -
-		mdesc[3].base;
+		0x00100000UL;
 
 	mdesc[4].type = fw_free;
-	mdesc[4].base = CPHYSADDR(PFN_ALIGN(&_end));
-	mdesc[4].size = memsize - mdesc[4].base;
+	mdesc[4].base = mdesc[0].base + CPHYSADDR(PFN_ALIGN(&_end));
+	mdesc[4].size = memsize - CPHYSADDR(mdesc[4].base);
 
 	return &mdesc[0];
 }
-- 
1.8.5.3

[PATCH 54/58] MIPS: malta: malta-memory: Use the PHYS_OFFSET to build the memory map

[Markos Chandras]

PHYS_OFFSET is used to denote the physical start address of the
first bank of RAM. When the Malta board is in EVA mode, the physical
start address of RAM is shifted to 0x80000000 so it's necessary to use
this macro in order to make the code EVA agnostic.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/mti-malta/malta-memory.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/arch/mips/mti-malta/malta-memory.c b/arch/mips/mti-malta/malta-memory.c
index cd04008..9235aee8 100644
--- a/arch/mips/mti-malta/malta-memory.c
+++ b/arch/mips/mti-malta/malta-memory.c
@@ -81,11 +81,11 @@ fw_memblock_t * __init fw_getmdesc(int eva)
 	memset(mdesc, 0, sizeof(mdesc));
 
 	mdesc[0].type = fw_dontuse;
-	mdesc[0].base = 0x00000000;
+	mdesc[0].base = PHYS_OFFSET;
 	mdesc[0].size = 0x00001000;
 
 	mdesc[1].type = fw_code;
-	mdesc[1].base = 0x00001000;
+	mdesc[1].base = mdesc[0].base + 0x00001000UL;
 	mdesc[1].size = 0x000ef000;
 
 	/*
@@ -96,17 +96,17 @@ fw_memblock_t * __init fw_getmdesc(int eva)
 	 * devices.
 	 */
 	mdesc[2].type = fw_dontuse;
-	mdesc[2].base = 0x000f0000;
+	mdesc[2].base = mdesc[0].base + 0x000f0000UL;
 	mdesc[2].size = 0x00010000;
 
 	mdesc[3].type = fw_dontuse;
-	mdesc[3].base = 0x00100000;
+	mdesc[3].base = mdesc[0].base + 0x00100000UL;
 	mdesc[3].size = CPHYSADDR(PFN_ALIGN((unsigned long)&_end)) -
-		mdesc[3].base;
+		0x00100000UL;
 
 	mdesc[4].type = fw_free;
-	mdesc[4].base = CPHYSADDR(PFN_ALIGN(&_end));
-	mdesc[4].size = memsize - mdesc[4].base;
+	mdesc[4].base = mdesc[0].base + CPHYSADDR(PFN_ALIGN(&_end));
+	mdesc[4].size = memsize - CPHYSADDR(mdesc[4].base);
 
 	return &mdesc[0];
 }
-- 
1.8.5.3

[PATCH 55/58] MIPS: malta: malta-memory: Add free_init_pages_eva() callback

[Markos Chandras]

Use a Malta specific function to free the init section once the
kernel has booted. When operating in EVA mode, the physical memory
is shifted to 0x80000000. Kernel is loaded into 0x80000000 (virtual)
so the offset between physical and virtual addresses is 0.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/mti-malta/malta-memory.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/arch/mips/mti-malta/malta-memory.c b/arch/mips/mti-malta/malta-memory.c
index 9235aee8..6d0f4ab 100644
--- a/arch/mips/mti-malta/malta-memory.c
+++ b/arch/mips/mti-malta/malta-memory.c
@@ -111,6 +111,12 @@ fw_memblock_t * __init fw_getmdesc(int eva)
 	return &mdesc[0];
 }
 
+static void free_init_pages_eva_malta(void *begin, void *end)
+{
+	free_init_pages("unused kernel", __pa_symbol((unsigned long *)begin),
+			__pa_symbol((unsigned long *)end));
+}
+
 static int __init fw_memtype_classify(unsigned int type)
 {
 	switch (type) {
@@ -128,6 +134,8 @@ void __init fw_meminit(void)
 	fw_memblock_t *p;
 
 	p = fw_getmdesc(config_enabled(CONFIG_EVA));
+	free_init_pages_eva = (config_enabled(CONFIG_EVA) ?
+			       free_init_pages_eva_malta : NULL);
 
 	while (p->size) {
 		long type;
-- 
1.8.5.3

[PATCH 55/58] MIPS: malta: malta-memory: Add free_init_pages_eva() callback

[Markos Chandras]

Use a Malta specific function to free the init section once the
kernel has booted. When operating in EVA mode, the physical memory
is shifted to 0x80000000. Kernel is loaded into 0x80000000 (virtual)
so the offset between physical and virtual addresses is 0.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/mti-malta/malta-memory.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/arch/mips/mti-malta/malta-memory.c b/arch/mips/mti-malta/malta-memory.c
index 9235aee8..6d0f4ab 100644
--- a/arch/mips/mti-malta/malta-memory.c
+++ b/arch/mips/mti-malta/malta-memory.c
@@ -111,6 +111,12 @@ fw_memblock_t * __init fw_getmdesc(int eva)
 	return &mdesc[0];
 }
 
+static void free_init_pages_eva_malta(void *begin, void *end)
+{
+	free_init_pages("unused kernel", __pa_symbol((unsigned long *)begin),
+			__pa_symbol((unsigned long *)end));
+}
+
 static int __init fw_memtype_classify(unsigned int type)
 {
 	switch (type) {
@@ -128,6 +134,8 @@ void __init fw_meminit(void)
 	fw_memblock_t *p;
 
 	p = fw_getmdesc(config_enabled(CONFIG_EVA));
+	free_init_pages_eva = (config_enabled(CONFIG_EVA) ?
+			       free_init_pages_eva_malta : NULL);
 
 	while (p->size) {
 		long type;
-- 
1.8.5.3

[PATCH 56/58] MIPS: malta: malta-init: Fix System Controller memory mapping for EVA

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

Shift System Controller memory mapping to 0x80000000

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/mti-malta/malta-init.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/arch/mips/mti-malta/malta-init.c b/arch/mips/mti-malta/malta-init.c
index fcebfce..deb1d7b 100644
--- a/arch/mips/mti-malta/malta-init.c
+++ b/arch/mips/mti-malta/malta-init.c
@@ -238,10 +238,23 @@ mips_pci_controller:
 			  MSC01_PCI_SWAP_BYTESWAP << MSC01_PCI_SWAP_MEM_SHF |
 			  MSC01_PCI_SWAP_BYTESWAP << MSC01_PCI_SWAP_BAR0_SHF);
 #endif
+#ifndef CONFIG_EVA
 		/* Fix up target memory mapping.  */
 		MSC_READ(MSC01_PCI_BAR0, mask);
 		MSC_WRITE(MSC01_PCI_P2SCMSKL, mask & MSC01_PCI_BAR0_SIZE_MSK);
+#else
+		/*
+		 * Setup the Malta max (2GB) memory for PCI DMA in host bridge
+		 * in transparent addressing mode, starting from 0x80000000.
+		 */
+		mask = PHYS_OFFSET | (1<<3);
+		MSC_WRITE(MSC01_PCI_BAR0, mask);
 
+		mask = PHYS_OFFSET;
+		MSC_WRITE(MSC01_PCI_HEAD4, mask);
+		MSC_WRITE(MSC01_PCI_P2SCMSKL, mask);
+		MSC_WRITE(MSC01_PCI_P2SCMAPL, mask);
+#endif
 		/* Don't handle target retries indefinitely.  */
 		if ((data & MSC01_PCI_CFG_MAXRTRY_MSK) ==
 		    MSC01_PCI_CFG_MAXRTRY_MSK)
-- 
1.8.5.3

[PATCH 56/58] MIPS: malta: malta-init: Fix System Controller memory mapping for EVA

[Markos Chandras]

From: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>

Shift System Controller memory mapping to 0x80000000

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/mti-malta/malta-init.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/arch/mips/mti-malta/malta-init.c b/arch/mips/mti-malta/malta-init.c
index fcebfce..deb1d7b 100644
--- a/arch/mips/mti-malta/malta-init.c
+++ b/arch/mips/mti-malta/malta-init.c
@@ -238,10 +238,23 @@ mips_pci_controller:
 			  MSC01_PCI_SWAP_BYTESWAP << MSC01_PCI_SWAP_MEM_SHF |
 			  MSC01_PCI_SWAP_BYTESWAP << MSC01_PCI_SWAP_BAR0_SHF);
 #endif
+#ifndef CONFIG_EVA
 		/* Fix up target memory mapping.  */
 		MSC_READ(MSC01_PCI_BAR0, mask);
 		MSC_WRITE(MSC01_PCI_P2SCMSKL, mask & MSC01_PCI_BAR0_SIZE_MSK);
+#else
+		/*
+		 * Setup the Malta max (2GB) memory for PCI DMA in host bridge
+		 * in transparent addressing mode, starting from 0x80000000.
+		 */
+		mask = PHYS_OFFSET | (1<<3);
+		MSC_WRITE(MSC01_PCI_BAR0, mask);
 
+		mask = PHYS_OFFSET;
+		MSC_WRITE(MSC01_PCI_HEAD4, mask);
+		MSC_WRITE(MSC01_PCI_P2SCMSKL, mask);
+		MSC_WRITE(MSC01_PCI_P2SCMAPL, mask);
+#endif
 		/* Don't handle target retries indefinitely.  */
 		if ((data & MSC01_PCI_CFG_MAXRTRY_MSK) ==
 		    MSC01_PCI_CFG_MAXRTRY_MSK)
-- 
1.8.5.3

[PATCH 57/58] MIPS: malta: Add support for SMP EVA

[Markos Chandras]

Allow secondary cores to program their segment control registers
during smp bootstrap code. This enables EVA on Malta SMP
configurations

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/mach-malta/kernel-entry-init.h | 6 ++++++
 arch/mips/kernel/head.S                              | 2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/arch/mips/include/asm/mach-malta/kernel-entry-init.h b/arch/mips/include/asm/mach-malta/kernel-entry-init.h
index 9bace9c..7c5e17a 100644
--- a/arch/mips/include/asm/mach-malta/kernel-entry-init.h
+++ b/arch/mips/include/asm/mach-malta/kernel-entry-init.h
@@ -154,6 +154,12 @@ nonsc_processor:
  * Do SMP slave processor setup necessary before we can safely execute C code.
  */
 	.macro	smp_slave_setup
+#ifdef CONFIG_EVA
+	sync
+	ehb
+	mfc0    t1, CP0_CONFIG
+	eva_entry
+#endif
 	.endm
 
 #endif /* __ASM_MACH_MIPS_KERNEL_ENTRY_INIT_H */
diff --git a/arch/mips/kernel/head.S b/arch/mips/kernel/head.S
index 7b6a5b3..e712dcf 100644
--- a/arch/mips/kernel/head.S
+++ b/arch/mips/kernel/head.S
@@ -175,8 +175,8 @@ NESTED(smp_bootstrap, 16, sp)
 	DMT	10	# dmt t2 /* t0, t1 are used by CLI and setup_c0_status() */
 	jal	mips_ihb
 #endif /* CONFIG_MIPS_MT_SMTC */
-	setup_c0_status_sec
 	smp_slave_setup
+	setup_c0_status_sec
 #ifdef CONFIG_MIPS_MT_SMTC
 	andi	t2, t2, VPECONTROL_TE
 	beqz	t2, 2f
-- 
1.8.5.3

[PATCH 57/58] MIPS: malta: Add support for SMP EVA

[Markos Chandras]

Allow secondary cores to program their segment control registers
during smp bootstrap code. This enables EVA on Malta SMP
configurations

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/include/asm/mach-malta/kernel-entry-init.h | 6 ++++++
 arch/mips/kernel/head.S                              | 2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/arch/mips/include/asm/mach-malta/kernel-entry-init.h b/arch/mips/include/asm/mach-malta/kernel-entry-init.h
index 9bace9c..7c5e17a 100644
--- a/arch/mips/include/asm/mach-malta/kernel-entry-init.h
+++ b/arch/mips/include/asm/mach-malta/kernel-entry-init.h
@@ -154,6 +154,12 @@ nonsc_processor:
  * Do SMP slave processor setup necessary before we can safely execute C code.
  */
 	.macro	smp_slave_setup
+#ifdef CONFIG_EVA
+	sync
+	ehb
+	mfc0    t1, CP0_CONFIG
+	eva_entry
+#endif
 	.endm
 
 #endif /* __ASM_MACH_MIPS_KERNEL_ENTRY_INIT_H */
diff --git a/arch/mips/kernel/head.S b/arch/mips/kernel/head.S
index 7b6a5b3..e712dcf 100644
--- a/arch/mips/kernel/head.S
+++ b/arch/mips/kernel/head.S
@@ -175,8 +175,8 @@ NESTED(smp_bootstrap, 16, sp)
 	DMT	10	# dmt t2 /* t0, t1 are used by CLI and setup_c0_status() */
 	jal	mips_ihb
 #endif /* CONFIG_MIPS_MT_SMTC */
-	setup_c0_status_sec
 	smp_slave_setup
+	setup_c0_status_sec
 #ifdef CONFIG_MIPS_MT_SMTC
 	andi	t2, t2, VPECONTROL_TE
 	beqz	t2, 2f
-- 
1.8.5.3

[PATCH 58/58] MIPS: Enable MIPS 3.5 features on Malta

[Markos Chandras]

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index 4230c7a..607792b 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -318,6 +318,7 @@ config MIPS_MALTA
 	select SWAP_IO_SPACE
 	select SYS_HAS_CPU_MIPS32_R1
 	select SYS_HAS_CPU_MIPS32_R2
+	select SYS_HAS_CPU_MIPS32_R3_5
 	select SYS_HAS_CPU_MIPS64_R1
 	select SYS_HAS_CPU_MIPS64_R2
 	select SYS_HAS_CPU_NEVADA
-- 
1.8.5.3

[PATCH 58/58] MIPS: Enable MIPS 3.5 features on Malta

[Markos Chandras]

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
 arch/mips/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index 4230c7a..607792b 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -318,6 +318,7 @@ config MIPS_MALTA
 	select SWAP_IO_SPACE
 	select SYS_HAS_CPU_MIPS32_R1
 	select SYS_HAS_CPU_MIPS32_R2
+	select SYS_HAS_CPU_MIPS32_R3_5
 	select SYS_HAS_CPU_MIPS64_R1
 	select SYS_HAS_CPU_MIPS64_R2
 	select SYS_HAS_CPU_NEVADA
-- 
1.8.5.3

Re: [PATCH 20/58] MIPS: lib: memset: Whitespace fixes

[Markos Chandras]

On 01/27/2014 09:43 PM, Sergei Shtylyov wrote:
> ....
> Hello.
>
> On 01/27/2014 11:19 PM, Markos Chandras wrote:
>
>> Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
>
>     You're spoiling intentionally marked delay slots, not fixing
> anything here. This is a convention.
>
> WBR, Sergei
>
>
Hi Sergei,

Apologies. I was not aware of this convention. I will drop this patch 
and fix the next one. I will also double check if I have made the same 
mistake in any of the other patches.

-- 
markos

Re: [PATCH 20/58] MIPS: lib: memset: Whitespace fixes

[Markos Chandras]

On 01/27/2014 09:43 PM, Sergei Shtylyov wrote:
> ....
> Hello.
>
> On 01/27/2014 11:19 PM, Markos Chandras wrote:
>
>> Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
>
>     You're spoiling intentionally marked delay slots, not fixing
> anything here. This is a convention.
>
> WBR, Sergei
>
>
Hi Sergei,

Apologies. I was not aware of this convention. I will drop this patch 
and fix the next one. I will also double check if I have made the same 
mistake in any of the other patches.

-- 
markos

Re: [PATCH 20/58] MIPS: lib: memset: Whitespace fixes

[Sergei Shtylyov]

Hello.

On 01/27/2014 11:19 PM, Markos Chandras wrote:

> Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>

    You're spoiling intentionally marked delay slots, not fixing anything 
here. This is a convention.

WBR, Sergei

Re: [PATCH 00/58] Add support for Enhanced Virtual Addressing

[David Daney]

This whole thing seems very messy.  I see a couple of problems:

1) There are if(CONFIG_EVA) ... else ... endif  all over the place.  It 
is very ugly.

2) You cannot have a signel kernel with both EVA and non-EVA support.

Have you considered just tagging all the instructions that touch the 
user address space, and patching them at system boot with their EVA 
equivalents if EVA support is desired?

You might even be able to use the existing exception tables and not have 
to add a special table for EVA.  The instructions that can trap are 
exactly those where the EVA version should be used.

David Daney



On 01/27/2014 12:18 PM, Markos Chandras wrote:
> Hi,
>
> This patchset adds support for the Enhanced Virtual Addressing feature
> of the MIPS 3.5 ISA currently present in the *Aptiv processors.
> In Release 3, the memory architecture defines the partitioning of memory
> space into fixed‐size segments, whose access properties and cacheability
> attributes are programmable. The "mapped" and "unmapped" segment‐access
> properties can now be modified so as to extend system memory space
> (kseg0/kseg1) to include unmapped access to user space (xkseg), a
> feature called Enhanced Virtual Addressing (EVA). Instructions that
> facilitate system read/write access to xkseg have also been added to the
> R3 ISA (lwe, swe, lbue, etc).
>
> This patchset also adds EVA support for the Malta board. The unmapped
> address space (kseg0/kseg1) has been extended to 2GB.
>
> If you want to try it, you can use the 'eva-next' branch on
> the following git repository
>
> https://github.com/hwoarang/linux.git eva-next
>
> The patchset applies on top of the upstream-sfr/mips-for-linux-next tree.
>
> Leonid Yegoshin (18):
>    MIPS: Kconfig: Add Kconfig symbols for EVA support
>    MIPS: asm: Add prefetch instruction for EVA
>    MIPS: uapi: inst: Add new EVA opcodes
>    MIPS: uapi: inst: Add instruction format for SPECIAL3 instructions
>    MIPS: traps: Set correct address limit for breakpoints and traps
>    MIPS: asm: uaccess: Disable unaligned access macros for EVA
>    MIPS: kernel: unaligned: Handle unaligned accesses for EVA
>    MIPS: asm: checksum: Split kernel and user copy operations
>    MIPS: asm: checksum: Add MIPS specific csum_and_copy_from_user
>      function
>    MIPS: kernel: signal: Prevent save/restore FPU context in user memory
>    MIPS: asm: r4kcache: Build flushing code for instruction cache
>    MIPS: asm: r4kcache: Add protected cache operation for EVA
>    MIPS: asm: r4kcache: Add EVA cache flushing functions
>    MIPS: kernel: {ftrace,kgdb}: Set correct address limit for cache
>      flushes
>    MIPS: asm: page: Allow __pa_symbol overrides
>    MIPS: mm: c-r4k: Build EVA {d,i}cache flushing functions
>    MIPS: mm: c-r4k: Flush scache to avoid cache aliases
>    MIPS: malta: malta-init: Fix System Controller memory mapping for EVA
>
> Markos Chandras (40):
>    MIPS: asm: Add wrappers for EVA/non-EVA instructions
>    MIPS: futex: Add EVA support for futex operations
>    MIPS: kernel: scall32-o32: Use EVA wrappers to fetch syscall arguments
>    MIPS: kernel: traps: Whitespace clean up
>    MIPS: lib: strnlen_user: Use macro to build the strnlen_user symbol
>    MIPS: lib: strnlen_user: Add EVA support
>    MIPS: lib: strlen_user: Use macro to build the strlen_user symbol
>    MIPS: lib: strlen_user: Add EVA support
>    MIPS: lib: strncpy_user: Use macro to build the strncpy_from_user
>      symbol
>    MIPS: lib: strncpy_user: Add EVA support
>    MIPS: lib: memcpy: Merge EXC and load/store macros
>    MIPS: lib: memcpy: Split source and destination prefetch macros
>    MIPS: lib: memcpy: Use macro to build the copy_user code
>    MIPS: lib: memcpy: Add EVA support
>    MIPS: lib: memset: Whitespace fixes
>    MIPS: lib: memset: Use macro to build the __bzero symbol
>    MIPS: lib: memset: Add EVA support for the __bzero function.
>    MIPS: asm: uaccess: Add instruction argument to __{put,get}_user_asm
>    MIPS: asm: uaccess: Move duplicated code to common function
>    MIPS: asm: uaccess: Use EVA instructions wrappers
>    MIPS: asm: uaccess: Rename {get,put}_user_asm macros
>    MIPS: asm: uaccess: Add EVA support to copy_{in, to,from}_user
>    MIPS: asm: uaccess: Add EVA support for str*_user operations
>    MIPS: kernel: unaligned: Add EVA instruction wrappers
>    MIPS: checksum: Split the 'copy_user' symbol
>    MIPS: lib: csum_partial: Merge EXC and load/store macros
>    MIPS: lib: csum_partial: Add macro to build csum_partial symbols
>    MIPS: lib: csum_partial: Add EVA support
>    MIPS: asm: cpu: Add cpu flag for Enhanced Virtual Addressing
>    MIPS: kernel: cpu-probe: Enable EVA option on supported cores
>    MIPS: kernel: proc: Add EVA to the list of CPU features
>    MIPS: mm: init: Add free_init_pages() callback for EVA
>    MIPS: mm: c-r4k: Add support for flushing user pages from cache
>    MIPS: malta: Configure Segment Control registers for EVA boot
>    MIPS: malta: spaces.h: Add spaces.h file for Malta (EVA)
>    MIPS: malta: malta-memory: Add support for the 'ememsize' variable
>    MIPS: malta: malta-memory: Use the PHYS_OFFSET to build the memory map
>    MIPS: malta: malta-memory: Add free_init_pages_eva() callback
>    MIPS: malta: Add support for SMP EVA
>    MIPS: Enable MIPS 3.5 features on Malta
>
>   arch/mips/Kconfig                                  |  29 +-
>   arch/mips/include/asm/asm.h                        | 130 +++++
>   arch/mips/include/asm/bootinfo.h                   |   2 +
>   arch/mips/include/asm/checksum.h                   |  44 +-
>   arch/mips/include/asm/cpu-features.h               |   4 +-
>   arch/mips/include/asm/cpu.h                        |   1 +
>   arch/mips/include/asm/futex.h                      |   9 +-
>   arch/mips/include/asm/fw/fw.h                      |   2 +-
>   .../include/asm/mach-malta/kernel-entry-init.h     | 115 ++++-
>   arch/mips/include/asm/mach-malta/spaces.h          |  46 ++
>   arch/mips/include/asm/page.h                       |   2 +
>   arch/mips/include/asm/r4kcache.h                   | 173 ++++++-
>   arch/mips/include/asm/uaccess.h                    | 559 ++++++++++++++++-----
>   arch/mips/include/uapi/asm/inst.h                  |  25 +-
>   arch/mips/kernel/cpu-probe.c                       |   3 +
>   arch/mips/kernel/ftrace.c                          |   4 +
>   arch/mips/kernel/head.S                            |   2 +-
>   arch/mips/kernel/kgdb.c                            |  17 +-
>   arch/mips/kernel/mips_ksyms.c                      |  24 +-
>   arch/mips/kernel/proc.c                            |   1 +
>   arch/mips/kernel/scall32-o32.S                     |   9 +-
>   arch/mips/kernel/signal.c                          |  26 +
>   arch/mips/kernel/traps.c                           |  29 +-
>   arch/mips/kernel/unaligned.c                       | 135 ++++-
>   arch/mips/lib/csum_partial.S                       | 282 +++++++----
>   arch/mips/lib/memcpy.S                             | 416 ++++++++++-----
>   arch/mips/lib/memset.S                             | 146 ++++--
>   arch/mips/lib/strlen_user.S                        |  36 +-
>   arch/mips/lib/strncpy_user.S                       |  40 +-
>   arch/mips/lib/strnlen_user.S                       |  36 +-
>   arch/mips/mm/c-r4k.c                               |  64 ++-
>   arch/mips/mm/init.c                                |  12 +-
>   arch/mips/mti-malta/malta-init.c                   |  13 +
>   arch/mips/mti-malta/malta-memory.c                 |  58 ++-
>   arch/mips/mti-malta/malta-setup.c                  |   4 +
>   35 files changed, 1974 insertions(+), 524 deletions(-)
>   create mode 100644 arch/mips/include/asm/mach-malta/spaces.h
>

Re: [PATCH 00/58] Add support for Enhanced Virtual Addressing

[Markos Chandras]

Hi David,

I have some comments

On 01/29/2014 06:37 PM, David Daney wrote:
> This whole thing seems very messy.  I see a couple of problems:
>
> 1) There are if(CONFIG_EVA) ... else ... endif  all over the place.  It
> is very ugly.
>
> 2) You cannot have a signel kernel with both EVA and non-EVA support.


Unfortunately, I don't think using eva=1 or whatever in the kernel 
command line would work. First of all, there is board specific init code 
long before the command line is available. See patch #51. At that point, 
the kernel command line is not 'exposed' yet so there is no way to tell 
if you are booting in EVA or non-EVA mode.

>
> Have you considered just tagging all the instructions that touch the
> user address space, and patching them at system boot with their EVA
> equivalents if EVA support is desired?

I don't think this is possible (or maybe i don't understand your 
proposal). Consider for example the copy_user function in memcpy.S. This 
function does copy_{to,from,in} user/kernel depending on the supplied 
virtual address. There is no way to tell in advance if a 'lb' 
instruction will load from kernel or user space. This is only possible 
during runtime by examining the get_fs() == get_ds() status. Therefore, 
it's necessary to have 4 variants of this functions replacing only the 
'store' or 'load' instructions that you really need to. For example

- load from user, store to kernel
- load from kernel, store to user
- load from user, store to user
- load from kernel, store to kernel

As you can see, static replacement of the instructions with the EVA ones 
during boot will not work as expected.

This is the reason I converted these functions to macros, so all the 
variants will be a simple 3 line assembler code that only replaces the 
instructions you are really interested in.

-- 
markos

Re: [PATCH 00/58] Add support for Enhanced Virtual Addressing

[Markos Chandras]

Hi David,

I have some comments

On 01/29/2014 06:37 PM, David Daney wrote:
> This whole thing seems very messy.  I see a couple of problems:
>
> 1) There are if(CONFIG_EVA) ... else ... endif  all over the place.  It
> is very ugly.
>
> 2) You cannot have a signel kernel with both EVA and non-EVA support.


Unfortunately, I don't think using eva=1 or whatever in the kernel 
command line would work. First of all, there is board specific init code 
long before the command line is available. See patch #51. At that point, 
the kernel command line is not 'exposed' yet so there is no way to tell 
if you are booting in EVA or non-EVA mode.

>
> Have you considered just tagging all the instructions that touch the
> user address space, and patching them at system boot with their EVA
> equivalents if EVA support is desired?

I don't think this is possible (or maybe i don't understand your 
proposal). Consider for example the copy_user function in memcpy.S. This 
function does copy_{to,from,in} user/kernel depending on the supplied 
virtual address. There is no way to tell in advance if a 'lb' 
instruction will load from kernel or user space. This is only possible 
during runtime by examining the get_fs() == get_ds() status. Therefore, 
it's necessary to have 4 variants of this functions replacing only the 
'store' or 'load' instructions that you really need to. For example

- load from user, store to kernel
- load from kernel, store to user
- load from user, store to user
- load from kernel, store to kernel

As you can see, static replacement of the instructions with the EVA ones 
during boot will not work as expected.

This is the reason I converted these functions to macros, so all the 
variants will be a simple 3 line assembler code that only replaces the 
instructions you are really interested in.

-- 
markos

[PATCH v2 03/58] MIPS: asm: Add wrappers for EVA/non-EVA instructions

[Markos Chandras]

EVA uses specific instructions for accessing user memory.
Instead of polluting the kernel with numerous #ifdef CONFIG_EVA
we add wrappers for all the instructions that need special
handling when EVA is enabled.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
Changes since v1:
- Move EVA instruction wrappers to a new header file to avoid problematic
inclusions of the asm/asm.h header file.

This is already included in the upstream-sfr/mips-for-linux-next tree.
The files that have previously been modified to include <asm/asm.h> have been
updated to include <asm/asm-eva.h>. The fixes are also in the upstream-sfr/
mips-for-linux-next tree but I will not post new versions of these patches
because the change is rather trivial.

For this patch, no further action is required apart from updating the
patchwork URL for this commit in the mips-for-linux-next branch.
---

 arch/mips/include/asm/asm-eva.h | 135 ++++++++++++++++++++++++++++++++++++++++
 arch/mips/include/asm/asm.h     |   1 +
 2 files changed, 136 insertions(+)
 create mode 100644 arch/mips/include/asm/asm-eva.h

diff --git a/arch/mips/include/asm/asm-eva.h b/arch/mips/include/asm/asm-eva.h
new file mode 100644
index 0000000..e41c56e
--- /dev/null
+++ b/arch/mips/include/asm/asm-eva.h
@@ -0,0 +1,135 @@
+/*
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License.  See the file "COPYING" in the main directory of this archive
+ * for more details.
+ *
+ * Copyright (C) 2014 Imagination Technologies Ltd.
+ *
+ */
+
+#ifndef __ASM_ASM_EVA_H
+#define __ASM_ASM_EVA_H
+
+#ifndef __ASSEMBLY__
+#ifdef CONFIG_EVA
+
+#define __BUILD_EVA_INSN(insn, reg, addr)				\
+				"	.set	push\n"			\
+				"	.set	mips0\n"		\
+				"	.set	eva\n"			\
+				"	"insn" "reg", "addr "\n"	\
+				"	.set	pop\n"
+
+#define user_cache(op, base)		__BUILD_EVA_INSN("cachee", op, base)
+#define user_ll(reg, addr)		__BUILD_EVA_INSN("lle", reg, addr)
+#define user_sc(reg, addr)		__BUILD_EVA_INSN("sce", reg, addr)
+#define user_lw(reg, addr)		__BUILD_EVA_INSN("lwe", reg, addr)
+#define user_lwl(reg, addr)		__BUILD_EVA_INSN("lwle", reg, addr)
+#define user_lwr(reg, addr)		__BUILD_EVA_INSN("lwre", reg, addr)
+#define user_lh(reg, addr)		__BUILD_EVA_INSN("lhe", reg, addr)
+#define user_lb(reg, addr)		__BUILD_EVA_INSN("lbe", reg, addr)
+#define user_lbu(reg, addr)		__BUILD_EVA_INSN("lbue", reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_ld(reg, addr)		user_lw(reg, addr)
+#define user_sw(reg, addr)		__BUILD_EVA_INSN("swe", reg, addr)
+#define user_swl(reg, addr)		__BUILD_EVA_INSN("swle", reg, addr)
+#define user_swr(reg, addr)		__BUILD_EVA_INSN("swre", reg, addr)
+#define user_sh(reg, addr)		__BUILD_EVA_INSN("she", reg, addr)
+#define user_sb(reg, addr)		__BUILD_EVA_INSN("sbe", reg, addr)
+/* No 64-bit EVA instruction for storing double words */
+#define user_sd(reg, addr)		user_sw(reg, addr)
+
+#else
+
+#define user_cache(op, base)		"cache " op ", " base "\n"
+#define user_ll(reg, addr)		"ll " reg ", " addr "\n"
+#define user_sc(reg, addr)		"sc " reg ", " addr "\n"
+#define user_lw(reg, addr)		"lw " reg ", " addr "\n"
+#define user_lwl(reg, addr)		"lwl " reg ", " addr "\n"
+#define user_lwr(reg, addr)		"lwr " reg ", " addr "\n"
+#define user_lh(reg, addr)		"lh " reg ", " addr "\n"
+#define user_lb(reg, addr)		"lb " reg ", " addr "\n"
+#define user_lbu(reg, addr)		"lbu " reg ", " addr "\n"
+#define user_sw(reg, addr)		"sw " reg ", " addr "\n"
+#define user_swl(reg, addr)		"swl " reg ", " addr "\n"
+#define user_swr(reg, addr)		"swr " reg ", " addr "\n"
+#define user_sh(reg, addr)		"sh " reg ", " addr "\n"
+#define user_sb(reg, addr)		"sb " reg ", " addr "\n"
+
+#ifdef CONFIG_32BIT
+/*
+ * No 'sd' or 'ld' instructions in 32-bit but the code will
+ * do the correct thing
+ */
+#define user_sd(reg, addr)		user_sw(reg, addr)
+#define user_ld(reg, addr)		user_lw(reg, addr)
+#else
+#define user_sd(reg, addr)		"sd " reg", " addr "\n"
+#define user_ld(reg, addr)		"ld " reg", " addr "\n"
+#endif /* CONFIG_32BIT */
+
+#endif /* CONFIG_EVA */
+
+#else /* __ASSEMBLY__ */
+
+#ifdef CONFIG_EVA
+
+#define __BUILD_EVA_INSN(insn, reg, addr)			\
+				.set	push;			\
+				.set	mips0;			\
+				.set	eva;			\
+				insn reg, addr;			\
+				.set	pop;
+
+#define user_cache(op, base)		__BUILD_EVA_INSN(cachee, op, base)
+#define user_ll(reg, addr)		__BUILD_EVA_INSN(lle, reg, addr)
+#define user_sc(reg, addr)		__BUILD_EVA_INSN(sce, reg, addr)
+#define user_lw(reg, addr)		__BUILD_EVA_INSN(lwe, reg, addr)
+#define user_lwl(reg, addr)		__BUILD_EVA_INSN(lwle, reg, addr)
+#define user_lwr(reg, addr)		__BUILD_EVA_INSN(lwre, reg, addr)
+#define user_lh(reg, addr)		__BUILD_EVA_INSN(lhe, reg, addr)
+#define user_lb(reg, addr)		__BUILD_EVA_INSN(lbe, reg, addr)
+#define user_lbu(reg, addr)		__BUILD_EVA_INSN(lbue, reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_ld(reg, addr)		user_lw(reg, addr)
+#define user_sw(reg, addr)		__BUILD_EVA_INSN(swe, reg, addr)
+#define user_swl(reg, addr)		__BUILD_EVA_INSN(swle, reg, addr)
+#define user_swr(reg, addr)		__BUILD_EVA_INSN(swre, reg, addr)
+#define user_sh(reg, addr)		__BUILD_EVA_INSN(she, reg, addr)
+#define user_sb(reg, addr)		__BUILD_EVA_INSN(sbe, reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_sd(reg, addr)		user_sw(reg, addr)
+#else
+
+#define user_cache(op, base)		cache op, base
+#define user_ll(reg, addr)		ll reg, addr
+#define user_sc(reg, addr)		sc reg, addr
+#define user_lw(reg, addr)		lw reg, addr
+#define user_lwl(reg, addr)		lwl reg, addr
+#define user_lwr(reg, addr)		lwr reg, addr
+#define user_lh(reg, addr)		lh reg, addr
+#define user_lb(reg, addr)		lb reg, addr
+#define user_lbu(reg, addr)		lbu reg, addr
+#define user_sw(reg, addr)		sw reg, addr
+#define user_swl(reg, addr)		swl reg, addr
+#define user_swr(reg, addr)		swr reg, addr
+#define user_sh(reg, addr)		sh reg, addr
+#define user_sb(reg, addr)		sb reg, addr
+
+#ifdef CONFIG_32BIT
+/*
+ * No 'sd' or 'ld' instructions in 32-bit but the code will
+ * do the correct thing
+ */
+#define user_sd(reg, addr)		user_sw(reg, addr)
+#define user_ld(reg, addr)		user_lw(reg, addr)
+#else
+#define user_sd(reg, addr)		sd reg, addr
+#define user_ld(reg, addr)		ld reg, addr
+#endif /* CONFIG_32BIT */
+
+#endif /* CONFIG_EVA */
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* __ASM_ASM_EVA_H */
diff --git a/arch/mips/include/asm/asm.h b/arch/mips/include/asm/asm.h
index b79be18a..b153e79 100644
--- a/arch/mips/include/asm/asm.h
+++ b/arch/mips/include/asm/asm.h
@@ -18,6 +18,7 @@
 #define __ASM_ASM_H
 
 #include <asm/sgidefs.h>
+#include <asm/asm-eva.h>
 
 #ifndef CAT
 #ifdef __STDC__
-- 
1.9.0

[PATCH v2 03/58] MIPS: asm: Add wrappers for EVA/non-EVA instructions

[Markos Chandras]

EVA uses specific instructions for accessing user memory.
Instead of polluting the kernel with numerous #ifdef CONFIG_EVA
we add wrappers for all the instructions that need special
handling when EVA is enabled.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
---
Changes since v1:
- Move EVA instruction wrappers to a new header file to avoid problematic
inclusions of the asm/asm.h header file.

This is already included in the upstream-sfr/mips-for-linux-next tree.
The files that have previously been modified to include <asm/asm.h> have been
updated to include <asm/asm-eva.h>. The fixes are also in the upstream-sfr/
mips-for-linux-next tree but I will not post new versions of these patches
because the change is rather trivial.

For this patch, no further action is required apart from updating the
patchwork URL for this commit in the mips-for-linux-next branch.
---

 arch/mips/include/asm/asm-eva.h | 135 ++++++++++++++++++++++++++++++++++++++++
 arch/mips/include/asm/asm.h     |   1 +
 2 files changed, 136 insertions(+)
 create mode 100644 arch/mips/include/asm/asm-eva.h

diff --git a/arch/mips/include/asm/asm-eva.h b/arch/mips/include/asm/asm-eva.h
new file mode 100644
index 0000000..e41c56e
--- /dev/null
+++ b/arch/mips/include/asm/asm-eva.h
@@ -0,0 +1,135 @@
+/*
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License.  See the file "COPYING" in the main directory of this archive
+ * for more details.
+ *
+ * Copyright (C) 2014 Imagination Technologies Ltd.
+ *
+ */
+
+#ifndef __ASM_ASM_EVA_H
+#define __ASM_ASM_EVA_H
+
+#ifndef __ASSEMBLY__
+#ifdef CONFIG_EVA
+
+#define __BUILD_EVA_INSN(insn, reg, addr)				\
+				"	.set	push\n"			\
+				"	.set	mips0\n"		\
+				"	.set	eva\n"			\
+				"	"insn" "reg", "addr "\n"	\
+				"	.set	pop\n"
+
+#define user_cache(op, base)		__BUILD_EVA_INSN("cachee", op, base)
+#define user_ll(reg, addr)		__BUILD_EVA_INSN("lle", reg, addr)
+#define user_sc(reg, addr)		__BUILD_EVA_INSN("sce", reg, addr)
+#define user_lw(reg, addr)		__BUILD_EVA_INSN("lwe", reg, addr)
+#define user_lwl(reg, addr)		__BUILD_EVA_INSN("lwle", reg, addr)
+#define user_lwr(reg, addr)		__BUILD_EVA_INSN("lwre", reg, addr)
+#define user_lh(reg, addr)		__BUILD_EVA_INSN("lhe", reg, addr)
+#define user_lb(reg, addr)		__BUILD_EVA_INSN("lbe", reg, addr)
+#define user_lbu(reg, addr)		__BUILD_EVA_INSN("lbue", reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_ld(reg, addr)		user_lw(reg, addr)
+#define user_sw(reg, addr)		__BUILD_EVA_INSN("swe", reg, addr)
+#define user_swl(reg, addr)		__BUILD_EVA_INSN("swle", reg, addr)
+#define user_swr(reg, addr)		__BUILD_EVA_INSN("swre", reg, addr)
+#define user_sh(reg, addr)		__BUILD_EVA_INSN("she", reg, addr)
+#define user_sb(reg, addr)		__BUILD_EVA_INSN("sbe", reg, addr)
+/* No 64-bit EVA instruction for storing double words */
+#define user_sd(reg, addr)		user_sw(reg, addr)
+
+#else
+
+#define user_cache(op, base)		"cache " op ", " base "\n"
+#define user_ll(reg, addr)		"ll " reg ", " addr "\n"
+#define user_sc(reg, addr)		"sc " reg ", " addr "\n"
+#define user_lw(reg, addr)		"lw " reg ", " addr "\n"
+#define user_lwl(reg, addr)		"lwl " reg ", " addr "\n"
+#define user_lwr(reg, addr)		"lwr " reg ", " addr "\n"
+#define user_lh(reg, addr)		"lh " reg ", " addr "\n"
+#define user_lb(reg, addr)		"lb " reg ", " addr "\n"
+#define user_lbu(reg, addr)		"lbu " reg ", " addr "\n"
+#define user_sw(reg, addr)		"sw " reg ", " addr "\n"
+#define user_swl(reg, addr)		"swl " reg ", " addr "\n"
+#define user_swr(reg, addr)		"swr " reg ", " addr "\n"
+#define user_sh(reg, addr)		"sh " reg ", " addr "\n"
+#define user_sb(reg, addr)		"sb " reg ", " addr "\n"
+
+#ifdef CONFIG_32BIT
+/*
+ * No 'sd' or 'ld' instructions in 32-bit but the code will
+ * do the correct thing
+ */
+#define user_sd(reg, addr)		user_sw(reg, addr)
+#define user_ld(reg, addr)		user_lw(reg, addr)
+#else
+#define user_sd(reg, addr)		"sd " reg", " addr "\n"
+#define user_ld(reg, addr)		"ld " reg", " addr "\n"
+#endif /* CONFIG_32BIT */
+
+#endif /* CONFIG_EVA */
+
+#else /* __ASSEMBLY__ */
+
+#ifdef CONFIG_EVA
+
+#define __BUILD_EVA_INSN(insn, reg, addr)			\
+				.set	push;			\
+				.set	mips0;			\
+				.set	eva;			\
+				insn reg, addr;			\
+				.set	pop;
+
+#define user_cache(op, base)		__BUILD_EVA_INSN(cachee, op, base)
+#define user_ll(reg, addr)		__BUILD_EVA_INSN(lle, reg, addr)
+#define user_sc(reg, addr)		__BUILD_EVA_INSN(sce, reg, addr)
+#define user_lw(reg, addr)		__BUILD_EVA_INSN(lwe, reg, addr)
+#define user_lwl(reg, addr)		__BUILD_EVA_INSN(lwle, reg, addr)
+#define user_lwr(reg, addr)		__BUILD_EVA_INSN(lwre, reg, addr)
+#define user_lh(reg, addr)		__BUILD_EVA_INSN(lhe, reg, addr)
+#define user_lb(reg, addr)		__BUILD_EVA_INSN(lbe, reg, addr)
+#define user_lbu(reg, addr)		__BUILD_EVA_INSN(lbue, reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_ld(reg, addr)		user_lw(reg, addr)
+#define user_sw(reg, addr)		__BUILD_EVA_INSN(swe, reg, addr)
+#define user_swl(reg, addr)		__BUILD_EVA_INSN(swle, reg, addr)
+#define user_swr(reg, addr)		__BUILD_EVA_INSN(swre, reg, addr)
+#define user_sh(reg, addr)		__BUILD_EVA_INSN(she, reg, addr)
+#define user_sb(reg, addr)		__BUILD_EVA_INSN(sbe, reg, addr)
+/* No 64-bit EVA instruction for loading double words */
+#define user_sd(reg, addr)		user_sw(reg, addr)
+#else
+
+#define user_cache(op, base)		cache op, base
+#define user_ll(reg, addr)		ll reg, addr
+#define user_sc(reg, addr)		sc reg, addr
+#define user_lw(reg, addr)		lw reg, addr
+#define user_lwl(reg, addr)		lwl reg, addr
+#define user_lwr(reg, addr)		lwr reg, addr
+#define user_lh(reg, addr)		lh reg, addr
+#define user_lb(reg, addr)		lb reg, addr
+#define user_lbu(reg, addr)		lbu reg, addr
+#define user_sw(reg, addr)		sw reg, addr
+#define user_swl(reg, addr)		swl reg, addr
+#define user_swr(reg, addr)		swr reg, addr
+#define user_sh(reg, addr)		sh reg, addr
+#define user_sb(reg, addr)		sb reg, addr
+
+#ifdef CONFIG_32BIT
+/*
+ * No 'sd' or 'ld' instructions in 32-bit but the code will
+ * do the correct thing
+ */
+#define user_sd(reg, addr)		user_sw(reg, addr)
+#define user_ld(reg, addr)		user_lw(reg, addr)
+#else
+#define user_sd(reg, addr)		sd reg, addr
+#define user_ld(reg, addr)		ld reg, addr
+#endif /* CONFIG_32BIT */
+
+#endif /* CONFIG_EVA */
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* __ASM_ASM_EVA_H */
diff --git a/arch/mips/include/asm/asm.h b/arch/mips/include/asm/asm.h
index b79be18a..b153e79 100644
--- a/arch/mips/include/asm/asm.h
+++ b/arch/mips/include/asm/asm.h
@@ -18,6 +18,7 @@
 #define __ASM_ASM_H
 
 #include <asm/sgidefs.h>
+#include <asm/asm-eva.h>
 
 #ifndef CAT
 #ifdef __STDC__
-- 
1.9.0