[PATCH v2 0/5] MIPS: Move cache setup earlier.

[PATCH v2 0/5] MIPS: Move cache setup earlier.

[David Daney]

From: David Daney <david.daney@cavium.com>

v2: Rebased to 3.4-rc7

>From v1:

Found on 3.2-rc4, if jump label things are enabled, the kernel will
not boot on MIPS.

As noted in patch 5/5, this was caused by: commit
97ce2c88f9ad42e3c60a9beb9fca87abf3639faa (jump-label: initialize
jump-label subsystem much earlier).

The fix is to make flush_icache_range() usable earlier.  I do this by
calling cpu_cache_init() from setup_arch().  For the boot CPU we can
no longer call this from per_cpu_trap_init(), so I add a flag to that
function so we can skip the call if on the boot CPU.

Some of the code in the various cpu_cache_init() functions however
could not be called this early, so I moved it into functions called by
the new board_cache_error_setup() hook.

Tested on Octeon and ip32-R5000

David Daney (5):
  MIPS: Introduce board_cache_error_setup() hook.
  MIPS: Make set_handler() __cpuinit.
  MIPS: Octeon: Use board_cache_error_setup for cache error handler
    setup.
  MIPS: Use board_cache_error_setup for r4k cache error handler setup.
  MIPS: Move cache setup to setup_arch().

 arch/mips/include/asm/setup.h |    3 ++-
 arch/mips/include/asm/traps.h |    1 +
 arch/mips/kernel/setup.c      |    2 ++
 arch/mips/kernel/smp.c        |    2 +-
 arch/mips/kernel/traps.c      |   16 ++++++++++------
 arch/mips/mm/c-octeon.c       |   14 ++++++++------
 arch/mips/mm/c-r4k.c          |   14 ++++++++++----
 7 files changed, 34 insertions(+), 18 deletions(-)

-- 
1.7.2.3

[PATCH v2 0/5] MIPS: Move cache setup earlier.

[David Daney]

From: David Daney <david.daney@cavium.com>

v2: Rebased to 3.4-rc7

From v1:

Found on 3.2-rc4, if jump label things are enabled, the kernel will
not boot on MIPS.

As noted in patch 5/5, this was caused by: commit
97ce2c88f9ad42e3c60a9beb9fca87abf3639faa (jump-label: initialize
jump-label subsystem much earlier).

The fix is to make flush_icache_range() usable earlier.  I do this by
calling cpu_cache_init() from setup_arch().  For the boot CPU we can
no longer call this from per_cpu_trap_init(), so I add a flag to that
function so we can skip the call if on the boot CPU.

Some of the code in the various cpu_cache_init() functions however
could not be called this early, so I moved it into functions called by
the new board_cache_error_setup() hook.

Tested on Octeon and ip32-R5000

David Daney (5):
  MIPS: Introduce board_cache_error_setup() hook.
  MIPS: Make set_handler() __cpuinit.
  MIPS: Octeon: Use board_cache_error_setup for cache error handler
    setup.
  MIPS: Use board_cache_error_setup for r4k cache error handler setup.
  MIPS: Move cache setup to setup_arch().

 arch/mips/include/asm/setup.h |    3 ++-
 arch/mips/include/asm/traps.h |    1 +
 arch/mips/kernel/setup.c      |    2 ++
 arch/mips/kernel/smp.c        |    2 +-
 arch/mips/kernel/traps.c      |   16 ++++++++++------
 arch/mips/mm/c-octeon.c       |   14 ++++++++------
 arch/mips/mm/c-r4k.c          |   14 ++++++++++----
 7 files changed, 34 insertions(+), 18 deletions(-)

-- 
1.7.2.3

[PATCH v2 1/5] MIPS: Introduce board_cache_error_setup() hook.

[David Daney]

From: David Daney <david.daney@cavium.com>

This is used in subsequent patches.

Signed-off-by: David Daney <david.daney@cavium.com>
---
 arch/mips/include/asm/traps.h |    1 +
 arch/mips/kernel/traps.c      |    5 ++++-
 2 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/arch/mips/include/asm/traps.h b/arch/mips/include/asm/traps.h
index ff74aec..420ca06 100644
--- a/arch/mips/include/asm/traps.h
+++ b/arch/mips/include/asm/traps.h
@@ -25,6 +25,7 @@ extern void (*board_nmi_handler_setup)(void);
 extern void (*board_ejtag_handler_setup)(void);
 extern void (*board_bind_eic_interrupt)(int irq, int regset);
 extern void (*board_ebase_setup)(void);
+extern void (*board_cache_error_setup)(void);
 
 extern int register_nmi_notifier(struct notifier_block *nb);
 
diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
index 9fd636b..b931eba 100644
--- a/arch/mips/kernel/traps.c
+++ b/arch/mips/kernel/traps.c
@@ -105,7 +105,7 @@ void (*board_nmi_handler_setup)(void);
 void (*board_ejtag_handler_setup)(void);
 void (*board_bind_eic_interrupt)(int irq, int regset);
 void (*board_ebase_setup)(void);
-
+void __cpuinitdata(*board_cache_error_setup)(void);
 
 static void show_raw_backtrace(unsigned long reg29)
 {
@@ -1845,6 +1845,9 @@ void __init trap_init(void)
 
 	set_except_vector(26, handle_dsp);
 
+	if (board_cache_error_setup)
+		board_cache_error_setup();
+
 	if (cpu_has_vce)
 		/* Special exception: R4[04]00 uses also the divec space. */
 		memcpy((void *)(ebase + 0x180), &except_vec3_r4000, 0x100);
-- 
1.7.2.3

[PATCH v2 2/5] MIPS: Make set_handler() __cpuinit.

[David Daney]

From: David Daney <david.daney@cavium.com>

Follow-on patched require this.

Signed-off-by: David Daney <david.daney@cavium.com>
---
 arch/mips/kernel/traps.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
index b931eba..2b5675b 100644
--- a/arch/mips/kernel/traps.c
+++ b/arch/mips/kernel/traps.c
@@ -1680,7 +1680,7 @@ void __cpuinit per_cpu_trap_init(void)
 }
 
 /* Install CPU exception handler */
-void __init set_handler(unsigned long offset, void *addr, unsigned long size)
+void __cpuinit set_handler(unsigned long offset, void *addr, unsigned long size)
 {
 	memcpy((void *)(ebase + offset), addr, size);
 	local_flush_icache_range(ebase + offset, ebase + offset + size);
-- 
1.7.2.3

Re: [PATCH v2 2/5] MIPS: Make set_handler() __cpuinit.

[Sergei Shtylyov]

Hello.

On 15-05-2012 4:04, David Daney wrote:

> From: David Daney<david.daney@cavium.com>

> Follow-on patched require this.

    s/patched/patches/
    Maybe Ralf would correct this...

> Signed-off-by: David Daney<david.daney@cavium.com>

WBR, Sergei

[PATCH v2 3/5] MIPS: Octeon: Use board_cache_error_setup for cache error handler setup.

[David Daney]

From: David Daney <david.daney@cavium.com>

Signed-off-by: David Daney <david.daney@cavium.com>
---
 arch/mips/mm/c-octeon.c |   14 ++++++++------
 1 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/arch/mips/mm/c-octeon.c b/arch/mips/mm/c-octeon.c
index 47037ec..44e69e7 100644
--- a/arch/mips/mm/c-octeon.c
+++ b/arch/mips/mm/c-octeon.c
@@ -21,6 +21,7 @@
 #include <asm/page.h>
 #include <asm/pgtable.h>
 #include <asm/r4kcache.h>
+#include <asm/traps.h>
 #include <asm/mmu_context.h>
 #include <asm/war.h>
 
@@ -248,6 +249,11 @@ static void __cpuinit probe_octeon(void)
 	}
 }
 
+static void  __cpuinit octeon_cache_error_setup(void)
+{
+	extern char except_vec2_octeon;
+	set_handler(0x100, &except_vec2_octeon, 0x80);
+}
 
 /**
  * Setup the Octeon cache flush routines
@@ -255,12 +261,6 @@ static void __cpuinit probe_octeon(void)
  */
 void __cpuinit octeon_cache_init(void)
 {
-	extern unsigned long ebase;
-	extern char except_vec2_octeon;
-
-	memcpy((void *)(ebase + 0x100), &except_vec2_octeon, 0x80);
-	octeon_flush_cache_sigtramp(ebase + 0x100);
-
 	probe_octeon();
 
 	shm_align_mask = PAGE_SIZE - 1;
@@ -280,6 +280,8 @@ void __cpuinit octeon_cache_init(void)
 
 	build_clear_page();
 	build_copy_page();
+
+	board_cache_error_setup = octeon_cache_error_setup;
 }
 
 /**
-- 
1.7.2.3

[PATCH v2 4/5] MIPS: Use board_cache_error_setup for r4k cache error handler setup.

[David Daney]

From: David Daney <david.daney@cavium.com>

Signed-off-by: David Daney <david.daney@cavium.com>
---
 arch/mips/mm/c-r4k.c |   14 ++++++++++----
 1 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index bda8eb2..5109be9 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -32,7 +32,7 @@
 #include <asm/mmu_context.h>
 #include <asm/war.h>
 #include <asm/cacheflush.h> /* for run_uncached() */
-
+#include <asm/traps.h>
 
 /*
  * Special Variant of smp_call_function for use by cache functions:
@@ -1385,10 +1385,8 @@ static int __init setcoherentio(char *str)
 __setup("coherentio", setcoherentio);
 #endif
 
-void __cpuinit r4k_cache_init(void)
+static void __cpuinit r4k_cache_error_setup(void)
 {
-	extern void build_clear_page(void);
-	extern void build_copy_page(void);
 	extern char __weak except_vec2_generic;
 	extern char __weak except_vec2_sb1;
 	struct cpuinfo_mips *c = &current_cpu_data;
@@ -1403,6 +1401,13 @@ void __cpuinit r4k_cache_init(void)
 		set_uncached_handler(0x100, &except_vec2_generic, 0x80);
 		break;
 	}
+}
+
+void __cpuinit r4k_cache_init(void)
+{
+	extern void build_clear_page(void);
+	extern void build_copy_page(void);
+	struct cpuinfo_mips *c = &current_cpu_data;
 
 	probe_pcache();
 	setup_scache();
@@ -1465,4 +1470,5 @@ void __cpuinit r4k_cache_init(void)
 	local_r4k___flush_cache_all(NULL);
 #endif
 	coherency_setup();
+	board_cache_error_setup = r4k_cache_error_setup;
 }
-- 
1.7.2.3

[PATCH v2 5/5] MIPS: Move cache setup to setup_arch().

[David Daney]

From: David Daney <david.daney@cavium.com>

commit 97ce2c88f9ad42e3c60a9beb9fca87abf3639faa (jump-label: initialize
jump-label subsystem much earlier) breaks MIPS.  The jump_label_init()
call was moved before trap_init() which is where we initialize
flush_icache_range().

In order to be good citizens, we move cache initialization earlier so
that we don't jump through a null flush_icache_range function pointer
when doing the jump label initialization.

Signed-off-by: David Daney <david.daney@cavium.com>
---
 arch/mips/include/asm/setup.h |    3 ++-
 arch/mips/kernel/setup.c      |    2 ++
 arch/mips/kernel/smp.c        |    2 +-
 arch/mips/kernel/traps.c      |    9 +++++----
 4 files changed, 10 insertions(+), 6 deletions(-)

diff --git a/arch/mips/include/asm/setup.h b/arch/mips/include/asm/setup.h
index 6dce6d8..2560b6b 100644
--- a/arch/mips/include/asm/setup.h
+++ b/arch/mips/include/asm/setup.h
@@ -14,7 +14,8 @@ extern void *set_vi_handler(int n, vi_handler_t addr);
 
 extern void *set_except_vector(int n, void *addr);
 extern unsigned long ebase;
-extern void per_cpu_trap_init(void);
+extern void per_cpu_trap_init(bool);
+extern void cpu_cache_init(void);
 
 #endif /* __KERNEL__ */
 
diff --git a/arch/mips/kernel/setup.c b/arch/mips/kernel/setup.c
index c504b21..a53f8ec 100644
--- a/arch/mips/kernel/setup.c
+++ b/arch/mips/kernel/setup.c
@@ -605,6 +605,8 @@ void __init setup_arch(char **cmdline_p)
 
 	resource_init();
 	plat_smp_setup();
+
+	cpu_cache_init();
 }
 
 unsigned long kernelsp[NR_CPUS];
diff --git a/arch/mips/kernel/smp.c b/arch/mips/kernel/smp.c
index ba9376b..dc019a1 100644
--- a/arch/mips/kernel/smp.c
+++ b/arch/mips/kernel/smp.c
@@ -106,7 +106,7 @@ asmlinkage __cpuinit void start_secondary(void)
 #endif /* CONFIG_MIPS_MT_SMTC */
 	cpu_probe();
 	cpu_report();
-	per_cpu_trap_init();
+	per_cpu_trap_init(false);
 	mips_clockevent_init();
 	mp_ops->init_secondary();
 
diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
index 2b5675b..0ba66c0 100644
--- a/arch/mips/kernel/traps.c
+++ b/arch/mips/kernel/traps.c
@@ -1538,7 +1538,6 @@ void *set_vi_handler(int n, vi_handler_t addr)
 	return set_vi_srs_handler(n, addr, 0);
 }
 
-extern void cpu_cache_init(void);
 extern void tlb_init(void);
 extern void flush_tlb_handlers(void);
 
@@ -1565,7 +1564,7 @@ static int __init ulri_disable(char *s)
 }
 __setup("noulri", ulri_disable);
 
-void __cpuinit per_cpu_trap_init(void)
+void __cpuinit per_cpu_trap_init(bool is_boot_cpu)
 {
 	unsigned int cpu = smp_processor_id();
 	unsigned int status_set = ST0_CU0;
@@ -1664,7 +1663,9 @@ void __cpuinit per_cpu_trap_init(void)
 #ifdef CONFIG_MIPS_MT_SMTC
 	if (bootTC) {
 #endif /* CONFIG_MIPS_MT_SMTC */
-		cpu_cache_init();
+		/* Boot CPU's cache setup in setup_arch(). */
+		if (!is_boot_cpu)
+			cpu_cache_init();
 		tlb_init();
 #ifdef CONFIG_MIPS_MT_SMTC
 	} else if (!secondaryTC) {
@@ -1741,7 +1742,7 @@ void __init trap_init(void)
 
 	if (board_ebase_setup)
 		board_ebase_setup();
-	per_cpu_trap_init();
+	per_cpu_trap_init(true);
 
 	/*
 	 * Copy the generic exception handlers to their final destination.
-- 
1.7.2.3

Re: [PATCH v2 5/5] MIPS: Move cache setup to setup_arch().

[Shane McDonald]

I've run into a problem in linux-3.5-rc1, and I've tracked it down
to this patch, MIPS: Move cache setup to setup_arch().,
commit 6650df3c380e0db558dbfec63ed860402c6afb2a.

On Mon, May 14, 2012 at 6:04 PM, David Daney <ddaney.cavm@gmail.com> wrote:
> From: David Daney <david.daney@cavium.com>
>
> commit 97ce2c88f9ad42e3c60a9beb9fca87abf3639faa (jump-label: initialize
> jump-label subsystem much earlier) breaks MIPS.  The jump_label_init()
> call was moved before trap_init() which is where we initialize
> flush_icache_range().
>
> In order to be good citizens, we move cache initialization earlier so
> that we don't jump through a null flush_icache_range function pointer
> when doing the jump label initialization.
>
> Signed-off-by: David Daney <david.daney@cavium.com>
> ---
>  arch/mips/include/asm/setup.h |    3 ++-
>  arch/mips/kernel/setup.c      |    2 ++
>  arch/mips/kernel/smp.c        |    2 +-
>  arch/mips/kernel/traps.c      |    9 +++++----
>  4 files changed, 10 insertions(+), 6 deletions(-)
>
> diff --git a/arch/mips/include/asm/setup.h b/arch/mips/include/asm/setup.h
> index 6dce6d8..2560b6b 100644
> --- a/arch/mips/include/asm/setup.h
> +++ b/arch/mips/include/asm/setup.h
> @@ -14,7 +14,8 @@ extern void *set_vi_handler(int n, vi_handler_t addr);
>
>  extern void *set_except_vector(int n, void *addr);
>  extern unsigned long ebase;
> -extern void per_cpu_trap_init(void);
> +extern void per_cpu_trap_init(bool);
> +extern void cpu_cache_init(void);
>
>  #endif /* __KERNEL__ */
>
> diff --git a/arch/mips/kernel/setup.c b/arch/mips/kernel/setup.c
> index c504b21..a53f8ec 100644
> --- a/arch/mips/kernel/setup.c
> +++ b/arch/mips/kernel/setup.c
> @@ -605,6 +605,8 @@ void __init setup_arch(char **cmdline_p)
>
>        resource_init();
>        plat_smp_setup();
> +
> +       cpu_cache_init();
>  }
>
>  unsigned long kernelsp[NR_CPUS];
> diff --git a/arch/mips/kernel/smp.c b/arch/mips/kernel/smp.c
> index ba9376b..dc019a1 100644
> --- a/arch/mips/kernel/smp.c
> +++ b/arch/mips/kernel/smp.c
> @@ -106,7 +106,7 @@ asmlinkage __cpuinit void start_secondary(void)
>  #endif /* CONFIG_MIPS_MT_SMTC */
>        cpu_probe();
>        cpu_report();
> -       per_cpu_trap_init();
> +       per_cpu_trap_init(false);
>        mips_clockevent_init();
>        mp_ops->init_secondary();
>
> diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
> index 2b5675b..0ba66c0 100644
> --- a/arch/mips/kernel/traps.c
> +++ b/arch/mips/kernel/traps.c
> @@ -1538,7 +1538,6 @@ void *set_vi_handler(int n, vi_handler_t addr)
>        return set_vi_srs_handler(n, addr, 0);
>  }
>
> -extern void cpu_cache_init(void);
>  extern void tlb_init(void);
>  extern void flush_tlb_handlers(void);
>
> @@ -1565,7 +1564,7 @@ static int __init ulri_disable(char *s)
>  }
>  __setup("noulri", ulri_disable);
>
> -void __cpuinit per_cpu_trap_init(void)
> +void __cpuinit per_cpu_trap_init(bool is_boot_cpu)
>  {
>        unsigned int cpu = smp_processor_id();
>        unsigned int status_set = ST0_CU0;
> @@ -1664,7 +1663,9 @@ void __cpuinit per_cpu_trap_init(void)
>  #ifdef CONFIG_MIPS_MT_SMTC
>        if (bootTC) {
>  #endif /* CONFIG_MIPS_MT_SMTC */
> -               cpu_cache_init();
> +               /* Boot CPU's cache setup in setup_arch(). */
> +               if (!is_boot_cpu)
> +                       cpu_cache_init();
>                tlb_init();
>  #ifdef CONFIG_MIPS_MT_SMTC
>        } else if (!secondaryTC) {
> @@ -1741,7 +1742,7 @@ void __init trap_init(void)
>
>        if (board_ebase_setup)
>                board_ebase_setup();
> -       per_cpu_trap_init();
> +       per_cpu_trap_init(true);
>
>        /*
>         * Copy the generic exception handlers to their final destination.
> --
> 1.7.2.3

I'm running a single-CPU, PMC-Sierra RM7035C-based system.

Before applying this patch, cca_setup() in arch/mips/mm/c-r4k.c
is called before coherency_setup() (called from rk4_cache_init()).
After applying the patch, it is called afterwards.  Because
coherency_setup() relies on cca_setup() properly setting the
variable cca, it won't use the value of cca supplied on the
kernel command line.

I haven't verified it, but I suspect the same problem will occur
with the call to setcoherentio(), also in c-r4k.c.

Unfortunately, I don't have the knowledge to formulate a patch
to this problem, but I wanted to raise the issue.

Shane McDonald

Re: [PATCH v2 5/5] MIPS: Move cache setup to setup_arch().

[David Daney]

On 06/10/2012 09:16 PM, Shane McDonald wrote:
> I've run into a problem in linux-3.5-rc1, and I've tracked it down
> to this patch, MIPS: Move cache setup to setup_arch().,
> commit 6650df3c380e0db558dbfec63ed860402c6afb2a.
>

Are you permitted to describe the problem in any additional detail?

Knowing what type of system is affected and the nature of the problem 
would be useful in trying to arrive at a good solution.

David Daney

> On Mon, May 14, 2012 at 6:04 PM, David Daney<ddaney.cavm@gmail.com>  wrote:
>> From: David Daney<david.daney@cavium.com>
>>
>> commit 97ce2c88f9ad42e3c60a9beb9fca87abf3639faa (jump-label: initialize
>> jump-label subsystem much earlier) breaks MIPS.  The jump_label_init()
>> call was moved before trap_init() which is where we initialize
>> flush_icache_range().
>>
>> In order to be good citizens, we move cache initialization earlier so
>> that we don't jump through a null flush_icache_range function pointer
>> when doing the jump label initialization.
>>
>> Signed-off-by: David Daney<david.daney@cavium.com>
>> ---
>>   arch/mips/include/asm/setup.h |    3 ++-
>>   arch/mips/kernel/setup.c      |    2 ++
>>   arch/mips/kernel/smp.c        |    2 +-
>>   arch/mips/kernel/traps.c      |    9 +++++----
>>   4 files changed, 10 insertions(+), 6 deletions(-)
>>
>> diff --git a/arch/mips/include/asm/setup.h b/arch/mips/include/asm/setup.h
>> index 6dce6d8..2560b6b 100644
>> --- a/arch/mips/include/asm/setup.h
>> +++ b/arch/mips/include/asm/setup.h
>> @@ -14,7 +14,8 @@ extern void *set_vi_handler(int n, vi_handler_t addr);
>>
>>   extern void *set_except_vector(int n, void *addr);
>>   extern unsigned long ebase;
>> -extern void per_cpu_trap_init(void);
>> +extern void per_cpu_trap_init(bool);
>> +extern void cpu_cache_init(void);
>>
>>   #endif /* __KERNEL__ */
>>
>> diff --git a/arch/mips/kernel/setup.c b/arch/mips/kernel/setup.c
>> index c504b21..a53f8ec 100644
>> --- a/arch/mips/kernel/setup.c
>> +++ b/arch/mips/kernel/setup.c
>> @@ -605,6 +605,8 @@ void __init setup_arch(char **cmdline_p)
>>
>>         resource_init();
>>         plat_smp_setup();
>> +
>> +       cpu_cache_init();
>>   }
>>
>>   unsigned long kernelsp[NR_CPUS];
>> diff --git a/arch/mips/kernel/smp.c b/arch/mips/kernel/smp.c
>> index ba9376b..dc019a1 100644
>> --- a/arch/mips/kernel/smp.c
>> +++ b/arch/mips/kernel/smp.c
>> @@ -106,7 +106,7 @@ asmlinkage __cpuinit void start_secondary(void)
>>   #endif /* CONFIG_MIPS_MT_SMTC */
>>         cpu_probe();
>>         cpu_report();
>> -       per_cpu_trap_init();
>> +       per_cpu_trap_init(false);
>>         mips_clockevent_init();
>>         mp_ops->init_secondary();
>>
>> diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
>> index 2b5675b..0ba66c0 100644
>> --- a/arch/mips/kernel/traps.c
>> +++ b/arch/mips/kernel/traps.c
>> @@ -1538,7 +1538,6 @@ void *set_vi_handler(int n, vi_handler_t addr)
>>         return set_vi_srs_handler(n, addr, 0);
>>   }
>>
>> -extern void cpu_cache_init(void);
>>   extern void tlb_init(void);
>>   extern void flush_tlb_handlers(void);
>>
>> @@ -1565,7 +1564,7 @@ static int __init ulri_disable(char *s)
>>   }
>>   __setup("noulri", ulri_disable);
>>
>> -void __cpuinit per_cpu_trap_init(void)
>> +void __cpuinit per_cpu_trap_init(bool is_boot_cpu)
>>   {
>>         unsigned int cpu = smp_processor_id();
>>         unsigned int status_set = ST0_CU0;
>> @@ -1664,7 +1663,9 @@ void __cpuinit per_cpu_trap_init(void)
>>   #ifdef CONFIG_MIPS_MT_SMTC
>>         if (bootTC) {
>>   #endif /* CONFIG_MIPS_MT_SMTC */
>> -               cpu_cache_init();
>> +               /* Boot CPU's cache setup in setup_arch(). */
>> +               if (!is_boot_cpu)
>> +                       cpu_cache_init();
>>                 tlb_init();
>>   #ifdef CONFIG_MIPS_MT_SMTC
>>         } else if (!secondaryTC) {
>> @@ -1741,7 +1742,7 @@ void __init trap_init(void)
>>
>>         if (board_ebase_setup)
>>                 board_ebase_setup();
>> -       per_cpu_trap_init();
>> +       per_cpu_trap_init(true);
>>
>>         /*
>>          * Copy the generic exception handlers to their final destination.
>> --
>> 1.7.2.3
>
> I'm running a single-CPU, PMC-Sierra RM7035C-based system.
>
> Before applying this patch, cca_setup() in arch/mips/mm/c-r4k.c
> is called before coherency_setup() (called from rk4_cache_init()).
> After applying the patch, it is called afterwards.  Because
> coherency_setup() relies on cca_setup() properly setting the
> variable cca, it won't use the value of cca supplied on the
> kernel command line.
>
> I haven't verified it, but I suspect the same problem will occur
> with the call to setcoherentio(), also in c-r4k.c.
>
> Unfortunately, I don't have the knowledge to formulate a patch
> to this problem, but I wanted to raise the issue.
>
> Shane McDonald
>

Re: [PATCH v2 5/5] MIPS: Move cache setup to setup_arch().

[David Daney]

On 06/11/2012 09:21 AM, David Daney wrote:
> On 06/10/2012 09:16 PM, Shane McDonald wrote:
>> I've run into a problem in linux-3.5-rc1, and I've tracked it down
>> to this patch, MIPS: Move cache setup to setup_arch().,
>> commit 6650df3c380e0db558dbfec63ed860402c6afb2a.
>>
>
> Are you permitted to describe the problem in any additional detail?
>
> Knowing what type of system is affected and the nature of the problem
> would be useful in trying to arrive at a good solution.
>

OK, for some reason I missed this bit...

[...]
>>
>> I'm running a single-CPU, PMC-Sierra RM7035C-based system.
>>
>> Before applying this patch, cca_setup() in arch/mips/mm/c-r4k.c
>> is called before coherency_setup() (called from rk4_cache_init()).
>> After applying the patch, it is called afterwards. Because
>> coherency_setup() relies on cca_setup() properly setting the
>> variable cca, it won't use the value of cca supplied on the
>> kernel command line.
>>
>> I haven't verified it, but I suspect the same problem will occur
>> with the call to setcoherentio(), also in c-r4k.c.
>>
>> Unfortunately, I don't have the knowledge to formulate a patch
>> to this problem, but I wanted to raise the issue.
>>
>> Shane McDonald
>>
>

I will think about how to fix it.

David Daney

Re: [PATCH v2 5/5] MIPS: Move cache setup to setup_arch().

[Shane McDonald]

Hi David:

  A little more info, if it's useful...

On Mon, Jun 11, 2012 at 10:39 AM, David Daney <ddaney.cavm@gmail.com> wrote:
>>> I've run into a problem in linux-3.5-rc1, and I've tracked it down
>>> to this patch, MIPS: Move cache setup to setup_arch().,
>>> commit 6650df3c380e0db558dbfec63ed860402c6afb2a.
>>>
>>
>> Are you permitted to describe the problem in any additional detail?
>>
>> Knowing what type of system is affected and the nature of the problem
>> would be useful in trying to arrive at a good solution.
>
>>> I'm running a single-CPU, PMC-Sierra RM7035C-based system.
>>>
>>> Before applying this patch, cca_setup() in arch/mips/mm/c-r4k.c
>>> is called before coherency_setup() (called from rk4_cache_init()).
>>> After applying the patch, it is called afterwards. Because
>>> coherency_setup() relies on cca_setup() properly setting the
>>> variable cca, it won't use the value of cca supplied on the
>>> kernel command line.
>>>
>>> I haven't verified it, but I suspect the same problem will occur
>>> with the call to setcoherentio(), also in c-r4k.c.
>>>
>>> Unfortunately, I don't have the knowledge to formulate a patch
>>> to this problem, but I wanted to raise the issue.

The board is the PMC-Sierra "Xiao Hu", described at
http://www.linuxjournal.com/article/7854.  The version I have
has the 600 MHz RM7035C processor.  The code supporting
the board is not in-tree, unfortunately.  I believe PMC-Sierra
is no longer maintaining their BSP, but I've been keeping
a copy of the kernel code up-to-date.

Here is the info on the processor reported by PMON:

CPU type RM7065.  Rev 5.3.  600 MHz/100 MHz.
Memory size 128 MB.
Icache size  16 KB, 32/line (4 way)
Dcache size  16 KB, 32/line (4 way)
Scache size 256 KB, 32/line (4 way)

I typically boot with the command-line options:
root=/dev/hda1 cca=3

CPU information displayed upon booting is:

CPU revision is: 00002753 (RM7000)
FPU revision is: 00002750

Cache information displayed upon booting is:

Primary instruction cache 16kB, VIPT, 4-way, linesize 32 bytes.
Primary data cache 16kB, 4-way, VIPT, no aliases, linesize 32 bytes
Secondary cache size 256K, linesize 32 bytes.

Here are some snippets from the .config file (I've removed lines
that are probably irrelevant to this problem):

CONFIG_PMC_XIAOHU=y
CONFIG_RWSEM_GENERIC_SPINLOCK=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_CEVT_R4K_LIB=y
CONFIG_CEVT_R4K=y
CONFIG_CSRC_R4K_LIB=y
CONFIG_CSRC_R4K=y
# CONFIG_ARCH_DMA_ADDR_T_64BIT is not set
CONFIG_DMA_NONCOHERENT=y

CONFIG_CPU_LITTLE_ENDIAN=y
CONFIG_SYS_SUPPORTS_LITTLE_ENDIAN=y
CONFIG_IRQ_CPU=y
CONFIG_MIPS_L1_CACHE_SHIFT=5

#
# CPU selection
#
CONFIG_CPU_RM7000=y
CONFIG_SYS_HAS_CPU_RM7000=y
CONFIG_SYS_SUPPORTS_32BIT_KERNEL=y
CONFIG_SYS_SUPPORTS_64BIT_KERNEL=y
CONFIG_CPU_SUPPORTS_32BIT_KERNEL=y
CONFIG_CPU_SUPPORTS_64BIT_KERNEL=y

#
# Kernel type
#
CONFIG_32BIT=y
# CONFIG_64BIT is not set
CONFIG_PAGE_SIZE_4KB=y
# CONFIG_PAGE_SIZE_16KB is not set
# CONFIG_PAGE_SIZE_64KB is not set
CONFIG_FORCE_MAX_ZONEORDER=11
CONFIG_BOARD_SCACHE=y
CONFIG_RM7000_CPU_SCACHE=y
CONFIG_CPU_HAS_PREFETCH=y
CONFIG_MIPS_MT_DISABLED=y
# CONFIG_ARCH_PHYS_ADDR_T_64BIT is not set


After adding in some debugging statements, I see that after applying
the subject patch, coherency_setup() is called before cca_setup()
is called.  Before applying the patch, coherency_setup() is called
after cca_setup().

cca_setup() grabs the value of the cca= command line parameter,
and stores it in the variable cca.  If no such command line parameter
is given, cca uses its default value of -1.  coherency_setup() then uses
the cca variable when setting up the coherency mode.
that value

> I will think about how to fix it.
>
> David Daney

Thanks.  There is a line:

__setup("cca=", cca_setup);

that seems to be used to call cca_setup().  I don't know how
the __setup() works, so I'm a little lost on the solution myself.

Note that, besides the cca_setup(), there is also a routine
setcoherentio() that is defined the same way as cca_setup().
I suspect that suffers from the same problem as cca_setup().

If there is any other information I can provide,
please let me know!  Thanks.

Shane McDonald

Re: [PATCH v2 5/5] MIPS: Move cache setup to setup_arch().

[Shane McDonald]

On Mon, Jun 11, 2012 at 12:32 PM, Shane McDonald
<mcdonald.shane@gmail.com> wrote:
> There is a line:
>
> __setup("cca=", cca_setup);
>
> that seems to be used to call cca_setup().  I don't know how
> the __setup() works, so I'm a little lost on the solution myself.
>
> Note that, besides the cca_setup(), there is also a routine
> setcoherentio() that is defined the same way as cca_setup().
> I suspect that suffers from the same problem as cca_setup().

I've been doing a little learning on how the __setup() macro works.
A proposed solution I have is to change from using the __setup()
macro to using early_param() to mark the call to cca_setup().
Functions marked with __setup() are executed late in the boot
process, whereas those marked with early_param() occur
very early in the process.  I have tried this out,
and it solves my problem, but I'm looking for feedback on
whether this is the correct solution.

Unless I get any different feedback, I'll send out a patch with
my change later today.

Shane McDonald

Re: [PATCH v2 5/5] MIPS: Move cache setup to setup_arch().

[David Daney]

On 06/13/2012 08:44 AM, Shane McDonald wrote:
> On Mon, Jun 11, 2012 at 12:32 PM, Shane McDonald
> <mcdonald.shane@gmail.com>  wrote:
>> There is a line:
>>
>> __setup("cca=", cca_setup);
>>
>> that seems to be used to call cca_setup().  I don't know how
>> the __setup() works, so I'm a little lost on the solution myself.
>>
>> Note that, besides the cca_setup(), there is also a routine
>> setcoherentio() that is defined the same way as cca_setup().
>> I suspect that suffers from the same problem as cca_setup().
>
> I've been doing a little learning on how the __setup() macro works.
> A proposed solution I have is to change from using the __setup()
> macro to using early_param() to mark the call to cca_setup().

This is the exact change I was going to suggest.

> Functions marked with __setup() are executed late in the boot
> process, whereas those marked with early_param() occur
> very early in the process.  I have tried this out,
> and it solves my problem, but I'm looking for feedback on
> whether this is the correct solution.
>
> Unless I get any different feedback, I'll send out a patch with
> my change later today.

Assuming that such a patch passes checkpatch.pl and is otherwise clean, 
you could add:

Acked-by: David Daney <david.daney@cavium.com>

Re: [PATCH v2 5/5] MIPS: Move cache setup to setup_arch().

[Ralf Baechle]

On Wed, Jun 13, 2012 at 09:29:50AM -0700, David Daney wrote:

> On 06/13/2012 08:44 AM, Shane McDonald wrote:
> >On Mon, Jun 11, 2012 at 12:32 PM, Shane McDonald
> ><mcdonald.shane@gmail.com>  wrote:
> >>There is a line:
> >>
> >>__setup("cca=", cca_setup);
> >>
> >>that seems to be used to call cca_setup().  I don't know how
> >>the __setup() works, so I'm a little lost on the solution myself.
> >>
> >>Note that, besides the cca_setup(), there is also a routine
> >>setcoherentio() that is defined the same way as cca_setup().
> >>I suspect that suffers from the same problem as cca_setup().
> >
> >I've been doing a little learning on how the __setup() macro works.
> >A proposed solution I have is to change from using the __setup()
> >macro to using early_param() to mark the call to cca_setup().
> 
> This is the exact change I was going to suggest.
> 
> >Functions marked with __setup() are executed late in the boot
> >process, whereas those marked with early_param() occur
> >very early in the process.  I have tried this out,
> >and it solves my problem, but I'm looking for feedback on
> >whether this is the correct solution.
> >
> >Unless I get any different feedback, I'll send out a patch with
> >my change later today.
> 
> Assuming that such a patch passes checkpatch.pl and is otherwise
> clean, you could add:

Sounds fine to me too, as long as the it's still being called early enough.
The value is being used for all TLB mappings so it should be ready
latest when the TLB mappings or pagetable entries are computed.

  Ralf