[PATCH 6/7] sparc32,leon: operate on boot-cpu IRQ controller registers

[PATCH 6/7] sparc32,leon: operate on boot-cpu IRQ controller registers

[Daniel Hellstrom]

Each CPU has a separate set of IRQ controller registers, this
patch makes sure that the boot-cpu registers are used instead
of CPU0's. Note that there are other parts of the SPARC32/LEON
port which does not support booting on other than CPU0 anyway,
however this this cleans up the IRQ controller layer in that
regard.

Signed-off-by: Daniel Hellstrom <daniel@gaisler.com>
---
 arch/sparc/include/asm/leon.h   |    1 +
 arch/sparc/kernel/leon_kernel.c |   14 ++++++++------
 2 files changed, 9 insertions(+), 6 deletions(-)

diff --git a/arch/sparc/include/asm/leon.h b/arch/sparc/include/asm/leon.h
index 31fb2ac..1776f71 100644
--- a/arch/sparc/include/asm/leon.h
+++ b/arch/sparc/include/asm/leon.h
@@ -335,6 +335,7 @@ extern int leon_flush_needed(void);
 extern void leon_switch_mm(void);
 extern int srmmu_swprobe_trace;
 extern int leon3_ticker_irq;
+extern int leon3_boot_cpu;
 
 #ifdef CONFIG_SMP
 extern int leon_smp_nrcpus(void);
diff --git a/arch/sparc/kernel/leon_kernel.c b/arch/sparc/kernel/leon_kernel.c
index 80ab271..26acc75 100644
--- a/arch/sparc/kernel/leon_kernel.c
+++ b/arch/sparc/kernel/leon_kernel.c
@@ -31,6 +31,7 @@ int leondebug_irq_disable;
 int leon_debug_irqout;
 static int dummy_master_l10_counter;
 unsigned long amba_system_id;
+int leon3_boot_cpu;
 static DEFINE_SPINLOCK(leon_irq_lock);
 
 unsigned long leon3_gptimer_irq; /* interrupt controller irq number */
@@ -79,8 +80,8 @@ void leon_eirq_setup(unsigned int eirq)
 	 */
 	irq_link(veirq);
 	mask = 1 << eirq;
-	oldmask = LEON3_BYPASS_LOAD_PA(LEON_IMASK(0));
-	LEON3_BYPASS_STORE_PA(LEON_IMASK(0), (oldmask | mask));
+	oldmask = LEON3_BYPASS_LOAD_PA(LEON_IMASK(leon3_boot_cpu));
+	LEON3_BYPASS_STORE_PA(LEON_IMASK(leon3_boot_cpu), (oldmask | mask));
 	sparc_leon_eirq = eirq;
 }
 
@@ -105,8 +106,8 @@ static void leon_unmask_irq(struct irq_data *data)
 
 	mask = (unsigned long)data->chip_data;
 	spin_lock_irqsave(&leon_irq_lock, flags);
-	oldmask = LEON3_BYPASS_LOAD_PA(LEON_IMASK(0));
-	LEON3_BYPASS_STORE_PA(LEON_IMASK(0), (oldmask | mask));
+	oldmask = LEON3_BYPASS_LOAD_PA(LEON_IMASK(leon3_boot_cpu));
+	LEON3_BYPASS_STORE_PA(LEON_IMASK(leon3_boot_cpu), (oldmask | mask));
 	spin_unlock_irqrestore(&leon_irq_lock, flags);
 }
 
@@ -116,8 +117,8 @@ static void leon_mask_irq(struct irq_data *data)
 
 	mask = (unsigned long)data->chip_data;
 	spin_lock_irqsave(&leon_irq_lock, flags);
-	oldmask = LEON3_BYPASS_LOAD_PA(LEON_IMASK(0));
-	LEON3_BYPASS_STORE_PA(LEON_IMASK(0), (oldmask & ~mask));
+	oldmask = LEON3_BYPASS_LOAD_PA(LEON_IMASK(leon3_boot_cpu));
+	LEON3_BYPASS_STORE_PA(LEON_IMASK(leon3_boot_cpu), (oldmask & ~mask));
 	spin_unlock_irqrestore(&leon_irq_lock, flags);
 }
 
@@ -205,6 +206,7 @@ void __init leon_init_timers(irq_handler_t counter_fn)
 	leon_debug_irqout = 0;
 	master_l10_counter = (unsigned int *)&dummy_master_l10_counter;
 	dummy_master_l10_counter = 0;
+	leon3_boot_cpu = hard_smp_processor_id();
 
 	rootnp = of_find_node_by_path("/ambapp0");
 	if (!rootnp)
-- 
1.5.4

Re: [PATCH 6/7] sparc32,leon: operate on boot-cpu IRQ controller

[Sam Ravnborg]

On Tue, Apr 19, 2011 at 06:07:18PM +0200, Daniel Hellstrom wrote:
> Each CPU has a separate set of IRQ controller registers, this
> patch makes sure that the boot-cpu registers are used instead
> of CPU0's. Note that there are other parts of the SPARC32/LEON
> port which does not support booting on other than CPU0 anyway,
> however this this cleans up the IRQ controller layer in that
> regard.
> 
> Signed-off-by: Daniel Hellstrom <daniel@gaisler.com>
> ---
>  arch/sparc/include/asm/leon.h   |    1 +
>  arch/sparc/kernel/leon_kernel.c |   14 ++++++++------
>  2 files changed, 9 insertions(+), 6 deletions(-)
> 
> diff --git a/arch/sparc/include/asm/leon.h b/arch/sparc/include/asm/leon.h
> index 31fb2ac..1776f71 100644
> --- a/arch/sparc/include/asm/leon.h
> +++ b/arch/sparc/include/asm/leon.h
> @@ -335,6 +335,7 @@ extern int leon_flush_needed(void);
>  extern void leon_switch_mm(void);
>  extern int srmmu_swprobe_trace;
>  extern int leon3_ticker_irq;
> +extern int leon3_boot_cpu;

We already have boot_cpu_id - defiend in smp_32.c.
Could it be used rather than a leon specific variable?

We would need to define boot_cpu_id also in the non-SMP case
to do so - but this could also clean up code
in sun4d_irq.

Note: leon actually set this variable in head_32.S already.

	Sam

Re: [PATCH 6/7] sparc32,leon: operate on boot-cpu IRQ controller

[Daniel Hellstrom]

Sam Ravnborg wrote:

>On Tue, Apr 19, 2011 at 06:07:18PM +0200, Daniel Hellstrom wrote:
>  
>
>>Each CPU has a separate set of IRQ controller registers, this
>>patch makes sure that the boot-cpu registers are used instead
>>of CPU0's. Note that there are other parts of the SPARC32/LEON
>>port which does not support booting on other than CPU0 anyway,
>>however this this cleans up the IRQ controller layer in that
>>regard.
>>
>>Signed-off-by: Daniel Hellstrom <daniel@gaisler.com>
>>---
>> arch/sparc/include/asm/leon.h   |    1 +
>> arch/sparc/kernel/leon_kernel.c |   14 ++++++++------
>> 2 files changed, 9 insertions(+), 6 deletions(-)
>>
>>diff --git a/arch/sparc/include/asm/leon.h b/arch/sparc/include/asm/leon.h
>>index 31fb2ac..1776f71 100644
>>--- a/arch/sparc/include/asm/leon.h
>>+++ b/arch/sparc/include/asm/leon.h
>>@@ -335,6 +335,7 @@ extern int leon_flush_needed(void);
>> extern void leon_switch_mm(void);
>> extern int srmmu_swprobe_trace;
>> extern int leon3_ticker_irq;
>>+extern int leon3_boot_cpu;
>>    
>>
>
>We already have boot_cpu_id - defiend in smp_32.c.
>Could it be used rather than a leon specific variable?
>
>We would need to define boot_cpu_id also in the non-SMP case
>to do so - but this could also clean up code
>in sun4d_irq.
>  
>
Yes, it might remove a ifdef to the expence of a bigger footprint.

>Note: leon actually set this variable in head_32.S already.
>  
>
I agree that it would be nicer, however I think the SUNs will never be 
booted on CPU1 anyway? Anyways, I think this should go into another 
patch series. I'm very interested in supporting booting on CPU1 for 
LEON, that is really nice in AMP systems where for example RTEMS is 
running on CPU0 and Linux on CPU1. Today RTEMS supports running on 
CPU!=0, so it is not a big issue right now but is not very flexible.

As I state in the comment this patch does not aim to fix everything just 
parts of the IRQ Controller code, there are a lot of other stuff that 
needs to be done in order to support this, not only in Linux the PROM as 
well. I plan to come back later with a patch set for this.

Sam, is that acceptable for now?

Daniel

Re: [PATCH 6/7] sparc32,leon: operate on boot-cpu IRQ controller

[crn]

On Wed, 20 Apr 2011, Daniel Hellstrom wrote:

>> 
> I agree that it would be nicer, however I think the SUNs will never be booted 
> on CPU1 anyway? Anyways, I think this should go into another patch series.

You cannot assume this, sun4d and AFAIK all 64bit systems will boot on
the first non-blacklisted CPU found.
For example, it is perfectly OK to have no CPU present on board0 slot0
in which case the SS1000 will boot using CPU1.

Re: [PATCH 6/7] sparc32,leon: operate on boot-cpu IRQ controller

[Daniel Hellstrom]

crn@post.netunix.com wrote:

>
>
> On Wed, 20 Apr 2011, Daniel Hellstrom wrote:
>
>>>
>> I agree that it would be nicer, however I think the SUNs will never 
>> be booted on CPU1 anyway? Anyways, I think this should go into 
>> another patch series.
>
>
> You cannot assume this, sun4d and AFAIK all 64bit systems will boot on
> the first non-blacklisted CPU found.
> For example, it is perfectly OK to have no CPU present on board0 slot0
> in which case the SS1000 will boot using CPU1.

Sorry I meant the UP 32-bit SUNs, some if not all SMP 32-bit SUNs 
apparently use the boot_cpu_id Sam is talking about. Have you tried 
booting on CPU1 with a UP Linux kernel on the SS1000 machine?

I still believe that the changes should go into a separate patch series 
though.

Thanks,
Daniel

Re: [PATCH 6/7] sparc32,leon: operate on boot-cpu IRQ controller

[David Miller]

From: Daniel Hellstrom <daniel@gaisler.com>
Date: Wed, 20 Apr 2011 10:12:25 +0200

> Sorry I meant the UP 32-bit SUNs, some if not all SMP 32-bit SUNs
> apparently use the boot_cpu_id Sam is talking about. Have you tried
> booting on CPU1 with a UP Linux kernel on the SS1000 machine?

Regardless, we should make it work.

It was terrible when I had to fix this on sparc64.

Re: [PATCH 6/7] sparc32,leon: operate on boot-cpu IRQ controller

[Daniel Hellstrom]

David Miller wrote:

>From: Daniel Hellstrom <daniel@gaisler.com>
>Date: Wed, 20 Apr 2011 10:12:25 +0200
>
>  
>
>>Sorry I meant the UP 32-bit SUNs, some if not all SMP 32-bit SUNs
>>apparently use the boot_cpu_id Sam is talking about. Have you tried
>>booting on CPU1 with a UP Linux kernel on the SS1000 machine?
>>    
>>
>
>Regardless, we should make it work.
>  
>
I totally agree with you. And it is important for the LEON family as 
well. New quad-core LEONs are designed to be able to run different 
operating systems on different CPUs, or two linux instances in parallel. 
So the hardware supports it, and I believe the SUNs do too.

>It was terrible when I had to fix this on sparc64.
>  
>
I think it is a quite big job, and not really related. I can drop this 
cleanup patch now and make a proper implementation later this year. What 
I want is get more important stuff working with genirq first.

Daniel

Re: [PATCH 6/7] sparc32,leon: operate on boot-cpu IRQ controller

[David Miller]

From: Daniel Hellstrom <daniel@gaisler.com>
Date: Wed, 20 Apr 2011 10:35:55 +0200

> David Miller wrote:
> 
>>It was terrible when I had to fix this on sparc64.
>>  
> I think it is a quite big job, and not really related. I can drop this
> cleanup patch now and make a proper implementation later this
> year. What I want is get more important stuff working with genirq
> first.

So like you said, leave this cleanup for later.

Re: [PATCH 6/7] sparc32,leon: operate on boot-cpu IRQ controller

[Daniel Hellstrom]

David Miller wrote:

>From: Daniel Hellstrom <daniel@gaisler.com>
>Date: Wed, 20 Apr 2011 10:35:55 +0200
>
>  
>
>>David Miller wrote:
>>
>>    
>>
>>>It was terrible when I had to fix this on sparc64.
>>> 
>>>      
>>>
>>I think it is a quite big job, and not really related. I can drop this
>>cleanup patch now and make a proper implementation later this
>>year. What I want is get more important stuff working with genirq
>>first.
>>    
>>
>
>So like you said, leave this cleanup for later.
>  
>
Ok, since the affinity patch depends on it, I will update according to 
Sams comments and resend it. The affinity patch will be patch 7/7 and 
6/7 should be dropped.

Daniel

Re: [PATCH 6/7] sparc32,leon: operate on boot-cpu IRQ controller

[David Miller]

From: Daniel Hellstrom <daniel@gaisler.com>
Date: Wed, 20 Apr 2011 10:44:26 +0200

> Ok, since the affinity patch depends on it, I will update according to
> Sams comments and resend it. The affinity patch will be patch 7/7 and
> 6/7 should be dropped.

Ok.

Re: [PATCH 6/7] sparc32,leon: operate on boot-cpu IRQ controller

[crn]

On Wed, 20 Apr 2011, Daniel Hellstrom wrote:
> crn@post.netunix.com wrote:
>> On Wed, 20 Apr 2011, Daniel Hellstrom wrote:
>>>> 
>>> I agree that it would be nicer, however I think the SUNs will never be 
>>> booted on CPU1 anyway? Anyways, I think this should go into another patch 
>>> series.
>> 
>> You cannot assume this, sun4d and AFAIK all 64bit systems will boot on
>> the first non-blacklisted CPU found.
>> For example, it is perfectly OK to have no CPU present on board0 slot0
>> in which case the SS1000 will boot using CPU1.
>
> Sorry I meant the UP 32-bit SUNs, some if not all SMP 32-bit SUNs apparently 
> use the boot_cpu_id Sam is talking about. Have you tried booting on CPU1 with 
> a UP Linux kernel on the SS1000 machine?
>
> I still believe that the changes should go into a separate patch series 
> though.

It is legal hardware and OBP wise to have only one CPU fitted in any
random slot and the OBP will report this.
We should support this (slightly unusual but valid) configuration.
In the case of the ss1000 the kernel has a bad case of bit rot and
it would be a very good idea to get a UP kernel working before
trying to fix SMP.

Re: [PATCH 6/7] sparc32,leon: operate on boot-cpu IRQ controller

[Sam Ravnborg]

>>
> Yes, it might remove a ifdef to the expence of a bigger footprint.
>
>> Note: leon actually set this variable in head_32.S already.
>>  
>>
> I agree that it would be nicer, however I think the SUNs will never be  
> booted on CPU1 anyway? Anyways, I think this should go into another  
> patch series.

I have implmented what I have in mind in following patch.
You can add this to your serie and we have this prepared
for further cleanup.

Just add your sob below mine and forward to David.

	Sam

[PATCH]sparc32: always define boot_cpu_id

This patch increases the data footprint with a few bytes in the
expense of more strightforward code in some places.
We may benefit more from this later as we will need the
boot_cpu_id in leon irq functions

Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
---
diff --git a/arch/sparc/include/asm/setup.h b/arch/sparc/include/asm/setup.h
index 2643c62..64718ba 100644
--- a/arch/sparc/include/asm/setup.h
+++ b/arch/sparc/include/asm/setup.h
@@ -11,4 +11,16 @@
 # define COMMAND_LINE_SIZE 256
 #endif
 
+#ifdef __KERNEL__
+
+#ifdef CONFIG_SPARC32
+/* The CPU that was used for booting
+ * Only sun4d + leon may have boot_cpu_id != 0
+ */
+extern unsigned char boot_cpu_id;
+extern unsigned char boot_cpu_id4;
+#endif
+
+#endif /* __KERNEL__ */
+
 #endif /* _SPARC_SETUP_H */
diff --git a/arch/sparc/kernel/leon_smp.c b/arch/sparc/kernel/leon_smp.c
index 8f5de4a..9a2e9b5 100644
--- a/arch/sparc/kernel/leon_smp.c
+++ b/arch/sparc/kernel/leon_smp.c
@@ -50,7 +50,6 @@
 extern ctxd_t *srmmu_ctx_table_phys;
 static int smp_processors_ready;
 extern volatile unsigned long cpu_callin_map[NR_CPUS];
-extern unsigned char boot_cpu_id;
 extern cpumask_t smp_commenced_mask;
 void __init leon_configure_cache_smp(void);
 
diff --git a/arch/sparc/kernel/setup_32.c b/arch/sparc/kernel/setup_32.c
index 7b8b76c..7e1fce8 100644
--- a/arch/sparc/kernel/setup_32.c
+++ b/arch/sparc/kernel/setup_32.c
@@ -103,6 +103,10 @@ static unsigned int boot_flags __initdata = 0;
 /* Exported for mm/init.c:paging_init. */
 unsigned long cmdline_memory_size __initdata = 0;
 
+/* which CPU booted us */
+unsigned char boot_cpu_id;
+unsigned char boot_cpu_id4; /* boot_cpu_id << 2 */
+
 static void
 prom_console_write(struct console *con, const char *s, unsigned n)
 {
diff --git a/arch/sparc/kernel/smp_32.c b/arch/sparc/kernel/smp_32.c
index 91c10fb..4a1d5b7 100644
--- a/arch/sparc/kernel/smp_32.c
+++ b/arch/sparc/kernel/smp_32.c
@@ -37,8 +37,6 @@
 #include "irq.h"
 
 volatile unsigned long cpu_callin_map[NR_CPUS] __cpuinitdata = {0,};
-unsigned char boot_cpu_id = 0;
-unsigned char boot_cpu_id4 = 0; /* boot_cpu_id << 2 */
 
 cpumask_t smp_commenced_mask = CPU_MASK_NONE;
 
diff --git a/arch/sparc/kernel/sun4d_irq.c b/arch/sparc/kernel/sun4d_irq.c
index 14a0435..89cc63c 100644
--- a/arch/sparc/kernel/sun4d_irq.c
+++ b/arch/sparc/kernel/sun4d_irq.c
@@ -438,11 +438,9 @@ static void __init sun4d_init_timers(irq_handler_t counter_fn)
 void __init sun4d_init_sbi_irq(void)
 {
 	struct device_node *dp;
-	int target_cpu = 0;
+	int target_cpu;
 
-#ifdef CONFIG_SMP
 	target_cpu = boot_cpu_id;
-#endif
 	for_each_node_by_name(dp, "sbi") {
 		int devid = of_getintprop_default(dp, "device-id", 0);
 		int board = of_getintprop_default(dp, "board#", 0);

Re: [PATCH 6/7] sparc32,leon: operate on boot-cpu IRQ controller

[Josip Rodin]

On Wed, Apr 20, 2011 at 11:06:54AM +0200, Sam Ravnborg wrote:
> [PATCH]sparc32: always define boot_cpu_id
> 
> +extern unsigned char boot_cpu_id;
> +unsigned char boot_cpu_id;
> -unsigned char boot_cpu_id = 0;
> -	int target_cpu = 0;
> +	int target_cpu;
>  	target_cpu = boot_cpu_id;

You declared it, but you didn't actually define it?

-- 
     2. That which causes joy or happiness.

Re: [PATCH 6/7] sparc32,leon: operate on boot-cpu IRQ controller

[Sam Ravnborg]

On Wed, Apr 20, 2011 at 11:38:26AM +0200, Josip Rodin wrote:
> On Wed, Apr 20, 2011 at 11:06:54AM +0200, Sam Ravnborg wrote:
> > [PATCH]sparc32: always define boot_cpu_id
> > 
> > +extern unsigned char boot_cpu_id;
> > +unsigned char boot_cpu_id;
> > -unsigned char boot_cpu_id = 0;
> > -	int target_cpu = 0;
> > +	int target_cpu;
> >  	target_cpu = boot_cpu_id;
> 
> You declared it, but you didn't actually define it?

It is defined in setup_32.c

	Sam

Re: [PATCH 6/7] sparc32,leon: operate on boot-cpu IRQ controller

[Daniel Hellstrom]

Josip Rodin wrote:

>On Wed, Apr 20, 2011 at 11:06:54AM +0200, Sam Ravnborg wrote:
>  
>
>>[PATCH]sparc32: always define boot_cpu_id
>>
>>+extern unsigned char boot_cpu_id;
>>+unsigned char boot_cpu_id;
>>-unsigned char boot_cpu_id = 0;
>>-	int target_cpu = 0;
>>+	int target_cpu;
>> 	target_cpu = boot_cpu_id;
>>    
>>
>
>You declared it, but you didn't actually define it?
>  
>
There is an error in this patch, setup.h include is missing. I will repost.

Daniel

Re: [PATCH 6/7] sparc32,leon: operate on boot-cpu IRQ controller

[Daniel Hellstrom]

Daniel Hellstrom wrote:

> Josip Rodin wrote:
>
>> On Wed, Apr 20, 2011 at 11:06:54AM +0200, Sam Ravnborg wrote:
>>  
>>
>>> [PATCH]sparc32: always define boot_cpu_id
>>>
>>> +extern unsigned char boot_cpu_id;
>>> +unsigned char boot_cpu_id;
>>> -unsigned char boot_cpu_id = 0;
>>> -    int target_cpu = 0;
>>> +    int target_cpu;
>>>     target_cpu = boot_cpu_id;
>>>   
>>
>>
>> You declared it, but you didn't actually define it?
>>  
>>
> There is an error in this patch, setup.h include is missing. I will 
> repost.

I will not repost at this point. I think this is broken in sparc32 
anyway. boot_cpu_id is always 0 because it is initialized before BSS is 
cleared, and boot_cpu_id is part of BSS. see arch/sparc/kernel/head_32.S

Daniel