[RFC][patch 2/10] Multiple vector domain support - cpu and domain

[RFC][patch 2/10] Multiple vector domain support - cpu and domain

[Kenji Kaneshige]

This patch add the code to handle the relationship between cpu and
domains. We need more consideration about how to separate vector
domains.

Signed-off-by: Kenji Kaneshige <kaneshige.kenji@jp.fujitsu.com>

---

 linux-2.6.13-rc1-kanesige/arch/ia64/kernel/irq_ia64.c |   24 ++++++++++++++++++
 linux-2.6.13-rc1-kanesige/arch/ia64/kernel/setup.c    |    2 +
 linux-2.6.13-rc1-kanesige/arch/ia64/kernel/smpboot.c  |    2 +
 linux-2.6.13-rc1-kanesige/include/asm-ia64/hw_irq.h   |   13 +++++++++
 4 files changed, 41 insertions(+)

diff -puN arch/ia64/kernel/irq_ia64.c~vector-domain-ia64-attach-cpu arch/ia64/kernel/irq_ia64.c
--- linux-2.6.13-rc1/arch/ia64/kernel/irq_ia64.c~vector-domain-ia64-attach-cpu	2005-07-13 14:51:40.000000000 +0900
+++ linux-2.6.13-rc1-kanesige/arch/ia64/kernel/irq_ia64.c	2005-07-13 14:51:40.000000000 +0900
@@ -286,3 +286,27 @@ ia64_send_ipi (int cpu, int vector, int 
 
 	writeq(ipi_data, ipi_addr);
 }
+
+void __init
+ia64_vector_domain_init (void)
+{
+	/* Attach BSP to domain #0 */
+	ia64_attach_cpu_to_domain(0);
+}
+
+#ifdef CONFIG_VECTOR_DOMAIN
+int ia64_cpu_domain_map[NR_CPUS];
+cpumask_t ia64_domain_cpumask[NR_VECTOR_DOMAINS];
+
+void __devinit
+ia64_attach_cpu_to_domain (int cpu)
+{
+	static int domain = -1;
+
+	/* Simple Round Robin for now */
+	if (++domain >= NR_VECTOR_DOMAINS)
+		domain = 0;
+	ia64_cpu_domain_map[cpu] = domain;
+	cpu_set(cpu, ia64_domain_cpumask[domain]);
+}
+#endif /* CONFIG_VECTOR_DOMAIN */
diff -puN arch/ia64/kernel/setup.c~vector-domain-ia64-attach-cpu arch/ia64/kernel/setup.c
--- linux-2.6.13-rc1/arch/ia64/kernel/setup.c~vector-domain-ia64-attach-cpu	2005-07-13 14:51:40.000000000 +0900
+++ linux-2.6.13-rc1-kanesige/arch/ia64/kernel/setup.c	2005-07-13 14:51:40.000000000 +0900
@@ -406,6 +406,8 @@ setup_arch (char **cmdline_p)
 
 	cpu_init();	/* initialize the bootstrap CPU */
 
+	ia64_vector_domain_init();
+
 #ifdef CONFIG_ACPI_BOOT
 	acpi_boot_init();
 #endif
diff -puN arch/ia64/kernel/smpboot.c~vector-domain-ia64-attach-cpu arch/ia64/kernel/smpboot.c
--- linux-2.6.13-rc1/arch/ia64/kernel/smpboot.c~vector-domain-ia64-attach-cpu	2005-07-13 14:51:40.000000000 +0900
+++ linux-2.6.13-rc1-kanesige/arch/ia64/kernel/smpboot.c	2005-07-13 14:51:40.000000000 +0900
@@ -766,6 +766,8 @@ __cpu_up (unsigned int cpu)
 	int ret;
 	int sapicid;
 
+	ia64_attach_cpu_to_domain(cpu);
+
 	sapicid = ia64_cpu_to_sapicid[cpu];
 	if (sapicid = -1)
 		return -EINVAL;
diff -puN include/asm-ia64/hw_irq.h~vector-domain-ia64-attach-cpu include/asm-ia64/hw_irq.h
--- linux-2.6.13-rc1/include/asm-ia64/hw_irq.h~vector-domain-ia64-attach-cpu	2005-07-13 14:51:40.000000000 +0900
+++ linux-2.6.13-rc1-kanesige/include/asm-ia64/hw_irq.h	2005-07-13 14:51:40.000000000 +0900
@@ -79,6 +79,18 @@ enum {
 extern __u8 isa_irq_to_vector_map[16];
 #define isa_irq_to_vector(x)	isa_irq_to_vector_map[(x)]
 
+#ifdef CONFIG_VECTOR_DOMAIN
+extern void ia64_attach_cpu_to_domain(int cpu);
+extern int ia64_cpu_domain_map[NR_CPUS];
+extern cpumask_t ia64_domain_cpumask[NR_VECTOR_DOMAINS];
+#define ia64_cpu_to_domain(cpu)			(ia64_cpu_domain_map[cpu])
+#define ia64_domain_to_cpumask(domain)		(ia64_domain_cpumask[domain])
+#else
+#define ia64_attach_cpu_to_domain(cpu)		do {} while (0)
+#define ia64_cpu_to_domain(cpu)			(cpu^cpu)
+#define ia64_domain_to_cpumask(domain)		(cpu_online_map)
+#endif /* CONFIG_VECTOR_DOMAIN */
+
 extern struct hw_interrupt_type irq_type_ia64_lsapic;	/* CPU-internal interrupt controller */
 
 extern int assign_irq_vector_nopanic (int irq); /* allocate a free vector without panic */
@@ -86,6 +98,7 @@ extern int assign_irq_vector (int irq);	
 extern void free_irq_vector (int vector);
 extern void ia64_send_ipi (int cpu, int vector, int delivery_mode, int redirect);
 extern void register_percpu_irq (ia64_vector vec, struct irqaction *action);
+extern void __init ia64_vector_domain_init(void);
 
 static inline void
 hw_resend_irq (struct hw_interrupt_type *h, unsigned int vector)

_

Re: [RFC][patch 2/10] Multiple vector domain support - cpu and domain management

[Ashok Raj]

On Thu, Jul 14, 2005 at 06:20:51PM +0900, Kenji Kaneshige wrote:
> 
> This patch add the code to handle the relationship between cpu and
> domains. We need more consideration about how to separate vector
> domains.
> 
> Signed-off-by: Kenji Kaneshige <kaneshige.kenji@jp.fujitsu.com>
> 
> ---
> 

Should we consider this based on some mach-* like schemes, so for e.g in
NUMA case we could use node as a VECTOR_DOMAIN? ... or maybe we could
somehow tie this into CPUSETS?..
 
> +
> +#ifdef CONFIG_VECTOR_DOMAIN
> +int ia64_cpu_domain_map[NR_CPUS];
> +cpumask_t ia64_domain_cpumask[NR_VECTOR_DOMAINS];
> +
> +void __devinit
> +ia64_attach_cpu_to_domain (int cpu)
> +{
> +	static int domain = -1;
> +
> +	/* Simple Round Robin for now */
> +	if (++domain >= NR_VECTOR_DOMAINS)
> +		domain = 0;
> +	ia64_cpu_domain_map[cpu] = domain;
> +	cpu_set(cpu, ia64_domain_cpumask[domain]);
> +}
> +#endif /* CONFIG_VECTOR_DOMAIN */

-- 
Cheers,
Ashok Raj
- Open Source Technology Center

Re: [RFC][patch 2/10] Multiple vector domain support - cpu and domain management

[david mosberger]

I didn't have time to look at these patches yet, but what I was
wondering: SGI Altix already does per-CPU interrupts.  Do we really
need another abstraction or could you use the same approach as SGI?

  --david

On 7/14/05, Ashok Raj <ashok.raj@intel.com> wrote:
> On Thu, Jul 14, 2005 at 06:20:51PM +0900, Kenji Kaneshige wrote:
> >
> > This patch add the code to handle the relationship between cpu and
> > domains. We need more consideration about how to separate vector
> > domains.
> >
> > Signed-off-by: Kenji Kaneshige <kaneshige.kenji@jp.fujitsu.com>
> >
> > ---
> >
> 
> Should we consider this based on some mach-* like schemes, so for e.g in
> NUMA case we could use node as a VECTOR_DOMAIN? ... or maybe we could
> somehow tie this into CPUSETS?..
> 
> > +
> > +#ifdef CONFIG_VECTOR_DOMAIN
> > +int ia64_cpu_domain_map[NR_CPUS];
> > +cpumask_t ia64_domain_cpumask[NR_VECTOR_DOMAINS];
> > +
> > +void __devinit
> > +ia64_attach_cpu_to_domain (int cpu)
> > +{
> > +     static int domain = -1;
> > +
> > +     /* Simple Round Robin for now */
> > +     if (++domain >= NR_VECTOR_DOMAINS)
> > +             domain = 0;
> > +     ia64_cpu_domain_map[cpu] = domain;
> > +     cpu_set(cpu, ia64_domain_cpumask[domain]);
> > +}
> > +#endif /* CONFIG_VECTOR_DOMAIN */
> 
> --
> Cheers,
> Ashok Raj
> - Open Source Technology Center
> -
> To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [RFC][patch 2/10] Multiple vector domain support - cpu and domain

[Zwane Mwaikambo]

On Thu, 14 Jul 2005, Ashok Raj wrote:

> On Thu, Jul 14, 2005 at 06:20:51PM +0900, Kenji Kaneshige wrote:
> > 
> > This patch add the code to handle the relationship between cpu and
> > domains. We need more consideration about how to separate vector
> > domains.
> > 
> > Signed-off-by: Kenji Kaneshige <kaneshige.kenji@jp.fujitsu.com>
> > 
> > ---
> > 
> 
> Should we consider this based on some mach-* like schemes, so for e.g in
> NUMA case we could use node as a VECTOR_DOMAIN? ... or maybe we could
> somehow tie this into CPUSETS?..

There is, on some hardware, a requirement that only specific processors 
may service specific devices due to interrupt controller dispatching, 
these are usually tied together as nodes and it'd be a lot easier 
determining interrupt domain participants if we didn't have to deal with 
arbitrary cpu masks but had a fixed group of processors and hence could 
figure out which node they all belonged to.

	Zwane

Re: [RFC][patch 2/10] Multiple vector domain support - cpu and domain

[Kenji Kaneshige]

Ashok Raj wrote:
> On Thu, Jul 14, 2005 at 06:20:51PM +0900, Kenji Kaneshige wrote:
> 
>>This patch add the code to handle the relationship between cpu and
>>domains. We need more consideration about how to separate vector
>>domains.
>>
>>Signed-off-by: Kenji Kaneshige <kaneshige.kenji@jp.fujitsu.com>
>>
>>---
>>
> 
> 
> Should we consider this based on some mach-* like schemes, so for e.g in
> NUMA case we could use node as a VECTOR_DOMAIN? ... or maybe we could
> somehow tie this into CPUSETS?..
>  
> 

I think some hardware implements hardware interrupt
redirection on each node. For such system, I think it
is good idea to use NUMA node as a VECTOR_DOMAIN to
make it possible to use hardware interrupt redirection.
Currently I don't have other reasons to use the set of
processors as a vector domain instead of per-CPU approach.

Does anyone have other reasons to use other approach than
per-CPU approach?

Thanks,
Kenji Kaneshige