IRQ setup on multicore systems (routing, balancing, etc)

IRQ setup on multicore systems (routing, balancing, etc)

[Mason]

Hello everyone,

I have a few very naive questions about interrupts.

How are interrupts set up on multicore systems?

If I write a device tree node for some peripheral, am I supposed
to specify which core each interrupt should be routed to?

On my system, there is a custom interrupt controller, but the ARM
chip also provides a Generic Interrupt Controller (GIC).

Am I supposed to use both, or can I use just the GIC?
(I suspect the answer is very platform-dependent.)

I've seen a lot of articles discussing interrupt "management"
on x86 (with APIC) but my search-foo is failing me for more
generic Linux "Howto set up". Are there good references?

I suppose I should take a look at these?
Documentation/devicetree/bindings/interrupt-controller/*
Documentation/devicetree/bindings/arm/gic.txt


For my own reference:

ARM Generic Interrupt Controller Architecture Specification v1.0 (IHI0048A)
Cortex-A9 MPCore (Revision: r3p0) Technical Reference Manual

Regards.

IRQ setup on multicore systems (routing, balancing, etc)

[Mason]

On 04/08/2015 15:41, Mason wrote:

> I have a few very naive questions about interrupts.
> 
> How are interrupts set up on multicore systems?
> 
> If I write a device tree node for some peripheral, am I supposed
> to specify which core each interrupt should be routed to?
> 
> On my system, there is a custom interrupt controller, but the ARM
> chip also provides a Generic Interrupt Controller (GIC).
> 
> Am I supposed to use both, or can I use just the GIC?
> (I suspect the answer is very platform-dependent.)
> 
> I've seen a lot of articles discussing interrupt "management"
> on x86 (with APIC) but my search-foo is failing me for more
> generic Linux "Howto set up". Are there good references?
> 
> I suppose I should take a look at these?
> Documentation/devicetree/bindings/interrupt-controller/*
> Documentation/devicetree/bindings/arm/gic.txt
> 
> 
> For my own reference:
> 
> ARM Generic Interrupt Controller Architecture Specification v1.0 (IHI0048A)
> Cortex-A9 MPCore (Revision: r3p0) Technical Reference Manual

This thread seems relevant:

[RFC] ARM: Let GIC to route IRQs to any allowed CPUs
http://thread.gmane.org/gmane.linux.ports.arm.kernel/102251

Russell's answer particularly so:
http://article.gmane.org/gmane.linux.ports.arm.kernel/102289

On 2011-01-12, Russell King wrote:

> There's quite a bit of history behind interrupt balancing across CPU
> cores, and it's not a simple issue to get to grips with.
> 
> I believe the x86 kernel uses software to balance interrupts across the
> cores using an algorithm in the kernel.  This tries to distribute the
> interrupts between the cores.  When this was tried on ARM, although it
> moved interrupts around the cores, it was no better than having all
> interrupts routed to one core.
> 
> On x86 they now do away with their kernel algorithm, and instead run a
> boot-time utility (irqbalance) which does a one-time distribution of
> interrupts across the cores.  The idea is that it is more important to
> keep an interrupt handler running on the same core than it is to
> constantly switch it between the cores.
> 
> If a handler keeps switching between the cores, you have to migrate cache
> lines between the cores, which adds to the cache coherency traffic, and
> on x86 results in a reduction in performance.
> 
> So, with all that in mind, when I was sorting out the initial SMP merging,
> I tried out various algorithms for automatic interrupt distribution, and
> never got any of them to work satisfactorily.  In light of discussions with
> x86 folk, particularly Arjan van de Ven, I decided not to merge any of them
> and leave it as a matter for userspace policy to control how interrupts are
> distributed to the cores - just like the majority of x86 platforms now do.
> 
> AFAIK, there's nothing stopping anyone running 'irqbalance' (the x86
> utility) on ARM - it should just be accessing procfs files.  See
> http://irqbalance.org/ for more information on the program, and on the
> issues surrounding IRQ distribution in SMP systems.

Regards.

IRQ setup on multicore systems (routing, balancing, etc)

[Mason]

On 04/08/2015 15:41, Mason wrote:

> I have a few very naive questions about [snip]

[META]

Are there perhaps more suitable mailing lists than this one
for such beginner questions?

(It looks like LAKML is for experienced Linux devs proposing kernel
changes, or submitting patches for existing ports.)

I lurked in the #devicetree IRC channel, but there was no activity
in ~6 hours (despite 40 people connected).

Hmmm, I'll give kernelnewbies a try.
http://kernelnewbies.org/mailinglistguidelines

Regards.

IRQ setup on multicore systems (routing, balancing, etc)

[Russell King - ARM Linux]

On Thu, Aug 06, 2015 at 05:17:06PM +0200, Mason wrote:
> On 04/08/2015 15:41, Mason wrote:
> 
> > I have a few very naive questions about [snip]
> 
> [META]
> 
> Are there perhaps more suitable mailing lists than this one
> for such beginner questions?

I thought your message yesterday indicated you'd found the answer.

You don't setup any kind of routing between CPU cores and devices.
At boot, initially CPU 0 will "own" all interrupts.

You then run a (fixed) version of irqbalanced (preferably the older
version which is not so x86-centric, eg, 0.56) which then distributes
the interrupts between the cores according to what the interrupt is
used for and the load it is imposing on the system.

-- 
FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up
according to speedtest.net.

IRQ setup on multicore systems (routing, balancing, etc)

[Mason]

On 06/08/2015 17:26, Russell King - ARM Linux wrote:
> On Thu, Aug 06, 2015 at 05:17:06PM +0200, Mason wrote:
>> On 04/08/2015 15:41, Mason wrote:
>>
>>> I have a few very naive questions about [snip]
>>
>> [META]
>>
>> Are there perhaps more suitable mailing lists than this one
>> for such beginner questions?
> 
> I thought your message yesterday indicated you'd found the answer.

I have so many questions, I could write a (very boring) book.

> You don't setup any kind of routing between CPU cores and devices.
> At boot, initially CPU 0 will "own" all interrupts.

OK.

It doesn't help that the port I'm working on doesn't follow
"established guidelines". In this case, interrupt routing is
hard-coded in the platform code:

static inline void tangox_enable(struct irq_data *data)
{
	int bit = data->irq - IRQ_CONTROLLER_IRQ_BASE;
#ifdef ENABLE_INTERRUPT_ON_CPU1
	if(route_interrupt_to_cpu1(bit))
		tangox_fiq_enable(bit);
	else
#endif
		tangox_irq_enable(bit);
}

(tangox_enable is called in the .irq_startup struct irq_chip callback.)

Throw weird naming in the mix (I thought FIQ was reserved for
the secure world, and my Linux runs in non-secure) and my
confusion level is up to 11.

> You then run a (fixed) version of irqbalanced (preferably the older
> version which is not so x86-centric, eg, 0.56) which then distributes
> the interrupts between the cores according to what the interrupt is
> used for and the load it is imposing on the system.

OK. Thanks for pointing that out.

Unfortunately, I guess I should have asked the more fundamental
questions first. Balancing IRQs only makes sense once IRQs are
working.

I don't understand how to convert my board file mess into the
equivalent device tree, and the platform IRQ code is ugly.

I still haven't found a good reference explaining "DT for dummies".
I don't know how the GIC and platform INTC interact.

I tried randomly copy/pasting DT snippets, but unsurprisingly,
nothing has worked so far.

(I use Mans Rullgard's port as a reference as much as I can.)
https://github.com/mansr/linux-tangox

/dts-v1/;

/ {
	compatible = "sigma,tango4-soc";

	#address-cells = <1>;
	#size-cells = <1>;

	gic: interrupt-controller at 20001000 {
		compatible = "arm,cortex-a9-gic";
		interrupt-controller;
		#interrupt-cells = <3>;
		reg = <0x20001000 0x1000>,
		      <0x20000100 0x0100>;
	};

	soc {
		compatible = "simple-bus";
		interrupt-parent = <&irqintc>;
		#address-cells = <1>;
		#size-cells = <1>;
		ranges;

		uart0: serial at 10700 {
			compatible = "ns16550a";
			reg = <0x10700 0x100>;
			clock-frequency = <7372800>;
			reg-shift = <2>;
			no-loopback-test;
		};

		eth0: emac at 26000 {
			compatible = "sigma,smp8640-emac";
			reg = <0x26000 0x800>;
			interrupts = <38>;
			clocks = <396000000>;
		};
	};

	cpublock: cpublock {
		compatible = "simple-bus";
		reg = <0x60000 0x10000>;
		ranges = <0x0 0x60000 0x10000>;
		interrupt-parent = <&irqintc>;
		#address-cells = <1>;
		#size-cells = <1>;

		intc: intc at e000 {
			compatible = "sigma,smp8640-intc";
			reg = <0xe000 0x1000>;
			ranges = <0x0 0xe000 0x1000>;
			interrupt-parent = <&gic>;
			interrupt-controller;
			#address-cells = <1>;
			#size-cells = <1>;

			irqintc: irq at 000 {
				reg = <0x000 0x100>;
				#interrupt-cells = <2>;
				interrupts = <2>;
				label = "IRQ";
			};

			fiqintc: fiq at 100 {
				reg = <0x100 0x100>;
				#interrupt-cells = <2>;
				interrupts = <3>;
				label = "FIQ";
			};

			iiqintc: iiq at 300 {
				reg = <0x300 0x100>;
				#interrupt-cells = <2>;
				interrupts = <4>;
				label = "IIQ";
			};
		};
	};

};

Bleh! I need a break. Be back in a few weeks.

Regards.

IRQ setup on multicore systems (routing, balancing, etc)

[Russell King - ARM Linux]

On Thu, Aug 06, 2015 at 06:14:46PM +0200, Mason wrote:
> It doesn't help that the port I'm working on doesn't follow
> "established guidelines". In this case, interrupt routing is
> hard-coded in the platform code:
> 
> static inline void tangox_enable(struct irq_data *data)
> {
> 	int bit = data->irq - IRQ_CONTROLLER_IRQ_BASE;
> #ifdef ENABLE_INTERRUPT_ON_CPU1
> 	if(route_interrupt_to_cpu1(bit))
> 		tangox_fiq_enable(bit);
> 	else
> #endif
> 		tangox_irq_enable(bit);
> }
> 
> (tangox_enable is called in the .irq_startup struct irq_chip callback.)
> 
> Throw weird naming in the mix (I thought FIQ was reserved for
> the secure world, and my Linux runs in non-secure) and my
> confusion level is up to 11.

Looking at the above code, I must say that it looks like the code is on
some kind of drugs.  The code implies that FIQs are routed to CPU1, which
is obviously a totally insane.  I think I'd be as confused as you over
that.

Are you basing your idea that certain interrupts need to be routed to a
certain CPU on this code, or on practical experience?  It may be better
to experiment with the hardware to find out what's possible rather than
rely on hacky undocumented code such as the above.

-- 
FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up
according to speedtest.net.