[PATCH] powerpc: iSeries has only 256 IRQs

[PATCH] powerpc: iSeries has only 256 IRQs

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 5167 bytes --]

The iSeries Hypervisor only allows us to specify IRQ numbers up to 255 (it
has a u8 field to pass it in).  This patch allows platforms to specify a
maximum to the virtual IRQ numbers we will use and has iSeries set that
to 255.  If not set, the maximum is NR_IRQS - 1 (as before).

Signed-off-by: Stephen Rothwell <sfr@canb.auug.org.au>
---

 arch/powerpc/kernel/irq.c              |   36 ++++++++++++++++++++------------
 arch/powerpc/platforms/iseries/setup.c |    7 ++++++
 include/asm-powerpc/irq.h              |    7 ++++++
 3 files changed, 36 insertions(+), 14 deletions(-)

This has been booted on iSeries partitions that have hardware that would
have previously tried to use IRQ numbers above 255.

-- 
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au

7d01c880856bae31502095bc68784c1518a680cb
diff --git a/arch/powerpc/kernel/irq.c b/arch/powerpc/kernel/irq.c
index bb5c950..57d560c 100644
--- a/arch/powerpc/kernel/irq.c
+++ b/arch/powerpc/kernel/irq.c
@@ -272,18 +272,26 @@ unsigned int virt_irq_to_real_map[NR_IRQ
  * Don't use virtual irqs 0, 1, 2 for devices.
  * The pcnet32 driver considers interrupt numbers < 2 to be invalid,
  * and 2 is the XICS IPI interrupt.
- * We limit virtual irqs to 17 less than NR_IRQS so that when we
- * offset them by 16 (to reserve the first 16 for ISA interrupts)
- * we don't end up with an interrupt number >= NR_IRQS.
+ * We limit virtual irqs to __irq_offet_value less than virt_irq_max so
+ * that when we offset them we don't end up with an interrupt
+ * number > virt_irq_max.
  */
 #define MIN_VIRT_IRQ	3
-#define MAX_VIRT_IRQ	(NR_IRQS - NUM_ISA_INTERRUPTS - 1)
-#define NR_VIRT_IRQS	(MAX_VIRT_IRQ - MIN_VIRT_IRQ + 1)
+
+unsigned int virt_irq_max;
+static unsigned int max_virt_irq;
+static unsigned int nr_virt_irqs;
 
 void
 virt_irq_init(void)
 {
 	int i;
+
+	if ((virt_irq_max == 0) || (virt_irq_max > (NR_IRQS - 1)))
+		virt_irq_max = NR_IRQS - 1;
+	max_virt_irq = virt_irq_max - __irq_offset_value;
+	nr_virt_irqs = max_virt_irq - MIN_VIRT_IRQ + 1;
+
 	for (i = 0; i < NR_IRQS; i++)
 		virt_irq_to_real_map[i] = UNDEFINED_IRQ;
 }
@@ -308,17 +316,17 @@ int virt_irq_create_mapping(unsigned int
 		return real_irq;
 	}
 
-	/* map to a number between MIN_VIRT_IRQ and MAX_VIRT_IRQ */
+	/* map to a number between MIN_VIRT_IRQ and max_virt_irq */
 	virq = real_irq;
-	if (virq > MAX_VIRT_IRQ)
-		virq = (virq % NR_VIRT_IRQS) + MIN_VIRT_IRQ;
+	if (virq > max_virt_irq)
+		virq = (virq % nr_virt_irqs) + MIN_VIRT_IRQ;
 
 	/* search for this number or a free slot */
 	first_virq = virq;
 	while (virt_irq_to_real_map[virq] != UNDEFINED_IRQ) {
 		if (virt_irq_to_real_map[virq] == real_irq)
 			return virq;
-		if (++virq > MAX_VIRT_IRQ)
+		if (++virq > max_virt_irq)
 			virq = MIN_VIRT_IRQ;
 		if (virq == first_virq)
 			goto nospace;	/* oops, no free slots */
@@ -330,8 +338,8 @@ int virt_irq_create_mapping(unsigned int
  nospace:
 	if (!warned) {
 		printk(KERN_CRIT "Interrupt table is full\n");
-		printk(KERN_CRIT "Increase NR_IRQS (currently %d) "
-		       "in your kernel sources and rebuild.\n", NR_IRQS);
+		printk(KERN_CRIT "Increase virt_irq_max (currently %d) "
+		       "in your kernel sources and rebuild.\n", virt_irq_max);
 		warned = 1;
 	}
 	return NO_IRQ;
@@ -349,8 +357,8 @@ unsigned int real_irq_to_virt_slowpath(u
 
 	virq = real_irq;
 
-	if (virq > MAX_VIRT_IRQ)
-		virq = (virq % NR_VIRT_IRQS) + MIN_VIRT_IRQ;
+	if (virq > max_virt_irq)
+		virq = (virq % nr_virt_irqs) + MIN_VIRT_IRQ;
 
 	first_virq = virq;
 
@@ -360,7 +368,7 @@ unsigned int real_irq_to_virt_slowpath(u
 
 		virq++;
 
-		if (virq >= MAX_VIRT_IRQ)
+		if (virq >= max_virt_irq)
 			virq = 0;
 
 	} while (first_virq != virq);
diff --git a/arch/powerpc/platforms/iseries/setup.c b/arch/powerpc/platforms/iseries/setup.c
index 6ce8a40..a6fd9be 100644
--- a/arch/powerpc/platforms/iseries/setup.c
+++ b/arch/powerpc/platforms/iseries/setup.c
@@ -54,6 +54,7 @@
 #include <asm/iseries/hv_lp_event.h>
 #include <asm/iseries/lpar_map.h>
 #include <asm/udbg.h>
+#include <asm/irq.h>
 
 #include "naca.h"
 #include "setup.h"
@@ -684,6 +685,12 @@ static int __init iseries_probe(void)
 	powerpc_firmware_features |= FW_FEATURE_ISERIES;
 	powerpc_firmware_features |= FW_FEATURE_LPAR;
 
+	/*
+	 * The Hypervisor only allows us up to 256 interrupt
+	 * sources (the irq number is passed in a u8).
+	 */
+	virt_irq_max = 255;
+
 	return 1;
 }
 
diff --git a/include/asm-powerpc/irq.h b/include/asm-powerpc/irq.h
index 51f87d9..7bc6d73 100644
--- a/include/asm-powerpc/irq.h
+++ b/include/asm-powerpc/irq.h
@@ -54,6 +54,13 @@
  */
 extern unsigned int virt_irq_to_real_map[NR_IRQS];
 
+/* The maximum virtual IRQ number that we support.  This
+ * can be set by the platform and will be reduced by the
+ * value of __irq_offset_value.  It defaults to and is
+ * capped by (NR_IRQS - 1).
+ */
+extern unsigned int virt_irq_max;
+
 /* Create a mapping for a real_irq if it doesn't already exist.
  * Return the virtual irq as a convenience.
  */
-- 
1.2.4


[-- Attachment #2: Type: application/pgp-signature, Size: 191 bytes --]

Re: [PATCH] powerpc: iSeries has only 256 IRQs

[Olaf Hering]

 On Tue, Apr 04, Stephen Rothwell wrote:

> The iSeries Hypervisor only allows us to specify IRQ numbers up to 255 (it
> has a u8 field to pass it in).  This patch allows platforms to specify a
> maximum to the virtual IRQ numbers we will use and has iSeries set that
> to 255.  If not set, the maximum is NR_IRQS - 1 (as before).

I tried this patch, and the partition dies that way:

vio_register_driver: driver iseries_veth registering
iseries_veth: eth0 attached to iSeries vlan 0 (LPAR map = 0x0005)
Emulex LightPulse Fibre Channel SCSI driver 8.1.4
Copyright(c) 2004-2006 Emulex.  All rights reserved.
PCI: Enabling device: (0000:18:12.0), cmd 3
scsi0 :  on PCI bus 18 device 90 irq 212
PCI: RDL: Device 0x0018:90  I/O Error( 1): 0x0101
PCI: RDL: Device 0x0018:90  I/O Error( 2): 0x0102
PCI: RDL: Device 0x0018:90  I/O Error( 3): 0x0102
PCI: RDL: Device 0x0018:90  I/O Error( 4): 0x0102
Kernel panic - not syncing: PCI: Hardware I/O Error, SRC B6000103, Automatic Reboot Disabled.

 <3>Badness in smp_call_function at arch/powerpc/kernel/smp.c:228
Call Trace:
[C00000000F836E50] [C00000000000E444] .show_stack+0x68/0x1b0 (unreliable)
[C00000000F836EF0] [C000000000264FDC] .program_check_exception+0x1cc/0x5a8
[C00000000F836FC0] [C000000000004A70] program_check_common+0xf0/0x100
--- Exception: 700 at .smp_call_function+0x34/0x1d8
    LR = .smp_call_function+0x28/0x1d8
[C00000000F8372B0] [C000000000334800] zone_names+0x818/0xf18 (unreliable)
[C00000000F837360] [C0000000000376F4] .panic+0xa0/0x194
[C00000000F837400] [C00000000002C354] .CheckReturnCode+0xac/0xc4
[C00000000F837480] [C00000000002CEB0] .iSeries_Read_Long+0x160/0x19c
[C00000000F837530] [D00000000021F5A0] .lpfc_sli_brdrestart+0xc8/0x1c0 [lpfc]
[C00000000F8375D0] [D00000000021F70C] .lpfc_sli_hba_setup+0x74/0x3e0 [lpfc]
[C00000000F837680] [D0000000002328C4] .lpfc_pci_probe_one+0x650/0xa7c [lpfc]
[C00000000F837740] [C0000000001970C4] .pci_device_probe+0x80/0xc4
[C00000000F8377D0] [C0000000001B0DE8] .driver_probe_device+0xa0/0x15c
[C00000000F837870] [C0000000001B1078] .__driver_attach+0xdc/0x164
[C00000000F837900] [C0000000001B051C] .bus_for_each_dev+0x7c/0xd4
[C00000000F8379C0] [C0000000001B0C8C] .driver_attach+0x28/0x40
[C00000000F837A40] [C0000000001AFFBC] .bus_add_driver+0x90/0x170
[C00000000F837AE0] [C0000000001B14A8] .driver_register+0xbc/0xd8
[C00000000F837B70] [C000000000196EA0] .__pci_register_driver+0x8c/0xd8
[C00000000F837C10] [D00000000023C028] .lpfc_init+0x60/0x2828 [lpfc]
[C00000000F837CA0] [C00000000005C564] .sys_init_module+0x1898/0x1bc4
[C00000000F837E30] [C000000000008534] syscall_exit+0x0/0x40
...

Before, it just did not find the card:
...
vio_register_driver: driver iseries_veth registering
iseries_veth: eth0 attached to iSeries vlan 0 (LPAR map = 0x0005)
Emulex LightPulse Fibre Channel SCSI driver 8.1.4
Copyright(c) 2004-2006 Emulex.  All rights reserved.
PCI: Enabling device: (0000:18:12.0), cmd 3
scsi0 :  on PCI bus 18 device 90 irq 235
lpfc 0000:18:12.0: 0:0451 Enable interrupt handler failed
lpfc: probe of 0000:18:12.0 failed with error -38
...

So, there is likely more breakage.

Re: [PATCH] powerpc: iSeries has only 256 IRQs

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 641 bytes --]

On Tue, 4 Apr 2006 15:36:14 +0200 Olaf Hering <olh@suse.de> wrote:
>
>  On Tue, Apr 04, Stephen Rothwell wrote:
> 
> > The iSeries Hypervisor only allows us to specify IRQ numbers up to 255 (it
> > has a u8 field to pass it in).  This patch allows platforms to specify a
> > maximum to the virtual IRQ numbers we will use and has iSeries set that
> > to 255.  If not set, the maximum is NR_IRQS - 1 (as before).
> 
> I tried this patch, and the partition dies that way:

Thanks for the test, I will have a further look.

-- 
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au
http://www.canb.auug.org.au/~sfr/

[-- Attachment #2: Type: application/pgp-signature, Size: 191 bytes --]

Re: [PATCH] powerpc: iSeries has only 256 IRQs

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 824 bytes --]

Hi Olaf,

On Tue, 4 Apr 2006 15:36:14 +0200 Olaf Hering <olh@suse.de> wrote:
>
>  On Tue, Apr 04, Stephen Rothwell wrote:
> 
> > The iSeries Hypervisor only allows us to specify IRQ numbers up to 255 (it
> > has a u8 field to pass it in).  This patch allows platforms to specify a
> > maximum to the virtual IRQ numbers we will use and has iSeries set that
> > to 255.  If not set, the maximum is NR_IRQS - 1 (as before).
> 
> I tried this patch, and the partition dies that way:

If you have time, can you try (without this patch) just changing NR_IRQS in
include/asm-powerpc/irq.h from 512 to 256 and see if that kernel will boot
and what changes (if any) you get to the boot messages.

Thanks.

-- 
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au
http://www.canb.auug.org.au/~sfr/

[-- Attachment #2: Type: application/pgp-signature, Size: 191 bytes --]

Re: [PATCH] powerpc: iSeries has only 256 IRQs

[Olaf Hering]

 On Wed, Apr 05, Stephen Rothwell wrote:

> If you have time, can you try (without this patch) just changing NR_IRQS in
> include/asm-powerpc/irq.h from 512 to 256 and see if that kernel will boot
> and what changes (if any) you get to the boot messages.

This results in a different irq:

vio_register_driver: driver iseries_veth registering
iseries_veth: eth0 attached to iSeries vlan 0 (LPAR map = 0x0005)
Emulex LightPulse Fibre Channel SCSI driver 8.1.4
Copyright(c) 2004-2006 Emulex.  All rights reserved.
PCI: Enabling device: (0000:18:12.0), cmd 3
scsi0 :  on PCI bus 18 device 90 irq 55
PCI: RDL: Device 0x0018:90  I/O Error( 1): 0x0101
PCI: RDL: Device 0x0018:90  I/O Error( 2): 0x0102
PCI: RDL: Device 0x0018:90  I/O Error( 3): 0x0102
PCI: RDL: Device 0x0018:90  I/O Error( 4): 0x0102
Kernel panic - not syncing: PCI: Hardware I/O Error, SRC B6000103, Automatic Reboot Disabled.

 <3>Badness in smp_call_function at /usr/src/linux-2.6.16-8/arch/powerpc/kernel/smp.c:228
Call Trace:
[C00000003E8DAF40] [C00000000000E3E4] .show_stack+0x68/0x1b0 (unreliable)
[C00000003E8DAFE0] [C000000000264BF4] .program_check_exception+0x1cc/0x5a8
[C00000003E8DB0B0] [C000000000004A70] program_check_common+0xf0/0x100
--- Exception: 700 at .smp_call_function+0x34/0x1d8
    LR = .smp_call_function+0x28/0x1d8
[C00000003E8DB3A0] [C000000000330000] zone_names+0x818/0xf18 (unreliable)
[C00000003E8DB450] [C0000000000371EC] .panic+0xa0/0x194
[C00000003E8DB4F0] [C00000000002BF48] .CheckReturnCode+0xac/0xc4
[C00000003E8DB570] [C00000000002CAA4] .iSeries_Read_Long+0x160/0x19c
[C00000003E8DB620] [D0000000002215A0] .lpfc_sli_brdrestart+0xc8/0x1c0 [lpfc]
[C00000003E8DB6C0] [D00000000022170C] .lpfc_sli_hba_setup+0x74/0x3e0 [lpfc]
[C00000003E8DB770] [D0000000002348C0] .lpfc_pci_probe_one+0x650/0xa7c [lpfc]
[C00000003E8DB830] [C000000000196D00] .pci_device_probe+0x80/0xc4
[C00000003E8DB8C0] [C0000000001B0A24] .driver_probe_device+0xa0/0x15c
[C00000003E8DB960] [C0000000001B0CB4] .__driver_attach+0xdc/0x164
[C00000003E8DB9F0] [C0000000001B0158] .bus_for_each_dev+0x7c/0xd4
[C00000003E8DBAB0] [C0000000001B08C8] .driver_attach+0x28/0x40
[C00000003E8DBB30] [C0000000001AFBF8] .bus_add_driver+0x90/0x170
[C00000003E8DBBD0] [C0000000001B10E4] .driver_register+0xbc/0xd8
[C00000003E8DBC60] [C000000000196ADC] .__pci_register_driver+0x8c/0xd8
[C00000003E8DBD00] [D00000000023E024] .lpfc_init+0x60/0x282c [lpfc]
[C00000003E8DBD90] [C00000000005C5E4] .sys_init_module+0x1e4/0x448


Great news is that 2.6.5 does it that way:

nst-sys:/ # modprobe  -v lpfc
insmod /lib/modules/2.6.5-override-iseries64/initrd/scsi_transport_fc.ko 
insmod /lib/modules/2.6.5-override-iseries64/initrd/lpfc.ko 
Kernel panic: PCI: Hardware I/O Error, SRC B6000103, Automatic Reboot Disabled.
 

Re: [PATCH] powerpc: iSeries has only 256 IRQs

[Doug Maxey]

On Tue, 04 Apr 2006 20:34:42 +0200, Olaf Hering wrote:
>Great news is that 2.6.5 does it that way:
>
>nst-sys:/ # modprobe  -v lpfc
>insmod /lib/modules/2.6.5-override-iseries64/initrd/scsi_transport_fc.ko 
>insmod /lib/modules/2.6.5-override-iseries64/initrd/lpfc.ko 
>Kernel panic: PCI: Hardware I/O Error, SRC B6000103, Automatic Reboot Disabled.
> 

What firmware level?

++doug

Re: [PATCH] powerpc: iSeries has only 256 IRQs

[Olaf Hering]

 On Tue, Apr 04, Doug Maxey wrote:

> What firmware level?

Of the card? No idea. At some point I had to upgrade it to make it
usable with lpfc in SLES9 SP2. But right now I couldnt upgrade it, kind
of chicken/egg problem. Driver has to be loaded to upgrade the firmware.
I will try to put that card into another box to upgrade the firmware.

I think the irq change itself is not the cause for the crash here.

Re: [PATCH] powerpc: iSeries has only 256 IRQs

[Doug Maxey]

On Tue, 04 Apr 2006 21:02:11 +0200, Olaf Hering wrote:
> On Tue, Apr 04, Doug Maxey wrote:
>
>> What firmware level?
>
>Of the card? No idea. At some point I had to upgrade it to make it

Both the card and the system level please.  Sorry if missed the info.

>usable with lpfc in SLES9 SP2. But right now I couldnt upgrade it, kind
>of chicken/egg problem. Driver has to be loaded to upgrade the firmware.
>I will try to put that card into another box to upgrade the firmware.
>
>I think the irq change itself is not the cause for the crash here.
>