[PATCH 0/2] Base pSeries PCIe support

[PATCH 0/2] Base pSeries PCIe support

[Jake Moilanen]

Here are the two base patches for PCIe on pSeries for 2.6.17.

This is enough to probe, find, configure, and run basic I/O to PCIe
enabled devices.

The patches do not include PCIe hotplug, nor multiple MSI vectors per
adapter.

These patches have had limited testing, as firmware still is working on
a few MSI bugs of their own.

[PATCH 1/2] Base pSeries PCIe support

[Jake Moilanen]

Added recognition of device_type: "pciex" and made PCIe portdrv
buildable on PPC64.


Signed-off-by: Jake Moilanen <moilanen@austin.ibm.com>

---

Index: 2.6.16/arch/powerpc/Kconfig
===================================================================
--- 2.6.16.orig/arch/powerpc/Kconfig	2006-03-31 15:11:20.000000000 -0600
+++ 2.6.16/arch/powerpc/Kconfig	2006-03-31 15:15:35.000000000 -0600
@@ -808,6 +808,8 @@
 
 endchoice
 
+source "drivers/pci/pcie/Kconfig"
+
 source "drivers/pci/Kconfig"
 
 source "drivers/pcmcia/Kconfig"
Index: 2.6.16/arch/powerpc/kernel/pci_64.c
===================================================================
--- 2.6.16.orig/arch/powerpc/kernel/pci_64.c	2006-03-31 15:11:20.000000000 -0600
+++ 2.6.16/arch/powerpc/kernel/pci_64.c	2006-03-31 15:15:35.000000000 -0600
@@ -395,7 +395,7 @@
 
 	dev->current_state = 4;		/* unknown power state */
 
-	if (!strcmp(type, "pci")) {
+	if (!strcmp(type, "pci") || !strcmp(type, "pciex")) {
 		/* a PCI-PCI bridge */
 		dev->hdr_type = PCI_HEADER_TYPE_BRIDGE;
 		dev->rom_base_reg = PCI_ROM_ADDRESS1;
Index: 2.6.16/arch/powerpc/kernel/rtas_pci.c
===================================================================
--- 2.6.16.orig/arch/powerpc/kernel/rtas_pci.c	2006-03-31 15:11:20.000000000 -0600
+++ 2.6.16/arch/powerpc/kernel/rtas_pci.c	2006-03-31 15:15:35.000000000 -0600
@@ -315,7 +315,8 @@
 	for (node = of_get_next_child(root, NULL);
 	     node != NULL;
 	     node = of_get_next_child(root, node)) {
-		if (node->type == NULL || strcmp(node->type, "pci") != 0)
+
+		if (node->type == NULL || ((strcmp(node->type, "pci") != 0) && (strcmp(node->type, "pciex") != 0)))
 			continue;
 
 		phb = pcibios_alloc_controller(node);

[PATCH 2/2] Base pSeries PCIe support

[Jake Moilanen]

This patch hooks our current interrupt subsystem and sets up a single
vector MSI as if it was a LSI.  Multiple MSI vectors is coming in the
future.

The NR_IRQS got bumped up to 1024, as vectors can go much higher.
Unfortunately, this number was arbitrarily picked as there is no claim
at what the max number really is by either the firmware team, or the
PAPR+.

Signed-off-by: Jake Moilanen <moilanen@austin.ibm.com>

---

Index: 2.6.16/arch/powerpc/kernel/prom.c
===================================================================
--- 2.6.16.orig/arch/powerpc/kernel/prom.c	2006-03-31 15:11:20.000000000 -0600
+++ 2.6.16/arch/powerpc/kernel/prom.c	2006-03-31 15:37:10.000000000 -0600
@@ -88,6 +88,8 @@
 struct device_node *dflt_interrupt_controller;
 int num_interrupt_controllers;
 
+void __init finish_msi_node(struct device_node *dn, unsigned long *mem_start, int measure_only);
+
 /*
  * Wrapper for allocating memory for various data that needs to be
  * attached to device nodes as they are processed at boot or when
@@ -120,6 +122,22 @@
 	return NULL;
 }
 
+int
+property_present(struct device_node *np, const char *name)
+{
+	struct property *pp;
+
+	read_lock(&devtree_lock);
+	for (pp = np->properties; pp != 0; pp = pp->next)
+		if (strcmp(pp->name, name) == 0) {
+			return 1;
+		}
+	read_unlock(&devtree_lock);
+
+	return 0;
+}
+
+
 /*
  * Find the interrupt parent of a node.
  */
@@ -341,6 +359,14 @@
 		return 0;
 	}
 
+#ifdef CONFIG_PPC_PSERIES
+	if(property_present(np, "ibm,req#msi")) {
+		printk(KERN_EMERG "Found MSI\n");
+		finish_msi_node(np, mem_start, measure_only);
+		return 0;
+	}
+#endif
+
 	ints = (unsigned int *) get_property(np, "interrupts", &intlen);
 	TRACE("ints=%p, intlen=%d\n", ints, intlen);
 	if (ints == NULL)
@@ -503,6 +529,80 @@
 	DBG(" <- finish_device_tree\n");
 }
 
+void __init finish_msi_node(struct device_node *dn, unsigned long *mem_start, int measure_only)
+{
+	static int seq_num = 1;
+	int i;
+	int rc;
+	int query_token = rtas_token("ibm,query-interrupt-source-number");
+	int devfn;
+	int busno;
+	u32 *reg;
+	int reglen;
+	int ret[3];
+	unsigned int virq;
+	unsigned int addr;
+	unsigned long buid = -1;
+	unsigned long wait_time;
+
+	reg = (u32 *) get_property(dn, "reg", &reglen);
+	if (reg == NULL || reglen < 20)
+		return;
+
+	devfn = (reg[0] >> 8) & 0xff;
+	busno = (reg[0] >> 16) & 0xff;
+
+	buid = get_phb_buid(dn->parent);
+	addr = (busno << 16) | (devfn << 8);
+
+	while (1) {
+		rc = rtas_call(rtas_token("ibm,change-msi"), 6, 3, ret, addr,
+			       buid >> 32, buid & 0xffffffff,
+			       1, 1, seq_num);
+
+		if (!rc)
+			break;
+		else if (rc == RTAS_BUSY)
+			udelay(1);
+		else if (rtas_is_extended_busy(rc)) {
+			wait_time = rtas_extended_busy_delay_time(rc);
+			udelay(wait_time * 1000);
+		} else {
+			printk(KERN_WARNING "error[%d]: getting the number of"
+			       "MSI interrupts for %s\n", rc, dn->name);
+			return;
+		}
+
+		seq_num = ret[1];
+	}
+
+	dn->n_intrs = ret[0];
+
+	dn->intrs = prom_alloc(dn->n_intrs * sizeof(*(dn->intrs)), mem_start);
+	if (!dn->intrs) {
+		printk(KERN_EMERG "finish_msi_node: can't allocate space\n");
+		return;
+	}
+
+	if (measure_only)
+		return;
+
+	for (i = 0; i < dn->n_intrs; i++) {
+		rc = rtas_call(query_token, 4, 3, ret, addr, buid >> 32, buid & 0xffffffff, i);
+
+		if (!rc) {
+			virq = virt_irq_create_mapping(ret[0]);
+
+			dn->intrs[i].line = irq_offset_up(virq);
+			dn->intrs[i].sense = ret[1];
+		} else {
+			printk(KERN_WARNING "error[%d]: query-interrupt-source-number for %s\n",
+			       rc, dn->name);
+		}
+	}
+
+}
+
 static inline char *find_flat_dt_string(u32 offset)
 {
 	return ((char *)initial_boot_params) +
Index: 2.6.16/include/asm-powerpc/irq.h
===================================================================
--- 2.6.16.orig/include/asm-powerpc/irq.h	2006-03-31 15:11:12.000000000 -0600
+++ 2.6.16/include/asm-powerpc/irq.h	2006-03-31 15:15:46.000000000 -0600
@@ -47,7 +47,7 @@
 /*
  * Maximum number of interrupt sources that we can handle.
  */
-#define NR_IRQS		512
+#define NR_IRQS		1024
 
 /* Interrupt numbers are virtual in case they are sparsely
  * distributed by the hardware.
Index: 2.6.16/include/asm-powerpc/pci-bridge.h
===================================================================
--- 2.6.16.orig/include/asm-powerpc/pci-bridge.h	2006-03-31 15:11:36.000000000 -0600
+++ 2.6.16/include/asm-powerpc/pci-bridge.h	2006-03-31 15:34:50.000000000 -0600
@@ -141,6 +141,7 @@
 void pcibios_fixup_new_pci_devices(struct pci_bus *bus, int fix_bus);
 
 extern int pcibios_remove_root_bus(struct pci_controller *phb);
+extern unsigned long __devinit get_phb_buid (struct device_node *phb);
 
 static inline struct pci_controller *pci_bus_to_host(struct pci_bus *bus)
 {

Re: [PATCH 1/2] Base pSeries PCIe support

[Dave Hansen]

On Fri, 2006-03-31 at 16:05 -0600, Jake Moilanen wrote:
> --- 2.6.16.orig/arch/powerpc/kernel/rtas_pci.c  2006-03-31 15:11:20.000000000 -0600
> +++ 2.6.16/arch/powerpc/kernel/rtas_pci.c       2006-03-31 15:15:35.000000000 -0600
> @@ -315,7 +315,8 @@
>         for (node = of_get_next_child(root, NULL);
>              node != NULL;
>              node = of_get_next_child(root, node)) {
> -               if (node->type == NULL || strcmp(node->type, "pci") != 0)
> +
> +               if (node->type == NULL || ((strcmp(node->type, "pci") != 0) && (strcmp(node->type, "pciex") != 0)))
>                         continue; 

That one looks to have gone a wee bit over 80 columns.

-- Dave

Re: [PATCH 1/2] Base pSeries PCIe support

[Jake Moilanen]

On Fri, 31 Mar 2006 14:15:11 -0800
Dave Hansen <haveblue@us.ibm.com> wrote:

> On Fri, 2006-03-31 at 16:05 -0600, Jake Moilanen wrote:
> > --- 2.6.16.orig/arch/powerpc/kernel/rtas_pci.c  2006-03-31 15:11:20.000000000 -0600
> > +++ 2.6.16/arch/powerpc/kernel/rtas_pci.c       2006-03-31 15:15:35.000000000 -0600
> > @@ -315,7 +315,8 @@
> >         for (node = of_get_next_child(root, NULL);
> >              node != NULL;
> >              node = of_get_next_child(root, node)) {
> > -               if (node->type == NULL || strcmp(node->type, "pci") != 0)
> > +
> > +               if (node->type == NULL || ((strcmp(node->type, "pci") != 0) && (strcmp(node->type, "pciex") != 0)))
> >                         continue; 
> 
> That one looks to have gone a wee bit over 80 columns.

I need to stop developing on my widescreen.  :)

Signed-off-by: Jake Moilanen <moilanen@austin.ibm.com>

---

Index: 2.6.16/arch/powerpc/Kconfig
===================================================================
--- 2.6.16.orig/arch/powerpc/Kconfig	2006-03-31 16:13:26.000000000 -0600
+++ 2.6.16/arch/powerpc/Kconfig	2006-03-31 16:13:36.000000000 -0600
@@ -808,6 +808,8 @@
 
 endchoice
 
+source "drivers/pci/pcie/Kconfig"
+
 source "drivers/pci/Kconfig"
 
 source "drivers/pcmcia/Kconfig"
Index: 2.6.16/arch/powerpc/kernel/pci_64.c
===================================================================
--- 2.6.16.orig/arch/powerpc/kernel/pci_64.c	2006-03-31 16:13:26.000000000 -0600
+++ 2.6.16/arch/powerpc/kernel/pci_64.c	2006-03-31 16:13:36.000000000 -0600
@@ -395,7 +395,7 @@
 
 	dev->current_state = 4;		/* unknown power state */
 
-	if (!strcmp(type, "pci")) {
+	if (!strcmp(type, "pci") || !strcmp(type, "pciex")) {
 		/* a PCI-PCI bridge */
 		dev->hdr_type = PCI_HEADER_TYPE_BRIDGE;
 		dev->rom_base_reg = PCI_ROM_ADDRESS1;
Index: 2.6.16/arch/powerpc/kernel/rtas_pci.c
===================================================================
--- 2.6.16.orig/arch/powerpc/kernel/rtas_pci.c	2006-03-31 16:13:26.000000000 -0600
+++ 2.6.16/arch/powerpc/kernel/rtas_pci.c	2006-03-31 16:14:10.000000000 -0600
@@ -315,7 +315,9 @@
 	for (node = of_get_next_child(root, NULL);
 	     node != NULL;
 	     node = of_get_next_child(root, node)) {
-		if (node->type == NULL || strcmp(node->type, "pci") != 0)
+
+		if (node->type == NULL || ((strcmp(node->type, "pci") != 0) &&
+					   (strcmp(node->type, "pciex") != 0)))
 			continue;
 
 		phb = pcibios_alloc_controller(node);

Re: [PATCH 0/2] Base pSeries PCIe support

[Arnd Bergmann]

Am Saturday 01 April 2006 00:02 schrieb Jake Moilanen:
> Here are the two base patches for PCIe on pSeries for 2.6.17.
>
> This is enough to probe, find, configure, and run basic I/O to PCIe
> enabled devices.

Does this use the draft PCIe bindings from Sun, or something else?
Is it compatible with the way that Apple represents PCIe devices
as plain PCI?

I'm just curious.

	Arnd <><

Re: [PATCH 0/2] Base pSeries PCIe support

[Jake Moilanen]

On Sat, 1 Apr 2006 00:28:43 +0200
Arnd Bergmann <arnd@arndb.de> wrote:

> Am Saturday 01 April 2006 00:02 schrieb Jake Moilanen:
> > Here are the two base patches for PCIe on pSeries for 2.6.17.
> >
> > This is enough to probe, find, configure, and run basic I/O to PCIe
> > enabled devices.
> 
> Does this use the draft PCIe bindings from Sun, or something else?
> Is it compatible with the way that Apple represents PCIe devices
> as plain PCI?
> 
> I'm just curious.

You can find some info on it:

	http://playground.sun.com/1275/bindings/pci/pci-express.txt

Jake

Re: [PATCH 0/2] Base pSeries PCIe support

[Doug Maxey]

On Fri, 31 Mar 2006 16:02:03 CST, Jake Moilanen wrote:
>Here are the two base patches for PCIe on pSeries for 2.6.17.
>
>This is enough to probe, find, configure, and run basic I/O to PCIe
>enabled devices.
>

Where are the bindings for pcie?  In the papr?

Have you heard of any effort to publish?

++doug

Re: [PATCH 2/2] Base pSeries PCIe support

[Olof Johansson]

On Fri, Mar 31, 2006 at 04:13:30PM -0600, Jake Moilanen wrote:
> This patch hooks our current interrupt subsystem and sets up a single
> vector MSI as if it was a LSI.  Multiple MSI vectors is coming in the
> future.
> 
> The NR_IRQS got bumped up to 1024, as vectors can go much higher.
> Unfortunately, this number was arbitrarily picked as there is no claim
> at what the max number really is by either the firmware team, or the
> PAPR+.

What does this do to the kernel size?

> Signed-off-by: Jake Moilanen <moilanen@austin.ibm.com>
> 
> ---
> 
> Index: 2.6.16/arch/powerpc/kernel/prom.c
> ===================================================================
> --- 2.6.16.orig/arch/powerpc/kernel/prom.c	2006-03-31 15:11:20.000000000 -0600
> +++ 2.6.16/arch/powerpc/kernel/prom.c	2006-03-31 15:37:10.000000000 -0600
> @@ -88,6 +88,8 @@
>  struct device_node *dflt_interrupt_controller;
>  int num_interrupt_controllers;
>  
> +void __init finish_msi_node(struct device_node *dn, unsigned long *mem_start, int measure_only);

The other one is called "finish_node_interrupts", "finish_node_msi"
would make sense here.

> +
>  /*
>   * Wrapper for allocating memory for various data that needs to be
>   * attached to device nodes as they are processed at boot or when
> @@ -120,6 +122,22 @@
>  	return NULL;
>  }
>  
> +int
> +property_present(struct device_node *np, const char *name)
> +{
> +	struct property *pp;
> +
> +	read_lock(&devtree_lock);
> +	for (pp = np->properties; pp != 0; pp = pp->next)
> +		if (strcmp(pp->name, name) == 0) {
> +			return 1;
> +		}
> +	read_unlock(&devtree_lock);
> +
> +	return 0;
> +}

Please use of_find_property() instead of reimplementing it, i.e. do what
get_property() does now, just don't return the value.

I assume you can't use get_property() because the properties you look
for lack values, right?

> +
>  /*
>   * Find the interrupt parent of a node.
>   */
> @@ -341,6 +359,14 @@
>  		return 0;
>  	}
>  
> +#ifdef CONFIG_PPC_PSERIES
> +	if(property_present(np, "ibm,req#msi")) {
> +		printk(KERN_EMERG "Found MSI\n");

KERN_DEBUG, if anything, right?

Also, get rid of the ifdef.

> +		finish_msi_node(np, mem_start, measure_only);
> +		return 0;
> +	}
> +#endif
> +
>  	ints = (unsigned int *) get_property(np, "interrupts", &intlen);
>  	TRACE("ints=%p, intlen=%d\n", ints, intlen);
>  	if (ints == NULL)
> @@ -503,6 +529,80 @@
>  	DBG(" <- finish_device_tree\n");
>  }
>  
> +void __init finish_msi_node(struct device_node *dn, unsigned long *mem_start, int measure_only)

I know you said no hotplug support yet, but in preparation, it might
make sense to mark it as __devinit instead.

> +{
> +	static int seq_num = 1;
> +	int i;
> +	int rc;
> +	int query_token = rtas_token("ibm,query-interrupt-source-number");
> +	int devfn;
> +	int busno;
> +	u32 *reg;
> +	int reglen;
> +	int ret[3];
> +	unsigned int virq;
> +	unsigned int addr;
> +	unsigned long buid = -1;
> +	unsigned long wait_time;
> +
> +	reg = (u32 *) get_property(dn, "reg", &reglen);
> +	if (reg == NULL || reglen < 20)
> +		return;
> +
> +	devfn = (reg[0] >> 8) & 0xff;
> +	busno = (reg[0] >> 16) & 0xff;
> +
> +	buid = get_phb_buid(dn->parent);
> +	addr = (busno << 16) | (devfn << 8);
> +
> +	while (1) {
> +		rc = rtas_call(rtas_token("ibm,change-msi"), 6, 3, ret, addr,
> +			       buid >> 32, buid & 0xffffffff,
> +			       1, 1, seq_num);
> +
> +		if (!rc)
> +			break;
> +		else if (rc == RTAS_BUSY)
> +			udelay(1);
> +		else if (rtas_is_extended_busy(rc)) {
> +			wait_time = rtas_extended_busy_delay_time(rc);
> +			udelay(wait_time * 1000);
> +		} else {
> +			printk(KERN_WARNING "error[%d]: getting the number of"
> +			       "MSI interrupts for %s\n", rc, dn->name);
> +			return;
> +		}

This is reimplemented with a few variations in a few places now. They
should be consolidated into a single function that will handle the
delay itself.

> +
> +		seq_num = ret[1];
> +	}
> +
> +	dn->n_intrs = ret[0];
> +
> +	dn->intrs = prom_alloc(dn->n_intrs * sizeof(*(dn->intrs)), mem_start);
> +	if (!dn->intrs) {
> +		printk(KERN_EMERG "finish_msi_node: can't allocate space\n");
> +		return;
> +	}
> +
> +	if (measure_only)
> +		return;

You probably want to return from measure_only _before_ you allocate and
populate intrs...

> +
> +	for (i = 0; i < dn->n_intrs; i++) {
> +		rc = rtas_call(query_token, 4, 3, ret, addr, buid >> 32, buid & 0xffffffff, i);
> +
> +		if (!rc) {
> +			virq = virt_irq_create_mapping(ret[0]);
> +
> +			dn->intrs[i].line = irq_offset_up(virq);
> +			dn->intrs[i].sense = ret[1];
> +		} else {
> +			printk(KERN_WARNING "error[%d]: query-interrupt-source-number for %s\n",
> +			       rc, dn->name);
> +		}
> +	}
> +
> +}
> +
>  static inline char *find_flat_dt_string(u32 offset)
>  {
>  	return ((char *)initial_boot_params) +
> Index: 2.6.16/include/asm-powerpc/irq.h
> ===================================================================
> --- 2.6.16.orig/include/asm-powerpc/irq.h	2006-03-31 15:11:12.000000000 -0600
> +++ 2.6.16/include/asm-powerpc/irq.h	2006-03-31 15:15:46.000000000 -0600
> @@ -47,7 +47,7 @@
>  /*
>   * Maximum number of interrupt sources that we can handle.
>   */
> -#define NR_IRQS		512
> +#define NR_IRQS		1024
>  
>  /* Interrupt numbers are virtual in case they are sparsely
>   * distributed by the hardware.
> Index: 2.6.16/include/asm-powerpc/pci-bridge.h
> ===================================================================
> --- 2.6.16.orig/include/asm-powerpc/pci-bridge.h	2006-03-31 15:11:36.000000000 -0600
> +++ 2.6.16/include/asm-powerpc/pci-bridge.h	2006-03-31 15:34:50.000000000 -0600
> @@ -141,6 +141,7 @@
>  void pcibios_fixup_new_pci_devices(struct pci_bus *bus, int fix_bus);
>  
>  extern int pcibios_remove_root_bus(struct pci_controller *phb);
> +extern unsigned long __devinit get_phb_buid (struct device_node *phb);

I think the get_phb_buid() call from prom.c might break 32-bit, since
rtas_pci.c isn't always built there.  An empty declaration here for
#ifndef CONFIG_<x> would take care of it.

>  
>  static inline struct pci_controller *pci_bus_to_host(struct pci_bus *bus)
>  {
> 
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-dev

Re: [PATCH 2/2] Base pSeries PCIe support

[Olof Johansson]

On Fri, Mar 31, 2006 at 04:48:58PM -0600, Olof Johansson wrote:

> > +	dn->intrs = prom_alloc(dn->n_intrs * sizeof(*(dn->intrs)), mem_start);
> > +	if (!dn->intrs) {
> > +		printk(KERN_EMERG "finish_msi_node: can't allocate space\n");
> > +		return;
> > +	}
> > +
> > +	if (measure_only)
> > +		return;
> 
> You probably want to return from measure_only _before_ you allocate and
> populate intrs...

Nevermind, this is correct as-is.


-Olof

Re: [PATCH 2/2] Base pSeries PCIe support

[Paul Mackerras]

Jake Moilanen writes:

> The NR_IRQS got bumped up to 1024, as vectors can go much higher.
> Unfortunately, this number was arbitrarily picked as there is no claim
> at what the max number really is by either the firmware team, or the
> PAPR+.

What matters is the number of different vectors, not the actual value
of the vectors, because we remap interrupt numbers that the firmware
gives us to logical Linux irq numbers between 0 and NR_IRQS-1.  We had
to do that when the POWER5 systems came out, because the interrupt
numbers there occupy 24 bits.

Paul.

Re: [PATCH 2/2] Base pSeries PCIe support

[Paul Mackerras]

Jake Moilanen writes:

> +property_present(struct device_node *np, const char *name)
> +{
> +	struct property *pp;
> +
> +	read_lock(&devtree_lock);
> +	for (pp = np->properties; pp != 0; pp = pp->next)
> +		if (strcmp(pp->name, name) == 0) {
> +			return 1;

Ummm, did you mean to return with the devtree_lock held?  I suspect
not.  In any case, why not just use get_property or of_find_property?

Paul.

Re: [PATCH 2/2] Base pSeries PCIe support

[Jake Moilanen]

On Sat, 1 Apr 2006 19:36:07 +1100
Paul Mackerras <paulus@samba.org> wrote:

> Jake Moilanen writes:
> 
> > The NR_IRQS got bumped up to 1024, as vectors can go much higher.
> > Unfortunately, this number was arbitrarily picked as there is no claim
> > at what the max number really is by either the firmware team, or the
> > PAPR+.
> 
> What matters is the number of different vectors, not the actual value
> of the vectors, because we remap interrupt numbers that the firmware
> gives us to logical Linux irq numbers between 0 and NR_IRQS-1.  We had
> to do that when the POWER5 systems came out, because the interrupt
> numbers there occupy 24 bits.

Ah.  That sounds right.  I haven't had a chance to test this version of
the patch.  Firmware is currently broken on my machine.

Here's try #2

I ended up putting #ifdef CONFIG_PPC_RTAS around this, as non-rtas
kernels won't like all the rtas calls.

Signed-off-by: Jake Moilanen <moilanen@austin.ibm.com>

---

Index: 2.6.16/arch/powerpc/kernel/prom.c
===================================================================
--- 2.6.16.orig/arch/powerpc/kernel/prom.c	2006-04-01 16:34:45.000000000 -0600
+++ 2.6.16/arch/powerpc/kernel/prom.c	2006-04-01 16:52:53.000000000 -0600
@@ -88,6 +88,8 @@
 struct device_node *dflt_interrupt_controller;
 int num_interrupt_controllers;
 
+void __init finish_node_msi(struct device_node *dn, unsigned long *mem_start, int measure_only);
+
 /*
  * Wrapper for allocating memory for various data that needs to be
  * attached to device nodes as they are processed at boot or when
@@ -295,6 +297,7 @@
 {
 	unsigned int *ints;
 	int intlen, intrcells, intrcount;
+	int dummy;
 	int i, j, n, sense;
 	unsigned int *irq, virq;
 	struct device_node *ic;
@@ -341,6 +344,13 @@
 		return 0;
 	}
 
+#ifdef CONFIG_PPC_RTAS
+	if(of_find_property(np, "ibm,req#msi", &dummy)) {
+		finish_node_msi(np, mem_start, measure_only);
+		return 0;
+	}
+#endif
+
 	ints = (unsigned int *) get_property(np, "interrupts", &intlen);
 	TRACE("ints=%p, intlen=%d\n", ints, intlen);
 	if (ints == NULL)
@@ -503,6 +513,82 @@
 	DBG(" <- finish_device_tree\n");
 }
 
+#ifdef CONFIG_PPC_RTAS
+void __devinit finish_node_msi(struct device_node *dn, unsigned long *mem_start, int measure_only)
+{
+	static int seq_num = 1;
+	int i;
+	int rc;
+	int query_token = rtas_token("ibm,query-interrupt-source-number");
+	int devfn;
+	int busno;
+	u32 *reg;
+	int reglen;
+	int ret[3];
+	unsigned int virq;
+	unsigned int addr;
+	unsigned long buid = -1;
+	unsigned long wait_time;
+
+	reg = (u32 *) get_property(dn, "reg", &reglen);
+	if (reg == NULL || reglen < 20)
+		return;
+
+	devfn = (reg[0] >> 8) & 0xff;
+	busno = (reg[0] >> 16) & 0xff;
+
+	buid = get_phb_buid(dn->parent);
+	addr = (busno << 16) | (devfn << 8);
+
+	while (1) {
+		rc = rtas_call(rtas_token("ibm,change-msi"), 6, 3, ret, addr,
+			       buid >> 32, buid & 0xffffffff,
+			       1, 1, seq_num);
+
+		if (!rc)
+			break;
+		else if (rc == RTAS_BUSY)
+			udelay(1);
+		else if (rtas_is_extended_busy(rc)) {
+			wait_time = rtas_extended_busy_delay_time(rc);
+			udelay(wait_time * 1000);
+		} else {
+			printk(KERN_WARNING "error[%d]: getting the number of"
+			       "MSI interrupts for %s\n", rc, dn->name);
+			return;
+		}
+
+		seq_num = ret[1];
+	}
+
+	dn->n_intrs = ret[0];
+
+	dn->intrs = prom_alloc(dn->n_intrs * sizeof(*(dn->intrs)), mem_start);
+	if (!dn->intrs) {
+		printk(KERN_WARNING "finish_node_msi: can't allocate space\n");
+		return;
+	}
+
+	if (measure_only)
+		return;
+
+	for (i = 0; i < dn->n_intrs; i++) {
+		rc = rtas_call(query_token, 4, 3, ret, addr, buid >> 32, buid & 0xffffffff, i);
+
+		if (!rc) {
+			virq = virt_irq_create_mapping(ret[0]);
+
+			dn->intrs[i].line = irq_offset_up(virq);
+			dn->intrs[i].sense = ret[1];
+		} else {
+			printk(KERN_WARNING "error[%d]: query-interrupt-source-number for %s\n",
+			       rc, dn->name);
+		}
+	}
+
+}
+#endif /* #ifdef CONFIG_PPC_RTAS */
+
 static inline char *find_flat_dt_string(u32 offset)
 {
 	return ((char *)initial_boot_params) +
Index: 2.6.16/include/asm-powerpc/pci-bridge.h
===================================================================
--- 2.6.16.orig/include/asm-powerpc/pci-bridge.h	2006-04-01 16:34:45.000000000 -0600
+++ 2.6.16/include/asm-powerpc/pci-bridge.h	2006-04-01 16:52:38.000000000 -0600
@@ -141,6 +141,7 @@
 void pcibios_fixup_new_pci_devices(struct pci_bus *bus, int fix_bus);
 
 extern int pcibios_remove_root_bus(struct pci_controller *phb);
+extern unsigned long __devinit get_phb_buid (struct device_node *phb);
 
 static inline struct pci_controller *pci_bus_to_host(struct pci_bus *bus)
 {

Re: [PATCH 2/2] Base pSeries PCIe support

[Jake Moilanen]

On Sat, 1 Apr 2006 16:57:23 -0600
Jake Moilanen <moilanen@austin.ibm.com> wrote:

> On Sat, 1 Apr 2006 19:36:07 +1100
> Paul Mackerras <paulus@samba.org> wrote:
> 
> > Jake Moilanen writes:
> > 
> > > The NR_IRQS got bumped up to 1024, as vectors can go much higher.
> > > Unfortunately, this number was arbitrarily picked as there is no claim
> > > at what the max number really is by either the firmware team, or the
> > > PAPR+.
> > 
> > What matters is the number of different vectors, not the actual value
> > of the vectors, because we remap interrupt numbers that the firmware
> > gives us to logical Linux irq numbers between 0 and NR_IRQS-1.  We had
> > to do that when the POWER5 systems came out, because the interrupt
> > numbers there occupy 24 bits.
> 
> Ah.  That sounds right.  I haven't had a chance to test this version of
> the patch.  Firmware is currently broken on my machine.

I was able to validate that these patches do work, and we are receiving
MSI interrupts correctly.

Is it too late to get into 2.6.17?

Jake

Re: [PATCH 2/2] Base pSeries PCIe support

[Benjamin Herrenschmidt]

> I was able to validate that these patches do work, and we are receiving
> MSI interrupts correctly.
> 
> Is it too late to get into 2.6.17?

Yes and it's totally broken anyway. The whole idea of "hiding" MSIs as
if they were LSIs is totally irrealistic in practice. Our firmware might
do it but we have to undo it.

Linux defines the semantics of a driver probe() routine to be called
with MSIs disabled by default.

This is very important as for a lot of devices (I'd say the vast
majority of the devices that claim to be MSI capable), enabling MSI is
either broken (various HW bugs related to using them, mostly ordering
issues) or requires all sort of additional tweaking on the device side
in addition to the standard config space stuffs.

Thus, we need to help whoever is currently ripping off the
drivers/pci/msi.c code which is way too Intel-centric (there's an SGI
guy working on it already) and implement an arch specific implementation
that undoes what the firmware did, and then re-assign MSIs as requested
by drviers.

Some of the Boeblingen folks have already been looking into it as part
of some bare-metal/bringup effort, but that doesn't include PAPR
implementation.

Ben.

Re: [PATCH 2/2] Base pSeries PCIe support

[Christoph Hellwig]

On Fri, Mar 31, 2006 at 04:13:30PM -0600, Jake Moilanen wrote:
> This patch hooks our current interrupt subsystem and sets up a single
> vector MSI as if it was a LSI.  Multiple MSI vectors is coming in the
> future.

This is broken.  Linux drivers expect MSI to be disabled on ->probe.
There's at least two reasons for that:

 (1) Many devices that claim to implement MSI are actually broken in
     more or less subtile ways. and thus must use traditition INTx pins.
 (2) MSI defines relaxed semantics for dma synchronization.  Silently
     enabling MSI could cause subtile data corruption.

Re: [PATCH 1/2] Base pSeries PCIe support

[Christoph Hellwig]

On Fri, Mar 31, 2006 at 04:05:45PM -0600, Jake Moilanen wrote:
> +source "drivers/pci/pcie/Kconfig"

does it actually work on ppc64?  last time I checked that code was
utterly x86-centric.

> -		if (node->type == NULL || strcmp(node->type, "pci") != 0)
> +
> +		if (node->type == NULL || ((strcmp(node->type, "pci") != 0) && (strcmp(node->type, "pciex") != 0)))
>  			continue;

please don't add superflous braces