PCIe hotplug on far PCIe switch ports

PCIe hotplug on far PCIe switch ports

[Thomas Schäfer]

Hi,

I'm using a Pentium M processor module with Intel E7320 (Lindenhurst) chipset and linux 2.6.14-rc4 from kernel.org. One of the chipsets PCIe ports has a PEX8532 PCIe switch attached and one of the switch ports hosts a network adapter. I want to perform PCIe hotplugging with the network adapter and tried loading the pciehp.ko driver, but without success.

   <snip>

   pciehp: add_host_bridge: status 5
   pciehp: add_host_bridge: osc_run_status 1
   pciehp: acpi_pciehprm: Registered PCI HOST Bridge(00)    on s:b:d:f(00:00:00:00) [\_SB_.PCI0]
   pciehp: acpi_pciehprm: P2P(0-1) on pci=b:d:f(0:2:0) acpi=b:d:f(0:2:0) [\_SB_.PCI0.EPA0]
   pciehp: acpi_pciehprm: Registered PCI  P2P Bridge(00-01) on s:b:d:f(00:00:02:00) [\_SB_.PCI0.EPA0]
   pciehp: acpi_pciehprm: P2P(0-2) on pci=b:d:f(0:3:0) acpi=b:d:f(0:3:0) [\_SB_.PCI0.EPA1]
   pciehp: acpi_pciehprm: Registered PCI  P2P Bridge(00-02) on s:b:d:f(00:00:03:00) [\_SB_.PCI0.EPA1]
   pciehp: acpi_pciehprm: P2P(0-9) on pci=b:d:f(0:1c:0) acpi=b:d:f(0:1c:0) [\_SB_.PCI0.P0P3]
   pciehp: acpi_pciehprm: Registered PCI  P2P Bridge(00-09) on s:b:d:f(00:00:1c:00) [\_SB_.PCI0.P0P3]
   pciehp: acpi_pciehprm: P2P(0-a) on pci=b:d:f(0:1e:0) acpi=b:d:f(0:1e:0) [\_SB_.PCI0.P0P4]
   pciehp: acpi_pciehprm: Registered PCI  P2P Bridge(00-0a) on s:b:d:f(00:00:1e:00) [\_SB_.PCI0.P0P4]
   pciehp: Fails to gain control of native hot-plug
   pciehp: pciehprm ACPI init fail

The driver can only find the root bridges of the chipset (EPA0, EPA1, P0P3, P0P4), but not the bridges associated to the switch. In my understanding this is due to the ACPI BIOS which fails to create a namespace holding the entire PCI tree.

In a second try, I built a kernel without ACPI support. Now the PCIe hotplug driver is loaded properly and installs the ports of the switch as hotplug slots. When pressing the attention button, the driver of the network adapter is unloaded and the adapter is properly removed from the PCI tree. But when pressing the attention button again, reinsertion of the network adapter in the PCI tree works, but loading the driver fails:

   <snip>

   pciehp: PCI function Hot-Added s:b:d:f(00:06:00:00)
   pciehp: returning from configure_new_device
   pciehp: board_added: back from configure_new_device
   pciehp: pciehp_resource_sort_and_combine: head = f70782f0, *head = f7283b60
   pciehp: *head->next = 00000000
   pciehp: pciehp_resource_sort_and_combine: head = f70782f4, *head = f7d7f8a0
   pciehp: *head->next = 00000000
   pciehp: pciehp_resource_sort_and_combine: head = f70782f8, *head = 00000000
   pciehp: pciehp_resource_sort_and_combine: head = f70782fc, *head = f7d7ff40
   pciehp: *head->next = 00000000
   pciehp: pciehp_slot_find: bus 6 device 0 index 0
   pciehp: pciehp_slot_find: func-> bus 6 device 0 function 0 pci_dev 00000000
   pciehp: board_added:call pci_hp_configure_dev, func 1
   pciehp: pciehp_configure_device: pci_dev still null. do pci_scan_slot
   PCI: Device 0000:06:00.0 not available because of resource collisions
   tg3: Cannot enable PCI device, aborting.
   tg3: probe of 0000:06:00.0 failed with error -22
   pciehp: pciehp_slot_find: bus 6 device 0 index 1
   pciehp: pciehp_slot_find: func-> bus 6 device 0 function 0 pci_dev f7d85400

   <snip>

The resource conflict seems to be the number of the interrupt line. Immediately after booting the network adapter has interrupt number 16:

   [root@amceval ~]# lspci -vx -s 6:0.0
   06:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express (rev 11)
           Subsystem: Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express
           Flags: bus master, fast devsel, latency 0, IRQ 16
           Memory at fe9f0000 (64-bit, non-prefetchable) [sizedK]
           Expansion ROM at fe9e0000 [disabled] [sizedK]
           Capabilities: [48] Power Management version 2
           Capabilities: [50] Vital Product Data
           Capabilities: [58] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable-
           Capabilities: [d0] Express Endpoint IRQ 0
   00: e4 14 59 16 06 00 10 00 11 00 00 02 10 00 00 00
   10: 04 00 9f fe 00 00 00 00 00 00 00 00 00 00 00 00
   20: 00 00 00 00 00 00 00 00 07 00 00 00 e4 14 59 16
   30: 00 00 9e fe 48 00 00 00 00 00 00 00 0a 01 00 00

Note that 0x0A is entered as interrupt number in offset 0x3C.

After the hotplug cycle, the network adapter has interrupt number 10:

[root@amceval hotplug]# lspci -vx -s 6:0.0
06:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express (rev 11)
        Subsystem: Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express
        Flags: bus master, fast devsel, latency 0, IRQ 10
        Memory at fe900000 (64-bit, non-prefetchable) [sizedK]
        Capabilities: [48] Power Management version 2
        Capabilities: [50] Vital Product Data
        Capabilities: [58] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable-
        Capabilities: [d0] Express Endpoint IRQ 0
00: e4 14 59 16 46 01 10 00 11 00 00 02 08 00 00 00
10: 04 00 90 fe 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 07 00 00 00 e4 14 59 16
30: 00 00 00 00 48 00 00 00 00 00 00 00 0a 01 00 00

My question is:

How should I proceed to get the entire PCIe hotplug cycle working? Is it necessary to fix the ACPI BIOS or is it possible to succeed without ACPI? Or am I missing something else?

Best regards,

Thomas Schäfer

____________________________________

Kontron Modular Computers GmbH

Konrad-Zuse-Str. 9
66115 Saarbrücken

Tel.: + 49 (0)681 / 95916 - 203
Fax:  + 49 (0)681 / 95916 - 100
E-mail: thomas.schaefer@kontron.com 


-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: PCIe hotplug on far PCIe switch ports

[Greg KH]

On Wed, Oct 12, 2005 at 07:07:19PM +0200, Thomas Sch?fer wrote:
> Hi,
> 
> I'm using a Pentium M processor module with Intel E7320 (Lindenhurst)
> chipset and linux 2.6.14-rc4 from kernel.org. One of the chipsets PCIe
> ports has a PEX8532 PCIe switch attached and one of the switch ports
> hosts a network adapter. I want to perform PCIe hotplugging with the
> network adapter and tried loading the pciehp.ko driver, but without
> success.

Try asking this on the pci hotplug mailing list, you might get a better
response there.

thanks,

greg k-h


-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: PCIe hotplug on far PCIe switch ports

[Rajesh Shah]

On Wed, Oct 12, 2005 at 07:07:19PM +0200, Thomas Schäfer wrote:
> 
> When pressing the attention button, the driver of the network adapter is unloaded and the adapter is properly removed from the PCI tree. But when pressing the attention button again, reinsertion of the network adapter in the PCI tree works, but loading the driver fails:
> 
<snip>
>    PCI: Device 0000:06:00.0 not available because of resource collisions
>    tg3: Cannot enable PCI device, aborting.
>    tg3: probe of 0000:06:00.0 failed with error -22
> 
I think the kernel is flagging a bogus resource collision error
for the disabled rom on the hot-added device. lspci output after
boot shows that the device has a disabled ROM.
> 
>    [root@amceval ~]# lspci -vx -s 6:0.0
>    06:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express (rev 11)
>            Subsystem: Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express
>            Flags: bus master, fast devsel, latency 0, IRQ 16
>            Memory at fe9f0000 (64-bit, non-prefetchable) [sizedK]
>            Expansion ROM at fe9e0000 [disabled] [sizedK]
>            Capabilities: [48] Power Management version 2
>            Capabilities: [50] Vital Product Data
>            Capabilities: [58] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable-
>            Capabilities: [d0] Express Endpoint IRQ 0
>    00: e4 14 59 16 06 00 10 00 11 00 00 02 10 00 00 00
>    10: 04 00 9f fe 00 00 00 00 00 00 00 00 00 00 00 00
>    20: 00 00 00 00 00 00 00 00 07 00 00 00 e4 14 59 16
>    30: 00 00 9e fe 48 00 00 00 00 00 00 00 0a 01 00 00
> 
> 
When attempting the hot-add, the pciehp driver clobbers the rom
base address to 0. When pciehp later calls pci_scan_slot() ->
pci_read_bases(), the device's resource structure records a
0 rom base address but a non-zero size. This in turn causes
the kernel to believe this was done to flag a resource conflict.

Does this patch fix the problem?

Signed-off-by: rajesh.shah@intel.com

 arch/i386/pci/i386.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

Index: linux-2.6.14-rc4/arch/i386/pci/i386.c
=================================--- linux-2.6.14-rc4.orig/arch/i386/pci/i386.c
+++ linux-2.6.14-rc4/arch/i386/pci/i386.c
@@ -221,6 +221,9 @@ int pcibios_enable_resources(struct pci_
 			continue;
 
 		r = &dev->resource[idx];
+		if ((idx = PCI_ROM_RESOURCE) &&
+				(!(r->flags & IORESOURCE_ROM_ENABLE)))
+			continue;
 		if (!r->start && r->end) {
 			printk(KERN_ERR "PCI: Device %s not available because of resource collisions\n", pci_name(dev));
 			return -EINVAL;
@@ -230,8 +233,6 @@ int pcibios_enable_resources(struct pci_
 		if (r->flags & IORESOURCE_MEM)
 			cmd |= PCI_COMMAND_MEMORY;
 	}
-	if (dev->resource[PCI_ROM_RESOURCE].start)
-		cmd |= PCI_COMMAND_MEMORY;
 	if (cmd != old_cmd) {
 		printk("PCI: Enabling device %s (%04x -> %04x)\n", pci_name(dev), old_cmd, cmd);
 		pci_write_config_word(dev, PCI_COMMAND, cmd);


-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: PCIe hotplug on far PCIe switch ports

[Rajat Jain]

On 10/13/05, Thomas Schäfer <thomas.schaefer@kontron.com> wrote:
> Hi,
>
> I'm using a Pentium M processor module with Intel E7320 (Lindenhurst) chipset and linux 2.6.14-rc4 from kernel.org. One of the chipsets PCIe ports has a PEX8532 PCIe switch attached and one of the switch ports hosts a network adapter. I want to perform PCIe hotplugging with the network adapter and tried loading the pciehp.ko driver, but without success.
>
>   <snip>
>
>   pciehp: add_host_bridge: status 5
>   pciehp: add_host_bridge: osc_run_status 1
>   pciehp: acpi_pciehprm: Registered PCI HOST Bridge(00)    on s:b:d:f(00:00:00:00) [\_SB_.PCI0]
>   pciehp: acpi_pciehprm: P2P(0-1) on pci=b:d:f(0:2:0) acpi=b:d:f(0:2:0) [\_SB_.PCI0.EPA0]
>   pciehp: acpi_pciehprm: Registered PCI  P2P Bridge(00-01) on s:b:d:f(00:00:02:00) [\_SB_.PCI0.EPA0]
>   pciehp: acpi_pciehprm: P2P(0-2) on pci=b:d:f(0:3:0) acpi=b:d:f(0:3:0) [\_SB_.PCI0.EPA1]
>   pciehp: acpi_pciehprm: Registered PCI  P2P Bridge(00-02) on s:b:d:f(00:00:03:00) [\_SB_.PCI0.EPA1]
>   pciehp: acpi_pciehprm: P2P(0-9) on pci=b:d:f(0:1c:0) acpi=b:d:f(0:1c:0) [\_SB_.PCI0.P0P3]
>   pciehp: acpi_pciehprm: Registered PCI  P2P Bridge(00-09) on s:b:d:f(00:00:1c:00) [\_SB_.PCI0.P0P3]
>   pciehp: acpi_pciehprm: P2P(0-a) on pci=b:d:f(0:1e:0) acpi=b:d:f(0:1e:0) [\_SB_.PCI0.P0P4]
>   pciehp: acpi_pciehprm: Registered PCI  P2P Bridge(00-0a) on s:b:d:f(00:00:1e:00) [\_SB_.PCI0.P0P4]
>   pciehp: Fails to gain control of native hot-plug
>   pciehp: pciehprm ACPI init fail
>
> The driver can only find the root bridges of the chipset (EPA0, EPA1, P0P3, P0P4), but not the bridges associated to the switch. In my understanding this is due to the ACPI BIOS which fails to create a namespace holding the entire PCI tree.
>

I am facing an exactly similar problem. But not sure how to get around
it. Disabling ACPI sure works for me, but that is not a permanent
solution. If you get any pointers, would appreciate if you could share
them.

Thanks,

Rajat


-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

RE: PCIe hotplug on far PCIe switch ports

[Thomas Schäfer]

 

> -----Original Message-----
> From: Greg KH [mailto:greg@kroah.com] 
> Sent: Wednesday, October 12, 2005 8:28 PM
> To: Thomas Schäfer
> Cc: linux-hotplug-devel@lists.sourceforge.net
> Subject: Re: PCIe hotplug on far PCIe switch ports
> 
> Try asking this on the pci hotplug mailing list, you might 
> get a better response there.
> 
> thanks,
> 
> greg k-h
> 
> 

Where is this list hosted? I've already seen some hints to the pci hotplug mailing list before but was not able to find it. The linux-hotplug page at SF only points to the linux-hotplug-devel list, so I decided to post here.

Best regards,

Thomas Schäfer


-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

RE: PCIe hotplug on far PCIe switch ports

[Thomas Schäfer]

 

> -----Original Message-----
> From: Rajesh Shah [mailto:rajesh.shah@intel.com] 
> Sent: Thursday, October 13, 2005 12:17 AM
> To: Thomas Schäfer
> Cc: linux-hotplug-devel@lists.sourceforge.net; 
> linux-kernel@vger.kernel.org
> Subject: Re: PCIe hotplug on far PCIe switch ports
> 
> When attempting the hot-add, the pciehp driver clobbers the 
> rom base address to 0. When pciehp later calls 
> pci_scan_slot() -> pci_read_bases(), the device's resource 
> structure records a 0 rom base address but a non-zero size. 
> This in turn causes the kernel to believe this was done to 
> flag a resource conflict.
> 
> Does this patch fix the problem?
> 
> Signed-off-by: rajesh.shah@intel.com
> 
>  arch/i386/pci/i386.c |    5 +++--
>  1 files changed, 3 insertions(+), 2 deletions(-)
> 
> Index: linux-2.6.14-rc4/arch/i386/pci/i386.c
> =================================> --- linux-2.6.14-rc4.orig/arch/i386/pci/i386.c
> +++ linux-2.6.14-rc4/arch/i386/pci/i386.c
> @@ -221,6 +221,9 @@ int pcibios_enable_resources(struct pci_
>  			continue;
>  
>  		r = &dev->resource[idx];
> +		if ((idx = PCI_ROM_RESOURCE) &&
> +				(!(r->flags & IORESOURCE_ROM_ENABLE)))
> +			continue;
>  		if (!r->start && r->end) {
>  			printk(KERN_ERR "PCI: Device %s not 
> available because of resource collisions\n", pci_name(dev));
>  			return -EINVAL;
> @@ -230,8 +233,6 @@ int pcibios_enable_resources(struct pci_
>  		if (r->flags & IORESOURCE_MEM)
>  			cmd |= PCI_COMMAND_MEMORY;
>  	}
> -	if (dev->resource[PCI_ROM_RESOURCE].start)
> -		cmd |= PCI_COMMAND_MEMORY;
>  	if (cmd != old_cmd) {
>  		printk("PCI: Enabling device %s (%04x -> 
> %04x)\n", pci_name(dev), old_cmd, cmd);
>  		pci_write_config_word(dev, PCI_COMMAND, cmd);
> 
> 

Hi Rajesh,

Yes, the patch solves the resource conflicts, but now I'm running into IRQ mapping problems:

   pciehp: !!!!event_thread sleeping
   irq 16: nobody cared (try booting with the "irqpoll" option)
    [<c0139954>] __report_bad_irq+0x24/0x80
    [<c0139a6b>] note_interrupt+0x8b/0xe0
    [<c013934a>] __do_IRQ+0xda/0xf0
    [<c0104b76>] do_IRQ+0x36/0x70
    [<c0104b7b>] do_IRQ+0x3b/0x70
    [<c01035d6>] common_interrupt+0x1a/0x20
    [<c0100c50>] default_idle+0x0/0x30
    [<c0100c73>] default_idle+0x23/0x30
    [<c0100d02>] cpu_idle+0x42/0x60
    [<c03bc83f>] start_kernel+0x16f/0x1b0
    [<c03bc380>] unknown_bootoption+0x0/0x1e0
   handlers:
   [<c0271c50>] (usb_hcd_irq+0x0/0x50)
   [<f8aadea0>] (pcie_isr+0x0/0x690 [pciehp])
   Disabling IRQ #16

   Message from syslogd@amceval at Thu Oct 13 10:38:36 2005 ...
   amceval kernel: Disabling IRQ #16

   [root@amceval hotplug]# cat /proc/interrupts 
              CPU0       
     0:     609080    IO-APIC-edge  timer
     2:          0          XT-PIC  cascade
     4:       2457    IO-APIC-edge  serial
     8:          1    IO-APIC-edge  rtc
    10:          0          XT-PIC  eth0
    14:          0    IO-APIC-edge  libata
    15:       8540    IO-APIC-edge  ide1
    16:     200000   IO-APIC-level  uhci_hcd:usb2, pciehp
    17:          2   IO-APIC-level  pciehp, pciehp
    18:          4   IO-APIC-level  pciehp, pciehp
    23:          0   IO-APIC-level  ehci_hcd:usb1
   NMI:          0 
   LOC:     608585 
   ERR:          1
   MIS:          0
   [root@amceval hotplug]#

Interrupt 10 is registered by the eth0 driver, but the network adapter still asserts interrupt 16. I saw the same behaviour when using kernel 2.6.13.2.

Any ideas?

Thanks,

Thomas Schäfer

P.S: Perhaps we should continue this discussion on the pci hotplug mailing list (where is it hosted?).


-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: PCIe hotplug on far PCIe switch ports

[Gary Hade]

On Thu, Oct 13, 2005 at 09:54:19AM +0200, Thomas Schäfer wrote:
>  
> 
> > -----Original Message-----
> > From: Greg KH [mailto:greg@kroah.com] 
> > Sent: Wednesday, October 12, 2005 8:28 PM
> > To: Thomas Schäfer
> > Cc: linux-hotplug-devel@lists.sourceforge.net
> > Subject: Re: PCIe hotplug on far PCIe switch ports
> > 
> > Try asking this on the pci hotplug mailing list, you might 
> > get a better response there.
> > 
> > thanks,
> > 
> > greg k-h
> > 
> > 
> 
> Where is this list hosted? I've already seen some hints to the pci hotplug mailing list before but was not able to find it. The linux-hotplug page at SF only points to the linux-hotplug-devel list, so I decided to post here.

https://lists.sourceforge.net/lists/listinfo/pcihpd-discuss

Gary

-- 
Gary Hade
IBM Linux Technology Center
503-578-4503  IBM T/L: 775-4503
garyhade@us.ibm.com
http://www.ibm.com/linux/ltc


-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel