pci-e hotplug on Intel SE7520HPAF2 board

pci-e hotplug on Intel SE7520HPAF2 board

[Tom Duffy]

[-- Attachment #1: Type: text/plain, Size: 4626 bytes --]

I recently got the new hotplug board from Intel with the 7520
(lindehurst) chipset.  I am running the latest Fedora Core 3 kernel
2.6.10-1.770_FC3smp.

Hot removal seems to work OK.  Before I pull the card, I see this on my
lspci list:

08:00.0 Ethernet controller: Intel Corp.: Unknown device 105f (rev 03)
08:00.1 Ethernet controller: Intel Corp.: Unknown device 105f (rev 03)

I pull the card, and it no longer shows up in lspci.  Also, the e1000
driver deallocates that card.  I see this in dmesg:

divert: freeing divert_blk for eth2
divert: freeing divert_blk for eth3

And ifconfig -a no longer shows those.  Great.

HOWEVER, I am not getting anything on hot insert.

When I load the acpiphp driver, I get these messages:

acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.4
acpiphp: Slot [4] registered
acpiphp: Slot [3] registered

I assume that means it likes those slots.  Here is lspci's view of them:

00:04.0 PCI bridge: Intel Corp. E7525/E7520 PCI Express Port B (rev 0c) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size 10
	Bus: primary=00, secondary=08, subordinate=09, sec-latency=0
	I/O behind bridge: 0000a000-0000afff
	Memory behind bridge: fd900000-fe0fffff
	Prefetchable memory behind bridge: 00000000f6200000-00000000f8100000
	Secondary status: 66Mhz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Message Signalled Interrupts: 64bit- Queue=0/1 Enable-
		Address: fee00000  Data: 0000
	Capabilities: [64] Express Root Port (Slot+) IRQ 0
		Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
		Device: Latency L0s <64ns, L1 <1us
		Device: Errors: Correctable+ Non-Fatal+ Fatal+ Unsupported-
		Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
		Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
		Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 4
		Link: Latency L0s <4us, L1 unlimited
		Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
		Link: Speed 2.5Gb/s, Width x8
		Slot: AtnBtn- PwrCtrl+ MRL- AtnInd+ PwrInd+ HotPlug+ Surpise-
		Slot: Number 4, PowerLimit 25.000000
		Slot: Enabled AtnBtn- PwrFlt+ MRL- PresDet+ CmdCplt- HPIrq+
		Slot: AttnInd Off, PwrInd Off, Power+
		Root: Correctable+ Non-Fatal+ Fatal+ PME+
	Capabilities: [100] Advanced Error Reporting

00:06.0 PCI bridge: Intel Corp. E7520 PCI Express Port C (rev 0c) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size 10
	Bus: primary=00, secondary=06, subordinate=07, sec-latency=0
	I/O behind bridge: 00009000-00009fff
	Memory behind bridge: fd100000-fd8fffff
	Prefetchable memory behind bridge: 00000000f4200000-00000000f6100000
	Secondary status: 66Mhz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Message Signalled Interrupts: 64bit- Queue=0/1 Enable-
		Address: fee00000  Data: 0000
	Capabilities: [64] Express Root Port (Slot+) IRQ 0
		Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
		Device: Latency L0s <64ns, L1 <1us
		Device: Errors: Correctable+ Non-Fatal+ Fatal+ Unsupported-
		Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
		Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
		Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 6
		Link: Latency L0s <4us, L1 unlimited
		Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
		Link: Speed 2.5Gb/s, Width x8
		Slot: AtnBtn- PwrCtrl+ MRL- AtnInd+ PwrInd+ HotPlug+ Surpise-
		Slot: Number 3, PowerLimit 25.000000
		Slot: Enabled AtnBtn- PwrFlt+ MRL- PresDet+ CmdCplt- HPIrq+
		Slot: AttnInd Off, PwrInd Off, Power+
		Root: Correctable+ Non-Fatal+ Fatal+ PME+
	Capabilities: [100] Advanced Error Reporting

What do I need to do to get hot insert working?

Thanks,

-tduffy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: pci-e hotplug on Intel SE7520HPAF2 board

[Greg KH]

On Mon, Mar 07, 2005 at 04:58:27PM -0800, Tom Duffy wrote:
> 
> What do I need to do to get hot insert working?

Did you tell the kernel to enable power to these slots?

I do not think that hot add and hot remove are supported, you need to
disable the power to them (or enable them) by either using a hardware
switch on the latch of the case, or by using the sysfs interface (under
/sys/bus/pci/slots/).

Also, there is a pci hotplug specific mailing list, if you are
interested.

thanks,

greg k-h


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: pci-e hotplug on Intel SE7520HPAF2 board

[Tom Duffy]

[-- Attachment #1: Type: text/plain, Size: 947 bytes --]

On Mon, 2005-03-07 at 22:55 -0800, Greg KH wrote:
> On Mon, Mar 07, 2005 at 04:58:27PM -0800, Tom Duffy wrote:
> > 
> > What do I need to do to get hot insert working?
> 
> Did you tell the kernel to enable power to these slots?
> 
> I do not think that hot add and hot remove are supported, you need to
> disable the power to them (or enable them) by either using a hardware
> switch on the latch of the case, or by using the sysfs interface (under
> /sys/bus/pci/slots/).

[root@intlhotp-1 ~]# cat /sys/bus/pci/slots/3/power
1
[root@intlhotp-1 ~]# cat /sys/bus/pci/slots/4/power
1
[root@intlhotp-1 ~]# echo 1 > /sys/bus/pci/slots/3/power
acpiphp_glue: No new device found
[root@intlhotp-1 ~]# echo 1 > /sys/bus/pci/slots/4/power
acpiphp_glue: No new device found

> Also, there is a pci hotplug specific mailing list, if you are
> interested.

Ok, we can move discussion there if you like.  What is the list?

-tduffy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: pci-e hotplug on Intel SE7520HPAF2 board

[Greg KH]

On Tue, Mar 08, 2005 at 09:09:08AM -0800, Tom Duffy wrote:
> On Mon, 2005-03-07 at 22:55 -0800, Greg KH wrote:
> > On Mon, Mar 07, 2005 at 04:58:27PM -0800, Tom Duffy wrote:
> > > 
> > > What do I need to do to get hot insert working?
> > 
> > Did you tell the kernel to enable power to these slots?
> > 
> > I do not think that hot add and hot remove are supported, you need to
> > disable the power to them (or enable them) by either using a hardware
> > switch on the latch of the case, or by using the sysfs interface (under
> > /sys/bus/pci/slots/).
> 
> [root@intlhotp-1 ~]# cat /sys/bus/pci/slots/3/power
> 1
> [root@intlhotp-1 ~]# cat /sys/bus/pci/slots/4/power
> 1

So the power is already enabled?

> [root@intlhotp-1 ~]# echo 1 > /sys/bus/pci/slots/3/power
> acpiphp_glue: No new device found
> [root@intlhotp-1 ~]# echo 1 > /sys/bus/pci/slots/4/power
> acpiphp_glue: No new device found

That's not good.  But you are using the acpi driver, not the pciexpress
driver, is that what you mean to do?

> > Also, there is a pci hotplug specific mailing list, if you are
> > interested.
> 
> Ok, we can move discussion there if you like.  What is the list?

pcihpd-discuss@lists.sourceforge.net

thanks,

greg k-h


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: pci-e hotplug on Intel SE7520HPAF2 board

[Tom Duffy]

[-- Attachment #1: Type: text/plain, Size: 2424 bytes --]

[ moving discussion to pcihpd-discuss ]

On Tue, 2005-03-08 at 09:33 -0800, Greg KH wrote:
> So the power is already enabled?

I guess so, although the LED does not illuminate unless I cold boot it
with a card inserted.

> > [root@intlhotp-1 ~]# echo 1 > /sys/bus/pci/slots/3/power
> > acpiphp_glue: No new device found
> > [root@intlhotp-1 ~]# echo 1 > /sys/bus/pci/slots/4/power
> > acpiphp_glue: No new device found
> 
> That's not good.  But you are using the acpi driver, not the pciexpress
> driver, is that what you mean to do?

I was hoping that I could use the ACPI driver since the pci express
native one will not work on the nvidia CK804 which is the real reason I
am trying out pci express hotplug.

But, it seems using the native driver does make the Intel board happier.

[root@intlhotp-1 ~]# rmmod acpiphp
[root@intlhotp-1 ~]# modprobe pciehp
pciehp: add_host_bridge: status 5
pciehp: acpi_pciehprm:\_SB_.PCI0.EPB0 OSHP fails=0x5
pciehp: acpi_pciehprm:\_SB_.PCI0.EPB0 OSHP fails=0x5
pciehp: acpi_pciehprm:\_SB_.PCI0.EPB0 OSHP fails=0x5
pciehp: acpi_pciehprm:\_SB_.PCI0.EPB0 OSHP fails=0x5
pciehp: acpi_pciehprm:\_SB_.PCI0.EPB0 OSHP fails=0x5
pciehp: acpi_pciehprm:\_SB_.PCI0.EPB0 OSHP fails=0x5
pciehp: acpi_pciehprm:\_SB_.PCI0.EPB0 OSHP fails=0x5
pciehp: acpi_pciehprm:\_SB_.PCI0.EPB0 OSHP fails=0x5
pciehp: acpi_pciehprm:\_SB_.PCI0.EPC0 OSHP fails=0x5
pciehp: acpi_pciehprm:\_SB_.PCI0.EPC0 OSHP fails=0x5
pciehp: acpi_pciehprm:\_SB_.PCI0.EPC0 OSHP fails=0x5
pciehp: acpi_pciehprm:\_SB_.PCI0.EPC0 OSHP fails=0x5
pciehp: acpi_pciehprm:\_SB_.PCI0.EPC0 OSHP fails=0x5
pciehp: acpi_pciehprm:\_SB_.PCI0.EPC0 OSHP fails=0x5
pciehp: acpi_pciehprm:\_SB_.PCI0.EPC0 OSHP fails=0x5
pciehp: acpi_pciehprm:\_SB_.PCI0.EPC0 OSHP fails=0x5

Not sure what those messages are telling me...

But, I can enable the slot:
[root@intlhotp-1 ~]# cat /sys/bus/pci/slots/4/power
0
[root@intlhotp-1 ~]# echo 1 > /sys/bus/pci/slots/4/power
PCI: Enabling device 0000:08:00.0 (0000 -> 0003)
PCI: Enabling device 0000:08:00.1 (0000 -> 0003)
[root@intlhotp-1 ~]# cat /sys/bus/pci/slots/4/power
1
[root@intlhotp-1 ~]# lspci
<snip>
08:00.0 Ethernet controller: Intel Corp.: Unknown device 105f (rev 03)
08:00.1 Ethernet controller: Intel Corp.: Unknown device 105f (rev 03)
<snip>

Cool.  The driver loads and works.

But, I would like to get the ACPI driver working...

-tduffy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: pci-e hotplug on Intel SE7520HPAF2 board

[Rajesh Shah]

On Tue, Mar 08, 2005 at 10:02:51AM -0800, Tom Duffy wrote:
> 
> > > [root@intlhotp-1 ~]# echo 1 > /sys/bus/pci/slots/3/power
> > > acpiphp_glue: No new device found
> > > [root@intlhotp-1 ~]# echo 1 > /sys/bus/pci/slots/4/power
> > > acpiphp_glue: No new device found
> > 
> > That's not good.  But you are using the acpi driver, not the pciexpress
> > driver, is that what you mean to do?
> 
I didn't think this board (firmware) supported ACPI based hotplug
but you did get a couple of slots registered messages. You could
load the acpiphp driver with debug enabled so it prints more info
about where it found the acpi hotpluggable slots. Perhaps the
hotpluggable slots it found are not where you're plugging the
card in.

Looking at the system specs, it looks like some slots are pci-e
slots and should be hotpluggable using pciehp, while other slots
are pci(x) and hotpluggable using shpchp. Could you also try to
load and use shpchp if you were using one of those slots?

> I was hoping that I could use the ACPI driver since the pci express
> native one will not work on the nvidia CK804 which is the real reason I
> am trying out pci express hotplug.
> 
Hmm... the hotplug drivers shouldn't care which card you are
hotplugging. They should simply pci-initialize the card and
cause the hotplug event to get the corresponding driver to load.
Do you know why the nvidia cards would only work with ACPI hotplug?

> 
> pciehp: acpi_pciehprm:\_SB_.PCI0.EPC0 OSHP fails=0x5
> pciehp: acpi_pciehprm:\_SB_.PCI0.EPC0 OSHP fails=0x5
> 
> Not sure what those messages are telling me...
> 
It's telling you the firmware hasn't defined the ACPI OSHP methods
for those hotpluggable slots. OSHP is used to tell firmware that
the OS supports native pcie/shpc hotplug, so it shouldn't try to
emulate acpi hotplug on the slot. Lack of this method is harmless
if the firmware never supported acpi hotplug on this slot anyway.

I'm CC'ing Dely who may know more about what's going on.

Rajesh


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

RE: pci-e hotplug on Intel SE7520HPAF2 board

[Sy, Dely L]

On Monday, March 07, 2005 4:58 PM, Tom Duffy wrote:
> I recently got the new hotplug board from Intel with the 7520
> (lindehurst) chipset.
> Hot removal seems to work OK.  Before I pull the card, I see this on 
> my lspci list:
> HOWEVER, I am not getting anything on hot insert.

When using acpiphp driver for hot-plug, one has to use attention button
to initiate power-up of the slot for there is no defined ACPI method
that 
the acpiphp driver can call to power-up slot.  The driver call _PS0 when

the user uses CLI (by echo 1 > /sys/pci/bus/slots/X).  However, this 
method is not defined to do the powering-up sequence for PCI hot-plug.
Maybe a standard method should be defined for such purpose so one can
use
CLI to power-up slot.

> > pciehp: acpi_pciehprm:\_SB_.PCI0.EPC0 OSHP fails=0x5
> > pciehp: acpi_pciehprm:\_SB_.PCI0.EPC0 OSHP fails=0x5
> > 
> > Not sure what those messages are telling me...
> > 
> It's telling you the firmware hasn't defined the ACPI OSHP methods
> for those hotpluggable slots. OSHP is used to tell firmware that
> the OS supports native pcie/shpc hotplug, so it shouldn't try to
> emulate acpi hotplug on the slot. Lack of this method is harmless
> if the firmware never supported acpi hotplug on this slot anyway.

As Rajesh mentioned, the OSHP method is called to gain control of native
hot-plug if the OS is ACPI enabled.  0x5 means that OSHP was not 
implemented for those devices. The driver traverses the host-bridge and 
p2p bridges and executes OSHP when it encountered hot-pluggable slots.  
Don't know why this BIOS didn't implemented OSHP on hot-pluggable slots.
Can you check cat /proc/interrupts to see if interrupts occurred when
you
do hot-plug using pciehp?

Thanks,
Dely


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id\x14396&opÌk
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

RE: pci-e hotplug on Intel SE7520HPAF2 board

[Tom Duffy]

[-- Attachment #1: Type: text/plain, Size: 3368 bytes --]

On Tue, 2005-03-08 at 12:44 -0800, Sy, Dely L wrote:
> When using acpiphp driver for hot-plug, one has to use attention button
> to initiate power-up of the slot for there is no defined ACPI method
> that 
> the acpiphp driver can call to power-up slot.  The driver call _PS0 when
> 
> the user uses CLI (by echo 1 > /sys/pci/bus/slots/X).  However, this 
> method is not defined to do the powering-up sequence for PCI hot-plug.
> Maybe a standard method should be defined for such purpose so one can
> use
> CLI to power-up slot.

Would this be implemented in the Linux driver?  Or would this need to be
defined in the ACPI hotplug spec?

> Can you check cat /proc/interrupts to see if interrupts occurred when
> you
> do hot-plug using pciehp?

[root@intlhotp-1 ~]# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
  0:   20167864   20177040   20177525   20239359    IO-APIC-edge  timer
  1:         26          2          0          8    IO-APIC-edge  i8042
  4:       2903        827         59       4992    IO-APIC-edge  serial
  8:          0          0          0          1    IO-APIC-edge  rtc
  9:          1          4          3          1   IO-APIC-level  acpi
 12:          0         28        960         82    IO-APIC-edge  i8042
 14:          1     362803     362757         35    IO-APIC-edge  ide0
169:          0          0          0          0   IO-APIC-level  uhci_hcd
177:      14278       2334         91      19521   IO-APIC-level  ioc0
185:          0          0          0         30   IO-APIC-level  ioc1
193:     198396          0          0       1289   IO-APIC-level  eth0
209:          0          0          0          0   IO-APIC-level  ehci_hcd
217:        779          1          2          2   IO-APIC-level  uhci_hcd, pciehp, pciehp
225:          0          0          0          0   IO-APIC-level  uhci_hcd
NMI:          0          0          0          0
LOC:   80766982   80766981   80766980   80766979
ERR:          0
MIS:          0
[root@intlhotp-1 ~]# echo 1 > /sys/bus/pci/slots/4/power
[root@intlhotp-1 ~]# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
  0:   20187870   20197047   20191221   20255956    IO-APIC-edge  timer
  1:         26          2          0          8    IO-APIC-edge  i8042
  4:       2903        827         59       4992    IO-APIC-edge  serial
  8:          0          0          0          1    IO-APIC-edge  rtc
  9:          1          5          5          1   IO-APIC-level  acpi
 12:          0         28        960         82    IO-APIC-edge  i8042
 14:          1     363073     363117         35    IO-APIC-edge  ide0
169:          0          0          0          0   IO-APIC-level  uhci_hcd
177:      14310       2354         91      19524   IO-APIC-level  ioc0
185:          0          0          0         30   IO-APIC-level  ioc1
193:     198724          0          0       1289   IO-APIC-level  eth0
209:          0          0          0          0   IO-APIC-level  ehci_hcd
217:        779          4          2          2   IO-APIC-level  uhci_hcd, pciehp, pciehp
225:          0          0          0          0   IO-APIC-level  uhci_hcd
NMI:          0          0          0          0
LOC:   80837292   80837291   80837290   80837289
ERR:          0
MIS:          0


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]