PCI HotPlug and Adding Resources after Linux Boots

PCI HotPlug and Adding Resources after Linux Boots

[Morrison, Tom]

[-- Attachment #1: Type: text/plain, Size: 4725 bytes --]

I am not exactly sure who to direct this question to (general Linux kernel or LinuxPPC),
so I am directing to both - in hopes that someone will recognize this problem - and perhaps
give me some suggestions on how to proceed...

I am running Linux (2.6.23x (and 2.6.27.x)) on a MPC8572 based system.

I have an 8616 switch that has a Port (6) connected to a FPGA that is
NOT loaded at before Linux boots (note: this port is configured for HOTPLUG
events - which we do get after FPGA  is loaded). We are NOT using a
static device tree map (because the devices in the system are very dynamic).

We use instead the pci auto scan mechanism(s) to scan/assign resources
(including into the BAR registers) at bootup to all of the devices that are
attached to this MPC8572...

Here is the port that is attached to the device (note: there are NO
resources assigned at this point this port):

-------------------------------------------------------------------------------------------------
02:06.0 PCI bridge: PLX Technology, Inc.: Unknown device 8616 (rev bb) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=02, secondary=05, subordinate=05, sec-latency=0
        Capabilities: [40] Power Management version 3
        Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/2 Enable+
        Capabilities: [68] #10 [0162]
        Capabilities: [a4] #0d [0000]

root@slave7 ~ # lspci -t
-+-[01]---00.0-[02-05]--+-01.0
 |                      +-04.0-[03]--
 |                      +-05.0-[04]--
 |                      \-06.0-[05]-

-------------------------------------------------------------------------------------------------

Later, after I detect there is an FPGA to load - I load it. At completion of the
loading of the FPGA - the 8616  detects the FPGA - and creates a HotPlug
event that the PCI Express HotPlug Driver handles:
-------------------------------------------------------------------------------------------------

root@slave7 ~ # pciehp: pcie_isr: intr_loc 8
pciehp: pciehp:  Presence/Notify input change.
pciehp: Card present on Slot(0005_0070)
pciehp: Surprise Removal
pciehp: hpc_get_power_status: SLOTCTRL 80 value read 8
pciehp: hpc_get_attention_status: SLOTCTRL 80, value read 8
pciehp: board_added: slot device, slot offset, hp slot = 0, 0 ,0
pciehp: hpc_check_lnk_status: lnk_status = 2021
PCI: Found 0000:05:00.0 [1172/0004] 00ff00 00
PCI: Calling quirk c0012d3c for 0000:05:00.0
program_fw_provided_values: Could not get hotplug parameters
entering assign resources (size: 2000000)
PCI: Failed to allocate mem resource #0:2000000@0 for 0000:05:00.0
bus pci: add device 0000:05:00.0
entering uevent
pci: Trying to Match Device 0000:05:00.0 with Driver pcieport-driver
pci: Trying to Match Device 0000:05:00.0 with Driver serial
pci: Trying to Match Device 0000:05:00.0 with Driver pexntb
pciehp: hpc_get_power_status: SLOTCTRL 80 value read 8
pciehp: hpc_get_attention_status: SLOTCTRL 80, value read 8

02:06.0 PCI bridge: PLX Technology, Inc.: Unknown device 8616 (rev bb) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=02, secondary=05, subordinate=05, sec-latency=0
        Capabilities: [40] Power Management version 3
        Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/2 Enable+
        Capabilities: [68] #10 [0162]
        Capabilities: [a4] #0d [0000]

05:00.0 Class ff00: Altera Corporation: Unknown device 0004 (rev 01)
        Subsystem: Altera Corporation: Unknown device 0004
        Flags: fast devsel
        Capabilities: [50] Message Signalled Interrupts: 64bit+ Queue=0/5 Enable-
        Capabilities: [78] Power Management version 3
        Capabilities: [80] #10 [0001]

root@slave7 ~ # lspci -t
-+-[01]---00.0-[02-05]--+-01.0
 |                      +-04.0-[03]--
 |                      +-05.0-[04]--
 |                      \-06.0-[05]----00.0
 \-[00]---00.0

-------------------------------------------------------------------------------------------------

So, as you can see - the device has been read - and it requires 32M of resources, but
because its parent doesn't have any resources allocated - it seemingly can't allocate and
use any additional resources.

How do I 'customize' and/or add resources at this point for this device (using semi-standard mechanisms)?

Thanks in advance for any/all ideas...


I


Tom Morrison
Principal Software Engineer
EMPIRIX
20 Crosby Drive - Bedford, MA  01730
p: 781.266.3567 f: 781.266.3670
email: tmorrison@empirix.com<mailto:tmorrison@empirix.com>
www.empirix.com<http://www.empirix.com/>




[-- Attachment #2: Type: text/html, Size: 24653 bytes --]

Re: PCI HotPlug and Adding Resources after Linux Boots

[Benjamin Herrenschmidt]

On Tue, 2009-09-22 at 15:36 -0400, Morrison, Tom wrote:
> I am not exactly sure who to direct this question to (general Linux
> kernel or LinuxPPC),

PCI Hotplug is reasonably arch specific at the moment so I suppose
here's is as good as anywhere else to ask :-)

> so I am directing to both – in hopes that someone will recognize this
> problem – and perhaps
> 
> give me some suggestions on how to proceed…

There's a few things you can do, though I don't have time just right now
to give you a detailed answer. I'll try again later.

In the meantime, some of the answers could be around not using full
automatic resource assignment, but instead, pre-initializing the top
bridge with some resources that are going to be enough for the device.

You can also try to get the bridge to re-allocate. There's various funky
locking issues with doing that though as long as it's during boot time,
it's not too much of a problem.

There are other more or less hackish ways to do it, but I'll have to
give it more thought.

I'm quite stretched at the moment so if you don't hear back from me in
the upcoming few days, don't hesitate to ping me again.

Cheers,
Ben. 

> 
> I am running Linux (2.6.23x (and 2.6.27.x)) on a MPC8572 based system.
> 
>  
> 
> I have an 8616 switch that has a Port (6) connected to a FPGA that is 
> 
> NOT loaded at before Linux boots (note: this port is configured for
> HOTPLUG
> 
> events - which we do get after FPGA  is loaded). We are NOT using a 
> 
> static device tree map (because the devices in the system are very
> dynamic).
> 
> 
> We use instead the pci auto scan mechanism(s) to scan/assign
> resources 
> 
> (including into the BAR registers) at bootup to all of the devices
> that are 
> 
> attached to this MPC8572…
> 
>  
> 
> Here is the port that is attached to the device (note: there are NO 
> 
> resources assigned at this point this port):
> 
>  
> 
> -------------------------------------------------------------------------------------------------
> 
> 02:06.0 PCI bridge: PLX Technology, Inc.: Unknown device 8616 (rev bb)
> (prog-if 00 [Normal decode])
> 
>         Flags: bus master, fast devsel, latency 0
> 
>         Bus: primary=02, secondary=05, subordinate=05, sec-latency=0
> 
>         Capabilities: [40] Power Management version 3
> 
>         Capabilities: [48] Message Signalled Interrupts: 64bit+
> Queue=0/2 Enable+
> 
>         Capabilities: [68] #10 [0162]
> 
>         Capabilities: [a4] #0d [0000]
> 
>  
> 
> root@slave7 ~ # lspci -t
> 
> -+-[01]---00.0-[02-05]--+-01.0
> 
>  |                      +-04.0-[03]--
> 
>  |                      +-05.0-[04]--
> 
>  |                      \-06.0-[05]—
> 
>  
> 
> -------------------------------------------------------------------------------------------------
> 
>  
> 
> Later, after I detect there is an FPGA to load - I load it. At
> completion of the
> 
> loading of the FPGA - the 8616  detects the FPGA – and creates a
> HotPlug 
> 
> event that the PCI Express HotPlug Driver handles:
> 
> -------------------------------------------------------------------------------------------------
> 
>  
> 
> root@slave7 ~ # pciehp: pcie_isr: intr_loc 8
> 
> pciehp: pciehp:  Presence/Notify input change.
> 
> pciehp: Card present on Slot(0005_0070)
> 
> pciehp: Surprise Removal
> 
> pciehp: hpc_get_power_status: SLOTCTRL 80 value read 8
> 
> pciehp: hpc_get_attention_status: SLOTCTRL 80, value read 8
> 
> pciehp: board_added: slot device, slot offset, hp slot = 0, 0 ,0
> 
> pciehp: hpc_check_lnk_status: lnk_status = 2021
> 
> PCI: Found 0000:05:00.0 [1172/0004] 00ff00 00
> 
> PCI: Calling quirk c0012d3c for 0000:05:00.0
> 
> program_fw_provided_values: Could not get hotplug parameters
> 
> entering assign resources (size: 2000000)
> 
> PCI: Failed to allocate mem resource #0:2000000@0 for 0000:05:00.0
> 
> bus pci: add device 0000:05:00.0
> 
> entering uevent
> 
> pci: Trying to Match Device 0000:05:00.0 with Driver pcieport-driver
> 
> pci: Trying to Match Device 0000:05:00.0 with Driver serial
> 
> pci: Trying to Match Device 0000:05:00.0 with Driver pexntb
> 
> pciehp: hpc_get_power_status: SLOTCTRL 80 value read 8
> 
> pciehp: hpc_get_attention_status: SLOTCTRL 80, value read 8
> 
>  
> 
> 02:06.0 PCI bridge: PLX Technology, Inc.: Unknown device 8616 (rev bb)
> (prog-if 00 [Normal decode])
> 
>         Flags: bus master, fast devsel, latency 0
> 
>         Bus: primary=02, secondary=05, subordinate=05, sec-latency=0
> 
>         Capabilities: [40] Power Management version 3
> 
>         Capabilities: [48] Message Signalled Interrupts: 64bit+
> Queue=0/2 Enable+
> 
>         Capabilities: [68] #10 [0162]
> 
>         Capabilities: [a4] #0d [0000]
> 
>  
> 
> 05:00.0 Class ff00: Altera Corporation: Unknown device 0004 (rev 01)
> 
>         Subsystem: Altera Corporation: Unknown device 0004
> 
>         Flags: fast devsel
> 
>         Capabilities: [50] Message Signalled Interrupts: 64bit+
> Queue=0/5 Enable-
> 
>         Capabilities: [78] Power Management version 3
> 
>         Capabilities: [80] #10 [0001]
> 
>  
> 
> root@slave7 ~ # lspci -t
> 
> -+-[01]---00.0-[02-05]--+-01.0
> 
>  |                      +-04.0-[03]--
> 
>  |                      +-05.0-[04]--
> 
>  |                      \-06.0-[05]----00.0
> 
>  \-[00]---00.0
> 
>  
> 
> -------------------------------------------------------------------------------------------------
> 
>  
> 
> So, as you can see – the device has been read – and it requires 32M of
> resources, but
> 
> because its parent doesn’t have any resources allocated – it seemingly
> can’t allocate and
> 
> use any additional resources.
> 
>  
> 
> How do I ‘customize’ and/or add resources at this point for this
> device (using semi-standard mechanisms)?
> 
>  
> 
> Thanks in advance for any/all ideas…
> 
>  
> 
>  
> 
> I 
> 
>  
> 
> Tom Morrison
> Principal Software Engineer
> 
> EMPIRIX 
> 20 Crosby Drive - Bedford, MA  01730
> p: 781.266.3567 f: 781.266.3670 
> email: tmorrison@empirix.com 
> www.empirix.com
> 
> 
> 
> 
>  
> 
> 
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev

RE: PCI HotPlug and Adding Resources after Linux Boots

[Morrison, Tom]

Thank you for taking the time in confirming some of the=20
potential strategies I was already thinking about...:-)

I am going to try to pre-allocate some resources for
the specific port/slot/bridge device (because we know
at all times that port #6 (bus #5) is going to have
a FPGA end device associated with it...

Any other thoughts - whenever you can make them - would
be more than welcome!

Tom


>> -----Original Message-----
>> From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org]
>> Sent: Wednesday, September 23, 2009 5:42 AM
>> To: Morrison, Tom
>> Cc: linuxppc-dev@ozlabs.org; linux-kernel@vger.kernel.org
>> Subject: Re: PCI HotPlug and Adding Resources after Linux Boots
>>=20
>> On Tue, 2009-09-22 at 15:36 -0400, Morrison, Tom wrote:
>> > I am not exactly sure who to direct this question to (general Linux
>> > kernel or LinuxPPC),
>>=20
>> PCI Hotplug is reasonably arch specific at the moment so I suppose
>> here's is as good as anywhere else to ask :-)
>>=20
>> > so I am directing to both - in hopes that someone will recognize this
>> > problem - and perhaps
>> >
>> > give me some suggestions on how to proceed...
>>=20
>> There's a few things you can do, though I don't have time just right now
>> to give you a detailed answer. I'll try again later.
>>=20
>> In the meantime, some of the answers could be around not using full
>> automatic resource assignment, but instead, pre-initializing the top
>> bridge with some resources that are going to be enough for the device.
>>=20
>> You can also try to get the bridge to re-allocate. There's various funky
>> locking issues with doing that though as long as it's during boot time,
>> it's not too much of a problem.
>>=20
>> There are other more or less hackish ways to do it, but I'll have to
>> give it more thought.
>>=20
>> I'm quite stretched at the moment so if you don't hear back from me in
>> the upcoming few days, don't hesitate to ping me again.
>>=20
>> Cheers,
>> Ben.
>> >
>> > I am running Linux (2.6.23x (and 2.6.27.x)) on a MPC8572 based system.
>>=20
[Morrison, Tom]=20
<snip the rest original msg>

RE: PCI HotPlug and Adding Resources after Linux Boots

[Morrison, Tom]

Ben/all,

I have had some luck reserving some memory (via in this autoscan
(which reserves the resources)) which in this hack code - I reserve
an appropriate amount of resources for the bridge (by detecting=20
this special type of port).

The problem comes later on - when the Hotplug event comes - and it
still can't allocate the resources...

A member of this group (who is away from the office right now) had=20
the following comments:

>> In your case, if the only problem that you are running into is=20
>> that the resources cannot be assigned to the FPGA, it may be=20
>> sufficient to hardcode the forwarding addresses and subordinate=20
>> bridge number for port 6 of your 8616.  The reason you are seeing=20
>> that error message is because the parent bridge device for the=20
>> detected FPGA (port 6 of the 8616) does not have forwarded=20
>> resource regions that match what the FPGA is trying to claim.

I guess the question I am asking is my friend's statement possibly
true. And, if he is correct - I am a little confused as to exactly=20
where/how do I determine and configure this forwarding=20
addresses/subordinate bridge number (I'm really a newbie at this
level of PCI configuration).

Thank you in advance to any/all who can help me figure this out!

Tom



>> -----Original Message-----

[Morrison, Tom]=20
>>=20
>> There's a few things you can do, though I don't have time just right now
>> to give you a detailed answer. I'll try again later.
>>=20
>> In the meantime, some of the answers could be around not using full
>> automatic resource assignment, but instead, pre-initializing the top
>> bridge with some resources that are going to be enough for the device.
>>=20
>> You can also try to get the bridge to re-allocate. There's various funky
>> locking issues with doing that though as long as it's during boot time,
>> it's not too much of a problem.
>>=20
>> There are other more or less hackish ways to do it, but I'll have to
>> give it more thought.
>>=20
>> I'm quite stretched at the moment so if you don't hear back from me in
>> the upcoming few days, don't hesitate to ping me again.
>>=20
>> Cheers,
>> Ben.
>>=20
>> >
>> > I am running Linux (2.6.23x (and 2.6.27.x)) on a MPC8572 based system.
>> >
>> >
>> >
>> > I have an 8616 switch that has a Port (6) connected to a FPGA that is
>> >
>> > NOT loaded at before Linux boots (note: this port is configured for
>> > HOTPLUG
>> >
>> > events - which we do get after FPGA  is loaded). We are NOT using a
>> >
>> > static device tree map (because the devices in the system are very
>> > dynamic).
>> >
>> >
>> > We use instead the pci auto scan mechanism(s) to scan/assign
>> > resources
>> >
>> > (including into the BAR registers) at bootup to all of the devices
>> > that are
>> >
>> > attached to this MPC8572...
>> >
>> >
>> >
>> > Here is the port that is attached to the device (note: there are NO
>> >
>> > resources assigned at this point this port):
>> >
>> >
>> >
>> > ----------------------------------------------------------------------=
-
>> --------------------------
>> >
>> > 02:06.0 PCI bridge: PLX Technology, Inc.: Unknown device 8616 (rev bb)
>> > (prog-if 00 [Normal decode])
>> >
>> >         Flags: bus master, fast devsel, latency 0
>> >
>> >         Bus: primary=3D02, secondary=3D05, subordinate=3D05, sec-laten=
cy=3D0
>> >
>> >         Capabilities: [40] Power Management version 3
>> >
>> >         Capabilities: [48] Message Signalled Interrupts: 64bit+
>> > Queue=3D0/2 Enable+
>> >
>> >         Capabilities: [68] #10 [0162]
>> >
>> >         Capabilities: [a4] #0d [0000]
>> >
>> >
>> >
>> > root@slave7 ~ # lspci -t
>> >
>> > -+-[01]---00.0-[02-05]--+-01.0
>> >
>> >  |                      +-04.0-[03]--
>> >
>> >  |                      +-05.0-[04]--
>> >
>> >  |                      \-06.0-[05]-
>> >
>> >
>> >
>> > ----------------------------------------------------------------------=
-
>> --------------------------
>> >
>> >
>> >
>> > Later, after I detect there is an FPGA to load - I load it. At
>> > completion of the
>> >
>> > loading of the FPGA - the 8616  detects the FPGA - and creates a
>> > HotPlug
>> >
>> > event that the PCI Express HotPlug Driver handles:
>> >
>> > ----------------------------------------------------------------------=
-
>> --------------------------
>> >
>> >
>> >
>> > root@slave7 ~ # pciehp: pcie_isr: intr_loc 8
>> >
>> > pciehp: pciehp:  Presence/Notify input change.
>> >
>> > pciehp: Card present on Slot(0005_0070)
>> >
>> > pciehp: Surprise Removal
>> >
>> > pciehp: hpc_get_power_status: SLOTCTRL 80 value read 8
>> >
>> > pciehp: hpc_get_attention_status: SLOTCTRL 80, value read 8
>> >
>> > pciehp: board_added: slot device, slot offset, hp slot =3D 0, 0 ,0
>> >
>> > pciehp: hpc_check_lnk_status: lnk_status =3D 2021
>> >
>> > PCI: Found 0000:05:00.0 [1172/0004] 00ff00 00
>> >
>> > PCI: Calling quirk c0012d3c for 0000:05:00.0
>> >
>> > program_fw_provided_values: Could not get hotplug parameters
>> >
>> > entering assign resources (size: 2000000)
>> >
>> > PCI: Failed to allocate mem resource #0:2000000@0 for 0000:05:00.0
>> >
>> > bus pci: add device 0000:05:00.0
>> >
>> > entering uevent
>> >
>> > pci: Trying to Match Device 0000:05:00.0 with Driver pcieport-driver
>> >
>> > pci: Trying to Match Device 0000:05:00.0 with Driver serial
>> >
>> > pci: Trying to Match Device 0000:05:00.0 with Driver pexntb
>> >
>> > pciehp: hpc_get_power_status: SLOTCTRL 80 value read 8
>> >
>> > pciehp: hpc_get_attention_status: SLOTCTRL 80, value read 8
>> >
>> >
>> >
>> > 02:06.0 PCI bridge: PLX Technology, Inc.: Unknown device 8616 (rev bb)
>> > (prog-if 00 [Normal decode])
>> >
>> >         Flags: bus master, fast devsel, latency 0
>> >
>> >         Bus: primary=3D02, secondary=3D05, subordinate=3D05, sec-laten=
cy=3D0
>> >
>> >         Capabilities: [40] Power Management version 3
>> >
>> >         Capabilities: [48] Message Signalled Interrupts: 64bit+
>> > Queue=3D0/2 Enable+
>> >
>> >         Capabilities: [68] #10 [0162]
>> >
>> >         Capabilities: [a4] #0d [0000]
>> >
>> >
>> >
>> > 05:00.0 Class ff00: Altera Corporation: Unknown device 0004 (rev 01)
>> >
>> >         Subsystem: Altera Corporation: Unknown device 0004
>> >
>> >         Flags: fast devsel
>> >
>> >         Capabilities: [50] Message Signalled Interrupts: 64bit+
>> > Queue=3D0/5 Enable-
>> >
>> >         Capabilities: [78] Power Management version 3
>> >
>> >         Capabilities: [80] #10 [0001]
>> >
>> >
>> >
>> > root@slave7 ~ # lspci -t
>> >
>> > -+-[01]---00.0-[02-05]--+-01.0
>> >
>> >  |                      +-04.0-[03]--
>> >
>> >  |                      +-05.0-[04]--
>> >
>> >  |                      \-06.0-[05]----00.0
>> >
>> >  \-[00]---00.0
>> >
>> >
>> >
>> > ----------------------------------------------------------------------=
-
>> --------------------------
>> >
>> >
>> >
>> > So, as you can see - the device has been read - and it requires 32M of
>> > resources, but
>> >
>> > because its parent doesn't have any resources allocated - it seemingly
>> > can't allocate and
>> >
>> > use any additional resources.
>> >
>> >
>> >
>> > How do I 'customize' and/or add resources at this point for this
>> > device (using semi-standard mechanisms)?
>> >
>> >
>> >
>> > Thanks in advance for any/all ideas...
>> >
>> >
>> >
>> >
>> >
>> > I
>> >
>> >
>> >
>> > Tom Morrison
>> > Principal Software Engineer
>> >
>> > EMPIRIX
>> > 20 Crosby Drive - Bedford, MA  01730
>> > p: 781.266.3567 f: 781.266.3670
>> > email: tmorrison@empirix.com
>> > www.empirix.com
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > Linuxppc-dev mailing list
>> > Linuxppc-dev@lists.ozlabs.org
>> > https://lists.ozlabs.org/listinfo/linuxppc-dev
>>=20