* Re: pci express switch driver
2004-10-10 2:15 pci express switch driver Sadik, Pallathu
@ 2004-10-30 4:37 ` Greg KH
2004-11-02 1:58 ` Rajesh Shah
` (5 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Greg KH @ 2004-10-30 4:37 UTC (permalink / raw)
To: linux-hotplug
On Sat, Oct 09, 2004 at 07:15:37PM -0700, Sadik, Pallathu wrote:
> Hello,
>
> I am am familiar with linux kernel. I have ported
> linux to IDT's MIPS based platforms and have written device
> drivers for their integrated ethernet controllers.
>
> Currently, I am working on developing a driver
> for a PCI express switch that have one upstream and
> multiple downstream ports with hotplug capability.
I've waited so long in responding, as I would hope the Intel people I
know would pipe up. But they didn't :(
Anyway, some other people are currently working on adding this kind of
functionality to Linux. But I have no idea when they will release it to
the public...
> I am totally new to PCI express and hotplug. I have
> gone through various kernel documents, source code
> and tutorials to understand the linux support for PCI
> express, particularly the hotplug feature support.
> I have some understanding of a PCI/PCI express driver
> structure in linux (from pm.txt, pci.txt and power/pci.txt
> etc of ~linux/Documentation). This clearly explains how
> to write a driver for a PCI/PCIexpress device/endpoint.
> However, I am not able to figure out whether these
> are applicable in the case of a switch.
>
> My hardware setup is as follows:
>
> PCI express switch:
> As I have mentioned above, the switch is going to
> have one upstream and multiple downstream ports.
> The downstream ports have implemented hotpulg feature
> (Attention indicator, power indicator, attention button,
> MRL sensor etc). They are capable of generating interrupts
> on the hotplug events.
And the current pci express hotplug driver does not work for you? If
not, I'm sure the author of that code would be interested in working
with you. Did you contact them?
> 1. Do I need to write a driver for this device? I couldn't locate a
> driver for PCI to PCI bridge in the linux code and functionally a
> PCI express switch is not that much different from a PCI bridge.
I think the kernel already has such a driver.
> I have gone through the source code and found different kinds of
> hotplug drivers under ~linux/drivers/pci/hotplug directory, namely
> pci_hotplug, shpchp, pciehp, acpiphp. I experimentally found out
> that the pci_hotplug module is required for the rest of the modules
> to load. However, I am not sure which one I should select.
Try them all, one of them should work :)
Seriously, the pciehp should be what you are looking for. If not, try
the shpchp driver. I've seen boxes that need both.
> As I understand acpiphp is for ACPI based hotplug control and
> pciehp is for PCI express native hotplug. Is that true? Do I need
> both of them for the correct operation?
Don't know, depends on how your chipset works.
> 4. Application code.
>
> I looked at Greg's website (www.kroah.com), and see that the pcihpview
> can be used to display all hotplug slots and enable/disable particular
> slots. Will this utility automatically detect the switch and control
> a switch?
If a pci hotplug driver is bound to that hardware, yes it will work. I
need to get an updated program out there one of these days, the last
released version doesn't even work on 2.6 (hint, grab the bk version, it
does.)
Hope this helps,
greg k-h
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_idU88&alloc_id\x12065&op=click
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: pci express switch driver
2004-10-10 2:15 pci express switch driver Sadik, Pallathu
2004-10-30 4:37 ` Greg KH
@ 2004-11-02 1:58 ` Rajesh Shah
2004-11-03 0:42 ` Sy, Dely L
` (4 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Rajesh Shah @ 2004-11-02 1:58 UTC (permalink / raw)
To: linux-hotplug
On Fri, Oct 29, 2004 at 09:37:51PM -0700, Greg KH wrote:
> On Sat, Oct 09, 2004 at 07:15:37PM -0700, Sadik, Pallathu wrote:
> >
> > Currently, I am working on developing a driver
> > for a PCI express switch that have one upstream and
> > multiple downstream ports with hotplug capability.
>
> I've waited so long in responding, as I would hope the Intel people I
> know would pipe up. But they didn't :(
>
That probably would be me (plus several other who discussed this
with Greg at OLS) :-). Sorry, my mail system seems to be
screwed up and I never saw the original post.
> Anyway, some other people are currently working on adding this kind of
> functionality to Linux. But I have no idea when they will release it to
> the public...
What we are working on is ACPI based hot-plug of an IO complex
rooted on a host bridge or a P2P bridge. If the switch you
describe looks like a p2p bridge to Linux, what we're working
on would probably overlap with what you need. We still
need to resolve some ACPI/namespace issues, so we're not quite
ready to send patches yet.
>
> > However, I am not able to figure out whether these
> > are applicable in the case of a switch.
> >
> > My hardware setup is as follows:
> >
> > PCI express switch:
> > As I have mentioned above, the switch is going to
> > have one upstream and multiple downstream ports.
> > The downstream ports have implemented hotpulg feature
> > (Attention indicator, power indicator, attention button,
> > MRL sensor etc). They are capable of generating interrupts
> > on the hotplug events.
>
What hotplug technology do you plan to use to hotplug the switch
itself? This is a platform/firmware capability. For example, if
you want to use SHPC based hotplug, there must be an SHPC
controller north of the switch you want to hotplug. If you
want to support ACPI based hotplug, your firmware has to
describe it in the ACPI namespace, generate the appropriate
notification etc.
Once the switch itself is hotplugged, downstream IO devices
can be hotplugged using SHPC/PCI-E mechanisms if the switch
has hotplug capabilities. Looks like your switch does
support SHPC/PCI-E hotplug.
>
> > 1. Do I need to write a driver for this device? I couldn't locate a
> > driver for PCI to PCI bridge in the linux code and functionally a
> > PCI express switch is not that much different from a PCI bridge.
>
Since we will be using ACPI based hotplug, we'll be extending
acpiphp.
>
> > pci_hotplug, shpchp, pciehp, acpiphp. I experimentally found out
> > that the pci_hotplug module is required for the rest of the modules
> > to load. However, I am not sure which one I should select.
>
You need to select the ones that your platform hardware and
firmware supports.
>
> > As I understand acpiphp is for ACPI based hotplug control and
> > pciehp is for PCI express native hotplug. Is that true? Do I need
> > both of them for the correct operation?
>
Again, depends on your platform capability.
Rajesh
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_idU88&alloc_id\x12065&op=click
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
^ permalink raw reply [flat|nested] 8+ messages in thread* RE: pci express switch driver
2004-10-10 2:15 pci express switch driver Sadik, Pallathu
2004-10-30 4:37 ` Greg KH
2004-11-02 1:58 ` Rajesh Shah
@ 2004-11-03 0:42 ` Sy, Dely L
2004-11-04 1:52 ` Sadik, Pallathu
` (3 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Sy, Dely L @ 2004-11-03 0:42 UTC (permalink / raw)
To: linux-hotplug
On Saturday, October 30, 2004 12:38 AM, Greg KH wrote:
>On Sat, Oct 09, 2004 at 07:15:37PM -0700, Sadik, Pallathu wrote:
>> Currently, I am working on developing a driver
>> for a PCI express switch that have one upstream and
>> multiple downstream ports with hotplug capability.
>
>I've waited so long in responding, as I would hope the Intel people I
>know would pipe up. But they didn't :(
>
I didn't see this email thread until someone pointed out to me
yesterday.
>> PCI express switch:
>> As I have mentioned above, the switch is going to
>> have one upstream and multiple downstream ports.
>> The downstream ports have implemented hotpulg feature
>> (Attention indicator, power indicator, attention button,
>> MRL sensor etc). They are capable of generating interrupts
>> on the hotplug events.
>
>And the current pci express hotplug driver does not work for you? If
>not, I'm sure the author of that code would be interested in working
>with you. Did you contact them?
>
The current pciehp driver in the kernel is for hot-plug of PCI Express
endpoint devices. Therefore, it should work for the multiple downstream
ports with hot-plug capability. However, more work needs to be done to
get it working on hot-plugging the PCI express switch; that is doing
hot-plug on the upstream port.
>> 1. Do I need to write a driver for this device? I couldn't locate a
>> driver for PCI to PCI bridge in the linux code and functionally a
>> PCI express switch is not that much different from a PCI bridge.
>
>I think the kernel already has such a driver.
>
>> I have gone through the source code and found different kinds of
>> hotplug drivers under ~linux/drivers/pci/hotplug directory, namely
>> pci_hotplug, shpchp, pciehp, acpiphp. I experimentally found out
>> that the pci_hotplug module is required for the rest of the modules
>> to load. However, I am not sure which one I should select.
>
>Try them all, one of them should work :)
>
>Seriously, the pciehp should be what you are looking for. If not, try
>the shpchp driver. I've seen boxes that need both.
>
If the downstream ports have PCI Express hot-pluggable slots, the pciehp
driver is used to do hot-plug of PCI Express endpoint devices. If the
downstream port has another PCI Express to PCI-X bridge attached to it,
and
this bridge has a SHPC (as specified in SHPC spec.) integrated, then the
shpchp is used to hot-plug PCI or PCI-X devices. Both drivers can run
on
the same system.
>> As I understand acpiphp is for ACPI based hotplug control and
>> pciehp is for PCI express native hotplug. Is that true? Do I need
>> both of them for the correct operation?
>
>Don't know, depends on how your chipset works.
>
To do hot-plug of PCI Express endpoint devices, one can use either
pciehp,
which uses system interrupts for hot-plug events, or acpihp, which uses
ACPI
based hot-plug control.
Thanks,
Dely
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_idU88&alloc_id\x12065&opÌk
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
^ permalink raw reply [flat|nested] 8+ messages in thread* RE: pci express switch driver
2004-10-10 2:15 pci express switch driver Sadik, Pallathu
` (2 preceding siblings ...)
2004-11-03 0:42 ` Sy, Dely L
@ 2004-11-04 1:52 ` Sadik, Pallathu
2004-11-04 22:22 ` Rajesh Shah
` (2 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Sadik, Pallathu @ 2004-11-04 1:52 UTC (permalink / raw)
To: linux-hotplug
Dely, Rajesh and Greg,
Thanks for the mail. Your answers cleared most of my doubts.
However I still have a few questions that I would like to clarify.
> The current pciehp driver in the kernel is for hot-plug of
> PCI Express endpoint devices. Therefore, it should work for
> the multiple downstream
>
> ports with hot-plug capability. However, more work needs to
> be done to
> get it working on hot-plugging the PCI express switch; that is doing
> hot-plug on the upstream port.
Why special care is needed for hot-plugging the upstream port?
As I understand, even if hot-plug signals are provided on the upstream
port of a switch/endpoint, the generated events gets consumed by
the downstream port of the root complex/switch where this card is plugged.
Hence, the mechanism to handle hot-plugging of the upstream port of
a switch should be same as hot-plugging of the downstream port of the root
port.
Our switch don't support hot-plug on the upstream port. However,
it does provide attention button that can generate Attention Button
Pressed event on the upstream port.
The motherboard I am using is from Supermicro: please see the URL
http://www.supermicro.com/products/motherboard/Xeon800/E7520/X6DH8-G2.cfm
Unfortunately, not much detail is awailable regarding the motherboard,
except that, it uses Intel Lindenhurst chipsets: E7520 (MCH) and
82801ER (ICH). This motherboard provides two x8 and one x4 PCIe slots.
I don't think these slots support hot-plug, since they have not
provided any hotplug signals (Attention Button etc). Hence, what
we plan to do is to plug two switches to this motherboard, one
directly to the root port and the second to the downstream port
of the first switch. Now, say, the user presses the
Attention Button on the upstream port of the second switch, it
generates an Attention_button_pressed event, that gets consumed
by the downstream port of the first switch. This switch in turn
generates an interrupt and the pciehp hotplug driver can take
care of this. The same thing would happen if the Attention Button
is pressed on the downstream port of the first switch, right?
Hence, the question of why special care is needed for upstream
hot-plug.
> If the downstream ports have PCI Express hot-pluggable slots,
> the pciehp driver is used to do hot-plug of PCI Express
> endpoint devices. If the
> downstream port has another PCI Express to PCI-X bridge
> attached to it, and this bridge has a SHPC (as specified in
> SHPC spec.) integrated, then the
>
> shpchp is used to hot-plug PCI or PCI-X devices. Both
> drivers can run on the same system.
As I mentioned above, we are planning to use two switches.
Do I still need the shpchp in this case?
> To do hot-plug of PCI Express endpoint devices, one can use
> either pciehp,
> which uses system interrupts for hot-plug events, or acpihp,
> which uses ACPI
> based hot-plug control.
The PCIe hotplug protocol specifies that, when a device is
in D1, D2 or D3hot state, the hot-plug controller generates
a wakeup event (PM_PME) instead of an interrupt. Please see
section 6.7.7 of PCI Exress Base specification, Rev 1.0a.
The Intel E7520 user manual says, when it receives a PM_PME
packet, it will forward a single MCHPME# output signal directly
to the PME# input of the ICH (82801ER). Please see section
5.9.6.4 of Intel E7520 Memory Controller Hub (MCH) Data sheet
available from
http://developer.intel.com/design/chipsets/datashts/index.htm
Now the ICH (82801ER)manual says, the PME# input will
cause an SCI to be generated. Please see section 5.13.5,
table 62 of the ICH manual. This manual can be obtained
from the above link. The SCI events are processed
by the ACPI layer, right? Hence, in order for the hotplug
to work under D1, D2 or D3hot state, the acpi hotplugging
is also needed, right?
The PCIe specification says that Firmware should
redirect System Interrupt to a GPE so that ACPI
can service the interrupts. It also says that
an apropriate _Exx GPE handler should be provided.
Is there a way to figure out whether the motherboard
supports ACPI tables for PCIe hotplug in the above
fashion? Is it possible to see it from the ACPI
tables that I can get using 'acpidump' command? If it is
not supported, is there a way I can add them?
(I have searched the acpi-devel archive, but couldn't
find any thread related to this topic. I am not sure
whether I should ask this question in that mailing list.)
Thanks and regards,
Sadik.
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_idU88&alloc_id\x12065&op=click
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: pci express switch driver
2004-10-10 2:15 pci express switch driver Sadik, Pallathu
` (3 preceding siblings ...)
2004-11-04 1:52 ` Sadik, Pallathu
@ 2004-11-04 22:22 ` Rajesh Shah
2004-11-04 23:40 ` Sadik, Pallathu
2004-11-05 21:57 ` Rajesh Shah
6 siblings, 0 replies; 8+ messages in thread
From: Rajesh Shah @ 2004-11-04 22:22 UTC (permalink / raw)
To: linux-hotplug
On Wed, Nov 03, 2004 at 05:52:00PM -0800, Sadik, Pallathu wrote:
> Dely, Rajesh and Greg,
>
> Why special care is needed for hot-plugging the upstream port?
> As I understand, even if hot-plug signals are provided on the upstream
> port of a switch/endpoint, the generated events gets consumed by
> the downstream port of the root complex/switch where this card is plugged.
> Hence, the mechanism to handle hot-plugging of the upstream port of
> a switch should be same as hot-plugging of the downstream port of the root
> port.
>
When you hot-insert the (upstream port of the) 2nd switch, it may
already have some devices connected to it (downstream). Also,
the switch _may_ contain hot-pluggable slots (that doesn't
appear to be the case with your motherboard). Someone needs to
collect resource requirements, program non-conflicting values
in the end devices and the summed up resource values in the
switch. This is not an issue during boot, since the system
firmware has done this already. Also, the hot-added switch
may contain IO-APICs (as it does in your case). These need
to be added too. Current hotplug drivers don't have the
code to support this.
> >
> > shpchp is used to hot-plug PCI or PCI-X devices. Both
> > drivers can run on the same system.
>
> As I mentioned above, we are planning to use two switches.
> Do I still need the shpchp in this case?
>
Your switch contains an SHPC controller, so you do need this
driver to hot-plug PCI/PCI-X devices under the switch. However,
your motherboard doesn't appear to support this.
> The PCIe hotplug protocol specifies that, when a device is
> in D1, D2 or D3hot state, the hot-plug controller generates
> a wakeup event (PM_PME) instead of an interrupt. Please see
Hmm... I thought you said the motherboard does not support
hotplug of IO devices (no attention button/indicator...).
What are you trying to do here? Put the switch and all
devices under it to a low-power state and then initiate
a hot-plug operation on an IO device?
> Is there a way to figure out whether the motherboard
> supports ACPI tables for PCIe hotplug in the above
> fashion? Is it possible to see it from the ACPI
> tables that I can get using 'acpidump' command? If it is
> not supported, is there a way I can add them?
Yes, you can dump your ACPI tables, but changing them
to support ACPI based hot-plug is non-trivial. Plus,
just updating your DSDT may not be enough, hardware
support may be needed.
Rajesh
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_idU88&alloc_id\x12065&op=click
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
^ permalink raw reply [flat|nested] 8+ messages in thread* RE: pci express switch driver
2004-10-10 2:15 pci express switch driver Sadik, Pallathu
` (4 preceding siblings ...)
2004-11-04 22:22 ` Rajesh Shah
@ 2004-11-04 23:40 ` Sadik, Pallathu
2004-11-05 21:57 ` Rajesh Shah
6 siblings, 0 replies; 8+ messages in thread
From: Sadik, Pallathu @ 2004-11-04 23:40 UTC (permalink / raw)
To: linux-hotplug
>
> Hmm... I thought you said the motherboard does not support
> hotplug of IO devices (no attention button/indicator...).
> What are you trying to do here? Put the switch and all
> devices under it to a low-power state and then initiate
> a hot-plug operation on an IO device?
My motherboard don't support hotplug. Hence, I am trying
to demonstrate the hot-plugging using two of our switches,
one connected to the downstream of the other one. Since
the downstream ports of the switches are capable of
hot-plug, I was hoping that I could demonstrate hot-plug
of the second switch.
Let us say, I have put the second switch and the endpoints
associated with it to low power mode (D3hot). Then I press
the Attention button on a downstream port of the second
switch. This should generate PM_PME event. This event
should be handled by the ACPI, right? Hence the
question.
Sadik.
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_idU88&alloc_id\x12065&op=click
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: pci express switch driver
2004-10-10 2:15 pci express switch driver Sadik, Pallathu
` (5 preceding siblings ...)
2004-11-04 23:40 ` Sadik, Pallathu
@ 2004-11-05 21:57 ` Rajesh Shah
6 siblings, 0 replies; 8+ messages in thread
From: Rajesh Shah @ 2004-11-05 21:57 UTC (permalink / raw)
To: linux-hotplug
On Thu, Nov 04, 2004 at 03:40:54PM -0800, Sadik, Pallathu wrote:
>
> Let us say, I have put the second switch and the endpoints
> associated with it to low power mode (D3hot). Then I press
> the Attention button on a downstream port of the second
> switch. This should generate PM_PME event. This event
> should be handled by the ACPI, right? Hence the
> question.
>
I _think_ I understand your question better now - you are trying
to see if acpi based hotplug will work if you force an acpi
interrupt by putting the switch and devices in the low power
state before you hotplug, correct?
In the case you mentioned, just because an ACPI SCI is asserted
does not mean you will be able to use acpiphp based hotplug.
For ACPI based hotplug to work, your firmware has to create a
namespace object for the p2p bridge that you are trying to
hotadd so that acpiphp can install a notify handler on it. When
you hotplug the switch and cause the SCI, the corresponding Lxx
or Exx method for the GPE must issue an ACPI Notify on the
switch object. Then acpiphp would get a chance to run and add
the switch.
As I said before, acpiphp, shpc and pcie drivers do not
currently have all the code to deal with switch hotplug so even
if you hack up the ACPI namespace to do this, the current
hotplug code will not be able to deal with it anyway.
Rajesh
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_idU88&alloc_id\x12065&op=click
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
^ permalink raw reply [flat|nested] 8+ messages in thread