Calling PCI Autoconfig again

Calling PCI Autoconfig again

[Wyse, Chris]

[-- Attachment #1: Type: text/plain, Size: 657 bytes --]

Hi,
 
I'm working on a board (PPC440GX CPU) that has an FPGA on it.  The FPGA
software has a PCI interface.  If I load the FPGA, then boot the Linux
kernel, the FPGA is recognized as a device on the PCI bus.  However, I'd
like to load the FPGA after the kernel has booted, then run autoconfig
again to detect the device.  What's the best way to do this?  
 
Please send responses directly to me (in addition to the list), since I
am not subscribed to the list.

Chris Wyse
Member of Technical Staff
Embedded Technologies
860-749-1556 office
860-978-0849 cell
413-778-9101 fax
http://www.windriver.com <http://www.windriver.com/> 
  

 

[-- Attachment #2: Type: text/html, Size: 1677 bytes --]

Re: Calling PCI Autoconfig again

[David Hawkins]

Wyse, Chris wrote:
> Hi,
>  
> I'm working on a board (PPC440GX CPU) that has an FPGA on it.  The FPGA 
> software has a PCI interface.  If I load the FPGA, then boot the Linux 
> kernel, the FPGA is recognized as a device on the PCI bus.  However, I'd 
> like to load the FPGA after the kernel has booted, then run autoconfig 
> again to detect the device.  What's the best way to do this? 
>  
> Please send responses directly to me (in addition to the list), since I 
> am not subscribed to the list.

Hi Chris,

I haven't had to do this, but I've thought about how I might do it.

In compact PCI, you can hot-swap a PCI device, and the Linux host
is supposed to go and re-enumerate the PCI bus. There is a
pci (host-side) hotplug skeleton in the source;

linux-2.6/drivers/pci/hotplug/pcihp_skeleton.c

I believe you can do the following;

   - consider the 440GX is a hotplug controller
     (and write a hotplug driver appropriately)
   - when Linux loads the FPGA, the FPGA generates a hotplug
     event, i.e., it just got plugged in
     (I'm not sure what signal is generated, perhaps a
      power-management interrupt PME#?)
   - the 440GX hotplug controller re-enumerates the PCI bus
     and finds the device

I'm sure you can get an idea from the hotplug skeleton what
re-enumeration entails.

Cheers
Dave

Re: Calling PCI Autoconfig again

[Michael Richardson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


You need some kind of hotplug PCI interface to make this work.

When I last tried this -- for the same reason --- it was hard because we
couldn't get the physical memory map from the (PC) BIOS.  (This was
before ACPI 2.0...)

Since you have complete control over the board, you should have this
information already. 

- -- 
]       ON HUMILITY: to err is human. To moo, bovine.           |  firewalls  [
]   Michael Richardson,    Xelerance Corporation, Ottawa, ON    |net architect[
] mcr@xelerance.com      http://www.sandelman.ottawa.on.ca/mcr/ |device driver[
] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Finger me for keys

iQEVAwUBRA2rNICLcPvd0N1lAQLchQf9E6B6ISMnpam5rgCXe67RQz3IWseOMtAb
8m65WKBDwTUxwKIwiOygY0AnLudL4+dSKp8QaswXsjl4g7BvjVBwjWH0itmIvWew
1V0BUVJUNMpzYH7I9fEtbJXjQvN/8VyJRX+7HIndvH1CWpeDzHgQTAVu+CIVfyGI
SEEZsgwzgq8mzbopawxPORd+rKv9F6tlnbqAU8P1eHB8wcUEu+WZQ+oA3WGS+Zhn
5ZjK3dX+QXI2JykicyS7VVUT5D6vaZFon4Ikk4Ao0JKfNHmAD7fN4BHhORrpGYyR
+dHlRZlZiRfw3AEoqg6XZZrcGWfciBK8mD6utPnSHomMsaTk80jbtQ==
=o2Wv
-----END PGP SIGNATURE-----

RE: Calling PCI Autoconfig again

[Wyse, Chris]

Hi Dave,

Thanks for the input.

I was thinking about making the FPGA a hotplug device.  I was really
hoping that I could just enable CONFIG_HOTPLUG in the kernel build, but
after some research, I don't believe that will work.  The hotplug
approach was more work than I wanted, but after your response, I decided
to look into it some more, specifically looking for an FPGA
implementation similar to ours.  I searched the linux-hotplug-devel list
for FPGA, and found this:

http://marc.theaimsgroup.com/?l=3Dlinux-hotplug-devel&m=3D101318310111131=
&w=3D
2

It implements a rescan of the PCI bus for a newly loaded FPGA device
(although despite the group name, it doesn't use HotPlug).

I'm going to try to use the code from the above link rather than
implement a hotplug driver. =20

Thanks again for the help.


Chris Wyse
Member of Technical Staff
Embedded Technologies
860-749-1556 office
860-978-0849 cell
413-778-9101 fax
http://www.windriver.com
 =20


-----Original Message-----
From: David Hawkins [mailto:dwh@ovro.caltech.edu]=20
Sent: Monday, March 06, 2006 11:33 PM
To: Wyse, Chris
Cc: linuxppc-embedded@ozlabs.org
Subject: Re: Calling PCI Autoconfig again

Wyse, Chris wrote:
> Hi,
> =20
> I'm working on a board (PPC440GX CPU) that has an FPGA on it.  The=20
> FPGA software has a PCI interface.  If I load the FPGA, then boot the=20
> Linux kernel, the FPGA is recognized as a device on the PCI bus. =20
> However, I'd like to load the FPGA after the kernel has booted, then=20
> run autoconfig again to detect the device.  What's the best way to do
this?
> =20
> Please send responses directly to me (in addition to the list), since=20
> I am not subscribed to the list.

Hi Chris,

I haven't had to do this, but I've thought about how I might do it.

In compact PCI, you can hot-swap a PCI device, and the Linux host is
supposed to go and re-enumerate the PCI bus. There is a pci (host-side)
hotplug skeleton in the source;

linux-2.6/drivers/pci/hotplug/pcihp_skeleton.c

I believe you can do the following;

   - consider the 440GX is a hotplug controller
     (and write a hotplug driver appropriately)
   - when Linux loads the FPGA, the FPGA generates a hotplug
     event, i.e., it just got plugged in
     (I'm not sure what signal is generated, perhaps a
      power-management interrupt PME#?)
   - the 440GX hotplug controller re-enumerates the PCI bus
     and finds the device

I'm sure you can get an idea from the hotplug skeleton what
re-enumeration entails.

Cheers
Dave

RE: Calling PCI Autoconfig again

[Wyse, Chris]

Hi,

This code will do the rescan of the PCI that we need for the FPGA.
However, it currently only supports a single execution after the FPGA
gets loaded.  Additional calls will continue to add FPGA devices to the
PCI device list data structure.  I'm going to make some changes to it to
handle multiple calls and to support multiple device ids.  I'm expecting
it to take a day or two, but I'll make an initial cut at it today that
will let us load the Spartan FPGA (only once).  This driver will be
loaded when the kernel boots, so I believe that we'll just need to make
a call into it (we won't have to dynamically load it like the phob
driver). =20

Michael,

For the FPGA loader, you should load the FPGA, then make a call to
pci_rescan_bus0() (I'm renaming check_hw() to pci_rescan_bus0()).  Next,
you need to do a pci_find_device() to detect the newly loaded FPGA.  If
the pci_find_device() fails, assume that the FPGA load has not
completed, so delay for some time frame (100 ms??), then rescan again.
If still not found, rescan up to MAX_PCI_RESCANS (probably set it to 3)
times, then return an error that the FPGA load failed.=20


Chris Wyse
Member of Technical Staff
Embedded Technologies
860-749-1556 office
860-978-0849 cell
413-778-9101 fax
http://www.windriver.com
 =20


-----Original Message-----
From: Wyse, Chris=20
Sent: Tuesday, March 07, 2006 8:49 AM
To: 'David Hawkins'
Cc: 'linuxppc-embedded@ozlabs.org'; Touron, Emmanuel; McComb, Michael
Subject: RE: Calling PCI Autoconfig again

Hi Dave,

Thanks for the input.

I was thinking about making the FPGA a hotplug device.  I was really
hoping that I could just enable CONFIG_HOTPLUG in the kernel build, but
after some research, I don't believe that will work.  The hotplug
approach was more work than I wanted, but after your response, I decided
to look into it some more, specifically looking for an FPGA
implementation similar to ours.  I searched the linux-hotplug-devel list
for FPGA, and found this:

http://marc.theaimsgroup.com/?l=3Dlinux-hotplug-devel&m=3D101318310111131=
&w=3D
2

It implements a rescan of the PCI bus for a newly loaded FPGA device
(although despite the group name, it doesn't use HotPlug).

I'm going to try to use the code from the above link rather than
implement a hotplug driver. =20

Thanks again for the help.


Chris Wyse
Member of Technical Staff
Embedded Technologies
860-749-1556 office
860-978-0849 cell
413-778-9101 fax
http://www.windriver.com
 =20


-----Original Message-----
From: David Hawkins [mailto:dwh@ovro.caltech.edu]
Sent: Monday, March 06, 2006 11:33 PM
To: Wyse, Chris
Cc: linuxppc-embedded@ozlabs.org
Subject: Re: Calling PCI Autoconfig again

Wyse, Chris wrote:
> Hi,
> =20
> I'm working on a board (PPC440GX CPU) that has an FPGA on it.  The=20
> FPGA software has a PCI interface.  If I load the FPGA, then boot the=20
> Linux kernel, the FPGA is recognized as a device on the PCI bus.
> However, I'd like to load the FPGA after the kernel has booted, then=20
> run autoconfig again to detect the device.  What's the best way to do
this?
> =20
> Please send responses directly to me (in addition to the list), since=20
> I am not subscribed to the list.

Hi Chris,

I haven't had to do this, but I've thought about how I might do it.

In compact PCI, you can hot-swap a PCI device, and the Linux host is
supposed to go and re-enumerate the PCI bus. There is a pci (host-side)
hotplug skeleton in the source;

linux-2.6/drivers/pci/hotplug/pcihp_skeleton.c

I believe you can do the following;

   - consider the 440GX is a hotplug controller
     (and write a hotplug driver appropriately)
   - when Linux loads the FPGA, the FPGA generates a hotplug
     event, i.e., it just got plugged in
     (I'm not sure what signal is generated, perhaps a
      power-management interrupt PME#?)
   - the 440GX hotplug controller re-enumerates the PCI bus
     and finds the device

I'm sure you can get an idea from the hotplug skeleton what
re-enumeration entails.

Cheers
Dave

Re: Calling PCI Autoconfig again

[Kumar Gala]

On Mar 7, 2006, at 8:17 AM, Wyse, Chris wrote:

> Hi,
>
> This code will do the rescan of the PCI that we need for the FPGA.
> However, it currently only supports a single execution after the FPGA
> gets loaded.  Additional calls will continue to add FPGA devices to  
> the
> PCI device list data structure.  I'm going to make some changes to  
> it to
> handle multiple calls and to support multiple device ids.  I'm  
> expecting
> it to take a day or two, but I'll make an initial cut at it today that
> will let us load the Spartan FPGA (only once).  This driver will be
> loaded when the kernel boots, so I believe that we'll just need to  
> make
> a call into it (we won't have to dynamically load it like the phob
> driver).

I'm dealing with this exact same situation and having a discussion on  
lkml regarding the proper way to do this.  In your case, do you care  
what addresses the FPGA is assigned at?

http://marc.theaimsgroup.com/?l=linux-kernel&m=114140791428032&w=2

> Michael,
>
> For the FPGA loader, you should load the FPGA, then make a call to
> pci_rescan_bus0() (I'm renaming check_hw() to pci_rescan_bus0()).   
> Next,
> you need to do a pci_find_device() to detect the newly loaded  
> FPGA.  If
> the pci_find_device() fails, assume that the FPGA load has not
> completed, so delay for some time frame (100 ms??), then rescan again.
> If still not found, rescan up to MAX_PCI_RESCANS (probably set it  
> to 3)
> times, then return an error that the FPGA load failed.

Do you not know when the FPGA load is done?  I assume its under  
software control so the "fpga loader" should just call the PCI setup  
code once it has completed successfully.

- kumar

Re: Calling PCI Autoconfig again

[Mark Chambers]

> I'm dealing with this exact same situation and having a discussion on  
> lkml regarding the proper way to do this.  In your case, do you care  
> what addresses the FPGA is assigned at?
> 
> http://marc.theaimsgroup.com/?l=linux-kernel&m=114140791428032&w=2
> 

Why does the PCI device have to have a fixed address?   And how do
you load an FPGA from user space - mmap?

Mark C.

Re: Calling PCI Autoconfig again

[Kumar Gala]

On Mar 7, 2006, at 10:53 AM, Mark Chambers wrote:

>> I'm dealing with this exact same situation and having a discussion  
>> on  lkml regarding the proper way to do this.  In your case, do  
>> you care  what addresses the FPGA is assigned at?
>> http://marc.theaimsgroup.com/?l=linux-kernel&m=114140791428032&w=2
>
> Why does the PCI device have to have a fixed address?   And how do
> you load an FPGA from user space - mmap?

In my case it does because we have other devices (DSPs) that are hard  
coded to know where in PCI address space the FPGA is.

As for how does one load it.  We have GPIO pins connected to the FPGA  
and load it from a file in user space with a kernel driver  
implementing the bit "protocol" over GPIO.

- kumar

Re: Calling PCI Autoconfig again

[Mark Chambers]

> 
> On Mar 7, 2006, at 10:53 AM, Mark Chambers wrote:
> 
>>> I'm dealing with this exact same situation and having a discussion  
>>> on  lkml regarding the proper way to do this.  In your case, do  
>>> you care  what addresses the FPGA is assigned at?
>>> http://marc.theaimsgroup.com/?l=linux-kernel&m=114140791428032&w=2
>>
>> Why does the PCI device have to have a fixed address?   And how do
>> you load an FPGA from user space - mmap?
> 
> In my case it does because we have other devices (DSPs) that are hard  
> coded to know where in PCI address space the FPGA is.
> 

Ah, ok.  So I'm guessing we're talking about DSP bus-master operations
where the DSP is putting out physical PCI space addresses that the
MMU can't help map.  I've run into a similar problem where I'd like to
be able to have a DSP do CPM/BD-like operations from SDRAM, but
it's exceedingly difficult to get the physical address of user-space data.

Well, FWIW, seems to me that something like what Greg K-H referred to,
where you pre-reserve space in PCI land, and then put the device there
when it becomes available, is the way to go.  Otherwise you have this
un-managed hole in PCI space allocation that might cause other problems 
with true hot-swap PCI, or break with kernel revisions.

Mark

RE: Calling PCI Autoconfig again

[Wyse, Chris]

Hi Kumar,

Thanks for the response.  Also, I apologize to the group - I didn't mean
to CC: the list with the last note.  However, I am glad that it got a
response from you.

Regarding the FPGA address assignment - it doesn't matter at all to me,
I just need to be able to access it on the PCI once the load is
complete.

As for the FPGA loader, the intent is that it will make the call to
rescan the PCI after the load is complete.  In fact, I intend to use it
as a way of determining if the load completed successfully, since we
don't have access to the DONE signal from the FPGA (we ran out of GPIO
pins). =20

If you missed this link from a previous post, please check it out.  It
implements a rescan of the PCI.

http://marc.theaimsgroup.com/?l=3Dlinux-hotplug-devel&m=3D101318310111131=
&w=3D
2=20

I see that you have a requirement for a fixed BAR address.  I think you
could modify the code from the above link to load a new BAR address
after the PCI has been rescanned.  You could probably add a routine to
register a BAR address for a device, and change the check_hw() routine
to see if a BAR address is registered.  If so, it would free the
allocated space for the device, allocate space at the desired address,
then write the new address to the BAR.

Hope this helps.  I'll be monitoring your thread for future
developments.


Chris Wyse
Member of Technical Staff
Embedded Technologies
860-749-1556 office
860-978-0849 cell
413-778-9101 fax
http://www.windriver.com
 =20


-----Original Message-----
From: Kumar Gala [mailto:galak@kernel.crashing.org]=20
Sent: Tuesday, March 07, 2006 11:27 AM
To: Wyse, Chris
Cc: Touron, Emmanuel; McComb, Michael; linuxppc-embedded@ozlabs.org
Subject: Re: Calling PCI Autoconfig again


On Mar 7, 2006, at 8:17 AM, Wyse, Chris wrote:

> Hi,
>
> This code will do the rescan of the PCI that we need for the FPGA.
> However, it currently only supports a single execution after the FPGA=20
> gets loaded.  Additional calls will continue to add FPGA devices to=20
> the PCI device list data structure.  I'm going to make some changes to

> it to handle multiple calls and to support multiple device ids.  I'm=20
> expecting it to take a day or two, but I'll make an initial cut at it=20
> today that will let us load the Spartan FPGA (only once).  This driver

> will be loaded when the kernel boots, so I believe that we'll just=20
> need to make a call into it (we won't have to dynamically load it like

> the phob driver).

I'm dealing with this exact same situation and having a discussion on
lkml regarding the proper way to do this.  In your case, do you care
what addresses the FPGA is assigned at?

http://marc.theaimsgroup.com/?l=3Dlinux-kernel&m=3D114140791428032&w=3D2

> Michael,
>
> For the FPGA loader, you should load the FPGA, then make a call to
> pci_rescan_bus0() (I'm renaming check_hw() to pci_rescan_bus0()).  =20
> Next,
> you need to do a pci_find_device() to detect the newly loaded FPGA. =20
> If the pci_find_device() fails, assume that the FPGA load has not=20
> completed, so delay for some time frame (100 ms??), then rescan again.
> If still not found, rescan up to MAX_PCI_RESCANS (probably set it to=20
> 3) times, then return an error that the FPGA load failed.

Do you not know when the FPGA load is done?  I assume its under software
control so the "fpga loader" should just call the PCI setup code once it
has completed successfully.

- kumar

Re: Calling PCI Autoconfig again

[Kumar Gala]

On Mar 7, 2006, at 1:25 PM, Wyse, Chris wrote:

> Hi Kumar,
>
> Thanks for the response.  Also, I apologize to the group - I didn't  
> mean
> to CC: the list with the last note.  However, I am glad that it got a
> response from you.
>
> Regarding the FPGA address assignment - it doesn't matter at all to  
> me,
> I just need to be able to access it on the PCI once the load is
> complete.
>
> As for the FPGA loader, the intent is that it will make the call to
> rescan the PCI after the load is complete.  In fact, I intend to  
> use it
> as a way of determining if the load completed successfully, since we
> don't have access to the DONE signal from the FPGA (we ran out of GPIO
> pins).
>
> If you missed this link from a previous post, please check it out.  It
> implements a rescan of the PCI.
>
> http://marc.theaimsgroup.com/?l=linux-hotplug- 
> devel&m=101318310111131&w=
> 2

I saw this link and felt that the code was doing alot of work that  
was already handled by the PCI subsystem for you.  For example, there  
isn't a need for you to kmalloc your own dev and go through all the  
effort.

For your case it should be as simple as the following:

bus = pci_find_bus(0, bus_of_dev);
dev = pci_scan_single_device(bus, devfn);
pci_bus_alloc_resource(...);
pci_bus_add_devices(bus);

You could also look at using the pci_scan_slot() / pci_find_slot()  
instead of pci_scan_single_device().

- kumar