proper way to assign fixed PCI resources to a "hotplug" device

proper way to assign fixed PCI resources to a "hotplug" device

[Kumar Gala]

I was wondering what the proper way to assign and setup a single PCI  
device that comes into existence after the system has booted.  I have  
an FPGA that we load from user space at which time it shows up on the  
PCI bus.

It has a single BAR and I need to assign it at a fixed address in PCI  
MMIO space.

All of the exported interfaces I see have to do with having the  
kernel assign the BAR automatically for me.

the following looks like what I want to do:

bus = pci_find_bus(0, 3);
dev = pci_scan_single_device(bus, devfn);
pci_bus_alloc_resource(...);
pci_update_resource(dev, dev->resource[0], 0);
pci_bus_add_devices(bus);

However, pci_update_resource() is not an exported symbol, so I could  
replace that code with the need updates to the actual BAR.

Is this the "right" way to go about this or is there a better  
mechanism to do this.

thanks

- kumar

Re: proper way to assign fixed PCI resources to a "hotplug" device

[Greg KH]

On Fri, Mar 03, 2006 at 11:42:03AM -0600, Kumar Gala wrote:
> I was wondering what the proper way to assign and setup a single PCI  
> device that comes into existence after the system has booted.  I have  
> an FPGA that we load from user space at which time it shows up on the  
> PCI bus.

Idealy your BIOS would set up this information :)

> It has a single BAR and I need to assign it at a fixed address in PCI  
> MMIO space.
> 
> All of the exported interfaces I see have to do with having the  
> kernel assign the BAR automatically for me.
> 
> the following looks like what I want to do:
> 
> bus = pci_find_bus(0, 3);
> dev = pci_scan_single_device(bus, devfn);
> pci_bus_alloc_resource(...);
> pci_update_resource(dev, dev->resource[0], 0);
> pci_bus_add_devices(bus);
> 
> However, pci_update_resource() is not an exported symbol, so I could  
> replace that code with the need updates to the actual BAR.
> 
> Is this the "right" way to go about this or is there a better  
> mechanism to do this.

Take a look at how the compat pci hotplug driver does this, you probably
just need to do the same as it.

thanks,

greg k-h

Re: proper way to assign fixed PCI resources to a "hotplug" device

[Kumar Gala]

On Fri, 3 Mar 2006, Greg KH wrote:

> On Fri, Mar 03, 2006 at 11:42:03AM -0600, Kumar Gala wrote:
> > I was wondering what the proper way to assign and setup a single PCI  
> > device that comes into existence after the system has booted.  I have  
> > an FPGA that we load from user space at which time it shows up on the  
> > PCI bus.
> 
> Idealy your BIOS would set up this information :)

How would my BIOS know about a device that didn't exist when it booted.  
Or do you mean my BIOS would load the FPGA as well so it existed.

> > It has a single BAR and I need to assign it at a fixed address in PCI  
> > MMIO space.
> > 
> > All of the exported interfaces I see have to do with having the  
> > kernel assign the BAR automatically for me.
> > 
> > the following looks like what I want to do:
> > 
> > bus = pci_find_bus(0, 3);
> > dev = pci_scan_single_device(bus, devfn);
> > pci_bus_alloc_resource(...);
> > pci_update_resource(dev, dev->resource[0], 0);
> > pci_bus_add_devices(bus);
> > 
> > However, pci_update_resource() is not an exported symbol, so I could  
> > replace that code with the need updates to the actual BAR.
> > 
> > Is this the "right" way to go about this or is there a better  
> > mechanism to do this.
> 
> Take a look at how the compat pci hotplug driver does this, you probably
> just need to do the same as it.

I'll take a look.  How about something like the following patch:

diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c
index ea9277b..8d5caec 100644
--- a/drivers/pci/setup-res.c
+++ b/drivers/pci/setup-res.c
@@ -155,6 +155,42 @@ int pci_assign_resource(struct pci_dev *
 	return ret;
 }
 
+int pci_assign_resource_fixed(struct pci_dev *dev, int resno)
+{
+	struct pci_bus *bus = dev->bus;
+	struct resource *res = dev->resource + resno;
+	unsigned int type_mask;
+	int i, ret = -EBUSY;
+
+	type_mask = IORESOURCE_IO | IORESOURCE_MEM | IORESOURCE_PREFETCH;
+
+	for (i = 0; i < PCI_BUS_NUM_RESOURCES; i++) {
+		struct resource *r = bus->resource[i];
+		if (!r)
+			continue;
+
+		/* type_mask must match */
+		if ((res->flags ^ r->flags) & type_mask)
+			continue;
+
+		ret = request_resource(r, res);
+
+		if (ret == 0)
+			break;
+	}
+
+	if (ret) {
+		printk(KERN_ERR "PCI: Failed to allocate %s resource #%d:%lx@%lx for %s\n",
+		       res->flags & IORESOURCE_IO ? "I/O" : "mem",
+		       resno, res->end - res->start + 1, res->start, pci_name(dev));
+	} else if (resno < PCI_BRIDGE_RESOURCES) {
+		pci_update_resource(dev, res, resno);
+	}
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(pci_assign_resource_fixed);
+
 /* Sort resources by alignment */
 void __devinit
 pdev_sort_resources(struct pci_dev *dev, struct resource_list *head)

Re: proper way to assign fixed PCI resources to a "hotplug" device

[Greg KH]

On Fri, Mar 03, 2006 at 04:39:52PM -0600, Kumar Gala wrote:
> On Fri, 3 Mar 2006, Greg KH wrote:
> 
> > On Fri, Mar 03, 2006 at 11:42:03AM -0600, Kumar Gala wrote:
> > > I was wondering what the proper way to assign and setup a single PCI  
> > > device that comes into existence after the system has booted.  I have  
> > > an FPGA that we load from user space at which time it shows up on the  
> > > PCI bus.
> > 
> > Idealy your BIOS would set up this information :)
> 
> How would my BIOS know about a device that didn't exist when it booted.  

According to the PCI Hotplug spec, your BIOS needs to take that into
consideration at boot time.  Yeah, it's a wierd thing, I agree, but is
how this works for x86 systems.  The space and resources are reserved
at boot time by the pci hotplug controller in anticipation of a device
being added sometime in the future.

Other arches do this differently (ppc64 has the stuff reserverd by the
hypervisor), and then compat pci does it by just plain guessing.  It
sounds like your situation is just like this one.

> Or do you mean my BIOS would load the FPGA as well so it existed.

No, see above.

> > > It has a single BAR and I need to assign it at a fixed address in PCI  
> > > MMIO space.
> > > 
> > > All of the exported interfaces I see have to do with having the  
> > > kernel assign the BAR automatically for me.
> > > 
> > > the following looks like what I want to do:
> > > 
> > > bus = pci_find_bus(0, 3);
> > > dev = pci_scan_single_device(bus, devfn);
> > > pci_bus_alloc_resource(...);
> > > pci_update_resource(dev, dev->resource[0], 0);
> > > pci_bus_add_devices(bus);
> > > 
> > > However, pci_update_resource() is not an exported symbol, so I could  
> > > replace that code with the need updates to the actual BAR.
> > > 
> > > Is this the "right" way to go about this or is there a better  
> > > mechanism to do this.
> > 
> > Take a look at how the compat pci hotplug driver does this, you probably
> > just need to do the same as it.
> 
> I'll take a look.  How about something like the following patch:

Hm, I don't think this is needed, see how the cpcihp drivers do it in
drivers/pci/hotplug/cpcihp*.c.  I think you can do things the same way.

But if not, I don't have any objection to adding this patch, you just
need to prove you really need it :)

thanks,

greg k-h

Re: proper way to assign fixed PCI resources to a "hotplug" device

[Kumar Gala]

On Mar 3, 2006, at 5:18 PM, Greg KH wrote:

> On Fri, Mar 03, 2006 at 04:39:52PM -0600, Kumar Gala wrote:
>> On Fri, 3 Mar 2006, Greg KH wrote:
>>
>>> On Fri, Mar 03, 2006 at 11:42:03AM -0600, Kumar Gala wrote:
>>>> I was wondering what the proper way to assign and setup a single  
>>>> PCI
>>>> device that comes into existence after the system has booted.  I  
>>>> have
>>>> an FPGA that we load from user space at which time it shows up  
>>>> on the
>>>> PCI bus.
>>>
>>> Idealy your BIOS would set up this information :)
>>
>> How would my BIOS know about a device that didn't exist when it  
>> booted.
>
> According to the PCI Hotplug spec, your BIOS needs to take that into
> consideration at boot time.  Yeah, it's a wierd thing, I agree, but is
> how this works for x86 systems.  The space and resources are reserved
> at boot time by the pci hotplug controller in anticipation of a device
> being added sometime in the future.
>
> Other arches do this differently (ppc64 has the stuff reserverd by the
> hypervisor), and then compat pci does it by just plain guessing.  It
> sounds like your situation is just like this one.

Well I reserve space for the device in my "BIOS".  However this is an  
embedded system so there isn't any calling out to the "BIOS" after  
linux has booted.  Is there some additional "work" that the x86  
systems do beyond ensure proper holes in the memory map exist for  
future devices to be placed into?

>> Or do you mean my BIOS would load the FPGA as well so it existed.
>
> No, see above.
>
>>>> It has a single BAR and I need to assign it at a fixed address  
>>>> in PCI
>>>> MMIO space.
>>>>
>>>> All of the exported interfaces I see have to do with having the
>>>> kernel assign the BAR automatically for me.
>>>>
>>>> the following looks like what I want to do:
>>>>
>>>> bus = pci_find_bus(0, 3);
>>>> dev = pci_scan_single_device(bus, devfn);
>>>> pci_bus_alloc_resource(...);
>>>> pci_update_resource(dev, dev->resource[0], 0);
>>>> pci_bus_add_devices(bus);
>>>>
>>>> However, pci_update_resource() is not an exported symbol, so I  
>>>> could
>>>> replace that code with the need updates to the actual BAR.
>>>>
>>>> Is this the "right" way to go about this or is there a better
>>>> mechanism to do this.
>>>
>>> Take a look at how the compat pci hotplug driver does this, you  
>>> probably
>>> just need to do the same as it.
>>
>> I'll take a look.  How about something like the following patch:
>
> Hm, I don't think this is needed, see how the cpcihp drivers do it in
> drivers/pci/hotplug/cpcihp*.c.  I think you can do things the same  
> way.

Ahh, looked at cpqphp_* and only found cpqhp_configure_device() of  
any use.  I'll take a look at cpcihp*.c

> But if not, I don't have any objection to adding this patch, you just
> need to prove you really need it :)

No problem, I asked because I'd prefer to use existing code to solve  
my problem.  Just need to understand how to get there.



- kumar

Re: proper way to assign fixed PCI resources to a "hotplug" device

[Scott Murray]

On Fri, 3 Mar 2006, Kumar Gala wrote:

> 
> On Mar 3, 2006, at 5:18 PM, Greg KH wrote:
> 
> > On Fri, Mar 03, 2006 at 04:39:52PM -0600, Kumar Gala wrote:
> >> On Fri, 3 Mar 2006, Greg KH wrote:
> >>
> >>> On Fri, Mar 03, 2006 at 11:42:03AM -0600, Kumar Gala wrote:
> >>>> I was wondering what the proper way to assign and setup a single  
> >>>> PCI
> >>>> device that comes into existence after the system has booted.  I  
> >>>> have
> >>>> an FPGA that we load from user space at which time it shows up  
> >>>> on the
> >>>> PCI bus.
> >>>
> >>> Idealy your BIOS would set up this information :)
> >>
> >> How would my BIOS know about a device that didn't exist when it  
> >> booted.
> >
> > According to the PCI Hotplug spec, your BIOS needs to take that into
> > consideration at boot time.  Yeah, it's a wierd thing, I agree, but is
> > how this works for x86 systems.  The space and resources are reserved
> > at boot time by the pci hotplug controller in anticipation of a device
> > being added sometime in the future.
> >
> > Other arches do this differently (ppc64 has the stuff reserverd by the
> > hypervisor), and then compat pci does it by just plain guessing.  It
> > sounds like your situation is just like this one.
> 
> Well I reserve space for the device in my "BIOS".  However this is an  
> embedded system so there isn't any calling out to the "BIOS" after  
> linux has booted.  Is there some additional "work" that the x86  
> systems do beyond ensure proper holes in the memory map exist for  
> future devices to be placed into?

For CompactPCI, most boards I've encountered actually do not reserve any 
extra resources on the bridge to the hotplug bus.  I have had a patch for
a while that allows reallocating the resources for a specified bridge and 
all its children at boot time, and recently ported it to 2.6.  It doesn't
sound like you need it since you've got your own bootloader, but let me
know if you want to take a look at it.

> Ahh, looked at cpqphp_* and only found cpqhp_configure_device() of  
> any use.  I'll take a look at cpcihp*.c

Just a warning, cpci_configure_slot in cpci_hotplug_pci.c is somewhat
broken atm, I've got a small patch I was going to clean up and send to 
Greg for 2.6.17.  I can provide it if you want to compare a working 
cpci_configure_slot against cpqhp_configure_device.

Scott


-- 
==============================================================================
Scott Murray, scott@spiteful.org

     "Good, bad ... I'm the guy with the gun." - Ash, "Army of Darkness"

Re: proper way to assign fixed PCI resources to a "hotplug" device

[Kumar Gala]

On Mar 3, 2006, at 5:18 PM, Greg KH wrote:

> On Fri, Mar 03, 2006 at 04:39:52PM -0600, Kumar Gala wrote:
>> On Fri, 3 Mar 2006, Greg KH wrote:
>>
>>> On Fri, Mar 03, 2006 at 11:42:03AM -0600, Kumar Gala wrote:
>>>> I was wondering what the proper way to assign and setup a single  
>>>> PCI
>>>> device that comes into existence after the system has booted.  I  
>>>> have
>>>> an FPGA that we load from user space at which time it shows up  
>>>> on the
>>>> PCI bus.
>>>
>>> Idealy your BIOS would set up this information :)
>>
>> How would my BIOS know about a device that didn't exist when it  
>> booted.
>
> According to the PCI Hotplug spec, your BIOS needs to take that into
> consideration at boot time.  Yeah, it's a wierd thing, I agree, but is
> how this works for x86 systems.  The space and resources are reserved
> at boot time by the pci hotplug controller in anticipation of a device
> being added sometime in the future.
>
> Other arches do this differently (ppc64 has the stuff reserverd by the
> hypervisor), and then compat pci does it by just plain guessing.  It
> sounds like your situation is just like this one.
>
>> Or do you mean my BIOS would load the FPGA as well so it existed.
>
> No, see above.
>
>>>> It has a single BAR and I need to assign it at a fixed address  
>>>> in PCI
>>>> MMIO space.
>>>>
>>>> All of the exported interfaces I see have to do with having the
>>>> kernel assign the BAR automatically for me.
>>>>
>>>> the following looks like what I want to do:
>>>>
>>>> bus = pci_find_bus(0, 3);
>>>> dev = pci_scan_single_device(bus, devfn);
>>>> pci_bus_alloc_resource(...);
>>>> pci_update_resource(dev, dev->resource[0], 0);
>>>> pci_bus_add_devices(bus);
>>>>
>>>> However, pci_update_resource() is not an exported symbol, so I  
>>>> could
>>>> replace that code with the need updates to the actual BAR.
>>>>
>>>> Is this the "right" way to go about this or is there a better
>>>> mechanism to do this.
>>>
>>> Take a look at how the compat pci hotplug driver does this, you  
>>> probably
>>> just need to do the same as it.
>>
>> I'll take a look.  How about something like the following patch:
>
> Hm, I don't think this is needed, see how the cpcihp drivers do it in
> drivers/pci/hotplug/cpcihp*.c.  I think you can do things the same  
> way.
>
> But if not, I don't have any objection to adding this patch, you just
> need to prove you really need it :)

Haven't seen any responses on this thread or my query on the hotplug  
list on how to assign fixed BARs/resources to a PCI device.  As far  
as I can tell there isn't a way to do this in the kernel today.   
Obviously, I'm in a more embedded situation where I have a fixed PCI  
memory map and know where devices are expected to be.

Not sure what else you are looking for in the way of proving the need  
for the new interface :)

- kumar

Re: proper way to assign fixed PCI resources to a "hotplug" device

[Jeff Garzik]

I have a similar situation:

BIOS initializes PCI device to mode A, I need to switch it to mode B. 
To do this, I must assign a value to an MMIO PCI BAR that was not 
initialized at boot.

How to do this?

	Jeff

Re: proper way to assign fixed PCI resources to a "hotplug" device

[Greg KH]

On Fri, Mar 03, 2006 at 06:18:32PM -0500, Jeff Garzik wrote:
> I have a similar situation:
> 
> BIOS initializes PCI device to mode A, I need to switch it to mode B. 
> To do this, I must assign a value to an MMIO PCI BAR that was not 
> initialized at boot.
> 
> How to do this?

I really don't know, what kind of device wants to do this?

thanks,

greg k-h

Re: proper way to assign fixed PCI resources to a "hotplug" device

[Tejun Heo]

Greg KH wrote:
> On Fri, Mar 03, 2006 at 06:18:32PM -0500, Jeff Garzik wrote:
> 
>>I have a similar situation:
>>
>>BIOS initializes PCI device to mode A, I need to switch it to mode B. 
>>To do this, I must assign a value to an MMIO PCI BAR that was not 
>>initialized at boot.
>>
>>How to do this?
> 
> 
> I really don't know, what kind of device wants to do this?
> 

Jeff is probably talking about ABAR of ICH controllers. ABAR (AHCI BAR, 
memory mapped IO region covering all AHCI registers) isn't needed for 
IDE mode operation and the BAR register is disabled when the chip is in 
IDE mode. However, ABAR becomes necessary for 1. accessing SCR registers 
(for SATA phy monitor and control) or 2. switching on AHCI mode manually 
(some notebook BIOSes always initalize ICH6/7m's into IDE mode even when 
the controller does support AHCI mode.

So, the problem is that the chip actually disables the PCI BAR if 
certain switches aren't turned on and thus BIOSes are likely not to 
reserve mmio address for the BAR. We can turn on proper switches during 
driver initialization but we don't know how to wiggle the BAR into mmio 
address space.

-- 
tejun

Re: proper way to assign fixed PCI resources to a "hotplug" device

[Greg KH]

On Wed, Mar 08, 2006 at 11:31:31AM +0900, Tejun Heo wrote:
> Greg KH wrote:
> >On Fri, Mar 03, 2006 at 06:18:32PM -0500, Jeff Garzik wrote:
> >
> >>I have a similar situation:
> >>
> >>BIOS initializes PCI device to mode A, I need to switch it to mode B. 
> >>To do this, I must assign a value to an MMIO PCI BAR that was not 
> >>initialized at boot.
> >>
> >>How to do this?
> >
> >
> >I really don't know, what kind of device wants to do this?
> >
> 
> Jeff is probably talking about ABAR of ICH controllers. ABAR (AHCI BAR, 
> memory mapped IO region covering all AHCI registers) isn't needed for 
> IDE mode operation and the BAR register is disabled when the chip is in 
> IDE mode. However, ABAR becomes necessary for 1. accessing SCR registers 
> (for SATA phy monitor and control) or 2. switching on AHCI mode manually 
> (some notebook BIOSes always initalize ICH6/7m's into IDE mode even when 
> the controller does support AHCI mode.
> 
> So, the problem is that the chip actually disables the PCI BAR if 
> certain switches aren't turned on and thus BIOSes are likely not to 
> reserve mmio address for the BAR. We can turn on proper switches during 
> driver initialization but we don't know how to wiggle the BAR into mmio 
> address space.

Thanks for the explaination, that makes more sense.  Unfortunatly I do
not know how to do this right now :(

Anyone with any ideas?

thanks,

greg k-h

Re: proper way to assign fixed PCI resources to a "hotplug" device

[Ivan Kokshaysky]

On Tue, Mar 07, 2006 at 09:27:23PM -0800, Greg KH wrote:
> On Wed, Mar 08, 2006 at 11:31:31AM +0900, Tejun Heo wrote:
> > So, the problem is that the chip actually disables the PCI BAR if 
> > certain switches aren't turned on and thus BIOSes are likely not to 
> > reserve mmio address for the BAR. We can turn on proper switches during 
> > driver initialization but we don't know how to wiggle the BAR into mmio 
> > address space.
> 
> Thanks for the explaination, that makes more sense.  Unfortunatly I do
> not know how to do this right now :(
> 
> Anyone with any ideas?

We have 'pci_fixup_early' stuff exactly for that sort of hardware.
IOW, just add a quirk routine that turns on desired mode of the
device and use DECLARE_PCI_FIXUP_EARLY() for it.
The new BAR will be discovered an assigned automatically then.

Ivan.

Re: proper way to assign fixed PCI resources to a "hotplug" device

[thockin]

On Wed, Mar 08, 2006 at 02:39:52PM +0300, Ivan Kokshaysky wrote:
> On Tue, Mar 07, 2006 at 09:27:23PM -0800, Greg KH wrote:
> > On Wed, Mar 08, 2006 at 11:31:31AM +0900, Tejun Heo wrote:
> > > So, the problem is that the chip actually disables the PCI BAR if 
> > > certain switches aren't turned on and thus BIOSes are likely not to 
> > > reserve mmio address for the BAR. We can turn on proper switches during 
> > > driver initialization but we don't know how to wiggle the BAR into mmio 
> > > address space.
> > 
> > Thanks for the explaination, that makes more sense.  Unfortunatly I do
> > not know how to do this right now :(
> > 
> > Anyone with any ideas?
> 
> We have 'pci_fixup_early' stuff exactly for that sort of hardware.
> IOW, just add a quirk routine that turns on desired mode of the
> device and use DECLARE_PCI_FIXUP_EARLY() for it.
> The new BAR will be discovered an assigned automatically then.

Assigned from what pool?  BIOS most likely sizes the hole to be a pretty
tight fit for all the resources it knows about.  If there is suddenly a
new resource, you're in trouble.

The problem is that we don't know explicitly where or how big the hole is,
how it is allocated (prefetch or non).  Then, even if we *did* know, we
don't know that there's enough space for an arbitrary device to show up
later.

In BIOSes I control we enable the BARs and allocate regions for them and
then turn them off if need be.

There really isn't a good generic answer to this that will work with
existing BIOSes.  My first instinct is to call this a BIOS bug.

We could teach linux about chipsets and let Linux re-do the whole
PCI-allocation process.   But that's not an easy task, and is probably a
contentious idea.

Tim

Re: proper way to assign fixed PCI resources to a "hotplug" device

[Ivan Kokshaysky]

On Wed, Mar 08, 2006 at 08:40:42AM -0800, thockin@hockin.org wrote:
> Assigned from what pool?  BIOS most likely sizes the hole to be a pretty
> tight fit for all the resources it knows about.  If there is suddenly a
> new resource, you're in trouble.

This can only be true when device in question is behind a pci-to-pci
bridge (which is obviously not the case for ICH controllers you mentioned).
Otherwise we have plenty of MMIO space.

> We could teach linux about chipsets and let Linux re-do the whole
> PCI-allocation process.   But that's not an easy task, and is probably a
> contentious idea.

Linux knows how to do this for years. Actually, this is the way how alpha
and some other platforms work. Since 2.6.13, this pretty much applies to
x86 as well (we do respect BIOS PCI allocations, but we clean the things
up after BIOS quite aggressively - see pci_assign_unassigned_resources() call).

Ivan.

Re: proper way to assign fixed PCI resources to a "hotplug" device

[thockin]

On Thu, Mar 09, 2006 at 12:21:53AM +0300, Ivan Kokshaysky wrote:
> On Wed, Mar 08, 2006 at 08:40:42AM -0800, thockin@hockin.org wrote:
> > Assigned from what pool?  BIOS most likely sizes the hole to be a pretty
> > tight fit for all the resources it knows about.  If there is suddenly a
> > new resource, you're in trouble.
> 
> This can only be true when device in question is behind a pci-to-pci
> bridge (which is obviously not the case for ICH controllers you mentioned).
> Otherwise we have plenty of MMIO space.

Not true.  Plenty of root bridges have the same base/limit style
configuration registers, but they are non-standard.  Even worse - the MMIO
hole thatthe chipset carves out, is not guaranteed to be big enough for
some new random allocation.

> > We could teach linux about chipsets and let Linux re-do the whole
> > PCI-allocation process.   But that's not an easy task, and is probably a
> > contentious idea.
> 
> Linux knows how to do this for years. Actually, this is the way how alpha
> and some other platforms work. Since 2.6.13, this pretty much applies to
> x86 as well (we do respect BIOS PCI allocations, but we clean the things
> up after BIOS quite aggressively - see pci_assign_unassigned_resources() call).

Cleaning up and re-doing are not the same thing.  The plethora of x86
chipsets makes this unpleasant at best and more likely unworkable.

Re: proper way to assign fixed PCI resources to a "hotplug" device

[Ivan Kokshaysky]

On Wed, Mar 08, 2006 at 01:57:34PM -0800, thockin@hockin.org wrote:
> > Otherwise we have plenty of MMIO space.
> 
> Not true.  Plenty of root bridges have the same base/limit style
> configuration registers, but they are non-standard.  Even worse - the MMIO
> hole thatthe chipset carves out, is not guaranteed to be big enough for
> some new random allocation.

I'm intrigued. Care to give us an example of such system (where the
root bridge window is too small), please? lspci -vxxx?

> Cleaning up and re-doing are not the same thing.  The plethora of x86
> chipsets makes this unpleasant at best and more likely unworkable.

Yes, re-doing is a LOT simpler. ;-)
If needed, we could introduce 'pci=totallyingnorefsckingbiossettings'
boot option - it would be 10 or less lines of code.

Ivan.

Re: proper way to assign fixed PCI resources to a "hotplug" device

[thockin]

On Thu, Mar 09, 2006 at 01:11:47AM +0300, Ivan Kokshaysky wrote:
> On Wed, Mar 08, 2006 at 01:57:34PM -0800, thockin@hockin.org wrote:
> > > Otherwise we have plenty of MMIO space.
> > 
> > Not true.  Plenty of root bridges have the same base/limit style
> > configuration registers, but they are non-standard.  Even worse - the MMIO
> > hole thatthe chipset carves out, is not guaranteed to be big enough for
> > some new random allocation.
> 
> I'm intrigued. Care to give us an example of such system (where the
> root bridge window is too small), please? lspci -vxxx?

I have systems where the BIOS sets up the IO hole to be the size of the
BARs it finds at PCI enum time.  If I have a device that has a hidden BAR,
the hole won't cover it.  Period.

> > Cleaning up and re-doing are not the same thing.  The plethora of x86
> > chipsets makes this unpleasant at best and more likely unworkable.
> 
> Yes, re-doing is a LOT simpler. ;-)
> If needed, we could introduce 'pci=totallyingnorefsckingbiossettings'
> boot option - it would be 10 or less lines of code.

Except for that whole chipset code thing.  Have you dealt with the
fugliness that is the chipset?  You're really glossing over it.

Re: proper way to assign fixed PCI resources to a "hotplug" device

[Kumar Gala]

On Mar 3, 2006, at 4:07 PM, Greg KH wrote:

> On Fri, Mar 03, 2006 at 11:42:03AM -0600, Kumar Gala wrote:
>> I was wondering what the proper way to assign and setup a single PCI
>> device that comes into existence after the system has booted.  I have
>> an FPGA that we load from user space at which time it shows up on the
>> PCI bus.
>
> Idealy your BIOS would set up this information :)
>
>> It has a single BAR and I need to assign it at a fixed address in PCI
>> MMIO space.
>>
>> All of the exported interfaces I see have to do with having the
>> kernel assign the BAR automatically for me.
>>
>> the following looks like what I want to do:
>>
>> bus = pci_find_bus(0, 3);
>> dev = pci_scan_single_device(bus, devfn);
>> pci_bus_alloc_resource(...);
>> pci_update_resource(dev, dev->resource[0], 0);
>> pci_bus_add_devices(bus);
>>
>> However, pci_update_resource() is not an exported symbol, so I could
>> replace that code with the need updates to the actual BAR.
>>
>> Is this the "right" way to go about this or is there a better
>> mechanism to do this.
>
> Take a look at how the compat pci hotplug driver does this, you  
> probably
> just need to do the same as it.

I found cpqhp_configure_device(), but I dont see anything about how  
to handle assigned a fixed address to the BAR.

I see I could use pci_find_slot()/pci_scan_slot().  Not sure I see  
how that is much better than pci_find_bus()/pci_scan_single_device().

- kumar

Re: proper way to assign fixed PCI resources to a "hotplug" device

[Greg KH]

On Fri, Mar 03, 2006 at 05:13:55PM -0600, Kumar Gala wrote:
> 
> On Mar 3, 2006, at 4:07 PM, Greg KH wrote:
> 
> >On Fri, Mar 03, 2006 at 11:42:03AM -0600, Kumar Gala wrote:
> >>I was wondering what the proper way to assign and setup a single PCI
> >>device that comes into existence after the system has booted.  I have
> >>an FPGA that we load from user space at which time it shows up on the
> >>PCI bus.
> >
> >Idealy your BIOS would set up this information :)
> >
> >>It has a single BAR and I need to assign it at a fixed address in PCI
> >>MMIO space.
> >>
> >>All of the exported interfaces I see have to do with having the
> >>kernel assign the BAR automatically for me.
> >>
> >>the following looks like what I want to do:
> >>
> >>bus = pci_find_bus(0, 3);
> >>dev = pci_scan_single_device(bus, devfn);
> >>pci_bus_alloc_resource(...);
> >>pci_update_resource(dev, dev->resource[0], 0);
> >>pci_bus_add_devices(bus);
> >>
> >>However, pci_update_resource() is not an exported symbol, so I could
> >>replace that code with the need updates to the actual BAR.
> >>
> >>Is this the "right" way to go about this or is there a better
> >>mechanism to do this.
> >
> >Take a look at how the compat pci hotplug driver does this, you  
> >probably
> >just need to do the same as it.
> 
> I found cpqhp_configure_device(), but I dont see anything about how  
> to handle assigned a fixed address to the BAR.

I don't know either, try asking on the pci hotplug mailing list and CC:
Scott, the author of that driver for how his devices work around that.

thanks,

greg k-h

Re: proper way to assign fixed PCI resources to a "hotplug" device

[Scott Murray]

On Fri, 3 Mar 2006, Greg KH wrote:

> On Fri, Mar 03, 2006 at 05:13:55PM -0600, Kumar Gala wrote:
> > I found cpqhp_configure_device(), but I dont see anything about how  
> > to handle assigned a fixed address to the BAR.
> 
> I don't know either, try asking on the pci hotplug mailing list and CC:
> Scott, the author of that driver for how his devices work around that.

Heh, I'm the Compact PCI hotplug guy, not the Compaq PCI hotplug guy. :)

Scott


-- 
==============================================================================
Scott Murray, scott@spiteful.org

     "Good, bad ... I'm the guy with the gun." - Ash, "Army of Darkness"