[PATCH] pci: only release that resource index is less than 3

[PATCH] pci: only release that resource index is less than 3

[Yinghai Lu]

after

| commit 308cf8e13f42f476dfd6552aeff58fdc0788e566
|
|    PCI: get larger bridge ranges when space is available

found one of resource of peer root bus (0x00) get released from root
resource. later one hotplug device can not get big range anymore.
other peer root buses is ok.

it turns out it is from transparent path.

those resources will be used for pci bridge BAR updated.
so need to limit it to 3.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>

---
 drivers/pci/setup-bus.c |    9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

Index: linux-2.6/drivers/pci/setup-bus.c
===================================================================
--- linux-2.6.orig/drivers/pci/setup-bus.c
+++ linux-2.6/drivers/pci/setup-bus.c
@@ -344,9 +344,14 @@ static struct resource *find_free_bus_re
 			 * if there is no child under that, we should release
 			 * and use it. don't need to reset it, pbus_size_* will
 			 * set it again
+			 * need to be less 3, otherwise can not write it to
+			 * bridge, also need to avoid releasing it from
+			 * transparent bus path
 			 */
-			if (!r->child && !release_resource(r))
-				return r;
+			if (i < 3 && !r->child) {
+				if (!release_resource(r))
+					return r;
+			}
 		}
 	}
 	return NULL;

Re: [PATCH] pci: only release that resource index is less than 3

[Bjorn Helgaas]

On Saturday 24 October 2009 03:25:59 am Yinghai Lu wrote:
> after
> 
> | commit 308cf8e13f42f476dfd6552aeff58fdc0788e566
> |
> |    PCI: get larger bridge ranges when space is available
> 
> found one of resource of peer root bus (0x00) get released from root
> resource. later one hotplug device can not get big range anymore.
> other peer root buses is ok.
> 
> it turns out it is from transparent path.
> 
> those resources will be used for pci bridge BAR updated.
> so need to limit it to 3.
> 
> Signed-off-by: Yinghai Lu <yinghai@kernel.org>
> 
> ---
>  drivers/pci/setup-bus.c |    9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
> 
> Index: linux-2.6/drivers/pci/setup-bus.c
> ===================================================================
> --- linux-2.6.orig/drivers/pci/setup-bus.c
> +++ linux-2.6/drivers/pci/setup-bus.c
> @@ -344,9 +344,14 @@ static struct resource *find_free_bus_re
>  			 * if there is no child under that, we should release
>  			 * and use it. don't need to reset it, pbus_size_* will
>  			 * set it again
> +			 * need to be less 3, otherwise can not write it to
> +			 * bridge, also need to avoid releasing it from
> +			 * transparent bus path
>  			 */
> -			if (!r->child && !release_resource(r))
> -				return r;
> +			if (i < 3 && !r->child) {
> +				if (!release_resource(r))
> +					return r;
> +			}

I am bewildered.

308cf8e13f42f added the release_resource() call here in
find_free_bus_resource().  I don't understand why the release
should be there in the first place -- it doesn't seem to
logically fit there.  A "release" should be connected to an
event, maybe a hot-remove or a move of a device from one place
to another.  It shouldn't be something we do as a side-effect
of searching for a free resource.

Now you're adding the magic number "3", which seems even less
related to the job of "finding an available bus resource."  I'm
guessing the "3" is related to PCI_BRIDGE_RESOURCES or something,
but that should be made explicit, and I really don't think it
belongs in this function.

Bjorn

Re: [PATCH] pci: only release that resource index is less than 3

[Jesse Barnes]

On Mon, 26 Oct 2009 10:32:57 -0600
Bjorn Helgaas <bjorn.helgaas@hp.com> wrote:

> On Saturday 24 October 2009 03:25:59 am Yinghai Lu wrote:
> > after
> > 
> > | commit 308cf8e13f42f476dfd6552aeff58fdc0788e566
> > |
> > |    PCI: get larger bridge ranges when space is available
> > 
> > found one of resource of peer root bus (0x00) get released from root
> > resource. later one hotplug device can not get big range anymore.
> > other peer root buses is ok.
> > 
> > it turns out it is from transparent path.
> > 
> > those resources will be used for pci bridge BAR updated.
> > so need to limit it to 3.
> > 
> > Signed-off-by: Yinghai Lu <yinghai@kernel.org>
> > 
> > ---
> >  drivers/pci/setup-bus.c |    9 +++++++--
> >  1 file changed, 7 insertions(+), 2 deletions(-)
> > 
> > Index: linux-2.6/drivers/pci/setup-bus.c
> > ===================================================================
> > --- linux-2.6.orig/drivers/pci/setup-bus.c
> > +++ linux-2.6/drivers/pci/setup-bus.c
> > @@ -344,9 +344,14 @@ static struct resource *find_free_bus_re
> >  			 * if there is no child under that, we
> > should release
> >  			 * and use it. don't need to reset it,
> > pbus_size_* will
> >  			 * set it again
> > +			 * need to be less 3, otherwise can not
> > write it to
> > +			 * bridge, also need to avoid releasing it
> > from
> > +			 * transparent bus path
> >  			 */
> > -			if (!r->child && !release_resource(r))
> > -				return r;
> > +			if (i < 3 && !r->child) {
> > +				if (!release_resource(r))
> > +					return r;
> > +			}
> 
> I am bewildered.
> 
> 308cf8e13f42f added the release_resource() call here in
> find_free_bus_resource().  I don't understand why the release
> should be there in the first place -- it doesn't seem to
> logically fit there.  A "release" should be connected to an
> event, maybe a hot-remove or a move of a device from one place
> to another.  It shouldn't be something we do as a side-effect
> of searching for a free resource.
> 
> Now you're adding the magic number "3", which seems even less
> related to the job of "finding an available bus resource."  I'm
> guessing the "3" is related to PCI_BRIDGE_RESOURCES or something,
> but that should be made explicit, and I really don't think it
> belongs in this function.

I agree, the 3 is a bit of magic.  The free could probably be shuffled
around a bit too, though it really is related to finding bus
resources.  I was a bit ambivalent about putting it in
find_free_bus_resources but when I started a reply to Yinghai and
looked for a better place it made some sense to leave it as-is.

But if we're going to be adding more logic here, we should probably
figure out a better way of doing it, maybe by adding a new pass to do
the freeing?  I know we've avoided modifying bus resources too much in
the past, but I think that'll be harder and harder to avoid as Windows
moves to that model and platforms begin to expect it.

-- 
Jesse Barnes, Intel Open Source Technology Center

Re: [PATCH] pci: only release that resource index is less than 3

[Yinghai Lu]

Jesse Barnes wrote:
> On Mon, 26 Oct 2009 10:32:57 -0600
> Bjorn Helgaas <bjorn.helgaas@hp.com> wrote:
> 
>> On Saturday 24 October 2009 03:25:59 am Yinghai Lu wrote:
>>> after
>>>
>>> | commit 308cf8e13f42f476dfd6552aeff58fdc0788e566
>>> |
>>> |    PCI: get larger bridge ranges when space is available
>>>
>>> found one of resource of peer root bus (0x00) get released from root
>>> resource. later one hotplug device can not get big range anymore.
>>> other peer root buses is ok.
>>>
>>> it turns out it is from transparent path.
>>>
>>> those resources will be used for pci bridge BAR updated.
>>> so need to limit it to 3.
>>>
>>> Signed-off-by: Yinghai Lu <yinghai@kernel.org>
>>>
>>> ---
>>>  drivers/pci/setup-bus.c |    9 +++++++--
>>>  1 file changed, 7 insertions(+), 2 deletions(-)
>>>
>>> Index: linux-2.6/drivers/pci/setup-bus.c
>>> ===================================================================
>>> --- linux-2.6.orig/drivers/pci/setup-bus.c
>>> +++ linux-2.6/drivers/pci/setup-bus.c
>>> @@ -344,9 +344,14 @@ static struct resource *find_free_bus_re
>>>  			 * if there is no child under that, we
>>> should release
>>>  			 * and use it. don't need to reset it,
>>> pbus_size_* will
>>>  			 * set it again
>>> +			 * need to be less 3, otherwise can not
>>> write it to
>>> +			 * bridge, also need to avoid releasing it
>>> from
>>> +			 * transparent bus path
>>>  			 */
>>> -			if (!r->child && !release_resource(r))
>>> -				return r;
>>> +			if (i < 3 && !r->child) {
>>> +				if (!release_resource(r))
>>> +					return r;
>>> +			}
>> I am bewildered.
>>
>> 308cf8e13f42f added the release_resource() call here in
>> find_free_bus_resource().  I don't understand why the release
>> should be there in the first place -- it doesn't seem to
>> logically fit there.  A "release" should be connected to an
>> event, maybe a hot-remove or a move of a device from one place
>> to another.  It shouldn't be something we do as a side-effect
>> of searching for a free resource.
>>
>> Now you're adding the magic number "3", which seems even less
>> related to the job of "finding an available bus resource."  I'm
>> guessing the "3" is related to PCI_BRIDGE_RESOURCES or something,
>> but that should be made explicit, and I really don't think it
>> belongs in this function.
> 
> I agree, the 3 is a bit of magic.  The free could probably be shuffled
> around a bit too, though it really is related to finding bus
> resources.  I was a bit ambivalent about putting it in
> find_free_bus_resources but when I started a reply to Yinghai and
> looked for a better place it made some sense to leave it as-is.
> 
> But if we're going to be adding more logic here, we should probably
> figure out a better way of doing it, maybe by adding a new pass to do
> the freeing?  I know we've avoided modifying bus resources too much in
> the past, but I think that'll be harder and harder to avoid as Windows
> moves to that model and platforms begin to expect it.

not sure if you like this version. it need more lines changes.

[PATCH] pci: only release that resource index is less than 3 -v2


after

| commit 308cf8e13f42f476dfd6552aeff58fdc0788e566
|
|    PCI: get larger bridge ranges when space is available

found one of resource of peer root bus (0x00) get released from root
resource. later one hotplug device can not get big range anymore.
other peer root buses is ok.

it turns out it is from transparent path.

those resources will be used for pci bridge BAR updated.
so need to limit it to 3.

v2: Jesse doesn't like it is in find_free_bus_resource...
    try to move out of pci_bus_size_bridges loop.
need to apply after:
	[PATCH] pci: pciehp update the slot bridge res to get big range for pcie devices - v3

Signed-off-by: Yinghai Lu <yinghai@kernel.org>

---
 drivers/pci/hotplug/pciehp_pci.c |    1 
 drivers/pci/setup-bus.c          |   83 +++++++++++++++++++++++++++++++++------
 include/linux/pci.h              |    1 
 3 files changed, 74 insertions(+), 11 deletions(-)

Index: linux-2.6/drivers/pci/setup-bus.c
===================================================================
--- linux-2.6.orig/drivers/pci/setup-bus.c
+++ linux-2.6/drivers/pci/setup-bus.c
@@ -322,33 +322,54 @@ static void pci_bridge_check_ranges(stru
 	}
 }
 
-/* Helper function for sizing routines: find first available
-   bus resource of a given type. Note: we intentionally skip
-   the bus resources which have already been assigned (that is,
-   have non-NULL parent resource). */
-static struct resource *find_free_bus_resource(struct pci_bus *bus, unsigned long type)
+void pci_bridge_release_not_used_res(struct pci_bus *bus)
 {
 	int i;
 	struct resource *r;
 	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
 				  IORESOURCE_PREFETCH;
 
-	for (i = 0; i < PCI_BUS_NUM_RESOURCES; i++) {
+	/* for pci bridges res only */
+	for (i = 0; i < 3; i++) {
 		r = bus->resource[i];
-		if (r == &ioport_resource || r == &iomem_resource)
-			continue;
-		if (r && (r->flags & type_mask) == type) {
+		if (r && (r->flags & type_mask)) {
 			if (!r->parent)
-				return r;
+				continue;
 			/*
 			 * if there is no child under that, we should release
 			 * and use it. don't need to reset it, pbus_size_* will
 			 * set it again
 			 */
 			if (!r->child && !release_resource(r))
-				return r;
+				dev_printk(KERN_DEBUG, &bus->dev,
+					   "resource %d %s %pR released\n", i,
+					   (r->flags & IORESOURCE_IO) ? "io: " :
+					    ((r->flags & IORESOURCE_PREFETCH)?
+						 "pref mem":"mem:"),
+					   r);
 		}
 	}
+}
+EXPORT_SYMBOL(pci_bridge_release_not_used_res);
+
+/* Helper function for sizing routines: find first available
+   bus resource of a given type. Note: we intentionally skip
+   the bus resources which have already been assigned (that is,
+   have non-NULL parent resource). */
+static struct resource *find_free_bus_resource(struct pci_bus *bus, unsigned long type)
+{
+	int i;
+	struct resource *r;
+	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
+				  IORESOURCE_PREFETCH;
+
+	for (i = 0; i < PCI_BUS_NUM_RESOURCES; i++) {
+		r = bus->resource[i];
+		if (r == &ioport_resource || r == &iomem_resource)
+			continue;
+		if (r && (r->flags & type_mask) == type && !r->parent)
+			return r;
+	}
 	return NULL;
 }
 
@@ -588,6 +609,41 @@ void __ref pci_bus_size_bridges(struct p
 }
 EXPORT_SYMBOL(pci_bus_size_bridges);
 
+static void pci_bus_release_bridges_not_used_res(struct pci_bus *bus)
+{
+	struct pci_dev *dev;
+
+	list_for_each_entry(dev, &bus->devices, bus_list) {
+		struct pci_bus *b = dev->subordinate;
+		if (!b)
+			continue;
+
+		switch (dev->class >> 8) {
+		case PCI_CLASS_BRIDGE_CARDBUS:
+			break;
+
+		case PCI_CLASS_BRIDGE_PCI:
+		default:
+			pci_bus_release_bridges_not_used_res(b);
+			break;
+		}
+	}
+
+	/* The root bus? */
+	if (!bus->self)
+		return;
+
+	switch (bus->self->class >> 8) {
+	case PCI_CLASS_BRIDGE_CARDBUS:
+		break;
+
+	case PCI_CLASS_BRIDGE_PCI:
+	default:
+		pci_bridge_release_not_used_res(bus);
+		break;
+	}
+}
+
 void __ref pci_bridge_assign_resources(const struct pci_dev *bridge)
 {
 	struct pci_bus *b;
@@ -687,6 +743,11 @@ pci_assign_unassigned_resources(void)
 {
 	struct pci_bus *bus;
 
+	/* Try to release bridge resources, that there is not child under it */
+	list_for_each_entry(bus, &pci_root_buses, node) {
+		pci_bus_release_bridges_not_used_res(bus);
+	}
+
 	/* Depth first, calculate sizes and alignments of all
 	   subordinate buses. */
 	list_for_each_entry(bus, &pci_root_buses, node) {
Index: linux-2.6/drivers/pci/hotplug/pciehp_pci.c
===================================================================
--- linux-2.6.orig/drivers/pci/hotplug/pciehp_pci.c
+++ linux-2.6/drivers/pci/hotplug/pciehp_pci.c
@@ -171,6 +171,7 @@ int pciehp_unconfigure_device(struct slo
 		}
 		pci_dev_put(temp);
 	}
+	pci_bridge_release_not_used_res(parent);
 
 	return rc;
 }
Index: linux-2.6/include/linux/pci.h
===================================================================
--- linux-2.6.orig/include/linux/pci.h
+++ linux-2.6/include/linux/pci.h
@@ -759,6 +759,7 @@ int pci_vpd_truncate(struct pci_dev *dev
 void pci_bridge_assign_resources(const struct pci_dev *bridge);
 void pci_bus_assign_resources(const struct pci_bus *bus);
 void pci_bus_size_bridges(struct pci_bus *bus);
+void pci_bridge_release_not_used_res(struct pci_bus *bus);
 int pci_claim_resource(struct pci_dev *, int);
 void pci_assign_unassigned_resources(void);
 void pdev_enable_device(struct pci_dev *);

Re: [PATCH] pci: only release that resource index is less than 3

[Bjorn Helgaas]

On Mon, 2009-10-26 at 14:23 -0700, Yinghai Lu wrote:

> ===================================================================
> --- linux-2.6.orig/drivers/pci/setup-bus.c
> +++ linux-2.6/drivers/pci/setup-bus.c
> @@ -322,33 +322,54 @@ static void pci_bridge_check_ranges(stru
>  	}
>  }
>  
> -/* Helper function for sizing routines: find first available
> -   bus resource of a given type. Note: we intentionally skip
> -   the bus resources which have already been assigned (that is,
> -   have non-NULL parent resource). */
> -static struct resource *find_free_bus_resource(struct pci_bus *bus, unsigned long type)
> +void pci_bridge_release_not_used_res(struct pci_bus *bus)
>  {
>  	int i;
>  	struct resource *r;
>  	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
>  				  IORESOURCE_PREFETCH;
>  
> -	for (i = 0; i < PCI_BUS_NUM_RESOURCES; i++) {
> +	/* for pci bridges res only */
> +	for (i = 0; i < 3; i++) {

I think the "i = 0; i < 3" part is to check the bridge apertures, i.e.,
the I/O base/limit, the memory base/limit, and the prefetchable memory
base/limit.  Right?  We need some way to indicate that more clearly than
using a hard-coded "3".

>  		r = bus->resource[i];
> -		if (r == &ioport_resource || r == &iomem_resource)
> -			continue;
> -		if (r && (r->flags & type_mask) == type) {
> +		if (r && (r->flags & type_mask)) {
>  			if (!r->parent)
> -				return r;
> +				continue;
>  			/*
>  			 * if there is no child under that, we should release
>  			 * and use it. don't need to reset it, pbus_size_* will
>  			 * set it again
>  			 */
>  			if (!r->child && !release_resource(r))

We got this resource pointer out of a struct pci_bus, and we release it
here.  We must have previously done a request_resource(),
allocate_resource(), or similar.  Where does that happen?  Are the
requests and releases nested correctly?

I would think that somewhere, we would be doing a request_resource() and
assigning the resource to pci_bus->resource[x].  But there are very few
assignments to the pci_bus resources:
  setup_resource (only for "pci=use_crs")
  pci_read_bridge_bases (just a copy from upstream bus resources)
  pci_alloc_child_bus (copy from upstream bridge resources)
  pci_create_bus (set to &ioport_resource or &iomem_resource)

My guess is that this release_resource() releases something we copied
from the bridge in pci_alloc_child_bus().  But that doesn't seem right,
because we aren't changing the bridge programming here.

Maybe I'm being confused by the fact that we find the resource pointer
from the pci_bus table, but that's only a pointer to the actual struct
resource that lives in the pci_dev of the upstream bridge.

Bjorn

> -				return r;
> +				dev_printk(KERN_DEBUG, &bus->dev,
> +					   "resource %d %s %pR released\n", i,
> +					   (r->flags & IORESOURCE_IO) ? "io: " :
> +					    ((r->flags & IORESOURCE_PREFETCH)?
> +						 "pref mem":"mem:"),
> +					   r);
>  		}
>  	}
> +}
> +EXPORT_SYMBOL(pci_bridge_release_not_used_res);
> +
> +/* Helper function for sizing routines: find first available
> +   bus resource of a given type. Note: we intentionally skip
> +   the bus resources which have already been assigned (that is,
> +   have non-NULL parent resource). */
> +static struct resource *find_free_bus_resource(struct pci_bus *bus, unsigned long type)
> +{
> +	int i;
> +	struct resource *r;
> +	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
> +				  IORESOURCE_PREFETCH;
> +
> +	for (i = 0; i < PCI_BUS_NUM_RESOURCES; i++) {
> +		r = bus->resource[i];
> +		if (r == &ioport_resource || r == &iomem_resource)
> +			continue;
> +		if (r && (r->flags & type_mask) == type && !r->parent)
> +			return r;
> +	}
>  	return NULL;
>  }
>  
> @@ -588,6 +609,41 @@ void __ref pci_bus_size_bridges(struct p
>  }
>  EXPORT_SYMBOL(pci_bus_size_bridges);
>  
> +static void pci_bus_release_bridges_not_used_res(struct pci_bus *bus)
> +{
> +	struct pci_dev *dev;
> +
> +	list_for_each_entry(dev, &bus->devices, bus_list) {
> +		struct pci_bus *b = dev->subordinate;
> +		if (!b)
> +			continue;
> +
> +		switch (dev->class >> 8) {
> +		case PCI_CLASS_BRIDGE_CARDBUS:
> +			break;
> +
> +		case PCI_CLASS_BRIDGE_PCI:
> +		default:
> +			pci_bus_release_bridges_not_used_res(b);
> +			break;
> +		}
> +	}
> +
> +	/* The root bus? */
> +	if (!bus->self)
> +		return;
> +
> +	switch (bus->self->class >> 8) {
> +	case PCI_CLASS_BRIDGE_CARDBUS:
> +		break;
> +
> +	case PCI_CLASS_BRIDGE_PCI:
> +	default:
> +		pci_bridge_release_not_used_res(bus);
> +		break;
> +	}
> +}
> +
>  void __ref pci_bridge_assign_resources(const struct pci_dev *bridge)
>  {
>  	struct pci_bus *b;
> @@ -687,6 +743,11 @@ pci_assign_unassigned_resources(void)
>  {
>  	struct pci_bus *bus;
>  
> +	/* Try to release bridge resources, that there is not child under it */
> +	list_for_each_entry(bus, &pci_root_buses, node) {
> +		pci_bus_release_bridges_not_used_res(bus);
> +	}
> +
>  	/* Depth first, calculate sizes and alignments of all
>  	   subordinate buses. */
>  	list_for_each_entry(bus, &pci_root_buses, node) {
> Index: linux-2.6/drivers/pci/hotplug/pciehp_pci.c
> ===================================================================
> --- linux-2.6.orig/drivers/pci/hotplug/pciehp_pci.c
> +++ linux-2.6/drivers/pci/hotplug/pciehp_pci.c
> @@ -171,6 +171,7 @@ int pciehp_unconfigure_device(struct slo
>  		}
>  		pci_dev_put(temp);
>  	}
> +	pci_bridge_release_not_used_res(parent);
>  
>  	return rc;
>  }
> Index: linux-2.6/include/linux/pci.h
> ===================================================================
> --- linux-2.6.orig/include/linux/pci.h
> +++ linux-2.6/include/linux/pci.h
> @@ -759,6 +759,7 @@ int pci_vpd_truncate(struct pci_dev *dev
>  void pci_bridge_assign_resources(const struct pci_dev *bridge);
>  void pci_bus_assign_resources(const struct pci_bus *bus);
>  void pci_bus_size_bridges(struct pci_bus *bus);
> +void pci_bridge_release_not_used_res(struct pci_bus *bus);
>  int pci_claim_resource(struct pci_dev *, int);
>  void pci_assign_unassigned_resources(void);
>  void pdev_enable_device(struct pci_dev *);

Re: [PATCH] pci: only release that resource index is less than 3

[Yinghai Lu]

Bjorn Helgaas wrote:
> On Mon, 2009-10-26 at 14:23 -0700, Yinghai Lu wrote:
> 
>> ===================================================================
>> --- linux-2.6.orig/drivers/pci/setup-bus.c
>> +++ linux-2.6/drivers/pci/setup-bus.c
>> @@ -322,33 +322,54 @@ static void pci_bridge_check_ranges(stru
>>  	}
>>  }
>>  
>> -/* Helper function for sizing routines: find first available
>> -   bus resource of a given type. Note: we intentionally skip
>> -   the bus resources which have already been assigned (that is,
>> -   have non-NULL parent resource). */
>> -static struct resource *find_free_bus_resource(struct pci_bus *bus, unsigned long type)
>> +void pci_bridge_release_not_used_res(struct pci_bus *bus)
>>  {
>>  	int i;
>>  	struct resource *r;
>>  	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
>>  				  IORESOURCE_PREFETCH;
>>  
>> -	for (i = 0; i < PCI_BUS_NUM_RESOURCES; i++) {
>> +	/* for pci bridges res only */
>> +	for (i = 0; i < 3; i++) {
> 
> I think the "i = 0; i < 3" part is to check the bridge apertures, i.e.,
> the I/O base/limit, the memory base/limit, and the prefetchable memory
> base/limit.  Right?  We need some way to indicate that more clearly than
> using a hard-coded "3".

yes. the 0x1c, 0x20, and 0x28

> 
>>  		r = bus->resource[i];
>> -		if (r == &ioport_resource || r == &iomem_resource)
>> -			continue;
>> -		if (r && (r->flags & type_mask) == type) {
>> +		if (r && (r->flags & type_mask)) {
>>  			if (!r->parent)
>> -				return r;
>> +				continue;
>>  			/*
>>  			 * if there is no child under that, we should release
>>  			 * and use it. don't need to reset it, pbus_size_* will
>>  			 * set it again
>>  			 */
>>  			if (!r->child && !release_resource(r))
> 
> We got this resource pointer out of a struct pci_bus, and we release it
> here.  We must have previously done a request_resource(),
> allocate_resource(), or similar.  Where does that happen?  Are the
> requests and releases nested correctly?
> 
> I would think that somewhere, we would be doing a request_resource() and
> assigning the resource to pci_bus->resource[x].  But there are very few
> assignments to the pci_bus resources:
>   setup_resource (only for "pci=use_crs")
>   pci_read_bridge_bases (just a copy from upstream bus resources)
>   pci_alloc_child_bus (copy from upstream bridge resources)
>   pci_create_bus (set to &ioport_resource or &iomem_resource)
> 
> My guess is that this release_resource() releases something we copied
> from the bridge in pci_alloc_child_bus().  But that doesn't seem right,
> because we aren't changing the bridge programming here.

in arch/x86/pci/i386.c::
pcibios_resource_survey() ==> pcibios_allocate_bus_resources()


YH

Re: [PATCH] pci: only release that resource index is less than 3

[Bjorn Helgaas]

On Monday 26 October 2009 06:20:28 pm Yinghai Lu wrote:
> Bjorn Helgaas wrote:
> > On Mon, 2009-10-26 at 14:23 -0700, Yinghai Lu wrote:
> > 
> >> ===================================================================
> >> --- linux-2.6.orig/drivers/pci/setup-bus.c
> >> +++ linux-2.6/drivers/pci/setup-bus.c
> >> @@ -322,33 +322,54 @@ static void pci_bridge_check_ranges(stru
> >>  	}
> >>  }
> >>  
> >> -/* Helper function for sizing routines: find first available
> >> -   bus resource of a given type. Note: we intentionally skip
> >> -   the bus resources which have already been assigned (that is,
> >> -   have non-NULL parent resource). */
> >> -static struct resource *find_free_bus_resource(struct pci_bus *bus, unsigned long type)
> >> +void pci_bridge_release_not_used_res(struct pci_bus *bus)
> >>  {
> >>  	int i;
> >>  	struct resource *r;
> >>  	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
> >>  				  IORESOURCE_PREFETCH;
> >>  
> >> -	for (i = 0; i < PCI_BUS_NUM_RESOURCES; i++) {
> >> +	/* for pci bridges res only */
> >> +	for (i = 0; i < 3; i++) {
> > 
> > I think the "i = 0; i < 3" part is to check the bridge apertures, i.e.,
> > the I/O base/limit, the memory base/limit, and the prefetchable memory
> > base/limit.  Right?  We need some way to indicate that more clearly than
> > using a hard-coded "3".
> 
> yes. the 0x1c, 0x20, and 0x28

Do you have a proposal for something other than "3"?  The magic number
"3" does appear other places, e.g., pci_read_bridge_bases() but I don't
think we should add new uses of it.  Make up a descriptive #define if
nothing else.

> >>  		r = bus->resource[i];
> >> -		if (r == &ioport_resource || r == &iomem_resource)
> >> -			continue;
> >> -		if (r && (r->flags & type_mask) == type) {
> >> +		if (r && (r->flags & type_mask)) {
> >>  			if (!r->parent)
> >> -				return r;
> >> +				continue;
> >>  			/*
> >>  			 * if there is no child under that, we should release
> >>  			 * and use it. don't need to reset it, pbus_size_* will
> >>  			 * set it again
> >>  			 */
> >>  			if (!r->child && !release_resource(r))
> > 
> > We got this resource pointer out of a struct pci_bus, and we release it
> > here.  We must have previously done a request_resource(),
> > allocate_resource(), or similar.  Where does that happen?  Are the
> > requests and releases nested correctly?
> > 
> > I would think that somewhere, we would be doing a request_resource() and
> > assigning the resource to pci_bus->resource[x].  But there are very few
> > assignments to the pci_bus resources:
> >   setup_resource (only for "pci=use_crs")
> >   pci_read_bridge_bases (just a copy from upstream bus resources)
> >   pci_alloc_child_bus (copy from upstream bridge resources)
> >   pci_create_bus (set to &ioport_resource or &iomem_resource)
> > 
> > My guess is that this release_resource() releases something we copied
> > from the bridge in pci_alloc_child_bus().  But that doesn't seem right,
> > because we aren't changing the bridge programming here.
> 
> in arch/x86/pci/i386.c::
> pcibios_resource_survey() ==> pcibios_allocate_bus_resources()

The asymmetry here bothers me.  The pci_claim_resource() in
pcibios_allocate_bus_resources() is done on a *device*, while the
release_resource() you added is done on a *bus*.  And the claim is
done in arch-specific code, while the release is done in generic
code.  And there's an interval where the bridge device resources
don't match the hardware apertures, and it's hard to figure out
when they are in sync again.

This might all work fine, but it's harder to understand than it
should be.

Bjorn

Re: [PATCH] pci: only release that resource index is less than 3

[Yinghai Lu]

[PATCH 3/4] pci: only release that resource index is less than 3 -v4

after

| commit 308cf8e13f42f476dfd6552aeff58fdc0788e566
|
|    PCI: get larger bridge ranges when space is available

found one of resource of peer root bus (0x00) get released from root
resource. later one hotplug device can not get big range anymore.
other peer root buses is ok.

it turns out it is from transparent path.

those resources will be used for pci bridge BAR updated.
so need to limit it to 3.

v2: Jesse doesn't like it is in find_free_bus_resource...
    try to move out of pci_bus_size_bridges loop.
need to apply after:
	[PATCH] pci: pciehp update the slot bridge res to get big range for pcie devices - v4
v3: add pci_setup_bridge calling after pci_bridge_release_not_used_res.
    only clear release those res for x86.
v4: Bjorn want to release use dev instead of bus.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>

---
 arch/x86/pci/i386.c              |    6 +++
 drivers/pci/hotplug/pciehp_pci.c |    2 +
 drivers/pci/setup-bus.c          |   75 ++++++++++++++++++++++++++++++++++++++-
 include/linux/pci.h              |    2 +
 4 files changed, 84 insertions(+), 1 deletion(-)

Index: linux-2.6/drivers/pci/setup-bus.c
===================================================================
--- linux-2.6.orig/drivers/pci/setup-bus.c
+++ linux-2.6/drivers/pci/setup-bus.c
@@ -319,6 +319,42 @@ static void pci_bridge_check_ranges(stru
 	}
 }
 
+void pci_bridge_release_not_used_res(struct pci_bus *bus)
+{
+	int idx;
+	bool changed = false;
+	struct pci_dev *dev;
+	struct resource *r;
+	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
+				  IORESOURCE_PREFETCH;
+
+	/* for pci bridges res only */
+	dev = bus->self;
+	for (idx = PCI_BRIDGE_RESOURCES; idx < PCI_BRIDGE_RESOURCES + 3;
+	    idx++) {
+		r = &dev->resource[idx];
+		if (r->flags & type_mask) {
+			if (!r->parent)
+				continue;
+			/*
+			 * if there is no child under that, we should release
+			 * and use it.
+			 */
+			if (!r->child && !release_resource(r)) {
+				dev_info(&dev->dev,
+					 "resource %d %pRt released\n",
+					 idx, r);
+				r->flags = 0;
+				changed = true;
+			}
+		}
+	}
+
+	if (changed)
+		pci_setup_bridge(bus);
+}
+EXPORT_SYMBOL(pci_bridge_release_not_used_res);
+
 /* Helper function for sizing routines: find first available
    bus resource of a given type. Note: we intentionally skip
    the bus resources which have already been assigned (that is,
@@ -576,6 +612,42 @@ void __ref pci_bus_size_bridges(struct p
 }
 EXPORT_SYMBOL(pci_bus_size_bridges);
 
+void __ref pci_bus_release_bridges_not_used_res(struct pci_bus *bus)
+{
+	struct pci_dev *dev;
+
+	list_for_each_entry(dev, &bus->devices, bus_list) {
+		struct pci_bus *b = dev->subordinate;
+		if (!b)
+			continue;
+
+		switch (dev->class >> 8) {
+		case PCI_CLASS_BRIDGE_CARDBUS:
+			break;
+
+		case PCI_CLASS_BRIDGE_PCI:
+		default:
+			pci_bus_release_bridges_not_used_res(b);
+			break;
+		}
+	}
+
+	/* The root bus? */
+	if (!bus->self)
+		return;
+
+	switch (bus->self->class >> 8) {
+	case PCI_CLASS_BRIDGE_CARDBUS:
+		break;
+
+	case PCI_CLASS_BRIDGE_PCI:
+	default:
+		pci_bridge_release_not_used_res(bus);
+		break;
+	}
+}
+EXPORT_SYMBOL(pci_bus_release_bridges_not_used_res);
+
 void __ref pci_bridge_assign_resources(const struct pci_dev *bridge)
 {
 	struct pci_bus *b;
@@ -644,7 +716,8 @@ static void pci_bus_dump_res(struct pci_
 
         for (i = 0; i < PCI_BUS_NUM_RESOURCES; i++) {
                 struct resource *res = bus->resource[i];
-                if (!res || !res->end)
+
+		if (!res || !res->end || !res->flags)
                         continue;
 
 		dev_printk(KERN_DEBUG, &bus->dev, "resource %d %pRt\n", i, res);
Index: linux-2.6/drivers/pci/hotplug/pciehp_pci.c
===================================================================
--- linux-2.6.orig/drivers/pci/hotplug/pciehp_pci.c
+++ linux-2.6/drivers/pci/hotplug/pciehp_pci.c
@@ -98,6 +98,7 @@ int pciehp_configure_device(struct slot
 		pci_dev_put(dev);
 	}
 
+	pci_bridge_release_not_used_res(parent);
 	pci_bus_size_bridges(parent);
 	pci_clear_master(bridge);
 	pci_bridge_assign_resources(bridge);
@@ -171,6 +172,7 @@ int pciehp_unconfigure_device(struct slo
 		}
 		pci_dev_put(temp);
 	}
+	pci_bridge_release_not_used_res(parent);
 
 	return rc;
 }
Index: linux-2.6/include/linux/pci.h
===================================================================
--- linux-2.6.orig/include/linux/pci.h
+++ linux-2.6/include/linux/pci.h
@@ -759,6 +759,8 @@ int pci_vpd_truncate(struct pci_dev *dev
 void pci_bridge_assign_resources(const struct pci_dev *bridge);
 void pci_bus_assign_resources(const struct pci_bus *bus);
 void pci_bus_size_bridges(struct pci_bus *bus);
+void pci_bus_release_bridges_not_used_res(struct pci_bus *bus);
+void pci_bridge_release_not_used_res(struct pci_bus *bus);
 int pci_claim_resource(struct pci_dev *, int);
 void pci_assign_unassigned_resources(void);
 void pdev_enable_device(struct pci_dev *);
Index: linux-2.6/arch/x86/pci/i386.c
===================================================================
--- linux-2.6.orig/arch/x86/pci/i386.c
+++ linux-2.6/arch/x86/pci/i386.c
@@ -194,6 +194,7 @@ static void __init pcibios_allocate_reso
 static int __init pcibios_assign_resources(void)
 {
 	struct pci_dev *dev = NULL;
+	struct pci_bus *bus;
 	struct resource *r;
 
 	if (!(pci_probe & PCI_ASSIGN_ROMS)) {
@@ -213,6 +214,11 @@ static int __init pcibios_assign_resourc
 		}
 	}
 
+	/* Try to release bridge resources, that there is not child under it */
+	list_for_each_entry(bus, &pci_root_buses, node) {
+		pci_bus_release_bridges_not_used_res(bus);
+	}
+
 	pci_assign_unassigned_resources();
 
 	return 0;

Re: [PATCH] pci: only release that resource index is less than 3

[Yinghai Lu]

Bjorn Helgaas wrote:
> On Saturday 24 October 2009 03:25:59 am Yinghai Lu wrote:
>> after
>>
>> | commit 308cf8e13f42f476dfd6552aeff58fdc0788e566
>> |
>> |    PCI: get larger bridge ranges when space is available
>>
>> found one of resource of peer root bus (0x00) get released from root
>> resource. later one hotplug device can not get big range anymore.
>> other peer root buses is ok.
>>
>> it turns out it is from transparent path.
>>
>> those resources will be used for pci bridge BAR updated.
>> so need to limit it to 3.
>>
>> Signed-off-by: Yinghai Lu <yinghai@kernel.org>
>>
>> ---
>>  drivers/pci/setup-bus.c |    9 +++++++--
>>  1 file changed, 7 insertions(+), 2 deletions(-)
>>
>> Index: linux-2.6/drivers/pci/setup-bus.c
>> ===================================================================
>> --- linux-2.6.orig/drivers/pci/setup-bus.c
>> +++ linux-2.6/drivers/pci/setup-bus.c
>> @@ -344,9 +344,14 @@ static struct resource *find_free_bus_re
>>  			 * if there is no child under that, we should release
>>  			 * and use it. don't need to reset it, pbus_size_* will
>>  			 * set it again
>> +			 * need to be less 3, otherwise can not write it to
>> +			 * bridge, also need to avoid releasing it from
>> +			 * transparent bus path
>>  			 */
>> -			if (!r->child && !release_resource(r))
>> -				return r;
>> +			if (i < 3 && !r->child) {
>> +				if (!release_resource(r))
>> +					return r;
>> +			}
> 
> I am bewildered.
> 
> 308cf8e13f42f added the release_resource() call here in
> find_free_bus_resource().  I don't understand why the release
> should be there in the first place -- it doesn't seem to
> logically fit there.  A "release" should be connected to an
> event, maybe a hot-remove or a move of a device from one place
> to another.  It shouldn't be something we do as a side-effect
> of searching for a free resource.

system with several IOHs, and one pci bridge under one IOH,

one of device under that bridge request big range 256M, and the bridge can not find range it, so the device BAR is not initialized. and the bridge already some range preallocated ( about 8M ) for future hotplug support.

so need to release the resource used by that bridge, so it could get new one.

> 
> Now you're adding the magic number "3", which seems even less
> related to the job of "finding an available bus resource."  I'm
> guessing the "3" is related to PCI_BRIDGE_RESOURCES or something,
> but that should be made explicit, and I really don't think it
> belongs in this function.

find_free_bus_resource is only needed by pbus_size_io and bus_size_mem with pci bridge (not card bus) to find position of 
bridge BAR, so later could update corresponding BAR.

YH