[PATCH v3 00/12] PCI: Double removing fix and allocate 64bit mmio pref

[PATCH v3 00/12] PCI: Double removing fix and allocate 64bit mmio pref

[Yinghai Lu]

First 4 are for Gu Zheng <guz.fnst@cn.fujitsu.com> to help double pci
device removing via sysfs.
For patch 4, rafael like to have use lock instead of atomic.

Patch 6 are for not enabling realloc auto when 64bit mmio is not there for
root bus.

Others are about mmio 64 allocation that could help Guo Chao <yan@linux.vnet.ibm.com> on powerpc mmio allocation.
It will try to assign 64 bit resource above 4g at first.

Patch 1-4 and Patch 5-12 could be appliey seperatedly.

Could be found:
        git://git.kernel.org/pub/scm/linux/kernel/git/yinghai/linux-yinghai.git for-pci-3.14

And it is based on current pci/for-linus.

-v2: update after patch that move device_del down to pci_destroy_dev.
     add "Try best to allocate pref mmio 64bit above 4G"

-v3: refresh and send out after pci_clip_resource() changes,
     as Bjorn is not happy with attachments.

Yinghai Lu (12):
  PCI: Use device_release_driver in pci_stop_root_bus
  PCI: Move back pci_proc_attach_devices calling
  PCI: Move resources and bus_list releasing to pci_release_dev
  PCI: Destroy pci dev only once
  PCI: pcibus address to resource converting take bus instead of dev
  PCI: Only enable realloc auto when root bus has 64bit mmio
  PCI: Don't use 4G bus address directly in resource allocation
  PCI: Try to allocate mem64 above 4G at first
  PCI: Kill PCIBIOS_MAX_MEM_32
  PCI: Try best to allocate pref mmio 64bit above 4g
  PCI: Sort pci root bus resources list
  intel-gtt: Read 64bit for gmar_bus_addr

 arch/x86/include/asm/pci.h   |   1 -
 drivers/char/agp/intel-gtt.c |  14 ++--
 drivers/pci/bus.c            |  74 ++++++++++++++++---
 drivers/pci/host-bridge.c    |  34 +++++----
 drivers/pci/probe.c          |  23 +++++-
 drivers/pci/remove.c         |  31 ++------
 drivers/pci/setup-bus.c      | 165 ++++++++++++++++++++++++++++++-------------
 drivers/pci/setup-res.c      |  14 +++-
 include/linux/pci.h          |  10 +--
 9 files changed, 257 insertions(+), 109 deletions(-)

-- 
1.8.1.4

[PATCH v3 01/12] PCI: Use device_release_driver in pci_stop_root_bus

[Yinghai Lu]

To be consistent with change in
| PCI: Move device_del() from pci_stop_dev() to pci_destroy_dev()

Use device_release_driver for root bus/hostbridge.

Also use device_unregister() in pci_remove_root_bus() instead of
device_del/put_device, that will be corresponding device_register()
for pci_create_root_bus for hostbridge.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
 drivers/pci/remove.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/remove.c b/drivers/pci/remove.c
index cc9337a..692f4c3 100644
--- a/drivers/pci/remove.c
+++ b/drivers/pci/remove.c
@@ -128,7 +128,7 @@ void pci_stop_root_bus(struct pci_bus *bus)
 		pci_stop_bus_device(child);
 
 	/* stop the host bridge */
-	device_del(&host_bridge->dev);
+	device_release_driver(&host_bridge->dev);
 }
 
 void pci_remove_root_bus(struct pci_bus *bus)
@@ -147,5 +147,5 @@ void pci_remove_root_bus(struct pci_bus *bus)
 	host_bridge->bus = NULL;
 
 	/* remove the host bridge */
-	put_device(&host_bridge->dev);
+	device_unregister(&host_bridge->dev);
 }
-- 
1.8.1.4

[PATCH v3 02/12] PCI: Move back pci_proc_attach_devices calling

[Yinghai Lu]

We stop detach proc when pci_stop_device.
So should attach that during pci_bus_add_device.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>
---
 drivers/pci/bus.c   | 1 +
 drivers/pci/probe.c | 2 --
 2 files changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
index fc1b740..1ffd95b 100644
--- a/drivers/pci/bus.c
+++ b/drivers/pci/bus.c
@@ -175,6 +175,7 @@ int pci_bus_add_device(struct pci_dev *dev)
 	 * are not assigned yet for some devices.
 	 */
 	pci_fixup_device(pci_fixup_final, dev);
+	pci_proc_attach_device(dev);
 	pci_create_sysfs_dev_files(dev);
 
 	dev->match_driver = true;
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 38e403d..173a9cf 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1381,8 +1381,6 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
 	dev->match_driver = false;
 	ret = device_add(&dev->dev);
 	WARN_ON(ret < 0);
-
-	pci_proc_attach_device(dev);
 }
 
 struct pci_dev *__ref pci_scan_single_device(struct pci_bus *bus, int devfn)
-- 
1.8.1.4

[PATCH v3 03/12] PCI: Move resources and bus_list releasing to pci_release_dev

[Yinghai Lu]

We should not release resource in pci_destroy that is too early
as there could be still other use hold reference.

release them or remove it from bus devices list at last
in pci_release_dev instead.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>
---
 drivers/pci/probe.c  | 21 +++++++++++++++++++--
 drivers/pci/remove.c | 19 -------------------
 2 files changed, 19 insertions(+), 21 deletions(-)

diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 173a9cf..12ec56c 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1154,6 +1154,18 @@ static void pci_release_capabilities(struct pci_dev *dev)
 	pci_free_cap_save_buffers(dev);
 }
 
+static void pci_free_resources(struct pci_dev *dev)
+{
+	int i;
+
+	pci_cleanup_rom(dev);
+	for (i = 0; i < PCI_NUM_RESOURCES; i++) {
+		struct resource *res = dev->resource + i;
+		if (res->parent)
+			release_resource(res);
+	}
+}
+
 /**
  * pci_release_dev - free a pci device structure when all users of it are finished.
  * @dev: device that's been disconnected
@@ -1163,9 +1175,14 @@ static void pci_release_capabilities(struct pci_dev *dev)
  */
 static void pci_release_dev(struct device *dev)
 {
-	struct pci_dev *pci_dev;
+	struct pci_dev *pci_dev = to_pci_dev(dev);
+
+	down_write(&pci_bus_sem);
+	list_del(&pci_dev->bus_list);
+	up_write(&pci_bus_sem);
+
+	pci_free_resources(pci_dev);
 
-	pci_dev = to_pci_dev(dev);
 	pci_release_capabilities(pci_dev);
 	pci_release_of_node(pci_dev);
 	pcibios_release_device(pci_dev);
diff --git a/drivers/pci/remove.c b/drivers/pci/remove.c
index 692f4c3..f452148 100644
--- a/drivers/pci/remove.c
+++ b/drivers/pci/remove.c
@@ -3,20 +3,6 @@
 #include <linux/pci-aspm.h>
 #include "pci.h"
 
-static void pci_free_resources(struct pci_dev *dev)
-{
-	int i;
-
-	msi_remove_pci_irq_vectors(dev);
-
-	pci_cleanup_rom(dev);
-	for (i = 0; i < PCI_NUM_RESOURCES; i++) {
-		struct resource *res = dev->resource + i;
-		if (res->parent)
-			release_resource(res);
-	}
-}
-
 static void pci_stop_dev(struct pci_dev *dev)
 {
 	pci_pme_active(dev, false);
@@ -36,11 +22,6 @@ static void pci_destroy_dev(struct pci_dev *dev)
 {
 	device_del(&dev->dev);
 
-	down_write(&pci_bus_sem);
-	list_del(&dev->bus_list);
-	up_write(&pci_bus_sem);
-
-	pci_free_resources(dev);
 	put_device(&dev->dev);
 }
 
-- 
1.8.1.4

[PATCH v3 04/12] PCI: Destroy pci dev only once

[Yinghai Lu]

Mutliple removing via /sys will call pci_destroy_dev two times.

| When concurent removing pci devices which are in the same pci subtree
| via sysfs, such as:
| echo -n 1 > /sys/bus/pci/devices/0000\:10\:00.0/remove ; echo -n 1 >
| /sys/bus/pci/devices/0000\:1a\:01.0/remove
| (1a:01.0 device is downstream from the 10:00.0 bridge)
|
| the following warning will show:
| [ 1799.280918] ------------[ cut here ]------------
| [ 1799.336199] WARNING: CPU: 7 PID: 126 at lib/list_debug.c:53 __list_del_entry+0x63/0xd0()
| [ 1799.433093] list_del corruption, ffff8807b4a7c000->next is LIST_POISON1 (dead000000100100)
| [ 1800.276623] CPU: 7 PID: 126 Comm: kworker/u512:1 Tainted: G        W    3.12.0-rc5+ #196
| [ 1800.508918] Workqueue: sysfsd sysfs_schedule_callback_work
| [ 1800.574703]  0000000000000009 ffff8807adbadbd8 ffffffff8168b26c ffff8807c27d08a8
| [ 1800.663860]  ffff8807adbadc28 ffff8807adbadc18 ffffffff810711dc ffff8807adbadc68
| [ 1800.753130]  ffff8807b4a7c000 ffff8807b4a7c000 ffff8807ad089c00 0000000000000000
| [ 1800.842282] Call Trace:
| [ 1800.871651]  [<ffffffff8168b26c>] dump_stack+0x55/0x76
| [ 1800.933301]  [<ffffffff810711dc>] warn_slowpath_common+0x8c/0xc0
| [ 1801.005283]  [<ffffffff810712c6>] warn_slowpath_fmt+0x46/0x50
| [ 1801.074081]  [<ffffffff8135a343>] __list_del_entry+0x63/0xd0
| [ 1801.141839]  [<ffffffff8135a3c1>] list_del+0x11/0x40
| [ 1801.201320]  [<ffffffff813734da>] pci_remove_bus_device+0x6a/0xe0
| [ 1801.274279]  [<ffffffff8137356e>] pci_stop_and_remove_bus_device+0x1e/0x30
| [ 1801.356606]  [<ffffffff8137b20b>] remove_callback+0x2b/0x40
| [ 1801.423412]  [<ffffffff81251848>] sysfs_schedule_callback_work+0x18/0x60
| [ 1801.503744]  [<ffffffff8108eab5>] process_one_work+0x1f5/0x540
| [ 1801.573640]  [<ffffffff8108ea53>] ? process_one_work+0x193/0x540
| [ 1801.645616]  [<ffffffff8108f2ac>] worker_thread+0x11c/0x370
| [ 1801.712337]  [<ffffffff8108f190>] ? rescuer_thread+0x350/0x350
| [ 1801.782178]  [<ffffffff8109731d>] kthread+0xed/0x100
| [ 1801.841661]  [<ffffffff81097230>] ? kthread_create_on_node+0x160/0x160
| [ 1801.919919]  [<ffffffff8169cc3c>] ret_from_fork+0x7c/0xb0
| [ 1801.984608]  [<ffffffff81097230>] ? kthread_create_on_node+0x160/0x160
| [ 1802.062825] ---[ end trace d77f2054de000fb7 ]---
|
| This issue is related to the bug 54411:
| https://bugzilla.kernel.org/show_bug.cgi?id=54411

Add is_removed to record if pci_destroy_dev is called already.

During second calling, still have extra dev ref hold via
device_schedule_call, so we are safe to check dev->is_removed.

It fixes the problem In Gu's test.

-v2: add partial changelog from Gu Zheng <guz.fnst@cn.fujitsu.com>
     refresh after patch of moving device_del from Rafael.
-v3: use atomic operations to prevent racing that Rafael and Bjorn concern.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>
---
 drivers/pci/probe.c  | 2 ++
 drivers/pci/remove.c | 8 +++++---
 include/linux/pci.h  | 1 +
 3 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 12ec56c..edab71d 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1398,6 +1398,8 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
 	dev->match_driver = false;
 	ret = device_add(&dev->dev);
 	WARN_ON(ret < 0);
+
+	atomic_set(&dev->removed_count, -1);
 }
 
 struct pci_dev *__ref pci_scan_single_device(struct pci_bus *bus, int devfn)
diff --git a/drivers/pci/remove.c b/drivers/pci/remove.c
index f452148..122de10 100644
--- a/drivers/pci/remove.c
+++ b/drivers/pci/remove.c
@@ -20,9 +20,11 @@ static void pci_stop_dev(struct pci_dev *dev)
 
 static void pci_destroy_dev(struct pci_dev *dev)
 {
-	device_del(&dev->dev);
-
-	put_device(&dev->dev);
+	if (atomic_inc_and_test(&dev->removed_count)) {
+		device_del(&dev->dev);
+		put_device(&dev->dev);
+	} else
+		atomic_dec(&dev->removed_count);
 }
 
 void pci_remove_bus(struct pci_bus *bus)
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 1084a15..1ca5438 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -316,6 +316,7 @@ struct pci_dev {
 	struct resource resource[DEVICE_COUNT_RESOURCE]; /* I/O and memory regions + expansion ROMs */
 
 	bool match_driver;		/* Skip attaching driver */
+	atomic_t	removed_count;	/* pci_destroy_dev is called */
 	/* These fields are used by common fixups */
 	unsigned int	transparent:1;	/* Subtractive decode PCI bridge */
 	unsigned int	multifunction:1;/* Part of multi-function device */
-- 
1.8.1.4

Re: [PATCH v3 04/12] PCI: Destroy pci dev only once

[Rafael J. Wysocki]

On Saturday, November 30, 2013 02:40:30 PM Yinghai Lu wrote:
> Mutliple removing via /sys will call pci_destroy_dev two times.
> 
> | When concurent removing pci devices which are in the same pci subtree
> | via sysfs, such as:
> | echo -n 1 > /sys/bus/pci/devices/0000\:10\:00.0/remove ; echo -n 1 >
> | /sys/bus/pci/devices/0000\:1a\:01.0/remove
> | (1a:01.0 device is downstream from the 10:00.0 bridge)
> |
> | the following warning will show:
> | [ 1799.280918] ------------[ cut here ]------------
> | [ 1799.336199] WARNING: CPU: 7 PID: 126 at lib/list_debug.c:53 __list_del_entry+0x63/0xd0()
> | [ 1799.433093] list_del corruption, ffff8807b4a7c000->next is LIST_POISON1 (dead000000100100)
> | [ 1800.276623] CPU: 7 PID: 126 Comm: kworker/u512:1 Tainted: G        W    3.12.0-rc5+ #196
> | [ 1800.508918] Workqueue: sysfsd sysfs_schedule_callback_work
> | [ 1800.574703]  0000000000000009 ffff8807adbadbd8 ffffffff8168b26c ffff8807c27d08a8
> | [ 1800.663860]  ffff8807adbadc28 ffff8807adbadc18 ffffffff810711dc ffff8807adbadc68
> | [ 1800.753130]  ffff8807b4a7c000 ffff8807b4a7c000 ffff8807ad089c00 0000000000000000
> | [ 1800.842282] Call Trace:
> | [ 1800.871651]  [<ffffffff8168b26c>] dump_stack+0x55/0x76
> | [ 1800.933301]  [<ffffffff810711dc>] warn_slowpath_common+0x8c/0xc0
> | [ 1801.005283]  [<ffffffff810712c6>] warn_slowpath_fmt+0x46/0x50
> | [ 1801.074081]  [<ffffffff8135a343>] __list_del_entry+0x63/0xd0
> | [ 1801.141839]  [<ffffffff8135a3c1>] list_del+0x11/0x40
> | [ 1801.201320]  [<ffffffff813734da>] pci_remove_bus_device+0x6a/0xe0
> | [ 1801.274279]  [<ffffffff8137356e>] pci_stop_and_remove_bus_device+0x1e/0x30
> | [ 1801.356606]  [<ffffffff8137b20b>] remove_callback+0x2b/0x40
> | [ 1801.423412]  [<ffffffff81251848>] sysfs_schedule_callback_work+0x18/0x60
> | [ 1801.503744]  [<ffffffff8108eab5>] process_one_work+0x1f5/0x540
> | [ 1801.573640]  [<ffffffff8108ea53>] ? process_one_work+0x193/0x540
> | [ 1801.645616]  [<ffffffff8108f2ac>] worker_thread+0x11c/0x370
> | [ 1801.712337]  [<ffffffff8108f190>] ? rescuer_thread+0x350/0x350
> | [ 1801.782178]  [<ffffffff8109731d>] kthread+0xed/0x100
> | [ 1801.841661]  [<ffffffff81097230>] ? kthread_create_on_node+0x160/0x160
> | [ 1801.919919]  [<ffffffff8169cc3c>] ret_from_fork+0x7c/0xb0
> | [ 1801.984608]  [<ffffffff81097230>] ? kthread_create_on_node+0x160/0x160
> | [ 1802.062825] ---[ end trace d77f2054de000fb7 ]---
> |
> | This issue is related to the bug 54411:
> | https://bugzilla.kernel.org/show_bug.cgi?id=54411
> 
> Add is_removed to record if pci_destroy_dev is called already.
> 
> During second calling, still have extra dev ref hold via
> device_schedule_call, so we are safe to check dev->is_removed.
> 
> It fixes the problem In Gu's test.
> 
> -v2: add partial changelog from Gu Zheng <guz.fnst@cn.fujitsu.com>
>      refresh after patch of moving device_del from Rafael.
> -v3: use atomic operations to prevent racing that Rafael and Bjorn concern.

And this is not enough as I said, can you please stop ignoring me openly?

You've already commented the last patch I posted, so why don't we finish
that discussion first?

Rafael

Re: [PATCH v3 04/12] PCI: Destroy pci dev only once

[Yinghai Lu]

On Sat, Nov 30, 2013 at 5:05 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> On Saturday, November 30, 2013 02:40:30 PM Yinghai Lu wrote:
>
> And this is not enough as I said, can you please stop ignoring me openly?
>
> You've already commented the last patch I posted, so why don't we finish
> that discussion first?

No, i did not mean that.

I resent the whole patchset as I added pci_clip_resource() and that
will need to revise several critial patches, and Bjorn does not like
to have patches to be attached format.

Also in the patcheset head mail, i did state that you have another
solution using
lock.

And your patch is quite big, please give me some time to digest it.

Thanks

Yinghai

Re: [PATCH v3 04/12] PCI: Destroy pci dev only once

[Rafael J. Wysocki]

On Tuesday, December 03, 2013 06:19:28 PM Yinghai Lu wrote:
> On Sat, Nov 30, 2013 at 5:05 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> > On Saturday, November 30, 2013 02:40:30 PM Yinghai Lu wrote:
> >
> > And this is not enough as I said, can you please stop ignoring me openly?
> >
> > You've already commented the last patch I posted, so why don't we finish
> > that discussion first?
> 
> No, i did not mean that.
> 
> I resent the whole patchset as I added pci_clip_resource() and that
> will need to revise several critial patches, and Bjorn does not like
> to have patches to be attached format.

I see.

> Also in the patcheset head mail, i did state that you have another
> solution using
> lock.
> 
> And your patch is quite big, please give me some time to digest it.

Yes, it is, please take your time. :-)

Thanks,
Rafael

[PATCH v3 05/12] PCI: pcibus address to resource converting take bus instead of dev

[Yinghai Lu]

For allocating resource under bus path, we do not have dev to pass along,
and we only have bus to use instead.

-v2: drop pcibios_bus_addr_to_resource().

Signed-off-by: Yinghai Lu <yinghai@kernel.org>
---
 drivers/pci/host-bridge.c | 34 +++++++++++++++++++++-------------
 include/linux/pci.h       |  5 +++++
 2 files changed, 26 insertions(+), 13 deletions(-)

diff --git a/drivers/pci/host-bridge.c b/drivers/pci/host-bridge.c
index a68dc61..c988a30 100644
--- a/drivers/pci/host-bridge.c
+++ b/drivers/pci/host-bridge.c
@@ -9,22 +9,19 @@
 
 #include "pci.h"
 
-static struct pci_bus *find_pci_root_bus(struct pci_dev *dev)
+static struct pci_bus *find_pci_root_bus(struct pci_bus *bus)
 {
-	struct pci_bus *bus;
-
-	bus = dev->bus;
 	while (bus->parent)
 		bus = bus->parent;
 
 	return bus;
 }
 
-static struct pci_host_bridge *find_pci_host_bridge(struct pci_dev *dev)
+static struct pci_host_bridge *find_pci_host_bridge(struct pci_bus *bus)
 {
-	struct pci_bus *bus = find_pci_root_bus(dev);
+	struct pci_bus *root_bus = find_pci_root_bus(bus);
 
-	return to_pci_host_bridge(bus->bridge);
+	return to_pci_host_bridge(root_bus->bridge);
 }
 
 void pci_set_host_bridge_release(struct pci_host_bridge *bridge,
@@ -40,10 +37,11 @@ static bool resource_contains(struct resource *res1, struct resource *res2)
 	return res1->start <= res2->start && res1->end >= res2->end;
 }
 
-void pcibios_resource_to_bus(struct pci_dev *dev, struct pci_bus_region *region,
-			     struct resource *res)
+void __pcibios_resource_to_bus(struct pci_bus *bus,
+				      struct pci_bus_region *region,
+				      struct resource *res)
 {
-	struct pci_host_bridge *bridge = find_pci_host_bridge(dev);
+	struct pci_host_bridge *bridge = find_pci_host_bridge(bus);
 	struct pci_host_bridge_window *window;
 	resource_size_t offset = 0;
 
@@ -60,6 +58,11 @@ void pcibios_resource_to_bus(struct pci_dev *dev, struct pci_bus_region *region,
 	region->start = res->start - offset;
 	region->end = res->end - offset;
 }
+void pcibios_resource_to_bus(struct pci_dev *dev, struct pci_bus_region *region,
+			     struct resource *res)
+{
+	__pcibios_resource_to_bus(dev->bus, region, res);
+}
 EXPORT_SYMBOL(pcibios_resource_to_bus);
 
 static bool region_contains(struct pci_bus_region *region1,
@@ -68,10 +71,10 @@ static bool region_contains(struct pci_bus_region *region1,
 	return region1->start <= region2->start && region1->end >= region2->end;
 }
 
-void pcibios_bus_to_resource(struct pci_dev *dev, struct resource *res,
-			     struct pci_bus_region *region)
+void __pcibios_bus_to_resource(struct pci_bus *bus, struct resource *res,
+				      struct pci_bus_region *region)
 {
-	struct pci_host_bridge *bridge = find_pci_host_bridge(dev);
+	struct pci_host_bridge *bridge = find_pci_host_bridge(bus);
 	struct pci_host_bridge_window *window;
 	resource_size_t offset = 0;
 
@@ -93,4 +96,9 @@ void pcibios_bus_to_resource(struct pci_dev *dev, struct resource *res,
 	res->start = region->start + offset;
 	res->end = region->end + offset;
 }
+void pcibios_bus_to_resource(struct pci_dev *dev, struct resource *res,
+			     struct pci_bus_region *region)
+{
+	__pcibios_bus_to_resource(dev->bus, res, region);
+}
 EXPORT_SYMBOL(pcibios_bus_to_resource);
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 1ca5438..a51b343 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -738,8 +738,13 @@ void pci_fixup_cardbus(struct pci_bus *);
 
 /* Generic PCI functions used internally */
 
+void __pcibios_resource_to_bus(struct pci_bus *bus,
+			       struct pci_bus_region *region,
+			       struct resource *res);
 void pcibios_resource_to_bus(struct pci_dev *dev, struct pci_bus_region *region,
 			     struct resource *res);
+void __pcibios_bus_to_resource(struct pci_bus *bus, struct resource *res,
+				      struct pci_bus_region *region);
 void pcibios_bus_to_resource(struct pci_dev *dev, struct resource *res,
 			     struct pci_bus_region *region);
 void pcibios_scan_specific_bus(int busn);
-- 
1.8.1.4

[PATCH v3 06/12] PCI: Only enable realloc auto when root bus has 64bit mmio

[Yinghai Lu]

Joseph found
| commit b07f2ebc109b607789f648dedcff4b125f9afec6
| Date:   Thu Feb 23 19:23:32 2012 -0800
|
|    PCI: add a PCI resource reallocation config option

cause one system can not load driver for Intel x520 NIC's.

The root resource:
[    1.212470] PCI host bridge to bus 0000:20
[    1.212475] pci_bus 0000:20: root bus resource [bus 20-3e]
[    1.212479] pci_bus 0000:20: root bus resource [io  0xc000-0xdfff]
[    1.212483] pci_bus 0000:20: root bus resource [mem 0xfecc0000-0xfecfffff]
[    1.212487] pci_bus 0000:20: root bus resource [mem 0xe9400000-0xe97fffff]

and bios does not assign sriov, also have two function ROM bar point to same
position.
[    1.213197] pci 0000:22:00.0: [8086:10fb] type 00 class 0x020000
...
[    1.213240] pci 0000:22:00.0: reg 0x30: [mem 0xe9500000-0xe957ffff pref]
[    1.213303] pci 0000:22:00.0: reg 0x184: [mem 0x00000000-0x00003fff 64bit]
[    1.213317] pci 0000:22:00.0: reg 0x190: [mem 0x00000000-0x00003fff 64bit]
[    1.213366] pci 0000:22:00.1: [8086:10fb] type 00 class 0x020000
...
[    1.213408] pci 0000:22:00.1: reg 0x30: [mem 0xe9500000-0xe957ffff pref]
[    1.213468] pci 0000:22:00.1: reg 0x184: [mem 0x00000000-0x00003fff 64bit]
[    1.213481] pci 0000:22:00.1: reg 0x190: [mem 0x00000000-0x00003fff 64bit]
[    1.218527] pci 0000:20:03.0: PCI bridge to [bus 22]
[    1.218534] pci 0000:20:03.0:   bridge window [io  0xd000-0xdfff]
[    1.218537] pci 0000:20:03.0:   bridge window [mem 0xe9400000-0xe95fffff]
...
[    1.254103] pci 0000:22:00.1: address space collision: [mem 0xe9500000-0xe957ffff pref] conflicts with 0000:22:00.0 [mem 0xe9500000-0xe957ffff pref]
[    1.254111] pci 0000:23:00.1: address space collision: [mem 0xe9700000-0xe977ffff pref] conflicts with 0000:23:00.0 [mem 0xe9700000-0xe977ffff pref]

We don't need to enable realloc for this case, as we can not alter root bus mmio
range to get big one to hold two rom bar, and sriov under 4G.

Add checking if pci root bus have 4G above mmio res, and don't enable
realloc auto accordingly.

bug report at: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1245938

Reported-by: Joseph Salisbury <joseph.salisbury@canonical.com>
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
---
 drivers/pci/setup-bus.c | 32 +++++++++++++++++++++++++++-----
 1 file changed, 27 insertions(+), 5 deletions(-)

diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index 219a410..f9e6efb 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -1432,17 +1432,39 @@ static int iov_resources_unassigned(struct pci_dev *dev, void *data)
 	return 0;
 }
 
+static bool pci_bus_mem_above_4g(struct pci_bus *bus)
+{
+	int i;
+	struct resource *res;
+
+	pci_bus_for_each_resource(bus, res, i) {
+		struct pci_bus_region region;
+
+		if (!res || !(res->flags & IORESOURCE_MEM))
+			continue;
+
+		__pcibios_resource_to_bus(bus, &region, res);
+		if (region.end > 0xffffffff)
+			return true;
+	}
+
+	return false;
+}
+
 static enum enable_type pci_realloc_detect(struct pci_bus *bus,
 			 enum enable_type enable_local)
 {
-	bool unassigned = false;
-
 	if (enable_local != undefined)
 		return enable_local;
 
-	pci_walk_bus(bus, iov_resources_unassigned, &unassigned);
-	if (unassigned)
-		return auto_enabled;
+	/* only enable auto when root bus does support 64bit mmio */
+	if (pci_bus_mem_above_4g(bus)) {
+		bool unassigned = false;
+
+		pci_walk_bus(bus, iov_resources_unassigned, &unassigned);
+		if (unassigned)
+			return auto_enabled;
+	}
 
 	return enable_local;
 }
-- 
1.8.1.4

[PATCH v3 07/12] PCI: Don't use 4G bus address directly in resource allocation

[Yinghai Lu]

Current we are using PCIBIOS_MAX_MEM_32 (4G limit) directly in the
pci_bus_alloc_resource to make sure that don't allocate too high
pref 64bit above 4G in the system that does not support that.

That is not right, as allocate_resource() should take resource limit.

Add pci_clip_resource() and use it check the pci bus address limit.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>
---
 drivers/pci/bus.c | 41 ++++++++++++++++++++++++++++++++++-------
 1 file changed, 34 insertions(+), 7 deletions(-)

diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
index 1ffd95b..e75bb17 100644
--- a/drivers/pci/bus.c
+++ b/drivers/pci/bus.c
@@ -98,6 +98,25 @@ void pci_bus_remove_resources(struct pci_bus *bus)
 	}
 }
 
+static struct pci_bus_region pci_mem_32 = {0, 0xffffffff};
+
+static void pci_clip_resource(struct resource *res, struct pci_bus *bus,
+			      struct pci_bus_region *region)
+{
+	struct pci_bus_region r;
+
+	__pcibios_resource_to_bus(bus, &r, res);
+	if (r.start < region->start)
+		r.start = region->start;
+	if (r.end > region->end)
+		r.end = region->end;
+
+	if (r.end < r.start)
+		res->end = res->start - 1;
+	else
+		__pcibios_bus_to_resource(bus, res, &r);
+}
+
 /**
  * pci_bus_alloc_resource - allocate a resource from a parent bus
  * @bus: PCI bus
@@ -125,15 +144,12 @@ pci_bus_alloc_resource(struct pci_bus *bus, struct resource *res,
 {
 	int i, ret = -ENOMEM;
 	struct resource *r;
-	resource_size_t max = -1;
 
 	type_mask |= IORESOURCE_IO | IORESOURCE_MEM;
 
-	/* don't allocate too high if the pref mem doesn't support 64bit*/
-	if (!(res->flags & IORESOURCE_MEM_64))
-		max = PCIBIOS_MAX_MEM_32;
-
 	pci_bus_for_each_resource(bus, r, i) {
+		struct resource avail;
+
 		if (!r)
 			continue;
 
@@ -147,10 +163,21 @@ pci_bus_alloc_resource(struct pci_bus *bus, struct resource *res,
 		    !(res->flags & IORESOURCE_PREFETCH))
 			continue;
 
+		/*
+		 * don't allocate too high if the pref mem doesn't
+		 * support 64bit.
+		 */
+		avail = *r;
+		if (!(res->flags & IORESOURCE_MEM_64)) {
+			pci_clip_resource(&avail, bus, &pci_mem_32);
+			if (!resource_size(&avail))
+				continue;
+		}
+
 		/* Ok, try it out.. */
 		ret = allocate_resource(r, res, size,
-					r->start ? : min,
-					max, align,
+					max(avail.start, r->start ? : min),
+					avail.end, align,
 					alignf, alignf_data);
 		if (ret == 0)
 			break;
-- 
1.8.1.4

[PATCH v3 08/12] PCI: Try to allocate mem64 above 4G at first

[Yinghai Lu]

On system with more pcie cards, we do not have enough range under 4G
to allocate those pci devices.

On 64bit system, we could try to allocate mem64 above 4G at first,
and fall back to below 4g if it can not find any above 4g.

x86 32bit without X86_PAE support will have bottom set to 0, because
resource_size_t is 32bit.
For 32bit kernel that resource_size_t is 64bit when pae is support.
we are safe because iomem_resource is limited to 32bit according to
x86_phys_bits.

-v2: update bottom assigning to make it clear for non-pae support machine.
-v3: Bjorn's change:
        use MAX_RESOURCE instead of -1
        use start/end instead of bottom/max
        for all arch instead of just x86_64
-v4: updated after PCI_MAX_RESOURCE_32 change.
-v5: restore io handling to use PCI_MAX_RESOURCE_32 as limit.
-v6: checking pcibios_resource_to_bus return for every bus res, to decide it
	if we need to try high at first.
     It supports all arches instead of just x86_64.
-v7: split 4G limit change out to another patch according to Bjorn.
     also use pci_clip_resource instead.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>
---
 drivers/pci/bus.c | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
index e75bb17..82eb234 100644
--- a/drivers/pci/bus.c
+++ b/drivers/pci/bus.c
@@ -99,6 +99,8 @@ void pci_bus_remove_resources(struct pci_bus *bus)
 }
 
 static struct pci_bus_region pci_mem_32 = {0, 0xffffffff};
+static struct pci_bus_region pci_mem_64 = {(resource_size_t)(1ULL<<32),
+					   (resource_size_t)(-1ULL)};
 
 static void pci_clip_resource(struct resource *res, struct pci_bus *bus,
 			      struct pci_bus_region *region)
@@ -149,6 +151,7 @@ pci_bus_alloc_resource(struct pci_bus *bus, struct resource *res,
 
 	pci_bus_for_each_resource(bus, r, i) {
 		struct resource avail;
+		int try_again = 0;
 
 		if (!r)
 			continue;
@@ -165,15 +168,23 @@ pci_bus_alloc_resource(struct pci_bus *bus, struct resource *res,
 
 		/*
 		 * don't allocate too high if the pref mem doesn't
-		 * support 64bit.
+		 * support 64bit, also if this is a 64-bit mem
+		 * resource, try above 4GB first
 		 */
 		avail = *r;
-		if (!(res->flags & IORESOURCE_MEM_64)) {
+		if (res->flags & IORESOURCE_MEM_64) {
+			pci_clip_resource(&avail, bus, &pci_mem_64);
+			if (!resource_size(&avail))
+				avail = *r;
+			else
+				try_again = 1;
+		} else {
 			pci_clip_resource(&avail, bus, &pci_mem_32);
 			if (!resource_size(&avail))
 				continue;
 		}
 
+again:
 		/* Ok, try it out.. */
 		ret = allocate_resource(r, res, size,
 					max(avail.start, r->start ? : min),
@@ -181,6 +192,12 @@ pci_bus_alloc_resource(struct pci_bus *bus, struct resource *res,
 					alignf, alignf_data);
 		if (ret == 0)
 			break;
+
+		if (try_again) {
+			avail = *r;
+			try_again = 0;
+			goto again;
+		}
 	}
 	return ret;
 }
-- 
1.8.1.4

[PATCH v3 09/12] PCI: Kill PCIBIOS_MAX_MEM_32

[Yinghai Lu]

No user anymore.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>
---
 arch/x86/include/asm/pci.h | 1 -
 include/linux/pci.h        | 4 ----
 2 files changed, 5 deletions(-)

diff --git a/arch/x86/include/asm/pci.h b/arch/x86/include/asm/pci.h
index 947b5c4..122c299 100644
--- a/arch/x86/include/asm/pci.h
+++ b/arch/x86/include/asm/pci.h
@@ -125,7 +125,6 @@ int setup_msi_irq(struct pci_dev *dev, struct msi_desc *msidesc,
 
 /* generic pci stuff */
 #include <asm-generic/pci.h>
-#define PCIBIOS_MAX_MEM_32 0xffffffff
 
 #ifdef CONFIG_NUMA
 /* Returns the node based on pci bus */
diff --git a/include/linux/pci.h b/include/linux/pci.h
index a51b343..46bf01c 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1491,10 +1491,6 @@ static inline struct pci_dev *pci_dev_get(struct pci_dev *dev)
 
 #include <asm/pci.h>
 
-#ifndef PCIBIOS_MAX_MEM_32
-#define PCIBIOS_MAX_MEM_32 (-1)
-#endif
-
 /* these helpers provide future and backwards compatibility
  * for accessing popular PCI BAR info */
 #define pci_resource_start(dev, bar)	((dev)->resource[(bar)].start)
-- 
1.8.1.4

[PATCH v3 10/12] PCI: Try best to allocate pref mmio 64bit above 4g

[Yinghai Lu]

When one of children resources does not support MEM_64, MEM_64 for
bridge get reset, so pull down whole pref resource on the bridge under 4G.

If the bridge support pref mem 64, will only allocate that with pref mem64 to
children that support it.
For children resources if they only support pref mem 32, will allocate them
from non pref mem instead.

If the bridge only support 32bit pref mmio, will still have all children pref
mmio under that.

-v2: Add release bridge res support with bridge mem res for pref_mem children res.
-v3: refresh and make it can be applied early before for_each_dev_res patchset.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Tested-by: Guo Chao <yan@linux.vnet.ibm.com>
---
 drivers/pci/setup-bus.c | 133 ++++++++++++++++++++++++++++++++----------------
 drivers/pci/setup-res.c |  14 ++++-
 2 files changed, 101 insertions(+), 46 deletions(-)

diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index f9e6efb..6f38e66 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -711,12 +711,11 @@ static void pci_bridge_check_ranges(struct pci_bus *bus)
    bus resource of a given type. Note: we intentionally skip
    the bus resources which have already been assigned (that is,
    have non-NULL parent resource). */
-static struct resource *find_free_bus_resource(struct pci_bus *bus, unsigned long type)
+static struct resource *find_free_bus_resource(struct pci_bus *bus,
+			 unsigned long type_mask, unsigned long type)
 {
 	int i;
 	struct resource *r;
-	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
-				  IORESOURCE_PREFETCH;
 
 	pci_bus_for_each_resource(bus, r, i) {
 		if (r == &ioport_resource || r == &iomem_resource)
@@ -813,7 +812,8 @@ static void pbus_size_io(struct pci_bus *bus, resource_size_t min_size,
 		resource_size_t add_size, struct list_head *realloc_head)
 {
 	struct pci_dev *dev;
-	struct resource *b_res = find_free_bus_resource(bus, IORESOURCE_IO);
+	struct resource *b_res = find_free_bus_resource(bus, IORESOURCE_IO,
+							IORESOURCE_IO);
 	resource_size_t size = 0, size0 = 0, size1 = 0;
 	resource_size_t children_add_size = 0;
 	resource_size_t min_align, align;
@@ -913,15 +913,16 @@ static inline resource_size_t calculate_mem_align(resource_size_t *aligns,
  * guarantees that all child resources fit in this size.
  */
 static int pbus_size_mem(struct pci_bus *bus, unsigned long mask,
-			 unsigned long type, resource_size_t min_size,
-			resource_size_t add_size,
-			struct list_head *realloc_head)
+			 unsigned long type, unsigned long type2,
+			 resource_size_t min_size, resource_size_t add_size,
+			 struct list_head *realloc_head)
 {
 	struct pci_dev *dev;
 	resource_size_t min_align, align, size, size0, size1;
 	resource_size_t aligns[12];	/* Alignments from 1Mb to 2Gb */
 	int order, max_order;
-	struct resource *b_res = find_free_bus_resource(bus, type);
+	struct resource *b_res = find_free_bus_resource(bus,
+					 mask | IORESOURCE_PREFETCH, type);
 	unsigned int mem64_mask = 0;
 	resource_size_t children_add_size = 0;
 
@@ -942,7 +943,8 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask,
 			struct resource *r = &dev->resource[i];
 			resource_size_t r_size;
 
-			if (r->parent || (r->flags & mask) != type)
+			if (r->parent || ((r->flags & mask) != type &&
+					  (r->flags & mask) != type2))
 				continue;
 			r_size = resource_size(r);
 #ifdef CONFIG_PCI_IOV
@@ -1115,8 +1117,9 @@ void __ref __pci_bus_size_bridges(struct pci_bus *bus,
 			struct list_head *realloc_head)
 {
 	struct pci_dev *dev;
-	unsigned long mask, prefmask;
+	unsigned long mask, prefmask, type2 = 0;
 	resource_size_t additional_mem_size = 0, additional_io_size = 0;
+	struct resource *b_res;
 
 	list_for_each_entry(dev, &bus->devices, bus_list) {
 		struct pci_bus *b = dev->subordinate;
@@ -1161,15 +1164,31 @@ void __ref __pci_bus_size_bridges(struct pci_bus *bus,
 		   has already been allocated by arch code, try
 		   non-prefetchable range for both types of PCI memory
 		   resources. */
+		b_res = &bus->self->resource[PCI_BRIDGE_RESOURCES];
 		mask = IORESOURCE_MEM;
 		prefmask = IORESOURCE_MEM | IORESOURCE_PREFETCH;
-		if (pbus_size_mem(bus, prefmask, prefmask,
+		if (b_res[2].flags & IORESOURCE_MEM_64) {
+			prefmask |= IORESOURCE_MEM_64;
+			if (pbus_size_mem(bus, prefmask, prefmask, prefmask,
 				  realloc_head ? 0 : additional_mem_size,
-				  additional_mem_size, realloc_head))
-			mask = prefmask; /* Success, size non-prefetch only. */
-		else
-			additional_mem_size += additional_mem_size;
-		pbus_size_mem(bus, mask, IORESOURCE_MEM,
+				  additional_mem_size, realloc_head)) {
+					/* Success, size non-pref64 only. */
+					mask = prefmask;
+					type2 = prefmask & ~IORESOURCE_MEM_64;
+			}
+		}
+		if (!type2) {
+			prefmask &= ~IORESOURCE_MEM_64;
+			if (pbus_size_mem(bus, prefmask, prefmask, prefmask,
+					 realloc_head ? 0 : additional_mem_size,
+					 additional_mem_size, realloc_head)) {
+				/* Success, size non-prefetch only. */
+				mask = prefmask;
+			} else
+				additional_mem_size += additional_mem_size;
+			type2 = IORESOURCE_MEM;
+		}
+		pbus_size_mem(bus, mask, IORESOURCE_MEM, type2,
 				realloc_head ? 0 : additional_mem_size,
 				additional_mem_size, realloc_head);
 		break;
@@ -1255,42 +1274,66 @@ static void __ref __pci_bridge_assign_resources(const struct pci_dev *bridge,
 static void pci_bridge_release_resources(struct pci_bus *bus,
 					  unsigned long type)
 {
-	int idx;
-	bool changed = false;
-	struct pci_dev *dev;
+	struct pci_dev *dev = bus->self;
 	struct resource *r;
 	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
-				  IORESOURCE_PREFETCH;
+				  IORESOURCE_PREFETCH | IORESOURCE_MEM_64;
+	unsigned old_flags = 0;
+	struct resource *b_res;
+	int idx = 1;
 
-	dev = bus->self;
-	for (idx = PCI_BRIDGE_RESOURCES; idx <= PCI_BRIDGE_RESOURCE_END;
-	     idx++) {
-		r = &dev->resource[idx];
-		if ((r->flags & type_mask) != type)
-			continue;
-		if (!r->parent)
-			continue;
-		/*
-		 * if there are children under that, we should release them
-		 *  all
-		 */
-		release_child_resources(r);
-		if (!release_resource(r)) {
-			dev_printk(KERN_DEBUG, &dev->dev,
-				 "resource %d %pR released\n", idx, r);
-			/* keep the old size */
-			r->end = resource_size(r) - 1;
-			r->start = 0;
-			r->flags = 0;
-			changed = true;
-		}
-	}
+	b_res = &dev->resource[PCI_BRIDGE_RESOURCES];
+
+	/*
+	 *     1. if there is io port assign fail, will release bridge
+	 *	  io port.
+	 *     2. if there is non pref mmio assign fail, release bridge
+	 *	  nonpref mmio.
+	 *     3. if there is 64bit pref mmio assign fail, and bridge pref
+	 *	  is 64bit, release bridge pref mmio.
+	 *     4. if there is pref mmio assign fail, and bridge pref is
+	 *	  32bit mmio, release bridge pref mmio
+	 *     5. if there is pref mmio assign fail, and bridge pref is not
+	 *	  assigned, release bridge nonpref mmio.
+	 */
+	if (type & IORESOURCE_IO)
+		idx = 0;
+	else if (!(type & IORESOURCE_PREFETCH))
+		idx = 1;
+	else if ((type & IORESOURCE_MEM_64) &&
+		 (b_res[2].flags & IORESOURCE_MEM_64))
+		idx = 2;
+	else if (!(b_res[2].flags & IORESOURCE_MEM_64) &&
+		 (b_res[2].flags & IORESOURCE_PREFETCH))
+		idx = 2;
+	else
+		idx = 1;
+
+	r = &b_res[idx];
+
+	if (!r->parent)
+		return;
+
+	/*
+	 * if there are children under that, we should release them
+	 *  all
+	 */
+	release_child_resources(r);
+	if (!release_resource(r)) {
+		type = old_flags = r->flags & type_mask;
+		dev_printk(KERN_DEBUG, &dev->dev, "resource %d %pR released\n",
+					PCI_BRIDGE_RESOURCES + idx, r);
+		/* keep the old size */
+		r->end = resource_size(r) - 1;
+		r->start = 0;
+		r->flags = 0;
 
-	if (changed) {
 		/* avoiding touch the one without PREF */
 		if (type & IORESOURCE_PREFETCH)
 			type = IORESOURCE_PREFETCH;
 		__pci_setup_bridge(bus, type);
+		/* for next child res under same bridge */
+		r->flags = old_flags;
 	}
 }
 
@@ -1491,7 +1534,7 @@ void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus)
 	LIST_HEAD(fail_head);
 	struct pci_dev_resource *fail_res;
 	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
-				  IORESOURCE_PREFETCH;
+				  IORESOURCE_PREFETCH | IORESOURCE_MEM_64;
 	int pci_try_num = 1;
 	enum enable_type enable_local;
 
diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c
index 83c4d3b..e968412 100644
--- a/drivers/pci/setup-res.c
+++ b/drivers/pci/setup-res.c
@@ -208,9 +208,21 @@ static int __pci_assign_resource(struct pci_bus *bus, struct pci_dev *dev,
 
 	/* First, try exact prefetching match.. */
 	ret = pci_bus_alloc_resource(bus, res, size, align, min,
-				     IORESOURCE_PREFETCH,
+				     IORESOURCE_PREFETCH | IORESOURCE_MEM_64,
 				     pcibios_align_resource, dev);
 
+	if (ret < 0 &&
+	    (res->flags & (IORESOURCE_PREFETCH | IORESOURCE_MEM_64))) {
+		/*
+		 * That failed.
+		 *
+		 * Try below 4g pref
+		 */
+		ret = pci_bus_alloc_resource(bus, res, size, align, min,
+					     IORESOURCE_PREFETCH,
+					     pcibios_align_resource, dev);
+	}
+
 	if (ret < 0 && (res->flags & IORESOURCE_PREFETCH)) {
 		/*
 		 * That failed.
-- 
1.8.1.4

[PATCH v3 11/12] PCI: Sort pci root bus resources list

[Yinghai Lu]

Some x86 systems expose above 4G 64bit mmio in _CRS as non-pref mmio range.
[   49.415281] PCI host bridge to bus 0000:00
[   49.419921] pci_bus 0000:00: root bus resource [bus 00-1e]
[   49.426107] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7]
[   49.433041] pci_bus 0000:00: root bus resource [io  0x1000-0x5fff]
[   49.440010] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff]
[   49.447768] pci_bus 0000:00: root bus resource [mem 0xfed8c000-0xfedfffff]
[   49.455532] pci_bus 0000:00: root bus resource [mem 0x90000000-0x9fffbfff]
[   49.463259] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x381fffffffff]

During assign unassigned 64bit mmio resource, it will go through
every non-pref mmio for root bus in pci_bus_alloc_resource().
As the loop is with pci_bus_for_each_resource(), and could have chance
to use under 4G mmio range instead of above 4G mmio range if the requested
range is not big enough, even it could handle above 4G 64bit pref mmio.

For root bus, we can order list from high to low in pci_add_resource_offset(),
during creating root bus, it will still keep the same order in final bus
resource list.
	pci_acpi_scan_root
		==> add_resources
			==> pci_add_resource_offset: # Add to temp resources
		==> pci_create_root_bus
			==> pci_bus_add_resource # add to final bus resources.

After that, we can make sure 64bit pref mmio for pci bridges will be allocated
higest of mmio non-pref, and in this case it is above 4G instead of under 4G.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>
---
 drivers/pci/bus.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
index 82eb234..30993ab 100644
--- a/drivers/pci/bus.c
+++ b/drivers/pci/bus.c
@@ -21,7 +21,8 @@
 void pci_add_resource_offset(struct list_head *resources, struct resource *res,
 			     resource_size_t offset)
 {
-	struct pci_host_bridge_window *window;
+	struct pci_host_bridge_window *window, *tmp;
+	struct list_head *n;
 
 	window = kzalloc(sizeof(struct pci_host_bridge_window), GFP_KERNEL);
 	if (!window) {
@@ -31,7 +32,17 @@ void pci_add_resource_offset(struct list_head *resources, struct resource *res,
 
 	window->res = res;
 	window->offset = offset;
-	list_add_tail(&window->list, resources);
+
+	/* sorted it according to res end */
+	n = resources;
+	list_for_each_entry(tmp, resources, list)
+		if (window->res->end > tmp->res->end) {
+			n = &tmp->list;
+			break;
+		}
+
+	/* Insert it just before n */
+	list_add_tail(&window->list, n);
 }
 EXPORT_SYMBOL(pci_add_resource_offset);
 
-- 
1.8.1.4

Re: [PATCH v3 11/12] PCI: Sort pci root bus resources list

[Bjorn Helgaas]

On Sat, Nov 30, 2013 at 3:40 PM, Yinghai Lu <yinghai@kernel.org> wrote:
> Some x86 systems expose above 4G 64bit mmio in _CRS as non-pref mmio range.
> [   49.415281] PCI host bridge to bus 0000:00
> [   49.419921] pci_bus 0000:00: root bus resource [bus 00-1e]
> [   49.426107] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7]
> [   49.433041] pci_bus 0000:00: root bus resource [io  0x1000-0x5fff]
> [   49.440010] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff]
> [   49.447768] pci_bus 0000:00: root bus resource [mem 0xfed8c000-0xfedfffff]
> [   49.455532] pci_bus 0000:00: root bus resource [mem 0x90000000-0x9fffbfff]
> [   49.463259] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x381fffffffff]
>
> During assign unassigned 64bit mmio resource, it will go through
> every non-pref mmio for root bus in pci_bus_alloc_resource().
> As the loop is with pci_bus_for_each_resource(), and could have chance
> to use under 4G mmio range instead of above 4G mmio range if the requested
> range is not big enough, even it could handle above 4G 64bit pref mmio.
>
> For root bus, we can order list from high to low in pci_add_resource_offset(),
> during creating root bus, it will still keep the same order in final bus
> resource list.
>         pci_acpi_scan_root
>                 ==> add_resources
>                         ==> pci_add_resource_offset: # Add to temp resources
>                 ==> pci_create_root_bus
>                         ==> pci_bus_add_resource # add to final bus resources.
>
> After that, we can make sure 64bit pref mmio for pci bridges will be allocated
> higest of mmio non-pref, and in this case it is above 4G instead of under 4G.

It really irritates me when I ask a question [1] and you just repost
the patch without even trying to answer it.

[1] http://lkml.kernel.org/r/CAErSpo4r6eJhgmfpth7haKDiKzDB+ZnEq0p_qdfTPo+kqySGgg@mail.gmail.com

> Signed-off-by: Yinghai Lu <yinghai@kernel.org>
> ---
>  drivers/pci/bus.c | 15 +++++++++++++--
>  1 file changed, 13 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
> index 82eb234..30993ab 100644
> --- a/drivers/pci/bus.c
> +++ b/drivers/pci/bus.c
> @@ -21,7 +21,8 @@
>  void pci_add_resource_offset(struct list_head *resources, struct resource *res,
>                              resource_size_t offset)
>  {
> -       struct pci_host_bridge_window *window;
> +       struct pci_host_bridge_window *window, *tmp;
> +       struct list_head *n;
>
>         window = kzalloc(sizeof(struct pci_host_bridge_window), GFP_KERNEL);
>         if (!window) {
> @@ -31,7 +32,17 @@ void pci_add_resource_offset(struct list_head *resources, struct resource *res,
>
>         window->res = res;
>         window->offset = offset;
> -       list_add_tail(&window->list, resources);
> +
> +       /* sorted it according to res end */
> +       n = resources;
> +       list_for_each_entry(tmp, resources, list)
> +               if (window->res->end > tmp->res->end) {
> +                       n = &tmp->list;
> +                       break;
> +               }
> +
> +       /* Insert it just before n */
> +       list_add_tail(&window->list, n);
>  }
>  EXPORT_SYMBOL(pci_add_resource_offset);
>
> --
> 1.8.1.4
>

Re: [PATCH v3 11/12] PCI: Sort pci root bus resources list

[Yinghai Lu]

On Mon, Dec 2, 2013 at 3:11 PM, Bjorn Helgaas <bhelgaas@google.com> wrote:
> On Sat, Nov 30, 2013 at 3:40 PM, Yinghai Lu <yinghai@kernel.org> wrote:
>> Some x86 systems expose above 4G 64bit mmio in _CRS as non-pref mmio range.
>> [   49.415281] PCI host bridge to bus 0000:00
>> [   49.419921] pci_bus 0000:00: root bus resource [bus 00-1e]
>> [   49.426107] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7]
>> [   49.433041] pci_bus 0000:00: root bus resource [io  0x1000-0x5fff]
>> [   49.440010] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff]
>> [   49.447768] pci_bus 0000:00: root bus resource [mem 0xfed8c000-0xfedfffff]
>> [   49.455532] pci_bus 0000:00: root bus resource [mem 0x90000000-0x9fffbfff]
>> [   49.463259] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x381fffffffff]
>>
>> During assign unassigned 64bit mmio resource, it will go through
>> every non-pref mmio for root bus in pci_bus_alloc_resource().
>> As the loop is with pci_bus_for_each_resource(), and could have chance
>> to use under 4G mmio range instead of above 4G mmio range if the requested
>> range is not big enough, even it could handle above 4G 64bit pref mmio.
>>
>> For root bus, we can order list from high to low in pci_add_resource_offset(),
>> during creating root bus, it will still keep the same order in final bus
>> resource list.
>>         pci_acpi_scan_root
>>                 ==> add_resources
>>                         ==> pci_add_resource_offset: # Add to temp resources
>>                 ==> pci_create_root_bus
>>                         ==> pci_bus_add_resource # add to final bus resources.
>>
>> After that, we can make sure 64bit pref mmio for pci bridges will be allocated
>> higest of mmio non-pref, and in this case it is above 4G instead of under 4G.
>
> It really irritates me when I ask a question [1] and you just repost
> the patch without even trying to answer it.
>
> [1] http://lkml.kernel.org/r/CAErSpo4r6eJhgmfpth7haKDiKzDB+ZnEq0p_qdfTPo+kqySGgg@mail.gmail.com

I expanded the changelog quite a bit, and thought it should have
answered your concern.

The old nehalem-ex and westmere-ex platform have specify above 4g mmio as
pref, but new ivybridge-ex platform only keep above 4g mmio as mmio as non-pref.

so we need allocation code to try the above 4G range for pref
allocation at first to
avoid mmio pref taking under 4G mmio range.

We may need to backport it to old kernel or stable kernel, but should
wait a while after it get into linus's tree for a while.

Thanks

Yinghai

[PATCH v3 12/12] intel-gtt: Read 64bit for gmar_bus_addr

[Yinghai Lu]

That bar could be 64bit pref mem and above 4G.

-v2: refresh to 3.13-rc1

Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Cc: David Airlie <airlied@linux.ie>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
---
 drivers/char/agp/intel-gtt.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/drivers/char/agp/intel-gtt.c b/drivers/char/agp/intel-gtt.c
index b8e2014..b929e9d 100644
--- a/drivers/char/agp/intel-gtt.c
+++ b/drivers/char/agp/intel-gtt.c
@@ -609,8 +609,10 @@ static bool intel_gtt_can_wc(void)
 static int intel_gtt_init(void)
 {
 	u32 gma_addr;
+	u32 addr_hi = 0;
 	u32 gtt_map_size;
 	int ret;
+	int pos;
 
 	ret = intel_private.driver->setup();
 	if (ret != 0)
@@ -660,13 +662,17 @@ static int intel_gtt_init(void)
 	}
 
 	if (INTEL_GTT_GEN <= 2)
-		pci_read_config_dword(intel_private.pcidev, I810_GMADDR,
-				      &gma_addr);
+		pos = I810_GMADDR;
 	else
-		pci_read_config_dword(intel_private.pcidev, I915_GMADDR,
-				      &gma_addr);
+		pos = I915_GMADDR;
+
+	pci_read_config_dword(intel_private.pcidev, pos, &gma_addr);
+
+	if (gma_addr & PCI_BASE_ADDRESS_MEM_TYPE_64)
+		pci_read_config_dword(intel_private.pcidev, pos + 4, &addr_hi);
 
 	intel_private.gma_bus_addr = (gma_addr & PCI_BASE_ADDRESS_MEM_MASK);
+	intel_private.gma_bus_addr |= (u64)addr_hi << 32;
 
 	return 0;
 }
-- 
1.8.1.4

Re: [PATCH v3 00/12] PCI: Double removing fix and allocate 64bit mmio pref

[Myron Stowe]

Yinghai:

Would you please consider *not* bundling unrelated, or at least
non-dependent, patches together in one series.  As a distrobution
subsystem maintainer, this type of bundling really complicates trying
to identify and isolate key material for bringing in.  It also makes
trying to identify the core content for key fixes, which can occur
many months or years later, significantly more difficult when they
were buried within a series with other unrelated content

Thanks,
 Myron.

On Sat, Nov 30, 2013 at 3:40 PM, Yinghai Lu <yinghai@kernel.org> wrote:
> First 4 are for Gu Zheng <guz.fnst@cn.fujitsu.com> to help double pci
> device removing via sysfs.
> For patch 4, rafael like to have use lock instead of atomic.
>
> Patch 6 are for not enabling realloc auto when 64bit mmio is not there for
> root bus.
>
> Others are about mmio 64 allocation that could help Guo Chao <yan@linux.vnet.ibm.com> on powerpc mmio allocation.
> It will try to assign 64 bit resource above 4g at first.
>
> Patch 1-4 and Patch 5-12 could be appliey seperatedly.
>
> Could be found:
>         git://git.kernel.org/pub/scm/linux/kernel/git/yinghai/linux-yinghai.git for-pci-3.14
>
> And it is based on current pci/for-linus.
>
> -v2: update after patch that move device_del down to pci_destroy_dev.
>      add "Try best to allocate pref mmio 64bit above 4G"
>
> -v3: refresh and send out after pci_clip_resource() changes,
>      as Bjorn is not happy with attachments.
>
> Yinghai Lu (12):
>   PCI: Use device_release_driver in pci_stop_root_bus
>   PCI: Move back pci_proc_attach_devices calling
>   PCI: Move resources and bus_list releasing to pci_release_dev
>   PCI: Destroy pci dev only once
>   PCI: pcibus address to resource converting take bus instead of dev
>   PCI: Only enable realloc auto when root bus has 64bit mmio
>   PCI: Don't use 4G bus address directly in resource allocation
>   PCI: Try to allocate mem64 above 4G at first
>   PCI: Kill PCIBIOS_MAX_MEM_32
>   PCI: Try best to allocate pref mmio 64bit above 4g
>   PCI: Sort pci root bus resources list
>   intel-gtt: Read 64bit for gmar_bus_addr
>
>  arch/x86/include/asm/pci.h   |   1 -
>  drivers/char/agp/intel-gtt.c |  14 ++--
>  drivers/pci/bus.c            |  74 ++++++++++++++++---
>  drivers/pci/host-bridge.c    |  34 +++++----
>  drivers/pci/probe.c          |  23 +++++-
>  drivers/pci/remove.c         |  31 ++------
>  drivers/pci/setup-bus.c      | 165 ++++++++++++++++++++++++++++++-------------
>  drivers/pci/setup-res.c      |  14 +++-
>  include/linux/pci.h          |  10 +--
>  9 files changed, 257 insertions(+), 109 deletions(-)
>
> --
> 1.8.1.4
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 00/12] PCI: Double removing fix and allocate 64bit mmio pref

[Yinghai Lu]

On Sun, Dec 1, 2013 at 9:05 AM, Myron Stowe <myron.stowe@gmail.com> wrote:
> Would you please consider *not* bundling unrelated, or at least
> non-dependent, patches together in one series.  As a distrobution
> subsystem maintainer, this type of bundling really complicates trying
> to identify and isolate key material for bringing in.  It also makes
> trying to identify the core content for key fixes, which can occur
> many months or years later, significantly more difficult when they
> were buried within a series with other unrelated content

ok.

Will resend them in three patchset.

Thanks

Yinghai