[PATCH v9 00/26] powerpc/powernv: PCI hotplug preparation

[PATCH v9 00/26] powerpc/powernv: PCI hotplug preparation

[Gavin Shan]

The series is split from "[PATCH v8 00/45] powerpc/powernv: PCI hotplug
support". The series does couple of things as below. The patches are
required to support PCI hotplug on PowerNV platforms. However, the
patches refactor the code with the goal: not affecting current logic.

   * Code cleanup and refactoring.
   * Track IO/M32/M64 segments consumed by one particular PE.
   * Remove DMA32 list and improve DMA32 segment tracking.
   * M64 support for IODA1 so that we have unified basis for the
     subsequent patches to support PCI hotplug.
   * Couple of fixes to PCI hotplug (used by EEH).
   * Exported functions to be used by PCI hotplug.

Gavin Shan (26):
  powerpc/pci: Cleanup on struct pci_controller_ops
  powerpc/powernv: Cleanup on pci_controller_ops instances
  powerpc/powernv: Drop phb->bdfn_to_pe()
  powerpc/powernv: Reorder fields in struct pnv_phb
  powerpc/powernv: Rename PE# fields in struct pnv_phb
  powerpc/powernv: Data type unsigned int for PE number
  powerpc/powernv: Fix initial IO and M32 segmap
  powerpc/powernv: Simplify pnv_ioda_setup_pe_seg()
  powerpc/powernv: IO and M32 mapping based on PCI device resources
  powerpc/powernv: Track M64 segment consumption
  powerpc/powernv: Rename M64 related functions
  powerpc/powernv/ioda1: M64 support on P7IOC
  powerpc/powernv/ioda1: Rename pnv_pci_ioda_setup_dma_pe()
  powerpc/powernv/ioda1: Introduce PNV_IODA1_DMA32_SEGSIZE
  powerpc/powernv: Remove DMA32 PE list
  powerpc/powernv/ioda1: Improve DMA32 segment track
  powerpc/powernv: Use PE instead of number during setup and release
  powerpc/pci: Rename pcibios_{add,remove}_pci_devices()
  powerpc/pci: Rename pcibios_find_pci_bus()
  powerpc/pci: Move pci_find_bus_by_node() around
  powerpc/pci: Export pci_add_device_node_info()
  powerpc/pci: Introduce pci_remove_device_node_info()
  powerpc/pci: Export pci_traverse_device_nodes()
  powerpc/pci: Don't scan empty slot
  powerpc/powernv: Simplify pnv_eeh_reset()
  powerpc/powernv: Exclude root bus in pnv_pci_reset_secondary_bus()

 arch/powerpc/include/asm/pci-bridge.h        |  41 +-
 arch/powerpc/include/asm/ppc-pci.h           |   6 +-
 arch/powerpc/kernel/eeh_driver.c             |  12 +-
 arch/powerpc/kernel/pci-hotplug.c            |  47 +-
 arch/powerpc/kernel/pci_dn.c                 |  66 ++-
 arch/powerpc/platforms/powernv/eeh-powernv.c |  81 ++-
 arch/powerpc/platforms/powernv/pci-ioda.c    | 719 +++++++++++++++------------
 arch/powerpc/platforms/powernv/pci.c         |   4 +-
 arch/powerpc/platforms/powernv/pci.h         |  41 +-
 arch/powerpc/platforms/pseries/msi.c         |   4 +-
 arch/powerpc/platforms/pseries/pci_dlpar.c   |  32 --
 arch/powerpc/platforms/pseries/setup.c       |   2 +-
 drivers/pci/hotplug/rpadlpar_core.c          |   8 +-
 drivers/pci/hotplug/rpaphp_core.c            |   4 +-
 drivers/pci/hotplug/rpaphp_pci.c             |   4 +-
 15 files changed, 592 insertions(+), 479 deletions(-)

-- 
2.1.0

[PATCH v9 01/26] powerpc/pci: Cleanup on struct pci_controller_ops

[Gavin Shan]

Each PHB has one instance of "struct pci_controller_ops" that includes
various callbacks called by PCI subsystem. In the definition of this
struct, some callbacks have explicit names for its arguments, but the
left don't have.

This adds all explicit names of the arguments to the callbacks in
"struct pci_controller_ops" so that the code looks consistent. Also,
argument name @dev is replaced by @pdev as the later one is the
preferred name for PCI device.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
Reviewed-by: Daniel Axtens <dja@axtens.net>
Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/include/asm/pci-bridge.h | 25 +++++++++++++------------
 1 file changed, 13 insertions(+), 12 deletions(-)

diff --git a/arch/powerpc/include/asm/pci-bridge.h b/arch/powerpc/include/asm/pci-bridge.h
index f5056e3..023c8c8 100644
--- a/arch/powerpc/include/asm/pci-bridge.h
+++ b/arch/powerpc/include/asm/pci-bridge.h
@@ -17,33 +17,34 @@ struct device_node;
  * PCI controller operations
  */
 struct pci_controller_ops {
-	void		(*dma_dev_setup)(struct pci_dev *dev);
+	void		(*dma_dev_setup)(struct pci_dev *pdev);
 	void		(*dma_bus_setup)(struct pci_bus *bus);
 
-	int		(*probe_mode)(struct pci_bus *);
+	int		(*probe_mode)(struct pci_bus *bus);
 
 	/* Called when pci_enable_device() is called. Returns true to
 	 * allow assignment/enabling of the device. */
-	bool		(*enable_device_hook)(struct pci_dev *);
+	bool		(*enable_device_hook)(struct pci_dev *pdev);
 
-	void		(*disable_device)(struct pci_dev *);
+	void		(*disable_device)(struct pci_dev *pdev);
 
-	void		(*release_device)(struct pci_dev *);
+	void		(*release_device)(struct pci_dev *pdev);
 
 	/* Called during PCI resource reassignment */
-	resource_size_t (*window_alignment)(struct pci_bus *, unsigned long type);
-	void		(*reset_secondary_bus)(struct pci_dev *dev);
+	resource_size_t (*window_alignment)(struct pci_bus *bus,
+					    unsigned long type);
+	void		(*reset_secondary_bus)(struct pci_dev *pdev);
 
 #ifdef CONFIG_PCI_MSI
-	int		(*setup_msi_irqs)(struct pci_dev *dev,
+	int		(*setup_msi_irqs)(struct pci_dev *pdev,
 					  int nvec, int type);
-	void		(*teardown_msi_irqs)(struct pci_dev *dev);
+	void		(*teardown_msi_irqs)(struct pci_dev *pdev);
 #endif
 
-	int             (*dma_set_mask)(struct pci_dev *dev, u64 dma_mask);
-	u64		(*dma_get_required_mask)(struct pci_dev *dev);
+	int             (*dma_set_mask)(struct pci_dev *pdev, u64 dma_mask);
+	u64		(*dma_get_required_mask)(struct pci_dev *pdev);
 
-	void		(*shutdown)(struct pci_controller *);
+	void		(*shutdown)(struct pci_controller *hose);
 };
 
 /*
-- 
2.1.0

[PATCH v9 02/26] powerpc/powernv: Cleanup on pci_controller_ops instances

[Gavin Shan]

This cleans up on below data struct instances to use tab instead of
space indent of statement to avoid complains from scripts/checkpatch.pl.
No logical changes introduced.

  @pnv_pci_ioda_controller_ops
  @pnv_npu_ioda_controller_ops

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
Reviewed-by: Daniel Axtens <dja@axtens.net>
Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 36 +++++++++++++++----------------
 1 file changed, 18 insertions(+), 18 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index c5baaf3..524c9c7 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -3210,31 +3210,31 @@ static void pnv_pci_ioda_shutdown(struct pci_controller *hose)
 }
 
 static const struct pci_controller_ops pnv_pci_ioda_controller_ops = {
-       .dma_dev_setup = pnv_pci_dma_dev_setup,
-       .dma_bus_setup = pnv_pci_dma_bus_setup,
+	.dma_dev_setup		= pnv_pci_dma_dev_setup,
+	.dma_bus_setup		= pnv_pci_dma_bus_setup,
 #ifdef CONFIG_PCI_MSI
-       .setup_msi_irqs = pnv_setup_msi_irqs,
-       .teardown_msi_irqs = pnv_teardown_msi_irqs,
+	.setup_msi_irqs		= pnv_setup_msi_irqs,
+	.teardown_msi_irqs	= pnv_teardown_msi_irqs,
 #endif
-       .enable_device_hook = pnv_pci_enable_device_hook,
-       .window_alignment = pnv_pci_window_alignment,
-       .reset_secondary_bus = pnv_pci_reset_secondary_bus,
-       .dma_set_mask = pnv_pci_ioda_dma_set_mask,
-       .dma_get_required_mask = pnv_pci_ioda_dma_get_required_mask,
-       .shutdown = pnv_pci_ioda_shutdown,
+	.enable_device_hook	= pnv_pci_enable_device_hook,
+	.window_alignment	= pnv_pci_window_alignment,
+	.reset_secondary_bus	= pnv_pci_reset_secondary_bus,
+	.dma_set_mask		= pnv_pci_ioda_dma_set_mask,
+	.dma_get_required_mask	= pnv_pci_ioda_dma_get_required_mask,
+	.shutdown		= pnv_pci_ioda_shutdown,
 };
 
 static const struct pci_controller_ops pnv_npu_ioda_controller_ops = {
-	.dma_dev_setup = pnv_pci_dma_dev_setup,
+	.dma_dev_setup		= pnv_pci_dma_dev_setup,
 #ifdef CONFIG_PCI_MSI
-	.setup_msi_irqs = pnv_setup_msi_irqs,
-	.teardown_msi_irqs = pnv_teardown_msi_irqs,
+	.setup_msi_irqs		= pnv_setup_msi_irqs,
+	.teardown_msi_irqs	= pnv_teardown_msi_irqs,
 #endif
-	.enable_device_hook = pnv_pci_enable_device_hook,
-	.window_alignment = pnv_pci_window_alignment,
-	.reset_secondary_bus = pnv_pci_reset_secondary_bus,
-	.dma_set_mask = pnv_npu_dma_set_mask,
-	.shutdown = pnv_pci_ioda_shutdown,
+	.enable_device_hook	= pnv_pci_enable_device_hook,
+	.window_alignment	= pnv_pci_window_alignment,
+	.reset_secondary_bus	= pnv_pci_reset_secondary_bus,
+	.dma_set_mask		= pnv_npu_dma_set_mask,
+	.shutdown		= pnv_pci_ioda_shutdown,
 };
 
 static void __init pnv_pci_init_ioda_phb(struct device_node *np,
-- 
2.1.0

[PATCH v9 03/26] powerpc/powernv: Drop phb->bdfn_to_pe()

[Gavin Shan]

This drops struct pnv_phb::bdfn_to_pe() as nobody uses it.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 9 ---------
 arch/powerpc/platforms/powernv/pci.h      | 1 -
 2 files changed, 10 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 524c9c7..10ecd97 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -3195,12 +3195,6 @@ static bool pnv_pci_enable_device_hook(struct pci_dev *dev)
 	return true;
 }
 
-static u32 pnv_ioda_bdfn_to_pe(struct pnv_phb *phb, struct pci_bus *bus,
-			       u32 devfn)
-{
-	return phb->ioda.pe_rmap[(bus->number << 8) | devfn];
-}
-
 static void pnv_pci_ioda_shutdown(struct pci_controller *hose)
 {
 	struct pnv_phb *phb = hose->private_data;
@@ -3377,9 +3371,6 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 	phb->freeze_pe = pnv_ioda_freeze_pe;
 	phb->unfreeze_pe = pnv_ioda_unfreeze_pe;
 
-	/* Setup RID -> PE mapping function */
-	phb->bdfn_to_pe = pnv_ioda_bdfn_to_pe;
-
 	/* Setup TCEs */
 	phb->dma_dev_setup = pnv_pci_ioda_dma_dev_setup;
 
diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
index 3f814f3..78f035e 100644
--- a/arch/powerpc/platforms/powernv/pci.h
+++ b/arch/powerpc/platforms/powernv/pci.h
@@ -110,7 +110,6 @@ struct pnv_phb {
 			 unsigned int is_64, struct msi_msg *msg);
 	void (*dma_dev_setup)(struct pnv_phb *phb, struct pci_dev *pdev);
 	void (*fixup_phb)(struct pci_controller *hose);
-	u32 (*bdfn_to_pe)(struct pnv_phb *phb, struct pci_bus *bus, u32 devfn);
 	int (*init_m64)(struct pnv_phb *phb);
 	void (*reserve_m64_pe)(struct pci_bus *bus,
 			       unsigned long *pe_bitmap, bool all);
-- 
2.1.0

[PATCH v9 04/26] powerpc/powernv: Reorder fields in struct pnv_phb

[Gavin Shan]

This moves those fields in struct pnv_phb that are related to PE
allocation around. No logical change.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/platforms/powernv/pci.h | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
index 78f035e..f2a1452 100644
--- a/arch/powerpc/platforms/powernv/pci.h
+++ b/arch/powerpc/platforms/powernv/pci.h
@@ -140,15 +140,14 @@ struct pnv_phb {
 		unsigned int		io_segsize;
 		unsigned int		io_pci_base;
 
-		/* PE allocation bitmap */
-		unsigned long		*pe_alloc;
-		/* PE allocation mutex */
+		/* PE allocation */
 		struct mutex		pe_alloc_mutex;
+		unsigned long		*pe_alloc;
+		struct pnv_ioda_pe	*pe_array;
 
 		/* M32 & IO segment maps */
 		unsigned int		*m32_segmap;
 		unsigned int		*io_segmap;
-		struct pnv_ioda_pe	*pe_array;
 
 		/* IRQ chip */
 		int			irq_chip_init;
-- 
2.1.0

[PATCH v9 05/26] powerpc/powernv: Rename PE# fields in struct pnv_phb

[Gavin Shan]

This renames the fields related to PE number in "struct pnv_phb"
for better reflecting of their usages as Alexey suggested. No
logical changes introduced.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/platforms/powernv/eeh-powernv.c |  2 +-
 arch/powerpc/platforms/powernv/pci-ioda.c    | 58 ++++++++++++++--------------
 arch/powerpc/platforms/powernv/pci.c         |  2 +-
 arch/powerpc/platforms/powernv/pci.h         |  4 +-
 4 files changed, 33 insertions(+), 33 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/eeh-powernv.c b/arch/powerpc/platforms/powernv/eeh-powernv.c
index 950b3e5..69e41ce 100644
--- a/arch/powerpc/platforms/powernv/eeh-powernv.c
+++ b/arch/powerpc/platforms/powernv/eeh-powernv.c
@@ -75,7 +75,7 @@ static int pnv_eeh_init(void)
 		 * and P7IOC separately. So we should regard
 		 * PE#0 as valid for PHB3 and P7IOC.
 		 */
-		if (phb->ioda.reserved_pe != 0)
+		if (phb->ioda.reserved_pe_idx != 0)
 			eeh_add_flag(EEH_VALID_PE_ZERO);
 
 		break;
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 10ecd97..1d2514f 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -124,7 +124,7 @@ static inline bool pnv_pci_is_mem_pref_64(unsigned long flags)
 
 static void pnv_ioda_reserve_pe(struct pnv_phb *phb, int pe_no)
 {
-	if (!(pe_no >= 0 && pe_no < phb->ioda.total_pe)) {
+	if (!(pe_no >= 0 && pe_no < phb->ioda.total_pe_num)) {
 		pr_warn("%s: Invalid PE %d on PHB#%x\n",
 			__func__, pe_no, phb->hose->global_number);
 		return;
@@ -144,8 +144,8 @@ static int pnv_ioda_alloc_pe(struct pnv_phb *phb)
 
 	do {
 		pe = find_next_zero_bit(phb->ioda.pe_alloc,
-					phb->ioda.total_pe, 0);
-		if (pe >= phb->ioda.total_pe)
+					phb->ioda.total_pe_num, 0);
+		if (pe >= phb->ioda.total_pe_num)
 			return IODA_INVALID_PE;
 	} while(test_and_set_bit(pe, phb->ioda.pe_alloc));
 
@@ -199,13 +199,13 @@ static int pnv_ioda2_init_m64(struct pnv_phb *phb)
 	 * expected to be 0 or last one of PE capabicity.
 	 */
 	r = &phb->hose->mem_resources[1];
-	if (phb->ioda.reserved_pe == 0)
+	if (phb->ioda.reserved_pe_idx == 0)
 		r->start += phb->ioda.m64_segsize;
-	else if (phb->ioda.reserved_pe == (phb->ioda.total_pe - 1))
+	else if (phb->ioda.reserved_pe_idx == (phb->ioda.total_pe_num - 1))
 		r->end -= phb->ioda.m64_segsize;
 	else
 		pr_warn("  Cannot strip M64 segment for reserved PE#%d\n",
-			phb->ioda.reserved_pe);
+			phb->ioda.reserved_pe_idx);
 
 	return 0;
 
@@ -274,7 +274,7 @@ static int pnv_ioda2_pick_m64_pe(struct pci_bus *bus, bool all)
 		return IODA_INVALID_PE;
 
 	/* Allocate bitmap */
-	size = _ALIGN_UP(phb->ioda.total_pe / 8, sizeof(unsigned long));
+	size = _ALIGN_UP(phb->ioda.total_pe_num / 8, sizeof(unsigned long));
 	pe_alloc = kzalloc(size, GFP_KERNEL);
 	if (!pe_alloc) {
 		pr_warn("%s: Out of memory !\n",
@@ -290,7 +290,7 @@ static int pnv_ioda2_pick_m64_pe(struct pci_bus *bus, bool all)
 	 * contributed by its child buses. For the case, we needn't
 	 * pick M64 dependent PE#.
 	 */
-	if (bitmap_empty(pe_alloc, phb->ioda.total_pe)) {
+	if (bitmap_empty(pe_alloc, phb->ioda.total_pe_num)) {
 		kfree(pe_alloc);
 		return IODA_INVALID_PE;
 	}
@@ -301,8 +301,8 @@ static int pnv_ioda2_pick_m64_pe(struct pci_bus *bus, bool all)
 	 */
 	master_pe = NULL;
 	i = -1;
-	while ((i = find_next_bit(pe_alloc, phb->ioda.total_pe, i + 1)) <
-		phb->ioda.total_pe) {
+	while ((i = find_next_bit(pe_alloc, phb->ioda.total_pe_num, i + 1)) <
+		phb->ioda.total_pe_num) {
 		pe = &phb->ioda.pe_array[i];
 
 		if (!master_pe) {
@@ -355,7 +355,7 @@ static void __init pnv_ioda_parse_m64_window(struct pnv_phb *phb)
 	hose->mem_offset[1] = res->start - pci_addr;
 
 	phb->ioda.m64_size = resource_size(res);
-	phb->ioda.m64_segsize = phb->ioda.m64_size / phb->ioda.total_pe;
+	phb->ioda.m64_segsize = phb->ioda.m64_size / phb->ioda.total_pe_num;
 	phb->ioda.m64_base = pci_addr;
 
 	pr_info(" MEM64 0x%016llx..0x%016llx -> 0x%016llx\n",
@@ -456,7 +456,7 @@ static int pnv_ioda_get_pe_state(struct pnv_phb *phb, int pe_no)
 	s64 rc;
 
 	/* Sanity check on PE number */
-	if (pe_no < 0 || pe_no >= phb->ioda.total_pe)
+	if (pe_no < 0 || pe_no >= phb->ioda.total_pe_num)
 		return OPAL_EEH_STOPPED_PERM_UNAVAIL;
 
 	/*
@@ -1088,7 +1088,7 @@ static struct pnv_ioda_pe *pnv_ioda_setup_npu_PE(struct pci_dev *npu_pdev)
 	 * same GPU get assigned the same PE.
 	 */
 	gpu_pdev = pnv_pci_get_gpu_dev(npu_pdev);
-	for (pe_num = 0; pe_num < phb->ioda.total_pe; pe_num++) {
+	for (pe_num = 0; pe_num < phb->ioda.total_pe_num; pe_num++) {
 		pe = &phb->ioda.pe_array[pe_num];
 		if (!pe->pdev)
 			continue;
@@ -1537,9 +1537,9 @@ int pnv_pci_sriov_enable(struct pci_dev *pdev, u16 num_vfs)
 		} else {
 			mutex_lock(&phb->ioda.pe_alloc_mutex);
 			*pdn->pe_num_map = bitmap_find_next_zero_area(
-				phb->ioda.pe_alloc, phb->ioda.total_pe,
+				phb->ioda.pe_alloc, phb->ioda.total_pe_num,
 				0, num_vfs, 0);
-			if (*pdn->pe_num_map >= phb->ioda.total_pe) {
+			if (*pdn->pe_num_map >= phb->ioda.total_pe_num) {
 				mutex_unlock(&phb->ioda.pe_alloc_mutex);
 				dev_info(&pdev->dev, "Failed to enable VF%d\n", num_vfs);
 				kfree(pdn->pe_num_map);
@@ -2858,7 +2858,7 @@ static void pnv_pci_ioda_fixup_iov_resources(struct pci_dev *pdev)
 	pdn->m64_single_mode = false;
 
 	total_vfs = pci_sriov_get_totalvfs(pdev);
-	mul = phb->ioda.total_pe;
+	mul = phb->ioda.total_pe_num;
 	total_vf_bar_sz = 0;
 
 	for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
@@ -2960,7 +2960,7 @@ static void pnv_ioda_setup_pe_seg(struct pci_controller *hose,
 			region.end   = res->end - phb->ioda.io_pci_base;
 			index = region.start / phb->ioda.io_segsize;
 
-			while (index < phb->ioda.total_pe &&
+			while (index < phb->ioda.total_pe_num &&
 			       region.start <= region.end) {
 				phb->ioda.io_segmap[index] = pe->pe_number;
 				rc = opal_pci_map_pe_mmio_window(phb->opal_id,
@@ -2985,7 +2985,7 @@ static void pnv_ioda_setup_pe_seg(struct pci_controller *hose,
 				       phb->ioda.m32_pci_base;
 			index = region.start / phb->ioda.m32_segsize;
 
-			while (index < phb->ioda.total_pe &&
+			while (index < phb->ioda.total_pe_num &&
 			       region.start <= region.end) {
 				phb->ioda.m32_segmap[index] = pe->pe_number;
 				rc = opal_pci_map_pe_mmio_window(phb->opal_id,
@@ -3300,13 +3300,13 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 		pr_err("  Failed to map registers !\n");
 
 	/* Initialize more IODA stuff */
-	phb->ioda.total_pe = 1;
+	phb->ioda.total_pe_num = 1;
 	prop32 = of_get_property(np, "ibm,opal-num-pes", NULL);
 	if (prop32)
-		phb->ioda.total_pe = be32_to_cpup(prop32);
+		phb->ioda.total_pe_num = be32_to_cpup(prop32);
 	prop32 = of_get_property(np, "ibm,opal-reserved-pe", NULL);
 	if (prop32)
-		phb->ioda.reserved_pe = be32_to_cpup(prop32);
+		phb->ioda.reserved_pe_idx = be32_to_cpup(prop32);
 
 	/* Parse 64-bit MMIO range */
 	pnv_ioda_parse_m64_window(phb);
@@ -3315,29 +3315,29 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 	/* FW Has already off top 64k of M32 space (MSI space) */
 	phb->ioda.m32_size += 0x10000;
 
-	phb->ioda.m32_segsize = phb->ioda.m32_size / phb->ioda.total_pe;
+	phb->ioda.m32_segsize = phb->ioda.m32_size / phb->ioda.total_pe_num;
 	phb->ioda.m32_pci_base = hose->mem_resources[0].start - hose->mem_offset[0];
 	phb->ioda.io_size = hose->pci_io_size;
-	phb->ioda.io_segsize = phb->ioda.io_size / phb->ioda.total_pe;
+	phb->ioda.io_segsize = phb->ioda.io_size / phb->ioda.total_pe_num;
 	phb->ioda.io_pci_base = 0; /* XXX calculate this ? */
 
 	/* Allocate aux data & arrays. We don't have IO ports on PHB3 */
-	size = _ALIGN_UP(phb->ioda.total_pe / 8, sizeof(unsigned long));
+	size = _ALIGN_UP(phb->ioda.total_pe_num / 8, sizeof(unsigned long));
 	m32map_off = size;
-	size += phb->ioda.total_pe * sizeof(phb->ioda.m32_segmap[0]);
+	size += phb->ioda.total_pe_num * sizeof(phb->ioda.m32_segmap[0]);
 	if (phb->type == PNV_PHB_IODA1) {
 		iomap_off = size;
-		size += phb->ioda.total_pe * sizeof(phb->ioda.io_segmap[0]);
+		size += phb->ioda.total_pe_num * sizeof(phb->ioda.io_segmap[0]);
 	}
 	pemap_off = size;
-	size += phb->ioda.total_pe * sizeof(struct pnv_ioda_pe);
+	size += phb->ioda.total_pe_num * sizeof(struct pnv_ioda_pe);
 	aux = memblock_virt_alloc(size, 0);
 	phb->ioda.pe_alloc = aux;
 	phb->ioda.m32_segmap = aux + m32map_off;
 	if (phb->type == PNV_PHB_IODA1)
 		phb->ioda.io_segmap = aux + iomap_off;
 	phb->ioda.pe_array = aux + pemap_off;
-	set_bit(phb->ioda.reserved_pe, phb->ioda.pe_alloc);
+	set_bit(phb->ioda.reserved_pe_idx, phb->ioda.pe_alloc);
 
 	INIT_LIST_HEAD(&phb->ioda.pe_dma_list);
 	INIT_LIST_HEAD(&phb->ioda.pe_list);
@@ -3356,7 +3356,7 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 #endif
 
 	pr_info("  %03d (%03d) PE's M32: 0x%x [segment=0x%x]\n",
-		phb->ioda.total_pe, phb->ioda.reserved_pe,
+		phb->ioda.total_pe_num, phb->ioda.reserved_pe_idx,
 		phb->ioda.m32_size, phb->ioda.m32_segsize);
 	if (phb->ioda.m64_size)
 		pr_info("                 M64: 0x%lx [segment=0x%lx]\n",
diff --git a/arch/powerpc/platforms/powernv/pci.c b/arch/powerpc/platforms/powernv/pci.c
index 73c8dc2..afbaa1c 100644
--- a/arch/powerpc/platforms/powernv/pci.c
+++ b/arch/powerpc/platforms/powernv/pci.c
@@ -380,7 +380,7 @@ static void pnv_pci_config_check_eeh(struct pci_dn *pdn)
 	 */
 	pe_no = pdn->pe_number;
 	if (pe_no == IODA_INVALID_PE) {
-		pe_no = phb->ioda.reserved_pe;
+		pe_no = phb->ioda.reserved_pe_idx;
 	}
 
 	/*
diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
index f2a1452..784882a 100644
--- a/arch/powerpc/platforms/powernv/pci.h
+++ b/arch/powerpc/platforms/powernv/pci.h
@@ -120,8 +120,8 @@ struct pnv_phb {
 
 	struct {
 		/* Global bridge info */
-		unsigned int		total_pe;
-		unsigned int		reserved_pe;
+		unsigned int		total_pe_num;
+		unsigned int		reserved_pe_idx;
 
 		/* 32-bit MMIO window */
 		unsigned int		m32_size;
-- 
2.1.0

[PATCH v9 06/26] powerpc/powernv: Data type unsigned int for PE number

[Gavin Shan]

This changes the data type of PE number from "int" to "unsigned int"
in order to match the fact PE number is never negative:

   * The number of PE to which the specified PCI device is attached.
   * The PE number map for SRIOV VFs.
   * The returned PE number from pnv_ioda_alloc_pe().
   * The returned PE number from pnv_ioda2_pick_m64_pe().

Suggested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/pci-bridge.h     | 6 +++---
 arch/powerpc/platforms/powernv/pci-ioda.c | 8 ++++----
 arch/powerpc/platforms/powernv/pci.c      | 2 +-
 arch/powerpc/platforms/powernv/pci.h      | 2 +-
 4 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/arch/powerpc/include/asm/pci-bridge.h b/arch/powerpc/include/asm/pci-bridge.h
index 023c8c8..220129f 100644
--- a/arch/powerpc/include/asm/pci-bridge.h
+++ b/arch/powerpc/include/asm/pci-bridge.h
@@ -209,14 +209,14 @@ struct pci_dn {
 #ifdef CONFIG_EEH
 	struct eeh_dev *edev;		/* eeh device */
 #endif
-#define IODA_INVALID_PE		(-1)
+#define IODA_INVALID_PE		0xFFFFFFFF
 #ifdef CONFIG_PPC_POWERNV
-	int	pe_number;
+	unsigned int pe_number;
 	int     vf_index;		/* VF index in the PF */
 #ifdef CONFIG_PCI_IOV
 	u16     vfs_expanded;		/* number of VFs IOV BAR expanded */
 	u16     num_vfs;		/* number of VFs enabled*/
-	int     *pe_num_map;		/* PE# for the first VF PE or array */
+	unsigned int *pe_num_map;	/* PE# for the first VF PE or array */
 	bool    m64_single_mode;	/* Use M64 BAR in Single Mode */
 #define IODA_INVALID_M64        (-1)
 	int     (*m64_map)[PCI_SRIOV_NUM_BARS];
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 1d2514f..4aa6cdf 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -138,7 +138,7 @@ static void pnv_ioda_reserve_pe(struct pnv_phb *phb, int pe_no)
 	phb->ioda.pe_array[pe_no].pe_number = pe_no;
 }
 
-static int pnv_ioda_alloc_pe(struct pnv_phb *phb)
+static unsigned int pnv_ioda_alloc_pe(struct pnv_phb *phb)
 {
 	unsigned long pe;
 
@@ -261,7 +261,7 @@ static void pnv_ioda2_reserve_m64_pe(struct pci_bus *bus,
 	}
 }
 
-static int pnv_ioda2_pick_m64_pe(struct pci_bus *bus, bool all)
+static unsigned int pnv_ioda2_pick_m64_pe(struct pci_bus *bus, bool all)
 {
 	struct pci_controller *hose = pci_bus_to_host(bus);
 	struct pnv_phb *phb = hose->private_data;
@@ -919,7 +919,7 @@ static struct pnv_ioda_pe *pnv_ioda_setup_dev_PE(struct pci_dev *dev)
 	struct pnv_phb *phb = hose->private_data;
 	struct pci_dn *pdn = pci_get_pdn(dev);
 	struct pnv_ioda_pe *pe;
-	int pe_num;
+	unsigned int pe_num;
 
 	if (!pdn) {
 		pr_err("%s: Device tree node not associated properly\n",
@@ -1010,7 +1010,7 @@ static void pnv_ioda_setup_bus_PE(struct pci_bus *bus, bool all)
 	struct pci_controller *hose = pci_bus_to_host(bus);
 	struct pnv_phb *phb = hose->private_data;
 	struct pnv_ioda_pe *pe;
-	int pe_num = IODA_INVALID_PE;
+	unsigned int pe_num = IODA_INVALID_PE;
 
 	/* Check if PE is determined by M64 */
 	if (phb->pick_m64_pe)
diff --git a/arch/powerpc/platforms/powernv/pci.c b/arch/powerpc/platforms/powernv/pci.c
index afbaa1c..8827461 100644
--- a/arch/powerpc/platforms/powernv/pci.c
+++ b/arch/powerpc/platforms/powernv/pci.c
@@ -370,7 +370,7 @@ static void pnv_pci_config_check_eeh(struct pci_dn *pdn)
 	struct pnv_phb *phb = pdn->phb->private_data;
 	u8	fstate;
 	__be16	pcierr;
-	int	pe_no;
+	unsigned int pe_no;
 	s64	rc;
 
 	/*
diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
index 784882a..66f2569 100644
--- a/arch/powerpc/platforms/powernv/pci.h
+++ b/arch/powerpc/platforms/powernv/pci.h
@@ -113,7 +113,7 @@ struct pnv_phb {
 	int (*init_m64)(struct pnv_phb *phb);
 	void (*reserve_m64_pe)(struct pci_bus *bus,
 			       unsigned long *pe_bitmap, bool all);
-	int (*pick_m64_pe)(struct pci_bus *bus, bool all);
+	unsigned int (*pick_m64_pe)(struct pci_bus *bus, bool all);
 	int (*get_pe_state)(struct pnv_phb *phb, int pe_no);
 	void (*freeze_pe)(struct pnv_phb *phb, int pe_no);
 	int (*unfreeze_pe)(struct pnv_phb *phb, int pe_no, int opt);
-- 
2.1.0

[PATCH v9 07/26] powerpc/powernv: Fix initial IO and M32 segmap

[Gavin Shan]

There are two arrays for IO and M32 segment maps on every PHB.
The index of the arrays are segment number and the value stored
in the corresponding element is PE number, indicating the segment
is assigned to the PE. Initially, all elements in those two arrays
are zeroes, meaning all segments are assigned to PE#0. It's wrong.

This fixes the initial values in the elements of those two arrays
to IODA_INVALID_PE, meaning all segments aren't assigned to any
PE.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 4aa6cdf..59b20e5 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -3240,6 +3240,7 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 	const __be64 *prop64;
 	const __be32 *prop32;
 	int len;
+	unsigned int segno;
 	u64 phb_id;
 	void *aux;
 	long rc;
@@ -3334,8 +3335,13 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 	aux = memblock_virt_alloc(size, 0);
 	phb->ioda.pe_alloc = aux;
 	phb->ioda.m32_segmap = aux + m32map_off;
-	if (phb->type == PNV_PHB_IODA1)
+	for (segno = 0; segno < phb->ioda.total_pe_num; segno++)
+		phb->ioda.m32_segmap[segno] = IODA_INVALID_PE;
+	if (phb->type == PNV_PHB_IODA1) {
 		phb->ioda.io_segmap = aux + iomap_off;
+		for (segno = 0; segno < phb->ioda.total_pe_num; segno++)
+			phb->ioda.io_segmap[segno] = IODA_INVALID_PE;
+	}
 	phb->ioda.pe_array = aux + pemap_off;
 	set_bit(phb->ioda.reserved_pe_idx, phb->ioda.pe_alloc);
 
-- 
2.1.0

[PATCH v9 08/26] powerpc/powernv: Simplify pnv_ioda_setup_pe_seg()

[Gavin Shan]

pnv_ioda_setup_pe_seg() associates the IO and M32 segments with the
owner PE. The code mapping segments should be fixed and immune from
logic changes introduced to pnv_ioda_setup_pe_seg().

This moves the code mapping segments to helper pnv_ioda_setup_pe_res().
The data type for @rc is changed to "int64_t". Also, argument @hose is
removed from pnv_ioda_setup_pe() as it can be got from @pe. No functional
changes introduced.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 121 +++++++++++++++---------------
 1 file changed, 62 insertions(+), 59 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 59b20e5..b954fbc 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -2929,19 +2929,72 @@ truncate_iov:
 }
 #endif /* CONFIG_PCI_IOV */
 
+static void pnv_ioda_setup_pe_res(struct pnv_ioda_pe *pe,
+				  struct resource *res)
+{
+	struct pnv_phb *phb = pe->phb;
+	struct pci_bus_region region;
+	int index;
+	int64_t rc;
+
+	if (!res || !res->flags || res->start > res->end)
+		return;
+
+	if (res->flags & IORESOURCE_IO) {
+		region.start = res->start - phb->ioda.io_pci_base;
+		region.end   = res->end - phb->ioda.io_pci_base;
+		index = region.start / phb->ioda.io_segsize;
+
+		while (index < phb->ioda.total_pe_num &&
+		       region.start <= region.end) {
+			phb->ioda.io_segmap[index] = pe->pe_number;
+			rc = opal_pci_map_pe_mmio_window(phb->opal_id,
+				pe->pe_number, OPAL_IO_WINDOW_TYPE, 0, index);
+			if (rc != OPAL_SUCCESS) {
+				pr_err("%s: Error %lld mapping IO segment#%d to PE#%d\n",
+				       __func__, rc, index, pe->pe_number);
+				break;
+			}
+
+			region.start += phb->ioda.io_segsize;
+			index++;
+		}
+	} else if ((res->flags & IORESOURCE_MEM) &&
+		   !pnv_pci_is_mem_pref_64(res->flags)) {
+		region.start = res->start -
+			       phb->hose->mem_offset[0] -
+			       phb->ioda.m32_pci_base;
+		region.end   = res->end -
+			       phb->hose->mem_offset[0] -
+			       phb->ioda.m32_pci_base;
+		index = region.start / phb->ioda.m32_segsize;
+
+		while (index < phb->ioda.total_pe_num &&
+		       region.start <= region.end) {
+			phb->ioda.m32_segmap[index] = pe->pe_number;
+			rc = opal_pci_map_pe_mmio_window(phb->opal_id,
+				pe->pe_number, OPAL_M32_WINDOW_TYPE, 0, index);
+			if (rc != OPAL_SUCCESS) {
+				pr_err("%s: Error %lld mapping M32 segment#%d to PE#%d",
+				       __func__, rc, index, pe->pe_number);
+				break;
+			}
+
+			region.start += phb->ioda.m32_segsize;
+			index++;
+		}
+	}
+}
+
 /*
  * This function is supposed to be called on basis of PE from top
  * to bottom style. So the the I/O or MMIO segment assigned to
  * parent PE could be overrided by its child PEs if necessary.
  */
-static void pnv_ioda_setup_pe_seg(struct pci_controller *hose,
-				  struct pnv_ioda_pe *pe)
+static void pnv_ioda_setup_pe_seg(struct pnv_ioda_pe *pe)
 {
-	struct pnv_phb *phb = hose->private_data;
-	struct pci_bus_region region;
 	struct resource *res;
-	int i, index;
-	int rc;
+	int i;
 
 	/*
 	 * NOTE: We only care PCI bus based PE for now. For PCI
@@ -2950,58 +3003,8 @@ static void pnv_ioda_setup_pe_seg(struct pci_controller *hose,
 	 */
 	BUG_ON(!(pe->flags & (PNV_IODA_PE_BUS | PNV_IODA_PE_BUS_ALL)));
 
-	pci_bus_for_each_resource(pe->pbus, res, i) {
-		if (!res || !res->flags ||
-		    res->start > res->end)
-			continue;
-
-		if (res->flags & IORESOURCE_IO) {
-			region.start = res->start - phb->ioda.io_pci_base;
-			region.end   = res->end - phb->ioda.io_pci_base;
-			index = region.start / phb->ioda.io_segsize;
-
-			while (index < phb->ioda.total_pe_num &&
-			       region.start <= region.end) {
-				phb->ioda.io_segmap[index] = pe->pe_number;
-				rc = opal_pci_map_pe_mmio_window(phb->opal_id,
-					pe->pe_number, OPAL_IO_WINDOW_TYPE, 0, index);
-				if (rc != OPAL_SUCCESS) {
-					pr_err("%s: OPAL error %d when mapping IO "
-					       "segment #%d to PE#%d\n",
-					       __func__, rc, index, pe->pe_number);
-					break;
-				}
-
-				region.start += phb->ioda.io_segsize;
-				index++;
-			}
-		} else if ((res->flags & IORESOURCE_MEM) &&
-			   !pnv_pci_is_mem_pref_64(res->flags)) {
-			region.start = res->start -
-				       hose->mem_offset[0] -
-				       phb->ioda.m32_pci_base;
-			region.end   = res->end -
-				       hose->mem_offset[0] -
-				       phb->ioda.m32_pci_base;
-			index = region.start / phb->ioda.m32_segsize;
-
-			while (index < phb->ioda.total_pe_num &&
-			       region.start <= region.end) {
-				phb->ioda.m32_segmap[index] = pe->pe_number;
-				rc = opal_pci_map_pe_mmio_window(phb->opal_id,
-					pe->pe_number, OPAL_M32_WINDOW_TYPE, 0, index);
-				if (rc != OPAL_SUCCESS) {
-					pr_err("%s: OPAL error %d when mapping M32 "
-					       "segment#%d to PE#%d",
-					       __func__, rc, index, pe->pe_number);
-					break;
-				}
-
-				region.start += phb->ioda.m32_segsize;
-				index++;
-			}
-		}
-	}
+	pci_bus_for_each_resource(pe->pbus, res, i)
+		pnv_ioda_setup_pe_res(pe, res);
 }
 
 static void pnv_pci_ioda_setup_seg(void)
@@ -3018,7 +3021,7 @@ static void pnv_pci_ioda_setup_seg(void)
 			continue;
 
 		list_for_each_entry(pe, &phb->ioda.pe_list, list) {
-			pnv_ioda_setup_pe_seg(hose, pe);
+			pnv_ioda_setup_pe_seg(pe);
 		}
 	}
 }
-- 
2.1.0

[PATCH v9 09/26] powerpc/powernv: IO and M32 mapping based on PCI device resources

[Gavin Shan]

Currently, the IO and M32 segments are mapped to the corresponding
PE based on the windows of the parent bridge of PE's primary bus.
It's not going to work when the windows of root port or upstream
port of the PCIe switch behind root port are extended to PHB's
apertures in order to support hotplug in subsequent patch.

This fixes the issue by mapping IO and M32 segments based on the
resources of the PCI devices included in the PE, instead of the
windows of the parent bridge of the PE's primary bus.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index b954fbc..904790b 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -2993,7 +2993,7 @@ static void pnv_ioda_setup_pe_res(struct pnv_ioda_pe *pe,
  */
 static void pnv_ioda_setup_pe_seg(struct pnv_ioda_pe *pe)
 {
-	struct resource *res;
+	struct pci_dev *pdev;
 	int i;
 
 	/*
@@ -3003,8 +3003,21 @@ static void pnv_ioda_setup_pe_seg(struct pnv_ioda_pe *pe)
 	 */
 	BUG_ON(!(pe->flags & (PNV_IODA_PE_BUS | PNV_IODA_PE_BUS_ALL)));
 
-	pci_bus_for_each_resource(pe->pbus, res, i)
-		pnv_ioda_setup_pe_res(pe, res);
+	list_for_each_entry(pdev, &pe->pbus->devices, bus_list) {
+		for (i = 0; i <= PCI_ROM_RESOURCE; i++)
+			pnv_ioda_setup_pe_res(pe, &pdev->resource[i]);
+
+		/*
+		 * If the PE contains all subordinate PCI buses, the
+		 * windows of the child bridges should be mapped to
+		 * the PE as well.
+		 */
+		if (!(pe->flags & PNV_IODA_PE_BUS_ALL) || !pci_is_bridge(pdev))
+			continue;
+		for (i = 0; i < PCI_BRIDGE_RESOURCE_NUM; i++)
+			pnv_ioda_setup_pe_res(pe,
+				&pdev->resource[PCI_BRIDGE_RESOURCES + i]);
+	}
 }
 
 static void pnv_pci_ioda_setup_seg(void)
-- 
2.1.0

[PATCH v9 10/26] powerpc/powernv: Track M64 segment consumption

[Gavin Shan]

When unplugging PCI devices, their parent PEs might be offline.
The consumed M64 resource by the PEs should be released at that
time. As we track M32 segment consumption, this introduces an
array to the PHB to track the mapping between M64 segment and
PE number.

Note: M64 mapping isn't covered by pnv_ioda_setup_pe_seg() as
IODA2 doesn't support the mapping explicitly while it's supported
on IODA1. Until now, no M64 is supported on IODA1 in software.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 10 ++++++++--
 arch/powerpc/platforms/powernv/pci.h      |  1 +
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 904790b..832b430 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -305,6 +305,7 @@ static unsigned int pnv_ioda2_pick_m64_pe(struct pci_bus *bus, bool all)
 		phb->ioda.total_pe_num) {
 		pe = &phb->ioda.pe_array[i];
 
+		phb->ioda.m64_segmap[pe->pe_number] = pe->pe_number;
 		if (!master_pe) {
 			pe->flags |= PNV_IODA_PE_MASTER;
 			INIT_LIST_HEAD(&pe->slaves);
@@ -3252,7 +3253,7 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 {
 	struct pci_controller *hose;
 	struct pnv_phb *phb;
-	unsigned long size, m32map_off, pemap_off, iomap_off = 0;
+	unsigned long size, m64map_off, m32map_off, pemap_off, iomap_off = 0;
 	const __be64 *prop64;
 	const __be32 *prop32;
 	int len;
@@ -3340,6 +3341,8 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 
 	/* Allocate aux data & arrays. We don't have IO ports on PHB3 */
 	size = _ALIGN_UP(phb->ioda.total_pe_num / 8, sizeof(unsigned long));
+	m64map_off = size;
+	size += phb->ioda.total_pe_num * sizeof(phb->ioda.m64_segmap[0]);
 	m32map_off = size;
 	size += phb->ioda.total_pe_num * sizeof(phb->ioda.m32_segmap[0]);
 	if (phb->type == PNV_PHB_IODA1) {
@@ -3350,9 +3353,12 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 	size += phb->ioda.total_pe_num * sizeof(struct pnv_ioda_pe);
 	aux = memblock_virt_alloc(size, 0);
 	phb->ioda.pe_alloc = aux;
+	phb->ioda.m64_segmap = aux + m64map_off;
 	phb->ioda.m32_segmap = aux + m32map_off;
-	for (segno = 0; segno < phb->ioda.total_pe_num; segno++)
+	for (segno = 0; segno < phb->ioda.total_pe_num; segno++) {
+		phb->ioda.m64_segmap[segno] = IODA_INVALID_PE;
 		phb->ioda.m32_segmap[segno] = IODA_INVALID_PE;
+	}
 	if (phb->type == PNV_PHB_IODA1) {
 		phb->ioda.io_segmap = aux + iomap_off;
 		for (segno = 0; segno < phb->ioda.total_pe_num; segno++)
diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
index 66f2569..a409165 100644
--- a/arch/powerpc/platforms/powernv/pci.h
+++ b/arch/powerpc/platforms/powernv/pci.h
@@ -146,6 +146,7 @@ struct pnv_phb {
 		struct pnv_ioda_pe	*pe_array;
 
 		/* M32 & IO segment maps */
+		unsigned int		*m64_segmap;
 		unsigned int		*m32_segmap;
 		unsigned int		*io_segmap;
 
-- 
2.1.0

[PATCH v9 11/26] powerpc/powernv: Rename M64 related functions

[Gavin Shan]

This renames those functions picking PE number based on consumed
M64 segments, mapping M64 segments to PEs as those functions are
going to be shared by IODA1/IODA2 in next patch. No logical changes
introduced.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 832b430..37f22b0 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -219,7 +219,7 @@ fail:
 	return -EIO;
 }
 
-static void pnv_ioda2_reserve_dev_m64_pe(struct pci_dev *pdev,
+static void pnv_ioda_reserve_dev_m64_pe(struct pci_dev *pdev,
 					 unsigned long *pe_bitmap)
 {
 	struct pci_controller *hose = pci_bus_to_host(pdev->bus);
@@ -246,22 +246,22 @@ static void pnv_ioda2_reserve_dev_m64_pe(struct pci_dev *pdev,
 	}
 }
 
-static void pnv_ioda2_reserve_m64_pe(struct pci_bus *bus,
-				     unsigned long *pe_bitmap,
-				     bool all)
+static void pnv_ioda_reserve_m64_pe(struct pci_bus *bus,
+				    unsigned long *pe_bitmap,
+				    bool all)
 {
 	struct pci_dev *pdev;
 
 	list_for_each_entry(pdev, &bus->devices, bus_list) {
-		pnv_ioda2_reserve_dev_m64_pe(pdev, pe_bitmap);
+		pnv_ioda_reserve_dev_m64_pe(pdev, pe_bitmap);
 
 		if (all && pdev->subordinate)
-			pnv_ioda2_reserve_m64_pe(pdev->subordinate,
-						 pe_bitmap, all);
+			pnv_ioda_reserve_m64_pe(pdev->subordinate,
+						pe_bitmap, all);
 	}
 }
 
-static unsigned int pnv_ioda2_pick_m64_pe(struct pci_bus *bus, bool all)
+static unsigned int pnv_ioda_pick_m64_pe(struct pci_bus *bus, bool all)
 {
 	struct pci_controller *hose = pci_bus_to_host(bus);
 	struct pnv_phb *phb = hose->private_data;
@@ -283,7 +283,7 @@ static unsigned int pnv_ioda2_pick_m64_pe(struct pci_bus *bus, bool all)
 	}
 
 	/* Figure out reserved PE numbers by the PE */
-	pnv_ioda2_reserve_m64_pe(bus, pe_alloc, all);
+	pnv_ioda_reserve_m64_pe(bus, pe_alloc, all);
 
 	/*
 	 * the current bus might not own M64 window and that's all
@@ -365,8 +365,8 @@ static void __init pnv_ioda_parse_m64_window(struct pnv_phb *phb)
 	/* Use last M64 BAR to cover M64 window */
 	phb->ioda.m64_bar_idx = 15;
 	phb->init_m64 = pnv_ioda2_init_m64;
-	phb->reserve_m64_pe = pnv_ioda2_reserve_m64_pe;
-	phb->pick_m64_pe = pnv_ioda2_pick_m64_pe;
+	phb->reserve_m64_pe = pnv_ioda_reserve_m64_pe;
+	phb->pick_m64_pe = pnv_ioda_pick_m64_pe;
 }
 
 static void pnv_ioda_freeze_pe(struct pnv_phb *phb, int pe_no)
-- 
2.1.0

[PATCH v9 12/26] powerpc/powernv/ioda1: M64 support on P7IOC

[Gavin Shan]

This enables M64 window on P7IOC, which has been enabled on PHB3.
Different from PHB3 where 16 M64 BARs are supported and each of
them can be owned by one particular PE# exclusively or divided
evenly to 256 segments, every P7IOC PHB has 16 M64 BARs and each
of them are divided to 8 segments. So every P7IOC PHB supports
128 M64 segments in total. P7IOC has M64DT, which helps mapping
one particular M64 segment# to arbitrary PE#. PHB3 doesn't have
M64DT, indicating that one M64 segment can only be pinned to the
fixed PE#.

In order to unified M64 support M64 on P7IOC and PHB3, we just
provide 128 M64 segments on every P7IOC PHB and each of them is
pinned to the fixed PE# by bypassing the function of M64DT. In
turn, we just need different phb->init_m64() for P7IOC and PHB3
and maps M64 segment in pnv_ioda_reserve_m64_pe() for P7IOC, most
of the code are shared by them.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 89 +++++++++++++++++++++++++++++--
 1 file changed, 86 insertions(+), 3 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 37f22b0..a1b74ec 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -48,6 +48,9 @@
 #include "powernv.h"
 #include "pci.h"
 
+#define PNV_IODA1_M64_NUM	16	/* Number of M64 BARs	*/
+#define PNV_IODA1_M64_SEGS	8	/* Segments per M64 BAR	*/
+
 /* 256M DMA window, 4K TCE pages, 8 bytes TCE */
 #define TCE32_TABLE_SIZE	((0x10000000 / 0x1000) * 8)
 
@@ -246,6 +249,64 @@ static void pnv_ioda_reserve_dev_m64_pe(struct pci_dev *pdev,
 	}
 }
 
+static int pnv_ioda1_init_m64(struct pnv_phb *phb)
+{
+	struct resource *r;
+	int index;
+
+	/*
+	 * There are 16 M64 BARs, each of which has 8 segments. So
+	 * there are as many M64 segments as the maximum number of
+	 * PEs, which is 128.
+	 */
+	for (index = 0; index < PNV_IODA1_M64_NUM; index++) {
+		unsigned long base, segsz = phb->ioda.m64_segsize;
+		int64_t rc;
+
+		base = phb->ioda.m64_base +
+		       index * PNV_IODA1_M64_SEGS * segsz;
+		rc = opal_pci_set_phb_mem_window(phb->opal_id,
+				OPAL_M64_WINDOW_TYPE, index, base, 0,
+				PNV_IODA1_M64_SEGS * segsz);
+		if (rc != OPAL_SUCCESS) {
+			pr_warn("  Error %lld setting M64 PHB#%d-BAR#%d\n",
+				rc, phb->hose->global_number, index);
+			goto fail;
+		}
+
+		rc = opal_pci_phb_mmio_enable(phb->opal_id,
+				OPAL_M64_WINDOW_TYPE, index,
+				OPAL_ENABLE_M64_SPLIT);
+		if (rc != OPAL_SUCCESS) {
+			pr_warn("  Error %lld enabling M64 PHB#%d-BAR#%d\n",
+				rc, phb->hose->global_number, index);
+			goto fail;
+		}
+	}
+
+	/*
+	 * Exclude the segment used by the reserved PE, which
+	 * is expected to be 0 or last supported PE#.
+	 */
+	r = &phb->hose->mem_resources[1];
+	if (phb->ioda.reserved_pe_idx == 0)
+		r->start += phb->ioda.m64_segsize;
+	else if (phb->ioda.reserved_pe_idx == (phb->ioda.total_pe_num - 1))
+		r->end -= phb->ioda.m64_segsize;
+	else
+		pr_warn("  Cannot cut M64 segment for reserved PE#%d\n",
+			phb->ioda.reserved_pe_idx);
+
+	return 0;
+
+fail:
+	for ( ; index >= 0; index--)
+		opal_pci_phb_mmio_enable(phb->opal_id,
+			OPAL_M64_WINDOW_TYPE, index, OPAL_DISABLE_M64);
+
+	return -EIO;
+}
+
 static void pnv_ioda_reserve_m64_pe(struct pci_bus *bus,
 				    unsigned long *pe_bitmap,
 				    bool all)
@@ -315,6 +376,26 @@ static unsigned int pnv_ioda_pick_m64_pe(struct pci_bus *bus, bool all)
 			pe->master = master_pe;
 			list_add_tail(&pe->list, &master_pe->slaves);
 		}
+
+		/*
+		 * P7IOC supports M64DT, which helps mapping M64 segment
+		 * to one particular PE#. However, PHB3 has fixed mapping
+		 * between M64 segment and PE#. In order to have same logic
+		 * for P7IOC and PHB3, we enforce fixed mapping between M64
+		 * segment and PE# on P7IOC.
+		 */
+		if (phb->type == PNV_PHB_IODA1) {
+			int64_t rc;
+
+			rc = opal_pci_map_pe_mmio_window(phb->opal_id,
+					pe->pe_number, OPAL_M64_WINDOW_TYPE,
+					pe->pe_number / PNV_IODA1_M64_SEGS,
+					pe->pe_number % PNV_IODA1_M64_SEGS);
+			if (rc != OPAL_SUCCESS)
+				pr_warn("%s: Error %lld mapping M64 for PHB#%d-PE#%d\n",
+					__func__, rc, phb->hose->global_number,
+					pe->pe_number);
+		}
 	}
 
 	kfree(pe_alloc);
@@ -329,8 +410,7 @@ static void __init pnv_ioda_parse_m64_window(struct pnv_phb *phb)
 	const u32 *r;
 	u64 pci_addr;
 
-	/* FIXME: Support M64 for P7IOC */
-	if (phb->type != PNV_PHB_IODA2) {
+	if (phb->type != PNV_PHB_IODA1 && phb->type != PNV_PHB_IODA2) {
 		pr_info("  Not support M64 window\n");
 		return;
 	}
@@ -364,7 +444,10 @@ static void __init pnv_ioda_parse_m64_window(struct pnv_phb *phb)
 
 	/* Use last M64 BAR to cover M64 window */
 	phb->ioda.m64_bar_idx = 15;
-	phb->init_m64 = pnv_ioda2_init_m64;
+	if (phb->type == PNV_PHB_IODA1)
+		phb->init_m64 = pnv_ioda1_init_m64;
+	else
+		phb->init_m64 = pnv_ioda2_init_m64;
 	phb->reserve_m64_pe = pnv_ioda_reserve_m64_pe;
 	phb->pick_m64_pe = pnv_ioda_pick_m64_pe;
 }
-- 
2.1.0

[PATCH v9 13/26] powerpc/powernv/ioda1: Rename pnv_pci_ioda_setup_dma_pe()

[Gavin Shan]

This renames pnv_pci_ioda_setup_dma_pe() to pnv_pci_ioda1_setup_dma_pe()
as it's the counter-part of IODA2's pnv_pci_ioda2_setup_dma_pe().
No logical changes introduced.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index a1b74ec..f27a62d 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -2029,9 +2029,10 @@ static struct iommu_table_ops pnv_ioda2_iommu_ops = {
 	.free = pnv_ioda2_table_free,
 };
 
-static void pnv_pci_ioda_setup_dma_pe(struct pnv_phb *phb,
-				      struct pnv_ioda_pe *pe, unsigned int base,
-				      unsigned int segs)
+static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
+				       struct pnv_ioda_pe *pe,
+				       unsigned int base,
+				       unsigned int segs)
 {
 
 	struct page *tce_mem = NULL;
@@ -2619,7 +2620,7 @@ static void pnv_ioda_setup_dma(struct pnv_phb *phb)
 		if (phb->type == PNV_PHB_IODA1) {
 			pe_info(pe, "DMA weight %d, assigned %d DMA32 segments\n",
 				pe->dma_weight, segs);
-			pnv_pci_ioda_setup_dma_pe(phb, pe, base, segs);
+			pnv_pci_ioda1_setup_dma_pe(phb, pe, base, segs);
 		} else if (phb->type == PNV_PHB_IODA2) {
 			pe_info(pe, "Assign DMA32 space\n");
 			segs = 0;
-- 
2.1.0

[PATCH v9 14/26] powerpc/powernv/ioda1: Introduce PNV_IODA1_DMA32_SEGSIZE

[Gavin Shan]

Currently, there is one macro (TCE32_TABLE_SIZE) representing the
TCE table size for one DMA32 segment. The constant representing
the DMA32 segment size (1 << 28) is still used in the code.

This defines PNV_IODA1_DMA32_SEGSIZE representing one DMA32
segment size. the TCE table size can be calcualted when the page
has fixed 4KB size. So all the related calculation depends on one
macro (PNV_IODA1_DMA32_SEGSIZE). No logical changes introduced.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 31 ++++++++++++++++++-------------
 1 file changed, 18 insertions(+), 13 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index f27a62d..ca034b8 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -50,9 +50,7 @@
 
 #define PNV_IODA1_M64_NUM	16	/* Number of M64 BARs	*/
 #define PNV_IODA1_M64_SEGS	8	/* Segments per M64 BAR	*/
-
-/* 256M DMA window, 4K TCE pages, 8 bytes TCE */
-#define TCE32_TABLE_SIZE	((0x10000000 / 0x1000) * 8)
+#define PNV_IODA1_DMA32_SEGSIZE	0x10000000
 
 #define POWERNV_IOMMU_DEFAULT_LEVELS	1
 #define POWERNV_IOMMU_MAX_LEVELS	5
@@ -2037,7 +2035,7 @@ static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
 
 	struct page *tce_mem = NULL;
 	struct iommu_table *tbl;
-	unsigned int i;
+	unsigned int tce32_segsz, i;
 	int64_t rc;
 	void *addr;
 
@@ -2057,29 +2055,34 @@ static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
 	/* Grab a 32-bit TCE table */
 	pe->tce32_seg = base;
 	pe_info(pe, " Setting up 32-bit TCE table at %08x..%08x\n",
-		(base << 28), ((base + segs) << 28) - 1);
+		base * PNV_IODA1_DMA32_SEGSIZE,
+		(base + segs) * PNV_IODA1_DMA32_SEGSIZE - 1);
 
 	/* XXX Currently, we allocate one big contiguous table for the
 	 * TCEs. We only really need one chunk per 256M of TCE space
 	 * (ie per segment) but that's an optimization for later, it
 	 * requires some added smarts with our get/put_tce implementation
+	 *
+	 * Each TCE page is 4KB in size and each TCE entry occupies 8
+	 * bytes
 	 */
+	tce32_segsz = PNV_IODA1_DMA32_SEGSIZE >> (IOMMU_PAGE_SHIFT_4K - 3);
 	tce_mem = alloc_pages_node(phb->hose->node, GFP_KERNEL,
-				   get_order(TCE32_TABLE_SIZE * segs));
+				   get_order(tce32_segsz * segs));
 	if (!tce_mem) {
 		pe_err(pe, " Failed to allocate a 32-bit TCE memory\n");
 		goto fail;
 	}
 	addr = page_address(tce_mem);
-	memset(addr, 0, TCE32_TABLE_SIZE * segs);
+	memset(addr, 0, tce32_segsz * segs);
 
 	/* Configure HW */
 	for (i = 0; i < segs; i++) {
 		rc = opal_pci_map_pe_dma_window(phb->opal_id,
 					      pe->pe_number,
 					      base + i, 1,
-					      __pa(addr) + TCE32_TABLE_SIZE * i,
-					      TCE32_TABLE_SIZE, 0x1000);
+					      __pa(addr) + tce32_segsz * i,
+					      tce32_segsz, IOMMU_PAGE_SIZE_4K);
 		if (rc) {
 			pe_err(pe, " Failed to configure 32-bit TCE table,"
 			       " err %ld\n", rc);
@@ -2088,8 +2091,9 @@ static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
 	}
 
 	/* Setup linux iommu table */
-	pnv_pci_setup_iommu_table(tbl, addr, TCE32_TABLE_SIZE * segs,
-				  base << 28, IOMMU_PAGE_SHIFT_4K);
+	pnv_pci_setup_iommu_table(tbl, addr, tce32_segsz * segs,
+				  base * PNV_IODA1_DMA32_SEGSIZE,
+				  IOMMU_PAGE_SHIFT_4K);
 
 	/* OPAL variant of P7IOC SW invalidated TCEs */
 	if (phb->ioda.tce_inval_reg)
@@ -2119,7 +2123,7 @@ static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
 	if (pe->tce32_seg >= 0)
 		pe->tce32_seg = -1;
 	if (tce_mem)
-		__free_pages(tce_mem, get_order(TCE32_TABLE_SIZE * segs));
+		__free_pages(tce_mem, get_order(tce32_segsz * segs));
 	if (tbl) {
 		pnv_pci_unlink_table_and_group(tbl, &pe->table_group);
 		iommu_free_table(tbl, "pnv");
@@ -3456,7 +3460,8 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 	mutex_init(&phb->ioda.pe_list_mutex);
 
 	/* Calculate how many 32-bit TCE segments we have */
-	phb->ioda.tce32_count = phb->ioda.m32_pci_base >> 28;
+	phb->ioda.tce32_count = phb->ioda.m32_pci_base /
+				PNV_IODA1_DMA32_SEGSIZE;
 
 #if 0 /* We should really do that ... */
 	rc = opal_pci_set_phb_mem_window(opal->phb_id,
-- 
2.1.0

[PATCH v9 15/26] powerpc/powernv: Remove DMA32 PE list

[Gavin Shan]

PEs are put into PHB DMA32 list (phb->ioda.pe_dma_list) according
to their DMA32 weight. The PEs on the list are iterated to setup
their TCE32 tables at system booting time. The list is used for
once at boot time and no need to keep it.

This moves the logic calculating DMA32 weight of PHB and PE to
pnv_ioda_setup_dma() to drop PHB's DMA32 list. Also, every PE
traces the consumed DMA32 segment by @tce32_seg and @tce32_segcount
are useless and they're removed.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 171 ++++++++++++++----------------
 arch/powerpc/platforms/powernv/pci.h      |  19 ----
 2 files changed, 78 insertions(+), 112 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index ca034b8..f70a4e0 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -890,44 +890,6 @@ out:
 	return 0;
 }
 
-static void pnv_ioda_link_pe_by_weight(struct pnv_phb *phb,
-				       struct pnv_ioda_pe *pe)
-{
-	struct pnv_ioda_pe *lpe;
-
-	list_for_each_entry(lpe, &phb->ioda.pe_dma_list, dma_link) {
-		if (lpe->dma_weight < pe->dma_weight) {
-			list_add_tail(&pe->dma_link, &lpe->dma_link);
-			return;
-		}
-	}
-	list_add_tail(&pe->dma_link, &phb->ioda.pe_dma_list);
-}
-
-static unsigned int pnv_ioda_dma_weight(struct pci_dev *dev)
-{
-	/* This is quite simplistic. The "base" weight of a device
-	 * is 10. 0 means no DMA is to be accounted for it.
-	 */
-
-	/* If it's a bridge, no DMA */
-	if (dev->hdr_type != PCI_HEADER_TYPE_NORMAL)
-		return 0;
-
-	/* Reduce the weight of slow USB controllers */
-	if (dev->class == PCI_CLASS_SERIAL_USB_UHCI ||
-	    dev->class == PCI_CLASS_SERIAL_USB_OHCI ||
-	    dev->class == PCI_CLASS_SERIAL_USB_EHCI)
-		return 3;
-
-	/* Increase the weight of RAID (includes Obsidian) */
-	if ((dev->class >> 8) == PCI_CLASS_STORAGE_RAID)
-		return 15;
-
-	/* Default */
-	return 10;
-}
-
 #ifdef CONFIG_PCI_IOV
 static int pnv_pci_vf_resource_shift(struct pci_dev *dev, int offset)
 {
@@ -1032,7 +994,6 @@ static struct pnv_ioda_pe *pnv_ioda_setup_dev_PE(struct pci_dev *dev)
 	pe->flags = PNV_IODA_PE_DEV;
 	pe->pdev = dev;
 	pe->pbus = NULL;
-	pe->tce32_seg = -1;
 	pe->mve_number = -1;
 	pe->rid = dev->bus->number << 8 | pdn->devfn;
 
@@ -1048,16 +1009,6 @@ static struct pnv_ioda_pe *pnv_ioda_setup_dev_PE(struct pci_dev *dev)
 		return NULL;
 	}
 
-	/* Assign a DMA weight to the device */
-	pe->dma_weight = pnv_ioda_dma_weight(dev);
-	if (pe->dma_weight != 0) {
-		phb->ioda.dma_weight += pe->dma_weight;
-		phb->ioda.dma_pe_count++;
-	}
-
-	/* Link the PE */
-	pnv_ioda_link_pe_by_weight(phb, pe);
-
 	return pe;
 }
 
@@ -1075,7 +1026,6 @@ static void pnv_ioda_setup_same_PE(struct pci_bus *bus, struct pnv_ioda_pe *pe)
 		}
 		pdn->pcidev = dev;
 		pdn->pe_number = pe->pe_number;
-		pe->dma_weight += pnv_ioda_dma_weight(dev);
 		if ((pe->flags & PNV_IODA_PE_BUS_ALL) && dev->subordinate)
 			pnv_ioda_setup_same_PE(dev->subordinate, pe);
 	}
@@ -1112,10 +1062,8 @@ static void pnv_ioda_setup_bus_PE(struct pci_bus *bus, bool all)
 	pe->flags |= (all ? PNV_IODA_PE_BUS_ALL : PNV_IODA_PE_BUS);
 	pe->pbus = bus;
 	pe->pdev = NULL;
-	pe->tce32_seg = -1;
 	pe->mve_number = -1;
 	pe->rid = bus->busn_res.start << 8;
-	pe->dma_weight = 0;
 
 	if (all)
 		pe_info(pe, "Secondary bus %d..%d associated with PE#%d\n",
@@ -1137,17 +1085,6 @@ static void pnv_ioda_setup_bus_PE(struct pci_bus *bus, bool all)
 
 	/* Put PE to the list */
 	list_add_tail(&pe->list, &phb->ioda.pe_list);
-
-	/* Account for one DMA PE if at least one DMA capable device exist
-	 * below the bridge
-	 */
-	if (pe->dma_weight != 0) {
-		phb->ioda.dma_weight += pe->dma_weight;
-		phb->ioda.dma_pe_count++;
-	}
-
-	/* Link the PE */
-	pnv_ioda_link_pe_by_weight(phb, pe);
 }
 
 static struct pnv_ioda_pe *pnv_ioda_setup_npu_PE(struct pci_dev *npu_pdev)
@@ -1188,7 +1125,6 @@ static struct pnv_ioda_pe *pnv_ioda_setup_npu_PE(struct pci_dev *npu_pdev)
 			rid = npu_pdev->bus->number << 8 | npu_pdn->devfn;
 			npu_pdn->pcidev = npu_pdev;
 			npu_pdn->pe_number = pe_num;
-			pe->dma_weight += pnv_ioda_dma_weight(npu_pdev);
 			phb->ioda.pe_rmap[rid] = pe->pe_number;
 
 			/* Map the PE to this link */
@@ -1536,7 +1472,6 @@ static void pnv_ioda_setup_vf_PE(struct pci_dev *pdev, u16 num_vfs)
 		pe->flags = PNV_IODA_PE_VF;
 		pe->pbus = NULL;
 		pe->parent_dev = pdev;
-		pe->tce32_seg = -1;
 		pe->mve_number = -1;
 		pe->rid = (pci_iov_virtfn_bus(pdev, vf_index) << 8) |
 			   pci_iov_virtfn_devfn(pdev, vf_index);
@@ -2027,6 +1962,54 @@ static struct iommu_table_ops pnv_ioda2_iommu_ops = {
 	.free = pnv_ioda2_table_free,
 };
 
+static int pnv_pci_ioda_dev_dma_weight(struct pci_dev *dev, void *data)
+{
+	unsigned int *weight = (unsigned int *)data;
+
+	/* This is quite simplistic. The "base" weight of a device
+	 * is 10. 0 means no DMA is to be accounted for it.
+	 */
+	if (dev->hdr_type != PCI_HEADER_TYPE_NORMAL)
+		return 0;
+
+	if (dev->class == PCI_CLASS_SERIAL_USB_UHCI ||
+	    dev->class == PCI_CLASS_SERIAL_USB_OHCI ||
+	    dev->class == PCI_CLASS_SERIAL_USB_EHCI)
+		*weight += 3;
+	else if ((dev->class >> 8) == PCI_CLASS_STORAGE_RAID)
+		*weight += 15;
+	else
+		*weight += 10;
+
+	return 0;
+}
+
+static unsigned int pnv_pci_ioda_pe_dma_weight(struct pnv_ioda_pe *pe)
+{
+	unsigned int weight = 0;
+
+	/* SRIOV VF has same DMA32 weight as its PF */
+#ifdef CONFIG_PCI_IOV
+	if ((pe->flags & PNV_IODA_PE_VF) && pe->parent_dev) {
+		pnv_pci_ioda_dev_dma_weight(pe->parent_dev, &weight);
+		return weight;
+	}
+#endif
+
+	if ((pe->flags & PNV_IODA_PE_DEV) && pe->pdev) {
+		pnv_pci_ioda_dev_dma_weight(pe->pdev, &weight);
+	} else if ((pe->flags & PNV_IODA_PE_BUS) && pe->pbus) {
+		struct pci_dev *pdev;
+
+		list_for_each_entry(pdev, &pe->pbus->devices, bus_list)
+			pnv_pci_ioda_dev_dma_weight(pdev, &weight);
+	} else if ((pe->flags & PNV_IODA_PE_BUS_ALL) && pe->pbus) {
+		pci_walk_bus(pe->pbus, pnv_pci_ioda_dev_dma_weight, &weight);
+	}
+
+	return weight;
+}
+
 static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
 				       struct pnv_ioda_pe *pe,
 				       unsigned int base,
@@ -2043,17 +2026,12 @@ static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
 	/* XXX FIXME: Provide 64-bit DMA facilities & non-4K TCE tables etc.. */
 	/* XXX FIXME: Allocate multi-level tables on PHB3 */
 
-	/* We shouldn't already have a 32-bit DMA associated */
-	if (WARN_ON(pe->tce32_seg >= 0))
-		return;
-
 	tbl = pnv_pci_table_alloc(phb->hose->node);
 	iommu_register_group(&pe->table_group, phb->hose->global_number,
 			pe->pe_number);
 	pnv_pci_link_table_and_group(phb->hose->node, 0, tbl, &pe->table_group);
 
 	/* Grab a 32-bit TCE table */
-	pe->tce32_seg = base;
 	pe_info(pe, " Setting up 32-bit TCE table at %08x..%08x\n",
 		base * PNV_IODA1_DMA32_SEGSIZE,
 		(base + segs) * PNV_IODA1_DMA32_SEGSIZE - 1);
@@ -2120,8 +2098,6 @@ static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
 	return;
  fail:
 	/* XXX Failure: Try to fallback to 64-bit only ? */
-	if (pe->tce32_seg >= 0)
-		pe->tce32_seg = -1;
 	if (tce_mem)
 		__free_pages(tce_mem, get_order(tce32_segsz * segs));
 	if (tbl) {
@@ -2532,10 +2508,6 @@ static void pnv_pci_ioda2_setup_dma_pe(struct pnv_phb *phb,
 {
 	int64_t rc;
 
-	/* We shouldn't already have a 32-bit DMA associated */
-	if (WARN_ON(pe->tce32_seg >= 0))
-		return;
-
 	/* TVE #1 is selected by PCI address bit 59 */
 	pe->tce_bypass_base = 1ull << 59;
 
@@ -2543,7 +2515,6 @@ static void pnv_pci_ioda2_setup_dma_pe(struct pnv_phb *phb,
 			pe->pe_number);
 
 	/* The PE will reserve all possible 32-bits space */
-	pe->tce32_seg = 0;
 	pe_info(pe, "Setting up 32-bit TCE table at 0..%08x\n",
 		phb->ioda.m32_pci_base);
 
@@ -2559,11 +2530,8 @@ static void pnv_pci_ioda2_setup_dma_pe(struct pnv_phb *phb,
 #endif
 
 	rc = pnv_pci_ioda2_setup_default_config(pe);
-	if (rc) {
-		if (pe->tce32_seg >= 0)
-			pe->tce32_seg = -1;
+	if (rc)
 		return;
-	}
 
 	if (pe->flags & PNV_IODA_PE_DEV)
 		iommu_add_device(&pe->pdev->dev);
@@ -2574,24 +2542,35 @@ static void pnv_pci_ioda2_setup_dma_pe(struct pnv_phb *phb,
 static void pnv_ioda_setup_dma(struct pnv_phb *phb)
 {
 	struct pci_controller *hose = phb->hose;
-	unsigned int residual, remaining, segs, tw, base;
+	unsigned int weight, total_weight, dma_pe_count;
+	unsigned int residual, remaining, segs, base;
 	struct pnv_ioda_pe *pe;
 
+	total_weight = 0;
+	pci_walk_bus(phb->hose->bus, pnv_pci_ioda_dev_dma_weight,
+		     &total_weight);
+
+	dma_pe_count = 0;
+	list_for_each_entry(pe, &phb->ioda.pe_list, list) {
+		weight = pnv_pci_ioda_pe_dma_weight(pe);
+		if (weight > 0)
+			dma_pe_count++;
+	}
+
 	/* If we have more PE# than segments available, hand out one
 	 * per PE until we run out and let the rest fail. If not,
 	 * then we assign at least one segment per PE, plus more based
 	 * on the amount of devices under that PE
 	 */
-	if (phb->ioda.dma_pe_count > phb->ioda.tce32_count)
+	if (dma_pe_count > phb->ioda.tce32_count)
 		residual = 0;
 	else
-		residual = phb->ioda.tce32_count -
-			phb->ioda.dma_pe_count;
+		residual = phb->ioda.tce32_count - dma_pe_count;
 
 	pr_info("PCI: Domain %04x has %ld available 32-bit DMA segments\n",
 		hose->global_number, phb->ioda.tce32_count);
 	pr_info("PCI: %d PE# for a total weight of %d\n",
-		phb->ioda.dma_pe_count, phb->ioda.dma_weight);
+		dma_pe_count, total_weight);
 
 	pnv_pci_ioda_setup_opal_tce_kill(phb);
 
@@ -2600,18 +2579,20 @@ static void pnv_ioda_setup_dma(struct pnv_phb *phb)
 	 * weight
 	 */
 	remaining = phb->ioda.tce32_count;
-	tw = phb->ioda.dma_weight;
 	base = 0;
-	list_for_each_entry(pe, &phb->ioda.pe_dma_list, dma_link) {
-		if (!pe->dma_weight)
+	list_for_each_entry(pe, &phb->ioda.pe_list, list) {
+		weight = pnv_pci_ioda_pe_dma_weight(pe);
+		if (!weight)
 			continue;
+
 		if (!remaining) {
 			pe_warn(pe, "No DMA32 resources available\n");
 			continue;
 		}
 		segs = 1;
 		if (residual) {
-			segs += ((pe->dma_weight * residual)  + (tw / 2)) / tw;
+			segs += ((weight * residual) + (total_weight / 2)) /
+				total_weight;
 			if (segs > remaining)
 				segs = remaining;
 		}
@@ -2623,7 +2604,7 @@ static void pnv_ioda_setup_dma(struct pnv_phb *phb)
 		 */
 		if (phb->type == PNV_PHB_IODA1) {
 			pe_info(pe, "DMA weight %d, assigned %d DMA32 segments\n",
-				pe->dma_weight, segs);
+				weight, segs);
 			pnv_pci_ioda1_setup_dma_pe(phb, pe, base, segs);
 		} else if (phb->type == PNV_PHB_IODA2) {
 			pe_info(pe, "Assign DMA32 space\n");
@@ -3167,13 +3148,18 @@ static void pnv_npu_ioda_fixup(void)
 	struct pci_controller *hose, *tmp;
 	struct pnv_phb *phb;
 	struct pnv_ioda_pe *pe;
+	unsigned int weight;
 
 	list_for_each_entry_safe(hose, tmp, &hose_list, list_node) {
 		phb = hose->private_data;
 		if (phb->type != PNV_PHB_NPU)
 			continue;
 
-		list_for_each_entry(pe, &phb->ioda.pe_dma_list, dma_link) {
+		list_for_each_entry(pe, &phb->ioda.pe_list, list) {
+			weight = pnv_pci_ioda_pe_dma_weight(pe);
+			if (WARN_ON(!weight))
+				continue;
+
 			enable_bypass = dma_get_mask(&pe->pdev->dev) ==
 				DMA_BIT_MASK(64);
 			pnv_npu_init_dma_pe(pe);
@@ -3455,7 +3441,6 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 	phb->ioda.pe_array = aux + pemap_off;
 	set_bit(phb->ioda.reserved_pe_idx, phb->ioda.pe_alloc);
 
-	INIT_LIST_HEAD(&phb->ioda.pe_dma_list);
 	INIT_LIST_HEAD(&phb->ioda.pe_list);
 	mutex_init(&phb->ioda.pe_list_mutex);
 
diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
index a409165..117cfcd 100644
--- a/arch/powerpc/platforms/powernv/pci.h
+++ b/arch/powerpc/platforms/powernv/pci.h
@@ -53,14 +53,7 @@ struct pnv_ioda_pe {
 	/* PE number */
 	unsigned int		pe_number;
 
-	/* "Weight" assigned to the PE for the sake of DMA resource
-	 * allocations
-	 */
-	unsigned int		dma_weight;
-
 	/* "Base" iommu table, ie, 4K TCEs, 32-bit DMA */
-	int			tce32_seg;
-	int			tce32_segcount;
 	struct iommu_table_group table_group;
 
 	/* 64-bit TCE bypass region */
@@ -78,7 +71,6 @@ struct pnv_ioda_pe {
 	struct list_head	slaves;
 
 	/* Link in list of PE#s */
-	struct list_head	dma_link;
 	struct list_head	list;
 };
 
@@ -169,17 +161,6 @@ struct pnv_phb {
 		/* 32-bit TCE tables allocation */
 		unsigned long		tce32_count;
 
-		/* Total "weight" for the sake of DMA resources
-		 * allocation
-		 */
-		unsigned int		dma_weight;
-		unsigned int		dma_pe_count;
-
-		/* Sorted list of used PE's, sorted at
-		 * boot for resource allocation purposes
-		 */
-		struct list_head	pe_dma_list;
-
 		/* TCE cache invalidate registers (physical and
 		 * remapped)
 		 */
-- 
2.1.0

[PATCH v9 16/26] powerpc/powernv/ioda1: Improve DMA32 segment track

[Gavin Shan]

In current implementation, the DMA32 segments required by one specific
PE isn't calculated with the information hold in the PE independently.
It conflicts with the PCI hotplug design: PE centralized, meaning the
PE's DMA32 segments should be calculated from the information hold in
the PE independently.

This introduces an array (@dma32_segmap) for every PHB to track the
DMA32 segmeng usage. Besides, this moves the logic calculating PE's
consumed DMA32 segments to pnv_pci_ioda1_setup_dma_pe() so that PE's
DMA32 segments are calculated/allocated from the information hold in
the PE (DMA32 weight). Also the logic is improved: we try to allocate
as much DMA32 segments as we can. It's acceptable that number of DMA32
segments less than the expected number are allocated.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 110 ++++++++++++++++--------------
 arch/powerpc/platforms/powernv/pci.h      |   7 +-
 2 files changed, 61 insertions(+), 56 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index f70a4e0..cfd2906 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -2011,27 +2011,57 @@ static unsigned int pnv_pci_ioda_pe_dma_weight(struct pnv_ioda_pe *pe)
 }
 
 static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
-				       struct pnv_ioda_pe *pe,
-				       unsigned int base,
-				       unsigned int segs)
+				       struct pnv_ioda_pe *pe)
 {
 
 	struct page *tce_mem = NULL;
 	struct iommu_table *tbl;
-	unsigned int tce32_segsz, i;
+	unsigned int weight, total_weight = 0;
+	unsigned int tce32_segsz, base, segs, i;
 	int64_t rc;
 	void *addr;
 
 	/* XXX FIXME: Handle 64-bit only DMA devices */
 	/* XXX FIXME: Provide 64-bit DMA facilities & non-4K TCE tables etc.. */
 	/* XXX FIXME: Allocate multi-level tables on PHB3 */
+	pci_walk_bus(phb->hose->bus, pnv_pci_ioda_dev_dma_weight,
+		     &total_weight);
+	weight = pnv_pci_ioda_pe_dma_weight(pe);
+
+	segs = (weight * phb->ioda.dma32_count) / total_weight;
+	if (!segs)
+		segs = 1;
+
+	/*
+	 * Allocate contiguous DMA32 segments. We begin with the expected
+	 * number of segments. With one more attempt, the number of DMA32
+	 * segments to be allocated is decreased by one until one segment
+	 * is allocated successfully.
+	 */
+	do {
+		for (base = 0; base <= phb->ioda.dma32_count - segs; base++) {
+			for (i = base; i < base + segs; i++) {
+				if (phb->ioda.dma32_segmap[i] !=
+				    IODA_INVALID_PE)
+					goto found;
+			}
+		}
+	} while (segs--);
+
+	if (!segs) {
+		pe_warn(pe, "No available DMA32 segments\n");
+		return;
+	}
 
+found:
 	tbl = pnv_pci_table_alloc(phb->hose->node);
 	iommu_register_group(&pe->table_group, phb->hose->global_number,
 			pe->pe_number);
 	pnv_pci_link_table_and_group(phb->hose->node, 0, tbl, &pe->table_group);
 
 	/* Grab a 32-bit TCE table */
+	pe_info(pe, "DMA weight %d (%d), assigned (%d) %d DMA32 segments\n",
+		weight, total_weight, base, segs);
 	pe_info(pe, " Setting up 32-bit TCE table at %08x..%08x\n",
 		base * PNV_IODA1_DMA32_SEGSIZE,
 		(base + segs) * PNV_IODA1_DMA32_SEGSIZE - 1);
@@ -2068,6 +2098,10 @@ static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
 		}
 	}
 
+	/* Setup DMA32 segment mapping */
+	for (i = base; i < base + segs; i++)
+		phb->ioda.dma32_segmap[i] = pe->pe_number;
+
 	/* Setup linux iommu table */
 	pnv_pci_setup_iommu_table(tbl, addr, tce32_segsz * segs,
 				  base * PNV_IODA1_DMA32_SEGSIZE,
@@ -2542,73 +2576,34 @@ static void pnv_pci_ioda2_setup_dma_pe(struct pnv_phb *phb,
 static void pnv_ioda_setup_dma(struct pnv_phb *phb)
 {
 	struct pci_controller *hose = phb->hose;
-	unsigned int weight, total_weight, dma_pe_count;
-	unsigned int residual, remaining, segs, base;
 	struct pnv_ioda_pe *pe;
-
-	total_weight = 0;
-	pci_walk_bus(phb->hose->bus, pnv_pci_ioda_dev_dma_weight,
-		     &total_weight);
-
-	dma_pe_count = 0;
-	list_for_each_entry(pe, &phb->ioda.pe_list, list) {
-		weight = pnv_pci_ioda_pe_dma_weight(pe);
-		if (weight > 0)
-			dma_pe_count++;
-	}
+	unsigned int weight;
 
 	/* If we have more PE# than segments available, hand out one
 	 * per PE until we run out and let the rest fail. If not,
 	 * then we assign at least one segment per PE, plus more based
 	 * on the amount of devices under that PE
 	 */
-	if (dma_pe_count > phb->ioda.tce32_count)
-		residual = 0;
-	else
-		residual = phb->ioda.tce32_count - dma_pe_count;
-
-	pr_info("PCI: Domain %04x has %ld available 32-bit DMA segments\n",
-		hose->global_number, phb->ioda.tce32_count);
-	pr_info("PCI: %d PE# for a total weight of %d\n",
-		dma_pe_count, total_weight);
+	pr_info("PCI: Domain %04x has %d available 32-bit DMA segments\n",
+		hose->global_number, phb->ioda.dma32_count);
 
 	pnv_pci_ioda_setup_opal_tce_kill(phb);
 
-	/* Walk our PE list and configure their DMA segments, hand them
-	 * out one base segment plus any residual segments based on
-	 * weight
-	 */
-	remaining = phb->ioda.tce32_count;
-	base = 0;
+	/* Walk our PE list and configure their DMA segments */
 	list_for_each_entry(pe, &phb->ioda.pe_list, list) {
 		weight = pnv_pci_ioda_pe_dma_weight(pe);
 		if (!weight)
 			continue;
 
-		if (!remaining) {
-			pe_warn(pe, "No DMA32 resources available\n");
-			continue;
-		}
-		segs = 1;
-		if (residual) {
-			segs += ((weight * residual) + (total_weight / 2)) /
-				total_weight;
-			if (segs > remaining)
-				segs = remaining;
-		}
-
 		/*
 		 * For IODA2 compliant PHB3, we needn't care about the weight.
 		 * The all available 32-bits DMA space will be assigned to
 		 * the specific PE.
 		 */
 		if (phb->type == PNV_PHB_IODA1) {
-			pe_info(pe, "DMA weight %d, assigned %d DMA32 segments\n",
-				weight, segs);
-			pnv_pci_ioda1_setup_dma_pe(phb, pe, base, segs);
+			pnv_pci_ioda1_setup_dma_pe(phb, pe);
 		} else if (phb->type == PNV_PHB_IODA2) {
 			pe_info(pe, "Assign DMA32 space\n");
-			segs = 0;
 			pnv_pci_ioda2_setup_dma_pe(phb, pe);
 		} else if (phb->type == PNV_PHB_NPU) {
 			/*
@@ -2618,9 +2613,6 @@ static void pnv_ioda_setup_dma(struct pnv_phb *phb)
 			 * as the PHB3 TVT.
 			 */
 		}
-
-		remaining -= segs;
-		base += segs;
 	}
 }
 
@@ -3327,7 +3319,8 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 {
 	struct pci_controller *hose;
 	struct pnv_phb *phb;
-	unsigned long size, m64map_off, m32map_off, pemap_off, iomap_off = 0;
+	unsigned long size, m64map_off, m32map_off, pemap_off;
+	unsigned long iomap_off = 0, dma32map_off = 0;
 	const __be64 *prop64;
 	const __be32 *prop32;
 	int len;
@@ -3413,6 +3406,10 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 	phb->ioda.io_segsize = phb->ioda.io_size / phb->ioda.total_pe_num;
 	phb->ioda.io_pci_base = 0; /* XXX calculate this ? */
 
+	/* Calculate how many 32-bit TCE segments we have */
+	phb->ioda.dma32_count = phb->ioda.m32_pci_base /
+				PNV_IODA1_DMA32_SEGSIZE;
+
 	/* Allocate aux data & arrays. We don't have IO ports on PHB3 */
 	size = _ALIGN_UP(phb->ioda.total_pe_num / 8, sizeof(unsigned long));
 	m64map_off = size;
@@ -3422,6 +3419,9 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 	if (phb->type == PNV_PHB_IODA1) {
 		iomap_off = size;
 		size += phb->ioda.total_pe_num * sizeof(phb->ioda.io_segmap[0]);
+		dma32map_off = size;
+		size += phb->ioda.dma32_count *
+			sizeof(phb->ioda.dma32_segmap[0]);
 	}
 	pemap_off = size;
 	size += phb->ioda.total_pe_num * sizeof(struct pnv_ioda_pe);
@@ -3437,6 +3437,10 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 		phb->ioda.io_segmap = aux + iomap_off;
 		for (segno = 0; segno < phb->ioda.total_pe_num; segno++)
 			phb->ioda.io_segmap[segno] = IODA_INVALID_PE;
+
+		phb->ioda.dma32_segmap = aux + dma32map_off;
+		for (segno = 0; segno < phb->ioda.dma32_count; segno++)
+			phb->ioda.dma32_segmap[segno] = IODA_INVALID_PE;
 	}
 	phb->ioda.pe_array = aux + pemap_off;
 	set_bit(phb->ioda.reserved_pe_idx, phb->ioda.pe_alloc);
@@ -3445,7 +3449,7 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 	mutex_init(&phb->ioda.pe_list_mutex);
 
 	/* Calculate how many 32-bit TCE segments we have */
-	phb->ioda.tce32_count = phb->ioda.m32_pci_base /
+	phb->ioda.dma32_count = phb->ioda.m32_pci_base /
 				PNV_IODA1_DMA32_SEGSIZE;
 
 #if 0 /* We should really do that ... */
diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
index 117cfcd..14d9391 100644
--- a/arch/powerpc/platforms/powernv/pci.h
+++ b/arch/powerpc/platforms/powernv/pci.h
@@ -142,6 +142,10 @@ struct pnv_phb {
 		unsigned int		*m32_segmap;
 		unsigned int		*io_segmap;
 
+		/* DMA32 segment maps - IODA1 only */
+		unsigned int		dma32_count;
+		unsigned int		*dma32_segmap;
+
 		/* IRQ chip */
 		int			irq_chip_init;
 		struct irq_chip		irq_chip;
@@ -158,9 +162,6 @@ struct pnv_phb {
 		 */
 		unsigned char		pe_rmap[0x10000];
 
-		/* 32-bit TCE tables allocation */
-		unsigned long		tce32_count;
-
 		/* TCE cache invalidate registers (physical and
 		 * remapped)
 		 */
-- 
2.1.0

[PATCH v9 17/26] powerpc/powernv: Use PE instead of number during setup and release

[Gavin Shan]

In current implementation, the PEs that are allocated or picked
from the reserved list are identified by PE number. The PE instance
has to be picked according to the PE number eventually. We have
same issue when PE is released.

For pnv_ioda_pick_m64_pe() and pnv_ioda_alloc_pe(), this returns
PE instance so that pnv_ioda_setup_bus_PE() can use the allocated
or reserved PE instance directly. Also, pnv_ioda_setup_bus_PE()
returns the reserved/allocated PE instance to be used in subsequent
patches. On the other hand, pnv_ioda_free_pe() uses PE instance
(not number) as its argument. No logical changes introduced.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 104 +++++++++++++++++-------------
 arch/powerpc/platforms/powernv/pci.h      |   2 +-
 2 files changed, 59 insertions(+), 47 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index cfd2906..5ee8a57 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -123,6 +123,14 @@ static inline bool pnv_pci_is_mem_pref_64(unsigned long flags)
 		(IORESOURCE_MEM_64 | IORESOURCE_PREFETCH));
 }
 
+static struct pnv_ioda_pe *pnv_ioda_init_pe(struct pnv_phb *phb, int pe_no)
+{
+	phb->ioda.pe_array[pe_no].phb = phb;
+	phb->ioda.pe_array[pe_no].pe_number = pe_no;
+
+	return &phb->ioda.pe_array[pe_no];
+}
+
 static void pnv_ioda_reserve_pe(struct pnv_phb *phb, int pe_no)
 {
 	if (!(pe_no >= 0 && pe_no < phb->ioda.total_pe_num)) {
@@ -135,11 +143,10 @@ static void pnv_ioda_reserve_pe(struct pnv_phb *phb, int pe_no)
 		pr_debug("%s: PE %d was reserved on PHB#%x\n",
 			 __func__, pe_no, phb->hose->global_number);
 
-	phb->ioda.pe_array[pe_no].phb = phb;
-	phb->ioda.pe_array[pe_no].pe_number = pe_no;
+	pnv_ioda_init_pe(phb, pe_no);
 }
 
-static unsigned int pnv_ioda_alloc_pe(struct pnv_phb *phb)
+static struct pnv_ioda_pe *pnv_ioda_alloc_pe(struct pnv_phb *phb)
 {
 	unsigned long pe;
 
@@ -147,20 +154,20 @@ static unsigned int pnv_ioda_alloc_pe(struct pnv_phb *phb)
 		pe = find_next_zero_bit(phb->ioda.pe_alloc,
 					phb->ioda.total_pe_num, 0);
 		if (pe >= phb->ioda.total_pe_num)
-			return IODA_INVALID_PE;
+			return NULL;
 	} while(test_and_set_bit(pe, phb->ioda.pe_alloc));
 
-	phb->ioda.pe_array[pe].phb = phb;
-	phb->ioda.pe_array[pe].pe_number = pe;
-	return pe;
+	return pnv_ioda_init_pe(phb, pe);
 }
 
-static void pnv_ioda_free_pe(struct pnv_phb *phb, int pe)
+static void pnv_ioda_free_pe(struct pnv_ioda_pe *pe)
 {
-	WARN_ON(phb->ioda.pe_array[pe].pdev);
+	struct pnv_phb *phb = pe->phb;
 
-	memset(&phb->ioda.pe_array[pe], 0, sizeof(struct pnv_ioda_pe));
-	clear_bit(pe, phb->ioda.pe_alloc);
+	WARN_ON(pe->pdev);
+
+	memset(pe, 0, sizeof(struct pnv_ioda_pe));
+	clear_bit(pe->pe_number, phb->ioda.pe_alloc);
 }
 
 /* The default M64 BAR is shared by all PEs */
@@ -320,7 +327,7 @@ static void pnv_ioda_reserve_m64_pe(struct pci_bus *bus,
 	}
 }
 
-static unsigned int pnv_ioda_pick_m64_pe(struct pci_bus *bus, bool all)
+static struct pnv_ioda_pe *pnv_ioda_pick_m64_pe(struct pci_bus *bus, bool all)
 {
 	struct pci_controller *hose = pci_bus_to_host(bus);
 	struct pnv_phb *phb = hose->private_data;
@@ -330,7 +337,7 @@ static unsigned int pnv_ioda_pick_m64_pe(struct pci_bus *bus, bool all)
 
 	/* Root bus shouldn't use M64 */
 	if (pci_is_root_bus(bus))
-		return IODA_INVALID_PE;
+		return NULL;
 
 	/* Allocate bitmap */
 	size = _ALIGN_UP(phb->ioda.total_pe_num / 8, sizeof(unsigned long));
@@ -338,7 +345,7 @@ static unsigned int pnv_ioda_pick_m64_pe(struct pci_bus *bus, bool all)
 	if (!pe_alloc) {
 		pr_warn("%s: Out of memory !\n",
 			__func__);
-		return IODA_INVALID_PE;
+		return NULL;
 	}
 
 	/* Figure out reserved PE numbers by the PE */
@@ -351,7 +358,7 @@ static unsigned int pnv_ioda_pick_m64_pe(struct pci_bus *bus, bool all)
 	 */
 	if (bitmap_empty(pe_alloc, phb->ioda.total_pe_num)) {
 		kfree(pe_alloc);
-		return IODA_INVALID_PE;
+		return NULL;
 	}
 
 	/*
@@ -397,7 +404,7 @@ static unsigned int pnv_ioda_pick_m64_pe(struct pci_bus *bus, bool all)
 	}
 
 	kfree(pe_alloc);
-	return master_pe->pe_number;
+	return master_pe;
 }
 
 static void __init pnv_ioda_parse_m64_window(struct pnv_phb *phb)
@@ -963,7 +970,6 @@ static struct pnv_ioda_pe *pnv_ioda_setup_dev_PE(struct pci_dev *dev)
 	struct pnv_phb *phb = hose->private_data;
 	struct pci_dn *pdn = pci_get_pdn(dev);
 	struct pnv_ioda_pe *pe;
-	unsigned int pe_num;
 
 	if (!pdn) {
 		pr_err("%s: Device tree node not associated properly\n",
@@ -973,8 +979,8 @@ static struct pnv_ioda_pe *pnv_ioda_setup_dev_PE(struct pci_dev *dev)
 	if (pdn->pe_number != IODA_INVALID_PE)
 		return NULL;
 
-	pe_num = pnv_ioda_alloc_pe(phb);
-	if (pe_num == IODA_INVALID_PE) {
+	pe = pnv_ioda_alloc_pe(phb);
+	if (!pe) {
 		pr_warning("%s: Not enough PE# available, disabling device\n",
 			   pci_name(dev));
 		return NULL;
@@ -987,10 +993,9 @@ static struct pnv_ioda_pe *pnv_ioda_setup_dev_PE(struct pci_dev *dev)
 	 *
 	 * At some point we want to remove the PDN completely anyways
 	 */
-	pe = &phb->ioda.pe_array[pe_num];
 	pci_dev_get(dev);
 	pdn->pcidev = dev;
-	pdn->pe_number = pe_num;
+	pdn->pe_number = pe->pe_number;
 	pe->flags = PNV_IODA_PE_DEV;
 	pe->pdev = dev;
 	pe->pbus = NULL;
@@ -1001,8 +1006,7 @@ static struct pnv_ioda_pe *pnv_ioda_setup_dev_PE(struct pci_dev *dev)
 
 	if (pnv_ioda_configure_pe(phb, pe)) {
 		/* XXX What do we do here ? */
-		if (pe_num)
-			pnv_ioda_free_pe(phb, pe_num);
+		pnv_ioda_free_pe(pe);
 		pdn->pe_number = IODA_INVALID_PE;
 		pe->pdev = NULL;
 		pci_dev_put(dev);
@@ -1037,28 +1041,26 @@ static void pnv_ioda_setup_same_PE(struct pci_bus *bus, struct pnv_ioda_pe *pe)
  * subordinate PCI devices and buses. The second type of PE is normally
  * orgiriated by PCIe-to-PCI bridge or PLX switch downstream ports.
  */
-static void pnv_ioda_setup_bus_PE(struct pci_bus *bus, bool all)
+static struct pnv_ioda_pe *pnv_ioda_setup_bus_PE(struct pci_bus *bus, bool all)
 {
 	struct pci_controller *hose = pci_bus_to_host(bus);
 	struct pnv_phb *phb = hose->private_data;
-	struct pnv_ioda_pe *pe;
-	unsigned int pe_num = IODA_INVALID_PE;
+	struct pnv_ioda_pe *pe = NULL;
 
 	/* Check if PE is determined by M64 */
 	if (phb->pick_m64_pe)
-		pe_num = phb->pick_m64_pe(bus, all);
+		pe = phb->pick_m64_pe(bus, all);
 
 	/* The PE number isn't pinned by M64 */
-	if (pe_num == IODA_INVALID_PE)
-		pe_num = pnv_ioda_alloc_pe(phb);
+	if (!pe)
+		pe = pnv_ioda_alloc_pe(phb);
 
-	if (pe_num == IODA_INVALID_PE) {
+	if (!pe) {
 		pr_warning("%s: Not enough PE# available for PCI bus %04x:%02x\n",
 			__func__, pci_domain_nr(bus), bus->number);
-		return;
+		return NULL;
 	}
 
-	pe = &phb->ioda.pe_array[pe_num];
 	pe->flags |= (all ? PNV_IODA_PE_BUS_ALL : PNV_IODA_PE_BUS);
 	pe->pbus = bus;
 	pe->pdev = NULL;
@@ -1067,17 +1069,16 @@ static void pnv_ioda_setup_bus_PE(struct pci_bus *bus, bool all)
 
 	if (all)
 		pe_info(pe, "Secondary bus %d..%d associated with PE#%d\n",
-			bus->busn_res.start, bus->busn_res.end, pe_num);
+			bus->busn_res.start, bus->busn_res.end, pe->pe_number);
 	else
 		pe_info(pe, "Secondary bus %d associated with PE#%d\n",
-			bus->busn_res.start, pe_num);
+			bus->busn_res.start, pe->pe_number);
 
 	if (pnv_ioda_configure_pe(phb, pe)) {
 		/* XXX What do we do here ? */
-		if (pe_num)
-			pnv_ioda_free_pe(phb, pe_num);
+		pnv_ioda_free_pe(pe);
 		pe->pbus = NULL;
-		return;
+		return NULL;
 	}
 
 	/* Associate it with all child devices */
@@ -1085,6 +1086,8 @@ static void pnv_ioda_setup_bus_PE(struct pci_bus *bus, bool all)
 
 	/* Put PE to the list */
 	list_add_tail(&pe->list, &phb->ioda.pe_list);
+
+	return pe;
 }
 
 static struct pnv_ioda_pe *pnv_ioda_setup_npu_PE(struct pci_dev *npu_pdev)
@@ -1396,7 +1399,7 @@ static void pnv_ioda_release_vf_PE(struct pci_dev *pdev)
 
 		pnv_ioda_deconfigure_pe(phb, pe);
 
-		pnv_ioda_free_pe(phb, pe->pe_number);
+		pnv_ioda_free_pe(pe);
 	}
 }
 
@@ -1405,6 +1408,7 @@ void pnv_pci_sriov_disable(struct pci_dev *pdev)
 	struct pci_bus        *bus;
 	struct pci_controller *hose;
 	struct pnv_phb        *phb;
+	struct pnv_ioda_pe    *pe;
 	struct pci_dn         *pdn;
 	struct pci_sriov      *iov;
 	u16                    num_vfs, i;
@@ -1429,8 +1433,11 @@ void pnv_pci_sriov_disable(struct pci_dev *pdev)
 		/* Release PE numbers */
 		if (pdn->m64_single_mode) {
 			for (i = 0; i < num_vfs; i++) {
-				if (pdn->pe_num_map[i] != IODA_INVALID_PE)
-					pnv_ioda_free_pe(phb, pdn->pe_num_map[i]);
+				if (pdn->pe_num_map[i] == IODA_INVALID_PE)
+					continue;
+
+				pe = &phb->ioda.pe_array[pdn->pe_num_map[i]];
+				pnv_ioda_free_pe(pe);
 			}
 		} else
 			bitmap_clear(phb->ioda.pe_alloc, *pdn->pe_num_map, num_vfs);
@@ -1483,8 +1490,7 @@ static void pnv_ioda_setup_vf_PE(struct pci_dev *pdev, u16 num_vfs)
 
 		if (pnv_ioda_configure_pe(phb, pe)) {
 			/* XXX What do we do here ? */
-			if (pe_num)
-				pnv_ioda_free_pe(phb, pe_num);
+			pnv_ioda_free_pe(pe);
 			pe->pdev = NULL;
 			continue;
 		}
@@ -1503,6 +1509,7 @@ int pnv_pci_sriov_enable(struct pci_dev *pdev, u16 num_vfs)
 	struct pci_bus        *bus;
 	struct pci_controller *hose;
 	struct pnv_phb        *phb;
+	struct pnv_ioda_pe    *pe;
 	struct pci_dn         *pdn;
 	int                    ret;
 	u16                    i;
@@ -1545,11 +1552,13 @@ int pnv_pci_sriov_enable(struct pci_dev *pdev, u16 num_vfs)
 		/* Calculate available PE for required VFs */
 		if (pdn->m64_single_mode) {
 			for (i = 0; i < num_vfs; i++) {
-				pdn->pe_num_map[i] = pnv_ioda_alloc_pe(phb);
-				if (pdn->pe_num_map[i] == IODA_INVALID_PE) {
+				pe = pnv_ioda_alloc_pe(phb);
+				if (!pe) {
 					ret = -EBUSY;
 					goto m64_failed;
 				}
+
+				pdn->pe_num_map[i] = pe->pe_number;
 			}
 		} else {
 			mutex_lock(&phb->ioda.pe_alloc_mutex);
@@ -1594,8 +1603,11 @@ int pnv_pci_sriov_enable(struct pci_dev *pdev, u16 num_vfs)
 m64_failed:
 	if (pdn->m64_single_mode) {
 		for (i = 0; i < num_vfs; i++) {
-			if (pdn->pe_num_map[i] != IODA_INVALID_PE)
-				pnv_ioda_free_pe(phb, pdn->pe_num_map[i]);
+			if (pdn->pe_num_map[i] == IODA_INVALID_PE)
+				continue;
+
+			pe = &phb->ioda.pe_array[pdn->pe_num_map[i]];
+			pnv_ioda_free_pe(pe);
 		}
 	} else
 		bitmap_clear(phb->ioda.pe_alloc, *pdn->pe_num_map, num_vfs);
diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
index 14d9391..904f60b 100644
--- a/arch/powerpc/platforms/powernv/pci.h
+++ b/arch/powerpc/platforms/powernv/pci.h
@@ -105,7 +105,7 @@ struct pnv_phb {
 	int (*init_m64)(struct pnv_phb *phb);
 	void (*reserve_m64_pe)(struct pci_bus *bus,
 			       unsigned long *pe_bitmap, bool all);
-	unsigned int (*pick_m64_pe)(struct pci_bus *bus, bool all);
+	struct pnv_ioda_pe *(*pick_m64_pe)(struct pci_bus *bus, bool all);
 	int (*get_pe_state)(struct pnv_phb *phb, int pe_no);
 	void (*freeze_pe)(struct pnv_phb *phb, int pe_no);
 	int (*unfreeze_pe)(struct pnv_phb *phb, int pe_no, int opt);
-- 
2.1.0

[PATCH v9 18/26] powerpc/pci: Rename pcibios_{add, remove}_pci_devices()

[Gavin Shan]

This renames pcibios_{add,remove}_pci_devices() to avoid conflicts
with names of the weak functions in PCI subsystem, which have the
prefix "pcibios". No logical changes introduced.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/pci-bridge.h |  4 ++--
 arch/powerpc/kernel/eeh_driver.c      | 12 ++++++------
 arch/powerpc/kernel/pci-hotplug.c     | 15 +++++++--------
 drivers/pci/hotplug/rpadlpar_core.c   |  2 +-
 drivers/pci/hotplug/rpaphp_core.c     |  4 ++--
 drivers/pci/hotplug/rpaphp_pci.c      |  2 +-
 6 files changed, 19 insertions(+), 20 deletions(-)

diff --git a/arch/powerpc/include/asm/pci-bridge.h b/arch/powerpc/include/asm/pci-bridge.h
index 220129f..99027b8 100644
--- a/arch/powerpc/include/asm/pci-bridge.h
+++ b/arch/powerpc/include/asm/pci-bridge.h
@@ -260,10 +260,10 @@ static inline struct eeh_dev *pdn_to_eeh_dev(struct pci_dn *pdn)
 extern struct pci_bus *pcibios_find_pci_bus(struct device_node *dn);
 
 /** Remove all of the PCI devices under this bus */
-extern void pcibios_remove_pci_devices(struct pci_bus *bus);
+extern void pci_hp_remove_devices(struct pci_bus *bus);
 
 /** Discover new pci devices under this bus, and add them */
-extern void pcibios_add_pci_devices(struct pci_bus *bus);
+extern void pci_hp_add_devices(struct pci_bus *bus);
 
 
 extern void isa_bridge_find_early(struct pci_controller *hose);
diff --git a/arch/powerpc/kernel/eeh_driver.c b/arch/powerpc/kernel/eeh_driver.c
index fb6207d..618d13c 100644
--- a/arch/powerpc/kernel/eeh_driver.c
+++ b/arch/powerpc/kernel/eeh_driver.c
@@ -621,7 +621,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus,
 	 * We don't remove the corresponding PE instances because
 	 * we need the information afterwords. The attached EEH
 	 * devices are expected to be attached soon when calling
-	 * into pcibios_add_pci_devices().
+	 * into pci_hp_add_devices().
 	 */
 	eeh_pe_state_mark(pe, EEH_PE_KEEP);
 	if (bus) {
@@ -630,7 +630,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus,
 		} else {
 			eeh_pe_state_clear(pe, EEH_PE_PRI_BUS);
 			pci_lock_rescan_remove();
-			pcibios_remove_pci_devices(bus);
+			pci_hp_remove_devices(bus);
 			pci_unlock_rescan_remove();
 		}
 	} else if (frozen_bus) {
@@ -681,7 +681,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus,
 		if (pe->type & EEH_PE_VF)
 			eeh_add_virt_device(edev, NULL);
 		else
-			pcibios_add_pci_devices(bus);
+			pci_hp_add_devices(bus);
 	} else if (frozen_bus && rmv_data->removed) {
 		pr_info("EEH: Sleep 5s ahead of partial hotplug\n");
 		ssleep(5);
@@ -691,7 +691,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus,
 		if (pe->type & EEH_PE_VF)
 			eeh_add_virt_device(edev, NULL);
 		else
-			pcibios_add_pci_devices(frozen_bus);
+			pci_hp_add_devices(frozen_bus);
 	}
 	eeh_pe_state_clear(pe, EEH_PE_KEEP);
 
@@ -896,7 +896,7 @@ perm_error:
 			eeh_pe_dev_mode_mark(pe, EEH_DEV_REMOVED);
 
 			pci_lock_rescan_remove();
-			pcibios_remove_pci_devices(frozen_bus);
+			pci_hp_remove_devices(frozen_bus);
 			pci_unlock_rescan_remove();
 		}
 	}
@@ -981,7 +981,7 @@ static void eeh_handle_special_event(void)
 				bus = eeh_pe_bus_get(phb_pe);
 				eeh_pe_dev_traverse(pe,
 					eeh_report_failure, NULL);
-				pcibios_remove_pci_devices(bus);
+				pci_hp_remove_devices(bus);
 			}
 			pci_unlock_rescan_remove();
 		}
diff --git a/arch/powerpc/kernel/pci-hotplug.c b/arch/powerpc/kernel/pci-hotplug.c
index 59c4361..2d108e5 100644
--- a/arch/powerpc/kernel/pci-hotplug.c
+++ b/arch/powerpc/kernel/pci-hotplug.c
@@ -38,20 +38,20 @@ void pcibios_release_device(struct pci_dev *dev)
 }
 
 /**
- * pcibios_remove_pci_devices - remove all devices under this bus
+ * pci_hp_remove_devices - remove all devices under this bus
  * @bus: the indicated PCI bus
  *
  * Remove all of the PCI devices under this bus both from the
  * linux pci device tree, and from the powerpc EEH address cache.
  */
-void pcibios_remove_pci_devices(struct pci_bus *bus)
+void pci_hp_remove_devices(struct pci_bus *bus)
 {
 	struct pci_dev *dev, *tmp;
 	struct pci_bus *child_bus;
 
 	/* First go down child busses */
 	list_for_each_entry(child_bus, &bus->children, node)
-		pcibios_remove_pci_devices(child_bus);
+		pci_hp_remove_devices(child_bus);
 
 	pr_debug("PCI: Removing devices on bus %04x:%02x\n",
 		 pci_domain_nr(bus),  bus->number);
@@ -60,11 +60,10 @@ void pcibios_remove_pci_devices(struct pci_bus *bus)
 		pci_stop_and_remove_bus_device(dev);
 	}
 }
-
-EXPORT_SYMBOL_GPL(pcibios_remove_pci_devices);
+EXPORT_SYMBOL_GPL(pci_hp_remove_devices);
 
 /**
- * pcibios_add_pci_devices - adds new pci devices to bus
+ * pci_hp_add_devices - adds new pci devices to bus
  * @bus: the indicated PCI bus
  *
  * This routine will find and fixup new pci devices under
@@ -74,7 +73,7 @@ EXPORT_SYMBOL_GPL(pcibios_remove_pci_devices);
  * is how this routine differs from other, similar pcibios
  * routines.)
  */
-void pcibios_add_pci_devices(struct pci_bus * bus)
+void pci_hp_add_devices(struct pci_bus *bus)
 {
 	int slotno, mode, pass, max;
 	struct pci_dev *dev;
@@ -114,4 +113,4 @@ void pcibios_add_pci_devices(struct pci_bus * bus)
 	}
 	pcibios_finish_adding_to_bus(bus);
 }
-EXPORT_SYMBOL_GPL(pcibios_add_pci_devices);
+EXPORT_SYMBOL_GPL(pci_hp_add_devices);
diff --git a/drivers/pci/hotplug/rpadlpar_core.c b/drivers/pci/hotplug/rpadlpar_core.c
index b46b57d..b770d6d 100644
--- a/drivers/pci/hotplug/rpadlpar_core.c
+++ b/drivers/pci/hotplug/rpadlpar_core.c
@@ -380,7 +380,7 @@ int dlpar_remove_pci_slot(char *drc_name, struct device_node *dn)
 	}
 
 	/* Remove all devices below slot */
-	pcibios_remove_pci_devices(bus);
+	pci_hp_remove_devices(bus);
 
 	/* Unmap PCI IO space */
 	if (pcibios_unmap_io_space(bus)) {
diff --git a/drivers/pci/hotplug/rpaphp_core.c b/drivers/pci/hotplug/rpaphp_core.c
index 611f605..8d13202 100644
--- a/drivers/pci/hotplug/rpaphp_core.c
+++ b/drivers/pci/hotplug/rpaphp_core.c
@@ -404,7 +404,7 @@ static int enable_slot(struct hotplug_slot *hotplug_slot)
 
 	if (state == PRESENT) {
 		pci_lock_rescan_remove();
-		pcibios_add_pci_devices(slot->bus);
+		pci_hp_add_devices(slot->bus);
 		pci_unlock_rescan_remove();
 		slot->state = CONFIGURED;
 	} else if (state == EMPTY) {
@@ -426,7 +426,7 @@ static int disable_slot(struct hotplug_slot *hotplug_slot)
 		return -EINVAL;
 
 	pci_lock_rescan_remove();
-	pcibios_remove_pci_devices(slot->bus);
+	pci_hp_remove_devices(slot->bus);
 	pci_unlock_rescan_remove();
 	vm_unmap_aliases();
 
diff --git a/drivers/pci/hotplug/rpaphp_pci.c b/drivers/pci/hotplug/rpaphp_pci.c
index 7836d69..137a892 100644
--- a/drivers/pci/hotplug/rpaphp_pci.c
+++ b/drivers/pci/hotplug/rpaphp_pci.c
@@ -116,7 +116,7 @@ int rpaphp_enable_slot(struct slot *slot)
 		}
 
 		if (list_empty(&bus->devices))
-			pcibios_add_pci_devices(bus);
+			pci_hp_add_devices(bus);
 
 		if (!list_empty(&bus->devices)) {
 			info->adapter_status = CONFIGURED;
-- 
2.1.0

[PATCH v9 19/26] powerpc/pci: Rename pcibios_find_pci_bus()

[Gavin Shan]

This renames pcibios_find_pci_bus() to pci_find_bus_by_node() to
avoid conflicts with those PCI subsystem weak function names, which
have prefix "pcibios". No logical changes introduced.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/include/asm/pci-bridge.h      | 2 +-
 arch/powerpc/platforms/pseries/pci_dlpar.c | 5 ++---
 drivers/pci/hotplug/rpadlpar_core.c        | 6 +++---
 drivers/pci/hotplug/rpaphp_pci.c           | 2 +-
 4 files changed, 7 insertions(+), 8 deletions(-)

diff --git a/arch/powerpc/include/asm/pci-bridge.h b/arch/powerpc/include/asm/pci-bridge.h
index 99027b8..f0a5b16 100644
--- a/arch/powerpc/include/asm/pci-bridge.h
+++ b/arch/powerpc/include/asm/pci-bridge.h
@@ -257,7 +257,7 @@ static inline struct eeh_dev *pdn_to_eeh_dev(struct pci_dn *pdn)
 #endif
 
 /** Find the bus corresponding to the indicated device node */
-extern struct pci_bus *pcibios_find_pci_bus(struct device_node *dn);
+extern struct pci_bus *pci_find_bus_by_node(struct device_node *dn);
 
 /** Remove all of the PCI devices under this bus */
 extern void pci_hp_remove_devices(struct pci_bus *bus);
diff --git a/arch/powerpc/platforms/pseries/pci_dlpar.c b/arch/powerpc/platforms/pseries/pci_dlpar.c
index 5d4a3df..aee22b4 100644
--- a/arch/powerpc/platforms/pseries/pci_dlpar.c
+++ b/arch/powerpc/platforms/pseries/pci_dlpar.c
@@ -54,8 +54,7 @@ find_bus_among_children(struct pci_bus *bus,
 	return child;
 }
 
-struct pci_bus *
-pcibios_find_pci_bus(struct device_node *dn)
+struct pci_bus *pci_find_bus_by_node(struct device_node *dn)
 {
 	struct pci_dn *pdn = dn->data;
 
@@ -64,7 +63,7 @@ pcibios_find_pci_bus(struct device_node *dn)
 
 	return find_bus_among_children(pdn->phb->bus, dn);
 }
-EXPORT_SYMBOL_GPL(pcibios_find_pci_bus);
+EXPORT_SYMBOL_GPL(pci_find_bus_by_node);
 
 struct pci_controller *init_phb_dynamic(struct device_node *dn)
 {
diff --git a/drivers/pci/hotplug/rpadlpar_core.c b/drivers/pci/hotplug/rpadlpar_core.c
index b770d6d..dc67f39 100644
--- a/drivers/pci/hotplug/rpadlpar_core.c
+++ b/drivers/pci/hotplug/rpadlpar_core.c
@@ -175,7 +175,7 @@ static int dlpar_add_pci_slot(char *drc_name, struct device_node *dn)
 	struct pci_dev *dev;
 	struct pci_controller *phb;
 
-	if (pcibios_find_pci_bus(dn))
+	if (pci_find_bus_by_node(dn))
 		return -EINVAL;
 
 	/* Add pci bus */
@@ -212,7 +212,7 @@ static int dlpar_remove_phb(char *drc_name, struct device_node *dn)
 	struct pci_dn *pdn;
 	int rc = 0;
 
-	if (!pcibios_find_pci_bus(dn))
+	if (!pci_find_bus_by_node(dn))
 		return -EINVAL;
 
 	/* If pci slot is hotpluggable, use hotplug to remove it */
@@ -356,7 +356,7 @@ int dlpar_remove_pci_slot(char *drc_name, struct device_node *dn)
 
 	pci_lock_rescan_remove();
 
-	bus = pcibios_find_pci_bus(dn);
+	bus = pci_find_bus_by_node(dn);
 	if (!bus) {
 		ret = -EINVAL;
 		goto out;
diff --git a/drivers/pci/hotplug/rpaphp_pci.c b/drivers/pci/hotplug/rpaphp_pci.c
index 137a892..ea41ea1 100644
--- a/drivers/pci/hotplug/rpaphp_pci.c
+++ b/drivers/pci/hotplug/rpaphp_pci.c
@@ -93,7 +93,7 @@ int rpaphp_enable_slot(struct slot *slot)
 	if (rc)
 		return rc;
 
-	bus = pcibios_find_pci_bus(slot->dn);
+	bus = pci_find_bus_by_node(slot->dn);
 	if (!bus) {
 		err("%s: no pci_bus for dn %s\n", __func__, slot->dn->full_name);
 		return -EINVAL;
-- 
2.1.0

[PATCH v9 20/26] powerpc/pci: Move pci_find_bus_by_node() around

[Gavin Shan]

This moves pci_find_bus_by_node() from arch/powerpc/platforms/
pseries/pci_dlpar.c to arch/powerpc/kernel/pci-hotplug.c so that
the function can be used by pSeries and PowerNV platform at the
same time. Also, below cleanup applied. No functional changes
introduced.

   * Remove variable "busdn" in find_bus_among_children()
   * Use PCI_DN() to convert device node to pci_dn

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
Acked-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
 arch/powerpc/kernel/pci-hotplug.c          | 29 ++++++++++++++++++++++++++++
 arch/powerpc/platforms/pseries/pci_dlpar.c | 31 ------------------------------
 2 files changed, 29 insertions(+), 31 deletions(-)

diff --git a/arch/powerpc/kernel/pci-hotplug.c b/arch/powerpc/kernel/pci-hotplug.c
index 2d108e5..46587a1 100644
--- a/arch/powerpc/kernel/pci-hotplug.c
+++ b/arch/powerpc/kernel/pci-hotplug.c
@@ -21,6 +21,35 @@
 #include <asm/firmware.h>
 #include <asm/eeh.h>
 
+static struct pci_bus *find_bus_among_children(struct pci_bus *bus,
+					       struct device_node *dn)
+{
+	struct pci_bus *child = NULL;
+	struct pci_bus *tmp;
+
+	if (pci_bus_to_OF_node(bus) == dn)
+		return bus;
+
+	list_for_each_entry(tmp, &bus->children, node) {
+		child = find_bus_among_children(tmp, dn);
+		if (child)
+			break;
+	}
+
+	return child;
+}
+
+struct pci_bus *pci_find_bus_by_node(struct device_node *dn)
+{
+	struct pci_dn *pdn = PCI_DN(dn);
+
+	if (!pdn  || !pdn->phb || !pdn->phb->bus)
+		return NULL;
+
+	return find_bus_among_children(pdn->phb->bus, dn);
+}
+EXPORT_SYMBOL_GPL(pci_find_bus_by_node);
+
 /**
  * pcibios_release_device - release PCI device
  * @dev: PCI device
diff --git a/arch/powerpc/platforms/pseries/pci_dlpar.c b/arch/powerpc/platforms/pseries/pci_dlpar.c
index aee22b4..906dbaa 100644
--- a/arch/powerpc/platforms/pseries/pci_dlpar.c
+++ b/arch/powerpc/platforms/pseries/pci_dlpar.c
@@ -34,37 +34,6 @@
 
 #include "pseries.h"
 
-static struct pci_bus *
-find_bus_among_children(struct pci_bus *bus,
-                        struct device_node *dn)
-{
-	struct pci_bus *child = NULL;
-	struct pci_bus *tmp;
-	struct device_node *busdn;
-
-	busdn = pci_bus_to_OF_node(bus);
-	if (busdn == dn)
-		return bus;
-
-	list_for_each_entry(tmp, &bus->children, node) {
-		child = find_bus_among_children(tmp, dn);
-		if (child)
-			break;
-	};
-	return child;
-}
-
-struct pci_bus *pci_find_bus_by_node(struct device_node *dn)
-{
-	struct pci_dn *pdn = dn->data;
-
-	if (!pdn  || !pdn->phb || !pdn->phb->bus)
-		return NULL;
-
-	return find_bus_among_children(pdn->phb->bus, dn);
-}
-EXPORT_SYMBOL_GPL(pci_find_bus_by_node);
-
 struct pci_controller *init_phb_dynamic(struct device_node *dn)
 {
 	struct pci_controller *phb;
-- 
2.1.0

[PATCH v9 21/26] powerpc/pci: Export pci_add_device_node_info()

[Gavin Shan]

This renames update_dn_pci_info() to pci_add_device_node_info()
with corresponding adjustment on the parameter type and exports it.
The function is used to create pdn (struct pci_dn) for the indicated
device node. Another function add_pdn(), almost wrapper of
pci_add_device_node_info(), to be used in traverse_pci_devices(). No
logical changes introduced.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/include/asm/pci-bridge.h  |  3 ++-
 arch/powerpc/kernel/pci_dn.c           | 30 +++++++++++++++++++-----------
 arch/powerpc/platforms/pseries/setup.c |  2 +-
 3 files changed, 22 insertions(+), 13 deletions(-)

diff --git a/arch/powerpc/include/asm/pci-bridge.h b/arch/powerpc/include/asm/pci-bridge.h
index f0a5b16..07b94ec 100644
--- a/arch/powerpc/include/asm/pci-bridge.h
+++ b/arch/powerpc/include/asm/pci-bridge.h
@@ -235,7 +235,8 @@ extern struct pci_dn *pci_get_pdn_by_devfn(struct pci_bus *bus,
 extern struct pci_dn *pci_get_pdn(struct pci_dev *pdev);
 extern struct pci_dn *add_dev_pci_data(struct pci_dev *pdev);
 extern void remove_dev_pci_data(struct pci_dev *pdev);
-extern void *update_dn_pci_info(struct device_node *dn, void *data);
+extern struct pci_dn *pci_add_device_node_info(struct pci_controller *hose,
+					       struct device_node *dn);
 
 static inline int pci_device_from_OF_node(struct device_node *np,
 					  u8 *bus, u8 *devfn)
diff --git a/arch/powerpc/kernel/pci_dn.c b/arch/powerpc/kernel/pci_dn.c
index 38102cb..0a249ff 100644
--- a/arch/powerpc/kernel/pci_dn.c
+++ b/arch/powerpc/kernel/pci_dn.c
@@ -282,13 +282,9 @@ void remove_dev_pci_data(struct pci_dev *pdev)
 #endif /* CONFIG_PCI_IOV */
 }
 
-/*
- * Traverse_func that inits the PCI fields of the device node.
- * NOTE: this *must* be done before read/write config to the device.
- */
-void *update_dn_pci_info(struct device_node *dn, void *data)
+struct pci_dn *pci_add_device_node_info(struct pci_controller *hose,
+					struct device_node *dn)
 {
-	struct pci_controller *phb = data;
 	const __be32 *type = of_get_property(dn, "ibm,pci-config-space-type", NULL);
 	const __be32 *regs;
 	struct device_node *parent;
@@ -299,7 +295,7 @@ void *update_dn_pci_info(struct device_node *dn, void *data)
 		return NULL;
 	dn->data = pdn;
 	pdn->node = dn;
-	pdn->phb = phb;
+	pdn->phb = hose;
 #ifdef CONFIG_PPC_POWERNV
 	pdn->pe_number = IODA_INVALID_PE;
 #endif
@@ -331,8 +327,9 @@ void *update_dn_pci_info(struct device_node *dn, void *data)
 	if (pdn->parent)
 		list_add_tail(&pdn->list, &pdn->parent->child_list);
 
-	return NULL;
+	return pdn;
 }
+EXPORT_SYMBOL_GPL(pci_add_device_node_info);
 
 /*
  * Traverse a device tree stopping each PCI device in the tree.
@@ -432,6 +429,18 @@ void *traverse_pci_dn(struct pci_dn *root,
 	return NULL;
 }
 
+static void *add_pdn(struct device_node *dn, void *data)
+{
+	struct pci_controller *hose = data;
+	struct pci_dn *pdn;
+
+	pdn = pci_add_device_node_info(hose, dn);
+	if (!pdn)
+		return ERR_PTR(-ENOMEM);
+
+	return NULL;
+}
+
 /** 
  * pci_devs_phb_init_dynamic - setup pci devices under this PHB
  * phb: pci-to-host bridge (top-level bridge connecting to cpu)
@@ -446,8 +455,7 @@ void pci_devs_phb_init_dynamic(struct pci_controller *phb)
 	struct pci_dn *pdn;
 
 	/* PHB nodes themselves must not match */
-	update_dn_pci_info(dn, phb);
-	pdn = dn->data;
+	pdn = pci_add_device_node_info(phb, dn);
 	if (pdn) {
 		pdn->devfn = pdn->busno = -1;
 		pdn->vendor_id = pdn->device_id = pdn->class_code = 0;
@@ -456,7 +464,7 @@ void pci_devs_phb_init_dynamic(struct pci_controller *phb)
 	}
 
 	/* Update dn->phb ptrs for new phb and children devices */
-	traverse_pci_devices(dn, update_dn_pci_info, phb);
+	traverse_pci_devices(dn, add_pdn, phb);
 }
 
 /** 
diff --git a/arch/powerpc/platforms/pseries/setup.c b/arch/powerpc/platforms/pseries/setup.c
index 6e944fc..cd739da 100644
--- a/arch/powerpc/platforms/pseries/setup.c
+++ b/arch/powerpc/platforms/pseries/setup.c
@@ -265,7 +265,7 @@ static int pci_dn_reconfig_notifier(struct notifier_block *nb, unsigned long act
 		pdn = parent ? PCI_DN(parent) : NULL;
 		if (pdn) {
 			/* Create pdn and EEH device */
-			update_dn_pci_info(np, pdn->phb);
+			pci_add_device_node_info(pdn->phb, np);
 			eeh_dev_init(PCI_DN(np), pdn->phb);
 		}
 
-- 
2.1.0

[PATCH v9 22/26] powerpc/pci: Introduce pci_remove_device_node_info()

[Gavin Shan]

This implements and exports pci_remove_device_node_info(). It's
used to remove the pdn (struct pci_dn) for the indicated device
node. The function is going to be used by PowerNV PCI hotplug
driver.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/include/asm/pci-bridge.h |  1 +
 arch/powerpc/kernel/pci_dn.c          | 23 +++++++++++++++++++++++
 2 files changed, 24 insertions(+)

diff --git a/arch/powerpc/include/asm/pci-bridge.h b/arch/powerpc/include/asm/pci-bridge.h
index 07b94ec..467c0b0 100644
--- a/arch/powerpc/include/asm/pci-bridge.h
+++ b/arch/powerpc/include/asm/pci-bridge.h
@@ -237,6 +237,7 @@ extern struct pci_dn *add_dev_pci_data(struct pci_dev *pdev);
 extern void remove_dev_pci_data(struct pci_dev *pdev);
 extern struct pci_dn *pci_add_device_node_info(struct pci_controller *hose,
 					       struct device_node *dn);
+extern void pci_remove_device_node_info(struct device_node *dn);
 
 static inline int pci_device_from_OF_node(struct device_node *np,
 					  u8 *bus, u8 *devfn)
diff --git a/arch/powerpc/kernel/pci_dn.c b/arch/powerpc/kernel/pci_dn.c
index 0a249ff..ce10281 100644
--- a/arch/powerpc/kernel/pci_dn.c
+++ b/arch/powerpc/kernel/pci_dn.c
@@ -331,6 +331,29 @@ struct pci_dn *pci_add_device_node_info(struct pci_controller *hose,
 }
 EXPORT_SYMBOL_GPL(pci_add_device_node_info);
 
+void pci_remove_device_node_info(struct device_node *dn)
+{
+	struct pci_dn *pdn = dn ? PCI_DN(dn) : NULL;
+#ifdef CONFIG_EEH
+	struct eeh_dev *edev = pdn_to_eeh_dev(pdn);
+
+	if (edev)
+		edev->pdn = NULL;
+#endif
+
+	if (!pdn)
+		return;
+
+	WARN_ON(!list_empty(&pdn->child_list));
+	list_del(&pdn->list);
+	if (pdn->parent)
+		of_node_put(pdn->parent->node);
+
+	dn->data = NULL;
+	kfree(pdn);
+}
+EXPORT_SYMBOL_GPL(pci_remove_device_node_info);
+
 /*
  * Traverse a device tree stopping each PCI device in the tree.
  * This is done depth first.  As each node is processed, a "pre"
-- 
2.1.0

[PATCH v9 23/26] powerpc/pci: Export pci_traverse_device_nodes()

[Gavin Shan]

This renames traverse_pci_devices() to pci_traverse_device_nodes().
The function traverses all subordinate device nodes of the specified
one. Also, below cleanup applied to the function. No logical changes
introduced.

   * Rename "pre" to "fn".
   * Avoid assignment in if condition reported from checkpatch.pl.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/include/asm/ppc-pci.h   |  6 +++---
 arch/powerpc/kernel/pci_dn.c         | 15 ++++++++++-----
 arch/powerpc/platforms/pseries/msi.c |  4 ++--
 3 files changed, 15 insertions(+), 10 deletions(-)

diff --git a/arch/powerpc/include/asm/ppc-pci.h b/arch/powerpc/include/asm/ppc-pci.h
index ca0c5bf..8753e4e 100644
--- a/arch/powerpc/include/asm/ppc-pci.h
+++ b/arch/powerpc/include/asm/ppc-pci.h
@@ -33,9 +33,9 @@ extern struct pci_dev *isa_bridge_pcidev;	/* may be NULL if no ISA bus */
 struct device_node;
 struct pci_dn;
 
-typedef void *(*traverse_func)(struct device_node *me, void *data);
-void *traverse_pci_devices(struct device_node *start, traverse_func pre,
-		void *data);
+void *pci_traverse_device_nodes(struct device_node *start,
+				void *(*fn)(struct device_node *, void *),
+				void *data);
 void *traverse_pci_dn(struct pci_dn *root,
 		      void *(*fn)(struct pci_dn *, void *),
 		      void *data);
diff --git a/arch/powerpc/kernel/pci_dn.c b/arch/powerpc/kernel/pci_dn.c
index ce10281..ecdccce 100644
--- a/arch/powerpc/kernel/pci_dn.c
+++ b/arch/powerpc/kernel/pci_dn.c
@@ -372,8 +372,9 @@ EXPORT_SYMBOL_GPL(pci_remove_device_node_info);
  * one of these nodes we also assume its siblings are non-pci for
  * performance.
  */
-void *traverse_pci_devices(struct device_node *start, traverse_func pre,
-		void *data)
+void *pci_traverse_device_nodes(struct device_node *start,
+				void *(*fn)(struct device_node *, void *),
+				void *data)
 {
 	struct device_node *dn, *nextdn;
 	void *ret;
@@ -388,8 +389,11 @@ void *traverse_pci_devices(struct device_node *start, traverse_func pre,
 		if (classp)
 			class = of_read_number(classp, 1);
 
-		if (pre && ((ret = pre(dn, data)) != NULL))
-			return ret;
+		if (fn) {
+			ret = fn(dn, data);
+			if (ret)
+				return ret;
+		}
 
 		/* If we are a PCI bridge, go down */
 		if (dn->child && ((class >> 8) == PCI_CLASS_BRIDGE_PCI ||
@@ -411,6 +415,7 @@ void *traverse_pci_devices(struct device_node *start, traverse_func pre,
 	}
 	return NULL;
 }
+EXPORT_SYMBOL_GPL(pci_traverse_device_nodes);
 
 static struct pci_dn *pci_dn_next_one(struct pci_dn *root,
 				      struct pci_dn *pdn)
@@ -487,7 +492,7 @@ void pci_devs_phb_init_dynamic(struct pci_controller *phb)
 	}
 
 	/* Update dn->phb ptrs for new phb and children devices */
-	traverse_pci_devices(dn, add_pdn, phb);
+	pci_traverse_device_nodes(dn, add_pdn, phb);
 }
 
 /** 
diff --git a/arch/powerpc/platforms/pseries/msi.c b/arch/powerpc/platforms/pseries/msi.c
index 272e9ec..543a638 100644
--- a/arch/powerpc/platforms/pseries/msi.c
+++ b/arch/powerpc/platforms/pseries/msi.c
@@ -305,7 +305,7 @@ static int msi_quota_for_device(struct pci_dev *dev, int request)
 	memset(&counts, 0, sizeof(struct msi_counts));
 
 	/* Work out how many devices we have below this PE */
-	traverse_pci_devices(pe_dn, count_non_bridge_devices, &counts);
+	pci_traverse_device_nodes(pe_dn, count_non_bridge_devices, &counts);
 
 	if (counts.num_devices == 0) {
 		pr_err("rtas_msi: found 0 devices under PE for %s\n",
@@ -320,7 +320,7 @@ static int msi_quota_for_device(struct pci_dev *dev, int request)
 	/* else, we have some more calculating to do */
 	counts.requestor = pci_device_to_OF_node(dev);
 	counts.request = request;
-	traverse_pci_devices(pe_dn, count_spare_msis, &counts);
+	pci_traverse_device_nodes(pe_dn, count_spare_msis, &counts);
 
 	/* If the quota isn't an integer multiple of the total, we can
 	 * use the remainder as spare MSIs for anyone that wants them. */
-- 
2.1.0

[PATCH v9 24/26] powerpc/pci: Don't scan empty slot

[Gavin Shan]

In hotplug case, function pci_add_pci_devices() is called to rescan
the specified PCI bus, which might not have any child devices. Access
to the PCI bus's child device node will cause kernel crash without
exception.

This adds one more check to skip scanning PCI bus that doesn't have
any subordinate devices from device-tree, in order to avoid kernel
crash.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/kernel/pci-hotplug.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/powerpc/kernel/pci-hotplug.c b/arch/powerpc/kernel/pci-hotplug.c
index 46587a1..2d71269 100644
--- a/arch/powerpc/kernel/pci-hotplug.c
+++ b/arch/powerpc/kernel/pci-hotplug.c
@@ -120,7 +120,8 @@ void pci_hp_add_devices(struct pci_bus *bus)
 	if (mode == PCI_PROBE_DEVTREE) {
 		/* use ofdt-based probe */
 		of_rescan_bus(dn, bus);
-	} else if (mode == PCI_PROBE_NORMAL) {
+	} else if (mode == PCI_PROBE_NORMAL &&
+		   dn->child && PCI_DN(dn->child)) {
 		/*
 		 * Use legacy probe. In the partial hotplug case, we
 		 * probably have grandchildren devices unplugged. So
-- 
2.1.0

[PATCH v9 25/26] powerpc/powernv: Simplify pnv_eeh_reset()

[Gavin Shan]

This drops unnecessary nested if statements in pnv_eeh_reset() to
improve the code readability. After the changes, the unused local
variable "ret" is dropped as well. No logical changes introduced.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/platforms/powernv/eeh-powernv.c | 67 +++++++++++++---------------
 1 file changed, 31 insertions(+), 36 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/eeh-powernv.c b/arch/powerpc/platforms/powernv/eeh-powernv.c
index 69e41ce..9226df1 100644
--- a/arch/powerpc/platforms/powernv/eeh-powernv.c
+++ b/arch/powerpc/platforms/powernv/eeh-powernv.c
@@ -1009,8 +1009,9 @@ static int pnv_eeh_reset_vf_pe(struct eeh_pe *pe, int option)
 static int pnv_eeh_reset(struct eeh_pe *pe, int option)
 {
 	struct pci_controller *hose = pe->phb;
+	struct pnv_phb *phb;
 	struct pci_bus *bus;
-	int ret;
+	int64_t rc;
 
 	/*
 	 * For PHB reset, we always have complete reset. For those PEs whose
@@ -1026,45 +1027,39 @@ static int pnv_eeh_reset(struct eeh_pe *pe, int option)
 	 * reset. The side effect is that EEH core has to clear the frozen
 	 * state explicitly after BAR restore.
 	 */
-	if (pe->type & EEH_PE_PHB) {
-		ret = pnv_eeh_phb_reset(hose, option);
-	} else {
-		struct pnv_phb *phb;
-		s64 rc;
+	if (pe->type & EEH_PE_PHB)
+		return pnv_eeh_phb_reset(hose, option);
 
-		/*
-		 * The frozen PE might be caused by PAPR error injection
-		 * registers, which are expected to be cleared after hitting
-		 * frozen PE as stated in the hardware spec. Unfortunately,
-		 * that's not true on P7IOC. So we have to clear it manually
-		 * to avoid recursive EEH errors during recovery.
-		 */
-		phb = hose->private_data;
-		if (phb->model == PNV_PHB_MODEL_P7IOC &&
-		    (option == EEH_RESET_HOT ||
-		    option == EEH_RESET_FUNDAMENTAL)) {
-			rc = opal_pci_reset(phb->opal_id,
-					    OPAL_RESET_PHB_ERROR,
-					    OPAL_ASSERT_RESET);
-			if (rc != OPAL_SUCCESS) {
-				pr_warn("%s: Failure %lld clearing "
-					"error injection registers\n",
-					__func__, rc);
-				return -EIO;
-			}
+	/*
+	 * The frozen PE might be caused by PAPR error injection
+	 * registers, which are expected to be cleared after hitting
+	 * frozen PE as stated in the hardware spec. Unfortunately,
+	 * that's not true on P7IOC. So we have to clear it manually
+	 * to avoid recursive EEH errors during recovery.
+	 */
+	phb = hose->private_data;
+	if (phb->model == PNV_PHB_MODEL_P7IOC &&
+	    (option == EEH_RESET_HOT ||
+	     option == EEH_RESET_FUNDAMENTAL)) {
+		rc = opal_pci_reset(phb->opal_id,
+				    OPAL_RESET_PHB_ERROR,
+				    OPAL_ASSERT_RESET);
+		if (rc != OPAL_SUCCESS) {
+			pr_warn("%s: Failure %lld clearing error injection registers\n",
+				__func__, rc);
+			return -EIO;
 		}
-
-		bus = eeh_pe_bus_get(pe);
-		if (pe->type & EEH_PE_VF)
-			ret = pnv_eeh_reset_vf_pe(pe, option);
-		else if (pci_is_root_bus(bus) ||
-			pci_is_root_bus(bus->parent))
-			ret = pnv_eeh_root_reset(hose, option);
-		else
-			ret = pnv_eeh_bridge_reset(bus->self, option);
 	}
 
-	return ret;
+	bus = eeh_pe_bus_get(pe);
+	if (pe->type & EEH_PE_VF)
+		return pnv_eeh_reset_vf_pe(pe, option);
+
+	if (pci_is_root_bus(bus) ||
+	    pci_is_root_bus(bus->parent))
+		return pnv_eeh_root_reset(hose, option);
+
+	return pnv_eeh_bridge_reset(bus->self, option);
 }
 
 /**
-- 
2.1.0

[PATCH v9 26/26] powerpc/powernv: Exclude root bus in pnv_pci_reset_secondary_bus()

[Gavin Shan]

The function pnv_pci_reset_secondary_bus() is called like below.
It's impossible for call the function on root bus. So it's safe
to remove the root bus case in the function. No functional changes
introduced.

   pci_parent_bus_reset() / pci_bus_reset() / pci_try_reset_bus()
   pci_reset_bridge_secondary_bus()
   pcibios_reset_secondary_bus()
   pnv_pci_reset_secondary_bus()

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
Reviewed-by: Daniel Axtens <dja@axtens.net>
Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/platforms/powernv/eeh-powernv.c | 12 ++----------
 1 file changed, 2 insertions(+), 10 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/eeh-powernv.c b/arch/powerpc/platforms/powernv/eeh-powernv.c
index 9226df1..593b8dc 100644
--- a/arch/powerpc/platforms/powernv/eeh-powernv.c
+++ b/arch/powerpc/platforms/powernv/eeh-powernv.c
@@ -868,16 +868,8 @@ static int pnv_eeh_bridge_reset(struct pci_dev *dev, int option)
 
 void pnv_pci_reset_secondary_bus(struct pci_dev *dev)
 {
-	struct pci_controller *hose;
-
-	if (pci_is_root_bus(dev->bus)) {
-		hose = pci_bus_to_host(dev->bus);
-		pnv_eeh_root_reset(hose, EEH_RESET_HOT);
-		pnv_eeh_root_reset(hose, EEH_RESET_DEACTIVATE);
-	} else {
-		pnv_eeh_bridge_reset(dev, EEH_RESET_HOT);
-		pnv_eeh_bridge_reset(dev, EEH_RESET_DEACTIVATE);
-	}
+	pnv_eeh_bridge_reset(dev, EEH_RESET_HOT);
+	pnv_eeh_bridge_reset(dev, EEH_RESET_DEACTIVATE);
 }
 
 static void pnv_eeh_wait_for_pending(struct pci_dn *pdn, const char *type,
-- 
2.1.0

Re: [PATCH v9 07/26] powerpc/powernv: Fix initial IO and M32 segmap

[Alistair Popple]

On Tue, 3 May 2016 15:41:26 Gavin Shan wrote:
> There are two arrays for IO and M32 segment maps on every PHB.
> The index of the arrays are segment number and the value stored
> in the corresponding element is PE number, indicating the segment
> is assigned to the PE. Initially, all elements in those two arrays
> are zeroes, meaning all segments are assigned to PE#0. It's wrong.
> 
> This fixes the initial values in the elements of those two arrays
> to IODA_INVALID_PE, meaning all segments aren't assigned to any
> PE.
> 
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> ---
>  arch/powerpc/platforms/powernv/pci-ioda.c | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c 
b/arch/powerpc/platforms/powernv/pci-ioda.c
> index 4aa6cdf..59b20e5 100644
> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
> @@ -3240,6 +3240,7 @@ static void __init pnv_pci_init_ioda_phb(struct 
device_node *np,
>  	const __be64 *prop64;
>  	const __be32 *prop32;
>  	int len;
> +	unsigned int segno;
>  	u64 phb_id;
>  	void *aux;
>  	long rc;
> @@ -3334,8 +3335,13 @@ static void __init pnv_pci_init_ioda_phb(struct 
device_node *np,
>  	aux = memblock_virt_alloc(size, 0);
>  	phb->ioda.pe_alloc = aux;
>  	phb->ioda.m32_segmap = aux + m32map_off;
> -	if (phb->type == PNV_PHB_IODA1)
> +	for (segno = 0; segno < phb->ioda.total_pe_num; segno++)

These arrays are indexed by segment number but the upper bound is the total 
number of PEs. Does IODA1 & IODA2 hardware always have the same number of PE#s 
and segments? Is there any chance there could be more or less PE#s 
(total_pe_num) than segments?

- Alistair

> +		phb->ioda.m32_segmap[segno] = IODA_INVALID_PE;
> +	if (phb->type == PNV_PHB_IODA1) {
>  		phb->ioda.io_segmap = aux + iomap_off;
> +		for (segno = 0; segno < phb->ioda.total_pe_num; segno++)
> +			phb->ioda.io_segmap[segno] = IODA_INVALID_PE;
> +	}
>  	phb->ioda.pe_array = aux + pemap_off;
>  	set_bit(phb->ioda.reserved_pe_idx, phb->ioda.pe_alloc);
>  
> 

Re: [PATCH v9 06/26] powerpc/powernv: Data type unsigned int for PE number

[Alistair Popple]

I had a quick look at the surrounding code and couldn't see anything obvious 
that would result in a behaviour change.

Reviewed-By: Alistair Popple <alistair@popple.id.au>

On Tue, 3 May 2016 15:41:25 Gavin Shan wrote:
> This changes the data type of PE number from "int" to "unsigned int"
> in order to match the fact PE number is never negative:
> 
>    * The number of PE to which the specified PCI device is attached.
>    * The PE number map for SRIOV VFs.
>    * The returned PE number from pnv_ioda_alloc_pe().
>    * The returned PE number from pnv_ioda2_pick_m64_pe().
> 
> Suggested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> ---
>  arch/powerpc/include/asm/pci-bridge.h     | 6 +++---
>  arch/powerpc/platforms/powernv/pci-ioda.c | 8 ++++----
>  arch/powerpc/platforms/powernv/pci.c      | 2 +-
>  arch/powerpc/platforms/powernv/pci.h      | 2 +-
>  4 files changed, 9 insertions(+), 9 deletions(-)
> 
> diff --git a/arch/powerpc/include/asm/pci-bridge.h 
b/arch/powerpc/include/asm/pci-bridge.h
> index 023c8c8..220129f 100644
> --- a/arch/powerpc/include/asm/pci-bridge.h
> +++ b/arch/powerpc/include/asm/pci-bridge.h
> @@ -209,14 +209,14 @@ struct pci_dn {
>  #ifdef CONFIG_EEH
>  	struct eeh_dev *edev;		/* eeh device */
>  #endif
> -#define IODA_INVALID_PE		(-1)
> +#define IODA_INVALID_PE		0xFFFFFFFF
>  #ifdef CONFIG_PPC_POWERNV
> -	int	pe_number;
> +	unsigned int pe_number;
>  	int     vf_index;		/* VF index in the PF */
>  #ifdef CONFIG_PCI_IOV
>  	u16     vfs_expanded;		/* number of VFs IOV BAR expanded */
>  	u16     num_vfs;		/* number of VFs enabled*/
> -	int     *pe_num_map;		/* PE# for the first VF PE or array */
> +	unsigned int *pe_num_map;	/* PE# for the first VF PE or array */
>  	bool    m64_single_mode;	/* Use M64 BAR in Single Mode */
>  #define IODA_INVALID_M64        (-1)
>  	int     (*m64_map)[PCI_SRIOV_NUM_BARS];
> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c 
b/arch/powerpc/platforms/powernv/pci-ioda.c
> index 1d2514f..4aa6cdf 100644
> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
> @@ -138,7 +138,7 @@ static void pnv_ioda_reserve_pe(struct pnv_phb *phb, int 
pe_no)
>  	phb->ioda.pe_array[pe_no].pe_number = pe_no;
>  }
>  
> -static int pnv_ioda_alloc_pe(struct pnv_phb *phb)
> +static unsigned int pnv_ioda_alloc_pe(struct pnv_phb *phb)
>  {
>  	unsigned long pe;
>  
> @@ -261,7 +261,7 @@ static void pnv_ioda2_reserve_m64_pe(struct pci_bus 
*bus,
>  	}
>  }
>  
> -static int pnv_ioda2_pick_m64_pe(struct pci_bus *bus, bool all)
> +static unsigned int pnv_ioda2_pick_m64_pe(struct pci_bus *bus, bool all)
>  {
>  	struct pci_controller *hose = pci_bus_to_host(bus);
>  	struct pnv_phb *phb = hose->private_data;
> @@ -919,7 +919,7 @@ static struct pnv_ioda_pe *pnv_ioda_setup_dev_PE(struct 
pci_dev *dev)
>  	struct pnv_phb *phb = hose->private_data;
>  	struct pci_dn *pdn = pci_get_pdn(dev);
>  	struct pnv_ioda_pe *pe;
> -	int pe_num;
> +	unsigned int pe_num;
>  
>  	if (!pdn) {
>  		pr_err("%s: Device tree node not associated properly\n",
> @@ -1010,7 +1010,7 @@ static void pnv_ioda_setup_bus_PE(struct pci_bus *bus, 
bool all)
>  	struct pci_controller *hose = pci_bus_to_host(bus);
>  	struct pnv_phb *phb = hose->private_data;
>  	struct pnv_ioda_pe *pe;
> -	int pe_num = IODA_INVALID_PE;
> +	unsigned int pe_num = IODA_INVALID_PE;
>  
>  	/* Check if PE is determined by M64 */
>  	if (phb->pick_m64_pe)
> diff --git a/arch/powerpc/platforms/powernv/pci.c 
b/arch/powerpc/platforms/powernv/pci.c
> index afbaa1c..8827461 100644
> --- a/arch/powerpc/platforms/powernv/pci.c
> +++ b/arch/powerpc/platforms/powernv/pci.c
> @@ -370,7 +370,7 @@ static void pnv_pci_config_check_eeh(struct pci_dn *pdn)
>  	struct pnv_phb *phb = pdn->phb->private_data;
>  	u8	fstate;
>  	__be16	pcierr;
> -	int	pe_no;
> +	unsigned int pe_no;
>  	s64	rc;
>  
>  	/*
> diff --git a/arch/powerpc/platforms/powernv/pci.h 
b/arch/powerpc/platforms/powernv/pci.h
> index 784882a..66f2569 100644
> --- a/arch/powerpc/platforms/powernv/pci.h
> +++ b/arch/powerpc/platforms/powernv/pci.h
> @@ -113,7 +113,7 @@ struct pnv_phb {
>  	int (*init_m64)(struct pnv_phb *phb);
>  	void (*reserve_m64_pe)(struct pci_bus *bus,
>  			       unsigned long *pe_bitmap, bool all);
> -	int (*pick_m64_pe)(struct pci_bus *bus, bool all);
> +	unsigned int (*pick_m64_pe)(struct pci_bus *bus, bool all);
>  	int (*get_pe_state)(struct pnv_phb *phb, int pe_no);
>  	void (*freeze_pe)(struct pnv_phb *phb, int pe_no);
>  	int (*unfreeze_pe)(struct pnv_phb *phb, int pe_no, int opt);
> 

Re: [PATCH v9 08/26] powerpc/powernv: Simplify pnv_ioda_setup_pe_seg()

[Alistair Popple]

On Tue, 3 May 2016 15:41:27 Gavin Shan wrote:
> pnv_ioda_setup_pe_seg() associates the IO and M32 segments with the
> owner PE. The code mapping segments should be fixed and immune from
> logic changes introduced to pnv_ioda_setup_pe_seg().
> 
> This moves the code mapping segments to helper pnv_ioda_setup_pe_res().
> The data type for @rc is changed to "int64_t". Also, argument @hose is
> removed from pnv_ioda_setup_pe() as it can be got from @pe. No functional
> changes introduced.
> 
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>

Reviewed-By: Alistair Popple <alistair@popple.id.au>

> ---
>  arch/powerpc/platforms/powernv/pci-ioda.c | 121 +++++++++++++++---------------
>  1 file changed, 62 insertions(+), 59 deletions(-)
> 
> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
> index 59b20e5..b954fbc 100644
> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
> @@ -2929,19 +2929,72 @@ truncate_iov:
>  }
>  #endif /* CONFIG_PCI_IOV */
>  
> +static void pnv_ioda_setup_pe_res(struct pnv_ioda_pe *pe,
> +				  struct resource *res)
> +{
> +	struct pnv_phb *phb = pe->phb;
> +	struct pci_bus_region region;
> +	int index;
> +	int64_t rc;
> +
> +	if (!res || !res->flags || res->start > res->end)
> +		return;
> +
> +	if (res->flags & IORESOURCE_IO) {
> +		region.start = res->start - phb->ioda.io_pci_base;
> +		region.end   = res->end - phb->ioda.io_pci_base;
> +		index = region.start / phb->ioda.io_segsize;
> +
> +		while (index < phb->ioda.total_pe_num &&
> +		       region.start <= region.end) {
> +			phb->ioda.io_segmap[index] = pe->pe_number;
> +			rc = opal_pci_map_pe_mmio_window(phb->opal_id,
> +				pe->pe_number, OPAL_IO_WINDOW_TYPE, 0, index);
> +			if (rc != OPAL_SUCCESS) {
> +				pr_err("%s: Error %lld mappin
> g IO segment#%d to PE#%d\n",
> +				       __func__, rc, index, pe->pe_number);
> +				break;
> +			}
> +
> +			region.start += phb->ioda.io_segsize;
> +			index++;
> +		}
> +	} else if ((res->flags & IORESOURCE_MEM) &&
> +		   !pnv_pci_is_mem_pref_64(res->flags)) {
> +		region.start = res->start -
> +			       phb->hose->mem_offset[0] -
> +			       phb->ioda.m32_pci_base;
> +		region.end   = res->end -
> +			       phb->hose->mem_offset[0] -
> +			       phb->ioda.m32_pci_base;
> +		index = region.start / phb->ioda.m32_segsize;
> +
> +		while (index < phb->ioda.total_pe_num &&
> +		       region.start <= region.end) {
> +			phb->ioda.m32_segmap[index] = pe->pe_number;
> +			rc = opal_pci_map_pe_mmio_window(phb->opal_id,
> +				pe->pe_number, OPAL_M32_WINDOW_TYPE, 0, index);
> +			if (rc != OPAL_SUCCESS) {
> +				pr_err("%s: Error %lld mapping M32 segment#%d to PE#%d",
> +				       __func__, rc, index, pe->pe_number);
> +				break;
> +			}
> +
> +			region.start += phb->ioda.m32_segsize;
> +			index++;
> +		}
> +	}
> +}
> +
>  /*
>   * This function is supposed to be called on basis of PE from top
>   * to bottom style. So the the I/O or MMIO segment assigned to
>   * parent PE could be overrided by its child PEs if necessary.
>   */
> -static void pnv_ioda_setup_pe_seg(struct pci_controller *hose,
> -				  struct pnv_ioda_pe *pe)
> +static void pnv_ioda_setup_pe_seg(struct pnv_ioda_pe *pe)
>  {
> -	struct pnv_phb *phb = hose->private_data;
> -	struct pci_bus_region region;
>  	struct resource *res;
> -	int i, index;
> -	int rc;
> +	int i;
>  
>  	/*
>  	 * NOTE: We only care PCI bus based PE for now. For PCI
> @@ -2950,58 +3003,8 @@ static void pnv_ioda_setup_pe_seg(struct pci_controller *hose,
>  	 */
>  	BUG_ON(!(pe->flags & (PNV_IODA_PE_BUS | PNV_IODA_PE_BUS_ALL)));
>  
> -	pci_bus_for_each_resource(pe->pbus, res, i) {
> -		if (!res || !res->flags ||
> -		    res->start > res->end)
> -			continue;
> -
> -		if (res->flags & IORESOURCE_IO) {
> -			region.start = res->start - phb->ioda.io_pci_base;
> -			region.end   = res->end - phb->ioda.io_pci_base;
> -			index = region.start / phb->ioda.io_segsize;
> -
> -			while (index < phb->ioda.total_pe_num &&
> -			       region.start <= region.end) {
> -				phb->ioda.io_segmap[index] = pe->pe_number;
> -				rc = opal_pci_map_pe_mmio_window(phb->opal_id,
> -					pe->pe_number, OPAL_IO_WINDOW_TYPE, 0, index);
> -				if (rc != OPAL_SUCCESS) {
> -					pr_err("%s: OPAL error %d when mapping IO "
> -					       "segment #%d to PE#%d\n",
> -					       __func__, rc, index, pe->pe_number);
> -					break;
> -				}
> -
> -				region.start += phb->ioda.io_segsize;
> -				index++;
> -			}
> -		} else if ((res->flags & IORESOURCE_MEM) &&
> -			   !pnv_pci_is_mem_pref_64(res->flags)) {
> -			region.start = res->start -
> -				       hose->mem_offset[0] -
> -				       phb->ioda.m32_pci_base;
> -			region.end   = res->end -
> -				       hose->mem_offset[0] -
> -				       phb->ioda.m32_pci_base;
> -			index = region.start / phb->ioda.m32_segsize;
> -
> -			while (index < phb->ioda.total_pe_num &&
> -			       region.start <= region.end) {
> -				phb->ioda.m32_segmap[index] = pe->pe_number;
> -				rc = opal_pci_map_pe_mmio_window(phb->opal_id,
> -					pe->pe_number, OPAL_M32_WINDOW_TYPE, 0, index);
> -				if (rc != OPAL_SUCCESS) {
> -					pr_err("%s: OPAL error %d when mapping M32 "
> -					       "segment#%d to PE#%d",
> -					       __func__, rc, index, pe->pe_number);
> -					break;
> -				}
> -
> -				region.start += phb->ioda.m32_segsize;
> -				index++;
> -			}
> -		}
> -	}
> +	pci_bus_for_each_resource(pe->pbus, res, i)
> +		pnv_ioda_setup_pe_res(pe, res);
>  }
>  
>  static void pnv_pci_ioda_setup_seg(void)
> @@ -3018,7 +3021,7 @@ static void pnv_pci_ioda_setup_seg(void)
>  			continue;
>  
>  		list_for_each_entry(pe, &phb->ioda.pe_list, list) {
> -			pnv_ioda_setup_pe_seg(hose, pe);
> +			pnv_ioda_setup_pe_seg(pe);
>  		}
>  	}
>  }
> 

Re: [PATCH v9 14/26] powerpc/powernv/ioda1: Introduce PNV_IODA1_DMA32_SEGSIZE

[Alistair Popple]

On Tue, 3 May 2016 15:41:33 Gavin Shan wrote:
> Currently, there is one macro (TCE32_TABLE_SIZE) representing the
> TCE table size for one DMA32 segment. The constant representing
> the DMA32 segment size (1 << 28) is still used in the code.
> 
> This defines PNV_IODA1_DMA32_SEGSIZE representing one DMA32
> segment size. the TCE table size can be calcualted when the page
> has fixed 4KB size. So all the related calculation depends on one
> macro (PNV_IODA1_DMA32_SEGSIZE). No logical changes introduced.

Looks ok to me.

Reviewed-By: Alistair Popple <alistair@popple.id.au>

> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> ---
>  arch/powerpc/platforms/powernv/pci-ioda.c | 31 ++++++++++++++++++-------------
>  1 file changed, 18 insertions(+), 13 deletions(-)
> 
> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
> index f27a62d..ca034b8 100644
> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
> @@ -50,9 +50,7 @@
>  
>  #define PNV_IODA1_M64_NUM	16	/* Number of M64 BARs	*/
>  #define PNV_IODA1_M64_SEGS	8	/* Segments per M64 BAR	*/
> -
> -/* 256M DMA window, 4K TCE pages, 8 bytes TCE */
> -#define TCE32_TABLE_SIZE	((0x10000000 / 0x1000) * 8)
> +#define PNV_IODA1_DMA32_SEGSIZE	0x10000000
>  
>  #define POWERNV_IOMMU_DEFAULT_LEVELS	1
>  #define POWERNV_IOMMU_MAX_LEVELS	5
> @@ -2037,7 +2035,7 @@ static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
>  
>  	struct page *tce_mem = NULL;
>  	struct iommu_table *tbl;
> -	unsigned int i;
> +	unsigned int tce32_segsz, i;
>  	int64_t rc;
>  	void *addr;
>  
> @@ -2057,29 +2055,34 @@ static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
>  	/* Grab a 32-bit TCE table */
>  	pe->tce32_seg = base;
>  	pe_info(pe, " Setting up 32-bit TCE table at %08x..%08x\n",
> -		(base << 28), ((base + segs) << 28) - 1);
> +		base * PNV_IODA1_DMA32_SEGSIZE,
> +		(base + segs) * PNV_IODA1_DMA32_SEGSIZE - 1);
>  
>  	/* XXX Currently, we allocate one big contiguous table for the
>  	 * TCEs. We only really need one chunk per 256M of TCE space
>  	 * (ie per segment) but that's an optimization for later, it
>  	 * requires some added smarts with our get/put_tce implementation
> +	 *
> +	 * Each TCE page is 4KB in size and each TCE entry occupies 8
> +	 * bytes
>  	 */
> +	tce32_segsz = PNV_IODA1_DMA32_SEGSIZE >> (IOMMU_PAGE_SHIFT_4K - 3);
>  	tce_mem = alloc_pages_node(phb->hose->node, GFP_KERNEL,
> -				   get_order(TCE32_TABLE_SIZE * segs));
> +				   get_order(tce32_segsz * segs));
>  	if (!tce_mem) {
>  		pe_err(pe, " Failed to allocate a 32-bit TCE memory\n");
>  		goto fail;
>  	}
>  	addr = page_address(tce_mem);
> -	memset(addr, 0, TCE32_TABLE_SIZE * segs);
> +	memset(addr, 0, tce32_segsz * segs);
>  
>  	/* Configure HW */
>  	for (i = 0; i < segs; i++) {
>  		rc = opal_pci_map_pe_dma_window(phb->opal_id,
>  					      pe->pe_number,
>  					      base + i, 1,
> -					      __pa(addr) + TCE32_TABLE_SIZE * i,
> -					      TCE32_TABLE_SIZE, 0x1000);
> +					      __pa(addr) + tce32_segsz * i,
> +					      tce32_segsz, IOMMU_PAGE_SIZE_4K);
>  		if (rc) {
>  			pe_err(pe, " Failed to configure 32-bit TCE table,"
>  			       " err %ld\n", rc);
> @@ -2088,8 +2091,9 @@ static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
>  	}
>  
>  	/* Setup linux iommu table */
> -	pnv_pci_setup_iommu_table(tbl, addr, TCE32_TABLE_SIZE * segs,
> -				  base << 28, IOMMU_PAGE_SHIFT_4K);
> +	pnv_pci_setup_iommu_table(tbl, addr, tce32_segsz * segs,
> +				  base * PNV_IODA1_DMA32_SEGSIZE,
> +				  IOMMU_PAGE_SHIFT_4K);
>  
>  	/* OPAL variant of P7IOC SW invalidated TCEs */
>  	if (phb->ioda.tce_inval_reg)
> @@ -2119,7 +2123,7 @@ static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
>  	if (pe->tce32_seg >= 0)
>  		pe->tce32_seg = -1;
>  	if (tce_mem)
> -		__free_pages(tce_mem, get_order(TCE32_TABLE_SIZE * segs));
> +		__free_pages(tce_mem, get_order(tce32_segsz * segs));
>  	if (tbl) {
>  		pnv_pci_unlink_table_and_group(tbl, &pe->table_group);
>  		iommu_free_table(tbl, "pnv");
> @@ -3456,7 +3460,8 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
>  	mutex_init(&phb->ioda.pe_list_mutex);
>  
>  	/* Calculate how many 32-bit TCE segments we have */
> -	phb->ioda.tce32_count = phb->ioda.m32_pci_base >> 28;
> +	phb->ioda.tce32_count = phb->ioda.m32_pci_base /
> +				PNV_IODA1_DMA32_SEGSIZE;
>  
>  #if 0 /* We should really do that ... */
>  	rc = opal_pci_set_phb_mem_window(opal->phb_id,
> 

Re: [PATCH v9 18/26] powerpc/pci: Rename pcibios_{add, remove}_pci_devices()

[Alistair Popple]

On Tue, 3 May 2016 15:41:37 Gavin Shan wrote:
> This renames pcibios_{add,remove}_pci_devices() to avoid conflicts
> with names of the weak functions in PCI subsystem, which have the
> prefix "pcibios". No logical changes introduced.

I'm guessing this is just protecting against future weak PCI subsystem 
functions that may get these names? In other words none of these functions 
currently exist as weak PCI subsystem functions and hence aren't called from 
the PCI subsystem?

If so this just looks like a simple rename.

Reviewed-By: Alistair Popple <alistair@popple.id.au>

> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> ---
>  arch/powerpc/include/asm/pci-bridge.h |  4 ++--
>  arch/powerpc/kernel/eeh_driver.c      | 12 ++++++------
>  arch/powerpc/kernel/pci-hotplug.c     | 15 +++++++--------
>  drivers/pci/hotplug/rpadlpar_core.c   |  2 +-
>  drivers/pci/hotplug/rpaphp_core.c     |  4 ++--
>  drivers/pci/hotplug/rpaphp_pci.c      |  2 +-
>  6 files changed, 19 insertions(+), 20 deletions(-)
> 
> diff --git a/arch/powerpc/include/asm/pci-bridge.h 
b/arch/powerpc/include/asm/pci-bridge.h
> index 220129f..99027b8 100644
> --- a/arch/powerpc/include/asm/pci-bridge.h
> +++ b/arch/powerpc/include/asm/pci-bridge.h
> @@ -260,10 +260,10 @@ static inline struct eeh_dev *pdn_to_eeh_dev(struct 
pci_dn *pdn)
>  extern struct pci_bus *pcibios_find_pci_bus(struct device_node *dn);
>  
>  /** Remove all of the PCI devices under this bus */
> -extern void pcibios_remove_pci_devices(struct pci_bus *bus);
> +extern void pci_hp_remove_devices(struct pci_bus *bus);
>  
>  /** Discover new pci devices under this bus, and add them */
> -extern void pcibios_add_pci_devices(struct pci_bus *bus);
> +extern void pci_hp_add_devices(struct pci_bus *bus);
>  
>  
>  extern void isa_bridge_find_early(struct pci_controller *hose);
> diff --git a/arch/powerpc/kernel/eeh_driver.c 
b/arch/powerpc/kernel/eeh_driver.c
> index fb6207d..618d13c 100644
> --- a/arch/powerpc/kernel/eeh_driver.c
> +++ b/arch/powerpc/kernel/eeh_driver.c
> @@ -621,7 +621,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct 
pci_bus *bus,
>  	 * We don't remove the corresponding PE instances because
>  	 * we need the information afterwords. The attached EEH
>  	 * devices are expected to be attached soon when calling
> -	 * into pcibios_add_pci_devices().
> +	 * into pci_hp_add_devices().
>  	 */
>  	eeh_pe_state_mark(pe, EEH_PE_KEEP);
>  	if (bus) {
> @@ -630,7 +630,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct 
pci_bus *bus,
>  		} else {
>  			eeh_pe_state_clear(pe, EEH_PE_PRI_BUS);
>  			pci_lock_rescan_remove();
> -			pcibios_remove_pci_devices(bus);
> +			pci_hp_remove_devices(bus);
>  			pci_unlock_rescan_remove();
>  		}
>  	} else if (frozen_bus) {
> @@ -681,7 +681,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct 
pci_bus *bus,
>  		if (pe->type & EEH_PE_VF)
>  			eeh_add_virt_device(edev, NULL);
>  		else
> -			pcibios_add_pci_devices(bus);
> +			pci_hp_add_devices(bus);
>  	} else if (frozen_bus && rmv_data->removed) {
>  		pr_info("EEH: Sleep 5s ahead of partial hotplug\n");
>  		ssleep(5);
> @@ -691,7 +691,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct 
pci_bus *bus,
>  		if (pe->type & EEH_PE_VF)
>  			eeh_add_virt_device(edev, NULL);
>  		else
> -			pcibios_add_pci_devices(frozen_bus);
> +			pci_hp_add_devices(frozen_bus);
>  	}
>  	eeh_pe_state_clear(pe, EEH_PE_KEEP);
>  
> @@ -896,7 +896,7 @@ perm_error:
>  			eeh_pe_dev_mode_mark(pe, EEH_DEV_REMOVED);
>  
>  			pci_lock_rescan_remove();
> -			pcibios_remove_pci_devices(frozen_bus);
> +			pci_hp_remove_devices(frozen_bus);
>  			pci_unlock_rescan_remove();
>  		}
>  	}
> @@ -981,7 +981,7 @@ static void eeh_handle_special_event(void)
>  				bus = eeh_pe_bus_get(phb_pe);
>  				eeh_pe_dev_traverse(pe,
>  					eeh_report_failure, NULL);
> -				pcibios_remove_pci_devices(bus);
> +				pci_hp_remove_devices(bus);
>  			}
>  			pci_unlock_rescan_remove();
>  		}
> diff --git a/arch/powerpc/kernel/pci-hotplug.c b/arch/powerpc/kernel/pci-
hotplug.c
> index 59c4361..2d108e5 100644
> --- a/arch/powerpc/kernel/pci-hotplug.c
> +++ b/arch/powerpc/kernel/pci-hotplug.c
> @@ -38,20 +38,20 @@ void pcibios_release_device(struct pci_dev *dev)
>  }
>  
>  /**
> - * pcibios_remove_pci_devices - remove all devices under this bus
> + * pci_hp_remove_devices - remove all devices under this bus
>   * @bus: the indicated PCI bus
>   *
>   * Remove all of the PCI devices under this bus both from the
>   * linux pci device tree, and from the powerpc EEH address cache.
>   */
> -void pcibios_remove_pci_devices(struct pci_bus *bus)
> +void pci_hp_remove_devices(struct pci_bus *bus)
>  {
>  	struct pci_dev *dev, *tmp;
>  	struct pci_bus *child_bus;
>  
>  	/* First go down child busses */
>  	list_for_each_entry(child_bus, &bus->children, node)
> -		pcibios_remove_pci_devices(child_bus);
> +		pci_hp_remove_devices(child_bus);
>  
>  	pr_debug("PCI: Removing devices on bus %04x:%02x\n",
>  		 pci_domain_nr(bus),  bus->number);
> @@ -60,11 +60,10 @@ void pcibios_remove_pci_devices(struct pci_bus *bus)
>  		pci_stop_and_remove_bus_device(dev);
>  	}
>  }
> -
> -EXPORT_SYMBOL_GPL(pcibios_remove_pci_devices);
> +EXPORT_SYMBOL_GPL(pci_hp_remove_devices);
>  
>  /**
> - * pcibios_add_pci_devices - adds new pci devices to bus
> + * pci_hp_add_devices - adds new pci devices to bus
>   * @bus: the indicated PCI bus
>   *
>   * This routine will find and fixup new pci devices under
> @@ -74,7 +73,7 @@ EXPORT_SYMBOL_GPL(pcibios_remove_pci_devices);
>   * is how this routine differs from other, similar pcibios
>   * routines.)
>   */
> -void pcibios_add_pci_devices(struct pci_bus * bus)
> +void pci_hp_add_devices(struct pci_bus *bus)
>  {
>  	int slotno, mode, pass, max;
>  	struct pci_dev *dev;
> @@ -114,4 +113,4 @@ void pcibios_add_pci_devices(struct pci_bus * bus)
>  	}
>  	pcibios_finish_adding_to_bus(bus);
>  }
> -EXPORT_SYMBOL_GPL(pcibios_add_pci_devices);
> +EXPORT_SYMBOL_GPL(pci_hp_add_devices);
> diff --git a/drivers/pci/hotplug/rpadlpar_core.c 
b/drivers/pci/hotplug/rpadlpar_core.c
> index b46b57d..b770d6d 100644
> --- a/drivers/pci/hotplug/rpadlpar_core.c
> +++ b/drivers/pci/hotplug/rpadlpar_core.c
> @@ -380,7 +380,7 @@ int dlpar_remove_pci_slot(char *drc_name, struct 
device_node *dn)
>  	}
>  
>  	/* Remove all devices below slot */
> -	pcibios_remove_pci_devices(bus);
> +	pci_hp_remove_devices(bus);
>  
>  	/* Unmap PCI IO space */
>  	if (pcibios_unmap_io_space(bus)) {
> diff --git a/drivers/pci/hotplug/rpaphp_core.c 
b/drivers/pci/hotplug/rpaphp_core.c
> index 611f605..8d13202 100644
> --- a/drivers/pci/hotplug/rpaphp_core.c
> +++ b/drivers/pci/hotplug/rpaphp_core.c
> @@ -404,7 +404,7 @@ static int enable_slot(struct hotplug_slot 
*hotplug_slot)
>  
>  	if (state == PRESENT) {
>  		pci_lock_rescan_remove();
> -		pcibios_add_pci_devices(slot->bus);
> +		pci_hp_add_devices(slot->bus);
>  		pci_unlock_rescan_remove();
>  		slot->state = CONFIGURED;
>  	} else if (state == EMPTY) {
> @@ -426,7 +426,7 @@ static int disable_slot(struct hotplug_slot 
*hotplug_slot)
>  		return -EINVAL;
>  
>  	pci_lock_rescan_remove();
> -	pcibios_remove_pci_devices(slot->bus);
> +	pci_hp_remove_devices(slot->bus);
>  	pci_unlock_rescan_remove();
>  	vm_unmap_aliases();
>  
> diff --git a/drivers/pci/hotplug/rpaphp_pci.c 
b/drivers/pci/hotplug/rpaphp_pci.c
> index 7836d69..137a892 100644
> --- a/drivers/pci/hotplug/rpaphp_pci.c
> +++ b/drivers/pci/hotplug/rpaphp_pci.c
> @@ -116,7 +116,7 @@ int rpaphp_enable_slot(struct slot *slot)
>  		}
>  
>  		if (list_empty(&bus->devices))
> -			pcibios_add_pci_devices(bus);
> +			pci_hp_add_devices(bus);
>  
>  		if (!list_empty(&bus->devices)) {
>  			info->adapter_status = CONFIGURED;
> 

Re: [PATCH v9 07/26] powerpc/powernv: Fix initial IO and M32 segmap

[Gavin Shan]

On Wed, May 04, 2016 at 01:31:04PM +1000, Alistair Popple wrote:
>On Tue, 3 May 2016 15:41:26 Gavin Shan wrote:
>> There are two arrays for IO and M32 segment maps on every PHB.
>> The index of the arrays are segment number and the value stored
>> in the corresponding element is PE number, indicating the segment
>> is assigned to the PE. Initially, all elements in those two arrays
>> are zeroes, meaning all segments are assigned to PE#0. It's wrong.
>> 
>> This fixes the initial values in the elements of those two arrays
>> to IODA_INVALID_PE, meaning all segments aren't assigned to any
>> PE.
>> 
>> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>> ---
>>  arch/powerpc/platforms/powernv/pci-ioda.c | 8 +++++++-
>>  1 file changed, 7 insertions(+), 1 deletion(-)
>> 
>> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c 
>b/arch/powerpc/platforms/powernv/pci-ioda.c
>> index 4aa6cdf..59b20e5 100644
>> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
>> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
>> @@ -3240,6 +3240,7 @@ static void __init pnv_pci_init_ioda_phb(struct 
>device_node *np,
>>  	const __be64 *prop64;
>>  	const __be32 *prop32;
>>  	int len;
>> +	unsigned int segno;
>>  	u64 phb_id;
>>  	void *aux;
>>  	long rc;
>> @@ -3334,8 +3335,13 @@ static void __init pnv_pci_init_ioda_phb(struct 
>device_node *np,
>>  	aux = memblock_virt_alloc(size, 0);
>>  	phb->ioda.pe_alloc = aux;
>>  	phb->ioda.m32_segmap = aux + m32map_off;
>> -	if (phb->type == PNV_PHB_IODA1)
>> +	for (segno = 0; segno < phb->ioda.total_pe_num; segno++)
>
>These arrays are indexed by segment number but the upper bound is the total 
>number of PEs. Does IODA1 & IODA2 hardware always have the same number of PE#s 
>and segments? Is there any chance there could be more or less PE#s 
>(total_pe_num) than segments?
>

Alistair, thanks for review the code in time. The total number of M32 segments
and PEs are always equal on IODA1/IODA2.

Thanks,
Gavin

>- Alistair
>
>> +		phb->ioda.m32_segmap[segno] = IODA_INVALID_PE;
>> +	if (phb->type == PNV_PHB_IODA1) {
>>  		phb->ioda.io_segmap = aux + iomap_off;
>> +		for (segno = 0; segno < phb->ioda.total_pe_num; segno++)
>> +			phb->ioda.io_segmap[segno] = IODA_INVALID_PE;
>> +	}
>>  	phb->ioda.pe_array = aux + pemap_off;
>>  	set_bit(phb->ioda.reserved_pe_idx, phb->ioda.pe_alloc);
>>  
>> 
>

Re: [PATCH v9 18/26] powerpc/pci: Rename pcibios_{add, remove}_pci_devices()

[Andrew Donnellan]

On 03/05/16 15:41, Gavin Shan wrote:
> This renames pcibios_{add,remove}_pci_devices() to avoid conflicts
> with names of the weak functions in PCI subsystem, which have the
> prefix "pcibios". No logical changes introduced.
>
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>

Looks fine to me.

Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>

> ---
>   arch/powerpc/include/asm/pci-bridge.h |  4 ++--
>   arch/powerpc/kernel/eeh_driver.c      | 12 ++++++------
>   arch/powerpc/kernel/pci-hotplug.c     | 15 +++++++--------
>   drivers/pci/hotplug/rpadlpar_core.c   |  2 +-
>   drivers/pci/hotplug/rpaphp_core.c     |  4 ++--
>   drivers/pci/hotplug/rpaphp_pci.c      |  2 +-
>   6 files changed, 19 insertions(+), 20 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/pci-bridge.h b/arch/powerpc/include/asm/pci-bridge.h
> index 220129f..99027b8 100644
> --- a/arch/powerpc/include/asm/pci-bridge.h
> +++ b/arch/powerpc/include/asm/pci-bridge.h
> @@ -260,10 +260,10 @@ static inline struct eeh_dev *pdn_to_eeh_dev(struct pci_dn *pdn)
>   extern struct pci_bus *pcibios_find_pci_bus(struct device_node *dn);
>
>   /** Remove all of the PCI devices under this bus */
> -extern void pcibios_remove_pci_devices(struct pci_bus *bus);
> +extern void pci_hp_remove_devices(struct pci_bus *bus);
>
>   /** Discover new pci devices under this bus, and add them */
> -extern void pcibios_add_pci_devices(struct pci_bus *bus);
> +extern void pci_hp_add_devices(struct pci_bus *bus);
>
>
>   extern void isa_bridge_find_early(struct pci_controller *hose);
> diff --git a/arch/powerpc/kernel/eeh_driver.c b/arch/powerpc/kernel/eeh_driver.c
> index fb6207d..618d13c 100644
> --- a/arch/powerpc/kernel/eeh_driver.c
> +++ b/arch/powerpc/kernel/eeh_driver.c
> @@ -621,7 +621,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus,
>   	 * We don't remove the corresponding PE instances because
>   	 * we need the information afterwords. The attached EEH
>   	 * devices are expected to be attached soon when calling
> -	 * into pcibios_add_pci_devices().
> +	 * into pci_hp_add_devices().
>   	 */
>   	eeh_pe_state_mark(pe, EEH_PE_KEEP);
>   	if (bus) {
> @@ -630,7 +630,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus,
>   		} else {
>   			eeh_pe_state_clear(pe, EEH_PE_PRI_BUS);
>   			pci_lock_rescan_remove();
> -			pcibios_remove_pci_devices(bus);
> +			pci_hp_remove_devices(bus);
>   			pci_unlock_rescan_remove();
>   		}
>   	} else if (frozen_bus) {
> @@ -681,7 +681,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus,
>   		if (pe->type & EEH_PE_VF)
>   			eeh_add_virt_device(edev, NULL);
>   		else
> -			pcibios_add_pci_devices(bus);
> +			pci_hp_add_devices(bus);
>   	} else if (frozen_bus && rmv_data->removed) {
>   		pr_info("EEH: Sleep 5s ahead of partial hotplug\n");
>   		ssleep(5);
> @@ -691,7 +691,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus,
>   		if (pe->type & EEH_PE_VF)
>   			eeh_add_virt_device(edev, NULL);
>   		else
> -			pcibios_add_pci_devices(frozen_bus);
> +			pci_hp_add_devices(frozen_bus);
>   	}
>   	eeh_pe_state_clear(pe, EEH_PE_KEEP);
>
> @@ -896,7 +896,7 @@ perm_error:
>   			eeh_pe_dev_mode_mark(pe, EEH_DEV_REMOVED);
>
>   			pci_lock_rescan_remove();
> -			pcibios_remove_pci_devices(frozen_bus);
> +			pci_hp_remove_devices(frozen_bus);
>   			pci_unlock_rescan_remove();
>   		}
>   	}
> @@ -981,7 +981,7 @@ static void eeh_handle_special_event(void)
>   				bus = eeh_pe_bus_get(phb_pe);
>   				eeh_pe_dev_traverse(pe,
>   					eeh_report_failure, NULL);
> -				pcibios_remove_pci_devices(bus);
> +				pci_hp_remove_devices(bus);
>   			}
>   			pci_unlock_rescan_remove();
>   		}
> diff --git a/arch/powerpc/kernel/pci-hotplug.c b/arch/powerpc/kernel/pci-hotplug.c
> index 59c4361..2d108e5 100644
> --- a/arch/powerpc/kernel/pci-hotplug.c
> +++ b/arch/powerpc/kernel/pci-hotplug.c
> @@ -38,20 +38,20 @@ void pcibios_release_device(struct pci_dev *dev)
>   }
>
>   /**
> - * pcibios_remove_pci_devices - remove all devices under this bus
> + * pci_hp_remove_devices - remove all devices under this bus
>    * @bus: the indicated PCI bus
>    *
>    * Remove all of the PCI devices under this bus both from the
>    * linux pci device tree, and from the powerpc EEH address cache.
>    */
> -void pcibios_remove_pci_devices(struct pci_bus *bus)
> +void pci_hp_remove_devices(struct pci_bus *bus)
>   {
>   	struct pci_dev *dev, *tmp;
>   	struct pci_bus *child_bus;
>
>   	/* First go down child busses */
>   	list_for_each_entry(child_bus, &bus->children, node)
> -		pcibios_remove_pci_devices(child_bus);
> +		pci_hp_remove_devices(child_bus);
>
>   	pr_debug("PCI: Removing devices on bus %04x:%02x\n",
>   		 pci_domain_nr(bus),  bus->number);
> @@ -60,11 +60,10 @@ void pcibios_remove_pci_devices(struct pci_bus *bus)
>   		pci_stop_and_remove_bus_device(dev);
>   	}
>   }
> -
> -EXPORT_SYMBOL_GPL(pcibios_remove_pci_devices);
> +EXPORT_SYMBOL_GPL(pci_hp_remove_devices);
>
>   /**
> - * pcibios_add_pci_devices - adds new pci devices to bus
> + * pci_hp_add_devices - adds new pci devices to bus
>    * @bus: the indicated PCI bus
>    *
>    * This routine will find and fixup new pci devices under
> @@ -74,7 +73,7 @@ EXPORT_SYMBOL_GPL(pcibios_remove_pci_devices);
>    * is how this routine differs from other, similar pcibios
>    * routines.)
>    */
> -void pcibios_add_pci_devices(struct pci_bus * bus)
> +void pci_hp_add_devices(struct pci_bus *bus)
>   {
>   	int slotno, mode, pass, max;
>   	struct pci_dev *dev;
> @@ -114,4 +113,4 @@ void pcibios_add_pci_devices(struct pci_bus * bus)
>   	}
>   	pcibios_finish_adding_to_bus(bus);
>   }
> -EXPORT_SYMBOL_GPL(pcibios_add_pci_devices);
> +EXPORT_SYMBOL_GPL(pci_hp_add_devices);
> diff --git a/drivers/pci/hotplug/rpadlpar_core.c b/drivers/pci/hotplug/rpadlpar_core.c
> index b46b57d..b770d6d 100644
> --- a/drivers/pci/hotplug/rpadlpar_core.c
> +++ b/drivers/pci/hotplug/rpadlpar_core.c
> @@ -380,7 +380,7 @@ int dlpar_remove_pci_slot(char *drc_name, struct device_node *dn)
>   	}
>
>   	/* Remove all devices below slot */
> -	pcibios_remove_pci_devices(bus);
> +	pci_hp_remove_devices(bus);
>
>   	/* Unmap PCI IO space */
>   	if (pcibios_unmap_io_space(bus)) {
> diff --git a/drivers/pci/hotplug/rpaphp_core.c b/drivers/pci/hotplug/rpaphp_core.c
> index 611f605..8d13202 100644
> --- a/drivers/pci/hotplug/rpaphp_core.c
> +++ b/drivers/pci/hotplug/rpaphp_core.c
> @@ -404,7 +404,7 @@ static int enable_slot(struct hotplug_slot *hotplug_slot)
>
>   	if (state == PRESENT) {
>   		pci_lock_rescan_remove();
> -		pcibios_add_pci_devices(slot->bus);
> +		pci_hp_add_devices(slot->bus);
>   		pci_unlock_rescan_remove();
>   		slot->state = CONFIGURED;
>   	} else if (state == EMPTY) {
> @@ -426,7 +426,7 @@ static int disable_slot(struct hotplug_slot *hotplug_slot)
>   		return -EINVAL;
>
>   	pci_lock_rescan_remove();
> -	pcibios_remove_pci_devices(slot->bus);
> +	pci_hp_remove_devices(slot->bus);
>   	pci_unlock_rescan_remove();
>   	vm_unmap_aliases();
>
> diff --git a/drivers/pci/hotplug/rpaphp_pci.c b/drivers/pci/hotplug/rpaphp_pci.c
> index 7836d69..137a892 100644
> --- a/drivers/pci/hotplug/rpaphp_pci.c
> +++ b/drivers/pci/hotplug/rpaphp_pci.c
> @@ -116,7 +116,7 @@ int rpaphp_enable_slot(struct slot *slot)
>   		}
>
>   		if (list_empty(&bus->devices))
> -			pcibios_add_pci_devices(bus);
> +			pci_hp_add_devices(bus);
>
>   		if (!list_empty(&bus->devices)) {
>   			info->adapter_status = CONFIGURED;
>

-- 
Andrew Donnellan              OzLabs, ADL Canberra
andrew.donnellan@au1.ibm.com  IBM Australia Limited

Re: [PATCH v9 20/26] powerpc/pci: Move pci_find_bus_by_node() around

[Andrew Donnellan]

On 03/05/16 15:41, Gavin Shan wrote:
> This moves pci_find_bus_by_node() from arch/powerpc/platforms/
> pseries/pci_dlpar.c to arch/powerpc/kernel/pci-hotplug.c so that
> the function can be used by pSeries and PowerNV platform at the
> same time. Also, below cleanup applied. No functional changes
> introduced.
>
>     * Remove variable "busdn" in find_bus_among_children()
>     * Use PCI_DN() to convert device node to pci_dn
>
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> Acked-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>

Looks good to me

Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>

> ---
>   arch/powerpc/kernel/pci-hotplug.c          | 29 ++++++++++++++++++++++++++++
>   arch/powerpc/platforms/pseries/pci_dlpar.c | 31 ------------------------------
>   2 files changed, 29 insertions(+), 31 deletions(-)
>
> diff --git a/arch/powerpc/kernel/pci-hotplug.c b/arch/powerpc/kernel/pci-hotplug.c
> index 2d108e5..46587a1 100644
> --- a/arch/powerpc/kernel/pci-hotplug.c
> +++ b/arch/powerpc/kernel/pci-hotplug.c
> @@ -21,6 +21,35 @@
>   #include <asm/firmware.h>
>   #include <asm/eeh.h>
>
> +static struct pci_bus *find_bus_among_children(struct pci_bus *bus,
> +					       struct device_node *dn)
> +{
> +	struct pci_bus *child = NULL;
> +	struct pci_bus *tmp;
> +
> +	if (pci_bus_to_OF_node(bus) == dn)
> +		return bus;
> +
> +	list_for_each_entry(tmp, &bus->children, node) {
> +		child = find_bus_among_children(tmp, dn);
> +		if (child)
> +			break;
> +	}
> +
> +	return child;
> +}
> +
> +struct pci_bus *pci_find_bus_by_node(struct device_node *dn)
> +{
> +	struct pci_dn *pdn = PCI_DN(dn);
> +
> +	if (!pdn  || !pdn->phb || !pdn->phb->bus)
> +		return NULL;
> +
> +	return find_bus_among_children(pdn->phb->bus, dn);
> +}
> +EXPORT_SYMBOL_GPL(pci_find_bus_by_node);
> +
>   /**
>    * pcibios_release_device - release PCI device
>    * @dev: PCI device
> diff --git a/arch/powerpc/platforms/pseries/pci_dlpar.c b/arch/powerpc/platforms/pseries/pci_dlpar.c
> index aee22b4..906dbaa 100644
> --- a/arch/powerpc/platforms/pseries/pci_dlpar.c
> +++ b/arch/powerpc/platforms/pseries/pci_dlpar.c
> @@ -34,37 +34,6 @@
>
>   #include "pseries.h"
>
> -static struct pci_bus *
> -find_bus_among_children(struct pci_bus *bus,
> -                        struct device_node *dn)
> -{
> -	struct pci_bus *child = NULL;
> -	struct pci_bus *tmp;
> -	struct device_node *busdn;
> -
> -	busdn = pci_bus_to_OF_node(bus);
> -	if (busdn == dn)
> -		return bus;
> -
> -	list_for_each_entry(tmp, &bus->children, node) {
> -		child = find_bus_among_children(tmp, dn);
> -		if (child)
> -			break;
> -	};
> -	return child;
> -}
> -
> -struct pci_bus *pci_find_bus_by_node(struct device_node *dn)
> -{
> -	struct pci_dn *pdn = dn->data;
> -
> -	if (!pdn  || !pdn->phb || !pdn->phb->bus)
> -		return NULL;
> -
> -	return find_bus_among_children(pdn->phb->bus, dn);
> -}
> -EXPORT_SYMBOL_GPL(pci_find_bus_by_node);
> -
>   struct pci_controller *init_phb_dynamic(struct device_node *dn)
>   {
>   	struct pci_controller *phb;
>

-- 
Andrew Donnellan              OzLabs, ADL Canberra
andrew.donnellan@au1.ibm.com  IBM Australia Limited

Re: [PATCH v9 18/26] powerpc/pci: Rename pcibios_{add,remove}_pci_devices()

[Gavin Shan]

On Wed, May 04, 2016 at 02:10:47PM +1000, Alistair Popple wrote:
>On Tue, 3 May 2016 15:41:37 Gavin Shan wrote:
>> This renames pcibios_{add,remove}_pci_devices() to avoid conflicts
>> with names of the weak functions in PCI subsystem, which have the
>> prefix "pcibios". No logical changes introduced.
>
>I'm guessing this is just protecting against future weak PCI subsystem 
>functions that may get these names? In other words none of these functions 
>currently exist as weak PCI subsystem functions and hence aren't called from 
>the PCI subsystem?
>
>If so this just looks like a simple rename.
>

Thanks for review, Alistair. Yes, it's simple rename. In PCI subsystem,
the weak funtion names have prefix pcibios_. Those functions are not weak
functions. Also, The functions are exported so that they can be used in PCI
hotplug driver (module) and it's worthy to have better names for them.

Those functions are called in PCI hotplug path to accomodate request from
EEH or PCI hotplug driver.

Thanks,
Gavin

>Reviewed-By: Alistair Popple <alistair@popple.id.au>
>
>> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>> ---
>>  arch/powerpc/include/asm/pci-bridge.h |  4 ++--
>>  arch/powerpc/kernel/eeh_driver.c      | 12 ++++++------
>>  arch/powerpc/kernel/pci-hotplug.c     | 15 +++++++--------
>>  drivers/pci/hotplug/rpadlpar_core.c   |  2 +-
>>  drivers/pci/hotplug/rpaphp_core.c     |  4 ++--
>>  drivers/pci/hotplug/rpaphp_pci.c      |  2 +-
>>  6 files changed, 19 insertions(+), 20 deletions(-)
>> 
>> diff --git a/arch/powerpc/include/asm/pci-bridge.h 
>b/arch/powerpc/include/asm/pci-bridge.h
>> index 220129f..99027b8 100644
>> --- a/arch/powerpc/include/asm/pci-bridge.h
>> +++ b/arch/powerpc/include/asm/pci-bridge.h
>> @@ -260,10 +260,10 @@ static inline struct eeh_dev *pdn_to_eeh_dev(struct 
>pci_dn *pdn)
>>  extern struct pci_bus *pcibios_find_pci_bus(struct device_node *dn);
>>  
>>  /** Remove all of the PCI devices under this bus */
>> -extern void pcibios_remove_pci_devices(struct pci_bus *bus);
>> +extern void pci_hp_remove_devices(struct pci_bus *bus);
>>  
>>  /** Discover new pci devices under this bus, and add them */
>> -extern void pcibios_add_pci_devices(struct pci_bus *bus);
>> +extern void pci_hp_add_devices(struct pci_bus *bus);
>>  
>>  
>>  extern void isa_bridge_find_early(struct pci_controller *hose);
>> diff --git a/arch/powerpc/kernel/eeh_driver.c 
>b/arch/powerpc/kernel/eeh_driver.c
>> index fb6207d..618d13c 100644
>> --- a/arch/powerpc/kernel/eeh_driver.c
>> +++ b/arch/powerpc/kernel/eeh_driver.c
>> @@ -621,7 +621,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct 
>pci_bus *bus,
>>  	 * We don't remove the corresponding PE instances because
>>  	 * we need the information afterwords. The attached EEH
>>  	 * devices are expected to be attached soon when calling
>> -	 * into pcibios_add_pci_devices().
>> +	 * into pci_hp_add_devices().
>>  	 */
>>  	eeh_pe_state_mark(pe, EEH_PE_KEEP);
>>  	if (bus) {
>> @@ -630,7 +630,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct 
>pci_bus *bus,
>>  		} else {
>>  			eeh_pe_state_clear(pe, EEH_PE_PRI_BUS);
>>  			pci_lock_rescan_remove();
>> -			pcibios_remove_pci_devices(bus);
>> +			pci_hp_remove_devices(bus);
>>  			pci_unlock_rescan_remove();
>>  		}
>>  	} else if (frozen_bus) {
>> @@ -681,7 +681,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct 
>pci_bus *bus,
>>  		if (pe->type & EEH_PE_VF)
>>  			eeh_add_virt_device(edev, NULL);
>>  		else
>> -			pcibios_add_pci_devices(bus);
>> +			pci_hp_add_devices(bus);
>>  	} else if (frozen_bus && rmv_data->removed) {
>>  		pr_info("EEH: Sleep 5s ahead of partial hotplug\n");
>>  		ssleep(5);
>> @@ -691,7 +691,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct 
>pci_bus *bus,
>>  		if (pe->type & EEH_PE_VF)
>>  			eeh_add_virt_device(edev, NULL);
>>  		else
>> -			pcibios_add_pci_devices(frozen_bus);
>> +			pci_hp_add_devices(frozen_bus);
>>  	}
>>  	eeh_pe_state_clear(pe, EEH_PE_KEEP);
>>  
>> @@ -896,7 +896,7 @@ perm_error:
>>  			eeh_pe_dev_mode_mark(pe, EEH_DEV_REMOVED);
>>  
>>  			pci_lock_rescan_remove();
>> -			pcibios_remove_pci_devices(frozen_bus);
>> +			pci_hp_remove_devices(frozen_bus);
>>  			pci_unlock_rescan_remove();
>>  		}
>>  	}
>> @@ -981,7 +981,7 @@ static void eeh_handle_special_event(void)
>>  				bus = eeh_pe_bus_get(phb_pe);
>>  				eeh_pe_dev_traverse(pe,
>>  					eeh_report_failure, NULL);
>> -				pcibios_remove_pci_devices(bus);
>> +				pci_hp_remove_devices(bus);
>>  			}
>>  			pci_unlock_rescan_remove();
>>  		}
>> diff --git a/arch/powerpc/kernel/pci-hotplug.c b/arch/powerpc/kernel/pci-
>hotplug.c
>> index 59c4361..2d108e5 100644
>> --- a/arch/powerpc/kernel/pci-hotplug.c
>> +++ b/arch/powerpc/kernel/pci-hotplug.c
>> @@ -38,20 +38,20 @@ void pcibios_release_device(struct pci_dev *dev)
>>  }
>>  
>>  /**
>> - * pcibios_remove_pci_devices - remove all devices under this bus
>> + * pci_hp_remove_devices - remove all devices under this bus
>>   * @bus: the indicated PCI bus
>>   *
>>   * Remove all of the PCI devices under this bus both from the
>>   * linux pci device tree, and from the powerpc EEH address cache.
>>   */
>> -void pcibios_remove_pci_devices(struct pci_bus *bus)
>> +void pci_hp_remove_devices(struct pci_bus *bus)
>>  {
>>  	struct pci_dev *dev, *tmp;
>>  	struct pci_bus *child_bus;
>>  
>>  	/* First go down child busses */
>>  	list_for_each_entry(child_bus, &bus->children, node)
>> -		pcibios_remove_pci_devices(child_bus);
>> +		pci_hp_remove_devices(child_bus);
>>  
>>  	pr_debug("PCI: Removing devices on bus %04x:%02x\n",
>>  		 pci_domain_nr(bus),  bus->number);
>> @@ -60,11 +60,10 @@ void pcibios_remove_pci_devices(struct pci_bus *bus)
>>  		pci_stop_and_remove_bus_device(dev);
>>  	}
>>  }
>> -
>> -EXPORT_SYMBOL_GPL(pcibios_remove_pci_devices);
>> +EXPORT_SYMBOL_GPL(pci_hp_remove_devices);
>>  
>>  /**
>> - * pcibios_add_pci_devices - adds new pci devices to bus
>> + * pci_hp_add_devices - adds new pci devices to bus
>>   * @bus: the indicated PCI bus
>>   *
>>   * This routine will find and fixup new pci devices under
>> @@ -74,7 +73,7 @@ EXPORT_SYMBOL_GPL(pcibios_remove_pci_devices);
>>   * is how this routine differs from other, similar pcibios
>>   * routines.)
>>   */
>> -void pcibios_add_pci_devices(struct pci_bus * bus)
>> +void pci_hp_add_devices(struct pci_bus *bus)
>>  {
>>  	int slotno, mode, pass, max;
>>  	struct pci_dev *dev;
>> @@ -114,4 +113,4 @@ void pcibios_add_pci_devices(struct pci_bus * bus)
>>  	}
>>  	pcibios_finish_adding_to_bus(bus);
>>  }
>> -EXPORT_SYMBOL_GPL(pcibios_add_pci_devices);
>> +EXPORT_SYMBOL_GPL(pci_hp_add_devices);
>> diff --git a/drivers/pci/hotplug/rpadlpar_core.c 
>b/drivers/pci/hotplug/rpadlpar_core.c
>> index b46b57d..b770d6d 100644
>> --- a/drivers/pci/hotplug/rpadlpar_core.c
>> +++ b/drivers/pci/hotplug/rpadlpar_core.c
>> @@ -380,7 +380,7 @@ int dlpar_remove_pci_slot(char *drc_name, struct 
>device_node *dn)
>>  	}
>>  
>>  	/* Remove all devices below slot */
>> -	pcibios_remove_pci_devices(bus);
>> +	pci_hp_remove_devices(bus);
>>  
>>  	/* Unmap PCI IO space */
>>  	if (pcibios_unmap_io_space(bus)) {
>> diff --git a/drivers/pci/hotplug/rpaphp_core.c 
>b/drivers/pci/hotplug/rpaphp_core.c
>> index 611f605..8d13202 100644
>> --- a/drivers/pci/hotplug/rpaphp_core.c
>> +++ b/drivers/pci/hotplug/rpaphp_core.c
>> @@ -404,7 +404,7 @@ static int enable_slot(struct hotplug_slot 
>*hotplug_slot)
>>  
>>  	if (state == PRESENT) {
>>  		pci_lock_rescan_remove();
>> -		pcibios_add_pci_devices(slot->bus);
>> +		pci_hp_add_devices(slot->bus);
>>  		pci_unlock_rescan_remove();
>>  		slot->state = CONFIGURED;
>>  	} else if (state == EMPTY) {
>> @@ -426,7 +426,7 @@ static int disable_slot(struct hotplug_slot 
>*hotplug_slot)
>>  		return -EINVAL;
>>  
>>  	pci_lock_rescan_remove();
>> -	pcibios_remove_pci_devices(slot->bus);
>> +	pci_hp_remove_devices(slot->bus);
>>  	pci_unlock_rescan_remove();
>>  	vm_unmap_aliases();
>>  
>> diff --git a/drivers/pci/hotplug/rpaphp_pci.c 
>b/drivers/pci/hotplug/rpaphp_pci.c
>> index 7836d69..137a892 100644
>> --- a/drivers/pci/hotplug/rpaphp_pci.c
>> +++ b/drivers/pci/hotplug/rpaphp_pci.c
>> @@ -116,7 +116,7 @@ int rpaphp_enable_slot(struct slot *slot)
>>  		}
>>  
>>  		if (list_empty(&bus->devices))
>> -			pcibios_add_pci_devices(bus);
>> +			pci_hp_add_devices(bus);
>>  
>>  		if (!list_empty(&bus->devices)) {
>>  			info->adapter_status = CONFIGURED;
>> 
>

Re: [PATCH v9 12/26] powerpc/powernv/ioda1: M64 support on P7IOC

[Alistair Popple]

On Tue, 3 May 2016 15:41:31 Gavin Shan wrote:
> This enables M64 window on P7IOC, which has been enabled on PHB3.

Have we tested that this works with an adaptor? This looks to be enabling 
support for something that didn't previously work (64-bit BARs on P7IOC)?

Regards,

Alistair

> Different from PHB3 where 16 M64 BARs are supported and each of
> them can be owned by one particular PE# exclusively or divided
> evenly to 256 segments, every P7IOC PHB has 16 M64 BARs and each
> of them are divided to 8 segments. So every P7IOC PHB supports
> 128 M64 segments in total. P7IOC has M64DT, which helps mapping
> one particular M64 segment# to arbitrary PE#. PHB3 doesn't have
> M64DT, indicating that one M64 segment can only be pinned to the
> fixed PE#.
> 
> In order to unified M64 support M64 on P7IOC and PHB3, we just
> provide 128 M64 segments on every P7IOC PHB and each of them is
> pinned to the fixed PE# by bypassing the function of M64DT. In
> turn, we just need different phb->init_m64() for P7IOC and PHB3
> and maps M64 segment in pnv_ioda_reserve_m64_pe() for P7IOC, most
> of the code are shared by them.
> 
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> ---
>  arch/powerpc/platforms/powernv/pci-ioda.c | 89 
+++++++++++++++++++++++++++++--
>  1 file changed, 86 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c 
b/arch/powerpc/platforms/powernv/pci-ioda.c
> index 37f22b0..a1b74ec 100644
> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
> @@ -48,6 +48,9 @@
>  #include "powernv.h"
>  #include "pci.h"
>  
> +#define PNV_IODA1_M64_NUM	16	/* Number of M64 BARs	*/
> +#define PNV_IODA1_M64_SEGS	8	/* Segments per M64 BAR	*/
> +
>  /* 256M DMA window, 4K TCE pages, 8 bytes TCE */
>  #define TCE32_TABLE_SIZE	((0x10000000 / 0x1000) * 8)
>  
> @@ -246,6 +249,64 @@ static void pnv_ioda_reserve_dev_m64_pe(struct pci_dev 
*pdev,
>  	}
>  }
>  
> +static int pnv_ioda1_init_m64(struct pnv_phb *phb)
> +{
> +	struct resource *r;
> +	int index;
> +
> +	/*
> +	 * There are 16 M64 BARs, each of which has 8 segments. So
> +	 * there are as many M64 segments as the maximum number of
> +	 * PEs, which is 128.
> +	 */
> +	for (index = 0; index < PNV_IODA1_M64_NUM; index++) {
> +		unsigned long base, segsz = phb->ioda.m64_segsize;
> +		int64_t rc;
> +
> +		base = phb->ioda.m64_base +
> +		       index * PNV_IODA1_M64_SEGS * segsz;
> +		rc = opal_pci_set_phb_mem_window(phb->opal_id,
> +				OPAL_M64_WINDOW_TYPE, index, base, 0,
> +				PNV_IODA1_M64_SEGS * segsz);
> +		if (rc != OPAL_SUCCESS) {
> +			pr_warn("  Error %lld setting M64 PHB#%d-BAR#%d\n",
> +				rc, phb->hose->global_number, index);
> +			goto fail;
> +		}
> +
> +		rc = opal_pci_phb_mmio_enable(phb->opal_id,
> +				OPAL_M64_WINDOW_TYPE, index,
> +				OPAL_ENABLE_M64_SPLIT);
> +		if (rc != OPAL_SUCCESS) {
> +			pr_warn("  Error %lld enabling M64 PHB#%d-BAR#%d\n",
> +				rc, phb->hose->global_number, index);
> +			goto fail;
> +		}
> +	}
> +
> +	/*
> +	 * Exclude the segment used by the reserved PE, which
> +	 * is expected to be 0 or last supported PE#.
> +	 */
> +	r = &phb->hose->mem_resources[1];
> +	if (phb->ioda.reserved_pe_idx == 0)
> +		r->start += phb->ioda.m64_segsize;
> +	else if (phb->ioda.reserved_pe_idx == (phb->ioda.total_pe_num - 1))
> +		r->end -= phb->ioda.m64_segsize;
> +	else
> +		pr_warn("  Cannot cut M64 segment for reserved PE#%d\n",
> +			phb->ioda.reserved_pe_idx);
> +
> +	return 0;
> +
> +fail:
> +	for ( ; index >= 0; index--)
> +		opal_pci_phb_mmio_enable(phb->opal_id,
> +			OPAL_M64_WINDOW_TYPE, index, OPAL_DISABLE_M64);
> +
> +	return -EIO;
> +}
> +
>  static void pnv_ioda_reserve_m64_pe(struct pci_bus *bus,
>  				    unsigned long *pe_bitmap,
>  				    bool all)
> @@ -315,6 +376,26 @@ static unsigned int pnv_ioda_pick_m64_pe(struct pci_bus 
*bus, bool all)
>  			pe->master = master_pe;
>  			list_add_tail(&pe->list, &master_pe->slaves);
>  		}
> +
> +		/*
> +		 * P7IOC supports M64DT, which helps mapping M64 segment
> +		 * to one particular PE#. However, PHB3 has fixed mapping
> +		 * between M64 segment and PE#. In order to have same logic
> +		 * for P7IOC and PHB3, we enforce fixed mapping between M64
> +		 * segment and PE# on P7IOC.
> +		 */
> +		if (phb->type == PNV_PHB_IODA1) {
> +			int64_t rc;
> +
> +			rc = opal_pci_map_pe_mmio_window(phb->opal_id,
> +					pe->pe_number, OPAL_M64_WINDOW_TYPE,
> +					pe->pe_number / PNV_IODA1_M64_SEGS,
> +					pe->pe_number % PNV_IODA1_M64_SEGS);
> +			if (rc != OPAL_SUCCESS)
> +				pr_warn("%s: Error %lld mapping M64 for 
PHB#%d-PE#%d\n",
> +					__func__, rc, phb->hose-
>global_number,
> +					pe->pe_number);
> +		}
>  	}
>  
>  	kfree(pe_alloc);
> @@ -329,8 +410,7 @@ static void __init pnv_ioda_parse_m64_window(struct 
pnv_phb *phb)
>  	const u32 *r;
>  	u64 pci_addr;
>  
> -	/* FIXME: Support M64 for P7IOC */
> -	if (phb->type != PNV_PHB_IODA2) {
> +	if (phb->type != PNV_PHB_IODA1 && phb->type != PNV_PHB_IODA2) {
>  		pr_info("  Not support M64 window\n");
>  		return;
>  	}
> @@ -364,7 +444,10 @@ static void __init pnv_ioda_parse_m64_window(struct 
pnv_phb *phb)
>  
>  	/* Use last M64 BAR to cover M64 window */
>  	phb->ioda.m64_bar_idx = 15;
> -	phb->init_m64 = pnv_ioda2_init_m64;
> +	if (phb->type == PNV_PHB_IODA1)
> +		phb->init_m64 = pnv_ioda1_init_m64;
> +	else
> +		phb->init_m64 = pnv_ioda2_init_m64;
>  	phb->reserve_m64_pe = pnv_ioda_reserve_m64_pe;
>  	phb->pick_m64_pe = pnv_ioda_pick_m64_pe;
>  }
> 

Re: [PATCH v9 12/26] powerpc/powernv/ioda1: M64 support on P7IOC

[Gavin Shan]

On Wed, May 04, 2016 at 03:17:51PM +1000, Alistair Popple wrote:
>On Tue, 3 May 2016 15:41:31 Gavin Shan wrote:
>> This enables M64 window on P7IOC, which has been enabled on PHB3.
>
>Have we tested that this works with an adaptor? This looks to be enabling 
>support for something that didn't previously work (64-bit BARs on P7IOC)?
>

The M64 isn't supported on P7IOC before, meaning the PCI device's M64
BAR is covered by PHB's M32 window. With the patch applied, the PCI
device's M64 BAR is covered by PHB's M64 window as below log I got
from vpl4. The kernel including the series of patches boots successfully
on vpl4 and all looks normal.

[    0.246110] pci 0000:60:00.0: BAR 2: assigned [mem 0x3da810000000-0x3da810ffffff 64bit pref]
[    0.246218] pci 0000:60:00.0: BAR 0: assigned [mem 0x3da081000000-0x3da08103ffff 64bit]
[    0.246306] pci 0000:60:00.0: BAR 6: assigned [mem 0x3da081040000-0x3da08105ffff pref]
[    0.246392] pci 0000:60     : [PE# 001] Secondary bus 96 associated with PE#1
[    0.246484] pci 0000:60     : [PE# 001] DMA weight 15 (64), assigned (0) 1 DMA32 segments
[    0.246552] pci 0000:60     : [PE# 001]  Setting up 32-bit TCE table at 00000000..0fffffff

Thanks,
Gavin

>Regards,
>
>Alistair
>
>> Different from PHB3 where 16 M64 BARs are supported and each of
>> them can be owned by one particular PE# exclusively or divided
>> evenly to 256 segments, every P7IOC PHB has 16 M64 BARs and each
>> of them are divided to 8 segments. So every P7IOC PHB supports
>> 128 M64 segments in total. P7IOC has M64DT, which helps mapping
>> one particular M64 segment# to arbitrary PE#. PHB3 doesn't have
>> M64DT, indicating that one M64 segment can only be pinned to the
>> fixed PE#.
>> 
>> In order to unified M64 support M64 on P7IOC and PHB3, we just
>> provide 128 M64 segments on every P7IOC PHB and each of them is
>> pinned to the fixed PE# by bypassing the function of M64DT. In
>> turn, we just need different phb->init_m64() for P7IOC and PHB3
>> and maps M64 segment in pnv_ioda_reserve_m64_pe() for P7IOC, most
>> of the code are shared by them.
>> 
>> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>> ---
>>  arch/powerpc/platforms/powernv/pci-ioda.c | 89 
>+++++++++++++++++++++++++++++--
>>  1 file changed, 86 insertions(+), 3 deletions(-)
>> 
>> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c 
>b/arch/powerpc/platforms/powernv/pci-ioda.c
>> index 37f22b0..a1b74ec 100644
>> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
>> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
>> @@ -48,6 +48,9 @@
>>  #include "powernv.h"
>>  #include "pci.h"
>>  
>> +#define PNV_IODA1_M64_NUM	16	/* Number of M64 BARs	*/
>> +#define PNV_IODA1_M64_SEGS	8	/* Segments per M64 BAR	*/
>> +
>>  /* 256M DMA window, 4K TCE pages, 8 bytes TCE */
>>  #define TCE32_TABLE_SIZE	((0x10000000 / 0x1000) * 8)
>>  
>> @@ -246,6 +249,64 @@ static void pnv_ioda_reserve_dev_m64_pe(struct pci_dev 
>*pdev,
>>  	}
>>  }
>>  
>> +static int pnv_ioda1_init_m64(struct pnv_phb *phb)
>> +{
>> +	struct resource *r;
>> +	int index;
>> +
>> +	/*
>> +	 * There are 16 M64 BARs, each of which has 8 segments. So
>> +	 * there are as many M64 segments as the maximum number of
>> +	 * PEs, which is 128.
>> +	 */
>> +	for (index = 0; index < PNV_IODA1_M64_NUM; index++) {
>> +		unsigned long base, segsz = phb->ioda.m64_segsize;
>> +		int64_t rc;
>> +
>> +		base = phb->ioda.m64_base +
>> +		       index * PNV_IODA1_M64_SEGS * segsz;
>> +		rc = opal_pci_set_phb_mem_window(phb->opal_id,
>> +				OPAL_M64_WINDOW_TYPE, index, base, 0,
>> +				PNV_IODA1_M64_SEGS * segsz);
>> +		if (rc != OPAL_SUCCESS) {
>> +			pr_warn("  Error %lld setting M64 PHB#%d-BAR#%d\n",
>> +				rc, phb->hose->global_number, index);
>> +			goto fail;
>> +		}
>> +
>> +		rc = opal_pci_phb_mmio_enable(phb->opal_id,
>> +				OPAL_M64_WINDOW_TYPE, index,
>> +				OPAL_ENABLE_M64_SPLIT);
>> +		if (rc != OPAL_SUCCESS) {
>> +			pr_warn("  Error %lld enabling M64 PHB#%d-BAR#%d\n",
>> +				rc, phb->hose->global_number, index);
>> +			goto fail;
>> +		}
>> +	}
>> +
>> +	/*
>> +	 * Exclude the segment used by the reserved PE, which
>> +	 * is expected to be 0 or last supported PE#.
>> +	 */
>> +	r = &phb->hose->mem_resources[1];
>> +	if (phb->ioda.reserved_pe_idx == 0)
>> +		r->start += phb->ioda.m64_segsize;
>> +	else if (phb->ioda.reserved_pe_idx == (phb->ioda.total_pe_num - 1))
>> +		r->end -= phb->ioda.m64_segsize;
>> +	else
>> +		pr_warn("  Cannot cut M64 segment for reserved PE#%d\n",
>> +			phb->ioda.reserved_pe_idx);
>> +
>> +	return 0;
>> +
>> +fail:
>> +	for ( ; index >= 0; index--)
>> +		opal_pci_phb_mmio_enable(phb->opal_id,
>> +			OPAL_M64_WINDOW_TYPE, index, OPAL_DISABLE_M64);
>> +
>> +	return -EIO;
>> +}
>> +
>>  static void pnv_ioda_reserve_m64_pe(struct pci_bus *bus,
>>  				    unsigned long *pe_bitmap,
>>  				    bool all)
>> @@ -315,6 +376,26 @@ static unsigned int pnv_ioda_pick_m64_pe(struct pci_bus 
>*bus, bool all)
>>  			pe->master = master_pe;
>>  			list_add_tail(&pe->list, &master_pe->slaves);
>>  		}
>> +
>> +		/*
>> +		 * P7IOC supports M64DT, which helps mapping M64 segment
>> +		 * to one particular PE#. However, PHB3 has fixed mapping
>> +		 * between M64 segment and PE#. In order to have same logic
>> +		 * for P7IOC and PHB3, we enforce fixed mapping between M64
>> +		 * segment and PE# on P7IOC.
>> +		 */
>> +		if (phb->type == PNV_PHB_IODA1) {
>> +			int64_t rc;
>> +
>> +			rc = opal_pci_map_pe_mmio_window(phb->opal_id,
>> +					pe->pe_number, OPAL_M64_WINDOW_TYPE,
>> +					pe->pe_number / PNV_IODA1_M64_SEGS,
>> +					pe->pe_number % PNV_IODA1_M64_SEGS);
>> +			if (rc != OPAL_SUCCESS)
>> +				pr_warn("%s: Error %lld mapping M64 for 
>PHB#%d-PE#%d\n",
>> +					__func__, rc, phb->hose-
>>global_number,
>> +					pe->pe_number);
>> +		}
>>  	}
>>  
>>  	kfree(pe_alloc);
>> @@ -329,8 +410,7 @@ static void __init pnv_ioda_parse_m64_window(struct 
>pnv_phb *phb)
>>  	const u32 *r;
>>  	u64 pci_addr;
>>  
>> -	/* FIXME: Support M64 for P7IOC */
>> -	if (phb->type != PNV_PHB_IODA2) {
>> +	if (phb->type != PNV_PHB_IODA1 && phb->type != PNV_PHB_IODA2) {
>>  		pr_info("  Not support M64 window\n");
>>  		return;
>>  	}
>> @@ -364,7 +444,10 @@ static void __init pnv_ioda_parse_m64_window(struct 
>pnv_phb *phb)
>>  
>>  	/* Use last M64 BAR to cover M64 window */
>>  	phb->ioda.m64_bar_idx = 15;
>> -	phb->init_m64 = pnv_ioda2_init_m64;
>> +	if (phb->type == PNV_PHB_IODA1)
>> +		phb->init_m64 = pnv_ioda1_init_m64;
>> +	else
>> +		phb->init_m64 = pnv_ioda2_init_m64;
>>  	phb->reserve_m64_pe = pnv_ioda_reserve_m64_pe;
>>  	phb->pick_m64_pe = pnv_ioda_pick_m64_pe;
>>  }
>> 
>

Re: [PATCH v9 06/26] powerpc/powernv: Data type unsigned int for PE number

[Alexey Kardashevskiy]

On 05/03/2016 03:41 PM, Gavin Shan wrote:
> This changes the data type of PE number from "int" to "unsigned int"
> in order to match the fact PE number is never negative:
>
>    * The number of PE to which the specified PCI device is attached.
>    * The PE number map for SRIOV VFs.
>    * The returned PE number from pnv_ioda_alloc_pe().
>    * The returned PE number from pnv_ioda2_pick_m64_pe().
>
> Suggested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>

Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>

> ---
>  arch/powerpc/include/asm/pci-bridge.h     | 6 +++---
>  arch/powerpc/platforms/powernv/pci-ioda.c | 8 ++++----
>  arch/powerpc/platforms/powernv/pci.c      | 2 +-
>  arch/powerpc/platforms/powernv/pci.h      | 2 +-
>  4 files changed, 9 insertions(+), 9 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/pci-bridge.h b/arch/powerpc/include/asm/pci-bridge.h
> index 023c8c8..220129f 100644
> --- a/arch/powerpc/include/asm/pci-bridge.h
> +++ b/arch/powerpc/include/asm/pci-bridge.h
> @@ -209,14 +209,14 @@ struct pci_dn {
>  #ifdef CONFIG_EEH
>  	struct eeh_dev *edev;		/* eeh device */
>  #endif
> -#define IODA_INVALID_PE		(-1)
> +#define IODA_INVALID_PE		0xFFFFFFFF
>  #ifdef CONFIG_PPC_POWERNV
> -	int	pe_number;
> +	unsigned int pe_number;
>  	int     vf_index;		/* VF index in the PF */
>  #ifdef CONFIG_PCI_IOV
>  	u16     vfs_expanded;		/* number of VFs IOV BAR expanded */
>  	u16     num_vfs;		/* number of VFs enabled*/
> -	int     *pe_num_map;		/* PE# for the first VF PE or array */
> +	unsigned int *pe_num_map;	/* PE# for the first VF PE or array */
>  	bool    m64_single_mode;	/* Use M64 BAR in Single Mode */
>  #define IODA_INVALID_M64        (-1)
>  	int     (*m64_map)[PCI_SRIOV_NUM_BARS];
> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
> index 1d2514f..4aa6cdf 100644
> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
> @@ -138,7 +138,7 @@ static void pnv_ioda_reserve_pe(struct pnv_phb *phb, int pe_no)
>  	phb->ioda.pe_array[pe_no].pe_number = pe_no;
>  }
>
> -static int pnv_ioda_alloc_pe(struct pnv_phb *phb)
> +static unsigned int pnv_ioda_alloc_pe(struct pnv_phb *phb)
>  {
>  	unsigned long pe;
>
> @@ -261,7 +261,7 @@ static void pnv_ioda2_reserve_m64_pe(struct pci_bus *bus,
>  	}
>  }
>
> -static int pnv_ioda2_pick_m64_pe(struct pci_bus *bus, bool all)
> +static unsigned int pnv_ioda2_pick_m64_pe(struct pci_bus *bus, bool all)
>  {
>  	struct pci_controller *hose = pci_bus_to_host(bus);
>  	struct pnv_phb *phb = hose->private_data;
> @@ -919,7 +919,7 @@ static struct pnv_ioda_pe *pnv_ioda_setup_dev_PE(struct pci_dev *dev)
>  	struct pnv_phb *phb = hose->private_data;
>  	struct pci_dn *pdn = pci_get_pdn(dev);
>  	struct pnv_ioda_pe *pe;
> -	int pe_num;
> +	unsigned int pe_num;
>
>  	if (!pdn) {
>  		pr_err("%s: Device tree node not associated properly\n",
> @@ -1010,7 +1010,7 @@ static void pnv_ioda_setup_bus_PE(struct pci_bus *bus, bool all)
>  	struct pci_controller *hose = pci_bus_to_host(bus);
>  	struct pnv_phb *phb = hose->private_data;
>  	struct pnv_ioda_pe *pe;
> -	int pe_num = IODA_INVALID_PE;
> +	unsigned int pe_num = IODA_INVALID_PE;
>
>  	/* Check if PE is determined by M64 */
>  	if (phb->pick_m64_pe)
> diff --git a/arch/powerpc/platforms/powernv/pci.c b/arch/powerpc/platforms/powernv/pci.c
> index afbaa1c..8827461 100644
> --- a/arch/powerpc/platforms/powernv/pci.c
> +++ b/arch/powerpc/platforms/powernv/pci.c
> @@ -370,7 +370,7 @@ static void pnv_pci_config_check_eeh(struct pci_dn *pdn)
>  	struct pnv_phb *phb = pdn->phb->private_data;
>  	u8	fstate;
>  	__be16	pcierr;
> -	int	pe_no;
> +	unsigned int pe_no;
>  	s64	rc;
>
>  	/*
> diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
> index 784882a..66f2569 100644
> --- a/arch/powerpc/platforms/powernv/pci.h
> +++ b/arch/powerpc/platforms/powernv/pci.h
> @@ -113,7 +113,7 @@ struct pnv_phb {
>  	int (*init_m64)(struct pnv_phb *phb);
>  	void (*reserve_m64_pe)(struct pci_bus *bus,
>  			       unsigned long *pe_bitmap, bool all);
> -	int (*pick_m64_pe)(struct pci_bus *bus, bool all);
> +	unsigned int (*pick_m64_pe)(struct pci_bus *bus, bool all);
>  	int (*get_pe_state)(struct pnv_phb *phb, int pe_no);
>  	void (*freeze_pe)(struct pnv_phb *phb, int pe_no);
>  	int (*unfreeze_pe)(struct pnv_phb *phb, int pe_no, int opt);
>


-- 
Alexey

Re: [PATCH v9 16/26] powerpc/powernv/ioda1: Improve DMA32 segment track

[Gavin Shan]

On Tue, May 03, 2016 at 03:41:35PM +1000, Gavin Shan wrote:
>In current implementation, the DMA32 segments required by one specific
>PE isn't calculated with the information hold in the PE independently.
>It conflicts with the PCI hotplug design: PE centralized, meaning the
>PE's DMA32 segments should be calculated from the information hold in
>the PE independently.
>
>This introduces an array (@dma32_segmap) for every PHB to track the
>DMA32 segmeng usage. Besides, this moves the logic calculating PE's
>consumed DMA32 segments to pnv_pci_ioda1_setup_dma_pe() so that PE's
>DMA32 segments are calculated/allocated from the information hold in
>the PE (DMA32 weight). Also the logic is improved: we try to allocate
>as much DMA32 segments as we can. It's acceptable that number of DMA32
>segments less than the expected number are allocated.
>
>Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>

This can cause overlapped DMA32 segments to different PEs in some cases.
I already had the fix and hold for posting after discussion with Michael
on how to handle the series tomorrow.

Thanks,
Gavin

>---
> arch/powerpc/platforms/powernv/pci-ioda.c | 110 ++++++++++++++++--------------
> arch/powerpc/platforms/powernv/pci.h      |   7 +-
> 2 files changed, 61 insertions(+), 56 deletions(-)
>
>diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
>index f70a4e0..cfd2906 100644
>--- a/arch/powerpc/platforms/powernv/pci-ioda.c
>+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
>@@ -2011,27 +2011,57 @@ static unsigned int pnv_pci_ioda_pe_dma_weight(struct pnv_ioda_pe *pe)
> }
>
> static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
>-				       struct pnv_ioda_pe *pe,
>-				       unsigned int base,
>-				       unsigned int segs)
>+				       struct pnv_ioda_pe *pe)
> {
>
> 	struct page *tce_mem = NULL;
> 	struct iommu_table *tbl;
>-	unsigned int tce32_segsz, i;
>+	unsigned int weight, total_weight = 0;
>+	unsigned int tce32_segsz, base, segs, i;
> 	int64_t rc;
> 	void *addr;
>
> 	/* XXX FIXME: Handle 64-bit only DMA devices */
> 	/* XXX FIXME: Provide 64-bit DMA facilities & non-4K TCE tables etc.. */
> 	/* XXX FIXME: Allocate multi-level tables on PHB3 */
>+	pci_walk_bus(phb->hose->bus, pnv_pci_ioda_dev_dma_weight,
>+		     &total_weight);
>+	weight = pnv_pci_ioda_pe_dma_weight(pe);
>+
>+	segs = (weight * phb->ioda.dma32_count) / total_weight;
>+	if (!segs)
>+		segs = 1;
>+
>+	/*
>+	 * Allocate contiguous DMA32 segments. We begin with the expected
>+	 * number of segments. With one more attempt, the number of DMA32
>+	 * segments to be allocated is decreased by one until one segment
>+	 * is allocated successfully.
>+	 */
>+	do {
>+		for (base = 0; base <= phb->ioda.dma32_count - segs; base++) {
>+			for (i = base; i < base + segs; i++) {
>+				if (phb->ioda.dma32_segmap[i] !=
>+				    IODA_INVALID_PE)
>+					goto found;
>+			}
>+		}
>+	} while (segs--);
>+
>+	if (!segs) {
>+		pe_warn(pe, "No available DMA32 segments\n");
>+		return;
>+	}
>
>+found:
> 	tbl = pnv_pci_table_alloc(phb->hose->node);
> 	iommu_register_group(&pe->table_group, phb->hose->global_number,
> 			pe->pe_number);
> 	pnv_pci_link_table_and_group(phb->hose->node, 0, tbl, &pe->table_group);
>
> 	/* Grab a 32-bit TCE table */
>+	pe_info(pe, "DMA weight %d (%d), assigned (%d) %d DMA32 segments\n",
>+		weight, total_weight, base, segs);
> 	pe_info(pe, " Setting up 32-bit TCE table at %08x..%08x\n",
> 		base * PNV_IODA1_DMA32_SEGSIZE,
> 		(base + segs) * PNV_IODA1_DMA32_SEGSIZE - 1);
>@@ -2068,6 +2098,10 @@ static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
> 		}
> 	}
>
>+	/* Setup DMA32 segment mapping */
>+	for (i = base; i < base + segs; i++)
>+		phb->ioda.dma32_segmap[i] = pe->pe_number;
>+
> 	/* Setup linux iommu table */
> 	pnv_pci_setup_iommu_table(tbl, addr, tce32_segsz * segs,
> 				  base * PNV_IODA1_DMA32_SEGSIZE,
>@@ -2542,73 +2576,34 @@ static void pnv_pci_ioda2_setup_dma_pe(struct pnv_phb *phb,
> static void pnv_ioda_setup_dma(struct pnv_phb *phb)
> {
> 	struct pci_controller *hose = phb->hose;
>-	unsigned int weight, total_weight, dma_pe_count;
>-	unsigned int residual, remaining, segs, base;
> 	struct pnv_ioda_pe *pe;
>-
>-	total_weight = 0;
>-	pci_walk_bus(phb->hose->bus, pnv_pci_ioda_dev_dma_weight,
>-		     &total_weight);
>-
>-	dma_pe_count = 0;
>-	list_for_each_entry(pe, &phb->ioda.pe_list, list) {
>-		weight = pnv_pci_ioda_pe_dma_weight(pe);
>-		if (weight > 0)
>-			dma_pe_count++;
>-	}
>+	unsigned int weight;
>
> 	/* If we have more PE# than segments available, hand out one
> 	 * per PE until we run out and let the rest fail. If not,
> 	 * then we assign at least one segment per PE, plus more based
> 	 * on the amount of devices under that PE
> 	 */
>-	if (dma_pe_count > phb->ioda.tce32_count)
>-		residual = 0;
>-	else
>-		residual = phb->ioda.tce32_count - dma_pe_count;
>-
>-	pr_info("PCI: Domain %04x has %ld available 32-bit DMA segments\n",
>-		hose->global_number, phb->ioda.tce32_count);
>-	pr_info("PCI: %d PE# for a total weight of %d\n",
>-		dma_pe_count, total_weight);
>+	pr_info("PCI: Domain %04x has %d available 32-bit DMA segments\n",
>+		hose->global_number, phb->ioda.dma32_count);
>
> 	pnv_pci_ioda_setup_opal_tce_kill(phb);
>
>-	/* Walk our PE list and configure their DMA segments, hand them
>-	 * out one base segment plus any residual segments based on
>-	 * weight
>-	 */
>-	remaining = phb->ioda.tce32_count;
>-	base = 0;
>+	/* Walk our PE list and configure their DMA segments */
> 	list_for_each_entry(pe, &phb->ioda.pe_list, list) {
> 		weight = pnv_pci_ioda_pe_dma_weight(pe);
> 		if (!weight)
> 			continue;
>
>-		if (!remaining) {
>-			pe_warn(pe, "No DMA32 resources available\n");
>-			continue;
>-		}
>-		segs = 1;
>-		if (residual) {
>-			segs += ((weight * residual) + (total_weight / 2)) /
>-				total_weight;
>-			if (segs > remaining)
>-				segs = remaining;
>-		}
>-
> 		/*
> 		 * For IODA2 compliant PHB3, we needn't care about the weight.
> 		 * The all available 32-bits DMA space will be assigned to
> 		 * the specific PE.
> 		 */
> 		if (phb->type == PNV_PHB_IODA1) {
>-			pe_info(pe, "DMA weight %d, assigned %d DMA32 segments\n",
>-				weight, segs);
>-			pnv_pci_ioda1_setup_dma_pe(phb, pe, base, segs);
>+			pnv_pci_ioda1_setup_dma_pe(phb, pe);
> 		} else if (phb->type == PNV_PHB_IODA2) {
> 			pe_info(pe, "Assign DMA32 space\n");
>-			segs = 0;
> 			pnv_pci_ioda2_setup_dma_pe(phb, pe);
> 		} else if (phb->type == PNV_PHB_NPU) {
> 			/*
>@@ -2618,9 +2613,6 @@ static void pnv_ioda_setup_dma(struct pnv_phb *phb)
> 			 * as the PHB3 TVT.
> 			 */
> 		}
>-
>-		remaining -= segs;
>-		base += segs;
> 	}
> }
>
>@@ -3327,7 +3319,8 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
> {
> 	struct pci_controller *hose;
> 	struct pnv_phb *phb;
>-	unsigned long size, m64map_off, m32map_off, pemap_off, iomap_off = 0;
>+	unsigned long size, m64map_off, m32map_off, pemap_off;
>+	unsigned long iomap_off = 0, dma32map_off = 0;
> 	const __be64 *prop64;
> 	const __be32 *prop32;
> 	int len;
>@@ -3413,6 +3406,10 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
> 	phb->ioda.io_segsize = phb->ioda.io_size / phb->ioda.total_pe_num;
> 	phb->ioda.io_pci_base = 0; /* XXX calculate this ? */
>
>+	/* Calculate how many 32-bit TCE segments we have */
>+	phb->ioda.dma32_count = phb->ioda.m32_pci_base /
>+				PNV_IODA1_DMA32_SEGSIZE;
>+
> 	/* Allocate aux data & arrays. We don't have IO ports on PHB3 */
> 	size = _ALIGN_UP(phb->ioda.total_pe_num / 8, sizeof(unsigned long));
> 	m64map_off = size;
>@@ -3422,6 +3419,9 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
> 	if (phb->type == PNV_PHB_IODA1) {
> 		iomap_off = size;
> 		size += phb->ioda.total_pe_num * sizeof(phb->ioda.io_segmap[0]);
>+		dma32map_off = size;
>+		size += phb->ioda.dma32_count *
>+			sizeof(phb->ioda.dma32_segmap[0]);
> 	}
> 	pemap_off = size;
> 	size += phb->ioda.total_pe_num * sizeof(struct pnv_ioda_pe);
>@@ -3437,6 +3437,10 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
> 		phb->ioda.io_segmap = aux + iomap_off;
> 		for (segno = 0; segno < phb->ioda.total_pe_num; segno++)
> 			phb->ioda.io_segmap[segno] = IODA_INVALID_PE;
>+
>+		phb->ioda.dma32_segmap = aux + dma32map_off;
>+		for (segno = 0; segno < phb->ioda.dma32_count; segno++)
>+			phb->ioda.dma32_segmap[segno] = IODA_INVALID_PE;
> 	}
> 	phb->ioda.pe_array = aux + pemap_off;
> 	set_bit(phb->ioda.reserved_pe_idx, phb->ioda.pe_alloc);
>@@ -3445,7 +3449,7 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
> 	mutex_init(&phb->ioda.pe_list_mutex);
>
> 	/* Calculate how many 32-bit TCE segments we have */
>-	phb->ioda.tce32_count = phb->ioda.m32_pci_base /
>+	phb->ioda.dma32_count = phb->ioda.m32_pci_base /
> 				PNV_IODA1_DMA32_SEGSIZE;
>
> #if 0 /* We should really do that ... */
>diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
>index 117cfcd..14d9391 100644
>--- a/arch/powerpc/platforms/powernv/pci.h
>+++ b/arch/powerpc/platforms/powernv/pci.h
>@@ -142,6 +142,10 @@ struct pnv_phb {
> 		unsigned int		*m32_segmap;
> 		unsigned int		*io_segmap;
>
>+		/* DMA32 segment maps - IODA1 only */
>+		unsigned int		dma32_count;
>+		unsigned int		*dma32_segmap;
>+
> 		/* IRQ chip */
> 		int			irq_chip_init;
> 		struct irq_chip		irq_chip;
>@@ -158,9 +162,6 @@ struct pnv_phb {
> 		 */
> 		unsigned char		pe_rmap[0x10000];
>
>-		/* 32-bit TCE tables allocation */
>-		unsigned long		tce32_count;
>-
> 		/* TCE cache invalidate registers (physical and
> 		 * remapped)
> 		 */
>-- 
>2.1.0
>

Re: [PATCH v9 12/26] powerpc/powernv/ioda1: M64 support on P7IOC

[Alistair Popple]

On Wed, 4 May 2016 16:48:53 Gavin Shan wrote:
> On Wed, May 04, 2016 at 03:17:51PM +1000, Alistair Popple wrote:
> >On Tue, 3 May 2016 15:41:31 Gavin Shan wrote:
> >> This enables M64 window on P7IOC, which has been enabled on PHB3.
> >
> >Have we tested that this works with an adaptor? This looks to be enabling 
> >support for something that didn't previously work (64-bit BARs on P7IOC)?
> >
> 
> The M64 isn't supported on P7IOC before, meaning the PCI device's M64
> BAR is covered by PHB's M32 window. With the patch applied, the PCI
> device's M64 BAR is covered by PHB's M64 window as below log I got
> from vpl4. The kernel including the series of patches boots successfully
> on vpl4 and all looks normal.

So you're changing the way 64-bit BARs work on P7IOC to use a different bit of
the hardware/firmware? This means it could break older systems if there is
an issue with the M64 support. Other than the comments below the code looks
reasonable. However I am assuming you've tested that cards with 64-bit BARs
still function on P7 after this change as I'm not familiar enough to comment on
the specific hardware details, although your comments make sense and match the
code.

> [    0.246110] pci 0000:60:00.0: BAR 2: assigned [mem 0x3da810000000-0x3da810ffffff 64bit pref]
> [    0.246218] pci 0000:60:00.0: BAR 0: assigned [mem 0x3da081000000-0x3da08103ffff 64bit]
> [    0.246306] pci 0000:60:00.0: BAR 6: assigned [mem 0x3da081040000-0x3da08105ffff pref]
> [    0.246392] pci 0000:60     : [PE# 001] Secondary bus 96 associated with PE#1
> [    0.246484] pci 0000:60     : [PE# 001] DMA weight 15 (64), assigned (0) 1 DMA32 segments
> [    0.246552] pci 0000:60     : [PE# 001]  Setting up 32-bit TCE table at 00000000..0fffffff
> 
> Thanks,
> Gavin
> 
> >Regards,
> >
> >Alistair
> >
> >> Different from PHB3 where 16 M64 BARs are supported and each of
> >> them can be owned by one particular PE# exclusively or divided
> >> evenly to 256 segments, every P7IOC PHB has 16 M64 BARs and each
> >> of them are divided to 8 segments. So every P7IOC PHB supports
> >> 128 M64 segments in total. P7IOC has M64DT, which helps mapping
> >> one particular M64 segment# to arbitrary PE#. PHB3 doesn't have
> >> M64DT, indicating that one M64 segment can only be pinned to the
> >> fixed PE#.
> >> 
> >> In order to unified M64 support M64 on P7IOC and PHB3, we just
> >> provide 128 M64 segments on every P7IOC PHB and each of them is
> >> pinned to the fixed PE# by bypassing the function of M64DT. In
> >> turn, we just need different phb->init_m64() for P7IOC and PHB3
> >> and maps M64 segment in pnv_ioda_reserve_m64_pe() for P7IOC, most
> >> of the code are shared by them.
> >> 
> >> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> >> ---
> >>  arch/powerpc/platforms/powernv/pci-ioda.c | 89 
> >+++++++++++++++++++++++++++++--
> >>  1 file changed, 86 insertions(+), 3 deletions(-)
> >> 
> >> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c 
> >b/arch/powerpc/platforms/powernv/pci-ioda.c
> >> index 37f22b0..a1b74ec 100644
> >> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
> >> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
> >> @@ -48,6 +48,9 @@
> >>  #include "powernv.h"
> >>  #include "pci.h"
> >>  
> >> +#define PNV_IODA1_M64_NUM	16	/* Number of M64 BARs	*/
> >> +#define PNV_IODA1_M64_SEGS	8	/* Segments per M64 BAR	*/
> >> +
> >>  /* 256M DMA window, 4K TCE pages, 8 bytes TCE */
> >>  #define TCE32_TABLE_SIZE	((0x10000000 / 0x1000) * 8)
> >>  
> >> @@ -246,6 +249,64 @@ static void pnv_ioda_reserve_dev_m64_pe(struct pci_dev 
> >*pdev,
> >>  	}
> >>  }
> >>  
> >> +static int pnv_ioda1_init_m64(struct pnv_phb *phb)
> >> +{
> >> +	struct resource *r;
> >> +	int index;
> >> +
> >> +	/*
> >> +	 * There are 16 M64 BARs, each of which has 8 segments. So
> >> +	 * there are as many M64 segments as the maximum number of
> >> +	 * PEs, which is 128.
> >> +	 */
> >> +	for (index = 0; index < PNV_IODA1_M64_NUM; index++) {
> >> +		unsigned long base, segsz = phb->ioda.m64_segsize;
> >> +		int64_t rc;
> >> +
> >> +		base = phb->ioda.m64_base +
> >> +		       index * PNV_IODA1_M64_SEGS * segsz;
> >> +		rc = opal_pci_set_phb_mem_window(phb->opal_id,
> >> +				OPAL_M64_WINDOW_TYPE, index, base, 0,
> >> +				PNV_IODA1_M64_SEGS * segsz);

Has firmware always supported OPAL_M64_WINDOW_TYPE for P7IOC? If older versions
don't support it what happens? Do we gracefully fall back to the old mode of
allocating all BARs from the M32 window or does it break?

> >> +		if (rc != OPAL_SUCCESS) {
> >> +			pr_warn("  Error %lld setting M64 PHB#%d-BAR#%d\n",
> >> +				rc, phb->hose->global_number, index);
> >> +			goto fail;
> >> +		}
> >> +
> >> +		rc = opal_pci_phb_mmio_enable(phb->opal_id,
> >> +				OPAL_M64_WINDOW_TYPE, index,
> >> +				OPAL_ENABLE_M64_SPLIT);
> >> +		if (rc != OPAL_SUCCESS) {
> >> +			pr_warn("  Error %lld enabling M64 PHB#%d-BAR#%d\n",
> >> +				rc, phb->hose->global_number, index);
> >> +			goto fail;
> >> +		}
> >> +	}
> >> +
> >> +	/*
> >> +	 * Exclude the segment used by the reserved PE, which
> >> +	 * is expected to be 0 or last supported PE#.
> >> +	 */
> >> +	r = &phb->hose->mem_resources[1];
> >> +	if (phb->ioda.reserved_pe_idx == 0)
> >> +		r->start += phb->ioda.m64_segsize;
> >> +	else if (phb->ioda.reserved_pe_idx == (phb->ioda.total_pe_num - 1))
> >> +		r->end -= phb->ioda.m64_segsize;
> >> +	else
> >> +		pr_warn("  Cannot cut M64 segment for reserved PE#%d\n",
> >> +			phb->ioda.reserved_pe_idx);

Should this be a WARN_ON()? If this condition can only exist because a future
programmer changes the reserved_pe_idx then I think it should be a WARN_ON()
as the above message would be too easy to miss.

> >> +
> >> +	return 0;
> >> +
> >> +fail:
> >> +	for ( ; index >= 0; index--)
> >> +		opal_pci_phb_mmio_enable(phb->opal_id,
> >> +			OPAL_M64_WINDOW_TYPE, index, OPAL_DISABLE_M64);
> >> +
> >> +	return -EIO;
> >> +}
> >> +
> >>  static void pnv_ioda_reserve_m64_pe(struct pci_bus *bus,
> >>  				    unsigned long *pe_bitmap,
> >>  				    bool all)
> >> @@ -315,6 +376,26 @@ static unsigned int pnv_ioda_pick_m64_pe(struct pci_bus 
> >*bus, bool all)
> >>  			pe->master = master_pe;
> >>  			list_add_tail(&pe->list, &master_pe->slaves);
> >>  		}
> >> +
> >> +		/*
> >> +		 * P7IOC supports M64DT, which helps mapping M64 segment
> >> +		 * to one particular PE#. However, PHB3 has fixed mapping
> >> +		 * between M64 segment and PE#. In order to have same logic
> >> +		 * for P7IOC and PHB3, we enforce fixed mapping between M64
> >> +		 * segment and PE# on P7IOC.
> >> +		 */
> >> +		if (phb->type == PNV_PHB_IODA1) {
> >> +			int64_t rc;
> >> +
> >> +			rc = opal_pci_map_pe_mmio_window(phb->opal_id,
> >> +					pe->pe_number, OPAL_M64_WINDOW_TYPE,
> >> +					pe->pe_number / PNV_IODA1_M64_SEGS,
> >> +					pe->pe_number % PNV_IODA1_M64_SEGS);
> >> +			if (rc != OPAL_SUCCESS)
> >> +				pr_warn("%s: Error %lld mapping M64 for 
> >PHB#%d-PE#%d\n",
> >> +					__func__, rc, phb->hose-
> >>global_number,
> >> +					pe->pe_number);
> >> +		}
> >>  	}
> >>  
> >>  	kfree(pe_alloc);
> >> @@ -329,8 +410,7 @@ static void __init pnv_ioda_parse_m64_window(struct 
> >pnv_phb *phb)
> >>  	const u32 *r;
> >>  	u64 pci_addr;
> >>  
> >> -	/* FIXME: Support M64 for P7IOC */
> >> -	if (phb->type != PNV_PHB_IODA2) {
> >> +	if (phb->type != PNV_PHB_IODA1 && phb->type != PNV_PHB_IODA2) {
> >>  		pr_info("  Not support M64 window\n");
> >>  		return;
> >>  	}
> >> @@ -364,7 +444,10 @@ static void __init pnv_ioda_parse_m64_window(struct 
> >pnv_phb *phb)
> >>  
> >>  	/* Use last M64 BAR to cover M64 window */
> >>  	phb->ioda.m64_bar_idx = 15;
> >> -	phb->init_m64 = pnv_ioda2_init_m64;
> >> +	if (phb->type == PNV_PHB_IODA1)
> >> +		phb->init_m64 = pnv_ioda1_init_m64;
> >> +	else
> >> +		phb->init_m64 = pnv_ioda2_init_m64;
> >>  	phb->reserve_m64_pe = pnv_ioda_reserve_m64_pe;
> >>  	phb->pick_m64_pe = pnv_ioda_pick_m64_pe;
> >>  }
> >> 
> >
> 

Re: [PATCH v9 12/26] powerpc/powernv/ioda1: M64 support on P7IOC

[Gavin Shan]

On Thu, May 05, 2016 at 09:53:51AM +1000, Alistair Popple wrote:
>On Wed, 4 May 2016 16:48:53 Gavin Shan wrote:
>> On Wed, May 04, 2016 at 03:17:51PM +1000, Alistair Popple wrote:
>> >On Tue, 3 May 2016 15:41:31 Gavin Shan wrote:
>> >> This enables M64 window on P7IOC, which has been enabled on PHB3.
>> >
>> >Have we tested that this works with an adaptor? This looks to be enabling 
>> >support for something that didn't previously work (64-bit BARs on P7IOC)?
>> >
>> 
>> The M64 isn't supported on P7IOC before, meaning the PCI device's M64
>> BAR is covered by PHB's M32 window. With the patch applied, the PCI
>> device's M64 BAR is covered by PHB's M64 window as below log I got
>> from vpl4. The kernel including the series of patches boots successfully
>> on vpl4 and all looks normal.
>
>So you're changing the way 64-bit BARs work on P7IOC to use a different bit of
>the hardware/firmware? This means it could break older systems if there is
>an issue with the M64 support. Other than the comments below the code looks
>reasonable. However I am assuming you've tested that cards with 64-bit BARs
>still function on P7 after this change as I'm not familiar enough to comment on
>the specific hardware details, although your comments make sense and match the
>code.
>

Nope, it doesn't rely on changes in firmware. The old firmware enables and
exposes PHB M64 window. The kernel doesn't use it on P7IOC until this patch.
On VPL4, there is a IPR adapter (0000:60:00.0) as below kernel log indicates.
Its BAR#2 is M64 (64-bits prefetchable) BAR. The BAR#2 is covered by PHB's M64
window with this patch. The PHB's M64 window is [0x00003da800000000..0x00003dafffffffff].
After it's applied, the IPR and root filesystem resident in the disk drive out
of it work fine.

>> [    0.246110] pci 0000:60:00.0: BAR 2: assigned [mem 0x3da810000000-0x3da810ffffff 64bit pref]
>> [    0.246218] pci 0000:60:00.0: BAR 0: assigned [mem 0x3da081000000-0x3da08103ffff 64bit]
>> [    0.246306] pci 0000:60:00.0: BAR 6: assigned [mem 0x3da081040000-0x3da08105ffff pref]
>> [    0.246392] pci 0000:60     : [PE# 001] Secondary bus 96 associated with PE#1
>> [    0.246484] pci 0000:60     : [PE# 001] DMA weight 15 (64), assigned (0) 1 DMA32 segments
>> [    0.246552] pci 0000:60     : [PE# 001]  Setting up 32-bit TCE table at 00000000..0fffffff
>> 
>> >> Different from PHB3 where 16 M64 BARs are supported and each of
>> >> them can be owned by one particular PE# exclusively or divided
>> >> evenly to 256 segments, every P7IOC PHB has 16 M64 BARs and each
>> >> of them are divided to 8 segments. So every P7IOC PHB supports
>> >> 128 M64 segments in total. P7IOC has M64DT, which helps mapping
>> >> one particular M64 segment# to arbitrary PE#. PHB3 doesn't have
>> >> M64DT, indicating that one M64 segment can only be pinned to the
>> >> fixed PE#.
>> >> 
>> >> In order to unified M64 support M64 on P7IOC and PHB3, we just
>> >> provide 128 M64 segments on every P7IOC PHB and each of them is
>> >> pinned to the fixed PE# by bypassing the function of M64DT. In
>> >> turn, we just need different phb->init_m64() for P7IOC and PHB3
>> >> and maps M64 segment in pnv_ioda_reserve_m64_pe() for P7IOC, most
>> >> of the code are shared by them.
>> >> 
>> >> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>> >> ---
>> >>  arch/powerpc/platforms/powernv/pci-ioda.c | 89 
>> >+++++++++++++++++++++++++++++--
>> >>  1 file changed, 86 insertions(+), 3 deletions(-)
>> >> 
>> >> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c 
>> >b/arch/powerpc/platforms/powernv/pci-ioda.c
>> >> index 37f22b0..a1b74ec 100644
>> >> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
>> >> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
>> >> @@ -48,6 +48,9 @@
>> >>  #include "powernv.h"
>> >>  #include "pci.h"
>> >>  
>> >> +#define PNV_IODA1_M64_NUM	16	/* Number of M64 BARs	*/
>> >> +#define PNV_IODA1_M64_SEGS	8	/* Segments per M64 BAR	*/
>> >> +
>> >>  /* 256M DMA window, 4K TCE pages, 8 bytes TCE */
>> >>  #define TCE32_TABLE_SIZE	((0x10000000 / 0x1000) * 8)
>> >>  
>> >> @@ -246,6 +249,64 @@ static void pnv_ioda_reserve_dev_m64_pe(struct pci_dev 
>> >*pdev,
>> >>  	}
>> >>  }
>> >>  
>> >> +static int pnv_ioda1_init_m64(struct pnv_phb *phb)
>> >> +{
>> >> +	struct resource *r;
>> >> +	int index;
>> >> +
>> >> +	/*
>> >> +	 * There are 16 M64 BARs, each of which has 8 segments. So
>> >> +	 * there are as many M64 segments as the maximum number of
>> >> +	 * PEs, which is 128.
>> >> +	 */
>> >> +	for (index = 0; index < PNV_IODA1_M64_NUM; index++) {
>> >> +		unsigned long base, segsz = phb->ioda.m64_segsize;
>> >> +		int64_t rc;
>> >> +
>> >> +		base = phb->ioda.m64_base +
>> >> +		       index * PNV_IODA1_M64_SEGS * segsz;
>> >> +		rc = opal_pci_set_phb_mem_window(phb->opal_id,
>> >> +				OPAL_M64_WINDOW_TYPE, index, base, 0,
>> >> +				PNV_IODA1_M64_SEGS * segsz);
>
>Has firmware always supported OPAL_M64_WINDOW_TYPE for P7IOC? If older versions
>don't support it what happens? Do we gracefully fall back to the old mode of
>allocating all BARs from the M32 window or does it break?
>

Old firmware enables and exposes M64 window (OPAL_M64_WINDOW_TYPE), but kernel
doesn't use it until this patch. If the OPAL call fails, errcode is returned.
The caller, pnv_pci_init_ioda_phb(), disables the window. It means the PCI
device M64 BARs will fail back to be covered by M32 window. The path is same
on PHB3 and P7IOC.

>> >> +		if (rc != OPAL_SUCCESS) {
>> >> +			pr_warn("  Error %lld setting M64 PHB#%d-BAR#%d\n",
>> >> +				rc, phb->hose->global_number, index);
>> >> +			goto fail;
>> >> +		}
>> >> +
>> >> +		rc = opal_pci_phb_mmio_enable(phb->opal_id,
>> >> +				OPAL_M64_WINDOW_TYPE, index,
>> >> +				OPAL_ENABLE_M64_SPLIT);
>> >> +		if (rc != OPAL_SUCCESS) {
>> >> +			pr_warn("  Error %lld enabling M64 PHB#%d-BAR#%d\n",
>> >> +				rc, phb->hose->global_number, index);
>> >> +			goto fail;
>> >> +		}
>> >> +	}
>> >> +
>> >> +	/*
>> >> +	 * Exclude the segment used by the reserved PE, which
>> >> +	 * is expected to be 0 or last supported PE#.
>> >> +	 */
>> >> +	r = &phb->hose->mem_resources[1];
>> >> +	if (phb->ioda.reserved_pe_idx == 0)
>> >> +		r->start += phb->ioda.m64_segsize;
>> >> +	else if (phb->ioda.reserved_pe_idx == (phb->ioda.total_pe_num - 1))
>> >> +		r->end -= phb->ioda.m64_segsize;
>> >> +	else
>> >> +		pr_warn("  Cannot cut M64 segment for reserved PE#%d\n",
>> >> +			phb->ioda.reserved_pe_idx);
>
>Should this be a WARN_ON()? If this condition can only exist because a future
>programmer changes the reserved_pe_idx then I think it should be a WARN_ON()
>as the above message would be too easy to miss.
>

Yeah, a WARN_ON() should be better.

Thanks,
Gavin

>> >> +
>> >> +	return 0;
>> >> +
>> >> +fail:
>> >> +	for ( ; index >= 0; index--)
>> >> +		opal_pci_phb_mmio_enable(phb->opal_id,
>> >> +			OPAL_M64_WINDOW_TYPE, index, OPAL_DISABLE_M64);
>> >> +
>> >> +	return -EIO;
>> >> +}
>> >> +
>> >>  static void pnv_ioda_reserve_m64_pe(struct pci_bus *bus,
>> >>  				    unsigned long *pe_bitmap,
>> >>  				    bool all)
>> >> @@ -315,6 +376,26 @@ static unsigned int pnv_ioda_pick_m64_pe(struct pci_bus 
>> >*bus, bool all)
>> >>  			pe->master = master_pe;
>> >>  			list_add_tail(&pe->list, &master_pe->slaves);
>> >>  		}
>> >> +
>> >> +		/*
>> >> +		 * P7IOC supports M64DT, which helps mapping M64 segment
>> >> +		 * to one particular PE#. However, PHB3 has fixed mapping
>> >> +		 * between M64 segment and PE#. In order to have same logic
>> >> +		 * for P7IOC and PHB3, we enforce fixed mapping between M64
>> >> +		 * segment and PE# on P7IOC.
>> >> +		 */
>> >> +		if (phb->type == PNV_PHB_IODA1) {
>> >> +			int64_t rc;
>> >> +
>> >> +			rc = opal_pci_map_pe_mmio_window(phb->opal_id,
>> >> +					pe->pe_number, OPAL_M64_WINDOW_TYPE,
>> >> +					pe->pe_number / PNV_IODA1_M64_SEGS,
>> >> +					pe->pe_number % PNV_IODA1_M64_SEGS);
>> >> +			if (rc != OPAL_SUCCESS)
>> >> +				pr_warn("%s: Error %lld mapping M64 for 
>> >PHB#%d-PE#%d\n",
>> >> +					__func__, rc, phb->hose-
>> >>global_number,
>> >> +					pe->pe_number);
>> >> +		}
>> >>  	}
>> >>  
>> >>  	kfree(pe_alloc);
>> >> @@ -329,8 +410,7 @@ static void __init pnv_ioda_parse_m64_window(struct 
>> >pnv_phb *phb)
>> >>  	const u32 *r;
>> >>  	u64 pci_addr;
>> >>  
>> >> -	/* FIXME: Support M64 for P7IOC */
>> >> -	if (phb->type != PNV_PHB_IODA2) {
>> >> +	if (phb->type != PNV_PHB_IODA1 && phb->type != PNV_PHB_IODA2) {
>> >>  		pr_info("  Not support M64 window\n");
>> >>  		return;
>> >>  	}
>> >> @@ -364,7 +444,10 @@ static void __init pnv_ioda_parse_m64_window(struct 
>> >pnv_phb *phb)
>> >>  
>> >>  	/* Use last M64 BAR to cover M64 window */
>> >>  	phb->ioda.m64_bar_idx = 15;
>> >> -	phb->init_m64 = pnv_ioda2_init_m64;
>> >> +	if (phb->type == PNV_PHB_IODA1)
>> >> +		phb->init_m64 = pnv_ioda1_init_m64;
>> >> +	else
>> >> +		phb->init_m64 = pnv_ioda2_init_m64;
>> >>  	phb->reserve_m64_pe = pnv_ioda_reserve_m64_pe;
>> >>  	phb->pick_m64_pe = pnv_ioda_pick_m64_pe;
>> >>  }
>> >> 
>> >
>> 
>

Re: [PATCH v9 12/26] powerpc/powernv/ioda1: M64 support on P7IOC

[Alistair Popple]

Thanks for the clarifications Gavin. Aside from the WARN_ON() (which is not a
major thing) everything looks good.

Reviewed-By: Alistair Popple <alistair@popple.id.au>

On Thu, 5 May 2016 10:40:33 Gavin Shan wrote:
> On Thu, May 05, 2016 at 09:53:51AM +1000, Alistair Popple wrote:
> >On Wed, 4 May 2016 16:48:53 Gavin Shan wrote:
> >> On Wed, May 04, 2016 at 03:17:51PM +1000, Alistair Popple wrote:
> >> >On Tue, 3 May 2016 15:41:31 Gavin Shan wrote:
> >> >> This enables M64 window on P7IOC, which has been enabled on PHB3.
> >> >
> >> >Have we tested that this works with an adaptor? This looks to be enabling 
> >> >support for something that didn't previously work (64-bit BARs on P7IOC)?
> >> >
> >> 
> >> The M64 isn't supported on P7IOC before, meaning the PCI device's M64
> >> BAR is covered by PHB's M32 window. With the patch applied, the PCI
> >> device's M64 BAR is covered by PHB's M64 window as below log I got
> >> from vpl4. The kernel including the series of patches boots successfully
> >> on vpl4 and all looks normal.
> >
> >So you're changing the way 64-bit BARs work on P7IOC to use a different bit of
> >the hardware/firmware? This means it could break older systems if there is
> >an issue with the M64 support. Other than the comments below the code looks
> >reasonable. However I am assuming you've tested that cards with 64-bit BARs
> >still function on P7 after this change as I'm not familiar enough to comment on
> >the specific hardware details, although your comments make sense and match the
> >code.
> >
> 
> Nope, it doesn't rely on changes in firmware. The old firmware enables and
> exposes PHB M64 window. The kernel doesn't use it on P7IOC until this patch.
> On VPL4, there is a IPR adapter (0000:60:00.0) as below kernel log indicates.
> Its BAR#2 is M64 (64-bits prefetchable) BAR. The BAR#2 is covered by PHB's M64
> window with this patch. The PHB's M64 window is [0x00003da800000000..0x00003dafffffffff].
> After it's applied, the IPR and root filesystem resident in the disk drive out
> of it work fine.
> 
> >> [    0.246110] pci 0000:60:00.0: BAR 2: assigned [mem 0x3da810000000-0x3da810ffffff 64bit pref]
> >> [    0.246218] pci 0000:60:00.0: BAR 0: assigned [mem 0x3da081000000-0x3da08103ffff 64bit]
> >> [    0.246306] pci 0000:60:00.0: BAR 6: assigned [mem 0x3da081040000-0x3da08105ffff pref]
> >> [    0.246392] pci 0000:60     : [PE# 001] Secondary bus 96 associated with PE#1
> >> [    0.246484] pci 0000:60     : [PE# 001] DMA weight 15 (64), assigned (0) 1 DMA32 segments
> >> [    0.246552] pci 0000:60     : [PE# 001]  Setting up 32-bit TCE table at 00000000..0fffffff
> >> 
> >> >> Different from PHB3 where 16 M64 BARs are supported and each of
> >> >> them can be owned by one particular PE# exclusively or divided
> >> >> evenly to 256 segments, every P7IOC PHB has 16 M64 BARs and each
> >> >> of them are divided to 8 segments. So every P7IOC PHB supports
> >> >> 128 M64 segments in total. P7IOC has M64DT, which helps mapping
> >> >> one particular M64 segment# to arbitrary PE#. PHB3 doesn't have
> >> >> M64DT, indicating that one M64 segment can only be pinned to the
> >> >> fixed PE#.
> >> >> 
> >> >> In order to unified M64 support M64 on P7IOC and PHB3, we just
> >> >> provide 128 M64 segments on every P7IOC PHB and each of them is
> >> >> pinned to the fixed PE# by bypassing the function of M64DT. In
> >> >> turn, we just need different phb->init_m64() for P7IOC and PHB3
> >> >> and maps M64 segment in pnv_ioda_reserve_m64_pe() for P7IOC, most
> >> >> of the code are shared by them.
> >> >> 
> >> >> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> >> >> ---
> >> >>  arch/powerpc/platforms/powernv/pci-ioda.c | 89 
> >> >+++++++++++++++++++++++++++++--
> >> >>  1 file changed, 86 insertions(+), 3 deletions(-)
> >> >> 
> >> >> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c 
> >> >b/arch/powerpc/platforms/powernv/pci-ioda.c
> >> >> index 37f22b0..a1b74ec 100644
> >> >> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
> >> >> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
> >> >> @@ -48,6 +48,9 @@
> >> >>  #include "powernv.h"
> >> >>  #include "pci.h"
> >> >>  
> >> >> +#define PNV_IODA1_M64_NUM	16	/* Number of M64 BARs	*/
> >> >> +#define PNV_IODA1_M64_SEGS	8	/* Segments per M64 BAR	*/
> >> >> +
> >> >>  /* 256M DMA window, 4K TCE pages, 8 bytes TCE */
> >> >>  #define TCE32_TABLE_SIZE	((0x10000000 / 0x1000) * 8)
> >> >>  
> >> >> @@ -246,6 +249,64 @@ static void pnv_ioda_reserve_dev_m64_pe(struct pci_dev 
> >> >*pdev,
> >> >>  	}
> >> >>  }
> >> >>  
> >> >> +static int pnv_ioda1_init_m64(struct pnv_phb *phb)
> >> >> +{
> >> >> +	struct resource *r;
> >> >> +	int index;
> >> >> +
> >> >> +	/*
> >> >> +	 * There are 16 M64 BARs, each of which has 8 segments. So
> >> >> +	 * there are as many M64 segments as the maximum number of
> >> >> +	 * PEs, which is 128.
> >> >> +	 */
> >> >> +	for (index = 0; index < PNV_IODA1_M64_NUM; index++) {
> >> >> +		unsigned long base, segsz = phb->ioda.m64_segsize;
> >> >> +		int64_t rc;
> >> >> +
> >> >> +		base = phb->ioda.m64_base +
> >> >> +		       index * PNV_IODA1_M64_SEGS * segsz;
> >> >> +		rc = opal_pci_set_phb_mem_window(phb->opal_id,
> >> >> +				OPAL_M64_WINDOW_TYPE, index, base, 0,
> >> >> +				PNV_IODA1_M64_SEGS * segsz);
> >
> >Has firmware always supported OPAL_M64_WINDOW_TYPE for P7IOC? If older versions
> >don't support it what happens? Do we gracefully fall back to the old mode of
> >allocating all BARs from the M32 window or does it break?
> >
> 
> Old firmware enables and exposes M64 window (OPAL_M64_WINDOW_TYPE), but kernel
> doesn't use it until this patch. If the OPAL call fails, errcode is returned.
> The caller, pnv_pci_init_ioda_phb(), disables the window. It means the PCI
> device M64 BARs will fail back to be covered by M32 window. The path is same
> on PHB3 and P7IOC.
> 
> >> >> +		if (rc != OPAL_SUCCESS) {
> >> >> +			pr_warn("  Error %lld setting M64 PHB#%d-BAR#%d\n",
> >> >> +				rc, phb->hose->global_number, index);
> >> >> +			goto fail;
> >> >> +		}
> >> >> +
> >> >> +		rc = opal_pci_phb_mmio_enable(phb->opal_id,
> >> >> +				OPAL_M64_WINDOW_TYPE, index,
> >> >> +				OPAL_ENABLE_M64_SPLIT);
> >> >> +		if (rc != OPAL_SUCCESS) {
> >> >> +			pr_warn("  Error %lld enabling M64 PHB#%d-BAR#%d\n",
> >> >> +				rc, phb->hose->global_number, index);
> >> >> +			goto fail;
> >> >> +		}
> >> >> +	}
> >> >> +
> >> >> +	/*
> >> >> +	 * Exclude the segment used by the reserved PE, which
> >> >> +	 * is expected to be 0 or last supported PE#.
> >> >> +	 */
> >> >> +	r = &phb->hose->mem_resources[1];
> >> >> +	if (phb->ioda.reserved_pe_idx == 0)
> >> >> +		r->start += phb->ioda.m64_segsize;
> >> >> +	else if (phb->ioda.reserved_pe_idx == (phb->ioda.total_pe_num - 1))
> >> >> +		r->end -= phb->ioda.m64_segsize;
> >> >> +	else
> >> >> +		pr_warn("  Cannot cut M64 segment for reserved PE#%d\n",
> >> >> +			phb->ioda.reserved_pe_idx);
> >
> >Should this be a WARN_ON()? If this condition can only exist because a future
> >programmer changes the reserved_pe_idx then I think it should be a WARN_ON()
> >as the above message would be too easy to miss.
> >
> 
> Yeah, a WARN_ON() should be better.
> 
> Thanks,
> Gavin
> 
> >> >> +
> >> >> +	return 0;
> >> >> +
> >> >> +fail:
> >> >> +	for ( ; index >= 0; index--)
> >> >> +		opal_pci_phb_mmio_enable(phb->opal_id,
> >> >> +			OPAL_M64_WINDOW_TYPE, index, OPAL_DISABLE_M64);
> >> >> +
> >> >> +	return -EIO;
> >> >> +}
> >> >> +
> >> >>  static void pnv_ioda_reserve_m64_pe(struct pci_bus *bus,
> >> >>  				    unsigned long *pe_bitmap,
> >> >>  				    bool all)
> >> >> @@ -315,6 +376,26 @@ static unsigned int pnv_ioda_pick_m64_pe(struct pci_bus 
> >> >*bus, bool all)
> >> >>  			pe->master = master_pe;
> >> >>  			list_add_tail(&pe->list, &master_pe->slaves);
> >> >>  		}
> >> >> +
> >> >> +		/*
> >> >> +		 * P7IOC supports M64DT, which helps mapping M64 segment
> >> >> +		 * to one particular PE#. However, PHB3 has fixed mapping
> >> >> +		 * between M64 segment and PE#. In order to have same logic
> >> >> +		 * for P7IOC and PHB3, we enforce fixed mapping between M64
> >> >> +		 * segment and PE# on P7IOC.
> >> >> +		 */
> >> >> +		if (phb->type == PNV_PHB_IODA1) {
> >> >> +			int64_t rc;
> >> >> +
> >> >> +			rc = opal_pci_map_pe_mmio_window(phb->opal_id,
> >> >> +					pe->pe_number, OPAL_M64_WINDOW_TYPE,
> >> >> +					pe->pe_number / PNV_IODA1_M64_SEGS,
> >> >> +					pe->pe_number % PNV_IODA1_M64_SEGS);
> >> >> +			if (rc != OPAL_SUCCESS)
> >> >> +				pr_warn("%s: Error %lld mapping M64 for 
> >> >PHB#%d-PE#%d\n",
> >> >> +					__func__, rc, phb->hose-
> >> >>global_number,
> >> >> +					pe->pe_number);
> >> >> +		}
> >> >>  	}
> >> >>  
> >> >>  	kfree(pe_alloc);
> >> >> @@ -329,8 +410,7 @@ static void __init pnv_ioda_parse_m64_window(struct 
> >> >pnv_phb *phb)
> >> >>  	const u32 *r;
> >> >>  	u64 pci_addr;
> >> >>  
> >> >> -	/* FIXME: Support M64 for P7IOC */
> >> >> -	if (phb->type != PNV_PHB_IODA2) {
> >> >> +	if (phb->type != PNV_PHB_IODA1 && phb->type != PNV_PHB_IODA2) {
> >> >>  		pr_info("  Not support M64 window\n");
> >> >>  		return;
> >> >>  	}
> >> >> @@ -364,7 +444,10 @@ static void __init pnv_ioda_parse_m64_window(struct 
> >> >pnv_phb *phb)
> >> >>  
> >> >>  	/* Use last M64 BAR to cover M64 window */
> >> >>  	phb->ioda.m64_bar_idx = 15;
> >> >> -	phb->init_m64 = pnv_ioda2_init_m64;
> >> >> +	if (phb->type == PNV_PHB_IODA1)
> >> >> +		phb->init_m64 = pnv_ioda1_init_m64;
> >> >> +	else
> >> >> +		phb->init_m64 = pnv_ioda2_init_m64;
> >> >>  	phb->reserve_m64_pe = pnv_ioda_reserve_m64_pe;
> >> >>  	phb->pick_m64_pe = pnv_ioda_pick_m64_pe;
> >> >>  }
> >> >> 
> >> >
> >> 
> >
> 

Re: [PATCH v9 16/26] powerpc/powernv/ioda1: Improve DMA32 segment track

[Gavin Shan]

On Wed, May 04, 2016 at 11:20:01PM +1000, Gavin Shan wrote:
>On Tue, May 03, 2016 at 03:41:35PM +1000, Gavin Shan wrote:
>>In current implementation, the DMA32 segments required by one specific
>>PE isn't calculated with the information hold in the PE independently.
>>It conflicts with the PCI hotplug design: PE centralized, meaning the
>>PE's DMA32 segments should be calculated from the information hold in
>>the PE independently.
>>
>>This introduces an array (@dma32_segmap) for every PHB to track the
>>DMA32 segmeng usage. Besides, this moves the logic calculating PE's
>>consumed DMA32 segments to pnv_pci_ioda1_setup_dma_pe() so that PE's
>>DMA32 segments are calculated/allocated from the information hold in
>>the PE (DMA32 weight). Also the logic is improved: we try to allocate
>>as much DMA32 segments as we can. It's acceptable that number of DMA32
>>segments less than the expected number are allocated.
>>
>>Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>
>This can cause overlapped DMA32 segments to different PEs in some cases.
>I already had the fix and hold for posting after discussion with Michael
>on how to handle the series tomorrow.
>

I'm going to send v10 as a reply to this thread.

Thanks,
Gavin

>>---
>> arch/powerpc/platforms/powernv/pci-ioda.c | 110 ++++++++++++++++--------------
>> arch/powerpc/platforms/powernv/pci.h      |   7 +-
>> 2 files changed, 61 insertions(+), 56 deletions(-)
>>
>>diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
>>index f70a4e0..cfd2906 100644
>>--- a/arch/powerpc/platforms/powernv/pci-ioda.c
>>+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
>>@@ -2011,27 +2011,57 @@ static unsigned int pnv_pci_ioda_pe_dma_weight(struct pnv_ioda_pe *pe)
>> }
>>
>> static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
>>-				       struct pnv_ioda_pe *pe,
>>-				       unsigned int base,
>>-				       unsigned int segs)
>>+				       struct pnv_ioda_pe *pe)
>> {
>>
>> 	struct page *tce_mem = NULL;
>> 	struct iommu_table *tbl;
>>-	unsigned int tce32_segsz, i;
>>+	unsigned int weight, total_weight = 0;
>>+	unsigned int tce32_segsz, base, segs, i;
>> 	int64_t rc;
>> 	void *addr;
>>
>> 	/* XXX FIXME: Handle 64-bit only DMA devices */
>> 	/* XXX FIXME: Provide 64-bit DMA facilities & non-4K TCE tables etc.. */
>> 	/* XXX FIXME: Allocate multi-level tables on PHB3 */
>>+	pci_walk_bus(phb->hose->bus, pnv_pci_ioda_dev_dma_weight,
>>+		     &total_weight);
>>+	weight = pnv_pci_ioda_pe_dma_weight(pe);
>>+
>>+	segs = (weight * phb->ioda.dma32_count) / total_weight;
>>+	if (!segs)
>>+		segs = 1;
>>+
>>+	/*
>>+	 * Allocate contiguous DMA32 segments. We begin with the expected
>>+	 * number of segments. With one more attempt, the number of DMA32
>>+	 * segments to be allocated is decreased by one until one segment
>>+	 * is allocated successfully.
>>+	 */
>>+	do {
>>+		for (base = 0; base <= phb->ioda.dma32_count - segs; base++) {
>>+			for (i = base; i < base + segs; i++) {
>>+				if (phb->ioda.dma32_segmap[i] !=
>>+				    IODA_INVALID_PE)
>>+					goto found;
>>+			}
>>+		}
>>+	} while (segs--);
>>+
>>+	if (!segs) {
>>+		pe_warn(pe, "No available DMA32 segments\n");
>>+		return;
>>+	}
>>
>>+found:
>> 	tbl = pnv_pci_table_alloc(phb->hose->node);
>> 	iommu_register_group(&pe->table_group, phb->hose->global_number,
>> 			pe->pe_number);
>> 	pnv_pci_link_table_and_group(phb->hose->node, 0, tbl, &pe->table_group);
>>
>> 	/* Grab a 32-bit TCE table */
>>+	pe_info(pe, "DMA weight %d (%d), assigned (%d) %d DMA32 segments\n",
>>+		weight, total_weight, base, segs);
>> 	pe_info(pe, " Setting up 32-bit TCE table at %08x..%08x\n",
>> 		base * PNV_IODA1_DMA32_SEGSIZE,
>> 		(base + segs) * PNV_IODA1_DMA32_SEGSIZE - 1);
>>@@ -2068,6 +2098,10 @@ static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
>> 		}
>> 	}
>>
>>+	/* Setup DMA32 segment mapping */
>>+	for (i = base; i < base + segs; i++)
>>+		phb->ioda.dma32_segmap[i] = pe->pe_number;
>>+
>> 	/* Setup linux iommu table */
>> 	pnv_pci_setup_iommu_table(tbl, addr, tce32_segsz * segs,
>> 				  base * PNV_IODA1_DMA32_SEGSIZE,
>>@@ -2542,73 +2576,34 @@ static void pnv_pci_ioda2_setup_dma_pe(struct pnv_phb *phb,
>> static void pnv_ioda_setup_dma(struct pnv_phb *phb)
>> {
>> 	struct pci_controller *hose = phb->hose;
>>-	unsigned int weight, total_weight, dma_pe_count;
>>-	unsigned int residual, remaining, segs, base;
>> 	struct pnv_ioda_pe *pe;
>>-
>>-	total_weight = 0;
>>-	pci_walk_bus(phb->hose->bus, pnv_pci_ioda_dev_dma_weight,
>>-		     &total_weight);
>>-
>>-	dma_pe_count = 0;
>>-	list_for_each_entry(pe, &phb->ioda.pe_list, list) {
>>-		weight = pnv_pci_ioda_pe_dma_weight(pe);
>>-		if (weight > 0)
>>-			dma_pe_count++;
>>-	}
>>+	unsigned int weight;
>>
>> 	/* If we have more PE# than segments available, hand out one
>> 	 * per PE until we run out and let the rest fail. If not,
>> 	 * then we assign at least one segment per PE, plus more based
>> 	 * on the amount of devices under that PE
>> 	 */
>>-	if (dma_pe_count > phb->ioda.tce32_count)
>>-		residual = 0;
>>-	else
>>-		residual = phb->ioda.tce32_count - dma_pe_count;
>>-
>>-	pr_info("PCI: Domain %04x has %ld available 32-bit DMA segments\n",
>>-		hose->global_number, phb->ioda.tce32_count);
>>-	pr_info("PCI: %d PE# for a total weight of %d\n",
>>-		dma_pe_count, total_weight);
>>+	pr_info("PCI: Domain %04x has %d available 32-bit DMA segments\n",
>>+		hose->global_number, phb->ioda.dma32_count);
>>
>> 	pnv_pci_ioda_setup_opal_tce_kill(phb);
>>
>>-	/* Walk our PE list and configure their DMA segments, hand them
>>-	 * out one base segment plus any residual segments based on
>>-	 * weight
>>-	 */
>>-	remaining = phb->ioda.tce32_count;
>>-	base = 0;
>>+	/* Walk our PE list and configure their DMA segments */
>> 	list_for_each_entry(pe, &phb->ioda.pe_list, list) {
>> 		weight = pnv_pci_ioda_pe_dma_weight(pe);
>> 		if (!weight)
>> 			continue;
>>
>>-		if (!remaining) {
>>-			pe_warn(pe, "No DMA32 resources available\n");
>>-			continue;
>>-		}
>>-		segs = 1;
>>-		if (residual) {
>>-			segs += ((weight * residual) + (total_weight / 2)) /
>>-				total_weight;
>>-			if (segs > remaining)
>>-				segs = remaining;
>>-		}
>>-
>> 		/*
>> 		 * For IODA2 compliant PHB3, we needn't care about the weight.
>> 		 * The all available 32-bits DMA space will be assigned to
>> 		 * the specific PE.
>> 		 */
>> 		if (phb->type == PNV_PHB_IODA1) {
>>-			pe_info(pe, "DMA weight %d, assigned %d DMA32 segments\n",
>>-				weight, segs);
>>-			pnv_pci_ioda1_setup_dma_pe(phb, pe, base, segs);
>>+			pnv_pci_ioda1_setup_dma_pe(phb, pe);
>> 		} else if (phb->type == PNV_PHB_IODA2) {
>> 			pe_info(pe, "Assign DMA32 space\n");
>>-			segs = 0;
>> 			pnv_pci_ioda2_setup_dma_pe(phb, pe);
>> 		} else if (phb->type == PNV_PHB_NPU) {
>> 			/*
>>@@ -2618,9 +2613,6 @@ static void pnv_ioda_setup_dma(struct pnv_phb *phb)
>> 			 * as the PHB3 TVT.
>> 			 */
>> 		}
>>-
>>-		remaining -= segs;
>>-		base += segs;
>> 	}
>> }
>>
>>@@ -3327,7 +3319,8 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
>> {
>> 	struct pci_controller *hose;
>> 	struct pnv_phb *phb;
>>-	unsigned long size, m64map_off, m32map_off, pemap_off, iomap_off = 0;
>>+	unsigned long size, m64map_off, m32map_off, pemap_off;
>>+	unsigned long iomap_off = 0, dma32map_off = 0;
>> 	const __be64 *prop64;
>> 	const __be32 *prop32;
>> 	int len;
>>@@ -3413,6 +3406,10 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
>> 	phb->ioda.io_segsize = phb->ioda.io_size / phb->ioda.total_pe_num;
>> 	phb->ioda.io_pci_base = 0; /* XXX calculate this ? */
>>
>>+	/* Calculate how many 32-bit TCE segments we have */
>>+	phb->ioda.dma32_count = phb->ioda.m32_pci_base /
>>+				PNV_IODA1_DMA32_SEGSIZE;
>>+
>> 	/* Allocate aux data & arrays. We don't have IO ports on PHB3 */
>> 	size = _ALIGN_UP(phb->ioda.total_pe_num / 8, sizeof(unsigned long));
>> 	m64map_off = size;
>>@@ -3422,6 +3419,9 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
>> 	if (phb->type == PNV_PHB_IODA1) {
>> 		iomap_off = size;
>> 		size += phb->ioda.total_pe_num * sizeof(phb->ioda.io_segmap[0]);
>>+		dma32map_off = size;
>>+		size += phb->ioda.dma32_count *
>>+			sizeof(phb->ioda.dma32_segmap[0]);
>> 	}
>> 	pemap_off = size;
>> 	size += phb->ioda.total_pe_num * sizeof(struct pnv_ioda_pe);
>>@@ -3437,6 +3437,10 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
>> 		phb->ioda.io_segmap = aux + iomap_off;
>> 		for (segno = 0; segno < phb->ioda.total_pe_num; segno++)
>> 			phb->ioda.io_segmap[segno] = IODA_INVALID_PE;
>>+
>>+		phb->ioda.dma32_segmap = aux + dma32map_off;
>>+		for (segno = 0; segno < phb->ioda.dma32_count; segno++)
>>+			phb->ioda.dma32_segmap[segno] = IODA_INVALID_PE;
>> 	}
>> 	phb->ioda.pe_array = aux + pemap_off;
>> 	set_bit(phb->ioda.reserved_pe_idx, phb->ioda.pe_alloc);
>>@@ -3445,7 +3449,7 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
>> 	mutex_init(&phb->ioda.pe_list_mutex);
>>
>> 	/* Calculate how many 32-bit TCE segments we have */
>>-	phb->ioda.tce32_count = phb->ioda.m32_pci_base /
>>+	phb->ioda.dma32_count = phb->ioda.m32_pci_base /
>> 				PNV_IODA1_DMA32_SEGSIZE;
>>
>> #if 0 /* We should really do that ... */
>>diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
>>index 117cfcd..14d9391 100644
>>--- a/arch/powerpc/platforms/powernv/pci.h
>>+++ b/arch/powerpc/platforms/powernv/pci.h
>>@@ -142,6 +142,10 @@ struct pnv_phb {
>> 		unsigned int		*m32_segmap;
>> 		unsigned int		*io_segmap;
>>
>>+		/* DMA32 segment maps - IODA1 only */
>>+		unsigned int		dma32_count;
>>+		unsigned int		*dma32_segmap;
>>+
>> 		/* IRQ chip */
>> 		int			irq_chip_init;
>> 		struct irq_chip		irq_chip;
>>@@ -158,9 +162,6 @@ struct pnv_phb {
>> 		 */
>> 		unsigned char		pe_rmap[0x10000];
>>
>>-		/* 32-bit TCE tables allocation */
>>-		unsigned long		tce32_count;
>>-
>> 		/* TCE cache invalidate registers (physical and
>> 		 * remapped)
>> 		 */
>>-- 
>>2.1.0
>>

[PATCH v10 12/26] powerpc/powernv/ioda1: M64 support on P7IOC

[Gavin Shan]

This enables M64 window on P7IOC, which has been enabled on PHB3.
Different from PHB3 where 16 M64 BARs are supported and each of
them can be owned by one particular PE# exclusively or divided
evenly to 256 segments, every P7IOC PHB has 16 M64 BARs and each
of them are divided to 8 segments. So every P7IOC PHB supports
128 M64 segments in total. P7IOC has M64DT, which helps mapping
one particular M64 segment# to arbitrary PE#. PHB3 doesn't have
M64DT, indicating that one M64 segment can only be pinned to the
fixed PE#.

In order to unified M64 support M64 on P7IOC and PHB3, we just
provide 128 M64 segments on every P7IOC PHB and each of them is
pinned to the fixed PE# by bypassing the function of M64DT. In
turn, we just need different phb->init_m64() for P7IOC and PHB3
and maps M64 segment in pnv_ioda_reserve_m64_pe() for P7IOC, most
of the code are shared by them.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
Reviewed-by: Alistair Popple <alistair@popple.id.au>
---
v10: Replace pr_warn() with WARN() suggested by Alistair
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 89 +++++++++++++++++++++++++++++--
 1 file changed, 86 insertions(+), 3 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 37f22b0..30a6d79 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -48,6 +48,9 @@
 #include "powernv.h"
 #include "pci.h"
 
+#define PNV_IODA1_M64_NUM	16	/* Number of M64 BARs	*/
+#define PNV_IODA1_M64_SEGS	8	/* Segments per M64 BAR	*/
+
 /* 256M DMA window, 4K TCE pages, 8 bytes TCE */
 #define TCE32_TABLE_SIZE	((0x10000000 / 0x1000) * 8)
 
@@ -246,6 +249,64 @@ static void pnv_ioda_reserve_dev_m64_pe(struct pci_dev *pdev,
 	}
 }
 
+static int pnv_ioda1_init_m64(struct pnv_phb *phb)
+{
+	struct resource *r;
+	int index;
+
+	/*
+	 * There are 16 M64 BARs, each of which has 8 segments. So
+	 * there are as many M64 segments as the maximum number of
+	 * PEs, which is 128.
+	 */
+	for (index = 0; index < PNV_IODA1_M64_NUM; index++) {
+		unsigned long base, segsz = phb->ioda.m64_segsize;
+		int64_t rc;
+
+		base = phb->ioda.m64_base +
+		       index * PNV_IODA1_M64_SEGS * segsz;
+		rc = opal_pci_set_phb_mem_window(phb->opal_id,
+				OPAL_M64_WINDOW_TYPE, index, base, 0,
+				PNV_IODA1_M64_SEGS * segsz);
+		if (rc != OPAL_SUCCESS) {
+			pr_warn("  Error %lld setting M64 PHB#%d-BAR#%d\n",
+				rc, phb->hose->global_number, index);
+			goto fail;
+		}
+
+		rc = opal_pci_phb_mmio_enable(phb->opal_id,
+				OPAL_M64_WINDOW_TYPE, index,
+				OPAL_ENABLE_M64_SPLIT);
+		if (rc != OPAL_SUCCESS) {
+			pr_warn("  Error %lld enabling M64 PHB#%d-BAR#%d\n",
+				rc, phb->hose->global_number, index);
+			goto fail;
+		}
+	}
+
+	/*
+	 * Exclude the segment used by the reserved PE, which
+	 * is expected to be 0 or last supported PE#.
+	 */
+	r = &phb->hose->mem_resources[1];
+	if (phb->ioda.reserved_pe_idx == 0)
+		r->start += phb->ioda.m64_segsize;
+	else if (phb->ioda.reserved_pe_idx == (phb->ioda.total_pe_num - 1))
+		r->end -= phb->ioda.m64_segsize;
+	else
+		WARN(1, "Wrong reserved PE#%d on PHB#%d\n",
+		     phb->ioda.reserved_pe_idx, phb->hose->global_number);
+
+	return 0;
+
+fail:
+	for ( ; index >= 0; index--)
+		opal_pci_phb_mmio_enable(phb->opal_id,
+			OPAL_M64_WINDOW_TYPE, index, OPAL_DISABLE_M64);
+
+	return -EIO;
+}
+
 static void pnv_ioda_reserve_m64_pe(struct pci_bus *bus,
 				    unsigned long *pe_bitmap,
 				    bool all)
@@ -315,6 +376,26 @@ static unsigned int pnv_ioda_pick_m64_pe(struct pci_bus *bus, bool all)
 			pe->master = master_pe;
 			list_add_tail(&pe->list, &master_pe->slaves);
 		}
+
+		/*
+		 * P7IOC supports M64DT, which helps mapping M64 segment
+		 * to one particular PE#. However, PHB3 has fixed mapping
+		 * between M64 segment and PE#. In order to have same logic
+		 * for P7IOC and PHB3, we enforce fixed mapping between M64
+		 * segment and PE# on P7IOC.
+		 */
+		if (phb->type == PNV_PHB_IODA1) {
+			int64_t rc;
+
+			rc = opal_pci_map_pe_mmio_window(phb->opal_id,
+					pe->pe_number, OPAL_M64_WINDOW_TYPE,
+					pe->pe_number / PNV_IODA1_M64_SEGS,
+					pe->pe_number % PNV_IODA1_M64_SEGS);
+			if (rc != OPAL_SUCCESS)
+				pr_warn("%s: Error %lld mapping M64 for PHB#%d-PE#%d\n",
+					__func__, rc, phb->hose->global_number,
+					pe->pe_number);
+		}
 	}
 
 	kfree(pe_alloc);
@@ -329,8 +410,7 @@ static void __init pnv_ioda_parse_m64_window(struct pnv_phb *phb)
 	const u32 *r;
 	u64 pci_addr;
 
-	/* FIXME: Support M64 for P7IOC */
-	if (phb->type != PNV_PHB_IODA2) {
+	if (phb->type != PNV_PHB_IODA1 && phb->type != PNV_PHB_IODA2) {
 		pr_info("  Not support M64 window\n");
 		return;
 	}
@@ -364,7 +444,10 @@ static void __init pnv_ioda_parse_m64_window(struct pnv_phb *phb)
 
 	/* Use last M64 BAR to cover M64 window */
 	phb->ioda.m64_bar_idx = 15;
-	phb->init_m64 = pnv_ioda2_init_m64;
+	if (phb->type == PNV_PHB_IODA1)
+		phb->init_m64 = pnv_ioda1_init_m64;
+	else
+		phb->init_m64 = pnv_ioda2_init_m64;
 	phb->reserve_m64_pe = pnv_ioda_reserve_m64_pe;
 	phb->pick_m64_pe = pnv_ioda_pick_m64_pe;
 }
-- 
2.1.0

[PATCH v10 16/26] powerpc/powernv/ioda1: Improve DMA32 segment track

[Gavin Shan]

In current implementation, the DMA32 segments required by one specific
PE isn't calculated with the information hold in the PE independently.
It conflicts with the PCI hotplug design: PE centralized, meaning the
PE's DMA32 segments should be calculated from the information hold in
the PE independently.

This introduces an array (@dma32_segmap) for every PHB to track the
DMA32 segmeng usage. Besides, this moves the logic calculating PE's
consumed DMA32 segments to pnv_pci_ioda1_setup_dma_pe() so that PE's
DMA32 segments are calculated/allocated from the information hold in
the PE (DMA32 weight). Also the logic is improved: we try to allocate
as much DMA32 segments as we can. It's acceptable that number of DMA32
segments less than the expected number are allocated.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
---
v10: Fixed picking same DMA32 segment by different PEs in some cases
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 115 ++++++++++++++++--------------
 arch/powerpc/platforms/powernv/pci.h      |   7 +-
 2 files changed, 66 insertions(+), 56 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 30c825a..98879a0 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -2011,27 +2011,62 @@ static unsigned int pnv_pci_ioda_pe_dma_weight(struct pnv_ioda_pe *pe)
 }
 
 static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
-				       struct pnv_ioda_pe *pe,
-				       unsigned int base,
-				       unsigned int segs)
+				       struct pnv_ioda_pe *pe)
 {
 
 	struct page *tce_mem = NULL;
 	struct iommu_table *tbl;
-	unsigned int tce32_segsz, i;
+	unsigned int weight, total_weight = 0;
+	unsigned int tce32_segsz, base, segs, avail, i;
 	int64_t rc;
 	void *addr;
 
 	/* XXX FIXME: Handle 64-bit only DMA devices */
 	/* XXX FIXME: Provide 64-bit DMA facilities & non-4K TCE tables etc.. */
 	/* XXX FIXME: Allocate multi-level tables on PHB3 */
+	weight = pnv_pci_ioda_pe_dma_weight(pe);
+	if (!weight)
+		return;
+
+	pci_walk_bus(phb->hose->bus, pnv_pci_ioda_dev_dma_weight,
+		     &total_weight);
+	segs = (weight * phb->ioda.dma32_count) / total_weight;
+	if (!segs)
+		segs = 1;
 
+	/*
+	 * Allocate contiguous DMA32 segments. We begin with the expected
+	 * number of segments. With one more attempt, the number of DMA32
+	 * segments to be allocated is decreased by one until one segment
+	 * is allocated successfully.
+	 */
+	do {
+		for (base = 0; base <= phb->ioda.dma32_count - segs; base++) {
+			for (avail = 0, i = base; i < base + segs; i++) {
+				if (phb->ioda.dma32_segmap[i] ==
+				    IODA_INVALID_PE)
+					avail++;
+			}
+
+			if (avail == segs)
+				goto found;
+		}
+	} while (--segs);
+
+	if (!segs) {
+		pe_warn(pe, "No available DMA32 segments\n");
+		return;
+	}
+
+found:
 	tbl = pnv_pci_table_alloc(phb->hose->node);
 	iommu_register_group(&pe->table_group, phb->hose->global_number,
 			pe->pe_number);
 	pnv_pci_link_table_and_group(phb->hose->node, 0, tbl, &pe->table_group);
 
 	/* Grab a 32-bit TCE table */
+	pe_info(pe, "DMA weight %d (%d), assigned (%d) %d DMA32 segments\n",
+		weight, total_weight, base, segs);
 	pe_info(pe, " Setting up 32-bit TCE table at %08x..%08x\n",
 		base * PNV_IODA1_DMA32_SEGSIZE,
 		(base + segs) * PNV_IODA1_DMA32_SEGSIZE - 1);
@@ -2068,6 +2103,10 @@ static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
 		}
 	}
 
+	/* Setup DMA32 segment mapping */
+	for (i = base; i < base + segs; i++)
+		phb->ioda.dma32_segmap[i] = pe->pe_number;
+
 	/* Setup linux iommu table */
 	pnv_pci_setup_iommu_table(tbl, addr, tce32_segsz * segs,
 				  base * PNV_IODA1_DMA32_SEGSIZE,
@@ -2542,73 +2581,34 @@ static void pnv_pci_ioda2_setup_dma_pe(struct pnv_phb *phb,
 static void pnv_ioda_setup_dma(struct pnv_phb *phb)
 {
 	struct pci_controller *hose = phb->hose;
-	unsigned int weight, total_weight, dma_pe_count;
-	unsigned int residual, remaining, segs, base;
 	struct pnv_ioda_pe *pe;
-
-	total_weight = 0;
-	pci_walk_bus(phb->hose->bus, pnv_pci_ioda_dev_dma_weight,
-		     &total_weight);
-
-	dma_pe_count = 0;
-	list_for_each_entry(pe, &phb->ioda.pe_list, list) {
-		weight = pnv_pci_ioda_pe_dma_weight(pe);
-		if (weight > 0)
-			dma_pe_count++;
-	}
+	unsigned int weight;
 
 	/* If we have more PE# than segments available, hand out one
 	 * per PE until we run out and let the rest fail. If not,
 	 * then we assign at least one segment per PE, plus more based
 	 * on the amount of devices under that PE
 	 */
-	if (dma_pe_count > phb->ioda.tce32_count)
-		residual = 0;
-	else
-		residual = phb->ioda.tce32_count - dma_pe_count;
-
-	pr_info("PCI: Domain %04x has %ld available 32-bit DMA segments\n",
-		hose->global_number, phb->ioda.tce32_count);
-	pr_info("PCI: %d PE# for a total weight of %d\n",
-		dma_pe_count, total_weight);
+	pr_info("PCI: Domain %04x has %d available 32-bit DMA segments\n",
+		hose->global_number, phb->ioda.dma32_count);
 
 	pnv_pci_ioda_setup_opal_tce_kill(phb);
 
-	/* Walk our PE list and configure their DMA segments, hand them
-	 * out one base segment plus any residual segments based on
-	 * weight
-	 */
-	remaining = phb->ioda.tce32_count;
-	base = 0;
+	/* Walk our PE list and configure their DMA segments */
 	list_for_each_entry(pe, &phb->ioda.pe_list, list) {
 		weight = pnv_pci_ioda_pe_dma_weight(pe);
 		if (!weight)
 			continue;
 
-		if (!remaining) {
-			pe_warn(pe, "No DMA32 resources available\n");
-			continue;
-		}
-		segs = 1;
-		if (residual) {
-			segs += ((weight * residual) + (total_weight / 2)) /
-				total_weight;
-			if (segs > remaining)
-				segs = remaining;
-		}
-
 		/*
 		 * For IODA2 compliant PHB3, we needn't care about the weight.
 		 * The all available 32-bits DMA space will be assigned to
 		 * the specific PE.
 		 */
 		if (phb->type == PNV_PHB_IODA1) {
-			pe_info(pe, "DMA weight %d, assigned %d DMA32 segments\n",
-				weight, segs);
-			pnv_pci_ioda1_setup_dma_pe(phb, pe, base, segs);
+			pnv_pci_ioda1_setup_dma_pe(phb, pe);
 		} else if (phb->type == PNV_PHB_IODA2) {
 			pe_info(pe, "Assign DMA32 space\n");
-			segs = 0;
 			pnv_pci_ioda2_setup_dma_pe(phb, pe);
 		} else if (phb->type == PNV_PHB_NPU) {
 			/*
@@ -2618,9 +2618,6 @@ static void pnv_ioda_setup_dma(struct pnv_phb *phb)
 			 * as the PHB3 TVT.
 			 */
 		}
-
-		remaining -= segs;
-		base += segs;
 	}
 }
 
@@ -3327,7 +3324,8 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 {
 	struct pci_controller *hose;
 	struct pnv_phb *phb;
-	unsigned long size, m64map_off, m32map_off, pemap_off, iomap_off = 0;
+	unsigned long size, m64map_off, m32map_off, pemap_off;
+	unsigned long iomap_off = 0, dma32map_off = 0;
 	const __be64 *prop64;
 	const __be32 *prop32;
 	int len;
@@ -3413,6 +3411,10 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 	phb->ioda.io_segsize = phb->ioda.io_size / phb->ioda.total_pe_num;
 	phb->ioda.io_pci_base = 0; /* XXX calculate this ? */
 
+	/* Calculate how many 32-bit TCE segments we have */
+	phb->ioda.dma32_count = phb->ioda.m32_pci_base /
+				PNV_IODA1_DMA32_SEGSIZE;
+
 	/* Allocate aux data & arrays. We don't have IO ports on PHB3 */
 	size = _ALIGN_UP(phb->ioda.total_pe_num / 8, sizeof(unsigned long));
 	m64map_off = size;
@@ -3422,6 +3424,9 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 	if (phb->type == PNV_PHB_IODA1) {
 		iomap_off = size;
 		size += phb->ioda.total_pe_num * sizeof(phb->ioda.io_segmap[0]);
+		dma32map_off = size;
+		size += phb->ioda.dma32_count *
+			sizeof(phb->ioda.dma32_segmap[0]);
 	}
 	pemap_off = size;
 	size += phb->ioda.total_pe_num * sizeof(struct pnv_ioda_pe);
@@ -3437,6 +3442,10 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 		phb->ioda.io_segmap = aux + iomap_off;
 		for (segno = 0; segno < phb->ioda.total_pe_num; segno++)
 			phb->ioda.io_segmap[segno] = IODA_INVALID_PE;
+
+		phb->ioda.dma32_segmap = aux + dma32map_off;
+		for (segno = 0; segno < phb->ioda.dma32_count; segno++)
+			phb->ioda.dma32_segmap[segno] = IODA_INVALID_PE;
 	}
 	phb->ioda.pe_array = aux + pemap_off;
 	set_bit(phb->ioda.reserved_pe_idx, phb->ioda.pe_alloc);
@@ -3445,7 +3454,7 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 	mutex_init(&phb->ioda.pe_list_mutex);
 
 	/* Calculate how many 32-bit TCE segments we have */
-	phb->ioda.tce32_count = phb->ioda.m32_pci_base /
+	phb->ioda.dma32_count = phb->ioda.m32_pci_base /
 				PNV_IODA1_DMA32_SEGSIZE;
 
 #if 0 /* We should really do that ... */
diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
index 117cfcd..14d9391 100644
--- a/arch/powerpc/platforms/powernv/pci.h
+++ b/arch/powerpc/platforms/powernv/pci.h
@@ -142,6 +142,10 @@ struct pnv_phb {
 		unsigned int		*m32_segmap;
 		unsigned int		*io_segmap;
 
+		/* DMA32 segment maps - IODA1 only */
+		unsigned int		dma32_count;
+		unsigned int		*dma32_segmap;
+
 		/* IRQ chip */
 		int			irq_chip_init;
 		struct irq_chip		irq_chip;
@@ -158,9 +162,6 @@ struct pnv_phb {
 		 */
 		unsigned char		pe_rmap[0x10000];
 
-		/* 32-bit TCE tables allocation */
-		unsigned long		tce32_count;
-
 		/* TCE cache invalidate registers (physical and
 		 * remapped)
 		 */
-- 
2.1.0

Re: [PATCH v9 07/26] powerpc/powernv: Fix initial IO and M32 segmap

[Alexey Kardashevskiy]

On 05/03/2016 03:41 PM, Gavin Shan wrote:
> There are two arrays for IO and M32 segment maps on every PHB.
> The index of the arrays are segment number and the value stored
> in the corresponding element is PE number, indicating the segment
> is assigned to the PE. Initially, all elements in those two arrays
> are zeroes, meaning all segments are assigned to PE#0. It's wrong.
>
> This fixes the initial values in the elements of those two arrays
> to IODA_INVALID_PE, meaning all segments aren't assigned to any
> PE.
>
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>

Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>

> ---
>  arch/powerpc/platforms/powernv/pci-ioda.c | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
> index 4aa6cdf..59b20e5 100644
> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
> @@ -3240,6 +3240,7 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
>  	const __be64 *prop64;
>  	const __be32 *prop32;
>  	int len;
> +	unsigned int segno;
>  	u64 phb_id;
>  	void *aux;
>  	long rc;
> @@ -3334,8 +3335,13 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
>  	aux = memblock_virt_alloc(size, 0);
>  	phb->ioda.pe_alloc = aux;
>  	phb->ioda.m32_segmap = aux + m32map_off;
> -	if (phb->type == PNV_PHB_IODA1)
> +	for (segno = 0; segno < phb->ioda.total_pe_num; segno++)
> +		phb->ioda.m32_segmap[segno] = IODA_INVALID_PE;
> +	if (phb->type == PNV_PHB_IODA1) {
>  		phb->ioda.io_segmap = aux + iomap_off;
> +		for (segno = 0; segno < phb->ioda.total_pe_num; segno++)
> +			phb->ioda.io_segmap[segno] = IODA_INVALID_PE;
> +	}
>  	phb->ioda.pe_array = aux + pemap_off;
>  	set_bit(phb->ioda.reserved_pe_idx, phb->ioda.pe_alloc);
>
>


-- 
Alexey

Re: [PATCH v9 08/26] powerpc/powernv: Simplify pnv_ioda_setup_pe_seg()

[Alexey Kardashevskiy]

On 05/03/2016 03:41 PM, Gavin Shan wrote:
> pnv_ioda_setup_pe_seg() associates the IO and M32 segments with the
> owner PE. The code mapping segments should be fixed and immune from
> logic changes introduced to pnv_ioda_setup_pe_seg().
>
> This moves the code mapping segments to helper pnv_ioda_setup_pe_res().
> The data type for @rc is changed to "int64_t". Also, argument @hose is
> removed from pnv_ioda_setup_pe() as it can be got from @pe. No functional
> changes introduced.
>
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>

Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>

> ---
>  arch/powerpc/platforms/powernv/pci-ioda.c | 121 +++++++++++++++---------------
>  1 file changed, 62 insertions(+), 59 deletions(-)
>
> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
> index 59b20e5..b954fbc 100644
> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
> @@ -2929,19 +2929,72 @@ truncate_iov:
>  }
>  #endif /* CONFIG_PCI_IOV */
>
> +static void pnv_ioda_setup_pe_res(struct pnv_ioda_pe *pe,
> +				  struct resource *res)
> +{
> +	struct pnv_phb *phb = pe->phb;
> +	struct pci_bus_region region;
> +	int index;
> +	int64_t rc;
> +
> +	if (!res || !res->flags || res->start > res->end)
> +		return;
> +
> +	if (res->flags & IORESOURCE_IO) {
> +		region.start = res->start - phb->ioda.io_pci_base;
> +		region.end   = res->end - phb->ioda.io_pci_base;
> +		index = region.start / phb->ioda.io_segsize;
> +
> +		while (index < phb->ioda.total_pe_num &&
> +		       region.start <= region.end) {
> +			phb->ioda.io_segmap[index] = pe->pe_number;
> +			rc = opal_pci_map_pe_mmio_window(phb->opal_id,
> +				pe->pe_number, OPAL_IO_WINDOW_TYPE, 0, index);
> +			if (rc != OPAL_SUCCESS) {
> +				pr_err("%s: Error %lld mapping IO segment#%d to PE#%d\n",
> +				       __func__, rc, index, pe->pe_number);
> +				break;
> +			}
> +
> +			region.start += phb->ioda.io_segsize;
> +			index++;
> +		}
> +	} else if ((res->flags & IORESOURCE_MEM) &&
> +		   !pnv_pci_is_mem_pref_64(res->flags)) {
> +		region.start = res->start -
> +			       phb->hose->mem_offset[0] -
> +			       phb->ioda.m32_pci_base;
> +		region.end   = res->end -
> +			       phb->hose->mem_offset[0] -
> +			       phb->ioda.m32_pci_base;
> +		index = region.start / phb->ioda.m32_segsize;
> +
> +		while (index < phb->ioda.total_pe_num &&
> +		       region.start <= region.end) {
> +			phb->ioda.m32_segmap[index] = pe->pe_number;
> +			rc = opal_pci_map_pe_mmio_window(phb->opal_id,
> +				pe->pe_number, OPAL_M32_WINDOW_TYPE, 0, index);
> +			if (rc != OPAL_SUCCESS) {
> +				pr_err("%s: Error %lld mapping M32 segment#%d to PE#%d",
> +				       __func__, rc, index, pe->pe_number);
> +				break;
> +			}
> +
> +			region.start += phb->ioda.m32_segsize;
> +			index++;
> +		}
> +	}
> +}
> +
>  /*
>   * This function is supposed to be called on basis of PE from top
>   * to bottom style. So the the I/O or MMIO segment assigned to
>   * parent PE could be overrided by its child PEs if necessary.
>   */
> -static void pnv_ioda_setup_pe_seg(struct pci_controller *hose,
> -				  struct pnv_ioda_pe *pe)
> +static void pnv_ioda_setup_pe_seg(struct pnv_ioda_pe *pe)
>  {
> -	struct pnv_phb *phb = hose->private_data;
> -	struct pci_bus_region region;
>  	struct resource *res;
> -	int i, index;
> -	int rc;
> +	int i;
>
>  	/*
>  	 * NOTE: We only care PCI bus based PE for now. For PCI
> @@ -2950,58 +3003,8 @@ static void pnv_ioda_setup_pe_seg(struct pci_controller *hose,
>  	 */
>  	BUG_ON(!(pe->flags & (PNV_IODA_PE_BUS | PNV_IODA_PE_BUS_ALL)));
>
> -	pci_bus_for_each_resource(pe->pbus, res, i) {
> -		if (!res || !res->flags ||
> -		    res->start > res->end)
> -			continue;
> -
> -		if (res->flags & IORESOURCE_IO) {
> -			region.start = res->start - phb->ioda.io_pci_base;
> -			region.end   = res->end - phb->ioda.io_pci_base;
> -			index = region.start / phb->ioda.io_segsize;
> -
> -			while (index < phb->ioda.total_pe_num &&
> -			       region.start <= region.end) {
> -				phb->ioda.io_segmap[index] = pe->pe_number;
> -				rc = opal_pci_map_pe_mmio_window(phb->opal_id,
> -					pe->pe_number, OPAL_IO_WINDOW_TYPE, 0, index);
> -				if (rc != OPAL_SUCCESS) {
> -					pr_err("%s: OPAL error %d when mapping IO "
> -					       "segment #%d to PE#%d\n",
> -					       __func__, rc, index, pe->pe_number);
> -					break;
> -				}
> -
> -				region.start += phb->ioda.io_segsize;
> -				index++;
> -			}
> -		} else if ((res->flags & IORESOURCE_MEM) &&
> -			   !pnv_pci_is_mem_pref_64(res->flags)) {
> -			region.start = res->start -
> -				       hose->mem_offset[0] -
> -				       phb->ioda.m32_pci_base;
> -			region.end   = res->end -
> -				       hose->mem_offset[0] -
> -				       phb->ioda.m32_pci_base;
> -			index = region.start / phb->ioda.m32_segsize;
> -
> -			while (index < phb->ioda.total_pe_num &&
> -			       region.start <= region.end) {
> -				phb->ioda.m32_segmap[index] = pe->pe_number;
> -				rc = opal_pci_map_pe_mmio_window(phb->opal_id,
> -					pe->pe_number, OPAL_M32_WINDOW_TYPE, 0, index);
> -				if (rc != OPAL_SUCCESS) {
> -					pr_err("%s: OPAL error %d when mapping M32 "
> -					       "segment#%d to PE#%d",
> -					       __func__, rc, index, pe->pe_number);
> -					break;
> -				}
> -
> -				region.start += phb->ioda.m32_segsize;
> -				index++;
> -			}
> -		}
> -	}
> +	pci_bus_for_each_resource(pe->pbus, res, i)
> +		pnv_ioda_setup_pe_res(pe, res);
>  }
>
>  static void pnv_pci_ioda_setup_seg(void)
> @@ -3018,7 +3021,7 @@ static void pnv_pci_ioda_setup_seg(void)
>  			continue;
>
>  		list_for_each_entry(pe, &phb->ioda.pe_list, list) {
> -			pnv_ioda_setup_pe_seg(hose, pe);
> +			pnv_ioda_setup_pe_seg(pe);
>  		}
>  	}
>  }
>


-- 
Alexey

Re: [PATCH v9 12/26] powerpc/powernv/ioda1: M64 support on P7IOC

[Gavin Shan]

On Thu, May 05, 2016 at 11:03:28AM +1000, Alistair Popple wrote:
>Thanks for the clarifications Gavin. Aside from the WARN_ON() (which is not a
>major thing) everything looks good.
>
>Reviewed-By: Alistair Popple <alistair@popple.id.au>
>

Thanks, Alistair. I'm going to send a updated revision (v10) replacing pr_warn()
with WARN(). The detailed message is as below:

	WARN(1, "Wrong reserved PE#%d on PHB#%d\n",
	     phb->ioda.reserved_pe_idx, phb->hose->global_number);


Thanks,
Gavin

>On Thu, 5 May 2016 10:40:33 Gavin Shan wrote:
>> On Thu, May 05, 2016 at 09:53:51AM +1000, Alistair Popple wrote:
>> >On Wed, 4 May 2016 16:48:53 Gavin Shan wrote:
>> >> On Wed, May 04, 2016 at 03:17:51PM +1000, Alistair Popple wrote:
>> >> >On Tue, 3 May 2016 15:41:31 Gavin Shan wrote:
>> >> >> This enables M64 window on P7IOC, which has been enabled on PHB3.
>> >> >
>> >> >Have we tested that this works with an adaptor? This looks to be enabling 
>> >> >support for something that didn't previously work (64-bit BARs on P7IOC)?
>> >> >
>> >> 
>> >> The M64 isn't supported on P7IOC before, meaning the PCI device's M64
>> >> BAR is covered by PHB's M32 window. With the patch applied, the PCI
>> >> device's M64 BAR is covered by PHB's M64 window as below log I got
>> >> from vpl4. The kernel including the series of patches boots successfully
>> >> on vpl4 and all looks normal.
>> >
>> >So you're changing the way 64-bit BARs work on P7IOC to use a different bit of
>> >the hardware/firmware? This means it could break older systems if there is
>> >an issue with the M64 support. Other than the comments below the code looks
>> >reasonable. However I am assuming you've tested that cards with 64-bit BARs
>> >still function on P7 after this change as I'm not familiar enough to comment on
>> >the specific hardware details, although your comments make sense and match the
>> >code.
>> >
>> 
>> Nope, it doesn't rely on changes in firmware. The old firmware enables and
>> exposes PHB M64 window. The kernel doesn't use it on P7IOC until this patch.
>> On VPL4, there is a IPR adapter (0000:60:00.0) as below kernel log indicates.
>> Its BAR#2 is M64 (64-bits prefetchable) BAR. The BAR#2 is covered by PHB's M64
>> window with this patch. The PHB's M64 window is [0x00003da800000000..0x00003dafffffffff].
>> After it's applied, the IPR and root filesystem resident in the disk drive out
>> of it work fine.
>> 
>> >> [    0.246110] pci 0000:60:00.0: BAR 2: assigned [mem 0x3da810000000-0x3da810ffffff 64bit pref]
>> >> [    0.246218] pci 0000:60:00.0: BAR 0: assigned [mem 0x3da081000000-0x3da08103ffff 64bit]
>> >> [    0.246306] pci 0000:60:00.0: BAR 6: assigned [mem 0x3da081040000-0x3da08105ffff pref]
>> >> [    0.246392] pci 0000:60     : [PE# 001] Secondary bus 96 associated with PE#1
>> >> [    0.246484] pci 0000:60     : [PE# 001] DMA weight 15 (64), assigned (0) 1 DMA32 segments
>> >> [    0.246552] pci 0000:60     : [PE# 001]  Setting up 32-bit TCE table at 00000000..0fffffff
>> >> 
>> >> >> Different from PHB3 where 16 M64 BARs are supported and each of
>> >> >> them can be owned by one particular PE# exclusively or divided
>> >> >> evenly to 256 segments, every P7IOC PHB has 16 M64 BARs and each
>> >> >> of them are divided to 8 segments. So every P7IOC PHB supports
>> >> >> 128 M64 segments in total. P7IOC has M64DT, which helps mapping
>> >> >> one particular M64 segment# to arbitrary PE#. PHB3 doesn't have
>> >> >> M64DT, indicating that one M64 segment can only be pinned to the
>> >> >> fixed PE#.
>> >> >> 
>> >> >> In order to unified M64 support M64 on P7IOC and PHB3, we just
>> >> >> provide 128 M64 segments on every P7IOC PHB and each of them is
>> >> >> pinned to the fixed PE# by bypassing the function of M64DT. In
>> >> >> turn, we just need different phb->init_m64() for P7IOC and PHB3
>> >> >> and maps M64 segment in pnv_ioda_reserve_m64_pe() for P7IOC, most
>> >> >> of the code are shared by them.
>> >> >> 
>> >> >> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>> >> >> ---
>> >> >>  arch/powerpc/platforms/powernv/pci-ioda.c | 89 
>> >> >+++++++++++++++++++++++++++++--
>> >> >>  1 file changed, 86 insertions(+), 3 deletions(-)
>> >> >> 
>> >> >> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c 
>> >> >b/arch/powerpc/platforms/powernv/pci-ioda.c
>> >> >> index 37f22b0..a1b74ec 100644
>> >> >> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
>> >> >> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
>> >> >> @@ -48,6 +48,9 @@
>> >> >>  #include "powernv.h"
>> >> >>  #include "pci.h"
>> >> >>  
>> >> >> +#define PNV_IODA1_M64_NUM	16	/* Number of M64 BARs	*/
>> >> >> +#define PNV_IODA1_M64_SEGS	8	/* Segments per M64 BAR	*/
>> >> >> +
>> >> >>  /* 256M DMA window, 4K TCE pages, 8 bytes TCE */
>> >> >>  #define TCE32_TABLE_SIZE	((0x10000000 / 0x1000) * 8)
>> >> >>  
>> >> >> @@ -246,6 +249,64 @@ static void pnv_ioda_reserve_dev_m64_pe(struct pci_dev 
>> >> >*pdev,
>> >> >>  	}
>> >> >>  }
>> >> >>  
>> >> >> +static int pnv_ioda1_init_m64(struct pnv_phb *phb)
>> >> >> +{
>> >> >> +	struct resource *r;
>> >> >> +	int index;
>> >> >> +
>> >> >> +	/*
>> >> >> +	 * There are 16 M64 BARs, each of which has 8 segments. So
>> >> >> +	 * there are as many M64 segments as the maximum number of
>> >> >> +	 * PEs, which is 128.
>> >> >> +	 */
>> >> >> +	for (index = 0; index < PNV_IODA1_M64_NUM; index++) {
>> >> >> +		unsigned long base, segsz = phb->ioda.m64_segsize;
>> >> >> +		int64_t rc;
>> >> >> +
>> >> >> +		base = phb->ioda.m64_base +
>> >> >> +		       index * PNV_IODA1_M64_SEGS * segsz;
>> >> >> +		rc = opal_pci_set_phb_mem_window(phb->opal_id,
>> >> >> +				OPAL_M64_WINDOW_TYPE, index, base, 0,
>> >> >> +				PNV_IODA1_M64_SEGS * segsz);
>> >
>> >Has firmware always supported OPAL_M64_WINDOW_TYPE for P7IOC? If older versions
>> >don't support it what happens? Do we gracefully fall back to the old mode of
>> >allocating all BARs from the M32 window or does it break?
>> >
>> 
>> Old firmware enables and exposes M64 window (OPAL_M64_WINDOW_TYPE), but kernel
>> doesn't use it until this patch. If the OPAL call fails, errcode is returned.
>> The caller, pnv_pci_init_ioda_phb(), disables the window. It means the PCI
>> device M64 BARs will fail back to be covered by M32 window. The path is same
>> on PHB3 and P7IOC.
>> 
>> >> >> +		if (rc != OPAL_SUCCESS) {
>> >> >> +			pr_warn("  Error %lld setting M64 PHB#%d-BAR#%d\n",
>> >> >> +				rc, phb->hose->global_number, index);
>> >> >> +			goto fail;
>> >> >> +		}
>> >> >> +
>> >> >> +		rc = opal_pci_phb_mmio_enable(phb->opal_id,
>> >> >> +				OPAL_M64_WINDOW_TYPE, index,
>> >> >> +				OPAL_ENABLE_M64_SPLIT);
>> >> >> +		if (rc != OPAL_SUCCESS) {
>> >> >> +			pr_warn("  Error %lld enabling M64 PHB#%d-BAR#%d\n",
>> >> >> +				rc, phb->hose->global_number, index);
>> >> >> +			goto fail;
>> >> >> +		}
>> >> >> +	}
>> >> >> +
>> >> >> +	/*
>> >> >> +	 * Exclude the segment used by the reserved PE, which
>> >> >> +	 * is expected to be 0 or last supported PE#.
>> >> >> +	 */
>> >> >> +	r = &phb->hose->mem_resources[1];
>> >> >> +	if (phb->ioda.reserved_pe_idx == 0)
>> >> >> +		r->start += phb->ioda.m64_segsize;
>> >> >> +	else if (phb->ioda.reserved_pe_idx == (phb->ioda.total_pe_num - 1))
>> >> >> +		r->end -= phb->ioda.m64_segsize;
>> >> >> +	else
>> >> >> +		pr_warn("  Cannot cut M64 segment for reserved PE#%d\n",
>> >> >> +			phb->ioda.reserved_pe_idx);
>> >
>> >Should this be a WARN_ON()? If this condition can only exist because a future
>> >programmer changes the reserved_pe_idx then I think it should be a WARN_ON()
>> >as the above message would be too easy to miss.
>> >
>> 
>> Yeah, a WARN_ON() should be better.
>> 
>> Thanks,
>> Gavin
>> 
>> >> >> +
>> >> >> +	return 0;
>> >> >> +
>> >> >> +fail:
>> >> >> +	for ( ; index >= 0; index--)
>> >> >> +		opal_pci_phb_mmio_enable(phb->opal_id,
>> >> >> +			OPAL_M64_WINDOW_TYPE, index, OPAL_DISABLE_M64);
>> >> >> +
>> >> >> +	return -EIO;
>> >> >> +}
>> >> >> +
>> >> >>  static void pnv_ioda_reserve_m64_pe(struct pci_bus *bus,
>> >> >>  				    unsigned long *pe_bitmap,
>> >> >>  				    bool all)
>> >> >> @@ -315,6 +376,26 @@ static unsigned int pnv_ioda_pick_m64_pe(struct pci_bus 
>> >> >*bus, bool all)
>> >> >>  			pe->master = master_pe;
>> >> >>  			list_add_tail(&pe->list, &master_pe->slaves);
>> >> >>  		}
>> >> >> +
>> >> >> +		/*
>> >> >> +		 * P7IOC supports M64DT, which helps mapping M64 segment
>> >> >> +		 * to one particular PE#. However, PHB3 has fixed mapping
>> >> >> +		 * between M64 segment and PE#. In order to have same logic
>> >> >> +		 * for P7IOC and PHB3, we enforce fixed mapping between M64
>> >> >> +		 * segment and PE# on P7IOC.
>> >> >> +		 */
>> >> >> +		if (phb->type == PNV_PHB_IODA1) {
>> >> >> +			int64_t rc;
>> >> >> +
>> >> >> +			rc = opal_pci_map_pe_mmio_window(phb->opal_id,
>> >> >> +					pe->pe_number, OPAL_M64_WINDOW_TYPE,
>> >> >> +					pe->pe_number / PNV_IODA1_M64_SEGS,
>> >> >> +					pe->pe_number % PNV_IODA1_M64_SEGS);
>> >> >> +			if (rc != OPAL_SUCCESS)
>> >> >> +				pr_warn("%s: Error %lld mapping M64 for 
>> >> >PHB#%d-PE#%d\n",
>> >> >> +					__func__, rc, phb->hose-
>> >> >>global_number,
>> >> >> +					pe->pe_number);
>> >> >> +		}
>> >> >>  	}
>> >> >>  
>> >> >>  	kfree(pe_alloc);
>> >> >> @@ -329,8 +410,7 @@ static void __init pnv_ioda_parse_m64_window(struct 
>> >> >pnv_phb *phb)
>> >> >>  	const u32 *r;
>> >> >>  	u64 pci_addr;
>> >> >>  
>> >> >> -	/* FIXME: Support M64 for P7IOC */
>> >> >> -	if (phb->type != PNV_PHB_IODA2) {
>> >> >> +	if (phb->type != PNV_PHB_IODA1 && phb->type != PNV_PHB_IODA2) {
>> >> >>  		pr_info("  Not support M64 window\n");
>> >> >>  		return;
>> >> >>  	}
>> >> >> @@ -364,7 +444,10 @@ static void __init pnv_ioda_parse_m64_window(struct 
>> >> >pnv_phb *phb)
>> >> >>  
>> >> >>  	/* Use last M64 BAR to cover M64 window */
>> >> >>  	phb->ioda.m64_bar_idx = 15;
>> >> >> -	phb->init_m64 = pnv_ioda2_init_m64;
>> >> >> +	if (phb->type == PNV_PHB_IODA1)
>> >> >> +		phb->init_m64 = pnv_ioda1_init_m64;
>> >> >> +	else
>> >> >> +		phb->init_m64 = pnv_ioda2_init_m64;
>> >> >>  	phb->reserve_m64_pe = pnv_ioda_reserve_m64_pe;
>> >> >>  	phb->pick_m64_pe = pnv_ioda_pick_m64_pe;
>> >> >>  }
>> >> >> 
>> >> >
>> >> 
>> >
>> 
>

Re: [PATCH v10 12/26] powerpc/powernv/ioda1: M64 support on P7IOC

[Alexey Kardashevskiy]

On 05/05/2016 12:02 PM, Gavin Shan wrote:
> This enables M64 window on P7IOC, which has been enabled on PHB3.
> Different from PHB3 where 16 M64 BARs are supported and each of
> them can be owned by one particular PE# exclusively or divided
> evenly to 256 segments, every P7IOC PHB has 16 M64 BARs and each
> of them are divided to 8 segments. So every P7IOC PHB supports
> 128 M64 segments in total. P7IOC has M64DT, which helps mapping
> one particular M64 segment# to arbitrary PE#. PHB3 doesn't have
> M64DT, indicating that one M64 segment can only be pinned to the
> fixed PE#.
>
> In order to unified M64 support M64 on P7IOC and PHB3, we just
> provide 128 M64 segments on every P7IOC PHB and each of them is
> pinned to the fixed PE# by bypassing the function of M64DT. In
> turn, we just need different phb->init_m64() for P7IOC and PHB3
> and maps M64 segment in pnv_ioda_reserve_m64_pe() for P7IOC, most
> of the code are shared by them.
>
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> Reviewed-by: Alistair Popple <alistair@popple.id.au>

Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>

> ---
> v10: Replace pr_warn() with WARN() suggested by Alistair
> ---
>  arch/powerpc/platforms/powernv/pci-ioda.c | 89 +++++++++++++++++++++++++++++--
>  1 file changed, 86 insertions(+), 3 deletions(-)
>
> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
> index 37f22b0..30a6d79 100644
> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
> @@ -48,6 +48,9 @@
>  #include "powernv.h"
>  #include "pci.h"
>
> +#define PNV_IODA1_M64_NUM	16	/* Number of M64 BARs	*/
> +#define PNV_IODA1_M64_SEGS	8	/* Segments per M64 BAR	*/
> +
>  /* 256M DMA window, 4K TCE pages, 8 bytes TCE */
>  #define TCE32_TABLE_SIZE	((0x10000000 / 0x1000) * 8)
>
> @@ -246,6 +249,64 @@ static void pnv_ioda_reserve_dev_m64_pe(struct pci_dev *pdev,
>  	}
>  }
>
> +static int pnv_ioda1_init_m64(struct pnv_phb *phb)
> +{
> +	struct resource *r;
> +	int index;
> +
> +	/*
> +	 * There are 16 M64 BARs, each of which has 8 segments. So
> +	 * there are as many M64 segments as the maximum number of
> +	 * PEs, which is 128.
> +	 */
> +	for (index = 0; index < PNV_IODA1_M64_NUM; index++) {
> +		unsigned long base, segsz = phb->ioda.m64_segsize;
> +		int64_t rc;
> +
> +		base = phb->ioda.m64_base +
> +		       index * PNV_IODA1_M64_SEGS * segsz;
> +		rc = opal_pci_set_phb_mem_window(phb->opal_id,
> +				OPAL_M64_WINDOW_TYPE, index, base, 0,
> +				PNV_IODA1_M64_SEGS * segsz);
> +		if (rc != OPAL_SUCCESS) {
> +			pr_warn("  Error %lld setting M64 PHB#%d-BAR#%d\n",
> +				rc, phb->hose->global_number, index);
> +			goto fail;
> +		}
> +
> +		rc = opal_pci_phb_mmio_enable(phb->opal_id,
> +				OPAL_M64_WINDOW_TYPE, index,
> +				OPAL_ENABLE_M64_SPLIT);
> +		if (rc != OPAL_SUCCESS) {
> +			pr_warn("  Error %lld enabling M64 PHB#%d-BAR#%d\n",
> +				rc, phb->hose->global_number, index);
> +			goto fail;
> +		}
> +	}
> +
> +	/*
> +	 * Exclude the segment used by the reserved PE, which
> +	 * is expected to be 0 or last supported PE#.
> +	 */
> +	r = &phb->hose->mem_resources[1];
> +	if (phb->ioda.reserved_pe_idx == 0)
> +		r->start += phb->ioda.m64_segsize;
> +	else if (phb->ioda.reserved_pe_idx == (phb->ioda.total_pe_num - 1))
> +		r->end -= phb->ioda.m64_segsize;
> +	else
> +		WARN(1, "Wrong reserved PE#%d on PHB#%d\n",
> +		     phb->ioda.reserved_pe_idx, phb->hose->global_number);
> +
> +	return 0;
> +
> +fail:
> +	for ( ; index >= 0; index--)
> +		opal_pci_phb_mmio_enable(phb->opal_id,
> +			OPAL_M64_WINDOW_TYPE, index, OPAL_DISABLE_M64);
> +
> +	return -EIO;
> +}
> +
>  static void pnv_ioda_reserve_m64_pe(struct pci_bus *bus,
>  				    unsigned long *pe_bitmap,
>  				    bool all)
> @@ -315,6 +376,26 @@ static unsigned int pnv_ioda_pick_m64_pe(struct pci_bus *bus, bool all)
>  			pe->master = master_pe;
>  			list_add_tail(&pe->list, &master_pe->slaves);
>  		}
> +
> +		/*
> +		 * P7IOC supports M64DT, which helps mapping M64 segment
> +		 * to one particular PE#. However, PHB3 has fixed mapping
> +		 * between M64 segment and PE#. In order to have same logic
> +		 * for P7IOC and PHB3, we enforce fixed mapping between M64
> +		 * segment and PE# on P7IOC.
> +		 */
> +		if (phb->type == PNV_PHB_IODA1) {
> +			int64_t rc;
> +
> +			rc = opal_pci_map_pe_mmio_window(phb->opal_id,
> +					pe->pe_number, OPAL_M64_WINDOW_TYPE,
> +					pe->pe_number / PNV_IODA1_M64_SEGS,
> +					pe->pe_number % PNV_IODA1_M64_SEGS);
> +			if (rc != OPAL_SUCCESS)
> +				pr_warn("%s: Error %lld mapping M64 for PHB#%d-PE#%d\n",
> +					__func__, rc, phb->hose->global_number,
> +					pe->pe_number);
> +		}
>  	}
>
>  	kfree(pe_alloc);
> @@ -329,8 +410,7 @@ static void __init pnv_ioda_parse_m64_window(struct pnv_phb *phb)
>  	const u32 *r;
>  	u64 pci_addr;
>
> -	/* FIXME: Support M64 for P7IOC */
> -	if (phb->type != PNV_PHB_IODA2) {
> +	if (phb->type != PNV_PHB_IODA1 && phb->type != PNV_PHB_IODA2) {
>  		pr_info("  Not support M64 window\n");
>  		return;
>  	}
> @@ -364,7 +444,10 @@ static void __init pnv_ioda_parse_m64_window(struct pnv_phb *phb)
>
>  	/* Use last M64 BAR to cover M64 window */
>  	phb->ioda.m64_bar_idx = 15;
> -	phb->init_m64 = pnv_ioda2_init_m64;
> +	if (phb->type == PNV_PHB_IODA1)
> +		phb->init_m64 = pnv_ioda1_init_m64;
> +	else
> +		phb->init_m64 = pnv_ioda2_init_m64;
>  	phb->reserve_m64_pe = pnv_ioda_reserve_m64_pe;
>  	phb->pick_m64_pe = pnv_ioda_pick_m64_pe;
>  }
>


-- 
Alexey

Re: [PATCH v9 14/26] powerpc/powernv/ioda1: Introduce PNV_IODA1_DMA32_SEGSIZE

[Alexey Kardashevskiy]

On 05/03/2016 03:41 PM, Gavin Shan wrote:
> Currently, there is one macro (TCE32_TABLE_SIZE) representing the
> TCE table size for one DMA32 segment. The constant representing
> the DMA32 segment size (1 << 28) is still used in the code.
>
> This defines PNV_IODA1_DMA32_SEGSIZE representing one DMA32
> segment size. the TCE table size can be calcualted when the page
> has fixed 4KB size. So all the related calculation depends on one
> macro (PNV_IODA1_DMA32_SEGSIZE). No logical changes introduced.
>
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>

Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>

> ---
>  arch/powerpc/platforms/powernv/pci-ioda.c | 31 ++++++++++++++++++-------------
>  1 file changed, 18 insertions(+), 13 deletions(-)
>
> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
> index f27a62d..ca034b8 100644
> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
> @@ -50,9 +50,7 @@
>
>  #define PNV_IODA1_M64_NUM	16	/* Number of M64 BARs	*/
>  #define PNV_IODA1_M64_SEGS	8	/* Segments per M64 BAR	*/
> -
> -/* 256M DMA window, 4K TCE pages, 8 bytes TCE */
> -#define TCE32_TABLE_SIZE	((0x10000000 / 0x1000) * 8)
> +#define PNV_IODA1_DMA32_SEGSIZE	0x10000000
>
>  #define POWERNV_IOMMU_DEFAULT_LEVELS	1
>  #define POWERNV_IOMMU_MAX_LEVELS	5
> @@ -2037,7 +2035,7 @@ static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
>
>  	struct page *tce_mem = NULL;
>  	struct iommu_table *tbl;
> -	unsigned int i;
> +	unsigned int tce32_segsz, i;
>  	int64_t rc;
>  	void *addr;
>
> @@ -2057,29 +2055,34 @@ static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
>  	/* Grab a 32-bit TCE table */
>  	pe->tce32_seg = base;
>  	pe_info(pe, " Setting up 32-bit TCE table at %08x..%08x\n",
> -		(base << 28), ((base + segs) << 28) - 1);
> +		base * PNV_IODA1_DMA32_SEGSIZE,
> +		(base + segs) * PNV_IODA1_DMA32_SEGSIZE - 1);
>
>  	/* XXX Currently, we allocate one big contiguous table for the
>  	 * TCEs. We only really need one chunk per 256M of TCE space
>  	 * (ie per segment) but that's an optimization for later, it
>  	 * requires some added smarts with our get/put_tce implementation
> +	 *
> +	 * Each TCE page is 4KB in size and each TCE entry occupies 8
> +	 * bytes
>  	 */
> +	tce32_segsz = PNV_IODA1_DMA32_SEGSIZE >> (IOMMU_PAGE_SHIFT_4K - 3);
>  	tce_mem = alloc_pages_node(phb->hose->node, GFP_KERNEL,
> -				   get_order(TCE32_TABLE_SIZE * segs));
> +				   get_order(tce32_segsz * segs));
>  	if (!tce_mem) {
>  		pe_err(pe, " Failed to allocate a 32-bit TCE memory\n");
>  		goto fail;
>  	}
>  	addr = page_address(tce_mem);
> -	memset(addr, 0, TCE32_TABLE_SIZE * segs);
> +	memset(addr, 0, tce32_segsz * segs);
>
>  	/* Configure HW */
>  	for (i = 0; i < segs; i++) {
>  		rc = opal_pci_map_pe_dma_window(phb->opal_id,
>  					      pe->pe_number,
>  					      base + i, 1,
> -					      __pa(addr) + TCE32_TABLE_SIZE * i,
> -					      TCE32_TABLE_SIZE, 0x1000);
> +					      __pa(addr) + tce32_segsz * i,
> +					      tce32_segsz, IOMMU_PAGE_SIZE_4K);
>  		if (rc) {
>  			pe_err(pe, " Failed to configure 32-bit TCE table,"
>  			       " err %ld\n", rc);
> @@ -2088,8 +2091,9 @@ static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
>  	}
>
>  	/* Setup linux iommu table */
> -	pnv_pci_setup_iommu_table(tbl, addr, TCE32_TABLE_SIZE * segs,
> -				  base << 28, IOMMU_PAGE_SHIFT_4K);
> +	pnv_pci_setup_iommu_table(tbl, addr, tce32_segsz * segs,
> +				  base * PNV_IODA1_DMA32_SEGSIZE,
> +				  IOMMU_PAGE_SHIFT_4K);
>
>  	/* OPAL variant of P7IOC SW invalidated TCEs */
>  	if (phb->ioda.tce_inval_reg)
> @@ -2119,7 +2123,7 @@ static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
>  	if (pe->tce32_seg >= 0)
>  		pe->tce32_seg = -1;
>  	if (tce_mem)
> -		__free_pages(tce_mem, get_order(TCE32_TABLE_SIZE * segs));
> +		__free_pages(tce_mem, get_order(tce32_segsz * segs));
>  	if (tbl) {
>  		pnv_pci_unlink_table_and_group(tbl, &pe->table_group);
>  		iommu_free_table(tbl, "pnv");
> @@ -3456,7 +3460,8 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
>  	mutex_init(&phb->ioda.pe_list_mutex);
>
>  	/* Calculate how many 32-bit TCE segments we have */
> -	phb->ioda.tce32_count = phb->ioda.m32_pci_base >> 28;
> +	phb->ioda.tce32_count = phb->ioda.m32_pci_base /
> +				PNV_IODA1_DMA32_SEGSIZE;
>
>  #if 0 /* We should really do that ... */
>  	rc = opal_pci_set_phb_mem_window(opal->phb_id,
>


-- 
Alexey

Re: [PATCH v9 09/26] powerpc/powernv: IO and M32 mapping based on PCI device resources

[Alexey Kardashevskiy]

On 05/03/2016 03:41 PM, Gavin Shan wrote:
> Currently, the IO and M32 segments are mapped to the corresponding
> PE based on the windows of the parent bridge of PE's primary bus.
> It's not going to work when the windows of root port or upstream
> port of the PCIe switch behind root port are extended to PHB's
> apertures in order to support hotplug in subsequent patch.
>
> This fixes the issue by mapping IO and M32 segments based on the
> resources of the PCI devices included in the PE, instead of the
> windows of the parent bridge of the PE's primary bus.
>
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>

Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>

> ---
>  arch/powerpc/platforms/powernv/pci-ioda.c | 19 ++++++++++++++++---
>  1 file changed, 16 insertions(+), 3 deletions(-)
>
> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
> index b954fbc..904790b 100644
> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
> @@ -2993,7 +2993,7 @@ static void pnv_ioda_setup_pe_res(struct pnv_ioda_pe *pe,
>   */
>  static void pnv_ioda_setup_pe_seg(struct pnv_ioda_pe *pe)
>  {
> -	struct resource *res;
> +	struct pci_dev *pdev;
>  	int i;
>
>  	/*
> @@ -3003,8 +3003,21 @@ static void pnv_ioda_setup_pe_seg(struct pnv_ioda_pe *pe)
>  	 */
>  	BUG_ON(!(pe->flags & (PNV_IODA_PE_BUS | PNV_IODA_PE_BUS_ALL)));
>
> -	pci_bus_for_each_resource(pe->pbus, res, i)
> -		pnv_ioda_setup_pe_res(pe, res);
> +	list_for_each_entry(pdev, &pe->pbus->devices, bus_list) {
> +		for (i = 0; i <= PCI_ROM_RESOURCE; i++)
> +			pnv_ioda_setup_pe_res(pe, &pdev->resource[i]);
> +
> +		/*
> +		 * If the PE contains all subordinate PCI buses, the
> +		 * windows of the child bridges should be mapped to
> +		 * the PE as well.
> +		 */
> +		if (!(pe->flags & PNV_IODA_PE_BUS_ALL) || !pci_is_bridge(pdev))
> +			continue;
> +		for (i = 0; i < PCI_BRIDGE_RESOURCE_NUM; i++)
> +			pnv_ioda_setup_pe_res(pe,
> +				&pdev->resource[PCI_BRIDGE_RESOURCES + i]);
> +	}
>  }
>
>  static void pnv_pci_ioda_setup_seg(void)
>


-- 
Alexey

Re: [PATCH v9 18/26] powerpc/pci: Rename pcibios_{add,remove}_pci_devices()

[Alexey Kardashevskiy]

On 05/03/2016 03:41 PM, Gavin Shan wrote:
> This renames pcibios_{add,remove}_pci_devices() to avoid conflicts
> with names of the weak functions in PCI subsystem, which have the
> prefix "pcibios". No logical changes introduced.
>
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>

Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>

> ---
>  arch/powerpc/include/asm/pci-bridge.h |  4 ++--
>  arch/powerpc/kernel/eeh_driver.c      | 12 ++++++------
>  arch/powerpc/kernel/pci-hotplug.c     | 15 +++++++--------
>  drivers/pci/hotplug/rpadlpar_core.c   |  2 +-
>  drivers/pci/hotplug/rpaphp_core.c     |  4 ++--
>  drivers/pci/hotplug/rpaphp_pci.c      |  2 +-
>  6 files changed, 19 insertions(+), 20 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/pci-bridge.h b/arch/powerpc/include/asm/pci-bridge.h
> index 220129f..99027b8 100644
> --- a/arch/powerpc/include/asm/pci-bridge.h
> +++ b/arch/powerpc/include/asm/pci-bridge.h
> @@ -260,10 +260,10 @@ static inline struct eeh_dev *pdn_to_eeh_dev(struct pci_dn *pdn)
>  extern struct pci_bus *pcibios_find_pci_bus(struct device_node *dn);
>
>  /** Remove all of the PCI devices under this bus */
> -extern void pcibios_remove_pci_devices(struct pci_bus *bus);
> +extern void pci_hp_remove_devices(struct pci_bus *bus);
>
>  /** Discover new pci devices under this bus, and add them */
> -extern void pcibios_add_pci_devices(struct pci_bus *bus);
> +extern void pci_hp_add_devices(struct pci_bus *bus);
>
>
>  extern void isa_bridge_find_early(struct pci_controller *hose);
> diff --git a/arch/powerpc/kernel/eeh_driver.c b/arch/powerpc/kernel/eeh_driver.c
> index fb6207d..618d13c 100644
> --- a/arch/powerpc/kernel/eeh_driver.c
> +++ b/arch/powerpc/kernel/eeh_driver.c
> @@ -621,7 +621,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus,
>  	 * We don't remove the corresponding PE instances because
>  	 * we need the information afterwords. The attached EEH
>  	 * devices are expected to be attached soon when calling
> -	 * into pcibios_add_pci_devices().
> +	 * into pci_hp_add_devices().
>  	 */
>  	eeh_pe_state_mark(pe, EEH_PE_KEEP);
>  	if (bus) {
> @@ -630,7 +630,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus,
>  		} else {
>  			eeh_pe_state_clear(pe, EEH_PE_PRI_BUS);
>  			pci_lock_rescan_remove();
> -			pcibios_remove_pci_devices(bus);
> +			pci_hp_remove_devices(bus);
>  			pci_unlock_rescan_remove();
>  		}
>  	} else if (frozen_bus) {
> @@ -681,7 +681,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus,
>  		if (pe->type & EEH_PE_VF)
>  			eeh_add_virt_device(edev, NULL);
>  		else
> -			pcibios_add_pci_devices(bus);
> +			pci_hp_add_devices(bus);
>  	} else if (frozen_bus && rmv_data->removed) {
>  		pr_info("EEH: Sleep 5s ahead of partial hotplug\n");
>  		ssleep(5);
> @@ -691,7 +691,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus,
>  		if (pe->type & EEH_PE_VF)
>  			eeh_add_virt_device(edev, NULL);
>  		else
> -			pcibios_add_pci_devices(frozen_bus);
> +			pci_hp_add_devices(frozen_bus);
>  	}
>  	eeh_pe_state_clear(pe, EEH_PE_KEEP);
>
> @@ -896,7 +896,7 @@ perm_error:
>  			eeh_pe_dev_mode_mark(pe, EEH_DEV_REMOVED);
>
>  			pci_lock_rescan_remove();
> -			pcibios_remove_pci_devices(frozen_bus);
> +			pci_hp_remove_devices(frozen_bus);
>  			pci_unlock_rescan_remove();
>  		}
>  	}
> @@ -981,7 +981,7 @@ static void eeh_handle_special_event(void)
>  				bus = eeh_pe_bus_get(phb_pe);
>  				eeh_pe_dev_traverse(pe,
>  					eeh_report_failure, NULL);
> -				pcibios_remove_pci_devices(bus);
> +				pci_hp_remove_devices(bus);
>  			}
>  			pci_unlock_rescan_remove();
>  		}
> diff --git a/arch/powerpc/kernel/pci-hotplug.c b/arch/powerpc/kernel/pci-hotplug.c
> index 59c4361..2d108e5 100644
> --- a/arch/powerpc/kernel/pci-hotplug.c
> +++ b/arch/powerpc/kernel/pci-hotplug.c
> @@ -38,20 +38,20 @@ void pcibios_release_device(struct pci_dev *dev)
>  }
>
>  /**
> - * pcibios_remove_pci_devices - remove all devices under this bus
> + * pci_hp_remove_devices - remove all devices under this bus
>   * @bus: the indicated PCI bus
>   *
>   * Remove all of the PCI devices under this bus both from the
>   * linux pci device tree, and from the powerpc EEH address cache.
>   */
> -void pcibios_remove_pci_devices(struct pci_bus *bus)
> +void pci_hp_remove_devices(struct pci_bus *bus)
>  {
>  	struct pci_dev *dev, *tmp;
>  	struct pci_bus *child_bus;
>
>  	/* First go down child busses */
>  	list_for_each_entry(child_bus, &bus->children, node)
> -		pcibios_remove_pci_devices(child_bus);
> +		pci_hp_remove_devices(child_bus);
>
>  	pr_debug("PCI: Removing devices on bus %04x:%02x\n",
>  		 pci_domain_nr(bus),  bus->number);
> @@ -60,11 +60,10 @@ void pcibios_remove_pci_devices(struct pci_bus *bus)
>  		pci_stop_and_remove_bus_device(dev);
>  	}
>  }
> -
> -EXPORT_SYMBOL_GPL(pcibios_remove_pci_devices);
> +EXPORT_SYMBOL_GPL(pci_hp_remove_devices);
>
>  /**
> - * pcibios_add_pci_devices - adds new pci devices to bus
> + * pci_hp_add_devices - adds new pci devices to bus
>   * @bus: the indicated PCI bus
>   *
>   * This routine will find and fixup new pci devices under
> @@ -74,7 +73,7 @@ EXPORT_SYMBOL_GPL(pcibios_remove_pci_devices);
>   * is how this routine differs from other, similar pcibios
>   * routines.)
>   */
> -void pcibios_add_pci_devices(struct pci_bus * bus)
> +void pci_hp_add_devices(struct pci_bus *bus)
>  {
>  	int slotno, mode, pass, max;
>  	struct pci_dev *dev;
> @@ -114,4 +113,4 @@ void pcibios_add_pci_devices(struct pci_bus * bus)
>  	}
>  	pcibios_finish_adding_to_bus(bus);
>  }
> -EXPORT_SYMBOL_GPL(pcibios_add_pci_devices);
> +EXPORT_SYMBOL_GPL(pci_hp_add_devices);
> diff --git a/drivers/pci/hotplug/rpadlpar_core.c b/drivers/pci/hotplug/rpadlpar_core.c
> index b46b57d..b770d6d 100644
> --- a/drivers/pci/hotplug/rpadlpar_core.c
> +++ b/drivers/pci/hotplug/rpadlpar_core.c
> @@ -380,7 +380,7 @@ int dlpar_remove_pci_slot(char *drc_name, struct device_node *dn)
>  	}
>
>  	/* Remove all devices below slot */
> -	pcibios_remove_pci_devices(bus);
> +	pci_hp_remove_devices(bus);
>
>  	/* Unmap PCI IO space */
>  	if (pcibios_unmap_io_space(bus)) {
> diff --git a/drivers/pci/hotplug/rpaphp_core.c b/drivers/pci/hotplug/rpaphp_core.c
> index 611f605..8d13202 100644
> --- a/drivers/pci/hotplug/rpaphp_core.c
> +++ b/drivers/pci/hotplug/rpaphp_core.c
> @@ -404,7 +404,7 @@ static int enable_slot(struct hotplug_slot *hotplug_slot)
>
>  	if (state == PRESENT) {
>  		pci_lock_rescan_remove();
> -		pcibios_add_pci_devices(slot->bus);
> +		pci_hp_add_devices(slot->bus);
>  		pci_unlock_rescan_remove();
>  		slot->state = CONFIGURED;
>  	} else if (state == EMPTY) {
> @@ -426,7 +426,7 @@ static int disable_slot(struct hotplug_slot *hotplug_slot)
>  		return -EINVAL;
>
>  	pci_lock_rescan_remove();
> -	pcibios_remove_pci_devices(slot->bus);
> +	pci_hp_remove_devices(slot->bus);
>  	pci_unlock_rescan_remove();
>  	vm_unmap_aliases();
>
> diff --git a/drivers/pci/hotplug/rpaphp_pci.c b/drivers/pci/hotplug/rpaphp_pci.c
> index 7836d69..137a892 100644
> --- a/drivers/pci/hotplug/rpaphp_pci.c
> +++ b/drivers/pci/hotplug/rpaphp_pci.c
> @@ -116,7 +116,7 @@ int rpaphp_enable_slot(struct slot *slot)
>  		}
>
>  		if (list_empty(&bus->devices))
> -			pcibios_add_pci_devices(bus);
> +			pci_hp_add_devices(bus);
>
>  		if (!list_empty(&bus->devices)) {
>  			info->adapter_status = CONFIGURED;
>


-- 
Alexey

Re: [PATCH v9 20/26] powerpc/pci: Move pci_find_bus_by_node() around

[Alexey Kardashevskiy]

On 05/03/2016 03:41 PM, Gavin Shan wrote:
> This moves pci_find_bus_by_node() from arch/powerpc/platforms/
> pseries/pci_dlpar.c to arch/powerpc/kernel/pci-hotplug.c so that
> the function can be used by pSeries and PowerNV platform at the
> same time. Also, below cleanup applied. No functional changes
> introduced.
>
>    * Remove variable "busdn" in find_bus_among_children()
>    * Use PCI_DN() to convert device node to pci_dn
>
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> Acked-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>

Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>

> ---
>  arch/powerpc/kernel/pci-hotplug.c          | 29 ++++++++++++++++++++++++++++
>  arch/powerpc/platforms/pseries/pci_dlpar.c | 31 ------------------------------
>  2 files changed, 29 insertions(+), 31 deletions(-)
>
> diff --git a/arch/powerpc/kernel/pci-hotplug.c b/arch/powerpc/kernel/pci-hotplug.c
> index 2d108e5..46587a1 100644
> --- a/arch/powerpc/kernel/pci-hotplug.c
> +++ b/arch/powerpc/kernel/pci-hotplug.c
> @@ -21,6 +21,35 @@
>  #include <asm/firmware.h>
>  #include <asm/eeh.h>
>
> +static struct pci_bus *find_bus_among_children(struct pci_bus *bus,
> +					       struct device_node *dn)
> +{
> +	struct pci_bus *child = NULL;
> +	struct pci_bus *tmp;
> +
> +	if (pci_bus_to_OF_node(bus) == dn)
> +		return bus;
> +
> +	list_for_each_entry(tmp, &bus->children, node) {
> +		child = find_bus_among_children(tmp, dn);
> +		if (child)
> +			break;
> +	}
> +
> +	return child;
> +}
> +
> +struct pci_bus *pci_find_bus_by_node(struct device_node *dn)
> +{
> +	struct pci_dn *pdn = PCI_DN(dn);
> +
> +	if (!pdn  || !pdn->phb || !pdn->phb->bus)
> +		return NULL;
> +
> +	return find_bus_among_children(pdn->phb->bus, dn);
> +}
> +EXPORT_SYMBOL_GPL(pci_find_bus_by_node);
> +
>  /**
>   * pcibios_release_device - release PCI device
>   * @dev: PCI device
> diff --git a/arch/powerpc/platforms/pseries/pci_dlpar.c b/arch/powerpc/platforms/pseries/pci_dlpar.c
> index aee22b4..906dbaa 100644
> --- a/arch/powerpc/platforms/pseries/pci_dlpar.c
> +++ b/arch/powerpc/platforms/pseries/pci_dlpar.c
> @@ -34,37 +34,6 @@
>
>  #include "pseries.h"
>
> -static struct pci_bus *
> -find_bus_among_children(struct pci_bus *bus,
> -                        struct device_node *dn)
> -{
> -	struct pci_bus *child = NULL;
> -	struct pci_bus *tmp;
> -	struct device_node *busdn;
> -
> -	busdn = pci_bus_to_OF_node(bus);
> -	if (busdn == dn)
> -		return bus;
> -
> -	list_for_each_entry(tmp, &bus->children, node) {
> -		child = find_bus_among_children(tmp, dn);
> -		if (child)
> -			break;
> -	};
> -	return child;
> -}
> -
> -struct pci_bus *pci_find_bus_by_node(struct device_node *dn)
> -{
> -	struct pci_dn *pdn = dn->data;
> -
> -	if (!pdn  || !pdn->phb || !pdn->phb->bus)
> -		return NULL;
> -
> -	return find_bus_among_children(pdn->phb->bus, dn);
> -}
> -EXPORT_SYMBOL_GPL(pci_find_bus_by_node);
> -
>  struct pci_controller *init_phb_dynamic(struct device_node *dn)
>  {
>  	struct pci_controller *phb;
>


-- 
Alexey

Re: [PATCH v10 16/26] powerpc/powernv/ioda1: Improve DMA32 segment track

[Alexey Kardashevskiy]

On 05/05/2016 12:04 PM, Gavin Shan wrote:
> In current implementation, the DMA32 segments required by one specific
> PE isn't calculated with the information hold in the PE independently.
> It conflicts with the PCI hotplug design: PE centralized, meaning the
> PE's DMA32 segments should be calculated from the information hold in
> the PE independently.
>
> This introduces an array (@dma32_segmap) for every PHB to track the
> DMA32 segmeng usage. Besides, this moves the logic calculating PE's
> consumed DMA32 segments to pnv_pci_ioda1_setup_dma_pe() so that PE's
> DMA32 segments are calculated/allocated from the information hold in
> the PE (DMA32 weight). Also the logic is improved: we try to allocate
> as much DMA32 segments as we can. It's acceptable that number of DMA32
> segments less than the expected number are allocated.
>
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>

Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>

> ---
> v10: Fixed picking same DMA32 segment by different PEs in some cases
> ---
>  arch/powerpc/platforms/powernv/pci-ioda.c | 115 ++++++++++++++++--------------
>  arch/powerpc/platforms/powernv/pci.h      |   7 +-
>  2 files changed, 66 insertions(+), 56 deletions(-)
>
> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
> index 30c825a..98879a0 100644
> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
> @@ -2011,27 +2011,62 @@ static unsigned int pnv_pci_ioda_pe_dma_weight(struct pnv_ioda_pe *pe)
>  }
>
>  static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
> -				       struct pnv_ioda_pe *pe,
> -				       unsigned int base,
> -				       unsigned int segs)
> +				       struct pnv_ioda_pe *pe)
>  {
>
>  	struct page *tce_mem = NULL;
>  	struct iommu_table *tbl;
> -	unsigned int tce32_segsz, i;
> +	unsigned int weight, total_weight = 0;
> +	unsigned int tce32_segsz, base, segs, avail, i;
>  	int64_t rc;
>  	void *addr;
>
>  	/* XXX FIXME: Handle 64-bit only DMA devices */
>  	/* XXX FIXME: Provide 64-bit DMA facilities & non-4K TCE tables etc.. */
>  	/* XXX FIXME: Allocate multi-level tables on PHB3 */
> +	weight = pnv_pci_ioda_pe_dma_weight(pe);
> +	if (!weight)
> +		return;
> +
> +	pci_walk_bus(phb->hose->bus, pnv_pci_ioda_dev_dma_weight,
> +		     &total_weight);
> +	segs = (weight * phb->ioda.dma32_count) / total_weight;
> +	if (!segs)
> +		segs = 1;
>
> +	/*
> +	 * Allocate contiguous DMA32 segments. We begin with the expected
> +	 * number of segments. With one more attempt, the number of DMA32
> +	 * segments to be allocated is decreased by one until one segment
> +	 * is allocated successfully.
> +	 */
> +	do {
> +		for (base = 0; base <= phb->ioda.dma32_count - segs; base++) {
> +			for (avail = 0, i = base; i < base + segs; i++) {
> +				if (phb->ioda.dma32_segmap[i] ==
> +				    IODA_INVALID_PE)
> +					avail++;
> +			}
> +
> +			if (avail == segs)
> +				goto found;
> +		}
> +	} while (--segs);
> +
> +	if (!segs) {
> +		pe_warn(pe, "No available DMA32 segments\n");
> +		return;
> +	}
> +
> +found:
>  	tbl = pnv_pci_table_alloc(phb->hose->node);
>  	iommu_register_group(&pe->table_group, phb->hose->global_number,
>  			pe->pe_number);
>  	pnv_pci_link_table_and_group(phb->hose->node, 0, tbl, &pe->table_group);
>
>  	/* Grab a 32-bit TCE table */
> +	pe_info(pe, "DMA weight %d (%d), assigned (%d) %d DMA32 segments\n",
> +		weight, total_weight, base, segs);
>  	pe_info(pe, " Setting up 32-bit TCE table at %08x..%08x\n",
>  		base * PNV_IODA1_DMA32_SEGSIZE,
>  		(base + segs) * PNV_IODA1_DMA32_SEGSIZE - 1);
> @@ -2068,6 +2103,10 @@ static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
>  		}
>  	}
>
> +	/* Setup DMA32 segment mapping */
> +	for (i = base; i < base + segs; i++)
> +		phb->ioda.dma32_segmap[i] = pe->pe_number;
> +
>  	/* Setup linux iommu table */
>  	pnv_pci_setup_iommu_table(tbl, addr, tce32_segsz * segs,
>  				  base * PNV_IODA1_DMA32_SEGSIZE,
> @@ -2542,73 +2581,34 @@ static void pnv_pci_ioda2_setup_dma_pe(struct pnv_phb *phb,
>  static void pnv_ioda_setup_dma(struct pnv_phb *phb)
>  {
>  	struct pci_controller *hose = phb->hose;
> -	unsigned int weight, total_weight, dma_pe_count;
> -	unsigned int residual, remaining, segs, base;
>  	struct pnv_ioda_pe *pe;
> -
> -	total_weight = 0;
> -	pci_walk_bus(phb->hose->bus, pnv_pci_ioda_dev_dma_weight,
> -		     &total_weight);
> -
> -	dma_pe_count = 0;
> -	list_for_each_entry(pe, &phb->ioda.pe_list, list) {
> -		weight = pnv_pci_ioda_pe_dma_weight(pe);
> -		if (weight > 0)
> -			dma_pe_count++;
> -	}
> +	unsigned int weight;
>
>  	/* If we have more PE# than segments available, hand out one
>  	 * per PE until we run out and let the rest fail. If not,
>  	 * then we assign at least one segment per PE, plus more based
>  	 * on the amount of devices under that PE
>  	 */
> -	if (dma_pe_count > phb->ioda.tce32_count)
> -		residual = 0;
> -	else
> -		residual = phb->ioda.tce32_count - dma_pe_count;
> -
> -	pr_info("PCI: Domain %04x has %ld available 32-bit DMA segments\n",
> -		hose->global_number, phb->ioda.tce32_count);
> -	pr_info("PCI: %d PE# for a total weight of %d\n",
> -		dma_pe_count, total_weight);
> +	pr_info("PCI: Domain %04x has %d available 32-bit DMA segments\n",
> +		hose->global_number, phb->ioda.dma32_count);
>
>  	pnv_pci_ioda_setup_opal_tce_kill(phb);
>
> -	/* Walk our PE list and configure their DMA segments, hand them
> -	 * out one base segment plus any residual segments based on
> -	 * weight
> -	 */
> -	remaining = phb->ioda.tce32_count;
> -	base = 0;
> +	/* Walk our PE list and configure their DMA segments */
>  	list_for_each_entry(pe, &phb->ioda.pe_list, list) {
>  		weight = pnv_pci_ioda_pe_dma_weight(pe);
>  		if (!weight)
>  			continue;
>
> -		if (!remaining) {
> -			pe_warn(pe, "No DMA32 resources available\n");
> -			continue;
> -		}
> -		segs = 1;
> -		if (residual) {
> -			segs += ((weight * residual) + (total_weight / 2)) /
> -				total_weight;
> -			if (segs > remaining)
> -				segs = remaining;
> -		}
> -
>  		/*
>  		 * For IODA2 compliant PHB3, we needn't care about the weight.
>  		 * The all available 32-bits DMA space will be assigned to
>  		 * the specific PE.
>  		 */
>  		if (phb->type == PNV_PHB_IODA1) {
> -			pe_info(pe, "DMA weight %d, assigned %d DMA32 segments\n",
> -				weight, segs);
> -			pnv_pci_ioda1_setup_dma_pe(phb, pe, base, segs);
> +			pnv_pci_ioda1_setup_dma_pe(phb, pe);
>  		} else if (phb->type == PNV_PHB_IODA2) {
>  			pe_info(pe, "Assign DMA32 space\n");
> -			segs = 0;
>  			pnv_pci_ioda2_setup_dma_pe(phb, pe);
>  		} else if (phb->type == PNV_PHB_NPU) {
>  			/*
> @@ -2618,9 +2618,6 @@ static void pnv_ioda_setup_dma(struct pnv_phb *phb)
>  			 * as the PHB3 TVT.
>  			 */
>  		}
> -
> -		remaining -= segs;
> -		base += segs;
>  	}
>  }
>
> @@ -3327,7 +3324,8 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
>  {
>  	struct pci_controller *hose;
>  	struct pnv_phb *phb;
> -	unsigned long size, m64map_off, m32map_off, pemap_off, iomap_off = 0;
> +	unsigned long size, m64map_off, m32map_off, pemap_off;
> +	unsigned long iomap_off = 0, dma32map_off = 0;
>  	const __be64 *prop64;
>  	const __be32 *prop32;
>  	int len;
> @@ -3413,6 +3411,10 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
>  	phb->ioda.io_segsize = phb->ioda.io_size / phb->ioda.total_pe_num;
>  	phb->ioda.io_pci_base = 0; /* XXX calculate this ? */
>
> +	/* Calculate how many 32-bit TCE segments we have */
> +	phb->ioda.dma32_count = phb->ioda.m32_pci_base /
> +				PNV_IODA1_DMA32_SEGSIZE;
> +
>  	/* Allocate aux data & arrays. We don't have IO ports on PHB3 */
>  	size = _ALIGN_UP(phb->ioda.total_pe_num / 8, sizeof(unsigned long));
>  	m64map_off = size;
> @@ -3422,6 +3424,9 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
>  	if (phb->type == PNV_PHB_IODA1) {
>  		iomap_off = size;
>  		size += phb->ioda.total_pe_num * sizeof(phb->ioda.io_segmap[0]);
> +		dma32map_off = size;
> +		size += phb->ioda.dma32_count *
> +			sizeof(phb->ioda.dma32_segmap[0]);
>  	}
>  	pemap_off = size;
>  	size += phb->ioda.total_pe_num * sizeof(struct pnv_ioda_pe);
> @@ -3437,6 +3442,10 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
>  		phb->ioda.io_segmap = aux + iomap_off;
>  		for (segno = 0; segno < phb->ioda.total_pe_num; segno++)
>  			phb->ioda.io_segmap[segno] = IODA_INVALID_PE;
> +
> +		phb->ioda.dma32_segmap = aux + dma32map_off;
> +		for (segno = 0; segno < phb->ioda.dma32_count; segno++)
> +			phb->ioda.dma32_segmap[segno] = IODA_INVALID_PE;
>  	}
>  	phb->ioda.pe_array = aux + pemap_off;
>  	set_bit(phb->ioda.reserved_pe_idx, phb->ioda.pe_alloc);
> @@ -3445,7 +3454,7 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
>  	mutex_init(&phb->ioda.pe_list_mutex);
>
>  	/* Calculate how many 32-bit TCE segments we have */
> -	phb->ioda.tce32_count = phb->ioda.m32_pci_base /
> +	phb->ioda.dma32_count = phb->ioda.m32_pci_base /
>  				PNV_IODA1_DMA32_SEGSIZE;
>
>  #if 0 /* We should really do that ... */
> diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
> index 117cfcd..14d9391 100644
> --- a/arch/powerpc/platforms/powernv/pci.h
> +++ b/arch/powerpc/platforms/powernv/pci.h
> @@ -142,6 +142,10 @@ struct pnv_phb {
>  		unsigned int		*m32_segmap;
>  		unsigned int		*io_segmap;
>
> +		/* DMA32 segment maps - IODA1 only */
> +		unsigned int		dma32_count;
> +		unsigned int		*dma32_segmap;
> +
>  		/* IRQ chip */
>  		int			irq_chip_init;
>  		struct irq_chip		irq_chip;
> @@ -158,9 +162,6 @@ struct pnv_phb {
>  		 */
>  		unsigned char		pe_rmap[0x10000];
>
> -		/* 32-bit TCE tables allocation */
> -		unsigned long		tce32_count;
> -
>  		/* TCE cache invalidate registers (physical and
>  		 * remapped)
>  		 */
>


-- 
Alexey

Re: [PATCH v9 02/26] powerpc/powernv: Cleanup on pci_controller_ops instances

[Alexey Kardashevskiy]

On 05/03/2016 03:41 PM, Gavin Shan wrote:
> This cleans up on below data struct instances to use tab instead of
> space indent of statement to avoid complains from scripts/checkpatch.pl.
> No logical changes introduced.
>
>   @pnv_pci_ioda_controller_ops
>   @pnv_npu_ioda_controller_ops
>
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> Reviewed-by: Daniel Axtens <dja@axtens.net>
> Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>


Spaces to tabs conversion is correct, aligning "=" is not to _my_ 
_personal_ _taste_.


Acked-by: Alexey Kardashevskiy <aik@ozlabs.ru>




> ---
>  arch/powerpc/platforms/powernv/pci-ioda.c | 36 +++++++++++++++----------------
>  1 file changed, 18 insertions(+), 18 deletions(-)
>
> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
> index c5baaf3..524c9c7 100644
> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
> @@ -3210,31 +3210,31 @@ static void pnv_pci_ioda_shutdown(struct pci_controller *hose)
>  }
>
>  static const struct pci_controller_ops pnv_pci_ioda_controller_ops = {
> -       .dma_dev_setup = pnv_pci_dma_dev_setup,
> -       .dma_bus_setup = pnv_pci_dma_bus_setup,
> +	.dma_dev_setup		= pnv_pci_dma_dev_setup,
> +	.dma_bus_setup		= pnv_pci_dma_bus_setup,
>  #ifdef CONFIG_PCI_MSI
> -       .setup_msi_irqs = pnv_setup_msi_irqs,
> -       .teardown_msi_irqs = pnv_teardown_msi_irqs,
> +	.setup_msi_irqs		= pnv_setup_msi_irqs,
> +	.teardown_msi_irqs	= pnv_teardown_msi_irqs,
>  #endif
> -       .enable_device_hook = pnv_pci_enable_device_hook,
> -       .window_alignment = pnv_pci_window_alignment,
> -       .reset_secondary_bus = pnv_pci_reset_secondary_bus,
> -       .dma_set_mask = pnv_pci_ioda_dma_set_mask,
> -       .dma_get_required_mask = pnv_pci_ioda_dma_get_required_mask,
> -       .shutdown = pnv_pci_ioda_shutdown,
> +	.enable_device_hook	= pnv_pci_enable_device_hook,
> +	.window_alignment	= pnv_pci_window_alignment,
> +	.reset_secondary_bus	= pnv_pci_reset_secondary_bus,
> +	.dma_set_mask		= pnv_pci_ioda_dma_set_mask,
> +	.dma_get_required_mask	= pnv_pci_ioda_dma_get_required_mask,
> +	.shutdown		= pnv_pci_ioda_shutdown,
>  };
>
>  static const struct pci_controller_ops pnv_npu_ioda_controller_ops = {
> -	.dma_dev_setup = pnv_pci_dma_dev_setup,
> +	.dma_dev_setup		= pnv_pci_dma_dev_setup,
>  #ifdef CONFIG_PCI_MSI
> -	.setup_msi_irqs = pnv_setup_msi_irqs,
> -	.teardown_msi_irqs = pnv_teardown_msi_irqs,
> +	.setup_msi_irqs		= pnv_setup_msi_irqs,
> +	.teardown_msi_irqs	= pnv_teardown_msi_irqs,
>  #endif
> -	.enable_device_hook = pnv_pci_enable_device_hook,
> -	.window_alignment = pnv_pci_window_alignment,
> -	.reset_secondary_bus = pnv_pci_reset_secondary_bus,
> -	.dma_set_mask = pnv_npu_dma_set_mask,
> -	.shutdown = pnv_pci_ioda_shutdown,
> +	.enable_device_hook	= pnv_pci_enable_device_hook,
> +	.window_alignment	= pnv_pci_window_alignment,
> +	.reset_secondary_bus	= pnv_pci_reset_secondary_bus,
> +	.dma_set_mask		= pnv_npu_dma_set_mask,
> +	.shutdown		= pnv_pci_ioda_shutdown,
>  };
>
>  static void __init pnv_pci_init_ioda_phb(struct device_node *np,
>


-- 
Alexey

Re: [v9,01/26] powerpc/pci: Cleanup on struct pci_controller_ops

[Michael Ellerman]

On Tue, 2016-03-05 at 05:41:20 UTC, Gavin Shan wrote:
> Each PHB has one instance of "struct pci_controller_ops" that includes
> various callbacks called by PCI subsystem. In the definition of this
> struct, some callbacks have explicit names for its arguments, but the
> left don't have.
> 
> This adds all explicit names of the arguments to the callbacks in
> "struct pci_controller_ops" so that the code looks consistent. Also,
> argument name @dev is replaced by @pdev as the later one is the
> preferred name for PCI device.
> 
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> Reviewed-by: Daniel Axtens <dja@axtens.net>
> Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
> Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>

Series applied to powerpc next, thanks.

https://git.kernel.org/powerpc/c/ec73f9a8dfdfce0112ee041705

cheers

Re: [PATCH v9 26/26] powerpc/powernv: Exclude root bus in pnv_pci_reset_secondary_bus()

[Gavin Shan]

On Tue, May 03, 2016 at 03:41:45PM +1000, Gavin Shan wrote:
>The function pnv_pci_reset_secondary_bus() is called like below.
>It's impossible for call the function on root bus. So it's safe
>to remove the root bus case in the function. No functional changes
>introduced.
>
>   pci_parent_bus_reset() / pci_bus_reset() / pci_try_reset_bus()
>   pci_reset_bridge_secondary_bus()
>   pcibios_reset_secondary_bus()
>   pnv_pci_reset_secondary_bus()
>
>Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>Reviewed-by: Daniel Axtens <dja@axtens.net>
>Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>---
> arch/powerpc/platforms/powernv/eeh-powernv.c | 12 ++----------
> 1 file changed, 2 insertions(+), 10 deletions(-)
>
>diff --git a/arch/powerpc/platforms/powernv/eeh-powernv.c b/arch/powerpc/platforms/powernv/eeh-powernv.c
>index 9226df1..593b8dc 100644
>--- a/arch/powerpc/platforms/powernv/eeh-powernv.c
>+++ b/arch/powerpc/platforms/powernv/eeh-powernv.c
>@@ -868,16 +868,8 @@ static int pnv_eeh_bridge_reset(struct pci_dev *dev, int option)
>
> void pnv_pci_reset_secondary_bus(struct pci_dev *dev)
> {
>-	struct pci_controller *hose;
>-
>-	if (pci_is_root_bus(dev->bus)) {
>-		hose = pci_bus_to_host(dev->bus);
>-		pnv_eeh_root_reset(hose, EEH_RESET_HOT);
>-		pnv_eeh_root_reset(hose, EEH_RESET_DEACTIVATE);
>-	} else {
>-		pnv_eeh_bridge_reset(dev, EEH_RESET_HOT);
>-		pnv_eeh_bridge_reset(dev, EEH_RESET_DEACTIVATE);
>-	}
>+	pnv_eeh_bridge_reset(dev, EEH_RESET_HOT);
>+	pnv_eeh_bridge_reset(dev, EEH_RESET_DEACTIVATE);
> }

Michael, please revert this one as it is already in linux-ppc-next
branch. Sorry for the overhead.

Obviously, I missed the truth that it affects the PCI passthrou path as
reported by Alexey: When passing GPU (0003:01:00.0) which seats behind
the root port, the reset request is routed to skiboot in original code.
In skiboot, the link bouncing events are masked during the reset. So we
don't see EEH (freeze all) error even link bouncing happens. With the
changes included, the reset is done by kernel and the link bouncing
events aren't masked by altering content of PHB3 (or P7IOC) specific
hardware registers which are invisible to kernel (skiboot hides the
hardware specific). It means the link bouncing is seen by the root port
and it causes a EEH (freeze all) error. The PCI passthrough on GPU device
cannot work.

Thanks,
Gavin

>
> static void pnv_eeh_wait_for_pending(struct pci_dn *pdn, const char *type,
>-- 
>2.1.0
>

Re: [PATCH v9 26/26] powerpc/powernv: Exclude root bus in pnv_pci_reset_secondary_bus()

[Michael Ellerman]

On Thu, 2016-05-12 at 13:48 +1000, Gavin Shan wrote:

> On Tue, May 03, 2016 at 03:41:45PM +1000, Gavin Shan wrote:

> > The function pnv_pci_reset_secondary_bus() is called like below.
> > It's impossible for call the function on root bus. So it's safe
> > to remove the root bus case in the function. No functional changes
> > introduced.
> > 
> >   pci_parent_bus_reset() / pci_bus_reset() / pci_try_reset_bus()
> >   pci_reset_bridge_secondary_bus()
> >   pcibios_reset_secondary_bus()
> >   pnv_pci_reset_secondary_bus()
> > 
> > Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> > Reviewed-by: Daniel Axtens <dja@axtens.net>
> > Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> > ---
> > arch/powerpc/platforms/powernv/eeh-powernv.c | 12 ++----------
> > 1 file changed, 2 insertions(+), 10 deletions(-)
> > 
> > diff --git a/arch/powerpc/platforms/powernv/eeh-powernv.c b/arch/powerpc/platforms/powernv/eeh-powernv.c
> > index 9226df1..593b8dc 100644
> > --- a/arch/powerpc/platforms/powernv/eeh-powernv.c
> > +++ b/arch/powerpc/platforms/powernv/eeh-powernv.c
> > @@ -868,16 +868,8 @@ static int pnv_eeh_bridge_reset(struct pci_dev *dev, int option)
> > 
> > void pnv_pci_reset_secondary_bus(struct pci_dev *dev)
> > {
> > -	struct pci_controller *hose;
> > -
> > -	if (pci_is_root_bus(dev->bus)) {
> > -		hose = pci_bus_to_host(dev->bus);
> > -		pnv_eeh_root_reset(hose, EEH_RESET_HOT);
> > -		pnv_eeh_root_reset(hose, EEH_RESET_DEACTIVATE);
> > -	} else {
> > -		pnv_eeh_bridge_reset(dev, EEH_RESET_HOT);
> > -		pnv_eeh_bridge_reset(dev, EEH_RESET_DEACTIVATE);
> > -	}
> > +	pnv_eeh_bridge_reset(dev, EEH_RESET_HOT);
> > +	pnv_eeh_bridge_reset(dev, EEH_RESET_DEACTIVATE);
> > }
> 
> Michael, please revert this one as it is already in linux-ppc-next
> branch. Sorry for the overhead.

Done.

https://git.kernel.org/cgit/linux/kernel/git/powerpc/linux.git/commit/?h=next&id=848912e547c4569445a61203a7df402646a88c25

cheers