[PATCH v1 00/10] powerpc/eeh: Remove eeh_mutex

[PATCH v1 00/10] powerpc/eeh: Remove eeh_mutex

[Gavin Shan]

The series of patches are follow-up in order to make EEH workable for PowerNV
platform on Juno-IOC-L machine. Couple of issues have been fixed with help of
Ben:
	- eeh_lock() and eeh_unlock() were introduced to protect the PE hierarchy
	  tree. However, we already had one kthread ("eehd"). So that's not necessary
	  any more.
	- When PHB gets fenced, we need do complete reset for the PHB in order
	  for recovery. However, we never checked the downstream PCIe links are
	  ready again.
	- Introduce mechanism to block accessing to PCI-CFG and MMIO. The hardware
	  should return 0xFF's while the PHB is fenced. So we needn't access PCI-CFG
	  and MMIO during the stage (before the PHB gets complete reset).
	- EEH address cache wasn't populated on PowerNV.
	- PCI-CFG for PCI bridges (PCI_COMMAND) wasn't restored correctly.
	- While PHB gets fenced, TG3 driver is still trying to access MMIO with loop.
	  That's unnecessary.

The series of patches have been verified on Juno-IOC-L machine:

Trigger frozen PE:

        echo 0x0000000002000000 > /sys/kernel/debug/powerpc/PCI0000/err_injct
        sleep 1
        echo 0x0 > /sys/kernel/debug/powerpc/PCI0000/err_injct

Trigger fenced PHB:

	echo 0x8000000000000000 > /sys/kernel/debug/powerpc/PCI0000/err_injct

---

arch/powerpc/include/asm/eeh.h               |  251 ++++++++++++++++++++------
arch/powerpc/include/asm/io.h                |   67 ++++---
arch/powerpc/kernel/eeh.c                    |   85 ++++++---
arch/powerpc/kernel/eeh_cache.c              |    2 +-
arch/powerpc/kernel/eeh_driver.c             |   11 ++
arch/powerpc/kernel/eeh_event.c              |    3 +-
arch/powerpc/kernel/eeh_pe.c                 |  150 ++++++++++++----
arch/powerpc/platforms/powernv/eeh-ioda.c    |   88 +++++++++-
arch/powerpc/platforms/powernv/eeh-powernv.c |   42 ++++-
arch/powerpc/platforms/powernv/pci-ioda.c    |    1 +
arch/powerpc/platforms/powernv/pci.c         |   63 +++++--
arch/powerpc/platforms/powernv/pci.h         |   16 ++-
arch/powerpc/platforms/pseries/eeh_pseries.c |   44 +++++
drivers/net/ethernet/broadcom/tg3.c          |   36 ++++
14 files changed, 685 insertions(+), 174 deletions(-)

Thanks,
Gavin

[PATCH 01/10] powerpc/eeh: Remove eeh_mutex

[Gavin Shan]

Originally, eeh_mutex was introduced to protect the PE hierarchy
tree and the attached EEH devices because EEH core was possiblly
running with multiple threads to access the PE hierarchy tree.
However, we now have only one kthread in EEH core. So we needn't
the eeh_mutex and just remove it. The patch also allows to be
interrupted while waiting on EEH event semaphore.

Signed-off-by: Gavin Shan <shangw@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/eeh.h  |   14 --------------
 arch/powerpc/kernel/eeh.c       |    3 ---
 arch/powerpc/kernel/eeh_event.c |    3 ++-
 arch/powerpc/kernel/eeh_pe.c    |   30 +-----------------------------
 4 files changed, 3 insertions(+), 47 deletions(-)

diff --git a/arch/powerpc/include/asm/eeh.h b/arch/powerpc/include/asm/eeh.h
index a0b11fb..dd65e31 100644
--- a/arch/powerpc/include/asm/eeh.h
+++ b/arch/powerpc/include/asm/eeh.h
@@ -151,7 +151,6 @@ struct eeh_ops {
 
 extern struct eeh_ops *eeh_ops;
 extern int eeh_subsystem_enabled;
-extern struct mutex eeh_mutex;
 extern raw_spinlock_t confirm_error_lock;
 extern int eeh_probe_mode;
 
@@ -173,16 +172,6 @@ static inline int eeh_probe_mode_dev(void)
 	return (eeh_probe_mode == EEH_PROBE_MODE_DEV);
 }
 
-static inline void eeh_lock(void)
-{
-	mutex_lock(&eeh_mutex);
-}
-
-static inline void eeh_unlock(void)
-{
-	mutex_unlock(&eeh_mutex);
-}
-
 static inline void eeh_serialize_lock(unsigned long *flags)
 {
 	raw_spin_lock_irqsave(&confirm_error_lock, *flags);
@@ -271,9 +260,6 @@ static inline void eeh_add_sysfs_files(struct pci_bus *bus) { }
 
 static inline void eeh_remove_bus_device(struct pci_dev *dev, int purge_pe) { }
 
-static inline void eeh_lock(void) { }
-static inline void eeh_unlock(void) { }
-
 #define EEH_POSSIBLE_ERROR(val, type) (0)
 #define EEH_IO_ERROR_VALUE(size) (-1UL)
 #endif /* CONFIG_EEH */
diff --git a/arch/powerpc/kernel/eeh.c b/arch/powerpc/kernel/eeh.c
index 7c567be..951a632 100644
--- a/arch/powerpc/kernel/eeh.c
+++ b/arch/powerpc/kernel/eeh.c
@@ -103,9 +103,6 @@ EXPORT_SYMBOL(eeh_subsystem_enabled);
  */
 int eeh_probe_mode;
 
-/* Global EEH mutex */
-DEFINE_MUTEX(eeh_mutex);
-
 /* Lock to avoid races due to multiple reports of an error */
 DEFINE_RAW_SPINLOCK(confirm_error_lock);
 
diff --git a/arch/powerpc/kernel/eeh_event.c b/arch/powerpc/kernel/eeh_event.c
index 39bcd81..165943e 100644
--- a/arch/powerpc/kernel/eeh_event.c
+++ b/arch/powerpc/kernel/eeh_event.c
@@ -55,7 +55,8 @@ static int eeh_event_handler(void * dummy)
 	struct eeh_pe *pe;
 
 	while (!kthread_should_stop()) {
-		down(&eeh_eventlist_sem);
+		 if (down_interruptible(&eeh_eventlist_sem))
+			break;
 
 		/* Fetch EEH event from the queue */
 		spin_lock_irqsave(&eeh_eventlist_lock, flags);
diff --git a/arch/powerpc/kernel/eeh_pe.c b/arch/powerpc/kernel/eeh_pe.c
index ae75722..55943fc 100644
--- a/arch/powerpc/kernel/eeh_pe.c
+++ b/arch/powerpc/kernel/eeh_pe.c
@@ -78,9 +78,7 @@ int eeh_phb_pe_create(struct pci_controller *phb)
 	}
 
 	/* Put it into the list */
-	eeh_lock();
 	list_add_tail(&pe->child, &eeh_phb_pe);
-	eeh_unlock();
 
 	pr_debug("EEH: Add PE for PHB#%d\n", phb->global_number);
 
@@ -185,21 +183,15 @@ void *eeh_pe_dev_traverse(struct eeh_pe *root,
 		return NULL;
 	}
 
-	eeh_lock();
-
 	/* Traverse root PE */
 	for (pe = root; pe; pe = eeh_pe_next(pe, root)) {
 		eeh_pe_for_each_dev(pe, edev) {
 			ret = fn(edev, flag);
-			if (ret) {
-				eeh_unlock();
+			if (ret)
 				return ret;
-			}
 		}
 	}
 
-	eeh_unlock();
-
 	return NULL;
 }
 
@@ -305,8 +297,6 @@ int eeh_add_to_parent_pe(struct eeh_dev *edev)
 {
 	struct eeh_pe *pe, *parent;
 
-	eeh_lock();
-
 	/*
 	 * Search the PE has been existing or not according
 	 * to the PE address. If that has been existing, the
@@ -316,7 +306,6 @@ int eeh_add_to_parent_pe(struct eeh_dev *edev)
 	pe = eeh_pe_get(edev);
 	if (pe && !(pe->type & EEH_PE_INVALID)) {
 		if (!edev->pe_config_addr) {
-			eeh_unlock();
 			pr_err("%s: PE with addr 0x%x already exists\n",
 				__func__, edev->config_addr);
 			return -EEXIST;
@@ -328,7 +317,6 @@ int eeh_add_to_parent_pe(struct eeh_dev *edev)
 
 		/* Put the edev to PE */
 		list_add_tail(&edev->list, &pe->edevs);
-		eeh_unlock();
 		pr_debug("EEH: Add %s to Bus PE#%x\n",
 			edev->dn->full_name, pe->addr);
 
@@ -347,7 +335,6 @@ int eeh_add_to_parent_pe(struct eeh_dev *edev)
 			parent->type &= ~EEH_PE_INVALID;
 			parent = parent->parent;
 		}
-		eeh_unlock();
 		pr_debug("EEH: Add %s to Device PE#%x, Parent PE#%x\n",
 			edev->dn->full_name, pe->addr, pe->parent->addr);
 
@@ -357,7 +344,6 @@ int eeh_add_to_parent_pe(struct eeh_dev *edev)
 	/* Create a new EEH PE */
 	pe = eeh_pe_alloc(edev->phb, EEH_PE_DEVICE);
 	if (!pe) {
-		eeh_unlock();
 		pr_err("%s: out of memory!\n", __func__);
 		return -ENOMEM;
 	}
@@ -385,7 +371,6 @@ int eeh_add_to_parent_pe(struct eeh_dev *edev)
 	if (!parent) {
 		parent = eeh_phb_pe_get(edev->phb);
 		if (!parent) {
-			eeh_unlock();
 			pr_err("%s: No PHB PE is found (PHB Domain=%d)\n",
 				__func__, edev->phb->global_number);
 			edev->pe = NULL;
@@ -402,7 +387,6 @@ int eeh_add_to_parent_pe(struct eeh_dev *edev)
 	list_add_tail(&pe->child, &parent->child_list);
 	list_add_tail(&edev->list, &pe->edevs);
 	edev->pe = pe;
-	eeh_unlock();
 	pr_debug("EEH: Add %s to Device PE#%x, Parent PE#%x\n",
 		edev->dn->full_name, pe->addr, pe->parent->addr);
 
@@ -430,8 +414,6 @@ int eeh_rmv_from_parent_pe(struct eeh_dev *edev, int purge_pe)
 		return -EEXIST;
 	}
 
-	eeh_lock();
-
 	/* Remove the EEH device */
 	pe = edev->pe;
 	edev->pe = NULL;
@@ -476,8 +458,6 @@ int eeh_rmv_from_parent_pe(struct eeh_dev *edev, int purge_pe)
 		pe = parent;
 	}
 
-	eeh_unlock();
-
 	return 0;
 }
 
@@ -550,9 +530,7 @@ static void *__eeh_pe_state_mark(void *data, void *flag)
  */
 void eeh_pe_state_mark(struct eeh_pe *pe, int state)
 {
-	eeh_lock();
 	eeh_pe_traverse(pe, __eeh_pe_state_mark, &state);
-	eeh_unlock();
 }
 
 /**
@@ -586,9 +564,7 @@ static void *__eeh_pe_state_clear(void *data, void *flag)
  */
 void eeh_pe_state_clear(struct eeh_pe *pe, int state)
 {
-	eeh_lock();
 	eeh_pe_traverse(pe, __eeh_pe_state_clear, &state);
-	eeh_unlock();
 }
 
 /**
@@ -673,8 +649,6 @@ struct pci_bus *eeh_pe_bus_get(struct eeh_pe *pe)
 	struct eeh_dev *edev;
 	struct pci_dev *pdev;
 
-	eeh_lock();
-
 	if (pe->type & EEH_PE_PHB) {
 		bus = pe->phb->bus;
 	} else if (pe->type & EEH_PE_BUS ||
@@ -691,7 +665,5 @@ struct pci_bus *eeh_pe_bus_get(struct eeh_pe *pe)
 	}
 
 out:
-	eeh_unlock();
-
 	return bus;
 }
-- 
1.7.5.4

[PATCH 02/10] powerpc/eeh: Don't collect PCI-CFG data on PHB

[Gavin Shan]

When the PHB is fenced or dead, it's pointless to collect the data
from PCI config space of subordinate PCI devices since it should
return 0xFF's. It also has potential risk to incur additional errors.
The patch avoids collecting PCI-CFG data while PHB is in fenced or
dead state.

Signed-off-by: Gavin Shan <shangw@linux.vnet.ibm.com>
---
 arch/powerpc/kernel/eeh.c |   34 ++++++++++++++++++++++++----------
 1 files changed, 24 insertions(+), 10 deletions(-)

diff --git a/arch/powerpc/kernel/eeh.c b/arch/powerpc/kernel/eeh.c
index 951a632..65320fd 100644
--- a/arch/powerpc/kernel/eeh.c
+++ b/arch/powerpc/kernel/eeh.c
@@ -232,16 +232,30 @@ void eeh_slot_error_detail(struct eeh_pe *pe, int severity)
 {
 	size_t loglen = 0;
 	struct eeh_dev *edev;
+	bool valid_cfg_log = true;
 
-	eeh_pci_enable(pe, EEH_OPT_THAW_MMIO);
-	eeh_ops->configure_bridge(pe);
-	eeh_pe_restore_bars(pe);
-
-	pci_regs_buf[0] = 0;
-	eeh_pe_for_each_dev(pe, edev) {
-		loglen += eeh_gather_pci_data(edev, pci_regs_buf,
-				EEH_PCI_REGS_LOG_LEN);
-        }
+	/*
+	 * When the PHB is fenced or dead, it's pointless to collect
+	 * the data from PCI config space because it should return
+	 * 0xFF's. The potential risk of that is introducing additional
+	 * errors.
+	 */
+	if (eeh_probe_mode_dev() &&
+	    (pe->type & EEH_PE_PHB) &&
+	    (pe->state & (EEH_PE_ISOLATED | EEH_PE_PHB_DEAD)))
+		valid_cfg_log = false;
+
+	if (valid_cfg_log) {
+		eeh_pci_enable(pe, EEH_OPT_THAW_MMIO);
+		eeh_ops->configure_bridge(pe);
+		eeh_pe_restore_bars(pe);
+
+		pci_regs_buf[0] = 0;
+		eeh_pe_for_each_dev(pe, edev) {
+			loglen += eeh_gather_pci_data(edev, pci_regs_buf,
+						      EEH_PCI_REGS_LOG_LEN);
+		}
+	}
 
 	eeh_ops->get_log(pe, severity, pci_regs_buf, loglen);
 }
-- 
1.7.5.4

[PATCH 03/10] powerpc/eeh: Check PCIe link after reset

[Gavin Shan]

After reset (e.g. complete reset) in order to bring the fenced PHB
back, the PCIe link might not be ready yet. The patch intends to
make sure the PCIe link is ready before accessing its subordinate
PCI devices. The patch also fixes that wrong values restored to
PCI_COMMAND register for PCI bridges.

Signed-off-by: Gavin Shan <shangw@linux.vnet.ibm.com>
---
 arch/powerpc/kernel/eeh_pe.c |  120 ++++++++++++++++++++++++++++++++++++++----
 1 files changed, 110 insertions(+), 10 deletions(-)

diff --git a/arch/powerpc/kernel/eeh_pe.c b/arch/powerpc/kernel/eeh_pe.c
index 55943fc..db83ada 100644
--- a/arch/powerpc/kernel/eeh_pe.c
+++ b/arch/powerpc/kernel/eeh_pe.c
@@ -22,6 +22,7 @@
  * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA
  */
 
+#include <linux/delay.h>
 #include <linux/export.h>
 #include <linux/gfp.h>
 #include <linux/init.h>
@@ -567,6 +568,88 @@ void eeh_pe_state_clear(struct eeh_pe *pe, int state)
 	eeh_pe_traverse(pe, __eeh_pe_state_clear, &state);
 }
 
+/*
+ * Some PCI bridges (e.g. PLX bridges) have primary/secondary
+ * buses assigned explicitly by firmware, and we probably have
+ * lost that after reset. So we have to delay the check until
+ * the PCI-CFG registers have been restored for the parent
+ * bridge.
+ *
+ * Don't use normal PCI-CFG accessors, which probably has been
+ * blocked on normal path during the stage. So we need utilize
+ * eeh operations, which is always permitted.
+ */
+static void eeh_bridge_check_link(struct device_node *dn,
+				  struct pci_dev *pdev)
+{
+	int cap;
+	uint32_t val;
+	int timeout = 0;
+
+	/*
+	 * We only check root port and downstream ports of
+	 * PCIe switches
+	 */
+	if (!pci_is_pcie(pdev) ||
+	    (pci_pcie_type(pdev) != PCI_EXP_TYPE_ROOT_PORT &&
+	     pci_pcie_type(pdev) != PCI_EXP_TYPE_DOWNSTREAM))
+		return;
+
+	pr_debug("%s: Check PCIe link for %s ...\n",
+		 __func__, pci_name(pdev));
+
+	/* Check slot status */
+	cap = pdev->pcie_cap;
+	eeh_ops->read_config(dn, cap + PCI_EXP_SLTSTA, 2, &val);
+	if (!(val & PCI_EXP_SLTSTA_PDS)) {
+		pr_debug("  No card in the slot (0x%04x) !\n", val);
+		return;
+	}
+
+	/* Check power status if we have the capability */
+	eeh_ops->read_config(dn, cap + PCI_EXP_SLTCAP, 2, &val);
+	if (val & PCI_EXP_SLTCAP_PCP) {
+		eeh_ops->read_config(dn, cap + PCI_EXP_SLTCTL, 2, &val);
+		if (val & PCI_EXP_SLTCTL_PCC) {
+			pr_debug("  In power-off state, power it on ...\n");
+			val &= ~(PCI_EXP_SLTCTL_PCC | PCI_EXP_SLTCTL_PIC);
+			val |= (0x0100 & PCI_EXP_SLTCTL_PIC);
+			eeh_ops->write_config(dn, cap + PCI_EXP_SLTCTL, 2, val);
+			msleep(2 * 1000);
+		}
+	}
+
+	/* Enable link */
+	eeh_ops->read_config(dn, cap + PCI_EXP_LNKCTL, 2, &val);
+	val &= ~PCI_EXP_LNKCTL_LD;
+	eeh_ops->write_config(dn, cap + PCI_EXP_LNKCTL, 2, val);
+
+	/* Check link */
+	eeh_ops->read_config(dn, cap + PCI_EXP_LNKCAP, 4, &val);
+	if (!(val & PCI_EXP_LNKCAP_DLLLARC)) {
+		pr_debug("  No link reporting capability (0x%08x) \n", val);
+		msleep(1000);
+		return;
+	}
+
+	/* Wait the link is up until timeout (5s) */
+	timeout = 0;
+	while (timeout < 5000) {
+		msleep(20);
+		timeout += 20;
+
+		eeh_ops->read_config(dn, cap + PCI_EXP_LNKSTA, 2, &val);
+		if (val & PCI_EXP_LNKSTA_DLLLA)
+			break;
+	}
+
+	if (val & PCI_EXP_LNKSTA_DLLLA)
+		pr_debug("  Link up (%s)\n",
+			 (val & PCI_EXP_LNKSTA_CLS_2_5GB) ? "2.5GB" : "5GB");
+	else
+		pr_debug("  Link not ready (0x%04x)\n", val);
+}
+
 /**
  * eeh_restore_one_device_bars - Restore the Base Address Registers for one device
  * @data: EEH device
@@ -580,9 +663,17 @@ static void *eeh_restore_one_device_bars(void *data, void *flag)
 {
 	int i;
 	u32 cmd;
+	struct pci_dev *pdev = NULL;
 	struct eeh_dev *edev = (struct eeh_dev *)data;
 	struct device_node *dn = eeh_dev_to_of_node(edev);
 
+	/* Trace the PCI bridge */
+	if (eeh_probe_mode_dev()) {
+		pdev = eeh_dev_to_pci_dev(edev);
+		if (pdev->hdr_type != PCI_HEADER_TYPE_BRIDGE)
+			pdev = NULL;
+	}
+
 	for (i = 4; i < 10; i++)
 		eeh_ops->write_config(dn, i*4, 4, edev->config_space[i]);
 	/* 12 == Expansion ROM Address */
@@ -603,16 +694,25 @@ static void *eeh_restore_one_device_bars(void *data, void *flag)
 	 * Restore PERR & SERR bits, some devices require it,
 	 * don't touch the other command bits
 	 */
-	eeh_ops->read_config(dn, PCI_COMMAND, 4, &cmd);
-	if (edev->config_space[1] & PCI_COMMAND_PARITY)
-		cmd |= PCI_COMMAND_PARITY;
-	else
-		cmd &= ~PCI_COMMAND_PARITY;
-	if (edev->config_space[1] & PCI_COMMAND_SERR)
-		cmd |= PCI_COMMAND_SERR;
-	else
-		cmd &= ~PCI_COMMAND_SERR;
-	eeh_ops->write_config(dn, PCI_COMMAND, 4, cmd);
+	if (pdev) {
+		eeh_ops->write_config(dn, PCI_COMMAND, 4,
+				      edev->config_space[1]);
+	} else {
+		eeh_ops->read_config(dn, PCI_COMMAND, 4, &cmd);
+		if (edev->config_space[1] & PCI_COMMAND_PARITY)
+			cmd |= PCI_COMMAND_PARITY;
+		else
+			cmd &= ~PCI_COMMAND_PARITY;
+		if (edev->config_space[1] & PCI_COMMAND_SERR)
+			cmd |= PCI_COMMAND_SERR;
+		else
+			cmd &= ~PCI_COMMAND_SERR;
+		eeh_ops->write_config(dn, PCI_COMMAND, 4, cmd);
+	}
+
+	/* Check the PCIe link for bridge */
+	if (pdev)
+		eeh_bridge_check_link(dn, pdev);
 
 	return NULL;
 }
-- 
1.7.5.4

[PATCH 04/10] powerpc/eeh: Backends to get/set settings

[Gavin Shan]

When the PHB gets fenced, 0xFF's returns from PCI config space and
MMIO space in the hardware. The operations writting to them should
be dropped. The patch introduce backends allow to set/get flags that
indicate the access to PCI-CFG and MMIO should be blocked.

Signed-off-by: Gavin Shan <shangw@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/eeh.h               |    6 +++
 arch/powerpc/platforms/pseries/eeh_pseries.c |   44 ++++++++++++++++++++++++++
 2 files changed, 50 insertions(+), 0 deletions(-)

diff --git a/arch/powerpc/include/asm/eeh.h b/arch/powerpc/include/asm/eeh.h
index dd65e31..de821c1 100644
--- a/arch/powerpc/include/asm/eeh.h
+++ b/arch/powerpc/include/asm/eeh.h
@@ -131,6 +131,10 @@ static inline struct pci_dev *eeh_dev_to_pci_dev(struct eeh_dev *edev)
 #define EEH_LOG_TEMP		1	/* EEH temporary error log	*/
 #define EEH_LOG_PERM		2	/* EEH permanent error log	*/
 
+/* Settings for platforms */
+#define EEH_SETTING_BLOCK_CFG	1	/* Blocked PCI config access	*/
+#define EEH_SETTING_BLOCK_IO	2	/* Blocked MMIO access		*/
+
 struct eeh_ops {
 	char *name;
 	int (*init)(void);
@@ -146,6 +150,8 @@ struct eeh_ops {
 	int (*configure_bridge)(struct eeh_pe *pe);
 	int (*read_config)(struct device_node *dn, int where, int size, u32 *val);
 	int (*write_config)(struct device_node *dn, int where, int size, u32 val);
+	int (*get_setting)(int option, int *value, void *data);
+	int (*set_setting)(int option, int value, void *data);
 	int (*next_error)(struct eeh_pe **pe);
 };
 
diff --git a/arch/powerpc/platforms/pseries/eeh_pseries.c b/arch/powerpc/platforms/pseries/eeh_pseries.c
index 62415f2..8c9509b 100644
--- a/arch/powerpc/platforms/pseries/eeh_pseries.c
+++ b/arch/powerpc/platforms/pseries/eeh_pseries.c
@@ -612,6 +612,48 @@ static int pseries_eeh_write_config(struct device_node *dn, int where, int size,
 	return rtas_write_config(pdn, where, size, val);
 }
 
+/**
+ * pseries_eeh_get_setting - Retrieve settings that affect EEH core
+ * @option: option
+ * @value: value
+ * @data: dependent data
+ *
+ * Retrieve the settings from the platform in order to affect the
+ * behaviour of EEH core. We don't block PCI config or MMIO access
+ * on pSeries platform.
+ */
+static int pseries_eeh_get_setting(int option, int *value, void *data)
+{
+	int ret = 0;
+
+	switch (option) {
+	case EEH_SETTING_BLOCK_CFG:
+	case EEH_SETTING_BLOCK_IO:
+		*value = 0;
+		break;
+	default:
+		pr_warning("%s: Unrecognized option (%d)\n",
+			   __func__, option);
+		ret = -EINVAL;
+	}
+
+	return ret;
+}
+
+/**
+ * pseries_eeh_set_setting - Configure settings to affect EEH core
+ * @option: option
+ * @value: value
+ * @data: dependent data
+ *
+ * Configure the settings for the platform in order to affect the
+ * behaviour of EEH core.
+ */
+static int pseries_eeh_set_setting(int option, int value, void *data)
+{
+	return 0;
+}
+
 static struct eeh_ops pseries_eeh_ops = {
 	.name			= "pseries",
 	.init			= pseries_eeh_init,
@@ -626,6 +668,8 @@ static struct eeh_ops pseries_eeh_ops = {
 	.configure_bridge       = pseries_eeh_configure_bridge,
 	.read_config		= pseries_eeh_read_config,
 	.write_config		= pseries_eeh_write_config,
+	.get_setting		= pseries_eeh_get_setting,
+	.set_setting		= pseries_eeh_set_setting,
 	.next_error		= NULL
 };
 
-- 
1.7.5.4

[PATCH 05/10] powerpc/powernv: Support set/get EEH settings

[Gavin Shan]

The patch implements PowerNV backends to support set/get settings.
Also, we needn't maintain multiple fields in "struct pnv_phb" to
trace different EEH states. The patch merges all EEH states to one
field "eeh_state".

Signed-off-by: Gavin Shan <shangw@linux.vnet.ibm.com>
---
 arch/powerpc/platforms/powernv/eeh-ioda.c    |   82 ++++++++++++++++++++++++-
 arch/powerpc/platforms/powernv/eeh-powernv.c |   34 +++++++++++
 arch/powerpc/platforms/powernv/pci.c         |    4 +-
 arch/powerpc/platforms/powernv/pci.h         |   12 +++-
 4 files changed, 124 insertions(+), 8 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/eeh-ioda.c b/arch/powerpc/platforms/powernv/eeh-ioda.c
index 84f3036..64c3d1e 100644
--- a/arch/powerpc/platforms/powernv/eeh-ioda.c
+++ b/arch/powerpc/platforms/powernv/eeh-ioda.c
@@ -132,7 +132,7 @@ static int ioda_eeh_post_init(struct pci_controller *hose)
 					    &ioda_eeh_dbgfs_ops);
 #endif
 
-		phb->eeh_enabled = 1;
+		phb->eeh_state |= PNV_EEH_STATE_ENABLED;
 	}
 
 	return 0;
@@ -583,6 +583,78 @@ static int ioda_eeh_configure_bridge(struct eeh_pe *pe)
 	return 0;
 }
 
+/**
+ * ioda_eeh_set_setting - Configure the settings to affect EEH core
+ * @option: option
+ * @value: value
+ * @data: dependent data
+ *
+ * Configure the settings to affect EEH core.
+ */
+static int ioda_eeh_set_setting(int option, int value, void *data)
+{
+	struct pci_controller *hose = (struct pci_controller *)data;
+	struct pnv_phb *phb = hose->private_data;
+	int ret = 0;
+
+	switch (option) {
+	case EEH_SETTING_BLOCK_CFG:
+		if (value)
+			phb->eeh_state |= PNV_EEH_STATE_CFG_BLOCKED;
+		else
+			phb->eeh_state &= ~PNV_EEH_STATE_CFG_BLOCKED;
+		break;
+	case EEH_SETTING_BLOCK_IO:
+		if (value)
+			phb->eeh_state |= PNV_EEH_STATE_IO_BLOCKED;
+		else
+			phb->eeh_state &= ~PNV_EEH_STATE_IO_BLOCKED;
+		break;
+	default:
+		pr_warning("%s: Unrecognized option (%d)\n",
+			   __func__, option);
+		ret = -EINVAL;
+	}
+
+	return ret;
+}
+
+/**
+ * ioda_eeh_get_setting - Retrieve the settings to affect EEH core
+ * @option: option
+ * @value: value
+ * @data: dependent data
+ *
+ * EEH core retrieves the settings and utilize them.
+ */
+static int ioda_eeh_get_setting(int option, int *value, void *data)
+{
+	struct pci_controller *hose = (struct pci_controller *)data;
+	struct pnv_phb *phb = hose->private_data;
+	int ret = 0;
+
+	switch (option) {
+	case EEH_SETTING_BLOCK_CFG:
+		if (phb->eeh_state & PNV_EEH_STATE_CFG_BLOCKED)
+			*value = 1;
+		else
+			*value = 0;
+		break;
+	case EEH_SETTING_BLOCK_IO:
+		if (phb->eeh_state & PNV_EEH_STATE_IO_BLOCKED)
+			*value = 1;
+		else
+			*value = 0;
+		break;
+	default:
+		pr_warning("%s: Unrecognized option (%d)\n",
+			   __func__, option);
+		ret = -EINVAL;
+	}
+
+	return ret;
+}
+
 static void ioda_eeh_hub_diag_common(struct OpalIoP7IOCErrorData *data)
 {
 	/* GEM */
@@ -815,7 +887,7 @@ static int ioda_eeh_next_error(struct eeh_pe **pe)
 		 * removed, we needn't take care of it any more.
 		 */
 		phb = hose->private_data;
-		if (phb->removed)
+		if (phb->eeh_state & PNV_EEH_STATE_REMOVED)
 			continue;
 
 		rc = opal_pci_next_error(phb->opal_id,
@@ -850,7 +922,7 @@ static int ioda_eeh_next_error(struct eeh_pe **pe)
 				list_for_each_entry_safe(hose, tmp,
 						&hose_list, list_node) {
 					phb = hose->private_data;
-					phb->removed = 1;
+					phb->eeh_state |= PNV_EEH_STATE_REMOVED;
 				}
 
 				WARN(1, "EEH: dead IOC detected\n");
@@ -867,7 +939,7 @@ static int ioda_eeh_next_error(struct eeh_pe **pe)
 
 				WARN(1, "EEH: dead PHB#%x detected\n",
 				     hose->global_number);
-				phb->removed = 1;
+				phb->eeh_state |= PNV_EEH_STATE_REMOVED;
 				ret = 3;
 				goto out;
 			} else if (severity == OPAL_EEH_SEV_PHB_FENCED) {
@@ -905,5 +977,7 @@ struct pnv_eeh_ops ioda_eeh_ops = {
 	.reset			= ioda_eeh_reset,
 	.get_log		= ioda_eeh_get_log,
 	.configure_bridge	= ioda_eeh_configure_bridge,
+	.set_setting		= ioda_eeh_set_setting,
+	.get_setting		= ioda_eeh_get_setting,
 	.next_error		= ioda_eeh_next_error
 };
diff --git a/arch/powerpc/platforms/powernv/eeh-powernv.c b/arch/powerpc/platforms/powernv/eeh-powernv.c
index 9559115..cac5e18 100644
--- a/arch/powerpc/platforms/powernv/eeh-powernv.c
+++ b/arch/powerpc/platforms/powernv/eeh-powernv.c
@@ -355,6 +355,38 @@ static int powernv_eeh_write_config(struct device_node *dn, int where,
 }
 
 /**
+ * powernv_eeh_set_setting - Configure setting to affect EEH core
+ * @option: option
+ * @value: value
+ * @data: option dependent data
+ *
+ * Configure setting to affect the behaviour of EEH core.
+ */
+static int powernv_eeh_set_setting(int option, int value, void *data)
+{
+	struct pci_controller *hose = data;
+	struct pnv_phb *phb = hose->private_data;
+
+	return phb->set_setting(option, value, data);
+}
+
+/**
+ * powernv_eeh_get_setting - Retrieve settings to affect EEH core
+ * @option: option
+ * @value: value
+ * @data: option dependent data
+ *
+ * Retrieve setting to affect the behaviour of EEH core
+ */
+static int powernv_eeh_get_setting(int option, int *value, void *data)
+{
+	struct pci_controller *hose = data;
+	struct pnv_phb *phb = hose->private_data;
+
+	return phb->get_setting(option, value, data);
+}
+
+/**
  * powernv_eeh_next_error - Retrieve next EEH error to handle
  * @pe: Affected PE
  *
@@ -391,6 +423,8 @@ static struct eeh_ops powernv_eeh_ops = {
 	.configure_bridge       = powernv_eeh_configure_bridge,
 	.read_config            = powernv_eeh_read_config,
 	.write_config           = powernv_eeh_write_config,
+	.set_setting		= powernv_eeh_set_setting,
+	.get_setting		= powernv_eeh_get_setting,
 	.next_error		= powernv_eeh_next_error
 };
 
diff --git a/arch/powerpc/platforms/powernv/pci.c b/arch/powerpc/platforms/powernv/pci.c
index 6d9a506..1f31826 100644
--- a/arch/powerpc/platforms/powernv/pci.c
+++ b/arch/powerpc/platforms/powernv/pci.c
@@ -308,7 +308,7 @@ static int pnv_pci_read_config(struct pci_bus *bus,
 	if (phb_pe && (phb_pe->state & EEH_PE_ISOLATED))
 		return PCIBIOS_SUCCESSFUL;
 
-	if (phb->eeh_enabled) {
+	if (phb->eeh_state & PNV_EEH_STATE_ENABLED) {
 		if (*val == EEH_IO_ERROR_VALUE(size)) {
 			busdn = pci_bus_to_OF_node(bus);
 			for (dn = busdn->child; dn; dn = dn->sibling) {
@@ -358,7 +358,7 @@ static int pnv_pci_write_config(struct pci_bus *bus,
 
 	/* Check if the PHB got frozen due to an error (no response) */
 #ifdef CONFIG_EEH
-	if (!phb->eeh_enabled)
+	if (!(phb->eeh_state & PNV_EEH_STATE_ENABLED))
 		pnv_pci_config_check_eeh(phb, bus, bdfn);
 #else
 	pnv_pci_config_check_eeh(phb, bus, bdfn);
diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
index 43906e3..a281a1c 100644
--- a/arch/powerpc/platforms/powernv/pci.h
+++ b/arch/powerpc/platforms/powernv/pci.h
@@ -76,8 +76,17 @@ struct pnv_eeh_ops {
 	int (*get_log)(struct eeh_pe *pe, int severity,
 		       char *drv_log, unsigned long len);
 	int (*configure_bridge)(struct eeh_pe *pe);
+	int (*set_setting)(int option, int value, void *data);
+	int (*get_setting)(int option, int *value, void *data);
 	int (*next_error)(struct eeh_pe **pe);
 };
+
+/* EEH states maintained by PCI hose */
+#define PNV_EEH_STATE_ENABLED		(1 << 0)	/* EEH enabled	*/
+#define PNV_EEH_STATE_REMOVED		(1 << 1)	/* PHB removed	*/
+#define PNV_EEH_STATE_CFG_BLOCKED	(1 << 2)	/* PHB PCI-CFG blocked */
+#define PNV_EEH_STATE_IO_BLOCKED	(1 << 3)	/* PHB MMIO blocked */
+
 #endif /* CONFIG_EEH */
 
 struct pnv_phb {
@@ -92,8 +101,7 @@ struct pnv_phb {
 
 #ifdef CONFIG_EEH
 	struct pnv_eeh_ops	*eeh_ops;
-	int			eeh_enabled;
-	int			removed;
+	int			eeh_state;
 #endif
 
 #ifdef CONFIG_DEBUG_FS
-- 
1.7.5.4

[PATCH 06/10] powerpc/eeh: Support blocked IO access

[Gavin Shan]

The patch intends to support blocking IO access. Basically, if
the EEH core detects that the IO access has been blocked on one
specific PHB, we will simply return 0xFF's for reading and drop
writing.

Signed-off-by: Gavin Shan <shangw@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/eeh.h               |  231 +++++++++++++++++++++-----
 arch/powerpc/include/asm/io.h                |   67 +++++---
 arch/powerpc/kernel/eeh.c                    |   50 ++++--
 arch/powerpc/platforms/powernv/eeh-powernv.c |    4 +-
 4 files changed, 269 insertions(+), 83 deletions(-)

diff --git a/arch/powerpc/include/asm/eeh.h b/arch/powerpc/include/asm/eeh.h
index de821c1..a8dd983 100644
--- a/arch/powerpc/include/asm/eeh.h
+++ b/arch/powerpc/include/asm/eeh.h
@@ -211,7 +211,9 @@ void eeh_dev_phb_init_dynamic(struct pci_controller *phb);
 int __init eeh_init(void);
 int __init eeh_ops_register(struct eeh_ops *ops);
 int __exit eeh_ops_unregister(const char *name);
-unsigned long eeh_check_failure(const volatile void __iomem *token,
+int eeh_check_blocked_io(const volatile void __iomem *token,
+			 void **pedev);
+unsigned long eeh_check_failure(struct eeh_dev *edev,
 				unsigned long val);
 int eeh_dev_check_failure(struct eeh_dev *edev);
 void __init eeh_addr_cache_build(void);
@@ -249,7 +251,13 @@ static inline void *eeh_dev_init(struct device_node *dn, void *data)
 
 static inline void eeh_dev_phb_init_dynamic(struct pci_controller *phb) { }
 
-static inline unsigned long eeh_check_failure(const volatile void __iomem *token, unsigned long val)
+int eeh_check_blocked_io(const volatile void __iomem *token,
+			 void **pedev)
+{
+	return 0;
+}
+
+static inline unsigned long eeh_check_failure(void *data, unsigned long val)
 {
 	return val;
 }
@@ -276,57 +284,99 @@ static inline void eeh_remove_bus_device(struct pci_dev *dev, int purge_pe) { }
  */
 static inline u8 eeh_readb(const volatile void __iomem *addr)
 {
-	u8 val = in_8(addr);
-	if (EEH_POSSIBLE_ERROR(val, u8))
-		return eeh_check_failure(addr, val);
+	u8 val = 0xFF;
+	void *edev;
+
+	if (!eeh_check_blocked_io(addr, &edev)) {
+		val = in_8(addr);
+		if (EEH_POSSIBLE_ERROR(val, u8))
+			return eeh_check_failure(edev, val);
+	}
+
 	return val;
 }
 
 static inline u16 eeh_readw(const volatile void __iomem *addr)
 {
-	u16 val = in_le16(addr);
-	if (EEH_POSSIBLE_ERROR(val, u16))
-		return eeh_check_failure(addr, val);
+	u16 val = 0xFFFF;
+	void *edev;
+
+	if (!eeh_check_blocked_io(addr, &edev)) {
+		val = in_le16(addr);
+		if (EEH_POSSIBLE_ERROR(val, u16))
+			return eeh_check_failure(edev, val);
+	}
+
 	return val;
 }
 
 static inline u32 eeh_readl(const volatile void __iomem *addr)
 {
-	u32 val = in_le32(addr);
-	if (EEH_POSSIBLE_ERROR(val, u32))
-		return eeh_check_failure(addr, val);
+	u32 val = 0xFFFFFFFF;
+	void *edev;
+
+	if (!eeh_check_blocked_io(addr, &edev)) {
+		val = in_le32(addr);
+		if (EEH_POSSIBLE_ERROR(val, u32))
+			return eeh_check_failure(edev, val);
+	}
+
 	return val;
 }
 
 static inline u64 eeh_readq(const volatile void __iomem *addr)
 {
-	u64 val = in_le64(addr);
-	if (EEH_POSSIBLE_ERROR(val, u64))
-		return eeh_check_failure(addr, val);
+	u64 val = 0xFFFFFFFFFFFFFFFF;
+	void *edev;
+
+	if (!eeh_check_blocked_io(addr, &edev)) {
+		val = in_le64(addr);
+		if (EEH_POSSIBLE_ERROR(val, u64))
+			return eeh_check_failure(edev, val);
+	}
+
 	return val;
 }
 
 static inline u16 eeh_readw_be(const volatile void __iomem *addr)
 {
-	u16 val = in_be16(addr);
-	if (EEH_POSSIBLE_ERROR(val, u16))
-		return eeh_check_failure(addr, val);
+	u16 val = 0xFFFF;
+	void *edev;
+
+	if (!eeh_check_blocked_io(addr, &edev)) {
+		val = in_be16(addr);
+		if (EEH_POSSIBLE_ERROR(val, u16))
+			return eeh_check_failure(edev, val);
+	}
+
 	return val;
 }
 
 static inline u32 eeh_readl_be(const volatile void __iomem *addr)
 {
-	u32 val = in_be32(addr);
-	if (EEH_POSSIBLE_ERROR(val, u32))
-		return eeh_check_failure(addr, val);
+	u32 val = 0xFFFFFFFF;
+	void *edev;
+
+	if (!eeh_check_blocked_io(addr, &edev)) {
+		val = in_be32(addr);
+		if (EEH_POSSIBLE_ERROR(val, u32))
+			return eeh_check_failure(edev, val);
+	}
+
 	return val;
 }
 
 static inline u64 eeh_readq_be(const volatile void __iomem *addr)
 {
-	u64 val = in_be64(addr);
-	if (EEH_POSSIBLE_ERROR(val, u64))
-		return eeh_check_failure(addr, val);
+	u64 val = 0xFFFFFFFFFFFFFFFF;
+	void *edev;
+
+	if (!eeh_check_blocked_io(addr, &edev)) {
+		val = in_be64(addr);
+		if (EEH_POSSIBLE_ERROR(val, u64))
+			return eeh_check_failure(edev, val);
+	}
+
 	return val;
 }
 
@@ -334,40 +384,145 @@ static inline void eeh_memcpy_fromio(void *dest, const
 				     volatile void __iomem *src,
 				     unsigned long n)
 {
-	_memcpy_fromio(dest, src, n);
+	void *edev;
+
+	memset(dest, 0xFF, n);
+
+	if (!eeh_check_blocked_io(src, &edev)) {
+		_memcpy_fromio(dest, src, n);
 
-	/* Look for ffff's here at dest[n].  Assume that at least 4 bytes
-	 * were copied. Check all four bytes.
-	 */
-	if (n >= 4 && EEH_POSSIBLE_ERROR(*((u32 *)(dest + n - 4)), u32))
-		eeh_check_failure(src, *((u32 *)(dest + n - 4)));
+		/*
+		 * Look for ffff's here at dest[n]. Assume that at
+		 * least 4 bytes were copied. Check all four bytes.
+		 */
+		if (n >= 4 && EEH_POSSIBLE_ERROR(*((u32 *)(dest + n - 4)), u32))
+			eeh_check_failure(edev, *((u32 *)(dest + n - 4)));
+	}
 }
 
 /* in-string eeh macros */
 static inline void eeh_readsb(const volatile void __iomem *addr, void * buf,
 			      int ns)
 {
-	_insb(addr, buf, ns);
-	if (EEH_POSSIBLE_ERROR((*(((u8*)buf)+ns-1)), u8))
-		eeh_check_failure(addr, *(u8*)buf);
+	void *edev;
+
+	memset(buf, 0xFF, ns);
+
+	if (!eeh_check_blocked_io(addr, &edev)) {
+		_insb(addr, buf, ns);
+		if (EEH_POSSIBLE_ERROR((*(((u8*)buf)+ns-1)), u8))
+			eeh_check_failure(edev, *(u8*)buf);
+	}
 }
 
 static inline void eeh_readsw(const volatile void __iomem *addr, void * buf,
 			      int ns)
 {
-	_insw(addr, buf, ns);
-	if (EEH_POSSIBLE_ERROR((*(((u16*)buf)+ns-1)), u16))
-		eeh_check_failure(addr, *(u16*)buf);
+	void *edev;
+
+	memset(buf, 0xFF, ns * sizeof(u16));
+
+	if (!eeh_check_blocked_io(addr, &edev)) {
+		_insw(addr, buf, ns);
+		if (EEH_POSSIBLE_ERROR((*(((u16*)buf)+ns-1)), u16))
+			eeh_check_failure(edev, *(u16*)buf);
+	}
 }
 
 static inline void eeh_readsl(const volatile void __iomem *addr, void * buf,
 			      int nl)
 {
-	_insl(addr, buf, nl);
-	if (EEH_POSSIBLE_ERROR((*(((u32*)buf)+nl-1)), u32))
-		eeh_check_failure(addr, *(u32*)buf);
+	void *edev;
+
+	memset(buf, 0xFF, nl * sizeof(u32));
+
+	if (!eeh_check_blocked_io(addr, &edev)) {
+		_insl(addr, buf, nl);
+		if (EEH_POSSIBLE_ERROR((*(((u32*)buf)+nl-1)), u32))
+			eeh_check_failure(edev, *(u32*)buf);
+	}
+}
+
+/* MMIO write */
+static inline void eeh_writeb(volatile u8 __iomem *addr, u8 val)
+{
+	if (!eeh_check_blocked_io(addr, NULL))
+		out_8(addr, val);
+}
+
+static inline void eeh_writew(volatile u16 __iomem *addr, u16 val)
+{
+	if (!eeh_check_blocked_io(addr, NULL))
+		out_le16(addr, val);
 }
 
+static inline void eeh_writel(volatile u32 __iomem *addr, u32 val)
+{
+	if (!eeh_check_blocked_io(addr, NULL))
+		out_le32(addr, val);
+}
+
+static inline void eeh_writeq(volatile u64 __iomem *addr, u64 val)
+{
+	if (!eeh_check_blocked_io(addr, NULL))
+		out_le64(addr, val);
+}
+
+static inline void eeh_writew_be(volatile u16 __iomem *addr, u16 val)
+{
+	if (!eeh_check_blocked_io(addr, NULL))
+		out_be16(addr, val);
+}
+
+static inline void eeh_writel_be(volatile u32 __iomem *addr, u32 val)
+{
+	if (!eeh_check_blocked_io(addr, NULL))
+		out_be32(addr, val);
+}
+
+static inline void eeh_writeq_be(volatile u64 __iomem *addr, u64 val)
+{
+	if (!eeh_check_blocked_io(addr, NULL))
+		out_be64(addr, val);
+}
+
+static inline void eeh_writesb(volatile u8 __iomem *addr,
+			       const void *buf, long count)
+{
+	if (!eeh_check_blocked_io(addr, NULL))
+		_outsb(addr, buf, count);
+}
+
+static inline void eeh_writesw(volatile u16 __iomem *addr,
+			       const void *buf, long count)
+{
+	if (!eeh_check_blocked_io(addr, NULL))
+		_outsw(addr, buf, count);
+}
+
+static inline void eeh_writesl(volatile u32 __iomem *addr,
+			       const void *buf, long count)
+{
+	if (!eeh_check_blocked_io(addr, NULL))
+		_outsl(addr, buf, count);
+}
+
+static inline void eeh_memset_io(volatile void __iomem *addr,
+				 int c, unsigned long n)
+{
+	if (!eeh_check_blocked_io(addr, NULL))
+		_memset_io(addr, c, n);
+}
+
+static inline void eeh_memcpy_toio(volatile void __iomem *dest,
+				   const void *src, unsigned long n)
+{
+	if (!eeh_check_blocked_io(dest, NULL))
+		_memcpy_toio(dest, src, n);
+}
+
+
+
 #endif /* CONFIG_PPC64 */
 #endif /* __KERNEL__ */
 #endif /* _POWERPC_EEH_H */
diff --git a/arch/powerpc/include/asm/io.h b/arch/powerpc/include/asm/io.h
index dd15e5e..fa58c45 100644
--- a/arch/powerpc/include/asm/io.h
+++ b/arch/powerpc/include/asm/io.h
@@ -412,30 +412,37 @@ __do_out_asm(_rec_outl, "stwbrx")
  * possible to hook directly at the toplevel PIO operation if they have to
  * be handled differently
  */
-#define __do_writeb(val, addr)	out_8(PCI_FIX_ADDR(addr), val)
-#define __do_writew(val, addr)	out_le16(PCI_FIX_ADDR(addr), val)
-#define __do_writel(val, addr)	out_le32(PCI_FIX_ADDR(addr), val)
-#define __do_writeq(val, addr)	out_le64(PCI_FIX_ADDR(addr), val)
-#define __do_writew_be(val, addr) out_be16(PCI_FIX_ADDR(addr), val)
-#define __do_writel_be(val, addr) out_be32(PCI_FIX_ADDR(addr), val)
-#define __do_writeq_be(val, addr) out_be64(PCI_FIX_ADDR(addr), val)
 
 #ifdef CONFIG_EEH
-#define __do_readb(addr)	eeh_readb(PCI_FIX_ADDR(addr))
-#define __do_readw(addr)	eeh_readw(PCI_FIX_ADDR(addr))
-#define __do_readl(addr)	eeh_readl(PCI_FIX_ADDR(addr))
-#define __do_readq(addr)	eeh_readq(PCI_FIX_ADDR(addr))
-#define __do_readw_be(addr)	eeh_readw_be(PCI_FIX_ADDR(addr))
-#define __do_readl_be(addr)	eeh_readl_be(PCI_FIX_ADDR(addr))
-#define __do_readq_be(addr)	eeh_readq_be(PCI_FIX_ADDR(addr))
+#define __do_readb(addr)		eeh_readb(PCI_FIX_ADDR(addr))
+#define __do_readw(addr)		eeh_readw(PCI_FIX_ADDR(addr))
+#define __do_readl(addr)		eeh_readl(PCI_FIX_ADDR(addr))
+#define __do_readq(addr)		eeh_readq(PCI_FIX_ADDR(addr))
+#define __do_readw_be(addr)		eeh_readw_be(PCI_FIX_ADDR(addr))
+#define __do_readl_be(addr)		eeh_readl_be(PCI_FIX_ADDR(addr))
+#define __do_readq_be(addr)		eeh_readq_be(PCI_FIX_ADDR(addr))
+#define __do_writeb(val, addr)		eeh_writeb(PCI_FIX_ADDR(addr), val)
+#define __do_writew(val, addr)          eeh_writew(PCI_FIX_ADDR(addr), val)
+#define __do_writel(val, addr)          eeh_writel(PCI_FIX_ADDR(addr), val)
+#define __do_writeq(val, addr)          eeh_writeq(PCI_FIX_ADDR(addr), val)
+#define __do_writew_be(val, addr)       eeh_writew_be(PCI_FIX_ADDR(addr), val)
+#define __do_writel_be(val, addr)       eeh_writel_be(PCI_FIX_ADDR(addr), val)
+#define __do_writeq_be(val, addr)       eeh_writeq_be(PCI_FIX_ADDR(addr), val)
 #else /* CONFIG_EEH */
-#define __do_readb(addr)	in_8(PCI_FIX_ADDR(addr))
-#define __do_readw(addr)	in_le16(PCI_FIX_ADDR(addr))
-#define __do_readl(addr)	in_le32(PCI_FIX_ADDR(addr))
-#define __do_readq(addr)	in_le64(PCI_FIX_ADDR(addr))
-#define __do_readw_be(addr)	in_be16(PCI_FIX_ADDR(addr))
-#define __do_readl_be(addr)	in_be32(PCI_FIX_ADDR(addr))
-#define __do_readq_be(addr)	in_be64(PCI_FIX_ADDR(addr))
+#define __do_readb(addr)		in_8(PCI_FIX_ADDR(addr))
+#define __do_readw(addr)		in_le16(PCI_FIX_ADDR(addr))
+#define __do_readl(addr)		in_le32(PCI_FIX_ADDR(addr))
+#define __do_readq(addr)		in_le64(PCI_FIX_ADDR(addr))
+#define __do_readw_be(addr)		in_be16(PCI_FIX_ADDR(addr))
+#define __do_readl_be(addr)		in_be32(PCI_FIX_ADDR(addr))
+#define __do_readq_be(addr)		in_be64(PCI_FIX_ADDR(addr))
+#define __do_writeb(val, addr)		out_8(PCI_FIX_ADDR(addr), val)
+#define __do_writew(val, addr)		out_le16(PCI_FIX_ADDR(addr), val)
+#define __do_writel(val, addr)		out_le32(PCI_FIX_ADDR(addr), val)
+#define __do_writeq(val, addr)		out_le64(PCI_FIX_ADDR(addr), val)
+#define __do_writew_be(val, addr)	out_be16(PCI_FIX_ADDR(addr), val)
+#define __do_writel_be(val, addr)	out_be32(PCI_FIX_ADDR(addr), val)
+#define __do_writeq_be(val, addr)	out_be64(PCI_FIX_ADDR(addr), val)
 #endif /* !defined(CONFIG_EEH) */
 
 #ifdef CONFIG_PPC32
@@ -458,14 +465,17 @@ __do_out_asm(_rec_outl, "stwbrx")
 #define __do_readsb(a, b, n)	eeh_readsb(PCI_FIX_ADDR(a), (b), (n))
 #define __do_readsw(a, b, n)	eeh_readsw(PCI_FIX_ADDR(a), (b), (n))
 #define __do_readsl(a, b, n)	eeh_readsl(PCI_FIX_ADDR(a), (b), (n))
+#define __do_writesb(a, b, n)   eeh_writesb(PCI_FIX_ADDR(a),(b),(n))
+#define __do_writesw(a, b, n)   eeh_writesw(PCI_FIX_ADDR(a),(b),(n))
+#define __do_writesl(a, b, n)   eeh_writesl(PCI_FIX_ADDR(a),(b),(n))
 #else /* CONFIG_EEH */
 #define __do_readsb(a, b, n)	_insb(PCI_FIX_ADDR(a), (b), (n))
 #define __do_readsw(a, b, n)	_insw(PCI_FIX_ADDR(a), (b), (n))
 #define __do_readsl(a, b, n)	_insl(PCI_FIX_ADDR(a), (b), (n))
-#endif /* !CONFIG_EEH */
 #define __do_writesb(a, b, n)	_outsb(PCI_FIX_ADDR(a),(b),(n))
 #define __do_writesw(a, b, n)	_outsw(PCI_FIX_ADDR(a),(b),(n))
 #define __do_writesl(a, b, n)	_outsl(PCI_FIX_ADDR(a),(b),(n))
+#endif /* !CONFIG_EEH */
 
 #define __do_insb(p, b, n)	readsb((PCI_IO_ADDR)_IO_BASE+(p), (b), (n))
 #define __do_insw(p, b, n)	readsw((PCI_IO_ADDR)_IO_BASE+(p), (b), (n))
@@ -474,17 +484,20 @@ __do_out_asm(_rec_outl, "stwbrx")
 #define __do_outsw(p, b, n)	writesw((PCI_IO_ADDR)_IO_BASE+(p),(b),(n))
 #define __do_outsl(p, b, n)	writesl((PCI_IO_ADDR)_IO_BASE+(p),(b),(n))
 
-#define __do_memset_io(addr, c, n)	\
-				_memset_io(PCI_FIX_ADDR(addr), c, n)
-#define __do_memcpy_toio(dst, src, n)	\
-				_memcpy_toio(PCI_FIX_ADDR(dst), src, n)
-
 #ifdef CONFIG_EEH
 #define __do_memcpy_fromio(dst, src, n)	\
 				eeh_memcpy_fromio(dst, PCI_FIX_ADDR(src), n)
+#define __do_memset_io(addr, c, n)	\
+				eeh_memset_io(PCI_FIX_ADDR(addr), c, n)
+#define __do_memcpy_toio(dst, src, n)	\
+				eeh_memcpy_toio(PCI_FIX_ADDR(dst), src, n)
 #else /* CONFIG_EEH */
 #define __do_memcpy_fromio(dst, src, n)	\
 				_memcpy_fromio(dst,PCI_FIX_ADDR(src),n)
+#define __do_memset_io(addr, c, n)	\
+				_memset_io(PCI_FIX_ADDR(addr), c, n)
+#define __do_memcpy_toio(dst, src, n)	\
+				_memcpy_toio(PCI_FIX_ADDR(dst), src, n)
 #endif /* !CONFIG_EEH */
 
 #ifdef CONFIG_PPC_INDIRECT_PIO
diff --git a/arch/powerpc/kernel/eeh.c b/arch/powerpc/kernel/eeh.c
index 65320fd..b46f1ca 100644
--- a/arch/powerpc/kernel/eeh.c
+++ b/arch/powerpc/kernel/eeh.c
@@ -464,39 +464,57 @@ dn_unlock:
 EXPORT_SYMBOL_GPL(eeh_dev_check_failure);
 
 /**
- * eeh_check_failure - Check if all 1's data is due to EEH slot freeze
- * @token: I/O token, should be address in the form 0xA....
- * @val: value, should be all 1's (XXX why do we need this arg??)
- *
- * Check for an EEH failure at the given token address.  Call this
- * routine if the result of a read was all 0xff's and you want to
- * find out if this is due to an EEH slot freeze event.  This routine
- * will query firmware for the EEH status.
+ * eeh_check_blocked_io - Check if the I/O access has been blocked
+ * @token: I/O token
+ * @pedev: EEH device
  *
- * Note this routine is safe to call in an interrupt context.
+ * Check if the I/O access has been blocked. If that's the case, we
+ * should return 0xFF's on read, or sliently drop the write.
  */
-unsigned long eeh_check_failure(const volatile void __iomem *token, unsigned long val)
+int eeh_check_blocked_io(const volatile void __iomem *token,
+			 void **pedev)
 {
-	unsigned long addr;
 	struct eeh_dev *edev;
+	unsigned long addr;
+	int rc;
 
-	/* Finding the phys addr + pci device; this is pretty quick. */
+	/* Don't block I/O ranges which aren't traced by cache */
 	addr = eeh_token_to_phys((unsigned long __force) token);
 	edev = eeh_addr_cache_get_dev(addr);
 	if (!edev) {
+		if (pedev) *pedev = NULL;
 		eeh_stats.no_device++;
-		return val;
+		return 0;
 	}
 
-	eeh_dev_check_failure(edev);
+	if (pedev) *pedev = edev;
+	eeh_ops->get_setting(EEH_SETTING_BLOCK_IO, &rc, edev->phb);
+	return rc;
+}
+
+EXPORT_SYMBOL(eeh_check_blocked_io);
+
+/**
+ * eeh_check_failure - Check if all 1's data is due to EEH slot freeze
+ * @edev: EEH device
+ * @val: value
+ *
+ * Check if the EEH device has been frozen.
+ *
+ * Note this routine is safe to call in an interrupt context.
+ */
+unsigned long eeh_check_failure(struct eeh_dev *edev, unsigned long val)
+{
+	if (edev) {
+		eeh_dev_check_failure(edev);
+		pci_dev_put(eeh_dev_to_pci_dev(edev));
+	}
 
-	pci_dev_put(eeh_dev_to_pci_dev(edev));
 	return val;
 }
 
 EXPORT_SYMBOL(eeh_check_failure);
 
-
 /**
  * eeh_pci_enable - Enable MMIO or DMA transfers for this slot
  * @pe: EEH PE
diff --git a/arch/powerpc/platforms/powernv/eeh-powernv.c b/arch/powerpc/platforms/powernv/eeh-powernv.c
index cac5e18..20a7865 100644
--- a/arch/powerpc/platforms/powernv/eeh-powernv.c
+++ b/arch/powerpc/platforms/powernv/eeh-powernv.c
@@ -367,7 +367,7 @@ static int powernv_eeh_set_setting(int option, int value, void *data)
 	struct pci_controller *hose = data;
 	struct pnv_phb *phb = hose->private_data;
 
-	return phb->set_setting(option, value, data);
+	return phb->eeh_ops->set_setting(option, value, data);
 }
 
 /**
@@ -383,7 +383,7 @@ static int powernv_eeh_get_setting(int option, int *value, void *data)
 	struct pci_controller *hose = data;
 	struct pnv_phb *phb = hose->private_data;
 
-	return phb->get_setting(option, value, data);
+	return phb->eeh_ops->get_setting(option, value, data);
 }
 
 /**
-- 
1.7.5.4

[PATCH 07/10] powerpc/powernv: Block PCI-CFG access if necessary

[Gavin Shan]

If the PCI-CFG access on the specific PHB, to return 0xFF's for
reading and drop writing. The patch implements that for PowerNV
platform. The patch also removes the check on "hose == NULL"
for PCI-CFG accessors since the kernel should stop while fetching
platform-dependent PHB (struct pnv_phb).

Signed-off-by: Gavin Shan <shangw@linux.vnet.ibm.com>
---
 arch/powerpc/platforms/powernv/eeh-powernv.c |   10 ++---
 arch/powerpc/platforms/powernv/pci.c         |   59 ++++++++++++++++++++------
 arch/powerpc/platforms/powernv/pci.h         |    4 ++
 3 files changed, 54 insertions(+), 19 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/eeh-powernv.c b/arch/powerpc/platforms/powernv/eeh-powernv.c
index 20a7865..249798e 100644
--- a/arch/powerpc/platforms/powernv/eeh-powernv.c
+++ b/arch/powerpc/platforms/powernv/eeh-powernv.c
@@ -328,9 +328,9 @@ static int powernv_eeh_read_config(struct device_node *dn, int where,
 {
 	struct eeh_dev *edev = of_node_to_eeh_dev(dn);
 	struct pci_dev *dev = eeh_dev_to_pci_dev(edev);
-	struct pci_controller *hose = edev->phb;
 
-	return hose->ops->read(dev->bus, dev->devfn, where, size, val);
+	return pnv_pci_cfg_read(dev->bus, dev->devfn,
+				where, size, val, false);
 }
 
 /**
@@ -347,11 +347,9 @@ static int powernv_eeh_write_config(struct device_node *dn, int where,
 {
 	struct eeh_dev *edev = of_node_to_eeh_dev(dn);
 	struct pci_dev *dev = eeh_dev_to_pci_dev(edev);
-	struct pci_controller *hose = edev->phb;
 
-	hose = pci_bus_to_host(dev->bus);
-
-	return hose->ops->write(dev->bus, dev->devfn, where, size, val);
+	return pnv_pci_cfg_write(dev->bus, dev->devfn,
+				 where, size, val, false);
 }
 
 /**
diff --git a/arch/powerpc/platforms/powernv/pci.c b/arch/powerpc/platforms/powernv/pci.c
index 1f31826..47fa921 100644
--- a/arch/powerpc/platforms/powernv/pci.c
+++ b/arch/powerpc/platforms/powernv/pci.c
@@ -255,21 +255,30 @@ static void pnv_pci_config_check_eeh(struct pnv_phb *phb, struct pci_bus *bus,
 		pnv_pci_handle_eeh_config(phb, pe_no);
 }
 
-static int pnv_pci_read_config(struct pci_bus *bus,
-			       unsigned int devfn,
-			       int where, int size, u32 *val)
+int pnv_pci_cfg_read(struct pci_bus *bus,
+		     unsigned int devfn,
+		     int where, int size,
+		     u32 *val, bool check)
 {
 	struct pci_controller *hose = pci_bus_to_host(bus);
 	struct pnv_phb *phb = hose->private_data;
+	u32 bdfn = (((uint64_t)bus->number) << 8) | devfn;
+	s64 rc;
 #ifdef CONFIG_EEH
 	struct device_node *busdn, *dn;
 	struct eeh_pe *phb_pe = NULL;
-#endif
-	u32 bdfn = (((uint64_t)bus->number) << 8) | devfn;
-	s64 rc;
 
-	if (hose == NULL)
+	/*
+	 * If PCI-CFG access has been blocked, we simply
+	 * return 0xFF's here.
+	 */
+	if (check &&
+	    (phb->eeh_state & PNV_EEH_STATE_ENABLED) &&
+	    (phb->eeh_state & PNV_EEH_STATE_CFG_BLOCKED)) {
+		*val = 0xFFFFFFFF;
 		return PCIBIOS_DEVICE_NOT_FOUND;
+	}
+#endif
 
 	switch (size) {
 	case 1: {
@@ -329,19 +338,26 @@ static int pnv_pci_read_config(struct pci_bus *bus,
 	return PCIBIOS_SUCCESSFUL;
 }
 
-static int pnv_pci_write_config(struct pci_bus *bus,
-				unsigned int devfn,
-				int where, int size, u32 val)
+int pnv_pci_cfg_write(struct pci_bus *bus,
+		      unsigned int devfn,
+		      int where, int size,
+		      u32 val, bool check)
 {
 	struct pci_controller *hose = pci_bus_to_host(bus);
 	struct pnv_phb *phb = hose->private_data;
 	u32 bdfn = (((uint64_t)bus->number) << 8) | devfn;
 
-	if (hose == NULL)
-		return PCIBIOS_DEVICE_NOT_FOUND;
-
 	cfg_dbg("pnv_pci_write_config bus: %x devfn: %x +%x/%x -> %08x\n",
 		bus->number, devfn, where, size, val);
+
+#ifdef CONFIG_EEH
+	/* If PCI-CFG access has been blocked, drop it */
+	if (check &&
+	    (phb->eeh_state & PNV_EEH_STATE_ENABLED) &&
+	    (phb->eeh_state & PNV_EEH_STATE_CFG_BLOCKED))
+		return PCIBIOS_DEVICE_NOT_FOUND;
+#endif
+
 	switch (size) {
 	case 1:
 		opal_pci_config_write_byte(phb->opal_id, bdfn, where, val);
@@ -367,6 +383,23 @@ static int pnv_pci_write_config(struct pci_bus *bus,
 	return PCIBIOS_SUCCESSFUL;
 }
 
+static int pnv_pci_read_config(struct pci_bus *bus,
+			       unsigned int devfn,
+			       int where, int size, u32 *val)
+{
+	return pnv_pci_cfg_read(bus, devfn, where,
+				size, val, true);
+}
+
+static int pnv_pci_write_config(struct pci_bus *bus,
+				unsigned int devfn,
+				int where, int size,
+				u32 val)
+{
+	return pnv_pci_cfg_write(bus, devfn, where,
+				 size, val, true);
+}
+
 struct pci_ops pnv_pci_ops = {
 	.read = pnv_pci_read_config,
 	.write = pnv_pci_write_config,
diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
index a281a1c..8624f8f 100644
--- a/arch/powerpc/platforms/powernv/pci.h
+++ b/arch/powerpc/platforms/powernv/pci.h
@@ -187,6 +187,10 @@ extern struct pci_ops pnv_pci_ops;
 extern struct pnv_eeh_ops ioda_eeh_ops;
 #endif
 
+extern int pnv_pci_cfg_read(struct pci_bus *bus, unsigned int devfn,
+			    int where, int size, u32 *val, bool check);
+extern int pnv_pci_cfg_write(struct pci_bus *bus, unsigned int devfn,
+			     int where, int size, u32 val, bool check);
 extern void pnv_pci_setup_iommu_table(struct iommu_table *tbl,
 				      void *tce_mem, u64 tce_size,
 				      u64 dma_offset);
-- 
1.7.5.4

[PATCH 08/10] powerpc/powernv: Hold PCI-CFG and I/O access

[Gavin Shan]

While doing recovery from fenced PHB, we need hold the PCI-CFG and
I/O access until the complete PHB reset and BARs restore are done.
The patch addresses that.

Signed-off-by: Gavin Shan <shangw@linux.vnet.ibm.com>
---
 arch/powerpc/kernel/eeh_driver.c          |   11 +++++++++++
 arch/powerpc/platforms/powernv/eeh-ioda.c |   10 ++++++++--
 2 files changed, 19 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/kernel/eeh_driver.c b/arch/powerpc/kernel/eeh_driver.c
index 0974e13..944e225 100644
--- a/arch/powerpc/kernel/eeh_driver.c
+++ b/arch/powerpc/kernel/eeh_driver.c
@@ -349,12 +349,14 @@ static void *eeh_report_failure(void *data, void *userdata)
  */
 static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus)
 {
+	struct pci_controller *hose;
 	struct timeval tstamp;
 	int cnt, rc;
 
 	/* pcibios will clear the counter; save the value */
 	cnt = pe->freeze_count;
 	tstamp = pe->tstamp;
+	hose = (pe->type & EEH_PE_PHB) ? pe->phb : NULL;
 
 	/*
 	 * We don't remove the corresponding PE instances because
@@ -377,6 +379,15 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus)
 	eeh_ops->configure_bridge(pe);
 	eeh_pe_restore_bars(pe);
 
+	/*
+	 * If we're recovering fenced PHB, the PCI-CFG and I/O should
+	 * have been blocked. We need reenable that.
+	 */
+	if (hose) {
+		eeh_ops->set_setting(EEH_SETTING_BLOCK_CFG, 0, hose);
+		eeh_ops->set_setting(EEH_SETTING_BLOCK_IO,  0, hose);
+	}
+
 	/* Give the system 5 seconds to finish running the user-space
 	 * hotplug shutdown scripts, e.g. ifdown for ethernet.  Yes,
 	 * this is a hack, but if we don't do this, and try to bring
diff --git a/arch/powerpc/platforms/powernv/eeh-ioda.c b/arch/powerpc/platforms/powernv/eeh-ioda.c
index 64c3d1e..23c2442 100644
--- a/arch/powerpc/platforms/powernv/eeh-ioda.c
+++ b/arch/powerpc/platforms/powernv/eeh-ioda.c
@@ -922,7 +922,9 @@ static int ioda_eeh_next_error(struct eeh_pe **pe)
 				list_for_each_entry_safe(hose, tmp,
 						&hose_list, list_node) {
 					phb = hose->private_data;
-					phb->eeh_state |= PNV_EEH_STATE_REMOVED;
+					phb->eeh_state |= (PNV_EEH_STATE_REMOVED |
+							   PNV_EEH_STATE_IO_BLOCKED |
+							   PNV_EEH_STATE_IO_BLOCKED);
 				}
 
 				WARN(1, "EEH: dead IOC detected\n");
@@ -939,7 +941,9 @@ static int ioda_eeh_next_error(struct eeh_pe **pe)
 
 				WARN(1, "EEH: dead PHB#%x detected\n",
 				     hose->global_number);
-				phb->eeh_state |= PNV_EEH_STATE_REMOVED;
+				phb->eeh_state |= (PNV_EEH_STATE_REMOVED |
+						   PNV_EEH_STATE_CFG_BLOCKED |
+						   PNV_EEH_STATE_IO_BLOCKED);
 				ret = 3;
 				goto out;
 			} else if (severity == OPAL_EEH_SEV_PHB_FENCED) {
@@ -948,6 +952,8 @@ static int ioda_eeh_next_error(struct eeh_pe **pe)
 
 				WARN(1, "EEH: fenced PHB#%x detected\n",
 				     hose->global_number);
+				phb->eeh_state |= (PNV_EEH_STATE_CFG_BLOCKED |
+						   PNV_EEH_STATE_IO_BLOCKED);
 				ret = 2;
 				goto out;
 			} else if (severity == OPAL_EEH_SEV_INF)
-- 
1.7.5.4

[PATCH 09/10] powerpc/eeh: Fix address catch for PowerNV

[Gavin Shan]

On the PowerNV platform, the EEH address cache isn't built correctly
because we skipped the EEH devices without binding PE. The patch
fixes that.

Signed-off-by: Gavin Shan <shangw@linux.vnet.ibm.com>
---
 arch/powerpc/kernel/eeh_cache.c           |    2 +-
 arch/powerpc/platforms/powernv/pci-ioda.c |    1 +
 2 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/arch/powerpc/kernel/eeh_cache.c b/arch/powerpc/kernel/eeh_cache.c
index 1d5d9a6..858ebea 100644
--- a/arch/powerpc/kernel/eeh_cache.c
+++ b/arch/powerpc/kernel/eeh_cache.c
@@ -194,7 +194,7 @@ static void __eeh_addr_cache_insert_dev(struct pci_dev *dev)
 	}
 
 	/* Skip any devices for which EEH is not enabled. */
-	if (!edev->pe) {
+	if (!eeh_probe_mode_dev() && !edev->pe) {
 #ifdef DEBUG
 		pr_info("PCI: skip building address cache for=%s - %s\n",
 			pci_name(dev), dn->full_name);
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 3e5c3d5..0ff9a3a 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -998,6 +998,7 @@ static void pnv_pci_ioda_fixup(void)
 	pnv_pci_ioda_create_dbgfs();
 
 #ifdef CONFIG_EEH
+	eeh_probe_mode_set(EEH_PROBE_MODE_DEV);
 	eeh_addr_cache_build();
 	eeh_init();
 #endif
-- 
1.7.5.4

[PATCH 10/10] net/tg3: Avoid delay during MMIO access

[Gavin Shan]

When the driver is encountering EEH errors, which might be caused
by frozen PCI host controller, the driver needn't keep reading on
MMIO until timeout. For the case, 0xFF's should be returned from
hardware. Otherwise, it possibly trigger soft-lockup. The patch
adds more check on that by pci_channel_offline(), thus to avoid
the possible soft-lockup.

Signed-off-by: Gavin Shan <shangw@linux.vnet.ibm.com>
---
 drivers/net/ethernet/broadcom/tg3.c |   36 +++++++++++++++++++++++++++++++++++
 1 files changed, 36 insertions(+), 0 deletions(-)

diff --git a/drivers/net/ethernet/broadcom/tg3.c b/drivers/net/ethernet/broadcom/tg3.c
index c777b90..a13463e 100644
--- a/drivers/net/ethernet/broadcom/tg3.c
+++ b/drivers/net/ethernet/broadcom/tg3.c
@@ -744,6 +744,9 @@ static int tg3_ape_lock(struct tg3 *tp, int locknum)
 		status = tg3_ape_read32(tp, gnt + off);
 		if (status == bit)
 			break;
+		if (pci_channel_offline(tp->pdev))
+			break;
+
 		udelay(10);
 	}
 
@@ -1635,6 +1638,9 @@ static void tg3_wait_for_event_ack(struct tg3 *tp)
 	for (i = 0; i < delay_cnt; i++) {
 		if (!(tr32(GRC_RX_CPU_EVENT) & GRC_RX_CPU_DRIVER_EVENT))
 			break;
+		if (pci_channel_offline(tp->pdev))
+			break;
+
 		udelay(8);
 	}
 }
@@ -1813,6 +1819,9 @@ static int tg3_poll_fw(struct tg3 *tp)
 		for (i = 0; i < 200; i++) {
 			if (tr32(VCPU_STATUS) & VCPU_STATUS_INIT_DONE)
 				return 0;
+			if (pci_channel_offline(tp->pdev))
+				return -ENODEV;
+
 			udelay(100);
 		}
 		return -ENODEV;
@@ -1823,6 +1832,15 @@ static int tg3_poll_fw(struct tg3 *tp)
 		tg3_read_mem(tp, NIC_SRAM_FIRMWARE_MBOX, &val);
 		if (val == ~NIC_SRAM_FIRMWARE_MBOX_MAGIC1)
 			break;
+		if (pci_channel_offline(tp->pdev)) {
+			if (!tg3_flag(tp, NO_FWARE_REPORTED)) {
+				tg3_flag_set(tp, NO_FWARE_REPORTED);
+				netdev_info(tp->dev, "No firmware running\n");
+			}
+
+			break;
+		}
+
 		udelay(10);
 	}
 
@@ -3520,6 +3538,8 @@ static int tg3_pause_cpu(struct tg3 *tp, u32 cpu_base)
 		tw32(cpu_base + CPU_MODE,  CPU_MODE_HALT);
 		if (tr32(cpu_base + CPU_MODE) & CPU_MODE_HALT)
 			break;
+		if (pci_channel_offline(tp->pdev))
+			return -EBUSY;
 	}
 
 	return (i == iters) ? -EBUSY : 0;
@@ -8589,6 +8609,14 @@ static int tg3_stop_block(struct tg3 *tp, unsigned long ofs, u32 enable_bit, boo
 	tw32_f(ofs, val);
 
 	for (i = 0; i < MAX_WAIT_CNT; i++) {
+		if (pci_channel_offline(tp->pdev)) {
+			dev_err(&tp->pdev->dev,
+				"tg3_stop_block device offline, "
+				"ofs=%lx enable_bit=%x\n",
+				ofs, enable_bit);
+			return -ENODEV;
+		}
+
 		udelay(100);
 		val = tr32(ofs);
 		if ((val & enable_bit) == 0)
@@ -8612,6 +8640,13 @@ static int tg3_abort_hw(struct tg3 *tp, bool silent)
 
 	tg3_disable_ints(tp);
 
+	if (pci_channel_offline(tp->pdev)) {
+		tp->rx_mode &= ~(RX_MODE_ENABLE | TX_MODE_ENABLE);
+		tp->mac_mode &= ~MAC_MODE_TDE_ENABLE;
+		err = -ENODEV;
+		goto err_no_dev;
+	}
+
 	tp->rx_mode &= ~RX_MODE_ENABLE;
 	tw32_f(MAC_RX_MODE, tp->rx_mode);
 	udelay(10);
@@ -8660,6 +8695,7 @@ static int tg3_abort_hw(struct tg3 *tp, bool silent)
 	err |= tg3_stop_block(tp, BUFMGR_MODE, BUFMGR_MODE_ENABLE, silent);
 	err |= tg3_stop_block(tp, MEMARB_MODE, MEMARB_MODE_ENABLE, silent);
 
+err_no_dev:
 	for (i = 0; i < tp->irq_cnt; i++) {
 		struct tg3_napi *tnapi = &tp->napi[i];
 		if (tnapi->hw_status)
-- 
1.7.5.4

Re: [PATCH 03/10] powerpc/eeh: Check PCIe link after reset

[Benjamin Herrenschmidt]

On Tue, 2013-06-25 at 13:55 +0800, Gavin Shan wrote:
>         * don't touch the other command bits
>          */
> -       eeh_ops->read_config(dn, PCI_COMMAND, 4, &cmd);
> -       if (edev->config_space[1] & PCI_COMMAND_PARITY)
> -               cmd |= PCI_COMMAND_PARITY;
> -       else
> -               cmd &= ~PCI_COMMAND_PARITY;
> -       if (edev->config_space[1] & PCI_COMMAND_SERR)
> -               cmd |= PCI_COMMAND_SERR;
> -       else
> -               cmd &= ~PCI_COMMAND_SERR;
> -       eeh_ops->write_config(dn, PCI_COMMAND, 4, cmd);
> +       if (pdev) {
> +               eeh_ops->write_config(dn, PCI_COMMAND, 4,
> +                                     edev->config_space[1]);
> +       } else {

That needs a much better comment. Why are you doing that instead
of what's below ? In fact there is more to restore in a bridge
right ? (windows etc...). Do you do that ? Should we just have a
different function to restore a device vs. a bridge ?

I also don't see a need to do thing differently between phyp and
powernv. Bridges inside partitions would suffer the same fate in
both cases.

Ben.

> +               eeh_ops->read_config(dn, PCI_COMMAND, 4, &cmd);
> +               if (edev->config_space[1] & PCI_COMMAND_PARITY)
> +                       cmd |= PCI_COMMAND_PARITY;
> +               else
> +                       cmd &= ~PCI_COMMAND_PARITY;
> +               if (edev->config_space[1] & PCI_COMMAND_SERR)
> +                       cmd |= PCI_COMMAND_SERR;
> +               else
> +                       cmd &= ~PCI_COMMAND_SERR;
> +               eeh_ops->write_config(dn, PCI_COMMAND, 4, cmd);
> +       }
> +

Re: [PATCH 04/10] powerpc/eeh: Backends to get/set settings

[Benjamin Herrenschmidt]

On Tue, 2013-06-25 at 13:55 +0800, Gavin Shan wrote:
> When the PHB gets fenced, 0xFF's returns from PCI config space and
> MMIO space in the hardware. The operations writting to them should
> be dropped. The patch introduce backends allow to set/get flags that
> indicate the access to PCI-CFG and MMIO should be blocked.

We can't block MMIO without massive overhead. Config space can be
blocked inside the firmware, can't it ?

Cheers,
Ben.

> Signed-off-by: Gavin Shan <shangw@linux.vnet.ibm.com>
> ---
>  arch/powerpc/include/asm/eeh.h               |    6 +++
>  arch/powerpc/platforms/pseries/eeh_pseries.c |   44 ++++++++++++++++++++++++++
>  2 files changed, 50 insertions(+), 0 deletions(-)
> 
> diff --git a/arch/powerpc/include/asm/eeh.h b/arch/powerpc/include/asm/eeh.h
> index dd65e31..de821c1 100644
> --- a/arch/powerpc/include/asm/eeh.h
> +++ b/arch/powerpc/include/asm/eeh.h
> @@ -131,6 +131,10 @@ static inline struct pci_dev *eeh_dev_to_pci_dev(struct eeh_dev *edev)
>  #define EEH_LOG_TEMP		1	/* EEH temporary error log	*/
>  #define EEH_LOG_PERM		2	/* EEH permanent error log	*/
>  
> +/* Settings for platforms */
> +#define EEH_SETTING_BLOCK_CFG	1	/* Blocked PCI config access	*/
> +#define EEH_SETTING_BLOCK_IO	2	/* Blocked MMIO access		*/
> +
>  struct eeh_ops {
>  	char *name;
>  	int (*init)(void);
> @@ -146,6 +150,8 @@ struct eeh_ops {
>  	int (*configure_bridge)(struct eeh_pe *pe);
>  	int (*read_config)(struct device_node *dn, int where, int size, u32 *val);
>  	int (*write_config)(struct device_node *dn, int where, int size, u32 val);
> +	int (*get_setting)(int option, int *value, void *data);
> +	int (*set_setting)(int option, int value, void *data);
>  	int (*next_error)(struct eeh_pe **pe);
>  };
>  
> diff --git a/arch/powerpc/platforms/pseries/eeh_pseries.c b/arch/powerpc/platforms/pseries/eeh_pseries.c
> index 62415f2..8c9509b 100644
> --- a/arch/powerpc/platforms/pseries/eeh_pseries.c
> +++ b/arch/powerpc/platforms/pseries/eeh_pseries.c
> @@ -612,6 +612,48 @@ static int pseries_eeh_write_config(struct device_node *dn, int where, int size,
>  	return rtas_write_config(pdn, where, size, val);
>  }
>  
> +/**
> + * pseries_eeh_get_setting - Retrieve settings that affect EEH core
> + * @option: option
> + * @value: value
> + * @data: dependent data
> + *
> + * Retrieve the settings from the platform in order to affect the
> + * behaviour of EEH core. We don't block PCI config or MMIO access
> + * on pSeries platform.
> + */
> +static int pseries_eeh_get_setting(int option, int *value, void *data)
> +{
> +	int ret = 0;
> +
> +	switch (option) {
> +	case EEH_SETTING_BLOCK_CFG:
> +	case EEH_SETTING_BLOCK_IO:
> +		*value = 0;
> +		break;
> +	default:
> +		pr_warning("%s: Unrecognized option (%d)\n",
> +			   __func__, option);
> +		ret = -EINVAL;
> +	}
> +
> +	return ret;
> +}
> +
> +/**
> + * pseries_eeh_set_setting - Configure settings to affect EEH core
> + * @option: option
> + * @value: value
> + * @data: dependent data
> + *
> + * Configure the settings for the platform in order to affect the
> + * behaviour of EEH core.
> + */
> +static int pseries_eeh_set_setting(int option, int value, void *data)
> +{
> +	return 0;
> +}
> +
>  static struct eeh_ops pseries_eeh_ops = {
>  	.name			= "pseries",
>  	.init			= pseries_eeh_init,
> @@ -626,6 +668,8 @@ static struct eeh_ops pseries_eeh_ops = {
>  	.configure_bridge       = pseries_eeh_configure_bridge,
>  	.read_config		= pseries_eeh_read_config,
>  	.write_config		= pseries_eeh_write_config,
> +	.get_setting		= pseries_eeh_get_setting,
> +	.set_setting		= pseries_eeh_set_setting,
>  	.next_error		= NULL
>  };
>  

Re: [PATCH 10/10] net/tg3: Avoid delay during MMIO access

[Benjamin Herrenschmidt]

On Tue, 2013-06-25 at 13:55 +0800, Gavin Shan wrote:
> When the driver is encountering EEH errors, which might be caused
> by frozen PCI host controller, the driver needn't keep reading on
> MMIO until timeout. For the case, 0xFF's should be returned from
> hardware. Otherwise, it possibly trigger soft-lockup. The patch
> adds more check on that by pci_channel_offline(), thus to avoid
> the possible soft-lockup.

Can you resend this patch "standalone" (not part of a series)
to the maintainer/author of this driver and CC the netdev list on
vger.kernel.org ?

For the CC list, check the author of the original EEH support.

Also maybe improve the explanation above explaining something like:

"When the EEH error is the result of a fenced host bridge, MMIO
accesses can be very slow (milliseconds) to timeout and return all 1's,
thus causing the driver various timeout loops to take way too long and
trigger soft-lockup warnings (in addition to taking minutes to recover).

It might be worthwhile to check if for any of these cases, ffffffff is
a valid possible value, and if not, bail early since that means the HW
is either gone or isolated.

In the meantime, checking that the PCI channel is offline will
workaround the problem".

Or something like that...

> Signed-off-by: Gavin Shan <shangw@linux.vnet.ibm.com>
> ---
>  drivers/net/ethernet/broadcom/tg3.c |   36 +++++++++++++++++++++++++++++++++++
>  1 files changed, 36 insertions(+), 0 deletions(-)
> 
> diff --git a/drivers/net/ethernet/broadcom/tg3.c b/drivers/net/ethernet/broadcom/tg3.c
> index c777b90..a13463e 100644
> --- a/drivers/net/ethernet/broadcom/tg3.c
> +++ b/drivers/net/ethernet/broadcom/tg3.c
> @@ -744,6 +744,9 @@ static int tg3_ape_lock(struct tg3 *tp, int locknum)
>  		status = tg3_ape_read32(tp, gnt + off);
>  		if (status == bit)
>  			break;
> +		if (pci_channel_offline(tp->pdev))
> +			break;
> +
>  		udelay(10);
>  	}
>  
> @@ -1635,6 +1638,9 @@ static void tg3_wait_for_event_ack(struct tg3 *tp)
>  	for (i = 0; i < delay_cnt; i++) {
>  		if (!(tr32(GRC_RX_CPU_EVENT) & GRC_RX_CPU_DRIVER_EVENT))
>  			break;
> +		if (pci_channel_offline(tp->pdev))
> +			break;
> +
>  		udelay(8);
>  	}
>  }
> @@ -1813,6 +1819,9 @@ static int tg3_poll_fw(struct tg3 *tp)
>  		for (i = 0; i < 200; i++) {
>  			if (tr32(VCPU_STATUS) & VCPU_STATUS_INIT_DONE)
>  				return 0;
> +			if (pci_channel_offline(tp->pdev))
> +				return -ENODEV;
> +
>  			udelay(100);
>  		}
>  		return -ENODEV;
> @@ -1823,6 +1832,15 @@ static int tg3_poll_fw(struct tg3 *tp)
>  		tg3_read_mem(tp, NIC_SRAM_FIRMWARE_MBOX, &val);
>  		if (val == ~NIC_SRAM_FIRMWARE_MBOX_MAGIC1)
>  			break;
> +		if (pci_channel_offline(tp->pdev)) {
> +			if (!tg3_flag(tp, NO_FWARE_REPORTED)) {
> +				tg3_flag_set(tp, NO_FWARE_REPORTED);
> +				netdev_info(tp->dev, "No firmware running\n");
> +			}
> +
> +			break;
> +		}
> +
>  		udelay(10);
>  	}
>  
> @@ -3520,6 +3538,8 @@ static int tg3_pause_cpu(struct tg3 *tp, u32 cpu_base)
>  		tw32(cpu_base + CPU_MODE,  CPU_MODE_HALT);
>  		if (tr32(cpu_base + CPU_MODE) & CPU_MODE_HALT)
>  			break;
> +		if (pci_channel_offline(tp->pdev))
> +			return -EBUSY;
>  	}
>  
>  	return (i == iters) ? -EBUSY : 0;
> @@ -8589,6 +8609,14 @@ static int tg3_stop_block(struct tg3 *tp, unsigned long ofs, u32 enable_bit, boo
>  	tw32_f(ofs, val);
>  
>  	for (i = 0; i < MAX_WAIT_CNT; i++) {
> +		if (pci_channel_offline(tp->pdev)) {
> +			dev_err(&tp->pdev->dev,
> +				"tg3_stop_block device offline, "
> +				"ofs=%lx enable_bit=%x\n",
> +				ofs, enable_bit);
> +			return -ENODEV;
> +		}
> +
>  		udelay(100);
>  		val = tr32(ofs);
>  		if ((val & enable_bit) == 0)
> @@ -8612,6 +8640,13 @@ static int tg3_abort_hw(struct tg3 *tp, bool silent)
>  
>  	tg3_disable_ints(tp);
>  
> +	if (pci_channel_offline(tp->pdev)) {
> +		tp->rx_mode &= ~(RX_MODE_ENABLE | TX_MODE_ENABLE);
> +		tp->mac_mode &= ~MAC_MODE_TDE_ENABLE;
> +		err = -ENODEV;
> +		goto err_no_dev;
> +	}
> +
>  	tp->rx_mode &= ~RX_MODE_ENABLE;
>  	tw32_f(MAC_RX_MODE, tp->rx_mode);
>  	udelay(10);
> @@ -8660,6 +8695,7 @@ static int tg3_abort_hw(struct tg3 *tp, bool silent)
>  	err |= tg3_stop_block(tp, BUFMGR_MODE, BUFMGR_MODE_ENABLE, silent);
>  	err |= tg3_stop_block(tp, MEMARB_MODE, MEMARB_MODE_ENABLE, silent);
>  
> +err_no_dev:
>  	for (i = 0; i < tp->irq_cnt; i++) {
>  		struct tg3_napi *tnapi = &tp->napi[i];
>  		if (tnapi->hw_status)

Re: [PATCH 04/10] powerpc/eeh: Backends to get/set settings

[Gavin Shan]

On Tue, Jun 25, 2013 at 04:07:24PM +1000, Benjamin Herrenschmidt wrote:
>On Tue, 2013-06-25 at 13:55 +0800, Gavin Shan wrote:
>> When the PHB gets fenced, 0xFF's returns from PCI config space and
>> MMIO space in the hardware. The operations writting to them should
>> be dropped. The patch introduce backends allow to set/get flags that
>> indicate the access to PCI-CFG and MMIO should be blocked.
>
>We can't block MMIO without massive overhead. Config space can be
>blocked inside the firmware, can't it ?
>

Yep. The config space has been blocked on fenced PHB by firmware. I
almostly forgot that (struct p7ioc_phb::use_asb) :-)

Thanks,
Gavin

>
>> Signed-off-by: Gavin Shan <shangw@linux.vnet.ibm.com>
>> ---
>>  arch/powerpc/include/asm/eeh.h               |    6 +++
>>  arch/powerpc/platforms/pseries/eeh_pseries.c |   44 ++++++++++++++++++++++++++
>>  2 files changed, 50 insertions(+), 0 deletions(-)
>> 
>> diff --git a/arch/powerpc/include/asm/eeh.h b/arch/powerpc/include/asm/eeh.h
>> index dd65e31..de821c1 100644
>> --- a/arch/powerpc/include/asm/eeh.h
>> +++ b/arch/powerpc/include/asm/eeh.h
>> @@ -131,6 +131,10 @@ static inline struct pci_dev *eeh_dev_to_pci_dev(struct eeh_dev *edev)
>>  #define EEH_LOG_TEMP		1	/* EEH temporary error log	*/
>>  #define EEH_LOG_PERM		2	/* EEH permanent error log	*/
>>  
>> +/* Settings for platforms */
>> +#define EEH_SETTING_BLOCK_CFG	1	/* Blocked PCI config access	*/
>> +#define EEH_SETTING_BLOCK_IO	2	/* Blocked MMIO access		*/
>> +
>>  struct eeh_ops {
>>  	char *name;
>>  	int (*init)(void);
>> @@ -146,6 +150,8 @@ struct eeh_ops {
>>  	int (*configure_bridge)(struct eeh_pe *pe);
>>  	int (*read_config)(struct device_node *dn, int where, int size, u32 *val);
>>  	int (*write_config)(struct device_node *dn, int where, int size, u32 val);
>> +	int (*get_setting)(int option, int *value, void *data);
>> +	int (*set_setting)(int option, int value, void *data);
>>  	int (*next_error)(struct eeh_pe **pe);
>>  };
>>  
>> diff --git a/arch/powerpc/platforms/pseries/eeh_pseries.c b/arch/powerpc/platforms/pseries/eeh_pseries.c
>> index 62415f2..8c9509b 100644
>> --- a/arch/powerpc/platforms/pseries/eeh_pseries.c
>> +++ b/arch/powerpc/platforms/pseries/eeh_pseries.c
>> @@ -612,6 +612,48 @@ static int pseries_eeh_write_config(struct device_node *dn, int where, int size,
>>  	return rtas_write_config(pdn, where, size, val);
>>  }
>>  
>> +/**
>> + * pseries_eeh_get_setting - Retrieve settings that affect EEH core
>> + * @option: option
>> + * @value: value
>> + * @data: dependent data
>> + *
>> + * Retrieve the settings from the platform in order to affect the
>> + * behaviour of EEH core. We don't block PCI config or MMIO access
>> + * on pSeries platform.
>> + */
>> +static int pseries_eeh_get_setting(int option, int *value, void *data)
>> +{
>> +	int ret = 0;
>> +
>> +	switch (option) {
>> +	case EEH_SETTING_BLOCK_CFG:
>> +	case EEH_SETTING_BLOCK_IO:
>> +		*value = 0;
>> +		break;
>> +	default:
>> +		pr_warning("%s: Unrecognized option (%d)\n",
>> +			   __func__, option);
>> +		ret = -EINVAL;
>> +	}
>> +
>> +	return ret;
>> +}
>> +
>> +/**
>> + * pseries_eeh_set_setting - Configure settings to affect EEH core
>> + * @option: option
>> + * @value: value
>> + * @data: dependent data
>> + *
>> + * Configure the settings for the platform in order to affect the
>> + * behaviour of EEH core.
>> + */
>> +static int pseries_eeh_set_setting(int option, int value, void *data)
>> +{
>> +	return 0;
>> +}
>> +
>>  static struct eeh_ops pseries_eeh_ops = {
>>  	.name			= "pseries",
>>  	.init			= pseries_eeh_init,
>> @@ -626,6 +668,8 @@ static struct eeh_ops pseries_eeh_ops = {
>>  	.configure_bridge       = pseries_eeh_configure_bridge,
>>  	.read_config		= pseries_eeh_read_config,
>>  	.write_config		= pseries_eeh_write_config,
>> +	.get_setting		= pseries_eeh_get_setting,
>> +	.set_setting		= pseries_eeh_set_setting,
>>  	.next_error		= NULL
>>  };
>>  
>
>

Re: [PATCH 03/10] powerpc/eeh: Check PCIe link after reset

[Gavin Shan]

On Tue, Jun 25, 2013 at 04:06:24PM +1000, Benjamin Herrenschmidt wrote:
>On Tue, 2013-06-25 at 13:55 +0800, Gavin Shan wrote:
>>         * don't touch the other command bits
>>          */
>> -       eeh_ops->read_config(dn, PCI_COMMAND, 4, &cmd);
>> -       if (edev->config_space[1] & PCI_COMMAND_PARITY)
>> -               cmd |= PCI_COMMAND_PARITY;
>> -       else
>> -               cmd &= ~PCI_COMMAND_PARITY;
>> -       if (edev->config_space[1] & PCI_COMMAND_SERR)
>> -               cmd |= PCI_COMMAND_SERR;
>> -       else
>> -               cmd &= ~PCI_COMMAND_SERR;
>> -       eeh_ops->write_config(dn, PCI_COMMAND, 4, cmd);
>> +       if (pdev) {
>> +               eeh_ops->write_config(dn, PCI_COMMAND, 4,
>> +                                     edev->config_space[1]);
>> +       } else {
>
>That needs a much better comment. Why are you doing that instead
>of what's below ? In fact there is more to restore in a bridge
>right ? (windows etc...). Do you do that ? Should we just have a
>different function to restore a device vs. a bridge ?
>

Yeah, We should have one separate function to do that for bridge.
I'll do that in next revision.

>I also don't see a need to do thing differently between phyp and
>powernv. Bridges inside partitions would suffer the same fate in
>both cases.
>

If we just have complete reset for fenced PHB, we need restore it
from the cache (edev->config_space[1]) instead of reading that from
hardware. Fenced PHB is the special case on PowerNV :-)

Thanks,
Gavin

Re: [PATCH 03/10] powerpc/eeh: Check PCIe link after reset

[Benjamin Herrenschmidt]

On Tue, 2013-06-25 at 15:47 +0800, Gavin Shan wrote:
> If we just have complete reset for fenced PHB, we need restore it
> from the cache (edev->config_space[1]) instead of reading that from
> hardware. Fenced PHB is the special case on PowerNV :-)

Well not really...

In general we can also end up doing a hard reset under pHyp, and bridges
can lose their state as well, which means they need to be restored from
cache.

We don't see the real PHB, but we might see the bridges if we have a PE
that contains a bridge, for example, a PCIe card with a switch on it.

If we hard reset that (because the driver requested it) or if pHyp did a
reset due to a fence behind the scene, that bridge *will* have lost its
state and will need to be reconfigured too... or is RTAS doing it all ?

Cheers,
Ben.

Re: [PATCH 03/10] powerpc/eeh: Check PCIe link after reset

[Gavin Shan]

On Tue, Jun 25, 2013 at 05:57:44PM +1000, Benjamin Herrenschmidt wrote:
>On Tue, 2013-06-25 at 15:47 +0800, Gavin Shan wrote:
>> If we just have complete reset for fenced PHB, we need restore it
>> from the cache (edev->config_space[1]) instead of reading that from
>> hardware. Fenced PHB is the special case on PowerNV :-)
>
>Well not really...
>
>In general we can also end up doing a hard reset under pHyp, and bridges
>can lose their state as well, which means they need to be restored from
>cache.
>
>We don't see the real PHB, but we might see the bridges if we have a PE
>that contains a bridge, for example, a PCIe card with a switch on it.
>
>If we hard reset that (because the driver requested it) or if pHyp did a
>reset due to a fence behind the scene, that bridge *will* have lost its
>state and will need to be reconfigured too... or is RTAS doing it all ?
>

Ok. So that would be job of eeh_ops->configure_bridge(). On pSeries, it
should have done with that.

Thanks,
Gavin