[RFC PATCH 0/2] cxl: Add support for disabling CAPP when unloading CXL

[RFC PATCH 0/2] cxl: Add support for disabling CAPP when unloading CXL

[Vaibhav Jain]

Recent updates to OPAL have implemented necessary infrastructure [1] to disable
CAPP and switch PHB back to PCIE mode during fast reset. This small patch-set
uses the same OPAL infrastructure to force disable of CAPP when CXL module is
unloaded via rmmod.

References:
[1]: https://lists.ozlabs.org/pipermail/skiboot/2019-January/013063.html

Vaibhav Jain (2):
  powerpc/powernv: Add support for CXL mode switch that need PHB reset
  cxl: Force a CAPP reset when unloading CXL module

 arch/powerpc/platforms/powernv/pci-cxl.c | 71 +++++++++++++++++++++---
 drivers/misc/cxl/cxl.h                   |  1 +
 drivers/misc/cxl/main.c                  |  3 +
 drivers/misc/cxl/pci.c                   | 25 ++++++++-
 4 files changed, 91 insertions(+), 9 deletions(-)

-- 
2.20.1

[RFC PATCH 1/2] powerpc/powernv: Add support for CXL mode switch that need PHB reset

[Vaibhav Jain]

Recent updates to OPAL [1] have provided support for new CXL modes on
PHB that need to force a cold reset on the bridge (CRESET). However
PHB CRESET is a multi step process and cannot be completed
synchronously as expected by current kernel implementation that issues
opal call opal_pci_set_phb_cxl_mode().

Hence this patch updates pnv_phb_to_cxl_mode() to implement a polling
loop that handles specific error codes (OPAL_BUSY) returned from
opal_pci_set_phb_cxl_mode() and drive the OPAL pci-state machine, if the
requested CXL mode needs a CRESET.

The patch also updates pnv_phb_to_cxl_mode() to convert and return
OPAL error codes into kernel error codes. This removes a previous
issue where callers to this function would have to include
'opal-api.h' to check for specific OPAL error codes.

References:
[1]: https://lists.ozlabs.org/pipermail/skiboot/2019-January/013063.html

Signed-off-by: Vaibhav Jain <vaibhav@linux.ibm.com>
---
 arch/powerpc/platforms/powernv/pci-cxl.c | 71 +++++++++++++++++++++---
 1 file changed, 63 insertions(+), 8 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/pci-cxl.c b/arch/powerpc/platforms/powernv/pci-cxl.c
index 1b18111453d7..d33d662c6212 100644
--- a/arch/powerpc/platforms/powernv/pci-cxl.c
+++ b/arch/powerpc/platforms/powernv/pci-cxl.c
@@ -10,6 +10,7 @@
 #include <linux/module.h>
 #include <asm/pnv-pci.h>
 #include <asm/opal.h>
+#include <linux/delay.h>
 
 #include "pci.h"
 
@@ -18,21 +19,75 @@ int pnv_phb_to_cxl_mode(struct pci_dev *dev, uint64_t mode)
 	struct pci_controller *hose = pci_bus_to_host(dev->bus);
 	struct pnv_phb *phb = hose->private_data;
 	struct pnv_ioda_pe *pe;
+	unsigned long starttime, endtime;
 	int rc;
 
 	pe = pnv_ioda_get_pe(dev);
 	if (!pe)
-		return -ENODEV;
+		return -ENOENT;
+
+	pe_info(pe, "Switching PHB to CXL mode=%d\n", mode);
+
+	/*
+	 * Use a 15 second timeout for mode switch. Value arrived after
+	 * limited testing and may need more tweaking.
+	 */
+	starttime = jiffies;
+	endtime = starttime + HZ * 15;
+
+	do {
+		rc = opal_pci_set_phb_cxl_mode(phb->opal_id, mode,
+					       pe->pe_number);
+
+		/* Wait until mode transistion done */
+		if (rc != OPAL_BUSY && rc != OPAL_BUSY_EVENT)
+			break;
+
+		/* Check if we timedout */
+		if (time_after(jiffies, endtime)) {
+			rc = OPAL_TIMEOUT;
+			break;
+		}
 
-	pe_info(pe, "Switching PHB to CXL\n");
+		/* Opal Busy with mode switch. Run pci state-machine */
+		rc = opal_pci_poll(phb->opal_id);
+		if (rc >= 0) {
+			/* wait for some time */
+			if (rc > 0)
+				msleep(rc);
+			opal_poll_events(NULL);
+			rc = OPAL_BUSY;
+			/* Continue with the mode switch */
+		}
+	} while (rc == OPAL_BUSY || rc == OPAL_BUSY_EVENT);
+
+	pe_level_printk(pe, KERN_DEBUG, "CXL mode switch finished in %u-msecs.",
+			jiffies_to_msecs(jiffies - starttime));
 
-	rc = opal_pci_set_phb_cxl_mode(phb->opal_id, mode, pe->pe_number);
-	if (rc == OPAL_UNSUPPORTED)
-		dev_err(&dev->dev, "Required cxl mode not supported by firmware - update skiboot\n");
-	else if (rc)
-		dev_err(&dev->dev, "opal_pci_set_phb_cxl_mode failed: %i\n", rc);
+	/* Check OPAL errors and convert them to kernel error codes */
+	switch (rc) {
+	case OPAL_SUCCESS:
+		return 0;
 
-	return rc;
+	case OPAL_PARAMETER:
+		dev_err(&dev->dev, "CXL not supported on this PHB\n");
+		return -ENOENT;
+
+	case OPAL_UNSUPPORTED:
+		dev_err(&dev->dev,
+			"Required cxl mode not supported by firmware"
+			" - update skiboot\n");
+		return -ENODEV;
+
+	case OPAL_TIMEOUT:
+		dev_err(&dev->dev, "opal_pci_set_phb_cxl_mode Timedout\n");
+		return -ETIME;
+
+	default:
+		dev_err(&dev->dev,
+			"opal_pci_set_phb_cxl_mode failed: %i\n", rc);
+		return -EIO;
+	};
 }
 EXPORT_SYMBOL(pnv_phb_to_cxl_mode);
 
-- 
2.20.1

Re: [RFC PATCH 1/2] powerpc/powernv: Add support for CXL mode switch that need PHB reset

[Frederic Barrat]

Le 25/01/2019 à 06:11, Vaibhav Jain a écrit :
> Recent updates to OPAL [1] have provided support for new CXL modes on
> PHB that need to force a cold reset on the bridge (CRESET). However
> PHB CRESET is a multi step process and cannot be completed
> synchronously as expected by current kernel implementation that issues
> opal call opal_pci_set_phb_cxl_mode().
> 
> Hence this patch updates pnv_phb_to_cxl_mode() to implement a polling
> loop that handles specific error codes (OPAL_BUSY) returned from
> opal_pci_set_phb_cxl_mode() and drive the OPAL pci-state machine, if the
> requested CXL mode needs a CRESET.
> 
> The patch also updates pnv_phb_to_cxl_mode() to convert and return
> OPAL error codes into kernel error codes. This removes a previous
> issue where callers to this function would have to include
> 'opal-api.h' to check for specific OPAL error codes.
> 
> References:
> [1]: https://lists.ozlabs.org/pipermail/skiboot/2019-January/013063.html
> 
> Signed-off-by: Vaibhav Jain <vaibhav@linux.ibm.com>
> ---
>   arch/powerpc/platforms/powernv/pci-cxl.c | 71 +++++++++++++++++++++---
>   1 file changed, 63 insertions(+), 8 deletions(-)
> 
> diff --git a/arch/powerpc/platforms/powernv/pci-cxl.c b/arch/powerpc/platforms/powernv/pci-cxl.c
> index 1b18111453d7..d33d662c6212 100644
> --- a/arch/powerpc/platforms/powernv/pci-cxl.c
> +++ b/arch/powerpc/platforms/powernv/pci-cxl.c
> @@ -10,6 +10,7 @@
>   #include <linux/module.h>
>   #include <asm/pnv-pci.h>
>   #include <asm/opal.h>
> +#include <linux/delay.h>
>   
>   #include "pci.h"
>   
> @@ -18,21 +19,75 @@ int pnv_phb_to_cxl_mode(struct pci_dev *dev, uint64_t mode)
>   	struct pci_controller *hose = pci_bus_to_host(dev->bus);
>   	struct pnv_phb *phb = hose->private_data;
>   	struct pnv_ioda_pe *pe;
> +	unsigned long starttime, endtime;
>   	int rc;
>   
>   	pe = pnv_ioda_get_pe(dev);
>   	if (!pe)
> -		return -ENODEV;
> +		return -ENOENT;
> +
> +	pe_info(pe, "Switching PHB to CXL mode=%d\n", mode);
> +
> +	/*
> +	 * Use a 15 second timeout for mode switch. Value arrived after
> +	 * limited testing and may need more tweaking.
> +	 */
> +	starttime = jiffies;
> +	endtime = starttime + HZ * 15;
> +
> +	do {
> +		rc = opal_pci_set_phb_cxl_mode(phb->opal_id, mode,
> +					       pe->pe_number);
> +
> +		/* Wait until mode transistion done */
> +		if (rc != OPAL_BUSY && rc != OPAL_BUSY_EVENT)
> +			break;
> +
> +		/* Check if we timedout */
> +		if (time_after(jiffies, endtime)) {
> +			rc = OPAL_TIMEOUT;
> +			break;
> +		}
>   
> -	pe_info(pe, "Switching PHB to CXL\n");
> +		/* Opal Busy with mode switch. Run pci state-machine */
> +		rc = opal_pci_poll(phb->opal_id);
> +		if (rc >= 0) {
> +			/* wait for some time */
> +			if (rc > 0)
> +				msleep(rc);
> +			opal_poll_events(NULL);

Why is a call to opal_poll_events() needed?

   Fred


> +			rc = OPAL_BUSY;
> +			/* Continue with the mode switch */
> +		}
> +	} while (rc == OPAL_BUSY || rc == OPAL_BUSY_EVENT);
> +
> +	pe_level_printk(pe, KERN_DEBUG, "CXL mode switch finished in %u-msecs.",
> +			jiffies_to_msecs(jiffies - starttime));
>   
> -	rc = opal_pci_set_phb_cxl_mode(phb->opal_id, mode, pe->pe_number);
> -	if (rc == OPAL_UNSUPPORTED)
> -		dev_err(&dev->dev, "Required cxl mode not supported by firmware - update skiboot\n");
> -	else if (rc)
> -		dev_err(&dev->dev, "opal_pci_set_phb_cxl_mode failed: %i\n", rc);
> +	/* Check OPAL errors and convert them to kernel error codes */
> +	switch (rc) {
> +	case OPAL_SUCCESS:
> +		return 0;
>   
> -	return rc;
> +	case OPAL_PARAMETER:
> +		dev_err(&dev->dev, "CXL not supported on this PHB\n");
> +		return -ENOENT;
> +
> +	case OPAL_UNSUPPORTED:
> +		dev_err(&dev->dev,
> +			"Required cxl mode not supported by firmware"
> +			" - update skiboot\n");
> +		return -ENODEV;
> +
> +	case OPAL_TIMEOUT:
> +		dev_err(&dev->dev, "opal_pci_set_phb_cxl_mode Timedout\n");
> +		return -ETIME;
> +
> +	default:
> +		dev_err(&dev->dev,
> +			"opal_pci_set_phb_cxl_mode failed: %i\n", rc);
> +		return -EIO;
> +	};
>   }
>   EXPORT_SYMBOL(pnv_phb_to_cxl_mode);
>   
> 

Re: [RFC PATCH 1/2] powerpc/powernv: Add support for CXL mode switch that need PHB reset

[Vaibhav Jain]

Frederic Barrat <fbarrat@linux.ibm.com> writes:
>> +			opal_poll_events(NULL);
>
> Why is a call to opal_poll_events() needed?

Trying to make sure that opal pollers are run on the current CPU and any
opal timer are executed.

-- 
Vaibhav Jain <vaibhav@linux.ibm.com>
Linux Technology Center, IBM India Pvt. Ltd.

Re: [RFC PATCH 1/2] powerpc/powernv: Add support for CXL mode switch that need PHB reset

[christophe lombard]

On 25/01/2019 06:11, Vaibhav Jain wrote:
> Recent updates to OPAL [1] have provided support for new CXL modes on
> PHB that need to force a cold reset on the bridge (CRESET). However
> PHB CRESET is a multi step process and cannot be completed
> synchronously as expected by current kernel implementation that issues
> opal call opal_pci_set_phb_cxl_mode().
> 
> Hence this patch updates pnv_phb_to_cxl_mode() to implement a polling
> loop that handles specific error codes (OPAL_BUSY) returned from
> opal_pci_set_phb_cxl_mode() and drive the OPAL pci-state machine, if the
> requested CXL mode needs a CRESET.
> 
> The patch also updates pnv_phb_to_cxl_mode() to convert and return
> OPAL error codes into kernel error codes. This removes a previous
> issue where callers to this function would have to include
> 'opal-api.h' to check for specific OPAL error codes.
> 
> References:
> [1]: https://lists.ozlabs.org/pipermail/skiboot/2019-January/013063.html
> 
> Signed-off-by: Vaibhav Jain <vaibhav@linux.ibm.com>
> ---
>   arch/powerpc/platforms/powernv/pci-cxl.c | 71 +++++++++++++++++++++---
>   1 file changed, 63 insertions(+), 8 deletions(-)
> 
> diff --git a/arch/powerpc/platforms/powernv/pci-cxl.c b/arch/powerpc/platforms/powernv/pci-cxl.c
> index 1b18111453d7..d33d662c6212 100644
> --- a/arch/powerpc/platforms/powernv/pci-cxl.c
> +++ b/arch/powerpc/platforms/powernv/pci-cxl.c
> @@ -10,6 +10,7 @@
>   #include <linux/module.h>
>   #include <asm/pnv-pci.h>
>   #include <asm/opal.h>
> +#include <linux/delay.h>
> 
>   #include "pci.h"
> 
> @@ -18,21 +19,75 @@ int pnv_phb_to_cxl_mode(struct pci_dev *dev, uint64_t mode)
>   	struct pci_controller *hose = pci_bus_to_host(dev->bus);
>   	struct pnv_phb *phb = hose->private_data;
>   	struct pnv_ioda_pe *pe;
> +	unsigned long starttime, endtime;
>   	int rc;
> 
>   	pe = pnv_ioda_get_pe(dev);
>   	if (!pe)
> -		return -ENODEV;
> +		return -ENOENT;

The return code of pnv_phb_to_cxl_mode() is also returned by an api in 
the cxllib librarie. So, hoping that nobody test the value !!

> +
> +	pe_info(pe, "Switching PHB to CXL mode=%d\n", mode);
> +
> +	/*
> +	 * Use a 15 second timeout for mode switch. Value arrived after
> +	 * limited testing and may need more tweaking.
> +	 */
> +	starttime = jiffies;
> +	endtime = starttime + HZ * 15;
> +
> +	do {
> +		rc = opal_pci_set_phb_cxl_mode(phb->opal_id, mode,
> +					       pe->pe_number);
> +
> +		/* Wait until mode transistion done */
> +		if (rc != OPAL_BUSY && rc != OPAL_BUSY_EVENT)
> +			break;
> +
> +		/* Check if we timedout */
> +		if (time_after(jiffies, endtime)) {
> +			rc = OPAL_TIMEOUT;
> +			break;
> +		}
> 
> -	pe_info(pe, "Switching PHB to CXL\n");
> +		/* Opal Busy with mode switch. Run pci state-machine */
> +		rc = opal_pci_poll(phb->opal_id);
> +		if (rc >= 0) {
> +			/* wait for some time */
> +			if (rc > 0)
> +				msleep(rc);
> +			opal_poll_events(NULL);
> +			rc = OPAL_BUSY;
> +			/* Continue with the mode switch */
> +		}
> +	} while (rc == OPAL_BUSY || rc == OPAL_BUSY_EVENT);
> +
> +	pe_level_printk(pe, KERN_DEBUG, "CXL mode switch finished in %u-msecs.",
> +			jiffies_to_msecs(jiffies - starttime));
> 
> -	rc = opal_pci_set_phb_cxl_mode(phb->opal_id, mode, pe->pe_number);
> -	if (rc == OPAL_UNSUPPORTED)
> -		dev_err(&dev->dev, "Required cxl mode not supported by firmware - update skiboot\n");
> -	else if (rc)
> -		dev_err(&dev->dev, "opal_pci_set_phb_cxl_mode failed: %i\n", rc);
> +	/* Check OPAL errors and convert them to kernel error codes */
> +	switch (rc) {
> +	case OPAL_SUCCESS:
> +		return 0;
> 
> -	return rc;
> +	case OPAL_PARAMETER:
> +		dev_err(&dev->dev, "CXL not supported on this PHB\n");
> +		return -ENOENT;
> +
> +	case OPAL_UNSUPPORTED:
> +		dev_err(&dev->dev,
> +			"Required cxl mode not supported by firmware"
> +			" - update skiboot\n");
> +		return -ENODEV;
> +
> +	case OPAL_TIMEOUT:
> +		dev_err(&dev->dev, "opal_pci_set_phb_cxl_mode Timedout\n");
> +		return -ETIME;
> +
> +	default:
> +		dev_err(&dev->dev,
> +			"opal_pci_set_phb_cxl_mode failed: %i\n", rc);
> +		return -EIO;
> +	};
>   }
>   EXPORT_SYMBOL(pnv_phb_to_cxl_mode);
> 

Re: [RFC PATCH 1/2] powerpc/powernv: Add support for CXL mode switch that need PHB reset

[Vaibhav Jain]

Thanks for reviewing this patch Christophe,

christophe lombard <clombard@linux.vnet.ibm.com> writes:

>> 
>>   	pe = pnv_ioda_get_pe(dev);
>>   	if (!pe)
>> -		return -ENODEV;
>> +		return -ENOENT;
>
> The return code of pnv_phb_to_cxl_mode() is also returned by an api in 
> the cxllib librarie. So, hoping that nobody test the value !!

Agreed. I did peek into cxllib_switch_phb_mode() before sending the
patch and saw two conflicting cases. While switching to CXL_MODE_PCI we
make sure that we return kernel error codes and while switching to
CXL_MODE_CXL we return OPAL error codes.

I havent seen how CX5 handles return values from this function but I am
betting thats its the usual zero & non-zero return value check, which
then should work with the proposed change.

-- 
Vaibhav Jain <vaibhav@linux.ibm.com>
Linux Technology Center, IBM India Pvt. Ltd.

[RFC PATCH 2/2] cxl: Force a CAPP reset when unloading CXL module

[Vaibhav Jain]

This patch forces shutdown of CAPP when CXL module is unloaded. This
is accomplished via a call to pnv_phb_to_cxl_mode() with mode ==
OPAL_PHB_CAPI_MODE_PCIE.

Signed-off-by: Vaibhav Jain <vaibhav@linux.ibm.com>
---
 drivers/misc/cxl/cxl.h  |  1 +
 drivers/misc/cxl/main.c |  3 +++
 drivers/misc/cxl/pci.c  | 25 ++++++++++++++++++++++++-
 3 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/drivers/misc/cxl/cxl.h b/drivers/misc/cxl/cxl.h
index d1d927ccb589..e545c2b81faf 100644
--- a/drivers/misc/cxl/cxl.h
+++ b/drivers/misc/cxl/cxl.h
@@ -1136,4 +1136,5 @@ void cxl_context_mm_count_get(struct cxl_context *ctx);
 /* Decrements the reference count to "struct mm_struct" */
 void cxl_context_mm_count_put(struct cxl_context *ctx);
 
+void cxl_pci_shutdown_capp(void);
 #endif
diff --git a/drivers/misc/cxl/main.c b/drivers/misc/cxl/main.c
index f35406be465a..f14ff0dcf231 100644
--- a/drivers/misc/cxl/main.c
+++ b/drivers/misc/cxl/main.c
@@ -372,6 +372,9 @@ static void exit_cxl(void)
 	if (cxl_is_power8())
 		unregister_cxl_calls(&cxl_calls);
 	idr_destroy(&cxl_adapter_idr);
+
+	if (cpu_has_feature(CPU_FTR_HVMODE))
+		cxl_pci_shutdown_capp();
 }
 
 module_init(init_cxl);
diff --git a/drivers/misc/cxl/pci.c b/drivers/misc/cxl/pci.c
index c79ba1c699ad..01be2e2d1069 100644
--- a/drivers/misc/cxl/pci.c
+++ b/drivers/misc/cxl/pci.c
@@ -25,7 +25,7 @@
 
 #include "cxl.h"
 #include <misc/cxl.h>
-
+#include <misc/cxllib.h>
 
 #define CXL_PCI_VSEC_ID	0x1280
 #define CXL_VSEC_MIN_SIZE 0x80
@@ -2065,6 +2065,29 @@ static void cxl_pci_resume(struct pci_dev *pdev)
 	}
 }
 
+void cxl_pci_shutdown_capp(void)
+{
+	struct pci_dev *pdev;
+	struct pci_bus *root_bus;
+	int rc;
+
+	/* Iterate over all CAPP supported PHB's and force them to PCI mode */
+	list_for_each_entry(root_bus, &pci_root_buses, node) {
+		for_each_pci_bridge(pdev, root_bus) {
+
+			if (!cxllib_slot_is_supported(pdev, 0))
+				continue;
+
+			rc = pnv_phb_to_cxl_mode(pdev,
+						 OPAL_PHB_CAPI_MODE_PCIE);
+			if (rc)
+				dev_err(&pdev->dev,
+					"cxl: Error resetting CAPP. Err=%d\n",
+					rc);
+		}
+	}
+}
+
 static const struct pci_error_handlers cxl_err_handler = {
 	.error_detected = cxl_pci_error_detected,
 	.slot_reset = cxl_pci_slot_reset,
-- 
2.20.1

Re: [RFC PATCH 2/2] cxl: Force a CAPP reset when unloading CXL module

[Frederic Barrat]

Le 25/01/2019 à 06:11, Vaibhav Jain a écrit :
> This patch forces shutdown of CAPP when CXL module is unloaded. This
> is accomplished via a call to pnv_phb_to_cxl_mode() with mode ==
> OPAL_PHB_CAPI_MODE_PCIE.
> 
> Signed-off-by: Vaibhav Jain <vaibhav@linux.ibm.com>
> ---
>   drivers/misc/cxl/cxl.h  |  1 +
>   drivers/misc/cxl/main.c |  3 +++
>   drivers/misc/cxl/pci.c  | 25 ++++++++++++++++++++++++-
>   3 files changed, 28 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/misc/cxl/cxl.h b/drivers/misc/cxl/cxl.h
> index d1d927ccb589..e545c2b81faf 100644
> --- a/drivers/misc/cxl/cxl.h
> +++ b/drivers/misc/cxl/cxl.h
> @@ -1136,4 +1136,5 @@ void cxl_context_mm_count_get(struct cxl_context *ctx);
>   /* Decrements the reference count to "struct mm_struct" */
>   void cxl_context_mm_count_put(struct cxl_context *ctx);
>   
> +void cxl_pci_shutdown_capp(void);
>   #endif
> diff --git a/drivers/misc/cxl/main.c b/drivers/misc/cxl/main.c
> index f35406be465a..f14ff0dcf231 100644
> --- a/drivers/misc/cxl/main.c
> +++ b/drivers/misc/cxl/main.c
> @@ -372,6 +372,9 @@ static void exit_cxl(void)
>   	if (cxl_is_power8())
>   		unregister_cxl_calls(&cxl_calls);
>   	idr_destroy(&cxl_adapter_idr);
> +
> +	if (cpu_has_feature(CPU_FTR_HVMODE))
> +		cxl_pci_shutdown_capp();
>   }
>   
>   module_init(init_cxl);
> diff --git a/drivers/misc/cxl/pci.c b/drivers/misc/cxl/pci.c
> index c79ba1c699ad..01be2e2d1069 100644
> --- a/drivers/misc/cxl/pci.c
> +++ b/drivers/misc/cxl/pci.c
> @@ -25,7 +25,7 @@
>   
>   #include "cxl.h"
>   #include <misc/cxl.h>
> -
> +#include <misc/cxllib.h>
>   
>   #define CXL_PCI_VSEC_ID	0x1280
>   #define CXL_VSEC_MIN_SIZE 0x80
> @@ -2065,6 +2065,29 @@ static void cxl_pci_resume(struct pci_dev *pdev)
>   	}
>   }
>   
> +void cxl_pci_shutdown_capp(void)
> +{
> +	struct pci_dev *pdev;
> +	struct pci_bus *root_bus;
> +	int rc;
> +
> +	/* Iterate over all CAPP supported PHB's and force them to PCI mode */
> +	list_for_each_entry(root_bus, &pci_root_buses, node) {
> +		for_each_pci_bridge(pdev, root_bus) {
> +
> +			if (!cxllib_slot_is_supported(pdev, 0))
> +				continue;
> +
> +			rc = pnv_phb_to_cxl_mode(pdev,
> +						 OPAL_PHB_CAPI_MODE_PCIE);
> +			if (rc)
> +				dev_err(&pdev->dev,
> +					"cxl: Error resetting CAPP. Err=%d\n",
> +					rc);
> +		}


That's the part I don't like. We're iterating over quite a few PCI 
devices, we basically don't know the ones we need to reset.
If we have a per-adapter property on /sys to activate the 
reset-on-unload, then we could move the call to 
pnv_phb_to_cxl_mode(OPAL_PHB_CAPI_MODE_PCIE) on the cxl_remove() 
callback, and only do it for the adapters we've been asked.

   Fred



> +	}
> +}
> +
>   static const struct pci_error_handlers cxl_err_handler = {
>   	.error_detected = cxl_pci_error_detected,
>   	.slot_reset = cxl_pci_slot_reset,
> 

Re: [RFC PATCH 0/2] cxl: Add support for disabling CAPP when unloading CXL

[Frederic Barrat]

Hi Vaibhav,

I think there's value in there, as I'm hearing Mellanox would prefer to 
start from a "clean" state, as in pci mode, when they load their driver 
after a card FW update.

However, I'm pretty reluctant to make it the default behaviour. It's not 
like we haven't had our share of problems on reset before :-) 
Furthermore, for non-mlx5 case, we don't really have a reason for this.
I'm thinking we could activate it through a per-adapter property, on 
/sys. See comment in one of the patches.

   Fred


Le 25/01/2019 à 06:11, Vaibhav Jain a écrit :
> Recent updates to OPAL have implemented necessary infrastructure [1] to disable
> CAPP and switch PHB back to PCIE mode during fast reset. This small patch-set
> uses the same OPAL infrastructure to force disable of CAPP when CXL module is
> unloaded via rmmod.
> 
> References:
> [1]: https://lists.ozlabs.org/pipermail/skiboot/2019-January/013063.html
> 
> Vaibhav Jain (2):
>    powerpc/powernv: Add support for CXL mode switch that need PHB reset
>    cxl: Force a CAPP reset when unloading CXL module
> 
>   arch/powerpc/platforms/powernv/pci-cxl.c | 71 +++++++++++++++++++++---
>   drivers/misc/cxl/cxl.h                   |  1 +
>   drivers/misc/cxl/main.c                  |  3 +
>   drivers/misc/cxl/pci.c                   | 25 ++++++++-
>   4 files changed, 91 insertions(+), 9 deletions(-)
>