Re: [PATCH v12 17/25] cxl/pci: Introduce CXL Endpoint protocol error handlers

["Bowman, Terry" <terry.bowman@amd.com>]

On 10/3/2025 3:12 PM, Cheatham, Benjamin wrote:
> On 9/25/2025 5:34 PM, Terry Bowman wrote:
>> CXL Endpoint protocol errors are currently handled using PCI error
>> handlers. The CXL Endpoint requires CXL specific handling in the case of
>> uncorrectable error (UCE) handling not provided by the PCI handlers.
>>
>> Add CXL specific handlers for CXL Endpoints. Rename the existing
>> cxl_error_handlers to be pci_error_handlers to more correctly indicate
>> the error type and follow naming consistency.
>>
>> The PCI handlers will be called if the CXL device is not trained for
>> alternate protocol (CXL). Update the CXL Endpoint PCI handlers to call the
>> CXL UCE handlers.
>>
>> The existing EP UCE handler includes checks for various results. These are
>> no longer needed because CXL UCE recovery will not be attempted. Implement
>> cxl_handle_ras() to return PCI_ERS_RESULT_NONE or PCI_ERS_RESULT_PANIC. The
>> CXL UCE handler is called by cxl_do_recovery() that acts on the return
>> value. In the case of the PCI handler path, call panic() if the result is
>> PCI_ERS_RESULT_PANIC.
>>
>> Signed-off-by: Terry Bowman <terry.bowman@amd.com>
>> Reviewed-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
>>
>> ---
>>
>> Changes in v11->v12:
>> - None
>>
>> Changes in v10->v11:
>> - cxl_error_detected() - Change handlers' scoped_guard() to guard() (Jonathan)
>> - cxl_error_detected() - Remove extra line (Shiju)
>> - Changes moved to core/ras.c (Terry)
>> - cxl_error_detected(), remove 'ue' and return with function call. (Jonathan)
>> - Remove extra space in documentation for PCI_ERS_RESULT_PANIC definition
>> - Move #include "pci.h from cxl.h to core.h (Terry)
>> - Remove unnecessary includes of cxl.h and core.h in mem.c (Terry)
>> ---
>>  drivers/cxl/core/core.h |  17 +++++++
>>  drivers/cxl/core/ras.c  | 110 +++++++++++++++++++---------------------
>>  drivers/cxl/cxlpci.h    |  15 ------
>>  drivers/cxl/pci.c       |   9 ++--
>>  4 files changed, 75 insertions(+), 76 deletions(-)
>>
>> diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
>> index 8c51a2631716..74c64d458f12 100644
>> --- a/drivers/cxl/core/core.h
>> +++ b/drivers/cxl/core/core.h
>> @@ -6,6 +6,7 @@
>>  
>>  #include <cxl/mailbox.h>
>>  #include <linux/rwsem.h>
>> +#include <linux/pci.h>
>>  
>>  extern const struct device_type cxl_nvdimm_bridge_type;
>>  extern const struct device_type cxl_nvdimm_type;
>> @@ -150,6 +151,11 @@ void cxl_ras_exit(void);
>>  void cxl_switch_port_init_ras(struct cxl_port *port);
>>  void cxl_endpoint_port_init_ras(struct cxl_port *ep);
>>  void cxl_dport_init_ras_reporting(struct cxl_dport *dport, struct device *host);
>> +pci_ers_result_t pci_error_detected(struct pci_dev *pdev,
>> +				    pci_channel_state_t error);
>> +void pci_cor_error_detected(struct pci_dev *pdev);
>> +void cxl_cor_error_detected(struct device *dev);
>> +pci_ers_result_t cxl_error_detected(struct device *dev);
>>  #else
>>  static inline int cxl_ras_init(void)
>>  {
>> @@ -163,6 +169,17 @@ static inline void cxl_switch_port_init_ras(struct cxl_port *port) { }
>>  static inline void cxl_endpoint_port_init_ras(struct cxl_port *ep) { }
>>  static inline void cxl_dport_init_ras_reporting(struct cxl_dport *dport,
>>  						struct device *host) { }
>> +static inline pci_ers_result_t pci_error_detected(struct pci_dev *pdev,
>> +						  pci_channel_state_t error)
>> +{
>> +	return PCI_ERS_RESULT_NONE;
>> +}
>> +static inline void pci_cor_error_detected(struct pci_dev *pdev) { }
>> +static inline void cxl_cor_error_detected(struct device *dev) { }
>> +static inline pci_ers_result_t cxl_error_detected(struct device *dev)
>> +{
>> +	return PCI_ERS_RESULT_NONE;
> My understanding is this only occurs for uncorrectable errors, so should this be upgraded to
> a PCI_ERS_RESULT_PANIC? If uncorrectable errors == system panic, I would expect that to be the
> case even if we don't have the code to handle the error built.
>
> I guess it's really a question of how safe you want to be. Is it ok to let uncorrectable errors
> propagate when the support is missing, or do we always panic regardless of handling code?

Here the CONFIG_CXL_RAS Kconfig is disabled and these function stubs allow the linker to complete
the build.PCI_ERS_RESULT_PANIC isn't returned because it implies handling but handling is 
disabled through unset CONFIG_CXL_RAS. If CONFIG_CXL_RAS is disabled then the interrupts
and CXL RAS logic should be disabled.

>> +}
>>  #endif // CONFIG_CXL_RAS
>>  
>>  int cxl_gpf_port_setup(struct cxl_dport *dport);
>> diff --git a/drivers/cxl/core/ras.c b/drivers/cxl/core/ras.c
>> index 14a434bd68f0..39472d82d586 100644
>> --- a/drivers/cxl/core/ras.c
>> +++ b/drivers/cxl/core/ras.c
>> @@ -129,7 +129,7 @@ void cxl_ras_exit(void)
>>  }
>>  
>>  static void cxl_handle_cor_ras(struct device *dev, u64 serial, void __iomem *ras_base);
>> -static bool cxl_handle_ras(struct device *dev, u64 serial, void __iomem *ras_base);
>> +static pci_ers_result_t cxl_handle_ras(struct device *dev, u64 serial, void __iomem *ras_base);
>>  
>>  #ifdef CONFIG_CXL_RCH_RAS
>>  static void cxl_dport_map_rch_aer(struct cxl_dport *dport)
>> @@ -371,7 +371,7 @@ static void header_log_copy(void __iomem *ras_base, u32 *log)
>>   * Log the state of the RAS status registers and prepare them to log the
>>   * next error status. Return 1 if reset needed.
>>   */
>> -static bool cxl_handle_ras(struct device *dev, u64 serial, void __iomem *ras_base)
>> +static pci_ers_result_t cxl_handle_ras(struct device *dev, u64 serial, void __iomem *ras_base)
>>  {
>>  	u32 hl[CXL_HEADERLOG_SIZE_U32];
>>  	void __iomem *addr;
>> @@ -380,13 +380,13 @@ static bool cxl_handle_ras(struct device *dev, u64 serial, void __iomem *ras_bas
>>  
>>  	if (!ras_base) {
>>  		dev_warn_once(dev, "CXL RAS register block is not mapped");
>> -		return false;
>> +		return PCI_ERS_RESULT_NONE;
> Same idea as above. I would assume since we can't tell the severity of the error we would
> just treat it as the worst case scenario.
>
The RAS UCE status needs to be read and verified as UCE before handling with a panic.

Terry