Re: [PATCH v2 02/15] cxl: add function for type2 cxl regs setup

[Zhi Wang <zhiw@nvidia.com>]

On Thu, 15 Aug 2024 17:40:35 +0100
Jonathan Cameron <Jonathan.Cameron@Huawei.com> wrote:

> On Wed, 14 Aug 2024 08:56:35 +0100
> Alejandro Lucero Palau <alucerop@amd.com> wrote:
> 
> > On 8/4/24 18:15, Jonathan Cameron wrote:
> > > On Mon, 15 Jul 2024 18:28:22 +0100
> > > alejandro.lucero-palau@amd.com wrote:
> > >  
> > >> From: Alejandro Lucero <alucerop@amd.com>
> > >>
> > >> Create a new function for a type2 device initialising the opaque
> > >> cxl_dev_state struct regarding cxl regs setup and mapping.
> > >>
> > >> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
> > >> ---
> > >>   drivers/cxl/pci.c                  | 28
> > >> ++++++++++++++++++++++++++++ drivers/net/ethernet/sfc/efx_cxl.c
> > >> |  3 +++ include/linux/cxl_accel_mem.h      |  1 +
> > >>   3 files changed, 32 insertions(+)
> > >>
> > >> diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
> > >> index e53646e9f2fb..b34d6259faf4 100644
> > >> --- a/drivers/cxl/pci.c
> > >> +++ b/drivers/cxl/pci.c
> > >> @@ -11,6 +11,7 @@
> > >>   #include <linux/pci.h>
> > >>   #include <linux/aer.h>
> > >>   #include <linux/io.h>
> > >> +#include <linux/cxl_accel_mem.h>
> > >>   #include "cxlmem.h"
> > >>   #include "cxlpci.h"
> > >>   #include "cxl.h"
> > >> @@ -521,6 +522,33 @@ static int cxl_pci_setup_regs(struct
> > >> pci_dev *pdev, enum cxl_regloc_type type, return
> > >> cxl_setup_regs(map); }
> > >>   
> > >> +int cxl_pci_accel_setup_regs(struct pci_dev *pdev, struct
> > >> cxl_dev_state *cxlds) +{
> > >> +	struct cxl_register_map map;
> > >> +	int rc;
> > >> +
> > >> +	rc = cxl_pci_setup_regs(pdev, CXL_REGLOC_RBI_MEMDEV,
> > >> &map);
> > >> +	if (rc)
> > >> +		return rc;
> > >> +
> > >> +	rc = cxl_map_device_regs(&map,
> > >> &cxlds->regs.device_regs);
> > >> +	if (rc)
> > >> +		return rc;
> > >> +
> > >> +	rc = cxl_pci_setup_regs(pdev, CXL_REGLOC_RBI_COMPONENT,
> > >> +				&cxlds->reg_map);
> > >> +	if (rc)
> > >> +		dev_warn(&pdev->dev, "No component registers
> > >> (%d)\n", rc);  
> > > Not fatal?  If we think it will happen on real devices, then
> > > dev_warn is too strong.  
> > 
> > 
> > This is more complex than what it seems, and it is not properly
> > handled with the current code.
> > 
> > I will cover it in another patch in more detail, but the fact is
> > those calls to cxl_pci_setup_regs need to be handled better,
> > because Type2 has some of these registers as optional.
> 
> I'd argue you don't have to support all type 2 devices with your
> first code.  Things like optionality of registers can come in when
> a device shows up where they aren't present.
> 
> Jonathan
> 

I think it is more like we need to change those register
probe routines to probe and return the result, but not decide
if the result is fatal or not. Let the caller decide it. E.g. type-3
assumes some registers group must be present, then the caller of type-3
can throw a fatal. While, type-2 just need to remember if the register
group is present or not. A register group is missing might not be fatal
to a type-2.

E.g.

1) moving the judges out of cxl_probe_regs() and wrap them into a
function. e.g. cxl_check_check_device_regs():
        case CXL_REGLOC_RBI_MEMDEV:
                dev_map = &map->device_map;
                cxl_probe_device_regs(host, base, dev_map);

		/* Moving the judeges out of here. */
                if (!dev_map->status.valid ||
                    ((caps & CXL_DRIVER_CAP_MBOX) &&
                !dev_map->mbox.valid) || !dev_map->memdev.valid) {
                        dev_err(host, "registers not found: %s%s%s\n",
                                !dev_map->status.valid ? "status " : "",
                                ((caps & CXL_DRIVER_CAP_MBOX) &&
                !dev_map->mbox.valid) ? "mbox " : "",
                !dev_map->memdev.valid ? "memdev " : ""); return -ENXIO;
                }

2) At the top caller for type-3 cxl_pci_probe():

        rc = cxl_pci_setup_regs(pdev, CXL_REGLOC_RBI_MEMDEV, &map,
                                cxlds->capabilities);
        if (rc)
                return rc;

	/* call cxl_check_device_regs() here, if fail, throw fatal! */

3) At the top caller for type-2 cxl_pci_accel_setup_regs():

	rc = cxl_pci_setup_regs(pdev, CXL_REGLOC_RBI_MEMDEV, &map,
                                cxlds->capabilities);
        if (rc)
                return rc;

/* call cxl_check_device_regs() here,
 * if succeed, map the registers
 * if fail, move on, no need to throw fatal.
 */
	rc = cxl_map_device_regs(&map, &cxlds->regs.device_regs);
        if (rc)
                return rc;

With the changes, we can let the CXL core detects what the registers the
device has, maybe the driver even doesn't need to tell the CXL core,
what caps the driver/device has, then we don't need to introduce the
cxlds->capabilities? the CXL core just go to check if a register group's
vaddr mapping is present, then it knows if the device has a
register group or not, after the cxl_pci_accel_setup_regs().

Thanks,
Zhi.