[PATCH v3 net-next 0/3] amd-xgbe: add support for AMD Crater

[PATCH v3 net-next 0/3] amd-xgbe: add support for AMD Crater

[Raju Rangoju]

Add support for a new AMD Ethernet device called "Crater". It has a new
PCI ID, add this to the current list of supported devices in the
amd-xgbe devices.Also, the BAR1 addresses cannot be used to access the
PCS registers on Crater platform, use the indirect addressing via SMN
instead.

Changes since v2:
 - Club patches 2 and 3 to avoid build warnings.
 - Fix the smatch warnings.
 - Fix the build errors on arm64


Raju Rangoju (3):
  amd-xgbe: reorganize the code of XPCS access
  amd-xgbe: add support for Crater ethernet device
  amd-xgbe: use smn functions to avoid race

 drivers/net/ethernet/amd/xgbe/xgbe-common.h |   5 +
 drivers/net/ethernet/amd/xgbe/xgbe-dev.c    | 119 ++++++++++++++++----
 drivers/net/ethernet/amd/xgbe/xgbe-pci.c    |  51 ++++++++-
 drivers/net/ethernet/amd/xgbe/xgbe-smn.h    |  27 +++++
 drivers/net/ethernet/amd/xgbe/xgbe.h        |   6 +
 5 files changed, 180 insertions(+), 28 deletions(-)
 create mode 100644 drivers/net/ethernet/amd/xgbe/xgbe-smn.h

-- 
2.34.1

[PATCH v3 net-next 1/3] amd-xgbe: reorganize the code of XPCS access

[Raju Rangoju]

The xgbe_{read/write}_mmd_regs_v* functions have common code which can
be moved to helper functions. Also, the xgbe_pci_probe() needs
reorganization.

Add new helper functions to calculate the mmd_address for v1/v2 of xpcs
access. And, convert if/else statements in xgbe_pci_probe() to switch
case. This helps code look cleaner.

Signed-off-by: Raju Rangoju <Raju.Rangoju@amd.com>
---
 drivers/net/ethernet/amd/xgbe/xgbe-dev.c | 43 ++++++++++++------------
 drivers/net/ethernet/amd/xgbe/xgbe-pci.c | 18 +++++++---
 2 files changed, 34 insertions(+), 27 deletions(-)

diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
index f393228d41c7..6cd003c24a64 100644
--- a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
+++ b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
@@ -1150,6 +1150,21 @@ static int xgbe_set_gpio(struct xgbe_prv_data *pdata, unsigned int gpio)
 	return 0;
 }
 
+static unsigned int get_mmd_address(struct xgbe_prv_data *pdata, int mmd_reg)
+{
+	return (mmd_reg & XGBE_ADDR_C45) ?
+		mmd_reg & ~XGBE_ADDR_C45 :
+		(pdata->mdio_mmd << 16) | (mmd_reg & 0xffff);
+}
+
+static unsigned int get_index_offset(struct xgbe_prv_data *pdata, unsigned int mmd_address,
+				     unsigned int *index)
+{
+	mmd_address <<= 1;
+	*index = mmd_address & ~pdata->xpcs_window_mask;
+	return pdata->xpcs_window + (mmd_address & pdata->xpcs_window_mask);
+}
+
 static int xgbe_read_mmd_regs_v2(struct xgbe_prv_data *pdata, int prtad,
 				 int mmd_reg)
 {
@@ -1157,10 +1172,7 @@ static int xgbe_read_mmd_regs_v2(struct xgbe_prv_data *pdata, int prtad,
 	unsigned int mmd_address, index, offset;
 	int mmd_data;
 
-	if (mmd_reg & XGBE_ADDR_C45)
-		mmd_address = mmd_reg & ~XGBE_ADDR_C45;
-	else
-		mmd_address = (pdata->mdio_mmd << 16) | (mmd_reg & 0xffff);
+	mmd_address = get_mmd_address(pdata, mmd_reg);
 
 	/* The PCS registers are accessed using mmio. The underlying
 	 * management interface uses indirect addressing to access the MMD
@@ -1171,9 +1183,7 @@ static int xgbe_read_mmd_regs_v2(struct xgbe_prv_data *pdata, int prtad,
 	 * register offsets must therefore be adjusted by left shifting the
 	 * offset 1 bit and reading 16 bits of data.
 	 */
-	mmd_address <<= 1;
-	index = mmd_address & ~pdata->xpcs_window_mask;
-	offset = pdata->xpcs_window + (mmd_address & pdata->xpcs_window_mask);
+	offset = get_index_offset(pdata, mmd_address, &index);
 
 	spin_lock_irqsave(&pdata->xpcs_lock, flags);
 	XPCS32_IOWRITE(pdata, pdata->xpcs_window_sel_reg, index);
@@ -1189,10 +1199,7 @@ static void xgbe_write_mmd_regs_v2(struct xgbe_prv_data *pdata, int prtad,
 	unsigned long flags;
 	unsigned int mmd_address, index, offset;
 
-	if (mmd_reg & XGBE_ADDR_C45)
-		mmd_address = mmd_reg & ~XGBE_ADDR_C45;
-	else
-		mmd_address = (pdata->mdio_mmd << 16) | (mmd_reg & 0xffff);
+	mmd_address = get_mmd_address(pdata, mmd_reg);
 
 	/* The PCS registers are accessed using mmio. The underlying
 	 * management interface uses indirect addressing to access the MMD
@@ -1203,9 +1210,7 @@ static void xgbe_write_mmd_regs_v2(struct xgbe_prv_data *pdata, int prtad,
 	 * register offsets must therefore be adjusted by left shifting the
 	 * offset 1 bit and writing 16 bits of data.
 	 */
-	mmd_address <<= 1;
-	index = mmd_address & ~pdata->xpcs_window_mask;
-	offset = pdata->xpcs_window + (mmd_address & pdata->xpcs_window_mask);
+	offset = get_index_offset(pdata, mmd_address, &index);
 
 	spin_lock_irqsave(&pdata->xpcs_lock, flags);
 	XPCS32_IOWRITE(pdata, pdata->xpcs_window_sel_reg, index);
@@ -1220,10 +1225,7 @@ static int xgbe_read_mmd_regs_v1(struct xgbe_prv_data *pdata, int prtad,
 	unsigned int mmd_address;
 	int mmd_data;
 
-	if (mmd_reg & XGBE_ADDR_C45)
-		mmd_address = mmd_reg & ~XGBE_ADDR_C45;
-	else
-		mmd_address = (pdata->mdio_mmd << 16) | (mmd_reg & 0xffff);
+	mmd_address = get_mmd_address(pdata, mmd_reg);
 
 	/* The PCS registers are accessed using mmio. The underlying APB3
 	 * management interface uses indirect addressing to access the MMD
@@ -1248,10 +1250,7 @@ static void xgbe_write_mmd_regs_v1(struct xgbe_prv_data *pdata, int prtad,
 	unsigned int mmd_address;
 	unsigned long flags;
 
-	if (mmd_reg & XGBE_ADDR_C45)
-		mmd_address = mmd_reg & ~XGBE_ADDR_C45;
-	else
-		mmd_address = (pdata->mdio_mmd << 16) | (mmd_reg & 0xffff);
+	mmd_address = get_mmd_address(pdata, mmd_reg);
 
 	/* The PCS registers are accessed using mmio. The underlying APB3
 	 * management interface uses indirect addressing to access the MMD
diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-pci.c b/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
index f409d7bd1f1e..8b0c1e450b7e 100644
--- a/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
+++ b/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
@@ -274,12 +274,18 @@ static int xgbe_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 
 	/* Set the PCS indirect addressing definition registers */
 	rdev = pci_get_domain_bus_and_slot(0, 0, PCI_DEVFN(0, 0));
-	if (rdev &&
-	    (rdev->vendor == PCI_VENDOR_ID_AMD) && (rdev->device == 0x15d0)) {
+
+	if (!(rdev && rdev->vendor == PCI_VENDOR_ID_AMD)) {
+		ret = -ENODEV;
+		goto err_pci_enable;
+	}
+
+	switch (rdev->device) {
+	case 0x15d0:
 		pdata->xpcs_window_def_reg = PCS_V2_RV_WINDOW_DEF;
 		pdata->xpcs_window_sel_reg = PCS_V2_RV_WINDOW_SELECT;
-	} else if (rdev && (rdev->vendor == PCI_VENDOR_ID_AMD) &&
-		   (rdev->device == 0x14b5)) {
+		break;
+	case 0x14b5:
 		pdata->xpcs_window_def_reg = PCS_V2_YC_WINDOW_DEF;
 		pdata->xpcs_window_sel_reg = PCS_V2_YC_WINDOW_SELECT;
 
@@ -288,9 +294,11 @@ static int xgbe_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 
 		/* Yellow Carp devices do not need rrc */
 		pdata->vdata->enable_rrc = 0;
-	} else {
+		break;
+	default:
 		pdata->xpcs_window_def_reg = PCS_V2_WINDOW_DEF;
 		pdata->xpcs_window_sel_reg = PCS_V2_WINDOW_SELECT;
+		break;
 	}
 	pci_dev_put(rdev);
 
-- 
2.34.1

Re: [PATCH v3 net-next 1/3] amd-xgbe: reorganize the code of XPCS access

[Kalesh Anakkur Purayil]

[-- Attachment #1.1: Type: text/plain, Size: 7281 bytes --]

On Tue, Dec 12, 2023 at 11:08 AM Raju Rangoju <Raju.Rangoju@amd.com> wrote:

> The xgbe_{read/write}_mmd_regs_v* functions have common code which can
> be moved to helper functions. Also, the xgbe_pci_probe() needs
> reorganization.
>
> Add new helper functions to calculate the mmd_address for v1/v2 of xpcs
> access. And, convert if/else statements in xgbe_pci_probe() to switch
> case. This helps code look cleaner.
>
> Signed-off-by: Raju Rangoju <Raju.Rangoju@amd.com>
> ---
>  drivers/net/ethernet/amd/xgbe/xgbe-dev.c | 43 ++++++++++++------------
>  drivers/net/ethernet/amd/xgbe/xgbe-pci.c | 18 +++++++---
>  2 files changed, 34 insertions(+), 27 deletions(-)
>
> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
> b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
> index f393228d41c7..6cd003c24a64 100644
> --- a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
> +++ b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
> @@ -1150,6 +1150,21 @@ static int xgbe_set_gpio(struct xgbe_prv_data
> *pdata, unsigned int gpio)
>         return 0;
>  }
>
> +static unsigned int get_mmd_address(struct xgbe_prv_data *pdata, int
> mmd_reg)
> +{
> +       return (mmd_reg & XGBE_ADDR_C45) ?
> +               mmd_reg & ~XGBE_ADDR_C45 :
> +               (pdata->mdio_mmd << 16) | (mmd_reg & 0xffff);
> +}
> +
> +static unsigned int get_index_offset(struct xgbe_prv_data *pdata,
> unsigned int mmd_address,
> +                                    unsigned int *index)
> +{
> +       mmd_address <<= 1;
> +       *index = mmd_address & ~pdata->xpcs_window_mask;
> +       return pdata->xpcs_window + (mmd_address &
> pdata->xpcs_window_mask);
> +}
> +
>  static int xgbe_read_mmd_regs_v2(struct xgbe_prv_data *pdata, int prtad,
>                                  int mmd_reg)
>  {
> @@ -1157,10 +1172,7 @@ static int xgbe_read_mmd_regs_v2(struct
> xgbe_prv_data *pdata, int prtad,
>         unsigned int mmd_address, index, offset;
>         int mmd_data;
>
> -       if (mmd_reg & XGBE_ADDR_C45)
> -               mmd_address = mmd_reg & ~XGBE_ADDR_C45;
> -       else
> -               mmd_address = (pdata->mdio_mmd << 16) | (mmd_reg & 0xffff);
> +       mmd_address = get_mmd_address(pdata, mmd_reg);
>
>         /* The PCS registers are accessed using mmio. The underlying
>          * management interface uses indirect addressing to access the MMD
> @@ -1171,9 +1183,7 @@ static int xgbe_read_mmd_regs_v2(struct
> xgbe_prv_data *pdata, int prtad,
>          * register offsets must therefore be adjusted by left shifting the
>          * offset 1 bit and reading 16 bits of data.
>          */
> -       mmd_address <<= 1;
> -       index = mmd_address & ~pdata->xpcs_window_mask;
> -       offset = pdata->xpcs_window + (mmd_address &
> pdata->xpcs_window_mask);
> +       offset = get_index_offset(pdata, mmd_address, &index);
>
>         spin_lock_irqsave(&pdata->xpcs_lock, flags);
>         XPCS32_IOWRITE(pdata, pdata->xpcs_window_sel_reg, index);
> @@ -1189,10 +1199,7 @@ static void xgbe_write_mmd_regs_v2(struct
> xgbe_prv_data *pdata, int prtad,
>         unsigned long flags;
>         unsigned int mmd_address, index, offset;
>
> -       if (mmd_reg & XGBE_ADDR_C45)
> -               mmd_address = mmd_reg & ~XGBE_ADDR_C45;
> -       else
> -               mmd_address = (pdata->mdio_mmd << 16) | (mmd_reg & 0xffff);
> +       mmd_address = get_mmd_address(pdata, mmd_reg);
>
>         /* The PCS registers are accessed using mmio. The underlying
>          * management interface uses indirect addressing to access the MMD
> @@ -1203,9 +1210,7 @@ static void xgbe_write_mmd_regs_v2(struct
> xgbe_prv_data *pdata, int prtad,
>          * register offsets must therefore be adjusted by left shifting the
>          * offset 1 bit and writing 16 bits of data.
>          */
> -       mmd_address <<= 1;
> -       index = mmd_address & ~pdata->xpcs_window_mask;
> -       offset = pdata->xpcs_window + (mmd_address &
> pdata->xpcs_window_mask);
> +       offset = get_index_offset(pdata, mmd_address, &index);
>
>         spin_lock_irqsave(&pdata->xpcs_lock, flags);
>         XPCS32_IOWRITE(pdata, pdata->xpcs_window_sel_reg, index);
> @@ -1220,10 +1225,7 @@ static int xgbe_read_mmd_regs_v1(struct
> xgbe_prv_data *pdata, int prtad,
>         unsigned int mmd_address;
>         int mmd_data;
>
> -       if (mmd_reg & XGBE_ADDR_C45)
> -               mmd_address = mmd_reg & ~XGBE_ADDR_C45;
> -       else
> -               mmd_address = (pdata->mdio_mmd << 16) | (mmd_reg & 0xffff);
> +       mmd_address = get_mmd_address(pdata, mmd_reg);
>
>         /* The PCS registers are accessed using mmio. The underlying APB3
>          * management interface uses indirect addressing to access the MMD
> @@ -1248,10 +1250,7 @@ static void xgbe_write_mmd_regs_v1(struct
> xgbe_prv_data *pdata, int prtad,
>         unsigned int mmd_address;
>         unsigned long flags;
>
> -       if (mmd_reg & XGBE_ADDR_C45)
> -               mmd_address = mmd_reg & ~XGBE_ADDR_C45;
> -       else
> -               mmd_address = (pdata->mdio_mmd << 16) | (mmd_reg & 0xffff);
> +       mmd_address = get_mmd_address(pdata, mmd_reg);
>
>         /* The PCS registers are accessed using mmio. The underlying APB3
>          * management interface uses indirect addressing to access the MMD
> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
> b/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
> index f409d7bd1f1e..8b0c1e450b7e 100644
> --- a/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
> +++ b/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
> @@ -274,12 +274,18 @@ static int xgbe_pci_probe(struct pci_dev *pdev,
> const struct pci_device_id *id)
>
>         /* Set the PCS indirect addressing definition registers */
>         rdev = pci_get_domain_bus_and_slot(0, 0, PCI_DEVFN(0, 0));
> -       if (rdev &&
> -           (rdev->vendor == PCI_VENDOR_ID_AMD) && (rdev->device ==
> 0x15d0)) {
> +
> +       if (!(rdev && rdev->vendor == PCI_VENDOR_ID_AMD)) {
> +               ret = -ENODEV;
> +               goto err_pci_enable;
> +       }
> +
> +       switch (rdev->device) {
> +       case 0x15d0:
>                 pdata->xpcs_window_def_reg = PCS_V2_RV_WINDOW_DEF;
>                 pdata->xpcs_window_sel_reg = PCS_V2_RV_WINDOW_SELECT;
> -       } else if (rdev && (rdev->vendor == PCI_VENDOR_ID_AMD) &&
> -                  (rdev->device == 0x14b5)) {
> +               break;
> +       case 0x14b5:
>
[Kalesh]: Can you use macros for 0x15d0 and 0x14b5

>                 pdata->xpcs_window_def_reg = PCS_V2_YC_WINDOW_DEF;
>                 pdata->xpcs_window_sel_reg = PCS_V2_YC_WINDOW_SELECT;
>
> @@ -288,9 +294,11 @@ static int xgbe_pci_probe(struct pci_dev *pdev, const
> struct pci_device_id *id)
>
>                 /* Yellow Carp devices do not need rrc */
>                 pdata->vdata->enable_rrc = 0;
> -       } else {
> +               break;
> +       default:
>                 pdata->xpcs_window_def_reg = PCS_V2_WINDOW_DEF;
>                 pdata->xpcs_window_sel_reg = PCS_V2_WINDOW_SELECT;
> +               break;
>         }
>         pci_dev_put(rdev);
>
> --
> 2.34.1
>
>
>

-- 
Regards,
Kalesh A P

[-- Attachment #1.2: Type: text/html, Size: 9003 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4239 bytes --]

Re: [PATCH v3 net-next 1/3] amd-xgbe: reorganize the code of XPCS access

[Tom Lendacky]

On 12/11/23 23:37, Raju Rangoju wrote:
> The xgbe_{read/write}_mmd_regs_v* functions have common code which can
> be moved to helper functions. Also, the xgbe_pci_probe() needs
> reorganization.
> 
> Add new helper functions to calculate the mmd_address for v1/v2 of xpcs
> access. And, convert if/else statements in xgbe_pci_probe() to switch
> case. This helps code look cleaner.
> 
> Signed-off-by: Raju Rangoju <Raju.Rangoju@amd.com>
> ---
>   drivers/net/ethernet/amd/xgbe/xgbe-dev.c | 43 ++++++++++++------------
>   drivers/net/ethernet/amd/xgbe/xgbe-pci.c | 18 +++++++---
>   2 files changed, 34 insertions(+), 27 deletions(-)
> 
> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
> index f393228d41c7..6cd003c24a64 100644
> --- a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
> +++ b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
> @@ -1150,6 +1150,21 @@ static int xgbe_set_gpio(struct xgbe_prv_data *pdata, unsigned int gpio)
>   	return 0;
>   }
>   
> +static unsigned int get_mmd_address(struct xgbe_prv_data *pdata, int mmd_reg)
> +{
> +	return (mmd_reg & XGBE_ADDR_C45) ?
> +		mmd_reg & ~XGBE_ADDR_C45 :
> +		(pdata->mdio_mmd << 16) | (mmd_reg & 0xffff);
> +}
> +
> +static unsigned int get_index_offset(struct xgbe_prv_data *pdata, unsigned int mmd_address,
> +				     unsigned int *index)

Just my opinion, but this looks confusing to me by updating index and returning
offset. And the name is confusing, too. I think it would read better as:

static void get_pcs_index_and_offset(struct xgbe_prv_data *pdata, unsigned int mmd_address,
				     unsigned int *index, unsigned int *offset)
{
	mmd_address <<= 1;
	*index = mmd_address & ~pdata->xpcs_window_mask;
	*offset = pdata->xpcs_window + (mmd_address & pdata->xpcs_window_mask);
}

Or break this into two functions:

static unsigned int get_pcs_index(struct xgbe_prv_data *pdata, unsigned int mmd_address)
{
	return (mmd_address << 1) & ~pdata->xpcs_window_mask;
}

static unsigned int get_pcs_offset(struct xgbe_prv_data *pdata, unsigned int mmd_address)
{
	return pdata->xpcs_window + ((mmd_address << 1) & pdata->xpcs_window_mask);
}

> +{
> +	mmd_address <<= 1;
> +	*index = mmd_address & ~pdata->xpcs_window_mask;
> +	return pdata->xpcs_window + (mmd_address & pdata->xpcs_window_mask);
> +}
> +
>   static int xgbe_read_mmd_regs_v2(struct xgbe_prv_data *pdata, int prtad,
>   				 int mmd_reg)
>   {
> @@ -1157,10 +1172,7 @@ static int xgbe_read_mmd_regs_v2(struct xgbe_prv_data *pdata, int prtad,
>   	unsigned int mmd_address, index, offset;
>   	int mmd_data;
>   
> -	if (mmd_reg & XGBE_ADDR_C45)
> -		mmd_address = mmd_reg & ~XGBE_ADDR_C45;
> -	else
> -		mmd_address = (pdata->mdio_mmd << 16) | (mmd_reg & 0xffff);
> +	mmd_address = get_mmd_address(pdata, mmd_reg);
>   
>   	/* The PCS registers are accessed using mmio. The underlying
>   	 * management interface uses indirect addressing to access the MMD
> @@ -1171,9 +1183,7 @@ static int xgbe_read_mmd_regs_v2(struct xgbe_prv_data *pdata, int prtad,
>   	 * register offsets must therefore be adjusted by left shifting the
>   	 * offset 1 bit and reading 16 bits of data.
>   	 */
> -	mmd_address <<= 1;
> -	index = mmd_address & ~pdata->xpcs_window_mask;
> -	offset = pdata->xpcs_window + (mmd_address & pdata->xpcs_window_mask);
> +	offset = get_index_offset(pdata, mmd_address, &index);

The comment above this code should be moved into the new helper and then
removed here and below.

>   
>   	spin_lock_irqsave(&pdata->xpcs_lock, flags);
>   	XPCS32_IOWRITE(pdata, pdata->xpcs_window_sel_reg, index);
> @@ -1189,10 +1199,7 @@ static void xgbe_write_mmd_regs_v2(struct xgbe_prv_data *pdata, int prtad,
>   	unsigned long flags;
>   	unsigned int mmd_address, index, offset;
>   
> -	if (mmd_reg & XGBE_ADDR_C45)
> -		mmd_address = mmd_reg & ~XGBE_ADDR_C45;
> -	else
> -		mmd_address = (pdata->mdio_mmd << 16) | (mmd_reg & 0xffff);
> +	mmd_address = get_mmd_address(pdata, mmd_reg);
>   
>   	/* The PCS registers are accessed using mmio. The underlying
>   	 * management interface uses indirect addressing to access the MMD
> @@ -1203,9 +1210,7 @@ static void xgbe_write_mmd_regs_v2(struct xgbe_prv_data *pdata, int prtad,
>   	 * register offsets must therefore be adjusted by left shifting the
>   	 * offset 1 bit and writing 16 bits of data.
>   	 */
> -	mmd_address <<= 1;
> -	index = mmd_address & ~pdata->xpcs_window_mask;
> -	offset = pdata->xpcs_window + (mmd_address & pdata->xpcs_window_mask);
> +	offset = get_index_offset(pdata, mmd_address, &index);
>   
>   	spin_lock_irqsave(&pdata->xpcs_lock, flags);
>   	XPCS32_IOWRITE(pdata, pdata->xpcs_window_sel_reg, index);
> @@ -1220,10 +1225,7 @@ static int xgbe_read_mmd_regs_v1(struct xgbe_prv_data *pdata, int prtad,
>   	unsigned int mmd_address;
>   	int mmd_data;
>   
> -	if (mmd_reg & XGBE_ADDR_C45)
> -		mmd_address = mmd_reg & ~XGBE_ADDR_C45;
> -	else
> -		mmd_address = (pdata->mdio_mmd << 16) | (mmd_reg & 0xffff);
> +	mmd_address = get_mmd_address(pdata, mmd_reg);
>   
>   	/* The PCS registers are accessed using mmio. The underlying APB3
>   	 * management interface uses indirect addressing to access the MMD
> @@ -1248,10 +1250,7 @@ static void xgbe_write_mmd_regs_v1(struct xgbe_prv_data *pdata, int prtad,
>   	unsigned int mmd_address;
>   	unsigned long flags;
>   
> -	if (mmd_reg & XGBE_ADDR_C45)
> -		mmd_address = mmd_reg & ~XGBE_ADDR_C45;
> -	else
> -		mmd_address = (pdata->mdio_mmd << 16) | (mmd_reg & 0xffff);
> +	mmd_address = get_mmd_address(pdata, mmd_reg);
>   
>   	/* The PCS registers are accessed using mmio. The underlying APB3
>   	 * management interface uses indirect addressing to access the MMD
> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-pci.c b/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
> index f409d7bd1f1e..8b0c1e450b7e 100644
> --- a/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
> +++ b/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
> @@ -274,12 +274,18 @@ static int xgbe_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>   
>   	/* Set the PCS indirect addressing definition registers */
>   	rdev = pci_get_domain_bus_and_slot(0, 0, PCI_DEVFN(0, 0));
> -	if (rdev &&
> -	    (rdev->vendor == PCI_VENDOR_ID_AMD) && (rdev->device == 0x15d0)) {
> +
> +	if (!(rdev && rdev->vendor == PCI_VENDOR_ID_AMD)) {
> +		ret = -ENODEV;
> +		goto err_pci_enable;
> +	}

This is different behavior compared to today. Today, everything would
default to the final "else" statement. With this patch you have a
possibility of failing probe now.

Thanks,
Tom

> +
> +	switch (rdev->device) {
> +	case 0x15d0:
>   		pdata->xpcs_window_def_reg = PCS_V2_RV_WINDOW_DEF;
>   		pdata->xpcs_window_sel_reg = PCS_V2_RV_WINDOW_SELECT;
> -	} else if (rdev && (rdev->vendor == PCI_VENDOR_ID_AMD) &&
> -		   (rdev->device == 0x14b5)) {
> +		break;
> +	case 0x14b5:
>   		pdata->xpcs_window_def_reg = PCS_V2_YC_WINDOW_DEF;
>   		pdata->xpcs_window_sel_reg = PCS_V2_YC_WINDOW_SELECT;
>   
> @@ -288,9 +294,11 @@ static int xgbe_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>   
>   		/* Yellow Carp devices do not need rrc */
>   		pdata->vdata->enable_rrc = 0;
> -	} else {
> +		break;
> +	default:
>   		pdata->xpcs_window_def_reg = PCS_V2_WINDOW_DEF;
>   		pdata->xpcs_window_sel_reg = PCS_V2_WINDOW_SELECT;
> +		break;
>   	}
>   	pci_dev_put(rdev);
>   

Re: [PATCH v3 net-next 1/3] amd-xgbe: reorganize the code of XPCS access

[Raju Rangoju]

On 12/12/2023 8:03 PM, Tom Lendacky wrote:
> On 12/11/23 23:37, Raju Rangoju wrote:
>> The xgbe_{read/write}_mmd_regs_v* functions have common code which can
>> be moved to helper functions. Also, the xgbe_pci_probe() needs
>> reorganization.
>>
>> Add new helper functions to calculate the mmd_address for v1/v2 of xpcs
>> access. And, convert if/else statements in xgbe_pci_probe() to switch
>> case. This helps code look cleaner.
>>
>> Signed-off-by: Raju Rangoju <Raju.Rangoju@amd.com>
>> ---
>>   drivers/net/ethernet/amd/xgbe/xgbe-dev.c | 43 ++++++++++++------------
>>   drivers/net/ethernet/amd/xgbe/xgbe-pci.c | 18 +++++++---
>>   2 files changed, 34 insertions(+), 27 deletions(-)
>>
>> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c 
>> b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
>> index f393228d41c7..6cd003c24a64 100644
>> --- a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
>> +++ b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
>> @@ -1150,6 +1150,21 @@ static int xgbe_set_gpio(struct xgbe_prv_data 
>> *pdata, unsigned int gpio)
>>       return 0;
>>   }
>> +static unsigned int get_mmd_address(struct xgbe_prv_data *pdata, int 
>> mmd_reg)
>> +{
>> +    return (mmd_reg & XGBE_ADDR_C45) ?
>> +        mmd_reg & ~XGBE_ADDR_C45 :
>> +        (pdata->mdio_mmd << 16) | (mmd_reg & 0xffff);
>> +}
>> +
>> +static unsigned int get_index_offset(struct xgbe_prv_data *pdata, 
>> unsigned int mmd_address,
>> +                     unsigned int *index)
> 
> Just my opinion, but this looks confusing to me by updating index and 
> returning
> offset. And the name is confusing, too. I think it would read better as:
> 
> static void get_pcs_index_and_offset(struct xgbe_prv_data *pdata, 
> unsigned int mmd_address,
>                       unsigned int *index, unsigned int *offset)
> {
>      mmd_address <<= 1;
>      *index = mmd_address & ~pdata->xpcs_window_mask;
>      *offset = pdata->xpcs_window + (mmd_address & 
> pdata->xpcs_window_mask);
> }
> 

Sure

> Or break this into two functions:
> 
> static unsigned int get_pcs_index(struct xgbe_prv_data *pdata, unsigned 
> int mmd_address)
> {
>      return (mmd_address << 1) & ~pdata->xpcs_window_mask;
> }
> 
> static unsigned int get_pcs_offset(struct xgbe_prv_data *pdata, unsigned 
> int mmd_address)
> {
>      return pdata->xpcs_window + ((mmd_address << 1) & 
> pdata->xpcs_window_mask);
> }
> 
>> +{
>> +    mmd_address <<= 1;
>> +    *index = mmd_address & ~pdata->xpcs_window_mask;
>> +    return pdata->xpcs_window + (mmd_address & pdata->xpcs_window_mask);
>> +}
>> +
>>   static int xgbe_read_mmd_regs_v2(struct xgbe_prv_data *pdata, int 
>> prtad,
>>                    int mmd_reg)
>>   {
>> @@ -1157,10 +1172,7 @@ static int xgbe_read_mmd_regs_v2(struct 
>> xgbe_prv_data *pdata, int prtad,
>>       unsigned int mmd_address, index, offset;
>>       int mmd_data;
>> -    if (mmd_reg & XGBE_ADDR_C45)
>> -        mmd_address = mmd_reg & ~XGBE_ADDR_C45;
>> -    else
>> -        mmd_address = (pdata->mdio_mmd << 16) | (mmd_reg & 0xffff);
>> +    mmd_address = get_mmd_address(pdata, mmd_reg);
>>       /* The PCS registers are accessed using mmio. The underlying
>>        * management interface uses indirect addressing to access the MMD
>> @@ -1171,9 +1183,7 @@ static int xgbe_read_mmd_regs_v2(struct 
>> xgbe_prv_data *pdata, int prtad,
>>        * register offsets must therefore be adjusted by left shifting the
>>        * offset 1 bit and reading 16 bits of data.
>>        */
>> -    mmd_address <<= 1;
>> -    index = mmd_address & ~pdata->xpcs_window_mask;
>> -    offset = pdata->xpcs_window + (mmd_address & 
>> pdata->xpcs_window_mask);
>> +    offset = get_index_offset(pdata, mmd_address, &index);
> 
> The comment above this code should be moved into the new helper and then
> removed here and below.

Sure, I'll take care of this.

> 
>>       spin_lock_irqsave(&pdata->xpcs_lock, flags);
>>       XPCS32_IOWRITE(pdata, pdata->xpcs_window_sel_reg, index);
>> @@ -1189,10 +1199,7 @@ static void xgbe_write_mmd_regs_v2(struct 
>> xgbe_prv_data *pdata, int prtad,
>>       unsigned long flags;
>>       unsigned int mmd_address, index, offset;
>> -    if (mmd_reg & XGBE_ADDR_C45)
>> -        mmd_address = mmd_reg & ~XGBE_ADDR_C45;
>> -    else
>> -        mmd_address = (pdata->mdio_mmd << 16) | (mmd_reg & 0xffff);
>> +    mmd_address = get_mmd_address(pdata, mmd_reg);
>>       /* The PCS registers are accessed using mmio. The underlying
>>        * management interface uses indirect addressing to access the MMD
>> @@ -1203,9 +1210,7 @@ static void xgbe_write_mmd_regs_v2(struct 
>> xgbe_prv_data *pdata, int prtad,
>>        * register offsets must therefore be adjusted by left shifting the
>>        * offset 1 bit and writing 16 bits of data.
>>        */
>> -    mmd_address <<= 1;
>> -    index = mmd_address & ~pdata->xpcs_window_mask;
>> -    offset = pdata->xpcs_window + (mmd_address & 
>> pdata->xpcs_window_mask);
>> +    offset = get_index_offset(pdata, mmd_address, &index);
>>       spin_lock_irqsave(&pdata->xpcs_lock, flags);
>>       XPCS32_IOWRITE(pdata, pdata->xpcs_window_sel_reg, index);
>> @@ -1220,10 +1225,7 @@ static int xgbe_read_mmd_regs_v1(struct 
>> xgbe_prv_data *pdata, int prtad,
>>       unsigned int mmd_address;
>>       int mmd_data;
>> -    if (mmd_reg & XGBE_ADDR_C45)
>> -        mmd_address = mmd_reg & ~XGBE_ADDR_C45;
>> -    else
>> -        mmd_address = (pdata->mdio_mmd << 16) | (mmd_reg & 0xffff);
>> +    mmd_address = get_mmd_address(pdata, mmd_reg);
>>       /* The PCS registers are accessed using mmio. The underlying APB3
>>        * management interface uses indirect addressing to access the MMD
>> @@ -1248,10 +1250,7 @@ static void xgbe_write_mmd_regs_v1(struct 
>> xgbe_prv_data *pdata, int prtad,
>>       unsigned int mmd_address;
>>       unsigned long flags;
>> -    if (mmd_reg & XGBE_ADDR_C45)
>> -        mmd_address = mmd_reg & ~XGBE_ADDR_C45;
>> -    else
>> -        mmd_address = (pdata->mdio_mmd << 16) | (mmd_reg & 0xffff);
>> +    mmd_address = get_mmd_address(pdata, mmd_reg);
>>       /* The PCS registers are accessed using mmio. The underlying APB3
>>        * management interface uses indirect addressing to access the MMD
>> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-pci.c 
>> b/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
>> index f409d7bd1f1e..8b0c1e450b7e 100644
>> --- a/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
>> +++ b/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
>> @@ -274,12 +274,18 @@ static int xgbe_pci_probe(struct pci_dev *pdev, 
>> const struct pci_device_id *id)
>>       /* Set the PCS indirect addressing definition registers */
>>       rdev = pci_get_domain_bus_and_slot(0, 0, PCI_DEVFN(0, 0));
>> -    if (rdev &&
>> -        (rdev->vendor == PCI_VENDOR_ID_AMD) && (rdev->device == 
>> 0x15d0)) {
>> +
>> +    if (!(rdev && rdev->vendor == PCI_VENDOR_ID_AMD)) {
>> +        ret = -ENODEV;
>> +        goto err_pci_enable;
>> +    }
> 
> This is different behavior compared to today. Today, everything would
> default to the final "else" statement. With this patch you have a
> possibility of failing probe now.

This was done to skip the cases where rdev is NULL or vendor != AMD. Not 
sure if I'm missing something.

> 
> Thanks,
> Tom
> 
>> +
>> +    switch (rdev->device) {
>> +    case 0x15d0:
>>           pdata->xpcs_window_def_reg = PCS_V2_RV_WINDOW_DEF;
>>           pdata->xpcs_window_sel_reg = PCS_V2_RV_WINDOW_SELECT;
>> -    } else if (rdev && (rdev->vendor == PCI_VENDOR_ID_AMD) &&
>> -           (rdev->device == 0x14b5)) {
>> +        break;
>> +    case 0x14b5:
>>           pdata->xpcs_window_def_reg = PCS_V2_YC_WINDOW_DEF;
>>           pdata->xpcs_window_sel_reg = PCS_V2_YC_WINDOW_SELECT;
>> @@ -288,9 +294,11 @@ static int xgbe_pci_probe(struct pci_dev *pdev, 
>> const struct pci_device_id *id)
>>           /* Yellow Carp devices do not need rrc */
>>           pdata->vdata->enable_rrc = 0;
>> -    } else {
>> +        break;
>> +    default:
>>           pdata->xpcs_window_def_reg = PCS_V2_WINDOW_DEF;
>>           pdata->xpcs_window_sel_reg = PCS_V2_WINDOW_SELECT;
>> +        break;
>>       }
>>       pci_dev_put(rdev);

Re: [PATCH v3 net-next 1/3] amd-xgbe: reorganize the code of XPCS access

[Tom Lendacky]

On 12/13/23 09:35, Raju Rangoju wrote:
> On 12/12/2023 8:03 PM, Tom Lendacky wrote:
>> On 12/11/23 23:37, Raju Rangoju wrote:
>>> The xgbe_{read/write}_mmd_regs_v* functions have common code which can
>>> be moved to helper functions. Also, the xgbe_pci_probe() needs
>>> reorganization.
>>>
>>> Add new helper functions to calculate the mmd_address for v1/v2 of xpcs
>>> access. And, convert if/else statements in xgbe_pci_probe() to switch
>>> case. This helps code look cleaner.
>>>
>>> Signed-off-by: Raju Rangoju <Raju.Rangoju@amd.com>
>>> ---
>>>   drivers/net/ethernet/amd/xgbe/xgbe-dev.c | 43 ++++++++++++------------
>>>   drivers/net/ethernet/amd/xgbe/xgbe-pci.c | 18 +++++++---
>>>   2 files changed, 34 insertions(+), 27 deletions(-)
>>>

>>> @@ -274,12 +274,18 @@ static int xgbe_pci_probe(struct pci_dev *pdev, 
>>> const struct pci_device_id *id)
>>>       /* Set the PCS indirect addressing definition registers */
>>>       rdev = pci_get_domain_bus_and_slot(0, 0, PCI_DEVFN(0, 0));
>>> -    if (rdev &&
>>> -        (rdev->vendor == PCI_VENDOR_ID_AMD) && (rdev->device == 
>>> 0x15d0)) {
>>> +
>>> +    if (!(rdev && rdev->vendor == PCI_VENDOR_ID_AMD)) {
>>> +        ret = -ENODEV;
>>> +        goto err_pci_enable;
>>> +    }
>>
>> This is different behavior compared to today. Today, everything would
>> default to the final "else" statement. With this patch you have a
>> possibility of failing probe now.
> 
> This was done to skip the cases where rdev is NULL or vendor != AMD. Not 
> sure if I'm missing something.

Right, I'm just pointing out that if rdev == NULL or vendor != AMD then, 
today, the else path is taken. With this patch you'll fail the probe with 
-ENODEV.

Thanks,
Tom

> 

[PATCH v3 net-next 2/3] amd-xgbe: add support for Crater ethernet device

[Raju Rangoju]

Add the necessary support to enable Crater ethernet device. Since the
BAR1 address cannot be used to access the XPCS registers on Crater, use
the pci_{read/write}_config_dword calls. Also, include the new pci device
id 0x1641 to register Crater device with PCIe.

Co-developed-by: Sudheesh Mavila <sudheesh.mavila@amd.com>
Signed-off-by: Sudheesh Mavila <sudheesh.mavila@amd.com>
Signed-off-by: Raju Rangoju <Raju.Rangoju@amd.com>
---
 drivers/net/ethernet/amd/xgbe/xgbe-common.h |  5 ++
 drivers/net/ethernet/amd/xgbe/xgbe-dev.c    | 93 +++++++++++++++++++++
 drivers/net/ethernet/amd/xgbe/xgbe-pci.c    | 35 +++++++-
 drivers/net/ethernet/amd/xgbe/xgbe.h        |  6 ++
 4 files changed, 137 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-common.h b/drivers/net/ethernet/amd/xgbe/xgbe-common.h
index 3b70f6737633..e1f70f0528ef 100644
--- a/drivers/net/ethernet/amd/xgbe/xgbe-common.h
+++ b/drivers/net/ethernet/amd/xgbe/xgbe-common.h
@@ -900,6 +900,11 @@
 #define PCS_V2_RV_WINDOW_SELECT		0x1064
 #define PCS_V2_YC_WINDOW_DEF		0x18060
 #define PCS_V2_YC_WINDOW_SELECT		0x18064
+#define PCS_V2_RN_WINDOW_DEF		0xF8078
+#define PCS_V2_RN_WINDOW_SELECT		0xF807c
+
+#define PCS_RN_SMN_BASE_ADDR		0x11E00000
+#define PCS_RN_PORT_ADDR_SIZE		0x100000
 
 /* PCS register entry bit positions and sizes */
 #define PCS_V2_WINDOW_DEF_OFFSET_INDEX	6
diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
index 6cd003c24a64..a9eb2ffa9f73 100644
--- a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
+++ b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
@@ -120,6 +120,7 @@
 #include <linux/bitrev.h>
 #include <linux/crc32.h>
 #include <linux/crc32poly.h>
+#include <linux/pci.h>
 
 #include "xgbe.h"
 #include "xgbe-common.h"
@@ -1165,6 +1166,92 @@ static unsigned int get_index_offset(struct xgbe_prv_data *pdata, unsigned int m
 	return pdata->xpcs_window + (mmd_address & pdata->xpcs_window_mask);
 }
 
+static int xgbe_read_mmd_regs_v3(struct xgbe_prv_data *pdata, int prtad,
+				 int mmd_reg)
+{
+	unsigned int mmd_address, index, offset;
+	struct pci_dev *rdev;
+	unsigned long flags;
+	int mmd_data;
+
+	rdev = pci_get_domain_bus_and_slot(0, 0, PCI_DEVFN(0, 0));
+	if (!rdev)
+		return 0;
+
+	mmd_address = get_mmd_address(pdata, mmd_reg);
+
+	/* The PCS registers are accessed using mmio. The underlying
+	 * management interface uses indirect addressing to access the MMD
+	 * register sets. This requires accessing of the PCS register in two
+	 * phases, an address phase and a data phase.
+	 *
+	 * The mmio interface is based on 16-bit offsets and values. All
+	 * register offsets must therefore be adjusted by left shifting the
+	 * offset 1 bit and reading 16 bits of data.
+	 */
+	offset = get_index_offset(pdata, mmd_address, &index);
+
+	spin_lock_irqsave(&pdata->xpcs_lock, flags);
+	pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + pdata->xpcs_window_sel_reg));
+	pci_write_config_dword(rdev, 0x64, index);
+	pci_write_config_dword(rdev, 0x60, pdata->xphy_base + offset);
+	pci_read_config_dword(rdev, 0x64, &mmd_data);
+	mmd_data = (offset % 4) ? FIELD_GET(XGBE_GEN_HI_MASK, mmd_data) :
+				  FIELD_GET(XGBE_GEN_LO_MASK, mmd_data);
+	pci_dev_put(rdev);
+
+	spin_unlock_irqrestore(&pdata->xpcs_lock, flags);
+
+	return mmd_data;
+}
+
+static void xgbe_write_mmd_regs_v3(struct xgbe_prv_data *pdata, int prtad,
+				   int mmd_reg, int mmd_data)
+{
+	unsigned int mmd_address, index, offset, ctr_mmd_data;
+	struct pci_dev *rdev;
+	unsigned long flags;
+
+	rdev = pci_get_domain_bus_and_slot(0, 0, PCI_DEVFN(0, 0));
+	if (!rdev)
+		return;
+
+	mmd_address = get_mmd_address(pdata, mmd_reg);
+
+	/* The PCS registers are accessed using mmio. The underlying
+	 * management interface uses indirect addressing to access the MMD
+	 * register sets. This requires accessing of the PCS register in two
+	 * phases, an address phase and a data phase.
+	 *
+	 * The mmio interface is based on 16-bit offsets and values. All
+	 * register offsets must therefore be adjusted by left shifting the
+	 * offset 1 bit and writing 16 bits of data.
+	 */
+	offset = get_index_offset(pdata, mmd_address, &index);
+
+	spin_lock_irqsave(&pdata->xpcs_lock, flags);
+	pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + pdata->xpcs_window_sel_reg));
+	pci_write_config_dword(rdev, 0x64, index);
+	pci_write_config_dword(rdev, 0x60, pdata->xphy_base + offset);
+	pci_read_config_dword(rdev, 0x64, &ctr_mmd_data);
+	if (offset % 4) {
+		ctr_mmd_data = FIELD_PREP(XGBE_GEN_HI_MASK, mmd_data) |
+			       FIELD_GET(XGBE_GEN_LO_MASK, ctr_mmd_data);
+	} else {
+		ctr_mmd_data = FIELD_PREP(XGBE_GEN_HI_MASK,
+					  FIELD_GET(XGBE_GEN_HI_MASK, ctr_mmd_data)) |
+			       FIELD_GET(XGBE_GEN_LO_MASK, mmd_data);
+	}
+
+	pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + pdata->xpcs_window_sel_reg));
+	pci_write_config_dword(rdev, 0x64, index);
+	pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + offset));
+	pci_write_config_dword(rdev, 0x64, ctr_mmd_data);
+	pci_dev_put(rdev);
+
+	spin_unlock_irqrestore(&pdata->xpcs_lock, flags);
+}
+
 static int xgbe_read_mmd_regs_v2(struct xgbe_prv_data *pdata, int prtad,
 				 int mmd_reg)
 {
@@ -1274,6 +1361,9 @@ static int xgbe_read_mmd_regs(struct xgbe_prv_data *pdata, int prtad,
 	case XGBE_XPCS_ACCESS_V1:
 		return xgbe_read_mmd_regs_v1(pdata, prtad, mmd_reg);
 
+	case XGBE_XPCS_ACCESS_V3:
+		return xgbe_read_mmd_regs_v3(pdata, prtad, mmd_reg);
+
 	case XGBE_XPCS_ACCESS_V2:
 	default:
 		return xgbe_read_mmd_regs_v2(pdata, prtad, mmd_reg);
@@ -1287,6 +1377,9 @@ static void xgbe_write_mmd_regs(struct xgbe_prv_data *pdata, int prtad,
 	case XGBE_XPCS_ACCESS_V1:
 		return xgbe_write_mmd_regs_v1(pdata, prtad, mmd_reg, mmd_data);
 
+	case XGBE_XPCS_ACCESS_V3:
+		return xgbe_write_mmd_regs_v3(pdata, prtad, mmd_reg, mmd_data);
+
 	case XGBE_XPCS_ACCESS_V2:
 	default:
 		return xgbe_write_mmd_regs_v2(pdata, prtad, mmd_reg, mmd_data);
diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-pci.c b/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
index 8b0c1e450b7e..db3e8aac3339 100644
--- a/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
+++ b/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
@@ -295,15 +295,28 @@ static int xgbe_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 		/* Yellow Carp devices do not need rrc */
 		pdata->vdata->enable_rrc = 0;
 		break;
+	case 0x1630:
+		pdata->xpcs_window_def_reg = PCS_V2_RN_WINDOW_DEF;
+		pdata->xpcs_window_sel_reg = PCS_V2_RN_WINDOW_SELECT;
+		break;
 	default:
 		pdata->xpcs_window_def_reg = PCS_V2_WINDOW_DEF;
 		pdata->xpcs_window_sel_reg = PCS_V2_WINDOW_SELECT;
 		break;
 	}
-	pci_dev_put(rdev);
 
 	/* Configure the PCS indirect addressing support */
-	reg = XPCS32_IOREAD(pdata, pdata->xpcs_window_def_reg);
+	if (pdata->vdata->xpcs_access == XGBE_XPCS_ACCESS_V3) {
+		reg = XP_IOREAD(pdata, XP_PROP_0);
+		pdata->xphy_base = PCS_RN_SMN_BASE_ADDR +
+				   (PCS_RN_PORT_ADDR_SIZE * XP_GET_BITS(reg, XP_PROP_0, PORT_ID));
+		pci_write_config_dword(rdev, 0x60, pdata->xphy_base + (pdata->xpcs_window_def_reg));
+		pci_read_config_dword(rdev, 0x64, &reg);
+	} else {
+		reg = XPCS32_IOREAD(pdata, pdata->xpcs_window_def_reg);
+	}
+	pci_dev_put(rdev);
+
 	pdata->xpcs_window = XPCS_GET_BITS(reg, PCS_V2_WINDOW_DEF, OFFSET);
 	pdata->xpcs_window <<= 6;
 	pdata->xpcs_window_size = XPCS_GET_BITS(reg, PCS_V2_WINDOW_DEF, SIZE);
@@ -481,6 +494,22 @@ static int __maybe_unused xgbe_pci_resume(struct device *dev)
 	return ret;
 }
 
+static struct xgbe_version_data xgbe_v3 = {
+	.init_function_ptrs_phy_impl	= xgbe_init_function_ptrs_phy_v2,
+	.xpcs_access			= XGBE_XPCS_ACCESS_V3,
+	.mmc_64bit			= 1,
+	.tx_max_fifo_size		= 65536,
+	.rx_max_fifo_size		= 65536,
+	.tx_tstamp_workaround		= 1,
+	.ecc_support			= 1,
+	.i2c_support			= 1,
+	.irq_reissue_support		= 1,
+	.tx_desc_prefetch		= 5,
+	.rx_desc_prefetch		= 5,
+	.an_cdr_workaround		= 0,
+	.enable_rrc			= 0,
+};
+
 static struct xgbe_version_data xgbe_v2a = {
 	.init_function_ptrs_phy_impl	= xgbe_init_function_ptrs_phy_v2,
 	.xpcs_access			= XGBE_XPCS_ACCESS_V2,
@@ -518,6 +547,8 @@ static const struct pci_device_id xgbe_pci_table[] = {
 	  .driver_data = (kernel_ulong_t)&xgbe_v2a },
 	{ PCI_VDEVICE(AMD, 0x1459),
 	  .driver_data = (kernel_ulong_t)&xgbe_v2b },
+	{ PCI_VDEVICE(AMD, 0x1641),
+	  .driver_data = (kernel_ulong_t)&xgbe_v3 },
 	/* Last entry must be zero */
 	{ 0, }
 };
diff --git a/drivers/net/ethernet/amd/xgbe/xgbe.h b/drivers/net/ethernet/amd/xgbe/xgbe.h
index ad136ed493ed..dbb1faaf6185 100644
--- a/drivers/net/ethernet/amd/xgbe/xgbe.h
+++ b/drivers/net/ethernet/amd/xgbe/xgbe.h
@@ -347,6 +347,10 @@
 		    (_src)->link_modes._sname,		\
 		    __ETHTOOL_LINK_MODE_MASK_NBITS)
 
+/* Generic low and high masks */
+#define XGBE_GEN_HI_MASK	GENMASK(31, 16)
+#define XGBE_GEN_LO_MASK	GENMASK(15, 0)
+
 struct xgbe_prv_data;
 
 struct xgbe_packet_data {
@@ -565,6 +569,7 @@ enum xgbe_speed {
 enum xgbe_xpcs_access {
 	XGBE_XPCS_ACCESS_V1 = 0,
 	XGBE_XPCS_ACCESS_V2,
+	XGBE_XPCS_ACCESS_V3,
 };
 
 enum xgbe_an_mode {
@@ -1056,6 +1061,7 @@ struct xgbe_prv_data {
 	struct device *dev;
 	struct platform_device *phy_platdev;
 	struct device *phy_dev;
+	unsigned int xphy_base;
 
 	/* Version related data */
 	struct xgbe_version_data *vdata;
-- 
2.34.1

Re: [PATCH v3 net-next 2/3] amd-xgbe: add support for Crater ethernet device

[Tom Lendacky]

On 12/11/23 23:37, Raju Rangoju wrote:
> Add the necessary support to enable Crater ethernet device. Since the
> BAR1 address cannot be used to access the XPCS registers on Crater, use
> the pci_{read/write}_config_dword calls. Also, include the new pci device
> id 0x1641 to register Crater device with PCIe.
> 
> Co-developed-by: Sudheesh Mavila <sudheesh.mavila@amd.com>
> Signed-off-by: Sudheesh Mavila <sudheesh.mavila@amd.com>
> Signed-off-by: Raju Rangoju <Raju.Rangoju@amd.com>
> ---
>   drivers/net/ethernet/amd/xgbe/xgbe-common.h |  5 ++
>   drivers/net/ethernet/amd/xgbe/xgbe-dev.c    | 93 +++++++++++++++++++++
>   drivers/net/ethernet/amd/xgbe/xgbe-pci.c    | 35 +++++++-
>   drivers/net/ethernet/amd/xgbe/xgbe.h        |  6 ++
>   4 files changed, 137 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-common.h b/drivers/net/ethernet/amd/xgbe/xgbe-common.h
> index 3b70f6737633..e1f70f0528ef 100644
> --- a/drivers/net/ethernet/amd/xgbe/xgbe-common.h
> +++ b/drivers/net/ethernet/amd/xgbe/xgbe-common.h
> @@ -900,6 +900,11 @@
>   #define PCS_V2_RV_WINDOW_SELECT		0x1064
>   #define PCS_V2_YC_WINDOW_DEF		0x18060
>   #define PCS_V2_YC_WINDOW_SELECT		0x18064
> +#define PCS_V2_RN_WINDOW_DEF		0xF8078
> +#define PCS_V2_RN_WINDOW_SELECT		0xF807c

Should this be PCS_V3_ (here and below) ??

> +
> +#define PCS_RN_SMN_BASE_ADDR		0x11E00000
> +#define PCS_RN_PORT_ADDR_SIZE		0x100000

All hex characters should be consistent and in lower case.

>   
>   /* PCS register entry bit positions and sizes */
>   #define PCS_V2_WINDOW_DEF_OFFSET_INDEX	6
> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
> index 6cd003c24a64..a9eb2ffa9f73 100644
> --- a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
> +++ b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
> @@ -120,6 +120,7 @@
>   #include <linux/bitrev.h>
>   #include <linux/crc32.h>
>   #include <linux/crc32poly.h>
> +#include <linux/pci.h>
>   
>   #include "xgbe.h"
>   #include "xgbe-common.h"
> @@ -1165,6 +1166,92 @@ static unsigned int get_index_offset(struct xgbe_prv_data *pdata, unsigned int m
>   	return pdata->xpcs_window + (mmd_address & pdata->xpcs_window_mask);
>   }
>   
> +static int xgbe_read_mmd_regs_v3(struct xgbe_prv_data *pdata, int prtad,
> +				 int mmd_reg)
> +{
> +	unsigned int mmd_address, index, offset;
> +	struct pci_dev *rdev;
> +	unsigned long flags;
> +	int mmd_data;
> +
> +	rdev = pci_get_domain_bus_and_slot(0, 0, PCI_DEVFN(0, 0));
> +	if (!rdev)
> +		return 0;
> +
> +	mmd_address = get_mmd_address(pdata, mmd_reg);
> +
> +	/* The PCS registers are accessed using mmio. The underlying
> +	 * management interface uses indirect addressing to access the MMD
> +	 * register sets. This requires accessing of the PCS register in two
> +	 * phases, an address phase and a data phase.
> +	 *
> +	 * The mmio interface is based on 16-bit offsets and values. All
> +	 * register offsets must therefore be adjusted by left shifting the
> +	 * offset 1 bit and reading 16 bits of data.
> +	 */
> +	offset = get_index_offset(pdata, mmd_address, &index);
> +
> +	spin_lock_irqsave(&pdata->xpcs_lock, flags);
> +	pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + pdata->xpcs_window_sel_reg));
> +	pci_write_config_dword(rdev, 0x64, index);
> +	pci_write_config_dword(rdev, 0x60, pdata->xphy_base + offset);
> +	pci_read_config_dword(rdev, 0x64, &mmd_data);
> +	mmd_data = (offset % 4) ? FIELD_GET(XGBE_GEN_HI_MASK, mmd_data) :
> +				  FIELD_GET(XGBE_GEN_LO_MASK, mmd_data);
> +	pci_dev_put(rdev);
> +
> +	spin_unlock_irqrestore(&pdata->xpcs_lock, flags);
> +
> +	return mmd_data;
> +}
> +
> +static void xgbe_write_mmd_regs_v3(struct xgbe_prv_data *pdata, int prtad,
> +				   int mmd_reg, int mmd_data)
> +{
> +	unsigned int mmd_address, index, offset, ctr_mmd_data;
> +	struct pci_dev *rdev;
> +	unsigned long flags;
> +
> +	rdev = pci_get_domain_bus_and_slot(0, 0, PCI_DEVFN(0, 0));
> +	if (!rdev)
> +		return;
> +
> +	mmd_address = get_mmd_address(pdata, mmd_reg);
> +
> +	/* The PCS registers are accessed using mmio. The underlying
> +	 * management interface uses indirect addressing to access the MMD
> +	 * register sets. This requires accessing of the PCS register in two
> +	 * phases, an address phase and a data phase.
> +	 *
> +	 * The mmio interface is based on 16-bit offsets and values. All
> +	 * register offsets must therefore be adjusted by left shifting the
> +	 * offset 1 bit and writing 16 bits of data.
> +	 */
> +	offset = get_index_offset(pdata, mmd_address, &index);
> +
> +	spin_lock_irqsave(&pdata->xpcs_lock, flags);
> +	pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + pdata->xpcs_window_sel_reg));
> +	pci_write_config_dword(rdev, 0x64, index);
> +	pci_write_config_dword(rdev, 0x60, pdata->xphy_base + offset);
> +	pci_read_config_dword(rdev, 0x64, &ctr_mmd_data);
> +	if (offset % 4) {
> +		ctr_mmd_data = FIELD_PREP(XGBE_GEN_HI_MASK, mmd_data) |
> +			       FIELD_GET(XGBE_GEN_LO_MASK, ctr_mmd_data);
> +	} else {
> +		ctr_mmd_data = FIELD_PREP(XGBE_GEN_HI_MASK,
> +					  FIELD_GET(XGBE_GEN_HI_MASK, ctr_mmd_data)) |
> +			       FIELD_GET(XGBE_GEN_LO_MASK, mmd_data);
> +	}

Braces aren't necessary.

Also, just curious, what is the "ctr" prefix meant to imply here?


> +
> +	pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + pdata->xpcs_window_sel_reg));
> +	pci_write_config_dword(rdev, 0x64, index);
> +	pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + offset));
> +	pci_write_config_dword(rdev, 0x64, ctr_mmd_data);
> +	pci_dev_put(rdev);
> +
> +	spin_unlock_irqrestore(&pdata->xpcs_lock, flags);
> +}
> +
>   static int xgbe_read_mmd_regs_v2(struct xgbe_prv_data *pdata, int prtad,
>   				 int mmd_reg)
>   {
> @@ -1274,6 +1361,9 @@ static int xgbe_read_mmd_regs(struct xgbe_prv_data *pdata, int prtad,
>   	case XGBE_XPCS_ACCESS_V1:
>   		return xgbe_read_mmd_regs_v1(pdata, prtad, mmd_reg);
>   
> +	case XGBE_XPCS_ACCESS_V3:
> +		return xgbe_read_mmd_regs_v3(pdata, prtad, mmd_reg);
> +
>   	case XGBE_XPCS_ACCESS_V2:
>   	default:
>   		return xgbe_read_mmd_regs_v2(pdata, prtad, mmd_reg);
> @@ -1287,6 +1377,9 @@ static void xgbe_write_mmd_regs(struct xgbe_prv_data *pdata, int prtad,
>   	case XGBE_XPCS_ACCESS_V1:
>   		return xgbe_write_mmd_regs_v1(pdata, prtad, mmd_reg, mmd_data);
>   
> +	case XGBE_XPCS_ACCESS_V3:
> +		return xgbe_write_mmd_regs_v3(pdata, prtad, mmd_reg, mmd_data);
> +
>   	case XGBE_XPCS_ACCESS_V2:
>   	default:
>   		return xgbe_write_mmd_regs_v2(pdata, prtad, mmd_reg, mmd_data);
> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-pci.c b/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
> index 8b0c1e450b7e..db3e8aac3339 100644
> --- a/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
> +++ b/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
> @@ -295,15 +295,28 @@ static int xgbe_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>   		/* Yellow Carp devices do not need rrc */
>   		pdata->vdata->enable_rrc = 0;
>   		break;
> +	case 0x1630:

What PCI ID is this, it doesn't match the 0x1641 added below?

Thanks,
Tom

> +		pdata->xpcs_window_def_reg = PCS_V2_RN_WINDOW_DEF;
> +		pdata->xpcs_window_sel_reg = PCS_V2_RN_WINDOW_SELECT;
> +		break;
>   	default:
>   		pdata->xpcs_window_def_reg = PCS_V2_WINDOW_DEF;
>   		pdata->xpcs_window_sel_reg = PCS_V2_WINDOW_SELECT;
>   		break;
>   	}
> -	pci_dev_put(rdev);
>   
>   	/* Configure the PCS indirect addressing support */
> -	reg = XPCS32_IOREAD(pdata, pdata->xpcs_window_def_reg);
> +	if (pdata->vdata->xpcs_access == XGBE_XPCS_ACCESS_V3) {
> +		reg = XP_IOREAD(pdata, XP_PROP_0);
> +		pdata->xphy_base = PCS_RN_SMN_BASE_ADDR +
> +				   (PCS_RN_PORT_ADDR_SIZE * XP_GET_BITS(reg, XP_PROP_0, PORT_ID));
> +		pci_write_config_dword(rdev, 0x60, pdata->xphy_base + (pdata->xpcs_window_def_reg));
> +		pci_read_config_dword(rdev, 0x64, &reg);
> +	} else {
> +		reg = XPCS32_IOREAD(pdata, pdata->xpcs_window_def_reg);
> +	}
> +	pci_dev_put(rdev);
> +
>   	pdata->xpcs_window = XPCS_GET_BITS(reg, PCS_V2_WINDOW_DEF, OFFSET);
>   	pdata->xpcs_window <<= 6;
>   	pdata->xpcs_window_size = XPCS_GET_BITS(reg, PCS_V2_WINDOW_DEF, SIZE);
> @@ -481,6 +494,22 @@ static int __maybe_unused xgbe_pci_resume(struct device *dev)
>   	return ret;
>   }
>   
> +static struct xgbe_version_data xgbe_v3 = {
> +	.init_function_ptrs_phy_impl	= xgbe_init_function_ptrs_phy_v2,
> +	.xpcs_access			= XGBE_XPCS_ACCESS_V3,
> +	.mmc_64bit			= 1,
> +	.tx_max_fifo_size		= 65536,
> +	.rx_max_fifo_size		= 65536,
> +	.tx_tstamp_workaround		= 1,
> +	.ecc_support			= 1,
> +	.i2c_support			= 1,
> +	.irq_reissue_support		= 1,
> +	.tx_desc_prefetch		= 5,
> +	.rx_desc_prefetch		= 5,
> +	.an_cdr_workaround		= 0,
> +	.enable_rrc			= 0,
> +};
> +
>   static struct xgbe_version_data xgbe_v2a = {
>   	.init_function_ptrs_phy_impl	= xgbe_init_function_ptrs_phy_v2,
>   	.xpcs_access			= XGBE_XPCS_ACCESS_V2,
> @@ -518,6 +547,8 @@ static const struct pci_device_id xgbe_pci_table[] = {
>   	  .driver_data = (kernel_ulong_t)&xgbe_v2a },
>   	{ PCI_VDEVICE(AMD, 0x1459),
>   	  .driver_data = (kernel_ulong_t)&xgbe_v2b },
> +	{ PCI_VDEVICE(AMD, 0x1641),
> +	  .driver_data = (kernel_ulong_t)&xgbe_v3 },
>   	/* Last entry must be zero */
>   	{ 0, }
>   };
> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe.h b/drivers/net/ethernet/amd/xgbe/xgbe.h
> index ad136ed493ed..dbb1faaf6185 100644
> --- a/drivers/net/ethernet/amd/xgbe/xgbe.h
> +++ b/drivers/net/ethernet/amd/xgbe/xgbe.h
> @@ -347,6 +347,10 @@
>   		    (_src)->link_modes._sname,		\
>   		    __ETHTOOL_LINK_MODE_MASK_NBITS)
>   
> +/* Generic low and high masks */
> +#define XGBE_GEN_HI_MASK	GENMASK(31, 16)
> +#define XGBE_GEN_LO_MASK	GENMASK(15, 0)
> +
>   struct xgbe_prv_data;
>   
>   struct xgbe_packet_data {
> @@ -565,6 +569,7 @@ enum xgbe_speed {
>   enum xgbe_xpcs_access {
>   	XGBE_XPCS_ACCESS_V1 = 0,
>   	XGBE_XPCS_ACCESS_V2,
> +	XGBE_XPCS_ACCESS_V3,
>   };
>   
>   enum xgbe_an_mode {
> @@ -1056,6 +1061,7 @@ struct xgbe_prv_data {
>   	struct device *dev;
>   	struct platform_device *phy_platdev;
>   	struct device *phy_dev;
> +	unsigned int xphy_base;
>   
>   	/* Version related data */
>   	struct xgbe_version_data *vdata;

Re: [PATCH v3 net-next 2/3] amd-xgbe: add support for Crater ethernet device

[Raju Rangoju]

On 12/12/2023 9:02 PM, Tom Lendacky wrote:
> On 12/11/23 23:37, Raju Rangoju wrote:
>> Add the necessary support to enable Crater ethernet device. Since the
>> BAR1 address cannot be used to access the XPCS registers on Crater, use
>> the pci_{read/write}_config_dword calls. Also, include the new pci device
>> id 0x1641 to register Crater device with PCIe.
>>
>> Co-developed-by: Sudheesh Mavila <sudheesh.mavila@amd.com>
>> Signed-off-by: Sudheesh Mavila <sudheesh.mavila@amd.com>
>> Signed-off-by: Raju Rangoju <Raju.Rangoju@amd.com>
>> ---
>>   drivers/net/ethernet/amd/xgbe/xgbe-common.h |  5 ++
>>   drivers/net/ethernet/amd/xgbe/xgbe-dev.c    | 93 +++++++++++++++++++++
>>   drivers/net/ethernet/amd/xgbe/xgbe-pci.c    | 35 +++++++-
>>   drivers/net/ethernet/amd/xgbe/xgbe.h        |  6 ++
>>   4 files changed, 137 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-common.h 
>> b/drivers/net/ethernet/amd/xgbe/xgbe-common.h
>> index 3b70f6737633..e1f70f0528ef 100644
>> --- a/drivers/net/ethernet/amd/xgbe/xgbe-common.h
>> +++ b/drivers/net/ethernet/amd/xgbe/xgbe-common.h
>> @@ -900,6 +900,11 @@
>>   #define PCS_V2_RV_WINDOW_SELECT        0x1064
>>   #define PCS_V2_YC_WINDOW_DEF        0x18060
>>   #define PCS_V2_YC_WINDOW_SELECT        0x18064
>> +#define PCS_V2_RN_WINDOW_DEF        0xF8078
>> +#define PCS_V2_RN_WINDOW_SELECT        0xF807c
> 
> Should this be PCS_V3_ (here and below) ??

Yes, it should be PCS_V3. I'll take care of it in next version.

> 
>> +
>> +#define PCS_RN_SMN_BASE_ADDR        0x11E00000
>> +#define PCS_RN_PORT_ADDR_SIZE        0x100000
> 
> All hex characters should be consistent and in lower case.

Sure

> 
>>   /* PCS register entry bit positions and sizes */
>>   #define PCS_V2_WINDOW_DEF_OFFSET_INDEX    6
>> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c 
>> b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
>> index 6cd003c24a64..a9eb2ffa9f73 100644
>> --- a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
>> +++ b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
>> @@ -120,6 +120,7 @@
>>   #include <linux/bitrev.h>
>>   #include <linux/crc32.h>
>>   #include <linux/crc32poly.h>
>> +#include <linux/pci.h>
>>   #include "xgbe.h"
>>   #include "xgbe-common.h"
>> @@ -1165,6 +1166,92 @@ static unsigned int get_index_offset(struct 
>> xgbe_prv_data *pdata, unsigned int m
>>       return pdata->xpcs_window + (mmd_address & 
>> pdata->xpcs_window_mask);
>>   }
>> +static int xgbe_read_mmd_regs_v3(struct xgbe_prv_data *pdata, int prtad,
>> +                 int mmd_reg)
>> +{
>> +    unsigned int mmd_address, index, offset;
>> +    struct pci_dev *rdev;
>> +    unsigned long flags;
>> +    int mmd_data;
>> +
>> +    rdev = pci_get_domain_bus_and_slot(0, 0, PCI_DEVFN(0, 0));
>> +    if (!rdev)
>> +        return 0;
>> +
>> +    mmd_address = get_mmd_address(pdata, mmd_reg);
>> +
>> +    /* The PCS registers are accessed using mmio. The underlying
>> +     * management interface uses indirect addressing to access the MMD
>> +     * register sets. This requires accessing of the PCS register in two
>> +     * phases, an address phase and a data phase.
>> +     *
>> +     * The mmio interface is based on 16-bit offsets and values. All
>> +     * register offsets must therefore be adjusted by left shifting the
>> +     * offset 1 bit and reading 16 bits of data.
>> +     */
>> +    offset = get_index_offset(pdata, mmd_address, &index);
>> +
>> +    spin_lock_irqsave(&pdata->xpcs_lock, flags);
>> +    pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + 
>> pdata->xpcs_window_sel_reg));
>> +    pci_write_config_dword(rdev, 0x64, index);
>> +    pci_write_config_dword(rdev, 0x60, pdata->xphy_base + offset);
>> +    pci_read_config_dword(rdev, 0x64, &mmd_data);
>> +    mmd_data = (offset % 4) ? FIELD_GET(XGBE_GEN_HI_MASK, mmd_data) :
>> +                  FIELD_GET(XGBE_GEN_LO_MASK, mmd_data);
>> +    pci_dev_put(rdev);
>> +
>> +    spin_unlock_irqrestore(&pdata->xpcs_lock, flags);
>> +
>> +    return mmd_data;
>> +}
>> +
>> +static void xgbe_write_mmd_regs_v3(struct xgbe_prv_data *pdata, int 
>> prtad,
>> +                   int mmd_reg, int mmd_data)
>> +{
>> +    unsigned int mmd_address, index, offset, ctr_mmd_data;
>> +    struct pci_dev *rdev;
>> +    unsigned long flags;
>> +
>> +    rdev = pci_get_domain_bus_and_slot(0, 0, PCI_DEVFN(0, 0));
>> +    if (!rdev)
>> +        return;
>> +
>> +    mmd_address = get_mmd_address(pdata, mmd_reg);
>> +
>> +    /* The PCS registers are accessed using mmio. The underlying
>> +     * management interface uses indirect addressing to access the MMD
>> +     * register sets. This requires accessing of the PCS register in two
>> +     * phases, an address phase and a data phase.
>> +     *
>> +     * The mmio interface is based on 16-bit offsets and values. All
>> +     * register offsets must therefore be adjusted by left shifting the
>> +     * offset 1 bit and writing 16 bits of data.
>> +     */
>> +    offset = get_index_offset(pdata, mmd_address, &index);
>> +
>> +    spin_lock_irqsave(&pdata->xpcs_lock, flags);
>> +    pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + 
>> pdata->xpcs_window_sel_reg));
>> +    pci_write_config_dword(rdev, 0x64, index);
>> +    pci_write_config_dword(rdev, 0x60, pdata->xphy_base + offset);
>> +    pci_read_config_dword(rdev, 0x64, &ctr_mmd_data);
>> +    if (offset % 4) {
>> +        ctr_mmd_data = FIELD_PREP(XGBE_GEN_HI_MASK, mmd_data) |
>> +                   FIELD_GET(XGBE_GEN_LO_MASK, ctr_mmd_data);
>> +    } else {
>> +        ctr_mmd_data = FIELD_PREP(XGBE_GEN_HI_MASK,
>> +                      FIELD_GET(XGBE_GEN_HI_MASK, ctr_mmd_data)) |
>> +                   FIELD_GET(XGBE_GEN_LO_MASK, mmd_data);
>> +    }
> 
> Braces aren't necessary.
> 
> Also, just curious, what is the "ctr" prefix meant to imply here?

ctr = crater. Will use the full name instead to avoid confusion.

> 
> 
>> +
>> +    pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + 
>> pdata->xpcs_window_sel_reg));
>> +    pci_write_config_dword(rdev, 0x64, index);
>> +    pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + offset));
>> +    pci_write_config_dword(rdev, 0x64, ctr_mmd_data);
>> +    pci_dev_put(rdev);
>> +
>> +    spin_unlock_irqrestore(&pdata->xpcs_lock, flags);
>> +}
>> +
>>   static int xgbe_read_mmd_regs_v2(struct xgbe_prv_data *pdata, int 
>> prtad,
>>                    int mmd_reg)
>>   {
>> @@ -1274,6 +1361,9 @@ static int xgbe_read_mmd_regs(struct 
>> xgbe_prv_data *pdata, int prtad,
>>       case XGBE_XPCS_ACCESS_V1:
>>           return xgbe_read_mmd_regs_v1(pdata, prtad, mmd_reg);
>> +    case XGBE_XPCS_ACCESS_V3:
>> +        return xgbe_read_mmd_regs_v3(pdata, prtad, mmd_reg);
>> +
>>       case XGBE_XPCS_ACCESS_V2:
>>       default:
>>           return xgbe_read_mmd_regs_v2(pdata, prtad, mmd_reg);
>> @@ -1287,6 +1377,9 @@ static void xgbe_write_mmd_regs(struct 
>> xgbe_prv_data *pdata, int prtad,
>>       case XGBE_XPCS_ACCESS_V1:
>>           return xgbe_write_mmd_regs_v1(pdata, prtad, mmd_reg, mmd_data);
>> +    case XGBE_XPCS_ACCESS_V3:
>> +        return xgbe_write_mmd_regs_v3(pdata, prtad, mmd_reg, mmd_data);
>> +
>>       case XGBE_XPCS_ACCESS_V2:
>>       default:
>>           return xgbe_write_mmd_regs_v2(pdata, prtad, mmd_reg, mmd_data);
>> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-pci.c 
>> b/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
>> index 8b0c1e450b7e..db3e8aac3339 100644
>> --- a/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
>> +++ b/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
>> @@ -295,15 +295,28 @@ static int xgbe_pci_probe(struct pci_dev *pdev, 
>> const struct pci_device_id *id)
>>           /* Yellow Carp devices do not need rrc */
>>           pdata->vdata->enable_rrc = 0;
>>           break;
>> +    case 0x1630:
> 
> What PCI ID is this, it doesn't match the 0x1641 added below?

0x1630 is root hub

> 
> Thanks,
> Tom
> 
>> +        pdata->xpcs_window_def_reg = PCS_V2_RN_WINDOW_DEF;
>> +        pdata->xpcs_window_sel_reg = PCS_V2_RN_WINDOW_SELECT;
>> +        break;
>>       default:
>>           pdata->xpcs_window_def_reg = PCS_V2_WINDOW_DEF;
>>           pdata->xpcs_window_sel_reg = PCS_V2_WINDOW_SELECT;
>>           break;
>>       }
>> -    pci_dev_put(rdev);
>>       /* Configure the PCS indirect addressing support */
>> -    reg = XPCS32_IOREAD(pdata, pdata->xpcs_window_def_reg);
>> +    if (pdata->vdata->xpcs_access == XGBE_XPCS_ACCESS_V3) {
>> +        reg = XP_IOREAD(pdata, XP_PROP_0);
>> +        pdata->xphy_base = PCS_RN_SMN_BASE_ADDR +
>> +                   (PCS_RN_PORT_ADDR_SIZE * XP_GET_BITS(reg, 
>> XP_PROP_0, PORT_ID));
>> +        pci_write_config_dword(rdev, 0x60, pdata->xphy_base + 
>> (pdata->xpcs_window_def_reg));
>> +        pci_read_config_dword(rdev, 0x64, &reg);
>> +    } else {
>> +        reg = XPCS32_IOREAD(pdata, pdata->xpcs_window_def_reg);
>> +    }
>> +    pci_dev_put(rdev);
>> +
>>       pdata->xpcs_window = XPCS_GET_BITS(reg, PCS_V2_WINDOW_DEF, OFFSET);
>>       pdata->xpcs_window <<= 6;
>>       pdata->xpcs_window_size = XPCS_GET_BITS(reg, PCS_V2_WINDOW_DEF, 
>> SIZE);
>> @@ -481,6 +494,22 @@ static int __maybe_unused xgbe_pci_resume(struct 
>> device *dev)
>>       return ret;
>>   }
>> +static struct xgbe_version_data xgbe_v3 = {
>> +    .init_function_ptrs_phy_impl    = xgbe_init_function_ptrs_phy_v2,
>> +    .xpcs_access            = XGBE_XPCS_ACCESS_V3,
>> +    .mmc_64bit            = 1,
>> +    .tx_max_fifo_size        = 65536,
>> +    .rx_max_fifo_size        = 65536,
>> +    .tx_tstamp_workaround        = 1,
>> +    .ecc_support            = 1,
>> +    .i2c_support            = 1,
>> +    .irq_reissue_support        = 1,
>> +    .tx_desc_prefetch        = 5,
>> +    .rx_desc_prefetch        = 5,
>> +    .an_cdr_workaround        = 0,
>> +    .enable_rrc            = 0,
>> +};
>> +
>>   static struct xgbe_version_data xgbe_v2a = {
>>       .init_function_ptrs_phy_impl    = xgbe_init_function_ptrs_phy_v2,
>>       .xpcs_access            = XGBE_XPCS_ACCESS_V2,
>> @@ -518,6 +547,8 @@ static const struct pci_device_id xgbe_pci_table[] 
>> = {
>>         .driver_data = (kernel_ulong_t)&xgbe_v2a },
>>       { PCI_VDEVICE(AMD, 0x1459),
>>         .driver_data = (kernel_ulong_t)&xgbe_v2b },
>> +    { PCI_VDEVICE(AMD, 0x1641),
>> +      .driver_data = (kernel_ulong_t)&xgbe_v3 },
>>       /* Last entry must be zero */
>>       { 0, }
>>   };
>> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe.h 
>> b/drivers/net/ethernet/amd/xgbe/xgbe.h
>> index ad136ed493ed..dbb1faaf6185 100644
>> --- a/drivers/net/ethernet/amd/xgbe/xgbe.h
>> +++ b/drivers/net/ethernet/amd/xgbe/xgbe.h
>> @@ -347,6 +347,10 @@
>>               (_src)->link_modes._sname,        \
>>               __ETHTOOL_LINK_MODE_MASK_NBITS)
>> +/* Generic low and high masks */
>> +#define XGBE_GEN_HI_MASK    GENMASK(31, 16)
>> +#define XGBE_GEN_LO_MASK    GENMASK(15, 0)
>> +
>>   struct xgbe_prv_data;
>>   struct xgbe_packet_data {
>> @@ -565,6 +569,7 @@ enum xgbe_speed {
>>   enum xgbe_xpcs_access {
>>       XGBE_XPCS_ACCESS_V1 = 0,
>>       XGBE_XPCS_ACCESS_V2,
>> +    XGBE_XPCS_ACCESS_V3,
>>   };
>>   enum xgbe_an_mode {
>> @@ -1056,6 +1061,7 @@ struct xgbe_prv_data {
>>       struct device *dev;
>>       struct platform_device *phy_platdev;
>>       struct device *phy_dev;
>> +    unsigned int xphy_base;
>>       /* Version related data */
>>       struct xgbe_version_data *vdata;

Re: [PATCH v3 net-next 2/3] amd-xgbe: add support for Crater ethernet device

[Tom Lendacky]

On 12/13/23 10:25, Raju Rangoju wrote:
> On 12/12/2023 9:02 PM, Tom Lendacky wrote:
>> On 12/11/23 23:37, Raju Rangoju wrote:
>>> Add the necessary support to enable Crater ethernet device. Since the
>>> BAR1 address cannot be used to access the XPCS registers on Crater, use
>>> the pci_{read/write}_config_dword calls. Also, include the new pci device
>>> id 0x1641 to register Crater device with PCIe.
>>>
>>> Co-developed-by: Sudheesh Mavila <sudheesh.mavila@amd.com>
>>> Signed-off-by: Sudheesh Mavila <sudheesh.mavila@amd.com>
>>> Signed-off-by: Raju Rangoju <Raju.Rangoju@amd.com>
>>> ---
>>>   drivers/net/ethernet/amd/xgbe/xgbe-common.h |  5 ++
>>>   drivers/net/ethernet/amd/xgbe/xgbe-dev.c    | 93 +++++++++++++++++++++
>>>   drivers/net/ethernet/amd/xgbe/xgbe-pci.c    | 35 +++++++-
>>>   drivers/net/ethernet/amd/xgbe/xgbe.h        |  6 ++
>>>   4 files changed, 137 insertions(+), 2 deletions(-)
>>>

>>> +
>>> +    spin_lock_irqsave(&pdata->xpcs_lock, flags);
>>> +    pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + 
>>> pdata->xpcs_window_sel_reg));
>>> +    pci_write_config_dword(rdev, 0x64, index);
>>> +    pci_write_config_dword(rdev, 0x60, pdata->xphy_base + offset);
>>> +    pci_read_config_dword(rdev, 0x64, &ctr_mmd_data);
>>> +    if (offset % 4) {
>>> +        ctr_mmd_data = FIELD_PREP(XGBE_GEN_HI_MASK, mmd_data) |
>>> +                   FIELD_GET(XGBE_GEN_LO_MASK, ctr_mmd_data);
>>> +    } else {
>>> +        ctr_mmd_data = FIELD_PREP(XGBE_GEN_HI_MASK,
>>> +                      FIELD_GET(XGBE_GEN_HI_MASK, ctr_mmd_data)) |
>>> +                   FIELD_GET(XGBE_GEN_LO_MASK, mmd_data);
>>> +    }
>>
>> Braces aren't necessary.
>>
>> Also, just curious, what is the "ctr" prefix meant to imply here?
> 
> ctr = crater. Will use the full name instead to avoid confusion.

You should probably make it generic and use pci_ or tmp_ (or smn_ if you 
end up squashing patches 2 and 3), since this may not be "crater" specific 
if this is used by another device ID in the future.

> 
>>
>>

>>> @@ -295,15 +295,28 @@ static int xgbe_pci_probe(struct pci_dev *pdev, 
>>> const struct pci_device_id *id)
>>>           /* Yellow Carp devices do not need rrc */
>>>           pdata->vdata->enable_rrc = 0;
>>>           break;
>>> +    case 0x1630:
>>
>> What PCI ID is this, it doesn't match the 0x1641 added below?
> 
> 0x1630 is root hub

Ah, right, sorry for the noise.

Thanks,
Tom

> 

[PATCH v3 net-next 3/3] amd-xgbe: use smn functions to avoid race

[Raju Rangoju]

Some of the ethernet add-in-cards have dual PHY but share a single MDIO
line (between the ports). In such cases, link inconsistencies are
noticed during the heavy traffic and during reboot stress tests.

So, use the SMN calls to avoid the race conditions.

Signed-off-by: Raju Rangoju <Raju.Rangoju@amd.com>
---
 drivers/net/ethernet/amd/xgbe/xgbe-dev.c | 33 ++++++------------------
 drivers/net/ethernet/amd/xgbe/xgbe-pci.c | 10 +++----
 drivers/net/ethernet/amd/xgbe/xgbe-smn.h | 27 +++++++++++++++++++
 drivers/net/ethernet/amd/xgbe/xgbe.h     |  2 +-
 4 files changed, 41 insertions(+), 31 deletions(-)
 create mode 100644 drivers/net/ethernet/amd/xgbe/xgbe-smn.h

diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
index a9eb2ffa9f73..8d8876ab258c 100644
--- a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
+++ b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
@@ -124,6 +124,7 @@
 
 #include "xgbe.h"
 #include "xgbe-common.h"
+#include "xgbe-smn.h"
 
 static inline unsigned int xgbe_get_max_frame(struct xgbe_prv_data *pdata)
 {
@@ -1170,14 +1171,9 @@ static int xgbe_read_mmd_regs_v3(struct xgbe_prv_data *pdata, int prtad,
 				 int mmd_reg)
 {
 	unsigned int mmd_address, index, offset;
-	struct pci_dev *rdev;
 	unsigned long flags;
 	int mmd_data;
 
-	rdev = pci_get_domain_bus_and_slot(0, 0, PCI_DEVFN(0, 0));
-	if (!rdev)
-		return 0;
-
 	mmd_address = get_mmd_address(pdata, mmd_reg);
 
 	/* The PCS registers are accessed using mmio. The underlying
@@ -1192,13 +1188,10 @@ static int xgbe_read_mmd_regs_v3(struct xgbe_prv_data *pdata, int prtad,
 	offset = get_index_offset(pdata, mmd_address, &index);
 
 	spin_lock_irqsave(&pdata->xpcs_lock, flags);
-	pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + pdata->xpcs_window_sel_reg));
-	pci_write_config_dword(rdev, 0x64, index);
-	pci_write_config_dword(rdev, 0x60, pdata->xphy_base + offset);
-	pci_read_config_dword(rdev, 0x64, &mmd_data);
+	amd_smn_write(0, (pdata->smn_base + pdata->xpcs_window_sel_reg), index);
+	amd_smn_read(0, pdata->smn_base + offset, &mmd_data);
 	mmd_data = (offset % 4) ? FIELD_GET(XGBE_GEN_HI_MASK, mmd_data) :
 				  FIELD_GET(XGBE_GEN_LO_MASK, mmd_data);
-	pci_dev_put(rdev);
 
 	spin_unlock_irqrestore(&pdata->xpcs_lock, flags);
 
@@ -1209,13 +1202,8 @@ static void xgbe_write_mmd_regs_v3(struct xgbe_prv_data *pdata, int prtad,
 				   int mmd_reg, int mmd_data)
 {
 	unsigned int mmd_address, index, offset, ctr_mmd_data;
-	struct pci_dev *rdev;
 	unsigned long flags;
 
-	rdev = pci_get_domain_bus_and_slot(0, 0, PCI_DEVFN(0, 0));
-	if (!rdev)
-		return;
-
 	mmd_address = get_mmd_address(pdata, mmd_reg);
 
 	/* The PCS registers are accessed using mmio. The underlying
@@ -1230,10 +1218,9 @@ static void xgbe_write_mmd_regs_v3(struct xgbe_prv_data *pdata, int prtad,
 	offset = get_index_offset(pdata, mmd_address, &index);
 
 	spin_lock_irqsave(&pdata->xpcs_lock, flags);
-	pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + pdata->xpcs_window_sel_reg));
-	pci_write_config_dword(rdev, 0x64, index);
-	pci_write_config_dword(rdev, 0x60, pdata->xphy_base + offset);
-	pci_read_config_dword(rdev, 0x64, &ctr_mmd_data);
+	amd_smn_write(0, (pdata->smn_base + pdata->xpcs_window_sel_reg), index);
+	amd_smn_read(0, pdata->smn_base + offset, &ctr_mmd_data);
+
 	if (offset % 4) {
 		ctr_mmd_data = FIELD_PREP(XGBE_GEN_HI_MASK, mmd_data) |
 			       FIELD_GET(XGBE_GEN_LO_MASK, ctr_mmd_data);
@@ -1243,12 +1230,8 @@ static void xgbe_write_mmd_regs_v3(struct xgbe_prv_data *pdata, int prtad,
 			       FIELD_GET(XGBE_GEN_LO_MASK, mmd_data);
 	}
 
-	pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + pdata->xpcs_window_sel_reg));
-	pci_write_config_dword(rdev, 0x64, index);
-	pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + offset));
-	pci_write_config_dword(rdev, 0x64, ctr_mmd_data);
-	pci_dev_put(rdev);
-
+	amd_smn_write(0, (pdata->smn_base + pdata->xpcs_window_sel_reg), index);
+	amd_smn_write(0, (pdata->smn_base + offset), ctr_mmd_data);
 	spin_unlock_irqrestore(&pdata->xpcs_lock, flags);
 }
 
diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-pci.c b/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
index db3e8aac3339..135128b5be90 100644
--- a/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
+++ b/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
@@ -121,6 +121,7 @@
 
 #include "xgbe.h"
 #include "xgbe-common.h"
+#include "xgbe-smn.h"
 
 static int xgbe_config_multi_msi(struct xgbe_prv_data *pdata)
 {
@@ -304,18 +305,17 @@ static int xgbe_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 		pdata->xpcs_window_sel_reg = PCS_V2_WINDOW_SELECT;
 		break;
 	}
+	pci_dev_put(rdev);
 
 	/* Configure the PCS indirect addressing support */
 	if (pdata->vdata->xpcs_access == XGBE_XPCS_ACCESS_V3) {
 		reg = XP_IOREAD(pdata, XP_PROP_0);
-		pdata->xphy_base = PCS_RN_SMN_BASE_ADDR +
-				   (PCS_RN_PORT_ADDR_SIZE * XP_GET_BITS(reg, XP_PROP_0, PORT_ID));
-		pci_write_config_dword(rdev, 0x60, pdata->xphy_base + (pdata->xpcs_window_def_reg));
-		pci_read_config_dword(rdev, 0x64, &reg);
+		pdata->smn_base = PCS_RN_SMN_BASE_ADDR +
+				  (PCS_RN_PORT_ADDR_SIZE * XP_GET_BITS(reg, XP_PROP_0, PORT_ID));
+		amd_smn_read(0, pdata->smn_base + (pdata->xpcs_window_def_reg), &reg);
 	} else {
 		reg = XPCS32_IOREAD(pdata, pdata->xpcs_window_def_reg);
 	}
-	pci_dev_put(rdev);
 
 	pdata->xpcs_window = XPCS_GET_BITS(reg, PCS_V2_WINDOW_DEF, OFFSET);
 	pdata->xpcs_window <<= 6;
diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-smn.h b/drivers/net/ethernet/amd/xgbe/xgbe-smn.h
new file mode 100644
index 000000000000..bd25ddc7c869
--- /dev/null
+++ b/drivers/net/ethernet/amd/xgbe/xgbe-smn.h
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * AMD 10Gb Ethernet driver
+ *
+ * Copyright (c) 2023, Advanced Micro Devices, Inc.
+ * All Rights Reserved.
+ *
+ * Author: Raju Rangoju <Raju.Rangoju@amd.com>
+ */
+
+#ifdef CONFIG_AMD_NB
+
+#include <asm/amd_nb.h>
+
+#else
+
+static inline int amd_smn_write(u16 node, u32 address, u32 value)
+{
+	return -ENODEV;
+}
+
+static inline int amd_smn_read(u16 node, u32 address, u32 *value)
+{
+	return -ENODEV;
+}
+
+#endif
diff --git a/drivers/net/ethernet/amd/xgbe/xgbe.h b/drivers/net/ethernet/amd/xgbe/xgbe.h
index dbb1faaf6185..ba45ab0adb8c 100644
--- a/drivers/net/ethernet/amd/xgbe/xgbe.h
+++ b/drivers/net/ethernet/amd/xgbe/xgbe.h
@@ -1061,7 +1061,7 @@ struct xgbe_prv_data {
 	struct device *dev;
 	struct platform_device *phy_platdev;
 	struct device *phy_dev;
-	unsigned int xphy_base;
+	unsigned int smn_base;
 
 	/* Version related data */
 	struct xgbe_version_data *vdata;
-- 
2.34.1

Re: [PATCH v3 net-next 3/3] amd-xgbe: use smn functions to avoid race

[Tom Lendacky]

On 12/11/23 23:37, Raju Rangoju wrote:
> Some of the ethernet add-in-cards have dual PHY but share a single MDIO
> line (between the ports). In such cases, link inconsistencies are
> noticed during the heavy traffic and during reboot stress tests.
> 
> So, use the SMN calls to avoid the race conditions.

So this patch replaces all the PCI accesses you added in patch #2, so why 
not just do this from the start?

> 
> Signed-off-by: Raju Rangoju <Raju.Rangoju@amd.com>
> ---
>   drivers/net/ethernet/amd/xgbe/xgbe-dev.c | 33 ++++++------------------
>   drivers/net/ethernet/amd/xgbe/xgbe-pci.c | 10 +++----
>   drivers/net/ethernet/amd/xgbe/xgbe-smn.h | 27 +++++++++++++++++++
>   drivers/net/ethernet/amd/xgbe/xgbe.h     |  2 +-
>   4 files changed, 41 insertions(+), 31 deletions(-)
>   create mode 100644 drivers/net/ethernet/amd/xgbe/xgbe-smn.h
> 
> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
> index a9eb2ffa9f73..8d8876ab258c 100644
> --- a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
> +++ b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
> @@ -124,6 +124,7 @@
>   
>   #include "xgbe.h"
>   #include "xgbe-common.h"
> +#include "xgbe-smn.h"
>   
>   static inline unsigned int xgbe_get_max_frame(struct xgbe_prv_data *pdata)
>   {
> @@ -1170,14 +1171,9 @@ static int xgbe_read_mmd_regs_v3(struct xgbe_prv_data *pdata, int prtad,
>   				 int mmd_reg)
>   {
>   	unsigned int mmd_address, index, offset;
> -	struct pci_dev *rdev;
>   	unsigned long flags;
>   	int mmd_data;
>   
> -	rdev = pci_get_domain_bus_and_slot(0, 0, PCI_DEVFN(0, 0));
> -	if (!rdev)
> -		return 0;
> -
>   	mmd_address = get_mmd_address(pdata, mmd_reg);
>   
>   	/* The PCS registers are accessed using mmio. The underlying
> @@ -1192,13 +1188,10 @@ static int xgbe_read_mmd_regs_v3(struct xgbe_prv_data *pdata, int prtad,
>   	offset = get_index_offset(pdata, mmd_address, &index);
>   
>   	spin_lock_irqsave(&pdata->xpcs_lock, flags);
> -	pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + pdata->xpcs_window_sel_reg));
> -	pci_write_config_dword(rdev, 0x64, index);
> -	pci_write_config_dword(rdev, 0x60, pdata->xphy_base + offset);
> -	pci_read_config_dword(rdev, 0x64, &mmd_data);
> +	amd_smn_write(0, (pdata->smn_base + pdata->xpcs_window_sel_reg), index);
> +	amd_smn_read(0, pdata->smn_base + offset, &mmd_data);
>   	mmd_data = (offset % 4) ? FIELD_GET(XGBE_GEN_HI_MASK, mmd_data) :
>   				  FIELD_GET(XGBE_GEN_LO_MASK, mmd_data);
> -	pci_dev_put(rdev);
>   
>   	spin_unlock_irqrestore(&pdata->xpcs_lock, flags);
>   
> @@ -1209,13 +1202,8 @@ static void xgbe_write_mmd_regs_v3(struct xgbe_prv_data *pdata, int prtad,
>   				   int mmd_reg, int mmd_data)
>   {
>   	unsigned int mmd_address, index, offset, ctr_mmd_data;
> -	struct pci_dev *rdev;
>   	unsigned long flags;
>   
> -	rdev = pci_get_domain_bus_and_slot(0, 0, PCI_DEVFN(0, 0));
> -	if (!rdev)
> -		return;
> -
>   	mmd_address = get_mmd_address(pdata, mmd_reg);
>   
>   	/* The PCS registers are accessed using mmio. The underlying
> @@ -1230,10 +1218,9 @@ static void xgbe_write_mmd_regs_v3(struct xgbe_prv_data *pdata, int prtad,
>   	offset = get_index_offset(pdata, mmd_address, &index);
>   
>   	spin_lock_irqsave(&pdata->xpcs_lock, flags);
> -	pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + pdata->xpcs_window_sel_reg));
> -	pci_write_config_dword(rdev, 0x64, index);
> -	pci_write_config_dword(rdev, 0x60, pdata->xphy_base + offset);
> -	pci_read_config_dword(rdev, 0x64, &ctr_mmd_data);
> +	amd_smn_write(0, (pdata->smn_base + pdata->xpcs_window_sel_reg), index);
> +	amd_smn_read(0, pdata->smn_base + offset, &ctr_mmd_data);
> +
>   	if (offset % 4) {
>   		ctr_mmd_data = FIELD_PREP(XGBE_GEN_HI_MASK, mmd_data) |
>   			       FIELD_GET(XGBE_GEN_LO_MASK, ctr_mmd_data);
> @@ -1243,12 +1230,8 @@ static void xgbe_write_mmd_regs_v3(struct xgbe_prv_data *pdata, int prtad,
>   			       FIELD_GET(XGBE_GEN_LO_MASK, mmd_data);
>   	}
>   
> -	pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + pdata->xpcs_window_sel_reg));
> -	pci_write_config_dword(rdev, 0x64, index);
> -	pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + offset));
> -	pci_write_config_dword(rdev, 0x64, ctr_mmd_data);
> -	pci_dev_put(rdev);
> -
> +	amd_smn_write(0, (pdata->smn_base + pdata->xpcs_window_sel_reg), index);
> +	amd_smn_write(0, (pdata->smn_base + offset), ctr_mmd_data);
>   	spin_unlock_irqrestore(&pdata->xpcs_lock, flags);
>   }
>   
> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-pci.c b/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
> index db3e8aac3339..135128b5be90 100644
> --- a/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
> +++ b/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
> @@ -121,6 +121,7 @@
>   
>   #include "xgbe.h"
>   #include "xgbe-common.h"
> +#include "xgbe-smn.h"
>   
>   static int xgbe_config_multi_msi(struct xgbe_prv_data *pdata)
>   {
> @@ -304,18 +305,17 @@ static int xgbe_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>   		pdata->xpcs_window_sel_reg = PCS_V2_WINDOW_SELECT;
>   		break;
>   	}
> +	pci_dev_put(rdev);
>   
>   	/* Configure the PCS indirect addressing support */
>   	if (pdata->vdata->xpcs_access == XGBE_XPCS_ACCESS_V3) {
>   		reg = XP_IOREAD(pdata, XP_PROP_0);
> -		pdata->xphy_base = PCS_RN_SMN_BASE_ADDR +
> -				   (PCS_RN_PORT_ADDR_SIZE * XP_GET_BITS(reg, XP_PROP_0, PORT_ID));
> -		pci_write_config_dword(rdev, 0x60, pdata->xphy_base + (pdata->xpcs_window_def_reg));
> -		pci_read_config_dword(rdev, 0x64, &reg);
> +		pdata->smn_base = PCS_RN_SMN_BASE_ADDR +
> +				  (PCS_RN_PORT_ADDR_SIZE * XP_GET_BITS(reg, XP_PROP_0, PORT_ID));
> +		amd_smn_read(0, pdata->smn_base + (pdata->xpcs_window_def_reg), &reg);
>   	} else {
>   		reg = XPCS32_IOREAD(pdata, pdata->xpcs_window_def_reg);
>   	}
> -	pci_dev_put(rdev);
>   
>   	pdata->xpcs_window = XPCS_GET_BITS(reg, PCS_V2_WINDOW_DEF, OFFSET);
>   	pdata->xpcs_window <<= 6;
> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-smn.h b/drivers/net/ethernet/amd/xgbe/xgbe-smn.h
> new file mode 100644
> index 000000000000..bd25ddc7c869
> --- /dev/null
> +++ b/drivers/net/ethernet/amd/xgbe/xgbe-smn.h
> @@ -0,0 +1,27 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * AMD 10Gb Ethernet driver
> + *
> + * Copyright (c) 2023, Advanced Micro Devices, Inc.
> + * All Rights Reserved.
> + *
> + * Author: Raju Rangoju <Raju.Rangoju@amd.com>
> + */

Shouldn't this license match the license in all the other files?

Also, you need header protection here, e.g.:

#ifndef __XGBE_SMN_H__
#define __XGBE_SMN_H__

and a #endif at the end.

Thanks,
Tom

> +
> +#ifdef CONFIG_AMD_NB
> +
> +#include <asm/amd_nb.h>
> +
> +#else
> +
> +static inline int amd_smn_write(u16 node, u32 address, u32 value)
> +{
> +	return -ENODEV;
> +}
> +
> +static inline int amd_smn_read(u16 node, u32 address, u32 *value)
> +{
> +	return -ENODEV;
> +}
> +
> +#endif
> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe.h b/drivers/net/ethernet/amd/xgbe/xgbe.h
> index dbb1faaf6185..ba45ab0adb8c 100644
> --- a/drivers/net/ethernet/amd/xgbe/xgbe.h
> +++ b/drivers/net/ethernet/amd/xgbe/xgbe.h
> @@ -1061,7 +1061,7 @@ struct xgbe_prv_data {
>   	struct device *dev;
>   	struct platform_device *phy_platdev;
>   	struct device *phy_dev;
> -	unsigned int xphy_base;
> +	unsigned int smn_base;
>   
>   	/* Version related data */
>   	struct xgbe_version_data *vdata;

Re: [PATCH v3 net-next 3/3] amd-xgbe: use smn functions to avoid race

[Raju Rangoju]

On 12/12/2023 9:08 PM, Tom Lendacky wrote:
> On 12/11/23 23:37, Raju Rangoju wrote:
>> Some of the ethernet add-in-cards have dual PHY but share a single MDIO
>> line (between the ports). In such cases, link inconsistencies are
>> noticed during the heavy traffic and during reboot stress tests.
>>
>> So, use the SMN calls to avoid the race conditions.
> 
> So this patch replaces all the PCI accesses you added in patch #2, so 
> why not just do this from the start?

Yes, that is correct. It was done to maintain the history and that will 
be reference as to why SMN is used over regular PCI accesses in this case.

> 
>>
>> Signed-off-by: Raju Rangoju <Raju.Rangoju@amd.com>
>> ---
>>   drivers/net/ethernet/amd/xgbe/xgbe-dev.c | 33 ++++++------------------
>>   drivers/net/ethernet/amd/xgbe/xgbe-pci.c | 10 +++----
>>   drivers/net/ethernet/amd/xgbe/xgbe-smn.h | 27 +++++++++++++++++++
>>   drivers/net/ethernet/amd/xgbe/xgbe.h     |  2 +-
>>   4 files changed, 41 insertions(+), 31 deletions(-)
>>   create mode 100644 drivers/net/ethernet/amd/xgbe/xgbe-smn.h
>>
>> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c 
>> b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
>> index a9eb2ffa9f73..8d8876ab258c 100644
>> --- a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
>> +++ b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c
>> @@ -124,6 +124,7 @@
>>   #include "xgbe.h"
>>   #include "xgbe-common.h"
>> +#include "xgbe-smn.h"
>>   static inline unsigned int xgbe_get_max_frame(struct xgbe_prv_data 
>> *pdata)
>>   {
>> @@ -1170,14 +1171,9 @@ static int xgbe_read_mmd_regs_v3(struct 
>> xgbe_prv_data *pdata, int prtad,
>>                    int mmd_reg)
>>   {
>>       unsigned int mmd_address, index, offset;
>> -    struct pci_dev *rdev;
>>       unsigned long flags;
>>       int mmd_data;
>> -    rdev = pci_get_domain_bus_and_slot(0, 0, PCI_DEVFN(0, 0));
>> -    if (!rdev)
>> -        return 0;
>> -
>>       mmd_address = get_mmd_address(pdata, mmd_reg);
>>       /* The PCS registers are accessed using mmio. The underlying
>> @@ -1192,13 +1188,10 @@ static int xgbe_read_mmd_regs_v3(struct 
>> xgbe_prv_data *pdata, int prtad,
>>       offset = get_index_offset(pdata, mmd_address, &index);
>>       spin_lock_irqsave(&pdata->xpcs_lock, flags);
>> -    pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + 
>> pdata->xpcs_window_sel_reg));
>> -    pci_write_config_dword(rdev, 0x64, index);
>> -    pci_write_config_dword(rdev, 0x60, pdata->xphy_base + offset);
>> -    pci_read_config_dword(rdev, 0x64, &mmd_data);
>> +    amd_smn_write(0, (pdata->smn_base + pdata->xpcs_window_sel_reg), 
>> index);
>> +    amd_smn_read(0, pdata->smn_base + offset, &mmd_data);
>>       mmd_data = (offset % 4) ? FIELD_GET(XGBE_GEN_HI_MASK, mmd_data) :
>>                     FIELD_GET(XGBE_GEN_LO_MASK, mmd_data);
>> -    pci_dev_put(rdev);
>>       spin_unlock_irqrestore(&pdata->xpcs_lock, flags);
>> @@ -1209,13 +1202,8 @@ static void xgbe_write_mmd_regs_v3(struct 
>> xgbe_prv_data *pdata, int prtad,
>>                      int mmd_reg, int mmd_data)
>>   {
>>       unsigned int mmd_address, index, offset, ctr_mmd_data;
>> -    struct pci_dev *rdev;
>>       unsigned long flags;
>> -    rdev = pci_get_domain_bus_and_slot(0, 0, PCI_DEVFN(0, 0));
>> -    if (!rdev)
>> -        return;
>> -
>>       mmd_address = get_mmd_address(pdata, mmd_reg);
>>       /* The PCS registers are accessed using mmio. The underlying
>> @@ -1230,10 +1218,9 @@ static void xgbe_write_mmd_regs_v3(struct 
>> xgbe_prv_data *pdata, int prtad,
>>       offset = get_index_offset(pdata, mmd_address, &index);
>>       spin_lock_irqsave(&pdata->xpcs_lock, flags);
>> -    pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + 
>> pdata->xpcs_window_sel_reg));
>> -    pci_write_config_dword(rdev, 0x64, index);
>> -    pci_write_config_dword(rdev, 0x60, pdata->xphy_base + offset);
>> -    pci_read_config_dword(rdev, 0x64, &ctr_mmd_data);
>> +    amd_smn_write(0, (pdata->smn_base + pdata->xpcs_window_sel_reg), 
>> index);
>> +    amd_smn_read(0, pdata->smn_base + offset, &ctr_mmd_data);
>> +
>>       if (offset % 4) {
>>           ctr_mmd_data = FIELD_PREP(XGBE_GEN_HI_MASK, mmd_data) |
>>                      FIELD_GET(XGBE_GEN_LO_MASK, ctr_mmd_data);
>> @@ -1243,12 +1230,8 @@ static void xgbe_write_mmd_regs_v3(struct 
>> xgbe_prv_data *pdata, int prtad,
>>                      FIELD_GET(XGBE_GEN_LO_MASK, mmd_data);
>>       }
>> -    pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + 
>> pdata->xpcs_window_sel_reg));
>> -    pci_write_config_dword(rdev, 0x64, index);
>> -    pci_write_config_dword(rdev, 0x60, (pdata->xphy_base + offset));
>> -    pci_write_config_dword(rdev, 0x64, ctr_mmd_data);
>> -    pci_dev_put(rdev);
>> -
>> +    amd_smn_write(0, (pdata->smn_base + pdata->xpcs_window_sel_reg), 
>> index);
>> +    amd_smn_write(0, (pdata->smn_base + offset), ctr_mmd_data);
>>       spin_unlock_irqrestore(&pdata->xpcs_lock, flags);
>>   }
>> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-pci.c 
>> b/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
>> index db3e8aac3339..135128b5be90 100644
>> --- a/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
>> +++ b/drivers/net/ethernet/amd/xgbe/xgbe-pci.c
>> @@ -121,6 +121,7 @@
>>   #include "xgbe.h"
>>   #include "xgbe-common.h"
>> +#include "xgbe-smn.h"
>>   static int xgbe_config_multi_msi(struct xgbe_prv_data *pdata)
>>   {
>> @@ -304,18 +305,17 @@ static int xgbe_pci_probe(struct pci_dev *pdev, 
>> const struct pci_device_id *id)
>>           pdata->xpcs_window_sel_reg = PCS_V2_WINDOW_SELECT;
>>           break;
>>       }
>> +    pci_dev_put(rdev);
>>       /* Configure the PCS indirect addressing support */
>>       if (pdata->vdata->xpcs_access == XGBE_XPCS_ACCESS_V3) {
>>           reg = XP_IOREAD(pdata, XP_PROP_0);
>> -        pdata->xphy_base = PCS_RN_SMN_BASE_ADDR +
>> -                   (PCS_RN_PORT_ADDR_SIZE * XP_GET_BITS(reg, 
>> XP_PROP_0, PORT_ID));
>> -        pci_write_config_dword(rdev, 0x60, pdata->xphy_base + 
>> (pdata->xpcs_window_def_reg));
>> -        pci_read_config_dword(rdev, 0x64, &reg);
>> +        pdata->smn_base = PCS_RN_SMN_BASE_ADDR +
>> +                  (PCS_RN_PORT_ADDR_SIZE * XP_GET_BITS(reg, 
>> XP_PROP_0, PORT_ID));
>> +        amd_smn_read(0, pdata->smn_base + 
>> (pdata->xpcs_window_def_reg), &reg);
>>       } else {
>>           reg = XPCS32_IOREAD(pdata, pdata->xpcs_window_def_reg);
>>       }
>> -    pci_dev_put(rdev);
>>       pdata->xpcs_window = XPCS_GET_BITS(reg, PCS_V2_WINDOW_DEF, OFFSET);
>>       pdata->xpcs_window <<= 6;
>> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-smn.h 
>> b/drivers/net/ethernet/amd/xgbe/xgbe-smn.h
>> new file mode 100644
>> index 000000000000..bd25ddc7c869
>> --- /dev/null
>> +++ b/drivers/net/ethernet/amd/xgbe/xgbe-smn.h
>> @@ -0,0 +1,27 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * AMD 10Gb Ethernet driver
>> + *
>> + * Copyright (c) 2023, Advanced Micro Devices, Inc.
>> + * All Rights Reserved.
>> + *
>> + * Author: Raju Rangoju <Raju.Rangoju@amd.com>
>> + */
> 
> Shouldn't this license match the license in all the other files?
> 
> Also, you need header protection here, e.g.:
> 
> #ifndef __XGBE_SMN_H__
> #define __XGBE_SMN_H__
> 
> and a #endif at the end.

Sure Tom, I'll take care of this.

> 
> Thanks,
> Tom
> 
>> +
>> +#ifdef CONFIG_AMD_NB
>> +
>> +#include <asm/amd_nb.h>
>> +
>> +#else
>> +
>> +static inline int amd_smn_write(u16 node, u32 address, u32 value)
>> +{
>> +    return -ENODEV;
>> +}
>> +
>> +static inline int amd_smn_read(u16 node, u32 address, u32 *value)
>> +{
>> +    return -ENODEV;
>> +}
>> +
>> +#endif
>> diff --git a/drivers/net/ethernet/amd/xgbe/xgbe.h 
>> b/drivers/net/ethernet/amd/xgbe/xgbe.h
>> index dbb1faaf6185..ba45ab0adb8c 100644
>> --- a/drivers/net/ethernet/amd/xgbe/xgbe.h
>> +++ b/drivers/net/ethernet/amd/xgbe/xgbe.h
>> @@ -1061,7 +1061,7 @@ struct xgbe_prv_data {
>>       struct device *dev;
>>       struct platform_device *phy_platdev;
>>       struct device *phy_dev;
>> -    unsigned int xphy_base;
>> +    unsigned int smn_base;
>>       /* Version related data */
>>       struct xgbe_version_data *vdata;

Re: [PATCH v3 net-next 3/3] amd-xgbe: use smn functions to avoid race

[Tom Lendacky]

On 12/14/23 22:35, Raju Rangoju wrote:
> 
> 
> On 12/12/2023 9:08 PM, Tom Lendacky wrote:
>> On 12/11/23 23:37, Raju Rangoju wrote:
>>> Some of the ethernet add-in-cards have dual PHY but share a single MDIO
>>> line (between the ports). In such cases, link inconsistencies are
>>> noticed during the heavy traffic and during reboot stress tests.
>>>
>>> So, use the SMN calls to avoid the race conditions.
>>
>> So this patch replaces all the PCI accesses you added in patch #2, so 
>> why not just do this from the start?
> 
> Yes, that is correct. It was done to maintain the history and that will be 
> reference as to why SMN is used over regular PCI accesses in this case.

Seems unnecessary to me. Adding a comment in the commit log and the code 
that states why SMN is used instead of PCI and how it fixes the race 
condition would be enough. Your call... but since patch #2 enables the 
device, this could cause a bisect issue.

Thanks,
Tom

>