[PATCH net-next v2 00/13] nfp: add support for multi-pf configuration

[PATCH net-next v2 00/13] nfp: add support for multi-pf configuration

[Louis Peens]

This patch series is introducing multiple PFs for multiple ports NIC
assembled with NFP3800 chip. This is done since the NFP3800 can
support up to 4 PFs, and is more in-line with the modern expectation
that each port/netdev is associated with a unique PF.

For compatibility concern with NFP4000/6000 cards, and older management
firmware on NFP3800, multiple ports sharing single PF is still supported
with this change. Whether it's multi-PF setup or single-PF setup is
determined by management firmware, and driver will notify the
application firmware of the setup so that both are well handled.

* Patch 1/13 and 2/13 are to support new management firmware with bumped
  major version.
* Patch 3/13, 4/13, 6/13, adjust the application firmware loading
  and unloading mechanism since multi PFs share the same application
  firmware.
* Patch 5/16 is a small fix to fix an issue sparse is complaining about.
* Patch 7/13 is a small fix to avoid reclaiming resources by mistake in
  multi-PF setup.
* Patch 8/13 re-formats the symbols to communicate with application
  firmware to adapt multi-PF setup.
* Patch 9/13 applies one port/netdev per PF.
* Patch 10/13 is to support both single-PF and multi-PF setup by a
  configuration in application firmware.
* Patch 11/13, 12/13, 13/13 are some necessary adaption to use SR-IOV
  for multi-PF setup.


Since v1:
    Modify 64-bit non-atomic write functions to avoid sparse
    warning

As part of v1 there was also some partially finished discussion about
devlink allowing to bind to multiple bus devices. This series creates a
devlink instance per PF, and the comment was asking if this should maybe
change to be a single instance, since it is still a single device. For
the moment we feel that this is a parallel issue to this specific
series, as it seems to be already implemented this way in other places,
and this series would be matching that.

We are curious about this idea though, as it does seem to make sense if
the original devlink idea was that it should have a one-to-one
correspondence per ASIC. Not sure where one would start with this
though, on first glance it looks like the assumption that devlink is
only connected to a single bus device is embedded quite deep. This
probably needs commenting/discussion with somebody that has pretty good
knowledge of devlink core.

Tianyu Yuan (4):
  nsp: generate nsp command with variable nsp major version
  nfp: bump the nsp major version to support multi-PF
  nfp: apply one port per PF for multi-PF setup
  nfp: configure VF total count for each PF

Yinjun Zhang (9):
  nfp: change application firmware loading flow in multi-PF setup
  nfp: don't skip firmware loading when it's pxe firmware in running
  io-64-nonatomic: truncate bits explicitly to avoid warning
  nfp: introduce keepalive mechanism for multi-PF setup
  nfp: avoid reclaiming resource mutex by mistake
  nfp: redefine PF id used to format symbols
  nfp: enable multi-PF in application firmware if supported
  nfp: configure VF split info into application firmware
  nfp: use absolute vf id for multi-PF case

 drivers/net/ethernet/netronome/nfp/abm/ctrl.c |   2 +-
 drivers/net/ethernet/netronome/nfp/abm/main.c |   2 +-
 drivers/net/ethernet/netronome/nfp/bpf/main.c |   2 +-
 .../net/ethernet/netronome/nfp/flower/main.c  |  19 +-
 drivers/net/ethernet/netronome/nfp/nfp_main.c | 227 ++++++++++++++++--
 drivers/net/ethernet/netronome/nfp/nfp_main.h |  28 +++
 .../net/ethernet/netronome/nfp/nfp_net_ctrl.h |   1 +
 .../net/ethernet/netronome/nfp/nfp_net_main.c | 166 ++++++++++---
 .../ethernet/netronome/nfp/nfp_net_sriov.c    |  39 ++-
 .../ethernet/netronome/nfp/nfp_net_sriov.h    |   5 +
 drivers/net/ethernet/netronome/nfp/nfp_port.c |   4 +-
 .../net/ethernet/netronome/nfp/nfpcore/nfp.h  |   4 +
 .../ethernet/netronome/nfp/nfpcore/nfp_dev.c  |   2 +
 .../ethernet/netronome/nfp/nfpcore/nfp_dev.h  |   1 +
 .../netronome/nfp/nfpcore/nfp_mutex.c         |  21 +-
 .../ethernet/netronome/nfp/nfpcore/nfp_nffw.h |   4 +
 .../ethernet/netronome/nfp/nfpcore/nfp_nsp.c  |  18 +-
 .../netronome/nfp/nfpcore/nfp_rtsym.c         |  16 +-
 drivers/net/ethernet/netronome/nfp/nic/main.c |   3 +-
 include/linux/io-64-nonatomic-hi-lo.h         |   8 +-
 include/linux/io-64-nonatomic-lo-hi.h         |   8 +-
 21 files changed, 482 insertions(+), 98 deletions(-)

-- 
2.34.1

[PATCH net-next v2 01/13] nsp: generate nsp command with variable nsp major version

[Louis Peens]

From: Tianyu Yuan <tianyu.yuan@corigine.com>

The most significant 4 bits of nsp command code should carry the
ABI major version so that nsp command can be responded correctly.
It is working well since current major version is 0.

However management firmware is going to bump the major version to
support multi-PF feature. So change the code to explicitly contain
the major version into nsp command code.

Signed-off-by: Tianyu Yuan <tianyu.yuan@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
 drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c
index 7136bc48530b..ee934663c6d9 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c
@@ -37,7 +37,8 @@
 
 #define NSP_COMMAND		0x08
 #define   NSP_COMMAND_OPTION	GENMASK_ULL(63, 32)
-#define   NSP_COMMAND_CODE	GENMASK_ULL(31, 16)
+#define   NSP_COMMAND_CODE_MJ_VER	GENMASK_ULL(31, 28)
+#define   NSP_COMMAND_CODE	GENMASK_ULL(27, 16)
 #define   NSP_COMMAND_DMA_BUF	BIT_ULL(1)
 #define   NSP_COMMAND_START	BIT_ULL(0)
 
@@ -380,6 +381,7 @@ __nfp_nsp_command(struct nfp_nsp *state, const struct nfp_nsp_command_arg *arg)
 
 	err = nfp_cpp_writeq(cpp, nsp_cpp, nsp_command,
 			     FIELD_PREP(NSP_COMMAND_OPTION, arg->option) |
+			     FIELD_PREP(NSP_COMMAND_CODE_MJ_VER, state->ver.major) |
 			     FIELD_PREP(NSP_COMMAND_CODE, arg->code) |
 			     FIELD_PREP(NSP_COMMAND_DMA_BUF, arg->dma) |
 			     FIELD_PREP(NSP_COMMAND_START, 1));
-- 
2.34.1

[PATCH net-next v2 02/13] nfp: bump the nsp major version to support multi-PF

[Louis Peens]

From: Tianyu Yuan <tianyu.yuan@corigine.com>

Currently NFP NICs implement single PF with multiple ports
instantiated. While NFP3800 can support multiple PFs and
one port per PF is more up-to-date, the management firmware
will start to support multi-PF. Since it's incompatible with
current implementation, the ABI major version is bumped.

A new flag is also introduced to indicate whether it's
multi-PF setup or single-PF setup.

Signed-off-by: Tianyu Yuan <tianyu.yuan@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
 drivers/net/ethernet/netronome/nfp/nfp_main.c      |  3 +++
 drivers/net/ethernet/netronome/nfp/nfp_main.h      |  6 ++++++
 .../net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c   | 14 ++++++++++----
 3 files changed, 19 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.c b/drivers/net/ethernet/netronome/nfp/nfp_main.c
index 71301dbd8fb5..39c1327625fa 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.c
@@ -625,6 +625,9 @@ static int nfp_nsp_init(struct pci_dev *pdev, struct nfp_pf *pf)
 		return err;
 	}
 
+	pf->multi_pf.en = pdev->multifunction;
+	dev_info(&pdev->dev, "%s-PF detected\n", pf->multi_pf.en ? "Multi" : "Single");
+
 	err = nfp_nsp_wait(nsp);
 	if (err < 0)
 		goto exit_close_nsp;
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.h b/drivers/net/ethernet/netronome/nfp/nfp_main.h
index 14a751bfe1fe..72ea3b83d313 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.h
@@ -84,6 +84,8 @@ struct nfp_dumpspec {
  * @port_refresh_work:	Work entry for taking netdevs out
  * @shared_bufs:	Array of shared buffer structures if FW has any SBs
  * @num_shared_bufs:	Number of elements in @shared_bufs
+ * @multi_pf:		Used in multi-PF setup
+ * @multi_pf.en:	True if it's a NIC with multiple PFs
  *
  * Fields which may change after proble are protected by devlink instance lock.
  */
@@ -141,6 +143,10 @@ struct nfp_pf {
 
 	struct nfp_shared_buf *shared_bufs;
 	unsigned int num_shared_bufs;
+
+	struct {
+		bool en;
+	} multi_pf;
 };
 
 extern struct pci_driver nfp_netvf_pci_driver;
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c
index ee934663c6d9..3098a9e52138 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c
@@ -59,7 +59,13 @@
 #define NFP_CAP_CMD_DMA_SG	0x28
 
 #define NSP_MAGIC		0xab10
-#define NSP_MAJOR		0
+/* ABI major version is bumped separately without resetting minor
+ * version when the change in NSP is not compatible to old driver.
+ */
+#define NSP_MAJOR		1
+/* ABI minor version is bumped when new feature is introduced
+ * while old driver can still work without this new feature.
+ */
 #define NSP_MINOR		8
 
 #define NSP_CODE_MAJOR		GENMASK(15, 12)
@@ -248,14 +254,14 @@ static int nfp_nsp_check(struct nfp_nsp *state)
 	state->ver.major = FIELD_GET(NSP_STATUS_MAJOR, reg);
 	state->ver.minor = FIELD_GET(NSP_STATUS_MINOR, reg);
 
-	if (state->ver.major != NSP_MAJOR) {
+	if (state->ver.major > NSP_MAJOR) {
 		nfp_err(cpp, "Unsupported ABI %hu.%hu\n",
 			state->ver.major, state->ver.minor);
 		return -EINVAL;
 	}
 	if (state->ver.minor < NSP_MINOR) {
-		nfp_err(cpp, "ABI too old to support NIC operation (%u.%hu < %u.%u), please update the management FW on the flash\n",
-			NSP_MAJOR, state->ver.minor, NSP_MAJOR, NSP_MINOR);
+		nfp_err(cpp, "ABI too old to support NIC operation (x.%u < x.%u), please update the management FW on the flash\n",
+			state->ver.minor, NSP_MINOR);
 		return -EINVAL;
 	}
 
-- 
2.34.1

[PATCH net-next v2 03/13] nfp: change application firmware loading flow in multi-PF setup

[Louis Peens]

From: Yinjun Zhang <yinjun.zhang@corigine.com>

In multi-PF setup, all PFs share the single application firmware.
Each PF is treated equally, and first-come-first-served. So the
first step is to check firmware is loaded or not. And also loading
firmware from disk and flash are treated consistently, both
propagating the failure and setting `fw_loaded` flag. At last,
firmware shouldn't be unloaded in this setup. The following commit
will introduce a keepalive mechanism to let management firmware
manage unloading.

The flow is not changed in non-multi-PF setup.

Signed-off-by: Yinjun Zhang <yinjun.zhang@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
 drivers/net/ethernet/netronome/nfp/nfp_main.c | 33 +++++++++++++++----
 1 file changed, 26 insertions(+), 7 deletions(-)

diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.c b/drivers/net/ethernet/netronome/nfp/nfp_main.c
index 39c1327625fa..c81784a626a6 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.c
@@ -528,6 +528,10 @@ nfp_fw_load(struct pci_dev *pdev, struct nfp_pf *pf, struct nfp_nsp *nsp)
 	if (err)
 		return err;
 
+	/* Skip firmware loading in multi-PF setup if firmware is loaded. */
+	if (pf->multi_pf.en && nfp_nsp_fw_loaded(nsp))
+		return 1;
+
 	fw = nfp_net_fw_find(pdev, pf);
 	do_reset = reset == NFP_NSP_DRV_RESET_ALWAYS ||
 		   (fw && reset == NFP_NSP_DRV_RESET_DISK);
@@ -556,16 +560,30 @@ nfp_fw_load(struct pci_dev *pdev, struct nfp_pf *pf, struct nfp_nsp *nsp)
 		fw_loaded = true;
 	} else if (policy != NFP_NSP_APP_FW_LOAD_DISK &&
 		   nfp_nsp_has_stored_fw_load(nsp)) {
+		err = nfp_nsp_load_stored_fw(nsp);
 
-		/* Don't propagate this error to stick with legacy driver
+		/* For single-PF setup:
+		 * Don't propagate this error to stick with legacy driver
 		 * behavior, failure will be detected later during init.
+		 * Don't flag the fw_loaded in this case since other devices
+		 * may reuse the firmware when configured this way.
+		 *
+		 * For multi-PF setup:
+		 * We only reach here when firmware is freshly loaded from
+		 * flash, so need propagate the error and flow the fw_loaded
+		 * as it does when loading firmware from disk.
 		 */
-		if (!nfp_nsp_load_stored_fw(nsp))
+		if (!err) {
 			dev_info(&pdev->dev, "Finished loading stored FW image\n");
 
-		/* Don't flag the fw_loaded in this case since other devices
-		 * may reuse the firmware when configured this way
-		 */
+			if (pf->multi_pf.en)
+				fw_loaded = true;
+		} else {
+			if (pf->multi_pf.en)
+				dev_err(&pdev->dev, "Stored FW loading failed: %d\n", err);
+			else
+				err = 0;
+		}
 	} else {
 		dev_warn(&pdev->dev, "Didn't load firmware, please update flash or reconfigure card\n");
 	}
@@ -575,9 +593,10 @@ nfp_fw_load(struct pci_dev *pdev, struct nfp_pf *pf, struct nfp_nsp *nsp)
 
 	/* We don't want to unload firmware when other devices may still be
 	 * dependent on it, which could be the case if there are multiple
-	 * devices that could load firmware.
+	 * devices that could load firmware or the case multiple PFs are
+	 * running.
 	 */
-	if (fw_loaded && ifcs == 1)
+	if (fw_loaded && ifcs == 1 && !pf->multi_pf.en)
 		pf->unload_fw_on_remove = true;
 
 	return err < 0 ? err : fw_loaded;
-- 
2.34.1

[PATCH net-next v2 04/13] nfp: don't skip firmware loading when it's pxe firmware in running

[Louis Peens]

From: Yinjun Zhang <yinjun.zhang@corigine.com>

In pxe boot case, the pxe firmware is not unloaded in some systems
when booting completes. Driver needs to detect it so that it has
chance to load the correct firmware.

Signed-off-by: Yinjun Zhang <yinjun.zhang@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
 drivers/net/ethernet/netronome/nfp/nfp_main.c | 25 +++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.c b/drivers/net/ethernet/netronome/nfp/nfp_main.c
index c81784a626a6..778f21dfbbd5 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.c
@@ -469,6 +469,28 @@ nfp_get_fw_policy_value(struct pci_dev *pdev, struct nfp_nsp *nsp,
 	return err;
 }
 
+static bool
+nfp_skip_fw_load(struct nfp_pf *pf, struct nfp_nsp *nsp)
+{
+	const struct nfp_mip *mip;
+	bool skip;
+
+	if (!pf->multi_pf.en || nfp_nsp_fw_loaded(nsp) <= 0)
+		return false;
+
+	mip = nfp_mip_open(pf->cpp);
+	if (!mip)
+		return false;
+
+	/* For the case that system boots from pxe, we need
+	 * reload FW if pxe FW is running.
+	 */
+	skip = !!strncmp(nfp_mip_name(mip), "pxe", 3);
+	nfp_mip_close(mip);
+
+	return skip;
+}
+
 /**
  * nfp_fw_load() - Load the firmware image
  * @pdev:       PCI Device structure
@@ -528,8 +550,7 @@ nfp_fw_load(struct pci_dev *pdev, struct nfp_pf *pf, struct nfp_nsp *nsp)
 	if (err)
 		return err;
 
-	/* Skip firmware loading in multi-PF setup if firmware is loaded. */
-	if (pf->multi_pf.en && nfp_nsp_fw_loaded(nsp))
+	if (nfp_skip_fw_load(pf, nsp))
 		return 1;
 
 	fw = nfp_net_fw_find(pdev, pf);
-- 
2.34.1

[PATCH net-next v2 05/13] io-64-nonatomic: truncate bits explicitly to avoid warning

[Louis Peens]

From: Yinjun Zhang <yinjun.zhang@corigine.com>

Sparse complains when passing constant value to these nonatomic io
write functions:

include/linux/io-64-nonatomic-hi-lo.h:22:16: sparse: sparse: cast truncates bits from constant value

Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202307300422.oPy5E1hB-lkp@intel.com/
Signed-off-by: Yinjun Zhang <yinjun.zhang@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
 include/linux/io-64-nonatomic-hi-lo.h | 8 ++++----
 include/linux/io-64-nonatomic-lo-hi.h | 8 ++++----
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/include/linux/io-64-nonatomic-hi-lo.h b/include/linux/io-64-nonatomic-hi-lo.h
index f32522bb3aa5..60bb802d4f23 100644
--- a/include/linux/io-64-nonatomic-hi-lo.h
+++ b/include/linux/io-64-nonatomic-hi-lo.h
@@ -19,7 +19,7 @@ static inline __u64 hi_lo_readq(const volatile void __iomem *addr)
 static inline void hi_lo_writeq(__u64 val, volatile void __iomem *addr)
 {
 	writel(val >> 32, addr + 4);
-	writel(val, addr);
+	writel(val & U32_MAX, addr);
 }
 
 static inline __u64 hi_lo_readq_relaxed(const volatile void __iomem *addr)
@@ -36,7 +36,7 @@ static inline __u64 hi_lo_readq_relaxed(const volatile void __iomem *addr)
 static inline void hi_lo_writeq_relaxed(__u64 val, volatile void __iomem *addr)
 {
 	writel_relaxed(val >> 32, addr + 4);
-	writel_relaxed(val, addr);
+	writel_relaxed(val & U32_MAX, addr);
 }
 
 #ifndef readq
@@ -73,7 +73,7 @@ static inline u64 ioread64_hi_lo(const void __iomem *addr)
 static inline void iowrite64_hi_lo(u64 val, void __iomem *addr)
 {
 	iowrite32(val >> 32, addr + sizeof(u32));
-	iowrite32(val, addr);
+	iowrite32(val & U32_MAX, addr);
 }
 #endif
 
@@ -95,7 +95,7 @@ static inline u64 ioread64be_hi_lo(const void __iomem *addr)
 static inline void iowrite64be_hi_lo(u64 val, void __iomem *addr)
 {
 	iowrite32be(val >> 32, addr);
-	iowrite32be(val, addr + sizeof(u32));
+	iowrite32be(val & U32_MAX, addr + sizeof(u32));
 }
 #endif
 
diff --git a/include/linux/io-64-nonatomic-lo-hi.h b/include/linux/io-64-nonatomic-lo-hi.h
index 448a21435dba..b73acd377507 100644
--- a/include/linux/io-64-nonatomic-lo-hi.h
+++ b/include/linux/io-64-nonatomic-lo-hi.h
@@ -18,7 +18,7 @@ static inline __u64 lo_hi_readq(const volatile void __iomem *addr)
 
 static inline void lo_hi_writeq(__u64 val, volatile void __iomem *addr)
 {
-	writel(val, addr);
+	writel(val & U32_MAX, addr);
 	writel(val >> 32, addr + 4);
 }
 
@@ -35,7 +35,7 @@ static inline __u64 lo_hi_readq_relaxed(const volatile void __iomem *addr)
 
 static inline void lo_hi_writeq_relaxed(__u64 val, volatile void __iomem *addr)
 {
-	writel_relaxed(val, addr);
+	writel_relaxed(val & U32_MAX, addr);
 	writel_relaxed(val >> 32, addr + 4);
 }
 
@@ -72,7 +72,7 @@ static inline u64 ioread64_lo_hi(const void __iomem *addr)
 #define iowrite64_lo_hi iowrite64_lo_hi
 static inline void iowrite64_lo_hi(u64 val, void __iomem *addr)
 {
-	iowrite32(val, addr);
+	iowrite32(val & U32_MAX, addr);
 	iowrite32(val >> 32, addr + sizeof(u32));
 }
 #endif
@@ -94,7 +94,7 @@ static inline u64 ioread64be_lo_hi(const void __iomem *addr)
 #define iowrite64be_lo_hi iowrite64be_lo_hi
 static inline void iowrite64be_lo_hi(u64 val, void __iomem *addr)
 {
-	iowrite32be(val, addr + sizeof(u32));
+	iowrite32be(val & U32_MAX, addr + sizeof(u32));
 	iowrite32be(val >> 32, addr);
 }
 #endif
-- 
2.34.1

[PATCH net-next v2 06/13] nfp: introduce keepalive mechanism for multi-PF setup

[Louis Peens]

From: Yinjun Zhang <yinjun.zhang@corigine.com>

In multi-PF setup, management firmware is in charge of application
firmware unloading instead of driver by keepalive mechanism.

A new NSP resource area is allocated for keepalive use with name
"nfp.beat". Driver sets the magic number when keepalive is needed
and periodically updates the PF's corresponding qword in "nfp.beat".
Management firmware checks these PFs' qwords to learn whether and
which PFs are alive, and will unload the application firmware if
no PF is running. This only works when magic number is correct.

Signed-off-by: Yinjun Zhang <yinjun.zhang@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
 drivers/net/ethernet/netronome/nfp/nfp_main.c | 88 +++++++++++++++++++
 drivers/net/ethernet/netronome/nfp/nfp_main.h |  8 ++
 .../net/ethernet/netronome/nfp/nfpcore/nfp.h  |  4 +
 3 files changed, 100 insertions(+)

diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.c b/drivers/net/ethernet/netronome/nfp/nfp_main.c
index 778f21dfbbd5..489113c53596 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.c
@@ -469,6 +469,77 @@ nfp_get_fw_policy_value(struct pci_dev *pdev, struct nfp_nsp *nsp,
 	return err;
 }
 
+static void
+nfp_nsp_beat_timer(struct timer_list *t)
+{
+	struct nfp_pf *pf = from_timer(pf, t, multi_pf.beat_timer);
+	u8 __iomem *addr;
+
+	/* Each PF has corresponding qword to beat:
+	 * offset | usage
+	 *   0    | magic number
+	 *   8    | beat qword of pf0
+	 *   16   | beat qword of pf1
+	 */
+	addr = pf->multi_pf.beat_addr + ((pf->multi_pf.id + 1) << 3);
+	writeq(jiffies, addr);
+	/* Beat once per second. */
+	mod_timer(&pf->multi_pf.beat_timer, jiffies + HZ);
+}
+
+/**
+ * nfp_nsp_keepalive_start() - Start keepalive mechanism if needed
+ * @pf:		NFP PF Device structure
+ *
+ * Return 0 if no error, errno otherwise
+ */
+static int
+nfp_nsp_keepalive_start(struct nfp_pf *pf)
+{
+	struct nfp_resource *res;
+	u8 __iomem *base;
+	int err = 0;
+	u64 addr;
+	u32 cpp;
+
+	if (!pf->multi_pf.en)
+		return 0;
+
+	res = nfp_resource_acquire(pf->cpp, NFP_KEEPALIVE);
+	if (IS_ERR(res))
+		return PTR_ERR(res);
+
+	cpp = nfp_resource_cpp_id(res);
+	addr = nfp_resource_address(res);
+
+	/* Allocate a fixed area for keepalive. */
+	base = nfp_cpp_map_area(pf->cpp, "keepalive", cpp, addr,
+				nfp_resource_size(res), &pf->multi_pf.beat_area);
+	if (IS_ERR(base)) {
+		nfp_err(pf->cpp, "Failed to map area for keepalive\n");
+		err = PTR_ERR(base);
+		goto res_release;
+	}
+
+	pf->multi_pf.beat_addr = base;
+	timer_setup(&pf->multi_pf.beat_timer, nfp_nsp_beat_timer, 0);
+	mod_timer(&pf->multi_pf.beat_timer, jiffies);
+
+res_release:
+	nfp_resource_release(res);
+	return err;
+}
+
+static void
+nfp_nsp_keepalive_stop(struct nfp_pf *pf)
+{
+	if (!pf->multi_pf.beat_area)
+		return;
+
+	del_timer_sync(&pf->multi_pf.beat_timer);
+	nfp_cpp_area_release_free(pf->multi_pf.beat_area);
+}
+
 static bool
 nfp_skip_fw_load(struct nfp_pf *pf, struct nfp_nsp *nsp)
 {
@@ -550,6 +621,10 @@ nfp_fw_load(struct pci_dev *pdev, struct nfp_pf *pf, struct nfp_nsp *nsp)
 	if (err)
 		return err;
 
+	err = nfp_nsp_keepalive_start(pf);
+	if (err)
+		return err;
+
 	if (nfp_skip_fw_load(pf, nsp))
 		return 1;
 
@@ -620,6 +695,16 @@ nfp_fw_load(struct pci_dev *pdev, struct nfp_pf *pf, struct nfp_nsp *nsp)
 	if (fw_loaded && ifcs == 1 && !pf->multi_pf.en)
 		pf->unload_fw_on_remove = true;
 
+	/* Only setting magic number when fw is freshly loaded here. NSP
+	 * won't unload fw when heartbeat stops if the magic number is not
+	 * correct. It's used when firmware is preloaded and shouldn't be
+	 * unloaded when driver exits.
+	 */
+	if (err < 0)
+		nfp_nsp_keepalive_stop(pf);
+	else if (fw_loaded && pf->multi_pf.en)
+		writeq(NFP_KEEPALIVE_MAGIC, pf->multi_pf.beat_addr);
+
 	return err < 0 ? err : fw_loaded;
 }
 
@@ -666,6 +751,7 @@ static int nfp_nsp_init(struct pci_dev *pdev, struct nfp_pf *pf)
 	}
 
 	pf->multi_pf.en = pdev->multifunction;
+	pf->multi_pf.id = PCI_FUNC(pdev->devfn);
 	dev_info(&pdev->dev, "%s-PF detected\n", pf->multi_pf.en ? "Multi" : "Single");
 
 	err = nfp_nsp_wait(nsp);
@@ -913,6 +999,7 @@ static int nfp_pci_probe(struct pci_dev *pdev,
 err_net_remove:
 	nfp_net_pci_remove(pf);
 err_fw_unload:
+	nfp_nsp_keepalive_stop(pf);
 	kfree(pf->rtbl);
 	nfp_mip_close(pf->mip);
 	if (pf->unload_fw_on_remove)
@@ -952,6 +1039,7 @@ static void __nfp_pci_shutdown(struct pci_dev *pdev, bool unload_fw)
 	nfp_net_pci_remove(pf);
 
 	vfree(pf->dumpspec);
+	nfp_nsp_keepalive_stop(pf);
 	kfree(pf->rtbl);
 	nfp_mip_close(pf->mip);
 	if (unload_fw && pf->unload_fw_on_remove)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.h b/drivers/net/ethernet/netronome/nfp/nfp_main.h
index 72ea3b83d313..c58849a332b0 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.h
@@ -86,6 +86,10 @@ struct nfp_dumpspec {
  * @num_shared_bufs:	Number of elements in @shared_bufs
  * @multi_pf:		Used in multi-PF setup
  * @multi_pf.en:	True if it's a NIC with multiple PFs
+ * @multi_pf.id:	PF index
+ * @multi_pf.beat_timer:Timer for beat to keepalive
+ * @multi_pf.beat_area:	Pointer to CPP area for beat to keepalive
+ * @multi_pf.beat_addr:	Pointer to mapped beat address used for keepalive
  *
  * Fields which may change after proble are protected by devlink instance lock.
  */
@@ -146,6 +150,10 @@ struct nfp_pf {
 
 	struct {
 		bool en;
+		u8 id;
+		struct timer_list beat_timer;
+		struct nfp_cpp_area *beat_area;
+		u8 __iomem *beat_addr;
 	} multi_pf;
 };
 
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp.h b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp.h
index db94b0bddc92..89a131cffc48 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp.h
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp.h
@@ -64,6 +64,10 @@ int nfp_nsp_read_sensors(struct nfp_nsp *state, unsigned int sensor_mask,
 /* MAC Statistics Accumulator */
 #define NFP_RESOURCE_MAC_STATISTICS	"mac.stat"
 
+/* Keepalive */
+#define NFP_KEEPALIVE			"nfp.beat"
+#define NFP_KEEPALIVE_MAGIC		0x6e66702e62656174ULL /* ASCII of "nfp.beat" */
+
 int nfp_resource_table_init(struct nfp_cpp *cpp);
 
 struct nfp_resource *
-- 
2.34.1

[PATCH net-next v2 07/13] nfp: avoid reclaiming resource mutex by mistake

[Louis Peens]

From: Yinjun Zhang <yinjun.zhang@corigine.com>

Multiple PFs of the same controller use the same interface
id. So we shouldn't unconditionally reclaim resource mutex
when probing, because the mutex may be held by another PF
from the same controller.

Now give it some time to release the mutex, and reclaim it
if timeout.

Signed-off-by: Yinjun Zhang <yinjun.zhang@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
 .../netronome/nfp/nfpcore/nfp_mutex.c         | 21 +++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)

diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_mutex.c b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_mutex.c
index 7bc17b94ac60..1b9170d9da77 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_mutex.c
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_mutex.c
@@ -343,6 +343,7 @@ int nfp_cpp_mutex_reclaim(struct nfp_cpp *cpp, int target,
 {
 	const u32 mur = NFP_CPP_ID(target, 3, 0);	/* atomic_read */
 	const u32 muw = NFP_CPP_ID(target, 4, 0);	/* atomic_write */
+	unsigned long timeout = jiffies + 2 * HZ;
 	u16 interface = nfp_cpp_interface(cpp);
 	int err;
 	u32 tmp;
@@ -351,13 +352,21 @@ int nfp_cpp_mutex_reclaim(struct nfp_cpp *cpp, int target,
 	if (err)
 		return err;
 
-	/* Check lock */
-	err = nfp_cpp_readl(cpp, mur, address, &tmp);
-	if (err < 0)
-		return err;
+	/* Check lock. Note that PFs from the same controller use same interface ID.
+	 * So considering that the lock may be held by other PFs from the same
+	 * controller, we give it some time to release the lock, and only reclaim it
+	 * if timeout.
+	 */
+	while (time_is_after_jiffies(timeout)) {
+		err = nfp_cpp_readl(cpp, mur, address, &tmp);
+		if (err < 0)
+			return err;
 
-	if (nfp_mutex_is_unlocked(tmp) || nfp_mutex_owner(tmp) != interface)
-		return 0;
+		if (nfp_mutex_is_unlocked(tmp) || nfp_mutex_owner(tmp) != interface)
+			return 0;
+
+		msleep_interruptible(10);
+	}
 
 	/* Bust the lock */
 	err = nfp_cpp_writel(cpp, muw, address, nfp_mutex_unlocked(interface));
-- 
2.34.1

[PATCH net-next v2 08/13] nfp: redefine PF id used to format symbols

[Louis Peens]

From: Yinjun Zhang <yinjun.zhang@corigine.com>

Taking account that NFP3800 supports 4 physical functions per
controller, now recalculate PF id to be used to format symbols
to communicate with application firmware.

Signed-off-by: Yinjun Zhang <yinjun.zhang@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
 drivers/net/ethernet/netronome/nfp/abm/ctrl.c  |  2 +-
 .../net/ethernet/netronome/nfp/flower/main.c   |  2 +-
 drivers/net/ethernet/netronome/nfp/nfp_main.c  | 18 +++++++++++-------
 drivers/net/ethernet/netronome/nfp/nfp_main.h  |  2 ++
 .../ethernet/netronome/nfp/nfpcore/nfp_dev.c   |  2 ++
 .../ethernet/netronome/nfp/nfpcore/nfp_dev.h   |  1 +
 6 files changed, 18 insertions(+), 9 deletions(-)

diff --git a/drivers/net/ethernet/netronome/nfp/abm/ctrl.c b/drivers/net/ethernet/netronome/nfp/abm/ctrl.c
index 69e84ff7f2e5..41d18df97c85 100644
--- a/drivers/net/ethernet/netronome/nfp/abm/ctrl.c
+++ b/drivers/net/ethernet/netronome/nfp/abm/ctrl.c
@@ -362,7 +362,7 @@ int nfp_abm_ctrl_find_addrs(struct nfp_abm *abm)
 	const struct nfp_rtsym *sym;
 	int res;
 
-	abm->pf_id = nfp_cppcore_pcie_unit(pf->cpp);
+	abm->pf_id = nfp_get_pf_id(pf);
 
 	/* Check if Qdisc offloads are supported */
 	res = nfp_pf_rtsym_read_optional(pf, NFP_RED_SUPPORT_SYM_NAME, 1);
diff --git a/drivers/net/ethernet/netronome/nfp/flower/main.c b/drivers/net/ethernet/netronome/nfp/flower/main.c
index 83eaa5ae3cd4..565987f0a595 100644
--- a/drivers/net/ethernet/netronome/nfp/flower/main.c
+++ b/drivers/net/ethernet/netronome/nfp/flower/main.c
@@ -378,10 +378,10 @@ nfp_flower_spawn_vnic_reprs(struct nfp_app *app,
 			    enum nfp_flower_cmsg_port_vnic_type vnic_type,
 			    enum nfp_repr_type repr_type, unsigned int cnt)
 {
-	u8 nfp_pcie = nfp_cppcore_pcie_unit(app->pf->cpp);
 	struct nfp_flower_priv *priv = app->priv;
 	atomic_t *replies = &priv->reify_replies;
 	struct nfp_flower_repr_priv *repr_priv;
+	u8 nfp_pcie = nfp_get_pf_id(app->pf);
 	enum nfp_port_type port_type;
 	struct nfp_repr *nfp_repr;
 	struct nfp_reprs *reprs;
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.c b/drivers/net/ethernet/netronome/nfp/nfp_main.c
index 489113c53596..74767729e542 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.c
@@ -69,6 +69,13 @@ static const struct pci_device_id nfp_pci_device_ids[] = {
 };
 MODULE_DEVICE_TABLE(pci, nfp_pci_device_ids);
 
+u8 nfp_get_pf_id(struct nfp_pf *pf)
+{
+	return nfp_cppcore_pcie_unit(pf->cpp) *
+	       pf->dev_info->pf_num_per_unit +
+	       pf->multi_pf.id;
+}
+
 int nfp_pf_rtsym_read_optional(struct nfp_pf *pf, const char *format,
 			       unsigned int default_val)
 {
@@ -76,7 +83,7 @@ int nfp_pf_rtsym_read_optional(struct nfp_pf *pf, const char *format,
 	int err = 0;
 	u64 val;
 
-	snprintf(name, sizeof(name), format, nfp_cppcore_pcie_unit(pf->cpp));
+	snprintf(name, sizeof(name), format, nfp_get_pf_id(pf));
 
 	val = nfp_rtsym_read_le(pf->rtbl, name, &err);
 	if (err) {
@@ -95,8 +102,7 @@ nfp_pf_map_rtsym(struct nfp_pf *pf, const char *name, const char *sym_fmt,
 {
 	char pf_symbol[256];
 
-	snprintf(pf_symbol, sizeof(pf_symbol), sym_fmt,
-		 nfp_cppcore_pcie_unit(pf->cpp));
+	snprintf(pf_symbol, sizeof(pf_symbol), sym_fmt, nfp_get_pf_id(pf));
 
 	return nfp_rtsym_map(pf->rtbl, pf_symbol, name, min_size, area);
 }
@@ -803,10 +809,8 @@ static void nfp_fw_unload(struct nfp_pf *pf)
 
 static int nfp_pf_find_rtsyms(struct nfp_pf *pf)
 {
+	unsigned int pf_id = nfp_get_pf_id(pf);
 	char pf_symbol[256];
-	unsigned int pf_id;
-
-	pf_id = nfp_cppcore_pcie_unit(pf->cpp);
 
 	/* Optional per-PCI PF mailbox */
 	snprintf(pf_symbol, sizeof(pf_symbol), NFP_MBOX_SYM_NAME, pf_id);
@@ -832,7 +836,7 @@ static u64 nfp_net_pf_get_app_cap(struct nfp_pf *pf)
 	int err = 0;
 	u64 val;
 
-	snprintf(name, sizeof(name), "_pf%u_net_app_cap", nfp_cppcore_pcie_unit(pf->cpp));
+	snprintf(name, sizeof(name), "_pf%u_net_app_cap", nfp_get_pf_id(pf));
 
 	val = nfp_rtsym_read_le(pf->rtbl, name, &err);
 	if (err) {
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.h b/drivers/net/ethernet/netronome/nfp/nfp_main.h
index c58849a332b0..7f76c718fef8 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.h
@@ -208,4 +208,6 @@ void nfp_devlink_params_unregister(struct nfp_pf *pf);
 
 unsigned int nfp_net_lr2speed(unsigned int linkrate);
 unsigned int nfp_net_speed2lr(unsigned int speed);
+
+u8 nfp_get_pf_id(struct nfp_pf *pf);
 #endif /* NFP_MAIN_H */
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_dev.c b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_dev.c
index 0725b51c2a95..8a7c5de0de77 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_dev.c
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_dev.c
@@ -19,6 +19,7 @@ const struct nfp_dev_info nfp_dev_info[NFP_DEV_CNT] = {
 		.pcie_cfg_expbar_offset	= 0x0a00,
 		.pcie_expl_offset	= 0xd000,
 		.qc_area_sz		= 0x100000,
+		.pf_num_per_unit	= 4,
 	},
 	[NFP_DEV_NFP3800_VF] = {
 		.dma_mask		= DMA_BIT_MASK(48),
@@ -38,6 +39,7 @@ const struct nfp_dev_info nfp_dev_info[NFP_DEV_CNT] = {
 		.pcie_cfg_expbar_offset	= 0x0400,
 		.pcie_expl_offset	= 0x1000,
 		.qc_area_sz		= 0x80000,
+		.pf_num_per_unit	= 1,
 	},
 	[NFP_DEV_NFP6000_VF] = {
 		.dma_mask		= DMA_BIT_MASK(40),
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_dev.h b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_dev.h
index e4d38178de0f..d948c9c4a09a 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_dev.h
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_dev.h
@@ -35,6 +35,7 @@ struct nfp_dev_info {
 	u32 pcie_cfg_expbar_offset;
 	u32 pcie_expl_offset;
 	u32 qc_area_sz;
+	u8 pf_num_per_unit;
 };
 
 extern const struct nfp_dev_info nfp_dev_info[NFP_DEV_CNT];
-- 
2.34.1

[PATCH net-next v2 09/13] nfp: apply one port per PF for multi-PF setup

[Louis Peens]

From: Tianyu Yuan <tianyu.yuan@corigine.com>

Only one port per PF is allowed in multi-PF setup. While eth_table
still carries the total port info, each PF need bind itself with
correct port according to PF id.

Signed-off-by: Tianyu Yuan <tianyu.yuan@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
 drivers/net/ethernet/netronome/nfp/abm/main.c   |  2 +-
 drivers/net/ethernet/netronome/nfp/bpf/main.c   |  2 +-
 .../net/ethernet/netronome/nfp/flower/main.c    | 17 ++++++++++-------
 drivers/net/ethernet/netronome/nfp/nfp_main.h   |  6 ++++++
 .../net/ethernet/netronome/nfp/nfp_net_main.c   | 11 ++++++++---
 drivers/net/ethernet/netronome/nfp/nfp_port.c   |  4 +++-
 drivers/net/ethernet/netronome/nfp/nic/main.c   |  3 ++-
 7 files changed, 31 insertions(+), 14 deletions(-)

diff --git a/drivers/net/ethernet/netronome/nfp/abm/main.c b/drivers/net/ethernet/netronome/nfp/abm/main.c
index 5d3df28c648f..d4acaa15629d 100644
--- a/drivers/net/ethernet/netronome/nfp/abm/main.c
+++ b/drivers/net/ethernet/netronome/nfp/abm/main.c
@@ -451,7 +451,7 @@ static int nfp_abm_init(struct nfp_app *app)
 		nfp_err(pf->cpp, "ABM NIC requires ETH table\n");
 		return -EINVAL;
 	}
-	if (pf->max_data_vnics != pf->eth_tbl->count) {
+	if (pf->max_data_vnics != pf->eth_tbl->count && !pf->multi_pf.en) {
 		nfp_err(pf->cpp, "ETH entries don't match vNICs (%d vs %d)\n",
 			pf->max_data_vnics, pf->eth_tbl->count);
 		return -EINVAL;
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/main.c b/drivers/net/ethernet/netronome/nfp/bpf/main.c
index f469950c7265..3d928dfba114 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/main.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/main.c
@@ -70,7 +70,7 @@ nfp_bpf_vnic_alloc(struct nfp_app *app, struct nfp_net *nn, unsigned int id)
 		nfp_err(pf->cpp, "No ETH table\n");
 		return -EINVAL;
 	}
-	if (pf->max_data_vnics != pf->eth_tbl->count) {
+	if (pf->max_data_vnics != pf->eth_tbl->count && !pf->multi_pf.en) {
 		nfp_err(pf->cpp, "ETH entries don't match vNICs (%d vs %d)\n",
 			pf->max_data_vnics, pf->eth_tbl->count);
 		return -EINVAL;
diff --git a/drivers/net/ethernet/netronome/nfp/flower/main.c b/drivers/net/ethernet/netronome/nfp/flower/main.c
index 565987f0a595..2e79b6d981de 100644
--- a/drivers/net/ethernet/netronome/nfp/flower/main.c
+++ b/drivers/net/ethernet/netronome/nfp/flower/main.c
@@ -428,10 +428,10 @@ nfp_flower_spawn_vnic_reprs(struct nfp_app *app,
 			goto err_reprs_clean;
 		}
 		if (repr_type == NFP_REPR_TYPE_PF) {
-			port->pf_id = i;
+			port->pf_id = nfp_net_get_id(app->pf, i);
 			port->vnic = priv->nn->dp.ctrl_bar;
 		} else {
-			port->pf_id = 0;
+			port->pf_id = nfp_net_get_id(app->pf, 0);
 			port->vf_id = i;
 			port->vnic =
 				app->pf->vf_cfg_mem + i * NFP_NET_CFG_BAR_SZ;
@@ -496,28 +496,31 @@ nfp_flower_spawn_phy_reprs(struct nfp_app *app, struct nfp_flower_priv *priv)
 	struct nfp_eth_table *eth_tbl = app->pf->eth_tbl;
 	atomic_t *replies = &priv->reify_replies;
 	struct nfp_flower_repr_priv *repr_priv;
+	int err, reify_cnt, phy_reprs_num;
 	struct nfp_repr *nfp_repr;
 	struct sk_buff *ctrl_skb;
 	struct nfp_reprs *reprs;
-	int err, reify_cnt;
 	unsigned int i;
 
 	ctrl_skb = nfp_flower_cmsg_mac_repr_start(app, eth_tbl->count);
 	if (!ctrl_skb)
 		return -ENOMEM;
 
+	phy_reprs_num = app->pf->multi_pf.en ? app->pf->max_data_vnics : eth_tbl->count;
 	reprs = nfp_reprs_alloc(eth_tbl->max_index + 1);
 	if (!reprs) {
 		err = -ENOMEM;
 		goto err_free_ctrl_skb;
 	}
 
-	for (i = 0; i < eth_tbl->count; i++) {
-		unsigned int phys_port = eth_tbl->ports[i].index;
+	for (i = 0; i < phy_reprs_num; i++) {
+		int idx = nfp_net_get_id(app->pf, i);
 		struct net_device *repr;
+		unsigned int phys_port;
 		struct nfp_port *port;
 		u32 cmsg_port_id;
 
+		phys_port = eth_tbl->ports[idx].index;
 		repr = nfp_repr_alloc(app);
 		if (!repr) {
 			err = -ENOMEM;
@@ -542,7 +545,7 @@ nfp_flower_spawn_phy_reprs(struct nfp_app *app, struct nfp_flower_priv *priv)
 			nfp_repr_free(repr);
 			goto err_reprs_clean;
 		}
-		err = nfp_port_init_phy_port(app->pf, app, port, i);
+		err = nfp_port_init_phy_port(app->pf, app, port, idx);
 		if (err) {
 			kfree(repr_priv);
 			nfp_port_free(port);
@@ -609,7 +612,7 @@ nfp_flower_spawn_phy_reprs(struct nfp_app *app, struct nfp_flower_priv *priv)
 static int nfp_flower_vnic_alloc(struct nfp_app *app, struct nfp_net *nn,
 				 unsigned int id)
 {
-	if (id > 0) {
+	if (id > 0 && !app->pf->multi_pf.en) {
 		nfp_warn(app->cpp, "FlowerNIC doesn't support more than one data vNIC\n");
 		goto err_invalid_port;
 	}
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.h b/drivers/net/ethernet/netronome/nfp/nfp_main.h
index 7f76c718fef8..4f6763ca1c92 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.h
@@ -210,4 +210,10 @@ unsigned int nfp_net_lr2speed(unsigned int linkrate);
 unsigned int nfp_net_speed2lr(unsigned int speed);
 
 u8 nfp_get_pf_id(struct nfp_pf *pf);
+
+static inline unsigned int nfp_net_get_id(const struct nfp_pf *pf, unsigned int id)
+{
+	return pf->multi_pf.en ? pf->multi_pf.id : id;
+}
+
 #endif /* NFP_MAIN_H */
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
index cbe4972ba104..98e155d79eb8 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
@@ -141,7 +141,7 @@ nfp_net_pf_init_vnic(struct nfp_pf *pf, struct nfp_net *nn, unsigned int id)
 {
 	int err;
 
-	nn->id = id;
+	nn->id = nfp_net_get_id(pf, id);
 
 	if (nn->port) {
 		err = nfp_devlink_port_register(pf->app, nn->port);
@@ -183,8 +183,8 @@ nfp_net_pf_alloc_vnics(struct nfp_pf *pf, void __iomem *ctrl_bar,
 	int err;
 
 	for (i = 0; i < pf->max_data_vnics; i++) {
-		nn = nfp_net_pf_alloc_vnic(pf, true, ctrl_bar, qc_bar,
-					   stride, i);
+		nn = nfp_net_pf_alloc_vnic(pf, true, ctrl_bar, qc_bar, stride,
+					   nfp_net_get_id(pf, i));
 		if (IS_ERR(nn)) {
 			err = PTR_ERR(nn);
 			goto err_free_prev;
@@ -707,6 +707,11 @@ int nfp_net_pci_probe(struct nfp_pf *pf)
 	if ((int)pf->max_data_vnics < 0)
 		return pf->max_data_vnics;
 
+	if (pf->multi_pf.en && pf->max_data_vnics != 1) {
+		nfp_err(pf->cpp, "Only one data_vnic per PF is supported in multiple PF setup.\n");
+		return -EINVAL;
+	}
+
 	err = nfp_net_pci_map_mem(pf);
 	if (err)
 		return err;
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_port.c b/drivers/net/ethernet/netronome/nfp/nfp_port.c
index 54640bcb70fb..c1612a464b5d 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_port.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_port.c
@@ -189,7 +189,9 @@ int nfp_port_init_phy_port(struct nfp_pf *pf, struct nfp_app *app,
 
 	port->eth_port = &pf->eth_tbl->ports[id];
 	port->eth_id = pf->eth_tbl->ports[id].index;
-	port->netdev->dev_port = id;
+	if (!pf->multi_pf.en)
+		port->netdev->dev_port = id;
+
 	if (pf->mac_stats_mem)
 		port->eth_stats =
 			pf->mac_stats_mem + port->eth_id * NFP_MAC_STATS_SIZE;
diff --git a/drivers/net/ethernet/netronome/nfp/nic/main.c b/drivers/net/ethernet/netronome/nfp/nic/main.c
index 9dd5afe37f6e..e7a2d01bcbff 100644
--- a/drivers/net/ethernet/netronome/nfp/nic/main.c
+++ b/drivers/net/ethernet/netronome/nfp/nic/main.c
@@ -12,7 +12,8 @@ static int nfp_nic_init(struct nfp_app *app)
 {
 	struct nfp_pf *pf = app->pf;
 
-	if (pf->eth_tbl && pf->max_data_vnics != pf->eth_tbl->count) {
+	if (pf->eth_tbl && pf->max_data_vnics != pf->eth_tbl->count &&
+	    !pf->multi_pf.en) {
 		nfp_err(pf->cpp, "ETH entries don't match vNICs (%d vs %d)\n",
 			pf->max_data_vnics, pf->eth_tbl->count);
 		return -EINVAL;
-- 
2.34.1

[PATCH net-next v2 10/13] nfp: enable multi-PF in application firmware if supported

[Louis Peens]

From: Yinjun Zhang <yinjun.zhang@corigine.com>

For backward compatibility concern, the new application firmware
is designed to support both single-PF setup and multi-PF setup.
Thus driver should inform application firmware which setup current
is. This should be done as early as possible since the setup may
affect some configurations exposed by firmware.

Signed-off-by: Yinjun Zhang <yinjun.zhang@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
 .../net/ethernet/netronome/nfp/nfp_net_ctrl.h |   1 +
 .../net/ethernet/netronome/nfp/nfp_net_main.c | 129 ++++++++++++++----
 2 files changed, 100 insertions(+), 30 deletions(-)

diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h b/drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h
index 3e63f6d6a563..d6b127f13ed3 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h
@@ -268,6 +268,7 @@
 #define   NFP_NET_CFG_CTRL_PKT_TYPE	  (0x1 << 0) /* Pkttype offload */
 #define   NFP_NET_CFG_CTRL_IPSEC	  (0x1 << 1) /* IPsec offload */
 #define   NFP_NET_CFG_CTRL_MCAST_FILTER	  (0x1 << 2) /* Multicast Filter */
+#define   NFP_NET_CFG_CTRL_MULTI_PF	  (0x1 << 5) /* Multi PF */
 #define   NFP_NET_CFG_CTRL_FREELIST_EN	  (0x1 << 6) /* Freelist enable flag bit */
 
 #define NFP_NET_CFG_CAP_WORD1		0x00a4
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
index 98e155d79eb8..f6f4fea0a791 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
@@ -684,15 +684,108 @@ int nfp_net_refresh_eth_port(struct nfp_port *port)
 	return ret;
 }
 
+/**
+ * nfp_net_pre_init() - Some necessary check and configuration
+ * in firmware before fully utilizing it.
+ * @pf: NFP PF handler
+ * @stride: queue stride
+ *
+ * Return: 0 on success, a negative error code otherwise.
+ */
+static int nfp_net_pre_init(struct nfp_pf *pf, int *stride)
+{
+	struct nfp_net_fw_version fw_ver;
+	struct nfp_cpp_area *area;
+	u8 __iomem *ctrl_bar;
+	int err = 0;
+
+	ctrl_bar = nfp_pf_map_rtsym(pf, NULL, "_pf%d_net_bar0", NFP_PF_CSR_SLICE_SIZE, &area);
+	if (IS_ERR(ctrl_bar)) {
+		nfp_err(pf->cpp, "Failed to find data vNIC memory symbol\n");
+		return PTR_ERR(ctrl_bar);
+	}
+
+	nfp_net_get_fw_version(&fw_ver, ctrl_bar);
+	if (fw_ver.extend & NFP_NET_CFG_VERSION_RESERVED_MASK ||
+	    fw_ver.class != NFP_NET_CFG_VERSION_CLASS_GENERIC) {
+		nfp_err(pf->cpp, "Unknown Firmware ABI %d.%d.%d.%d\n",
+			fw_ver.extend, fw_ver.class,
+			fw_ver.major, fw_ver.minor);
+		err = -EINVAL;
+		goto end;
+	}
+
+	/* Determine stride */
+	if (nfp_net_fw_ver_eq(&fw_ver, 0, 0, 0, 1)) {
+		*stride = 2;
+		nfp_warn(pf->cpp, "OBSOLETE Firmware detected - VF isolation not available\n");
+	} else {
+		switch (fw_ver.major) {
+		case 1 ... 5:
+			*stride = 4;
+			break;
+		default:
+			nfp_err(pf->cpp, "Unsupported Firmware ABI %d.%d.%d.%d\n",
+				fw_ver.extend, fw_ver.class,
+				fw_ver.major, fw_ver.minor);
+			err = -EINVAL;
+			goto end;
+		}
+	}
+
+	if (!pf->multi_pf.en)
+		goto end;
+
+	/* Enable multi-PF. */
+	if (readl(ctrl_bar + NFP_NET_CFG_CAP_WORD1) & NFP_NET_CFG_CTRL_MULTI_PF) {
+		unsigned long long addr;
+		u32 cfg_q, cpp_id, ret;
+		unsigned long timeout;
+
+		writel(NFP_NET_CFG_CTRL_MULTI_PF, ctrl_bar + NFP_NET_CFG_CTRL_WORD1);
+		writel(NFP_NET_CFG_UPDATE_GEN, ctrl_bar + NFP_NET_CFG_UPDATE);
+
+		/* Config queue is next to txq. */
+		cfg_q = readl(ctrl_bar + NFP_NET_CFG_START_TXQ) + 1;
+		addr = nfp_qcp_queue_offset(pf->dev_info, cfg_q) + NFP_QCP_QUEUE_ADD_WPTR;
+		cpp_id = NFP_CPP_ISLAND_ID(0, NFP_CPP_ACTION_RW, 0, 0);
+		err = nfp_cpp_writel(pf->cpp, cpp_id, addr, 1);
+		if (err)
+			goto end;
+
+		timeout = jiffies + HZ * NFP_NET_POLL_TIMEOUT;
+		while ((ret = readl(ctrl_bar + NFP_NET_CFG_UPDATE))) {
+			if (ret & NFP_NET_CFG_UPDATE_ERR) {
+				nfp_err(pf->cpp, "Enable multi-PF failed\n");
+				err = -EIO;
+				break;
+			}
+
+			usleep_range(250, 500);
+			if (time_is_before_eq_jiffies(timeout)) {
+				nfp_err(pf->cpp, "Enable multi-PF timeout\n");
+				err = -ETIMEDOUT;
+				break;
+			}
+		};
+	} else {
+		nfp_err(pf->cpp, "Loaded firmware doesn't support multi-PF\n");
+		err = -EINVAL;
+	}
+
+end:
+	nfp_cpp_area_release_free(area);
+	return err;
+}
+
 /*
  * PCI device functions
  */
 int nfp_net_pci_probe(struct nfp_pf *pf)
 {
 	struct devlink *devlink = priv_to_devlink(pf);
-	struct nfp_net_fw_version fw_ver;
 	u8 __iomem *ctrl_bar, *qc_bar;
-	int stride;
+	int stride = 0;
 	int err;
 
 	INIT_WORK(&pf->port_refresh_work, nfp_net_refresh_vnics);
@@ -703,6 +796,10 @@ int nfp_net_pci_probe(struct nfp_pf *pf)
 		return -EINVAL;
 	}
 
+	err = nfp_net_pre_init(pf, &stride);
+	if (err)
+		return err;
+
 	pf->max_data_vnics = nfp_net_pf_get_num_ports(pf);
 	if ((int)pf->max_data_vnics < 0)
 		return pf->max_data_vnics;
@@ -723,34 +820,6 @@ int nfp_net_pci_probe(struct nfp_pf *pf)
 		goto err_unmap;
 	}
 
-	nfp_net_get_fw_version(&fw_ver, ctrl_bar);
-	if (fw_ver.extend & NFP_NET_CFG_VERSION_RESERVED_MASK ||
-	    fw_ver.class != NFP_NET_CFG_VERSION_CLASS_GENERIC) {
-		nfp_err(pf->cpp, "Unknown Firmware ABI %d.%d.%d.%d\n",
-			fw_ver.extend, fw_ver.class,
-			fw_ver.major, fw_ver.minor);
-		err = -EINVAL;
-		goto err_unmap;
-	}
-
-	/* Determine stride */
-	if (nfp_net_fw_ver_eq(&fw_ver, 0, 0, 0, 1)) {
-		stride = 2;
-		nfp_warn(pf->cpp, "OBSOLETE Firmware detected - VF isolation not available\n");
-	} else {
-		switch (fw_ver.major) {
-		case 1 ... 5:
-			stride = 4;
-			break;
-		default:
-			nfp_err(pf->cpp, "Unsupported Firmware ABI %d.%d.%d.%d\n",
-				fw_ver.extend, fw_ver.class,
-				fw_ver.major, fw_ver.minor);
-			err = -EINVAL;
-			goto err_unmap;
-		}
-	}
-
 	err = nfp_net_pf_app_init(pf, qc_bar, stride);
 	if (err)
 		goto err_unmap;
-- 
2.34.1

[PATCH net-next v2 11/13] nfp: configure VF total count for each PF

[Louis Peens]

From: Tianyu Yuan <tianyu.yuan@corigine.com>

By default, PFs share the total 64 VFs equally, i.e., 32 VFs
for each PF in two port NIC, which is initialized in each
PF’s SR-IOV capability register by management firmware.

And a new hwinfo `abi_total_vf` is introduced to make each
PF’s VF total count configurable. Management firmware reads the
hwinfo and configures it in SR-IOV capability register during
boot process. So reboot is required to make the configuration
take effect. This is not touched in driver code.

Driver then modifies each PF’s `sriov_totalvf` according to
maximum VF count supported by the loaded application firmware.
Here we apply the rule that the PF with smaller id is satisfied
first if total configured count exceeds the limitation.

Signed-off-by: Tianyu Yuan <tianyu.yuan@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
 drivers/net/ethernet/netronome/nfp/nfp_main.c | 49 +++++++++++++++++--
 1 file changed, 45 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.c b/drivers/net/ethernet/netronome/nfp/nfp_main.c
index 74767729e542..b15b5fe0c1c9 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.c
@@ -224,11 +224,48 @@ static int nfp_pf_board_state_wait(struct nfp_pf *pf)
 	return 0;
 }
 
+static unsigned int nfp_pf_get_limit_vfs(struct nfp_pf *pf,
+					 unsigned int limit_vfs_rtsym)
+{
+	u16 pos, offset, total;
+
+	if (!pf->multi_pf.en || !limit_vfs_rtsym)
+		return limit_vfs_rtsym;
+
+	pos = pci_find_ext_capability(pf->pdev, PCI_EXT_CAP_ID_SRIOV);
+	if (!pos)
+		return 0;
+
+	/* Management firmware ensures that SR-IOV capability registers
+	 * are initialized correctly.
+	 */
+	pci_read_config_word(pf->pdev, pos + PCI_SRIOV_VF_OFFSET, &offset);
+	pci_read_config_word(pf->pdev, pos + PCI_SRIOV_TOTAL_VF, &total);
+	if (!total)
+		return 0;
+
+	/* Offset of first VF is relative to its PF. */
+	offset += pf->multi_pf.id;
+	if (offset < pf->dev_info->pf_num_per_unit)
+		return 0;
+
+	/* For multi-PF device, VF is numbered from max PF count. */
+	offset -= pf->dev_info->pf_num_per_unit;
+	if (offset >= limit_vfs_rtsym)
+		return 0;
+
+	if (offset + total > limit_vfs_rtsym)
+		return limit_vfs_rtsym - offset;
+
+	return total;
+}
+
 static int nfp_pcie_sriov_read_nfd_limit(struct nfp_pf *pf)
 {
+	unsigned int limit_vfs_rtsym;
 	int err;
 
-	pf->limit_vfs = nfp_rtsym_read_le(pf->rtbl, "nfd_vf_cfg_max_vfs", &err);
+	limit_vfs_rtsym = nfp_rtsym_read_le(pf->rtbl, "nfd_vf_cfg_max_vfs", &err);
 	if (err) {
 		/* For backwards compatibility if symbol not found allow all */
 		pf->limit_vfs = ~0;
@@ -239,9 +276,13 @@ static int nfp_pcie_sriov_read_nfd_limit(struct nfp_pf *pf)
 		return err;
 	}
 
-	err = pci_sriov_set_totalvfs(pf->pdev, pf->limit_vfs);
-	if (err)
-		nfp_warn(pf->cpp, "Failed to set VF count in sysfs: %d\n", err);
+	pf->limit_vfs = nfp_pf_get_limit_vfs(pf, limit_vfs_rtsym);
+	if (pci_sriov_get_totalvfs(pf->pdev) != pf->limit_vfs) {
+		err = pci_sriov_set_totalvfs(pf->pdev, pf->limit_vfs);
+		if (err)
+			nfp_warn(pf->cpp, "Failed to set VF count in sysfs: %d\n", err);
+	}
+
 	return 0;
 }
 
-- 
2.34.1

[PATCH net-next v2 12/13] nfp: configure VF split info into application firmware

[Louis Peens]

From: Yinjun Zhang <yinjun.zhang@corigine.com>

In multi-PF case, all PFs share total 64 VFs. To support the VF
count of each PF configurable, driver needs to write the VF count
and the first VF id into application firmware, so that firmware
can initialize and allocate relevant resource accordingly.

Signed-off-by: Yinjun Zhang <yinjun.zhang@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
 drivers/net/ethernet/netronome/nfp/nfp_main.c |  1 +
 drivers/net/ethernet/netronome/nfp/nfp_main.h |  2 ++
 .../net/ethernet/netronome/nfp/nfp_net_main.c | 16 ++++++++++++
 .../ethernet/netronome/nfp/nfp_net_sriov.c    | 25 +++++++++++++++++++
 .../ethernet/netronome/nfp/nfp_net_sriov.h    |  5 ++++
 5 files changed, 49 insertions(+)

diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.c b/drivers/net/ethernet/netronome/nfp/nfp_main.c
index b15b5fe0c1c9..70e140e7d93b 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.c
@@ -254,6 +254,7 @@ static unsigned int nfp_pf_get_limit_vfs(struct nfp_pf *pf,
 	if (offset >= limit_vfs_rtsym)
 		return 0;
 
+	pf->multi_pf.vf_fid = offset;
 	if (offset + total > limit_vfs_rtsym)
 		return limit_vfs_rtsym - offset;
 
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.h b/drivers/net/ethernet/netronome/nfp/nfp_main.h
index 4f6763ca1c92..e7f125a3f884 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.h
@@ -87,6 +87,7 @@ struct nfp_dumpspec {
  * @multi_pf:		Used in multi-PF setup
  * @multi_pf.en:	True if it's a NIC with multiple PFs
  * @multi_pf.id:	PF index
+ * @multi_pf.vf_fid:	Id of first VF that belongs to this PF
  * @multi_pf.beat_timer:Timer for beat to keepalive
  * @multi_pf.beat_area:	Pointer to CPP area for beat to keepalive
  * @multi_pf.beat_addr:	Pointer to mapped beat address used for keepalive
@@ -151,6 +152,7 @@ struct nfp_pf {
 	struct {
 		bool en;
 		u8 id;
+		u8 vf_fid;
 		struct timer_list beat_timer;
 		struct nfp_cpp_area *beat_area;
 		u8 __iomem *beat_addr;
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
index f6f4fea0a791..eb7b0ecd65df 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
@@ -293,6 +293,16 @@ static int nfp_net_pf_init_vnics(struct nfp_pf *pf)
 	return err;
 }
 
+static void nfp_net_pf_clean_vnics(struct nfp_pf *pf)
+{
+	struct nfp_net *nn;
+
+	list_for_each_entry(nn, &pf->vnics, vnic_list) {
+		if (nfp_net_is_data_vnic(nn))
+			nfp_net_pf_clean_vnic(pf, nn);
+	}
+}
+
 static int
 nfp_net_pf_app_init(struct nfp_pf *pf, u8 __iomem *qc_bar, unsigned int stride)
 {
@@ -852,11 +862,17 @@ int nfp_net_pci_probe(struct nfp_pf *pf)
 	if (err)
 		goto err_stop_app;
 
+	err = nfp_net_pf_init_sriov(pf);
+	if (err)
+		goto err_clean_vnics;
+
 	devl_unlock(devlink);
 	devlink_register(devlink);
 
 	return 0;
 
+err_clean_vnics:
+	nfp_net_pf_clean_vnics(pf);
 err_stop_app:
 	nfp_net_pf_app_stop(pf);
 err_free_irqs:
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c b/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c
index 6eeeb0fda91f..f516ba7a429e 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c
@@ -14,6 +14,9 @@
 #include "nfp_net.h"
 #include "nfp_net_sriov.h"
 
+/* The configurations that precede VF creating. */
+#define NFP_NET_VF_PRE_CONFIG	NFP_NET_VF_CFG_MB_CAP_SPLIT
+
 static int
 nfp_net_sriov_check(struct nfp_app *app, int vf, u16 cap, const char *msg, bool warn)
 {
@@ -29,6 +32,10 @@ nfp_net_sriov_check(struct nfp_app *app, int vf, u16 cap, const char *msg, bool
 		return -EOPNOTSUPP;
 	}
 
+	/* No need to check vf for the pre-configurations. */
+	if (cap & NFP_NET_VF_PRE_CONFIG)
+		return 0;
+
 	if (vf < 0 || vf >= app->pf->num_vfs) {
 		if (warn)
 			nfp_warn(app->pf->cpp, "invalid VF id %d\n", vf);
@@ -309,3 +316,21 @@ int nfp_app_get_vf_config(struct net_device *netdev, int vf,
 
 	return 0;
 }
+
+int nfp_net_pf_init_sriov(struct nfp_pf *pf)
+{
+	int err;
+
+	if (!pf->multi_pf.en || !pf->limit_vfs)
+		return 0;
+
+	err = nfp_net_sriov_check(pf->app, 0, NFP_NET_VF_CFG_MB_CAP_SPLIT, "split", true);
+	if (err)
+		return err;
+
+	writeb(pf->limit_vfs, pf->vfcfg_tbl2 + NFP_NET_VF_CFG_MB_VF_CNT);
+
+	/* Reuse NFP_NET_VF_CFG_MB_VF_NUM to pass vf_fid to FW. */
+	return nfp_net_sriov_update(pf->app, pf->multi_pf.vf_fid,
+				    NFP_NET_VF_CFG_MB_UPD_SPLIT, "split");
+}
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.h b/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.h
index 2d445fa199dc..8de959018819 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.h
@@ -21,6 +21,7 @@
 #define   NFP_NET_VF_CFG_MB_CAP_TRUST			  (0x1 << 4)
 #define   NFP_NET_VF_CFG_MB_CAP_VLAN_PROTO		  (0x1 << 5)
 #define   NFP_NET_VF_CFG_MB_CAP_RATE			  (0x1 << 6)
+#define   NFP_NET_VF_CFG_MB_CAP_SPLIT			  (0x1 << 8)
 #define NFP_NET_VF_CFG_MB_RET				0x2
 #define NFP_NET_VF_CFG_MB_UPD				0x4
 #define   NFP_NET_VF_CFG_MB_UPD_MAC			  (0x1 << 0)
@@ -30,6 +31,8 @@
 #define   NFP_NET_VF_CFG_MB_UPD_TRUST			  (0x1 << 4)
 #define   NFP_NET_VF_CFG_MB_UPD_VLAN_PROTO		  (0x1 << 5)
 #define   NFP_NET_VF_CFG_MB_UPD_RATE			  (0x1 << 6)
+#define   NFP_NET_VF_CFG_MB_UPD_SPLIT			  (0x1 << 8)
+#define NFP_NET_VF_CFG_MB_VF_CNT			0x6
 #define NFP_NET_VF_CFG_MB_VF_NUM			0x7
 
 /* VF config entry
@@ -68,4 +71,6 @@ int nfp_app_set_vf_link_state(struct net_device *netdev, int vf,
 int nfp_app_get_vf_config(struct net_device *netdev, int vf,
 			  struct ifla_vf_info *ivi);
 
+int nfp_net_pf_init_sriov(struct nfp_pf *pf);
+
 #endif /* _NFP_NET_SRIOV_H_ */
-- 
2.34.1

[PATCH net-next v2 13/13] nfp: use absolute vf id for multi-PF case

[Louis Peens]

From: Yinjun Zhang <yinjun.zhang@corigine.com>

In multi-PF setup, absolute VF id is required to configure attributes
for corresponding VF.

Add helper function to map rtsym with specified offset. With PF's first
VF as base offset, we can access `vf_cfg_mem` as before.

Signed-off-by: Yinjun Zhang <yinjun.zhang@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
 drivers/net/ethernet/netronome/nfp/nfp_main.c    | 14 +++++++++++---
 drivers/net/ethernet/netronome/nfp/nfp_main.h    |  4 ++++
 .../net/ethernet/netronome/nfp/nfp_net_main.c    | 10 ++++++----
 .../net/ethernet/netronome/nfp/nfp_net_sriov.c   | 14 ++++++++++----
 .../ethernet/netronome/nfp/nfpcore/nfp_nffw.h    |  4 ++++
 .../ethernet/netronome/nfp/nfpcore/nfp_rtsym.c   | 16 ++++++++++++----
 6 files changed, 47 insertions(+), 15 deletions(-)

diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.c b/drivers/net/ethernet/netronome/nfp/nfp_main.c
index 70e140e7d93b..139499d891c1 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.c
@@ -97,14 +97,22 @@ int nfp_pf_rtsym_read_optional(struct nfp_pf *pf, const char *format,
 }
 
 u8 __iomem *
-nfp_pf_map_rtsym(struct nfp_pf *pf, const char *name, const char *sym_fmt,
-		 unsigned int min_size, struct nfp_cpp_area **area)
+nfp_pf_map_rtsym_offset(struct nfp_pf *pf, const char *name, const char *sym_fmt,
+			unsigned int offset, unsigned int min_size,
+			struct nfp_cpp_area **area)
 {
 	char pf_symbol[256];
 
 	snprintf(pf_symbol, sizeof(pf_symbol), sym_fmt, nfp_get_pf_id(pf));
 
-	return nfp_rtsym_map(pf->rtbl, pf_symbol, name, min_size, area);
+	return nfp_rtsym_map_offset(pf->rtbl, pf_symbol, name, offset, min_size, area);
+}
+
+u8 __iomem *
+nfp_pf_map_rtsym(struct nfp_pf *pf, const char *name, const char *sym_fmt,
+		 unsigned int min_size, struct nfp_cpp_area **area)
+{
+	return nfp_pf_map_rtsym_offset(pf, name, sym_fmt, 0, min_size, area);
 }
 
 /* Callers should hold the devlink instance lock */
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.h b/drivers/net/ethernet/netronome/nfp/nfp_main.h
index e7f125a3f884..4f1623917c4e 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.h
@@ -179,6 +179,10 @@ int nfp_pf_rtsym_read_optional(struct nfp_pf *pf, const char *format,
 			       unsigned int default_val);
 int nfp_net_pf_get_app_id(struct nfp_pf *pf);
 u8 __iomem *
+nfp_pf_map_rtsym_offset(struct nfp_pf *pf, const char *name, const char *sym_fmt,
+			unsigned int offset, unsigned int min_size,
+			struct nfp_cpp_area **area);
+u8 __iomem *
 nfp_pf_map_rtsym(struct nfp_pf *pf, const char *name, const char *sym_fmt,
 		 unsigned int min_size, struct nfp_cpp_area **area);
 int nfp_mbox_cmd(struct nfp_pf *pf, u32 cmd, void *in_data, u64 in_length,
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
index eb7b0ecd65df..f68fd01dac60 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
@@ -473,9 +473,10 @@ static int nfp_net_pci_map_mem(struct nfp_pf *pf)
 		}
 	}
 
-	pf->vf_cfg_mem = nfp_pf_map_rtsym(pf, "net.vfcfg", "_pf%d_net_vf_bar",
-					  NFP_NET_CFG_BAR_SZ * pf->limit_vfs,
-					  &pf->vf_cfg_bar);
+	pf->vf_cfg_mem = nfp_pf_map_rtsym_offset(pf, "net.vfcfg", "_pf%d_net_vf_bar",
+						 NFP_NET_CFG_BAR_SZ * pf->multi_pf.vf_fid,
+						 NFP_NET_CFG_BAR_SZ * pf->limit_vfs,
+						 &pf->vf_cfg_bar);
 	if (IS_ERR(pf->vf_cfg_mem)) {
 		if (PTR_ERR(pf->vf_cfg_mem) != -ENOENT) {
 			err = PTR_ERR(pf->vf_cfg_mem);
@@ -484,7 +485,8 @@ static int nfp_net_pci_map_mem(struct nfp_pf *pf)
 		pf->vf_cfg_mem = NULL;
 	}
 
-	min_size = NFP_NET_VF_CFG_SZ * pf->limit_vfs + NFP_NET_VF_CFG_MB_SZ;
+	min_size = NFP_NET_VF_CFG_SZ * (pf->limit_vfs + pf->multi_pf.vf_fid) +
+		   NFP_NET_VF_CFG_MB_SZ;
 	pf->vfcfg_tbl2 = nfp_pf_map_rtsym(pf, "net.vfcfg_tbl2",
 					  "_pf%d_net_vf_cfg2",
 					  min_size, &pf->vfcfg_tbl2_area);
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c b/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c
index f516ba7a429e..67aea9445aa2 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c
@@ -72,7 +72,7 @@ int nfp_app_set_vf_mac(struct net_device *netdev, int vf, u8 *mac)
 {
 	struct nfp_app *app = nfp_app_from_netdev(netdev);
 	unsigned int vf_offset;
-	int err;
+	int err, abs_vf;
 
 	err = nfp_net_sriov_check(app, vf, NFP_NET_VF_CFG_MB_CAP_MAC, "mac", true);
 	if (err)
@@ -85,13 +85,14 @@ int nfp_app_set_vf_mac(struct net_device *netdev, int vf, u8 *mac)
 		return -EINVAL;
 	}
 
+	abs_vf = vf + app->pf->multi_pf.vf_fid;
 	/* Write MAC to VF entry in VF config symbol */
-	vf_offset = NFP_NET_VF_CFG_MB_SZ + vf * NFP_NET_VF_CFG_SZ;
+	vf_offset = NFP_NET_VF_CFG_MB_SZ + abs_vf * NFP_NET_VF_CFG_SZ;
 	writel(get_unaligned_be32(mac), app->pf->vfcfg_tbl2 + vf_offset);
 	writew(get_unaligned_be16(mac + 4),
 	       app->pf->vfcfg_tbl2 + vf_offset + NFP_NET_VF_CFG_MAC_LO);
 
-	err = nfp_net_sriov_update(app, vf, NFP_NET_VF_CFG_MB_UPD_MAC, "MAC");
+	err = nfp_net_sriov_update(app, abs_vf, NFP_NET_VF_CFG_MB_UPD_MAC, "MAC");
 	if (!err)
 		nfp_info(app->pf->cpp,
 			 "MAC %pM set on VF %d, reload the VF driver to make this change effective.\n",
@@ -145,6 +146,7 @@ int nfp_app_set_vf_vlan(struct net_device *netdev, int vf, u16 vlan, u8 qos,
 	if (vlan_tag && is_proto_sup)
 		vlan_tag |= FIELD_PREP(NFP_NET_VF_CFG_VLAN_PROT, ntohs(vlan_proto));
 
+	vf += app->pf->multi_pf.vf_fid;
 	vf_offset = NFP_NET_VF_CFG_MB_SZ + vf * NFP_NET_VF_CFG_SZ;
 	writel(vlan_tag, app->pf->vfcfg_tbl2 + vf_offset + NFP_NET_VF_CFG_VLAN);
 
@@ -169,6 +171,7 @@ int nfp_app_set_vf_rate(struct net_device *netdev, int vf,
 		return -EINVAL;
 	}
 
+	vf += app->pf->multi_pf.vf_fid;
 	vf_offset = NFP_NET_VF_CFG_MB_SZ + vf * NFP_NET_VF_CFG_SZ;
 	ratevalue = FIELD_PREP(NFP_NET_VF_CFG_MAX_RATE,
 			       max_tx_rate ? max_tx_rate :
@@ -195,6 +198,7 @@ int nfp_app_set_vf_spoofchk(struct net_device *netdev, int vf, bool enable)
 		return err;
 
 	/* Write spoof check control bit to VF entry in VF config symbol */
+	vf += app->pf->multi_pf.vf_fid;
 	vf_offset = NFP_NET_VF_CFG_MB_SZ + vf * NFP_NET_VF_CFG_SZ +
 		NFP_NET_VF_CFG_CTRL;
 	vf_ctrl = readb(app->pf->vfcfg_tbl2 + vf_offset);
@@ -219,6 +223,7 @@ int nfp_app_set_vf_trust(struct net_device *netdev, int vf, bool enable)
 		return err;
 
 	/* Write trust control bit to VF entry in VF config symbol */
+	vf += app->pf->multi_pf.vf_fid;
 	vf_offset = NFP_NET_VF_CFG_MB_SZ + vf * NFP_NET_VF_CFG_SZ +
 		NFP_NET_VF_CFG_CTRL;
 	vf_ctrl = readb(app->pf->vfcfg_tbl2 + vf_offset);
@@ -253,6 +258,7 @@ int nfp_app_set_vf_link_state(struct net_device *netdev, int vf,
 	}
 
 	/* Write link state to VF entry in VF config symbol */
+	vf += app->pf->multi_pf.vf_fid;
 	vf_offset = NFP_NET_VF_CFG_MB_SZ + vf * NFP_NET_VF_CFG_SZ +
 		NFP_NET_VF_CFG_CTRL;
 	vf_ctrl = readb(app->pf->vfcfg_tbl2 + vf_offset);
@@ -278,7 +284,7 @@ int nfp_app_get_vf_config(struct net_device *netdev, int vf,
 	if (err)
 		return err;
 
-	vf_offset = NFP_NET_VF_CFG_MB_SZ + vf * NFP_NET_VF_CFG_SZ;
+	vf_offset = NFP_NET_VF_CFG_MB_SZ + (vf + app->pf->multi_pf.vf_fid) * NFP_NET_VF_CFG_SZ;
 
 	mac_hi = readl(app->pf->vfcfg_tbl2 + vf_offset);
 	mac_lo = readw(app->pf->vfcfg_tbl2 + vf_offset + NFP_NET_VF_CFG_MAC_LO);
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nffw.h b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nffw.h
index 49a4d3f56b56..4042352f83b0 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nffw.h
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nffw.h
@@ -101,6 +101,10 @@ u64 nfp_rtsym_read_le(struct nfp_rtsym_table *rtbl, const char *name,
 int nfp_rtsym_write_le(struct nfp_rtsym_table *rtbl, const char *name,
 		       u64 value);
 u8 __iomem *
+nfp_rtsym_map_offset(struct nfp_rtsym_table *rtbl, const char *name, const char *id,
+		     unsigned int offset, unsigned int min_size,
+		     struct nfp_cpp_area **area);
+u8 __iomem *
 nfp_rtsym_map(struct nfp_rtsym_table *rtbl, const char *name, const char *id,
 	      unsigned int min_size, struct nfp_cpp_area **area);
 
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_rtsym.c b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_rtsym.c
index 2260c2403a83..97a4417a1c1b 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_rtsym.c
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_rtsym.c
@@ -520,8 +520,9 @@ int nfp_rtsym_write_le(struct nfp_rtsym_table *rtbl, const char *name,
 }
 
 u8 __iomem *
-nfp_rtsym_map(struct nfp_rtsym_table *rtbl, const char *name, const char *id,
-	      unsigned int min_size, struct nfp_cpp_area **area)
+nfp_rtsym_map_offset(struct nfp_rtsym_table *rtbl, const char *name, const char *id,
+		     unsigned int offset, unsigned int min_size,
+		     struct nfp_cpp_area **area)
 {
 	const struct nfp_rtsym *sym;
 	u8 __iomem *mem;
@@ -540,12 +541,12 @@ nfp_rtsym_map(struct nfp_rtsym_table *rtbl, const char *name, const char *id,
 		return (u8 __iomem *)ERR_PTR(err);
 	}
 
-	if (sym->size < min_size) {
+	if (sym->size < min_size + offset) {
 		nfp_err(rtbl->cpp, "rtsym '%s': too small\n", name);
 		return (u8 __iomem *)ERR_PTR(-EINVAL);
 	}
 
-	mem = nfp_cpp_map_area(rtbl->cpp, id, cpp_id, addr, sym->size, area);
+	mem = nfp_cpp_map_area(rtbl->cpp, id, cpp_id, addr + offset, sym->size - offset, area);
 	if (IS_ERR(mem)) {
 		nfp_err(rtbl->cpp, "rtysm '%s': failed to map: %ld\n",
 			name, PTR_ERR(mem));
@@ -554,3 +555,10 @@ nfp_rtsym_map(struct nfp_rtsym_table *rtbl, const char *name, const char *id,
 
 	return mem;
 }
+
+u8 __iomem *
+nfp_rtsym_map(struct nfp_rtsym_table *rtbl, const char *name, const char *id,
+	      unsigned int min_size, struct nfp_cpp_area **area)
+{
+	return nfp_rtsym_map_offset(rtbl, name, id, 0, min_size, area);
+}
-- 
2.34.1

Re: [PATCH net-next v2 00/13] nfp: add support for multi-pf configuration

[Jakub Kicinski]

On Wed, 16 Aug 2023 16:38:59 +0200 Louis Peens wrote:
> As part of v1 there was also some partially finished discussion about
> devlink allowing to bind to multiple bus devices. This series creates a
> devlink instance per PF, and the comment was asking if this should maybe
> change to be a single instance, since it is still a single device. For
> the moment we feel that this is a parallel issue to this specific
> series, as it seems to be already implemented this way in other places,
> and this series would be matching that.
> 
> We are curious about this idea though, as it does seem to make sense if
> the original devlink idea was that it should have a one-to-one
> correspondence per ASIC. Not sure where one would start with this
> though, on first glance it looks like the assumption that devlink is
> only connected to a single bus device is embedded quite deep. This
> probably needs commenting/discussion with somebody that has pretty good
> knowledge of devlink core.

How do you suggest we move forward? This is a community project after
all, _someone_ has to start the discussion and then write the code.

Re: [PATCH net-next v2 00/13] nfp: add support for multi-pf configuration

[Louis Peens]

On Thu, Aug 17, 2023 at 07:22:05PM -0700, Jakub Kicinski wrote:
> On Wed, 16 Aug 2023 16:38:59 +0200 Louis Peens wrote:
> > As part of v1 there was also some partially finished discussion about
> > devlink allowing to bind to multiple bus devices. This series creates a
> > devlink instance per PF, and the comment was asking if this should maybe
> > change to be a single instance, since it is still a single device. For
> > the moment we feel that this is a parallel issue to this specific
> > series, as it seems to be already implemented this way in other places,
> > and this series would be matching that.
> > 
> > We are curious about this idea though, as it does seem to make sense if
> > the original devlink idea was that it should have a one-to-one
> > correspondence per ASIC. Not sure where one would start with this
> > though, on first glance it looks like the assumption that devlink is
> > only connected to a single bus device is embedded quite deep. This
> > probably needs commenting/discussion with somebody that has pretty good
> > knowledge of devlink core.
> 
> How do you suggest we move forward? This is a community project after
> all, _someone_ has to start the discussion and then write the code.

This is a good point, it would have been nice if things could just be
wished into existence. I will try to negotiate for some time to be spend
on this from our side, and then raise a new topic for discussion.