* [PATCH net-next 01/12] nsp: generate nsp command with variable nsp major version
2023-07-24 9:48 [PATCH net-next 00/12] nfp: add support for multi-pf configuration Louis Peens
@ 2023-07-24 9:48 ` Louis Peens
2023-07-24 9:48 ` [PATCH net-next 02/12] nfp: bump the nsp major version to support multi-PF Louis Peens
` (11 subsequent siblings)
12 siblings, 0 replies; 22+ messages in thread
From: Louis Peens @ 2023-07-24 9:48 UTC (permalink / raw)
To: David Miller, Jakub Kicinski, Paolo Abeni
Cc: Simon Horman, Yinjun Zhang, Tianyu Yuan, netdev, oss-drivers
From: Tianyu Yuan <tianyu.yuan@corigine.com>
The most significant 4 bits of nsp command code should carry the
ABI major version so that nsp command can be responded correctly.
It is working well since current major version is 0.
However management firmware is going to bump the major version to
support multi-PF feature. So change the code to explicitly contain
the major version into nsp command code.
Signed-off-by: Tianyu Yuan <tianyu.yuan@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c
index 7136bc48530b..ee934663c6d9 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c
@@ -37,7 +37,8 @@
#define NSP_COMMAND 0x08
#define NSP_COMMAND_OPTION GENMASK_ULL(63, 32)
-#define NSP_COMMAND_CODE GENMASK_ULL(31, 16)
+#define NSP_COMMAND_CODE_MJ_VER GENMASK_ULL(31, 28)
+#define NSP_COMMAND_CODE GENMASK_ULL(27, 16)
#define NSP_COMMAND_DMA_BUF BIT_ULL(1)
#define NSP_COMMAND_START BIT_ULL(0)
@@ -380,6 +381,7 @@ __nfp_nsp_command(struct nfp_nsp *state, const struct nfp_nsp_command_arg *arg)
err = nfp_cpp_writeq(cpp, nsp_cpp, nsp_command,
FIELD_PREP(NSP_COMMAND_OPTION, arg->option) |
+ FIELD_PREP(NSP_COMMAND_CODE_MJ_VER, state->ver.major) |
FIELD_PREP(NSP_COMMAND_CODE, arg->code) |
FIELD_PREP(NSP_COMMAND_DMA_BUF, arg->dma) |
FIELD_PREP(NSP_COMMAND_START, 1));
--
2.34.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH net-next 02/12] nfp: bump the nsp major version to support multi-PF
2023-07-24 9:48 [PATCH net-next 00/12] nfp: add support for multi-pf configuration Louis Peens
2023-07-24 9:48 ` [PATCH net-next 01/12] nsp: generate nsp command with variable nsp major version Louis Peens
@ 2023-07-24 9:48 ` Louis Peens
2023-07-24 9:48 ` [PATCH net-next 03/12] nfp: change application firmware loading flow in multi-PF setup Louis Peens
` (10 subsequent siblings)
12 siblings, 0 replies; 22+ messages in thread
From: Louis Peens @ 2023-07-24 9:48 UTC (permalink / raw)
To: David Miller, Jakub Kicinski, Paolo Abeni
Cc: Simon Horman, Yinjun Zhang, Tianyu Yuan, netdev, oss-drivers
From: Tianyu Yuan <tianyu.yuan@corigine.com>
Currently NFP NICs implement single PF with multiple ports
instantiated. While NFP3800 can support multiple PFs and
one port per PF is more up-to-date, the management firmware
will start to support multi-PF. Since it's incompatible with
current implementation, the ABI major version is bumped.
A new flag is also introduced to indicate whether it's
multi-PF setup or single-PF setup.
Signed-off-by: Tianyu Yuan <tianyu.yuan@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
drivers/net/ethernet/netronome/nfp/nfp_main.c | 3 +++
drivers/net/ethernet/netronome/nfp/nfp_main.h | 6 ++++++
.../net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c | 14 ++++++++++----
3 files changed, 19 insertions(+), 4 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.c b/drivers/net/ethernet/netronome/nfp/nfp_main.c
index 71301dbd8fb5..39c1327625fa 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.c
@@ -625,6 +625,9 @@ static int nfp_nsp_init(struct pci_dev *pdev, struct nfp_pf *pf)
return err;
}
+ pf->multi_pf.en = pdev->multifunction;
+ dev_info(&pdev->dev, "%s-PF detected\n", pf->multi_pf.en ? "Multi" : "Single");
+
err = nfp_nsp_wait(nsp);
if (err < 0)
goto exit_close_nsp;
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.h b/drivers/net/ethernet/netronome/nfp/nfp_main.h
index 14a751bfe1fe..72ea3b83d313 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.h
@@ -84,6 +84,8 @@ struct nfp_dumpspec {
* @port_refresh_work: Work entry for taking netdevs out
* @shared_bufs: Array of shared buffer structures if FW has any SBs
* @num_shared_bufs: Number of elements in @shared_bufs
+ * @multi_pf: Used in multi-PF setup
+ * @multi_pf.en: True if it's a NIC with multiple PFs
*
* Fields which may change after proble are protected by devlink instance lock.
*/
@@ -141,6 +143,10 @@ struct nfp_pf {
struct nfp_shared_buf *shared_bufs;
unsigned int num_shared_bufs;
+
+ struct {
+ bool en;
+ } multi_pf;
};
extern struct pci_driver nfp_netvf_pci_driver;
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c
index ee934663c6d9..3098a9e52138 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c
@@ -59,7 +59,13 @@
#define NFP_CAP_CMD_DMA_SG 0x28
#define NSP_MAGIC 0xab10
-#define NSP_MAJOR 0
+/* ABI major version is bumped separately without resetting minor
+ * version when the change in NSP is not compatible to old driver.
+ */
+#define NSP_MAJOR 1
+/* ABI minor version is bumped when new feature is introduced
+ * while old driver can still work without this new feature.
+ */
#define NSP_MINOR 8
#define NSP_CODE_MAJOR GENMASK(15, 12)
@@ -248,14 +254,14 @@ static int nfp_nsp_check(struct nfp_nsp *state)
state->ver.major = FIELD_GET(NSP_STATUS_MAJOR, reg);
state->ver.minor = FIELD_GET(NSP_STATUS_MINOR, reg);
- if (state->ver.major != NSP_MAJOR) {
+ if (state->ver.major > NSP_MAJOR) {
nfp_err(cpp, "Unsupported ABI %hu.%hu\n",
state->ver.major, state->ver.minor);
return -EINVAL;
}
if (state->ver.minor < NSP_MINOR) {
- nfp_err(cpp, "ABI too old to support NIC operation (%u.%hu < %u.%u), please update the management FW on the flash\n",
- NSP_MAJOR, state->ver.minor, NSP_MAJOR, NSP_MINOR);
+ nfp_err(cpp, "ABI too old to support NIC operation (x.%u < x.%u), please update the management FW on the flash\n",
+ state->ver.minor, NSP_MINOR);
return -EINVAL;
}
--
2.34.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH net-next 03/12] nfp: change application firmware loading flow in multi-PF setup
2023-07-24 9:48 [PATCH net-next 00/12] nfp: add support for multi-pf configuration Louis Peens
2023-07-24 9:48 ` [PATCH net-next 01/12] nsp: generate nsp command with variable nsp major version Louis Peens
2023-07-24 9:48 ` [PATCH net-next 02/12] nfp: bump the nsp major version to support multi-PF Louis Peens
@ 2023-07-24 9:48 ` Louis Peens
2023-07-24 9:48 ` [PATCH net-next 04/12] nfp: don't skip firmware loading when it's pxe firmware in running Louis Peens
` (9 subsequent siblings)
12 siblings, 0 replies; 22+ messages in thread
From: Louis Peens @ 2023-07-24 9:48 UTC (permalink / raw)
To: David Miller, Jakub Kicinski, Paolo Abeni
Cc: Simon Horman, Yinjun Zhang, Tianyu Yuan, netdev, oss-drivers
From: Yinjun Zhang <yinjun.zhang@corigine.com>
In multi-PF setup, all PFs share the single application firmware.
Each PF is treated equally, and first-come-first-served. So the
first step is to check firmware is loaded or not. And also loading
firmware from disk and flash are treated consistently, both
propagating the failure and setting `fw_loaded` flag. At last,
firmware shouldn't be unloaded in this setup. The following commit
will introduce a keepalive mechanism to let management firmware
manage unloading.
The flow is not changed in non-multi-PF setup.
Signed-off-by: Yinjun Zhang <yinjun.zhang@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
drivers/net/ethernet/netronome/nfp/nfp_main.c | 33 +++++++++++++++----
1 file changed, 26 insertions(+), 7 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.c b/drivers/net/ethernet/netronome/nfp/nfp_main.c
index 39c1327625fa..c81784a626a6 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.c
@@ -528,6 +528,10 @@ nfp_fw_load(struct pci_dev *pdev, struct nfp_pf *pf, struct nfp_nsp *nsp)
if (err)
return err;
+ /* Skip firmware loading in multi-PF setup if firmware is loaded. */
+ if (pf->multi_pf.en && nfp_nsp_fw_loaded(nsp))
+ return 1;
+
fw = nfp_net_fw_find(pdev, pf);
do_reset = reset == NFP_NSP_DRV_RESET_ALWAYS ||
(fw && reset == NFP_NSP_DRV_RESET_DISK);
@@ -556,16 +560,30 @@ nfp_fw_load(struct pci_dev *pdev, struct nfp_pf *pf, struct nfp_nsp *nsp)
fw_loaded = true;
} else if (policy != NFP_NSP_APP_FW_LOAD_DISK &&
nfp_nsp_has_stored_fw_load(nsp)) {
+ err = nfp_nsp_load_stored_fw(nsp);
- /* Don't propagate this error to stick with legacy driver
+ /* For single-PF setup:
+ * Don't propagate this error to stick with legacy driver
* behavior, failure will be detected later during init.
+ * Don't flag the fw_loaded in this case since other devices
+ * may reuse the firmware when configured this way.
+ *
+ * For multi-PF setup:
+ * We only reach here when firmware is freshly loaded from
+ * flash, so need propagate the error and flow the fw_loaded
+ * as it does when loading firmware from disk.
*/
- if (!nfp_nsp_load_stored_fw(nsp))
+ if (!err) {
dev_info(&pdev->dev, "Finished loading stored FW image\n");
- /* Don't flag the fw_loaded in this case since other devices
- * may reuse the firmware when configured this way
- */
+ if (pf->multi_pf.en)
+ fw_loaded = true;
+ } else {
+ if (pf->multi_pf.en)
+ dev_err(&pdev->dev, "Stored FW loading failed: %d\n", err);
+ else
+ err = 0;
+ }
} else {
dev_warn(&pdev->dev, "Didn't load firmware, please update flash or reconfigure card\n");
}
@@ -575,9 +593,10 @@ nfp_fw_load(struct pci_dev *pdev, struct nfp_pf *pf, struct nfp_nsp *nsp)
/* We don't want to unload firmware when other devices may still be
* dependent on it, which could be the case if there are multiple
- * devices that could load firmware.
+ * devices that could load firmware or the case multiple PFs are
+ * running.
*/
- if (fw_loaded && ifcs == 1)
+ if (fw_loaded && ifcs == 1 && !pf->multi_pf.en)
pf->unload_fw_on_remove = true;
return err < 0 ? err : fw_loaded;
--
2.34.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH net-next 04/12] nfp: don't skip firmware loading when it's pxe firmware in running
2023-07-24 9:48 [PATCH net-next 00/12] nfp: add support for multi-pf configuration Louis Peens
` (2 preceding siblings ...)
2023-07-24 9:48 ` [PATCH net-next 03/12] nfp: change application firmware loading flow in multi-PF setup Louis Peens
@ 2023-07-24 9:48 ` Louis Peens
2023-07-24 9:48 ` [PATCH net-next 05/12] nfp: introduce keepalive mechanism for multi-PF setup Louis Peens
` (8 subsequent siblings)
12 siblings, 0 replies; 22+ messages in thread
From: Louis Peens @ 2023-07-24 9:48 UTC (permalink / raw)
To: David Miller, Jakub Kicinski, Paolo Abeni
Cc: Simon Horman, Yinjun Zhang, Tianyu Yuan, netdev, oss-drivers
From: Yinjun Zhang <yinjun.zhang@corigine.com>
In pxe boot case, the pxe firmware is not unloaded in some systems
when booting completes. Driver needs to detect it so that it has
chance to load the correct firmware.
Signed-off-by: Yinjun Zhang <yinjun.zhang@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
drivers/net/ethernet/netronome/nfp/nfp_main.c | 25 +++++++++++++++++--
1 file changed, 23 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.c b/drivers/net/ethernet/netronome/nfp/nfp_main.c
index c81784a626a6..778f21dfbbd5 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.c
@@ -469,6 +469,28 @@ nfp_get_fw_policy_value(struct pci_dev *pdev, struct nfp_nsp *nsp,
return err;
}
+static bool
+nfp_skip_fw_load(struct nfp_pf *pf, struct nfp_nsp *nsp)
+{
+ const struct nfp_mip *mip;
+ bool skip;
+
+ if (!pf->multi_pf.en || nfp_nsp_fw_loaded(nsp) <= 0)
+ return false;
+
+ mip = nfp_mip_open(pf->cpp);
+ if (!mip)
+ return false;
+
+ /* For the case that system boots from pxe, we need
+ * reload FW if pxe FW is running.
+ */
+ skip = !!strncmp(nfp_mip_name(mip), "pxe", 3);
+ nfp_mip_close(mip);
+
+ return skip;
+}
+
/**
* nfp_fw_load() - Load the firmware image
* @pdev: PCI Device structure
@@ -528,8 +550,7 @@ nfp_fw_load(struct pci_dev *pdev, struct nfp_pf *pf, struct nfp_nsp *nsp)
if (err)
return err;
- /* Skip firmware loading in multi-PF setup if firmware is loaded. */
- if (pf->multi_pf.en && nfp_nsp_fw_loaded(nsp))
+ if (nfp_skip_fw_load(pf, nsp))
return 1;
fw = nfp_net_fw_find(pdev, pf);
--
2.34.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH net-next 05/12] nfp: introduce keepalive mechanism for multi-PF setup
2023-07-24 9:48 [PATCH net-next 00/12] nfp: add support for multi-pf configuration Louis Peens
` (3 preceding siblings ...)
2023-07-24 9:48 ` [PATCH net-next 04/12] nfp: don't skip firmware loading when it's pxe firmware in running Louis Peens
@ 2023-07-24 9:48 ` Louis Peens
2023-07-29 20:20 ` kernel test robot
2023-07-24 9:48 ` [PATCH net-next 06/12] nfp: avoid reclaiming resource mutex by mistake Louis Peens
` (7 subsequent siblings)
12 siblings, 1 reply; 22+ messages in thread
From: Louis Peens @ 2023-07-24 9:48 UTC (permalink / raw)
To: David Miller, Jakub Kicinski, Paolo Abeni
Cc: Simon Horman, Yinjun Zhang, Tianyu Yuan, netdev, oss-drivers
From: Yinjun Zhang <yinjun.zhang@corigine.com>
In multi-PF setup, management firmware is in charge of application
firmware unloading instead of driver by keepalive mechanism.
A new NSP resource area is allocated for keepalive use with name
"nfp.beat". Driver sets the magic number when keepalive is needed
and periodically updates the PF's corresponding qword in "nfp.beat".
Management firmware checks these PFs' qwords to learn whether and
which PFs are alive, and will unload the application firmware if
no PF is running. This only works when magic number is correct.
Signed-off-by: Yinjun Zhang <yinjun.zhang@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
drivers/net/ethernet/netronome/nfp/nfp_main.c | 88 +++++++++++++++++++
drivers/net/ethernet/netronome/nfp/nfp_main.h | 8 ++
.../net/ethernet/netronome/nfp/nfpcore/nfp.h | 4 +
3 files changed, 100 insertions(+)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.c b/drivers/net/ethernet/netronome/nfp/nfp_main.c
index 778f21dfbbd5..489113c53596 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.c
@@ -469,6 +469,77 @@ nfp_get_fw_policy_value(struct pci_dev *pdev, struct nfp_nsp *nsp,
return err;
}
+static void
+nfp_nsp_beat_timer(struct timer_list *t)
+{
+ struct nfp_pf *pf = from_timer(pf, t, multi_pf.beat_timer);
+ u8 __iomem *addr;
+
+ /* Each PF has corresponding qword to beat:
+ * offset | usage
+ * 0 | magic number
+ * 8 | beat qword of pf0
+ * 16 | beat qword of pf1
+ */
+ addr = pf->multi_pf.beat_addr + ((pf->multi_pf.id + 1) << 3);
+ writeq(jiffies, addr);
+ /* Beat once per second. */
+ mod_timer(&pf->multi_pf.beat_timer, jiffies + HZ);
+}
+
+/**
+ * nfp_nsp_keepalive_start() - Start keepalive mechanism if needed
+ * @pf: NFP PF Device structure
+ *
+ * Return 0 if no error, errno otherwise
+ */
+static int
+nfp_nsp_keepalive_start(struct nfp_pf *pf)
+{
+ struct nfp_resource *res;
+ u8 __iomem *base;
+ int err = 0;
+ u64 addr;
+ u32 cpp;
+
+ if (!pf->multi_pf.en)
+ return 0;
+
+ res = nfp_resource_acquire(pf->cpp, NFP_KEEPALIVE);
+ if (IS_ERR(res))
+ return PTR_ERR(res);
+
+ cpp = nfp_resource_cpp_id(res);
+ addr = nfp_resource_address(res);
+
+ /* Allocate a fixed area for keepalive. */
+ base = nfp_cpp_map_area(pf->cpp, "keepalive", cpp, addr,
+ nfp_resource_size(res), &pf->multi_pf.beat_area);
+ if (IS_ERR(base)) {
+ nfp_err(pf->cpp, "Failed to map area for keepalive\n");
+ err = PTR_ERR(base);
+ goto res_release;
+ }
+
+ pf->multi_pf.beat_addr = base;
+ timer_setup(&pf->multi_pf.beat_timer, nfp_nsp_beat_timer, 0);
+ mod_timer(&pf->multi_pf.beat_timer, jiffies);
+
+res_release:
+ nfp_resource_release(res);
+ return err;
+}
+
+static void
+nfp_nsp_keepalive_stop(struct nfp_pf *pf)
+{
+ if (!pf->multi_pf.beat_area)
+ return;
+
+ del_timer_sync(&pf->multi_pf.beat_timer);
+ nfp_cpp_area_release_free(pf->multi_pf.beat_area);
+}
+
static bool
nfp_skip_fw_load(struct nfp_pf *pf, struct nfp_nsp *nsp)
{
@@ -550,6 +621,10 @@ nfp_fw_load(struct pci_dev *pdev, struct nfp_pf *pf, struct nfp_nsp *nsp)
if (err)
return err;
+ err = nfp_nsp_keepalive_start(pf);
+ if (err)
+ return err;
+
if (nfp_skip_fw_load(pf, nsp))
return 1;
@@ -620,6 +695,16 @@ nfp_fw_load(struct pci_dev *pdev, struct nfp_pf *pf, struct nfp_nsp *nsp)
if (fw_loaded && ifcs == 1 && !pf->multi_pf.en)
pf->unload_fw_on_remove = true;
+ /* Only setting magic number when fw is freshly loaded here. NSP
+ * won't unload fw when heartbeat stops if the magic number is not
+ * correct. It's used when firmware is preloaded and shouldn't be
+ * unloaded when driver exits.
+ */
+ if (err < 0)
+ nfp_nsp_keepalive_stop(pf);
+ else if (fw_loaded && pf->multi_pf.en)
+ writeq(NFP_KEEPALIVE_MAGIC, pf->multi_pf.beat_addr);
+
return err < 0 ? err : fw_loaded;
}
@@ -666,6 +751,7 @@ static int nfp_nsp_init(struct pci_dev *pdev, struct nfp_pf *pf)
}
pf->multi_pf.en = pdev->multifunction;
+ pf->multi_pf.id = PCI_FUNC(pdev->devfn);
dev_info(&pdev->dev, "%s-PF detected\n", pf->multi_pf.en ? "Multi" : "Single");
err = nfp_nsp_wait(nsp);
@@ -913,6 +999,7 @@ static int nfp_pci_probe(struct pci_dev *pdev,
err_net_remove:
nfp_net_pci_remove(pf);
err_fw_unload:
+ nfp_nsp_keepalive_stop(pf);
kfree(pf->rtbl);
nfp_mip_close(pf->mip);
if (pf->unload_fw_on_remove)
@@ -952,6 +1039,7 @@ static void __nfp_pci_shutdown(struct pci_dev *pdev, bool unload_fw)
nfp_net_pci_remove(pf);
vfree(pf->dumpspec);
+ nfp_nsp_keepalive_stop(pf);
kfree(pf->rtbl);
nfp_mip_close(pf->mip);
if (unload_fw && pf->unload_fw_on_remove)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.h b/drivers/net/ethernet/netronome/nfp/nfp_main.h
index 72ea3b83d313..c58849a332b0 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.h
@@ -86,6 +86,10 @@ struct nfp_dumpspec {
* @num_shared_bufs: Number of elements in @shared_bufs
* @multi_pf: Used in multi-PF setup
* @multi_pf.en: True if it's a NIC with multiple PFs
+ * @multi_pf.id: PF index
+ * @multi_pf.beat_timer:Timer for beat to keepalive
+ * @multi_pf.beat_area: Pointer to CPP area for beat to keepalive
+ * @multi_pf.beat_addr: Pointer to mapped beat address used for keepalive
*
* Fields which may change after proble are protected by devlink instance lock.
*/
@@ -146,6 +150,10 @@ struct nfp_pf {
struct {
bool en;
+ u8 id;
+ struct timer_list beat_timer;
+ struct nfp_cpp_area *beat_area;
+ u8 __iomem *beat_addr;
} multi_pf;
};
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp.h b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp.h
index db94b0bddc92..89a131cffc48 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp.h
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp.h
@@ -64,6 +64,10 @@ int nfp_nsp_read_sensors(struct nfp_nsp *state, unsigned int sensor_mask,
/* MAC Statistics Accumulator */
#define NFP_RESOURCE_MAC_STATISTICS "mac.stat"
+/* Keepalive */
+#define NFP_KEEPALIVE "nfp.beat"
+#define NFP_KEEPALIVE_MAGIC 0x6e66702e62656174ULL /* ASCII of "nfp.beat" */
+
int nfp_resource_table_init(struct nfp_cpp *cpp);
struct nfp_resource *
--
2.34.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* Re: [PATCH net-next 05/12] nfp: introduce keepalive mechanism for multi-PF setup
2023-07-24 9:48 ` [PATCH net-next 05/12] nfp: introduce keepalive mechanism for multi-PF setup Louis Peens
@ 2023-07-29 20:20 ` kernel test robot
2023-07-30 4:51 ` Yinjun Zhang
0 siblings, 1 reply; 22+ messages in thread
From: kernel test robot @ 2023-07-29 20:20 UTC (permalink / raw)
To: Louis Peens, David Miller, Jakub Kicinski, Paolo Abeni
Cc: oe-kbuild-all, netdev, Simon Horman, Yinjun Zhang, Tianyu Yuan,
oss-drivers
Hi Louis,
kernel test robot noticed the following build warnings:
[auto build test WARNING on net-next/main]
url: https://github.com/intel-lab-lkp/linux/commits/Louis-Peens/nsp-generate-nsp-command-with-variable-nsp-major-version/20230724-180015
base: net-next/main
patch link: https://lore.kernel.org/r/20230724094821.14295-6-louis.peens%40corigine.com
patch subject: [PATCH net-next 05/12] nfp: introduce keepalive mechanism for multi-PF setup
config: openrisc-randconfig-r081-20230730 (https://download.01.org/0day-ci/archive/20230730/202307300422.oPy5E1hB-lkp@intel.com/config)
compiler: or1k-linux-gcc (GCC) 12.3.0
reproduce: (https://download.01.org/0day-ci/archive/20230730/202307300422.oPy5E1hB-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202307300422.oPy5E1hB-lkp@intel.com/
sparse warnings: (new ones prefixed by >>)
drivers/net/ethernet/netronome/nfp/nfp_main.c: note: in included file (through drivers/net/ethernet/netronome/nfp/nfp_net.h):
>> include/linux/io-64-nonatomic-hi-lo.h:22:16: sparse: sparse: cast truncates bits from constant value (6e66702e62656174 becomes 62656174)
vim +22 include/linux/io-64-nonatomic-hi-lo.h
797a796a13df6b include/asm-generic/io-64-nonatomic-hi-lo.h Hitoshi Mitake 2012-02-07 18
3a044178cccfeb include/asm-generic/io-64-nonatomic-hi-lo.h Jason Baron 2014-07-04 19 static inline void hi_lo_writeq(__u64 val, volatile void __iomem *addr)
797a796a13df6b include/asm-generic/io-64-nonatomic-hi-lo.h Hitoshi Mitake 2012-02-07 20 {
797a796a13df6b include/asm-generic/io-64-nonatomic-hi-lo.h Hitoshi Mitake 2012-02-07 21 writel(val >> 32, addr + 4);
797a796a13df6b include/asm-generic/io-64-nonatomic-hi-lo.h Hitoshi Mitake 2012-02-07 @22 writel(val, addr);
797a796a13df6b include/asm-generic/io-64-nonatomic-hi-lo.h Hitoshi Mitake 2012-02-07 23 }
3a044178cccfeb include/asm-generic/io-64-nonatomic-hi-lo.h Jason Baron 2014-07-04 24
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [PATCH net-next 05/12] nfp: introduce keepalive mechanism for multi-PF setup
2023-07-29 20:20 ` kernel test robot
@ 2023-07-30 4:51 ` Yinjun Zhang
2023-08-07 2:10 ` Yinjun Zhang
0 siblings, 1 reply; 22+ messages in thread
From: Yinjun Zhang @ 2023-07-30 4:51 UTC (permalink / raw)
To: lkp
Cc: davem, kuba, louis.peens, netdev, oe-kbuild-all, oss-drivers,
pabeni, simon.horman, tianyu.yuan, yinjun.zhang
On Sun, 30 Jul 2023 04:20:57 +0800, kernel test robot wrote:
> Hi Louis,
>
> kernel test robot noticed the following build warnings:
>
> [auto build test WARNING on net-next/main]
>
> url: https://github.com/intel-lab-lkp/linux/commits/Louis-Peens/nsp-generate-nsp-command-with-variable-nsp-major-version/20230724-180015
> base: net-next/main
> patch link: https://lore.kernel.org/r/20230724094821.14295-6-louis.peens%40corigine.com
> patch subject: [PATCH net-next 05/12] nfp: introduce keepalive mechanism for multi-PF setup
> config: openrisc-randconfig-r081-20230730 (https://download.01.org/0day-ci/archive/20230730/202307300422.oPy5E1hB-lkp@intel.com/config)
> compiler: or1k-linux-gcc (GCC) 12.3.0
> reproduce: (https://download.01.org/0day-ci/archive/20230730/202307300422.oPy5E1hB-lkp@intel.com/reproduce)
>
> If you fix the issue in a separate patch/commit (i.e. not just a new version of
> the same patch/commit), kindly add following tags
> | Reported-by: kernel test robot <lkp@intel.com>
> | Closes: https://lore.kernel.org/oe-kbuild-all/202307300422.oPy5E1hB-lkp@intel.com/
>
> sparse warnings: (new ones prefixed by >>)
> drivers/net/ethernet/netronome/nfp/nfp_main.c: note: in included file (through drivers/net/ethernet/netronome/nfp/nfp_net.h):
> >> include/linux/io-64-nonatomic-hi-lo.h:22:16: sparse: sparse: cast truncates bits from constant value (6e66702e62656174 becomes 62656174)
I think it's more like a callee's problem instead of the caller's.
`writeq` is supposed to be able to be fed with a constant. WDYT?
>
> vim +22 include/linux/io-64-nonatomic-hi-lo.h
>
> 797a796a13df6b include/asm-generic/io-64-nonatomic-hi-lo.h Hitoshi Mitake 2012-02-07 18
> 3a044178cccfeb include/asm-generic/io-64-nonatomic-hi-lo.h Jason Baron 2014-07-04 19 static inline void hi_lo_writeq(__u64 val, volatile void __iomem *addr)
> 797a796a13df6b include/asm-generic/io-64-nonatomic-hi-lo.h Hitoshi Mitake 2012-02-07 20 {
> 797a796a13df6b include/asm-generic/io-64-nonatomic-hi-lo.h Hitoshi Mitake 2012-02-07 21 writel(val >> 32, addr + 4);
> 797a796a13df6b include/asm-generic/io-64-nonatomic-hi-lo.h Hitoshi Mitake 2012-02-07 @22 writel(val, addr);
> 797a796a13df6b include/asm-generic/io-64-nonatomic-hi-lo.h Hitoshi Mitake 2012-02-07 23 }
> 3a044178cccfeb include/asm-generic/io-64-nonatomic-hi-lo.h Jason Baron 2014-07-04 24
>
> --
> 0-DAY CI Kernel Test Service
> https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 22+ messages in thread* RE: [PATCH net-next 05/12] nfp: introduce keepalive mechanism for multi-PF setup
2023-07-30 4:51 ` Yinjun Zhang
@ 2023-08-07 2:10 ` Yinjun Zhang
0 siblings, 0 replies; 22+ messages in thread
From: Yinjun Zhang @ 2023-08-07 2:10 UTC (permalink / raw)
To: lkp@intel.com
Cc: davem@davemloft.net, kuba@kernel.org, Louis Peens,
netdev@vger.kernel.org, oe-kbuild-all@lists.linux.dev,
oss-drivers, pabeni@redhat.com, Simon Horman, Tianyu Yuan
On Sunday, July 30, 2023 12:52 PM, Yinjun Zhang wrote:
> On Sun, 30 Jul 2023 04:20:57 +0800, kernel test robot wrote:
> > Hi Louis,
> >
> > kernel test robot noticed the following build warnings:
> >
> > [auto build test WARNING on net-next/main]
> >
> > url: https://github.com/intel-lab-lkp/linux/commits/Louis-Peens/nsp-
> generate-nsp-command-with-variable-nsp-major-version/20230724-180015
> > base: net-next/main
> > patch link: https://lore.kernel.org/r/20230724094821.14295-6-
> louis.peens%40corigine.com
> > patch subject: [PATCH net-next 05/12] nfp: introduce keepalive mechanism
> for multi-PF setup
> > config: openrisc-randconfig-r081-20230730 (https://download.01.org/0day-
> ci/archive/20230730/202307300422.oPy5E1hB-lkp@intel.com/config)
> > compiler: or1k-linux-gcc (GCC) 12.3.0
> > reproduce: (https://download.01.org/0day-
> ci/archive/20230730/202307300422.oPy5E1hB-lkp@intel.com/reproduce)
> >
> > If you fix the issue in a separate patch/commit (i.e. not just a new version of
> > the same patch/commit), kindly add following tags
> > | Reported-by: kernel test robot <lkp@intel.com>
> > | Closes: https://lore.kernel.org/oe-kbuild-all/202307300422.oPy5E1hB-
> lkp@intel.com/
> >
> > sparse warnings: (new ones prefixed by >>)
> > drivers/net/ethernet/netronome/nfp/nfp_main.c: note: in included file
> (through drivers/net/ethernet/netronome/nfp/nfp_net.h):
> > >> include/linux/io-64-nonatomic-hi-lo.h:22:16: sparse: sparse: cast
> truncates bits from constant value (6e66702e62656174 becomes 62656174)
>
> I think it's more like a callee's problem instead of the caller's.
> `writeq` is supposed to be able to be fed with a constant. WDYT?
There's no response for one week. To compromise, I'm going to change
the caller in driver side. Let me know if anybody has some other comments.
>
> >
> > vim +22 include/linux/io-64-nonatomic-hi-lo.h
> >
> > 797a796a13df6b include/asm-generic/io-64-nonatomic-hi-lo.h Hitoshi
> Mitake 2012-02-07 18
> > 3a044178cccfeb include/asm-generic/io-64-nonatomic-hi-lo.h Jason Baron
> 2014-07-04 19 static inline void hi_lo_writeq(__u64 val, volatile void
> __iomem *addr)
> > 797a796a13df6b include/asm-generic/io-64-nonatomic-hi-lo.h Hitoshi
> Mitake 2012-02-07 20 {
> > 797a796a13df6b include/asm-generic/io-64-nonatomic-hi-lo.h Hitoshi
> Mitake 2012-02-07 21 writel(val >> 32, addr + 4);
> > 797a796a13df6b include/asm-generic/io-64-nonatomic-hi-lo.h Hitoshi
> Mitake 2012-02-07 @22 writel(val, addr);
> > 797a796a13df6b include/asm-generic/io-64-nonatomic-hi-lo.h Hitoshi
> Mitake 2012-02-07 23 }
> > 3a044178cccfeb include/asm-generic/io-64-nonatomic-hi-lo.h Jason Baron
> 2014-07-04 24
> >
> > --
> > 0-DAY CI Kernel Test Service
> > https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH net-next 06/12] nfp: avoid reclaiming resource mutex by mistake
2023-07-24 9:48 [PATCH net-next 00/12] nfp: add support for multi-pf configuration Louis Peens
` (4 preceding siblings ...)
2023-07-24 9:48 ` [PATCH net-next 05/12] nfp: introduce keepalive mechanism for multi-PF setup Louis Peens
@ 2023-07-24 9:48 ` Louis Peens
2023-07-24 9:48 ` [PATCH net-next 07/12] nfp: redefine PF id used to format symbols Louis Peens
` (6 subsequent siblings)
12 siblings, 0 replies; 22+ messages in thread
From: Louis Peens @ 2023-07-24 9:48 UTC (permalink / raw)
To: David Miller, Jakub Kicinski, Paolo Abeni
Cc: Simon Horman, Yinjun Zhang, Tianyu Yuan, netdev, oss-drivers
From: Yinjun Zhang <yinjun.zhang@corigine.com>
Multiple PFs of the same controller use the same interface
id. So we shouldn't unconditionally reclaim resource mutex
when probing, because the mutex may be held by another PF
from the same controller.
Now give it some time to release the mutex, and reclaim it
if timeout.
Signed-off-by: Yinjun Zhang <yinjun.zhang@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
.../netronome/nfp/nfpcore/nfp_mutex.c | 21 +++++++++++++------
1 file changed, 15 insertions(+), 6 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_mutex.c b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_mutex.c
index 7bc17b94ac60..1b9170d9da77 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_mutex.c
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_mutex.c
@@ -343,6 +343,7 @@ int nfp_cpp_mutex_reclaim(struct nfp_cpp *cpp, int target,
{
const u32 mur = NFP_CPP_ID(target, 3, 0); /* atomic_read */
const u32 muw = NFP_CPP_ID(target, 4, 0); /* atomic_write */
+ unsigned long timeout = jiffies + 2 * HZ;
u16 interface = nfp_cpp_interface(cpp);
int err;
u32 tmp;
@@ -351,13 +352,21 @@ int nfp_cpp_mutex_reclaim(struct nfp_cpp *cpp, int target,
if (err)
return err;
- /* Check lock */
- err = nfp_cpp_readl(cpp, mur, address, &tmp);
- if (err < 0)
- return err;
+ /* Check lock. Note that PFs from the same controller use same interface ID.
+ * So considering that the lock may be held by other PFs from the same
+ * controller, we give it some time to release the lock, and only reclaim it
+ * if timeout.
+ */
+ while (time_is_after_jiffies(timeout)) {
+ err = nfp_cpp_readl(cpp, mur, address, &tmp);
+ if (err < 0)
+ return err;
- if (nfp_mutex_is_unlocked(tmp) || nfp_mutex_owner(tmp) != interface)
- return 0;
+ if (nfp_mutex_is_unlocked(tmp) || nfp_mutex_owner(tmp) != interface)
+ return 0;
+
+ msleep_interruptible(10);
+ }
/* Bust the lock */
err = nfp_cpp_writel(cpp, muw, address, nfp_mutex_unlocked(interface));
--
2.34.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH net-next 07/12] nfp: redefine PF id used to format symbols
2023-07-24 9:48 [PATCH net-next 00/12] nfp: add support for multi-pf configuration Louis Peens
` (5 preceding siblings ...)
2023-07-24 9:48 ` [PATCH net-next 06/12] nfp: avoid reclaiming resource mutex by mistake Louis Peens
@ 2023-07-24 9:48 ` Louis Peens
2023-07-24 9:48 ` [PATCH net-next 08/12] nfp: apply one port per PF for multi-PF setup Louis Peens
` (5 subsequent siblings)
12 siblings, 0 replies; 22+ messages in thread
From: Louis Peens @ 2023-07-24 9:48 UTC (permalink / raw)
To: David Miller, Jakub Kicinski, Paolo Abeni
Cc: Simon Horman, Yinjun Zhang, Tianyu Yuan, netdev, oss-drivers
From: Yinjun Zhang <yinjun.zhang@corigine.com>
Taking account that NFP3800 supports 4 physical functions per
controller, now recalculate PF id to be used to format symbols
to communicate with application firmware.
Signed-off-by: Yinjun Zhang <yinjun.zhang@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
drivers/net/ethernet/netronome/nfp/abm/ctrl.c | 2 +-
.../net/ethernet/netronome/nfp/flower/main.c | 2 +-
drivers/net/ethernet/netronome/nfp/nfp_main.c | 18 +++++++++++-------
drivers/net/ethernet/netronome/nfp/nfp_main.h | 2 ++
.../ethernet/netronome/nfp/nfpcore/nfp_dev.c | 2 ++
.../ethernet/netronome/nfp/nfpcore/nfp_dev.h | 1 +
6 files changed, 18 insertions(+), 9 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/abm/ctrl.c b/drivers/net/ethernet/netronome/nfp/abm/ctrl.c
index 69e84ff7f2e5..41d18df97c85 100644
--- a/drivers/net/ethernet/netronome/nfp/abm/ctrl.c
+++ b/drivers/net/ethernet/netronome/nfp/abm/ctrl.c
@@ -362,7 +362,7 @@ int nfp_abm_ctrl_find_addrs(struct nfp_abm *abm)
const struct nfp_rtsym *sym;
int res;
- abm->pf_id = nfp_cppcore_pcie_unit(pf->cpp);
+ abm->pf_id = nfp_get_pf_id(pf);
/* Check if Qdisc offloads are supported */
res = nfp_pf_rtsym_read_optional(pf, NFP_RED_SUPPORT_SYM_NAME, 1);
diff --git a/drivers/net/ethernet/netronome/nfp/flower/main.c b/drivers/net/ethernet/netronome/nfp/flower/main.c
index 83eaa5ae3cd4..565987f0a595 100644
--- a/drivers/net/ethernet/netronome/nfp/flower/main.c
+++ b/drivers/net/ethernet/netronome/nfp/flower/main.c
@@ -378,10 +378,10 @@ nfp_flower_spawn_vnic_reprs(struct nfp_app *app,
enum nfp_flower_cmsg_port_vnic_type vnic_type,
enum nfp_repr_type repr_type, unsigned int cnt)
{
- u8 nfp_pcie = nfp_cppcore_pcie_unit(app->pf->cpp);
struct nfp_flower_priv *priv = app->priv;
atomic_t *replies = &priv->reify_replies;
struct nfp_flower_repr_priv *repr_priv;
+ u8 nfp_pcie = nfp_get_pf_id(app->pf);
enum nfp_port_type port_type;
struct nfp_repr *nfp_repr;
struct nfp_reprs *reprs;
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.c b/drivers/net/ethernet/netronome/nfp/nfp_main.c
index 489113c53596..74767729e542 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.c
@@ -69,6 +69,13 @@ static const struct pci_device_id nfp_pci_device_ids[] = {
};
MODULE_DEVICE_TABLE(pci, nfp_pci_device_ids);
+u8 nfp_get_pf_id(struct nfp_pf *pf)
+{
+ return nfp_cppcore_pcie_unit(pf->cpp) *
+ pf->dev_info->pf_num_per_unit +
+ pf->multi_pf.id;
+}
+
int nfp_pf_rtsym_read_optional(struct nfp_pf *pf, const char *format,
unsigned int default_val)
{
@@ -76,7 +83,7 @@ int nfp_pf_rtsym_read_optional(struct nfp_pf *pf, const char *format,
int err = 0;
u64 val;
- snprintf(name, sizeof(name), format, nfp_cppcore_pcie_unit(pf->cpp));
+ snprintf(name, sizeof(name), format, nfp_get_pf_id(pf));
val = nfp_rtsym_read_le(pf->rtbl, name, &err);
if (err) {
@@ -95,8 +102,7 @@ nfp_pf_map_rtsym(struct nfp_pf *pf, const char *name, const char *sym_fmt,
{
char pf_symbol[256];
- snprintf(pf_symbol, sizeof(pf_symbol), sym_fmt,
- nfp_cppcore_pcie_unit(pf->cpp));
+ snprintf(pf_symbol, sizeof(pf_symbol), sym_fmt, nfp_get_pf_id(pf));
return nfp_rtsym_map(pf->rtbl, pf_symbol, name, min_size, area);
}
@@ -803,10 +809,8 @@ static void nfp_fw_unload(struct nfp_pf *pf)
static int nfp_pf_find_rtsyms(struct nfp_pf *pf)
{
+ unsigned int pf_id = nfp_get_pf_id(pf);
char pf_symbol[256];
- unsigned int pf_id;
-
- pf_id = nfp_cppcore_pcie_unit(pf->cpp);
/* Optional per-PCI PF mailbox */
snprintf(pf_symbol, sizeof(pf_symbol), NFP_MBOX_SYM_NAME, pf_id);
@@ -832,7 +836,7 @@ static u64 nfp_net_pf_get_app_cap(struct nfp_pf *pf)
int err = 0;
u64 val;
- snprintf(name, sizeof(name), "_pf%u_net_app_cap", nfp_cppcore_pcie_unit(pf->cpp));
+ snprintf(name, sizeof(name), "_pf%u_net_app_cap", nfp_get_pf_id(pf));
val = nfp_rtsym_read_le(pf->rtbl, name, &err);
if (err) {
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.h b/drivers/net/ethernet/netronome/nfp/nfp_main.h
index c58849a332b0..7f76c718fef8 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.h
@@ -208,4 +208,6 @@ void nfp_devlink_params_unregister(struct nfp_pf *pf);
unsigned int nfp_net_lr2speed(unsigned int linkrate);
unsigned int nfp_net_speed2lr(unsigned int speed);
+
+u8 nfp_get_pf_id(struct nfp_pf *pf);
#endif /* NFP_MAIN_H */
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_dev.c b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_dev.c
index 0725b51c2a95..8a7c5de0de77 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_dev.c
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_dev.c
@@ -19,6 +19,7 @@ const struct nfp_dev_info nfp_dev_info[NFP_DEV_CNT] = {
.pcie_cfg_expbar_offset = 0x0a00,
.pcie_expl_offset = 0xd000,
.qc_area_sz = 0x100000,
+ .pf_num_per_unit = 4,
},
[NFP_DEV_NFP3800_VF] = {
.dma_mask = DMA_BIT_MASK(48),
@@ -38,6 +39,7 @@ const struct nfp_dev_info nfp_dev_info[NFP_DEV_CNT] = {
.pcie_cfg_expbar_offset = 0x0400,
.pcie_expl_offset = 0x1000,
.qc_area_sz = 0x80000,
+ .pf_num_per_unit = 1,
},
[NFP_DEV_NFP6000_VF] = {
.dma_mask = DMA_BIT_MASK(40),
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_dev.h b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_dev.h
index e4d38178de0f..d948c9c4a09a 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_dev.h
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_dev.h
@@ -35,6 +35,7 @@ struct nfp_dev_info {
u32 pcie_cfg_expbar_offset;
u32 pcie_expl_offset;
u32 qc_area_sz;
+ u8 pf_num_per_unit;
};
extern const struct nfp_dev_info nfp_dev_info[NFP_DEV_CNT];
--
2.34.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH net-next 08/12] nfp: apply one port per PF for multi-PF setup
2023-07-24 9:48 [PATCH net-next 00/12] nfp: add support for multi-pf configuration Louis Peens
` (6 preceding siblings ...)
2023-07-24 9:48 ` [PATCH net-next 07/12] nfp: redefine PF id used to format symbols Louis Peens
@ 2023-07-24 9:48 ` Louis Peens
2023-07-24 9:48 ` [PATCH net-next 09/12] nfp: enable multi-PF in application firmware if supported Louis Peens
` (4 subsequent siblings)
12 siblings, 0 replies; 22+ messages in thread
From: Louis Peens @ 2023-07-24 9:48 UTC (permalink / raw)
To: David Miller, Jakub Kicinski, Paolo Abeni
Cc: Simon Horman, Yinjun Zhang, Tianyu Yuan, netdev, oss-drivers
From: Tianyu Yuan <tianyu.yuan@corigine.com>
Only one port per PF is allowed in multi-PF setup. While eth_table
still carries the total port info, each PF need bind itself with
correct port according to PF id.
Signed-off-by: Tianyu Yuan <tianyu.yuan@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
drivers/net/ethernet/netronome/nfp/abm/main.c | 2 +-
drivers/net/ethernet/netronome/nfp/bpf/main.c | 2 +-
.../net/ethernet/netronome/nfp/flower/main.c | 17 ++++++++++-------
drivers/net/ethernet/netronome/nfp/nfp_main.h | 6 ++++++
.../net/ethernet/netronome/nfp/nfp_net_main.c | 11 ++++++++---
drivers/net/ethernet/netronome/nfp/nfp_port.c | 4 +++-
drivers/net/ethernet/netronome/nfp/nic/main.c | 3 ++-
7 files changed, 31 insertions(+), 14 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/abm/main.c b/drivers/net/ethernet/netronome/nfp/abm/main.c
index 5d3df28c648f..d4acaa15629d 100644
--- a/drivers/net/ethernet/netronome/nfp/abm/main.c
+++ b/drivers/net/ethernet/netronome/nfp/abm/main.c
@@ -451,7 +451,7 @@ static int nfp_abm_init(struct nfp_app *app)
nfp_err(pf->cpp, "ABM NIC requires ETH table\n");
return -EINVAL;
}
- if (pf->max_data_vnics != pf->eth_tbl->count) {
+ if (pf->max_data_vnics != pf->eth_tbl->count && !pf->multi_pf.en) {
nfp_err(pf->cpp, "ETH entries don't match vNICs (%d vs %d)\n",
pf->max_data_vnics, pf->eth_tbl->count);
return -EINVAL;
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/main.c b/drivers/net/ethernet/netronome/nfp/bpf/main.c
index f469950c7265..3d928dfba114 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/main.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/main.c
@@ -70,7 +70,7 @@ nfp_bpf_vnic_alloc(struct nfp_app *app, struct nfp_net *nn, unsigned int id)
nfp_err(pf->cpp, "No ETH table\n");
return -EINVAL;
}
- if (pf->max_data_vnics != pf->eth_tbl->count) {
+ if (pf->max_data_vnics != pf->eth_tbl->count && !pf->multi_pf.en) {
nfp_err(pf->cpp, "ETH entries don't match vNICs (%d vs %d)\n",
pf->max_data_vnics, pf->eth_tbl->count);
return -EINVAL;
diff --git a/drivers/net/ethernet/netronome/nfp/flower/main.c b/drivers/net/ethernet/netronome/nfp/flower/main.c
index 565987f0a595..2e79b6d981de 100644
--- a/drivers/net/ethernet/netronome/nfp/flower/main.c
+++ b/drivers/net/ethernet/netronome/nfp/flower/main.c
@@ -428,10 +428,10 @@ nfp_flower_spawn_vnic_reprs(struct nfp_app *app,
goto err_reprs_clean;
}
if (repr_type == NFP_REPR_TYPE_PF) {
- port->pf_id = i;
+ port->pf_id = nfp_net_get_id(app->pf, i);
port->vnic = priv->nn->dp.ctrl_bar;
} else {
- port->pf_id = 0;
+ port->pf_id = nfp_net_get_id(app->pf, 0);
port->vf_id = i;
port->vnic =
app->pf->vf_cfg_mem + i * NFP_NET_CFG_BAR_SZ;
@@ -496,28 +496,31 @@ nfp_flower_spawn_phy_reprs(struct nfp_app *app, struct nfp_flower_priv *priv)
struct nfp_eth_table *eth_tbl = app->pf->eth_tbl;
atomic_t *replies = &priv->reify_replies;
struct nfp_flower_repr_priv *repr_priv;
+ int err, reify_cnt, phy_reprs_num;
struct nfp_repr *nfp_repr;
struct sk_buff *ctrl_skb;
struct nfp_reprs *reprs;
- int err, reify_cnt;
unsigned int i;
ctrl_skb = nfp_flower_cmsg_mac_repr_start(app, eth_tbl->count);
if (!ctrl_skb)
return -ENOMEM;
+ phy_reprs_num = app->pf->multi_pf.en ? app->pf->max_data_vnics : eth_tbl->count;
reprs = nfp_reprs_alloc(eth_tbl->max_index + 1);
if (!reprs) {
err = -ENOMEM;
goto err_free_ctrl_skb;
}
- for (i = 0; i < eth_tbl->count; i++) {
- unsigned int phys_port = eth_tbl->ports[i].index;
+ for (i = 0; i < phy_reprs_num; i++) {
+ int idx = nfp_net_get_id(app->pf, i);
struct net_device *repr;
+ unsigned int phys_port;
struct nfp_port *port;
u32 cmsg_port_id;
+ phys_port = eth_tbl->ports[idx].index;
repr = nfp_repr_alloc(app);
if (!repr) {
err = -ENOMEM;
@@ -542,7 +545,7 @@ nfp_flower_spawn_phy_reprs(struct nfp_app *app, struct nfp_flower_priv *priv)
nfp_repr_free(repr);
goto err_reprs_clean;
}
- err = nfp_port_init_phy_port(app->pf, app, port, i);
+ err = nfp_port_init_phy_port(app->pf, app, port, idx);
if (err) {
kfree(repr_priv);
nfp_port_free(port);
@@ -609,7 +612,7 @@ nfp_flower_spawn_phy_reprs(struct nfp_app *app, struct nfp_flower_priv *priv)
static int nfp_flower_vnic_alloc(struct nfp_app *app, struct nfp_net *nn,
unsigned int id)
{
- if (id > 0) {
+ if (id > 0 && !app->pf->multi_pf.en) {
nfp_warn(app->cpp, "FlowerNIC doesn't support more than one data vNIC\n");
goto err_invalid_port;
}
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.h b/drivers/net/ethernet/netronome/nfp/nfp_main.h
index 7f76c718fef8..4f6763ca1c92 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.h
@@ -210,4 +210,10 @@ unsigned int nfp_net_lr2speed(unsigned int linkrate);
unsigned int nfp_net_speed2lr(unsigned int speed);
u8 nfp_get_pf_id(struct nfp_pf *pf);
+
+static inline unsigned int nfp_net_get_id(const struct nfp_pf *pf, unsigned int id)
+{
+ return pf->multi_pf.en ? pf->multi_pf.id : id;
+}
+
#endif /* NFP_MAIN_H */
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
index cbe4972ba104..98e155d79eb8 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
@@ -141,7 +141,7 @@ nfp_net_pf_init_vnic(struct nfp_pf *pf, struct nfp_net *nn, unsigned int id)
{
int err;
- nn->id = id;
+ nn->id = nfp_net_get_id(pf, id);
if (nn->port) {
err = nfp_devlink_port_register(pf->app, nn->port);
@@ -183,8 +183,8 @@ nfp_net_pf_alloc_vnics(struct nfp_pf *pf, void __iomem *ctrl_bar,
int err;
for (i = 0; i < pf->max_data_vnics; i++) {
- nn = nfp_net_pf_alloc_vnic(pf, true, ctrl_bar, qc_bar,
- stride, i);
+ nn = nfp_net_pf_alloc_vnic(pf, true, ctrl_bar, qc_bar, stride,
+ nfp_net_get_id(pf, i));
if (IS_ERR(nn)) {
err = PTR_ERR(nn);
goto err_free_prev;
@@ -707,6 +707,11 @@ int nfp_net_pci_probe(struct nfp_pf *pf)
if ((int)pf->max_data_vnics < 0)
return pf->max_data_vnics;
+ if (pf->multi_pf.en && pf->max_data_vnics != 1) {
+ nfp_err(pf->cpp, "Only one data_vnic per PF is supported in multiple PF setup.\n");
+ return -EINVAL;
+ }
+
err = nfp_net_pci_map_mem(pf);
if (err)
return err;
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_port.c b/drivers/net/ethernet/netronome/nfp/nfp_port.c
index 54640bcb70fb..c1612a464b5d 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_port.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_port.c
@@ -189,7 +189,9 @@ int nfp_port_init_phy_port(struct nfp_pf *pf, struct nfp_app *app,
port->eth_port = &pf->eth_tbl->ports[id];
port->eth_id = pf->eth_tbl->ports[id].index;
- port->netdev->dev_port = id;
+ if (!pf->multi_pf.en)
+ port->netdev->dev_port = id;
+
if (pf->mac_stats_mem)
port->eth_stats =
pf->mac_stats_mem + port->eth_id * NFP_MAC_STATS_SIZE;
diff --git a/drivers/net/ethernet/netronome/nfp/nic/main.c b/drivers/net/ethernet/netronome/nfp/nic/main.c
index 9dd5afe37f6e..e7a2d01bcbff 100644
--- a/drivers/net/ethernet/netronome/nfp/nic/main.c
+++ b/drivers/net/ethernet/netronome/nfp/nic/main.c
@@ -12,7 +12,8 @@ static int nfp_nic_init(struct nfp_app *app)
{
struct nfp_pf *pf = app->pf;
- if (pf->eth_tbl && pf->max_data_vnics != pf->eth_tbl->count) {
+ if (pf->eth_tbl && pf->max_data_vnics != pf->eth_tbl->count &&
+ !pf->multi_pf.en) {
nfp_err(pf->cpp, "ETH entries don't match vNICs (%d vs %d)\n",
pf->max_data_vnics, pf->eth_tbl->count);
return -EINVAL;
--
2.34.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH net-next 09/12] nfp: enable multi-PF in application firmware if supported
2023-07-24 9:48 [PATCH net-next 00/12] nfp: add support for multi-pf configuration Louis Peens
` (7 preceding siblings ...)
2023-07-24 9:48 ` [PATCH net-next 08/12] nfp: apply one port per PF for multi-PF setup Louis Peens
@ 2023-07-24 9:48 ` Louis Peens
2023-07-24 9:48 ` [PATCH net-next 10/12] nfp: configure VF total count for each PF Louis Peens
` (3 subsequent siblings)
12 siblings, 0 replies; 22+ messages in thread
From: Louis Peens @ 2023-07-24 9:48 UTC (permalink / raw)
To: David Miller, Jakub Kicinski, Paolo Abeni
Cc: Simon Horman, Yinjun Zhang, Tianyu Yuan, netdev, oss-drivers
From: Yinjun Zhang <yinjun.zhang@corigine.com>
For backward compatibility concern, the new application firmware
is designed to support both single-PF setup and multi-PF setup.
Thus driver should inform application firmware which setup current
is. This should be done as early as possible since the setup may
affect some configurations exposed by firmware.
Signed-off-by: Yinjun Zhang <yinjun.zhang@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
.../net/ethernet/netronome/nfp/nfp_net_ctrl.h | 1 +
.../net/ethernet/netronome/nfp/nfp_net_main.c | 129 ++++++++++++++----
2 files changed, 100 insertions(+), 30 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h b/drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h
index 3e63f6d6a563..d6b127f13ed3 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h
@@ -268,6 +268,7 @@
#define NFP_NET_CFG_CTRL_PKT_TYPE (0x1 << 0) /* Pkttype offload */
#define NFP_NET_CFG_CTRL_IPSEC (0x1 << 1) /* IPsec offload */
#define NFP_NET_CFG_CTRL_MCAST_FILTER (0x1 << 2) /* Multicast Filter */
+#define NFP_NET_CFG_CTRL_MULTI_PF (0x1 << 5) /* Multi PF */
#define NFP_NET_CFG_CTRL_FREELIST_EN (0x1 << 6) /* Freelist enable flag bit */
#define NFP_NET_CFG_CAP_WORD1 0x00a4
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
index 98e155d79eb8..f6f4fea0a791 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
@@ -684,15 +684,108 @@ int nfp_net_refresh_eth_port(struct nfp_port *port)
return ret;
}
+/**
+ * nfp_net_pre_init() - Some necessary check and configuration
+ * in firmware before fully utilizing it.
+ * @pf: NFP PF handler
+ * @stride: queue stride
+ *
+ * Return: 0 on success, a negative error code otherwise.
+ */
+static int nfp_net_pre_init(struct nfp_pf *pf, int *stride)
+{
+ struct nfp_net_fw_version fw_ver;
+ struct nfp_cpp_area *area;
+ u8 __iomem *ctrl_bar;
+ int err = 0;
+
+ ctrl_bar = nfp_pf_map_rtsym(pf, NULL, "_pf%d_net_bar0", NFP_PF_CSR_SLICE_SIZE, &area);
+ if (IS_ERR(ctrl_bar)) {
+ nfp_err(pf->cpp, "Failed to find data vNIC memory symbol\n");
+ return PTR_ERR(ctrl_bar);
+ }
+
+ nfp_net_get_fw_version(&fw_ver, ctrl_bar);
+ if (fw_ver.extend & NFP_NET_CFG_VERSION_RESERVED_MASK ||
+ fw_ver.class != NFP_NET_CFG_VERSION_CLASS_GENERIC) {
+ nfp_err(pf->cpp, "Unknown Firmware ABI %d.%d.%d.%d\n",
+ fw_ver.extend, fw_ver.class,
+ fw_ver.major, fw_ver.minor);
+ err = -EINVAL;
+ goto end;
+ }
+
+ /* Determine stride */
+ if (nfp_net_fw_ver_eq(&fw_ver, 0, 0, 0, 1)) {
+ *stride = 2;
+ nfp_warn(pf->cpp, "OBSOLETE Firmware detected - VF isolation not available\n");
+ } else {
+ switch (fw_ver.major) {
+ case 1 ... 5:
+ *stride = 4;
+ break;
+ default:
+ nfp_err(pf->cpp, "Unsupported Firmware ABI %d.%d.%d.%d\n",
+ fw_ver.extend, fw_ver.class,
+ fw_ver.major, fw_ver.minor);
+ err = -EINVAL;
+ goto end;
+ }
+ }
+
+ if (!pf->multi_pf.en)
+ goto end;
+
+ /* Enable multi-PF. */
+ if (readl(ctrl_bar + NFP_NET_CFG_CAP_WORD1) & NFP_NET_CFG_CTRL_MULTI_PF) {
+ unsigned long long addr;
+ u32 cfg_q, cpp_id, ret;
+ unsigned long timeout;
+
+ writel(NFP_NET_CFG_CTRL_MULTI_PF, ctrl_bar + NFP_NET_CFG_CTRL_WORD1);
+ writel(NFP_NET_CFG_UPDATE_GEN, ctrl_bar + NFP_NET_CFG_UPDATE);
+
+ /* Config queue is next to txq. */
+ cfg_q = readl(ctrl_bar + NFP_NET_CFG_START_TXQ) + 1;
+ addr = nfp_qcp_queue_offset(pf->dev_info, cfg_q) + NFP_QCP_QUEUE_ADD_WPTR;
+ cpp_id = NFP_CPP_ISLAND_ID(0, NFP_CPP_ACTION_RW, 0, 0);
+ err = nfp_cpp_writel(pf->cpp, cpp_id, addr, 1);
+ if (err)
+ goto end;
+
+ timeout = jiffies + HZ * NFP_NET_POLL_TIMEOUT;
+ while ((ret = readl(ctrl_bar + NFP_NET_CFG_UPDATE))) {
+ if (ret & NFP_NET_CFG_UPDATE_ERR) {
+ nfp_err(pf->cpp, "Enable multi-PF failed\n");
+ err = -EIO;
+ break;
+ }
+
+ usleep_range(250, 500);
+ if (time_is_before_eq_jiffies(timeout)) {
+ nfp_err(pf->cpp, "Enable multi-PF timeout\n");
+ err = -ETIMEDOUT;
+ break;
+ }
+ };
+ } else {
+ nfp_err(pf->cpp, "Loaded firmware doesn't support multi-PF\n");
+ err = -EINVAL;
+ }
+
+end:
+ nfp_cpp_area_release_free(area);
+ return err;
+}
+
/*
* PCI device functions
*/
int nfp_net_pci_probe(struct nfp_pf *pf)
{
struct devlink *devlink = priv_to_devlink(pf);
- struct nfp_net_fw_version fw_ver;
u8 __iomem *ctrl_bar, *qc_bar;
- int stride;
+ int stride = 0;
int err;
INIT_WORK(&pf->port_refresh_work, nfp_net_refresh_vnics);
@@ -703,6 +796,10 @@ int nfp_net_pci_probe(struct nfp_pf *pf)
return -EINVAL;
}
+ err = nfp_net_pre_init(pf, &stride);
+ if (err)
+ return err;
+
pf->max_data_vnics = nfp_net_pf_get_num_ports(pf);
if ((int)pf->max_data_vnics < 0)
return pf->max_data_vnics;
@@ -723,34 +820,6 @@ int nfp_net_pci_probe(struct nfp_pf *pf)
goto err_unmap;
}
- nfp_net_get_fw_version(&fw_ver, ctrl_bar);
- if (fw_ver.extend & NFP_NET_CFG_VERSION_RESERVED_MASK ||
- fw_ver.class != NFP_NET_CFG_VERSION_CLASS_GENERIC) {
- nfp_err(pf->cpp, "Unknown Firmware ABI %d.%d.%d.%d\n",
- fw_ver.extend, fw_ver.class,
- fw_ver.major, fw_ver.minor);
- err = -EINVAL;
- goto err_unmap;
- }
-
- /* Determine stride */
- if (nfp_net_fw_ver_eq(&fw_ver, 0, 0, 0, 1)) {
- stride = 2;
- nfp_warn(pf->cpp, "OBSOLETE Firmware detected - VF isolation not available\n");
- } else {
- switch (fw_ver.major) {
- case 1 ... 5:
- stride = 4;
- break;
- default:
- nfp_err(pf->cpp, "Unsupported Firmware ABI %d.%d.%d.%d\n",
- fw_ver.extend, fw_ver.class,
- fw_ver.major, fw_ver.minor);
- err = -EINVAL;
- goto err_unmap;
- }
- }
-
err = nfp_net_pf_app_init(pf, qc_bar, stride);
if (err)
goto err_unmap;
--
2.34.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH net-next 10/12] nfp: configure VF total count for each PF
2023-07-24 9:48 [PATCH net-next 00/12] nfp: add support for multi-pf configuration Louis Peens
` (8 preceding siblings ...)
2023-07-24 9:48 ` [PATCH net-next 09/12] nfp: enable multi-PF in application firmware if supported Louis Peens
@ 2023-07-24 9:48 ` Louis Peens
2023-07-24 9:48 ` [PATCH net-next 11/12] nfp: configure VF split info into application firmware Louis Peens
` (2 subsequent siblings)
12 siblings, 0 replies; 22+ messages in thread
From: Louis Peens @ 2023-07-24 9:48 UTC (permalink / raw)
To: David Miller, Jakub Kicinski, Paolo Abeni
Cc: Simon Horman, Yinjun Zhang, Tianyu Yuan, netdev, oss-drivers
From: Tianyu Yuan <tianyu.yuan@corigine.com>
By default, PFs share the total 64 VFs equally, i.e., 32 VFs
for each PF in two port NIC, which is initialized in each
PF’s SR-IOV capability register by management firmware.
And a new hwinfo `abi_total_vf` is introduced to make each
PF’s VF total count configurable. Management firmware reads the
hwinfo and configures it in SR-IOV capability register during
boot process. So reboot is required to make the configuration
take effect. This is not touched in driver code.
Driver then modifies each PF’s `sriov_totalvf` according to
maximum VF count supported by the loaded application firmware.
Here we apply the rule that the PF with smaller id is satisfied
first if total configured count exceeds the limitation.
Signed-off-by: Tianyu Yuan <tianyu.yuan@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
drivers/net/ethernet/netronome/nfp/nfp_main.c | 49 +++++++++++++++++--
1 file changed, 45 insertions(+), 4 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.c b/drivers/net/ethernet/netronome/nfp/nfp_main.c
index 74767729e542..b15b5fe0c1c9 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.c
@@ -224,11 +224,48 @@ static int nfp_pf_board_state_wait(struct nfp_pf *pf)
return 0;
}
+static unsigned int nfp_pf_get_limit_vfs(struct nfp_pf *pf,
+ unsigned int limit_vfs_rtsym)
+{
+ u16 pos, offset, total;
+
+ if (!pf->multi_pf.en || !limit_vfs_rtsym)
+ return limit_vfs_rtsym;
+
+ pos = pci_find_ext_capability(pf->pdev, PCI_EXT_CAP_ID_SRIOV);
+ if (!pos)
+ return 0;
+
+ /* Management firmware ensures that SR-IOV capability registers
+ * are initialized correctly.
+ */
+ pci_read_config_word(pf->pdev, pos + PCI_SRIOV_VF_OFFSET, &offset);
+ pci_read_config_word(pf->pdev, pos + PCI_SRIOV_TOTAL_VF, &total);
+ if (!total)
+ return 0;
+
+ /* Offset of first VF is relative to its PF. */
+ offset += pf->multi_pf.id;
+ if (offset < pf->dev_info->pf_num_per_unit)
+ return 0;
+
+ /* For multi-PF device, VF is numbered from max PF count. */
+ offset -= pf->dev_info->pf_num_per_unit;
+ if (offset >= limit_vfs_rtsym)
+ return 0;
+
+ if (offset + total > limit_vfs_rtsym)
+ return limit_vfs_rtsym - offset;
+
+ return total;
+}
+
static int nfp_pcie_sriov_read_nfd_limit(struct nfp_pf *pf)
{
+ unsigned int limit_vfs_rtsym;
int err;
- pf->limit_vfs = nfp_rtsym_read_le(pf->rtbl, "nfd_vf_cfg_max_vfs", &err);
+ limit_vfs_rtsym = nfp_rtsym_read_le(pf->rtbl, "nfd_vf_cfg_max_vfs", &err);
if (err) {
/* For backwards compatibility if symbol not found allow all */
pf->limit_vfs = ~0;
@@ -239,9 +276,13 @@ static int nfp_pcie_sriov_read_nfd_limit(struct nfp_pf *pf)
return err;
}
- err = pci_sriov_set_totalvfs(pf->pdev, pf->limit_vfs);
- if (err)
- nfp_warn(pf->cpp, "Failed to set VF count in sysfs: %d\n", err);
+ pf->limit_vfs = nfp_pf_get_limit_vfs(pf, limit_vfs_rtsym);
+ if (pci_sriov_get_totalvfs(pf->pdev) != pf->limit_vfs) {
+ err = pci_sriov_set_totalvfs(pf->pdev, pf->limit_vfs);
+ if (err)
+ nfp_warn(pf->cpp, "Failed to set VF count in sysfs: %d\n", err);
+ }
+
return 0;
}
--
2.34.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH net-next 11/12] nfp: configure VF split info into application firmware
2023-07-24 9:48 [PATCH net-next 00/12] nfp: add support for multi-pf configuration Louis Peens
` (9 preceding siblings ...)
2023-07-24 9:48 ` [PATCH net-next 10/12] nfp: configure VF total count for each PF Louis Peens
@ 2023-07-24 9:48 ` Louis Peens
2023-07-24 9:48 ` [PATCH net-next 12/12] nfp: use absolute vf id for multi-PF case Louis Peens
2023-07-25 0:01 ` [PATCH net-next 00/12] nfp: add support for multi-pf configuration Jakub Kicinski
12 siblings, 0 replies; 22+ messages in thread
From: Louis Peens @ 2023-07-24 9:48 UTC (permalink / raw)
To: David Miller, Jakub Kicinski, Paolo Abeni
Cc: Simon Horman, Yinjun Zhang, Tianyu Yuan, netdev, oss-drivers
From: Yinjun Zhang <yinjun.zhang@corigine.com>
In multi-PF case, all PFs share total 64 VFs. To support the VF
count of each PF configurable, driver needs to write the VF count
and the first VF id into application firmware, so that firmware
can initialize and allocate relevant resource accordingly.
Signed-off-by: Yinjun Zhang <yinjun.zhang@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
drivers/net/ethernet/netronome/nfp/nfp_main.c | 1 +
drivers/net/ethernet/netronome/nfp/nfp_main.h | 2 ++
.../net/ethernet/netronome/nfp/nfp_net_main.c | 16 ++++++++++++
.../ethernet/netronome/nfp/nfp_net_sriov.c | 25 +++++++++++++++++++
.../ethernet/netronome/nfp/nfp_net_sriov.h | 5 ++++
5 files changed, 49 insertions(+)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.c b/drivers/net/ethernet/netronome/nfp/nfp_main.c
index b15b5fe0c1c9..70e140e7d93b 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.c
@@ -254,6 +254,7 @@ static unsigned int nfp_pf_get_limit_vfs(struct nfp_pf *pf,
if (offset >= limit_vfs_rtsym)
return 0;
+ pf->multi_pf.vf_fid = offset;
if (offset + total > limit_vfs_rtsym)
return limit_vfs_rtsym - offset;
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.h b/drivers/net/ethernet/netronome/nfp/nfp_main.h
index 4f6763ca1c92..e7f125a3f884 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.h
@@ -87,6 +87,7 @@ struct nfp_dumpspec {
* @multi_pf: Used in multi-PF setup
* @multi_pf.en: True if it's a NIC with multiple PFs
* @multi_pf.id: PF index
+ * @multi_pf.vf_fid: Id of first VF that belongs to this PF
* @multi_pf.beat_timer:Timer for beat to keepalive
* @multi_pf.beat_area: Pointer to CPP area for beat to keepalive
* @multi_pf.beat_addr: Pointer to mapped beat address used for keepalive
@@ -151,6 +152,7 @@ struct nfp_pf {
struct {
bool en;
u8 id;
+ u8 vf_fid;
struct timer_list beat_timer;
struct nfp_cpp_area *beat_area;
u8 __iomem *beat_addr;
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
index f6f4fea0a791..eb7b0ecd65df 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
@@ -293,6 +293,16 @@ static int nfp_net_pf_init_vnics(struct nfp_pf *pf)
return err;
}
+static void nfp_net_pf_clean_vnics(struct nfp_pf *pf)
+{
+ struct nfp_net *nn;
+
+ list_for_each_entry(nn, &pf->vnics, vnic_list) {
+ if (nfp_net_is_data_vnic(nn))
+ nfp_net_pf_clean_vnic(pf, nn);
+ }
+}
+
static int
nfp_net_pf_app_init(struct nfp_pf *pf, u8 __iomem *qc_bar, unsigned int stride)
{
@@ -852,11 +862,17 @@ int nfp_net_pci_probe(struct nfp_pf *pf)
if (err)
goto err_stop_app;
+ err = nfp_net_pf_init_sriov(pf);
+ if (err)
+ goto err_clean_vnics;
+
devl_unlock(devlink);
devlink_register(devlink);
return 0;
+err_clean_vnics:
+ nfp_net_pf_clean_vnics(pf);
err_stop_app:
nfp_net_pf_app_stop(pf);
err_free_irqs:
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c b/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c
index 6eeeb0fda91f..f516ba7a429e 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c
@@ -14,6 +14,9 @@
#include "nfp_net.h"
#include "nfp_net_sriov.h"
+/* The configurations that precede VF creating. */
+#define NFP_NET_VF_PRE_CONFIG NFP_NET_VF_CFG_MB_CAP_SPLIT
+
static int
nfp_net_sriov_check(struct nfp_app *app, int vf, u16 cap, const char *msg, bool warn)
{
@@ -29,6 +32,10 @@ nfp_net_sriov_check(struct nfp_app *app, int vf, u16 cap, const char *msg, bool
return -EOPNOTSUPP;
}
+ /* No need to check vf for the pre-configurations. */
+ if (cap & NFP_NET_VF_PRE_CONFIG)
+ return 0;
+
if (vf < 0 || vf >= app->pf->num_vfs) {
if (warn)
nfp_warn(app->pf->cpp, "invalid VF id %d\n", vf);
@@ -309,3 +316,21 @@ int nfp_app_get_vf_config(struct net_device *netdev, int vf,
return 0;
}
+
+int nfp_net_pf_init_sriov(struct nfp_pf *pf)
+{
+ int err;
+
+ if (!pf->multi_pf.en || !pf->limit_vfs)
+ return 0;
+
+ err = nfp_net_sriov_check(pf->app, 0, NFP_NET_VF_CFG_MB_CAP_SPLIT, "split", true);
+ if (err)
+ return err;
+
+ writeb(pf->limit_vfs, pf->vfcfg_tbl2 + NFP_NET_VF_CFG_MB_VF_CNT);
+
+ /* Reuse NFP_NET_VF_CFG_MB_VF_NUM to pass vf_fid to FW. */
+ return nfp_net_sriov_update(pf->app, pf->multi_pf.vf_fid,
+ NFP_NET_VF_CFG_MB_UPD_SPLIT, "split");
+}
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.h b/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.h
index 2d445fa199dc..8de959018819 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.h
@@ -21,6 +21,7 @@
#define NFP_NET_VF_CFG_MB_CAP_TRUST (0x1 << 4)
#define NFP_NET_VF_CFG_MB_CAP_VLAN_PROTO (0x1 << 5)
#define NFP_NET_VF_CFG_MB_CAP_RATE (0x1 << 6)
+#define NFP_NET_VF_CFG_MB_CAP_SPLIT (0x1 << 8)
#define NFP_NET_VF_CFG_MB_RET 0x2
#define NFP_NET_VF_CFG_MB_UPD 0x4
#define NFP_NET_VF_CFG_MB_UPD_MAC (0x1 << 0)
@@ -30,6 +31,8 @@
#define NFP_NET_VF_CFG_MB_UPD_TRUST (0x1 << 4)
#define NFP_NET_VF_CFG_MB_UPD_VLAN_PROTO (0x1 << 5)
#define NFP_NET_VF_CFG_MB_UPD_RATE (0x1 << 6)
+#define NFP_NET_VF_CFG_MB_UPD_SPLIT (0x1 << 8)
+#define NFP_NET_VF_CFG_MB_VF_CNT 0x6
#define NFP_NET_VF_CFG_MB_VF_NUM 0x7
/* VF config entry
@@ -68,4 +71,6 @@ int nfp_app_set_vf_link_state(struct net_device *netdev, int vf,
int nfp_app_get_vf_config(struct net_device *netdev, int vf,
struct ifla_vf_info *ivi);
+int nfp_net_pf_init_sriov(struct nfp_pf *pf);
+
#endif /* _NFP_NET_SRIOV_H_ */
--
2.34.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH net-next 12/12] nfp: use absolute vf id for multi-PF case
2023-07-24 9:48 [PATCH net-next 00/12] nfp: add support for multi-pf configuration Louis Peens
` (10 preceding siblings ...)
2023-07-24 9:48 ` [PATCH net-next 11/12] nfp: configure VF split info into application firmware Louis Peens
@ 2023-07-24 9:48 ` Louis Peens
2023-07-25 0:01 ` [PATCH net-next 00/12] nfp: add support for multi-pf configuration Jakub Kicinski
12 siblings, 0 replies; 22+ messages in thread
From: Louis Peens @ 2023-07-24 9:48 UTC (permalink / raw)
To: David Miller, Jakub Kicinski, Paolo Abeni
Cc: Simon Horman, Yinjun Zhang, Tianyu Yuan, netdev, oss-drivers
From: Yinjun Zhang <yinjun.zhang@corigine.com>
In multi-PF setup, absolute VF id is required to configure attributes
for corresponding VF.
Add helper function to map rtsym with specified offset. With PF's first
VF as base offset, we can access `vf_cfg_mem` as before.
Signed-off-by: Yinjun Zhang <yinjun.zhang@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Acked-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Louis Peens <louis.peens@corigine.com>
---
drivers/net/ethernet/netronome/nfp/nfp_main.c | 14 +++++++++++---
drivers/net/ethernet/netronome/nfp/nfp_main.h | 4 ++++
.../net/ethernet/netronome/nfp/nfp_net_main.c | 10 ++++++----
.../net/ethernet/netronome/nfp/nfp_net_sriov.c | 14 ++++++++++----
.../ethernet/netronome/nfp/nfpcore/nfp_nffw.h | 4 ++++
.../ethernet/netronome/nfp/nfpcore/nfp_rtsym.c | 16 ++++++++++++----
6 files changed, 47 insertions(+), 15 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.c b/drivers/net/ethernet/netronome/nfp/nfp_main.c
index 70e140e7d93b..139499d891c1 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.c
@@ -97,14 +97,22 @@ int nfp_pf_rtsym_read_optional(struct nfp_pf *pf, const char *format,
}
u8 __iomem *
-nfp_pf_map_rtsym(struct nfp_pf *pf, const char *name, const char *sym_fmt,
- unsigned int min_size, struct nfp_cpp_area **area)
+nfp_pf_map_rtsym_offset(struct nfp_pf *pf, const char *name, const char *sym_fmt,
+ unsigned int offset, unsigned int min_size,
+ struct nfp_cpp_area **area)
{
char pf_symbol[256];
snprintf(pf_symbol, sizeof(pf_symbol), sym_fmt, nfp_get_pf_id(pf));
- return nfp_rtsym_map(pf->rtbl, pf_symbol, name, min_size, area);
+ return nfp_rtsym_map_offset(pf->rtbl, pf_symbol, name, offset, min_size, area);
+}
+
+u8 __iomem *
+nfp_pf_map_rtsym(struct nfp_pf *pf, const char *name, const char *sym_fmt,
+ unsigned int min_size, struct nfp_cpp_area **area)
+{
+ return nfp_pf_map_rtsym_offset(pf, name, sym_fmt, 0, min_size, area);
}
/* Callers should hold the devlink instance lock */
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.h b/drivers/net/ethernet/netronome/nfp/nfp_main.h
index e7f125a3f884..4f1623917c4e 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.h
@@ -179,6 +179,10 @@ int nfp_pf_rtsym_read_optional(struct nfp_pf *pf, const char *format,
unsigned int default_val);
int nfp_net_pf_get_app_id(struct nfp_pf *pf);
u8 __iomem *
+nfp_pf_map_rtsym_offset(struct nfp_pf *pf, const char *name, const char *sym_fmt,
+ unsigned int offset, unsigned int min_size,
+ struct nfp_cpp_area **area);
+u8 __iomem *
nfp_pf_map_rtsym(struct nfp_pf *pf, const char *name, const char *sym_fmt,
unsigned int min_size, struct nfp_cpp_area **area);
int nfp_mbox_cmd(struct nfp_pf *pf, u32 cmd, void *in_data, u64 in_length,
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
index eb7b0ecd65df..f68fd01dac60 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
@@ -473,9 +473,10 @@ static int nfp_net_pci_map_mem(struct nfp_pf *pf)
}
}
- pf->vf_cfg_mem = nfp_pf_map_rtsym(pf, "net.vfcfg", "_pf%d_net_vf_bar",
- NFP_NET_CFG_BAR_SZ * pf->limit_vfs,
- &pf->vf_cfg_bar);
+ pf->vf_cfg_mem = nfp_pf_map_rtsym_offset(pf, "net.vfcfg", "_pf%d_net_vf_bar",
+ NFP_NET_CFG_BAR_SZ * pf->multi_pf.vf_fid,
+ NFP_NET_CFG_BAR_SZ * pf->limit_vfs,
+ &pf->vf_cfg_bar);
if (IS_ERR(pf->vf_cfg_mem)) {
if (PTR_ERR(pf->vf_cfg_mem) != -ENOENT) {
err = PTR_ERR(pf->vf_cfg_mem);
@@ -484,7 +485,8 @@ static int nfp_net_pci_map_mem(struct nfp_pf *pf)
pf->vf_cfg_mem = NULL;
}
- min_size = NFP_NET_VF_CFG_SZ * pf->limit_vfs + NFP_NET_VF_CFG_MB_SZ;
+ min_size = NFP_NET_VF_CFG_SZ * (pf->limit_vfs + pf->multi_pf.vf_fid) +
+ NFP_NET_VF_CFG_MB_SZ;
pf->vfcfg_tbl2 = nfp_pf_map_rtsym(pf, "net.vfcfg_tbl2",
"_pf%d_net_vf_cfg2",
min_size, &pf->vfcfg_tbl2_area);
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c b/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c
index f516ba7a429e..67aea9445aa2 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c
@@ -72,7 +72,7 @@ int nfp_app_set_vf_mac(struct net_device *netdev, int vf, u8 *mac)
{
struct nfp_app *app = nfp_app_from_netdev(netdev);
unsigned int vf_offset;
- int err;
+ int err, abs_vf;
err = nfp_net_sriov_check(app, vf, NFP_NET_VF_CFG_MB_CAP_MAC, "mac", true);
if (err)
@@ -85,13 +85,14 @@ int nfp_app_set_vf_mac(struct net_device *netdev, int vf, u8 *mac)
return -EINVAL;
}
+ abs_vf = vf + app->pf->multi_pf.vf_fid;
/* Write MAC to VF entry in VF config symbol */
- vf_offset = NFP_NET_VF_CFG_MB_SZ + vf * NFP_NET_VF_CFG_SZ;
+ vf_offset = NFP_NET_VF_CFG_MB_SZ + abs_vf * NFP_NET_VF_CFG_SZ;
writel(get_unaligned_be32(mac), app->pf->vfcfg_tbl2 + vf_offset);
writew(get_unaligned_be16(mac + 4),
app->pf->vfcfg_tbl2 + vf_offset + NFP_NET_VF_CFG_MAC_LO);
- err = nfp_net_sriov_update(app, vf, NFP_NET_VF_CFG_MB_UPD_MAC, "MAC");
+ err = nfp_net_sriov_update(app, abs_vf, NFP_NET_VF_CFG_MB_UPD_MAC, "MAC");
if (!err)
nfp_info(app->pf->cpp,
"MAC %pM set on VF %d, reload the VF driver to make this change effective.\n",
@@ -145,6 +146,7 @@ int nfp_app_set_vf_vlan(struct net_device *netdev, int vf, u16 vlan, u8 qos,
if (vlan_tag && is_proto_sup)
vlan_tag |= FIELD_PREP(NFP_NET_VF_CFG_VLAN_PROT, ntohs(vlan_proto));
+ vf += app->pf->multi_pf.vf_fid;
vf_offset = NFP_NET_VF_CFG_MB_SZ + vf * NFP_NET_VF_CFG_SZ;
writel(vlan_tag, app->pf->vfcfg_tbl2 + vf_offset + NFP_NET_VF_CFG_VLAN);
@@ -169,6 +171,7 @@ int nfp_app_set_vf_rate(struct net_device *netdev, int vf,
return -EINVAL;
}
+ vf += app->pf->multi_pf.vf_fid;
vf_offset = NFP_NET_VF_CFG_MB_SZ + vf * NFP_NET_VF_CFG_SZ;
ratevalue = FIELD_PREP(NFP_NET_VF_CFG_MAX_RATE,
max_tx_rate ? max_tx_rate :
@@ -195,6 +198,7 @@ int nfp_app_set_vf_spoofchk(struct net_device *netdev, int vf, bool enable)
return err;
/* Write spoof check control bit to VF entry in VF config symbol */
+ vf += app->pf->multi_pf.vf_fid;
vf_offset = NFP_NET_VF_CFG_MB_SZ + vf * NFP_NET_VF_CFG_SZ +
NFP_NET_VF_CFG_CTRL;
vf_ctrl = readb(app->pf->vfcfg_tbl2 + vf_offset);
@@ -219,6 +223,7 @@ int nfp_app_set_vf_trust(struct net_device *netdev, int vf, bool enable)
return err;
/* Write trust control bit to VF entry in VF config symbol */
+ vf += app->pf->multi_pf.vf_fid;
vf_offset = NFP_NET_VF_CFG_MB_SZ + vf * NFP_NET_VF_CFG_SZ +
NFP_NET_VF_CFG_CTRL;
vf_ctrl = readb(app->pf->vfcfg_tbl2 + vf_offset);
@@ -253,6 +258,7 @@ int nfp_app_set_vf_link_state(struct net_device *netdev, int vf,
}
/* Write link state to VF entry in VF config symbol */
+ vf += app->pf->multi_pf.vf_fid;
vf_offset = NFP_NET_VF_CFG_MB_SZ + vf * NFP_NET_VF_CFG_SZ +
NFP_NET_VF_CFG_CTRL;
vf_ctrl = readb(app->pf->vfcfg_tbl2 + vf_offset);
@@ -278,7 +284,7 @@ int nfp_app_get_vf_config(struct net_device *netdev, int vf,
if (err)
return err;
- vf_offset = NFP_NET_VF_CFG_MB_SZ + vf * NFP_NET_VF_CFG_SZ;
+ vf_offset = NFP_NET_VF_CFG_MB_SZ + (vf + app->pf->multi_pf.vf_fid) * NFP_NET_VF_CFG_SZ;
mac_hi = readl(app->pf->vfcfg_tbl2 + vf_offset);
mac_lo = readw(app->pf->vfcfg_tbl2 + vf_offset + NFP_NET_VF_CFG_MAC_LO);
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nffw.h b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nffw.h
index 49a4d3f56b56..4042352f83b0 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nffw.h
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nffw.h
@@ -101,6 +101,10 @@ u64 nfp_rtsym_read_le(struct nfp_rtsym_table *rtbl, const char *name,
int nfp_rtsym_write_le(struct nfp_rtsym_table *rtbl, const char *name,
u64 value);
u8 __iomem *
+nfp_rtsym_map_offset(struct nfp_rtsym_table *rtbl, const char *name, const char *id,
+ unsigned int offset, unsigned int min_size,
+ struct nfp_cpp_area **area);
+u8 __iomem *
nfp_rtsym_map(struct nfp_rtsym_table *rtbl, const char *name, const char *id,
unsigned int min_size, struct nfp_cpp_area **area);
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_rtsym.c b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_rtsym.c
index 2260c2403a83..97a4417a1c1b 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_rtsym.c
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_rtsym.c
@@ -520,8 +520,9 @@ int nfp_rtsym_write_le(struct nfp_rtsym_table *rtbl, const char *name,
}
u8 __iomem *
-nfp_rtsym_map(struct nfp_rtsym_table *rtbl, const char *name, const char *id,
- unsigned int min_size, struct nfp_cpp_area **area)
+nfp_rtsym_map_offset(struct nfp_rtsym_table *rtbl, const char *name, const char *id,
+ unsigned int offset, unsigned int min_size,
+ struct nfp_cpp_area **area)
{
const struct nfp_rtsym *sym;
u8 __iomem *mem;
@@ -540,12 +541,12 @@ nfp_rtsym_map(struct nfp_rtsym_table *rtbl, const char *name, const char *id,
return (u8 __iomem *)ERR_PTR(err);
}
- if (sym->size < min_size) {
+ if (sym->size < min_size + offset) {
nfp_err(rtbl->cpp, "rtsym '%s': too small\n", name);
return (u8 __iomem *)ERR_PTR(-EINVAL);
}
- mem = nfp_cpp_map_area(rtbl->cpp, id, cpp_id, addr, sym->size, area);
+ mem = nfp_cpp_map_area(rtbl->cpp, id, cpp_id, addr + offset, sym->size - offset, area);
if (IS_ERR(mem)) {
nfp_err(rtbl->cpp, "rtysm '%s': failed to map: %ld\n",
name, PTR_ERR(mem));
@@ -554,3 +555,10 @@ nfp_rtsym_map(struct nfp_rtsym_table *rtbl, const char *name, const char *id,
return mem;
}
+
+u8 __iomem *
+nfp_rtsym_map(struct nfp_rtsym_table *rtbl, const char *name, const char *id,
+ unsigned int min_size, struct nfp_cpp_area **area)
+{
+ return nfp_rtsym_map_offset(rtbl, name, id, 0, min_size, area);
+}
--
2.34.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* Re: [PATCH net-next 00/12] nfp: add support for multi-pf configuration
2023-07-24 9:48 [PATCH net-next 00/12] nfp: add support for multi-pf configuration Louis Peens
` (11 preceding siblings ...)
2023-07-24 9:48 ` [PATCH net-next 12/12] nfp: use absolute vf id for multi-PF case Louis Peens
@ 2023-07-25 0:01 ` Jakub Kicinski
2023-07-25 1:28 ` Yinjun Zhang
12 siblings, 1 reply; 22+ messages in thread
From: Jakub Kicinski @ 2023-07-25 0:01 UTC (permalink / raw)
To: Louis Peens
Cc: David Miller, Paolo Abeni, Simon Horman, Yinjun Zhang,
Tianyu Yuan, netdev, oss-drivers
On Mon, 24 Jul 2023 11:48:09 +0200 Louis Peens wrote:
> This patch series is introducing multiple PFs for multiple ports NIC
> assembled with NFP3800 chip. This is done since the NFP3800 can
> support up to 4 PFs, and is more in-line with the modern expectation
> that each port/netdev is associated with a unique PF.
>
> For compatibility concern with NFP4000/6000 cards, and older management
> firmware on NFP3800, multiple ports sharing single PF is still supported
> with this change. Whether it's multi-PF setup or single-PF setup is
> determined by management firmware, and driver will notify the
> application firmware of the setup so that both are well handled.
So every PF will have its own devlink instance?
Can you show devlink dev info output?
^ permalink raw reply [flat|nested] 22+ messages in thread* RE: [PATCH net-next 00/12] nfp: add support for multi-pf configuration
2023-07-25 0:01 ` [PATCH net-next 00/12] nfp: add support for multi-pf configuration Jakub Kicinski
@ 2023-07-25 1:28 ` Yinjun Zhang
2023-07-25 18:59 ` Jakub Kicinski
0 siblings, 1 reply; 22+ messages in thread
From: Yinjun Zhang @ 2023-07-25 1:28 UTC (permalink / raw)
To: Jakub Kicinski, Louis Peens
Cc: David Miller, Paolo Abeni, Simon Horman, Tianyu Yuan,
netdev@vger.kernel.org, oss-drivers
On Tuesday, July 25, 2023 8:01 AM, Jakub Kicinski wrote:
> On Mon, 24 Jul 2023 11:48:09 +0200 Louis Peens wrote:
> > This patch series is introducing multiple PFs for multiple ports NIC
> > assembled with NFP3800 chip. This is done since the NFP3800 can
> > support up to 4 PFs, and is more in-line with the modern expectation
> > that each port/netdev is associated with a unique PF.
> >
> > For compatibility concern with NFP4000/6000 cards, and older management
> > firmware on NFP3800, multiple ports sharing single PF is still supported
> > with this change. Whether it's multi-PF setup or single-PF setup is
> > determined by management firmware, and driver will notify the
> > application firmware of the setup so that both are well handled.
>
> So every PF will have its own devlink instance?
> Can you show devlink dev info output?
Yes, here it is:
pci/0000:01:00.0:
driver nfp
serial_number UKAAMDA2000-100122190023
versions:
fixed:
board.id AMDA2000-1001
board.rev 01
board.manufacture UKA
board.model schubert
running:
fw.mgmt 23.07-1
fw.cpld 0x1000001
fw.app sri-23.07.0-0
chip.init AMDA-2000-1001 20230321150349
stored:
fw.mgmt 23.07-1
fw.cpld 0x0
chip.init AMDA-2000-1001 20230321150349
pci/0000:01:00.1:
driver nfp
serial_number UKAAMDA2000-100122190023
versions:
fixed:
board.id AMDA2000-1001
board.rev 01
board.manufacture UKA
board.model schubert
running:
fw.mgmt 23.07-1
fw.cpld 0x1000001
fw.app sri-23.07.0-0
chip.init AMDA-2000-1001 20230321150349
stored:
fw.mgmt 23.07-1
fw.cpld 0x0
chip.init AMDA-2000-1001 20230321150349
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [PATCH net-next 00/12] nfp: add support for multi-pf configuration
2023-07-25 1:28 ` Yinjun Zhang
@ 2023-07-25 18:59 ` Jakub Kicinski
2023-07-26 2:00 ` Yinjun Zhang
0 siblings, 1 reply; 22+ messages in thread
From: Jakub Kicinski @ 2023-07-25 18:59 UTC (permalink / raw)
To: Yinjun Zhang
Cc: Louis Peens, David Miller, Paolo Abeni, Simon Horman, Tianyu Yuan,
netdev@vger.kernel.org, oss-drivers
On Tue, 25 Jul 2023 01:28:34 +0000 Yinjun Zhang wrote:
> On Tuesday, July 25, 2023 8:01 AM, Jakub Kicinski wrote:
> > On Mon, 24 Jul 2023 11:48:09 +0200 Louis Peens wrote:
> > > This patch series is introducing multiple PFs for multiple ports NIC
> > > assembled with NFP3800 chip. This is done since the NFP3800 can
> > > support up to 4 PFs, and is more in-line with the modern expectation
> > > that each port/netdev is associated with a unique PF.
> > >
> > > For compatibility concern with NFP4000/6000 cards, and older management
> > > firmware on NFP3800, multiple ports sharing single PF is still supported
> > > with this change. Whether it's multi-PF setup or single-PF setup is
> > > determined by management firmware, and driver will notify the
> > > application firmware of the setup so that both are well handled.
> >
> > So every PF will have its own devlink instance?
> > Can you show devlink dev info output?
>
> Yes, here it is:
> serial_number UKAAMDA2000-100122190023
> serial_number UKAAMDA2000-100122190023
Since it's clearly a single ASIC shouldn't it have a single devlink
instance?
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: [PATCH net-next 00/12] nfp: add support for multi-pf configuration
2023-07-25 18:59 ` Jakub Kicinski
@ 2023-07-26 2:00 ` Yinjun Zhang
2023-07-26 4:17 ` Jakub Kicinski
0 siblings, 1 reply; 22+ messages in thread
From: Yinjun Zhang @ 2023-07-26 2:00 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Louis Peens, David Miller, Paolo Abeni, Simon Horman, Tianyu Yuan,
netdev@vger.kernel.org, oss-drivers
On Wednesday, July 26, 2023 3:00 AM, Jakub Kicinski wrote:
> On Tue, 25 Jul 2023 01:28:34 +0000 Yinjun Zhang wrote:
> > On Tuesday, July 25, 2023 8:01 AM, Jakub Kicinski wrote:
> > > On Mon, 24 Jul 2023 11:48:09 +0200 Louis Peens wrote:
> > > > This patch series is introducing multiple PFs for multiple ports NIC
> > > > assembled with NFP3800 chip. This is done since the NFP3800 can
> > > > support up to 4 PFs, and is more in-line with the modern expectation
> > > > that each port/netdev is associated with a unique PF.
> > > >
> > > > For compatibility concern with NFP4000/6000 cards, and older
> management
> > > > firmware on NFP3800, multiple ports sharing single PF is still supported
> > > > with this change. Whether it's multi-PF setup or single-PF setup is
> > > > determined by management firmware, and driver will notify the
> > > > application firmware of the setup so that both are well handled.
> > >
> > > So every PF will have its own devlink instance?
> > > Can you show devlink dev info output?
> >
> > Yes, here it is:
>
> > serial_number UKAAMDA2000-100122190023
>
> > serial_number UKAAMDA2000-100122190023
>
> Since it's clearly a single ASIC shouldn't it have a single devlink
> instance?
But there're more than one PCI device now. Isn't it universal implementation
to register a devlink for each PCI device?
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH net-next 00/12] nfp: add support for multi-pf configuration
2023-07-26 2:00 ` Yinjun Zhang
@ 2023-07-26 4:17 ` Jakub Kicinski
2023-07-26 7:28 ` Yinjun Zhang
0 siblings, 1 reply; 22+ messages in thread
From: Jakub Kicinski @ 2023-07-26 4:17 UTC (permalink / raw)
To: Yinjun Zhang
Cc: Louis Peens, David Miller, Paolo Abeni, Simon Horman, Tianyu Yuan,
netdev@vger.kernel.org, oss-drivers
On Wed, 26 Jul 2023 02:00:30 +0000 Yinjun Zhang wrote:
> > > serial_number UKAAMDA2000-100122190023
> >
> > > serial_number UKAAMDA2000-100122190023
> >
> > Since it's clearly a single ASIC shouldn't it have a single devlink
> > instance?
>
> But there're more than one PCI device now. Isn't it universal implementation
> to register a devlink for each PCI device?
It's only the prevailing implementation because people are too lazy to
implement things correctly, if you ask me. devlink doesn't have the
ability to bind to multiple bus devices, that would need to be
addressed.
^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: [PATCH net-next 00/12] nfp: add support for multi-pf configuration
2023-07-26 4:17 ` Jakub Kicinski
@ 2023-07-26 7:28 ` Yinjun Zhang
0 siblings, 0 replies; 22+ messages in thread
From: Yinjun Zhang @ 2023-07-26 7:28 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Louis Peens, David Miller, Paolo Abeni, Simon Horman, Tianyu Yuan,
netdev@vger.kernel.org, oss-drivers
On Wednesday, July 26, 2023 12:17 PM, Jakub Kicinski wrote:
> On Wed, 26 Jul 2023 02:00:30 +0000 Yinjun Zhang wrote:
> > > > serial_number UKAAMDA2000-100122190023
> > >
> > > > serial_number UKAAMDA2000-100122190023
> > >
> > > Since it's clearly a single ASIC shouldn't it have a single devlink
> > > instance?
> >
> > But there're more than one PCI device now. Isn't it universal
> implementation
> > to register a devlink for each PCI device?
>
> It's only the prevailing implementation because people are too lazy to
> implement things correctly, if you ask me. devlink doesn't have the
> ability to bind to multiple bus devices, that would need to be
> addressed.
It sounds like a separate topic to refine devlink infrastructure.
^ permalink raw reply [flat|nested] 22+ messages in thread