* Re: [PATCH 4/8] enic: enable rq extended cq support
@ 2025-03-02 14:29 kernel test robot
0 siblings, 0 replies; 3+ messages in thread
From: kernel test robot @ 2025-03-02 14:29 UTC (permalink / raw)
To: oe-kbuild; +Cc: lkp, Dan Carpenter
BCC: lkp@intel.com
CC: oe-kbuild-all@lists.linux.dev
In-Reply-To: <20250227-enic_cleanup_and_ext_cq-v1-4-c314f95812bb@cisco.com>
References: <20250227-enic_cleanup_and_ext_cq-v1-4-c314f95812bb@cisco.com>
TO: Satish Kharat via B4 Relay <devnull+satishkh.cisco.com@kernel.org>
TO: Christian Benvenuti <benve@cisco.com>
TO: Andrew Lunn <andrew+netdev@lunn.ch>
TO: "David S. Miller" <davem@davemloft.net>
CC: netdev@vger.kernel.org
TO: Eric Dumazet <edumazet@google.com>
TO: Jakub Kicinski <kuba@kernel.org>
TO: Paolo Abeni <pabeni@redhat.com>
CC: linux-kernel@vger.kernel.org
CC: Satish Kharat <satishkh@cisco.com>
CC: Nelson Escobar <neescoba@cisco.com>
CC: John Daley <johndale@cisco.com>
Hi Satish,
kernel test robot noticed the following build warnings:
[auto build test WARNING on de7a88b639d488607352a270ef2e052c4442b1b3]
url: https://github.com/intel-lab-lkp/linux/commits/Satish-Kharat-via-B4-Relay/enic-Move-function-from-header-file-to-c-file/20250228-083517
base: de7a88b639d488607352a270ef2e052c4442b1b3
patch link: https://lore.kernel.org/r/20250227-enic_cleanup_and_ext_cq-v1-4-c314f95812bb%40cisco.com
patch subject: [PATCH 4/8] enic: enable rq extended cq support
:::::: branch date: 3 days ago
:::::: commit date: 3 days ago
config: sparc-randconfig-r071-20250302 (https://download.01.org/0day-ci/archive/20250302/202503022204.u6MW8MpW-lkp@intel.com/config)
compiler: sparc-linux-gcc (GCC) 14.2.0
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Reported-by: Dan Carpenter <error27@gmail.com>
| Closes: https://lore.kernel.org/r/202503022204.u6MW8MpW-lkp@intel.com/
smatch warnings:
drivers/net/ethernet/cisco/enic/enic_rq.c:94 enic_rq_cq_desc_dec() error: uninitialized symbol 'type_color'.
vim +/type_color +94 drivers/net/ethernet/cisco/enic/enic_rq.c
fe57762c649058 John Daley 2025-02-05 23
180b3e15104ce9 Satish Kharat 2025-02-27 24 static void enic_rq_cq_desc_dec(void *cq_desc, u8 cq_desc_size, u8 *type,
34d3abffbe44db Satish Kharat 2025-02-27 25 u8 *color, u16 *q_number, u16 *completed_index)
fe57762c649058 John Daley 2025-02-05 26 {
34d3abffbe44db Satish Kharat 2025-02-27 27 /* type_color is the last field for all cq structs */
180b3e15104ce9 Satish Kharat 2025-02-27 28 u8 type_color;
180b3e15104ce9 Satish Kharat 2025-02-27 29
180b3e15104ce9 Satish Kharat 2025-02-27 30 switch (cq_desc_size) {
180b3e15104ce9 Satish Kharat 2025-02-27 31 case VNIC_RQ_CQ_ENTRY_SIZE_16: {
180b3e15104ce9 Satish Kharat 2025-02-27 32 struct cq_enet_rq_desc *desc =
180b3e15104ce9 Satish Kharat 2025-02-27 33 (struct cq_enet_rq_desc *)cq_desc;
180b3e15104ce9 Satish Kharat 2025-02-27 34 type_color = desc->type_color;
180b3e15104ce9 Satish Kharat 2025-02-27 35
180b3e15104ce9 Satish Kharat 2025-02-27 36 /* Make sure color bit is read from desc *before* other fields
180b3e15104ce9 Satish Kharat 2025-02-27 37 * are read from desc. Hardware guarantees color bit is last
180b3e15104ce9 Satish Kharat 2025-02-27 38 * bit (byte) written. Adding the rmb() prevents the compiler
180b3e15104ce9 Satish Kharat 2025-02-27 39 * and/or CPU from reordering the reads which would potentially
180b3e15104ce9 Satish Kharat 2025-02-27 40 * result in reading stale values.
180b3e15104ce9 Satish Kharat 2025-02-27 41 */
180b3e15104ce9 Satish Kharat 2025-02-27 42 rmb();
180b3e15104ce9 Satish Kharat 2025-02-27 43
180b3e15104ce9 Satish Kharat 2025-02-27 44 *q_number = le16_to_cpu(desc->q_number_rss_type_flags) &
180b3e15104ce9 Satish Kharat 2025-02-27 45 CQ_DESC_Q_NUM_MASK;
180b3e15104ce9 Satish Kharat 2025-02-27 46 *completed_index = le16_to_cpu(desc->completed_index_flags) &
180b3e15104ce9 Satish Kharat 2025-02-27 47 CQ_DESC_COMP_NDX_MASK;
180b3e15104ce9 Satish Kharat 2025-02-27 48 break;
180b3e15104ce9 Satish Kharat 2025-02-27 49 }
180b3e15104ce9 Satish Kharat 2025-02-27 50 case VNIC_RQ_CQ_ENTRY_SIZE_32: {
180b3e15104ce9 Satish Kharat 2025-02-27 51 struct cq_enet_rq_desc_32 *desc =
180b3e15104ce9 Satish Kharat 2025-02-27 52 (struct cq_enet_rq_desc_32 *)cq_desc;
180b3e15104ce9 Satish Kharat 2025-02-27 53 type_color = desc->type_color;
180b3e15104ce9 Satish Kharat 2025-02-27 54
180b3e15104ce9 Satish Kharat 2025-02-27 55 /* Make sure color bit is read from desc *before* other fields
180b3e15104ce9 Satish Kharat 2025-02-27 56 * are read from desc. Hardware guarantees color bit is last
180b3e15104ce9 Satish Kharat 2025-02-27 57 * bit (byte) written. Adding the rmb() prevents the compiler
180b3e15104ce9 Satish Kharat 2025-02-27 58 * and/or CPU from reordering the reads which would potentially
180b3e15104ce9 Satish Kharat 2025-02-27 59 * result in reading stale values.
180b3e15104ce9 Satish Kharat 2025-02-27 60 */
180b3e15104ce9 Satish Kharat 2025-02-27 61 rmb();
180b3e15104ce9 Satish Kharat 2025-02-27 62
180b3e15104ce9 Satish Kharat 2025-02-27 63 *q_number = le16_to_cpu(desc->q_number_rss_type_flags) &
180b3e15104ce9 Satish Kharat 2025-02-27 64 CQ_DESC_Q_NUM_MASK;
180b3e15104ce9 Satish Kharat 2025-02-27 65 *completed_index = le16_to_cpu(desc->completed_index_flags) &
180b3e15104ce9 Satish Kharat 2025-02-27 66 CQ_DESC_COMP_NDX_MASK;
180b3e15104ce9 Satish Kharat 2025-02-27 67 *completed_index |= (desc->fetch_index_flags & CQ_DESC_32_FI_MASK) <<
180b3e15104ce9 Satish Kharat 2025-02-27 68 CQ_DESC_COMP_NDX_BITS;
180b3e15104ce9 Satish Kharat 2025-02-27 69 break;
180b3e15104ce9 Satish Kharat 2025-02-27 70 }
180b3e15104ce9 Satish Kharat 2025-02-27 71 case VNIC_RQ_CQ_ENTRY_SIZE_64: {
180b3e15104ce9 Satish Kharat 2025-02-27 72 struct cq_enet_rq_desc_64 *desc =
180b3e15104ce9 Satish Kharat 2025-02-27 73 (struct cq_enet_rq_desc_64 *)cq_desc;
180b3e15104ce9 Satish Kharat 2025-02-27 74 type_color = desc->type_color;
34d3abffbe44db Satish Kharat 2025-02-27 75
34d3abffbe44db Satish Kharat 2025-02-27 76 /* Make sure color bit is read from desc *before* other fields
34d3abffbe44db Satish Kharat 2025-02-27 77 * are read from desc. Hardware guarantees color bit is last
34d3abffbe44db Satish Kharat 2025-02-27 78 * bit (byte) written. Adding the rmb() prevents the compiler
34d3abffbe44db Satish Kharat 2025-02-27 79 * and/or CPU from reordering the reads which would potentially
34d3abffbe44db Satish Kharat 2025-02-27 80 * result in reading stale values.
34d3abffbe44db Satish Kharat 2025-02-27 81 */
34d3abffbe44db Satish Kharat 2025-02-27 82 rmb();
34d3abffbe44db Satish Kharat 2025-02-27 83
34d3abffbe44db Satish Kharat 2025-02-27 84 *q_number = le16_to_cpu(desc->q_number_rss_type_flags) &
34d3abffbe44db Satish Kharat 2025-02-27 85 CQ_DESC_Q_NUM_MASK;
34d3abffbe44db Satish Kharat 2025-02-27 86 *completed_index = le16_to_cpu(desc->completed_index_flags) &
34d3abffbe44db Satish Kharat 2025-02-27 87 CQ_DESC_COMP_NDX_MASK;
180b3e15104ce9 Satish Kharat 2025-02-27 88 *completed_index |= (desc->fetch_index_flags & CQ_DESC_64_FI_MASK) <<
180b3e15104ce9 Satish Kharat 2025-02-27 89 CQ_DESC_COMP_NDX_BITS;
180b3e15104ce9 Satish Kharat 2025-02-27 90 break;
180b3e15104ce9 Satish Kharat 2025-02-27 91 }
180b3e15104ce9 Satish Kharat 2025-02-27 92 }
180b3e15104ce9 Satish Kharat 2025-02-27 93
34d3abffbe44db Satish Kharat 2025-02-27 @94 *color = (type_color >> CQ_DESC_COLOR_SHIFT) & CQ_DESC_COLOR_MASK;
34d3abffbe44db Satish Kharat 2025-02-27 95 *type = type_color & CQ_DESC_TYPE_MASK;
fe57762c649058 John Daley 2025-02-05 96 }
fe57762c649058 John Daley 2025-02-05 97
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 3+ messages in thread* [PATCH 0/8] enic:enable 32, 64 byte cqes and get max rx/tx ring size from hw
@ 2025-02-28 0:30 Satish Kharat
2025-02-28 0:30 ` Satish Kharat via B4 Relay
0 siblings, 1 reply; 3+ messages in thread
From: Satish Kharat @ 2025-02-28 0:30 UTC (permalink / raw)
To: Christian Benvenuti, Andrew Lunn, David S. Miller, Eric Dumazet,
Jakub Kicinski, Paolo Abeni
Cc: netdev, linux-kernel, Satish Kharat, Nelson Escobar, John Daley
This series enables using the max rx and tx ring sizes read from hw.
For newer hw that can be up to 16k entries. This requires bigger
completion entries for rx queues. This series enables the use of the
32 and 64 byte completion queues entries for enic rx queues on
supported hw versions. This is in addition to the exiting (default)
16 byte rx cqes.
Signed-off-by: Satish Kharat <satishkh@cisco.com>
---
Satish Kharat (8):
enic: Move function from header file to c file
enic: enic rq code reorg
enic: enic rq extended cq defines
enic: enable rq extended cq support
enic : remove unused function cq_enet_wq_desc_dec
enic : added enic_wq.c and enic_wq.h
enic : cleanup of enic wq request completion path
enic : get max rq & wq entries supported by hw, 16K queues
drivers/net/ethernet/cisco/enic/Makefile | 2 +-
drivers/net/ethernet/cisco/enic/cq_desc.h | 25 +--
drivers/net/ethernet/cisco/enic/cq_enet_desc.h | 142 ++++++--------
drivers/net/ethernet/cisco/enic/enic.h | 13 ++
drivers/net/ethernet/cisco/enic/enic_ethtool.c | 12 +-
drivers/net/ethernet/cisco/enic/enic_main.c | 69 +------
drivers/net/ethernet/cisco/enic/enic_res.c | 87 +++++++--
drivers/net/ethernet/cisco/enic/enic_res.h | 11 +-
drivers/net/ethernet/cisco/enic/enic_rq.c | 246 ++++++++++++++++++++++---
drivers/net/ethernet/cisco/enic/enic_rq.h | 6 +-
drivers/net/ethernet/cisco/enic/enic_wq.c | 121 ++++++++++++
drivers/net/ethernet/cisco/enic/enic_wq.h | 6 +
drivers/net/ethernet/cisco/enic/vnic_cq.h | 41 -----
drivers/net/ethernet/cisco/enic/vnic_devcmd.h | 19 ++
drivers/net/ethernet/cisco/enic/vnic_enet.h | 5 +
drivers/net/ethernet/cisco/enic/vnic_rq.h | 2 +-
drivers/net/ethernet/cisco/enic/vnic_wq.h | 2 +-
17 files changed, 545 insertions(+), 264 deletions(-)
---
base-commit: de7a88b639d488607352a270ef2e052c4442b1b3
change-id: 20250218-enic_cleanup_and_ext_cq-f21868bbde07
Best regards,
--
Satish Kharat <satishkh@cisco.com>
^ permalink raw reply [flat|nested] 3+ messages in thread* [PATCH 4/8] enic: enable rq extended cq support 2025-02-28 0:30 [PATCH 0/8] enic:enable 32, 64 byte cqes and get max rx/tx ring size from hw Satish Kharat @ 2025-02-28 0:30 ` Satish Kharat via B4 Relay 0 siblings, 0 replies; 3+ messages in thread From: Satish Kharat @ 2025-02-28 0:30 UTC (permalink / raw) To: Christian Benvenuti, Andrew Lunn, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni Cc: netdev, linux-kernel, Satish Kharat, Nelson Escobar, John Daley Enables getting from hw all the supported rq cq sizes and uses the highest supported cq size. Co-developed-by: Nelson Escobar <neescoba@cisco.com> Signed-off-by: Nelson Escobar <neescoba@cisco.com> Co-developed-by: John Daley <johndale@cisco.com> Signed-off-by: John Daley <johndale@cisco.com> Signed-off-by: Satish Kharat <satishkh@cisco.com> --- drivers/net/ethernet/cisco/enic/cq_desc.h | 3 + drivers/net/ethernet/cisco/enic/enic.h | 9 +++ drivers/net/ethernet/cisco/enic/enic_main.c | 4 ++ drivers/net/ethernet/cisco/enic/enic_res.c | 58 +++++++++++++++- drivers/net/ethernet/cisco/enic/enic_rq.c | 101 +++++++++++++++++++++------- 5 files changed, 149 insertions(+), 26 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/cq_desc.h b/drivers/net/ethernet/cisco/enic/cq_desc.h index 462c5435a206b4cc93b3734fdc96a2192b53a235..8fc313b6ed0434bd55b8e10bf3086ef848acbdf1 100644 --- a/drivers/net/ethernet/cisco/enic/cq_desc.h +++ b/drivers/net/ethernet/cisco/enic/cq_desc.h @@ -40,6 +40,9 @@ struct cq_desc { #define CQ_DESC_COMP_NDX_BITS 12 #define CQ_DESC_COMP_NDX_MASK ((1 << CQ_DESC_COMP_NDX_BITS) - 1) +#define CQ_DESC_32_FI_MASK (BIT(0) | BIT(1)) +#define CQ_DESC_64_FI_MASK (BIT(0) | BIT(1)) + static inline void cq_desc_dec(const struct cq_desc *desc_arg, u8 *type, u8 *color, u16 *q_number, u16 *completed_index) { diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h index 305ed12aa0311ca6cc53bfbffcc300182a8011a7..d60e55accafd0e4f83728524da4f167a474d6213 100644 --- a/drivers/net/ethernet/cisco/enic/enic.h +++ b/drivers/net/ethernet/cisco/enic/enic.h @@ -31,6 +31,13 @@ #define ENIC_AIC_LARGE_PKT_DIFF 3 +enum ext_cq { + ENIC_RQ_CQ_ENTRY_SIZE_16, + ENIC_RQ_CQ_ENTRY_SIZE_32, + ENIC_RQ_CQ_ENTRY_SIZE_64, + ENIC_RQ_CQ_ENTRY_SIZE_MAX, +}; + struct enic_msix_entry { int requested; char devname[IFNAMSIZ + 8]; @@ -228,6 +235,7 @@ struct enic { struct enic_rfs_flw_tbl rfs_h; u8 rss_key[ENIC_RSS_LEN]; struct vnic_gen_stats gen_stats; + enum ext_cq ext_cq; }; static inline struct net_device *vnic_get_netdev(struct vnic_dev *vdev) @@ -349,5 +357,6 @@ int enic_is_valid_vf(struct enic *enic, int vf); int enic_is_dynamic(struct enic *enic); void enic_set_ethtool_ops(struct net_device *netdev); int __enic_set_rsskey(struct enic *enic); +void enic_ext_cq(struct enic *enic); #endif /* _ENIC_H_ */ diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c index 080234ef4c2bb53c19e26601ca9bb38d26a738b7..d716514366dfc56b4e08260d18d78fddd23f6253 100644 --- a/drivers/net/ethernet/cisco/enic/enic_main.c +++ b/drivers/net/ethernet/cisco/enic/enic_main.c @@ -2192,6 +2192,7 @@ static void enic_reset(struct work_struct *work) enic_init_vnic_resources(enic); enic_set_rss_nic_cfg(enic); enic_dev_set_ig_vlan_rewrite_mode(enic); + enic_ext_cq(enic); enic_open(enic->netdev); /* Allow infiniband to fiddle with the device again */ @@ -2218,6 +2219,7 @@ static void enic_tx_hang_reset(struct work_struct *work) enic_init_vnic_resources(enic); enic_set_rss_nic_cfg(enic); enic_dev_set_ig_vlan_rewrite_mode(enic); + enic_ext_cq(enic); enic_open(enic->netdev); /* Allow infiniband to fiddle with the device again */ @@ -2592,6 +2594,8 @@ static int enic_dev_init(struct enic *enic) enic_get_res_counts(enic); + enic_ext_cq(enic); + err = enic_alloc_enic_resources(enic); if (err) { dev_err(dev, "Failed to allocate enic resources\n"); diff --git a/drivers/net/ethernet/cisco/enic/enic_res.c b/drivers/net/ethernet/cisco/enic/enic_res.c index 1261251998330c8b8363c4dd2db1ccc25847476c..a7179cc4b5296cfbce137c54a9e17e6b358a19ae 100644 --- a/drivers/net/ethernet/cisco/enic/enic_res.c +++ b/drivers/net/ethernet/cisco/enic/enic_res.c @@ -312,6 +312,7 @@ void enic_init_vnic_resources(struct enic *enic) int enic_alloc_vnic_resources(struct enic *enic) { enum vnic_dev_intr_mode intr_mode; + int rq_cq_desc_size; unsigned int i; int err; @@ -326,6 +327,24 @@ int enic_alloc_vnic_resources(struct enic *enic) intr_mode == VNIC_DEV_INTR_MODE_MSIX ? "MSI-X" : "unknown"); + switch (enic->ext_cq) { + case ENIC_RQ_CQ_ENTRY_SIZE_16: + rq_cq_desc_size = 16; + break; + case ENIC_RQ_CQ_ENTRY_SIZE_32: + rq_cq_desc_size = 32; + break; + case ENIC_RQ_CQ_ENTRY_SIZE_64: + rq_cq_desc_size = 64; + break; + default: + dev_err(enic_get_dev(enic), + "Unable to determine rq cq desc size: %d", + enic->ext_cq); + err = -ENODEV; + goto err_out; + } + /* Allocate queue resources */ @@ -348,8 +367,8 @@ int enic_alloc_vnic_resources(struct enic *enic) for (i = 0; i < enic->cq_count; i++) { if (i < enic->rq_count) err = vnic_cq_alloc(enic->vdev, &enic->cq[i], i, - enic->config.rq_desc_count, - sizeof(struct cq_enet_rq_desc)); + enic->config.rq_desc_count, + rq_cq_desc_size); else err = vnic_cq_alloc(enic->vdev, &enic->cq[i], i, enic->config.wq_desc_count, @@ -380,6 +399,39 @@ int enic_alloc_vnic_resources(struct enic *enic) err_out_cleanup: enic_free_vnic_resources(enic); - +err_out: return err; } + +/* + * CMD_CQ_ENTRY_SIZE_SET can fail on older hw generations that don't support + * that command + */ +void enic_ext_cq(struct enic *enic) +{ + u64 a0 = CMD_CQ_ENTRY_SIZE_SET, a1 = 0; + int wait = 1000; + int ret; + + spin_lock_bh(&enic->devcmd_lock); + ret = vnic_dev_cmd(enic->vdev, CMD_CAPABILITY, &a0, &a1, wait); + if (ret || a0) { + dev_info(&enic->pdev->dev, + "CMD_CQ_ENTRY_SIZE_SET not supported."); + enic->ext_cq = ENIC_RQ_CQ_ENTRY_SIZE_16; + goto out; + } + a1 &= VNIC_RQ_CQ_ENTRY_SIZE_ALL_BIT; + enic->ext_cq = fls(a1) - 1; + a0 = VNIC_RQ_ALL; + a1 = enic->ext_cq; + ret = vnic_dev_cmd(enic->vdev, CMD_CQ_ENTRY_SIZE_SET, &a0, &a1, wait); + if (ret) { + dev_info(&enic->pdev->dev, "CMD_CQ_ENTRY_SIZE_SET failed."); + enic->ext_cq = ENIC_RQ_CQ_ENTRY_SIZE_16; + } +out: + spin_unlock_bh(&enic->devcmd_lock); + dev_info(&enic->pdev->dev, "CQ entry size set to %d bytes", + 16 << enic->ext_cq); +} diff --git a/drivers/net/ethernet/cisco/enic/enic_rq.c b/drivers/net/ethernet/cisco/enic/enic_rq.c index 96eeb0e0d69602e8338630eab048dc960b161bbb..0013f98ccd0bfe006aea7b8a676a7872e345b8b6 100644 --- a/drivers/net/ethernet/cisco/enic/enic_rq.c +++ b/drivers/net/ethernet/cisco/enic/enic_rq.c @@ -21,24 +21,76 @@ static void enic_intr_update_pkt_size(struct vnic_rx_bytes_counter *pkt_size, pkt_size->small_pkt_bytes_cnt += pkt_len; } -static void enic_rq_cq_desc_dec(struct cq_enet_rq_desc *desc, u8 *type, +static void enic_rq_cq_desc_dec(void *cq_desc, u8 cq_desc_size, u8 *type, u8 *color, u16 *q_number, u16 *completed_index) { /* type_color is the last field for all cq structs */ - u8 type_color = desc->type_color; + u8 type_color; + + switch (cq_desc_size) { + case VNIC_RQ_CQ_ENTRY_SIZE_16: { + struct cq_enet_rq_desc *desc = + (struct cq_enet_rq_desc *)cq_desc; + type_color = desc->type_color; + + /* Make sure color bit is read from desc *before* other fields + * are read from desc. Hardware guarantees color bit is last + * bit (byte) written. Adding the rmb() prevents the compiler + * and/or CPU from reordering the reads which would potentially + * result in reading stale values. + */ + rmb(); - /* Make sure color bit is read from desc *before* other fields - * are read from desc. Hardware guarantees color bit is last - * bit (byte) written. Adding the rmb() prevents the compiler - * and/or CPU from reordering the reads which would potentially - * result in reading stale values. - */ - rmb(); + *q_number = le16_to_cpu(desc->q_number_rss_type_flags) & + CQ_DESC_Q_NUM_MASK; + *completed_index = le16_to_cpu(desc->completed_index_flags) & + CQ_DESC_COMP_NDX_MASK; + break; + } + case VNIC_RQ_CQ_ENTRY_SIZE_32: { + struct cq_enet_rq_desc_32 *desc = + (struct cq_enet_rq_desc_32 *)cq_desc; + type_color = desc->type_color; + + /* Make sure color bit is read from desc *before* other fields + * are read from desc. Hardware guarantees color bit is last + * bit (byte) written. Adding the rmb() prevents the compiler + * and/or CPU from reordering the reads which would potentially + * result in reading stale values. + */ + rmb(); + + *q_number = le16_to_cpu(desc->q_number_rss_type_flags) & + CQ_DESC_Q_NUM_MASK; + *completed_index = le16_to_cpu(desc->completed_index_flags) & + CQ_DESC_COMP_NDX_MASK; + *completed_index |= (desc->fetch_index_flags & CQ_DESC_32_FI_MASK) << + CQ_DESC_COMP_NDX_BITS; + break; + } + case VNIC_RQ_CQ_ENTRY_SIZE_64: { + struct cq_enet_rq_desc_64 *desc = + (struct cq_enet_rq_desc_64 *)cq_desc; + type_color = desc->type_color; + + /* Make sure color bit is read from desc *before* other fields + * are read from desc. Hardware guarantees color bit is last + * bit (byte) written. Adding the rmb() prevents the compiler + * and/or CPU from reordering the reads which would potentially + * result in reading stale values. + */ + rmb(); + + *q_number = le16_to_cpu(desc->q_number_rss_type_flags) & + CQ_DESC_Q_NUM_MASK; + *completed_index = le16_to_cpu(desc->completed_index_flags) & + CQ_DESC_COMP_NDX_MASK; + *completed_index |= (desc->fetch_index_flags & CQ_DESC_64_FI_MASK) << + CQ_DESC_COMP_NDX_BITS; + break; + } + } - *q_number = le16_to_cpu(desc->q_number_rss_type_flags) & - CQ_DESC_Q_NUM_MASK; - *completed_index = le16_to_cpu(desc->completed_index_flags) & - CQ_DESC_COMP_NDX_MASK; *color = (type_color >> CQ_DESC_COLOR_SHIFT) & CQ_DESC_COLOR_MASK; *type = type_color & CQ_DESC_TYPE_MASK; } @@ -113,6 +165,10 @@ static void enic_rq_set_skb_flags(struct vnic_rq *vrq, u8 type, u32 rss_hash, } } +/* + * cq_enet_rq_desc accesses section uses only the 1st 15 bytes of the cq which is identical + * for all type (16,32 and 64 byte) of cqs. + */ static inline void cq_enet_rq_desc_dec(struct cq_enet_rq_desc *desc, u8 *ingress_port, u8 *fcoe, u8 *eop, u8 *sop, u8 *rss_type, u8 *csum_not_calc, u32 *rss_hash, u16 *bytes_written, u8 *packet_error, @@ -256,7 +312,7 @@ void enic_free_rq_buf(struct vnic_rq *rq, struct vnic_rq_buf *buf) } static void enic_rq_indicate_buf(struct enic *enic, struct vnic_rq *rq, - struct vnic_rq_buf *buf, struct cq_enet_rq_desc *cq_desc, + struct vnic_rq_buf *buf, void *cq_desc, u8 type, u16 q_number, u16 completed_index) { struct sk_buff *skb; @@ -274,7 +330,7 @@ static void enic_rq_indicate_buf(struct enic *enic, struct vnic_rq *rq, rqstats->packets++; - cq_enet_rq_desc_dec(cq_desc, &ingress_port, + cq_enet_rq_desc_dec((struct cq_enet_rq_desc *)cq_desc, &ingress_port, &fcoe, &eop, &sop, &rss_type, &csum_not_calc, &rss_hash, &bytes_written, &packet_error, &vlan_stripped, &vlan_tci, &checksum, &fcoe_sof, @@ -326,7 +382,7 @@ static void enic_rq_indicate_buf(struct enic *enic, struct vnic_rq *rq, } } -static void enic_rq_service(struct enic *enic, struct cq_enet_rq_desc *cq_desc, u8 type, +static void enic_rq_service(struct enic *enic, void *cq_desc, u8 type, u16 q_number, u16 completed_index) { struct vnic_rq *vrq = &enic->rq[q_number].vrq; @@ -358,10 +414,9 @@ unsigned int enic_rq_cq_service(struct enic *enic, unsigned int cq_index, unsigned int work_done = 0; struct vnic_cq *cq = &enic->cq[cq_index]; - struct cq_enet_rq_desc *cq_desc = (struct cq_enet_rq_desc *)((u8 *)cq->ring.descs - + cq->ring.desc_size * cq->to_clean); + void *cq_desc = (u8 *)cq->ring.descs + cq->ring.desc_size * cq->to_clean; - enic_rq_cq_desc_dec(cq_desc, &type, &color, &q_number, + enic_rq_cq_desc_dec(cq_desc, enic->ext_cq, &type, &color, &q_number, &completed_index); while (color != cq->last_color) { @@ -377,10 +432,10 @@ unsigned int enic_rq_cq_service(struct enic *enic, unsigned int cq_index, if (++work_done >= work_to_do) break; - cq_desc = (struct cq_enet_rq_desc *)((u8 *)cq->ring.descs - + cq->ring.desc_size * cq->to_clean); - enic_rq_cq_desc_dec(cq_desc, &type, &color, &q_number, - &completed_index); + cq_desc = ((u8 *)cq->ring.descs + + cq->ring.desc_size * cq->to_clean); + enic_rq_cq_desc_dec(cq_desc, enic->ext_cq, &type, &color, + &q_number, &completed_index); } return work_done; -- 2.48.1 ^ permalink raw reply related [flat|nested] 3+ messages in thread
* [PATCH 4/8] enic: enable rq extended cq support @ 2025-02-28 0:30 ` Satish Kharat via B4 Relay 0 siblings, 0 replies; 3+ messages in thread From: Satish Kharat via B4 Relay @ 2025-02-28 0:30 UTC (permalink / raw) To: Christian Benvenuti, Andrew Lunn, David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni Cc: netdev, linux-kernel, Satish Kharat, Nelson Escobar, John Daley From: Satish Kharat <satishkh@cisco.com> Enables getting from hw all the supported rq cq sizes and uses the highest supported cq size. Co-developed-by: Nelson Escobar <neescoba@cisco.com> Signed-off-by: Nelson Escobar <neescoba@cisco.com> Co-developed-by: John Daley <johndale@cisco.com> Signed-off-by: John Daley <johndale@cisco.com> Signed-off-by: Satish Kharat <satishkh@cisco.com> --- drivers/net/ethernet/cisco/enic/cq_desc.h | 3 + drivers/net/ethernet/cisco/enic/enic.h | 9 +++ drivers/net/ethernet/cisco/enic/enic_main.c | 4 ++ drivers/net/ethernet/cisco/enic/enic_res.c | 58 +++++++++++++++- drivers/net/ethernet/cisco/enic/enic_rq.c | 101 +++++++++++++++++++++------- 5 files changed, 149 insertions(+), 26 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/cq_desc.h b/drivers/net/ethernet/cisco/enic/cq_desc.h index 462c5435a206b4cc93b3734fdc96a2192b53a235..8fc313b6ed0434bd55b8e10bf3086ef848acbdf1 100644 --- a/drivers/net/ethernet/cisco/enic/cq_desc.h +++ b/drivers/net/ethernet/cisco/enic/cq_desc.h @@ -40,6 +40,9 @@ struct cq_desc { #define CQ_DESC_COMP_NDX_BITS 12 #define CQ_DESC_COMP_NDX_MASK ((1 << CQ_DESC_COMP_NDX_BITS) - 1) +#define CQ_DESC_32_FI_MASK (BIT(0) | BIT(1)) +#define CQ_DESC_64_FI_MASK (BIT(0) | BIT(1)) + static inline void cq_desc_dec(const struct cq_desc *desc_arg, u8 *type, u8 *color, u16 *q_number, u16 *completed_index) { diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h index 305ed12aa0311ca6cc53bfbffcc300182a8011a7..d60e55accafd0e4f83728524da4f167a474d6213 100644 --- a/drivers/net/ethernet/cisco/enic/enic.h +++ b/drivers/net/ethernet/cisco/enic/enic.h @@ -31,6 +31,13 @@ #define ENIC_AIC_LARGE_PKT_DIFF 3 +enum ext_cq { + ENIC_RQ_CQ_ENTRY_SIZE_16, + ENIC_RQ_CQ_ENTRY_SIZE_32, + ENIC_RQ_CQ_ENTRY_SIZE_64, + ENIC_RQ_CQ_ENTRY_SIZE_MAX, +}; + struct enic_msix_entry { int requested; char devname[IFNAMSIZ + 8]; @@ -228,6 +235,7 @@ struct enic { struct enic_rfs_flw_tbl rfs_h; u8 rss_key[ENIC_RSS_LEN]; struct vnic_gen_stats gen_stats; + enum ext_cq ext_cq; }; static inline struct net_device *vnic_get_netdev(struct vnic_dev *vdev) @@ -349,5 +357,6 @@ int enic_is_valid_vf(struct enic *enic, int vf); int enic_is_dynamic(struct enic *enic); void enic_set_ethtool_ops(struct net_device *netdev); int __enic_set_rsskey(struct enic *enic); +void enic_ext_cq(struct enic *enic); #endif /* _ENIC_H_ */ diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c index 080234ef4c2bb53c19e26601ca9bb38d26a738b7..d716514366dfc56b4e08260d18d78fddd23f6253 100644 --- a/drivers/net/ethernet/cisco/enic/enic_main.c +++ b/drivers/net/ethernet/cisco/enic/enic_main.c @@ -2192,6 +2192,7 @@ static void enic_reset(struct work_struct *work) enic_init_vnic_resources(enic); enic_set_rss_nic_cfg(enic); enic_dev_set_ig_vlan_rewrite_mode(enic); + enic_ext_cq(enic); enic_open(enic->netdev); /* Allow infiniband to fiddle with the device again */ @@ -2218,6 +2219,7 @@ static void enic_tx_hang_reset(struct work_struct *work) enic_init_vnic_resources(enic); enic_set_rss_nic_cfg(enic); enic_dev_set_ig_vlan_rewrite_mode(enic); + enic_ext_cq(enic); enic_open(enic->netdev); /* Allow infiniband to fiddle with the device again */ @@ -2592,6 +2594,8 @@ static int enic_dev_init(struct enic *enic) enic_get_res_counts(enic); + enic_ext_cq(enic); + err = enic_alloc_enic_resources(enic); if (err) { dev_err(dev, "Failed to allocate enic resources\n"); diff --git a/drivers/net/ethernet/cisco/enic/enic_res.c b/drivers/net/ethernet/cisco/enic/enic_res.c index 1261251998330c8b8363c4dd2db1ccc25847476c..a7179cc4b5296cfbce137c54a9e17e6b358a19ae 100644 --- a/drivers/net/ethernet/cisco/enic/enic_res.c +++ b/drivers/net/ethernet/cisco/enic/enic_res.c @@ -312,6 +312,7 @@ void enic_init_vnic_resources(struct enic *enic) int enic_alloc_vnic_resources(struct enic *enic) { enum vnic_dev_intr_mode intr_mode; + int rq_cq_desc_size; unsigned int i; int err; @@ -326,6 +327,24 @@ int enic_alloc_vnic_resources(struct enic *enic) intr_mode == VNIC_DEV_INTR_MODE_MSIX ? "MSI-X" : "unknown"); + switch (enic->ext_cq) { + case ENIC_RQ_CQ_ENTRY_SIZE_16: + rq_cq_desc_size = 16; + break; + case ENIC_RQ_CQ_ENTRY_SIZE_32: + rq_cq_desc_size = 32; + break; + case ENIC_RQ_CQ_ENTRY_SIZE_64: + rq_cq_desc_size = 64; + break; + default: + dev_err(enic_get_dev(enic), + "Unable to determine rq cq desc size: %d", + enic->ext_cq); + err = -ENODEV; + goto err_out; + } + /* Allocate queue resources */ @@ -348,8 +367,8 @@ int enic_alloc_vnic_resources(struct enic *enic) for (i = 0; i < enic->cq_count; i++) { if (i < enic->rq_count) err = vnic_cq_alloc(enic->vdev, &enic->cq[i], i, - enic->config.rq_desc_count, - sizeof(struct cq_enet_rq_desc)); + enic->config.rq_desc_count, + rq_cq_desc_size); else err = vnic_cq_alloc(enic->vdev, &enic->cq[i], i, enic->config.wq_desc_count, @@ -380,6 +399,39 @@ int enic_alloc_vnic_resources(struct enic *enic) err_out_cleanup: enic_free_vnic_resources(enic); - +err_out: return err; } + +/* + * CMD_CQ_ENTRY_SIZE_SET can fail on older hw generations that don't support + * that command + */ +void enic_ext_cq(struct enic *enic) +{ + u64 a0 = CMD_CQ_ENTRY_SIZE_SET, a1 = 0; + int wait = 1000; + int ret; + + spin_lock_bh(&enic->devcmd_lock); + ret = vnic_dev_cmd(enic->vdev, CMD_CAPABILITY, &a0, &a1, wait); + if (ret || a0) { + dev_info(&enic->pdev->dev, + "CMD_CQ_ENTRY_SIZE_SET not supported."); + enic->ext_cq = ENIC_RQ_CQ_ENTRY_SIZE_16; + goto out; + } + a1 &= VNIC_RQ_CQ_ENTRY_SIZE_ALL_BIT; + enic->ext_cq = fls(a1) - 1; + a0 = VNIC_RQ_ALL; + a1 = enic->ext_cq; + ret = vnic_dev_cmd(enic->vdev, CMD_CQ_ENTRY_SIZE_SET, &a0, &a1, wait); + if (ret) { + dev_info(&enic->pdev->dev, "CMD_CQ_ENTRY_SIZE_SET failed."); + enic->ext_cq = ENIC_RQ_CQ_ENTRY_SIZE_16; + } +out: + spin_unlock_bh(&enic->devcmd_lock); + dev_info(&enic->pdev->dev, "CQ entry size set to %d bytes", + 16 << enic->ext_cq); +} diff --git a/drivers/net/ethernet/cisco/enic/enic_rq.c b/drivers/net/ethernet/cisco/enic/enic_rq.c index 96eeb0e0d69602e8338630eab048dc960b161bbb..0013f98ccd0bfe006aea7b8a676a7872e345b8b6 100644 --- a/drivers/net/ethernet/cisco/enic/enic_rq.c +++ b/drivers/net/ethernet/cisco/enic/enic_rq.c @@ -21,24 +21,76 @@ static void enic_intr_update_pkt_size(struct vnic_rx_bytes_counter *pkt_size, pkt_size->small_pkt_bytes_cnt += pkt_len; } -static void enic_rq_cq_desc_dec(struct cq_enet_rq_desc *desc, u8 *type, +static void enic_rq_cq_desc_dec(void *cq_desc, u8 cq_desc_size, u8 *type, u8 *color, u16 *q_number, u16 *completed_index) { /* type_color is the last field for all cq structs */ - u8 type_color = desc->type_color; + u8 type_color; + + switch (cq_desc_size) { + case VNIC_RQ_CQ_ENTRY_SIZE_16: { + struct cq_enet_rq_desc *desc = + (struct cq_enet_rq_desc *)cq_desc; + type_color = desc->type_color; + + /* Make sure color bit is read from desc *before* other fields + * are read from desc. Hardware guarantees color bit is last + * bit (byte) written. Adding the rmb() prevents the compiler + * and/or CPU from reordering the reads which would potentially + * result in reading stale values. + */ + rmb(); - /* Make sure color bit is read from desc *before* other fields - * are read from desc. Hardware guarantees color bit is last - * bit (byte) written. Adding the rmb() prevents the compiler - * and/or CPU from reordering the reads which would potentially - * result in reading stale values. - */ - rmb(); + *q_number = le16_to_cpu(desc->q_number_rss_type_flags) & + CQ_DESC_Q_NUM_MASK; + *completed_index = le16_to_cpu(desc->completed_index_flags) & + CQ_DESC_COMP_NDX_MASK; + break; + } + case VNIC_RQ_CQ_ENTRY_SIZE_32: { + struct cq_enet_rq_desc_32 *desc = + (struct cq_enet_rq_desc_32 *)cq_desc; + type_color = desc->type_color; + + /* Make sure color bit is read from desc *before* other fields + * are read from desc. Hardware guarantees color bit is last + * bit (byte) written. Adding the rmb() prevents the compiler + * and/or CPU from reordering the reads which would potentially + * result in reading stale values. + */ + rmb(); + + *q_number = le16_to_cpu(desc->q_number_rss_type_flags) & + CQ_DESC_Q_NUM_MASK; + *completed_index = le16_to_cpu(desc->completed_index_flags) & + CQ_DESC_COMP_NDX_MASK; + *completed_index |= (desc->fetch_index_flags & CQ_DESC_32_FI_MASK) << + CQ_DESC_COMP_NDX_BITS; + break; + } + case VNIC_RQ_CQ_ENTRY_SIZE_64: { + struct cq_enet_rq_desc_64 *desc = + (struct cq_enet_rq_desc_64 *)cq_desc; + type_color = desc->type_color; + + /* Make sure color bit is read from desc *before* other fields + * are read from desc. Hardware guarantees color bit is last + * bit (byte) written. Adding the rmb() prevents the compiler + * and/or CPU from reordering the reads which would potentially + * result in reading stale values. + */ + rmb(); + + *q_number = le16_to_cpu(desc->q_number_rss_type_flags) & + CQ_DESC_Q_NUM_MASK; + *completed_index = le16_to_cpu(desc->completed_index_flags) & + CQ_DESC_COMP_NDX_MASK; + *completed_index |= (desc->fetch_index_flags & CQ_DESC_64_FI_MASK) << + CQ_DESC_COMP_NDX_BITS; + break; + } + } - *q_number = le16_to_cpu(desc->q_number_rss_type_flags) & - CQ_DESC_Q_NUM_MASK; - *completed_index = le16_to_cpu(desc->completed_index_flags) & - CQ_DESC_COMP_NDX_MASK; *color = (type_color >> CQ_DESC_COLOR_SHIFT) & CQ_DESC_COLOR_MASK; *type = type_color & CQ_DESC_TYPE_MASK; } @@ -113,6 +165,10 @@ static void enic_rq_set_skb_flags(struct vnic_rq *vrq, u8 type, u32 rss_hash, } } +/* + * cq_enet_rq_desc accesses section uses only the 1st 15 bytes of the cq which is identical + * for all type (16,32 and 64 byte) of cqs. + */ static inline void cq_enet_rq_desc_dec(struct cq_enet_rq_desc *desc, u8 *ingress_port, u8 *fcoe, u8 *eop, u8 *sop, u8 *rss_type, u8 *csum_not_calc, u32 *rss_hash, u16 *bytes_written, u8 *packet_error, @@ -256,7 +312,7 @@ void enic_free_rq_buf(struct vnic_rq *rq, struct vnic_rq_buf *buf) } static void enic_rq_indicate_buf(struct enic *enic, struct vnic_rq *rq, - struct vnic_rq_buf *buf, struct cq_enet_rq_desc *cq_desc, + struct vnic_rq_buf *buf, void *cq_desc, u8 type, u16 q_number, u16 completed_index) { struct sk_buff *skb; @@ -274,7 +330,7 @@ static void enic_rq_indicate_buf(struct enic *enic, struct vnic_rq *rq, rqstats->packets++; - cq_enet_rq_desc_dec(cq_desc, &ingress_port, + cq_enet_rq_desc_dec((struct cq_enet_rq_desc *)cq_desc, &ingress_port, &fcoe, &eop, &sop, &rss_type, &csum_not_calc, &rss_hash, &bytes_written, &packet_error, &vlan_stripped, &vlan_tci, &checksum, &fcoe_sof, @@ -326,7 +382,7 @@ static void enic_rq_indicate_buf(struct enic *enic, struct vnic_rq *rq, } } -static void enic_rq_service(struct enic *enic, struct cq_enet_rq_desc *cq_desc, u8 type, +static void enic_rq_service(struct enic *enic, void *cq_desc, u8 type, u16 q_number, u16 completed_index) { struct vnic_rq *vrq = &enic->rq[q_number].vrq; @@ -358,10 +414,9 @@ unsigned int enic_rq_cq_service(struct enic *enic, unsigned int cq_index, unsigned int work_done = 0; struct vnic_cq *cq = &enic->cq[cq_index]; - struct cq_enet_rq_desc *cq_desc = (struct cq_enet_rq_desc *)((u8 *)cq->ring.descs - + cq->ring.desc_size * cq->to_clean); + void *cq_desc = (u8 *)cq->ring.descs + cq->ring.desc_size * cq->to_clean; - enic_rq_cq_desc_dec(cq_desc, &type, &color, &q_number, + enic_rq_cq_desc_dec(cq_desc, enic->ext_cq, &type, &color, &q_number, &completed_index); while (color != cq->last_color) { @@ -377,10 +432,10 @@ unsigned int enic_rq_cq_service(struct enic *enic, unsigned int cq_index, if (++work_done >= work_to_do) break; - cq_desc = (struct cq_enet_rq_desc *)((u8 *)cq->ring.descs - + cq->ring.desc_size * cq->to_clean); - enic_rq_cq_desc_dec(cq_desc, &type, &color, &q_number, - &completed_index); + cq_desc = ((u8 *)cq->ring.descs + + cq->ring.desc_size * cq->to_clean); + enic_rq_cq_desc_dec(cq_desc, enic->ext_cq, &type, &color, + &q_number, &completed_index); } return work_done; -- 2.48.1 ^ permalink raw reply related [flat|nested] 3+ messages in thread
end of thread, other threads:[~2025-03-02 14:29 UTC | newest] Thread overview: 3+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2025-03-02 14:29 [PATCH 4/8] enic: enable rq extended cq support kernel test robot -- strict thread matches above, loose matches on Subject: below -- 2025-02-28 0:30 [PATCH 0/8] enic:enable 32, 64 byte cqes and get max rx/tx ring size from hw Satish Kharat 2025-02-28 0:30 ` [PATCH 4/8] enic: enable rq extended cq support Satish Kharat 2025-02-28 0:30 ` Satish Kharat via B4 Relay
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.