[PATCH net-next V2 00/10] net/mlx5e: Use multiple doorbells

[Tariq Toukan <tariqt@nvidia.com>]

Hi,

This series by Cosmin adds multiple doorbells usage in mlx5e driver.
See detailed description by Cosmin below [1].

Find V1 here:
https://lore.kernel.org/all/1757499891-596641-1-git-send-email-tariqt@nvidia.com/

Regards,
Tariq

V1->V2:
- added numbers to cover letter.
- fixed mlx5.rst nested list error.
- removed newline from NL_SET_ERR_MSG_FMT_MOD.

[1]
mlx5e uses a single MMIO-mapped doorbell per netdevice for all send and
receive operations. Writes to the doorbell go over the PCIe bus directly
to the device, which then services the indicated queues.

On certain architectures and with sufficiently high volume of doorbell
ringing (many cores, many active channels, small MTU, no GSO, etc.), the
MMIO-mapped doorbell address can become contended, leading to delays in
servicing writes to that address and a global slowdown of all traffic
for that netdevice.

mlx5 NICs have supported using multiple doorbells for many years, the
mlx5_ib driver for the same hardware has been using multiple doorbells
traditionally.

This patch series extends the mlx5 Ethernet driver to also use multiple
doorbells to solve the MMIO contention issues. By allocating and using
more doorbells for all channel queues (TX and RX), the MMIO contention
on any particular doorbell address is reduced significantly.

The first patches are cleanups:
net/mlx5: Fix typo of MLX5_EQ_DOORBEL_OFFSET
net/mlx5: Remove unused 'offset' field from struct mlx5_sq_bfreg'
net/mlx5e: Remove unused 'xsk' param of mlx5e_build_xdpsq_param

The next patch separates the global doorbell from Ethernet-specific
resources:
net/mlx5: Store the global doorbell in mlx5_priv

Next, plumbing to allow a different doorbell to be used for channel TX
and RX queues:
net/mlx5e: Prepare for using multiple TX doorbells
net/mlx5e: Prepare for using different CQ doorbells

Then, enable using multiple doorbells for channel queues:
net/mlx5e: Use multiple TX doorbells
net/mlx5e: Use multiple CQ doorbells

Finally, introduce a devlink parameter to control this:
devlink: Add a 'num_doorbells' driverinit param
net/mlx5e: Use the 'num_doorbells' devlink param

Some performance results, done with the Linux pktgen script, running b2b
over Connect-X 8 NICs:
samples/pktgen/pktgen_sample02_multiqueue.sh -i $NIC -s 64 -d $DST_IP \
  -m $MAC -t 64

Baseline (1 doorbell): 9 Mpps
This series (8 doorbells): 56 Mpps

Note that pktgen without 'burst' rings the doorbell after every packet,
while real packet TX using NAPI usually batches multiple pending packets
with the xmit_more mechanism. So this is in essence a micro-benchmark
showcasing the improvement of using multiple doorbells on platforms
affected by MMIO contention. Real life traffic usually sees little
movement either way.


Cosmin Ratiu (10):
  net/mlx5: Fix typo of MLX5_EQ_DOORBEL_OFFSET
  net/mlx5: Remove unused 'offset' field from mlx5_sq_bfreg
  net/mlx5e: Remove unused 'xsk' param of mlx5e_build_xdpsq_param
  net/mlx5: Store the global doorbell in mlx5_priv
  net/mlx5e: Prepare for using multiple TX doorbells
  net/mlx5e: Prepare for using different CQ doorbells
  net/mlx5e: Use multiple TX doorbells
  net/mlx5e: Use multiple CQ doorbells
  devlink: Add a 'num_doorbells' driverinit param
  net/mlx5e: Use the 'num_doorbells' devlink param

 .../networking/devlink/devlink-params.rst     |  3 ++
 Documentation/networking/devlink/mlx5.rst     |  9 ++++
 drivers/infiniband/hw/mlx5/cq.c               |  4 +-
 drivers/net/ethernet/mellanox/mlx5/core/cq.c  |  1 -
 .../net/ethernet/mellanox/mlx5/core/devlink.c | 26 +++++++++++
 drivers/net/ethernet/mellanox/mlx5/core/en.h  |  3 ++
 .../ethernet/mellanox/mlx5/core/en/params.c   |  6 +--
 .../ethernet/mellanox/mlx5/core/en/params.h   |  2 +-
 .../net/ethernet/mellanox/mlx5/core/en/ptp.c  |  6 ++-
 .../net/ethernet/mellanox/mlx5/core/en/ptp.h  |  1 +
 .../net/ethernet/mellanox/mlx5/core/en/trap.c |  1 +
 .../net/ethernet/mellanox/mlx5/core/en/txrx.h |  5 +--
 .../mellanox/mlx5/core/en/xsk/setup.c         |  2 +-
 .../ethernet/mellanox/mlx5/core/en_common.c   | 45 ++++++++++++++++---
 .../net/ethernet/mellanox/mlx5/core/en_main.c | 37 ++++++++++++---
 drivers/net/ethernet/mellanox/mlx5/core/eq.c  |  8 ++--
 .../ethernet/mellanox/mlx5/core/fpga/conn.c   |  1 -
 .../net/ethernet/mellanox/mlx5/core/lib/aso.c |  8 ++--
 .../net/ethernet/mellanox/mlx5/core/main.c    | 11 +++--
 .../mellanox/mlx5/core/steering/hws/send.c    |  8 ++--
 .../mellanox/mlx5/core/steering/sws/dr_send.c |  1 -
 drivers/net/ethernet/mellanox/mlx5/core/wc.c  | 16 ++++---
 include/linux/mlx5/cq.h                       |  1 -
 include/linux/mlx5/driver.h                   |  8 ++--
 include/net/devlink.h                         |  4 ++
 net/devlink/param.c                           |  5 +++
 26 files changed, 163 insertions(+), 59 deletions(-)


base-commit: a4ab91f470c5c84ce0c17197c1562d29aa032340
-- 
2.31.1