From: Xuanqiang Luo <xuanqiang.luo@linux.dev>
To: netdev@vger.kernel.org
Cc: saeedm@nvidia.com, leon@kernel.org, tariqt@nvidia.com,
mbloch@nvidia.com, andrew+netdev@lunn.ch, davem@davemloft.net,
edumazet@kernel.org, kuba@kernel.org, pabeni@redhat.com,
linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org,
Xuanqiang Luo <luoxuanqiang@kylinos.cn>
Subject: [PATCH net-next v1] net/mlx5e: TX, Do not open a multi-packet WQE for a single packet
Date: Tue, 17 Nov 2026 17:50:36 +0900 [thread overview]
Message-ID: <20261117085036.3138779-1-xuanqiang.luo@linux.dev> (raw)
From: Xuanqiang Luo <luoxuanqiang@kylinos.cn>
A multi-packet WQE (MPWQE, added by commit 5af75c747e2a ("net/mlx5e:
Enhanced TX MPWQE for SKBs")) lets several packets share one WQE, so the
control and Ethernet segments are fetched by the device once per batch
instead of once per packet. This saves PCIe bandwidth and CPU cycles, but
only if the WQE carries more than one packet.
With skb_tx_mpwqe enabled, which is the default, mlx5e_xmit() sends every
skb accepted by mlx5e_tx_skb_supports_mpwqe() through an MPWQE session.
A session stays open between two calls only while the stack sets
xmit_more. A packet without xmit_more closes the session and rings the
doorbell. Hence a packet that comes without xmit_more while no session is
open opens a session just for itself and closes it right away. Such a WQE
takes one WQEBB, as a regular WQE does for the same packet, and the
doorbell is rung for each packet either way, so nothing is saved.
This is the case for the IP fragments of a large UDP datagram and for
UDP request/response traffic, which reach the driver one packet at a
time. With pktgen and burst 1 every MPWQE carries one packet (mpwqe_blks
equals mpwqe_pkts, xmit_more is 0) and the packet rate is 15% lower than
with skb_tx_mpwqe turned off.
Use the regular WQE path for a packet that has no xmit_more and no open
session. Packets with xmit_more still open and extend a session, and the
last packet of a batch still closes it, so batching is unchanged.
The only visible change besides the performance is in the statistics:
tx*_mpwqe_blks and tx*_mpwqe_pkts no longer count a packet sent alone.
Such a packet is counted in tx*_packets, as any packet sent with a
regular WQE.
pktgen, 64 B packets on one TX queue, between two 10G ConnectX-5 hosts,
2-socket Hygon C86-4G 7490 sender on net-next, the patch switched at run
time by a temporary module parameter (not part of this patch) so that
before and after run in the same boot, mean of 3 boots x 3 rounds:
test before after change
pktgen 64 B burst 1, kpps 1494 1740 +16.5%
pktgen 64 B burst 16, kpps 3236 3236 +0.0%
Batches do not regress: with burst 16, which uses xmit_more, the rate is
unchanged and 15% above skb_tx_mpwqe off, so the gain of MPWQE is kept.
The price is that the regular path runs a little more code for a packet
than a single packet MPWQE does (mlx5e_sq_calc_wqe_attr(),
mlx5e_accel_tx_finish()); the packet rate above shows that it is more
than compensated.
Signed-off-by: Xuanqiang Luo <luoxuanqiang@kylinos.cn>
---
pktgen, 64 B UDP packets, one TX queue, 30M packets per run, sender and
IRQs pinned, interrupt coalescing fixed, before -> after, in pps:
burst 1 (no xmit_more) 1493583 -> 1739855 (+16.5%)
burst 4 2578336 -> 2576508 (-0.1%)
burst 16 3235640 -> 3235746 (+0.0%)
After vs before, 3 boots, paired t-test over the boots (2 degrees of
freedom, so a low power), the change of each boot in brackets:
- burst 1: +16.5% (p<0.001) [+16.2 +15.9 +17.3]
- burst 4: -0.1% (p=0.86) [+0.6 -0.4 -0.5]
- burst 16: +0.0% (p=0.99) [+0.3 -0.0 -0.3]
With the patch, burst 1 is within 1% of skb_tx_mpwqe turned off (1752020
pps). With burst 4 and 16 turning it off is 1% and 13% slower (2550781
and 2810510 pps), so the gain of MPWQE for batches is kept.
Single packet MPWQEs, counted with tx*_mpwqe_blks and tx*_mpwqe_pkts:
with burst 1, before the patch mpwqe_blks and mpwqe_pkts are both
29993000 for 29993000 packets, after the patch they are 0. With burst 4
(7519017 blocks, 29972000 packets) and burst 16 (1984750 blocks, 29888000
packets) the counters are the same before and after.
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
index 14479da98f42..7c1fcd8fb294 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
@@ -646,6 +646,7 @@ netdev_tx_t mlx5e_xmit(struct sk_buff *skb, struct net_device *dev)
{
struct mlx5e_priv *priv = netdev_priv(dev);
struct mlx5e_accel_tx_state accel = {};
+ bool xmit_more = netdev_xmit_more();
struct mlx5e_tx_wqe_attr wqe_attr;
struct mlx5e_tx_attr attr;
struct mlx5e_tx_wqe *wqe;
@@ -678,11 +679,13 @@ netdev_tx_t mlx5e_xmit(struct sk_buff *skb, struct net_device *dev)
mlx5e_sq_xmit_prepare(sq, skb, &accel, &attr);
if (test_bit(MLX5E_SQ_STATE_MPWQE, &sq->state)) {
- if (mlx5e_tx_skb_supports_mpwqe(skb, &attr)) {
+ /* An MPWQE for a lone packet saves nothing, use a regular WQE. */
+ if (mlx5e_tx_skb_supports_mpwqe(skb, &attr) &&
+ (xmit_more || mlx5e_tx_mpwqe_session_is_active(sq))) {
struct mlx5_wqe_eth_seg eseg = {};
mlx5e_txwqe_build_eseg(priv, sq, skb, &accel, &eseg, attr.ihs);
- mlx5e_sq_xmit_mpwqe(sq, skb, &eseg, netdev_xmit_more());
+ mlx5e_sq_xmit_mpwqe(sq, skb, &eseg, xmit_more);
return NETDEV_TX_OK;
}
@@ -697,7 +700,7 @@ netdev_tx_t mlx5e_xmit(struct sk_buff *skb, struct net_device *dev)
mlx5e_accel_tx_finish(sq, wqe, &accel,
(struct mlx5_wqe_inline_seg *)(wqe->data + wqe_attr.ds_cnt_inl));
mlx5e_txwqe_build_eseg(priv, sq, skb, &accel, &wqe->eth, attr.ihs);
- mlx5e_sq_xmit_wqe(sq, skb, &attr, &wqe_attr, wqe, pi, netdev_xmit_more());
+ mlx5e_sq_xmit_wqe(sq, skb, &attr, &wqe_attr, wqe, pi, xmit_more);
return NETDEV_TX_OK;
}
--
2.43.0
reply other threads:[~2026-11-17 8:50 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261117085036.3138779-1-xuanqiang.luo@linux.dev \
--to=xuanqiang.luo@linux.dev \
--cc=andrew+netdev@lunn.ch \
--cc=davem@davemloft.net \
--cc=edumazet@kernel.org \
--cc=kuba@kernel.org \
--cc=leon@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=luoxuanqiang@kylinos.cn \
--cc=mbloch@nvidia.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=saeedm@nvidia.com \
--cc=tariqt@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox