Re: [PATCH net-next] net/mlx5e: bound TX CQ poll softirq residency with a time budget

["Jose Fernandez (Anthropic)" <jose.fernandez@linux.dev>]

Hi Dragos,

Thanks for the review.

On Mon, Jul 06, 2026 at 01:48:41PM +0200, Dragos Tatulea wrote:
> This is interesting. I can think of 2 cases here which can overlap:
>
> 1) Packets are sent with xmit_more which trigger this many WQEs per
>   CQE scenario.
>
> 2) TSO packets can have many fragments so even a single WQE will have
>    more than one DMA unmap. For the default GSO size of 64K each TSO
>    skb will have 2 fragments on your platform. If the GSO is bumped to
>    256K then you usually get ~5 fragments.
>
> Let's first understand if this is a case of 1) (small packets sent
> with xmit_more) or 2) (many TSO packets) or 1+2) (many TSO packets sent
> with xmit_more).

Likely both, mostly 1). The workload is bursty bulk transfers, and
the stalls hit while processing a burst's completions: one CQE covers
a whole xmit_more batch, each TSO skb adds its 2 frag unmaps, so the
CQE budget bounds entries but not the unmap work behind them.

I don't have the tx_xmit_more / tx_tso_packets ratios handy. I'll
collect them across a burst window and report back.

> If xmit_more is used then you can actually tune the BQL to avoid this
> behavior which seems to wreak havoc on your configuration.

We considered that. BQL bounds bytes in flight, but bytes only
translate into poll time at some assumed per-unmap cost, and that
cost is the unstable part here: a few us normally, ~100x worse when
many CPUs contend on the shared SMMU command queue. A BQL setting
small enough for the worst case would hurt throughput the rest of
the time.

> The bounded poll is faster because it is interrupted earlier or is
> it faster because it results in less contention on the IOMMU?

Both, and we tried to separate them. Sweeping the yield cadence
barely moves the numbers, so it is not just doing less per poll. And
in A/B runs with the budget armed, the competing side's per-unmap
cost drops 15-20%, so there is a real contention reduction on top of
the transmit path getting the core back sooner.

> This change is too invasive in the driver for all the other cases. We have to figure
> another way to go about this issue.

Fair enough, and the module parameter should not have been there anyway.
The direction I would rather take: fix the cost instead of capping it,
by converting the mlx5e TX path to the tso_dma_map helpers (as bnxt
did), so a TSO skb maps as one IOVA range where the DMA IOVA API is
available, completing with one sync instead of one per frag.

Once the counters confirm the frag-unmap volume I will prototype that,
unless you or Tariq/Saeed see a problem with the direction. Happy to
test on the affected boxes. We have a repro harness.

Thanks,
Jose