Re: AVX512 block copy performance?

[Bruce Richardson <bruce.richardson@intel.com>]

On Mon, Jul 20, 2026 at 12:10:34PM +0200, Morten Brørup wrote:
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Leyi Rong
> > Sent: Thursday, 14 January 2021 07.40
> > 
> > Optimize Tx path by using AVX512 instructions and vectorize the
> > tx free bufs process.
> > 
> > Signed-off-by: Leyi Rong <leyi.rong@intel.com>
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > ---
> 
> [...]
> 
> > +static __rte_always_inline int
> > +i40e_tx_free_bufs_avx512(struct i40e_tx_queue *txq)
> > +{
> > +	struct i40e_vec_tx_entry *txep;
> > +	uint32_t n;
> > +	uint32_t i;
> > +	int nb_free = 0;
> > +	struct rte_mbuf *m, *free[RTE_I40E_TX_MAX_FREE_BUF_SZ];
> > +
> > +	/* check DD bits on threshold descriptor */
> > +	if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz &
> > +			rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) !=
> > +			rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE))
> > +		return 0;
> > +
> > +	n = txq->tx_rs_thresh;
> > +
> > +	 /* first buffer to free from S/W ring is at index
> > +	  * tx_next_dd - (tx_rs_thresh-1)
> > +	  */
> > +	txep = (void *)txq->sw_ring;
> > +	txep += txq->tx_next_dd - (n - 1);
> > +
> > +	if (txq->offloads & DEV_TX_OFFLOAD_MBUF_FAST_FREE && (n & 31) ==
> > 0) {
> > +		struct rte_mempool *mp = txep[0].mbuf->pool;
> > +		void **cache_objs;
> > +		struct rte_mempool_cache *cache =
> > rte_mempool_default_cache(mp,
> > +				rte_lcore_id());
> > +
> > +		if (!cache || cache->len == 0)
> > +			goto normal;
> > +
> > +		cache_objs = &cache->objs[cache->len];
> > +
> > +		if (n > RTE_MEMPOOL_CACHE_MAX_SIZE) {
> > +			rte_mempool_ops_enqueue_bulk(mp, (void *)txep, n);
> > +			goto done;
> > +		}
> > +
> > +		/* The cache follows the following algorithm
> > +		 *   1. Add the objects to the cache
> > +		 *   2. Anything greater than the cache min value (if it
> > +		 *   crosses the cache flush threshold) is flushed to the
> > ring.
> > +		 */
> > +		/* Add elements back into the cache */
> > +		uint32_t copied = 0;
> > +		/* n is multiple of 32 */
> > +		while (copied < n) {
> > +			const __m512i a = _mm512_load_si512(&txep[copied]);
> > +			const __m512i b = _mm512_load_si512(&txep[copied +
> > 8]);
> > +			const __m512i c = _mm512_load_si512(&txep[copied +
> > 16]);
> > +			const __m512i d = _mm512_load_si512(&txep[copied +
> > 24]);
> > +
> > +			_mm512_storeu_si512(&cache_objs[copied], a);
> > +			_mm512_storeu_si512(&cache_objs[copied + 8], b);
> > +			_mm512_storeu_si512(&cache_objs[copied + 16], c);
> > +			_mm512_storeu_si512(&cache_objs[copied + 24], d);
> > +			copied += 32;
> > +		}
> 
> Do you have any indications about how this copy loop performs, compared to e.g. the 64-byte block copy in rte_memcpy() [1]?
> 
> [1]: https://github.com/DPDK/dpdk/blob/v26.07-rc4/lib/eal/x86/include/rte_memcpy.h#L657
> 
> I'm wondering if it would be worth adding something similar to the mempool library, for speeding up mbuf alloc/free.
> 
Possibly, but you also need to put in an additional checks for AVX-512 at
runtime. The reason that we are doing the 64-byte copies explicitly here is
that we are already in a function which is already using AVX-512 and for
which we have no penalty or branch to check for its presence. The check is
done at function pointer selection time for us. For inline copies, you may
have a cost for that, though hopefully it could be very minimal. Only way to
know is to test it out!

/Bruce