AVX512 block copy performance?

["Morten Brørup" <mb@smartsharesystems.com>]

> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Leyi Rong
> Sent: Thursday, 14 January 2021 07.40
> 
> Optimize Tx path by using AVX512 instructions and vectorize the
> tx free bufs process.
> 
> Signed-off-by: Leyi Rong <leyi.rong@intel.com>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---

[...]

> +static __rte_always_inline int
> +i40e_tx_free_bufs_avx512(struct i40e_tx_queue *txq)
> +{
> +	struct i40e_vec_tx_entry *txep;
> +	uint32_t n;
> +	uint32_t i;
> +	int nb_free = 0;
> +	struct rte_mbuf *m, *free[RTE_I40E_TX_MAX_FREE_BUF_SZ];
> +
> +	/* check DD bits on threshold descriptor */
> +	if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz &
> +			rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) !=
> +			rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE))
> +		return 0;
> +
> +	n = txq->tx_rs_thresh;
> +
> +	 /* first buffer to free from S/W ring is at index
> +	  * tx_next_dd - (tx_rs_thresh-1)
> +	  */
> +	txep = (void *)txq->sw_ring;
> +	txep += txq->tx_next_dd - (n - 1);
> +
> +	if (txq->offloads & DEV_TX_OFFLOAD_MBUF_FAST_FREE && (n & 31) ==
> 0) {
> +		struct rte_mempool *mp = txep[0].mbuf->pool;
> +		void **cache_objs;
> +		struct rte_mempool_cache *cache =
> rte_mempool_default_cache(mp,
> +				rte_lcore_id());
> +
> +		if (!cache || cache->len == 0)
> +			goto normal;
> +
> +		cache_objs = &cache->objs[cache->len];
> +
> +		if (n > RTE_MEMPOOL_CACHE_MAX_SIZE) {
> +			rte_mempool_ops_enqueue_bulk(mp, (void *)txep, n);
> +			goto done;
> +		}
> +
> +		/* The cache follows the following algorithm
> +		 *   1. Add the objects to the cache
> +		 *   2. Anything greater than the cache min value (if it
> +		 *   crosses the cache flush threshold) is flushed to the
> ring.
> +		 */
> +		/* Add elements back into the cache */
> +		uint32_t copied = 0;
> +		/* n is multiple of 32 */
> +		while (copied < n) {
> +			const __m512i a = _mm512_load_si512(&txep[copied]);
> +			const __m512i b = _mm512_load_si512(&txep[copied +
> 8]);
> +			const __m512i c = _mm512_load_si512(&txep[copied +
> 16]);
> +			const __m512i d = _mm512_load_si512(&txep[copied +
> 24]);
> +
> +			_mm512_storeu_si512(&cache_objs[copied], a);
> +			_mm512_storeu_si512(&cache_objs[copied + 8], b);
> +			_mm512_storeu_si512(&cache_objs[copied + 16], c);
> +			_mm512_storeu_si512(&cache_objs[copied + 24], d);
> +			copied += 32;
> +		}

Do you have any indications about how this copy loop performs, compared to e.g. the 64-byte block copy in rte_memcpy() [1]?

[1]: https://github.com/DPDK/dpdk/blob/v26.07-rc4/lib/eal/x86/include/rte_memcpy.h#L657

I'm wondering if it would be worth adding something similar to the mempool library, for speeding up mbuf alloc/free.

> +		cache->len += n;
> +
> +		if (cache->len >= cache->flushthresh) {
> +			rte_mempool_ops_enqueue_bulk
> +				(mp, &cache->objs[cache->size],
> +				cache->len - cache->size);
> +			cache->len = cache->size;
> +		}
> +		goto done;
> +	}
> +
> +normal: