Re: [PATCH net-next v2 3/3] net/mlx5e: use indirect calls wrapper for the rx packet handler

[Paolo Abeni <pabeni@redhat.com>]

Hi,

On Fri, 2019-06-07 at 18:09 +0000, Saeed Mahameed wrote:
> On Thu, 2019-06-06 at 23:56 +0200, Paolo Abeni wrote:
> > We can avoid another indirect call per packet wrapping the rx
> > handler call with the proper helper.
> > 
> > To ensure that even the last listed direct call experience
> > measurable gain, despite the additional conditionals we must
> > traverse before reaching it, I tested reversing the order of the
> > listed options, with performance differences below noise level.
> > 
> > Together with the previous indirect call patch, this gives
> > ~6% performance improvement in raw UDP tput.
> > 
> > v1 -> v2:
> >  - update the direct call list and use a macro to define it,
> >    as per Saeed suggestion. An intermediated additional
> >    macro is needed to allow arg list expansion
> > 
> > Signed-off-by: Paolo Abeni <pabeni@redhat.com>
> > ---
> >  drivers/net/ethernet/mellanox/mlx5/core/en.h    | 4 ++++
> >  drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 5 ++++-
> >  2 files changed, 8 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h
> > b/drivers/net/ethernet/mellanox/mlx5/core/en.h
> > index 3a183d690e23..52bcdc87cbe2 100644
> > --- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
> > +++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
> > @@ -148,6 +148,10 @@ struct page_pool;
> >  
> >  #define MLX5E_MSG_LEVEL			NETIF_MSG_LINK
> >  
> > +#define MLX5_RX_INDIRECT_CALL_LIST \
> > +	mlx5e_handle_rx_cqe_mpwrq, mlx5e_handle_rx_cqe,
> > mlx5i_handle_rx_cqe, \
> > +	mlx5e_ipsec_handle_rx_cqe
> > +
> >  #define mlx5e_dbg(mlevel, priv, format, ...)                    \
> >  do {                                                            \
> >  	if (NETIF_MSG_##mlevel & (priv)->msglevel)              \
> > diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
> > b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
> > index 0fe5f13d07cc..7faf643eb1b9 100644
> > --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
> > +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
> > @@ -1303,6 +1303,8 @@ void mlx5e_handle_rx_cqe_mpwrq(struct mlx5e_rq
> > *rq, struct mlx5_cqe64 *cqe)
> >  	mlx5_wq_ll_pop(wq, cqe->wqe_id, &wqe->next.next_wqe_index);
> >  }
> >  
> > +#define INDIRECT_CALL_LIST(f, list, ...) INDIRECT_CALL_4(f, list,
> > __VA_ARGS__)
> > +
> 
> Hi Paolo, 
> 
> This patch produces some compiler errors:
> 
> Please note that mlx5e_ipsec_handle_rx_cqe is only defined when
> CONFIG_MLX5_EN_IPSEC is enabled.

I'm sorry, I dumbly did not fuzz vs mlx5 build options.

It looks like that, to cope with all the possible mixes, a not-so-nice
macro maze is required; something alike the following:

#if defined(CONFIG_MLX5_EN_IPSEC) && defined (CONFIG_MLX5_CORE_IPOIB)

#define MLX5_RX_INDIRECT_CALL_LIST \
	mlx5e_handle_rx_cqe_mpwrq, mlx5e_handle_rx_cqe, mlx5i_handle_rx_cqe, \
	mlx5e_ipsec_handle_rx_cqe
#define INDIRECT_CALL_LIST(f, list, ...) INDIRECT_CALL_4(f, list, __VA_ARGS__)

#elif defined(CONFIG_MLX5_EN_IPSEC)

#define MLX5_RX_INDIRECT_CALL_LIST \
	mlx5e_handle_rx_cqe_mpwrq, mlx5e_handle_rx_cqe, \
	mlx5e_ipsec_handle_rx_cqe
#define INDIRECT_CALL_LIST(f, list, ...) INDIRECT_CALL_3(f, list, __VA_ARGS__)

#elif defined(CONFIG_MLX5_CORE_IPOIB)

#define MLX5_RX_INDIRECT_CALL_LIST \
	mlx5e_handle_rx_cqe_mpwrq, mlx5e_handle_rx_cqe, mlx5i_handle_rx_cqe
#define INDIRECT_CALL_LIST(f, list, ...) INDIRECT_CALL_3(f, list, __VA_ARGS__)

#else

#define MLX5_RX_INDIRECT_CALL_LIST \
	mlx5e_handle_rx_cqe_mpwrq, mlx5e_handle_rx_cqe
#define INDIRECT_CALL_LIST(f, list, ...) INDIRECT_CALL_2(f, list, __VA_ARGS__)

#endif

If you are ok with the above, I can include it in v3, otherwise I can
either:

* drop patch 2/3 and use only the 2 alternatives
(mlx5e_handle_rx_cqe_mpwrq, mlx5e_handle_rx_cqe) that are available
regardless of the driver build options

* drop both patches 2/3 and 3/3

Any feedback welcome, thanks!

Paolo