Re: [PATCH] RDMA/ipoib: distribute cq completion vector better

Jinpu Wang
2020-10-28T23:38:58Z

On Wed, Oct 28, 2020 at 2:59 PM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> On Wed, Oct 28, 2020 at 02:57:57PM +0100, Jinpu Wang wrote:
> > On Wed, Oct 28, 2020 at 2:53 PM Jason Gunthorpe <jgg@nvidia.com> wrote:
> > >
> > > On Wed, Oct 28, 2020 at 02:48:01PM +0100, Jinpu Wang wrote:
> > > > On Wed, Oct 28, 2020 at 2:44 PM Jason Gunthorpe <jgg@nvidia.com> wrote:
> > > > >
> > > > > On Tue, Oct 13, 2020 at 09:43:42AM +0200, Jack Wang wrote:
> > > > > > Currently ipoib choose cq completion vector based on port number,
> > > > > > when HCA only have one port, all the interface recv queue completion
> > > > > > are bind to cq completion vector 0.
> > > > > >
> > > > > > To better distribute the load, use same method as __ib_alloc_cq_any
> > > > > > to choose completion vector, with the change, each interface now use
> > > > > > different completion vectors.
> > > > > >
> > > > > > Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
> > > > > > Reviewed-by: Gioh Kim <gi-oh.kim@cloud.ionos.com>
> > > > > > drivers/infiniband/ulp/ipoib/ipoib_verbs.c | 4 ++--
> > > > > > 1 file changed, 2 insertions(+), 2 deletions(-)
> > > > >
> > > > > If you care about IPoIB performance you should be using the
> > > > > accelerated IPoIB stuff, the drivers implementing that all provide
> > > > > much better handling of CQ affinity.
> > > > >
> > > > > What scenario does this patch make a difference?
> > > >
> > > > AFAIK the enhance mode is only for datagram mode, we are using connected mode.
> > >
> > > And you are using child interfaces or multiple cards?
> > we are using multiple child interfaces on Connect x5 HCA
> > (MCX556A-ECAT)
>
> Ok.. Do you think it would be better to change IPoIB to use the new
> shared CQ pool API?
>
> Jason
I thought about it, but CQ pool API doesn't work with IB_POLL_DIRECT,
ipoib is using NAPI,
and it's non-trivial to convert to CQ pool API. I'm not confident to
do it properly,