Re: [PATCH] RDMA/ipoib: distribute cq completion vector better

Jinpu Wang
2020-10-28T23:20:58Z

On Wed, Oct 28, 2020 at 2:53 PM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> On Wed, Oct 28, 2020 at 02:48:01PM +0100, Jinpu Wang wrote:
> > On Wed, Oct 28, 2020 at 2:44 PM Jason Gunthorpe <jgg@nvidia.com> wrote:
> > >
> > > On Tue, Oct 13, 2020 at 09:43:42AM +0200, Jack Wang wrote:
> > > > Currently ipoib choose cq completion vector based on port number,
> > > > when HCA only have one port, all the interface recv queue completion
> > > > are bind to cq completion vector 0.
> > > >
> > > > To better distribute the load, use same method as __ib_alloc_cq_any
> > > > to choose completion vector, with the change, each interface now use
> > > > different completion vectors.
> > > >
> > > > Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
> > > > Reviewed-by: Gioh Kim <gi-oh.kim@cloud.ionos.com>
> > > > drivers/infiniband/ulp/ipoib/ipoib_verbs.c | 4 ++--
> > > > 1 file changed, 2 insertions(+), 2 deletions(-)
> > >
> > > If you care about IPoIB performance you should be using the
> > > accelerated IPoIB stuff, the drivers implementing that all provide
> > > much better handling of CQ affinity.
> > >
> > > What scenario does this patch make a difference?
> >
> > AFAIK the enhance mode is only for datagram mode, we are using connected mode.
>
> And you are using child interfaces or multiple cards?
we are using multiple child interfaces on Connect x5 HCA (MCX556A-ECAT)
ibstat
CA 'mlx5_0'
CA type: MT4119
Number of ports: 1
Firmware version: 16.28.2006
Hardware version: 0
Node GUID: 0x98039b03006c7912
System image GUID: 0x98039b03006c7912
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 38
LMC: 0
SM lid: 4
Capability mask: 0x2651e848
Port GUID: 0x98039b03006c7912
Link layer: InfiniBand
CA 'mlx5_1'
CA type: MT4119
Number of ports: 1
Firmware version: 16.28.2006
Hardware version: 0
Node GUID: 0x98039b03006c7913
System image GUID: 0x98039b03006c7912
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 134
LMC: 0
SM lid: 224
Capability mask: 0x2651e848
Port GUID: 0x98039b03006c7913
Link layer: InfiniBand

>
> Jason