mlx4: IB_EVENT_PATH_MIG not generated on path migration

mlx4: IB_EVENT_PATH_MIG not generated on path migration

[Fabian Holler]

Hello,

We are using Linux 3.14.29 with mlx4 drivers and trying to use automatic path
migration for RC connections.
The path migration works, but no IB_EVENT_PATH_MIG* events are generated when
it happens.

APM is configured on machine A by the following code snippet:
        [..]
        err = ib_query_qp(qp, &qp_attr, 0, &init_attr);                                                                                            
        if (err)                                                                                                                                   
                return err;                                                                                                                        
                                                                                                                                                   
        memcpy(&qp_attr.alt_ah_attr, &qp_attr.ah_attr,
               sizeof(qp_attr.alt_ah_attr));
                                                                                                                                                   
        qp_attr.alt_pkey_index          = qp_attr.pkey_index;
        qp_attr.alt_timeout             = qp_attr.timeout;
        qp_attr.path_mig_state          = IB_MIG_REARM;
        qp_attr.alt_ah_attr.dlid        = dlid; /* peer's 2. port dlid */

        err = ib_modify_qp(con->qp, &qp_attr,
                           IB_QP_ALT_PATH | IB_QP_PATH_MIG_STATE);

(I experiment with setting also the ah_attr.port_num and alt_port_num
attributes, they seem to have no effect.)

The QP event handler is registered when calling rdma_create_qp():
        [..]
        init_attr.event_handler = qp_event_handler;
        init_attr.qp_context = con;
        ret = rdma_create_qp(cm_id, pd, &init_attr);


Both machines have MT26428 HCAs.
1 Port of machine A, both ports of machine B are connected to a switch.
We establish the connection to the 1. port of machine B, unplug the
corresponding cable and the path is migrated transparently to the second port
of machine B.
The connection can be used like nothing happens.

As soon as the migration happens I expect that a IB_EVENT_PATH_MIG* is
generated on the QP event handler, but I don't receive any QP events.

We need the event notification to rearm the path migration state machine for
another migration.

Does anybody have an idea what could be wrong?
Are the PATH_MIG* notifications with mlx4 drivers are working for somebody?


thanks

Fabian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mlx4: IB_EVENT_PATH_MIG not generated on path migration

[Jason Gunthorpe]

On Tue, Feb 10, 2015 at 04:56:43PM +0100, Fabian Holler wrote:

> Does anybody have an idea what could be wrong?
> Are the PATH_MIG* notifications with mlx4 drivers are working for somebody?

IIRC rdmacm does not forward these events.

There is a lot missing to support APM with RDMA CM, it doesn't do any
of the CM state machine operations to coordinate loading alternate
paths, for instance.

> As soon as the migration happens I expect that a IB_EVENT_PATH_MIG* is
> generated on the QP event handler, but I don't receive any QP events.

qp_event_handler is a RDMA CM event handler, it handles
RDMA_CM_EVENT_* and there is no RDMA_CM_EVENT_PATH_MIGRATED.

You'd need to listen to the verbs level events directly, I don't
recall if this is possible with RDMA CM or not, sorry.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mlx4: IB_EVENT_PATH_MIG not generated on path migration

[Jack Wang]

2015-02-10 19:00 GMT+01:00 Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>:
> On Tue, Feb 10, 2015 at 04:56:43PM +0100, Fabian Holler wrote:
>
>> Does anybody have an idea what could be wrong?
>> Are the PATH_MIG* notifications with mlx4 drivers are working for somebody?
>
> IIRC rdmacm does not forward these events.
Thanks Jason for reply.

We are working in kernel module, we subscribe qp_event_handler when
create qp, we've checked mlx4 driver(mlx4_core & mlx4_ib), looks they
do report qp event. But strangely we never receive the call in our
callback function.

Forgot to mention, the switch is from Qlogic, in case it matters.

Regards,
Jack

>
> There is a lot missing to support APM with RDMA CM, it doesn't do any
> of the CM state machine operations to coordinate loading alternate
> paths, for instance.
>
>> As soon as the migration happens I expect that a IB_EVENT_PATH_MIG* is
>> generated on the QP event handler, but I don't receive any QP events.
>
> qp_event_handler is a RDMA CM event handler, it handles
> RDMA_CM_EVENT_* and there is no RDMA_CM_EVENT_PATH_MIGRATED.
>
> You'd need to listen to the verbs level events directly, I don't
> recall if this is possible with RDMA CM or not, sorry.
>
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mlx4: IB_EVENT_PATH_MIG not generated on path migration

[Fabian Holler]

Hello Jason,

thanks for your answer.

On Tue, Feb 10, 2015 at 11:00:39AM -0700, Jason Gunthorpe wrote:
> On Tue, Feb 10, 2015 at 04:56:43PM +0100, Fabian Holler wrote:
> 
> > Does anybody have an idea what could be wrong?
> > Are the PATH_MIG* notifications with mlx4 drivers are working for somebody?
> 
> IIRC rdmacm does not forward these events.
> 
> There is a lot missing to support APM with RDMA CM, it doesn't do any
> of the CM state machine operations to coordinate loading alternate
> paths, for instance.

Alternate paths can be set directly at the QP.

> > As soon as the migration happens I expect that a IB_EVENT_PATH_MIG* is
> > generated on the QP event handler, but I don't receive any QP events.
> 
> qp_event_handler is a RDMA CM event handler, it handles
> RDMA_CM_EVENT_* and there is no RDMA_CM_EVENT_PATH_MIGRATED.
> You'd need to listen to the verbs level events directly, I don't
> recall if this is possible with RDMA CM or not, sorry.

The RDMA CM event handler is registered by the rdma_create_id() call.

The event handler specified in the struct ib_qp_init_attr * parameter of
rdma_create_qp() is registered as QP event handler.
rdma_create_qp() calls ib_create_qp() and sets qp->event_handler to
ib_qp_init_attr->event_handler.


Fabian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mlx4: IB_EVENT_PATH_MIG not generated on path migration

[Jason Gunthorpe]

On Wed, Feb 11, 2015 at 10:21:36AM +0100, Fabian Holler wrote:
> Hello Jason,
> 
> thanks for your answer.
> 
> On Tue, Feb 10, 2015 at 11:00:39AM -0700, Jason Gunthorpe wrote:
> > On Tue, Feb 10, 2015 at 04:56:43PM +0100, Fabian Holler wrote:
> > 
> > > Does anybody have an idea what could be wrong?
> > > Are the PATH_MIG* notifications with mlx4 drivers are working for somebody?
> > 
> > IIRC rdmacm does not forward these events.
> > 
> > There is a lot missing to support APM with RDMA CM, it doesn't do any
> > of the CM state machine operations to coordinate loading alternate
> > paths, for instance.
> 
> Alternate paths can be set directly at the QP.

APM requires a certain amount of synchronization between the local and
remote QP state, particularly if you want it to work more than
once. That is supposed to be done with CMA mads.

> The event handler specified in the struct ib_qp_init_attr * parameter of
> rdma_create_qp() is registered as QP event handler.
> rdma_create_qp() calls ib_create_qp() and sets qp->event_handler to
> ib_qp_init_attr->event_handler.

Okay, right, I was thinking of something else.

We had exactly this working a few months ago for a demo, in kernel
using Lustre and RDMA CM. I checked the patches we made and yes, the
event_handler was being properly called with IB_EVENT_PATH_MIG on
mlx4 and qib cards.

IIRC we may have needed the latest firmware on the mlx side.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mlx4: IB_EVENT_PATH_MIG not generated on path migration

[Fabian Holler]

Hi Jason,

On Wed, Feb 11, 2015 at 10:38:38AM -0700, Jason Gunthorpe wrote:
> On Wed, Feb 11, 2015 at 10:21:36AM +0100, Fabian Holler wrote:
> > Hello Jason,
> > 
> > thanks for your answer.
> > 
> > On Tue, Feb 10, 2015 at 11:00:39AM -0700, Jason Gunthorpe wrote:
> > > On Tue, Feb 10, 2015 at 04:56:43PM +0100, Fabian Holler wrote:
> > > 
> > > > Does anybody have an idea what could be wrong?
> > > > Are the PATH_MIG* notifications with mlx4 drivers are working for somebody?
> > > 
> > > IIRC rdmacm does not forward these events.
> > > 
> > > There is a lot missing to support APM with RDMA CM, it doesn't do any
> > > of the CM state machine operations to coordinate loading alternate
> > > paths, for instance.
> > 
> > Alternate paths can be set directly at the QP.
> 
> APM requires a certain amount of synchronization between the local and
> remote QP state, particularly if you want it to work more than
> once. That is supposed to be done with CMA mads.

Ok, we discovered functions to configure APM in ib_cm.h.
We will investigate if we can use them to setup APM.

> > The event handler specified in the struct ib_qp_init_attr * parameter of 
> > rdma_create_qp() is registered as QP event handler.
> > rdma_create_qp() calls ib_create_qp() and sets qp->event_handler to
> > ib_qp_init_attr->event_handler.
> 
> Okay, right, I was thinking of something else.
> 
> We had exactly this working a few months ago for a demo, in kernel
> using Lustre and RDMA CM. I checked the patches we made and yes, the
> event_handler was being properly called with IB_EVENT_PATH_MIG on
> mlx4 and qib cards.
> 
> IIRC we may have needed the latest firmware on the mlx side.

We will also try a firmware upgrade.


thanks a lot!

Fabian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mlx4: IB_EVENT_PATH_MIG not generated on path migration

[Fabian Holler]

[-- Attachment #1: Type: text/plain, Size: 1108 bytes --]

Hello Jason,

On Wed, Feb 11, 2015 at 10:38:38AM -0700, Jason Gunthorpe wrote:
> On Wed, Feb 11, 2015 at 10:21:36AM +0100, Fabian Holler wrote:
> > On Tue, Feb 10, 2015 at 11:00:39AM -0700, Jason Gunthorpe wrote:
> > > On Tue, Feb 10, 2015 at 04:56:43PM +0100, Fabian Holler wrote:
> > > 
> > > > Does anybody have an idea what could be wrong?
> > > > Are the PATH_MIG* notifications with mlx4 drivers are working for somebody?
[..]
> > The event handler specified in the struct ib_qp_init_attr * parameter of
> > rdma_create_qp() is registered as QP event handler.
> > rdma_create_qp() calls ib_create_qp() and sets qp->event_handler to
> > ib_qp_init_attr->event_handler.
> 
> Okay, right, I was thinking of something else.
> 
> We had exactly this working a few months ago for a demo, in kernel
> using Lustre and RDMA CM. I checked the patches we made and yes, the
> event_handler was being properly called with IB_EVENT_PATH_MIG on
> mlx4 and qib cards.

are your patches for APM support with RDMA CM available somewhere?
We would be glad to test them :-)


regards

Fabian

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]