cq error timeout issue

cq error timeout issue

[Vlad Weinbaum]

Hi,

I try to use ibv_poll_cq to identify connectivity problems. The
scenario is following, based on modified rping example:

1) preliminary steps done and rdma connection established between
Client and Server, retry_count in rdma_conn_param is set 1;
2) Server lost its link (corresponding switch port disabled), Client
is still connected to the switch;
3) Client calls ibv_post_send
4) Client polls cq with ibv_poll_cq and gets expected
IBV_WC_RETRY_EXC_ERR after about 1 second.

Can this timeout be decreased? If it is impossible, can you suggest
something else?

Used software is Ofed-1.5.3.1, IB device is Mellanox MT26428.

Thanks,
Vlad
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: cq error timeout issue

[Hefty, Sean]

> I try to use ibv_poll_cq to identify connectivity problems. The
> scenario is following, based on modified rping example:
> 
> 1) preliminary steps done and rdma connection established between
> Client and Server, retry_count in rdma_conn_param is set 1;
> 2) Server lost its link (corresponding switch port disabled), Client
> is still connected to the switch;
> 3) Client calls ibv_post_send
> 4) Client polls cq with ibv_poll_cq and gets expected
> IBV_WC_RETRY_EXC_ERR after about 1 second.
> 
> Can this timeout be decreased? If it is impossible, can you suggest
> something else?

I don't believe easily.  The timeout is based on the path record returned by the SM, which is really what an app should use.  If you can adjust the timeout at the SM, that would be best.

If you can use a newer kernel, another alternative is to use rdma_set_option to provide your own path record as input in place of calling rdma_resolve_route.

Btw, with a small timeout and few retries, if you're not using QoS, you may want to enable that to prevent false timeouts.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: cq error timeout issue

[Vlad Weinbaum]

Thank you Sean,

I have an access to OpenSM configuration on the switch in my testing
environment. I tried to reduce the timout there (subnet_timeout,
packet_life_time), but unsuccessfully.
Btw, I found detail that I cannot explain. I query the QP after
connect and get timeout value 16, that must be 4 us * 2^ 16 = 256 ms,
but I get about 800 ms.

Thanks,
Vlad

On Mon, Oct 31, 2011 at 8:51 AM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>> I try to use ibv_poll_cq to identify connectivity problems. The
>> scenario is following, based on modified rping example:
>>
>> 1) preliminary steps done and rdma connection established between
>> Client and Server, retry_count in rdma_conn_param is set 1;
>> 2) Server lost its link (corresponding switch port disabled), Client
>> is still connected to the switch;
>> 3) Client calls ibv_post_send
>> 4) Client polls cq with ibv_poll_cq and gets expected
>> IBV_WC_RETRY_EXC_ERR after about 1 second.
>>
>> Can this timeout be decreased? If it is impossible, can you suggest
>> something else?
>
> I don't believe easily.  The timeout is based on the path record returned by the SM, which is really what an app should use.  If you can adjust the timeout at the SM, that would be best.
>
> If you can use a newer kernel, another alternative is to use rdma_set_option to provide your own path record as input in place of calling rdma_resolve_route.
>
> Btw, with a small timeout and few retries, if you're not using QoS, you may want to enable that to prevent false timeouts.
>
> - Sean
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: cq error timeout issue

[Or Gerlitz]

On Mon, Oct 31, 2011 at 9:08 AM, Vlad Weinbaum
<vlad.weinbaum-adSD+vzF2QdBDgjK7y7TUQ@public.gmane.org> wrote:
> [...] I found detail that I cannot explain. I query the QP after connect and get timeout value 16,
>  that must be 4 us * 2^16 = 256 ms, but I get about 800 ms.

As Sean indicated, the timeout is **based** on the packet_life_time,
in case you're configuring
your QP through the rdma-cm, the IB stack code actually adds one to
the packet_life_time quantity
as a rough estimate for the local hca ack delay, still maybe there is
a hole here and the query qp verb
isn't reporting correctly, what was the value you configured on the sm
side, and how many retries did
you use for the qp? each retry will double the time you should be
observing in practice.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: cq error timeout issue

[Vlad Weinbaum]

Hi,

packet_life_time is 12 in sm.conf.

I'm debugging the polling process and see that ibv_poll_cq returns
error when mlx4_cqe->owner_sr_opcode gets MLX4_CQE_OPCODE_ERROR value
(in libmlx4, mlx4_poll_one). Where this value came from?

Thanks,
Vlad


On Mon, Oct 31, 2011 at 3:00 PM, Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Mon, Oct 31, 2011 at 9:08 AM, Vlad Weinbaum
> <vlad.weinbaum-adSD+vzF2QdBDgjK7y7TUQ@public.gmane.org> wrote:
>> [...] I found detail that I cannot explain. I query the QP after connect and get timeout value 16,
>>  that must be 4 us * 2^16 = 256 ms, but I get about 800 ms.
>
> As Sean indicated, the timeout is **based** on the packet_life_time,
> in case you're configuring
> your QP through the rdma-cm, the IB stack code actually adds one to
> the packet_life_time quantity
> as a rough estimate for the local hca ack delay, still maybe there is
> a hole here and the query qp verb
> isn't reporting correctly, what was the value you configured on the sm
> side, and how many retries did
> you use for the qp? each retry will double the time you should be
> observing in practice.
>
> Or.
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: cq error timeout issue

[Hal Rosenstock]

Just to clarify:

On 11/1/2011 9:25 AM, Vlad Weinbaum wrote:
> packet_life_time is 12 in sm.conf.

OpenSM packet_life_time setting has nothing to do with this (but is used
for SwitchInfo:LifeTimeValue setting):

# The code of maximal time a packet can live in a switch
# The actual time is 4.096usec * 2^<packet_life_time>
# The value 0x14 disables this mechanism

The rate in SA PathRecord comes from subnet timeout in the absence of QoS:
        /*
         * Set packet lifetime.
         * According to spec definition IBA 1.2 Table 205
         * PacketLifeTime description, for loopback paths,
         * packetLifeTime shall be zero.
         */
        if (p_src_port == p_dest_port)
                pkt_life = 0;
        else if (p_qos_level && p_qos_level->pkt_life_set)
                pkt_life = p_qos_level->pkt_life;
        else
                pkt_life = sa->p_subn->opt.subnet_timeout;

-- Hal

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html