A strange problem when using IB to transfer things

A strange problem when using IB to transfer things

[Ding Dinghua]

Hi all:
         I'm using RDMA to do fs-metadata mirror between nodes. I
encountered a strange problem when the program was running:
Complete queue handler reported that the  RDMA-Write operation failed,
 the status of  corresponding "struct ib_wc" is "IB_WC_RETRY_EXC_ERR".
The problem is encountered randomly. I don't know the meaning of this
error code as well as what to do next. Would anyone give me some tips?
thanks a lot.
       My dev env is Centos5.3, IB card is "01:00.0 InfiniBand:
Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev 20)".
       Please cc to me, since I haven't subscribe the list.

-- 
Ding Dinghua
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: A strange problem when using IB to transfer things

[Dotan Barak]

Hi.

On 11/06/2010 10:51, Ding Dinghua wrote:
> Hi all:
>           I'm using RDMA to do fs-metadata mirror between nodes. I
> encountered a strange problem when the program was running:
> Complete queue handler reported that the  RDMA-Write operation failed,
>   the status of  corresponding "struct ib_wc" is "IB_WC_RETRY_EXC_ERR".
> The problem is encountered randomly. I don't know the meaning of this
> error code as well as what to do next. Would anyone give me some tips?
> thanks a lot.
>    

Do you sync between the sides before closing the QPs?

Retry exceeded mans that a message was sent without a response
(maybe you need to increase the timeout in the QP attributes:  timeout 
or retry_cnt).

I hope this helped you ..

Dotan
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: A strange problem when using IB to transfer things

[Ding Dinghua]

2010/6/11 Dotan Barak <dotanba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
> Hi.
>
> On 11/06/2010 10:51, Ding Dinghua wrote:
>>
>> Hi all:
>>          I'm using RDMA to do fs-metadata mirror between nodes. I
>> encountered a strange problem when the program was running:
>> Complete queue handler reported that the  RDMA-Write operation failed,
>>  the status of  corresponding "struct ib_wc" is "IB_WC_RETRY_EXC_ERR".
>> The problem is encountered randomly. I don't know the meaning of this
>> error code as well as what to do next. Would anyone give me some tips?
>> thanks a lot.
>>
>
> Do you sync between the sides before closing the QPs?

Can you say it more detail? thanks.

>
> Retry exceeded mans that a message was sent without a response
> (maybe you need to increase the timeout in the QP attributes:  timeout or
> retry_cnt).
>
> I hope this helped you ..
>
> Dotan
>



-- 
Ding Dinghua
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

P

[Dotan Barak]

On 12/06/2010 03:22, Ding Dinghua wrote:
> 2010/6/11 Dotan Barak<dotanba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
>    
>> Hi.
>>
>> On 11/06/2010 10:51, Ding Dinghua wrote:
>>      
>>> Hi all:
>>>           I'm using RDMA to do fs-metadata mirror between nodes. I
>>> encountered a strange problem when the program was running:
>>> Complete queue handler reported that the  RDMA-Write operation failed,
>>>   the status of  corresponding "struct ib_wc" is "IB_WC_RETRY_EXC_ERR".
>>> The problem is encountered randomly. I don't know the meaning of this
>>> error code as well as what to do next. Would anyone give me some tips?
>>> thanks a lot.
>>>
>>>        
>> Do you sync between the sides before closing the QPs?
>>      
> Can you say it more detail? thanks.
>    
If you try to send a message from local QP to a remote QP before the 
remote QP is in RTR state (or after it was closed/transferred to the 
ERROR state),
you may get RETRY EXCEEDED, because there isn't any QP in the remote 
side that can accept your message (and send a response).

How do you connect the QPs? (And how do you close the connection between 
them)

Dotan
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: P

[Ding Dinghua]

Sorry for late reply.

2010/6/12 Dotan Barak <dotanba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
> On 12/06/2010 03:22, Ding Dinghua wrote:
>>
>> 2010/6/11 Dotan Barak<dotanba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
>>
>>>
>>> Hi.
>>>
>>> On 11/06/2010 10:51, Ding Dinghua wrote:
>>>
>>>>
>>>> Hi all:
>>>>          I'm using RDMA to do fs-metadata mirror between nodes. I
>>>> encountered a strange problem when the program was running:
>>>> Complete queue handler reported that the  RDMA-Write operation failed,
>>>>  the status of  corresponding "struct ib_wc" is "IB_WC_RETRY_EXC_ERR".
>>>> The problem is encountered randomly. I don't know the meaning of this
>>>> error code as well as what to do next. Would anyone give me some tips?
>>>> thanks a lot.
>>>>
>>>>
>>>
>>> Do you sync between the sides before closing the QPs?
>>>
>>
>> Can you say it more detail? thanks.
>>
>
> If you try to send a message from local QP to a remote QP before the remote
> QP is in RTR state (or after it was closed/transferred to the ERROR state),
> you may get RETRY EXCEEDED, because there isn't any QP in the remote side
> that can accept your message (and send a response).
>
> How do you connect the QPs? (And how do you close the connection between
> them)
>
I call rdma_create_id to create an ib id, then do resolve remote addr,
resolve route work, then
setup qp and call rdma_connect to setup connection, before ack or
error replies, the thread will
wait on a wait queue. The listening ib id of remote node will catch
the connect request,
setup qp, allocate and map pages to construct the RDMA-WRITE space,
and call rdma_accept to reply
the request.

Some other information which may be useful:
1.All the "RETRY EXCEEDED" problems happened when there were two
connections which use RDMA-WRITE to transfer things.
And the latter connection had a high possibility to get into this problem.
2. All the "RETRY EXCEEDED" problems happened when the RMDA-WRITE
space is 256MB each(that is, for two connections, consumes 512MB mem),
when the RDMA-WRITE  space is 64MB, this problem never happened in our
test. Remote node's total memory is 2GB.

Thanks a lot.


> Dotan
>



-- 
Ding Dinghua
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: P

[Dotan Barak]

> I call rdma_create_id to create an ib id, then do resolve remote addr,
> resolve route work, then
> setup qp and call rdma_connect to setup connection, before ack or
> error replies, the thread will
> wait on a wait queue. The listening ib id of remote node will catch
> the connect request,
> setup qp, allocate and map pages to construct the RDMA-WRITE space,
> and call rdma_accept to reply
> the request.
>
> Some other information which may be useful:
> 1.All the "RETRY EXCEEDED" problems happened when there were two
> connections which use RDMA-WRITE to transfer things.
> And the latter connection had a high possibility to get into this problem.
> 2. All the "RETRY EXCEEDED" problems happened when the RMDA-WRITE
> space is 256MB each(that is, for two connections, consumes 512MB mem),
> when the RDMA-WRITE  space is 64MB, this problem never happened in our
> test. Remote node's total memory is 2GB.
>
> Thanks a lot.
>    
Some more questions:
* Is the WR that "produces" the RETRY EXCEEDED is the first one/last 
one/in the middle?
* Which values are you using in the QP context for retry exceeded 
counter + retry timeout?
* Did you try to increase those values?
* How many more QPs do you have between those nodes and which operations 
do they use
    (only RDMA-WRITEs?)

Thanks
Dotan
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: P

[Ding Dinghua]

hello,

2010/6/19 Dotan Barak <dotanba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
>
>> I call rdma_create_id to create an ib id, then do resolve remote addr,
>> resolve route work, then
>> setup qp and call rdma_connect to setup connection, before ack or
>> error replies, the thread will
>> wait on a wait queue. The listening ib id of remote node will catch
>> the connect request,
>> setup qp, allocate and map pages to construct the RDMA-WRITE space,
>> and call rdma_accept to reply
>> the request.
>>
>> Some other information which may be useful:
>> 1.All the "RETRY EXCEEDED" problems happened when there were two
>> connections which use RDMA-WRITE to transfer things.
>> And the latter connection had a high possibility to get into this problem.
>> 2. All the "RETRY EXCEEDED" problems happened when the RMDA-WRITE
>> space is 256MB each(that is, for two connections, consumes 512MB mem),
>> when the RDMA-WRITE  space is 64MB, this problem never happened in our
>> test. Remote node's total memory is 2GB.
>>
>> Thanks a lot.
>>
>
> Some more questions:
> * Is the WR that "produces" the RETRY EXCEEDED is the first one/last one/in
> the middle?

it's the first one

> * Which values are you using in the QP context for retry exceeded counter +
> retry timeout?
> * Did you try to increase those values?

I haven't set these values(actually  I don't know where to set these
values), i just set max_send_wr and max_send_sge
fields of struct ib_qp_cap when creating qp.

> * How many more QPs do you have between those nodes and which operations do
> they use
>   (only RDMA-WRITEs?)
>

4096 QPs for each connection,  only do RDMA-WRITES.

> Thanks
> Dotan
>



-- 
Ding Dinghua
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: P

[Dotan Barak]

On 20/06/2010 07:51, Ding Dinghua wrote:
> hello,
>
> 2010/6/19 Dotan Barak<dotanba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
>    
>>      
>>> I call rdma_create_id to create an ib id, then do resolve remote addr,
>>> resolve route work, then
>>> setup qp and call rdma_connect to setup connection, before ack or
>>> error replies, the thread will
>>> wait on a wait queue. The listening ib id of remote node will catch
>>> the connect request,
>>> setup qp, allocate and map pages to construct the RDMA-WRITE space,
>>> and call rdma_accept to reply
>>> the request.
>>>
>>> Some other information which may be useful:
>>> 1.All the "RETRY EXCEEDED" problems happened when there were two
>>> connections which use RDMA-WRITE to transfer things.
>>> And the latter connection had a high possibility to get into this problem.
>>> 2. All the "RETRY EXCEEDED" problems happened when the RMDA-WRITE
>>> space is 256MB each(that is, for two connections, consumes 512MB mem),
>>> when the RDMA-WRITE  space is 64MB, this problem never happened in our
>>> test. Remote node's total memory is 2GB.
>>>
>>> Thanks a lot.
>>>
>>>        
>> Some more questions:
>> * Is the WR that "produces" the RETRY EXCEEDED is the first one/last one/in
>> the middle?
>>      
> it's the first one
>
>    
>> * Which values are you using in the QP context for retry exceeded counter +
>> retry timeout?
>> * Did you try to increase those values?
>>      
> I haven't set these values(actually  I don't know where to set these
> values), i just set max_send_wr and max_send_sge
> fields of struct ib_qp_cap when creating qp.
>
>    
Can you perform query QP after establishing a connection between the QPs 
and check those values?

>> * How many more QPs do you have between those nodes and which operations do
>> they use
>>    (only RDMA-WRITEs?)
>>
>>      
> 4096 QPs for each connection,  only do RDMA-WRITES.
>    
So, you send in parallel total of 4K (QPs) * 64M (Bytes)  = 256 GB
(am i missing something, or this is the amount of data that will be sent 
between two nodes?)

Dotan
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: P

[Ding Dinghua]

2010/6/21 Dotan Barak <dotanba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
> On 20/06/2010 07:51, Ding Dinghua wrote:
>>
>> hello,
>>
>> 2010/6/19 Dotan Barak<dotanba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
>>
>>>
>>>
>>>>
>>>> I call rdma_create_id to create an ib id, then do resolve remote addr,
>>>> resolve route work, then
>>>> setup qp and call rdma_connect to setup connection, before ack or
>>>> error replies, the thread will
>>>> wait on a wait queue. The listening ib id of remote node will catch
>>>> the connect request,
>>>> setup qp, allocate and map pages to construct the RDMA-WRITE space,
>>>> and call rdma_accept to reply
>>>> the request.
>>>>
>>>> Some other information which may be useful:
>>>> 1.All the "RETRY EXCEEDED" problems happened when there were two
>>>> connections which use RDMA-WRITE to transfer things.
>>>> And the latter connection had a high possibility to get into this
>>>> problem.
>>>> 2. All the "RETRY EXCEEDED" problems happened when the RMDA-WRITE
>>>> space is 256MB each(that is, for two connections, consumes 512MB mem),
>>>> when the RDMA-WRITE  space is 64MB, this problem never happened in our
>>>> test. Remote node's total memory is 2GB.
>>>>
>>>> Thanks a lot.
>>>>
>>>>
>>>
>>> Some more questions:
>>> * Is the WR that "produces" the RETRY EXCEEDED is the first one/last
>>> one/in
>>> the middle?
>>>
>>
>> it's the first one
>>
>>
>>>
>>> * Which values are you using in the QP context for retry exceeded counter
>>> +
>>> retry timeout?
>>> * Did you try to increase those values?
>>>
>>
>> I haven't set these values(actually  I don't know where to set these
>> values), i just set max_send_wr and max_send_sge
>> fields of struct ib_qp_cap when creating qp.
>>
>>
>
> Can you perform query QP after establishing a connection between the QPs and
> check those values?
>

All the QPs (local and remote, 2 connections) :
3 qp_state, 19 retry_cnt, 7 timeout, that's, all QPs' qp_state is
IB_QPS_RTS(Should remote QP's state be this, or IB_QPS_RTR?
But QP's state of first connection is the same and it can work...).

>>> * How many more QPs do you have between those nodes and which operations
>>> do
>>> they use
>>>   (only RDMA-WRITEs?)
>>>
>>>
>>
>> 4096 QPs for each connection,  only do RDMA-WRITES.
>>
>
> So, you send in parallel total of 4K (QPs) * 64M (Bytes)  = 256 GB
> (am i missing something, or this is the amount of data that will be sent
> between two nodes?)
>
if  RDMA-WRITE space is 64M(Bytes),  this means upper level applications send
at most 64M(Bytes)  to remote node one time. These QPs may send to different
piece of the 64M space in parallel.

> Dotan
>



-- 
Ding Dinghua
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: P

[Ding Dinghua]

2010/6/12 Dotan Barak <dotanba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
> On 12/06/2010 03:22, Ding Dinghua wrote:
>>
>> 2010/6/11 Dotan Barak<dotanba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
>>
>>>
>>> Hi.
>>>
>>> On 11/06/2010 10:51, Ding Dinghua wrote:
>>>
>>>>
>>>> Hi all:
>>>>          I'm using RDMA to do fs-metadata mirror between nodes. I
>>>> encountered a strange problem when the program was running:
>>>> Complete queue handler reported that the  RDMA-Write operation failed,
>>>>  the status of  corresponding "struct ib_wc" is "IB_WC_RETRY_EXC_ERR".
>>>> The problem is encountered randomly. I don't know the meaning of this
>>>> error code as well as what to do next. Would anyone give me some tips?
>>>> thanks a lot.
>>>>
>>>>
>>>
>>> Do you sync between the sides before closing the QPs?
>>>
>>
>> Can you say it more detail? thanks.
>>
>
> If you try to send a message from local QP to a remote QP before the remote
> QP is in RTR state (or after it was closed/transferred to the ERROR state),
> you may get RETRY EXCEEDED, because there isn't any QP in the remote side
> that can accept your message (and send a response).
>
> How do you connect the QPs? (And how do you close the connection between
> them)
>
Sorry i forget the close issue.

1. Local node call ib_poll_cq to process the remaining complete queue entry,
2. Local node call rdma_disconnect to destroy connection, before
remote side ack, the thread will wait on a wait queue.
3. After catching this request, the remote node will also call
ib_poll_cq to process the remainning complete queue entry,
then do some resource-release work, then send a reply.
4. Local node was waken up and do resource-release work.


> Dotan
>



-- 
Ding Dinghua
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html