Re: State of ipoib cm mode

Re: State of ipoib cm mode

[Nikolay Borisov]

[Resending with the linux-rdma list cc'ed + some additional information]

On 07/27/2016 02:54 PM, Michael S. Tsirkin wrote:
> On Wed, Jul 27, 2016 at 01:41:53PM +0300, Nikolay Borisov wrote:
>> Hello,
>>
>> I've been running some production servers with ipoib cm but have
>> observed various hangs, e.g. :
>>
>> http://www.spinics.net/lists/linux-rdma/msg34577.html
>> http://www.spinics.net/lists/linux-rdma/msg37011.html
>> http://thread.gmane.org/gmane.linux.drivers.rdma/38899
>>
>> Other people have also confirmed that there is a latent bug, which is
>> very hard to debug (e.g. here:
>> http://www.spinics.net/lists/linux-rdma/msg37022.html). Essentially
>>
>> As the person who originally wrote the code and considering that git
>> blame indicates most of it hasn't been touched does that mean it's
>> considered stable? Also do you happen to have a hunch as to what might
>> be causing such stalls?
>>
>> Regards,
>> Nikolay
> 
> Please repost copying a mailing list.
> I have a general policy against responding to off-list mail.

Ok.

In addition to that, here is the state of a node which has been hung for
about 2 days now - no infiniband multicast connectivity, this is similar
to the issue observed in the first mailing list entry I have referenced,
but this time I managed to obtain the state of the ipoib_cm_rx and
ib_cm_id structs (as well as any other structs which are referenced from
those):


struct ipoib_cm_rx {
  id = 0xffff8802128fa600,
  qp = 0xffff880100e94000,
  rx_ring = 0x0,
  list = {
    next = 0xffff88055f02bdd8,
    prev = 0xffff88055f02bdd8
  },
  dev = 0xffff880661f68000,
  jiffies = 4367003834,
  state = IPOIB_CM_RX_FLUSH,
  recv_count = 0
}

struct ib_cm_id {
  cm_handler = 0xffffffffa01e7b60 <ipoib_cm_rx_handler>,
  context = 0xffff880660f11780,
  device = 0xffff8800378e4000,
  service_id = 216172782113783824,
  service_mask = 18446744073709551615,
  state = IB_CM_IDLE,
  lap_state = IB_CM_LAP_UNINIT,
  local_id = 1741978561,
  remote_id = 3782023797,
  remote_cm_qpn = 1
}

And the backtrace is like that:

PID: 28224  TASK: ffff88064bdb5280  CPU: 5   COMMAND: "kworker/u24:2"
 #0 [ffff88055f02bc28] __schedule at ffffffff8160fc6a
 #1 [ffff88055f02bc70] schedule at ffffffff816103dc
 #2 [ffff88055f02bc88] schedule_timeout at ffffffff81613642
 #3 [ffff88055f02bd08] wait_for_completion at ffffffff816118df
 #4 [ffff88055f02bd68] cm_destroy_id at ffffffffa01d3759 [ib_cm]
 #5 [ffff88055f02bdc0] ib_destroy_cm_id at ffffffffa01d3a10 [ib_cm]
 #6 [ffff88055f02bdd0] ipoib_cm_free_rx_reap_list at ffffffffa01e7675
[ib_ipoib]
 #7 [ffff88055f02be18] ipoib_cm_rx_reap at ffffffffa01e7705 [ib_ipoib]
 #8 [ffff88055f02be28] process_one_work at ffffffff8106bdf9
 #9 [ffff88055f02be68] worker_thread at ffffffff8106c4a9
#10 [ffff88055f02bed0] kthread at ffffffff8107161f
#11 [ffff88055f02bf50] ret_from_fork at ffffffff816149ff

ffffffffa01d3759 is wait_for_completion(&cm_id_priv->comp);

Can you advise what other information might be helpful to debug this ?

Regards,
Nikolay
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: State of ipoib cm mode

[Michael S. Tsirkin]

On Wed, Jul 27, 2016 at 03:05:52PM +0300, Nikolay Borisov wrote:
> [Resending with the linux-rdma list cc'ed + some additional information]
> 
> On 07/27/2016 02:54 PM, Michael S. Tsirkin wrote:
> > On Wed, Jul 27, 2016 at 01:41:53PM +0300, Nikolay Borisov wrote:
> >> Hello,
> >>
> >> I've been running some production servers with ipoib cm but have
> >> observed various hangs, e.g. :
> >>
> >> http://www.spinics.net/lists/linux-rdma/msg34577.html
> >> http://www.spinics.net/lists/linux-rdma/msg37011.html
> >> http://thread.gmane.org/gmane.linux.drivers.rdma/38899
> >>
> >> Other people have also confirmed that there is a latent bug, which is
> >> very hard to debug (e.g. here:
> >> http://www.spinics.net/lists/linux-rdma/msg37022.html). Essentially
> >>
> >> As the person who originally wrote the code and considering that git
> >> blame indicates most of it hasn't been touched does that mean it's
> >> considered stable? Also do you happen to have a hunch as to what might
> >> be causing such stalls?
> >>
> >> Regards,
> >> Nikolay
> > 
> > Please repost copying a mailing list.
> > I have a general policy against responding to off-list mail.
> 
> Ok.
> 
> In addition to that, here is the state of a node which has been hung for
> about 2 days now - no infiniband multicast connectivity, this is similar
> to the issue observed in the first mailing list entry I have referenced,
> but this time I managed to obtain the state of the ipoib_cm_rx and
> ib_cm_id structs (as well as any other structs which are referenced from
> those):
> 
> 
> struct ipoib_cm_rx {
>   id = 0xffff8802128fa600,
>   qp = 0xffff880100e94000,
>   rx_ring = 0x0,
>   list = {
>     next = 0xffff88055f02bdd8,
>     prev = 0xffff88055f02bdd8
>   },
>   dev = 0xffff880661f68000,
>   jiffies = 4367003834,
>   state = IPOIB_CM_RX_FLUSH,
>   recv_count = 0
> }
> 
> struct ib_cm_id {
>   cm_handler = 0xffffffffa01e7b60 <ipoib_cm_rx_handler>,
>   context = 0xffff880660f11780,
>   device = 0xffff8800378e4000,
>   service_id = 216172782113783824,
>   service_mask = 18446744073709551615,
>   state = IB_CM_IDLE,
>   lap_state = IB_CM_LAP_UNINIT,
>   local_id = 1741978561,
>   remote_id = 3782023797,
>   remote_cm_qpn = 1
> }
> 
> And the backtrace is like that:
> 
> PID: 28224  TASK: ffff88064bdb5280  CPU: 5   COMMAND: "kworker/u24:2"
>  #0 [ffff88055f02bc28] __schedule at ffffffff8160fc6a
>  #1 [ffff88055f02bc70] schedule at ffffffff816103dc
>  #2 [ffff88055f02bc88] schedule_timeout at ffffffff81613642
>  #3 [ffff88055f02bd08] wait_for_completion at ffffffff816118df
>  #4 [ffff88055f02bd68] cm_destroy_id at ffffffffa01d3759 [ib_cm]
>  #5 [ffff88055f02bdc0] ib_destroy_cm_id at ffffffffa01d3a10 [ib_cm]
>  #6 [ffff88055f02bdd0] ipoib_cm_free_rx_reap_list at ffffffffa01e7675
> [ib_ipoib]
>  #7 [ffff88055f02be18] ipoib_cm_rx_reap at ffffffffa01e7705 [ib_ipoib]
>  #8 [ffff88055f02be28] process_one_work at ffffffff8106bdf9
>  #9 [ffff88055f02be68] worker_thread at ffffffff8106c4a9
> #10 [ffff88055f02bed0] kthread at ffffffff8107161f
> #11 [ffff88055f02bf50] ret_from_fork at ffffffff816149ff
> 
> ffffffffa01d3759 is wait_for_completion(&cm_id_priv->comp);
> 
> Can you advise what other information might be helpful to debug this ?
> 
> Regards,
> Nikolay

I haven't looked at infiniband for ages, and won't be able to help you
much. The links provided seem to indicate issues when SM or CM is not
responsive.  Try introducing delays by pausing the SM once in a while,
or dropping packets to/from SM, or CM packets. Maybe add a mode that drops
some of these packets once in a while.

-- 
MST
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: State of ipoib cm mode

[Nikolay Borisov]

On Wed, Jul 27, 2016 at 7:05 PM, Serge Ryabchun
<serge.ryabchun-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Hi Nikolay,
>
> very similar behavior we have experienced an half a year ago with CX2 and
> CX3 and QDR Mellanox switches.
> It was fixed by this patch -
> http://www.spinics.net/lists/linux-rdma/msg23811.html. Not fixed really but
> at least it can move multicast QP from SQE to RTS state and restore
> connectivity.

Thanks for chiming in. According to git describe this patch made it to
4.1 and the kernel I'm using is 4.4. So in my case this behavior is
happening despite this patch being applied. One other element is that
I'm seeing this with qlogic cards (ib_qib driver). Unfortunately I'm
not able to pinpoint whether this is a problem of the card driver or
with the middleware ib_ipoib driver.

>
> It really was fixed replacing PSUs in the chassis by the more powerful. It
> appeared that Mellanox ASIC is very sensitive to the power. Under heavy
> loading those PSUs became slightly unstable and as a result built-in switch
> on the same PSUs produced damaged frames.
>
> --
> Regards,
> Serge
>
>
> On Wed, Jul 27, 2016 at 2:05 PM, Nikolay Borisov <kernel-6AxghH7DbtA@public.gmane.org> wrote:
>>
>> [Resending with the linux-rdma list cc'ed + some additional information]
>>
>> On 07/27/2016 02:54 PM, Michael S. Tsirkin wrote:
>> > On Wed, Jul 27, 2016 at 01:41:53PM +0300, Nikolay Borisov wrote:
>> >> Hello,
>> >>
>> >> I've been running some production servers with ipoib cm but have
>> >> observed various hangs, e.g. :
>> >>
>> >> http://www.spinics.net/lists/linux-rdma/msg34577.html
>> >> http://www.spinics.net/lists/linux-rdma/msg37011.html
>> >> http://thread.gmane.org/gmane.linux.drivers.rdma/38899
>> >>
>> >> Other people have also confirmed that there is a latent bug, which is
>> >> very hard to debug (e.g. here:
>> >> http://www.spinics.net/lists/linux-rdma/msg37022.html). Essentially
>> >>
>> >> As the person who originally wrote the code and considering that git
>> >> blame indicates most of it hasn't been touched does that mean it's
>> >> considered stable? Also do you happen to have a hunch as to what might
>> >> be causing such stalls?
>> >>
>> >> Regards,
>> >> Nikolay
>> >
>> > Please repost copying a mailing list.
>> > I have a general policy against responding to off-list mail.
>>
>> Ok.
>>
>> In addition to that, here is the state of a node which has been hung for
>> about 2 days now - no infiniband multicast connectivity, this is similar
>> to the issue observed in the first mailing list entry I have referenced,
>> but this time I managed to obtain the state of the ipoib_cm_rx and
>> ib_cm_id structs (as well as any other structs which are referenced from
>> those):
>>
>>
>> struct ipoib_cm_rx {
>>   id = 0xffff8802128fa600,
>>   qp = 0xffff880100e94000,
>>   rx_ring = 0x0,
>>   list = {
>>     next = 0xffff88055f02bdd8,
>>     prev = 0xffff88055f02bdd8
>>   },
>>   dev = 0xffff880661f68000,
>>   jiffies = 4367003834,
>>   state = IPOIB_CM_RX_FLUSH,
>>   recv_count = 0
>> }
>>
>> struct ib_cm_id {
>>   cm_handler = 0xffffffffa01e7b60 <ipoib_cm_rx_handler>,
>>   context = 0xffff880660f11780,
>>   device = 0xffff8800378e4000,
>>   service_id = 216172782113783824,
>>   service_mask = 18446744073709551615,
>>   state = IB_CM_IDLE,
>>   lap_state = IB_CM_LAP_UNINIT,
>>   local_id = 1741978561,
>>   remote_id = 3782023797,
>>   remote_cm_qpn = 1
>> }
>>
>> And the backtrace is like that:
>>
>> PID: 28224  TASK: ffff88064bdb5280  CPU: 5   COMMAND: "kworker/u24:2"
>>  #0 [ffff88055f02bc28] __schedule at ffffffff8160fc6a
>>  #1 [ffff88055f02bc70] schedule at ffffffff816103dc
>>  #2 [ffff88055f02bc88] schedule_timeout at ffffffff81613642
>>  #3 [ffff88055f02bd08] wait_for_completion at ffffffff816118df
>>  #4 [ffff88055f02bd68] cm_destroy_id at ffffffffa01d3759 [ib_cm]
>>  #5 [ffff88055f02bdc0] ib_destroy_cm_id at ffffffffa01d3a10 [ib_cm]
>>  #6 [ffff88055f02bdd0] ipoib_cm_free_rx_reap_list at ffffffffa01e7675
>> [ib_ipoib]
>>  #7 [ffff88055f02be18] ipoib_cm_rx_reap at ffffffffa01e7705 [ib_ipoib]
>>  #8 [ffff88055f02be28] process_one_work at ffffffff8106bdf9
>>  #9 [ffff88055f02be68] worker_thread at ffffffff8106c4a9
>> #10 [ffff88055f02bed0] kthread at ffffffff8107161f
>> #11 [ffff88055f02bf50] ret_from_fork at ffffffff816149ff
>>
>> ffffffffa01d3759 is wait_for_completion(&cm_id_priv->comp);
>>
>> Can you advise what other information might be helpful to debug this ?
>>
>> Regards,
>> Nikolay
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: State of ipoib cm mode

[Erez Shitrit]

Hi Nikolay,

There are few issues here, the tx_timeout is part of the error flow of
the driver, it happened mostly because the HW is much faster than the
SW/CPU, it is temporary occasion that should be freed after a short
time (not including bugs ..)

according your issue, please check with your HW vendor, i think that
it misses a completion.
that can lead to a livelock ..

Thanks, Erez


On Wed, Jul 27, 2016 at 7:35 PM, Nikolay Borisov
<n.borisov-/eCPMmvKun9pLGFMi4vTTA@public.gmane.org> wrote:
> On Wed, Jul 27, 2016 at 7:05 PM, Serge Ryabchun
> <serge.ryabchun-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> Hi Nikolay,
>>
>> very similar behavior we have experienced an half a year ago with CX2 and
>> CX3 and QDR Mellanox switches.
>> It was fixed by this patch -
>> http://www.spinics.net/lists/linux-rdma/msg23811.html. Not fixed really but
>> at least it can move multicast QP from SQE to RTS state and restore
>> connectivity.
>
> Thanks for chiming in. According to git describe this patch made it to
> 4.1 and the kernel I'm using is 4.4. So in my case this behavior is
> happening despite this patch being applied. One other element is that
> I'm seeing this with qlogic cards (ib_qib driver). Unfortunately I'm
> not able to pinpoint whether this is a problem of the card driver or
> with the middleware ib_ipoib driver.
>
>>
>> It really was fixed replacing PSUs in the chassis by the more powerful. It
>> appeared that Mellanox ASIC is very sensitive to the power. Under heavy
>> loading those PSUs became slightly unstable and as a result built-in switch
>> on the same PSUs produced damaged frames.
>>
>> --
>> Regards,
>> Serge
>>
>>
>> On Wed, Jul 27, 2016 at 2:05 PM, Nikolay Borisov <kernel-6AxghH7DbtA@public.gmane.org> wrote:
>>>
>>> [Resending with the linux-rdma list cc'ed + some additional information]
>>>
>>> On 07/27/2016 02:54 PM, Michael S. Tsirkin wrote:
>>> > On Wed, Jul 27, 2016 at 01:41:53PM +0300, Nikolay Borisov wrote:
>>> >> Hello,
>>> >>
>>> >> I've been running some production servers with ipoib cm but have
>>> >> observed various hangs, e.g. :
>>> >>
>>> >> http://www.spinics.net/lists/linux-rdma/msg34577.html
>>> >> http://www.spinics.net/lists/linux-rdma/msg37011.html
>>> >> http://thread.gmane.org/gmane.linux.drivers.rdma/38899
>>> >>
>>> >> Other people have also confirmed that there is a latent bug, which is
>>> >> very hard to debug (e.g. here:
>>> >> http://www.spinics.net/lists/linux-rdma/msg37022.html). Essentially
>>> >>
>>> >> As the person who originally wrote the code and considering that git
>>> >> blame indicates most of it hasn't been touched does that mean it's
>>> >> considered stable? Also do you happen to have a hunch as to what might
>>> >> be causing such stalls?
>>> >>
>>> >> Regards,
>>> >> Nikolay
>>> >
>>> > Please repost copying a mailing list.
>>> > I have a general policy against responding to off-list mail.
>>>
>>> Ok.
>>>
>>> In addition to that, here is the state of a node which has been hung for
>>> about 2 days now - no infiniband multicast connectivity, this is similar
>>> to the issue observed in the first mailing list entry I have referenced,
>>> but this time I managed to obtain the state of the ipoib_cm_rx and
>>> ib_cm_id structs (as well as any other structs which are referenced from
>>> those):
>>>
>>>
>>> struct ipoib_cm_rx {
>>>   id = 0xffff8802128fa600,
>>>   qp = 0xffff880100e94000,
>>>   rx_ring = 0x0,
>>>   list = {
>>>     next = 0xffff88055f02bdd8,
>>>     prev = 0xffff88055f02bdd8
>>>   },
>>>   dev = 0xffff880661f68000,
>>>   jiffies = 4367003834,
>>>   state = IPOIB_CM_RX_FLUSH,
>>>   recv_count = 0
>>> }
>>>
>>> struct ib_cm_id {
>>>   cm_handler = 0xffffffffa01e7b60 <ipoib_cm_rx_handler>,
>>>   context = 0xffff880660f11780,
>>>   device = 0xffff8800378e4000,
>>>   service_id = 216172782113783824,
>>>   service_mask = 18446744073709551615,
>>>   state = IB_CM_IDLE,
>>>   lap_state = IB_CM_LAP_UNINIT,
>>>   local_id = 1741978561,
>>>   remote_id = 3782023797,
>>>   remote_cm_qpn = 1
>>> }
>>>
>>> And the backtrace is like that:
>>>
>>> PID: 28224  TASK: ffff88064bdb5280  CPU: 5   COMMAND: "kworker/u24:2"
>>>  #0 [ffff88055f02bc28] __schedule at ffffffff8160fc6a
>>>  #1 [ffff88055f02bc70] schedule at ffffffff816103dc
>>>  #2 [ffff88055f02bc88] schedule_timeout at ffffffff81613642
>>>  #3 [ffff88055f02bd08] wait_for_completion at ffffffff816118df
>>>  #4 [ffff88055f02bd68] cm_destroy_id at ffffffffa01d3759 [ib_cm]
>>>  #5 [ffff88055f02bdc0] ib_destroy_cm_id at ffffffffa01d3a10 [ib_cm]
>>>  #6 [ffff88055f02bdd0] ipoib_cm_free_rx_reap_list at ffffffffa01e7675
>>> [ib_ipoib]
>>>  #7 [ffff88055f02be18] ipoib_cm_rx_reap at ffffffffa01e7705 [ib_ipoib]
>>>  #8 [ffff88055f02be28] process_one_work at ffffffff8106bdf9
>>>  #9 [ffff88055f02be68] worker_thread at ffffffff8106c4a9
>>> #10 [ffff88055f02bed0] kthread at ffffffff8107161f
>>> #11 [ffff88055f02bf50] ret_from_fork at ffffffff816149ff
>>>
>>> ffffffffa01d3759 is wait_for_completion(&cm_id_priv->comp);
>>>
>>> Can you advise what other information might be helpful to debug this ?
>>>
>>> Regards,
>>> Nikolay
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html