problem with duplicate resends

problem with duplicate resends

[Hebenstreit, Michael]

Hello

We have a problem in a Linux cluster using Omnipath 100 and GPFS. Typically, after a complete reboot the cluster works correctly for 10-14 days. Then problems start, happening about once ever 2-3 days. This makes the problem very hard to debug.

The problem starts with one or more storage nodes (A, B, C...) being unable to write to a "bad" storage node X. A/B/C/... would then throw an IBV_WC_RETRY_EXC_ERR error and close the QP pair. In response NodeX would also close the connection. Afterwards GPFS cannot re-establish a new connection fast enough and everything goes south until the NodeX is rebooted. GPFS is NOT my question here though.

During the last crash thanks to a new monitoring system, we discovered that NodeA/B/C/.. would execute 6 RDMA retries and accordingly the RcResend counters on the hfi1 driver would go up. But on NodeX the RcDupRew counter would go up in step with all the RcResends. That indicates the resends are incorrect and had already been previously acknowledged.

The operating system is RedHat EL 8.10 with a very old rdma-core version 48.

My question - is there any known bug in libibverbs/libhfi1verbs-rdmav34 that could explain this behavior?

Thanks
Michael

------------------------------------------------------------------------------
Michael Hebenstreit         Principal Performance Engineer
Cornelis Networks           Performance Team
Tel.:+1-385-393-5444        E-mail: michael.hebenstreit@cornelisnetworks.com

External recipient

Re: problem with duplicate resends

[Leon Romanovsky]

On Fri, Feb 20, 2026 at 06:48:21PM +0000, Hebenstreit, Michael wrote:
> Hello
> 
> We have a problem in a Linux cluster using Omnipath 100 and GPFS. Typically, after a complete reboot the cluster works correctly for 10-14 days. Then problems start, happening about once ever 2-3 days. This makes the problem very hard to debug.
> 
> The problem starts with one or more storage nodes (A, B, C...) being unable to write to a "bad" storage node X. A/B/C/... would then throw an IBV_WC_RETRY_EXC_ERR error and close the QP pair. In response NodeX would also close the connection. Afterwards GPFS cannot re-establish a new connection fast enough and everything goes south until the NodeX is rebooted. GPFS is NOT my question here though.
> 
> During the last crash thanks to a new monitoring system, we discovered that NodeA/B/C/.. would execute 6 RDMA retries and accordingly the RcResend counters on the hfi1 driver would go up. But on NodeX the RcDupRew counter would go up in step with all the RcResends. That indicates the resends are incorrect and had already been previously acknowledged.
> 
> The operating system is RedHat EL 8.10 with a very old rdma-core version 48.
> 
> My question - is there any known bug in libibverbs/libhfi1verbs-rdmav34 that could explain this behavior?

From the upstream perspective, the answer is no. We are not aware of any  
related issues or discussions.

Thanks

> 
> Thanks
> Michael
> 
> ------------------------------------------------------------------------------
> Michael Hebenstreit         Principal Performance Engineer
> Cornelis Networks           Performance Team
> Tel.:+1-385-393-5444        E-mail: michael.hebenstreit@cornelisnetworks.com
> 
> External recipient
>