[PATCH] DAPL v2.0: common: increase default IB ack timer from 16 to 20

[PATCH] DAPL v2.0: common: increase default IB ack timer from 16 to 20

[Davis, Arlin R]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 1408 bytes --]

For larger, more congested fabrics, a larger ACK timer is needed.
Consumers can still change default with environment variable
DAPL_ACK_TIMER if they need to increase or decrease.

This applies to SCM and UCM providers only. The CMA provider, which
uses rdma_cm, has no way to control ack timer with current API.

Signed-off-by: Arlin Davis <arlin.r.davis@intel.com>
---
 dapl/openib_common/dapl_ib_common.h |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/dapl/openib_common/dapl_ib_common.h b/dapl/openib_common/dapl_ib_common.h
index a0dd772..e757b65 100644
--- a/dapl/openib_common/dapl_ib_common.h
+++ b/dapl/openib_common/dapl_ib_common.h
@@ -158,8 +158,8 @@ typedef uint16_t		ib_hca_port_t;
 #define DAT_UD_QKEY	0x78654321
 
 /* RC timer - retry count defaults */
-#define DCM_ACK_TIMER	16 /* 5 bits, 4.096us*2^ack_timer. 16== 268ms */
-#define DCM_ACK_RETRY	7  /* 3 bits, 7 * 268ms = 1.8 seconds */
+#define DCM_ACK_TIMER	20 /* 5 bits, 4.096us*2^ack_timer. 16== 268ms, 20==4.2s */
+#define DCM_ACK_RETRY	7  /* 3 bits, 7 * 4.2  == 30 seconds */
 #define DCM_RNR_TIMER	12 /* 5 bits, 12 =.64ms, 28 =163ms, 31 =491ms */
 #define DCM_RNR_RETRY	7  /* 3 bits, 7 == infinite */
 #define DCM_IB_MTU	2048
-- 
1.7.3



N‹§²æìr¸›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±­ÙšŠ{ayº\x1dʇڙë,j\a­¢f£¢·hš‹»öì\x17/oSc¾™Ú³9˜uÀ¦æå‰È&jw¨®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿïêäz¹Þ–Šàþf£¢·hšˆ§~ˆmš

RE: [PATCH] DAPL v2.0: common: increase default IB ack timer from 16 to 20

[Hefty, Sean]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 628 bytes --]

> For larger, more congested fabrics, a larger ACK timer is needed.
> Consumers can still change default with environment variable
> DAPL_ACK_TIMER if they need to increase or decrease.
> 
> This applies to SCM and UCM providers only. The CMA provider, which
> uses rdma_cm, has no way to control ack timer with current API.

The rdma_cm uses the path lifetime to determine the QP timeout value.  The path record can be manipulated by the user when calling rdma_create_ep().

- Sean

N‹§²æìr¸›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±­ÙšŠ{ayº\x1dʇڙë,j\a­¢f£¢·hš‹»öì\x17/oSc¾™Ú³9˜uÀ¦æå‰È&jw¨®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿïêäz¹Þ–Šàþf£¢·hšˆ§~ˆmš

RE: [PATCH] DAPL v2.0: common: increase default IB ack timer from 16 to 20

[Davis, Arlin R]

> > For larger, more congested fabrics, a larger ACK timer is needed.
> > Consumers can still change default with environment variable
> > DAPL_ACK_TIMER if they need to increase or decrease.
> >
> > This applies to SCM and UCM providers only. The CMA provider, which
> > uses rdma_cm, has no way to control ack timer with current API.
> 
> The rdma_cm uses the path lifetime to determine the QP timeout value.
> The path record can be manipulated by the user when calling
> rdma_create_ep().

I stand corrected. I guess I should have said "has no direct way". 
Yes, a user can modify path lifetime via rdma_create_ep() but there 
is no way for the user to know how much that will be manipulated and 
increased in the IB CM driver. 

RE: [PATCH] DAPL v2.0: common: increase default IB ack timer from 16 to 20

[Hefty, Sean]

> Yes, a user can modify path lifetime via rdma_create_ep() but there
> is no way for the user to know how much that will be manipulated and
> increased in the IB CM driver.

Sure there is.  It's an open source driver.  :)

The ib_cm calculates the "correct" timeout based on the packet lifetime provided by the SA and the CA ack timeout.  If user modifications to this value are necessary, then there is a bug in either the SA or HCA driver. 

Re: [PATCH] DAPL v2.0: common: increase default IB ack timer from 16 to 20

[Davis, Arlin R]

> > Yes, a user can modify path lifetime via rdma_create_ep() but there
> > is no way for the user to know how much that will be manipulated and
> > increased in the IB CM driver.
> 
> Sure there is.  It's an open source driver.  :)
> 
> The ib_cm calculates the "correct" timeout based on the packet lifetime
> provided by the SA and the CA ack timeout.  If user modifications to
> this value are necessary, then there is a bug in either the SA or HCA
> driver.

or there could be congestion on the fabric that the SA or HCA driver is unaware of. 

Re: [PATCH] DAPL v2.0: common: increase default IB ack timer from 16 to 20

[Jason Gunthorpe]

On Wed, Nov 02, 2011 at 11:31:38PM +0000, Davis, Arlin R wrote:
> > > Yes, a user can modify path lifetime via rdma_create_ep() but there
> > > is no way for the user to know how much that will be manipulated and
> > > increased in the IB CM driver.
> > 
> > Sure there is.  It's an open source driver.  :)
> > 
> > The ib_cm calculates the "correct" timeout based on the packet lifetime
> > provided by the SA and the CA ack timeout.  If user modifications to
> > this value are necessary, then there is a bug in either the SA or HCA
> > driver.
> 
> or there could be congestion on the fabric that the SA or HCA driver
> is unaware of.

That is what packet life time is supposed to model, and the policy
knob for it belongs in the SA, not every app..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html