Re: [PATCH v2] rdma/ib_pack.h: add new bth opcodes

Re: [PATCH v2] rdma/ib_pack.h: add new bth opcodes

[Or Gerlitz]

Bob Pearson wrote:
> I was wondering if I could use this to cause ConnectX RDMAoE senders to slow down 
> in response to these packets. There is a challenge managing fast ROCE senders 
> in networks that may not fully implement per priority pause.
>   
Hi Bob,

QCN (IEEE 802.1 based Ethernet congestion control mechanism) can apply 
for IBoE traffic, in the same manner it would for FCoE, IP etc. Is there 
a specific reason you wanted to apply the IB mechanism and not use the 
Ethernet one? Yep, PFC is helpful.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] rdma/ib_pack.h: add new bth opcodes

[Bob Pearson]

On 08/12/2010 01:35 AM, Or Gerlitz wrote:
> Bob Pearson wrote:
>> I was wondering if I could use this to cause ConnectX RDMAoE senders 
>> to slow down in response to these packets. There is a challenge 
>> managing fast ROCE senders in networks that may not fully implement 
>> per priority pause.
> Hi Bob,
>
> QCN (IEEE 802.1 based Ethernet congestion control mechanism) can apply 
> for IBoE traffic, in the same manner it would for FCoE, IP etc. Is 
> there a specific reason you wanted to apply the IB mechanism and not 
> use the Ethernet one? Yep, PFC is helpful.
>
> Or.
>
Hi Or,

As I mentioned I did not pursue this. I was curious to see if I could 
force a ConnectX device to slow down from a remote application. But 
since the MADs have been crippled for IBOE there is no way to configure it.

Bob
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] rdma/ib_pack.h: add new bth opcodes

[Or Gerlitz]

Bob Pearson wrote:
> I was curious to see if I could force a ConnectX device to slow down from a remote application.
> But since the MADs have been crippled for IBOE there is no way to configure it.

QP1 MADs are working for ConnectX, e.g the IB CM is fully functional for IBoE, and I don't think the 
mad layer was modified to emulate MADs for the CM over regular UD QP, UDP or their such, Eli, 
am I correct in that? For some reason the PMA (QP1 performance counters) service isn't exposed, but
it should be working (and helpful) as well.

Or.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] rdma/ib_pack.h: add new bth opcodes

[Eli Cohen]

On Thu, Aug 19, 2010 at 11:54:17AM +0300, Or Gerlitz wrote:
> Bob Pearson wrote:
> > I was curious to see if I could force a ConnectX device to slow down from a remote application.
> > But since the MADs have been crippled for IBOE there is no way to configure it.
> 
> QP1 MADs are working for ConnectX, e.g the IB CM is fully functional for IBoE, and I don't think the 
> mad layer was modified to emulate MADs for the CM over regular UD QP, UDP or their such, Eli, 
> am I correct in that?
Right. QP1 is supported for kernel consumers.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IBoE and congestion management

[Or Gerlitz]

Fredy Neeser wrote:
> Not sure what you mean by "IB CM is fully functional for IBoE"  with 
> ConnectX.
>
The IB CM (Connection Manager) is used for the connection establishment, 
it has nothing do to with Congestion Management. I mentioned it in the 
context of IB MAD (Management Datagrams) support by IBoE. I believe that 
the RXE driver (IBoE hw emulation on regular Eth nics) emulates mads for 
the CM to work.
> Is your solution based on IB CCA?  If so, how will it interact with 
> 802.1Qau Congestion Notification?
>
I believe that IB CCA is not the correct path to go, but rather QCN 
which you've mentioned.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

FW: [PATCH v2] rdma/ib_pack.h: add new bth opcodes

[Robert Pearson]

Resending. Managed to mangle the patch in the first send.
Please ignore the previous send.

-----Original Message-----
From: Robert Pearson [mailto:rpearson-klaOcWyJdxkshyMvu7JE4pqQE7yCjDx5@public.gmane.org] 
Sent: Monday, August 09, 2010 12:00 AM
To: 'linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org'
Subject: [PATCH] rdma/ib_pack.h: add new bth opcodes

Several new opcodes have been added since the last time ib_pack.h was
updated.
These changes add them.

Signed-off-by: Bob Pearson <rpearson-klaOcWyJdxkshyMvu7JE4pqQE7yCjDx5@public.gmane.org>
---

diff --git a/include/rdma/ib_pack.h b/include/rdma/ib_pack.h
index cbb50f4..df10acc 100644
--- a/include/rdma/ib_pack.h
+++ b/include/rdma/ib_pack.h
@@ -73,8 +73,11 @@ enum {
        IB_OPCODE_UC                                = 0x20,
        IB_OPCODE_RD                                = 0x40,
        IB_OPCODE_UD                                = 0x60,
+       IB_OPCODE_CN                                = 0x80,
+       IB_OPCODE_XRC                               = 0xA0,

        /* operations -- just used to define real constants */
+       IB_OPCODE_CNP                               = 0x00,
        IB_OPCODE_SEND_FIRST                        = 0x00,
        IB_OPCODE_SEND_MIDDLE                       = 0x01,
        IB_OPCODE_SEND_LAST                         = 0x02,
@@ -96,6 +99,9 @@ enum {
        IB_OPCODE_ATOMIC_ACKNOWLEDGE                = 0x12,
        IB_OPCODE_COMPARE_SWAP                      = 0x13,
        IB_OPCODE_FETCH_ADD                         = 0x14,
+       IB_OPCODE_RESYNC                            = 0x15,
+       IB_OPCODE_SEND_LAST_WITH_INVALIDATE         = 0x16,
+       IB_OPCODE_SEND_ONLY_WITH_INVALIDATE         = 0x17,

        /* real constants follow -- see comment about above IB_OPCODE()
           macro for more details */
@@ -122,6 +128,8 @@ enum {
        IB_OPCODE(RC, ATOMIC_ACKNOWLEDGE),
        IB_OPCODE(RC, COMPARE_SWAP),
        IB_OPCODE(RC, FETCH_ADD),
+       IB_OPCODE(RC, SEND_LAST_WITH_INVALIDATE),
+       IB_OPCODE(RC, SEND_ONLY_WITH_INVALIDATE),

        /* UC */
        IB_OPCODE(UC, SEND_FIRST),
@@ -159,10 +167,39 @@ enum {
        IB_OPCODE(RD, ATOMIC_ACKNOWLEDGE),
        IB_OPCODE(RD, COMPARE_SWAP),
        IB_OPCODE(RD, FETCH_ADD),
+       IB_OPCODE(RD, RESYNC),

        /* UD */
        IB_OPCODE(UD, SEND_ONLY),
-       IB_OPCODE(UD, SEND_ONLY_WITH_IMMEDIATE)
+       IB_OPCODE(UD, SEND_ONLY_WITH_IMMEDIATE),
+
+       /* CN */
+       IB_OPCODE(CN, CNP),
+
+       /* XRC */
+       IB_OPCODE(XRC, SEND_FIRST),
+       IB_OPCODE(XRC, SEND_MIDDLE),
+       IB_OPCODE(XRC, SEND_LAST),
+       IB_OPCODE(XRC, SEND_LAST_WITH_IMMEDIATE),
+       IB_OPCODE(XRC, SEND_ONLY),
+       IB_OPCODE(XRC, SEND_ONLY_WITH_IMMEDIATE),
+       IB_OPCODE(XRC, RDMA_WRITE_FIRST),
+       IB_OPCODE(XRC, RDMA_WRITE_MIDDLE),
+       IB_OPCODE(XRC, RDMA_WRITE_LAST),
+       IB_OPCODE(XRC, RDMA_WRITE_LAST_WITH_IMMEDIATE),
+       IB_OPCODE(XRC, RDMA_WRITE_ONLY),
+       IB_OPCODE(XRC, RDMA_WRITE_ONLY_WITH_IMMEDIATE),
+       IB_OPCODE(XRC, RDMA_READ_REQUEST),
+       IB_OPCODE(XRC, RDMA_READ_RESPONSE_FIRST),
+       IB_OPCODE(XRC, RDMA_READ_RESPONSE_MIDDLE),
+       IB_OPCODE(XRC, RDMA_READ_RESPONSE_LAST),
+       IB_OPCODE(XRC, RDMA_READ_RESPONSE_ONLY),
+       IB_OPCODE(XRC, ACKNOWLEDGE),
+       IB_OPCODE(XRC, ATOMIC_ACKNOWLEDGE),
+       IB_OPCODE(XRC, COMPARE_SWAP),
+       IB_OPCODE(XRC, FETCH_ADD),
+       IB_OPCODE(XRC, SEND_LAST_WITH_INVALIDATE),
+       IB_OPCODE(XRC, SEND_ONLY_WITH_INVALIDATE),
 };

 enum {

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2] rdma/ib_pack.h: add new bth opcodes

[Hefty, Sean]

> Several new opcodes have been added since the last time ib_pack.h was
> updated.
> These changes add them.

Will anything make use of these?

> diff --git a/include/rdma/ib_pack.h b/include/rdma/ib_pack.h
> index cbb50f4..df10acc 100644
> --- a/include/rdma/ib_pack.h
> +++ b/include/rdma/ib_pack.h
> @@ -73,8 +73,11 @@ enum {
>         IB_OPCODE_UC                                = 0x20,
>         IB_OPCODE_RD                                = 0x40,
>         IB_OPCODE_UD                                = 0x60,
> +       IB_OPCODE_CN                                = 0x80,
> +       IB_OPCODE_XRC                               = 0xA0,

The XRC and RD values all look correct, but I must have fallen asleep and missed something.  What's CN?

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] rdma/ib_pack.h: add new bth opcodes

[Bob Pearson]

On 08/09/2010 06:36 PM, Hefty, Sean wrote:
>> Several new opcodes have been added since the last time ib_pack.h was
>> updated.
>> These changes add them.
>>      
> Will anything make use of these?
>
>    
>> diff --git a/include/rdma/ib_pack.h b/include/rdma/ib_pack.h
>> index cbb50f4..df10acc 100644
>> --- a/include/rdma/ib_pack.h
>> +++ b/include/rdma/ib_pack.h
>> @@ -73,8 +73,11 @@ enum {
>>          IB_OPCODE_UC                                = 0x20,
>>          IB_OPCODE_RD                                = 0x40,
>>          IB_OPCODE_UD                                = 0x60,
>> +       IB_OPCODE_CN                                = 0x80,
>> +       IB_OPCODE_XRC                               = 0xA0,
>>      
> The XRC and RD values all look correct, but I must have fallen asleep and missed something.  What's CN
>    
> - Sean
>
>    

Or also asked this question. Thanks to both of you.

CN is (backwards) congestion notification. It has a single opcode and 
caries no payload. It is described in an annex in IBA Vol 1.
I started adding the opcodes for send_xxx_with_invalidate but figured we 
may as well capture all of them.
My interest is supporting the rxe driver, a software implementation of 
the IB transport over Ethernet, and we have
looked at supporting xrc as an option, and I spent a little time looking 
at trying to exploit congestion notification to see if it
would bu useful in this context.

Bob Pearson

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] rdma/ib_pack.h: add new bth opcodes

[Or Gerlitz]

Bob Pearson wrote:
> My interest is supporting the rxe driver, a software implementation of
> the IB transport over Ethernet, [...] I spent a little time looking at
> trying to exploit congestion notification to see if it would bu useful in this context.

Hi Bob,

As the IB congestion control / notification has the part of the IB switches marking 
packets with FECNs, I don't see how does IB CCA fits into IBoE scheme, Paul?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html