[PATCH] libibverbs: add path record definitions to sa.h

[PATCH] libibverbs: add path record definitions to sa.h

[Sean Hefty]

Add definitions for path record wire definition.  This
will be used by the librdmacm and ib_acm service, and is
exchanged with the kernel using the newer set and query
route functionality.

Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
I went with the simpler definition, rather than defining bitfields.
Once a definition is accepted into the libibverbs git tree, I will
request that the same definition be pulled into the OFED 1.5.2 codebase.

 include/infiniband/sa.h |   37 +++++++++++++++++++++++++++++++++++++
 1 files changed, 37 insertions(+), 0 deletions(-)

diff --git a/include/infiniband/sa.h b/include/infiniband/sa.h
index ec90f14..f475c66 100644
--- a/include/infiniband/sa.h
+++ b/include/infiniband/sa.h
@@ -97,4 +97,41 @@ struct ibv_sa_service_rec {
 	uint64_t      data64[2];
 };
 
+#define IB_PATH_RECORD_REVERSIBLE 0x80
+
+struct ib_path_record
+{
+	uint64_t        service_id;
+	union ibv_gid   dgid;
+	union ibv_gid   sgid;
+	uint16_t        dlid;
+	uint16_t        slid;
+	uint32_t        flowlabel_hoplimit; /* resv-31:28 flow label-27:8 hop limit-7:0*/
+	uint8_t         tclass;
+	uint8_t         reversible_numpath; /* reversible-7:7 num path-6:0 */
+	uint16_t        pkey;
+	uint16_t        qosclass_sl;        /* qos class-15:4 sl-3:0 */
+	uint8_t         mtu;                /* mtu selector-7:6 mtu-5:0 */
+	uint8_t         rate;               /* rate selector-7:6 rate-5:0 */
+	uint8_t         packetlifetime;     /* lifetime selector-7:6 lifetime-5:0 */
+	uint8_t         preference;
+	uint8_t         reserved[6];
+};
+
+#define IB_PATH_FLAG_GMP             (1<<0)
+#define IB_PATH_FLAG_PRIMARY         (1<<1)
+#define IB_PATH_FLAG_ALTERNATE       (1<<2)
+#define IB_PATH_FLAG_OUTBOUND        (1<<3)
+#define IB_PATH_FLAG_INBOUND         (1<<4)
+#define IB_PATH_FLAG_INBOUND_REVERSE (1<<5)
+#define IB_PATH_FLAG_BIDIRECTIONAL   (IB_PATH_FLAG_OUTBOUND |     \
+                                      IB_PATH_FLAG_INBOUND_REVERSE)
+
+struct ib_path_data
+{
+	uint32_t              flags;
+	uint32_t              reserved;
+	struct ib_path_record path;
+};
+
 #endif /* INFINIBAND_SA_H */



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] libibverbs: add path record definitions to sa.h

[Roland Dreier]

 > +#define IB_PATH_RECORD_REVERSIBLE 0x80
 > +
 > +struct ib_path_record

Can we use IBV_ / ibv_ prefixes here to match the rest of libibverbs?
Also having struct ibv_sa_path_rec and struct ibv_path_record is sort of
unfortunate.  Not sure if I can think of a better name here though --
ibv_raw_path_record?  ibv_packed_path_record?  Oh well...
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2] libibverbs: add path record definitions to sa.h

[Sean Hefty]

Add definitions for path record wire definition.  This
will be used by the librdmacm and ib_acm service, and is
exchanged with the kernel using the newer set and query
route functionality.

Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
Change from v1:
Used ibv/IBV prefix to match rest of libibverbs.
I used/kept the name 'path_record' to match the attribute name listed
in the spec.  I just avoided the StudlyCaps.  I don't have a strong
opinion on this.  And, FWIW, the IB management code uses ib_path_rec.

 include/infiniband/sa.h |   37 +++++++++++++++++++++++++++++++++++++
 1 files changed, 37 insertions(+), 0 deletions(-)

diff --git a/include/infiniband/sa.h b/include/infiniband/sa.h
index ec90f14..5a5f6e3 100644
--- a/include/infiniband/sa.h
+++ b/include/infiniband/sa.h
@@ -97,4 +97,41 @@ struct ibv_sa_service_rec {
 	uint64_t      data64[2];
 };
 
+#define IBV_PATH_RECORD_REVERSIBLE 0x80
+
+struct ibv_path_record
+{
+	uint64_t        service_id;
+	union ibv_gid   dgid;
+	union ibv_gid   sgid;
+	uint16_t        dlid;
+	uint16_t        slid;
+	uint32_t        flowlabel_hoplimit; /* resv-31:28 flow label-27:8 hop limit-7:0*/
+	uint8_t         tclass;
+	uint8_t         reversible_numpath; /* reversible-7:7 num path-6:0 */
+	uint16_t        pkey;
+	uint16_t        qosclass_sl;        /* qos class-15:4 sl-3:0 */
+	uint8_t         mtu;                /* mtu selector-7:6 mtu-5:0 */
+	uint8_t         rate;               /* rate selector-7:6 rate-5:0 */
+	uint8_t         packetlifetime;     /* lifetime selector-7:6 lifetime-5:0 */
+	uint8_t         preference;
+	uint8_t         reserved[6];
+};
+
+#define IBV_PATH_FLAG_GMP             (1<<0)
+#define IBV_PATH_FLAG_PRIMARY         (1<<1)
+#define IBV_PATH_FLAG_ALTERNATE       (1<<2)
+#define IBV_PATH_FLAG_OUTBOUND        (1<<3)
+#define IBV_PATH_FLAG_INBOUND         (1<<4)
+#define IBV_PATH_FLAG_INBOUND_REVERSE (1<<5)
+#define IBV_PATH_FLAG_BIDIRECTIONAL   (IBV_PATH_FLAG_OUTBOUND |     \
+                                       IBV_PATH_FLAG_INBOUND_REVERSE)
+
+struct ibv_path_data
+{
+	uint32_t               flags;
+	uint32_t               reserved;
+	struct ibv_path_record path;
+};
+
 #endif /* INFINIBAND_SA_H */



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2] libibverbs: add path record definitions to sa.h

[Sean Hefty]

Roland,

I'd like to release a new version of librdmacm that can support the user space
SA query feature in 2.6.33, which will also be part of OFED 1.5.2.  Currently,
there's a dependency on the path record definition being part of libibverbs.  Do
you have any opinions on the best way to handle this?  I guess, ideally, I'd
like to see a released version of libibverbs include this, but I can think of
ways around this.

- Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] libibverbs: add path record definitions to sa.h

[Roland Dreier]

 > I'd like to release a new version of librdmacm that can support the user space
 > SA query feature in 2.6.33, which will also be part of OFED 1.5.2.  Currently,
 > there's a dependency on the path record definition being part of libibverbs.  Do
 > you have any opinions on the best way to handle this?  I guess, ideally, I'd
 > like to see a released version of libibverbs include this, but I can think of
 > ways around this.

I can release an updated version of libibverbs (I have enough stuff
pending that this is probably a good idea anyway).  However could you do
some autoconf stuff so librdmacm works against older libibverbs (but
doesn't enable the stuff that can't be done without the missing stuff)?
Or maybe it's not worth it.
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2] libibverbs: add path record definitions to sa.h

[Sean Hefty]

>I can release an updated version of libibverbs (I have enough stuff
>pending that this is probably a good idea anyway).  However could you do
>some autoconf stuff so librdmacm works against older libibverbs (but
>doesn't enable the stuff that can't be done without the missing stuff)?
>Or maybe it's not worth it.

I'll see if I can figure out how to update the autoconf stuff correctly.
Thanks.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] libibverbs: add path record definitions to sa.h

[Roland Dreier]

thanks, applied this.  Will roll a new libibverbs release early next week.
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] libibverbs: add path record definitions to sa.h

[Steve Wise]

Roland Dreier wrote:
> thanks, applied this.  Will roll a new libibverbs release early next week.
>   

Can you add the RAW_ETY qp type in this release as well?


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] libibverbs: add path record definitions to sa.h

[Roland Dreier]

 > Can you add the RAW_ETY qp type in this release as well?

To be honest I haven't looked at the iWARP datagram stuff at all.  I'm
not sure overloading the RAW_ETY QP type is necessarily the right thing
to do -- it has quite different (never implemented) semantics in the IB
case.  Is there any overview of what you guys are planning as far as
how work requests are created for such QPs?
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] libibverbs: add path record definitions to sa.h

[Steve Wise]

Roland Dreier wrote:
>  > Can you add the RAW_ETY qp type in this release as well?
>
> To be honest I haven't looked at the iWARP datagram stuff at all.  I'm
> not sure overloading the RAW_ETY QP type is necessarily the right thing
> to do -- it has quite different (never implemented) semantics in the IB
> case.  Is there any overview of what you guys are planning as far as
> how work requests are created for such QPs?
>   

The RAW_ETY qp would be just that:  A kernel-bypass/user mode qp that 
allows sending/receiving ethernet packets.   It would also provide a way 
for user applications to join/leave ethernet mcast groups (which 
requires an rdma core kernel change that Intel posted too).  What the 
iWARP vendors are doing on top of that is implementing some form of UDP 
in user mode.  The main goal here is to provide an ultra low latency UDP 
multicast and unicast channel for important market segments that desire 
this paradigm.  Also, due to the nature of this (send/recv raw eth 
frames), the interface would be privileged.

If you want to wait, then later I'll post patches on how this is being 
done for cxgb4.  But I thought adding the RAW_ETY was definitely a 
common requirement for Intel and Chelsio.


Steve.



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2] libibverbs: add path record definitions to sa.h

[Walukiewicz, Miroslaw]

Hello Steve, 

Do you plan some changes in the core code related to RAW_QPT? 

Could you explain me better what means "priviledged interface" for you?

Regards, 

Mirek
-----Original Message-----
From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Steve Wise
Sent: Tuesday, May 18, 2010 4:04 PM
To: Roland Dreier
Cc: Hefty, Sean; linux-rdma
Subject: Re: [PATCH v2] libibverbs: add path record definitions to sa.h

Roland Dreier wrote:
>  > Can you add the RAW_ETY qp type in this release as well?
>
> To be honest I haven't looked at the iWARP datagram stuff at all.  I'm
> not sure overloading the RAW_ETY QP type is necessarily the right thing
> to do -- it has quite different (never implemented) semantics in the IB
> case.  Is there any overview of what you guys are planning as far as
> how work requests are created for such QPs?
>   

The RAW_ETY qp would be just that:  A kernel-bypass/user mode qp that 
allows sending/receiving ethernet packets.   It would also provide a way 
for user applications to join/leave ethernet mcast groups (which 
requires an rdma core kernel change that Intel posted too).  What the 
iWARP vendors are doing on top of that is implementing some form of UDP 
in user mode.  The main goal here is to provide an ultra low latency UDP 
multicast and unicast channel for important market segments that desire 
this paradigm.  Also, due to the nature of this (send/recv raw eth 
frames), the interface would be privileged.

If you want to wait, then later I'll post patches on how this is being 
done for cxgb4.  But I thought adding the RAW_ETY was definitely a 
common requirement for Intel and Chelsio.


Steve.



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] libibverbs: add path record definitions to sa.h

[Steve Wise]

Walukiewicz, Miroslaw wrote:
> Hello Steve, 
>
> Do you plan some changes in the core code related to RAW_QPT? 
>
>   


The only changes I see needed to the kernel core is the mcast change you 
already proposed to allow mcast attach/detach for RAW_ETY qps...


> Could you explain me better what means "priviledged interface" for you?
>
>   


I just mean that allocating these raw qps should only be allowed by 
effective UID 0.  This is analogous to PF_PACKET sockets which are 
privileged as well.



Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] libibverbs: add path record definitions to sa.h

[Steve Wise]

Steve Wise wrote:
> Walukiewicz, Miroslaw wrote:
>> Hello Steve,
>> Do you plan some changes in the core code related to RAW_QPT?
>>   
>
>
> The only changes I see needed to the kernel core is the mcast change 
> you already proposed to allow mcast attach/detach for RAW_ETY qps...
>


Also, There is a rdmacm kernel change to pass up iwarp L2 addresses once 
a cm_id has resolved the addresses.  I posted it earlier and I think 
Sean is going to integrate it for 2.6.36.

Steve.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2] libibverbs: add path record definitions to sa.h

[Sean Hefty]

>Also, There is a rdmacm kernel change to pass up iwarp L2 addresses once
>a cm_id has resolved the addresses.  I posted it earlier and I think
>Sean is going to integrate it for 2.6.36.

I believe that the patch you posted earlier is sufficient for 2.6.35.  I just
need to update my patch set for AF_IB support, which were written assuming that
iWarp devices did not return L2 addresses.

- Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2] libibverbs: add path record definitions to sa.h

[Walukiewicz, Miroslaw]

Hello Steve, 

I want to add a change preventing creation of the L2 RAW_QPT from user priviledge (uid = 0 will be able to do such operation) 

What is the best place to do such change: ibv_create_qp in libibverbs(verbs.c) or  allowing to  decide for NIC vendors if they want to enable such API to user or root. In that case the change is requested only for libnes library?

Regards,

Mirek

-----Original Message-----
From: Steve Wise [mailto:swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org] 
Sent: Wednesday, May 19, 2010 6:00 PM
To: Walukiewicz, Miroslaw
Cc: Roland Dreier; Hefty, Sean; linux-rdma
Subject: Re: [PATCH v2] libibverbs: add path record definitions to sa.h

Walukiewicz, Miroslaw wrote:
> Hello Steve, 
>
> Do you plan some changes in the core code related to RAW_QPT? 
>
>   


The only changes I see needed to the kernel core is the mcast change you 
already proposed to allow mcast attach/detach for RAW_ETY qps...


> Could you explain me better what means "priviledged interface" for you?
>
>   


I just mean that allocating these raw qps should only be allowed by 
effective UID 0.  This is analogous to PF_PACKET sockets which are 
privileged as well.



Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] libibverbs: add path record definitions to sa.h

[Steve Wise]

Walukiewicz, Miroslaw wrote:
> Hello Steve, 
>
> I want to add a change preventing creation of the L2 RAW_QPT from user priviledge (uid = 0 will be able to do such operation) 
>
> What is the best place to do such change: ibv_create_qp in libibverbs(verbs.c) or  allowing to  decide for NIC vendors if they want to enable such API to user or root. In that case the change is requested only for libnes library?
>
>   

I think it should be done in your kernel driver in your create_qp 
function.  The user space libraries really aren't trusted.  IE someone 
would write their own library to call into the kernel to create this 
qp.  So only in the kernel can you enforce privs.

Steve.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html