Write Packets to InfiniBand HCA

Write Packets to InfiniBand HCA

[Greg I Kerr]

Hi,

Does anyone know if it is or isn't possible to put the HCA in some
kind of "raw" mode where I can compose a packet in software and write
it to the card? This is obviously in comparison to calling
ibv_post_send and having a work request converted into a packet.

Thanks,

Greg Kerr
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Write Packets to InfiniBand HCA

[Mike Heinz]

That would depend on which HCA you are using. I know that you can use QLogic's iPath interface to do what you want, but I don't think it is possible through the verbs interface available through stock OFED.

-----Original Message-----
From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Greg I Kerr
Sent: Monday, January 02, 2012 9:50 PM
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: Write Packets to InfiniBand HCA

Hi,

Does anyone know if it is or isn't possible to put the HCA in some kind of "raw" mode where I can compose a packet in software and write it to the card? This is obviously in comparison to calling ibv_post_send and having a work request converted into a packet.

Thanks,

Greg Kerr
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at  http://vger.kernel.org/majordomo-info.html


This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Write Packets to InfiniBand HCA

[Greg I Kerr]

Yes I should have mentioned that I am using a Mellanox Connect-X
adapter. Do you know where I can find documentation for QLogic's
iPath? A quick google search didn't seem to turn anything up.

Thanks for the information.

- Greg Kerr

On Tue, Jan 3, 2012 at 9:53 AM, Mike Heinz <michael.heinz-h88ZbnxC6KDQT0dZR+AlfA@public.gmane.org> wrote:
> That would depend on which HCA you are using. I know that you can use QLogic's iPath interface to do what you want, but I don't think it is possible through the verbs interface available through stock OFED.
>
> -----Original Message-----
> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner@vger.kernel.org] On Behalf Of Greg I Kerr
> Sent: Monday, January 02, 2012 9:50 PM
> To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Subject: Write Packets to InfiniBand HCA
>
> Hi,
>
> Does anyone know if it is or isn't possible to put the HCA in some kind of "raw" mode where I can compose a packet in software and write it to the card? This is obviously in comparison to calling ibv_post_send and having a work request converted into a packet.
>
> Thanks,
>
> Greg Kerr
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
> This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Write Packets to InfiniBand HCA

[Roland Dreier]

It is possible with ConnectX (cf "MLX QPs" in the kernel driver).
However I don't know what documentation
is available and some hacking would be needed to use this for
something more general than sending
MADs on special QPs.

On Tue, Jan 3, 2012 at 8:43 AM, Greg I Kerr <kerr.g-movQPkccWJngpn9g0Uvcdg@public.gmane.org> wrote:
> Yes I should have mentioned that I am using a Mellanox Connect-X
> adapter. Do you know where I can find documentation for QLogic's
> iPath? A quick google search didn't seem to turn anything up.
>
> Thanks for the information.
>
> - Greg Kerr
>
> On Tue, Jan 3, 2012 at 9:53 AM, Mike Heinz <michael.heinz-h88ZbnxC6KDQT0dZR+AlfA@public.gmane.org> wrote:
>> That would depend on which HCA you are using. I know that you can use QLogic's iPath interface to do what you want, but I don't think it is possible through the verbs interface available through stock OFED.
>>
>> -----Original Message-----
>> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner@vger.kernel.org] On Behalf Of Greg I Kerr
>> Sent: Monday, January 02, 2012 9:50 PM
>> To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> Subject: Write Packets to InfiniBand HCA
>>
>> Hi,
>>
>> Does anyone know if it is or isn't possible to put the HCA in some kind of "raw" mode where I can compose a packet in software and write it to the card? This is obviously in comparison to calling ibv_post_send and having a work request converted into a packet.
>>
>> Thanks,
>>
>> Greg Kerr
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>> This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Write Packets to InfiniBand HCA

[Mike Heinz]

Greg,

I don't think the iPath interface is documented outside the headers, which are included as part of the QLogic PSM package. I believe that package is included with stock OFED. iPath is basically the hardware interface at the very bottom of the QLogic software stack, it wasn't intended for normal use.

-----Original Message-----
From: Roland Dreier [mailto:roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org]
Sent: Tuesday, January 03, 2012 6:41 PM
To: Greg I Kerr
Cc: Mike Heinz; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: Re: Write Packets to InfiniBand HCA

It is possible with ConnectX (cf "MLX QPs" in the kernel driver).
However I don't know what documentation
is available and some hacking would be needed to use this for something more general than sending MADs on special QPs.

On Tue, Jan 3, 2012 at 8:43 AM, Greg I Kerr <kerr.g-movQPkccWJngpn9g0Uvcdg@public.gmane.org> wrote:
> Yes I should have mentioned that I am using a Mellanox Connect-X
> adapter. Do you know where I can find documentation for QLogic's
> iPath? A quick google search didn't seem to turn anything up.
>
> Thanks for the information.
>
> - Greg Kerr
>
> On Tue, Jan 3, 2012 at 9:53 AM, Mike Heinz <michael.heinz-h88ZbnxC6KDQT0dZR+AlfA@public.gmane.org> wrote:
>> That would depend on which HCA you are using. I know that you can use QLogic's iPath interface to do what you want, but I don't think it is possible through the verbs interface available through stock OFED.
>>
>> -----Original Message-----
>> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Greg I Kerr
>> Sent: Monday, January 02, 2012 9:50 PM
>> To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> Subject: Write Packets to InfiniBand HCA
>>
>> Hi,
>>
>> Does anyone know if it is or isn't possible to put the HCA in some kind of "raw" mode where I can compose a packet in software and write it to the card? This is obviously in comparison to calling ibv_post_send and having a work request converted into a packet.
>>
>> Thanks,
>>
>> Greg Kerr
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma"
>> in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo
>> info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>> This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma"
> in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo
> info at  http://vger.kernel.org/majordomo-info.html


This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Write Packets to InfiniBand HCA

[Or Gerlitz]

Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:
> It is possible with ConnectX (cf "MLX QPs" in the kernel driver). However I don't
> know what documentation is available and some hacking would be needed to use this for
> something more general than sending MADs on special QPs.

Greg, RAW Ethernet QPs (soon to be re-submitted upstream) which to be
implemented within the ConnnetX / mlx4 driver as MLX transport QPs
would allow you to do  that - over Ethernet.

If you're asking on IB, I would suggest using UD QPs which are
available today from user space, do you have any issue with them?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Write Packets to InfiniBand HCA

[Greg I Kerr]

Thanks for the help everyone.
>
> Greg, RAW Ethernet QPs (soon to be re-submitted upstream) which to be
> implemented within the ConnnetX / mlx4 driver as MLX transport QPs
> would allow you to do  that - over Ethernet.
>
> If you're asking on IB, I would suggest using UD QPs which are
> available today from user space, do you have any issue with them?
>

Yes I want to do this with IB. I wasn't aware that I could use a UD QP
to write my own packets to the wire. Is this what you're suggesting I
can use a UD QP for?

Thanks,

Greg

On Wed, Jan 4, 2012 at 4:03 PM, Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:
>> It is possible with ConnectX (cf "MLX QPs" in the kernel driver). However I don't
>> know what documentation is available and some hacking would be needed to use this for
>> something more general than sending MADs on special QPs.

> Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Write Packets to InfiniBand HCA

[Or Gerlitz]

On 1/4/2012 11:23 PM, Greg I Kerr wrote:
> Yes I want to do this with IB. I wasn't aware that I could use a UD QP 
> to write my own packets to the wire. Is this what you're suggesting I 
> can use a UD QP for?

You'll have to further clarify the "my own packets" part of your 
sentence... IB have some layers which come into play in various elements 
(software/hardware of HCAs/switches) during the life cycle of the 
packet. Specifically each IB packet would have an IB L2 header called 
LRH, IB also has L3 header named GRH which is optional unless you need 
multicast (or IBoE). The UD thing I mentioned is IB's L4 - the 
transport. The most common transports are UD and RC. So if you go UD the 
packet will have L2 (LRH) and L4 UD headers (called BTH plus DTH) which 
on top of you add the payload.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html