Size of Infiniband packets

Size of Infiniband packets

[Anuj Kalia]

Hi.

I'm trying to do a back-of-the-hand calculation of the number of
packets per second that can be transmitted over 40Gbps Infiniband. I
need some information about the packet size to do this calculation.

Suppose I do an 8 byte RDMA write over RC transport. What is the
packet size that is  transmitted on the wire? This should include
Infiniband headers and the work request.

I read that 256 bytes is the minimum packet size that can be
transferred. Quote from
http://www.info-ab.uclm.es/personal/FcoAlfaro/publi/ccece02.pdf:

"As already mentioned, packets with 256 and 4096 bytes have been
considered. These sizes correspond to minimum and maximum packet size
allowable in InfiniBand."

Is this correct?

Thanks for your help!

--Anuj
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Size of Infiniband packets

[Jason Gunthorpe]

On Thu, Dec 05, 2013 at 06:22:57PM -0500, Anuj Kalia wrote:

> I'm trying to do a back-of-the-hand calculation of the number of
> packets per second that can be transmitted over 40Gbps Infiniband. I
> need some information about the packet size to do this calculation.
> 
> Suppose I do an 8 byte RDMA write over RC transport. What is the
> packet size that is  transmitted on the wire? This should include
> Infiniband headers and the work request.

See page 257 (Figure 76) of IB Spec volume 1.

A RDMA write will contain:
 LRH 8
 BTH 12
 RETH 16
 data 8 
 ICRC 4 
 VCRC 2

Plus 2 bytes of framing is 52 bytes. IB is defined to sustain
continuous back-to-back packets, so
  3200MB/sec / 52 bytes/packet = 615 million pkts/sec

-- 
Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>        (780)4406067x832
Chief Technology Officer, Obsidian Research Corp         Edmonton, Canada
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Size of Infiniband packets

[Suresh Shelvapille]

It is more like 61.5Milpkts, but this is assuming PCIe-2 overhead 26Gpbs I think, but with pci3.0 we should get more
like 70mil pkts/sec I believe.
 

> -----Original Message-----
> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Jason
> Gunthorpe
> Sent: Thursday, December 05, 2013 6:54 PM
> To: Anuj Kalia
> Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Subject: Re: Size of Infiniband packets
> 
> On Thu, Dec 05, 2013 at 06:22:57PM -0500, Anuj Kalia wrote:
> 
> > I'm trying to do a back-of-the-hand calculation of the number of
> > packets per second that can be transmitted over 40Gbps Infiniband. I
> > need some information about the packet size to do this calculation.
> >
> > Suppose I do an 8 byte RDMA write over RC transport. What is the
> > packet size that is  transmitted on the wire? This should include
> > Infiniband headers and the work request.
> 
> See page 257 (Figure 76) of IB Spec volume 1.
> 
> A RDMA write will contain:
>  LRH 8
>  BTH 12
>  RETH 16
>  data 8
>  ICRC 4
>  VCRC 2
> 
> Plus 2 bytes of framing is 52 bytes. IB is defined to sustain
> continuous back-to-back packets, so
>   3200MB/sec / 52 bytes/packet = 615 million pkts/sec
> 
> --
> Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>        (780)4406067x832
> Chief Technology Officer, Obsidian Research Corp         Edmonton, Canada
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Size of Infiniband packets

[Anuj Kalia]

@Jason, Thanks! I'll read about the headers now.

@Suresh, Thanks for the correction. Sorry for the duplicate email..

On Fri, Dec 6, 2013 at 10:31 AM, Suresh Shelvapille
<suri-lFb5ksp6isWhEniVeURVKkEOCMrvLtNR@public.gmane.org> wrote:
> It is more like 61.5Milpkts, but this is assuming PCIe-2 overhead 26Gpbs I think, but with pci3.0 we should get more
> like 70mil pkts/sec I believe.
>
>
>> -----Original Message-----
>> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Jason
>> Gunthorpe
>> Sent: Thursday, December 05, 2013 6:54 PM
>> To: Anuj Kalia
>> Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> Subject: Re: Size of Infiniband packets
>>
>> On Thu, Dec 05, 2013 at 06:22:57PM -0500, Anuj Kalia wrote:
>>
>> > I'm trying to do a back-of-the-hand calculation of the number of
>> > packets per second that can be transmitted over 40Gbps Infiniband. I
>> > need some information about the packet size to do this calculation.
>> >
>> > Suppose I do an 8 byte RDMA write over RC transport. What is the
>> > packet size that is  transmitted on the wire? This should include
>> > Infiniband headers and the work request.
>>
>> See page 257 (Figure 76) of IB Spec volume 1.
>>
>> A RDMA write will contain:
>>  LRH 8
>>  BTH 12
>>  RETH 16
>>  data 8
>>  ICRC 4
>>  VCRC 2
>>
>> Plus 2 bytes of framing is 52 bytes. IB is defined to sustain
>> continuous back-to-back packets, so
>>   3200MB/sec / 52 bytes/packet = 615 million pkts/sec
>>
>> --
>> Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>        (780)4406067x832
>> Chief Technology Officer, Obsidian Research Corp         Edmonton, Canada
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html