Distributing IPoIB interrupts

Distributing IPoIB interrupts

[Christopher Mitchell]

Hi,

One of my colleagues has been trying to benchmark the maximum
performance of Mellanox ConnectX-3 Infiniband cards using IP over IB.
He was able to use smp_affinity to distribute Ethernet packet-receive
interrupts across many cores, and specify on which cores the
interrupts occurred. With IPoIB, smp_affinity successfully moved
interrupts to a specific core, but he was unable to distribute the
IPoIB interrupts onto multiple cores. Is there a design detail of how
IPoIB works that prevents this? Does anyone know of a setting,
configuration value, or feature that we're overlooking to allow this,
or is it impossible? Thanks in advance for your feedback.

Sincerely,
Christopher
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Distributing IPoIB interrupts

[Christopher Mitchell]

Ido,

These are dual-processor AMD Opteron systems with 8 physical cores per
processor. I have libibverbs 1.1.6-1ubuntu1, libmlx4-1.0.4, and
librdmacm 1.0.15-1 and friends, which appears to correspond to OFED
3.2 (cf. http://www.openfabrics.org/downloads/OFED/release_notes/OFED_3.2_rc1_release_notes).
I misspoke: these are ConnectX-2, not 3, HCAs. Does that change
anything?

Thanks in advance,
Christopher

On Sat, Jan 12, 2013 at 5:17 AM, Ido Shamai <idos-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
> On 1/12/2013 11:46 AM, Ido Shamai wrote:
>
> On 1/11/2013 10:48 PM, Christopher Mitchell wrote:
>
> Hi,
>
> Hi Christopher,
>
> Which OFED version is being used in this case?
> Are those Sandy Brdige PCIx gen3 systems?
>
> The required technique is basically what you've described.
> Assuming Sandy Bridge:
>
> Find out which CPU NUMA node ConnectX3 sits on (for dual socket systems).
> Spread IPoIB interrupts (their names depends on OFED version usually
> something with mlx) onto cores of this node.
> Run test application affinity on those nodes as well.
> This way you should the maximum out of your system.
>
> One of my colleagues has been trying to benchmark the maximum
> performance of Mellanox ConnectX-3 Infiniband cards using IP over IB.
> He was able to use smp_affinity to distribute Ethernet packet-receive
> interrupts across many cores, and specify on which cores the
> interrupts occurred. With IPoIB, smp_affinity successfully moved
> interrupts to a specific core, but he was unable to distribute the
> IPoIB interrupts onto multiple cores. Is there a design detail of how
> IPoIB works that prevents this? Does anyone know of a setting,
> configuration value, or feature that we're overlooking to allow this,
> or is it impossible? Thanks in advance for your feedback.
>
> Sincerely,
> Christopher
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Distributing IPoIB interrupts

[Ido Shamai]

On 1/12/2013 6:52 PM, Christopher Mitchell wrote:
> Ido,
>
> These are dual-processor AMD Opteron systems with 8 physical cores per
> processor. I have libibverbs 1.1.6-1ubuntu1, libmlx4-1.0.4, and
> librdmacm 1.0.15-1 and friends, which appears to correspond to OFED
> 3.2 (cf. http://www.openfabrics.org/downloads/OFED/release_notes/OFED_3.2_rc1_release_notes).
> I misspoke: these are ConnectX-2, not 3, HCAs. Does that change
> anything?
Right.

Although in IPoIB case, those libraries are not taken in action, since 
it uses the ib_ipoib module in the kernel.
> Thanks in advance,
> Christopher
>
> On Sat, Jan 12, 2013 at 5:17 AM, Ido Shamai <idos-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
>> On 1/12/2013 11:46 AM, Ido Shamai wrote:
>>
>> On 1/11/2013 10:48 PM, Christopher Mitchell wrote:
>>
>> Hi,
>>
>> Hi Christopher,
>>
>> Which OFED version is being used in this case?
>> Are those Sandy Brdige PCIx gen3 systems?
>>
>> The required technique is basically what you've described.
>> Assuming Sandy Bridge:
>>
>> Find out which CPU NUMA node ConnectX3 sits on (for dual socket systems).
>> Spread IPoIB interrupts (their names depends on OFED version usually
>> something with mlx) onto cores of this node.
>> Run test application affinity on those nodes as well.
>> This way you should the maximum out of your system.
>>
>> One of my colleagues has been trying to benchmark the maximum
>> performance of Mellanox ConnectX-3 Infiniband cards using IP over IB.
>> He was able to use smp_affinity to distribute Ethernet packet-receive
>> interrupts across many cores, and specify on which cores the
>> interrupts occurred. With IPoIB, smp_affinity successfully moved
>> interrupts to a specific core, but he was unable to distribute the
>> IPoIB interrupts onto multiple cores. Is there a design detail of how
>> IPoIB works that prevents this? Does anyone know of a setting,
>> configuration value, or feature that we're overlooking to allow this,
>> or is it impossible? Thanks in advance for your feedback.
>>
>> Sincerely,
>> Christopher
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>>

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html