Re: intel iommu causing performance drop in mlx4 ipoib

Nikolay Borisov
2016-05-16T12:37:00Z

On 05/16/2016 01:54 PM, David Woodhouse wrote:
> On Mon, 2016-05-16 at 13:39 +0300, Nikolay Borisov wrote:
>>
>> I've observed a strange performance pathology with it when running ipoib
>> and using a naive iperf test. My setup has multiple machines with a mix
>> of qlogic/mellanox cards, connected via an QLogic 12300 switch. All of
>> the nodes are running on 4x 10Gbps. When I run a performance test and
>> the mellanox card is a server i.e it is receiving data I get very bad
>> performance. By this I mean I cannot get more than 4 gigabits per
>> second - very low. 'perf top' clearly shows that the culprit is
>> intel_map_page which is being called form the receive path
>> of the mellanox adapter:
>>
>> 84.26% 0.04% ksoftirqd/0 [kernel.kallsyms] [k] intel_map_page
>> |
>> --- intel_map_page
>> |
>> |--98.38%-- ipoib_cm_alloc_rx_skb
>
> Are you *sure* it's disabled? Can you be more specific about where the
> time is spent? intel_map_page() doesn't really do much except calling
> in to __intel_map_single()... which should return fairly much
> immediately.

Oops, turned out I had intel_iommu=on, though I'm sure some days ago I
didn't and performance suffered. I now tried with this option removed
and performance is again back to normal. All in all this driver +
intel_iommu is a bad combination I guess. Anyway, sorry for the noise
and thanks for the prompt reply.

Regards,
Nikolay