Slow performance of IPoIB on Qlogic card

Slow performance of IPoIB on Qlogic card

[Nikolay Borisov]

Hello,


I have 2 machines which are connected via an infiniband switch and are
using the QLogic 12300 switch and InfiniPath_QLE7340 adapter. I'm
running a very naive iperf test between the 2 machines. When running an
upstream elrepo kernel 4.4.4-1.el6.elrepo.x86_64 I can do around
15-18gbit which is sort of the max in my setup. At the same time I'm
also seeing a lot of sdmaP interrupts being served.

When I boot the client iperf machine with my custom built machine I'm
barely able to get more than 4gbit/s. At the same time I'm seeing a lot
of sdmaI interrupts being served.

After some investigation it turned out that changing CONFIG_HZ to 1000
(from 100) put the performance of my custom built kernel on par with the
elrepo one. Despite iperf showing a 4x increase in bandwidth I still see
a lot of sdmaI interrupt and not much sdmaP.

Another interesting this is the output of 'perf top -g'. With
CONFIG_HZ=100 the profile is dominated by 'cpuidle_enter' while iperf is
running i.e. that my machine is almost idle. With CONFIG_HZ=1000 the
profile is dominated by qib_verbs_send/qib_do_send/worker_thread which I
believe is the expected behavior.

Given that I have the following questions:

1. What does the sdmaI/sdmaP interrupts mean. Looking around the code
and ML it seems they I stands for Idle and P for progress. But other
than that this doesn't mean much to me, can you expand on their meaning?

2. Is it bad that more sdmaI interrupts are coming rather than sdmaP.
Currently they seem to be on the same core(0)

3. Why does changing the timer tick-rate have such a tremendous impact
on the performance?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html