Re: Why our soft-RoCE throughput is quite low compared with TCP

[Leon Romanovsky <leon@kernel.org>]

On Fri, Nov 15, 2019 at 09:26:41PM +0800, QWang wrote:
> Dear experts on RDMA,
>       We are sorry to disturb you. Because of a project, we need to
> integrate soft-RoCE in our system. However ,we are very confused by our
> soft-RoCE throughput results, which are quite low compared with TCP
> throughput. The throughput of soft-RoCE in our tests measured by ib_send_bw
> and ib_read_bw is only 2 Gbps (the net link bandwidth is 100 Gbps and the
> two Xeon E5 servers with Mellanox ConnectX-4 cards are connected via
> back-to-back, the OS is ubuntu16.04 with kernel 4.15.0-041500-generic). The
> throughput of hard-RoCE and TCP are normal, which are 100 Gbps and 20 Gbps,
> respectively. But in the figure 6 in the attached paper "A Performance
> Comparison of Container Networking Alternatives", the throughput of
> soft-RoCE can be up to 23 Gbps.  In our tests, we get the open-source
> soft-RoCE from github in https://github.com/linux-rdma. Do you know how can
> we get such high bandwidth? Do we need to configure some OS system settings?
>       We find that in 2017, someone finds the same problem and he posts all
> his detailed results on https://bugzilla.kernel.org/show_bug.cgi?id=190951  
> . But it remains unsolved. His results are nearly the same with our's. For
> simplicity,  we do not post our results in this email. You can get very
> detailed information in the web page listed above.
>       We are very confused by our results. We will very appreciate it if we
> can receive your early reply. Best wishes,
> Wang Qi

Can you please fix your email client?
The email text looks like one big sentence.

From the perf report attached to this bugzilla, looks like RXE does a
lot of CRC32 calculations and it is consistent with what Matan said
a long time ago, RXE "stuck" in ICRC calculations required by spec.

I'm curios what are your CONFIG_CRYPTO_* configs?

Thanks

>