Re: [question]Why our soft-RoCE throughput is quite low compared with TCP

[Leon Romanovsky <leon@kernel.org>]

On Mon, Nov 18, 2019 at 08:56:35PM +0800, wangqi wrote:
>
> On 2019/11/18 下午8:28, Leon Romanovsky wrote:
> > On Mon, Nov 18, 2019 at 06:13:07PM +0800, wangqi wrote:
> >> On 2019/11/18 下午5:49, Leon Romanovsky wrote:
> >>> On Mon, Nov 18, 2019 at 02:38:19PM +0800, wangqi wrote:
> >>>> On 2019/11/16 上午12:07, Leon Romanovsky wrote:
> >>>>
> >>>>> On Fri, Nov 15, 2019 at 09:26:41PM +0800, QWang wrote:
> >>>>>> Dear experts on RDMA,
> >>>>>>       We are sorry to disturb you. Because of a project, we need to
> >>>>>> integrate soft-RoCE in our system. However ,we are very confused by our
> >>>>>> soft-RoCE throughput results, which are quite low compared with TCP
> >>>>>> throughput. The throughput of soft-RoCE in our tests measured by ib_send_bw
> >>>>>> and ib_read_bw is only 2 Gbps (the net link bandwidth is 100 Gbps and the
> >>>>>> two Xeon E5 servers with Mellanox ConnectX-4 cards are connected via
> >>>>>> back-to-back, the OS is ubuntu16.04 with kernel 4.15.0-041500-generic). The
> >>>>>> throughput of hard-RoCE and TCP are normal, which are 100 Gbps and 20 Gbps,
> >>>>>> respectively. But in the figure 6 in the attached paper "A Performance
> >>>>>> Comparison of Container Networking Alternatives", the throughput of
> >>>>>> soft-RoCE can be up to 23 Gbps.  In our tests, we get the open-source
> >>>>>> soft-RoCE from github in https://github.com/linux-rdma. Do you know how can
> >>>>>> we get such high bandwidth? Do we need to configure some OS system settings?
> >>>>>>       We find that in 2017, someone finds the same problem and he posts all
> >>>>>> his detailed results on https://bugzilla.kernel.org/show_bug.cgi?id=190951  
> >>>>>> . But it remains unsolved. His results are nearly the same with our's. For
> >>>>>> simplicity,  we do not post our results in this email. You can get very
> >>>>>> detailed information in the web page listed above.
> >>>>>>       We are very confused by our results. We will very appreciate it if we
> >>>>>> can receive your early reply. Best wishes,
> >>>>>> Wang Qi
> >>>>> Can you please fix your email client?
> >>>>> The email text looks like one big sentence.
> >>>>>
> >>>>> From the perf report attached to this bugzilla, looks like RXE does a
> >>>>> lot of CRC32 calculations and it is consistent with what Matan said
> >>>>> a long time ago, RXE "stuck" in ICRC calculations required by spec.
> >>>>>
> >>>>> I'm curios what are your CONFIG_CRYPTO_* configs?
> >>>>>
> >>>>> ThanksCONFIG_CRYPTO_* configs
> >>>>>
> >>>>>
> >>>> I'm sorry for the editor problem in my last email. Now I use another editor.
> >>> Now your email has extra line between lines.
> >>>
> >>>> We get our rdma-core and perftest from
> >>>>
> >>>> https://github.com/linux-rdma/rdma-core/archive/v25.0.tar.gz
> >>>> and https://github.com/linux-rdma/perftest/archive/4.4-0.8.tar.gz, respectively.
> >>>>
> >>>> We attach five files to clarify our problem.
> >>>>
> >>>> * The first file "server_tcp_vs_softroce_performance.txt" is the results of TCP
> >>>>
> >>>> and softroce throughput in our two servers (connected via back to back).
> >>>>
> >>>> * The second file "server_CONFIG_CRYPTO_result.txt" is the
> >>>>
> >>>> CONFIG_CRYPTO_* config results in the two servers..
> >>>>
> >>>> * The third file "server_perf.txt" is the "ib_send_bw - n 10000 192.168.0.20
> >>>>
> >>>> & perf record -ags sleep 10 & wait" results in our two servers (we use
> >>>>
> >>>> "perf report --header >perf" to make the file).
> >>>>
> >>>> * The fourth file "vm_tcp_vs_softroce_performance.txt" is the results of TCP
> >>>>
> >>>> and softroce throughput in two virtual machines with the latest linux kernel
> >>>>
> >>>> 5.4.0-rc7
> >>>>
> >>>> (we get the kernel from https://github.com/torvalds/linux/archive/v5.4-rc7.zip).
> >>>>
> >>>> * The fifth  file "vm_CONFIG_CRYPTO_result.txt" is the result in two virtual
> >>>>
> >>>> machines.
> >>>>
> >>>> * The sixth file "vm_perf.txt" is the "ib_send_bw - n 10000 192.168.122.228
> >>>>
> >>>> & perf record -ags sleep 10 & wait " result in the two virtual machines.
> >>>>
> >>>> On the other side, we tried to use the rxe command "rxe_cfg crc disable"
> >>> I don't see any parsing of "crc disable" in upstream variant of rxe_cfg
> >>> and there is no such module parameter in the kernel.
> >>>
> >>> Thanks
> >>
> >> We get the command "rxe_cfg crc disable" from the following webpages:
> >>
> >> https://www.systutorials.com/docs/linux/man/8-rxe_cfg/
> >>
> >> https://www.reflectionsofthevoid.com/2011/08/
> >>
> >> It may be removed in the present soft-roce edition.
> > It was never existed in upstream variant and in the kernel you are testing.
> >
> >> Can you figure out why our softroce throughput is so low from the six
> > According to the logs, it is ICRC.
> >
> >> files in our last email? We hope to get a much higher softroce throughput,
> >>
> >> like 20 Gbps in our systems (now it's only 2 Gbps, and hard-roce can be
> >>
> >> up to 100 Gbps in our system).
> >>
> >> Qi
> >>
> >>
> We try to use "rxe_cfg icrc disable" and "rxe_cfg ICRC disable", but it
>
> seems that the performance doesn't change at all.

Why are you continuing to try "disable" if your kernel and rdma-core don't support it?

Thanks

>
> Qi
>
>
>