From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.6 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8D09C432C0 for ; Tue, 19 Nov 2019 07:39:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7455C2186D for ; Tue, 19 Nov 2019 07:39:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1574149140; bh=DhlSvKBrxz4Njc9o1R2La335E0GDrmcz/MzyQYVftBs=; h=Date:From:To:Cc:Subject:References:In-Reply-To:List-ID:From; b=oq12qb64Nd7zSQKYuuueD2KmlNhN2TxQNcGdeCIBx7ykbT9TOSMFD1CGxTcxkPneb 1J2hdhMiPAU/QkNzkBDPWQ8rp8hAa+/gc0af7jnBu4jHYYC3jcJlE9ZCr+uY6IY98W jqdl+HYCorxxZp9TtxDwxQvvyAa3cYkoELyfLriU= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727345AbfKSHjA (ORCPT ); Tue, 19 Nov 2019 02:39:00 -0500 Received: from mail.kernel.org ([198.145.29.99]:60374 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725536AbfKSHi7 (ORCPT ); Tue, 19 Nov 2019 02:38:59 -0500 Received: from localhost (unknown [193.47.165.251]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 5B4AE21850; Tue, 19 Nov 2019 07:38:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1574149139; bh=DhlSvKBrxz4Njc9o1R2La335E0GDrmcz/MzyQYVftBs=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=xMIizeLive0Ny05YRpZLV96JJtTaXzpeS9/tq0RojVPMql8Xzp3u46TBwdhuYlsj0 1dbjbbvd6bd2iU6c1cs1Pk3l0nFYZdcRGgUiE9TelTEDQcBnPxsNxsOCsqRa0Rg0Zg yX52h2Emu/CFcQvo7ntoQrCotWyyTUMXLxx6+g2M= Date: Tue, 19 Nov 2019 09:38:54 +0200 From: Leon Romanovsky To: wangqi <3100102071@zju.edu.cn> Cc: linux-rdma@vger.kernel.org Subject: Re: [question]Why our soft-RoCE throughput is quite low compared with TCP Message-ID: <20191119073854.GF52766@unreal> References: <20191115160707.GG6763@unreal> <20191118094924.GA52766@unreal> <0bb80672-3980-04e7-5cf1-846b517ad53e@zju.edu.cn> <20191118122803.GC52766@unreal> <20191118144155.GE52766@unreal> <95220ca0-5df5-b70c-8f29-30410fd568c7@zju.edu.cn> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <95220ca0-5df5-b70c-8f29-30410fd568c7@zju.edu.cn> User-Agent: Mutt/1.12.1 (2019-06-15) Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On Tue, Nov 19, 2019 at 08:22:34AM +0800, wangqi wrote: > > On 2019/11/18 下午10:41, Leon Romanovsky wrote: > > On Mon, Nov 18, 2019 at 08:56:35PM +0800, wangqi wrote: > >> On 2019/11/18 下午8:28, Leon Romanovsky wrote: > >>> On Mon, Nov 18, 2019 at 06:13:07PM +0800, wangqi wrote: > >>>> On 2019/11/18 下午5:49, Leon Romanovsky wrote: > >>>>> On Mon, Nov 18, 2019 at 02:38:19PM +0800, wangqi wrote: > >>>>>> On 2019/11/16 上午12:07, Leon Romanovsky wrote: > >>>>>> > >>>>>>> On Fri, Nov 15, 2019 at 09:26:41PM +0800, QWang wrote: > >>>>>>>> Dear experts on RDMA, > >>>>>>>>       We are sorry to disturb you. Because of a project, we need to > >>>>>>>> integrate soft-RoCE in our system. However ,we are very confused by our > >>>>>>>> soft-RoCE throughput results, which are quite low compared with TCP > >>>>>>>> throughput. The throughput of soft-RoCE in our tests measured by ib_send_bw > >>>>>>>> and ib_read_bw is only 2 Gbps (the net link bandwidth is 100 Gbps and the > >>>>>>>> two Xeon E5 servers with Mellanox ConnectX-4 cards are connected via > >>>>>>>> back-to-back, the OS is ubuntu16.04 with kernel 4.15.0-041500-generic). The > >>>>>>>> throughput of hard-RoCE and TCP are normal, which are 100 Gbps and 20 Gbps, > >>>>>>>> respectively. But in the figure 6 in the attached paper "A Performance > >>>>>>>> Comparison of Container Networking Alternatives", the throughput of > >>>>>>>> soft-RoCE can be up to 23 Gbps.  In our tests, we get the open-source > >>>>>>>> soft-RoCE from github in https://github.com/linux-rdma. Do you know how can > >>>>>>>> we get such high bandwidth? Do we need to configure some OS system settings? > >>>>>>>>       We find that in 2017, someone finds the same problem and he posts all > >>>>>>>> his detailed results on https://bugzilla.kernel.org/show_bug.cgi?id=190951   > >>>>>>>> . But it remains unsolved. His results are nearly the same with our's. For > >>>>>>>> simplicity,  we do not post our results in this email. You can get very > >>>>>>>> detailed information in the web page listed above. > >>>>>>>>       We are very confused by our results. We will very appreciate it if we > >>>>>>>> can receive your early reply. Best wishes, > >>>>>>>> Wang Qi > >>>>>>> Can you please fix your email client? > >>>>>>> The email text looks like one big sentence. > >>>>>>> > >>>>>>> From the perf report attached to this bugzilla, looks like RXE does a > >>>>>>> lot of CRC32 calculations and it is consistent with what Matan said > >>>>>>> a long time ago, RXE "stuck" in ICRC calculations required by spec. > >>>>>>> > >>>>>>> I'm curios what are your CONFIG_CRYPTO_* configs? > >>>>>>> > >>>>>>> ThanksCONFIG_CRYPTO_* configs > >>>>>>> > >>>>>>> > >>>>>> I'm sorry for the editor problem in my last email. Now I use another editor. > >>>>> Now your email has extra line between lines. > >>>>> > >>>>>> We get our rdma-core and perftest from > >>>>>> > >>>>>> https://github.com/linux-rdma/rdma-core/archive/v25.0.tar.gz > >>>>>> and https://github.com/linux-rdma/perftest/archive/4.4-0.8.tar.gz, respectively. > >>>>>> > >>>>>> We attach five files to clarify our problem. > >>>>>> > >>>>>> * The first file "server_tcp_vs_softroce_performance.txt" is the results of TCP > >>>>>> > >>>>>> and softroce throughput in our two servers (connected via back to back). > >>>>>> > >>>>>> * The second file "server_CONFIG_CRYPTO_result.txt" is the > >>>>>> > >>>>>> CONFIG_CRYPTO_* config results in the two servers.. > >>>>>> > >>>>>> * The third file "server_perf.txt" is the "ib_send_bw - n 10000 192.168.0.20 > >>>>>> > >>>>>> & perf record -ags sleep 10 & wait" results in our two servers (we use > >>>>>> > >>>>>> "perf report --header >perf" to make the file). > >>>>>> > >>>>>> * The fourth file "vm_tcp_vs_softroce_performance.txt" is the results of TCP > >>>>>> > >>>>>> and softroce throughput in two virtual machines with the latest linux kernel > >>>>>> > >>>>>> 5.4.0-rc7 > >>>>>> > >>>>>> (we get the kernel from https://github.com/torvalds/linux/archive/v5.4-rc7.zip). > >>>>>> > >>>>>> * The fifth  file "vm_CONFIG_CRYPTO_result.txt" is the result in two virtual > >>>>>> > >>>>>> machines. > >>>>>> > >>>>>> * The sixth file "vm_perf.txt" is the "ib_send_bw - n 10000 192.168.122.228 > >>>>>> > >>>>>> & perf record -ags sleep 10 & wait " result in the two virtual machines. > >>>>>> > >>>>>> On the other side, we tried to use the rxe command "rxe_cfg crc disable" > >>>>> I don't see any parsing of "crc disable" in upstream variant of rxe_cfg > >>>>> and there is no such module parameter in the kernel. > >>>>> > >>>>> Thanks > >>>> We get the command "rxe_cfg crc disable" from the following webpages: > >>>> > >>>> https://www.systutorials.com/docs/linux/man/8-rxe_cfg/ > >>>> > >>>> https://www.reflectionsofthevoid.com/2011/08/ > >>>> > >>>> It may be removed in the present soft-roce edition. > >>> It was never existed in upstream variant and in the kernel you are testing. > >>> > >>>> Can you figure out why our softroce throughput is so low from the six > >>> According to the logs, it is ICRC. > >>> > >>>> files in our last email? We hope to get a much higher softroce throughput, > >>>> > >>>> like 20 Gbps in our systems (now it's only 2 Gbps, and hard-roce can be > >>>> > >>>> up to 100 Gbps in our system). > >>>> > >>>> Qi > >>>> > >>>> > >> We try to use "rxe_cfg icrc disable" and "rxe_cfg ICRC disable", but it > >> > >> seems that the performance doesn't change at all. > > Why are you continuing to try "disable" if your kernel and rdma-core don't support it? > > > > Thanks > > > >> Qi > >> > >> > >> > > Thank you. > > Can you tell us the edition of OS, kernel and softroce in your system? > > So we can install the same edition and make our softroce work well. > > By the way, can you tell me the softroce throughput (measured by > > ib_send_bw or ib_read_bw) and the TCP throughput (measured by > > iperf) in your system? I'm sorry, but I'm not running anything performance related on RXE, so I imagine that my results will be aligned with yours. RXE is used as my rapid development platform. Thanks > > Qi >