From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.9 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C7B5C47083 for ; Wed, 2 Jun 2021 12:49:14 +0000 (UTC) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 08834613D6 for ; Wed, 2 Jun 2021 12:49:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 08834613D6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id C3CFD824A8; Wed, 2 Jun 2021 12:49:13 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4TT2P5Tm2Jrx; Wed, 2 Jun 2021 12:49:09 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTP id 483A382A57; Wed, 2 Jun 2021 12:49:09 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1F30AC000E; Wed, 2 Jun 2021 12:49:09 +0000 (UTC) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 53A55C0001 for ; Wed, 2 Jun 2021 12:49:07 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 3ACDF402E2 for ; Wed, 2 Jun 2021 12:49:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RyjT2hI-59ap for ; Wed, 2 Jun 2021 12:49:03 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp2.osuosl.org (Postfix) with ESMTP id 0C0FE4002B for ; Wed, 2 Jun 2021 12:49:02 +0000 (UTC) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 07AC46D; Wed, 2 Jun 2021 05:49:02 -0700 (PDT) Received: from [10.57.73.64] (unknown [10.57.73.64]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 080E53F719; Wed, 2 Jun 2021 05:48:59 -0700 (PDT) Subject: Re: Regression 5.12.0-rc4 net: ice: significant throughput drop To: Daniel Borkmann , Jussi Maki References: <7f048c57-423b-68ba-eede-7e194c1fea4e@arm.com> <7c04eeea-22d3-c265-8e1e-b3f173f2179f@iogearbox.net> From: Robin Murphy Message-ID: <705f90c3-b933-8863-2124-3fea7fdbd81a@arm.com> Date: Wed, 2 Jun 2021 13:48:54 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Thunderbird/78.10.1 MIME-Version: 1.0 In-Reply-To: <7c04eeea-22d3-c265-8e1e-b3f173f2179f@iogearbox.net> Content-Language: en-GB Cc: jroedel@suse.de, netdev@vger.kernel.org, jesse.brandeburg@intel.com, hch@lst.de, iommu@lists.linux-foundation.org, anthony.l.nguyen@intel.com, gregkh@linuxfoundation.org, intel-wired-lan@lists.osuosl.org, bpf , davem@davemloft.net X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" On 2021-06-02 09:09, Daniel Borkmann wrote: > On 6/1/21 7:42 PM, Jussi Maki wrote: >> Hi Robin, >> >> On Tue, Jun 1, 2021 at 2:39 PM Robin Murphy wrote: >>>>> The regression shows as a significant drop in throughput as measured >>>>> with "super_netperf" [0], >>>>> with measured bandwidth of ~95Gbps before and ~35Gbps after: >>> >>> I guess that must be the difference between using the flush queue >>> vs. strict invalidation. On closer inspection, it seems to me that >>> there's a subtle pre-existing bug in the AMD IOMMU driver, in that >>> amd_iommu_init_dma_ops() actually runs *after* amd_iommu_init_api() >>> has called bus_set_iommu(). Does the patch below work? >> >> Thanks for the quick response & patch. I tried it out and indeed it >> does solve the issue: Cool, thanks Jussi. May I infer a Tested-by tag from that? >> # uname -a >> Linux zh-lab-node-3 5.13.0-rc3-amd-iommu+ #31 SMP Tue Jun 1 17:12:57 >> UTC 2021 x86_64 x86_64 x86_64 GNU/Linux >> root@zh-lab-node-3:~# ./super_netperf 32 -H 172.18.0.2 >> 95341.2 >> >> root@zh-lab-node-3:~# uname -a >> Linux zh-lab-node-3 5.13.0-rc3-amd-iommu-unpatched #32 SMP Tue Jun 1 >> 17:29:34 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux >> root@zh-lab-node-3:~# ./super_netperf 32 -H 172.18.0.2 >> 33989.5 > > Robin, probably goes without saying, but please make sure to include ... > > Fixes: a250c23f15c2 ("iommu: remove DOMAIN_ATTR_DMA_USE_FLUSH_QUEUE") > > ... to your fix in [0], maybe along with another Fixes tag pointing to > the original > commit adding this issue. But certainly a250c23f15c2 would be good given > the regression > was uncovered on that one first, so that Greg et al have a chance to > pick this fix up > for stable kernels. Given that the race looks to have been pretty theoretical until now, I'm not convinced it's worth the bother of digging through the long history of default domain and DMA ops movement to figure where it started, much less attempt invasive backports. The flush queue change which made it apparent only landed in 5.13-rc1, so as long as we can get this in as a fix in the current cycle we should be golden - in the meantime, note that booting with "iommu.strict=0" should also restore the expected behaviour. FWIW I do still plan to resend the patch "properly" soon (in all honesty it wasn't even compile-tested!) Cheers, Robin. _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu