From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5637CC433EF for ; Tue, 5 Apr 2022 16:53:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=W8aCHRUwkLv8cApJajOh8LHlwUscR9B7YjiPyAq8DXw=; b=PaXJJOOgJh3rzc GPQ643sBkKI2TntiMB6bbimrb+htcvcLJhP124fRAovmjZudwPFiKjzlps4Z2g6xMiEFBt4E8d1a5 Ioyf3uusaLUu3zFrNsO7Mt/ecmqfAWyHA7MfpMMmBh+EJXSLzIpa9JO+I8JCMwMLakwJ1hNWy5GMZ 51iE4bB/a0aAChF5aPKyBKd2sSdUHZ8sLEoW52GEqxXYUvilSsUjJGPjFq66mKpvbWuu8dHA39mcD INjy6if6+vqHXm2T4bPjeFNGEVx05FfpdtX6TBGR1tRTCHSk/05e0BEGg2UzOw3PAFFDUNHPHUAa8 QjpoW9zin3SwZ3diZXXQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nbmPB-0021N7-CU; Tue, 05 Apr 2022 16:51:45 +0000 Received: from mail-qt1-x829.google.com ([2607:f8b0:4864:20::829]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nbmP7-0021ML-4i for linux-arm-kernel@lists.infradead.org; Tue, 05 Apr 2022 16:51:42 +0000 Received: by mail-qt1-x829.google.com with SMTP id j21so12144941qta.0 for ; Tue, 05 Apr 2022 09:51:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=DZaRsWyX31QD4hA+oZZqyItqPWIM69HLGIh6hYg0oQE=; b=SKkJTRBQcHwyVWbzY1Q0fFJTZmB73lmVPAnSYhBqVVK1scWpN4Ob8A00UKCaM387r4 AfciYNdmHkeH/JlcNCEIyV1xeDIc/xO8CTsUpaQ9Nfm8m45bDswB5zCngl2vpuILNw3p CQ1zD0Vrk72EXS9N1dpOrZ0cMsRqoFy9Avyd7aRjyfzieiSJVwJ72RCA3g7oJFF6OHxk gtWdCx9YOOio9pgRwfesVCtbshpogXUCTFcLuqPtNJmArFM1fFUWc0MMu7tetlaOELml FSV82Fy2929wV82QLNafqtOMhBmRY2JlpLyGnh5N53amH3Rg+Lv9Du2jfa44f15e9Vn8 HFxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=DZaRsWyX31QD4hA+oZZqyItqPWIM69HLGIh6hYg0oQE=; b=tkEZF2vFSsk71k3JTQdV33jsLyyqGq3qGHYU5IQkZnvbU1OZl5uQX8D9EtLYpTOdSP t+sqRIGwJkBeO7vkRSrSEoXewgxdGvd7l5TjARnzAuD/QEmAsQt1+HUcn+xTN8KkHbw9 ThPssUzQE98L5EmV2NanLpCGkbiTD/83fHeAvRJ+ZntHpsK5VYNkKa3h6iLOBUByhVa9 wpELGELjxEJoF3g/BzVTXG6FX8gUsyD6/lcjI4eFU1xegR01ktMT2bKqhJi1R/iP8qWc ilCCD1GYAQDzYnvST2oYaz9p8SIP1DulXrZpTzWkShLawjRt2ovfTGZU7PiK2oWUOuiU 00bg== X-Gm-Message-State: AOAM532lReXvzccvDXL5U1PAhVo+kKPTwQBwtMUi9YlQyxdf75ihHoq/ 3YmB8rU5m/Htn9XziiALtQXCnQ== X-Google-Smtp-Source: ABdhPJyqiRVuBFpx9bRzvsG3XiF+NGW1XPncA04INZy3n5mHw4hVYOyIPXq4yrbKNu2ipSi3tYvOvg== X-Received: by 2002:a05:622a:1d5:b0:2e1:a8b8:3ee8 with SMTP id t21-20020a05622a01d500b002e1a8b83ee8mr3675968qtw.346.1649177497004; Tue, 05 Apr 2022 09:51:37 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-162-113-129.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.113.129]) by smtp.gmail.com with ESMTPSA id s13-20020a05620a0bcd00b0067afe7dd3ffsm9182055qki.49.2022.04.05.09.51.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Apr 2022 09:51:36 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1nbmP0-00D9J9-QM; Tue, 05 Apr 2022 13:51:34 -0300 Date: Tue, 5 Apr 2022 13:51:34 -0300 From: Jason Gunthorpe To: Marc Zyngier Cc: xieming , sashal@kernel.org, catalin.marinas@arm.com, linux@armlinux.org.uk, linux-kernel@vger.kernel.org, alex.williamson@redhat.com, will@kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org Subject: Re: [PATCH v2] kvm/arm64: fixed passthrough gpu into vm on arm64 Message-ID: <20220405165134.GS64706@ziepe.ca> References: <20220401090828.614167-1-xieming@kylinos.cn> <87tubcbvgk.wl-maz@kernel.org> <20220404132405.GQ64706@ziepe.ca> <87o81gc3dc.wl-maz@kernel.org> <20220404170202.GR64706@ziepe.ca> <87mtgzblez.wl-maz@kernel.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <87mtgzblez.wl-maz@kernel.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220405_095141_411991_CA3BC0E9 X-CRM114-Status: GOOD ( 34.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Apr 05, 2022 at 04:27:16PM +0100, Marc Zyngier wrote: > On Mon, 04 Apr 2022 18:02:02 +0100, > Jason Gunthorpe wrote: > > > > On Mon, Apr 04, 2022 at 03:47:11PM +0100, Marc Zyngier wrote: > > > > I'm guessing it will turn into a SBSA like thing where the ARM ARM is > > > > kind of vauge but a SOC has to implement Normal-NC in a certain way to > > > > be functional for the server market. > > > > > > The main issue is that this equivalence isn't architected, so people > > > can build whatever they want. SBSA means nothing to KVM (or Linux at > > > large), and there is currently no way to describe which devices are > > > safe to map as Normal-NC vs Device. > > > > And people have, we know of some ARM SOC's that don't work fully with > > NORMAL_NC for this usage. That is already a problem for baremetal > > Linux, let alone KVM.. > > > > That is why I likened it to SBSA - if you want to build a server SOC > > that works with existing server software, you have to support > > NORMAL_NC in this way. Even if it isn't architected. > > I see it the other way around. If it isn't architected (and in this > case not even detectable in a scalable way), it simply isn't > supportable by SW. Except the software already supports it. Catalin decided NORMAL_NC would be how Linux works in 2014 in commit de2db7432917 ("arm64: Make DMA coherent and strongly ordered mappings not executable") There are 47 places under drivers/ that call pgprot_writecombine() already, and if you make a "server" kind of chip you are likely to encounter these drivers and must support them. Linux has created a de-facto spec here. While I agree the current situation in ARM64 is not nice and could be improved, it has been supported by SW this way for a long time already. > > I didn't quite understand your other remarks though - is there a > > problem here? It seems like yes from the other thread you pointed at? > > The main issue is that we have no idea what the behaviour is on a > given implementation, and no way to even detect that for a given > device, NORMAL_NC is a memory type that won't cause any issue. I agree with this, but that is a driver problem for calling pgprot_writecombine() not a KVM problem. vfio is just another driver in this sense. We already have arch_can_pci_mmap_wc() which is a half attempt to solve this problem, but ARM64 doesn't wire it up. We've also gone far enough down this path for long enough that we can't break all the existing systems that are working this way already. So I expect any future accomodation would be some FW indication that NORMAL_NC doesn't work for pgprot_writecombine(), probably in DT and probably for an embedded focused chip. Maybe combined with a quirk list of non-working CPU IDs or something. Wire it up to arch_can_pci_mmap_wc() and you hvae something reasonable - except that none of the 47 drivers actually use this call today. Sigh. Thanks, Jason _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel