From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FE40C433F5 for ; Mon, 4 Apr 2022 14:47:20 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id BB1C04B2A3; Mon, 4 Apr 2022 10:47:19 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@kernel.org Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4zMitFtRnyDo; Mon, 4 Apr 2022 10:47:18 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 72E414B2BB; Mon, 4 Apr 2022 10:47:18 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 77E884B295 for ; Mon, 4 Apr 2022 10:47:17 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wQYLW4N9dlEb for ; Mon, 4 Apr 2022 10:47:16 -0400 (EDT) Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 3C0B24B23F for ; Mon, 4 Apr 2022 10:47:16 -0400 (EDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 52F4E6157F; Mon, 4 Apr 2022 14:47:15 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id ACF52C340EE; Mon, 4 Apr 2022 14:47:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1649083634; bh=UBiyTVY3jPw/kwQGsGmG7CA9LhSqHRjUGhLmpR6GJSw=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=GS/UArGSQ0mETeN+At6tSC8ePgdvJBSwHTE9iT/5gdMwzo987FY7nWSCqUY524rPP +gwMBz5gj94MRY1pTgOhczQ5r47bOtpYUihQeZF1cFA3oNZaCVMHcgzt9YRnW7ZP3r 9NcTUeaWXvbSZZ8DYujxu7lzECSIupm08EeFr05/U8QHXdmPKpNIXZjq3cNZDf9muJ kXuz4/gX7Cd3e8JJa4UHAwsjBNaXeZD00KYXDnmuaY9/CUlHtahHxt6tWQzHD7Q06k AXEBfb7Kz+UXaU8OxmkoA31/nQwXZK8rS2QEfL5E/0l0itPNQHEyWX7F6Z4zUhVpJk kGXLORkMShjCA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nbNz5-001Wuy-Np; Mon, 04 Apr 2022 15:47:12 +0100 Date: Mon, 04 Apr 2022 15:47:11 +0100 Message-ID: <87o81gc3dc.wl-maz@kernel.org> From: Marc Zyngier To: Jason Gunthorpe Subject: Re: [PATCH v2] kvm/arm64: fixed passthrough gpu into vm on arm64 In-Reply-To: <20220404132405.GQ64706@ziepe.ca> References: <20220401090828.614167-1-xieming@kylinos.cn> <87tubcbvgk.wl-maz@kernel.org> <20220404132405.GQ64706@ziepe.ca> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: jgg@ziepe.ca, xieming@kylinos.cn, sashal@kernel.org, catalin.marinas@arm.com, linux@armlinux.org.uk, linux-kernel@vger.kernel.org, alex.williamson@redhat.com, will@kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Cc: sashal@kernel.org, catalin.marinas@arm.com, linux@armlinux.org.uk, linux-kernel@vger.kernel.org, xieming , alex.williamson@redhat.com, will@kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Mon, 04 Apr 2022 14:24:05 +0100, Jason Gunthorpe wrote: > > On Fri, Apr 01, 2022 at 05:48:59PM +0100, Marc Zyngier wrote: > > > NAK. For a start, there is no such thing as 'write-combine' in the ARM > > architecture, and I'm not convinced you can equate WC to Normal-NC. > > See the previous discussion at [1]. > > > > [1] https://lore.kernel.org/r/20210429162906.32742-1-sdonthineni@nvidia.com > > We've had a lot of discussions with ARM related to how this works with > drivers like mlx5 that use WC. > > ARM has now published some guidance on this: > > https://community.arm.com/arm-research/m/resources/1012 Nicely buried where nobody would dare looking. > > As an ecosystem we seem to be drifting toward Normal-NC for this > behavior (largely because it is what Linux does). At least that is > what we are testing and qualifing ARM CPUs against mlx5 with. > > I'm guessing it will turn into a SBSA like thing where the ARM ARM is > kind of vauge but a SOC has to implement Normal-NC in a certain way to > be functional for the server market. The main issue is that this equivalence isn't architected, so people can build whatever they want. SBSA means nothing to KVM (or Linux at large), and there is currently no way to describe which devices are safe to map as Normal-NC vs Device. We either have to take userspace's word for it, or rely on some other heuristics (do this for PCIe, but not anything else). None of which are entirely safe. Not to mention that no currently available CPU implements FEAT_DGH. M. -- Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EFD33C433EF for ; Mon, 4 Apr 2022 14:48:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Subject:Cc:To:From:Message-ID:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=wQ/mBgTVf0+auHWxVyE4OO1CeX+tU0jpkiNnXJbxGhI=; b=vxaDjXE7ILPyJ7 E8k9nRzzTBCy6+rwmvUpKKY7nGfqn7BCVG4P0/0/kJgs3qGQfjaaj+EkMRhF81CsxF8sCSpbxSRts M2fpFRJZ0OKWb03YiVmqnYu2EUlg9rCKA0wt+SlO4vIl+ivxVZJL8tGefWChoZ2v31jQxf0gIzzDv UbGrEcEUAqLUyTPpcY5UBL9vo8FD08bOrE4xpBEZxvlf/4KLJDmwUGNGTdDkObGb0oc1piqR++PVs teO9kuodD8Z/Sjjx8Myan9mAKh85lAu3x0ZfFxSMn8Jrzn7cuZfs5i2i3I3Sa+RVO6aXC8mGn+kZH FDXTfukfDaoUgC5tU+0w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nbNzE-00FPnu-BL; Mon, 04 Apr 2022 14:47:20 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nbNzA-00FPn0-HD for linux-arm-kernel@lists.infradead.org; Mon, 04 Apr 2022 14:47:18 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 52F4E6157F; Mon, 4 Apr 2022 14:47:15 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id ACF52C340EE; Mon, 4 Apr 2022 14:47:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1649083634; bh=UBiyTVY3jPw/kwQGsGmG7CA9LhSqHRjUGhLmpR6GJSw=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=GS/UArGSQ0mETeN+At6tSC8ePgdvJBSwHTE9iT/5gdMwzo987FY7nWSCqUY524rPP +gwMBz5gj94MRY1pTgOhczQ5r47bOtpYUihQeZF1cFA3oNZaCVMHcgzt9YRnW7ZP3r 9NcTUeaWXvbSZZ8DYujxu7lzECSIupm08EeFr05/U8QHXdmPKpNIXZjq3cNZDf9muJ kXuz4/gX7Cd3e8JJa4UHAwsjBNaXeZD00KYXDnmuaY9/CUlHtahHxt6tWQzHD7Q06k AXEBfb7Kz+UXaU8OxmkoA31/nQwXZK8rS2QEfL5E/0l0itPNQHEyWX7F6Z4zUhVpJk kGXLORkMShjCA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nbNz5-001Wuy-Np; Mon, 04 Apr 2022 15:47:12 +0100 Date: Mon, 04 Apr 2022 15:47:11 +0100 Message-ID: <87o81gc3dc.wl-maz@kernel.org> From: Marc Zyngier To: Jason Gunthorpe Cc: xieming , sashal@kernel.org, catalin.marinas@arm.com, linux@armlinux.org.uk, linux-kernel@vger.kernel.org, alex.williamson@redhat.com, will@kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org Subject: Re: [PATCH v2] kvm/arm64: fixed passthrough gpu into vm on arm64 In-Reply-To: <20220404132405.GQ64706@ziepe.ca> References: <20220401090828.614167-1-xieming@kylinos.cn> <87tubcbvgk.wl-maz@kernel.org> <20220404132405.GQ64706@ziepe.ca> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: jgg@ziepe.ca, xieming@kylinos.cn, sashal@kernel.org, catalin.marinas@arm.com, linux@armlinux.org.uk, linux-kernel@vger.kernel.org, alex.williamson@redhat.com, will@kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220404_074716_679732_22A68BC4 X-CRM114-Status: GOOD ( 26.61 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, 04 Apr 2022 14:24:05 +0100, Jason Gunthorpe wrote: > > On Fri, Apr 01, 2022 at 05:48:59PM +0100, Marc Zyngier wrote: > > > NAK. For a start, there is no such thing as 'write-combine' in the ARM > > architecture, and I'm not convinced you can equate WC to Normal-NC. > > See the previous discussion at [1]. > > > > [1] https://lore.kernel.org/r/20210429162906.32742-1-sdonthineni@nvidia.com > > We've had a lot of discussions with ARM related to how this works with > drivers like mlx5 that use WC. > > ARM has now published some guidance on this: > > https://community.arm.com/arm-research/m/resources/1012 Nicely buried where nobody would dare looking. > > As an ecosystem we seem to be drifting toward Normal-NC for this > behavior (largely because it is what Linux does). At least that is > what we are testing and qualifing ARM CPUs against mlx5 with. > > I'm guessing it will turn into a SBSA like thing where the ARM ARM is > kind of vauge but a SOC has to implement Normal-NC in a certain way to > be functional for the server market. The main issue is that this equivalence isn't architected, so people can build whatever they want. SBSA means nothing to KVM (or Linux at large), and there is currently no way to describe which devices are safe to map as Normal-NC vs Device. We either have to take userspace's word for it, or rely on some other heuristics (do this for PCIe, but not anything else). None of which are entirely safe. Not to mention that no currently available CPU implements FEAT_DGH. M. -- Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD6E6C433F5 for ; Mon, 4 Apr 2022 14:48:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378133AbiDDOuU (ORCPT ); Mon, 4 Apr 2022 10:50:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33480 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378117AbiDDOuB (ORCPT ); Mon, 4 Apr 2022 10:50:01 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31BE910FC0 for ; Mon, 4 Apr 2022 07:47:17 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id E82BFB8101C for ; Mon, 4 Apr 2022 14:47:15 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id ACF52C340EE; Mon, 4 Apr 2022 14:47:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1649083634; bh=UBiyTVY3jPw/kwQGsGmG7CA9LhSqHRjUGhLmpR6GJSw=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=GS/UArGSQ0mETeN+At6tSC8ePgdvJBSwHTE9iT/5gdMwzo987FY7nWSCqUY524rPP +gwMBz5gj94MRY1pTgOhczQ5r47bOtpYUihQeZF1cFA3oNZaCVMHcgzt9YRnW7ZP3r 9NcTUeaWXvbSZZ8DYujxu7lzECSIupm08EeFr05/U8QHXdmPKpNIXZjq3cNZDf9muJ kXuz4/gX7Cd3e8JJa4UHAwsjBNaXeZD00KYXDnmuaY9/CUlHtahHxt6tWQzHD7Q06k AXEBfb7Kz+UXaU8OxmkoA31/nQwXZK8rS2QEfL5E/0l0itPNQHEyWX7F6Z4zUhVpJk kGXLORkMShjCA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nbNz5-001Wuy-Np; Mon, 04 Apr 2022 15:47:12 +0100 Date: Mon, 04 Apr 2022 15:47:11 +0100 Message-ID: <87o81gc3dc.wl-maz@kernel.org> From: Marc Zyngier To: Jason Gunthorpe Cc: xieming , sashal@kernel.org, catalin.marinas@arm.com, linux@armlinux.org.uk, linux-kernel@vger.kernel.org, alex.williamson@redhat.com, will@kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org Subject: Re: [PATCH v2] kvm/arm64: fixed passthrough gpu into vm on arm64 In-Reply-To: <20220404132405.GQ64706@ziepe.ca> References: <20220401090828.614167-1-xieming@kylinos.cn> <87tubcbvgk.wl-maz@kernel.org> <20220404132405.GQ64706@ziepe.ca> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: jgg@ziepe.ca, xieming@kylinos.cn, sashal@kernel.org, catalin.marinas@arm.com, linux@armlinux.org.uk, linux-kernel@vger.kernel.org, alex.williamson@redhat.com, will@kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 04 Apr 2022 14:24:05 +0100, Jason Gunthorpe wrote: > > On Fri, Apr 01, 2022 at 05:48:59PM +0100, Marc Zyngier wrote: > > > NAK. For a start, there is no such thing as 'write-combine' in the ARM > > architecture, and I'm not convinced you can equate WC to Normal-NC. > > See the previous discussion at [1]. > > > > [1] https://lore.kernel.org/r/20210429162906.32742-1-sdonthineni@nvidia.com > > We've had a lot of discussions with ARM related to how this works with > drivers like mlx5 that use WC. > > ARM has now published some guidance on this: > > https://community.arm.com/arm-research/m/resources/1012 Nicely buried where nobody would dare looking. > > As an ecosystem we seem to be drifting toward Normal-NC for this > behavior (largely because it is what Linux does). At least that is > what we are testing and qualifing ARM CPUs against mlx5 with. > > I'm guessing it will turn into a SBSA like thing where the ARM ARM is > kind of vauge but a SOC has to implement Normal-NC in a certain way to > be functional for the server market. The main issue is that this equivalence isn't architected, so people can build whatever they want. SBSA means nothing to KVM (or Linux at large), and there is currently no way to describe which devices are safe to map as Normal-NC vs Device. We either have to take userspace's word for it, or rely on some other heuristics (do this for PCIe, but not anything else). None of which are entirely safe. Not to mention that no currently available CPU implements FEAT_DGH. M. -- Without deviation from the norm, progress is not possible.