From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8AB66EE801D for ; Fri, 8 Sep 2023 16:41:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ZWmLls2S61h5DxzDx8eTfOlWXmBhtPNUsXB6a+C8Hw4=; b=aw+J2ziyNTx1A2 e6upBMgNICWZ5ozLMb3FNcwmtITd1Yd7IPuk+v7gs2BkyflDAwmZwQ38Io5CJFL0JI1O/RP1IU17C wT/aJ1Z7qiMu/4ppmXJd92Vt2r0FAyvaJpldPmRdzhUkzi5cp+7fSxfGlhvxpc6DMcEWQs/rKdVoo bTIVm/327jh9e6ZYq4NGvCbUuqIQeR9HNY7hEdxGEEg2jNNjWMJAXFJ4lGzPAztHTDekRGnRxQDZI Lv8X6HTd++Ty6G4G18Hb6QpMl6KJPhK1ifaDsZghmmBL86cNHSO48uI4OgYphRdp6P0Y/orKhPV4N oo4amJTKY8M8VXg132jQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qeeXf-00E68i-2q; Fri, 08 Sep 2023 16:41:11 +0000 Received: from sin.source.kernel.org ([145.40.73.55]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qeeXZ-00E67Z-21 for linux-arm-kernel@lists.infradead.org; Fri, 08 Sep 2023 16:41:10 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id E1BB3CE1C29; Fri, 8 Sep 2023 16:41:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4839DC433C9; Fri, 8 Sep 2023 16:40:58 +0000 (UTC) Date: Fri, 8 Sep 2023 17:40:55 +0100 From: Catalin Marinas To: ankita@nvidia.com Cc: jgg@nvidia.com, maz@kernel.org, oliver.upton@linux.dev, will@kernel.org, aniketa@nvidia.com, cjia@nvidia.com, kwankhede@nvidia.com, targupta@nvidia.com, vsethi@nvidia.com, acurrid@nvidia.com, apopple@nvidia.com, jhubbard@nvidia.com, danw@nvidia.com, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, Lorenzo Pieralisi Subject: Re: [PATCH v1 2/2] KVM: arm64: allow the VM to select DEVICE_* and NORMAL_NC for IO memory Message-ID: References: <20230907181459.18145-1-ankita@nvidia.com> <20230907181459.18145-3-ankita@nvidia.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20230907181459.18145-3-ankita@nvidia.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230908_094105_868778_6028ABCB X-CRM114-Status: GOOD ( 22.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org + Lorenzo On Thu, Sep 07, 2023 at 11:14:59AM -0700, ankita@nvidia.com wrote: > From: Ankit Agrawal > = > Linux allows device drivers to map IO memory on a per-page basis using > "write combining" or WC. This is often done using > pgprot_writecombing(). The driver knows which pages can support WC > access and the proper programming model to generate this IO. Generally > the use case is to boost performance by using write combining to > generate larger PCIe MemWr TLPs. > = > Allow VMs to select DEVICE_* or NORMAL_NC on a page by page basis for > all IO memory. This puts the VM in charge of the memory attributes, > and removes the KVM override to DEVICE_nGnRE. > = > Ultimately this makes pgprot_writecombing() work correctly in VMs and > allows drivers like mlx5 to fully operate their HW. > = > After some discussions with ARM and CPU architects we reached the > conclusion there was no need for KVM to prevent the VM from selecting > between DEVICE_* and NORMAL_NC for IO memory in VMs. There was a fear > that NORMAL_NC could result in uncontained failures, but upon deeper > analysis it turns out there are already possible cases for uncontained > failures with DEVICE types too. Ultimately the platform must be > implemented in a way that ensures that all DEVICE_* and NORMAL_NC > accesses have no uncontained failures. > = > Fortunately real platforms do tend to implement this. > = > This patch makes the VM's memory attributes behave as follows: > = > =A0S1=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 |=A0=A0 S2=A0=A0=A0=A0=A0=A0=A0=A0=A0= |=A0 Result > =A0NORMAL-WB=A0=A0=A0=A0|=A0 NORMAL-NC=A0=A0=A0=A0|=A0 NORMAL-NC > =A0NORMAL-WT=A0=A0=A0=A0|=A0 NORMAL-NC=A0=A0=A0=A0|=A0 NORMAL-NC > =A0NORMAL-NC=A0=A0=A0=A0|=A0 NORMAL-NC=A0=A0=A0=A0|=A0 NORMAL-NC > =A0DEVICE=A0|=A0 NORMAL-NC=A0=A0=A0=A0|=A0 DEVICE > = > See section D8.5.5 of DDI0487_I_a_a-profile_architecture_reference_manual= .pdf > for details. > = > Signed-off-by: Ankit Agrawal >From the discussions with the hardware people in Arm and Nvidia, we indeed concluded that relaxing S2 to Normal-NC is not any worse than Device (and actually Device memory is more prone to generating uncontained errors, something to do with the Device memory ordering semantics). For this change: Reviewed-by: Catalin Marinas -- = Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel