From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DB7FCC3A5A7 for ; Tue, 6 Dec 2022 09:49:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Subject:CC:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=cxIphPuL7T+z/AHxlzGTdWutfFX80UsvhNEfKU1+OVo=; b=seotMrQKfjJmo6 y9pkUInvqbhO/wuCKJnucTrhcwXy81V4jOET/aNp6ssqSYBzPgDapOsgGDZhLmvc3gs51yEKX+nXN MNRlMU17D2O+eT/rGtJZPUEE6C6eADWenNYMtuHsWIaJkfmij1uIO/KEvsjXJU53kv2e9NkFc5/SO +QJkGw8CMkBXXOZ7wvm3kfAalWS8eUSg+fcZ1jnbDC5TltrvsIDKAVcBpQYjvd6AXoYKyUMbhthDg fL254BQ4ryYHQqFBqOgw9o+K/p7NF47wMR6a7VhKOYiNAO6TD0KnKioi81pBWNKFFLOHWYfe7wUbS DI2cc3YlJ2xlzWj7SjmQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1p2UYn-006Rbx-QY; Tue, 06 Dec 2022 09:48:22 +0000 Received: from frasgout.his.huawei.com ([185.176.79.56]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1p2UYD-006R1w-Ph for linux-arm-kernel@lists.infradead.org; Tue, 06 Dec 2022 09:47:48 +0000 Received: from lhrpeml500005.china.huawei.com (unknown [172.18.147.226]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4NRFr96sD3z686fV; Tue, 6 Dec 2022 17:44:41 +0800 (CST) Received: from localhost (10.45.155.47) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.34; Tue, 6 Dec 2022 09:47:34 +0000 Date: Tue, 6 Dec 2022 09:47:33 +0000 From: Jonathan Cameron To: Dan Williams CC: Davidlohr Bueso , , , , , James Morse , Will Deacon , , Anshuman Khandual , , Subject: Re: [PATCH 5/5] cxl/region: Manage CPU caches relative to DPA invalidation events Message-ID: <20221206094733.00007ed2@Huawei.com> In-Reply-To: <638e502e1904a_3cbe0294e2@dwillia2-xfh.jf.intel.com.notmuch> References: <166993219354.1995348.12912519920112533797.stgit@dwillia2-xfh.jf.intel.com> <166993222098.1995348.16604163596374520890.stgit@dwillia2-xfh.jf.intel.com> <20221205192054.mwhzyjrfwfn3tma5@offworld> <638e502e1904a_3cbe0294e2@dwillia2-xfh.jf.intel.com.notmuch> Organization: Huawei Technologies Research and Development (UK) Ltd. X-Mailer: Claws Mail 4.1.0 (GTK 3.24.33; x86_64-w64-mingw32) MIME-Version: 1.0 X-Originating-IP: [10.45.155.47] X-ClientProxiedBy: lhrpeml500004.china.huawei.com (7.191.163.9) To lhrpeml500005.china.huawei.com (7.191.163.240) X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221206_014746_207032_1018C31B X-CRM114-Status: GOOD ( 27.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, 5 Dec 2022 12:10:22 -0800 Dan Williams wrote: > [ add linux-arm-kernel@lists.infradead.org ] > > Background for ARM folks, CXL can dynamically reconfigure the target > devices that back a given physical memory region. When that happens the > CPU cache can be holding cache data from a previous configuration. The > mitigation for that scenario on x86 is wbinvd, ARM does not have an > equivalent. The result, dynamic region creation is disabled on ARM. In > the near term, most CXL is configured pre-boot, but going forward this > restriction is untenable. > > Davidlohr Bueso wrote: > > On Thu, 01 Dec 2022, Dan Williams wrote: > > > > >A "DPA invalidation event" is any scenario where the contents of a DPA > > >(Device Physical Address) is modified in a way that is incoherent with > > >CPU caches, or if the HPA (Host Physical Address) to DPA association > > >changes due to a remapping event. > > > > > >PMEM security events like Unlock and Passphrase Secure Erase already > > >manage caches through LIBNVDIMM, > > > > Just to be clear, is this is why you get rid of the explicit flushing > > for the respective commands in security.c? > > Correct, because those commands can only be executed through libnvdimm. > > > > > >so that leaves HPA to DPA remap events > > >that need cache management by the CXL core. Those only happen when the > > >boot time CXL configuration has changed. That event occurs when > > >userspace attaches an endpoint decoder to a region configuration, and > > >that region is subsequently activated. > > > > > >The implications of not invalidating caches between remap events is that > > >reads from the region at different points in time may return different > > >results due to stale cached data from the previous HPA to DPA mapping. > > >Without a guarantee that the region contents after cxl_region_probe() > > >are written before being read (a layering-violation assumption that > > >cxl_region_probe() can not make) the CXL subsystem needs to ensure that > > >reads that precede writes see consistent results. > > > > Hmm where does this leave us remaping under arm64 which is doesn't have > > ARCH_HAS_CPU_CACHE_INVALIDATE_MEMREGION? > > > > Back when we were discussing this it was all related to the security stuff, > > which under arm it could just be easily discarded as not available feature. > > I can throw out a few strawman options, but really need help from ARM > folks to decide where to go next. +Cc bunch of relevant people. There are discussions underway but I'm not sure anyone will want to give more details here yet. > > 1/ Map and loop cache flushing line by line. It works, but for Terabytes > of CXL the cost is 10s of seconds of latency to reconfigure a region. > That said, region configuration, outside of test scenarios, is typically > a "once per bare metal provisioning" event. > > 2/ Set a configuration dependency that mandates that all CXL memory be > routed through the page allocator where it is guaranteed that the memory > will be written (zeroed) before use. This restricts some planned use > cases for the "Dynamic Capacity Device" capability. This is the only case that's really a problem (to my mind) I hope we will have a more general solution before there is much hardware out there, particularly where sharing is involved. > > 3/ Work with the CXL consortium to extend the back-invalidate concept > for general purpose usage to make devices capable of invalidating caches > for a new memory region they joined, and mandate it for ARM. This one > has a long lead time and a gap for every device in flight currently. There are significant disadvantages in doing this that I suspect will mean this never happens for some classes of device, or is turned off for performance reasons. For anyone curious, go look at the protocol requirements of back invalidate in the CXL 3.0 spec. Jonathan _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel