From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E69BDC00140 for ; Wed, 10 Aug 2022 18:13:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:Reply-To:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=z10nwrt0msmIzrb4QAgTbNPZiHFKjrNgs4E75yao7ow=; b=hda0B8GHcAC+Zn reQZV2fLBUrniUy7R7OZj/pWEz9ow4IYLY4H8X//f8DOg4fQSx/rsNuZVwuHHR8BP7Z4zU0MAUIMg wJ9G42ob9ojCtZxZp6DVltX0Roum2O2OFzztYNo7yWWDELZYUkGnBgBEh/ZKeG30XNVjm2MB+JpIZ 9vPzwviRAm1wnx5qOF7qvE2nFse7YFmYWhnH83jCrSGruNINeRg+NeWOArISIvzW+fBhBNHCU5Joh CnUH4fl1Lkc0Bd1nEushujSN3wEG54Mw1KDOfEg6MLGw3WnUDtpYB2hCxbOhDvmqGKaTcbhRMh52R i6uPSNXQzJ1sw6hnrS6w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oLqBM-00DcZC-I6; Wed, 10 Aug 2022 18:11:52 +0000 Received: from mailsrv.cs.umass.edu ([128.119.240.136]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oLqBH-00DcV6-Oc for linux-arm-kernel@lists.infradead.org; Wed, 10 Aug 2022 18:11:51 +0000 Received: from [192.168.1.218] (cpe-74-75-141-195.maine.res.rr.com [74.75.141.195]) by mailsrv.cs.umass.edu (Postfix) with ESMTPSA id 924D24010524; Wed, 10 Aug 2022 14:11:43 -0400 (EDT) Subject: Re: [PATCH RFC 10/15] x86: add an arch helper function to invalidate all cache for nvdimm To: Mark Rutland Cc: Dave Jiang , Jonathan Cameron , Davidlohr Bueso , linux-cxl@vger.kernel.org, nvdimm@lists.linux.dev, dan.j.williams@intel.com, bwidawsk@kernel.org, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, a.manzanares@samsung.com, linux-arch@vger.kernel.org, Arnd Bergmann , linux-arm-kernel@lists.infradead.org References: <165791918718.2491387.4203738301057301285.stgit@djiang5-desk3.ch.intel.com> <165791937063.2491387.15277418618265930924.stgit@djiang5-desk3.ch.intel.com> <20220718053039.5whjdcxynukildlo@offworld> <4bedc81d-62fa-7091-029e-a2e56b4f8f7a@intel.com> <20220803183729.00002183@huawei.com> <9f3705e1-de21-0f3c-12af-fd011b6d613d@intel.com> From: Eliot Moss Message-ID: Date: Wed, 10 Aug 2022 14:11:41 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Thunderbird/68.12.1 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220810_111148_033767_FB31A4E6 X-CRM114-Status: GOOD ( 31.30 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: moss@cs.umass.edu Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 8/10/2022 2:09 PM, Mark Rutland wrote: > On Wed, Aug 10, 2022 at 10:31:12AM -0400, Eliot Moss wrote: >> On 8/10/2022 10:15 AM, Mark Rutland wrote: >>> On Tue, Aug 09, 2022 at 02:47:06PM -0700, Dave Jiang wrote: >>>> >>>> On 8/3/2022 10:37 AM, Jonathan Cameron wrote: >>>>> On Tue, 19 Jul 2022 12:07:03 -0700 >>>>> Dave Jiang wrote: >>>>> >>>>>> On 7/17/2022 10:30 PM, Davidlohr Bueso wrote: >>>>>>> On Fri, 15 Jul 2022, Dave Jiang wrote: >>>>>>>> The original implementation to flush all cache after unlocking the >>>>>>>> nvdimm >>>>>>>> resides in drivers/acpi/nfit/intel.c. This is a temporary stop gap until >>>>>>>> nvdimm with security operations arrives on other archs. With support CXL >>>>>>>> pmem supporting security operations, specifically "unlock" dimm, the >>>>>>>> need >>>>>>>> for an arch supported helper function to invalidate all CPU cache for >>>>>>>> nvdimm has arrived. Remove original implementation from acpi/nfit and >>>>>>>> add >>>>>>>> cross arch support for this operation. >>>>>>>> >>>>>>>> Add CONFIG_ARCH_HAS_NVDIMM_INVAL_CACHE Kconfig and allow x86_64 to >>>>>>>> opt in >>>>>>>> and provide the support via wbinvd_on_all_cpus() call. >>>>>>> So the 8.2.9.5.5 bits will also need wbinvd - and I guess arm64 will need >>>>>>> its own semantics (iirc there was a flush all call in the past). Cc'ing >>>>>>> Jonathan as well. >>>>>>> >>>>>>> Anyway, I think this call should not be defined in any place other >>>>>>> than core >>>>>>> kernel headers, and not in pat/nvdimm. I was trying to make it fit in >>>>>>> smp.h, >>>>>>> for example, but conviniently we might be able to hijack >>>>>>> flush_cache_all() >>>>>>> for our purposes as of course neither x86-64 arm64 uses it :) >>>>>>> >>>>>>> And I see this as safe (wrt not adding a big hammer on unaware >>>>>>> drivers) as >>>>>>> the 32bit archs that define the call are mostly contained thin their >>>>>>> arch/, >>>>>>> and the few in drivers/ are still specific to those archs. >>>>>>> >>>>>>> Maybe something like the below. >>>>>> Ok. I'll replace my version with yours. >>>>> Careful with flush_cache_all(). The stub version in >>>>> include/asm-generic/cacheflush.h has a comment above it that would >>>>> need updating at very least (I think). >>>>> Note there 'was' a flush_cache_all() for ARM64, but: >>>>> https://patchwork.kernel.org/project/linux-arm-kernel/patch/1429521875-16893-1-git-send-email-mark.rutland@arm.com/ >>>> >>>> >>>> flush_and_invalidate_cache_all() instead given it calls wbinvd on x86? I >>>> think other archs, at least ARM, those are separate instructions aren't >>>> they? >>> >>> On arm and arm64 there is no way to perform maintenance on *all* caches; it has >>> to be done in cacheline increments by address. It's not realistic to do that >>> for the entire address space, so we need to know the relevant address ranges >>> (as per the commit referenced above). >>> >>> So we probably need to think a bit harder about the geenric interface, since >>> "all" isn't possible to implement. :/ >> >> Can you not do flushing by set and way on each cache, >> probably working outwards from L1? > > Unfortunately, for a number of reasons, that doeesn't work. For better or > worse, the *only* way which is guaranteed to work is to do this by address. > > If you look at the latest ARM ARM (ARM DDI 0487H.a): > > https://developer.arm.com/documentation/ddi0487/ha/ > > ... on page D4-4754, in the block "Example code for cache maintenance > instructions", there's note with a treatise on this. > > The gist is that: > > * Set/Way ops are only guaranteed to affect the caches local to the CPU > issuing them, and are not guaranteed to affect caches owned by other CPUs. > > * Set/Way ops are not guaranteed to affect system-level caches, which are > fairly popular these days (whereas VA ops are required to affect those). > > * Set/Way ops race with the natural behaviour of caches (so e.g. a line could > bounce between layers of cache, or between caches in the system, and avoid > being operated upon). > > So unless you're on a single CPU system, with translation disabled, and you > *know* that there are no system-level caches, you can't rely upon Set/Way ops > to do anything useful. > > They're really there for firmware to use for IMPLEMENTATION DEFINED power-up > and power-down sequences, and aren'y useful to portable code. Thanks for the explanation. Really does seem that ARM could use the equivalent on wbnoinvd/wbinvd/invd. Regards - Eliot _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel