From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EB548C38145 for ; Wed, 7 Sep 2022 17:51:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:From:References:Cc:To:Subject: MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=6UCy1Z3rixKJuVcsvQGyD/8VSkiz+TORfwqDYZ3r9Vk=; b=3QEB5AiU/ZE7ha XwbFUbZdt3AF3BgJfYYfYX+bGlzdbJpP5FrZkppvFIFITbKEr6dURQbWlvmoYIcQVNuA3ywPLe00V i6eC5ei9jTnXo85p1qn+Fhaa194qqhxKZsRwOGyV8ATlgbVQP/ZsQrfwkMhzZ4J1KdwvpCF9VTu/h gog7fjwaquZbhtjQCDTiN8XAlcdv4LVmM1pDaFS0UNuxtJRdivjgGHuQbmEUX1u1hsIgQS3z1WlWo l2sHOE3o/aQDeNfglu6Gv2ejpA0ug0fm9lVby0UtTQo41fPyQ74vu5tKG3fK7W9MnIMNJnaWN4yP/ Oe3MaVRqfl7MaFKjN3hg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oVzC8-008MDW-3q; Wed, 07 Sep 2022 17:50:36 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oVzC4-008MBe-3Z for linux-arm-kernel@lists.infradead.org; Wed, 07 Sep 2022 17:50:34 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1DE17106F; Wed, 7 Sep 2022 10:50:34 -0700 (PDT) Received: from [10.57.15.197] (unknown [10.57.15.197]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id B0ADF3F7B4; Wed, 7 Sep 2022 10:50:25 -0700 (PDT) Message-ID: <97fface8-e40e-072c-4335-c94094884e93@arm.com> Date: Wed, 7 Sep 2022 18:50:20 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Thunderbird/102.2.1 Subject: Re: [PATCH] arm64: dma: Drop cache invalidation from arch_dma_prep_coherent() Content-Language: en-GB To: Will Deacon Cc: Christoph Hellwig , linux-arm-kernel@lists.infradead.org, Catalin Marinas , Mark Rutland , Ard Biesheuvel References: <20220823122111.17439-1-will@kernel.org> <20220907090305.GA30704@lst.de> <5d856574-4cd7-70d0-adcb-3e284fef315f@arm.com> <20220907162543.GA30558@willie-the-truck> From: Robin Murphy In-Reply-To: <20220907162543.GA30558@willie-the-truck> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220907_105032_278434_C04CA611 X-CRM114-Status: GOOD ( 39.82 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 2022-09-07 17:25, Will Deacon wrote: > On Wed, Sep 07, 2022 at 10:27:45AM +0100, Robin Murphy wrote: >> On 2022-09-07 10:03, Christoph Hellwig wrote: >>> On Tue, Aug 23, 2022 at 01:21:11PM +0100, Will Deacon wrote: >>>> arch_dma_prep_coherent() is called when preparing a non-cacheable region >>>> for a consistent DMA buffer allocation. Since the buffer pages may >>>> previously have been written via a cacheable mapping and consequently >>>> allocated as dirty cachelines, the purpose of this function is to remove >>>> these dirty lines from the cache, writing them back so that the >>>> non-coherent device is able to see them. >>> >>> Yes. >>> >>>> I'm slightly wary about this change as other architectures seem to do >>>> clean+invalidate here, but I'd like to hear what others think in any >>>> case. >>> >>> If arm64 is fine with having clean but present cachelines when creating >>> an uncached mapping for a cache line, the invalidate is not required. >>> >>> But isn't it better for the cache if these by definition useless >>> cachelines get evicted? >>> >>> My biggest concern here is that we're now moving from consolidating >>> these semantics in all the different architectures to different ones, >>> making a centralization of the policies even harder. >> >> FWIW I agree with Ard in not being entirely confident with this change. The >> impression I had (which may be wrong) was that the architecture never >> actually ruled out unexpected cache hits in the case of mismatched >> attributes, it just quietly stopped mentioning it at all. And even if the >> architecture did rule them out, how confident are we about errata that might >> still allow them to happen? > > The architecture does rule this out as long as the cacheable alias is not > dirty (i.e. has not been written to). If this was not the case, then our > non-cacheable DMA buffers would be unreliable because speculative fills > from the linear map could "mask" the actual DMA data. The presence of the > linear map also means that the lines could immediately be refetched by > speculation, rendering the invalidation pointless in the current code. > > Regardless of this patch, a CPU erratum violating the architectural > guarantee would be broken. It is the presence of the linear alias that > would need to be resolved. Oh have I got a treat for you... but that's not relevant here ;) > The architecture text is buried in the infamous "B2.8 Mismatched memory > attributes" section of the Arm ARM, but it's also worth talking to Mark R, > Catalin and the handful of folks in ATG who grok this to get a summary > which is easier to digest. I'm looking at DDI0487I.a and nothing stands out as different from previous readings. In particular, "A read of the memory Location by one agent might not return the value most recently written to that memory Location by the same agent" seems entirely consistent with non-cacheable reads being allowed to unexpectedly hit clean (but stale) cache entries even if non-cacheable writes don't. And frankly I don't see how "There might be a loss of coherency when multiple agents attempt to access a memory Location." helps rule *anything* out. The bit about writeback does seem to imply that non-cacheable writes can't dirty a cache entry, so I can at least accept that unexpected hits for writes are off the table. Furthermore, the bit about mismatched shareability - which I believe applies here since our cacheable mappings are ISh while non-cacheable is inherently OSh - explicitly mentions: "Software running on a PE cleans and invalidates a Location from cache before and after each read or write to that Location by that PE." If the architecture saying to clean and invalidate is supposed to be a guarantee that invalidation is unnecessary, I give up. >> It seems like we don't stand to gain much by removing the invalidation - >> since the overhead will still be in the clean - other than the potential for >> a slightly increased chance of rare and hard-to-debug memory corruption :/ > > I just find it odd that we rely on the CPU not hitting the cacheable alias > in other places, yet we're using an invalidation for this path. It's > inconsistent and difficult to explain to people. As I said, I'm happy to > add a comment to the existing code instead of the change here, but I don't > know what to say other than something like: > > /* > * The architecture says we only need a clean here, but invalidate as > * well just in case. > */ > > Which feels deeply unsatisfactory. Well, the chances of the alias being speculated back into caches by access to nearby linear map pages still seem a lot smaller than the chances of caches hanging on to lines we've just finished accessing with cacheable attributes. Christoph makes a good point that at worst it's a concise hint for caches with LRU replacement policy that might otherwise think we could be coming back for our freshly-written clean zeros. Cheers, Robin. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel