From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7665C433E0 for ; Thu, 4 Mar 2021 00:34:06 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 74E5D64EEF for ; Thu, 4 Mar 2021 00:34:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 74E5D64EEF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=5aImQWC01S3dPzoh/ga66HQglqBHgR50hO5gezoeRqM=; b=qu79k4rQHHs/eWmnpGsP2Tj2K qbLiNccsKd117ahMPe68FAoNecWUfsuX/G5nUeYW6jxTiW4kWfbyP3oZPM2yDrXjJHJsAUd6iW3HQ g5AnrjBQCo4BwLTzxPbDIiVu3lRiEOGcxgjIvlfjzkVQc9toIytMpuF9ozyOQOHEr9Pbm+CQoIgUA IHIoRR2JRT1f8igDdHU7VpEkNy64ls5/zNjddqK0YasReXLawVLBCTGURhxhCXiVNxm4nvRzCCoTU xx1A7F886WBljxcJ8XvczizyasdtnktCRCVqBkylgO3izVzaGAmD8P+9WVv6aQ8QvCnITJT4q57wX /dWSsuwsQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHbqM-0079fi-8l; Thu, 04 Mar 2021 00:27:54 +0000 Received: from mail.kernel.org ([198.145.29.99]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHWQg-0063pT-3P for linux-arm-kernel@lists.infradead.org; Wed, 03 Mar 2021 18:41:05 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id F3CF764ECF; Wed, 3 Mar 2021 18:40:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1614796860; bh=UbAe4tMSrFH8b1h1l/9oQZXjKonAI2NyL6PU9QeISX8=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=VEK6x18ah8MiEwJIMxJW2YQGQP4P8P6UlP7YJ2oIcftXVTD4QOE2y9bn2BNbvkLtj S7ctJS931O5hCb8GelPp5mfWWWQcsY2xTgSAoA6h78WKN4OSI1eb/axjP/ZNlwHesJ VgucGy2LNhaGkSi6itpATVCz9LeT5faeVBS7BXj9R0avqWFzYHNlS0Rp5aOe8omRRI kaAoIoI3QnbLAZ6RKt6dKMM5+nPSN0x3g/kCR4g8PSXc27o3F4501wLxIvBxmUWhq7 tw5h2qDZgXwOeatdKaoZmbw38g272sOCH1MAHiPSMkCxWqW7xjHk6U9HdXdWCRKePk 1V1QZA1MRyOzg== Date: Wed, 3 Mar 2021 18:40:55 +0000 From: Will Deacon To: Hector Martin Cc: Linux ARM , Marc Zyngier , Mark Rutland , Peter Zijlstra , Boqun Feng , Catalin Marinas Subject: Re: LSE atomic op ordering is weaker than intended? Message-ID: <20210303184055.GA19891@willie-the-truck> References: <90ea3e27-b2ef-41ac-75c7-5f0686603b2a@marcan.st> <20210303153619.GA19247@willie-the-truck> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, Mar 04, 2021 at 03:04:20AM +0900, Hector Martin wrote: > On 04/03/2021 00.36, Will Deacon wrote: > > > Did I miss something, or is this in fact an issue? > > > > Both. The -AL atomics are actually special-cased in the > > "barrier-ordered-before" relation in the Arm ARM: > > > > [RW1 is barrier-ordered-before if] > > * RW1 appears in program order before an atomic instruction with both > > Acquire and Release semantics that appears in program order before > > RW2. > > > > However, that isn't sufficient to order prior accesses with the "load part" > > of the RmW and later accesses with the "store part" of the RmW, as you have > > observed in your test. I'm aware of some pending proposals in this area of > > the architecture, so I'm reluctant to make any changes until that's > > bottomed-out, but I'll make a note to chase that up. > > I had actually seen that part of the spec, and looked at it sideways a few > times, but concluded it wasn't giving me the ordering guarantees I was > looking for (this was before I wrote the litmus test). You're right, it does > nonetheless make it stronger than the mere combination of _acquire and > _release semantics. > > Glad to hear this is something being worked on! I've been giving myself a > crash course in memory model minutiae over the past few weeks :) > > > > (And while I'm talking to the right people: this issue aside, do atomic ops > > > on Normal memory create ordering with Device memory ops, or are there no > > > guarantees there due to the fact that Normal memory is mapped > > > inner-shareable and the ordering guarantees thus do not extend to > > > outer-shareable Device accesses? My currenty understanding is the latter, > > > but I find the ARM ARM wording hard to conclusively grok here.) > > > > Outer-shareable is a superset of inner-shareable, but I think this would be > > easier with a specific example. I'll go and look at the AIC patch, since > > this is all a lot easier to talk about in the context of some real code. > > > > Which is the latest version I should look at? > > I'm just about to send a v3 tomorrow, so I'll CC you on that patch (don't > bother with v2, this part of the code is changing a lot). That said, it's > basically the following two sequences: > > A: > > // ...stuff that needs to be ordered prior to the atomic here > ret = atomic_fetch_or_release(flags...) > if (condition on ret and unrelated stuff) { > writel(reg_send, ...) // includes pre-barrier Looks you have a control dependency here, so I think that can be writel_relaxed() [with a comment!]. > } > > B: > > writel_relaxed(reg_ack, ...) Interesting, is there no MMIO read on the IRQ path? > dma_wmb() // need a post-barrier > atomic_fetch_andnot_acquire(flags...) > // ...stuff that needs to be ordered after the atomic here This looks pretty dodgy to me as a later read could drift up before teh writel_relaxed(). > My current understanding is that I cannot drop the dma_wmb() in B and use > _relaxed in A() and instead use full-ordered atomic ops, because the atomic > ops, operating on normal IS memory, would not make any statements regarding > ordering with device OS memory. I need the I/O writes to be ordered with > regard to the atomics. If this is all to do with IPIs between CPUs and the "stuff that needs to be ordered the atomic" is all normal, cacheable memory then I dont't think you need to worry about the outer-shareable domain at all. The only reason we have that in dma_*mb() is because the coherent DMA buffers might be non-cacheable to deal with non-coherent DMA. But hack something together, and I'll look at your v3 to see what's going on. Will _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel