From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,LONGWORDS, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D1A4C43333 for ; Thu, 4 Mar 2021 00:51:26 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BBB2764FAB for ; Thu, 4 Mar 2021 00:51:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BBB2764FAB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=marcan.st Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:Message-ID:Subject: From:References:Cc:To:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=53tmXRWOQuycP+QbcZLwZNbivnZfjKxe6X8qd4gkWIY=; b=B/qRm34yEYeUVW/9nPsQMTt5D pZos0hY3j+IvJFagFMkLqRhVuqsppjnr7C90ceiPopczB1cUQaN7MXnW2JedTpz4mGC6auLZ3gOZS 10Wl3efhV/Gu62YXeEGnznYLlCudEZISowN9qMHQh3iiyuOS451bo5eJrmg2uHHZnJYJ5dx5/x9x8 99t/47nF/h3ookYYQkMQ6IroCJ9TtIf3Ahsn/qPBBdC96GNf7r9kGF2CplOoM96Zhh4LBjlqQG5Kz ppSsuzM7xpIjSOUoLUSjvZpvSdBnY3FkKdqJrO7Mcp2xZIi0MbBdjLpxd+Zm/e0bIc4OhXxRvM9Es O9XqV4i5A==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHc7p-007DjS-Lb; Thu, 04 Mar 2021 00:45:57 +0000 Received: from marcansoft.com ([212.63.210.85] helo=mail.marcansoft.com) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHXJU-006FSA-W9 for linux-arm-kernel@lists.infradead.org; Wed, 03 Mar 2021 19:37:43 +0000 Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: marcan@marcan.st) by mail.marcansoft.com (Postfix) with ESMTPSA id 8C9B43FA1B; Wed, 3 Mar 2021 19:37:33 +0000 (UTC) To: Will Deacon Cc: Linux ARM , Marc Zyngier , Mark Rutland , Peter Zijlstra , Boqun Feng , Catalin Marinas References: <90ea3e27-b2ef-41ac-75c7-5f0686603b2a@marcan.st> <20210303153619.GA19247@willie-the-truck> <20210303184055.GA19891@willie-the-truck> From: Hector Martin Subject: Re: LSE atomic op ordering is weaker than intended? Message-ID: <0d194019-deb9-8d9b-823c-4226f2d2dce2@marcan.st> Date: Thu, 4 Mar 2021 04:37:31 +0900 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.6.0 MIME-Version: 1.0 In-Reply-To: <20210303184055.GA19891@willie-the-truck> Content-Language: es-ES X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 04/03/2021 03.40, Will Deacon wrote: >> // ...stuff that needs to be ordered prior to the atomic here >> ret = atomic_fetch_or_release(flags...) >> if (condition on ret and unrelated stuff) { >> writel(reg_send, ...) // includes pre-barrier > > Looks you have a control dependency here, so I think that can be > writel_relaxed() [with a comment!]. Wouldn't the control dependency be strictly on the *read* portion of the atomic, while I'm looking to order on the write part? Though, actually, I omitted something; the unrelated stuff includes an atomic_read_acquire that will create order against the write-release portion of the atomic (not per the Linux cross-platform docs, but yes per ARMv8 B2.3.3) and *that* should be a suitable control dependency if I'm not mistaken. >> writel_relaxed(reg_ack, ...) > > Interesting, is there no MMIO read on the IRQ path? There's a prior read, but that doesn't help here; the important part is that this writel (ACKing the hardware IPI) cannot move after the atomic op after it (figuring out which software IPIs it represents), because if it does we might ACK a subsequent IPI we're not about to handle. >> dma_wmb() // need a post-barrier >> atomic_fetch_andnot_acquire(flags...) >> // ...stuff that needs to be ordered after the atomic here > > This looks pretty dodgy to me as a later read could drift up before teh > writel_relaxed(). That's fine, as long as those reads are ordered after the atomic fetch portion itself (which they should be since it's an _acquire). AIUI the worst case ordering this can result in is: - read portion of atomic - the entire rest of the code (if it does no writes) - writel_relaxed() - write portion of atomic That's fine: the result of the atomic read gives us license to run the rest of the code; the writel doesn't need to have happened. The important part is that the write portion of the atomic can't happen before the writel. > If this is all to do with IPIs between CPUs and the "stuff that needs to be > ordered the atomic" is all normal, cacheable memory then I > dont't think you need to worry about the outer-shareable domain at all. The > only reason we have that in dma_*mb() is because the coherent DMA buffers > might be non-cacheable to deal with non-coherent DMA. What worried me is whether the order created by atomics on normal memory might not extend to MMIO ops on device memory, which are treated as outer-shareable (per D5.5.1). I don't fully understand how the entire B2.3 section on ordering interacts across memory regions having different attributes. That section talks about a Common Shareability Domain, and also explicitly mentions only Normal memory locations, so I'm having a lot of trouble piecing together how MMIO ops are treated within the entire memory ordering framework (and all the litmus test stuff seems to assume everythign is IS...). All the actual memory involved in everything but the writel() stuff is all normal cacheable memory; what I need to ensure is that I can transitively create memory ordering constraints between normal memory accesses through MMIO writes. That is, that I can create two ordered sequences on two PEs: (normal access 1; mmio write 1) (mmio write 2; normal access 2) Such that I can assert that at least one of the following orderings are true: (normal access 1; normal access 2) (as observed by the PEs) or: (mmio write 2; mmio write 1) (as observed by the MMIO device) But never this combination: (normal access 2; normal access 1) (mmio write 1; mmio write 2) > But hack something together, and I'll look at your v3 to see what's going > on. Please do, hopefully the full code makes more sense. -- Hector Martin (marcan@marcan.st) Public Key: https://mrcn.st/pub _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel