From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 99F60C54EE9 for ; Wed, 7 Sep 2022 17:54:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=F3LTb60z988HAV7ViBLX2yFb+Wnf02B6MOIzHovrBKU=; b=Cpa0ILRkBx8SVj k9U1DdUsNeTlzlf9E4o92lS+8sZ/7eFxZ5yHfUSJXiasik7aAmKvVVVrpzNBR1WVqCQEPyk0ZKrd7 qFzX8k6OzzZGxE0G7LUCnssf1Y0KY+idTSD4Fq5T/jpJEckhBtwj0sV8YN3DgF18n1/SiZf2XxW+C dLWMomPXGbuBm0v/5TwvusVYU5iTo9k/lZQtKQ1jFrQOkwa6ocAB9nDtA0DGfP07HWB8JYeqfk873 3kW0uTAYPE6fuG4LSCeTM4zySoj0WPA6nN3863TKyf5BaHbNYRrw/FbvAaTVg/oP/S89qoUT9fydu gxkRx7uFHxKCyeyC8hMQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oVzFM-008NWP-CT; Wed, 07 Sep 2022 17:53:56 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oVzFI-008NVQ-N0 for linux-arm-kernel@lists.infradead.org; Wed, 07 Sep 2022 17:53:54 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 66E6161196; Wed, 7 Sep 2022 17:53:51 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B01D2C433D6; Wed, 7 Sep 2022 17:53:48 +0000 (UTC) Date: Wed, 7 Sep 2022 18:53:43 +0100 From: Catalin Marinas To: Mark Zhang Cc: will@kernel.org, linux-arm-kernel@lists.infradead.org, Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Leon Romanovsky , Michael Guralnik , Michael Berezin , yong.xu@arm.com, Eran Ben Elisha Subject: Re: Should we use "dsb" or "dmb" between write to buffer and write to register Message-ID: References: <2bea1a7b-935e-695d-ddaa-13eacda5672c@nvidia.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <2bea1a7b-935e-695d-ddaa-13eacda5672c@nvidia.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220907_105352_850421_38ED1DD2 X-CRM114-Status: GOOD ( 28.22 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, Aug 22, 2022 at 03:53:42PM +0800, Mark Zhang wrote: > May I consult when to use dsb or dmb in our device driver, thanks: > > For example when send a command a FW/HW, usually we do it with 3 steps: > 1. memcpy(buff, src, size); > 2. wmb(); > 3. write64(ctrl, reg_addr); > > IIUC in kernel wmb() is implemented with "dsb st". When we change this to > "dmb st" then we get better performance, but we are not sure if it's safe. I > have read Will's post[1] but still not sure. > > So our questions are: > 1. can we use "dmb" here? > 2. If we can then should we use "dmb st", or "dmb oshst"? > > Thank you very much. > > [1] https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git/commit/?id=22ec71615d824f4f11d38d0e55a88d8956b7e45f Will convinced me at the time that it's sufficient, though every time I revisit this I get confused ;). Not sure whether we have updated the memory model since to cover such scenarios. In practice at least from what I recall that should be safe. IIRC, the logic is that if an observer in the same shareability domain is seeing the write64 (3), it should have observed the memcpy (1) as well given the DMB. The device in question is one of the observers observing the memcpy to 'buff' (but it doesn't 'observe' the write64 itself). In a multi-copy atomic world, if a third observer is seeing the write64 and therefore the memcpy, it means that the device should have observed the memcpy as well (the multi-copy atomicity requirement). That's where it looks a bit like Schrodinger's cat to me (the state of the cat being whether the device observed the memcpy or not). You can't be sure until you have a third observer seeing the write64 to device. In the absence of such hypothetical observer, the device might or might not have seen the new data in 'buff' since it cannot observe the write64 to its control register (and from the commit log, this seems to be the case with peripherals private to a CPU). I guess the question is what does it mean for the device that a third observer saw the write64. In one interpretation of observability, another write64 from the third observer is ordered after the original write64 but to me it still doesn't help clarify any order imposed on the device access to 'buff': Initial state: buff=0 ctrl=0 P0: P1: Device: Wbuff=1 Wctrl=2 Ry=buff DMB DMB Wctrl=1 Rx=buff If the final 'ctrl' register value is 2 then x==1. But I don't see how y==0 or 1 is influenced by Wctrl=2. If x==1 on P1, any other observer, including the device, should see the buff value of 1 but this assumes that there is some other ordering for when Ry=buff is issued. So, as you can see, I'm even more confused than when I started writing this email ;). I'd leave this to Will to explain and, of course, if your hardware folks disagree, they should let us know. -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel