From mboxrd@z Thu Jan 1 00:00:00 1970 From: Catalin Marinas Subject: Re: [PATCH v2] sata_sil24: Use memory barriers before issuing commands Date: Tue, 15 Jun 2010 12:10:53 +0100 Message-ID: <1276600253.26369.46.camel@e102109-lin.cambridge.arm.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: Sender: linux-ide-owner@vger.kernel.org To: Robert Hancock Cc: Nick Piggin , Tejun Heo , linux-ide@vger.kernel.org, linux-kernel@vger.kernel.org, Colin Tuckley , Jeff Garzik , linux-arch List-Id: linux-arch.vger.kernel.org On Sat, 2010-06-12 at 02:30 +0100, Robert Hancock wrote: > On Fri, Jun 11, 2010 at 5:04 AM, Catalin Marinas > wrote: > > On Fri, 2010-06-11 at 11:11 +0100, Nick Piggin wrote: > >> On Fri, Jun 11, 2010 at 10:41:46AM +0100, Catalin Marinas wrote: > >> > The only reference of DMA buffers vs I/O I found in the DMA-API.txt > >> > file: > >> > > >> > Consistent memory is memory for which a write by either the > >> > device or the processor can immediately be read by the processor > >> > or device without having to worry about caching effects. (You > >> > may however need to make sure to flush the processor's write > >> > buffers before telling devices to read that memory.) > >> > > >> > But there is no API for "flushing the processor's write buffers". Does > >> > it mean that this should be taken care of in writel()? We would make the > >> > I/O accessors pretty expensive on some architectures. > >> > >> The APIs for that are mb/wmb/rmb ones. > > > > So if that's the API for the above case and we are strictly referring to > > the sata_sil24 patch I sent - shouldn't we just add wmb() in the driver > > between the write to the DMA buffer and the writel() to start the DMA > > transfer? Do we need to move the wmb() to the writel() macro? > > I think it would be best if writel, etc. did the write buffer flushing > by default. As Nick said, if there are some performance critical areas > then those can use the relaxed versions but it's safest if the default > behavior works as drivers expect. I went through the past discussion pointed to by Fujita (thanks!) but I wouldn't say it resulted in a definitive guideline on how architectures should implement the I/O accessors. >From an ARM perspective, I would prefer to add wmb() in the drivers where it matters - basically only those using DMA coherent buffers. A lot of drivers already have this in place and that's already documented in DMA-API.txt (maybe with a bit of clarification). Some statistics - grepping drivers/ for alloc_coherent shows 285 files. Of these, 69 already use barriers. It's not trivial to go through 200+ drivers and add barriers but I wouldn't say that's impossible. If we go the other route of adding mb() in writel() (though I don't prefer it), there are two additional issues: (1) how relaxed would the "writel_relaxed" etc. accessors be? Are they relaxed only with regards to coherent DMA buffers or relaxed with other I/O operations as well? Can the compiler reorder them? (2) do we go through all the drivers that currently have *mb() and remove them? A quick grep in drivers/ shows over 1600 occurrences of *mb(). -- Catalin From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from cam-admin0.cambridge.arm.com ([217.140.96.50]:35126 "EHLO cam-admin0.cambridge.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754849Ab0FOLLK (ORCPT ); Tue, 15 Jun 2010 07:11:10 -0400 Subject: Re: [PATCH v2] sata_sil24: Use memory barriers before issuing commands From: Catalin Marinas In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Date: Tue, 15 Jun 2010 12:10:53 +0100 Message-ID: <1276600253.26369.46.camel@e102109-lin.cambridge.arm.com> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-arch-owner@vger.kernel.org List-ID: To: Robert Hancock Cc: Nick Piggin , Tejun Heo , linux-ide@vger.kernel.org, linux-kernel@vger.kernel.org, Colin Tuckley , Jeff Garzik , linux-arch Message-ID: <20100615111053.y9CQovTc5pUsuVatWuL7OL293i6F3TegOn6IOkTyR4U@z> On Sat, 2010-06-12 at 02:30 +0100, Robert Hancock wrote: > On Fri, Jun 11, 2010 at 5:04 AM, Catalin Marinas > wrote: > > On Fri, 2010-06-11 at 11:11 +0100, Nick Piggin wrote: > >> On Fri, Jun 11, 2010 at 10:41:46AM +0100, Catalin Marinas wrote: > >> > The only reference of DMA buffers vs I/O I found in the DMA-API.txt > >> > file: > >> > > >> > Consistent memory is memory for which a write by either the > >> > device or the processor can immediately be read by the processor > >> > or device without having to worry about caching effects. (You > >> > may however need to make sure to flush the processor's write > >> > buffers before telling devices to read that memory.) > >> > > >> > But there is no API for "flushing the processor's write buffers". Does > >> > it mean that this should be taken care of in writel()? We would make the > >> > I/O accessors pretty expensive on some architectures. > >> > >> The APIs for that are mb/wmb/rmb ones. > > > > So if that's the API for the above case and we are strictly referring to > > the sata_sil24 patch I sent - shouldn't we just add wmb() in the driver > > between the write to the DMA buffer and the writel() to start the DMA > > transfer? Do we need to move the wmb() to the writel() macro? > > I think it would be best if writel, etc. did the write buffer flushing > by default. As Nick said, if there are some performance critical areas > then those can use the relaxed versions but it's safest if the default > behavior works as drivers expect. I went through the past discussion pointed to by Fujita (thanks!) but I wouldn't say it resulted in a definitive guideline on how architectures should implement the I/O accessors. >From an ARM perspective, I would prefer to add wmb() in the drivers where it matters - basically only those using DMA coherent buffers. A lot of drivers already have this in place and that's already documented in DMA-API.txt (maybe with a bit of clarification). Some statistics - grepping drivers/ for alloc_coherent shows 285 files. Of these, 69 already use barriers. It's not trivial to go through 200+ drivers and add barriers but I wouldn't say that's impossible. If we go the other route of adding mb() in writel() (though I don't prefer it), there are two additional issues: (1) how relaxed would the "writel_relaxed" etc. accessors be? Are they relaxed only with regards to coherent DMA buffers or relaxed with other I/O operations as well? Can the compiler reorder them? (2) do we go through all the drivers that currently have *mb() and remove them? A quick grep in drivers/ shows over 1600 occurrences of *mb(). -- Catalin