From mboxrd@z Thu Jan 1 00:00:00 1970 From: Roland Dreier Subject: [ofa-general] mthca use of dma_sync_single is bogus Date: Mon, 09 Jul 2007 14:16:19 -0700 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: general-bounces@lists.openfabrics.org Errors-To: general-bounces@lists.openfabrics.org To: mst@mellanox.co.il, general@lists.openfabrics.org Cc: Lukas Hejtmanek , xen-devel@lists.xensource.com, Keir Fraser List-Id: xen-devel@lists.xenproject.org It seems the problems running mthca in a Xen domU have uncovered a bug in mthca: mthca uses dma_sync_single in mthca_arbel_write_mtt_seg() and mthca_arbel_map_phys_fmr() to sync the MTTs that get written. However, Documentation/DMA-API.txt says: void dma_sync_single(struct device *dev, dma_addr_t dma_handle, size_t size, enum dma_data_direction direction) synchronise a single contiguous or scatter/gather mapping. All the parameters must be the same as those passed into the single mapping API. and mthca is *not* following this clear rule: it is trying to sync only a subrange of the mapping. Later on in the document, there is: void dma_sync_single_range(struct device *dev, dma_addr_t dma_handle, unsigned long offset, size_t size, enum dma_data_direction direction) does a partial sync. starting at offset and continuing for size. You must be careful to observe the cache alignment and width when doing anything like this. You must also be extra careful about accessing memory you intend to sync partially. but that is in a section dealing with non-consistent memory so it's not entirely clear to me whether it's kosher to use this as mthca wants. The other alternative is to put the MTT table in coherent memory just like the MPT table. That might be the best solution I suppose... Michael, anyone else, thoughts on this? - R.