From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chen Baozi Subject: Re: Question about DMA on 1:1 mapping dom0 of arm64 Date: Sat, 18 Apr 2015 16:29:18 +0800 Message-ID: <20150418082918.GA9106@cbz-thinkpad> References: <20150417112411.GA13595@cbz-thinkpad> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Stefano Stabellini Cc: Konrad Rzeszutek Wilk , julien.grall@linaro.org, Ian Campbell , xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org On Fri, Apr 17, 2015 at 03:32:20PM +0100, Stefano Stabellini wrote: > On Fri, 17 Apr 2015, Chen Baozi wrote: > > Hi all, > > > > According to my recent experience, there might be some problems of swiotlb > > dma map on 1:1 mapping arm64 dom0 with large memory. The issue is like below: > > > > For those arm64 server with large memory, it is possible to set dom0_mem > > > 4G (e.g. I have one set with 16G). In this case, according to my understanding, > > there is chance that the dom0 kernel needs to map some buffers above 4G to do > > DMA operations (e.g. in snps,dwmac ethernet driver). However, most DMA engines > > support only 32-bit physical address, thus aren't able to operate directly on > > those memory. IIUC, swiotlb is implemented to solve this (using bounce buffer), > > if there is no IOMMU or IOMMU is not enabled on the system. Sadly, it seems > > that xen_swiotlb_map_page in my dom0 kernel allocates > > (start_dma_addr = 0x944800000) the buffers for DMA above 4G which fails > > dma_capable() checking and was then unable to return from xen_swiotlb_map_page() > > successfully. > > > > If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory > > of dom0 below 4G, everything goes fine. > > > > I am not familiar with swiotlb-xen, so there would be misunderstanding about > > the current situation. Fix me if I did/understood anything wrong. > > > > Any ideas? > > I think that the problem is that xen_swiotlb_init doesn't necessarely allocate > memory under 4G on arm/arm64. > > xen_swiotlb_init calls __get_free_pages to allocate memory, so the pages > could easily be above 4G. Subsequently xen_swiotlb_fixup is called on > the allocated memory range, calling xen_create_contiguous_region and > passing an address_bits mask. However xen_create_contiguous_region > doesn't actually do anything at all on ARM. > > I think that given that dom0 is mapped 1:1 on ARM, the easiest and best > fix would be to simply allocate memory under 4G to begin with. Something > like (maybe with an ifdef ARM around it): > > diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c > index 810ad41..22ac33a 100644 > --- a/drivers/xen/swiotlb-xen.c > +++ b/drivers/xen/swiotlb-xen.c > @@ -235,7 +235,7 @@ retry: > #define SLABS_PER_PAGE (1 << (PAGE_SHIFT - IO_TLB_SHIFT)) > #define IO_TLB_MIN_SLABS ((1<<20) >> IO_TLB_SHIFT) > while ((SLABS_PER_PAGE << order) > IO_TLB_MIN_SLABS) { > - xen_io_tlb_start = (void *)__get_free_pages(__GFP_NOWARN, order); > + xen_io_tlb_start = (void *)__get_free_pages(__GFP_NOWARN|__GFP_DMA32, order); ^^ __GFP_DMA works on arm64 Cheers, Baozi.