Re: Question about DMA on 1:1 mapping dom0 of arm64

[Chen Baozi <cbz@baozis.org>]

On Fri, Apr 17, 2015 at 03:32:20PM +0100, Stefano Stabellini wrote:
> On Fri, 17 Apr 2015, Chen Baozi wrote:
> > Hi all,
> > 
> > According to my recent experience, there might be some problems of swiotlb
> > dma map on 1:1 mapping arm64 dom0 with large memory. The issue is like below:
> > 
> > For those arm64 server with large memory, it is possible to set dom0_mem >
> > 4G (e.g. I have one set with 16G). In this case, according to my understanding,
> > there is chance that the dom0 kernel needs to map some buffers above 4G to do
> > DMA operations (e.g. in snps,dwmac ethernet driver). However, most DMA engines
> > support only 32-bit physical address, thus aren't able to operate directly on
> > those memory. IIUC, swiotlb is implemented to solve this (using bounce buffer),
> > if there is no IOMMU or IOMMU is not enabled on the system. Sadly, it seems
> > that xen_swiotlb_map_page in my dom0 kernel allocates
> > (start_dma_addr = 0x944800000) the buffers for DMA above 4G which fails
> > dma_capable() checking and was then unable to return from xen_swiotlb_map_page()
> > successfully.
> >
> > If I set dom0_mem to a small value (e.g. 512M), which makes all physical memory
> > of dom0 below 4G, everything goes fine.
> > 
> > I am not familiar with swiotlb-xen, so there would be misunderstanding about
> > the current situation. Fix me if I did/understood anything wrong.
> > 
> > Any ideas?
> 
> I think that the problem is that xen_swiotlb_init doesn't necessarely allocate
> memory under 4G on arm/arm64.
> 
> xen_swiotlb_init calls __get_free_pages to allocate memory, so the pages
> could easily be above 4G.  Subsequently xen_swiotlb_fixup is called on
> the allocated memory range, calling xen_create_contiguous_region and
> passing an address_bits mask. However xen_create_contiguous_region
> doesn't actually do anything at all on ARM.
> 
> I think that given that dom0 is mapped 1:1 on ARM, the easiest and best
> fix would be to simply allocate memory under 4G to begin with. Something
> like (maybe with an ifdef ARM around it):
> 
> diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
> index 810ad41..22ac33a 100644
> --- a/drivers/xen/swiotlb-xen.c
> +++ b/drivers/xen/swiotlb-xen.c
> @@ -235,7 +235,7 @@ retry:
>  #define SLABS_PER_PAGE (1 << (PAGE_SHIFT - IO_TLB_SHIFT))
>  #define IO_TLB_MIN_SLABS ((1<<20) >> IO_TLB_SHIFT)
>  		while ((SLABS_PER_PAGE << order) > IO_TLB_MIN_SLABS) {
> -			xen_io_tlb_start = (void *)__get_free_pages(__GFP_NOWARN, order);
> +			xen_io_tlb_start = (void *)__get_free_pages(__GFP_NOWARN|__GFP_DMA32, order);
                                                                                 ^^ __GFP_DMA works on arm64

Cheers,

Baozi.