Re: Upstream Dom0 DRM problems regarding swiotlb

From: Christoph Hellwig <hch@infradead.org>
To: Michael Labriola <michael.d.labriola@gmail.com>
Cc: Juergen Gross <jgross@suse.com>,
	Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>,
	Andrew Cooper <andrew.cooper3@citrix.com>,
	xen-devel@lists.xen.org, Christoph Hellwig <hch@infradead.org>,
	Paul Durrant <Paul.Durrant@citrix.com>,
	Jan Beulich <JBeulich@suse.com>
Subject: Re: Upstream Dom0 DRM problems regarding swiotlb
Date: Fri, 15 Feb 2019 08:42:01 -0800	[thread overview]
Message-ID: <20190215164201.GA12704@infradead.org> (raw)
In-Reply-To: <CAOQxz3yhQsRhSTWBSwVhHP-BPqr3uTDpqa8bO44YTgUCYEkbXw@mail.gmail.com>

On Fri, Feb 15, 2019 at 11:07:22AM -0500, Michael Labriola wrote:
> > > But the latter text seems to agree with that.  So what is the actual
> > > problem that started this discussion?
> > >
> >
> > See https://lists.xen.org/archives/html/xen-devel/2019-02/threads.html#00818
> 
> I believe the actual problem is either:
> 
> 1) The radeon/amdgpu drivers are calling ttm_populate_and_map_pages()
> which *should* work on a Xen PV host, but doesn't and needs to be
> fixed.
> 
> or
> 
> 2) The radeon/amdgpu drivers are calling ttm_populate_and_map_pages()
> which *cannot* work in Xen, and they should go back to calling
> ttm_dma_populate() in that case.
> 
> I'm having a hard time figuring out which of those is correct.

I think the answer is neither 1 or 2 (or a bit of both).

ttm_populate_and_map_pages uses dma_map_page to map GPU memory,
and from what I can tell from your report potentially lots of it.

So this does "properly" use the DMA API for some amount of "properly".
The problem here is that ttm_populate_and_map_pages first allocates
memory and then maps it in a way where it bounce buffers a lot,
leading to a swiotlb buffer exaustion, as seen in your report.

ttm_dma_populate also sort of "properly" uses the DMA API in that it
uses the dma_alloc_coherent allocator.  The benefit of that allocator is
that is always returns addressable memory without exhausing the swiotlb
buffer.  The dowside of ttm_dma_populate / dma_alloc_coherent is that
on architectures where PCIe is not cache coherent it pointlessly
up other resources, as coherent DMA memory can be a very finite resource
there.

So for a short term fix forcing the dma_alloc_coherent route on
Xen/x86 is the right thing.  On Xen/arm and Xen/arm64 is might already
be problemeatic due to the explanation above unfortunately.

The real fix is to finally get broadly available non-coherent device
memory allocator into mainlaine and with ttm_populate_and_map_pages
to use it.  Note that for non-coherent devices it seems like
ttm_populate_and_map_pages also seems to miss proper ownership
management but that is another issue.

My proposal for such an allocator here:

   https://lwn.net/Articles/774429/

unfortunately doesn't seem to go anywhere as the DRM folks generally
seem to prefer bolting band-aid ontop of band-aid instead of actually
fixing such problems.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel