Re: GPU passthrough issue when VM is configured with 4G memory

[Gordan Bobic <gordan@bobich.net>]

On 05/22/2013 07:11 PM, Andrew Bobulsky wrote:
> On Wed, May 22, 2013 at 11:17 AM, Gordan Bobic <gordan@bobich.net
> <mailto:gordan@bobich.net>> wrote:
>
>     On Tue, 21 May 2013 04:09:25 +0000, Hanweidong
>     <hanweidong@huawei.com <mailto:hanweidong@huawei.com>> wrote:
>
>             -----Original Message-----
>             From: Gordan Bobic [mailto:gordan@bobich.net
>             <mailto:gordan@bobich.net>]
>             Sent: 2013年5月21日 2:43
>             To: George Dunlap
>             Cc: Hanweidong; Stefano Stabellini; xudong.hao@intel.com
>             <mailto:xudong.hao@intel.com>; Yanqiangjun;
>             Luonengjun; Wangzhenguo; Yangxiaowei; Gonglei (Arei);
>             Anthony Perard;
>             xen-devel@lists.xen.org <mailto:xen-devel@lists.xen.org>;
>             xiantao.zhang@intel.com <mailto:xiantao.zhang@intel.com>
>             Subject: Re: [Xen-devel] GPU passthrough issue when VM is
>             configured
>             with 4G memory
>
>             I'd also like to stress this is not only an issue for > 4GB
>             of RAM in
>             domU - I am seeing the issue with > 2GB of RAM in domU.
>
>
>         What's the total mmio size of you domU?
>
>
>     How can I find that out?
>
>
>         When RAM of domU overlaps
>         with pci_mem_start, hvmloader will relocate RAM, and it will cause
>         problem due to hvmloader and QEMU don't setup the memory layout
>         consistently. I suspect the mmio size of your domU is close to 2GB.
>         When you configured RAM > 2G, then RAM of your domU overlapped with
>         pci_mem_start, and resulted in failure.
>
>
>     Is there a way to force a hole between, say, 2GB and 4GB explicitly
>     to avoid the PCI memory being clobbered?
>
> I'll second that question!
> Gordan, one thing I didn't think of until now: the card I was having
> trouble with, a Radeon 6990, is a 4GB card, laid out as two 2GB GPUs.
> The card with which I don't have issues (aside from the occasional need
> to reboot the whole system to get it to work right), a Radeon 5850, is a
> 1GB card.  If as this thread has suggested, along with what I've read
> about the PCI Hole, the memory on the GPU determines the "clobber size"
> (yes I made that term up :P),

I'm pretty sure the GPU RAM size is not relevant. I've seen this with 
the 1GB Nvidia Quadro 2000, 1GB ATI 7450 and the 6GB 7970. The GPU 
memory size does not affect the aperture size or location. I for one 
would love it if all of the GPU RAM was in fact mapped as a BAR - that 
would open up some very funky approaches to processing large amounts of 
data, e.g. by just mapping the data straight to the GPU. It would also 
make it trivially easy to use all that RAM as a RAM disk when the GPU 
isn't being used (e.g. you can use the RAM for fast swap for normal 
desktop tasks, and free it up when gaming). but that's a whole different 
topic.

Gordan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel