Re: GPU passthrough issue when VM is configured with 4G memory

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Wed, May 29, 2013 at 05:18:24PM +0100, Stefano Stabellini wrote:
> On Thu, 25 Apr 2013, Hanweidong wrote:
> > > -----Original Message-----
> > > From: xen-devel-bounces@lists.xen.org [mailto:xen-devel-
> > > bounces@lists.xen.org] On Behalf Of Hanweidong
> > > Sent: 2013年3月26日 17:38
> > > To: Stefano Stabellini
> > > Cc: George Dunlap; xudong.hao@intel.com; Yanqiangjun; Luonengjun;
> > > Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-
> > > devel@lists.xen.org; xiantao.zhang@intel.com
> > > Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
> > > with 4G memory
> > > 
> > > 
> > > > -----Original Message-----
> > > > From: Stefano Stabellini [mailto:stefano.stabellini@eu.citrix.com]
> > > > Sent: 2013年3月18日 20:02
> > > > To: Hanweidong
> > > > Cc: George Dunlap; Stefano Stabellini; Yanqiangjun; Luonengjun;
> > > > Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-
> > > > devel@lists.xen.org; xudong.hao@intel.com; xiantao.zhang@intel.com
> > > > Subject: RE: [Xen-devel] GPU passthrough issue when VM is configured
> > > > with 4G memory
> > > >
> > > > On Wed, 13 Mar 2013, Hanweidong wrote:
> > > > > MMIO HOLE was adjusted to e0000000 - fc000000. But QEMU uses below
> > > > code to init
> > > > > RAM in xen_ram_init:
> > > > >
> > > > >     ...
> > > > >     block_len = ram_size;
> > > > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> > > > >         /* Xen does not allocate the memory continuously, and keep
> > > a
> > > > hole at
> > > > >          * HVM_BELOW_4G_MMIO_START of HVM_BELOW_4G_MMIO_LENGTH
> > > > >          */
> > > > >         block_len += HVM_BELOW_4G_MMIO_LENGTH;
> > > > >     }
> > > > >     memory_region_init_ram(&ram_memory, "xen.ram", block_len);
> > > > >     vmstate_register_ram_global(&ram_memory);
> > > > >
> > > > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> > > > >         above_4g_mem_size = ram_size - HVM_BELOW_4G_RAM_END;
> > > > >         below_4g_mem_size = HVM_BELOW_4G_RAM_END;
> > > > >     } else {
> > > > >         below_4g_mem_size = ram_size;
> > > > >     }
> > > > >     ...
> > > > >
> > > > > HVM_BELOW_4G_RAM_END is f0000000. If we change HVM_BELOW_4G_RAM_END
> > > > to e0000000,
> > > > > Which it's consistent with hvmloader when assigning a GPU, and then
> > > > guest worked
> > > > > for us. So we wondering that xen_ram_init in QEMU should be
> > > > consistent with
> > > > > hvmloader.
> > > > >
> > > > > In addition, we found QEMU uses hardcode 0xe0000000 in pc_init1()
> > > as
> > > > below.
> > > > > Should keep these places handle the consistent mmio hole or not?
> > > > >
> > > > >     if (ram_size >= 0xe0000000 ) {
> > > > >         above_4g_mem_size = ram_size - 0xe0000000;
> > > > >         below_4g_mem_size = 0xe0000000;
> > > > >     } else {
> > > > >         above_4g_mem_size = 0;
> > > > >         below_4g_mem_size = ram_size;
> > > > >     }
> > > >
> > > > The guys at Intel sent a couple of patches recently to fix this issue:
> > > >
> > > > http://marc.info/?l=xen-devel&m=136150317011027
> > > > http://marc.info/?l=qemu-devel&m=136177475215360&w=2
> > > >
> > > > Do they solve your problem?
> > > 
> > > These two patches didn't solve our problem.
> > > 
> > 
> > I debugged this issue with above two patches. I want to share some information and discuss solution here. This issue is actually caused by that a VM has a large pci hole (mmio size) which results in QEMU sets memory regions inconsistently with hvmloader (QEMU uses hardcode 0xe0000000 in pc_init1 and xen_ram_init). I created a virtual device with 1GB mmio size to debug this issue. Firstly, QEMU set memory regions except pci hole region in pc_init1() and xen_ram_init(), then hvmloader calculated pci_mem_start as 0x80000000, and wrote it to TOM register, which triggered QEMU to update pci hole region with 0x80000000 using i440fx_update_pci_mem_hole(). Finally the windows 7 VM (configured 8G) crashed with BSOD code 0x00000024. If I hardcode in QEMU pc_init1 and xen_ram_init to match hvmloader's. Then the problem was gone. 
> > 
> > Althrough above two patches will pass actual pci hole start address to QEMU, but it's too late, QEMU pc_init1() and xen_ram_init() already set the other memory regions, and obviously the pci hole might overlap with ram regions in this case. So I think hvmloader should setup pci devices and calculate pci hole first, then QEMU can map memory regions correctly from the beginning.  
> > 
> 
> Thank you very much for your detailed analysis of the problem.
> 
> After reading this, I wonder how is possible that qemu-xen-traditional
> does not have this issue, considering that AFAIK there is no way for
> hvmloader to tell qemu-xen-traditional where the PCI hole starts.
> 
> The only difference between upstream QEMU and qemu-xen-traditional is
> that the former would start the PCI hole at 0xf0000000 while the latter
> would start the PCI hole at 0xe0000000.
> 
> So I would expect that your test, where hvmloader is updating the PCI
> hole region to start at 0x80000000, would fail on qemu-xen-traditional
> too.
> 
> Of course having the PCI hole starting unconditionally at 0xf0000000
> makes it much easier to run into problems than starting it at
> 0xe0000000.
> 
> 
> Assuming that everything above is correct, this is what I would do:
> 
> 1) modify upstream QEMU to start the PCI hole at 0xe0000000, to match
> qemu-xen-unstable in terms of configuration and not to introduce any
> regressions. Do this for the Xen 4.3 release.
> 
> 2) for Xen 4.4 rework the two patches above and improve
> i440fx_update_pci_mem_hole: resizing the pci_hole subregion is not
> enough, it also needs to be able to resize the system memory region
> (xen.ram) to make room for the bigger pci_hole


Would that make migration more difficult - meaning if you have now two
different QEMU versions where the PCI hole is different on them? Or is
that not an issue and QEMU handles setting the layout nicely? Or is
the 0xe0000000 the norm in Xen 4.1, and Xen 4.2?

I am assuming you unplug the PCI device before you migrate of course.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel