Re: GPU passthrough issue when VM is configured with 4G memory

[youenn.gestin@grim-is-a-geek.fr]

Le 13.03.2013 14:23, Hanweidong a écrit :
>> -----Original Message-----
>> From: dunlapg@gmail.com [mailto:dunlapg@gmail.com] On Behalf Of 
>> George
>> Dunlap
>> Sent: 2013年3月12日 18:40
>> To: Hanweidong
>> Cc: Stefano Stabellini; Yanqiangjun; Luonengjun; Wangzhenguo;
>> Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-devel@lists.xen.org
>> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
>> with 4G memory
>>
>> On Tue, Mar 12, 2013 at 5:45 AM, Hanweidong <hanweidong@huawei.com>
>> wrote:
>> > I don't think it's a pass-through issue. We also encountered 
>> similar
>> > issue when using ivshmem device with 512M mmio. The problem is
>> hvmloader
>> > adjusted pci_mem_start according to actual mmio size of VM, but 
>> QEMU
>> does
>> > not know the actual mmio size of VM, it just uses the
>> HVM_BELOW_4G_RAM_END
>> > and HVM_BELOW_4G_MMIO_LENGTH in xen_ram_init. This results in 
>> memory
>> layout
>> > mismatch between hvmloader and QEMU. I think it needs to make QEMU
>> know
>> > actual mmio size, and then it can make the same memory layout as
>> hvmloader.
>>
>> So I'm not an expert in this codepath, but it was my understanding
>> that resizing the MMIO hole is actually the job of the guest BIOS: 
>> it
>> is supposed to query the PCI devices to find out how much memory 
>> they
>> need, and then relocate memory as appropriate.  XenServer never had
>> any problem with the MMIO hole not being big enough when passing 
>> cards
>> to the guest; the only problem we've ever had (as far as I know) is
>> the issue I mentioned before, where there was a bug in some pci 
>> bridge
>> cards that breaks VT-d if the guest physical address overlaps the 
>> host
>> physical address range of another device.
>>
>> Can you maybe boot Linux in an HVM domain and do an lspci or look at
>> Linux's e820 map, and check to see whether the BIOS has in fact
>> relocated the guest memory, or whether the MMIO hole does indeed 
>> start
>> only at HVM_BELOW_4G_RAM_END?
>>
>
> We tried Linux guest, but it stopped at grub screen, cannot boot up. 
> Posted
> "xl dmesg" output as below:
>
> (XEN) HVM13: HVM Loader
> (XEN) HVM13: Detected Xen v4.3-unstable
> (XEN) HVM13: Xenbus rings @0xfeffc000, event channel 3
> (XEN) HVM13: System requested SeaBIOS
> (XEN) HVM13: CPU speed is 2500 MHz
> (XEN) irq.c:270: Dom13 PCI link 0 changed 0 -> 5
> (XEN) HVM13: PCI-ISA link 0 routed to IRQ5
> (XEN) irq.c:270: Dom13 PCI link 1 changed 0 -> 10
> (XEN) HVM13: PCI-ISA link 1 routed to IRQ10
> (XEN) irq.c:270: Dom13 PCI link 2 changed 0 -> 11
> (XEN) HVM13: PCI-ISA link 2 routed to IRQ11
> (XEN) irq.c:270: Dom13 PCI link 3 changed 0 -> 5
> (XEN) HVM13: PCI-ISA link 3 routed to IRQ5
> (XEN) HVM13: pci dev 01:2 INTD->IRQ5
> (XEN) HVM13: pci dev 01:3 INTA->IRQ10
> (XEN) HVM13: pci dev 03:0 INTA->IRQ5
> (XEN) HVM13: pci dev 04:0 INTA->IRQ5
> (XEN) HVM13: pci dev 05:0 INTA->IRQ10
> (XEN) HVM13: pci dev 05:0 bar 14 size lx: 08000000
> (XEN) memory_map:add: dom13 gfn=e0000 mfn=e8000 nr=8000
> (XEN) memory_map:add: dom13 gfn=e8000 mfn=f0000 nr=4000
> (XEN) HVM13: pci dev 05:0 bar 1c size lx: 04000000
> (XEN) HVM13: pci dev 02:0 bar 10 size lx: 02000000
> (XEN) memory_map:add: dom13 gfn=ee000 mfn=bc000 nr=2000
> (XEN) HVM13: pci dev 05:0 bar 10 size lx: 02000000
> (XEN) HVM13: pci dev 03:0 bar 14 size lx: 01000000
> (XEN) HVM13: pci dev 05:0 bar 30 size lx: 00080000
> (XEN) HVM13: pci dev 02:0 bar 30 size lx: 00010000
> (XEN) HVM13: pci dev 04:0 bar 30 size lx: 00010000
> (XEN) HVM13: pci dev 02:0 bar 14 size lx: 00001000
> (XEN) HVM13: pci dev 03:0 bar 10 size lx: 00000100
> (XEN) HVM13: pci dev 04:0 bar 10 size lx: 00000100
> (XEN) HVM13: pci dev 04:0 bar 14 size lx: 00000100
> (XEN) HVM13: pci dev 05:0 bar 24 size lx: 00000080
> (XEN) ioport_map:add: dom13 gport=c200 mport=2000 nr=80
> (XEN) HVM13: pci dev 01:2 bar 20 size lx: 00000020
> (XEN) HVM13: pci dev 01:1 bar 20 size lx: 00000010
> (XEN) HVM13: Multiprocessor initialisation:
> (XEN) HVM13:  - CPU0 ... 46-bit phys ... fixed MTRRs ... var MTRRs
> [3/8] ... done.
> (XEN) HVM13: Testing HVM environment:
> (XEN) HVM13:  - REP INSB across page boundaries ... passed
> (XEN) HVM13:  - GS base MSRs and SWAPGS ... passed
> (XEN) HVM13: Passed 2 of 2 tests
> (XEN) HVM13: Writing SMBIOS tables ...
> (XEN) HVM13: Loading SeaBIOS ...
> (XEN) HVM13: Creating MP tables ...
> (XEN) HVM13: Loading ACPI ...
> (XEN) HVM13: vm86 TSS at fc00a000
> (XEN) HVM13: BIOS map:
> (XEN) HVM13:  10000-100d3: Scratch space
> (XEN) HVM13:  e0000-fffff: Main BIOS
> (XEN) HVM13: E820 table:
> (XEN) HVM13:  [00]: 00000000:00000000 - 00000000:000a0000: RAM
> (XEN) HVM13:  HOLE: 00000000:000a0000 - 00000000:000e0000
> (XEN) HVM13:  [01]: 00000000:000e0000 - 00000000:00100000: RESERVED
> (XEN) HVM13:  [02]: 00000000:00100000 - 00000000:e0000000: RAM
> (XEN) HVM13:  HOLE: 00000000:e0000000 - 00000000:fc000000
> (XEN) HVM13:  [03]: 00000000:fc000000 - 00000001:00000000: RESERVED
> (XEN) HVM13:  [04]: 00000001:00000000 - 00000001:1f800000: RAM
> (XEN) HVM13: Invoking SeaBIOS ...
> (XEN) HVM13: SeaBIOS (version 
> rel-1.7.1-0-g51755c3-20130227_164001-linux-DPMZPO)
> (XEN) HVM13:
> (XEN) HVM13: Found Xen hypervisor signature at 40000000
> (XEN) HVM13: xen: copy e820...
> (XEN) HVM13: Ram Size=0xe0000000 (0x000000001f800000 high)
> (XEN) HVM13: Relocating low data from 0x000e2490 to 0x000ef790 (size 
> 2156)
> (XEN) HVM13: Relocating init from 0x000e2cfc to 0xdffe20f0 (size 
> 56804)
> (XEN) HVM13: CPU Mhz=2501
> (XEN) HVM13: Found 9 PCI devices (max PCI bus is 00)
> (XEN) HVM13: Allocated Xen hypercall page at dffff000
> (XEN) HVM13: Detected Xen v4.3-unstable
> (XEN) HVM13: Found 1 cpu(s) max supported 1 cpu(s)
> (XEN) HVM13: xen: copy BIOS tables...
> (XEN) HVM13: Copying SMBIOS entry point from 0x00010010 to 0x000fdb10
> (XEN) HVM13: Copying MPTABLE from 0xfc001140/fc001150 to 0x000fda30
> (XEN) HVM13: Copying PIR from 0x00010030 to 0x000fd9b0
> (XEN) HVM13: Copying ACPI RSDP from 0x000100b0 to 0x000fd980
> (XEN) HVM13: Scan for VGA option rom
> (XEN) HVM13: Running option rom at c000:0003
> (XEN) stdvga.c:147:d13 entering stdvga and caching modes
> (XEN) HVM13: Turning on vga text mode console
> (XEN) HVM13: SeaBIOS (version 
> rel-1.7.1-0-g51755c3-20130227_164001-linux-DPMZPO)
> (XEN) HVM13:
> (XEN) HVM13: UHCI init on dev 00:01.2 (io=c280)
> (XEN) HVM13: Found 1 lpt ports
> (XEN) HVM13: Found 1 serial ports
> (XEN) HVM13: ATA controller 1 at 1f0/3f4/c2a0 (irq 14 dev 9)
> (XEN) HVM13: ATA controller 2 at 170/374/c2a8 (irq 15 dev 9)
> (XEN) HVM13: ata0-0: QEMU HARDDISK ATA-7 Hard-Disk (20480 MiBytes)
> (XEN) HVM13: Searching bootorder for: /pci@i0cf8/*@1,1/drive@0/disk@0
> (XEN) HVM13: DVD/CD [ata1-0: QEMU DVD-ROM ATAPI-4 DVD/CD]
> (XEN) HVM13: Searching bootorder for: /pci@i0cf8/*@1,1/drive@1/disk@0
> (XEN) HVM13: PS2 keyboard initialized
> (XEN) HVM13: All threads complete.
> (XEN) HVM13: Scan for option roms
> (XEN) HVM13: Running option rom at c900:0003
> (XEN) HVM13: pmm call arg1=1
> (XEN) HVM13: pmm call arg1=0
> (XEN) HVM13: pmm call arg1=1
> (XEN) HVM13: pmm call arg1=0
> (XEN) HVM13: Searching bootorder for: /pci@i0cf8/*@4
> (XEN) HVM13: Press F12 for boot menu.
> (XEN) HVM13:
> (XEN) HVM13: drive 0x000fd930: PCHS=16383/16/63 translation=lba
> LCHS=1024/255/63 s=41943040
> (XEN) HVM13:
> (XEN) HVM13: Space available for UMB: 000ca000-000ee800
> (XEN) HVM13: Returned 61440 bytes of ZoneHigh
> (XEN) HVM13: e820 map has 7 items:
> (XEN) HVM13:   0: 0000000000000000 - 000000000009fc00 = 1 RAM
> (XEN) HVM13:   1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
> (XEN) HVM13:   2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
> (XEN) HVM13:   3: 0000000000100000 - 00000000dffff000 = 1 RAM
> (XEN) HVM13:   4: 00000000dffff000 - 00000000e0000000 = 2 RESERVED
> (XEN) HVM13:   5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
> (XEN) HVM13:   6: 0000000100000000 - 000000011f800000 = 1 RAM
> (XEN) HVM13: enter handle_19:
> (XEN) HVM13:   NULL
> (XEN) HVM13: Booting from Hard Disk...
> (XEN) HVM13: Booting from 0000:7c00
> (XEN) stdvga.c:151:d13 leaving stdvga
> (XEN) stdvga.c:147:d13 entering stdvga and caching modes
>
> MMIO HOLE was adjusted to e0000000 - fc000000. But QEMU uses below 
> code to init
> RAM in xen_ram_init:
>
>     ...
>     block_len = ram_size;
>     if (ram_size >= HVM_BELOW_4G_RAM_END) {
>         /* Xen does not allocate the memory continuously, and keep a 
> hole at
>          * HVM_BELOW_4G_MMIO_START of HVM_BELOW_4G_MMIO_LENGTH
>          */
>         block_len += HVM_BELOW_4G_MMIO_LENGTH;
>     }
>     memory_region_init_ram(&ram_memory, "xen.ram", block_len);
>     vmstate_register_ram_global(&ram_memory);
>
>     if (ram_size >= HVM_BELOW_4G_RAM_END) {
>         above_4g_mem_size = ram_size - HVM_BELOW_4G_RAM_END;
>         below_4g_mem_size = HVM_BELOW_4G_RAM_END;
>     } else {
>         below_4g_mem_size = ram_size;
>     }
>     ...
>
> HVM_BELOW_4G_RAM_END is f0000000. If we change HVM_BELOW_4G_RAM_END
> to e0000000,
> Which it's consistent with hvmloader when assigning a GPU, and then
> guest worked
> for us. So we wondering that xen_ram_init in QEMU should be 
> consistent with
> hvmloader.
>
> In addition, we found QEMU uses hardcode 0xe0000000 in pc_init1() as 
> below.
> Should keep these places handle the consistent mmio hole or not?
>
>     if (ram_size >= 0xe0000000 ) {
>         above_4g_mem_size = ram_size - 0xe0000000;
>         below_4g_mem_size = 0xe0000000;
>     } else {
>         above_4g_mem_size = 0;
>         below_4g_mem_size = ram_size;
>     }
>
> --weidong
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Hello,

What can of bug we can encounter by hardcoding the HVM_BELOW_4G_RAM_END 
to e0000000 ?

By doing this I'm able to boot on Win7 with gpu passthrough of an 
nvidia 560TI an 12Go of ram.

Regards and thanks for all your work !

Youenn.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel