Re: [Qemu-devel] VFIO-VGA Issue

[deniv <deniv@lavabit.com>]

Alex Williamson:
> On Thu, 2013-04-25 at 11:38 +0000, deniv wrote:
>> Sorry for the long delay.
>>
>> Alex Williamson:
>>> On Thu, 2013-04-11 at 13:59 -0400, deniv@lavabit.com wrote:
>>>>> On Wed, 2013-04-10 at 16:32 -0400, deniv@lavabit.com wrote:
>>>>>>>> However, turning gfx_passthru off did
>>>>>>>> the trick. Win7 started loading with cirrus and switched to HD7750
>>>>>>>> halfway
>>>>>>>> through boot proccess. I didn't do any testing just let Windows
>>>>>>>> calculate
>>>>>>>> its score. The result was 7.4 and Aero was working.
>>>>>>>
>>>>>>> You should be able to do this with vfio too, use -vga cirrus and don't
>>>>>>> use the x-vga option on pci-assign.  The x-vga enables legacy VGA
>>>>>>> support for boot and primary console, as a secondary head normal PCI
>>>>>>> device assignment should be sufficient.
>>>>>>>
>>>>>> Oh, how I wish it was true! Trying to load with cirrus and vfio-pci
>>>>>> results in BSOD:
>>>>>> Attemp to reset the display driver and recover from timeout failed.
>>>>>
>>>>> Is this a fresh windows install?  Windows doesn't like change and will
>>>>> BSOD pretty easily when attaching graphics to an existing image.
>>>>>
>>>>>> Trying the old pci-assign with kvm results in non-working GFX. Windows
>>>>>> shows code 10 and sometimes code 43 for the card.
>>>>>
>>>>> What happens if you don't use a q35 machine?  Windows is very particular
>>>>> about the PCIe type of a device and will often show Code 10 if it
>>>>> doesn't have a type compatible with a root complex.  Alternatively you
>>>>> can use the q35 config in the docs directory with the -readconfig option
>>>>> and the bus= option on the pci-assign device to place the graphics
>>>>> behind a root port.
>>>>>
>>>>
>>>> After many attempts I have VGA passthrough working. Each test has been
>>>> performed in a clean Win7 install with kvm enabled. Installation was done
>>>> with i440fx machine and changed to q35 after virtio drivers were
>>>> installed. I did that because there's no AHCI driver for q35 yet and win7
>>>> doesn't have drivers for lsi scsi.
>>>>
>>>> 1. q35/pc, vga=none, vfio-pci, x-vga=on. Nothing on the VM's screen, debug
>>>> output can be seen in the previous mails.
>>>> 2. q35/pc, vga=cirrus, vfio-pci, x-vga=on. Screen corruption on host
>>>> (correction to the previous mails: corruption also happens even in kms
>>>> console). Nothing on the VM's screen.
>>>> 3. q35/pc, vga=cirrus, vfio-pci, no x-vga. BSOD: Attempt to reset the
>>>> display driver and recover from timeout failed.
>>>> 4. q35, vga=cirrus, pci-assign. Windows boots, the graphic card doesn't
>>>> start: Code 10.
>>>> 5. pc, vga=cirrus, pci-assign. IT'S ALIVE! Ironically, this is the oldest
>>>> way to assign pci devices in kvm. Why I could make it work previously?
>>>> Silly me!
>>>
>>> What does your /sys/kernel/iommu_groups look like for the group
>>> containing your VGA device?  I'm curious if it includes the PCIe root
>>> port and whether you're attaching those to vfio-pci or leaving them
>>> bound to pcieport.  If the former, that may contribute to why you're
>>> having problems with vfio-pci.
>> Yes, the group containing the VGA device includes PCIe root port. No, I
>> did not attach it to vfio-pci. I tried this right now, it didn't make
>> any difference.
> 
> You were right by not attaching it to vfio-pci.  Previous versions of
> vfio-pci still required it, but we've since found that causes more
> problems than it solves.  I wanted to make sure it wasn't attached.
> 
>> For what it's worth, I also noticed errors in dmesg output when Windows
>> BSODs (q35, vga=cirrus, vfio-pci, no x-vga). There are about ten
>> thousand lines of
>> ---
>> dmar: DRHD: handling fault status reg 3
>> dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 1200b6000
>> DMAR:[fault reason 06] PTE Read access is not set
>> ---
>> Fault addresses start at 11fff6000 (always the same) and go to about
>> 1201b3000 (varies on each start).
>>
>> Those read faults are followed a bunch of
>> ---
>> dmar: DRHD: handling fault status reg 3
>> dmar: DMAR:[DMA Write] Request device [01:00.0] fault addr 11ffed000
>> DMAR:[fault reason 05] PTE Write access is not set
>> ---
>> Fault addresses 11fef3000-11fff0000 (the last address varies).
> 
> And only with vfio-pci?  Very odd.
> 
> I've since gotten my HD7850 working with vfio-pci,x-vga=on with my Intel
> VT-d system.  Gleb is working on a patch to fix the emulator bug, but it
> can be avoided using the emulate_invalid_guest_state=0 module option to
> kvm_intel.
Okay. This time I used your next branch. kernel command line included
"kvm_intel.emulate_invalid_guest_state=0"

> I see we've already mentioned it in this thread, but I'll re-iterate the
> importance of not loading host drivers for graphics cards until we can
> reset them better.  Another user was having trouble with nvidia cards
> that was cleared by not loading the host nvidia kernel driver.
As far as I know, nothing touches HD7750 before qemu comes in. My kernel
config: http://pastebin.com/sE73CMgH

> I believe you're using the stock upstream kernel, 3.9-rc5 last I saw in
> this thread.  One potential difference between vfio-pci and pci-assign
> is that vfio does not allow access to unarchitected PCI config space.
> That is, regions not covered by capabilities.  This has also been shown
> to be a problem, so it's being fixed, but is not yet upstream.  I plan
> to push this in for 3.10, but it's already in my next branch:
I reran some test with your branch.

pc/q35, vga=none, vfio-pci, x-vga=on. As previously, without 'if (1 ||
data == quirk->data)' in hw/misc/vfio.c qemu freezes early. With the
patch it goes a bit further, corrupts the main screen, and freezes again.

pc, vga=cirrus, vfio-pci, no x-vga.
[  104.469331] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[  104.494873] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270
[  125.895842] vfio-pci 0000:01:00.0: irq 48 for MSI/MSI-X
And then BSOD: Attempt to reset the display driver and recover from
timeout failed.

q35, vga=cirrus, vfio-pci, no x-vga. It's like the previous config, but
with many DMAR faults.
[   73.235513] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[   73.260776] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270
[   95.919689] vfio-pci 0000:01:00.0: irq 48 for MSI/MSI-X
[   97.642747] dmar: DRHD: handling fault status reg 3
[   97.642752] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr
11fff6000
[   97.642752] DMAR:[fault reason 06] PTE Read access is not set
..
+ BSOD.

q35, vga=cirrus, pci-assign. Windows boots, the GPU doesn't start (code 10).
pc, vga=cirrus, pci-assign. This is the working config.