[Bug 107561] New: 4.2 breaks PCI passthrough in QEMU/KVM

[Bug 107561] New: 4.2 breaks PCI passthrough in QEMU/KVM

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=107561

            Bug ID: 107561
           Summary: 4.2 breaks PCI passthrough in QEMU/KVM
           Product: Virtualization
           Version: unspecified
    Kernel Version: 4.2
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: kvm
          Assignee: virtualization_kvm@kernel-bugs.osdl.org
          Reporter: fhorn50@gmail.com
        Regression: No

QEMU/KVM VM's using PCI pass-through are taking a long time to boot, or never
booting. The issue is only present when the VM has more then ~2.5G of ram. I
was able to get a VM with 6G to boot, but I had to let it run overnight.

More info can be found in this thread:
https://bbs.archlinux.org/viewtopic.php?id=203240

Someone in the forum seems to have narrowed it down to a a set of commits
regarding kvm and mtrr in 4.1-rc2 that are in the mainline kernel as of 4.2.

Specifically commit fa61213746a706dd975661151c35795ca4dd82c2 KVM: MTRR:
simplify kvm_mtrr_get_guest_memory_type.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=107561

John Brooks <kbt@fastquake.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kbt@fastquake.com

--- Comment #1 from John Brooks <kbt@fastquake.com> ---
I can confirm this bug. As soon as I enabled PCI passthrough on my VM while
using kernel 4.2.2, my VM ran extremely slowly, and increasing the amount of
RAM made the problem worse. Switching to kernel 4.1.12 fixed the problem.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=107561

Aaron Campbell <aaron@monkey.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |aaron@monkey.org

--- Comment #2 from Aaron Campbell <aaron@monkey.org> ---
I can also confirm this bug.  After upgrading the kernel on a Fedora box from
4.0.4 to 4.2.5, my Windows 8.1 VM hung on start.  The VM would only start if
the GPU passthrough configuration was removed, or if the VM's memory size was
reduced to 2GB.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=107561

brenix@gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |brenix@gmail.com

--- Comment #3 from brenix@gmail.com ---
Issue is still present in 4.4-rc1 and same conditions exist. After lowering it
below 2.5G of ram, the VM boots normally. Anything over 2.5G will cause it to
boot really slow and utilize a lot of cpu.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=107561

schefister@gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |schefister@gmail.com

--- Comment #4 from schefister@gmail.com ---
I narrowed it down earlier to "simplify kvm_mtrr_get_guest_memory_type".
Actually this was the commit to break it, but it is only a small part of the
feauture set implementing vMTRR. To my knowledge 4.1.x versions didn't have
this code set at all. It was introduced in 4.2.
What kvm_mtrr_get_guest_memory_type() does is basically decides what cache type
to use (Uncachable, Write-Back, or Write-Through) based on the function
mtrr_for_each_mem_type(). This kvm_mtrr_get_guest_memory_type function is only
called by vmx_get_mt_mask. This is all related to how the hardware is set up
regarding IOMMU, MMIO, VT-d and EPT andress translation.

For experimenting purposes I simply modified kvm_mtrr_get_guest_memory_type to
always return MTRR_TYPE_WRBACK and that fixed the issue. Guest boots correctly. 
Although I know, that hardcoding Write-back mode is not a viable long-term
solution. It seems as the new function (in 4.2 and onward) does not return the
proper cache type in case of pci passthrough configurations. 

I would also like to point out, that
https://bugzilla.kernel.org/show_bug.cgi?id=107921 may be related. The other
bug mentions very similar hardware to mine, and issues started the same time
(upgrading to 4.2.x) with a pci passthrough configuration.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=107561

Paolo Bonzini <bonzini@gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bonzini@gnu.org

--- Comment #5 from Paolo Bonzini <bonzini@gnu.org> ---
Please capture a trace according to http://www.linux-kvm.org/page/Tracing, so
that I can see your MTRR settings and figure out why that patch breaks things.

If possible, capture a trace with 2 GB of memory and one with 3 GB.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=107561

--- Comment #6 from Paolo Bonzini <bonzini@gnu.org> ---
Ah, the traces are huge. You can xz them and send them through private email.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=107561

Paolo Bonzini <bonzini@gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |tajjada@gmail.com

--- Comment #7 from Paolo Bonzini <bonzini@gnu.org> ---
*** Bug 107921 has been marked as a duplicate of this bug. ***

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=107561

--- Comment #8 from schefister@gmail.com ---
I captured the traces, they take more than half a GB uncompressed. They are
available in xz under
https://drive.google.com/folderview?id=0B8ebX_WjVHnGNlN4eTEzU2xtMEk&usp=sharing

To make things clear: I have two hosts. HostB is a testing machine. VGA
passthrough with EDKII worked out of the box on 4.1 kernels. It broke with with
several 4.2 versions and also 4.3. Tried several repo versions and untouched
kernel.org versions, that was back about a month ago. Than I tested again back
a few days, and it works again with 4.2.6-301.fc23 fedora repo kernel.

HostA is the problematic one. I did the trace as requested, with 2G and 3G
guest ram. It worked with 4.1 kernels, but still doesn't work with
4.2.6-301.fc23. I can make it work with either lowering RAM to under 2.5 GB or
with my beforementioned modification (kvm_mtrr_get_guest_memory_type to always
return MTRR_TYPE_WRBACK). Of course I made the traces with unmodified kernel,
and only the 2G guest actually booted.
The exact symptom is (when I say it doesn't work), that the guest is extremly
slow. I tried booting live Linux guest, after about 15 minutes a saw messages,
but even after two hours I still couldn't get to a shell. Windows guest only
shows the white dots under the logo circling around very slowly. Once I got a
blue screen, and letters came up one-by-one, like if the error message was
written with a typewriter. Sometimes the guest just shuts down, qemu
terminates. UEFI shell, UEFI setup in guest works perfectly at all conditions,
slowness starts when booting an OS.

I can also provide the traces with working kernel version on HostA and also on
HostB, if requested for comparison.

In the shared folders you can find MTRR, PAT, and lspci -vvv info for each
host, along with traces for HostA as requested (2GB and 3GB). One of the
members on the original Arch Linux thread suggested I put a printk in the
problematic function. The dmesg files in each folder show the arguments of
vmx_get_mt_mask and what kvm_mtrr_get_guest_memory_type returns to it. 
The added line was (just before the return statement in vmx_get_mt_mask):

printk(KERN_INFO "vmx_get_mt_mask got the following: cpu=%d, vcpu=%d, gfn=%x,
MMIO=%d, cache=%x", vcpu->cpu, vcpu->vcpu_id, gfn, is_mmio, cache);

It is visible from dmesg files that in case of large guest ram, the function
doesn't even get called for vcpus other than 0. On the other hand it is called
for all in case of small memory. 
The traces are NOT from the same run as the dmesg files, as they have been
created before your post about tracing.

Please ask if any more info, traces or dumps are needed. I would be glad to
provide any help.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=107561

--- Comment #9 from Paolo Bonzini <bonzini@gnu.org> ---
Can you try the following on HostA:

1) providing *host* dmesg

2) disabling each MTRR starting from the first, until things are fast again. 
Then reboot and try disabling only the MTRR that made things fast again.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=107561

--- Comment #10 from Paolo Bonzini <bonzini@gnu.org> ---
Another test on the 3GB machine: make it fast by nuking
kvm_mtrr_get_guest_memory_type (return MTRR_TYPE_WRBACK), then cat /proc/mtrr
from the guest.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=107561

--- Comment #11 from Paolo Bonzini <bonzini@gnu.org> ---
Created attachment 197071
  --> https://bugzilla.kernel.org/attachment.cgi?id=197071&action=edit
OVMF log

> 1) providing host dmesg

Of course this makes no sense, you already did.  And from the traces I suspect
that /proc/mtrr is the same on the two guests.  The culprit seems to be this
MTRR:

 qemu-system-x86-2228  [002]   127.763684: kvm_msr: msr_read 201 = 0xff80000800
 qemu-system-x86-2228  [002]   127.763685: kvm_msr: msr_read 200 = 0x80000000

which is set up by OVMF.  I'm attaching the OVMF output for reference (it can
be extracted from the traces with "trace-cmd report trace-3GBmem.dat |grep
0x402 | awk --non-decimal-data '{printf "%c", 0+$NF}' > OVMF.log").

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=107561

--- Comment #12 from Paolo Bonzini <bonzini@gnu.org> ---
Created attachment 197081
  --> https://bugzilla.kernel.org/attachment.cgi?id=197081&action=edit
patch to be tested

The bug happens when the VM is created with a maxphyaddr that doesn't match the
host.  This patch should fix your bug.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=107561

--- Comment #13 from schefister@gmail.com ---
Sorry for the delay. I couldn't check sooner.
Unfortunately the guest doesn't start with the patch applied. Nothing displayed
(no sync signal from passthrough card). Host dmesg doesn't show any errors.
Usually, even with slow guest I get the following on host dmesg when starting
VM:

...
VFIO - User Level meta-driver version: 0.3
kernel: vgaarb: device changed decodes:
PCI:0000:01:00.0,olddecodes=io+mem,decodes=io+mem:owns=io+mem
kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e@0x258
kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
kernel: pmd_set_huge: Cannot satisfy [mem 0xe0000000-0xe0200000] with a
huge-page mapping due to MTRR override.
- than some USB resets for keyboard passthrough -
kernel: kvm: zapping shadow pages for mmio generation wraparound
...and so on, the guest screen displays UEFI startup by now

With the patch applied it simly stops at the pmd_set_huge line, and nothing
further, no usb, no guest start, no error messages.

I also did follow up on the other tests.
Disabling host mtrrs one-by-one did not yield any results. Guest is still slow
even if disabling all the hosts mtrrs.
Tried nukeing kvm_mtrr_get_guest_memory_type as requested and started a live
distro. Guest did boot, but did not have a /proc/mtrr file. (Graphics scrolling
was slow, I guess on account of lack of cacheing, but otherwise guest was
speedy as normal) I saved some info from the guest, like dmesg, lspci -vvv, and
some sysfs files. You can find them in the same share as before.
https://drive.google.com/folderview?id=0B8ebX_WjVHnGNlN4eTEzU2xtMEk&usp=sharing

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=107561

--- Comment #14 from Jasen Borisov <tajjada@gmail.com> ---
Can confirm. I am also hit by this bug, and my virtual machine doesn't start
either.

I tried applying the patch to both 4.3.0 and linux-next, and neither of those
worked.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 107561] 4.2 breaks PCI passthrough in QEMU/KVM

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=107561

Paolo Bonzini <bonzini@gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #197081|0                           |1
        is obsolete|                            |
             Status|NEW                         |NEEDINFO
           Assignee|virtualization_kvm@kernel-b |bonzini@gnu.org
                   |ugs.osdl.org                |

--- Comment #15 from Paolo Bonzini <bonzini@gnu.org> ---
Created attachment 197381
  --> https://bugzilla.kernel.org/attachment.cgi?id=197381&action=edit
patch to be tested v2

There is indeed a two-character mistake in the patch.  Can you guys please test
this one instead?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.