[Bug 103321] New: NPT page attribute support causes extreme slowdown

[Bug 103321] New: NPT page attribute support causes extreme slowdown

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=103321

            Bug ID: 103321
           Summary: NPT page attribute support causes extreme slowdown
           Product: Virtualization
           Version: unspecified
    Kernel Version: 4.2-rc3+
          Hardware: x86-64
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: kvm
          Assignee: virtualization_kvm@kernel-bugs.osdl.org
          Reporter: dracon@ewetel.net
        Regression: No

Created attachment 185501
  --> https://bugzilla.kernel.org/attachment.cgi?id=185501&action=edit
dmesg

For some reason the new NPT attribute support that has been introduced with
4.3-RC3 causes my VM to run awfully slow, I managed to track this issue to
commit 3c2e7f7de3 ("KVM: SVM: use NPT page attributes").

The VM runs Win7 64bit via Qemu and uses PCI passthrough with several devices
(primary GPU, USB, audio) on a AMD 970/SB950 based Asrock 970 EXTREME3 2.0
motherboard and an AMD Phenom II X6 CPU.

There are no error/warning messages whatsoever, booting the VM takes several
minutes just to get past the first boot screen.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 103321] NPT page attribute support causes extreme slowdown

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=103321

--- Comment #1 from Sebastian Schütte <dracon@ewetel.net> ---
Created attachment 185511
  --> https://bugzilla.kernel.org/attachment.cgi?id=185511&action=edit
cpuinfo

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 103321] NPT page attribute support causes extreme slowdown

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=103321

--- Comment #2 from Sebastian Schütte <dracon@ewetel.net> ---
Created attachment 185521
  --> https://bugzilla.kernel.org/attachment.cgi?id=185521&action=edit
lspci

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

Re: [Bug 103321] New: NPT page attribute support causes extreme slowdown

[Paolo Bonzini]

On 23/08/2015 11:53, Sebastian Schütte wrote:
>> What version of SeaBIOS?
> 1.8.2 stable, from the Arch repo
>> Can you try cherry-picking commit 5492830370171b6a4ede8a3bfba687a8d0f25fa5?
> Tried that, but it doesn't seem to make a difference.
>> If that fails, can you trace the execution of the VM until the beginning
>> of the Linux kernel boot (http://www.linux-kvm.org/page/Tracing) and
>> send it compressed to me by email?
> I have attached a trace up to the point when the Windows bootloader
> kicks in. I could still set up a Linux VM if you think its trace would
> be more helpful, though.

No, it seems to be in the BIOS.  Please try this:

diff --git a/arch/x86/kvm/svm.c b/arch/x86/kvm/svm.c
index 74d825716f4f..f8f69cffecbf 100644
--- a/arch/x86/kvm/svm.c
+++ b/arch/x86/kvm/svm.c
@@ -882,7 +882,8 @@ static u8 fallback_mtrr_type(int mtrr)
 	case MTRR_TYPE_WRTHROUGH:
 		return MTRR_TYPE_UNCACHABLE;
 	case MTRR_TYPE_WRPROT:
-		return MTRR_TYPE_UC_MINUS;
+		/* Assume it's backed by RAM in the host.  */
+		return MTRR_TYPE_WRBACK;
 	default:
 		BUG();
 	}
@@ -3054,6 +3055,7 @@ static int cr_interception(struct vcpu_svm *svm)
 	if (cr >= 16) { /* mov to cr */
 		cr -= 16;
 		val = kvm_register_read(&svm->vcpu, reg);
+		trace_kvm_cr_write(cr, val);
 		switch (cr) {
 		case 0:
 			if (!check_selective_cr0_intercepted(svm, val))


This is a bit of a hack, but it can be enough for a test.  For a
real patch one would make mtrr2protval bidimensional, accessing it
like "return mtrr2protval[is_mmio][mtrr]" in svm_get_mt_mask.
Then you can return MTRR_TYPE_UC_MINUS if is_mmio, and
MTRR_TYPE_WRBACK if !is_mmio.

Paolo

Re: [Bug 103321] New: NPT page attribute support causes extreme slowdown

[Sebastian Schütte]

> Please try this:
Still no difference

I guess the trace_kvm_cr_write() call in that patch was supposed to
trigger kvm_cr entries while tracing? I couldn't find any, though, the
only entries containing "cr" within the output of "trace-cmd report"
were kvm_exit ones that looked quite similar to the previous dump.

If you still think it's worth it I'll send you the whole ~10MB
compressed trace.

Re: [Bug 103321] New: NPT page attribute support causes extreme slowdown

[Sebastian Schütte]

I inserted some printk() lines into init_vmcb() around the call of svm_set_guest_pat() to print out the g_pat value as well as svm->vcpu.vcpu_id and noticed that something was off:

Initially, the PATs of all VCPUs are set to 0x0606060606060606. However, after attaching some devices (vfio-pci enabling device and vfio_ecap_init lines are being printed) init_vmcb() is only called again for vcpu_id > 0. Unless g_pat is changed somewhere else, VCPU #0 remains set to 0x0606060606060606 (according to comments in svm_set_guest_pat() this is bad for assigned devices) while all other VCPUs use 0x0007040600070406.

I'd guess that could explain the slowdown.

Re: [Bug 103321] New: NPT page attribute support causes extreme slowdown

[Paolo Bonzini]

On 02/09/2015 21:01, Sebastian Schütte wrote:
> I inserted some printk() lines into init_vmcb() around the call of
> svm_set_guest_pat() to print out the g_pat value as well as
> svm->vcpu.vcpu_id and noticed that something was off:
> 
> Initially, the PATs of all VCPUs are set to 0x0606060606060606.
> However, after attaching some devices (vfio-pci enabling device and
> vfio_ecap_init lines are being printed) init_vmcb() is only called
> again for vcpu_id > 0. Unless g_pat is changed somewhere else, VCPU
> #0 remains set to 0x0606060606060606 (according to comments in
> svm_set_guest_pat() this is bad for assigned devices) while all other
> VCPUs use 0x0007040600070406.
> 
> I'd guess that could explain the slowdown.
> 

Hi, sorry for the delay.  If you change KVM like this

diff --git a/arch/x86/kvm/svm.c b/arch/x86/kvm/svm.c
index 27f57fc05bc7..3ce878c5fde8 100644
--- a/arch/x86/kvm/svm.c
+++ b/arch/x86/kvm/svm.c
@@ -1155,10 +1155,7 @@ static void svm_set_guest_pat(struct vcpu_svm *svm, u64 *g_pat)
 	 * have assigned devices, however, we cannot force WB for RAM
 	 * pages only, so use the guest PAT directly.
 	 */
-	if (!kvm_arch_has_assigned_device(vcpu->kvm))
-		*g_pat = 0x0606060606060606;
-	else
-		*g_pat = vcpu->arch.pat;
+	*g_pat = vcpu->arch.pat;
 }
 
 static u64 svm_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)

(obviously not a valid patch for upstream patch, but good enough for
testing) do you get normal speed?

Paolo

Re: [Bug 103321] New: NPT page attribute support causes extreme slowdown

[Sebastian Schütte]

> do you get normal speed?
Nope, still slow... I somehow missed that svm_set_msr() calls svm_set_guest_pat() as well, it does seem to properly change g_pat. The only thing that catches my eyes is that later on all WT entries in g_pat are replaced by WC (0x0007040600070406 -> 0x0007010600070106), but that doesn't seem to make a difference.

If it wasn't for some frustrating and badly timed libvirt update breaking my old VM config some days ago I would have noticed that earlier :/

[Bug 103321] NPT page attribute support causes extreme slowdown

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=103321

Jimi <JimiJames.Bove@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |JimiJames.Bove@gmail.com

--- Comment #3 from Jimi <JimiJames.Bove@gmail.com> ---
Before discovering this bug page, I started a thread on the Arch forums before
coming across a bug that's either the same one as this or something very
similar. We've done some extra investigation over there. Here's a link:
https://bbs.archlinux.org/viewtopic.php?pid=1574465

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 103321] NPT page attribute support causes extreme slowdown

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=103321

--- Comment #4 from Jimi <JimiJames.Bove@gmail.com> ---
(In reply to Jimi from comment #3)
> Before discovering this bug page, I started a thread on the Arch forums
> before coming across a bug that's either the same one as this or something
> very similar. We've done some extra investigation over there. Here's a link:
> https://bbs.archlinux.org/viewtopic.php?pid=1574465

Er, that is, by "very similar" I mean the behavior. Obviously, they could have
wildly different causes.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 103321] NPT page attribute support causes extreme slowdown

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=103321

Sebastian Schütte <dracon@ewetel.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |CODE_FIX

--- Comment #5 from Sebastian Schütte <dracon@ewetel.net> ---
As far as I am concerned, RC4 fixes the issue (should've marked it as fixed
earlier) by reverting said commit. This change has also been backported to 4.2
stable about 2 weeks ago, AFAIK.

Note that this bug is actually AMD specific and only affects CPUs with NPT
support.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 103321] NPT page attribute support causes extreme slowdown

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=103321

--- Comment #6 from Jimi <JimiJames.Bove@gmail.com> ---
Oh. Definitely not the same bug, then. Ours still isn't fixed and affects more
than just AMD.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.