From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andrew Theurer Subject: windows workload: many ept_violation and mmio exits Date: Thu, 03 Dec 2009 07:46:19 -0600 Message-ID: <4B17C12B.4020300@linux.vnet.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit To: kvm@vger.kernel.org Return-path: Received: from e7.ny.us.ibm.com ([32.97.182.137]:59008 "EHLO e7.ny.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751226AbZLCNqT (ORCPT ); Thu, 3 Dec 2009 08:46:19 -0500 Received: from d01relay01.pok.ibm.com (d01relay01.pok.ibm.com [9.56.227.233]) by e7.ny.us.ibm.com (8.14.3/8.13.1) with ESMTP id nB3DfirE004460 for ; Thu, 3 Dec 2009 08:41:44 -0500 Received: from d01av01.pok.ibm.com (d01av01.pok.ibm.com [9.56.224.215]) by d01relay01.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id nB3DkPTf106570 for ; Thu, 3 Dec 2009 08:46:25 -0500 Received: from d01av01.pok.ibm.com (loopback [127.0.0.1]) by d01av01.pok.ibm.com (8.14.3/8.13.1/NCO v10.0 AVout) with ESMTP id nB3DkP9w004743 for ; Thu, 3 Dec 2009 08:46:25 -0500 Received: from [127.0.0.1] (sig-9-65-5-221.mts.ibm.com [9.65.5.221]) by d01av01.pok.ibm.com (8.14.3/8.13.1/NCO v10.0 AVin) with ESMTP id nB3DkNin004353 for ; Thu, 3 Dec 2009 08:46:24 -0500 Sender: kvm-owner@vger.kernel.org List-ID: I am running a windows workload which has 26 windows VMs running many instances of a J2EE workload. There are 13 pairs of an application server VM and database server VM. There seem to be quite a bit of vm_exits, and it looks over a third of them are mmio_exit: > efer_relo 0 > exits 337139 > fpu_reloa 247321 > halt_exit 19092 > halt_wake 18611 > host_stat 247332 > hypercall 0 > insn_emul 184265 > insn_emul 184265 > invlpg 0 > io_exits 69184 > irq_exits 52953 > irq_injec 48115 > irq_windo 2411 > largepage 19 > mmio_exit 123554 > mmu_cache 0 > mmu_flood 0 > mmu_pde_z 0 > mmu_pte_u 0 > mmu_pte_w 0 > mmu_recyc 0 > mmu_shado 0 > mmu_unsyn 0 > nmi_injec 0 > nmi_windo 0 > pf_fixed 19 > pf_guest 0 > remote_tl 0 > request_i 0 > signal_ex 0 > tlb_flush 0 I collected a kvmtrace, and below is a very small portion of that. Is there a way I can figure out what device the mmio's are for? Also, is it normal to have lots of ept_violations? This is a 2 socket Nehalem system with SMT on. > qemu-system-x86-19673 [014] 213577.939614: kvm_entry: vcpu 0 > qemu-system-x86-19673 [014] 213577.939624: kvm_exit: reason ept_violation rip 0xfffff8000160ef8e > qemu-system-x86-19673 [014] 213577.939624: kvm_page_fault: address fed000f0 error_code 181 > qemu-system-x86-19673 [014] 213577.939627: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 > qemu-system-x86-19673 [014] 213577.939629: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0xfb8f214d > qemu-system-x86-19673 [014] 213577.939631: kvm_entry: vcpu 0 > qemu-system-x86-19673 [014] 213577.939633: kvm_exit: reason ept_violation rip 0xfffff8000160ef8e > qemu-system-x86-19673 [014] 213577.939634: kvm_page_fault: address fed000f0 error_code 181 > qemu-system-x86-19673 [014] 213577.939636: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 > qemu-system-x86-19332 [008] 213577.939637: kvm_entry: vcpu 0 > qemu-system-x86-19673 [014] 213577.939638: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0xfb8f24e2 > qemu-system-x86-19673 [014] 213577.939640: kvm_entry: vcpu 0 > qemu-system-x86-19211 [010] 213577.939663: kvm_set_irq: gsi 11 level 1 source 0 > qemu-system-x86-19211 [010] 213577.939664: kvm_pic_set_irq: chip 1 pin 3 (level|masked) > qemu-system-x86-19211 [010] 213577.939665: kvm_apic_accept_irq: apicid 0 vec 130 (LowPrio|level) > qemu-system-x86-19211 [010] 213577.939666: kvm_ioapic_set_irq: pin 11 dst 1 vec=130 (LowPrio|logical|level) > qemu-system-x86-19673 [014] 213577.939692: kvm_exit: reason ept_violation rip 0xfffff8000160ef8e > qemu-system-x86-19673 [014] 213577.939693: kvm_page_fault: address fed000f0 error_code 181 > qemu-system-x86-19673 [014] 213577.939696: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 > qemu-system-x86-19332 [008] 213577.939699: kvm_exit: reason ept_violation rip 0xfffff80001b3af8e > qemu-system-x86-19332 [008] 213577.939700: kvm_page_fault: address fed000f0 error_code 181 > qemu-system-x86-19673 [014] 213577.939702: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0xfb8f3da6 > qemu-system-x86-19563 [010] 213577.939702: kvm_set_irq: gsi 11 level 1 source 0 > qemu-system-x86-19563 [010] 213577.939703: kvm_pic_set_irq: chip 1 pin 3 (level|masked) > qemu-system-x86-19673 [014] 213577.939704: kvm_entry: vcpu 0 > qemu-system-x86-19563 [010] 213577.939705: kvm_apic_accept_irq: apicid 0 vec 130 (LowPrio|level) > qemu-system-x86-19332 [008] 213577.939706: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 > qemu-system-x86-19563 [010] 213577.939707: kvm_ioapic_set_irq: pin 11 dst 1 vec=130 (LowPrio|logical|level) > qemu-system-x86-19332 [008] 213577.939713: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0x29a105de > qemu-system-x86-19332 [008] 213577.939715: kvm_entry: vcpu 0 > qemu-system-x86-19201 [011] 213577.939716: kvm_exit: reason exception rip 0x1162412 > qemu-system-x86-19332 [008] 213577.939717: kvm_exit: reason halt rip 0xfffffa6000fae7a1 > qemu-system-x86-19201 [011] 213577.939717: kvm_entry: vcpu 0 > qemu-system-x86-19673 [014] 213577.939761: kvm_exit: reason ept_violation rip 0xfffff8000160ef8e > qemu-system-x86-19673 [014] 213577.939762: kvm_page_fault: address fed000f0 error_code 181 > qemu-system-x86-19673 [014] 213577.939766: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 > qemu-system-x86-19673 [014] 213577.939772: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0xfb8f58dd > qemu-system-x86-19673 [014] 213577.939774: kvm_entry: vcpu 0 > qemu-system-x86-19673 [014] 213577.939776: kvm_exit: reason ept_violation rip 0xfffff8000160ef8e > qemu-system-x86-19673 [014] 213577.939776: kvm_page_fault: address fed000f0 error_code 181 > qemu-system-x86-19673 [014] 213577.939779: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 > qemu-system-x86-19673 [014] 213577.939782: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0xfb8f5d09 > qemu-system-x86-19673 [014] 213577.939784: kvm_entry: vcpu 0 > qemu-system-x86-19673 [014] 213577.939791: kvm_exit: reason ept_violation rip 0xfffff8000160ef8e > qemu-system-x86-19673 [014] 213577.939791: kvm_page_fault: address fed000f0 error_code 181 > qemu-system-x86-19673 [014] 213577.939794: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 > qemu-system-x86-19673 [014] 213577.939798: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0xfb8f62fb > qemu-system-x86-19673 [014] 213577.939799: kvm_entry: vcpu 0 > qemu-system-x86-19673 [014] 213577.939802: kvm_exit: reason ept_violation rip 0xfffff8000160ef8e > qemu-system-x86-19673 [014] 213577.939802: kvm_page_fault: address fed000f0 error_code 181 > qemu-system-x86-19673 [014] 213577.939805: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 > qemu-system-x86-19673 [014] 213577.939808: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0xfb8f66f4 > qemu-system-x86-19673 [014] 213577.939809: kvm_entry: vcpu 0 > qemu-system-x86-19673 [014] 213577.939836: kvm_exit: reason ept_violation rip 0xfffff8000160ef8e > qemu-system-x86-19673 [014] 213577.939837: kvm_page_fault: address fed000f0 error_code 181 > qemu-system-x86-19673 [014] 213577.939845: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 > qemu-system-x86-19661 [014] 213577.939875: kvm_set_irq: gsi 11 level 1 source 0 > qemu-system-x86-19661 [014] 213577.939876: kvm_pic_set_irq: chip 1 pin 3 (level|masked) > qemu-system-x86-19661 [014] 213577.939876: kvm_apic_accept_irq: apicid 0 vec 130 (LowPrio|level) > qemu-system-x86-19661 [014] 213577.939877: kvm_ioapic_set_irq: pin 11 dst 1 vec=130 (LowPrio|logical|level) > qemu-system-x86-19320 [008] 213577.939895: kvm_set_irq: gsi 11 level 1 source 0 > qemu-system-x86-19320 [008] 213577.939896: kvm_pic_set_irq: chip 1 pin 3 (level|masked) > qemu-system-x86-19320 [008] 213577.939897: kvm_apic_accept_irq: apicid 0 vec 130 (LowPrio|level) > qemu-system-x86-19673 [014] 213577.939898: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0xfb8f89f2 > qemu-system-x86-19320 [008] 213577.939899: kvm_ioapic_set_irq: pin 11 dst 1 vec=130 (LowPrio|logical|level) > qemu-system-x86-19673 [014] 213577.939900: kvm_inj_virq: irq 130 > qemu-system-x86-19673 [014] 213577.939901: kvm_entry: vcpu 0 > qemu-system-x86-19673 [014] 213577.939904: kvm_exit: reason io_instruction rip 0xfffffa6001acb435 > qemu-system-x86-19673 [014] 213577.939904: kvm_pio: pio_read at 0xc033 size 1 count 1 > qemu-system-x86-19673 [014] 213577.939907: kvm_set_irq: gsi 11 level 0 source 0 > qemu-system-x86-19673 [014] 213577.939907: kvm_pic_set_irq: chip 1 pin 3 (level|masked) > qemu-system-x86-19673 [014] 213577.939908: kvm_ioapic_set_irq: pin 11 dst 1 vec=130 (LowPrio|logical|level) > qemu-system-x86-19673 [014] 213577.939910: kvm_entry: vcpu 0 > qemu-system-x86-19673 [014] 213577.939912: kvm_exit: reason apic_access rip 0xfffff800016a050c > qemu-system-x86-19673 [014] 213577.939914: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 > qemu-system-x86-19673 [014] 213577.939914: kvm_apic: apic_write APIC_EOI = 0x0 > qemu-system-x86-19673 [014] 213577.939914: kvm_ack_irq: irqchip IOAPIC pin 11 > qemu-system-x86-19673 [014] 213577.939915: kvm_entry: vcpu 0 > qemu-system-x86-19673 [014] 213577.939918: kvm_exit: reason ext_irq rip 0xfffff800016a12f0 > qemu-system-x86-19661 [014] 213577.939934: kvm_set_irq: gsi 11 level 1 source 0 > qemu-system-x86-19661 [014] 213577.939935: kvm_pic_set_irq: chip 1 pin 3 (level|masked) > qemu-system-x86-19661 [014] 213577.939936: kvm_apic_accept_irq: apicid 0 vec 130 (LowPrio|level) > qemu-system-x86-19661 [014] 213577.939936: kvm_ioapic_set_irq: pin 11 dst 1 vec=130 (LowPrio|logical|level) > qemu-system-x86-19332 [008] 213577.939940: kvm_inj_virq: irq 130 > qemu-system-x86-19332 [008] 213577.939941: kvm_entry: vcpu 0 > qemu-system-x86-19332 [008] 213577.939944: kvm_exit: reason io_instruction rip 0xfffffa6000f9d435 > qemu-system-x86-19332 [008] 213577.939944: kvm_pio: pio_read at 0xc033 size 1 count 1 > qemu-system-x86-19332 [008] 213577.939948: kvm_set_irq: gsi 11 level 0 source 0 > qemu-system-x86-19332 [008] 213577.939949: kvm_pic_set_irq: chip 1 pin 3 (level|masked) > qemu-system-x86-19332 [008] 213577.939949: kvm_ioapic_set_irq: pin 11 dst 1 vec=130 (LowPrio|logical|level) > qemu-system-x86-19673 [014] 213577.939950: kvm_inj_virq: irq 130 > qemu-system-x86-19673 [014] 213577.939951: kvm_entry: vcpu 0 > qemu-system-x86-19332 [008] 213577.939953: kvm_entry: vcpu 0 > qemu-system-x86-19673 [014] 213577.939953: kvm_exit: reason io_instruction rip 0xfffffa6001acb435 > qemu-system-x86-19673 [014] 213577.939954: kvm_pio: pio_read at 0xc033 size 1 count 1 > qemu-system-x86-19332 [008] 213577.939955: kvm_exit: reason apic_access rip 0xfffff8000166e50c > qemu-system-x86-19673 [014] 213577.939957: kvm_set_irq: gsi 11 level 0 source 0 > qemu-system-x86-19332 [008] 213577.939958: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 > qemu-system-x86-19673 [014] 213577.939958: kvm_pic_set_irq: chip 1 pin 3 (level|masked) > qemu-system-x86-19332 [008] 213577.939958: kvm_apic: apic_write APIC_EOI = 0x0 > qemu-system-x86-19673 [014] 213577.939958: kvm_ioapic_set_irq: pin 11 dst 1 vec=130 (LowPrio|logical|level) > qemu-system-x86-19332 [008] 213577.939958: kvm_ack_irq: irqchip IOAPIC pin 11 > qemu-system-x86-19332 [008] 213577.939959: kvm_entry: vcpu 0 > qemu-system-x86-19332 [008] 213577.939961: kvm_exit: reason ept_violation rip 0xfffff80001b3af8e > qemu-system-x86-19332 [008] 213577.939961: kvm_page_fault: address fed000f0 error_code 181 > qemu-system-x86-19673 [014] 213577.939962: kvm_entry: vcpu 0 > qemu-system-x86-19673 [014] 213577.939964: kvm_exit: reason apic_access rip 0xfffff800016a050c > qemu-system-x86-19332 [008] 213577.939965: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 > qemu-system-x86-19673 [014] 213577.939966: kvm_mmio: mmio write len 4 gpa 0xfee000b0 val 0x0 > qemu-system-x86-19673 [014] 213577.939967: kvm_apic: apic_write APIC_EOI = 0x0 > qemu-system-x86-19673 [014] 213577.939967: kvm_ack_irq: irqchip IOAPIC pin 11 > qemu-system-x86-19673 [014] 213577.939968: kvm_entry: vcpu 0 > qemu-system-x86-19332 [008] 213577.939969: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0x29a16a40 > qemu-system-x86-19332 [008] 213577.939971: kvm_entry: vcpu 0 > qemu-system-x86-19332 [008] 213577.939981: kvm_exit: reason ept_violation rip 0xfffff80001b3af8e > qemu-system-x86-19332 [008] 213577.939982: kvm_page_fault: address fed000f0 error_code 181 > qemu-system-x86-19673 [014] 213577.939982: kvm_exit: reason ept_violation rip 0xfffff8000160ef8e > qemu-system-x86-19673 [014] 213577.939983: kvm_page_fault: address fed000f0 error_code 181 > qemu-system-x86-19332 [008] 213577.939985: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 > qemu-system-x86-19673 [014] 213577.939987: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 > qemu-system-x86-19332 [008] 213577.939989: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0x29a1722f > qemu-system-x86-19673 [014] 213577.939991: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0xfb8fae83 > qemu-system-x86-19332 [008] 213577.939991: kvm_entry: vcpu 0 > qemu-system-x86-19673 [014] 213577.939993: kvm_entry: vcpu 0 > qemu-system-x86-19673 [014] 213577.940010: kvm_exit: reason cr_access rip 0xfffff800016ee2b2 > qemu-system-x86-19673 [014] 213577.940011: kvm_cr: cr_write 4 = 0x678 > qemu-system-x86-19673 [014] 213577.940017: kvm_entry: vcpu 0 > qemu-system-x86-19673 [014] 213577.940019: kvm_exit: reason cr_access rip 0xfffff800016ee2b5 > qemu-system-x86-19673 [014] 213577.940019: kvm_cr: cr_write 4 = 0x6f8 > qemu-system-x86-19673 [014] 213577.940021: kvm_entry: vcpu 0 > qemu-system-x86-19673 [014] 213577.940048: kvm_exit: reason exception rip 0xfffff800016a0620 > qemu-system-x86-19673 [014] 213577.940049: kvm_entry: vcpu 0 > qemu-system-x86-19673 [014] 213577.940079: kvm_exit: reason ept_violation rip 0xfffff8000160ef8e > qemu-system-x86-19673 [014] 213577.940080: kvm_page_fault: address fed000f0 error_code 181 > qemu-system-x86-19673 [014] 213577.940083: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 Here is oprofile: > 4117817 62.2029 kvm-intel.ko kvm-intel.ko vmx_vcpu_run > 338198 5.1087 qemu-system-x86_64 qemu-system-x86_64 /usr/local/qemu/48bb360cc687b89b74dfb1cac0f6e8812b64841c/bin/qemu-system-x86_64 > 62449 0.9433 kvm.ko kvm.ko kvm_arch_vcpu_ioctl_run > 56512 0.8537 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 copy_user_generic_string > 52373 0.7911 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 native_write_msr_safe > 34847 0.5264 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 schedule > 34678 0.5238 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 fget_light > 29894 0.4516 kvm.ko kvm.ko paging64_walk_addr > 27778 0.4196 kvm.ko kvm.ko gfn_to_hva > 24563 0.3710 kvm.ko kvm.ko x86_decode_insn > 23900 0.3610 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 do_select > 21123 0.3191 libc-2.10.90.so libc-2.10.90.so memcpy > 20694 0.3126 kvm.ko kvm.ko x86_emulate_insn > 19862 0.3000 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 kfree > 19107 0.2886 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 __switch_to > 18319 0.2767 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 update_curr > 17981 0.2716 libc-2.10.90.so libc-2.10.90.so ioctl > 17934 0.2709 librt-2.10.90.so librt-2.10.90.so clock_gettime > 17874 0.2700 ioatdma.ko ioatdma.ko ioat2_issue_pending > 17578 0.2655 libpthread-2.10.90.so libpthread-2.10.90.so pthread_mutex_lock > 17041 0.2574 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 task_rq_lock > 15806 0.2388 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 native_read_msr_safe > 15292 0.2310 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 fput > 14197 0.2145 libc-2.10.90.so libc-2.10.90.so memset > 14167 0.2140 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 __up_read > 13974 0.2111 kvm.ko kvm.ko kvm_arch_vcpu_put > 13885 0.2097 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 select_task_rq_fair > 13766 0.2079 bnx2.ko bnx2.ko bnx2_poll_work > 13349 0.2016 kvm.ko kvm.ko find_highest_vector > 13121 0.1982 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 __down_read > 12518 0.1891 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 do_vfs_ioctl > 12184 0.1840 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 vmlinux-2.6.32-rc7-5e8cb552cb8b48244b6d07bff984b3c4080d4bc9-autokern1 try_to_wake_up > 12095 0.1827 kvm.ko kvm.ko gfn_to_memslot > 11870 0.1793 kvm.ko kvm.ko kvm_read_guest > 11657 0.1761 tun.ko tun.ko tun_chr_aio_read -Andrew