From: Paolo Bonzini <pbonzini@redhat.com>
To: Kashyap Chamarthy <kashyap.cv@gmail.com>
Cc: Gleb Natapov <gleb@redhat.com>,
"kvm@vger.kernel.org" <kvm@vger.kernel.org>
Subject: Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot
Date: Tue, 08 Oct 2013 15:03:20 +0200 [thread overview]
Message-ID: <52540298.2030807@redhat.com> (raw)
In-Reply-To: <CAOaxAcYXHxUPRO9V16HAPDJic_Nvvxe4YC7nZj5FTfzq9dXfVA@mail.gmail.com>
Il 08/10/2013 07:38, Kashyap Chamarthy ha scritto:
> On Mon, Oct 7, 2013 at 6:29 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
>> Gleb, so I just did a trace of KVM MMU to try to understand why L2 is
>> stuck with shadow on EPT
>
> Paolo, were you able to reproduce this again? Yesterday, on #qemu you
> mentioned you'll test it again :-)
Yes, I could reproduce it too.
>> Boot L2 guest:
Here L2 doesn't go past the second instruction. It gets a page fault
even though the spte is present, and KVM then loops on a page fault
for 0xfe05b.
Here is an annotated function_graph trace of L1.
It's possible that L0 is injecting the same fault repeatedly, i.e.
they are not different faults from the processor. I'll get an L0
trace next.
---- KVM executes at 0xfffffff0 via emulation
kvm_cpu_has_pending_timer [kvm]() {
apic_has_pending_timer [kvm]();
}
kvm_check_async_pf_completion [kvm]();
vmx_save_host_state [kvm_intel]();
__srcu_read_unlock();
rcu_note_context_switch();
/* kvm_entry: vcpu 0 */
vmx_vcpu_run [kvm_intel]();
vmx_read_l1_tsc [kvm_intel]();
vmx_handle_external_intr [kvm_intel]() {
}
__srcu_read_lock();
vmx_handle_exit [kvm_intel]() {
guest_state_valid.part.27 [kvm_intel]() {
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
}
vmx_interrupt_allowed [kvm_intel]();
x86_emulate_instruction [kvm]() {
init_emulate_ctxt [kvm]() {
vmx_get_cs_db_l_bits [kvm_intel]() {
vmx_read_guest_seg_ar [kvm_intel]();
}
vmx_get_rflags [kvm_intel]();
}
x86_decode_insn [kvm]() {
do_insn_fetch [kvm]() {
...
kvm_fetch_guest_virt [kvm]() {
vmx_get_cpl [kvm_intel]();
kvm_read_guest_virt_helper [kvm]() {
nonpaging_gva_to_gpa [kvm]();
kvm_read_guest [kvm]() {
kvm_read_guest_page [kvm]() {
gfn_to_hva_prot [kvm]() {
__gfn_to_hva_many [kvm]();
}
}
}
} /* kvm_read_guest_virt_helper [kvm] */
}
}
...
}
x86_emulate_insn [kvm]() {
/* kvm_emulate_insn: ffff0000:fff0:ea 5b e0 00 f0 (real) */
em_jmp_far [kvm]() {
load_segment_descriptor [kvm]() {
emulator_get_segment [kvm]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
}
emulator_set_segment [kvm]() {
vmx_set_segment [kvm_intel]() {
fix_rmode_seg [kvm_intel]() {
vmcs_writel [kvm_intel]();
vmcs_writel [kvm_intel]();
vmcs_writel [kvm_intel]();
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
vmcs_writel [kvm_intel]();
}
emulation_required [kvm_intel]() {
guest_state_valid.part.27 [kvm_intel]() {
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
} /* rmode_segment_valid [kvm_intel] */
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
}
}
}
}
}
}
writeback [kvm]();
writeback_registers [kvm]();
}
vmx_get_interrupt_shadow [kvm_intel]();
vmx_set_interrupt_shadow [kvm_intel]();
vmx_get_rflags [kvm_intel]();
kvm_set_rflags [kvm]() {
vmx_set_rflags [kvm_intel]() {
vmcs_writel [kvm_intel]();
}
}
}
guest_state_valid.part.27 [kvm_intel]() {
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
}
emulation_required [kvm_intel]() {
guest_state_valid.part.27 [kvm_intel]() {
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
rmode_segment_valid [kvm_intel]() {
vmx_get_segment [kvm_intel]() {
vmx_read_guest_seg_selector [kvm_intel]();
}
vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
}
}
}
}
---- KVM executes at f000:e05b and gets a page fault
kvm_cpu_has_pending_timer [kvm]() {
apic_has_pending_timer [kvm]();
}
kvm_check_async_pf_completion [kvm]();
kvm_apic_accept_events [kvm]();
kvm_cpu_has_injectable_intr [kvm]() {
kvm_apic_accept_pic_intr [kvm]();
}
vmx_interrupt_allowed [kvm_intel]();
kvm_cpu_has_injectable_intr [kvm]() {
kvm_apic_accept_pic_intr [kvm]();
}
enable_irq_window [kvm_intel]() {
vmcs_writel [kvm_intel]();
}
vmx_save_host_state [kvm_intel]();
__srcu_read_unlock();
rcu_note_context_switch();
/* kvm_entry: vcpu 0 */
vmx_vcpu_run [kvm_intel]() {
vmcs_writel [kvm_intel]();
vmcs_writel [kvm_intel]();
perf_guest_get_msrs();
/* kvm_exit: reason EXCEPTION_NMI rip 0xe05b info fe05b 80000b0e */
}
vmx_read_l1_tsc [kvm_intel]();
vmx_handle_external_intr [kvm_intel]();
__srcu_read_lock();
vmx_handle_exit [kvm_intel]() {
handle_exception [kvm_intel]() {
/* kvm_page_fault: address fe05b error_code 14 */
kvm_mmu_page_fault [kvm]() {
nonpaging_page_fault [kvm]() {
mmu_topup_memory_caches [kvm]();
gfn_to_memslot_dirty_bitmap.isra.67 [kvm]() {
gfn_to_memslot [kvm]();
}
mapping_level.isra.86 [kvm]() {
kvm_host_page_size [kvm]() {
gfn_to_hva [kvm]() {
__gfn_to_hva_many [kvm]();
}
down_read() {
_cond_resched();
}
find_vma();
vma_kernel_pagesize();
up_read();
}
}
try_async_pf [kvm]() {
gfn_to_pfn_async [kvm]() {
__gfn_to_pfn [kvm]() {
__gfn_to_pfn_memslot [kvm]() {
__gfn_to_hva_many [kvm]();
__get_user_pages_fast() {
gup_pud_range() {
gup_pte_range();
}
}
}
}
}
}
handle_abnormal_pfn [kvm]();
_raw_spin_lock();
make_mmu_pages_available.isra.80 [kvm]();
transparent_hugepage_adjust.isra.91 [kvm]() {
kvm_is_mmio_pfn [kvm]();
}
__direct_map.isra.104 [kvm]() {
shadow_walk_init [kvm]();
kvm_mmu_get_page [kvm]() {
pte_list_add [kvm]();
/* kvm_mmu_get_page: sp gen 2 gfn 0 1 q0 direct wux !nxe root 0 sync new */
}
link_shadow_page.isra.63 [kvm]() {
mmu_spte_set [kvm]();
}
mmu_set_spte [kvm]() {
set_spte [kvm]() {
mark_page_dirty [kvm]();
mmu_spte_update [kvm]() {
mmu_spte_set [kvm]();
}
}
kvm_mmu_page_get_gfn [kvm]();
gfn_to_rmap [kvm]() {
gfn_to_memslot [kvm]();
}
pte_list_add [kvm]();
kvm_release_pfn_clean [kvm]() {
kvm_is_mmio_pfn [kvm]();
put_page();
}
}
__direct_pte_prefetch [kvm]() {
kvm_mmu_page_get_gfn [kvm]();
gfn_to_memslot_dirty_bitmap.isra.67 [kvm]() {
gfn_to_memslot [kvm]();
}
gfn_to_page_many_atomic [kvm]() {
__gfn_to_hva_many [kvm]();
__get_user_pages_fast() {
gup_pud_range() {
gup_pte_range();
}
}
}
mmu_set_spte [kvm]() {
set_spte [kvm]() {
mark_page_dirty [kvm]();
mmu_spte_update [kvm]() {
mmu_spte_set [kvm]();
}
}
kvm_mmu_page_get_gfn [kvm]();
gfn_to_rmap [kvm]() {
gfn_to_memslot [kvm]();
}
pte_list_add [kvm]();
kvm_release_pfn_clean [kvm]() {
kvm_is_mmio_pfn [kvm]();
put_page();
}
}
mmu_set_spte [kvm]() {
set_spte [kvm]() {
mark_page_dirty [kvm]();
mmu_spte_update [kvm]() {
mmu_spte_set [kvm]();
}
}
kvm_mmu_page_get_gfn [kvm]();
gfn_to_rmap [kvm]() {
gfn_to_memslot [kvm]();
}
pte_list_add [kvm]();
kvm_release_pfn_clean [kvm]() {
kvm_is_mmio_pfn [kvm]();
put_page();
}
}
mmu_set_spte [kvm]() {
set_spte [kvm]() {
mark_page_dirty [kvm]();
mmu_spte_update [kvm]() {
mmu_spte_set [kvm]();
}
}
kvm_mmu_page_get_gfn [kvm]();
gfn_to_rmap [kvm]() {
gfn_to_memslot [kvm]();
}
pte_list_add [kvm]();
kvm_release_pfn_clean [kvm]() {
kvm_is_mmio_pfn [kvm]();
put_page();
}
}
mmu_set_spte [kvm]() {
set_spte [kvm]() {
mark_page_dirty [kvm]();
mmu_spte_update [kvm]() {
mmu_spte_set [kvm]();
}
}
kvm_mmu_page_get_gfn [kvm]();
gfn_to_rmap [kvm]() {
gfn_to_memslot [kvm]();
}
pte_list_add [kvm]();
kvm_release_pfn_clean [kvm]() {
kvm_is_mmio_pfn [kvm]();
put_page();
}
}
mmu_set_spte [kvm]() {
set_spte [kvm]() {
mark_page_dirty [kvm]();
mmu_spte_update [kvm]() {
mmu_spte_set [kvm]();
}
}
kvm_mmu_page_get_gfn [kvm]();
gfn_to_rmap [kvm]() {
gfn_to_memslot [kvm]();
}
pte_list_add [kvm]();
kvm_release_pfn_clean [kvm]() {
kvm_is_mmio_pfn [kvm]();
put_page();
}
}
mmu_set_spte [kvm]() {
set_spte [kvm]() {
mark_page_dirty [kvm]();
mmu_spte_update [kvm]() {
mmu_spte_set [kvm]();
}
}
kvm_mmu_page_get_gfn [kvm]();
gfn_to_rmap [kvm]() {
gfn_to_memslot [kvm]();
}
pte_list_add [kvm]();
kvm_release_pfn_clean [kvm]() {
kvm_is_mmio_pfn [kvm]();
put_page();
}
}
}
}
}
}
}
}
---- up to this point the trace is identical for shadow-on-shadow (working)
---- and shadow-on-EPT (not working)
----
---- for shadow-on-EPT L1 gets another page fault even though the spte is
---- present. The error code is identical.
kvm_cpu_has_pending_timer [kvm]() {
apic_has_pending_timer [kvm]();
}
kvm_check_async_pf_completion [kvm]();
vmx_save_host_state [kvm_intel]();
__srcu_read_unlock();
rcu_note_context_switch();
/* kvm_entry: vcpu 0 */
vmx_vcpu_run [kvm_intel]() {
vmcs_writel [kvm_intel]();
vmcs_writel [kvm_intel]();
perf_guest_get_msrs();
/* kvm_exit: reason EXCEPTION_NMI rip 0xe05b info fe05b 80000b0e */
}
vmx_read_l1_tsc [kvm_intel]();
vmx_handle_external_intr [kvm_intel]();
__srcu_read_lock();
vmx_handle_exit [kvm_intel]() {
handle_exception [kvm_intel]() {
/* kvm_page_fault: address fe05b error_code 14 */
kvm_mmu_page_fault [kvm]() {
nonpaging_page_fault [kvm]() {
mmu_topup_memory_caches [kvm]();
gfn_to_memslot_dirty_bitmap.isra.67 [kvm]() {
gfn_to_memslot [kvm]();
}
mapping_level.isra.86 [kvm]() {
kvm_host_page_size [kvm]() {
gfn_to_hva [kvm]() {
__gfn_to_hva_many [kvm]();
}
down_read() {
_cond_resched();
}
find_vma();
vma_kernel_pagesize();
up_read();
}
}
try_async_pf [kvm]() {
gfn_to_pfn_async [kvm]() {
__gfn_to_pfn [kvm]() {
__gfn_to_pfn_memslot [kvm]() {
__gfn_to_hva_many [kvm]();
__get_user_pages_fast() {
gup_pud_range() {
gup_pte_range();
}
}
}
}
}
}
handle_abnormal_pfn [kvm]();
_raw_spin_lock();
make_mmu_pages_available.isra.80 [kvm]();
transparent_hugepage_adjust.isra.91 [kvm]() {
kvm_is_mmio_pfn [kvm]();
}
__direct_map.isra.104 [kvm]() {
shadow_walk_init [kvm]();
>>>>>> here starts the difference with the second part,
>>>>>> this shows that the spte is present
mmu_set_spte [kvm]() {
set_spte [kvm]() {
mark_page_dirty [kvm]();
mmu_spte_update [kvm]() {
spte_has_volatile_bits [kvm]();
}
}
kvm_release_pfn_clean [kvm]() {
kvm_is_mmio_pfn [kvm]();
put_page();
}
}
__direct_pte_prefetch [kvm]();
}
}
}
}
}
---- this part of the trace then loops endlessly
Paolo
next prev parent reply other threads:[~2013-10-08 13:03 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-10-04 8:31 [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot Kashyap Chamarthy
2013-10-04 9:33 ` Kashyap Chamarthy
2013-10-04 9:39 ` Gleb Natapov
2013-10-04 12:33 ` Kashyap Chamarthy
2013-10-04 13:05 ` Gleb Natapov
2013-10-04 13:08 ` Gleb Natapov
2013-10-04 15:28 ` Kashyap Chamarthy
2013-10-07 12:59 ` Kashyap Chamarthy
2013-10-07 15:18 ` Gleb Natapov
2013-10-08 5:38 ` Kashyap Chamarthy
2013-10-08 13:03 ` Paolo Bonzini [this message]
2013-10-08 14:48 ` Paolo Bonzini
2013-10-09 6:22 ` Kashyap Chamarthy
2013-10-09 8:16 ` Gleb Natapov
2013-10-09 17:46 ` Kashyap Chamarthy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=52540298.2030807@redhat.com \
--to=pbonzini@redhat.com \
--cc=gleb@redhat.com \
--cc=kashyap.cv@gmail.com \
--cc=kvm@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.