From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christian Ehrhardt Date: Thu, 13 Mar 2008 17:53:23 +0000 Subject: Re: [kvm-ppc-devel] Guests oops when trying to mount initramfs Message-Id: <47D96A13.2090509@linux.vnet.ibm.com> MIME-Version: 1 Content-Type: multipart/mixed; boundary="------------040709060302020705090704" List-Id: References: <47D7C7FF.2060306@linux.vnet.ibm.com> In-Reply-To: <47D7C7FF.2060306@linux.vnet.ibm.com> To: kvm-ppc@vger.kernel.org This is a multi-part message in MIME format. --------------040709060302020705090704 Content-Type: text/plain; charset="iso-8859-1"; format="flowed" Content-Transfer-Encoding: quoted-printable Christian Ehrhardt wrote: [...] > And if it reproducible in your case we can use it to find the reason for = it, here a short summary from my past discussions about that issue: > - effectively what you see is that the guest has an itlb miss for it's ke= rnel mapping. > - we only deliver that to the guest if we can't find that mapping in the = guest tlb > - this should never happen because the guest should never remove it's own= kernel mapping and therefor we should be able to cover all itlb misses in = the host by using the guest tlb > - but it happens so the question is when&why is the guest kernel mapping = removed from the guest tlb > - afaik we have only one place that might remove that mapping which is th= e function kvmppc_emul_tlbwe > - we also know the kernel mapping which is usually a single 16mb mapping = we initialize for the guest in kvm_arch_vcpu_setup (I might be wrong here -= Hollis?) > =3D> we could track that tlbwe that should never happen by inserting a wa= rn_on into kvmppc_emul_tlbwe if it overwrites a guest tlb kernel mapping an= d starting from there debug whats going on > ok for Hollis to continue and anyone else that might have comments -> tracking any eviction of the guest kernel mapping (=3D0xC..) did not tri= gger -> tracking the delivery of a itlb/dtlb miss with that address did not trig= ger -> I inserted a BUG statement in the guests report of "Unable to handle ker= nel paging request ..." That triggers a emulation of a trap in the host and there I have a dump_= vcpu&dump_tlb -> for the case that someone wants to reproduce that I attached debug patch= es for host and guest kernel=20 Here is the output when hitting the trap: Looking up port of RPC 100005/1 on 192.168.1.2 VFS: Mounted root (nfs filesystem). Freeing unused kernel memory: 116k init Unable to handle kernel paging request for instruction fetch Faulting instruction address: 0xc0000ac0 trap! pc: c000fcbc msr: 00021002 lr: c000fcbc ctr: c012dc54 srr0: c012d638 srr1: 00021002 exceptions: 00000000 gpr00: c000fcbc bff2aaf0 c8810400 0000002c gpr04: 00000001 00000001 00000000 00000004 gpr08: 00000001 c025ca00 00001250 c0260000 gpr12: 65930e5e 1001f2f8 00000000 00000000 gpr16: 00000000 00000000 00000000 c01f0000 gpr20: c0240000 c0240000 00000000 00000000 gpr24: c0240000 c01f0000 00000001 48026e60 gpr28: 48027a10 bff2af04 00000001 bff2ab10 vcpu 0 TLB dump: | nr | tid | word0 | word1 | word2 | G 1 | 00000001 | 0FF3F210 | 08FAB000 | 08FA016D | G 2 | 00000001 | 0FEAD210 | 08FDC000 | 08FD0149 | G 3 | 00000001 | 48026210 | 08FFB000 | 08FF0349 | G 5 | 00000001 | 10003210 | 0022D000 | 0022016D | G 6 | 00000000 | D1012210 | EF600000 | EF600703 | G 7 | 00000001 | 4800C210 | 003D7000 | 003D016D | G 8 | 00000001 | 10000210 | 00227000 | 00220149 | G 9 | 00000001 | 0FFE8210 | 08FE6000 | 08FE0349 | G11 | 00000001 | 0FEC5210 | 08FD4000 | 08FD016D | G13 | 00000001 | 0FFEA210 | 08FCD000 | 08FC036F | G14 | 00000001 | 0FFEC210 | 08FE5000 | 08FE035B | G15 | 00000001 | 0FFEF210 | 08FDE000 | 08FD035B | G16 | 00000000 | D1021210 | E8001000 | E8000703 | G18 | 00000001 | 0FF70210 | 003EB000 | 003E016D | G20 | 00000001 | 0FF20210 | 003F8000 | 003F016D | G22 | 00000001 | 0FF1F210 | 003F9000 | 003F016D | G23 | 00000001 | 0FFE7210 | 08FCB000 | 08FC0349 | G25 | 00000001 | 0FF17210 | 08F85000 | 08F8016D | G26 | 00000001 | 48013210 | 00235000 | 0023016D | G28 | 00000001 | 10001210 | 00228000 | 0022016D | G31 | 00000001 | 10017210 | 00242000 | 0024037F | G32 | 00000001 | 48012210 | 00234000 | 0023016D | G33 | 00000001 | 0FEA9210 | 003DC000 | 003D0149 | G34 | 00000001 | 0FEB3210 | 08FEB000 | 08FE0149 | G35 | 00000001 | 0FEB9210 | 08FEC000 | 08FE0149 | G36 | 00000001 | 0FEBA210 | 08FC2000 | 08FC0149 | G37 | 00000001 | 48009210 | 08FFA000 | 08FF016D | G39 | 00000001 | 0FEDD210 | 08F8A000 | 08F8016D | G40 | 00000001 | 0FFED210 | 08FCA000 | 08FC035B | G42 | 00000001 | 10005210 | 0022F000 | 0022016D | G43 | 00000001 | 10006210 | 00230000 | 0023016D | G45 | 00000001 | 0FEDA210 | 08F8D000 | 08F8016D | G46 | 00000001 | 4801C210 | 08FC9000 | 08FC035B | G47 | 00000001 | 0FEB4210 | 08FC1000 | 08FC0149 | G48 | 00000001 | 0FEAB210 | 08FD7000 | 08FD0149 | G49 | 00000001 | 0FEB0210 | 08FD9000 | 08FD0149 | G50 | 00000001 | 0FEB7210 | 08FEE000 | 08FE0149 | G51 | 00000001 | 0FEAF210 | 08FDA000 | 08FD0149 | G52 | 00000001 | 0FEB6210 | 08FEF000 | 08FE0149 | G53 | 00000001 | 0FEAA210 | 003DB000 | 003D0149 | G55 | 00000001 | 0FF61210 | 08F97000 | 08F9016D | G56 | 00000001 | 0FEB2210 | 08FE7000 | 08FE0149 | G57 | 00000001 | 0FEB8210 | 08FED000 | 08FE0149 | G58 | 00000001 | 0FEAC210 | 08FDD000 | 08FD0149 | G59 | 00000001 | 0FEB5210 | 08FC0000 | 08FC0149 | G60 | 00000001 | 48008210 | 08FFC000 | 08FF016D | G61 | 00000001 | 48027210 | 00241000 | 0024037F | G62 | 00000001 | BFF2A210 | 00243000 | 0024035B | G63 | 00000000 | C0000290 | 00000000 | 00000107 | S17 | 00000001 | 10003310 | 0E3B8000 | 0000002F | S18 | 00000001 | BFF2A310 | 0E3CE000 | 0000001F | S19 | 00000001 | 10017310 | 0E3CD000 | 0000003F | S20 | 00000000 | C0000310 | 0E38B000 | 0000003F | S21 | 00000000 | C0246310 | 0E3D1000 | 0000003F | S22 | 00000000 | C000C310 | 0E3E7000 | 0000003F | S23 | 00000000 | C8810310 | 0E9BA000 | 0000003F | S24 | 00000000 | C000D310 | 0E3E8000 | 0000003F | S25 | 00000000 | C000F310 | 0E3EA000 | 0000003F | S26 | 00000000 | C0264310 | 0DC31000 | 0000003F | S27 | 00000000 | C0037310 | 0DCF2000 | 0000003F | S28 | 00000000 | C0106310 | 0E9D1000 | 0000003F | S29 | 00000000 | C0223310 | 0E3AE000 | 0000003F | S30 | 00000000 | C0047310 | 0DD02000 | 0000003F | S31 | 00000000 | C024A310 | 0E3D5000 | 0000003F | S32 | 00000000 | C0024310 | 0DC1F000 | 0000003F | S33 | 00000000 | C0248310 | 0E3D3000 | 0000003F | S34 | 00000000 | C010D310 | 0E9D8000 | 0000003F | S35 | 00000000 | C010C310 | 0E9D7000 | 0000003F | S36 | 00000000 | C01F6310 | 0E201000 | 0000003F | S37 | 00000000 | C026B310 | 0DC38000 | 0000003F | S38 | 00000000 | C026C310 | 0DC39000 | 0000003F | S39 | 00000000 | C0023310 | 0DC1E000 | 0000003F | S40 | 00000000 | C0257310 | 0DC22000 | 0000003F | S41 | 00000000 | C012D310 | 0E0B8000 | 0000003F | S42 | 00000000 | C004A310 | 0DD05000 | 0000003F | S43 | 00000000 | C001E310 | 0DC19000 | 0000003F | S44 | 00000000 | C001D310 | 0DC18000 | 0000003F | S45 | 00000000 | C000A310 | 0E3E5000 | 0000003F | S46 | 00000000 | C0262310 | 0DC2D000 | 0000003F | S47 | 00000000 | C027C310 | 0DC69000 | 0000003F | S48 | 00000000 | C01D2310 | 0E13D000 | 0000003F | S49 | 00000000 | C8835310 | 0E06B000 | 0000003F | S50 | 00000000 | C0001310 | 0E38C000 | 0000003F | S51 | 00000000 | C025C310 | 0DC27000 | 0000003F | S52 | 00000000 | C027D310 | 0DC6A000 | 0000003F | S53 | 00000000 | C012A310 | 0E0B5000 | 0000003F | S54 | 00000000 | C0130310 | 0E0BB000 | 0000003F | S55 | 00000000 | C01D0310 | 0E13B000 | 0000003F | S56 | 00000000 | C0256310 | 0DC21000 | 0000003F | Oops: Exception in kernel mode, sig: 4 [#1] Bamboo Modules linked in: NIP: c000fcbc LR: c000fcbc CTR: c012dc54 REGS: bff2aa40 TRAP: 0700 Not tainted (2.6.25-rc3) MSR: 00021002 CR: 24002022 XER: 00000000 TASK =3D c8810400[1] 'init' THREAD: c881e000 GPR00: c000fcbc bff2aaf0 c8810400 0000002c 00000001 00000001 00000000 00000= 004 GPR08: 00000001 c025ca00 00001250 c0260000 65930e5e 1001f2f8 00000000 00000= 000 GPR16: 00000000 00000000 00000000 c01f0000 c0240000 c0240000 00000000 00000= 000 GPR24: c0240000 c01f0000 00000001 48026e60 48027a10 bff2af04 00000001 bff2a= b10 NIP [c000fcbc] bad_page_fault+0x7c/0xb4 LR [c000fcbc] bad_page_fault+0x7c/0xb4 Call Trace: Instruction dump: 2b800380 419d0034 2f800300 40be003c 3c60c01f 809f00a4 38636580 48015271 809f0080 3c60c01f 3863664c 48015261 <0fe00000> 48000000 2f800400 419e001c ---[ end trace 9c05eabdb79d9d2c ]--- Kernel panic - not syncing: Attempted to kill init! Rebooting in 180 seconds.. --=20 Gr=FCsse / regards,=20 Christian Ehrhardt IBM Linux Technology Center, Open Virtualization --------------040709060302020705090704 Content-Type: text/x-patch; name="debug-guest-kernel-mapping-guestpatch.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="debug-guest-kernel-mapping-guestpatch.diff" diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -432,5 +432,8 @@ void bad_page_fault(struct pt_regs *regs printk(KERN_ALERT "Faulting instruction address: 0x%08lx\n", regs->nip); + // to trigger kvmppc host trap! + BUG(); + die("Kernel access of bad area", regs, sig); } --------------040709060302020705090704 Content-Type: text/x-patch; name="debug-guest-kernel-mapping-hostpatch.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="debug-guest-kernel-mapping-hostpatch.diff" diff --git a/arch/powerpc/kvm/44x_tlb.c b/arch/powerpc/kvm/44x_tlb.c --- a/arch/powerpc/kvm/44x_tlb.c +++ b/arch/powerpc/kvm/44x_tlb.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include "44x_tlb.h" @@ -29,6 +30,36 @@ #define PPC44x_TLB_SUPER_PERM_MASK (PPC44x_TLB_SX|PPC44x_TLB_SR|PPC44x_TLB_SW) static unsigned int kvmppc_tlb_44x_pos; + +void kvmppc_dump_tlb(struct kvm_vcpu *vcpu) +{ + struct tlbe *tlbe; + int i; + + printk("vcpu %d TLB dump:\n", vcpu->vcpu_id); + printk("| %2s | %8s | %8s | %8s | %8s |\n", + "nr", " tid ", "word0", "word1", "word2"); + + for (i = 0; i < PPC44x_TLB_SIZE; i++) + { + tlbe = &vcpu->arch.guest_tlb[i]; + if (tlbe->word0 & PPC44x_TLB_VALID) + printk("G%2d | %08X | %08X | %08X | %08X |\n", + i, tlbe->tid, tlbe->word0, tlbe->word1, tlbe->word2); + } + + msleep(500); + + for (i = 0; i < PPC44x_TLB_SIZE; i++) + { + tlbe = &vcpu->arch.shadow_tlb[i]; + if (tlbe->word0 & PPC44x_TLB_VALID) + printk("S%2d | %08X | %08X | %08X | %08X |\n", + i, tlbe->tid, tlbe->word0, tlbe->word1, tlbe->word2); + } + + msleep(500); +} static u32 kvmppc_44x_tlb_shadow_attrib(u32 attrib, int usermode) { diff --git a/arch/powerpc/kvm/44x_tlb.h b/arch/powerpc/kvm/44x_tlb.h --- a/arch/powerpc/kvm/44x_tlb.h +++ b/arch/powerpc/kvm/44x_tlb.h @@ -27,6 +27,8 @@ extern int kvmppc_44x_tlb_index(struct k unsigned int pid, unsigned int as); extern struct tlbe *kvmppc_44x_dtlb_search(struct kvm_vcpu *vcpu, gva_t eaddr); extern struct tlbe *kvmppc_44x_itlb_search(struct kvm_vcpu *vcpu, gva_t eaddr); + +extern void kvmppc_dump_tlb(struct kvm_vcpu *vcpu); /* TLB helper functions */ static inline unsigned int get_tlb_size(const struct tlbe *tlbe) diff --git a/arch/powerpc/kvm/emulate.c b/arch/powerpc/kvm/emulate.c --- a/arch/powerpc/kvm/emulate.c +++ b/arch/powerpc/kvm/emulate.c @@ -129,6 +129,7 @@ static int kvmppc_emul_tlbwe(struct kvm_ if (index > PPC44x_TLB_SIZE) { printk("%s: index %d\n", __func__, index); kvmppc_dump_vcpu(vcpu); + kvmppc_dump_tlb(vcpu); return EMULATE_FAIL; } @@ -138,6 +139,14 @@ static int kvmppc_emul_tlbwe(struct kvm_ #endif tlbe = &vcpu->arch.guest_tlb[index]; + + if ((get_tlb_eaddr(tlbe) >> 30) == 0xc) { + printk("evicting %02d: %08x %08x %08x %08x\n", index, + tlbe->tid, tlbe->word0, tlbe->word1, tlbe->word2); + kvmppc_dump_vcpu(vcpu); + WARN_ON(1); + } + /* Invalidate shadow mappings for the about-to-be-clobbered TLBE. */ if (tlbe->word0 & PPC44x_TLB_VALID) { @@ -250,6 +259,8 @@ int kvmppc_emulate_instruction(struct kv switch (get_op(inst)) { case 3: /* trap */ printk("trap!\n"); + kvmppc_dump_vcpu(vcpu); + kvmppc_dump_tlb(vcpu); kvmppc_queue_exception(vcpu, BOOKE_INTERRUPT_PROGRAM); advance = 0; break; diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c --- a/arch/powerpc/kvm/powerpc.c +++ b/arch/powerpc/kvm/powerpc.c @@ -218,6 +218,7 @@ int kvmppc_handle_exit(struct kvm_run *r case BOOKE_INTERRUPT_MACHINE_CHECK: printk("MACHINE CHECK: %lx\n", mfspr(SPRN_MCSR)); kvmppc_dump_vcpu(vcpu); + kvmppc_dump_tlb(vcpu); r = RESUME_HOST; break; @@ -310,6 +311,12 @@ int kvmppc_handle_exit(struct kvm_run *r gtlbe = kvmppc_44x_dtlb_search(vcpu, eaddr); if (!gtlbe) { /* The guest didn't have a mapping for it. */ + if ((eaddr >> 30) == 0xc) { + printk("Guest mapping f0r 0xc not found!\n"); + kvmppc_dump_vcpu(vcpu); + kvmppc_dump_tlb(vcpu); + } + kvmppc_queue_exception(vcpu, exit_nr); vcpu->arch.dear = vcpu->arch.fault_dear; vcpu->arch.esr = vcpu->arch.fault_esr; @@ -354,6 +361,12 @@ int kvmppc_handle_exit(struct kvm_run *r gtlbe = kvmppc_44x_itlb_search(vcpu, eaddr); if (!gtlbe) { /* The guest didn't have a mapping for it. */ + if ((eaddr >> 30) == 0xc) { + printk("Guest mapping f0r 0xc not found!\n"); + kvmppc_dump_vcpu(vcpu); + kvmppc_dump_tlb(vcpu); + } + kvmppc_queue_exception(vcpu, exit_nr); r = RESUME_GUEST; break; --------------040709060302020705090704 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ --------------040709060302020705090704 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ kvm-ppc-devel mailing list kvm-ppc-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel --------------040709060302020705090704--