Re: [kvm-ppc-devel] Guests oops when trying to mount initramfs

[Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>]

[-- Attachment #1: Type: text/plain, Size: 8053 bytes --]

Christian Ehrhardt wrote:
[...]
> And if it reproducible in your case we can use it to find the reason for it, here a short summary from my past discussions about that issue:
> - effectively what you see is that the guest has an itlb miss for it's kernel mapping.
> - we only deliver that to the guest if we can't find that mapping in the guest tlb
> - this should never happen because the guest should never remove it's own kernel mapping and therefor we should be able to cover all itlb misses in the host by using the guest tlb
> - but it happens so the question is when&why is the guest kernel mapping removed from the guest tlb
> - afaik we have only one place that might remove that mapping which is the function kvmppc_emul_tlbwe
> - we also know the kernel mapping which is usually a single 16mb mapping we initialize for the guest in kvm_arch_vcpu_setup (I might be wrong here - Hollis?)
> => we could track that tlbwe that should never happen by inserting a warn_on into kvmppc_emul_tlbwe if it overwrites a guest tlb kernel mapping and starting from there debug whats going on
>

ok for Hollis to continue and anyone else that might have comments
-> tracking any eviction of the guest kernel mapping (=0xC..) did not trigger
-> tracking the delivery of a itlb/dtlb miss with that address did not trigger
-> I inserted a BUG statement in the guests report of "Unable to handle kernel paging request ..."
   That triggers a emulation of a trap in the host and there I have a dump_vcpu&dump_tlb
-> for the case that someone wants to reproduce that I attached debug patches for host and guest kernel 

   Here is the output when hitting the trap:

Looking up port of RPC 100005/1 on 192.168.1.2
VFS: Mounted root (nfs filesystem).
Freeing unused kernel memory: 116k init
Unable to handle kernel paging request for instruction fetch
Faulting instruction address: 0xc0000ac0
trap!
pc:   c000fcbc msr:  00021002
lr:   c000fcbc ctr:  c012dc54
srr0: c012d638 srr1: 00021002
exceptions: 00000000
gpr00: c000fcbc bff2aaf0 c8810400 0000002c
gpr04: 00000001 00000001 00000000 00000004
gpr08: 00000001 c025ca00 00001250 c0260000
gpr12: 65930e5e 1001f2f8 00000000 00000000
gpr16: 00000000 00000000 00000000 c01f0000
gpr20: c0240000 c0240000 00000000 00000000
gpr24: c0240000 c01f0000 00000001 48026e60
gpr28: 48027a10 bff2af04 00000001 bff2ab10
vcpu 0 TLB dump:
| nr |     tid  |    word0 |    word1 |    word2 |
G 1 | 00000001 | 0FF3F210 | 08FAB000 | 08FA016D |
G 2 | 00000001 | 0FEAD210 | 08FDC000 | 08FD0149 |
G 3 | 00000001 | 48026210 | 08FFB000 | 08FF0349 |
G 5 | 00000001 | 10003210 | 0022D000 | 0022016D |
G 6 | 00000000 | D1012210 | EF600000 | EF600703 |
G 7 | 00000001 | 4800C210 | 003D7000 | 003D016D |
G 8 | 00000001 | 10000210 | 00227000 | 00220149 |
G 9 | 00000001 | 0FFE8210 | 08FE6000 | 08FE0349 |
G11 | 00000001 | 0FEC5210 | 08FD4000 | 08FD016D |
G13 | 00000001 | 0FFEA210 | 08FCD000 | 08FC036F |
G14 | 00000001 | 0FFEC210 | 08FE5000 | 08FE035B |
G15 | 00000001 | 0FFEF210 | 08FDE000 | 08FD035B |
G16 | 00000000 | D1021210 | E8001000 | E8000703 |
G18 | 00000001 | 0FF70210 | 003EB000 | 003E016D |
G20 | 00000001 | 0FF20210 | 003F8000 | 003F016D |
G22 | 00000001 | 0FF1F210 | 003F9000 | 003F016D |
G23 | 00000001 | 0FFE7210 | 08FCB000 | 08FC0349 |
G25 | 00000001 | 0FF17210 | 08F85000 | 08F8016D |
G26 | 00000001 | 48013210 | 00235000 | 0023016D |
G28 | 00000001 | 10001210 | 00228000 | 0022016D |
G31 | 00000001 | 10017210 | 00242000 | 0024037F |
G32 | 00000001 | 48012210 | 00234000 | 0023016D |
G33 | 00000001 | 0FEA9210 | 003DC000 | 003D0149 |
G34 | 00000001 | 0FEB3210 | 08FEB000 | 08FE0149 |
G35 | 00000001 | 0FEB9210 | 08FEC000 | 08FE0149 |
G36 | 00000001 | 0FEBA210 | 08FC2000 | 08FC0149 |
G37 | 00000001 | 48009210 | 08FFA000 | 08FF016D |
G39 | 00000001 | 0FEDD210 | 08F8A000 | 08F8016D |
G40 | 00000001 | 0FFED210 | 08FCA000 | 08FC035B |
G42 | 00000001 | 10005210 | 0022F000 | 0022016D |
G43 | 00000001 | 10006210 | 00230000 | 0023016D |
G45 | 00000001 | 0FEDA210 | 08F8D000 | 08F8016D |
G46 | 00000001 | 4801C210 | 08FC9000 | 08FC035B |
G47 | 00000001 | 0FEB4210 | 08FC1000 | 08FC0149 |
G48 | 00000001 | 0FEAB210 | 08FD7000 | 08FD0149 |
G49 | 00000001 | 0FEB0210 | 08FD9000 | 08FD0149 |
G50 | 00000001 | 0FEB7210 | 08FEE000 | 08FE0149 |
G51 | 00000001 | 0FEAF210 | 08FDA000 | 08FD0149 |
G52 | 00000001 | 0FEB6210 | 08FEF000 | 08FE0149 |
G53 | 00000001 | 0FEAA210 | 003DB000 | 003D0149 |
G55 | 00000001 | 0FF61210 | 08F97000 | 08F9016D |
G56 | 00000001 | 0FEB2210 | 08FE7000 | 08FE0149 |
G57 | 00000001 | 0FEB8210 | 08FED000 | 08FE0149 |
G58 | 00000001 | 0FEAC210 | 08FDD000 | 08FD0149 |
G59 | 00000001 | 0FEB5210 | 08FC0000 | 08FC0149 |
G60 | 00000001 | 48008210 | 08FFC000 | 08FF016D |
G61 | 00000001 | 48027210 | 00241000 | 0024037F |
G62 | 00000001 | BFF2A210 | 00243000 | 0024035B |
G63 | 00000000 | C0000290 | 00000000 | 00000107 |
S17 | 00000001 | 10003310 | 0E3B8000 | 0000002F |
S18 | 00000001 | BFF2A310 | 0E3CE000 | 0000001F |
S19 | 00000001 | 10017310 | 0E3CD000 | 0000003F |
S20 | 00000000 | C0000310 | 0E38B000 | 0000003F |
S21 | 00000000 | C0246310 | 0E3D1000 | 0000003F |
S22 | 00000000 | C000C310 | 0E3E7000 | 0000003F |
S23 | 00000000 | C8810310 | 0E9BA000 | 0000003F |
S24 | 00000000 | C000D310 | 0E3E8000 | 0000003F |
S25 | 00000000 | C000F310 | 0E3EA000 | 0000003F |
S26 | 00000000 | C0264310 | 0DC31000 | 0000003F |
S27 | 00000000 | C0037310 | 0DCF2000 | 0000003F |
S28 | 00000000 | C0106310 | 0E9D1000 | 0000003F |
S29 | 00000000 | C0223310 | 0E3AE000 | 0000003F |
S30 | 00000000 | C0047310 | 0DD02000 | 0000003F |
S31 | 00000000 | C024A310 | 0E3D5000 | 0000003F |
S32 | 00000000 | C0024310 | 0DC1F000 | 0000003F |
S33 | 00000000 | C0248310 | 0E3D3000 | 0000003F |
S34 | 00000000 | C010D310 | 0E9D8000 | 0000003F |
S35 | 00000000 | C010C310 | 0E9D7000 | 0000003F |
S36 | 00000000 | C01F6310 | 0E201000 | 0000003F |
S37 | 00000000 | C026B310 | 0DC38000 | 0000003F |
S38 | 00000000 | C026C310 | 0DC39000 | 0000003F |
S39 | 00000000 | C0023310 | 0DC1E000 | 0000003F |
S40 | 00000000 | C0257310 | 0DC22000 | 0000003F |
S41 | 00000000 | C012D310 | 0E0B8000 | 0000003F |
S42 | 00000000 | C004A310 | 0DD05000 | 0000003F |
S43 | 00000000 | C001E310 | 0DC19000 | 0000003F |
S44 | 00000000 | C001D310 | 0DC18000 | 0000003F |
S45 | 00000000 | C000A310 | 0E3E5000 | 0000003F |
S46 | 00000000 | C0262310 | 0DC2D000 | 0000003F |
S47 | 00000000 | C027C310 | 0DC69000 | 0000003F |
S48 | 00000000 | C01D2310 | 0E13D000 | 0000003F |
S49 | 00000000 | C8835310 | 0E06B000 | 0000003F |
S50 | 00000000 | C0001310 | 0E38C000 | 0000003F |
S51 | 00000000 | C025C310 | 0DC27000 | 0000003F |
S52 | 00000000 | C027D310 | 0DC6A000 | 0000003F |
S53 | 00000000 | C012A310 | 0E0B5000 | 0000003F |
S54 | 00000000 | C0130310 | 0E0BB000 | 0000003F |
S55 | 00000000 | C01D0310 | 0E13B000 | 0000003F |
S56 | 00000000 | C0256310 | 0DC21000 | 0000003F |
Oops: Exception in kernel mode, sig: 4 [#1]
Bamboo
Modules linked in:
NIP: c000fcbc LR: c000fcbc CTR: c012dc54
REGS: bff2aa40 TRAP: 0700   Not tainted  (2.6.25-rc3)
MSR: 00021002 <ME>  CR: 24002022  XER: 00000000
TASK = c8810400[1] 'init' THREAD: c881e000
GPR00: c000fcbc bff2aaf0 c8810400 0000002c 00000001 00000001 00000000 00000004
GPR08: 00000001 c025ca00 00001250 c0260000 65930e5e 1001f2f8 00000000 00000000
GPR16: 00000000 00000000 00000000 c01f0000 c0240000 c0240000 00000000 00000000
GPR24: c0240000 c01f0000 00000001 48026e60 48027a10 bff2af04 00000001 bff2ab10
NIP [c000fcbc] bad_page_fault+0x7c/0xb4
LR [c000fcbc] bad_page_fault+0x7c/0xb4
Call Trace:
Instruction dump:
2b800380 419d0034 2f800300 40be003c 3c60c01f 809f00a4 38636580 48015271
809f0080 3c60c01f 3863664c 48015261 <0fe00000> 48000000 2f800400 419e001c
---[ end trace 9c05eabdb79d9d2c ]---
Kernel panic - not syncing: Attempted to kill init!
Rebooting in 180 seconds..


-- 

Grüsse / regards, 
Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization

[-- Attachment #2: debug-guest-kernel-mapping-guestpatch.diff --]
[-- Type: text/x-patch, Size: 359 bytes --]

diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
--- a/arch/powerpc/mm/fault.c
+++ b/arch/powerpc/mm/fault.c
@@ -432,5 +432,8 @@ void bad_page_fault(struct pt_regs *regs
 	printk(KERN_ALERT "Faulting instruction address: 0x%08lx\n",
 		regs->nip);
 
+	// to trigger kvmppc host trap!
+	BUG();
+
 	die("Kernel access of bad area", regs, sig);
 }

[-- Attachment #3: debug-guest-kernel-mapping-hostpatch.diff --]
[-- Type: text/x-patch, Size: 4162 bytes --]

diff --git a/arch/powerpc/kvm/44x_tlb.c b/arch/powerpc/kvm/44x_tlb.c
--- a/arch/powerpc/kvm/44x_tlb.c
+++ b/arch/powerpc/kvm/44x_tlb.c
@@ -21,6 +21,7 @@
 #include <linux/string.h>
 #include <linux/kvm_host.h>
 #include <linux/highmem.h>
+#include <linux/delay.h>
 #include <asm/mmu-44x.h>
 
 #include "44x_tlb.h"
@@ -29,6 +30,36 @@
 #define PPC44x_TLB_SUPER_PERM_MASK (PPC44x_TLB_SX|PPC44x_TLB_SR|PPC44x_TLB_SW)
 
 static unsigned int kvmppc_tlb_44x_pos;
+
+void kvmppc_dump_tlb(struct kvm_vcpu *vcpu)
+{
+	struct tlbe *tlbe;
+	int i;
+
+	printk("vcpu %d TLB dump:\n", vcpu->vcpu_id);
+	printk("| %2s | %8s | %8s | %8s | %8s |\n",
+		"nr", " tid ", "word0", "word1", "word2");
+
+	for (i = 0; i < PPC44x_TLB_SIZE; i++)
+	{
+		tlbe = &vcpu->arch.guest_tlb[i];
+		if (tlbe->word0 & PPC44x_TLB_VALID)
+			printk("G%2d | %08X | %08X | %08X | %08X |\n",
+			i, tlbe->tid, tlbe->word0, tlbe->word1,	tlbe->word2);
+	}
+
+	msleep(500);
+
+	for (i = 0; i < PPC44x_TLB_SIZE; i++)
+	{
+		tlbe = &vcpu->arch.shadow_tlb[i];
+		if (tlbe->word0 & PPC44x_TLB_VALID)
+			printk("S%2d | %08X | %08X | %08X | %08X |\n",
+			i, tlbe->tid, tlbe->word0, tlbe->word1,	tlbe->word2);
+	}
+
+	msleep(500);
+}
 
 static u32 kvmppc_44x_tlb_shadow_attrib(u32 attrib, int usermode)
 {
diff --git a/arch/powerpc/kvm/44x_tlb.h b/arch/powerpc/kvm/44x_tlb.h
--- a/arch/powerpc/kvm/44x_tlb.h
+++ b/arch/powerpc/kvm/44x_tlb.h
@@ -27,6 +27,8 @@ extern int kvmppc_44x_tlb_index(struct k
                                 unsigned int pid, unsigned int as);
 extern struct tlbe *kvmppc_44x_dtlb_search(struct kvm_vcpu *vcpu, gva_t eaddr);
 extern struct tlbe *kvmppc_44x_itlb_search(struct kvm_vcpu *vcpu, gva_t eaddr);
+
+extern void kvmppc_dump_tlb(struct kvm_vcpu *vcpu);
 
 /* TLB helper functions */
 static inline unsigned int get_tlb_size(const struct tlbe *tlbe)
diff --git a/arch/powerpc/kvm/emulate.c b/arch/powerpc/kvm/emulate.c
--- a/arch/powerpc/kvm/emulate.c
+++ b/arch/powerpc/kvm/emulate.c
@@ -129,6 +129,7 @@ static int kvmppc_emul_tlbwe(struct kvm_
 	if (index > PPC44x_TLB_SIZE) {
 		printk("%s: index %d\n", __func__, index);
 		kvmppc_dump_vcpu(vcpu);
+		kvmppc_dump_tlb(vcpu);
 		return EMULATE_FAIL;
 	}
 
@@ -138,6 +139,14 @@ static int kvmppc_emul_tlbwe(struct kvm_
 #endif
 
 	tlbe = &vcpu->arch.guest_tlb[index];
+
+	if ((get_tlb_eaddr(tlbe) >> 30) == 0xc) {
+		printk("evicting %02d: %08x %08x %08x %08x\n", index,
+		tlbe->tid, tlbe->word0, tlbe->word1, tlbe->word2);
+		kvmppc_dump_vcpu(vcpu);
+		WARN_ON(1);
+	}
+
 
 	/* Invalidate shadow mappings for the about-to-be-clobbered TLBE. */
 	if (tlbe->word0 & PPC44x_TLB_VALID) {
@@ -250,6 +259,8 @@ int kvmppc_emulate_instruction(struct kv
 	switch (get_op(inst)) {
 	case 3:                                                 /* trap */
 		printk("trap!\n");
+		kvmppc_dump_vcpu(vcpu);
+		kvmppc_dump_tlb(vcpu);
 		kvmppc_queue_exception(vcpu, BOOKE_INTERRUPT_PROGRAM);
 		advance = 0;
 		break;
diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
--- a/arch/powerpc/kvm/powerpc.c
+++ b/arch/powerpc/kvm/powerpc.c
@@ -218,6 +218,7 @@ int kvmppc_handle_exit(struct kvm_run *r
 	case BOOKE_INTERRUPT_MACHINE_CHECK:
 		printk("MACHINE CHECK: %lx\n", mfspr(SPRN_MCSR));
 		kvmppc_dump_vcpu(vcpu);
+		kvmppc_dump_tlb(vcpu);
 		r = RESUME_HOST;
 		break;
 
@@ -310,6 +311,12 @@ int kvmppc_handle_exit(struct kvm_run *r
 		gtlbe = kvmppc_44x_dtlb_search(vcpu, eaddr);
 		if (!gtlbe) {
 			/* The guest didn't have a mapping for it. */
+			if ((eaddr >> 30) == 0xc) {
+				printk("Guest mapping f0r 0xc not found!\n");
+				kvmppc_dump_vcpu(vcpu);
+				kvmppc_dump_tlb(vcpu);
+			}
+
 			kvmppc_queue_exception(vcpu, exit_nr);
 			vcpu->arch.dear = vcpu->arch.fault_dear;
 			vcpu->arch.esr = vcpu->arch.fault_esr;
@@ -354,6 +361,12 @@ int kvmppc_handle_exit(struct kvm_run *r
 		gtlbe = kvmppc_44x_itlb_search(vcpu, eaddr);
 		if (!gtlbe) {
 			/* The guest didn't have a mapping for it. */
+			if ((eaddr >> 30) == 0xc) {
+				printk("Guest mapping f0r 0xc not found!\n");
+				kvmppc_dump_vcpu(vcpu);
+				kvmppc_dump_tlb(vcpu);
+			}
+
 			kvmppc_queue_exception(vcpu, exit_nr);
 			r = RESUME_GUEST;
 			break;

[-- Attachment #4: Type: text/plain, Size: 228 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[-- Attachment #5: Type: text/plain, Size: 170 bytes --]

_______________________________________________
kvm-ppc-devel mailing list
kvm-ppc-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel