Re: [kvm-ppc-devel] Guests oops when trying to mount initramfs

Re: [kvm-ppc-devel] Guests oops when trying to mount initramfs

[Christian Ehrhardt]

Hi,
I already chatted with Hollis a bit about that, but wanted to ask you about it. I also put kvm-ppc-devel on cc because it seem to be a common issue.
I as well as Hollis saw that kind of bug in the past occasionally but never in a reproducible way to be able to debug it.
So the first question is - is your scenario running into that bug to ~>95% ?

And if it reproducible in your case we can use it to find the reason for it, here a short summary from my past discussions about that issue:
- effectively what you see is that the guest has an itlb miss for it's kernel mapping.
- we only deliver that to the guest if we can't find that mapping in the guest tlb
- this should never happen because the guest should never remove it's own kernel mapping and therefor we should be able to cover all itlb misses in the host by using the guest tlb
- but it happens so the question is when&why is the guest kernel mapping removed from the guest tlb
- afaik we have only one place that might remove that mapping which is the function kvmppc_emul_tlbwe
- we also know the kernel mapping which is usually a single 16mb mapping we initialize for the guest in kvm_arch_vcpu_setup (I might be wrong here - Hollis?)
=> we could track that tlbwe that should never happen by inserting a warn_on into kvmppc_emul_tlbwe if it overwrites a guest tlb kernel mapping and starting from there debug whats going on

Any comments or other suggestions are welcome

Jerone Young wrote:
> I'm looking into this. But got a guest to load without the device tree
> mainpulation code. I have an cuImage with a prefilled device tree built
> in & initramfs integrated. 
> 
> It oopses at Decrementor() 
> 
> Does not mount the initramfs 
> 
> Here is the trace:
> 
[...]
> RPC: Registered udp transport module.
> RPC: Registered tcp transport module.
> Freeing unused kernel memory: 1500k init
> Unable to handle kernel paging request for instruction fetch
> Faulting instruction address: 0xc0000ac0
> Oops: Kernel access of bad area, sig: 1 [#1]
> Bamboo
> Modules linked in:
> NIP: c0000ac0 LR: 48003fdc CTR: 00000000
> REGS: bfcfe910 TRAP: 0400   Not tainted  (2.6.25-rc3-dirty)
> MSR: 00021000 <ME>  CR: 42000028  XER: 00000000
> TASK = c8810400[1] 'init' THREAD: c881e000
> GPR00: 00000001 bfcfe9c0 00000000 480177b0 4801795c 00000001 00000000
> 48011f34 
> GPR08: 0002d002 48026e64 00000000 bfcfe9c0 c881e000 00000000 00000000
> 00000000 
> GPR16: 00000000 48018700 00000001 c0240000 c0240000 bfcfea48 00000007
> 10000034 
> GPR24: 00000000 6fffff40 48017000 48027f4c 00000000 480177b0 480272e4
> bfcfe9c0 
> NIP [c0000ac0] Decrementer+0x0/0xc0
> LR [48003fdc] 0x48003fdc
> Call Trace:
> Instruction dump:
> 914b00b0 3d400002 614a1002 512a0420 4800c6b5 c000b378 c000d924 60000000 
> 60000000 60000000 60000000 60000000 <7d5043a6> 7d7143a6 7c3443a6
> 7d400026 
> ---[ end trace 59dad6c3a6e54fa0 ]---


-- 

Grüsse / regards, 
Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-ppc-devel mailing list
kvm-ppc-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel

Re: [kvm-ppc-devel] Guests oops when trying to mount initramfs

[Hollis Blanchard]

Yup, this is definitely the right direction. I actually hadn't looked at
tlbwe yet, but that's probably where the issue is.

I'm also looking at adding some "relay" debugfs code to export all of
KVM's TLB operations to userspace for analysis. There are a *lot* of
them, and printk is not suitable for this data. :) This will be of more
general use down the road when we consider more sophisticated shadow TLB
eviction policies (i.e. not round robin)...

-- 
Hollis Blanchard
IBM Linux Technology Center

On Wed, 2008-03-12 at 13:09 +0100, Christian Ehrhardt wrote:
> Hi,
> I already chatted with Hollis a bit about that, but wanted to ask you about it. I also put kvm-ppc-devel on cc because it seem to be a common issue.
> I as well as Hollis saw that kind of bug in the past occasionally but never in a reproducible way to be able to debug it.
> So the first question is - is your scenario running into that bug to ~>95% ?
> 
> And if it reproducible in your case we can use it to find the reason for it, here a short summary from my past discussions about that issue:
> - effectively what you see is that the guest has an itlb miss for it's kernel mapping.
> - we only deliver that to the guest if we can't find that mapping in the guest tlb
> - this should never happen because the guest should never remove it's own kernel mapping and therefor we should be able to cover all itlb misses in the host by using the guest tlb
> - but it happens so the question is when&why is the guest kernel mapping removed from the guest tlb
> - afaik we have only one place that might remove that mapping which is the function kvmppc_emul_tlbwe
> - we also know the kernel mapping which is usually a single 16mb mapping we initialize for the guest in kvm_arch_vcpu_setup (I might be wrong here - Hollis?)
> => we could track that tlbwe that should never happen by inserting a warn_on into kvmppc_emul_tlbwe if it overwrites a guest tlb kernel mapping and starting from there debug whats going on
> 
> Any comments or other suggestions are welcome
> 
> Jerone Young wrote:
> > I'm looking into this. But got a guest to load without the device tree
> > mainpulation code. I have an cuImage with a prefilled device tree built
> > in & initramfs integrated. 
> > 
> > It oopses at Decrementor() 
> > 
> > Does not mount the initramfs 
> > 
> > Here is the trace:
> > 
> [...]
> > RPC: Registered udp transport module.
> > RPC: Registered tcp transport module.
> > Freeing unused kernel memory: 1500k init
> > Unable to handle kernel paging request for instruction fetch
> > Faulting instruction address: 0xc0000ac0
> > Oops: Kernel access of bad area, sig: 1 [#1]
> > Bamboo
> > Modules linked in:
> > NIP: c0000ac0 LR: 48003fdc CTR: 00000000
> > REGS: bfcfe910 TRAP: 0400   Not tainted  (2.6.25-rc3-dirty)
> > MSR: 00021000 <ME>  CR: 42000028  XER: 00000000
> > TASK = c8810400[1] 'init' THREAD: c881e000
> > GPR00: 00000001 bfcfe9c0 00000000 480177b0 4801795c 00000001 00000000
> > 48011f34 
> > GPR08: 0002d002 48026e64 00000000 bfcfe9c0 c881e000 00000000 00000000
> > 00000000 
> > GPR16: 00000000 48018700 00000001 c0240000 c0240000 bfcfea48 00000007
> > 10000034 
> > GPR24: 00000000 6fffff40 48017000 48027f4c 00000000 480177b0 480272e4
> > bfcfe9c0 
> > NIP [c0000ac0] Decrementer+0x0/0xc0
> > LR [48003fdc] 0x48003fdc
> > Call Trace:
> > Instruction dump:
> > 914b00b0 3d400002 614a1002 512a0420 4800c6b5 c000b378 c000d924 60000000 
> > 60000000 60000000 60000000 60000000 <7d5043a6> 7d7143a6 7c3443a6
> > 7d400026 
> > ---[ end trace 59dad6c3a6e54fa0 ]---
> 



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-ppc-devel mailing list
kvm-ppc-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel

Re: [kvm-ppc-devel] Guests oops when trying to mount initramfs

[Christian Ehrhardt]

[-- Attachment #1: Type: text/plain, Size: 8053 bytes --]

Christian Ehrhardt wrote:
[...]
> And if it reproducible in your case we can use it to find the reason for it, here a short summary from my past discussions about that issue:
> - effectively what you see is that the guest has an itlb miss for it's kernel mapping.
> - we only deliver that to the guest if we can't find that mapping in the guest tlb
> - this should never happen because the guest should never remove it's own kernel mapping and therefor we should be able to cover all itlb misses in the host by using the guest tlb
> - but it happens so the question is when&why is the guest kernel mapping removed from the guest tlb
> - afaik we have only one place that might remove that mapping which is the function kvmppc_emul_tlbwe
> - we also know the kernel mapping which is usually a single 16mb mapping we initialize for the guest in kvm_arch_vcpu_setup (I might be wrong here - Hollis?)
> => we could track that tlbwe that should never happen by inserting a warn_on into kvmppc_emul_tlbwe if it overwrites a guest tlb kernel mapping and starting from there debug whats going on
>

ok for Hollis to continue and anyone else that might have comments
-> tracking any eviction of the guest kernel mapping (=0xC..) did not trigger
-> tracking the delivery of a itlb/dtlb miss with that address did not trigger
-> I inserted a BUG statement in the guests report of "Unable to handle kernel paging request ..."
   That triggers a emulation of a trap in the host and there I have a dump_vcpu&dump_tlb
-> for the case that someone wants to reproduce that I attached debug patches for host and guest kernel 

   Here is the output when hitting the trap:

Looking up port of RPC 100005/1 on 192.168.1.2
VFS: Mounted root (nfs filesystem).
Freeing unused kernel memory: 116k init
Unable to handle kernel paging request for instruction fetch
Faulting instruction address: 0xc0000ac0
trap!
pc:   c000fcbc msr:  00021002
lr:   c000fcbc ctr:  c012dc54
srr0: c012d638 srr1: 00021002
exceptions: 00000000
gpr00: c000fcbc bff2aaf0 c8810400 0000002c
gpr04: 00000001 00000001 00000000 00000004
gpr08: 00000001 c025ca00 00001250 c0260000
gpr12: 65930e5e 1001f2f8 00000000 00000000
gpr16: 00000000 00000000 00000000 c01f0000
gpr20: c0240000 c0240000 00000000 00000000
gpr24: c0240000 c01f0000 00000001 48026e60
gpr28: 48027a10 bff2af04 00000001 bff2ab10
vcpu 0 TLB dump:
| nr |     tid  |    word0 |    word1 |    word2 |
G 1 | 00000001 | 0FF3F210 | 08FAB000 | 08FA016D |
G 2 | 00000001 | 0FEAD210 | 08FDC000 | 08FD0149 |
G 3 | 00000001 | 48026210 | 08FFB000 | 08FF0349 |
G 5 | 00000001 | 10003210 | 0022D000 | 0022016D |
G 6 | 00000000 | D1012210 | EF600000 | EF600703 |
G 7 | 00000001 | 4800C210 | 003D7000 | 003D016D |
G 8 | 00000001 | 10000210 | 00227000 | 00220149 |
G 9 | 00000001 | 0FFE8210 | 08FE6000 | 08FE0349 |
G11 | 00000001 | 0FEC5210 | 08FD4000 | 08FD016D |
G13 | 00000001 | 0FFEA210 | 08FCD000 | 08FC036F |
G14 | 00000001 | 0FFEC210 | 08FE5000 | 08FE035B |
G15 | 00000001 | 0FFEF210 | 08FDE000 | 08FD035B |
G16 | 00000000 | D1021210 | E8001000 | E8000703 |
G18 | 00000001 | 0FF70210 | 003EB000 | 003E016D |
G20 | 00000001 | 0FF20210 | 003F8000 | 003F016D |
G22 | 00000001 | 0FF1F210 | 003F9000 | 003F016D |
G23 | 00000001 | 0FFE7210 | 08FCB000 | 08FC0349 |
G25 | 00000001 | 0FF17210 | 08F85000 | 08F8016D |
G26 | 00000001 | 48013210 | 00235000 | 0023016D |
G28 | 00000001 | 10001210 | 00228000 | 0022016D |
G31 | 00000001 | 10017210 | 00242000 | 0024037F |
G32 | 00000001 | 48012210 | 00234000 | 0023016D |
G33 | 00000001 | 0FEA9210 | 003DC000 | 003D0149 |
G34 | 00000001 | 0FEB3210 | 08FEB000 | 08FE0149 |
G35 | 00000001 | 0FEB9210 | 08FEC000 | 08FE0149 |
G36 | 00000001 | 0FEBA210 | 08FC2000 | 08FC0149 |
G37 | 00000001 | 48009210 | 08FFA000 | 08FF016D |
G39 | 00000001 | 0FEDD210 | 08F8A000 | 08F8016D |
G40 | 00000001 | 0FFED210 | 08FCA000 | 08FC035B |
G42 | 00000001 | 10005210 | 0022F000 | 0022016D |
G43 | 00000001 | 10006210 | 00230000 | 0023016D |
G45 | 00000001 | 0FEDA210 | 08F8D000 | 08F8016D |
G46 | 00000001 | 4801C210 | 08FC9000 | 08FC035B |
G47 | 00000001 | 0FEB4210 | 08FC1000 | 08FC0149 |
G48 | 00000001 | 0FEAB210 | 08FD7000 | 08FD0149 |
G49 | 00000001 | 0FEB0210 | 08FD9000 | 08FD0149 |
G50 | 00000001 | 0FEB7210 | 08FEE000 | 08FE0149 |
G51 | 00000001 | 0FEAF210 | 08FDA000 | 08FD0149 |
G52 | 00000001 | 0FEB6210 | 08FEF000 | 08FE0149 |
G53 | 00000001 | 0FEAA210 | 003DB000 | 003D0149 |
G55 | 00000001 | 0FF61210 | 08F97000 | 08F9016D |
G56 | 00000001 | 0FEB2210 | 08FE7000 | 08FE0149 |
G57 | 00000001 | 0FEB8210 | 08FED000 | 08FE0149 |
G58 | 00000001 | 0FEAC210 | 08FDD000 | 08FD0149 |
G59 | 00000001 | 0FEB5210 | 08FC0000 | 08FC0149 |
G60 | 00000001 | 48008210 | 08FFC000 | 08FF016D |
G61 | 00000001 | 48027210 | 00241000 | 0024037F |
G62 | 00000001 | BFF2A210 | 00243000 | 0024035B |
G63 | 00000000 | C0000290 | 00000000 | 00000107 |
S17 | 00000001 | 10003310 | 0E3B8000 | 0000002F |
S18 | 00000001 | BFF2A310 | 0E3CE000 | 0000001F |
S19 | 00000001 | 10017310 | 0E3CD000 | 0000003F |
S20 | 00000000 | C0000310 | 0E38B000 | 0000003F |
S21 | 00000000 | C0246310 | 0E3D1000 | 0000003F |
S22 | 00000000 | C000C310 | 0E3E7000 | 0000003F |
S23 | 00000000 | C8810310 | 0E9BA000 | 0000003F |
S24 | 00000000 | C000D310 | 0E3E8000 | 0000003F |
S25 | 00000000 | C000F310 | 0E3EA000 | 0000003F |
S26 | 00000000 | C0264310 | 0DC31000 | 0000003F |
S27 | 00000000 | C0037310 | 0DCF2000 | 0000003F |
S28 | 00000000 | C0106310 | 0E9D1000 | 0000003F |
S29 | 00000000 | C0223310 | 0E3AE000 | 0000003F |
S30 | 00000000 | C0047310 | 0DD02000 | 0000003F |
S31 | 00000000 | C024A310 | 0E3D5000 | 0000003F |
S32 | 00000000 | C0024310 | 0DC1F000 | 0000003F |
S33 | 00000000 | C0248310 | 0E3D3000 | 0000003F |
S34 | 00000000 | C010D310 | 0E9D8000 | 0000003F |
S35 | 00000000 | C010C310 | 0E9D7000 | 0000003F |
S36 | 00000000 | C01F6310 | 0E201000 | 0000003F |
S37 | 00000000 | C026B310 | 0DC38000 | 0000003F |
S38 | 00000000 | C026C310 | 0DC39000 | 0000003F |
S39 | 00000000 | C0023310 | 0DC1E000 | 0000003F |
S40 | 00000000 | C0257310 | 0DC22000 | 0000003F |
S41 | 00000000 | C012D310 | 0E0B8000 | 0000003F |
S42 | 00000000 | C004A310 | 0DD05000 | 0000003F |
S43 | 00000000 | C001E310 | 0DC19000 | 0000003F |
S44 | 00000000 | C001D310 | 0DC18000 | 0000003F |
S45 | 00000000 | C000A310 | 0E3E5000 | 0000003F |
S46 | 00000000 | C0262310 | 0DC2D000 | 0000003F |
S47 | 00000000 | C027C310 | 0DC69000 | 0000003F |
S48 | 00000000 | C01D2310 | 0E13D000 | 0000003F |
S49 | 00000000 | C8835310 | 0E06B000 | 0000003F |
S50 | 00000000 | C0001310 | 0E38C000 | 0000003F |
S51 | 00000000 | C025C310 | 0DC27000 | 0000003F |
S52 | 00000000 | C027D310 | 0DC6A000 | 0000003F |
S53 | 00000000 | C012A310 | 0E0B5000 | 0000003F |
S54 | 00000000 | C0130310 | 0E0BB000 | 0000003F |
S55 | 00000000 | C01D0310 | 0E13B000 | 0000003F |
S56 | 00000000 | C0256310 | 0DC21000 | 0000003F |
Oops: Exception in kernel mode, sig: 4 [#1]
Bamboo
Modules linked in:
NIP: c000fcbc LR: c000fcbc CTR: c012dc54
REGS: bff2aa40 TRAP: 0700   Not tainted  (2.6.25-rc3)
MSR: 00021002 <ME>  CR: 24002022  XER: 00000000
TASK = c8810400[1] 'init' THREAD: c881e000
GPR00: c000fcbc bff2aaf0 c8810400 0000002c 00000001 00000001 00000000 00000004
GPR08: 00000001 c025ca00 00001250 c0260000 65930e5e 1001f2f8 00000000 00000000
GPR16: 00000000 00000000 00000000 c01f0000 c0240000 c0240000 00000000 00000000
GPR24: c0240000 c01f0000 00000001 48026e60 48027a10 bff2af04 00000001 bff2ab10
NIP [c000fcbc] bad_page_fault+0x7c/0xb4
LR [c000fcbc] bad_page_fault+0x7c/0xb4
Call Trace:
Instruction dump:
2b800380 419d0034 2f800300 40be003c 3c60c01f 809f00a4 38636580 48015271
809f0080 3c60c01f 3863664c 48015261 <0fe00000> 48000000 2f800400 419e001c
---[ end trace 9c05eabdb79d9d2c ]---
Kernel panic - not syncing: Attempted to kill init!
Rebooting in 180 seconds..


-- 

Grüsse / regards, 
Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization

[-- Attachment #2: debug-guest-kernel-mapping-guestpatch.diff --]
[-- Type: text/x-patch, Size: 359 bytes --]

diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c
--- a/arch/powerpc/mm/fault.c
+++ b/arch/powerpc/mm/fault.c
@@ -432,5 +432,8 @@ void bad_page_fault(struct pt_regs *regs
 	printk(KERN_ALERT "Faulting instruction address: 0x%08lx\n",
 		regs->nip);
 
+	// to trigger kvmppc host trap!
+	BUG();
+
 	die("Kernel access of bad area", regs, sig);
 }

[-- Attachment #3: debug-guest-kernel-mapping-hostpatch.diff --]
[-- Type: text/x-patch, Size: 4162 bytes --]

diff --git a/arch/powerpc/kvm/44x_tlb.c b/arch/powerpc/kvm/44x_tlb.c
--- a/arch/powerpc/kvm/44x_tlb.c
+++ b/arch/powerpc/kvm/44x_tlb.c
@@ -21,6 +21,7 @@
 #include <linux/string.h>
 #include <linux/kvm_host.h>
 #include <linux/highmem.h>
+#include <linux/delay.h>
 #include <asm/mmu-44x.h>
 
 #include "44x_tlb.h"
@@ -29,6 +30,36 @@
 #define PPC44x_TLB_SUPER_PERM_MASK (PPC44x_TLB_SX|PPC44x_TLB_SR|PPC44x_TLB_SW)
 
 static unsigned int kvmppc_tlb_44x_pos;
+
+void kvmppc_dump_tlb(struct kvm_vcpu *vcpu)
+{
+	struct tlbe *tlbe;
+	int i;
+
+	printk("vcpu %d TLB dump:\n", vcpu->vcpu_id);
+	printk("| %2s | %8s | %8s | %8s | %8s |\n",
+		"nr", " tid ", "word0", "word1", "word2");
+
+	for (i = 0; i < PPC44x_TLB_SIZE; i++)
+	{
+		tlbe = &vcpu->arch.guest_tlb[i];
+		if (tlbe->word0 & PPC44x_TLB_VALID)
+			printk("G%2d | %08X | %08X | %08X | %08X |\n",
+			i, tlbe->tid, tlbe->word0, tlbe->word1,	tlbe->word2);
+	}
+
+	msleep(500);
+
+	for (i = 0; i < PPC44x_TLB_SIZE; i++)
+	{
+		tlbe = &vcpu->arch.shadow_tlb[i];
+		if (tlbe->word0 & PPC44x_TLB_VALID)
+			printk("S%2d | %08X | %08X | %08X | %08X |\n",
+			i, tlbe->tid, tlbe->word0, tlbe->word1,	tlbe->word2);
+	}
+
+	msleep(500);
+}
 
 static u32 kvmppc_44x_tlb_shadow_attrib(u32 attrib, int usermode)
 {
diff --git a/arch/powerpc/kvm/44x_tlb.h b/arch/powerpc/kvm/44x_tlb.h
--- a/arch/powerpc/kvm/44x_tlb.h
+++ b/arch/powerpc/kvm/44x_tlb.h
@@ -27,6 +27,8 @@ extern int kvmppc_44x_tlb_index(struct k
                                 unsigned int pid, unsigned int as);
 extern struct tlbe *kvmppc_44x_dtlb_search(struct kvm_vcpu *vcpu, gva_t eaddr);
 extern struct tlbe *kvmppc_44x_itlb_search(struct kvm_vcpu *vcpu, gva_t eaddr);
+
+extern void kvmppc_dump_tlb(struct kvm_vcpu *vcpu);
 
 /* TLB helper functions */
 static inline unsigned int get_tlb_size(const struct tlbe *tlbe)
diff --git a/arch/powerpc/kvm/emulate.c b/arch/powerpc/kvm/emulate.c
--- a/arch/powerpc/kvm/emulate.c
+++ b/arch/powerpc/kvm/emulate.c
@@ -129,6 +129,7 @@ static int kvmppc_emul_tlbwe(struct kvm_
 	if (index > PPC44x_TLB_SIZE) {
 		printk("%s: index %d\n", __func__, index);
 		kvmppc_dump_vcpu(vcpu);
+		kvmppc_dump_tlb(vcpu);
 		return EMULATE_FAIL;
 	}
 
@@ -138,6 +139,14 @@ static int kvmppc_emul_tlbwe(struct kvm_
 #endif
 
 	tlbe = &vcpu->arch.guest_tlb[index];
+
+	if ((get_tlb_eaddr(tlbe) >> 30) == 0xc) {
+		printk("evicting %02d: %08x %08x %08x %08x\n", index,
+		tlbe->tid, tlbe->word0, tlbe->word1, tlbe->word2);
+		kvmppc_dump_vcpu(vcpu);
+		WARN_ON(1);
+	}
+
 
 	/* Invalidate shadow mappings for the about-to-be-clobbered TLBE. */
 	if (tlbe->word0 & PPC44x_TLB_VALID) {
@@ -250,6 +259,8 @@ int kvmppc_emulate_instruction(struct kv
 	switch (get_op(inst)) {
 	case 3:                                                 /* trap */
 		printk("trap!\n");
+		kvmppc_dump_vcpu(vcpu);
+		kvmppc_dump_tlb(vcpu);
 		kvmppc_queue_exception(vcpu, BOOKE_INTERRUPT_PROGRAM);
 		advance = 0;
 		break;
diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
--- a/arch/powerpc/kvm/powerpc.c
+++ b/arch/powerpc/kvm/powerpc.c
@@ -218,6 +218,7 @@ int kvmppc_handle_exit(struct kvm_run *r
 	case BOOKE_INTERRUPT_MACHINE_CHECK:
 		printk("MACHINE CHECK: %lx\n", mfspr(SPRN_MCSR));
 		kvmppc_dump_vcpu(vcpu);
+		kvmppc_dump_tlb(vcpu);
 		r = RESUME_HOST;
 		break;
 
@@ -310,6 +311,12 @@ int kvmppc_handle_exit(struct kvm_run *r
 		gtlbe = kvmppc_44x_dtlb_search(vcpu, eaddr);
 		if (!gtlbe) {
 			/* The guest didn't have a mapping for it. */
+			if ((eaddr >> 30) == 0xc) {
+				printk("Guest mapping f0r 0xc not found!\n");
+				kvmppc_dump_vcpu(vcpu);
+				kvmppc_dump_tlb(vcpu);
+			}
+
 			kvmppc_queue_exception(vcpu, exit_nr);
 			vcpu->arch.dear = vcpu->arch.fault_dear;
 			vcpu->arch.esr = vcpu->arch.fault_esr;
@@ -354,6 +361,12 @@ int kvmppc_handle_exit(struct kvm_run *r
 		gtlbe = kvmppc_44x_itlb_search(vcpu, eaddr);
 		if (!gtlbe) {
 			/* The guest didn't have a mapping for it. */
+			if ((eaddr >> 30) == 0xc) {
+				printk("Guest mapping f0r 0xc not found!\n");
+				kvmppc_dump_vcpu(vcpu);
+				kvmppc_dump_tlb(vcpu);
+			}
+
 			kvmppc_queue_exception(vcpu, exit_nr);
 			r = RESUME_GUEST;
 			break;

[-- Attachment #4: Type: text/plain, Size: 228 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[-- Attachment #5: Type: text/plain, Size: 170 bytes --]

_______________________________________________
kvm-ppc-devel mailing list
kvm-ppc-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel

Re: [kvm-ppc-devel] Guests oops when trying to mount initramfs

[Hollis Blanchard]

On Thu, 2008-03-13 at 18:53 +0100, Christian Ehrhardt wrote:
> ok for Hollis to continue and anyone else that might have comments
> -> tracking any eviction of the guest kernel mapping (=0xC..) did not
> trigger
> -> tracking the delivery of a itlb/dtlb miss with that address did not
> trigger
> -> I inserted a BUG statement in the guests report of "Unable to
> handle kernel paging request ..."
>    That triggers a emulation of a trap in the host and there I have a
> dump_vcpu&dump_tlb
> 
I think this just means our debug tests in the host kernel aren't
working. The *only* way the guest interrupt vectors are invoked is by
the host kernel...

-- 
Hollis Blanchard
IBM Linux Technology Center


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-ppc-devel mailing list
kvm-ppc-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel

Re: [kvm-ppc-devel] Guests oops when trying to mount initramfs

[Christian Ehrhardt]

[-- Attachment #1: Type: text/plain, Size: 4374 bytes --]

Hollis Blanchard wrote:
> On Thu, 2008-03-13 at 18:53 +0100, Christian Ehrhardt wrote:
>> ok for Hollis to continue and anyone else that might have comments
>> -> tracking any eviction of the guest kernel mapping (=0xC..) did not
>> trigger
>> -> tracking the delivery of a itlb/dtlb miss with that address did not
>> trigger
>> -> I inserted a BUG statement in the guests report of "Unable to
>> handle kernel paging request ..."
>>    That triggers a emulation of a trap in the host and there I have a
>> dump_vcpu&dump_tlb
>>
> I think this just means our debug tests in the host kernel aren't
> working. The *only* way the guest interrupt vectors are invoked is by
> the host kernel...
> 
right - I checked and changed the eaddr matching and now get useful triggers.
And you now can see the initial programming of the Kernel mapping done by the guest (3x tlbwe). I add the output here for reference, but unfortunately now that I have triggers that are known to work the actual issue does no more occur :-(. I run into some soft lockup all the time now (12/12 tests).
Maybe we need to fix that first (if is not related anyway)?
I attach an updated version of the debug patch for the host kernel.

I only post vcpu (all zero gpr lines removed) and tlb dump, not the stack trace because this tlb programming is expected:

kvmppc_emul_tlbwe - writing or evicting guest kernel mapping
 63: tid 00000000 w0 00000000 w1 00000000 w2 00000000,
 inst: ra 00000000 rs 00000003 ws 00000000
 guest address: 0x0
pc:   000000a4 msr:  00000040
lr:   0000003c ctr:  00000000
srr0: 00000000 srr1: 00000000
dear: 00000000 esr:  00000000
exceptions: 00000000
gpr00: 0000003f 00fffff8 00000000 c0000290
gpr04: 00000000 00000107 00000000 00000000
gpr28: 00000000 00000000 00000000 00263100
vcpu 0 TLB dump:
| nr |     tid  |    word0 |    word1 |    word2 |
G 0 | 00000000 | 00000270 | 00000000 | 00000007 |
S42 | 00000000 | 00000310 | 0E622000 | 0000003F |
[...]
kvmppc_emul_tlbwe - writing or evicting guest kernel mapping
 63: tid 00000000 w0 c0000290 w1 00000000 w2 00000000,
 inst: ra 00000000 rs 00000004 ws 00000001
pc:   000000a8 msr:  00000040
lr:   0000003c ctr:  00000000
srr0: 00000000 srr1: 00000000
dear: 00000000 esr:  00000000
exceptions: 00000000
gpr00: 0000003f 00fffff8 00000000 c0000290
gpr04: 00000000 00000107 00000000 00000000
gpr28: 00000000 00000000 00000000 00263100
vcpu 0 TLB dump:
| nr |     tid  |    word0 |    word1 |    word2 |
G 0 | 00000000 | 00000270 | 00000000 | 00000007 |
G63 | 00000000 | C0000290 | 00000000 | 00000000 |
S42 | 00000000 | 00000310 | 0E622000 | 0000003F |
S43 | 00000000 | C0000310 | 0E622000 | 00000007 |
[...]
kvmppc_emul_tlbwe - writing or evicting guest kernel mapping
 63: tid 00000000 w0 c0000290 w1 00000000 w2 00000000,
 inst: ra 00000000 rs 00000005 ws 00000002
pc:   000000ac msr:  00000040
lr:   0000003c ctr:  00000000
srr0: 00000000 srr1: 00000000
dear: 00000000 esr:  00000000
exceptions: 00000000
gpr00: 0000003f 00fffff8 00000000 c0000290
gpr04: 00000000 00000107 00000000 00000000
gpr28: 00000000 00000000 00000000 00263100
vcpu 0 TLB dump:
| nr |     tid  |    word0 |    word1 |    word2 |
G 0 | 00000000 | 00000270 | 00000000 | 00000007 |
G63 | 00000000 | C0000290 | 00000000 | 00000000 |
S42 | 00000000 | 00000310 | 0E622000 | 0000003F |
S44 | 00000000 | C0000310 | 0E622000 | 00000007 |


And later only the soft lockup in 12/12 guest test runs (without that debug code, even with the non triggering one, at least 1/3 triggered the tlb issue).
Heisenberg affects bugs as it does with quantum effects ;-)
 
BUG: soft lockup - CPU#0 stuck for 61s! [modprobe:677]
NIP: c0000ac0 LR: 48008904 CTR: 00001fe6
REGS: bf896750 TRAP: 0901   Not tainted  (2.6.25-rc3)
MSR: 00021000 <ME>  CR: 44004028  XER: 20000000
TASK = c89de000[677] 'modprobe' THREAD: c89b0000
GPR00: 00000003 bf896800 00000000 0feb86a2 0feb5d45 00000073 0000005f feff0000
GPR08: 00000000 00000000 00000000 0fea919c 42000024
NIP [c0000ac0] Decrementer+0x0/0xc0
LR [48008904] 0x48008904
Call Trace:
Instruction dump:
914b00b0 3d400002 614a1002 512a0420 4800c471 c000ba60 c000d6e0 60000000
60000000 60000000 60000000 60000000 <7d5043a6> 7d7143a6 7c3443a6 7d400026


-- 

Grüsse / regards, 
Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization

[-- Attachment #2: debug-tlbwe-kernel-evict --]
[-- Type: text/plain, Size: 5168 bytes --]

diff --git a/arch/powerpc/kvm/44x_tlb.c b/arch/powerpc/kvm/44x_tlb.c
--- a/arch/powerpc/kvm/44x_tlb.c
+++ b/arch/powerpc/kvm/44x_tlb.c
@@ -21,6 +21,7 @@
 #include <linux/string.h>
 #include <linux/kvm_host.h>
 #include <linux/highmem.h>
+#include <linux/delay.h>
 #include <asm/mmu-44x.h>
 
 #include "44x_tlb.h"
@@ -29,6 +30,36 @@
 #define PPC44x_TLB_SUPER_PERM_MASK (PPC44x_TLB_SX|PPC44x_TLB_SR|PPC44x_TLB_SW)
 
 static unsigned int kvmppc_tlb_44x_pos;
+
+void kvmppc_dump_tlb(struct kvm_vcpu *vcpu)
+{
+	struct tlbe *tlbe;
+	int i;
+
+	printk("vcpu %d TLB dump:\n", vcpu->vcpu_id);
+	printk("| %2s | %8s | %8s | %8s | %8s |\n",
+		"nr", " tid ", "word0", "word1", "word2");
+
+	for (i = 0; i < PPC44x_TLB_SIZE; i++)
+	{
+		tlbe = &vcpu->arch.guest_tlb[i];
+		if (tlbe->word0 & PPC44x_TLB_VALID)
+			printk("G%2d | %08X | %08X | %08X | %08X |\n",
+			i, tlbe->tid, tlbe->word0, tlbe->word1,	tlbe->word2);
+	}
+
+	msleep(500);
+
+	for (i = 0; i < PPC44x_TLB_SIZE; i++)
+	{
+		tlbe = &vcpu->arch.shadow_tlb[i];
+		if (tlbe->word0 & PPC44x_TLB_VALID)
+			printk("S%2d | %08X | %08X | %08X | %08X |\n",
+			i, tlbe->tid, tlbe->word0, tlbe->word1,	tlbe->word2);
+	}
+
+	msleep(500);
+}
 
 static u32 kvmppc_44x_tlb_shadow_attrib(u32 attrib, int usermode)
 {
diff --git a/arch/powerpc/kvm/44x_tlb.h b/arch/powerpc/kvm/44x_tlb.h
--- a/arch/powerpc/kvm/44x_tlb.h
+++ b/arch/powerpc/kvm/44x_tlb.h
@@ -27,6 +27,8 @@ extern int kvmppc_44x_tlb_index(struct k
                                 unsigned int pid, unsigned int as);
 extern struct tlbe *kvmppc_44x_dtlb_search(struct kvm_vcpu *vcpu, gva_t eaddr);
 extern struct tlbe *kvmppc_44x_itlb_search(struct kvm_vcpu *vcpu, gva_t eaddr);
+
+extern void kvmppc_dump_tlb(struct kvm_vcpu *vcpu);
 
 /* TLB helper functions */
 static inline unsigned int get_tlb_size(const struct tlbe *tlbe)
diff --git a/arch/powerpc/kvm/emulate.c b/arch/powerpc/kvm/emulate.c
--- a/arch/powerpc/kvm/emulate.c
+++ b/arch/powerpc/kvm/emulate.c
@@ -21,6 +21,7 @@
 #include <linux/timer.h>
 #include <linux/types.h>
 #include <linux/string.h>
+#include <linux/delay.h>
 #include <linux/kvm_host.h>
 
 #include <asm/dcr.h>
@@ -129,6 +130,7 @@ static int kvmppc_emul_tlbwe(struct kvm_
 	if (index > PPC44x_TLB_SIZE) {
 		printk("%s: index %d\n", __func__, index);
 		kvmppc_dump_vcpu(vcpu);
+		kvmppc_dump_tlb(vcpu);
 		return EMULATE_FAIL;
 	}
 
@@ -138,6 +140,25 @@ static int kvmppc_emul_tlbwe(struct kvm_
 #endif
 
 	tlbe = &vcpu->arch.guest_tlb[index];
+
+	if (	((get_tlb_eaddr(tlbe) & 0xf0000000) == 0xc0000000)
+		 ||
+		((vcpu->arch.gpr[rs] & 0xf0000000) == 0xc0000000) ) {
+		printk("%s - writing or evicting guest kernel mapping"
+			"\n %02d: tid %08x w0 %08x w1 %08x w2 %08x,"
+			"\n inst: ra %08x rs %08x ws %08x \n",
+		__func__, index, 
+		tlbe->tid, tlbe->word0, tlbe->word1, tlbe->word2,
+		ra, rs, ws);
+		msleep(500);
+		kvmppc_dump_vcpu(vcpu);
+		msleep(500);
+		kvmppc_dump_tlb(vcpu);
+		msleep(500);
+		WARN_ON(1);
+		msleep(500);
+	}
+
 
 	/* Invalidate shadow mappings for the about-to-be-clobbered TLBE. */
 	if (tlbe->word0 & PPC44x_TLB_VALID) {
@@ -250,6 +271,8 @@ int kvmppc_emulate_instruction(struct kv
 	switch (get_op(inst)) {
 	case 3:                                                 /* trap */
 		printk("trap!\n");
+		kvmppc_dump_vcpu(vcpu);
+		kvmppc_dump_tlb(vcpu);
 		kvmppc_queue_exception(vcpu, BOOKE_INTERRUPT_PROGRAM);
 		advance = 0;
 		break;
diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
--- a/arch/powerpc/kvm/powerpc.c
+++ b/arch/powerpc/kvm/powerpc.c
@@ -70,6 +70,7 @@ void kvmppc_dump_vcpu(struct kvm_vcpu *v
 	printk("pc:   %08x msr:  %08x\n", vcpu->arch.pc, vcpu->arch.msr);
 	printk("lr:   %08x ctr:  %08x\n", vcpu->arch.lr, vcpu->arch.ctr);
 	printk("srr0: %08x srr1: %08x\n", vcpu->arch.srr0, vcpu->arch.srr1);
+	printk("dear: %08x esr:  %08x\n", vcpu->arch.dear, vcpu->arch.esr);
 
 	printk("exceptions: %08lx\n", vcpu->arch.pending_exceptions);
 
@@ -218,6 +219,7 @@ int kvmppc_handle_exit(struct kvm_run *r
 	case BOOKE_INTERRUPT_MACHINE_CHECK:
 		printk("MACHINE CHECK: %lx\n", mfspr(SPRN_MCSR));
 		kvmppc_dump_vcpu(vcpu);
+		kvmppc_dump_tlb(vcpu);
 		r = RESUME_HOST;
 		break;
 
@@ -309,7 +311,13 @@ int kvmppc_handle_exit(struct kvm_run *r
 		/* Check the guest TLB. */
 		gtlbe = kvmppc_44x_dtlb_search(vcpu, eaddr);
 		if (!gtlbe) {
-			/* The guest didn't have a mapping for it. */
+		/* The guest didn't have a mapping for it. */
+			if ((eaddr & 0xf0000000) == 0xc0000000) {
+				printk("DTLBMiss - Guest mapping f0r 0xc not found!\n");
+				kvmppc_dump_vcpu(vcpu);
+				kvmppc_dump_tlb(vcpu);
+			}
+
 			kvmppc_queue_exception(vcpu, exit_nr);
 			vcpu->arch.dear = vcpu->arch.fault_dear;
 			vcpu->arch.esr = vcpu->arch.fault_esr;
@@ -354,6 +362,12 @@ int kvmppc_handle_exit(struct kvm_run *r
 		gtlbe = kvmppc_44x_itlb_search(vcpu, eaddr);
 		if (!gtlbe) {
 			/* The guest didn't have a mapping for it. */
+			if ((eaddr & 0xf0000000) == 0xc0000000) {
+				printk("ITLBMiss - Guest mapping f0r 0xc not found!\n");
+				kvmppc_dump_vcpu(vcpu);
+				kvmppc_dump_tlb(vcpu);
+			}
+
 			kvmppc_queue_exception(vcpu, exit_nr);
 			r = RESUME_GUEST;
 			break;

[-- Attachment #3: Type: text/plain, Size: 228 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[-- Attachment #4: Type: text/plain, Size: 170 bytes --]

_______________________________________________
kvm-ppc-devel mailing list
kvm-ppc-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel

Re: [kvm-ppc-devel] Guests oops when trying to mount initramfs

[Christian Ehrhardt]

Christian Ehrhardt wrote:
> 
> And later only the soft lockup in 12/12 guest test runs (without that 
> debug code, even with the non triggering one, at least 1/3 triggered the 
> tlb issue).
> Heisenberg affects bugs as it does with quantum effects ;-)
> 
To update this thread and keep it's completeness - with a bit luck I got the issue again while running with the debug code and can now confirm that we are not delivering a ITLB Miss for the guest kernel mapping prior to the issue.

-- 

Grüsse / regards, 
Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-ppc-devel mailing list
kvm-ppc-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/kvm-ppc-devel