oops in kvm:gfn_to_rmap

oops in kvm:gfn_to_rmap

[Adrian Bridgett]

We've had two machines crash (at a weekend of course! \o/ for IPMI
remote reboot!) - both times the machines were pingable but ssh failed
(no ssh banner via telnet). This time there was an oops.

host is debian lenny:
linux-image-2.6.26-1-amd64
kvm: 83+dfsg-1

box has several SMP guests on it and had been stable until quite
recently (I think it's a pure load issue).  Quad core Xeon E5420, 16GB
RAM.  Box oops is from has these guests:
- Windows SBS 2008 (4GB - no virtio yet)
- Centos 5.2 (4GB, no virtio (will change that!))
- Vista 32bit (1GB virtio)
- Windows 2003 web edition (1GB virtio)

Googling shows a patch from last year which I presume is in kvm-83:
http://www.mail-archive.com/kvm@vger.kernel.org/msg01312.html

Oops follows:

PGD 27a874067 PUD 4ba84d067 PMD 0 
CPU 2 
Modules linked in: ipmi_watchdog ipmi_devintf ipmi_si ipmi_msghandler tun kvm_intel kvm ipv6 bridge loop snd_pcm snd_timer snd soundcore snd_page_alloc iTCO_wdt pcspkr serio_raw psmouse rng_core usbhid hid i5000_edac ff_memless edac_core button shpchp pci_hotplug evdev dcdbas ext3 jbd mbcache dm_mirror dm_log dm_snapshot dm_mod sg sr_mod cdrom ide_pci_generic ide_core ata_generic sd_mod ata_piix libata dock ehci_hcd uhci_hcd mptsas mptscsih mptbase scsi_transport_sas scsi_mod bnx2 firmware_class thermal processor fan thermal_sys
Pid: 1999, comm: kvm Not tainted 2.6.26-1-amd64 #1
RIP: 0010:[<ffffffffa02935c4>]  [<ffffffffa02935c4>] :kvm:gfn_to_rmap+0x17/0x49
RSP: 0018:ffff81000a59ba38  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 00000000000fee01 RSI: fffffffffffff001 RDI: ffff810449894000
RBP: fffffffffffff001 R08: 000000000000000a R09: 000000000000000a
R10: 0000000000411ef1 R11: 000000000000000b R12: ffff810174746ea0
R13: ffff810449894000 R14: ffff8101c90bc000 R15: ffffffffa02a9e00
FS:  0000000040bfa950(0000) GS:ffff8104bccfff40(0000) knlGS:0000000000000000
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 0000000000000000 CR3: 000000048dff5000 CR4: 00000000000026e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kvm (pid: 1999, threadinfo ffff81000a59a000, task ffff8104bcec6280)
Stack:  00000000002486dd 00000000002486dd ffff8101c90bc038 ffffffffa02936a5
 000000000000000b ffff8101c90bc038 ffff810174746ea0 ffff810449894000
 0000000000000038 ffffffffa029383a 0000000000000010 ffff81000a5614d8
Call Trace:
 [<ffffffffa02936a5>] ? :kvm:rmap_remove+0xaf/0x198
 [<ffffffffa029383a>] ? :kvm:kvm_mmu_zap_page+0x8a/0x270
 [<ffffffffa0294f1a>] ? :kvm:kvm_mmu_pte_write+0x2aa/0x721
 [<ffffffffa0295c86>] ? :kvm:paging64_gva_to_gpa+0x21/0x44
 [<ffffffffa028c79b>] ? :kvm:mark_page_dirty+0x1c/0x3f
 [<ffffffffa028f87a>] ? :kvm:emulator_write_phys+0x36/0x46
 [<ffffffffa0290c93>] ? :kvm:emulator_write_emulated_onepage+0x68/0xe6
 [<ffffffffa029ab36>] ? :kvm:x86_emulate_insn+0x378f/0x4245
 [<ffffffffa02972b2>] ? :kvm:x86_decode_insn+0x9e4/0xad9
 [<ffffffffa028f711>] ? :kvm:emulate_instruction+0x1a0/0x2d3
 [<ffffffffa0294c35>] ? :kvm:kvm_mmu_page_fault+0x46/0x81
 [<ffffffffa029231e>] ? :kvm:kvm_arch_vcpu_ioctl_run+0x463/0x615
 [<ffffffffa028d6ce>] ? :kvm:kvm_vcpu_ioctl+0xf3/0x3a9
 [<ffffffff80228496>] ? __wake_up_common+0x41/0x74
 [<ffffffff8022adc9>] ? __wake_up+0x38/0x4f
 [<ffffffff8024f82b>] ? wake_futex+0x1f/0x29
 [<ffffffff8024fb2e>] ? futex_wake+0xbb/0xfa
 [<ffffffff8024ac3e>] ? getnstimeofday+0x39/0x98
 [<ffffffff80248d42>] ? ktime_get_ts+0x22/0x4b
 [<ffffffff8023e200>] ? __dequeue_signal+0x19/0x15a
 [<ffffffff8023de87>] ? recalc_sigpending+0xe/0x38
 [<ffffffff8023f17b>] ? dequeue_signal+0x8d/0x111
 [<ffffffff8023fbe4>] ? sys_rt_sigtimedwait+0xf1/0x25f
 [<ffffffff8022f07f>] ? hrtick_set+0x88/0xf7
 [<ffffffff802a6621>] ? vfs_ioctl+0x21/0x6b
 [<ffffffff802a68b3>] ? do_vfs_ioctl+0x248/0x261
 [<ffffffff802a691d>] ? sys_ioctl+0x51/0x70
 [<ffffffff8020beca>] ? system_call_after_swapgs+0x8a/0x8f


 RSP <ffff81000a59ba38>
---[ end trace 543d7cb274e6e19e ]---

Thanks,

Adrian
-- 
Email: adrian@smop.co.uk  -*-  GPG key available on public key servers
Debian GNU/Linux - the maintainable distribution   -*-  www.debian.org

Re: oops in kvm:gfn_to_rmap

[Adrian Bridgett]

In fact, we were running the kvm modules from the kernel tree, I've
just swapped to KVM-83 modules - we'll see how it goes (it's just
crashed twice today).

Thanks,

Adrian

Re: oops in kvm:gfn_to_rmap

[Marcelo Tosatti]

On Mon, Mar 02, 2009 at 08:48:16AM +0000, Adrian Bridgett wrote:
> We've had two machines crash (at a weekend of course! \o/ for IPMI
> remote reboot!) - both times the machines were pingable but ssh failed
> (no ssh banner via telnet). This time there was an oops.
> 
> host is debian lenny:
> linux-image-2.6.26-1-amd64
> kvm: 83+dfsg-1
> 
> box has several SMP guests on it and had been stable until quite
> recently (I think it's a pure load issue).  Quad core Xeon E5420, 16GB
> RAM.  Box oops is from has these guests:
> - Windows SBS 2008 (4GB - no virtio yet)
> - Centos 5.2 (4GB, no virtio (will change that!))
> - Vista 32bit (1GB virtio)
> - Windows 2003 web edition (1GB virtio)
> 
> Googling shows a patch from last year which I presume is in kvm-83:
> http://www.mail-archive.com/kvm@vger.kernel.org/msg01312.html
> 
> Oops follows:

Adrian,

Please paste a few lines above

PGD 27a874067 PUD 4ba84d067 PMD 0 CPU 2

Thanks

> 
> PGD 27a874067 PUD 4ba84d067 PMD 0 
> CPU 2 
> Modules linked in: ipmi_watchdog ipmi_devintf ipmi_si ipmi_msghandler tun kvm_intel kvm ipv6 bridge loop snd_pcm snd_timer snd soundcore snd_page_alloc iTCO_wdt pcspkr serio_raw psmouse rng_core usbhid hid i5000_edac ff_memless edac_core button shpchp pci_hotplug evdev dcdbas ext3 jbd mbcache dm_mirror dm_log dm_snapshot dm_mod sg sr_mod cdrom ide_pci_generic ide_core ata_generic sd_mod ata_piix libata dock ehci_hcd uhci_hcd mptsas mptscsih mptbase scsi_transport_sas scsi_mod bnx2 firmware_class thermal processor fan thermal_sys
> Pid: 1999, comm: kvm Not tainted 2.6.26-1-amd64 #1
> RIP: 0010:[<ffffffffa02935c4>]  [<ffffffffa02935c4>] :kvm:gfn_to_rmap+0x17/0x49
> RSP: 0018:ffff81000a59ba38  EFLAGS: 00010246
> RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> RDX: 00000000000fee01 RSI: fffffffffffff001 RDI: ffff810449894000
> RBP: fffffffffffff001 R08: 000000000000000a R09: 000000000000000a
> R10: 0000000000411ef1 R11: 000000000000000b R12: ffff810174746ea0
> R13: ffff810449894000 R14: ffff8101c90bc000 R15: ffffffffa02a9e00
> FS:  0000000040bfa950(0000) GS:ffff8104bccfff40(0000) knlGS:0000000000000000
> CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
> CR2: 0000000000000000 CR3: 000000048dff5000 CR4: 00000000000026e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process kvm (pid: 1999, threadinfo ffff81000a59a000, task ffff8104bcec6280)
> Stack:  00000000002486dd 00000000002486dd ffff8101c90bc038 ffffffffa02936a5
>  000000000000000b ffff8101c90bc038 ffff810174746ea0 ffff810449894000
>  0000000000000038 ffffffffa029383a 0000000000000010 ffff81000a5614d8
> Call Trace:
>  [<ffffffffa02936a5>] ? :kvm:rmap_remove+0xaf/0x198
>  [<ffffffffa029383a>] ? :kvm:kvm_mmu_zap_page+0x8a/0x270
>  [<ffffffffa0294f1a>] ? :kvm:kvm_mmu_pte_write+0x2aa/0x721
>  [<ffffffffa0295c86>] ? :kvm:paging64_gva_to_gpa+0x21/0x44
>  [<ffffffffa028c79b>] ? :kvm:mark_page_dirty+0x1c/0x3f
>  [<ffffffffa028f87a>] ? :kvm:emulator_write_phys+0x36/0x46
>  [<ffffffffa0290c93>] ? :kvm:emulator_write_emulated_onepage+0x68/0xe6
>  [<ffffffffa029ab36>] ? :kvm:x86_emulate_insn+0x378f/0x4245
>  [<ffffffffa02972b2>] ? :kvm:x86_decode_insn+0x9e4/0xad9
>  [<ffffffffa028f711>] ? :kvm:emulate_instruction+0x1a0/0x2d3
>  [<ffffffffa0294c35>] ? :kvm:kvm_mmu_page_fault+0x46/0x81

Re: oops in kvm:gfn_to_rmap

[Adrian Bridgett]

On Mon, Mar  2, 2009 at 12:12:09 -0300 (-0300), Marcelo Tosatti wrote:
> Adrian,
> 
> Please paste a few lines above
> 
> PGD 27a874067 PUD 4ba84d067 PMD 0 CPU 2

Oops! Missed those - sorry!

BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
IP: [<ffffffffa02935c4>] :kvm:gfn_to_rmap+0x17/0x49

Since switching to the -83 modules it's been up most of the day
(crashed twice this morning).  Definitely too early to tell if it's
fixed it, really need to leave it a week or so to be sure.

Thanks

Adrian