Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36

xen-devel.lists.xenproject.org archive mirror
 help / color / mirror / Atom feed

* Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36
@ 2011-04-13 21:02 Gerd Jakobovitsch
  2011-04-14 13:15 ` Konrad Rzeszutek Wilk
  0 siblings, 1 reply; 11+ messages in thread
From: Gerd Jakobovitsch @ 2011-04-13 21:02 UTC (permalink / raw)
  To: xen-devel

I'm trying to run several VMs (linux hvm, with tapdisk:aio disks at a 
storage over nfs) on a CentOS system, using the up-to-date version of 
xen 4.0 / kernel pvops 2.6.32.x stable. With a configuration without 
(most of) debug activated, I can start several instances - I'm running 7 
of them - but shortly afterwards the system stops responding. I can't 
find any information on this.

Activating several debug configuration items, among them 
DEBUG_PAGEALLOC, I get an exception as soon as I try to start up a VM. 
The system reboots.

Below the log from /var/log/messages:

Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Created 
/dev/xen/blktap-2/control device
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Created 
/dev/xen/blktap-2/blktap0 device
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Created 
/dev/xen/blktap-2/tapdev0 device
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: new interface: ring: 251, 
device: 253, minor: 0
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: I/O queue driver: lio
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: block-aio 
open('/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda')
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: 
open(/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda) 
with O_DIRECT
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Image size:       pre 
sector_shift  [134217728]   post sector_shift [262144]
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: opened image 
/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda (1 
users, state: 0x00000001, type: 0)
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: VBD CHAIN:
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: 
/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda: 0
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.158549] block tda: 
sector-size: 512 capacity: 262144
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200514] general protection 
fault: 0000 [#1] SMP DEBUG_PAGEALLOC
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200703] last sysfs file: 
/sys/block/tda/removable
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200761] CPU 0
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200847] Modules linked in: 
bridge stp bonding bnx2i libiscsi scsi_transport_iscsi cnic uio bnx2 
megaraid_sas
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201363] Pid: 4988, comm: 
tapdisk2 Not tainted 2.6.32.36 #3 PowerEdge M610
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201424] RIP: 
e030:[<ffffffff812b9c24>]  [<ffffffff812b9c24>] 
blktap_device_end_request+0x49/0x5e
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201543] RSP: 
e02b:ffff88006a7f7cd8  EFLAGS: 00010046
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201600] RAX: 
6b6b6b6b6b6b6b6b RBX: ffff88006a6fc000 RCX: ffff88006a7f7c38
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201662] RDX: 
0000000000000000 RSI: 0000000000000000 RDI: ffff88006a5c3500
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201723] RBP: 
ffff88006a7f7cf8 R08: ffffffff818383c0 R09: ffff88006a7f7c38
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201784] R10: 
0000000000000000 R11: ffff88007b697b18 R12: ffff88007b697b18
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201845] R13: 
ffff88006a5c3360 R14: 0000000000000000 R15: ffff88006a5c3370
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201910] FS:  
00007f50a9445730(0000) GS:ffff8800280c7000(0000) knlGS:0000000000000000
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201974] CS:  e033 DS: 0000 
ES: 0000 CR0: 000000008005003b
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202032] CR2: 
00007fb35d12e6e8 CR3: 000000006a4ce000 CR4: 0000000000002660
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202093] DR0: 
0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202154] DR3: 
0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202436] Process tapdisk2 
(pid: 4988, threadinfo ffff88006a7f6000, task ffff88006b5a0000)
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202941] Stack:
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.203206]  ffff88006b5a0000 
0000000000000000 0000000000000000 0000000000000000
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.203609] <0> 
ffff88006a7f7e88 ffffffff812b9416 ffff88006a6c80f8 0000000100000000
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.204310] <0> 
00000000ffffffff ffff88006a5c3360 000000017edd7ab0 0000000000000000
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.205284] Call Trace:
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.205553]  
[<ffffffff812b9416>] blktap_ring_ioctl+0x183/0x2d8
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.205838]  
[<ffffffff81209a64>] ? inode_has_perm+0xa1/0xb3
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206120]  
[<ffffffff8157641f>] ? _spin_unlock+0x26/0x2a
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206400]  
[<ffffffff81126ff9>] ? aio_read_evt+0x56/0xe0
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206678]  
[<ffffffff81127071>] ? aio_read_evt+0xce/0xe0
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206957]  
[<ffffffff8124f5c1>] ? _raw_spin_lock+0x77/0x12d
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.207236]  
[<ffffffff81209bf8>] ? file_has_perm+0xb4/0xc6
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.207516]  
[<ffffffff8110464e>] vfs_ioctl+0x5e/0x77
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.207793]  
[<ffffffff81104b63>] do_vfs_ioctl+0x484/0x4d5
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.208069]  
[<ffffffff81104c0b>] sys_ioctl+0x57/0x7a
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.208346]  
[<ffffffff81012cc2>] system_call_fastpath+0x16/0x1b
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.208621] Code: 89 de 4c 89 
ef e8 60 f4 ff ff 49 8b 44 24 40 48 8b b8 90 04 00 00 e8 41 c9 2b 00 44 
89 f6 4c 89 e7 e8 39 fc ff ff 49 8b 44 24 40 <48> 8b b8 90 04 00 00 e8 
66 c7 2b 00 5b 41 5c 41 5d 41 5e c9 c3
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.211986] RIP  
[<ffffffff812b9c24>] blktap_device_end_request+0x49/0x5e
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.212306]  RSP <ffff88006a7f7cd8>
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.212579] ---[ end trace 
b97070122f44735d ]---
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: Created 
/dev/xen/blktap-2/blktap1 device
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: Created 
/dev/xen/blktap-2/tapdev1 device
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: new interface: ring: 251, 
device: 253, minor: 1
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: I/O queue driver: lio
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: block-aio 
open('/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda')
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: 
open(/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda) 
with O_DIRECT
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: Image size:       pre 
sector_shift  [10737418240]         post sector_shift [20971520]
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: opened image 
/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda (1 
users, state: 0x00000001, type: 0)
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: VBD CHAIN:
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: 
/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda: 0
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.317931] block tdb: 
sector-size: 512 capacity: 20971520

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36
  2011-04-13 21:02 Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36 Gerd Jakobovitsch
@ 2011-04-14 13:15 ` Konrad Rzeszutek Wilk
  2011-04-14 16:38   ` Daniel Stodden
  2011-04-15 14:21   ` xen 4.0.2rc3/kernel 2.6.32.36: BUG: unable to handle kernel paging request Gerd Jakobovitsch
  0 siblings, 2 replies; 11+ messages in thread
From: Konrad Rzeszutek Wilk @ 2011-04-14 13:15 UTC (permalink / raw)
  To: Gerd Jakobovitsch, daniel.stodden; +Cc: xen-devel

On Wed, Apr 13, 2011 at 06:02:13PM -0300, Gerd Jakobovitsch wrote:
> I'm trying to run several VMs (linux hvm, with tapdisk:aio disks at
> a storage over nfs) on a CentOS system, using the up-to-date version
> of xen 4.0 / kernel pvops 2.6.32.x stable. With a configuration
> without (most of) debug activated, I can start several instances -
> I'm running 7 of them - but shortly afterwards the system stops
> responding. I can't find any information on this.

First time I see it.
> 
> Activating several debug configuration items, among them
> DEBUG_PAGEALLOC, I get an exception as soon as I try to start up a
> VM. The system reboots.

Oooh, and is the log below from that situation?

Daniel, any thoughs?
> 
> Below the log from /var/log/messages:
> 
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Created
> /dev/xen/blktap-2/control device
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Created
> /dev/xen/blktap-2/blktap0 device
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Created
> /dev/xen/blktap-2/tapdev0 device
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: new interface: ring:
> 251, device: 253, minor: 0
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: I/O queue driver: lio
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: block-aio open('/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda')
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: open(/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda)
> with O_DIRECT
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Image size:       pre
> sector_shift  [134217728]   post sector_shift [262144]
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: opened image
> /storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda (1
> users, state: 0x00000001, type: 0)
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: VBD CHAIN:
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]:
> /storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda: 0
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.158549] block tda:
> sector-size: 512 capacity: 262144
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200514] general
> protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200703] last sysfs
> file: /sys/block/tda/removable
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200761] CPU 0
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200847] Modules linked
> in: bridge stp bonding bnx2i libiscsi scsi_transport_iscsi cnic uio
> bnx2 megaraid_sas
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201363] Pid: 4988,
> comm: tapdisk2 Not tainted 2.6.32.36 #3 PowerEdge M610
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201424] RIP:
> e030:[<ffffffff812b9c24>]  [<ffffffff812b9c24>]
> blktap_device_end_request+0x49/0x5e
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201543] RSP:
> e02b:ffff88006a7f7cd8  EFLAGS: 00010046
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201600] RAX:
> 6b6b6b6b6b6b6b6b RBX: ffff88006a6fc000 RCX: ffff88006a7f7c38
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201662] RDX:
> 0000000000000000 RSI: 0000000000000000 RDI: ffff88006a5c3500
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201723] RBP:
> ffff88006a7f7cf8 R08: ffffffff818383c0 R09: ffff88006a7f7c38
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201784] R10:
> 0000000000000000 R11: ffff88007b697b18 R12: ffff88007b697b18
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201845] R13:
> ffff88006a5c3360 R14: 0000000000000000 R15: ffff88006a5c3370
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201910] FS:
> 00007f50a9445730(0000) GS:ffff8800280c7000(0000)
> knlGS:0000000000000000
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201974] CS:  e033 DS:
> 0000 ES: 0000 CR0: 000000008005003b
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202032] CR2:
> 00007fb35d12e6e8 CR3: 000000006a4ce000 CR4: 0000000000002660
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202093] DR0:
> 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202154] DR3:
> 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202436] Process
> tapdisk2 (pid: 4988, threadinfo ffff88006a7f6000, task
> ffff88006b5a0000)
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202941] Stack:
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.203206]
> ffff88006b5a0000 0000000000000000 0000000000000000 0000000000000000
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.203609] <0>
> ffff88006a7f7e88 ffffffff812b9416 ffff88006a6c80f8 0000000100000000
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.204310] <0>
> 00000000ffffffff ffff88006a5c3360 000000017edd7ab0 0000000000000000
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.205284] Call Trace:
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.205553]
> [<ffffffff812b9416>] blktap_ring_ioctl+0x183/0x2d8
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.205838]
> [<ffffffff81209a64>] ? inode_has_perm+0xa1/0xb3
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206120]
> [<ffffffff8157641f>] ? _spin_unlock+0x26/0x2a
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206400]
> [<ffffffff81126ff9>] ? aio_read_evt+0x56/0xe0
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206678]
> [<ffffffff81127071>] ? aio_read_evt+0xce/0xe0
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206957]
> [<ffffffff8124f5c1>] ? _raw_spin_lock+0x77/0x12d
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.207236]
> [<ffffffff81209bf8>] ? file_has_perm+0xb4/0xc6
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.207516]
> [<ffffffff8110464e>] vfs_ioctl+0x5e/0x77
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.207793]
> [<ffffffff81104b63>] do_vfs_ioctl+0x484/0x4d5
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.208069]
> [<ffffffff81104c0b>] sys_ioctl+0x57/0x7a
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.208346]
> [<ffffffff81012cc2>] system_call_fastpath+0x16/0x1b
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.208621] Code: 89 de 4c
> 89 ef e8 60 f4 ff ff 49 8b 44 24 40 48 8b b8 90 04 00 00 e8 41 c9 2b
> 00 44 89 f6 4c 89 e7 e8 39 fc ff ff 49 8b 44 24 40 <48> 8b b8 90 04
> 00 00 e8 66 c7 2b 00 5b 41 5c 41 5d 41 5e c9 c3
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.211986] RIP
> [<ffffffff812b9c24>] blktap_device_end_request+0x49/0x5e
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.212306]  RSP <ffff88006a7f7cd8>
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.212579] ---[ end trace
> b97070122f44735d ]---
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: Created
> /dev/xen/blktap-2/blktap1 device
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: Created
> /dev/xen/blktap-2/tapdev1 device
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: new interface: ring:
> 251, device: 253, minor: 1
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: I/O queue driver: lio
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: block-aio open('/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda')
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: open(/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda)
> with O_DIRECT
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: Image size:       pre
> sector_shift  [10737418240]         post sector_shift [20971520]
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: opened image
> /storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda (1
> users, state: 0x00000001, type: 0)
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: VBD CHAIN:
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]:
> /storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda: 0
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.317931] block tdb:
> sector-size: 512 capacity: 20971520
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36
  2011-04-14 13:15 ` Konrad Rzeszutek Wilk
@ 2011-04-14 16:38   ` Daniel Stodden
  2011-04-14 17:32     ` Daniel Stodden
  2011-04-14 17:43     ` Daniel Stodden
  2011-04-15 14:21   ` xen 4.0.2rc3/kernel 2.6.32.36: BUG: unable to handle kernel paging request Gerd Jakobovitsch
  1 sibling, 2 replies; 11+ messages in thread
From: Daniel Stodden @ 2011-04-14 16:38 UTC (permalink / raw)
  To: Konrad Rzeszutek Wilk; +Cc: Gerd Jakobovitsch, xen-devel@lists.xensource.com

On Thu, 2011-04-14 at 09:15 -0400, Konrad Rzeszutek Wilk wrote:
> On Wed, Apr 13, 2011 at 06:02:13PM -0300, Gerd Jakobovitsch wrote:
> > I'm trying to run several VMs (linux hvm, with tapdisk:aio disks at
> > a storage over nfs) on a CentOS system, using the up-to-date version
> > of xen 4.0 / kernel pvops 2.6.32.x stable. With a configuration
> > without (most of) debug activated, I can start several instances -
> > I'm running 7 of them - but shortly afterwards the system stops
> > responding. I can't find any information on this.
> 
> First time I see it.
> > 
> > Activating several debug configuration items, among them
> > DEBUG_PAGEALLOC, I get an exception as soon as I try to start up a
> > VM. The system reboots.
> 
> Oooh, and is the log below from that situation?
> 
> Daniel, any thoughs?

---
	  Unmap pages from the kernel linear mapping after free_pages().
	  This results in a large slowdown, but helps to find certain types
	  of memory corruption.

Stunning. Our I/O page allocator is a sort of twisted mempool. Unless
the allocation is explicitly modified in sysfs/, everything should stay
pinned. We might be just tripping over debug code alone, but I didn't
figure it out yet.

Daniel

> > 
> > Below the log from /var/log/messages:
> > 
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Created
> > /dev/xen/blktap-2/control device
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Created
> > /dev/xen/blktap-2/blktap0 device
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Created
> > /dev/xen/blktap-2/tapdev0 device
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: new interface: ring:
> > 251, device: 253, minor: 0
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: I/O queue driver: lio
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: block-aio open('/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda')
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: open(/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda)
> > with O_DIRECT
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Image size:       pre
> > sector_shift  [134217728]   post sector_shift [262144]
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: opened image
> > /storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda (1
> > users, state: 0x00000001, type: 0)
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: VBD CHAIN:
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]:
> > /storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda: 0
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.158549] block tda:
> > sector-size: 512 capacity: 262144
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200514] general
> > protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200703] last sysfs
> > file: /sys/block/tda/removable
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200761] CPU 0
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200847] Modules linked
> > in: bridge stp bonding bnx2i libiscsi scsi_transport_iscsi cnic uio
> > bnx2 megaraid_sas
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201363] Pid: 4988,
> > comm: tapdisk2 Not tainted 2.6.32.36 #3 PowerEdge M610
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201424] RIP:
> > e030:[<ffffffff812b9c24>]  [<ffffffff812b9c24>]
> > blktap_device_end_request+0x49/0x5e
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201543] RSP:
> > e02b:ffff88006a7f7cd8  EFLAGS: 00010046
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201600] RAX:
> > 6b6b6b6b6b6b6b6b RBX: ffff88006a6fc000 RCX: ffff88006a7f7c38
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201662] RDX:
> > 0000000000000000 RSI: 0000000000000000 RDI: ffff88006a5c3500
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201723] RBP:
> > ffff88006a7f7cf8 R08: ffffffff818383c0 R09: ffff88006a7f7c38
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201784] R10:
> > 0000000000000000 R11: ffff88007b697b18 R12: ffff88007b697b18
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201845] R13:
> > ffff88006a5c3360 R14: 0000000000000000 R15: ffff88006a5c3370
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201910] FS:
> > 00007f50a9445730(0000) GS:ffff8800280c7000(0000)
> > knlGS:0000000000000000
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201974] CS:  e033 DS:
> > 0000 ES: 0000 CR0: 000000008005003b
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202032] CR2:
> > 00007fb35d12e6e8 CR3: 000000006a4ce000 CR4: 0000000000002660
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202093] DR0:
> > 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202154] DR3:
> > 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202436] Process
> > tapdisk2 (pid: 4988, threadinfo ffff88006a7f6000, task
> > ffff88006b5a0000)
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202941] Stack:
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.203206]
> > ffff88006b5a0000 0000000000000000 0000000000000000 0000000000000000
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.203609] <0>
> > ffff88006a7f7e88 ffffffff812b9416 ffff88006a6c80f8 0000000100000000
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.204310] <0>
> > 00000000ffffffff ffff88006a5c3360 000000017edd7ab0 0000000000000000
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.205284] Call Trace:
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.205553]
> > [<ffffffff812b9416>] blktap_ring_ioctl+0x183/0x2d8
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.205838]
> > [<ffffffff81209a64>] ? inode_has_perm+0xa1/0xb3
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206120]
> > [<ffffffff8157641f>] ? _spin_unlock+0x26/0x2a
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206400]
> > [<ffffffff81126ff9>] ? aio_read_evt+0x56/0xe0
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206678]
> > [<ffffffff81127071>] ? aio_read_evt+0xce/0xe0
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206957]
> > [<ffffffff8124f5c1>] ? _raw_spin_lock+0x77/0x12d
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.207236]
> > [<ffffffff81209bf8>] ? file_has_perm+0xb4/0xc6
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.207516]
> > [<ffffffff8110464e>] vfs_ioctl+0x5e/0x77
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.207793]
> > [<ffffffff81104b63>] do_vfs_ioctl+0x484/0x4d5
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.208069]
> > [<ffffffff81104c0b>] sys_ioctl+0x57/0x7a
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.208346]
> > [<ffffffff81012cc2>] system_call_fastpath+0x16/0x1b
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.208621] Code: 89 de 4c
> > 89 ef e8 60 f4 ff ff 49 8b 44 24 40 48 8b b8 90 04 00 00 e8 41 c9 2b
> > 00 44 89 f6 4c 89 e7 e8 39 fc ff ff 49 8b 44 24 40 <48> 8b b8 90 04
> > 00 00 e8 66 c7 2b 00 5b 41 5c 41 5d 41 5e c9 c3
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.211986] RIP
> > [<ffffffff812b9c24>] blktap_device_end_request+0x49/0x5e
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.212306]  RSP <ffff88006a7f7cd8>
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.212579] ---[ end trace
> > b97070122f44735d ]---
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: Created
> > /dev/xen/blktap-2/blktap1 device
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: Created
> > /dev/xen/blktap-2/tapdev1 device
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: new interface: ring:
> > 251, device: 253, minor: 1
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: I/O queue driver: lio
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: block-aio open('/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda')
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: open(/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda)
> > with O_DIRECT
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: Image size:       pre
> > sector_shift  [10737418240]         post sector_shift [20971520]
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: opened image
> > /storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda (1
> > users, state: 0x00000001, type: 0)
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: VBD CHAIN:
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]:
> > /storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda: 0
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.317931] block tdb:
> > sector-size: 512 capacity: 20971520
> > 
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36
  2011-04-14 16:38   ` Daniel Stodden
@ 2011-04-14 17:32     ` Daniel Stodden
  2011-04-14 17:43     ` Daniel Stodden
  1 sibling, 0 replies; 11+ messages in thread
From: Daniel Stodden @ 2011-04-14 17:32 UTC (permalink / raw)
  To: Konrad Rzeszutek Wilk; +Cc: Gerd Jakobovitsch, xen-devel@lists.xensource.com

On Thu, 2011-04-14 at 12:38 -0400, Daniel Stodden wrote:
> On Thu, 2011-04-14 at 09:15 -0400, Konrad Rzeszutek Wilk wrote:
> > On Wed, Apr 13, 2011 at 06:02:13PM -0300, Gerd Jakobovitsch wrote:
> > > I'm trying to run several VMs (linux hvm, with tapdisk:aio disks at
> > > a storage over nfs) on a CentOS system, using the up-to-date version
> > > of xen 4.0 / kernel pvops 2.6.32.x stable. With a configuration
> > > without (most of) debug activated, I can start several instances -
> > > I'm running 7 of them - but shortly afterwards the system stops
> > > responding. I can't find any information on this.
> > 
> > First time I see it.
> > > 
> > > Activating several debug configuration items, among them
> > > DEBUG_PAGEALLOC, I get an exception as soon as I try to start up a
> > > VM. The system reboots.
> > 
> > Oooh, and is the log below from that situation?
> > 
> > Daniel, any thoughs?
> 
> ---
> 	  Unmap pages from the kernel linear mapping after free_pages().
> 	  This results in a large slowdown, but helps to find certain types
> 	  of memory corruption.
> 
> Stunning. Our I/O page allocator is a sort of twisted mempool. Unless
> the allocation is explicitly modified in sysfs/, everything should stay
> pinned. We might be just tripping over debug code alone, but I didn't
> figure it out yet.

Stuff worked fine with 2.6.32.32, but gets immediately reproducible
with .32.36.

Daniel

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36
  2011-04-14 16:38   ` Daniel Stodden
  2011-04-14 17:32     ` Daniel Stodden
@ 2011-04-14 17:43     ` Daniel Stodden
  2011-04-14 18:05       ` Gerd Jakobovitsch
  1 sibling, 1 reply; 11+ messages in thread
From: Daniel Stodden @ 2011-04-14 17:43 UTC (permalink / raw)
  To: Konrad Rzeszutek Wilk; +Cc: Gerd Jakobovitsch, xen-devel@lists.xensource.com

On Thu, 2011-04-14 at 12:38 -0400, Daniel Stodden wrote:
> On Thu, 2011-04-14 at 09:15 -0400, Konrad Rzeszutek Wilk wrote:
> > On Wed, Apr 13, 2011 at 06:02:13PM -0300, Gerd Jakobovitsch wrote:
> > > I'm trying to run several VMs (linux hvm, with tapdisk:aio disks at
> > > a storage over nfs) on a CentOS system, using the up-to-date version
> > > of xen 4.0 / kernel pvops 2.6.32.x stable. With a configuration
> > > without (most of) debug activated, I can start several instances -
> > > I'm running 7 of them - but shortly afterwards the system stops
> > > responding. I can't find any information on this.
> > 
> > First time I see it.
> > > 
> > > Activating several debug configuration items, among them
> > > DEBUG_PAGEALLOC, I get an exception as soon as I try to start up a
> > > VM. The system reboots.
> > 
> > Oooh, and is the log below from that situation?
> > 
> > Daniel, any thoughs?
> 
> ---
> 	  Unmap pages from the kernel linear mapping after free_pages().
> 	  This results in a large slowdown, but helps to find certain types
> 	  of memory corruption.
> 
> Stunning. Our I/O page allocator is a sort of twisted mempool. Unless
> the allocation is explicitly modified in sysfs/, everything should stay
> pinned. We might be just tripping over debug code alone, but I didn't
> figure it out yet.

Ah, that's just missing Dominic's spinlock fix. 

http://xenbits.xen.org/gitweb/?p=people/dstodden/linux.git;a=commit;h=a765257af7e28c41bd776c3e03615539597eb592

Daniel

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36
  2011-04-14 17:43     ` Daniel Stodden
@ 2011-04-14 18:05       ` Gerd Jakobovitsch
  2011-04-14 23:42         ` Daniel Stodden
  0 siblings, 1 reply; 11+ messages in thread
From: Gerd Jakobovitsch @ 2011-04-14 18:05 UTC (permalink / raw)
  To: Daniel Stodden; +Cc: xen-devel@lists.xensource.com, Konrad Rzeszutek Wilk

Hello Daniel:

I applied the patch and the bug at VM startup was solved. Thank you for 
your help.

Regards

Gerd

On 04/14/2011 02:43 PM, Daniel Stodden wrote:
> On Thu, 2011-04-14 at 12:38 -0400, Daniel Stodden wrote:
>> On Thu, 2011-04-14 at 09:15 -0400, Konrad Rzeszutek Wilk wrote:
>>> On Wed, Apr 13, 2011 at 06:02:13PM -0300, Gerd Jakobovitsch wrote:
>>>> I'm trying to run several VMs (linux hvm, with tapdisk:aio disks at
>>>> a storage over nfs) on a CentOS system, using the up-to-date version
>>>> of xen 4.0 / kernel pvops 2.6.32.x stable. With a configuration
>>>> without (most of) debug activated, I can start several instances -
>>>> I'm running 7 of them - but shortly afterwards the system stops
>>>> responding. I can't find any information on this.
>>> First time I see it.
>>>> Activating several debug configuration items, among them
>>>> DEBUG_PAGEALLOC, I get an exception as soon as I try to start up a
>>>> VM. The system reboots.
>>> Oooh, and is the log below from that situation?
>>>
>>> Daniel, any thoughs?
>> ---
>> 	  Unmap pages from the kernel linear mapping after free_pages().
>> 	  This results in a large slowdown, but helps to find certain types
>> 	  of memory corruption.
>>
>> Stunning. Our I/O page allocator is a sort of twisted mempool. Unless
>> the allocation is explicitly modified in sysfs/, everything should stay
>> pinned. We might be just tripping over debug code alone, but I didn't
>> figure it out yet.
> Ah, that's just missing Dominic's spinlock fix.
>
> http://xenbits.xen.org/gitweb/?p=people/dstodden/linux.git;a=commit;h=a765257af7e28c41bd776c3e03615539597eb592
>
> Daniel
>

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36
  2011-04-14 18:05       ` Gerd Jakobovitsch
@ 2011-04-14 23:42         ` Daniel Stodden
  0 siblings, 0 replies; 11+ messages in thread
From: Daniel Stodden @ 2011-04-14 23:42 UTC (permalink / raw)
  To: Jeremy Fitzhardinge
  Cc: Jakobovitsch, xen-devel@lists.xensource.com, Gerd,
	Konrad Rzeszutek Wilk

Jeremy,

it wants a manual patch into drivers/xen/blktap/device.c because it's
rolled into that combo of renaming plus overall feature upgrades I have
in that series. 

It's the only pending bugfix, quite an obvious one actually. It's been
rare enough unless provoked like Gerd did, but we found it first in XCP
so it actually tends to happen.

Can you just pick it, and push through to stable straight away?

On Thu, 2011-04-14 at 14:05 -0400, Gerd Jakobovitsch wrote:
> Hello Daniel:
> 
> I applied the patch and the bug at VM startup was solved. Thank you for 
> your help.
> 
> Regards
> 
> Gerd

> > Ah, that's just missing Dominic's spinlock fix.
> >
> > http://xenbits.xen.org/gitweb/?p=people/dstodden/linux.git;a=commit;h=a765257af7e28c41bd776c3e03615539597eb592

Thanks very much.

Daniel

^ permalink raw reply	[flat|nested] 11+ messages in thread

* xen 4.0.2rc3/kernel 2.6.32.36: BUG: unable to handle kernel paging request
  2011-04-14 13:15 ` Konrad Rzeszutek Wilk
  2011-04-14 16:38   ` Daniel Stodden
@ 2011-04-15 14:21   ` Gerd Jakobovitsch
  2011-04-15 15:44     ` Pasi Kärkkäinen
  1 sibling, 1 reply; 11+ messages in thread
From: Gerd Jakobovitsch @ 2011-04-15 14:21 UTC (permalink / raw)
  To: xen-devel@lists.xensource.com

[-- Attachment #1: Type: text/plain, Size: 4859 bytes --]

Reporting a new bug that appeared during stress tests. The scenario is 
the same as reported below, with patches applied:

On 04/14/2011 10:15 AM, Konrad Rzeszutek Wilk wrote:
> On Wed, Apr 13, 2011 at 06:02:13PM -0300, Gerd Jakobovitsch wrote:
>> I'm trying to run several VMs (linux hvm, with tapdisk:aio disks at
>> a storage over nfs) on a CentOS system, using the up-to-date version
>> of xen 4.0 / kernel pvops 2.6.32.x stable. With a configuration
>> without (most of) debug activated, I can start several instances -
>> I'm running 7 of them - but shortly afterwards the system stops
>> responding. I can't find any information on this.
> First time I see it.
>> Activating several debug configuration items, among them
>> DEBUG_PAGEALLOC, I get an exception as soon as I try to start up a
>> VM. The system reboots.
>>
With the debug information still set, I'm running 42 VMs - mixed Linux 
(several distros) and Windows, most of them running benchmarks for CPU 
and disk usage. After roughly 15 hours, a bug message appeared at dmesg. 
It affected xm commands - it seems to be related to a specific VM - but 
xl commands still work. VMs are running.

# xm list
Error: (5, 'Input/output error, while reading 
/local/domain/33/console/vnc-port')
Usage: xm list [options] [Domain, ...]

After killing the VM that reported error, xm commands are working again.

The BUG message at dmesg:

[66007.135552] BUG: unable to handle kernel paging request at 
ffff8800004ca458
[66007.135567] IP: [<ffffffff8100d4ae>] xen_set_pte+0x3e/0x4b
[66007.135580] PGD 1002067 PUD 1006067 PMD 2d78067 PTE 100000004ca025
[66007.135675] Oops: 0003 [#1] SMP DEBUG_PAGEALLOC
[66007.135686] last sysfs file: 
/sys/class/net/virtbr/bridge/topology_change_detected
[66007.135693] CPU 4
[66007.135698] Modules linked in: arptable_filter arp_tables bridge stp 
bonding bnx2i libiscsi scsi_transport_iscsi cnic uio bnx2 megaraid_sas
[66007.135729] Pid: 683, comm: pageattr-test Not tainted 2.6.32.36 #7 
PowerEdge M610
[66007.135735] RIP: e030:[<ffffffff8100d4ae>]  [<ffffffff8100d4ae>] 
xen_set_pte+0x3e/0x4b
[66007.135746] RSP: e02b:ffff88007c8edbb0  EFLAGS: 00010202
[66007.135751] RAX: 0000000000e32cb6 RBX: 0000000000e32cb6 RCX: 
0000000000000001
[66007.135757] RDX: 0000000000000000 RSI: 8010000800569267 RDI: 
ffff8800004ca458
[66007.135764] RBP: ffff88007c8edbd0 R08: 0000000000000001 R09: 
0000000000000000
[66007.135770] R10: ffffffff818385f8 R11: ffffffff818385e0 R12: 
8010000800569267
[66007.135776] R13: ffff8800004ca458 R14: 8010000416569067 R15: 
8010000800569267
[66007.135786] FS:  00007f0eeede66e0(0000) GS:ffff88002813f000(0000) 
knlGS:0000000000000000
[66007.135792] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[66007.135797] CR2: ffff8800004ca458 CR3: 000000007b663000 CR4: 
0000000000002660
[66007.135804] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[66007.135810] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[66007.135816] Process pageattr-test (pid: 683, threadinfo 
ffff88007c8ec000, task ffff88007e4ce480)
[66007.135822] Stack:
[66007.135825]  0000000000000000 8010000004569067 0000000000004569 
ffff88007c8edd20
[66007.135835] <0> ffff88007c8edbe0 ffffffff81034740 ffff88007c8edbf0 
ffffffff8103474d
[66007.135848] <0> ffff88007c8edcf0 ffffffff81034e77 000000017c8edc40 
ffffffff818385e0
[66007.135860] Call Trace:
[66007.135868]  [<ffffffff81034740>] set_pte+0x17/0x1b
[66007.135875]  [<ffffffff8103474d>] set_pte_atomic+0x9/0xb
[66007.135882]  [<ffffffff81034e77>] __change_page_attr_set_clr+0x186/0x82d
[66007.135936]  [<ffffffff8124f4a0>] ? _raw_spin_unlock+0xab/0xb1
[66007.135951]  [<ffffffff8157641f>] ? _spin_unlock+0x26/0x2a
[66007.135961]  [<ffffffff810e587d>] ? vm_unmap_aliases+0x151/0x160
[66007.135969]  [<ffffffff81035695>] change_page_attr_set_clr+0x177/0x360
[66007.135976]  [<ffffffff8103597a>] change_page_attr_set+0x27/0x29
[66007.135983]  [<ffffffff810348e2>] ? pte_flags+0x9/0x18
[66007.135990]  [<ffffffff81035c01>] do_pageattr_test+0x285/0x4b1
[66007.135998]  [<ffffffff8103597c>] ? do_pageattr_test+0x0/0x4b1
[66007.136097]  [<ffffffff8106a9c3>] kthread+0x69/0x71
[66007.136105]  [<ffffffff81013daa>] child_rip+0xa/0x20
[66007.136112]  [<ffffffff81012ee6>] ? int_ret_from_sys_call+0x7/0x1b
[66007.136119]  [<ffffffff81013726>] ? retint_restore_args+0x5/0x6
[66007.136127]  [<ffffffff81013da0>] ? child_rip+0x0/0x20
[66007.136131] Code: e8 3c ff ff ff ff 05 b6 5c 94 00 e8 31 ff ff ff 8b 
1d b3 5c 94 00 e8 a2 23 02 00 ff c8 0f 94 c0 0f b6 c0 01 d8 89 05 9e 5c 
94 00 <4d> 89 65 00 41 59 5b 41 5c 41 5d c9 c3 55 48 89 e5 53 89 fb 48
[66007.136273] RIP  [<ffffffff8100d4ae>] xen_set_pte+0x3e/0x4b
[66007.136281]  RSP <ffff88007c8edbb0>
[66007.136285] CR2: ffff8800004ca458
[66007.136574] ---[ end trace 4e200a271895cc90 ]---

Attached errors registered in xm dmesg and xend.log.

[-- Attachment #2: bug_paging_xend-log.txt --]
[-- Type: text/plain, Size: 5501 bytes --]

[2011-04-15 05:32:06 4415] ERROR (xmlrpclib2:181) Internal error handling xend.domain
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line 134, in _marshaled_dispatch
    response = self._dispatch(method, params)
  File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line 406, in _dispatch
    return func(*params)
  File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 58, in domain
    info = lookup(domid)
  File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 50, in lookup
    info = XendDomain.instance().domain_lookup(domid)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 543, in domain_lookup
    self._refresh(refresh_shutdown = False)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 417, in _refresh
    self._refreshTxn(txn, refresh_shutdown)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 430, in _refreshTxn
    self.domains[domid].update(dom, refresh_shutdown, transaction)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 3523, in update
    self._update_consoles(transaction)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1656, in _update_consoles
    vnc_port = self.readDomTxn(transaction, 'console/vnc-port')
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1720, in readDomTxn
    return transaction.read(*paths)
  File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 72, in read
    return self._read(args[0])
  File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 59, in _read
    '%s, while reading %s' % (ex.args[1], path))
RuntimeError: (5, 'Input/output error, while reading /local/domain/33/console/vnc-port')
[2011-04-15 05:32:12 4415] ERROR (xmlrpclib2:181) Internal error handling xend.domain
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line 134, in _marshaled_dispatch
    response = self._dispatch(method, params)
  File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line 406, in _dispatch
    return func(*params)
  File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 58, in domain
    info = lookup(domid)
  File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 50, in lookup
    info = XendDomain.instance().domain_lookup(domid)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 543, in domain_lookup
    self._refresh(refresh_shutdown = False)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 417, in _refresh
    self._refreshTxn(txn, refresh_shutdown)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 430, in _refreshTxn
    self.domains[domid].update(dom, refresh_shutdown, transaction)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 3523, in update
    self._update_consoles(transaction)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1656, in _update_consoles
    vnc_port = self.readDomTxn(transaction, 'console/vnc-port')
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1720, in readDomTxn
    return transaction.read(*paths)
  File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 72, in read
    return self._read(args[0])
  File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 59, in _read
    '%s, while reading %s' % (ex.args[1], path))
RuntimeError: (5, 'Input/output error, while reading /local/domain/33/console/vnc-port')
[2011-04-15 05:32:17 4415] ERROR (xmlrpclib2:181) Internal error handling xend.domain
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line 134, in _marshaled_dispatch
    response = self._dispatch(method, params)
  File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line 406, in _dispatch
    return func(*params)
  File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 58, in domain
    info = lookup(domid)
  File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 50, in lookup
    info = XendDomain.instance().domain_lookup(domid)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 543, in domain_lookup
    self._refresh(refresh_shutdown = False)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 417, in _refresh
    self._refreshTxn(txn, refresh_shutdown)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 430, in _refreshTxn
    self.domains[domid].update(dom, refresh_shutdown, transaction)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 3523, in update
    self._update_consoles(transaction)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1656, in _update_consoles
    vnc_port = self.readDomTxn(transaction, 'console/vnc-port')
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1720, in readDomTxn
    return transaction.read(*paths)
  File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 72, in read
    return self._read(args[0])
  File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 59, in _read
    '%s, while reading %s' % (ex.args[1], path))


[-- Attachment #3: bug_paging_xm_dmesg.txt --]
[-- Type: text/plain, Size: 6306 bytes --]

(XEN) HVM43: Booting from 0000:7c00
(XEN) HVM44: Press F12 for boot menu.
(XEN) HVM44: 
(XEN) HVM44: Booting from Hard Disk...
(XEN) HVM44: Booting from 0000:7c00
(XEN) HVM44: int13_harddisk: function 42. LBA out of range
(XEN) stdvga.c:151:d43 leaving stdvga
(XEN) stdvga.c:147:d43 entering stdvga and caching modes
(XEN) irq.c:243: Dom43 PCI link 0 changed 5 -> 0
(XEN) irq.c:243: Dom43 PCI link 1 changed 10 -> 0
(XEN) irq.c:243: Dom43 PCI link 2 changed 11 -> 0
(XEN) irq.c:243: Dom43 PCI link 3 changed 5 -> 0
(XEN) stdvga.c:151:d43 leaving stdvga
(XEN) stdvga.c:151:d42 leaving stdvga
(XEN) mm.c:3617:d0 PTE entry 81244b for address ffff88007caa0000 doesn't match frame 621c4b
(XEN) mm.c:3617:d0 PTE entry 812bed for address ffff88007caa1000 doesn't match frame 6217ed
(XEN) mm.c:3617:d0 PTE entry 812426 for address ffff88007caa2000 doesn't match frame 621c26
(XEN) mm.c:3617:d0 PTE entry 812590 for address ffff88007caa3000 doesn't match frame 621d90
(XEN) mm.c:3617:d0 PTE entry 812591 for address ffff88007caa4000 doesn't match frame 621d91
(XEN) mm.c:3617:d0 PTE entry 812492 for address ffff88007caa5000 doesn't match frame 621c92
(XEN) mm.c:3617:d0 PTE entry 812493 for address ffff88007caa6000 doesn't match frame 621c93
(XEN) mm.c:3617:d0 PTE entry 8124ca for address ffff88007caa7000 doesn't match frame 621cca
(XEN) mm.c:3617:d0 PTE entry 8124cb for address ffff88007caa8000 doesn't match frame 621ccb
(XEN) mm.c:3617:d0 PTE entry 80e8b0 for address ffff88007caa9000 doesn't match frame 6258b0
(XEN) mm.c:3617:d0 PTE entry 80e8b1 for address ffff88007caaa000 doesn't match frame 6258b1
(XEN) mm.c:3617:d0 PTE entry 803fe2 for address ffff88007caab000 doesn't match frame 5cfde2
(XEN) mm.c:3617:d0 PTE entry 80b3fd for address ffff88007ca99000 doesn't match frame 46b1fd
(XEN) mm.c:3617:d0 PTE entry 80b3fe for address ffff88007ca9a000 doesn't match frame 46b1fe
(XEN) mm.c:3617:d0 PTE entry 80ac10 for address ffff88007ca9b000 doesn't match frame 46b610
(XEN) mm.c:3617:d0 PTE entry 80c01e for address ffff88007ca9c000 doesn't match frame 46a21e
(XEN) mm.c:3617:d0 PTE entry 80ab16 for address ffff88007ca9d000 doesn't match frame 46b916
(XEN) mm.c:3617:d0 PTE entry 80b0f6 for address ffff88007ca9e000 doesn't match frame 46b2f6
(XEN) mm.c:3617:d0 PTE entry 809c8c for address ffff88007ca9f000 doesn't match frame 46c68c
(XEN) mm.c:3617:d0 PTE entry 8017e0 for address ffff88007cb9f000 doesn't match frame 5b19e0
(XEN) mm.c:3617:d0 PTE entry 8017e1 for address ffff88007cba0000 doesn't match frame 5b19e1
(XEN) mm.c:3617:d0 PTE entry 801fc4 for address ffff88007cba1000 doesn't match frame 5b11c4
(XEN) mm.c:3617:d0 PTE entry 801fc5 for address ffff88007cba2000 doesn't match frame 5b11c5
(XEN) mm.c:3617:d0 PTE entry 801f2a for address ffff88007cba3000 doesn't match frame 5b112a
(XEN) mm.c:3617:d0 PTE entry 801f2b for address ffff88007cba4000 doesn't match frame 5b112b
(XEN) mm.c:3617:d0 PTE entry 801ee8 for address ffff88007cba5000 doesn't match frame 5b10e8
(XEN) mm.c:3617:d0 PTE entry 801ee9 for address ffff88007cba6000 doesn't match frame 5b10e9
(XEN) mm.c:3617:d0 PTE entry 803fde for address ffff88007cba7000 doesn't match frame 5af1de
(XEN) mm.c:3617:d0 PTE entry 8120ec for address ffff88007cb9e000 doesn't match frame 9660ec
(XEN) mm.c:3617:d0 PTE entry 8120ed for address ffff88007cb9f000 doesn't match frame 9660ed
(XEN) mm.c:3617:d0 PTE entry 810e0a for address ffff88007cba0000 doesn't match frame 96720a
(XEN) mm.c:3617:d0 PTE entry 810e0b for address ffff88007cba1000 doesn't match frame 96720b
(XEN) mm.c:3617:d0 PTE entry 81080e for address ffff88007cba2000 doesn't match frame 96780e
(XEN) mm.c:3617:d0 PTE entry 81080f for address ffff88007cba3000 doesn't match frame 96780f
(XEN) mm.c:3617:d0 PTE entry 80ef3e for address ffff88007cba4000 doesn't match frame 96933e
(XEN) mm.c:3617:d0 PTE entry 802fd7 for address ffff88007c433000 doesn't match frame 2111d7
(XEN) mm.c:3617:d0 PTE entry 805131 for address ffff88007c434000 doesn't match frame 20ef31
(XEN) mm.c:3617:d0 PTE entry 80c1ea for address ffff88007c435000 doesn't match frame 207fea
(XEN) mm.c:3617:d0 PTE entry 814ae9 for address ffff88007c9b0000 doesn't match frame 8866e9
(XEN) mm.c:3617:d0 PTE entry 8113b9 for address ffff88007ca20000 doesn't match frame 9277b9
(XEN) mm.c:3617:d0 PTE entry 808ab2 for address ffff88007ca21000 doesn't match frame 92feb2
(XEN) mm.c:3617:d0 PTE entry 803bb3 for address ffff88007ca22000 doesn't match frame 934fb3
(XEN) mm.c:3617:d0 PTE entry 814747 for address ffff88007ca23000 doesn't match frame 924347
(XEN) mm.c:3617:d0 PTE entry 819873 for address ffff88007ca24000 doesn't match frame 91f073
(XEN) mm.c:3617:d0 PTE entry 80302c for address ffff88007ca25000 doesn't match frame 93582c
(XEN) mm.c:3617:d0 PTE entry 81b75c for address ffff88007ca26000 doesn't match frame 91d35c
(XEN) mm.c:3617:d0 PTE entry c5fd0b for address ffff88007ca86000 doesn't match frame 95830b
(XEN) mm.c:3617:d0 PTE entry 81a093 for address ffff88007ca87000 doesn't match frame 9bde93
(XEN) mm.c:3617:d0 PTE entry 815f11 for address ffff88007ca88000 doesn't match frame 942111
(XEN) mm.c:3617:d0 PTE entry 815bc5 for address ffff88007ca89000 doesn't match frame 9425c5
(XEN) mm.c:3617:d0 PTE entry 816875 for address ffff88007cb9b000 doesn't match frame 59c675
(XEN) mm.c:3617:d0 PTE entry 8158a8 for address ffff88007cb9c000 doesn't match frame 59d6a8
(XEN) mm.c:3617:d0 PTE entry 82fa1c for address ffff88007c926000 doesn't match frame 5e321c
(XEN) mm.c:3617:d0 PTE entry 82655e for address ffff88007c927000 doesn't match frame 5ec95e
(XEN) mm.c:3617:d0 PTE entry c5ce72 for address ffff88007c926000 doesn't match frame 95b072
(XEN) mm.c:3617:d0 PTE entry c5cea9 for address ffff88007c927000 doesn't match frame 95b0a9
(XEN) mm.c:2399:d0 Bad type (saw 1c00000000000002 != exp 7000000000000000) for mfn 800569 (pfn 4569)
(XEN) mm.c:878:d0 Error getting mfn 800569 (pfn 4569) from L1 entry 8010000800569267 for l1e_owner=0, pg_owner=0
(XEN) mm.c:4664:d0 ptwr_emulate: could not get_page_from_l1e()
(XEN) mm.c:3617:d0 PTE entry 80e284 for address ffff88007ca89000 doesn't match frame 468084
(XEN) mm.c:3617:d0 PTE entry 817918 for address ffff88007ca8a000 doesn't match frame 47f118
(XEN) common.c:3722:d0 tracking VRAM f0000 - f0160
(XEN) common.c:3722:d0 tracking VRAM f0000 - f0160

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: xen 4.0.2rc3/kernel 2.6.32.36: BUG: unable to handle kernel paging request
  2011-04-15 14:21   ` xen 4.0.2rc3/kernel 2.6.32.36: BUG: unable to handle kernel paging request Gerd Jakobovitsch
@ 2011-04-15 15:44     ` Pasi Kärkkäinen
  2011-04-15 15:47       ` Gerd Jakobovitsch
  0 siblings, 1 reply; 11+ messages in thread
From: Pasi Kärkkäinen @ 2011-04-15 15:44 UTC (permalink / raw)
  To: Gerd Jakobovitsch; +Cc: xen-devel@lists.xensource.com

On Fri, Apr 15, 2011 at 11:21:40AM -0300, Gerd Jakobovitsch wrote:
> Reporting a new bug that appeared during stress tests. The scenario is  
> the same as reported below, with patches applied:
>

Hello,

Can you clarify which exact kernel version/changeset you used for this test?

Does your kernel include the latest bugfix (from 3 days ago)?
http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=ae333e97552c81ab10395ad1ffc6d6daaadb144a

changelog:
http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=xen/stable-2.6.32.x

-- Pasi


> On 04/14/2011 10:15 AM, Konrad Rzeszutek Wilk wrote:
>> On Wed, Apr 13, 2011 at 06:02:13PM -0300, Gerd Jakobovitsch wrote:
>>> I'm trying to run several VMs (linux hvm, with tapdisk:aio disks at
>>> a storage over nfs) on a CentOS system, using the up-to-date version
>>> of xen 4.0 / kernel pvops 2.6.32.x stable. With a configuration
>>> without (most of) debug activated, I can start several instances -
>>> I'm running 7 of them - but shortly afterwards the system stops
>>> responding. I can't find any information on this.
>> First time I see it.
>>> Activating several debug configuration items, among them
>>> DEBUG_PAGEALLOC, I get an exception as soon as I try to start up a
>>> VM. The system reboots.
>>>
> With the debug information still set, I'm running 42 VMs - mixed Linux  
> (several distros) and Windows, most of them running benchmarks for CPU  
> and disk usage. After roughly 15 hours, a bug message appeared at dmesg.  
> It affected xm commands - it seems to be related to a specific VM - but  
> xl commands still work. VMs are running.
>
> # xm list
> Error: (5, 'Input/output error, while reading  
> /local/domain/33/console/vnc-port')
> Usage: xm list [options] [Domain, ...]
>
> After killing the VM that reported error, xm commands are working again.
>
> The BUG message at dmesg:
>
> [66007.135552] BUG: unable to handle kernel paging request at  
> ffff8800004ca458
> [66007.135567] IP: [<ffffffff8100d4ae>] xen_set_pte+0x3e/0x4b
> [66007.135580] PGD 1002067 PUD 1006067 PMD 2d78067 PTE 100000004ca025
> [66007.135675] Oops: 0003 [#1] SMP DEBUG_PAGEALLOC
> [66007.135686] last sysfs file:  
> /sys/class/net/virtbr/bridge/topology_change_detected
> [66007.135693] CPU 4
> [66007.135698] Modules linked in: arptable_filter arp_tables bridge stp  
> bonding bnx2i libiscsi scsi_transport_iscsi cnic uio bnx2 megaraid_sas
> [66007.135729] Pid: 683, comm: pageattr-test Not tainted 2.6.32.36 #7  
> PowerEdge M610
> [66007.135735] RIP: e030:[<ffffffff8100d4ae>]  [<ffffffff8100d4ae>]  
> xen_set_pte+0x3e/0x4b
> [66007.135746] RSP: e02b:ffff88007c8edbb0  EFLAGS: 00010202
> [66007.135751] RAX: 0000000000e32cb6 RBX: 0000000000e32cb6 RCX:  
> 0000000000000001
> [66007.135757] RDX: 0000000000000000 RSI: 8010000800569267 RDI:  
> ffff8800004ca458
> [66007.135764] RBP: ffff88007c8edbd0 R08: 0000000000000001 R09:  
> 0000000000000000
> [66007.135770] R10: ffffffff818385f8 R11: ffffffff818385e0 R12:  
> 8010000800569267
> [66007.135776] R13: ffff8800004ca458 R14: 8010000416569067 R15:  
> 8010000800569267
> [66007.135786] FS:  00007f0eeede66e0(0000) GS:ffff88002813f000(0000)  
> knlGS:0000000000000000
> [66007.135792] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [66007.135797] CR2: ffff8800004ca458 CR3: 000000007b663000 CR4:  
> 0000000000002660
> [66007.135804] DR0: 0000000000000000 DR1: 0000000000000000 DR2:  
> 0000000000000000
> [66007.135810] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:  
> 0000000000000400
> [66007.135816] Process pageattr-test (pid: 683, threadinfo  
> ffff88007c8ec000, task ffff88007e4ce480)
> [66007.135822] Stack:
> [66007.135825]  0000000000000000 8010000004569067 0000000000004569  
> ffff88007c8edd20
> [66007.135835] <0> ffff88007c8edbe0 ffffffff81034740 ffff88007c8edbf0  
> ffffffff8103474d
> [66007.135848] <0> ffff88007c8edcf0 ffffffff81034e77 000000017c8edc40  
> ffffffff818385e0
> [66007.135860] Call Trace:
> [66007.135868]  [<ffffffff81034740>] set_pte+0x17/0x1b
> [66007.135875]  [<ffffffff8103474d>] set_pte_atomic+0x9/0xb
> [66007.135882]  [<ffffffff81034e77>] __change_page_attr_set_clr+0x186/0x82d
> [66007.135936]  [<ffffffff8124f4a0>] ? _raw_spin_unlock+0xab/0xb1
> [66007.135951]  [<ffffffff8157641f>] ? _spin_unlock+0x26/0x2a
> [66007.135961]  [<ffffffff810e587d>] ? vm_unmap_aliases+0x151/0x160
> [66007.135969]  [<ffffffff81035695>] change_page_attr_set_clr+0x177/0x360
> [66007.135976]  [<ffffffff8103597a>] change_page_attr_set+0x27/0x29
> [66007.135983]  [<ffffffff810348e2>] ? pte_flags+0x9/0x18
> [66007.135990]  [<ffffffff81035c01>] do_pageattr_test+0x285/0x4b1
> [66007.135998]  [<ffffffff8103597c>] ? do_pageattr_test+0x0/0x4b1
> [66007.136097]  [<ffffffff8106a9c3>] kthread+0x69/0x71
> [66007.136105]  [<ffffffff81013daa>] child_rip+0xa/0x20
> [66007.136112]  [<ffffffff81012ee6>] ? int_ret_from_sys_call+0x7/0x1b
> [66007.136119]  [<ffffffff81013726>] ? retint_restore_args+0x5/0x6
> [66007.136127]  [<ffffffff81013da0>] ? child_rip+0x0/0x20
> [66007.136131] Code: e8 3c ff ff ff ff 05 b6 5c 94 00 e8 31 ff ff ff 8b  
> 1d b3 5c 94 00 e8 a2 23 02 00 ff c8 0f 94 c0 0f b6 c0 01 d8 89 05 9e 5c  
> 94 00 <4d> 89 65 00 41 59 5b 41 5c 41 5d c9 c3 55 48 89 e5 53 89 fb 48
> [66007.136273] RIP  [<ffffffff8100d4ae>] xen_set_pte+0x3e/0x4b
> [66007.136281]  RSP <ffff88007c8edbb0>
> [66007.136285] CR2: ffff8800004ca458
> [66007.136574] ---[ end trace 4e200a271895cc90 ]---
>
> Attached errors registered in xm dmesg and xend.log.

> [2011-04-15 05:32:06 4415] ERROR (xmlrpclib2:181) Internal error handling xend.domain
> Traceback (most recent call last):
>   File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line 134, in _marshaled_dispatch
>     response = self._dispatch(method, params)
>   File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line 406, in _dispatch
>     return func(*params)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 58, in domain
>     info = lookup(domid)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 50, in lookup
>     info = XendDomain.instance().domain_lookup(domid)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 543, in domain_lookup
>     self._refresh(refresh_shutdown = False)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 417, in _refresh
>     self._refreshTxn(txn, refresh_shutdown)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 430, in _refreshTxn
>     self.domains[domid].update(dom, refresh_shutdown, transaction)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 3523, in update
>     self._update_consoles(transaction)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1656, in _update_consoles
>     vnc_port = self.readDomTxn(transaction, 'console/vnc-port')
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1720, in readDomTxn
>     return transaction.read(*paths)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 72, in read
>     return self._read(args[0])
>   File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 59, in _read
>     '%s, while reading %s' % (ex.args[1], path))
> RuntimeError: (5, 'Input/output error, while reading /local/domain/33/console/vnc-port')
> [2011-04-15 05:32:12 4415] ERROR (xmlrpclib2:181) Internal error handling xend.domain
> Traceback (most recent call last):
>   File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line 134, in _marshaled_dispatch
>     response = self._dispatch(method, params)
>   File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line 406, in _dispatch
>     return func(*params)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 58, in domain
>     info = lookup(domid)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 50, in lookup
>     info = XendDomain.instance().domain_lookup(domid)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 543, in domain_lookup
>     self._refresh(refresh_shutdown = False)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 417, in _refresh
>     self._refreshTxn(txn, refresh_shutdown)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 430, in _refreshTxn
>     self.domains[domid].update(dom, refresh_shutdown, transaction)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 3523, in update
>     self._update_consoles(transaction)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1656, in _update_consoles
>     vnc_port = self.readDomTxn(transaction, 'console/vnc-port')
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1720, in readDomTxn
>     return transaction.read(*paths)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 72, in read
>     return self._read(args[0])
>   File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 59, in _read
>     '%s, while reading %s' % (ex.args[1], path))
> RuntimeError: (5, 'Input/output error, while reading /local/domain/33/console/vnc-port')
> [2011-04-15 05:32:17 4415] ERROR (xmlrpclib2:181) Internal error handling xend.domain
> Traceback (most recent call last):
>   File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line 134, in _marshaled_dispatch
>     response = self._dispatch(method, params)
>   File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line 406, in _dispatch
>     return func(*params)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 58, in domain
>     info = lookup(domid)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 50, in lookup
>     info = XendDomain.instance().domain_lookup(domid)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 543, in domain_lookup
>     self._refresh(refresh_shutdown = False)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 417, in _refresh
>     self._refreshTxn(txn, refresh_shutdown)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 430, in _refreshTxn
>     self.domains[domid].update(dom, refresh_shutdown, transaction)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 3523, in update
>     self._update_consoles(transaction)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1656, in _update_consoles
>     vnc_port = self.readDomTxn(transaction, 'console/vnc-port')
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1720, in readDomTxn
>     return transaction.read(*paths)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 72, in read
>     return self._read(args[0])
>   File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 59, in _read
>     '%s, while reading %s' % (ex.args[1], path))
> 

> (XEN) HVM43: Booting from 0000:7c00
> (XEN) HVM44: Press F12 for boot menu.
> (XEN) HVM44: 
> (XEN) HVM44: Booting from Hard Disk...
> (XEN) HVM44: Booting from 0000:7c00
> (XEN) HVM44: int13_harddisk: function 42. LBA out of range
> (XEN) stdvga.c:151:d43 leaving stdvga
> (XEN) stdvga.c:147:d43 entering stdvga and caching modes
> (XEN) irq.c:243: Dom43 PCI link 0 changed 5 -> 0
> (XEN) irq.c:243: Dom43 PCI link 1 changed 10 -> 0
> (XEN) irq.c:243: Dom43 PCI link 2 changed 11 -> 0
> (XEN) irq.c:243: Dom43 PCI link 3 changed 5 -> 0
> (XEN) stdvga.c:151:d43 leaving stdvga
> (XEN) stdvga.c:151:d42 leaving stdvga
> (XEN) mm.c:3617:d0 PTE entry 81244b for address ffff88007caa0000 doesn't match frame 621c4b
> (XEN) mm.c:3617:d0 PTE entry 812bed for address ffff88007caa1000 doesn't match frame 6217ed
> (XEN) mm.c:3617:d0 PTE entry 812426 for address ffff88007caa2000 doesn't match frame 621c26
> (XEN) mm.c:3617:d0 PTE entry 812590 for address ffff88007caa3000 doesn't match frame 621d90
> (XEN) mm.c:3617:d0 PTE entry 812591 for address ffff88007caa4000 doesn't match frame 621d91
> (XEN) mm.c:3617:d0 PTE entry 812492 for address ffff88007caa5000 doesn't match frame 621c92
> (XEN) mm.c:3617:d0 PTE entry 812493 for address ffff88007caa6000 doesn't match frame 621c93
> (XEN) mm.c:3617:d0 PTE entry 8124ca for address ffff88007caa7000 doesn't match frame 621cca
> (XEN) mm.c:3617:d0 PTE entry 8124cb for address ffff88007caa8000 doesn't match frame 621ccb
> (XEN) mm.c:3617:d0 PTE entry 80e8b0 for address ffff88007caa9000 doesn't match frame 6258b0
> (XEN) mm.c:3617:d0 PTE entry 80e8b1 for address ffff88007caaa000 doesn't match frame 6258b1
> (XEN) mm.c:3617:d0 PTE entry 803fe2 for address ffff88007caab000 doesn't match frame 5cfde2
> (XEN) mm.c:3617:d0 PTE entry 80b3fd for address ffff88007ca99000 doesn't match frame 46b1fd
> (XEN) mm.c:3617:d0 PTE entry 80b3fe for address ffff88007ca9a000 doesn't match frame 46b1fe
> (XEN) mm.c:3617:d0 PTE entry 80ac10 for address ffff88007ca9b000 doesn't match frame 46b610
> (XEN) mm.c:3617:d0 PTE entry 80c01e for address ffff88007ca9c000 doesn't match frame 46a21e
> (XEN) mm.c:3617:d0 PTE entry 80ab16 for address ffff88007ca9d000 doesn't match frame 46b916
> (XEN) mm.c:3617:d0 PTE entry 80b0f6 for address ffff88007ca9e000 doesn't match frame 46b2f6
> (XEN) mm.c:3617:d0 PTE entry 809c8c for address ffff88007ca9f000 doesn't match frame 46c68c
> (XEN) mm.c:3617:d0 PTE entry 8017e0 for address ffff88007cb9f000 doesn't match frame 5b19e0
> (XEN) mm.c:3617:d0 PTE entry 8017e1 for address ffff88007cba0000 doesn't match frame 5b19e1
> (XEN) mm.c:3617:d0 PTE entry 801fc4 for address ffff88007cba1000 doesn't match frame 5b11c4
> (XEN) mm.c:3617:d0 PTE entry 801fc5 for address ffff88007cba2000 doesn't match frame 5b11c5
> (XEN) mm.c:3617:d0 PTE entry 801f2a for address ffff88007cba3000 doesn't match frame 5b112a
> (XEN) mm.c:3617:d0 PTE entry 801f2b for address ffff88007cba4000 doesn't match frame 5b112b
> (XEN) mm.c:3617:d0 PTE entry 801ee8 for address ffff88007cba5000 doesn't match frame 5b10e8
> (XEN) mm.c:3617:d0 PTE entry 801ee9 for address ffff88007cba6000 doesn't match frame 5b10e9
> (XEN) mm.c:3617:d0 PTE entry 803fde for address ffff88007cba7000 doesn't match frame 5af1de
> (XEN) mm.c:3617:d0 PTE entry 8120ec for address ffff88007cb9e000 doesn't match frame 9660ec
> (XEN) mm.c:3617:d0 PTE entry 8120ed for address ffff88007cb9f000 doesn't match frame 9660ed
> (XEN) mm.c:3617:d0 PTE entry 810e0a for address ffff88007cba0000 doesn't match frame 96720a
> (XEN) mm.c:3617:d0 PTE entry 810e0b for address ffff88007cba1000 doesn't match frame 96720b
> (XEN) mm.c:3617:d0 PTE entry 81080e for address ffff88007cba2000 doesn't match frame 96780e
> (XEN) mm.c:3617:d0 PTE entry 81080f for address ffff88007cba3000 doesn't match frame 96780f
> (XEN) mm.c:3617:d0 PTE entry 80ef3e for address ffff88007cba4000 doesn't match frame 96933e
> (XEN) mm.c:3617:d0 PTE entry 802fd7 for address ffff88007c433000 doesn't match frame 2111d7
> (XEN) mm.c:3617:d0 PTE entry 805131 for address ffff88007c434000 doesn't match frame 20ef31
> (XEN) mm.c:3617:d0 PTE entry 80c1ea for address ffff88007c435000 doesn't match frame 207fea
> (XEN) mm.c:3617:d0 PTE entry 814ae9 for address ffff88007c9b0000 doesn't match frame 8866e9
> (XEN) mm.c:3617:d0 PTE entry 8113b9 for address ffff88007ca20000 doesn't match frame 9277b9
> (XEN) mm.c:3617:d0 PTE entry 808ab2 for address ffff88007ca21000 doesn't match frame 92feb2
> (XEN) mm.c:3617:d0 PTE entry 803bb3 for address ffff88007ca22000 doesn't match frame 934fb3
> (XEN) mm.c:3617:d0 PTE entry 814747 for address ffff88007ca23000 doesn't match frame 924347
> (XEN) mm.c:3617:d0 PTE entry 819873 for address ffff88007ca24000 doesn't match frame 91f073
> (XEN) mm.c:3617:d0 PTE entry 80302c for address ffff88007ca25000 doesn't match frame 93582c
> (XEN) mm.c:3617:d0 PTE entry 81b75c for address ffff88007ca26000 doesn't match frame 91d35c
> (XEN) mm.c:3617:d0 PTE entry c5fd0b for address ffff88007ca86000 doesn't match frame 95830b
> (XEN) mm.c:3617:d0 PTE entry 81a093 for address ffff88007ca87000 doesn't match frame 9bde93
> (XEN) mm.c:3617:d0 PTE entry 815f11 for address ffff88007ca88000 doesn't match frame 942111
> (XEN) mm.c:3617:d0 PTE entry 815bc5 for address ffff88007ca89000 doesn't match frame 9425c5
> (XEN) mm.c:3617:d0 PTE entry 816875 for address ffff88007cb9b000 doesn't match frame 59c675
> (XEN) mm.c:3617:d0 PTE entry 8158a8 for address ffff88007cb9c000 doesn't match frame 59d6a8
> (XEN) mm.c:3617:d0 PTE entry 82fa1c for address ffff88007c926000 doesn't match frame 5e321c
> (XEN) mm.c:3617:d0 PTE entry 82655e for address ffff88007c927000 doesn't match frame 5ec95e
> (XEN) mm.c:3617:d0 PTE entry c5ce72 for address ffff88007c926000 doesn't match frame 95b072
> (XEN) mm.c:3617:d0 PTE entry c5cea9 for address ffff88007c927000 doesn't match frame 95b0a9
> (XEN) mm.c:2399:d0 Bad type (saw 1c00000000000002 != exp 7000000000000000) for mfn 800569 (pfn 4569)
> (XEN) mm.c:878:d0 Error getting mfn 800569 (pfn 4569) from L1 entry 8010000800569267 for l1e_owner=0, pg_owner=0
> (XEN) mm.c:4664:d0 ptwr_emulate: could not get_page_from_l1e()
> (XEN) mm.c:3617:d0 PTE entry 80e284 for address ffff88007ca89000 doesn't match frame 468084
> (XEN) mm.c:3617:d0 PTE entry 817918 for address ffff88007ca8a000 doesn't match frame 47f118
> (XEN) common.c:3722:d0 tracking VRAM f0000 - f0160
> (XEN) common.c:3722:d0 tracking VRAM f0000 - f0160

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: xen 4.0.2rc3/kernel 2.6.32.36: BUG: unable to handle kernel paging request
  2011-04-15 15:44     ` Pasi Kärkkäinen
@ 2011-04-15 15:47       ` Gerd Jakobovitsch
  2011-04-18 21:05         ` Konrad Rzeszutek Wilk
  0 siblings, 1 reply; 11+ messages in thread
From: Gerd Jakobovitsch @ 2011-04-15 15:47 UTC (permalink / raw)
  To: Pasi Kärkkäinen; +Cc: xen-devel@lists.xensource.com


[-- Attachment #1.1: Type: text/plain, Size: 19138 bytes --]

Hello Pasi:

Yes, the kernel includes the last bugfix, and additionally the patch 
Daniel Stodden indicated yesterday:

http://xenbits.xen.org/gitweb/?p=people/dstodden/linux.git;a=commit;h=a765257af7e28c41bd776c3e03615539597eb592



On 04/15/2011 12:44 PM, Pasi Kärkkäinen wrote:
> On Fri, Apr 15, 2011 at 11:21:40AM -0300, Gerd Jakobovitsch wrote:
>> Reporting a new bug that appeared during stress tests. The scenario is
>> the same as reported below, with patches applied:
>>
> Hello,
>
> Can you clarify which exact kernel version/changeset you used for this test?
>
> Does your kernel include the latest bugfix (from 3 days ago)?
> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=ae333e97552c81ab10395ad1ffc6d6daaadb144a
>
> changelog:
> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=xen/stable-2.6.32.x
>
> -- Pasi
>
>
>> On 04/14/2011 10:15 AM, Konrad Rzeszutek Wilk wrote:
>>> On Wed, Apr 13, 2011 at 06:02:13PM -0300, Gerd Jakobovitsch wrote:
>>>> I'm trying to run several VMs (linux hvm, with tapdisk:aio disks at
>>>> a storage over nfs) on a CentOS system, using the up-to-date version
>>>> of xen 4.0 / kernel pvops 2.6.32.x stable. With a configuration
>>>> without (most of) debug activated, I can start several instances -
>>>> I'm running 7 of them - but shortly afterwards the system stops
>>>> responding. I can't find any information on this.
>>> First time I see it.
>>>> Activating several debug configuration items, among them
>>>> DEBUG_PAGEALLOC, I get an exception as soon as I try to start up a
>>>> VM. The system reboots.
>>>>
>> With the debug information still set, I'm running 42 VMs - mixed Linux
>> (several distros) and Windows, most of them running benchmarks for CPU
>> and disk usage. After roughly 15 hours, a bug message appeared at dmesg.
>> It affected xm commands - it seems to be related to a specific VM - but
>> xl commands still work. VMs are running.
>>
>> # xm list
>> Error: (5, 'Input/output error, while reading
>> /local/domain/33/console/vnc-port')
>> Usage: xm list [options] [Domain, ...]
>>
>> After killing the VM that reported error, xm commands are working again.
>>
>> The BUG message at dmesg:
>>
>> [66007.135552] BUG: unable to handle kernel paging request at
>> ffff8800004ca458
>> [66007.135567] IP: [<ffffffff8100d4ae>] xen_set_pte+0x3e/0x4b
>> [66007.135580] PGD 1002067 PUD 1006067 PMD 2d78067 PTE 100000004ca025
>> [66007.135675] Oops: 0003 [#1] SMP DEBUG_PAGEALLOC
>> [66007.135686] last sysfs file:
>> /sys/class/net/virtbr/bridge/topology_change_detected
>> [66007.135693] CPU 4
>> [66007.135698] Modules linked in: arptable_filter arp_tables bridge stp
>> bonding bnx2i libiscsi scsi_transport_iscsi cnic uio bnx2 megaraid_sas
>> [66007.135729] Pid: 683, comm: pageattr-test Not tainted 2.6.32.36 #7
>> PowerEdge M610
>> [66007.135735] RIP: e030:[<ffffffff8100d4ae>]  [<ffffffff8100d4ae>]
>> xen_set_pte+0x3e/0x4b
>> [66007.135746] RSP: e02b:ffff88007c8edbb0  EFLAGS: 00010202
>> [66007.135751] RAX: 0000000000e32cb6 RBX: 0000000000e32cb6 RCX:
>> 0000000000000001
>> [66007.135757] RDX: 0000000000000000 RSI: 8010000800569267 RDI:
>> ffff8800004ca458
>> [66007.135764] RBP: ffff88007c8edbd0 R08: 0000000000000001 R09:
>> 0000000000000000
>> [66007.135770] R10: ffffffff818385f8 R11: ffffffff818385e0 R12:
>> 8010000800569267
>> [66007.135776] R13: ffff8800004ca458 R14: 8010000416569067 R15:
>> 8010000800569267
>> [66007.135786] FS:  00007f0eeede66e0(0000) GS:ffff88002813f000(0000)
>> knlGS:0000000000000000
>> [66007.135792] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> [66007.135797] CR2: ffff8800004ca458 CR3: 000000007b663000 CR4:
>> 0000000000002660
>> [66007.135804] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [66007.135810] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400
>> [66007.135816] Process pageattr-test (pid: 683, threadinfo
>> ffff88007c8ec000, task ffff88007e4ce480)
>> [66007.135822] Stack:
>> [66007.135825]  0000000000000000 8010000004569067 0000000000004569
>> ffff88007c8edd20
>> [66007.135835]<0>  ffff88007c8edbe0 ffffffff81034740 ffff88007c8edbf0
>> ffffffff8103474d
>> [66007.135848]<0>  ffff88007c8edcf0 ffffffff81034e77 000000017c8edc40
>> ffffffff818385e0
>> [66007.135860] Call Trace:
>> [66007.135868]  [<ffffffff81034740>] set_pte+0x17/0x1b
>> [66007.135875]  [<ffffffff8103474d>] set_pte_atomic+0x9/0xb
>> [66007.135882]  [<ffffffff81034e77>] __change_page_attr_set_clr+0x186/0x82d
>> [66007.135936]  [<ffffffff8124f4a0>] ? _raw_spin_unlock+0xab/0xb1
>> [66007.135951]  [<ffffffff8157641f>] ? _spin_unlock+0x26/0x2a
>> [66007.135961]  [<ffffffff810e587d>] ? vm_unmap_aliases+0x151/0x160
>> [66007.135969]  [<ffffffff81035695>] change_page_attr_set_clr+0x177/0x360
>> [66007.135976]  [<ffffffff8103597a>] change_page_attr_set+0x27/0x29
>> [66007.135983]  [<ffffffff810348e2>] ? pte_flags+0x9/0x18
>> [66007.135990]  [<ffffffff81035c01>] do_pageattr_test+0x285/0x4b1
>> [66007.135998]  [<ffffffff8103597c>] ? do_pageattr_test+0x0/0x4b1
>> [66007.136097]  [<ffffffff8106a9c3>] kthread+0x69/0x71
>> [66007.136105]  [<ffffffff81013daa>] child_rip+0xa/0x20
>> [66007.136112]  [<ffffffff81012ee6>] ? int_ret_from_sys_call+0x7/0x1b
>> [66007.136119]  [<ffffffff81013726>] ? retint_restore_args+0x5/0x6
>> [66007.136127]  [<ffffffff81013da0>] ? child_rip+0x0/0x20
>> [66007.136131] Code: e8 3c ff ff ff ff 05 b6 5c 94 00 e8 31 ff ff ff 8b
>> 1d b3 5c 94 00 e8 a2 23 02 00 ff c8 0f 94 c0 0f b6 c0 01 d8 89 05 9e 5c
>> 94 00<4d>  89 65 00 41 59 5b 41 5c 41 5d c9 c3 55 48 89 e5 53 89 fb 48
>> [66007.136273] RIP  [<ffffffff8100d4ae>] xen_set_pte+0x3e/0x4b
>> [66007.136281]  RSP<ffff88007c8edbb0>
>> [66007.136285] CR2: ffff8800004ca458
>> [66007.136574] ---[ end trace 4e200a271895cc90 ]---
>>
>> Attached errors registered in xm dmesg and xend.log.
>> [2011-04-15 05:32:06 4415] ERROR (xmlrpclib2:181) Internal error handling xend.domain
>> Traceback (most recent call last):
>>    File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line 134, in _marshaled_dispatch
>>      response = self._dispatch(method, params)
>>    File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line 406, in _dispatch
>>      return func(*params)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 58, in domain
>>      info = lookup(domid)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 50, in lookup
>>      info = XendDomain.instance().domain_lookup(domid)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 543, in domain_lookup
>>      self._refresh(refresh_shutdown = False)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 417, in _refresh
>>      self._refreshTxn(txn, refresh_shutdown)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 430, in _refreshTxn
>>      self.domains[domid].update(dom, refresh_shutdown, transaction)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 3523, in update
>>      self._update_consoles(transaction)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1656, in _update_consoles
>>      vnc_port = self.readDomTxn(transaction, 'console/vnc-port')
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1720, in readDomTxn
>>      return transaction.read(*paths)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 72, in read
>>      return self._read(args[0])
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 59, in _read
>>      '%s, while reading %s' % (ex.args[1], path))
>> RuntimeError: (5, 'Input/output error, while reading /local/domain/33/console/vnc-port')
>> [2011-04-15 05:32:12 4415] ERROR (xmlrpclib2:181) Internal error handling xend.domain
>> Traceback (most recent call last):
>>    File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line 134, in _marshaled_dispatch
>>      response = self._dispatch(method, params)
>>    File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line 406, in _dispatch
>>      return func(*params)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 58, in domain
>>      info = lookup(domid)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 50, in lookup
>>      info = XendDomain.instance().domain_lookup(domid)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 543, in domain_lookup
>>      self._refresh(refresh_shutdown = False)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 417, in _refresh
>>      self._refreshTxn(txn, refresh_shutdown)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 430, in _refreshTxn
>>      self.domains[domid].update(dom, refresh_shutdown, transaction)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 3523, in update
>>      self._update_consoles(transaction)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1656, in _update_consoles
>>      vnc_port = self.readDomTxn(transaction, 'console/vnc-port')
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1720, in readDomTxn
>>      return transaction.read(*paths)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 72, in read
>>      return self._read(args[0])
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 59, in _read
>>      '%s, while reading %s' % (ex.args[1], path))
>> RuntimeError: (5, 'Input/output error, while reading /local/domain/33/console/vnc-port')
>> [2011-04-15 05:32:17 4415] ERROR (xmlrpclib2:181) Internal error handling xend.domain
>> Traceback (most recent call last):
>>    File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line 134, in _marshaled_dispatch
>>      response = self._dispatch(method, params)
>>    File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line 406, in _dispatch
>>      return func(*params)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 58, in domain
>>      info = lookup(domid)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 50, in lookup
>>      info = XendDomain.instance().domain_lookup(domid)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 543, in domain_lookup
>>      self._refresh(refresh_shutdown = False)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 417, in _refresh
>>      self._refreshTxn(txn, refresh_shutdown)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 430, in _refreshTxn
>>      self.domains[domid].update(dom, refresh_shutdown, transaction)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 3523, in update
>>      self._update_consoles(transaction)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1656, in _update_consoles
>>      vnc_port = self.readDomTxn(transaction, 'console/vnc-port')
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1720, in readDomTxn
>>      return transaction.read(*paths)
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 72, in read
>>      return self._read(args[0])
>>    File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 59, in _read
>>      '%s, while reading %s' % (ex.args[1], path))
>>
>> (XEN) HVM43: Booting from 0000:7c00
>> (XEN) HVM44: Press F12 for boot menu.
>> (XEN) HVM44:
>> (XEN) HVM44: Booting from Hard Disk...
>> (XEN) HVM44: Booting from 0000:7c00
>> (XEN) HVM44: int13_harddisk: function 42. LBA out of range
>> (XEN) stdvga.c:151:d43 leaving stdvga
>> (XEN) stdvga.c:147:d43 entering stdvga and caching modes
>> (XEN) irq.c:243: Dom43 PCI link 0 changed 5 ->  0
>> (XEN) irq.c:243: Dom43 PCI link 1 changed 10 ->  0
>> (XEN) irq.c:243: Dom43 PCI link 2 changed 11 ->  0
>> (XEN) irq.c:243: Dom43 PCI link 3 changed 5 ->  0
>> (XEN) stdvga.c:151:d43 leaving stdvga
>> (XEN) stdvga.c:151:d42 leaving stdvga
>> (XEN) mm.c:3617:d0 PTE entry 81244b for address ffff88007caa0000 doesn't match frame 621c4b
>> (XEN) mm.c:3617:d0 PTE entry 812bed for address ffff88007caa1000 doesn't match frame 6217ed
>> (XEN) mm.c:3617:d0 PTE entry 812426 for address ffff88007caa2000 doesn't match frame 621c26
>> (XEN) mm.c:3617:d0 PTE entry 812590 for address ffff88007caa3000 doesn't match frame 621d90
>> (XEN) mm.c:3617:d0 PTE entry 812591 for address ffff88007caa4000 doesn't match frame 621d91
>> (XEN) mm.c:3617:d0 PTE entry 812492 for address ffff88007caa5000 doesn't match frame 621c92
>> (XEN) mm.c:3617:d0 PTE entry 812493 for address ffff88007caa6000 doesn't match frame 621c93
>> (XEN) mm.c:3617:d0 PTE entry 8124ca for address ffff88007caa7000 doesn't match frame 621cca
>> (XEN) mm.c:3617:d0 PTE entry 8124cb for address ffff88007caa8000 doesn't match frame 621ccb
>> (XEN) mm.c:3617:d0 PTE entry 80e8b0 for address ffff88007caa9000 doesn't match frame 6258b0
>> (XEN) mm.c:3617:d0 PTE entry 80e8b1 for address ffff88007caaa000 doesn't match frame 6258b1
>> (XEN) mm.c:3617:d0 PTE entry 803fe2 for address ffff88007caab000 doesn't match frame 5cfde2
>> (XEN) mm.c:3617:d0 PTE entry 80b3fd for address ffff88007ca99000 doesn't match frame 46b1fd
>> (XEN) mm.c:3617:d0 PTE entry 80b3fe for address ffff88007ca9a000 doesn't match frame 46b1fe
>> (XEN) mm.c:3617:d0 PTE entry 80ac10 for address ffff88007ca9b000 doesn't match frame 46b610
>> (XEN) mm.c:3617:d0 PTE entry 80c01e for address ffff88007ca9c000 doesn't match frame 46a21e
>> (XEN) mm.c:3617:d0 PTE entry 80ab16 for address ffff88007ca9d000 doesn't match frame 46b916
>> (XEN) mm.c:3617:d0 PTE entry 80b0f6 for address ffff88007ca9e000 doesn't match frame 46b2f6
>> (XEN) mm.c:3617:d0 PTE entry 809c8c for address ffff88007ca9f000 doesn't match frame 46c68c
>> (XEN) mm.c:3617:d0 PTE entry 8017e0 for address ffff88007cb9f000 doesn't match frame 5b19e0
>> (XEN) mm.c:3617:d0 PTE entry 8017e1 for address ffff88007cba0000 doesn't match frame 5b19e1
>> (XEN) mm.c:3617:d0 PTE entry 801fc4 for address ffff88007cba1000 doesn't match frame 5b11c4
>> (XEN) mm.c:3617:d0 PTE entry 801fc5 for address ffff88007cba2000 doesn't match frame 5b11c5
>> (XEN) mm.c:3617:d0 PTE entry 801f2a for address ffff88007cba3000 doesn't match frame 5b112a
>> (XEN) mm.c:3617:d0 PTE entry 801f2b for address ffff88007cba4000 doesn't match frame 5b112b
>> (XEN) mm.c:3617:d0 PTE entry 801ee8 for address ffff88007cba5000 doesn't match frame 5b10e8
>> (XEN) mm.c:3617:d0 PTE entry 801ee9 for address ffff88007cba6000 doesn't match frame 5b10e9
>> (XEN) mm.c:3617:d0 PTE entry 803fde for address ffff88007cba7000 doesn't match frame 5af1de
>> (XEN) mm.c:3617:d0 PTE entry 8120ec for address ffff88007cb9e000 doesn't match frame 9660ec
>> (XEN) mm.c:3617:d0 PTE entry 8120ed for address ffff88007cb9f000 doesn't match frame 9660ed
>> (XEN) mm.c:3617:d0 PTE entry 810e0a for address ffff88007cba0000 doesn't match frame 96720a
>> (XEN) mm.c:3617:d0 PTE entry 810e0b for address ffff88007cba1000 doesn't match frame 96720b
>> (XEN) mm.c:3617:d0 PTE entry 81080e for address ffff88007cba2000 doesn't match frame 96780e
>> (XEN) mm.c:3617:d0 PTE entry 81080f for address ffff88007cba3000 doesn't match frame 96780f
>> (XEN) mm.c:3617:d0 PTE entry 80ef3e for address ffff88007cba4000 doesn't match frame 96933e
>> (XEN) mm.c:3617:d0 PTE entry 802fd7 for address ffff88007c433000 doesn't match frame 2111d7
>> (XEN) mm.c:3617:d0 PTE entry 805131 for address ffff88007c434000 doesn't match frame 20ef31
>> (XEN) mm.c:3617:d0 PTE entry 80c1ea for address ffff88007c435000 doesn't match frame 207fea
>> (XEN) mm.c:3617:d0 PTE entry 814ae9 for address ffff88007c9b0000 doesn't match frame 8866e9
>> (XEN) mm.c:3617:d0 PTE entry 8113b9 for address ffff88007ca20000 doesn't match frame 9277b9
>> (XEN) mm.c:3617:d0 PTE entry 808ab2 for address ffff88007ca21000 doesn't match frame 92feb2
>> (XEN) mm.c:3617:d0 PTE entry 803bb3 for address ffff88007ca22000 doesn't match frame 934fb3
>> (XEN) mm.c:3617:d0 PTE entry 814747 for address ffff88007ca23000 doesn't match frame 924347
>> (XEN) mm.c:3617:d0 PTE entry 819873 for address ffff88007ca24000 doesn't match frame 91f073
>> (XEN) mm.c:3617:d0 PTE entry 80302c for address ffff88007ca25000 doesn't match frame 93582c
>> (XEN) mm.c:3617:d0 PTE entry 81b75c for address ffff88007ca26000 doesn't match frame 91d35c
>> (XEN) mm.c:3617:d0 PTE entry c5fd0b for address ffff88007ca86000 doesn't match frame 95830b
>> (XEN) mm.c:3617:d0 PTE entry 81a093 for address ffff88007ca87000 doesn't match frame 9bde93
>> (XEN) mm.c:3617:d0 PTE entry 815f11 for address ffff88007ca88000 doesn't match frame 942111
>> (XEN) mm.c:3617:d0 PTE entry 815bc5 for address ffff88007ca89000 doesn't match frame 9425c5
>> (XEN) mm.c:3617:d0 PTE entry 816875 for address ffff88007cb9b000 doesn't match frame 59c675
>> (XEN) mm.c:3617:d0 PTE entry 8158a8 for address ffff88007cb9c000 doesn't match frame 59d6a8
>> (XEN) mm.c:3617:d0 PTE entry 82fa1c for address ffff88007c926000 doesn't match frame 5e321c
>> (XEN) mm.c:3617:d0 PTE entry 82655e for address ffff88007c927000 doesn't match frame 5ec95e
>> (XEN) mm.c:3617:d0 PTE entry c5ce72 for address ffff88007c926000 doesn't match frame 95b072
>> (XEN) mm.c:3617:d0 PTE entry c5cea9 for address ffff88007c927000 doesn't match frame 95b0a9
>> (XEN) mm.c:2399:d0 Bad type (saw 1c00000000000002 != exp 7000000000000000) for mfn 800569 (pfn 4569)
>> (XEN) mm.c:878:d0 Error getting mfn 800569 (pfn 4569) from L1 entry 8010000800569267 for l1e_owner=0, pg_owner=0
>> (XEN) mm.c:4664:d0 ptwr_emulate: could not get_page_from_l1e()
>> (XEN) mm.c:3617:d0 PTE entry 80e284 for address ffff88007ca89000 doesn't match frame 468084
>> (XEN) mm.c:3617:d0 PTE entry 817918 for address ffff88007ca8a000 doesn't match frame 47f118
>> (XEN) common.c:3722:d0 tracking VRAM f0000 - f0160
>> (XEN) common.c:3722:d0 tracking VRAM f0000 - f0160
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel

-- 

*Gerd Jakobovitsch
Engenheiro de Produto **
---------------------------------------------------------* *
**ALOG Data Centers do Brasil**
**Excelência em Projetos de Hosting*
Rua Dr. Miguel Couto, 58 -- 01008-010 -- São Paulo - SP
Telefone: (11) 3524-4970 / (11) 7152-0815
*http://www.alog.com.br* <http://www.alog.com.br/>


*"Como estão nossos serviços? Clique aqui 
<https://www.surveymonkey.com/s/faleconosco> e nos conte. Queremos 
escutar a sua opinião!"*






[-- Attachment #1.2.1: Type: text/html, Size: 22455 bytes --]

[-- Attachment #1.2.2: alog --]
[-- Type: image/png, Size: 32911 bytes --]

[-- Attachment #1.2.3: graphics2 --]
[-- Type: image/png, Size: 42793 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: xen 4.0.2rc3/kernel 2.6.32.36: BUG: unable to handle kernel paging request
  2011-04-15 15:47       ` Gerd Jakobovitsch
@ 2011-04-18 21:05         ` Konrad Rzeszutek Wilk
  0 siblings, 0 replies; 11+ messages in thread
From: Konrad Rzeszutek Wilk @ 2011-04-18 21:05 UTC (permalink / raw)
  To: Gerd Jakobovitsch; +Cc: xen-devel@lists.xensource.com

On Fri, Apr 15, 2011 at 12:47:51PM -0300, Gerd Jakobovitsch wrote:
> Hello Pasi:
> 
> Yes, the kernel includes the last bugfix, and additionally the patch
> Daniel Stodden indicated yesterday:
> >>The BUG message at dmesg:

.. snip..
> >>
> >>[66007.135552] BUG: unable to handle kernel paging request at
> >>ffff8800004ca458
> >>[66007.135567] IP: [<ffffffff8100d4ae>] xen_set_pte+0x3e/0x4b
> >>[66007.135580] PGD 1002067 PUD 1006067 PMD 2d78067 PTE 100000004ca025
> >>[66007.135675] Oops: 0003 [#1] SMP DEBUG_PAGEALLOC

.. snip..
> >>[66007.135860] Call Trace:
> >>[66007.135868]  [<ffffffff81034740>] set_pte+0x17/0x1b
> >>[66007.135875]  [<ffffffff8103474d>] set_pte_atomic+0x9/0xb
> >>[66007.135882]  [<ffffffff81034e77>] __change_page_attr_set_clr+0x186/0x82d
> >>[66007.135936]  [<ffffffff8124f4a0>] ? _raw_spin_unlock+0xab/0xb1
> >>[66007.135951]  [<ffffffff8157641f>] ? _spin_unlock+0x26/0x2a
> >>[66007.135961]  [<ffffffff810e587d>] ? vm_unmap_aliases+0x151/0x160
> >>[66007.135969]  [<ffffffff81035695>] change_page_attr_set_clr+0x177/0x360
> >>[66007.135976]  [<ffffffff8103597a>] change_page_attr_set+0x27/0x29
> >>[66007.135983]  [<ffffffff810348e2>] ? pte_flags+0x9/0x18
> >>[66007.135990]  [<ffffffff81035c01>] do_pageattr_test+0x285/0x4b1
> >>[66007.135998]  [<ffffffff8103597c>] ? do_pageattr_test+0x0/0x4b1
> >>[66007.136097]  [<ffffffff8106a9c3>] kthread+0x69/0x71
> >>[66007.136105]  [<ffffffff81013daa>] child_rip+0xa/0x20
> >>[66007.136112]  [<ffffffff81012ee6>] ? int_ret_from_sys_call+0x7/0x1b
> >>[66007.136119]  [<ffffffff81013726>] ? retint_restore_args+0x5/0x6
> >>[66007.136127]  [<ffffffff81013da0>] ? child_rip+0x0/0x20

.. snip..
> >>(XEN) HVM43: Booting from 0000:7c00
> >>(XEN) HVM44: Press F12 for boot menu.
> >>(XEN) HVM44:
> >>(XEN) HVM44: Booting from Hard Disk...
> >>(XEN) HVM44: Booting from 0000:7c00
> >>(XEN) HVM44: int13_harddisk: function 42. LBA out of range
> >>(XEN) stdvga.c:151:d43 leaving stdvga
> >>(XEN) stdvga.c:147:d43 entering stdvga and caching modes
> >>(XEN) irq.c:243: Dom43 PCI link 0 changed 5 ->  0
> >>(XEN) irq.c:243: Dom43 PCI link 1 changed 10 ->  0
> >>(XEN) irq.c:243: Dom43 PCI link 2 changed 11 ->  0
> >>(XEN) irq.c:243: Dom43 PCI link 3 changed 5 ->  0
> >>(XEN) stdvga.c:151:d43 leaving stdvga
> >>(XEN) stdvga.c:151:d42 leaving stdvga
> >>(XEN) mm.c:3617:d0 PTE entry 81244b for address ffff88007caa0000 doesn't match frame 621c4b
> >>(XEN) mm.c:3617:d0 PTE entry 812bed for address ffff88007caa1000 doesn't match frame 6217ed
> >>(XEN) mm.c:3617:d0 PTE entry 812426 for address ffff88007caa2000 doesn't match frame 621c26
> >>(XEN) mm.c:3617:d0 PTE entry 812590 for address ffff88007caa3000 doesn't match frame 621d90
> >>(XEN) mm.c:3617:d0 PTE entry 812591 for address ffff88007caa4000 doesn't match frame 621d91
> >>(XEN) mm.c:3617:d0 PTE entry 812492 for address ffff88007caa5000 doesn't match frame 621c92
> >>(XEN) mm.c:3617:d0 PTE entry 812493 for address ffff88007caa6000 doesn't match frame 621c93
> >>(XEN) mm.c:3617:d0 PTE entry 8124ca for address ffff88007caa7000 doesn't match frame 621cca
> >>(XEN) mm.c:3617:d0 PTE entry 8124cb for address ffff88007caa8000 doesn't match frame 621ccb
> >>(XEN) mm.c:3617:d0 PTE entry 80e8b0 for address ffff88007caa9000 doesn't match frame 6258b0
> >>(XEN) mm.c:3617:d0 PTE entry 80e8b1 for address ffff88007caaa000 doesn't match frame 6258b1
> >>(XEN) mm.c:3617:d0 PTE entry 803fe2 for address ffff88007caab000 doesn't match frame 5cfde2
> >>(XEN) mm.c:3617:d0 PTE entry 80b3fd for address ffff88007ca99000 doesn't match frame 46b1fd
> >>(XEN) mm.c:3617:d0 PTE entry 80b3fe for address ffff88007ca9a000 doesn't match frame 46b1fe
> >>(XEN) mm.c:3617:d0 PTE entry 80ac10 for address ffff88007ca9b000 doesn't match frame 46b610
> >>(XEN) mm.c:3617:d0 PTE entry 80c01e for address ffff88007ca9c000 doesn't match frame 46a21e
> >>(XEN) mm.c:3617:d0 PTE entry 80ab16 for address ffff88007ca9d000 doesn't match frame 46b916
> >>(XEN) mm.c:3617:d0 PTE entry 80b0f6 for address ffff88007ca9e000 doesn't match frame 46b2f6
> >>(XEN) mm.c:3617:d0 PTE entry 809c8c for address ffff88007ca9f000 doesn't match frame 46c68c
> >>(XEN) mm.c:3617:d0 PTE entry 8017e0 for address ffff88007cb9f000 doesn't match frame 5b19e0
> >>(XEN) mm.c:3617:d0 PTE entry 8017e1 for address ffff88007cba0000 doesn't match frame 5b19e1
> >>(XEN) mm.c:3617:d0 PTE entry 801fc4 for address ffff88007cba1000 doesn't match frame 5b11c4
> >>(XEN) mm.c:3617:d0 PTE entry 801fc5 for address ffff88007cba2000 doesn't match frame 5b11c5
> >>(XEN) mm.c:3617:d0 PTE entry 801f2a for address ffff88007cba3000 doesn't match frame 5b112a
> >>(XEN) mm.c:3617:d0 PTE entry 801f2b for address ffff88007cba4000 doesn't match frame 5b112b
> >>(XEN) mm.c:3617:d0 PTE entry 801ee8 for address ffff88007cba5000 doesn't match frame 5b10e8
> >>(XEN) mm.c:3617:d0 PTE entry 801ee9 for address ffff88007cba6000 doesn't match frame 5b10e9
> >>(XEN) mm.c:3617:d0 PTE entry 803fde for address ffff88007cba7000 doesn't match frame 5af1de
> >>(XEN) mm.c:3617:d0 PTE entry 8120ec for address ffff88007cb9e000 doesn't match frame 9660ec
> >>(XEN) mm.c:3617:d0 PTE entry 8120ed for address ffff88007cb9f000 doesn't match frame 9660ed
> >>(XEN) mm.c:3617:d0 PTE entry 810e0a for address ffff88007cba0000 doesn't match frame 96720a
> >>(XEN) mm.c:3617:d0 PTE entry 810e0b for address ffff88007cba1000 doesn't match frame 96720b
> >>(XEN) mm.c:3617:d0 PTE entry 81080e for address ffff88007cba2000 doesn't match frame 96780e
> >>(XEN) mm.c:3617:d0 PTE entry 81080f for address ffff88007cba3000 doesn't match frame 96780f
> >>(XEN) mm.c:3617:d0 PTE entry 80ef3e for address ffff88007cba4000 doesn't match frame 96933e
> >>(XEN) mm.c:3617:d0 PTE entry 802fd7 for address ffff88007c433000 doesn't match frame 2111d7
> >>(XEN) mm.c:3617:d0 PTE entry 805131 for address ffff88007c434000 doesn't match frame 20ef31
> >>(XEN) mm.c:3617:d0 PTE entry 80c1ea for address ffff88007c435000 doesn't match frame 207fea
> >>(XEN) mm.c:3617:d0 PTE entry 814ae9 for address ffff88007c9b0000 doesn't match frame 8866e9
> >>(XEN) mm.c:3617:d0 PTE entry 8113b9 for address ffff88007ca20000 doesn't match frame 9277b9
> >>(XEN) mm.c:3617:d0 PTE entry 808ab2 for address ffff88007ca21000 doesn't match frame 92feb2
> >>(XEN) mm.c:3617:d0 PTE entry 803bb3 for address ffff88007ca22000 doesn't match frame 934fb3
> >>(XEN) mm.c:3617:d0 PTE entry 814747 for address ffff88007ca23000 doesn't match frame 924347
> >>(XEN) mm.c:3617:d0 PTE entry 819873 for address ffff88007ca24000 doesn't match frame 91f073
> >>(XEN) mm.c:3617:d0 PTE entry 80302c for address ffff88007ca25000 doesn't match frame 93582c
> >>(XEN) mm.c:3617:d0 PTE entry 81b75c for address ffff88007ca26000 doesn't match frame 91d35c
> >>(XEN) mm.c:3617:d0 PTE entry c5fd0b for address ffff88007ca86000 doesn't match frame 95830b
> >>(XEN) mm.c:3617:d0 PTE entry 81a093 for address ffff88007ca87000 doesn't match frame 9bde93
> >>(XEN) mm.c:3617:d0 PTE entry 815f11 for address ffff88007ca88000 doesn't match frame 942111
> >>(XEN) mm.c:3617:d0 PTE entry 815bc5 for address ffff88007ca89000 doesn't match frame 9425c5
> >>(XEN) mm.c:3617:d0 PTE entry 816875 for address ffff88007cb9b000 doesn't match frame 59c675
> >>(XEN) mm.c:3617:d0 PTE entry 8158a8 for address ffff88007cb9c000 doesn't match frame 59d6a8
> >>(XEN) mm.c:3617:d0 PTE entry 82fa1c for address ffff88007c926000 doesn't match frame 5e321c
> >>(XEN) mm.c:3617:d0 PTE entry 82655e for address ffff88007c927000 doesn't match frame 5ec95e
> >>(XEN) mm.c:3617:d0 PTE entry c5ce72 for address ffff88007c926000 doesn't match frame 95b072
> >>(XEN) mm.c:3617:d0 PTE entry c5cea9 for address ffff88007c927000 doesn't match frame 95b0a9
> >>(XEN) mm.c:2399:d0 Bad type (saw 1c00000000000002 != exp 7000000000000000) for mfn 800569 (pfn 4569)
> >>(XEN) mm.c:878:d0 Error getting mfn 800569 (pfn 4569) from L1 entry 8010000800569267 for l1e_owner=0, pg_owner=0
> >>(XEN) mm.c:4664:d0 ptwr_emulate: could not get_page_from_l1e()
> >>(XEN) mm.c:3617:d0 PTE entry 80e284 for address ffff88007ca89000 doesn't match frame 468084
> >>(XEN) mm.c:3617:d0 PTE entry 817918 for address ffff88007ca8a000 doesn't match frame 47f118
> >>(XEN) common.c:3722:d0 tracking VRAM f0000 - f0160
> >>(XEN) common.c:3722:d0 tracking VRAM f0000 - f0160

So to reproduce this, what should I be doing? Can you give me a rough idea of what
your guests are doing, how big the NFS files are, etc?

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2011-04-18 21:05 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-04-13 21:02 Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36 Gerd Jakobovitsch
2011-04-14 13:15 ` Konrad Rzeszutek Wilk
2011-04-14 16:38   ` Daniel Stodden
2011-04-14 17:32     ` Daniel Stodden
2011-04-14 17:43     ` Daniel Stodden
2011-04-14 18:05       ` Gerd Jakobovitsch
2011-04-14 23:42         ` Daniel Stodden
2011-04-15 14:21   ` xen 4.0.2rc3/kernel 2.6.32.36: BUG: unable to handle kernel paging request Gerd Jakobovitsch
2011-04-15 15:44     ` Pasi Kärkkäinen
2011-04-15 15:47       ` Gerd Jakobovitsch
2011-04-18 21:05         ` Konrad Rzeszutek Wilk

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).