From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: BUG at xen4.1/kernel 2.6.32.35 at a CentOS 5.5 when starting a VM Date: Tue, 12 Apr 2011 06:59:46 -0400 Message-ID: <20110412105946.GA23127@dumpdata.com> References: <4D933ADB.8060106@alog.com.br> <4D9F247C.6030801@alog.com.br> <4D9F805E.2020806@alog.com.br> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: Content-Disposition: inline In-Reply-To: <4D9F805E.2020806@alog.com.br> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Gerd Jakobovitsch Cc: xen-devel@lists.xensource.com, Teck Choon Giam List-Id: xen-devel@lists.xenproject.org On Fri, Apr 08, 2011 at 06:38:38PM -0300, Gerd Jakobovitsch wrote: > One more follow-up: >=20 > Another kernel bug report, with no kernel debug activated: This is just with the guest starting right? Not running for a long time? >=20 > r2b16ch2x28p2 kernel: [ 3243.777796] CR2: 00000000000002f0 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.761622] BUG: unable to > handle kernel NULL pointer dereference at 00000000000002f0 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.761781] IP: > [] blktap_device_end_request+0x4e/0x6c > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.761892] PGD 710a3067 > PUD 724c6067 PMD 0 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.762076] Oops: 0000 [#1] SM= P > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.762212] last sysfs > file: /sys/devices/vbd-6-51712/statistics/wr_sect > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.762271] CPU 5 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.762363] Modules linked > in: bnx2 xt_mac bridge stp nfs fscache nfs_acl auth_rpcgss > arptable_filter arp_tables xt_esp ipt_ah xt_physdev xt_multiport > lockd sunrpc bonding dm_multipath megaraid_sas [last unloaded: bnx2] > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.763350] Pid: 7781, > comm: tapdisk2 Not tainted 2.6.32.36 #5 PowerEdge M610 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.763410] RIP: > e030:[] [] > blktap_device_end_request+0x4e/0x6c > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.763519] RSP: > e02b:ffff88006ed49cf8 EFLAGS: 00010046 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.763574] RAX: > 0000000000000000 RBX: ffff88005e6fc3e0 RCX: ffffffff811a3de6 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.763635] RDX: > ffff88005e6fc3e0 RSI: ffffffff8149bed6 RDI: ffff880070f42178 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.763694] RBP: > ffff880070f42010 R08: ffffffff81661840 R09: 00000001002c9435 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.763971] R10: > 0000000000000000 R11: ffff88005e6378f0 R12: ffff88005e6378f0 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.764244] R13: > ffff880070f42000 R14: 0000000000000000 R15: 0000000000000001 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.764522] FS: > 00007fcfd34b0730(0000) GS:ffff880015fe7000(0000) > knlGS:0000000000000000 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.765011] CS: e033 DS: > 0000 ES: 0000 CR0: 000000008005003b > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.765280] CR2: > 00000000000002f0 CR3: 000000006ed46000 CR4: 0000000000002660 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.765554] DR0: > 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.765829] DR3: > 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.766105] Process > tapdisk2 (pid: 7781, threadinfo ffff88006ed48000, task > ffff880079486270) > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.766602] Stack: > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.766868] > ffff88007a81dee0 ffff880079486270 0000000000000000 0000000000000001 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.767266] <0> > 0000000000000001 ffffffff8121e0b9 0000000000000003 0000000000000000 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.767967] <0> > ffff880070f42000 0000bda50000bda7 ffff88005e6fc3e0 fffffffd00000000 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.768921] Call Trace: > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.769193] > [] ? blktap_ring_ioctl+0x159/0x290 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.769474] > [] ? error_exit+0x2a/0x60 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.769750] > [] ? retint_restore_args+0x5/0x6 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.770030] > [] ? hypercall_page+0x3aa/0x1001 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.770308] > [] ? hypercall_page+0x3aa/0x1001 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.770592] > [] ? selinux_file_ioctl+0x0/0x3d > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.770874] > [] ? xen_force_evtchn_callback+0x9/0xa > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.771157] > [] ? check_events+0x12/0x20 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.771435] > [] ? selinux_file_ioctl+0x0/0x3d > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.771715] > [] ? xen_restore_fl_direct_end+0x0/0x1 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.771993] > [] ? xen_spin_lock_slow+0xb7/0xf8 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.772275] > [] ? vfs_ioctl+0x55/0x6b > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.772554] > [] ? do_vfs_ioctl+0x492/0x4e5 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.772834] > [] ? xen_restore_fl_direct_end+0x0/0x1 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.773116] > [] ? sys_ioctl+0x51/0x70 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.773392] > [] ? system_call_fastpath+0x16/0x1b > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.773671] Code: e8 39 f6 > ff ff 49 8b 44 24 40 48 8b b8 f0 02 00 00 e8 e6 d6 27 00 4c 89 e7 41 > 8b 54 24 60 44 89 f6 e8 b9 59 f8 ff 49 8b 44 24 40 <48> 8b b8 f0 02 > 00 00 e8 02 14 df ff 66 90 ff 14 25 78 8d 65 81 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.777198] RIP > [] blktap_device_end_request+0x4e/0x6c > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.777524] RSP > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.777796] CR2: 0000000000000= 2f0 > Apr 8 18:41:58 r2b16ch2x28p2 kernel: [ 3243.778067] ---[ end trace > a71b80c14de09da1 ]--- >=20 > On 04/08/2011 12:44 PM, Teck Choon Giam wrote: > > > > > >On Fri, Apr 8, 2011 at 11:06 PM, Gerd Jakobovitsch > >> wrote: > > > > On 03/30/2011 11:44 PM, Teck Choon Giam wrote: > >> On Wed, Mar 30, 2011 at 10:14 PM, Gerd Jakobovitsch wrote: > >>> Hello all, > >>> > >>> I used to run xen4.0 kernel 2.6.32.24 over CentOS 5.5, with a re= lative success, but the bug at mmu.c appeared once at a while. Therefore,= I'm looking for a more stable option. > >>> I compiled and ran the newly released xen 4.1, with kernel PVOPS= 2.6.32.35 over CentOS 5.5. When trying to start a VM, the following bugs= appeared at dmesg. After that, xl and xm commands do not longer respond: > >>> > >>> [ 145.749573] alloc irq_desc for 2209 on node -1 > >>> [ 145.749581] alloc kstat_irqs on node -1 > >>> [ 145.883515] block tda: sector-size: 512 capacity: 262144 > >>> [ 145.889952] general protection fault: 0000 [#1] SMP > >>> [ 145.890109] last sysfs file: /sys/block/tda/removable > >>> [ 145.890164] CPU 7 > >>> [ 145.890252] Modules linked in: bridge stp nfs fscache nfs_acl= auth_rpcgss arptable_filter arp_tables xt_esp ipt_ah xt_physdev xt_multi= port lockd sunrpc bonding dm_multipath bnx2 megaraid_sas > >>> [ 145.891125] Pid: 5179, comm: tapdisk2 Not tainted 2.6.32.35 #= 1 PowerEdge M610 > >>> [ 145.891184] RIP: e030:[] [] blktap_device_end_request+0x4e/0x63 > >>> [ 145.891296] RSP: e02b:ffff880064061cd8 EFLAGS: 00010046 > >>> [ 145.891351] RAX: 6b6b6b6b6b6b6b6b RBX: ffff88007d264690 RCX: = 0000000000000028 > >>> [ 145.891410] RDX: 0000000000000000 RSI: 0000000000000000 RDI: = 0000000000000000 > >>> [ 145.891469] RBP: ffff880064061cf8 R08: 0000000064061c98 R09: = ffff88007da42948 > >>> [ 145.891528] R10: ffffea0000000008 R11: 0000000001f60080 R12: = ffff88007da427f8 > >>> [ 145.891587] R13: ffff88007c75f398 R14: 0000000000000000 R15: = ffff88007c75f3a8 > >>> [ 145.891651] FS: 00007ff33d9a4730(0000) GS:ffff8800189e5000(0= 000) knlGS:0000000000000000 > >>> [ 145.891714] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > >>> [ 145.891771] CR2: 0000000002594cc8 CR3: 000000007be61000 CR4: = 0000000000002660 > >>> [ 145.891830] DR0: 0000000000000000 DR1: 0000000000000000 DR2: = 0000000000000000 > >>> [ 145.891890] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: = 0000000000000400 > >>> [ 145.892171] Process tapdisk2 (pid: 5179, threadinfo ffff88006= 4060000, task ffff88007c272d60) > >>> [ 145.892669] Stack: > >>> [ 145.892934] ffff88007c272d60 0000000000000000 00000000000000= 00 0000000000000000 > >>> [ 145.893334]<0> ffff880064061e88 ffffffff812815ae ffff8800640= 61e58 ffffffff811d234f > >>> [ 145.894035]<0> ffff88007e9bbfc0 ffff88007c75f398 00000001fff= fffff 0000000000000000 > >>> [ 145.895015] Call Trace: > >>> [ 145.895286] [] blktap_ring_ioctl+0x183/0x2= d8 > >>> [ 145.895566] [] ? inode_has_perm+0x77/0x89 > >>> [ 145.895844] [] ? inode_has_perm+0x77/0x89 > >>> [ 145.896124] [] ? _raw_spin_lock+0x77/0x12f > >>> [ 145.896403] [] ? _raw_spin_unlock+0xab/0xb= 2 > >>> [ 145.896682] [] ? _spin_unlock+0x9/0xb > >>> [ 145.896958] [] ? _raw_spin_lock+0x77/0x12f > >>> [ 145.897234] [] ? file_has_perm+0xb4/0xc6 > >>> [ 145.897513] [] vfs_ioctl+0x5e/0x77 > >>> [ 145.897786] [] do_vfs_ioctl+0x484/0x4d5 > >>> [ 145.898060] [] sys_ioctl+0x57/0x7a > >>> [ 145.898338] [] system_call_fastpath+0x16/0= x1b > >>> [ 145.898614] Code: e8 5f f4 ff ff 49 8b 44 24 40 48 8b b8 80 0= 3 00 00 e8 64 75 2a 00 41 8b 54 24 60 44 89 f6 4c 89 e7 e8 b5 89 f7 ff 49= 8b 44 24 40<48> 8b b8 80 03 00 00 e8 23 74 2a 00 5b 41 5c 41 5d 41 5e c= 9 c3 > >>> [ 145.902008] RIP [] blktap_device_end_reque= st+0x4e/0x63 > >>> [ 145.902321] RSP > >>> [ 145.902585] ---[ end trace 2800cfa5aa85ca0a ]--- > >>> [ 262.100689] BUG: spinlock lockup on CPU#4, vol_id/5181, ffff8= 8007c75f520 > >>> [ 262.100965] Pid: 5181, comm: vol_id Tainted: G D 2.6.= 32.35 #1 > >>> [ 262.101232] Call Trace: > >>> [ 262.101497] [] _raw_spin_lock+0x101/0x12f > >>> [ 262.101762] [] _spin_lock_irq+0x1e/0x20 > >>> [ 262.102028] [] __make_request+0x5e/0x402 > >>> [ 262.102294] [] ? xen_restore_fl_direct_end= +0x0/0x1 > >>> [ 262.102563] [] generic_make_request+0x258/= 0x2f4 > >>> [ 262.102832] [] ? bio_init+0x18/0x32 > >>> [ 262.103099] [] submit_bio+0xd0/0xd9 > >>> [ 262.103366] [] submit_bh+0xf7/0x11a > >>> [ 262.103631] [] block_read_full_page+0x246/= 0x264 > >>> [ 262.103898] [] ? blkdev_get_block+0x0/0x4d > >>> [ 262.104165] [] ? _spin_unlock_irq+0x1e/0x2= 0 > >>> [ 262.104433] [] ? add_to_page_cache_locked+= 0xa0/0xca > >>> [ 262.104702] [] blkdev_readpage+0x13/0x15 > >>> [ 262.104972] [] __do_page_cache_readahead+0= x144/0x177 > >>> [ 262.105240] [] ondemand_readahead+0x126/0x= 18e > >>> [ 262.105507] [] page_cache_sync_readahead+0= x38/0x3a > >>> [ 262.105778] [] generic_file_aio_read+0x24c= /0x5c1 > >>> [ 262.106045] [] do_sync_read+0xe2/0x126 > >>> [ 262.106315] [] ? autoremove_wake_function+= 0x0/0x38 > >>> [ 262.106584] [] ? selinux_file_permission+0= x5c/0x10e > >>> [ 262.106854] [] ? security_file_permission+= 0x11/0x13 > >>> [ 262.107120] [] vfs_read+0xab/0x167 > >>> [ 262.107385] [] sys_read+0x47/0x70 > >>> [ 262.107652] [] system_call_fastpath+0x16/0= x1b > >>> [ 262.107918] sending NMI to all CPUs: > >>> [ 262.108189] BUG: unable to handle kernel paging request at ff= ffffffff5fb310 > >>> [ 262.108526] IP: [] flat_send_IPI_mask+0x6a/= 0xc0 > >>> [ 262.108832] PGD 1003067 PUD 1004067 PMD 18b7067 PTE 0 > >>> [ 262.109235] Oops: 0002 [#2] SMP > >>> [ 262.109565] last sysfs file: /sys/class/blktap2/blktap1/dev > >>> [ 262.109830] CPU 4 > >>> [ 262.110121] Modules linked in: bridge stp nfs fscache nfs_acl= auth_rpcgss arptable_filter arp_tables xt_esp ipt_ah xt_physdev xt_multi= port lockd sunrpc bonding dm_multipath bnx2 megaraid_sas > >>> [ 262.111520] Pid: 5181, comm: vol_id Tainted: G D 2.6.= 32.35 #1 PowerEdge M610 > >>> [ 262.112008] RIP: e030:[] [] flat_send_IPI_mask+0x6a/0xc0 > >>> [ 262.112535] RSP: e02b:ffff88006778f968 EFLAGS: 00010086 > >>> [ 262.112800] RAX: 00000000ff000000 RBX: ffffffff81790060 RCX: = 00000000000160a0 > >>> [ 262.113068] RDX: ffff88001898e000 RSI: 0000000000000002 RDI: = ffffffff81816020 > >>> [ 262.113337] RBP: ffff88006778f988 R08: 0000000000000000 R09: = 0000000000000004 > >>> [ 262.113605] R10: 0000000000000002 R11: 0000000000000004 R12: = 0000000000000002 > >>> [ 262.113877] R13: 0000000000000800 R14: 00000000000000ff R15: = 0000000000000000 > >>> [ 262.114149] FS: 00007fa78bcc5710(0063) GS:ffff88001898e000(0= 000) knlGS:0000000000000000 > >>> [ 262.114636] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > >>> [ 262.114902] CR2: ffffffffff5fb310 CR3: 00000000641b4000 CR4: = 0000000000002660 > >>> [ 262.115171] DR0: 0000000000000000 DR1: 0000000000000000 DR2: = 0000000000000000 > >>> [ 262.115438] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: = 0000000000000400 > >>> [ 262.115707] Process vol_id (pid: 5181, threadinfo ffff8800677= 8e000, task ffff88007db86250) > >>> [ 262.116194] Stack: > >>> [ 262.116451] 0000000000000000 0000000076e9ecd0 00000000000000= 00 0000000076e9ecd0 > >>> [ 262.116825]<0> ffff88006778f998 ffffffff8102c841 ffff8800677= 8f9b8 ffffffff81029f0d > >>> [ 262.117485]<0> ffff88007c75f520 ffff88007c75f520 ffff8800677= 8f9f8 ffffffff81219eb3 > >>> [ 262.118396] Call Trace: > >>> [ 262.118657] [] flat_send_IPI_all+0x1a/0x56 > >>> [ 262.118925] [] arch_trigger_all_cpu_backtr= ace+0x45/0x66 > >>> [ 262.119195] [] _raw_spin_lock+0x106/0x12f > >>> [ 262.119463] [] _spin_lock_irq+0x1e/0x20 > >>> [ 262.119730] [] __make_request+0x5e/0x402 > >>> [ 262.119996] [] ? xen_restore_fl_direct_end= +0x0/0x1 > >>> [ 262.120264] [] generic_make_request+0x258/= 0x2f4 > >>> [ 262.120532] [] ? bio_init+0x18/0x32 > >>> [ 262.120799] [] submit_bio+0xd0/0xd9 > >>> [ 262.121066] [] submit_bh+0xf7/0x11a > >>> [ 262.121333] [] block_read_full_page+0x246/= 0x264 > >>> [ 262.121602] [] ? blkdev_get_block+0x0/0x4d > >>> [ 262.121870] [] ? _spin_unlock_irq+0x1e/0x2= 0 > >>> [ 262.122137] [] ? add_to_page_cache_locked+= 0xa0/0xca > >>> [ 262.127766] [] blkdev_readpage+0x13/0x15 > >>> [ 262.128025] [] __do_page_cache_readahead+0= x144/0x177 > >>> [ 262.128288] [] ondemand_readahead+0x126/0x= 18e > >>> [ 262.128548] [] page_cache_sync_readahead+0= x38/0x3a > >>> [ 262.128810] [] generic_file_aio_read+0x24c= /0x5c1 > >>> [ 262.129070] [] do_sync_read+0xe2/0x126 > >>> [ 262.129329] [] ? autoremove_wake_function+= 0x0/0x38 > >>> [ 262.129590] [] ? selinux_file_permission+0= x5c/0x10e > >>> [ 262.129851] [] ? security_file_permission+= 0x11/0x13 > >>> [ 262.130110] [] vfs_read+0xab/0x167 > >>> [ 262.130368] [] sys_read+0x47/0x70 > >>> [ 262.130624] [] system_call_fastpath+0x16/0= x1b > >>> [ 262.130883] Code: 8b 05 b4 95 7e 00 83 fe 02 44 8b 68 34 75 0= a ff 90 58 01 00 00 eb 0e f3 90 8b 04 25 00 b3 5f ff f6 c4 10 75 f2 44 89= f0 c1 e0 18<89> 04 25 10 b3 5f ff 41 83 fc 02 74 08 44 89 e0 44 09 e8 e= b 06 > >>> [ 262.133866] RIP [] flat_send_IPI_mask+0x6a= /0xc0 > >>> [ 262.134164] RSP > >>> [ 262.134419] CR2: ffffffffff5fb310 > >>> [ 262.134673] ---[ end trace 2800cfa5aa85ca0b ]--- > >>> > >> Can you try to recompile your PVOPS kernel with CONFIG_DEBUG_PAGE= ALLOC=3Dy? > >> > >> You can read more about this BUG at > >> http://lists.xensource.com/archives/html/xen-devel/2011-03/msg017= 56.html > >> > >> I initially hit this BUG sometime Dec 2010... ... > >> http://lists.xensource.com/archives/html/xen-devel/2010-12/msg015= 01.html > >> > >> Thanks. > >> > >> Kindest regards, > >> Giam Teck Choon > > Sorry for the delayed answer. The problem I'm facing now is not > > related to the mmu bug - that one I still am seeing at systems > > with xen 4.0.2 / kernel 2.6.32.24. Newer kernels have new bugs, > > much more troublesome, since I can not run a single VM instance. > > > > Adding DEBUG_PAGEALLOC, the main difference is that the system is > > rebooting shortly after trying to start up a VM: > > > > > >Ok. Sorry, didn't see your log message carefully previously. > > > > > > Apr 8 12:00:43 r2b16ch2x28p2 tapdisk2[15563]: I/O queue driver: l= io > > Apr 8 12:00:43 r2b16ch2x28p2 tapdisk2[15563]: received 'attach' > > message (uuid =3D 0) > > Apr 8 12:00:43 r2b16ch2x28p2 tapdisk2[15563]: sending 'attach > > response' message (uuid =3D 0) > > Apr 8 12:00:43 r2b16ch2x28p2 tapdisk2[15563]: received 'open' > > message (uuid =3D 0) > > Apr 8 12:00:43 r2b16ch2x28p2 tapdisk2[15563]: block-aio > > open('/storage5/linux-centos-5-64b-base-7253/hda') > > Apr 8 12:00:43 r2b16ch2x28p2 tapdisk2[15563]: > > open(/storage5/linux-centos-5-64b-base-7253/hda) with O_DIRECT > > Apr 8 12:00:43 r2b16ch2x28p2 tapdisk2[15563]: Image size: > >pre sector_shift [134217728] post sector_shift [262144] > > Apr 8 12:00:43 r2b16ch2x28p2 tapdisk2[15563]: opened image > > /storage5/linux-centos-5-64b-base-rip/hda (1 users, state: > > 0x00000001, type: 0) > > Apr 8 12:00:43 r2b16ch2x28p2 tapdisk2[15563]: VBD CHAIN: > > Apr 8 12:00:43 r2b16ch2x28p2 tapdisk2[15563]: > > /storage5/linux-centos-5-64b-base/hda: 0 > > Apr 8 12:00:43 r2b16ch2x28p2 tapdisk2[15563]: sending 'open > > response' message (uuid =3D 0) > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.644887] block tda: > > sector-size: 512 capacity: 262144 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.657328] general > > protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.657379] last sysfs > > file: /sys/block/tda/removable > > > > > >Just curious... ... what type of storage you are using for your VMs? > > > > > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.657400] CPU 0 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.657421] Modules > > linked in: nfs fscache nfs_acl auth_rpcgss bridge stp ocfs2 > > ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager > > ocfs2_stackglue configfs arptable_filter arp_tables xt_esp ipt_ah > > xt_physdev xt_multiport dm_round_robin lockd sunrpc crc32c bonding > > iscsi_tcp libiscsi_tcp bnx2i libiscsi scsi_transport_iscsi cnic > > uio dm_multipath bnx2 megaraid_sas > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.657736] Pid: 15566, > > comm: tapdisk2 Not tainted 2.6.32.36 #4 PowerEdge M610 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.657763] RIP: > > e030:[] [] > > blktap_device_end_request+0x4e/0x63 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.657808] RSP: > > e02b:ffff88006da5dcd8 EFLAGS: 00010046 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.657833] RAX: > > 6b6b6b6b6b6b6b6b RBX: ffff88006566c000 RCX: 0000000000000000 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.657860] RDX: > > 0000000000000000 RSI: 0000000000000000 RDI: ffff88006d8c7980 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.657887] RBP: > > ffff88006da5dcf8 R08: ffffffff817e66a0 R09: ffff88007d775790 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.658136] R10: > > ffffffff810ccfe4 R11: ffff8800280d8f60 R12: ffff88006720e7f8 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.658384] R13: > > ffff88006d8c77e0 R14: 0000000000000000 R15: ffff88006d8c77f0 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.658635] FS: > >00007f86d75a3730(0000) GS:ffff8800280c7000(0000) > > knlGS:0000000000000000 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.659107] CS: e033 DS: > > 0000 ES: 0000 CR0: 000000008005003b > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.659351] CR2: > > 0000000045614ed8 CR3: 000000006d911000 CR4: 0000000000002660 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.659600] DR0: > > 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.659847] DR3: > > 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.660100] Process > > tapdisk2 (pid: 15566, threadinfo ffff88006da5c000, task > > ffff88007d7750f0) > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.660574] Stack: > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.660807] > >ffff88007d7750f0 0000000000000000 0000000000000000 > >0000000000000000 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.661074] <0> > > ffff88006da5de88 ffffffff8129f2c0 ffffffff8100fedd 000000016da5ddc= 8 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.661574] <0> > > 00000000ffffffff ffff88006d8c77e0 00000001816fab27 000000000000000= 0 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.662287] Call Trace: > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.662522] > >[] blktap_ring_ioctl+0x183/0x2d8 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.662767] > >[] ? xen_force_evtchn_callback+0xd/0xf > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.663012] > >[] ? inode_has_perm+0xa1/0xb3 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.663260] > >[] ? xen_restore_fl_direct_end+0x0/0x1 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.663509] > >[] ? lock_release+0x1b8/0x1c3 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.663756] > >[] ? _raw_spin_unlock+0xab/0xb2 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.663999] > >[] ? _spin_unlock+0x26/0x2a > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.664248] > >[] ? aio_read_evt+0x87/0x13a > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.664493] > >[] ? aio_read_evt+0x11c/0x13a > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.664740] > >[] ? _raw_spin_lock+0x77/0x12f > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.664986] > >[] ? file_has_perm+0xb4/0xc6 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.665237] > >[] vfs_ioctl+0x5e/0x77 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.665480] > >[] do_vfs_ioctl+0x484/0x4d5 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.665726] > >[] sys_ioctl+0x57/0x7a > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.665970] > >[] system_call_fastpath+0x16/0x1b > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.666214] Code: e8 61 > > f4 ff ff 49 8b 44 24 40 48 8b b8 70 04 00 00 e8 79 67 2b 00 41 8b > > 54 24 60 44 89 f6 4c 89 e7 e8 76 3b f7 ff 49 8b 44 24 40 <48> 8b > > b8 70 04 00 00 e8 1d 65 2b 00 5b 41 5c 41 5d 41 5e c9 c3 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.667320] RIP > >[] blktap_device_end_request+0x4e/0x63 > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.667577] RSP > > > > Apr 8 12:00:43 r2b16ch2x28p2 kernel: [ 8879.668069] ---[ end > > trace da218b929afc63f7 ]--- > > Apr 8 12:00:43 r2b16ch2x28p2 tapdisk2[15584]: I/O queue driver: l= io > > Apr 8 12:00:48 r2b16ch2x28p2 tap-ctl: > > tap-err:tap_ctl_read_message: failure reading message > > Apr 8 12:00:48 r2b16ch2x28p2 tap-ctl: > > tap-err:tap_ctl_send_and_receive: failed to receive 'unknown' mess= age > > > > > >Looks like it is related to blktap/blktap2 drivers related issue > >to me... so you are right... ... this is different BUG from what I > >encountered as the logs are different. Sorry, didn't read your > >log carefully before replying previously. Have you try to use > >normal LVM for your VM to reproduce the BUG as I guess you are > >using different storage? > > > >Thanks. > > > >Kindest regards, > >Giam Teck Choon >=20 > --=20 >=20 > *Gerd Jakobovitsch > Engenheiro de Produto ** > ---------------------------------------------------------* * > **ALOG Data Centers do Brasil** > **Excel=EAncia em Projetos de Hosting* > Rua Dr. Miguel Couto, 58 -- 01008-010 -- S=E3o Paulo - SP > Telefone: (11) 3524-4970 / (11) 7152-0815 > *http://www.alog.com.br* >=20 >=20 > *"Como est=E3o nossos servi=E7os? Clique aqui > e nos conte. Queremos > escutar a sua opini=E3o!"* >=20 >=20 >=20 >=20 >=20 > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel