From mboxrd@z Thu Jan 1 00:00:00 1970 From: Gerd Jakobovitsch Subject: xen 4.0.2rc3/kernel 2.6.32.36: BUG: unable to handle kernel paging request Date: Fri, 15 Apr 2011 11:21:40 -0300 Message-ID: <4DA85474.6060002@alog.com.br> References: <4DA60F55.4000604@alog.com.br> <20110414131543.GE5548@dumpdata.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------050805060306020307040706" Return-path: In-Reply-To: <20110414131543.GE5548@dumpdata.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --------------050805060306020307040706 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Reporting a new bug that appeared during stress tests. The scenario is the same as reported below, with patches applied: On 04/14/2011 10:15 AM, Konrad Rzeszutek Wilk wrote: > On Wed, Apr 13, 2011 at 06:02:13PM -0300, Gerd Jakobovitsch wrote: >> I'm trying to run several VMs (linux hvm, with tapdisk:aio disks at >> a storage over nfs) on a CentOS system, using the up-to-date version >> of xen 4.0 / kernel pvops 2.6.32.x stable. With a configuration >> without (most of) debug activated, I can start several instances - >> I'm running 7 of them - but shortly afterwards the system stops >> responding. I can't find any information on this. > First time I see it. >> Activating several debug configuration items, among them >> DEBUG_PAGEALLOC, I get an exception as soon as I try to start up a >> VM. The system reboots. >> With the debug information still set, I'm running 42 VMs - mixed Linux (several distros) and Windows, most of them running benchmarks for CPU and disk usage. After roughly 15 hours, a bug message appeared at dmesg. It affected xm commands - it seems to be related to a specific VM - but xl commands still work. VMs are running. # xm list Error: (5, 'Input/output error, while reading /local/domain/33/console/vnc-port') Usage: xm list [options] [Domain, ...] After killing the VM that reported error, xm commands are working again. The BUG message at dmesg: [66007.135552] BUG: unable to handle kernel paging request at ffff8800004ca458 [66007.135567] IP: [] xen_set_pte+0x3e/0x4b [66007.135580] PGD 1002067 PUD 1006067 PMD 2d78067 PTE 100000004ca025 [66007.135675] Oops: 0003 [#1] SMP DEBUG_PAGEALLOC [66007.135686] last sysfs file: /sys/class/net/virtbr/bridge/topology_change_detected [66007.135693] CPU 4 [66007.135698] Modules linked in: arptable_filter arp_tables bridge stp bonding bnx2i libiscsi scsi_transport_iscsi cnic uio bnx2 megaraid_sas [66007.135729] Pid: 683, comm: pageattr-test Not tainted 2.6.32.36 #7 PowerEdge M610 [66007.135735] RIP: e030:[] [] xen_set_pte+0x3e/0x4b [66007.135746] RSP: e02b:ffff88007c8edbb0 EFLAGS: 00010202 [66007.135751] RAX: 0000000000e32cb6 RBX: 0000000000e32cb6 RCX: 0000000000000001 [66007.135757] RDX: 0000000000000000 RSI: 8010000800569267 RDI: ffff8800004ca458 [66007.135764] RBP: ffff88007c8edbd0 R08: 0000000000000001 R09: 0000000000000000 [66007.135770] R10: ffffffff818385f8 R11: ffffffff818385e0 R12: 8010000800569267 [66007.135776] R13: ffff8800004ca458 R14: 8010000416569067 R15: 8010000800569267 [66007.135786] FS: 00007f0eeede66e0(0000) GS:ffff88002813f000(0000) knlGS:0000000000000000 [66007.135792] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b [66007.135797] CR2: ffff8800004ca458 CR3: 000000007b663000 CR4: 0000000000002660 [66007.135804] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [66007.135810] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [66007.135816] Process pageattr-test (pid: 683, threadinfo ffff88007c8ec000, task ffff88007e4ce480) [66007.135822] Stack: [66007.135825] 0000000000000000 8010000004569067 0000000000004569 ffff88007c8edd20 [66007.135835] <0> ffff88007c8edbe0 ffffffff81034740 ffff88007c8edbf0 ffffffff8103474d [66007.135848] <0> ffff88007c8edcf0 ffffffff81034e77 000000017c8edc40 ffffffff818385e0 [66007.135860] Call Trace: [66007.135868] [] set_pte+0x17/0x1b [66007.135875] [] set_pte_atomic+0x9/0xb [66007.135882] [] __change_page_attr_set_clr+0x186/0x82d [66007.135936] [] ? _raw_spin_unlock+0xab/0xb1 [66007.135951] [] ? _spin_unlock+0x26/0x2a [66007.135961] [] ? vm_unmap_aliases+0x151/0x160 [66007.135969] [] change_page_attr_set_clr+0x177/0x360 [66007.135976] [] change_page_attr_set+0x27/0x29 [66007.135983] [] ? pte_flags+0x9/0x18 [66007.135990] [] do_pageattr_test+0x285/0x4b1 [66007.135998] [] ? do_pageattr_test+0x0/0x4b1 [66007.136097] [] kthread+0x69/0x71 [66007.136105] [] child_rip+0xa/0x20 [66007.136112] [] ? int_ret_from_sys_call+0x7/0x1b [66007.136119] [] ? retint_restore_args+0x5/0x6 [66007.136127] [] ? child_rip+0x0/0x20 [66007.136131] Code: e8 3c ff ff ff ff 05 b6 5c 94 00 e8 31 ff ff ff 8b 1d b3 5c 94 00 e8 a2 23 02 00 ff c8 0f 94 c0 0f b6 c0 01 d8 89 05 9e 5c 94 00 <4d> 89 65 00 41 59 5b 41 5c 41 5d c9 c3 55 48 89 e5 53 89 fb 48 [66007.136273] RIP [] xen_set_pte+0x3e/0x4b [66007.136281] RSP [66007.136285] CR2: ffff8800004ca458 [66007.136574] ---[ end trace 4e200a271895cc90 ]--- Attached errors registered in xm dmesg and xend.log. --------------050805060306020307040706 Content-Type: text/plain; name="bug_paging_xend-log.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="bug_paging_xend-log.txt" [2011-04-15 05:32:06 4415] ERROR (xmlrpclib2:181) Internal error handling xend.domain Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line 134, in _marshaled_dispatch response = self._dispatch(method, params) File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line 406, in _dispatch return func(*params) File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 58, in domain info = lookup(domid) File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 50, in lookup info = XendDomain.instance().domain_lookup(domid) File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 543, in domain_lookup self._refresh(refresh_shutdown = False) File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 417, in _refresh self._refreshTxn(txn, refresh_shutdown) File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 430, in _refreshTxn self.domains[domid].update(dom, refresh_shutdown, transaction) File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 3523, in update self._update_consoles(transaction) File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1656, in _update_consoles vnc_port = self.readDomTxn(transaction, 'console/vnc-port') File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1720, in readDomTxn return transaction.read(*paths) File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 72, in read return self._read(args[0]) File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 59, in _read '%s, while reading %s' % (ex.args[1], path)) RuntimeError: (5, 'Input/output error, while reading /local/domain/33/console/vnc-port') [2011-04-15 05:32:12 4415] ERROR (xmlrpclib2:181) Internal error handling xend.domain Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line 134, in _marshaled_dispatch response = self._dispatch(method, params) File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line 406, in _dispatch return func(*params) File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 58, in domain info = lookup(domid) File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 50, in lookup info = XendDomain.instance().domain_lookup(domid) File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 543, in domain_lookup self._refresh(refresh_shutdown = False) File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 417, in _refresh self._refreshTxn(txn, refresh_shutdown) File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 430, in _refreshTxn self.domains[domid].update(dom, refresh_shutdown, transaction) File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 3523, in update self._update_consoles(transaction) File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1656, in _update_consoles vnc_port = self.readDomTxn(transaction, 'console/vnc-port') File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1720, in readDomTxn return transaction.read(*paths) File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 72, in read return self._read(args[0]) File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 59, in _read '%s, while reading %s' % (ex.args[1], path)) RuntimeError: (5, 'Input/output error, while reading /local/domain/33/console/vnc-port') [2011-04-15 05:32:17 4415] ERROR (xmlrpclib2:181) Internal error handling xend.domain Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line 134, in _marshaled_dispatch response = self._dispatch(method, params) File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line 406, in _dispatch return func(*params) File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 58, in domain info = lookup(domid) File "/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py", line 50, in lookup info = XendDomain.instance().domain_lookup(domid) File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 543, in domain_lookup self._refresh(refresh_shutdown = False) File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 417, in _refresh self._refreshTxn(txn, refresh_shutdown) File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 430, in _refreshTxn self.domains[domid].update(dom, refresh_shutdown, transaction) File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 3523, in update self._update_consoles(transaction) File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1656, in _update_consoles vnc_port = self.readDomTxn(transaction, 'console/vnc-port') File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1720, in readDomTxn return transaction.read(*paths) File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 72, in read return self._read(args[0]) File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line 59, in _read '%s, while reading %s' % (ex.args[1], path)) --------------050805060306020307040706 Content-Type: text/plain; name="bug_paging_xm_dmesg.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="bug_paging_xm_dmesg.txt" (XEN) HVM43: Booting from 0000:7c00 (XEN) HVM44: Press F12 for boot menu. (XEN) HVM44: (XEN) HVM44: Booting from Hard Disk... (XEN) HVM44: Booting from 0000:7c00 (XEN) HVM44: int13_harddisk: function 42. LBA out of range (XEN) stdvga.c:151:d43 leaving stdvga (XEN) stdvga.c:147:d43 entering stdvga and caching modes (XEN) irq.c:243: Dom43 PCI link 0 changed 5 -> 0 (XEN) irq.c:243: Dom43 PCI link 1 changed 10 -> 0 (XEN) irq.c:243: Dom43 PCI link 2 changed 11 -> 0 (XEN) irq.c:243: Dom43 PCI link 3 changed 5 -> 0 (XEN) stdvga.c:151:d43 leaving stdvga (XEN) stdvga.c:151:d42 leaving stdvga (XEN) mm.c:3617:d0 PTE entry 81244b for address ffff88007caa0000 doesn't match frame 621c4b (XEN) mm.c:3617:d0 PTE entry 812bed for address ffff88007caa1000 doesn't match frame 6217ed (XEN) mm.c:3617:d0 PTE entry 812426 for address ffff88007caa2000 doesn't match frame 621c26 (XEN) mm.c:3617:d0 PTE entry 812590 for address ffff88007caa3000 doesn't match frame 621d90 (XEN) mm.c:3617:d0 PTE entry 812591 for address ffff88007caa4000 doesn't match frame 621d91 (XEN) mm.c:3617:d0 PTE entry 812492 for address ffff88007caa5000 doesn't match frame 621c92 (XEN) mm.c:3617:d0 PTE entry 812493 for address ffff88007caa6000 doesn't match frame 621c93 (XEN) mm.c:3617:d0 PTE entry 8124ca for address ffff88007caa7000 doesn't match frame 621cca (XEN) mm.c:3617:d0 PTE entry 8124cb for address ffff88007caa8000 doesn't match frame 621ccb (XEN) mm.c:3617:d0 PTE entry 80e8b0 for address ffff88007caa9000 doesn't match frame 6258b0 (XEN) mm.c:3617:d0 PTE entry 80e8b1 for address ffff88007caaa000 doesn't match frame 6258b1 (XEN) mm.c:3617:d0 PTE entry 803fe2 for address ffff88007caab000 doesn't match frame 5cfde2 (XEN) mm.c:3617:d0 PTE entry 80b3fd for address ffff88007ca99000 doesn't match frame 46b1fd (XEN) mm.c:3617:d0 PTE entry 80b3fe for address ffff88007ca9a000 doesn't match frame 46b1fe (XEN) mm.c:3617:d0 PTE entry 80ac10 for address ffff88007ca9b000 doesn't match frame 46b610 (XEN) mm.c:3617:d0 PTE entry 80c01e for address ffff88007ca9c000 doesn't match frame 46a21e (XEN) mm.c:3617:d0 PTE entry 80ab16 for address ffff88007ca9d000 doesn't match frame 46b916 (XEN) mm.c:3617:d0 PTE entry 80b0f6 for address ffff88007ca9e000 doesn't match frame 46b2f6 (XEN) mm.c:3617:d0 PTE entry 809c8c for address ffff88007ca9f000 doesn't match frame 46c68c (XEN) mm.c:3617:d0 PTE entry 8017e0 for address ffff88007cb9f000 doesn't match frame 5b19e0 (XEN) mm.c:3617:d0 PTE entry 8017e1 for address ffff88007cba0000 doesn't match frame 5b19e1 (XEN) mm.c:3617:d0 PTE entry 801fc4 for address ffff88007cba1000 doesn't match frame 5b11c4 (XEN) mm.c:3617:d0 PTE entry 801fc5 for address ffff88007cba2000 doesn't match frame 5b11c5 (XEN) mm.c:3617:d0 PTE entry 801f2a for address ffff88007cba3000 doesn't match frame 5b112a (XEN) mm.c:3617:d0 PTE entry 801f2b for address ffff88007cba4000 doesn't match frame 5b112b (XEN) mm.c:3617:d0 PTE entry 801ee8 for address ffff88007cba5000 doesn't match frame 5b10e8 (XEN) mm.c:3617:d0 PTE entry 801ee9 for address ffff88007cba6000 doesn't match frame 5b10e9 (XEN) mm.c:3617:d0 PTE entry 803fde for address ffff88007cba7000 doesn't match frame 5af1de (XEN) mm.c:3617:d0 PTE entry 8120ec for address ffff88007cb9e000 doesn't match frame 9660ec (XEN) mm.c:3617:d0 PTE entry 8120ed for address ffff88007cb9f000 doesn't match frame 9660ed (XEN) mm.c:3617:d0 PTE entry 810e0a for address ffff88007cba0000 doesn't match frame 96720a (XEN) mm.c:3617:d0 PTE entry 810e0b for address ffff88007cba1000 doesn't match frame 96720b (XEN) mm.c:3617:d0 PTE entry 81080e for address ffff88007cba2000 doesn't match frame 96780e (XEN) mm.c:3617:d0 PTE entry 81080f for address ffff88007cba3000 doesn't match frame 96780f (XEN) mm.c:3617:d0 PTE entry 80ef3e for address ffff88007cba4000 doesn't match frame 96933e (XEN) mm.c:3617:d0 PTE entry 802fd7 for address ffff88007c433000 doesn't match frame 2111d7 (XEN) mm.c:3617:d0 PTE entry 805131 for address ffff88007c434000 doesn't match frame 20ef31 (XEN) mm.c:3617:d0 PTE entry 80c1ea for address ffff88007c435000 doesn't match frame 207fea (XEN) mm.c:3617:d0 PTE entry 814ae9 for address ffff88007c9b0000 doesn't match frame 8866e9 (XEN) mm.c:3617:d0 PTE entry 8113b9 for address ffff88007ca20000 doesn't match frame 9277b9 (XEN) mm.c:3617:d0 PTE entry 808ab2 for address ffff88007ca21000 doesn't match frame 92feb2 (XEN) mm.c:3617:d0 PTE entry 803bb3 for address ffff88007ca22000 doesn't match frame 934fb3 (XEN) mm.c:3617:d0 PTE entry 814747 for address ffff88007ca23000 doesn't match frame 924347 (XEN) mm.c:3617:d0 PTE entry 819873 for address ffff88007ca24000 doesn't match frame 91f073 (XEN) mm.c:3617:d0 PTE entry 80302c for address ffff88007ca25000 doesn't match frame 93582c (XEN) mm.c:3617:d0 PTE entry 81b75c for address ffff88007ca26000 doesn't match frame 91d35c (XEN) mm.c:3617:d0 PTE entry c5fd0b for address ffff88007ca86000 doesn't match frame 95830b (XEN) mm.c:3617:d0 PTE entry 81a093 for address ffff88007ca87000 doesn't match frame 9bde93 (XEN) mm.c:3617:d0 PTE entry 815f11 for address ffff88007ca88000 doesn't match frame 942111 (XEN) mm.c:3617:d0 PTE entry 815bc5 for address ffff88007ca89000 doesn't match frame 9425c5 (XEN) mm.c:3617:d0 PTE entry 816875 for address ffff88007cb9b000 doesn't match frame 59c675 (XEN) mm.c:3617:d0 PTE entry 8158a8 for address ffff88007cb9c000 doesn't match frame 59d6a8 (XEN) mm.c:3617:d0 PTE entry 82fa1c for address ffff88007c926000 doesn't match frame 5e321c (XEN) mm.c:3617:d0 PTE entry 82655e for address ffff88007c927000 doesn't match frame 5ec95e (XEN) mm.c:3617:d0 PTE entry c5ce72 for address ffff88007c926000 doesn't match frame 95b072 (XEN) mm.c:3617:d0 PTE entry c5cea9 for address ffff88007c927000 doesn't match frame 95b0a9 (XEN) mm.c:2399:d0 Bad type (saw 1c00000000000002 != exp 7000000000000000) for mfn 800569 (pfn 4569) (XEN) mm.c:878:d0 Error getting mfn 800569 (pfn 4569) from L1 entry 8010000800569267 for l1e_owner=0, pg_owner=0 (XEN) mm.c:4664:d0 ptwr_emulate: could not get_page_from_l1e() (XEN) mm.c:3617:d0 PTE entry 80e284 for address ffff88007ca89000 doesn't match frame 468084 (XEN) mm.c:3617:d0 PTE entry 817918 for address ffff88007ca8a000 doesn't match frame 47f118 (XEN) common.c:3722:d0 tracking VRAM f0000 - f0160 (XEN) common.c:3722:d0 tracking VRAM f0000 - f0160 --------------050805060306020307040706 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --------------050805060306020307040706--