From mboxrd@z Thu Jan 1 00:00:00 1970 From: Kip Macy Subject: Re: grant table unmap failure makes guest unreapable and causes xen oops Date: Wed, 6 Jul 2005 11:18:07 -0700 Message-ID: References: Reply-To: Kip Macy Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: Content-Disposition: inline List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: xen-devel List-Id: xen-devel@lists.xenproject.org It looks like some of these problems may have been fixed by check-ins in the last couple of hours. I'm doing a make world right now. -Kip =20 On 7/6/05, Kip Macy wrote: > I just hit this so I don't fully understand it yet, but it looks like > there may be some race condition with grant_table unmap requests and > garbage collection of domain memory on crashed guests. >=20 > My centos4 domU isn't finding its init (this may be the breakage in > file-backed VBDs that Mark mentioned - it was finding it a couple of > days ago) and thus calls HYPERVISOR_crash: >=20 > Freeing unused kernel memory: 92k freed > Kernel panic - not syncing: No init found. Try passing init=3D option to= kernel. >=20 > [root@rs0 ~]# xm list > Name Id Mem(MB) CPU VCPU(s) State Time(s) Console > Domain-0 0 251 0 1 r---- 1013= .3 > rhel4_0 1 0 3 1 ----c > 0.5 9601 >=20 > The following errors show up on the console: >=20 > (XEN) (file=3Dgrant_table.c, line=3D500) Bad handle (0). > (XEN) (file=3Dgrant_table.c, line=3D500) Bad handle (49152). > (XEN) (file=3Dgrant_table.c, line=3D500) Bad handle (49792). > (XEN) (file=3Dgrant_table.c, line=3D500) Bad handle (0). > (XEN) (file=3Dgrant_table.c, line=3D500) Bad handle (61440). >=20 > And the guest never goes away. >=20 > [root@rs0 ~]# xm destroy 1 > [root@rs0 ~]# xm list > Name Id Mem(MB) CPU VCPU(s) State Time(s) Console > Domain-0 0 251 0 1 r---- 1208.= 2 > rhel4_0 1 0 3 1 > ----c 0.5 9601 >=20 > restarting xend here is interesting: >=20 > [root@rs0 ~]# xend start > DBMap>introduceDomain> 1 69067 > /domain/4042ebcc-778d-4488-a0bd-6152c42ba98b > Traceback (most recent call last): > > RuntimeError: (9, 'Bad file descriptor') >=20 > Message from syslogd@rs0 at Wed Jul 6 10:59:17 2005 ... > rs0 xenstored: xenstored corruption: connection id 0: err Bad address: > Unknown error 14 (Bad address) > Exception starting xend: (9, 'Bad file descriptor') >=20 > On the console we see: > (XEN) (file=3D/build/kmacy/xen/xen-unstable.hg/xen/include/asm/mm.h, > line=3D187) Error pfn be9: rd=3Dffbf8a80, od=3Dffbf8a80, caf=3D00000000, > taf=3Df0000001 > (XEN) (file=3D/build/kmacy/xen/xen-unstable.hg/xen/include/asm/mm.h, > line=3D187) Error pfn 10dcb: rd=3Dffbf8a80, od=3D00000000, caf=3D00000000= , > taf=3Df0000000 > [ERR] corruptxenstored corruption: connection id 0: err Bad address: > Unknown error 14 (Bad address) >=20 > *NOW* comes the fun part: > [root@rs0 ~]# /sbin/shutdown -r now >=20 > Broadcast message from root (pts/1) (Wed Jul 6 11:01:00 2005): >=20 > The system is going down for reboot NOW! > INIT: Sending processes the TERM signal > (XEN) CPU: 0 > (XEN) EIP: e008:[] > (XEN) EFLAGS: 00210202 CONTEXT: hypervisor > (XEN) eax: 0000000a ebx: 00000000 ecx: 00000000 edx: 00000003 > (XEN) esi: 00000001 edi: ffbf2700 ebp: ffbf1004 esp: ff103e04 > (XEN) cr0: 8005003b cr3: 181cd000 > (XEN) ds: e010 es: e010 fs: 0000 gs: 0000 ss: e010 cs: e008 > (XEN) Xen stack trace from esp=3Dff103e04: > (XEN) 00000001 00000052 00000000 00000400 fec6b000 ff1a9900 fc400000 0= 0000f00 > (XEN) 00000000 00000000 00000001 000a0067 fc400f00 ff1a9900 > ff1a7080 [ff1269de] > (XEN) ff1a7080 ff1a9900 000000a0 00000000 fec6b000 32ff0001 ff1a9900 0= 000067c > (XEN) 00000261 000a0067 fec72000 [ff12a110] 000a0067 ff1a9900 > 00000000 [ff13b9ef] > (XEN) 181cd000 ff103fb4 00000001 c7910985 ff1a9b80 fec72000 > ff1a9900 [ff12a266] > (XEN) ff1a9900 fec72000 ff1b2000 [ff12ba14] fec71000 ff1a9b80 > ff103fb4 ff1a9b90 > (XEN) c7910985 fe31e440 00000000 00000000 00000000 00000000 > ff1a9900 [ff12d55e] > (XEN) ff1a9900 00000000 0000000c 00200286 ff103fb4 ff1a9900 > [ff13b9ef] 181cd000 > (XEN) 001446c9 00000000 c7910984 ff1a9900 00018ef0 18ef0061 > [ff12eb6a] 00018ef0 > (XEN) ffffffff 00000010 ff1a9900 00000007 c873e000 00010000 c0568ee0 0= 00002db > (XEN) 32db0001 ff103fb4 ff1a9b80 ff1a9900 00000000 fe3f8b6c ff1a9900 f= f103fb4 > (XEN) c7910984 ffbf3080 [ff13e867] 00000000 00000000 00000000 > 00000000 00000001 > (XEN) 00000005 00000020 ee000000 ffbf3080 ffbf3bf8 ffbf3080 ffbf3080 f= fbf3080 > (XEN) 00007ff0 c8623284 b6e69000 [ff14a8f3] c8683eec 00000001 > 00000000 00007ff0 > (XEN) c8623284 b6e69000 0000001a 000e0003 c0115b33 00000061 00200282 c= 8683eec > (XEN) 00000069 0000007b 0000007b 00000000 00000000 00000000 ffbf3080 > (XEN) Xen call trace from esp=3Dff103e04: > (XEN) [] [] [] [] > [] [] > (XEN) [] [] [] [] >=20 > **************************************** > Panic on CPU0: > CPU0 FATAL PAGE FAULT > [error_code=3D0000] > Faulting linear address: 00000004 > **************************************** >=20 > Reboot in five seconds... >=20 >=20 > Line 910 of "grant_table.c" starts at address 0xff10b87f > > and ends at 0xff10b888 . > <...> > ( readonly ? 1 : (!(map->ref_and_flags & GNTMAP_readonly)))) > { > ref =3D (map->ref_and_flags >> MAPTRACK_REF_SHIFT); > act =3D &rgt->active[ref]; <- line 910 >=20 > spin_lock(&rgt->lock); >=20 > if ( act->frame !=3D frame ) > <...> > 0xff10b882 : mov 0x4(%ecx),%eax > 0xff1269de : test %eax,%eax > 0xff12a110 : jmp 0xff12a090 > 0xff13b9ef <__flush_tlb_mask+239>: mov 0x44(%ebx),%eax > 0xff12a266 : mov %edi,(%esp) > 0xff12d55e : jmp 0xff12d17a > 0xff13b9ef <__flush_tlb_mask+239>: mov 0x44(%ebx),%eax > 0xff12eb6a : mov %eax,%esi > 0xff13e867 : test %eax,%eax > 0xff14a8f3 : mov %eax,0x18(%esp) >