domU crashing, trying to determine why

domU crashing, trying to determine why

[Micah Anderson]

[-- Attachment #1.1: Type: text/plain, Size: 2234 bytes --]


I'm having a domU that is regularly crashing, without any obvious reason
why. I set 'on_crash=preserve' to try and get a crash to try and get
some clues, but what I got doesn't too obvious to me. I was hoping
someone here would be able to give me some clues. 

Thanks for any clues anyone can give! Here is what I managed to eke out:

root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 0
eip: c01013a7 hypercall_page+0x3a7 flags: 00001246 i z p
esp: c0389fbc
eax: 00000000	ebx: 00000001	ecx: 00000000	edx: c0389fc8
esi: 00000000	edi: 00000000	ebp: 00000000
 cs: 00000061	 ds: 0000007b	 fs: 000000d8	 gs: 00000000

Stack:
 c0105f52 ffffffff 00000000 c06ca9dd 0009cf8a 00000000 00000020 ffffffff
 c01028ab c0102810 00000020 c1631000 00000000 c038f88b c03ac580 0702080b
 00000000

Code:
cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 cd 82 <c3> cc cc cc cc cc cc cc cc cc cc 

Call Trace:
  [<c01013a7>] hypercall_page+0x3a7  <--
  [<c0105f52>] xen_safe_halt+0x9f 
  [<c01028ab>] xen_idle+0x1b 
  [<c0102810>] cpu_idle+0xa8 
  [<c038f88b>] start_kernel+0x2f5 
root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 1
eip: c0170f39 file_kill+0x27 flags: 00001202 i nz
esp: e74cbf2c
eax: ec825e98	ebx: d7033a80	ecx: ee07207f	edx: e7672e80
esi: d7033a80	edi: ed3942c0	ebp: ed396be8
 cs: 00000061	 ds: 0000007b	 fs: 000000d8	 gs: 00000000

Stack:
 00000010 c01710ae ed420f00 e81b3a78 e82d5cfc e7218b00 e74ca000 c016080d
 c123f020 e81b3a78 c01608e7 000001e8 c123f020 e7218b00 e7218b4c 00000000
 c011c723 00000000 e59bba80 c01214aa 00000001 e7447a80 e7688580 00000000
 e76885c0 00000000 ed4ebd00 00000000 e74ca000 c0121998 00000000 00000000

Code:
13 8b 43 04 89 42 04 89 10 89 5b 04 89 1b f0 fe 05 40 42 38 c0 <0f> b7 15 40 42 38 c0 38 f2 0f 95 

Call Trace:
  [<c0170f39>] file_kill+0x27  <--
  [<c01710ae>] __fput+0xd2 
  [<c016080d>] remove_vma+0x25 
  [<c01608e7>] exit_mmap+0xbc 
  [<c011c723>] mmput+0x24 
  [<c01214aa>] do_exit+0x1ad 
  [<c0121998>] do_group_exit+0x94 
  [<c01219ce>] sys_exit_group+0xd 
  [<c0103f76>] syscall_call+0x7 
  [<c02c0000>] zone_wait_table_init+0x47 



[-- Attachment #1.2: Type: application/pgp-signature, Size: 835 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: domU crashing, trying to determine why

[Pasi Kärkkäinen]

On Sat, Jan 01, 2011 at 02:16:55PM -0500, Micah Anderson wrote:
> 
> I'm having a domU that is regularly crashing, without any obvious reason
> why. I set 'on_crash=preserve' to try and get a crash to try and get
> some clues, but what I got doesn't too obvious to me. I was hoping
> someone here would be able to give me some clues. 
> 
> Thanks for any clues anyone can give! Here is what I managed to eke out:
> 
> root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 0

So the domU has only 1 vcpu? 

-- Pasi

> eip: c01013a7 hypercall_page+0x3a7 flags: 00001246 i z p
> esp: c0389fbc
> eax: 00000000	ebx: 00000001	ecx: 00000000	edx: c0389fc8
> esi: 00000000	edi: 00000000	ebp: 00000000
>  cs: 00000061	 ds: 0000007b	 fs: 000000d8	 gs: 00000000
> 
> Stack:
>  c0105f52 ffffffff 00000000 c06ca9dd 0009cf8a 00000000 00000020 ffffffff
>  c01028ab c0102810 00000020 c1631000 00000000 c038f88b c03ac580 0702080b
>  00000000
> 
> Code:
> cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 cd 82 <c3> cc cc cc cc cc cc cc cc cc cc 
> 
> Call Trace:
>   [<c01013a7>] hypercall_page+0x3a7  <--
>   [<c0105f52>] xen_safe_halt+0x9f 
>   [<c01028ab>] xen_idle+0x1b 
>   [<c0102810>] cpu_idle+0xa8 
>   [<c038f88b>] start_kernel+0x2f5 
> root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 1
> eip: c0170f39 file_kill+0x27 flags: 00001202 i nz
> esp: e74cbf2c
> eax: ec825e98	ebx: d7033a80	ecx: ee07207f	edx: e7672e80
> esi: d7033a80	edi: ed3942c0	ebp: ed396be8
>  cs: 00000061	 ds: 0000007b	 fs: 000000d8	 gs: 00000000
> 
> Stack:
>  00000010 c01710ae ed420f00 e81b3a78 e82d5cfc e7218b00 e74ca000 c016080d
>  c123f020 e81b3a78 c01608e7 000001e8 c123f020 e7218b00 e7218b4c 00000000
>  c011c723 00000000 e59bba80 c01214aa 00000001 e7447a80 e7688580 00000000
>  e76885c0 00000000 ed4ebd00 00000000 e74ca000 c0121998 00000000 00000000
> 
> Code:
> 13 8b 43 04 89 42 04 89 10 89 5b 04 89 1b f0 fe 05 40 42 38 c0 <0f> b7 15 40 42 38 c0 38 f2 0f 95 
> 
> Call Trace:
>   [<c0170f39>] file_kill+0x27  <--
>   [<c01710ae>] __fput+0xd2 
>   [<c016080d>] remove_vma+0x25 
>   [<c01608e7>] exit_mmap+0xbc 
>   [<c011c723>] mmput+0x24 
>   [<c01214aa>] do_exit+0x1ad 
>   [<c0121998>] do_group_exit+0x94 
>   [<c01219ce>] sys_exit_group+0xd 
>   [<c0103f76>] syscall_call+0x7 
>   [<c02c0000>] zone_wait_table_init+0x47 
> 
> 



> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: domU crashing, trying to determine why

[micah anderson]

[-- Attachment #1.1: Type: text/plain, Size: 1918 bytes --]

On Sat, 1 Jan 2011 23:41:06 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> On Sat, Jan 01, 2011 at 02:16:55PM -0500, Micah Anderson wrote:
> > 
> > I'm having a domU that is regularly crashing, without any obvious reason
> > why. I set 'on_crash=preserve' to try and get a crash to try and get
> > some clues, but what I got doesn't too obvious to me. I was hoping
> > someone here would be able to give me some clues. 
> > 
> > Thanks for any clues anyone can give! Here is what I managed to eke out:
> > 
> > root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 0
> 
> So the domU has only 1 vcpu? 

No, I did this for both vcpus, the second one is below:

> > root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 1
> > eip: c0170f39 file_kill+0x27 flags: 00001202 i nz
> > esp: e74cbf2c
> > eax: ec825e98	ebx: d7033a80	ecx: ee07207f	edx: e7672e80
> > esi: d7033a80	edi: ed3942c0	ebp: ed396be8
> >  cs: 00000061	 ds: 0000007b	 fs: 000000d8	 gs: 00000000
> > 
> > Stack:
> >  00000010 c01710ae ed420f00 e81b3a78 e82d5cfc e7218b00 e74ca000 c016080d
> >  c123f020 e81b3a78 c01608e7 000001e8 c123f020 e7218b00 e7218b4c 00000000
> >  c011c723 00000000 e59bba80 c01214aa 00000001 e7447a80 e7688580 00000000
> >  e76885c0 00000000 ed4ebd00 00000000 e74ca000 c0121998 00000000 00000000
> > 
> > Code:
> > 13 8b 43 04 89 42 04 89 10 89 5b 04 89 1b f0 fe 05 40 42 38 c0 <0f> b7 15 40 42 38 c0 38 f2 0f 95 
> > 
> > Call Trace:
> >   [<c0170f39>] file_kill+0x27  <--
> >   [<c01710ae>] __fput+0xd2 
> >   [<c016080d>] remove_vma+0x25 
> >   [<c01608e7>] exit_mmap+0xbc 
> >   [<c011c723>] mmput+0x24 
> >   [<c01214aa>] do_exit+0x1ad 
> >   [<c0121998>] do_group_exit+0x94 
> >   [<c01219ce>] sys_exit_group+0xd 
> >   [<c0103f76>] syscall_call+0x7 
> >   [<c02c0000>] zone_wait_table_init+0x47 


[-- Attachment #1.2: Type: application/pgp-signature, Size: 835 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: domU crashing, trying to determine why

[Pasi Kärkkäinen]

On Sat, Jan 01, 2011 at 07:43:32PM -0500, micah anderson wrote:
> On Sat, 1 Jan 2011 23:41:06 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> > On Sat, Jan 01, 2011 at 02:16:55PM -0500, Micah Anderson wrote:
> > > 
> > > I'm having a domU that is regularly crashing, without any obvious reason
> > > why. I set 'on_crash=preserve' to try and get a crash to try and get
> > > some clues, but what I got doesn't too obvious to me. I was hoping
> > > someone here would be able to give me some clues. 
> > > 
> > > Thanks for any clues anyone can give! Here is what I managed to eke out:
> > > 
> > > root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 0
> > 
> > So the domU has only 1 vcpu? 
> 
> No, I did this for both vcpus, the second one is below:
> 

True, I somehow missed the other one.
Did you check debian bugreports if there are other reports like this? 

-- Pasi

> > > root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 1
> > > eip: c0170f39 file_kill+0x27 flags: 00001202 i nz
> > > esp: e74cbf2c
> > > eax: ec825e98	ebx: d7033a80	ecx: ee07207f	edx: e7672e80
> > > esi: d7033a80	edi: ed3942c0	ebp: ed396be8
> > >  cs: 00000061	 ds: 0000007b	 fs: 000000d8	 gs: 00000000
> > > 
> > > Stack:
> > >  00000010 c01710ae ed420f00 e81b3a78 e82d5cfc e7218b00 e74ca000 c016080d
> > >  c123f020 e81b3a78 c01608e7 000001e8 c123f020 e7218b00 e7218b4c 00000000
> > >  c011c723 00000000 e59bba80 c01214aa 00000001 e7447a80 e7688580 00000000
> > >  e76885c0 00000000 ed4ebd00 00000000 e74ca000 c0121998 00000000 00000000
> > > 
> > > Code:
> > > 13 8b 43 04 89 42 04 89 10 89 5b 04 89 1b f0 fe 05 40 42 38 c0 <0f> b7 15 40 42 38 c0 38 f2 0f 95 
> > > 
> > > Call Trace:
> > >   [<c0170f39>] file_kill+0x27  <--
> > >   [<c01710ae>] __fput+0xd2 
> > >   [<c016080d>] remove_vma+0x25 
> > >   [<c01608e7>] exit_mmap+0xbc 
> > >   [<c011c723>] mmput+0x24 
> > >   [<c01214aa>] do_exit+0x1ad 
> > >   [<c0121998>] do_group_exit+0x94 
> > >   [<c01219ce>] sys_exit_group+0xd 
> > >   [<c0103f76>] syscall_call+0x7 
> > >   [<c02c0000>] zone_wait_table_init+0x47 
> 

Re: domU crashing, trying to determine why

[micah anderson]

[-- Attachment #1.1: Type: text/plain, Size: 1099 bytes --]

On Sun, 2 Jan 2011 14:59:13 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> On Sat, Jan 01, 2011 at 07:43:32PM -0500, micah anderson wrote:
> > On Sat, 1 Jan 2011 23:41:06 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> > > On Sat, Jan 01, 2011 at 02:16:55PM -0500, Micah Anderson wrote:
> > > > 
> > > > I'm having a domU that is regularly crashing, without any obvious reason
> > > > why. I set 'on_crash=preserve' to try and get a crash to try and get
> > > > some clues, but what I got doesn't too obvious to me. I was hoping
> > > > someone here would be able to give me some clues. 
> > > > 
> > > > Thanks for any clues anyone can give! Here is what I managed to eke out:
> > > > 
> > > > root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 0
> > > 
> > > So the domU has only 1 vcpu? 
> > 
> > No, I did this for both vcpus, the second one is below:
> > 
> 
> True, I somehow missed the other one.
> Did you check debian bugreports if there are other reports like this? 

I did, but I didn't find anything relevant.

m


[-- Attachment #1.2: Type: application/pgp-signature, Size: 835 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: domU crashing, trying to determine why

[Pasi Kärkkäinen]

On Sun, Jan 02, 2011 at 01:15:22PM -0500, micah anderson wrote:
> On Sun, 2 Jan 2011 14:59:13 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> > On Sat, Jan 01, 2011 at 07:43:32PM -0500, micah anderson wrote:
> > > On Sat, 1 Jan 2011 23:41:06 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> > > > On Sat, Jan 01, 2011 at 02:16:55PM -0500, Micah Anderson wrote:
> > > > > 
> > > > > I'm having a domU that is regularly crashing, without any obvious reason
> > > > > why. I set 'on_crash=preserve' to try and get a crash to try and get
> > > > > some clues, but what I got doesn't too obvious to me. I was hoping
> > > > > someone here would be able to give me some clues. 
> > > > > 
> > > > > Thanks for any clues anyone can give! Here is what I managed to eke out:
> > > > > 
> > > > > root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 0
> > > > 
> > > > So the domU has only 1 vcpu? 
> > > 
> > > No, I did this for both vcpus, the second one is below:
> > > 
> > 
> > True, I somehow missed the other one.
> > Did you check debian bugreports if there are other reports like this? 
> 
> I did, but I didn't find anything relevant.
> 

Can you list the steps to reproduce the bug/crash? Does it happen every time?
Does it happen if you have only 1 vcpu for the domU?

-- Pasi

Re: domU crashing, trying to determine why

[micah anderson]

[-- Attachment #1.1: Type: text/plain, Size: 1875 bytes --]

On Sun, 2 Jan 2011 20:39:09 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> On Sun, Jan 02, 2011 at 01:15:22PM -0500, micah anderson wrote:
> > On Sun, 2 Jan 2011 14:59:13 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> > > On Sat, Jan 01, 2011 at 07:43:32PM -0500, micah anderson wrote:
> > > > On Sat, 1 Jan 2011 23:41:06 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> > > > > On Sat, Jan 01, 2011 at 02:16:55PM -0500, Micah Anderson wrote:
> > > > > > 
> > > > > > I'm having a domU that is regularly crashing, without any obvious reason
> > > > > > why. I set 'on_crash=preserve' to try and get a crash to try and get
> > > > > > some clues, but what I got doesn't too obvious to me. I was hoping
> > > > > > someone here would be able to give me some clues. 
> > > > > > 
> > > > > > Thanks for any clues anyone can give! Here is what I managed to eke out:
> > > > > > 
> > > > > > root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 0
> > > > > 
> > > > > So the domU has only 1 vcpu? 
> > > > 
> > > > No, I did this for both vcpus, the second one is below:
> > > > 
> > > 
> > > True, I somehow missed the other one.
> > > Did you check debian bugreports if there are other reports like this? 
> > 
> > I did, but I didn't find anything relevant.
> > 
> 
> Can you list the steps to reproduce the bug/crash? Does it happen every time?
> Does it happen if you have only 1 vcpu for the domU?

There really aren't any steps. I 'xm create' the domain, and then what
seems like a random amount of time later, it crashes. Usually that
random amount of time is somewhere between one week and four weeks, but
I've had it happen twice in a weekend. 

I've got this setup:

vcpus=2
cpus=[1,3]

I really need available CPU power of 2 CPUs, or the machine will be too
underpowered... 

m

[-- Attachment #1.2: Type: application/pgp-signature, Size: 835 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: domU crashing, trying to determine why

[Pasi Kärkkäinen]

On Sun, Jan 02, 2011 at 10:41:24PM -0500, micah anderson wrote:
> On Sun, 2 Jan 2011 20:39:09 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> > On Sun, Jan 02, 2011 at 01:15:22PM -0500, micah anderson wrote:
> > > On Sun, 2 Jan 2011 14:59:13 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> > > > On Sat, Jan 01, 2011 at 07:43:32PM -0500, micah anderson wrote:
> > > > > On Sat, 1 Jan 2011 23:41:06 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> > > > > > On Sat, Jan 01, 2011 at 02:16:55PM -0500, Micah Anderson wrote:
> > > > > > > 
> > > > > > > I'm having a domU that is regularly crashing, without any obvious reason
> > > > > > > why. I set 'on_crash=preserve' to try and get a crash to try and get
> > > > > > > some clues, but what I got doesn't too obvious to me. I was hoping
> > > > > > > someone here would be able to give me some clues. 
> > > > > > > 
> > > > > > > Thanks for any clues anyone can give! Here is what I managed to eke out:
> > > > > > > 
> > > > > > > root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 0
> > > > > > 
> > > > > > So the domU has only 1 vcpu? 
> > > > > 
> > > > > No, I did this for both vcpus, the second one is below:
> > > > > 
> > > > 
> > > > True, I somehow missed the other one.
> > > > Did you check debian bugreports if there are other reports like this? 
> > > 
> > > I did, but I didn't find anything relevant.
> > > 
> > 
> > Can you list the steps to reproduce the bug/crash? Does it happen every time?
> > Does it happen if you have only 1 vcpu for the domU?
> 
> There really aren't any steps. I 'xm create' the domain, and then what
> seems like a random amount of time later, it crashes. Usually that
> random amount of time is somewhere between one week and four weeks, but
> I've had it happen twice in a weekend. 
> 

What kind of workload is running in the domU ?

> I've got this setup:
> 
> vcpus=2
> cpus=[1,3]
> 
> I really need available CPU power of 2 CPUs, or the machine will be too
> underpowered... 
> 

Ok.

-- Pasi

Re: domU crashing, trying to determine why

[micah anderson]

[-- Attachment #1.1: Type: text/plain, Size: 2199 bytes --]

On Mon, 3 Jan 2011 07:34:35 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> On Sun, Jan 02, 2011 at 10:41:24PM -0500, micah anderson wrote:
> > On Sun, 2 Jan 2011 20:39:09 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> > > On Sun, Jan 02, 2011 at 01:15:22PM -0500, micah anderson wrote:
> > > > On Sun, 2 Jan 2011 14:59:13 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> > > > > On Sat, Jan 01, 2011 at 07:43:32PM -0500, micah anderson wrote:
> > > > > > On Sat, 1 Jan 2011 23:41:06 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> > > > > > > On Sat, Jan 01, 2011 at 02:16:55PM -0500, Micah Anderson wrote:
> > > > > > > > 
> > > > > > > > I'm having a domU that is regularly crashing, without any obvious reason
> > > > > > > > why. I set 'on_crash=preserve' to try and get a crash to try and get
> > > > > > > > some clues, but what I got doesn't too obvious to me. I was hoping
> > > > > > > > someone here would be able to give me some clues. 
> > > > > > > > 
> > > > > > > > Thanks for any clues anyone can give! Here is what I managed to eke out:
> > > > > > > > 
> > > > > > > > root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 0
> > > > > > > 
> > > > > > > So the domU has only 1 vcpu? 
> > > > > > 
> > > > > > No, I did this for both vcpus, the second one is below:
> > > > > > 
> > > > > 
> > > > > True, I somehow missed the other one.
> > > > > Did you check debian bugreports if there are other reports like this? 
> > > > 
> > > > I did, but I didn't find anything relevant.
> > > > 
> > > 
> > > Can you list the steps to reproduce the bug/crash? Does it happen every time?
> > > Does it happen if you have only 1 vcpu for the domU?
> > 
> > There really aren't any steps. I 'xm create' the domain, and then what
> > seems like a random amount of time later, it crashes. Usually that
> > random amount of time is somewhere between one week and four weeks, but
> > I've had it happen twice in a weekend. 
> > 
> 
> What kind of workload is running in the domU ?

Its a database server, only running mysql. Its used fairly heavily for
user authentication, and email account verification.


[-- Attachment #1.2: Type: application/pgp-signature, Size: 835 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: domU crashing, trying to determine why

[Konrad Rzeszutek Wilk]

> > I've got this setup:
> > 
> > vcpus=2
> > cpus=[1,3]

Is there anything in the Xen hypervisor ring buffer? You might need to enable the 
full logging of the guest and dom0 to get anything there.

Re: domU crashing, trying to determine why

[micah anderson]

[-- Attachment #1.1: Type: text/plain, Size: 432 bytes --]

On Mon, 3 Jan 2011 11:40:31 -0500, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> > > I've got this setup:
> > > 
> > > vcpus=2
> > > cpus=[1,3]
> 
> Is there anything in the Xen hypervisor ring buffer? You might need to enable the 
> full logging of the guest and dom0 to get anything there.

How do I get at the ring buffer, and what specifically should I enable
to get all of that?

thanks!
micah

-- 


[-- Attachment #1.2: Type: application/pgp-signature, Size: 835 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: domU crashing, trying to determine why

[Konrad Rzeszutek Wilk]

On Mon, Jan 03, 2011 at 12:03:34PM -0500, micah anderson wrote:
> On Mon, 3 Jan 2011 11:40:31 -0500, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> > > > I've got this setup:
> > > > 
> > > > vcpus=2
> > > > cpus=[1,3]
> > 
> > Is there anything in the Xen hypervisor ring buffer? You might need to enable the 
> > full logging of the guest and dom0 to get anything there.
> 
> How do I get at the ring buffer, and what specifically should I enable

'xm dmesg'

> to get all of that?

In http://wiki.xensource.com/xenwiki/XenParavirtOps
"Are there more debugging options I could enable to troubleshoot booting problems?"

> 
> thanks!
> micah
> 
> -- 
> 

Re: domU crashing, trying to determine why

[Ian Campbell]

On Mon, 2011-01-03 at 16:40 +0000, Konrad Rzeszutek Wilk wrote:
> > > I've got this setup:
> > > 
> > > vcpus=2
> > > cpus=[1,3]
> 
> Is there anything in the Xen hypervisor ring buffer?

Equally was there anything on the guest console? If it is dying before
you can look then you can enable logging via XENCONSOLED_TRACE=guest
in /etc/sysconfig/xencommons or /etc/default/xencommons (depending on
distro). Logs show up in /var/log/xen/guest/

If the guest is still running but deadlocked or similar then it might be
worth sending a few SysRQs to it. To do this press Ctrl-O on the guest
console and press one of the magic letters, 'h' will give help.
't' (tasks) would be a useful start as would 'p' (processor state).

It might be worth compiling your guest kernel with lockdep support and
seeing if it complains about anything.

Also, I notice you seem to be running the Debian Lenny classic-Xen
forward port. Are you running the most recent version of that kernel?
Unfortunately there is not much developer interest in these classic-Xen
forwards ports these days.

You could also try the -686-bigmem kernel flavour from Lenny which is a
PVops enabled kernel, or perhaps a Squeeze kernel from backports.org
(either -686-bigmem or -xen-686, both are pvops kernels). If your
problem persists with these then there is more chance of it getting
fixed than with the Lenny -xen-686 flavour.

Ian.

>  You might need to enable the 
> full logging of the guest and dom0 to get anything there.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: domU crashing, trying to determine why

[micah anderson]

[-- Attachment #1.1: Type: text/plain, Size: 2378 bytes --]

On Tue, 4 Jan 2011 10:54:44 +0000, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Mon, 2011-01-03 at 16:40 +0000, Konrad Rzeszutek Wilk wrote:
> > > > I've got this setup:
> > > > 
> > > > vcpus=2
> > > > cpus=[1,3]
> > 
> > Is there anything in the Xen hypervisor ring buffer?
> 
> Equally was there anything on the guest console? If it is dying before
> you can look then you can enable logging via XENCONSOLED_TRACE=guest
> in /etc/sysconfig/xencommons or /etc/default/xencommons (depending on
> distro). Logs show up in /var/log/xen/guest/

Because I configured 'on_crash = preserve' I can see the console after
it crashes, but no, there is nothing on the guest console, except for
the login prompt (which of course no longer works).

> If the guest is still running but deadlocked or similar then it might be
> worth sending a few SysRQs to it. To do this press Ctrl-O on the guest
> console and press one of the magic letters, 'h' will give help.
> 't' (tasks) would be a useful start as would 'p' (processor state).

I'll try this on the next crash. It crashed this morning, so I took the
liberty of adding the kernel log parameters so I can get xen hypervisor
ring buffer debugging information for the next crash.

> It might be worth compiling your guest kernel with lockdep support and
> seeing if it complains about anything.

Is this a .config option? If so, what is it?

> Also, I notice you seem to be running the Debian Lenny classic-Xen
> forward port. Are you running the most recent version of that kernel?
> Unfortunately there is not much developer interest in these classic-Xen
> forwards ports these days.

Yes, I am running the up-to-date Debian Lenny kernel. One solution of
course is to upgrade the system to Squeeze to get an overhaul of the
whole subsystem, getting a newer kernel/xen completely... but of course
that is a bigger change, and would not give us any clue as to what is
going on.

> You could also try the -686-bigmem kernel flavour from Lenny which is a
> PVops enabled kernel, or perhaps a Squeeze kernel from backports.org
> (either -686-bigmem or -xen-686, both are pvops kernels). If your
> problem persists with these then there is more chance of it getting
> fixed than with the Lenny -xen-686 flavour.

Do you mean run that kernel in the Dom0, or the DomU? 

thanks,
micah

[-- Attachment #1.2: Type: application/pgp-signature, Size: 835 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: domU crashing, trying to determine why

[Ian Campbell]

On Tue, 2011-01-04 at 19:49 +0000, micah anderson wrote:
> On Tue, 4 Jan 2011 10:54:44 +0000, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Mon, 2011-01-03 at 16:40 +0000, Konrad Rzeszutek Wilk wrote:

> > It might be worth compiling your guest kernel with lockdep support and
> > seeing if it complains about anything.
> 
> Is this a .config option? If so, what is it?

It's CONFIG_LOCKDEP. It always takes me several minutes of trial and
error to figure out what dependencies I need to turn it on...

> > You could also try the -686-bigmem kernel flavour from Lenny which is a
> > PVops enabled kernel, or perhaps a Squeeze kernel from backports.org
> > (either -686-bigmem or -xen-686, both are pvops kernels). If your
> > problem persists with these then there is more chance of it getting
> > fixed than with the Lenny -xen-686 flavour.
> 
> Do you mean run that kernel in the Dom0, or the DomU? 

In the domU.

Currently only the -xen-686 flavours work for dom0.

Ian.

Re: domU crashing, trying to determine why

[micah anderson]

[-- Attachment #1.1: Type: text/plain, Size: 1622 bytes --]

On Mon, 3 Jan 2011 12:14:25 -0500, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> On Mon, Jan 03, 2011 at 12:03:34PM -0500, micah anderson wrote:
> > On Mon, 3 Jan 2011 11:40:31 -0500, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> > > > > I've got this setup:
> > > > > 
> > > > > vcpus=2
> > > > > cpus=[1,3]
> > > 
> > > Is there anything in the Xen hypervisor ring buffer? You might need to enable the 
> > > full logging of the guest and dom0 to get anything there.
> > 
> > How do I get at the ring buffer, and what specifically should I enable
> 
> 'xm dmesg'

(XEN) mm.c:645:d2 Non-privileged (2) attempt to map I/O space 000000f0
(XEN) traps.c:2544: GPF (0068): ff1886c2 -> ff1887c9
(XEN) domain_crash_sync called from entry.S (ff188600)
(XEN) Domain 2 (vcpu#1) crashed on cpu#3:
(XEN) ----[ Xen-3.2-1  x86_32p  debug=n  Tainted:    C ]----
(XEN) CPU:    3
(XEN) EIP:    0061:[<c01013a7>]
(XEN) EFLAGS: 00000246   CONTEXT: guest
(XEN) eax: 00000000   ebx: 00000001   ecx: 00000000   edx: ed447f90
(XEN) esi: 00000001   edi: 00000001   ebp: 00000000   esp: ed447f84
(XEN) cr0: 8005003b   cr4: 000006f0   cr3: 00bdac80   cr2: 095346dc
(XEN) ds: 007b   es: 007b   fs: 00d8   gs: 0000   ss: 0069   cs: 0061
(XEN) Guest stack trace from esp=ed447f84:
(XEN)    c0105f52 ffffffff 00000001 673bfa5d 0002dba1 00000000 00000000 ffffffff
(XEN)    c01028ab c0102810 00000000 00000000 00000000 00000000 00000000 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 000000d8 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000


[-- Attachment #1.2: Type: application/pgp-signature, Size: 835 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: domU crashing, trying to determine why

[Konrad Rzeszutek Wilk]

On Fri, Jan 14, 2011 at 12:31:22PM -0500, micah anderson wrote:
> On Mon, 3 Jan 2011 12:14:25 -0500, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> > On Mon, Jan 03, 2011 at 12:03:34PM -0500, micah anderson wrote:
> > > On Mon, 3 Jan 2011 11:40:31 -0500, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> > > > > > I've got this setup:
> > > > > > 
> > > > > > vcpus=2
> > > > > > cpus=[1,3]
> > > > 
> > > > Is there anything in the Xen hypervisor ring buffer? You might need to enable the 
> > > > full logging of the guest and dom0 to get anything there.
> > > 
> > > How do I get at the ring buffer, and what specifically should I enable
> > 
> > 'xm dmesg'
> 
> (XEN) mm.c:645:d2 Non-privileged (2) attempt to map I/O space 000000f0

Hmm, that should not happen anymore, as there are is a bunch of
code guarding domains from trying to touch 0->0x100. Unless your guest
has the old stuff.

> (XEN) traps.c:2544: GPF (0068): ff1886c2 -> ff1887c9
> (XEN) domain_crash_sync called from entry.S (ff188600)
> (XEN) Domain 2 (vcpu#1) crashed on cpu#3:
> (XEN) ----[ Xen-3.2-1  x86_32p  debug=n  Tainted:    C ]----

Ugh, ancient hypervisor.

> (XEN) CPU:    3
> (XEN) EIP:    0061:[<c01013a7>]

Now use gdb to lookup that EIP value. Or use your guest System.map
to see what function this could map too.

> (XEN) EFLAGS: 00000246   CONTEXT: guest
> (XEN) eax: 00000000   ebx: 00000001   ecx: 00000000   edx: ed447f90
> (XEN) esi: 00000001   edi: 00000001   ebp: 00000000   esp: ed447f84
> (XEN) cr0: 8005003b   cr4: 000006f0   cr3: 00bdac80   cr2: 095346dc
> (XEN) ds: 007b   es: 007b   fs: 00d8   gs: 0000   ss: 0069   cs: 0061
> (XEN) Guest stack trace from esp=ed447f84:
> (XEN)    c0105f52 ffffffff 00000001 673bfa5d 0002dba1 00000000 00000000 ffffffff
> (XEN)    c01028ab c0102810 00000000 00000000 00000000 00000000 00000000 00000000

And potentially c01028ab  c0105f52 .. 
> (XEN)    00000000 00000000 00000000 00000000 00000000 00000000 000000d8 00000000
> (XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000
>