* domU crashing, trying to determine why @ 2011-01-01 19:16 Micah Anderson 2011-01-01 21:41 ` Pasi Kärkkäinen 0 siblings, 1 reply; 17+ messages in thread From: Micah Anderson @ 2011-01-01 19:16 UTC (permalink / raw) To: xen-devel [-- Attachment #1.1: Type: text/plain, Size: 2234 bytes --] I'm having a domU that is regularly crashing, without any obvious reason why. I set 'on_crash=preserve' to try and get a crash to try and get some clues, but what I got doesn't too obvious to me. I was hoping someone here would be able to give me some clues. Thanks for any clues anyone can give! Here is what I managed to eke out: root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 0 eip: c01013a7 hypercall_page+0x3a7 flags: 00001246 i z p esp: c0389fbc eax: 00000000 ebx: 00000001 ecx: 00000000 edx: c0389fc8 esi: 00000000 edi: 00000000 ebp: 00000000 cs: 00000061 ds: 0000007b fs: 000000d8 gs: 00000000 Stack: c0105f52 ffffffff 00000000 c06ca9dd 0009cf8a 00000000 00000020 ffffffff c01028ab c0102810 00000020 c1631000 00000000 c038f88b c03ac580 0702080b 00000000 Code: cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 cd 82 <c3> cc cc cc cc cc cc cc cc cc cc Call Trace: [<c01013a7>] hypercall_page+0x3a7 <-- [<c0105f52>] xen_safe_halt+0x9f [<c01028ab>] xen_idle+0x1b [<c0102810>] cpu_idle+0xa8 [<c038f88b>] start_kernel+0x2f5 root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 1 eip: c0170f39 file_kill+0x27 flags: 00001202 i nz esp: e74cbf2c eax: ec825e98 ebx: d7033a80 ecx: ee07207f edx: e7672e80 esi: d7033a80 edi: ed3942c0 ebp: ed396be8 cs: 00000061 ds: 0000007b fs: 000000d8 gs: 00000000 Stack: 00000010 c01710ae ed420f00 e81b3a78 e82d5cfc e7218b00 e74ca000 c016080d c123f020 e81b3a78 c01608e7 000001e8 c123f020 e7218b00 e7218b4c 00000000 c011c723 00000000 e59bba80 c01214aa 00000001 e7447a80 e7688580 00000000 e76885c0 00000000 ed4ebd00 00000000 e74ca000 c0121998 00000000 00000000 Code: 13 8b 43 04 89 42 04 89 10 89 5b 04 89 1b f0 fe 05 40 42 38 c0 <0f> b7 15 40 42 38 c0 38 f2 0f 95 Call Trace: [<c0170f39>] file_kill+0x27 <-- [<c01710ae>] __fput+0xd2 [<c016080d>] remove_vma+0x25 [<c01608e7>] exit_mmap+0xbc [<c011c723>] mmput+0x24 [<c01214aa>] do_exit+0x1ad [<c0121998>] do_group_exit+0x94 [<c01219ce>] sys_exit_group+0xd [<c0103f76>] syscall_call+0x7 [<c02c0000>] zone_wait_table_init+0x47 [-- Attachment #1.2: Type: application/pgp-signature, Size: 835 bytes --] [-- Attachment #2: Type: text/plain, Size: 138 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: domU crashing, trying to determine why 2011-01-01 19:16 domU crashing, trying to determine why Micah Anderson @ 2011-01-01 21:41 ` Pasi Kärkkäinen 2011-01-02 0:43 ` micah anderson 0 siblings, 1 reply; 17+ messages in thread From: Pasi Kärkkäinen @ 2011-01-01 21:41 UTC (permalink / raw) To: Micah Anderson; +Cc: xen-devel On Sat, Jan 01, 2011 at 02:16:55PM -0500, Micah Anderson wrote: > > I'm having a domU that is regularly crashing, without any obvious reason > why. I set 'on_crash=preserve' to try and get a crash to try and get > some clues, but what I got doesn't too obvious to me. I was hoping > someone here would be able to give me some clues. > > Thanks for any clues anyone can give! Here is what I managed to eke out: > > root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 0 So the domU has only 1 vcpu? -- Pasi > eip: c01013a7 hypercall_page+0x3a7 flags: 00001246 i z p > esp: c0389fbc > eax: 00000000 ebx: 00000001 ecx: 00000000 edx: c0389fc8 > esi: 00000000 edi: 00000000 ebp: 00000000 > cs: 00000061 ds: 0000007b fs: 000000d8 gs: 00000000 > > Stack: > c0105f52 ffffffff 00000000 c06ca9dd 0009cf8a 00000000 00000020 ffffffff > c01028ab c0102810 00000020 c1631000 00000000 c038f88b c03ac580 0702080b > 00000000 > > Code: > cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 cd 82 <c3> cc cc cc cc cc cc cc cc cc cc > > Call Trace: > [<c01013a7>] hypercall_page+0x3a7 <-- > [<c0105f52>] xen_safe_halt+0x9f > [<c01028ab>] xen_idle+0x1b > [<c0102810>] cpu_idle+0xa8 > [<c038f88b>] start_kernel+0x2f5 > root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 1 > eip: c0170f39 file_kill+0x27 flags: 00001202 i nz > esp: e74cbf2c > eax: ec825e98 ebx: d7033a80 ecx: ee07207f edx: e7672e80 > esi: d7033a80 edi: ed3942c0 ebp: ed396be8 > cs: 00000061 ds: 0000007b fs: 000000d8 gs: 00000000 > > Stack: > 00000010 c01710ae ed420f00 e81b3a78 e82d5cfc e7218b00 e74ca000 c016080d > c123f020 e81b3a78 c01608e7 000001e8 c123f020 e7218b00 e7218b4c 00000000 > c011c723 00000000 e59bba80 c01214aa 00000001 e7447a80 e7688580 00000000 > e76885c0 00000000 ed4ebd00 00000000 e74ca000 c0121998 00000000 00000000 > > Code: > 13 8b 43 04 89 42 04 89 10 89 5b 04 89 1b f0 fe 05 40 42 38 c0 <0f> b7 15 40 42 38 c0 38 f2 0f 95 > > Call Trace: > [<c0170f39>] file_kill+0x27 <-- > [<c01710ae>] __fput+0xd2 > [<c016080d>] remove_vma+0x25 > [<c01608e7>] exit_mmap+0xbc > [<c011c723>] mmput+0x24 > [<c01214aa>] do_exit+0x1ad > [<c0121998>] do_group_exit+0x94 > [<c01219ce>] sys_exit_group+0xd > [<c0103f76>] syscall_call+0x7 > [<c02c0000>] zone_wait_table_init+0x47 > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: domU crashing, trying to determine why 2011-01-01 21:41 ` Pasi Kärkkäinen @ 2011-01-02 0:43 ` micah anderson 2011-01-02 12:59 ` Pasi Kärkkäinen 0 siblings, 1 reply; 17+ messages in thread From: micah anderson @ 2011-01-02 0:43 UTC (permalink / raw) To: Pasi Kärkkäinen; +Cc: xen-devel [-- Attachment #1.1: Type: text/plain, Size: 1918 bytes --] On Sat, 1 Jan 2011 23:41:06 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote: > On Sat, Jan 01, 2011 at 02:16:55PM -0500, Micah Anderson wrote: > > > > I'm having a domU that is regularly crashing, without any obvious reason > > why. I set 'on_crash=preserve' to try and get a crash to try and get > > some clues, but what I got doesn't too obvious to me. I was hoping > > someone here would be able to give me some clues. > > > > Thanks for any clues anyone can give! Here is what I managed to eke out: > > > > root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 0 > > So the domU has only 1 vcpu? No, I did this for both vcpus, the second one is below: > > root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 1 > > eip: c0170f39 file_kill+0x27 flags: 00001202 i nz > > esp: e74cbf2c > > eax: ec825e98 ebx: d7033a80 ecx: ee07207f edx: e7672e80 > > esi: d7033a80 edi: ed3942c0 ebp: ed396be8 > > cs: 00000061 ds: 0000007b fs: 000000d8 gs: 00000000 > > > > Stack: > > 00000010 c01710ae ed420f00 e81b3a78 e82d5cfc e7218b00 e74ca000 c016080d > > c123f020 e81b3a78 c01608e7 000001e8 c123f020 e7218b00 e7218b4c 00000000 > > c011c723 00000000 e59bba80 c01214aa 00000001 e7447a80 e7688580 00000000 > > e76885c0 00000000 ed4ebd00 00000000 e74ca000 c0121998 00000000 00000000 > > > > Code: > > 13 8b 43 04 89 42 04 89 10 89 5b 04 89 1b f0 fe 05 40 42 38 c0 <0f> b7 15 40 42 38 c0 38 f2 0f 95 > > > > Call Trace: > > [<c0170f39>] file_kill+0x27 <-- > > [<c01710ae>] __fput+0xd2 > > [<c016080d>] remove_vma+0x25 > > [<c01608e7>] exit_mmap+0xbc > > [<c011c723>] mmput+0x24 > > [<c01214aa>] do_exit+0x1ad > > [<c0121998>] do_group_exit+0x94 > > [<c01219ce>] sys_exit_group+0xd > > [<c0103f76>] syscall_call+0x7 > > [<c02c0000>] zone_wait_table_init+0x47 [-- Attachment #1.2: Type: application/pgp-signature, Size: 835 bytes --] [-- Attachment #2: Type: text/plain, Size: 138 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: domU crashing, trying to determine why 2011-01-02 0:43 ` micah anderson @ 2011-01-02 12:59 ` Pasi Kärkkäinen 2011-01-02 18:15 ` micah anderson 0 siblings, 1 reply; 17+ messages in thread From: Pasi Kärkkäinen @ 2011-01-02 12:59 UTC (permalink / raw) To: micah anderson; +Cc: xen-devel On Sat, Jan 01, 2011 at 07:43:32PM -0500, micah anderson wrote: > On Sat, 1 Jan 2011 23:41:06 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote: > > On Sat, Jan 01, 2011 at 02:16:55PM -0500, Micah Anderson wrote: > > > > > > I'm having a domU that is regularly crashing, without any obvious reason > > > why. I set 'on_crash=preserve' to try and get a crash to try and get > > > some clues, but what I got doesn't too obvious to me. I was hoping > > > someone here would be able to give me some clues. > > > > > > Thanks for any clues anyone can give! Here is what I managed to eke out: > > > > > > root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 0 > > > > So the domU has only 1 vcpu? > > No, I did this for both vcpus, the second one is below: > True, I somehow missed the other one. Did you check debian bugreports if there are other reports like this? -- Pasi > > > root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 1 > > > eip: c0170f39 file_kill+0x27 flags: 00001202 i nz > > > esp: e74cbf2c > > > eax: ec825e98 ebx: d7033a80 ecx: ee07207f edx: e7672e80 > > > esi: d7033a80 edi: ed3942c0 ebp: ed396be8 > > > cs: 00000061 ds: 0000007b fs: 000000d8 gs: 00000000 > > > > > > Stack: > > > 00000010 c01710ae ed420f00 e81b3a78 e82d5cfc e7218b00 e74ca000 c016080d > > > c123f020 e81b3a78 c01608e7 000001e8 c123f020 e7218b00 e7218b4c 00000000 > > > c011c723 00000000 e59bba80 c01214aa 00000001 e7447a80 e7688580 00000000 > > > e76885c0 00000000 ed4ebd00 00000000 e74ca000 c0121998 00000000 00000000 > > > > > > Code: > > > 13 8b 43 04 89 42 04 89 10 89 5b 04 89 1b f0 fe 05 40 42 38 c0 <0f> b7 15 40 42 38 c0 38 f2 0f 95 > > > > > > Call Trace: > > > [<c0170f39>] file_kill+0x27 <-- > > > [<c01710ae>] __fput+0xd2 > > > [<c016080d>] remove_vma+0x25 > > > [<c01608e7>] exit_mmap+0xbc > > > [<c011c723>] mmput+0x24 > > > [<c01214aa>] do_exit+0x1ad > > > [<c0121998>] do_group_exit+0x94 > > > [<c01219ce>] sys_exit_group+0xd > > > [<c0103f76>] syscall_call+0x7 > > > [<c02c0000>] zone_wait_table_init+0x47 > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: domU crashing, trying to determine why 2011-01-02 12:59 ` Pasi Kärkkäinen @ 2011-01-02 18:15 ` micah anderson 2011-01-02 18:39 ` Pasi Kärkkäinen 0 siblings, 1 reply; 17+ messages in thread From: micah anderson @ 2011-01-02 18:15 UTC (permalink / raw) To: Pasi Kärkkäinen; +Cc: xen-devel [-- Attachment #1.1: Type: text/plain, Size: 1099 bytes --] On Sun, 2 Jan 2011 14:59:13 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote: > On Sat, Jan 01, 2011 at 07:43:32PM -0500, micah anderson wrote: > > On Sat, 1 Jan 2011 23:41:06 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote: > > > On Sat, Jan 01, 2011 at 02:16:55PM -0500, Micah Anderson wrote: > > > > > > > > I'm having a domU that is regularly crashing, without any obvious reason > > > > why. I set 'on_crash=preserve' to try and get a crash to try and get > > > > some clues, but what I got doesn't too obvious to me. I was hoping > > > > someone here would be able to give me some clues. > > > > > > > > Thanks for any clues anyone can give! Here is what I managed to eke out: > > > > > > > > root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 0 > > > > > > So the domU has only 1 vcpu? > > > > No, I did this for both vcpus, the second one is below: > > > > True, I somehow missed the other one. > Did you check debian bugreports if there are other reports like this? I did, but I didn't find anything relevant. m [-- Attachment #1.2: Type: application/pgp-signature, Size: 835 bytes --] [-- Attachment #2: Type: text/plain, Size: 138 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: domU crashing, trying to determine why 2011-01-02 18:15 ` micah anderson @ 2011-01-02 18:39 ` Pasi Kärkkäinen 2011-01-03 3:41 ` micah anderson 0 siblings, 1 reply; 17+ messages in thread From: Pasi Kärkkäinen @ 2011-01-02 18:39 UTC (permalink / raw) To: micah anderson; +Cc: xen-devel On Sun, Jan 02, 2011 at 01:15:22PM -0500, micah anderson wrote: > On Sun, 2 Jan 2011 14:59:13 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote: > > On Sat, Jan 01, 2011 at 07:43:32PM -0500, micah anderson wrote: > > > On Sat, 1 Jan 2011 23:41:06 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote: > > > > On Sat, Jan 01, 2011 at 02:16:55PM -0500, Micah Anderson wrote: > > > > > > > > > > I'm having a domU that is regularly crashing, without any obvious reason > > > > > why. I set 'on_crash=preserve' to try and get a crash to try and get > > > > > some clues, but what I got doesn't too obvious to me. I was hoping > > > > > someone here would be able to give me some clues. > > > > > > > > > > Thanks for any clues anyone can give! Here is what I managed to eke out: > > > > > > > > > > root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 0 > > > > > > > > So the domU has only 1 vcpu? > > > > > > No, I did this for both vcpus, the second one is below: > > > > > > > True, I somehow missed the other one. > > Did you check debian bugreports if there are other reports like this? > > I did, but I didn't find anything relevant. > Can you list the steps to reproduce the bug/crash? Does it happen every time? Does it happen if you have only 1 vcpu for the domU? -- Pasi ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: domU crashing, trying to determine why 2011-01-02 18:39 ` Pasi Kärkkäinen @ 2011-01-03 3:41 ` micah anderson 2011-01-03 5:34 ` Pasi Kärkkäinen 0 siblings, 1 reply; 17+ messages in thread From: micah anderson @ 2011-01-03 3:41 UTC (permalink / raw) To: Pasi Kärkkäinen; +Cc: xen-devel [-- Attachment #1.1: Type: text/plain, Size: 1875 bytes --] On Sun, 2 Jan 2011 20:39:09 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote: > On Sun, Jan 02, 2011 at 01:15:22PM -0500, micah anderson wrote: > > On Sun, 2 Jan 2011 14:59:13 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote: > > > On Sat, Jan 01, 2011 at 07:43:32PM -0500, micah anderson wrote: > > > > On Sat, 1 Jan 2011 23:41:06 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote: > > > > > On Sat, Jan 01, 2011 at 02:16:55PM -0500, Micah Anderson wrote: > > > > > > > > > > > > I'm having a domU that is regularly crashing, without any obvious reason > > > > > > why. I set 'on_crash=preserve' to try and get a crash to try and get > > > > > > some clues, but what I got doesn't too obvious to me. I was hoping > > > > > > someone here would be able to give me some clues. > > > > > > > > > > > > Thanks for any clues anyone can give! Here is what I managed to eke out: > > > > > > > > > > > > root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 0 > > > > > > > > > > So the domU has only 1 vcpu? > > > > > > > > No, I did this for both vcpus, the second one is below: > > > > > > > > > > True, I somehow missed the other one. > > > Did you check debian bugreports if there are other reports like this? > > > > I did, but I didn't find anything relevant. > > > > Can you list the steps to reproduce the bug/crash? Does it happen every time? > Does it happen if you have only 1 vcpu for the domU? There really aren't any steps. I 'xm create' the domain, and then what seems like a random amount of time later, it crashes. Usually that random amount of time is somewhere between one week and four weeks, but I've had it happen twice in a weekend. I've got this setup: vcpus=2 cpus=[1,3] I really need available CPU power of 2 CPUs, or the machine will be too underpowered... m [-- Attachment #1.2: Type: application/pgp-signature, Size: 835 bytes --] [-- Attachment #2: Type: text/plain, Size: 138 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: domU crashing, trying to determine why 2011-01-03 3:41 ` micah anderson @ 2011-01-03 5:34 ` Pasi Kärkkäinen 2011-01-03 6:32 ` micah anderson 2011-01-03 16:40 ` Konrad Rzeszutek Wilk 0 siblings, 2 replies; 17+ messages in thread From: Pasi Kärkkäinen @ 2011-01-03 5:34 UTC (permalink / raw) To: micah anderson; +Cc: xen-devel On Sun, Jan 02, 2011 at 10:41:24PM -0500, micah anderson wrote: > On Sun, 2 Jan 2011 20:39:09 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote: > > On Sun, Jan 02, 2011 at 01:15:22PM -0500, micah anderson wrote: > > > On Sun, 2 Jan 2011 14:59:13 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote: > > > > On Sat, Jan 01, 2011 at 07:43:32PM -0500, micah anderson wrote: > > > > > On Sat, 1 Jan 2011 23:41:06 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote: > > > > > > On Sat, Jan 01, 2011 at 02:16:55PM -0500, Micah Anderson wrote: > > > > > > > > > > > > > > I'm having a domU that is regularly crashing, without any obvious reason > > > > > > > why. I set 'on_crash=preserve' to try and get a crash to try and get > > > > > > > some clues, but what I got doesn't too obvious to me. I was hoping > > > > > > > someone here would be able to give me some clues. > > > > > > > > > > > > > > Thanks for any clues anyone can give! Here is what I managed to eke out: > > > > > > > > > > > > > > root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 0 > > > > > > > > > > > > So the domU has only 1 vcpu? > > > > > > > > > > No, I did this for both vcpus, the second one is below: > > > > > > > > > > > > > True, I somehow missed the other one. > > > > Did you check debian bugreports if there are other reports like this? > > > > > > I did, but I didn't find anything relevant. > > > > > > > Can you list the steps to reproduce the bug/crash? Does it happen every time? > > Does it happen if you have only 1 vcpu for the domU? > > There really aren't any steps. I 'xm create' the domain, and then what > seems like a random amount of time later, it crashes. Usually that > random amount of time is somewhere between one week and four weeks, but > I've had it happen twice in a weekend. > What kind of workload is running in the domU ? > I've got this setup: > > vcpus=2 > cpus=[1,3] > > I really need available CPU power of 2 CPUs, or the machine will be too > underpowered... > Ok. -- Pasi ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: domU crashing, trying to determine why 2011-01-03 5:34 ` Pasi Kärkkäinen @ 2011-01-03 6:32 ` micah anderson 2011-01-03 16:40 ` Konrad Rzeszutek Wilk 1 sibling, 0 replies; 17+ messages in thread From: micah anderson @ 2011-01-03 6:32 UTC (permalink / raw) To: Pasi Kärkkäinen; +Cc: xen-devel [-- Attachment #1.1: Type: text/plain, Size: 2199 bytes --] On Mon, 3 Jan 2011 07:34:35 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote: > On Sun, Jan 02, 2011 at 10:41:24PM -0500, micah anderson wrote: > > On Sun, 2 Jan 2011 20:39:09 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote: > > > On Sun, Jan 02, 2011 at 01:15:22PM -0500, micah anderson wrote: > > > > On Sun, 2 Jan 2011 14:59:13 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote: > > > > > On Sat, Jan 01, 2011 at 07:43:32PM -0500, micah anderson wrote: > > > > > > On Sat, 1 Jan 2011 23:41:06 +0200, Pasi Kärkkäinen <pasik@iki.fi> wrote: > > > > > > > On Sat, Jan 01, 2011 at 02:16:55PM -0500, Micah Anderson wrote: > > > > > > > > > > > > > > > > I'm having a domU that is regularly crashing, without any obvious reason > > > > > > > > why. I set 'on_crash=preserve' to try and get a crash to try and get > > > > > > > > some clues, but what I got doesn't too obvious to me. I was hoping > > > > > > > > someone here would be able to give me some clues. > > > > > > > > > > > > > > > > Thanks for any clues anyone can give! Here is what I managed to eke out: > > > > > > > > > > > > > > > > root@guillemot:/etc/xen# /usr/lib/xen-3.2-1/bin/xenctx -s /boot/System.map-2.6.26-2-xen-686 11 0 > > > > > > > > > > > > > > So the domU has only 1 vcpu? > > > > > > > > > > > > No, I did this for both vcpus, the second one is below: > > > > > > > > > > > > > > > > True, I somehow missed the other one. > > > > > Did you check debian bugreports if there are other reports like this? > > > > > > > > I did, but I didn't find anything relevant. > > > > > > > > > > Can you list the steps to reproduce the bug/crash? Does it happen every time? > > > Does it happen if you have only 1 vcpu for the domU? > > > > There really aren't any steps. I 'xm create' the domain, and then what > > seems like a random amount of time later, it crashes. Usually that > > random amount of time is somewhere between one week and four weeks, but > > I've had it happen twice in a weekend. > > > > What kind of workload is running in the domU ? Its a database server, only running mysql. Its used fairly heavily for user authentication, and email account verification. [-- Attachment #1.2: Type: application/pgp-signature, Size: 835 bytes --] [-- Attachment #2: Type: text/plain, Size: 138 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: domU crashing, trying to determine why 2011-01-03 5:34 ` Pasi Kärkkäinen 2011-01-03 6:32 ` micah anderson @ 2011-01-03 16:40 ` Konrad Rzeszutek Wilk 2011-01-03 17:03 ` micah anderson 2011-01-04 10:54 ` Ian Campbell 1 sibling, 2 replies; 17+ messages in thread From: Konrad Rzeszutek Wilk @ 2011-01-03 16:40 UTC (permalink / raw) To: Pasi Kärkkäinen; +Cc: micah anderson, xen-devel > > I've got this setup: > > > > vcpus=2 > > cpus=[1,3] Is there anything in the Xen hypervisor ring buffer? You might need to enable the full logging of the guest and dom0 to get anything there. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: domU crashing, trying to determine why 2011-01-03 16:40 ` Konrad Rzeszutek Wilk @ 2011-01-03 17:03 ` micah anderson 2011-01-03 17:14 ` Konrad Rzeszutek Wilk 2011-01-04 10:54 ` Ian Campbell 1 sibling, 1 reply; 17+ messages in thread From: micah anderson @ 2011-01-03 17:03 UTC (permalink / raw) To: Konrad Rzeszutek Wilk, Pasi Kärkkäinen; +Cc: xen-devel [-- Attachment #1.1: Type: text/plain, Size: 432 bytes --] On Mon, 3 Jan 2011 11:40:31 -0500, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote: > > > I've got this setup: > > > > > > vcpus=2 > > > cpus=[1,3] > > Is there anything in the Xen hypervisor ring buffer? You might need to enable the > full logging of the guest and dom0 to get anything there. How do I get at the ring buffer, and what specifically should I enable to get all of that? thanks! micah -- [-- Attachment #1.2: Type: application/pgp-signature, Size: 835 bytes --] [-- Attachment #2: Type: text/plain, Size: 138 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: domU crashing, trying to determine why 2011-01-03 17:03 ` micah anderson @ 2011-01-03 17:14 ` Konrad Rzeszutek Wilk 2011-01-14 17:31 ` micah anderson 0 siblings, 1 reply; 17+ messages in thread From: Konrad Rzeszutek Wilk @ 2011-01-03 17:14 UTC (permalink / raw) To: micah anderson; +Cc: xen-devel On Mon, Jan 03, 2011 at 12:03:34PM -0500, micah anderson wrote: > On Mon, 3 Jan 2011 11:40:31 -0500, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote: > > > > I've got this setup: > > > > > > > > vcpus=2 > > > > cpus=[1,3] > > > > Is there anything in the Xen hypervisor ring buffer? You might need to enable the > > full logging of the guest and dom0 to get anything there. > > How do I get at the ring buffer, and what specifically should I enable 'xm dmesg' > to get all of that? In http://wiki.xensource.com/xenwiki/XenParavirtOps "Are there more debugging options I could enable to troubleshoot booting problems?" > > thanks! > micah > > -- > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: domU crashing, trying to determine why 2011-01-03 17:14 ` Konrad Rzeszutek Wilk @ 2011-01-14 17:31 ` micah anderson 2011-01-14 17:45 ` Konrad Rzeszutek Wilk 0 siblings, 1 reply; 17+ messages in thread From: micah anderson @ 2011-01-14 17:31 UTC (permalink / raw) To: Konrad Rzeszutek Wilk; +Cc: xen-devel [-- Attachment #1.1: Type: text/plain, Size: 1622 bytes --] On Mon, 3 Jan 2011 12:14:25 -0500, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote: > On Mon, Jan 03, 2011 at 12:03:34PM -0500, micah anderson wrote: > > On Mon, 3 Jan 2011 11:40:31 -0500, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote: > > > > > I've got this setup: > > > > > > > > > > vcpus=2 > > > > > cpus=[1,3] > > > > > > Is there anything in the Xen hypervisor ring buffer? You might need to enable the > > > full logging of the guest and dom0 to get anything there. > > > > How do I get at the ring buffer, and what specifically should I enable > > 'xm dmesg' (XEN) mm.c:645:d2 Non-privileged (2) attempt to map I/O space 000000f0 (XEN) traps.c:2544: GPF (0068): ff1886c2 -> ff1887c9 (XEN) domain_crash_sync called from entry.S (ff188600) (XEN) Domain 2 (vcpu#1) crashed on cpu#3: (XEN) ----[ Xen-3.2-1 x86_32p debug=n Tainted: C ]---- (XEN) CPU: 3 (XEN) EIP: 0061:[<c01013a7>] (XEN) EFLAGS: 00000246 CONTEXT: guest (XEN) eax: 00000000 ebx: 00000001 ecx: 00000000 edx: ed447f90 (XEN) esi: 00000001 edi: 00000001 ebp: 00000000 esp: ed447f84 (XEN) cr0: 8005003b cr4: 000006f0 cr3: 00bdac80 cr2: 095346dc (XEN) ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0069 cs: 0061 (XEN) Guest stack trace from esp=ed447f84: (XEN) c0105f52 ffffffff 00000001 673bfa5d 0002dba1 00000000 00000000 ffffffff (XEN) c01028ab c0102810 00000000 00000000 00000000 00000000 00000000 00000000 (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 000000d8 00000000 (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [-- Attachment #1.2: Type: application/pgp-signature, Size: 835 bytes --] [-- Attachment #2: Type: text/plain, Size: 138 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: domU crashing, trying to determine why 2011-01-14 17:31 ` micah anderson @ 2011-01-14 17:45 ` Konrad Rzeszutek Wilk 0 siblings, 0 replies; 17+ messages in thread From: Konrad Rzeszutek Wilk @ 2011-01-14 17:45 UTC (permalink / raw) To: micah anderson; +Cc: xen-devel On Fri, Jan 14, 2011 at 12:31:22PM -0500, micah anderson wrote: > On Mon, 3 Jan 2011 12:14:25 -0500, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote: > > On Mon, Jan 03, 2011 at 12:03:34PM -0500, micah anderson wrote: > > > On Mon, 3 Jan 2011 11:40:31 -0500, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote: > > > > > > I've got this setup: > > > > > > > > > > > > vcpus=2 > > > > > > cpus=[1,3] > > > > > > > > Is there anything in the Xen hypervisor ring buffer? You might need to enable the > > > > full logging of the guest and dom0 to get anything there. > > > > > > How do I get at the ring buffer, and what specifically should I enable > > > > 'xm dmesg' > > (XEN) mm.c:645:d2 Non-privileged (2) attempt to map I/O space 000000f0 Hmm, that should not happen anymore, as there are is a bunch of code guarding domains from trying to touch 0->0x100. Unless your guest has the old stuff. > (XEN) traps.c:2544: GPF (0068): ff1886c2 -> ff1887c9 > (XEN) domain_crash_sync called from entry.S (ff188600) > (XEN) Domain 2 (vcpu#1) crashed on cpu#3: > (XEN) ----[ Xen-3.2-1 x86_32p debug=n Tainted: C ]---- Ugh, ancient hypervisor. > (XEN) CPU: 3 > (XEN) EIP: 0061:[<c01013a7>] Now use gdb to lookup that EIP value. Or use your guest System.map to see what function this could map too. > (XEN) EFLAGS: 00000246 CONTEXT: guest > (XEN) eax: 00000000 ebx: 00000001 ecx: 00000000 edx: ed447f90 > (XEN) esi: 00000001 edi: 00000001 ebp: 00000000 esp: ed447f84 > (XEN) cr0: 8005003b cr4: 000006f0 cr3: 00bdac80 cr2: 095346dc > (XEN) ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0069 cs: 0061 > (XEN) Guest stack trace from esp=ed447f84: > (XEN) c0105f52 ffffffff 00000001 673bfa5d 0002dba1 00000000 00000000 ffffffff > (XEN) c01028ab c0102810 00000000 00000000 00000000 00000000 00000000 00000000 And potentially c01028ab c0105f52 .. > (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 000000d8 00000000 > (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: domU crashing, trying to determine why 2011-01-03 16:40 ` Konrad Rzeszutek Wilk 2011-01-03 17:03 ` micah anderson @ 2011-01-04 10:54 ` Ian Campbell 2011-01-04 19:49 ` micah anderson 1 sibling, 1 reply; 17+ messages in thread From: Ian Campbell @ 2011-01-04 10:54 UTC (permalink / raw) To: Konrad Rzeszutek Wilk; +Cc: micah anderson, xen-devel@lists.xensource.com On Mon, 2011-01-03 at 16:40 +0000, Konrad Rzeszutek Wilk wrote: > > > I've got this setup: > > > > > > vcpus=2 > > > cpus=[1,3] > > Is there anything in the Xen hypervisor ring buffer? Equally was there anything on the guest console? If it is dying before you can look then you can enable logging via XENCONSOLED_TRACE=guest in /etc/sysconfig/xencommons or /etc/default/xencommons (depending on distro). Logs show up in /var/log/xen/guest/ If the guest is still running but deadlocked or similar then it might be worth sending a few SysRQs to it. To do this press Ctrl-O on the guest console and press one of the magic letters, 'h' will give help. 't' (tasks) would be a useful start as would 'p' (processor state). It might be worth compiling your guest kernel with lockdep support and seeing if it complains about anything. Also, I notice you seem to be running the Debian Lenny classic-Xen forward port. Are you running the most recent version of that kernel? Unfortunately there is not much developer interest in these classic-Xen forwards ports these days. You could also try the -686-bigmem kernel flavour from Lenny which is a PVops enabled kernel, or perhaps a Squeeze kernel from backports.org (either -686-bigmem or -xen-686, both are pvops kernels). If your problem persists with these then there is more chance of it getting fixed than with the Lenny -xen-686 flavour. Ian. > You might need to enable the > full logging of the guest and dom0 to get anything there. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: domU crashing, trying to determine why 2011-01-04 10:54 ` Ian Campbell @ 2011-01-04 19:49 ` micah anderson 2011-01-05 13:27 ` Ian Campbell 0 siblings, 1 reply; 17+ messages in thread From: micah anderson @ 2011-01-04 19:49 UTC (permalink / raw) To: Ian Campbell, Konrad Rzeszutek Wilk; +Cc: xen-devel@lists.xensource.com [-- Attachment #1.1: Type: text/plain, Size: 2378 bytes --] On Tue, 4 Jan 2011 10:54:44 +0000, Ian Campbell <Ian.Campbell@citrix.com> wrote: > On Mon, 2011-01-03 at 16:40 +0000, Konrad Rzeszutek Wilk wrote: > > > > I've got this setup: > > > > > > > > vcpus=2 > > > > cpus=[1,3] > > > > Is there anything in the Xen hypervisor ring buffer? > > Equally was there anything on the guest console? If it is dying before > you can look then you can enable logging via XENCONSOLED_TRACE=guest > in /etc/sysconfig/xencommons or /etc/default/xencommons (depending on > distro). Logs show up in /var/log/xen/guest/ Because I configured 'on_crash = preserve' I can see the console after it crashes, but no, there is nothing on the guest console, except for the login prompt (which of course no longer works). > If the guest is still running but deadlocked or similar then it might be > worth sending a few SysRQs to it. To do this press Ctrl-O on the guest > console and press one of the magic letters, 'h' will give help. > 't' (tasks) would be a useful start as would 'p' (processor state). I'll try this on the next crash. It crashed this morning, so I took the liberty of adding the kernel log parameters so I can get xen hypervisor ring buffer debugging information for the next crash. > It might be worth compiling your guest kernel with lockdep support and > seeing if it complains about anything. Is this a .config option? If so, what is it? > Also, I notice you seem to be running the Debian Lenny classic-Xen > forward port. Are you running the most recent version of that kernel? > Unfortunately there is not much developer interest in these classic-Xen > forwards ports these days. Yes, I am running the up-to-date Debian Lenny kernel. One solution of course is to upgrade the system to Squeeze to get an overhaul of the whole subsystem, getting a newer kernel/xen completely... but of course that is a bigger change, and would not give us any clue as to what is going on. > You could also try the -686-bigmem kernel flavour from Lenny which is a > PVops enabled kernel, or perhaps a Squeeze kernel from backports.org > (either -686-bigmem or -xen-686, both are pvops kernels). If your > problem persists with these then there is more chance of it getting > fixed than with the Lenny -xen-686 flavour. Do you mean run that kernel in the Dom0, or the DomU? thanks, micah [-- Attachment #1.2: Type: application/pgp-signature, Size: 835 bytes --] [-- Attachment #2: Type: text/plain, Size: 138 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: domU crashing, trying to determine why 2011-01-04 19:49 ` micah anderson @ 2011-01-05 13:27 ` Ian Campbell 0 siblings, 0 replies; 17+ messages in thread From: Ian Campbell @ 2011-01-05 13:27 UTC (permalink / raw) To: micah anderson; +Cc: xen-devel@lists.xensource.com, Konrad Rzeszutek Wilk On Tue, 2011-01-04 at 19:49 +0000, micah anderson wrote: > On Tue, 4 Jan 2011 10:54:44 +0000, Ian Campbell <Ian.Campbell@citrix.com> wrote: > > On Mon, 2011-01-03 at 16:40 +0000, Konrad Rzeszutek Wilk wrote: > > It might be worth compiling your guest kernel with lockdep support and > > seeing if it complains about anything. > > Is this a .config option? If so, what is it? It's CONFIG_LOCKDEP. It always takes me several minutes of trial and error to figure out what dependencies I need to turn it on... > > You could also try the -686-bigmem kernel flavour from Lenny which is a > > PVops enabled kernel, or perhaps a Squeeze kernel from backports.org > > (either -686-bigmem or -xen-686, both are pvops kernels). If your > > problem persists with these then there is more chance of it getting > > fixed than with the Lenny -xen-686 flavour. > > Do you mean run that kernel in the Dom0, or the DomU? In the domU. Currently only the -xen-686 flavours work for dom0. Ian. ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2011-01-14 17:45 UTC | newest] Thread overview: 17+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-01-01 19:16 domU crashing, trying to determine why Micah Anderson 2011-01-01 21:41 ` Pasi Kärkkäinen 2011-01-02 0:43 ` micah anderson 2011-01-02 12:59 ` Pasi Kärkkäinen 2011-01-02 18:15 ` micah anderson 2011-01-02 18:39 ` Pasi Kärkkäinen 2011-01-03 3:41 ` micah anderson 2011-01-03 5:34 ` Pasi Kärkkäinen 2011-01-03 6:32 ` micah anderson 2011-01-03 16:40 ` Konrad Rzeszutek Wilk 2011-01-03 17:03 ` micah anderson 2011-01-03 17:14 ` Konrad Rzeszutek Wilk 2011-01-14 17:31 ` micah anderson 2011-01-14 17:45 ` Konrad Rzeszutek Wilk 2011-01-04 10:54 ` Ian Campbell 2011-01-04 19:49 ` micah anderson 2011-01-05 13:27 ` Ian Campbell
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).