* KVM Guest Lock up (100%) again! [not found] <2122448695.1031741.1365086071426.JavaMail.root@innovot.com> @ 2013-04-04 14:36 ` Phil Daws 2013-04-08 20:02 ` Phil Daws 0 siblings, 1 reply; 6+ messages in thread From: Phil Daws @ 2013-04-04 14:36 UTC (permalink / raw) To: kvm One of my KVM guests locked up again at 100% CPU! Any thoughts on how I can diagnose it ? We would love to put into production but am very concerned about the current stability. I have tried to re-direct the console, through screen, to see whether there is a spin lock that is causing the problem; though all I got in the log file was a login prompt. What is the correct way of redirecting the console in KVM on a CentOS 6.4 system please ? Thanks. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: KVM Guest Lock up (100%) again! 2013-04-04 14:36 ` KVM Guest Lock up (100%) again! Phil Daws @ 2013-04-08 20:02 ` Phil Daws 2013-04-10 14:10 ` Gleb Natapov 0 siblings, 1 reply; 6+ messages in thread From: Phil Daws @ 2013-04-08 20:02 UTC (permalink / raw) To: kvm Hello all, Another lock up again this evening :( am wondering whether should consider upgrading the kernel to 3.7.10 and the latest version of KVM. Thoughts ? Thanks. ----- Original Message ----- To: kvm@vger.kernel.org Sent: Thursday, 4 April, 2013 3:36:11 PM Subject: KVM Guest Lock up (100%) again! One of my KVM guests locked up again at 100% CPU! Any thoughts on how I can diagnose it ? We would love to put into production but am very concerned about the current stability. I have tried to re-direct the console, through screen, to see whether there is a spin lock that is causing the problem; though all I got in the log file was a login prompt. What is the correct way of redirecting the console in KVM on a CentOS 6.4 system please ? Thanks. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: KVM Guest Lock up (100%) again! 2013-04-08 20:02 ` Phil Daws @ 2013-04-10 14:10 ` Gleb Natapov 2013-04-12 14:10 ` Phil Daws 0 siblings, 1 reply; 6+ messages in thread From: Gleb Natapov @ 2013-04-10 14:10 UTC (permalink / raw) To: Phil Daws; +Cc: kvm On Mon, Apr 08, 2013 at 09:02:04PM +0100, Phil Daws wrote: > Hello all, > > Another lock up again this evening :( am wondering whether should consider upgrading the kernel to 3.7.10 and the latest version of KVM. Thoughts ? > Please try with latest kvm/qemu. This is not the list to report vendor specific problems. > Thanks. > > ----- Original Message ----- > To: kvm@vger.kernel.org > Sent: Thursday, 4 April, 2013 3:36:11 PM > Subject: KVM Guest Lock up (100%) again! > > One of my KVM guests locked up again at 100% CPU! Any thoughts on how I can diagnose it ? We would love to put into production but am very concerned about the current stability. I have tried to re-direct the console, through screen, to see whether there is a spin lock that is causing the problem; though all I got in the log file was a login prompt. What is the correct way of redirecting the console in KVM on a CentOS 6.4 system please ? Thanks. > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Gleb. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: KVM Guest Lock up (100%) again! 2013-04-10 14:10 ` Gleb Natapov @ 2013-04-12 14:10 ` Phil Daws 2013-04-12 15:13 ` Gleb Natapov 0 siblings, 1 reply; 6+ messages in thread From: Phil Daws @ 2013-04-12 14:10 UTC (permalink / raw) To: Gleb Natapov; +Cc: kvm Well this is still happening ... I have tried to isolate what could be causing but not much luck yet. Thought the VMs may have been IO bound but that not the case and even tried upping the vCPU allocation from one to two as plenty of head room. When it locks up I see this on a strace: [pid 1343] read(14, 0x7fff82aeb360, 4096) = -1 EAGAIN (Resource temporarily unavailable) [pid 1343] read(7, "\0", 512) = 1 [pid 1343] read(7, 0x7fff82aec160, 512) = -1 EAGAIN (Resource temporarily unavailable) [pid 1343] select(26, [7 10 13 14 16 17 22 25], [], [], {1, 0}) = 1 (in [16], left {0, 999981}) [pid 1343] read(16, "\16\0\0\0\0\0\0\0\376\377\377\377\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0"..., 128) = 128 [pid 1343] rt_sigaction(SIGALRM, NULL, {0x7f210b2c0510, ~[KILL STOP RTMIN RT_1], SA_RESTORER, 0x7f210ac22500}, 8) = 0 [pid 1343] write(8, "\0", 1) = 1 [pid 1343] write(15, "\1\0\0\0\0\0\0\0", 8) = 8 [pid 1343] read(16, 0x7fff82aec2d0, 128) = -1 EAGAIN (Resource temporarily unavailable) [pid 1343] timer_gettime(0x1, {it_interval={0, 0}, it_value={0, 0}}) = 0 [pid 1343] timer_settime(0x1, 0, {it_interval={0, 0}, it_value={0, 656000000}}, NULL) = 0 [pid 1343] select(26, [7 10 13 14 16 17 22 25], [], [], {1, 0}) = 2 (in [7 14], left {0, 999998}) [pid 1343] read(14, "\1\0\0\0\0\0\0\0", 4096) = 8 [pid 1343] read(14, 0x7fff82aeb360, 4096) = -1 EAGAIN (Resource temporarily unavailable) [pid 1343] read(7, "\0", 512) = 1 [pid 1343] read(7, 0x7fff82aec160, 512) = -1 EAGAIN (Resource temporarily unavailable) Does that shed any light ? Trying to find a how to for upgrading to the latest KVM/QEMU. Thanks. ----- Original Message ----- From: "Gleb Natapov" <gleb@redhat.com> To: "Phil Daws" <uxbod@splatnix.net> Cc: kvm@vger.kernel.org Sent: Wednesday, 10 April, 2013 3:10:27 PM Subject: Re: KVM Guest Lock up (100%) again! On Mon, Apr 08, 2013 at 09:02:04PM +0100, Phil Daws wrote: > Hello all, > > Another lock up again this evening :( am wondering whether should consider upgrading the kernel to 3.7.10 and the latest version of KVM. Thoughts ? > Please try with latest kvm/qemu. This is not the list to report vendor specific problems. > Thanks. > > ----- Original Message ----- > To: kvm@vger.kernel.org > Sent: Thursday, 4 April, 2013 3:36:11 PM > Subject: KVM Guest Lock up (100%) again! > > One of my KVM guests locked up again at 100% CPU! Any thoughts on how I can diagnose it ? We would love to put into production but am very concerned about the current stability. I have tried to re-direct the console, through screen, to see whether there is a spin lock that is causing the problem; though all I got in the log file was a login prompt. What is the correct way of redirecting the console in KVM on a CentOS 6.4 system please ? Thanks. > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Gleb. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: KVM Guest Lock up (100%) again! 2013-04-12 14:10 ` Phil Daws @ 2013-04-12 15:13 ` Gleb Natapov 2013-04-12 16:31 ` Phil Daws 0 siblings, 1 reply; 6+ messages in thread From: Gleb Natapov @ 2013-04-12 15:13 UTC (permalink / raw) To: Phil Daws; +Cc: kvm On Fri, Apr 12, 2013 at 03:10:43PM +0100, Phil Daws wrote: > Well this is still happening ... I have tried to isolate what could be causing but not much luck yet. Thought the VMs may have been IO bound but that not the case and even tried upping the vCPU allocation from one to two as plenty of head room. When it locks up I see this on a strace: > > [pid 1343] read(14, 0x7fff82aeb360, 4096) = -1 EAGAIN (Resource temporarily unavailable) > [pid 1343] read(7, "\0", 512) = 1 > [pid 1343] read(7, 0x7fff82aec160, 512) = -1 EAGAIN (Resource temporarily unavailable) > [pid 1343] select(26, [7 10 13 14 16 17 22 25], [], [], {1, 0}) = 1 (in [16], left {0, 999981}) > [pid 1343] read(16, "\16\0\0\0\0\0\0\0\376\377\377\377\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0"..., 128) = 128 > [pid 1343] rt_sigaction(SIGALRM, NULL, {0x7f210b2c0510, ~[KILL STOP RTMIN RT_1], SA_RESTORER, 0x7f210ac22500}, 8) = 0 > [pid 1343] write(8, "\0", 1) = 1 > [pid 1343] write(15, "\1\0\0\0\0\0\0\0", 8) = 8 > [pid 1343] read(16, 0x7fff82aec2d0, 128) = -1 EAGAIN (Resource temporarily unavailable) > [pid 1343] timer_gettime(0x1, {it_interval={0, 0}, it_value={0, 0}}) = 0 > [pid 1343] timer_settime(0x1, 0, {it_interval={0, 0}, it_value={0, 656000000}}, NULL) = 0 > [pid 1343] select(26, [7 10 13 14 16 17 22 25], [], [], {1, 0}) = 2 (in [7 14], left {0, 999998}) > [pid 1343] read(14, "\1\0\0\0\0\0\0\0", 4096) = 8 > [pid 1343] read(14, 0x7fff82aeb360, 4096) = -1 EAGAIN (Resource temporarily unavailable) > [pid 1343] read(7, "\0", 512) = 1 > [pid 1343] read(7, 0x7fff82aec160, 512) = -1 EAGAIN (Resource temporarily unavailable) > > Does that shed any light ? Trying to find a how to for upgrading to the latest KVM/QEMU. > Is the lockup with upstream now? strace is not very helpful to diagnose kvm problems. Try to run ftrace: http://www.linux-kvm.org/page/Tracing -- Gleb. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: KVM Guest Lock up (100%) again! 2013-04-12 15:13 ` Gleb Natapov @ 2013-04-12 16:31 ` Phil Daws 0 siblings, 0 replies; 6+ messages in thread From: Phil Daws @ 2013-04-12 16:31 UTC (permalink / raw) To: Gleb Natapov; +Cc: kvm Was running two guests on k3.7.10 but have now switched one to stock 2.6.32; and neither have crashed yet. Will leave running as and if it stays stable will switch out the other kernel to stock as-well. Am wondering if have hit a kernel buglet. Thank you for the ftrace info. Have a great weekend. ----- Original Message ----- From: "Gleb Natapov" <gleb@redhat.com> To: "Phil Daws" <uxbod@splatnix.net> Cc: kvm@vger.kernel.org Sent: Friday, 12 April, 2013 4:13:16 PM Subject: Re: KVM Guest Lock up (100%) again! On Fri, Apr 12, 2013 at 03:10:43PM +0100, Phil Daws wrote: > Well this is still happening ... I have tried to isolate what could be causing but not much luck yet. Thought the VMs may have been IO bound but that not the case and even tried upping the vCPU allocation from one to two as plenty of head room. When it locks up I see this on a strace: > > [pid 1343] read(14, 0x7fff82aeb360, 4096) = -1 EAGAIN (Resource temporarily unavailable) > [pid 1343] read(7, "\0", 512) = 1 > [pid 1343] read(7, 0x7fff82aec160, 512) = -1 EAGAIN (Resource temporarily unavailable) > [pid 1343] select(26, [7 10 13 14 16 17 22 25], [], [], {1, 0}) = 1 (in [16], left {0, 999981}) > [pid 1343] read(16, "\16\0\0\0\0\0\0\0\376\377\377\377\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0"..., 128) = 128 > [pid 1343] rt_sigaction(SIGALRM, NULL, {0x7f210b2c0510, ~[KILL STOP RTMIN RT_1], SA_RESTORER, 0x7f210ac22500}, 8) = 0 > [pid 1343] write(8, "\0", 1) = 1 > [pid 1343] write(15, "\1\0\0\0\0\0\0\0", 8) = 8 > [pid 1343] read(16, 0x7fff82aec2d0, 128) = -1 EAGAIN (Resource temporarily unavailable) > [pid 1343] timer_gettime(0x1, {it_interval={0, 0}, it_value={0, 0}}) = 0 > [pid 1343] timer_settime(0x1, 0, {it_interval={0, 0}, it_value={0, 656000000}}, NULL) = 0 > [pid 1343] select(26, [7 10 13 14 16 17 22 25], [], [], {1, 0}) = 2 (in [7 14], left {0, 999998}) > [pid 1343] read(14, "\1\0\0\0\0\0\0\0", 4096) = 8 > [pid 1343] read(14, 0x7fff82aeb360, 4096) = -1 EAGAIN (Resource temporarily unavailable) > [pid 1343] read(7, "\0", 512) = 1 > [pid 1343] read(7, 0x7fff82aec160, 512) = -1 EAGAIN (Resource temporarily unavailable) > > Does that shed any light ? Trying to find a how to for upgrading to the latest KVM/QEMU. > Is the lockup with upstream now? strace is not very helpful to diagnose kvm problems. Try to run ftrace: http://www.linux-kvm.org/page/Tracing -- Gleb. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-04-12 16:33 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <2122448695.1031741.1365086071426.JavaMail.root@innovot.com>
2013-04-04 14:36 ` KVM Guest Lock up (100%) again! Phil Daws
2013-04-08 20:02 ` Phil Daws
2013-04-10 14:10 ` Gleb Natapov
2013-04-12 14:10 ` Phil Daws
2013-04-12 15:13 ` Gleb Natapov
2013-04-12 16:31 ` Phil Daws
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox