setting time in guest with ntpdate results in VM hang

setting time in guest with ntpdate results in VM hang

[Dale Swanston]

Hello.

We are running a guest OS of CentOS 4.4 (kernel 2.6.12) for legacy 
reasons, upgrading is not an option.   NTP is running on the host and 
synching with a local GPS NTP server.  But due to frequency drift in the 
guest it restarts itself periodically and upon start up performs an 
ntpdate to force a time jump on the guest.

I have seen 2 occasions now (over 2 months) where the VM hangs right as 
the ntpdate command alters the guest clock (based on output in 
/var/log/messages).

 From the host's perspective the VM is still running but it appears to 
be using very high CPU percentage (more than typical).  The only 
recovery option is to force shutdown of the VM and restart it.

1.  Are there any known issues with ntpdate and VMs hanging?  Any 
workarounds?
2.  Are there any debugging tools further characterise the problem?

Thanks in advance,
Dale


Further specifics, let me know if more details are needed:

Host:
# uname -a
Linux s1pscu01host 2.6.32-220.el6.x86_64 #1 SMP Sat Dec 10 17:04:11 CST 
2011 x86_64 x86_64 x86_64 GNU/Linux

on  Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz

with qemu-kvm-0.12.1.2-2.209.el6.x86_64

Guest:
I don't use any kernel time modifiers in the guest ...
kernel /vmlinuz-2.6.12 ro root=/dev/VolGroup00/LogVol00 rhgb quiet

CentOS 4.4

Re: setting time in guest with ntpdate results in VM hang

[Marcelo Tosatti]

On Fri, Aug 24, 2012 at 09:57:35AM -0600, Dale Swanston wrote:
> Hello.
> 
> We are running a guest OS of CentOS 4.4 (kernel 2.6.12) for legacy
> reasons, upgrading is not an option.   NTP is running on the host
> and synching with a local GPS NTP server.  But due to frequency
> drift in the guest it restarts itself periodically and upon start up
> performs an ntpdate to force a time jump on the guest.
> 
> I have seen 2 occasions now (over 2 months) where the VM hangs right
> as the ntpdate command alters the guest clock (based on output in
> /var/log/messages).
> 
> From the host's perspective the VM is still running but it appears
> to be using very high CPU percentage (more than typical).  The only
> recovery option is to force shutdown of the VM and restart it.

This should not happen.

> 1.  Are there any known issues with ntpdate and VMs hanging?  Any
> workarounds?
> 2.  Are there any debugging tools further characterise the problem?

Upgrading the guest kernel is not an option? At least install recent
kernel in guest to confirm that its not an already fixed bug.

> Thanks in advance,
> Dale
> 
> 
> Further specifics, let me know if more details are needed:
> 
> Host:
> # uname -a
> Linux s1pscu01host 2.6.32-220.el6.x86_64 #1 SMP Sat Dec 10 17:04:11
> CST 2011 x86_64 x86_64 x86_64 GNU/Linux
> 
> on  Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
> 
> with qemu-kvm-0.12.1.2-2.209.el6.x86_64
> 
> Guest:
> I don't use any kernel time modifiers in the guest ...
> kernel /vmlinuz-2.6.12 ro root=/dev/VolGroup00/LogVol00 rhgb quiet
> 
> CentOS 4.4
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: setting time in guest with ntpdate results in VM hang

[Dale Swanston]

On 8/24/2012 1:43 PM, Marcelo Tosatti wrote:
> On Fri, Aug 24, 2012 at 09:57:35AM -0600, Dale Swanston wrote:
>> Hello.
>>
>> We are running a guest OS of CentOS 4.4 (kernel 2.6.12) for legacy
>> reasons, upgrading is not an option.   NTP is running on the host
>> and synching with a local GPS NTP server.  But due to frequency
>> drift in the guest it restarts itself periodically and upon start up
>> performs an ntpdate to force a time jump on the guest.
>>
>> I have seen 2 occasions now (over 2 months) where the VM hangs right
>> as the ntpdate command alters the guest clock (based on output in
>> /var/log/messages).
>>
>>  From the host's perspective the VM is still running but it appears
>> to be using very high CPU percentage (more than typical).  The only
>> recovery option is to force shutdown of the VM and restart it.
> This should not happen.
>
>> 1.  Are there any known issues with ntpdate and VMs hanging?  Any
>> workarounds?
>> 2.  Are there any debugging tools further characterise the problem?
> Upgrading the guest kernel is not an option? At least install recent
> kernel in guest to confirm that its not an already fixed bug.

Good idea.  I'll try that.

But are there any tools available to determine what the VM is doing when 
it appears hung?  I've looked but haven't found much on debug or 
diagnostics on a running VM.  Any links?

Is it possible the guest kernel is panicking? What would the VM do if 
that happened?  Would it shutdown?

Thanks again.

Re: setting time in guest with ntpdate results in VM hang

[David Ahern]

On 8/27/12 10:58 AM, Dale Swanston wrote:
> Good idea.  I'll try that.
>
> But are there any tools available to determine what the VM is doing when
> it appears hung?  I've looked but haven't found much on debug or
> diagnostics on a running VM.  Any links?

If you have the vmlinux, enable the gdbserver stub via Qemu's monitor. 
Then use 'gdb vmlinux', connect to the VM 'target remote host:port' and 
look at the backtrace.

I have seen something similar using kvm-clock in a guest running 2.6.27.

David

Re: setting time in guest with ntpdate results in VM hang

[Marcelo Tosatti]

On Mon, Aug 27, 2012 at 01:23:05PM -0600, David Ahern wrote:
> On 8/27/12 10:58 AM, Dale Swanston wrote:
> >Good idea.  I'll try that.
> >
> >But are there any tools available to determine what the VM is doing when
> >it appears hung?  I've looked but haven't found much on debug or
> >diagnostics on a running VM.  Any links?
> 
> If you have the vmlinux, enable the gdbserver stub via Qemu's
> monitor. Then use 'gdb vmlinux', connect to the VM 'target remote
> host:port' and look at the backtrace.

Another option is to boot the host with profile=kvm, wait for the guest to hang,
then do:

readprofile -r ; readprofile -m System-map-of-guest.map

> I have seen something similar using kvm-clock in a guest running 2.6.27.
> 
> David