KVM host freezing

KVM host freezing

[Marc Haber]

Hi,

I have just started deploying a host doing virtualization with KVM.
The box has an Athlon 64 X2, 4 GB RAM and is running Debian squeeze
with a locally built 2.6,39 kernel and backported versions of qemu-kvm
(0.14.0) and libvirt (0.9.0) from Debian sid. The box is currently
hosting five VMs, all of them Debian systems as well and rather
unloaded. The only time when there is significant load is when all VMs
are simultaneously starting up their cron jobs.

When the host starts up, it immediately spews the following lines to
the console:

kvm: 2865: cpu0 unhandled rdmsr: 0xc0010048
kvm: 2865: cpu0 unhandled wrmsr: 0xc0010048 data 2100000401
kvm: 2865: cpu0 unhandled rdmsr: 0xc0010001
kvm: 2849: cpu0 unhandled rdmsr: 0xc0010048
kvm: 2849: cpu0 unhandled wrmsr: 0xc0010048 data c0579f7cc0010448
kvm: 2849: cpu0 unhandled rdmsr: 0xc0010001
kvm: 2950: cpu0 unhandled rdmsr: 0xc0010048
kvm: 2950: cpu0 unhandled wrmsr: 0xc0010048 data c0579f7cc0010448
kvm: 2849: cpu1 unhandled rdmsr: 0xc0010048
kvm: 2963: cpu0 unhandled rdmsr: 0xc0010112
kvm: 2963: cpu0 unhandled rdmsr: 0xc0010048
kvm: 2963: cpu0 unhandled wrmsr: 0xc0010048 data 2100000401
kvm: 2963: cpu0 unhandled rdmsr: 0xc0010001
kvm: 2963: cpu1 unhandled rdmsr: 0xc0010048
kvm: 2963: cpu1 unhandled wrmsr: 0xc0010048 data 2100000401

Every few days, the system stops dead in its tracks and needs a hard
reset to be revived. I have a serial console, which unfortunately
disconnects me after a few minutes of inactivity, and only caches the
last few lines of activity. Whenever I connect to the serial console
of the frozen system, I have a few lines of the same "unhandled
(rd|wr)msr" messages.

The syslog doesn't show anything strange. The system just stops dead
in its tracks.

Is there any possibility that the freezes have to do with the
"unhandles (rd|wr)msr" messages? When else could be the cause?

In the mean time, I have taken the box offline and am running memtest.
Up to now, everything seems to be fine.

Any hints will be appreciated.

Greetings
Marc



-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Mannheim, Germany  |  lose things."    Winona Ryder | Fon: *49 621 72739834
Nordisch by Nature |  How to make an American Quilt | Fax: *49 3221 2323190

Re: KVM host freezing

[Avi Kivity]

On 06/02/2011 11:25 AM, Marc Haber wrote:
> Hi,
>
> I have just started deploying a host doing virtualization with KVM.
> The box has an Athlon 64 X2, 4 GB RAM and is running Debian squeeze
> with a locally built 2.6,39 kernel and backported versions of qemu-kvm
> (0.14.0) and libvirt (0.9.0) from Debian sid. The box is currently
> hosting five VMs, all of them Debian systems as well and rather
> unloaded. The only time when there is significant load is when all VMs
> are simultaneously starting up their cron jobs.
>
> When the host starts up, it immediately spews the following lines to
> the console:
>
> kvm: 2865: cpu0 unhandled rdmsr: 0xc0010048
> kvm: 2865: cpu0 unhandled wrmsr: 0xc0010048 data 2100000401
> kvm: 2865: cpu0 unhandled rdmsr: 0xc0010001
> kvm: 2849: cpu0 unhandled rdmsr: 0xc0010048
> kvm: 2849: cpu0 unhandled wrmsr: 0xc0010048 data c0579f7cc0010448
> kvm: 2849: cpu0 unhandled rdmsr: 0xc0010001
> kvm: 2950: cpu0 unhandled rdmsr: 0xc0010048
> kvm: 2950: cpu0 unhandled wrmsr: 0xc0010048 data c0579f7cc0010448
> kvm: 2849: cpu1 unhandled rdmsr: 0xc0010048
> kvm: 2963: cpu0 unhandled rdmsr: 0xc0010112
> kvm: 2963: cpu0 unhandled rdmsr: 0xc0010048
> kvm: 2963: cpu0 unhandled wrmsr: 0xc0010048 data 2100000401
> kvm: 2963: cpu0 unhandled rdmsr: 0xc0010001
> kvm: 2963: cpu1 unhandled rdmsr: 0xc0010048
> kvm: 2963: cpu1 unhandled wrmsr: 0xc0010048 data 2100000401
>
> Every few days, the system stops dead in its tracks and needs a hard
> reset to be revived. I have a serial console, which unfortunately
> disconnects me after a few minutes of inactivity, and only caches the
> last few lines of activity. Whenever I connect to the serial console
> of the frozen system, I have a few lines of the same "unhandled
> (rd|wr)msr" messages.
>
> The syslog doesn't show anything strange. The system just stops dead
> in its tracks.
>
> Is there any possibility that the freezes have to do with the
> "unhandles (rd|wr)msr" messages?

Very unlikely.

> When else could be the cause?
>
> In the mean time, I have taken the box offline and am running memtest.
> Up to now, everything seems to be fine.
>
> Any hints will be appreciated.

You might try setting up netconsole to get reliable logging.

Do you have NMIs?  'grep NMI /proc/interrupts'.

Does running 'perf top -F 10000' make the hang come sooner?

-- 
error compiling committee.c: too many arguments to function

Re: KVM host freezing

[Marc Haber]

On Thu, Jun 02, 2011 at 04:41:30PM +0300, Avi Kivity wrote:
> On 06/02/2011 11:25 AM, Marc Haber wrote:
> >Is there any possibility that the freezes have to do with the
> >"unhandles (rd|wr)msr" messages?
> 
> Very unlikely.

What does that mean anyway? It's looked with sufficiently high
priority to get spewed onto the console.

> >When else could be the cause?
> >
> >In the mean time, I have taken the box offline and am running memtest.
> >Up to now, everything seems to be fine.
> >
> >Any hints will be appreciated.
> 
> You might try setting up netconsole to get reliable logging.

logging of what? Of things written to syslog before the freeze occurs
so that they don't reach the disk reliably? I could set up logging
(which facility/priority?) to the serial console.

> Do you have NMIs?  'grep NMI /proc/interrupts'.

about one per minute.

> Does running 'perf top -F 10000' make the hang come sooner?

If that's important, I'll make some effort to compile a statically
linked perf for 2.6.39. Is it?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Mannheim, Germany  |  lose things."    Winona Ryder | Fon: *49 621 72739834
Nordisch by Nature |  How to make an American Quilt | Fax: *49 3221 2323190

Re: KVM host freezing

[Avi Kivity]

On 06/02/2011 11:36 PM, Marc Haber wrote:
> On Thu, Jun 02, 2011 at 04:41:30PM +0300, Avi Kivity wrote:
> >  On 06/02/2011 11:25 AM, Marc Haber wrote:
> >  >Is there any possibility that the freezes have to do with the
> >  >"unhandles (rd|wr)msr" messages?
> >
> >  Very unlikely.
>
> What does that mean anyway?

The guest read or wrote a model specific register which kvm does not 
implement.

> It's looked with sufficiently high
> priority to get spewed onto the console.

It's done so if the problem is the cause of a guest malfunction, we'll 
have a clue.

> >  >When else could be the cause?
> >  >
> >  >In the mean time, I have taken the box offline and am running memtest.
> >  >Up to now, everything seems to be fine.
> >  >
> >  >Any hints will be appreciated.
> >
> >  You might try setting up netconsole to get reliable logging.
>
> logging of what? Of things written to syslog before the freeze occurs
> so that they don't reach the disk reliably? I could set up logging
> (which facility/priority?) to the serial console.

dmesg.  Syslog is unlikely to get anything during a hard freeze.

> >  Do you have NMIs?  'grep NMI /proc/interrupts'.
>
> about one per minute.
>
> >  Does running 'perf top -F 10000' make the hang come sooner?
>
> If that's important, I'll make some effort to compile a statically
> linked perf for 2.6.39. Is it?
>

Yes.  But why statically linked?  The default should work fine.

-- 
error compiling committee.c: too many arguments to function