Re: [Qemu-devel] Hang on reboot in FreeBSD guest on Linux KVM host

Re: [Qemu-devel] Hang on reboot in FreeBSD guest on Linux KVM host

[John Nielsen]

[Adding -devel to CC.]

I've opened a bug regarding this issue:

https://bugs.launchpad.net/qemu/+bug/1329956

I was unable to duplicate the issue in a different hypervisor, so it seems likely the problem (or at least its solution) is in Qemu. Any ideas or feedback still very much appreciated.

JN

On May 9, 2014, at 2:12 PM, John Nielsen <lists@jnielsen.net> wrote:

> I am trying to solve a problem with x86_64 FreeBSD virtual machines running on a Linux+libvirt+KVM hypervisor. To be honest I'm not sure if the problem is in FreeBSD or the hypervisor, but I'm trying to approach it from both directions.
> 
> The _second_ time FreeBSD boots in a virtual machine with more than one core, the boot hangs just before the kernel would normally bring up the additional processors. The VM will boot fine a first time, but running either "shutdown -r now" OR "reboot" will lead to a hung second boot. Stopping and starting the host qemu-kvm process is the only way to continue.
> 
> Interestingly the problem does not manifest itself after a "virsh reset" on the host. I can also avoid it by patching the guest kernel to skip the SMP part of the shutdown routine.
> 
> Clearly something in the normal FreeBSD shutdown code ls leaving the VM in a bad state that hinders the next boot, but I haven't been able to identify what exactly. Can someone on the list suggest ways to debug this further? Unless it's a FreeBSD bug, I'd like to find a solution or workaround that doesn't involve modifying the guest OS.
> 
> One more thing: the problem only appears on one of two clusters of host machines. The hosts within each cluster are identical, and the two clusters are _nearly_ identical to each other. All are running the same software, including:
>  CentOS 6.5
>  kernel 3.12.13 (custom)
>  libvirt-1.1.4-2.el6
>  qemu-kvm-1.7.0-2.el6
>  seabios-1.7.3.1-1.el6
> 
> The only substantial difference on the hardware side is the CPU. The hosts where the problem occurs use "Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz", while the hosts that don't show the problem use the prior revision, "Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz".
> 
> All ideas appreciated.

Re: [Qemu-devel] Hang on reboot in FreeBSD guest on Linux KVM host

[Paolo Bonzini]

Il 16/06/2014 18:09, John Nielsen ha scritto:
>>> The only substantial difference on the hardware side is the CPU.
>>> The hosts where the problem occurs use "Intel(R) Xeon(R) CPU
>>> E5-2650 v2 @ 2.60GHz", while the hosts that don't show the
>>> problem use the prior revision, "Intel(R) Xeon(R) CPU E5-2650 0 @
>>> 2.00GHz".

Can you do "grep . /sys/module/kvm_intel/parameters/*" on both hosts please?

Paolo

Re: [Qemu-devel] Hang on reboot in FreeBSD guest on Linux KVM host

[John Nielsen]

On Jun 16, 2014, at 10:39 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:

> Il 16/06/2014 18:09, John Nielsen ha scritto:
>>>> The only substantial difference on the hardware side is the CPU.
>>>> The hosts where the problem occurs use "Intel(R) Xeon(R) CPU
>>>> E5-2650 v2 @ 2.60GHz", while the hosts that don't show the
>>>> problem use the prior revision, "Intel(R) Xeon(R) CPU E5-2650 0 @
>>>> 2.00GHz".
> 
> Can you do "grep . /sys/module/kvm_intel/parameters/*" on both hosts please?

No differences that I can see. Output below.

Working host:
Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
# grep . /sys/module/kvm_intel/parameters/*
/sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
/sys/module/kvm_intel/parameters/enable_apicv:N
/sys/module/kvm_intel/parameters/enable_shadow_vmcs:N
/sys/module/kvm_intel/parameters/ept:Y
/sys/module/kvm_intel/parameters/eptad:N
/sys/module/kvm_intel/parameters/fasteoi:Y
/sys/module/kvm_intel/parameters/flexpriority:Y
/sys/module/kvm_intel/parameters/nested:N
/sys/module/kvm_intel/parameters/ple_gap:128
/sys/module/kvm_intel/parameters/ple_window:4096
/sys/module/kvm_intel/parameters/unrestricted_guest:Y
/sys/module/kvm_intel/parameters/vmm_exclusive:Y
/sys/module/kvm_intel/parameters/vpid:Y

Problem host:
Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
# grep . /sys/module/kvm_intel/parameters/*
/sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
/sys/module/kvm_intel/parameters/enable_apicv:Y
/sys/module/kvm_intel/parameters/enable_shadow_vmcs:N
/sys/module/kvm_intel/parameters/ept:Y
/sys/module/kvm_intel/parameters/eptad:N
/sys/module/kvm_intel/parameters/fasteoi:Y
/sys/module/kvm_intel/parameters/flexpriority:Y
/sys/module/kvm_intel/parameters/nested:N
/sys/module/kvm_intel/parameters/ple_gap:128
/sys/module/kvm_intel/parameters/ple_window:4096
/sys/module/kvm_intel/parameters/unrestricted_guest:Y
/sys/module/kvm_intel/parameters/vmm_exclusive:Y
/sys/module/kvm_intel/parameters/vpid:Y

Re: [Qemu-devel] Hang on reboot in FreeBSD guest on Linux KVM host

[Paolo Bonzini]

Il 16/06/2014 18:47, John Nielsen ha scritto:
> On Jun 16, 2014, at 10:39 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>> Il 16/06/2014 18:09, John Nielsen ha scritto:
>>>>> The only substantial difference on the hardware side is the CPU.
>>>>> The hosts where the problem occurs use "Intel(R) Xeon(R) CPU
>>>>> E5-2650 v2 @ 2.60GHz", while the hosts that don't show the
>>>>> problem use the prior revision, "Intel(R) Xeon(R) CPU E5-2650 0 @
>>>>> 2.00GHz".
>>
>> Can you do "grep . /sys/module/kvm_intel/parameters/*" on both hosts please?
>
> No differences that I can see. Output below.

Not really:

> Working host:
> Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
> # grep . /sys/module/kvm_intel/parameters/*
> /sys/module/kvm_intel/parameters/enable_apicv:N
>
> Problem host:
> Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
> # grep . /sys/module/kvm_intel/parameters/*
> /sys/module/kvm_intel/parameters/enable_apicv:Y

So we have a clue.  Let me study the code more, I'll try to get back 
with a suggestion.

In the meanwhile, I'm CCing the KVM list and BCCing QEMU, so that 
follow-ups come to the KVM list.

Paolo