Re: [Qemu-devel] Hang on reboot in FreeBSD guest on Linux KVM host

Re: [Qemu-devel] Hang on reboot in FreeBSD guest on Linux KVM host

[John Nielsen]

[Adding -devel to CC.]

I've opened a bug regarding this issue:

https://bugs.launchpad.net/qemu/+bug/1329956

I was unable to duplicate the issue in a different hypervisor, so it seems likely the problem (or at least its solution) is in Qemu. Any ideas or feedback still very much appreciated.

JN

On May 9, 2014, at 2:12 PM, John Nielsen <lists@jnielsen.net> wrote:

> I am trying to solve a problem with x86_64 FreeBSD virtual machines running on a Linux+libvirt+KVM hypervisor. To be honest I'm not sure if the problem is in FreeBSD or the hypervisor, but I'm trying to approach it from both directions.
> 
> The _second_ time FreeBSD boots in a virtual machine with more than one core, the boot hangs just before the kernel would normally bring up the additional processors. The VM will boot fine a first time, but running either "shutdown -r now" OR "reboot" will lead to a hung second boot. Stopping and starting the host qemu-kvm process is the only way to continue.
> 
> Interestingly the problem does not manifest itself after a "virsh reset" on the host. I can also avoid it by patching the guest kernel to skip the SMP part of the shutdown routine.
> 
> Clearly something in the normal FreeBSD shutdown code ls leaving the VM in a bad state that hinders the next boot, but I haven't been able to identify what exactly. Can someone on the list suggest ways to debug this further? Unless it's a FreeBSD bug, I'd like to find a solution or workaround that doesn't involve modifying the guest OS.
> 
> One more thing: the problem only appears on one of two clusters of host machines. The hosts within each cluster are identical, and the two clusters are _nearly_ identical to each other. All are running the same software, including:
>  CentOS 6.5
>  kernel 3.12.13 (custom)
>  libvirt-1.1.4-2.el6
>  qemu-kvm-1.7.0-2.el6
>  seabios-1.7.3.1-1.el6
> 
> The only substantial difference on the hardware side is the CPU. The hosts where the problem occurs use "Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz", while the hosts that don't show the problem use the prior revision, "Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz".
> 
> All ideas appreciated.

Re: [Qemu-devel] Hang on reboot in FreeBSD guest on Linux KVM host

[Paolo Bonzini]

Il 16/06/2014 18:09, John Nielsen ha scritto:
>>> The only substantial difference on the hardware side is the CPU.
>>> The hosts where the problem occurs use "Intel(R) Xeon(R) CPU
>>> E5-2650 v2 @ 2.60GHz", while the hosts that don't show the
>>> problem use the prior revision, "Intel(R) Xeon(R) CPU E5-2650 0 @
>>> 2.00GHz".

Can you do "grep . /sys/module/kvm_intel/parameters/*" on both hosts please?

Paolo

Re: [Qemu-devel] Hang on reboot in FreeBSD guest on Linux KVM host

[John Nielsen]

On Jun 16, 2014, at 10:39 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:

> Il 16/06/2014 18:09, John Nielsen ha scritto:
>>>> The only substantial difference on the hardware side is the CPU.
>>>> The hosts where the problem occurs use "Intel(R) Xeon(R) CPU
>>>> E5-2650 v2 @ 2.60GHz", while the hosts that don't show the
>>>> problem use the prior revision, "Intel(R) Xeon(R) CPU E5-2650 0 @
>>>> 2.00GHz".
> 
> Can you do "grep . /sys/module/kvm_intel/parameters/*" on both hosts please?

No differences that I can see. Output below.

Working host:
Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
# grep . /sys/module/kvm_intel/parameters/*
/sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
/sys/module/kvm_intel/parameters/enable_apicv:N
/sys/module/kvm_intel/parameters/enable_shadow_vmcs:N
/sys/module/kvm_intel/parameters/ept:Y
/sys/module/kvm_intel/parameters/eptad:N
/sys/module/kvm_intel/parameters/fasteoi:Y
/sys/module/kvm_intel/parameters/flexpriority:Y
/sys/module/kvm_intel/parameters/nested:N
/sys/module/kvm_intel/parameters/ple_gap:128
/sys/module/kvm_intel/parameters/ple_window:4096
/sys/module/kvm_intel/parameters/unrestricted_guest:Y
/sys/module/kvm_intel/parameters/vmm_exclusive:Y
/sys/module/kvm_intel/parameters/vpid:Y

Problem host:
Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
# grep . /sys/module/kvm_intel/parameters/*
/sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
/sys/module/kvm_intel/parameters/enable_apicv:Y
/sys/module/kvm_intel/parameters/enable_shadow_vmcs:N
/sys/module/kvm_intel/parameters/ept:Y
/sys/module/kvm_intel/parameters/eptad:N
/sys/module/kvm_intel/parameters/fasteoi:Y
/sys/module/kvm_intel/parameters/flexpriority:Y
/sys/module/kvm_intel/parameters/nested:N
/sys/module/kvm_intel/parameters/ple_gap:128
/sys/module/kvm_intel/parameters/ple_window:4096
/sys/module/kvm_intel/parameters/unrestricted_guest:Y
/sys/module/kvm_intel/parameters/vmm_exclusive:Y
/sys/module/kvm_intel/parameters/vpid:Y

Re: Hang on reboot in FreeBSD guest on Linux KVM host

[Paolo Bonzini]

Il 16/06/2014 18:47, John Nielsen ha scritto:
> On Jun 16, 2014, at 10:39 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>> Il 16/06/2014 18:09, John Nielsen ha scritto:
>>>>> The only substantial difference on the hardware side is the CPU.
>>>>> The hosts where the problem occurs use "Intel(R) Xeon(R) CPU
>>>>> E5-2650 v2 @ 2.60GHz", while the hosts that don't show the
>>>>> problem use the prior revision, "Intel(R) Xeon(R) CPU E5-2650 0 @
>>>>> 2.00GHz".
>>
>> Can you do "grep . /sys/module/kvm_intel/parameters/*" on both hosts please?
>
> No differences that I can see. Output below.

Not really:

> Working host:
> Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
> # grep . /sys/module/kvm_intel/parameters/*
> /sys/module/kvm_intel/parameters/enable_apicv:N
>
> Problem host:
> Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
> # grep . /sys/module/kvm_intel/parameters/*
> /sys/module/kvm_intel/parameters/enable_apicv:Y

So we have a clue.  Let me study the code more, I'll try to get back 
with a suggestion.

In the meanwhile, I'm CCing the KVM list and BCCing QEMU, so that 
follow-ups come to the KVM list.

Paolo

Re: [Qemu-devel] Hang on reboot in FreeBSD guest on Linux KVM host

[Paolo Bonzini]

Il 16/06/2014 18:47, John Nielsen ha scritto:
> On Jun 16, 2014, at 10:39 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>> Il 16/06/2014 18:09, John Nielsen ha scritto:
>>>>> The only substantial difference on the hardware side is the CPU.
>>>>> The hosts where the problem occurs use "Intel(R) Xeon(R) CPU
>>>>> E5-2650 v2 @ 2.60GHz", while the hosts that don't show the
>>>>> problem use the prior revision, "Intel(R) Xeon(R) CPU E5-2650 0 @
>>>>> 2.00GHz".
>>
>> Can you do "grep . /sys/module/kvm_intel/parameters/*" on both hosts please?
>
> No differences that I can see. Output below.

Not really:

> Working host:
> Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
> # grep . /sys/module/kvm_intel/parameters/*
> /sys/module/kvm_intel/parameters/enable_apicv:N
>
> Problem host:
> Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
> # grep . /sys/module/kvm_intel/parameters/*
> /sys/module/kvm_intel/parameters/enable_apicv:Y

So we have a clue.  Let me study the code more, I'll try to get back 
with a suggestion.

In the meanwhile, I'm CCing the KVM list and BCCing QEMU, so that 
follow-ups come to the KVM list.

Paolo

Re: Hang on reboot in FreeBSD guest on Linux KVM host

[Gleb Natapov]

On Tue, Jun 17, 2014 at 06:21:23AM +0200, Paolo Bonzini wrote:
> Il 16/06/2014 18:47, John Nielsen ha scritto:
> >On Jun 16, 2014, at 10:39 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> >
> >>Il 16/06/2014 18:09, John Nielsen ha scritto:
> >>>>>The only substantial difference on the hardware side is the CPU.
> >>>>>The hosts where the problem occurs use "Intel(R) Xeon(R) CPU
> >>>>>E5-2650 v2 @ 2.60GHz", while the hosts that don't show the
> >>>>>problem use the prior revision, "Intel(R) Xeon(R) CPU E5-2650 0 @
> >>>>>2.00GHz".
> >>
> >>Can you do "grep . /sys/module/kvm_intel/parameters/*" on both hosts please?
> >
> >No differences that I can see. Output below.
> 
> Not really:
> 
> >Working host:
> >Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
> ># grep . /sys/module/kvm_intel/parameters/*
> >/sys/module/kvm_intel/parameters/enable_apicv:N
> >
> >Problem host:
> >Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
> ># grep . /sys/module/kvm_intel/parameters/*
> >/sys/module/kvm_intel/parameters/enable_apicv:Y
> 
> So we have a clue.  Let me study the code more, I'll try to get back with a
> suggestion.
> 
Does disabling apicv on E5-2650 v2 make reboot problem go away?

--
			Gleb.

Re: Hang on reboot in FreeBSD guest on Linux KVM host

[John Nielsen]

On Jun 17, 2014, at 12:05 AM, Gleb Natapov <gleb@kernel.org> wrote:

> On Tue, Jun 17, 2014 at 06:21:23AM +0200, Paolo Bonzini wrote:
>> Il 16/06/2014 18:47, John Nielsen ha scritto:
>>> On Jun 16, 2014, at 10:39 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>> 
>>>> Il 16/06/2014 18:09, John Nielsen ha scritto:
>>>>>>> The only substantial difference on the hardware side is the CPU.
>>>>>>> The hosts where the problem occurs use "Intel(R) Xeon(R) CPU
>>>>>>> E5-2650 v2 @ 2.60GHz", while the hosts that don't show the
>>>>>>> problem use the prior revision, "Intel(R) Xeon(R) CPU E5-2650 0 @
>>>>>>> 2.00GHz".
>>>> 
>>>> Can you do "grep . /sys/module/kvm_intel/parameters/*" on both hosts please?
>>> 
>>> No differences that I can see. Output below.
>> 
>> Not really:
>> 
>>> Working host:
>>> Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
>>> # grep . /sys/module/kvm_intel/parameters/*
>>> /sys/module/kvm_intel/parameters/enable_apicv:N
>>> 
>>> Problem host:
>>> Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
>>> # grep . /sys/module/kvm_intel/parameters/*
>>> /sys/module/kvm_intel/parameters/enable_apicv:Y
>> 
>> So we have a clue.  Let me study the code more, I'll try to get back with a
>> suggestion.

Wow, can't believe I missed that. Good catch!

> Does disabling apicv on E5-2650 v2 make reboot problem go away?

Yes it does!

# modprobe kvm_intel
/sys/module/kvm_intel/parameters/enable_apicv:Y
# /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512 -smp 2,sockets=1,cores=1,threads=2 -drive file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2 -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none

[problem occurs]

# rmmod kvm_intel
# modprobe kvm_intel enable_apicv=N
/sys/module/kvm_intel/parameters/enable_apicv:N
# /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512 -smp 2,sockets=1,cores=1,threads=2 -drive file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2 -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none

[problem does not occur]

Thank you. This both narrows the problem considerably and provides an acceptable workaround. It would still be nice to see it fixed, of course. Keep me CC'ed as I'm not on the KVM list.

JN

Re: Hang on reboot in FreeBSD guest on Linux KVM host

[John Nielsen]

On Jun 17, 2014, at 10:48 AM, John Nielsen <lists@jnielsen.net> wrote:

> On Jun 17, 2014, at 12:05 AM, Gleb Natapov <gleb@kernel.org> wrote:
> 
>> On Tue, Jun 17, 2014 at 06:21:23AM +0200, Paolo Bonzini wrote:
>>> Il 16/06/2014 18:47, John Nielsen ha scritto:
>>>> On Jun 16, 2014, at 10:39 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>>> 
>>>>> Il 16/06/2014 18:09, John Nielsen ha scritto:
>>>>>>>> The only substantial difference on the hardware side is the CPU.
>>>>>>>> The hosts where the problem occurs use "Intel(R) Xeon(R) CPU
>>>>>>>> E5-2650 v2 @ 2.60GHz", while the hosts that don't show the
>>>>>>>> problem use the prior revision, "Intel(R) Xeon(R) CPU E5-2650 0 @
>>>>>>>> 2.00GHz".
>>>>> 
>>>>> Can you do "grep . /sys/module/kvm_intel/parameters/*" on both hosts please?
>>>> 
>>>> No differences that I can see. Output below.
>>> 
>>> Not really:
>>> 
>>>> Working host:
>>>> Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
>>>> # grep . /sys/module/kvm_intel/parameters/*
>>>> /sys/module/kvm_intel/parameters/enable_apicv:N
>>>> 
>>>> Problem host:
>>>> Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
>>>> # grep . /sys/module/kvm_intel/parameters/*
>>>> /sys/module/kvm_intel/parameters/enable_apicv:Y
>>> 
>>> So we have a clue.  Let me study the code more, I'll try to get back with a
>>> suggestion.
> 
> Wow, can't believe I missed that. Good catch!
> 
>> Does disabling apicv on E5-2650 v2 make reboot problem go away?
> 
> Yes it does!
> 
> # modprobe kvm_intel
> /sys/module/kvm_intel/parameters/enable_apicv:Y
> # /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512 -smp 2,sockets=1,cores=1,threads=2 -drive file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2 -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none
> 
> [problem occurs]
> 
> # rmmod kvm_intel
> # modprobe kvm_intel enable_apicv=N
> /sys/module/kvm_intel/parameters/enable_apicv:N
> # /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512 -smp 2,sockets=1,cores=1,threads=2 -drive file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2 -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none
> 
> [problem does not occur]
> 
> Thank you. This both narrows the problem considerably and provides an acceptable workaround. It would still be nice to see it fixed, of course. Keep me CC'ed as I'm not on the KVM list.

I’m resurrecting an old thread since I haven’t heard anything in a while. Has anyone looked in to the KVM+apicv bug documented above as well as here:

https://bugs.launchpad.net/qemu/+bug/1329956 ?

If appropriate, where should I go to file a KVM bug (since this isn’t really Qemu’s problem)?

Thanks,

JN--
To unsubscribe from this list: send the line "unsubscribe kvm" in

Re: Hang on reboot in FreeBSD guest on Linux KVM host

[Bandan Das]

John Nielsen <lists@jnielsen.net> writes:

> On Jun 17, 2014, at 10:48 AM, John Nielsen <lists@jnielsen.net> wrote:
>
>> On Jun 17, 2014, at 12:05 AM, Gleb Natapov <gleb@kernel.org> wrote:
>> 
>>> On Tue, Jun 17, 2014 at 06:21:23AM +0200, Paolo Bonzini wrote:
>>>> Il 16/06/2014 18:47, John Nielsen ha scritto:
>>>>> On Jun 16, 2014, at 10:39 AM, Paolo Bonzini <pbonzini@redhat.com>
>>>>> wrote:
>>>>> 
>>>>>> Il 16/06/2014 18:09, John Nielsen ha scritto:
>>>>>>>>> The only substantial difference on the hardware side is the
>>>>>>>>> CPU.  The hosts where the problem occurs use "Intel(R)
>>>>>>>>> Xeon(R) CPU E5-2650 v2 @ 2.60GHz", while the hosts that don't
>>>>>>>>> show the problem use the prior revision, "Intel(R) Xeon(R)
>>>>>>>>> CPU E5-2650 0 @ 2.00GHz".
>>>>>>  Can you do "grep . /sys/module/kvm_intel/parameters/*" on both
>>>>>> hosts please?
>>>>>  No differences that I can see. Output below.
>>>>  Not really:
>>>> 
>>>>> Working host: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz # grep
>>>>> . /sys/module/kvm_intel/parameters/*
>>>>> /sys/module/kvm_intel/parameters/enable_apicv:N
>>>>> 
>>>>> Problem host: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz # grep
>>>>> . /sys/module/kvm_intel/parameters/*
>>>>> /sys/module/kvm_intel/parameters/enable_apicv:Y
>>>>  So we have a clue.  Let me study the code more, I'll try to get
>>>> back with a suggestion.
>>  Wow, can't believe I missed that. Good catch!
>> 
>>> Does disabling apicv on E5-2650 v2 make reboot problem go away?
>>  Yes it does!
>> 
>> # modprobe kvm_intel /sys/module/kvm_intel/parameters/enable_apicv:Y
>> # /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m
>> 512 -smp 2,sockets=1,cores=1,threads=2 -drive
>> file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2
>> -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net
>> none
>> 
>> [problem occurs]
>> 
>> # rmmod kvm_intel # modprobe kvm_intel enable_apicv=N
>> /sys/module/kvm_intel/parameters/enable_apicv:N #
>> /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512
>> -smp 2,sockets=1,cores=1,threads=2 -drive
>> file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2
>> -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net
>> none
>> 
>> [problem does not occur]
>> 
>> Thank you. This both narrows the problem considerably and provides
>> an acceptable workaround. It would still be nice to see it fixed, of
>> course. Keep me CC'ed as I'm not on the KVM list.
>
> I’m resurrecting an old thread since I haven’t heard anything in a
> while. Has anyone looked in to the KVM+apicv bug documented above as
> well as here:
>
> https://bugs.launchpad.net/qemu/+bug/1329956 ?
>
> If appropriate, where should I go to file a KVM bug (since this isn’t
> really Qemu’s problem)?

Hi John, does this happen with the latest upstream kernel version ?

Bandan

> Thanks,
>
> JN-- To unsubscribe from this list: send the line "unsubscribe kvm" in
--
To unsubscribe from this list: send the line "unsubscribe kvm" in

Re: Hang on reboot in FreeBSD guest on Linux KVM host

[John Nielsen]

On Jun 22, 2015, at 3:48 PM, Bandan Das <bsd@redhat.com> wrote:

> John Nielsen <lists@jnielsen.net> writes:
> 
>> On Jun 17, 2014, at 10:48 AM, John Nielsen <lists@jnielsen.net> wrote:
>> 
>>> On Jun 17, 2014, at 12:05 AM, Gleb Natapov <gleb@kernel.org> wrote:
>>> 
>>>> On Tue, Jun 17, 2014 at 06:21:23AM +0200, Paolo Bonzini wrote:
>>>>> Il 16/06/2014 18:47, John Nielsen ha scritto:
>>>>>> On Jun 16, 2014, at 10:39 AM, Paolo Bonzini <pbonzini@redhat.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> Il 16/06/2014 18:09, John Nielsen ha scritto:
>>>>>>>>>> The only substantial difference on the hardware side is the
>>>>>>>>>> CPU.  The hosts where the problem occurs use "Intel(R)
>>>>>>>>>> Xeon(R) CPU E5-2650 v2 @ 2.60GHz", while the hosts that don't
>>>>>>>>>> show the problem use the prior revision, "Intel(R) Xeon(R)
>>>>>>>>>> CPU E5-2650 0 @ 2.00GHz".
>>>>>>> Can you do "grep . /sys/module/kvm_intel/parameters/*" on both
>>>>>>> hosts please?
>>>>>> No differences that I can see. Output below.
>>>>> Not really:
>>>>> 
>>>>>> Working host: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz # grep
>>>>>> . /sys/module/kvm_intel/parameters/*
>>>>>> /sys/module/kvm_intel/parameters/enable_apicv:N
>>>>>> 
>>>>>> Problem host: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz # grep
>>>>>> . /sys/module/kvm_intel/parameters/*
>>>>>> /sys/module/kvm_intel/parameters/enable_apicv:Y
>>>>> So we have a clue.  Let me study the code more, I'll try to get
>>>>> back with a suggestion.
>>> Wow, can't believe I missed that. Good catch!
>>> 
>>>> Does disabling apicv on E5-2650 v2 make reboot problem go away?
>>> Yes it does!
>>> 
>>> # modprobe kvm_intel /sys/module/kvm_intel/parameters/enable_apicv:Y
>>> # /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m
>>> 512 -smp 2,sockets=1,cores=1,threads=2 -drive
>>> file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2
>>> -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net
>>> none
>>> 
>>> [problem occurs]
>>> 
>>> # rmmod kvm_intel # modprobe kvm_intel enable_apicv=N
>>> /sys/module/kvm_intel/parameters/enable_apicv:N #
>>> /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512
>>> -smp 2,sockets=1,cores=1,threads=2 -drive
>>> file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2
>>> -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net
>>> none
>>> 
>>> [problem does not occur]
>>> 
>>> Thank you. This both narrows the problem considerably and provides
>>> an acceptable workaround. It would still be nice to see it fixed, of
>>> course. Keep me CC'ed as I'm not on the KVM list.
>> 
>> I’m resurrecting an old thread since I haven’t heard anything in a
>> while. Has anyone looked in to the KVM+apicv bug documented above as
>> well as here:
>> 
>> https://bugs.launchpad.net/qemu/+bug/1329956 ?
>> 
>> If appropriate, where should I go to file a KVM bug (since this isn’t
>> really Qemu’s problem)?
> 
> Hi John, does this happen with the latest upstream kernel version ?

I know for sure it happens with 4.0.4 and I’m not aware of any newer changes that would affect it.--
To unsubscribe from this list: send the line "unsubscribe kvm" in

Re: Hang on reboot in FreeBSD guest on Linux KVM host

[Paolo Bonzini]

On 23/06/2015 00:08, John Nielsen wrote:
>>>> Thank you. This both narrows the problem considerably and provides
>>>> an acceptable workaround. It would still be nice to see it fixed, of
>>>> course. Keep me CC'ed as I'm not on the KVM list.
>>>
>>> I’m resurrecting an old thread since I haven’t heard anything in a
>>> while. Has anyone looked in to the KVM+apicv bug documented above as
>>> well as here:
>>>
>>> https://bugs.launchpad.net/qemu/+bug/1329956 ?
>>>
>>> If appropriate, where should I go to file a KVM bug (since this isn’t
>>> really Qemu’s problem)?
>>
>> Hi John, does this happen with the latest upstream kernel version ?
> 
> I know for sure it happens with 4.0.4 and I’m not aware of any newer changes that would affect it.--
> To unsubscribe from this list: send the line "unsubscribe kvm" in

Hi John, I now have APICv hardware much more easily available (read: at
home) so I can look at this again.

Paolo

Re: Hang on reboot in FreeBSD guest on Linux KVM host

[Paolo Bonzini]

On 23/06/2015 00:08, John Nielsen wrote:
>>> I’m resurrecting an old thread since I haven’t heard anything in a
>>> while. Has anyone looked in to the KVM+apicv bug documented above as
>>> well as here:
>>>
>>> https://bugs.launchpad.net/qemu/+bug/1329956 ?
>>>
>>> If appropriate, where should I go to file a KVM bug (since this isn’t
>>> really Qemu’s problem)?
>>
>> Hi John, does this happen with the latest upstream kernel version ?
> 
> I know for sure it happens with 4.0.4 and I’m not aware of any newer changes that would affect it.--
> To unsubscribe from this list: send the line "unsubscribe kvm" in

Can you reproduce it with 10.1?

I did this:

1) download
http://download.pcbsd.org/iso/10.1-RELEASE/amd64/PCBSD10.1.2-x64-trueos-server.raw.xz
and unpack it

2) run it with "qemu-kvm -drive
if=virtio,PCBSD10.1.2-x64-trueos-server.raw -smp 2"

3) login as root/pcbsd, type "reboot"

I would like to know if I'm doing anything wrong.  My machine is a Xeon
E5 v3 (Haswell).  My SeaBIOS build doesn't have the atkbd0 bug, but just
to rule that out, can you send me your Seabios binary
(/usr/share/qemu/bios*.bin) as well?

Paolo

Re: Hang on reboot in FreeBSD guest on Linux KVM host

[Bandan Das]

Paolo Bonzini <pbonzini@redhat.com> writes:
...
>
> I did this:
>
> 1) download
> http://download.pcbsd.org/iso/10.1-RELEASE/amd64/PCBSD10.1.2-x64-trueos-server.raw.xz
> and unpack it
>
> 2) run it with "qemu-kvm -drive
> if=virtio,PCBSD10.1.2-x64-trueos-server.raw -smp 2"
>
> 3) login as root/pcbsd, type "reboot"
>
> I would like to know if I'm doing anything wrong.  My machine is a Xeon
> E5 v3 (Haswell).  My SeaBIOS build doesn't have the atkbd0 bug, but just

I thought (from the bug report) that the bug is only reproducible
on v2 Xeons ?

> to rule that out, can you send me your Seabios binary
> (/usr/share/qemu/bios*.bin) as well?
>
> Paolo
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Hang on reboot in FreeBSD guest on Linux KVM host

[Paolo Bonzini]

On 24/06/2015 17:57, Bandan Das wrote:
> Paolo Bonzini <pbonzini@redhat.com> writes:
> ...
>>
>> I did this:
>>
>> 1) download
>> http://download.pcbsd.org/iso/10.1-RELEASE/amd64/PCBSD10.1.2-x64-trueos-server.raw.xz
>> and unpack it
>>
>> 2) run it with "qemu-kvm -drive
>> if=virtio,PCBSD10.1.2-x64-trueos-server.raw -smp 2"
>>
>> 3) login as root/pcbsd, type "reboot"
>>
>> I would like to know if I'm doing anything wrong.  My machine is a Xeon
>> E5 v3 (Haswell).  My SeaBIOS build doesn't have the atkbd0 bug, but just
> 
> I thought (from the bug report) that the bug is only reproducible
> on v2 Xeons ?

It is related to APICv.  I think when the thread died out, v3 Xeons
didn't exist yet.

Paolo

>> to rule that out, can you send me your Seabios binary
>> (/usr/share/qemu/bios*.bin) as well?
>>
>> Paolo
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Hang on reboot in FreeBSD guest on Linux KVM host

[John Nielsen]

On Jun 24, 2015, at 9:50 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 23/06/2015 00:08, John Nielsen wrote:
>>>> I’m resurrecting an old thread since I haven’t heard anything in a
>>>> while. Has anyone looked in to the KVM+apicv bug documented above as
>>>> well as here:
>>>> 
>>>> https://bugs.launchpad.net/qemu/+bug/1329956 ?
>>>> 
>>>> If appropriate, where should I go to file a KVM bug (since this isn’t
>>>> really Qemu’s problem)?
>>> 
>>> Hi John, does this happen with the latest upstream kernel version ?
>> 
>> I know for sure it happens with 4.0.4 and I’m not aware of any newer changes that would affect it.--
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
> 
> Can you reproduce it with 10.1?
> 
> I did this:
> 
> 1) download
> http://download.pcbsd.org/iso/10.1-RELEASE/amd64/PCBSD10.1.2-x64-trueos-server.raw.xz
> and unpack it
> 
> 2) run it with "qemu-kvm -drive
> if=virtio,PCBSD10.1.2-x64-trueos-server.raw -smp 2"
> 
> 3) login as root/pcbsd, type "reboot"
> 
> I would like to know if I'm doing anything wrong.  My machine is a Xeon
> E5 v3 (Haswell).  My SeaBIOS build doesn't have the atkbd0 bug, but just
> to rule that out, can you send me your Seabios binary
> (/usr/share/qemu/bios*.bin) as well?

Interesting. Using the same PC-BSD image I am able to reproduce on a server running slightly older software but I can not reproduce running bleeding edge. I verified enable_apicv=Y on both. In both cases I ran
qemu-kvm -drive if=virtio,file=PCBSD10.1.2-x64-trueos-server.raw -smp 2 -vnc 0.0.0.0:0

Specifically:

Breaks (VM hangs during boot after pressing ctrl-alt-del):
kernel 3.12.22
qemu-kvm-1.7.0-3.el6.x86_64
seabios-1.7.3.1-1.el6.noarch
Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz

Works (VM reboots normally):
kernel 4.0.4
qemu-kvm-2.3.0-6.el7.centos.x86_64
seabios-bin-1.8.1-1.el7.centos.noarch
Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz


Unfortunately I no longer have the test environment I used a few days ago to reproduce this issue so I can’t verify the software versions that were in use. It’s possible I was mistaken about the kernel version (I thought it was 4.0.4). Perhaps it really is fixed in the newer kernel? In any case, this is great news! I would be interested in identifying the patch(es) that fixed the issue to make back-porting them easier, but I won’t have time to do a binary search anytime soon.

Thanks for looking in to this again. If anyone else is interested in identifying what specifically fixed the issue please let me know if I can do anything to help.

JN

Re: Hang on reboot in FreeBSD guest on Linux KVM host

[Paolo Bonzini]

On 24/06/2015 19:07, John Nielsen wrote:
> 
> Unfortunately I no longer have the test environment I used a few days
> ago to reproduce this issue so I can’t verify the software versions
> that were in use. It’s possible I was mistaken about the kernel
> version (I thought it was 4.0.4). Perhaps it really is fixed in the
> newer kernel? In any case, this is great news! I would be interested
> in identifying the patch(es) that fixed the issue to make
> back-porting them easier, but I won’t have time to do a binary search
> anytime soon.

Random guess:

commit a323b409820c8afd33fbd841f5534eb84b406e8d
Author: Radim Krčmář <rkrcmar@redhat.com>
Date:   Thu Oct 30 15:06:46 2014 +0100

    KVM: x86: detect LVTT changes under APICv

commit e462755cae2bf8297a663278935ad4d59812d2f3
Author: Radim Krčmář <rkrcmar@redhat.com>
Date:   Thu Oct 30 15:06:45 2014 +0100

    KVM: x86: detect SPIV changes under APICv

Paolo

Re: Hang on reboot in FreeBSD guest on Linux KVM host

[Jack Wang]

From: Wincy Van <fanwenyi0529@gmail.com>

On Thu, Jun 25, 2015 at 1:07 AM, John Nielsen <lists@jnielsen.net> wrot=
e:
> Interesting. Using the same PC-BSD image I am able to reproduce on a =
server running slightly older software but I can not reproduce running =
bleeding edge. I verified enable_apicv=3DY on both. In both cases I ran
> qemu-kvm -drive if=3Dvirtio,file=3DPCBSD10.1.2-x64-trueos-server.raw =
-smp 2 -vnc 0.0.0.0:0
>
> Specifically:
>
> Breaks (VM hangs during boot after pressing ctrl-alt-del):
> kernel 3.12.22
> qemu-kvm-1.7.0-3.el6.x86_64
> seabios-1.7.3.1-1.el6.noarch
> Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
>
> Works (VM reboots normally):
> kernel 4.0.4
> qemu-kvm-2.3.0-6.el7.centos.x86_64
> seabios-bin-1.8.1-1.el7.centos.noarch
> Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
>
>
> Unfortunately I no longer have the test environment I used a few days=
 ago to reproduce this issue so I can=E2=80=99t verify the software ver=
sions that were in use. It=E2=80=99s possible I was mistaken about the =
kernel version (I thought it was 4.0.4). Perhaps it really is fixed in =
the newer kernel? In any case, this is great news! I would be intereste=
d in identifying the patch(es) that fixed the issue to make back-portin=
g them easier, but I won=E2=80=99t have time to do a binary search anyt=
ime soon.
>
> Thanks for looking in to this again. If anyone else is interested in =
identifying what specifically fixed the issue please let me know if I c=
an do anything to help.
>

John,

This commit may work for you.

commit 4114c27d450bef228be9c7b0c40a888e18a3a636
Author: Wei Wang <wei.w.wang@intel.com>
Date:   Wed Nov 5 10:53:43 2014 +0800

    KVM: x86: reset RVI upon system reset


Thanks
Wincy

Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.

---

### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**

**Summary:**
Multi-cores FreeBSD guests hang during reboot under KVM on systems with Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The same configuration works correctly on Ice Lake (Xeon Gold 6338).

---

#### **Environment**

**Host:**

* OS: Debian 12 (Bookworm)
* Kernel versions tested: 6.1.118 and 6.12.47 (both affected)
* QEMU versions tested: 8.2.10 and 9.2.4
* Firmware: SeaBIOS 1.16.2-1 and OVMF 2024.11.1
* CPU: Intel(R) Xeon(R) 6710E (SierraForest)
* KVM module: `kvm_intel`
* Command-line (simplified example):

  ```
  qemu-system-x86_64 -m 2048 -enable-kvm -cpu host -smp 4 -hda freebsd14.img
  ```

**Guest:**

* OS: FreeBSD 14.3-RELEASE
* SMP guests multi-cores

---

#### **Steps to Reproduce**

1. Start a FreeBSD 14.3 guest under KVM on a SierraForest host.
2. Log in and run:

   ```
   # reboot
   ```

   or press `Ctrl+Alt+Del`.
3. Observe that the VM hangs during reboot — it never returns to BIOS/UEFI.

---

#### **Expected Result**

FreeBSD should reboot cleanly and return to login prompt.

---

#### **Actual Result**

Guest hangs indefinitely during reboot.
QEMU remains running but guest output is frozen.
No host kernel warnings or errors in `dmesg`.

---

#### **Workaround**

Disable APICv on the host before starting QEMU:

```
modprobe kvm_intel enable_apicv=N
```

With `enable_apicv=N`, the FreeBSD guest reboots normally every time.

---

#### **Additional Information**

* The issue **does not reproduce** on an Ice Lake host (Intel Xeon Gold 6338) with identical kernel, QEMU, and guest image.
* Collected `trace-cmd` data for `kvm*` events during both:

  * **Good reboot** (with `enable_apicv=N`)
  * **Bad reboot** (default APICv enabled)

Traces are available for analysis upon request.

---

#### **Preliminary Analysis**

Disabling APICv avoids the hang, suggesting the problem is related to APIC virtualization on SierraForest.
Possible causes:

* Regression in APICv or posted-interrupt handling on new Xeon platforms.
* Microcode or MSR-related difference in APICv behavior on SierraForest.
* Incorrect EOI or interrupt delivery sequence during guest reboot.

---

#### **Reproducibility**

* 100% reproducible on SierraForest systems
* Not reproducible on Ice Lake

---

#### **Request for Guidance**

* Is this a known KVM or Intel virtualization issue on SierraForest?
* Would you like `trace-cmd` traces, MSR dumps, or `perf kvm` data for further diagnosis?
* Is there a kernel parameter or patch to selectively disable APICv on affected models?

Re: Hang on reboot in FreeBSD guest on Linux KVM host

[Jack Wang]

From: Wincy Van <fanwenyi0529@gmail.com>

On Thu, Jun 25, 2015 at 1:07 AM, John Nielsen <lists@jnielsen.net> wrot=
e:
> Interesting. Using the same PC-BSD image I am able to reproduce on a =
server running slightly older software but I can not reproduce running =
bleeding edge. I verified enable_apicv=3DY on both. In both cases I ran
> qemu-kvm -drive if=3Dvirtio,file=3DPCBSD10.1.2-x64-trueos-server.raw =
-smp 2 -vnc 0.0.0.0:0
>
> Specifically:
>
> Breaks (VM hangs during boot after pressing ctrl-alt-del):
> kernel 3.12.22
> qemu-kvm-1.7.0-3.el6.x86_64
> seabios-1.7.3.1-1.el6.noarch
> Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
>
> Works (VM reboots normally):
> kernel 4.0.4
> qemu-kvm-2.3.0-6.el7.centos.x86_64
> seabios-bin-1.8.1-1.el7.centos.noarch
> Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
>
>
> Unfortunately I no longer have the test environment I used a few days=
 ago to reproduce this issue so I can=E2=80=99t verify the software ver=
sions that were in use. It=E2=80=99s possible I was mistaken about the =
kernel version (I thought it was 4.0.4). Perhaps it really is fixed in =
the newer kernel? In any case, this is great news! I would be intereste=
d in identifying the patch(es) that fixed the issue to make back-portin=
g them easier, but I won=E2=80=99t have time to do a binary search anyt=
ime soon.
>
> Thanks for looking in to this again. If anyone else is interested in =
identifying what specifically fixed the issue please let me know if I c=
an do anything to help.
>

John,

This commit may work for you.

commit 4114c27d450bef228be9c7b0c40a888e18a3a636
Author: Wei Wang <wei.w.wang@intel.com>
Date:   Wed Nov 5 10:53:43 2014 +0800

    KVM: x86: reset RVI upon system reset


Thanks
Wincy

Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.

---

### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**

**Summary:**
Multi-cores FreeBSD guests hang during reboot under KVM on systems with Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The same configuration works correctly on Ice Lake (Xeon Gold 6338).

---

#### **Environment**

**Host:**

* OS: Debian 12 (Bookworm)
* Kernel versions tested: 6.1.118 and 6.12.47 (both affected)
* QEMU versions tested: 8.2.10 and 9.2.4
* Firmware: SeaBIOS 1.16.2-1 and OVMF 2024.11.1
* CPU: Intel(R) Xeon(R) 6710E (SierraForest)
* KVM module: `kvm_intel`
* Command-line (simplified example):

  ```
  qemu-system-x86_64 -m 2048 -enable-kvm -cpu host -smp 4 -hda freebsd14.img
  ```

**Guest:**

* OS: FreeBSD 14.3-RELEASE
* SMP guests multi-cores

---

#### **Steps to Reproduce**

1. Start a FreeBSD 14.3 guest under KVM on a SierraForest host.
2. Log in and run:

   ```
   # reboot
   ```

   or press `Ctrl+Alt+Del`.
3. Observe that the VM hangs during reboot — it never returns to BIOS/UEFI.

---

#### **Expected Result**

FreeBSD should reboot cleanly and return to login prompt.

---

#### **Actual Result**

Guest hangs indefinitely during reboot.
QEMU remains running but guest output is frozen.
No host kernel warnings or errors in `dmesg`.

---

#### **Workaround**

Disable APICv on the host before starting QEMU:

```
modprobe kvm_intel enable_apicv=N
```

With `enable_apicv=N`, the FreeBSD guest reboots normally every time.

---

#### **Additional Information**

* The issue **does not reproduce** on an Ice Lake host (Intel Xeon Gold 6338) with identical kernel, QEMU, and guest image.
* Collected `trace-cmd` data for `kvm*` events during both:

  * **Good reboot** (with `enable_apicv=N`)
  * **Bad reboot** (default APICv enabled)

Traces are available for analysis upon request.

---

#### **Preliminary Analysis**

Disabling APICv avoids the hang, suggesting the problem is related to APIC virtualization on SierraForest.
Possible causes:

* Regression in APICv or posted-interrupt handling on new Xeon platforms.
* Microcode or MSR-related difference in APICv behavior on SierraForest.
* Incorrect EOI or interrupt delivery sequence during guest reboot.

---

#### **Reproducibility**

* 100% reproducible on SierraForest systems
* Not reproducible on Ice Lake

---

#### **Request for Guidance**

* Is this a known KVM or Intel virtualization issue on SierraForest?
* Would you like `trace-cmd` traces, MSR dumps, or `perf kvm` data for further diagnosis?
* Is there a kernel parameter or patch to selectively disable APICv on affected models?

Re: Hang on reboot in FreeBSD guest on Linux KVM host

[Jack Wang]

From: Wincy Van <fanwenyi0529@gmail.com>

On Thu, Jun 25, 2015 at 1:07 AM, John Nielsen <lists@jnielsen.net> wrot=
e:
> Interesting. Using the same PC-BSD image I am able to reproduce on a =
server running slightly older software but I can not reproduce running =
bleeding edge. I verified enable_apicv=3DY on both. In both cases I ran
> qemu-kvm -drive if=3Dvirtio,file=3DPCBSD10.1.2-x64-trueos-server.raw =
-smp 2 -vnc 0.0.0.0:0
>
> Specifically:
>
> Breaks (VM hangs during boot after pressing ctrl-alt-del):
> kernel 3.12.22
> qemu-kvm-1.7.0-3.el6.x86_64
> seabios-1.7.3.1-1.el6.noarch
> Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
>
> Works (VM reboots normally):
> kernel 4.0.4
> qemu-kvm-2.3.0-6.el7.centos.x86_64
> seabios-bin-1.8.1-1.el7.centos.noarch
> Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
>
>
> Unfortunately I no longer have the test environment I used a few days=
 ago to reproduce this issue so I can=E2=80=99t verify the software ver=
sions that were in use. It=E2=80=99s possible I was mistaken about the =
kernel version (I thought it was 4.0.4). Perhaps it really is fixed in =
the newer kernel? In any case, this is great news! I would be intereste=
d in identifying the patch(es) that fixed the issue to make back-portin=
g them easier, but I won=E2=80=99t have time to do a binary search anyt=
ime soon.
>
> Thanks for looking in to this again. If anyone else is interested in =
identifying what specifically fixed the issue please let me know if I c=
an do anything to help.
>

John,

This commit may work for you.

commit 4114c27d450bef228be9c7b0c40a888e18a3a636
Author: Wei Wang <wei.w.wang@intel.com>
Date:   Wed Nov 5 10:53:43 2014 +0800

    KVM: x86: reset RVI upon system reset


Thanks
Wincy

Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.

---

### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**

**Summary:**
Multi-cores FreeBSD guests hang during reboot under KVM on systems with Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The same configuration works correctly on Ice Lake (Xeon Gold 6338).

---

#### **Environment**

**Host:**

* OS: Debian 12 (Bookworm)
* Kernel versions tested: 6.1.118 and 6.12.47 (both affected)
* QEMU versions tested: 8.2.10 and 9.2.4
* Firmware: SeaBIOS 1.16.2-1 and OVMF 2024.11.1
* CPU: Intel(R) Xeon(R) 6710E (SierraForest)
* KVM module: `kvm_intel`
* Command-line (simplified example):

  ```
  qemu-system-x86_64 -m 2048 -enable-kvm -cpu host -smp 4 -hda freebsd14.img
  ```

**Guest:**

* OS: FreeBSD 14.3-RELEASE
* SMP guests multi-cores

---

#### **Steps to Reproduce**

1. Start a FreeBSD 14.3 guest under KVM on a SierraForest host.
2. Log in and run:

   ```
   # reboot
   ```

   or press `Ctrl+Alt+Del`.
3. Observe that the VM hangs during reboot — it never returns to BIOS/UEFI.

---

#### **Expected Result**

FreeBSD should reboot cleanly and return to login prompt.

---

#### **Actual Result**

Guest hangs indefinitely during reboot.
QEMU remains running but guest output is frozen.
No host kernel warnings or errors in `dmesg`.

---

#### **Workaround**

Disable APICv on the host before starting QEMU:

```
modprobe kvm_intel enable_apicv=N
```

With `enable_apicv=N`, the FreeBSD guest reboots normally every time.

---

#### **Additional Information**

* The issue **does not reproduce** on an Ice Lake host (Intel Xeon Gold 6338) with identical kernel, QEMU, and guest image.
* Collected `trace-cmd` data for `kvm*` events during both:

  * **Good reboot** (with `enable_apicv=N`)
  * **Bad reboot** (default APICv enabled)

Traces are available for analysis upon request.

---

#### **Preliminary Analysis**

Disabling APICv avoids the hang, suggesting the problem is related to APIC virtualization on SierraForest.
Possible causes:

* Regression in APICv or posted-interrupt handling on new Xeon platforms.
* Microcode or MSR-related difference in APICv behavior on SierraForest.
* Incorrect EOI or interrupt delivery sequence during guest reboot.

---

#### **Reproducibility**

* 100% reproducible on SierraForest systems
* Not reproducible on Ice Lake

---

#### **Request for Guidance**

* Is this a known KVM or Intel virtualization issue on SierraForest?
* Would you like `trace-cmd` traces, MSR dumps, or `perf kvm` data for further diagnosis?
* Is there a kernel parameter or patch to selectively disable APICv on affected models?

Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU

[Jinpu Wang]

[-- Attachment #1: Type: text/plain, Size: 5273 bytes --]

On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote:
>
> From: Wincy Van <fanwenyi0529@gmail.com>
>
> On Thu, Jun 25, 2015 at 1:07 AM, John Nielsen <lists@jnielsen.net> wrot=
> e:
> > Interesting. Using the same PC-BSD image I am able to reproduce on a =
> server running slightly older software but I can not reproduce running =
> bleeding edge. I verified enable_apicv=3DY on both. In both cases I ran
> > qemu-kvm -drive if=3Dvirtio,file=3DPCBSD10.1.2-x64-trueos-server.raw =
> -smp 2 -vnc 0.0.0.0:0
> >
> > Specifically:
> >
> > Breaks (VM hangs during boot after pressing ctrl-alt-del):
> > kernel 3.12.22
> > qemu-kvm-1.7.0-3.el6.x86_64
> > seabios-1.7.3.1-1.el6.noarch
> > Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
> >
> > Works (VM reboots normally):
> > kernel 4.0.4
> > qemu-kvm-2.3.0-6.el7.centos.x86_64
> > seabios-bin-1.8.1-1.el7.centos.noarch
> > Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
> >
> >
> > Unfortunately I no longer have the test environment I used a few days=
>  ago to reproduce this issue so I can=E2=80=99t verify the software ver=
> sions that were in use. It=E2=80=99s possible I was mistaken about the =
> kernel version (I thought it was 4.0.4). Perhaps it really is fixed in =
> the newer kernel? In any case, this is great news! I would be intereste=
> d in identifying the patch(es) that fixed the issue to make back-portin=
> g them easier, but I won=E2=80=99t have time to do a binary search anyt=
> ime soon.
> >
> > Thanks for looking in to this again. If anyone else is interested in =
> identifying what specifically fixed the issue please let me know if I c=
> an do anything to help.
> >
>
> John,
>
> This commit may work for you.
>
> commit 4114c27d450bef228be9c7b0c40a888e18a3a636
> Author: Wei Wang <wei.w.wang@intel.com>
> Date:   Wed Nov 5 10:53:43 2014 +0800
>
>     KVM: x86: reset RVI upon system reset
>
>
> Thanks
> Wincy
>
> Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
> machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
>
> ---
>
> ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
>
> **Summary:**
> Multi-cores FreeBSD guests hang during reboot under KVM on systems with Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The same configuration works correctly on Ice Lake (Xeon Gold 6338).
>
> ---
>
> #### **Environment**
>
> **Host:**
>
> * OS: Debian 12 (Bookworm)
> * Kernel versions tested: 6.1.118 and 6.12.47 (both affected)
> * QEMU versions tested: 8.2.10 and 9.2.4
> * Firmware: SeaBIOS 1.16.2-1 and OVMF 2024.11.1
> * CPU: Intel(R) Xeon(R) 6710E (SierraForest)
> * KVM module: `kvm_intel`
> * Command-line (simplified example):
>
>   ```
>   qemu-system-x86_64 -m 2048 -enable-kvm -cpu host -smp 4 -hda freebsd14.img
>   ```
>
> **Guest:**
>
> * OS: FreeBSD 14.3-RELEASE
> * SMP guests multi-cores
>
> ---
>
> #### **Steps to Reproduce**
>
> 1. Start a FreeBSD 14.3 guest under KVM on a SierraForest host.
> 2. Log in and run:
>
>    ```
>    # reboot
>    ```
>
>    or press `Ctrl+Alt+Del`.
> 3. Observe that the VM hangs during reboot — it never returns to BIOS/UEFI.
>
> ---
>
> #### **Expected Result**
>
> FreeBSD should reboot cleanly and return to login prompt.
>
> ---
>
> #### **Actual Result**
>
> Guest hangs indefinitely during reboot.
> QEMU remains running but guest output is frozen.
> No host kernel warnings or errors in `dmesg`.
>
> ---
>
> #### **Workaround**
>
> Disable APICv on the host before starting QEMU:
>
> ```
> modprobe kvm_intel enable_apicv=N
> ```
>
> With `enable_apicv=N`, the FreeBSD guest reboots normally every time.
>
> ---
>
> #### **Additional Information**
>
> * The issue **does not reproduce** on an Ice Lake host (Intel Xeon Gold 6338) with identical kernel, QEMU, and guest image.
> * Collected `trace-cmd` data for `kvm*` events during both:
>
>   * **Good reboot** (with `enable_apicv=N`)
>   * **Bad reboot** (default APICv enabled)
>
> Traces are available for analysis upon request.
>
> ---
>
> #### **Preliminary Analysis**
>
> Disabling APICv avoids the hang, suggesting the problem is related to APIC virtualization on SierraForest.
> Possible causes:
>
> * Regression in APICv or posted-interrupt handling on new Xeon platforms.
> * Microcode or MSR-related difference in APICv behavior on SierraForest.
> * Incorrect EOI or interrupt delivery sequence during guest reboot.
>
> ---
>
> #### **Reproducibility**
>
> * 100% reproducible on SierraForest systems
> * Not reproducible on Ice Lake
>
> ---
>
> #### **Request for Guidance**
>
> * Is this a known KVM or Intel virtualization issue on SierraForest?
> * Would you like `trace-cmd` traces, MSR dumps, or `perf kvm` data for further diagnosis?
> * Is there a kernel parameter or patch to selectively disable APICv on affected models?


I've attached report from "trace-cmd trace-cmd record -e kvm -e
kvm_mmio -e kvm_userspace_exit"  in good boot and bad reboot cases,
maybe folks could find something interesting.

[-- Attachment #2: SRF_bug.tar.xz --]
[-- Type: application/x-xz, Size: 3784793 bytes --]

Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU

[Sean Christopherson]

Trimmed Cc: to drop people from the original thread.  In the future, just start
a new bug report.  Piggybacking a 10 year old bug just because the symptoms are
similar does more harm than good.  Whatever the old thread was chasing was already
fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed
the problem.  I.e. whatever they were chasing _can't_ be the same root cause,
because even if it's literally the same code bug, it would require a code change
and thus a regression between v4.0 and v6.1.

On Wed, Oct 08, 2025, Jinpu Wang wrote:
> On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote:
> > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
> > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.

Are there any host kernels that _do_ work?  E.g. have you tried a bleeding edge
host kernel?

> > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
> >
> > **Summary:**
> > Multi-cores FreeBSD guests hang during reboot under KVM on systems with
> > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with
> > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The
> > same configuration works correctly on Ice Lake (Xeon Gold 6338).

Does Sierra Forest have IPI virtualization?  If so, you could try running with
APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization.

Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU

[Jinpu Wang]

Hi Sean,

On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@google.com> wrote:
>
> Trimmed Cc: to drop people from the original thread.  In the future, just start
> a new bug report.  Piggybacking a 10 year old bug just because the symptoms are
> similar does more harm than good.  Whatever the old thread was chasing was already
> fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed
> the problem.  I.e. whatever they were chasing _can't_ be the same root cause,
> because even if it's literally the same code bug, it would require a code change
> and thus a regression between v4.0 and v6.1.
Thx for the reply,  it makes sense. I will remember this next time.
>
> On Wed, Oct 08, 2025, Jinpu Wang wrote:
> > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote:
> > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
> > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
>
> Are there any host kernels that _do_ work?  E.g. have you tried a bleeding edge
> host kernel?
I will try linus/master today.
>
> > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
> > >
> > > **Summary:**
> > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with
> > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with
> > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The
> > > same configuration works correctly on Ice Lake (Xeon Gold 6338).
>
> Does Sierra Forest have IPI virtualization?  If so, you could try running with
> APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization.
Yes, it does:
$  grep . /sys/module/kvm_intel/parameters/*
/sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N
/sys/module/kvm_intel/parameters/dump_invalid_vmcs:N
/sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
/sys/module/kvm_intel/parameters/enable_apicv:Y
/sys/module/kvm_intel/parameters/enable_ipiv:Y
/sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y
/sys/module/kvm_intel/parameters/ept:Y
/sys/module/kvm_intel/parameters/eptad:Y
/sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y
/sys/module/kvm_intel/parameters/fasteoi:Y
/sys/module/kvm_intel/parameters/flexpriority:Y
/sys/module/kvm_intel/parameters/nested:Y
/sys/module/kvm_intel/parameters/nested_early_check:N
/sys/module/kvm_intel/parameters/ple_gap:128
/sys/module/kvm_intel/parameters/ple_window:4096
/sys/module/kvm_intel/parameters/ple_window_grow:2
/sys/module/kvm_intel/parameters/ple_window_max:4294967295
/sys/module/kvm_intel/parameters/ple_window_shrink:0
/sys/module/kvm_intel/parameters/pml:Y
/sys/module/kvm_intel/parameters/preemption_timer:Y
/sys/module/kvm_intel/parameters/sgx:N
/sys/module/kvm_intel/parameters/unrestricted_guest:Y
/sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required
/sys/module/kvm_intel/parameters/vnmi:Y
/sys/module/kvm_intel/parameters/vpid:Y

I tried to disable ipiv, but it doesn't help. freebsd hang on reboot.
sudo modprobe -r kvm_intel
sudo modprobe  kvm_intel enable_ipiv=N
/sys/module/kvm_intel/parameters/enable_ipiv:N

Thx!

Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU

[Jinpu Wang]

On Thu, Oct 9, 2025 at 5:44 AM Jinpu Wang <jinpu.wang@ionos.com> wrote:
>
> Hi Sean,
>
> On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@google.com> wrote:
> >
> > Trimmed Cc: to drop people from the original thread.  In the future, just start
> > a new bug report.  Piggybacking a 10 year old bug just because the symptoms are
> > similar does more harm than good.  Whatever the old thread was chasing was already
> > fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed
> > the problem.  I.e. whatever they were chasing _can't_ be the same root cause,
> > because even if it's literally the same code bug, it would require a code change
> > and thus a regression between v4.0 and v6.1.
> Thx for the reply,  it makes sense. I will remember this next time.
> >
> > On Wed, Oct 08, 2025, Jinpu Wang wrote:
> > > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote:
> > > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
> > > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
> >
> > Are there any host kernels that _do_ work?  E.g. have you tried a bleeding edge
> > host kernel?
> I will try linus/master today.
> >
> > > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
> > > >
> > > > **Summary:**
> > > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with
> > > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with
> > > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The
> > > > same configuration works correctly on Ice Lake (Xeon Gold 6338).
> >
> > Does Sierra Forest have IPI virtualization?  If so, you could try running with
> > APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization.
> Yes, it does:
> $  grep . /sys/module/kvm_intel/parameters/*
> /sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N
> /sys/module/kvm_intel/parameters/dump_invalid_vmcs:N
> /sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
> /sys/module/kvm_intel/parameters/enable_apicv:Y
> /sys/module/kvm_intel/parameters/enable_ipiv:Y
> /sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y
> /sys/module/kvm_intel/parameters/ept:Y
> /sys/module/kvm_intel/parameters/eptad:Y
> /sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y
> /sys/module/kvm_intel/parameters/fasteoi:Y
> /sys/module/kvm_intel/parameters/flexpriority:Y
> /sys/module/kvm_intel/parameters/nested:Y
> /sys/module/kvm_intel/parameters/nested_early_check:N
> /sys/module/kvm_intel/parameters/ple_gap:128
> /sys/module/kvm_intel/parameters/ple_window:4096
> /sys/module/kvm_intel/parameters/ple_window_grow:2
> /sys/module/kvm_intel/parameters/ple_window_max:4294967295
> /sys/module/kvm_intel/parameters/ple_window_shrink:0
> /sys/module/kvm_intel/parameters/pml:Y
> /sys/module/kvm_intel/parameters/preemption_timer:Y
> /sys/module/kvm_intel/parameters/sgx:N
> /sys/module/kvm_intel/parameters/unrestricted_guest:Y
> /sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required
> /sys/module/kvm_intel/parameters/vnmi:Y
> /sys/module/kvm_intel/parameters/vpid:Y
>
> I tried to disable ipiv, but it doesn't help. freebsd hang on reboot.
> sudo modprobe -r kvm_intel
> sudo modprobe  kvm_intel enable_ipiv=N
> /sys/module/kvm_intel/parameters/enable_ipiv:N
>
> Thx!
+cc Vitaly
Sorry, I missed one detail, we are use hyper-V enlightment features:
"+hv-relaxed,+hv-vapic,+hv-time,+hv-runtime,hv-spinlocks=0x1fff,+hv-vpindex,+hv-synic,+hv-stimer,+hv-tlbflush,hv-ipi."

did a lot tests with different features, and looks the hang is related
to  +hv-synic,+hv-stimer.  hv-synic seems the key which causes boot
hang of Freebsd 14.

But the problem seems fixed with FreeBSD 15?  I guess it's this fix:
https://github.com/systemd/systemd/issues/28001

Seems it's a bug from freebsd side, rather than on kvm side to me, but
I'm puzzled by disable apicv helps?

Thx!

Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU

[Jinpu Wang]

On Thu, Oct 9, 2025 at 1:21 PM Jinpu Wang <jinpu.wang@ionos.com> wrote:
>
> On Thu, Oct 9, 2025 at 5:44 AM Jinpu Wang <jinpu.wang@ionos.com> wrote:
> >
> > Hi Sean,
> >
> > On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@google.com> wrote:
> > >
> > > Trimmed Cc: to drop people from the original thread.  In the future, just start
> > > a new bug report.  Piggybacking a 10 year old bug just because the symptoms are
> > > similar does more harm than good.  Whatever the old thread was chasing was already
> > > fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed
> > > the problem.  I.e. whatever they were chasing _can't_ be the same root cause,
> > > because even if it's literally the same code bug, it would require a code change
> > > and thus a regression between v4.0 and v6.1.
> > Thx for the reply,  it makes sense. I will remember this next time.
> > >
> > > On Wed, Oct 08, 2025, Jinpu Wang wrote:
> > > > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote:
> > > > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
> > > > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
> > >
> > > Are there any host kernels that _do_ work?  E.g. have you tried a bleeding edge
> > > host kernel?
> > I will try linus/master today.
> > >
> > > > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
> > > > >
> > > > > **Summary:**
> > > > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with
> > > > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with
> > > > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The
> > > > > same configuration works correctly on Ice Lake (Xeon Gold 6338).
> > >
> > > Does Sierra Forest have IPI virtualization?  If so, you could try running with
> > > APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization.
> > Yes, it does:
> > $  grep . /sys/module/kvm_intel/parameters/*
> > /sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N
> > /sys/module/kvm_intel/parameters/dump_invalid_vmcs:N
> > /sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
> > /sys/module/kvm_intel/parameters/enable_apicv:Y
> > /sys/module/kvm_intel/parameters/enable_ipiv:Y
> > /sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y
> > /sys/module/kvm_intel/parameters/ept:Y
> > /sys/module/kvm_intel/parameters/eptad:Y
> > /sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y
> > /sys/module/kvm_intel/parameters/fasteoi:Y
> > /sys/module/kvm_intel/parameters/flexpriority:Y
> > /sys/module/kvm_intel/parameters/nested:Y
> > /sys/module/kvm_intel/parameters/nested_early_check:N
> > /sys/module/kvm_intel/parameters/ple_gap:128
> > /sys/module/kvm_intel/parameters/ple_window:4096
> > /sys/module/kvm_intel/parameters/ple_window_grow:2
> > /sys/module/kvm_intel/parameters/ple_window_max:4294967295
> > /sys/module/kvm_intel/parameters/ple_window_shrink:0
> > /sys/module/kvm_intel/parameters/pml:Y
> > /sys/module/kvm_intel/parameters/preemption_timer:Y
> > /sys/module/kvm_intel/parameters/sgx:N
> > /sys/module/kvm_intel/parameters/unrestricted_guest:Y
> > /sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required
> > /sys/module/kvm_intel/parameters/vnmi:Y
> > /sys/module/kvm_intel/parameters/vpid:Y
> >
> > I tried to disable ipiv, but it doesn't help. freebsd hang on reboot.
> > sudo modprobe -r kvm_intel
> > sudo modprobe  kvm_intel enable_ipiv=N
> > /sys/module/kvm_intel/parameters/enable_ipiv:N
> >
> > Thx!
> +cc Vitaly
> Sorry, I missed one detail, we are use hyper-V enlightment features:
> "+hv-relaxed,+hv-vapic,+hv-time,+hv-runtime,hv-spinlocks=0x1fff,+hv-vpindex,+hv-synic,+hv-stimer,+hv-tlbflush,hv-ipi."
>
> did a lot tests with different features, and looks the hang is related
> to  +hv-synic,+hv-stimer.  hv-synic seems the key which causes boot
> hang of Freebsd 14.
>
> But the problem seems fixed with FreeBSD 15?  I guess it's this fix:
https://reviews.freebsd.org/D43508

>
> Seems it's a bug from freebsd side, rather than on kvm side to me, but
> I'm puzzled by disable apicv helps?
>
> Thx!

Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU

[Vitaly Kuznetsov]

Jinpu Wang <jinpu.wang@ionos.com> writes:

> On Thu, Oct 9, 2025 at 1:21 PM Jinpu Wang <jinpu.wang@ionos.com> wrote:
>>
>> On Thu, Oct 9, 2025 at 5:44 AM Jinpu Wang <jinpu.wang@ionos.com> wrote:
>> >
>> > Hi Sean,
>> >
>> > On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@google.com> wrote:
>> > >
>> > > Trimmed Cc: to drop people from the original thread.  In the future, just start
>> > > a new bug report.  Piggybacking a 10 year old bug just because the symptoms are
>> > > similar does more harm than good.  Whatever the old thread was chasing was already
>> > > fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed
>> > > the problem.  I.e. whatever they were chasing _can't_ be the same root cause,
>> > > because even if it's literally the same code bug, it would require a code change
>> > > and thus a regression between v4.0 and v6.1.
>> > Thx for the reply,  it makes sense. I will remember this next time.
>> > >
>> > > On Wed, Oct 08, 2025, Jinpu Wang wrote:
>> > > > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote:
>> > > > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
>> > > > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
>> > >
>> > > Are there any host kernels that _do_ work?  E.g. have you tried a bleeding edge
>> > > host kernel?
>> > I will try linus/master today.
>> > >
>> > > > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
>> > > > >
>> > > > > **Summary:**
>> > > > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with
>> > > > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with
>> > > > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The
>> > > > > same configuration works correctly on Ice Lake (Xeon Gold 6338).
>> > >
>> > > Does Sierra Forest have IPI virtualization?  If so, you could try running with
>> > > APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization.
>> > Yes, it does:
>> > $  grep . /sys/module/kvm_intel/parameters/*
>> > /sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N
>> > /sys/module/kvm_intel/parameters/dump_invalid_vmcs:N
>> > /sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
>> > /sys/module/kvm_intel/parameters/enable_apicv:Y
>> > /sys/module/kvm_intel/parameters/enable_ipiv:Y
>> > /sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y
>> > /sys/module/kvm_intel/parameters/ept:Y
>> > /sys/module/kvm_intel/parameters/eptad:Y
>> > /sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y
>> > /sys/module/kvm_intel/parameters/fasteoi:Y
>> > /sys/module/kvm_intel/parameters/flexpriority:Y
>> > /sys/module/kvm_intel/parameters/nested:Y
>> > /sys/module/kvm_intel/parameters/nested_early_check:N
>> > /sys/module/kvm_intel/parameters/ple_gap:128
>> > /sys/module/kvm_intel/parameters/ple_window:4096
>> > /sys/module/kvm_intel/parameters/ple_window_grow:2
>> > /sys/module/kvm_intel/parameters/ple_window_max:4294967295
>> > /sys/module/kvm_intel/parameters/ple_window_shrink:0
>> > /sys/module/kvm_intel/parameters/pml:Y
>> > /sys/module/kvm_intel/parameters/preemption_timer:Y
>> > /sys/module/kvm_intel/parameters/sgx:N
>> > /sys/module/kvm_intel/parameters/unrestricted_guest:Y
>> > /sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required
>> > /sys/module/kvm_intel/parameters/vnmi:Y
>> > /sys/module/kvm_intel/parameters/vpid:Y
>> >
>> > I tried to disable ipiv, but it doesn't help. freebsd hang on reboot.
>> > sudo modprobe -r kvm_intel
>> > sudo modprobe  kvm_intel enable_ipiv=N
>> > /sys/module/kvm_intel/parameters/enable_ipiv:N
>> >
>> > Thx!
>> +cc Vitaly
>> Sorry, I missed one detail, we are use hyper-V enlightment features:
>> "+hv-relaxed,+hv-vapic,+hv-time,+hv-runtime,hv-spinlocks=0x1fff,+hv-vpindex,+hv-synic,+hv-stimer,+hv-tlbflush,hv-ipi."
>>
>> did a lot tests with different features, and looks the hang is related
>> to  +hv-synic,+hv-stimer.  hv-synic seems the key which causes boot
>> hang of Freebsd 14.
>>
>> But the problem seems fixed with FreeBSD 15?  I guess it's this fix:
> https://reviews.freebsd.org/D43508
>
>>
>> Seems it's a bug from freebsd side, rather than on kvm side to me, but
>> I'm puzzled by disable apicv helps?

In theory, FreeBSD should work well even if KVM is misdetected as
genuine Hyper-V. Apparently, our emulation is not 1:1 and there are
subtle differences which cause the hang. I did not look at FreeBSD code
at all but my wild guess is that SynIC/stimer are not disabled properly
upon reboot and this causes the problem. If we somehow manage to find
how genuine Hyper-V's behavior is different, it would make sense to
update KVM/QEMU to match.

-- 
Vitaly

Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU

[Jinpu Wang]

On Thu, Oct 9, 2025 at 2:53 PM Vitaly Kuznetsov <vkuznets@redhat.com> wrote:
>
> Jinpu Wang <jinpu.wang@ionos.com> writes:
>
> > On Thu, Oct 9, 2025 at 1:21 PM Jinpu Wang <jinpu.wang@ionos.com> wrote:
> >>
> >> On Thu, Oct 9, 2025 at 5:44 AM Jinpu Wang <jinpu.wang@ionos.com> wrote:
> >> >
> >> > Hi Sean,
> >> >
> >> > On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@google.com> wrote:
> >> > >
> >> > > Trimmed Cc: to drop people from the original thread.  In the future, just start
> >> > > a new bug report.  Piggybacking a 10 year old bug just because the symptoms are
> >> > > similar does more harm than good.  Whatever the old thread was chasing was already
> >> > > fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed
> >> > > the problem.  I.e. whatever they were chasing _can't_ be the same root cause,
> >> > > because even if it's literally the same code bug, it would require a code change
> >> > > and thus a regression between v4.0 and v6.1.
> >> > Thx for the reply,  it makes sense. I will remember this next time.
> >> > >
> >> > > On Wed, Oct 08, 2025, Jinpu Wang wrote:
> >> > > > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote:
> >> > > > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
> >> > > > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
> >> > >
> >> > > Are there any host kernels that _do_ work?  E.g. have you tried a bleeding edge
> >> > > host kernel?
> >> > I will try linus/master today.
> >> > >
> >> > > > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
> >> > > > >
> >> > > > > **Summary:**
> >> > > > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with
> >> > > > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with
> >> > > > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The
> >> > > > > same configuration works correctly on Ice Lake (Xeon Gold 6338).
> >> > >
> >> > > Does Sierra Forest have IPI virtualization?  If so, you could try running with
> >> > > APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization.
> >> > Yes, it does:
> >> > $  grep . /sys/module/kvm_intel/parameters/*
> >> > /sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N
> >> > /sys/module/kvm_intel/parameters/dump_invalid_vmcs:N
> >> > /sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
> >> > /sys/module/kvm_intel/parameters/enable_apicv:Y
> >> > /sys/module/kvm_intel/parameters/enable_ipiv:Y
> >> > /sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y
> >> > /sys/module/kvm_intel/parameters/ept:Y
> >> > /sys/module/kvm_intel/parameters/eptad:Y
> >> > /sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y
> >> > /sys/module/kvm_intel/parameters/fasteoi:Y
> >> > /sys/module/kvm_intel/parameters/flexpriority:Y
> >> > /sys/module/kvm_intel/parameters/nested:Y
> >> > /sys/module/kvm_intel/parameters/nested_early_check:N
> >> > /sys/module/kvm_intel/parameters/ple_gap:128
> >> > /sys/module/kvm_intel/parameters/ple_window:4096
> >> > /sys/module/kvm_intel/parameters/ple_window_grow:2
> >> > /sys/module/kvm_intel/parameters/ple_window_max:4294967295
> >> > /sys/module/kvm_intel/parameters/ple_window_shrink:0
> >> > /sys/module/kvm_intel/parameters/pml:Y
> >> > /sys/module/kvm_intel/parameters/preemption_timer:Y
> >> > /sys/module/kvm_intel/parameters/sgx:N
> >> > /sys/module/kvm_intel/parameters/unrestricted_guest:Y
> >> > /sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required
> >> > /sys/module/kvm_intel/parameters/vnmi:Y
> >> > /sys/module/kvm_intel/parameters/vpid:Y
> >> >
> >> > I tried to disable ipiv, but it doesn't help. freebsd hang on reboot.
> >> > sudo modprobe -r kvm_intel
> >> > sudo modprobe  kvm_intel enable_ipiv=N
> >> > /sys/module/kvm_intel/parameters/enable_ipiv:N
> >> >
> >> > Thx!
> >> +cc Vitaly
> >> Sorry, I missed one detail, we are use hyper-V enlightment features:
> >> "+hv-relaxed,+hv-vapic,+hv-time,+hv-runtime,hv-spinlocks=0x1fff,+hv-vpindex,+hv-synic,+hv-stimer,+hv-tlbflush,hv-ipi."
> >>
> >> did a lot tests with different features, and looks the hang is related
> >> to  +hv-synic,+hv-stimer.  hv-synic seems the key which causes boot
> >> hang of Freebsd 14.
> >>
> >> But the problem seems fixed with FreeBSD 15?  I guess it's this fix:
> > https://reviews.freebsd.org/D43508
> >
> >>
> >> Seems it's a bug from freebsd side, rather than on kvm side to me, but
> >> I'm puzzled by disable apicv helps?
>
> In theory, FreeBSD should work well even if KVM is misdetected as
> genuine Hyper-V. Apparently, our emulation is not 1:1 and there are
> subtle differences which cause the hang. I did not look at FreeBSD code
> at all but my wild guess is that SynIC/stimer are not disabled properly
> upon reboot and this causes the problem. If we somehow manage to find
> how genuine Hyper-V's behavior is different, it would make sense to
> update KVM/QEMU to match.
>
> --
> Vitaly
Hi Vitaly,
Thx for the reply, that's another view.
I spent a few hours to build freebsd stable/14 with commit
6744fd8e7503 ("x86/cpu: improve hypervisor detection") applied, the
bug is fixed, no boot hung, and boot is much faster ~ 40 seconds, as
Freebsd 15, without the fix Freebsd takes 150 seconds if sucessfully
boots, and often hang forever during reboot.

Jinpu
>

Re: Hang on reboot in FreeBSD guest on Linux KVM host

[John Nielsen]

On Jun 16, 2014, at 10:21 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:

> Il 16/06/2014 18:47, John Nielsen ha scritto:
>> On Jun 16, 2014, at 10:39 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> 
>>> Il 16/06/2014 18:09, John Nielsen ha scritto:
>>>>>> The only substantial difference on the hardware side is the CPU.
>>>>>> The hosts where the problem occurs use "Intel(R) Xeon(R) CPU
>>>>>> E5-2650 v2 @ 2.60GHz", while the hosts that don't show the
>>>>>> problem use the prior revision, "Intel(R) Xeon(R) CPU E5-2650 0 @
>>>>>> 2.00GHz".
>>> 
>>> Can you do "grep . /sys/module/kvm_intel/parameters/*" on both hosts please?
>> 
>> No differences that I can see. Output below.
> 
> Not really:
> 
>> Working host:
>> Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
>> # grep . /sys/module/kvm_intel/parameters/*
>> /sys/module/kvm_intel/parameters/enable_apicv:N
>> 
>> Problem host:
>> Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
>> # grep . /sys/module/kvm_intel/parameters/*
>> /sys/module/kvm_intel/parameters/enable_apicv:Y
> 
> So we have a clue.  Let me study the code more, I'll try to get back with a suggestion.

Paolo, have you had an opportunity to look in to this some more?

Thanks,

JN

Re: Hang on reboot in FreeBSD guest on Linux KVM host

[Paolo Bonzini]

Il 20/06/2014 17:41, John Nielsen ha scritto:
>> >
>> > So we have a clue.  Let me study the code more, I'll try to get back with a suggestion.
> Paolo, have you had an opportunity to look in to this some more?

Not yet, sorry.

One possibility is this though.  Can you try migrating (or 
saving/restoring) the guest when it's hung, and see if it resuscitates?

Paolo

Re: Hang on reboot in FreeBSD guest on Linux KVM host

[John Nielsen]

On Jun 20, 2014, at 1:53 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:

> Il 20/06/2014 17:41, John Nielsen ha scritto:
>>> >
>>> > So we have a clue.  Let me study the code more, I'll try to get back with a suggestion.
>> Paolo, have you had an opportunity to look in to this some more?
> 
> Not yet, sorry.
> 
> One possibility is this though.  Can you try migrating (or saving/restoring) the guest when it's hung, and see if it resuscitates?

The guest is still hung after a save/restore.

# /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512 -smp 2,sockets=1,cores=1,threads=2 -drive file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2 -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none -monitor stdio
QEMU 2.0.50 monitor - type 'help' for more information
(qemu) stop
(qemu) savevm smphang
(qemu) q
# /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512 -smp 2,sockets=1,cores=1,threads=2 -drive file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2 -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none -monitor stdio -loadvm smphang
QEMU 2.0.50 monitor - type 'help' for more information
(qemu) 

[The VNC console shows the same hung kernel screen as when I ran savevm]

JN