* Re: Hang on reboot in FreeBSD guest on Linux KVM host
[not found] ` <4F14D859-D641-4AB5-B749-83D9D82F1DEA@jnielsen.net>
@ 2014-06-17 4:21 ` Paolo Bonzini
2014-06-17 6:05 ` Gleb Natapov
2014-06-20 15:41 ` Hang on reboot in FreeBSD guest on Linux KVM host John Nielsen
0 siblings, 2 replies; 25+ messages in thread
From: Paolo Bonzini @ 2014-06-17 4:21 UTC (permalink / raw)
To: John Nielsen; +Cc: kvm@vger.kernel.org
Il 16/06/2014 18:47, John Nielsen ha scritto:
> On Jun 16, 2014, at 10:39 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>> Il 16/06/2014 18:09, John Nielsen ha scritto:
>>>>> The only substantial difference on the hardware side is the CPU.
>>>>> The hosts where the problem occurs use "Intel(R) Xeon(R) CPU
>>>>> E5-2650 v2 @ 2.60GHz", while the hosts that don't show the
>>>>> problem use the prior revision, "Intel(R) Xeon(R) CPU E5-2650 0 @
>>>>> 2.00GHz".
>>
>> Can you do "grep . /sys/module/kvm_intel/parameters/*" on both hosts please?
>
> No differences that I can see. Output below.
Not really:
> Working host:
> Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
> # grep . /sys/module/kvm_intel/parameters/*
> /sys/module/kvm_intel/parameters/enable_apicv:N
>
> Problem host:
> Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
> # grep . /sys/module/kvm_intel/parameters/*
> /sys/module/kvm_intel/parameters/enable_apicv:Y
So we have a clue. Let me study the code more, I'll try to get back
with a suggestion.
In the meanwhile, I'm CCing the KVM list and BCCing QEMU, so that
follow-ups come to the KVM list.
Paolo
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in FreeBSD guest on Linux KVM host
2014-06-17 4:21 ` Hang on reboot in FreeBSD guest on Linux KVM host Paolo Bonzini
@ 2014-06-17 6:05 ` Gleb Natapov
2014-06-17 16:48 ` John Nielsen
2014-06-20 15:41 ` Hang on reboot in FreeBSD guest on Linux KVM host John Nielsen
1 sibling, 1 reply; 25+ messages in thread
From: Gleb Natapov @ 2014-06-17 6:05 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: John Nielsen, kvm@vger.kernel.org
On Tue, Jun 17, 2014 at 06:21:23AM +0200, Paolo Bonzini wrote:
> Il 16/06/2014 18:47, John Nielsen ha scritto:
> >On Jun 16, 2014, at 10:39 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> >
> >>Il 16/06/2014 18:09, John Nielsen ha scritto:
> >>>>>The only substantial difference on the hardware side is the CPU.
> >>>>>The hosts where the problem occurs use "Intel(R) Xeon(R) CPU
> >>>>>E5-2650 v2 @ 2.60GHz", while the hosts that don't show the
> >>>>>problem use the prior revision, "Intel(R) Xeon(R) CPU E5-2650 0 @
> >>>>>2.00GHz".
> >>
> >>Can you do "grep . /sys/module/kvm_intel/parameters/*" on both hosts please?
> >
> >No differences that I can see. Output below.
>
> Not really:
>
> >Working host:
> >Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
> ># grep . /sys/module/kvm_intel/parameters/*
> >/sys/module/kvm_intel/parameters/enable_apicv:N
> >
> >Problem host:
> >Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
> ># grep . /sys/module/kvm_intel/parameters/*
> >/sys/module/kvm_intel/parameters/enable_apicv:Y
>
> So we have a clue. Let me study the code more, I'll try to get back with a
> suggestion.
>
Does disabling apicv on E5-2650 v2 make reboot problem go away?
--
Gleb.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in FreeBSD guest on Linux KVM host
2014-06-17 6:05 ` Gleb Natapov
@ 2014-06-17 16:48 ` John Nielsen
2015-06-22 18:02 ` John Nielsen
0 siblings, 1 reply; 25+ messages in thread
From: John Nielsen @ 2014-06-17 16:48 UTC (permalink / raw)
To: Gleb Natapov; +Cc: Paolo Bonzini, kvm@vger.kernel.org
On Jun 17, 2014, at 12:05 AM, Gleb Natapov <gleb@kernel.org> wrote:
> On Tue, Jun 17, 2014 at 06:21:23AM +0200, Paolo Bonzini wrote:
>> Il 16/06/2014 18:47, John Nielsen ha scritto:
>>> On Jun 16, 2014, at 10:39 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>>
>>>> Il 16/06/2014 18:09, John Nielsen ha scritto:
>>>>>>> The only substantial difference on the hardware side is the CPU.
>>>>>>> The hosts where the problem occurs use "Intel(R) Xeon(R) CPU
>>>>>>> E5-2650 v2 @ 2.60GHz", while the hosts that don't show the
>>>>>>> problem use the prior revision, "Intel(R) Xeon(R) CPU E5-2650 0 @
>>>>>>> 2.00GHz".
>>>>
>>>> Can you do "grep . /sys/module/kvm_intel/parameters/*" on both hosts please?
>>>
>>> No differences that I can see. Output below.
>>
>> Not really:
>>
>>> Working host:
>>> Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
>>> # grep . /sys/module/kvm_intel/parameters/*
>>> /sys/module/kvm_intel/parameters/enable_apicv:N
>>>
>>> Problem host:
>>> Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
>>> # grep . /sys/module/kvm_intel/parameters/*
>>> /sys/module/kvm_intel/parameters/enable_apicv:Y
>>
>> So we have a clue. Let me study the code more, I'll try to get back with a
>> suggestion.
Wow, can't believe I missed that. Good catch!
> Does disabling apicv on E5-2650 v2 make reboot problem go away?
Yes it does!
# modprobe kvm_intel
/sys/module/kvm_intel/parameters/enable_apicv:Y
# /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512 -smp 2,sockets=1,cores=1,threads=2 -drive file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2 -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none
[problem occurs]
# rmmod kvm_intel
# modprobe kvm_intel enable_apicv=N
/sys/module/kvm_intel/parameters/enable_apicv:N
# /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512 -smp 2,sockets=1,cores=1,threads=2 -drive file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2 -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none
[problem does not occur]
Thank you. This both narrows the problem considerably and provides an acceptable workaround. It would still be nice to see it fixed, of course. Keep me CC'ed as I'm not on the KVM list.
JN
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in FreeBSD guest on Linux KVM host
2014-06-17 4:21 ` Hang on reboot in FreeBSD guest on Linux KVM host Paolo Bonzini
2014-06-17 6:05 ` Gleb Natapov
@ 2014-06-20 15:41 ` John Nielsen
2014-06-20 19:53 ` Paolo Bonzini
1 sibling, 1 reply; 25+ messages in thread
From: John Nielsen @ 2014-06-20 15:41 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: kvm@vger.kernel.org
On Jun 16, 2014, at 10:21 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 16/06/2014 18:47, John Nielsen ha scritto:
>> On Jun 16, 2014, at 10:39 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>
>>> Il 16/06/2014 18:09, John Nielsen ha scritto:
>>>>>> The only substantial difference on the hardware side is the CPU.
>>>>>> The hosts where the problem occurs use "Intel(R) Xeon(R) CPU
>>>>>> E5-2650 v2 @ 2.60GHz", while the hosts that don't show the
>>>>>> problem use the prior revision, "Intel(R) Xeon(R) CPU E5-2650 0 @
>>>>>> 2.00GHz".
>>>
>>> Can you do "grep . /sys/module/kvm_intel/parameters/*" on both hosts please?
>>
>> No differences that I can see. Output below.
>
> Not really:
>
>> Working host:
>> Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
>> # grep . /sys/module/kvm_intel/parameters/*
>> /sys/module/kvm_intel/parameters/enable_apicv:N
>>
>> Problem host:
>> Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
>> # grep . /sys/module/kvm_intel/parameters/*
>> /sys/module/kvm_intel/parameters/enable_apicv:Y
>
> So we have a clue. Let me study the code more, I'll try to get back with a suggestion.
Paolo, have you had an opportunity to look in to this some more?
Thanks,
JN
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in FreeBSD guest on Linux KVM host
2014-06-20 15:41 ` Hang on reboot in FreeBSD guest on Linux KVM host John Nielsen
@ 2014-06-20 19:53 ` Paolo Bonzini
2014-06-20 21:39 ` John Nielsen
0 siblings, 1 reply; 25+ messages in thread
From: Paolo Bonzini @ 2014-06-20 19:53 UTC (permalink / raw)
To: John Nielsen; +Cc: kvm@vger.kernel.org
Il 20/06/2014 17:41, John Nielsen ha scritto:
>> >
>> > So we have a clue. Let me study the code more, I'll try to get back with a suggestion.
> Paolo, have you had an opportunity to look in to this some more?
Not yet, sorry.
One possibility is this though. Can you try migrating (or
saving/restoring) the guest when it's hung, and see if it resuscitates?
Paolo
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in FreeBSD guest on Linux KVM host
2014-06-20 19:53 ` Paolo Bonzini
@ 2014-06-20 21:39 ` John Nielsen
0 siblings, 0 replies; 25+ messages in thread
From: John Nielsen @ 2014-06-20 21:39 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: kvm@vger.kernel.org
On Jun 20, 2014, at 1:53 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 20/06/2014 17:41, John Nielsen ha scritto:
>>> >
>>> > So we have a clue. Let me study the code more, I'll try to get back with a suggestion.
>> Paolo, have you had an opportunity to look in to this some more?
>
> Not yet, sorry.
>
> One possibility is this though. Can you try migrating (or saving/restoring) the guest when it's hung, and see if it resuscitates?
The guest is still hung after a save/restore.
# /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512 -smp 2,sockets=1,cores=1,threads=2 -drive file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2 -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none -monitor stdio
QEMU 2.0.50 monitor - type 'help' for more information
(qemu) stop
(qemu) savevm smphang
(qemu) q
# /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512 -smp 2,sockets=1,cores=1,threads=2 -drive file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2 -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none -monitor stdio -loadvm smphang
QEMU 2.0.50 monitor - type 'help' for more information
(qemu)
[The VNC console shows the same hung kernel screen as when I ran savevm]
JN
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in FreeBSD guest on Linux KVM host
2014-06-17 16:48 ` John Nielsen
@ 2015-06-22 18:02 ` John Nielsen
2015-06-22 21:48 ` Bandan Das
0 siblings, 1 reply; 25+ messages in thread
From: John Nielsen @ 2015-06-22 18:02 UTC (permalink / raw)
To: Gleb Natapov; +Cc: Paolo Bonzini, kvm@vger.kernel.org, John Nielsen
On Jun 17, 2014, at 10:48 AM, John Nielsen <lists@jnielsen.net> wrote:
> On Jun 17, 2014, at 12:05 AM, Gleb Natapov <gleb@kernel.org> wrote:
>
>> On Tue, Jun 17, 2014 at 06:21:23AM +0200, Paolo Bonzini wrote:
>>> Il 16/06/2014 18:47, John Nielsen ha scritto:
>>>> On Jun 16, 2014, at 10:39 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>>>
>>>>> Il 16/06/2014 18:09, John Nielsen ha scritto:
>>>>>>>> The only substantial difference on the hardware side is the CPU.
>>>>>>>> The hosts where the problem occurs use "Intel(R) Xeon(R) CPU
>>>>>>>> E5-2650 v2 @ 2.60GHz", while the hosts that don't show the
>>>>>>>> problem use the prior revision, "Intel(R) Xeon(R) CPU E5-2650 0 @
>>>>>>>> 2.00GHz".
>>>>>
>>>>> Can you do "grep . /sys/module/kvm_intel/parameters/*" on both hosts please?
>>>>
>>>> No differences that I can see. Output below.
>>>
>>> Not really:
>>>
>>>> Working host:
>>>> Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
>>>> # grep . /sys/module/kvm_intel/parameters/*
>>>> /sys/module/kvm_intel/parameters/enable_apicv:N
>>>>
>>>> Problem host:
>>>> Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
>>>> # grep . /sys/module/kvm_intel/parameters/*
>>>> /sys/module/kvm_intel/parameters/enable_apicv:Y
>>>
>>> So we have a clue. Let me study the code more, I'll try to get back with a
>>> suggestion.
>
> Wow, can't believe I missed that. Good catch!
>
>> Does disabling apicv on E5-2650 v2 make reboot problem go away?
>
> Yes it does!
>
> # modprobe kvm_intel
> /sys/module/kvm_intel/parameters/enable_apicv:Y
> # /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512 -smp 2,sockets=1,cores=1,threads=2 -drive file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2 -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none
>
> [problem occurs]
>
> # rmmod kvm_intel
> # modprobe kvm_intel enable_apicv=N
> /sys/module/kvm_intel/parameters/enable_apicv:N
> # /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512 -smp 2,sockets=1,cores=1,threads=2 -drive file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2 -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none
>
> [problem does not occur]
>
> Thank you. This both narrows the problem considerably and provides an acceptable workaround. It would still be nice to see it fixed, of course. Keep me CC'ed as I'm not on the KVM list.
I’m resurrecting an old thread since I haven’t heard anything in a while. Has anyone looked in to the KVM+apicv bug documented above as well as here:
https://bugs.launchpad.net/qemu/+bug/1329956 ?
If appropriate, where should I go to file a KVM bug (since this isn’t really Qemu’s problem)?
Thanks,
JN--
To unsubscribe from this list: send the line "unsubscribe kvm" in
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in FreeBSD guest on Linux KVM host
2015-06-22 18:02 ` John Nielsen
@ 2015-06-22 21:48 ` Bandan Das
2015-06-22 22:08 ` John Nielsen
0 siblings, 1 reply; 25+ messages in thread
From: Bandan Das @ 2015-06-22 21:48 UTC (permalink / raw)
To: John Nielsen
Cc: Gleb Natapov, Paolo Bonzini, kvm@vger.kernel.org, John Nielsen
John Nielsen <lists@jnielsen.net> writes:
> On Jun 17, 2014, at 10:48 AM, John Nielsen <lists@jnielsen.net> wrote:
>
>> On Jun 17, 2014, at 12:05 AM, Gleb Natapov <gleb@kernel.org> wrote:
>>
>>> On Tue, Jun 17, 2014 at 06:21:23AM +0200, Paolo Bonzini wrote:
>>>> Il 16/06/2014 18:47, John Nielsen ha scritto:
>>>>> On Jun 16, 2014, at 10:39 AM, Paolo Bonzini <pbonzini@redhat.com>
>>>>> wrote:
>>>>>
>>>>>> Il 16/06/2014 18:09, John Nielsen ha scritto:
>>>>>>>>> The only substantial difference on the hardware side is the
>>>>>>>>> CPU. The hosts where the problem occurs use "Intel(R)
>>>>>>>>> Xeon(R) CPU E5-2650 v2 @ 2.60GHz", while the hosts that don't
>>>>>>>>> show the problem use the prior revision, "Intel(R) Xeon(R)
>>>>>>>>> CPU E5-2650 0 @ 2.00GHz".
>>>>>> Can you do "grep . /sys/module/kvm_intel/parameters/*" on both
>>>>>> hosts please?
>>>>> No differences that I can see. Output below.
>>>> Not really:
>>>>
>>>>> Working host: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz # grep
>>>>> . /sys/module/kvm_intel/parameters/*
>>>>> /sys/module/kvm_intel/parameters/enable_apicv:N
>>>>>
>>>>> Problem host: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz # grep
>>>>> . /sys/module/kvm_intel/parameters/*
>>>>> /sys/module/kvm_intel/parameters/enable_apicv:Y
>>>> So we have a clue. Let me study the code more, I'll try to get
>>>> back with a suggestion.
>> Wow, can't believe I missed that. Good catch!
>>
>>> Does disabling apicv on E5-2650 v2 make reboot problem go away?
>> Yes it does!
>>
>> # modprobe kvm_intel /sys/module/kvm_intel/parameters/enable_apicv:Y
>> # /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m
>> 512 -smp 2,sockets=1,cores=1,threads=2 -drive
>> file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2
>> -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net
>> none
>>
>> [problem occurs]
>>
>> # rmmod kvm_intel # modprobe kvm_intel enable_apicv=N
>> /sys/module/kvm_intel/parameters/enable_apicv:N #
>> /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512
>> -smp 2,sockets=1,cores=1,threads=2 -drive
>> file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2
>> -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net
>> none
>>
>> [problem does not occur]
>>
>> Thank you. This both narrows the problem considerably and provides
>> an acceptable workaround. It would still be nice to see it fixed, of
>> course. Keep me CC'ed as I'm not on the KVM list.
>
> I’m resurrecting an old thread since I haven’t heard anything in a
> while. Has anyone looked in to the KVM+apicv bug documented above as
> well as here:
>
> https://bugs.launchpad.net/qemu/+bug/1329956 ?
>
> If appropriate, where should I go to file a KVM bug (since this isn’t
> really Qemu’s problem)?
Hi John, does this happen with the latest upstream kernel version ?
Bandan
> Thanks,
>
> JN-- To unsubscribe from this list: send the line "unsubscribe kvm" in
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in FreeBSD guest on Linux KVM host
2015-06-22 21:48 ` Bandan Das
@ 2015-06-22 22:08 ` John Nielsen
2015-06-23 8:14 ` Paolo Bonzini
2015-06-24 15:50 ` Paolo Bonzini
0 siblings, 2 replies; 25+ messages in thread
From: John Nielsen @ 2015-06-22 22:08 UTC (permalink / raw)
To: Bandan Das; +Cc: Gleb Natapov, Paolo Bonzini, kvm@vger.kernel.org, John Nielsen
On Jun 22, 2015, at 3:48 PM, Bandan Das <bsd@redhat.com> wrote:
> John Nielsen <lists@jnielsen.net> writes:
>
>> On Jun 17, 2014, at 10:48 AM, John Nielsen <lists@jnielsen.net> wrote:
>>
>>> On Jun 17, 2014, at 12:05 AM, Gleb Natapov <gleb@kernel.org> wrote:
>>>
>>>> On Tue, Jun 17, 2014 at 06:21:23AM +0200, Paolo Bonzini wrote:
>>>>> Il 16/06/2014 18:47, John Nielsen ha scritto:
>>>>>> On Jun 16, 2014, at 10:39 AM, Paolo Bonzini <pbonzini@redhat.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Il 16/06/2014 18:09, John Nielsen ha scritto:
>>>>>>>>>> The only substantial difference on the hardware side is the
>>>>>>>>>> CPU. The hosts where the problem occurs use "Intel(R)
>>>>>>>>>> Xeon(R) CPU E5-2650 v2 @ 2.60GHz", while the hosts that don't
>>>>>>>>>> show the problem use the prior revision, "Intel(R) Xeon(R)
>>>>>>>>>> CPU E5-2650 0 @ 2.00GHz".
>>>>>>> Can you do "grep . /sys/module/kvm_intel/parameters/*" on both
>>>>>>> hosts please?
>>>>>> No differences that I can see. Output below.
>>>>> Not really:
>>>>>
>>>>>> Working host: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz # grep
>>>>>> . /sys/module/kvm_intel/parameters/*
>>>>>> /sys/module/kvm_intel/parameters/enable_apicv:N
>>>>>>
>>>>>> Problem host: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz # grep
>>>>>> . /sys/module/kvm_intel/parameters/*
>>>>>> /sys/module/kvm_intel/parameters/enable_apicv:Y
>>>>> So we have a clue. Let me study the code more, I'll try to get
>>>>> back with a suggestion.
>>> Wow, can't believe I missed that. Good catch!
>>>
>>>> Does disabling apicv on E5-2650 v2 make reboot problem go away?
>>> Yes it does!
>>>
>>> # modprobe kvm_intel /sys/module/kvm_intel/parameters/enable_apicv:Y
>>> # /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m
>>> 512 -smp 2,sockets=1,cores=1,threads=2 -drive
>>> file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2
>>> -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net
>>> none
>>>
>>> [problem occurs]
>>>
>>> # rmmod kvm_intel # modprobe kvm_intel enable_apicv=N
>>> /sys/module/kvm_intel/parameters/enable_apicv:N #
>>> /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512
>>> -smp 2,sockets=1,cores=1,threads=2 -drive
>>> file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2
>>> -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net
>>> none
>>>
>>> [problem does not occur]
>>>
>>> Thank you. This both narrows the problem considerably and provides
>>> an acceptable workaround. It would still be nice to see it fixed, of
>>> course. Keep me CC'ed as I'm not on the KVM list.
>>
>> I’m resurrecting an old thread since I haven’t heard anything in a
>> while. Has anyone looked in to the KVM+apicv bug documented above as
>> well as here:
>>
>> https://bugs.launchpad.net/qemu/+bug/1329956 ?
>>
>> If appropriate, where should I go to file a KVM bug (since this isn’t
>> really Qemu’s problem)?
>
> Hi John, does this happen with the latest upstream kernel version ?
I know for sure it happens with 4.0.4 and I’m not aware of any newer changes that would affect it.--
To unsubscribe from this list: send the line "unsubscribe kvm" in
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in FreeBSD guest on Linux KVM host
2015-06-22 22:08 ` John Nielsen
@ 2015-06-23 8:14 ` Paolo Bonzini
2015-06-24 15:50 ` Paolo Bonzini
1 sibling, 0 replies; 25+ messages in thread
From: Paolo Bonzini @ 2015-06-23 8:14 UTC (permalink / raw)
To: John Nielsen, Bandan Das; +Cc: Gleb Natapov, kvm@vger.kernel.org, John Nielsen
On 23/06/2015 00:08, John Nielsen wrote:
>>>> Thank you. This both narrows the problem considerably and provides
>>>> an acceptable workaround. It would still be nice to see it fixed, of
>>>> course. Keep me CC'ed as I'm not on the KVM list.
>>>
>>> I’m resurrecting an old thread since I haven’t heard anything in a
>>> while. Has anyone looked in to the KVM+apicv bug documented above as
>>> well as here:
>>>
>>> https://bugs.launchpad.net/qemu/+bug/1329956 ?
>>>
>>> If appropriate, where should I go to file a KVM bug (since this isn’t
>>> really Qemu’s problem)?
>>
>> Hi John, does this happen with the latest upstream kernel version ?
>
> I know for sure it happens with 4.0.4 and I’m not aware of any newer changes that would affect it.--
> To unsubscribe from this list: send the line "unsubscribe kvm" in
Hi John, I now have APICv hardware much more easily available (read: at
home) so I can look at this again.
Paolo
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in FreeBSD guest on Linux KVM host
2015-06-22 22:08 ` John Nielsen
2015-06-23 8:14 ` Paolo Bonzini
@ 2015-06-24 15:50 ` Paolo Bonzini
2015-06-24 15:57 ` Bandan Das
2015-06-24 17:07 ` John Nielsen
1 sibling, 2 replies; 25+ messages in thread
From: Paolo Bonzini @ 2015-06-24 15:50 UTC (permalink / raw)
To: John Nielsen, Bandan Das; +Cc: Gleb Natapov, kvm@vger.kernel.org, John Nielsen
On 23/06/2015 00:08, John Nielsen wrote:
>>> I’m resurrecting an old thread since I haven’t heard anything in a
>>> while. Has anyone looked in to the KVM+apicv bug documented above as
>>> well as here:
>>>
>>> https://bugs.launchpad.net/qemu/+bug/1329956 ?
>>>
>>> If appropriate, where should I go to file a KVM bug (since this isn’t
>>> really Qemu’s problem)?
>>
>> Hi John, does this happen with the latest upstream kernel version ?
>
> I know for sure it happens with 4.0.4 and I’m not aware of any newer changes that would affect it.--
> To unsubscribe from this list: send the line "unsubscribe kvm" in
Can you reproduce it with 10.1?
I did this:
1) download
http://download.pcbsd.org/iso/10.1-RELEASE/amd64/PCBSD10.1.2-x64-trueos-server.raw.xz
and unpack it
2) run it with "qemu-kvm -drive
if=virtio,PCBSD10.1.2-x64-trueos-server.raw -smp 2"
3) login as root/pcbsd, type "reboot"
I would like to know if I'm doing anything wrong. My machine is a Xeon
E5 v3 (Haswell). My SeaBIOS build doesn't have the atkbd0 bug, but just
to rule that out, can you send me your Seabios binary
(/usr/share/qemu/bios*.bin) as well?
Paolo
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in FreeBSD guest on Linux KVM host
2015-06-24 15:50 ` Paolo Bonzini
@ 2015-06-24 15:57 ` Bandan Das
2015-06-24 15:58 ` Paolo Bonzini
2015-06-24 17:07 ` John Nielsen
1 sibling, 1 reply; 25+ messages in thread
From: Bandan Das @ 2015-06-24 15:57 UTC (permalink / raw)
To: Paolo Bonzini
Cc: John Nielsen, Gleb Natapov, kvm@vger.kernel.org, John Nielsen
Paolo Bonzini <pbonzini@redhat.com> writes:
...
>
> I did this:
>
> 1) download
> http://download.pcbsd.org/iso/10.1-RELEASE/amd64/PCBSD10.1.2-x64-trueos-server.raw.xz
> and unpack it
>
> 2) run it with "qemu-kvm -drive
> if=virtio,PCBSD10.1.2-x64-trueos-server.raw -smp 2"
>
> 3) login as root/pcbsd, type "reboot"
>
> I would like to know if I'm doing anything wrong. My machine is a Xeon
> E5 v3 (Haswell). My SeaBIOS build doesn't have the atkbd0 bug, but just
I thought (from the bug report) that the bug is only reproducible
on v2 Xeons ?
> to rule that out, can you send me your Seabios binary
> (/usr/share/qemu/bios*.bin) as well?
>
> Paolo
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in FreeBSD guest on Linux KVM host
2015-06-24 15:57 ` Bandan Das
@ 2015-06-24 15:58 ` Paolo Bonzini
0 siblings, 0 replies; 25+ messages in thread
From: Paolo Bonzini @ 2015-06-24 15:58 UTC (permalink / raw)
To: Bandan Das; +Cc: John Nielsen, Gleb Natapov, kvm@vger.kernel.org, John Nielsen
On 24/06/2015 17:57, Bandan Das wrote:
> Paolo Bonzini <pbonzini@redhat.com> writes:
> ...
>>
>> I did this:
>>
>> 1) download
>> http://download.pcbsd.org/iso/10.1-RELEASE/amd64/PCBSD10.1.2-x64-trueos-server.raw.xz
>> and unpack it
>>
>> 2) run it with "qemu-kvm -drive
>> if=virtio,PCBSD10.1.2-x64-trueos-server.raw -smp 2"
>>
>> 3) login as root/pcbsd, type "reboot"
>>
>> I would like to know if I'm doing anything wrong. My machine is a Xeon
>> E5 v3 (Haswell). My SeaBIOS build doesn't have the atkbd0 bug, but just
>
> I thought (from the bug report) that the bug is only reproducible
> on v2 Xeons ?
It is related to APICv. I think when the thread died out, v3 Xeons
didn't exist yet.
Paolo
>> to rule that out, can you send me your Seabios binary
>> (/usr/share/qemu/bios*.bin) as well?
>>
>> Paolo
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in FreeBSD guest on Linux KVM host
2015-06-24 15:50 ` Paolo Bonzini
2015-06-24 15:57 ` Bandan Das
@ 2015-06-24 17:07 ` John Nielsen
2015-06-24 17:15 ` Paolo Bonzini
2015-06-25 2:40 ` Wincy Van
1 sibling, 2 replies; 25+ messages in thread
From: John Nielsen @ 2015-06-24 17:07 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: Bandan Das, Gleb Natapov, kvm@vger.kernel.org
On Jun 24, 2015, at 9:50 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> On 23/06/2015 00:08, John Nielsen wrote:
>>>> I’m resurrecting an old thread since I haven’t heard anything in a
>>>> while. Has anyone looked in to the KVM+apicv bug documented above as
>>>> well as here:
>>>>
>>>> https://bugs.launchpad.net/qemu/+bug/1329956 ?
>>>>
>>>> If appropriate, where should I go to file a KVM bug (since this isn’t
>>>> really Qemu’s problem)?
>>>
>>> Hi John, does this happen with the latest upstream kernel version ?
>>
>> I know for sure it happens with 4.0.4 and I’m not aware of any newer changes that would affect it.--
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>
> Can you reproduce it with 10.1?
>
> I did this:
>
> 1) download
> http://download.pcbsd.org/iso/10.1-RELEASE/amd64/PCBSD10.1.2-x64-trueos-server.raw.xz
> and unpack it
>
> 2) run it with "qemu-kvm -drive
> if=virtio,PCBSD10.1.2-x64-trueos-server.raw -smp 2"
>
> 3) login as root/pcbsd, type "reboot"
>
> I would like to know if I'm doing anything wrong. My machine is a Xeon
> E5 v3 (Haswell). My SeaBIOS build doesn't have the atkbd0 bug, but just
> to rule that out, can you send me your Seabios binary
> (/usr/share/qemu/bios*.bin) as well?
Interesting. Using the same PC-BSD image I am able to reproduce on a server running slightly older software but I can not reproduce running bleeding edge. I verified enable_apicv=Y on both. In both cases I ran
qemu-kvm -drive if=virtio,file=PCBSD10.1.2-x64-trueos-server.raw -smp 2 -vnc 0.0.0.0:0
Specifically:
Breaks (VM hangs during boot after pressing ctrl-alt-del):
kernel 3.12.22
qemu-kvm-1.7.0-3.el6.x86_64
seabios-1.7.3.1-1.el6.noarch
Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
Works (VM reboots normally):
kernel 4.0.4
qemu-kvm-2.3.0-6.el7.centos.x86_64
seabios-bin-1.8.1-1.el7.centos.noarch
Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
Unfortunately I no longer have the test environment I used a few days ago to reproduce this issue so I can’t verify the software versions that were in use. It’s possible I was mistaken about the kernel version (I thought it was 4.0.4). Perhaps it really is fixed in the newer kernel? In any case, this is great news! I would be interested in identifying the patch(es) that fixed the issue to make back-porting them easier, but I won’t have time to do a binary search anytime soon.
Thanks for looking in to this again. If anyone else is interested in identifying what specifically fixed the issue please let me know if I can do anything to help.
JN
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in FreeBSD guest on Linux KVM host
2015-06-24 17:07 ` John Nielsen
@ 2015-06-24 17:15 ` Paolo Bonzini
2015-06-25 2:40 ` Wincy Van
1 sibling, 0 replies; 25+ messages in thread
From: Paolo Bonzini @ 2015-06-24 17:15 UTC (permalink / raw)
To: John Nielsen; +Cc: Bandan Das, Gleb Natapov, kvm@vger.kernel.org
On 24/06/2015 19:07, John Nielsen wrote:
>
> Unfortunately I no longer have the test environment I used a few days
> ago to reproduce this issue so I can’t verify the software versions
> that were in use. It’s possible I was mistaken about the kernel
> version (I thought it was 4.0.4). Perhaps it really is fixed in the
> newer kernel? In any case, this is great news! I would be interested
> in identifying the patch(es) that fixed the issue to make
> back-porting them easier, but I won’t have time to do a binary search
> anytime soon.
Random guess:
commit a323b409820c8afd33fbd841f5534eb84b406e8d
Author: Radim Krčmář <rkrcmar@redhat.com>
Date: Thu Oct 30 15:06:46 2014 +0100
KVM: x86: detect LVTT changes under APICv
commit e462755cae2bf8297a663278935ad4d59812d2f3
Author: Radim Krčmář <rkrcmar@redhat.com>
Date: Thu Oct 30 15:06:45 2014 +0100
KVM: x86: detect SPIV changes under APICv
Paolo
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in FreeBSD guest on Linux KVM host
2015-06-24 17:07 ` John Nielsen
2015-06-24 17:15 ` Paolo Bonzini
@ 2015-06-25 2:40 ` Wincy Van
2025-10-08 12:42 ` Jack Wang
` (2 more replies)
1 sibling, 3 replies; 25+ messages in thread
From: Wincy Van @ 2015-06-25 2:40 UTC (permalink / raw)
To: John Nielsen; +Cc: Paolo Bonzini, Bandan Das, Gleb Natapov, kvm@vger.kernel.org
On Thu, Jun 25, 2015 at 1:07 AM, John Nielsen <lists@jnielsen.net> wrote:
> Interesting. Using the same PC-BSD image I am able to reproduce on a server running slightly older software but I can not reproduce running bleeding edge. I verified enable_apicv=Y on both. In both cases I ran
> qemu-kvm -drive if=virtio,file=PCBSD10.1.2-x64-trueos-server.raw -smp 2 -vnc 0.0.0.0:0
>
> Specifically:
>
> Breaks (VM hangs during boot after pressing ctrl-alt-del):
> kernel 3.12.22
> qemu-kvm-1.7.0-3.el6.x86_64
> seabios-1.7.3.1-1.el6.noarch
> Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
>
> Works (VM reboots normally):
> kernel 4.0.4
> qemu-kvm-2.3.0-6.el7.centos.x86_64
> seabios-bin-1.8.1-1.el7.centos.noarch
> Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
>
>
> Unfortunately I no longer have the test environment I used a few days ago to reproduce this issue so I can’t verify the software versions that were in use. It’s possible I was mistaken about the kernel version (I thought it was 4.0.4). Perhaps it really is fixed in the newer kernel? In any case, this is great news! I would be interested in identifying the patch(es) that fixed the issue to make back-porting them easier, but I won’t have time to do a binary search anytime soon.
>
> Thanks for looking in to this again. If anyone else is interested in identifying what specifically fixed the issue please let me know if I can do anything to help.
>
John,
This commit may work for you.
commit 4114c27d450bef228be9c7b0c40a888e18a3a636
Author: Wei Wang <wei.w.wang@intel.com>
Date: Wed Nov 5 10:53:43 2014 +0800
KVM: x86: reset RVI upon system reset
Thanks
Wincy
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in FreeBSD guest on Linux KVM host
2015-06-25 2:40 ` Wincy Van
@ 2025-10-08 12:42 ` Jack Wang
2025-10-08 12:44 ` Jack Wang
2025-10-08 12:54 ` Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU Jinpu Wang
2 siblings, 0 replies; 25+ messages in thread
From: Jack Wang @ 2025-10-08 12:42 UTC (permalink / raw)
To: fanwenyi0529, John Nielsen; +Cc: bsd, gleb, kvm, pbonzini
From: Wincy Van <fanwenyi0529@gmail.com>
On Thu, Jun 25, 2015 at 1:07 AM, John Nielsen <lists@jnielsen.net> wrot=
e:
> Interesting. Using the same PC-BSD image I am able to reproduce on a =
server running slightly older software but I can not reproduce running =
bleeding edge. I verified enable_apicv=3DY on both. In both cases I ran
> qemu-kvm -drive if=3Dvirtio,file=3DPCBSD10.1.2-x64-trueos-server.raw =
-smp 2 -vnc 0.0.0.0:0
>
> Specifically:
>
> Breaks (VM hangs during boot after pressing ctrl-alt-del):
> kernel 3.12.22
> qemu-kvm-1.7.0-3.el6.x86_64
> seabios-1.7.3.1-1.el6.noarch
> Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
>
> Works (VM reboots normally):
> kernel 4.0.4
> qemu-kvm-2.3.0-6.el7.centos.x86_64
> seabios-bin-1.8.1-1.el7.centos.noarch
> Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
>
>
> Unfortunately I no longer have the test environment I used a few days=
ago to reproduce this issue so I can=E2=80=99t verify the software ver=
sions that were in use. It=E2=80=99s possible I was mistaken about the =
kernel version (I thought it was 4.0.4). Perhaps it really is fixed in =
the newer kernel? In any case, this is great news! I would be intereste=
d in identifying the patch(es) that fixed the issue to make back-portin=
g them easier, but I won=E2=80=99t have time to do a binary search anyt=
ime soon.
>
> Thanks for looking in to this again. If anyone else is interested in =
identifying what specifically fixed the issue please let me know if I c=
an do anything to help.
>
John,
This commit may work for you.
commit 4114c27d450bef228be9c7b0c40a888e18a3a636
Author: Wei Wang <wei.w.wang@intel.com>
Date: Wed Nov 5 10:53:43 2014 +0800
KVM: x86: reset RVI upon system reset
Thanks
Wincy
Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
---
### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
**Summary:**
Multi-cores FreeBSD guests hang during reboot under KVM on systems with Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The same configuration works correctly on Ice Lake (Xeon Gold 6338).
---
#### **Environment**
**Host:**
* OS: Debian 12 (Bookworm)
* Kernel versions tested: 6.1.118 and 6.12.47 (both affected)
* QEMU versions tested: 8.2.10 and 9.2.4
* Firmware: SeaBIOS 1.16.2-1 and OVMF 2024.11.1
* CPU: Intel(R) Xeon(R) 6710E (SierraForest)
* KVM module: `kvm_intel`
* Command-line (simplified example):
```
qemu-system-x86_64 -m 2048 -enable-kvm -cpu host -smp 4 -hda freebsd14.img
```
**Guest:**
* OS: FreeBSD 14.3-RELEASE
* SMP guests multi-cores
---
#### **Steps to Reproduce**
1. Start a FreeBSD 14.3 guest under KVM on a SierraForest host.
2. Log in and run:
```
# reboot
```
or press `Ctrl+Alt+Del`.
3. Observe that the VM hangs during reboot — it never returns to BIOS/UEFI.
---
#### **Expected Result**
FreeBSD should reboot cleanly and return to login prompt.
---
#### **Actual Result**
Guest hangs indefinitely during reboot.
QEMU remains running but guest output is frozen.
No host kernel warnings or errors in `dmesg`.
---
#### **Workaround**
Disable APICv on the host before starting QEMU:
```
modprobe kvm_intel enable_apicv=N
```
With `enable_apicv=N`, the FreeBSD guest reboots normally every time.
---
#### **Additional Information**
* The issue **does not reproduce** on an Ice Lake host (Intel Xeon Gold 6338) with identical kernel, QEMU, and guest image.
* Collected `trace-cmd` data for `kvm*` events during both:
* **Good reboot** (with `enable_apicv=N`)
* **Bad reboot** (default APICv enabled)
Traces are available for analysis upon request.
---
#### **Preliminary Analysis**
Disabling APICv avoids the hang, suggesting the problem is related to APIC virtualization on SierraForest.
Possible causes:
* Regression in APICv or posted-interrupt handling on new Xeon platforms.
* Microcode or MSR-related difference in APICv behavior on SierraForest.
* Incorrect EOI or interrupt delivery sequence during guest reboot.
---
#### **Reproducibility**
* 100% reproducible on SierraForest systems
* Not reproducible on Ice Lake
---
#### **Request for Guidance**
* Is this a known KVM or Intel virtualization issue on SierraForest?
* Would you like `trace-cmd` traces, MSR dumps, or `perf kvm` data for further diagnosis?
* Is there a kernel parameter or patch to selectively disable APICv on affected models?
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in FreeBSD guest on Linux KVM host
2015-06-25 2:40 ` Wincy Van
2025-10-08 12:42 ` Jack Wang
@ 2025-10-08 12:44 ` Jack Wang
2025-10-08 12:54 ` Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU Jinpu Wang
2 siblings, 0 replies; 25+ messages in thread
From: Jack Wang @ 2025-10-08 12:44 UTC (permalink / raw)
To: fanwenyi0529, John Nielsen; +Cc: bsd, gleb, kvm, pbonzini, xiaoyao.li
From: Wincy Van <fanwenyi0529@gmail.com>
On Thu, Jun 25, 2015 at 1:07 AM, John Nielsen <lists@jnielsen.net> wrot=
e:
> Interesting. Using the same PC-BSD image I am able to reproduce on a =
server running slightly older software but I can not reproduce running =
bleeding edge. I verified enable_apicv=3DY on both. In both cases I ran
> qemu-kvm -drive if=3Dvirtio,file=3DPCBSD10.1.2-x64-trueos-server.raw =
-smp 2 -vnc 0.0.0.0:0
>
> Specifically:
>
> Breaks (VM hangs during boot after pressing ctrl-alt-del):
> kernel 3.12.22
> qemu-kvm-1.7.0-3.el6.x86_64
> seabios-1.7.3.1-1.el6.noarch
> Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
>
> Works (VM reboots normally):
> kernel 4.0.4
> qemu-kvm-2.3.0-6.el7.centos.x86_64
> seabios-bin-1.8.1-1.el7.centos.noarch
> Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
>
>
> Unfortunately I no longer have the test environment I used a few days=
ago to reproduce this issue so I can=E2=80=99t verify the software ver=
sions that were in use. It=E2=80=99s possible I was mistaken about the =
kernel version (I thought it was 4.0.4). Perhaps it really is fixed in =
the newer kernel? In any case, this is great news! I would be intereste=
d in identifying the patch(es) that fixed the issue to make back-portin=
g them easier, but I won=E2=80=99t have time to do a binary search anyt=
ime soon.
>
> Thanks for looking in to this again. If anyone else is interested in =
identifying what specifically fixed the issue please let me know if I c=
an do anything to help.
>
John,
This commit may work for you.
commit 4114c27d450bef228be9c7b0c40a888e18a3a636
Author: Wei Wang <wei.w.wang@intel.com>
Date: Wed Nov 5 10:53:43 2014 +0800
KVM: x86: reset RVI upon system reset
Thanks
Wincy
Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
---
### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
**Summary:**
Multi-cores FreeBSD guests hang during reboot under KVM on systems with Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The same configuration works correctly on Ice Lake (Xeon Gold 6338).
---
#### **Environment**
**Host:**
* OS: Debian 12 (Bookworm)
* Kernel versions tested: 6.1.118 and 6.12.47 (both affected)
* QEMU versions tested: 8.2.10 and 9.2.4
* Firmware: SeaBIOS 1.16.2-1 and OVMF 2024.11.1
* CPU: Intel(R) Xeon(R) 6710E (SierraForest)
* KVM module: `kvm_intel`
* Command-line (simplified example):
```
qemu-system-x86_64 -m 2048 -enable-kvm -cpu host -smp 4 -hda freebsd14.img
```
**Guest:**
* OS: FreeBSD 14.3-RELEASE
* SMP guests multi-cores
---
#### **Steps to Reproduce**
1. Start a FreeBSD 14.3 guest under KVM on a SierraForest host.
2. Log in and run:
```
# reboot
```
or press `Ctrl+Alt+Del`.
3. Observe that the VM hangs during reboot — it never returns to BIOS/UEFI.
---
#### **Expected Result**
FreeBSD should reboot cleanly and return to login prompt.
---
#### **Actual Result**
Guest hangs indefinitely during reboot.
QEMU remains running but guest output is frozen.
No host kernel warnings or errors in `dmesg`.
---
#### **Workaround**
Disable APICv on the host before starting QEMU:
```
modprobe kvm_intel enable_apicv=N
```
With `enable_apicv=N`, the FreeBSD guest reboots normally every time.
---
#### **Additional Information**
* The issue **does not reproduce** on an Ice Lake host (Intel Xeon Gold 6338) with identical kernel, QEMU, and guest image.
* Collected `trace-cmd` data for `kvm*` events during both:
* **Good reboot** (with `enable_apicv=N`)
* **Bad reboot** (default APICv enabled)
Traces are available for analysis upon request.
---
#### **Preliminary Analysis**
Disabling APICv avoids the hang, suggesting the problem is related to APIC virtualization on SierraForest.
Possible causes:
* Regression in APICv or posted-interrupt handling on new Xeon platforms.
* Microcode or MSR-related difference in APICv behavior on SierraForest.
* Incorrect EOI or interrupt delivery sequence during guest reboot.
---
#### **Reproducibility**
* 100% reproducible on SierraForest systems
* Not reproducible on Ice Lake
---
#### **Request for Guidance**
* Is this a known KVM or Intel virtualization issue on SierraForest?
* Would you like `trace-cmd` traces, MSR dumps, or `perf kvm` data for further diagnosis?
* Is there a kernel parameter or patch to selectively disable APICv on affected models?
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU
2015-06-25 2:40 ` Wincy Van
2025-10-08 12:42 ` Jack Wang
2025-10-08 12:44 ` Jack Wang
@ 2025-10-08 12:54 ` Jinpu Wang
2025-10-08 15:55 ` Sean Christopherson
2 siblings, 1 reply; 25+ messages in thread
From: Jinpu Wang @ 2025-10-08 12:54 UTC (permalink / raw)
To: fanwenyi0529, John Nielsen; +Cc: bsd, gleb, kvm, Paolo Bonzini, Xiaoyao Li
[-- Attachment #1: Type: text/plain, Size: 5273 bytes --]
On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote:
>
> From: Wincy Van <fanwenyi0529@gmail.com>
>
> On Thu, Jun 25, 2015 at 1:07 AM, John Nielsen <lists@jnielsen.net> wrot=
> e:
> > Interesting. Using the same PC-BSD image I am able to reproduce on a =
> server running slightly older software but I can not reproduce running =
> bleeding edge. I verified enable_apicv=3DY on both. In both cases I ran
> > qemu-kvm -drive if=3Dvirtio,file=3DPCBSD10.1.2-x64-trueos-server.raw =
> -smp 2 -vnc 0.0.0.0:0
> >
> > Specifically:
> >
> > Breaks (VM hangs during boot after pressing ctrl-alt-del):
> > kernel 3.12.22
> > qemu-kvm-1.7.0-3.el6.x86_64
> > seabios-1.7.3.1-1.el6.noarch
> > Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
> >
> > Works (VM reboots normally):
> > kernel 4.0.4
> > qemu-kvm-2.3.0-6.el7.centos.x86_64
> > seabios-bin-1.8.1-1.el7.centos.noarch
> > Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
> >
> >
> > Unfortunately I no longer have the test environment I used a few days=
> ago to reproduce this issue so I can=E2=80=99t verify the software ver=
> sions that were in use. It=E2=80=99s possible I was mistaken about the =
> kernel version (I thought it was 4.0.4). Perhaps it really is fixed in =
> the newer kernel? In any case, this is great news! I would be intereste=
> d in identifying the patch(es) that fixed the issue to make back-portin=
> g them easier, but I won=E2=80=99t have time to do a binary search anyt=
> ime soon.
> >
> > Thanks for looking in to this again. If anyone else is interested in =
> identifying what specifically fixed the issue please let me know if I c=
> an do anything to help.
> >
>
> John,
>
> This commit may work for you.
>
> commit 4114c27d450bef228be9c7b0c40a888e18a3a636
> Author: Wei Wang <wei.w.wang@intel.com>
> Date: Wed Nov 5 10:53:43 2014 +0800
>
> KVM: x86: reset RVI upon system reset
>
>
> Thanks
> Wincy
>
> Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
> machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
>
> ---
>
> ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
>
> **Summary:**
> Multi-cores FreeBSD guests hang during reboot under KVM on systems with Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The same configuration works correctly on Ice Lake (Xeon Gold 6338).
>
> ---
>
> #### **Environment**
>
> **Host:**
>
> * OS: Debian 12 (Bookworm)
> * Kernel versions tested: 6.1.118 and 6.12.47 (both affected)
> * QEMU versions tested: 8.2.10 and 9.2.4
> * Firmware: SeaBIOS 1.16.2-1 and OVMF 2024.11.1
> * CPU: Intel(R) Xeon(R) 6710E (SierraForest)
> * KVM module: `kvm_intel`
> * Command-line (simplified example):
>
> ```
> qemu-system-x86_64 -m 2048 -enable-kvm -cpu host -smp 4 -hda freebsd14.img
> ```
>
> **Guest:**
>
> * OS: FreeBSD 14.3-RELEASE
> * SMP guests multi-cores
>
> ---
>
> #### **Steps to Reproduce**
>
> 1. Start a FreeBSD 14.3 guest under KVM on a SierraForest host.
> 2. Log in and run:
>
> ```
> # reboot
> ```
>
> or press `Ctrl+Alt+Del`.
> 3. Observe that the VM hangs during reboot — it never returns to BIOS/UEFI.
>
> ---
>
> #### **Expected Result**
>
> FreeBSD should reboot cleanly and return to login prompt.
>
> ---
>
> #### **Actual Result**
>
> Guest hangs indefinitely during reboot.
> QEMU remains running but guest output is frozen.
> No host kernel warnings or errors in `dmesg`.
>
> ---
>
> #### **Workaround**
>
> Disable APICv on the host before starting QEMU:
>
> ```
> modprobe kvm_intel enable_apicv=N
> ```
>
> With `enable_apicv=N`, the FreeBSD guest reboots normally every time.
>
> ---
>
> #### **Additional Information**
>
> * The issue **does not reproduce** on an Ice Lake host (Intel Xeon Gold 6338) with identical kernel, QEMU, and guest image.
> * Collected `trace-cmd` data for `kvm*` events during both:
>
> * **Good reboot** (with `enable_apicv=N`)
> * **Bad reboot** (default APICv enabled)
>
> Traces are available for analysis upon request.
>
> ---
>
> #### **Preliminary Analysis**
>
> Disabling APICv avoids the hang, suggesting the problem is related to APIC virtualization on SierraForest.
> Possible causes:
>
> * Regression in APICv or posted-interrupt handling on new Xeon platforms.
> * Microcode or MSR-related difference in APICv behavior on SierraForest.
> * Incorrect EOI or interrupt delivery sequence during guest reboot.
>
> ---
>
> #### **Reproducibility**
>
> * 100% reproducible on SierraForest systems
> * Not reproducible on Ice Lake
>
> ---
>
> #### **Request for Guidance**
>
> * Is this a known KVM or Intel virtualization issue on SierraForest?
> * Would you like `trace-cmd` traces, MSR dumps, or `perf kvm` data for further diagnosis?
> * Is there a kernel parameter or patch to selectively disable APICv on affected models?
I've attached report from "trace-cmd trace-cmd record -e kvm -e
kvm_mmio -e kvm_userspace_exit" in good boot and bad reboot cases,
maybe folks could find something interesting.
[-- Attachment #2: SRF_bug.tar.xz --]
[-- Type: application/x-xz, Size: 3784793 bytes --]
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU
2025-10-08 12:54 ` Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU Jinpu Wang
@ 2025-10-08 15:55 ` Sean Christopherson
2025-10-09 3:44 ` Jinpu Wang
0 siblings, 1 reply; 25+ messages in thread
From: Sean Christopherson @ 2025-10-08 15:55 UTC (permalink / raw)
To: Jinpu Wang; +Cc: fanwenyi0529, kvm, Paolo Bonzini, Xiaoyao Li, linux-kernel
Trimmed Cc: to drop people from the original thread. In the future, just start
a new bug report. Piggybacking a 10 year old bug just because the symptoms are
similar does more harm than good. Whatever the old thread was chasing was already
fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed
the problem. I.e. whatever they were chasing _can't_ be the same root cause,
because even if it's literally the same code bug, it would require a code change
and thus a regression between v4.0 and v6.1.
On Wed, Oct 08, 2025, Jinpu Wang wrote:
> On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote:
> > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
> > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
Are there any host kernels that _do_ work? E.g. have you tried a bleeding edge
host kernel?
> > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
> >
> > **Summary:**
> > Multi-cores FreeBSD guests hang during reboot under KVM on systems with
> > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with
> > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The
> > same configuration works correctly on Ice Lake (Xeon Gold 6338).
Does Sierra Forest have IPI virtualization? If so, you could try running with
APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU
2025-10-08 15:55 ` Sean Christopherson
@ 2025-10-09 3:44 ` Jinpu Wang
2025-10-09 11:21 ` Jinpu Wang
0 siblings, 1 reply; 25+ messages in thread
From: Jinpu Wang @ 2025-10-09 3:44 UTC (permalink / raw)
To: Sean Christopherson
Cc: fanwenyi0529, kvm, Paolo Bonzini, Xiaoyao Li, linux-kernel
Hi Sean,
On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@google.com> wrote:
>
> Trimmed Cc: to drop people from the original thread. In the future, just start
> a new bug report. Piggybacking a 10 year old bug just because the symptoms are
> similar does more harm than good. Whatever the old thread was chasing was already
> fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed
> the problem. I.e. whatever they were chasing _can't_ be the same root cause,
> because even if it's literally the same code bug, it would require a code change
> and thus a regression between v4.0 and v6.1.
Thx for the reply, it makes sense. I will remember this next time.
>
> On Wed, Oct 08, 2025, Jinpu Wang wrote:
> > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote:
> > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
> > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
>
> Are there any host kernels that _do_ work? E.g. have you tried a bleeding edge
> host kernel?
I will try linus/master today.
>
> > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
> > >
> > > **Summary:**
> > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with
> > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with
> > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The
> > > same configuration works correctly on Ice Lake (Xeon Gold 6338).
>
> Does Sierra Forest have IPI virtualization? If so, you could try running with
> APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization.
Yes, it does:
$ grep . /sys/module/kvm_intel/parameters/*
/sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N
/sys/module/kvm_intel/parameters/dump_invalid_vmcs:N
/sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
/sys/module/kvm_intel/parameters/enable_apicv:Y
/sys/module/kvm_intel/parameters/enable_ipiv:Y
/sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y
/sys/module/kvm_intel/parameters/ept:Y
/sys/module/kvm_intel/parameters/eptad:Y
/sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y
/sys/module/kvm_intel/parameters/fasteoi:Y
/sys/module/kvm_intel/parameters/flexpriority:Y
/sys/module/kvm_intel/parameters/nested:Y
/sys/module/kvm_intel/parameters/nested_early_check:N
/sys/module/kvm_intel/parameters/ple_gap:128
/sys/module/kvm_intel/parameters/ple_window:4096
/sys/module/kvm_intel/parameters/ple_window_grow:2
/sys/module/kvm_intel/parameters/ple_window_max:4294967295
/sys/module/kvm_intel/parameters/ple_window_shrink:0
/sys/module/kvm_intel/parameters/pml:Y
/sys/module/kvm_intel/parameters/preemption_timer:Y
/sys/module/kvm_intel/parameters/sgx:N
/sys/module/kvm_intel/parameters/unrestricted_guest:Y
/sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required
/sys/module/kvm_intel/parameters/vnmi:Y
/sys/module/kvm_intel/parameters/vpid:Y
I tried to disable ipiv, but it doesn't help. freebsd hang on reboot.
sudo modprobe -r kvm_intel
sudo modprobe kvm_intel enable_ipiv=N
/sys/module/kvm_intel/parameters/enable_ipiv:N
Thx!
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU
2025-10-09 3:44 ` Jinpu Wang
@ 2025-10-09 11:21 ` Jinpu Wang
2025-10-09 12:36 ` Jinpu Wang
0 siblings, 1 reply; 25+ messages in thread
From: Jinpu Wang @ 2025-10-09 11:21 UTC (permalink / raw)
To: Sean Christopherson
Cc: fanwenyi0529, kvm, Paolo Bonzini, Xiaoyao Li, linux-kernel,
vkuznets
On Thu, Oct 9, 2025 at 5:44 AM Jinpu Wang <jinpu.wang@ionos.com> wrote:
>
> Hi Sean,
>
> On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@google.com> wrote:
> >
> > Trimmed Cc: to drop people from the original thread. In the future, just start
> > a new bug report. Piggybacking a 10 year old bug just because the symptoms are
> > similar does more harm than good. Whatever the old thread was chasing was already
> > fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed
> > the problem. I.e. whatever they were chasing _can't_ be the same root cause,
> > because even if it's literally the same code bug, it would require a code change
> > and thus a regression between v4.0 and v6.1.
> Thx for the reply, it makes sense. I will remember this next time.
> >
> > On Wed, Oct 08, 2025, Jinpu Wang wrote:
> > > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote:
> > > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
> > > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
> >
> > Are there any host kernels that _do_ work? E.g. have you tried a bleeding edge
> > host kernel?
> I will try linus/master today.
> >
> > > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
> > > >
> > > > **Summary:**
> > > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with
> > > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with
> > > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The
> > > > same configuration works correctly on Ice Lake (Xeon Gold 6338).
> >
> > Does Sierra Forest have IPI virtualization? If so, you could try running with
> > APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization.
> Yes, it does:
> $ grep . /sys/module/kvm_intel/parameters/*
> /sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N
> /sys/module/kvm_intel/parameters/dump_invalid_vmcs:N
> /sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
> /sys/module/kvm_intel/parameters/enable_apicv:Y
> /sys/module/kvm_intel/parameters/enable_ipiv:Y
> /sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y
> /sys/module/kvm_intel/parameters/ept:Y
> /sys/module/kvm_intel/parameters/eptad:Y
> /sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y
> /sys/module/kvm_intel/parameters/fasteoi:Y
> /sys/module/kvm_intel/parameters/flexpriority:Y
> /sys/module/kvm_intel/parameters/nested:Y
> /sys/module/kvm_intel/parameters/nested_early_check:N
> /sys/module/kvm_intel/parameters/ple_gap:128
> /sys/module/kvm_intel/parameters/ple_window:4096
> /sys/module/kvm_intel/parameters/ple_window_grow:2
> /sys/module/kvm_intel/parameters/ple_window_max:4294967295
> /sys/module/kvm_intel/parameters/ple_window_shrink:0
> /sys/module/kvm_intel/parameters/pml:Y
> /sys/module/kvm_intel/parameters/preemption_timer:Y
> /sys/module/kvm_intel/parameters/sgx:N
> /sys/module/kvm_intel/parameters/unrestricted_guest:Y
> /sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required
> /sys/module/kvm_intel/parameters/vnmi:Y
> /sys/module/kvm_intel/parameters/vpid:Y
>
> I tried to disable ipiv, but it doesn't help. freebsd hang on reboot.
> sudo modprobe -r kvm_intel
> sudo modprobe kvm_intel enable_ipiv=N
> /sys/module/kvm_intel/parameters/enable_ipiv:N
>
> Thx!
+cc Vitaly
Sorry, I missed one detail, we are use hyper-V enlightment features:
"+hv-relaxed,+hv-vapic,+hv-time,+hv-runtime,hv-spinlocks=0x1fff,+hv-vpindex,+hv-synic,+hv-stimer,+hv-tlbflush,hv-ipi."
did a lot tests with different features, and looks the hang is related
to +hv-synic,+hv-stimer. hv-synic seems the key which causes boot
hang of Freebsd 14.
But the problem seems fixed with FreeBSD 15? I guess it's this fix:
https://github.com/systemd/systemd/issues/28001
Seems it's a bug from freebsd side, rather than on kvm side to me, but
I'm puzzled by disable apicv helps?
Thx!
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU
2025-10-09 11:21 ` Jinpu Wang
@ 2025-10-09 12:36 ` Jinpu Wang
2025-10-09 12:52 ` Vitaly Kuznetsov
0 siblings, 1 reply; 25+ messages in thread
From: Jinpu Wang @ 2025-10-09 12:36 UTC (permalink / raw)
To: Sean Christopherson
Cc: fanwenyi0529, kvm, Paolo Bonzini, Xiaoyao Li, linux-kernel,
vkuznets
On Thu, Oct 9, 2025 at 1:21 PM Jinpu Wang <jinpu.wang@ionos.com> wrote:
>
> On Thu, Oct 9, 2025 at 5:44 AM Jinpu Wang <jinpu.wang@ionos.com> wrote:
> >
> > Hi Sean,
> >
> > On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@google.com> wrote:
> > >
> > > Trimmed Cc: to drop people from the original thread. In the future, just start
> > > a new bug report. Piggybacking a 10 year old bug just because the symptoms are
> > > similar does more harm than good. Whatever the old thread was chasing was already
> > > fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed
> > > the problem. I.e. whatever they were chasing _can't_ be the same root cause,
> > > because even if it's literally the same code bug, it would require a code change
> > > and thus a regression between v4.0 and v6.1.
> > Thx for the reply, it makes sense. I will remember this next time.
> > >
> > > On Wed, Oct 08, 2025, Jinpu Wang wrote:
> > > > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote:
> > > > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
> > > > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
> > >
> > > Are there any host kernels that _do_ work? E.g. have you tried a bleeding edge
> > > host kernel?
> > I will try linus/master today.
> > >
> > > > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
> > > > >
> > > > > **Summary:**
> > > > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with
> > > > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with
> > > > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The
> > > > > same configuration works correctly on Ice Lake (Xeon Gold 6338).
> > >
> > > Does Sierra Forest have IPI virtualization? If so, you could try running with
> > > APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization.
> > Yes, it does:
> > $ grep . /sys/module/kvm_intel/parameters/*
> > /sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N
> > /sys/module/kvm_intel/parameters/dump_invalid_vmcs:N
> > /sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
> > /sys/module/kvm_intel/parameters/enable_apicv:Y
> > /sys/module/kvm_intel/parameters/enable_ipiv:Y
> > /sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y
> > /sys/module/kvm_intel/parameters/ept:Y
> > /sys/module/kvm_intel/parameters/eptad:Y
> > /sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y
> > /sys/module/kvm_intel/parameters/fasteoi:Y
> > /sys/module/kvm_intel/parameters/flexpriority:Y
> > /sys/module/kvm_intel/parameters/nested:Y
> > /sys/module/kvm_intel/parameters/nested_early_check:N
> > /sys/module/kvm_intel/parameters/ple_gap:128
> > /sys/module/kvm_intel/parameters/ple_window:4096
> > /sys/module/kvm_intel/parameters/ple_window_grow:2
> > /sys/module/kvm_intel/parameters/ple_window_max:4294967295
> > /sys/module/kvm_intel/parameters/ple_window_shrink:0
> > /sys/module/kvm_intel/parameters/pml:Y
> > /sys/module/kvm_intel/parameters/preemption_timer:Y
> > /sys/module/kvm_intel/parameters/sgx:N
> > /sys/module/kvm_intel/parameters/unrestricted_guest:Y
> > /sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required
> > /sys/module/kvm_intel/parameters/vnmi:Y
> > /sys/module/kvm_intel/parameters/vpid:Y
> >
> > I tried to disable ipiv, but it doesn't help. freebsd hang on reboot.
> > sudo modprobe -r kvm_intel
> > sudo modprobe kvm_intel enable_ipiv=N
> > /sys/module/kvm_intel/parameters/enable_ipiv:N
> >
> > Thx!
> +cc Vitaly
> Sorry, I missed one detail, we are use hyper-V enlightment features:
> "+hv-relaxed,+hv-vapic,+hv-time,+hv-runtime,hv-spinlocks=0x1fff,+hv-vpindex,+hv-synic,+hv-stimer,+hv-tlbflush,hv-ipi."
>
> did a lot tests with different features, and looks the hang is related
> to +hv-synic,+hv-stimer. hv-synic seems the key which causes boot
> hang of Freebsd 14.
>
> But the problem seems fixed with FreeBSD 15? I guess it's this fix:
https://reviews.freebsd.org/D43508
>
> Seems it's a bug from freebsd side, rather than on kvm side to me, but
> I'm puzzled by disable apicv helps?
>
> Thx!
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU
2025-10-09 12:36 ` Jinpu Wang
@ 2025-10-09 12:52 ` Vitaly Kuznetsov
2025-10-10 5:04 ` Jinpu Wang
0 siblings, 1 reply; 25+ messages in thread
From: Vitaly Kuznetsov @ 2025-10-09 12:52 UTC (permalink / raw)
To: Jinpu Wang, Sean Christopherson
Cc: fanwenyi0529, kvm, Paolo Bonzini, Xiaoyao Li, linux-kernel
Jinpu Wang <jinpu.wang@ionos.com> writes:
> On Thu, Oct 9, 2025 at 1:21 PM Jinpu Wang <jinpu.wang@ionos.com> wrote:
>>
>> On Thu, Oct 9, 2025 at 5:44 AM Jinpu Wang <jinpu.wang@ionos.com> wrote:
>> >
>> > Hi Sean,
>> >
>> > On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@google.com> wrote:
>> > >
>> > > Trimmed Cc: to drop people from the original thread. In the future, just start
>> > > a new bug report. Piggybacking a 10 year old bug just because the symptoms are
>> > > similar does more harm than good. Whatever the old thread was chasing was already
>> > > fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed
>> > > the problem. I.e. whatever they were chasing _can't_ be the same root cause,
>> > > because even if it's literally the same code bug, it would require a code change
>> > > and thus a regression between v4.0 and v6.1.
>> > Thx for the reply, it makes sense. I will remember this next time.
>> > >
>> > > On Wed, Oct 08, 2025, Jinpu Wang wrote:
>> > > > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote:
>> > > > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
>> > > > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
>> > >
>> > > Are there any host kernels that _do_ work? E.g. have you tried a bleeding edge
>> > > host kernel?
>> > I will try linus/master today.
>> > >
>> > > > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
>> > > > >
>> > > > > **Summary:**
>> > > > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with
>> > > > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with
>> > > > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The
>> > > > > same configuration works correctly on Ice Lake (Xeon Gold 6338).
>> > >
>> > > Does Sierra Forest have IPI virtualization? If so, you could try running with
>> > > APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization.
>> > Yes, it does:
>> > $ grep . /sys/module/kvm_intel/parameters/*
>> > /sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N
>> > /sys/module/kvm_intel/parameters/dump_invalid_vmcs:N
>> > /sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
>> > /sys/module/kvm_intel/parameters/enable_apicv:Y
>> > /sys/module/kvm_intel/parameters/enable_ipiv:Y
>> > /sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y
>> > /sys/module/kvm_intel/parameters/ept:Y
>> > /sys/module/kvm_intel/parameters/eptad:Y
>> > /sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y
>> > /sys/module/kvm_intel/parameters/fasteoi:Y
>> > /sys/module/kvm_intel/parameters/flexpriority:Y
>> > /sys/module/kvm_intel/parameters/nested:Y
>> > /sys/module/kvm_intel/parameters/nested_early_check:N
>> > /sys/module/kvm_intel/parameters/ple_gap:128
>> > /sys/module/kvm_intel/parameters/ple_window:4096
>> > /sys/module/kvm_intel/parameters/ple_window_grow:2
>> > /sys/module/kvm_intel/parameters/ple_window_max:4294967295
>> > /sys/module/kvm_intel/parameters/ple_window_shrink:0
>> > /sys/module/kvm_intel/parameters/pml:Y
>> > /sys/module/kvm_intel/parameters/preemption_timer:Y
>> > /sys/module/kvm_intel/parameters/sgx:N
>> > /sys/module/kvm_intel/parameters/unrestricted_guest:Y
>> > /sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required
>> > /sys/module/kvm_intel/parameters/vnmi:Y
>> > /sys/module/kvm_intel/parameters/vpid:Y
>> >
>> > I tried to disable ipiv, but it doesn't help. freebsd hang on reboot.
>> > sudo modprobe -r kvm_intel
>> > sudo modprobe kvm_intel enable_ipiv=N
>> > /sys/module/kvm_intel/parameters/enable_ipiv:N
>> >
>> > Thx!
>> +cc Vitaly
>> Sorry, I missed one detail, we are use hyper-V enlightment features:
>> "+hv-relaxed,+hv-vapic,+hv-time,+hv-runtime,hv-spinlocks=0x1fff,+hv-vpindex,+hv-synic,+hv-stimer,+hv-tlbflush,hv-ipi."
>>
>> did a lot tests with different features, and looks the hang is related
>> to +hv-synic,+hv-stimer. hv-synic seems the key which causes boot
>> hang of Freebsd 14.
>>
>> But the problem seems fixed with FreeBSD 15? I guess it's this fix:
> https://reviews.freebsd.org/D43508
>
>>
>> Seems it's a bug from freebsd side, rather than on kvm side to me, but
>> I'm puzzled by disable apicv helps?
In theory, FreeBSD should work well even if KVM is misdetected as
genuine Hyper-V. Apparently, our emulation is not 1:1 and there are
subtle differences which cause the hang. I did not look at FreeBSD code
at all but my wild guess is that SynIC/stimer are not disabled properly
upon reboot and this causes the problem. If we somehow manage to find
how genuine Hyper-V's behavior is different, it would make sense to
update KVM/QEMU to match.
--
Vitaly
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU
2025-10-09 12:52 ` Vitaly Kuznetsov
@ 2025-10-10 5:04 ` Jinpu Wang
0 siblings, 0 replies; 25+ messages in thread
From: Jinpu Wang @ 2025-10-10 5:04 UTC (permalink / raw)
To: Vitaly Kuznetsov
Cc: Sean Christopherson, fanwenyi0529, kvm, Paolo Bonzini, Xiaoyao Li,
linux-kernel
On Thu, Oct 9, 2025 at 2:53 PM Vitaly Kuznetsov <vkuznets@redhat.com> wrote:
>
> Jinpu Wang <jinpu.wang@ionos.com> writes:
>
> > On Thu, Oct 9, 2025 at 1:21 PM Jinpu Wang <jinpu.wang@ionos.com> wrote:
> >>
> >> On Thu, Oct 9, 2025 at 5:44 AM Jinpu Wang <jinpu.wang@ionos.com> wrote:
> >> >
> >> > Hi Sean,
> >> >
> >> > On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@google.com> wrote:
> >> > >
> >> > > Trimmed Cc: to drop people from the original thread. In the future, just start
> >> > > a new bug report. Piggybacking a 10 year old bug just because the symptoms are
> >> > > similar does more harm than good. Whatever the old thread was chasing was already
> >> > > fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed
> >> > > the problem. I.e. whatever they were chasing _can't_ be the same root cause,
> >> > > because even if it's literally the same code bug, it would require a code change
> >> > > and thus a regression between v4.0 and v6.1.
> >> > Thx for the reply, it makes sense. I will remember this next time.
> >> > >
> >> > > On Wed, Oct 08, 2025, Jinpu Wang wrote:
> >> > > > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote:
> >> > > > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
> >> > > > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
> >> > >
> >> > > Are there any host kernels that _do_ work? E.g. have you tried a bleeding edge
> >> > > host kernel?
> >> > I will try linus/master today.
> >> > >
> >> > > > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
> >> > > > >
> >> > > > > **Summary:**
> >> > > > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with
> >> > > > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with
> >> > > > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The
> >> > > > > same configuration works correctly on Ice Lake (Xeon Gold 6338).
> >> > >
> >> > > Does Sierra Forest have IPI virtualization? If so, you could try running with
> >> > > APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization.
> >> > Yes, it does:
> >> > $ grep . /sys/module/kvm_intel/parameters/*
> >> > /sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N
> >> > /sys/module/kvm_intel/parameters/dump_invalid_vmcs:N
> >> > /sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
> >> > /sys/module/kvm_intel/parameters/enable_apicv:Y
> >> > /sys/module/kvm_intel/parameters/enable_ipiv:Y
> >> > /sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y
> >> > /sys/module/kvm_intel/parameters/ept:Y
> >> > /sys/module/kvm_intel/parameters/eptad:Y
> >> > /sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y
> >> > /sys/module/kvm_intel/parameters/fasteoi:Y
> >> > /sys/module/kvm_intel/parameters/flexpriority:Y
> >> > /sys/module/kvm_intel/parameters/nested:Y
> >> > /sys/module/kvm_intel/parameters/nested_early_check:N
> >> > /sys/module/kvm_intel/parameters/ple_gap:128
> >> > /sys/module/kvm_intel/parameters/ple_window:4096
> >> > /sys/module/kvm_intel/parameters/ple_window_grow:2
> >> > /sys/module/kvm_intel/parameters/ple_window_max:4294967295
> >> > /sys/module/kvm_intel/parameters/ple_window_shrink:0
> >> > /sys/module/kvm_intel/parameters/pml:Y
> >> > /sys/module/kvm_intel/parameters/preemption_timer:Y
> >> > /sys/module/kvm_intel/parameters/sgx:N
> >> > /sys/module/kvm_intel/parameters/unrestricted_guest:Y
> >> > /sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required
> >> > /sys/module/kvm_intel/parameters/vnmi:Y
> >> > /sys/module/kvm_intel/parameters/vpid:Y
> >> >
> >> > I tried to disable ipiv, but it doesn't help. freebsd hang on reboot.
> >> > sudo modprobe -r kvm_intel
> >> > sudo modprobe kvm_intel enable_ipiv=N
> >> > /sys/module/kvm_intel/parameters/enable_ipiv:N
> >> >
> >> > Thx!
> >> +cc Vitaly
> >> Sorry, I missed one detail, we are use hyper-V enlightment features:
> >> "+hv-relaxed,+hv-vapic,+hv-time,+hv-runtime,hv-spinlocks=0x1fff,+hv-vpindex,+hv-synic,+hv-stimer,+hv-tlbflush,hv-ipi."
> >>
> >> did a lot tests with different features, and looks the hang is related
> >> to +hv-synic,+hv-stimer. hv-synic seems the key which causes boot
> >> hang of Freebsd 14.
> >>
> >> But the problem seems fixed with FreeBSD 15? I guess it's this fix:
> > https://reviews.freebsd.org/D43508
> >
> >>
> >> Seems it's a bug from freebsd side, rather than on kvm side to me, but
> >> I'm puzzled by disable apicv helps?
>
> In theory, FreeBSD should work well even if KVM is misdetected as
> genuine Hyper-V. Apparently, our emulation is not 1:1 and there are
> subtle differences which cause the hang. I did not look at FreeBSD code
> at all but my wild guess is that SynIC/stimer are not disabled properly
> upon reboot and this causes the problem. If we somehow manage to find
> how genuine Hyper-V's behavior is different, it would make sense to
> update KVM/QEMU to match.
>
> --
> Vitaly
Hi Vitaly,
Thx for the reply, that's another view.
I spent a few hours to build freebsd stable/14 with commit
6744fd8e7503 ("x86/cpu: improve hypervisor detection") applied, the
bug is fixed, no boot hung, and boot is much faster ~ 40 seconds, as
Freebsd 15, without the fix Freebsd takes 150 seconds if sucessfully
boots, and often hang forever during reboot.
Jinpu
>
^ permalink raw reply [flat|nested] 25+ messages in thread
end of thread, other threads:[~2025-10-10 5:04 UTC | newest]
Thread overview: 25+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <6DBD3DBB-24B1-4564-B524-E8E73508BBC5@jnielsen.net>
[not found] ` <42870B81-CA29-4161-9BCE-F6D6020C3D2C@jnielsen.net>
[not found] ` <539F1DC0.4020604@redhat.com>
[not found] ` <4F14D859-D641-4AB5-B749-83D9D82F1DEA@jnielsen.net>
2014-06-17 4:21 ` Hang on reboot in FreeBSD guest on Linux KVM host Paolo Bonzini
2014-06-17 6:05 ` Gleb Natapov
2014-06-17 16:48 ` John Nielsen
2015-06-22 18:02 ` John Nielsen
2015-06-22 21:48 ` Bandan Das
2015-06-22 22:08 ` John Nielsen
2015-06-23 8:14 ` Paolo Bonzini
2015-06-24 15:50 ` Paolo Bonzini
2015-06-24 15:57 ` Bandan Das
2015-06-24 15:58 ` Paolo Bonzini
2015-06-24 17:07 ` John Nielsen
2015-06-24 17:15 ` Paolo Bonzini
2015-06-25 2:40 ` Wincy Van
2025-10-08 12:42 ` Jack Wang
2025-10-08 12:44 ` Jack Wang
2025-10-08 12:54 ` Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU Jinpu Wang
2025-10-08 15:55 ` Sean Christopherson
2025-10-09 3:44 ` Jinpu Wang
2025-10-09 11:21 ` Jinpu Wang
2025-10-09 12:36 ` Jinpu Wang
2025-10-09 12:52 ` Vitaly Kuznetsov
2025-10-10 5:04 ` Jinpu Wang
2014-06-20 15:41 ` Hang on reboot in FreeBSD guest on Linux KVM host John Nielsen
2014-06-20 19:53 ` Paolo Bonzini
2014-06-20 21:39 ` John Nielsen
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).