Re: Hang on reboot in multi-core FreeBSD guest on Linux KVM host with Intel Sierra Forest CPU

[Vitaly Kuznetsov <vkuznets@redhat.com>]

Jinpu Wang <jinpu.wang@ionos.com> writes:

> On Thu, Oct 9, 2025 at 1:21 PM Jinpu Wang <jinpu.wang@ionos.com> wrote:
>>
>> On Thu, Oct 9, 2025 at 5:44 AM Jinpu Wang <jinpu.wang@ionos.com> wrote:
>> >
>> > Hi Sean,
>> >
>> > On Wed, Oct 8, 2025 at 5:55 PM Sean Christopherson <seanjc@google.com> wrote:
>> > >
>> > > Trimmed Cc: to drop people from the original thread.  In the future, just start
>> > > a new bug report.  Piggybacking a 10 year old bug just because the symptoms are
>> > > similar does more harm than good.  Whatever the old thread was chasing was already
>> > > fixed, _10 years_ ago; they were just trying to identy exactly what commit fixed
>> > > the problem.  I.e. whatever they were chasing _can't_ be the same root cause,
>> > > because even if it's literally the same code bug, it would require a code change
>> > > and thus a regression between v4.0 and v6.1.
>> > Thx for the reply,  it makes sense. I will remember this next time.
>> > >
>> > > On Wed, Oct 08, 2025, Jinpu Wang wrote:
>> > > > On Wed, Oct 8, 2025 at 2:44 PM Jack Wang <jinpu.wang@ionos.com> wrote:
>> > > > > Sorry for bump this old thread, we hit same issue on Intel Sierra Forest
>> > > > > machines with LTS kernel 6.1/6.12, maybe KVM comunity could help fix it.
>> > >
>> > > Are there any host kernels that _do_ work?  E.g. have you tried a bleeding edge
>> > > host kernel?
>> > I will try linus/master today.
>> > >
>> > > > > ### **[BUG] Hang on FreeBSD Guest Reboot under KVM on Intel SierraForest (Xeon 6710E)**
>> > > > >
>> > > > > **Summary:**
>> > > > > Multi-cores FreeBSD guests hang during reboot under KVM on systems with
>> > > > > Intel(R) Xeon(R) 6710E (SierraForest). The issue is fully reproducible with
>> > > > > APICv enabled and disappears when disabling APICv (`enable_apicv=N`). The
>> > > > > same configuration works correctly on Ice Lake (Xeon Gold 6338).
>> > >
>> > > Does Sierra Forest have IPI virtualization?  If so, you could try running with
>> > > APICv enabled, but enable_ipiv=false to specifically disable IPI virtualization.
>> > Yes, it does:
>> > $  grep . /sys/module/kvm_intel/parameters/*
>> > /sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr:N
>> > /sys/module/kvm_intel/parameters/dump_invalid_vmcs:N
>> > /sys/module/kvm_intel/parameters/emulate_invalid_guest_state:Y
>> > /sys/module/kvm_intel/parameters/enable_apicv:Y
>> > /sys/module/kvm_intel/parameters/enable_ipiv:Y
>> > /sys/module/kvm_intel/parameters/enable_shadow_vmcs:Y
>> > /sys/module/kvm_intel/parameters/ept:Y
>> > /sys/module/kvm_intel/parameters/eptad:Y
>> > /sys/module/kvm_intel/parameters/error_on_inconsistent_vmcs_config:Y
>> > /sys/module/kvm_intel/parameters/fasteoi:Y
>> > /sys/module/kvm_intel/parameters/flexpriority:Y
>> > /sys/module/kvm_intel/parameters/nested:Y
>> > /sys/module/kvm_intel/parameters/nested_early_check:N
>> > /sys/module/kvm_intel/parameters/ple_gap:128
>> > /sys/module/kvm_intel/parameters/ple_window:4096
>> > /sys/module/kvm_intel/parameters/ple_window_grow:2
>> > /sys/module/kvm_intel/parameters/ple_window_max:4294967295
>> > /sys/module/kvm_intel/parameters/ple_window_shrink:0
>> > /sys/module/kvm_intel/parameters/pml:Y
>> > /sys/module/kvm_intel/parameters/preemption_timer:Y
>> > /sys/module/kvm_intel/parameters/sgx:N
>> > /sys/module/kvm_intel/parameters/unrestricted_guest:Y
>> > /sys/module/kvm_intel/parameters/vmentry_l1d_flush:not required
>> > /sys/module/kvm_intel/parameters/vnmi:Y
>> > /sys/module/kvm_intel/parameters/vpid:Y
>> >
>> > I tried to disable ipiv, but it doesn't help. freebsd hang on reboot.
>> > sudo modprobe -r kvm_intel
>> > sudo modprobe  kvm_intel enable_ipiv=N
>> > /sys/module/kvm_intel/parameters/enable_ipiv:N
>> >
>> > Thx!
>> +cc Vitaly
>> Sorry, I missed one detail, we are use hyper-V enlightment features:
>> "+hv-relaxed,+hv-vapic,+hv-time,+hv-runtime,hv-spinlocks=0x1fff,+hv-vpindex,+hv-synic,+hv-stimer,+hv-tlbflush,hv-ipi."
>>
>> did a lot tests with different features, and looks the hang is related
>> to  +hv-synic,+hv-stimer.  hv-synic seems the key which causes boot
>> hang of Freebsd 14.
>>
>> But the problem seems fixed with FreeBSD 15?  I guess it's this fix:
> https://reviews.freebsd.org/D43508
>
>>
>> Seems it's a bug from freebsd side, rather than on kvm side to me, but
>> I'm puzzled by disable apicv helps?

In theory, FreeBSD should work well even if KVM is misdetected as
genuine Hyper-V. Apparently, our emulation is not 1:1 and there are
subtle differences which cause the hang. I did not look at FreeBSD code
at all but my wild guess is that SynIC/stimer are not disabled properly
upon reboot and this causes the problem. If we somehow manage to find
how genuine Hyper-V's behavior is different, it would make sense to
update KVM/QEMU to match.

-- 
Vitaly