Nested virtualization not working with hyperv guest/windows 11

Nested virtualization not working with hyperv guest/windows 11

[Michał Zegan]

Hi,

I've sent this some time ago, but was not subscribed here, so unsure if 
I didn't get a reply or maybe missed it, so repeating:

I have a linux host with cpu intel core i7 12700h, kernel currently 6.2, 
fedora37.

I have a kvm/qemu/libvirt virtual machine, cpu model set to host, 
machine type q35, uefi with secureboot enabled, smm on.

The kvm_intel module has nested=y set in parameters so nested 
virtualization is enabled on host.

The virtual machine has windows11 pro guest installed.

When I install hyperv/virtualization platform/other similar functions, 
after reboot, the windows does not boot. Namely it reboots three times 
and then goes to recovery.

The only way to fix the problem is disabling nested virtualization or 
vmx cpu feature, or uninstalling hyperv. However I wanted nested 
virtualization to be working.

Any clue what might be going on and how to fix it or even see the problem?

Re: Nested virtualization not working with hyperv guest/windows 11

[Sean Christopherson]

On Fri, Mar 24, 2023, Michał Zegan wrote:
> Hi,
> 
> I've sent this some time ago, but was not subscribed here, so unsure if I
> didn't get a reply or maybe missed it, so repeating:
> 
> I have a linux host with cpu intel core i7 12700h, kernel currently 6.2,
> fedora37.
> 
> I have a kvm/qemu/libvirt virtual machine, cpu model set to host, machine
> type q35, uefi with secureboot enabled, smm on.
> 
> The kvm_intel module has nested=y set in parameters so nested virtualization
> is enabled on host.
> 
> The virtual machine has windows11 pro guest installed.
> 
> When I install hyperv/virtualization platform/other similar functions, after
> reboot, the windows does not boot. Namely it reboots three times and then
> goes to recovery.

This is going to be nearly impossible to debug without more information.  Assuming
you can't extract more information from the guest, can you try enabling KVM
tracepoints?  E.g. to see if KVM is injecting an exception or a nested VM-Entry
failure that leads to the reboot.

I.e. enable tracing

    echo 1 > /sys/kernel/debug/tracing/tracing_on

and then to get the full blast from the trace firehose:

    echo 1 > /sys/kernel/debug/tracing/events/kvm/enable

or to get slightly less noisy log:

    echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_entry/enable
    echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_exit/enable
    echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_inj_exception/enable
    echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_nested_intercepts/enable
    echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_nested_intr_vmexit/enable
    echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_nested_vmenter_failed/enable
    echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_nested_vmexit/enable
    echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_nested_vmexit_inject/enable
    echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_nested_vmenter/enable

To capture something useful, you may need to (significantly) increase the size of
the buffer, 

    echo 131072 > /sys/kernel/debug/tracing/buffer_size_kb

The log itself can be found at

    /sys/kernel/debug/tracing/trace

Re: Nested virtualization not working with hyperv guest/windows 11

[Michał Zegan]

Hello,

Thanks for the reply.

doesn't qemu actually panic on entry failure? note i don't have crashes 
nor anything in logs, I've tried to enable ignore_msrs in kvm parameters 
to see if maybe this will report something, but no luck.

Tracing data are here: https://transfer.sh/1lfnMc/kvmtrace.gz

Also unsure if I didn't lose some data in the process, although if this 
was actually three reboots, not just instantly going to recovery, it 
shouldn't matter.

I definitely cannot get anything useful from it.

W dniu 24.03.2023 o 15:40, Sean Christopherson pisze:
> On Fri, Mar 24, 2023, Michał Zegan wrote:
>> Hi,
>>
>> I've sent this some time ago, but was not subscribed here, so unsure if I
>> didn't get a reply or maybe missed it, so repeating:
>>
>> I have a linux host with cpu intel core i7 12700h, kernel currently 6.2,
>> fedora37.
>>
>> I have a kvm/qemu/libvirt virtual machine, cpu model set to host, machine
>> type q35, uefi with secureboot enabled, smm on.
>>
>> The kvm_intel module has nested=y set in parameters so nested virtualization
>> is enabled on host.
>>
>> The virtual machine has windows11 pro guest installed.
>>
>> When I install hyperv/virtualization platform/other similar functions, after
>> reboot, the windows does not boot. Namely it reboots three times and then
>> goes to recovery.
> This is going to be nearly impossible to debug without more information.  Assuming
> you can't extract more information from the guest, can you try enabling KVM
> tracepoints?  E.g. to see if KVM is injecting an exception or a nested VM-Entry
> failure that leads to the reboot.
>
> I.e. enable tracing
>
>      echo 1 > /sys/kernel/debug/tracing/tracing_on
>
> and then to get the full blast from the trace firehose:
>
>      echo 1 > /sys/kernel/debug/tracing/events/kvm/enable
>
> or to get slightly less noisy log:
>
>      echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_entry/enable
>      echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_exit/enable
>      echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_inj_exception/enable
>      echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_nested_intercepts/enable
>      echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_nested_intr_vmexit/enable
>      echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_nested_vmenter_failed/enable
>      echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_nested_vmexit/enable
>      echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_nested_vmexit_inject/enable
>      echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_nested_vmenter/enable
>
> To capture something useful, you may need to (significantly) increase the size of
> the buffer,
>
>      echo 131072 > /sys/kernel/debug/tracing/buffer_size_kb
>
> The log itself can be found at
>
>      /sys/kernel/debug/tracing/trace

ping: Re: Nested virtualization not working with hyperv guest/windows 11

[Michał Zegan]

W dniu 24.03.2023 o 15:59, Michał Zegan pisze:
> Hello,
>
> Thanks for the reply.
>
> doesn't qemu actually panic on entry failure? note i don't have 
> crashes nor anything in logs, I've tried to enable ignore_msrs in kvm 
> parameters to see if maybe this will report something, but no luck.
>
> Tracing data are here: https://transfer.sh/1lfnMc/kvmtrace.gz
>
> Also unsure if I didn't lose some data in the process, although if 
> this was actually three reboots, not just instantly going to recovery, 
> it shouldn't matter.
>
> I definitely cannot get anything useful from it.
>
> W dniu 24.03.2023 o 15:40, Sean Christopherson pisze:
>> On Fri, Mar 24, 2023, Michał Zegan wrote:
>>> Hi,
>>>
>>> I've sent this some time ago, but was not subscribed here, so unsure 
>>> if I
>>> didn't get a reply or maybe missed it, so repeating:
>>>
>>> I have a linux host with cpu intel core i7 12700h, kernel currently 
>>> 6.2,
>>> fedora37.
>>>
>>> I have a kvm/qemu/libvirt virtual machine, cpu model set to host, 
>>> machine
>>> type q35, uefi with secureboot enabled, smm on.
>>>
>>> The kvm_intel module has nested=y set in parameters so nested 
>>> virtualization
>>> is enabled on host.
>>>
>>> The virtual machine has windows11 pro guest installed.
>>>
>>> When I install hyperv/virtualization platform/other similar 
>>> functions, after
>>> reboot, the windows does not boot. Namely it reboots three times and 
>>> then
>>> goes to recovery.
>> This is going to be nearly impossible to debug without more 
>> information.  Assuming
>> you can't extract more information from the guest, can you try 
>> enabling KVM
>> tracepoints?  E.g. to see if KVM is injecting an exception or a 
>> nested VM-Entry
>> failure that leads to the reboot.
>>
>> I.e. enable tracing
>>
>>      echo 1 > /sys/kernel/debug/tracing/tracing_on
>>
>> and then to get the full blast from the trace firehose:
>>
>>      echo 1 > /sys/kernel/debug/tracing/events/kvm/enable
>>
>> or to get slightly less noisy log:
>>
>>      echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_entry/enable
>>      echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_exit/enable
>>      echo 1 > 
>> /sys/kernel/debug/tracing/events/kvm/kvm_inj_exception/enable
>>      echo 1 > 
>> /sys/kernel/debug/tracing/events/kvm/kvm_nested_intercepts/enable
>>      echo 1 > 
>> /sys/kernel/debug/tracing/events/kvm/kvm_nested_intr_vmexit/enable
>>      echo 1 > 
>> /sys/kernel/debug/tracing/events/kvm/kvm_nested_vmenter_failed/enable
>>      echo 1 > 
>> /sys/kernel/debug/tracing/events/kvm/kvm_nested_vmexit/enable
>>      echo 1 > 
>> /sys/kernel/debug/tracing/events/kvm/kvm_nested_vmexit_inject/enable
>>      echo 1 > 
>> /sys/kernel/debug/tracing/events/kvm/kvm_nested_vmenter/enable
>>
>> To capture something useful, you may need to (significantly) increase 
>> the size of
>> the buffer,
>>
>>      echo 131072 > /sys/kernel/debug/tracing/buffer_size_kb
>>
>> The log itself can be found at
>>
>>      /sys/kernel/debug/tracing/trace

ping: Re: Nested virtualization not working with hyperv guest/windows 11

[Michał Zegan]

W dniu 30.03.2023 o 18:46, Michał Zegan pisze:
>
> W dniu 24.03.2023 o 15:59, Michał Zegan pisze:
>> Hello,
>>
>> Thanks for the reply.
>>
>> doesn't qemu actually panic on entry failure? note i don't have 
>> crashes nor anything in logs, I've tried to enable ignore_msrs in kvm 
>> parameters to see if maybe this will report something, but no luck.
>>
>> Tracing data are here: https://transfer.sh/1lfnMc/kvmtrace.gz
>>
>> Also unsure if I didn't lose some data in the process, although if 
>> this was actually three reboots, not just instantly going to 
>> recovery, it shouldn't matter.
>>
>> I definitely cannot get anything useful from it.
>>
>> W dniu 24.03.2023 o 15:40, Sean Christopherson pisze:
>>> On Fri, Mar 24, 2023, Michał Zegan wrote:
>>>> Hi,
>>>>
>>>> I've sent this some time ago, but was not subscribed here, so 
>>>> unsure if I
>>>> didn't get a reply or maybe missed it, so repeating:
>>>>
>>>> I have a linux host with cpu intel core i7 12700h, kernel currently 
>>>> 6.2,
>>>> fedora37.
>>>>
>>>> I have a kvm/qemu/libvirt virtual machine, cpu model set to host, 
>>>> machine
>>>> type q35, uefi with secureboot enabled, smm on.
>>>>
>>>> The kvm_intel module has nested=y set in parameters so nested 
>>>> virtualization
>>>> is enabled on host.
>>>>
>>>> The virtual machine has windows11 pro guest installed.
>>>>
>>>> When I install hyperv/virtualization platform/other similar 
>>>> functions, after
>>>> reboot, the windows does not boot. Namely it reboots three times 
>>>> and then
>>>> goes to recovery.
>>> This is going to be nearly impossible to debug without more 
>>> information.  Assuming
>>> you can't extract more information from the guest, can you try 
>>> enabling KVM
>>> tracepoints?  E.g. to see if KVM is injecting an exception or a 
>>> nested VM-Entry
>>> failure that leads to the reboot.
>>>
>>> I.e. enable tracing
>>>
>>>      echo 1 > /sys/kernel/debug/tracing/tracing_on
>>>
>>> and then to get the full blast from the trace firehose:
>>>
>>>      echo 1 > /sys/kernel/debug/tracing/events/kvm/enable
>>>
>>> or to get slightly less noisy log:
>>>
>>>      echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_entry/enable
>>>      echo 1 > /sys/kernel/debug/tracing/events/kvm/kvm_exit/enable
>>>      echo 1 > 
>>> /sys/kernel/debug/tracing/events/kvm/kvm_inj_exception/enable
>>>      echo 1 > 
>>> /sys/kernel/debug/tracing/events/kvm/kvm_nested_intercepts/enable
>>>      echo 1 > 
>>> /sys/kernel/debug/tracing/events/kvm/kvm_nested_intr_vmexit/enable
>>>      echo 1 > 
>>> /sys/kernel/debug/tracing/events/kvm/kvm_nested_vmenter_failed/enable
>>>      echo 1 > 
>>> /sys/kernel/debug/tracing/events/kvm/kvm_nested_vmexit/enable
>>>      echo 1 > 
>>> /sys/kernel/debug/tracing/events/kvm/kvm_nested_vmexit_inject/enable
>>>      echo 1 > 
>>> /sys/kernel/debug/tracing/events/kvm/kvm_nested_vmenter/enable
>>>
>>> To capture something useful, you may need to (significantly) 
>>> increase the size of
>>> the buffer,
>>>
>>>      echo 131072 > /sys/kernel/debug/tracing/buffer_size_kb
>>>
>>> The log itself can be found at
>>>
>>>      /sys/kernel/debug/tracing/trace