HvExtCallQueryCapabilities and HvExtCallGetBootZeroedMemory implementation in KVM for Windows guest

HvExtCallQueryCapabilities and HvExtCallGetBootZeroedMemory implementation in KVM for Windows guest

[Vipin Sharma]

Hi Vitaly, Yury, Sunil, Tianyu

Before I work on a patch series and send it out to the KVM mailing
list, I wanted to check with you a potential Windows VM optimization
and see if you have worked on it or if you know about some obvious
known blockers regarding this feature.

Hypervisor Top-Level Functional Specification v6.0b mentions a hypercall:

    HvExtCallGetBootZeroedMemory
    Call Code = 0x8002

This hypercall can be used by Windows guest to know which pages are
already zeroed and then guest can avoid zeroing them again during the
boot, resulting in Windows VM faster boot time and less memory usage.

KVM currently doesn't implement this feature. I am thinking of
implementing it, here is a rough code flow:
1. KVM will set bit 20 in EBX of CPUID leaf 0x40000003 to let the
Windows guest know that it can use the extended hypercall interface.
2. Guest during the boot will use hypercall HvExtCallQueryCapabilities
(Call Code = 0x8001) to see which extended calls are available.
3. KVM will respond to guest that the hypercall
HvExtCallGetBootZeroedMemory is available.
4. Guest will issue the hypercall HvExtCallGetBootZeroedMemory to know
which pages are zeroed.
5. KVM or userspace VMM will respond with GPA and page count to guest.
6. Guest will skip zeroing these pages, resulting in faster boot and
less memory utilization of guest.

This seems like a very easy win for KVM to increase Windows guest boot
performance but I am not sure if I am overlooking something. If you
are aware of any potential side effects of enabling these hypercalls
or some other issue I am not thinking about please let me know,
otherwise, I can start working on this feature and send RFC patches to
the mailing list.

Thanks
Vipin

Re: HvExtCallQueryCapabilities and HvExtCallGetBootZeroedMemory implementation in KVM for Windows guest

[Vitaly Kuznetsov]

Vipin Sharma <vipinsh@google.com> writes:

> Hi Vitaly, Yury, Sunil, Tianyu

Hi Vipin!

>
> Before I work on a patch series and send it out to the KVM mailing
> list, I wanted to check with you a potential Windows VM optimization
> and see if you have worked on it or if you know about some obvious
> known blockers regarding this feature.
>
> Hypervisor Top-Level Functional Specification v6.0b mentions a hypercall:
>
>     HvExtCallGetBootZeroedMemory
>     Call Code = 0x8002
>
> This hypercall can be used by Windows guest to know which pages are
> already zeroed and then guest can avoid zeroing them again during the
> boot, resulting in Windows VM faster boot time and less memory usage.
>
> KVM currently doesn't implement this feature. I am thinking of
> implementing it, here is a rough code flow:
> 1. KVM will set bit 20 in EBX of CPUID leaf 0x40000003 to let the
> Windows guest know that it can use the extended hypercall interface.
> 2. Guest during the boot will use hypercall HvExtCallQueryCapabilities
> (Call Code = 0x8001) to see which extended calls are available.
> 3. KVM will respond to guest that the hypercall
> HvExtCallGetBootZeroedMemory is available.
> 4. Guest will issue the hypercall HvExtCallGetBootZeroedMemory to know
> which pages are zeroed.
> 5. KVM or userspace VMM will respond with GPA and page count to guest.

I think it's VMM's responsibility. How would KVM know if the memory
allocated to the guest was zeroed or not?

The easiest solution would be to just pass through this hypercall to the
VMM and let it respond. Alternatively, we can probably add a flag to
KVM_SET_USER_MEMORY_REGION to either indicate that the memory is zeroed
or to actually ask KVM to zero it. This way we will have the required
information in KVM. I'm not sure if it's worth it, Windows probably
calls HvExtCallGetBootZeroedMemory just once upon boot so handling it in
the VMM is totally fine.

> 6. Guest will skip zeroing these pages, resulting in faster boot and
> less memory utilization of guest.
>
> This seems like a very easy win for KVM to increase Windows guest boot
> performance but I am not sure if I am overlooking something. If you
> are aware of any potential side effects of enabling these hypercalls
> or some other issue I am not thinking about please let me know,
> otherwise, I can start working on this feature and send RFC patches to
> the mailing list.

I dug through my git archives and found that I've actually tried
HvExtCallQueryCapabilities back in 2018 but for some reason Windows
versions I was testing didn't use it (hope it wasn't some silly mistake
like forgotten CPUID bit on my part :-) so I put it aside and never got
back to it. Thanks for picking this up!

-- 
Vitaly

Re: HvExtCallQueryCapabilities and HvExtCallGetBootZeroedMemory implementation in KVM for Windows guest

[Vipin Sharma]

On Thu, Oct 6, 2022 at 1:49 AM Vitaly Kuznetsov <vkuznets@redhat.com> wrote:
>
> Vipin Sharma <vipinsh@google.com> writes:
>
> > Hi Vitaly, Yury, Sunil, Tianyu
>
> Hi Vipin!
>
> >
> > Before I work on a patch series and send it out to the KVM mailing
> > list, I wanted to check with you a potential Windows VM optimization
> > and see if you have worked on it or if you know about some obvious
> > known blockers regarding this feature.
> >
> > Hypervisor Top-Level Functional Specification v6.0b mentions a hypercall:
> >
> >     HvExtCallGetBootZeroedMemory
> >     Call Code = 0x8002
> >
> > This hypercall can be used by Windows guest to know which pages are
> > already zeroed and then guest can avoid zeroing them again during the
> > boot, resulting in Windows VM faster boot time and less memory usage.
> >
> > KVM currently doesn't implement this feature. I am thinking of
> > implementing it, here is a rough code flow:
> > 1. KVM will set bit 20 in EBX of CPUID leaf 0x40000003 to let the
> > Windows guest know that it can use the extended hypercall interface.
> > 2. Guest during the boot will use hypercall HvExtCallQueryCapabilities
> > (Call Code = 0x8001) to see which extended calls are available.
> > 3. KVM will respond to guest that the hypercall
> > HvExtCallGetBootZeroedMemory is available.
> > 4. Guest will issue the hypercall HvExtCallGetBootZeroedMemory to know
> > which pages are zeroed.
> > 5. KVM or userspace VMM will respond with GPA and page count to guest.
>
> I think it's VMM's responsibility. How would KVM know if the memory
> allocated to the guest was zeroed or not?
>
> The easiest solution would be to just pass through this hypercall to the
> VMM and let it respond. Alternatively, we can probably add a flag to
> KVM_SET_USER_MEMORY_REGION to either indicate that the memory is zeroed
> or to actually ask KVM to zero it. This way we will have the required
> information in KVM. I'm not sure if it's worth it, Windows probably
> calls HvExtCallGetBootZeroedMemory just once upon boot so handling it in
> the VMM is totally fine.

I agree with you. Since, it is probably only upon boot time, handling
in VMM should be okay.

>
> > 6. Guest will skip zeroing these pages, resulting in faster boot and
> > less memory utilization of guest.
> >
> > This seems like a very easy win for KVM to increase Windows guest boot
> > performance but I am not sure if I am overlooking something. If you
> > are aware of any potential side effects of enabling these hypercalls
> > or some other issue I am not thinking about please let me know,
> > otherwise, I can start working on this feature and send RFC patches to
> > the mailing list.
>
> I dug through my git archives and found that I've actually tried
> HvExtCallQueryCapabilities back in 2018 but for some reason Windows
> versions I was testing didn't use it (hope it wasn't some silly mistake
> like forgotten CPUID bit on my part :-) so I put it aside and never got
> back to it. Thanks for picking this up!

Keeping my fingers crossed!

Thanks for the feedback.

>
> --
> Vitaly
>