* Re: [PATCH] x86/hyperv: Suspend/resume the VP assist page for hibernation
From: Vitaly Kuznetsov @ 2020-04-17 10:03 UTC (permalink / raw)
To: Dexuan Cui
Cc: bp, haiyangz, hpa, kys, linux-hyperv, linux-kernel, mingo,
sthemmin, tglx, x86, mikelley, wei.liu, kvm
In-Reply-To: <1587104999-28927-1-git-send-email-decui@microsoft.com>
Dexuan Cui <decui@microsoft.com> writes:
> Unlike the other CPUs, CPU0 is never offlined during hibernation. So in the
> resume path, the "new" kernel's VP assist page is not suspended (i.e.
> disabled), and later when we jump to the "old" kernel, the page is not
> properly re-enabled for CPU0 with the allocated page from the old kernel.
>
> So far, the VP assist page is only used by hv_apic_eoi_write().
No, not only for that ('git grep hv_get_vp_assist_page')
KVM on Hyper-V also needs VP assist page to use Enlightened VMCS. In
particular, Enlightened VMPTR is written there.
This makes me wonder: how does hibernation work with KVM in case we use
Enlightened VMCS and we have VMs running? We need to make sure VP Assist
page content is preserved.
--
Vitaly
^ permalink raw reply
* Re: [PATCH v2 0/1] x86/kvm/hyper-v: Add support to SYNIC exit on EOM
From: Roman Kagan @ 2020-04-17 10:42 UTC (permalink / raw)
To: Jon Doron; +Cc: kvm, linux-hyperv, vkuznets
In-Reply-To: <20200416125430.GL7606@jondnuc>
On Thu, Apr 16, 2020 at 03:54:30PM +0300, Jon Doron wrote:
> On 16/04/2020, Roman Kagan wrote:
> > On Thu, Apr 16, 2020 at 11:38:46AM +0300, Jon Doron wrote:
> > > According to the TLFS:
> > > "A write to the end of message (EOM) register by the guest causes the
> > > hypervisor to scan the internal message buffer queue(s) associated with
> > > the virtual processor.
> > >
> > > If a message buffer queue contains a queued message buffer, the hypervisor
> > > attempts to deliver the message.
> > >
> > > Message delivery succeeds if the SIM page is enabled and the message slot
> > > corresponding to the SINTx is empty (that is, the message type in the
> > > header is set to HvMessageTypeNone).
> > > If a message is successfully delivered, its corresponding internal message
> > > buffer is dequeued and marked free.
> > > If the corresponding SINTx is not masked, an edge-triggered interrupt is
> > > delivered (that is, the corresponding bit in the IRR is set).
> > >
> > > This register can be used by guests to poll for messages. It can also be
> > > used as a way to drain the message queue for a SINTx that has
> > > been disabled (that is, masked)."
> >
> > Doesn't this work already?
> >
>
> Well if you dont have SCONTROL and a GSI associated with the SINT then it
> does not...
Yes you do need both of these.
> > > So basically this means that we need to exit on EOM so the hypervisor
> > > will have a chance to send all the pending messages regardless of the
> > > SCONTROL mechnaisim.
> >
> > I might be misinterpreting the spec, but my understanding is that
> > SCONTROL {en,dis}ables the message queueing completely. What the quoted
> > part means is that a write to EOM should trigger the message source to
> > push a new message into the slot, regardless of whether the SINT was
> > masked or not.
> >
> > And this (I think, haven't tested) should already work. The userspace
> > just keeps using the SINT route as it normally does, posting
> > notifications to the corresponding irqfd when posting a message, and
> > waiting on the resamplerfd for the message slot to become free. If the
> > SINT is masked KVM will skip injecting the interrupt, that's it.
> >
> > Roman.
>
> That's what I was thinking originally as well, but then i noticed KDNET as a
> VMBus client (and it basically runs before anything else) is working in this
> polling mode, where SCONTROL is disabled and it just loops, and if it saw
> there is a PENDING message flag it will issue an EOM to indicate it has free
> the slot.
Who sets up the message page then? Doesn't it enabe SCONTROL as well?
Note that, even if you don't see it being enabled by Windows, it can be
enabled by the firmware and/or by the bootloader.
Can you perhaps try with the SeaBIOS from
https://src.openvz.org/projects/UP/repos/seabios branch hv-scsi? It
enables SCONTROL and leaves it that way.
I'd also suggest tracing kvm_msr events (both reads and writes) for
SCONTROL and SIMP msrs, to better understand the picture.
So far the change you propose appears too heavy to work around the
problem of disabled SCONTROL. You seem to be better off just making
sure it's enabled (either by the firmware or slighly violating the spec
and initializing to enabled from the start), and sticking to the
existing infrastructure for posting messages.
> (There are a bunch of patches i sent on the QEMU mailing list as well where
> i CCed you, I will probably revise it a bit but was hoping to get KVM
> sorted out first).
I'll look through the archive, should be there, thanks.
Roman.
^ permalink raw reply
* Re: [PATCH] x86/hyperv: Suspend/resume the VP assist page for hibernation
From: Wei Liu @ 2020-04-17 10:55 UTC (permalink / raw)
To: Vitaly Kuznetsov
Cc: Dexuan Cui, bp, haiyangz, hpa, kys, linux-hyperv, linux-kernel,
mingo, sthemmin, tglx, x86, mikelley, wei.liu, kvm
In-Reply-To: <87blnqv389.fsf@vitty.brq.redhat.com>
On Fri, Apr 17, 2020 at 12:03:18PM +0200, Vitaly Kuznetsov wrote:
> Dexuan Cui <decui@microsoft.com> writes:
>
> > Unlike the other CPUs, CPU0 is never offlined during hibernation. So in the
> > resume path, the "new" kernel's VP assist page is not suspended (i.e.
> > disabled), and later when we jump to the "old" kernel, the page is not
> > properly re-enabled for CPU0 with the allocated page from the old kernel.
> >
> > So far, the VP assist page is only used by hv_apic_eoi_write().
>
> No, not only for that ('git grep hv_get_vp_assist_page')
>
> KVM on Hyper-V also needs VP assist page to use Enlightened VMCS. In
> particular, Enlightened VMPTR is written there.
>
> This makes me wonder: how does hibernation work with KVM in case we use
> Enlightened VMCS and we have VMs running? We need to make sure VP Assist
> page content is preserved.
The page itself is preserved, isn't it?
hv_cpu_die never frees the vp_assit page. It merely disables it.
hv_cpu_init only allocates a new page if necessary.
Wei.
>
> --
> Vitaly
>
^ permalink raw reply
* Re: [PATCH] x86/hyperv: Suspend/resume the VP assist page for hibernation
From: Wei Liu @ 2020-04-17 11:00 UTC (permalink / raw)
To: Dexuan Cui
Cc: bp, haiyangz, hpa, kys, linux-hyperv, linux-kernel, mingo,
sthemmin, tglx, x86, mikelley, vkuznets, wei.liu
In-Reply-To: <1587104999-28927-1-git-send-email-decui@microsoft.com>
On Thu, Apr 16, 2020 at 11:29:59PM -0700, Dexuan Cui wrote:
> Unlike the other CPUs, CPU0 is never offlined during hibernation. So in the
> resume path, the "new" kernel's VP assist page is not suspended (i.e.
> disabled), and later when we jump to the "old" kernel, the page is not
> properly re-enabled for CPU0 with the allocated page from the old kernel.
>
> So far, the VP assist page is only used by hv_apic_eoi_write(). When the
> page is not properly re-enabled, hvp->apic_assist is always 0, so the
> HV_X64_MSR_EOI MSR is always written. This is not ideal with respect to
> performance, but Hyper-V can still correctly handle this.
>
> The issue is: the hypervisor can corrupt the old kernel memory, and hence
> sometimes cause unexpected behaviors, e.g. when the old kernel's non-boot
> CPUs are being onlined in the resume path, the VM can hang or be killed
> due to virtual triple fault.
I don't quite follow here.
The first sentence is rather alarming -- why would Hyper-V corrupt
guest's memory (kernel or not)?
Secondly, code below only specifies cpu0. What does it do with non-boot
cpus on the resume path?
Wei.
>
> Fix the issue by calling hv_cpu_die()/hv_cpu_init() in the syscore ops.
>
> Without the fix, hibernation can fail at a rate of 1/300 ~ 1/500.
> With the fix, hibernation can pass a long-haul test of 2000 rounds.
>
> Fixes: 05bd330a7fd8 ("x86/hyperv: Suspend/resume the hypercall page for hibernation")
> Cc: stable@vger.kernel.org
> Signed-off-by: Dexuan Cui <decui@microsoft.com>
> ---
> arch/x86/hyperv/hv_init.c | 12 ++++++++++--
> 1 file changed, 10 insertions(+), 2 deletions(-)
>
> diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
> index b0da5320bcff..4d3ce86331a3 100644
> --- a/arch/x86/hyperv/hv_init.c
> +++ b/arch/x86/hyperv/hv_init.c
> @@ -72,7 +72,8 @@ static int hv_cpu_init(unsigned int cpu)
> struct page *pg;
>
> input_arg = (void **)this_cpu_ptr(hyperv_pcpu_input_arg);
> - pg = alloc_page(GFP_KERNEL);
> + /* hv_cpu_init() can be called with IRQs disabled from hv_resume() */
> + pg = alloc_page(GFP_ATOMIC);
> if (unlikely(!pg))
> return -ENOMEM;
> *input_arg = page_address(pg);
> @@ -253,6 +254,7 @@ static int __init hv_pci_init(void)
> static int hv_suspend(void)
> {
> union hv_x64_msr_hypercall_contents hypercall_msr;
> + int ret;
>
> /*
> * Reset the hypercall page as it is going to be invalidated
> @@ -269,12 +271,17 @@ static int hv_suspend(void)
> hypercall_msr.enable = 0;
> wrmsrl(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64);
>
> - return 0;
> + ret = hv_cpu_die(0);
> + return ret;
> }
>
> static void hv_resume(void)
> {
> union hv_x64_msr_hypercall_contents hypercall_msr;
> + int ret;
> +
> + ret = hv_cpu_init(0);
> + WARN_ON(ret);
>
> /* Re-enable the hypercall page */
> rdmsrl(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64);
> @@ -287,6 +294,7 @@ static void hv_resume(void)
> hv_hypercall_pg_saved = NULL;
> }
>
> +/* Note: when the ops are called, only CPU0 is online and IRQs are disabled. */
> static struct syscore_ops hv_syscore_ops = {
> .suspend = hv_suspend,
> .resume = hv_resume,
> --
> 2.19.1
>
^ permalink raw reply
* Re: [PATCH] x86/hyperv: Suspend/resume the VP assist page for hibernation
From: Vitaly Kuznetsov @ 2020-04-17 12:03 UTC (permalink / raw)
To: Wei Liu
Cc: Dexuan Cui, bp, haiyangz, hpa, kys, linux-hyperv, linux-kernel,
mingo, sthemmin, tglx, x86, mikelley, wei.liu, kvm
In-Reply-To: <20200417105558.2jkqq2lih6vvoip2@debian>
Wei Liu <wei.liu@kernel.org> writes:
> On Fri, Apr 17, 2020 at 12:03:18PM +0200, Vitaly Kuznetsov wrote:
>> Dexuan Cui <decui@microsoft.com> writes:
>>
>> > Unlike the other CPUs, CPU0 is never offlined during hibernation. So in the
>> > resume path, the "new" kernel's VP assist page is not suspended (i.e.
>> > disabled), and later when we jump to the "old" kernel, the page is not
>> > properly re-enabled for CPU0 with the allocated page from the old kernel.
>> >
>> > So far, the VP assist page is only used by hv_apic_eoi_write().
>>
>> No, not only for that ('git grep hv_get_vp_assist_page')
>>
>> KVM on Hyper-V also needs VP assist page to use Enlightened VMCS. In
>> particular, Enlightened VMPTR is written there.
>>
>> This makes me wonder: how does hibernation work with KVM in case we use
>> Enlightened VMCS and we have VMs running? We need to make sure VP Assist
>> page content is preserved.
>
> The page itself is preserved, isn't it?
>
Right, unlike hyperv_pcpu_input_arg is is not freed.
> hv_cpu_die never frees the vp_assit page. It merely disables it.
> hv_cpu_init only allocates a new page if necessary.
I'm not really sure that Hyper-V will like us when we disable VP Assist
page and have an active L2 guest using Enlightened VMCS, who knows what
it caches and when. I'll try to at least test if/how it works.
This all is not really related to Dexuan's patch)
--
Vitaly
^ permalink raw reply
* Re: [PATCH] x86/hyperv: Suspend/resume the VP assist page for hibernation
From: Wei Liu @ 2020-04-17 13:08 UTC (permalink / raw)
To: Vitaly Kuznetsov
Cc: Wei Liu, Dexuan Cui, bp, haiyangz, hpa, kys, linux-hyperv,
linux-kernel, mingo, sthemmin, tglx, x86, mikelley, kvm
In-Reply-To: <87wo6etj39.fsf@vitty.brq.redhat.com>
On Fri, Apr 17, 2020 at 02:03:38PM +0200, Vitaly Kuznetsov wrote:
> Wei Liu <wei.liu@kernel.org> writes:
>
> > On Fri, Apr 17, 2020 at 12:03:18PM +0200, Vitaly Kuznetsov wrote:
> >> Dexuan Cui <decui@microsoft.com> writes:
> >>
> >> > Unlike the other CPUs, CPU0 is never offlined during hibernation. So in the
> >> > resume path, the "new" kernel's VP assist page is not suspended (i.e.
> >> > disabled), and later when we jump to the "old" kernel, the page is not
> >> > properly re-enabled for CPU0 with the allocated page from the old kernel.
> >> >
> >> > So far, the VP assist page is only used by hv_apic_eoi_write().
> >>
> >> No, not only for that ('git grep hv_get_vp_assist_page')
> >>
> >> KVM on Hyper-V also needs VP assist page to use Enlightened VMCS. In
> >> particular, Enlightened VMPTR is written there.
> >>
> >> This makes me wonder: how does hibernation work with KVM in case we use
> >> Enlightened VMCS and we have VMs running? We need to make sure VP Assist
> >> page content is preserved.
> >
> > The page itself is preserved, isn't it?
> >
>
> Right, unlike hyperv_pcpu_input_arg is is not freed.
>
> > hv_cpu_die never frees the vp_assit page. It merely disables it.
> > hv_cpu_init only allocates a new page if necessary.
>
> I'm not really sure that Hyper-V will like us when we disable VP Assist
> page and have an active L2 guest using Enlightened VMCS, who knows what
> it caches and when. I'll try to at least test if/how it works.
>
I'm curious to know that as well. :-)
> This all is not really related to Dexuan's patch)
Right.
Wei.
>
> --
> Vitaly
>
^ permalink raw reply
* RE: [PATCH] x86/hyperv: Suspend/resume the VP assist page for hibernation
From: Dexuan Cui @ 2020-04-17 22:44 UTC (permalink / raw)
To: Wei Liu
Cc: bp@alien8.de, Haiyang Zhang, hpa@zytor.com, KY Srinivasan,
linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org,
mingo@redhat.com, Stephen Hemminger, tglx@linutronix.de,
x86@kernel.org, Michael Kelley, vkuznets
In-Reply-To: <20200417090748.r2c45se5paqz5766@debian>
> From: Wei Liu <wei.liu@kernel.org>
> Sent: Friday, April 17, 2020 2:08 AM
> > @@ -72,7 +72,8 @@ static int hv_cpu_init(unsigned int cpu)
> > struct page *pg;
> >
> > input_arg = (void **)this_cpu_ptr(hyperv_pcpu_input_arg);
> > - pg = alloc_page(GFP_KERNEL);
> > + /* hv_cpu_init() can be called with IRQs disabled from hv_resume() */
> > + pg = alloc_page(GFP_ATOMIC);
>
> IMHO it would be better to only tap into the reserve pool if so
> required, e.g.
>
> pg = alloc_page(irqs_disabled() ? GFP_ATOMIC : GFP_KERNEL);
>
> Wei.
Ok, I'll follow the suggestion.
BTW, there are indeed some usages like this, but not a lot:
grep irqs_disabled drivers/acpi include/acpi drivers/trace -nr |grep GFP_ATOMIC | grep GFP_KERNEL
Thanks,
-- Dexuan
^ permalink raw reply
* RE: [PATCH] x86/hyperv: Suspend/resume the VP assist page for hibernation
From: Dexuan Cui @ 2020-04-17 23:07 UTC (permalink / raw)
To: vkuznets, Wei Liu
Cc: bp@alien8.de, Haiyang Zhang, hpa@zytor.com, KY Srinivasan,
linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org,
mingo@redhat.com, Stephen Hemminger, tglx@linutronix.de,
x86@kernel.org, Michael Kelley, wei.liu@kernel.org,
kvm@vger.kernel.org
In-Reply-To: <87wo6etj39.fsf@vitty.brq.redhat.com>
> From: Vitaly Kuznetsov <vkuznets@redhat.com>
> Sent: Friday, April 17, 2020 5:04 AM
> To: Wei Liu <wei.liu@kernel.org>
>
> Wei Liu <wei.liu@kernel.org> writes:
>
> > On Fri, Apr 17, 2020 at 12:03:18PM +0200, Vitaly Kuznetsov wrote:
> >> Dexuan Cui <decui@microsoft.com> writes:
> >>
> >> > Unlike the other CPUs, CPU0 is never offlined during hibernation. So in the
> >> > resume path, the "new" kernel's VP assist page is not suspended (i.e.
> >> > disabled), and later when we jump to the "old" kernel, the page is not
> >> > properly re-enabled for CPU0 with the allocated page from the old kernel.
> >> >
> >> > So far, the VP assist page is only used by hv_apic_eoi_write().
> >>
> >> No, not only for that ('git grep hv_get_vp_assist_page')
Sorry, I unintentionally ignored that, as I have few knowledge about the
optimization for nested virtualization. :-)
> >> KVM on Hyper-V also needs VP assist page to use Enlightened VMCS. In
> >> particular, Enlightened VMPTR is written there.
> >>
> >> This makes me wonder: how does hibernation work with KVM in case we
> use
> >> Enlightened VMCS and we have VMs running? We need to make sure VP
> Assist
> >> page content is preserved.
> >
> > The page itself is preserved, isn't it?
> >
>
> Right, unlike hyperv_pcpu_input_arg is is not freed.
>
> > hv_cpu_die never frees the vp_assit page. It merely disables it.
> > hv_cpu_init only allocates a new page if necessary.
>
> I'm not really sure that Hyper-V will like us when we disable VP Assist
> page and have an active L2 guest using Enlightened VMCS, who knows what
> it caches and when. I'll try to at least test if/how it works.
>
> This all is not really related to Dexuan's patch)
> --
> Vitaly
It looks you imply that: if there is no active L2 guests, it should be safe to
disable/reenable the assist page upon hibernation?
Can you please write a patch for KVM (when KVM runs on Hyper-V) to abort
the hibernation request if there is any active L2 guest? The pm_notifier can
be used for this.
Thanks,
-- Dexuan
^ permalink raw reply
* RE: [PATCH] x86/hyperv: Suspend/resume the VP assist page for hibernation
From: Dexuan Cui @ 2020-04-17 23:47 UTC (permalink / raw)
To: Wei Liu
Cc: bp@alien8.de, Haiyang Zhang, hpa@zytor.com, KY Srinivasan,
linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org,
mingo@redhat.com, Stephen Hemminger, tglx@linutronix.de,
x86@kernel.org, Michael Kelley, vkuznets
In-Reply-To: <20200417110007.uzfo6musx2x2suw7@debian>
> From: Wei Liu <wei.liu@kernel.org>
> Sent: Friday, April 17, 2020 4:00 AM
> To: Dexuan Cui <decui@microsoft.com>
>
> On Thu, Apr 16, 2020 at 11:29:59PM -0700, Dexuan Cui wrote:
> > Unlike the other CPUs, CPU0 is never offlined during hibernation. So in the
> > resume path, the "new" kernel's VP assist page is not suspended (i.e.
> > disabled), and later when we jump to the "old" kernel, the page is not
> > properly re-enabled for CPU0 with the allocated page from the old kernel.
> >
> > So far, the VP assist page is only used by hv_apic_eoi_write(). When the
> > page is not properly re-enabled, hvp->apic_assist is always 0, so the
> > HV_X64_MSR_EOI MSR is always written. This is not ideal with respect to
> > performance, but Hyper-V can still correctly handle this.
> >
> > The issue is: the hypervisor can corrupt the old kernel memory, and hence
> > sometimes cause unexpected behaviors, e.g. when the old kernel's non-boot
> > CPUs are being onlined in the resume path, the VM can hang or be killed
> > due to virtual triple fault.
>
> I don't quite follow here.
>
> The first sentence is rather alarming -- why would Hyper-V corrupt
> guest's memory (kernel or not)?
Without this patch, after the VM resumes from hibernation, the hypervisor
still thinks the assist page of vCPU0 points to the physical page allocated by
the "new" kernel (the "new" kernel started up freshly, loaded the saved state
of the "old" kernel from disk into memory, and jumped to the "old" kernel),
but the same physical page can be allocated to store something different in
the "old" kernel (which is the currently running kernel, since the VM resumed).
Conceptually, it looks Hyper-V writes into the assist page from time to time,
e.g. for the EOI optimization. This "corrupts" the page for the "old" kernel.
I'm not absolutely sure if this explains the strange hang issue or triple fault
I occasionally saw in my long-haul hibernation test, but with this patch,
I never reproduce the strange hang/triple fault issue again, so I think this
patch works.
> Secondly, code below only specifies cpu0. What does it do with non-boot
> cpus on the resume path?
>
> Wei.
hyperv_init() registers hv_cpu_init()/hv_cpu_die() to the cpuhp framework:
cpuhp = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "x86/hyperv_init:online",
hv_cpu_init, hv_cpu_die);
In the hibernation procedure, the non-boot CPUs are automatically disabled
and reenabled, so hv_cpu_init()/hv_cpu_die() are automatically called for them,
e.g. in the resume path, see:
hibernation_restore()
resume_target_kernel()
hibernate_resume_nonboot_cpu_disable()
disable_nonboot_cpus()
syscore_suspend()
hv_cpu_die(0) // Added by this patch
swsusp_arch_resume()
relocate_restore_code()
restore_image()
jump to the old kernel and we return from
the swsusp_arch_suspend() in create_image()
syscore_resume()
hv_cpu_init(0) // Added by this patch.
suspend_enable_secondary_cpus()
dpm_resume_start()
...
Thanks,
-- Dexuan
^ permalink raw reply
* Re: [PATCH v2 0/1] x86/kvm/hyper-v: Add support to SYNIC exit on EOM
From: Jon Doron @ 2020-04-18 6:41 UTC (permalink / raw)
To: Roman Kagan, kvm, linux-hyperv, vkuznets
In-Reply-To: <20200417104251.GA3009@rvkaganb>
On 17/04/2020, Roman Kagan wrote:
>On Thu, Apr 16, 2020 at 03:54:30PM +0300, Jon Doron wrote:
>> On 16/04/2020, Roman Kagan wrote:
>> > On Thu, Apr 16, 2020 at 11:38:46AM +0300, Jon Doron wrote:
>> > > According to the TLFS:
>> > > "A write to the end of message (EOM) register by the guest causes the
>> > > hypervisor to scan the internal message buffer queue(s) associated with
>> > > the virtual processor.
>> > >
>> > > If a message buffer queue contains a queued message buffer, the hypervisor
>> > > attempts to deliver the message.
>> > >
>> > > Message delivery succeeds if the SIM page is enabled and the message slot
>> > > corresponding to the SINTx is empty (that is, the message type in the
>> > > header is set to HvMessageTypeNone).
>> > > If a message is successfully delivered, its corresponding internal message
>> > > buffer is dequeued and marked free.
>> > > If the corresponding SINTx is not masked, an edge-triggered interrupt is
>> > > delivered (that is, the corresponding bit in the IRR is set).
>> > >
>> > > This register can be used by guests to poll for messages. It can also be
>> > > used as a way to drain the message queue for a SINTx that has
>> > > been disabled (that is, masked)."
>> >
>> > Doesn't this work already?
>> >
>>
>> Well if you dont have SCONTROL and a GSI associated with the SINT then it
>> does not...
>
>Yes you do need both of these.
>
>> > > So basically this means that we need to exit on EOM so the hypervisor
>> > > will have a chance to send all the pending messages regardless of the
>> > > SCONTROL mechnaisim.
>> >
>> > I might be misinterpreting the spec, but my understanding is that
>> > SCONTROL {en,dis}ables the message queueing completely. What the quoted
>> > part means is that a write to EOM should trigger the message source to
>> > push a new message into the slot, regardless of whether the SINT was
>> > masked or not.
>> >
>> > And this (I think, haven't tested) should already work. The userspace
>> > just keeps using the SINT route as it normally does, posting
>> > notifications to the corresponding irqfd when posting a message, and
>> > waiting on the resamplerfd for the message slot to become free. If the
>> > SINT is masked KVM will skip injecting the interrupt, that's it.
>> >
>> > Roman.
>>
>> That's what I was thinking originally as well, but then i noticed KDNET as a
>> VMBus client (and it basically runs before anything else) is working in this
>> polling mode, where SCONTROL is disabled and it just loops, and if it saw
>> there is a PENDING message flag it will issue an EOM to indicate it has free
>> the slot.
>
>Who sets up the message page then? Doesn't it enabe SCONTROL as well?
>
KdNet is the one setting the SIMP and it's not setting the SCONTROL, ill
paste output of KVM traces for the relevant MSRs
>Note that, even if you don't see it being enabled by Windows, it can be
>enabled by the firmware and/or by the bootloader.
>
>Can you perhaps try with the SeaBIOS from
>https://src.openvz.org/projects/UP/repos/seabios branch hv-scsi? It
>enables SCONTROL and leaves it that way.
>
>I'd also suggest tracing kvm_msr events (both reads and writes) for
>SCONTROL and SIMP msrs, to better understand the picture.
>
>So far the change you propose appears too heavy to work around the
>problem of disabled SCONTROL. You seem to be better off just making
>sure it's enabled (either by the firmware or slighly violating the spec
>and initializing to enabled from the start), and sticking to the
>existing infrastructure for posting messages.
>
I guess there is something I'm missing here but let's say the BIOS would
have set the SCONTROL but the OS is not setting it, who is in charge of
handling the interrupts?
>> (There are a bunch of patches i sent on the QEMU mailing list as well where
>> i CCed you, I will probably revise it a bit but was hoping to get KVM
>> sorted out first).
>
>I'll look through the archive, should be there, thanks.
>
>Roman.
I tried testing with both the SeaBIOS branch you have suggested and the
EDK2, unfortunately I could not get the EDK2 build to identify my VM
drive to boot from (not sure why)
Here is an output of KVM trace for the relevant MSRs (SCONTROL and SIMP)
QEMU Default BIOS
-----------------
qemu-system-x86-613 [000] .... 1121.080722: kvm_hv_synic_set_msr: vcpu_id 0 msr 0x40000080 data 0x0 host 1
qemu-system-x86-613 [000] .... 1121.080722: kvm_hv_synic_set_msr: vcpu_id 0 msr 0x40000083 data 0x0 host 1
qemu-system-x86-613 [000] .N.. 1121.095592: kvm_hv_synic_set_msr: vcpu_id 0 msr 0x40000080 data 0x0 host 1
qemu-system-x86-613 [000] .N.. 1121.095592: kvm_hv_synic_set_msr: vcpu_id 0 msr 0x40000083 data 0x0 host 1
Choose Windows DebugEntry
qemu-system-x86-613 [001] .... 1165.185227: kvm_msr: msr_read 40000083 = 0x0
qemu-system-x86-613 [001] .... 1165.185255: kvm_hv_synic_set_msr: vcpu_id 0 msr 0x40000083 data 0xfa1001 host 0
qemu-system-x86-613 [001] .... 1165.185255: kvm_msr: msr_write 40000083 = 0xfa1001
qemu-system-x86-613 [001] .... 1165.193206: kvm_msr: msr_read 40000083 = 0xfa1001
qemu-system-x86-613 [001] .... 1165.193236: kvm_hv_synic_set_msr: vcpu_id 0 msr 0x40000083 data 0xfa1000 host 0
qemu-system-x86-613 [001] .... 1165.193237: kvm_msr: msr_write 40000083 = 0xfa1000
SeaBIOS hv-scsci
----------------
qemu-system-x86-656 [001] .... 1313.072714: kvm_hv_synic_set_msr: vcpu_id 0 msr 0x40000080 data 0x0 host 1
qemu-system-x86-656 [001] .... 1313.072714: kvm_hv_synic_set_msr: vcpu_id 0 msr 0x40000083 data 0x0 host 1
qemu-system-x86-656 [001] .... 1313.087752: kvm_hv_synic_set_msr: vcpu_id 0 msr 0x40000080 data 0x0 host 1
qemu-system-x86-656 [001] .... 1313.087752: kvm_hv_synic_set_msr: vcpu_id 0 msr 0x40000083 data 0x0 host 1
qemu-system-x86-656 [001] .... 1313.156675: kvm_msr: msr_read 40000083 = 0x0
qemu-system-x86-656 [001] .... 1313.156680: kvm_hv_synic_set_msr: vcpu_id 0 msr 0x40000083 data 0x7fffe001 host 0
Choose Windows DebugEntry
qemu-system-x86-656 [001] .... 1313.156680: kvm_msr: msr_write 40000083 = 0x7fffe001
qemu-system-x86-656 [001] .... 1313.162111: kvm_msr: msr_read 40000080 = 0x0
qemu-system-x86-656 [001] .... 1313.162118: kvm_hv_synic_set_msr: vcpu_id 0 msr 0x40000080 data 0x1 host 0
qemu-system-x86-656 [001] .... 1313.162119: kvm_msr: msr_write 40000080 = 0x1
qemu-system-x86-656 [001] .... 1313.246758: kvm_msr: msr_read 40000083 = 0x7fffe001
qemu-system-x86-656 [001] .... 1313.246764: kvm_hv_synic_set_msr: vcpu_id 0 msr 0x40000083 data 0x0 host 0
qemu-system-x86-656 [001] .... 1313.246764: kvm_msr: msr_write 40000083 = 0x0
qemu-system-x86-656 [001] .... 1348.904727: kvm_msr: msr_read 40000083 = 0x0
qemu-system-x86-656 [001] .... 1348.904771: kvm_hv_synic_set_msr: vcpu_id 0 msr 0x40000083 data 0xfa1001 host 0
qemu-system-x86-656 [001] .... 1348.904772: kvm_msr: msr_write 40000083 = 0xfa1001
qemu-system-x86-656 [001] .... 1348.919170: kvm_msr: msr_read 40000083 = 0xfa1001
qemu-system-x86-656 [001] .... 1348.919183: kvm_hv_synic_set_msr: vcpu_id 0 msr 0x40000083 data 0xfa1000 host 0
qemu-system-x86-656 [001] .... 1348.919183: kvm_msr: msr_write 40000083 = 0xfa1000
I could not get the EDK2 setup to work though
(https://src.openvz.org/projects/UP/repos/edk2 branch hv-scsi)
It does not detect my VM hard drive not sure why (this is how i
configured it:
-drive file=./win10.qcow2,format=qcow2,if=none,id=drive_disk0 \
-device virtio-blk-pci,drive=drive_disk0 \
(Is there something special i need to configure it order for it to
work?, I tried building EDK2 with and without SMM_REQUIRE and
SECURE_BOOT_ENABLE)
But in general it sounds like there is something I dont fully
understand when SCONTROL is enabled, then a GSI is associated with this
SintRoute.
Then when the guest triggers an EOI via the APIC we will trigger the GSI
notification, which will give us another go on trying to copy the
message into it's slot.
So is it the OS that is in charge of setting the EOI? If so then it
needs to be aware of SCONTROL being enabled and just having it left set
by the BIOS might not be enough?
Also in the TLFS (looking at v6) they mention that message queueing has
"3 exit conditions", which will cause the hypervisor to try and attempt
to deliver the additional messages.
The 3 exit conditions they refer to are:
* Another message buffer is queued.
* The guest indicates the “end of interrupt” by writing to the APIC’s
EOI register.
* The guest indicates the “end of message” by writing to the SynIC’s EOM
register.
Also notice this additional exit is only if there is a pending message
and not for every EOM.
Thanks,
-- Jon.
^ permalink raw reply
* Re: [PATCH 26/29] mm: remove vmalloc_user_node_flags
From: Geert Uytterhoeven @ 2020-04-20 9:26 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, K. Y. Srinivasan, Haiyang Zhang, Stephen Hemminger,
Wei Liu, the arch/x86 maintainers, David Airlie, Daniel Vetter,
Laura Abbott, Sumit Semwal, Sakari Ailus, Minchan Kim,
Nitin Gupta, Christophe Leroy, Linux-Arch, linux-hyperv,
linux-s390, Peter Zijlstra, linuxppc-dev,
Linux Kernel Mailing List, DRI Development, linaro-mm-sig,
Linux MM, Linux IOMMU, Johannes Weiner, bpf, Robin Murphy,
Linux ARM
In-Reply-To: <20200414131348.444715-27-hch@lst.de>
Hi Christoph,
On Tue, Apr 14, 2020 at 3:22 PM Christoph Hellwig <hch@lst.de> wrote:
> Open code it in __bpf_map_area_alloc, which is the only caller. Also
> clean up __bpf_map_area_alloc to have a single vmalloc call with
> slightly different flags instead of the current two different calls.
>
> For this to compile for the nommu case add a __vmalloc_node_range stub
> to nommu.c.
Apparently your nommu-cross-compilers are in quarantaine? ;-)
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> Acked-by: Johannes Weiner <hannes@cmpxchg.org>
> --- a/mm/nommu.c
> +++ b/mm/nommu.c
> @@ -150,6 +150,14 @@ void *__vmalloc(unsigned long size, gfp_t gfp_mask)
> }
> EXPORT_SYMBOL(__vmalloc);
>
> +void *__vmalloc_node_range(unsigned long size, unsigned long align,
> + unsigned long start, unsigned long end, gfp_t gfp_mask,
> + pgprot_t prot, unsigned long vm_flags, int node,
> + const void *caller)
> +{
> + return __vmalloc(size, flags);
On Mon, Apr 20, 2020 at 10:39 AM <noreply@ellerman.id.au> wrote:
> FAILED linux-next/m5272c3_defconfig/m68k-gcc8 Mon Apr 20, 18:38
>
> http://kisskb.ellerman.id.au/kisskb/buildresult/14213623/
>
> mm/nommu.c:158:25: error: 'flags' undeclared (first use in this function); did you mean 'class'?
"return __vmalloc(size, gfp_mask);", I assume?
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply
* Re: [PATCH 24/29] mm: remove __vmalloc_node_flags_caller
From: Geert Uytterhoeven @ 2020-04-20 9:27 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, K. Y. Srinivasan, Haiyang Zhang, Stephen Hemminger,
Wei Liu, the arch/x86 maintainers, David Airlie, Daniel Vetter,
Laura Abbott, Sumit Semwal, Sakari Ailus, Minchan Kim,
Nitin Gupta, Christophe Leroy, Linux-Arch, linux-hyperv,
linux-s390, Peter Zijlstra, linuxppc-dev,
Linux Kernel Mailing List, DRI Development, linaro-mm-sig,
Linux MM, Linux IOMMU, bpf, Robin Murphy, Linux ARM
In-Reply-To: <20200414131348.444715-25-hch@lst.de>
Hi Christoph,
On Tue, Apr 14, 2020 at 3:21 PM Christoph Hellwig <hch@lst.de> wrote:
> Just use __vmalloc_node instead which gets and extra argument. To be
> able to to use __vmalloc_node in all caller make it available outside
> of vmalloc and implement it in nommu.c.
>
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
One more nommu failure below...
> --- a/mm/nommu.c
> +++ b/mm/nommu.c
> @@ -150,8 +150,8 @@ void *__vmalloc(unsigned long size, gfp_t gfp_mask)
> }
> EXPORT_SYMBOL(__vmalloc);
>
> -void *__vmalloc_node_flags_caller(unsigned long size, int node, gfp_t flags,
> - void *caller)
> +void *__vmalloc_node(unsigned long size, unsigned long align, gfp_t gfp_mask,
> + int node, const void *caller)
> {
> return __vmalloc(size, flags);
On Mon, Apr 20, 2020 at 10:39 AM <noreply@ellerman.id.au> wrote:
> FAILED linux-next/m5272c3_defconfig/m68k-gcc8 Mon Apr 20, 18:38
>
> http://kisskb.ellerman.id.au/kisskb/buildresult/14213623/
>
> mm/nommu.c:164:25: error: 'flags' undeclared (first use in this function); did you mean 'class'?
"return __vmalloc(size, gfp_mask);"
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply
* Re: [PATCH] x86/hyperv: Suspend/resume the VP assist page for hibernation
From: Wei Liu @ 2020-04-20 12:08 UTC (permalink / raw)
To: Dexuan Cui
Cc: Wei Liu, bp@alien8.de, Haiyang Zhang, hpa@zytor.com,
KY Srinivasan, linux-hyperv@vger.kernel.org,
linux-kernel@vger.kernel.org, mingo@redhat.com, Stephen Hemminger,
tglx@linutronix.de, x86@kernel.org, Michael Kelley, vkuznets
In-Reply-To: <HK0P153MB0273A04F0585524883C46B0FBFD90@HK0P153MB0273.APCP153.PROD.OUTLOOK.COM>
On Fri, Apr 17, 2020 at 11:47:41PM +0000, Dexuan Cui wrote:
> > From: Wei Liu <wei.liu@kernel.org>
> > Sent: Friday, April 17, 2020 4:00 AM
> > To: Dexuan Cui <decui@microsoft.com>
> >
> > On Thu, Apr 16, 2020 at 11:29:59PM -0700, Dexuan Cui wrote:
> > > Unlike the other CPUs, CPU0 is never offlined during hibernation. So in the
> > > resume path, the "new" kernel's VP assist page is not suspended (i.e.
> > > disabled), and later when we jump to the "old" kernel, the page is not
> > > properly re-enabled for CPU0 with the allocated page from the old kernel.
> > >
> > > So far, the VP assist page is only used by hv_apic_eoi_write(). When the
> > > page is not properly re-enabled, hvp->apic_assist is always 0, so the
> > > HV_X64_MSR_EOI MSR is always written. This is not ideal with respect to
> > > performance, but Hyper-V can still correctly handle this.
> > >
> > > The issue is: the hypervisor can corrupt the old kernel memory, and hence
> > > sometimes cause unexpected behaviors, e.g. when the old kernel's non-boot
> > > CPUs are being onlined in the resume path, the VM can hang or be killed
> > > due to virtual triple fault.
> >
> > I don't quite follow here.
> >
> > The first sentence is rather alarming -- why would Hyper-V corrupt
> > guest's memory (kernel or not)?
>
> Without this patch, after the VM resumes from hibernation, the hypervisor
> still thinks the assist page of vCPU0 points to the physical page allocated by
> the "new" kernel (the "new" kernel started up freshly, loaded the saved state
> of the "old" kernel from disk into memory, and jumped to the "old" kernel),
> but the same physical page can be allocated to store something different in
> the "old" kernel (which is the currently running kernel, since the VM resumed).
>
> Conceptually, it looks Hyper-V writes into the assist page from time to time,
> e.g. for the EOI optimization. This "corrupts" the page for the "old" kernel.
>
> I'm not absolutely sure if this explains the strange hang issue or triple fault
> I occasionally saw in my long-haul hibernation test, but with this patch,
> I never reproduce the strange hang/triple fault issue again, so I think this
> patch works.
OK. I wouldn't be surprised if the corruption ends up changing code in
the kernel which further triggers triple fault etc.
I would suggest make this clear in the commit message to not give the
impression that Hyper-V has this weird behaviour of corrupting guest
memory for no reason.
We can replace the paragraph starting with "The issue is: ..." with:
---
Linux needs to update Hyper-V the correct VP assist page to prevent
Hyper-V from writing to a stale page, which causes guest memory
corruption. The memory corruption may have caused some of the hangs and
triple faults we saw during non-boot CPUs resume.
---
This What do you think?
Wei.
>
> > Secondly, code below only specifies cpu0. What does it do with non-boot
> > cpus on the resume path?
> >
> > Wei.
>
> hyperv_init() registers hv_cpu_init()/hv_cpu_die() to the cpuhp framework:
>
> cpuhp = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "x86/hyperv_init:online",
> hv_cpu_init, hv_cpu_die);
>
> In the hibernation procedure, the non-boot CPUs are automatically disabled
> and reenabled, so hv_cpu_init()/hv_cpu_die() are automatically called for them,
> e.g. in the resume path, see:
> hibernation_restore()
> resume_target_kernel()
> hibernate_resume_nonboot_cpu_disable()
> disable_nonboot_cpus()
> syscore_suspend()
> hv_cpu_die(0) // Added by this patch
> swsusp_arch_resume()
> relocate_restore_code()
> restore_image()
> jump to the old kernel and we return from
> the swsusp_arch_suspend() in create_image()
> syscore_resume()
> hv_cpu_init(0) // Added by this patch.
> suspend_enable_secondary_cpus()
> dpm_resume_start()
> ...
> Thanks,
> -- Dexuan
^ permalink raw reply
* RE: [PATCH] x86/hyperv: Suspend/resume the VP assist page for hibernation
From: Dexuan Cui @ 2020-04-20 16:40 UTC (permalink / raw)
To: Wei Liu
Cc: bp@alien8.de, Haiyang Zhang, hpa@zytor.com, KY Srinivasan,
linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org,
mingo@redhat.com, Stephen Hemminger, tglx@linutronix.de,
x86@kernel.org, Michael Kelley, vkuznets
In-Reply-To: <20200420120822.4bncj2iwgqbpoxei@debian>
> From: Wei Liu <wei.liu@kernel.org>
> Sent: Monday, April 20, 2020 5:08 AM
>
> I would suggest make this clear in the commit message to not give the
> impression that Hyper-V has this weird behaviour of corrupting guest
> memory for no reason.
>
> We can replace the paragraph starting with "The issue is: ..." with:
>
> ---
> Linux needs to update Hyper-V the correct VP assist page to prevent
> Hyper-V from writing to a stale page, which causes guest memory
> corruption. The memory corruption may have caused some of the hangs and
> triple faults we saw during non-boot CPUs resume.
> ---
> This What do you think?
This version is much better. I'll use it. Thanks, Wei!
I'll post a v2 with the updated comments. I'll document Vitaly's concern about
nested virtualization in the comment.
Thanks,
-- Dexuan
^ permalink raw reply
* [PATCH 1/1] Drivers: hv: Move AEOI determination to architecture dependent code
From: Michael Kelley @ 2020-04-20 16:49 UTC (permalink / raw)
To: kys, haiyangz, sthemmin, wei.liu, tglx, mingo, bp, x86, hpa,
linux-kernel, linux-hyperv
Cc: mikelley
Hyper-V on ARM64 doesn't provide a flag for the AEOI recommendation
in ms_hyperv.hints, so having the test in architecture independent
code doesn't work. Resolve this by moving the check of the flag
to an architecture dependent helper function. No functionality is
changed.
Signed-off-by: Michael Kelley <mikelley@microsoft.com>
---
arch/x86/include/asm/mshyperv.h | 2 ++
drivers/hv/hv.c | 6 +-----
2 files changed, 3 insertions(+), 5 deletions(-)
diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
index 1c42ecbe75cb..d30805ed323e 100644
--- a/arch/x86/include/asm/mshyperv.h
+++ b/arch/x86/include/asm/mshyperv.h
@@ -35,6 +35,8 @@ typedef int (*hyperv_fill_flush_list_func)(
rdmsrl(HV_X64_MSR_SINT0 + int_num, val)
#define hv_set_synint_state(int_num, val) \
wrmsrl(HV_X64_MSR_SINT0 + int_num, val)
+#define hv_recommend_using_aeoi() \
+ (!(ms_hyperv.hints & HV_DEPRECATING_AEOI_RECOMMENDED))
#define hv_get_crash_ctl(val) \
rdmsrl(HV_X64_MSR_CRASH_CTL, val)
diff --git a/drivers/hv/hv.c b/drivers/hv/hv.c
index 6098e0cbdb4b..533c8b82b344 100644
--- a/drivers/hv/hv.c
+++ b/drivers/hv/hv.c
@@ -184,11 +184,7 @@ void hv_synic_enable_regs(unsigned int cpu)
shared_sint.vector = HYPERVISOR_CALLBACK_VECTOR;
shared_sint.masked = false;
- if (ms_hyperv.hints & HV_DEPRECATING_AEOI_RECOMMENDED)
- shared_sint.auto_eoi = false;
- else
- shared_sint.auto_eoi = true;
-
+ shared_sint.auto_eoi = hv_recommend_using_aeoi();
hv_set_synint_state(VMBUS_MESSAGE_SINT, shared_sint.as_uint64);
/* Enable the global synic bit */
--
2.18.2
^ permalink raw reply related
* [PATCH 0/4] Split hyperv-tlfs.h into generic and arch specific files
From: Michael Kelley @ 2020-04-20 17:38 UTC (permalink / raw)
To: kys, haiyangz, sthemmin, wei.liu, tglx, mingo, bp, x86, hpa,
pbonzini, sean.j.christopherson, vkuznets, wanpengli, jmattson,
joro, kvm, linux-kernel, linux-hyperv
Cc: mikelley
This series splits hyperv-tlfs.h into architecture independent and
architecture specific files so that the arch independent portion can
be shared between the x86/x64 and ARM64 code for Hyper-V. While the
Hyper-V team has not released a version of the TLFS document that
clearly specifies which portions of the interface are arch independent,
we can make a fairly good assessment based on implementation work done
to support Linux guests on Hyper-V on ARM64, and on private communication
with the Hyper-V team. Definitions are considered arch independent if
they are implemented by Hyper-V on both architectures (x86/x64 and ARM64),
even if they are currently needed by Linux code only on one architecture.
Many definitions in hyperv-tlfs.h have historically contained "X64" in the
name, which doesn't make sense for architecture independent definitions.
While many of the occurrences of "X64" have already been removed, some
still remain in definitions that should be arch independent. The
split removes the "X64" from the definitions so that the arch
independent hyper-tlfs.h has no occurrences of "X64". However, to
keep this patch set separate from a wider change in the names, aliases
are added in the x86/x64 specific hyperv-tlfs.h so that existing code
continues to compile. The definitions can be fixed throughout the code
in a more incremental fashion in separate patches, and then the aliases
can be removed.
Where it is not clear if definitions are arch independent, they have been
kept in the x86/x64 specific file. The Hyper-V team is aiming to have a
version of the TLFS document covering ARM64 by the end of calendar 2020,
so additional definitions may be moved into the arch independent portion
after the new TLFS document is released.
The first two patches in the series clean up the existing hyperv-tlfs.h
file a bit by removing duplicate or unnecessary definitions so they are
not propagated across the split. The third patch does the split, and the
fourth patch adds new definitions that are needed on ARM64 but are generic.
These changes have no functional impact.
Michael Kelley (4):
KVM: x86: hyperv: Remove duplicate definitions of Reference TSC Page
x86/hyperv: Remove HV_PROCESSOR_POWER_STATE #defines
x86/hyperv: Split hyperv-tlfs.h into arch dependent and independent
files
asm-generic/hyperv: Add definitions for Get/SetVpRegister hypercalls
arch/x86/include/asm/hyperv-tlfs.h | 472 +++--------------------------
arch/x86/include/asm/kvm_host.h | 2 +-
arch/x86/kvm/hyperv.c | 4 +-
include/asm-generic/hyperv-tlfs.h | 470 ++++++++++++++++++++++++++++
4 files changed, 509 insertions(+), 439 deletions(-)
create mode 100644 include/asm-generic/hyperv-tlfs.h
--
2.18.2
^ permalink raw reply
* [PATCH 3/4] x86/hyperv: Split hyperv-tlfs.h into arch dependent and independent files
From: Michael Kelley @ 2020-04-20 17:38 UTC (permalink / raw)
To: kys, haiyangz, sthemmin, wei.liu, tglx, mingo, bp, x86, hpa,
pbonzini, sean.j.christopherson, vkuznets, wanpengli, jmattson,
joro, kvm, linux-kernel, linux-hyperv
Cc: mikelley
In-Reply-To: <20200420173838.24672-1-mikelley@microsoft.com>
In preparation for adding ARM64 support, split hyperv-tlfs.h into
architecture dependent and architecture independent files, similar
to what has been done with mshyperv.h. Move architecture independent
definitions into include/asm-generic/hyperv-tlfs.h. The split will
avoid duplicating significant lines of code in the ARM64 version of
hyperv-tlfs.h. The split has no functional impact.
Some of the common definitions have "X64" in the symbol name. Change
these to remove the "X64" in the architecture independent version of
hyperv-tlfs.h, but add aliases with the "X64" in the x86 version so
that x86 code will continue to compile. A later patch set will
change all the references and allow removal of the aliases.
Signed-off-by: Michael Kelley <mikelley@microsoft.com>
---
arch/x86/include/asm/hyperv-tlfs.h | 459 +++--------------------------
include/asm-generic/hyperv-tlfs.h | 442 +++++++++++++++++++++++++++
2 files changed, 478 insertions(+), 423 deletions(-)
create mode 100644 include/asm-generic/hyperv-tlfs.h
diff --git a/arch/x86/include/asm/hyperv-tlfs.h b/arch/x86/include/asm/hyperv-tlfs.h
index 2dd1ceb2bcf8..4e91f6118d5d 100644
--- a/arch/x86/include/asm/hyperv-tlfs.h
+++ b/arch/x86/include/asm/hyperv-tlfs.h
@@ -11,17 +11,6 @@
#include <linux/types.h>
#include <asm/page.h>
-
-/*
- * While not explicitly listed in the TLFS, Hyper-V always runs with a page size
- * of 4096. These definitions are used when communicating with Hyper-V using
- * guest physical pages and guest physical page addresses, since the guest page
- * size may not be 4096 on all architectures.
- */
-#define HV_HYP_PAGE_SHIFT 12
-#define HV_HYP_PAGE_SIZE BIT(HV_HYP_PAGE_SHIFT)
-#define HV_HYP_PAGE_MASK (~(HV_HYP_PAGE_SIZE - 1))
-
/*
* The below CPUID leaves are present if VersionAndFeatures.HypervisorPresent
* is set by CPUID(HvCpuIdFunctionVersionAndFeatures).
@@ -39,78 +28,41 @@
#define HYPERV_CPUID_MAX 0x4000ffff
/*
- * Feature identification. EAX indicates which features are available
- * to the partition based upon the current partition privileges.
- * These are HYPERV_CPUID_FEATURES.EAX bits.
+ * Aliases for Group A features that have X64 in the name.
+ * On x86/x64 these are HYPERV_CPUID_FEATURES.EAX bits.
*/
-/* VP Runtime (HV_X64_MSR_VP_RUNTIME) available */
-#define HV_X64_MSR_VP_RUNTIME_AVAILABLE BIT(0)
-/* Partition Reference Counter (HV_X64_MSR_TIME_REF_COUNT) available*/
-#define HV_MSR_TIME_REF_COUNT_AVAILABLE BIT(1)
-/*
- * Basic SynIC MSRs (HV_X64_MSR_SCONTROL through HV_X64_MSR_EOM
- * and HV_X64_MSR_SINT0 through HV_X64_MSR_SINT15) available
- */
-#define HV_X64_MSR_SYNIC_AVAILABLE BIT(2)
-/*
- * Synthetic Timer MSRs (HV_X64_MSR_STIMER0_CONFIG through
- * HV_X64_MSR_STIMER3_COUNT) available
- */
-#define HV_MSR_SYNTIMER_AVAILABLE BIT(3)
-/*
- * APIC access MSRs (HV_X64_MSR_EOI, HV_X64_MSR_ICR and HV_X64_MSR_TPR)
- * are available
- */
-#define HV_X64_MSR_APIC_ACCESS_AVAILABLE BIT(4)
-/* Hypercall MSRs (HV_X64_MSR_GUEST_OS_ID and HV_X64_MSR_HYPERCALL) available*/
-#define HV_X64_MSR_HYPERCALL_AVAILABLE BIT(5)
-/* Access virtual processor index MSR (HV_X64_MSR_VP_INDEX) available*/
-#define HV_X64_MSR_VP_INDEX_AVAILABLE BIT(6)
-/* Virtual system reset MSR (HV_X64_MSR_RESET) is available*/
-#define HV_X64_MSR_RESET_AVAILABLE BIT(7)
-/*
- * Access statistics pages MSRs (HV_X64_MSR_STATS_PARTITION_RETAIL_PAGE,
- * HV_X64_MSR_STATS_PARTITION_INTERNAL_PAGE, HV_X64_MSR_STATS_VP_RETAIL_PAGE,
- * HV_X64_MSR_STATS_VP_INTERNAL_PAGE) available
- */
-#define HV_X64_MSR_STAT_PAGES_AVAILABLE BIT(8)
-/* Partition reference TSC MSR is available */
-#define HV_MSR_REFERENCE_TSC_AVAILABLE BIT(9)
-/* Partition Guest IDLE MSR is available */
-#define HV_X64_MSR_GUEST_IDLE_AVAILABLE BIT(10)
-/*
- * There is a single feature flag that signifies if the partition has access
- * to MSRs with local APIC and TSC frequencies.
- */
-#define HV_X64_ACCESS_FREQUENCY_MSRS BIT(11)
-/* AccessReenlightenmentControls privilege */
-#define HV_X64_ACCESS_REENLIGHTENMENT BIT(13)
-/* AccessTscInvariantControls privilege */
-#define HV_X64_ACCESS_TSC_INVARIANT BIT(15)
+#define HV_X64_MSR_VP_RUNTIME_AVAILABLE \
+ HV_MSR_VP_RUNTIME_AVAILABLE
+#define HV_X64_MSR_SYNIC_AVAILABLE \
+ HV_MSR_SYNIC_AVAILABLE
+#define HV_X64_MSR_APIC_ACCESS_AVAILABLE \
+ HV_MSR_APIC_ACCESS_AVAILABLE
+#define HV_X64_MSR_HYPERCALL_AVAILABLE \
+ HV_MSR_HYPERCALL_AVAILABLE
+#define HV_X64_MSR_VP_INDEX_AVAILABLE \
+ HV_MSR_VP_INDEX_AVAILABLE
+#define HV_X64_MSR_RESET_AVAILABLE \
+ HV_MSR_RESET_AVAILABLE
+#define HV_X64_MSR_GUEST_IDLE_AVAILABLE \
+ HV_MSR_GUEST_IDLE_AVAILABLE
+#define HV_X64_ACCESS_FREQUENCY_MSRS \
+ HV_ACCESS_FREQUENCY_MSRS
+#define HV_X64_ACCESS_REENLIGHTENMENT \
+ HV_ACCESS_REENLIGHTENMENT
+#define HV_X64_ACCESS_TSC_INVARIANT \
+ HV_ACCESS_TSC_INVARIANT
/*
- * Feature identification: indicates which flags were specified at partition
- * creation. The format is the same as the partition creation flag structure
- * defined in section Partition Creation Flags.
- * These are HYPERV_CPUID_FEATURES.EBX bits.
+ * Aliases for Group B features that have X64 in the name.
+ * On x86/x64 these are HYPERV_CPUID_FEATURES.EBX bits.
*/
-#define HV_X64_CREATE_PARTITIONS BIT(0)
-#define HV_X64_ACCESS_PARTITION_ID BIT(1)
-#define HV_X64_ACCESS_MEMORY_POOL BIT(2)
-#define HV_X64_ADJUST_MESSAGE_BUFFERS BIT(3)
-#define HV_X64_POST_MESSAGES BIT(4)
-#define HV_X64_SIGNAL_EVENTS BIT(5)
-#define HV_X64_CREATE_PORT BIT(6)
-#define HV_X64_CONNECT_PORT BIT(7)
-#define HV_X64_ACCESS_STATS BIT(8)
-#define HV_X64_DEBUGGING BIT(11)
-#define HV_X64_CPU_POWER_MANAGEMENT BIT(12)
+#define HV_X64_POST_MESSAGES HV_POST_MESSAGES
+#define HV_X64_SIGNAL_EVENTS HV_SIGNAL_EVENTS
/*
- * Feature identification. EDX indicates which miscellaneous features
- * are available to the partition.
- * These are HYPERV_CPUID_FEATURES.EDX bits.
+ * Group D Features. The bit assignments are custom to each architecture.
+ * On x86/x64 these are HYPERV_CPUID_FEATURES.EDX bits.
*/
/* The MWAIT instruction is available (per section MONITOR / MWAIT) */
#define HV_X64_MWAIT_AVAILABLE BIT(0)
@@ -187,7 +139,7 @@
* processor, except for virtual processors that are reported as sibling SMT
* threads.
*/
-#define HV_X64_NO_NONARCH_CORESHARING BIT(18)
+#define HV_X64_NO_NONARCH_CORESHARING BIT(18)
/* Nested features. These are HYPERV_CPUID_NESTED_FEATURES.EAX bits. */
#define HV_X64_NESTED_DIRECT_FLUSH BIT(17)
@@ -295,42 +247,6 @@ union hv_x64_msr_hypercall_contents {
} __packed;
};
-/*
- * TSC page layout.
- */
-struct ms_hyperv_tsc_page {
- volatile u32 tsc_sequence;
- u32 reserved1;
- volatile u64 tsc_scale;
- volatile s64 tsc_offset;
-} __packed;
-
-/*
- * The guest OS needs to register the guest ID with the hypervisor.
- * The guest ID is a 64 bit entity and the structure of this ID is
- * specified in the Hyper-V specification:
- *
- * msdn.microsoft.com/en-us/library/windows/hardware/ff542653%28v=vs.85%29.aspx
- *
- * While the current guideline does not specify how Linux guest ID(s)
- * need to be generated, our plan is to publish the guidelines for
- * Linux and other guest operating systems that currently are hosted
- * on Hyper-V. The implementation here conforms to this yet
- * unpublished guidelines.
- *
- *
- * Bit(s)
- * 63 - Indicates if the OS is Open Source or not; 1 is Open Source
- * 62:56 - Os Type; Linux is 0x100
- * 55:48 - Distro specific identification
- * 47:16 - Linux kernel version number
- * 15:0 - Distro specific identification
- *
- *
- */
-
-#define HV_LINUX_VENDOR_ID 0x8100
-
struct hv_reenlightenment_control {
__u64 vector:8;
__u64 reserved1:8;
@@ -354,31 +270,12 @@ struct hv_tsc_emulation_status {
#define HV_X64_MSR_HYPERCALL_PAGE_ADDRESS_MASK \
(~((1ull << HV_X64_MSR_HYPERCALL_PAGE_ADDRESS_SHIFT) - 1))
-/*
- * Crash notification (HV_X64_MSR_CRASH_CTL) flags.
- */
-#define HV_CRASH_CTL_CRASH_NOTIFY_MSG BIT_ULL(62)
-#define HV_CRASH_CTL_CRASH_NOTIFY BIT_ULL(63)
#define HV_X64_MSR_CRASH_PARAMS \
(1 + (HV_X64_MSR_CRASH_P4 - HV_X64_MSR_CRASH_P0))
#define HV_IPI_LOW_VECTOR 0x10
#define HV_IPI_HIGH_VECTOR 0xff
-/* Declare the various hypercall operations. */
-#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE 0x0002
-#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST 0x0003
-#define HVCALL_NOTIFY_LONG_SPIN_WAIT 0x0008
-#define HVCALL_SEND_IPI 0x000b
-#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE_EX 0x0013
-#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST_EX 0x0014
-#define HVCALL_SEND_IPI_EX 0x0015
-#define HVCALL_POST_MESSAGE 0x005c
-#define HVCALL_SIGNAL_EVENT 0x005d
-#define HVCALL_RETARGET_INTERRUPT 0x007e
-#define HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_SPACE 0x00af
-#define HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_LIST 0x00b0
-
#define HV_X64_MSR_VP_ASSIST_PAGE_ENABLE 0x00000001
#define HV_X64_MSR_VP_ASSIST_PAGE_ADDRESS_SHIFT 12
#define HV_X64_MSR_VP_ASSIST_PAGE_ADDRESS_MASK \
@@ -390,63 +287,6 @@ struct hv_tsc_emulation_status {
#define HV_X64_MSR_TSC_REFERENCE_ENABLE 0x00000001
#define HV_X64_MSR_TSC_REFERENCE_ADDRESS_SHIFT 12
-#define HV_FLUSH_ALL_PROCESSORS BIT(0)
-#define HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES BIT(1)
-#define HV_FLUSH_NON_GLOBAL_MAPPINGS_ONLY BIT(2)
-#define HV_FLUSH_USE_EXTENDED_RANGE_FORMAT BIT(3)
-
-enum HV_GENERIC_SET_FORMAT {
- HV_GENERIC_SET_SPARSE_4K,
- HV_GENERIC_SET_ALL,
-};
-
-#define HV_PARTITION_ID_SELF ((u64)-1)
-
-#define HV_HYPERCALL_RESULT_MASK GENMASK_ULL(15, 0)
-#define HV_HYPERCALL_FAST_BIT BIT(16)
-#define HV_HYPERCALL_VARHEAD_OFFSET 17
-#define HV_HYPERCALL_REP_COMP_OFFSET 32
-#define HV_HYPERCALL_REP_COMP_MASK GENMASK_ULL(43, 32)
-#define HV_HYPERCALL_REP_START_OFFSET 48
-#define HV_HYPERCALL_REP_START_MASK GENMASK_ULL(59, 48)
-
-/* hypercall status code */
-#define HV_STATUS_SUCCESS 0
-#define HV_STATUS_INVALID_HYPERCALL_CODE 2
-#define HV_STATUS_INVALID_HYPERCALL_INPUT 3
-#define HV_STATUS_INVALID_ALIGNMENT 4
-#define HV_STATUS_INVALID_PARAMETER 5
-#define HV_STATUS_INSUFFICIENT_MEMORY 11
-#define HV_STATUS_INVALID_PORT_ID 17
-#define HV_STATUS_INVALID_CONNECTION_ID 18
-#define HV_STATUS_INSUFFICIENT_BUFFERS 19
-
-/*
- * The Hyper-V TimeRefCount register and the TSC
- * page provide a guest VM clock with 100ns tick rate
- */
-#define HV_CLOCK_HZ (NSEC_PER_SEC/100)
-
-/* Define the number of synthetic interrupt sources. */
-#define HV_SYNIC_SINT_COUNT (16)
-/* Define the expected SynIC version. */
-#define HV_SYNIC_VERSION_1 (0x1)
-/* Valid SynIC vectors are 16-255. */
-#define HV_SYNIC_FIRST_VALID_VECTOR (16)
-
-#define HV_SYNIC_CONTROL_ENABLE (1ULL << 0)
-#define HV_SYNIC_SIMP_ENABLE (1ULL << 0)
-#define HV_SYNIC_SIEFP_ENABLE (1ULL << 0)
-#define HV_SYNIC_SINT_MASKED (1ULL << 16)
-#define HV_SYNIC_SINT_AUTO_EOI (1ULL << 17)
-#define HV_SYNIC_SINT_VECTOR_MASK (0xFF)
-
-#define HV_SYNIC_STIMER_COUNT (4)
-
-/* Define synthetic interrupt controller message constants. */
-#define HV_MESSAGE_SIZE (256)
-#define HV_MESSAGE_PAYLOAD_BYTE_COUNT (240)
-#define HV_MESSAGE_PAYLOAD_QWORD_COUNT (30)
/* Define hypervisor message types. */
enum hv_message_type {
@@ -457,76 +297,25 @@ enum hv_message_type {
HVMSG_GPA_INTERCEPT = 0x80000001,
/* Timer notification messages. */
- HVMSG_TIMER_EXPIRED = 0x80000010,
+ HVMSG_TIMER_EXPIRED = 0x80000010,
/* Error messages. */
HVMSG_INVALID_VP_REGISTER_VALUE = 0x80000020,
HVMSG_UNRECOVERABLE_EXCEPTION = 0x80000021,
- HVMSG_UNSUPPORTED_FEATURE = 0x80000022,
+ HVMSG_UNSUPPORTED_FEATURE = 0x80000022,
/* Trace buffer complete messages. */
HVMSG_EVENTLOG_BUFFERCOMPLETE = 0x80000040,
/* Platform-specific processor intercept messages. */
- HVMSG_X64_IOPORT_INTERCEPT = 0x80010000,
+ HVMSG_X64_IOPORT_INTERCEPT = 0x80010000,
HVMSG_X64_MSR_INTERCEPT = 0x80010001,
- HVMSG_X64_CPUID_INTERCEPT = 0x80010002,
+ HVMSG_X64_CPUID_INTERCEPT = 0x80010002,
HVMSG_X64_EXCEPTION_INTERCEPT = 0x80010003,
- HVMSG_X64_APIC_EOI = 0x80010004,
- HVMSG_X64_LEGACY_FP_ERROR = 0x80010005
+ HVMSG_X64_APIC_EOI = 0x80010004,
+ HVMSG_X64_LEGACY_FP_ERROR = 0x80010005
};
-/* Define synthetic interrupt controller message flags. */
-union hv_message_flags {
- __u8 asu8;
- struct {
- __u8 msg_pending:1;
- __u8 reserved:7;
- } __packed;
-};
-
-/* Define port identifier type. */
-union hv_port_id {
- __u32 asu32;
- struct {
- __u32 id:24;
- __u32 reserved:8;
- } __packed u;
-};
-
-/* Define synthetic interrupt controller message header. */
-struct hv_message_header {
- __u32 message_type;
- __u8 payload_size;
- union hv_message_flags message_flags;
- __u8 reserved[2];
- union {
- __u64 sender;
- union hv_port_id port;
- };
-} __packed;
-
-/* Define synthetic interrupt controller message format. */
-struct hv_message {
- struct hv_message_header header;
- union {
- __u64 payload[HV_MESSAGE_PAYLOAD_QWORD_COUNT];
- } u;
-} __packed;
-
-/* Define the synthetic interrupt message page layout. */
-struct hv_message_page {
- struct hv_message sint_message[HV_SYNIC_SINT_COUNT];
-} __packed;
-
-/* Define timer message payload structure. */
-struct hv_timer_message_payload {
- __u32 timer_index;
- __u32 reserved;
- __u64 expiration_time; /* When the timer expired */
- __u64 delivery_time; /* When the message was delivered */
-} __packed;
-
struct hv_nested_enlightenments_control {
struct {
__u32 directhypercall:1;
@@ -754,187 +543,11 @@ struct hv_enlightened_vmcs {
#define HV_VMX_ENLIGHTENED_CLEAN_FIELD_ALL 0xFFFF
-/* Define synthetic interrupt controller flag constants. */
-#define HV_EVENT_FLAGS_COUNT (256 * 8)
-#define HV_EVENT_FLAGS_LONG_COUNT (256 / sizeof(unsigned long))
-
-/*
- * Synthetic timer configuration.
- */
-union hv_stimer_config {
- u64 as_uint64;
- struct {
- u64 enable:1;
- u64 periodic:1;
- u64 lazy:1;
- u64 auto_enable:1;
- u64 apic_vector:8;
- u64 direct_mode:1;
- u64 reserved_z0:3;
- u64 sintx:4;
- u64 reserved_z1:44;
- } __packed;
-};
-
-
-/* Define the synthetic interrupt controller event flags format. */
-union hv_synic_event_flags {
- unsigned long flags[HV_EVENT_FLAGS_LONG_COUNT];
-};
-
-/* Define SynIC control register. */
-union hv_synic_scontrol {
- u64 as_uint64;
- struct {
- u64 enable:1;
- u64 reserved:63;
- } __packed;
-};
-
-/* Define synthetic interrupt source. */
-union hv_synic_sint {
- u64 as_uint64;
- struct {
- u64 vector:8;
- u64 reserved1:8;
- u64 masked:1;
- u64 auto_eoi:1;
- u64 polling:1;
- u64 reserved2:45;
- } __packed;
-};
-
-/* Define the format of the SIMP register */
-union hv_synic_simp {
- u64 as_uint64;
- struct {
- u64 simp_enabled:1;
- u64 preserved:11;
- u64 base_simp_gpa:52;
- } __packed;
-};
-
-/* Define the format of the SIEFP register */
-union hv_synic_siefp {
- u64 as_uint64;
- struct {
- u64 siefp_enabled:1;
- u64 preserved:11;
- u64 base_siefp_gpa:52;
- } __packed;
-};
-
-struct hv_vpset {
- u64 format;
- u64 valid_bank_mask;
- u64 bank_contents[];
-} __packed;
-
-/* HvCallSendSyntheticClusterIpi hypercall */
-struct hv_send_ipi {
- u32 vector;
- u32 reserved;
- u64 cpu_mask;
-} __packed;
-
-/* HvCallSendSyntheticClusterIpiEx hypercall */
-struct hv_send_ipi_ex {
- u32 vector;
- u32 reserved;
- struct hv_vpset vp_set;
-} __packed;
-
-/* HvFlushGuestPhysicalAddressSpace hypercalls */
-struct hv_guest_mapping_flush {
- u64 address_space;
- u64 flags;
-} __packed;
-
-/*
- * HV_MAX_FLUSH_PAGES = "additional_pages" + 1. It's limited
- * by the bitwidth of "additional_pages" in union hv_gpa_page_range.
- */
-#define HV_MAX_FLUSH_PAGES (2048)
-
-/* HvFlushGuestPhysicalAddressList hypercall */
-union hv_gpa_page_range {
- u64 address_space;
- struct {
- u64 additional_pages:11;
- u64 largepage:1;
- u64 basepfn:52;
- } page;
-};
-
-/*
- * All input flush parameters should be in single page. The max flush
- * count is equal with how many entries of union hv_gpa_page_range can
- * be populated into the input parameter page.
- */
-#define HV_MAX_FLUSH_REP_COUNT ((HV_HYP_PAGE_SIZE - 2 * sizeof(u64)) / \
- sizeof(union hv_gpa_page_range))
-
-struct hv_guest_mapping_flush_list {
- u64 address_space;
- u64 flags;
- union hv_gpa_page_range gpa_list[HV_MAX_FLUSH_REP_COUNT];
-};
-
-/* HvFlushVirtualAddressSpace, HvFlushVirtualAddressList hypercalls */
-struct hv_tlb_flush {
- u64 address_space;
- u64 flags;
- u64 processor_mask;
- u64 gva_list[];
-} __packed;
-
-/* HvFlushVirtualAddressSpaceEx, HvFlushVirtualAddressListEx hypercalls */
-struct hv_tlb_flush_ex {
- u64 address_space;
- u64 flags;
- struct hv_vpset hv_vp_set;
- u64 gva_list[];
-} __packed;
-
struct hv_partition_assist_pg {
u32 tlb_lock_count;
};
-union hv_msi_entry {
- u64 as_uint64;
- struct {
- u32 address;
- u32 data;
- } __packed;
-};
-
-struct hv_interrupt_entry {
- u32 source; /* 1 for MSI(-X) */
- u32 reserved1;
- union hv_msi_entry msi_entry;
-} __packed;
-/*
- * flags for hv_device_interrupt_target.flags
- */
-#define HV_DEVICE_INTERRUPT_TARGET_MULTICAST 1
-#define HV_DEVICE_INTERRUPT_TARGET_PROCESSOR_SET 2
-
-struct hv_device_interrupt_target {
- u32 vector;
- u32 flags;
- union {
- u64 vp_mask;
- struct hv_vpset vp_set;
- };
-} __packed;
+#include <asm-generic/hyperv-tlfs.h>
-/* HvRetargetDeviceInterrupt hypercall */
-struct hv_retarget_device_interrupt {
- u64 partition_id; /* use "self" */
- u64 device_id;
- struct hv_interrupt_entry int_entry;
- u64 reserved2;
- struct hv_device_interrupt_target int_target;
-} __packed __aligned(8);
#endif
diff --git a/include/asm-generic/hyperv-tlfs.h b/include/asm-generic/hyperv-tlfs.h
new file mode 100644
index 000000000000..1f92ef92eb56
--- /dev/null
+++ b/include/asm-generic/hyperv-tlfs.h
@@ -0,0 +1,442 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * This file contains definitions from Hyper-V Hypervisor Top-Level Functional
+ * Specification (TLFS):
+ * https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs
+ */
+
+#ifndef _ASM_GENERIC_HYPERV_TLFS_H
+#define _ASM_GENERIC_HYPERV_TLFS_H
+
+#include <linux/types.h>
+#include <linux/bits.h>
+#include <linux/time64.h>
+
+/*
+ * While not explicitly listed in the TLFS, Hyper-V always runs with a page size
+ * of 4096. These definitions are used when communicating with Hyper-V using
+ * guest physical pages and guest physical page addresses, since the guest page
+ * size may not be 4096 on all architectures.
+ */
+#define HV_HYP_PAGE_SHIFT 12
+#define HV_HYP_PAGE_SIZE BIT(HV_HYP_PAGE_SHIFT)
+#define HV_HYP_PAGE_MASK (~(HV_HYP_PAGE_SIZE - 1))
+
+/*
+ * Hyper-V provides two categories of flags relevant to guest VMs. The
+ * "Features" category indicates specific functionality that is available
+ * to guests on this particular instance of Hyper-V. The "Features"
+ * are presented in four groups, each of which is 32 bits. The group A
+ * and B definitions are common across architectures and are listed here.
+ * However, not all flags are relevant on all architectures.
+ *
+ * Groups C and D vary across architectures and are listed in the
+ * architecture specific portion of hyperv-tlfs.h. Some of these flags exist
+ * on multiple architectures, but the bit positions are different so they
+ * cannot appear in the generic portion of hyperv-tlfs.h.
+ *
+ * The "Enlightenments" category provides recommendations on whether to use
+ * specific enlightenments that are available. The Enlighenments are a single
+ * group of 32 bits, but they vary across architectures and are listed in
+ * the architecture specific portion of hyperv-tlfs.h.
+ */
+
+/*
+ * Group A Features.
+ */
+
+/* VP Runtime register available */
+#define HV_MSR_VP_RUNTIME_AVAILABLE BIT(0)
+/* Partition Reference Counter available*/
+#define HV_MSR_TIME_REF_COUNT_AVAILABLE BIT(1)
+/* Basic SynIC register available */
+#define HV_MSR_SYNIC_AVAILABLE BIT(2)
+/* Synthetic Timer registers available */
+#define HV_MSR_SYNTIMER_AVAILABLE BIT(3)
+/* Virtual APIC assist and VP assist page registers available */
+#define HV_MSR_APIC_ACCESS_AVAILABLE BIT(4)
+/* Hypercall and Guest OS ID registers available*/
+#define HV_MSR_HYPERCALL_AVAILABLE BIT(5)
+/* Access virtual processor index register available*/
+#define HV_MSR_VP_INDEX_AVAILABLE BIT(6)
+/* Virtual system reset register available*/
+#define HV_MSR_RESET_AVAILABLE BIT(7)
+/* Access statistics page registers available */
+#define HV_MSR_STAT_PAGES_AVAILABLE BIT(8)
+/* Partition reference TSC register is available */
+#define HV_MSR_REFERENCE_TSC_AVAILABLE BIT(9)
+/* Partition Guest IDLE register is available */
+#define HV_MSR_GUEST_IDLE_AVAILABLE BIT(10)
+/* Partition local APIC and TSC frequency registers available */
+#define HV_ACCESS_FREQUENCY_MSRS BIT(11)
+/* AccessReenlightenmentControls privilege */
+#define HV_ACCESS_REENLIGHTENMENT BIT(13)
+/* AccessTscInvariantControls privilege */
+#define HV_ACCESS_TSC_INVARIANT BIT(15)
+
+/*
+ * Group B features.
+ */
+#define HV_CREATE_PARTITIONS BIT(0)
+#define HV_ACCESS_PARTITION_ID BIT(1)
+#define HV_ACCESS_MEMORY_POOL BIT(2)
+#define HV_ADJUST_MESSAGE_BUFFERS BIT(3)
+#define HV_POST_MESSAGES BIT(4)
+#define HV_SIGNAL_EVENTS BIT(5)
+#define HV_CREATE_PORT BIT(6)
+#define HV_CONNECT_PORT BIT(7)
+#define HV_ACCESS_STATS BIT(8)
+#define HV_DEBUGGING BIT(11)
+#define HV_CPU_POWER_MANAGEMENT BIT(12)
+
+
+/*
+ * TSC page layout.
+ */
+struct ms_hyperv_tsc_page {
+ volatile u32 tsc_sequence;
+ u32 reserved1;
+ volatile u64 tsc_scale;
+ volatile s64 tsc_offset;
+} __packed;
+
+/*
+ * The guest OS needs to register the guest ID with the hypervisor.
+ * The guest ID is a 64 bit entity and the structure of this ID is
+ * specified in the Hyper-V specification:
+ *
+ * msdn.microsoft.com/en-us/library/windows/hardware/ff542653%28v=vs.85%29.aspx
+ *
+ * While the current guideline does not specify how Linux guest ID(s)
+ * need to be generated, our plan is to publish the guidelines for
+ * Linux and other guest operating systems that currently are hosted
+ * on Hyper-V. The implementation here conforms to this yet
+ * unpublished guidelines.
+ *
+ *
+ * Bit(s)
+ * 63 - Indicates if the OS is Open Source or not; 1 is Open Source
+ * 62:56 - Os Type; Linux is 0x100
+ * 55:48 - Distro specific identification
+ * 47:16 - Linux kernel version number
+ * 15:0 - Distro specific identification
+ *
+ *
+ */
+
+#define HV_LINUX_VENDOR_ID 0x8100
+
+/*
+ * Crash notification flags.
+ */
+#define HV_CRASH_CTL_CRASH_NOTIFY_MSG BIT_ULL(62)
+#define HV_CRASH_CTL_CRASH_NOTIFY BIT_ULL(63)
+
+/* Declare the various hypercall operations. */
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE 0x0002
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST 0x0003
+#define HVCALL_NOTIFY_LONG_SPIN_WAIT 0x0008
+#define HVCALL_SEND_IPI 0x000b
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE_EX 0x0013
+#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST_EX 0x0014
+#define HVCALL_SEND_IPI_EX 0x0015
+#define HVCALL_POST_MESSAGE 0x005c
+#define HVCALL_SIGNAL_EVENT 0x005d
+#define HVCALL_RETARGET_INTERRUPT 0x007e
+#define HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_SPACE 0x00af
+#define HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_LIST 0x00b0
+
+#define HV_FLUSH_ALL_PROCESSORS BIT(0)
+#define HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES BIT(1)
+#define HV_FLUSH_NON_GLOBAL_MAPPINGS_ONLY BIT(2)
+#define HV_FLUSH_USE_EXTENDED_RANGE_FORMAT BIT(3)
+
+enum HV_GENERIC_SET_FORMAT {
+ HV_GENERIC_SET_SPARSE_4K,
+ HV_GENERIC_SET_ALL,
+};
+
+#define HV_PARTITION_ID_SELF ((u64)-1)
+#define HV_VP_INDEX_SELF ((u32)-2)
+
+#define HV_HYPERCALL_RESULT_MASK GENMASK_ULL(15, 0)
+#define HV_HYPERCALL_FAST_BIT BIT(16)
+#define HV_HYPERCALL_VARHEAD_OFFSET 17
+#define HV_HYPERCALL_REP_COMP_OFFSET 32
+#define HV_HYPERCALL_REP_COMP_1 BIT_ULL(32)
+#define HV_HYPERCALL_REP_COMP_MASK GENMASK_ULL(43, 32)
+#define HV_HYPERCALL_REP_START_OFFSET 48
+#define HV_HYPERCALL_REP_START_MASK GENMASK_ULL(59, 48)
+
+/* hypercall status code */
+#define HV_STATUS_SUCCESS 0
+#define HV_STATUS_INVALID_HYPERCALL_CODE 2
+#define HV_STATUS_INVALID_HYPERCALL_INPUT 3
+#define HV_STATUS_INVALID_ALIGNMENT 4
+#define HV_STATUS_INVALID_PARAMETER 5
+#define HV_STATUS_INSUFFICIENT_MEMORY 11
+#define HV_STATUS_INVALID_PORT_ID 17
+#define HV_STATUS_INVALID_CONNECTION_ID 18
+#define HV_STATUS_INSUFFICIENT_BUFFERS 19
+
+/*
+ * The Hyper-V TimeRefCount register and the TSC
+ * page provide a guest VM clock with 100ns tick rate
+ */
+#define HV_CLOCK_HZ (NSEC_PER_SEC/100)
+
+/* Define the number of synthetic interrupt sources. */
+#define HV_SYNIC_SINT_COUNT (16)
+/* Define the expected SynIC version. */
+#define HV_SYNIC_VERSION_1 (0x1)
+/* Valid SynIC vectors are 16-255. */
+#define HV_SYNIC_FIRST_VALID_VECTOR (16)
+
+#define HV_SYNIC_CONTROL_ENABLE (1ULL << 0)
+#define HV_SYNIC_SIMP_ENABLE (1ULL << 0)
+#define HV_SYNIC_SIEFP_ENABLE (1ULL << 0)
+#define HV_SYNIC_SINT_MASKED (1ULL << 16)
+#define HV_SYNIC_SINT_AUTO_EOI (1ULL << 17)
+#define HV_SYNIC_SINT_VECTOR_MASK (0xFF)
+
+#define HV_SYNIC_STIMER_COUNT (4)
+
+/* Define synthetic interrupt controller message constants. */
+#define HV_MESSAGE_SIZE (256)
+#define HV_MESSAGE_PAYLOAD_BYTE_COUNT (240)
+#define HV_MESSAGE_PAYLOAD_QWORD_COUNT (30)
+
+/* Define synthetic interrupt controller message flags. */
+union hv_message_flags {
+ __u8 asu8;
+ struct {
+ __u8 msg_pending:1;
+ __u8 reserved:7;
+ } __packed;
+};
+
+/* Define port identifier type. */
+union hv_port_id {
+ __u32 asu32;
+ struct {
+ __u32 id:24;
+ __u32 reserved:8;
+ } __packed u;
+};
+
+/* Define synthetic interrupt controller message header. */
+struct hv_message_header {
+ __u32 message_type;
+ __u8 payload_size;
+ union hv_message_flags message_flags;
+ __u8 reserved[2];
+ union {
+ __u64 sender;
+ union hv_port_id port;
+ };
+} __packed;
+
+/* Define synthetic interrupt controller message format. */
+struct hv_message {
+ struct hv_message_header header;
+ union {
+ __u64 payload[HV_MESSAGE_PAYLOAD_QWORD_COUNT];
+ } u;
+} __packed;
+
+/* Define the synthetic interrupt message page layout. */
+struct hv_message_page {
+ struct hv_message sint_message[HV_SYNIC_SINT_COUNT];
+} __packed;
+
+/* Define timer message payload structure. */
+struct hv_timer_message_payload {
+ __u32 timer_index;
+ __u32 reserved;
+ __u64 expiration_time; /* When the timer expired */
+ __u64 delivery_time; /* When the message was delivered */
+} __packed;
+
+
+/* Define synthetic interrupt controller flag constants. */
+#define HV_EVENT_FLAGS_COUNT (256 * 8)
+#define HV_EVENT_FLAGS_LONG_COUNT (256 / sizeof(unsigned long))
+
+/*
+ * Synthetic timer configuration.
+ */
+union hv_stimer_config {
+ u64 as_uint64;
+ struct {
+ u64 enable:1;
+ u64 periodic:1;
+ u64 lazy:1;
+ u64 auto_enable:1;
+ u64 apic_vector:8;
+ u64 direct_mode:1;
+ u64 reserved_z0:3;
+ u64 sintx:4;
+ u64 reserved_z1:44;
+ } __packed;
+};
+
+
+/* Define the synthetic interrupt controller event flags format. */
+union hv_synic_event_flags {
+ unsigned long flags[HV_EVENT_FLAGS_LONG_COUNT];
+};
+
+/* Define SynIC control register. */
+union hv_synic_scontrol {
+ u64 as_uint64;
+ struct {
+ u64 enable:1;
+ u64 reserved:63;
+ } __packed;
+};
+
+/* Define synthetic interrupt source. */
+union hv_synic_sint {
+ u64 as_uint64;
+ struct {
+ u64 vector:8;
+ u64 reserved1:8;
+ u64 masked:1;
+ u64 auto_eoi:1;
+ u64 polling:1;
+ u64 reserved2:45;
+ } __packed;
+};
+
+/* Define the format of the SIMP register */
+union hv_synic_simp {
+ u64 as_uint64;
+ struct {
+ u64 simp_enabled:1;
+ u64 preserved:11;
+ u64 base_simp_gpa:52;
+ } __packed;
+};
+
+/* Define the format of the SIEFP register */
+union hv_synic_siefp {
+ u64 as_uint64;
+ struct {
+ u64 siefp_enabled:1;
+ u64 preserved:11;
+ u64 base_siefp_gpa:52;
+ } __packed;
+};
+
+struct hv_vpset {
+ u64 format;
+ u64 valid_bank_mask;
+ u64 bank_contents[];
+} __packed;
+
+/* HvCallSendSyntheticClusterIpi hypercall */
+struct hv_send_ipi {
+ u32 vector;
+ u32 reserved;
+ u64 cpu_mask;
+} __packed;
+
+/* HvCallSendSyntheticClusterIpiEx hypercall */
+struct hv_send_ipi_ex {
+ u32 vector;
+ u32 reserved;
+ struct hv_vpset vp_set;
+} __packed;
+
+/* HvFlushGuestPhysicalAddressSpace hypercalls */
+struct hv_guest_mapping_flush {
+ u64 address_space;
+ u64 flags;
+} __packed;
+
+/*
+ * HV_MAX_FLUSH_PAGES = "additional_pages" + 1. It's limited
+ * by the bitwidth of "additional_pages" in union hv_gpa_page_range.
+ */
+#define HV_MAX_FLUSH_PAGES (2048)
+
+/* HvFlushGuestPhysicalAddressList hypercall */
+union hv_gpa_page_range {
+ u64 address_space;
+ struct {
+ u64 additional_pages:11;
+ u64 largepage:1;
+ u64 basepfn:52;
+ } page;
+};
+
+/*
+ * All input flush parameters should be in single page. The max flush
+ * count is equal with how many entries of union hv_gpa_page_range can
+ * be populated into the input parameter page.
+ */
+#define HV_MAX_FLUSH_REP_COUNT ((HV_HYP_PAGE_SIZE - 2 * sizeof(u64)) / \
+ sizeof(union hv_gpa_page_range))
+
+struct hv_guest_mapping_flush_list {
+ u64 address_space;
+ u64 flags;
+ union hv_gpa_page_range gpa_list[HV_MAX_FLUSH_REP_COUNT];
+};
+
+/* HvFlushVirtualAddressSpace, HvFlushVirtualAddressList hypercalls */
+struct hv_tlb_flush {
+ u64 address_space;
+ u64 flags;
+ u64 processor_mask;
+ u64 gva_list[];
+} __packed;
+
+/* HvFlushVirtualAddressSpaceEx, HvFlushVirtualAddressListEx hypercalls */
+struct hv_tlb_flush_ex {
+ u64 address_space;
+ u64 flags;
+ struct hv_vpset hv_vp_set;
+ u64 gva_list[];
+} __packed;
+
+/* HvRetargetDeviceInterrupt hypercall */
+union hv_msi_entry {
+ u64 as_uint64;
+ struct {
+ u32 address;
+ u32 data;
+ } __packed;
+};
+
+struct hv_interrupt_entry {
+ u32 source; /* 1 for MSI(-X) */
+ u32 reserved1;
+ union hv_msi_entry msi_entry;
+} __packed;
+
+/*
+ * flags for hv_device_interrupt_target.flags
+ */
+#define HV_DEVICE_INTERRUPT_TARGET_MULTICAST 1
+#define HV_DEVICE_INTERRUPT_TARGET_PROCESSOR_SET 2
+
+struct hv_device_interrupt_target {
+ u32 vector;
+ u32 flags;
+ union {
+ u64 vp_mask;
+ struct hv_vpset vp_set;
+ };
+} __packed;
+
+struct hv_retarget_device_interrupt {
+ u64 partition_id; /* use "self" */
+ u64 device_id;
+ struct hv_interrupt_entry int_entry;
+ u64 reserved2;
+ struct hv_device_interrupt_target int_target;
+} __packed __aligned(8);
+
+#endif
--
2.18.2
^ permalink raw reply related
* [PATCH 2/4] x86/hyperv: Remove HV_PROCESSOR_POWER_STATE #defines
From: Michael Kelley @ 2020-04-20 17:38 UTC (permalink / raw)
To: kys, haiyangz, sthemmin, wei.liu, tglx, mingo, bp, x86, hpa,
pbonzini, sean.j.christopherson, vkuznets, wanpengli, jmattson,
joro, kvm, linux-kernel, linux-hyperv
Cc: mikelley
In-Reply-To: <20200420173838.24672-1-mikelley@microsoft.com>
The HV_PROCESSOR_POWER_STATE_C<n> #defines date back to year 2010,
but they are not in the TLFS v6.0 document and are not used anywhere
in Linux. Remove them.
Signed-off-by: Michael Kelley <mikelley@microsoft.com>
---
arch/x86/include/asm/hyperv-tlfs.h | 5 -----
1 file changed, 5 deletions(-)
diff --git a/arch/x86/include/asm/hyperv-tlfs.h b/arch/x86/include/asm/hyperv-tlfs.h
index 0e4d76920957..2dd1ceb2bcf8 100644
--- a/arch/x86/include/asm/hyperv-tlfs.h
+++ b/arch/x86/include/asm/hyperv-tlfs.h
@@ -390,11 +390,6 @@ struct hv_tsc_emulation_status {
#define HV_X64_MSR_TSC_REFERENCE_ENABLE 0x00000001
#define HV_X64_MSR_TSC_REFERENCE_ADDRESS_SHIFT 12
-#define HV_PROCESSOR_POWER_STATE_C0 0
-#define HV_PROCESSOR_POWER_STATE_C1 1
-#define HV_PROCESSOR_POWER_STATE_C2 2
-#define HV_PROCESSOR_POWER_STATE_C3 3
-
#define HV_FLUSH_ALL_PROCESSORS BIT(0)
#define HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES BIT(1)
#define HV_FLUSH_NON_GLOBAL_MAPPINGS_ONLY BIT(2)
--
2.18.2
^ permalink raw reply related
* [PATCH 1/4] KVM: x86: hyperv: Remove duplicate definitions of Reference TSC Page
From: Michael Kelley @ 2020-04-20 17:38 UTC (permalink / raw)
To: kys, haiyangz, sthemmin, wei.liu, tglx, mingo, bp, x86, hpa,
pbonzini, sean.j.christopherson, vkuznets, wanpengli, jmattson,
joro, kvm, linux-kernel, linux-hyperv
Cc: mikelley
In-Reply-To: <20200420173838.24672-1-mikelley@microsoft.com>
The Hyper-V Reference TSC Page structure is defined twice. struct
ms_hyperv_tsc_page has padding out to a full 4 Kbyte page size. But
the padding is not needed because the declaration includes a union
with HV_HYP_PAGE_SIZE. KVM uses the second definition, which is
struct _HV_REFERENCE_TSC_PAGE, because it does not have the padding.
Fix the duplication by removing the padding from ms_hyperv_tsc_page.
Fix up the KVM code to use it. Remove the no longer used struct
_HV_REFERENCE_TSC_PAGE.
There is no functional change.
Signed-off-by: Michael Kelley <mikelley@microsoft.com>
---
arch/x86/include/asm/hyperv-tlfs.h | 8 --------
arch/x86/include/asm/kvm_host.h | 2 +-
arch/x86/kvm/hyperv.c | 4 ++--
3 files changed, 3 insertions(+), 11 deletions(-)
diff --git a/arch/x86/include/asm/hyperv-tlfs.h b/arch/x86/include/asm/hyperv-tlfs.h
index 29336574d0bc..0e4d76920957 100644
--- a/arch/x86/include/asm/hyperv-tlfs.h
+++ b/arch/x86/include/asm/hyperv-tlfs.h
@@ -303,7 +303,6 @@ struct ms_hyperv_tsc_page {
u32 reserved1;
volatile u64 tsc_scale;
volatile s64 tsc_offset;
- u64 reserved2[509];
} __packed;
/*
@@ -433,13 +432,6 @@ enum HV_GENERIC_SET_FORMAT {
*/
#define HV_CLOCK_HZ (NSEC_PER_SEC/100)
-typedef struct _HV_REFERENCE_TSC_PAGE {
- __u32 tsc_sequence;
- __u32 res1;
- __u64 tsc_scale;
- __s64 tsc_offset;
-} __packed HV_REFERENCE_TSC_PAGE, *PHV_REFERENCE_TSC_PAGE;
-
/* Define the number of synthetic interrupt sources. */
#define HV_SYNIC_SINT_COUNT (16)
/* Define the expected SynIC version. */
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 42a2d0d3984a..4698343b9a05 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -865,7 +865,7 @@ struct kvm_hv {
u64 hv_crash_param[HV_X64_MSR_CRASH_PARAMS];
u64 hv_crash_ctl;
- HV_REFERENCE_TSC_PAGE tsc_ref;
+ struct ms_hyperv_tsc_page tsc_ref;
struct idr conn_to_evt;
diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index bcefa9d4e57e..1f3c6fd3cdaa 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -900,7 +900,7 @@ static int kvm_hv_msr_set_crash_data(struct kvm_vcpu *vcpu,
* These two equivalencies are implemented in this function.
*/
static bool compute_tsc_page_parameters(struct pvclock_vcpu_time_info *hv_clock,
- HV_REFERENCE_TSC_PAGE *tsc_ref)
+ struct ms_hyperv_tsc_page *tsc_ref)
{
u64 max_mul;
@@ -941,7 +941,7 @@ void kvm_hv_setup_tsc_page(struct kvm *kvm,
u64 gfn;
BUILD_BUG_ON(sizeof(tsc_seq) != sizeof(hv->tsc_ref.tsc_sequence));
- BUILD_BUG_ON(offsetof(HV_REFERENCE_TSC_PAGE, tsc_sequence) != 0);
+ BUILD_BUG_ON(offsetof(struct ms_hyperv_tsc_page, tsc_sequence) != 0);
if (!(hv->hv_tsc_page & HV_X64_MSR_TSC_REFERENCE_ENABLE))
return;
--
2.18.2
^ permalink raw reply related
* [PATCH 4/4] asm-generic/hyperv: Add definitions for Get/SetVpRegister hypercalls
From: Michael Kelley @ 2020-04-20 17:38 UTC (permalink / raw)
To: kys, haiyangz, sthemmin, wei.liu, tglx, mingo, bp, x86, hpa,
pbonzini, sean.j.christopherson, vkuznets, wanpengli, jmattson,
joro, kvm, linux-kernel, linux-hyperv
Cc: mikelley
In-Reply-To: <20200420173838.24672-1-mikelley@microsoft.com>
Add definitions for GetVpRegister and SetVpRegister hypercalls, which
are implemented for both x86 and ARM64.
Signed-off-by: Michael Kelley <mikelley@microsoft.com>
---
include/asm-generic/hyperv-tlfs.h | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/include/asm-generic/hyperv-tlfs.h b/include/asm-generic/hyperv-tlfs.h
index 1f92ef92eb56..29b60f5b6323 100644
--- a/include/asm-generic/hyperv-tlfs.h
+++ b/include/asm-generic/hyperv-tlfs.h
@@ -141,6 +141,8 @@ struct ms_hyperv_tsc_page {
#define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE_EX 0x0013
#define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST_EX 0x0014
#define HVCALL_SEND_IPI_EX 0x0015
+#define HVCALL_GET_VP_REGISTERS 0x0050
+#define HVCALL_SET_VP_REGISTERS 0x0051
#define HVCALL_POST_MESSAGE 0x005c
#define HVCALL_SIGNAL_EVENT 0x005d
#define HVCALL_RETARGET_INTERRUPT 0x007e
@@ -439,4 +441,30 @@ struct hv_retarget_device_interrupt {
struct hv_device_interrupt_target int_target;
} __packed __aligned(8);
+
+/* HvGetVPRegister hypercall */
+struct hv_get_vp_register_input {
+ u64 partitionid;
+ u32 vpindex;
+ u8 inputvtl;
+ u8 padding[3];
+ u32 name0;
+ u32 name1;
+} __packed;
+
+struct hv_get_vp_register_output {
+ union {
+ struct {
+ u32 a;
+ u32 b;
+ u32 c;
+ u32 d;
+ } as32 __packed;
+ struct {
+ u64 low;
+ u64 high;
+ } as64 __packed;
+ };
+};
+
#endif
--
2.18.2
^ permalink raw reply related
* [PATCH v2] x86/hyperv: Suspend/resume the VP assist page for hibernation
From: Dexuan Cui @ 2020-04-21 2:46 UTC (permalink / raw)
To: bp, haiyangz, hpa, kys, linux-hyperv, linux-kernel, mingo,
sthemmin, tglx, x86, mikelley, vkuznets, wei.liu
Cc: Dexuan Cui, stable
Unlike the other CPUs, CPU0 is never offlined during hibernation, so in the
resume path, the "new" kernel's VP assist page is not suspended (i.e. not
disabled), and later when we jump to the "old" kernel, the page is not
properly re-enabled for CPU0 with the allocated page from the old kernel.
So far, the VP assist page is used by hv_apic_eoi_write(), and is also
used in the case of nested virtualization (running KVM atop Hyper-V).
For hv_apic_eoi_write(), when the page is not properly re-enabled,
hvp->apic_assist is always 0, so the HV_X64_MSR_EOI MSR is always written.
This is not ideal with respect to performance, but Hyper-V can still
correctly handle this according to the Hyper-V spec; nevertheless, Linux
still must update the Hyper-V hypervisor with the correct VP assist page
to prevent Hyper-V from writing to the stale page, which causes guest
memory corruption and consequently may have caused the hangs and triple
faults seen during non-boot CPUs resume.
Fix the issue by calling hv_cpu_die()/hv_cpu_init() in the syscore ops.
Without the fix, hibernation can fail at a rate of 1/300 ~ 1/500.
With the fix, hibernation can pass a long-haul test of 2000 runs.
In the case of nested virtualization, disabling/reenabling the assist
page upon hibernation may be unsafe if there are active L2 guests.
It looks KVM should be enhanced to abort the hibernation request if
there is any active L2 guest.
Fixes: 05bd330a7fd8 ("x86/hyperv: Suspend/resume the hypercall page for hibernation")
Cc: stable@vger.kernel.org
Signed-off-by: Dexuan Cui <decui@microsoft.com>
---
Changes in v2:
Used alloc_page(irqs_disabled() ? GFP_ATOMIC : GFP_KERNEL) [Wei Liu]
Improved the changelog per comments from Wei Liu and Vitaly Kuznetsov.
arch/x86/hyperv/hv_init.c | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
index b0da5320bcff..a151ec7feb4b 100644
--- a/arch/x86/hyperv/hv_init.c
+++ b/arch/x86/hyperv/hv_init.c
@@ -72,7 +72,8 @@ static int hv_cpu_init(unsigned int cpu)
struct page *pg;
input_arg = (void **)this_cpu_ptr(hyperv_pcpu_input_arg);
- pg = alloc_page(GFP_KERNEL);
+ /* hv_cpu_init() can be called with IRQs disabled from hv_resume() */
+ pg = alloc_page(irqs_disabled() ? GFP_ATOMIC : GFP_KERNEL);
if (unlikely(!pg))
return -ENOMEM;
*input_arg = page_address(pg);
@@ -253,6 +254,7 @@ static int __init hv_pci_init(void)
static int hv_suspend(void)
{
union hv_x64_msr_hypercall_contents hypercall_msr;
+ int ret;
/*
* Reset the hypercall page as it is going to be invalidated
@@ -269,12 +271,17 @@ static int hv_suspend(void)
hypercall_msr.enable = 0;
wrmsrl(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64);
- return 0;
+ ret = hv_cpu_die(0);
+ return ret;
}
static void hv_resume(void)
{
union hv_x64_msr_hypercall_contents hypercall_msr;
+ int ret;
+
+ ret = hv_cpu_init(0);
+ WARN_ON(ret);
/* Re-enable the hypercall page */
rdmsrl(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64);
@@ -287,6 +294,7 @@ static void hv_resume(void)
hv_hypercall_pg_saved = NULL;
}
+/* Note: when the ops are called, only CPU0 is online and IRQs are disabled. */
static struct syscore_ops hv_syscore_ops = {
.suspend = hv_suspend,
.resume = hv_resume,
--
2.19.1
^ permalink raw reply related
* Re: [PATCH 1/1] Drivers: hv: Move AEOI determination to architecture dependent code
From: Wei Liu @ 2020-04-21 9:07 UTC (permalink / raw)
To: Michael Kelley
Cc: kys, haiyangz, sthemmin, wei.liu, tglx, mingo, bp, x86, hpa,
linux-kernel, linux-hyperv
In-Reply-To: <20200420164926.24471-1-mikelley@microsoft.com>
On Mon, Apr 20, 2020 at 09:49:26AM -0700, Michael Kelley wrote:
> Hyper-V on ARM64 doesn't provide a flag for the AEOI recommendation
> in ms_hyperv.hints, so having the test in architecture independent
> code doesn't work. Resolve this by moving the check of the flag
> to an architecture dependent helper function. No functionality is
> changed.
>
> Signed-off-by: Michael Kelley <mikelley@microsoft.com>
Applied to hyperv-fixes. Thanks.
> ---
> arch/x86/include/asm/mshyperv.h | 2 ++
> drivers/hv/hv.c | 6 +-----
> 2 files changed, 3 insertions(+), 5 deletions(-)
>
> diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
> index 1c42ecbe75cb..d30805ed323e 100644
> --- a/arch/x86/include/asm/mshyperv.h
> +++ b/arch/x86/include/asm/mshyperv.h
> @@ -35,6 +35,8 @@ typedef int (*hyperv_fill_flush_list_func)(
> rdmsrl(HV_X64_MSR_SINT0 + int_num, val)
> #define hv_set_synint_state(int_num, val) \
> wrmsrl(HV_X64_MSR_SINT0 + int_num, val)
> +#define hv_recommend_using_aeoi() \
> + (!(ms_hyperv.hints & HV_DEPRECATING_AEOI_RECOMMENDED))
>
> #define hv_get_crash_ctl(val) \
> rdmsrl(HV_X64_MSR_CRASH_CTL, val)
> diff --git a/drivers/hv/hv.c b/drivers/hv/hv.c
> index 6098e0cbdb4b..533c8b82b344 100644
> --- a/drivers/hv/hv.c
> +++ b/drivers/hv/hv.c
> @@ -184,11 +184,7 @@ void hv_synic_enable_regs(unsigned int cpu)
>
> shared_sint.vector = HYPERVISOR_CALLBACK_VECTOR;
> shared_sint.masked = false;
> - if (ms_hyperv.hints & HV_DEPRECATING_AEOI_RECOMMENDED)
> - shared_sint.auto_eoi = false;
> - else
> - shared_sint.auto_eoi = true;
> -
> + shared_sint.auto_eoi = hv_recommend_using_aeoi();
> hv_set_synint_state(VMBUS_MESSAGE_SINT, shared_sint.as_uint64);
>
> /* Enable the global synic bit */
> --
> 2.18.2
>
^ permalink raw reply
* Re: [PATCH v2] x86/hyperv: Suspend/resume the VP assist page for hibernation
From: Wei Liu @ 2020-04-21 9:07 UTC (permalink / raw)
To: Dexuan Cui
Cc: bp, haiyangz, hpa, kys, linux-hyperv, linux-kernel, mingo,
sthemmin, tglx, x86, mikelley, vkuznets, wei.liu, stable
In-Reply-To: <1587437171-2472-1-git-send-email-decui@microsoft.com>
On Mon, Apr 20, 2020 at 07:46:11PM -0700, Dexuan Cui wrote:
> Unlike the other CPUs, CPU0 is never offlined during hibernation, so in the
> resume path, the "new" kernel's VP assist page is not suspended (i.e. not
> disabled), and later when we jump to the "old" kernel, the page is not
> properly re-enabled for CPU0 with the allocated page from the old kernel.
>
> So far, the VP assist page is used by hv_apic_eoi_write(), and is also
> used in the case of nested virtualization (running KVM atop Hyper-V).
>
> For hv_apic_eoi_write(), when the page is not properly re-enabled,
> hvp->apic_assist is always 0, so the HV_X64_MSR_EOI MSR is always written.
> This is not ideal with respect to performance, but Hyper-V can still
> correctly handle this according to the Hyper-V spec; nevertheless, Linux
> still must update the Hyper-V hypervisor with the correct VP assist page
> to prevent Hyper-V from writing to the stale page, which causes guest
> memory corruption and consequently may have caused the hangs and triple
> faults seen during non-boot CPUs resume.
>
> Fix the issue by calling hv_cpu_die()/hv_cpu_init() in the syscore ops.
> Without the fix, hibernation can fail at a rate of 1/300 ~ 1/500.
> With the fix, hibernation can pass a long-haul test of 2000 runs.
>
> In the case of nested virtualization, disabling/reenabling the assist
> page upon hibernation may be unsafe if there are active L2 guests.
> It looks KVM should be enhanced to abort the hibernation request if
> there is any active L2 guest.
>
> Fixes: 05bd330a7fd8 ("x86/hyperv: Suspend/resume the hypercall page for hibernation")
> Cc: stable@vger.kernel.org
> Signed-off-by: Dexuan Cui <decui@microsoft.com>
Applied to hyperv-fixes. Thanks.
^ permalink raw reply
* Re: [PATCH 1/4] KVM: x86: hyperv: Remove duplicate definitions of Reference TSC Page
From: Wei Liu @ 2020-04-21 9:29 UTC (permalink / raw)
To: Michael Kelley
Cc: kys, haiyangz, sthemmin, wei.liu, tglx, mingo, bp, x86, hpa,
pbonzini, sean.j.christopherson, vkuznets, wanpengli, jmattson,
joro, kvm, linux-kernel, linux-hyperv
In-Reply-To: <20200420173838.24672-2-mikelley@microsoft.com>
On Mon, Apr 20, 2020 at 10:38:35AM -0700, Michael Kelley wrote:
> The Hyper-V Reference TSC Page structure is defined twice. struct
> ms_hyperv_tsc_page has padding out to a full 4 Kbyte page size. But
> the padding is not needed because the declaration includes a union
> with HV_HYP_PAGE_SIZE. KVM uses the second definition, which is
> struct _HV_REFERENCE_TSC_PAGE, because it does not have the padding.
>
> Fix the duplication by removing the padding from ms_hyperv_tsc_page.
> Fix up the KVM code to use it. Remove the no longer used struct
> _HV_REFERENCE_TSC_PAGE.
>
> There is no functional change.
>
> Signed-off-by: Michael Kelley <mikelley@microsoft.com>
> ---
> arch/x86/include/asm/hyperv-tlfs.h | 8 --------
> arch/x86/include/asm/kvm_host.h | 2 +-
> arch/x86/kvm/hyperv.c | 4 ++--
Paolo, this patch touches KVM code. Let me know how you would like to
handle this.
Wei.
^ permalink raw reply
* Re: [PATCH 1/4] KVM: x86: hyperv: Remove duplicate definitions of Reference TSC Page
From: Paolo Bonzini @ 2020-04-21 11:16 UTC (permalink / raw)
To: Wei Liu, Michael Kelley
Cc: kys, haiyangz, sthemmin, tglx, mingo, bp, x86, hpa,
sean.j.christopherson, vkuznets, wanpengli, jmattson, joro, kvm,
linux-kernel, linux-hyperv
In-Reply-To: <20200421092925.rxb72yep4paruvi6@debian>
On 21/04/20 11:29, Wei Liu wrote:
> On Mon, Apr 20, 2020 at 10:38:35AM -0700, Michael Kelley wrote:
>> The Hyper-V Reference TSC Page structure is defined twice. struct
>> ms_hyperv_tsc_page has padding out to a full 4 Kbyte page size. But
>> the padding is not needed because the declaration includes a union
>> with HV_HYP_PAGE_SIZE. KVM uses the second definition, which is
>> struct _HV_REFERENCE_TSC_PAGE, because it does not have the padding.
>>
>> Fix the duplication by removing the padding from ms_hyperv_tsc_page.
>> Fix up the KVM code to use it. Remove the no longer used struct
>> _HV_REFERENCE_TSC_PAGE.
>>
>> There is no functional change.
>>
>> Signed-off-by: Michael Kelley <mikelley@microsoft.com>
>> ---
>> arch/x86/include/asm/hyperv-tlfs.h | 8 --------
>> arch/x86/include/asm/kvm_host.h | 2 +-
>> arch/x86/kvm/hyperv.c | 4 ++--
>
> Paolo, this patch touches KVM code. Let me know how you would like to
> handle this.
Just include it, I don't expect conflicts.
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Paolo
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox