* RE: [PATCH v1 5/6] x86/hyperv: Implement hypervisor ram collection into vmcore
From: Michael Kelley @ 2025-09-23 1:35 UTC (permalink / raw)
To: Mukesh R, linux-hyperv@vger.kernel.org,
linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org
Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org,
decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com,
bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org,
hpa@zytor.com, arnd@arndb.de
In-Reply-To: <eb755e6d-3adf-ee3b-2942-666bc1bedef5@linux.microsoft.com>
From: Mukesh R <mrathor@linux.microsoft.com> Sent: Friday, September 19, 2025 6:43 PM
>
> On 9/18/25 19:32, Mukesh R wrote:
> > On 9/18/25 16:53, Michael Kelley wrote:
> >> From: Mukesh R <mrathor@linux.microsoft.com> Sent: Tuesday, September 16, 2025 6:13 PM
> >>>
> >>> On 9/15/25 10:55, Michael Kelley wrote:
[snip]
> >>>>> +/*
> >>>>> + * Common function for all cpus before devirtualization.
> >>>>> + *
> >>>>> + * Hypervisor crash: all cpus get here in nmi context.
> >>>>> + * Linux crash: the panicing cpu gets here at base level, all others in nmi
> >>>>> + * context. Note, panicing cpu may not be the bsp.
> >>>>> + *
> >>>>> + * The function is not inlined so it will show on the stack. It is named so
> >>>>> + * because the crash cmd looks for certain well known function names on the
> >>>>> + * stack before looking into the cpu saved note in the elf section, and
> >>>>> + * that work is currently incomplete.
> >>>>> + *
> >>>>> + * Notes:
> >>>>> + * Hypervisor crash:
> >>>>> + * - the hypervisor is in a very restrictive mode at this point and any
> >>>>> + * vmexit it cannot handle would result in reboot. For example, console
> >>>>> + * output from here would result in synic ipi hcall, which would result
> >>>>> + * in reboot. So, no mumbo jumbo, just get to kexec as quickly as possible.
> >>>>> + *
> >>>>> + * Devirtualization is supported from the bsp only.
> >>>>> + */
> >>>>> +static noinline __noclone void crash_nmi_callback(struct pt_regs *regs)
> >>>>> +{
> >>>>> + struct hv_input_disable_hyp_ex *input;
> >>>>> + u64 status;
> >>>>> + int msecs = 1000, ccpu = smp_processor_id();
> >>>>> +
> >>>>> + if (ccpu == 0) {
> >>>>> + /* crash_save_cpu() will be done in the kexec path */
> >>>>> + cpu_emergency_stop_pt(); /* disable performance trace */
> >>>>> + atomic_inc(&crash_cpus_wait);
> >>>>> + } else {
> >>>>> + crash_save_cpu(regs, ccpu);
> >>>>> + cpu_emergency_stop_pt(); /* disable performance trace */
> >>>>> + atomic_inc(&crash_cpus_wait);
> >>>>> + for (;;); /* cause no vmexits */
> >>>>> + }
> >>>>> +
> >>>>> + while (atomic_read(&crash_cpus_wait) < num_online_cpus() && msecs--)
> >>>>> + mdelay(1);
> >>>>> +
> >>>>> + stop_nmi();
> >>>>> + if (!hv_has_crashed)
> >>>>> + hv_notify_prepare_hyp();
> >>>>> +
> >>>>> + if (crashing_cpu == -1)
> >>>>> + crashing_cpu = ccpu; /* crash cmd uses this */
> >>>>
> >>>> Could just be "crashing_cpu = 0" since only the BSP gets here.
> >>>
> >>> a code change request has been open for while to remove the requirement
> >>> of bsp..
> >>>
> >>>>> +
> >>>>> + hv_hvcrash_ctxt_save();
> >>>>> + hv_mark_tss_not_busy();
> >>>>> + hv_crash_fixup_kernpt();
> >>>>> +
> >>>>> + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
> >>>>> + memset(input, 0, sizeof(*input));
> >>>>> + input->rip = trampoline_pa; /* PA of hv_crash_asm32 */
> >>>>> + input->arg = devirt_cr3arg; /* PA of trampoline page table L4 */
> >>>>
> >>>> Is this comment correct? Isn't it the PA of struct hv_crash_tramp_data?
> >>>> And just for clarification, Hyper-V treats this "arg" value as opaque and does
> >>>> not access it. It only provides it in EDI when it invokes the trampoline
> >>>> function, right?
> >>>
> >>> comment is correct. cr3 always points to l4 (or l5 if 5 level page tables).
> >>
> >> Yes, the comment matches the name of the "devirt_cr3arg" variable.
> >> Unfortunately my previous comment was incomplete because the value
> >> stored in the static variable "devirt_cr3arg" isn?t the address of an L4 page
> >> table. It's not a CR3 value. The value stored in devirt_cr3arg is actually the
> >> PA of struct hv_crash_tramp_data. The CR3 value is stored in the
> >> tramp32_cr3 field (at offset 0) of that structure, so there's an additional level
> >> of indirection. The (corrected) comment in the header to hv_crash_asm32()
> >> describes EDI as containing "PA of struct hv_crash_tramp_data", which
> >> ought to match what is described here. I'd say that "devirt_cr3arg" ought
> >> to be renamed to "tramp_data_pa" or something else parallel to
> >> "trampoline_pa".
> >
> > hyp needs trampoline cr3 for transition, we pass it as an arg. we piggy
> > back extra information for ourselves needed in trampoline.S. so it's
> > all good.
>
> actually, what i said earlier was true, not above. that the arg is
> opaque and hyp does not use it (we are transitioning paging off after
> all!). i did this all almost two years ago, so had vague recollections
> but finally had time today to go back to square one and old notes,
> and remember things now. so final answer:
>
> the hypercall calls it TrampolineCr3, i guess this is how windows uses it
> (they have customized kernel code for core collection). doing that was
> becoming too intrusive on linux, so i decided to use the arg to pass the
> info i needed in the trampoline code. Since the hypercall calls the arg
> TrampolineCr3, i must have just used that name for the arg to match it,
> probably falsely assuming hypervisor somehow looked at it. (actually,
> the windows hypercall wrapper does look at it to make sure it is a
> ram address).
>
> since the hypercall doesn't use the arg, it could just call it
> devirtArg, but maybe in the past they used it somehow. in my latest
> version, i just call it devirt_arg.
OK. Good to get this all straightened out. Please leave a code
comment to the effect that the hypercall doesn't use the arg, and
that the value is provided solely to be passed to hv_crash_asm32()
for it to use. That means that struct hv_crash_tramp_data is owned
by Linux and can be changed/updated as needed.
The assignment statement to the hypercall input could look like:
input->arg = devirt_arg; /* PA of struct hv_crash_tramp_data */
which would align with the comment in the header of hv_crash_asm32().
Michael
^ permalink raw reply
* Re: [PATCH RFC] PCI: Convert devm_pci_alloc_host_bridge() users to error-pointer returns
From: AngeloGioacchino Del Regno @ 2025-09-22 10:55 UTC (permalink / raw)
To: Alok Tiwari, thomas.petazzoni, pali, lpieralisi, kwilczynski,
mani, robh, bhelgaas, joyce.ooi, alyssa, maz, jim2101024,
florian.fainelli, bcm-kernel-feedback-list, rjui, sbranden,
ryder.lee, jianjun.wang, sergio.paracuellos, matthias.bgg,
marek.vasut+renesas, yoshihiro.shimoda.uh, geert+renesas,
magnus.damm, shawn.lin, heiko, michal.simek, bharat.kumar.gogada,
will, kys, haiyangz, wei.liu, decui, linus.walleij,
thierry.reding, jonathanh, rric, nirmal.patel, toan,
jonathan.derrick, linux-pci
Cc: linux-arm-kernel, linux-kernel, linux-rpi-kernel, linux-mediatek,
linux-renesas-soc, linux-rockchip, linux-hyperv, linux-tegra
In-Reply-To: <20250921161434.1561770-1-alok.a.tiwari@oracle.com>
Il 21/09/25 18:14, Alok Tiwari ha scritto:
> devm_pci_alloc_host_bridge() and pci_alloc_host_bridge() previously
> returned NULL on failure, forcing callers to special-case NULL handling
> and often hardcode -ENOMEM as the error.
>
> This series updates devm_pci_alloc_host_bridge() to consistently return
> error pointers (ERR_PTR) with the actual error code, instead of NULL.
> All callers across PCI host controller drivers are updated to use
> IS_ERR_OR_NULL()/PTR_ERR() instead of NULL checks and hardcoded -ENOMEM.
>
> Benefits:
> - Standardizes error handling with Linux kernel ERR_PTR()/PTR_ERR()
> conventions.
> - Ensures that the actual error code from lower-level helpers is
> propagated back to the caller.
> - Removes ambiguity between NULL and error pointer returns.
>
> Touched drivers include:
> cadence (J721E, cadence-plat)
> dwc (designware, qcom)
> mobiveil (layerscape-gen4, mobiveil-plat)
> aardvark, ftpci100, ixp4xx, loongson, mvebu, rcar, tegra, v3-semi,
> versatile, xgene, altera, brcmstb, iproc, mediatek, mt7621, xilinx,
> plda, and others
>
> This patch updates error handling across these host controller drivers
> so that callers consistently receive ERR_PTR() instead of NULL.
>
I think that's a nice improvement - propagating the right error code looks good.
The only thing is - you have to make sure that it never returns NULL, so that
in the controller drivers you always check for `if (IS_ERR(x))` - otherwise with
the current IS_ERR_OR_NULL(x) most of the error paths are wrong.
Cheers,
Angelo
> Signed-off-by: Alok Tiwari <alok.a.tiwari@oracle.com>
> ---
> arch/mips/pci/pci-xtalk-bridge.c | 4 ++--
> drivers/pci/controller/cadence/pci-j721e.c | 4 ++--
> drivers/pci/controller/cadence/pcie-cadence-plat.c | 4 ++--
> drivers/pci/controller/dwc/pcie-designware-host.c | 4 ++--
> drivers/pci/controller/dwc/pcie-qcom.c | 4 ++--
> drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c | 4 ++--
> drivers/pci/controller/mobiveil/pcie-mobiveil-plat.c | 4 ++--
> drivers/pci/controller/pci-aardvark.c | 4 ++--
> drivers/pci/controller/pci-ftpci100.c | 4 ++--
> drivers/pci/controller/pci-host-common.c | 4 ++--
> drivers/pci/controller/pci-hyperv.c | 4 ++--
> drivers/pci/controller/pci-ixp4xx.c | 4 ++--
> drivers/pci/controller/pci-loongson.c | 4 ++--
> drivers/pci/controller/pci-mvebu.c | 4 ++--
> drivers/pci/controller/pci-rcar-gen2.c | 4 ++--
> drivers/pci/controller/pci-tegra.c | 4 ++--
> drivers/pci/controller/pci-v3-semi.c | 4 ++--
> drivers/pci/controller/pci-versatile.c | 4 ++--
> drivers/pci/controller/pci-xgene.c | 4 ++--
> drivers/pci/controller/pcie-altera.c | 4 ++--
> drivers/pci/controller/pcie-brcmstb.c | 4 ++--
> drivers/pci/controller/pcie-iproc-bcma.c | 4 ++--
> drivers/pci/controller/pcie-iproc-platform.c | 4 ++--
> drivers/pci/controller/pcie-mediatek-gen3.c | 4 ++--
> drivers/pci/controller/pcie-mediatek.c | 4 ++--
> drivers/pci/controller/pcie-mt7621.c | 4 ++--
> drivers/pci/controller/pcie-rcar-host.c | 4 ++--
> drivers/pci/controller/pcie-rockchip-host.c | 4 ++--
> drivers/pci/controller/pcie-xilinx-cpm.c | 4 ++--
> drivers/pci/controller/pcie-xilinx-dma-pl.c | 4 ++--
> drivers/pci/controller/pcie-xilinx-nwl.c | 4 ++--
> drivers/pci/controller/pcie-xilinx.c | 4 ++--
> drivers/pci/controller/plda/pcie-plda-host.c | 4 ++--
> drivers/pci/probe.c | 8 ++++----
> 34 files changed, 70 insertions(+), 70 deletions(-)
>
^ permalink raw reply
* Re: [PATCH RFC] PCI: Convert devm_pci_alloc_host_bridge() users to error-pointer returns
From: Marc Zyngier @ 2025-09-21 17:19 UTC (permalink / raw)
To: Alok Tiwari
Cc: thomas.petazzoni, pali, lpieralisi, kwilczynski, mani, robh,
bhelgaas, joyce.ooi, alyssa, jim2101024, florian.fainelli,
bcm-kernel-feedback-list, rjui, sbranden, ryder.lee, jianjun.wang,
sergio.paracuellos, matthias.bgg, angelogioacchino.delregno,
marek.vasut+renesas, yoshihiro.shimoda.uh, geert+renesas,
magnus.damm, shawn.lin, heiko, michal.simek, bharat.kumar.gogada,
will, kys, haiyangz, wei.liu, decui, linus.walleij,
thierry.reding, jonathanh, rric, nirmal.patel, toan,
jonathan.derrick, linux-pci, linux-arm-kernel, linux-kernel,
linux-rpi-kernel, linux-mediatek, linux-renesas-soc,
linux-rockchip, linux-hyperv, linux-tegra
In-Reply-To: <20250921161434.1561770-1-alok.a.tiwari@oracle.com>
On Sun, 21 Sep 2025 17:14:07 +0100,
Alok Tiwari <alok.a.tiwari@oracle.com> wrote:
>
> devm_pci_alloc_host_bridge() and pci_alloc_host_bridge() previously
> returned NULL on failure, forcing callers to special-case NULL handling
> and often hardcode -ENOMEM as the error.
>
> This series updates devm_pci_alloc_host_bridge() to consistently return
> error pointers (ERR_PTR) with the actual error code, instead of NULL.
> All callers across PCI host controller drivers are updated to use
> IS_ERR_OR_NULL()/PTR_ERR() instead of NULL checks and hardcoded -ENOMEM.
>
> Benefits:
> - Standardizes error handling with Linux kernel ERR_PTR()/PTR_ERR()
> conventions.
> - Ensures that the actual error code from lower-level helpers is
> propagated back to the caller.
> - Removes ambiguity between NULL and error pointer returns.
>
> Touched drivers include:
> cadence (J721E, cadence-plat)
> dwc (designware, qcom)
> mobiveil (layerscape-gen4, mobiveil-plat)
> aardvark, ftpci100, ixp4xx, loongson, mvebu, rcar, tegra, v3-semi,
> versatile, xgene, altera, brcmstb, iproc, mediatek, mt7621, xilinx,
> plda, and others
>
> This patch updates error handling across these host controller drivers
> so that callers consistently receive ERR_PTR() instead of NULL.
Not quite.
> diff --git a/arch/mips/pci/pci-xtalk-bridge.c b/arch/mips/pci/pci-xtalk-bridge.c
> index e00c38620d14..c2c8ed8ecac1 100644
> --- a/arch/mips/pci/pci-xtalk-bridge.c
> +++ b/arch/mips/pci/pci-xtalk-bridge.c
> @@ -636,8 +636,8 @@ static int bridge_probe(struct platform_device *pdev)
> pci_set_flags(PCI_PROBE_ONLY);
>
> host = devm_pci_alloc_host_bridge(dev, sizeof(*bc));
> - if (!host) {
> - err = -ENOMEM;
> + if (IS_ERR_OR_NULL(host)) {
> + err = PTR_ERR(host);
Under which circumstances can NULL still be returned? Because applying
PTR_ERR() to a NULL pointer looks like a pretty bad idea.
> goto err_remove_domain;
> }
>
[...]
> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> index f41128f91ca7..e627f36b7683 100644
> --- a/drivers/pci/probe.c
> +++ b/drivers/pci/probe.c
> @@ -686,18 +686,18 @@ struct pci_host_bridge *devm_pci_alloc_host_bridge(struct device *dev,
>
> bridge = pci_alloc_host_bridge(priv);
> if (!bridge)
> - return NULL;
> + return ERR_PTR(-ENOMEM);
>
> bridge->dev.parent = dev;
>
> ret = devm_add_action_or_reset(dev, devm_pci_alloc_host_bridge_release,
> bridge);
> if (ret)
> - return NULL;
> + return ERR_PTR(ret);
>
> ret = devm_of_pci_bridge_init(dev, bridge);
> if (ret)
> - return NULL;
> + return ERR_PTR(ret);
>
> return bridge;
> }
> @@ -3198,7 +3198,7 @@ struct pci_bus *pci_create_root_bus(struct device *parent, int bus,
>
> bridge = pci_alloc_host_bridge(0);
> if (!bridge)
> - return NULL;
> + return ERR_PTR(-ENOMEM);
>
> bridge->dev.parent = parent;
>
And what about the code that comes after that if we fail to register
the bus? The remaining "return NULL", which will then be interpreted
as 0 in any user of this function, leading to a worse situation than
what we have now.
Also, things like pci_scan_root_bus() have the following pattern:
b = pci_create_root_bus(parent, bus, ops, sysdata, resources);
if (!b)
return NULL;
which will end with prejudice given what you have introduced.
If you are going to touch this sort of things, at least make it
consistent, analyse *all* code paths, and provide documentation.
M.
--
Jazz isn't dead. It just smells funny.
^ permalink raw reply
* [PATCH RFC] PCI: Convert devm_pci_alloc_host_bridge() users to error-pointer returns
From: Alok Tiwari @ 2025-09-21 16:14 UTC (permalink / raw)
To: thomas.petazzoni, pali, lpieralisi, kwilczynski, mani, robh,
bhelgaas, joyce.ooi, alyssa, maz, jim2101024, florian.fainelli,
bcm-kernel-feedback-list, rjui, sbranden, ryder.lee, jianjun.wang,
sergio.paracuellos, matthias.bgg, angelogioacchino.delregno,
marek.vasut+renesas, yoshihiro.shimoda.uh, geert+renesas,
magnus.damm, shawn.lin, heiko, michal.simek, bharat.kumar.gogada,
will, kys, haiyangz, wei.liu, decui, linus.walleij,
thierry.reding, jonathanh, rric, nirmal.patel, toan,
jonathan.derrick, linux-pci
Cc: alok.a.tiwari, linux-arm-kernel, linux-kernel, linux-rpi-kernel,
linux-mediatek, linux-renesas-soc, linux-rockchip, linux-hyperv,
linux-tegra
devm_pci_alloc_host_bridge() and pci_alloc_host_bridge() previously
returned NULL on failure, forcing callers to special-case NULL handling
and often hardcode -ENOMEM as the error.
This series updates devm_pci_alloc_host_bridge() to consistently return
error pointers (ERR_PTR) with the actual error code, instead of NULL.
All callers across PCI host controller drivers are updated to use
IS_ERR_OR_NULL()/PTR_ERR() instead of NULL checks and hardcoded -ENOMEM.
Benefits:
- Standardizes error handling with Linux kernel ERR_PTR()/PTR_ERR()
conventions.
- Ensures that the actual error code from lower-level helpers is
propagated back to the caller.
- Removes ambiguity between NULL and error pointer returns.
Touched drivers include:
cadence (J721E, cadence-plat)
dwc (designware, qcom)
mobiveil (layerscape-gen4, mobiveil-plat)
aardvark, ftpci100, ixp4xx, loongson, mvebu, rcar, tegra, v3-semi,
versatile, xgene, altera, brcmstb, iproc, mediatek, mt7621, xilinx,
plda, and others
This patch updates error handling across these host controller drivers
so that callers consistently receive ERR_PTR() instead of NULL.
Signed-off-by: Alok Tiwari <alok.a.tiwari@oracle.com>
---
arch/mips/pci/pci-xtalk-bridge.c | 4 ++--
drivers/pci/controller/cadence/pci-j721e.c | 4 ++--
drivers/pci/controller/cadence/pcie-cadence-plat.c | 4 ++--
drivers/pci/controller/dwc/pcie-designware-host.c | 4 ++--
drivers/pci/controller/dwc/pcie-qcom.c | 4 ++--
drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c | 4 ++--
drivers/pci/controller/mobiveil/pcie-mobiveil-plat.c | 4 ++--
drivers/pci/controller/pci-aardvark.c | 4 ++--
drivers/pci/controller/pci-ftpci100.c | 4 ++--
drivers/pci/controller/pci-host-common.c | 4 ++--
drivers/pci/controller/pci-hyperv.c | 4 ++--
drivers/pci/controller/pci-ixp4xx.c | 4 ++--
drivers/pci/controller/pci-loongson.c | 4 ++--
drivers/pci/controller/pci-mvebu.c | 4 ++--
drivers/pci/controller/pci-rcar-gen2.c | 4 ++--
drivers/pci/controller/pci-tegra.c | 4 ++--
drivers/pci/controller/pci-v3-semi.c | 4 ++--
drivers/pci/controller/pci-versatile.c | 4 ++--
drivers/pci/controller/pci-xgene.c | 4 ++--
drivers/pci/controller/pcie-altera.c | 4 ++--
drivers/pci/controller/pcie-brcmstb.c | 4 ++--
drivers/pci/controller/pcie-iproc-bcma.c | 4 ++--
drivers/pci/controller/pcie-iproc-platform.c | 4 ++--
drivers/pci/controller/pcie-mediatek-gen3.c | 4 ++--
drivers/pci/controller/pcie-mediatek.c | 4 ++--
drivers/pci/controller/pcie-mt7621.c | 4 ++--
drivers/pci/controller/pcie-rcar-host.c | 4 ++--
drivers/pci/controller/pcie-rockchip-host.c | 4 ++--
drivers/pci/controller/pcie-xilinx-cpm.c | 4 ++--
drivers/pci/controller/pcie-xilinx-dma-pl.c | 4 ++--
drivers/pci/controller/pcie-xilinx-nwl.c | 4 ++--
drivers/pci/controller/pcie-xilinx.c | 4 ++--
drivers/pci/controller/plda/pcie-plda-host.c | 4 ++--
drivers/pci/probe.c | 8 ++++----
34 files changed, 70 insertions(+), 70 deletions(-)
diff --git a/arch/mips/pci/pci-xtalk-bridge.c b/arch/mips/pci/pci-xtalk-bridge.c
index e00c38620d14..c2c8ed8ecac1 100644
--- a/arch/mips/pci/pci-xtalk-bridge.c
+++ b/arch/mips/pci/pci-xtalk-bridge.c
@@ -636,8 +636,8 @@ static int bridge_probe(struct platform_device *pdev)
pci_set_flags(PCI_PROBE_ONLY);
host = devm_pci_alloc_host_bridge(dev, sizeof(*bc));
- if (!host) {
- err = -ENOMEM;
+ if (IS_ERR_OR_NULL(host)) {
+ err = PTR_ERR(host);
goto err_remove_domain;
}
diff --git a/drivers/pci/controller/cadence/pci-j721e.c b/drivers/pci/controller/cadence/pci-j721e.c
index 6c93f39d0288..3b8afaef21a6 100644
--- a/drivers/pci/controller/cadence/pci-j721e.c
+++ b/drivers/pci/controller/cadence/pci-j721e.c
@@ -475,8 +475,8 @@ static int j721e_pcie_probe(struct platform_device *pdev)
return -ENODEV;
bridge = devm_pci_alloc_host_bridge(dev, sizeof(*rc));
- if (!bridge)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
if (!data->byte_access_allowed)
bridge->ops = &cdns_ti_pcie_host_ops;
diff --git a/drivers/pci/controller/cadence/pcie-cadence-plat.c b/drivers/pci/controller/cadence/pcie-cadence-plat.c
index 0456845dabb9..7570cb5998f6 100644
--- a/drivers/pci/controller/cadence/pcie-cadence-plat.c
+++ b/drivers/pci/controller/cadence/pcie-cadence-plat.c
@@ -66,8 +66,8 @@ static int cdns_plat_pcie_probe(struct platform_device *pdev)
return -ENODEV;
bridge = devm_pci_alloc_host_bridge(dev, sizeof(*rc));
- if (!bridge)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
rc = pci_host_bridge_priv(bridge);
rc->pcie.dev = dev;
diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
index 952f8594b501..b2b99f275c19 100644
--- a/drivers/pci/controller/dwc/pcie-designware-host.c
+++ b/drivers/pci/controller/dwc/pcie-designware-host.c
@@ -467,8 +467,8 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)
raw_spin_lock_init(&pp->lock);
bridge = devm_pci_alloc_host_bridge(dev, 0);
- if (!bridge)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
pp->bridge = bridge;
diff --git a/drivers/pci/controller/dwc/pcie-qcom.c b/drivers/pci/controller/dwc/pcie-qcom.c
index 294babe1816e..34d35c925c62 100644
--- a/drivers/pci/controller/dwc/pcie-qcom.c
+++ b/drivers/pci/controller/dwc/pcie-qcom.c
@@ -1809,8 +1809,8 @@ static int qcom_pcie_probe(struct platform_device *pdev)
struct pci_config_window *cfg;
bridge = devm_pci_alloc_host_bridge(dev, 0);
- if (!bridge) {
- ret = -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge)) {
+ ret = PTR_ERR(bridge);
goto err_pm_runtime_put;
}
diff --git a/drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c b/drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c
index 4919b27eaf44..f9ebefc71be3 100644
--- a/drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c
+++ b/drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c
@@ -207,8 +207,8 @@ static int __init ls_g4_pcie_probe(struct platform_device *pdev)
}
bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
- if (!bridge)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
pcie = pci_host_bridge_priv(bridge);
mv_pci = &pcie->pci;
diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil-plat.c b/drivers/pci/controller/mobiveil/pcie-mobiveil-plat.c
index c5bb87ff6d9a..9d2e3b0bc866 100644
--- a/drivers/pci/controller/mobiveil/pcie-mobiveil-plat.c
+++ b/drivers/pci/controller/mobiveil/pcie-mobiveil-plat.c
@@ -27,8 +27,8 @@ static int mobiveil_pcie_probe(struct platform_device *pdev)
/* allocate the PCIe port */
bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
- if (!bridge)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
pcie = pci_host_bridge_priv(bridge);
pcie->rp.bridge = bridge;
diff --git a/drivers/pci/controller/pci-aardvark.c b/drivers/pci/controller/pci-aardvark.c
index e34bea1ff0ac..4b75a451efe4 100644
--- a/drivers/pci/controller/pci-aardvark.c
+++ b/drivers/pci/controller/pci-aardvark.c
@@ -1740,8 +1740,8 @@ static int advk_pcie_probe(struct platform_device *pdev)
int ret, irq;
bridge = devm_pci_alloc_host_bridge(dev, sizeof(struct advk_pcie));
- if (!bridge)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
pcie = pci_host_bridge_priv(bridge);
pcie->pdev = pdev;
diff --git a/drivers/pci/controller/pci-ftpci100.c b/drivers/pci/controller/pci-ftpci100.c
index 28e43831c0f1..0618d70fbdda 100644
--- a/drivers/pci/controller/pci-ftpci100.c
+++ b/drivers/pci/controller/pci-ftpci100.c
@@ -419,8 +419,8 @@ static int faraday_pci_probe(struct platform_device *pdev)
u32 val;
host = devm_pci_alloc_host_bridge(dev, sizeof(*p));
- if (!host)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
host->ops = &faraday_pci_ops;
p = pci_host_bridge_priv(host);
diff --git a/drivers/pci/controller/pci-host-common.c b/drivers/pci/controller/pci-host-common.c
index 810d1c8de24e..28c5d55062ed 100644
--- a/drivers/pci/controller/pci-host-common.c
+++ b/drivers/pci/controller/pci-host-common.c
@@ -60,8 +60,8 @@ int pci_host_common_init(struct platform_device *pdev,
struct pci_config_window *cfg;
bridge = devm_pci_alloc_host_bridge(dev, 0);
- if (!bridge)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
of_pci_check_probe_only();
diff --git a/drivers/pci/controller/pci-hyperv.c b/drivers/pci/controller/pci-hyperv.c
index d2b7e8ea710b..0b88e396c323 100644
--- a/drivers/pci/controller/pci-hyperv.c
+++ b/drivers/pci/controller/pci-hyperv.c
@@ -3759,8 +3759,8 @@ static int hv_pci_probe(struct hv_device *hdev,
int ret;
bridge = devm_pci_alloc_host_bridge(&hdev->device, 0);
- if (!bridge)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
hbus = kzalloc(sizeof(*hbus), GFP_KERNEL);
if (!hbus)
diff --git a/drivers/pci/controller/pci-ixp4xx.c b/drivers/pci/controller/pci-ixp4xx.c
index acb85e0d5675..422ec30757ee 100644
--- a/drivers/pci/controller/pci-ixp4xx.c
+++ b/drivers/pci/controller/pci-ixp4xx.c
@@ -528,8 +528,8 @@ static int __init ixp4xx_pci_probe(struct platform_device *pdev)
int i;
host = devm_pci_alloc_host_bridge(dev, sizeof(*p));
- if (!host)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
host->ops = &ixp4xx_pci_ops;
p = pci_host_bridge_priv(host);
diff --git a/drivers/pci/controller/pci-loongson.c b/drivers/pci/controller/pci-loongson.c
index bc630ab8a283..b832d79faf52 100644
--- a/drivers/pci/controller/pci-loongson.c
+++ b/drivers/pci/controller/pci-loongson.c
@@ -326,8 +326,8 @@ static int loongson_pci_probe(struct platform_device *pdev)
return -ENODEV;
bridge = devm_pci_alloc_host_bridge(dev, sizeof(*priv));
- if (!bridge)
- return -ENODEV;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
priv = pci_host_bridge_priv(bridge);
priv->pdev = pdev;
diff --git a/drivers/pci/controller/pci-mvebu.c b/drivers/pci/controller/pci-mvebu.c
index a72aa57591c0..c0fd8efaf540 100644
--- a/drivers/pci/controller/pci-mvebu.c
+++ b/drivers/pci/controller/pci-mvebu.c
@@ -1456,8 +1456,8 @@ static int mvebu_pcie_probe(struct platform_device *pdev)
int num, i, ret;
bridge = devm_pci_alloc_host_bridge(dev, sizeof(struct mvebu_pcie));
- if (!bridge)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
pcie = pci_host_bridge_priv(bridge);
pcie->pdev = pdev;
diff --git a/drivers/pci/controller/pci-rcar-gen2.c b/drivers/pci/controller/pci-rcar-gen2.c
index d29866485361..845347e0317e 100644
--- a/drivers/pci/controller/pci-rcar-gen2.c
+++ b/drivers/pci/controller/pci-rcar-gen2.c
@@ -284,8 +284,8 @@ static int rcar_pci_probe(struct platform_device *pdev)
void __iomem *reg;
bridge = devm_pci_alloc_host_bridge(dev, sizeof(*priv));
- if (!bridge)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
priv = pci_host_bridge_priv(bridge);
bridge->sysdata = priv;
diff --git a/drivers/pci/controller/pci-tegra.c b/drivers/pci/controller/pci-tegra.c
index 467ddc701adc..dc45692e9906 100644
--- a/drivers/pci/controller/pci-tegra.c
+++ b/drivers/pci/controller/pci-tegra.c
@@ -2568,8 +2568,8 @@ static int tegra_pcie_probe(struct platform_device *pdev)
int err;
host = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
- if (!host)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
pcie = pci_host_bridge_priv(host);
host->sysdata = pcie;
diff --git a/drivers/pci/controller/pci-v3-semi.c b/drivers/pci/controller/pci-v3-semi.c
index 460a825325dd..6f1f82e4228d 100644
--- a/drivers/pci/controller/pci-v3-semi.c
+++ b/drivers/pci/controller/pci-v3-semi.c
@@ -715,8 +715,8 @@ static int v3_pci_probe(struct platform_device *pdev)
int ret;
host = devm_pci_alloc_host_bridge(dev, sizeof(*v3));
- if (!host)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
host->ops = &v3_pci_ops;
v3 = pci_host_bridge_priv(host);
diff --git a/drivers/pci/controller/pci-versatile.c b/drivers/pci/controller/pci-versatile.c
index e9a6758fe2c1..b367c17db667 100644
--- a/drivers/pci/controller/pci-versatile.c
+++ b/drivers/pci/controller/pci-versatile.c
@@ -72,8 +72,8 @@ static int versatile_pci_probe(struct platform_device *pdev)
struct pci_host_bridge *bridge;
bridge = devm_pci_alloc_host_bridge(dev, 0);
- if (!bridge)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
versatile_pci_base = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(versatile_pci_base))
diff --git a/drivers/pci/controller/pci-xgene.c b/drivers/pci/controller/pci-xgene.c
index b95afa35201d..3b3a6e08d17b 100644
--- a/drivers/pci/controller/pci-xgene.c
+++ b/drivers/pci/controller/pci-xgene.c
@@ -622,8 +622,8 @@ static int xgene_pcie_probe(struct platform_device *pdev)
"MSI driver not ready\n");
bridge = devm_pci_alloc_host_bridge(dev, sizeof(*port));
- if (!bridge)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
port = pci_host_bridge_priv(bridge);
diff --git a/drivers/pci/controller/pcie-altera.c b/drivers/pci/controller/pcie-altera.c
index 3dbb7adc421c..92f976bea8ef 100644
--- a/drivers/pci/controller/pcie-altera.c
+++ b/drivers/pci/controller/pcie-altera.c
@@ -995,8 +995,8 @@ static int altera_pcie_probe(struct platform_device *pdev)
const struct altera_pcie_data *data;
bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
- if (!bridge)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
pcie = pci_host_bridge_priv(bridge);
pcie->pdev = pdev;
diff --git a/drivers/pci/controller/pcie-brcmstb.c b/drivers/pci/controller/pcie-brcmstb.c
index 9afbd02ded35..c683418c176c 100644
--- a/drivers/pci/controller/pcie-brcmstb.c
+++ b/drivers/pci/controller/pcie-brcmstb.c
@@ -1874,8 +1874,8 @@ static int brcm_pcie_probe(struct platform_device *pdev)
int ret;
bridge = devm_pci_alloc_host_bridge(&pdev->dev, sizeof(*pcie));
- if (!bridge)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
data = of_device_get_match_data(&pdev->dev);
if (!data) {
diff --git a/drivers/pci/controller/pcie-iproc-bcma.c b/drivers/pci/controller/pcie-iproc-bcma.c
index 99a99900444d..d2adc4162a6c 100644
--- a/drivers/pci/controller/pcie-iproc-bcma.c
+++ b/drivers/pci/controller/pcie-iproc-bcma.c
@@ -39,8 +39,8 @@ static int iproc_bcma_pcie_probe(struct bcma_device *bdev)
int ret;
bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
- if (!bridge)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
pcie = pci_host_bridge_priv(bridge);
diff --git a/drivers/pci/controller/pcie-iproc-platform.c b/drivers/pci/controller/pcie-iproc-platform.c
index 0cb78c583c7e..8f6843ce573e 100644
--- a/drivers/pci/controller/pcie-iproc-platform.c
+++ b/drivers/pci/controller/pcie-iproc-platform.c
@@ -46,8 +46,8 @@ static int iproc_pltfm_pcie_probe(struct platform_device *pdev)
int ret;
bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
- if (!bridge)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
pcie = pci_host_bridge_priv(bridge);
diff --git a/drivers/pci/controller/pcie-mediatek-gen3.c b/drivers/pci/controller/pcie-mediatek-gen3.c
index 97147f43e41c..e3e908236238 100644
--- a/drivers/pci/controller/pcie-mediatek-gen3.c
+++ b/drivers/pci/controller/pcie-mediatek-gen3.c
@@ -1175,8 +1175,8 @@ static int mtk_pcie_probe(struct platform_device *pdev)
int err;
host = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
- if (!host)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
pcie = pci_host_bridge_priv(host);
diff --git a/drivers/pci/controller/pcie-mediatek.c b/drivers/pci/controller/pcie-mediatek.c
index 24cc30a2ab6c..7a2c74996ace 100644
--- a/drivers/pci/controller/pcie-mediatek.c
+++ b/drivers/pci/controller/pcie-mediatek.c
@@ -1083,8 +1083,8 @@ static int mtk_pcie_probe(struct platform_device *pdev)
int err;
host = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
- if (!host)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
pcie = pci_host_bridge_priv(host);
diff --git a/drivers/pci/controller/pcie-mt7621.c b/drivers/pci/controller/pcie-mt7621.c
index 01ead2f92e87..9dfa5075b980 100644
--- a/drivers/pci/controller/pcie-mt7621.c
+++ b/drivers/pci/controller/pcie-mt7621.c
@@ -480,8 +480,8 @@ static int mt7621_pcie_probe(struct platform_device *pdev)
return -ENODEV;
bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
- if (!bridge)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
pcie = pci_host_bridge_priv(bridge);
pcie->dev = dev;
diff --git a/drivers/pci/controller/pcie-rcar-host.c b/drivers/pci/controller/pcie-rcar-host.c
index fe288fd770c4..47500ed59608 100644
--- a/drivers/pci/controller/pcie-rcar-host.c
+++ b/drivers/pci/controller/pcie-rcar-host.c
@@ -952,8 +952,8 @@ static int rcar_pcie_probe(struct platform_device *pdev)
int err;
bridge = devm_pci_alloc_host_bridge(dev, sizeof(*host));
- if (!bridge)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
host = pci_host_bridge_priv(bridge);
pcie = &host->pcie;
diff --git a/drivers/pci/controller/pcie-rockchip-host.c b/drivers/pci/controller/pcie-rockchip-host.c
index ee1822ca01db..225a5200f7a6 100644
--- a/drivers/pci/controller/pcie-rockchip-host.c
+++ b/drivers/pci/controller/pcie-rockchip-host.c
@@ -934,8 +934,8 @@ static int rockchip_pcie_probe(struct platform_device *pdev)
return -ENODEV;
bridge = devm_pci_alloc_host_bridge(dev, sizeof(*rockchip));
- if (!bridge)
- return -ENOMEM;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
rockchip = pci_host_bridge_priv(bridge);
diff --git a/drivers/pci/controller/pcie-xilinx-cpm.c b/drivers/pci/controller/pcie-xilinx-cpm.c
index d38f27e20761..1c14c5328ae0 100644
--- a/drivers/pci/controller/pcie-xilinx-cpm.c
+++ b/drivers/pci/controller/pcie-xilinx-cpm.c
@@ -574,8 +574,8 @@ static int xilinx_cpm_pcie_probe(struct platform_device *pdev)
int err;
bridge = devm_pci_alloc_host_bridge(dev, sizeof(*port));
- if (!bridge)
- return -ENODEV;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
port = pci_host_bridge_priv(bridge);
diff --git a/drivers/pci/controller/pcie-xilinx-dma-pl.c b/drivers/pci/controller/pcie-xilinx-dma-pl.c
index b037c8f315e4..0e68026671b8 100644
--- a/drivers/pci/controller/pcie-xilinx-dma-pl.c
+++ b/drivers/pci/controller/pcie-xilinx-dma-pl.c
@@ -771,8 +771,8 @@ static int xilinx_pl_dma_pcie_probe(struct platform_device *pdev)
int err;
bridge = devm_pci_alloc_host_bridge(dev, sizeof(*port));
- if (!bridge)
- return -ENODEV;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
port = pci_host_bridge_priv(bridge);
diff --git a/drivers/pci/controller/pcie-xilinx-nwl.c b/drivers/pci/controller/pcie-xilinx-nwl.c
index 05b8c205493c..a23f5e677c17 100644
--- a/drivers/pci/controller/pcie-xilinx-nwl.c
+++ b/drivers/pci/controller/pcie-xilinx-nwl.c
@@ -834,8 +834,8 @@ static int nwl_pcie_probe(struct platform_device *pdev)
int err;
bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
- if (!bridge)
- return -ENODEV;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
pcie = pci_host_bridge_priv(bridge);
platform_set_drvdata(pdev, pcie);
diff --git a/drivers/pci/controller/pcie-xilinx.c b/drivers/pci/controller/pcie-xilinx.c
index 937ea6ae1ac4..7631af1ef6af 100644
--- a/drivers/pci/controller/pcie-xilinx.c
+++ b/drivers/pci/controller/pcie-xilinx.c
@@ -574,8 +574,8 @@ static int xilinx_pcie_probe(struct platform_device *pdev)
return -ENODEV;
bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
- if (!bridge)
- return -ENODEV;
+ if (IS_ERR_OR_NULL(bridge))
+ return PTR_ERR(bridge);
pcie = pci_host_bridge_priv(bridge);
mutex_init(&pcie->map_lock);
diff --git a/drivers/pci/controller/plda/pcie-plda-host.c b/drivers/pci/controller/plda/pcie-plda-host.c
index 8e2db2e5b64b..28d638067adc 100644
--- a/drivers/pci/controller/plda/pcie-plda-host.c
+++ b/drivers/pci/controller/plda/pcie-plda-host.c
@@ -598,8 +598,8 @@ int plda_pcie_host_init(struct plda_pcie_rp *port, struct pci_ops *ops,
"failed to map config memory\n");
bridge = devm_pci_alloc_host_bridge(dev, 0);
- if (!bridge)
- return dev_err_probe(dev, -ENOMEM,
+ if (IS_ERR_OR_NULL(bridge))
+ return dev_err_probe(dev, PTR_ERR(bridge),
"failed to alloc bridge\n");
if (port->host_ops && port->host_ops->host_init) {
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index f41128f91ca7..e627f36b7683 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -686,18 +686,18 @@ struct pci_host_bridge *devm_pci_alloc_host_bridge(struct device *dev,
bridge = pci_alloc_host_bridge(priv);
if (!bridge)
- return NULL;
+ return ERR_PTR(-ENOMEM);
bridge->dev.parent = dev;
ret = devm_add_action_or_reset(dev, devm_pci_alloc_host_bridge_release,
bridge);
if (ret)
- return NULL;
+ return ERR_PTR(ret);
ret = devm_of_pci_bridge_init(dev, bridge);
if (ret)
- return NULL;
+ return ERR_PTR(ret);
return bridge;
}
@@ -3198,7 +3198,7 @@ struct pci_bus *pci_create_root_bus(struct device *parent, int bus,
bridge = pci_alloc_host_bridge(0);
if (!bridge)
- return NULL;
+ return ERR_PTR(-ENOMEM);
bridge->dev.parent = parent;
--
2.50.1
^ permalink raw reply related
* Re: [PATCH v1 5/6] x86/hyperv: Implement hypervisor ram collection into vmcore
From: Mukesh R @ 2025-09-20 1:42 UTC (permalink / raw)
To: Michael Kelley, linux-hyperv@vger.kernel.org,
linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org
Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org,
decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com,
bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org,
hpa@zytor.com, arnd@arndb.de
In-Reply-To: <58e4f6b3-6ae3-4bf4-3e1f-0981d6af91ea@linux.microsoft.com>
On 9/18/25 19:32, Mukesh R wrote:
> On 9/18/25 16:53, Michael Kelley wrote:
>> From: Mukesh R <mrathor@linux.microsoft.com> Sent: Tuesday, September 16, 2025 6:13 PM
>>>
>>> On 9/15/25 10:55, Michael Kelley wrote:
>>>> From: Mukesh Rathor <mrathor@linux.microsoft.com> Sent: Tuesday, September 9, 2025 5:10 PM
>>>>>
>>>>> Introduce a new file to implement collection of hypervisor ram into the
>>>>
>>>> s/ram/RAM/ (multiple places)
>>>
>>> a quick grep indicates using saying ram is common, i like ram over RAM
>>>
>>>>> vmcore collected by linux. By default, the hypervisor ram is locked, ie,
>>>>> protected via hw page table. Hyper-V implements a disable hypercall which
>>>>
>>>> The terminology here is a bit confusing since you have two names for
>>>> the same thing: "disable" hypervisor, and "devirtualize". Is it possible to
>>>> just use "devirtualize" everywhere, and drop the "disable" terminology?
>>>
>>> The concept is devirtualize and the actual hypercall was originally named
>>> disable. so intermixing is natural imo.
>>>
>>>>> essentially devirtualizes the system on the fly. This mechanism makes the
>>>>> hypervisor ram accessible to linux. Because the hypervisor ram is already
>>>>> mapped into linux address space (as reserved ram),
>>>>
>>>> Is the hypervisor RAM mapped into the VMM process user address space,
>>>> or somewhere in the kernel address space? If the latter, where in the kernel
>>>> code, or what mechanism, does that? Just curious, as I wasn't aware that
>>>> this is happening ....
>>>
>>> mapped in kernel as normal ram and we reserve it very early in boot. i
>>> see that patch has not made it here yet, should be coming very soon.
>>
>> OK, that's fine. The answer to my question is coming soon ....
>>
>>>
>>>>> it is automatically
>>>>> collected into the vmcore without extra work. More details of the
>>>>> implementation are available in the file prologue.
>>>>>
>>>>> Signed-off-by: Mukesh Rathor <mrathor@linux.microsoft.com>
>>>>> ---
>>>>> arch/x86/hyperv/hv_crash.c | 622 +++++++++++++++++++++++++++++++++++++
>>>>> 1 file changed, 622 insertions(+)
>>>>> create mode 100644 arch/x86/hyperv/hv_crash.c
>>>>>
>>>>> diff --git a/arch/x86/hyperv/hv_crash.c b/arch/x86/hyperv/hv_crash.c
>>>>> new file mode 100644
>>>>> index 000000000000..531bac79d598
>>>>> --- /dev/null
>>>>> +++ b/arch/x86/hyperv/hv_crash.c
>>>>> @@ -0,0 +1,622 @@
>>>>> +// SPDX-License-Identifier: GPL-2.0-only
>>>>> +/*
>>>>> + * X86 specific Hyper-V kdump/crash support module
>>>>> + *
>>>>> + * Copyright (C) 2025, Microsoft, Inc.
>>>>> + *
>>>>> + * This module implements hypervisor ram collection into vmcore for both
>>>>> + * cases of the hypervisor crash and linux dom0/root crash.
>>>>
>>>> For a hypervisor crash, does any of this apply to general guest VMs? I'm
>>>> thinking it does not. Hypervisor RAM is collected only into the vmcore
>>>> for the root partition, right? Maybe some additional clarification could be
>>>> added so there's no confusion in this regard.
>>>
>>> it would be odd for guests to collect hyp core, and target audience is
>>> assumed to be those who are somewhat familiar with basic concepts before
>>> getting here.
>>
>> I was unsure because I had not seen any code that adds the hypervisor memory
>> to the Linux memory map. Thought maybe something was going on I hadn?t
>> heard about, so I didn't know the scope of it.
>>
>> Of course, I'm one of those people who was *not* familiar with the basic concepts
>> before getting here. And given that there's no spec available from Hyper-V,
>> the comments in this patch set are all there is for anyone outside of Microsoft.
>> In that vein, I think it's reasonable to provide some description of how this
>> all works in the code comments. And you've done that, which is very
>> helpful. But I encountered a few places where I was confused or unclear, and
>> my suggestions here and in Patch 4 are just about making things as precise as
>> possible without adding a huge amount of additional verbiage. For someone
>> new, English text descriptions that the code can be checked against are
>> helpful, and drawing hard boundaries ("this is only applicable to the root
>> partition") is helpful.
>>
>> If you don't want to deal with it now, I could provide a follow-on patch later
>> that tweaks or augments the wording a bit to clarify some of these places.
>> You can review, like with any patch. I've done wording work over the years
>> to many places in the VMBus code, and more broadly in providing most of
>> the documentation in Documentation/virt/hyperv.
>
> with time, things will start making sense... i find comment pretty clear
> that it collects core for both cases of hv crash and dom0 crash, and no
> mention of guest implies has nothing to do with guests.
>
>>>
>>>> And what *does* happen to guest VMs after a hypervisor crash?
>>>
>>> they are gone... what else could we do?
>>>
>>>>> + * Hyper-V implements
>>>>> + * a devirtualization hypercall with a 32bit protected mode ABI callback. This
>>>>> + * mechanism must be used to unlock hypervisor ram. Since the hypervisor ram
>>>>> + * is already mapped in linux, it is automatically collected into linux vmcore,
>>>>> + * and can be examined by the crash command (raw ram dump) or windbg.
>>>>> + *
>>>>> + * At a high level:
>>>>> + *
>>>>> + * Hypervisor Crash:
>>>>> + * Upon crash, hypervisor goes into an emergency minimal dispatch loop, a
>>>>> + * restrictive mode with very limited hypercall and msr support.
>>>>
>>>> s/msr/MSR/
>>>
>>> msr is used all over, seems acceptable.
>>>
>>>>> + * Each cpu then injects NMIs into dom0/root vcpus.
>>>>
>>>> The "Each cpu" part of this sentence is confusing to me -- which CPUs does
>>>> this refer to? Maybe it would be better to say "It then injects an NMI into
>>>> each dom0/root partition vCPU." without being specific as to which CPUs do
>>>> the injecting since that seems more like a hypervisor implementation detail
>>>> that's not relevant here.
>>>
>>> all cpus in the system. there is a dedicated/pinned dom0 vcpu for each cpu.
>>
>> OK, that makes sense now that I think about it. Each physical CPU in the host
>> has a corresponding vCPU in the dom0/root partition. And each of the vCPUs
>> gets an NMI that sends it to the Linux-in-dom0 NMI handler, even if it was off
>> running a vCPU in some guest VM.
>>
>>>
>>>>> + * A shared page is used to check
>>>>> + * by linux in the nmi handler if the hypervisor has crashed. This shared
>>>>
>>>> s/nmi/NMI/ (multiple places)
>>>
>>>>> + * page is setup in hv_root_crash_init during boot.
>>>>> + *
>>>>> + * Linux Crash:
>>>>> + * In case of linux crash, the callback hv_crash_stop_other_cpus will send
>>>>> + * NMIs to all cpus, then proceed to the crash_nmi_callback where it waits
>>>>> + * for all cpus to be in NMI.
>>>>> + *
>>>>> + * NMI Handler (upon quorum):
>>>>> + * Eventually, in both cases, all cpus wil end up in the nmi hanlder.
>>>>
>>>> s/hanlder/handler/
>>>>
>>>> And maybe just drop the word "wil" (which is misspelled).
>>>>
>>>>> + * Hyper-V requires the disable hypervisor must be done from the bsp. So
>>>>
>>>> s/bsp/BSP (multiple places)
>>>>
>>>>> + * the bsp nmi handler saves current context, does some fixups and makes
>>>>> + * the hypercall to disable the hypervisor, ie, devirtualize. Hypervisor
>>>>> + * at that point will suspend all vcpus (except the bsp), unlock all its
>>>>> + * ram, and return to linux at the 32bit mode entry RIP.
>>>>> + *
>>>>> + * Linux 32bit entry trampoline will then restore long mode and call C
>>>>> + * function here to restore context and continue execution to crash kexec.
>>>>> + */
>>>>> +
>>>>> +#include <linux/delay.h>
>>>>> +#include <linux/kexec.h>
>>>>> +#include <linux/crash_dump.h>
>>>>> +#include <linux/panic.h>
>>>>> +#include <asm/apic.h>
>>>>> +#include <asm/desc.h>
>>>>> +#include <asm/page.h>
>>>>> +#include <asm/pgalloc.h>
>>>>> +#include <asm/mshyperv.h>
>>>>> +#include <asm/nmi.h>
>>>>> +#include <asm/idtentry.h>
>>>>> +#include <asm/reboot.h>
>>>>> +#include <asm/intel_pt.h>
>>>>> +
>>>>> +int hv_crash_enabled;
>>>>
>>>> Seems like this is conceptually a "bool", not an "int".
>>>
>>> yeah, can change it to bool if i do another iteration.
>>>
>>>>> +EXPORT_SYMBOL_GPL(hv_crash_enabled);
>>>>> +
>>>>> +struct hv_crash_ctxt {
>>>>> + ulong rsp;
>>>>> + ulong cr0;
>>>>> + ulong cr2;
>>>>> + ulong cr4;
>>>>> + ulong cr8;
>>>>> +
>>>>> + u16 cs;
>>>>> + u16 ss;
>>>>> + u16 ds;
>>>>> + u16 es;
>>>>> + u16 fs;
>>>>> + u16 gs;
>>>>> +
>>>>> + u16 gdt_fill;
>>>>> + struct desc_ptr gdtr;
>>>>> + char idt_fill[6];
>>>>> + struct desc_ptr idtr;
>>>>> +
>>>>> + u64 gsbase;
>>>>> + u64 efer;
>>>>> + u64 pat;
>>>>> +};
>>>>> +static struct hv_crash_ctxt hv_crash_ctxt;
>>>>> +
>>>>> +/* Shared hypervisor page that contains crash dump area we peek into.
>>>>> + * NB: windbg looks for "hv_cda" symbol so don't change it.
>>>>> + */
>>>>> +static struct hv_crashdump_area *hv_cda;
>>>>> +
>>>>> +static u32 trampoline_pa, devirt_cr3arg;
>>>>> +static atomic_t crash_cpus_wait;
>>>>> +static void *hv_crash_ptpgs[4];
>>>>> +static int hv_has_crashed, lx_has_crashed;
>>>>
>>>> These are conceptually "bool" as well.
>>>>
>>>>> +
>>>>> +/* This cannot be inlined as it needs stack */
>>>>> +static noinline __noclone void hv_crash_restore_tss(void)
>>>>> +{
>>>>> + load_TR_desc();
>>>>> +}
>>>>> +
>>>>> +/* This cannot be inlined as it needs stack */
>>>>> +static noinline void hv_crash_clear_kernpt(void)
>>>>> +{
>>>>> + pgd_t *pgd;
>>>>> + p4d_t *p4d;
>>>>> +
>>>>> + /* Clear entry so it's not confusing to someone looking at the core */
>>>>> + pgd = pgd_offset_k(trampoline_pa);
>>>>> + p4d = p4d_offset(pgd, trampoline_pa);
>>>>> + native_p4d_clear(p4d);
>>>>> +}
>>>>> +
>>>>> +/*
>>>>> + * This is the C entry point from the asm glue code after the devirt hypercall.
>>>>> + * We enter here in IA32-e long mode, ie, full 64bit mode running on kernel
>>>>> + * page tables with our below 4G page identity mapped, but using a temporary
>>>>> + * GDT. ds/fs/gs/es are null. ss is not usable. bp is null. stack is not
>>>>> + * available. We restore kernel GDT, and rest of the context, and continue
>>>>> + * to kexec.
>>>>> + */
>>>>> +static asmlinkage void __noreturn hv_crash_c_entry(void)
>>>>> +{
>>>>> + struct hv_crash_ctxt *ctxt = &hv_crash_ctxt;
>>>>> +
>>>>> + /* first thing, restore kernel gdt */
>>>>> + native_load_gdt(&ctxt->gdtr);
>>>>> +
>>>>> + asm volatile("movw %%ax, %%ss" : : "a"(ctxt->ss));
>>>>> + asm volatile("movq %0, %%rsp" : : "m"(ctxt->rsp));
>>>>> +
>>>>> + asm volatile("movw %%ax, %%ds" : : "a"(ctxt->ds));
>>>>> + asm volatile("movw %%ax, %%es" : : "a"(ctxt->es));
>>>>> + asm volatile("movw %%ax, %%fs" : : "a"(ctxt->fs));
>>>>> + asm volatile("movw %%ax, %%gs" : : "a"(ctxt->gs));
>>>>> +
>>>>> + native_wrmsrq(MSR_IA32_CR_PAT, ctxt->pat);
>>>>> + asm volatile("movq %0, %%cr0" : : "r"(ctxt->cr0));
>>>>> +
>>>>> + asm volatile("movq %0, %%cr8" : : "r"(ctxt->cr8));
>>>>> + asm volatile("movq %0, %%cr4" : : "r"(ctxt->cr4));
>>>>> + asm volatile("movq %0, %%cr2" : : "r"(ctxt->cr4));
>>>>> +
>>>>> + native_load_idt(&ctxt->idtr);
>>>>> + native_wrmsrq(MSR_GS_BASE, ctxt->gsbase);
>>>>> + native_wrmsrq(MSR_EFER, ctxt->efer);
>>>>> +
>>>>> + /* restore the original kernel CS now via far return */
>>>>> + asm volatile("movzwq %0, %%rax\n\t"
>>>>> + "pushq %%rax\n\t"
>>>>> + "pushq $1f\n\t"
>>>>> + "lretq\n\t"
>>>>> + "1:nop\n\t" : : "m"(ctxt->cs) : "rax");
>>>>> +
>>>>> + /* We are in asmlinkage without stack frame, hence make a C function
>>>>> + * call which will buy stack frame to restore the tss or clear PT entry.
>>>>> + */
>>>>> + hv_crash_restore_tss();
>>>>> + hv_crash_clear_kernpt();
>>>>> +
>>>>> + /* we are now fully in devirtualized normal kernel mode */
>>>>> + __crash_kexec(NULL);
>>>>
>>>> The comments for __crash_kexec() say that "panic_cpu" should be set to
>>>> the current CPU. I don't see that such is the case here.
>>>
>>> if linux panic, it would be set by vpanic, if hyp crash, that is
>>> irrelevant.
>>>
>>>>> +
>>>>> + for (;;)
>>>>> + cpu_relax();
>>>>
>>>> Is the intent that __crash_kexec() should never return, on any of the vCPUs,
>>>> because devirtualization isn't done unless there's a valid kdump image loaded?
>>>> I wonder if
>>>>
>>>> native_wrmsrq(HV_X64_MSR_RESET, 1);
>>>>
>>>> would be better than looping forever in case __crash_kexec() fails
>>>> somewhere along the way even if there's a kdump image loaded.
>>>
>>> yeah, i've gone thru all 3 possibilities here:
>>> o loop forever
>>> o reset
>>> o BUG() : this was in V0
>>>
>>> reset is just bad because system would just reboot without any indication
>>> if hyp crashes. with loop at least there is a hang, and one could make
>>> note of it, and if internal, attach debugger.
>>>
>>> BUG is best imo because with hyp gone linux will try to redo panic
>>> and we would print something extra to help. I think i'll just go
>>> back to my V0: BUG()
>>>
>>>>> +}
>>>>> +/* Tell gcc we are using lretq long jump in the above function intentionally */
>>>>> +STACK_FRAME_NON_STANDARD(hv_crash_c_entry);
>>>>> +
>>>>> +static void hv_mark_tss_not_busy(void)
>>>>> +{
>>>>> + struct desc_struct *desc = get_current_gdt_rw();
>>>>> + tss_desc tss;
>>>>> +
>>>>> + memcpy(&tss, &desc[GDT_ENTRY_TSS], sizeof(tss_desc));
>>>>> + tss.type = 0x9; /* available 64-bit TSS. 0xB is busy TSS */
>>>>> + write_gdt_entry(desc, GDT_ENTRY_TSS, &tss, DESC_TSS);
>>>>> +}
>>>>> +
>>>>> +/* Save essential context */
>>>>> +static void hv_hvcrash_ctxt_save(void)
>>>>> +{
>>>>> + struct hv_crash_ctxt *ctxt = &hv_crash_ctxt;
>>>>> +
>>>>> + asm volatile("movq %%rsp,%0" : "=m"(ctxt->rsp));
>>>>> +
>>>>> + ctxt->cr0 = native_read_cr0();
>>>>> + ctxt->cr4 = native_read_cr4();
>>>>> +
>>>>> + asm volatile("movq %%cr2, %0" : "=a"(ctxt->cr2));
>>>>> + asm volatile("movq %%cr8, %0" : "=a"(ctxt->cr8));
>>>>> +
>>>>> + asm volatile("movl %%cs, %%eax" : "=a"(ctxt->cs));
>>>>> + asm volatile("movl %%ss, %%eax" : "=a"(ctxt->ss));
>>>>> + asm volatile("movl %%ds, %%eax" : "=a"(ctxt->ds));
>>>>> + asm volatile("movl %%es, %%eax" : "=a"(ctxt->es));
>>>>> + asm volatile("movl %%fs, %%eax" : "=a"(ctxt->fs));
>>>>> + asm volatile("movl %%gs, %%eax" : "=a"(ctxt->gs));
>>>>> +
>>>>> + native_store_gdt(&ctxt->gdtr);
>>>>> + store_idt(&ctxt->idtr);
>>>>> +
>>>>> + ctxt->gsbase = __rdmsr(MSR_GS_BASE);
>>>>> + ctxt->efer = __rdmsr(MSR_EFER);
>>>>> + ctxt->pat = __rdmsr(MSR_IA32_CR_PAT);
>>>>> +}
>>>>> +
>>>>> +/* Add trampoline page to the kernel pagetable for transition to kernel PT */
>>>>> +static void hv_crash_fixup_kernpt(void)
>>>>> +{
>>>>> + pgd_t *pgd;
>>>>> + p4d_t *p4d;
>>>>> +
>>>>> + pgd = pgd_offset_k(trampoline_pa);
>>>>> + p4d = p4d_offset(pgd, trampoline_pa);
>>>>> +
>>>>> + /* trampoline_pa is below 4G, so no pre-existing entry to clobber */
>>>>> + p4d_populate(&init_mm, p4d, (pud_t *)hv_crash_ptpgs[1]);
>>>>> + p4d->p4d = p4d->p4d & ~(_PAGE_NX); /* enable execute */
>>>>> +}
>>>>> +
>>>>> +/*
>>>>> + * Now that all cpus are in nmi and spinning, we notify the hyp that linux has
>>>>> + * crashed and will collect core. This will cause the hyp to quiesce and
>>>>> + * suspend all VPs except the bsp. Called if linux crashed and not the hyp.
>>>>> + */
>>>>> +static void hv_notify_prepare_hyp(void)
>>>>> +{
>>>>> + u64 status;
>>>>> + struct hv_input_notify_partition_event *input;
>>>>> + struct hv_partition_event_root_crashdump_input *cda;
>>>>> +
>>>>> + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
>>>>> + cda = &input->input.crashdump_input;
>>>>
>>>> The code ordering here is a bit weird. I'd expect this line to be grouped
>>>> with cda->crashdump_action being set.
>>>
>>> we are setting two pointers, and using them later. setting pointers
>>> up front is pretty normal.
>>>
>>>>> + memset(input, 0, sizeof(*input));
>>>>> + input->event = HV_PARTITION_EVENT_ROOT_CRASHDUMP;
>>>>> +
>>>>> + cda->crashdump_action = HV_CRASHDUMP_ENTRY;
>>>>> + status = hv_do_hypercall(HVCALL_NOTIFY_PARTITION_EVENT, input, NULL);
>>>>> + if (!hv_result_success(status))
>>>>> + return;
>>>>> +
>>>>> + cda->crashdump_action = HV_CRASHDUMP_SUSPEND_ALL_VPS;
>>>>> + hv_do_hypercall(HVCALL_NOTIFY_PARTITION_EVENT, input, NULL);
>>>>> +}
>>>>> +
>>>>> +/*
>>>>> + * Common function for all cpus before devirtualization.
>>>>> + *
>>>>> + * Hypervisor crash: all cpus get here in nmi context.
>>>>> + * Linux crash: the panicing cpu gets here at base level, all others in nmi
>>>>> + * context. Note, panicing cpu may not be the bsp.
>>>>> + *
>>>>> + * The function is not inlined so it will show on the stack. It is named so
>>>>> + * because the crash cmd looks for certain well known function names on the
>>>>> + * stack before looking into the cpu saved note in the elf section, and
>>>>> + * that work is currently incomplete.
>>>>> + *
>>>>> + * Notes:
>>>>> + * Hypervisor crash:
>>>>> + * - the hypervisor is in a very restrictive mode at this point and any
>>>>> + * vmexit it cannot handle would result in reboot. For example, console
>>>>> + * output from here would result in synic ipi hcall, which would result
>>>>> + * in reboot. So, no mumbo jumbo, just get to kexec as quickly as possible.
>>>>> + *
>>>>> + * Devirtualization is supported from the bsp only.
>>>>> + */
>>>>> +static noinline __noclone void crash_nmi_callback(struct pt_regs *regs)
>>>>> +{
>>>>> + struct hv_input_disable_hyp_ex *input;
>>>>> + u64 status;
>>>>> + int msecs = 1000, ccpu = smp_processor_id();
>>>>> +
>>>>> + if (ccpu == 0) {
>>>>> + /* crash_save_cpu() will be done in the kexec path */
>>>>> + cpu_emergency_stop_pt(); /* disable performance trace */
>>>>> + atomic_inc(&crash_cpus_wait);
>>>>> + } else {
>>>>> + crash_save_cpu(regs, ccpu);
>>>>> + cpu_emergency_stop_pt(); /* disable performance trace */
>>>>> + atomic_inc(&crash_cpus_wait);
>>>>> + for (;;); /* cause no vmexits */
>>>>> + }
>>>>> +
>>>>> + while (atomic_read(&crash_cpus_wait) < num_online_cpus() && msecs--)
>>>>> + mdelay(1);
>>>>> +
>>>>> + stop_nmi();
>>>>> + if (!hv_has_crashed)
>>>>> + hv_notify_prepare_hyp();
>>>>> +
>>>>> + if (crashing_cpu == -1)
>>>>> + crashing_cpu = ccpu; /* crash cmd uses this */
>>>>
>>>> Could just be "crashing_cpu = 0" since only the BSP gets here.
>>>
>>> a code change request has been open for while to remove the requirement
>>> of bsp..
>>>
>>>>> +
>>>>> + hv_hvcrash_ctxt_save();
>>>>> + hv_mark_tss_not_busy();
>>>>> + hv_crash_fixup_kernpt();
>>>>> +
>>>>> + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
>>>>> + memset(input, 0, sizeof(*input));
>>>>> + input->rip = trampoline_pa; /* PA of hv_crash_asm32 */
>>>>> + input->arg = devirt_cr3arg; /* PA of trampoline page table L4 */
>>>>
>>>> Is this comment correct? Isn't it the PA of struct hv_crash_tramp_data?
>>>> And just for clarification, Hyper-V treats this "arg" value as opaque and does
>>>> not access it. It only provides it in EDI when it invokes the trampoline
>>>> function, right?
>>>
>>> comment is correct. cr3 always points to l4 (or l5 if 5 level page tables).
>>
>> Yes, the comment matches the name of the "devirt_cr3arg" variable.
>> Unfortunately my previous comment was incomplete because the value
>> stored in the static variable "devirt_cr3arg" isn?t the address of an L4 page
>> table. It's not a CR3 value. The value stored in devirt_cr3arg is actually the
>> PA of struct hv_crash_tramp_data. The CR3 value is stored in the
>> tramp32_cr3 field (at offset 0) of that structure, so there's an additional level
>> of indirection. The (corrected) comment in the header to hv_crash_asm32()
>> describes EDI as containing "PA of struct hv_crash_tramp_data", which
>> ought to match what is described here. I'd say that "devirt_cr3arg" ought
>> to be renamed to "tramp_data_pa" or something else parallel to
>> "trampoline_pa".
>
> hyp needs trampoline cr3 for transition, we pass it as an arg. we piggy
> back extra information for ourselves needed in trampoline.S. so it's
> all good.
actually, what i said earlier was true, not above. that the arg is
opaque and hyp does not use it (we are transitioning paging off after
all!). i did this all almost two years ago, so had vague recollections
but finally had time today to go back to square one and old notes,
and remember things now. so final answer:
the hypercall calls it TrampolineCr3, i guess this is how windows uses it
(they have customized kernel code for core collection). doing that was
becoming too intrusive on linux, so i decided to use the arg to pass the
info i needed in the trampoline code. Since the hypercall calls the arg
TrampolineCr3, i must have just used that name for the arg to match it,
probably falsely assuming hypervisor somehow looked at it. (actually,
the windows hypercall wrapper does look at it to make sure it is a
ram address).
since the hypercall doesn't use the arg, it could just call it
devirtArg, but maybe in the past they used it somehow. in my latest
version, i just call it devirt_arg.
>>> right, comes in edi, i don't know what EDI is (just kidding!)...
>>>
>>>>> +
>>>>> + status = hv_do_hypercall(HVCALL_DISABLE_HYP_EX, input, NULL);
>>>>> +
>>>>> + /* Devirt failed, just reboot as things are in very bad state now */
>>>>> + native_wrmsrq(HV_X64_MSR_RESET, 1); /* get hv to reboot */
>>>>> +}
>>>>> +
>>>>> +/*
>>>>> + * Generic nmi callback handler: could be called without any crash also.
>>>>> + * hv crash: hypervisor injects nmi's into all cpus
>>>>> + * lx crash: panicing cpu sends nmi to all but self via crash_stop_other_cpus
>>>>> + */
>>>>> +static int hv_crash_nmi_local(unsigned int cmd, struct pt_regs *regs)
>>>>> +{
>>>>> + int ccpu = smp_processor_id();
>>>>> +
>>>>> + if (!hv_has_crashed && hv_cda && hv_cda->cda_valid)
>>>>> + hv_has_crashed = 1;
>>>>> +
>>>>> + if (!hv_has_crashed && !lx_has_crashed)
>>>>> + return NMI_DONE; /* ignore the nmi */
>>>>> +
>>>>> + if (hv_has_crashed) {
>>>>> + if (!kexec_crash_loaded() || !hv_crash_enabled) {
>>>>> + if (ccpu == 0) {
>>>>> + native_wrmsrq(HV_X64_MSR_RESET, 1); /* reboot */
>>>>> + } else
>>>>> + for (;;); /* cause no vmexits */
>>>>> + }
>>>>> + }
>>>>> +
>>>>> + crash_nmi_callback(regs);
>>>>> +
>>>>> + return NMI_DONE;
>>>>
>>>> crash_nmi_callback() should never return, right? Normally one would
>>>> expect to return NMI_HANDLED here, but I guess it doesn't matter
>>>> if the return is never executed.
>>>
>>> correct.
>>>
>>>>> +}
>>>>> +
>>>>> +/*
>>>>> + * hv_crash_stop_other_cpus() == smp_ops.crash_stop_other_cpus
>>>>> + *
>>>>> + * On normal linux panic, this is called twice: first from panic and then again
>>>>> + * from native_machine_crash_shutdown.
>>>>> + *
>>>>> + * In case of mshv, 3 ways to get here:
>>>>> + * 1. hv crash (only bsp will get here):
>>>>> + * BSP : nmi callback -> DisableHv -> hv_crash_asm32 -> hv_crash_c_entry
>>>>> + * -> __crash_kexec -> native_machine_crash_shutdown
>>>>> + * -> crash_smp_send_stop -> smp_ops.crash_stop_other_cpus
>>>>> + * linux panic:
>>>>> + * 2. panic cpu x: panic() -> crash_smp_send_stop
>>>>> + * -> smp_ops.crash_stop_other_cpus
>>>>> + * 3. bsp: native_machine_crash_shutdown -> crash_smp_send_stop
>>>>> + *
>>>>> + * NB: noclone and non standard stack because of call to crash_setup_regs().
>>>>> + */
>>>>> +static void __noclone hv_crash_stop_other_cpus(void)
>>>>> +{
>>>>> + static int crash_stop_done;
>>>>> + struct pt_regs lregs;
>>>>> + int ccpu = smp_processor_id();
>>>>> +
>>>>> + if (hv_has_crashed)
>>>>> + return; /* all cpus already in nmi handler path */
>>>>> +
>>>>> + if (!kexec_crash_loaded())
>>>>> + return;
>>>>
>>>> If we're in a normal panic path (your Case #2 above) with no kdump kernel
>>>> loaded, why leave the other vCPUs running? Seems like that could violate
>>>> expectations in vpanic(), where it calls panic_other_cpus_shutdown() and
>>>> thereafter assumes other vCPUs are not running.
>>>
>>> no, there is lots of complexity here!
>>>
>>> if we hang vcpus here, hyp will note and may trigger its own watchdog.
>>> also, machine_crash_shutdown() does another ipi.
>>>
>>> I think the best thing to do here is go back to my V0 which did not
>>> have check for kexec_crash_loaded(), but had this in hv_crash_c_entry:
>>>
>>> + /* we are now fully in devirtualized normal kernel mode */
>>> + __crash_kexec(NULL);
>>> +
>>> + BUG();
>>>
>>>
>>> this way hyp would be disabled, ie, system devirtualized, and
>>> __crash_kernel() will return, resulting in BUG() that will cause
>>> it to go thru panic and honor panic= parameter with either hang
>>> or reset. instead of bug, i could just call panic() also.
>>>
>>>>> +
>>>>> + if (crash_stop_done)
>>>>> + return;
>>>>> + crash_stop_done = 1;
>>>>
>>>> Is crash_stop_done necessary? hv_crash_stop_other_cpus() is called
>>>> from crash_smp_send_stop(), which has its own static variable
>>>> "cpus_stopped" that does the same thing.
>>>
>>> yes. for error paths.
>>>
>>>>> +
>>>>> + /* linux has crashed: hv is healthy, we can ipi safely */
>>>>> + lx_has_crashed = 1;
>>>>> + wmb(); /* nmi handlers look at lx_has_crashed */
>>>>> +
>>>>> + apic->send_IPI_allbutself(NMI_VECTOR);
>>>>
>>>> The default .crash_stop_other_cpus function is kdump_nmi_shootdown_cpus().
>>>> In addition to sending the NMI IPI, it does disable_local_APIC(). I don't know, but
>>>> should disable_local_APIC() be done somewhere here as well?
>>>
>>> no, hyp does that.
>>
>> As part of the devirt operation initiated by the HVCALL_DISABLE_HYP_EX
>> hypercall in crash_nmi_callback()? This gets back to an earlier question/comment
>> where I was trying to figure out if the APIC is still enabled, and in what mode,
>> when hv_crash_asm32() is invoked.
>
>>>
>>>>> +
>>>>> + if (crashing_cpu == -1)
>>>>> + crashing_cpu = ccpu; /* crash cmd uses this */
>>>>> +
>>>>> + /* crash_setup_regs() happens in kexec also, but for the kexec cpu which
>>>>> + * is the bsp. We could be here on non-bsp cpu, collect regs if so.
>>>>> + */
>>>>> + if (ccpu)
>>>>> + crash_setup_regs(&lregs, NULL);
>>>>> +
>>>>> + crash_nmi_callback(&lregs);
>>>>> +}
>>>>> +STACK_FRAME_NON_STANDARD(hv_crash_stop_other_cpus);
>>>>> +
>>>>> +/* This GDT is accessed in IA32-e compat mode which uses 32bits addresses */
>>>>> +struct hv_gdtreg_32 {
>>>>> + u16 fill;
>>>>> + u16 limit;
>>>>> + u32 address;
>>>>> +} __packed;
>>>>> +
>>>>> +/* We need a CS with L bit to goto IA32-e long mode from 32bit compat mode */
>>>>> +struct hv_crash_tramp_gdt {
>>>>> + u64 null; /* index 0, selector 0, null selector */
>>>>> + u64 cs64; /* index 1, selector 8, cs64 selector */
>>>>> +} __packed;
>>>>> +
>>>>> +/* No stack, so jump via far ptr in memory to load the 64bit CS */
>>>>> +struct hv_cs_jmptgt {
>>>>> + u32 address;
>>>>> + u16 csval;
>>>>> + u16 fill;
>>>>> +} __packed;
>>>>> +
>>>>> +/* This trampoline data is copied onto the trampoline page after the asm code */
>>>>> +struct hv_crash_tramp_data {
>>>>> + u64 tramp32_cr3;
>>>>> + u64 kernel_cr3;
>>>>> + struct hv_gdtreg_32 gdtr32;
>>>>> + struct hv_crash_tramp_gdt tramp_gdt;
>>>>> + struct hv_cs_jmptgt cs_jmptgt;
>>>>> + u64 c_entry_addr;
>>>>> +} __packed;
>>>>> +
>>>>> +/*
>>>>> + * Setup a temporary gdt to allow the asm code to switch to the long mode.
>>>>> + * Since the asm code is relocated/copied to a below 4G page, it cannot use rip
>>>>> + * relative addressing, hence we must use trampoline_pa here. Also, save other
>>>>> + * info like jmp and C entry targets for same reasons.
>>>>> + *
>>>>> + * Returns: 0 on success, -1 on error
>>>>> + */
>>>>> +static int hv_crash_setup_trampdata(u64 trampoline_va)
>>>>> +{
>>>>> + int size, offs;
>>>>> + void *dest;
>>>>> + struct hv_crash_tramp_data *tramp;
>>>>> +
>>>>> + /* These must match exactly the ones in the corresponding asm file */
>>>>> + BUILD_BUG_ON(offsetof(struct hv_crash_tramp_data, tramp32_cr3) != 0);
>>>>> + BUILD_BUG_ON(offsetof(struct hv_crash_tramp_data, kernel_cr3) != 8);
>>>>> + BUILD_BUG_ON(offsetof(struct hv_crash_tramp_data, gdtr32.limit) != 18);
>>>>> + BUILD_BUG_ON(offsetof(struct hv_crash_tramp_data,
>>>>> + cs_jmptgt.address) != 40);
>>>>
>>>> It would be nice to pick up the constants from a #include file that is
>>>> shared with the asm code in Patch 4 of the series.
>>>
>>> yeah, could go either way, some don't like tiny headers... if there are
>>> no objections to new header for this, i could go that way too.
>>
>> Saw your follow-on comments about this as well. The tiny header
>> is ugly. It's a judgment call that can go either way, so go with your
>> preference.
>>
>>>
>>>>> +
>>>>> + /* hv_crash_asm_end is beyond last byte by 1 */
>>>>> + size = &hv_crash_asm_end - &hv_crash_asm32;
>>>>> + if (size + sizeof(struct hv_crash_tramp_data) > PAGE_SIZE) {
>>>>> + pr_err("%s: trampoline page overflow\n", __func__);
>>>>> + return -1;
>>>>> + }
>>>>> +
>>>>> + dest = (void *)trampoline_va;
>>>>> + memcpy(dest, &hv_crash_asm32, size);
>>>>> +
>>>>> + dest += size;
>>>>> + dest = (void *)round_up((ulong)dest, 16);
>>>>> + tramp = (struct hv_crash_tramp_data *)dest;
>>>>> +
>>>>> + /* see MAX_ASID_AVAILABLE in tlb.c: "PCID 0 is reserved for use by
>>>>> + * non-PCID-aware users". Build cr3 with pcid 0
>>>>> + */
>>>>> + tramp->tramp32_cr3 = __sme_pa(hv_crash_ptpgs[0]);
>>>>> +
>>>>> + /* Note, when restoring X86_CR4_PCIDE, cr3[11:0] must be zero */
>>>>> + tramp->kernel_cr3 = __sme_pa(init_mm.pgd);
>>>>> +
>>>>> + tramp->gdtr32.limit = sizeof(struct hv_crash_tramp_gdt);
>>>>> + tramp->gdtr32.address = trampoline_pa +
>>>>> + (ulong)&tramp->tramp_gdt - trampoline_va;
>>>>> +
>>>>> + /* base:0 limit:0xfffff type:b dpl:0 P:1 L:1 D:0 avl:0 G:1 */
>>>>> + tramp->tramp_gdt.cs64 = 0x00af9a000000ffff;
>>>>> +
>>>>> + tramp->cs_jmptgt.csval = 0x8;
>>>>> + offs = (ulong)&hv_crash_asm64_lbl - (ulong)&hv_crash_asm32;
>>>>> + tramp->cs_jmptgt.address = trampoline_pa + offs;
>>>>> +
>>>>> + tramp->c_entry_addr = (u64)&hv_crash_c_entry;
>>>>> +
>>>>> + devirt_cr3arg = trampoline_pa + (ulong)dest - trampoline_va;
>>>>> +
>>>>> + return 0;
>>>>> +}
>>>>> +
>>>>> +/*
>>>>> + * Build 32bit trampoline page table for transition from protected mode
>>>>> + * non-paging to long-mode paging. This transition needs pagetables below 4G.
>>>>> + */
>>>>> +static void hv_crash_build_tramp_pt(void)
>>>>> +{
>>>>> + p4d_t *p4d;
>>>>> + pud_t *pud;
>>>>> + pmd_t *pmd;
>>>>> + pte_t *pte;
>>>>> + u64 pa, addr = trampoline_pa;
>>>>> +
>>>>> + p4d = hv_crash_ptpgs[0] + pgd_index(addr) * sizeof(p4d);
>>>>> + pa = virt_to_phys(hv_crash_ptpgs[1]);
>>>>> + set_p4d(p4d, __p4d(_PAGE_TABLE | pa));
>>>>> + p4d->p4d &= ~(_PAGE_NX); /* disable no execute */
>>>>> +
>>>>> + pud = hv_crash_ptpgs[1] + pud_index(addr) * sizeof(pud);
>>>>> + pa = virt_to_phys(hv_crash_ptpgs[2]);
>>>>> + set_pud(pud, __pud(_PAGE_TABLE | pa));
>>>>> +
>>>>> + pmd = hv_crash_ptpgs[2] + pmd_index(addr) * sizeof(pmd);
>>>>> + pa = virt_to_phys(hv_crash_ptpgs[3]);
>>>>> + set_pmd(pmd, __pmd(_PAGE_TABLE | pa));
>>>>> +
>>>>> + pte = hv_crash_ptpgs[3] + pte_index(addr) * sizeof(pte);
>>>>> + set_pte(pte, pfn_pte(addr >> PAGE_SHIFT, PAGE_KERNEL_EXEC));
>>>>> +}
>>>>> +
>>>>> +/*
>>>>> + * Setup trampoline for devirtualization:
>>>>> + * - a page below 4G, ie 32bit addr containing asm glue code that mshv jmps to
>>>>> + * in protected mode.
>>>>> + * - 4 pages for a temporary page table that asm code uses to turn paging on
>>>>> + * - a temporary gdt to use in the compat mode.
>>>>> + *
>>>>> + * Returns: 0 on success
>>>>> + */
>>>>> +static int hv_crash_trampoline_setup(void)
>>>>> +{
>>>>> + int i, rc, order;
>>>>> + struct page *page;
>>>>> + u64 trampoline_va;
>>>>> + gfp_t flags32 = GFP_KERNEL | GFP_DMA32 | __GFP_ZERO;
>>>>> +
>>>>> + /* page for 32bit trampoline assembly code + hv_crash_tramp_data */
>>>>> + page = alloc_page(flags32);
>>>>> + if (page == NULL) {
>>>>> + pr_err("%s: failed to alloc asm stub page\n", __func__);
>>>>> + return -1;
>>>>> + }
>>>>> +
>>>>> + trampoline_va = (u64)page_to_virt(page);
>>>>> + trampoline_pa = (u32)page_to_phys(page);
>>>>> +
>>>>> + order = 2; /* alloc 2^2 pages */
>>>>> + page = alloc_pages(flags32, order);
>>>>> + if (page == NULL) {
>>>>> + pr_err("%s: failed to alloc pt pages\n", __func__);
>>>>> + free_page(trampoline_va);
>>>>> + return -1;
>>>>> + }
>>>>> +
>>>>> + for (i = 0; i < 4; i++, page++)
>>>>> + hv_crash_ptpgs[i] = page_to_virt(page);
>>>>> +
>>>>> + hv_crash_build_tramp_pt();
>>>>> +
>>>>> + rc = hv_crash_setup_trampdata(trampoline_va);
>>>>> + if (rc)
>>>>> + goto errout;
>>>>> +
>>>>> + return 0;
>>>>> +
>>>>> +errout:
>>>>> + free_page(trampoline_va);
>>>>> + free_pages((ulong)hv_crash_ptpgs[0], order);
>>>>> +
>>>>> + return rc;
>>>>> +}
>>>>> +
>>>>> +/* Setup for kdump kexec to collect hypervisor ram when running as mshv root */
>>>>> +void hv_root_crash_init(void)
>>>>> +{
>>>>> + int rc;
>>>>> + struct hv_input_get_system_property *input;
>>>>> + struct hv_output_get_system_property *output;
>>>>> + unsigned long flags;
>>>>> + u64 status;
>>>>> + union hv_pfn_range cda_info;
>>>>> +
>>>>> + if (pgtable_l5_enabled()) {
>>>>> + pr_err("Hyper-V: crash dump not yet supported on 5level PTs\n");
>>>>> + return;
>>>>> + }
>>>>> +
>>>>> + rc = register_nmi_handler(NMI_LOCAL, hv_crash_nmi_local, NMI_FLAG_FIRST,
>>>>> + "hv_crash_nmi");
>>>>> + if (rc) {
>>>>> + pr_err("Hyper-V: failed to register crash nmi handler\n");
>>>>> + return;
>>>>> + }
>>>>> +
>>>>> + local_irq_save(flags);
>>>>> + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
>>>>> + output = *this_cpu_ptr(hyperv_pcpu_output_arg);
>>>>> +
>>>>> + memset(input, 0, sizeof(*input));
>>>>> + memset(output, 0, sizeof(*output));
>>>>
>>>> Why zero the output area? This is one of those hypercall things that we're
>>>> inconsistent about. A few hypercall call sites zero the output area, and it's
>>>> not clear why they do. Hyper-V should be responsible for properly filling in
>>>> the output area. Linux should not need to do this zero'ing, unless there's some
>>>> known bug in Hyper-V for certain hypercalls, in which case there should be
>>>> a code comment stating "why".
>>>
>>> for the same reason sometimes you see char *p = NULL, either leftover
>>> code or someone was debugging or just copy and paste. this is just copy
>>> paste. i agree in general that we don't need to clear it at all, in fact,
>>> i'd like to remove them all! but i also understand people with different
>>> skills and junior members find it easier to debug, and also we were in
>>> early product development. for that reason, it doesn't have to be
>>> consistent either, if some complex hypercalls are failing repeatedly,
>>> just for ease of debug, one might leave it there temporarily. but
>>> now that things are stable, i think we should just remove them all and
>>> get used to a bit more inconvenient debugging...
>>
>> I see your point about debugging, but on balance I agree that they
>> should all be removed. If there's some debug case, add it back
>> temporarily to debug, but leave upstream without it. The zero'ing is
>> also unnecessary code in the interrupt disabled window, which you
>> have expressed concern about in a different thread.
>
> yeah, i've been extremely busy so not able to pay much attention to
> upstreaming, but imo they should have been removed before upstreaming.
> a simple patch that just removes memset of output would be welcome.
>
>>>
>>>>> + input->property_id = HV_SYSTEM_PROPERTY_CRASHDUMPAREA;
>>>>> +
>>>>> + status = hv_do_hypercall(HVCALL_GET_SYSTEM_PROPERTY, input, output);
>>>>> + cda_info.as_uint64 = output->hv_cda_info.as_uint64;
>>>>> + local_irq_restore(flags);
>>>>> +
>>>>> + if (!hv_result_success(status)) {
>>>>> + pr_err("Hyper-V: %s: property:%d %s\n", __func__,
>>>>> + input->property_id, hv_result_to_string(status));
>>>>> + goto err_out;
>>>>> + }
>>>>> +
>>>>> + if (cda_info.base_pfn == 0) {
>>>>> + pr_err("Hyper-V: hypervisor crash dump area pfn is 0\n");
>>>>> + goto err_out;
>>>>> + }
>>>>> +
>>>>> + hv_cda = phys_to_virt(cda_info.base_pfn << PAGE_SHIFT);
>>>>
>>>> Use HV_HYP_PAGE_SHIFT, since PFNs provided by Hyper-V are always in
>>>> terms of the Hyper-V page size, which isn't necessarily the guest page size.
>>>> Yes, on x86 there's no difference, but for future robustness ....
>>>
>>> i don't know about guests, but we won't even boot if dom0 pg size
>>> didn't match.. but easier to change than to make the case..
>>
>> FWIW, a normal Linux guest on ARM64 works just fine with a page
>> size of 16K or 64K, even though the underlying Hyper-V page size
>> is only 4K. That's why we have HV_HYP_PAGE_SHIFT and related in
>> the first place. Using it properly really matters for normal guests.
>> (Having the guest page size smaller than the Hyper-V page size
>> does *not* work, but there are no such use cases.)
>>
>> Even on ARM64, I know the root partition page size is required to
>> match the Hyper-V page size. But using HV_HYP_PAGE_SIZE is
>> still appropriate just to not leave code that will go wrong if the
>> match requirement should ever change.
>>
>>>
>>>>> +
>>>>> + rc = hv_crash_trampoline_setup();
>>>>> + if (rc)
>>>>> + goto err_out;
>>>>> +
>>>>> + smp_ops.crash_stop_other_cpus = hv_crash_stop_other_cpus;
>>>>> +
>>>>> + crash_kexec_post_notifiers = true;
>>>>> + hv_crash_enabled = 1;
>>>>> + pr_info("Hyper-V: linux and hv kdump support enabled\n");
>>>>
>>>> This message and the message below aren't consistent. One refers
>>>> to "hv kdump" and the other to "hyp kdump".
>>>
>>>>> +
>>>>> + return;
>>>>> +
>>>>> +err_out:
>>>>> + unregister_nmi_handler(NMI_LOCAL, "hv_crash_nmi");
>>>>> + pr_err("Hyper-V: only linux (but not hyp) kdump support enabled\n");
>>>>> +}
>>>>> --
>>>>> 2.36.1.vfs.0.0
^ permalink raw reply
* Re: [PATCH v3 5/5] mshv: Introduce new hypercall to map stats page for L1VH partitions
From: Nuno Das Neves @ 2025-09-19 22:44 UTC (permalink / raw)
To: Stanislav Kinsburskii
Cc: linux-hyperv, linux-kernel, prapal, easwar.hariharan, tiala,
anirudh, paekkaladevi, kys, haiyangz, wei.liu, decui, Jinank Jain
In-Reply-To: <aMxjORzTO0DgWq9q@skinsburskii.localdomain>
On 9/18/2025 12:53 PM, Stanislav Kinsburskii wrote:
> On Tue, Sep 16, 2025 at 04:44:22PM -0700, Nuno Das Neves wrote:
>> From: Jinank Jain <jinankjain@linux.microsoft.com>
>>
>
> <snip>
>
>> +static int hv_call_map_stats_page2(enum hv_stats_object_type type,
>> + const union hv_stats_object_identity *identity,
>> + u64 map_location)
>> +{
>> + unsigned long flags;
>> + struct hv_input_map_stats_page2 *input;
>> + u64 status;
>> + int ret;
>> +
>> + if (!map_location || !mshv_use_overlay_gpfn())
>> + return -EINVAL;
>> +
>> + do {
>> + local_irq_save(flags);
>> + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
>> +
>> + memset(input, 0, sizeof(*input));
>> + input->type = type;
>> + input->identity = *identity;
>> + input->map_location = map_location;
>> +
>> + status = hv_do_hypercall(HVCALL_MAP_STATS_PAGE2, input, NULL);
>> +
>> + local_irq_restore(flags);
>> + if (hv_result(status) != HV_STATUS_INSUFFICIENT_MEMORY) {
>> + if (hv_result_success(status))
>> + break;
>> + hv_status_debug(status, "\n");
>
> It looks more natural to check for success first and break the loop, and
> only then handle errors.
> Maybe even set ret for both success and error messages and break and
> handle only the unsufficient memory status.
>
Something like this?
local_irq_restore(flags);
ret = hv_result_to_errno(status);
if (!ret)
break;
if (hv_result(status) != HV_STATUS_INSUFFICIENT_MEMORY) {
hv_status_debug(status, "\n");
break;
}
ret = hv_call_deposit_pages(NUMA_NO_NODE,
hv_current_partition_id, 1);
>> @@ -865,6 +931,19 @@ int hv_call_unmap_stat_page(enum hv_stats_object_type type,
>> return hv_result_to_errno(status);
>> }
>>
>> +int hv_unmap_stats_page(enum hv_stats_object_type type, void *page_addr,
>> + const union hv_stats_object_identity *identity)
>> +{
>
> Should this function be type of void?
>
The return type is consistent with the other hypercall helpers. It's true that
in practice we don't ever check if the unmap succeeded. I think it's fine as-is.
> Thanks,
> Stanislav
^ permalink raw reply
* Re: [PATCH v3 3/5] mshv: Get the vmm capabilities offered by the hypervisor
From: Nuno Das Neves @ 2025-09-19 22:37 UTC (permalink / raw)
To: Stanislav Kinsburskii
Cc: linux-hyperv, linux-kernel, prapal, easwar.hariharan, tiala,
anirudh, paekkaladevi, kys, haiyangz, wei.liu, decui
In-Reply-To: <aMxUe7WLzMXJY16c@skinsburskii.localdomain>
On 9/18/2025 11:50 AM, Stanislav Kinsburskii wrote:
> On Tue, Sep 16, 2025 at 04:44:20PM -0700, Nuno Das Neves wrote:
>> From: Purna Pavan Chandra Aekkaladevi <paekkaladevi@linux.microsoft.com>
>>
>> Some hypervisor APIs are gated by feature bits in the
>> "vmm capabilities" partition property. Store the capabilities on
>> mshv_root module init, using HVCALL_GET_PARTITION_PROPERTY_EX.
>>
>> This is not supported on all hypervisors. In that case, just set the
>> capabilities to 0 and proceed as normal.
>>
>> Signed-off-by: Purna Pavan Chandra Aekkaladevi <paekkaladevi@linux.microsoft.com>
>> Signed-off-by: Nuno Das Neves <nunodasneves@linux.microsoft.com>
>> Reviewed-by: Praveen K Paladugu <prapal@linux.microsoft.com>
>> Reviewed-by: Easwar Hariharan <easwar.hariharan@linux.microsoft.com>
>> Reviewed-by: Tianyu Lan <tiala@microsoft.com>
>> ---
>> drivers/hv/mshv_root.h | 1 +
>> drivers/hv/mshv_root_main.c | 22 ++++++++++++++++++++++
>> 2 files changed, 23 insertions(+)
>>
>> diff --git a/drivers/hv/mshv_root.h b/drivers/hv/mshv_root.h
>> index 4aeb03bea6b6..0cb1e2589fe1 100644
>> --- a/drivers/hv/mshv_root.h
>> +++ b/drivers/hv/mshv_root.h
>> @@ -178,6 +178,7 @@ struct mshv_root {
>> struct hv_synic_pages __percpu *synic_pages;
>> spinlock_t pt_ht_lock;
>> DECLARE_HASHTABLE(pt_htable, MSHV_PARTITIONS_HASH_BITS);
>> + struct hv_partition_property_vmm_capabilities vmm_caps;
>> };
>>
>> /*
>> diff --git a/drivers/hv/mshv_root_main.c b/drivers/hv/mshv_root_main.c
>> index 24df47726363..f7738cefbdf3 100644
>> --- a/drivers/hv/mshv_root_main.c
>> +++ b/drivers/hv/mshv_root_main.c
>> @@ -2201,6 +2201,26 @@ static int __init mshv_root_partition_init(struct device *dev)
>> return err;
>> }
>>
>> +static void mshv_init_vmm_caps(struct device *dev)
>> +{
>> + int ret;
>
> nit: this is void function so ret looks redundant.
>
True, it's not needed.
>> +
>> + memset(&mshv_root.vmm_caps, 0, sizeof(mshv_root.vmm_caps));
>
> Zeroying is redundant as mshv_root is a statci variable.
>
Good point.
>> + ret = hv_call_get_partition_property_ex(HV_PARTITION_ID_SELF,
>> + HV_PARTITION_PROPERTY_VMM_CAPABILITIES,
>> + 0, &mshv_root.vmm_caps,
>
> Also, we align "slow" hypercalls by PAGE_SIZE. Why is it fine to not do
> it here?
>
I guess you're referring to the output argument of the hypercall?
Check the previous patch to see how hv_call_get_partition_property_ex()
is implemented. It uses the per-cpu input/output args as normal, which
are HV_HYP_PAGE_SIZE in size, and page-aligned.
> Thanks,
> Stanislav
>
>> + sizeof(mshv_root.vmm_caps));
>> +
>> + /*
>> + * HVCALL_GET_PARTITION_PROPERTY_EX or HV_PARTITION_PROPERTY_VMM_CAPABILITIES
>> + * may not be supported. Leave them as 0 in that case.
>> + */
>> + if (ret)
>> + dev_warn(dev, "Unable to get VMM capabilities\n");
>> +
>> + dev_dbg(dev, "vmm_caps=0x%llx\n", mshv_root.vmm_caps.as_uint64[0]);
>> +}
>> +
>> static int __init mshv_parent_partition_init(void)
>> {
>> int ret;
>> @@ -2253,6 +2273,8 @@ static int __init mshv_parent_partition_init(void)
>> if (ret)
>> goto remove_cpu_state;
>>
>> + mshv_init_vmm_caps(dev);
>> +
>> ret = mshv_irqfd_wq_init();
>> if (ret)
>> goto exit_partition;
>> --
>> 2.34.1
>>
^ permalink raw reply
* Re: [PATCH v3 2/5] mshv: Add the HVCALL_GET_PARTITION_PROPERTY_EX hypercall
From: Nuno Das Neves @ 2025-09-19 22:30 UTC (permalink / raw)
To: Stanislav Kinsburskii
Cc: linux-hyperv, linux-kernel, prapal, easwar.hariharan, tiala,
anirudh, paekkaladevi, kys, haiyangz, wei.liu, decui
In-Reply-To: <aMxS7Wh67SuF4LV2@skinsburskii.localdomain>
On 9/18/2025 11:43 AM, Stanislav Kinsburskii wrote:
> On Tue, Sep 16, 2025 at 04:44:19PM -0700, Nuno Das Neves wrote:
>
> <snip>
>
>> diff --git a/include/hyperv/hvhdk.h b/include/hyperv/hvhdk.h
>> index b4067ada02cf..b91358b9c929 100644
>> --- a/include/hyperv/hvhdk.h
>> +++ b/include/hyperv/hvhdk.h
>> @@ -376,6 +376,46 @@ struct hv_input_set_partition_property {
>> u64 property_value;
>> } __packed;
>>
>> +union hv_partition_property_arg {
>> + u64 as_uint64;
>> + struct {
>> + union {
>> + u32 arg;
>> + u32 vp_index;
>> + };
>> + u16 reserved0;
>> + u8 reserved1;
>> + u8 object_type;
>> + };
>> +} __packed;
>
> Shouldn't the struct be "packed" instead?
>
Indeed, I'll fix it, thanks.
>> +
>> +struct hv_input_get_partition_property_ex {
>> + u64 partition_id;
>> + u32 property_code; /* enum hv_partition_property_code */
>> + u32 padding;
>> + union {
>> + union hv_partition_property_arg arg_data;
>> + u64 arg;
>> + };
>> +} __packed;
>> +
>> +/*
>> + * NOTE: Should use hv_input_set_partition_property_ex_header to compute this
>> + * size, but hv_input_get_partition_property_ex is identical so it suffices
>> + */
>> +#define HV_PARTITION_PROPERTY_EX_MAX_VAR_SIZE \
>> + (HV_HYP_PAGE_SIZE - sizeof(struct hv_input_get_partition_property_ex))
>> +
>> +union hv_partition_property_ex {
>> + u8 buffer[HV_PARTITION_PROPERTY_EX_MAX_VAR_SIZE];
>> + struct hv_partition_property_vmm_capabilities vmm_capabilities;
>> + /* More fields to be filled in when needed */
>> +} __packed;
>
> Packing a union is redundant.
>
> Thanks,
> Stanislav
^ permalink raw reply
* RE: [PATCH v1 5/6] x86/hyperv: Implement hypervisor ram collection into vmcore
From: Michael Kelley @ 2025-09-19 19:48 UTC (permalink / raw)
To: Mukesh R, linux-hyperv@vger.kernel.org,
linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org
Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org,
decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com,
bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org,
hpa@zytor.com, arnd@arndb.de
In-Reply-To: <58e4f6b3-6ae3-4bf4-3e1f-0981d6af91ea@linux.microsoft.com>
From: Mukesh R <mrathor@linux.microsoft.com> Sent: Thursday, September 18, 2025 7:32 PM
>
> On 9/18/25 16:53, Michael Kelley wrote:
> > From: Mukesh R <mrathor@linux.microsoft.com> Sent: Tuesday, September 16, 2025 6:13 PM
> >>
> >> On 9/15/25 10:55, Michael Kelley wrote:
> >>> From: Mukesh Rathor <mrathor@linux.microsoft.com> Sent: Tuesday, September 9, 2025 5:10 PM
[snip]
> >>>> +
> >>>> +/*
> >>>> + * Common function for all cpus before devirtualization.
> >>>> + *
> >>>> + * Hypervisor crash: all cpus get here in nmi context.
> >>>> + * Linux crash: the panicing cpu gets here at base level, all others in nmi
> >>>> + * context. Note, panicing cpu may not be the bsp.
> >>>> + *
> >>>> + * The function is not inlined so it will show on the stack. It is named so
> >>>> + * because the crash cmd looks for certain well known function names on the
> >>>> + * stack before looking into the cpu saved note in the elf section, and
> >>>> + * that work is currently incomplete.
> >>>> + *
> >>>> + * Notes:
> >>>> + * Hypervisor crash:
> >>>> + * - the hypervisor is in a very restrictive mode at this point and any
> >>>> + * vmexit it cannot handle would result in reboot. For example, console
> >>>> + * output from here would result in synic ipi hcall, which would result
> >>>> + * in reboot. So, no mumbo jumbo, just get to kexec as quickly as possible.
> >>>> + *
> >>>> + * Devirtualization is supported from the bsp only.
> >>>> + */
> >>>> +static noinline __noclone void crash_nmi_callback(struct pt_regs *regs)
> >>>> +{
> >>>> + struct hv_input_disable_hyp_ex *input;
> >>>> + u64 status;
> >>>> + int msecs = 1000, ccpu = smp_processor_id();
> >>>> +
> >>>> + if (ccpu == 0) {
> >>>> + /* crash_save_cpu() will be done in the kexec path */
> >>>> + cpu_emergency_stop_pt(); /* disable performance trace */
> >>>> + atomic_inc(&crash_cpus_wait);
> >>>> + } else {
> >>>> + crash_save_cpu(regs, ccpu);
> >>>> + cpu_emergency_stop_pt(); /* disable performance trace */
> >>>> + atomic_inc(&crash_cpus_wait);
> >>>> + for (;;); /* cause no vmexits */
> >>>> + }
> >>>> +
> >>>> + while (atomic_read(&crash_cpus_wait) < num_online_cpus() && msecs--)
> >>>> + mdelay(1);
> >>>> +
> >>>> + stop_nmi();
> >>>> + if (!hv_has_crashed)
> >>>> + hv_notify_prepare_hyp();
> >>>> +
> >>>> + if (crashing_cpu == -1)
> >>>> + crashing_cpu = ccpu; /* crash cmd uses this */
> >>>
> >>> Could just be "crashing_cpu = 0" since only the BSP gets here.
> >>
> >> a code change request has been open for while to remove the requirement
> >> of bsp..
> >>
> >>>> +
> >>>> + hv_hvcrash_ctxt_save();
> >>>> + hv_mark_tss_not_busy();
> >>>> + hv_crash_fixup_kernpt();
> >>>> +
> >>>> + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
> >>>> + memset(input, 0, sizeof(*input));
> >>>> + input->rip = trampoline_pa; /* PA of hv_crash_asm32 */
> >>>> + input->arg = devirt_cr3arg; /* PA of trampoline page table L4 */
> >>>
> >>> Is this comment correct? Isn't it the PA of struct hv_crash_tramp_data?
> >>> And just for clarification, Hyper-V treats this "arg" value as opaque and does
> >>> not access it. It only provides it in EDI when it invokes the trampoline
> >>> function, right?
> >>
> >> comment is correct. cr3 always points to l4 (or l5 if 5 level page tables).
> >
> > Yes, the comment matches the name of the "devirt_cr3arg" variable.
> > Unfortunately my previous comment was incomplete because the value
> > stored in the static variable "devirt_cr3arg" isn?t the address of an L4 page
> > table. It's not a CR3 value. The value stored in devirt_cr3arg is actually the
> > PA of struct hv_crash_tramp_data. The CR3 value is stored in the
> > tramp32_cr3 field (at offset 0) of that structure, so there's an additional level
> > of indirection. The (corrected) comment in the header to hv_crash_asm32()
> > describes EDI as containing "PA of struct hv_crash_tramp_data", which
> > ought to match what is described here. I'd say that "devirt_cr3arg" ought
> > to be renamed to "tramp_data_pa" or something else parallel to
> > "trampoline_pa".
>
> hyp needs trampoline cr3 for transition, we pass it as an arg. we piggy
> back extra information for ourselves needed in trampoline.S. so it's
> all good.
>
That's a pretty important "detail" that hasn't heretofore been mentioned.
It means the layout of struct hv_crash_tramp_data is not entirely at Linux's
discretion. The tramp32_cr3 field must be first so the hypervisor finds it
where it expects it. Please add code comments describing that the
hypervisor uses the tramp32_cr3 field.
With this new information, I agree the code works. But the devirt_cr3arg
variable is still named incorrectly, and the "PA of trampoline page table L4"
comment is still incorrect. The value in "devirt_cr3arg" is the PA of a memory
location in the trampoline page that contains the devirt CR3 (which itself is
the PA of trampoline page table L4). The CR3 value is in the tramp32_cr3 field
of struct hv_crash_tramp_data in the trampoline page. The CR3 value is
not in static variable devirt_cr3arg, which is why I object to the naming of that
variable.
So rename devirt_cr3arg to devirt_cr3arg_pa. And the comment
becomes "PA of PA of trampoline page table L4", which is rather unwieldy, so
could be shortened to "PA of devirt CR3 value" or something similar. You could
also use "PA of struct hv_crash_tramp_data" as the comment, as I suggested
previously.
Michael
^ permalink raw reply
* Re: [PATCH v1 4/6] x86/hyperv: Add trampoline asm code to transition from hypervisor
From: Mukesh R @ 2025-09-19 19:09 UTC (permalink / raw)
To: Borislav Petkov, Michael Kelley
Cc: linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-arch@vger.kernel.org, kys@microsoft.com,
haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com,
tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com,
x86@kernel.org, hpa@zytor.com, arnd@arndb.de
In-Reply-To: <20250919090625.GBaM0dEegelsB724bZ@fat_crate.local>
On 9/19/25 02:06, Borislav Petkov wrote:
> On Thu, Sep 18, 2025 at 11:52:35PM +0000, Michael Kelley wrote:
>> From: Mukesh R <mrathor@linux.microsoft.com> Sent: Tuesday, September 16, 2025 2:31 PM
>>>
>>> On 9/15/25 10:55, Michael Kelley wrote:
>>>> From: Mukesh Rathor <mrathor@linux.microsoft.com> Sent: Tuesday, September 9, 2025 5:10 PM
>>>>>
>>>>> Introduce a small asm stub to transition from the hypervisor to linux
>>>>
>>>> I'd argue for capitalizing "Linux" here and in other places in commit
>>>> text and code comments throughout this patch set.
>>>
>>> I'd argue against it. A quick grep indicates it is a common practice,
>>> and in the code world goes easy on the eyes :).
>
> But not in commit messages.
>
> Commit messages should be maximally readable and things should start in
> capital letters if that is their common spelling.
>
> When it comes to "Linux", yeah, that's so widespread so you have both. If I'm
> referring to what Linux does as a policy or in general or so on, I'd spell it
> capitalized but I don't think we've enforced that too strictly...
>
>> I'll offer a final comment on this topic, and then let it be. There's
>> a history of Greg K-H, Marc Zyngier, Boris Petkov, Sean Christopherson,
>> and other maintainers giving comments to use the capitalized form
>> of "Linux", "MSR", "RAM", etc. See:
>
> MSR, RAM and other abbreviations are capitalized and that's the only correct
> way to spell them.
>
>>>>> upon devirtualization.
>
> What is "devirtualization"?
Hypervisor is disabled, and it transfer control to the root/dom0
partition, so essentially hypervisor is gone when control comes back
to root/dom0 Linux.
>>> since control comes back to linux at the callback here, i fail to
>>> understand what is vague about it. when hyp completes devirt,
>>> devirt is complete.
>
> This "speak" is what gets on my nerves. You're writing here as if everyone is
> in your head and everyone knows what "hyp" and "devirt" is.
that's just follow up conversation, commit comment says "hypervisor" and
"devirtualization".
> Commit mesages are not code and they should be maximally readable and
> accessible to the widest audience, not only to the three people who develop
> the feature.
>
> If this patch were aimed at the things I maintain, it'll need a serious commit
> message scrubbing and sanitizing first.
>
> HTH.
>
^ permalink raw reply
* Re: [PATCH v1 4/6] x86/hyperv: Add trampoline asm code to transition from hypervisor
From: Borislav Petkov @ 2025-09-19 9:06 UTC (permalink / raw)
To: Michael Kelley, Mukesh R
Cc: linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-arch@vger.kernel.org, kys@microsoft.com,
haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com,
tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com,
x86@kernel.org, hpa@zytor.com, arnd@arndb.de
In-Reply-To: <SN6PR02MB4157CAE4FA74E482A96471B1D416A@SN6PR02MB4157.namprd02.prod.outlook.com>
On Thu, Sep 18, 2025 at 11:52:35PM +0000, Michael Kelley wrote:
> From: Mukesh R <mrathor@linux.microsoft.com> Sent: Tuesday, September 16, 2025 2:31 PM
> >
> > On 9/15/25 10:55, Michael Kelley wrote:
> > > From: Mukesh Rathor <mrathor@linux.microsoft.com> Sent: Tuesday, September 9, 2025 5:10 PM
> > >>
> > >> Introduce a small asm stub to transition from the hypervisor to linux
> > >
> > > I'd argue for capitalizing "Linux" here and in other places in commit
> > > text and code comments throughout this patch set.
> >
> > I'd argue against it. A quick grep indicates it is a common practice,
> > and in the code world goes easy on the eyes :).
But not in commit messages.
Commit messages should be maximally readable and things should start in
capital letters if that is their common spelling.
When it comes to "Linux", yeah, that's so widespread so you have both. If I'm
referring to what Linux does as a policy or in general or so on, I'd spell it
capitalized but I don't think we've enforced that too strictly...
> I'll offer a final comment on this topic, and then let it be. There's
> a history of Greg K-H, Marc Zyngier, Boris Petkov, Sean Christopherson,
> and other maintainers giving comments to use the capitalized form
> of "Linux", "MSR", "RAM", etc. See:
MSR, RAM and other abbreviations are capitalized and that's the only correct
way to spell them.
> > >> upon devirtualization.
What is "devirtualization"?
> > since control comes back to linux at the callback here, i fail to
> > understand what is vague about it. when hyp completes devirt,
> > devirt is complete.
This "speak" is what gets on my nerves. You're writing here as if everyone is
in your head and everyone knows what "hyp" and "devirt" is.
Commit mesages are not code and they should be maximally readable and
accessible to the widest audience, not only to the three people who develop
the feature.
If this patch were aimed at the things I maintain, it'll need a serious commit
message scrubbing and sanitizing first.
HTH.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
^ permalink raw reply
* Re: [PATCH v5 00/10] x86/hyperv/hv_vtl: Use a wakeup mailbox to boot secondary CPUs
From: Ricardo Neri @ 2025-09-19 2:59 UTC (permalink / raw)
To: x86, Krzysztof Kozlowski, Conor Dooley, Rob Herring,
K. Y. Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui,
Michael Kelley, Rafael J. Wysocki
Cc: Saurabh Sengar, Chris Oo, Kirill A. Shutemov, linux-hyperv,
devicetree, linux-acpi, linux-kernel, Ricardo Neri, Yunhong Jiang,
Thomas Gleixner
In-Reply-To: <20250820231135.GA24797@ranerica-svr.sc.intel.com>
On Wed, Aug 20, 2025 at 04:11:35PM -0700, Ricardo Neri wrote:
> On Fri, Jun 27, 2025 at 08:35:06PM -0700, Ricardo Neri wrote:
> > Hi,
> >
> > Here is a new version of this series. Thanks to Rafael for his feedback!
> > I incorporated his feedback in this updated version. Please see the
> > changelog for details.
> >
> > If the DeviceTree bindings look good, then the patches should be ready for
> > review by the x86, ACPI, and Hyper-V maintainers.
> >
> > I did not change the cover letter but I included it here for completeness.
> >
> > Thanks in advance for your feedback!
>
> Hello,
>
> I would like to know what else is needed to move this patchset forward.
> Rafael and Rob have reviewed the DeviceTree bindings. Rafael has reviewed
> the relocation of the code that makes use of the mailbox.
>
> Would it be possible for the Hyper-V maintainers to take a look (Michael
> Kelley has reviewed the patches already)? Perhaps this could increase the
> confidence of the x86 maintainers.
Many thanks Dexuan for reviewing this series. Now that the relevant
subsystem maintainers have reviewed their portions of the series, perhaps
the x86 maintainers could take a look?
Thanks in advance!
BR,
Ricardo
^ permalink raw reply
* Re: [EXTERNAL] [PATCH v5 10/10] x86/hyperv/vtl: Use the wakeup mailbox to boot secondary CPUs
From: Ricardo Neri @ 2025-09-19 2:47 UTC (permalink / raw)
To: Dexuan Cui
Cc: x86@kernel.org, Krzysztof Kozlowski, Conor Dooley, Rob Herring,
KY Srinivasan, Haiyang Zhang, Wei Liu, Michael Kelley,
Rafael J. Wysocki, ssengar@linux.microsoft.com, Chris Oo,
Kirill A. Shutemov, linux-hyperv@vger.kernel.org,
devicetree@vger.kernel.org, linux-acpi@vger.kernel.org,
linux-kernel@vger.kernel.org, Ricardo Neri
In-Reply-To: <DS3PR21MB5878BD23A845865D898E3C4DBF08A@DS3PR21MB5878.namprd21.prod.outlook.com>
On Fri, Sep 12, 2025 at 08:47:52PM +0000, Dexuan Cui wrote:
> > From: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
> > Sent: Friday, June 27, 2025 8:35 PM
> > [...]
> > The hypervisor is an untrusted entity for TDX guests. It cannot be used
> > to boot secondary CPUs. The function hv_vtl_wakeup_secondary_cpu() cannot
> > be used.
> >
> > Instead, the virtual firmware boots the secondary CPUs and places them in
> > a state to transfer control to the kernel using the wakeup mailbox.
> >
> > The kernel updates the APIC callback wakeup_secondary_cpu_64() to use
> > the mailbox if detected early during boot (enumerated via either an ACPI
> > table or a DeviceTree node).
> >
> > Reviewed-by: Michael Kelley <mhklinux@outlook.com>
> > Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
> > ---
>
> LGTM
>
> Reviewed-by: Dexuan Cui <decui@microsoft.com>
Thank you for your review!
^ permalink raw reply
* Re: [EXTERNAL] [PATCH v5 05/10] x86/hyperv/vtl: Set real_mode_header in hv_vtl_init_platform()
From: Ricardo Neri @ 2025-09-19 2:46 UTC (permalink / raw)
To: Dexuan Cui
Cc: x86@kernel.org, Krzysztof Kozlowski, Conor Dooley, Rob Herring,
KY Srinivasan, Haiyang Zhang, Wei Liu, Michael Kelley,
Rafael J. Wysocki, ssengar@linux.microsoft.com, Chris Oo,
Kirill A. Shutemov, linux-hyperv@vger.kernel.org,
devicetree@vger.kernel.org, linux-acpi@vger.kernel.org,
linux-kernel@vger.kernel.org, Ricardo Neri, Yunhong Jiang,
Thomas Gleixner
In-Reply-To: <DS3PR21MB58781CFA8BEFF22D9BA407B1BF08A@DS3PR21MB5878.namprd21.prod.outlook.com>
On Fri, Sep 12, 2025 at 08:43:47PM +0000, Dexuan Cui wrote:
> > From: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
> > Sent: Friday, June 27, 2025 8:35 PM
> >[...]
> > From: Yunhong Jiang <yunhong.jiang@linux.intel.com>
> >
> > Hyper-V VTL clears x86_platform.realmode_{init(), reserve()} in
> > hv_vtl_platform_init() whereas it sets real_mode_header later in
>
> s/hv_vtl_platform_init/hv_vtl_init_platform/
Thanks for catching this. I will correct it.
>
> otherwise LGTM
>
> Reviewed-by: Dexuan Cui <decui@microsoft.com>
Thank you!
^ permalink raw reply
* Re: [PATCH v1 5/6] x86/hyperv: Implement hypervisor ram collection into vmcore
From: Mukesh R @ 2025-09-19 2:32 UTC (permalink / raw)
To: Michael Kelley, linux-hyperv@vger.kernel.org,
linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org
Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org,
decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com,
bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org,
hpa@zytor.com, arnd@arndb.de
In-Reply-To: <SN6PR02MB4157AFD23F088FC243A24C17D416A@SN6PR02MB4157.namprd02.prod.outlook.com>
On 9/18/25 16:53, Michael Kelley wrote:
> From: Mukesh R <mrathor@linux.microsoft.com> Sent: Tuesday, September 16, 2025 6:13 PM
>>
>> On 9/15/25 10:55, Michael Kelley wrote:
>>> From: Mukesh Rathor <mrathor@linux.microsoft.com> Sent: Tuesday, September 9, 2025 5:10 PM
>>>>
>>>> Introduce a new file to implement collection of hypervisor ram into the
>>>
>>> s/ram/RAM/ (multiple places)
>>
>> a quick grep indicates using saying ram is common, i like ram over RAM
>>
>>>> vmcore collected by linux. By default, the hypervisor ram is locked, ie,
>>>> protected via hw page table. Hyper-V implements a disable hypercall which
>>>
>>> The terminology here is a bit confusing since you have two names for
>>> the same thing: "disable" hypervisor, and "devirtualize". Is it possible to
>>> just use "devirtualize" everywhere, and drop the "disable" terminology?
>>
>> The concept is devirtualize and the actual hypercall was originally named
>> disable. so intermixing is natural imo.
>>
>>>> essentially devirtualizes the system on the fly. This mechanism makes the
>>>> hypervisor ram accessible to linux. Because the hypervisor ram is already
>>>> mapped into linux address space (as reserved ram),
>>>
>>> Is the hypervisor RAM mapped into the VMM process user address space,
>>> or somewhere in the kernel address space? If the latter, where in the kernel
>>> code, or what mechanism, does that? Just curious, as I wasn't aware that
>>> this is happening ....
>>
>> mapped in kernel as normal ram and we reserve it very early in boot. i
>> see that patch has not made it here yet, should be coming very soon.
>
> OK, that's fine. The answer to my question is coming soon ....
>
>>
>>>> it is automatically
>>>> collected into the vmcore without extra work. More details of the
>>>> implementation are available in the file prologue.
>>>>
>>>> Signed-off-by: Mukesh Rathor <mrathor@linux.microsoft.com>
>>>> ---
>>>> arch/x86/hyperv/hv_crash.c | 622 +++++++++++++++++++++++++++++++++++++
>>>> 1 file changed, 622 insertions(+)
>>>> create mode 100644 arch/x86/hyperv/hv_crash.c
>>>>
>>>> diff --git a/arch/x86/hyperv/hv_crash.c b/arch/x86/hyperv/hv_crash.c
>>>> new file mode 100644
>>>> index 000000000000..531bac79d598
>>>> --- /dev/null
>>>> +++ b/arch/x86/hyperv/hv_crash.c
>>>> @@ -0,0 +1,622 @@
>>>> +// SPDX-License-Identifier: GPL-2.0-only
>>>> +/*
>>>> + * X86 specific Hyper-V kdump/crash support module
>>>> + *
>>>> + * Copyright (C) 2025, Microsoft, Inc.
>>>> + *
>>>> + * This module implements hypervisor ram collection into vmcore for both
>>>> + * cases of the hypervisor crash and linux dom0/root crash.
>>>
>>> For a hypervisor crash, does any of this apply to general guest VMs? I'm
>>> thinking it does not. Hypervisor RAM is collected only into the vmcore
>>> for the root partition, right? Maybe some additional clarification could be
>>> added so there's no confusion in this regard.
>>
>> it would be odd for guests to collect hyp core, and target audience is
>> assumed to be those who are somewhat familiar with basic concepts before
>> getting here.
>
> I was unsure because I had not seen any code that adds the hypervisor memory
> to the Linux memory map. Thought maybe something was going on I hadn?t
> heard about, so I didn't know the scope of it.
>
> Of course, I'm one of those people who was *not* familiar with the basic concepts
> before getting here. And given that there's no spec available from Hyper-V,
> the comments in this patch set are all there is for anyone outside of Microsoft.
> In that vein, I think it's reasonable to provide some description of how this
> all works in the code comments. And you've done that, which is very
> helpful. But I encountered a few places where I was confused or unclear, and
> my suggestions here and in Patch 4 are just about making things as precise as
> possible without adding a huge amount of additional verbiage. For someone
> new, English text descriptions that the code can be checked against are
> helpful, and drawing hard boundaries ("this is only applicable to the root
> partition") is helpful.
>
> If you don't want to deal with it now, I could provide a follow-on patch later
> that tweaks or augments the wording a bit to clarify some of these places.
> You can review, like with any patch. I've done wording work over the years
> to many places in the VMBus code, and more broadly in providing most of
> the documentation in Documentation/virt/hyperv.
with time, things will start making sense... i find comment pretty clear
that it collects core for both cases of hv crash and dom0 crash, and no
mention of guest implies has nothing to do with guests.
>>
>>> And what *does* happen to guest VMs after a hypervisor crash?
>>
>> they are gone... what else could we do?
>>
>>>> + * Hyper-V implements
>>>> + * a devirtualization hypercall with a 32bit protected mode ABI callback. This
>>>> + * mechanism must be used to unlock hypervisor ram. Since the hypervisor ram
>>>> + * is already mapped in linux, it is automatically collected into linux vmcore,
>>>> + * and can be examined by the crash command (raw ram dump) or windbg.
>>>> + *
>>>> + * At a high level:
>>>> + *
>>>> + * Hypervisor Crash:
>>>> + * Upon crash, hypervisor goes into an emergency minimal dispatch loop, a
>>>> + * restrictive mode with very limited hypercall and msr support.
>>>
>>> s/msr/MSR/
>>
>> msr is used all over, seems acceptable.
>>
>>>> + * Each cpu then injects NMIs into dom0/root vcpus.
>>>
>>> The "Each cpu" part of this sentence is confusing to me -- which CPUs does
>>> this refer to? Maybe it would be better to say "It then injects an NMI into
>>> each dom0/root partition vCPU." without being specific as to which CPUs do
>>> the injecting since that seems more like a hypervisor implementation detail
>>> that's not relevant here.
>>
>> all cpus in the system. there is a dedicated/pinned dom0 vcpu for each cpu.
>
> OK, that makes sense now that I think about it. Each physical CPU in the host
> has a corresponding vCPU in the dom0/root partition. And each of the vCPUs
> gets an NMI that sends it to the Linux-in-dom0 NMI handler, even if it was off
> running a vCPU in some guest VM.
>
>>
>>>> + * A shared page is used to check
>>>> + * by linux in the nmi handler if the hypervisor has crashed. This shared
>>>
>>> s/nmi/NMI/ (multiple places)
>>
>>>> + * page is setup in hv_root_crash_init during boot.
>>>> + *
>>>> + * Linux Crash:
>>>> + * In case of linux crash, the callback hv_crash_stop_other_cpus will send
>>>> + * NMIs to all cpus, then proceed to the crash_nmi_callback where it waits
>>>> + * for all cpus to be in NMI.
>>>> + *
>>>> + * NMI Handler (upon quorum):
>>>> + * Eventually, in both cases, all cpus wil end up in the nmi hanlder.
>>>
>>> s/hanlder/handler/
>>>
>>> And maybe just drop the word "wil" (which is misspelled).
>>>
>>>> + * Hyper-V requires the disable hypervisor must be done from the bsp. So
>>>
>>> s/bsp/BSP (multiple places)
>>>
>>>> + * the bsp nmi handler saves current context, does some fixups and makes
>>>> + * the hypercall to disable the hypervisor, ie, devirtualize. Hypervisor
>>>> + * at that point will suspend all vcpus (except the bsp), unlock all its
>>>> + * ram, and return to linux at the 32bit mode entry RIP.
>>>> + *
>>>> + * Linux 32bit entry trampoline will then restore long mode and call C
>>>> + * function here to restore context and continue execution to crash kexec.
>>>> + */
>>>> +
>>>> +#include <linux/delay.h>
>>>> +#include <linux/kexec.h>
>>>> +#include <linux/crash_dump.h>
>>>> +#include <linux/panic.h>
>>>> +#include <asm/apic.h>
>>>> +#include <asm/desc.h>
>>>> +#include <asm/page.h>
>>>> +#include <asm/pgalloc.h>
>>>> +#include <asm/mshyperv.h>
>>>> +#include <asm/nmi.h>
>>>> +#include <asm/idtentry.h>
>>>> +#include <asm/reboot.h>
>>>> +#include <asm/intel_pt.h>
>>>> +
>>>> +int hv_crash_enabled;
>>>
>>> Seems like this is conceptually a "bool", not an "int".
>>
>> yeah, can change it to bool if i do another iteration.
>>
>>>> +EXPORT_SYMBOL_GPL(hv_crash_enabled);
>>>> +
>>>> +struct hv_crash_ctxt {
>>>> + ulong rsp;
>>>> + ulong cr0;
>>>> + ulong cr2;
>>>> + ulong cr4;
>>>> + ulong cr8;
>>>> +
>>>> + u16 cs;
>>>> + u16 ss;
>>>> + u16 ds;
>>>> + u16 es;
>>>> + u16 fs;
>>>> + u16 gs;
>>>> +
>>>> + u16 gdt_fill;
>>>> + struct desc_ptr gdtr;
>>>> + char idt_fill[6];
>>>> + struct desc_ptr idtr;
>>>> +
>>>> + u64 gsbase;
>>>> + u64 efer;
>>>> + u64 pat;
>>>> +};
>>>> +static struct hv_crash_ctxt hv_crash_ctxt;
>>>> +
>>>> +/* Shared hypervisor page that contains crash dump area we peek into.
>>>> + * NB: windbg looks for "hv_cda" symbol so don't change it.
>>>> + */
>>>> +static struct hv_crashdump_area *hv_cda;
>>>> +
>>>> +static u32 trampoline_pa, devirt_cr3arg;
>>>> +static atomic_t crash_cpus_wait;
>>>> +static void *hv_crash_ptpgs[4];
>>>> +static int hv_has_crashed, lx_has_crashed;
>>>
>>> These are conceptually "bool" as well.
>>>
>>>> +
>>>> +/* This cannot be inlined as it needs stack */
>>>> +static noinline __noclone void hv_crash_restore_tss(void)
>>>> +{
>>>> + load_TR_desc();
>>>> +}
>>>> +
>>>> +/* This cannot be inlined as it needs stack */
>>>> +static noinline void hv_crash_clear_kernpt(void)
>>>> +{
>>>> + pgd_t *pgd;
>>>> + p4d_t *p4d;
>>>> +
>>>> + /* Clear entry so it's not confusing to someone looking at the core */
>>>> + pgd = pgd_offset_k(trampoline_pa);
>>>> + p4d = p4d_offset(pgd, trampoline_pa);
>>>> + native_p4d_clear(p4d);
>>>> +}
>>>> +
>>>> +/*
>>>> + * This is the C entry point from the asm glue code after the devirt hypercall.
>>>> + * We enter here in IA32-e long mode, ie, full 64bit mode running on kernel
>>>> + * page tables with our below 4G page identity mapped, but using a temporary
>>>> + * GDT. ds/fs/gs/es are null. ss is not usable. bp is null. stack is not
>>>> + * available. We restore kernel GDT, and rest of the context, and continue
>>>> + * to kexec.
>>>> + */
>>>> +static asmlinkage void __noreturn hv_crash_c_entry(void)
>>>> +{
>>>> + struct hv_crash_ctxt *ctxt = &hv_crash_ctxt;
>>>> +
>>>> + /* first thing, restore kernel gdt */
>>>> + native_load_gdt(&ctxt->gdtr);
>>>> +
>>>> + asm volatile("movw %%ax, %%ss" : : "a"(ctxt->ss));
>>>> + asm volatile("movq %0, %%rsp" : : "m"(ctxt->rsp));
>>>> +
>>>> + asm volatile("movw %%ax, %%ds" : : "a"(ctxt->ds));
>>>> + asm volatile("movw %%ax, %%es" : : "a"(ctxt->es));
>>>> + asm volatile("movw %%ax, %%fs" : : "a"(ctxt->fs));
>>>> + asm volatile("movw %%ax, %%gs" : : "a"(ctxt->gs));
>>>> +
>>>> + native_wrmsrq(MSR_IA32_CR_PAT, ctxt->pat);
>>>> + asm volatile("movq %0, %%cr0" : : "r"(ctxt->cr0));
>>>> +
>>>> + asm volatile("movq %0, %%cr8" : : "r"(ctxt->cr8));
>>>> + asm volatile("movq %0, %%cr4" : : "r"(ctxt->cr4));
>>>> + asm volatile("movq %0, %%cr2" : : "r"(ctxt->cr4));
>>>> +
>>>> + native_load_idt(&ctxt->idtr);
>>>> + native_wrmsrq(MSR_GS_BASE, ctxt->gsbase);
>>>> + native_wrmsrq(MSR_EFER, ctxt->efer);
>>>> +
>>>> + /* restore the original kernel CS now via far return */
>>>> + asm volatile("movzwq %0, %%rax\n\t"
>>>> + "pushq %%rax\n\t"
>>>> + "pushq $1f\n\t"
>>>> + "lretq\n\t"
>>>> + "1:nop\n\t" : : "m"(ctxt->cs) : "rax");
>>>> +
>>>> + /* We are in asmlinkage without stack frame, hence make a C function
>>>> + * call which will buy stack frame to restore the tss or clear PT entry.
>>>> + */
>>>> + hv_crash_restore_tss();
>>>> + hv_crash_clear_kernpt();
>>>> +
>>>> + /* we are now fully in devirtualized normal kernel mode */
>>>> + __crash_kexec(NULL);
>>>
>>> The comments for __crash_kexec() say that "panic_cpu" should be set to
>>> the current CPU. I don't see that such is the case here.
>>
>> if linux panic, it would be set by vpanic, if hyp crash, that is
>> irrelevant.
>>
>>>> +
>>>> + for (;;)
>>>> + cpu_relax();
>>>
>>> Is the intent that __crash_kexec() should never return, on any of the vCPUs,
>>> because devirtualization isn't done unless there's a valid kdump image loaded?
>>> I wonder if
>>>
>>> native_wrmsrq(HV_X64_MSR_RESET, 1);
>>>
>>> would be better than looping forever in case __crash_kexec() fails
>>> somewhere along the way even if there's a kdump image loaded.
>>
>> yeah, i've gone thru all 3 possibilities here:
>> o loop forever
>> o reset
>> o BUG() : this was in V0
>>
>> reset is just bad because system would just reboot without any indication
>> if hyp crashes. with loop at least there is a hang, and one could make
>> note of it, and if internal, attach debugger.
>>
>> BUG is best imo because with hyp gone linux will try to redo panic
>> and we would print something extra to help. I think i'll just go
>> back to my V0: BUG()
>>
>>>> +}
>>>> +/* Tell gcc we are using lretq long jump in the above function intentionally */
>>>> +STACK_FRAME_NON_STANDARD(hv_crash_c_entry);
>>>> +
>>>> +static void hv_mark_tss_not_busy(void)
>>>> +{
>>>> + struct desc_struct *desc = get_current_gdt_rw();
>>>> + tss_desc tss;
>>>> +
>>>> + memcpy(&tss, &desc[GDT_ENTRY_TSS], sizeof(tss_desc));
>>>> + tss.type = 0x9; /* available 64-bit TSS. 0xB is busy TSS */
>>>> + write_gdt_entry(desc, GDT_ENTRY_TSS, &tss, DESC_TSS);
>>>> +}
>>>> +
>>>> +/* Save essential context */
>>>> +static void hv_hvcrash_ctxt_save(void)
>>>> +{
>>>> + struct hv_crash_ctxt *ctxt = &hv_crash_ctxt;
>>>> +
>>>> + asm volatile("movq %%rsp,%0" : "=m"(ctxt->rsp));
>>>> +
>>>> + ctxt->cr0 = native_read_cr0();
>>>> + ctxt->cr4 = native_read_cr4();
>>>> +
>>>> + asm volatile("movq %%cr2, %0" : "=a"(ctxt->cr2));
>>>> + asm volatile("movq %%cr8, %0" : "=a"(ctxt->cr8));
>>>> +
>>>> + asm volatile("movl %%cs, %%eax" : "=a"(ctxt->cs));
>>>> + asm volatile("movl %%ss, %%eax" : "=a"(ctxt->ss));
>>>> + asm volatile("movl %%ds, %%eax" : "=a"(ctxt->ds));
>>>> + asm volatile("movl %%es, %%eax" : "=a"(ctxt->es));
>>>> + asm volatile("movl %%fs, %%eax" : "=a"(ctxt->fs));
>>>> + asm volatile("movl %%gs, %%eax" : "=a"(ctxt->gs));
>>>> +
>>>> + native_store_gdt(&ctxt->gdtr);
>>>> + store_idt(&ctxt->idtr);
>>>> +
>>>> + ctxt->gsbase = __rdmsr(MSR_GS_BASE);
>>>> + ctxt->efer = __rdmsr(MSR_EFER);
>>>> + ctxt->pat = __rdmsr(MSR_IA32_CR_PAT);
>>>> +}
>>>> +
>>>> +/* Add trampoline page to the kernel pagetable for transition to kernel PT */
>>>> +static void hv_crash_fixup_kernpt(void)
>>>> +{
>>>> + pgd_t *pgd;
>>>> + p4d_t *p4d;
>>>> +
>>>> + pgd = pgd_offset_k(trampoline_pa);
>>>> + p4d = p4d_offset(pgd, trampoline_pa);
>>>> +
>>>> + /* trampoline_pa is below 4G, so no pre-existing entry to clobber */
>>>> + p4d_populate(&init_mm, p4d, (pud_t *)hv_crash_ptpgs[1]);
>>>> + p4d->p4d = p4d->p4d & ~(_PAGE_NX); /* enable execute */
>>>> +}
>>>> +
>>>> +/*
>>>> + * Now that all cpus are in nmi and spinning, we notify the hyp that linux has
>>>> + * crashed and will collect core. This will cause the hyp to quiesce and
>>>> + * suspend all VPs except the bsp. Called if linux crashed and not the hyp.
>>>> + */
>>>> +static void hv_notify_prepare_hyp(void)
>>>> +{
>>>> + u64 status;
>>>> + struct hv_input_notify_partition_event *input;
>>>> + struct hv_partition_event_root_crashdump_input *cda;
>>>> +
>>>> + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
>>>> + cda = &input->input.crashdump_input;
>>>
>>> The code ordering here is a bit weird. I'd expect this line to be grouped
>>> with cda->crashdump_action being set.
>>
>> we are setting two pointers, and using them later. setting pointers
>> up front is pretty normal.
>>
>>>> + memset(input, 0, sizeof(*input));
>>>> + input->event = HV_PARTITION_EVENT_ROOT_CRASHDUMP;
>>>> +
>>>> + cda->crashdump_action = HV_CRASHDUMP_ENTRY;
>>>> + status = hv_do_hypercall(HVCALL_NOTIFY_PARTITION_EVENT, input, NULL);
>>>> + if (!hv_result_success(status))
>>>> + return;
>>>> +
>>>> + cda->crashdump_action = HV_CRASHDUMP_SUSPEND_ALL_VPS;
>>>> + hv_do_hypercall(HVCALL_NOTIFY_PARTITION_EVENT, input, NULL);
>>>> +}
>>>> +
>>>> +/*
>>>> + * Common function for all cpus before devirtualization.
>>>> + *
>>>> + * Hypervisor crash: all cpus get here in nmi context.
>>>> + * Linux crash: the panicing cpu gets here at base level, all others in nmi
>>>> + * context. Note, panicing cpu may not be the bsp.
>>>> + *
>>>> + * The function is not inlined so it will show on the stack. It is named so
>>>> + * because the crash cmd looks for certain well known function names on the
>>>> + * stack before looking into the cpu saved note in the elf section, and
>>>> + * that work is currently incomplete.
>>>> + *
>>>> + * Notes:
>>>> + * Hypervisor crash:
>>>> + * - the hypervisor is in a very restrictive mode at this point and any
>>>> + * vmexit it cannot handle would result in reboot. For example, console
>>>> + * output from here would result in synic ipi hcall, which would result
>>>> + * in reboot. So, no mumbo jumbo, just get to kexec as quickly as possible.
>>>> + *
>>>> + * Devirtualization is supported from the bsp only.
>>>> + */
>>>> +static noinline __noclone void crash_nmi_callback(struct pt_regs *regs)
>>>> +{
>>>> + struct hv_input_disable_hyp_ex *input;
>>>> + u64 status;
>>>> + int msecs = 1000, ccpu = smp_processor_id();
>>>> +
>>>> + if (ccpu == 0) {
>>>> + /* crash_save_cpu() will be done in the kexec path */
>>>> + cpu_emergency_stop_pt(); /* disable performance trace */
>>>> + atomic_inc(&crash_cpus_wait);
>>>> + } else {
>>>> + crash_save_cpu(regs, ccpu);
>>>> + cpu_emergency_stop_pt(); /* disable performance trace */
>>>> + atomic_inc(&crash_cpus_wait);
>>>> + for (;;); /* cause no vmexits */
>>>> + }
>>>> +
>>>> + while (atomic_read(&crash_cpus_wait) < num_online_cpus() && msecs--)
>>>> + mdelay(1);
>>>> +
>>>> + stop_nmi();
>>>> + if (!hv_has_crashed)
>>>> + hv_notify_prepare_hyp();
>>>> +
>>>> + if (crashing_cpu == -1)
>>>> + crashing_cpu = ccpu; /* crash cmd uses this */
>>>
>>> Could just be "crashing_cpu = 0" since only the BSP gets here.
>>
>> a code change request has been open for while to remove the requirement
>> of bsp..
>>
>>>> +
>>>> + hv_hvcrash_ctxt_save();
>>>> + hv_mark_tss_not_busy();
>>>> + hv_crash_fixup_kernpt();
>>>> +
>>>> + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
>>>> + memset(input, 0, sizeof(*input));
>>>> + input->rip = trampoline_pa; /* PA of hv_crash_asm32 */
>>>> + input->arg = devirt_cr3arg; /* PA of trampoline page table L4 */
>>>
>>> Is this comment correct? Isn't it the PA of struct hv_crash_tramp_data?
>>> And just for clarification, Hyper-V treats this "arg" value as opaque and does
>>> not access it. It only provides it in EDI when it invokes the trampoline
>>> function, right?
>>
>> comment is correct. cr3 always points to l4 (or l5 if 5 level page tables).
>
> Yes, the comment matches the name of the "devirt_cr3arg" variable.
> Unfortunately my previous comment was incomplete because the value
> stored in the static variable "devirt_cr3arg" isn?t the address of an L4 page
> table. It's not a CR3 value. The value stored in devirt_cr3arg is actually the
> PA of struct hv_crash_tramp_data. The CR3 value is stored in the
> tramp32_cr3 field (at offset 0) of that structure, so there's an additional level
> of indirection. The (corrected) comment in the header to hv_crash_asm32()
> describes EDI as containing "PA of struct hv_crash_tramp_data", which
> ought to match what is described here. I'd say that "devirt_cr3arg" ought
> to be renamed to "tramp_data_pa" or something else parallel to
> "trampoline_pa".
hyp needs trampoline cr3 for transition, we pass it as an arg. we piggy
back extra information for ourselves needed in trampoline.S. so it's
all good.
>>
>> right, comes in edi, i don't know what EDI is (just kidding!)...
>>
>>>> +
>>>> + status = hv_do_hypercall(HVCALL_DISABLE_HYP_EX, input, NULL);
>>>> +
>>>> + /* Devirt failed, just reboot as things are in very bad state now */
>>>> + native_wrmsrq(HV_X64_MSR_RESET, 1); /* get hv to reboot */
>>>> +}
>>>> +
>>>> +/*
>>>> + * Generic nmi callback handler: could be called without any crash also.
>>>> + * hv crash: hypervisor injects nmi's into all cpus
>>>> + * lx crash: panicing cpu sends nmi to all but self via crash_stop_other_cpus
>>>> + */
>>>> +static int hv_crash_nmi_local(unsigned int cmd, struct pt_regs *regs)
>>>> +{
>>>> + int ccpu = smp_processor_id();
>>>> +
>>>> + if (!hv_has_crashed && hv_cda && hv_cda->cda_valid)
>>>> + hv_has_crashed = 1;
>>>> +
>>>> + if (!hv_has_crashed && !lx_has_crashed)
>>>> + return NMI_DONE; /* ignore the nmi */
>>>> +
>>>> + if (hv_has_crashed) {
>>>> + if (!kexec_crash_loaded() || !hv_crash_enabled) {
>>>> + if (ccpu == 0) {
>>>> + native_wrmsrq(HV_X64_MSR_RESET, 1); /* reboot */
>>>> + } else
>>>> + for (;;); /* cause no vmexits */
>>>> + }
>>>> + }
>>>> +
>>>> + crash_nmi_callback(regs);
>>>> +
>>>> + return NMI_DONE;
>>>
>>> crash_nmi_callback() should never return, right? Normally one would
>>> expect to return NMI_HANDLED here, but I guess it doesn't matter
>>> if the return is never executed.
>>
>> correct.
>>
>>>> +}
>>>> +
>>>> +/*
>>>> + * hv_crash_stop_other_cpus() == smp_ops.crash_stop_other_cpus
>>>> + *
>>>> + * On normal linux panic, this is called twice: first from panic and then again
>>>> + * from native_machine_crash_shutdown.
>>>> + *
>>>> + * In case of mshv, 3 ways to get here:
>>>> + * 1. hv crash (only bsp will get here):
>>>> + * BSP : nmi callback -> DisableHv -> hv_crash_asm32 -> hv_crash_c_entry
>>>> + * -> __crash_kexec -> native_machine_crash_shutdown
>>>> + * -> crash_smp_send_stop -> smp_ops.crash_stop_other_cpus
>>>> + * linux panic:
>>>> + * 2. panic cpu x: panic() -> crash_smp_send_stop
>>>> + * -> smp_ops.crash_stop_other_cpus
>>>> + * 3. bsp: native_machine_crash_shutdown -> crash_smp_send_stop
>>>> + *
>>>> + * NB: noclone and non standard stack because of call to crash_setup_regs().
>>>> + */
>>>> +static void __noclone hv_crash_stop_other_cpus(void)
>>>> +{
>>>> + static int crash_stop_done;
>>>> + struct pt_regs lregs;
>>>> + int ccpu = smp_processor_id();
>>>> +
>>>> + if (hv_has_crashed)
>>>> + return; /* all cpus already in nmi handler path */
>>>> +
>>>> + if (!kexec_crash_loaded())
>>>> + return;
>>>
>>> If we're in a normal panic path (your Case #2 above) with no kdump kernel
>>> loaded, why leave the other vCPUs running? Seems like that could violate
>>> expectations in vpanic(), where it calls panic_other_cpus_shutdown() and
>>> thereafter assumes other vCPUs are not running.
>>
>> no, there is lots of complexity here!
>>
>> if we hang vcpus here, hyp will note and may trigger its own watchdog.
>> also, machine_crash_shutdown() does another ipi.
>>
>> I think the best thing to do here is go back to my V0 which did not
>> have check for kexec_crash_loaded(), but had this in hv_crash_c_entry:
>>
>> + /* we are now fully in devirtualized normal kernel mode */
>> + __crash_kexec(NULL);
>> +
>> + BUG();
>>
>>
>> this way hyp would be disabled, ie, system devirtualized, and
>> __crash_kernel() will return, resulting in BUG() that will cause
>> it to go thru panic and honor panic= parameter with either hang
>> or reset. instead of bug, i could just call panic() also.
>>
>>>> +
>>>> + if (crash_stop_done)
>>>> + return;
>>>> + crash_stop_done = 1;
>>>
>>> Is crash_stop_done necessary? hv_crash_stop_other_cpus() is called
>>> from crash_smp_send_stop(), which has its own static variable
>>> "cpus_stopped" that does the same thing.
>>
>> yes. for error paths.
>>
>>>> +
>>>> + /* linux has crashed: hv is healthy, we can ipi safely */
>>>> + lx_has_crashed = 1;
>>>> + wmb(); /* nmi handlers look at lx_has_crashed */
>>>> +
>>>> + apic->send_IPI_allbutself(NMI_VECTOR);
>>>
>>> The default .crash_stop_other_cpus function is kdump_nmi_shootdown_cpus().
>>> In addition to sending the NMI IPI, it does disable_local_APIC(). I don't know, but
>>> should disable_local_APIC() be done somewhere here as well?
>>
>> no, hyp does that.
>
> As part of the devirt operation initiated by the HVCALL_DISABLE_HYP_EX
> hypercall in crash_nmi_callback()? This gets back to an earlier question/comment
> where I was trying to figure out if the APIC is still enabled, and in what mode,
> when hv_crash_asm32() is invoked.
>>
>>>> +
>>>> + if (crashing_cpu == -1)
>>>> + crashing_cpu = ccpu; /* crash cmd uses this */
>>>> +
>>>> + /* crash_setup_regs() happens in kexec also, but for the kexec cpu which
>>>> + * is the bsp. We could be here on non-bsp cpu, collect regs if so.
>>>> + */
>>>> + if (ccpu)
>>>> + crash_setup_regs(&lregs, NULL);
>>>> +
>>>> + crash_nmi_callback(&lregs);
>>>> +}
>>>> +STACK_FRAME_NON_STANDARD(hv_crash_stop_other_cpus);
>>>> +
>>>> +/* This GDT is accessed in IA32-e compat mode which uses 32bits addresses */
>>>> +struct hv_gdtreg_32 {
>>>> + u16 fill;
>>>> + u16 limit;
>>>> + u32 address;
>>>> +} __packed;
>>>> +
>>>> +/* We need a CS with L bit to goto IA32-e long mode from 32bit compat mode */
>>>> +struct hv_crash_tramp_gdt {
>>>> + u64 null; /* index 0, selector 0, null selector */
>>>> + u64 cs64; /* index 1, selector 8, cs64 selector */
>>>> +} __packed;
>>>> +
>>>> +/* No stack, so jump via far ptr in memory to load the 64bit CS */
>>>> +struct hv_cs_jmptgt {
>>>> + u32 address;
>>>> + u16 csval;
>>>> + u16 fill;
>>>> +} __packed;
>>>> +
>>>> +/* This trampoline data is copied onto the trampoline page after the asm code */
>>>> +struct hv_crash_tramp_data {
>>>> + u64 tramp32_cr3;
>>>> + u64 kernel_cr3;
>>>> + struct hv_gdtreg_32 gdtr32;
>>>> + struct hv_crash_tramp_gdt tramp_gdt;
>>>> + struct hv_cs_jmptgt cs_jmptgt;
>>>> + u64 c_entry_addr;
>>>> +} __packed;
>>>> +
>>>> +/*
>>>> + * Setup a temporary gdt to allow the asm code to switch to the long mode.
>>>> + * Since the asm code is relocated/copied to a below 4G page, it cannot use rip
>>>> + * relative addressing, hence we must use trampoline_pa here. Also, save other
>>>> + * info like jmp and C entry targets for same reasons.
>>>> + *
>>>> + * Returns: 0 on success, -1 on error
>>>> + */
>>>> +static int hv_crash_setup_trampdata(u64 trampoline_va)
>>>> +{
>>>> + int size, offs;
>>>> + void *dest;
>>>> + struct hv_crash_tramp_data *tramp;
>>>> +
>>>> + /* These must match exactly the ones in the corresponding asm file */
>>>> + BUILD_BUG_ON(offsetof(struct hv_crash_tramp_data, tramp32_cr3) != 0);
>>>> + BUILD_BUG_ON(offsetof(struct hv_crash_tramp_data, kernel_cr3) != 8);
>>>> + BUILD_BUG_ON(offsetof(struct hv_crash_tramp_data, gdtr32.limit) != 18);
>>>> + BUILD_BUG_ON(offsetof(struct hv_crash_tramp_data,
>>>> + cs_jmptgt.address) != 40);
>>>
>>> It would be nice to pick up the constants from a #include file that is
>>> shared with the asm code in Patch 4 of the series.
>>
>> yeah, could go either way, some don't like tiny headers... if there are
>> no objections to new header for this, i could go that way too.
>
> Saw your follow-on comments about this as well. The tiny header
> is ugly. It's a judgment call that can go either way, so go with your
> preference.
>
>>
>>>> +
>>>> + /* hv_crash_asm_end is beyond last byte by 1 */
>>>> + size = &hv_crash_asm_end - &hv_crash_asm32;
>>>> + if (size + sizeof(struct hv_crash_tramp_data) > PAGE_SIZE) {
>>>> + pr_err("%s: trampoline page overflow\n", __func__);
>>>> + return -1;
>>>> + }
>>>> +
>>>> + dest = (void *)trampoline_va;
>>>> + memcpy(dest, &hv_crash_asm32, size);
>>>> +
>>>> + dest += size;
>>>> + dest = (void *)round_up((ulong)dest, 16);
>>>> + tramp = (struct hv_crash_tramp_data *)dest;
>>>> +
>>>> + /* see MAX_ASID_AVAILABLE in tlb.c: "PCID 0 is reserved for use by
>>>> + * non-PCID-aware users". Build cr3 with pcid 0
>>>> + */
>>>> + tramp->tramp32_cr3 = __sme_pa(hv_crash_ptpgs[0]);
>>>> +
>>>> + /* Note, when restoring X86_CR4_PCIDE, cr3[11:0] must be zero */
>>>> + tramp->kernel_cr3 = __sme_pa(init_mm.pgd);
>>>> +
>>>> + tramp->gdtr32.limit = sizeof(struct hv_crash_tramp_gdt);
>>>> + tramp->gdtr32.address = trampoline_pa +
>>>> + (ulong)&tramp->tramp_gdt - trampoline_va;
>>>> +
>>>> + /* base:0 limit:0xfffff type:b dpl:0 P:1 L:1 D:0 avl:0 G:1 */
>>>> + tramp->tramp_gdt.cs64 = 0x00af9a000000ffff;
>>>> +
>>>> + tramp->cs_jmptgt.csval = 0x8;
>>>> + offs = (ulong)&hv_crash_asm64_lbl - (ulong)&hv_crash_asm32;
>>>> + tramp->cs_jmptgt.address = trampoline_pa + offs;
>>>> +
>>>> + tramp->c_entry_addr = (u64)&hv_crash_c_entry;
>>>> +
>>>> + devirt_cr3arg = trampoline_pa + (ulong)dest - trampoline_va;
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +/*
>>>> + * Build 32bit trampoline page table for transition from protected mode
>>>> + * non-paging to long-mode paging. This transition needs pagetables below 4G.
>>>> + */
>>>> +static void hv_crash_build_tramp_pt(void)
>>>> +{
>>>> + p4d_t *p4d;
>>>> + pud_t *pud;
>>>> + pmd_t *pmd;
>>>> + pte_t *pte;
>>>> + u64 pa, addr = trampoline_pa;
>>>> +
>>>> + p4d = hv_crash_ptpgs[0] + pgd_index(addr) * sizeof(p4d);
>>>> + pa = virt_to_phys(hv_crash_ptpgs[1]);
>>>> + set_p4d(p4d, __p4d(_PAGE_TABLE | pa));
>>>> + p4d->p4d &= ~(_PAGE_NX); /* disable no execute */
>>>> +
>>>> + pud = hv_crash_ptpgs[1] + pud_index(addr) * sizeof(pud);
>>>> + pa = virt_to_phys(hv_crash_ptpgs[2]);
>>>> + set_pud(pud, __pud(_PAGE_TABLE | pa));
>>>> +
>>>> + pmd = hv_crash_ptpgs[2] + pmd_index(addr) * sizeof(pmd);
>>>> + pa = virt_to_phys(hv_crash_ptpgs[3]);
>>>> + set_pmd(pmd, __pmd(_PAGE_TABLE | pa));
>>>> +
>>>> + pte = hv_crash_ptpgs[3] + pte_index(addr) * sizeof(pte);
>>>> + set_pte(pte, pfn_pte(addr >> PAGE_SHIFT, PAGE_KERNEL_EXEC));
>>>> +}
>>>> +
>>>> +/*
>>>> + * Setup trampoline for devirtualization:
>>>> + * - a page below 4G, ie 32bit addr containing asm glue code that mshv jmps to
>>>> + * in protected mode.
>>>> + * - 4 pages for a temporary page table that asm code uses to turn paging on
>>>> + * - a temporary gdt to use in the compat mode.
>>>> + *
>>>> + * Returns: 0 on success
>>>> + */
>>>> +static int hv_crash_trampoline_setup(void)
>>>> +{
>>>> + int i, rc, order;
>>>> + struct page *page;
>>>> + u64 trampoline_va;
>>>> + gfp_t flags32 = GFP_KERNEL | GFP_DMA32 | __GFP_ZERO;
>>>> +
>>>> + /* page for 32bit trampoline assembly code + hv_crash_tramp_data */
>>>> + page = alloc_page(flags32);
>>>> + if (page == NULL) {
>>>> + pr_err("%s: failed to alloc asm stub page\n", __func__);
>>>> + return -1;
>>>> + }
>>>> +
>>>> + trampoline_va = (u64)page_to_virt(page);
>>>> + trampoline_pa = (u32)page_to_phys(page);
>>>> +
>>>> + order = 2; /* alloc 2^2 pages */
>>>> + page = alloc_pages(flags32, order);
>>>> + if (page == NULL) {
>>>> + pr_err("%s: failed to alloc pt pages\n", __func__);
>>>> + free_page(trampoline_va);
>>>> + return -1;
>>>> + }
>>>> +
>>>> + for (i = 0; i < 4; i++, page++)
>>>> + hv_crash_ptpgs[i] = page_to_virt(page);
>>>> +
>>>> + hv_crash_build_tramp_pt();
>>>> +
>>>> + rc = hv_crash_setup_trampdata(trampoline_va);
>>>> + if (rc)
>>>> + goto errout;
>>>> +
>>>> + return 0;
>>>> +
>>>> +errout:
>>>> + free_page(trampoline_va);
>>>> + free_pages((ulong)hv_crash_ptpgs[0], order);
>>>> +
>>>> + return rc;
>>>> +}
>>>> +
>>>> +/* Setup for kdump kexec to collect hypervisor ram when running as mshv root */
>>>> +void hv_root_crash_init(void)
>>>> +{
>>>> + int rc;
>>>> + struct hv_input_get_system_property *input;
>>>> + struct hv_output_get_system_property *output;
>>>> + unsigned long flags;
>>>> + u64 status;
>>>> + union hv_pfn_range cda_info;
>>>> +
>>>> + if (pgtable_l5_enabled()) {
>>>> + pr_err("Hyper-V: crash dump not yet supported on 5level PTs\n");
>>>> + return;
>>>> + }
>>>> +
>>>> + rc = register_nmi_handler(NMI_LOCAL, hv_crash_nmi_local, NMI_FLAG_FIRST,
>>>> + "hv_crash_nmi");
>>>> + if (rc) {
>>>> + pr_err("Hyper-V: failed to register crash nmi handler\n");
>>>> + return;
>>>> + }
>>>> +
>>>> + local_irq_save(flags);
>>>> + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
>>>> + output = *this_cpu_ptr(hyperv_pcpu_output_arg);
>>>> +
>>>> + memset(input, 0, sizeof(*input));
>>>> + memset(output, 0, sizeof(*output));
>>>
>>> Why zero the output area? This is one of those hypercall things that we're
>>> inconsistent about. A few hypercall call sites zero the output area, and it's
>>> not clear why they do. Hyper-V should be responsible for properly filling in
>>> the output area. Linux should not need to do this zero'ing, unless there's some
>>> known bug in Hyper-V for certain hypercalls, in which case there should be
>>> a code comment stating "why".
>>
>> for the same reason sometimes you see char *p = NULL, either leftover
>> code or someone was debugging or just copy and paste. this is just copy
>> paste. i agree in general that we don't need to clear it at all, in fact,
>> i'd like to remove them all! but i also understand people with different
>> skills and junior members find it easier to debug, and also we were in
>> early product development. for that reason, it doesn't have to be
>> consistent either, if some complex hypercalls are failing repeatedly,
>> just for ease of debug, one might leave it there temporarily. but
>> now that things are stable, i think we should just remove them all and
>> get used to a bit more inconvenient debugging...
>
> I see your point about debugging, but on balance I agree that they
> should all be removed. If there's some debug case, add it back
> temporarily to debug, but leave upstream without it. The zero'ing is
> also unnecessary code in the interrupt disabled window, which you
> have expressed concern about in a different thread.
yeah, i've been extremely busy so not able to pay much attention to
upstreaming, but imo they should have been removed before upstreaming.
a simple patch that just removes memset of output would be welcome.
>>
>>>> + input->property_id = HV_SYSTEM_PROPERTY_CRASHDUMPAREA;
>>>> +
>>>> + status = hv_do_hypercall(HVCALL_GET_SYSTEM_PROPERTY, input, output);
>>>> + cda_info.as_uint64 = output->hv_cda_info.as_uint64;
>>>> + local_irq_restore(flags);
>>>> +
>>>> + if (!hv_result_success(status)) {
>>>> + pr_err("Hyper-V: %s: property:%d %s\n", __func__,
>>>> + input->property_id, hv_result_to_string(status));
>>>> + goto err_out;
>>>> + }
>>>> +
>>>> + if (cda_info.base_pfn == 0) {
>>>> + pr_err("Hyper-V: hypervisor crash dump area pfn is 0\n");
>>>> + goto err_out;
>>>> + }
>>>> +
>>>> + hv_cda = phys_to_virt(cda_info.base_pfn << PAGE_SHIFT);
>>>
>>> Use HV_HYP_PAGE_SHIFT, since PFNs provided by Hyper-V are always in
>>> terms of the Hyper-V page size, which isn't necessarily the guest page size.
>>> Yes, on x86 there's no difference, but for future robustness ....
>>
>> i don't know about guests, but we won't even boot if dom0 pg size
>> didn't match.. but easier to change than to make the case..
>
> FWIW, a normal Linux guest on ARM64 works just fine with a page
> size of 16K or 64K, even though the underlying Hyper-V page size
> is only 4K. That's why we have HV_HYP_PAGE_SHIFT and related in
> the first place. Using it properly really matters for normal guests.
> (Having the guest page size smaller than the Hyper-V page size
> does *not* work, but there are no such use cases.)
>
> Even on ARM64, I know the root partition page size is required to
> match the Hyper-V page size. But using HV_HYP_PAGE_SIZE is
> still appropriate just to not leave code that will go wrong if the
> match requirement should ever change.
>
>>
>>>> +
>>>> + rc = hv_crash_trampoline_setup();
>>>> + if (rc)
>>>> + goto err_out;
>>>> +
>>>> + smp_ops.crash_stop_other_cpus = hv_crash_stop_other_cpus;
>>>> +
>>>> + crash_kexec_post_notifiers = true;
>>>> + hv_crash_enabled = 1;
>>>> + pr_info("Hyper-V: linux and hv kdump support enabled\n");
>>>
>>> This message and the message below aren't consistent. One refers
>>> to "hv kdump" and the other to "hyp kdump".
>>
>>>> +
>>>> + return;
>>>> +
>>>> +err_out:
>>>> + unregister_nmi_handler(NMI_LOCAL, "hv_crash_nmi");
>>>> + pr_err("Hyper-V: only linux (but not hyp) kdump support enabled\n");
>>>> +}
>>>> --
>>>> 2.36.1.vfs.0.0
>>>>
>>>
>
^ permalink raw reply
* RE: [PATCH v1 6/6] x86/hyperv: Enable build of hypervisor crashdump collection files
From: Michael Kelley @ 2025-09-18 23:53 UTC (permalink / raw)
To: Mukesh R, linux-hyperv@vger.kernel.org,
linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org
Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org,
decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com,
bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org,
hpa@zytor.com, arnd@arndb.de
In-Reply-To: <daa1e607-26ca-887c-b32c-d39addd6fa44@linux.microsoft.com>
From: Mukesh R <mrathor@linux.microsoft.com> Sent: Tuesday, September 16, 2025 6:16 PM
>
> On 9/15/25 10:56, Michael Kelley wrote:
> > From: Mukesh Rathor <mrathor@linux.microsoft.com> Sent: Tuesday, September 9, 2025 5:10 PM
> >>
> >> Enable build of the new files introduced in the earlier commits and add
> >> call to do the setup during boot.
> >>
> >> Signed-off-by: Mukesh Rathor <mrathor@linux.microsoft.com>
> >> ---
> >> arch/x86/hyperv/Makefile | 6 ++++++
> >> arch/x86/hyperv/hv_init.c | 1 +
> >> include/asm-generic/mshyperv.h | 9 +++++++++
> >> 3 files changed, 16 insertions(+)
> >>
> >> diff --git a/arch/x86/hyperv/Makefile b/arch/x86/hyperv/Makefile
> >> index d55f494f471d..6f5d97cddd80 100644
> >> --- a/arch/x86/hyperv/Makefile
> >> +++ b/arch/x86/hyperv/Makefile
> >> @@ -5,4 +5,10 @@ obj-$(CONFIG_HYPERV_VTL_MODE) += hv_vtl.o
> >>
> >> ifdef CONFIG_X86_64
> >> obj-$(CONFIG_PARAVIRT_SPINLOCKS) += hv_spinlock.o
> >> +
> >> + ifdef CONFIG_MSHV_ROOT
> >> + CFLAGS_REMOVE_hv_trampoline.o += -pg
> >> + CFLAGS_hv_trampoline.o += -fno-stack-protector
> >> + obj-$(CONFIG_CRASH_DUMP) += hv_crash.o hv_trampoline.o
> >> + endif
> >> endif
> >> diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
> >> index afdbda2dd7b7..577bbd143527 100644
> >> --- a/arch/x86/hyperv/hv_init.c
> >> +++ b/arch/x86/hyperv/hv_init.c
> >> @@ -510,6 +510,7 @@ void __init hyperv_init(void)
> >> memunmap(src);
> >>
> >> hv_remap_tsc_clocksource();
> >> + hv_root_crash_init();
> >> } else {
> >> hypercall_msr.guest_physical_address = vmalloc_to_pfn(hv_hypercall_pg);
> >> wrmsrq(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64);
> >> diff --git a/include/asm-generic/mshyperv.h b/include/asm-generic/mshyperv.h
> >> index dbd4c2f3aee3..952c221765f5 100644
> >> --- a/include/asm-generic/mshyperv.h
> >> +++ b/include/asm-generic/mshyperv.h
> >> @@ -367,6 +367,15 @@ int hv_call_deposit_pages(int node, u64 partition_id, u32
> >> num_pages);
> >> int hv_call_add_logical_proc(int node, u32 lp_index, u32 acpi_id);
> >> int hv_call_create_vp(int node, u64 partition_id, u32 vp_index, u32 flags);
> >>
> >> +#if CONFIG_CRASH_DUMP
> >> +void hv_root_crash_init(void);
> >> +void hv_crash_asm32(void);
> >> +void hv_crash_asm64_lbl(void);
> >> +void hv_crash_asm_end(void);
> >> +#else /* CONFIG_CRASH_DUMP */
> >> +static inline void hv_root_crash_init(void) {}
> >> +#endif /* CONFIG_CRASH_DUMP */
> >> +
> >
> > The hv_crash_asm* functions are x86 specific. Seems like their
> > declarations should go in arch/x86/include/asm/mshyperv.h, not in
> > the architecture-neutral include/asm-generic/mshyperv.h.
>
> well, arm port is going on. i suppose i could move it to x86 and
> they can move it back here in their patch submissions. hopefully
> they will remember or someone will catch it.
I could see the ARM64 implementation implementing its own version
of hv_root_crash_init() since that's a generic name. But sharing the
"asm" function names across architectures seems more questionable.
I doubt there would be hv_crash_asm32() on ARM64. :-)
>
> >> #else /* CONFIG_MSHV_ROOT */
> >> static inline bool hv_root_partition(void) { return false; }
> >> static inline bool hv_l1vh_partition(void) { return false; }
> >> --
> >> 2.36.1.vfs.0.0
> >>
^ permalink raw reply
* RE: [PATCH v1 5/6] x86/hyperv: Implement hypervisor ram collection into vmcore
From: Michael Kelley @ 2025-09-18 23:53 UTC (permalink / raw)
To: Mukesh R, linux-hyperv@vger.kernel.org,
linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org
Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org,
decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com,
bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org,
hpa@zytor.com, arnd@arndb.de
In-Reply-To: <87cab5ec-ab76-b1cf-4891-30314e5dace6@linux.microsoft.com>
From: Mukesh R <mrathor@linux.microsoft.com> Sent: Tuesday, September 16, 2025 6:13 PM
>
> On 9/15/25 10:55, Michael Kelley wrote:
> > From: Mukesh Rathor <mrathor@linux.microsoft.com> Sent: Tuesday, September 9, 2025 5:10 PM
> >>
> >> Introduce a new file to implement collection of hypervisor ram into the
> >
> > s/ram/RAM/ (multiple places)
>
> a quick grep indicates using saying ram is common, i like ram over RAM
>
> >> vmcore collected by linux. By default, the hypervisor ram is locked, ie,
> >> protected via hw page table. Hyper-V implements a disable hypercall which
> >
> > The terminology here is a bit confusing since you have two names for
> > the same thing: "disable" hypervisor, and "devirtualize". Is it possible to
> > just use "devirtualize" everywhere, and drop the "disable" terminology?
>
> The concept is devirtualize and the actual hypercall was originally named
> disable. so intermixing is natural imo.
>
> >> essentially devirtualizes the system on the fly. This mechanism makes the
> >> hypervisor ram accessible to linux. Because the hypervisor ram is already
> >> mapped into linux address space (as reserved ram),
> >
> > Is the hypervisor RAM mapped into the VMM process user address space,
> > or somewhere in the kernel address space? If the latter, where in the kernel
> > code, or what mechanism, does that? Just curious, as I wasn't aware that
> > this is happening ....
>
> mapped in kernel as normal ram and we reserve it very early in boot. i
> see that patch has not made it here yet, should be coming very soon.
OK, that's fine. The answer to my question is coming soon ....
>
> >> it is automatically
> >> collected into the vmcore without extra work. More details of the
> >> implementation are available in the file prologue.
> >>
> >> Signed-off-by: Mukesh Rathor <mrathor@linux.microsoft.com>
> >> ---
> >> arch/x86/hyperv/hv_crash.c | 622 +++++++++++++++++++++++++++++++++++++
> >> 1 file changed, 622 insertions(+)
> >> create mode 100644 arch/x86/hyperv/hv_crash.c
> >>
> >> diff --git a/arch/x86/hyperv/hv_crash.c b/arch/x86/hyperv/hv_crash.c
> >> new file mode 100644
> >> index 000000000000..531bac79d598
> >> --- /dev/null
> >> +++ b/arch/x86/hyperv/hv_crash.c
> >> @@ -0,0 +1,622 @@
> >> +// SPDX-License-Identifier: GPL-2.0-only
> >> +/*
> >> + * X86 specific Hyper-V kdump/crash support module
> >> + *
> >> + * Copyright (C) 2025, Microsoft, Inc.
> >> + *
> >> + * This module implements hypervisor ram collection into vmcore for both
> >> + * cases of the hypervisor crash and linux dom0/root crash.
> >
> > For a hypervisor crash, does any of this apply to general guest VMs? I'm
> > thinking it does not. Hypervisor RAM is collected only into the vmcore
> > for the root partition, right? Maybe some additional clarification could be
> > added so there's no confusion in this regard.
>
> it would be odd for guests to collect hyp core, and target audience is
> assumed to be those who are somewhat familiar with basic concepts before
> getting here.
I was unsure because I had not seen any code that adds the hypervisor memory
to the Linux memory map. Thought maybe something was going on I hadn’t
heard about, so I didn't know the scope of it.
Of course, I'm one of those people who was *not* familiar with the basic concepts
before getting here. And given that there's no spec available from Hyper-V,
the comments in this patch set are all there is for anyone outside of Microsoft.
In that vein, I think it's reasonable to provide some description of how this
all works in the code comments. And you've done that, which is very
helpful. But I encountered a few places where I was confused or unclear, and
my suggestions here and in Patch 4 are just about making things as precise as
possible without adding a huge amount of additional verbiage. For someone
new, English text descriptions that the code can be checked against are
helpful, and drawing hard boundaries ("this is only applicable to the root
partition") is helpful.
If you don't want to deal with it now, I could provide a follow-on patch later
that tweaks or augments the wording a bit to clarify some of these places.
You can review, like with any patch. I've done wording work over the years
to many places in the VMBus code, and more broadly in providing most of
the documentation in Documentation/virt/hyperv.
>
> > And what *does* happen to guest VMs after a hypervisor crash?
>
> they are gone... what else could we do?
>
> >> + * Hyper-V implements
> >> + * a devirtualization hypercall with a 32bit protected mode ABI callback. This
> >> + * mechanism must be used to unlock hypervisor ram. Since the hypervisor ram
> >> + * is already mapped in linux, it is automatically collected into linux vmcore,
> >> + * and can be examined by the crash command (raw ram dump) or windbg.
> >> + *
> >> + * At a high level:
> >> + *
> >> + * Hypervisor Crash:
> >> + * Upon crash, hypervisor goes into an emergency minimal dispatch loop, a
> >> + * restrictive mode with very limited hypercall and msr support.
> >
> > s/msr/MSR/
>
> msr is used all over, seems acceptable.
>
> >> + * Each cpu then injects NMIs into dom0/root vcpus.
> >
> > The "Each cpu" part of this sentence is confusing to me -- which CPUs does
> > this refer to? Maybe it would be better to say "It then injects an NMI into
> > each dom0/root partition vCPU." without being specific as to which CPUs do
> > the injecting since that seems more like a hypervisor implementation detail
> > that's not relevant here.
>
> all cpus in the system. there is a dedicated/pinned dom0 vcpu for each cpu.
OK, that makes sense now that I think about it. Each physical CPU in the host
has a corresponding vCPU in the dom0/root partition. And each of the vCPUs
gets an NMI that sends it to the Linux-in-dom0 NMI handler, even if it was off
running a vCPU in some guest VM.
>
> >> + * A shared page is used to check
> >> + * by linux in the nmi handler if the hypervisor has crashed. This shared
> >
> > s/nmi/NMI/ (multiple places)
>
> >> + * page is setup in hv_root_crash_init during boot.
> >> + *
> >> + * Linux Crash:
> >> + * In case of linux crash, the callback hv_crash_stop_other_cpus will send
> >> + * NMIs to all cpus, then proceed to the crash_nmi_callback where it waits
> >> + * for all cpus to be in NMI.
> >> + *
> >> + * NMI Handler (upon quorum):
> >> + * Eventually, in both cases, all cpus wil end up in the nmi hanlder.
> >
> > s/hanlder/handler/
> >
> > And maybe just drop the word "wil" (which is misspelled).
> >
> >> + * Hyper-V requires the disable hypervisor must be done from the bsp. So
> >
> > s/bsp/BSP (multiple places)
> >
> >> + * the bsp nmi handler saves current context, does some fixups and makes
> >> + * the hypercall to disable the hypervisor, ie, devirtualize. Hypervisor
> >> + * at that point will suspend all vcpus (except the bsp), unlock all its
> >> + * ram, and return to linux at the 32bit mode entry RIP.
> >> + *
> >> + * Linux 32bit entry trampoline will then restore long mode and call C
> >> + * function here to restore context and continue execution to crash kexec.
> >> + */
> >> +
> >> +#include <linux/delay.h>
> >> +#include <linux/kexec.h>
> >> +#include <linux/crash_dump.h>
> >> +#include <linux/panic.h>
> >> +#include <asm/apic.h>
> >> +#include <asm/desc.h>
> >> +#include <asm/page.h>
> >> +#include <asm/pgalloc.h>
> >> +#include <asm/mshyperv.h>
> >> +#include <asm/nmi.h>
> >> +#include <asm/idtentry.h>
> >> +#include <asm/reboot.h>
> >> +#include <asm/intel_pt.h>
> >> +
> >> +int hv_crash_enabled;
> >
> > Seems like this is conceptually a "bool", not an "int".
>
> yeah, can change it to bool if i do another iteration.
>
> >> +EXPORT_SYMBOL_GPL(hv_crash_enabled);
> >> +
> >> +struct hv_crash_ctxt {
> >> + ulong rsp;
> >> + ulong cr0;
> >> + ulong cr2;
> >> + ulong cr4;
> >> + ulong cr8;
> >> +
> >> + u16 cs;
> >> + u16 ss;
> >> + u16 ds;
> >> + u16 es;
> >> + u16 fs;
> >> + u16 gs;
> >> +
> >> + u16 gdt_fill;
> >> + struct desc_ptr gdtr;
> >> + char idt_fill[6];
> >> + struct desc_ptr idtr;
> >> +
> >> + u64 gsbase;
> >> + u64 efer;
> >> + u64 pat;
> >> +};
> >> +static struct hv_crash_ctxt hv_crash_ctxt;
> >> +
> >> +/* Shared hypervisor page that contains crash dump area we peek into.
> >> + * NB: windbg looks for "hv_cda" symbol so don't change it.
> >> + */
> >> +static struct hv_crashdump_area *hv_cda;
> >> +
> >> +static u32 trampoline_pa, devirt_cr3arg;
> >> +static atomic_t crash_cpus_wait;
> >> +static void *hv_crash_ptpgs[4];
> >> +static int hv_has_crashed, lx_has_crashed;
> >
> > These are conceptually "bool" as well.
> >
> >> +
> >> +/* This cannot be inlined as it needs stack */
> >> +static noinline __noclone void hv_crash_restore_tss(void)
> >> +{
> >> + load_TR_desc();
> >> +}
> >> +
> >> +/* This cannot be inlined as it needs stack */
> >> +static noinline void hv_crash_clear_kernpt(void)
> >> +{
> >> + pgd_t *pgd;
> >> + p4d_t *p4d;
> >> +
> >> + /* Clear entry so it's not confusing to someone looking at the core */
> >> + pgd = pgd_offset_k(trampoline_pa);
> >> + p4d = p4d_offset(pgd, trampoline_pa);
> >> + native_p4d_clear(p4d);
> >> +}
> >> +
> >> +/*
> >> + * This is the C entry point from the asm glue code after the devirt hypercall.
> >> + * We enter here in IA32-e long mode, ie, full 64bit mode running on kernel
> >> + * page tables with our below 4G page identity mapped, but using a temporary
> >> + * GDT. ds/fs/gs/es are null. ss is not usable. bp is null. stack is not
> >> + * available. We restore kernel GDT, and rest of the context, and continue
> >> + * to kexec.
> >> + */
> >> +static asmlinkage void __noreturn hv_crash_c_entry(void)
> >> +{
> >> + struct hv_crash_ctxt *ctxt = &hv_crash_ctxt;
> >> +
> >> + /* first thing, restore kernel gdt */
> >> + native_load_gdt(&ctxt->gdtr);
> >> +
> >> + asm volatile("movw %%ax, %%ss" : : "a"(ctxt->ss));
> >> + asm volatile("movq %0, %%rsp" : : "m"(ctxt->rsp));
> >> +
> >> + asm volatile("movw %%ax, %%ds" : : "a"(ctxt->ds));
> >> + asm volatile("movw %%ax, %%es" : : "a"(ctxt->es));
> >> + asm volatile("movw %%ax, %%fs" : : "a"(ctxt->fs));
> >> + asm volatile("movw %%ax, %%gs" : : "a"(ctxt->gs));
> >> +
> >> + native_wrmsrq(MSR_IA32_CR_PAT, ctxt->pat);
> >> + asm volatile("movq %0, %%cr0" : : "r"(ctxt->cr0));
> >> +
> >> + asm volatile("movq %0, %%cr8" : : "r"(ctxt->cr8));
> >> + asm volatile("movq %0, %%cr4" : : "r"(ctxt->cr4));
> >> + asm volatile("movq %0, %%cr2" : : "r"(ctxt->cr4));
> >> +
> >> + native_load_idt(&ctxt->idtr);
> >> + native_wrmsrq(MSR_GS_BASE, ctxt->gsbase);
> >> + native_wrmsrq(MSR_EFER, ctxt->efer);
> >> +
> >> + /* restore the original kernel CS now via far return */
> >> + asm volatile("movzwq %0, %%rax\n\t"
> >> + "pushq %%rax\n\t"
> >> + "pushq $1f\n\t"
> >> + "lretq\n\t"
> >> + "1:nop\n\t" : : "m"(ctxt->cs) : "rax");
> >> +
> >> + /* We are in asmlinkage without stack frame, hence make a C function
> >> + * call which will buy stack frame to restore the tss or clear PT entry.
> >> + */
> >> + hv_crash_restore_tss();
> >> + hv_crash_clear_kernpt();
> >> +
> >> + /* we are now fully in devirtualized normal kernel mode */
> >> + __crash_kexec(NULL);
> >
> > The comments for __crash_kexec() say that "panic_cpu" should be set to
> > the current CPU. I don't see that such is the case here.
>
> if linux panic, it would be set by vpanic, if hyp crash, that is
> irrelevant.
>
> >> +
> >> + for (;;)
> >> + cpu_relax();
> >
> > Is the intent that __crash_kexec() should never return, on any of the vCPUs,
> > because devirtualization isn't done unless there's a valid kdump image loaded?
> > I wonder if
> >
> > native_wrmsrq(HV_X64_MSR_RESET, 1);
> >
> > would be better than looping forever in case __crash_kexec() fails
> > somewhere along the way even if there's a kdump image loaded.
>
> yeah, i've gone thru all 3 possibilities here:
> o loop forever
> o reset
> o BUG() : this was in V0
>
> reset is just bad because system would just reboot without any indication
> if hyp crashes. with loop at least there is a hang, and one could make
> note of it, and if internal, attach debugger.
>
> BUG is best imo because with hyp gone linux will try to redo panic
> and we would print something extra to help. I think i'll just go
> back to my V0: BUG()
>
> >> +}
> >> +/* Tell gcc we are using lretq long jump in the above function intentionally */
> >> +STACK_FRAME_NON_STANDARD(hv_crash_c_entry);
> >> +
> >> +static void hv_mark_tss_not_busy(void)
> >> +{
> >> + struct desc_struct *desc = get_current_gdt_rw();
> >> + tss_desc tss;
> >> +
> >> + memcpy(&tss, &desc[GDT_ENTRY_TSS], sizeof(tss_desc));
> >> + tss.type = 0x9; /* available 64-bit TSS. 0xB is busy TSS */
> >> + write_gdt_entry(desc, GDT_ENTRY_TSS, &tss, DESC_TSS);
> >> +}
> >> +
> >> +/* Save essential context */
> >> +static void hv_hvcrash_ctxt_save(void)
> >> +{
> >> + struct hv_crash_ctxt *ctxt = &hv_crash_ctxt;
> >> +
> >> + asm volatile("movq %%rsp,%0" : "=m"(ctxt->rsp));
> >> +
> >> + ctxt->cr0 = native_read_cr0();
> >> + ctxt->cr4 = native_read_cr4();
> >> +
> >> + asm volatile("movq %%cr2, %0" : "=a"(ctxt->cr2));
> >> + asm volatile("movq %%cr8, %0" : "=a"(ctxt->cr8));
> >> +
> >> + asm volatile("movl %%cs, %%eax" : "=a"(ctxt->cs));
> >> + asm volatile("movl %%ss, %%eax" : "=a"(ctxt->ss));
> >> + asm volatile("movl %%ds, %%eax" : "=a"(ctxt->ds));
> >> + asm volatile("movl %%es, %%eax" : "=a"(ctxt->es));
> >> + asm volatile("movl %%fs, %%eax" : "=a"(ctxt->fs));
> >> + asm volatile("movl %%gs, %%eax" : "=a"(ctxt->gs));
> >> +
> >> + native_store_gdt(&ctxt->gdtr);
> >> + store_idt(&ctxt->idtr);
> >> +
> >> + ctxt->gsbase = __rdmsr(MSR_GS_BASE);
> >> + ctxt->efer = __rdmsr(MSR_EFER);
> >> + ctxt->pat = __rdmsr(MSR_IA32_CR_PAT);
> >> +}
> >> +
> >> +/* Add trampoline page to the kernel pagetable for transition to kernel PT */
> >> +static void hv_crash_fixup_kernpt(void)
> >> +{
> >> + pgd_t *pgd;
> >> + p4d_t *p4d;
> >> +
> >> + pgd = pgd_offset_k(trampoline_pa);
> >> + p4d = p4d_offset(pgd, trampoline_pa);
> >> +
> >> + /* trampoline_pa is below 4G, so no pre-existing entry to clobber */
> >> + p4d_populate(&init_mm, p4d, (pud_t *)hv_crash_ptpgs[1]);
> >> + p4d->p4d = p4d->p4d & ~(_PAGE_NX); /* enable execute */
> >> +}
> >> +
> >> +/*
> >> + * Now that all cpus are in nmi and spinning, we notify the hyp that linux has
> >> + * crashed and will collect core. This will cause the hyp to quiesce and
> >> + * suspend all VPs except the bsp. Called if linux crashed and not the hyp.
> >> + */
> >> +static void hv_notify_prepare_hyp(void)
> >> +{
> >> + u64 status;
> >> + struct hv_input_notify_partition_event *input;
> >> + struct hv_partition_event_root_crashdump_input *cda;
> >> +
> >> + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
> >> + cda = &input->input.crashdump_input;
> >
> > The code ordering here is a bit weird. I'd expect this line to be grouped
> > with cda->crashdump_action being set.
>
> we are setting two pointers, and using them later. setting pointers
> up front is pretty normal.
>
> >> + memset(input, 0, sizeof(*input));
> >> + input->event = HV_PARTITION_EVENT_ROOT_CRASHDUMP;
> >> +
> >> + cda->crashdump_action = HV_CRASHDUMP_ENTRY;
> >> + status = hv_do_hypercall(HVCALL_NOTIFY_PARTITION_EVENT, input, NULL);
> >> + if (!hv_result_success(status))
> >> + return;
> >> +
> >> + cda->crashdump_action = HV_CRASHDUMP_SUSPEND_ALL_VPS;
> >> + hv_do_hypercall(HVCALL_NOTIFY_PARTITION_EVENT, input, NULL);
> >> +}
> >> +
> >> +/*
> >> + * Common function for all cpus before devirtualization.
> >> + *
> >> + * Hypervisor crash: all cpus get here in nmi context.
> >> + * Linux crash: the panicing cpu gets here at base level, all others in nmi
> >> + * context. Note, panicing cpu may not be the bsp.
> >> + *
> >> + * The function is not inlined so it will show on the stack. It is named so
> >> + * because the crash cmd looks for certain well known function names on the
> >> + * stack before looking into the cpu saved note in the elf section, and
> >> + * that work is currently incomplete.
> >> + *
> >> + * Notes:
> >> + * Hypervisor crash:
> >> + * - the hypervisor is in a very restrictive mode at this point and any
> >> + * vmexit it cannot handle would result in reboot. For example, console
> >> + * output from here would result in synic ipi hcall, which would result
> >> + * in reboot. So, no mumbo jumbo, just get to kexec as quickly as possible.
> >> + *
> >> + * Devirtualization is supported from the bsp only.
> >> + */
> >> +static noinline __noclone void crash_nmi_callback(struct pt_regs *regs)
> >> +{
> >> + struct hv_input_disable_hyp_ex *input;
> >> + u64 status;
> >> + int msecs = 1000, ccpu = smp_processor_id();
> >> +
> >> + if (ccpu == 0) {
> >> + /* crash_save_cpu() will be done in the kexec path */
> >> + cpu_emergency_stop_pt(); /* disable performance trace */
> >> + atomic_inc(&crash_cpus_wait);
> >> + } else {
> >> + crash_save_cpu(regs, ccpu);
> >> + cpu_emergency_stop_pt(); /* disable performance trace */
> >> + atomic_inc(&crash_cpus_wait);
> >> + for (;;); /* cause no vmexits */
> >> + }
> >> +
> >> + while (atomic_read(&crash_cpus_wait) < num_online_cpus() && msecs--)
> >> + mdelay(1);
> >> +
> >> + stop_nmi();
> >> + if (!hv_has_crashed)
> >> + hv_notify_prepare_hyp();
> >> +
> >> + if (crashing_cpu == -1)
> >> + crashing_cpu = ccpu; /* crash cmd uses this */
> >
> > Could just be "crashing_cpu = 0" since only the BSP gets here.
>
> a code change request has been open for while to remove the requirement
> of bsp..
>
> >> +
> >> + hv_hvcrash_ctxt_save();
> >> + hv_mark_tss_not_busy();
> >> + hv_crash_fixup_kernpt();
> >> +
> >> + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
> >> + memset(input, 0, sizeof(*input));
> >> + input->rip = trampoline_pa; /* PA of hv_crash_asm32 */
> >> + input->arg = devirt_cr3arg; /* PA of trampoline page table L4 */
> >
> > Is this comment correct? Isn't it the PA of struct hv_crash_tramp_data?
> > And just for clarification, Hyper-V treats this "arg" value as opaque and does
> > not access it. It only provides it in EDI when it invokes the trampoline
> > function, right?
>
> comment is correct. cr3 always points to l4 (or l5 if 5 level page tables).
Yes, the comment matches the name of the "devirt_cr3arg" variable.
Unfortunately my previous comment was incomplete because the value
stored in the static variable "devirt_cr3arg" isn’t the address of an L4 page
table. It's not a CR3 value. The value stored in devirt_cr3arg is actually the
PA of struct hv_crash_tramp_data. The CR3 value is stored in the
tramp32_cr3 field (at offset 0) of that structure, so there's an additional level
of indirection. The (corrected) comment in the header to hv_crash_asm32()
describes EDI as containing "PA of struct hv_crash_tramp_data", which
ought to match what is described here. I'd say that "devirt_cr3arg" ought
to be renamed to "tramp_data_pa" or something else parallel to
"trampoline_pa".
>
> right, comes in edi, i don't know what EDI is (just kidding!)...
>
> >> +
> >> + status = hv_do_hypercall(HVCALL_DISABLE_HYP_EX, input, NULL);
> >> +
> >> + /* Devirt failed, just reboot as things are in very bad state now */
> >> + native_wrmsrq(HV_X64_MSR_RESET, 1); /* get hv to reboot */
> >> +}
> >> +
> >> +/*
> >> + * Generic nmi callback handler: could be called without any crash also.
> >> + * hv crash: hypervisor injects nmi's into all cpus
> >> + * lx crash: panicing cpu sends nmi to all but self via crash_stop_other_cpus
> >> + */
> >> +static int hv_crash_nmi_local(unsigned int cmd, struct pt_regs *regs)
> >> +{
> >> + int ccpu = smp_processor_id();
> >> +
> >> + if (!hv_has_crashed && hv_cda && hv_cda->cda_valid)
> >> + hv_has_crashed = 1;
> >> +
> >> + if (!hv_has_crashed && !lx_has_crashed)
> >> + return NMI_DONE; /* ignore the nmi */
> >> +
> >> + if (hv_has_crashed) {
> >> + if (!kexec_crash_loaded() || !hv_crash_enabled) {
> >> + if (ccpu == 0) {
> >> + native_wrmsrq(HV_X64_MSR_RESET, 1); /* reboot */
> >> + } else
> >> + for (;;); /* cause no vmexits */
> >> + }
> >> + }
> >> +
> >> + crash_nmi_callback(regs);
> >> +
> >> + return NMI_DONE;
> >
> > crash_nmi_callback() should never return, right? Normally one would
> > expect to return NMI_HANDLED here, but I guess it doesn't matter
> > if the return is never executed.
>
> correct.
>
> >> +}
> >> +
> >> +/*
> >> + * hv_crash_stop_other_cpus() == smp_ops.crash_stop_other_cpus
> >> + *
> >> + * On normal linux panic, this is called twice: first from panic and then again
> >> + * from native_machine_crash_shutdown.
> >> + *
> >> + * In case of mshv, 3 ways to get here:
> >> + * 1. hv crash (only bsp will get here):
> >> + * BSP : nmi callback -> DisableHv -> hv_crash_asm32 -> hv_crash_c_entry
> >> + * -> __crash_kexec -> native_machine_crash_shutdown
> >> + * -> crash_smp_send_stop -> smp_ops.crash_stop_other_cpus
> >> + * linux panic:
> >> + * 2. panic cpu x: panic() -> crash_smp_send_stop
> >> + * -> smp_ops.crash_stop_other_cpus
> >> + * 3. bsp: native_machine_crash_shutdown -> crash_smp_send_stop
> >> + *
> >> + * NB: noclone and non standard stack because of call to crash_setup_regs().
> >> + */
> >> +static void __noclone hv_crash_stop_other_cpus(void)
> >> +{
> >> + static int crash_stop_done;
> >> + struct pt_regs lregs;
> >> + int ccpu = smp_processor_id();
> >> +
> >> + if (hv_has_crashed)
> >> + return; /* all cpus already in nmi handler path */
> >> +
> >> + if (!kexec_crash_loaded())
> >> + return;
> >
> > If we're in a normal panic path (your Case #2 above) with no kdump kernel
> > loaded, why leave the other vCPUs running? Seems like that could violate
> > expectations in vpanic(), where it calls panic_other_cpus_shutdown() and
> > thereafter assumes other vCPUs are not running.
>
> no, there is lots of complexity here!
>
> if we hang vcpus here, hyp will note and may trigger its own watchdog.
> also, machine_crash_shutdown() does another ipi.
>
> I think the best thing to do here is go back to my V0 which did not
> have check for kexec_crash_loaded(), but had this in hv_crash_c_entry:
>
> + /* we are now fully in devirtualized normal kernel mode */
> + __crash_kexec(NULL);
> +
> + BUG();
>
>
> this way hyp would be disabled, ie, system devirtualized, and
> __crash_kernel() will return, resulting in BUG() that will cause
> it to go thru panic and honor panic= parameter with either hang
> or reset. instead of bug, i could just call panic() also.
>
> >> +
> >> + if (crash_stop_done)
> >> + return;
> >> + crash_stop_done = 1;
> >
> > Is crash_stop_done necessary? hv_crash_stop_other_cpus() is called
> > from crash_smp_send_stop(), which has its own static variable
> > "cpus_stopped" that does the same thing.
>
> yes. for error paths.
>
> >> +
> >> + /* linux has crashed: hv is healthy, we can ipi safely */
> >> + lx_has_crashed = 1;
> >> + wmb(); /* nmi handlers look at lx_has_crashed */
> >> +
> >> + apic->send_IPI_allbutself(NMI_VECTOR);
> >
> > The default .crash_stop_other_cpus function is kdump_nmi_shootdown_cpus().
> > In addition to sending the NMI IPI, it does disable_local_APIC(). I don't know, but
> > should disable_local_APIC() be done somewhere here as well?
>
> no, hyp does that.
As part of the devirt operation initiated by the HVCALL_DISABLE_HYP_EX
hypercall in crash_nmi_callback()? This gets back to an earlier question/comment
where I was trying to figure out if the APIC is still enabled, and in what mode,
when hv_crash_asm32() is invoked.
>
> >> +
> >> + if (crashing_cpu == -1)
> >> + crashing_cpu = ccpu; /* crash cmd uses this */
> >> +
> >> + /* crash_setup_regs() happens in kexec also, but for the kexec cpu which
> >> + * is the bsp. We could be here on non-bsp cpu, collect regs if so.
> >> + */
> >> + if (ccpu)
> >> + crash_setup_regs(&lregs, NULL);
> >> +
> >> + crash_nmi_callback(&lregs);
> >> +}
> >> +STACK_FRAME_NON_STANDARD(hv_crash_stop_other_cpus);
> >> +
> >> +/* This GDT is accessed in IA32-e compat mode which uses 32bits addresses */
> >> +struct hv_gdtreg_32 {
> >> + u16 fill;
> >> + u16 limit;
> >> + u32 address;
> >> +} __packed;
> >> +
> >> +/* We need a CS with L bit to goto IA32-e long mode from 32bit compat mode */
> >> +struct hv_crash_tramp_gdt {
> >> + u64 null; /* index 0, selector 0, null selector */
> >> + u64 cs64; /* index 1, selector 8, cs64 selector */
> >> +} __packed;
> >> +
> >> +/* No stack, so jump via far ptr in memory to load the 64bit CS */
> >> +struct hv_cs_jmptgt {
> >> + u32 address;
> >> + u16 csval;
> >> + u16 fill;
> >> +} __packed;
> >> +
> >> +/* This trampoline data is copied onto the trampoline page after the asm code */
> >> +struct hv_crash_tramp_data {
> >> + u64 tramp32_cr3;
> >> + u64 kernel_cr3;
> >> + struct hv_gdtreg_32 gdtr32;
> >> + struct hv_crash_tramp_gdt tramp_gdt;
> >> + struct hv_cs_jmptgt cs_jmptgt;
> >> + u64 c_entry_addr;
> >> +} __packed;
> >> +
> >> +/*
> >> + * Setup a temporary gdt to allow the asm code to switch to the long mode.
> >> + * Since the asm code is relocated/copied to a below 4G page, it cannot use rip
> >> + * relative addressing, hence we must use trampoline_pa here. Also, save other
> >> + * info like jmp and C entry targets for same reasons.
> >> + *
> >> + * Returns: 0 on success, -1 on error
> >> + */
> >> +static int hv_crash_setup_trampdata(u64 trampoline_va)
> >> +{
> >> + int size, offs;
> >> + void *dest;
> >> + struct hv_crash_tramp_data *tramp;
> >> +
> >> + /* These must match exactly the ones in the corresponding asm file */
> >> + BUILD_BUG_ON(offsetof(struct hv_crash_tramp_data, tramp32_cr3) != 0);
> >> + BUILD_BUG_ON(offsetof(struct hv_crash_tramp_data, kernel_cr3) != 8);
> >> + BUILD_BUG_ON(offsetof(struct hv_crash_tramp_data, gdtr32.limit) != 18);
> >> + BUILD_BUG_ON(offsetof(struct hv_crash_tramp_data,
> >> + cs_jmptgt.address) != 40);
> >
> > It would be nice to pick up the constants from a #include file that is
> > shared with the asm code in Patch 4 of the series.
>
> yeah, could go either way, some don't like tiny headers... if there are
> no objections to new header for this, i could go that way too.
Saw your follow-on comments about this as well. The tiny header
is ugly. It's a judgment call that can go either way, so go with your
preference.
>
> >> +
> >> + /* hv_crash_asm_end is beyond last byte by 1 */
> >> + size = &hv_crash_asm_end - &hv_crash_asm32;
> >> + if (size + sizeof(struct hv_crash_tramp_data) > PAGE_SIZE) {
> >> + pr_err("%s: trampoline page overflow\n", __func__);
> >> + return -1;
> >> + }
> >> +
> >> + dest = (void *)trampoline_va;
> >> + memcpy(dest, &hv_crash_asm32, size);
> >> +
> >> + dest += size;
> >> + dest = (void *)round_up((ulong)dest, 16);
> >> + tramp = (struct hv_crash_tramp_data *)dest;
> >> +
> >> + /* see MAX_ASID_AVAILABLE in tlb.c: "PCID 0 is reserved for use by
> >> + * non-PCID-aware users". Build cr3 with pcid 0
> >> + */
> >> + tramp->tramp32_cr3 = __sme_pa(hv_crash_ptpgs[0]);
> >> +
> >> + /* Note, when restoring X86_CR4_PCIDE, cr3[11:0] must be zero */
> >> + tramp->kernel_cr3 = __sme_pa(init_mm.pgd);
> >> +
> >> + tramp->gdtr32.limit = sizeof(struct hv_crash_tramp_gdt);
> >> + tramp->gdtr32.address = trampoline_pa +
> >> + (ulong)&tramp->tramp_gdt - trampoline_va;
> >> +
> >> + /* base:0 limit:0xfffff type:b dpl:0 P:1 L:1 D:0 avl:0 G:1 */
> >> + tramp->tramp_gdt.cs64 = 0x00af9a000000ffff;
> >> +
> >> + tramp->cs_jmptgt.csval = 0x8;
> >> + offs = (ulong)&hv_crash_asm64_lbl - (ulong)&hv_crash_asm32;
> >> + tramp->cs_jmptgt.address = trampoline_pa + offs;
> >> +
> >> + tramp->c_entry_addr = (u64)&hv_crash_c_entry;
> >> +
> >> + devirt_cr3arg = trampoline_pa + (ulong)dest - trampoline_va;
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +/*
> >> + * Build 32bit trampoline page table for transition from protected mode
> >> + * non-paging to long-mode paging. This transition needs pagetables below 4G.
> >> + */
> >> +static void hv_crash_build_tramp_pt(void)
> >> +{
> >> + p4d_t *p4d;
> >> + pud_t *pud;
> >> + pmd_t *pmd;
> >> + pte_t *pte;
> >> + u64 pa, addr = trampoline_pa;
> >> +
> >> + p4d = hv_crash_ptpgs[0] + pgd_index(addr) * sizeof(p4d);
> >> + pa = virt_to_phys(hv_crash_ptpgs[1]);
> >> + set_p4d(p4d, __p4d(_PAGE_TABLE | pa));
> >> + p4d->p4d &= ~(_PAGE_NX); /* disable no execute */
> >> +
> >> + pud = hv_crash_ptpgs[1] + pud_index(addr) * sizeof(pud);
> >> + pa = virt_to_phys(hv_crash_ptpgs[2]);
> >> + set_pud(pud, __pud(_PAGE_TABLE | pa));
> >> +
> >> + pmd = hv_crash_ptpgs[2] + pmd_index(addr) * sizeof(pmd);
> >> + pa = virt_to_phys(hv_crash_ptpgs[3]);
> >> + set_pmd(pmd, __pmd(_PAGE_TABLE | pa));
> >> +
> >> + pte = hv_crash_ptpgs[3] + pte_index(addr) * sizeof(pte);
> >> + set_pte(pte, pfn_pte(addr >> PAGE_SHIFT, PAGE_KERNEL_EXEC));
> >> +}
> >> +
> >> +/*
> >> + * Setup trampoline for devirtualization:
> >> + * - a page below 4G, ie 32bit addr containing asm glue code that mshv jmps to
> >> + * in protected mode.
> >> + * - 4 pages for a temporary page table that asm code uses to turn paging on
> >> + * - a temporary gdt to use in the compat mode.
> >> + *
> >> + * Returns: 0 on success
> >> + */
> >> +static int hv_crash_trampoline_setup(void)
> >> +{
> >> + int i, rc, order;
> >> + struct page *page;
> >> + u64 trampoline_va;
> >> + gfp_t flags32 = GFP_KERNEL | GFP_DMA32 | __GFP_ZERO;
> >> +
> >> + /* page for 32bit trampoline assembly code + hv_crash_tramp_data */
> >> + page = alloc_page(flags32);
> >> + if (page == NULL) {
> >> + pr_err("%s: failed to alloc asm stub page\n", __func__);
> >> + return -1;
> >> + }
> >> +
> >> + trampoline_va = (u64)page_to_virt(page);
> >> + trampoline_pa = (u32)page_to_phys(page);
> >> +
> >> + order = 2; /* alloc 2^2 pages */
> >> + page = alloc_pages(flags32, order);
> >> + if (page == NULL) {
> >> + pr_err("%s: failed to alloc pt pages\n", __func__);
> >> + free_page(trampoline_va);
> >> + return -1;
> >> + }
> >> +
> >> + for (i = 0; i < 4; i++, page++)
> >> + hv_crash_ptpgs[i] = page_to_virt(page);
> >> +
> >> + hv_crash_build_tramp_pt();
> >> +
> >> + rc = hv_crash_setup_trampdata(trampoline_va);
> >> + if (rc)
> >> + goto errout;
> >> +
> >> + return 0;
> >> +
> >> +errout:
> >> + free_page(trampoline_va);
> >> + free_pages((ulong)hv_crash_ptpgs[0], order);
> >> +
> >> + return rc;
> >> +}
> >> +
> >> +/* Setup for kdump kexec to collect hypervisor ram when running as mshv root */
> >> +void hv_root_crash_init(void)
> >> +{
> >> + int rc;
> >> + struct hv_input_get_system_property *input;
> >> + struct hv_output_get_system_property *output;
> >> + unsigned long flags;
> >> + u64 status;
> >> + union hv_pfn_range cda_info;
> >> +
> >> + if (pgtable_l5_enabled()) {
> >> + pr_err("Hyper-V: crash dump not yet supported on 5level PTs\n");
> >> + return;
> >> + }
> >> +
> >> + rc = register_nmi_handler(NMI_LOCAL, hv_crash_nmi_local, NMI_FLAG_FIRST,
> >> + "hv_crash_nmi");
> >> + if (rc) {
> >> + pr_err("Hyper-V: failed to register crash nmi handler\n");
> >> + return;
> >> + }
> >> +
> >> + local_irq_save(flags);
> >> + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
> >> + output = *this_cpu_ptr(hyperv_pcpu_output_arg);
> >> +
> >> + memset(input, 0, sizeof(*input));
> >> + memset(output, 0, sizeof(*output));
> >
> > Why zero the output area? This is one of those hypercall things that we're
> > inconsistent about. A few hypercall call sites zero the output area, and it's
> > not clear why they do. Hyper-V should be responsible for properly filling in
> > the output area. Linux should not need to do this zero'ing, unless there's some
> > known bug in Hyper-V for certain hypercalls, in which case there should be
> > a code comment stating "why".
>
> for the same reason sometimes you see char *p = NULL, either leftover
> code or someone was debugging or just copy and paste. this is just copy
> paste. i agree in general that we don't need to clear it at all, in fact,
> i'd like to remove them all! but i also understand people with different
> skills and junior members find it easier to debug, and also we were in
> early product development. for that reason, it doesn't have to be
> consistent either, if some complex hypercalls are failing repeatedly,
> just for ease of debug, one might leave it there temporarily. but
> now that things are stable, i think we should just remove them all and
> get used to a bit more inconvenient debugging...
I see your point about debugging, but on balance I agree that they
should all be removed. If there's some debug case, add it back
temporarily to debug, but leave upstream without it. The zero'ing is
also unnecessary code in the interrupt disabled window, which you
have expressed concern about in a different thread.
>
> >> + input->property_id = HV_SYSTEM_PROPERTY_CRASHDUMPAREA;
> >> +
> >> + status = hv_do_hypercall(HVCALL_GET_SYSTEM_PROPERTY, input, output);
> >> + cda_info.as_uint64 = output->hv_cda_info.as_uint64;
> >> + local_irq_restore(flags);
> >> +
> >> + if (!hv_result_success(status)) {
> >> + pr_err("Hyper-V: %s: property:%d %s\n", __func__,
> >> + input->property_id, hv_result_to_string(status));
> >> + goto err_out;
> >> + }
> >> +
> >> + if (cda_info.base_pfn == 0) {
> >> + pr_err("Hyper-V: hypervisor crash dump area pfn is 0\n");
> >> + goto err_out;
> >> + }
> >> +
> >> + hv_cda = phys_to_virt(cda_info.base_pfn << PAGE_SHIFT);
> >
> > Use HV_HYP_PAGE_SHIFT, since PFNs provided by Hyper-V are always in
> > terms of the Hyper-V page size, which isn't necessarily the guest page size.
> > Yes, on x86 there's no difference, but for future robustness ....
>
> i don't know about guests, but we won't even boot if dom0 pg size
> didn't match.. but easier to change than to make the case..
FWIW, a normal Linux guest on ARM64 works just fine with a page
size of 16K or 64K, even though the underlying Hyper-V page size
is only 4K. That's why we have HV_HYP_PAGE_SHIFT and related in
the first place. Using it properly really matters for normal guests.
(Having the guest page size smaller than the Hyper-V page size
does *not* work, but there are no such use cases.)
Even on ARM64, I know the root partition page size is required to
match the Hyper-V page size. But using HV_HYP_PAGE_SIZE is
still appropriate just to not leave code that will go wrong if the
match requirement should ever change.
>
> >> +
> >> + rc = hv_crash_trampoline_setup();
> >> + if (rc)
> >> + goto err_out;
> >> +
> >> + smp_ops.crash_stop_other_cpus = hv_crash_stop_other_cpus;
> >> +
> >> + crash_kexec_post_notifiers = true;
> >> + hv_crash_enabled = 1;
> >> + pr_info("Hyper-V: linux and hv kdump support enabled\n");
> >
> > This message and the message below aren't consistent. One refers
> > to "hv kdump" and the other to "hyp kdump".
>
> >> +
> >> + return;
> >> +
> >> +err_out:
> >> + unregister_nmi_handler(NMI_LOCAL, "hv_crash_nmi");
> >> + pr_err("Hyper-V: only linux (but not hyp) kdump support enabled\n");
> >> +}
> >> --
> >> 2.36.1.vfs.0.0
> >>
> >
^ permalink raw reply
* RE: [PATCH v1 4/6] x86/hyperv: Add trampoline asm code to transition from hypervisor
From: Michael Kelley @ 2025-09-18 23:52 UTC (permalink / raw)
To: Mukesh R, linux-hyperv@vger.kernel.org,
linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org
Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org,
decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com,
bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org,
hpa@zytor.com, arnd@arndb.de
In-Reply-To: <79f5d0ac-0b3e-70fc-2cbe-8a2352642746@linux.microsoft.com>
From: Mukesh R <mrathor@linux.microsoft.com> Sent: Tuesday, September 16, 2025 2:31 PM
>
> On 9/15/25 10:55, Michael Kelley wrote:
> > From: Mukesh Rathor <mrathor@linux.microsoft.com> Sent: Tuesday, September 9, 2025 5:10 PM
> >>
> >> Introduce a small asm stub to transition from the hypervisor to linux
> >
> > I'd argue for capitalizing "Linux" here and in other places in commit
> > text and code comments throughout this patch set.
>
> I'd argue against it. A quick grep indicates it is a common practice,
> and in the code world goes easy on the eyes :).
I'll offer a final comment on this topic, and then let it be. There's
a history of Greg K-H, Marc Zyngier, Boris Petkov, Sean Christopherson,
and other maintainers giving comments to use the capitalized form
of "Linux", "MSR", "RAM", etc. See:
https://lore.kernel.org/lkml/Y+4WHGNdWTZ5Hc6Y@kroah.com/
https://lore.kernel.org/lkml/86o7u0dqzj.wl-maz@kernel.org/
https://lore.kernel.org/lkml/408e68d0-1ae1-6d56-d008-61de14214326@linaro.org/
https://lore.kernel.org/lkml/20250819215304.GMaKTyQBWi6YzqZ0bW@fat_crate.local/
https://lore.kernel.org/lkml/Y0CAHch5UR2Lp0tU@google.com/
https://lore.kernel.org/lkml/20240126214336.GA453589@bhelgaas/
https://lore.kernel.org/lkml/20161117155543.vg3domfqm3bhp4f7@pd.tnic/
>
> >> upon devirtualization.
> >
> > In this patch and subsequent patches, you've used the phrase "upon
> > devirtualization", which seems a little vague to me. Does this mean
> > "when devirtualization is complete" or perhaps "when the hypervisor
> > completes devirtualization"? Since there's no spec on any of this,
> > being as precise as possible will help future readers.
>
> since control comes back to linux at the callback here, i fail to
> understand what is vague about it. when hyp completes devirt,
> devirt is complete.
To me, the word "upon" is less precise than just "after". In temporal
contexts, "upon" might mean "at the same time as" or it might mean
"immediately after". I wrote this comment as I was trying to figure out
how the entire devirtualization process works. Eventually it became clear
and the ambiguity was resolved, but initially I was uncertain. See some
broader thoughts in my reply on Patch 5 of the series.
>
> >>
> >> At a high level, during panic of either the hypervisor or the dom0 (aka
> >> root), the nmi handler asks hypervisor to devirtualize.
> >
> > Suggest:
> >
> > At a high level, during panic of either the hypervisor or Linux running
> > in dom0 (a.k.a. the root partition), the Linux NMI handler asks the
> > hypervisor to devirtualize.
> >
> >> As part of that,
> >> the arguments include an entry point to return back to linux. This asm
> >> stub implements that entry point.
> >>
> >> The stub is entered in protected mode, uses temporary gdt and page table
> >> to enable long mode and get to kernel entry point which then restores full
> >> kernel context to resume execution to kexec.
> >>
> >> Signed-off-by: Mukesh Rathor <mrathor@linux.microsoft.com>
> >> ---
> >> arch/x86/hyperv/hv_trampoline.S | 105 ++++++++++++++++++++++++++++++++
> >> 1 file changed, 105 insertions(+)
> >> create mode 100644 arch/x86/hyperv/hv_trampoline.S
> >>
> >> diff --git a/arch/x86/hyperv/hv_trampoline.S b/arch/x86/hyperv/hv_trampoline.S
> >> new file mode 100644
> >> index 000000000000..27a755401a42
> >> --- /dev/null
> >> +++ b/arch/x86/hyperv/hv_trampoline.S
> >> @@ -0,0 +1,105 @@
> >> +/* SPDX-License-Identifier: GPL-2.0-only */
> >> +/*
> >> + * X86 specific Hyper-V kdump/crash related code.
> >
> > Add a qualification that this is for root partition only, and not for
> > general guests?
>
> i don't think it is needed, it would be odd for guests to collect hyp
> core. besides makefile/kconfig shows this is root vm only
>
> >> + *
> >> + * Copyright (C) 2025, Microsoft, Inc.
> >> + *
> >> + */
> >> +#include <linux/linkage.h>
> >> +#include <asm/alternative.h>
> >> +#include <asm/msr.h>
> >> +#include <asm/processor-flags.h>
> >> +#include <asm/nospec-branch.h>
> >> +
> >> +/*
> >> + * void noreturn hv_crash_asm32(arg1)
> >> + * arg1 == edi == 32bit PA of struct hv_crash_trdata
> >
> > I think this is "struct hv_crash_tramp_data".
>
> correct
>
> >> + *
> >> + * The hypervisor jumps here upon devirtualization in protected mode. This
> >> + * code gets copied to a page in the low 4G ie, 32bit space so it can run
> >> + * in the protected mode. Hence we cannot use any compile/link time offsets or
> >> + * addresses. It restores long mode via temporary gdt and page tables and
> >> + * eventually jumps to kernel code entry at HV_CRASHDATA_OFFS_C_entry.
> >> + *
> >> + * PreCondition (ie, Hypervisor call back ABI):
> >> + * o CR0 is set to 0x0021: PE(prot mode) and NE are set, paging is disabled
> >> + * o CR4 is set to 0x0
> >> + * o IA32_EFER is set to 0x901 (SCE and NXE are set)
> >> + * o EDI is set to the Arg passed to HVCALL_DISABLE_HYP_EX.
> >> + * o CS, DS, ES, FS, GS are all initialized with a base of 0 and limit 0xFFFF
> >> + * o IDTR, TR and GDTR are initialized with a base of 0 and limit of 0xFFFF
> >> + * o LDTR is initialized as invalid (limit of 0)
> >> + * o MSR PAT is power on default.
> >> + * o Other state/registers are cleared. All TLBs flushed.
> >
> > Clarification about "Other state/registers are cleared": What about
> > processor features that Linux may have enabled or disabled during its
> > initial boot? Are those still in the states Linux set? Or are they reset to
> > power-on defaults? For example, if Linux enabled x2apic, is x2apic
> > still enabled when the stub is entered?
>
> correct, if linux set x2apic, x2apic would still be enabled.
>
> >> + *
> >> + * See Intel SDM 10.8.5
> >
> > Hmmm. I downloaded the latest combined SDM, and section 10.8.5
> > in Volume 3A is about Microcode Update Resources, which doesn't
> > seem applicable here. Other volumes don't have a section 10.8.5.
>
> google ai found it right away upon searching: intel sdm 10.8.5 ia-32e
Unfortunately, Intel doesn't necessarily maintain the section numbering
across revisions of the SDM. This web page:
https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html
has a link to download the "Combined Volume Set", and currently provides
the version dated June 2025. The section "Initializing IA-32e Mode" is
numbered 11.8.5. The December 2024 version has the same 11.8.5
numbering. Are you finding an older version?
Presumably the section title is less likely to change unless Intel does a
major rewrite. So something like this would be more durable:
* See Intel SDM Volume 3A section "Initializing IA-32e Mode" (numbered
11.8.5 in the June 2025 version)
>
> >> + */
> >> +
> >> +#define HV_CRASHDATA_OFFS_TRAMPCR3 0x0 /* 0 */
> >> +#define HV_CRASHDATA_OFFS_KERNCR3 0x8 /* 8 */
> >> +#define HV_CRASHDATA_OFFS_GDTRLIMIT 0x12 /* 18 */
> >> +#define HV_CRASHDATA_OFFS_CS_JMPTGT 0x28 /* 40 */
> >> +#define HV_CRASHDATA_OFFS_C_entry 0x30 /* 48 */
> >
> > It seems like these offsets should go in a #include file along
> > with the definition of struct hv_crash_tramp_data. Then the
> > BUILD_BUG_ON() calls in hv_crash_setup_trampdata() could
> > check against these symbolic names instead of hardcoding
> > numbers that must match these.
>
> yeah, that works too and was the first cut. but given the small
> number of these, and that they are not used/needed anywhere else,
> and that they will almost never change, creating another tiny header
> in a non-driver directory didn't seem worth it.. but i could go
> either way.
>
> >> +#define HV_CRASHDATA_TRAMPOLINE_CS 0x8
> >
> > This #define isn't used anywhere.
>
> removed
>
> >> +
> >> + .text
> >> + .code32
> >> +
> >> +SYM_CODE_START(hv_crash_asm32)
> >> + UNWIND_HINT_UNDEFINED
> >> + ANNOTATE_NOENDBR
> >
> > No ENDBR here, presumably because this function is entered via other
> > than an indirect CALL or JMP instruction. Right?
> >
> >> + movl $X86_CR4_PAE, %ecx
> >> + movl %ecx, %cr4
> >> +
> >> + movl %edi, %ebx
> >> + add $HV_CRASHDATA_OFFS_TRAMPCR3, %ebx
> >> + movl %cs:(%ebx), %eax
> >> + movl %eax, %cr3
> >> +
> >> + # Setup EFER for long mode now.
> >> + movl $MSR_EFER, %ecx
> >> + rdmsr
> >> + btsl $_EFER_LME, %eax
> >> + wrmsr
> >> +
> >> + # Turn paging on using the temp 32bit trampoline page table.
> >> + movl %cr0, %eax
> >> + orl $(X86_CR0_PG), %eax
> >> + movl %eax, %cr0
> >> +
> >> + /* since kernel cr3 could be above 4G, we need to be in the long mode
> >> + * before we can load 64bits of the kernel cr3. We use a temp gdt for
> >> + * that with CS.L=1 and CS.D=0 */
> >> + mov %edi, %eax
> >> + add $HV_CRASHDATA_OFFS_GDTRLIMIT, %eax
> >> + lgdtl %cs:(%eax)
> >> +
> >> + /* not done yet, restore CS now to switch to CS.L=1 */
> >> + mov %edi, %eax
> >> + add $HV_CRASHDATA_OFFS_CS_JMPTGT, %eax
> >> + ljmp %cs:*(%eax)
> >> +SYM_CODE_END(hv_crash_asm32)
> >> +
> >> + /* we now run in full 64bit IA32-e long mode, CS.L=1 and CS.D=0 */
> >> + .code64
> >> + .balign 8
> >> +SYM_CODE_START(hv_crash_asm64)
> >> + UNWIND_HINT_UNDEFINED
> >> + ANNOTATE_NOENDBR
> >
> > But this *is* entered via an indirect JMP, right? So back to my
> > earlier question about the state of processor feature enablement.
> > If Linux enabled IBT, is it still enabled after devirtualization and
> > the hypervisor invokes this entry point? Linux guests on Hyper-V
> > have historically not enabled IBT, but patches that enable it are
> > now in linux-next, and will go into the 6.18 kernel. So maybe
> > this needs an ENDBR64.
>
> IBT would be disabled in the transition here.... so doesn't really
> matter. ENDBR ok too..
So does Hyper-V explicitly disable IBT before making the callback?
Or is the IBT disabling somehow a processor side effect of going back
to protected mode? I don't see anything in the SDM about the latter.
Not having a Hyper-V spec for all this is frustrating ...
Doing the ENDBR64 here might be safer in the long run in case
we ever do end up here with IBT enabled.
>
> >> +SYM_INNER_LABEL(hv_crash_asm64_lbl, SYM_L_GLOBAL)
> >> + /* restore kernel page tables so we can jump to kernel code */
> >> + mov %edi, %eax
> >> + add $HV_CRASHDATA_OFFS_KERNCR3, %eax
> >> + movq %cs:(%eax), %rbx
> >> + movq %rbx, %cr3
> >> +
> >> + mov %edi, %eax
> >> + add $HV_CRASHDATA_OFFS_C_entry, %eax
> >> + movq %cs:(%eax), %rbx
> >> + ANNOTATE_RETPOLINE_SAFE
> >> + jmp *%rbx
> >> +
> >> + int $3
> >> +
> >> +SYM_INNER_LABEL(hv_crash_asm_end, SYM_L_GLOBAL)
> >> +SYM_CODE_END(hv_crash_asm64)
> >> --
> >> 2.36.1.vfs.0.0
> >>
> >
^ permalink raw reply
* RE: [PATCH v1 3/6] hyperv: Add definitions for hypervisor crash dump support
From: Michael Kelley @ 2025-09-18 23:52 UTC (permalink / raw)
To: Mukesh R, linux-hyperv@vger.kernel.org,
linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org
Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org,
decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com,
bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org,
hpa@zytor.com, arnd@arndb.de
In-Reply-To: <cfcbdf49-33e6-685c-daed-4dd8f1523c49@linux.microsoft.com>
From: Mukesh R <mrathor@linux.microsoft.com> Sent: Monday, September 15, 2025 6:15 PM
>
> On 9/15/25 10:54, Michael Kelley wrote:
> > From: Mukesh Rathor <mrathor@linux.microsoft.com> Sent: Tuesday, September 9, 2025 5:10 PM
> >>
> >> Add data structures for hypervisor crash dump support to the hypervisor
> >> host ABI header file. Details of their usages are in subsequent commits.
> >>
> >> Signed-off-by: Mukesh Rathor <mrathor@linux.microsoft.com>
> >> ---
> >> include/hyperv/hvhdk_mini.h | 55 +++++++++++++++++++++++++++++++++++++
> >> 1 file changed, 55 insertions(+)
> >>
> >> diff --git a/include/hyperv/hvhdk_mini.h b/include/hyperv/hvhdk_mini.h
> >> index 858f6a3925b3..ad9a8048fb4e 100644
> >> --- a/include/hyperv/hvhdk_mini.h
> >> +++ b/include/hyperv/hvhdk_mini.h
> >>
[snip]
> >> +enum hv_crashdump_action {
> >> + HV_CRASHDUMP_NONE = 0,
> >> + HV_CRASHDUMP_SUSPEND_ALL_VPS,
> >> + HV_CRASHDUMP_PREPARE_FOR_STATE_SAVE,
> >> + HV_CRASHDUMP_STATE_SAVED,
> >> + HV_CRASHDUMP_ENTRY,
> >> +};
> >
> > Nit: Since these values are part of the ABI, it's probably better
> > to assign explicit values to each enum member in order to
> > ward off any mistaken reordering or additions in the middle
> > of the list.
>
> No, like I have mentioned in the past, we are mirroring hyp headers
> with the eventual goal of just consuming from there directly.
> Each change in ABI header is very carefully examined, we now have
> a process for it.
>
Acknowledged. I keep wanting to tighten up the ABI specification,
and sometimes forget that there are constraints on doing so.
Michael
^ permalink raw reply
* Re: [PATCH 5/5] x86/Hyper-V: Add Hyper-V specific hvcall to set backing page
From: Borislav Petkov @ 2025-09-18 19:59 UTC (permalink / raw)
To: Tianyu Lan
Cc: kys, haiyangz, wei.liu, decui, tglx, mingo, dave.hansen, x86, hpa,
arnd, Neeraj.Upadhyay, tiala, romank, linux-arch, linux-hyperv,
linux-kernel, Michael Kelley
In-Reply-To: <CAMvTesBwkFr7VYoEYq5hc7NJi5xdiz037bmzoaEV2eG__h-kTg@mail.gmail.com>
On Fri, Sep 19, 2025 at 01:11:02AM +0800, Tianyu Lan wrote:
> Could I move the check into savic_register_gpa() or add a stub function
> to check guest runs on Hyper-V or not and then call associated function
> to register APIC backing page?
You probably should do
static struct apic apic_x2apic_savic_hyperv
and copy the apic_x2apic_savic contents into it and overwrite the .setup
function with your variant.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
^ permalink raw reply
* Re: [PATCH v3 5/5] mshv: Introduce new hypercall to map stats page for L1VH partitions
From: Stanislav Kinsburskii @ 2025-09-18 19:53 UTC (permalink / raw)
To: Nuno Das Neves
Cc: linux-hyperv, linux-kernel, prapal, easwar.hariharan, tiala,
anirudh, paekkaladevi, kys, haiyangz, wei.liu, decui, Jinank Jain
In-Reply-To: <1758066262-15477-6-git-send-email-nunodasneves@linux.microsoft.com>
On Tue, Sep 16, 2025 at 04:44:22PM -0700, Nuno Das Neves wrote:
> From: Jinank Jain <jinankjain@linux.microsoft.com>
>
<snip>
> +static int hv_call_map_stats_page2(enum hv_stats_object_type type,
> + const union hv_stats_object_identity *identity,
> + u64 map_location)
> +{
> + unsigned long flags;
> + struct hv_input_map_stats_page2 *input;
> + u64 status;
> + int ret;
> +
> + if (!map_location || !mshv_use_overlay_gpfn())
> + return -EINVAL;
> +
> + do {
> + local_irq_save(flags);
> + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
> +
> + memset(input, 0, sizeof(*input));
> + input->type = type;
> + input->identity = *identity;
> + input->map_location = map_location;
> +
> + status = hv_do_hypercall(HVCALL_MAP_STATS_PAGE2, input, NULL);
> +
> + local_irq_restore(flags);
> + if (hv_result(status) != HV_STATUS_INSUFFICIENT_MEMORY) {
> + if (hv_result_success(status))
> + break;
> + hv_status_debug(status, "\n");
It looks more natural to check for success first and break the loop, and
only then handle errors.
Maybe even set ret for both success and error messages and break and
handle only the unsufficient memory status.
> @@ -865,6 +931,19 @@ int hv_call_unmap_stat_page(enum hv_stats_object_type type,
> return hv_result_to_errno(status);
> }
>
> +int hv_unmap_stats_page(enum hv_stats_object_type type, void *page_addr,
> + const union hv_stats_object_identity *identity)
> +{
Should this function be type of void?
Thanks,
Stanislav
^ permalink raw reply
* Re: [PATCH v3 4/5] mshv: Allocate vp state page for HVCALL_MAP_VP_STATE_PAGE on L1VH
From: Stanislav Kinsburskii @ 2025-09-18 18:58 UTC (permalink / raw)
To: Nuno Das Neves
Cc: linux-hyperv, linux-kernel, prapal, easwar.hariharan, tiala,
anirudh, paekkaladevi, kys, haiyangz, wei.liu, decui, Jinank Jain
In-Reply-To: <1758066262-15477-5-git-send-email-nunodasneves@linux.microsoft.com>
On Tue, Sep 16, 2025 at 04:44:21PM -0700, Nuno Das Neves wrote:
> From: Jinank Jain <jinankjain@linux.microsoft.com>
>
Reviewed-by: Stanislav Kinsburskii <skinsburskii@linux.microsoft.com>
> Introduce mshv_use_overlay_gpfn() to check if a page needs to be
> allocated and passed to the hypervisor to map VP state pages. This is
> only needed on L1VH, and only on some (newer) versions of the
> hypervisor, hence the need to check vmm_capabilities.
>
> Introduce functions hv_map/unmap_vp_state_page() to handle the
> allocation and freeing.
>
> Signed-off-by: Jinank Jain <jinankjain@linux.microsoft.com>
> Signed-off-by: Nuno Das Neves <nunodasneves@linux.microsoft.com>
> Reviewed-by: Praveen K Paladugu <prapal@linux.microsoft.com>
> Reviewed-by: Easwar Hariharan <easwar.hariharan@linux.microsoft.com>
> ---
> drivers/hv/mshv_root.h | 11 ++---
> drivers/hv/mshv_root_hv_call.c | 61 ++++++++++++++++++++++++---
> drivers/hv/mshv_root_main.c | 76 +++++++++++++++++-----------------
> 3 files changed, 98 insertions(+), 50 deletions(-)
>
> diff --git a/drivers/hv/mshv_root.h b/drivers/hv/mshv_root.h
> index 0cb1e2589fe1..dbe2d1d0b22f 100644
> --- a/drivers/hv/mshv_root.h
> +++ b/drivers/hv/mshv_root.h
> @@ -279,11 +279,12 @@ int hv_call_set_vp_state(u32 vp_index, u64 partition_id,
> /* Choose between pages and bytes */
> struct hv_vp_state_data state_data, u64 page_count,
> struct page **pages, u32 num_bytes, u8 *bytes);
> -int hv_call_map_vp_state_page(u64 partition_id, u32 vp_index, u32 type,
> - union hv_input_vtl input_vtl,
> - struct page **state_page);
> -int hv_call_unmap_vp_state_page(u64 partition_id, u32 vp_index, u32 type,
> - union hv_input_vtl input_vtl);
> +int hv_map_vp_state_page(u64 partition_id, u32 vp_index, u32 type,
> + union hv_input_vtl input_vtl,
> + struct page **state_page);
> +int hv_unmap_vp_state_page(u64 partition_id, u32 vp_index, u32 type,
> + struct page *state_page,
> + union hv_input_vtl input_vtl);
> int hv_call_create_port(u64 port_partition_id, union hv_port_id port_id,
> u64 connection_partition_id, struct hv_port_info *port_info,
> u8 port_vtl, u8 min_connection_vtl, int node);
> diff --git a/drivers/hv/mshv_root_hv_call.c b/drivers/hv/mshv_root_hv_call.c
> index 3fd3cce23f69..98c6278ff151 100644
> --- a/drivers/hv/mshv_root_hv_call.c
> +++ b/drivers/hv/mshv_root_hv_call.c
> @@ -526,9 +526,9 @@ int hv_call_set_vp_state(u32 vp_index, u64 partition_id,
> return ret;
> }
>
> -int hv_call_map_vp_state_page(u64 partition_id, u32 vp_index, u32 type,
> - union hv_input_vtl input_vtl,
> - struct page **state_page)
> +static int hv_call_map_vp_state_page(u64 partition_id, u32 vp_index, u32 type,
> + union hv_input_vtl input_vtl,
> + struct page **state_page)
> {
> struct hv_input_map_vp_state_page *input;
> struct hv_output_map_vp_state_page *output;
> @@ -547,7 +547,14 @@ int hv_call_map_vp_state_page(u64 partition_id, u32 vp_index, u32 type,
> input->type = type;
> input->input_vtl = input_vtl;
>
> - status = hv_do_hypercall(HVCALL_MAP_VP_STATE_PAGE, input, output);
> + if (*state_page) {
> + input->flags.map_location_provided = 1;
> + input->requested_map_location =
> + page_to_pfn(*state_page);
> + }
> +
> + status = hv_do_hypercall(HVCALL_MAP_VP_STATE_PAGE, input,
> + output);
>
> if (hv_result(status) != HV_STATUS_INSUFFICIENT_MEMORY) {
> if (hv_result_success(status))
> @@ -565,8 +572,39 @@ int hv_call_map_vp_state_page(u64 partition_id, u32 vp_index, u32 type,
> return ret;
> }
>
> -int hv_call_unmap_vp_state_page(u64 partition_id, u32 vp_index, u32 type,
> - union hv_input_vtl input_vtl)
> +static bool mshv_use_overlay_gpfn(void)
> +{
> + return hv_l1vh_partition() &&
> + mshv_root.vmm_caps.vmm_can_provide_overlay_gpfn;
> +}
> +
> +int hv_map_vp_state_page(u64 partition_id, u32 vp_index, u32 type,
> + union hv_input_vtl input_vtl,
> + struct page **state_page)
> +{
> + int ret = 0;
> + struct page *allocated_page = NULL;
> +
> + if (mshv_use_overlay_gpfn()) {
> + allocated_page = alloc_page(GFP_KERNEL);
> + if (!allocated_page)
> + return -ENOMEM;
> + *state_page = allocated_page;
> + } else {
> + *state_page = NULL;
> + }
> +
> + ret = hv_call_map_vp_state_page(partition_id, vp_index, type, input_vtl,
> + state_page);
> +
> + if (ret && allocated_page)
> + __free_page(allocated_page);
> +
> + return ret;
> +}
> +
> +static int hv_call_unmap_vp_state_page(u64 partition_id, u32 vp_index, u32 type,
> + union hv_input_vtl input_vtl)
> {
> unsigned long flags;
> u64 status;
> @@ -590,6 +628,17 @@ int hv_call_unmap_vp_state_page(u64 partition_id, u32 vp_index, u32 type,
> return hv_result_to_errno(status);
> }
>
> +int hv_unmap_vp_state_page(u64 partition_id, u32 vp_index, u32 type,
> + struct page *state_page, union hv_input_vtl input_vtl)
> +{
> + int ret = hv_call_unmap_vp_state_page(partition_id, vp_index, type, input_vtl);
> +
> + if (mshv_use_overlay_gpfn() && state_page)
> + __free_page(state_page);
> +
> + return ret;
> +}
> +
> int hv_call_get_partition_property_ex(u64 partition_id, u64 property_code,
> u64 arg, void *property_value,
> size_t property_value_sz)
> diff --git a/drivers/hv/mshv_root_main.c b/drivers/hv/mshv_root_main.c
> index f7738cefbdf3..52f69eace9b9 100644
> --- a/drivers/hv/mshv_root_main.c
> +++ b/drivers/hv/mshv_root_main.c
> @@ -890,7 +890,7 @@ mshv_partition_ioctl_create_vp(struct mshv_partition *partition,
> {
> struct mshv_create_vp args;
> struct mshv_vp *vp;
> - struct page *intercept_message_page, *register_page, *ghcb_page;
> + struct page *intercept_msg_page, *register_page, *ghcb_page;
> void *stats_pages[2];
> long ret;
>
> @@ -908,28 +908,25 @@ mshv_partition_ioctl_create_vp(struct mshv_partition *partition,
> if (ret)
> return ret;
>
> - ret = hv_call_map_vp_state_page(partition->pt_id, args.vp_index,
> - HV_VP_STATE_PAGE_INTERCEPT_MESSAGE,
> - input_vtl_zero,
> - &intercept_message_page);
> + ret = hv_map_vp_state_page(partition->pt_id, args.vp_index,
> + HV_VP_STATE_PAGE_INTERCEPT_MESSAGE,
> + input_vtl_zero, &intercept_msg_page);
> if (ret)
> goto destroy_vp;
>
> if (!mshv_partition_encrypted(partition)) {
> - ret = hv_call_map_vp_state_page(partition->pt_id, args.vp_index,
> - HV_VP_STATE_PAGE_REGISTERS,
> - input_vtl_zero,
> - ®ister_page);
> + ret = hv_map_vp_state_page(partition->pt_id, args.vp_index,
> + HV_VP_STATE_PAGE_REGISTERS,
> + input_vtl_zero, ®ister_page);
> if (ret)
> goto unmap_intercept_message_page;
> }
>
> if (mshv_partition_encrypted(partition) &&
> is_ghcb_mapping_available()) {
> - ret = hv_call_map_vp_state_page(partition->pt_id, args.vp_index,
> - HV_VP_STATE_PAGE_GHCB,
> - input_vtl_normal,
> - &ghcb_page);
> + ret = hv_map_vp_state_page(partition->pt_id, args.vp_index,
> + HV_VP_STATE_PAGE_GHCB,
> + input_vtl_normal, &ghcb_page);
> if (ret)
> goto unmap_register_page;
> }
> @@ -960,7 +957,7 @@ mshv_partition_ioctl_create_vp(struct mshv_partition *partition,
> atomic64_set(&vp->run.vp_signaled_count, 0);
>
> vp->vp_index = args.vp_index;
> - vp->vp_intercept_msg_page = page_to_virt(intercept_message_page);
> + vp->vp_intercept_msg_page = page_to_virt(intercept_msg_page);
> if (!mshv_partition_encrypted(partition))
> vp->vp_register_page = page_to_virt(register_page);
>
> @@ -993,21 +990,19 @@ mshv_partition_ioctl_create_vp(struct mshv_partition *partition,
> if (hv_scheduler_type == HV_SCHEDULER_TYPE_ROOT)
> mshv_vp_stats_unmap(partition->pt_id, args.vp_index);
> unmap_ghcb_page:
> - if (mshv_partition_encrypted(partition) && is_ghcb_mapping_available()) {
> - hv_call_unmap_vp_state_page(partition->pt_id, args.vp_index,
> - HV_VP_STATE_PAGE_GHCB,
> - input_vtl_normal);
> - }
> + if (mshv_partition_encrypted(partition) && is_ghcb_mapping_available())
> + hv_unmap_vp_state_page(partition->pt_id, args.vp_index,
> + HV_VP_STATE_PAGE_GHCB, ghcb_page,
> + input_vtl_normal);
> unmap_register_page:
> - if (!mshv_partition_encrypted(partition)) {
> - hv_call_unmap_vp_state_page(partition->pt_id, args.vp_index,
> - HV_VP_STATE_PAGE_REGISTERS,
> - input_vtl_zero);
> - }
> + if (!mshv_partition_encrypted(partition))
> + hv_unmap_vp_state_page(partition->pt_id, args.vp_index,
> + HV_VP_STATE_PAGE_REGISTERS,
> + register_page, input_vtl_zero);
> unmap_intercept_message_page:
> - hv_call_unmap_vp_state_page(partition->pt_id, args.vp_index,
> - HV_VP_STATE_PAGE_INTERCEPT_MESSAGE,
> - input_vtl_zero);
> + hv_unmap_vp_state_page(partition->pt_id, args.vp_index,
> + HV_VP_STATE_PAGE_INTERCEPT_MESSAGE,
> + intercept_msg_page, input_vtl_zero);
> destroy_vp:
> hv_call_delete_vp(partition->pt_id, args.vp_index);
> return ret;
> @@ -1748,24 +1743,27 @@ static void destroy_partition(struct mshv_partition *partition)
> mshv_vp_stats_unmap(partition->pt_id, vp->vp_index);
>
> if (vp->vp_register_page) {
> - (void)hv_call_unmap_vp_state_page(partition->pt_id,
> - vp->vp_index,
> - HV_VP_STATE_PAGE_REGISTERS,
> - input_vtl_zero);
> + (void)hv_unmap_vp_state_page(partition->pt_id,
> + vp->vp_index,
> + HV_VP_STATE_PAGE_REGISTERS,
> + virt_to_page(vp->vp_register_page),
> + input_vtl_zero);
> vp->vp_register_page = NULL;
> }
>
> - (void)hv_call_unmap_vp_state_page(partition->pt_id,
> - vp->vp_index,
> - HV_VP_STATE_PAGE_INTERCEPT_MESSAGE,
> - input_vtl_zero);
> + (void)hv_unmap_vp_state_page(partition->pt_id,
> + vp->vp_index,
> + HV_VP_STATE_PAGE_INTERCEPT_MESSAGE,
> + virt_to_page(vp->vp_intercept_msg_page),
> + input_vtl_zero);
> vp->vp_intercept_msg_page = NULL;
>
> if (vp->vp_ghcb_page) {
> - (void)hv_call_unmap_vp_state_page(partition->pt_id,
> - vp->vp_index,
> - HV_VP_STATE_PAGE_GHCB,
> - input_vtl_normal);
> + (void)hv_unmap_vp_state_page(partition->pt_id,
> + vp->vp_index,
> + HV_VP_STATE_PAGE_GHCB,
> + virt_to_page(vp->vp_ghcb_page),
> + input_vtl_normal);
> vp->vp_ghcb_page = NULL;
> }
>
> --
> 2.34.1
>
^ permalink raw reply
* Re: [PATCH v3 3/5] mshv: Get the vmm capabilities offered by the hypervisor
From: Stanislav Kinsburskii @ 2025-09-18 18:50 UTC (permalink / raw)
To: Nuno Das Neves
Cc: linux-hyperv, linux-kernel, prapal, easwar.hariharan, tiala,
anirudh, paekkaladevi, kys, haiyangz, wei.liu, decui
In-Reply-To: <1758066262-15477-4-git-send-email-nunodasneves@linux.microsoft.com>
On Tue, Sep 16, 2025 at 04:44:20PM -0700, Nuno Das Neves wrote:
> From: Purna Pavan Chandra Aekkaladevi <paekkaladevi@linux.microsoft.com>
>
> Some hypervisor APIs are gated by feature bits in the
> "vmm capabilities" partition property. Store the capabilities on
> mshv_root module init, using HVCALL_GET_PARTITION_PROPERTY_EX.
>
> This is not supported on all hypervisors. In that case, just set the
> capabilities to 0 and proceed as normal.
>
> Signed-off-by: Purna Pavan Chandra Aekkaladevi <paekkaladevi@linux.microsoft.com>
> Signed-off-by: Nuno Das Neves <nunodasneves@linux.microsoft.com>
> Reviewed-by: Praveen K Paladugu <prapal@linux.microsoft.com>
> Reviewed-by: Easwar Hariharan <easwar.hariharan@linux.microsoft.com>
> Reviewed-by: Tianyu Lan <tiala@microsoft.com>
> ---
> drivers/hv/mshv_root.h | 1 +
> drivers/hv/mshv_root_main.c | 22 ++++++++++++++++++++++
> 2 files changed, 23 insertions(+)
>
> diff --git a/drivers/hv/mshv_root.h b/drivers/hv/mshv_root.h
> index 4aeb03bea6b6..0cb1e2589fe1 100644
> --- a/drivers/hv/mshv_root.h
> +++ b/drivers/hv/mshv_root.h
> @@ -178,6 +178,7 @@ struct mshv_root {
> struct hv_synic_pages __percpu *synic_pages;
> spinlock_t pt_ht_lock;
> DECLARE_HASHTABLE(pt_htable, MSHV_PARTITIONS_HASH_BITS);
> + struct hv_partition_property_vmm_capabilities vmm_caps;
> };
>
> /*
> diff --git a/drivers/hv/mshv_root_main.c b/drivers/hv/mshv_root_main.c
> index 24df47726363..f7738cefbdf3 100644
> --- a/drivers/hv/mshv_root_main.c
> +++ b/drivers/hv/mshv_root_main.c
> @@ -2201,6 +2201,26 @@ static int __init mshv_root_partition_init(struct device *dev)
> return err;
> }
>
> +static void mshv_init_vmm_caps(struct device *dev)
> +{
> + int ret;
nit: this is void function so ret looks redundant.
> +
> + memset(&mshv_root.vmm_caps, 0, sizeof(mshv_root.vmm_caps));
Zeroying is redundant as mshv_root is a statci variable.
> + ret = hv_call_get_partition_property_ex(HV_PARTITION_ID_SELF,
> + HV_PARTITION_PROPERTY_VMM_CAPABILITIES,
> + 0, &mshv_root.vmm_caps,
Also, we align "slow" hypercalls by PAGE_SIZE. Why is it fine to not do
it here?
Thanks,
Stanislav
> + sizeof(mshv_root.vmm_caps));
> +
> + /*
> + * HVCALL_GET_PARTITION_PROPERTY_EX or HV_PARTITION_PROPERTY_VMM_CAPABILITIES
> + * may not be supported. Leave them as 0 in that case.
> + */
> + if (ret)
> + dev_warn(dev, "Unable to get VMM capabilities\n");
> +
> + dev_dbg(dev, "vmm_caps=0x%llx\n", mshv_root.vmm_caps.as_uint64[0]);
> +}
> +
> static int __init mshv_parent_partition_init(void)
> {
> int ret;
> @@ -2253,6 +2273,8 @@ static int __init mshv_parent_partition_init(void)
> if (ret)
> goto remove_cpu_state;
>
> + mshv_init_vmm_caps(dev);
> +
> ret = mshv_irqfd_wq_init();
> if (ret)
> goto exit_partition;
> --
> 2.34.1
>
^ permalink raw reply
* Re: [PATCH v3 2/5] mshv: Add the HVCALL_GET_PARTITION_PROPERTY_EX hypercall
From: Stanislav Kinsburskii @ 2025-09-18 18:43 UTC (permalink / raw)
To: Nuno Das Neves
Cc: linux-hyperv, linux-kernel, prapal, easwar.hariharan, tiala,
anirudh, paekkaladevi, kys, haiyangz, wei.liu, decui
In-Reply-To: <1758066262-15477-3-git-send-email-nunodasneves@linux.microsoft.com>
On Tue, Sep 16, 2025 at 04:44:19PM -0700, Nuno Das Neves wrote:
<snip>
> diff --git a/include/hyperv/hvhdk.h b/include/hyperv/hvhdk.h
> index b4067ada02cf..b91358b9c929 100644
> --- a/include/hyperv/hvhdk.h
> +++ b/include/hyperv/hvhdk.h
> @@ -376,6 +376,46 @@ struct hv_input_set_partition_property {
> u64 property_value;
> } __packed;
>
> +union hv_partition_property_arg {
> + u64 as_uint64;
> + struct {
> + union {
> + u32 arg;
> + u32 vp_index;
> + };
> + u16 reserved0;
> + u8 reserved1;
> + u8 object_type;
> + };
> +} __packed;
Shouldn't the struct be "packed" instead?
> +
> +struct hv_input_get_partition_property_ex {
> + u64 partition_id;
> + u32 property_code; /* enum hv_partition_property_code */
> + u32 padding;
> + union {
> + union hv_partition_property_arg arg_data;
> + u64 arg;
> + };
> +} __packed;
> +
> +/*
> + * NOTE: Should use hv_input_set_partition_property_ex_header to compute this
> + * size, but hv_input_get_partition_property_ex is identical so it suffices
> + */
> +#define HV_PARTITION_PROPERTY_EX_MAX_VAR_SIZE \
> + (HV_HYP_PAGE_SIZE - sizeof(struct hv_input_get_partition_property_ex))
> +
> +union hv_partition_property_ex {
> + u8 buffer[HV_PARTITION_PROPERTY_EX_MAX_VAR_SIZE];
> + struct hv_partition_property_vmm_capabilities vmm_capabilities;
> + /* More fields to be filled in when needed */
> +} __packed;
Packing a union is redundant.
Thanks,
Stanislav
^ permalink raw reply
* Re: [PATCH v3 1/5] mshv: Only map vp->vp_stats_pages if on root scheduler
From: Stanislav Kinsburskii @ 2025-09-18 18:38 UTC (permalink / raw)
To: Nuno Das Neves
Cc: linux-hyperv, linux-kernel, prapal, easwar.hariharan, tiala,
anirudh, paekkaladevi, kys, haiyangz, wei.liu, decui
In-Reply-To: <1758066262-15477-2-git-send-email-nunodasneves@linux.microsoft.com>
On Tue, Sep 16, 2025 at 04:44:18PM -0700, Nuno Das Neves wrote:
> This mapping is only used for checking if the dispatch thread is
> blocked. This is only relevant for the root scheduler, so check the
> scheduler type to determine whether to map/unmap these pages, instead of
> the current check, which is incorrect.
>
> Signed-off-by: Nuno Das Neves <nunodasneves@linux.microsoft.com>
> Reviewed-by: Anirudh Rayabharam <anirudh@anirudhrb.com>
> Reviewed-by: Praveen K Paladugu <prapal@linux.microsoft.com>
> Reviewed-by: Easwar Hariharan <easwar.hariharan@linux.microsoft.com>
> Reviewed-by: Tianyu Lan <tiala@microsoft.com>
> ---
> drivers/hv/mshv_root_main.c | 12 ++++++++----
> 1 file changed, 8 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/hv/mshv_root_main.c b/drivers/hv/mshv_root_main.c
> index e3b2bd417c46..24df47726363 100644
> --- a/drivers/hv/mshv_root_main.c
> +++ b/drivers/hv/mshv_root_main.c
> @@ -934,7 +934,11 @@ mshv_partition_ioctl_create_vp(struct mshv_partition *partition,
> goto unmap_register_page;
> }
>
> - if (hv_parent_partition()) {
> + /*
> + * This mapping of the stats page is for detecting if dispatch thread
> + * is blocked - only relevant for root scheduler
> + */
> + if (hv_scheduler_type == HV_SCHEDULER_TYPE_ROOT) {
> ret = mshv_vp_stats_map(partition->pt_id, args.vp_index,
> stats_pages);
> if (ret)
> @@ -963,7 +967,7 @@ mshv_partition_ioctl_create_vp(struct mshv_partition *partition,
> if (mshv_partition_encrypted(partition) && is_ghcb_mapping_available())
> vp->vp_ghcb_page = page_to_virt(ghcb_page);
>
> - if (hv_parent_partition())
> + if (hv_scheduler_type == HV_SCHEDULER_TYPE_ROOT)
> memcpy(vp->vp_stats_pages, stats_pages, sizeof(stats_pages));
>
> /*
> @@ -986,7 +990,7 @@ mshv_partition_ioctl_create_vp(struct mshv_partition *partition,
> free_vp:
> kfree(vp);
> unmap_stats_pages:
> - if (hv_parent_partition())
> + if (hv_scheduler_type == HV_SCHEDULER_TYPE_ROOT)
> mshv_vp_stats_unmap(partition->pt_id, args.vp_index);
> unmap_ghcb_page:
> if (mshv_partition_encrypted(partition) && is_ghcb_mapping_available()) {
> @@ -1740,7 +1744,7 @@ static void destroy_partition(struct mshv_partition *partition)
> if (!vp)
> continue;
>
> - if (hv_parent_partition())
> + if (hv_scheduler_type == HV_SCHEDULER_TYPE_ROOT)
> mshv_vp_stats_unmap(partition->pt_id, vp->vp_index);
>
> if (vp->vp_register_page) {
Acked-by: Stanislav Kinsburskii <skinsburskii@linux.microsoft.com>
> --
> 2.34.1
>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox