* [PATCH v7 8/9] x86/hyperv/vtl: Mark the wakeup mailbox page as private
From: Ricardo Neri @ 2025-11-17 17:02 UTC (permalink / raw)
To: x86, Krzysztof Kozlowski, Conor Dooley, Rob Herring,
K. Y. Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui,
Michael Kelley, Rafael J. Wysocki
Cc: Saurabh Sengar, Chris Oo, Kirill A. Shutemov, linux-hyperv,
devicetree, linux-acpi, linux-kernel, Ricardo Neri, Yunhong Jiang,
Ricardo Neri
In-Reply-To: <20251117-rneri-wakeup-mailbox-v7-0-4a8b82ab7c2c@linux.intel.com>
From: Yunhong Jiang <yunhong.jiang@linux.intel.com>
The current code maps MMIO devices as shared (decrypted) by default in a
confidential computing VM.
In a TDX environment, secondary CPUs are booted using the Multiprocessor
Wakeup Structure defined in the ACPI specification. The virtual firmware
and the operating system function in the guest context, without
intervention from the VMM. Map the physical memory of the mailbox as
private. Use the is_private_mmio() callback.
Signed-off-by: Yunhong Jiang <yunhong.jiang@linux.intel.com>
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
---
Changed since v7:
- Dropped check for !CONFIG_X86_MAILBOX_WAKEUP. The symbol is no longer
valid and now we have a stub for !CONFIG_ACPI.
- Dropped Reviewed-by tags from Dexuan and Michael as this patch
changed.
Changes in v6:
- Fixed a compile error with !CONFIG_X86_MAILBOX_WAKEUP.
- Added Reviewed-by tag from Dexuan. Thanks!
Changes in v5:
- None
Changes in v4:
- Updated to use the renamed function acpi_get_mp_wakeup_mailbox_paddr().
- Added Reviewed-by tag from Michael. Thanks!
Changes in v3:
- Use the new helper function get_mp_wakeup_mailbox_paddr().
- Edited the commit message for clarity.
Changes in v2:
- Added the helper function within_page() to improve readability
- Override the is_private_mmio() callback when detecting a TDX
environment. The address of the mailbox is checked in
hv_is_private_mmio_tdx().
---
arch/x86/hyperv/hv_vtl.c | 15 +++++++++++++++
1 file changed, 15 insertions(+)
diff --git a/arch/x86/hyperv/hv_vtl.c b/arch/x86/hyperv/hv_vtl.c
index ca0d23206e67..812d8a434966 100644
--- a/arch/x86/hyperv/hv_vtl.c
+++ b/arch/x86/hyperv/hv_vtl.c
@@ -6,6 +6,7 @@
* Saurabh Sengar <ssengar@microsoft.com>
*/
+#include <asm/acpi.h>
#include <asm/apic.h>
#include <asm/boot.h>
#include <asm/desc.h>
@@ -54,6 +55,18 @@ static void __noreturn hv_vtl_restart(char __maybe_unused *cmd)
hv_vtl_emergency_restart();
}
+static inline bool within_page(u64 addr, u64 start)
+{
+ return addr >= start && addr < (start + PAGE_SIZE);
+}
+
+static bool hv_vtl_is_private_mmio_tdx(u64 addr)
+{
+ u64 mb_addr = acpi_get_mp_wakeup_mailbox_paddr();
+
+ return mb_addr && within_page(addr, mb_addr);
+}
+
void __init hv_vtl_init_platform(void)
{
/*
@@ -66,6 +79,8 @@ void __init hv_vtl_init_platform(void)
/* There is no paravisor present if we are here. */
if (hv_isolation_type_tdx()) {
x86_init.resources.realmode_limit = SZ_4G;
+ x86_platform.hyper.is_private_mmio = hv_vtl_is_private_mmio_tdx;
+
} else {
x86_platform.realmode_reserve = x86_init_noop;
x86_platform.realmode_init = x86_init_noop;
--
2.43.0
^ permalink raw reply related
* [PATCH v7 9/9] x86/hyperv/vtl: Use the wakeup mailbox to boot secondary CPUs
From: Ricardo Neri @ 2025-11-17 17:02 UTC (permalink / raw)
To: x86, Krzysztof Kozlowski, Conor Dooley, Rob Herring,
K. Y. Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui,
Michael Kelley, Rafael J. Wysocki
Cc: Saurabh Sengar, Chris Oo, Kirill A. Shutemov, linux-hyperv,
devicetree, linux-acpi, linux-kernel, Ricardo Neri, Ricardo Neri
In-Reply-To: <20251117-rneri-wakeup-mailbox-v7-0-4a8b82ab7c2c@linux.intel.com>
The hypervisor is an untrusted entity for TDX guests. It cannot be used
to boot secondary CPUs. The function hv_vtl_wakeup_secondary_cpu() cannot
be used.
Instead, the virtual firmware boots the secondary CPUs and places them in
a state to transfer control to the kernel using the wakeup mailbox. The
firmware enumerates the mailbox via either an ACPI table or a DeviceTree
node.
If the wakeup mailbox is present, the kernel updates the APIC callback
wakeup_secondary_cpu_64() to use it.
Reviewed-by: Dexuan Cui <decui@microsoft.com>
Reviewed-by: Michael Kelley <mhklinux@outlook.com>
Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
---
Changes in v7:
- None
Changes in v6:
- Added Reviewed-by tag from Dexuan. Thanks!
Changes in v5:
- None
Changes in v4:
- Added Reviewed-by tag from Michael. Thanks!
Changes in v3:
- Unconditionally use the wakeup mailbox in a TDX confidential VM.
(Michael).
- Edited the commit message for clarity.
Changes in v2:
- None
---
arch/x86/hyperv/hv_vtl.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/arch/x86/hyperv/hv_vtl.c b/arch/x86/hyperv/hv_vtl.c
index 812d8a434966..431218ad6624 100644
--- a/arch/x86/hyperv/hv_vtl.c
+++ b/arch/x86/hyperv/hv_vtl.c
@@ -265,7 +265,15 @@ int __init hv_vtl_early_init(void)
panic("XSAVE has to be disabled as it is not supported by this module.\n"
"Please add 'noxsave' to the kernel command line.\n");
- apic_update_callback(wakeup_secondary_cpu_64, hv_vtl_wakeup_secondary_cpu);
+ /*
+ * TDX confidential VMs do not trust the hypervisor and cannot use it to
+ * boot secondary CPUs. Instead, they will be booted using the wakeup
+ * mailbox if detected during boot. See setup_arch().
+ *
+ * There is no paravisor present if we are here.
+ */
+ if (!hv_isolation_type_tdx())
+ apic_update_callback(wakeup_secondary_cpu_64, hv_vtl_wakeup_secondary_cpu);
return 0;
}
--
2.43.0
^ permalink raw reply related
* Re: [PATCH] Drivers: hv: ioctl for self targeted passthrough hvcalls
From: Wei Liu @ 2025-11-17 17:34 UTC (permalink / raw)
To: kernel test robot, nunodasneves
Cc: Anirudh Rayabharam, K. Y. Srinivasan, Haiyang Zhang, Wei Liu,
Dexuan Cui, Long Li, llvm, oe-kbuild-all, linux-hyperv,
linux-kernel
In-Reply-To: <202511161617.KcDzR4sA-lkp@intel.com>
+Nuno
On Sun, Nov 16, 2025 at 04:17:08PM +0800, kernel test robot wrote:
> Hi Anirudh,
>
> kernel test robot noticed the following build errors:
>
> [auto build test ERROR on linus/master]
> [also build test ERROR on v6.18-rc5]
> [cannot apply to next-20251114]
> [If your patch is applied to the wrong git tree, kindly drop us a note.
> And when submitting patch, we suggest to use '--base' as documented in
> https://git-scm.com/docs/git-format-patch#_base_tree_information]
>
> url: https://github.com/intel-lab-lkp/linux/commits/Anirudh-Rayabharam/Drivers-hv-ioctl-for-self-targeted-passthrough-hvcalls/20251114-182039
> base: linus/master
> patch link: https://lore.kernel.org/r/20251114095853.3482596-1-anirudh%40anirudhrb.com
> patch subject: [PATCH] Drivers: hv: ioctl for self targeted passthrough hvcalls
> config: x86_64-buildonly-randconfig-005-20251116 (https://download.01.org/0day-ci/archive/20251116/202511161617.KcDzR4sA-lkp@intel.com/config)
> compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
> reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251116/202511161617.KcDzR4sA-lkp@intel.com/reproduce)
>
> If you fix the issue in a separate patch/commit (i.e. not just a new version of
> the same patch/commit), kindly add following tags
> | Reported-by: kernel test robot <lkp@intel.com>
> | Closes: https://lore.kernel.org/oe-kbuild-all/202511161617.KcDzR4sA-lkp@intel.com/
>
> All errors (new ones prefixed by >>):
>
> >> drivers/hv/mshv_root_main.c:125:2: error: use of undeclared identifier 'HVCALL_GET_PARTITION_PROPERTY_EX'
> 125 | HVCALL_GET_PARTITION_PROPERTY_EX,
> | ^
Anirudh, please check this.
> >> drivers/hv/mshv_root_main.c:175:18: error: invalid application of 'sizeof' to an incomplete type 'u16[]' (aka 'unsigned short[]')
> 175 | for (i = 0; i < ARRAY_SIZE(mshv_passthru_hvcalls); ++i)
> | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is from the original patch. Perhaps adding the explicit declaration
of the array size would help.
Wei
^ permalink raw reply
* Re: [PATCH] Drivers: hv: ioctl for self targeted passthrough hvcalls
From: Nuno Das Neves @ 2025-11-17 17:41 UTC (permalink / raw)
To: Wei Liu, kernel test robot
Cc: Anirudh Rayabharam, K. Y. Srinivasan, Haiyang Zhang, Dexuan Cui,
Long Li, llvm, oe-kbuild-all, linux-hyperv, linux-kernel
In-Reply-To: <20251117173428.GB2380208@liuwe-devbox-debian-v2.local>
On 11/17/2025 9:34 AM, Wei Liu wrote:
> +Nuno
>
> On Sun, Nov 16, 2025 at 04:17:08PM +0800, kernel test robot wrote:
>> Hi Anirudh,
>>
>> kernel test robot noticed the following build errors:
>>
>> [auto build test ERROR on linus/master]
>> [also build test ERROR on v6.18-rc5]
>> [cannot apply to next-20251114]
>> [If your patch is applied to the wrong git tree, kindly drop us a note.
>> And when submitting patch, we suggest to use '--base' as documented in
>> https://git-scm.com/docs/git-format-patch#_base_tree_information]
>>
>> url: https://github.com/intel-lab-lkp/linux/commits/Anirudh-Rayabharam/Drivers-hv-ioctl-for-self-targeted-passthrough-hvcalls/20251114-182039
>> base: linus/master
>> patch link: https://lore.kernel.org/r/20251114095853.3482596-1-anirudh%40anirudhrb.com
>> patch subject: [PATCH] Drivers: hv: ioctl for self targeted passthrough hvcalls
>> config: x86_64-buildonly-randconfig-005-20251116 (https://download.01.org/0day-ci/archive/20251116/202511161617.KcDzR4sA-lkp@intel.com/config)
>> compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
>> reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251116/202511161617.KcDzR4sA-lkp@intel.com/reproduce)
>>
>> If you fix the issue in a separate patch/commit (i.e. not just a new version of
>> the same patch/commit), kindly add following tags
>> | Reported-by: kernel test robot <lkp@intel.com>
>> | Closes: https://lore.kernel.org/oe-kbuild-all/202511161617.KcDzR4sA-lkp@intel.com/
>>
>> All errors (new ones prefixed by >>):
>>
>>>> drivers/hv/mshv_root_main.c:125:2: error: use of undeclared identifier 'HVCALL_GET_PARTITION_PROPERTY_EX'
>> 125 | HVCALL_GET_PARTITION_PROPERTY_EX,
>> | ^
>
> Anirudh, please check this.
>
This is introduced in hyperv-next, in:
59aeea195948 mshv: Add the HVCALL_GET_PARTITION_PROPERTY_EX hypercall
But the bot applies the patch to linus/master, that is probably why.
>>>> drivers/hv/mshv_root_main.c:175:18: error: invalid application of 'sizeof' to an incomplete type 'u16[]' (aka 'unsigned short[]')
>> 175 | for (i = 0; i < ARRAY_SIZE(mshv_passthru_hvcalls); ++i)
>> | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> This is from the original patch. Perhaps adding the explicit declaration
> of the array size would help.
>
I think this is just due to the earlier error...
Nuno
> Wei
^ permalink raw reply
* [PATCH] Drivers: hv: adjust interrupt control structure for ARM64
From: Anirudh Rayabharam @ 2025-11-17 17:50 UTC (permalink / raw)
To: K. Y. Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li
Cc: anirudh, Jinank Jain, linux-hyperv, linux-kernel
From: Jinank Jain <jinankjain@microsoft.com>
Interrupt control structure (union hv_interupt_control) has different
fields when it comes to x86 vs ARM64. Bring in the correct structure
from HyperV header files and adjust the existing interrupt routing
code accordingly.
Signed-off-by: Jinank Jain <jinankjain@microsoft.com>
Signed-off-by: Anirudh Rayabharam (Microsoft) <anirudh@anirudhrb.com>
---
drivers/hv/mshv_eventfd.c | 6 ++++++
drivers/hv/mshv_irq.c | 4 ++++
drivers/hv/mshv_root_hv_call.c | 6 ++++++
include/hyperv/hvhdk.h | 6 ++++++
4 files changed, 22 insertions(+)
diff --git a/drivers/hv/mshv_eventfd.c b/drivers/hv/mshv_eventfd.c
index 2a80af1d610a..d93a18f09c76 100644
--- a/drivers/hv/mshv_eventfd.c
+++ b/drivers/hv/mshv_eventfd.c
@@ -163,8 +163,10 @@ static int mshv_try_assert_irq_fast(struct mshv_irqfd *irqfd)
if (hv_scheduler_type != HV_SCHEDULER_TYPE_ROOT)
return -EOPNOTSUPP;
+#if IS_ENABLED(CONFIG_X86)
if (irq->lapic_control.logical_dest_mode)
return -EOPNOTSUPP;
+#endif
vp = partition->pt_vp_array[irq->lapic_apic_id];
@@ -196,8 +198,10 @@ static void mshv_assert_irq_slow(struct mshv_irqfd *irqfd)
unsigned int seq;
int idx;
+#if IS_ENABLED(CONFIG_X86)
WARN_ON(irqfd->irqfd_resampler &&
!irq->lapic_control.level_triggered);
+#endif
idx = srcu_read_lock(&partition->pt_irq_srcu);
if (irqfd->irqfd_girq_ent.guest_irq_num) {
@@ -469,6 +473,7 @@ static int mshv_irqfd_assign(struct mshv_partition *pt,
init_poll_funcptr(&irqfd->irqfd_polltbl, mshv_irqfd_queue_proc);
spin_lock_irq(&pt->pt_irqfds_lock);
+#if IS_ENABLED(CONFIG_X86)
if (args->flags & BIT(MSHV_IRQFD_BIT_RESAMPLE) &&
!irqfd->irqfd_lapic_irq.lapic_control.level_triggered) {
/*
@@ -479,6 +484,7 @@ static int mshv_irqfd_assign(struct mshv_partition *pt,
ret = -EINVAL;
goto fail;
}
+#endif
ret = 0;
hlist_for_each_entry(tmp, &pt->pt_irqfds_list, irqfd_hnode) {
if (irqfd->irqfd_eventfd_ctx != tmp->irqfd_eventfd_ctx)
diff --git a/drivers/hv/mshv_irq.c b/drivers/hv/mshv_irq.c
index d0fb9ef734f4..798e7e1ab06e 100644
--- a/drivers/hv/mshv_irq.c
+++ b/drivers/hv/mshv_irq.c
@@ -119,6 +119,10 @@ void mshv_copy_girq_info(struct mshv_guest_irq_ent *ent,
lirq->lapic_vector = ent->girq_irq_data & 0xFF;
lirq->lapic_apic_id = (ent->girq_addr_lo >> 12) & 0xFF;
lirq->lapic_control.interrupt_type = (ent->girq_irq_data & 0x700) >> 8;
+#if IS_ENABLED(CONFIG_X86)
lirq->lapic_control.level_triggered = (ent->girq_irq_data >> 15) & 0x1;
lirq->lapic_control.logical_dest_mode = (ent->girq_addr_lo >> 2) & 0x1;
+#elif IS_ENABLED(CONFIG_ARM64)
+ lirq->lapic_control.asserted = 1;
+#endif
}
diff --git a/drivers/hv/mshv_root_hv_call.c b/drivers/hv/mshv_root_hv_call.c
index caf02cfa49c9..598eaff4ff29 100644
--- a/drivers/hv/mshv_root_hv_call.c
+++ b/drivers/hv/mshv_root_hv_call.c
@@ -388,7 +388,13 @@ int hv_call_assert_virtual_interrupt(u64 partition_id, u32 vector,
memset(input, 0, sizeof(*input));
input->partition_id = partition_id;
input->vector = vector;
+ /*
+ * NOTE: dest_addr only needs to be provided while asserting an
+ * interrupt on x86 platform
+ */
+#if IS_ENABLED(CONFIG_X86)
input->dest_addr = dest_addr;
+#endif
input->control = control;
status = hv_do_hypercall(HVCALL_ASSERT_VIRTUAL_INTERRUPT, input, NULL);
local_irq_restore(flags);
diff --git a/include/hyperv/hvhdk.h b/include/hyperv/hvhdk.h
index 416c0d45b793..469186df7826 100644
--- a/include/hyperv/hvhdk.h
+++ b/include/hyperv/hvhdk.h
@@ -579,9 +579,15 @@ union hv_interrupt_control {
u64 as_uint64;
struct {
u32 interrupt_type; /* enum hv_interrupt_type */
+#if IS_ENABLED(CONFIG_X86)
u32 level_triggered : 1;
u32 logical_dest_mode : 1;
u32 rsvd : 30;
+#elif IS_ENABLED(CONFIG_ARM64)
+ u32 rsvd1 : 2;
+ u32 asserted : 1;
+ u32 rsvd2 : 29;
+#endif
} __packed;
};
base-commit: db7df69995ffbe806d60ad46d5fb9d959da9e549
--
2.34.1
^ permalink raw reply related
* Re: [PATCH] Drivers: hv: ioctl for self targeted passthrough hvcalls
From: Anirudh Rayabharam @ 2025-11-17 17:57 UTC (permalink / raw)
To: Nuno Das Neves
Cc: Wei Liu, kernel test robot, K. Y. Srinivasan, Haiyang Zhang,
Dexuan Cui, Long Li, llvm, oe-kbuild-all, linux-hyperv,
linux-kernel
In-Reply-To: <522f1bc2-417f-434a-92f9-7b417744cef4@linux.microsoft.com>
On Mon, Nov 17, 2025 at 09:41:19AM -0800, Nuno Das Neves wrote:
> On 11/17/2025 9:34 AM, Wei Liu wrote:
> > +Nuno
> >
> > On Sun, Nov 16, 2025 at 04:17:08PM +0800, kernel test robot wrote:
> >> Hi Anirudh,
> >>
> >> kernel test robot noticed the following build errors:
> >>
> >> [auto build test ERROR on linus/master]
> >> [also build test ERROR on v6.18-rc5]
> >> [cannot apply to next-20251114]
> >> [If your patch is applied to the wrong git tree, kindly drop us a note.
> >> And when submitting patch, we suggest to use '--base' as documented in
> >> https://git-scm.com/docs/git-format-patch#_base_tree_information]
> >>
> >> url: https://github.com/intel-lab-lkp/linux/commits/Anirudh-Rayabharam/Drivers-hv-ioctl-for-self-targeted-passthrough-hvcalls/20251114-182039
> >> base: linus/master
> >> patch link: https://lore.kernel.org/r/20251114095853.3482596-1-anirudh%40anirudhrb.com
> >> patch subject: [PATCH] Drivers: hv: ioctl for self targeted passthrough hvcalls
> >> config: x86_64-buildonly-randconfig-005-20251116 (https://download.01.org/0day-ci/archive/20251116/202511161617.KcDzR4sA-lkp@intel.com/config)
> >> compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
> >> reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251116/202511161617.KcDzR4sA-lkp@intel.com/reproduce)
> >>
> >> If you fix the issue in a separate patch/commit (i.e. not just a new version of
> >> the same patch/commit), kindly add following tags
> >> | Reported-by: kernel test robot <lkp@intel.com>
> >> | Closes: https://lore.kernel.org/oe-kbuild-all/202511161617.KcDzR4sA-lkp@intel.com/
> >>
> >> All errors (new ones prefixed by >>):
> >>
> >>>> drivers/hv/mshv_root_main.c:125:2: error: use of undeclared identifier 'HVCALL_GET_PARTITION_PROPERTY_EX'
> >> 125 | HVCALL_GET_PARTITION_PROPERTY_EX,
> >> | ^
> >
> > Anirudh, please check this.
> >
> This is introduced in hyperv-next, in:
> 59aeea195948 mshv: Add the HVCALL_GET_PARTITION_PROPERTY_EX hypercall
>
> But the bot applies the patch to linus/master, that is probably why.
Right. In the v2 of this patch, I used the --base argument to git
format-patch as suggested by the tool above. That should help it apply
the patch in the right tree. Let's see if it still complains.
Anirudh
>
> >>>> drivers/hv/mshv_root_main.c:175:18: error: invalid application of 'sizeof' to an incomplete type 'u16[]' (aka 'unsigned short[]')
> >> 175 | for (i = 0; i < ARRAY_SIZE(mshv_passthru_hvcalls); ++i)
> >> | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > This is from the original patch. Perhaps adding the explicit declaration
> > of the array size would help.
> >
>
> I think this is just due to the earlier error...
>
> Nuno
>
> > Wei
>
^ permalink raw reply
* Re: [Intel-wired-lan] [PATCH v3] net: ethernet: fix uninitialized pointers with free attribute
From: Dan Carpenter @ 2025-11-17 18:11 UTC (permalink / raw)
To: Alexander Lobakin
Cc: Ally Heev, Tony Nguyen, Przemek Kitszel, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
K. Y. Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui,
Aleksandr Loktionov, intel-wired-lan, netdev, linux-kernel,
linux-hyperv
In-Reply-To: <dd88462f-19cb-4fde-b1f0-5caf7e6c6ce6@intel.com>
On Mon, Nov 17, 2025 at 03:37:30PM +0100, Alexander Lobakin wrote:
> From: Dan Carpenter <dan.carpenter@linaro.org>
> Date: Mon, 17 Nov 2025 16:11:32 +0300
>
> > On Thu, Nov 06, 2025 at 03:07:26PM +0100, Alexander Lobakin wrote:
> >>> diff --git a/drivers/net/ethernet/intel/ice/ice_flow.c b/drivers/net/ethernet/intel/ice/ice_flow.c
> >>> index 6d5c939dc8a515c252cd2b77d155b69fa264ee92..3590dacf3ee57879b3809d715e40bb290e40c4aa 100644
> >>> --- a/drivers/net/ethernet/intel/ice/ice_flow.c
> >>> +++ b/drivers/net/ethernet/intel/ice/ice_flow.c
> >>> @@ -1573,12 +1573,13 @@ ice_flow_set_parser_prof(struct ice_hw *hw, u16 dest_vsi, u16 fdir_vsi,
> >>> struct ice_parser_profile *prof, enum ice_block blk)
> >>> {
> >>> u64 id = find_first_bit(prof->ptypes, ICE_FLOW_PTYPE_MAX);
> >>> - struct ice_flow_prof_params *params __free(kfree);
> >>> u8 fv_words = hw->blk[blk].es.fvw;
> >>> int status;
> >>> int i, idx;
> >>>
> >>> - params = kzalloc(sizeof(*params), GFP_KERNEL);
> >>> + struct ice_flow_prof_params *params __free(kfree) =
> >>> + kzalloc(sizeof(*params), GFP_KERNEL);
> >>
> >> Please don't do it that way. It's not C++ with RAII and
> >> declare-where-you-use.
> >> Just leave the variable declarations where they are, but initialize them
> >> with `= NULL`.
> >>
> >> Variable declarations must be in one block and sorted from the longest
> >> to the shortest.
> >>
> >
> > These days, with __free the trend is to say yes this is RAII and we
> > should declare it where you use it. I personally don't have a strong
>
> Sorta, but we can't "declare it where you use it" since we don't allow
> declaration-after-statement in the kernel.
That changed when we merged cleanup.h. It is allowed now. I still don't
like to declare variables anywhere unless it's a __free() variable and I
think almost everyone else agrees. The only subsystem which I know that
completely moved to declaring variables willy-nilly was bcachefs.
regards,
dan carpenter
^ permalink raw reply
* Re: [PATCH v2] Drivers: hv: ioctl for self targeted passthrough hvcalls
From: Nuno Das Neves @ 2025-11-17 18:16 UTC (permalink / raw)
To: Anirudh Rayabharam, K. Y. Srinivasan, Haiyang Zhang, Wei Liu,
Dexuan Cui, Long Li
Cc: linux-hyperv, linux-kernel
In-Reply-To: <20251117095207.113502-1-anirudh@anirudhrb.com>
On 11/17/2025 1:52 AM, Anirudh Rayabharam wrote:
> From: Anirudh Rayabharam (Microsoft) <anirudh@anirudhrb.com>
>
> Allow MSHV_ROOT_HVCALL IOCTL on the /dev/mshv fd. This IOCTL would
> execute a passthrough hypercall targeting the root/parent partition
> i.e. HV_PARTITION_ID_SELF.
>
I think it's worth taking a moment to check and perhaps explain in
the commit message/a comment any security implications of the VMM
process being able to call these hypercalls on the root/parent
partition.
One implication would be: can the VMM process influence other
processes in the root partition via these hypercalls,
e.g. HVCALL_SET_VP_REGISTERS? I would think that the hypervisor
itself disallows this but we should check. We can ask the
hypervisor team what they think, and check the hypervisor code.
Specifically we should check on any hypercall that could possibly
influence partition state, i.e.:
HVCALL_SET_PARTITION_PROPERTY
HVCALL_SET_VP_REGISTERS
HVCALL_INSTALL_INTERCEPT
HVCALL_CLEAR_VIRTUAL_INTERRUPT
HVCALL_REGISTER_INTERCEPT_RESULT
HVCALL_ASSERT_VIRTUAL_INTERRUPT
HVCALL_SIGNAL_EVENT_DIRECT
HVCALL_POST_MESSAGE_DIRECT
If it turns out there is something risky we are enabling here, we can
introduce a new array of hypercalls to restrict which ones can be
called on HV_PARTITION_ID_SELF.
> This will be useful for the VMM to query things like supported
> synthetic processor features, supported VMM capabiliites etc.
>
> While at it, add HVCALL_GET_PARTITION_PROPERTY_EX to the allowed list of
> passthrough hypercalls.
>
> Signed-off-by: Anirudh Rayabharam (Microsoft) <anirudh@anirudhrb.com>
> ---
>
> v2: rebased on latest hyperv-next
>
> ---
> drivers/hv/mshv_root_main.c | 11 ++++++++---
> 1 file changed, 8 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/hv/mshv_root_main.c b/drivers/hv/mshv_root_main.c
> index 45c7a5fea1cf..671c4d18520a 100644
> --- a/drivers/hv/mshv_root_main.c
> +++ b/drivers/hv/mshv_root_main.c
> @@ -122,6 +122,7 @@ static struct miscdevice mshv_dev = {
> */
> static u16 mshv_passthru_hvcalls[] = {
> HVCALL_GET_PARTITION_PROPERTY,
> + HVCALL_GET_PARTITION_PROPERTY_EX,
> HVCALL_SET_PARTITION_PROPERTY,
> HVCALL_INSTALL_INTERCEPT,
> HVCALL_GET_VP_REGISTERS,
> @@ -160,6 +161,7 @@ static int mshv_ioctl_passthru_hvcall(struct mshv_partition *partition,
> void *input_pg = NULL;
> void *output_pg = NULL;
> u16 reps_completed;
> + u64 pt_id = partition ? partition->pt_id : HV_PARTITION_ID_SELF;
>
> if (copy_from_user(&args, user_args, sizeof(args)))
> return -EFAULT;
> @@ -181,7 +183,7 @@ static int mshv_ioctl_passthru_hvcall(struct mshv_partition *partition,
> is_async = mshv_hvcall_is_async(args.code);
> if (is_async) {
> /* async hypercalls can only be called from partition fd */
> - if (!partition_locked)
> + if (!partition || !partition_locked)
> return -EINVAL;
> ret = mshv_init_async_handler(partition);
> if (ret)
> @@ -209,7 +211,7 @@ static int mshv_ioctl_passthru_hvcall(struct mshv_partition *partition,
> * NOTE: This only works because all the allowed hypercalls' input
> * structs begin with a u64 partition_id field.
> */
> - *(u64 *)input_pg = partition->pt_id;
> + *(u64 *)input_pg = pt_id;
>
> reps_completed = 0;
> do {
> @@ -238,7 +240,7 @@ static int mshv_ioctl_passthru_hvcall(struct mshv_partition *partition,
> ret = hv_result_to_errno(status);
> else
> ret = hv_call_deposit_pages(NUMA_NO_NODE,
> - partition->pt_id, 1);
> + pt_id, 1);
nit: Can these arguments just all go on a single line now?
> } while (!ret);
>
> args.status = hv_result(status);
> @@ -2050,6 +2052,9 @@ static long mshv_dev_ioctl(struct file *filp, unsigned int ioctl,
> case MSHV_CREATE_PARTITION:
> return mshv_ioctl_create_partition((void __user *)arg,
> misc->this_device);
> + case MSHV_ROOT_HVCALL:
> + return mshv_ioctl_passthru_hvcall(NULL, false,
> + (void __user *)arg);
> }
>
> return -ENOTTY;
>
> base-commit: db7df69995ffbe806d60ad46d5fb9d959da9e549
^ permalink raw reply
* Re: [PATCH v2] Drivers: hv: ioctl for self targeted passthrough hvcalls
From: Wei Liu @ 2025-11-17 19:18 UTC (permalink / raw)
To: Nuno Das Neves
Cc: Anirudh Rayabharam, K. Y. Srinivasan, Haiyang Zhang, Wei Liu,
Dexuan Cui, Long Li, linux-hyperv, linux-kernel
In-Reply-To: <36ac7105-3aa7-4e53-b87d-b99438f65295@linux.microsoft.com>
On Mon, Nov 17, 2025 at 10:16:12AM -0800, Nuno Das Neves wrote:
> On 11/17/2025 1:52 AM, Anirudh Rayabharam wrote:
> > From: Anirudh Rayabharam (Microsoft) <anirudh@anirudhrb.com>
> >
> > Allow MSHV_ROOT_HVCALL IOCTL on the /dev/mshv fd. This IOCTL would
> > execute a passthrough hypercall targeting the root/parent partition
> > i.e. HV_PARTITION_ID_SELF.
> >
>
> I think it's worth taking a moment to check and perhaps explain in
> the commit message/a comment any security implications of the VMM
> process being able to call these hypercalls on the root/parent
> partition.
>
> One implication would be: can the VMM process influence other
> processes in the root partition via these hypercalls,
> e.g. HVCALL_SET_VP_REGISTERS? I would think that the hypervisor
> itself disallows this but we should check. We can ask the
> hypervisor team what they think, and check the hypervisor code.
>
> Specifically we should check on any hypercall that could possibly
> influence partition state, i.e.:
> HVCALL_SET_PARTITION_PROPERTY
> HVCALL_SET_VP_REGISTERS
> HVCALL_INSTALL_INTERCEPT
> HVCALL_CLEAR_VIRTUAL_INTERRUPT
> HVCALL_REGISTER_INTERCEPT_RESULT
> HVCALL_ASSERT_VIRTUAL_INTERRUPT
> HVCALL_SIGNAL_EVENT_DIRECT
> HVCALL_POST_MESSAGE_DIRECT
>
> If it turns out there is something risky we are enabling here, we can
> introduce a new array of hypercalls to restrict which ones can be
> called on HV_PARTITION_ID_SELF.
>
This is a good point. Please check with the hypervisor team.
Wei
^ permalink raw reply
* Re: [PATCH v2] Drivers: hv: ioctl for self targeted passthrough hvcalls
From: Wei Liu @ 2025-11-17 19:24 UTC (permalink / raw)
To: Nuno Das Neves
Cc: Anirudh Rayabharam, K. Y. Srinivasan, Haiyang Zhang, Wei Liu,
Dexuan Cui, Long Li, linux-hyperv, linux-kernel
In-Reply-To: <20251117191827.GC2380208@liuwe-devbox-debian-v2.local>
On Mon, Nov 17, 2025 at 07:18:27PM +0000, Wei Liu wrote:
> On Mon, Nov 17, 2025 at 10:16:12AM -0800, Nuno Das Neves wrote:
> > On 11/17/2025 1:52 AM, Anirudh Rayabharam wrote:
> > > From: Anirudh Rayabharam (Microsoft) <anirudh@anirudhrb.com>
> > >
> > > Allow MSHV_ROOT_HVCALL IOCTL on the /dev/mshv fd. This IOCTL would
> > > execute a passthrough hypercall targeting the root/parent partition
> > > i.e. HV_PARTITION_ID_SELF.
> > >
> >
> > I think it's worth taking a moment to check and perhaps explain in
> > the commit message/a comment any security implications of the VMM
> > process being able to call these hypercalls on the root/parent
> > partition.
> >
> > One implication would be: can the VMM process influence other
> > processes in the root partition via these hypercalls,
> > e.g. HVCALL_SET_VP_REGISTERS? I would think that the hypervisor
> > itself disallows this but we should check. We can ask the
> > hypervisor team what they think, and check the hypervisor code.
> >
> > Specifically we should check on any hypercall that could possibly
> > influence partition state, i.e.:
> > HVCALL_SET_PARTITION_PROPERTY
> > HVCALL_SET_VP_REGISTERS
> > HVCALL_INSTALL_INTERCEPT
> > HVCALL_CLEAR_VIRTUAL_INTERRUPT
> > HVCALL_REGISTER_INTERCEPT_RESULT
> > HVCALL_ASSERT_VIRTUAL_INTERRUPT
> > HVCALL_SIGNAL_EVENT_DIRECT
> > HVCALL_POST_MESSAGE_DIRECT
> >
> > If it turns out there is something risky we are enabling here, we can
> > introduce a new array of hypercalls to restrict which ones can be
> > called on HV_PARTITION_ID_SELF.
> >
>
> This is a good point. Please check with the hypervisor team.
I should add: it is always easier to relax restrictions later than to
add them back in, so if there is any doubt and we want this code in as
quickly as possible, we can start with a new array and expand it later.
Wei
^ permalink raw reply
* RE: [patch net-next] net: mana: Handle hardware reset events when probing the device
From: Long Li @ 2025-11-17 20:39 UTC (permalink / raw)
To: Haiyang Zhang, longli@linux.microsoft.com, KY Srinivasan, Wei Liu,
Dexuan Cui, David S. Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Shradha Gupta, Simon Horman, Konstantin Taranov,
Souradeep Chakrabarti, Erick Archer, linux-hyperv@vger.kernel.org,
netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-rdma@vger.kernel.org
In-Reply-To: <SA3PR21MB3867695C1B6C975ADFDE635BCAC8A@SA3PR21MB3867.namprd21.prod.outlook.com>
> Subject: RE: [patch net-next] net: mana: Handle hardware reset events when
> probing the device
>
>
>
> > -----Original Message-----
> > From: longli@linux.microsoft.com <longli@linux.microsoft.com>
> > Sent: Friday, November 14, 2025 9:29 PM
> > To: KY Srinivasan <kys@microsoft.com>; Haiyang Zhang
> > <haiyangz@microsoft.com>; Wei Liu <wei.liu@kernel.org>; Dexuan Cui
> > <DECUI@microsoft.com>; David S. Miller <davem@davemloft.net>; Eric
> > Dumazet <edumazet@google.com>; Jakub Kicinski <kuba@kernel.org>;
> Paolo
> > Abeni <pabeni@redhat.com>; Shradha Gupta
> > <shradhagupta@linux.microsoft.com>;
> > Simon Horman <horms@kernel.org>; Konstantin Taranov
> > <kotaranov@microsoft.com>; Souradeep Chakrabarti
> > <schakrabarti@linux.microsoft.com>; Erick Archer
> > <erick.archer@outlook.com>; linux-hyperv@vger.kernel.org;
> > netdev@vger.kernel.org; linux-kernel@vger.kernel.org; linux-
> > rdma@vger.kernel.org
> > Cc: Long Li <longli@microsoft.com>
> > Subject: [patch net-next] net: mana: Handle hardware reset events when
> > probing the device
> >
> > From: Long Li <longli@microsoft.com>
> >
> > When MANA is being probed, it's possible that hardware is in recovery
> > mode and the device may get GDMA_EQE_HWC_RESET_REQUEST over HWC
> in the
> > middle of the probe. Detect such condition and go through the recovery
> > service procedure.
> >
> > Fixes: fbe346ce9d62 ("net: mana: Handle Reset Request from MANA NIC")
> > Signed-off-by: Long Li <longli@microsoft.com>
> > ---
> > .../net/ethernet/microsoft/mana/gdma_main.c | 131 +++++++++++++++-
> --
> > include/net/mana/gdma.h | 9 +-
> > 2 files changed, 122 insertions(+), 18 deletions(-)
> >
> > diff --git a/drivers/net/ethernet/microsoft/mana/gdma_main.c
> > b/drivers/net/ethernet/microsoft/mana/gdma_main.c
> > index effe0a2f207a..1d9c2beb22b2 100644
> > --- a/drivers/net/ethernet/microsoft/mana/gdma_main.c
> > +++ b/drivers/net/ethernet/microsoft/mana/gdma_main.c
> > @@ -15,6 +15,12 @@
> >
> > struct dentry *mana_debugfs_root;
> >
> > +static struct mana_serv_delayed_work {
> > + struct delayed_work work;
> > + struct pci_dev *pdev;
> > + enum gdma_eqe_type type;
> > +} mns_delayed_wk;
> > +
> > static u32 mana_gd_r32(struct gdma_context *g, u64 offset) {
> > return readl(g->bar0_va + offset);
> > @@ -387,6 +393,25 @@ EXPORT_SYMBOL_NS(mana_gd_ring_cq,
> "NET_MANA");
> >
> > #define MANA_SERVICE_PERIOD 10
> >
> > +static void mana_serv_rescan(struct pci_dev *pdev) {
> > + struct pci_bus *parent;
> > +
> > + pci_lock_rescan_remove();
> > +
> > + parent = pdev->bus;
> > + if (!parent) {
> > + dev_err(&pdev->dev, "MANA service: no parent bus\n");
> > + goto out;
> > + }
> > +
> > + pci_stop_and_remove_bus_device(pdev);
> > + pci_rescan_bus(parent);
> > +
> > +out:
> > + pci_unlock_rescan_remove();
> > +}
> > +
> > static void mana_serv_fpga(struct pci_dev *pdev) {
> > struct pci_bus *bus, *parent;
> > @@ -419,9 +444,12 @@ static void mana_serv_reset(struct pci_dev *pdev)
> > {
> > struct gdma_context *gc = pci_get_drvdata(pdev);
> > struct hw_channel_context *hwc;
> > + int ret;
> >
> > if (!gc) {
> > - dev_err(&pdev->dev, "MANA service: no GC\n");
> > + /* Perform PCI rescan on device if GC is not set up */
> > + dev_err(&pdev->dev, "MANA service: GC not setup,
> > rescanning\n");
> > + mana_serv_rescan(pdev);
> > return;
> > }
> >
> > @@ -440,9 +468,18 @@ static void mana_serv_reset(struct pci_dev *pdev)
> >
> > msleep(MANA_SERVICE_PERIOD * 1000);
> >
> > - mana_gd_resume(pdev);
> > + ret = mana_gd_resume(pdev);
> > + if (ret == -ETIMEDOUT || ret == -EPROTO) {
> > + /* Perform PCI rescan on device if we failed on HWC */
> > + dev_err(&pdev->dev, "MANA service: resume failed,
> > rescanning\n");
> > + mana_serv_rescan(pdev);
> > + goto out;
> > + }
> >
> > - dev_info(&pdev->dev, "MANA reset cycle completed\n");
> > + if (ret)
> > + dev_info(&pdev->dev, "MANA reset cycle failed err %d\n",
> ret);
> > + else
> > + dev_info(&pdev->dev, "MANA reset cycle completed\n");
> >
> > out:
> > gc->in_service = false;
> > @@ -454,18 +491,9 @@ struct mana_serv_work {
> > enum gdma_eqe_type type;
> > };
> >
> > -static void mana_serv_func(struct work_struct *w)
> > +static void mana_do_service(enum gdma_eqe_type type, struct pci_dev
> > *pdev)
> > {
> > - struct mana_serv_work *mns_wk;
> > - struct pci_dev *pdev;
> > -
> > - mns_wk = container_of(w, struct mana_serv_work, serv_work);
> > - pdev = mns_wk->pdev;
> > -
> > - if (!pdev)
> > - goto out;
> > -
> > - switch (mns_wk->type) {
> > + switch (type) {
> > case GDMA_EQE_HWC_FPGA_RECONFIG:
> > mana_serv_fpga(pdev);
> > break;
> > @@ -475,12 +503,36 @@ static void mana_serv_func(struct work_struct
> *w)
> > break;
> >
> > default:
> > - dev_err(&pdev->dev, "MANA service: unknown type %d\n",
> > - mns_wk->type);
> > + dev_err(&pdev->dev, "MANA service: unknown type %d\n",
> type);
> > break;
> > }
> > +}
> > +
> > +static void mana_serv_delayed_func(struct work_struct *w) {
> > + struct mana_serv_delayed_work *dwork;
> > + struct pci_dev *pdev;
> > +
> > + dwork = container_of(w, struct mana_serv_delayed_work,
> work.work);
> > + pdev = dwork->pdev;
> > +
> > + if (pdev)
> > + mana_do_service(dwork->type, pdev);
> > +
> > + pci_dev_put(pdev);
> > +}
> > +
> > +static void mana_serv_func(struct work_struct *w) {
> > + struct mana_serv_work *mns_wk;
> > + struct pci_dev *pdev;
> > +
> > + mns_wk = container_of(w, struct mana_serv_work, serv_work);
> > + pdev = mns_wk->pdev;
> > +
> > + if (pdev)
> > + mana_do_service(mns_wk->type, pdev);
> >
> > -out:
> > pci_dev_put(pdev);
> > kfree(mns_wk);
> > module_put(THIS_MODULE);
> > @@ -541,6 +593,17 @@ static void mana_gd_process_eqe(struct
> gdma_queue
> > *eq)
> > case GDMA_EQE_HWC_RESET_REQUEST:
> > dev_info(gc->dev, "Recv MANA service type:%d\n", type);
> >
> > + if (atomic_inc_return(&gc->in_probe) == 1) {
>
> Since we don't care about how many times it entered probe/service,
> test_and_set_bit() should be sufficient here.
>
> > + /*
> > + * Device is in probe and we received an hardware
> reset
> > + * event, probe() will detect that "in_probe" has
> > + * changed and perform service procedure.
> > + */
> > + dev_info(gc->dev,
> > + "Service is to be processed in probe\n");
> > + break;
> > + }
> > +
> > if (gc->in_service) {
> > dev_info(gc->dev, "Already in service\n");
> > break;
> > @@ -1930,6 +1993,8 @@ static int mana_gd_probe(struct pci_dev *pdev,
> > const struct pci_device_id *ent)
> > gc->mana_pci_debugfs =
> debugfs_create_dir(pci_slot_name(pdev-
> > >slot),
> >
> mana_debugfs_root);
> >
> > + atomic_set(&gc->in_probe, 0);
> > +
> > err = mana_gd_setup(pdev);
> > if (err)
> > goto unmap_bar;
> > @@ -1942,8 +2007,19 @@ static int mana_gd_probe(struct pci_dev *pdev,
> > const struct pci_device_id *ent)
> > if (err)
> > goto cleanup_mana;
> >
> > + /*
> > + * If a hardware reset event has occurred over HWC during probe,
> > + * rollback and perform hardware reset procedure.
> > + */
> > + if (atomic_inc_return(&gc->in_probe) > 1) {
> > + err = -EPROTO;
> > + goto cleanup_mana_rdma;
> > + }
> > +
> > return 0;
> >
> > +cleanup_mana_rdma:
> > + mana_rdma_remove(&gc->mana_ib);
> > cleanup_mana:
> > mana_remove(&gc->mana, false);
> > cleanup_gd:
> > @@ -1967,6 +2043,25 @@ static int mana_gd_probe(struct pci_dev *pdev,
> > const struct pci_device_id *ent)
> > disable_dev:
> > pci_disable_device(pdev);
> > dev_err(&pdev->dev, "gdma probe failed: err = %d\n", err);
> > +
> > + /*
> > + * Hardware could be in recovery mode and the HWC returns
> TIMEDOUT
> > or
> > + * EPROTO from mana_gd_setup(), mana_probe() or
> mana_rdma_probe(),
> > or
> > + * we received a hardware reset event over HWC interrupt. In this
> > case,
> > + * perform the device recovery procedure after
> MANA_SERVICE_PERIOD
> > + * seconds.
> > + */
> > + if (err == -ETIMEDOUT || err == -EPROTO) {
> > + dev_info(&pdev->dev, "Start MANA recovery mode\n");
> > +
> > + mns_delayed_wk.pdev = pci_dev_get(pdev);
> > + mns_delayed_wk.type = GDMA_EQE_HWC_RESET_REQUEST;
> > +
> > + INIT_DELAYED_WORK(&mns_delayed_wk.work,
> > mana_serv_delayed_func);
>
> To avoid INIT_DELAYED_WORK potentially multiple times this should be in the
> mana_driver_init()
>
> > + schedule_delayed_work(&mns_delayed_wk.work,
> > + secs_to_jiffies(MANA_SERVICE_PERIOD));
> > + }
> > +
> > return err;
> > }
> >
> > @@ -2084,6 +2179,8 @@ static int __init mana_driver_init(void)
> >
> > static void __exit mana_driver_exit(void) {
> > + cancel_delayed_work_sync(&mns_delayed_wk.work);
>
> I think we should call disable_delayed_work_sync() to prevent the work
> scheduled again after this line.
Thank you. I will send v2 to address all the comments and support multiple PCI devices in BM mode.
Long
^ permalink raw reply
* [PATCH v5 0/3] Add support for clean shutdown with MSHV
From: Praveen K Paladugu @ 2025-11-17 21:08 UTC (permalink / raw)
To: kys, haiyangz, wei.liu, decui, tglx, mingo, linux-hyperv,
linux-kernel, bp, dave.hansen, x86, hpa, arnd
Cc: anbelski, prapal, easwar.hariharan, nunodasneves, skinsburskii
Add support for clean shutdown of the root partition when running on
MSHV Hypervisor.
v5:
- Fixed build errors
- Padded struct hv_input_set_system_property for alignment
- Dropped CONFIG_ACPI stub
v4:
- Adopted machine_ops to order invoking HV_ENTER_SLEEP_STATE as the
last step in shutdown sequence.
- This ensures rest of the cleanups are done before powering off
v3:
- Dropped acpi_sleep handlers as they are not used on mshv
- Applied ordering for hv_reboot_notifier
- Fixed build issues on i386, arm64 architectures
v2:
- Addressed review comments from v1.
- Moved all sleep state handling methods under CONFIG_ACPI stub
- - This fixes build issues on non-x86 architectures.
Praveen K Paladugu (3):
hyperv: Add definitions for MSHV sleep state configuration
hyperv: Use reboot notifier to configure sleep state
hyperv: Cleanly shutdown root partition with MSHV
arch/x86/hyperv/hv_init.c | 9 ++++
arch/x86/include/asm/mshyperv.h | 4 ++
drivers/hv/mshv_common.c | 96 +++++++++++++++++++++++++++++++++
include/hyperv/hvgdk_mini.h | 4 +-
include/hyperv/hvhdk_mini.h | 40 ++++++++++++++
5 files changed, 152 insertions(+), 1 deletion(-)
--
2.51.0
^ permalink raw reply
* [PATCH v5 1/3] hyperv: Add definitions for MSHV sleep state configuration
From: Praveen K Paladugu @ 2025-11-17 21:08 UTC (permalink / raw)
To: kys, haiyangz, wei.liu, decui, tglx, mingo, linux-hyperv,
linux-kernel, bp, dave.hansen, x86, hpa, arnd
Cc: anbelski, prapal, easwar.hariharan, nunodasneves, skinsburskii
In-Reply-To: <20251117210855.108126-1-prapal@linux.microsoft.com>
Add the definitions required to configure sleep states in mshv hypervsior.
Signed-off-by: Praveen K Paladugu <prapal@linux.microsoft.com>
Co-developed-by: Anatol Belski <anbelski@linux.microsoft.com>
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Reviewed-by: Easwar Hariharan <easwar.hariharan@linux.microsoft.com>
Reviewed-by: Nuno Das Neves <nunodasneves@linux.microsoft.com>
---
include/hyperv/hvgdk_mini.h | 4 +++-
include/hyperv/hvhdk_mini.h | 40 +++++++++++++++++++++++++++++++++++++
2 files changed, 43 insertions(+), 1 deletion(-)
diff --git a/include/hyperv/hvgdk_mini.h b/include/hyperv/hvgdk_mini.h
index 1d5ce11be8b6..04b18d0e37af 100644
--- a/include/hyperv/hvgdk_mini.h
+++ b/include/hyperv/hvgdk_mini.h
@@ -465,19 +465,21 @@ union hv_vp_assist_msr_contents { /* HV_REGISTER_VP_ASSIST_PAGE */
#define HVCALL_RESET_DEBUG_SESSION 0x006b
#define HVCALL_MAP_STATS_PAGE 0x006c
#define HVCALL_UNMAP_STATS_PAGE 0x006d
+#define HVCALL_SET_SYSTEM_PROPERTY 0x006f
#define HVCALL_ADD_LOGICAL_PROCESSOR 0x0076
#define HVCALL_GET_SYSTEM_PROPERTY 0x007b
#define HVCALL_MAP_DEVICE_INTERRUPT 0x007c
#define HVCALL_UNMAP_DEVICE_INTERRUPT 0x007d
#define HVCALL_RETARGET_INTERRUPT 0x007e
#define HVCALL_NOTIFY_PARTITION_EVENT 0x0087
+#define HVCALL_ENTER_SLEEP_STATE 0x0084
#define HVCALL_NOTIFY_PORT_RING_EMPTY 0x008b
#define HVCALL_REGISTER_INTERCEPT_RESULT 0x0091
#define HVCALL_ASSERT_VIRTUAL_INTERRUPT 0x0094
#define HVCALL_CREATE_PORT 0x0095
#define HVCALL_CONNECT_PORT 0x0096
#define HVCALL_START_VP 0x0099
-#define HVCALL_GET_VP_INDEX_FROM_APIC_ID 0x009a
+#define HVCALL_GET_VP_INDEX_FROM_APIC_ID 0x009a
#define HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_SPACE 0x00af
#define HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_LIST 0x00b0
#define HVCALL_SIGNAL_EVENT_DIRECT 0x00c0
diff --git a/include/hyperv/hvhdk_mini.h b/include/hyperv/hvhdk_mini.h
index f2d7b50de7a4..41a29bf8ec14 100644
--- a/include/hyperv/hvhdk_mini.h
+++ b/include/hyperv/hvhdk_mini.h
@@ -140,6 +140,7 @@ enum hv_snp_status {
enum hv_system_property {
/* Add more values when needed */
+ HV_SYSTEM_PROPERTY_SLEEP_STATE = 3,
HV_SYSTEM_PROPERTY_SCHEDULER_TYPE = 15,
HV_DYNAMIC_PROCESSOR_FEATURE_PROPERTY = 21,
HV_SYSTEM_PROPERTY_CRASHDUMPAREA = 47,
@@ -155,6 +156,19 @@ union hv_pfn_range { /* HV_SPA_PAGE_RANGE */
} __packed;
};
+enum hv_sleep_state {
+ HV_SLEEP_STATE_S1 = 1,
+ HV_SLEEP_STATE_S2 = 2,
+ HV_SLEEP_STATE_S3 = 3,
+ HV_SLEEP_STATE_S4 = 4,
+ HV_SLEEP_STATE_S5 = 5,
+ /*
+ * After hypervisor has received this, any follow up sleep
+ * state registration requests will be rejected.
+ */
+ HV_SLEEP_STATE_LOCK = 6
+};
+
enum hv_dynamic_processor_feature_property {
/* Add more values when needed */
HV_X64_DYNAMIC_PROCESSOR_FEATURE_MAX_ENCRYPTED_PARTITIONS = 13,
@@ -184,6 +198,32 @@ struct hv_output_get_system_property {
};
} __packed;
+struct hv_sleep_state_info {
+ u32 sleep_state; /* enum hv_sleep_state */
+ u8 pm1a_slp_typ;
+ u8 pm1b_slp_typ;
+} __packed;
+
+struct hv_input_set_system_property {
+ u32 property_id; /* enum hv_system_property */
+ u32 reserved;
+ union {
+ /* More fields to be filled in when needed */
+ struct hv_sleep_state_info set_sleep_state_info;
+
+ /*
+ * Add a reserved field to ensure the union is 8-byte aligned as
+ * existing members may not be. This is a temporary measure
+ * until all remaining members are added.
+ */
+ u64 reserved0[8];
+ };
+} __packed;
+
+struct hv_input_enter_sleep_state { /* HV_INPUT_ENTER_SLEEP_STATE */
+ u32 sleep_state; /* enum hv_sleep_state */
+} __packed;
+
struct hv_input_map_stats_page {
u32 type; /* enum hv_stats_object_type */
u32 padding;
--
2.51.0
^ permalink raw reply related
* [PATCH v5 2/3] hyperv: Use reboot notifier to configure sleep state
From: Praveen K Paladugu @ 2025-11-17 21:08 UTC (permalink / raw)
To: kys, haiyangz, wei.liu, decui, tglx, mingo, linux-hyperv,
linux-kernel, bp, dave.hansen, x86, hpa, arnd
Cc: anbelski, prapal, easwar.hariharan, nunodasneves, skinsburskii
In-Reply-To: <20251117210855.108126-1-prapal@linux.microsoft.com>
Configure sleep state information in mshv hypervisor. This sleep state
information from ACPI will be used by hypervisor to poweroff the host.
Signed-off-by: Praveen K Paladugu <prapal@linux.microsoft.com>
Co-developed-by: Anatol Belski <anbelski@linux.microsoft.com>
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Reviewed-by: Easwar Hariharan <easwar.hariharan@linux.microsoft.com>
---
arch/x86/hyperv/hv_init.c | 7 +++
arch/x86/include/asm/mshyperv.h | 2 +
drivers/hv/mshv_common.c | 78 +++++++++++++++++++++++++++++++++
3 files changed, 87 insertions(+)
diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
index e28737ec7054..645b52dd732e 100644
--- a/arch/x86/hyperv/hv_init.c
+++ b/arch/x86/hyperv/hv_init.c
@@ -555,6 +555,13 @@ void __init hyperv_init(void)
hv_remap_tsc_clocksource();
hv_root_crash_init();
+ /*
+ * The notifier registration might fail at various hops.
+ * Corresponding error messages will land in dmesg. There is
+ * otherwise nothing that can be specifically done to handle
+ * failures here.
+ */
+ hv_sleep_notifiers_register();
} else {
hypercall_msr.guest_physical_address = vmalloc_to_pfn(hv_hypercall_pg);
wrmsrq(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64);
diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
index 10037125099a..166053df0484 100644
--- a/arch/x86/include/asm/mshyperv.h
+++ b/arch/x86/include/asm/mshyperv.h
@@ -182,8 +182,10 @@ int hyperv_fill_flush_guest_mapping_list(
void hv_apic_init(void);
void __init hv_init_spinlocks(void);
bool hv_vcpu_is_preempted(int vcpu);
+void hv_sleep_notifiers_register(void);
#else
static inline void hv_apic_init(void) {}
+static inline void hv_sleep_notifiers_register(void) {};
#endif
struct irq_domain *hv_create_pci_msi_domain(void);
diff --git a/drivers/hv/mshv_common.c b/drivers/hv/mshv_common.c
index aa2be51979fd..ee733ba1575e 100644
--- a/drivers/hv/mshv_common.c
+++ b/drivers/hv/mshv_common.c
@@ -14,6 +14,9 @@
#include <asm/mshyperv.h>
#include <linux/resume_user_mode.h>
#include <linux/export.h>
+#include <linux/acpi.h>
+#include <linux/notifier.h>
+#include <linux/reboot.h>
#include "mshv.h"
@@ -138,3 +141,78 @@ int hv_call_get_partition_property(u64 partition_id,
return 0;
}
EXPORT_SYMBOL_GPL(hv_call_get_partition_property);
+
+/*
+ * Corresponding sleep states have to be initialized in order for a subsequent
+ * HVCALL_ENTER_SLEEP_STATE call to succeed. Currently only S5 state as per
+ * ACPI 6.4 chapter 7.4.2 is relevant, while S1, S2 and S3 can be supported.
+ *
+ * In order to pass proper PM values to mshv, ACPI should be initialized and
+ * should support S5 sleep state when this method is invoked.
+ */
+static int hv_initialize_sleep_states(void)
+{
+ u64 status;
+ unsigned long flags;
+ struct hv_input_set_system_property *in;
+ acpi_status acpi_status;
+ u8 sleep_type_a, sleep_type_b;
+
+ if (!acpi_sleep_state_supported(ACPI_STATE_S5)) {
+ pr_err("%s: S5 sleep state not supported.\n", __func__);
+ return -ENODEV;
+ }
+
+ acpi_status = acpi_get_sleep_type_data(ACPI_STATE_S5, &sleep_type_a,
+ &sleep_type_b);
+ if (ACPI_FAILURE(acpi_status))
+ return -ENODEV;
+
+ local_irq_save(flags);
+ in = *this_cpu_ptr(hyperv_pcpu_input_arg);
+ memset(in, 0, sizeof(*in));
+
+ in->property_id = HV_SYSTEM_PROPERTY_SLEEP_STATE;
+ in->set_sleep_state_info.sleep_state = HV_SLEEP_STATE_S5;
+ in->set_sleep_state_info.pm1a_slp_typ = sleep_type_a;
+ in->set_sleep_state_info.pm1b_slp_typ = sleep_type_b;
+
+ status = hv_do_hypercall(HVCALL_SET_SYSTEM_PROPERTY, in, NULL);
+ local_irq_restore(flags);
+
+ if (!hv_result_success(status)) {
+ hv_status_err(status, "\n");
+ return hv_result_to_errno(status);
+ }
+
+ return 0;
+}
+
+/*
+ * This notifier initializes sleep states in mshv hypervisor which will be
+ * used during power off.
+ */
+static int hv_reboot_notifier_handler(struct notifier_block *this,
+ unsigned long code, void *another)
+{
+ int ret = 0;
+
+ if (code == SYS_HALT || code == SYS_POWER_OFF)
+ ret = hv_initialize_sleep_states();
+
+ return ret ? NOTIFY_DONE : NOTIFY_OK;
+}
+
+static struct notifier_block hv_reboot_notifier = {
+ .notifier_call = hv_reboot_notifier_handler,
+};
+
+void hv_sleep_notifiers_register(void)
+{
+ int ret;
+
+ ret = register_reboot_notifier(&hv_reboot_notifier);
+ if (ret)
+ pr_err("%s: cannot register reboot notifier %d\n", __func__,
+ ret);
+}
--
2.51.0
^ permalink raw reply related
* [PATCH v5 3/3] hyperv: Cleanly shutdown root partition with MSHV
From: Praveen K Paladugu @ 2025-11-17 21:08 UTC (permalink / raw)
To: kys, haiyangz, wei.liu, decui, tglx, mingo, linux-hyperv,
linux-kernel, bp, dave.hansen, x86, hpa, arnd
Cc: anbelski, prapal, easwar.hariharan, nunodasneves, skinsburskii
In-Reply-To: <20251117210855.108126-1-prapal@linux.microsoft.com>
When a root partition running on MSHV is powered off, the default
behavior is to write ACPI registers to power-off. However, this ACPI
write is intercepted by MSHV and will result in a Machine Check
Exception(MCE).
The root partition eventually panics with a trace similar to:
[ 81.306348] reboot: Power down
[ 81.314709] mce: [Hardware Error]: CPU 0: Machine Check Exception: 4 Bank 0: b2000000c0060001
[ 81.314711] mce: [Hardware Error]: TSC 3b8cb60a66 PPIN 11d98332458e4ea9
[ 81.314713] mce: [Hardware Error]: PROCESSOR 0:606a6 TIME 1759339405 SOCKET 0 APIC 0 microcode ffffffff
[ 81.314715] mce: [Hardware Error]: Run the above through 'mcelog --ascii'
[ 81.314716] mce: [Hardware Error]: Machine check: Processor context corrupt
[ 81.314717] Kernel panic - not syncing: Fatal machine check
To correctly shutdown a root partition running on MSHV hypervisor, sleep
state information must be configured within the hypervsior. Later, the
HVCALL_ENTER_SLEEP_STATE hypercall should be invoked as the last step in
the shutdown sequence.
The previous patch configures the sleep state information and this patch
invokes HVCALL_ENTER_SLEEP_STATE hypercall to cleanly shutdown the root
partition.
Signed-off-by: Praveen K Paladugu <prapal@linux.microsoft.com>
Co-developed-by: Anatol Belski <anbelski@linux.microsoft.com>
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Reviewed-by: Easwar Hariharan <easwar.hariharan@linux.microsoft.com>
---
arch/x86/hyperv/hv_init.c | 2 ++
arch/x86/include/asm/mshyperv.h | 2 ++
drivers/hv/mshv_common.c | 18 ++++++++++++++++++
3 files changed, 22 insertions(+)
diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
index 645b52dd732e..24824534ff8d 100644
--- a/arch/x86/hyperv/hv_init.c
+++ b/arch/x86/hyperv/hv_init.c
@@ -34,6 +34,7 @@
#include <clocksource/hyperv_timer.h>
#include <linux/highmem.h>
#include <linux/export.h>
+#include <asm/reboot.h>
void *hv_hypercall_pg;
@@ -562,6 +563,7 @@ void __init hyperv_init(void)
* failures here.
*/
hv_sleep_notifiers_register();
+ machine_ops.power_off = hv_machine_power_off;
} else {
hypercall_msr.guest_physical_address = vmalloc_to_pfn(hv_hypercall_pg);
wrmsrq(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64);
diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
index 166053df0484..4c22f3257368 100644
--- a/arch/x86/include/asm/mshyperv.h
+++ b/arch/x86/include/asm/mshyperv.h
@@ -183,9 +183,11 @@ void hv_apic_init(void);
void __init hv_init_spinlocks(void);
bool hv_vcpu_is_preempted(int vcpu);
void hv_sleep_notifiers_register(void);
+void hv_machine_power_off(void);
#else
static inline void hv_apic_init(void) {}
static inline void hv_sleep_notifiers_register(void) {};
+static inline void hv_machine_power_off(void) {};
#endif
struct irq_domain *hv_create_pci_msi_domain(void);
diff --git a/drivers/hv/mshv_common.c b/drivers/hv/mshv_common.c
index ee733ba1575e..73505cbdc324 100644
--- a/drivers/hv/mshv_common.c
+++ b/drivers/hv/mshv_common.c
@@ -216,3 +216,21 @@ void hv_sleep_notifiers_register(void)
pr_err("%s: cannot register reboot notifier %d\n", __func__,
ret);
}
+
+/*
+ * Power off the machine by entering S5 sleep state via Hyper-V hypercall.
+ * This call does not return if successful.
+ */
+void hv_machine_power_off(void)
+{
+ unsigned long flags;
+ struct hv_input_enter_sleep_state *in;
+
+ local_irq_save(flags);
+ in = *this_cpu_ptr(hyperv_pcpu_input_arg);
+ in->sleep_state = HV_SLEEP_STATE_S5;
+
+ (void)hv_do_hypercall(HVCALL_ENTER_SLEEP_STATE, in, NULL);
+ local_irq_restore(flags);
+
+}
--
2.51.0
^ permalink raw reply related
* Re: [PATCH v4 1/3] hyperv: Add definitions for MSHV sleep state configuration
From: Praveen Paladugu @ 2025-11-17 21:10 UTC (permalink / raw)
To: Michael Kelley
Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org,
decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com,
linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org,
bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org,
hpa@zytor.com, arnd@arndb.de, anbelski@linux.microsoft.com,
easwar.hariharan@linux.microsoft.com,
nunodasneves@linux.microsoft.com,
skinsburskii@linux.microsoft.com
In-Reply-To: <SN6PR02MB4157FD3C208A593FB391932AD4CDA@SN6PR02MB4157.namprd02.prod.outlook.com>
On Thu, Nov 13, 2025 at 07:41:55PM +0000, Michael Kelley wrote:
> From: Praveen K Paladugu <prapal@linux.microsoft.com> Sent: Friday, November 7, 2025 2:17 PM
> >
> > Add the definitions required to configure sleep states in mshv hypervsior.
> >
> > Signed-off-by: Praveen K Paladugu <prapal@linux.microsoft.com>
> > Co-developed-by: Anatol Belski <anbelski@linux.microsoft.com>
> > Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
> > Reviewed-by: Easwar Hariharan <easwar.hariharan@linux.microsoft.com>
> > Reviewed-by: Nuno Das Neves <nunodasneves@linux.microsoft.com>
> > ---
> > include/hyperv/hvgdk_mini.h | 4 +++-
> > include/hyperv/hvhdk_mini.h | 33 +++++++++++++++++++++++++++++++++
> > 2 files changed, 36 insertions(+), 1 deletion(-)
> >
> > diff --git a/include/hyperv/hvgdk_mini.h b/include/hyperv/hvgdk_mini.h
> > index 7499a679e60a..b43fa1c9ed2c 100644
> > --- a/include/hyperv/hvgdk_mini.h
> > +++ b/include/hyperv/hvgdk_mini.h
> > @@ -465,19 +465,21 @@ union hv_vp_assist_msr_contents { /*
> > HV_REGISTER_VP_ASSIST_PAGE */
> > #define HVCALL_RESET_DEBUG_SESSION 0x006b
> > #define HVCALL_MAP_STATS_PAGE 0x006c
> > #define HVCALL_UNMAP_STATS_PAGE 0x006d
> > +#define HVCALL_SET_SYSTEM_PROPERTY 0x006f
> > #define HVCALL_ADD_LOGICAL_PROCESSOR 0x0076
> > #define HVCALL_GET_SYSTEM_PROPERTY 0x007b
> > #define HVCALL_MAP_DEVICE_INTERRUPT 0x007c
> > #define HVCALL_UNMAP_DEVICE_INTERRUPT 0x007d
> > #define HVCALL_RETARGET_INTERRUPT 0x007e
> > #define HVCALL_NOTIFY_PARTITION_EVENT 0x0087
> > +#define HVCALL_ENTER_SLEEP_STATE 0x0084
> > #define HVCALL_NOTIFY_PORT_RING_EMPTY 0x008b
> > #define HVCALL_REGISTER_INTERCEPT_RESULT 0x0091
> > #define HVCALL_ASSERT_VIRTUAL_INTERRUPT 0x0094
> > #define HVCALL_CREATE_PORT 0x0095
> > #define HVCALL_CONNECT_PORT 0x0096
> > #define HVCALL_START_VP 0x0099
> > -#define HVCALL_GET_VP_INDEX_FROM_APIC_ID 0x009a
> > +#define HVCALL_GET_VP_INDEX_FROM_APIC_ID 0x009a
> > #define HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_SPACE 0x00af
> > #define HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_LIST 0x00b0
> > #define HVCALL_SIGNAL_EVENT_DIRECT 0x00c0
> > diff --git a/include/hyperv/hvhdk_mini.h b/include/hyperv/hvhdk_mini.h
> > index f2d7b50de7a4..06cf30deb319 100644
> > --- a/include/hyperv/hvhdk_mini.h
> > +++ b/include/hyperv/hvhdk_mini.h
> > @@ -140,6 +140,7 @@ enum hv_snp_status {
> >
> > enum hv_system_property {
> > /* Add more values when needed */
> > + HV_SYSTEM_PROPERTY_SLEEP_STATE = 3,
> > HV_SYSTEM_PROPERTY_SCHEDULER_TYPE = 15,
> > HV_DYNAMIC_PROCESSOR_FEATURE_PROPERTY = 21,
> > HV_SYSTEM_PROPERTY_CRASHDUMPAREA = 47,
> > @@ -155,6 +156,19 @@ union hv_pfn_range { /* HV_SPA_PAGE_RANGE */
> > } __packed;
> > };
> >
> > +enum hv_sleep_state {
> > + HV_SLEEP_STATE_S1 = 1,
> > + HV_SLEEP_STATE_S2 = 2,
> > + HV_SLEEP_STATE_S3 = 3,
> > + HV_SLEEP_STATE_S4 = 4,
> > + HV_SLEEP_STATE_S5 = 5,
> > + /*
> > + * After hypervisor has received this, any follow up sleep
> > + * state registration requests will be rejected.
> > + */
> > + HV_SLEEP_STATE_LOCK = 6
> > +};
> > +
> > enum hv_dynamic_processor_feature_property {
> > /* Add more values when needed */
> > HV_X64_DYNAMIC_PROCESSOR_FEATURE_MAX_ENCRYPTED_PARTITIONS =
> > 13,
> > @@ -184,6 +198,25 @@ struct hv_output_get_system_property {
> > };
> > } __packed;
> >
> > +struct hv_sleep_state_info {
> > + u32 sleep_state; /* enum hv_sleep_state */
> > + u8 pm1a_slp_typ;
> > + u8 pm1b_slp_typ;
> > +} __packed;
> > +
> > +struct hv_input_set_system_property {
> > + u32 property_id; /* enum hv_system_property */
> > + u32 reserved;
> > + union {
> > + /* More fields to be filled in when needed */
> > + struct hv_sleep_state_info set_sleep_state_info;
> > + };
> > +} __packed;
>
> Because struct hv_sleep_state_info is 6 bytes long, this
> hypercall input struct ends up being 14 bytes long, which is
> unusual. Hyper-V almost always makes hypercall input size
> a multiple of 8 bytes, though in a few cases the last 4 bytes
> might be ignored, making it a multiple of 4 bytes. So I'm
> wondering if there should be a 2 byte "reserved" field in
> struct hv_sleep_state_info so that everything ends up
> being a multiple of 8 bytes.
>
> Since there aren't any fields after such a putative "reserved"
> field, there's nothing currently at the wrong offset. But it would
> affect how much space gets zero'ed in hv_initialize_sleep_states().
> Zero'ing those last 2 bytes might prove useful if future versions
> of the hypervisor were to use those 2 bytes for an additional
> field or two.
>
Nice catch Michael. I added a reserved field to the union to keep the struct
fully aligned and future proof.
> > +
> > +struct hv_input_enter_sleep_state { /* HV_INPUT_ENTER_SLEEP_STATE */
> > + u32 sleep_state; /* enum hv_sleep_state */
> > +} __packed;
> > +
> > struct hv_input_map_stats_page {
> > u32 type; /* enum hv_stats_object_type */
> > u32 padding;
> > --
> > 2.51.0
^ permalink raw reply
* Re: [PATCH v4 2/3] hyperv: Use reboot notifier to configure sleep state
From: Praveen Paladugu @ 2025-11-17 21:11 UTC (permalink / raw)
To: Michael Kelley
Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org,
decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com,
linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org,
bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org,
hpa@zytor.com, arnd@arndb.de, anbelski@linux.microsoft.com,
easwar.hariharan@linux.microsoft.com,
nunodasneves@linux.microsoft.com,
skinsburskii@linux.microsoft.com
In-Reply-To: <SN6PR02MB4157396FA3D70F49220C83FED4CDA@SN6PR02MB4157.namprd02.prod.outlook.com>
On Thu, Nov 13, 2025 at 07:42:00PM +0000, Michael Kelley wrote:
> From: Praveen K Paladugu <prapal@linux.microsoft.com> Sent: Friday, November 7, 2025 2:25 PM
> >
> > On 11/7/2025 4:16 PM, Praveen K Paladugu wrote:
> > > Configure sleep state information in mshv hypervisor. This sleep state
> > > information from ACPI will be used by hypervisor to poweroff the host.
> > >
> > > Signed-off-by: Praveen K Paladugu <prapal@linux.microsoft.com>
> > > Co-developed-by: Anatol Belski <anbelski@linux.microsoft.com>
> > > Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
> > > ---
> > > arch/x86/hyperv/hv_init.c | 7 +++
> > > arch/x86/include/asm/mshyperv.h | 2 +
> > > drivers/hv/mshv_common.c | 80 +++++++++++++++++++++++++++++++++
> > > 3 files changed, 89 insertions(+)
> > >
> > > diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
> > > index e28737ec7054..645b52dd732e 100644
> > > --- a/arch/x86/hyperv/hv_init.c
> > > +++ b/arch/x86/hyperv/hv_init.c
> > > @@ -555,6 +555,13 @@ void __init hyperv_init(void)
> > >
> > > hv_remap_tsc_clocksource();
> > > hv_root_crash_init();
> > > + /*
> > > + * The notifier registration might fail at various hops.
> > > + * Corresponding error messages will land in dmesg. There is
> > > + * otherwise nothing that can be specifically done to handle
> > > + * failures here.
> > > + */
> > > + hv_sleep_notifiers_register();
> > > } else {
> > > hypercall_msr.guest_physical_address = vmalloc_to_pfn(hv_hypercall_pg);
> > > wrmsrq(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64);
> > > diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
> > > index 1342d55c2545..fbc1233175ce 100644
> > > --- a/arch/x86/include/asm/mshyperv.h
> > > +++ b/arch/x86/include/asm/mshyperv.h
> > > @@ -181,8 +181,10 @@ int hyperv_fill_flush_guest_mapping_list(
> > > void hv_apic_init(void);
> > > void __init hv_init_spinlocks(void);
> > > bool hv_vcpu_is_preempted(int vcpu);
> > > +void hv_sleep_notifiers_register(void);
> > > #else
> > > static inline void hv_apic_init(void) {}
> > > +static inline void hv_sleep_notifiers_register(void) {};
> > > #endif
> > >
> > > struct irq_domain *hv_create_pci_msi_domain(void);
> > > diff --git a/drivers/hv/mshv_common.c b/drivers/hv/mshv_common.c
> > > index aa2be51979fd..d1a1daa52b65 100644
> > > --- a/drivers/hv/mshv_common.c
> > > +++ b/drivers/hv/mshv_common.c
> > > @@ -14,6 +14,9 @@
> > > #include <asm/mshyperv.h>
> > > #include <linux/resume_user_mode.h>
> > > #include <linux/export.h>
> > > +#include <linux/acpi.h>
> > > +#include <linux/notifier.h>
> > > +#include <linux/reboot.h>
> > >
> > > #include "mshv.h"
> > >
> > > @@ -138,3 +141,80 @@ int hv_call_get_partition_property(u64 partition_id,
> > > return 0;
> > > }
> > > EXPORT_SYMBOL_GPL(hv_call_get_partition_property);
> > > +
> > > +#if IS_ENABLED(CONFIG_ACPI)
>
> Is gating on CONFIG_ACPI necessary? I thought this might be leftover from earlier
> versions when you were calling acpi_os_set_prepare_sleep() and
> acpi_os_set_prepare_extended_sleep(). In my testing, this current version
> compiles OK if the gating is removed and CONFIG_ACPI=n, as the ACPI functions
> you are calling have stubs in that case.
>
I re-checked and confirmed this gating is no longer necessary. I dropped
it in v5.
> > > +/*
> > > + * Corresponding sleep states have to be initialized in order for a subsequent
> > > + * HVCALL_ENTER_SLEEP_STATE call to succeed. Currently only S5 state as per
> > > + * ACPI 6.4 chapter 7.4.2 is relevant, while S1, S2 and S3 can be supported.
> > > + *
> > > + * In order to pass proper PM values to mshv, ACPI should be initialized and
> > > + * should support S5 sleep state when this method is invoked.
> > > + */
> > > +static int hv_initialize_sleep_states(void)
> > > +{
> > > + u64 status;
> > > + unsigned long flags;
> > > + struct hv_input_set_system_property *in;
> > > + acpi_status acpi_status;
> > > + u8 sleep_type_a, sleep_type_b;
> > > +
> > > + if (!acpi_sleep_state_supported(ACPI_STATE_S5)) {
> > > + pr_err("%s: S5 sleep state not supported.\n", __func__);
> > > + return -ENODEV;
> > > + }
> > > +
> > > + acpi_status = acpi_get_sleep_type_data(ACPI_STATE_S5, &sleep_type_a,
> > > + &sleep_type_b);
> > > + if (ACPI_FAILURE(acpi_status))
> > > + return -ENODEV;
> > > +
> > > + local_irq_save(flags);
> > > + in = *this_cpu_ptr(hyperv_pcpu_input_arg);
> > > + memset(in, 0, sizeof(*in));
> > > +
> > > + in->property_id = HV_SYSTEM_PROPERTY_SLEEP_STATE;
> > > + in->set_sleep_state_info.sleep_state = HV_SLEEP_STATE_S5;
> > > + in->set_sleep_state_info.pm1a_slp_typ = sleep_type_a;
> > > + in->set_sleep_state_info.pm1b_slp_typ = sleep_type_b;
> > > +
> > > + status = hv_do_hypercall(HVCALL_SET_SYSTEM_PROPERTY, in, NULL);
> > > + local_irq_restore(flags);
> > > +
> > > + if (!hv_result_success(status)) {
> > > + hv_status_err(status, "\n");
> > > + return hv_result_to_errno(status);
> > > + }
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +/*
> > > + * This notifier initializes sleep states in mshv hypervisor which will be
> > > + * used during power off.
> > > + */
> > > +static int hv_reboot_notifier_handler(struct notifier_block *this,
> > > + unsigned long code, void *another)
> > > +{
> > > + int ret = 0;
> > > +
> > > + if (code == SYS_POWER_OFF)
> >
> > This patchset only addresses poweroff.
> > Reboot currently works via ACPI. Nothing more is needed there.
> >
> > `kernel_halt` needs the use of a different hypercall. Support for
> > Halting will be introduced later.
> > > + ret = hv_initialize_sleep_states();
> > > +
> > > + return ret ? NOTIFY_DONE : NOTIFY_OK;
> > > +}
> > > +
> > > +static struct notifier_block hv_reboot_notifier = {
> > > + .notifier_call = hv_reboot_notifier_handler,
> > > +};
> > > +
> > > +void hv_sleep_notifiers_register(void)
> > > +{
> > > + int ret;
> > > +
> > > + ret = register_reboot_notifier(&hv_reboot_notifier);
> > > + if (ret)
> > > + pr_err("%s: cannot register reboot notifier %d\n", __func__,
> > > + ret);
> > > +}
> > > +#endif
^ permalink raw reply
* Re: [PATCH v4 3/3] hyperv: Cleanly shutdown root partition with MSHV
From: Praveen Paladugu @ 2025-11-17 21:12 UTC (permalink / raw)
To: Michael Kelley
Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org,
decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com,
linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org,
bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org,
hpa@zytor.com, arnd@arndb.de, anbelski@linux.microsoft.com,
easwar.hariharan@linux.microsoft.com,
nunodasneves@linux.microsoft.com,
skinsburskii@linux.microsoft.com
In-Reply-To: <SN6PR02MB415764759BCA5070B8303AF2D4CDA@SN6PR02MB4157.namprd02.prod.outlook.com>
On Thu, Nov 13, 2025 at 07:42:10PM +0000, Michael Kelley wrote:
> From: Praveen K Paladugu <prapal@linux.microsoft.com>
> >
> > When a root partition running on MSHV is powered off, the default
> > behavior is to write ACPI registers to power-off. However, this ACPI
> > write is intercepted by MSHV and will result in a Machine Check
> > Exception(MCE).
> >
> > The root partition eventually panics with a trace similar to:
> >
> > [ 81.306348] reboot: Power down
> > [ 81.314709] mce: [Hardware Error]: CPU 0: Machine Check Exception: 4 Bank 0: b2000000c0060001
> > [ 81.314711] mce: [Hardware Error]: TSC 3b8cb60a66 PPIN 11d98332458e4ea9
> > [ 81.314713] mce: [Hardware Error]: PROCESSOR 0:606a6 TIME 1759339405 SOCKET 0 APIC 0 microcode ffffffff
> > [ 81.314715] mce: [Hardware Error]: Run the above through 'mcelog --ascii'
> > [ 81.314716] mce: [Hardware Error]: Machine check: Processor context corrupt
> > [ 81.314717] Kernel panic - not syncing: Fatal machine check
> >
> > To correctly shutdown a root partition running on MSHV, sleep state
> > information has be configured within mshv. Later HVCALL_ENTER_SLEEP_STATE
>
> s/has be/has to be/ --or-- s/has be/must be/
>
> Nit: Be consistent in capitalizing "MSHV" (or not capitalizing it).
>
> > should be invoked as the last step in the shutdown sequence.
> >
> > The previous patch configures the sleep state information and this patch
> > invokes HVCALL_ENTER_SLEEP_STATE to cleanly shutdown the root partition.
> >
> > Signed-off-by: Praveen K Paladugu <prapal@linux.microsoft.com>
> > Co-developed-by: Anatol Belski <anbelski@linux.microsoft.com>
> > Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
> > ---
> > arch/x86/hyperv/hv_init.c | 2 ++
> > arch/x86/include/asm/mshyperv.h | 2 ++
> > drivers/hv/mshv_common.c | 19 +++++++++++++++++++
> > 3 files changed, 23 insertions(+)
> >
> > diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
> > index 645b52dd732e..24824534ff8d 100644
> > --- a/arch/x86/hyperv/hv_init.c
> > +++ b/arch/x86/hyperv/hv_init.c
> > @@ -34,6 +34,7 @@
> > #include <clocksource/hyperv_timer.h>
> > #include <linux/highmem.h>
> > #include <linux/export.h>
> > +#include <asm/reboot.h>
> >
> > void *hv_hypercall_pg;
> >
> > @@ -562,6 +563,7 @@ void __init hyperv_init(void)
> > * failures here.
> > */
> > hv_sleep_notifiers_register();
> > + machine_ops.power_off = hv_machine_power_off;
> > } else {
> > hypercall_msr.guest_physical_address = vmalloc_to_pfn(hv_hypercall_pg);
> > wrmsrq(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64);
> > diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
> > index fbc1233175ce..9082d56103ce 100644
> > --- a/arch/x86/include/asm/mshyperv.h
> > +++ b/arch/x86/include/asm/mshyperv.h
> > @@ -182,9 +182,11 @@ void hv_apic_init(void);
> > void __init hv_init_spinlocks(void);
> > bool hv_vcpu_is_preempted(int vcpu);
> > void hv_sleep_notifiers_register(void);
> > +void hv_machine_power_off(void);
> > #else
> > static inline void hv_apic_init(void) {}
> > static inline void hv_sleep_notifiers_register(void) {};
> > +static inline void hv_machine_power_off(void) {};
> > #endif
> >
> > struct irq_domain *hv_create_pci_msi_domain(void);
> > diff --git a/drivers/hv/mshv_common.c b/drivers/hv/mshv_common.c
> > index d1a1daa52b65..0588d293a92a 100644
> > --- a/drivers/hv/mshv_common.c
> > +++ b/drivers/hv/mshv_common.c
> > @@ -217,4 +217,23 @@ void hv_sleep_notifiers_register(void)
> > pr_err("%s: cannot register reboot notifier %d\n", __func__,
> > ret);
> > }
> > +
> > +/*
> > + * Power off the machine by entering S5 sleep state via Hyper-V hypercall.
> > + * This call does not return if successful.
> > + */
> > +void hv_machine_power_off(void)
> > +{
> > + u64 status;
> > + unsigned long flags;
> > + struct hv_input_enter_sleep_state *in;
> > +
> > + local_irq_save(flags);
> > + in = *this_cpu_ptr(hyperv_pcpu_input_arg);
> > + in->sleep_state = HV_SLEEP_STATE_S5;
> > +
> > + status = hv_do_hypercall(HVCALL_ENTER_SLEEP_STATE, in, NULL);
>
> As flagged by the kernel test robot, this should be
>
> (void)hv_do_hypercall(HVCALL_ENTER_SLEEP_STATE, in, NULL);
>
> so that the intent to ignore the return value is explicit. And the local
> variable "status" can be removed.
>
Thanks for the feedback. I addressed it in v5.
> > + local_irq_restore(flags);
> > +
> > +}
> > #endif
> > --
> > 2.51.0
^ permalink raw reply
* Re: [PATCH v5 2/3] hyperv: Use reboot notifier to configure sleep state
From: Stanislav Kinsburskii @ 2025-11-17 23:37 UTC (permalink / raw)
To: Praveen K Paladugu
Cc: kys, haiyangz, wei.liu, decui, tglx, mingo, linux-hyperv,
linux-kernel, bp, dave.hansen, x86, hpa, arnd, anbelski,
easwar.hariharan, nunodasneves
In-Reply-To: <20251117210855.108126-3-prapal@linux.microsoft.com>
On Mon, Nov 17, 2025 at 03:08:17PM -0600, Praveen K Paladugu wrote:
> Configure sleep state information in mshv hypervisor. This sleep state
> information from ACPI will be used by hypervisor to poweroff the host.
>
> Signed-off-by: Praveen K Paladugu <prapal@linux.microsoft.com>
> Co-developed-by: Anatol Belski <anbelski@linux.microsoft.com>
> Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
> Reviewed-by: Easwar Hariharan <easwar.hariharan@linux.microsoft.com>
> ---
> arch/x86/hyperv/hv_init.c | 7 +++
> arch/x86/include/asm/mshyperv.h | 2 +
> drivers/hv/mshv_common.c | 78 +++++++++++++++++++++++++++++++++
> 3 files changed, 87 insertions(+)
>
> diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
> index e28737ec7054..645b52dd732e 100644
> --- a/arch/x86/hyperv/hv_init.c
> +++ b/arch/x86/hyperv/hv_init.c
> @@ -555,6 +555,13 @@ void __init hyperv_init(void)
>
> hv_remap_tsc_clocksource();
> hv_root_crash_init();
> + /*
> + * The notifier registration might fail at various hops.
> + * Corresponding error messages will land in dmesg. There is
> + * otherwise nothing that can be specifically done to handle
> + * failures here.
> + */
I'd suggest removing this comment: it doesn't bring much value.
Reviewed-by: Stansialv Kinsburskii <skinsburskii@linux.miscrosoft.com>
> + hv_sleep_notifiers_register();
> } else {
> hypercall_msr.guest_physical_address = vmalloc_to_pfn(hv_hypercall_pg);
> wrmsrq(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64);
> diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
> index 10037125099a..166053df0484 100644
> --- a/arch/x86/include/asm/mshyperv.h
> +++ b/arch/x86/include/asm/mshyperv.h
> @@ -182,8 +182,10 @@ int hyperv_fill_flush_guest_mapping_list(
> void hv_apic_init(void);
> void __init hv_init_spinlocks(void);
> bool hv_vcpu_is_preempted(int vcpu);
> +void hv_sleep_notifiers_register(void);
> #else
> static inline void hv_apic_init(void) {}
> +static inline void hv_sleep_notifiers_register(void) {};
> #endif
>
> struct irq_domain *hv_create_pci_msi_domain(void);
> diff --git a/drivers/hv/mshv_common.c b/drivers/hv/mshv_common.c
> index aa2be51979fd..ee733ba1575e 100644
> --- a/drivers/hv/mshv_common.c
> +++ b/drivers/hv/mshv_common.c
> @@ -14,6 +14,9 @@
> #include <asm/mshyperv.h>
> #include <linux/resume_user_mode.h>
> #include <linux/export.h>
> +#include <linux/acpi.h>
> +#include <linux/notifier.h>
> +#include <linux/reboot.h>
>
> #include "mshv.h"
>
> @@ -138,3 +141,78 @@ int hv_call_get_partition_property(u64 partition_id,
> return 0;
> }
> EXPORT_SYMBOL_GPL(hv_call_get_partition_property);
> +
> +/*
> + * Corresponding sleep states have to be initialized in order for a subsequent
> + * HVCALL_ENTER_SLEEP_STATE call to succeed. Currently only S5 state as per
> + * ACPI 6.4 chapter 7.4.2 is relevant, while S1, S2 and S3 can be supported.
> + *
> + * In order to pass proper PM values to mshv, ACPI should be initialized and
> + * should support S5 sleep state when this method is invoked.
> + */
> +static int hv_initialize_sleep_states(void)
> +{
> + u64 status;
> + unsigned long flags;
> + struct hv_input_set_system_property *in;
> + acpi_status acpi_status;
> + u8 sleep_type_a, sleep_type_b;
> +
> + if (!acpi_sleep_state_supported(ACPI_STATE_S5)) {
> + pr_err("%s: S5 sleep state not supported.\n", __func__);
> + return -ENODEV;
> + }
> +
> + acpi_status = acpi_get_sleep_type_data(ACPI_STATE_S5, &sleep_type_a,
> + &sleep_type_b);
> + if (ACPI_FAILURE(acpi_status))
> + return -ENODEV;
> +
> + local_irq_save(flags);
> + in = *this_cpu_ptr(hyperv_pcpu_input_arg);
> + memset(in, 0, sizeof(*in));
> +
> + in->property_id = HV_SYSTEM_PROPERTY_SLEEP_STATE;
> + in->set_sleep_state_info.sleep_state = HV_SLEEP_STATE_S5;
> + in->set_sleep_state_info.pm1a_slp_typ = sleep_type_a;
> + in->set_sleep_state_info.pm1b_slp_typ = sleep_type_b;
> +
> + status = hv_do_hypercall(HVCALL_SET_SYSTEM_PROPERTY, in, NULL);
> + local_irq_restore(flags);
> +
> + if (!hv_result_success(status)) {
> + hv_status_err(status, "\n");
> + return hv_result_to_errno(status);
> + }
> +
> + return 0;
> +}
> +
> +/*
> + * This notifier initializes sleep states in mshv hypervisor which will be
> + * used during power off.
> + */
> +static int hv_reboot_notifier_handler(struct notifier_block *this,
> + unsigned long code, void *another)
> +{
> + int ret = 0;
> +
> + if (code == SYS_HALT || code == SYS_POWER_OFF)
> + ret = hv_initialize_sleep_states();
> +
> + return ret ? NOTIFY_DONE : NOTIFY_OK;
> +}
> +
> +static struct notifier_block hv_reboot_notifier = {
> + .notifier_call = hv_reboot_notifier_handler,
> +};
> +
> +void hv_sleep_notifiers_register(void)
> +{
> + int ret;
> +
> + ret = register_reboot_notifier(&hv_reboot_notifier);
> + if (ret)
> + pr_err("%s: cannot register reboot notifier %d\n", __func__,
> + ret);
> +}
> --
> 2.51.0
^ permalink raw reply
* Re: [PATCH v2] Drivers: hv: ioctl for self targeted passthrough hvcalls
From: Nuno Das Neves @ 2025-11-17 23:42 UTC (permalink / raw)
To: Wei Liu
Cc: Anirudh Rayabharam, K. Y. Srinivasan, Haiyang Zhang, Dexuan Cui,
Long Li, linux-hyperv, linux-kernel
In-Reply-To: <20251117192402.GA2402579@liuwe-devbox-debian-v2.local>
On 11/17/2025 11:24 AM, Wei Liu wrote:
> On Mon, Nov 17, 2025 at 07:18:27PM +0000, Wei Liu wrote:
>> On Mon, Nov 17, 2025 at 10:16:12AM -0800, Nuno Das Neves wrote:
>>> On 11/17/2025 1:52 AM, Anirudh Rayabharam wrote:
>>>> From: Anirudh Rayabharam (Microsoft) <anirudh@anirudhrb.com>
>>>>
>>>> Allow MSHV_ROOT_HVCALL IOCTL on the /dev/mshv fd. This IOCTL would
>>>> execute a passthrough hypercall targeting the root/parent partition
>>>> i.e. HV_PARTITION_ID_SELF.
>>>>
>>>
>>> I think it's worth taking a moment to check and perhaps explain in
>>> the commit message/a comment any security implications of the VMM
>>> process being able to call these hypercalls on the root/parent
>>> partition.
>>>
>>> One implication would be: can the VMM process influence other
>>> processes in the root partition via these hypercalls,
>>> e.g. HVCALL_SET_VP_REGISTERS? I would think that the hypervisor
>>> itself disallows this but we should check. We can ask the
>>> hypervisor team what they think, and check the hypervisor code.
>>>
>>> Specifically we should check on any hypercall that could possibly
>>> influence partition state, i.e.:
>>> HVCALL_SET_PARTITION_PROPERTY
>>> HVCALL_SET_VP_REGISTERS
>>> HVCALL_INSTALL_INTERCEPT
>>> HVCALL_CLEAR_VIRTUAL_INTERRUPT
>>> HVCALL_REGISTER_INTERCEPT_RESULT
>>> HVCALL_ASSERT_VIRTUAL_INTERRUPT
>>> HVCALL_SIGNAL_EVENT_DIRECT
>>> HVCALL_POST_MESSAGE_DIRECT
>>>
>>> If it turns out there is something risky we are enabling here, we can
>>> introduce a new array of hypercalls to restrict which ones can be
>>> called on HV_PARTITION_ID_SELF.
>>>
>>
>> This is a good point. Please check with the hypervisor team.
>
> I should add: it is always easier to relax restrictions later than to
> add them back in, so if there is any doubt and we want this code in as
> quickly as possible, we can start with a new array and expand it later.
>
Agreed. I think that's a good approach here, we can just enable
HVCALL_GET_PARTITION_PROPERTY and HVCALL_GET_PARTITION_PROPERTY_EX for
self-targeted passthru hypercalls.
> Wei
^ permalink raw reply
* Re: [PATCH v5 3/3] hyperv: Cleanly shutdown root partition with MSHV
From: Stanislav Kinsburskii @ 2025-11-17 23:45 UTC (permalink / raw)
To: Praveen K Paladugu
Cc: kys, haiyangz, wei.liu, decui, tglx, mingo, linux-hyperv,
linux-kernel, bp, dave.hansen, x86, hpa, arnd, anbelski,
easwar.hariharan, nunodasneves
In-Reply-To: <20251117210855.108126-4-prapal@linux.microsoft.com>
On Mon, Nov 17, 2025 at 03:08:18PM -0600, Praveen K Paladugu wrote:
> When a root partition running on MSHV is powered off, the default
> behavior is to write ACPI registers to power-off. However, this ACPI
> write is intercepted by MSHV and will result in a Machine Check
> Exception(MCE).
>
> The root partition eventually panics with a trace similar to:
>
> [ 81.306348] reboot: Power down
> [ 81.314709] mce: [Hardware Error]: CPU 0: Machine Check Exception: 4 Bank 0: b2000000c0060001
> [ 81.314711] mce: [Hardware Error]: TSC 3b8cb60a66 PPIN 11d98332458e4ea9
> [ 81.314713] mce: [Hardware Error]: PROCESSOR 0:606a6 TIME 1759339405 SOCKET 0 APIC 0 microcode ffffffff
> [ 81.314715] mce: [Hardware Error]: Run the above through 'mcelog --ascii'
> [ 81.314716] mce: [Hardware Error]: Machine check: Processor context corrupt
> [ 81.314717] Kernel panic - not syncing: Fatal machine check
>
> To correctly shutdown a root partition running on MSHV hypervisor, sleep
> state information must be configured within the hypervsior. Later, the
> HVCALL_ENTER_SLEEP_STATE hypercall should be invoked as the last step in
> the shutdown sequence.
>
> The previous patch configures the sleep state information and this patch
> invokes HVCALL_ENTER_SLEEP_STATE hypercall to cleanly shutdown the root
> partition.
>
> Signed-off-by: Praveen K Paladugu <prapal@linux.microsoft.com>
> Co-developed-by: Anatol Belski <anbelski@linux.microsoft.com>
> Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
> Reviewed-by: Easwar Hariharan <easwar.hariharan@linux.microsoft.com>
> ---
> arch/x86/hyperv/hv_init.c | 2 ++
> arch/x86/include/asm/mshyperv.h | 2 ++
> drivers/hv/mshv_common.c | 18 ++++++++++++++++++
> 3 files changed, 22 insertions(+)
>
> diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
> index 645b52dd732e..24824534ff8d 100644
> --- a/arch/x86/hyperv/hv_init.c
> +++ b/arch/x86/hyperv/hv_init.c
> @@ -34,6 +34,7 @@
> #include <clocksource/hyperv_timer.h>
> #include <linux/highmem.h>
> #include <linux/export.h>
> +#include <asm/reboot.h>
>
> void *hv_hypercall_pg;
>
> @@ -562,6 +563,7 @@ void __init hyperv_init(void)
> * failures here.
> */
> hv_sleep_notifiers_register();
> + machine_ops.power_off = hv_machine_power_off;
It looks more natural to me to gather all the machine_ops hooks in one
place (meaning in ms_hyperv_init_platform).
It is better moving this assignment there and do the branching on the
partition type in the power_off callback instead.
Thanks,
Stanislav
> } else {
> hypercall_msr.guest_physical_address = vmalloc_to_pfn(hv_hypercall_pg);
> wrmsrq(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64);
> diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
> index 166053df0484..4c22f3257368 100644
> --- a/arch/x86/include/asm/mshyperv.h
> +++ b/arch/x86/include/asm/mshyperv.h
> @@ -183,9 +183,11 @@ void hv_apic_init(void);
> void __init hv_init_spinlocks(void);
> bool hv_vcpu_is_preempted(int vcpu);
> void hv_sleep_notifiers_register(void);
> +void hv_machine_power_off(void);
> #else
> static inline void hv_apic_init(void) {}
> static inline void hv_sleep_notifiers_register(void) {};
> +static inline void hv_machine_power_off(void) {};
> #endif
>
> struct irq_domain *hv_create_pci_msi_domain(void);
> diff --git a/drivers/hv/mshv_common.c b/drivers/hv/mshv_common.c
> index ee733ba1575e..73505cbdc324 100644
> --- a/drivers/hv/mshv_common.c
> +++ b/drivers/hv/mshv_common.c
> @@ -216,3 +216,21 @@ void hv_sleep_notifiers_register(void)
> pr_err("%s: cannot register reboot notifier %d\n", __func__,
> ret);
> }
> +
> +/*
> + * Power off the machine by entering S5 sleep state via Hyper-V hypercall.
> + * This call does not return if successful.
> + */
> +void hv_machine_power_off(void)
> +{
> + unsigned long flags;
> + struct hv_input_enter_sleep_state *in;
> +
> + local_irq_save(flags);
> + in = *this_cpu_ptr(hyperv_pcpu_input_arg);
> + in->sleep_state = HV_SLEEP_STATE_S5;
> +
> + (void)hv_do_hypercall(HVCALL_ENTER_SLEEP_STATE, in, NULL);
> + local_irq_restore(flags);
> +
> +}
> --
> 2.51.0
^ permalink raw reply
* Re: [PATCH v5 1/3] hyperv: Add definitions for MSHV sleep state configuration
From: Stanislav Kinsburskii @ 2025-11-17 23:46 UTC (permalink / raw)
To: Praveen K Paladugu
Cc: kys, haiyangz, wei.liu, decui, tglx, mingo, linux-hyperv,
linux-kernel, bp, dave.hansen, x86, hpa, arnd, anbelski,
easwar.hariharan, nunodasneves
In-Reply-To: <20251117210855.108126-2-prapal@linux.microsoft.com>
On Mon, Nov 17, 2025 at 03:08:16PM -0600, Praveen K Paladugu wrote:
> Add the definitions required to configure sleep states in mshv hypervsior.
>
> Signed-off-by: Praveen K Paladugu <prapal@linux.microsoft.com>
> Co-developed-by: Anatol Belski <anbelski@linux.microsoft.com>
> Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
> Reviewed-by: Easwar Hariharan <easwar.hariharan@linux.microsoft.com>
> Reviewed-by: Nuno Das Neves <nunodasneves@linux.microsoft.com>
> ---
> include/hyperv/hvgdk_mini.h | 4 +++-
> include/hyperv/hvhdk_mini.h | 40 +++++++++++++++++++++++++++++++++++++
> 2 files changed, 43 insertions(+), 1 deletion(-)
>
> diff --git a/include/hyperv/hvgdk_mini.h b/include/hyperv/hvgdk_mini.h
> index 1d5ce11be8b6..04b18d0e37af 100644
> --- a/include/hyperv/hvgdk_mini.h
> +++ b/include/hyperv/hvgdk_mini.h
> @@ -465,19 +465,21 @@ union hv_vp_assist_msr_contents { /* HV_REGISTER_VP_ASSIST_PAGE */
> #define HVCALL_RESET_DEBUG_SESSION 0x006b
> #define HVCALL_MAP_STATS_PAGE 0x006c
> #define HVCALL_UNMAP_STATS_PAGE 0x006d
> +#define HVCALL_SET_SYSTEM_PROPERTY 0x006f
> #define HVCALL_ADD_LOGICAL_PROCESSOR 0x0076
> #define HVCALL_GET_SYSTEM_PROPERTY 0x007b
> #define HVCALL_MAP_DEVICE_INTERRUPT 0x007c
> #define HVCALL_UNMAP_DEVICE_INTERRUPT 0x007d
> #define HVCALL_RETARGET_INTERRUPT 0x007e
> #define HVCALL_NOTIFY_PARTITION_EVENT 0x0087
> +#define HVCALL_ENTER_SLEEP_STATE 0x0084
> #define HVCALL_NOTIFY_PORT_RING_EMPTY 0x008b
> #define HVCALL_REGISTER_INTERCEPT_RESULT 0x0091
> #define HVCALL_ASSERT_VIRTUAL_INTERRUPT 0x0094
> #define HVCALL_CREATE_PORT 0x0095
> #define HVCALL_CONNECT_PORT 0x0096
> #define HVCALL_START_VP 0x0099
> -#define HVCALL_GET_VP_INDEX_FROM_APIC_ID 0x009a
> +#define HVCALL_GET_VP_INDEX_FROM_APIC_ID 0x009a
> #define HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_SPACE 0x00af
> #define HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_LIST 0x00b0
> #define HVCALL_SIGNAL_EVENT_DIRECT 0x00c0
> diff --git a/include/hyperv/hvhdk_mini.h b/include/hyperv/hvhdk_mini.h
> index f2d7b50de7a4..41a29bf8ec14 100644
> --- a/include/hyperv/hvhdk_mini.h
> +++ b/include/hyperv/hvhdk_mini.h
> @@ -140,6 +140,7 @@ enum hv_snp_status {
>
> enum hv_system_property {
> /* Add more values when needed */
> + HV_SYSTEM_PROPERTY_SLEEP_STATE = 3,
> HV_SYSTEM_PROPERTY_SCHEDULER_TYPE = 15,
> HV_DYNAMIC_PROCESSOR_FEATURE_PROPERTY = 21,
> HV_SYSTEM_PROPERTY_CRASHDUMPAREA = 47,
> @@ -155,6 +156,19 @@ union hv_pfn_range { /* HV_SPA_PAGE_RANGE */
> } __packed;
> };
>
> +enum hv_sleep_state {
> + HV_SLEEP_STATE_S1 = 1,
> + HV_SLEEP_STATE_S2 = 2,
> + HV_SLEEP_STATE_S3 = 3,
> + HV_SLEEP_STATE_S4 = 4,
Why adding the unused states?
It's not aligned with the common approach for this driver where we add
only those definitions which are used.
Thanks,
Stanislav
> + HV_SLEEP_STATE_S5 = 5,
> + /*
> + * After hypervisor has received this, any follow up sleep
> + * state registration requests will be rejected.
> + */
> + HV_SLEEP_STATE_LOCK = 6
> +};
> +
> enum hv_dynamic_processor_feature_property {
> /* Add more values when needed */
> HV_X64_DYNAMIC_PROCESSOR_FEATURE_MAX_ENCRYPTED_PARTITIONS = 13,
> @@ -184,6 +198,32 @@ struct hv_output_get_system_property {
> };
> } __packed;
>
> +struct hv_sleep_state_info {
> + u32 sleep_state; /* enum hv_sleep_state */
> + u8 pm1a_slp_typ;
> + u8 pm1b_slp_typ;
> +} __packed;
> +
> +struct hv_input_set_system_property {
> + u32 property_id; /* enum hv_system_property */
> + u32 reserved;
> + union {
> + /* More fields to be filled in when needed */
> + struct hv_sleep_state_info set_sleep_state_info;
> +
> + /*
> + * Add a reserved field to ensure the union is 8-byte aligned as
> + * existing members may not be. This is a temporary measure
> + * until all remaining members are added.
> + */
> + u64 reserved0[8];
> + };
> +} __packed;
> +
> +struct hv_input_enter_sleep_state { /* HV_INPUT_ENTER_SLEEP_STATE */
> + u32 sleep_state; /* enum hv_sleep_state */
> +} __packed;
> +
> struct hv_input_map_stats_page {
> u32 type; /* enum hv_stats_object_type */
> u32 padding;
> --
> 2.51.0
^ permalink raw reply
* Re: [PATCH v4] mshv: Extend create partition ioctl to support cpu features
From: Stanislav Kinsburskii @ 2025-11-17 23:58 UTC (permalink / raw)
To: Nuno Das Neves
Cc: linux-hyperv, linux-kernel, wei.liu, mhklinux, kys, haiyangz,
decui, longli, prapal, mrathor, muislam, anrayabh, Jinank Jain
In-Reply-To: <1762903194-25195-1-git-send-email-nunodasneves@linux.microsoft.com>
On Tue, Nov 11, 2025 at 03:19:54PM -0800, Nuno Das Neves wrote:
> From: Muminul Islam <muislam@microsoft.com>
>
> The existing mshv create partition ioctl does not provide a way to
> specify which cpu features are enabled in the guest. Instead, it
> attempts to enable all features and those that are not supported are
> silently disabled by the hypervisor.
>
> This was done to reduce unnecessary complexity and is sufficient for
> many cases. However, new scenarios require fine-grained control over
> these features.
>
> Define a new mshv_create_partition_v2 structure which supports
> passing the disabled processor and xsave feature bits through to the
> create partition hypercall directly.
>
> Introduce a new flag MSHV_PT_BIT_CPU_AND_XSAVE_FEATURES which enables
> the new structure. If unset, the original mshv_create_partition struct
> is used, with the old behavior of enabling all features.
>
> Co-developed-by: Jinank Jain <jinankjain@microsoft.com>
> Signed-off-by: Jinank Jain <jinankjain@microsoft.com>
> Signed-off-by: Muminul Islam <muislam@microsoft.com>
> Signed-off-by: Nuno Das Neves <nunodasneves@linux.microsoft.com>
> ---
> Changes in v4:
> - Change BIT() to BIT_ULL() [Michael Kelley]
> - Enforce pt_num_cpu_fbanks == MSHV_NUM_CPU_FEATURES_BANKS and expect
> that number to never change. In future, additional processor banks
> will be settable as 'early' partition properties. Remove redundant
> code that set default values for unspecified banks [Michael Kelley]
> - Set xsave features to 0 on arm64 [Michael Kelley]
> - Add clarifying comments in a few places
>
> Changes in v3:
> - Remove the new cpu features definitions in hvhdk.h, and retain the
> old behavior of enabling all features for the old struct. For the v2
> struct, still disable unspecified feature banks, since that makes it
> robust to future extensions.
> - Amend comments and commit message to reflect the above
> - Fix unused variable on arm64 [kernel test robot]
>
> Changes in v2:
> - Fix exposure of CONFIG_X86_64 to uapi [kernel test robot]
> - Fix compilation issue on arm64 [kernel test robot]
> ---
> drivers/hv/mshv_root_main.c | 113 +++++++++++++++++++++++++++++-------
> include/uapi/linux/mshv.h | 34 +++++++++++
> 2 files changed, 126 insertions(+), 21 deletions(-)
>
> diff --git a/drivers/hv/mshv_root_main.c b/drivers/hv/mshv_root_main.c
> index d542a0143bb8..9f9438289b60 100644
> --- a/drivers/hv/mshv_root_main.c
> +++ b/drivers/hv/mshv_root_main.c
> @@ -1900,43 +1900,114 @@ add_partition(struct mshv_partition *partition)
> return 0;
> }
>
> -static long
> -mshv_ioctl_create_partition(void __user *user_arg, struct device *module_dev)
> +static_assert(MSHV_NUM_CPU_FEATURES_BANKS ==
> + HV_PARTITION_PROCESSOR_FEATURES_BANKS);
> +
> +static long mshv_ioctl_process_pt_flags(void __user *user_arg, u64 *pt_flags,
> + struct hv_partition_creation_properties *cr_props,
> + union hv_partition_isolation_properties *isol_props)
> {
> - struct mshv_create_partition args;
> - u64 creation_flags;
> - struct hv_partition_creation_properties creation_properties = {};
> - union hv_partition_isolation_properties isolation_properties = {};
> - struct mshv_partition *partition;
> - struct file *file;
> - int fd;
> - long ret;
> + int i;
> + struct mshv_create_partition_v2 args;
> + union hv_partition_processor_features *disabled_procs;
> + union hv_partition_processor_xsave_features *disabled_xsave;
>
> - if (copy_from_user(&args, user_arg, sizeof(args)))
> + /* First, copy v1 struct in case user is on previous versions */
> + if (copy_from_user(&args, user_arg,
> + sizeof(struct mshv_create_partition)))
> return -EFAULT;
>
> if ((args.pt_flags & ~MSHV_PT_FLAGS_MASK) ||
> args.pt_isolation >= MSHV_PT_ISOLATION_COUNT)
> return -EINVAL;
>
> + disabled_procs = &cr_props->disabled_processor_features;
> + disabled_xsave = &cr_props->disabled_processor_xsave_features;
> +
> + /* Check if user provided newer struct with feature fields */
> + if (args.pt_flags & BIT_ULL(MSHV_PT_BIT_CPU_AND_XSAVE_FEATURES)) {
> + if (copy_from_user(&args, user_arg, sizeof(args)))
> + return -EFAULT;
> +
> + if (args.pt_num_cpu_fbanks != MSHV_NUM_CPU_FEATURES_BANKS ||
> + mshv_field_nonzero(args, pt_rsvd) ||
> + mshv_field_nonzero(args, pt_rsvd1))
> + return -EINVAL;
> +
> + /*
> + * Note this assumes MSHV_NUM_CPU_FEATURES_BANKS will never
> + * change and equals HV_PARTITION_PROCESSOR_FEATURES_BANKS
> + * (i.e. 2).
> + *
> + * Further banks (index >= 2) will be modifiable as 'early'
> + * properties via the set partition property hypercall.
> + */
> + for (i = 0; i < HV_PARTITION_PROCESSOR_FEATURES_BANKS; i++)
> + disabled_procs->as_uint64[i] = args.pt_cpu_fbanks[i];
> +
> +#if IS_ENABLED(CONFIG_X86_64)
> + disabled_xsave->as_uint64 = args.pt_disabled_xsave;
> +#else
> + /*
> + * In practice this field is ignored on arm64, but safer to
> + * zero it in case it is ever used.
> + */
> + disabled_xsave->as_uint64 = 0;
Why not explicitly treat non-zero value as invalid here instead?
Isn't it always better than implicitly (and silently) zeroing it?
> +
> + if (mshv_field_nonzero(args, pt_rsvd2))
> + return -EINVAL;
> +#endif
> + } else {
> + /*
> + * v1 behavior: try to enable everything. The hypervisor will
> + * disable features that are not supported. The banks can be
> + * queried via the get partition property hypercall.
> + */
> + for (i = 0; i < HV_PARTITION_PROCESSOR_FEATURES_BANKS; i++)
> + disabled_procs->as_uint64[i] = 0;
> +
> + disabled_xsave->as_uint64 = 0;
> + }
> +
> /* Only support EXO partitions */
> - creation_flags = HV_PARTITION_CREATION_FLAG_EXO_PARTITION |
> - HV_PARTITION_CREATION_FLAG_INTERCEPT_MESSAGE_PAGE_ENABLED;
> + *pt_flags = HV_PARTITION_CREATION_FLAG_EXO_PARTITION |
> + HV_PARTITION_CREATION_FLAG_INTERCEPT_MESSAGE_PAGE_ENABLED;
> +
> + if (args.pt_flags & BIT_ULL(MSHV_PT_BIT_LAPIC))
> + *pt_flags |= HV_PARTITION_CREATION_FLAG_LAPIC_ENABLED;
> + if (args.pt_flags & BIT_ULL(MSHV_PT_BIT_X2APIC))
> + *pt_flags |= HV_PARTITION_CREATION_FLAG_X2APIC_CAPABLE;
> + if (args.pt_flags & BIT_ULL(MSHV_PT_BIT_GPA_SUPER_PAGES))
> + *pt_flags |= HV_PARTITION_CREATION_FLAG_GPA_SUPER_PAGES_ENABLED;
>
> - if (args.pt_flags & BIT(MSHV_PT_BIT_LAPIC))
> - creation_flags |= HV_PARTITION_CREATION_FLAG_LAPIC_ENABLED;
> - if (args.pt_flags & BIT(MSHV_PT_BIT_X2APIC))
> - creation_flags |= HV_PARTITION_CREATION_FLAG_X2APIC_CAPABLE;
> - if (args.pt_flags & BIT(MSHV_PT_BIT_GPA_SUPER_PAGES))
> - creation_flags |= HV_PARTITION_CREATION_FLAG_GPA_SUPER_PAGES_ENABLED;
> + isol_props->as_uint64 = 0;
These properties are missing the nested one.
I'd recommend squeezing that change in this patch
Thanks,
Stanislav
>
> switch (args.pt_isolation) {
> case MSHV_PT_ISOLATION_NONE:
> - isolation_properties.isolation_type =
> - HV_PARTITION_ISOLATION_TYPE_NONE;
> + isol_props->isolation_type = HV_PARTITION_ISOLATION_TYPE_NONE;
> break;
> }
>
> + return 0;
> +}
> +
> +static long
> +mshv_ioctl_create_partition(void __user *user_arg, struct device *module_dev)
> +{
> + u64 creation_flags;
> + struct hv_partition_creation_properties creation_properties;
> + union hv_partition_isolation_properties isolation_properties;
> + struct mshv_partition *partition;
> + struct file *file;
> + int fd;
> + long ret;
> +
> + ret = mshv_ioctl_process_pt_flags(user_arg, &creation_flags,
> + &creation_properties,
> + &isolation_properties);
> + if (ret)
> + return ret;
> +
> partition = kzalloc(sizeof(*partition), GFP_KERNEL);
> if (!partition)
> return -ENOMEM;
> diff --git a/include/uapi/linux/mshv.h b/include/uapi/linux/mshv.h
> index 876bfe4e4227..cf904f3aa201 100644
> --- a/include/uapi/linux/mshv.h
> +++ b/include/uapi/linux/mshv.h
> @@ -26,6 +26,7 @@ enum {
> MSHV_PT_BIT_LAPIC,
> MSHV_PT_BIT_X2APIC,
> MSHV_PT_BIT_GPA_SUPER_PAGES,
> + MSHV_PT_BIT_CPU_AND_XSAVE_FEATURES,
> MSHV_PT_BIT_COUNT,
> };
>
> @@ -41,6 +42,8 @@ enum {
> * @pt_flags: Bitmask of 1 << MSHV_PT_BIT_*
> * @pt_isolation: MSHV_PT_ISOLATION_*
> *
> + * This is the initial/v1 version for backward compatibility.
> + *
> * Returns a file descriptor to act as a handle to a guest partition.
> * At this point the partition is not yet initialized in the hypervisor.
> * Some operations must be done with the partition in this state, e.g. setting
> @@ -52,6 +55,37 @@ struct mshv_create_partition {
> __u64 pt_isolation;
> };
>
> +#define MSHV_NUM_CPU_FEATURES_BANKS 2
> +
> +/**
> + * struct mshv_create_partition_v2
> + *
> + * This is extended version of the above initial MSHV_CREATE_PARTITION
> + * ioctl and allows for following additional parameters:
> + *
> + * @pt_num_cpu_fbanks: Must be set to MSHV_NUM_CPU_FEATURES_BANKS.
> + * @pt_cpu_fbanks: Disabled processor feature banks array.
> + * @pt_disabled_xsave: Disabled xsave feature bits.
> + *
> + * pt_cpu_fbanks and pt_disabled_xsave are passed through as-is to the create
> + * partition hypercall.
> + *
> + * Returns : same as above original mshv_create_partition
> + */
> +struct mshv_create_partition_v2 {
> + __u64 pt_flags;
> + __u64 pt_isolation;
> + __u16 pt_num_cpu_fbanks;
> + __u8 pt_rsvd[6]; /* MBZ */
> + __u64 pt_cpu_fbanks[MSHV_NUM_CPU_FEATURES_BANKS];
> + __u64 pt_rsvd1[2]; /* MBZ */
> +#if defined(__x86_64__)
> + __u64 pt_disabled_xsave;
> +#else
> + __u64 pt_rsvd2; /* MBZ */
> +#endif
> +} __packed;
> +
> /* /dev/mshv */
> #define MSHV_CREATE_PARTITION _IOW(MSHV_IOCTL, 0x00, struct mshv_create_partition)
>
> --
> 2.34.1
^ permalink raw reply
* [Patch net-next v2] net: mana: Handle hardware recovery events when probing the device
From: longli @ 2025-11-18 1:52 UTC (permalink / raw)
To: K. Y. Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Shradha Gupta, Simon Horman, Konstantin Taranov,
Souradeep Chakrabarti, Erick Archer, linux-hyperv, netdev,
linux-kernel, linux-rdma
Cc: Long Li
From: Long Li <longli@microsoft.com>
When MANA is being probed, it's possible that hardware is in recovery
mode and the device may get GDMA_EQE_HWC_RESET_REQUEST over HWC in the
middle of the probe. Detect such condition and go through the recovery
service procedure.
Fixes: fbe346ce9d62 ("net: mana: Handle Reset Request from MANA NIC")
Signed-off-by: Long Li <longli@microsoft.com>
---
Changes
v2: Use a list for handling multiple devices.
Use disable_delayed_work_sync() on driver exit.
Replace atomic_t with flags to detect if interrupt happens before probe finishes
.../net/ethernet/microsoft/mana/gdma_main.c | 172 ++++++++++++++++--
include/net/mana/gdma.h | 12 +-
2 files changed, 166 insertions(+), 18 deletions(-)
diff --git a/drivers/net/ethernet/microsoft/mana/gdma_main.c b/drivers/net/ethernet/microsoft/mana/gdma_main.c
index effe0a2f207a..57d58adf623a 100644
--- a/drivers/net/ethernet/microsoft/mana/gdma_main.c
+++ b/drivers/net/ethernet/microsoft/mana/gdma_main.c
@@ -15,6 +15,20 @@
struct dentry *mana_debugfs_root;
+struct mana_dev_recovery {
+ struct list_head list;
+ struct pci_dev *pdev;
+ enum gdma_eqe_type type;
+};
+
+static struct mana_dev_recovery_work {
+ struct list_head dev_list;
+ struct delayed_work work;
+
+ /* Lock for dev_list above */
+ spinlock_t lock;
+} mana_dev_recovery_work;
+
static u32 mana_gd_r32(struct gdma_context *g, u64 offset)
{
return readl(g->bar0_va + offset);
@@ -387,6 +401,25 @@ EXPORT_SYMBOL_NS(mana_gd_ring_cq, "NET_MANA");
#define MANA_SERVICE_PERIOD 10
+static void mana_serv_rescan(struct pci_dev *pdev)
+{
+ struct pci_bus *parent;
+
+ pci_lock_rescan_remove();
+
+ parent = pdev->bus;
+ if (!parent) {
+ dev_err(&pdev->dev, "MANA service: no parent bus\n");
+ goto out;
+ }
+
+ pci_stop_and_remove_bus_device(pdev);
+ pci_rescan_bus(parent);
+
+out:
+ pci_unlock_rescan_remove();
+}
+
static void mana_serv_fpga(struct pci_dev *pdev)
{
struct pci_bus *bus, *parent;
@@ -419,9 +452,12 @@ static void mana_serv_reset(struct pci_dev *pdev)
{
struct gdma_context *gc = pci_get_drvdata(pdev);
struct hw_channel_context *hwc;
+ int ret;
if (!gc) {
- dev_err(&pdev->dev, "MANA service: no GC\n");
+ /* Perform PCI rescan on device if GC is not set up */
+ dev_err(&pdev->dev, "MANA service: GC not setup, rescanning\n");
+ mana_serv_rescan(pdev);
return;
}
@@ -440,9 +476,18 @@ static void mana_serv_reset(struct pci_dev *pdev)
msleep(MANA_SERVICE_PERIOD * 1000);
- mana_gd_resume(pdev);
+ ret = mana_gd_resume(pdev);
+ if (ret == -ETIMEDOUT || ret == -EPROTO) {
+ /* Perform PCI rescan on device if we failed on HWC */
+ dev_err(&pdev->dev, "MANA service: resume failed, rescanning\n");
+ mana_serv_rescan(pdev);
+ goto out;
+ }
- dev_info(&pdev->dev, "MANA reset cycle completed\n");
+ if (ret)
+ dev_info(&pdev->dev, "MANA reset cycle failed err %d\n", ret);
+ else
+ dev_info(&pdev->dev, "MANA reset cycle completed\n");
out:
gc->in_service = false;
@@ -454,18 +499,9 @@ struct mana_serv_work {
enum gdma_eqe_type type;
};
-static void mana_serv_func(struct work_struct *w)
+static void mana_do_service(enum gdma_eqe_type type, struct pci_dev *pdev)
{
- struct mana_serv_work *mns_wk;
- struct pci_dev *pdev;
-
- mns_wk = container_of(w, struct mana_serv_work, serv_work);
- pdev = mns_wk->pdev;
-
- if (!pdev)
- goto out;
-
- switch (mns_wk->type) {
+ switch (type) {
case GDMA_EQE_HWC_FPGA_RECONFIG:
mana_serv_fpga(pdev);
break;
@@ -475,12 +511,46 @@ static void mana_serv_func(struct work_struct *w)
break;
default:
- dev_err(&pdev->dev, "MANA service: unknown type %d\n",
- mns_wk->type);
+ dev_err(&pdev->dev, "MANA service: unknown type %d\n", type);
break;
}
+}
+
+static void mana_recovery_delayed_func(struct work_struct *w)
+{
+ struct mana_dev_recovery_work *work;
+ struct mana_dev_recovery *dev, *tmp;
+ unsigned long flags;
+
+ work = container_of(w, struct mana_dev_recovery_work, work.work);
+
+ spin_lock_irqsave(&work->lock, flags);
+
+ list_for_each_entry_safe(dev, tmp, &work->dev_list, list) {
+ list_del(&dev->list);
+ spin_unlock_irqrestore(&work->lock, flags);
+
+ mana_do_service(dev->type, dev->pdev);
+ pci_dev_put(dev->pdev);
+ kfree(dev);
+
+ spin_lock_irqsave(&work->lock, flags);
+ }
+
+ spin_unlock_irqrestore(&work->lock, flags);
+}
+
+static void mana_serv_func(struct work_struct *w)
+{
+ struct mana_serv_work *mns_wk;
+ struct pci_dev *pdev;
+
+ mns_wk = container_of(w, struct mana_serv_work, serv_work);
+ pdev = mns_wk->pdev;
+
+ if (pdev)
+ mana_do_service(mns_wk->type, pdev);
-out:
pci_dev_put(pdev);
kfree(mns_wk);
module_put(THIS_MODULE);
@@ -541,6 +611,17 @@ static void mana_gd_process_eqe(struct gdma_queue *eq)
case GDMA_EQE_HWC_RESET_REQUEST:
dev_info(gc->dev, "Recv MANA service type:%d\n", type);
+ if (!test_and_set_bit(GC_PROBE_SUCCEEDED, &gc->flags)) {
+ /*
+ * Device is in probe and we received a hardware reset
+ * event, the probe function will detect that the flag
+ * has changed and perform service procedure.
+ */
+ dev_info(gc->dev,
+ "Service is to be processed in probe\n");
+ break;
+ }
+
if (gc->in_service) {
dev_info(gc->dev, "Already in service\n");
break;
@@ -1942,8 +2023,19 @@ static int mana_gd_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
if (err)
goto cleanup_mana;
+ /*
+ * If a hardware reset event has occurred over HWC during probe,
+ * rollback and perform hardware reset procedure.
+ */
+ if (test_and_set_bit(GC_PROBE_SUCCEEDED, &gc->flags)) {
+ err = -EPROTO;
+ goto cleanup_mana_rdma;
+ }
+
return 0;
+cleanup_mana_rdma:
+ mana_rdma_remove(&gc->mana_ib);
cleanup_mana:
mana_remove(&gc->mana, false);
cleanup_gd:
@@ -1967,6 +2059,35 @@ static int mana_gd_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
disable_dev:
pci_disable_device(pdev);
dev_err(&pdev->dev, "gdma probe failed: err = %d\n", err);
+
+ /*
+ * Hardware could be in recovery mode and the HWC returns TIMEDOUT or
+ * EPROTO from mana_gd_setup(), mana_probe() or mana_rdma_probe(), or
+ * we received a hardware reset event over HWC interrupt. In this case,
+ * perform the device recovery procedure after MANA_SERVICE_PERIOD
+ * seconds.
+ */
+ if (err == -ETIMEDOUT || err == -EPROTO) {
+ struct mana_dev_recovery *dev;
+ unsigned long flags;
+
+ dev_info(&pdev->dev, "Start MANA recovery mode\n");
+
+ dev = kzalloc(sizeof(*dev), GFP_KERNEL);
+ if (!dev)
+ return err;
+
+ dev->pdev = pci_dev_get(pdev);
+ dev->type = GDMA_EQE_HWC_RESET_REQUEST;
+
+ spin_lock_irqsave(&mana_dev_recovery_work.lock, flags);
+ list_add_tail(&dev->list, &mana_dev_recovery_work.dev_list);
+ spin_unlock_irqrestore(&mana_dev_recovery_work.lock, flags);
+
+ schedule_delayed_work(&mana_dev_recovery_work.work,
+ secs_to_jiffies(MANA_SERVICE_PERIOD));
+ }
+
return err;
}
@@ -2071,6 +2192,10 @@ static int __init mana_driver_init(void)
{
int err;
+ INIT_LIST_HEAD(&mana_dev_recovery_work.dev_list);
+ spin_lock_init(&mana_dev_recovery_work.lock);
+ INIT_DELAYED_WORK(&mana_dev_recovery_work.work, mana_recovery_delayed_func);
+
mana_debugfs_root = debugfs_create_dir("mana", NULL);
err = pci_register_driver(&mana_driver);
@@ -2084,6 +2209,19 @@ static int __init mana_driver_init(void)
static void __exit mana_driver_exit(void)
{
+ struct mana_dev_recovery *dev, *tmp;
+ unsigned long flags;
+
+ disable_delayed_work_sync(&mana_dev_recovery_work.work);
+
+ spin_lock_irqsave(&mana_dev_recovery_work.lock, flags);
+ list_for_each_entry_safe(dev, tmp, &mana_dev_recovery_work.dev_list, list) {
+ list_del(&dev->list);
+ pci_dev_put(dev->pdev);
+ kfree(dev);
+ }
+ spin_unlock_irqrestore(&mana_dev_recovery_work.lock, flags);
+
pci_unregister_driver(&mana_driver);
debugfs_remove(mana_debugfs_root);
diff --git a/include/net/mana/gdma.h b/include/net/mana/gdma.h
index 637f42485dba..bf3b32540786 100644
--- a/include/net/mana/gdma.h
+++ b/include/net/mana/gdma.h
@@ -382,6 +382,10 @@ struct gdma_irq_context {
char name[MANA_IRQ_NAME_SZ];
};
+enum gdma_context_flags {
+ GC_PROBE_SUCCEEDED = 0,
+};
+
struct gdma_context {
struct device *dev;
struct dentry *mana_pci_debugfs;
@@ -430,6 +434,8 @@ struct gdma_context {
u64 pf_cap_flags1;
struct workqueue_struct *service_wq;
+
+ unsigned long flags;
};
static inline bool mana_gd_is_mana(struct gdma_dev *gd)
@@ -592,6 +598,9 @@ enum {
#define GDMA_DRV_CAP_FLAG_1_HANDLE_RECONFIG_EQE BIT(17)
#define GDMA_DRV_CAP_FLAG_1_HW_VPORT_LINK_AWARE BIT(6)
+/* Driver can handle hardware recovery events during probe */
+#define GDMA_DRV_CAP_FLAG_1_PROBE_RECOVERY BIT(22)
+
#define GDMA_DRV_CAP_FLAGS1 \
(GDMA_DRV_CAP_FLAG_1_EQ_SHARING_MULTI_VPORT | \
GDMA_DRV_CAP_FLAG_1_NAPI_WKDONE_FIX | \
@@ -601,7 +610,8 @@ enum {
GDMA_DRV_CAP_FLAG_1_DYNAMIC_IRQ_ALLOC_SUPPORT | \
GDMA_DRV_CAP_FLAG_1_SELF_RESET_ON_EQE | \
GDMA_DRV_CAP_FLAG_1_HANDLE_RECONFIG_EQE | \
- GDMA_DRV_CAP_FLAG_1_HW_VPORT_LINK_AWARE)
+ GDMA_DRV_CAP_FLAG_1_HW_VPORT_LINK_AWARE | \
+ GDMA_DRV_CAP_FLAG_1_PROBE_RECOVERY)
#define GDMA_DRV_CAP_FLAGS2 0
--
2.43.0
^ permalink raw reply related
* [PATCH net-next v10 00/11] vsock: add namespace support to vhost-vsock and loopback
From: Bobby Eshleman @ 2025-11-18 2:00 UTC (permalink / raw)
To: Stefano Garzarella, David S. Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Simon Horman, Stefan Hajnoczi, Michael S. Tsirkin,
Jason Wang, Eugenio Pérez, Xuan Zhuo, K. Y. Srinivasan,
Haiyang Zhang, Wei Liu, Dexuan Cui, Bryan Tan, Vishnu Dasa,
Broadcom internal kernel review list, Shuah Khan
Cc: linux-kernel, virtualization, netdev, kvm, linux-hyperv,
linux-kselftest, Sargun Dhillon, Bobby Eshleman, berrange,
Bobby Eshleman
This series adds namespace support to vhost-vsock and loopback. It does
not add namespaces to any of the other guest transports (virtio-vsock,
hyperv, or vmci).
The current revision supports two modes: local and global. Local
mode is complete isolation of namespaces, while global mode is complete
sharing between namespaces of CIDs (the original behavior).
The mode is set using /proc/sys/net/vsock/ns_mode.
Modes are per-netns and write-once. This allows a system to configure
namespaces independently (some may share CIDs, others are completely
isolated). This also supports future possible mixed use cases, where
there may be namespaces in global mode spinning up VMs while there are
mixed mode namespaces that provide services to the VMs, but are not
allowed to allocate from the global CID pool (this mode is not
implemented in this series).
If a socket or VM is created when a namespace is global but the
namespace changes to local, the socket or VM will continue working
normally. That is, the socket or VM assumes the mode behavior of the
namespace at the time the socket/VM was created. The original mode is
captured in vsock_create() and so occurs at the time of socket(2) and
accept(2) for sockets and open(2) on /dev/vhost-vsock for VMs. This
prevents a socket/VM connection from suddenly breaking due to a
namespace mode change. Any new sockets/VMs created after the mode change
will adopt the new mode's behavior.
Additionally, added tests for the new namespace features:
tools/testing/selftests/vsock/vmtest.sh
1..29
ok 1 vm_server_host_client
ok 2 vm_client_host_server
ok 3 vm_loopback
ok 4 ns_guest_local_mode_rejected
ok 5 ns_host_vsock_ns_mode_ok
ok 6 ns_host_vsock_ns_mode_write_once_ok
ok 7 ns_global_same_cid_fails
ok 8 ns_local_same_cid_ok
ok 9 ns_global_local_same_cid_ok
ok 10 ns_local_global_same_cid_ok
ok 11 ns_diff_global_host_connect_to_global_vm_ok
ok 12 ns_diff_global_host_connect_to_local_vm_fails
ok 13 ns_diff_global_vm_connect_to_global_host_ok
ok 14 ns_diff_global_vm_connect_to_local_host_fails
ok 15 ns_diff_local_host_connect_to_local_vm_fails
ok 16 ns_diff_local_vm_connect_to_local_host_fails
ok 17 ns_diff_global_to_local_loopback_local_fails
ok 18 ns_diff_local_to_global_loopback_fails
ok 19 ns_diff_local_to_local_loopback_fails
ok 20 ns_diff_global_to_global_loopback_ok
ok 21 ns_same_local_loopback_ok
ok 22 ns_same_local_host_connect_to_local_vm_ok
ok 23 ns_same_local_vm_connect_to_local_host_ok
ok 24 ns_mode_change_connection_continue_vm_ok
ok 25 ns_mode_change_connection_continue_host_ok
ok 26 ns_mode_change_connection_continue_both_ok
ok 27 ns_delete_vm_ok
ok 28 ns_delete_host_ok
ok 29 ns_delete_both_ok
SUMMARY: PASS=29 SKIP=0 FAIL=0
Dependent on series:
https://lore.kernel.org/all/20251108-vsock-selftests-fixes-and-improvements-v4-0-d5e8d6c87289@meta.com/
Thanks again for everyone's help and reviews!
Suggested-by: Sargun Dhillon <sargun@sargun.me>
Signed-off-by: Bobby Eshleman <bobbyeshleman@gmail.com>
To: Stefano Garzarella <sgarzare@redhat.com>
To: Shuah Khan <shuah@kernel.org>
To: David S. Miller <davem@davemloft.net>
To: Eric Dumazet <edumazet@google.com>
To: Jakub Kicinski <kuba@kernel.org>
To: Paolo Abeni <pabeni@redhat.com>
To: Simon Horman <horms@kernel.org>
To: Stefan Hajnoczi <stefanha@redhat.com>
To: Michael S. Tsirkin <mst@redhat.com>
To: Jason Wang <jasowang@redhat.com>
To: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
To: Eugenio Pérez <eperezma@redhat.com>
To: K. Y. Srinivasan <kys@microsoft.com>
To: Haiyang Zhang <haiyangz@microsoft.com>
To: Wei Liu <wei.liu@kernel.org>
To: Dexuan Cui <decui@microsoft.com>
To: Bryan Tan <bryan-bt.tan@broadcom.com>
To: Vishnu Dasa <vishnu.dasa@broadcom.com>
To: Broadcom internal kernel review list <bcm-kernel-feedback-list@broadcom.com>
Cc: virtualization@lists.linux.dev
Cc: netdev@vger.kernel.org
Cc: linux-kselftest@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Cc: kvm@vger.kernel.org
Cc: linux-hyperv@vger.kernel.org
Cc: berrange@redhat.com
Cc: Sargun Dhillon <sargun@sargun.me>
Changes in v10:
- Combine virtio common patches into one (Stefano)
- Resolve vsock_loopback virtio_transport_reset_no_sock() issue
with info->vsk setting. This eliminates the need for skb->cb,
so remove skb->cb patches.
- many line width 80 fixes
- Link to v9: https://lore.kernel.org/all/20251111-vsock-vmtest-v9-0-852787a37bed@meta.com
Changes in v9:
- reorder loopback patch after patch for virtio transport common code
- remove module ordering tests patch because loopback no longer depends
on pernet ops
- major simplifications in vsock_loopback
- added a new patch for blocking local mode for guests, added test case
to check
- add net ref tracking to vsock_loopback patch
- Link to v8: https://lore.kernel.org/r/20251023-vsock-vmtest-v8-0-dea984d02bb0@meta.com
Changes in v8:
- Break generic cleanup/refactoring patches into standalone series,
remove those from this series
- Link to dependency: https://lore.kernel.org/all/20251022-vsock-selftests-fixes-and-improvements-v1-0-edeb179d6463@meta.com/
- Link to v7: https://lore.kernel.org/r/20251021-vsock-vmtest-v7-0-0661b7b6f081@meta.com
Changes in v7:
- fix hv_sock build
- break out vmtest patches into distinct, more well-scoped patches
- change `orig_net_mode` to `net_mode`
- many fixes and style changes in per-patch change sets (see individual
patches for specific changes)
- optimize `virtio_vsock_skb_cb` layout
- update commit messages with more useful descriptions
- vsock_loopback: use orig_net_mode instead of current net mode
- add tests for edge cases (ns deletion, mode changing, loopback module
load ordering)
- Link to v6: https://lore.kernel.org/r/20250916-vsock-vmtest-v6-0-064d2eb0c89d@meta.com
Changes in v6:
- define behavior when mode changes to local while socket/VM is alive
- af_vsock: clarify description of CID behavior
- af_vsock: use stronger langauge around CID rules (dont use "may")
- af_vsock: improve naming of buf/buffer
- af_vsock: improve string length checking on proc writes
- vsock_loopback: add space in struct to clarify lock protection
- vsock_loopback: do proper cleanup/unregister on vsock_loopback_exit()
- vsock_loopback: use virtio_vsock_skb_net() instead of sock_net()
- vsock_loopback: set loopback to NULL after kfree()
- vsock_loopback: use pernet_operations and remove callback mechanism
- vsock_loopback: add macros for "global" and "local"
- vsock_loopback: fix length checking
- vmtest.sh: check for namespace support in vmtest.sh
- Link to v5: https://lore.kernel.org/r/20250827-vsock-vmtest-v5-0-0ba580bede5b@meta.com
Changes in v5:
- /proc/net/vsock_ns_mode -> /proc/sys/net/vsock/ns_mode
- vsock_global_net -> vsock_global_dummy_net
- fix netns lookup in vhost_vsock to respect pid namespaces
- add callbacks for vsock_loopback to avoid circular dependency
- vmtest.sh loads vsock_loopback module
- remove vsock_net_mode_can_set()
- change vsock_net_write_mode() to return true/false based on success
- make vsock_net_mode enum instead of u8
- Link to v4: https://lore.kernel.org/r/20250805-vsock-vmtest-v4-0-059ec51ab111@meta.com
Changes in v4:
- removed RFC tag
- implemented loopback support
- renamed new tests to better reflect behavior
- completed suite of tests with permutations of ns modes and vsock_test
as guest/host
- simplified socat bridging with unix socket instead of tcp + veth
- only use vsock_test for success case, socat for failure case (context
in commit message)
- lots of cleanup
Changes in v3:
- add notion of "modes"
- add procfs /proc/net/vsock_ns_mode
- local and global modes only
- no /dev/vhost-vsock-netns
- vmtest.sh already merged, so new patch just adds new tests for NS
- Link to v2:
https://lore.kernel.org/kvm/20250312-vsock-netns-v2-0-84bffa1aa97a@gmail.com
Changes in v2:
- only support vhost-vsock namespaces
- all g2h namespaces retain old behavior, only common API changes
impacted by vhost-vsock changes
- add /dev/vhost-vsock-netns for "opt-in"
- leave /dev/vhost-vsock to old behavior
- removed netns module param
- Link to v1:
https://lore.kernel.org/r/20200116172428.311437-1-sgarzare@redhat.com
Changes in v1:
- added 'netns' module param to vsock.ko to enable the
network namespace support (disabled by default)
- added 'vsock_net_eq()' to check the "net" assigned to a socket
only when 'netns' support is enabled
- Link to RFC: https://patchwork.ozlabs.org/cover/1202235/
---
Bobby Eshleman (11):
vsock: a per-net vsock NS mode state
vsock: add netns to vsock core
vsock: reject bad VSOCK_NET_MODE_LOCAL configuration for G2H
vsock: add netns support to virtio transports
virtio: set skb owner of virtio_transport_reset_no_sock() reply
selftests/vsock: add namespace helpers to vmtest.sh
selftests/vsock: prepare vm management helpers for namespaces
selftests/vsock: add tests for proc sys vsock ns_mode
selftests/vsock: add namespace tests for CID collisions
selftests/vsock: add tests for host <-> vm connectivity with namespaces
selftests/vsock: add tests for namespace deletion and mode changes
MAINTAINERS | 1 +
drivers/vhost/vsock.c | 57 +-
include/linux/virtio_vsock.h | 8 +-
include/net/af_vsock.h | 58 +-
include/net/net_namespace.h | 4 +
include/net/netns/vsock.h | 17 +
net/vmw_vsock/af_vsock.c | 294 ++++++++-
net/vmw_vsock/hyperv_transport.c | 6 +
net/vmw_vsock/virtio_transport.c | 29 +-
net/vmw_vsock/virtio_transport_common.c | 69 +-
net/vmw_vsock/vmci_transport.c | 7 +
net/vmw_vsock/vsock_loopback.c | 20 +-
tools/testing/selftests/vsock/vmtest.sh | 1037 +++++++++++++++++++++++++++++--
13 files changed, 1514 insertions(+), 93 deletions(-)
---
base-commit: 962ac5ca99a5c3e7469215bf47572440402dfd59
change-id: 20250325-vsock-vmtest-b3a21d2102c2
prerequisite-message-id: <20251022-vsock-selftests-fixes-and-improvements-v1-0-edeb179d6463@meta.com>
prerequisite-patch-id: a2eecc3851f2509ed40009a7cab6990c6d7cfff5
prerequisite-patch-id: 501db2100636b9c8fcb3b64b8b1df797ccbede85
prerequisite-patch-id: ba1a2f07398a035bc48ef72edda41888614be449
prerequisite-patch-id: fd5cc5445aca9355ce678e6d2bfa89fab8a57e61
prerequisite-patch-id: 795ab4432ffb0843e22b580374782e7e0d99b909
prerequisite-patch-id: 1499d263dc933e75366c09e045d2125ca39f7ddd
prerequisite-patch-id: f92d99bb1d35d99b063f818a19dcda999152d74c
prerequisite-patch-id: e3296f38cdba6d903e061cff2bbb3e7615e8e671
prerequisite-patch-id: bc4662b4710d302d4893f58708820fc2a0624325
prerequisite-patch-id: f8991f2e98c2661a706183fde6b35e2b8d9aedcf
prerequisite-patch-id: 44bf9ed69353586d284e5ee63d6fffa30439a698
prerequisite-patch-id: d50621bc630eeaf608bbaf260370c8dabf6326df
Best regards,
--
Bobby Eshleman <bobbyeshleman@meta.com>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox