[PATCH v3] x86/hyperv: Use per cpu initial stack for vtl context

[PATCH v3] x86/hyperv: Use per cpu initial stack for vtl context

[Saurabh Sengar]

Currently, the secondary CPUs in Hyper-V VTL context lack support for
parallel startup. Therefore, relying on the single initial_stack fetched
from the current task structure suffices for all vCPUs.

However, common initial_stack risks stack corruption when parallel startup
is enabled. In order to facilitate parallel startup, use the initial_stack
from the per CPU idle thread instead of the current task.

Fixes: 18415f33e2ac ("cpu/hotplug: Allow "parallel" bringup up to CPUHP_BP_KICK_AP_STATE")
Signed-off-by: Saurabh Sengar <ssengar@linux.microsoft.com>
Reviewed-by: Michael Kelley <mhklinux@outlook.com>
---
[V3]
 - Added the VTL code dependency on SMP to fix kernel build error
   when SMP is disabled.

 arch/x86/hyperv/hv_vtl.c | 19 +++++++++++++++----
 drivers/hv/Kconfig       |  1 +
 2 files changed, 16 insertions(+), 4 deletions(-)

diff --git a/arch/x86/hyperv/hv_vtl.c b/arch/x86/hyperv/hv_vtl.c
index 804b629ea49d..b4e233954d0f 100644
--- a/arch/x86/hyperv/hv_vtl.c
+++ b/arch/x86/hyperv/hv_vtl.c
@@ -12,6 +12,7 @@
 #include <asm/i8259.h>
 #include <asm/mshyperv.h>
 #include <asm/realmode.h>
+#include <../kernel/smpboot.h>
 
 extern struct boot_params boot_params;
 static struct real_mode_header hv_vtl_real_mode_header;
@@ -58,7 +59,7 @@ static void hv_vtl_ap_entry(void)
 	((secondary_startup_64_fn)secondary_startup_64)(&boot_params, &boot_params);
 }
 
-static int hv_vtl_bringup_vcpu(u32 target_vp_index, u64 eip_ignored)
+static int hv_vtl_bringup_vcpu(u32 target_vp_index, int cpu, u64 eip_ignored)
 {
 	u64 status;
 	int ret = 0;
@@ -72,7 +73,9 @@ static int hv_vtl_bringup_vcpu(u32 target_vp_index, u64 eip_ignored)
 	struct ldttss_desc *ldt;
 	struct desc_struct *gdt;
 
-	u64 rsp = current->thread.sp;
+	struct task_struct *idle = idle_thread_get(cpu);
+	u64 rsp = (unsigned long)idle->thread.sp;
+
 	u64 rip = (u64)&hv_vtl_ap_entry;
 
 	native_store_gdt(&gdt_ptr);
@@ -199,7 +202,15 @@ static int hv_vtl_apicid_to_vp_id(u32 apic_id)
 
 static int hv_vtl_wakeup_secondary_cpu(u32 apicid, unsigned long start_eip)
 {
-	int vp_id;
+	int vp_id, cpu;
+
+	/* Find the logical CPU for the APIC ID */
+	for_each_present_cpu(cpu) {
+		if (arch_match_cpu_phys_id(cpu, apicid))
+			break;
+	}
+	if (cpu >= nr_cpu_ids)
+		return -EINVAL;
 
 	pr_debug("Bringing up CPU with APIC ID %d in VTL2...\n", apicid);
 	vp_id = hv_vtl_apicid_to_vp_id(apicid);
@@ -213,7 +224,7 @@ static int hv_vtl_wakeup_secondary_cpu(u32 apicid, unsigned long start_eip)
 		return -EINVAL;
 	}
 
-	return hv_vtl_bringup_vcpu(vp_id, start_eip);
+	return hv_vtl_bringup_vcpu(vp_id, cpu, start_eip);
 }
 
 int __init hv_vtl_early_init(void)
diff --git a/drivers/hv/Kconfig b/drivers/hv/Kconfig
index 00242107d62e..862c47b191af 100644
--- a/drivers/hv/Kconfig
+++ b/drivers/hv/Kconfig
@@ -16,6 +16,7 @@ config HYPERV
 config HYPERV_VTL_MODE
 	bool "Enable Linux to boot in VTL context"
 	depends on X86_64 && HYPERV
+	depends on SMP
 	default n
 	help
 	  Virtual Secure Mode (VSM) is a set of hypervisor capabilities and
-- 
2.34.1

Re: [PATCH v3] x86/hyperv: Use per cpu initial stack for vtl context

[Wei Liu]

On Sun, Mar 03, 2024 at 12:01:36AM -0800, Saurabh Sengar wrote:
> Currently, the secondary CPUs in Hyper-V VTL context lack support for
> parallel startup. Therefore, relying on the single initial_stack fetched
> from the current task structure suffices for all vCPUs.
> 
> However, common initial_stack risks stack corruption when parallel startup
> is enabled. In order to facilitate parallel startup, use the initial_stack
> from the per CPU idle thread instead of the current task.
> 
> Fixes: 18415f33e2ac ("cpu/hotplug: Allow "parallel" bringup up to CPUHP_BP_KICK_AP_STATE")

I don't think this patch is buggy. Instead, it exposes an assumption in
the VTL code. So this either should be dropped or point to the patch
which introduces the assumption.

Let me know what you would prefer.

Thanks,
Wei.

> Signed-off-by: Saurabh Sengar <ssengar@linux.microsoft.com>
> Reviewed-by: Michael Kelley <mhklinux@outlook.com>
> ---
> [V3]
>  - Added the VTL code dependency on SMP to fix kernel build error
>    when SMP is disabled.
> 
>  arch/x86/hyperv/hv_vtl.c | 19 +++++++++++++++----
>  drivers/hv/Kconfig       |  1 +
>  2 files changed, 16 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/x86/hyperv/hv_vtl.c b/arch/x86/hyperv/hv_vtl.c
> index 804b629ea49d..b4e233954d0f 100644
> --- a/arch/x86/hyperv/hv_vtl.c
> +++ b/arch/x86/hyperv/hv_vtl.c
> @@ -12,6 +12,7 @@
>  #include <asm/i8259.h>
>  #include <asm/mshyperv.h>
>  #include <asm/realmode.h>
> +#include <../kernel/smpboot.h>
>  
>  extern struct boot_params boot_params;
>  static struct real_mode_header hv_vtl_real_mode_header;
> @@ -58,7 +59,7 @@ static void hv_vtl_ap_entry(void)
>  	((secondary_startup_64_fn)secondary_startup_64)(&boot_params, &boot_params);
>  }
>  
> -static int hv_vtl_bringup_vcpu(u32 target_vp_index, u64 eip_ignored)
> +static int hv_vtl_bringup_vcpu(u32 target_vp_index, int cpu, u64 eip_ignored)
>  {
>  	u64 status;
>  	int ret = 0;
> @@ -72,7 +73,9 @@ static int hv_vtl_bringup_vcpu(u32 target_vp_index, u64 eip_ignored)
>  	struct ldttss_desc *ldt;
>  	struct desc_struct *gdt;
>  
> -	u64 rsp = current->thread.sp;
> +	struct task_struct *idle = idle_thread_get(cpu);
> +	u64 rsp = (unsigned long)idle->thread.sp;
> +
>  	u64 rip = (u64)&hv_vtl_ap_entry;
>  
>  	native_store_gdt(&gdt_ptr);
> @@ -199,7 +202,15 @@ static int hv_vtl_apicid_to_vp_id(u32 apic_id)
>  
>  static int hv_vtl_wakeup_secondary_cpu(u32 apicid, unsigned long start_eip)
>  {
> -	int vp_id;
> +	int vp_id, cpu;
> +
> +	/* Find the logical CPU for the APIC ID */
> +	for_each_present_cpu(cpu) {
> +		if (arch_match_cpu_phys_id(cpu, apicid))
> +			break;
> +	}
> +	if (cpu >= nr_cpu_ids)
> +		return -EINVAL;
>  
>  	pr_debug("Bringing up CPU with APIC ID %d in VTL2...\n", apicid);
>  	vp_id = hv_vtl_apicid_to_vp_id(apicid);
> @@ -213,7 +224,7 @@ static int hv_vtl_wakeup_secondary_cpu(u32 apicid, unsigned long start_eip)
>  		return -EINVAL;
>  	}
>  
> -	return hv_vtl_bringup_vcpu(vp_id, start_eip);
> +	return hv_vtl_bringup_vcpu(vp_id, cpu, start_eip);
>  }
>  
>  int __init hv_vtl_early_init(void)
> diff --git a/drivers/hv/Kconfig b/drivers/hv/Kconfig
> index 00242107d62e..862c47b191af 100644
> --- a/drivers/hv/Kconfig
> +++ b/drivers/hv/Kconfig
> @@ -16,6 +16,7 @@ config HYPERV
>  config HYPERV_VTL_MODE
>  	bool "Enable Linux to boot in VTL context"
>  	depends on X86_64 && HYPERV
> +	depends on SMP
>  	default n
>  	help
>  	  Virtual Secure Mode (VSM) is a set of hypervisor capabilities and
> -- 
> 2.34.1
> 

Re: [PATCH v3] x86/hyperv: Use per cpu initial stack for vtl context

[Saurabh Singh Sengar]

On Mon, Mar 04, 2024 at 06:24:27AM +0000, Wei Liu wrote:
> On Sun, Mar 03, 2024 at 12:01:36AM -0800, Saurabh Sengar wrote:
> > Currently, the secondary CPUs in Hyper-V VTL context lack support for
> > parallel startup. Therefore, relying on the single initial_stack fetched
> > from the current task structure suffices for all vCPUs.
> > 
> > However, common initial_stack risks stack corruption when parallel startup
> > is enabled. In order to facilitate parallel startup, use the initial_stack
> > from the per CPU idle thread instead of the current task.
> > 
> > Fixes: 18415f33e2ac ("cpu/hotplug: Allow "parallel" bringup up to CPUHP_BP_KICK_AP_STATE")
> 
> I don't think this patch is buggy. Instead, it exposes an assumption in
> the VTL code. So this either should be dropped or point to the patch
> which introduces the assumption.
> 
> Let me know what you would prefer.

The VTL code will crash if this fix is not present post above mentioned patch:
18415f33e2ac ("cpu/hotplug: Allow "parallel" bringup up to CPUHP_BP_KICK_AP_STATE").
So I would prefer a fixes which added the assumption in VTL:

Fixes: 3be1bc2fe9d2 ("x86/hyperv: VTL support for Hyper-V")

Please let me know if you need V4 for it.

> 
> Thanks,
> Wei.
> 
> > Signed-off-by: Saurabh Sengar <ssengar@linux.microsoft.com>
> > Reviewed-by: Michael Kelley <mhklinux@outlook.com>
> > ---
> > [V3]
> >  - Added the VTL code dependency on SMP to fix kernel build error
> >    when SMP is disabled.
> > 
> >  arch/x86/hyperv/hv_vtl.c | 19 +++++++++++++++----
> >  drivers/hv/Kconfig       |  1 +
> >  2 files changed, 16 insertions(+), 4 deletions(-)
> > 
> > diff --git a/arch/x86/hyperv/hv_vtl.c b/arch/x86/hyperv/hv_vtl.c
> > index 804b629ea49d..b4e233954d0f 100644
> > --- a/arch/x86/hyperv/hv_vtl.c
> > +++ b/arch/x86/hyperv/hv_vtl.c
> > @@ -12,6 +12,7 @@
> >  #include <asm/i8259.h>
> >  #include <asm/mshyperv.h>
> >  #include <asm/realmode.h>
> > +#include <../kernel/smpboot.h>
> >  
> >  extern struct boot_params boot_params;
> >  static struct real_mode_header hv_vtl_real_mode_header;
> > @@ -58,7 +59,7 @@ static void hv_vtl_ap_entry(void)
> >  	((secondary_startup_64_fn)secondary_startup_64)(&boot_params, &boot_params);
> >  }
> >  
> > -static int hv_vtl_bringup_vcpu(u32 target_vp_index, u64 eip_ignored)
> > +static int hv_vtl_bringup_vcpu(u32 target_vp_index, int cpu, u64 eip_ignored)
> >  {
> >  	u64 status;
> >  	int ret = 0;
> > @@ -72,7 +73,9 @@ static int hv_vtl_bringup_vcpu(u32 target_vp_index, u64 eip_ignored)
> >  	struct ldttss_desc *ldt;
> >  	struct desc_struct *gdt;
> >  
> > -	u64 rsp = current->thread.sp;
> > +	struct task_struct *idle = idle_thread_get(cpu);
> > +	u64 rsp = (unsigned long)idle->thread.sp;
> > +
> >  	u64 rip = (u64)&hv_vtl_ap_entry;
> >  
> >  	native_store_gdt(&gdt_ptr);
> > @@ -199,7 +202,15 @@ static int hv_vtl_apicid_to_vp_id(u32 apic_id)
> >  
> >  static int hv_vtl_wakeup_secondary_cpu(u32 apicid, unsigned long start_eip)
> >  {
> > -	int vp_id;
> > +	int vp_id, cpu;
> > +
> > +	/* Find the logical CPU for the APIC ID */
> > +	for_each_present_cpu(cpu) {
> > +		if (arch_match_cpu_phys_id(cpu, apicid))
> > +			break;
> > +	}
> > +	if (cpu >= nr_cpu_ids)
> > +		return -EINVAL;
> >  
> >  	pr_debug("Bringing up CPU with APIC ID %d in VTL2...\n", apicid);
> >  	vp_id = hv_vtl_apicid_to_vp_id(apicid);
> > @@ -213,7 +224,7 @@ static int hv_vtl_wakeup_secondary_cpu(u32 apicid, unsigned long start_eip)
> >  		return -EINVAL;
> >  	}
> >  
> > -	return hv_vtl_bringup_vcpu(vp_id, start_eip);
> > +	return hv_vtl_bringup_vcpu(vp_id, cpu, start_eip);
> >  }
> >  
> >  int __init hv_vtl_early_init(void)
> > diff --git a/drivers/hv/Kconfig b/drivers/hv/Kconfig
> > index 00242107d62e..862c47b191af 100644
> > --- a/drivers/hv/Kconfig
> > +++ b/drivers/hv/Kconfig
> > @@ -16,6 +16,7 @@ config HYPERV
> >  config HYPERV_VTL_MODE
> >  	bool "Enable Linux to boot in VTL context"
> >  	depends on X86_64 && HYPERV
> > +	depends on SMP
> >  	default n
> >  	help
> >  	  Virtual Secure Mode (VSM) is a set of hypervisor capabilities and
> > -- 
> > 2.34.1
> > 

Re: [PATCH v3] x86/hyperv: Use per cpu initial stack for vtl context

[Wei Liu]

On Sun, Mar 03, 2024 at 11:08:17PM -0800, Saurabh Singh Sengar wrote:
> On Mon, Mar 04, 2024 at 06:24:27AM +0000, Wei Liu wrote:
> > On Sun, Mar 03, 2024 at 12:01:36AM -0800, Saurabh Sengar wrote:
> > > Currently, the secondary CPUs in Hyper-V VTL context lack support for
> > > parallel startup. Therefore, relying on the single initial_stack fetched
> > > from the current task structure suffices for all vCPUs.
> > > 
> > > However, common initial_stack risks stack corruption when parallel startup
> > > is enabled. In order to facilitate parallel startup, use the initial_stack
> > > from the per CPU idle thread instead of the current task.
> > > 
> > > Fixes: 18415f33e2ac ("cpu/hotplug: Allow "parallel" bringup up to CPUHP_BP_KICK_AP_STATE")
> > 
> > I don't think this patch is buggy. Instead, it exposes an assumption in
> > the VTL code. So this either should be dropped or point to the patch
> > which introduces the assumption.
> > 
> > Let me know what you would prefer.
> 
> The VTL code will crash if this fix is not present post above mentioned patch:
> 18415f33e2ac ("cpu/hotplug: Allow "parallel" bringup up to CPUHP_BP_KICK_AP_STATE").
> So I would prefer a fixes which added the assumption in VTL:
> 
> Fixes: 3be1bc2fe9d2 ("x86/hyperv: VTL support for Hyper-V")
> 
> Please let me know if you need V4 for it.

No need to repost. I can change the commit message.

Thanks,
Wei.

Re: [PATCH v3] x86/hyperv: Use per cpu initial stack for vtl context

[Wei Liu]

On Tue, Mar 05, 2024 at 12:26:50AM +0000, Wei Liu wrote:
> On Sun, Mar 03, 2024 at 11:08:17PM -0800, Saurabh Singh Sengar wrote:
> > On Mon, Mar 04, 2024 at 06:24:27AM +0000, Wei Liu wrote:
> > > On Sun, Mar 03, 2024 at 12:01:36AM -0800, Saurabh Sengar wrote:
> > > > Currently, the secondary CPUs in Hyper-V VTL context lack support for
> > > > parallel startup. Therefore, relying on the single initial_stack fetched
> > > > from the current task structure suffices for all vCPUs.
> > > > 
> > > > However, common initial_stack risks stack corruption when parallel startup
> > > > is enabled. In order to facilitate parallel startup, use the initial_stack
> > > > from the per CPU idle thread instead of the current task.
> > > > 
> > > > Fixes: 18415f33e2ac ("cpu/hotplug: Allow "parallel" bringup up to CPUHP_BP_KICK_AP_STATE")
> > > 
> > > I don't think this patch is buggy. Instead, it exposes an assumption in
> > > the VTL code. So this either should be dropped or point to the patch
> > > which introduces the assumption.
> > > 
> > > Let me know what you would prefer.
> > 
> > The VTL code will crash if this fix is not present post above mentioned patch:
> > 18415f33e2ac ("cpu/hotplug: Allow "parallel" bringup up to CPUHP_BP_KICK_AP_STATE").
> > So I would prefer a fixes which added the assumption in VTL:
> > 
> > Fixes: 3be1bc2fe9d2 ("x86/hyperv: VTL support for Hyper-V")
> > 
> > Please let me know if you need V4 for it.
> 
> No need to repost. I can change the commit message.

Applied to hyperv-next with the new Fixes tag. Thanks.