[PATCH v3 04/10] x86/sev: Cache AP Jump Table Address

[PATCH v3 04/10] x86/sev: Cache AP Jump Table Address

[Sean Christopherson]

On Thu, Jan 27, 2022, Joerg Roedel wrote:
> From: Joerg Roedel <jroedel@suse.de>
> 
> Store the physical address of the AP jump table in kernel memory so
> that it does not need to be fetched from the Hypervisor again.

This doesn't explain why the kernel would retrieve the jump table more than once,
e.g. at this point in the series, this can only ever be called once.

> Signed-off-by: Joerg Roedel <jroedel@suse.de>
> ---
>  arch/x86/kernel/sev.c | 28 +++++++++++++++-------------
>  1 file changed, 15 insertions(+), 13 deletions(-)
> 
> diff --git a/arch/x86/kernel/sev.c b/arch/x86/kernel/sev.c
> index 8a4317fa699a..969ef9855bb5 100644
> --- a/arch/x86/kernel/sev.c
> +++ b/arch/x86/kernel/sev.c
> @@ -43,6 +43,9 @@ static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);
>   */
>  static struct ghcb __initdata *boot_ghcb;
>  
> +/* Cached AP jump table Address */
> +static phys_addr_t jump_table_pa;
> +
>  /* #VC handler runtime per-CPU data */
>  struct sev_es_runtime_data {
>  	struct ghcb ghcb_page;
> @@ -523,12 +526,14 @@ void noinstr __sev_es_nmi_complete(void)
>  	__sev_put_ghcb(&state);
>  }
>  
> -static u64 get_jump_table_addr(void)
> +static phys_addr_t get_jump_table_addr(void)

Not new, but I believe this can be tagged __init.

>  {
>  	struct ghcb_state state;
>  	unsigned long flags;
>  	struct ghcb *ghcb;
> -	u64 ret = 0;
> +
> +	if (jump_table_pa)
> +		return jump_table_pa;
>  
>  	local_irq_save(flags);
>  
> @@ -544,39 +549,36 @@ static u64 get_jump_table_addr(void)
>  
>  	if (ghcb_sw_exit_info_1_is_valid(ghcb) &&
>  	    ghcb_sw_exit_info_2_is_valid(ghcb))
> -		ret = ghcb->save.sw_exit_info_2;
> +		jump_table_pa = (phys_addr_t)ghcb->save.sw_exit_info_2;
>  
>  	__sev_put_ghcb(&state);
>  
>  	local_irq_restore(flags);
>  
> -	return ret;
> +	return jump_table_pa;
>  }
>  
>  int sev_es_setup_ap_jump_table(struct real_mode_header *rmh)

__init here too.

>  {
>  	u16 startup_cs, startup_ip;
> -	phys_addr_t jump_table_pa;
> -	u64 jump_table_addr;
>  	u16 __iomem *jump_table;
> +	phys_addr_t pa;
>  
> -	jump_table_addr = get_jump_table_addr();
> +	pa = get_jump_table_addr();
>  
>  	/* On UP guests there is no jump table so this is not a failure */

Does anything actually check that the jump table is valid for SMP guests?

> -	if (!jump_table_addr)
> +	if (!pa)

Using '0' for "not valid" is funky because '0' isn't technically an illegal GPA,
and because it means the address (or lack thereof) isn't cached on a single-vCPU
guest.

[PATCH v3 05/10] x86/sev: Setup code to park APs in the AP Jump Table

[Sean Christopherson]

On Thu, Jan 27, 2022, Joerg Roedel wrote:
> +static int __init sev_setup_ap_jump_table(void)

This name is really confusing.  AFAICT, it's specific to SEV-ES, but used only
"sev" for the namespace because sev_es_setup_ap_jump_table() already exists.
I assume this variant is purely for parking/offlining vCPUs?  Adding that in the
name would be helpful.

The two flows are also very, very similar, but apparently do slightly different
things.  Even more odd is that this version applies different sanity checks on
the address than the existing code.  It should be fairly simple to extract a
common helper.  That would likely help with naming problem too.

> +{
> +	size_t blob_size = rm_ap_jump_table_blob_end - rm_ap_jump_table_blob;
> +	u16 startup_cs, startup_ip;
> +	u16 __iomem *jump_table;
> +	phys_addr_t pa;
> +
> +	if (!cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT))
> +		return 0;
> +
> +	if (ghcb_info.vm_proto < 2) {
> +		pr_warn("AP jump table parking requires at least GHCB protocol version 2\n");
> +		return 0;
> +	}
> +
> +	pa = get_jump_table_addr();
> +
> +	/* On UP guests there is no jump table so this is not a failure */
> +	if (!pa)
> +		return 0;
> +
> +	/* Check overflow and size for untrusted jump table address */
> +	if (pa + PAGE_SIZE < pa || pa + PAGE_SIZE > SZ_4G) {
> +		pr_info("AP jump table is above 4GB or address overflow - not enabling AP jump table parking\n");
> +		return 0;
> +	}
> +
> +	jump_table = ioremap_encrypted(pa, PAGE_SIZE);
> +	if (WARN_ON(!jump_table))
> +		return -EINVAL;
> +
> +	/*
> +	 * Save reset vector to restore it later because the blob will
> +	 * overwrite it.
> +	 */
> +	startup_ip = jump_table[0];
> +	startup_cs = jump_table[1];
> +
> +	/* Install AP jump table Blob with real mode AP parking code */
> +	memcpy_toio(jump_table, rm_ap_jump_table_blob, blob_size);
> +
> +	/* Setup AP jump table GDT */
> +	sev_es_setup_ap_jump_table_data(jump_table, (u32)pa);
> +
> +	writew(startup_ip, &jump_table[0]);
> +	writew(startup_cs, &jump_table[1]);
> +
> +	iounmap(jump_table);
> +
> +	pr_info("AP jump table Blob successfully set up\n");
> +
> +	/* Mark AP jump table blob as available */
> +	sev_ap_jumptable_blob_installed = true;
> +
> +	return 0;
> +}
> +core_initcall(sev_setup_ap_jump_table);

[PATCH v3 00/10] x86/sev: KEXEC/KDUMP support for SEV-ES guests

[Tao Liu]

On Thu, Jan 27, 2022 at 11:10:34AM +0100, Joerg Roedel wrote:
> From: Joerg Roedel <jroedel@suse.de>
> 
> Hi,
> 
> here are changes to enable kexec/kdump in SEV-ES guests. The biggest
> problem for supporting kexec/kdump under SEV-ES is to find a way to
> hand the non-boot CPUs (APs) from one kernel to another.
> 
> Without SEV-ES the first kernel parks the CPUs in a HLT loop until
> they get reset by the kexec'ed kernel via an INIT-SIPI-SIPI sequence.
> For virtual machines the CPU reset is emulated by the hypervisor,
> which sets the vCPU registers back to reset state.
> 
> This does not work under SEV-ES, because the hypervisor has no access
> to the vCPU registers and can't make modifications to them. So an
> SEV-ES guest needs to reset the vCPU itself and park it using the
> AP-reset-hold protocol. Upon wakeup the guest needs to jump to
> real-mode and to the reset-vector configured in the AP-Jump-Table.
> 
> The code to do this is the main part of this patch-set. It works by
> placing code on the AP Jump-Table page itself to park the vCPU and for
> jumping to the reset vector upon wakeup. The code on the AP Jump Table
> runs in 16-bit protected mode with segment base set to the beginning
> of the page. The AP Jump-Table is usually not within the first 1MB of
> memory, so the code can't run in real-mode.
> 
> The AP Jump-Table is the best place to put the parking code, because
> the memory is owned, but read-only by the firmware and writeable by
> the OS. Only the first 4 bytes are used for the reset-vector, leaving
> the rest of the page for code/data/stack to park a vCPU. The code
> can't be in kernel memory because by the time the vCPU wakes up the
> memory will be owned by the new kernel, which might have overwritten it
> already.
> 
> The other patches add initial GHCB Version 2 protocol support, because
> kexec/kdump need the MSR-based (without a GHCB) AP-reset-hold VMGEXIT,
> which is a GHCB protocol version 2 feature.
> 
> The kexec'ed kernel is also entered via the decompressor and needs
> MMIO support there, so this patch-set also adds MMIO #VC support to
> the decompressor and support for handling CLFLUSH instructions.
> 
> Finally there is also code to disable kexec/kdump support at runtime
> when the environment does not support it (e.g. no GHCB protocol
> version 2 support or AP Jump Table over 4GB).
> 
> The diffstat looks big, but most of it is moving code for MMIO #VC
> support around to make it available to the decompressor.
> 
> The previous version of this patch-set can be found here:
> 
> 	https://lore.kernel.org/lkml/20210913155603.28383-1-joro at 8bytes.org/
> 
> Please review.
> 
> Thanks,
> 
> 	Joerg
> 
> Changes v2->v3:
> 
> 	- Rebased to v5.17-rc1
> 	- Applied most review comments by Boris
> 	- Use the name 'AP jump table' consistently
> 	- Make kexec-disabling for unsupported guests x86-specific
> 	- Cleanup and consolidate patches to detect GHCB v2 protocol
> 	  support
> 
> Joerg Roedel (10):
>   x86/kexec/64: Disable kexec when SEV-ES is active
>   x86/sev: Save and print negotiated GHCB protocol version
>   x86/sev: Set GHCB data structure version
>   x86/sev: Cache AP Jump Table Address
>   x86/sev: Setup code to park APs in the AP Jump Table
>   x86/sev: Park APs on AP Jump Table with GHCB protocol version 2
>   x86/sev: Use AP Jump Table blob to stop CPU
>   x86/sev: Add MMIO handling support to boot/compressed/ code
>   x86/sev: Handle CLFLUSH MMIO events
>   x86/kexec/64: Support kexec under SEV-ES with AP Jump Table Blob
> 
>  arch/x86/boot/compressed/sev.c          |  45 +-
>  arch/x86/include/asm/insn-eval.h        |   1 +
>  arch/x86/include/asm/realmode.h         |   5 +
>  arch/x86/include/asm/sev-ap-jumptable.h |  29 +
>  arch/x86/include/asm/sev.h              |  11 +-
>  arch/x86/kernel/machine_kexec_64.c      |  12 +
>  arch/x86/kernel/process.c               |   8 +
>  arch/x86/kernel/sev-shared.c            | 233 +++++-
>  arch/x86/kernel/sev.c                   | 404 +++++------
>  arch/x86/lib/insn-eval-shared.c         | 913 ++++++++++++++++++++++++
>  arch/x86/lib/insn-eval.c                | 909 +----------------------
>  arch/x86/realmode/Makefile              |   9 +-
>  arch/x86/realmode/rm/Makefile           |  11 +-
>  arch/x86/realmode/rm/header.S           |   3 +
>  arch/x86/realmode/rm/sev.S              |  85 +++
>  arch/x86/realmode/rmpiggy.S             |   6 +
>  arch/x86/realmode/sev/Makefile          |  33 +
>  arch/x86/realmode/sev/ap_jump_table.S   | 131 ++++
>  arch/x86/realmode/sev/ap_jump_table.lds |  24 +
>  19 files changed, 1730 insertions(+), 1142 deletions(-)
>  create mode 100644 arch/x86/include/asm/sev-ap-jumptable.h
>  create mode 100644 arch/x86/lib/insn-eval-shared.c
>  create mode 100644 arch/x86/realmode/rm/sev.S
>  create mode 100644 arch/x86/realmode/sev/Makefile
>  create mode 100644 arch/x86/realmode/sev/ap_jump_table.S
>  create mode 100644 arch/x86/realmode/sev/ap_jump_table.lds
> 
> 
> base-commit: e783362eb54cd99b2cac8b3a9aeac942e6f6ac07
> -- 
> 2.34.1
>

Hi Joerg,

I tried the patch set with 5.17.0-rc1 kernel, and I have a few questions:

1) Is it a bug or should qemu-kvm 6.2.0 be patched with specific patch? Because
   I found it will exit with 0 when I tried to reboot the VM with sev-es enabled.
   However with only sev enabled, the VM can do reboot with no problem:

[root at dell-per7525-03 ~]# virsh start TW-SEV-ES --console
....
Fedora Linux 35 (Server Edition)
Kernel 5.17.0-rc1 on an x86_64 (ttyS0)
....
[root at fedora ~]# reboot
.....
[   48.077682] reboot: Restarting system
[   48.078109] reboot: machine restart
                       ^^^^^^^^^^^^^^^ guest vm reached restart
[root at dell-per7525-03 ~]# echo $?
0
^^^ qemu-kvm exit with 0, no reboot back to normal VM kernel
[root at dell-per7525-03 ~]#

2) With sev-es enabled and the 2 patch sets applied: A) [PATCH v3 00/10] x86/sev:
KEXEC/KDUMP support for SEV-ES guests, and B) [PATCH v6 0/7] KVM: SVM: Add initial
GHCB protocol version 2 support. I can enable kdump and have vmcore generated:

[root at fedora ~]# dmesg|grep -i sev
[    0.030600] SEV: Hypervisor GHCB protocol version support: min=1 max=2
[    0.030602] SEV: Using GHCB protocol version 2
[    0.296144] AMD Memory Encryption Features active: SEV SEV-ES
[    0.450991] SEV: AP jump table Blob successfully set up
[root at fedora ~]# kdumpctl status
kdump: Kdump is operational

However without the 2 patch sets, I can also enable kdump and have vmcore generated:

[root at fedora ~]# dmesg|grep -i sev
[    0.295754] AMD Memory Encryption Features active: SEV SEV-ES
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ patch set A & B 
	       not applied, so only have this string.
[root at fedora ~]# echo c > /proc/sysrq-trigger
...
[    2.759403] kdump[549]: saving vmcore-dmesg.txt to /sysroot/var/crash/127.0.0.1-2022-04-18-05:58:50/
[    2.804355] kdump[555]: saving vmcore-dmesg.txt complete
[    2.806915] kdump[557]: saving vmcore
                           ^^^^^^^^^^^^^ vmcore can still be generated
...
[    7.068981] reboot: Restarting system
[    7.069340] reboot: machine restart

[root at dell-per7525-03 ~]# echo $?
0
^^^ same exit issue as question 1.

I doesn't have a complete technical background of the patch set, but isn't
it the issue which this patch set is trying to solve? Or I missed something?

Thanks,
Tao Liu
 
> _______________________________________________
> Virtualization mailing list
> Virtualization at lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH v3 00/10] x86/sev: KEXEC/KDUMP support for SEV-ES guests

[Tom Lendacky]

On 4/29/22 04:06, Tao Liu wrote:
> On Thu, Jan 27, 2022 at 11:10:34AM +0100, Joerg Roedel wrote:

> 
> Hi Joerg,
> 
> I tried the patch set with 5.17.0-rc1 kernel, and I have a few questions:
> 
> 1) Is it a bug or should qemu-kvm 6.2.0 be patched with specific patch? Because
>     I found it will exit with 0 when I tried to reboot the VM with sev-es enabled.
>     However with only sev enabled, the VM can do reboot with no problem:

Qemu was specifically patched to exit on reboot with SEV-ES guests. Qemu 
performs a reboot by resetting the vCPU state, which can't be done with an 
SEV-ES guest because the vCPU state is encrypted.

> 
> [root at dell-per7525-03 ~]# virsh start TW-SEV-ES --console
> ....
> Fedora Linux 35 (Server Edition)
> Kernel 5.17.0-rc1 on an x86_64 (ttyS0)
> ....
> [root at fedora ~]# reboot
> .....
> [   48.077682] reboot: Restarting system
> [   48.078109] reboot: machine restart
>                         ^^^^^^^^^^^^^^^ guest vm reached restart
> [root at dell-per7525-03 ~]# echo $?
> 0
> ^^^ qemu-kvm exit with 0, no reboot back to normal VM kernel
> [root at dell-per7525-03 ~]#
> 
> 2) With sev-es enabled and the 2 patch sets applied: A) [PATCH v3 00/10] x86/sev:
> KEXEC/KDUMP support for SEV-ES guests, and B) [PATCH v6 0/7] KVM: SVM: Add initial
> GHCB protocol version 2 support. I can enable kdump and have vmcore generated:
> 
> [root at fedora ~]# dmesg|grep -i sev
> [    0.030600] SEV: Hypervisor GHCB protocol version support: min=1 max=2
> [    0.030602] SEV: Using GHCB protocol version 2
> [    0.296144] AMD Memory Encryption Features active: SEV SEV-ES
> [    0.450991] SEV: AP jump table Blob successfully set up
> [root at fedora ~]# kdumpctl status
> kdump: Kdump is operational
> 
> However without the 2 patch sets, I can also enable kdump and have vmcore generated:
> 
> [root at fedora ~]# dmesg|grep -i sev
> [    0.295754] AMD Memory Encryption Features active: SEV SEV-ES
>                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ patch set A & B
> 	       not applied, so only have this string.
> [root at fedora ~]# echo c > /proc/sysrq-trigger
> ...
> [    2.759403] kdump[549]: saving vmcore-dmesg.txt to /sysroot/var/crash/127.0.0.1-2022-04-18-05:58:50/
> [    2.804355] kdump[555]: saving vmcore-dmesg.txt complete
> [    2.806915] kdump[557]: saving vmcore
>                             ^^^^^^^^^^^^^ vmcore can still be generated
> ...
> [    7.068981] reboot: Restarting system
> [    7.069340] reboot: machine restart
> 
> [root at dell-per7525-03 ~]# echo $?
> 0
> ^^^ same exit issue as question 1.
> 
> I doesn't have a complete technical background of the patch set, but isn't
> it the issue which this patch set is trying to solve? Or I missed something?

The main goal of this patch set is to really to solve the ability to 
perform a kexec. I would expect kdump to work since kdump shuts down all 
but the executing vCPU and performs its operations before "rebooting" 
(which will exit Qemu as I mentioned above). But kexec requires the need 
to restart the APs from within the guest after they have been stopped. 
That requires specific support and actions on the part of the guest kernel 
in how the APs are stopped and restarted.

Thanks,
Tom

> 
> Thanks,
> Tao Liu
>   
>> _______________________________________________
>> Virtualization mailing list
>> Virtualization at lists.linux-foundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/virtualization
> 

Re: [PATCH v3 00/10] x86/sev: KEXEC/KDUMP support for SEV-ES guests

[Tao Liu]

Hi Tom,

On Fri, Apr 29, 2022 at 08:08:28AM -0500, Tom Lendacky wrote:
> On 4/29/22 04:06, Tao Liu wrote:
> > On Thu, Jan 27, 2022 at 11:10:34AM +0100, Joerg Roedel wrote:
> 
> > 
> > Hi Joerg,
> > 
> > I tried the patch set with 5.17.0-rc1 kernel, and I have a few questions:
> > 
> > 1) Is it a bug or should qemu-kvm 6.2.0 be patched with specific patch? Because
> >     I found it will exit with 0 when I tried to reboot the VM with sev-es enabled.
> >     However with only sev enabled, the VM can do reboot with no problem:
> 
> Qemu was specifically patched to exit on reboot with SEV-ES guests. Qemu
> performs a reboot by resetting the vCPU state, which can't be done with an
> SEV-ES guest because the vCPU state is encrypted.
> 

Sorry for the late response, and thank you for the explanation!

> > 
> > [root@dell-per7525-03 ~]# virsh start TW-SEV-ES --console
> > ....
> > Fedora Linux 35 (Server Edition)
> > Kernel 5.17.0-rc1 on an x86_64 (ttyS0)
> > ....
> > [root@fedora ~]# reboot
> > .....
> > [   48.077682] reboot: Restarting system
> > [   48.078109] reboot: machine restart
> >                         ^^^^^^^^^^^^^^^ guest vm reached restart
> > [root@dell-per7525-03 ~]# echo $?
> > 0
> > ^^^ qemu-kvm exit with 0, no reboot back to normal VM kernel
> > [root@dell-per7525-03 ~]#
> > 
> > 2) With sev-es enabled and the 2 patch sets applied: A) [PATCH v3 00/10] x86/sev:
> > KEXEC/KDUMP support for SEV-ES guests, and B) [PATCH v6 0/7] KVM: SVM: Add initial
> > GHCB protocol version 2 support. I can enable kdump and have vmcore generated:
> > 
> > [root@fedora ~]# dmesg|grep -i sev
> > [    0.030600] SEV: Hypervisor GHCB protocol version support: min=1 max=2
> > [    0.030602] SEV: Using GHCB protocol version 2
> > [    0.296144] AMD Memory Encryption Features active: SEV SEV-ES
> > [    0.450991] SEV: AP jump table Blob successfully set up
> > [root@fedora ~]# kdumpctl status
> > kdump: Kdump is operational
> > 
> > However without the 2 patch sets, I can also enable kdump and have vmcore generated:
> > 
> > [root@fedora ~]# dmesg|grep -i sev
> > [    0.295754] AMD Memory Encryption Features active: SEV SEV-ES
> >                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ patch set A & B
> > 	       not applied, so only have this string.
> > [root@fedora ~]# echo c > /proc/sysrq-trigger
> > ...
> > [    2.759403] kdump[549]: saving vmcore-dmesg.txt to /sysroot/var/crash/127.0.0.1-2022-04-18-05:58:50/
> > [    2.804355] kdump[555]: saving vmcore-dmesg.txt complete
> > [    2.806915] kdump[557]: saving vmcore
> >                             ^^^^^^^^^^^^^ vmcore can still be generated
> > ...
> > [    7.068981] reboot: Restarting system
> > [    7.069340] reboot: machine restart
> > 
> > [root@dell-per7525-03 ~]# echo $?
> > 0
> > ^^^ same exit issue as question 1.
> > 
> > I doesn't have a complete technical background of the patch set, but isn't
> > it the issue which this patch set is trying to solve? Or I missed something?
> 
> The main goal of this patch set is to really to solve the ability to perform
> a kexec. I would expect kdump to work since kdump shuts down all but the
> executing vCPU and performs its operations before "rebooting" (which will
> exit Qemu as I mentioned above). But kexec requires the need to restart the
> APs from within the guest after they have been stopped. That requires
> specific support and actions on the part of the guest kernel in how the APs
> are stopped and restarted.

Recently I got one sev-es flaged machine borrowed and retested the patch, which
worked fine for kexec when sev-es enabled. With the patchset applied in 5.17.0-rc1, 
kexec'ed kernel can bring up all APs with no problem.

However as for kdump, I find one issue. Although kdump kernel can work well on one
cpu, but we can still enable multi-cpus by removing the "nr_cpus=1" kernel parameter
in kdump sysconfig. I was expecting kdump kernel can bring up all APs as kexec did,
however:

[    0.000000] Command line: elfcorehdr=0x5b000000 BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.17.0-rc1+ ro resume=/dev/mapper/rhel-swap biosdevname=0 net.ifnames=0 console=ttyS0 irqpoll reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug transparent_hugepage=never nokaslr novmcoredd hest_disable disable_cpu_apicid=0 iTCO_wdt.pretimeout=0
...
[    0.376663] smp: Bringing up secondary CPUs ...
[    0.377599] x86: Booting SMP configuration:
[    0.378342] .... node  #0, CPUs:      #1
[   10.377698] smpboot: do_boot_cpu failed(-1) to wakeup CPU#1
[   10.379882]  #2
[   20.379645] smpboot: do_boot_cpu failed(-1) to wakeup CPU#2
[   20.380648] smp: Brought up 1 node, 1 CPU
[   20.381600] smpboot: Max logical packages: 4
[   20.382597] smpboot: Total of 1 processors activated (4192.00 BogoMIPS)

Turns out for kdump, the APs were not stopped properly, so I modified the following code:

--- a/arch/x86/kernel/reboot.c
+++ b/arch/x86/kernel/reboot.c
@@ -26,6 +26,7 @@
 #include <asm/cpu.h>
 #include <asm/nmi.h>
 #include <asm/smp.h>
+#include <asm/sev.h>
 
 #include <linux/ctype.h>
 #include <linux/mc146818rtc.h>
@@ -821,6 +822,7 @@ static int crash_nmi_callback(unsigned int val, struct pt_regs *regs)
 
        atomic_dec(&waiting_for_crash_ipi);
        /* Assume hlt works */
+       sev_es_stop_this_cpu();
        halt();
        for (;;)
                cpu_relax();

[    0.000000] Command line: elfcorehdr=0x5b000000 BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.17.0-rc1-hack+ ro resume=/dev/mapper/rhel-swap biosdevname=0 net.ifnames=0 console=ttyS0 irqpoll reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug transparent_hugepage=never nokaslr novmcoredd hest_disable disable_cpu_apicid=0 iTCO_wdt.pretimeout=0
...
[    0.402618] smp: Bringing up secondary CPUs ...
[    0.403308] x86: Booting SMP configuration:
[    0.404171] .... node  #0, CPUs:      #1 #2 #3
[    0.407362] smp: Brought up 1 node, 4 CPUs
[    0.408907] smpboot: Max logical packages: 4
[    0.409172] smpboot: Total of 4 processors activated (16768.01 BogoMIPS)

Now all APs can work in kdump kernel.

Thanks,
Tao Liu

> 
> Thanks,
> Tom
> 
> > 
> > Thanks,
> > Tao Liu
> > > _______________________________________________
> > > Virtualization mailing list
> > > Virtualization@lists.linux-foundation.org
> > > https://lists.linuxfoundation.org/mailman/listinfo/virtualization
> > 
> 


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH v3 00/10] x86/sev: KEXEC/KDUMP support for SEV-ES guests

[Baoquan He]

Hi Joerg and Tom,

On 01/27/22 at 11:10am, Joerg Roedel wrote:
> From: Joerg Roedel <jroedel@suse.de>
> 
> Hi,
> 
> here are changes to enable kexec/kdump in SEV-ES guests. The biggest
> problem for supporting kexec/kdump under SEV-ES is to find a way to
> hand the non-boot CPUs (APs) from one kernel to another.
> 
> Without SEV-ES the first kernel parks the CPUs in a HLT loop until
> they get reset by the kexec'ed kernel via an INIT-SIPI-SIPI sequence.
> For virtual machines the CPU reset is emulated by the hypervisor,
> which sets the vCPU registers back to reset state.

Is there any plan for this patchset to proceed? Without this patchset,
it does fail kexec and kdump with multiple cpus.

Thanks
Baoquan

> 
> This does not work under SEV-ES, because the hypervisor has no access
> to the vCPU registers and can't make modifications to them. So an
> SEV-ES guest needs to reset the vCPU itself and park it using the
> AP-reset-hold protocol. Upon wakeup the guest needs to jump to
> real-mode and to the reset-vector configured in the AP-Jump-Table.
> 
> The code to do this is the main part of this patch-set. It works by
> placing code on the AP Jump-Table page itself to park the vCPU and for
> jumping to the reset vector upon wakeup. The code on the AP Jump Table
> runs in 16-bit protected mode with segment base set to the beginning
> of the page. The AP Jump-Table is usually not within the first 1MB of
> memory, so the code can't run in real-mode.
> 
> The AP Jump-Table is the best place to put the parking code, because
> the memory is owned, but read-only by the firmware and writeable by
> the OS. Only the first 4 bytes are used for the reset-vector, leaving
> the rest of the page for code/data/stack to park a vCPU. The code
> can't be in kernel memory because by the time the vCPU wakes up the
> memory will be owned by the new kernel, which might have overwritten it
> already.
> 
> The other patches add initial GHCB Version 2 protocol support, because
> kexec/kdump need the MSR-based (without a GHCB) AP-reset-hold VMGEXIT,
> which is a GHCB protocol version 2 feature.
> 
> The kexec'ed kernel is also entered via the decompressor and needs
> MMIO support there, so this patch-set also adds MMIO #VC support to
> the decompressor and support for handling CLFLUSH instructions.
> 
> Finally there is also code to disable kexec/kdump support at runtime
> when the environment does not support it (e.g. no GHCB protocol
> version 2 support or AP Jump Table over 4GB).
> 
> The diffstat looks big, but most of it is moving code for MMIO #VC
> support around to make it available to the decompressor.
> 
> The previous version of this patch-set can be found here:
> 
> 	https://lore.kernel.org/lkml/20210913155603.28383-1-joro@8bytes.org/
> 
> Please review.
> 
> Thanks,
> 
> 	Joerg
> 
> Changes v2->v3:
> 
> 	- Rebased to v5.17-rc1
> 	- Applied most review comments by Boris
> 	- Use the name 'AP jump table' consistently
> 	- Make kexec-disabling for unsupported guests x86-specific
> 	- Cleanup and consolidate patches to detect GHCB v2 protocol
> 	  support
> 
> Joerg Roedel (10):
>   x86/kexec/64: Disable kexec when SEV-ES is active
>   x86/sev: Save and print negotiated GHCB protocol version
>   x86/sev: Set GHCB data structure version
>   x86/sev: Cache AP Jump Table Address
>   x86/sev: Setup code to park APs in the AP Jump Table
>   x86/sev: Park APs on AP Jump Table with GHCB protocol version 2
>   x86/sev: Use AP Jump Table blob to stop CPU
>   x86/sev: Add MMIO handling support to boot/compressed/ code
>   x86/sev: Handle CLFLUSH MMIO events
>   x86/kexec/64: Support kexec under SEV-ES with AP Jump Table Blob
> 
>  arch/x86/boot/compressed/sev.c          |  45 +-
>  arch/x86/include/asm/insn-eval.h        |   1 +
>  arch/x86/include/asm/realmode.h         |   5 +
>  arch/x86/include/asm/sev-ap-jumptable.h |  29 +
>  arch/x86/include/asm/sev.h              |  11 +-
>  arch/x86/kernel/machine_kexec_64.c      |  12 +
>  arch/x86/kernel/process.c               |   8 +
>  arch/x86/kernel/sev-shared.c            | 233 +++++-
>  arch/x86/kernel/sev.c                   | 404 +++++------
>  arch/x86/lib/insn-eval-shared.c         | 913 ++++++++++++++++++++++++
>  arch/x86/lib/insn-eval.c                | 909 +----------------------
>  arch/x86/realmode/Makefile              |   9 +-
>  arch/x86/realmode/rm/Makefile           |  11 +-
>  arch/x86/realmode/rm/header.S           |   3 +
>  arch/x86/realmode/rm/sev.S              |  85 +++
>  arch/x86/realmode/rmpiggy.S             |   6 +
>  arch/x86/realmode/sev/Makefile          |  33 +
>  arch/x86/realmode/sev/ap_jump_table.S   | 131 ++++
>  arch/x86/realmode/sev/ap_jump_table.lds |  24 +
>  19 files changed, 1730 insertions(+), 1142 deletions(-)
>  create mode 100644 arch/x86/include/asm/sev-ap-jumptable.h
>  create mode 100644 arch/x86/lib/insn-eval-shared.c
>  create mode 100644 arch/x86/realmode/rm/sev.S
>  create mode 100644 arch/x86/realmode/sev/Makefile
>  create mode 100644 arch/x86/realmode/sev/ap_jump_table.S
>  create mode 100644 arch/x86/realmode/sev/ap_jump_table.lds
> 
> 
> base-commit: e783362eb54cd99b2cac8b3a9aeac942e6f6ac07
> -- 
> 2.34.1
> 


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec