Re: [Regression v5.19-rc1] crash kexec fails to boot the 2nd kernel (Re: [PATCH v12 38/46] x86/sev: Add SEV-SNP feature detection/setup)

[Michael Roth <michael.roth@amd.com>]

On Fri, Jun 24, 2022 at 12:44:52AM +0000, NOMURA JUNICHI(野村 淳一) wrote:
> I found crash kexec fails to boot the 2nd kernel since v5.19-rc1 and
> git bisect points to this as a bad commit:
> 
>   commit b190a043c49af4587f5e157053f909192820522a
>   Author: Michael Roth <michael.roth@amd.com>
>   Date:   Thu Feb 24 10:56:18 2022 -0600
> 
>     x86/sev: Add SEV-SNP feature detection/setup
> 
>     Initial/preliminary detection of SEV-SNP is done via the Confidential
>     Computing blob. Check for it prior to the normal SEV/SME feature
>     initialization, and add some sanity checks to confirm it agrees with
>     SEV-SNP CPUID/MSR bits.
> 
> The problem seems to occur when find_cc_blob_setup_data() walks setup_data
> chain.  If the code is modified to do nothing in find_cc_blob_setup_data(),
> the 2nd kernel boots fine.
> 
> On my system, the chain of setup_data looks like following on regular boot:
>   setup_data: type=0x3 addr=0x9e9e5018 next=0x9e9dc018
>   setup_data: type=0x3 addr=0x9e9dc018 next=0x9e9d2018
>   setup_data: type=0x3 addr=0x9e9d2018 next=0x8a27b018
>   setup_data: type=0x3 addr=0x8a27b018 next=0x8a218018
>   setup_data: type=0x3 addr=0x8a218018 next=0x9e9a0018
>   setup_data: type=0x3 addr=0x9e9a0018 next=0x8a1e6018
>   setup_data: type=0x3 addr=0x8a1e6018 next=0x8a1b4018
>   setup_data: type=0x3 addr=0x8a1b4018 next=0x8a182018
>   setup_data: type=0x3 addr=0x8a182018 next=0x8a056018
>   setup_data: type=0x3 addr=0x8a056018 next=0x8a020018
>   setup_data: type=0x3 addr=0x8a020018 next=0x89fea018
>   setup_data: type=0x3 addr=0x89fea018 next=0x0
> 
> OTOH, it looks like following on crash kexec boot:
>   setup_data: type=0x4 addr=0x2e000000 next=0x0

Hi,

Thanks for the debug info. I haven't been able to reproduce this on the
Milan or Cascade Lake systems I've tried, with kexec -l/-p, and well as
with/without -s, so there may be something hardware/environment-specific
going on here, so I could really use your help to test possible fixes.

> 
> Other places that parses setup_data uses early_memremap() before
> accessing the data (e.g. parse_setup_data()).  I wonder if the lack of
> remapping causes the problem but find_cc_blob is too early in the
> boot process for early_memremap to work.

I think this might be the case. Prior to early_memremap() being
available, we need to rely on the initialize identity map set up by the
decompression kernel. It has some stuff to add mappings for boot_params
and whatnot, but I don't see where boot_params->hdr.setup_data is
handled.

If you use kexec -s to force kexec_file_load, then the kernel sets it up
so that boot_params->hdr.setup_data points to some memory just after
boot_params, and boot/compressed uses 2M pages in its identity map, so
that generally ends up handling the whole range.

But if you use kexec's default kexec_load functionality, setup_data might
be allocated elsewhere, so in that case we might need explicit mapping. I
noticed on my systems boot_params->hdr.setup_data seems to generally end
up at 0x100000 for some reason, and maybe that addr just happens to
get mapped for other reasons so I don't end up hitting the crash.

Could you give it a shot with the kexec -s flag and so if that works?

If so, can you apply the below potential fix, and retry your original
reproducer?

diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index 44c350d627c7..c548950981a2 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -110,6 +110,7 @@ void kernel_add_identity_map(unsigned long start, unsigned long end)
 void initialize_identity_maps(void *rmode)
 {
        unsigned long cmdline;
+       struct setup_data *sd;

        /* Exclude the encryption mask from __PHYSICAL_MASK */
        physical_mask &= ~sme_me_mask;
@@ -163,6 +164,12 @@ void initialize_identity_maps(void *rmode)
        cmdline = get_cmd_line_ptr();
        kernel_add_identity_map(cmdline, cmdline + COMMAND_LINE_SIZE);

+       sd = (struct setup_data *)boot_params->hdr.setup_data;
+       while (sd) {
+               kernel_add_identity_map((unsigned long)sd, (unsigned long)(sizeof(*sd) + sd->len));
+               sd = (struct setup_data *)sd->next;
+       }
+
        sev_prep_identity_maps(top_level_pgt);

        /* Load the new page-table. */


Thanks!

-Mike

> 
> -- 
> Jun'ichi Nomura, NEC Corporation / NEC Solution Innovators, Ltd.
>