[tip:x86/sev] [x86/sev] 06685975c2: BUG:kernel_failed_in_early-boot_stage,last_printk:Booting_the_kernel(entry_offset:#)

[tip:x86/sev] [x86/sev] 06685975c2: BUG:kernel_failed_in_early-boot_stage,last_printk:Booting_the_kernel(entry_offset:#)

[kernel test robot]

Hello,

kernel test robot noticed "BUG:kernel_failed_in_early-boot_stage,last_printk:Booting_the_kernel(entry_offset:#)" on:

commit: 06685975c2090e180851a0ff175c140188b6b54a ("x86/sev: Move SEV compilation units")
https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git x86/sev

[test failed on linux-next/master 0b58e108042b0ed28a71cd7edf5175999955b233]

in testcase: boot

compiler: gcc-13
test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G

(please refer to attached dmesg/kmsg for entire log/backtrace)


we don't have idea how this commit could cause early boot issue, so we rebuild
kernel for both this commit and its parent with the config in [1], then rerun
the tests, the issue is still persistent while keeping clean on parent.

3991b04d4870fd33 06685975c2090e180851a0ff175
---------------- ---------------------------
       fail:runs  %reproduction    fail:runs
           |             |             |
           :200         25%          50:50    dmesg.BUG:kernel_failed_in_early-boot_stage,last_printk:Booting_the_kernel(entry_offset:#)

since the config in [1] is a random config, we also try another x86_64 config
which we use for various functional tests, still can reproduce the issue.

we also try two i386 configs, but not reproduce the issue. FYI


If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202407091342.46d7dbb-oliver.sang@intel.com



Decompressing Linux... No EFI environment detected.
Parsing ELF... done.
Booting the kernel (entry_offset: 0x0000000000000ffa).
convert early boot stage from reboot-without-warning to failed
BUG: kernel failed in early-boot stage, last printk: Booting the kernel (entry_offset: 0x0000000000000ffa).
Linux version 6.10.0-rc2-00015-g06685975c209 #1
Command line: ip=::::vm-meta-70::dhcp root=/dev/ram0 RESULT_ROOT=/result/boot/1/vm-snb/debian-12-x86_64-20240206.cgz/x86_64-randconfig-r016-20230701/gcc-13/06685975c2090e180851a0ff175c140188b6b54a/21 BOOT_IMAGE=/pkg/linux/x86_64-randconfig-r016-20230701/gcc-13/06685975c2090e180851a0ff175c140188b6b54a/vmlinuz-6.10.0-rc2-00015-g06685975c209 branch=tip/x86/sev job=/lkp/jobs/scheduled/vm-meta-70/boot-1-debian-12-x86_64-20240206.cgz-x86_64-randconfig-r016-20230701-06685975c209-20240708-45480-br2l55-20.yaml user=lkp ARCH=x86_64 kconfig=x86_64-randconfig-r016-20230701 commit=06685975c2090e180851a0ff175c140188b6b54a nmi_watchdog=0 intremap=posted_msi vmalloc=256M initramfs_async=0 page_owner=on carrier_timeout=60 max_uptime=600 LKP_SERVER=internal-lkp-server selinux=0 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 net.ifnames=0 printk.devkmsg=on panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 drbd.minor_count=8 systemd.log_level=err ignore_loglevel console=tty0 earlyprintk=ttyS0,115200 console=ttyS0,115200 vga=normal rw rcuperf.shutdown=0 rcuscale.shutdown=0 refscale.shutdown=0 watchdog_thresh=240 audit=0 kunit.enable=0 ia32_emulation=on riscv_isa_fallback=1

Kboot worker: lkp-worker32
Elapsed time: 60


[1]
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20240709/202407091342.46d7dbb-oliver.sang@intel.com



-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [tip:x86/sev] [x86/sev] 06685975c2: BUG:kernel_failed_in_early-boot_stage,last_printk:Booting_the_kernel(entry_offset:#)

[Borislav Petkov]

On Tue, Jul 09, 2024 at 02:43:29PM +0800, kernel test robot wrote:
> kernel test robot noticed "BUG:kernel_failed_in_early-boot_stage,last_printk:Booting_the_kernel(entry_offset:#)" on:
> 
> commit: 06685975c2090e180851a0ff175c140188b6b54a ("x86/sev: Move SEV compilation units")
> https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git x86/sev

Does it go away when you try two patches after it:

a7e7185ccd2a ("x86/sev: Move the instrumentation exclusion bits too")

?

I might have to merge the two if so.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [tip:x86/sev] [x86/sev] 06685975c2: BUG:kernel_failed_in_early-boot_stage,last_printk:Booting_the_kernel(entry_offset:#)

[Oliver Sang]

hi, Boris,

On Tue, Jul 09, 2024 at 11:03:23AM +0200, Borislav Petkov wrote:
> On Tue, Jul 09, 2024 at 02:43:29PM +0800, kernel test robot wrote:
> > kernel test robot noticed "BUG:kernel_failed_in_early-boot_stage,last_printk:Booting_the_kernel(entry_offset:#)" on:
> > 
> > commit: 06685975c2090e180851a0ff175c140188b6b54a ("x86/sev: Move SEV compilation units")
> > https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git x86/sev
> 
> Does it go away when you try two patches after it:
> 
> a7e7185ccd2a ("x86/sev: Move the instrumentation exclusion bits too")
> 
> ?

no. when this bisect done, we tested the branch tip 8434cf006ceae,
the issue is still persistent.


* 8434cf006ceae (tip/x86/sev) Documentation/ABI/configfs-tsm: Fix an unexpected indentation silly
* a7e7185ccd2ac x86/sev: Move the instrumentation exclusion bits too
* 6a7d32f3a205f x86/sev: Do RMP memory coverage check after max_pfn has been set
* 06685975c2090 x86/sev: Move SEV compilation units
* 3991b04d4870f virt: sev-guest: Mark driver struct with __refdata to prevent section mismatch


> 
> I might have to merge the two if so.
> 
> Thx.
> 
> -- 
> Regards/Gruss,
>     Boris.
> 
> https://people.kernel.org/tglx/notes-about-netiquette

Re: [tip:x86/sev] [x86/sev] 06685975c2: BUG:kernel_failed_in_early-boot_stage,last_printk:Booting_the_kernel(entry_offset:#)

[Borislav Petkov]

On Wed, Jul 10, 2024 at 09:47:36AM +0800, Oliver Sang wrote:
> no. when this bisect done, we tested the branch tip 8434cf006ceae,
> the issue is still persistent.

Ok, I think that should fix it:

From: "Borislav Petkov (AMD)" <bp@alien8.de>
Date: Wed, 10 Jul 2024 10:16:18 +0200
Subject: [PATCH] x86/coco/sev: Disable KCSAN too

The SEV guest code was moved to x86/coco/ along with the instrumentation
exclusion bits. However, there is a blanket KCSAN disable in
arch/x86/kernel/Makefile due to boot hangs.

Copy that disable to the coco's sev Makefile too.

Fixes:  Fixes: 06685975c209 ("x86/sev: Move SEV compilation units")
Closes: https://lore.kernel.org/oe-lkp/202407091342.46d7dbb-oliver.sang@intel.com
Reported-by: kernel test robot <oliver.sang@intel.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
---
 arch/x86/coco/sev/Makefile | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/arch/x86/coco/sev/Makefile b/arch/x86/coco/sev/Makefile
index 5f72e92b37ff..4e375e7305ac 100644
--- a/arch/x86/coco/sev/Makefile
+++ b/arch/x86/coco/sev/Makefile
@@ -9,3 +9,7 @@ endif
 KASAN_SANITIZE_core.o	:= n
 KMSAN_SANITIZE_core.o	:= n
 KCOV_INSTRUMENT_core.o	:= n
+
+# With some compiler versions the generated code results in boot hangs, caused
+# by several compilation units. To be safe, disable all instrumentation.
+KCSAN_SANITIZE		:= n
-- 
2.43.0

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [tip:x86/sev] [x86/sev] 06685975c2: BUG:kernel_failed_in_early-boot_stage,last_printk:Booting_the_kernel(entry_offset:#)

[Oliver Sang]

hi, Boris,

On Wed, Jul 10, 2024 at 10:28:38AM +0200, Borislav Petkov wrote:
> On Wed, Jul 10, 2024 at 09:47:36AM +0800, Oliver Sang wrote:
> > no. when this bisect done, we tested the branch tip 8434cf006ceae,
> > the issue is still persistent.
> 
> Ok, I think that should fix it:

yes, your patch fixed the issue.

Tested-by: kernel test robot <oliver.sang@intel.com>

> 
> From: "Borislav Petkov (AMD)" <bp@alien8.de>
> Date: Wed, 10 Jul 2024 10:16:18 +0200
> Subject: [PATCH] x86/coco/sev: Disable KCSAN too
> 
> The SEV guest code was moved to x86/coco/ along with the instrumentation
> exclusion bits. However, there is a blanket KCSAN disable in
> arch/x86/kernel/Makefile due to boot hangs.
> 
> Copy that disable to the coco's sev Makefile too.
> 
> Fixes:  Fixes: 06685975c209 ("x86/sev: Move SEV compilation units")
> Closes: https://lore.kernel.org/oe-lkp/202407091342.46d7dbb-oliver.sang@intel.com
> Reported-by: kernel test robot <oliver.sang@intel.com>
> Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
> ---
>  arch/x86/coco/sev/Makefile | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/arch/x86/coco/sev/Makefile b/arch/x86/coco/sev/Makefile
> index 5f72e92b37ff..4e375e7305ac 100644
> --- a/arch/x86/coco/sev/Makefile
> +++ b/arch/x86/coco/sev/Makefile
> @@ -9,3 +9,7 @@ endif
>  KASAN_SANITIZE_core.o	:= n
>  KMSAN_SANITIZE_core.o	:= n
>  KCOV_INSTRUMENT_core.o	:= n
> +
> +# With some compiler versions the generated code results in boot hangs, caused
> +# by several compilation units. To be safe, disable all instrumentation.
> +KCSAN_SANITIZE		:= n
> -- 
> 2.43.0
> 
> -- 
> Regards/Gruss,
>     Boris.
> 
> https://people.kernel.org/tglx/notes-about-netiquette

Re: [tip:x86/sev] [x86/sev] 06685975c2: BUG:kernel_failed_in_early-boot_stage,last_printk:Booting_the_kernel(entry_offset:#)

[Borislav Petkov]

On Thu, Jul 11, 2024 at 09:38:55AM +0800, Oliver Sang wrote:
> yes, your patch fixed the issue.
> 
> Tested-by: kernel test robot <oliver.sang@intel.com>

Thanks, I went and merged the other build issue reported by 0day into the
original patch so that there's no build failures during bisection.

We should be good to go now, knock on wood.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette