sev_es_trampoline_start undefined symbol referenced errors during kunit run

sev_es_trampoline_start undefined symbol referenced errors during kunit run

[Shuah Khan]

Hi Tom,

I have been seeing sev_es_trampoline_start undefined symbol referenced errors
during the following kunit test runs.

./tools/testing/kunit/kunit.py run --arch x86_64
./tools/testing/kunit/kunit.py run --alltests --arch x86_64

The error is here:

ERROR:root:ld:arch/x86/realmode/rm/realmode.lds:236: undefined symbol `sev_es_trampoline_start' referenced in expression
make[6]: *** [../arch/x86/realmode/rm/Makefile:49: arch/x86/realmode/rm/realmode.elf] Error 1
make[5]: *** [../arch/x86/realmode/Makefile:22: arch/x86/realmode/rm/realmode.bin] Error 2
make[4]: *** [../scripts/Makefile.build:461: arch/x86/realmode] Error 2

I made time to look into this error.

sev_es_trampoline_start is referenced in arch/x86/coco/sev/core.c twice:

- To override start_ip in  wakeup_cpu_via_vmgexit()
- In sev_es_setup_ap_jump_table() to compute startup_ip

sev_es_trampoline_start is defined if CONFIG_AMD_MEM_ENCRYPT is enabled
and all other references to it are under ifdef CONFIG_AMD_MEM_ENCRYPT
conditional except the two in arch/x86/coco/sev/core.c

git grep sev_es_trampoline_start
arch/x86/coco/sev/core.c:       start_ip = real_mode_header->sev_es_trampoline_start;
arch/x86/coco/sev/core.c:       startup_ip = (u16)(rmh->sev_es_trampoline_start -
arch/x86/include/asm/realmode.h:        u32     sev_es_trampoline_start;
arch/x86/realmode/rm/header.S:  .long   pa_sev_es_trampoline_start
arch/x86/realmode/rm/trampoline_64.S:SYM_CODE_START(sev_es_trampoline_start)
arch/x86/realmode/rm/trampoline_64.S:SYM_CODE_END(sev_es_trampoline_start)

Why are these references not under ifdef CONFIG_AMD_MEM_ENCRYPT conditional?

The following commits added the references to sev_es_trampoline_start
in arch/x86/coco/sev/core.c

0afb6b660a6b58cb336d1175ed687bf9525849a4
8940ac9ced8bc1c48c4e28b0784e3234c9d14469

thanks,
-- Shuah

Re: sev_es_trampoline_start undefined symbol referenced errors during kunit run

[Borislav Petkov]

On Mon, Apr 14, 2025 at 04:28:44PM -0600, Shuah Khan wrote:
> Hi Tom,
> 
> I have been seeing sev_es_trampoline_start undefined symbol referenced errors
> during the following kunit test runs.
> 
> ./tools/testing/kunit/kunit.py run --arch x86_64
> ./tools/testing/kunit/kunit.py run --alltests --arch x86_64
> 
> The error is here:
> 
> ERROR:root:ld:arch/x86/realmode/rm/realmode.lds:236: undefined symbol `sev_es_trampoline_start' referenced in expression
> make[6]: *** [../arch/x86/realmode/rm/Makefile:49: arch/x86/realmode/rm/realmode.elf] Error 1
> make[5]: *** [../arch/x86/realmode/Makefile:22: arch/x86/realmode/rm/realmode.bin] Error 2
> make[4]: *** [../scripts/Makefile.build:461: arch/x86/realmode] Error 2
> 
> I made time to look into this error.
> 
> sev_es_trampoline_start is referenced in arch/x86/coco/sev/core.c twice:
> 
> - To override start_ip in  wakeup_cpu_via_vmgexit()
> - In sev_es_setup_ap_jump_table() to compute startup_ip
> 
> sev_es_trampoline_start is defined if CONFIG_AMD_MEM_ENCRYPT is enabled
> and all other references to it are under ifdef CONFIG_AMD_MEM_ENCRYPT
> conditional except the two in arch/x86/coco/sev/core.c
> 
> git grep sev_es_trampoline_start
> arch/x86/coco/sev/core.c:       start_ip = real_mode_header->sev_es_trampoline_start;
> arch/x86/coco/sev/core.c:       startup_ip = (u16)(rmh->sev_es_trampoline_start -
> arch/x86/include/asm/realmode.h:        u32     sev_es_trampoline_start;
> arch/x86/realmode/rm/header.S:  .long   pa_sev_es_trampoline_start
> arch/x86/realmode/rm/trampoline_64.S:SYM_CODE_START(sev_es_trampoline_start)
> arch/x86/realmode/rm/trampoline_64.S:SYM_CODE_END(sev_es_trampoline_start)
> 
> Why are these references not under ifdef CONFIG_AMD_MEM_ENCRYPT conditional?

obj-$(CONFIG_AMD_MEM_ENCRYPT)   += sev/

in arch/x86/coco/Makefile

The real problem looks like that pasyms.h thing which gets included at the end
of realmode.lds and which contains that symbol.

How exactly can this be reproduced? Exact steps please.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: sev_es_trampoline_start undefined symbol referenced errors during kunit run

[David Gow]

[-- Attachment #1: Type: text/plain, Size: 1116 bytes --]

On Tue, 15 Apr 2025 at 07:01, Borislav Petkov <bp@alien8.de> wrote:
>
> On Mon, Apr 14, 2025 at 04:28:44PM -0600, Shuah Khan wrote:

... snip ...

>
> The real problem looks like that pasyms.h thing which gets included at the end
> of realmode.lds and which contains that symbol.

I had a look at this a few months ago, and came to the same
conclusion: that realmode.lds is where the issue is coming from. My
suspicion was that realmode.lds was being generated when the kernel
was built in-tree (with CONFIG_AMD_MEM_ENCRYPT), but somehow was not
being deleted when the source tree was being cleaned before an
out-of-tree build was being done (without CONFIG_AMD_MEM_ENCRYPT).

But, alas, I was never able to find a way to reproduce this to test it
out: realmode.lds seems to be properly deleted by make clean / make
mrproper here.

(My personal strategy is to always do out-of-tree builds, which may be
why I don't tend to randomly come across this. Equally, it may be
worth trying `--build_dir .` as an argument to kunit.py to force an
in-tree build if you're already doing in-tree builds often.)

Cheers,
-- David

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5281 bytes --]

Re: sev_es_trampoline_start undefined symbol referenced errors during kunit run

[Shuah Khan]

On 4/14/25 17:00, Borislav Petkov wrote:
> On Mon, Apr 14, 2025 at 04:28:44PM -0600, Shuah Khan wrote:
>> Hi Tom,
>>
>> I have been seeing sev_es_trampoline_start undefined symbol referenced errors
>> during the following kunit test runs.
>>
>> ./tools/testing/kunit/kunit.py run --arch x86_64
>> ./tools/testing/kunit/kunit.py run --alltests --arch x86_64
>>
>> The error is here:
>>
>> ERROR:root:ld:arch/x86/realmode/rm/realmode.lds:236: undefined symbol `sev_es_trampoline_start' referenced in expression
>> make[6]: *** [../arch/x86/realmode/rm/Makefile:49: arch/x86/realmode/rm/realmode.elf] Error 1
>> make[5]: *** [../arch/x86/realmode/Makefile:22: arch/x86/realmode/rm/realmode.bin] Error 2
>> make[4]: *** [../scripts/Makefile.build:461: arch/x86/realmode] Error 2
>>
>
> 
> The real problem looks like that pasyms.h thing which gets included at the end
> of realmode.lds and which contains that symbol.
> 
> How exactly can this be reproduced? Exact steps please.

Run these kunit tests - not out of tree. Tree has to be
clean to run these tests. Otherwise you are prompted to
run mrproper.

./tools/testing/kunit/kunit.py run --arch x86_64
or
./tools/testing/kunit/kunit.py run --alltests --arch x86_64

The tree I see this every single time I do my tree testing.

thanks,
-- Shuah

Re: sev_es_trampoline_start undefined symbol referenced errors during kunit run

[Borislav Petkov]

On Tue, Apr 15, 2025 at 08:25:09AM -0600, Shuah Khan wrote:
> ./tools/testing/kunit/kunit.py run --arch x86_64
> or
> ./tools/testing/kunit/kunit.py run --alltests --arch x86_64
> 
> The tree I see this every single time I do my tree testing.

Doesn't reproduce here:

# ./tools/testing/kunit/kunit.py run --arch x86_64 > log.00 2>&1
# ./tools/testing/kunit/kunit.py run --alltests --arch x86_64 > log.01 2>&1
# grep -i error log.*
log.00:[19:04:52] [PASSED] error_pointer
log.01:[19:44:06] [PASSED] error_pointer

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: sev_es_trampoline_start undefined symbol referenced errors during kunit run

[Shuah Khan]

On 4/15/25 12:01, Borislav Petkov wrote:
> On Tue, Apr 15, 2025 at 08:25:09AM -0600, Shuah Khan wrote:
>> ./tools/testing/kunit/kunit.py run --arch x86_64
>> or
>> ./tools/testing/kunit/kunit.py run --alltests --arch x86_64
>>
>> The tree I see this every single time I do my tree testing.
> 
> Doesn't reproduce here:
> 
> # ./tools/testing/kunit/kunit.py run --arch x86_64 > log.00 2>&1
> # ./tools/testing/kunit/kunit.py run --alltests --arch x86_64 > log.01 2>&1
> # grep -i error log.*
> log.00:[19:04:52] [PASSED] error_pointer
> log.01:[19:44:06] [PASSED] error_pointer
> 

Does your arch/x86/realmode/rm/pasyms.h has reference to sev_es_trampoline_start?

The one in my tree has it.

arch/x86/realmode/rm/pasyms.h:pa_sev_es_trampoline_start = sev_es_trampoline_start

thanks,
-- Shuah

Re: sev_es_trampoline_start undefined symbol referenced errors during kunit run

[Borislav Petkov]

On Tue, Apr 15, 2025 at 01:06:49PM -0600, Shuah Khan wrote:
> Does your arch/x86/realmode/rm/pasyms.h has reference to sev_es_trampoline_start?
> 
> The one in my tree has it.
> 
> arch/x86/realmode/rm/pasyms.h:pa_sev_es_trampoline_start = sev_es_trampoline_start


# ./tools/testing/kunit/kunit.py run --arch x86_64
...

[00:15:36] Elapsed time: 58.840s total, 2.096s configuring, 53.170s building, 3.487s running

# cat arch/x86/realmode/rm/pasyms.h
cat: arch/x86/realmode/rm/pasyms.h: No such file or directory

Could explain why I don't see the issue...

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: sev_es_trampoline_start undefined symbol referenced errors during kunit run

[Shuah Khan]

On 4/15/25 16:17, Borislav Petkov wrote:
> On Tue, Apr 15, 2025 at 01:06:49PM -0600, Shuah Khan wrote:
>> Does your arch/x86/realmode/rm/pasyms.h has reference to sev_es_trampoline_start?
>>
>> The one in my tree has it.
>>
>> arch/x86/realmode/rm/pasyms.h:pa_sev_es_trampoline_start = sev_es_trampoline_start
> 
> 
> # ./tools/testing/kunit/kunit.py run --arch x86_64
> ...
> 
> [00:15:36] Elapsed time: 58.840s total, 2.096s configuring, 53.170s building, 3.487s running
> 
> # cat arch/x86/realmode/rm/pasyms.h
> cat: arch/x86/realmode/rm/pasyms.h: No such file or directory
> 
> Could explain why I don't see the issue...
> 

I see arch/x86/realmode/rm/pasyms.h on my system. It is a generated
file from arch/x86/realmode/rm Makefile

Here is the target information.

REALMODE_OBJS = $(addprefix $(obj)/,$(realmode-y))

sed-pasyms := -n -r -e 's/^([0-9a-fA-F]+) [ABCDGRSTVW] (.+)$$/pa_\2 = \2;/p'

quiet_cmd_pasyms = PASYMS  $@
       cmd_pasyms = $(NM) $(real-prereqs) | sed $(sed-pasyms) | sort | uniq > $@

targets += pasyms.h
$(obj)/pasyms.h: $(REALMODE_OBJS) FORCE
         $(call if_changed,pasyms)

The key is how and why this file gets generated and why the reference

pa_sev_es_trampoline_start = sev_es_trampoline_start

is added unconditionally even when  CONFIG_AMD_MEM_ENCRYPT is not
enabled. I think the logic should be fixed to take AMD_MEM_ENCRYPT
enabled or disabled into account when this pasyms.h file is generated.

kunit test run starts with no config file and generates its own, so it
isn't coming from any existing config.

thanks,
-- Shuah

Re: sev_es_trampoline_start undefined symbol referenced errors during kunit run

[David Gow]

[-- Attachment #1: Type: text/plain, Size: 2327 bytes --]

On Wed, 16 Apr 2025 at 06:30, Shuah Khan <skhan@linuxfoundation.org> wrote:
>
> On 4/15/25 16:17, Borislav Petkov wrote:
> > On Tue, Apr 15, 2025 at 01:06:49PM -0600, Shuah Khan wrote:
> >> Does your arch/x86/realmode/rm/pasyms.h has reference to sev_es_trampoline_start?
> >>
> >> The one in my tree has it.
> >>
> >> arch/x86/realmode/rm/pasyms.h:pa_sev_es_trampoline_start = sev_es_trampoline_start
> >
> >
> > # ./tools/testing/kunit/kunit.py run --arch x86_64
> > ...
> >
> > [00:15:36] Elapsed time: 58.840s total, 2.096s configuring, 53.170s building, 3.487s running
> >
> > # cat arch/x86/realmode/rm/pasyms.h
> > cat: arch/x86/realmode/rm/pasyms.h: No such file or directory
> >
> > Could explain why I don't see the issue...
> >
>
> I see arch/x86/realmode/rm/pasyms.h on my system. It is a generated
> file from arch/x86/realmode/rm Makefile
>

Aha: I've been able to reproduce this, albeit by not cleaning the tree properly.

- make ARCH=x86_64 # Build an x86_64 kernel in-tree, with
CONFIG_AMD_MEM_ENCRYPT=y
- ./tools/testing/kunit/kunit.py run # Attempt to build and run KUnit
on a UML kernel (built out-of-tree in the .kunit directory)
# This will fail, telling you to clean the tree with 'make ARCH=um mrproper'
- make ARCH=um mrproper # Clean the source tree, but incompletely, as
the original kernel was built with ARCH=x86_64, not ARCH=um
# As a result, the pasyms.h file will be left in the tree, as it's not
part of the UML build
- ./tools/testing/kunit/kunit.py run --arch x86_64 # Attempt to
build/run an out-of-tree x86_64 kernel.
# This will not tell you to clean the source tree, as it was
(incorrectly) cleaned for the wrong architecture, but will fail due to
the wrong pasyms.h still being present.

I'm not sure if this is the same cause as what you're seeing, Shuah,
but it seems plausible enough. If so, this is really an issue with the
Makefiles suggesting the wrong make mrproper command (assuming that
the architecture hasn't changed), or failing to detect that the source
tree still isn't clean. Maybe that's something we could work around in
either the arch/um makefiles or in kunit.py (or at least the
documentation), if we don't want to rework how dirty trees are
detected.

Either way, this should work after running `make ARCH=x86_64
mrproper`. Does that work for you?

Cheers,
-- David

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5281 bytes --]

Re: sev_es_trampoline_start undefined symbol referenced errors during kunit run

[Shuah Khan]

On 4/15/25 18:32, David Gow wrote:
> On Wed, 16 Apr 2025 at 06:30, Shuah Khan <skhan@linuxfoundation.org> wrote:
>>
>> On 4/15/25 16:17, Borislav Petkov wrote:
>>> On Tue, Apr 15, 2025 at 01:06:49PM -0600, Shuah Khan wrote:
>>>> Does your arch/x86/realmode/rm/pasyms.h has reference to sev_es_trampoline_start?
>>>>
>>>> The one in my tree has it.
>>>>
>>>> arch/x86/realmode/rm/pasyms.h:pa_sev_es_trampoline_start = sev_es_trampoline_start
>>>
>>>
>>> # ./tools/testing/kunit/kunit.py run --arch x86_64
>>> ...
>>>
>>> [00:15:36] Elapsed time: 58.840s total, 2.096s configuring, 53.170s building, 3.487s running
>>>
>>> # cat arch/x86/realmode/rm/pasyms.h
>>> cat: arch/x86/realmode/rm/pasyms.h: No such file or directory
>>>
>>> Could explain why I don't see the issue...
>>>
>>
>> I see arch/x86/realmode/rm/pasyms.h on my system. It is a generated
>> file from arch/x86/realmode/rm Makefile
>>
> 
> Aha: I've been able to reproduce this, albeit by not cleaning the tree properly.
> 
> - make ARCH=x86_64 # Build an x86_64 kernel in-tree, with
> CONFIG_AMD_MEM_ENCRYPT=y
> - ./tools/testing/kunit/kunit.py run # Attempt to build and run KUnit
> on a UML kernel (built out-of-tree in the .kunit directory)
> # This will fail, telling you to clean the tree with 'make ARCH=um mrproper'
> - make ARCH=um mrproper # Clean the source tree, but incompletely, as
> the original kernel was built with ARCH=x86_64, not ARCH=um
> # As a result, the pasyms.h file will be left in the tree, as it's not
> part of the UML build
> - ./tools/testing/kunit/kunit.py run --arch x86_64 # Attempt to
> build/run an out-of-tree x86_64 kernel.
> # This will not tell you to clean the source tree, as it was
> (incorrectly) cleaned for the wrong architecture, but will fail due to
> the wrong pasyms.h still being present.
> 
> I'm not sure if this is the same cause as what you're seeing, Shuah,
> but it seems plausible enough. If so, this is really an issue with the
> Makefiles suggesting the wrong make mrproper command (assuming that
> the architecture hasn't changed), or failing to detect that the source
> tree still isn't clean. Maybe that's something we could work around in
> either the arch/um makefiles or in kunit.py (or at least the
> documentation), if we don't want to rework how dirty trees are
> detected.
> 

Yes I am seeing the same thing. However, it doesn't even ask you
to run mrproper because it doesn't seem to think the tree is dirty.

So the fix would be to detect that the tree is dirty?

I just tried make ARCH=x86_64 mrproper and then ran
kunit.py run --arch x86_64


> Either way, this should work after running `make ARCH=x86_64
> mrproper`. Does that work for you?

I just tried make ARCH=x86_64 mrproper and then ran
kunit.py run --arch x86_64

thanks,
-- Shuah

Re: sev_es_trampoline_start undefined symbol referenced errors during kunit run

[Borislav Petkov]

On Wed, Apr 16, 2025 at 08:32:08AM +0800, David Gow wrote:
> - make ARCH=um mrproper # Clean the source tree, but incompletely, as
> the original kernel was built with ARCH=x86_64, not ARCH=um
> # As a result, the pasyms.h file will be left in the tree, as it's not
> part of the UML build

Yeah, or you do:

$ git clean -dqfx

and the thing is gone. :-P

> - ./tools/testing/kunit/kunit.py run --arch x86_64 # Attempt to
> build/run an out-of-tree x86_64 kernel.
> # This will not tell you to clean the source tree, as it was
> (incorrectly) cleaned for the wrong architecture, but will fail due to
> the wrong pasyms.h still being present.
> 
> I'm not sure if this is the same cause as what you're seeing, Shuah,
> but it seems plausible enough. If so, this is really an issue with the
> Makefiles suggesting the wrong make mrproper command (assuming that
> the architecture hasn't changed), or failing to detect that the source
> tree still isn't clean. Maybe that's something we could work around in
> either the arch/um makefiles or in kunit.py (or at least the
> documentation), if we don't want to rework how dirty trees are
> detected.

ARCH=um has always been weird in that regard and we have had "bugs" like that in
the past.

I guess the simplest thing to do is to whack ARCH=um too when you clean, in
addition to x86_64. Or use the above git command if you're in a git repo...

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: sev_es_trampoline_start undefined symbol referenced errors during kunit run

[Shuah Khan]

On 4/15/25 16:29, Shuah Khan wrote:
> On 4/15/25 16:17, Borislav Petkov wrote:
>> On Tue, Apr 15, 2025 at 01:06:49PM -0600, Shuah Khan wrote:
>>> Does your arch/x86/realmode/rm/pasyms.h has reference to sev_es_trampoline_start?
>>>
>>> The one in my tree has it.
>>>
>>> arch/x86/realmode/rm/pasyms.h:pa_sev_es_trampoline_start = sev_es_trampoline_start
>>
>>
>> # ./tools/testing/kunit/kunit.py run --arch x86_64
>> ...
>>
>> [00:15:36] Elapsed time: 58.840s total, 2.096s configuring, 53.170s building, 3.487s running
>>
>> # cat arch/x86/realmode/rm/pasyms.h
>> cat: arch/x86/realmode/rm/pasyms.h: No such file or directory
>>
>> Could explain why I don't see the issue...
>>
> 
> I see arch/x86/realmode/rm/pasyms.h on my system. It is a generated
> file from arch/x86/realmode/rm Makefile
> 
> Here is the target information.
> 
> REALMODE_OBJS = $(addprefix $(obj)/,$(realmode-y))
> 
> sed-pasyms := -n -r -e 's/^([0-9a-fA-F]+) [ABCDGRSTVW] (.+)$$/pa_\2 = \2;/p'
> 
> quiet_cmd_pasyms = PASYMS  $@
>        cmd_pasyms = $(NM) $(real-prereqs) | sed $(sed-pasyms) | sort | uniq > $@
> 
> targets += pasyms.h
> $(obj)/pasyms.h: $(REALMODE_OBJS) FORCE
>          $(call if_changed,pasyms)
> 
> The key is how and why this file gets generated and why the reference
> 
> pa_sev_es_trampoline_start = sev_es_trampoline_start
> 
> is added unconditionally even when  CONFIG_AMD_MEM_ENCRYPT is not
> enabled. I think the logic should be fixed to take AMD_MEM_ENCRYPT
> enabled or disabled into account when this pasyms.h file is generated.
> 

Okay - I think the logic to extract symbols to include in pasyms.h
will have to take AMD_MEM_ENCRYPT enabled vs. disabled into account.

running git grep sev_es_trampoline_start in arch/x86/realmode/rm shows:

header.S:       .long   pa_sev_es_trampoline_start
trampoline_64.S:SYM_CODE_START(sev_es_trampoline_start)
trampoline_64.S:SYM_CODE_END(sev_es_trampoline_start)

All of the above are under ifdef AMD_MEM_ENCRYPT conditional.
Even if pasyms.h is generated sev_es_trampoline_start should not
be included in pasyms.h if AMD_MEM_ENCRYPT is disabled.

thanks,
-- Shuah