Re: [kvm-unit-tests PATCH 1/2] arm64: set sctlr_el1.SPAN

[Nikos Nikoleris <nikos.nikoleris@arm.com>]

On 17/07/2023 07:52, Andrew Jones wrote:
> On Mon, Jul 17, 2023 at 08:50:30AM +0200, Andrew Jones wrote:
>> On Fri, Jul 14, 2023 at 06:42:25PM +0000, Nadav Amit wrote:
>>>
>>>
>>>> On Jul 14, 2023, at 4:29 AM, Shaoqin Huang <shahuang@redhat.com> wrote:
>>>>
>>>> !! External Email
>>>>
>>>> Hi,
>>>>
>>>> On 7/14/23 18:31, Alexandru Elisei wrote:
>>>>> Hi,
>>>>>
>>>>> On Sat, Jun 17, 2023 at 01:31:37AM +0000, Nadav Amit wrote:
>>>>>> From: Nadav Amit <namit@vmware.com>
>>>>>>
>>>>>> Do not assume PAN is not supported or that sctlr_el1.SPAN is already set.
>>>>>
>>>>> In arm/cstart64.S
>>>>>
>>>>> .globl start
>>>>> start:
>>>>>          /* get our base address */
>>>>>       [..]
>>>>>
>>>>> 1:
>>>>>          /* zero BSS */
>>>>>       [..]
>>>>>
>>>>>          /* zero and set up stack */
>>>>>       [..]
>>>>>
>>>>>          /* set SCTLR_EL1 to a known value */
>>>>>          ldr     x4, =INIT_SCTLR_EL1_MMU_OFF
>>>>>       [..]
>>>>>
>>>>>          /* set up exception handling */
>>>>>          bl      exceptions_init
>>>>>       [..]
>>>>>
>>>>> Where in lib/arm64/asm/sysreg.h:
>>>>>
>>>>> #define SCTLR_EL1_RES1  (_BITUL(7) | _BITUL(8) | _BITUL(11) | _BITUL(20) | \
>>>>>                           _BITUL(22) | _BITUL(23) | _BITUL(28) | _BITUL(29))
>>>>> #define INIT_SCTLR_EL1_MMU_OFF  \
>>>>>                          SCTLR_EL1_RES1
>>>>>
>>>>> Look like bit 23 (SPAN) should be set.
>>>>>
>>>>> How are you seeing SCTLR_EL1.SPAN unset?
>>>>
>>>> Yeah. the sctlr_el1.SPAN has always been set by the above flow. So Nadav
>>>> you can describe what you encounter with more details. Like which tests
>>>> crash you encounter, and how to reproduce it.
>>>
>>> I am using Nikos’s work to run the test using EFI, not from QEMU.
>>>
>>> So the code that you mentioned - which is supposed to initialize SCTLR -
>>> is not executed (and actually not part of the EFI image).
>>>
>>> Note that using EFI, the entry point is _start [1], and not “start”.
>>>
>>> That is also the reason lack of BSS zeroing also caused me issues with the
>>> EFI setup, which I reported before.
>>
>> Nadav,
>>
>> Would you mind reposting this along with the BSS zeroing patch, the
>> way I proposed we do that, and anything else you've discovered when
>> trying to use the EFI unit tests without QEMU? We'll call that our
>> first non-QEMU EFI support series, since the first EFI series was
>> only targeting QEMU.
> 
> Oh, and I meant to mention that, when reposting this patch, maybe we
> can consider managing sctlr in a similar way to the non-efi start path?
> 

Nadav, if you are running baremetal, it might be worth checking what EL 
you're running in as well. If HW is implementing EL2, EFI will handover 
in EL2.

I was planning to rebase an old patch (more like rewrite it) but I 
haven't found the time yet [1]. If I remember correctly, we have to 
check what EL we're running in and if it's EL2 we have to add a stub 
EL2, drop to EL1 and setup EL1. But things have change since that patch 
and with the new structure, I am not sure if we would drop to EL1 right 
at the start (crt0-efi-aarch64.S) or somewhere in setup_efi().

In general, I think, it would be easier to deal with this in QEMU 
(-machine secure=on) and before we even start thinking about real 
hardware where it is very likely that we will have to address other 
issues (such as the problem with the BSS, cache maintenance) as well.

[1]: 
https://github.com/relokin/kvm-unit-tests/commit/1468abeee7be1d85140ed92cb91a42ee27a9bf1f

Thanks,

Nikos