Re: [PATCH v8 0/4] target/arm: Add FEAT_MEC to max cpu

[Pierrick Bouvier <pierrick.bouvier@linaro.org>]

On 7/14/25 4:31 PM, Gustavo Romero wrote:
> Hi folks,
> 
> Richard, thanks for v8. Pierrick, thanks for testing it. :)
> 
> On 7/14/25 14:09, Pierrick Bouvier wrote:
>> On 7/14/25 8:58 AM, Richard Henderson wrote:
>>> Changes for v8:
>>>     - Re-order SCTLR2 and TCR2 so that they are independent of MEC.
>>>     - Enable the SCTLR2 and TCR2 enable bits.
>>>     - Squash 3 smaller MEC patches together.
>>>
>>> This still fails the RME tests, because we still need TF-A rebuilt
>>> with ENABLE_FEAT_SCTLR2 and ENABLE_FEAT_TCR2.  Pierrick, since you
>>> have just done such a build, could you re-test with this series?
>>>
>>
>> I tested that on my local Realm enabled setup and I can confirm this solved the issue and current series works.
>> Both flags are needed in TF-A. ENABLE_FEAT_TCR2 is needed to boot host, and ENABLE_FEAT_SCTLR2 is needed to boot nested guest.
> 
> I'm a bit confused because the QEMU RME tests, afaics, uses OP-TEE, not TF-A. I've built TF-A
> using the scripts in [0], enabling ENABLE_FEAT_TCR2 and ENABLE_FEAT_SCTLR2, but no way to get
> it booting. I understand we can embed a OP-TEE into the TF_A via BL32=<optee_image> when
> building TF-A. Is that what you're using?
> 

I agree it's confusing. In short, no, OP-TEE is not used anywhere for 
Realms, only TF-A and RMM are used in our images. It seems that OP-TEE 
is a term used generically to represent any firmware running in secure 
mode, but it's a *totally* different software than TF-A + RMM. Naming 
OP-TEE like this is like if Linux would have been named "OP-kernel".

The RME tests we have are based on this excellent tutorial [0], and 
build is automated with 'qemu-rme-stack' [1], that simply follows those 
instructions.

[0] 
https://linaro.atlassian.net/wiki/spaces/QEMU/pages/30128767027/Device+Assignment+Enabled+RME+Stack+on+QEMU
[1] https://github.com/pbo-linaro/qemu-rme-stack

To add to the confusion (*get ready*), [0] uses OP-TEE build, which is 
simply a build system for op-tee + other things, but in the variant it 
generates, OP-TEE itself is not included. Yes, that's utterly confusing 
and took me quite some time to discover it, after talking with Mathieu 
himself. They removed it explicitely to make the stack more simple. In 
short, for Realms, forget about OP-TEE.

If you want to see it by yourself:

https://git.codelinaro.org/linaro/dcap/op-tee-4.2.0/build/-/blob/cca/v8/qemu_v8_cca.mk?ref_type=heads#L172

...
TF_A_FLAGS ?= \
	BL33=$(BL33_BIN) \
	PLAT=qemu \
	QEMU_USE_GIC_DRIVER=$(TFA_GIC_DRIVER) \
	DEBUG=$(TF_A_DEBUG) \
	LOG_LEVEL=$(TF_A_LOGLVL) \
	ENABLE_RME=1 \
	RMM=$(RMM_BIN)
...
TF_A_FLAGS_BL32_OPTEE  = BL32=$(OPTEE_OS_HEADER_V2_BIN)
TF_A_FLAGS_BL32_OPTEE += BL32_EXTRA1=$(OPTEE_OS_PAGER_V2_BIN)
TF_A_FLAGS_BL32_OPTEE += BL32_EXTRA2=$(OPTEE_OS_PAGEABLE_V2_BIN)
TF_A_FLAGS_SPMC_AT_EL_n  = $(TF_A_FLAGS_BL32_OPTEE) SPD=opteed
...
#TF_A_FLAGS += $(TF_A_FLAGS_SPMC_AT_EL_$(SPMC_AT_EL))

The last line shows that OP_TEE flags are **NOT** added to TF_A_FLAGS 
build flags.
The qemu_v8_cca.mk build file was copied from qemu_v8.mk, which itself 
has the OP-TEE inclusion.

---

Recently, I had to generate a custom rootfs, and I experimented 
generating it directly from docker images, to avoid rebuilding the world 
using Buildroot. Once it worked, I realized it was a good opportunity to 
rebuild the rest of the stack too. The result is 'qemu-linux-stack' [2].

Master branch has only tf-a + uboot (no Realm support), while rme branch 
[3], which supports Realm, uses tf-a + rmm + edk2 instead. I removed 
u-boot as I couldn't get it to boot, and I knew that edk2 worked.

One branch is only one configuration (and it will stay this way).

[2] https://github.com/pbo-linaro/qemu-linux-stack
[3] https://github.com/pbo-linaro/qemu-linux-stack/tree/rme

Beyond the personal knowledge I got through that, I hope it can be used 
for others for who it's confusing about what runs before start_kernel, 
and I guess I'm not the only one who didn't know about that. In the end, 
things are not too complicated, but as most of the build systems out 
there (OP-TEE build, shrinkwrap, ...) try to be "generic and versatile", 
it ends up being complicated. I prefer basic and straightforward script 
shells to lenghty documentation and wiki pages, but it's a personal choice.

---

Now, coming to the change introduced by this series, and supporting 
FEAT_SCTRL2 and FEAT_TCR2, all those images need to be updated [0], [1], 
[2], [3], because essentially, TF-A itself must be patched to support 
this. I was about to mention that, and mention that I can send a PR 
directly to it once we have this merged on QEMU side.

The change in arm-trusted-firmware is quite simple:

+diff --git a/plat/qemu/common/common.mk b/plat/qemu/common/common.mk
+index 751511cf8..6bc108492 100644
+--- a/plat/qemu/common/common.mk
++++ b/plat/qemu/common/common.mk
+@@ -122,6 +122,10 @@ ENABLE_FEAT_FGT         :=      2
+ # 8.7
+ ENABLE_FEAT_HCX             :=      2
+
++# 8.8
++ENABLE_FEAT_TCR2    :=      2
++ENABLE_FEAT_SCTLR2  :=      2
++

I'll push all that tomorrow on master, and rebase rmm and 
device_passthrough on top of it.
I'll add this to original RME images, it's less direct though, as it's 
needed to update the tf-a fork, and then the optee build system.

Sorry for this very long email. At least, all is there.

> Thanks.
> 
> 
> Cheers,
> Gustavo
> 
> [0] https://github.com/pbo-linaro/qemu-linux-stack.git
> 
>> As I'm off today, I'll update that properly tomorrow when I have time, and not rush things. I'll update RME images for sbsa and virt tests + device passthrough test, and post associated patches.
>>
>> Thanks,
>> Pierrick
>