[U-Boot] imx7d: CPU core issue in secure mode

[U-Boot] imx7d: CPU core issue in secure mode

[Tobias Junghans]

Hi,

I'm trying to get an imx7d-based Colibris board running in secure mode in 
order to be able to use the CAAM, especially the HWRNG. However it seems like 
it's currently not possible to boot a mainline kernel (4.19) in secure mode 
with both CPU cores powered up, likely due to the missing PSCI firmware in 
secure mode. When booting in nonsecure mode the kernel recognizes both CPU 
cores while CAAM isn't working. Basically it's the same issue as discussed at

https://www.spinics.net/lists/u-boot-v2/msg33873.html

I'm using the latest mainline U-Boot (2019.07-rc4) with 
CONFIG_ARMV7_BOOT_SEC_DEFAULT=y. Is there anything I can do about this issue?

Thank you and best regards

Tobias

[U-Boot] imx7d: CPU core issue in secure mode

[Philippe Schenker]

+ Stefan Agner

On Thu, 2019-07-04 at 10:13 +0200, Tobias Junghans wrote:
> Hi,
> 
> I'm trying to get an imx7d-based Colibris board running in secure mode in 
> order to be able to use the CAAM, especially the HWRNG. However it seems like 
> it's currently not possible to boot a mainline kernel (4.19) in secure mode 
> with both CPU cores powered up, likely due to the missing PSCI firmware in 
> secure mode. When booting in nonsecure mode the kernel recognizes both CPU 
> cores while CAAM isn't working. Basically it's the same issue as discussed at
> 
> https://www.spinics.net/lists/u-boot-v2/msg33873.html
> 
> I'm using the latest mainline U-Boot (2019.07-rc4) with 
> CONFIG_ARMV7_BOOT_SEC_DEFAULT=y. Is there anything I can do about this issue?
> 
> Thank you and best regards
> 
> Tobias
> 
> 
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> https://lists.denx.de/listinfo/u-boot

[U-Boot] imx7d: CPU core issue in secure mode

[Igor Opaniuk]

+ Peng

Hi Tobias, Peng,

On Thu, Jul 4, 2019 at 2:20 PM Tobias Junghans <tobias.junghans@veyon.io> wrote:
>
> Hi,
>
> I'm trying to get an imx7d-based Colibris board running in secure mode in
> order to be able to use the CAAM, especially the HWRNG. However it seems like
> it's currently not possible to boot a mainline kernel (4.19) in secure mode
> with both CPU cores powered up, likely due to the missing PSCI firmware in
> secure mode. When booting in nonsecure mode the kernel recognizes both CPU
> cores while CAAM isn't working. Basically it's the same issue as discussed at
>
> https://www.spinics.net/lists/u-boot-v2/msg33873.html
>
> I'm using the latest mainline U-Boot (2019.07-rc4) with
> CONFIG_ARMV7_BOOT_SEC_DEFAULT=y. Is there anything I can do about this issue?
>
> Thank you and best regards
>
> Tobias
>
>
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> https://lists.denx.de/listinfo/u-boot

I might be mistaken, but AFAIK there was on-going work done by Peng
Fan regarding
proper CAAM initialization in the OP-TEE and further usage in the
mainline kernel.

As I understood, the initial initialization of the jobrings is done in OP-TEE
(which is booted before U-boot) in secure world, and then linux kernel,
running in normal world, should be able to use it.
Regarding PSCI, frankly, I have no idea who particularly should provide it's
support here: U-boot or OP-TEE (taking into account that in this setup
U-boot is booted in non-secure PL2, so OP-TEE is the only one, who is
able to provide secure
runtime services, so-called secure monitor).

BTW, I also saw some setups, where similar things to do the same in
U-boot (when it's booted in secure mode),
which also does have it's own implementation of secure
monitor(subsequently PSCI) and CAAM driver,
which probably does the same type of initialization, as in OP-TEE.


Peng,
Could you please provide some comments regarding this? Thanks!

-- 
Best regards - Freundliche Grüsse - Meilleures salutations

Igor Opaniuk

mailto: igor.opaniuk at gmail.com
skype: igor.opanyuk
+380 (93) 836 40 67
http://ua.linkedin.com/in/iopaniuk

[U-Boot] imx7d: CPU core issue in secure mode

[Peng Fan]

> Subject: Re: [U-Boot] imx7d: CPU core issue in secure mode
> 
> + Peng
> 
> Hi Tobias, Peng,
> 
> On Thu, Jul 4, 2019 at 2:20 PM Tobias Junghans <tobias.junghans@veyon.io>
> wrote:
> >
> > Hi,
> >
> > I'm trying to get an imx7d-based Colibris board running in secure mode
> > in order to be able to use the CAAM, especially the HWRNG. However it
> > seems like it's currently not possible to boot a mainline kernel
> > (4.19) in secure mode with both CPU cores powered up, likely due to
> > the missing PSCI firmware in secure mode. When booting in nonsecure
> > mode the kernel recognizes both CPU cores while CAAM isn't working.
> > Basically it's the same issue as discussed at
> >
> >
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> >
> spinics.net%2Flists%2Fu-boot-v2%2Fmsg33873.html&amp;data=02%7C01%7
> Cpen
> >
> g.fan%40nxp.com%7C69f453b8841a47775d7608d705edd3ee%7C686ea1d3b
> c2b4c6fa
> >
> 92cd99c5c301635%7C0%7C0%7C636984391331231662&amp;sdata=MtD5x
> 15k3vvgBMr
> > vqBaZBY9G8AFD0WuE9J8XxIP%2Fz%2Bk%3D&amp;reserved=0
> >
> > I'm using the latest mainline U-Boot (2019.07-rc4) with
> > CONFIG_ARMV7_BOOT_SEC_DEFAULT=y. Is there anything I can do about
> this issue?


Try "setenv bootm_boot_mode nonsec" in U-Boot stage.

> >
> > Thank you and best regards
> >
> > Tobias
> >
> >
> > _______________________________________________
> > U-Boot mailing list
> > U-Boot at lists.denx.de
> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
> >
> s.denx.de%2Flistinfo%2Fu-boot&amp;data=02%7C01%7Cpeng.fan%40nxp.co
> m%7C
> >
> 69f453b8841a47775d7608d705edd3ee%7C686ea1d3bc2b4c6fa92cd99c5c30
> 1635%7C
> >
> 0%7C0%7C636984391331231662&amp;sdata=Ra4mzQpiZpANam1gyhhsy2g
> WMHNH3JRNr
> > ryP%2BPOiqsM%3D&amp;reserved=0
> 
> I might be mistaken, but AFAIK there was on-going work done by Peng Fan
> regarding proper CAAM initialization in the OP-TEE and further usage in the
> mainline kernel.

Silvano was doing the CAAM part in OP-TEE.

> 
> As I understood, the initial initialization of the jobrings is done in OP-TEE
> (which is booted before U-boot) in secure world, and then linux kernel,
> running in normal world, should be able to use it.
> Regarding PSCI, frankly, I have no idea who particularly should provide it's
> support here: U-boot or OP-TEE (taking into account that in this setup U-boot
> is booted in non-secure PL2, so OP-TEE is the only one, who is able to provide
> secure runtime services, so-called secure monitor).
> 
> BTW, I also saw some setups, where similar things to do the same in U-boot
> (when it's booted in secure mode), which also does have it's own
> implementation of secure monitor(subsequently PSCI) and CAAM driver,
> which probably does the same type of initialization, as in OP-TEE.
> 
> 
> Peng,
> Could you please provide some comments regarding this? Thanks!


There is psci services in U-Boot too. If want non-secure kernel without OP-TEE,
Need set "setenv bootm_boot_mode nonsec " in U-Boot stage.
If want run OP-TEE, not set the env.

Regards,
Peng.

> 
> --
> Best regards - Freundliche Grüsse - Meilleures salutations
> 
> Igor Opaniuk
> 
> mailto: igor.opaniuk at gmail.com
> skype: igor.opanyuk
> +380 (93) 836 40 67
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fua.linke
> din.com%2Fin%2Fiopaniuk&amp;data=02%7C01%7Cpeng.fan%40nxp.com%7
> C69f453b8841a47775d7608d705edd3ee%7C686ea1d3bc2b4c6fa92cd99c5c3
> 01635%7C0%7C0%7C636984391331231662&amp;sdata=%2B8TlRt9QP6mV
> wMhc3TtHxaZdM%2FvSx09Jz%2BpFhJOlgvg%3D&amp;reserved=0

[U-Boot] imx7d: CPU core issue in secure mode

[Tobias Junghans]

Hi Peng,

Am Freitag, 12. Juli 2019, 05:38:21 CEST schrieb Peng Fan:
> Try "setenv bootm_boot_mode nonsec" in U-Boot stage.

Unfortunately this does not help. I tried the following setups:

CONFIG_SECURE_BOOT=y
CONFIG_CPU_V7_HAS_NONSEC=y
CONFIG_CPU_V7_HAS_VIRT=y
CONFIG_ARCH_SUPPORT_PSCI=y
CONFIG_ARMV7_NONSEC=y
CONFIG_ARMV7_BOOT_SEC_DEFAULT=y
CONFIG_ARMV7_VIRT=y
CONFIG_ARMV7_PSCI=y
CONFIG_ARMV7_PSCI_NR_CPUS=2
CONFIG_FSL_CAAM=y
CONFIG_SYS_FSL_HAS_SEC=y
CONFIG_SYS_FSL_SEC_COMPAT_4=y
# CONFIG_SYS_FSL_SEC_BE is not set
CONFIG_SYS_FSL_SEC_COMPAT=4
CONFIG_SYS_FSL_SEC_LE=y


Booting with bootm_boot_mode=nonsec


U-Boot 2019.07 (Jul 12 2019 - 10:02:31 +0200)
CPU:   Freescale i.MX7D rev1.3 1000 MHz (running at 792 MHz)
..
SEC0: RNG instantiated
..


[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing 
instruction cache
[    0.000000] percpu: Embedded 16 pages/cpu s34380 r8192 d22964 u65536
[    0.000000] pcpu-alloc: s34380 r8192 d22964 u65536 alloc=16*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
..
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.0 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS migration not required
[    0.000000] psci: SMC Calling Convention v1.0
..
[    0.002872] CPU: Testing write buffer coherency: ok
[    0.003224] CPU0: update cpu_capacity 1024
[    0.003234] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.004687] smp: Bringing up secondary CPUs ...
[    0.005424] CPU1: update cpu_capacity 1024
[    0.005432] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[    0.005553] smp: Brought up 1 node, 2 CPUs
[    0.005568] CPU: All CPU(s) started in HYP mode.
[    0.005571] CPU: Virtualization extensions available.
..
[    0.185229] caam 30900000.caam: device ID = 0x0a16030000000000 (Era 8)
[    0.185240] caam 30900000.caam: job rings = 3, qi = 0
[    0.186894] caam_jr 30901000.jr0: failed to flush job ring 0
[    0.192721] caam_jr: probe of 30901000.jr0 failed with error -5
[    0.192846] caam_jr 30902000.jr1: failed to flush job ring 1
[    0.198796] caam_jr: probe of 30902000.jr1 failed with error -5
[    0.198989] caam_jr 30903000.jr1: failed to flush job ring 2
[    0.204957] caam_jr: probe of 30903000.jr1 failed with error -5
[    0.212619] Job Ring Device allocation for transform failed



Same configuration with 

setenv bootm_boot_mode=sec

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing 
instruction cache
[    0.000000] percpu: Embedded 16 pages/cpu s34380 r8192 d22964 u65536
[    0.000000] pcpu-alloc: s34380 r8192 d22964 u65536 alloc=16*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.002866] CPU: Testing write buffer coherency: ok
[    0.003217] CPU0: update cpu_capacity 1024
[    0.003226] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.004673] smp: Bringing up secondary CPUs ...
[    0.005174] smp: Brought up 1 node, 1 CPU
[    0.005188] CPU: All CPU(s) started in SVC mode.
..
[    0.185631] caam 30900000.caam: device ID = 0x0a16030000000000 (Era 8)
[    0.185643] caam 30900000.caam: job rings = 3, qi = 0
[    0.196909] caam algorithms registered in /proc/crypto
[    0.199620] caam_jr 30901000.jr0: registering rng-caam


=> only 1 CPU core up.


Now I tried to disable the CAAM driver and secure boot support in U-Boot

# CONFIG_SECURE_BOOT is not set
CONFIG_CPU_V7_HAS_NONSEC=y
CONFIG_CPU_V7_HAS_VIRT=y
CONFIG_ARCH_SUPPORT_PSCI=y
CONFIG_ARMV7_NONSEC=y
CONFIG_ARMV7_BOOT_SEC_DEFAULT=y
CONFIG_ARMV7_VIRT=y
CONFIG_ARMV7_PSCI=y
CONFIG_ARMV7_PSCI_NR_CPUS=2
# CONFIG_FSL_CAAM is not set
CONFIG_SYS_FSL_SEC_COMPAT_4=y
# CONFIG_SYS_FSL_SEC_BE is not set
CONFIG_SYS_FSL_SEC_LE=y

 and booting with bootm_boot_mode=nonsec

[    0.185233] caam 30900000.caam: Entropy delay = 3200
[    0.212342] caam 30900000.caam: failed to acquire DECO 0
[    0.217689] caam 30900000.caam: failed to instantiate RNG

=> both CPU cores are up in nonsec mode

Am I missing something?

Thank you and best regards

Tobias

[U-Boot] imx7d: CPU core issue in secure mode

[Fabio Estevam]

[Adding Bryan and Breno]

Hi Bryan,

I think you worked on allowing the CAAM driver in Linux to work on
i.MX7D running in non-secure when you created:
commit 22191ac35344 ("drivers/crypto/fsl: assign job-rings to non-TrustZone")

It was reverted later by Breno as it broke secure boot.

If I understand correctly the current solution is to let OP-TEE deal
with job-rings initialization.

Is there any other alternative to use the mainline kernel CAAM driver
in non-secure if someone is not using OP-TEE?

Thanks,

Fabio Estevam

On Fri, Jul 12, 2019 at 5:20 AM Tobias Junghans
<tobias.junghans@veyon.io> wrote:
>
> Hi Peng,
>
> Am Freitag, 12. Juli 2019, 05:38:21 CEST schrieb Peng Fan:
> > Try "setenv bootm_boot_mode nonsec" in U-Boot stage.
>
> Unfortunately this does not help. I tried the following setups:
>
> CONFIG_SECURE_BOOT=y
> CONFIG_CPU_V7_HAS_NONSEC=y
> CONFIG_CPU_V7_HAS_VIRT=y
> CONFIG_ARCH_SUPPORT_PSCI=y
> CONFIG_ARMV7_NONSEC=y
> CONFIG_ARMV7_BOOT_SEC_DEFAULT=y
> CONFIG_ARMV7_VIRT=y
> CONFIG_ARMV7_PSCI=y
> CONFIG_ARMV7_PSCI_NR_CPUS=2
> CONFIG_FSL_CAAM=y
> CONFIG_SYS_FSL_HAS_SEC=y
> CONFIG_SYS_FSL_SEC_COMPAT_4=y
> # CONFIG_SYS_FSL_SEC_BE is not set
> CONFIG_SYS_FSL_SEC_COMPAT=4
> CONFIG_SYS_FSL_SEC_LE=y
>
>
> Booting with bootm_boot_mode=nonsec
>
>
> U-Boot 2019.07 (Jul 12 2019 - 10:02:31 +0200)
> CPU:   Freescale i.MX7D rev1.3 1000 MHz (running at 792 MHz)
> ..
> SEC0: RNG instantiated
> ..
>
>
> [    0.000000] Booting Linux on physical CPU 0x0
> [    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
> [    0.000000] CPU: div instructions available: patching division code
> [    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing
> instruction cache
> [    0.000000] percpu: Embedded 16 pages/cpu s34380 r8192 d22964 u65536
> [    0.000000] pcpu-alloc: s34380 r8192 d22964 u65536 alloc=16*4096
> [    0.000000] pcpu-alloc: [0] 0 [0] 1
> [    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
> ..
> [    0.000000] psci: probing for conduit method from DT.
> [    0.000000] psci: PSCIv1.0 detected in firmware.
> [    0.000000] psci: Using standard PSCI v0.2 function IDs
> [    0.000000] psci: Trusted OS migration not required
> [    0.000000] psci: SMC Calling Convention v1.0
> ..
> [    0.002872] CPU: Testing write buffer coherency: ok
> [    0.003224] CPU0: update cpu_capacity 1024
> [    0.003234] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> [    0.004687] smp: Bringing up secondary CPUs ...
> [    0.005424] CPU1: update cpu_capacity 1024
> [    0.005432] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> [    0.005553] smp: Brought up 1 node, 2 CPUs
> [    0.005568] CPU: All CPU(s) started in HYP mode.
> [    0.005571] CPU: Virtualization extensions available.
> ..
> [    0.185229] caam 30900000.caam: device ID = 0x0a16030000000000 (Era 8)
> [    0.185240] caam 30900000.caam: job rings = 3, qi = 0
> [    0.186894] caam_jr 30901000.jr0: failed to flush job ring 0
> [    0.192721] caam_jr: probe of 30901000.jr0 failed with error -5
> [    0.192846] caam_jr 30902000.jr1: failed to flush job ring 1
> [    0.198796] caam_jr: probe of 30902000.jr1 failed with error -5
> [    0.198989] caam_jr 30903000.jr1: failed to flush job ring 2
> [    0.204957] caam_jr: probe of 30903000.jr1 failed with error -5
> [    0.212619] Job Ring Device allocation for transform failed
>
>
>
> Same configuration with
>
> setenv bootm_boot_mode=sec
>
> [    0.000000] Booting Linux on physical CPU 0x0
> [    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
> [    0.000000] CPU: div instructions available: patching division code
> [    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing
> instruction cache
> [    0.000000] percpu: Embedded 16 pages/cpu s34380 r8192 d22964 u65536
> [    0.000000] pcpu-alloc: s34380 r8192 d22964 u65536 alloc=16*4096
> [    0.000000] pcpu-alloc: [0] 0 [0] 1
> [    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
> [    0.002866] CPU: Testing write buffer coherency: ok
> [    0.003217] CPU0: update cpu_capacity 1024
> [    0.003226] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> [    0.004673] smp: Bringing up secondary CPUs ...
> [    0.005174] smp: Brought up 1 node, 1 CPU
> [    0.005188] CPU: All CPU(s) started in SVC mode.
> ..
> [    0.185631] caam 30900000.caam: device ID = 0x0a16030000000000 (Era 8)
> [    0.185643] caam 30900000.caam: job rings = 3, qi = 0
> [    0.196909] caam algorithms registered in /proc/crypto
> [    0.199620] caam_jr 30901000.jr0: registering rng-caam
>
>
> => only 1 CPU core up.
>
>
> Now I tried to disable the CAAM driver and secure boot support in U-Boot
>
> # CONFIG_SECURE_BOOT is not set
> CONFIG_CPU_V7_HAS_NONSEC=y
> CONFIG_CPU_V7_HAS_VIRT=y
> CONFIG_ARCH_SUPPORT_PSCI=y
> CONFIG_ARMV7_NONSEC=y
> CONFIG_ARMV7_BOOT_SEC_DEFAULT=y
> CONFIG_ARMV7_VIRT=y
> CONFIG_ARMV7_PSCI=y
> CONFIG_ARMV7_PSCI_NR_CPUS=2
> # CONFIG_FSL_CAAM is not set
> CONFIG_SYS_FSL_SEC_COMPAT_4=y
> # CONFIG_SYS_FSL_SEC_BE is not set
> CONFIG_SYS_FSL_SEC_LE=y
>
>  and booting with bootm_boot_mode=nonsec
>
> [    0.185233] caam 30900000.caam: Entropy delay = 3200
> [    0.212342] caam 30900000.caam: failed to acquire DECO 0
> [    0.217689] caam 30900000.caam: failed to instantiate RNG
>
> => both CPU cores are up in nonsec mode
>
> Am I missing something?
>
> Thank you and best regards
>
> Tobias