[PATCH] rockchip: Enable MMU and caches before relocation

[Oleg Ryjkov <oryjkov@gmail.com>]

Turn on the MMU and both caches at the start of U-Boot proper instead of
after relocation. This saves 1.1 to 1.6 s per boot on RK3566, RK3399 and
RK3588 boards:

                                  board_init_f to       dm_f
                                  board_init_r
  RK3588  ROCK 5B+, SPI flash     1.698 -> 0.085 s   0.999 -> 0.032 s
  RK3588  ROCK 5B+, SD card       1.711 -> 0.085 s   1.005 -> 0.032 s
  RK3399  ROCK Pi 4A, SD card     1.358 -> 0.081 s   0.941 -> 0.024 s
  RK3566  Radxa ZERO 3W, SD card  1.131 -> 0.056 s   0.688 -> 0.020 s

These are means of three boots per column from U-Boot's bootstage
report, with the option off and on in alternating boots. The kernel
starts 1.61, 1.63, 1.25 and 1.08 s earlier.

U-Boot proper is entered with the MMU and both caches off: the SPL turns
them on for itself and off again before it hands over to TF-A. They come
back only in initr_caches() after relocation, so everything in
board_init_f() runs uncached, above all the driver model scanning the
device tree and binding devices (dm_f) and the serial console probe with
its pinctrl and clock lookups. Turning on only the instruction cache
changes nothing; it is the data accesses.

mach_cpu_init() now turns both caches on, with page tables taken from
the pre-relocation malloc area. SYS_MALLOC_F_LEN grows by about their
size (PGTABLE_SIZE is 120 KiB on RK3588, 32 to 56 KiB on the others),
and the tables are taken only if at least 32 KiB of the area remain for
driver model and the rest of board_init_f(), which use about 10 KiB on
these boards. Without that room the caches stay off as before, without
a message. After relocation enable_caches() turns the caches off, which
cleans them, and dcache_enable() builds the final tables where
arch_reserve_mmu() has put them, as i.MX8M does since commit
ac9153c74f30 ("ARM: imx: Enable MMU and dcache very early on i.MX8M").

The mechanism is not specific to Rockchip: most TF-A platforms enter
U-Boot proper with the caches off. What is per platform is the size of
the early malloc area, and so the room for the tables. This starts with
the Rockchip SoCs it could be tested on; STM32MP and i.MX do the same in
their own code, and a generic ARMv8 option could replace these copies
later.

Risks:
- U-Boot entered with the MMU already on now moves to its own page
  tables in enable_caches(); before, it kept the ones it was handed.
- From arch_reserve_mmu() until enable_caches(), gd->arch.tlb_addr names
  tables that do not exist yet, so nothing may change MMU attributes in
  that window. Nothing on Rockchip does today.
- It is on by default only for the SoC families it was tested on. PX30,
  RK3308, RK3328, RK3528 and RK3576 build with it but have not run it;
  RK3368 boards have too small an early malloc area and keep the caches
  off.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Oleg Ryjkov <oryjkov@gmail.com>
---

Notes:
    How it was tested, on master (3cb2cbbc75a):
    
    - Boards: Radxa ROCK 5B+ (RK3588, 8 GiB; from SPI flash and from SD
      card), Radxa ROCK Pi 4A (RK3399, 1 GiB, SD card, mainline TPL) and
      Radxa ZERO 3W (RK3566, 2 GiB, SD card), mainline TF-A, Armbian userland.
    - Measurement: CONFIG_BOOTSTAGE, BOOTSTAGE_REPORT and BOOTSTAGE_FDT;
      three boots per variant, alternating off and on; each column of the
      table is the mean, the boots of a column agree within 0.1 ms
      (board_init_f to board_init_r) and 40 ms (kernel start).
    - Early malloc use, before / after mach_cpu_init() / end of
      board_init_f: RK3588 0x648 / 0x1f000 / 0x21988 of 0x30000,
      RK3399 0x648 / 0x9000 / 0xb520 of 0x20000, RK3566 0x648 / 0xb000 /
      0xd4e8 of 0x20000. With the option off the same boards use 0x2988,
      0x2520 and 0x24e8 bytes after the first 0x648.
    - Room check: the tables are taken only if, placed the way memalign()
      will place them, at least 32 KiB of the early malloc area remain. That
      leaves 36 KiB to spare on RK3588 (0x30000 area, 120 KiB tables) and on
      RK3576 (0x20000, 56 KiB), more on the others.
    - Fallback: the option on with SYS_MALLOC_F_LEN 0x10000 boots like the
      option off, without a message, on all three.
    - Entry with the MMU on: the image started with "go" from a running
      U-Boot with the caches on (after dcache/icache flush) boots Linux on
      all three, with the option on and off.
    - 20 boots in a row of the option on per board: all came back, U-Boot
      times within 0.1 ms, no new kernel log errors.
    - All 132 ARM64 Rockchip defconfigs configure; one per SoC family
      (evb-px30, rock-pi-s-rk3308, rock64-rk3328, geekbox,
      rock-pi-4-rk3399, generic-rk3528, rock-3a-rk3568,
      radxa-zero-3-rk3566, generic-rk3576, rock5b-rk3588) builds with
      -Werror, the untested ones also with the option turned on.
    
    Tested-by tags for PX30, RK3308, RK3328, RK3528 or RK3576 boards would
    let the default cover them too.
    
    Others have hit this phase:
    - "Rockchip RK3399 SOC is very slow" (December 2025): the RK3399 log
      there shows 1.4 s between the BL31 and U-Boot banners, which is this
      phase.
      https://lore.kernel.org/u-boot/0d8cb0a2-5581-4885-b7bc-34402a4ec3c8@arturia.com/
    - Simon Glass and Casey Connolly on the slow uncached pre-relocation FDT
      scan, in the "clk: port full Linux Common Clock Framework" RFC thread
      (March 2026).
      https://lore.kernel.org/u-boot/20260319-casey-ccf-upstream-v1-0-4df2ee2226da@linaro.org/
    - "rockchip: Skip serial pinctrl at pre-reloc phase" (August 2024)
      addressed one part of it, about 150 ms on a ROCK 5B; with the caches
      on the console probe takes about 12 ms.
      https://lore.kernel.org/u-boot/20240805084350.3895788-1-jonas@kwiboo.se/

 arch/arm/mach-rockchip/Kconfig | 20 +++++++++
 arch/arm/mach-rockchip/board.c | 79 ++++++++++++++++++++++++++++++++++
 2 files changed, 99 insertions(+)

diff --git a/arch/arm/mach-rockchip/Kconfig b/arch/arm/mach-rockchip/Kconfig
index 1a2e7847c9e..d133c88894f 100644
--- a/arch/arm/mach-rockchip/Kconfig
+++ b/arch/arm/mach-rockchip/Kconfig
@@ -628,6 +628,24 @@ config ROCKCHIP_COMMON_BOARD
 	  in charge of common process of board_init() and board_late_init() for
 	  U-Boot proper.
 
+config ROCKCHIP_EARLY_CACHES
+	bool "Enable the MMU and caches before relocation"
+	depends on ARM64 && ROCKCHIP_COMMON_BOARD && SYS_MALLOC_F
+	depends on !SYS_DCACHE_OFF && !SYS_ICACHE_OFF
+	default y if ROCKCHIP_RK3399 || ROCKCHIP_RK3568 || ROCKCHIP_RK3588
+	help
+	  Turn on the MMU and both caches at the start of U-Boot proper,
+	  instead of after relocation. Everything before relocation, the
+	  driver model setup and the console probe among it, then runs with
+	  the data cache on. The page tables for this time come from the
+	  pre-relocation malloc area, which grows by about their size
+	  (128 KiB on RK3588). Without that room, for example with a
+	  smaller SYS_MALLOC_F_LEN set by the board, the caches stay off
+	  until relocation, without a message.
+
+	  On by default for RK3399, RK356x and RK3588, where it was tested;
+	  nothing in it is specific to those SoCs.
+
 config SPL_ROCKCHIP_COMMON_BOARD
 	bool "Rockchip SPL common board file"
 	depends on SPL
@@ -806,6 +824,8 @@ config CUSTOM_SYS_INIT_SP_ADDR
 	default 0x03f00000 if SPL_TEXT_BASE = 0x00000000
 
 config SYS_MALLOC_F_LEN
+	default 0x30000 if ROCKCHIP_EARLY_CACHES && ROCKCHIP_RK3588
+	default 0x20000 if ROCKCHIP_EARLY_CACHES
 	default 0x10000 if CUSTOM_SYS_INIT_SP_ADDR = 0x63f00000
 	default 0x10000 if CUSTOM_SYS_INIT_SP_ADDR = 0x43f00000
 	default 0x10000 if CUSTOM_SYS_INIT_SP_ADDR = 0x03f00000
diff --git a/arch/arm/mach-rockchip/board.c b/arch/arm/mach-rockchip/board.c
index 98c890d3bdc..367d0315386 100644
--- a/arch/arm/mach-rockchip/board.c
+++ b/arch/arm/mach-rockchip/board.c
@@ -19,6 +19,7 @@
 #include <hash.h>
 #include <init.h>
 #include <log.h>
+#include <malloc.h>
 #include <mmc.h>
 #include <dm/uclass-internal.h>
 #include <misc.h>
@@ -29,11 +30,16 @@
 #include <u-boot/crc.h>
 #include <u-boot/sha256.h>
 #include <asm/cache.h>
+#include <asm/global_data.h>
+#include <asm/system.h>
 #include <asm/io.h>
 #include <asm/arch-rockchip/boot_mode.h>
 #include <asm/arch-rockchip/clock.h>
 #include <asm/arch-rockchip/periph.h>
 #include <power/regulator.h>
+#include <linux/sizes.h>
+
+DECLARE_GLOBAL_DATA_PTR;
 
 #if IS_ENABLED(CONFIG_EFI_HAVE_CAPSULE_SUPPORT) && IS_ENABLED(CONFIG_EFI_PARTITION) && \
 	IS_ENABLED(CONFIG_PARTITION_TYPE_GUID)
@@ -215,6 +221,79 @@ void enable_caches(void)
 }
 #endif
 
+#if IS_ENABLED(CONFIG_ROCKCHIP_EARLY_CACHES)
+/*
+ * U-Boot proper is entered with the MMU and caches off (SPL turns them off
+ * before it hands over to TF-A) and turns them on only after relocation,
+ * in initr_caches(). Until then every access to the device tree goes to
+ * uncached DRAM, which makes binding the devices and probing the console
+ * very slow. Turn the caches on here instead, with page tables from the
+ * pre-relocation malloc area; enable_caches() moves to the tables reserved
+ * by arch_reserve_mmu() after relocation.
+ */
+int mach_cpu_init(void)
+{
+	ulong size, start, limit;
+	void *tables;
+
+	if (mmu_status())
+		return 0;
+
+	size = PGTABLE_SIZE;
+	start = ALIGN(gd->malloc_base + gd->malloc_ptr, SZ_4K);
+	limit = gd->malloc_base + gd->malloc_limit;
+
+	/*
+	 * SYS_MALLOC_F_LEN is raised by about the size of the tables when
+	 * this option is on. Take them only if at least 32 KiB of the area
+	 * remain for driver model and the rest of board_init_f(), which use
+	 * about 10 KiB on RK3399, RK356x and RK3588. Check first, the way
+	 * memalign() will place them, so that a board without the room
+	 * skips this without an error.
+	 */
+	if (start + size + SZ_32K > limit) {
+		log_debug("No room for early page tables: %lx bytes free\n",
+			  limit - (gd->malloc_base + gd->malloc_ptr));
+		return 0;
+	}
+
+	tables = memalign(SZ_4K, size);
+	if (!tables)
+		return 0;
+
+	gd->arch.tlb_addr = (ulong)tables;
+	gd->arch.tlb_size = size;
+	icache_enable();
+	dcache_enable();
+
+	return 0;
+}
+
+void enable_caches(void)
+{
+	/*
+	 * The early page tables live in the pre-relocation malloc area,
+	 * which is left behind after relocation. Turn the caches off, which
+	 * cleans the data cache, and have dcache_enable() build the tables
+	 * at the place arch_reserve_mmu() has set in gd->arch.tlb_addr.
+	 *
+	 * From arch_reserve_mmu() until here, gd->arch.tlb_addr names the
+	 * final tables, which do not exist yet, while the MMU still runs on
+	 * the early ones that tlb_fillptr and tlb_emerg point into. Nothing
+	 * may change MMU attributes in that window
+	 * (mmu_set_region_dcache_behaviour() and the like).
+	 *
+	 * U-Boot entered with the MMU already on keeps running on the tables
+	 * it was handed until here; this moves it to its own as well.
+	 */
+	dcache_disable();
+	gd->arch.tlb_fillptr = 0;
+
+	icache_enable();
+	dcache_enable();
+}
+#endif
+
 #if IS_ENABLED(CONFIG_USB_GADGET)
 #include <usb.h>
 
-- 
2.54.0