* [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support
@ 2026-11-13 2:32 Alastair D'Silva
2026-11-13 2:32 ` [PATCH 01/26] power: regulator: axp: Fix AXP318W ELDO voltage mask Alastair D'Silva
` (27 more replies)
0 siblings, 28 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
This series adds initial support for the Allwinner A733 (sun60iw2 / sun60i)
SoC and the Xunlong Orange Pi 4 Pro board to U-Boot.
The A733 is an octa-core ARMv8 SoC (dual-cluster Cortex-A76 + Cortex-A55)
with LPDDR5 memory trained by an on-chip Synopsys ARC PMU, an AXP318W PMIC,
eMMC/SD (MMC0/1/2), USB 2.0 EHCI/OHCI, and a DesignWare PCIe Gen3 x1 host
controller paired with a Cadence Combo PHY.
Overview of the series:
- Patches 1-3: Fix AXP318W ELDO voltage mask and sysreset shutdown register,
and extend SPL PMIC support for DCDC6/7 and ALDO/BLDO/DLDO/ELDO rails
required to bring up LPDDR5 and board I/O.
- Patches 4-11: Add A733 CCU/R-CCU dt-bindings, GPIO, pinctrl, clock/reset,
MMC (including transfer-size-scaled data timeout), and UART console support.
- Patches 12-13: Add disposable .init.rodata support for AArch64 SPL (allowing
large DRAM init tables in SRAM A2 without exceeding the 0x58000 BL31 load
address) and ERET-based FEL return support for A733.
- Patches 14-21: Add freestanding ARC PMU LPDDR5 training firmware (built from
C via ARC_CROSS_COMPILE, or supplied via PMU_FW= / BINMAN_ALLOW_MISSING=1),
the A733 LPDDR5 DRAM driver, initial sun60i SoC support, and documentation
in doc/board/allwinner/a733-pmu.rst.
- Patches 22-24: Add the A733 Cadence Combo PHY driver, DesignWare PCIe
controller driver, and NVMe distro boot target support. Note that a small
SoC helper (sunxi_a733_pcie_soc_init) handles PCIe/SerDes CCU muxes,
RTC DCXO feed, SYS_PLL, and NSI QoS until U-Boot's gate-only clk_sunxi
driver gains rate/parent support for A733.
- Patches 25-26: Add the sun60i-a733.dtsi SoC device tree and Xunlong
Orange Pi 4 Pro board support (defconfig, DTS, and MAINTAINERS).
Tested on live Xunlong Orange Pi 4 Pro (6 GiB LPDDR5 @ 3600 MT/s) via FEL
boot with TF-A BL31 v2.15.0 + pending Allwinner A733 TF-A patches:
- https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/55200
(feat(allwinner): add A733 support)
- https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/55204
(feat(allwinner): a733: ungate security controllers)
- https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/55205
(fix(allwinner): restore core security on power-on)
Verified functionality:
- SPL DRAM training (6144 MiB LPDDR5, 3600 MT/s)
- SD card (mmc0: SD32G 29.7 GiB, 4-bit)
- PCIe Gen3 + NVMe SSD (UHPVN0153EQ0V2 238.4 GB, partition enumeration)
Alastair D'Silva (17):
power: regulator: axp: Fix AXP318W ELDO voltage mask
power: pmic: axp: Fix AXP318W sysreset shutdown register
power: axp: Extend AXP318W SPL support with DCDC6/7 and LDOs
mmc: sunxi: Scale data transfer timeout with transfer size
arm64: spl: Support disposable .init.rodata section after
_image_binary_end
sunxi: fel: Add Allwinner A733 ERET FEL return support
sunxi: dram: a733: Add ARC PMU register, ABI, and build definitions
sunxi: dram: a733: Add ARC PMU primitives, clock, telemetry, and CBT
sunxi: dram: a733: Add ARC PMU deskew and BIST training firmware
sunxi: dram: a733: Add ARC PMU 1D/2D eye training firmware
sunxi: dram: a733: Add ARC PMU training state machine
sunxi: dram: a733: Add Allwinner A733 LPDDR5 DRAM driver
doc: sunxi: Document Allwinner A733 LPDDR5 PMU architecture
phy: sunxi: Add Allwinner A733 Cadence Combo PHY driver
pci: sunxi: Add Allwinner A733 DesignWare PCIe host controller driver
sunxi: Add NVMe to distro boot targets
sunxi: Add Xunlong Orange Pi 4 Pro board support
Enzo Adriano (1):
arm64: dts: allwinner: add A733 SoC description
Junhui Liu (1):
dt-bindings: clk: sun60i-a733-ccu: Add Allwinner A733 support
Yixun Lan (7):
gpio: a733: add initial support
pinctrl: sunxi: a733: add initial support
clock: reset: a733: add initial support
clock: a733: implement support for mmc/uart
sunxi: mmc: enable support for A733
sunxi: a733: uart: setup serial console
sunxi: sun60i: initial support for A733 SoC
MAINTAINERS | 13 +
arch/arm/config.mk | 4 +
arch/arm/cpu/armv8/fel_utils.S | 113 +-
arch/arm/cpu/armv8/u-boot-spl.lds | 20 +
arch/arm/cpu/armv8/u-boot.lds | 1 +
arch/arm/dts/Makefile | 2 +
arch/arm/dts/sun60i-a733-orangepi-4-pro.dts | 237 +
arch/arm/dts/sun60i-a733.dtsi | 816 +++
arch/arm/include/asm/arch-sunxi/boot0.h | 20 +-
arch/arm/include/asm/arch-sunxi/clock.h | 1 +
.../include/asm/arch-sunxi/clock_sun50i_h6.h | 19 +
.../include/asm/arch-sunxi/cpu_sunxi_ncat2.h | 43 +-
arch/arm/include/asm/arch-sunxi/dram.h | 2 +
arch/arm/include/asm/arch-sunxi/serial.h | 15 +-
arch/arm/include/asm/arch-sunxi/spl.h | 3 +
arch/arm/include/asm/arch-sunxi/watchdog.h | 1 +
arch/arm/mach-sunxi/Kconfig | 47 +-
arch/arm/mach-sunxi/Makefile | 37 +
arch/arm/mach-sunxi/board.c | 16 +
arch/arm/mach-sunxi/clock_sun50i_h6.c | 149 +-
arch/arm/mach-sunxi/cpu_info.c | 2 +
arch/arm/mach-sunxi/dram_sun60i_a733.c | 1303 +++++
arch/arm/mach-sunxi/dram_sun60i_a733_tables.h | 941 ++++
arch/arm/mach-sunxi/pmu/build_pmu_fw.py | 257 +
arch/arm/mach-sunxi/pmu/dwc_ddrphy_pmu_regs.h | 463 ++
arch/arm/mach-sunxi/pmu/pmu.ld | 38 +
arch/arm/mach-sunxi/pmu/pmu_cbt.c | 417 ++
arch/arm/mach-sunxi/pmu/pmu_clock.c | 938 ++++
arch/arm/mach-sunxi/pmu/pmu_deskew.c | 4895 +++++++++++++++++
arch/arm/mach-sunxi/pmu/pmu_eye.c | 3224 +++++++++++
arch/arm/mach-sunxi/pmu/pmu_fw.h | 553 ++
arch/arm/mach-sunxi/pmu/pmu_primitives.c | 531 ++
arch/arm/mach-sunxi/pmu/pmu_telemetry.c | 619 +++
arch/arm/mach-sunxi/pmu/pmu_train.c | 2748 +++++++++
arch/arm/mach-sunxi/pmu/sunxi_pmu_abi.h | 251 +
arch/arm/mach-sunxi/rmr_switch.S | 22 +-
board/sunxi/MAINTAINERS | 6 +
board/sunxi/board.c | 48 +-
common/spl/Kconfig | 22 +
configs/orangepi_4_pro_defconfig | 36 +
doc/board/allwinner/a733-pmu.rst | 299 +
doc/board/allwinner/index.rst | 1 +
doc/board/allwinner/sunxi.rst | 14 +-
drivers/clk/sunxi/Kconfig | 14 +
drivers/clk/sunxi/Makefile | 2 +
drivers/clk/sunxi/clk_a733.c | 142 +
drivers/clk/sunxi/clk_a733_r.c | 45 +
drivers/clk/sunxi/clk_sunxi.c | 10 +
drivers/gpio/Kconfig | 7 +
drivers/gpio/sunxi_gpio.c | 49 +-
drivers/mmc/sunxi_mmc.c | 48 +-
drivers/pci/Kconfig | 11 +
drivers/pci/Makefile | 1 +
drivers/pci/pcie_dw_sunxi.c | 355 ++
drivers/phy/allwinner/Kconfig | 8 +
drivers/phy/allwinner/Makefile | 1 +
drivers/phy/allwinner/phy-sun60i-combophy.c | 293 +
drivers/pinctrl/sunxi/Kconfig | 18 +
drivers/pinctrl/sunxi/pinctrl-sunxi.c | 98 +-
drivers/power/Kconfig | 110 +-
drivers/power/axp_spl.c | 119 +-
drivers/power/pmic/axp.c | 4 +-
drivers/power/regulator/axp_regulator.c | 12 +-
include/axp_pmic.h | 5 +
include/configs/sun60i.h | 11 +
include/configs/sunxi-common.h | 7 +
.../clock/allwinner,sun60i-a733-ccu.h | 291 +
.../clock/allwinner,sun60i-a733-r-ccu.h | 38 +
.../reset/allwinner,sun60i-a733-ccu.h | 129 +
.../reset/allwinner,sun60i-a733-r-ccu.h | 23 +
include/sunxi_gpio.h | 10 +-
71 files changed, 20948 insertions(+), 100 deletions(-)
create mode 100644 arch/arm/dts/sun60i-a733-orangepi-4-pro.dts
create mode 100644 arch/arm/dts/sun60i-a733.dtsi
create mode 100644 arch/arm/mach-sunxi/dram_sun60i_a733.c
create mode 100644 arch/arm/mach-sunxi/dram_sun60i_a733_tables.h
create mode 100755 arch/arm/mach-sunxi/pmu/build_pmu_fw.py
create mode 100644 arch/arm/mach-sunxi/pmu/dwc_ddrphy_pmu_regs.h
create mode 100644 arch/arm/mach-sunxi/pmu/pmu.ld
create mode 100644 arch/arm/mach-sunxi/pmu/pmu_cbt.c
create mode 100644 arch/arm/mach-sunxi/pmu/pmu_clock.c
create mode 100644 arch/arm/mach-sunxi/pmu/pmu_deskew.c
create mode 100644 arch/arm/mach-sunxi/pmu/pmu_eye.c
create mode 100644 arch/arm/mach-sunxi/pmu/pmu_fw.h
create mode 100644 arch/arm/mach-sunxi/pmu/pmu_primitives.c
create mode 100644 arch/arm/mach-sunxi/pmu/pmu_telemetry.c
create mode 100644 arch/arm/mach-sunxi/pmu/pmu_train.c
create mode 100644 arch/arm/mach-sunxi/pmu/sunxi_pmu_abi.h
create mode 100644 configs/orangepi_4_pro_defconfig
create mode 100644 doc/board/allwinner/a733-pmu.rst
create mode 100644 drivers/clk/sunxi/clk_a733.c
create mode 100644 drivers/clk/sunxi/clk_a733_r.c
create mode 100644 drivers/pci/pcie_dw_sunxi.c
create mode 100644 drivers/phy/allwinner/phy-sun60i-combophy.c
create mode 100644 include/configs/sun60i.h
create mode 100644 include/dt-bindings/clock/allwinner,sun60i-a733-ccu.h
create mode 100644 include/dt-bindings/clock/allwinner,sun60i-a733-r-ccu.h
create mode 100644 include/dt-bindings/reset/allwinner,sun60i-a733-ccu.h
create mode 100644 include/dt-bindings/reset/allwinner,sun60i-a733-r-ccu.h
--
2.53.0
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH 01/26] power: regulator: axp: Fix AXP318W ELDO voltage mask
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-20 0:06 ` Andre Przywara
2026-11-13 2:32 ` [PATCH 02/26] power: pmic: axp: Fix AXP318W sysreset shutdown register Alastair D'Silva
` (26 subsequent siblings)
27 siblings, 1 reply; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
On AXP318W, ELDO1-ELDO6 span 500 mV to 1500 mV in 25 mV steps (40 steps,
selectors 0x00..0x28), requiring a 6-bit selector mask (0x3f) rather
than the 5-bit mask (0x1f, max 0x1f -> 1275 mV) currently listed in
axp318_regulators[].
Fixes: 71f07932756f ("power: regulator: add AXP318W support")
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
drivers/power/regulator/axp_regulator.c | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/drivers/power/regulator/axp_regulator.c b/drivers/power/regulator/axp_regulator.c
index 16d3a8f7f90..e425f3bf8b2 100644
--- a/drivers/power/regulator/axp_regulator.c
+++ b/drivers/power/regulator/axp_regulator.c
@@ -227,12 +227,12 @@ static const struct axp_regulator_plat axp318_regulators[] = {
{ "dldo4", 0x22, BIT(3), 0x37, 0x1f, 500, 3400, 100, NA },
{ "dldo5", 0x22, BIT(4), 0x38, 0x1f, 500, 3400, 100, NA },
{ "dldo6", 0x22, BIT(5), 0x39, 0x1f, 500, 3400, 100, NA },
- { "eldo1", 0x22, BIT(6), 0x3a, 0x1f, 500, 1500, 25, NA },
- { "eldo2", 0x22, BIT(7), 0x3b, 0x1f, 500, 1500, 25, NA },
- { "eldo3", 0x23, BIT(0), 0x3c, 0x1f, 500, 1500, 25, NA },
- { "eldo4", 0x23, BIT(1), 0x3d, 0x1f, 500, 1500, 25, NA },
- { "eldo5", 0x23, BIT(2), 0x3e, 0x1f, 500, 1500, 25, NA },
- { "eldo6", 0x23, BIT(3), 0x3f, 0x1f, 500, 1500, 25, NA },
+ { "eldo1", 0x22, BIT(6), 0x3a, 0x3f, 500, 1500, 25, NA },
+ { "eldo2", 0x22, BIT(7), 0x3b, 0x3f, 500, 1500, 25, NA },
+ { "eldo3", 0x23, BIT(0), 0x3c, 0x3f, 500, 1500, 25, NA },
+ { "eldo4", 0x23, BIT(1), 0x3d, 0x3f, 500, 1500, 25, NA },
+ { "eldo5", 0x23, BIT(2), 0x3e, 0x3f, 500, 1500, 25, NA },
+ { "eldo6", 0x23, BIT(3), 0x3f, 0x3f, 500, 1500, 25, NA },
{ "swout1", 0x11, BIT(3), NA, NA, NA, NA, NA, NA },
{ "swout2", 0x11, BIT(4), NA, NA, NA, NA, NA, NA },
{ }
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 02/26] power: pmic: axp: Fix AXP318W sysreset shutdown register
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
2026-11-13 2:32 ` [PATCH 01/26] power: regulator: axp: Fix AXP318W ELDO voltage mask Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 03/26] power: axp: Extend AXP318W SPL support with DCDC6/7 and LDOs Alastair D'Silva
` (25 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
AXP318W moves the PMIC poweroff control bit (BIT(7)) to register 0x55
because register 0x32 is repurposed as the CLDO4 voltage control
register. Define AXP318_SHUTDOWN (0x55) in axp_pmic.h and select it in
axp_sysreset_request() when driving AXP318_ID.
Fixes: 71f07932756f ("power: regulator: add AXP318W support")
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
drivers/power/pmic/axp.c | 4 +++-
include/axp_pmic.h | 2 ++
2 files changed, 5 insertions(+), 1 deletion(-)
diff --git a/drivers/power/pmic/axp.c b/drivers/power/pmic/axp.c
index 7d8348b0769..c24165ef67e 100644
--- a/drivers/power/pmic/axp.c
+++ b/drivers/power/pmic/axp.c
@@ -10,12 +10,14 @@
#if CONFIG_IS_ENABLED(SYSRESET)
static int axp_sysreset_request(struct udevice *dev, enum sysreset_t type)
{
+ ulong id = dev_get_driver_data(dev->parent);
+ u8 reg = (id == AXP318_ID) ? AXP318_SHUTDOWN : AXP152_SHUTDOWN;
int ret;
if (type != SYSRESET_POWER_OFF)
return -EPROTONOSUPPORT;
- ret = pmic_clrsetbits(dev->parent, AXP152_SHUTDOWN, 0, AXP152_POWEROFF);
+ ret = pmic_clrsetbits(dev->parent, reg, 0, AXP152_POWEROFF);
if (ret < 0)
return ret;
diff --git a/include/axp_pmic.h b/include/axp_pmic.h
index 2f547da74c2..5df0e269b9d 100644
--- a/include/axp_pmic.h
+++ b/include/axp_pmic.h
@@ -26,6 +26,8 @@
#define AXP_PMIC_SEC_DEVICE_ADDR 0x745
#define AXP_PMIC_SEC_RUNTIME_ADDR 0x3a
+#define AXP318_SHUTDOWN 0x55
+
enum {
AXP152_ID,
AXP202_ID,
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 03/26] power: axp: Extend AXP318W SPL support with DCDC6/7 and LDOs
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
2026-11-13 2:32 ` [PATCH 01/26] power: regulator: axp: Fix AXP318W ELDO voltage mask Alastair D'Silva
2026-11-13 2:32 ` [PATCH 02/26] power: pmic: axp: Fix AXP318W sysreset shutdown register Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 04/26] dt-bindings: clk: sun60i-a733-ccu: Add Allwinner A733 support Alastair D'Silva
` (24 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
Commit 61c2f29bde1d ("power: regulator: enable AWP318W SPL support")
introduced initial AXP318W DCDC1-DCDC5 support in axp_spl.c, but
Allwinner A733 boards using AXP318W also supply LPDDR5 VDDQ/VDD2 and
PLL/IO rails from DCDC6, DCDC7, ALDO, BLDO, DLDO, and ELDO regulators
during SPL bringup, and sunxi_board_init() did not yet invoke axp_init()
when CONFIG_AXP318W_POWER is enabled.
Extend axp_spl.c with read-modify-write selector updates (preserving
upper control bits outside volt_mask), -1 OTP-default skip semantics,
DCDC6/DCDC7 helpers, and ALDO/BLDO/DLDO/ELDO descriptors and setters for
AXP318W, and wire AXP318W_POWER, DCDC6/7, and BLDO1-4 into Kconfig and
sunxi_board_init().
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
board/sunxi/board.c | 23 +++++++-
drivers/power/Kconfig | 110 +++++++++++++++++++++++++++++++------
drivers/power/axp_spl.c | 119 +++++++++++++++++++++++++++++++++++++---
include/axp_pmic.h | 3 +
4 files changed, 229 insertions(+), 26 deletions(-)
diff --git a/board/sunxi/board.c b/board/sunxi/board.c
index 3d1afec7c66..299faa890b9 100644
--- a/board/sunxi/board.c
+++ b/board/sunxi/board.c
@@ -586,9 +586,10 @@ void sunxi_board_init(void)
defined CONFIG_AXP221_POWER || defined CONFIG_AXP305_POWER || \
defined CONFIG_AXP809_POWER || defined CONFIG_AXP818_POWER || \
defined CONFIG_AXP313_POWER || defined CONFIG_AXP717_POWER || \
- defined CONFIG_AXP803_POWER
+ defined CONFIG_AXP803_POWER || defined CONFIG_AXP318W_POWER
power_failed = axp_init();
+#ifdef AXP_POWER_STATUS
if (IS_ENABLED(CONFIG_AXP_DISABLE_BOOT_ON_POWERON) && !power_failed) {
u8 boot_reason;
@@ -598,6 +599,7 @@ void sunxi_board_init(void)
pmic_bus_write(0x32, BIT(7));
}
}
+#endif
#ifdef CONFIG_AXP_DCDC1_VOLT
power_failed |= axp_set_dcdc1(CONFIG_AXP_DCDC1_VOLT);
@@ -614,6 +616,12 @@ void sunxi_board_init(void)
#ifdef CONFIG_AXP_DCDC5_VOLT
power_failed |= axp_set_dcdc5(CONFIG_AXP_DCDC5_VOLT);
#endif
+#ifdef CONFIG_AXP_DCDC6_VOLT
+ power_failed |= axp_set_dcdc6(CONFIG_AXP_DCDC6_VOLT);
+#endif
+#ifdef CONFIG_AXP_DCDC7_VOLT
+ power_failed |= axp_set_dcdc7(CONFIG_AXP_DCDC7_VOLT);
+#endif
#ifdef CONFIG_AXP_ALDO1_VOLT
power_failed |= axp_set_aldo1(CONFIG_AXP_ALDO1_VOLT);
@@ -628,6 +636,19 @@ void sunxi_board_init(void)
power_failed |= axp_set_aldo4(CONFIG_AXP_ALDO4_VOLT);
#endif
+#ifdef CONFIG_AXP_BLDO1_VOLT
+ power_failed |= axp_set_bldo(1, CONFIG_AXP_BLDO1_VOLT);
+#endif
+#ifdef CONFIG_AXP_BLDO2_VOLT
+ power_failed |= axp_set_bldo(2, CONFIG_AXP_BLDO2_VOLT);
+#endif
+#ifdef CONFIG_AXP_BLDO3_VOLT
+ power_failed |= axp_set_bldo(3, CONFIG_AXP_BLDO3_VOLT);
+#endif
+#ifdef CONFIG_AXP_BLDO4_VOLT
+ power_failed |= axp_set_bldo(4, CONFIG_AXP_BLDO4_VOLT);
+#endif
+
#ifdef CONFIG_AXP_DLDO1_VOLT
power_failed |= axp_set_dldo(1, CONFIG_AXP_DLDO1_VOLT);
power_failed |= axp_set_dldo(2, CONFIG_AXP_DLDO2_VOLT);
diff --git a/drivers/power/Kconfig b/drivers/power/Kconfig
index b3fd0232938..e202790c4cc 100644
--- a/drivers/power/Kconfig
+++ b/drivers/power/Kconfig
@@ -171,12 +171,15 @@ config AXP_I2C_ADDRESS
config AXP_DCDC1_VOLT
int "axp pmic dcdc1 voltage"
- depends on AXP221_POWER || AXP809_POWER || AXP818_POWER || AXP803_POWER
+ depends on AXP221_POWER || AXP809_POWER || AXP818_POWER || \
+ AXP803_POWER || AXP318W_POWER
+ default -1 if AXP318W_POWER
default 3300 if AXP818_POWER || MACH_SUN8I_R40 || AXP803_POWER
default 3000 if MACH_SUN6I || MACH_SUN8I || MACH_SUN9I
help
Set the voltage (mV) to program the axp pmic dcdc1 at, set to 0 to
- disable dcdc1. On A23 / A31 / A33 (axp221) boards dcdc1 is used for
+ disable dcdc1, or -1 on AXP318W to leave the OTP default untouched.
+ On A23 / A31 / A33 (axp221) boards dcdc1 is used for
generic 3.3V IO voltage for external devices like the lcd-panal and
sdcard interfaces, etc. On most boards dcdc1 is undervolted to 3.0V to
save battery. On A31 devices dcdc1 is also used for VCC-IO. On A83T
@@ -185,7 +188,10 @@ config AXP_DCDC1_VOLT
config AXP_DCDC2_VOLT
int "axp pmic dcdc2 voltage"
- depends on AXP152_POWER || AXP209_POWER || AXP221_POWER || AXP809_POWER || AXP818_POWER || AXP313_POWER || AXP717_POWER || AXP803_POWER
+ depends on AXP152_POWER || AXP209_POWER || AXP221_POWER || \
+ AXP809_POWER || AXP818_POWER || AXP313_POWER || \
+ AXP717_POWER || AXP803_POWER || AXP318W_POWER
+ default -1 if AXP318W_POWER
default 900 if AXP818_POWER
default 1400 if AXP152_POWER || (AXP209_POWER && !MACH_SUN8I_V3S)
default 1000 if AXP313_POWER
@@ -209,7 +215,10 @@ config AXP_DCDC2_VOLT
config AXP_DCDC3_VOLT
int "axp pmic dcdc3 voltage"
- depends on AXP152_POWER || AXP209_POWER || AXP221_POWER || AXP809_POWER || AXP818_POWER || AXP313_POWER || AXP717_POWER
+ depends on AXP152_POWER || AXP209_POWER || AXP221_POWER || \
+ AXP809_POWER || AXP818_POWER || AXP313_POWER || \
+ AXP717_POWER || AXP318W_POWER
+ default -1 if AXP318W_POWER
default 900 if AXP809_POWER || AXP818_POWER
default 1500 if AXP152_POWER
default 1250 if (AXP209_POWER && !MACH_SUN8I_V3S)
@@ -233,7 +242,9 @@ config AXP_DCDC3_VOLT
config AXP_DCDC4_VOLT
int "axp pmic dcdc4 voltage"
- depends on AXP152_POWER || AXP221_POWER || AXP809_POWER || AXP305_POWER
+ depends on AXP152_POWER || AXP221_POWER || AXP809_POWER || \
+ AXP305_POWER || AXP318W_POWER
+ default -1 if AXP318W_POWER
default 1250 if AXP152_POWER
default 1200 if MACH_SUN6I
default 0 if MACH_SUN8I
@@ -251,7 +262,9 @@ config AXP_DCDC4_VOLT
config AXP_DCDC5_VOLT
int "axp pmic dcdc5 voltage"
- depends on AXP221_POWER || AXP809_POWER || AXP818_POWER || AXP803_POWER
+ depends on AXP221_POWER || AXP809_POWER || AXP818_POWER || \
+ AXP803_POWER || AXP318W_POWER
+ default -1 if AXP318W_POWER
default 1500 if MACH_SUN6I || MACH_SUN8I || MACH_SUN9I
help
Set the voltage (mV) to program the axp pmic dcdc5 at, set to 0 to
@@ -259,9 +272,28 @@ config AXP_DCDC5_VOLT
On A23 / A31 / A33 / A80 / A83T / R40 boards dcdc5 is VCC-DRAM and
should be 1.5V, 1.35V if DDR3L is used.
+config AXP_DCDC6_VOLT
+ int "axp pmic dcdc6 voltage"
+ depends on AXP318W_POWER
+ default -1
+ help
+ Set the voltage (mV) to program the axp pmic dcdc6 at, set to 0 to
+ disable dcdc6, or -1 to leave the OTP default untouched.
+ On A733 LPDDR5 boards dcdc6 typically supplies VDDQ-DRAM (0.5V-0.56V).
+
+config AXP_DCDC7_VOLT
+ int "axp pmic dcdc7 voltage"
+ depends on AXP318W_POWER
+ default -1
+ help
+ Set the voltage (mV) to program the axp pmic dcdc7 at, set to 0 to
+ disable dcdc7, or -1 to leave the OTP default untouched.
+ On A733 LPDDR5 boards dcdc7 typically supplies VDD2-DRAM (1.05V-1.08V).
+
config AXP_ALDO1_VOLT
int "axp pmic (a)ldo1 voltage"
- depends on AXP221_POWER || AXP809_POWER || AXP818_POWER
+ depends on AXP221_POWER || AXP809_POWER || AXP818_POWER || AXP318W_POWER
+ default -1 if AXP318W_POWER
default 0 if MACH_SUN6I || MACH_SUN8I_R40
default 1800 if MACH_SUN8I_A83T
default 3000 if MACH_SUN8I || MACH_SUN9I
@@ -276,7 +308,9 @@ config AXP_ALDO1_VOLT
config AXP_ALDO2_VOLT
int "axp pmic (a)ldo2 voltage"
- depends on AXP152_POWER || AXP209_POWER || AXP221_POWER || AXP809_POWER || AXP818_POWER
+ depends on AXP152_POWER || AXP209_POWER || AXP221_POWER || \
+ AXP809_POWER || AXP818_POWER || AXP318W_POWER
+ default -1 if AXP318W_POWER
default 3000 if AXP152_POWER || AXP209_POWER
default 0 if MACH_SUN6I || MACH_SUN9I
default 1800 if MACH_SUN8I_A83T
@@ -294,7 +328,9 @@ config AXP_ALDO2_VOLT
config AXP_ALDO3_VOLT
int "axp pmic (a)ldo3 voltage"
- depends on AXP209_POWER || AXP221_POWER || AXP809_POWER || AXP818_POWER
+ depends on AXP209_POWER || AXP221_POWER || AXP809_POWER || \
+ AXP818_POWER || AXP318W_POWER
+ default -1 if AXP318W_POWER
default 0 if AXP209_POWER || MACH_SUN9I
default 3000 if MACH_SUN6I || MACH_SUN8I
help
@@ -351,16 +387,50 @@ config AXP_ALDO3_INRUSH_QUIRK
config AXP_ALDO4_VOLT
int "axp pmic (a)ldo4 voltage"
- depends on AXP209_POWER
+ depends on AXP209_POWER || AXP318W_POWER
+ default -1 if AXP318W_POWER
default 0 if AXP209_POWER
help
Set the voltage (mV) to program the axp pmic aldo4 at, set to 0 to
disable aldo4.
On A10(s) / A13 / A20 boards aldo4 should be 2.8V.
+config AXP_BLDO1_VOLT
+ int "axp pmic bldo1 voltage"
+ depends on AXP318W_POWER
+ default -1
+ help
+ Set the voltage (mV) to program the axp pmic bldo1 at, set to 0 to
+ disable bldo1, or -1 to leave the OTP default untouched.
+
+config AXP_BLDO2_VOLT
+ int "axp pmic bldo2 voltage"
+ depends on AXP318W_POWER
+ default -1
+ help
+ Set the voltage (mV) to program the axp pmic bldo2 at, set to 0 to
+ disable bldo2, or -1 to leave the OTP default untouched.
+
+config AXP_BLDO3_VOLT
+ int "axp pmic bldo3 voltage"
+ depends on AXP318W_POWER
+ default -1
+ help
+ Set the voltage (mV) to program the axp pmic bldo3 at, set to 0 to
+ disable bldo3, or -1 to leave the OTP default untouched.
+
+config AXP_BLDO4_VOLT
+ int "axp pmic bldo4 voltage"
+ depends on AXP318W_POWER
+ default -1
+ help
+ Set the voltage (mV) to program the axp pmic bldo4 at, set to 0 to
+ disable bldo4, or -1 to leave the OTP default untouched.
+
config AXP_DLDO1_VOLT
int "axp pmic dldo1 voltage"
- depends on AXP221_POWER || AXP809_POWER || AXP818_POWER
+ depends on AXP221_POWER || AXP809_POWER || AXP818_POWER || AXP318W_POWER
+ default -1 if AXP318W_POWER
default 0
help
Set the voltage (mV) to program the axp pmic dldo1 at, set to 0 to
@@ -370,7 +440,8 @@ config AXP_DLDO1_VOLT
config AXP_DLDO2_VOLT
int "axp pmic dldo2 voltage"
- depends on AXP221_POWER || AXP809_POWER || AXP818_POWER
+ depends on AXP221_POWER || AXP809_POWER || AXP818_POWER || AXP318W_POWER
+ default -1 if AXP318W_POWER
default 3000 if MACH_SUN9I
default 0
help
@@ -380,7 +451,8 @@ config AXP_DLDO2_VOLT
config AXP_DLDO3_VOLT
int "axp pmic dldo3 voltage"
- depends on AXP221_POWER || AXP818_POWER
+ depends on AXP221_POWER || AXP818_POWER || AXP318W_POWER
+ default -1 if AXP318W_POWER
default 0
help
Set the voltage (mV) to program the axp pmic dldo3 at, set to 0 to
@@ -388,7 +460,8 @@ config AXP_DLDO3_VOLT
config AXP_DLDO4_VOLT
int "axp pmic dldo4 voltage"
- depends on AXP221_POWER || AXP818_POWER
+ depends on AXP221_POWER || AXP818_POWER || AXP318W_POWER
+ default -1 if AXP318W_POWER
default 0
help
Set the voltage (mV) to program the axp pmic dldo4 at, set to 0 to
@@ -396,7 +469,8 @@ config AXP_DLDO4_VOLT
config AXP_ELDO1_VOLT
int "axp pmic eldo1 voltage"
- depends on AXP221_POWER || AXP809_POWER || AXP818_POWER
+ depends on AXP221_POWER || AXP809_POWER || AXP818_POWER || AXP318W_POWER
+ default -1 if AXP318W_POWER
default 0
help
Set the voltage (mV) to program the axp pmic eldo1 at, set to 0 to
@@ -404,7 +478,8 @@ config AXP_ELDO1_VOLT
config AXP_ELDO2_VOLT
int "axp pmic eldo2 voltage"
- depends on AXP221_POWER || AXP809_POWER || AXP818_POWER
+ depends on AXP221_POWER || AXP809_POWER || AXP818_POWER || AXP318W_POWER
+ default -1 if AXP318W_POWER
default 0
help
Set the voltage (mV) to program the axp pmic eldo2 at, set to 0 to
@@ -412,7 +487,8 @@ config AXP_ELDO2_VOLT
config AXP_ELDO3_VOLT
int "axp pmic eldo3 voltage"
- depends on AXP221_POWER || AXP809_POWER || AXP818_POWER
+ depends on AXP221_POWER || AXP809_POWER || AXP818_POWER || AXP318W_POWER
+ default -1 if AXP318W_POWER
default 3000 if MACH_SUN9I
default 0
help
diff --git a/drivers/power/axp_spl.c b/drivers/power/axp_spl.c
index 0162ef5e4cc..af638e30ec5 100644
--- a/drivers/power/axp_spl.c
+++ b/drivers/power/axp_spl.c
@@ -81,10 +81,45 @@ static const struct axp_reg_desc_spl axp_spl_dcdc_regulators[] = {
{ 0x11, BIT(0), 0x1a, 0x7f, 500, 1840, 10, 70 },
};
+static const struct axp_reg_desc_spl axp_spl_aldo_regulators[] = {
+ { 0x20, BIT(0), 0x24, 0x1f, 500, 3400, 100, NA },
+ { 0x20, BIT(1), 0x25, 0x1f, 500, 3400, 100, NA },
+ { 0x20, BIT(2), 0x26, 0x1f, 500, 3400, 100, NA },
+ { 0x20, BIT(3), 0x27, 0x1f, 500, 3400, 100, NA },
+ { 0x20, BIT(4), 0x28, 0x1f, 500, 3400, 100, NA },
+ { 0x20, BIT(5), 0x29, 0x1f, 500, 3400, 100, NA },
+};
+
+static const struct axp_reg_desc_spl axp_spl_bldo_regulators[] = {
+ { 0x20, BIT(6), 0x2a, 0x1f, 500, 3400, 100, NA },
+ { 0x20, BIT(7), 0x2b, 0x1f, 500, 3400, 100, NA },
+ { 0x21, BIT(0), 0x2c, 0x1f, 500, 3400, 100, NA },
+ { 0x21, BIT(1), 0x2d, 0x1f, 500, 3400, 100, NA },
+ { 0x21, BIT(2), 0x2e, 0x1f, 500, 3400, 100, NA },
+};
+
+static const struct axp_reg_desc_spl axp_spl_dldo_regulators[] = {
+ { 0x22, BIT(0), 0x34, 0x1f, 500, 3400, 100, NA },
+ { 0x22, BIT(1), 0x35, 0x1f, 500, 3400, 100, NA },
+ { 0x22, BIT(2), 0x36, 0x1f, 500, 3400, 100, NA },
+ { 0x22, BIT(3), 0x37, 0x1f, 500, 3400, 100, NA },
+ { 0x22, BIT(4), 0x38, 0x1f, 500, 3400, 100, NA },
+ { 0x22, BIT(5), 0x39, 0x1f, 500, 3400, 100, NA },
+};
+
+static const struct axp_reg_desc_spl axp_spl_eldo_regulators[] = {
+ { 0x22, BIT(6), 0x3a, 0x3f, 500, 1500, 25, NA },
+ { 0x22, BIT(7), 0x3b, 0x3f, 500, 1500, 25, NA },
+ { 0x23, BIT(0), 0x3c, 0x3f, 500, 1500, 25, NA },
+ { 0x23, BIT(1), 0x3d, 0x3f, 500, 1500, 25, NA },
+ { 0x23, BIT(2), 0x3e, 0x3f, 500, 1500, 25, NA },
+ { 0x23, BIT(3), 0x3f, 0x3f, 500, 1500, 25, NA },
+};
+
#define AXP_CHIP_VERSION 0
#define AXP_CHIP_VERSION_MASK 0
#define AXP_CHIP_ID 0
-#define AXP_SHUTDOWN_REG 0x55
+#define AXP_SHUTDOWN_REG AXP318_SHUTDOWN
#define AXP_SHUTDOWN_MASK BIT(7)
#elif defined(CONFIG_AXP305_POWER) /* AXP305 */
@@ -127,26 +162,38 @@ static u8 axp_mvolt_to_cfg(int mvolt, const struct axp_reg_desc_spl *reg)
return reg->split + mvolt / (reg->step_mV * 2);
}
-static int axp_set_dcdc(int dcdc_num, unsigned int mvolt)
+static int axp_set_reg(const struct axp_reg_desc_spl *reg, unsigned int mvolt)
{
- const struct axp_reg_desc_spl *reg;
+ u8 cfg, val;
int ret;
- if (dcdc_num < 1 || dcdc_num > ARRAY_SIZE(axp_spl_dcdc_regulators))
- return -EINVAL;
-
- reg = &axp_spl_dcdc_regulators[dcdc_num - 1];
+ if (mvolt == (unsigned int)-1)
+ return 0;
if (mvolt == 0)
return pmic_bus_clrbits(reg->enable_reg, reg->enable_mask);
- ret = pmic_bus_write(reg->volt_reg, axp_mvolt_to_cfg(mvolt, reg));
+ cfg = axp_mvolt_to_cfg(mvolt, reg);
+ ret = pmic_bus_read(reg->volt_reg, &val);
+ if (ret)
+ return ret;
+
+ val = (val & ~reg->volt_mask) | (cfg & reg->volt_mask);
+ ret = pmic_bus_write(reg->volt_reg, val);
if (ret)
return ret;
return pmic_bus_setbits(reg->enable_reg, reg->enable_mask);
}
+static int axp_set_dcdc(int dcdc_num, unsigned int mvolt)
+{
+ if (dcdc_num < 1 || dcdc_num > ARRAY_SIZE(axp_spl_dcdc_regulators))
+ return -EINVAL;
+
+ return axp_set_reg(&axp_spl_dcdc_regulators[dcdc_num - 1], mvolt);
+}
+
int axp_set_dcdc1(unsigned int mvolt)
{
return axp_set_dcdc(1, mvolt);
@@ -172,6 +219,62 @@ int axp_set_dcdc5(unsigned int mvolt)
return axp_set_dcdc(5, mvolt);
}
+#ifdef CONFIG_AXP318W_POWER
+int axp_set_dcdc6(unsigned int mvolt)
+{
+ return axp_set_dcdc(6, mvolt);
+}
+
+int axp_set_dcdc7(unsigned int mvolt)
+{
+ return axp_set_dcdc(7, mvolt);
+}
+
+int axp_set_aldo1(unsigned int mvolt)
+{
+ return axp_set_reg(&axp_spl_aldo_regulators[0], mvolt);
+}
+
+int axp_set_aldo2(unsigned int mvolt)
+{
+ return axp_set_reg(&axp_spl_aldo_regulators[1], mvolt);
+}
+
+int axp_set_aldo3(unsigned int mvolt)
+{
+ return axp_set_reg(&axp_spl_aldo_regulators[2], mvolt);
+}
+
+int axp_set_aldo4(unsigned int mvolt)
+{
+ return axp_set_reg(&axp_spl_aldo_regulators[3], mvolt);
+}
+
+int axp_set_bldo(int bldo_num, unsigned int mvolt)
+{
+ if (bldo_num < 1 || bldo_num > ARRAY_SIZE(axp_spl_bldo_regulators))
+ return -EINVAL;
+
+ return axp_set_reg(&axp_spl_bldo_regulators[bldo_num - 1], mvolt);
+}
+
+int axp_set_dldo(int dldo_num, unsigned int mvolt)
+{
+ if (dldo_num < 1 || dldo_num > ARRAY_SIZE(axp_spl_dldo_regulators))
+ return -EINVAL;
+
+ return axp_set_reg(&axp_spl_dldo_regulators[dldo_num - 1], mvolt);
+}
+
+int axp_set_eldo(int eldo_num, unsigned int mvolt)
+{
+ if (eldo_num < 1 || eldo_num > ARRAY_SIZE(axp_spl_eldo_regulators))
+ return -EINVAL;
+
+ return axp_set_reg(&axp_spl_eldo_regulators[eldo_num - 1], mvolt);
+}
+#endif
+
int axp_init(void)
{
int ret = pmic_bus_init();
diff --git a/include/axp_pmic.h b/include/axp_pmic.h
index 5df0e269b9d..b2463876e52 100644
--- a/include/axp_pmic.h
+++ b/include/axp_pmic.h
@@ -49,10 +49,13 @@ int axp_set_dcdc2(unsigned int mvolt);
int axp_set_dcdc3(unsigned int mvolt);
int axp_set_dcdc4(unsigned int mvolt);
int axp_set_dcdc5(unsigned int mvolt);
+int axp_set_dcdc6(unsigned int mvolt);
+int axp_set_dcdc7(unsigned int mvolt);
int axp_set_aldo1(unsigned int mvolt);
int axp_set_aldo2(unsigned int mvolt);
int axp_set_aldo3(unsigned int mvolt);
int axp_set_aldo4(unsigned int mvolt);
+int axp_set_bldo(int bldo_num, unsigned int mvolt);
int axp_set_dldo(int dldo_num, unsigned int mvolt);
int axp_set_eldo(int eldo_num, unsigned int mvolt);
int axp_set_fldo(int fldo_num, unsigned int mvolt);
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 04/26] dt-bindings: clk: sun60i-a733-ccu: Add Allwinner A733 support
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (2 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 03/26] power: axp: Extend AXP318W SPL support with DCDC6/7 and LDOs Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 05/26] mmc: sunxi: Scale data transfer timeout with transfer size Alastair D'Silva
` (23 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva, Krzysztof Kozlowski
From: Junhui Liu <junhui.liu@pigmoral.tech>
The CCU and R-CCU (PRCM) modules provide clocks and reset functions for
the Allwinner A733 SoC. The clock architecture of the A733 is evolved
from the A523, though the root clocking strategy transitions from a
static oscillator frequency in the Devicetree to the "hosc" clock, which
is determined by choosing from three possible frequencies (19.2MHz,
24MHz, or 26MHz) by the RTC hardware, and finally feeds the CCU and
R-CCU.
Additionally, the MCU_CCU module found in previous designs is removed
from the A733, and the clock tree is expanded with more clock outputs
to support new functional modules.
Import the DT binding headers from v6 of the upstream Linux series.
Reviewed-by: Krzysztof Kozlowski <krzysztof.kozlowski@oss.qualcomm.com>
Signed-off-by: Junhui Liu <junhui.liu@pigmoral.tech>
Link: https://lore.kernel.org/r/20261009-a733-clk-v6-1-ae8178fce7e8@pigmoral.tech
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
.../clock/allwinner,sun60i-a733-ccu.h | 291 ++++++++++++++++++
.../clock/allwinner,sun60i-a733-r-ccu.h | 38 +++
.../reset/allwinner,sun60i-a733-ccu.h | 129 ++++++++
.../reset/allwinner,sun60i-a733-r-ccu.h | 23 ++
4 files changed, 481 insertions(+)
create mode 100644 include/dt-bindings/clock/allwinner,sun60i-a733-ccu.h
create mode 100644 include/dt-bindings/clock/allwinner,sun60i-a733-r-ccu.h
create mode 100644 include/dt-bindings/reset/allwinner,sun60i-a733-ccu.h
create mode 100644 include/dt-bindings/reset/allwinner,sun60i-a733-r-ccu.h
diff --git a/include/dt-bindings/clock/allwinner,sun60i-a733-ccu.h b/include/dt-bindings/clock/allwinner,sun60i-a733-ccu.h
new file mode 100644
index 00000000000..81924cba58e
--- /dev/null
+++ b/include/dt-bindings/clock/allwinner,sun60i-a733-ccu.h
@@ -0,0 +1,291 @@
+/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) */
+/*
+ * Copyright (C) 2026 Junhui Liu <junhui.liu@pigmoral.tech>
+ *
+ * The user manual uses PCLK, HCLK, and MCLK suffixes for some bus clocks.
+ * These are represented here as APB, AHB, and MBUS prefixes respectively,
+ * with the bus type moved to the front of the clock name.
+ */
+
+#ifndef _DT_BINDINGS_CLK_ALLWINNER_SUN60I_A733_CCU_H_
+#define _DT_BINDINGS_CLK_ALLWINNER_SUN60I_A733_CCU_H_
+
+#define CLK_PLL_REF 0
+#define CLK_SYS_24M 1
+#define CLK_PLL_DDR 2
+#define CLK_PLL_PERIPH0_4X 3
+#define CLK_PLL_PERIPH0_2X 4
+#define CLK_PLL_PERIPH0_800M 5
+#define CLK_PLL_PERIPH0_480M 6
+#define CLK_PLL_PERIPH0_600M 7
+#define CLK_PLL_PERIPH0_400M 8
+#define CLK_PLL_PERIPH0_300M 9
+#define CLK_PLL_PERIPH0_200M 10
+#define CLK_PLL_PERIPH0_160M 11
+#define CLK_PLL_PERIPH0_150M 12
+#define CLK_PLL_PERIPH1_4X 13
+#define CLK_PLL_PERIPH1_2X 14
+#define CLK_PLL_PERIPH1_800M 15
+#define CLK_PLL_PERIPH1_480M 16
+#define CLK_PLL_PERIPH1_600M 17
+#define CLK_PLL_PERIPH1_400M 18
+#define CLK_PLL_PERIPH1_300M 19
+#define CLK_PLL_PERIPH1_200M 20
+#define CLK_PLL_PERIPH1_160M 21
+#define CLK_PLL_PERIPH1_150M 22
+#define CLK_PLL_GPU0 23
+#define CLK_PLL_VIDEO0 24
+#define CLK_PLL_VIDEO0_4X 25
+#define CLK_PLL_VIDEO0_3X 26
+#define CLK_PLL_VIDEO1 27
+#define CLK_PLL_VIDEO1_4X 28
+#define CLK_PLL_VIDEO1_3X 29
+#define CLK_PLL_VIDEO2 30
+#define CLK_PLL_VIDEO2_4X 31
+#define CLK_PLL_VIDEO2_3X 32
+#define CLK_PLL_VE0 33
+#define CLK_PLL_VE1 34
+#define CLK_PLL_AUDIO0_4X 35
+#define CLK_PLL_AUDIO1 36
+#define CLK_PLL_AUDIO1_DIV2 37
+#define CLK_PLL_AUDIO1_DIV5 38
+#define CLK_PLL_NPU 39
+#define CLK_PLL_DE 40
+#define CLK_PLL_DE_4X 41
+#define CLK_PLL_DE_3X 42
+#define CLK_AHB 43
+#define CLK_APB0 44
+#define CLK_APB1 45
+#define CLK_APB_UART 46
+#define CLK_TRACE 47
+#define CLK_GIC 48
+#define CLK_BUS_ITS_PCIE0_ACLK 49
+#define CLK_MBUS 50
+#define CLK_MBUS_IOMMU0_SYS 51
+#define CLK_APB_IOMMU0_SYS 52
+#define CLK_AHB_IOMMU0_SYS 53
+#define CLK_BUS_MSI_LITE0 54
+#define CLK_BUS_MSI_LITE1 55
+#define CLK_BUS_MSI_LITE2 56
+#define CLK_MBUS_IOMMU1_SYS 57
+#define CLK_APB_IOMMU1_SYS 58
+#define CLK_AHB_IOMMU1_SYS 59
+#define CLK_AHB_VE_DEC 60
+#define CLK_AHB_VE_ENC 61
+#define CLK_AHB_VID_IN 62
+#define CLK_AHB_VID_COUT0 63
+#define CLK_AHB_VID_COUT1 64
+#define CLK_AHB_DE 65
+#define CLK_AHB_NPU 66
+#define CLK_AHB_GPU0 67
+#define CLK_AHB_SERDES 68
+#define CLK_AHB_USB_SYS 69
+#define CLK_AHB_MSI_LITE0 70
+#define CLK_AHB_STORE 71
+#define CLK_AHB_CPUS 72
+#define CLK_MBUS_IOMMU0 73
+#define CLK_MBUS_IOMMU1 74
+#define CLK_MBUS_DESYS 75
+#define CLK_MBUS_VE_ENC0_GATE 76
+#define CLK_MBUS_VE_DEC0_GATE 77
+#define CLK_MBUS_GPU0 78
+#define CLK_MBUS_NPU 79
+#define CLK_MBUS_VID_IN 80
+#define CLK_MBUS_SERDES 81
+#define CLK_MBUS_MSI_LITE0 82
+#define CLK_MBUS_STORE 83
+#define CLK_MBUS_MSI_LITE2 84
+#define CLK_MBUS_DMA0 85
+#define CLK_MBUS_VE_ENC0 86
+#define CLK_MBUS_CE 87
+#define CLK_MBUS_DMA1 88
+#define CLK_MBUS_NAND0 89
+#define CLK_MBUS_CSI 90
+#define CLK_MBUS_ISP 91
+#define CLK_MBUS_GMAC0 92
+#define CLK_MBUS_GMAC1 93
+#define CLK_MBUS_VE_DEC0 94
+#define CLK_BUS_DMA0 95
+#define CLK_BUS_DMA1 96
+#define CLK_BUS_SPINLOCK 97
+#define CLK_BUS_MSGBOX0 98
+#define CLK_BUS_PWM0 99
+#define CLK_BUS_PWM1 100
+#define CLK_BUS_DBG 101
+#define CLK_BUS_SYSDAP 102
+#define CLK_TIMER0 103
+#define CLK_TIMER1 104
+#define CLK_TIMER2 105
+#define CLK_TIMER3 106
+#define CLK_TIMER4 107
+#define CLK_TIMER5 108
+#define CLK_TIMER6 109
+#define CLK_TIMER7 110
+#define CLK_TIMER8 111
+#define CLK_TIMER9 112
+#define CLK_BUS_TIMER 113
+#define CLK_AVS 114
+#define CLK_DE0 115
+#define CLK_BUS_DE0 116
+#define CLK_DI 117
+#define CLK_BUS_DI 118
+#define CLK_G2D 119
+#define CLK_BUS_G2D 120
+#define CLK_EINK 121
+#define CLK_EINK_PANEL 122
+#define CLK_BUS_EINK 123
+#define CLK_VE_ENC0 124
+#define CLK_VE_DEC0 125
+#define CLK_BUS_VE_ENC0 126
+#define CLK_BUS_VE_DEC0 127
+#define CLK_CE 128
+#define CLK_BUS_CE 129
+#define CLK_BUS_CE_SYS 130
+#define CLK_NPU 131
+#define CLK_BUS_NPU 132
+#define CLK_GPU0 133
+#define CLK_BUS_GPU0 134
+#define CLK_DRAM0 135
+#define CLK_BUS_DRAM0 136
+#define CLK_NAND0_CLK0 137
+#define CLK_NAND0_CLK1 138
+#define CLK_BUS_NAND0 139
+#define CLK_MMC0 140
+#define CLK_BUS_MMC0 141
+#define CLK_MMC1 142
+#define CLK_BUS_MMC1 143
+#define CLK_MMC2 144
+#define CLK_BUS_MMC2 145
+#define CLK_MMC3 146
+#define CLK_BUS_MMC3 147
+#define CLK_UFS_AXI 148
+#define CLK_UFS_CFG 149
+#define CLK_BUS_UFS 150
+#define CLK_BUS_UART0 151
+#define CLK_BUS_UART1 152
+#define CLK_BUS_UART2 153
+#define CLK_BUS_UART3 154
+#define CLK_BUS_UART4 155
+#define CLK_BUS_UART5 156
+#define CLK_BUS_UART6 157
+#define CLK_BUS_I2C0 158
+#define CLK_BUS_I2C1 159
+#define CLK_BUS_I2C2 160
+#define CLK_BUS_I2C3 161
+#define CLK_BUS_I2C4 162
+#define CLK_BUS_I2C5 163
+#define CLK_BUS_I2C6 164
+#define CLK_BUS_I2C7 165
+#define CLK_BUS_I2C8 166
+#define CLK_BUS_I2C9 167
+#define CLK_BUS_I2C10 168
+#define CLK_BUS_I2C11 169
+#define CLK_BUS_I2C12 170
+#define CLK_SPI0 171
+#define CLK_BUS_SPI0 172
+#define CLK_SPI1 173
+#define CLK_BUS_SPI1 174
+#define CLK_SPI2 175
+#define CLK_BUS_SPI2 176
+#define CLK_SPIF 177
+#define CLK_BUS_SPIF 178
+#define CLK_SPI3 179
+#define CLK_BUS_SPI3 180
+#define CLK_SPI4 181
+#define CLK_BUS_SPI4 182
+#define CLK_GPADC0_24M 183
+#define CLK_BUS_GPADC0 184
+#define CLK_BUS_THS0 185
+#define CLK_IRRX 186
+#define CLK_BUS_IRRX 187
+#define CLK_IRTX 188
+#define CLK_BUS_IRTX 189
+#define CLK_BUS_LRADC 190
+#define CLK_SGPIO 191
+#define CLK_BUS_SGPIO 192
+#define CLK_LPC 193
+#define CLK_BUS_LPC 194
+#define CLK_I2SPCM0 195
+#define CLK_BUS_I2SPCM0 196
+#define CLK_I2SPCM1 197
+#define CLK_BUS_I2SPCM1 198
+#define CLK_I2SPCM2 199
+#define CLK_I2SPCM2_ASRC 200
+#define CLK_BUS_I2SPCM2 201
+#define CLK_I2SPCM3 202
+#define CLK_BUS_I2SPCM3 203
+#define CLK_I2SPCM4 204
+#define CLK_BUS_I2SPCM4 205
+#define CLK_SPDIF_TX 206
+#define CLK_SPDIF_RX 207
+#define CLK_BUS_SPDIF 208
+#define CLK_DMIC 209
+#define CLK_BUS_DMIC 210
+#define CLK_USB_OHCI0 211
+#define CLK_BUS_OHCI0 212
+#define CLK_BUS_EHCI0 213
+#define CLK_BUS_OTG 214
+#define CLK_USB_OHCI1 215
+#define CLK_BUS_OHCI1 216
+#define CLK_BUS_EHCI1 217
+#define CLK_USB01_REF 218
+#define CLK_USB2_U2_REF 219
+#define CLK_USB2_SUSPEND 220
+#define CLK_USB2_MF 221
+#define CLK_USB2_U3_UTMI 222
+#define CLK_USB2_U2_PIPE 223
+#define CLK_PCIE_AUX 224
+#define CLK_PCIE_AXI_SLV 225
+#define CLK_SERDES_PHY 226
+#define CLK_GMAC_PTP 227
+#define CLK_GMAC0_PHY 228
+#define CLK_BUS_GMAC0 229
+#define CLK_GMAC1_PHY 230
+#define CLK_BUS_GMAC1 231
+#define CLK_TCON_LCD0 232
+#define CLK_BUS_TCON_LCD0 233
+#define CLK_TCON_LCD1 234
+#define CLK_BUS_TCON_LCD1 235
+#define CLK_TCON_LCD2 236
+#define CLK_BUS_TCON_LCD2 237
+#define CLK_DSI0 238
+#define CLK_BUS_DSI0 239
+#define CLK_DSI1 240
+#define CLK_BUS_DSI1 241
+#define CLK_COMBPHY0 242
+#define CLK_COMBPHY1 243
+#define CLK_BUS_TCON_TV0 244
+#define CLK_BUS_TCON_TV1 245
+#define CLK_EDP_TV 246
+#define CLK_BUS_EDP_TV 247
+#define CLK_HDMI_CEC_32K 248
+#define CLK_HDMI_CEC 249
+#define CLK_HDMI_TV 250
+#define CLK_BUS_HDMI_TV 251
+#define CLK_HDMI_SFR 252
+#define CLK_HDCP_ESM 253
+#define CLK_BUS_DPSS_TOP0 254
+#define CLK_BUS_DPSS_TOP1 255
+#define CLK_LEDC 256
+#define CLK_BUS_LEDC 257
+#define CLK_BUS_DSC 258
+#define CLK_CSI_MASTER0 259
+#define CLK_CSI_MASTER1 260
+#define CLK_CSI_MASTER2 261
+#define CLK_CSI 262
+#define CLK_BUS_CSI 263
+#define CLK_ISP 264
+#define CLK_RES_DCAP_24M 265
+#define CLK_APB2JTAG 266
+#define CLK_FANOUT_24M 267
+#define CLK_FANOUT_12M 268
+#define CLK_FANOUT_16M 269
+#define CLK_FANOUT_25M 270
+#define CLK_FANOUT_27M 271
+#define CLK_FANOUT_PCLK 272
+#define CLK_FANOUT0 273
+#define CLK_FANOUT1 274
+#define CLK_FANOUT2 275
+#define CLK_FANOUT3 276
+
+#endif /* _DT_BINDINGS_CLK_ALLWINNER_SUN60I_A733_CCU_H_ */
diff --git a/include/dt-bindings/clock/allwinner,sun60i-a733-r-ccu.h b/include/dt-bindings/clock/allwinner,sun60i-a733-r-ccu.h
new file mode 100644
index 00000000000..401a1e66e80
--- /dev/null
+++ b/include/dt-bindings/clock/allwinner,sun60i-a733-r-ccu.h
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) */
+/*
+ * Copyright (C) 2026 Junhui Liu <junhui.liu@pigmoral.tech>
+ */
+
+#ifndef _DT_BINDINGS_CLK_ALLWINNER_SUN60I_A733_R_CCU_H_
+#define _DT_BINDINGS_CLK_ALLWINNER_SUN60I_A733_R_CCU_H_
+
+#define CLK_R_AHB 0
+#define CLK_R_APB0 1
+#define CLK_R_APB1 2
+#define CLK_R_TIMER0 3
+#define CLK_R_TIMER1 4
+#define CLK_R_TIMER2 5
+#define CLK_R_TIMER3 6
+#define CLK_BUS_R_TIMER 7
+#define CLK_R_PWM 8
+#define CLK_BUS_R_PWM 9
+#define CLK_R_SPI 10
+#define CLK_BUS_R_SPI 11
+#define CLK_BUS_R_MSGBOX 12
+#define CLK_BUS_R_UART0 13
+#define CLK_BUS_R_UART1 14
+#define CLK_BUS_R_I2C0 15
+#define CLK_BUS_R_I2C1 16
+#define CLK_BUS_R_I2C2 17
+#define CLK_BUS_R_PPU 18
+#define CLK_BUS_R_TZMA 19
+#define CLK_BUS_R_CPU_BIST 20
+#define CLK_R_IR_RX 21
+#define CLK_BUS_R_IR_RX 22
+#define CLK_BUS_R_RTC 23
+#define CLK_R_RISCV 24
+#define CLK_BUS_R_RISCV 25
+#define CLK_BUS_R_RISCV_CFG 26
+#define CLK_BUS_R_CPUCFG 27
+
+#endif /* _DT_BINDINGS_CLK_ALLWINNER_SUN60I_A733_R_CCU_H_ */
diff --git a/include/dt-bindings/reset/allwinner,sun60i-a733-ccu.h b/include/dt-bindings/reset/allwinner,sun60i-a733-ccu.h
new file mode 100644
index 00000000000..a4db2f673bd
--- /dev/null
+++ b/include/dt-bindings/reset/allwinner,sun60i-a733-ccu.h
@@ -0,0 +1,129 @@
+/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) */
+/*
+ * Copyright (C) 2026 Junhui Liu <junhui.liu@pigmoral.tech>
+ */
+
+#ifndef _DT_BINDINGS_RST_ALLWINNER_SUN60I_A733_CCU_H_
+#define _DT_BINDINGS_RST_ALLWINNER_SUN60I_A733_CCU_H_
+
+#define RST_BUS_ITS_PCIE0 0
+#define RST_BUS_IOMMU0_SYS 1
+#define RST_BUS_MSI_LITE0_AHB 2
+#define RST_BUS_MSI_LITE0_MBUS 3
+#define RST_BUS_MSI_LITE1_AHB 4
+#define RST_BUS_MSI_LITE1_MBUS 5
+#define RST_BUS_MSI_LITE2_AHB 6
+#define RST_BUS_MSI_LITE2_MBUS 7
+#define RST_BUS_IOMMU1_SYS 8
+#define RST_BUS_DMA0 9
+#define RST_BUS_DMA1 10
+#define RST_BUS_SPINLOCK 11
+#define RST_BUS_MSGBOX 12
+#define RST_BUS_PWM0 13
+#define RST_BUS_PWM1 14
+#define RST_BUS_DBG 15
+#define RST_BUS_SYSDAP 16
+#define RST_BUS_TIMER0 17
+#define RST_BUS_DE0 18
+#define RST_BUS_DI 19
+#define RST_BUS_G2D 20
+#define RST_BUS_EINK 21
+#define RST_BUS_DE_SYS 22
+#define RST_BUS_VE_ENC0 23
+#define RST_BUS_VE_DEC0 24
+#define RST_BUS_CE 25
+#define RST_BUS_CE_SYS 26
+#define RST_BUS_NPU_CORE 27
+#define RST_BUS_NPU_AXI 28
+#define RST_BUS_NPU_AHB 29
+#define RST_BUS_NPU_SRAM 30
+#define RST_BUS_GPU0 31
+#define RST_BUS_DRAM0 32
+#define RST_BUS_NAND0 33
+#define RST_BUS_MMC0 34
+#define RST_BUS_MMC1 35
+#define RST_BUS_MMC2 36
+#define RST_BUS_MMC3 37
+#define RST_BUS_UFS_AHB 38
+#define RST_BUS_UFS_AXI 39
+#define RST_BUS_UFS_PHY 40
+#define RST_BUS_UFS_CORE 41
+#define RST_BUS_UART0 42
+#define RST_BUS_UART1 43
+#define RST_BUS_UART2 44
+#define RST_BUS_UART3 45
+#define RST_BUS_UART4 46
+#define RST_BUS_UART5 47
+#define RST_BUS_UART6 48
+#define RST_BUS_I2C0 49
+#define RST_BUS_I2C1 50
+#define RST_BUS_I2C2 51
+#define RST_BUS_I2C3 52
+#define RST_BUS_I2C4 53
+#define RST_BUS_I2C5 54
+#define RST_BUS_I2C6 55
+#define RST_BUS_I2C7 56
+#define RST_BUS_I2C8 57
+#define RST_BUS_I2C9 58
+#define RST_BUS_I2C10 59
+#define RST_BUS_I2C11 60
+#define RST_BUS_I2C12 61
+#define RST_BUS_SPI0 62
+#define RST_BUS_SPI1 63
+#define RST_BUS_SPI2 64
+#define RST_BUS_SPIF 65
+#define RST_BUS_SPI3 66
+#define RST_BUS_SPI4 67
+#define RST_BUS_GPADC0 68
+#define RST_BUS_THS0 69
+#define RST_BUS_IRRX 70
+#define RST_BUS_IRTX 71
+#define RST_BUS_LRADC 72
+#define RST_BUS_SGPIO 73
+#define RST_BUS_LPC 74
+#define RST_BUS_I2SPCM0 75
+#define RST_BUS_I2SPCM1 76
+#define RST_BUS_I2SPCM2 77
+#define RST_BUS_I2SPCM3 78
+#define RST_BUS_I2SPCM4 79
+#define RST_BUS_SPDIF 80
+#define RST_BUS_DMIC 81
+#define RST_USB_PHY0 82
+#define RST_BUS_OHCI0 83
+#define RST_BUS_EHCI0 84
+#define RST_BUS_OTG 85
+#define RST_USB_PHY1 86
+#define RST_BUS_OHCI1 87
+#define RST_BUS_EHCI1 88
+#define RST_BUS_USB2 89
+#define RST_BUS_PCIE_PWRUP 90
+#define RST_BUS_PCIE 91
+#define RST_BUS_SERDES 92
+#define RST_BUS_GMAC0 93
+#define RST_BUS_GMAC0_AXI 94
+#define RST_BUS_GMAC1 95
+#define RST_BUS_GMAC1_AXI 96
+#define RST_BUS_TCON_LCD0 97
+#define RST_BUS_TCON_LCD1 98
+#define RST_BUS_TCON_LCD2 99
+#define RST_BUS_LVDS0 100
+#define RST_BUS_LVDS1 101
+#define RST_BUS_DSI0 102
+#define RST_BUS_DSI1 103
+#define RST_BUS_TCON_TV0 104
+#define RST_BUS_TCON_TV1 105
+#define RST_BUS_EDP 106
+#define RST_BUS_HDMI_MAIN 107
+#define RST_BUS_HDMI_SUB 108
+#define RST_BUS_HDMI_HDCP 109
+#define RST_BUS_DPSS_TOP0 110
+#define RST_BUS_DPSS_TOP1 111
+#define RST_BUS_VIDEO_OUT0 112
+#define RST_BUS_VIDEO_OUT1 113
+#define RST_BUS_LEDC 114
+#define RST_BUS_DSC 115
+#define RST_BUS_CSI 116
+#define RST_BUS_VIDEO_IN 117
+#define RST_BUS_APB2JTAG 118
+
+#endif /* _DT_BINDINGS_RST_ALLWINNER_SUN60I_A733_CCU_H_ */
diff --git a/include/dt-bindings/reset/allwinner,sun60i-a733-r-ccu.h b/include/dt-bindings/reset/allwinner,sun60i-a733-r-ccu.h
new file mode 100644
index 00000000000..69fc7d405e1
--- /dev/null
+++ b/include/dt-bindings/reset/allwinner,sun60i-a733-r-ccu.h
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) */
+/*
+ * Copyright (C) 2026 Junhui Liu <junhui.liu@pigmoral.tech>
+ */
+
+#ifndef _DT_BINDINGS_RST_ALLWINNER_SUN60I_A733_R_CCU_H_
+#define _DT_BINDINGS_RST_ALLWINNER_SUN60I_A733_R_CCU_H_
+
+#define RST_BUS_R_TIMER 0
+#define RST_BUS_R_PWM 1
+#define RST_BUS_R_SPI 2
+#define RST_BUS_R_MSGBOX 3
+#define RST_BUS_R_UART0 4
+#define RST_BUS_R_UART1 5
+#define RST_BUS_R_I2C0 6
+#define RST_BUS_R_I2C1 7
+#define RST_BUS_R_I2C2 8
+#define RST_BUS_R_IR_RX 9
+#define RST_BUS_R_RTC 10
+#define RST_BUS_R_RISCV_CFG 11
+#define RST_BUS_R_CPUCFG 12
+
+#endif /* _DT_BINDINGS_RST_ALLWINNER_SUN60I_A733_R_CCU_H_ */
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 05/26] mmc: sunxi: Scale data transfer timeout with transfer size
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (3 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 04/26] dt-bindings: clk: sun60i-a733-ccu: Add Allwinner A733 support Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 06/26] gpio: a733: add initial support Alastair D'Silva
` (22 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
sunxi_mmc_send_cmd_common() currently uses a hardcoded 120 ms timeout
when waiting for SUNXI_MMC_RINT_DATA_OVER or
SUNXI_MMC_RINT_AUTO_COMMAND_DONE. Large multi-block reads at the initial
400 kHz card identification clock (or across slower eMMC/SD bus modes)
easily exceed 120 ms and time out spuriously.
Scale the data completion timeout dynamically with transfer byte count
(~4 ms per KiB above the 120 ms floor), clamped to UINT_MAX to prevent
32-bit truncation, and fix the debug message typo/format specifier.
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
drivers/mmc/sunxi_mmc.c | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/drivers/mmc/sunxi_mmc.c b/drivers/mmc/sunxi_mmc.c
index 84f4e4c2198..12c92b1e2b6 100644
--- a/drivers/mmc/sunxi_mmc.c
+++ b/drivers/mmc/sunxi_mmc.c
@@ -417,8 +417,16 @@ static int sunxi_mmc_send_cmd_common(struct sunxi_mmc_priv *priv,
goto out;
if (data) {
- timeout_msecs = 120;
- debug("cacl timeout %x msec\n", timeout_msecs);
+ /*
+ * Scale timeout with transfer size (~4 ms per KiB) so large
+ * multi-block transfers at 400 kHz card-init clock do not
+ * hit a fixed 120 ms ceiling, clamping to UINT_MAX.
+ */
+ u64 calc_msecs = 120ULL + (((u64)bytecnt * 4ULL) >> 10);
+
+ timeout_msecs = (calc_msecs > UINT_MAX) ? UINT_MAX :
+ (unsigned int)calc_msecs;
+ debug("calc timeout %u msec\n", timeout_msecs);
error = mmc_rint_wait(priv, mmc, timeout_msecs,
data->blocks > 1 ?
SUNXI_MMC_RINT_AUTO_COMMAND_DONE :
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 06/26] gpio: a733: add initial support
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (4 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 05/26] mmc: sunxi: Scale data transfer timeout with transfer size Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 07/26] pinctrl: sunxi: " Alastair D'Silva
` (21 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
From: Yixun Lan <dlan@gentoo.org>
In A733 SoC, the main GPIO register layout has changed: Port B begins at
offset 0x100 (leaving a 0x80-byte virtual Port A slot at offset 0x80)
with a 0x80-byte bank stride and DRV/PULL offsets at 0x20/0x30, whereas
R-PIO (0x07025000) retains the NCAT2 0x30-byte bank stride and DRV/PULL
offsets at 0x14/0x24.
Introduce CONFIG_SUNXI_A733_PINCTRL to support both main PIO and R-PIO
bank addressing, 4-bit drive strength masks, and UART0 PB0/PB9 mux
constants.
Signed-off-by: Yixun Lan <dlan@gentoo.org>
Link: https://lore.kernel.org/r/20260113-01-a733-soc-support-v3-6-f19dcd098f60@gentoo.org
[Alastair: preserve NCAT2 DRV/PULL offsets for R-PIO, fix 4-bit DRV mask
and preprocessor guards, add PB0/PB9 UART0 mux defines]
Co-developed-by: Alastair D'Silva <alastair@d-silva.org>
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
drivers/gpio/Kconfig | 7 ++++++
drivers/gpio/sunxi_gpio.c | 49 +++++++++++++++++++++++++++++++--------
include/sunxi_gpio.h | 10 +++++++-
3 files changed, 55 insertions(+), 11 deletions(-)
diff --git a/drivers/gpio/Kconfig b/drivers/gpio/Kconfig
index 7f12e39dad8..725d5c3c277 100644
--- a/drivers/gpio/Kconfig
+++ b/drivers/gpio/Kconfig
@@ -472,6 +472,13 @@ config SUNXI_NEW_PINCTRL
The Allwinner D1 and other new SoCs use a different register map
for the GPIO block, which we need to know about in the SPL.
+config SUNXI_A733_PINCTRL
+ bool
+ depends on SUNXI_GPIO
+ help
+ The Allwinner A733 uses a 0x80-byte bank stride starting at offset
+ 0x80 for main PIO while retaining 0x30-byte banks for R-PIO.
+
config XILINX_GPIO
bool "Xilinx GPIO driver"
depends on DM_GPIO
diff --git a/drivers/gpio/sunxi_gpio.c b/drivers/gpio/sunxi_gpio.c
index 094c45a6927..a997d528fd6 100644
--- a/drivers/gpio/sunxi_gpio.c
+++ b/drivers/gpio/sunxi_gpio.c
@@ -38,22 +38,48 @@
#define GPIO_DAT_REG_OFFSET 0x10
-#define GPIO_DRV_REG_OFFSET 0x14
-
/* Newer SoCs use a slightly different register layout */
-#ifdef CONFIG_SUNXI_NEW_PINCTRL
+#if defined(CONFIG_SUNXI_NEW_PINCTRL) || defined(CONFIG_SUNXI_A733_PINCTRL)
/* pin drive strength: 4 bits per pin */
#define GPIO_DRV_INDEX(pin) ((pin) / 8)
#define GPIO_DRV_OFFSET(pin) (((pin) % 8) * 4)
+#define GPIO_DRV_MASK 0xfU
+
+/*
+ * On A733, main PIO (0x02000000) places DRV at 0x20 and PULL at 0x30, whereas
+ * R-PIO (0x07025000) retains the NCAT2 layout (DRV at 0x14, PULL at 0x24).
+ */
+static inline u32 gpio_drv_reg_offset(void *bank_base)
+{
+ if (IS_ENABLED(CONFIG_SUNXI_A733_PINCTRL) &&
+ (uintptr_t)bank_base < SUNXI_R_PIO_BASE)
+ return 0x20;
+ return 0x14;
+}
-#define GPIO_PULL_REG_OFFSET 0x24
+static inline u32 gpio_pull_reg_offset(void *bank_base)
+{
+ if (IS_ENABLED(CONFIG_SUNXI_A733_PINCTRL) &&
+ (uintptr_t)bank_base < SUNXI_R_PIO_BASE)
+ return 0x30;
+ return 0x24;
+}
#else /* older generation pin controllers */
/* pin drive strength: 2 bits per pin */
#define GPIO_DRV_INDEX(pin) ((pin) / 16)
#define GPIO_DRV_OFFSET(pin) (((pin) % 16) * 2)
+#define GPIO_DRV_MASK 0x3U
-#define GPIO_PULL_REG_OFFSET 0x1c
+static inline u32 gpio_drv_reg_offset(void *bank_base)
+{
+ return 0x14;
+}
+
+static inline u32 gpio_pull_reg_offset(void *bank_base)
+{
+ return 0x1c;
+}
#endif
#define GPIO_PULL_INDEX(pin) (((pin) & 0x1f) >> 4)
@@ -65,12 +91,15 @@ static void* BANK_TO_GPIO(int bank)
if (bank < SUNXI_GPIO_L) {
pio_base = (void *)(uintptr_t)SUNXI_PIO_BASE;
+ if (IS_ENABLED(CONFIG_SUNXI_A733_PINCTRL))
+ return pio_base + SUN60I_A733_PINCTRL_BANK_OFFSET +
+ bank * SUN60I_A733_PINCTRL_BANK_SIZE;
+ return pio_base + bank * SUNXI_PINCTRL_BANK_SIZE;
} else {
pio_base = (void *)(uintptr_t)SUNXI_R_PIO_BASE;
bank -= SUNXI_GPIO_L;
+ return pio_base + bank * SUNXI_PINCTRL_BANK_SIZE;
}
-
- return pio_base + bank * SUNXI_PINCTRL_BANK_SIZE;
}
void sunxi_gpio_set_cfgbank(void *bank_base, int pin_offset, u32 val)
@@ -136,8 +165,8 @@ void sunxi_gpio_set_drv_bank(void *bank_base, u32 pin_offset, u32 val)
u32 index = GPIO_DRV_INDEX(pin_offset);
u32 offset = GPIO_DRV_OFFSET(pin_offset);
- clrsetbits_le32(bank_base + GPIO_DRV_REG_OFFSET + index * 4,
- 0x3U << offset, val << offset);
+ clrsetbits_le32(bank_base + gpio_drv_reg_offset(bank_base) + index * 4,
+ GPIO_DRV_MASK << offset, val << offset);
}
void sunxi_gpio_set_pull(u32 pin, u32 val)
@@ -153,7 +182,7 @@ void sunxi_gpio_set_pull_bank(void *bank_base, int pin_offset, u32 val)
u32 index = GPIO_PULL_INDEX(pin_offset);
u32 offset = GPIO_PULL_OFFSET(pin_offset);
- clrsetbits_le32(bank_base + GPIO_PULL_REG_OFFSET + index * 4,
+ clrsetbits_le32(bank_base + gpio_pull_reg_offset(bank_base) + index * 4,
0x3U << offset, val << offset);
}
diff --git a/include/sunxi_gpio.h b/include/sunxi_gpio.h
index 12b54c8dda4..ff181d77bbd 100644
--- a/include/sunxi_gpio.h
+++ b/include/sunxi_gpio.h
@@ -19,6 +19,9 @@
#elif defined(CONFIG_SUN50I_GEN_H6)
#define SUNXI_PIO_BASE 0x0300b000
#define SUNXI_R_PIO_BASE 0x07022000
+#elif defined(CONFIG_MACH_SUN60I_A733)
+#define SUNXI_PIO_BASE 0x02000000
+#define SUNXI_R_PIO_BASE 0x07025000
#elif defined(CONFIG_SUNXI_GEN_NCAT2)
#define SUNXI_PIO_BASE 0x02000000
#define SUNXI_R_PIO_BASE 0x07022000
@@ -117,6 +120,8 @@ enum sunxi_gpio_number {
#define SUN8I_A83T_GPB_UART0 2
#define SUN8I_V3S_GPB_UART0 3
#define SUN50I_GPB_UART0 4
+#define SUN60I_A733_GPB0_UART0 3
+#define SUN60I_A733_GPB9_UART0 2
#define SUNXI_GPC_NAND 2
#define SUNXI_GPC_SPI0 3
@@ -169,7 +174,7 @@ enum sunxi_gpio_number {
#define SUN9I_GPN_R_RSB 3
-#ifdef CONFIG_SUNXI_NEW_PINCTRL
+#if defined(CONFIG_SUNXI_NEW_PINCTRL) || defined(CONFIG_SUNXI_A733_PINCTRL)
#define SUNXI_PINCTRL_BANK_SIZE 0x30
#define SUNXI_GPIO_DISABLE 0xf
#else
@@ -177,6 +182,9 @@ enum sunxi_gpio_number {
#define SUNXI_GPIO_DISABLE 0x7
#endif
+#define SUN60I_A733_PINCTRL_BANK_OFFSET 0x80
+#define SUN60I_A733_PINCTRL_BANK_SIZE 0x80
+
/* GPIO pin pull-up/down config */
#define SUNXI_GPIO_PULL_DISABLE 0
#define SUNXI_GPIO_PULL_UP 1
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 07/26] pinctrl: sunxi: a733: add initial support
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (5 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 06/26] gpio: a733: add initial support Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 08/26] clock: reset: " Alastair D'Silva
` (20 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
From: Yixun Lan <dlan@gentoo.org>
Add pin controller support for Allwinner A733 SoC, covering both main
PIO (allwinner,sun60i-a733-pinctrl) and R-PIO
(allwinner,sun60i-a733-r-pinctrl).
Because A733 main PIO starts Port B at offset 0x100 with a 0x80-byte
bank stride while R-PIO retains a 0x30-byte bank stride from offset 0x0,
track per-controller bank_offset and bank_size in sunxi_pinctrl_desc.
Signed-off-by: Yixun Lan <dlan@gentoo.org>
Link: https://lore.kernel.org/r/20260113-01-a733-soc-support-v3-7-f19dcd098f60@gentoo.org
[Alastair: add per-controller bank_size so R-PIO Bank M keeps 0x30
stride, select SUNXI_A733_PINCTRL and add UART0_PORT_B0 mux]
Co-developed-by: Alastair D'Silva <alastair@d-silva.org>
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
drivers/pinctrl/sunxi/Kconfig | 18 +++++
drivers/pinctrl/sunxi/pinctrl-sunxi.c | 98 ++++++++++++++++++++++++---
2 files changed, 106 insertions(+), 10 deletions(-)
diff --git a/drivers/pinctrl/sunxi/Kconfig b/drivers/pinctrl/sunxi/Kconfig
index 54314992299..c1cd8dc05d6 100644
--- a/drivers/pinctrl/sunxi/Kconfig
+++ b/drivers/pinctrl/sunxi/Kconfig
@@ -149,4 +149,22 @@ config PINCTRL_SUN55I_A523_R
default MACH_SUN55I_A523
select PINCTRL_SUNXI
+config PINCTRL_SUN60I_A733
+ bool "Support for the Allwinner A733 PIO"
+ depends on MACH_SUN60I_A733
+ default MACH_SUN60I_A733
+ select PINCTRL_SUNXI
+ select SUNXI_A733_PINCTRL if SUNXI_GPIO
+ help
+ Select this to enable the main pin controller (PIO) driver on
+ Allwinner A733 and T736 SoCs.
+
+config PINCTRL_SUN60I_A733_R
+ bool "Support for the Allwinner A733 R-PIO"
+ default MACH_SUN60I_A733
+ select PINCTRL_SUNXI
+ help
+ Select this to enable the PRCM pin controller (R-PIO) driver on
+ Allwinner A733 and T736 SoCs.
+
endif
diff --git a/drivers/pinctrl/sunxi/pinctrl-sunxi.c b/drivers/pinctrl/sunxi/pinctrl-sunxi.c
index ab4e88c9918..920c15ab864 100644
--- a/drivers/pinctrl/sunxi/pinctrl-sunxi.c
+++ b/drivers/pinctrl/sunxi/pinctrl-sunxi.c
@@ -33,12 +33,23 @@ struct sunxi_pinctrl_desc {
u8 num_functions;
u8 first_bank;
u8 num_banks;
+ u16 bank_offset;
+ u16 bank_size;
};
struct sunxi_pinctrl_plat {
void __iomem *base;
};
+static void *sunxi_pinctrl_bank_regs(struct sunxi_pinctrl_plat *plat,
+ const struct sunxi_pinctrl_desc *desc,
+ uint bank)
+{
+ u16 stride = desc->bank_size ? desc->bank_size : SUNXI_PINCTRL_BANK_SIZE;
+
+ return plat->base + desc->bank_offset + bank * stride;
+}
+
static int sunxi_pinctrl_get_pins_count(struct udevice *dev)
{
const struct sunxi_pinctrl_desc *desc = dev_get_priv(dev);
@@ -87,7 +98,7 @@ static int sunxi_pinctrl_pinmux_set(struct udevice *dev, uint pin_selector,
sunxi_pinctrl_get_function_name(dev, func_selector),
desc->functions[func_selector].mux);
- sunxi_gpio_set_cfgbank(plat->base + bank * SUNXI_PINCTRL_BANK_SIZE,
+ sunxi_gpio_set_cfgbank(sunxi_pinctrl_bank_regs(plat, desc, bank),
pin, desc->functions[func_selector].mux);
return 0;
@@ -100,20 +111,24 @@ static const struct pinconf_param sunxi_pinctrl_pinconf_params[] = {
{ "drive-strength", PIN_CONFIG_DRIVE_STRENGTH, 10 },
};
-static int sunxi_pinctrl_pinconf_set_pull(struct sunxi_pinctrl_plat *plat,
+static int sunxi_pinctrl_pinconf_set_pull(struct udevice *dev,
uint bank, uint pin, uint bias)
{
- void *regs = plat->base + bank * SUNXI_PINCTRL_BANK_SIZE;
+ const struct sunxi_pinctrl_desc *desc = dev_get_priv(dev);
+ struct sunxi_pinctrl_plat *plat = dev_get_plat(dev);
+ void *regs = sunxi_pinctrl_bank_regs(plat, desc, bank);
sunxi_gpio_set_pull_bank(regs, pin, bias);
return 0;
}
-static int sunxi_pinctrl_pinconf_set_drive(struct sunxi_pinctrl_plat *plat,
+static int sunxi_pinctrl_pinconf_set_drive(struct udevice *dev,
uint bank, uint pin, uint drive)
{
- void *regs = plat->base + bank * SUNXI_PINCTRL_BANK_SIZE;
+ const struct sunxi_pinctrl_desc *desc = dev_get_priv(dev);
+ struct sunxi_pinctrl_plat *plat = dev_get_plat(dev);
+ void *regs = sunxi_pinctrl_bank_regs(plat, desc, bank);
if (drive < 10 || drive > 40)
return -EINVAL;
@@ -127,7 +142,6 @@ static int sunxi_pinctrl_pinconf_set_drive(struct sunxi_pinctrl_plat *plat,
static int sunxi_pinctrl_pinconf_set(struct udevice *dev, uint pin_selector,
uint param, uint val)
{
- struct sunxi_pinctrl_plat *plat = dev_get_plat(dev);
int bank = pin_selector / SUNXI_GPIOS_PER_BANK;
int pin = pin_selector % SUNXI_GPIOS_PER_BANK;
@@ -135,9 +149,9 @@ static int sunxi_pinctrl_pinconf_set(struct udevice *dev, uint pin_selector,
case PIN_CONFIG_BIAS_DISABLE:
case PIN_CONFIG_BIAS_PULL_DOWN:
case PIN_CONFIG_BIAS_PULL_UP:
- return sunxi_pinctrl_pinconf_set_pull(plat, bank, pin, val);
+ return sunxi_pinctrl_pinconf_set_pull(dev, bank, pin, val);
case PIN_CONFIG_DRIVE_STRENGTH:
- return sunxi_pinctrl_pinconf_set_drive(plat, bank, pin, val);
+ return sunxi_pinctrl_pinconf_set_drive(dev, bank, pin, val);
}
return -EINVAL;
@@ -146,10 +160,12 @@ static int sunxi_pinctrl_pinconf_set(struct udevice *dev, uint pin_selector,
static int sunxi_pinctrl_get_pin_muxing(struct udevice *dev, uint pin_selector,
char *buf, int size)
{
+ const struct sunxi_pinctrl_desc *desc = dev_get_priv(dev);
struct sunxi_pinctrl_plat *plat = dev_get_plat(dev);
int bank = pin_selector / SUNXI_GPIOS_PER_BANK;
int pin = pin_selector % SUNXI_GPIOS_PER_BANK;
- int mux = sunxi_gpio_get_cfgbank(plat->base + bank * SUNXI_PINCTRL_BANK_SIZE, pin);
+ void *regs = sunxi_pinctrl_bank_regs(plat, desc, bank);
+ int mux = sunxi_gpio_get_cfgbank(regs, pin);
switch (mux) {
case SUNXI_GPIO_INPUT:
@@ -207,7 +223,7 @@ static int sunxi_pinctrl_bind(struct udevice *dev)
if (!gpio_plat)
return -ENOMEM;
- gpio_plat->regs = plat->base + i * SUNXI_PINCTRL_BANK_SIZE;
+ gpio_plat->regs = sunxi_pinctrl_bank_regs(plat, desc, i);
gpio_plat->bank_name[0] = 'P';
gpio_plat->bank_name[1] = 'A' + desc->first_bank + i;
gpio_plat->bank_name[2] = '\0';
@@ -784,6 +800,41 @@ static const struct sunxi_pinctrl_desc __maybe_unused sun55i_a523_pinctrl_desc =
.num_banks = 11,
};
+static const struct sunxi_pinctrl_function sun60i_a733_pinctrl_functions[] = {
+ { "gpio_in", 0 },
+ { "gpio_out", 1 },
+ { "gmac0", 5 }, /* PH0-PH15 */
+ { "gmac1", 5 }, /* PJ0-PJ15 */
+ { "mmc0", 2 }, /* PF0-PF5 */
+ { "mmc1", 2 }, /* PG0-PG5 */
+ { "mmc2", 3 }, /* PC0, PC1, PC5, PC6, PC8-PC11, PC13-PC16 */
+ { "mmc3", 4 }, /* PC0, PC1, PC5, PC6, PC8-PC11, PC13-PC16 */
+ { "pcie", 6 }, /* PD20 */
+ { "spi0", 5 }, /* PC2-PC4, PC7, PC12 */
+#if IS_ENABLED(CONFIG_UART0_PORT_F)
+ { "uart0", 3 }, /* PF2, PF4 */
+#elif IS_ENABLED(CONFIG_UART0_PORT_B0)
+ { "uart0", 3 }, /* PB0, PB1 */
+#else
+ { "uart0", 2 }, /* PB9, PB10 */
+#endif
+ { "uart1", 2 }, /* PG6, PG7 */
+};
+
+/*
+ * A733 main PIO omits Port A physically; Port B begins at offset 0x100 with a
+ * 0x80-byte bank stride, so .first_bank = SUNXI_GPIO_A reserves a virtual PA
+ * slot at +0x80 to keep linear bank indices aligned with Port B..K.
+ */
+static const struct sunxi_pinctrl_desc __maybe_unused sun60i_a733_pinctrl_desc = {
+ .functions = sun60i_a733_pinctrl_functions,
+ .num_functions = ARRAY_SIZE(sun60i_a733_pinctrl_functions),
+ .first_bank = SUNXI_GPIO_A,
+ .num_banks = 11,
+ .bank_offset = SUN60I_A733_PINCTRL_BANK_OFFSET,
+ .bank_size = SUN60I_A733_PINCTRL_BANK_SIZE,
+};
+
static const struct sunxi_pinctrl_function sun50i_h616_r_pinctrl_functions[] = {
{ "gpio_in", 0 },
{ "gpio_out", 1 },
@@ -849,6 +900,21 @@ static const struct sunxi_pinctrl_desc __maybe_unused sun55i_a523_r_pinctrl_desc
.num_banks = 2,
};
+static const struct sunxi_pinctrl_function sun60i_a733_r_pinctrl_functions[] = {
+ { "gpio_in", 0 },
+ { "gpio_out", 1 },
+ { "r_i2c0", 2 }, /* PL0-PL1 */
+ { "r_uart0", 3 }, /* PL2-PL3 */
+ { "r_uart1", 2 }, /* PL2-PL3 */
+};
+
+static const struct sunxi_pinctrl_desc __maybe_unused sun60i_a733_r_pinctrl_desc = {
+ .functions = sun60i_a733_r_pinctrl_functions,
+ .num_functions = ARRAY_SIZE(sun60i_a733_r_pinctrl_functions),
+ .first_bank = SUNXI_GPIO_L,
+ .num_banks = 2,
+};
+
static const struct udevice_id sunxi_pinctrl_ids[] = {
#ifdef CONFIG_PINCTRL_SUNIV_F1C100S
{
@@ -1035,6 +1101,18 @@ static const struct udevice_id sunxi_pinctrl_ids[] = {
.compatible = "allwinner,sun55i-a523-r-pinctrl",
.data = (ulong)&sun55i_a523_r_pinctrl_desc,
},
+#endif
+#ifdef CONFIG_PINCTRL_SUN60I_A733
+ {
+ .compatible = "allwinner,sun60i-a733-pinctrl",
+ .data = (ulong)&sun60i_a733_pinctrl_desc,
+ },
+#endif
+#ifdef CONFIG_PINCTRL_SUN60I_A733_R
+ {
+ .compatible = "allwinner,sun60i-a733-r-pinctrl",
+ .data = (ulong)&sun60i_a733_r_pinctrl_desc,
+ },
#endif
{}
};
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 08/26] clock: reset: a733: add initial support
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (6 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 07/26] pinctrl: sunxi: " Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 09/26] clock: a733: implement support for mmc/uart Alastair D'Silva
` (19 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
From: Yixun Lan <dlan@gentoo.org>
Implement clock gate and reset driver support for the Allwinner A733
CCU and PRCM (R-CCU), using the v6 upstream Linux DT binding headers.
Signed-off-by: Yixun Lan <dlan@gentoo.org>
Link: https://lore.kernel.org/r/20260113-01-a733-soc-support-v3-5-f19dcd098f60@gentoo.org
[Alastair: update to v6 allwinner,sun60i-a733-ccu.h bindings, fix I2C2/3
reset offsets, add MMC0-2/USB OTG/R_SPI gates, constify tables]
Co-developed-by: Alastair D'Silva <alastair@d-silva.org>
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
drivers/clk/sunxi/Kconfig | 14 ++++
drivers/clk/sunxi/Makefile | 2 +
drivers/clk/sunxi/clk_a733.c | 132 +++++++++++++++++++++++++++++++++
drivers/clk/sunxi/clk_a733_r.c | 45 +++++++++++
drivers/clk/sunxi/clk_sunxi.c | 10 +++
5 files changed, 203 insertions(+)
create mode 100644 drivers/clk/sunxi/clk_a733.c
create mode 100644 drivers/clk/sunxi/clk_a733_r.c
diff --git a/drivers/clk/sunxi/Kconfig b/drivers/clk/sunxi/Kconfig
index 1c1cc82719c..9858f4546ea 100644
--- a/drivers/clk/sunxi/Kconfig
+++ b/drivers/clk/sunxi/Kconfig
@@ -143,4 +143,18 @@ config CLK_SUN55I_A523_R
This enables common clock driver support for the PRCM
in Allwinner A523/T527 SoCs.
+config CLK_SUN60I_A733
+ bool "Clock driver for Allwinner A733/T736"
+ default MACH_SUN60I_A733
+ help
+ This enables common clock driver support for platforms based
+ on Allwinner A733/T736 SoC.
+
+config CLK_SUN60I_A733_R
+ bool "Clock driver for Allwinner A733 generation PRCM"
+ default MACH_SUN60I_A733
+ help
+ This enables common clock driver support for the PRCM
+ in Allwinner A733/T736 SoCs.
+
endif # CLK_SUNXI
diff --git a/drivers/clk/sunxi/Makefile b/drivers/clk/sunxi/Makefile
index 93b542cebcd..943e47612ce 100644
--- a/drivers/clk/sunxi/Makefile
+++ b/drivers/clk/sunxi/Makefile
@@ -27,3 +27,5 @@ obj-$(CONFIG_CLK_SUN50I_A64) += clk_a64.o
obj-$(CONFIG_CLK_SUN50I_A100) += clk_a100.o
obj-$(CONFIG_CLK_SUN55I_A523) += clk_a523.o
obj-$(CONFIG_CLK_SUN55I_A523_R) += clk_a523_r.o
+obj-$(CONFIG_CLK_SUN60I_A733) += clk_a733.o
+obj-$(CONFIG_CLK_SUN60I_A733_R) += clk_a733_r.o
diff --git a/drivers/clk/sunxi/clk_a733.c b/drivers/clk/sunxi/clk_a733.c
new file mode 100644
index 00000000000..5c942fd5d58
--- /dev/null
+++ b/drivers/clk/sunxi/clk_a733.c
@@ -0,0 +1,132 @@
+// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+/*
+ * Copyright (C) 2025-2026 Yixun Lan <dlan@gentoo.org>
+ * Copyright (C) 2026 Alastair D'Silva <alastair@d-silva.org>
+ */
+
+#include <clk/sunxi.h>
+#include <linux/bitops.h>
+#include <linux/kernel.h>
+
+#include <dt-bindings/clock/allwinner,sun60i-a733-ccu.h>
+#include <dt-bindings/reset/allwinner,sun60i-a733-ccu.h>
+
+static const struct ccu_clk_gate a733_gates[] = {
+ [CLK_SYS_24M] = GATE_DUMMY,
+ [CLK_PLL_PERIPH0_200M] = GATE_DUMMY,
+ [CLK_PLL_PERIPH0_300M] = GATE_DUMMY,
+ [CLK_PLL_PERIPH1_300M] = GATE_DUMMY,
+ [CLK_APB1] = GATE_DUMMY,
+ [CLK_APB_UART] = GATE_DUMMY,
+
+ [CLK_MBUS_GMAC0] = GATE(0x05e4, BIT(11)),
+ [CLK_BUS_CE] = GATE(0x0ac4, BIT(0)),
+
+ [CLK_MMC0] = GATE(0x0d00, BIT(31)),
+ [CLK_BUS_MMC0] = GATE(0x0d0c, BIT(0)),
+ [CLK_MMC1] = GATE(0x0d10, BIT(31)),
+ [CLK_BUS_MMC1] = GATE(0x0d1c, BIT(0)),
+ [CLK_MMC2] = GATE(0x0d20, BIT(31)),
+ [CLK_BUS_MMC2] = GATE(0x0d2c, BIT(0)),
+
+ [CLK_BUS_UART0] = GATE(0x0e00, BIT(0)),
+ [CLK_BUS_UART1] = GATE(0x0e04, BIT(0)),
+ [CLK_BUS_UART2] = GATE(0x0e08, BIT(0)),
+ [CLK_BUS_UART3] = GATE(0x0e0c, BIT(0)),
+ [CLK_BUS_UART4] = GATE(0x0e10, BIT(0)),
+ [CLK_BUS_UART5] = GATE(0x0e14, BIT(0)),
+ [CLK_BUS_UART6] = GATE(0x0e18, BIT(0)),
+
+ [CLK_BUS_I2C0] = GATE(0x0e80, BIT(0)),
+ [CLK_BUS_I2C1] = GATE(0x0e84, BIT(0)),
+ [CLK_BUS_I2C2] = GATE(0x0e88, BIT(0)),
+ [CLK_BUS_I2C3] = GATE(0x0e8c, BIT(0)),
+ [CLK_BUS_I2C4] = GATE(0x0e90, BIT(0)),
+ [CLK_BUS_I2C5] = GATE(0x0e94, BIT(0)),
+ [CLK_BUS_I2C6] = GATE(0x0e98, BIT(0)),
+ [CLK_BUS_I2C7] = GATE(0x0e9c, BIT(0)),
+ [CLK_BUS_I2C8] = GATE(0x0ea0, BIT(0)),
+ [CLK_BUS_I2C9] = GATE(0x0ea4, BIT(0)),
+ [CLK_BUS_I2C10] = GATE(0x0ea8, BIT(0)),
+ [CLK_BUS_I2C11] = GATE(0x0eac, BIT(0)),
+ [CLK_BUS_I2C12] = GATE(0x0eb0, BIT(0)),
+
+ [CLK_SPI0] = GATE(0x0f00, BIT(31)),
+ [CLK_BUS_SPI0] = GATE(0x0f04, BIT(0)),
+ [CLK_SPI1] = GATE(0x0f08, BIT(31)),
+ [CLK_BUS_SPI1] = GATE(0x0f0c, BIT(0)),
+ [CLK_SPI2] = GATE(0x0f10, BIT(31)),
+ [CLK_BUS_SPI2] = GATE(0x0f14, BIT(0)),
+
+ [CLK_GPADC0_24M] = GATE(0x0fc0, BIT(31)),
+ [CLK_BUS_GPADC0] = GATE(0x0fc4, BIT(0)),
+ [CLK_BUS_THS0] = GATE(0x0fe4, BIT(0)),
+ [CLK_BUS_LRADC] = GATE(0x1024, BIT(0)),
+
+ [CLK_USB_OHCI0] = GATE(0x1300, BIT(31)),
+ [CLK_BUS_OHCI0] = GATE(0x1304, BIT(0)),
+ [CLK_BUS_EHCI0] = GATE(0x1304, BIT(4)),
+ [CLK_BUS_OTG] = GATE(0x1304, BIT(8)),
+ [CLK_USB_OHCI1] = GATE(0x1308, BIT(31)),
+ [CLK_BUS_OHCI1] = GATE(0x130c, BIT(0)),
+ [CLK_BUS_EHCI1] = GATE(0x130c, BIT(4)),
+
+ [CLK_GMAC0_PHY] = GATE(0x1410, BIT(31)),
+ [CLK_BUS_GMAC0] = GATE(0x141c, BIT(0)),
+};
+
+static const struct ccu_reset a733_resets[] = {
+ [RST_BUS_CE] = RESET(0x0ac4, BIT(16)),
+
+ [RST_BUS_MMC0] = RESET(0x0d0c, BIT(16)),
+ [RST_BUS_MMC1] = RESET(0x0d1c, BIT(16)),
+ [RST_BUS_MMC2] = RESET(0x0d2c, BIT(16)),
+
+ [RST_BUS_UART0] = RESET(0x0e00, BIT(16)),
+ [RST_BUS_UART1] = RESET(0x0e04, BIT(16)),
+ [RST_BUS_UART2] = RESET(0x0e08, BIT(16)),
+ [RST_BUS_UART3] = RESET(0x0e0c, BIT(16)),
+ [RST_BUS_UART4] = RESET(0x0e10, BIT(16)),
+ [RST_BUS_UART5] = RESET(0x0e14, BIT(16)),
+ [RST_BUS_UART6] = RESET(0x0e18, BIT(16)),
+
+ [RST_BUS_I2C0] = RESET(0x0e80, BIT(16)),
+ [RST_BUS_I2C1] = RESET(0x0e84, BIT(16)),
+ [RST_BUS_I2C2] = RESET(0x0e88, BIT(16)),
+ [RST_BUS_I2C3] = RESET(0x0e8c, BIT(16)),
+ [RST_BUS_I2C4] = RESET(0x0e90, BIT(16)),
+ [RST_BUS_I2C5] = RESET(0x0e94, BIT(16)),
+ [RST_BUS_I2C6] = RESET(0x0e98, BIT(16)),
+ [RST_BUS_I2C7] = RESET(0x0e9c, BIT(16)),
+ [RST_BUS_I2C8] = RESET(0x0ea0, BIT(16)),
+ [RST_BUS_I2C9] = RESET(0x0ea4, BIT(16)),
+ [RST_BUS_I2C10] = RESET(0x0ea8, BIT(16)),
+ [RST_BUS_I2C11] = RESET(0x0eac, BIT(16)),
+ [RST_BUS_I2C12] = RESET(0x0eb0, BIT(16)),
+
+ [RST_BUS_SPI0] = RESET(0x0f04, BIT(16)),
+ [RST_BUS_SPI1] = RESET(0x0f0c, BIT(16)),
+ [RST_BUS_SPI2] = RESET(0x0f14, BIT(16)),
+
+ [RST_BUS_GPADC0] = RESET(0x0fc4, BIT(16)),
+ [RST_BUS_THS0] = RESET(0x0fe4, BIT(16)),
+ [RST_BUS_LRADC] = RESET(0x1024, BIT(16)),
+
+ [RST_USB_PHY0] = RESET(0x1300, BIT(30)),
+ [RST_BUS_OHCI0] = RESET(0x1304, BIT(16)),
+ [RST_BUS_EHCI0] = RESET(0x1304, BIT(20)),
+ [RST_BUS_OTG] = RESET(0x1304, BIT(24)),
+ [RST_USB_PHY1] = RESET(0x1308, BIT(30)),
+ [RST_BUS_OHCI1] = RESET(0x130c, BIT(16)),
+ [RST_BUS_EHCI1] = RESET(0x130c, BIT(20)),
+
+ [RST_BUS_GMAC0] = RESET(0x141c, BIT(16)),
+ [RST_BUS_GMAC0_AXI] = RESET(0x141c, BIT(17)),
+};
+
+const struct ccu_desc a733_ccu_desc = {
+ .gates = a733_gates,
+ .resets = a733_resets,
+ .num_gates = ARRAY_SIZE(a733_gates),
+ .num_resets = ARRAY_SIZE(a733_resets),
+};
diff --git a/drivers/clk/sunxi/clk_a733_r.c b/drivers/clk/sunxi/clk_a733_r.c
new file mode 100644
index 00000000000..ae02dd21de5
--- /dev/null
+++ b/drivers/clk/sunxi/clk_a733_r.c
@@ -0,0 +1,45 @@
+// SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+/*
+ * Copyright (C) 2025-2026 Yixun Lan <dlan@gentoo.org>
+ * Copyright (C) 2026 Alastair D'Silva <alastair@d-silva.org>
+ */
+
+#include <clk/sunxi.h>
+#include <linux/bitops.h>
+#include <linux/kernel.h>
+
+#include <dt-bindings/clock/allwinner,sun60i-a733-r-ccu.h>
+#include <dt-bindings/reset/allwinner,sun60i-a733-r-ccu.h>
+
+static const struct ccu_clk_gate a733_r_gates[] = {
+ [CLK_R_AHB] = GATE_DUMMY,
+ [CLK_R_APB0] = GATE_DUMMY,
+ [CLK_R_APB1] = GATE_DUMMY,
+ [CLK_R_SPI] = GATE(0x150, BIT(31)),
+ [CLK_BUS_R_SPI] = GATE(0x15c, BIT(0)),
+ [CLK_BUS_R_UART0] = GATE(0x18c, BIT(0)),
+ [CLK_BUS_R_UART1] = GATE(0x18c, BIT(1)),
+ [CLK_BUS_R_I2C0] = GATE(0x19c, BIT(0)),
+ [CLK_BUS_R_I2C1] = GATE(0x19c, BIT(1)),
+ [CLK_BUS_R_I2C2] = GATE(0x19c, BIT(2)),
+ [CLK_BUS_R_RTC] = GATE(0x20c, BIT(0)),
+ [CLK_BUS_R_CPUCFG] = GATE(0x22c, BIT(0)),
+};
+
+static const struct ccu_reset a733_r_resets[] = {
+ [RST_BUS_R_SPI] = RESET(0x15c, BIT(16)),
+ [RST_BUS_R_UART0] = RESET(0x18c, BIT(16)),
+ [RST_BUS_R_UART1] = RESET(0x18c, BIT(17)),
+ [RST_BUS_R_I2C0] = RESET(0x19c, BIT(16)),
+ [RST_BUS_R_I2C1] = RESET(0x19c, BIT(17)),
+ [RST_BUS_R_I2C2] = RESET(0x19c, BIT(18)),
+ [RST_BUS_R_RTC] = RESET(0x20c, BIT(16)),
+ [RST_BUS_R_CPUCFG] = RESET(0x22c, BIT(16)),
+};
+
+const struct ccu_desc a733_r_ccu_desc = {
+ .gates = a733_r_gates,
+ .resets = a733_r_resets,
+ .num_gates = ARRAY_SIZE(a733_r_gates),
+ .num_resets = ARRAY_SIZE(a733_r_resets),
+};
diff --git a/drivers/clk/sunxi/clk_sunxi.c b/drivers/clk/sunxi/clk_sunxi.c
index 046d5d1605a..6d55b4598a5 100644
--- a/drivers/clk/sunxi/clk_sunxi.c
+++ b/drivers/clk/sunxi/clk_sunxi.c
@@ -128,6 +128,8 @@ extern const struct ccu_desc r40_ccu_desc;
extern const struct ccu_desc v3s_ccu_desc;
extern const struct ccu_desc a523_ccu_desc;
extern const struct ccu_desc a523_r_ccu_desc;
+extern const struct ccu_desc a733_ccu_desc;
+extern const struct ccu_desc a733_r_ccu_desc;
static const struct udevice_id sunxi_clk_ids[] = {
#ifdef CONFIG_CLK_SUN4I_A10
@@ -233,6 +235,14 @@ static const struct udevice_id sunxi_clk_ids[] = {
#ifdef CONFIG_CLK_SUN55I_A523_R
{ .compatible = "allwinner,sun55i-a523-r-ccu",
.data = (ulong)&a523_r_ccu_desc },
+#endif
+#ifdef CONFIG_CLK_SUN60I_A733
+ { .compatible = "allwinner,sun60i-a733-ccu",
+ .data = (ulong)&a733_ccu_desc },
+#endif
+#ifdef CONFIG_CLK_SUN60I_A733_R
+ { .compatible = "allwinner,sun60i-a733-r-ccu",
+ .data = (ulong)&a733_r_ccu_desc },
#endif
{ }
};
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 09/26] clock: a733: implement support for mmc/uart
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (7 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 08/26] clock: reset: " Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 10/26] sunxi: mmc: enable support for A733 Alastair D'Silva
` (18 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
From: Yixun Lan <dlan@gentoo.org>
Add peripheral base addresses and SPL clock initialization support for
UART, I2C, MMC, MBUS, and ZQ/pad resistor calibration on the Allwinner
A733 SoC.
Link: https://lore.kernel.org/r/20260113-01-a733-soc-support-v3-4-f19dcd098f60@gentoo.org
Signed-off-by: Yixun Lan <dlan@gentoo.org>
[Alastair: fix CPUCFG/WDT bases and PLL6 factor, add MBUS/RESCAL,
I2C BGR, and PLL1 guard]
Co-developed-by: Alastair D'Silva <alastair@d-silva.org>
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
.../include/asm/arch-sunxi/clock_sun50i_h6.h | 19 +++++
.../include/asm/arch-sunxi/cpu_sunxi_ncat2.h | 42 ++++++++--
arch/arm/mach-sunxi/clock_sun50i_h6.c | 81 +++++++++++++++++--
3 files changed, 128 insertions(+), 14 deletions(-)
diff --git a/arch/arm/include/asm/arch-sunxi/clock_sun50i_h6.h b/arch/arm/include/asm/arch-sunxi/clock_sun50i_h6.h
index 8d1c7c18548..2443cf36eb3 100644
--- a/arch/arm/include/asm/arch-sunxi/clock_sun50i_h6.h
+++ b/arch/arm/include/asm/arch-sunxi/clock_sun50i_h6.h
@@ -29,13 +29,30 @@
#define CCU_NAND0_CLK_CFG 0x810
#define CCU_NAND1_CLK_CFG 0x814
#define CCU_H6_NAND_GATE_RESET 0x82c
+#if defined(CONFIG_MACH_SUN60I_A733)
+#define CCU_MMC0_CLK_CFG 0x0d00
+#define CCU_MMC1_CLK_CFG 0x0d10
+#define CCU_MMC2_CLK_CFG 0x0d20
+#else
#define CCU_MMC0_CLK_CFG 0x830
#define CCU_MMC1_CLK_CFG 0x834
#define CCU_MMC2_CLK_CFG 0x838
+#endif
#define CCU_H6_MMC_GATE_RESET 0x84c
#define CCU_H6_UART_GATE_RESET 0x90c
#define CCU_H6_I2C_GATE_RESET 0x91c
+/* A733 CCU register offsets */
+#define CCU_A733_PLL6_CFG 0x0a0
+#define CCU_A733_APB_UART_CFG 0x538
+#define CCU_A733_MMC_CLK_STRIDE 0x10
+#define CCU_A733_MMC_BGR_BASE 0x0d0c
+#define CCU_A733_MMC_BGR_REG(n) (CCU_A733_MMC_BGR_BASE + (n) * CCU_A733_MMC_CLK_STRIDE)
+#define CCU_A733_UART_BGR_BASE 0x0e00
+#define CCU_A733_UART_BGR_REG(n) (CCU_A733_UART_BGR_BASE + (n) * 4)
+#define CCU_A733_I2C_BGR_BASE 0x0e80
+#define CCU_A733_I2C_BGR_REG(n) (CCU_A733_I2C_BGR_BASE + (n) * 4)
+
/* A523 CPU PLL offsets */
#define CPC_CPUA_PLL_CTRL 0x04
#define CPC_DSU_PLL_CTRL 0x08
@@ -69,6 +86,8 @@
#define CCM_PLL5_CTRL_DIV2(div0) ((div0) << 1)
/* pll6 bit field */
+#define CCM_A733_PLL6_CTRL_P0_SHIFT 20
+#define CCM_A733_PLL6_CTRL_P0_MASK (0x7 << CCM_A733_PLL6_CTRL_P0_SHIFT)
#define CCM_PLL6_CTRL_P0_SHIFT 16
#define CCM_PLL6_CTRL_P0_MASK (0x7 << CCM_PLL6_CTRL_P0_SHIFT)
#define CCM_PLL6_CTRL_N_SHIFT 8
diff --git a/arch/arm/include/asm/arch-sunxi/cpu_sunxi_ncat2.h b/arch/arm/include/asm/arch-sunxi/cpu_sunxi_ncat2.h
index bcfdc0a41c5..260fed24741 100644
--- a/arch/arm/include/asm/arch-sunxi/cpu_sunxi_ncat2.h
+++ b/arch/arm/include/asm/arch-sunxi/cpu_sunxi_ncat2.h
@@ -7,19 +7,32 @@
#ifndef _SUNXI_CPU_SUNXI_NCAT2_H
#define _SUNXI_CPU_SUNXI_NCAT2_H
+#if defined(CONFIG_MACH_SUN60I_A733)
+#define SUNXI_CCM_BASE 0x02002000
+#define SUNXI_WDT_BASE 0x02050000
+#define SUNXI_TIMER_BASE 0x02052000
+#define SUNXI_TWI0_BASE 0x02510000
+#define SUNXI_TWI1_BASE 0x02511000
+#define SUNXI_TWI2_BASE 0x02512000
+#define SUNXI_TWI3_BASE 0x02513000
+#else
#define SUNXI_CCM_BASE 0x02001000
#define SUNXI_TIMER_BASE 0x02050000
-
#define SUNXI_TWI0_BASE 0x02502000
#define SUNXI_TWI1_BASE 0x02502400
#define SUNXI_TWI2_BASE 0x02502800
#define SUNXI_TWI3_BASE 0x02502C00
+#endif
#define SUNXI_SRAMC_BASE 0x03000000
/* SID address space starts at 0x03006000, but e-fuse is at offset 0x200 */
#define SUNXI_SIDC_BASE 0x03006000
#define SUNXI_SID_BASE 0x03006200
+#ifdef CONFIG_MACH_SUN60I_A733
+#define SUNXI_GIC600_BASE 0x03400000
+#else
#define SUNXI_GIC400_BASE 0x03020000
+#endif
#ifdef CONFIG_MACH_SUN55I_A523
#define SUNXI_DRAM_COM_BASE 0x03120000
@@ -31,24 +44,37 @@
#define SUNXI_MMC1_BASE 0x04021000
#define SUNXI_MMC2_BASE 0x04022000
-#ifndef CONFIG_MACH_SUN55I_A523
-#define SUNXI_R_CPUCFG_BASE 0x07000400
-#endif
#define SUNXI_PRCM_BASE 0x07010000
+#if defined(CONFIG_MACH_SUN60I_A733)
+#define SUNXI_R_WDOG_BASE 0x07021000
+#else
#define SUNXI_R_WDOG_BASE 0x07020400
-#ifdef CONFIG_MACH_SUN55I_A523
+#endif
+#if defined(CONFIG_MACH_SUN55I_A523) || defined(CONFIG_MACH_SUN60I_A733)
#define SUNXI_R_CPUCFG_BASE 0x07050000
+#else
+#define SUNXI_R_CPUCFG_BASE 0x07000400
#endif
+#if defined(CONFIG_MACH_SUN60I_A733)
+#define SUNXI_R_TWI_BASE 0x07083000
+#else
#define SUNXI_R_TWI_BASE 0x07081400
+#endif
#define SUNXI_RTC_BASE 0x07090000
-#ifdef CONFIG_MACH_SUN55I_A523
+#if defined(CONFIG_MACH_SUN60I_A733)
+#define SUNXI_CPUCFG_BASE 0x08000000
+#define SUNXI_CPU_DA_DDR_CTRL_REG (SUNXI_CPUCFG_BASE + 0x0200)
+#define SUNXI_CLU0_CPU0_CTRL_REG (SUNXI_CPUCFG_BASE + 0x1000)
+#define SUNXI_CLUSTER_CFG_BASE 0x08860000
+#define SUNXI_CPU_PLL_CFG_BASE 0x08870000
+#elif defined(CONFIG_MACH_SUN55I_A523)
#define SUNXI_CPUCFG_BASE 0x08815000
+#define SUNXI_CPU_PLL_CFG_BASE 0x08817000
#else
#define SUNXI_CPUCFG_BASE 0x09010000
-#endif
-
#define SUNXI_CPU_PLL_CFG_BASE 0x08817000
+#endif
#ifndef __ASSEMBLY__
void sunxi_board_init(void);
diff --git a/arch/arm/mach-sunxi/clock_sun50i_h6.c b/arch/arm/mach-sunxi/clock_sun50i_h6.c
index 80004f13a1e..517fdcb56ea 100644
--- a/arch/arm/mach-sunxi/clock_sun50i_h6.c
+++ b/arch/arm/mach-sunxi/clock_sun50i_h6.c
@@ -11,8 +11,47 @@
#endif
#ifdef CONFIG_XPL_BUILD
+#ifdef CONFIG_MACH_SUN60I_A733
+#define SUNXI_CCU_MBUS_MAT_CLK_GATING (SUNXI_CCM_BASE + 0x05c0UL)
+#define SUNXI_CCU_MBUS_MAT_CLK_GATING_VAL 0x110103ffU
+#define SUNXI_CCU_MBUS_MAT_CLK_ENABLE (SUNXI_CCM_BASE + 0x05e0UL)
+#define SUNXI_CCU_MBUS_MAT_CLK_ENABLE_VAL 0xf1055803U
+#define SUNXI_CCU_MBUS_MASTER_GATE (SUNXI_CCM_BASE + 0x05e4UL)
+#define SUNXI_CCU_MBUS_MASTER_GATE_VAL 0x0007ffffU
+
+#define SUNXI_SYSCTRL_RESCAL_CTRL (SUNXI_SRAMC_BASE + 0x0160UL)
+#define SUNXI_SYSCTRL_RESCAL_CTRL_EN BIT(8)
+#define SUNXI_SYSCTRL_RESCAL_CAL (SUNXI_SRAMC_BASE + 0x0168UL)
+#define SUNXI_SYSCTRL_RESCAL_CAL_MASK 0xffU
+
+/*
+ * On A733, the BROM configures PLL_PERIPH0 and APB/AXI root dividers before
+ * SPL entry, while TF-A BL31 (sunxi_security_setup) configures TrustZone SPC,
+ * CCU/PRCM security switches, and IOMMU bypass before Non-Secure EL2 entry.
+ * SPL only needs to ungate the MBUS matrix clocks and enable SYSCTRL ZQ/pad
+ * resistor calibration prior to LPDDR5 training.
+ */
+static void sunxi_a733_bus_and_rescal_init(void)
+{
+ writel(SUNXI_CCU_MBUS_MAT_CLK_GATING_VAL,
+ (void __iomem *)SUNXI_CCU_MBUS_MAT_CLK_GATING);
+ writel(SUNXI_CCU_MBUS_MAT_CLK_ENABLE_VAL,
+ (void __iomem *)SUNXI_CCU_MBUS_MAT_CLK_ENABLE);
+ writel(SUNXI_CCU_MBUS_MASTER_GATE_VAL,
+ (void __iomem *)SUNXI_CCU_MBUS_MASTER_GATE);
+
+ setbits_le32((void __iomem *)SUNXI_SYSCTRL_RESCAL_CTRL,
+ SUNXI_SYSCTRL_RESCAL_CTRL_EN);
+ clrbits_le32((void __iomem *)SUNXI_SYSCTRL_RESCAL_CAL,
+ SUNXI_SYSCTRL_RESCAL_CAL_MASK);
+}
+#endif
+
void clock_init_safe(void)
{
+#ifdef CONFIG_MACH_SUN60I_A733
+ sunxi_a733_bus_and_rescal_init();
+#else
void *const ccm = (void *)SUNXI_CCM_BASE;
void *const prcm = (void *)SUNXI_PRCM_BASE;
@@ -72,12 +111,26 @@ void clock_init_safe(void)
writel(MBUS_CLK_SRC_PLL6X2 | MBUS_CLK_M(3),
ccm + CCU_H6_MBUS_CFG);
}
+#endif
}
void clock_init_uart(void)
{
void *const ccm = (void *)SUNXI_CCM_BASE;
+ if (IS_ENABLED(CONFIG_MACH_SUN60I_A733)) {
+ /* A733: APB_UART clock source is OSC24M / 1 */
+ writel(APB2_CLK_SRC_OSC24M |
+ APB2_CLK_RATE_N_1 |
+ APB2_CLK_RATE_M(1),
+ ccm + CCU_A733_APB_UART_CFG);
+
+ /* Deassert reset and enable clock gate for console UART */
+ setbits_le32(ccm + CCU_A733_UART_BGR_REG(CONFIG_CONS_INDEX - 1),
+ BIT(RESET_SHIFT) | BIT(GATE_SHIFT));
+ return;
+ }
+
/* uart clock source is apb2 */
writel(APB2_CLK_SRC_OSC24M|
APB2_CLK_RATE_N_1|
@@ -199,6 +252,10 @@ void clock_set_pll1(unsigned int clk)
clk /= 24000000;
+ /* BROM and TF-A configure cluster PLLs on A733 */
+ if (IS_ENABLED(CONFIG_MACH_SUN60I_A733))
+ return;
+
if (IS_ENABLED(CONFIG_MACH_SUN55I_A523))
clock_a523_set_cpu_plls(clk);
else
@@ -217,6 +274,9 @@ int clock_twi_onoff(int port, int state)
if (port == 5) {
shift = 0;
ptr = prcm + CCU_PRCM_I2C_GATE_RESET;
+ } else if (IS_ENABLED(CONFIG_MACH_SUN60I_A733)) {
+ shift = 0;
+ ptr = ccm + CCU_A733_I2C_BGR_REG(port);
} else {
shift = port;
ptr = ccm + CCU_H6_I2C_GATE_RESET;
@@ -236,13 +296,22 @@ int clock_twi_onoff(int port, int state)
unsigned int clock_get_pll6(void)
{
void *const ccm = (void *)SUNXI_CCM_BASE;
- uint32_t rval = readl(ccm + CCU_H6_PLL6_CFG);
- int n = ((rval & CCM_PLL6_CTRL_N_MASK) >> CCM_PLL6_CTRL_N_SHIFT) + 1;
- int div2 = ((rval & CCM_PLL6_CTRL_DIV2_MASK) >>
- CCM_PLL6_CTRL_DIV2_SHIFT) + 1;
- int div1, m;
+ int div1, div2, m, n;
+ u32 rval;
+
+ if (IS_ENABLED(CONFIG_MACH_SUN60I_A733))
+ rval = readl(ccm + CCU_A733_PLL6_CFG);
+ else
+ rval = readl(ccm + CCU_H6_PLL6_CFG);
+
+ n = ((rval & CCM_PLL6_CTRL_N_MASK) >> CCM_PLL6_CTRL_N_SHIFT) + 1;
+ div2 = ((rval & CCM_PLL6_CTRL_DIV2_MASK) >>
+ CCM_PLL6_CTRL_DIV2_SHIFT) + 1;
- if (IS_ENABLED(CONFIG_SUNXI_GEN_NCAT2)) {
+ if (IS_ENABLED(CONFIG_MACH_SUN60I_A733)) {
+ div1 = ((rval & CCM_A733_PLL6_CTRL_P0_MASK) >>
+ CCM_A733_PLL6_CTRL_P0_SHIFT) + 1;
+ } else if (IS_ENABLED(CONFIG_SUNXI_GEN_NCAT2)) {
div1 = ((rval & CCM_PLL6_CTRL_P0_MASK) >>
CCM_PLL6_CTRL_P0_SHIFT) + 1;
} else {
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 10/26] sunxi: mmc: enable support for A733
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (8 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 09/26] clock: a733: implement support for mmc/uart Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 11/26] sunxi: a733: uart: setup serial console Alastair D'Silva
` (17 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
From: Yixun Lan <dlan@gentoo.org>
Add MMC clock and SPL bus-gate/reset support for the Allwinner A733 MMC
controllers, where SMHC0_CLK_REG starts at CCU offset 0x0d00 with a
per-channel stride of 0x10 bytes and MMC2 uses PLL_PERIPH0_800M.
Link: https://lore.kernel.org/r/20260113-01-a733-soc-support-v3-8-f19dcd098f60@gentoo.org
Signed-off-by: Yixun Lan <dlan@gentoo.org>
[Alastair: add MMC2 800 MHz factor and 8-bit SPL support, preserve SUN9I]
Co-developed-by: Alastair D'Silva <alastair@d-silva.org>
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
drivers/mmc/sunxi_mmc.c | 36 +++++++++++++++++++++++-------------
1 file changed, 23 insertions(+), 13 deletions(-)
diff --git a/drivers/mmc/sunxi_mmc.c b/drivers/mmc/sunxi_mmc.c
index 12c92b1e2b6..35e23dbafe2 100644
--- a/drivers/mmc/sunxi_mmc.c
+++ b/drivers/mmc/sunxi_mmc.c
@@ -101,13 +101,14 @@ static int mmc_set_mod_clk(struct sunxi_mmc_priv *priv, unsigned int hz)
pll_hz /= 2;
/*
- * The A523/T527 uses PERIPH0_400M as the MMC0/1 input clock,
- * and PERIPH0_800M for MMC2. There is also the hidden divider
- * of 2. The clock code reports 600 MHz for PERIPH0.
+ * The A523/T527 and A733 use PERIPH0_400M as the MMC0/1 input
+ * clock, and PERIPH0_800M for MMC2. There is also the hidden
+ * divider of 2. The clock code reports 600 MHz for PERIPH0.
* Adjust the calculation accordingly: 600 * hidden2 / 3 for
* MMC0/1, and 600 * hidden2 / 3 * 2 for MMC2.
*/
- if (IS_ENABLED(CONFIG_MACH_SUN55I_A523)) {
+ if (IS_ENABLED(CONFIG_MACH_SUN55I_A523) ||
+ IS_ENABLED(CONFIG_MACH_SUN60I_A733)) {
pll_hz /= 3;
if (priv->mmc_no == 2)
pll_hz *= 2;
@@ -166,8 +167,9 @@ static int mmc_set_mod_clk(struct sunxi_mmc_priv *priv, unsigned int hz)
CCM_MMC_CTRL_SCLK_DLY(sclk_dly);
}
- /* The A523 has a second divider, not a shift. */
- if (IS_ENABLED(CONFIG_MACH_SUN55I_A523))
+ /* The A523 and A733 have a second divider, not a shift. */
+ if (IS_ENABLED(CONFIG_MACH_SUN55I_A523) ||
+ IS_ENABLED(CONFIG_MACH_SUN60I_A733))
n = (1U << n) - 1;
writel(CCM_MMC_CTRL_ENABLE| pll | CCM_MMC_CTRL_N(n) |
@@ -585,7 +587,8 @@ struct mmc *sunxi_mmc_init(int sdc_no)
if ((IS_ENABLED(CONFIG_MACH_SUN50I) || IS_ENABLED(CONFIG_MACH_SUN8I) ||
IS_ENABLED(CONFIG_MACH_SUN9I) || IS_ENABLED(CONFIG_SUN50I_GEN_H6) ||
- IS_ENABLED(CONFIG_MACH_SUN55I_A523)) && (sdc_no == 2))
+ IS_ENABLED(CONFIG_MACH_SUN55I_A523) ||
+ IS_ENABLED(CONFIG_MACH_SUN60I_A733)) && (sdc_no == 2))
cfg->host_caps = MMC_MODE_8BIT;
cfg->host_caps |= MMC_MODE_HS_52MHz | MMC_MODE_HS;
@@ -611,6 +614,10 @@ struct mmc *sunxi_mmc_init(int sdc_no)
writel(SUNXI_MMC_COMMON_CLK_GATE | SUNXI_MMC_COMMON_RESET,
SUNXI_MMC_COMMON_BASE + 4 * sdc_no);
#endif
+#elif defined(CONFIG_MACH_SUN60I_A733)
+ /* A733 has individual BGR registers per MMC channel */
+ setbits_le32(ccm + CCU_A733_MMC_BGR_REG(sdc_no),
+ BIT(GATE_SHIFT) | BIT(RESET_SHIFT));
#else /* CONFIG_SUN50I_GEN_H6 */
setbits_le32(ccm + CCU_H6_MMC_GATE_RESET, 1 << sdc_no);
/* unassert reset */
@@ -669,16 +676,19 @@ static const struct dm_mmc_ops sunxi_mmc_ops = {
.get_cd = sunxi_mmc_getcd,
};
-static unsigned get_mclk_offset(void)
+static unsigned int get_mclk_offset(unsigned int mmc_no)
{
+ if (IS_ENABLED(CONFIG_MACH_SUN60I_A733))
+ return 0xd00 + mmc_no * 0x10;
+
if (IS_ENABLED(CONFIG_MACH_SUN9I))
- return 0x410;
+ return 0x410 + mmc_no * 4;
if (IS_ENABLED(CONFIG_SUN50I_GEN_H6) || IS_ENABLED(CONFIG_SUNXI_GEN_NCAT2))
- return 0x830;
+ return 0x830 + mmc_no * 4;
- return 0x88;
-};
+ return 0x88 + mmc_no * 4;
+}
static int sunxi_mmc_probe(struct udevice *dev)
{
@@ -715,7 +725,7 @@ static int sunxi_mmc_probe(struct udevice *dev)
ccu_reg = (u32 *)(uintptr_t)ofnode_get_addr(args.node);
priv->mmc_no = ((uintptr_t)priv->reg - SUNXI_MMC0_BASE) / 0x1000;
- priv->mclkreg = (void *)ccu_reg + get_mclk_offset() + priv->mmc_no * 4;
+ priv->mclkreg = (void *)ccu_reg + get_mclk_offset(priv->mmc_no);
ret = clk_get_by_name(dev, "ahb", &gate_clk);
if (!ret)
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 11/26] sunxi: a733: uart: setup serial console
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (9 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 10/26] sunxi: mmc: enable support for A733 Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 12/26] arm64: spl: Support disposable .init.rodata section after _image_binary_end Alastair D'Silva
` (16 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
From: Yixun Lan <dlan@gentoo.org>
Set up the UART stride, R_UART base address, and SPL UART0 serial
console pinmux (PB9/PB10 by default, or PB0/PB1 when CONFIG_UART0_PORT_B0
is enabled) for the Allwinner A733 SoC.
Link: https://lore.kernel.org/r/20260113-01-a733-soc-support-v3-3-f19dcd098f60@gentoo.org
Signed-off-by: Yixun Lan <dlan@gentoo.org>
[Alastair: add 0x1000 UART stride, R_UART base, and PB0/PB1 UART0 support]
Co-developed-by: Alastair D'Silva <alastair@d-silva.org>
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
arch/arm/include/asm/arch-sunxi/serial.h | 15 ++++++++++++---
arch/arm/mach-sunxi/board.c | 10 ++++++++++
2 files changed, 22 insertions(+), 3 deletions(-)
diff --git a/arch/arm/include/asm/arch-sunxi/serial.h b/arch/arm/include/asm/arch-sunxi/serial.h
index 9386287b65e..ad785ff71e9 100644
--- a/arch/arm/include/asm/arch-sunxi/serial.h
+++ b/arch/arm/include/asm/arch-sunxi/serial.h
@@ -17,6 +17,9 @@
#elif defined(CONFIG_MACH_SUNIV)
#define SUNXI_UART0_BASE 0x01c25000
#define SUNXI_R_UART_BASE 0
+#elif defined(CONFIG_MACH_SUN60I_A733)
+#define SUNXI_UART0_BASE 0x02500000
+#define SUNXI_R_UART_BASE 0x07080000
#elif defined(CONFIG_SUNXI_GEN_NCAT2)
#define SUNXI_UART0_BASE 0x02500000
#define SUNXI_R_UART_BASE 0 // 0x07080000 (?>
@@ -25,8 +28,14 @@
#define SUNXI_R_UART_BASE 0x01f02800
#endif
-#define SUNXI_UART1_BASE (SUNXI_UART0_BASE + 0x400)
-#define SUNXI_UART2_BASE (SUNXI_UART0_BASE + 0x800)
-#define SUNXI_UART3_BASE (SUNXI_UART0_BASE + 0xc00)
+#ifdef CONFIG_MACH_SUN60I_A733
+#define SUNXI_UART_STRIDE 0x1000
+#else
+#define SUNXI_UART_STRIDE 0x400
+#endif
+
+#define SUNXI_UART1_BASE (SUNXI_UART0_BASE + SUNXI_UART_STRIDE)
+#define SUNXI_UART2_BASE (SUNXI_UART0_BASE + 2 * SUNXI_UART_STRIDE)
+#define SUNXI_UART3_BASE (SUNXI_UART0_BASE + 3 * SUNXI_UART_STRIDE)
#endif /* SUNXI_SERIAL_MEMMAP_H */
diff --git a/arch/arm/mach-sunxi/board.c b/arch/arm/mach-sunxi/board.c
index 432b1c10f92..38936afae24 100644
--- a/arch/arm/mach-sunxi/board.c
+++ b/arch/arm/mach-sunxi/board.c
@@ -145,6 +145,16 @@ static int gpio_init(void)
sunxi_gpio_set_cfgpin(SUNXI_GPB(9), 2);
sunxi_gpio_set_cfgpin(SUNXI_GPB(10), 2);
sunxi_gpio_set_pull(SUNXI_GPB(10), SUNXI_GPIO_PULL_UP);
+#elif CONFIG_CONS_INDEX == 1 && defined(CONFIG_MACH_SUN60I_A733)
+#if IS_ENABLED(CONFIG_UART0_PORT_B0)
+ sunxi_gpio_set_cfgpin(SUNXI_GPB(0), SUN60I_A733_GPB0_UART0);
+ sunxi_gpio_set_cfgpin(SUNXI_GPB(1), SUN60I_A733_GPB0_UART0);
+ sunxi_gpio_set_pull(SUNXI_GPB(1), SUNXI_GPIO_PULL_UP);
+#else
+ sunxi_gpio_set_cfgpin(SUNXI_GPB(9), SUN60I_A733_GPB9_UART0);
+ sunxi_gpio_set_cfgpin(SUNXI_GPB(10), SUN60I_A733_GPB9_UART0);
+ sunxi_gpio_set_pull(SUNXI_GPB(10), SUNXI_GPIO_PULL_UP);
+#endif
#elif CONFIG_CONS_INDEX == 1 && defined(CONFIG_MACH_SUN8I_A83T)
sunxi_gpio_set_cfgpin(SUNXI_GPB(9), SUN8I_A83T_GPB_UART0);
sunxi_gpio_set_cfgpin(SUNXI_GPB(10), SUN8I_A83T_GPB_UART0);
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 12/26] arm64: spl: Support disposable .init.rodata section after _image_binary_end
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (10 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 11/26] sunxi: a733: uart: setup serial console Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 13/26] sunxi: fel: Add Allwinner A733 ERET FEL return support Alastair D'Silva
` (15 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
Some SoCs with tight SRAM budgets (such as Allwinner A733, where TF-A
BL31 loads into SRAM A2 at 0x00058000) carry large one-time DRAM
initialization tables and PHY firmware in SPL that are never referenced
after DRAM bringup completes.
Add CONFIG_SPL_DISPOSABLE_INIT_RODATA and CONFIG_SPL_ACTIVE_MAX_SIZE so
ARM64 SPL builds with separate BSS and no SPL DT append can emit
.init.rodata immediately after _image_binary_end, copy it into
u-boot-spl-nodtb.bin via SPL_OBJCFLAGS, and enforce a linker ASSERT on
the active SPL ceiling before the reclaimable region begins.
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
arch/arm/config.mk | 4 ++++
arch/arm/cpu/armv8/u-boot-spl.lds | 20 ++++++++++++++++++++
arch/arm/cpu/armv8/u-boot.lds | 1 +
common/spl/Kconfig | 18 ++++++++++++++++++
4 files changed, 43 insertions(+)
diff --git a/arch/arm/config.mk b/arch/arm/config.mk
index bce9a31e966..9f76e70bd21 100644
--- a/arch/arm/config.mk
+++ b/arch/arm/config.mk
@@ -165,6 +165,10 @@ endif
# 2) unit tests include device tree blobs
OBJCOPYFLAGS += -j .dtb.init.rodata
+ifeq ($(CONFIG_SPL_DISPOSABLE_INIT_RODATA),y)
+SPL_OBJCFLAGS += -j .init.rodata
+endif
+
ifdef CONFIG_EFI_LOADER
OBJCOPYFLAGS += -j .efi_runtime -j .efi_runtime_rel
endif
diff --git a/arch/arm/cpu/armv8/u-boot-spl.lds b/arch/arm/cpu/armv8/u-boot-spl.lds
index 39020a1fddd..3feeb73ff04 100644
--- a/arch/arm/cpu/armv8/u-boot-spl.lds
+++ b/arch/arm/cpu/armv8/u-boot-spl.lds
@@ -33,6 +33,9 @@ SECTIONS
.rodata : {
. = ALIGN(8);
*(SORT_BY_ALIGNMENT(SORT_BY_NAME(.rodata*)))
+#if !CONFIG_IS_ENABLED(DISPOSABLE_INIT_RODATA)
+ *(SORT_BY_ALIGNMENT(SORT_BY_NAME(.init.rodata*)))
+#endif
} >.sram
.data : {
@@ -67,6 +70,18 @@ SECTIONS
_end = .;
_image_binary_end = .;
+#if CONFIG_IS_ENABLED(DISPOSABLE_INIT_RODATA)
+ /*
+ * Disposable early-init read-only tables (e.g. DRAM PHY firmware/tables)
+ * placed after _image_binary_end so subsequent firmware loads (such as
+ * TF-A BL31) can reclaim this SRAM region once early bringup completes.
+ */
+ .init.rodata : {
+ . = ALIGN(8);
+ *(SORT_BY_ALIGNMENT(SORT_BY_NAME(.init.rodata*)))
+ } >.sram
+#endif
+
#ifdef CONFIG_SPL_SEPARATE_BSS
.bss : {
__bss_start = .;
@@ -102,3 +117,8 @@ ASSERT(_image_binary_end % 8 == 0, \
ASSERT(ADDR(.bss) % 8 == 0, \
".bss must be 8-byte aligned");
+
+#if CONFIG_IS_ENABLED(DISPOSABLE_INIT_RODATA) && CONFIG_SPL_ACTIVE_MAX_SIZE > 0
+ASSERT(_image_binary_end - __image_copy_start <= CONFIG_SPL_ACTIVE_MAX_SIZE, \
+ "SPL active binary exceeds CONFIG_SPL_ACTIVE_MAX_SIZE");
+#endif
diff --git a/arch/arm/cpu/armv8/u-boot.lds b/arch/arm/cpu/armv8/u-boot.lds
index 231ab28b50c..55ad4c5feea 100644
--- a/arch/arm/cpu/armv8/u-boot.lds
+++ b/arch/arm/cpu/armv8/u-boot.lds
@@ -120,6 +120,7 @@ SECTIONS
#endif
__start_rodata = .;
*(SORT_BY_ALIGNMENT(SORT_BY_NAME(.rodata*)))
+ *(SORT_BY_ALIGNMENT(SORT_BY_NAME(.init.rodata*)))
}
__u_boot_list ALIGN(8) : {
diff --git a/common/spl/Kconfig b/common/spl/Kconfig
index 4930ecb0861..170578ef895 100644
--- a/common/spl/Kconfig
+++ b/common/spl/Kconfig
@@ -86,6 +86,24 @@ config SPL_MAX_SIZE
sections), BSS excluded. When defined, the linker checks that the
actual size does not exceed it.
+config SPL_DISPOSABLE_INIT_RODATA
+ bool
+ depends on ARM64 && SPL_SEPARATE_BSS && !SPL_OF_CONTROL
+ help
+ Place .init.rodata* sections after _image_binary_end in the SPL
+ SRAM image so one-time initialization tables (such as DRAM PHY
+ firmware) can be reclaimed by subsequent firmware stages (e.g.
+ TF-A BL31) once early hardware bringup completes.
+
+config SPL_ACTIVE_MAX_SIZE
+ hex
+ depends on SPL_DISPOSABLE_INIT_RODATA
+ default 0x0
+ help
+ When non-zero, maximum size in bytes of the active SPL image
+ (_image_binary_end - __image_copy_start) before the disposable
+ .init.rodata region begins.
+
config SPL_PAD_TO
hex "Offset to which the SPL should be padded before appending the SPL payload"
default 0x7f8000 if ARCH_ROCKCHIP
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 13/26] sunxi: fel: Add Allwinner A733 ERET FEL return support
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (11 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 12/26] arm64: spl: Support disposable .init.rodata section after _image_binary_end Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-19 23:53 ` Andre Przywara
2026-11-13 2:32 ` [PATCH 14/26] sunxi: dram: a733: Add ARC PMU register, ABI, and build definitions Alastair D'Silva
` (14 subsequent siblings)
27 siblings, 1 reply; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
On the Allwinner A733, boot0.h latches CLU0_CPU0_CTRL_REG to AArch64
and sunxi-fel uses SRAM swap buffers across 0x47000. Rather than arming
R_CPUCFG hotplug magic and warm-resetting through BROM, drop directly
to Secure AArch32 SVC via ERET after draining UART0 TX, and install an
EL3 SMC vector in SH_SRAM (0x00080000) so sunxi-fel can re-enter AArch64
EL3 for BL31.
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
arch/arm/cpu/armv8/fel_utils.S | 113 ++++++++++++++++++++++++++++++++-
1 file changed, 112 insertions(+), 1 deletion(-)
diff --git a/arch/arm/cpu/armv8/fel_utils.S b/arch/arm/cpu/armv8/fel_utils.S
index ccddfaaf04c..fc0e0b93c37 100644
--- a/arch/arm/cpu/armv8/fel_utils.S
+++ b/arch/arm/cpu/armv8/fel_utils.S
@@ -11,6 +11,30 @@
#include <asm/system.h>
#include <linux/linkage.h>
#include <asm/arch/cpu.h>
+#include <asm/arch/serial.h>
+
+#ifdef CONFIG_MACH_SUN60I_A733
+#define A733_FEL_EL3_VBAR SUNXI_FEL_EL3_VBAR
+#define A733_FEL_EL3_A32_SYNC_OFF 0x600
+#define UART_LSR_OFFSET 0x14
+#define UART_LSR_TEMT_BIT 6
+#define SCR_EL3_IRQ_EN (1 << 1)
+#define SCR_EL3_FIQ_EN (1 << 2)
+#define SCR_EL3_FEL_A32_CLEAR (SCR_EL3_RW_AARCH64 | \
+ SCR_EL3_SMD_DIS | \
+ SCR_EL3_EA_EN | \
+ SCR_EL3_FIQ_EN | \
+ SCR_EL3_IRQ_EN | \
+ SCR_EL3_NS_EN)
+#define ICC_SRE_EL3_ENABLE_ALL 0xf
+#define ICC_SRE_EL1_ENABLE_ALL 0x7
+#define ICC_IGRPEN1_EL3_ENABLE_ALL 0x3
+#define SPSR_EL3_A32_SVC_MASKED (SPSR_EL_ASYN_MASK | \
+ SPSR_EL_IRQ_MASK | \
+ SPSR_EL_FIQ_MASK | \
+ SPSR_EL_M_AARCH32 | \
+ SPSR_EL_M_SVC)
+#endif
/*
* We don't overwrite save_boot_params() here, to save the FEL state upon
@@ -40,6 +64,89 @@ ENTRY(return_to_fel)
adr x1, fel_stash_addr // to find the fel_stash address in AA32
str w2, [x1]
+#if defined(CONFIG_MACH_SUN60I_A733)
+#if CONFIG_IS_ENABLED(SERIAL)
+ /* Drain UART0 TX FIFO before dropping to AArch32 */
+ ldr x2, =SUNXI_UART0_BASE
+ mov w1, #0x10000
+2: ldr w0, [x2, #UART_LSR_OFFSET]
+ tbnz w0, #UART_LSR_TEMT_BIT, 3f
+ subs w1, w1, #1
+ b.ne 2b
+3:
+#endif
+
+ /*
+ * On A733, boot0.h latches CLU0_CPU0_CTRL_REG to AArch64 and FEL uses
+ * SRAM swap buffers across 0x47000. Rather than warm-resetting through
+ * the BROM hotplug path, drop directly to Secure AArch32 SVC via ERET
+ * and install an EL3 SMC vector in SH_SRAM outside the FEL swap buffer
+ * so sunxi-fel can re-enter AArch64 EL3 for BL31.
+ */
+ adr x4, el3_a32_sync
+ adr x5, el3_a32_sync_end
+ ldr x6, =(A733_FEL_EL3_VBAR + A733_FEL_EL3_A32_SYNC_OFF)
+ mov x8, x6
+4: ldr w7, [x4], #4
+ str w7, [x6], #4
+ cmp x4, x5
+ b.lo 4b
+
+5: dc cvau, x8
+ add x8, x8, #64
+ cmp x8, x6
+ b.lo 5b
+ dsb ish
+ ic iallu
+ dsb ish
+ isb
+
+ ldr x0, =A733_FEL_EL3_VBAR
+ msr vbar_el3, x0
+
+ mrs x1, ICC_SRE_EL3
+ orr x1, x1, #ICC_SRE_EL3_ENABLE_ALL
+ msr ICC_SRE_EL3, x1
+ isb
+
+ mrs x1, ICC_SRE_EL1
+ orr x1, x1, #ICC_SRE_EL1_ENABLE_ALL
+ msr ICC_SRE_EL1, x1
+ mov x1, #ICC_IGRPEN1_EL3_ENABLE_ALL
+ msr ICC_IGRPEN1_EL3, x1
+
+ mrs x0, scr_el3
+ mov x1, #SCR_EL3_FEL_A32_CLEAR
+ bic x0, x0, x1
+ msr scr_el3, x0
+
+ mov x0, #SPSR_EL3_A32_SVC_MASKED
+ msr spsr_el3, x0
+
+ adr x0, back_in_32
+ msr elr_el3, x0
+
+ dsb sy
+ isb
+ eret
+
+ .align 2
+el3_a32_sync:
+ mrs x9, scr_el3
+ mov x10, #SCR_EL3_RW_AARCH64
+ orr x9, x9, x10
+ msr scr_el3, x9
+ isb
+ cbnz w0, 6f
+ movz w0, #(CONFIG_SUNXI_BL31_BASE & 0xffff)
+ movk w0, #(CONFIG_SUNXI_BL31_BASE >> 16), lsl #16
+6: uxtw x0, w0
+ ic iallu
+ dsb sy
+ isb
+ br x0
+el3_a32_sync_end:
+#else
ldr w0, =0xfa50392f // CPU hotplug magic
#if defined(CONFIG_MACH_SUN50I_H616) || defined(CONFIG_MACH_SUN50I_A133) || \
defined(CONFIG_MACH_SUN55I_A523)
@@ -63,6 +170,7 @@ ENTRY(return_to_fel)
isb sy
1: wfi
b 1b
+#endif
fel_stash_addr: // must immediately precede back_in_32:
.word 0x00000000 // receives fel_stash addr, by AA64 code above
@@ -71,6 +179,9 @@ fel_stash_addr: // must immediately precede back_in_32:
back_in_32:
.word 0xe51f000c // ldr r0, [pc, #-12] ; load fel_stash address
.word 0xe5901008 // ldr r1, [r0, #8]
+#ifdef CONFIG_MACH_SUN60I_A733
+ .word 0xe38110c0 // orr r1, r1, #0xc0 ; mask IRQ/FIQ in thunk
+#endif
.word 0xe129f001 // msr CPSR_fc, r1
.word 0xf57ff06f // isb
.word 0xe590d000 // ldr sp, [r0]
@@ -82,7 +193,7 @@ back_in_32:
.word 0xe590100c // ldr r1, [r0, #12]
.word 0xee011f10 // mcr 15, 0, r1, cr1, cr0, {0} ; SCTLR
.word 0xf57ff06f // isb
-#ifdef CONFIG_MACH_SUN55I_A523
+#if defined(CONFIG_MACH_SUN55I_A523) || defined(CONFIG_MACH_SUN60I_A733)
.word 0xe5901018 // ldr r1, [r0, #24]
.word 0xee041f16 // mcr 15, 0, r1, cr4, cr6, {0}; ICC_PMR
.word 0xe590101c // ldr r1, [r0, #28]
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 14/26] sunxi: dram: a733: Add ARC PMU register, ABI, and build definitions
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (12 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 13/26] sunxi: fel: Add Allwinner A733 ERET FEL return support Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 15/26] sunxi: dram: a733: Add ARC PMU primitives, clock, telemetry, and CBT Alastair D'Silva
` (13 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
Add the shared host/ARC ABI header, DesignWare LPDDR5 DDR PHY CSR and
DCCM offsets, internal firmware prototypes, ICCM execute-only linker
script, and single-pass ELF build helper for the Allwinner A733 DDR PHY
ARC EM4 microsequencer firmware.
The firmware in arch/arm/mach-sunxi/pmu/ is an original GPL-2.0+ C
implementation written under a strict two-team clean-room methodology:
an analysis team documented hardware register addresses, DCCM mailbox
protocol layouts, and JEDEC LPDDR5 training state transitions as
functional specifications without sharing code or control-flow idioms,
and an isolated implementation team wrote this C codebase solely from
those specifications and JEDEC JESD209-5.
build_pmu_fw.py locates an ARC toolchain via ARC_CROSS_COMPILE, PATH,
or ~/.buildman, supports --pmu-fw for externally built ICCM binaries,
and supports --allow-missing (wired to BINMAN_ALLOW_MISSING) so CI
builders without arc-*-gcc emit a non-functional stub cleanly.
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
arch/arm/mach-sunxi/pmu/build_pmu_fw.py | 257 ++++++++
arch/arm/mach-sunxi/pmu/dwc_ddrphy_pmu_regs.h | 463 +++++++++++++++
arch/arm/mach-sunxi/pmu/pmu.ld | 38 ++
arch/arm/mach-sunxi/pmu/pmu_fw.h | 553 ++++++++++++++++++
arch/arm/mach-sunxi/pmu/sunxi_pmu_abi.h | 251 ++++++++
5 files changed, 1562 insertions(+)
create mode 100755 arch/arm/mach-sunxi/pmu/build_pmu_fw.py
create mode 100644 arch/arm/mach-sunxi/pmu/dwc_ddrphy_pmu_regs.h
create mode 100644 arch/arm/mach-sunxi/pmu/pmu.ld
create mode 100644 arch/arm/mach-sunxi/pmu/pmu_fw.h
create mode 100644 arch/arm/mach-sunxi/pmu/sunxi_pmu_abi.h
diff --git a/arch/arm/mach-sunxi/pmu/build_pmu_fw.py b/arch/arm/mach-sunxi/pmu/build_pmu_fw.py
new file mode 100755
index 00000000000..8ce2bfdcd6c
--- /dev/null
+++ b/arch/arm/mach-sunxi/pmu/build_pmu_fw.py
@@ -0,0 +1,257 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0+
+# Copyright (C) 2026 Alastair D'Silva <alastair@d-silva.org>
+"""
+Single-pass ELF builder for Allwinner A733 LPDDR5 ARC EM4 PMU Firmware.
+"""
+
+import argparse
+import configparser
+import glob
+import os
+import shutil
+import struct
+import subprocess
+import sys
+import tempfile
+
+DEFAULT_SOURCES = (
+ "pmu_primitives.c",
+ "pmu_clock.c",
+ "pmu_telemetry.c",
+ "pmu_cbt.c",
+ "pmu_deskew.c",
+ "pmu_eye.c",
+ "pmu_train.c",
+)
+
+
+def resolve_arc_toolchain(preferred_prefix, explicit=False):
+ """Locate an ARC GCC toolchain prefix across PATH and U-Boot CI locations."""
+ if preferred_prefix:
+ if shutil.which(f"{preferred_prefix}gcc") and shutil.which(f"{preferred_prefix}objcopy"):
+ return preferred_prefix
+ if explicit:
+ raise RuntimeError(
+ f"Explicitly requested ARC toolchain '{preferred_prefix}gcc' not found"
+ )
+
+ candidates = [
+ "arc-linux-gnu-",
+ "arc-linux-",
+ "arc-elf32-",
+ ]
+ for bman_cfg in (
+ os.path.expanduser("~/.buildman"),
+ "/home/uboot/.buildman",
+ ):
+ if not os.path.isfile(bman_cfg):
+ continue
+ cfg = configparser.ConfigParser()
+ try:
+ cfg.read(bman_cfg)
+ if cfg.has_option("toolchain-prefix", "arc"):
+ candidates.append(os.path.expanduser(cfg.get("toolchain-prefix", "arc")))
+ if cfg.has_section("toolchain"):
+ for _, root in cfg.items("toolchain"):
+ root = os.path.expanduser(root)
+ for pat in ("bin/arc-*-gcc", "*/bin/arc-*-gcc"):
+ candidates.extend(sorted(glob.glob(os.path.join(root, pat)), reverse=True))
+ except configparser.Error:
+ pass
+
+ for pattern in (
+ "/opt/gcc-*-nolibc/arc-linux/bin/arc-linux-",
+ os.path.expanduser("~/.buildman-toolchains/gcc-*-nolibc/arc-linux/bin/arc-linux-"),
+ "/home/uboot/.buildman-toolchains/gcc-*-nolibc/arc-linux/bin/arc-linux-",
+ ):
+ candidates.extend(sorted(glob.glob(pattern), reverse=True))
+
+ for prefix in candidates:
+ if prefix.endswith("gcc"):
+ prefix = prefix[:-3]
+ if shutil.which(f"{prefix}gcc") and shutil.which(f"{prefix}objcopy"):
+ return prefix
+
+ return None
+
+
+def write_header(output_h_path, hwords, missing=False):
+ """Atomically emit the C header containing the PMU ICCM image or missing stub."""
+ out_dir = os.path.dirname(output_h_path)
+ os.makedirs(out_dir, exist_ok=True)
+ with tempfile.NamedTemporaryFile("w", dir=out_dir, delete=False, prefix=".pmu_fw_tmp_") as f:
+ tmp_name = f.name
+ f.write("/* SPDX-License-Identifier: GPL-2.0+ */\n")
+ f.write("/*\n")
+ if missing:
+ f.write(" * Auto-generated stub ARC EM4 PMU Firmware Header (missing toolchain).\n")
+ else:
+ f.write(
+ f" * Auto-generated ARC EM4 PMU Firmware Header "
+ f"({len(hwords) * 2} bytes / {len(hwords)} halfwords).\n"
+ )
+ f.write(" */\n\n")
+ f.write("#ifndef __DRAM_SUN60I_A733_PMU_FW_H__\n")
+ f.write("#define __DRAM_SUN60I_A733_PMU_FW_H__\n\n")
+ f.write("#include <linux/compiler_attributes.h>\n")
+ f.write("#include <linux/types.h>\n\n")
+ if missing:
+ f.write("#define SUN60I_A733_PMU_FW_MISSING 1\n\n")
+ f.write(
+ "static const u16 __section(\".init.rodata\") "
+ "sun60i_a733_pmu_imem[1] = { 0 };\n\n"
+ )
+ else:
+ f.write(
+ f"static const u16 __section(\".init.rodata\") "
+ f"sun60i_a733_pmu_imem[{len(hwords)}] = {{\n"
+ )
+ for i in range(0, len(hwords), 8):
+ hex_str = ", ".join(f"0x{x:04x}" for x in hwords[i:i + 8])
+ f.write(f"\t{hex_str},\n")
+ f.write("};\n\n")
+ f.write("#endif /* __DRAM_SUN60I_A733_PMU_FW_H__ */\n")
+
+ os.replace(tmp_name, output_h_path)
+
+
+def main():
+ pmu_dir_default = os.path.abspath(os.path.dirname(__file__))
+ repo_root = os.path.abspath(os.path.join(pmu_dir_default, "..", "..", "..", ".."))
+ default_out = os.path.join(repo_root, "arch/arm/mach-sunxi/dram_sun60i_a733_pmu_fw.h")
+
+ parser = argparse.ArgumentParser(
+ description="Build Allwinner A733 LPDDR5 ARC EM4 PMU firmware."
+ )
+ parser.add_argument("--cross-compile", default="", help="ARC toolchain prefix")
+ parser.add_argument("--pmu-fw", default="", help="Prebuilt raw PMU ICCM binary path")
+ parser.add_argument("--allow-missing", action="store_true",
+ help="Emit a non-functional stub header if ARC toolchain is absent")
+ parser.add_argument("--src-dir", default=pmu_dir_default, help="PMU firmware source directory")
+ parser.add_argument("--ld-script", default="", help="Linker script path")
+ parser.add_argument("--out", default=default_out, help="Output C header path")
+ parser.add_argument("sources", nargs="*", help="Explicit C source files to compile")
+ args = parser.parse_args()
+
+ src_dir = os.path.abspath(args.src_dir)
+ ld_script = os.path.abspath(args.ld_script) if args.ld_script else os.path.join(src_dir, "pmu.ld")
+ output_h_path = os.path.abspath(args.out)
+
+ if args.pmu_fw:
+ with open(args.pmu_fw, "rb") as bf:
+ raw = bf.read()
+ if not raw:
+ raise ValueError(f"PMU firmware '{args.pmu_fw}' is empty")
+ if len(raw) > 0x10000:
+ raise ValueError(
+ f"PMU firmware '{args.pmu_fw}' exceeds 64 KiB ICCM limit ({len(raw)} bytes)"
+ )
+ else:
+ explicit_cc = any(arg.startswith("--cross-compile") for arg in sys.argv[1:])
+ prefix = resolve_arc_toolchain(args.cross_compile, explicit=explicit_cc)
+ if prefix is None:
+ if args.allow_missing:
+ print(
+ "Image 'spl/u-boot-spl.bin' is missing ARC cross-compiler; "
+ "using non-functional stub PMU firmware",
+ file=sys.stderr,
+ )
+ write_header(output_h_path, [], missing=True)
+ return 0
+ print(
+ "Error: ARC toolchain not found (install arc-linux-gnu-gcc, pass PMU_FW=, "
+ "or build with BINMAN_ALLOW_MISSING=1)",
+ file=sys.stderr,
+ )
+ return 1
+
+ gcc = f"{prefix}gcc"
+ objcopy = f"{prefix}objcopy"
+ c_files = (
+ [os.path.abspath(p) for p in args.sources]
+ if args.sources
+ else [os.path.join(src_dir, name) for name in DEFAULT_SOURCES]
+ )
+
+ cflags = [
+ "-mcpu=em4",
+ "-mrf16",
+ "-mcode-density",
+ "-mbranch-index",
+ "-mcode-density-frame",
+ "-mdiv-rem",
+ "-mbarrel-shifter",
+ "-mmpy-option=mpy",
+ "-fno-branch-count-reg",
+ "-EL",
+ "-Os",
+ "-ffreestanding",
+ "-fno-caller-saves",
+ "-fno-ipa-ra",
+ "-ffixed-r26",
+ "-ffixed-r27",
+ "-fno-strict-aliasing",
+ "-fno-jump-tables",
+ "-fno-tree-switch-conversion",
+ "-fno-pic",
+ "-fno-pie",
+ "-fno-stack-protector",
+ "-fno-builtin",
+ "-ffunction-sections",
+ "-fdata-sections",
+ "-std=gnu11",
+ "-Wall",
+ "-Wextra",
+ f"-I{src_dir}",
+ ]
+
+ with tempfile.TemporaryDirectory(prefix="pmu_fw_") as tmpdir:
+ objs = []
+ for src_path in c_files:
+ base = os.path.splitext(os.path.basename(src_path))[0]
+ obj_path = os.path.join(tmpdir, f"{base}.o")
+ subprocess.check_call([gcc, *cflags, "-c", src_path, "-o", obj_path])
+ objs.append(obj_path)
+
+ elf_path = os.path.join(tmpdir, "pmu.elf")
+ bin_path = os.path.join(tmpdir, "pmu.bin")
+ ld_cmd = [
+ gcc,
+ "-mcpu=em4",
+ "-mrf16",
+ "-mcode-density",
+ "-mdiv-rem",
+ "-mbarrel-shifter",
+ "-mmpy-option=mpy",
+ "-EL",
+ "-Os",
+ "-nostdlib",
+ "-no-pie",
+ f"-Wl,-L,{src_dir}",
+ f"-Wl,-T,{ld_script}",
+ "-Wl,--gc-sections",
+ "-Wl,--build-id=none",
+ *objs,
+ "-o",
+ elf_path,
+ ]
+ subprocess.check_call(ld_cmd)
+ subprocess.check_call([objcopy, "-O", "binary", elf_path, bin_path])
+
+ with open(bin_path, "rb") as bf:
+ raw = bf.read()
+
+ if len(raw) % 2 != 0:
+ raw += b"\x00"
+
+ hwords = list(struct.unpack(f"<{len(raw) // 2}H", raw))
+ while hwords and hwords[-1] == 0:
+ hwords.pop()
+
+ write_header(output_h_path, hwords, missing=False)
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/arch/arm/mach-sunxi/pmu/dwc_ddrphy_pmu_regs.h b/arch/arm/mach-sunxi/pmu/dwc_ddrphy_pmu_regs.h
new file mode 100644
index 00000000000..119fbc9c644
--- /dev/null
+++ b/arch/arm/mach-sunxi/pmu/dwc_ddrphy_pmu_regs.h
@@ -0,0 +1,463 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Copyright (C) 2026 Alastair D'Silva <alastair@d-silva.org>
+ *
+ * Synopsys DesignWare DDR PHY PMU Microcontroller Register & Memory Definitions
+ * Target Architecture: Synopsys ARC EM4 (ARCv2 ISA)
+ * SoC: Allwinner A733 (Sun60i) / LPDDR4 & LPDDR5 PHY
+ *
+ * =============================================================================
+ * PHY SILICON ARCHITECTURE & TERMINOLOGY EXPLAINED
+ * =============================================================================
+ *
+ * Inside the Synopsys DesignWare DDR PHY Type 9 silicon on Allwinner A733:
+ *
+ * 1. DBYTE Slices (Data Byte Slices 0..3):
+ * Each DBYTE slice manages the physical analog signals for one 8-bit byte lane
+ * (DQ0..DQ7, Data Mask DM, and differential Data Strobe DQS/DQSN).
+ * Because the A733 uses dual 16-bit channels (Ch0 = 16-bit, Ch1 = 16-bit),
+ * there are 4 DBYTE slices in total:
+ * - Slice 0: Ch0 Byte Lane 0 (DQ0..DQ7)
+ * - Slice 1: Ch0 Byte Lane 1 (DQ8..DQ15)
+ * - Slice 2: Ch1 Byte Lane 0 (DQ0..DQ7)
+ * - Slice 3: Ch1 Byte Lane 1 (DQ8..DQ15)
+ *
+ * 2. AC Slices (Address/Command Slices 0..1):
+ * Manage the clock (CK/CKN) and Command/Address (CA0..CA6) signals sent to the
+ * LPDDR5 memory chips for each channel.
+ *
+ * 3. PUB (PHY Utility Block):
+ * The central control core containing master PLL configuration, clock gating,
+ * PMU execution mailboxes, analog bias generators, and host CPU APB interface.
+ *
+ * 4. LCDL (Local Clock Delay Line):
+ * Analog tapped delay lines consisting of coarse stages and fine vernier stages.
+ * Adjusting LCDL delay taps shifts signal transitions by picoseconds, enabling
+ * the PMU to place sampling clock edges precisely in the center of the data eye.
+ *
+ * 5. CBT (Command Bus Training):
+ * A JEDEC-standard training mode where the memory controller transmits command
+ * patterns to the DRAM, and the DRAM returns feedback on the DQ lines. This allows
+ * the PHY to align the CA signals to the memory clock with picosecond accuracy.
+ *
+ * 6. ZQ Calibration:
+ * Calibrates transmitter output impedance (Ron) and receiver on-die termination
+ * (ODT) against an external precision resistor to prevent transmission line
+ * signal reflections.
+ * =============================================================================
+ */
+
+#ifndef _DWC_DDRPHY_PMU_REGS_H
+#define _DWC_DDRPHY_PMU_REGS_H
+
+#include "sunxi_pmu_abi.h"
+
+/* ========================================================================== */
+
+/* ========================================================================== */
+/* ARC PMU DCCM Addresses (0x80000000 Base) */
+/* ========================================================================== */
+
+#define PMU_DMEM_BASE 0x80000000UL
+#define PMU_DMEM_CLK_GATE_FLAG 0x80000001UL /* Byte: Clock gate / broadcast flag */
+#define PMU_DMEM_PSTATE_FLAGS 0x80000004UL /* Byte: PState / channel status flags */
+#define PMU_DMEM_CAL_MODE_FLAG_ADDR 0x80000005UL /* Byte: Calibration parameter */
+#define PMU_DMEM_DRAM_FREQ_MHZ 0x80000006UL /* Half: Operating DRAM data rate (MT/s) */
+#define PMU_DMEM_PLL_RATIO 0x80000008UL /* Byte: WCK:CK ratio (4 = 4:1, 2 = 2:1) */
+#define PMU_DMEM_ASSERT_CODE 0x8000000cUL /* Word: Fatal assertion / panic code */
+#define PMU_DMEM_SIGNED_STEP 0x8000000eUL /* SByte: Signed AC timing step */
+#define PMU_DMEM_SEQ_CTRL 0x80000010UL /* Half: Training sequence control mask */
+#define PMU_DMEM_POST_THRESH 0x80000012UL /* Byte: Post mailbox threshold */
+#define PMU_DMEM_TRAIN_MODE_ADDR 0x80000014UL /* Half: Active training stage mode */
+#define PMU_DMEM_ANALOG_CFG 0x80000015UL /* Byte: Analog cfg flag */
+#define PMU_DMEM_RX_CLK_EN 0x80000019UL /* Byte: Receiver clock enable flag */
+#define PMU_DMEM_CAL_STEP_CONFIG 0x8000001aUL /* Half: Calibration step configuration */
+#define PMU_DMEM_DESKEW_LIM 0x8000001cUL /* Byte: Dual-rank deskew cfg */
+#define PMU_DMEM_CH0_DQ_WIDTH 0x80000024UL /* Byte: Total DQ bit width Ch0 */
+#define PMU_DMEM_CH0_ANALOG_EN 0x80000025UL /* Byte: Ch0 analog power enable */
+#define PMU_DMEM_CH0_DESKEW_COARSE 0x80000026UL /* Base: Ch0 coarse deskew diff */
+#define PMU_DMEM_ACTIVE_SLICE_CNT 0x80000028UL /* Byte: Active DBYTE slice cnt */
+#define PMU_DMEM_CH0_DESKEW_CROSS 0x8000002cUL /* Base: Ch0 cross deskew diff */
+#define PMU_DMEM_CH0_DESKEW_FINE 0x80000030UL /* Base: Ch0 fine deskew diff */
+#define PMU_DMEM_CH1_DQ_WIDTH 0x8000003fUL /* Byte: Total DQ bit width Ch1 */
+#define PMU_DMEM_CH1_ACTIVE_FLAG 0x80000040UL /* Byte: Ch1 active enable flag */
+#define PMU_DMEM_SLICE_PROFILE 0x8000005aUL /* Base: 29-entry slice profile */
+#define PMU_DMEM_MR3_ADDR 0x80000062UL /* Byte: Signed Vref training mode flags */
+#define PMU_DMEM_MR13_ADDR 0x80000072UL /* Byte: Vref DAC control configuration */
+#define PMU_DMEM_MR20_ADDR 0x8000008eUL /* Byte: JEDEC MR20 RDQS mode byte */
+#define PMU_DMEM_MR22_ADDR 0x80000096UL /* Byte: DRAM training config byte */
+#define PMU_DMEM_CAL_OVERRIDE 0x800000ceUL /* Byte: Eye margin override */
+#define PMU_DMEM_WCK_PHASE 0x800000d2UL /* Base: WCK align phase codes */
+#define PMU_DMEM_TEST_PAT 0x800000e4UL /* Half: Custom CBT test pattern */
+#define PMU_DMEM_DESKEW_OFFSET 0x800000e6UL /* Byte: Deskew offset override */
+#define PMU_DMEM_VREF_OVR_CTRL_ADDR 0x800000e7UL /* Byte: Vref DAC override control flags */
+#define PMU_DMEM_PLL_GATE_CH0_ADDR 0x800000e8UL /* Base: Per-rank shadow base */
+#define PMU_DMEM_PLL_GATE_CH1_ADDR 0x800000eaUL /* Half: Ch1 shadow base */
+#define PMU_DMEM_PLL_SEQ_EN_ADDR 0x800000f2UL /* Half: PLL param seq enable */
+#define PMU_DMEM_AC_PROFILE_EN_ADDR 0x800000f4UL /* Byte: AC slice profile en */
+#define PMU_DMEM_LANE_ERR_MASK 0x80000101UL /* Byte: Per-lane deskew error status mask */
+#define PMU_DMEM_TRAIN_FEAT_MASK_ADDR 0x80000102UL /* Byte: Eye sweep op17 param */
+#define PMU_DMEM_CA_VREF_RANK1_ADDR 0x80000103UL /* Byte: CA Vref state Rank 1 */
+#define PMU_DMEM_CA_VREF_RANK0_ADDR 0x80000104UL /* Byte: CA Vref state Rank 0 */
+#define PMU_DMEM_SHADOW_INIT_IDX_ADDR 0x80000108UL /* Half: Initial shadow idx */
+#define PMU_DMEM_SHADOW_PHASE_CNT_ADDR 0x8000010aUL /* Half: Phase shadow count */
+#define PMU_DMEM_SHADOW_MID_IDX_ADDR 0x8000010cUL /* Half: Midpoint shadow idx */
+#define PMU_DMEM_SHADOW_FINAL_CNT_ADDR 0x8000010eUL /* Half: Final shadow count */
+#define PMU_DMEM_GATE_DESC 0x80000118UL /* Base: Gate eye margin desc */
+#define PMU_DMEM_RFIFO_BAK 0x80000130UL /* Base: 7-halfword RFIFO backup */
+#define PMU_DMEM_CBT_LANE_LIM 0x80000403UL /* Byte: CBT strobe/lane limit */
+#define PMU_DMEM_MARGIN_STAT 0x80000405UL /* Byte: Stage 10 margin status byte */
+#define PMU_DMEM_STAGE_RANK_IDX_ADDR 0x80000406UL /* Byte: Stage active rank idx */
+#define PMU_DMEM_STAGE_SLICE_IDX_ADDR 0x80000407UL /* Byte: Stage active slice idx */
+#define PMU_DMEM_STAGE_LANE_IDX_ADDR 0x80000408UL /* Byte: Stage active lane idx */
+#define PMU_DMEM_COST_WEIGHT_2 0x80000410UL /* Half: Eye overflow weight */
+#define PMU_DMEM_SHADOW_TRACE_PTR 0x80000418UL /* Word: Shadow trace buffer ptr */
+#define PMU_DMEM_STEP_CFG_RAW 0x8000001aUL /* Half: Raw step config register */
+#define PMU_DMEM_ACTIVE_CH_OFFSET 0x8000041cUL /* Word: Active channel CSR offset */
+#define PMU_DMEM_COST_WEIGHT_1 0x80000422UL /* Half: Eye deviation weight */
+#define PMU_DMEM_CAL_SAVE_CFG_ADDR 0x8000042aUL /* Half: Saved cal status word */
+#define PMU_DMEM_DBYTE_DLY_SAVE 0x80000430UL /* Base: Saved DBYTE delay array */
+#define PMU_DMEM_CAL_TABLE 0x8000043cUL /* Half: 2D calibration parameter table */
+#define PMU_DMEM_AC_PROFILE_DLY_ADDR 0x80000444UL /* Base: AC profile step cfg A */
+#define PMU_DMEM_AC_PROFILE_CFG_ADDR 0x80000448UL /* Base: AC profile step cfg B */
+#define PMU_DMEM_CBT_STATE 0x8000045aUL /* Byte: CBT state latch */
+#define PMU_DMEM_CBT_CONFIG 0x8000046aUL /* Byte: CBT configuration value */
+#define PMU_DMEM_CAL_CMD_TABLE 0x8000046cUL /* Base: Shadow cal cmd tbl */
+#define PMU_DMEM_NORM_STEP_STRIDE 0x8000047cUL /* Half: Normalized step stride */
+#define PMU_DMEM_CAL_SEQ_MAP 0x80000480UL /* Base: 21-byte cal seq idx map */
+#define PMU_DMEM_CAL_STRUCT_ADDR 0x800009a4UL /* Base: 54B cal struct base */
+#define PMU_DMEM_CH_MASK 0x800009b8UL /* Byte: Active channel mask */
+#define PMU_DMEM_AC_CAL_ACTIVE 0x80000a7cUL /* Byte: AC calibration active flag */
+#define PMU_DMEM_AC_SIGNED_STEP 0x80000a7dUL /* Byte: AC signed step offset */
+#define PMU_DMEM_DQ_PIN_MAP_BASE 0x80000a7eUL /* Base: DBYTE DQ pin swizzle table */
+#define PMU_DMEM_SLICE_CNT_LIMIT 0x80000b20UL /* Base: Slice cnt limit array */
+#define PMU_DMEM_SLICE_IMP_CFG 0x80000b24UL /* Base: Sec rank slice map */
+#define PMU_DMEM_TIMING_LIM 0x80000b64UL /* Byte: Gate timing limit */
+#define PMU_DMEM_ACTIVE_RANK 0x80000b67UL /* Byte: Active rank / byte index */
+#define PMU_DMEM_CH_RANGE 0x80000b68UL /* Base: Active ch [min, max] */
+#define PMU_DMEM_CH_ACTIVE_RANGE 0x80000b6aUL /* Base: Primary ch [min, max] */
+#define PMU_DMEM_DESKEW_DIR 0x80000b6bUL /* Byte: Active deskew sweep direction */
+#define PMU_DMEM_CH_SEC_RANGE 0x80000b6cUL /* Base: Secondary ch [min, max] */
+#define PMU_DMEM_DESKEW_STAT_TBL 0x80000b6eUL /* Base: Deskew lane stat tbl */
+#define PMU_DMEM_ARCHIVED_CAL_STAT_ADDR 0x80000b7eUL /* Half: Archived cal status */
+#define PMU_DMEM_EYE_SWEEP_STAT 0x80000b82UL /* Base: Eye sweep status arr */
+#define PMU_DMEM_LANE_STATUS 0x80000b97UL /* Byte: Latched slice lane status */
+#define PMU_DMEM_ACTIVE_SLICE_MASK 0x80000b98UL /* Byte: Active slice bitmask */
+#define PMU_DMEM_DBYTE_BACKUP 0x80000ba0UL /* Base: DBYTE delay register backup buffer */
+#define PMU_DMEM_EYE_BACKUP 0x80000bc0UL /* Base: 2D eye table backup buffer */
+#define PMU_DMEM_ALT_TRACKER 0x80000d70UL
+#define PMU_DMEM_CAL_TRACKER 0x80000d9cUL /* Base: Calibration marker scratchpad state */
+#define PMU_DMEM_EYE_CLEAR_BUF 0x80000dacUL /* Base: Eye scratchpad buffer */
+#define PMU_DMEM_AC_TIMING_TBL 0x80000dc8UL /* Base: Primary AC timing tbl */
+#define PMU_DMEM_LANE_PARAM_TBL_ADDR 0x80000e48UL /* Base: Per-lane parameter tbl */
+#define PMU_DMEM_LANE_STEP_TBL_ADDR 0x80000e4aUL /* Base: Per-lane signed step tbl */
+#define PMU_DMEM_AC_TIMING_DESC 0x80003708UL /* Base: Sec AC timing desc tbl */
+#define PMU_DMEM_CAL_TAP_STAT 0x8000378bUL /* Base: Calibrated tap status tbl */
+#define PMU_DMEM_CBT_COEFF_A_ADDR 0x80006098UL /* Base: 10-lane CBT coeff tbl A */
+#define PMU_DMEM_CBT_COEFF_B_ADDR 0x80006138UL /* Base: 10-lane CBT coeff tbl B */
+#define PMU_DMEM_CBT_LANE_FLAGS_ADDR 0x800061d8UL /* Base: 10-lane CBT byte flags */
+#define PMU_DMEM_CBT_BASE_TBL_ADDR 0x80006228UL /* Base: 10-lane CBT base tbl */
+#define PMU_DMEM_CBT_INIT_STAT_ADDR 0x80006278UL /* Base: 10-lane CBT init stat tbl */
+#define PMU_DMEM_CBT_TRACK_STAT_ADDR 0x800062c8UL /* Base: 10-lane CBT track stat tbl */
+#define PMU_DMEM_RX_SCAN_MATRIX 0x80008000UL /* Base: 3.5 KiB RX gate scan scratch */
+#define PMU_DMEM_CBT_DELAY_ARR 0x8000e798UL /* Base: CBT per-slice delay arr */
+#define PMU_DMEM_DELAY_TAP_LUT 0x8000e7b8UL /* Base: Pad delay tap LUT */
+#define PMU_DMEM_DELAY_TAP_SUB1 0x8000e7c4UL
+
+/* ========================================================================== */
+/* ARC PMU DCCM Relative Byte Offsets */
+/* ========================================================================== */
+
+#define PMU_DMEM_DRAM_FREQ_OFF 0x06 /* Half: Clock in MHz offset */
+#define PMU_DMEM_DRAM_TYPE_FLAGS 0x0009
+#define PMU_DMEM_DRAM_CFG_FLAGS 0x0a /* Half: DRAM config bitmask */
+#define PMU_DMEM_CAL_SELECT_FLAGS 0x0b /* Byte: Calibration selector flags */
+#define PMU_DMEM_CAL_MISC_FLAGS 0x000d
+#define PMU_DMEM_TRAIN_MODE_HI 0x000f
+#define PMU_DMEM_SEQUENCE_CTRL 0x0010
+#define PMU_DMEM_TRAIN_MODE 0x0014
+#define PMU_DMEM_CAL_CTRL_HI 0x0017
+#define PMU_DMEM_CAL_MODE_EXT 0x0018
+#define PMU_DMEM_CAL_DISABLE 0x0019
+#define PMU_DMEM_CAL_MODE 0x001b
+#define PMU_DMEM_LANE_MASK 0x1c /* Byte: Lane active mask */
+#define PMU_DMEM_CH0_CTRL_LO 0x0020
+#define PMU_DMEM_CH0_CTRL_HI 0x0021
+#define PMU_DMEM_CH0_RANK_EN 0x0025
+#define PMU_DMEM_CH0_RANK_CFG 0x0032
+#define PMU_DMEM_SEARCH_WIN_COARSE 0x0037
+#define PMU_DMEM_SEARCH_WIN_FINE 0x0039
+#define PMU_DMEM_CH1_RANK_EN 0x0040
+#define PMU_DMEM_CH1_RANK_CFG 0x004d
+#define PMU_DMEM_LANE_SWEEP_CTRL 0x0059U
+#define PMU_DMEM_SLICE_PROFILE_TABLE 0x005aUL
+#define PMU_DMEM_VREF_FLAG_SIGNED 0x0062
+#define PMU_DMEM_VREF_DAC_CTRL 0x0072
+#define PMU_DMEM_MR19_OFF 0x8a /* Byte: DRAM protocol status offset */
+#define PMU_DMEM_TRAIN_PARAM_CTRL 0x0096
+#define PMU_DMEM_CAL_OVERRIDE_FLAG 0x00ce
+#define PMU_DMEM_WCK_TAP_C0R0 0x00d1U
+#define PMU_DMEM_WCK_TAP_C1R0 0x00d6U
+#define PMU_DMEM_WCK_TAP_C0R1 0x00dbU
+#define PMU_DMEM_WCK_TAP_C1R1 0x00e0U
+#define PMU_DMEM_BIST_PATTERN_LO 0x00e4
+#define PMU_DMEM_BIST_PATTERN_HI 0x00e5
+#define PMU_DMEM_DESKEW_OFFSET_OVERRIDE 0x00e6
+#define PMU_DMEM_TRAIN_STATUS 0x00e7
+#define PMU_DMEM_DQ_SWAP_BASE 0x0100
+#define PMU_DMEM_TRAIN_FEATURE_MASK 0x0102
+#define PMU_DMEM_DIFF_TAP_PAIR 0x0105
+#define PMU_DMEM_CAL_STEP 0x0106
+#define PMU_DMEM_SEARCH_WINDOW_BIAS_LO 0x0110
+#define PMU_DMEM_SEARCH_WINDOW_BIAS_HI 0x0111
+#define PMU_DMEM_HIGH_FREQ_FLAG 0x0401
+#define PMU_DMEM_CAL_ACTIVE_FLAG 0x0402
+#define PMU_DMEM_FREQ_MODE 0x0403
+#define PMU_DMEM_METRIC_TABLE_ENTRIES 0x0409
+#define PMU_DMEM_ACTIVE_CSR_OFFSET 0x041c
+#define PMU_DMEM_METRIC_COEFF_X 0x0420
+#define PMU_DMEM_METRIC_COEFF_Y 0x0422
+#define PMU_DMEM_ITER_COUNT 0x0428
+#define PMU_DMEM_2D_STEP_CTRL 0x0438
+#define PMU_DMEM_2D_STEP_PARAM 0x043a
+#define PMU_DMEM_CAL_STRIDE 0x047c
+#define PMU_DMEM_DESKEW_TARGET_DELAY 0x047e
+#define PMU_DMEM_CAL_STRUCT_BASE 0x09a4UL
+#define PMU_DMEM_AC_STEP_FLAG 0x0a7c
+#define PMU_DMEM_RANK_SLICE_COUNTS 0x0b20
+#define PMU_DMEM_RANK_SLICE_MAP 0x0b44
+#define PMU_DMEM_SEARCH_WINDOW_STEPS 0x0b64
+#define PMU_DMEM_SEARCH_WINDOW_SCALE 0x0b65
+#define PMU_DMEM_CAL_BYTE 0x0b67
+#define PMU_DMEM_SLICE_START 0x0b68
+#define PMU_DMEM_SLICE_END 0x0b69
+#define PMU_DMEM_RANK0_SLICE_START 0x0b6a
+#define PMU_DMEM_RANK_BOUNDARY 0x0b6b
+#define PMU_DMEM_RANK1_SLICE_START 0x0b6c
+#define PMU_DMEM_RANK1_SLICE_END 0x0b6d
+#define PMU_DMEM_DELAY_CAL_RESULTS 0x0b70
+#define PMU_DMEM_DESKEW_SWAP 0x0b72U
+#define PMU_DMEM_DESKEW_NORM 0x0b76U
+#define PMU_DMEM_BEST_TAP_ARR 0x0b7aU
+#define PMU_DMEM_LANE_ACTIVE_FLAGS 0x0b96
+#define PMU_DMEM_ACTIVE_SLICE_MASK_CAL 0x0b98
+#define PMU_DMEM_EYE_SAMPLE_BUF 0x0dc8
+#define PMU_DMEM_SWEEP_SAMPLE_BUF 0x6318
+#define PMU_DMEM_SWEEP_BLINK_OFF 0x6594U
+#define PMU_DMEM_CBT_DELAY_ARR_OFF 0xe798U
+
+/* ========================================================================== */
+/* Calibration Descriptor & Step Tracker Geometry */
+/* ========================================================================== */
+
+#define PMU_SLICE_DESC_SIZE 54U
+#define PMU_SLICE_DESC_COUNT 4U
+#define PMU_SLICE_DESC_TOTAL_BYTES (PMU_SLICE_DESC_COUNT * PMU_SLICE_DESC_SIZE)
+#define PMU_SLICE_DESC_MR1_OFF 0x01U
+#define PMU_SLICE_DESC_MR2_OFF 0x02U
+#define PMU_SLICE_DESC_MR3_OFF 0x03U
+#define PMU_SLICE_DESC_MR16_OFF 0x10U
+#define PMU_SLICE_DESC_MR41_OFF 0x29U
+#define PMU_TRACKER_SLOTS_BYTES 32U
+
+/* ========================================================================== */
+/* Synopsys DDR PHY Internal APB CSRs (0x90000000 Base) */
+/* ========================================================================== */
+
+#define PHY_DBYTE_SLICE_STRIDE 0x2000UL
+#define PHY_REG_PUB_BASE 0x90000000UL /* Half: PHY APB base aperture */
+#define PHY_REG_AC_COARSE_TIMING 0x90000020UL /* Half: PHY AC coarse timing reg */
+#define PHY_REG_AC_FINE_TIMING 0x90000024UL /* Half: PHY AC timing register */
+#define PHY_REG_STROBE_CAL_EN 0x90000148UL /* Half: Strobe cal mode enable */
+#define PHY_REG_STREAM_ALT_BASE 0x9001e000UL /* Half: Alt APB stream base */
+#define PHY_REG_DBYTE_BASE 0x90020000UL /* Base address for DBYTE slice CSRs */
+#define PHY_REG_OP_MODE 0x90020012UL /* Channel-indexed PHY operating mode */
+#define PHY_REG_DBYTE_XCVR_GATE 0x9002001eUL /* Half: DBYTE xcvr gate ctrl */
+#define PHY_REG_DBYTE_DELAY_CNT 0x90020028UL
+#define PHY_REG_DBYTE_ANALOG 0x90020054UL /* Half: DBYTE analog delay tap */
+#define PHY_REG_DBYTE_EYE_DELAY 0x9002009cUL /* Half: DBYTE eye sweep delay */
+#define PHY_REG_DBYTE_COARSE_STEP 0x900200e4UL /* Half: DBYTE coarse step reg */
+#define PHY_REG_DBYTE_DQ_SWIZZLE_BASE 0x90020100UL /* Half: DBYTE Slice 0 delay register base */
+#define PHY_REG_DBYTE_GATE_STAT 0x9002011aUL /* Half: DBYTE sample status */
+#define PHY_REG_DBYTE_WCK_DLY 0x90020146UL /* Half: DBYTE WCK delay reg */
+#define PHY_REG_DBYTE_COARSE 0x90020154UL /* Half: DBYTE coarse gate delay */
+#define PHY_REG_DBYTE_EYE_CTRL 0x90020158UL /* Half: DBYTE eye sweep ctrl */
+#define PHY_REG_DBYTE_DQ_DLY0 0x90020160UL /* Half: DBYTE DQ delay 0 reg */
+#define PHY_REG_DBYTE_SAMPLE_RDY 0x9002016aUL /* Half: DBYTE sample ready reg */
+#define PHY_REG_DBYTE_LCDL_STAT 0x9002016eUL /* Half: DBYTE LCDL gate status */
+#define PHY_REG_DBYTE_CAL_TRIG 0x90020176UL /* DBYTE calibration strobe sequence */
+#define PHY_REG_DBYTE_PERIOD 0x900201aaUL /* Half: DBYTE period meas reg */
+#define PHY_REG_DBYTE_PLL_TIER 0x900201b2UL /* Half: DBYTE PLL rate tier cfg */
+#define PHY_REG_DBYTE_PLL_LOCK 0x900201c0UL /* Half: DBYTE PLL lock status */
+#define PHY_REG_DBYTE_UPPER_GATE 0x90021f6aUL /* Half: DBYTE gate control reg */
+#define PHY_REG_DBYTE_BCAST_BASE 0x9003e000UL
+#define PHY_REG_DBYTE_BCAST_PULSE 0x9003e002UL /* Half: DBYTE bcast win valid */
+#define PHY_REG_DBYTE_BCAST_GATE 0x9003e00aUL /* Half: DBYTE bcast pulse ctrl */
+#define PHY_REG_DBYTE_BCAST_MODE 0x9003e012UL /* Half: DBYTE bcast mode reg */
+#define PHY_REG_DBYTE_BCAST_VREF 0x9003e014UL /* Half: DBYTE bcast Vref ctrl */
+#define PHY_REG_DBYTE_BCAST_WIDTH 0x9003e016UL /* Half: DBYTE bcast pulse width */
+#define PHY_REG_DBYTE_BCAST_XCVR_GATE 0x9003e01eUL /* Half: DBYTE bcast gate ctrl */
+#define PHY_REG_DBYTE_BCAST_TX_CTRL 0x9003e02cUL /* Half: DBYTE bcast ctrl */
+#define PHY_REG_DBYTE_BCAST_PWR_GATE 0x9003e034UL /* Half: DBYTE bcast power gate */
+#define PHY_REG_DBYTE_BCAST_STROBE 0x9003e080UL
+#define PHY_REG_DBYTE_BCAST_CBT 0x9003e144UL
+#define PHY_REG_DBYTE_BCAST_RX_EN 0x9003e14eUL
+#define PHY_REG_DBYTE_BCAST_EYE 0x9003e158UL
+#define PHY_REG_DBYTE_BCAST_DQ_DLY0 0x9003e160UL
+#define PHY_REG_DBYTE_BCAST_DQ_DLY1 0x9003e162UL
+#define PHY_REG_DBYTE_BCAST_DQ_DLY2 0x9003e166UL
+#define PHY_REG_DBYTE_BCAST_DQ_DLY3 0x9003e168UL
+#define PHY_REG_DBYTE_BCAST_SAMPLE_RDY 0x9003e16aUL
+#define PHY_REG_DBYTE_BCAST_DQS_DLY0 0x9003e172UL
+#define PHY_REG_DBYTE_BCAST_DQS_DLY1 0x9003e174UL
+#define PHY_REG_DBYTE_BCAST_TRIG 0x9003e176UL /* Half: DBYTE bcast gate commit */
+#define PHY_REG_DBYTE_BCAST_LANE 0x9003e17cUL
+#define PHY_REG_DBYTE_BCAST_WCK 0x9003e1b2UL
+#define PHY_REG_DBYTE_UPPER_SAMPLE_A 0x9003f16aUL /* Half: Upper DBYTE sample A */
+#define PHY_REG_DBYTE_UPPER_DLY0 0x9003fe20UL /* Half: DBYTE upper delay 0 */
+#define PHY_REG_DBYTE_UPPER_DLY1 0x9003fe22UL /* Half: DBYTE upper delay 1 */
+#define PHY_REG_DBYTE_UPPER_DLY2 0x9003fe24UL /* Half: DBYTE upper delay 2 */
+#define PHY_REG_DBYTE_UPPER_DLY3 0x9003fe26UL /* Half: DBYTE upper delay 3 */
+#define PHY_REG_DBYTE_UPPER_COARSE 0x9003ff54UL
+#define PHY_REG_DBYTE_UPPER_SAMPLE_B 0x9003ff6aUL /* Half: Upper DBYTE sample B */
+#define PHY_REG_DBYTE_UPPER_BCAST 0x9003ff7cUL
+#define PHY_REG_MASTER_BASE 0x90040000UL
+#define PHY_REG_CAL_STAT_CTRL0 0x90040008UL /* Half: PHY cal status ctrl A */
+#define PHY_REG_CAL_STAT_CTRL1 0x9004000cUL /* Half: PHY cal status ctrl */
+#define PHY_REG_MASTER_CFG 0x9004000eUL /* Master PUB configuration */
+#define PHY_REG_SLICE_STAT 0x90040018UL /* Slice / lane status register */
+#define PHY_REG_MASTER_DELAY 0x9004001aUL /* Half: Master profile delay */
+#define PHY_REG_CAL_STAT_CTRL2 0x90040022UL /* Half: PHY cal status ctrl */
+#define PHY_REG_PUB_ACK 0x9004002aUL /* Half: PUB status ack reg */
+#define PHY_REG_MASTER_PWR_GATE 0x90040034UL /* Half: Master power gate ctrl */
+#define PHY_REG_CLK_GATE 0x90040040UL /* Master PHY clock gate enable */
+#define PHY_REG_CLK_GATE_STAT 0x90040042UL /* Master clock gate acknowledge */
+#define PHY_REG_DESKEW_DLY0 0x90040044UL /* Deskew delay tap 0 */
+#define PHY_REG_DESKEW_DLY1 0x90040046UL /* Deskew delay tap 1 */
+#define PHY_REG_TIMER_DUMMY 0x90040048UL /* Half: Dummy APB turnaround read register */
+#define PHY_REG_CLK_TIMING_CTRL 0x9004004eUL /* Clock timing control */
+#define PHY_REG_PAD_CAL_CTRL 0x90040050UL /* Half: Master pad cal ctrl */
+#define PHY_REG_VREF_TRIM_CFG 0x90040054UL
+#define PHY_REG_CA_VREF_DAC 0x90040058UL /* Half: CA Vref DAC ctrl A */
+#define PHY_REG_PLL_CLK_CTRL 0x90040062UL /* Master PLL & clock control */
+#define PHY_REG_VREF_CTRL 0x90040082UL /* Receiver Vref control */
+#define PHY_REG_VREF_STAT 0x90040084UL /* Half: Gate settle timing reg */
+#define PHY_REG_DRAM_TYPE_CFG 0x90040088UL
+#define PHY_REG_GATE_STAT 0x90040092UL /* Half: Gate pulse status reg */
+#define PHY_REG_ZQ_STAT 0x90040094UL /* Half: ZQ telemetry status 0 */
+#define PHY_REG_PHY_STATUS 0x900400c0UL /* Master PHY status & receiver clock */
+#define PHY_REG_MASTER_CAL_CTRL 0x900400dcUL
+#define PHY_REG_CH0_ANALOG_PWR 0x900400e4UL /* Ch0 analog power configuration */
+#define PHY_REG_CH1_ANALOG_PWR 0x900400e6UL /* Ch1 analog power configuration */
+#define PHY_REG_RESET_TRIGGER 0x900400f2UL /* Soft reset pulse toggle */
+#define PHY_REG_GATE_MASK 0x900400f4UL
+#define PHY_REG_GATE_CTRL 0x900400fcUL
+#define PHY_REG_CH0_RANK_CFG 0x90040100UL
+#define PHY_REG_CH1_RANK_CFG 0x90040102UL
+#define PHY_REG_RFIFO_CFG 0x90040110UL /* Half: RFIFO cfg base */
+#define PHY_REG_RFIFO_SUPPRESS 0x90040116UL /* Half: RFIFO suppress ctrl */
+#define PHY_REG_MASTER_INIT_CTRL 0x90040120UL /* Half: Master init ctrl */
+#define PHY_REG_CBT_CTRL 0x90040144UL /* CBT control register */
+#define PHY_REG_CAL_TRIG 0x90040146UL /* Calibration strobe trigger */
+#define PHY_REG_CAL_STAT 0x90040148UL /* Calibration status & configuration */
+#define PHY_REG_MASTER_COARSE_CFG 0x90040154UL /* Half: Master coarse gate cfg */
+#define PHY_REG_CBT_STAT0 0x90040180UL /* Half: CBT status reg */
+#define PHY_REG_CBT_STAT1 0x90040182UL /* Half: CBT status reg */
+#define PHY_REG_CBT_STAT2 0x90040184UL /* Half: CBT status reg */
+#define PHY_REG_AC_PLL_CTRL 0x900401c0UL
+#define PHY_REG_CBT_PLL 0x900401f0UL /* CBT PLL control register */
+#define PHY_REG_CMD_DISPATCH 0x90040238UL /* Command dispatch strobe */
+#define PHY_REG_CLK_PHASE_BASE 0x90040600UL
+#define PHY_REG_CLK_PHASE_DELAY 0x90040628UL
+#define PHY_REG_PUB_SEC_BASE 0x9005e000UL
+#define PHY_REG_SEC_CLK_GATE 0x9005e0a4UL /* Secondary clock gate toggle */
+#define PHY_REG_SEC_CAL_CTRL 0x9005e0b2UL
+#define PHY_REG_SEC_PUB_CTRL0 0x9005e0eeUL /* Half: Sec PUB ctrl */
+#define PHY_REG_PUB_CAL_STROBE 0x9005e140UL /* Secondary calibration strobe */
+#define PHY_REG_SEC_PUB_CTRL1 0x9005e14aUL /* Half: Sec PUB ctrl */
+#define PHY_REG_AC_BASE 0x90060000UL /* Half: AC slice profile base */
+#define PHY_AC_PIN_DELAY_BASE 0x90060002UL /* AC pin delay base */
+#define PHY_AC_BASE_OFFSET_CTRL 0x90060004UL /* AC base offset control */
+#define PHY_REG_AC_SLICE_CFG 0x9006005cUL
+#define PHY_AC_SWIZZLE_S0_BASE 0x90060120UL /* AC slice 0 pin swizzle base */
+#define PHY_REG_AC_PLL_LOCK 0x9006017eUL /* Half: AC PLL lock status B */
+#define PHY_AC_MASTER_DESKEW0 0x90061002UL /* AC master deskew 0 */
+#define PHY_AC_MASTER_DESKEW1 0x90061202UL /* AC master deskew 1 */
+#define PHY_AC_GATE_CTRL_RESET 0x90061e02UL /* AC gate control reset */
+#define PHY_AC_SWIZZLE_S1_BASE 0x90062120UL /* AC slice 1 pin swizzle base */
+#define PHY_AC_PROFILE_LANE_BASE 0x9007e002UL /* AC profile lane base */
+#define PHY_AC_PROFILE_MASTER_CTRL 0x9007e004UL /* AC profile master control */
+#define PHY_REG_PLL_STATUS 0x9007e034UL /* PLL lock status */
+#define PHY_REG_AC_DRIVE_CFG0 0x9007e05cUL
+#define PHY_REG_AC_DRIVE_CFG1 0x9007e066UL
+#define PHY_REG_AC_DRIVE_CFG2 0x9007e068UL
+#define PHY_REG_AC_DRIVE_CFG3 0x9007e080UL
+#define PHY_REG_AC_DRIVE_CFG4 0x9007e082UL
+#define PHY_REG_AC_DRIVE_CFG5 0x9007e084UL
+#define PHY_REG_AC_DRIVE_CFG6 0x9007e086UL
+#define PHY_REG_AC_BCAST_GATE 0x9007e108UL
+#define PHY_REG_CSR_RESULT_STREAM 0x90082000UL /* Half: PHY CSR result stream */
+#define PHY_REG_BIST_PLL_CFG 0x900e0022UL /* Half: AC ctrl read source */
+#define PHY_REG_DESKEW_QUAD_TAP 0x900e0048UL /* Half: Deskew quad-tap base */
+#define PHY_REG_AC_PAT_CTRL 0x900e00c0UL
+#define PHY_REG_PLL_CFG 0x900e00c2UL /* Master PLL configuration parameter 0 */
+#define PHY_REG_AC_TX_EN_BCAST 0x900e00caUL /* AC transmitter broadcast enable */
+#define PHY_REG_AC_BCAST 0x900fe022UL /* Half: AC bcast write target */
+#define PHY_REG_BIST_CTRL_MODE 0x900fe0c0UL /* Half: AC bcast test pattern A */
+#define PHY_REG_AC_BCAST_PAT_DATA 0x900fe0caUL /* Half: AC bcast test pattern B */
+#define PHY_REG_AC_BCAST_PAT_CFG 0x900fe1acUL /* Half: AC bcast pattern ctrl B */
+#define PHY_REG_AC_BCAST_PAT_CTRL 0x900fe1aeUL /* Half: AC bcast pattern ctrl */
+#define PHY_REG_BIST_PAT_CTRL0 0x900ffe4cUL
+#define PHY_REG_BIST_PAT_CTRL1 0x900ffe4eUL
+#define PHY_REG_CSR_SCRATCH_BASE 0x90120052UL /* Half: PUB CSR scratchpad */
+#define PHY_REG_CAL_LATCH_STREAM 0x90121000UL
+#define PHY_REG_CAL_RESTORE_TRIG 0x9013e050UL
+#define PHY_REG_SHADOW_TRACK_BASE 0x90140000UL /* Half: Shadow track aperture */
+#define PHY_REG_MAILBOX_STAT 0x90180008UL /* Mailbox status */
+#define PHY_REG_MAILBOX_STREAM 0x90180064UL /* Mailbox stream low */
+#define PHY_REG_MAILBOX_TRIG 0x90180066UL /* Mailbox interrupt trigger */
+#define PHY_REG_PUB_CFG 0x90180100UL
+#define PHY_REG_PUB_CTL 0x901801dcUL /* PUB control */
+#define PHY_REG_SHADOW_CNT 0x9018021eUL /* Half: Shadow stream count reg */
+
+/* ========================================================================== */
+/* ARC EM4 Auxiliary Timer Registers */
+/* ========================================================================== */
+
+#define PMU_AUX_RTC_CTRL 0x103 /* AUX 259: RTC control / enable register */
+#define PMU_AUX_RTC_LOW 0x104 /* AUX 260: RTC low 32-bit counter */
+#define PMU_AUX_RTC_HIGH 0x105 /* AUX 261: RTC high 32-bit counter */
+
+/* ========================================================================== */
+/* PMU Firmware Identifiers & Assertion Error Codes */
+/* ========================================================================== */
+
+#define PMU_FW_VERSION_ID 0x230bU
+#define PHY_CSR_SCRATCH_MAGIC_LO 0x55aaU
+#define PHY_CSR_SCRATCH_MAGIC_HI 0xaa55U
+#define PMU_ERR_PLL_LOCK_TIMEOUT 0x00240000UL
+#define PMU_ERR_CLK_GATE_TIMEOUT 0x00250000UL
+#define PMU_ERR_ZQ_CAL_LOW_FREQ 0x00430000UL
+#define PMU_ERR_ZQ_CAL_HIGH_FREQ 0x00470000UL
+#define PMU_ERR_EYE_RANK_LIMIT_1 0x00920001UL
+#define PMU_ERR_EYE_RANK_MODE_INVALID 0x00930001UL
+#define PMU_ERR_RANK_BOUNDS_BASE 0x009d0003UL
+#define PMU_ERR_EYE_STEP_ALIGN_MISMATCH 0x00a40003UL
+#define PMU_ERR_DESKEW_SAMPLE_CHECK 0x01160000UL
+#define PMU_ERR_RX_FIFO_SCAN_OVERFLOW 0x011c0000UL
+#define PMU_ERR_LPCA_CLK_GATE_PRECOND 0x01280000UL
+#define PMU_ERR_AC_STEP_COORD_ADJUST 0x019f0003UL
+#define PMU_ERR_BIST_PATTERN_IDX_INVALID 0x01a40001UL
+#define PMU_ERR_BIST_VALIDATION_MODE 0x01a50000UL
+#define PMU_ERR_BIST_CMD_MODE_INVALID 0x01a80001UL
+#define PMU_ERR_EYE_SLICE_REG_SOURCE 0x01ab0000UL
+#define PMU_ERR_EYE_WINDOW_TOO_NARROW 0x01b10006UL
+#define PMU_ERR_EYE_SWEEP_CONVERGENCE 0x01f60000UL
+#define PMU_ERR_CMD_SUBCODE_INVALID 0x02190001UL
+#define PMU_ERR_CSR_BUS_ACCESS_FAULT 0x02240000UL
+#define PMU_ERR_WCK_EYE_ZERO_WIDTH 0x022e0003UL
+#define PMU_ERR_VREF_STEP_DIR_INVALID 0x02350000UL
+#define PMU_ERR_BIST_PATTERN_MODE_RANGE 0x027e0001UL
+#define PMU_ERR_COARSE_DELAY_OVERFLOW 0x02a80000UL
+
+#endif /* _DWC_DDRPHY_PMU_REGS_H */
diff --git a/arch/arm/mach-sunxi/pmu/pmu.ld b/arch/arm/mach-sunxi/pmu/pmu.ld
new file mode 100644
index 00000000000..79ff0f8c0c4
--- /dev/null
+++ b/arch/arm/mach-sunxi/pmu/pmu.ld
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Copyright (C) 2026 Alastair D'Silva <alastair@d-silva.org>
+ *
+ * Linker script for Allwinner A733 LPDDR5 ARC EM4 PMU Firmware
+ */
+
+OUTPUT_FORMAT("elf32-littlearc", "elf32-littlearc", "elf32-littlearc")
+OUTPUT_ARCH(arc)
+ENTRY(pmu_vector_table)
+
+SECTIONS
+{
+ .vectors 0x00000000 : {
+ KEEP(*(.vectors))
+ } = 0xe078
+
+ .text 0x00000080 : {
+ KEEP(*(.text.pmu_start))
+ KEEP(*(.text.pmu_default_exception_handler))
+ *(.text .text.*)
+ } = 0xe078
+
+ .rodata : { *(.rodata*) }
+ .data : { *(.data*) *(.sdata*) }
+ .bss : { *(.bss*) *(.sbss*) *(COMMON) }
+
+ /DISCARD/ : {
+ *(.comment)
+ *(.note*)
+ *(.eh_frame*)
+ *(.ARC.attributes)
+ }
+}
+
+ASSERT(SIZEOF(.rodata) == 0 && SIZEOF(.data) == 0 && SIZEOF(.bss) == 0,
+ "ARC ICCM is execute-only; .rodata/.data/.bss sections are forbidden")
+ASSERT(. <= 0x00010000, "PMU firmware exceeds 64 KiB ICCM")
diff --git a/arch/arm/mach-sunxi/pmu/pmu_fw.h b/arch/arm/mach-sunxi/pmu/pmu_fw.h
new file mode 100644
index 00000000000..e71955bdf5c
--- /dev/null
+++ b/arch/arm/mach-sunxi/pmu/pmu_fw.h
@@ -0,0 +1,553 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Copyright (C) 2026 Alastair D'Silva <alastair@d-silva.org>
+ *
+ * Common declarations and external symbol prototypes for standalone
+ * Allwinner A733 DDR PHY PMU firmware modules.
+ */
+
+#ifndef _SUNXI_PMU_FW_H
+#define _SUNXI_PMU_FW_H
+
+#include "dwc_ddrphy_pmu_regs.h"
+
+/*
+ * ARC EM4 global architectural state registers pinned across all translation
+ * units via -ffixed-r26 and -ffixed-r27:
+ * r26 (gp) = 0x80000400 (struct pmu_sdata base in DCCM)
+ * r27 (fp) = reserved by firmware startup so GCC never clobbers r27
+ */
+
+/**
+ * dmem_ptr() - Convert a DCCM byte offset into an ARC DMEM pointer
+ * @offset: Byte offset within DCCM (or full 0x8000xxxx address)
+ *
+ * Return: Pointer to the target DCCM address.
+ */
+static inline void *dmem_ptr(u32 offset)
+{
+ return (void *)(uintptr_t)(PMU_DMEM_BASE | offset);
+}
+
+/**
+ * dmem_read8() - Read an unsigned 8-bit byte from DCCM
+ * @offset: Byte offset within DCCM (or full 0x8000xxxx address)
+ *
+ * Return: 8-bit value read from DCCM.
+ */
+static inline u8 dmem_read8(u32 offset)
+{
+ return *(__volatile__ u8 *)dmem_ptr(offset);
+}
+
+/**
+ * dmem_read16() - Read an unsigned 16-bit halfword from DCCM
+ * @offset: Byte-aligned halfword offset within DCCM
+ *
+ * Return: 16-bit value read from DCCM.
+ */
+static inline u16 dmem_read16(u32 offset)
+{
+ return *(__volatile__ u16 *)dmem_ptr(offset);
+}
+
+/**
+ * dmem_read32() - Read an unsigned 32-bit word from DCCM
+ * @offset: Word-aligned offset within DCCM
+ *
+ * Return: 32-bit value read from DCCM.
+ */
+static inline u32 dmem_read32(u32 offset)
+{
+ return *(__volatile__ u32 *)dmem_ptr(offset);
+}
+
+/**
+ * dmem_write8() - Write an unsigned 8-bit byte to DCCM
+ * @offset: Byte offset within DCCM (or full 0x8000xxxx address)
+ * @val: 8-bit value to write
+ */
+static inline void dmem_write8(u32 offset, u8 val)
+{
+ *(__volatile__ u8 *)dmem_ptr(offset) = val;
+}
+
+/**
+ * dmem_write16() - Write an unsigned 16-bit halfword to DCCM
+ * @offset: Halfword-aligned offset within DCCM
+ * @val: 16-bit value to write
+ */
+static inline void dmem_write16(u32 offset, u16 val)
+{
+ *(__volatile__ u16 *)dmem_ptr(offset) = val;
+}
+
+/**
+ * dmem_write32() - Write an unsigned 32-bit word to DCCM
+ * @offset: Word-aligned offset within DCCM
+ * @val: 32-bit value to write
+ */
+static inline void dmem_write32(u32 offset, u32 val)
+{
+ *(__volatile__ u32 *)dmem_ptr(offset) = val;
+}
+
+/**
+ * phy_read16() - Read a 16-bit DDR PHY CSR register in ARC MMIO space
+ * @addr: ARC PHY register address (0x9000xxxx)
+ *
+ * Return: 16-bit CSR value.
+ */
+static inline u16 phy_read16(u32 addr)
+{
+ return *(__volatile__ u16 *)(uintptr_t)addr;
+}
+
+/**
+ * phy_write16() - Write a 16-bit DDR PHY CSR register in ARC MMIO space
+ * @addr: ARC PHY register address (0x9000xxxx)
+ * @val: 16-bit CSR value to write
+ */
+static inline void phy_write16(u32 addr, u16 val)
+{
+ *(__volatile__ u16 *)(uintptr_t)addr = val;
+}
+
+/**
+ * struct pmu_sdata - PMU small-data (.sdata) state block at DCCM 0x80000400
+ * @cbt_dly_init: Clamped CBT delay line initial offset (0x400)
+ * @high_freq_flag: High-frequency operating mode flag (0x401)
+ * @cal_active_flag: Active calibration stage flag (0x402)
+ * @freq_mode: Frequency mode / CBT lane limit code (0x403)
+ * @timer_shift: Hardware timer prescaler shift factor (0x404)
+ * @stage_step_mode: Stage 10 margin status / step mode (0x405)
+ * @stage_rank_idx: Active calibration rank index (0x406)
+ * @stage_slice_idx: Active calibration slice index & flags (0x407)
+ * @stage_lane_idx: Active calibration lane index & mode (0x408)
+ * @metric_tbl_entries: Calibration metric table entry count (0x409)
+ * @rsvd0: Alignment padding halfword (0x40a)
+ * @active_slice_cnt: Active DBYTE slice count (0x40c)
+ * @active_ch_cnt: Active channel count / quadratic cost weight W (0x410)
+ * @rsvd1: Reserved word (0x414)
+ * @shadow_trace_ptr: Shadow trace buffer pointer (0x418)
+ * @active_csr_offset: Active channel PHY CSR address offset (0x41c)
+ * @fw_version_id: Firmware version ID / quadratic cost weight X (0x420)
+ * @num_ranks: Active rank count / quadratic cost weight Y (0x422)
+ * @rsvd_424: Reserved alignment word (0x424)
+ * @iter_count: Calibration stage iteration counter (0x428)
+ * @cal_save_cfg: Saved hardware calibration configuration mask (0x42a)
+ * @dbyte_slice_cnt: Active DBYTE slice limit (0x42c)
+ * @rsvd2: Alignment padding bytes (0x42d..0x42f)
+ * @rfifo_dly: Per-slice RFIFO delay parameters across 4 slices (0x430..0x437)
+ * @step_2d_ctrl: 2D eye sweep step control (0x438)
+ * @step_2d_param: 2D eye sweep step parameter (0x43a)
+ * @rsvd3: Alignment padding byte (0x43b)
+ * @master_dly_tbl: Master delay table [channel][rank] (0x43c..0x443)
+ * @ac_profile_dly: AC profile delay halfwords (0x444..0x447)
+ * @ac_profile_cfg: AC profile step configuration bytes (0x448..0x453)
+ * @saved_cal_stat: Saved calibration status word (0x454)
+ * @cal_marker_a: Calibration step marker A (0x458)
+ * @cbt_state: CBT state latch (0x45a)
+ * @cbt_step_ext: CBT step extension flag (0x45b)
+ * @cbt_cal_active: CBT calibration active flag (0x45c)
+ * @rsvd4: Alignment padding byte (0x45d)
+ * @cal_marker_b: Calibration step marker B (0x45e)
+ * @active_slice_mask: Active slice bitmask (0x460)
+ * @tracker_popcnt_acc: Tracker popcount accumulator (0x464)
+ * @ch_range_lim: Channel active range limit (0x468)
+ * @step_mode_ovr: Channel step mode override (0x469)
+ * @cbt_config: CBT configuration value (0x46a)
+ * @cbt_step_stat: CBT step status flag (0x46b)
+ * @cal_cmd_tbl: Shadow calibration command table [0..3] (0x46c..0x46f)
+ * @active_slice_idx: Active slice index (byte 0 is cal_cmd_tbl[4]) (0x470)
+ * @slice_timing_desc: Per-rank/slice timing descriptor bytes (0x474..0x47b)
+ * @cal_stride: Normalized power-of-2 step stride (0x47c)
+ * @deskew_target_dly: Pad timing / deskew target delay (0x47e)
+ * @cal_seq_map: 21-byte calibration sequence index map (0x480..0x494)
+ * @rsvd5: Alignment padding to 0x4c0 (0x495..0x4bf)
+ * @rank_ch_tap_tbl: Per-rank/channel tap table (0x4c0..0x4c3)
+ * @best_tap: Best WCK read-gate tap per channel and rank (0x4c4..0x4c7)
+ */
+struct pmu_sdata {
+ u8 cbt_dly_init;
+ u8 high_freq_flag;
+ u8 cal_active_flag;
+ u8 freq_mode;
+ u8 timer_shift;
+ u8 stage_step_mode;
+ u8 stage_rank_idx;
+ u8 stage_slice_idx;
+ u8 stage_lane_idx;
+ u8 metric_tbl_entries;
+ u16 rsvd0;
+ u32 active_slice_cnt;
+ u32 active_ch_cnt;
+ u32 rsvd1;
+ u32 shadow_trace_ptr;
+ u32 active_csr_offset;
+ u16 fw_version_id;
+ u16 num_ranks;
+ u32 rsvd_424;
+ u16 iter_count;
+ u16 cal_save_cfg;
+ u8 dbyte_slice_cnt;
+ u8 rsvd2[3];
+ u16 rfifo_dly[4];
+ u16 step_2d_ctrl;
+ u8 step_2d_param;
+ u8 rsvd3;
+ u16 master_dly_tbl[2][2];
+ u16 ac_profile_dly[2];
+ u8 ac_profile_cfg[12];
+ u32 saved_cal_stat;
+ u16 cal_marker_a;
+ u8 cbt_state;
+ u8 cbt_step_ext;
+ u8 cbt_cal_active;
+ u8 rsvd4;
+ u16 cal_marker_b;
+ u32 active_slice_mask;
+ s32 tracker_popcnt_acc;
+ u8 ch_range_lim;
+ u8 step_mode_ovr;
+ u8 cbt_config;
+ u8 cbt_step_stat;
+ u8 cal_cmd_tbl[4];
+ u32 active_slice_idx;
+ u8 slice_timing_desc[8];
+ u16 cal_stride;
+ u16 deskew_target_dly;
+ u8 cal_seq_map[21];
+ u8 rsvd5[43];
+ u8 rank_ch_tap_tbl[2][2];
+ u8 best_tap[2][2];
+};
+
+/**
+ * struct pmu_cal_tracker - Multi-channel calibration step tracker (0x80000d9c)
+ * @slot0: Primary 8-halfword channel 0 command slot (0x00..0x0f)
+ * @slot1: Secondary 8-halfword channel 1 command slot (0x10..0x1f)
+ * @slot1_mode: Channel 1 step mode control halfword (0x20)
+ * @repeat_cnt: Step repeat count minus one (0x22)
+ * @slot0_mode: Channel 0 step mode control halfword (0x24)
+ * @ext_flag: Step extension flag halfword (0x26)
+ * @commit_flag: Pending step commit flag halfword (0x28)
+ * @rsvd0: Alignment padding halfword (0x2a)
+ */
+struct pmu_cal_tracker {
+ u16 slot0[8];
+ u16 slot1[8];
+ u16 slot1_mode;
+ u16 repeat_cnt;
+ u16 slot0_mode;
+ u16 ext_flag;
+ u16 commit_flag;
+ u16 rsvd0;
+};
+
+/**
+ * pmu_sdata() - Access the PMU small-data (.sdata) structure via r26 (gp)
+ *
+ * Return: Volatile pointer to struct pmu_sdata at DCCM 0x80000400.
+ */
+#define PMU_SDATA_BASE 0x80000400UL
+
+static inline __volatile__ struct pmu_sdata *pmu_sdata(void)
+{
+ return (__volatile__ struct pmu_sdata *)PMU_SDATA_BASE;
+}
+
+_Static_assert(sizeof(struct pmu_sdata) == 0xc8,
+ "struct pmu_sdata size mismatch");
+_Static_assert(offsetof(struct pmu_sdata, cbt_dly_init) == 0x00,
+ "cbt_dly_init offset mismatch");
+_Static_assert(offsetof(struct pmu_sdata, active_ch_cnt) == 0x10,
+ "active_ch_cnt offset mismatch");
+_Static_assert(offsetof(struct pmu_sdata, shadow_trace_ptr) == 0x18,
+ "shadow_trace_ptr offset mismatch");
+_Static_assert(offsetof(struct pmu_sdata, active_csr_offset) == 0x1c,
+ "active_csr_offset offset mismatch");
+_Static_assert(offsetof(struct pmu_sdata, fw_version_id) == 0x20,
+ "fw_version_id offset mismatch");
+_Static_assert(offsetof(struct pmu_sdata, num_ranks) == 0x22,
+ "num_ranks offset mismatch");
+_Static_assert(offsetof(struct pmu_sdata, best_tap) == 0xc4,
+ "best_tap offset mismatch");
+
+/**
+ * pmu_tracker() - Access the primary calibration step tracker in DCCM
+ *
+ * Return: Volatile pointer to struct pmu_cal_tracker at DCCM 0x80000d9c.
+ */
+static inline __volatile__ struct pmu_cal_tracker *pmu_tracker(void)
+{
+ return (__volatile__ struct pmu_cal_tracker *)PMU_DMEM_CAL_TRACKER;
+}
+
+/**
+ * struct pmu_cal_bist_cmd_ctx - BIST command strobe context block
+ * @ctrl0: Primary command control word 0
+ * @ctrl1: Secondary command control word 1
+ * @ctrl2: Tertiary command control word 2
+ * @strobe_cmd_a: Strobe command pattern word C
+ * @strobe_cmd_b: Strobe command pattern word 10
+ * @stride_minus_1: Loop stride count minus one
+ * @rsvd0: Alignment padding halfword
+ * @csr_offset: Target PHY CSR channel/slice address offset
+ * @slice_delays: Per-slice 16-bit delay values across 4 slices
+ */
+struct pmu_cal_bist_cmd_ctx {
+ u32 ctrl0;
+ u32 ctrl1;
+ u32 ctrl2;
+ u32 strobe_cmd_a;
+ u32 strobe_cmd_b;
+ u16 stride_minus_1;
+ u16 rsvd0;
+ u32 csr_offset;
+ u16 slice_delays[4];
+};
+
+/* Cross-Module PMU Firmware Function Prototypes */
+void *memcpy(void *dest, const void *src, size_t n);
+void *memset(void *s, int c, size_t n);
+void pmu_ac_lane_profile_setup(void);
+void pmu_adjust_step_limit(u8 *pair, u32 flag);
+void pmu_assert_or_halt(u32 cond, u32 assert_code);
+u32 pmu_bist_pattern_bit_get(u32 shift, u32 mask);
+void pmu_cal_bist_cmd_strobe_dispatch(uintptr_t ctx_addr,
+ u32 pattern_mode, u32 cmd_mode, u32 target_mask,
+ uintptr_t slice_flags_addr, u32 status_idx, u32 setup_flags);
+void pmu_cal_bist_pattern_setup(u32 pattern_idx, u32 slice_or_lane,
+ u32 enable_deskew_reset, u32 validation_mode);
+void pmu_cal_bist_search_win_setup(u32 stage);
+u32 pmu_cal_cbt_desc_unpack(u32 rank, u32 channel);
+void pmu_cal_channel_rank_bounds_verify(u32 err_id);
+u32 pmu_cal_channel_rank_config_get(u32 rank, u16 *out_cfg,
+ u8 *out_mode_sel, u8 *out_num_slices);
+void pmu_cal_dbyte_bcast_pulse_width_clear(const u8 *cfg);
+void pmu_cal_dbyte_delay_backup_all_ch(void);
+void pmu_cal_dbyte_deskew_fine_tune(u32 mode);
+void pmu_cal_dbyte_gate_mask_clear(void);
+void pmu_cal_dbyte_pattern_loop_exec(u32 reg_offset,
+ u16 pattern_mode, void *dest_buf, void *scratch_buf,
+ u32 bound_start, u32 bound_end, u32 flags);
+s16 pmu_cal_dbyte_rx_fifo_reset_poll(u32 channel, u32 mode_sel,
+ u32 enable_scan, void *unused_buf);
+void pmu_cal_dbyte_sample_ready_poll_all(u32 flags);
+u32 pmu_cal_delay_line_accum_dispatch(u32 count, u32 limit);
+void pmu_cal_delay_line_init(void);
+void pmu_cal_deskew_quad_tap_set(u32 ch, u16 tap0, u16 tap1,
+ u16 tap2, u16 tap3);
+u32 pmu_cal_eye_margin_quadratic_cost_calc(u32 val, u32 target,
+ u32 min_bound, u32 max_bound, u8 *out_overflow);
+u32 pmu_cal_eye_margin_sample_latch(u32 mask, u32 lane);
+void pmu_cal_eye_param_block_init(u16 *param_block, u32 p1, u32 p2,
+ u32 p3);
+void pmu_cal_eye_table_restore(u8 *dest, u32 offset);
+void pmu_cal_eye_table_save(const u8 *src, u32 offset);
+void pmu_cal_gate_eye_master_trigger(void);
+void pmu_cal_lane_deskew_sweep_exec(u32 channel);
+u32 pmu_cal_lane_error_status_check(u32 lane, u32 phase);
+void pmu_cal_lane_mask_and_all(u32 base, u32 offset, u32 mask);
+void pmu_cal_lane_mask_or_all(u32 base, u32 offset, u32 mask);
+void pmu_cal_lane_window_margin_eval(u32 mode);
+void pmu_cal_lane_window_sweep_coordinator(u32 ch_mask, u32 wide_range);
+void pmu_cal_lcdl_coarse_fine_pair_pack(u32 ch_idx, u32 threshold,
+ u32 limit, const u8 *pair_a,
+ const u8 *pair_b, const u8 *pair_c,
+ const u8 *pair_d, void *table_base);
+void pmu_cal_margin_dir_get(void);
+void pmu_cal_margin_matrix_scan_eval(u32 channel, u32 rank,
+ u32 slice, u32 mode, u32 flag, u32 step);
+u16 pmu_cal_marker_status_update(void);
+u16 pmu_cal_marker_verify(void);
+void pmu_cal_marker_verify_and_buffer_clear(u32 flags);
+void pmu_cal_marker_win_step_prog(u32 marker0, u32 marker1,
+ u32 win_param, u32 win_mode);
+void pmu_cal_multi_lane_deskew_sweep(u32 type, u32 ch_mask,
+ u32 lane_mask);
+void pmu_cal_multi_rank_step_commit(u32 rank_mode, u32 steps_table,
+ u32 eval_error_mask, u32 skip_mask_low, u32 skip_mask_high,
+ u32 pulse_coarse_pre, u32 check_settle_flags);
+void pmu_cal_multi_rank_timing_step(u32 rank_mode, u32 steps_table,
+ u32 eval_error_mask, u32 skip_mask_low, u32 skip_mask_high,
+ u32 check_settle_flags);
+void pmu_cal_pad_impedance_lock(void);
+void pmu_cal_pad_timing_delay_set(u32 bypass_min_clamp, u32 steps,
+ u32 use_fixed_unit);
+void pmu_cal_pad_timing_step_settle(void);
+u32 pmu_cal_pad_voltage_dac_set(u32 val, u32 mode);
+void pmu_cal_param_unpack(u32 val, u8 *out);
+void pmu_cal_phase_detector_edge_align(void);
+void pmu_cal_pll_lock_settle_spin(u16 *out_dbyte_pll, u16 *out_ac_pll);
+void pmu_cal_pulse_burst_send(u32 burst_type, u32 count, u32 slice);
+void pmu_cal_pulse_seq_coordinator(u32 ch_idx, u32 rank, u32 param1, u32 flags);
+void pmu_cal_rank_deskew_matrix_commit(u32 channel, u32 rank, void *buf);
+u32 pmu_cal_rank_margin_window_check(u32 tap_ptr, u32 channel,
+ u32 slice_mask);
+u8 pmu_cal_rank_offset_get(u32 rank, u32 channel);
+void pmu_cal_rank_param_commit(u32 freq_pt, u32 mode_cfg, u32 step_lim,
+ u32 type);
+void pmu_cal_rank_rx_en_delay_commit(u32 channel, u16 rank_flags,
+ u32 cal_stage_mode, u32 stage);
+void pmu_cal_rx_dqs_delay_center(void);
+void pmu_cal_search_win_init(u32 base);
+void pmu_cal_search_win_max_calc(u32 channel, u32 rank, u32 dir);
+void pmu_cal_seq_table_timed_replay(const void *seq_table,
+ u32 total_bytes);
+u32 pmu_cal_sequence_pulse_send(u32 cmd_flags, u32 pulse_type,
+ u32 phase_id, u32 param_code, u32 tap_idx, u32 slice_mask,
+ u32 extra);
+void pmu_cal_slice_deskew_result_commit(u32 channel, void *buf,
+ u32 flags);
+u32 pmu_cal_slice_mask_eval(void *table_buf);
+void pmu_cal_slice_step_diff_commit(u32 slice_idx, u16 *out_buf,
+ u32 cal_mode);
+void pmu_cal_slice_step_diff_latch(u32 timing, u32 wait_ack);
+void pmu_cal_slice_step_eval_sweep(void *results_buf, u32 sweep_mode,
+ u32 cal_mode, u32 channel_rank, u32 unused,
+ u32 timing_val, u32 overflow_flag,
+ const u16 *slice_delay_lut, u32 bound_flag,
+ u32 remap_flag, const void *sweep_ctx);
+void pmu_cal_slice_timing_latch(u32 timing, u32 wait_ack);
+void pmu_cal_delay_accum_dispatch(u32 dram_type, u32 count,
+ void *slice_base);
+u16 pmu_cal_stride_get(void);
+u32 pmu_cal_strobe_pulse_init(void);
+void pmu_cal_struct_mask_remap(void);
+void pmu_cal_timing_margin_envelope_scan(u32 channel, u32 rank, void *buf);
+void pmu_cal_timing_table_matrix_init(u8 *buf);
+u32 pmu_cal_tracker_csr_commit(__volatile__ void *tracker);
+void pmu_cal_tracker_reset_and_apb_zero(const void *buf, u32 len);
+u32 pmu_cal_tracker_step_advance(u32 marker, u32 count, u32 mode);
+void pmu_cal_training_flags_gate_update(u8 *descriptor);
+void pmu_cal_vref_dac_step_adjust(u32 stage_step);
+void pmu_cal_vref_search_center_update(void);
+void pmu_cal_vref_trim_cfg_latch(void);
+void pmu_cal_wck_ck_align_pulse_send(u32 ch, u32 rank);
+void pmu_cal_win_bounds_clear(void);
+u32 pmu_cal_window_params_dispatch(const u8 *params_buf,
+ u16 base_val, u32 mode_flags);
+void pmu_cbt_3phase_pulse_dispatch(u32 cal_mode,
+ const u8 *step_table, u32 step_idx, u32 flags);
+void pmu_cbt_3phase_pulse_seq(void);
+void pmu_cbt_cal_pulse_seq(u32 pulse_cmd);
+u32 pmu_cbt_delay_offset_lookup(u32 rank, u32 channel);
+void pmu_cbt_deskew_pulse_train(u32 pulse_cmd, u32 is_alternate);
+void pmu_cbt_strobe_pulse_train(u32 pulse_cmd, u32 is_stage2);
+void pmu_cbt_timing_pulse_coordinator(u16 cal_mode, u32 phase_sel);
+u32 pmu_channel_rank_avail_check(u32 channel, u32 rank);
+void pmu_channel_timing_deskew_reset(u32 dly_ac0, u32 dly_ac1,
+ u32 dly_ac2, u32 dly_ac3, u32 target_slice);
+u32 pmu_stage_slice_flag_check(void);
+u32 pmu_check_freq_lt_3200(void);
+u32 pmu_check_stage(u32 val);
+void pmu_cal_marker_clear(void);
+void pmu_clk_gate_handoff(void);
+void pmu_clk_ratio_pulse_trigger(void);
+void pmu_cmd_code_map_log(u32 cmd_code, u32 sub_code);
+void pmu_dbyte_cal_step_latch(u32 fine_val, u32 coarse_val,
+ u32 ch_flags);
+void pmu_dbyte_cal_strobe_pulse_seq(u32 mode, u32 val, u32 step);
+u32 pmu_dbyte_coarse_step_mask_sweep(u32 ch_idx, u32 step_val, u32 flag);
+void pmu_dbyte_deskew_phase_sample(u32 ch_base, u32 *out_mask,
+ u32 sel_mask);
+void pmu_dbyte_dq_deskew_regs_save_and_ramp(u8 *out_a, u8 *out_b,
+ u8 *out_c, u32 flag, u32 extra);
+void pmu_dbyte_lane_error_mask_calc(u32 mode, u32 slice_or_rank);
+void pmu_dbyte_rx_en_slice_pulse_eval(u32 mode, u32 slice_mask);
+void pmu_default_exception_handler(void);
+void pmu_delay_us(u32 count, u32 unit);
+u32 pmu_dly_line_repack(u32 val);
+u32 pmu_dly_reg_offset_select(u32 rank, u32 unused, u32 direction);
+void pmu_dmem_reg_stream_unpack(const void *cfg_desc);
+u8 pmu_dmem_training_flags_eval(void);
+u32 pmu_dram_cfg_bit13_check(u32 rank, u32 flag);
+void pmu_eye_centroid_avg_calc(const u16 *base_delays,
+ u16 *out_left, u16 *out_right, u16 *out_width);
+void pmu_eye_margin_step_align(u8 *margin_cur, u8 *margin_target,
+ u32 step_size);
+u32 pmu_eye_margin_window_search(u8 *samples, u32 reserved_arg);
+void pmu_cbt_cal_active_clear(void);
+void pmu_cbt_cal_active_set(void);
+void pmu_cbt_step_stat_set(void);
+u32 pmu_freq_delay_step_calc(u32 count, u32 unit);
+u32 pmu_freq_ratio_mult(u32 mult);
+u32 pmu_freq_scaled_timing_step_calc(void);
+u8 pmu_cbt_config_get(void);
+u8 pmu_cal_mode_flag_get(void);
+u32 pmu_lane_mask_bit_get(u32 word_idx, u32 bit_shift);
+u16 pmu_phy_cal_stat_read(void);
+u32 pmu_mr22_bit4_check(void);
+void pmu_hw_timer_delay(u32 count);
+u32 pmu_is_lpddr4(void);
+u32 pmu_is_lpddr5(void);
+void pmu_mailbox_post_cmd_dispatch(u32 cmd);
+int pmu_main_wrapper(void);
+u8 pmu_map_mask(u8 val, u32 index);
+s32 pmu_math_freq_scale_div(s32 val);
+s8 pmu_midpoint_deviation_calc(u32 val);
+void pmu_multiphase_pulse_seq_repeat(u32 phase_cmd, u32 param_val,
+ u32 slice, u32 count);
+void pmu_multirank_slice_cal_dispatch(void *table_ptr,
+ u32 eval_error_mask, u32 skip_mask_low, s32 skip_mask_high,
+ u32 ch_enable_mask, u32 check_settle_flags);
+u16 pmu_phy_ac_lane_timing_delay_set(u32 lane);
+void pmu_phy_analog_cfg_mode_set(u32 mode);
+void pmu_phy_cal_strobe_latch_setup(void);
+void pmu_phy_clk_gate_sync_pulse(void);
+void pmu_phy_clk_phase_toggle_pulse(void);
+void pmu_phy_dbyte_broadcast_enable(void);
+void pmu_phy_dbyte_cfg_unpack(void);
+u16 pmu_phy_delay_bound_margin_log(u32 slice);
+void pmu_phy_lcdl_delay_read(u16 *out_arr, u32 flag, u32 mode);
+void pmu_phy_mode_cfg_dispatch(u32 mode);
+void pmu_phy_pll_cfg_quad_write_stream(u32 reg_offset, u16 p0,
+ u16 p1, u32 p2, u32 flags);
+void pmu_phy_pll_clock_divider_align(void);
+void pmu_phy_pll_lock_init(void);
+void pmu_phy_profile_param_program(void);
+u32 pmu_phy_query_operating_mode(void);
+void pmu_phy_dbyte_cal_trig_pulse(u32 slice, u32 bit_idx);
+void pmu_phy_reg_buffer_stream(void *buf, u32 count, u32 is_save);
+void pmu_phy_reg_offset_diff_adjust(u32 rank);
+void pmu_pll_clk_gate_cfg(void);
+void pmu_pll_lock_freq_tier_cfg(void);
+void pmu_pll_quad_param_tuple_stream(const u16 *param_array,
+ u16 *out_ctr, u16 *base_ptr);
+void pmu_post_cmd_conditional_dispatch(u32 val);
+void pmu_phy_sec_cal_strobe_pulse(void);
+void pmu_phy_cal_trig_pulse(void);
+void pmu_phy_soft_reset_pulse(void);
+u16 pmu_rank_margin_channel_accum(u32 rank, u32 start_slice,
+ u32 end_slice, u32 flags, u32 swap_mode);
+void pmu_rank_mask_multiparam_dispatch(u32 channel, u32 rank,
+ u32 shadow_update);
+void pmu_rank_shadow_status_sample(void);
+void pmu_rank_slice_margin_eval(void);
+int pmu_sample_margin_center_eval(const u8 *samples,
+ u8 *out_margins, u8 *out_mode);
+void pmu_cal_marker_pair_set(u16 val);
+void pmu_cbt_config_set(u8 val);
+void pmu_phy_deskew_dly0_set(u16 val);
+void pmu_phy_deskew_dly1_set(u16 val);
+void pmu_phy_clk_timing_set(u16 val);
+void pmu_slice_delay_step_program(u32 reg_base_off,
+ const s16 *step_deltas, u32 format_mode, u32 shift_sel,
+ u32 stride_flag, u32 single_lane_flag);
+void pmu_slice_deskew_pulse_seq(u32 enable);
+void pmu_slice_lcdl_delay_collect(u32 param, void *out_buf, u32 count);
+void pmu_slice_metric_scan_and_program(void);
+void pmu_slice_lane_status_save(void);
+void pmu_start(void);
+u32 pmu_unpack_6(const u8 *p);
+u32 pmu_unpack_7(const u8 *p);
+void pmu_unpack_byte_pair(u32 val, u8 *out);
+__section(".vectors") void pmu_vector_table(void);
+void pmu_vref_signed_direction_pulse_dispatch(u32 slice_target,
+ u32 vref_step);
+u32 pmu_window_centroid_calc(void *table_buf);
+void pmu_phy_pll_cfg_quad_write(u16 v0);
+
+#endif /* _SUNXI_PMU_FW_H */
diff --git a/arch/arm/mach-sunxi/pmu/sunxi_pmu_abi.h b/arch/arm/mach-sunxi/pmu/sunxi_pmu_abi.h
new file mode 100644
index 00000000000..0625a6f46dd
--- /dev/null
+++ b/arch/arm/mach-sunxi/pmu/sunxi_pmu_abi.h
@@ -0,0 +1,251 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Copyright (C) 2026 Alastair D'Silva <alastair@d-silva.org>
+ *
+ * Synopsys DesignWare DDR PHY PMU Firmware & SPL Hardware ABI Definitions
+ * SoC: Allwinner A733 (Sun60i) / LPDDR5 PHY (Type 9)
+ *
+ * For detailed architecture, clean-room specification provenance, and memory
+ * map alignment, see doc/board/allwinner/a733-pmu.rst.
+ */
+
+#ifndef _SUNXI_PMU_ABI_H_
+#define _SUNXI_PMU_ABI_H_
+
+#if defined(__KERNEL__) || defined(__UBOOT__)
+#include <linux/types.h>
+#include <linux/bitops.h>
+#include <linux/compiler_attributes.h>
+#else
+#include <stdint.h>
+#include <stdbool.h>
+#include <stddef.h>
+
+#ifndef __packed
+#define __packed __attribute__((__packed__))
+#endif
+#ifndef __aligned
+#define __aligned(x) __attribute__((__aligned__(x)))
+#endif
+#ifndef __always_inline
+#define __always_inline inline __attribute__((__always_inline__))
+#endif
+#ifndef noinline
+#define noinline __attribute__((__noinline__))
+#endif
+#ifndef __section
+#define __section(s) __attribute__((__section__(s)))
+#endif
+
+#ifndef BIT
+#define BIT(nr) (1UL << (nr))
+#endif
+
+#ifndef _PMU_TYPES_DEFINED_
+#define _PMU_TYPES_DEFINED_
+typedef unsigned char u8;
+typedef unsigned short u16;
+typedef unsigned int u32;
+typedef unsigned long long u64;
+typedef signed char s8;
+typedef signed short s16;
+typedef signed int s32;
+typedef signed long long s64;
+#endif
+#endif
+
+/*
+ * Memory Map Base Addresses: Host ARM Physical vs ARC EM4 Microsequencer
+ */
+#define SUNXI_DRAM_PHY_CTRL_BASE 0x0aaa0000UL /* PHY CSRs (APB Slave interface) */
+#define SUNXI_DRAM_PHY_IMEM_BASE 0x0a9a0000UL /* ARC ICCM: Instruction Memory (64 KB) */
+#define SUNXI_DRAM_PHY_DMEM_BASE 0x0a9b0000UL /* ARC DCCM: Data Memory (64 KB) */
+
+/*
+ * PMU Control & Mailbox Register Offsets (relative to SUNXI_DRAM_PHY_CTRL_BASE / 0x0aaa0000)
+ * Note: The PHY APB slave interface is strictly 16-bit. All host accesses
+ * must use 16-bit readw / writew operations.
+ */
+#define PMU_REG_APB_MUX 0x0000 /* 0 = Host ARM owns APB, 1 = PMU ARC owns APB */
+#define PMU_REG_POLL_STAT 0x0008 /* Bit 0: 1 = PMU busy, 0 = msg pending / complete */
+#define PMU_REG_HANDOFF_0 0x0060 /* Handshake handoff 0 */
+#define PMU_REG_HANDOFF_1 0x0062 /* Handshake handoff 1 (write 1 to trigger, 0 to ack) */
+#define PMU_REG_STATUS_LO 0x0064 /* PMU status low 16 bits */
+#define PMU_REG_MAILBOX_INT 0x0066 /* Mailbox interrupt pulse / trigger */
+#define PMU_REG_STATUS_HI 0x0068 /* PMU status high 16 bits */
+#define PMU_REG_RESET 0x0132 /* MicroReset (0=run, bit 0=stall, bit 3=reset) */
+
+/*
+ * PMU Return Status Codes (read from PMU_REG_STATUS_LO | (PMU_REG_STATUS_HI << 16))
+ */
+#define PMU_STATUS_SUCCESS 0x00000007U /* Training completed successfully */
+#define PMU_STATUS_FATAL_ERROR 0x000000ffU /* Training failed assertion / aborted */
+
+/*
+ * PMU SequenceCtrl Flags (DMEM offset 0x10)
+ * Selects which calibration and training stages the PMU microsequencer executes.
+ */
+#define PMU_SEQ_DEV_INIT BIT(0) /* Bit 0: Device init, PLL lock, CBT entry, ZQ cal */
+#define PMU_SEQ_STAGE_VREF BIT(1) /* Bit 1: Receiver Vref voltage centering */
+#define PMU_SEQ_STAGE_DCD BIT(2) /* Bit 2: Duty cycle distortion correction */
+#define PMU_SEQ_STAGE_DQS BIT(3) /* Bit 3: Read/Write DQS strobe timing centering */
+#define PMU_SEQ_BIST_SEARCH_WIN BIT(4) /* Bit 4: BIST test pattern search window setup */
+#define PMU_SEQ_SEARCH_WIN BIT(6) /* Bit 6: High-resolution 2D eye margin search */
+#define PMU_SEQ_ZQ_CAL BIT(7) /* Bit 7: ZQ pad output impedance & ODT calibration */
+#define PMU_SEQ_DESKEW_SWEEP BIT(8) /* Bit 8: Per-bit DQ deskew delay line tuning */
+#define PMU_SEQ_MARGIN_STEP BIT(9) /* Bit 9: Margin envelope boundary evaluation */
+#define PMU_SEQ_LANE_WIN_SWEEP BIT(10) /* Bit 10: Lane window sweep & cross-lane matrix scan */
+#define PMU_SEQ_LPCA_INIT BIT(12) /* Bit 12: Low-Power Command/Address training & init */
+
+/*
+ * Operating Mode:
+ * - FULL_TRAIN: Complete multi-stage LPDDR5 calibration sweep.
+ */
+#define PMU_SEQ_FULL_TRAIN (PMU_SEQ_DEV_INIT | PMU_SEQ_STAGE_VREF | \
+ PMU_SEQ_STAGE_DCD | PMU_SEQ_STAGE_DQS | \
+ PMU_SEQ_BIST_SEARCH_WIN | PMU_SEQ_SEARCH_WIN | \
+ PMU_SEQ_MARGIN_STEP | PMU_SEQ_LPCA_INIT) /* 0x125f */
+
+/**
+ * struct pmu_message_block - Host-to-PMU parameter handshake mailbox
+ * @msg_misc: Miscellaneous control and debug flags
+ * @cfg_flags: Hardware configuration flags
+ * @rsvd0: Reserved padding at offset 0x02
+ * @pstate_flags: Power-state flags and channel offset (0x04)
+ * @rsvd1: Reserved padding at offset 0x05
+ * @dram_rate_mts: DRAM data rate in MT/s (3600 for 1800 MHz WCK)
+ * @pll_ratio: WCK:CK frequency ratio (0x08: 4 = 4:1, 2 = 2:1)
+ * @rsvd2: Reserved padding at offset 0x09
+ * @dfi_freq_ratio: DFI frequency divider ratio (0x0a)
+ * @assert_code: PMU 32-bit assert error code (overwritten on fatal error, 0x0c)
+ * @rsvd_0c: Reserved runtime byte at offset 0x0c
+ * @cal_misc_flags: Runtime calibration miscellaneous flags (0x0d)
+ * @ac_signed_step: Runtime signed AC timing step (0x0e)
+ * @train_mode_hi: Runtime upper training mode flags (0x0f)
+ * @sequence_ctrl: Training stage bitmask (PMU_SEQ_*)
+ * @post_threshold: Telemetry log filter threshold (0xff = disabled)
+ * @rsvd3: Reserved padding at offset 0x13
+ * @train_mode: Training mode / calibration state (0x14)
+ * @cal_ctrl: Calibration control flags (0x16)
+ * @rsvd4: Reserved padding at offsets 0x18..0x23
+ * @ch0_dq_width: Channel 0 total DQ bit width (16)
+ * @ch0_rank_en: Channel 0 active rank bitmask (0x03 = dual-rank)
+ * @rsvd5: Reserved padding at offsets 0x26..0x3e
+ * @ch1_dq_width: Channel 1 total DQ bit width (16)
+ * @ch1_rank_en: Channel 1 active rank bitmask (0x03 = dual-rank)
+ * @rsvd6: Reserved padding at offsets 0x41..0x59
+ * @mr1: Per-slice JEDEC MR1 values (clock and WL set config)
+ * @mr2: Per-slice JEDEC MR2 values (RL and nWR config)
+ * @mr3: Per-slice JEDEC MR3 values (PDDS, DBI, and bank org)
+ * @mr10: Per-slice JEDEC MR10 values (RDQS preamble/postamble)
+ * @mr11: Per-slice JEDEC MR11 values (CA and DQ ODT)
+ * @mr12: Per-slice JEDEC MR12 values (CA Vref)
+ * @mr13: Per-slice JEDEC MR13 values
+ * @mr14: Per-slice JEDEC MR14 values (DQ[7:0] Vref)
+ * @mr15: Per-slice JEDEC MR15 values (DQ[15:8] Vref)
+ * @mr16: Per-slice JEDEC MR16 values (FSP-WR/OP and CBT)
+ * @mr17: Per-slice JEDEC MR17 values (CK/CA ODT)
+ * @mr18: Per-slice JEDEC MR18 values (WCK ODT and CK-to-WCK sync)
+ * @mr19: Per-slice JEDEC MR19 values (DVFSC and WCK2DQ osc)
+ * @mr20: Per-slice JEDEC MR20 values (RDQS mode)
+ * @mr21: Per-slice JEDEC MR21 values
+ * @mr22: Per-slice JEDEC MR22 values
+ * @mr24: Per-slice JEDEC MR24 values (DFE)
+ * @mr25: Per-slice JEDEC MR25 values
+ * @mr26: Per-slice JEDEC MR26 values
+ * @mr27: Per-slice JEDEC MR27 values
+ * @mr28: Per-slice JEDEC MR28 values (ZQ reset/stop/interval)
+ * @mr30: Per-slice JEDEC MR30 values (DCA)
+ * @mr31: Per-slice JEDEC MR31 values
+ * @mr32: Per-slice JEDEC MR32 values
+ * @mr33: Per-slice JEDEC MR33 values
+ * @mr34: Per-slice JEDEC MR34 values
+ * @mr37: Per-slice JEDEC MR37 values (WCK2DQI interval)
+ * @mr40: Per-slice JEDEC MR40 values (WCK2DQO interval)
+ * @mr41: Per-slice JEDEC MR41 values (NT-ODT and PPRE)
+ * @rsvd7: Reserved padding at offsets 0xce..0xe7
+ * @pll_gate_ch: Per-channel PLL clock gate thresholds (0xe8..0xeb)
+ * @rsvd8: Reserved padding at offsets 0xec..0xff
+ * @dq_swap_en: Per-bit DQ pin swizzle enable flag (0x100)
+ * @lane_err_mask: Per-lane deskew error status mask (0x101)
+ * @train_feature_mask: Training feature override bitmask (0x102)
+ * @ca_vref_rank1: Initial CA Vref DAC setting for Rank 1 (0x103)
+ * @ca_vref_rank0: Initial CA Vref DAC setting for Rank 0 (0x104)
+ * @diff_tap_pair: Differential tap pair offset (0x105)
+ *
+ * Placed at the base of PMU DCCM (0x80000000 in ARC address space,
+ * 0x0a9b0000 in host physical address space). Written by U-Boot SPL
+ * prior to triggering PMU training execution.
+ */
+struct pmu_message_block {
+ u8 msg_misc; /* 0x000 */
+ u8 cfg_flags; /* 0x001 */
+ u16 rsvd0; /* 0x002 */
+ u8 pstate_flags; /* 0x004 */
+ u8 rsvd1; /* 0x005 */
+ u16 dram_rate_mts; /* 0x006 */
+ u8 pll_ratio; /* 0x008 */
+ u8 rsvd2; /* 0x009 */
+ u16 dfi_freq_ratio; /* 0x00a */
+ union {
+ u32 assert_code; /* 0x00c */
+ struct __packed {
+ u8 rsvd_0c; /* 0x00c */
+ u8 cal_misc_flags; /* 0x00d */
+ s8 ac_signed_step; /* 0x00e */
+ u8 train_mode_hi; /* 0x00f */
+ };
+ };
+ u16 sequence_ctrl; /* 0x010 */
+ u8 post_threshold; /* 0x012 */
+ u8 rsvd3; /* 0x013 */
+ u16 train_mode; /* 0x014 */
+ u16 cal_ctrl; /* 0x016 */
+ u8 rsvd4[12]; /* 0x018..0x023 */
+ u8 ch0_dq_width; /* 0x024 */
+ u8 ch0_rank_en; /* 0x025 */
+ u8 rsvd5[25]; /* 0x026..0x03e */
+ u8 ch1_dq_width; /* 0x03f */
+ u8 ch1_rank_en; /* 0x040 */
+ u8 rsvd6[25]; /* 0x041..0x059 */
+ u8 mr1[4]; /* 0x05a */
+ u8 mr2[4]; /* 0x05e */
+ u8 mr3[4]; /* 0x062 */
+ u8 mr10[4]; /* 0x066 */
+ u8 mr11[4]; /* 0x06a */
+ u8 mr12[4]; /* 0x06e */
+ u8 mr13[4]; /* 0x072 */
+ u8 mr14[4]; /* 0x076 */
+ u8 mr15[4]; /* 0x07a */
+ u8 mr16[4]; /* 0x07e */
+ u8 mr17[4]; /* 0x082 */
+ u8 mr18[4]; /* 0x086 */
+ u8 mr19[4]; /* 0x08a */
+ u8 mr20[4]; /* 0x08e */
+ u8 mr21[4]; /* 0x092 */
+ u8 mr22[4]; /* 0x096 */
+ u8 mr24[4]; /* 0x09a */
+ u8 mr25[4]; /* 0x09e */
+ u8 mr26[4]; /* 0x0a2 */
+ u8 mr27[4]; /* 0x0a6 */
+ u8 mr28[4]; /* 0x0aa */
+ u8 mr30[4]; /* 0x0ae */
+ u8 mr31[4]; /* 0x0b2 */
+ u8 mr32[4]; /* 0x0b6 */
+ u8 mr33[4]; /* 0x0ba */
+ u8 mr34[4]; /* 0x0be */
+ u8 mr37[4]; /* 0x0c2 */
+ u8 mr40[4]; /* 0x0c6 */
+ u8 mr41[4]; /* 0x0ca */
+ u8 rsvd7[26]; /* 0x0ce..0x0e7 */
+ u16 pll_gate_ch[2]; /* 0x0e8..0x0eb */
+ u8 rsvd8[20]; /* 0x0ec..0x0ff */
+ u8 dq_swap_en; /* 0x100 */
+ u8 lane_err_mask; /* 0x101 */
+ u8 train_feature_mask; /* 0x102 */
+ u8 ca_vref_rank1; /* 0x103 */
+ u8 ca_vref_rank0; /* 0x104 */
+ u8 diff_tap_pair; /* 0x105 */
+} __packed __aligned(2);
+
+#endif /* _SUNXI_PMU_ABI_H_ */
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 15/26] sunxi: dram: a733: Add ARC PMU primitives, clock, telemetry, and CBT
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (13 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 14/26] sunxi: dram: a733: Add ARC PMU register, ABI, and build definitions Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 16/26] sunxi: dram: a733: Add ARC PMU deskew and BIST training firmware Alastair D'Silva
` (12 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
Add the freestanding memory/division helpers, ARC auxiliary RTC delay
and PHY PLL clock-gate helpers, Synopsys mailbox IPC and panic handlers,
and JEDEC LPDDR5 Command Bus Training (CBT) routines for the Allwinner
A733 DDR PHY ARC EM4 firmware.
Written from clean-room hardware register and JEDEC JESD209-5 protocol
specifications (see doc/board/allwinner/a733-pmu.rst).
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
arch/arm/mach-sunxi/pmu/pmu_cbt.c | 417 ++++++++++
arch/arm/mach-sunxi/pmu/pmu_clock.c | 938 +++++++++++++++++++++++
arch/arm/mach-sunxi/pmu/pmu_primitives.c | 531 +++++++++++++
arch/arm/mach-sunxi/pmu/pmu_telemetry.c | 619 +++++++++++++++
4 files changed, 2505 insertions(+)
create mode 100644 arch/arm/mach-sunxi/pmu/pmu_cbt.c
create mode 100644 arch/arm/mach-sunxi/pmu/pmu_clock.c
create mode 100644 arch/arm/mach-sunxi/pmu/pmu_primitives.c
create mode 100644 arch/arm/mach-sunxi/pmu/pmu_telemetry.c
diff --git a/arch/arm/mach-sunxi/pmu/pmu_cbt.c b/arch/arm/mach-sunxi/pmu/pmu_cbt.c
new file mode 100644
index 00000000000..6c97ad8532e
--- /dev/null
+++ b/arch/arm/mach-sunxi/pmu/pmu_cbt.c
@@ -0,0 +1,417 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Copyright (C) 2026 Alastair D'Silva <alastair@d-silva.org>
+ *
+ * Allwinner A733 Synopsys DDR PHY PMU Command Bus Training (CBT) Helpers
+ */
+
+#include "pmu_fw.h"
+
+/**
+ * pmu_cal_rank_offset_get() - Compute JEDEC CBT read latency entry for rank and channel
+ * @rank: Active DRAM rank index (0 or 1)
+ * @channel: Active DRAM channel index (0 or 1)
+ *
+ * Extracts the 4-bit MR2[3:0] RL index from byte +2 of the 54-byte per-slice
+ * descriptor at DCCM 0x800009a4 + (@rank * 2 + @channel) * 54 and computes the
+ * corresponding JEDEC read latency (doubled when pmu_cbt_config_get() == 2).
+ *
+ * Return: 8-bit CBT read latency entry for @rank and @channel.
+ */
+u8 pmu_cal_rank_offset_get(u32 rank, u32 channel)
+{
+ u32 slice_idx = (rank << 1) + channel;
+ u32 desc_addr = PMU_DMEM_CAL_STRUCT_ADDR +
+ slice_idx * PMU_SLICE_DESC_SIZE;
+ u32 idx = dmem_read8(desc_addr + PMU_SLICE_DESC_MR2_OFF) & 0x0fU;
+ u32 rl;
+
+ if (idx >= 12U)
+ return 0U;
+
+ rl = idx + 3U + (idx >= 4U) + (idx >= 7U) + (idx >= 9U);
+ if (pmu_cbt_config_get() == 2U)
+ return (idx < 6U) ? (u8)(rl << 1) : 0U;
+
+ return (u8)rl;
+}
+
+/**
+ * pmu_cal_cbt_desc_unpack() - Unpack CBT step multiplier from slice descriptor
+ * @rank: Active DRAM rank index (0 or 1)
+ * @channel: Active DRAM channel index (0 or 1)
+ *
+ * Return: 8 or 4 when DCCM 0x80000008 == 2 (else 4 or 2) based on bit 4 of
+ * descriptor byte +3 at 0x800009a4 + (@rank * 2 + @channel) * 54.
+ */
+u32 pmu_cal_cbt_desc_unpack(u32 rank, u32 channel)
+{
+ u32 slice_idx = (rank << 1) + channel;
+ u8 byte_val = dmem_read8(PMU_DMEM_CAL_STRUCT_ADDR +
+ slice_idx * PMU_SLICE_DESC_SIZE +
+ PMU_SLICE_DESC_MR3_OFF);
+ u32 res = (byte_val & 0x10U) ? 2U : 4U;
+
+ if (dmem_read8(PMU_DMEM_PLL_RATIO) == 2U)
+ res <<= 1;
+
+ return res;
+}
+
+/**
+ * pmu_cal_search_win_max_calc() - Scan slice registers and update master delay table
+ * @channel: Target DRAM channel index (0 or 1)
+ * @rank: Target DRAM rank index (0 or 1)
+ * @dir: Direction step selector (0 adds +4 offset, non-zero adds 0)
+ */
+void pmu_cal_search_win_max_calc(u32 channel, u32 rank, u32 dir)
+{
+ u8 slice_start;
+ u8 slice_end;
+ s32 max_val = -1;
+ u32 addr;
+ s32 i;
+ u8 pll_ratio;
+ u32 shift;
+ s32 v;
+
+ if (rank == 1U && dmem_read8(PMU_DMEM_CH1_ACTIVE_FLAG) == 0U) {
+ __volatile__ u16 *tbl = pmu_sdata()->master_dly_tbl[channel];
+
+ tbl[1] = tbl[0];
+ return;
+ }
+
+ slice_start = dmem_read8((rank != 0U) ? PMU_DMEM_RANK1_SLICE_START :
+ PMU_DMEM_RANK0_SLICE_START);
+ slice_end = dmem_read8((rank != 0U) ? PMU_DMEM_RANK1_SLICE_END :
+ PMU_DMEM_RANK_BOUNDARY);
+ addr = PHY_REG_DBYTE_BASE |
+ (((pmu_sdata()->active_csr_offset |
+ (channel + (((dir == 0U) ? 1U : 0U) << 2) + 0x20U) |
+ ((u32)slice_start << 12)) << 1));
+
+ for (i = (s32)slice_end + 1 - (s32)slice_start; i > 0; i--) {
+ s16 val = (s16)phy_read16(addr);
+
+ if ((s32)val > max_val)
+ max_val = (s32)val;
+ addr += PHY_DBYTE_SLICE_STRIDE;
+ }
+
+ pll_ratio = dmem_read8(PMU_DMEM_PLL_RATIO);
+ shift = 511U >> (pll_ratio == 2U);
+
+ v = (max_val >> ((pll_ratio >> 1) + 7U)) +
+ (s8)dmem_read8(PMU_DMEM_SEARCH_WINDOW_BIAS_LO) +
+ (((u32)max_val & shift) != 0U ? 1 : 0) + 2;
+ pmu_sdata()->master_dly_tbl[channel][rank] = (u16)v;
+}
+
+/**
+ * pmu_phy_reg_offset_diff_adjust() - Program CA Vref DAC and trim registers for rank
+ * @rank: Target rank index (0 or non-zero)
+ */
+void pmu_phy_reg_offset_diff_adjust(u32 rank)
+{
+ __volatile__ u16 *apb =
+ (__volatile__ u16 *)(PHY_REG_CA_VREF_DAC +
+ (pmu_sdata()->active_csr_offset << 1));
+ u8 ca_vref_r1 = dmem_read8(PMU_DMEM_CA_VREF_RANK1_ADDR);
+ u8 ca_vref_r0 = dmem_read8(PMU_DMEM_CA_VREF_RANK0_ADDR);
+ u8 val = (rank == 0U) ? ca_vref_r0 : ca_vref_r1;
+ s32 diff;
+ u16 ca_reg_a, ca_reg_b, ca_reg_c;
+ u16 ca_reg_val;
+
+ apb[0] = (apb[0] & 0x3f00U) | val;
+ apb[1] = (apb[1] & 0x3f00U) | val;
+ apb[4] = (apb[4] & 0x3f00U) | val;
+
+ diff = (s32)ca_vref_r1 - (s32)ca_vref_r0;
+ if (rank == 0U)
+ diff = -diff;
+
+ ca_reg_a = apb[13];
+ ca_reg_val = (u16)((ca_reg_a & 0x3f00U) | (((ca_reg_a & 0x3fU) + diff) & 0x3fU));
+ apb[13] = ca_reg_val;
+ apb[14] = ca_reg_val;
+
+ ca_reg_b = apb[15];
+ apb[15] = (u16)((ca_reg_b & 0x3f00U) | (((ca_reg_b & 0x3fU) + diff) & 0x3fU));
+
+ ca_reg_c = apb[16];
+ apb[16] = (u16)((ca_reg_c & 0x3f00U) | (((ca_reg_c & 0x3fU) + diff) & 0x3fU));
+}
+
+/**
+ * pmu_dmem_reg_stream_unpack() - Clear CBT/deskew status CSRs and pulse reset
+ * @cfg_desc: Pointer to CBT configuration descriptor (32-bit offset at +0x18)
+ */
+void pmu_dmem_reg_stream_unpack(const void *cfg_desc)
+{
+ __volatile__ u16 *apb = (__volatile__ u16 *)PHY_REG_RESET_TRIGGER;
+ __volatile__ u16 *bcast_base = (__volatile__ u16 *)PHY_REG_DBYTE_BCAST_BASE;
+ const u8 *desc = (const u8 *)cfg_desc;
+ u32 rank_cfg_off = *(const u32 *)(desc + 0x18U);
+ u32 zero = 0U;
+ u16 val;
+
+ phy_write16(PHY_REG_DBYTE_UPPER_SAMPLE_B, (u16)zero);
+ bcast_base[176] = (u16)zero;
+ bcast_base[177] = (u16)zero;
+ apb[71] = (u16)zero;
+ apb[0] = (u16)zero;
+ bcast_base[rank_cfg_off] = (u16)zero;
+
+ val = phy_read16(PHY_REG_BIST_PLL_CFG);
+ phy_write16(PHY_REG_AC_BCAST, val & (u16)~0x0010U);
+
+ pmu_phy_cal_trig_pulse();
+ if (pmu_sdata()->freq_mode <= 1U)
+ pmu_phy_soft_reset_pulse();
+}
+
+/**
+ * pmu_cbt_delay_offset_lookup() - Compute JEDEC CBT write latency offset for rank and channel
+ * @rank: Active DRAM rank index (0 or 1)
+ * @channel: Active DRAM channel index (0 or 1)
+ *
+ * Extracts the 4-bit MR1[7:4] clock-frequency index and MR3[5] write-latency
+ * set selector from the 54-byte per-slice descriptor at DCCM 0x800009a4 and
+ * computes the corresponding JEDEC WL Set A or Set B delay entry.
+ *
+ * Return: 8-bit CBT write latency offset for @rank and @channel.
+ */
+u32 pmu_cbt_delay_offset_lookup(u32 rank, u32 channel)
+{
+ u32 slice_idx = (rank << 1) + channel;
+ u32 desc_addr = PMU_DMEM_CAL_STRUCT_ADDR +
+ slice_idx * PMU_SLICE_DESC_SIZE;
+ u32 idx = (u32)(dmem_read8(desc_addr + PMU_SLICE_DESC_MR1_OFF) >> 4);
+ u32 wl;
+
+ if (idx >= 12U)
+ return 0U;
+
+ if (dmem_read8(desc_addr + PMU_SLICE_DESC_MR3_OFF) & 0x20U)
+ wl = idx + 2U + (idx >= 4U) + (idx >= 7U) + (idx >= 9U);
+ else
+ wl = ((idx * 11U + 2U) >> 4) + 2U;
+
+ if (pmu_cbt_config_get() == 2U)
+ return (idx < 6U) ? (wl << 1) : 0U;
+
+ return wl;
+}
+
+/**
+ * pmu_cbt_cal_pulse_seq() - Execute 6-step CBT calibration pulse sequence
+ * @pulse_cmd: Pulse command flags for intermediate phases
+ */
+noinline void pmu_cbt_cal_pulse_seq(u32 pulse_cmd)
+{
+ u32 zero = 0U;
+
+ pmu_cal_sequence_pulse_send(0x41U << 19, 7U, 4U, 0U, zero, zero, zero);
+ pmu_cbt_cal_active_set();
+ pmu_cal_sequence_pulse_send(pulse_cmd, 7U, 4U, 0U, zero, zero, zero);
+ pmu_cbt_cal_active_clear();
+ pmu_cal_sequence_pulse_send(pulse_cmd, 7U, 4U, 0U, zero, zero, zero);
+ pmu_cal_sequence_pulse_send(0x80U | (1U << 19), 7U, 4U, 0U,
+ zero, zero, zero);
+}
+
+/**
+ * pmu_cbt_timing_pulse_coordinator() - Coordinate 4-channel CBT timing pulse sweep
+ * @cal_mode: Calibration mode halfword forwarded to pmu_cal_window_params_dispatch()
+ * @phase_sel: Phase pattern selector (0 for primary masks, non-zero for alternate)
+ */
+void pmu_cbt_timing_pulse_coordinator(u16 cal_mode, u32 phase_sel)
+{
+ u32 zero = 0U;
+ __volatile__ u16 *out_tbl =
+ (__volatile__ u16 *)PMU_DMEM_DELAY_TAP_LUT;
+ __volatile__ u16 *out_sub1 =
+ (__volatile__ u16 *)PMU_DMEM_DELAY_TAP_SUB1;
+ u32 mask_word;
+ u32 phase;
+
+ out_tbl[0] = 100U;
+
+ for (phase = 0U; phase < 4U; phase++) {
+ u32 pulse_mask;
+ u16 step;
+ u16 marker;
+
+ if (phase_sel == 0U) {
+ if (phase == 0U)
+ pulse_mask = 0x7f00U;
+ else if (phase == 1U)
+ pulse_mask = 0x7f0000U;
+ else if (phase == 2U)
+ pulse_mask = 0x7f007f7fUL;
+ else
+ pulse_mask = 0x7f7f007fUL;
+ } else {
+ if (phase == 0U)
+ pulse_mask = 0x7fU;
+ else if (phase == 1U)
+ pulse_mask = 0x7f000000UL;
+ else if (phase == 2U)
+ pulse_mask = 0x7f007f7fUL;
+ else
+ pulse_mask = 0x007f7f7fUL;
+ }
+
+ mask_word = pulse_mask;
+ step = out_tbl[phase];
+ pmu_cal_marker_pair_set((u16)(step << 2));
+
+ pmu_cal_sequence_pulse_send(0x41UL << 19, 7U, 8U, 0U,
+ zero, zero, zero);
+ pmu_cal_pad_timing_delay_set(0U, 4U, 1U);
+
+ pmu_cal_window_params_dispatch((const u8 *)&mask_word, 2U, cal_mode);
+
+ pmu_cal_pad_timing_delay_set(0U, 20U, 1U);
+ pmu_cal_sequence_pulse_send(1UL << 18, 7U, 4U, 0U,
+ zero, zero, zero);
+ pmu_cal_sequence_pulse_send((1UL << 19) | 0x80UL, 7U, 8U, 0U,
+ zero, zero, zero);
+
+ marker = pmu_cal_stride_get();
+ out_tbl[phase + 1U] = marker;
+ out_sub1[phase] = (u16)(marker - 1U);
+ }
+}
+
+/**
+ * pmu_cbt_strobe_pulse_train() - Generate CBT strobe pulse train
+ * @pulse_cmd: Slice / pulse command selector
+ * @is_stage2: Non-zero for stage 2 short sequence, 0 for full repeat sequence
+ */
+void pmu_cbt_strobe_pulse_train(u32 pulse_cmd, u32 is_stage2)
+{
+ u32 zero = 0U;
+ u8 pll_ratio = dmem_read8(PMU_DMEM_PLL_RATIO);
+ u32 repeat_cnt;
+ u32 i;
+
+ pmu_cal_strobe_pulse_init();
+ pmu_cal_sequence_pulse_send(0U, 7U, 4U, 0U, zero, zero, zero);
+ pmu_cbt_cal_active_set();
+
+ repeat_cnt = (pll_ratio == 2U) ? 4U : 2U;
+
+ if (is_stage2 != 0U)
+ pmu_cal_sequence_pulse_send(0U, 7U, 2U, 0U, zero, zero, 4U);
+
+ pmu_cal_sequence_pulse_send(0x200000U, 5U, 0U, 256U, zero,
+ pulse_cmd, zero);
+
+ if (is_stage2 == 0U) {
+ pmu_cal_sequence_pulse_send(0x1000004U, 0x2bU, repeat_cnt, 0U,
+ zero, pulse_cmd, zero);
+ pmu_cal_sequence_pulse_send(0x1000004U, 0x2bU, repeat_cnt, 0U,
+ zero, pulse_cmd, zero);
+ }
+
+ pmu_cal_sequence_pulse_send(0x1040000U, 0x2bU, repeat_cnt, 0U, zero,
+ pulse_cmd, zero);
+ pmu_cal_sequence_pulse_send(0U, 7U, 10U, 0U, zero, zero, zero);
+
+ for (i = 0U; i < 2U; i++) {
+ pmu_cal_sequence_pulse_send(1U << (15U + i), 7U, 0U, 0U,
+ zero, zero, zero);
+ pmu_cal_sequence_pulse_send(1U << (16U + i), 7U, 0U, 0U,
+ zero, zero, zero);
+ if (i == 0U)
+ pmu_cbt_cal_active_clear();
+ }
+
+ pmu_cal_sequence_pulse_send(0x80U | (1U << 19), 7U, 4U, 0U,
+ zero, zero, zero);
+}
+
+/**
+ * pmu_cbt_deskew_pulse_train() - Generate CBT deskew pulse train
+ * @pulse_cmd: Slice / pulse command selector
+ * @is_alternate: Non-zero to add 1 to second repeat count
+ */
+void pmu_cbt_deskew_pulse_train(u32 pulse_cmd, u32 is_alternate)
+{
+ u32 zero = 0U;
+ u8 pll_ratio = dmem_read8(PMU_DMEM_PLL_RATIO);
+ u32 repeat_cnt;
+ u32 second_cnt;
+
+ pmu_cal_strobe_pulse_init();
+ pmu_cal_sequence_pulse_send(0U, 7U, 0U, 0U, zero, zero, zero);
+ pmu_cal_sequence_pulse_send(0x480000U, 5U, 0U, 512U, zero,
+ pulse_cmd, zero);
+ pmu_cbt_cal_active_set();
+
+ repeat_cnt = (pll_ratio == 2U) ? 5U : 2U;
+ second_cnt = (is_alternate != 0U) ? (repeat_cnt + 1U) : repeat_cnt;
+
+ pmu_cal_sequence_pulse_send(0x0cU, 0x2bU, repeat_cnt, 0U, zero,
+ pulse_cmd, zero);
+ pmu_cal_sequence_pulse_send(0x0cU, 0x2bU, second_cnt, 0U, zero,
+ pulse_cmd, zero);
+ pmu_cal_sequence_pulse_send(0x40000U, 0x2bU, repeat_cnt, 2U, zero,
+ pulse_cmd, zero);
+ pmu_cal_sequence_pulse_send(0x0cU, 0x2bU, repeat_cnt, 2U, zero,
+ pulse_cmd, zero);
+
+ pmu_cal_sequence_pulse_send(0U, 7U, 4U, 0U, zero, zero, 10U);
+ pmu_cal_sequence_pulse_send(1U << 15, 7U, 4U, 0U, zero, zero, zero);
+ pmu_cal_sequence_pulse_send(0U, 7U, 8U, 0U, zero, zero, zero);
+ pmu_cbt_cal_active_clear();
+ pmu_cal_sequence_pulse_send(0U, 7U, 4U, 0U, zero, zero, zero);
+
+ pmu_cal_sequence_pulse_send(0U, 5U, 4U, 896U, zero, pulse_cmd, zero);
+ pmu_cal_sequence_pulse_send((1U << 19) | 0x80U, 7U, 0U, 0U, zero,
+ pulse_cmd, zero);
+}
+
+/**
+ * pmu_cbt_3phase_pulse_dispatch() - Dispatch 3-phase CBT calibration pulse sequence
+ * @cal_mode: Target calibration mode / rank selector
+ * @step_table: Base pointer to calibration step table
+ * @step_idx: Step index within @step_table
+ * @flags: Control bitmask (bit 0 skips profile program, bit 1 skips AC setup)
+ */
+void pmu_cbt_3phase_pulse_dispatch(u32 cal_mode, const u8 *step_table,
+ u32 step_idx, u32 flags)
+{
+ u8 val = step_table[step_idx];
+ u32 is_final_step = (step_idx == 16U) ? 1U : 0U;
+ u32 p2 = ((step_idx - 14U) < 2U) ? 0x20U : (is_final_step ? 4U : 6U);
+ u32 is_final_ovr = 0U;
+
+ pmu_cal_strobe_pulse_init();
+ if ((flags & 2U) == 0U) {
+ pmu_ac_lane_profile_setup();
+ pmu_cal_pad_timing_delay_set(0U, 5U, 1U);
+ }
+
+ pmu_dbyte_cal_step_latch(step_idx, val, cal_mode);
+ pmu_cal_sequence_pulse_send(0U, 7U, p2, 0U, 0U, is_final_step, 0U);
+
+ if (is_final_step != 0U && (val & 0x20U) != 0U) {
+ u32 arg = (dmem_read8(PMU_DMEM_CBT_LANE_LIM) == 0U) ?
+ (1U << 19) : 0U;
+
+ pmu_cbt_cal_pulse_seq(arg);
+ is_final_ovr = 1U;
+ }
+
+ pmu_cbt_3phase_pulse_seq();
+ pmu_cal_slice_step_diff_latch(0U, is_final_ovr ^ 1U);
+ if (is_final_ovr != 0U)
+ pmu_delay_us(0x1388U, 0x14U);
+ if ((flags & 1U) == 0U)
+ pmu_phy_profile_param_program();
+}
+
diff --git a/arch/arm/mach-sunxi/pmu/pmu_clock.c b/arch/arm/mach-sunxi/pmu/pmu_clock.c
new file mode 100644
index 00000000000..0ba564bb10b
--- /dev/null
+++ b/arch/arm/mach-sunxi/pmu/pmu_clock.c
@@ -0,0 +1,938 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Copyright (C) 2026 Alastair D'Silva <alastair@d-silva.org>
+ *
+ * Allwinner A733 Synopsys DDR PHY PMU Hardware Timer, PLL & Broadcast Helpers
+ */
+
+#include "pmu_fw.h"
+
+static inline void pmu_phy_clk_gate_clear(void);
+static void pmu_phy_pll_lock_poll(u32 timeout);
+static void pmu_profile_param_stride_cfg(u32 mode);
+static inline void pmu_phy_clk_gate_set(void);
+
+/**
+ * pmu_freq_ratio_mult() - Compute scaled DRAM frequency ratio multiplier
+ * @mult: Multiplier factor applied to DRAM frequency
+ *
+ * Return: (((u32)freq * @mult) / ((u32)type * 2000U)) + 1U.
+ */
+u32 pmu_freq_ratio_mult(u32 mult)
+{
+ u32 freq = dmem_read16(PMU_DMEM_DRAM_FREQ_OFF);
+ u32 type = dmem_read8(PMU_DMEM_DRAM_FREQ_OFF + 2U);
+
+ if (type == 0U)
+ type = 2U;
+
+ return ((freq * mult) / (type * 2000U)) + 1U;
+}
+
+/**
+ * pmu_phy_pll_lock_init() - Initialize CBT PLL lock and control registers
+ *
+ * Sets bit 0 of PHY register 0x90040144 into 0x900401f0, executes 4-tick and
+ * 31-tick hardware timer delays, and writes 0x0001 to 0x90040144.
+ */
+void pmu_phy_pll_lock_init(void)
+{
+ u16 val = *(__volatile__ u16 *)PHY_REG_CBT_CTRL;
+
+ *(__volatile__ u16 *)PHY_REG_CBT_PLL = val | 0x0001U;
+ pmu_hw_timer_delay(4U);
+ pmu_hw_timer_delay(31U);
+ *(__volatile__ u16 *)PHY_REG_CBT_CTRL = 0x0001U;
+}
+
+/**
+ * pmu_phy_dbyte_broadcast_enable() - Enable DBYTE broadcast mode after PLL lock
+ *
+ * Executes 31-tick and 4-tick hardware timer delays, clears PHY register
+ * 0x900401f0 to 0, and sets bit 1 of PHY register 0x90040144.
+ */
+void pmu_phy_dbyte_broadcast_enable(void)
+{
+ u16 val;
+
+ pmu_hw_timer_delay(31U);
+ pmu_hw_timer_delay(4U);
+ *(__volatile__ u16 *)PHY_REG_CBT_PLL = 0x0000U;
+ val = *(__volatile__ u16 *)PHY_REG_CBT_CTRL;
+ *(__volatile__ u16 *)PHY_REG_CBT_CTRL = val | 0x0002U;
+}
+
+/**
+ * pmu_hw_timer_delay() - Execute rollover-safe ARC auxiliary RTC delay
+ * @count: Requested raw delay count
+ *
+ * Scales @count by the shift factor in DCCM 0x80000404, ensures the ARC
+ * auxiliary RTC counter (0x104/0x105) is enabled, performs a fast APB dummy
+ * read turnaround when ticks < 16, or spins across the 64-bit counter with
+ * 32-bit rollover handling when ticks >= 16.
+ */
+noinline void pmu_hw_timer_delay(u32 count)
+{
+ u32 s = pmu_sdata()->timer_shift & 31U;
+ u32 ticks = (count == 0U) ? 0U : ((count + 1U) >> s);
+ u32 start_lo;
+ u32 start_hi;
+ u32 headroom;
+
+ if ((__builtin_arc_lr(PMU_AUX_RTC_CTRL) & 1U) == 0U) {
+ u32 two = 2U;
+ u32 one = 1U;
+
+ __builtin_arc_sr(two, PMU_AUX_RTC_CTRL);
+ __builtin_arc_sr(one, PMU_AUX_RTC_CTRL);
+ }
+
+ if (ticks < 16U) {
+ (void)*(__volatile__ u16 *)PHY_REG_TIMER_DUMMY;
+ return;
+ }
+
+ start_lo = __builtin_arc_lr(PMU_AUX_RTC_LOW);
+ start_hi = __builtin_arc_lr(PMU_AUX_RTC_HIGH);
+ headroom = ~start_lo;
+
+ if (headroom < ticks) {
+ u32 rem = ticks - headroom;
+
+ while (__builtin_arc_lr(PMU_AUX_RTC_LOW) >= start_lo)
+ ;
+ while (__builtin_arc_lr(PMU_AUX_RTC_LOW) < rem)
+ ;
+ } else {
+ u32 target = start_lo + ticks;
+
+ while (__builtin_arc_lr(PMU_AUX_RTC_LOW) < target &&
+ __builtin_arc_lr(PMU_AUX_RTC_HIGH) == start_hi)
+ ;
+ }
+}
+
+/**
+ * pmu_delay_us() - Convert microsecond/unit delay into timer ticks and wait
+ * @count: Delay duration count
+ * @unit: Delay unit selector passed to pmu_freq_delay_step_calc()
+ */
+void pmu_delay_us(u32 count, u32 unit)
+{
+ pmu_hw_timer_delay(pmu_freq_delay_step_calc(count, unit));
+}
+
+/**
+ * pmu_cal_strobe_pulse_init() - Reset PHY strobe registers and DCCM 0x458 markers
+ * Return: Always 0.
+ */
+u32 pmu_cal_strobe_pulse_init(void)
+{
+ pmu_phy_deskew_dly0_set(0U);
+ pmu_phy_deskew_dly1_set(0U);
+ pmu_cal_marker_pair_set(0U);
+ pmu_cal_marker_clear();
+ return 0U;
+}
+
+/**
+ * pmu_phy_pll_lock_poll() - Poll PHY clock gate status 0x90040042 with timeout
+ * @timeout: Maximum poll iterations before raising assertion 0x00240000
+ */
+static noinline void pmu_phy_pll_lock_poll(u32 timeout)
+{
+ u32 locked = 0U;
+ u32 i;
+
+ for (i = 0U; i < timeout; i++) {
+ if (*(__volatile__ u16 *)PHY_REG_CLK_GATE_STAT != 0U) {
+ locked = 1U;
+ break;
+ }
+ }
+
+ if (!locked)
+ pmu_assert_or_halt(0U, PMU_ERR_PLL_LOCK_TIMEOUT);
+}
+
+/**
+ * pmu_cal_slice_step_diff_latch() - Latch slice step difference and timing strobe
+ * @timing: 16-bit timing parameter written to PHY 0x9004004e
+ * @wait_ack: Non-zero to poll PHY 0x90040042 (timeout 25000) and clear 0x90040040
+ */
+void pmu_cal_slice_step_diff_latch(u32 timing, u32 wait_ack)
+{
+ pmu_phy_clk_timing_set((u16)(timing));
+ pmu_hw_timer_delay(6U);
+ pmu_phy_deskew_dly1_set((u16)(pmu_cal_stride_get()) - 1U);
+ pmu_phy_clk_gate_set();
+ if (wait_ack != 0U) {
+ pmu_phy_pll_lock_poll(25000U);
+ pmu_phy_clk_gate_clear();
+ }
+}
+
+/**
+ * pmu_cal_slice_timing_latch() - Latch slice timing value and strobe PHY 0x90040040
+ * @timing: 16-bit timing parameter written to PHY 0x9004004e
+ * @wait_ack: Non-zero to poll PHY 0x90040042 (timeout 25000) and clear 0x90040040
+ */
+void pmu_cal_slice_timing_latch(u32 timing, u32 wait_ack)
+{
+ pmu_phy_clk_timing_set((u16)(timing));
+ pmu_hw_timer_delay(6U);
+ pmu_phy_clk_gate_set();
+ if (wait_ack != 0U) {
+ pmu_phy_pll_lock_poll(25000U);
+ pmu_phy_clk_gate_clear();
+ }
+}
+
+/**
+ * pmu_cal_delay_line_accum_dispatch() - Compute frequency-scaled delay with floor
+ * @count: Raw delay duration count passed to pmu_freq_delay_step_calc()
+ * @limit: Minimum step floor returned when computed ticks are smaller
+ *
+ * Return: max(pmu_freq_delay_step_calc(@count, 0), @limit).
+ */
+u32 pmu_cal_delay_line_accum_dispatch(u32 count, u32 limit)
+{
+ u32 ticks = pmu_freq_delay_step_calc(count, 0U);
+
+ return (ticks < limit) ? limit : ticks;
+}
+
+/**
+ * pmu_freq_delay_step_calc() - Convert delay duration and unit into timer steps
+ * @count: Requested delay duration
+ * @unit: Calibration delay unit multiplier
+ *
+ * Scales @count by DRAM frequency (DCCM 0x80000006) and rounding bias
+ * (DCCM 0x80000008), using overflow-safe division for @count > 200000, and
+ * enforces the minimum calibration delay floor from DCCM 0x80000a7c/0x80000b64.
+ *
+ * Return: Computed timer step count.
+ */
+noinline u32 pmu_freq_delay_step_calc(u32 count, u32 unit)
+{
+ u32 f = dmem_read16(PMU_DMEM_DRAM_FREQ_MHZ);
+ u32 t = dmem_read8(PMU_DMEM_PLL_RATIO);
+ u32 cycles;
+ u32 cycle_delay;
+ u32 cal_delay;
+
+ if (t == 0U)
+ t = 2U;
+
+ if (count > 200000U)
+ cycles = (count / 100000U) * f;
+ else
+ cycles = (count * f) / 100000U;
+ cycles /= 20U;
+
+ cycle_delay = (cycles + t + 1U) / t;
+ cal_delay = unit + 1U;
+ if (dmem_read8(PMU_DMEM_AC_CAL_ACTIVE) != 0U &&
+ (dmem_read8(PMU_DMEM_CLK_GATE_FLAG) & 8U) == 0U) {
+ u32 w_steps = dmem_read8(PMU_DMEM_SEARCH_WINDOW_STEPS);
+
+ if (w_steps != 0U)
+ cal_delay = w_steps * unit *
+ dmem_read8(PMU_DMEM_SEARCH_WINDOW_SCALE);
+ }
+
+ return (cal_delay < cycle_delay) ? cycle_delay : cal_delay;
+}
+
+/**
+ * pmu_clk_gate_handoff() - Hand off PHY PLL clock gate and wait 5000 units
+ *
+ * Clears PHY register 0x90040062, spins until PHY status register 0x90040042 is
+ * non-zero, clears PHY clock gate register 0x90040040, and delays 5000 units.
+ */
+void pmu_clk_gate_handoff(void)
+{
+ u32 timeout = 100000U;
+
+ phy_write16(PHY_REG_PLL_CLK_CTRL, 0x0000U);
+ while (phy_read16(PHY_REG_CLK_GATE_STAT) == 0U) {
+ if (--timeout == 0U)
+ pmu_assert_or_halt(0U, PMU_ERR_CLK_GATE_TIMEOUT);
+ pmu_delay_us(1U, 0U);
+ }
+ phy_write16(PHY_REG_CLK_GATE, 0x0000U);
+ pmu_delay_us(5000U, 0U);
+}
+
+/**
+ * pmu_phy_analog_cfg_mode_set() - Program channel analog receiver bias registers
+ * @mode: Channel mode selector (0: Ch0 only, 1: Ch1 only, >=2: both Ch0 and Ch1)
+ */
+void pmu_phy_analog_cfg_mode_set(u32 mode)
+{
+ u8 m = (u8)mode;
+ u16 v0 = (m != 1U) ? dmem_read8(PMU_DMEM_CH0_ANALOG_EN) : 0U;
+ u16 v1 = (m != 0U) ? dmem_read8(PMU_DMEM_CH1_ACTIVE_FLAG) : 0U;
+
+ phy_write16(PHY_REG_CH0_ANALOG_PWR, v0);
+ phy_write16(PHY_REG_CH1_ANALOG_PWR, v1);
+}
+
+/**
+ * pmu_cal_pll_lock_settle_spin() - Interrogate AC and DBYTE PLL lock status
+ * @out_dbyte_pll: Output pointer receiving DBYTE PLL lock mask (0x900201c0 & 0x01ff)
+ * @out_ac_pll: Output pointer receiving AC PLL lock mask (0x9006017e & 0x01ff)
+ */
+void pmu_cal_pll_lock_settle_spin(u16 *out_dbyte_pll, u16 *out_ac_pll)
+{
+ __volatile__ u16 *p1 = (__volatile__ u16 *)PHY_REG_AC_PLL_CTRL;
+ __volatile__ u16 *p2;
+ u32 v0;
+ u32 v4;
+ u32 saved;
+
+ v0 = p1[3];
+ v4 = p1[5];
+ saved = (v4 << 16) | v0;
+ p1[5] = (u16)(v4 & ~0x001fU);
+ p1[3] = (u16)(v0 | 0x0800U);
+ p1[3] = (u16)(v0 | 0x0c00U);
+ pmu_hw_timer_delay(10U);
+ p2 = (__volatile__ u16 *)PHY_REG_AC_PLL_CTRL;
+ p2[3] = (u16)(saved | 0x0800U);
+ *out_ac_pll = *(__volatile__ const u16 *)PHY_REG_AC_PLL_LOCK &
+ 0x01ffU;
+ *out_dbyte_pll = *(__volatile__ const u16 *)PHY_REG_DBYTE_PLL_LOCK &
+ 0x01ffU;
+ p2[3] = (u16)saved;
+ p2[5] = (u16)(saved >> 16);
+}
+
+/**
+ * pmu_cal_seq_table_timed_replay() - Replay 6-byte [addr, data] sequence records
+ * @seq_table: Pointer to array of 6-byte [u32 addr, u16 data] sequence records
+ * @total_bytes: Total byte length of @seq_table (divided by 6 for record count)
+ */
+void pmu_cal_seq_table_timed_replay(const void *seq_table, u32 total_bytes)
+{
+ const u8 *ptr = (const u8 *)seq_table;
+ u32 count = total_bytes / 6U;
+ u32 i;
+
+ for (i = 0U; i < count; i++) {
+ u32 addr = (u32)ptr[0] | ((u32)ptr[1] << 8) |
+ ((u32)ptr[2] << 16) | ((u32)ptr[3] << 24);
+ u16 data = (u16)((u16)ptr[4] | ((u16)ptr[5] << 8));
+
+ ptr += 6U;
+ if (addr == 0xffffffffUL)
+ pmu_hw_timer_delay(data);
+ else
+ *(__volatile__ u16 *)(PHY_REG_PUB_BASE |
+ (addr << 1)) = data;
+ }
+}
+
+/**
+ * pmu_phy_pll_cfg_quad_write_stream() - Write 4 PLL words and append shadow trace
+ * @reg_offset: Register offset shifted left by 1 into 0x90140000
+ * @p0: First 16-bit parameter word
+ * @p1: Second 16-bit parameter word
+ * @p2: Third parameter word (low 16 bits written to +4, bits [18:16] to tag)
+ * @flags: Control flags shifted left by 3 into the 4th tag halfword
+ */
+void pmu_phy_pll_cfg_quad_write_stream(u32 reg_offset, u16 p0, u16 p1, u32 p2,
+ u32 flags)
+{
+ __volatile__ u16 *reg =
+ (__volatile__ u16 *)(PHY_REG_SHADOW_TRACK_BASE |
+ (reg_offset << 1));
+ u16 *trace = (u16 *)(*(__volatile__ u32 *)PMU_DMEM_SHADOW_TRACE_PTR);
+ u16 tag = (u16)(((flags & 0xffU) << 3) | ((p2 >> 16) & 0x07U));
+
+ reg[0] = p0;
+ reg[1] = p1;
+ reg[2] = (u16)p2;
+ reg[3] = tag;
+
+ trace[0] = p0;
+ trace[1] = p1;
+ trace[2] = (u16)p2;
+ trace[3] = tag;
+ *(__volatile__ u32 *)PMU_DMEM_SHADOW_TRACE_PTR = (u32)(trace + 4);
+}
+
+/**
+ * pmu_phy_clk_phase_toggle_pulse() - Toggle PHY clock phase and settle PLL aligners
+ *
+ * Pulses the master PHY clock phase and clear strobes around 0x90040606, delays
+ * for phase stabilization, and restores the original 8-bit configuration.
+ */
+void pmu_phy_clk_phase_toggle_pulse(void)
+{
+ __volatile__ u16 *p = (__volatile__ u16 *)PHY_REG_CLK_PHASE_BASE;
+ u16 orig;
+
+ orig = p[3];
+ p[16] = 1U;
+ p[3] = (orig & 0x00efU) | 0x0010U;
+ p[16] = 0U;
+ p[13] = 1U;
+ p[13] = 0U;
+ p[17] = 1U;
+ pmu_delay_us(0x07bfa480UL, 0U);
+ p[17] = 0U;
+ p[32] = 0U;
+ p[32] = 1U;
+ p[32] = 0U;
+ p[3] = (u8)orig;
+}
+
+/**
+ * pmu_cal_pad_timing_delay_set() - Stream pad timing delay halfword frames
+ * @bypass_min_clamp: Non-zero to force val = step regardless of @steps
+ * @steps: Target delay steps
+ * @use_fixed_unit: Non-zero for unit step 1, 0 for (DCCM 0x8000045a << 1)
+ */
+void pmu_cal_pad_timing_delay_set(u32 bypass_min_clamp, u32 steps,
+ u32 use_fixed_unit)
+{
+ u16 marker_val = pmu_sdata()->cal_marker_a;
+ u16 buf[8];
+ u32 step = (use_fixed_unit == 0U) ?
+ (u32)((u8)(pmu_sdata()->cbt_state << 1)) : 1U;
+ u32 val, div;
+
+ if (step == 0U)
+ step = 1U;
+ val = (bypass_min_clamp == 0U && steps > step) ? steps : step;
+ div = val / step;
+ u32 count;
+ u16 v3 = 0U;
+ u16 v7 = 0U;
+
+ if (val != div * step)
+ div++;
+ count = (div - 1U) & 0xffffU;
+
+ memset(buf, 0, sizeof(buf));
+ if ((marker_val & 0x7U) == 4U) {
+ memset((void *)pmu_tracker()->slot1, 0,
+ sizeof(pmu_tracker()->slot1));
+ pmu_cal_marker_status_update();
+ pmu_sdata()->cal_marker_a = marker_val + 4U;
+ }
+
+ while (count > 255U) {
+ u16 s0 = 0x1fU << 10;
+
+ buf[3] = (u16)(s0 - 0x100U);
+ buf[7] = s0;
+ pmu_cal_tracker_reset_and_apb_zero(buf, 8U);
+ count -= 255U;
+ }
+
+ if (count != 0U) {
+ v3 = (u16)(((count << 12) & 0x7000U) | 0x0b00U);
+ v7 = (u16)((count << 7) & 0x7c00U);
+ }
+ buf[3] = v3;
+ buf[7] = v7;
+
+ pmu_cal_tracker_reset_and_apb_zero(buf, 8U);
+}
+
+/**
+ * pmu_pll_quad_param_tuple_stream() - Stream 2x2 PLL quad parameter tuples
+ * @param_array: Input 8-halfword parameter array
+ * @out_ctr: In/out trace counter incremented per streamed tuple
+ * @base_ptr: In/out base register index incremented per outer iteration
+ */
+void pmu_pll_quad_param_tuple_stream(const u16 *param_array, u16 *out_ctr,
+ u16 *base_ptr)
+{
+ u32 iter;
+
+ for (iter = 0U; iter < 4U; iter++) {
+ if ((dmem_read32(PMU_DMEM_PLL_GATE_CH0_ADDR) |
+ dmem_read16(PMU_DMEM_PLL_SEQ_EN_ADDR)) != 0U) {
+ u32 target = ((u32)(*base_ptr) << 1) +
+ (iter & 1U) + 0x20800UL;
+
+ pmu_phy_pll_cfg_quad_write_stream((u32)(*out_ctr) << 2,
+ param_array[0],
+ param_array[1],
+ target & 0x7ffffUL,
+ 3U);
+ *out_ctr += 1U;
+ }
+ param_array += 2;
+ if ((iter & 1U) != 0U)
+ *base_ptr += 1U;
+ }
+}
+
+/**
+ * pmu_phy_mode_cfg_dispatch() - Configure PHY analog bias and channel bounds for mode
+ * @mode: Channel/rank dispatch mode (1 = Ch0, 2 = Ch1, else broadcast)
+ */
+void pmu_phy_mode_cfg_dispatch(u32 mode)
+{
+ __volatile__ u16 *apb = (__volatile__ u16 *)PHY_REG_CH0_ANALOG_PWR;
+ u8 m = (u8)mode;
+ u16 v0 = dmem_read8(PMU_DMEM_CH0_ANALOG_EN);
+ u16 v1 = dmem_read8(PMU_DMEM_CH1_ACTIVE_FLAG);
+
+ if (m == 2U)
+ v0 = 0U;
+ if (m == 1U)
+ v1 = 0U;
+ apb[0] = v0;
+ apb[1] = v1;
+ apb[12] = 0U;
+
+ if (m == 2U) {
+ dmem_write16(PMU_DMEM_CH_RANGE,
+ dmem_read16(PMU_DMEM_CH_SEC_RANGE));
+ apb[8] = 0x000fU;
+ } else if (m == 1U) {
+ dmem_write16(PMU_DMEM_CH_RANGE,
+ dmem_read16(PMU_DMEM_CH_ACTIVE_RANGE));
+ apb[8] = 0x00f0U;
+ } else {
+ apb[8] &= ~0x0011U;
+ dmem_write8(PMU_DMEM_SLICE_START,
+ dmem_read8(PMU_DMEM_RANK0_SLICE_START));
+ dmem_write8(PMU_DMEM_SLICE_END,
+ dmem_read8(PMU_DMEM_RANK1_SLICE_END));
+ apb[8] = 0U;
+ }
+
+ pmu_profile_param_stride_cfg(mode);
+}
+
+/**
+ * pmu_clk_ratio_pulse_trigger() - Trigger frequency-scaled 2000-step pad timing delay
+ */
+void pmu_clk_ratio_pulse_trigger(void)
+{
+ u32 scaled = pmu_freq_ratio_mult(2000U);
+
+ pmu_cal_pad_timing_delay_set(0U, (u16)scaled, 0U);
+}
+
+/**
+ * pmu_phy_pll_clock_divider_align() - Compute DBYTE slice bounds and active slice mask
+ */
+void pmu_phy_pll_clock_divider_align(void)
+{
+ u8 dq0 = dmem_read8(PMU_DMEM_CH0_DQ_WIDTH);
+ u8 dq1 = dmem_read8(PMU_DMEM_CH1_DQ_WIDTH);
+ u8 rank1_en = dmem_read8(PMU_DMEM_CH1_ACTIVE_FLAG);
+ u8 d1c = dmem_read8(PMU_DMEM_DESKEW_LIM);
+ u8 num_slices0 = (u8)((dq0 + 7U) >> 3);
+ u8 end_slice0 = num_slices0 - 1U;
+ u8 start_slice1 = num_slices0;
+ u8 end_slice1;
+ u8 mask;
+ int i;
+ u32 rank, ch;
+
+ dmem_write8(PMU_DMEM_RANK0_SLICE_START, 0U);
+ dmem_write8(PMU_DMEM_RANK_BOUNDARY, end_slice0);
+
+ if (rank1_en != 0U) {
+ start_slice1 = (u8)(pmu_sdata()->active_slice_cnt >> 1);
+ num_slices0 = (u8)(((dq1 + 7U) >> 3) + start_slice1);
+ end_slice1 = num_slices0 - 1U;
+ } else {
+ end_slice1 = end_slice0;
+ }
+ dmem_write8(PMU_DMEM_RANK1_SLICE_START, start_slice1);
+ dmem_write8(PMU_DMEM_RANK1_SLICE_END, end_slice1);
+
+ mask = dmem_read8(PMU_DMEM_ACTIVE_SLICE_MASK);
+ for (i = 0; i <= (int)end_slice0; i++)
+ mask |= (u8)(1U << i);
+ dmem_write8(PMU_DMEM_ACTIVE_SLICE_MASK, mask);
+
+ if (end_slice1 >= start_slice1) {
+ for (i = (int)start_slice1; i <= (int)end_slice1; i++)
+ mask |= (u8)(1U << i);
+ dmem_write8(PMU_DMEM_ACTIVE_SLICE_MASK, mask);
+ }
+
+ for (rank = 0U; rank < 2U; rank++) {
+ for (ch = 0U; ch < 2U; ch++) {
+ u8 dq = (ch == 0U) ? dq0 : dq1;
+ u8 shift = (u8)(ch * 2U);
+ u8 slice_cnt = ((d1c >> shift) & (1U << rank)) ?
+ (u8)((dq + 7U) >> 3) :
+ (u8)((dq + 15U) >> 4);
+
+ dmem_write8(PMU_DMEM_SLICE_CNT_LIMIT +
+ ch * 2U + rank, slice_cnt);
+ }
+ }
+}
+
+/**
+ * pmu_profile_param_stride_cfg() - Broadcast profile delay across slices and pulse strobe
+ * @mode: Channel/rank mode selector (3 = max across channels, else offset)
+ */
+static void pmu_profile_param_stride_cfg(u32 mode)
+{
+ u8 byte_idx;
+ u32 val;
+ u32 csr_ch_off;
+ u32 count;
+ u32 i;
+ u32 reg;
+ __volatile__ u16 *strobe;
+
+ if (dmem_read16(PMU_DMEM_SEQ_CTRL) == 1U)
+ return;
+
+ byte_idx = dmem_read8(PMU_DMEM_ACTIVE_RANK);
+ if (mode == 3U) {
+ const u16 *tbl = (const u16 *)
+ dmem_ptr(PMU_DMEM_CAL_TABLE + byte_idx * 4U);
+
+ val = tbl[0];
+ if (dmem_read8(PMU_DMEM_CH1_ACTIVE_FLAG) != 0U && tbl[1] > val)
+ val = tbl[1];
+ } else {
+ u32 offset = (mode & ~1U) + byte_idx * 4U;
+
+ val = dmem_read8(PMU_DMEM_CAL_TABLE + offset);
+ }
+
+ csr_ch_off = pmu_sdata()->active_csr_offset << 1;
+ val = (u16)(s8)val;
+ *(__volatile__ u16 *)(PHY_REG_MASTER_DELAY | csr_ch_off) = (u16)val;
+
+ count = pmu_sdata()->active_slice_cnt;
+ reg = PHY_REG_DBYTE_BASE | csr_ch_off;
+ for (i = 0U; i <= count; i++, reg += PHY_DBYTE_SLICE_STRIDE)
+ *(__volatile__ u16 *)reg = (u16)val;
+
+ strobe = (__volatile__ u16 *)PHY_REG_PUB_CAL_STROBE;
+ *strobe = 1U;
+ pmu_hw_timer_delay(0x20U);
+ *strobe = 0U;
+}
+
+/**
+ * pmu_cal_tracker_csr_commit() - Commit multi-channel calibration step tracker to CSR stream
+ * @tracker: Pointer to 22-halfword calibration step tracker structure
+ *
+ * Return: Updated 16-bit calibration marker from 0x8000045e.
+ */
+u32 pmu_cal_tracker_csr_commit(__volatile__ void *tracker)
+{
+ __volatile__ u16 *t = (__volatile__ u16 *)tracker;
+ u8 mask = (u8)pmu_sdata()->active_slice_mask;
+ u32 popcount = 0U;
+ u16 marker_val;
+ u8 step_ext_val;
+ u8 ch_range_val;
+ u16 has_step_ext;
+ u16 sample[8];
+ u32 base;
+ u32 i;
+
+ while (mask != 0U) {
+ mask &= (u8)(mask - 1U);
+ popcount++;
+ }
+
+ marker_val = pmu_sdata()->cal_marker_b;
+ step_ext_val = pmu_sdata()->cbt_step_ext;
+ ch_range_val = pmu_sdata()->ch_range_lim;
+ has_step_ext = (step_ext_val != 0U) ? 1U : 0U;
+
+ t[19] = has_step_ext;
+
+ sample[0] = (t[3] & 0x7fU) | ((t[4] & 0x7fU) << 7) | (t[5] << 14);
+ sample[1] = (t[6] & 0x0fU) | (t[7] << 4);
+ sample[2] = (t[7] >> 12) & 1U;
+ sample[3] = (u16)((u8)(t[2] << 7)) |
+ ((t[0] << 4) & 0x10U) |
+ ((t[1] << 5) & 0x60U) |
+ ((t[18] << 8) & 0x300U) |
+ ((t[20] << 11) & 0x800U) |
+ (has_step_ext << 10) |
+ ((t[17] << 12) & 0x7000U);
+
+ sample[4] = (t[11] & 0x7fU) | ((t[12] & 0x7fU) << 7) | (t[13] << 14);
+ sample[5] = (t[14] & 0x0fU) | (t[15] << 4);
+ sample[6] = (t[15] >> 12) & 1U;
+ sample[7] = (u16)((u8)(t[10] << 7)) |
+ ((t[8] << 4) & 0x10U) |
+ ((t[9] << 5) & 0x60U) |
+ ((t[16] << 8) & 0x300U) |
+ ((t[17] << 7) & 0x7c00U);
+
+ if (ch_range_val != 0U) {
+ s32 sum;
+
+ if (ch_range_val != 1U)
+ return (u32)marker_val;
+
+ sum = (s8)popcount + pmu_sdata()->tracker_popcnt_acc;
+ if (sum < 8) {
+ *(__volatile__ u8 *)&pmu_sdata()->active_slice_mask = 0U;
+ pmu_sdata()->tracker_popcnt_acc = sum % 8;
+ return (u32)marker_val;
+ }
+ }
+
+ base = PHY_REG_CSR_RESULT_STREAM | ((u32)marker_val << 1);
+ for (i = 0U; i < 8U; i++)
+ *(__volatile__ u16 *)(base + (i * 2U)) = sample[i];
+
+ marker_val += 8U;
+ pmu_sdata()->cal_marker_b = marker_val;
+
+ if ((t[18] & 3U) != 0U)
+ pmu_sdata()->step_mode_ovr = (u8)(1U & ~t[18]);
+
+ if (ch_range_val == 1U) {
+ s32 sum;
+
+ pmu_tracker()->slot1_mode = 0U;
+ sum = (s8)popcount + pmu_sdata()->tracker_popcnt_acc;
+ *(__volatile__ u8 *)&pmu_sdata()->active_slice_mask = 0U;
+ pmu_sdata()->tracker_popcnt_acc = sum % 8;
+ }
+
+ return (u32)marker_val;
+}
+
+/**
+ * pmu_pll_clk_gate_cfg() - Program PLL clock timing and PUB transceiver gates
+ */
+void pmu_pll_clk_gate_cfg(void)
+{
+ u32 csr_sh1 = pmu_sdata()->active_csr_offset << 1;
+ u16 val0 = *(__volatile__ const u16 *)(csr_sh1 | PHY_REG_DBYTE_XCVR_GATE);
+ __volatile__ u16 *phy_base = (__volatile__ u16 *)PHY_REG_MASTER_BASE;
+ u8 cal_mode_flag;
+
+ *(__volatile__ u16 *)(csr_sh1 | PHY_REG_DBYTE_BCAST_XCVR_GATE) =
+ (val0 & 0xff78U) | 6U;
+
+ phy_base[0x00c2 >> 1] = 1U;
+
+ cal_mode_flag = *(__volatile__ const u8 *)PMU_DMEM_CAL_MODE_FLAG_ADDR;
+ if (cal_mode_flag != 0U) {
+ pmu_delay_us(0U, 4U);
+ phy_base[0x00c2 >> 1] = 0U;
+ phy_base[0x01c2 >> 1] = 1U;
+ phy_base[0x01c0 >> 1] = 1U;
+ phy_base[0x0620 >> 1] = 0U;
+ pmu_delay_us(0x989680U, 0U);
+ phy_base[0x0186 >> 1] = 0U;
+ phy_base[0x0400 >> 1] = 0x33U;
+ phy_base[0x0144 >> 1] = 0U;
+ pmu_hw_timer_delay(0x20U);
+ phy_base[0x00f0 >> 1] = 1U;
+ phy_base[0x0144 >> 1] = 2U;
+ pmu_hw_timer_delay(1024U);
+ } else {
+ pmu_hw_timer_delay(4U);
+ phy_base[0x0092 >> 1] = 496U;
+ phy_base[0x00c2 >> 1] = 0U;
+ phy_base[0x01c2 >> 1] = 0U;
+ phy_base[0x0620 >> 1] = 0U;
+ phy_base[0x01c0 >> 1] = 0U;
+ phy_base[0x0186 >> 1] = 0U;
+ pmu_delay_us(0xf4240U, 0U);
+ phy_base[0x01c4 >> 1] = 3U;
+ pmu_delay_us(0xf4240U >> 1, 0U);
+ phy_base[0x01c4 >> 1] = 2U;
+ pmu_delay_us(0x989680U, 0U);
+ phy_base[0x01c4 >> 1] = 0U;
+ phy_base[0x00f0 >> 1] = 1U;
+ phy_base[0x0400 >> 1] = 0x33U;
+ phy_base[0x0144 >> 1] = 0U;
+ pmu_hw_timer_delay(6U);
+ phy_base[0x00f0 >> 1] = 2U;
+ pmu_delay_us(0U, 0x2710U);
+ phy_base[0x0144 >> 1] = 2U;
+ }
+
+ phy_base[0x00f0 >> 1] = 0U;
+ phy_base[0x0144 >> 1] = 0U;
+ pmu_delay_us(0U, 0x20U);
+ phy_base[0x0400 >> 1] = 0U;
+ phy_base[0x01f0 >> 1] = 0U;
+ phy_base[0x0144 >> 1] = 2U;
+}
+
+/**
+ * pmu_channel_rank_avail_check() - Dispatch mode configuration for active channel
+ * @channel: Target channel index (0 = Ch0, 1 = Ch1, >1 = broadcast)
+ * @rank: Rank bit index to test in Ch0 (0x80000025) or Ch1 (0x80000040)
+ *
+ * Return: 1 if dispatched (mode 1, 2, or 3), or 0 if rank bit is inactive.
+ */
+u32 pmu_channel_rank_avail_check(u32 channel, u32 rank)
+{
+ u32 mode = 3U;
+
+ if (channel <= 1U) {
+ u8 flags = (channel == 0U) ?
+ dmem_read8(PMU_DMEM_CH0_ANALOG_EN) :
+ dmem_read8(PMU_DMEM_CH1_ACTIVE_FLAG);
+
+ if (((flags >> rank) & 1U) == 0U)
+ return 0U;
+ mode = channel + 1U;
+ }
+
+ pmu_phy_mode_cfg_dispatch(mode);
+ return 1U;
+}
+
+/**
+ * pmu_phy_clk_gate_clear() - Clear PHY control register 0x90040040 to 0
+ */
+static inline void pmu_phy_clk_gate_clear(void)
+{
+ *(__volatile__ u16 *)PHY_REG_CLK_GATE = 0;
+}
+
+/**
+ * pmu_phy_cal_stat_read() - Read 16-bit value from PHY register 0x90040148
+ *
+ * Return: 16-bit register value at 0x90040148.
+ */
+u16 pmu_phy_cal_stat_read(void)
+{
+ return *(__volatile__ u16 *)PHY_REG_CAL_STAT;
+}
+
+/**
+ * pmu_phy_dbyte_cal_trig_pulse() - Execute 5-step strobe pulse on PHY reg 0x176
+ * @slice: Target DBYTE slice index (0..3)
+ * @bit_idx: Lane bit index (0..8, or 15 for all 9 lanes 0x1ff)
+ *
+ * Sequences 0 -> mask -> (mask | 0x200) -> mask -> 0 on the slice's PHY
+ * register at 0x90020176 + (slice << 13).
+ */
+void pmu_phy_dbyte_cal_trig_pulse(u32 slice, u32 bit_idx)
+{
+ __volatile__ u16 *reg =
+ (__volatile__ u16 *)(PHY_REG_DBYTE_CAL_TRIG + (slice << 13));
+ u32 mask = (bit_idx == 15) ? 511U : (1U << bit_idx);
+ u32 strobe = mask | (1U << 9);
+ u16 zero = 0;
+
+ *reg = zero;
+ *reg = (u16)mask;
+ *reg = (u16)strobe;
+ *reg = (u16)mask;
+ *reg = zero;
+}
+
+/**
+ * pmu_phy_sec_cal_strobe_pulse() - Pulse PHY register 0x9005e140 high for 32 timer ticks
+ */
+void pmu_phy_sec_cal_strobe_pulse(void)
+{
+ *(__volatile__ u16 *)PHY_REG_PUB_CAL_STROBE = 1;
+ pmu_hw_timer_delay(32);
+ *(__volatile__ u16 *)PHY_REG_PUB_CAL_STROBE = 0;
+}
+
+/**
+ * pmu_phy_cal_trig_pulse() - Pulse PHY register 0x90040146 high for 32 timer ticks
+ */
+void pmu_phy_cal_trig_pulse(void)
+{
+ *(__volatile__ u16 *)PHY_REG_CAL_TRIG = 1;
+ pmu_hw_timer_delay(32);
+ *(__volatile__ u16 *)PHY_REG_CAL_TRIG = 0;
+}
+
+/**
+ * pmu_phy_soft_reset_pulse() - Toggle PHY register 0x900400f2 high then low
+ */
+void pmu_phy_soft_reset_pulse(void)
+{
+ __volatile__ u16 *p = (__volatile__ u16 *)PHY_REG_RESET_TRIGGER;
+
+ *p = 1;
+ *p = 0;
+}
+
+/**
+ * pmu_phy_clk_gate_set() - Set PHY control register 0x90040040 to 1
+ */
+static inline void pmu_phy_clk_gate_set(void)
+{
+ *(__volatile__ u16 *)PHY_REG_CLK_GATE = 1;
+}
+
+/**
+ * pmu_phy_deskew_dly0_set() - Write 16-bit value to channel-indexed PHY reg 0x90040044
+ * @val: Halfword value to write
+ */
+void pmu_phy_deskew_dly0_set(u16 val)
+{
+ u32 ch_offset = pmu_sdata()->active_csr_offset;
+
+ *(__volatile__ u16 *)(PHY_REG_DESKEW_DLY0 + (ch_offset << 1)) = val;
+}
+
+/**
+ * pmu_phy_deskew_dly1_set() - Write 16-bit value to channel-indexed PHY reg 0x90040046
+ * @val: Halfword value to write
+ */
+void pmu_phy_deskew_dly1_set(u16 val)
+{
+ u32 ch_offset = pmu_sdata()->active_csr_offset;
+
+ *(__volatile__ u16 *)(PHY_REG_DESKEW_DLY1 + (ch_offset << 1)) = val;
+}
+
+/**
+ * pmu_phy_clk_timing_set() - Write 16-bit value to PHY register 0x9004004e
+ * @val: Halfword value to write
+ */
+void pmu_phy_clk_timing_set(u16 val)
+{
+ *(__volatile__ u16 *)PHY_REG_CLK_TIMING_CTRL = val;
+}
+
+/**
+ * pmu_phy_clk_gate_sync_pulse() - Conditionally pulse PHY clock gate reg 0x9005e0a4
+ *
+ * When bit 0 of DCCM 0x80000001 is set, asserts PHY register 0x9005e0a4 to 1,
+ * waits 10 hardware timer ticks, and clears it back to 0.
+ */
+void pmu_phy_clk_gate_sync_pulse(void)
+{
+ u8 flag = *(__volatile__ u8 *)PMU_DMEM_CLK_GATE_FLAG;
+
+ if ((flag & 1) != 0) {
+ *(__volatile__ u16 *)PHY_REG_SEC_CLK_GATE = 1;
+ pmu_hw_timer_delay(10);
+ *(__volatile__ u16 *)PHY_REG_SEC_CLK_GATE = 0;
+ }
+}
+
+/**
+ * pmu_phy_pll_cfg_quad_write() - Broadcast halfword mask to PHY 0x900e00c2..0x900e00c8
+ * @v0: Halfword mask written to 0x900e00c2, 0x900e00c4, 0x900e00c6, and 0x900e00c8
+ */
+void pmu_phy_pll_cfg_quad_write(u16 v0)
+{
+ __volatile__ u16 *p = (__volatile__ u16 *)PHY_REG_PLL_CFG;
+
+ p[0] = v0;
+ p[1] = v0;
+ p[2] = v0;
+ p[3] = v0;
+}
diff --git a/arch/arm/mach-sunxi/pmu/pmu_primitives.c b/arch/arm/mach-sunxi/pmu/pmu_primitives.c
new file mode 100644
index 00000000000..23277c7f7d3
--- /dev/null
+++ b/arch/arm/mach-sunxi/pmu/pmu_primitives.c
@@ -0,0 +1,531 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Copyright (C) 2026 Alastair D'Silva <alastair@d-silva.org>
+ *
+ * Allwinner A733 Synopsys DDR PHY PMU Foundation Memory Primitives
+ */
+
+#include "pmu_fw.h"
+
+/**
+ * memset() - Fill memory region with a constant byte value
+ * @s: Pointer to destination memory block
+ * @c: Byte value to replicate across @n bytes
+ * @n: Number of bytes to set
+ *
+ * Aligns the destination pointer to a 4-byte boundary using byte stores, fills
+ * 32-bit words using the replicated pattern (u8)@c * 0x01010101UL, and writes
+ * any trailing bytes.
+ *
+ * Return: Original destination pointer @s.
+ */
+void *memset(void *s, int c, size_t n)
+{
+ u8 *p = (u8 *)s;
+ u8 val = (u8)c;
+ u32 pat = (u32)val * 0x01010101UL;
+
+ while (n > 0 && ((uintptr_t)p & 3U) != 0U) {
+ *p++ = val;
+ n--;
+ }
+
+ while (n >= 4U) {
+ *(u32 *)p = pat;
+ p += 4;
+ n -= 4U;
+ }
+
+ while (n > 0) {
+ *p++ = val;
+ n--;
+ }
+
+ return s;
+}
+
+/**
+ * memcpy() - Copy memory region from source to destination
+ * @dest: Destination buffer pointer
+ * @src: Source buffer pointer
+ * @n: Number of bytes to copy
+ *
+ * Copies 32-bit words when both @dest and @src are 4-byte aligned, followed by
+ * any trailing bytes; otherwise copies byte-by-byte.
+ *
+ * Return: Original destination pointer @dest.
+ */
+void *memcpy(void *dest, const void *src, size_t n)
+{
+ u8 *d = (u8 *)dest;
+ const u8 *s = (const u8 *)src;
+
+ if ((((uintptr_t)d | (uintptr_t)s) & 3U) == 0U) {
+ while (n >= 4U) {
+ *(u32 *)d = *(const u32 *)s;
+ d += 4;
+ s += 4;
+ n -= 4U;
+ }
+ }
+
+ while (n > 0) {
+ *d++ = *s++;
+ n--;
+ }
+
+ return dest;
+}
+
+/**
+ * pmu_cal_tracker_step_advance() - Advance calibration step tracker across step count
+ * @marker: Initial 16-bit calibration marker
+ * @count: Number of step iterations to advance
+ * @mode: Step mode selector (1 for fast multi-step mode, 0 for single-step)
+ *
+ * Return: Updated 16-bit calibration marker.
+ */
+u32 pmu_cal_tracker_step_advance(u32 marker, u32 count, u32 mode)
+{
+ __volatile__ struct pmu_cal_tracker *trk = pmu_tracker();
+ __volatile__ u32 *slot0 = (__volatile__ u32 *)(uintptr_t)trk->slot0;
+ __volatile__ u32 *slot1 = (__volatile__ u32 *)(uintptr_t)trk->slot1;
+
+ while ((u16)count != 0U) {
+ if (count >= 4U && mode == 1U) {
+ if ((marker & 7U) != 0U) {
+ count--;
+ slot1[0] = 0U;
+ slot1[1] = 0U;
+ slot1[2] = 0U;
+ slot1[3] = 0U;
+ marker = pmu_cal_marker_status_update();
+ mode = 1U;
+ } else {
+ memset((void *)trk->slot0, 0,
+ PMU_TRACKER_SLOTS_BYTES);
+ if (count >= 513U) {
+ count -= 512U;
+ trk->repeat_cnt = 0x00ffU;
+ trk->slot0_mode = 0x0003U;
+ marker = pmu_cal_marker_status_update();
+ mode = 1U;
+ } else {
+ u32 half_cnt = (count >> 1) - 1U;
+
+ trk->repeat_cnt = (u16)half_cnt;
+ if ((count >> 1) == 0U) {
+ trk->slot0_mode = 0U;
+ marker = pmu_cal_marker_status_update();
+ goto reset_slot_next;
+ }
+ trk->slot0_mode = 0x0003U;
+ return (u16)(marker + 8U);
+ }
+ }
+ } else {
+ mode = 0U;
+ if ((marker & 7U) != 0U) {
+ slot1[0] = 0U;
+ slot1[1] = 0U;
+ slot1[2] = 0U;
+ slot1[3] = 0U;
+ if (count == 1U)
+ return (u16)(marker + 4U);
+ marker = pmu_cal_marker_status_update();
+ if (count == 2U)
+ goto reset_slot_next;
+ count--;
+ } else {
+ memset((void *)trk->slot0, 0,
+ PMU_TRACKER_SLOTS_BYTES);
+ if (count == 1U)
+ goto reset_slot_next;
+ if (count == 2U)
+ return (u16)(marker + 8U);
+ marker = pmu_cal_marker_status_update();
+ count -= 2U;
+ }
+ }
+ }
+
+ return pmu_sdata()->cal_marker_a;
+
+reset_slot_next:
+ slot0[0] = 0U;
+ slot0[1] = 0U;
+ slot0[2] = 0U;
+ slot0[3] = 0U;
+ return (u16)(marker + 4U);
+}
+
+/**
+ * pmu_default_exception_handler() - Dump ARC exception state to DCCM and halt
+ */
+__attribute__((naked))
+void pmu_default_exception_handler(void)
+{
+ __asm__ __volatile__("mov r2, 0x80000000\n\t"
+ "mov r3, 0x1234dead\n\t"
+ "st r3, [r2]\n\t"
+ "add_s r2, r2, 4\n\t"
+ "lr r3, [0x400]\n\t"
+ "st r3, [r2]\n\t"
+ "add_s r2, r2, 4\n\t"
+ "lr r3, [0x403]\n\t"
+ "st r3, [r2]\n\t"
+ "add_s r2, r2, 4\n\t"
+ "st r0, [r2]\n\t"
+ "add_s r2, r2, 4\n\t"
+ "st r1, [r2]\n\t"
+ "add_s r2, r2, 4\n\t"
+ "st sp, [r2]\n\t"
+ "add_s r2, r2, 4\n\t"
+ "st fp, [r2]\n\t"
+ "add_s r2, r2, 4\n\t"
+ "mov r3, 0xabcddead\n\t"
+ "st r3, [r2]\n\t"
+ "flag 1\n\t"
+ "nop_s\n\t");
+}
+
+/**
+ * pmu_start() - ARC EM4 CRT0 reset entry point
+ *
+ * Clears core registers, initializes the stack pointer (0x80010000) and
+ * small-data global pointer (0x80000400), invokes pmu_main_wrapper(),
+ * and halts the processor upon return.
+ */
+__attribute__((naked))
+void pmu_start(void)
+{
+ __asm__ __volatile__("mov_s r0, 0\n\t"
+ "mov_s r1, 0\n\t"
+ "mov_s r2, 0\n\t"
+ "mov_s r3, 0\n\t"
+ "mov_s r10, 0\n\t"
+ "mov_s r11, 0\n\t"
+ "mov_s r12, 0\n\t"
+ "mov_s r13, 0\n\t"
+ "mov_s r14, 0\n\t"
+ "mov_s r15, 0\n\t"
+ "mov_s gp, 0\n\t"
+ "mov_s fp, 0\n\t"
+ "mov_s sp, 0\n\t"
+ "mov ilink, 0\n\t"
+ "mov r30, 0\n\t"
+ "mov_s blink, 0\n\t"
+ "mov_s sp, 0x80010000\n\t"
+ "mov_s gp, 0x80000400\n\t"
+ "mov_s fp, 0\n\t"
+ "jl pmu_main_wrapper\n\t"
+ "1:\n\t"
+ "flag 1\n\t"
+ "nop_s\n\t"
+ "b_s 1b\n\t");
+}
+
+/**
+ * pmu_vector_table() - ARC EM4 interrupt and exception vector table
+ */
+__attribute__((naked))
+void pmu_vector_table(void)
+{
+ __asm__ __volatile__(".word pmu_start\n\t"
+ ".rept 15\n\t"
+ ".word pmu_default_exception_handler\n\t"
+ ".endr\n\t"
+ ".rept 4\n\t"
+ ".word 1f\n\t"
+ ".endr\n\t"
+ "1:\n\t"
+ "nop\n\t"
+ "nop\n\t"
+ "nop\n\t"
+ "nop\n\t"
+ "rtie\n\t");
+}
+
+/**
+ * pmu_cal_struct_mask_remap() - Remap calibration struct shadow masks
+ */
+void pmu_cal_struct_mask_remap(void)
+{
+ u32 base =
+ (PMU_DMEM_BASE | PMU_DMEM_CAL_STRUCT_BASE);
+
+ pmu_cal_eye_table_save((void *)(uintptr_t)(base), PMU_SLICE_DESC_MR16_OFF);
+ pmu_cal_lane_mask_or_all(base, PMU_SLICE_DESC_MR16_OFF, 0x41U);
+ pmu_cal_lane_mask_and_all(base, PMU_SLICE_DESC_MR16_OFF, 0xf3U);
+ pmu_cal_eye_table_save((void *)(uintptr_t)(base), PMU_SLICE_DESC_MR41_OFF);
+ pmu_cal_lane_mask_and_all(base, PMU_SLICE_DESC_MR41_OFF, 0x1fU);
+ pmu_multirank_slice_cal_dispatch((void *)(uintptr_t)base, 0U,
+ 0xfffeffffUL, (u32)-513, 0xffU, 0U);
+ pmu_delay_us(20000U, 0U);
+ pmu_cal_eye_table_restore((u8 *)(uintptr_t)base, PMU_SLICE_DESC_MR41_OFF);
+}
+
+/**
+ * pmu_adjust_step_limit() - Adjust coarse/fine step limit byte pair
+ * @pair: Two-byte array holding [0] coarse count and [1] fine tap offset
+ * @flag: Parity selector (non-zero adjusts odd counts; zero adjusts non-zero even counts)
+ *
+ * Decrements @pair[0] by 1 and adds 64 fine taps to @pair[1] when @pair[0]
+ * matches the parity condition selected by @flag.
+ */
+void pmu_adjust_step_limit(u8 *pair, u32 flag)
+{
+ u32 count = pair[0];
+
+ if (flag ? (count & 1U) : (count != 0U && !(count & 1U))) {
+ pair[0] = (u8)(count - 1U);
+ pair[1] = (u8)(pair[1] + 64U);
+ }
+}
+
+/**
+ * pmu_bist_pattern_bit_get() - Extract 1-bit truth value from 0xe4 lookup constant
+ * @shift: Left shift applied to @mask before indexing
+ * @mask: Input mask value
+ *
+ * Return: 1-bit lookup result ((0xe4 >> ((mask << shift) & 6)) & 1).
+ */
+u32 pmu_bist_pattern_bit_get(u32 shift, u32 mask)
+{
+ u32 lut;
+
+ mask = (mask << shift) & 6U;
+ lut = 0xe4U;
+ return (lut >> mask) & 1U;
+}
+
+struct map_pair {
+ u8 match;
+ u8 bit;
+};
+
+struct map_entry {
+ u8 base_mask;
+ u8 count;
+ struct map_pair pairs[8];
+};
+
+/**
+ * pmu_map_mask() - Remap bitmask through DCCM lookup table at 0x80000a7e
+ * @val: Input bitmask byte
+ * @index: Table entry index (0..N)
+ *
+ * Return: Remapped 8-bit mask combining base_mask and matched bit pairs.
+ */
+u8 pmu_map_mask(u8 val, u32 index)
+{
+ const struct map_entry *table =
+ (const struct map_entry *)PMU_DMEM_DQ_PIN_MAP_BASE;
+ const struct map_entry *entry = &table[index];
+ u8 result = entry->base_mask & val;
+ u32 i;
+
+ for (i = 0; i < entry->count; i++) {
+ if (val & entry->pairs[i].match)
+ result |= (1U << entry->pairs[i].bit);
+ }
+ return result;
+}
+
+/**
+ * pmu_math_freq_scale_div() - Divide signed value by scaled frequency factor
+ * @val: Signed dividend
+ *
+ * Divides @val by (dmem[0x80000008] << 7), rounding positive values toward
+ * +infinity and negative values toward 0.
+ *
+ * Return: Scaled signed quotient.
+ */
+s32 pmu_math_freq_scale_div(s32 val)
+{
+ u32 pll_ratio = dmem_read8(PMU_DMEM_PLL_RATIO);
+ u32 divisor;
+ u32 pos;
+
+ if (pll_ratio == 0U)
+ pll_ratio = 2U;
+ divisor = pll_ratio << 7;
+
+ if (val >= 1) {
+ u32 num = (u32)(val - 1);
+
+ return (s32)((num + divisor) / divisor);
+ }
+
+ pos = (u32)(-val);
+ return -(s32)(pos / divisor);
+}
+
+/**
+ * pmu_unpack_6() - Combine coarse and 6-bit fine delay pair into linear tap value
+ * @p: Two-byte array holding [0] coarse step and [1] 6-bit fine offset
+ *
+ * Return: Combined linear delay value ((p[0] << 6) + p[1]).
+ */
+u32 pmu_unpack_6(const u8 *p)
+{
+ return ((u32)p[0] << 6) + p[1];
+}
+
+/**
+ * pmu_unpack_7() - Combine coarse and 7-bit fine delay pair into linear tap value
+ * @p: Two-byte array holding [0] coarse step and [1] 7-bit fine offset
+ *
+ * Return: Combined linear delay value ((p[0] << 7) + p[1]).
+ */
+u32 pmu_unpack_7(const u8 *p)
+{
+ return ((u32)p[0] << 7) + p[1];
+}
+
+/**
+ * pmu_unpack_byte_pair() - Split packed delay value into coarse/fine byte pair
+ * @val: Packed delay value
+ * @out: Two-byte output buffer receiving [0] (val >> 6) and [1] (val & 0x3f)
+ */
+void pmu_unpack_byte_pair(u32 val, u8 *out)
+{
+ out[1] = (u8)(val & 0x3fU);
+ out[0] = (u8)(val >> 6);
+}
+
+/**
+ * pmu_cal_param_unpack() - Unpack fine and rounded coarse delay fields into byte pair
+ * @val: Packed 10-bit delay line parameter
+ * @out: Two-byte output array receiving [0] rounded coarse delay and [1] 6-bit fine delay
+ */
+void pmu_cal_param_unpack(u32 val, u8 *out)
+{
+ u32 c = val >> 6;
+ u32 f = val & 0x3fU;
+
+ out[1] = (u8)f;
+ out[0] = (u8)(((c >> 1) & 0x07U) + (c & 0x01U));
+}
+
+/**
+ * pmu_dly_line_repack() - Repack fine and shifted coarse delay fields
+ * @val: Input delay value
+ *
+ * Return: Recombined delay tap ((val & 0x3f) + (val & 0x40) + ((val >> 1) & 0x1c0)).
+ */
+u32 pmu_dly_line_repack(u32 val)
+{
+ return (val & 0x3fU) + (val & 0x40U) + ((val >> 1) & 0x1c0U);
+}
+
+/**
+ * pmu_midpoint_deviation_calc() - Compute signed LCDL tap deviation from nominal midpoint
+ * @val: Input LCDL tap value
+ *
+ * Return: (s8)(64 - val) when @val < 65, or (s8)((val << 2) + 48) when @val >= 65.
+ */
+s8 pmu_midpoint_deviation_calc(u32 val)
+{
+ u32 c = 64U;
+
+ if (val <= c)
+ return (s8)(c - val);
+
+ return (s8)((val << 2) + 48U);
+}
+
+/**
+ * pmu_cal_lane_mask_and_all() - Bitwise-AND byte across all 4 slice descriptors
+ * @base: Base DCCM address of the 4-slice (4 x 54-byte) descriptor table
+ * @offset: Byte offset within each 54-byte slice descriptor
+ * @mask: 8-bit mask applied via bitwise AND to each slice byte
+ */
+void pmu_cal_lane_mask_and_all(u32 base, u32 offset, u32 mask)
+{
+ __volatile__ u8 *p = (__volatile__ u8 *)(base + offset);
+ u32 i;
+
+ for (i = 0U; i < PMU_SLICE_DESC_TOTAL_BYTES; i += PMU_SLICE_DESC_SIZE)
+ p[i] &= (u8)mask;
+}
+
+/**
+ * pmu_cal_lane_mask_or_all() - Bitwise-OR byte across all 4 slice descriptors
+ * @base: Base DCCM address of the 4-slice (4 x 54-byte) descriptor table
+ * @offset: Byte offset within each 54-byte slice descriptor
+ * @mask: 8-bit mask applied via bitwise OR to each slice byte
+ */
+void pmu_cal_lane_mask_or_all(u32 base, u32 offset, u32 mask)
+{
+ __volatile__ u8 *p = (__volatile__ u8 *)(base + offset);
+ u32 i;
+
+ for (i = 0U; i < PMU_SLICE_DESC_TOTAL_BYTES; i += PMU_SLICE_DESC_SIZE)
+ p[i] |= (u8)mask;
+}
+
+/**
+ * pmu_cal_lcdl_coarse_fine_pair_pack() - Pack coarse/fine LCDL delay pairs into table
+ * @ch_idx: Channel index
+ * @threshold: Channel threshold value
+ * @limit: Channel limit value
+ * @pair_a: Primary coarse/fine pair A
+ * @pair_b: Primary coarse/fine pair B
+ * @pair_c: Secondary coarse/fine pair C
+ * @pair_d: Secondary coarse/fine pair D
+ * @table_base: Base pointer of destination LCDL delay table
+ */
+void pmu_cal_lcdl_coarse_fine_pair_pack(u32 ch_idx, u32 threshold, u32 limit,
+ const u8 *pair_a, const u8 *pair_b,
+ const u8 *pair_c, const u8 *pair_d,
+ void *table_base)
+{
+ u8 *entry = (u8 *)table_base + (ch_idx << 2);
+ bool use_sec = (u8)limit > (u8)threshold;
+ const u8 *pa = use_sec ? pair_c : pair_a;
+ const u8 *pb = use_sec ? pair_d : pair_b;
+ const u8 *src = entry + (use_sec ? 0x02U : 0x12U);
+ u16 h_a = *(const u16 *)src;
+ u16 h_b = *(const u16 *)(src + 8U);
+
+ *(u16 *)(entry + 0x22U) = h_a;
+ *(u16 *)(entry + 0x2aU) = h_b;
+ *(u16 *)(entry + 0x20U) = (u16)(((u16)pa[0] << 7) + pa[1]);
+ *(u16 *)(entry + 0x28U) = (u16)(((u16)pb[0] << 7) + pb[1]);
+}
+
+/**
+ * pmu_cal_eye_margin_quadratic_cost_calc() - Compute weighted quadratic eye-centering cost
+ * @val: Packed eye sample word (low byte: center offset, high byte: magnitude)
+ * @target: Signed 8-bit target center offset
+ * @min_bound: Minimum valid magnitude bound
+ * @max_bound: Maximum valid magnitude bound
+ * @out_overflow: Output pointer receiving 0x3f on violation or abs(delta) in-bounds
+ *
+ * Return: Weighted quadratic cost ((delta^2 * w1) + (overflow^2 * w2)).
+ */
+u32 pmu_cal_eye_margin_quadratic_cost_calc(u32 val, u32 target, u32 min_bound,
+ u32 max_bound, u8 *out_overflow)
+{
+ s32 delta = (s32)((s8)val) - (s32)((s8)target);
+ u32 mag = (u16)val >> 8;
+ u32 min_b = (u8)min_bound;
+ u32 max_b = (u8)max_bound;
+ s32 over;
+ u8 out = 0x3fU;
+
+ if (delta < 0)
+ delta = -delta;
+ over = (s32)(min_b - mag);
+ if (over <= 0) {
+ over = (s32)(mag - max_b);
+ if (over <= 0) {
+ over = 0;
+ out = (u8)delta;
+ }
+ }
+ *out_overflow = out;
+ return (u32)(u16)(delta * delta) * dmem_read16(PMU_DMEM_COST_WEIGHT_1) +
+ (u32)(u16)(over * over) * dmem_read16(PMU_DMEM_COST_WEIGHT_2);
+}
diff --git a/arch/arm/mach-sunxi/pmu/pmu_telemetry.c b/arch/arm/mach-sunxi/pmu/pmu_telemetry.c
new file mode 100644
index 00000000000..ea47e991fcf
--- /dev/null
+++ b/arch/arm/mach-sunxi/pmu/pmu_telemetry.c
@@ -0,0 +1,619 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Copyright (C) 2026 Alastair D'Silva <alastair@d-silva.org>
+ *
+ * Allwinner A733 Synopsys DDR PHY PMU Mailbox POST IPC & Assertion Handlers
+ */
+
+#include "pmu_fw.h"
+
+static void pmu_mailbox_post_exec_wait(void);
+
+/**
+ * pmu_mailbox_post_cmd_dispatch() - Dispatch 16-bit POST command to PHY mailbox
+ * @cmd: Command code to transmit (pulses 0x90040238 for terminal codes 0xff or 7)
+ */
+noinline void pmu_mailbox_post_cmd_dispatch(u32 cmd)
+{
+ if (cmd == 255U || cmd == 7U) {
+ phy_write16(PHY_REG_CMD_DISPATCH,
+ (cmd == 255U) ? 0x0004U : 0x0001U);
+ phy_write16(PHY_REG_CMD_DISPATCH, 0x0000U);
+ }
+
+ phy_write16(PHY_REG_MAILBOX_STREAM, (u16)cmd);
+ phy_write16(PHY_REG_MAILBOX_STREAM + 2U, 0x0000U);
+ pmu_mailbox_post_exec_wait();
+}
+
+/**
+ * pmu_mailbox_post_exec_wait() - Handshake PHY mailbox interrupt and busy status
+ *
+ * Waits for bit 0 of PHY register 0x90180008 to clear, triggers the mailbox
+ * interrupt at 0x90180066 with 1, and waits for bit 0 of 0x90180008 to assert.
+ */
+static noinline void pmu_mailbox_post_exec_wait(void)
+{
+ while ((phy_read16(PHY_REG_MAILBOX_STAT) & 1U) != 0U)
+ ;
+ phy_write16(PHY_REG_MAILBOX_TRIG, 0x0001U);
+ while ((phy_read16(PHY_REG_MAILBOX_STAT) & 1U) == 0U)
+ ;
+}
+
+/**
+ * pmu_assert_or_halt() - Validate condition or record panic code and halt PMU
+ * @cond: Condition value (non-zero continues execution; zero triggers fatal halt)
+ * @assert_code: 32-bit diagnostic code recorded at DCCM 0x8000000c on failure
+ */
+void pmu_assert_or_halt(u32 cond, u32 assert_code)
+{
+ if (cond != 0U)
+ return;
+
+ *(__volatile__ u32 *)PMU_DMEM_ASSERT_CODE = assert_code;
+ *(__volatile__ u8 *)PMU_DMEM_PSTATE_FLAGS |= 0x02U;
+ pmu_mailbox_post_cmd_dispatch(255U);
+ __builtin_arc_sr(0U, PMU_AUX_RTC_CTRL);
+ __builtin_arc_flag(1);
+ for (;;)
+ ;
+}
+
+/**
+ * pmu_ac_lane_profile_setup() - Program AC lane profile registers and gate settle delay
+ */
+void pmu_ac_lane_profile_setup(void)
+{
+ const u8 *ac_desc = (const u8 *)PMU_DMEM_AC_PROFILE_EN_ADDR;
+ u16 saved_gate_mask;
+ u16 saved_gate_ctrl;
+ u8 cal_status;
+ s32 delay_diff;
+
+ saved_gate_mask = phy_read16(PHY_REG_GATE_MASK);
+ saved_gate_ctrl = phy_read16(PHY_REG_GATE_CTRL);
+
+ phy_write16(PHY_REG_GATE_CTRL, 0U);
+ phy_write16(PHY_REG_GATE_MASK, 0x00ffU);
+
+ cal_status = *(const u8 *)PMU_DMEM_TIMING_LIM;
+ phy_write16(PHY_REG_VREF_STAT, (u16)cal_status | 0x0100U);
+
+ delay_diff = (s32)(*(const u8 *)PMU_DMEM_LANE_STATUS + cal_status) - 2;
+
+ if (ac_desc[0] != 0U) {
+ const u16 *ac_words = (const u16 *)(ac_desc + 4);
+ u32 csr_off = pmu_sdata()->active_csr_offset << 1;
+
+ phy_write16(PHY_REG_AC_DRIVE_CFG5 + csr_off, ac_words[0]);
+ phy_write16(PHY_REG_AC_DRIVE_CFG6 + csr_off, ac_words[1]);
+ phy_write16(PHY_REG_AC_DRIVE_CFG3 + csr_off, ac_words[2]);
+ phy_write16(PHY_REG_AC_DRIVE_CFG4 + csr_off, ac_words[3]);
+ phy_write16(PHY_REG_AC_DRIVE_CFG0 + csr_off, ac_desc[3]);
+ phy_write16(PHY_REG_AC_DRIVE_CFG1 + csr_off, ac_desc[1]);
+ phy_write16(PHY_REG_AC_DRIVE_CFG2 + csr_off, ac_desc[2]);
+ }
+
+ if (delay_diff < 0)
+ delay_diff += (s32)((u32)cal_status *
+ *(const u8 *)PMU_DMEM_PLL_RATIO);
+
+ phy_write16(PHY_REG_DRAM_TYPE_CFG, (u16)delay_diff);
+ pmu_delay_us(11000U, 0U);
+
+ phy_write16(PHY_REG_GATE_MASK, saved_gate_mask);
+ phy_write16(PHY_REG_GATE_CTRL, saved_gate_ctrl);
+
+ *(__volatile__ u8 *)PMU_DMEM_AC_SIGNED_STEP =
+ (u8)(((cal_status >> 1) & ~1U) - 1U);
+}
+
+/**
+ * pmu_cmd_code_map_log() - Map command/subcode to CBT lane limit code and log
+ * @cmd_code: Primary command code
+ * @sub_code: Sub-command selector (0 selects @cmd_code mapping)
+ */
+void pmu_cmd_code_map_log(u32 cmd_code, u32 sub_code)
+{
+ u8 map_code = 9U;
+
+ if (sub_code == 0U) {
+ u32 masked = cmd_code & 0x0fffU;
+
+ if (masked >= 257U && masked <= 261U)
+ map_code = (u8)(masked - 255U);
+ else if (masked == 272U)
+ map_code = 7U;
+ else if (masked == 512U)
+ map_code = 8U;
+ } else if (sub_code == 1U) {
+ map_code = 0U;
+ } else if (sub_code == 2U) {
+ map_code = 1U;
+ } else if (sub_code == 4U || sub_code == 17U) {
+ map_code = 2U;
+ } else if (sub_code == 18U) {
+ map_code = 3U;
+ } else if (sub_code == 8U || sub_code == 19U) {
+ map_code = 4U;
+ } else if (sub_code == 20U) {
+ map_code = 5U;
+ } else if (sub_code == 21U) {
+ map_code = 6U;
+ } else if (sub_code == 26U) {
+ map_code = 7U;
+ } else if (sub_code == 0x20U) {
+ map_code = 8U;
+ } else {
+ pmu_assert_or_halt(0U, PMU_ERR_CMD_SUBCODE_INVALID);
+ map_code = pmu_sdata()->freq_mode;
+ }
+
+ pmu_sdata()->freq_mode = map_code;
+}
+
+/**
+ * pmu_slice_metric_scan_and_program() - Scan slice metrics and program DBYTE delay CSRs
+ */
+void pmu_slice_metric_scan_and_program(void)
+{
+ u8 start_slice;
+ u8 end_slice;
+ u16 num_ranks;
+ u32 ch_offset;
+ u32 slice;
+
+ start_slice = dmem_read8(PMU_DMEM_SLICE_START);
+ end_slice = dmem_read8(PMU_DMEM_SLICE_END);
+ num_ranks = pmu_sdata()->num_ranks;
+ ch_offset = pmu_sdata()->active_csr_offset;
+
+ for (slice = start_slice; slice <= end_slice; slice++) {
+ const __volatile__ u16 *slice_metrics =
+ (const __volatile__ u16 *)(PMU_DMEM_CBT_DELAY_ARR +
+ (slice << 3));
+ u16 max_val = slice_metrics[0];
+ u16 min_val = max_val;
+ u32 k;
+ u32 midpoint;
+ u32 diff;
+ u16 reg_val_lo;
+ u16 reg_val_hi;
+ u32 slice_base;
+ u32 bank;
+
+ for (k = 0U; k < num_ranks; k++) {
+ u16 val = slice_metrics[k];
+
+ if (val > max_val)
+ max_val = val;
+ if (val < min_val)
+ min_val = val;
+ }
+
+ midpoint = (max_val + min_val) >> 1;
+ diff = 12U - midpoint;
+
+ reg_val_lo = (u16)((diff << 4) | midpoint);
+ reg_val_hi = (u16)(diff | (midpoint << 4));
+ slice_base = PHY_REG_DBYTE_BASE |
+ (((slice << 12) | ch_offset) << 1);
+
+ for (bank = 0U; bank < 2U; bank++) {
+ u32 reg_low = (bank != 0U) ? 0x5eU : 0x62U;
+ u32 reg_high = reg_low + 2U;
+ u16 w1 = (*(__volatile__ const u16 *)
+ (PHY_REG_DBYTE_BASE +
+ ((reg_low | ch_offset) << 1)) & 768U) |
+ reg_val_lo;
+ u16 w2 = (*(__volatile__ const u16 *)
+ (PHY_REG_DBYTE_BASE +
+ ((reg_high | ch_offset) << 1)) & 768U) |
+ reg_val_hi;
+ u32 a1 = slice_base + (reg_low << 1);
+ u32 a2 = slice_base + (reg_high << 1);
+
+ *(__volatile__ u16 *)a1 = w1;
+ *(__volatile__ u16 *)(a1 + 2U) = w1;
+ *(__volatile__ u16 *)a2 = w2;
+ *(__volatile__ u16 *)(a2 + 2U) = w2;
+ }
+ }
+}
+
+/**
+ * pmu_mr22_bit4_check() - Query bit 4 of training configuration byte 0x96
+ *
+ * Return: 1 if bit 4 of DCCM 0x80000096 is set, 0 otherwise.
+ */
+u32 pmu_mr22_bit4_check(void)
+{
+ return (*(__volatile__ u8 *)PMU_DMEM_MR22_ADDR >> 4) & 1;
+}
+
+/**
+ * pmu_stage_slice_flag_check() - Extract 2-bit mode field gated by DCCM 0x407 bit 6
+ *
+ * Return: Bits [4:3] of halfword at DCCM 0x80000408 if bit 6 of DCCM 0x80000407
+ * is set, or 0 otherwise.
+ */
+u32 pmu_stage_slice_flag_check(void)
+{
+ u32 flag = pmu_sdata()->stage_slice_idx;
+
+ if (!(flag & 0x40U))
+ return 0;
+
+ return (pmu_sdata()->stage_lane_idx >> 3) & 3U;
+}
+
+/**
+ * pmu_check_freq_lt_3200() - Check whether DRAM data rate is below 3200 MT/s
+ *
+ * Return: 1 if halfword at DCCM 0x80000006 is less than 3200, 0 otherwise.
+ */
+u32 pmu_check_freq_lt_3200(void)
+{
+ return dmem_read16(PMU_DMEM_DRAM_FREQ_OFF) < 3200U;
+}
+
+/**
+ * pmu_is_lpddr4() - Check if DRAM protocol configuration is set to LPDDR4
+ *
+ * Return: 1 if bits [1:0] of DCCM 0x8000008a equal 1, 0 otherwise.
+ */
+u32 pmu_is_lpddr4(void)
+{
+ return (dmem_read8(PMU_DMEM_MR19_OFF) & 3U) == 1U;
+}
+
+/**
+ * pmu_is_lpddr5() - Check if DRAM protocol mode bits [1:0] at 0x8e are zero
+ *
+ * Return: 1 if bits [1:0] of DCCM 0x8000008e equal 0, 0 otherwise.
+ */
+u32 pmu_is_lpddr5(void)
+{
+ return (dmem_read8(PMU_DMEM_MR20_ADDR) & 3U) == 0U;
+}
+
+/**
+ * pmu_check_stage() - Check whether stage index belongs to {0, 6, 14, 40}
+ * @val: Calibration stage index to test
+ *
+ * Return: 1 if @val is 0, 6, 14, or 40; 0 otherwise.
+ */
+u32 pmu_check_stage(u32 val)
+{
+ if (val <= 14)
+ return (0x4041U >> val) & 1;
+ return val == 40;
+}
+
+/**
+ * pmu_cal_marker_clear() - Clear 16-bit calibration state halfword at DCCM 0x80000458
+ */
+void pmu_cal_marker_clear(void)
+{
+ pmu_sdata()->cal_marker_a = 0;
+}
+
+/**
+ * pmu_cbt_cal_active_clear() - Clear byte flag at DCCM 0x8000045c to 0
+ */
+void pmu_cbt_cal_active_clear(void)
+{
+ pmu_sdata()->cbt_cal_active = 0;
+}
+
+/**
+ * pmu_cbt_cal_active_set() - Set byte flag at DCCM 0x8000045c to 1
+ */
+void pmu_cbt_cal_active_set(void)
+{
+ pmu_sdata()->cbt_cal_active = 1;
+}
+
+/**
+ * pmu_cbt_step_stat_set() - Set byte flag at DCCM 0x8000046b to 1
+ */
+void pmu_cbt_step_stat_set(void)
+{
+ pmu_sdata()->cbt_step_stat = 1;
+}
+
+/**
+ * pmu_cbt_config_get() - Read calibration status byte from DCCM 0x8000046a
+ *
+ * Return: 8-bit value stored at DCCM 0x8000046a.
+ */
+u8 pmu_cbt_config_get(void)
+{
+ return pmu_sdata()->cbt_config;
+}
+
+/**
+ * pmu_cal_mode_flag_get() - Read configuration byte from DCCM 0x80000005
+ *
+ * Return: 8-bit value stored at DCCM 0x80000005.
+ */
+u8 pmu_cal_mode_flag_get(void)
+{
+ return *(__volatile__ u8 *)PMU_DMEM_CAL_MODE_FLAG_ADDR;
+}
+
+/**
+ * pmu_lane_mask_bit_get() - Extract bit from DCCM 0x8000001c status byte
+ * @word_idx: 2-bit pair index (multiplied by 2)
+ * @bit_shift: Additional bit offset within the pair
+ *
+ * Return: Extracted bit ((dmem[0x1c] >> ((word_idx << 1) + bit_shift)) & 1).
+ */
+u32 pmu_lane_mask_bit_get(u32 word_idx, u32 bit_shift)
+{
+ u32 val = dmem_read8(PMU_DMEM_LANE_MASK);
+ u32 shift = (word_idx << 1) + bit_shift;
+
+ return (val >> shift) & 1U;
+}
+
+/**
+ * pmu_phy_query_operating_mode() - Query normalized PHY operating mode
+ *
+ * Return: Operating mode index (0..4) derived from PHY register 0x90020012.
+ *
+ * Reads the channel-adjusted PHY operating mode register at 0x90020012 and
+ * maps raw mode bits [2:0] into the normalized PMU operating mode code.
+ */
+u32 pmu_phy_query_operating_mode(void)
+{
+ u32 ch_offset = *(__volatile__ u32 *)PMU_DMEM_ACTIVE_CH_OFFSET;
+ u16 val = *(__volatile__ u16 *)(PHY_REG_OP_MODE | (ch_offset << 1));
+ u32 mode = val & 7;
+
+ if (mode <= 1)
+ return mode;
+
+ return ((val & 6) >> 1) + 1;
+}
+
+/**
+ * pmu_cal_marker_pair_set() - Write 16-bit value to DCCM 0x80000458 and 0x8000045e
+ * @val: Halfword value to store at both offsets
+ */
+void pmu_cal_marker_pair_set(u16 val)
+{
+ pmu_sdata()->cal_marker_a = val;
+ pmu_sdata()->cal_marker_b = val;
+}
+
+/**
+ * pmu_cbt_config_set() - Set DCCM 0x8000045a flag to 1 and store byte at 0x8000046a
+ * @val: Byte value to write to DCCM 0x8000046a
+ */
+void pmu_cbt_config_set(u8 val)
+{
+ pmu_sdata()->cbt_state = 1;
+ pmu_sdata()->cbt_config = val;
+}
+
+/**
+ * pmu_post_cmd_conditional_dispatch() - Conditionally dispatch post-command and sync clock gate
+ * @val: Command code to pass to pmu_mailbox_post_cmd_dispatch()
+ *
+ * Dispatches @val to pmu_mailbox_post_cmd_dispatch() when DCCM 0x80000012 is
+ * at most 0xc8, then pulses the PHY clock gate synchronizer.
+ */
+void pmu_post_cmd_conditional_dispatch(u32 val)
+{
+ u8 threshold = *(__volatile__ u8 *)PMU_DMEM_POST_THRESH;
+
+ if (threshold <= 0xc8)
+ pmu_mailbox_post_cmd_dispatch(val);
+
+ pmu_phy_clk_gate_sync_pulse();
+}
+
+/**
+ * pmu_cal_stride_get() - Read calibration step stride from DCCM 0x80000458
+ *
+ * Return: 16-bit halfword at DCCM 0x80000458 logically shifted right by 2 bits.
+ */
+u16 pmu_cal_stride_get(void)
+{
+ return pmu_sdata()->cal_marker_a >> 2;
+}
+
+/**
+ * pmu_cal_marker_verify() - Compare calibration markers A and B and dispatch fallback
+ *
+ * Return: 0 if halfwords at DCCM 0x80000458 and 0x8000045e match, or the result
+ * of pmu_cal_marker_status_update() when they differ.
+ */
+u16 pmu_cal_marker_verify(void)
+{
+ u32 marker_a;
+ u32 marker_b;
+ u32 diff;
+
+ marker_a = pmu_sdata()->cal_marker_a;
+ marker_b = pmu_sdata()->cal_marker_b;
+ diff = marker_a ^ marker_b;
+ if (diff == 0U)
+ return (u16)diff;
+
+ return pmu_cal_marker_status_update();
+}
+
+/**
+ * pmu_dram_cfg_bit13_check() - Query bit 13 of DRAM config flags at DCCM 0x0a
+ * @rank: Rank index (returns 0 if equal to 4)
+ * @flag: Secondary gating flag (returns 0 if non-zero)
+ *
+ * Return: Bit 13 of DCCM 0x8000000a when @rank != 4 and @flag == 0, else 0.
+ */
+u32 pmu_dram_cfg_bit13_check(u32 rank, u32 flag)
+{
+ u16 cfg;
+
+ if (rank == 4U || flag != 0U)
+ return 0U;
+
+ cfg = *(__volatile__ u16 *)(PMU_DMEM_BASE + PMU_DMEM_DRAM_CFG_FLAGS);
+ return (u32)((cfg >> 13) & 1U);
+}
+
+/**
+ * pmu_dmem_training_flags_eval() - Assemble 5-bit composite Vref/training mode byte
+ *
+ * Return: 5-bit composite flag byte combining DCCM 0x62, 0x72, and 0x96.
+ */
+u8 pmu_dmem_training_flags_eval(void)
+{
+ u32 mr3_val = dmem_read8(PMU_DMEM_MR3_ADDR);
+ u32 mr13_val = dmem_read8(PMU_DMEM_MR13_ADDR);
+ u32 mr22_val = dmem_read8(PMU_DMEM_MR22_ADDR);
+ u32 res = (mr3_val >> 7) | ((mr3_val >> 5) & 2U);
+
+ if (mr22_val & 0x40U)
+ res |= 4U;
+ if (mr22_val & 0x10U)
+ res |= 8U;
+ if (!(mr13_val & 0x20U))
+ res |= 16U;
+
+ return (u8)res;
+}
+
+/**
+ * pmu_cal_marker_status_update() - Recompute calibration marker A and clear tracker
+ *
+ * Return: Updated 16-bit calibration marker written to DCCM 0x80000458.
+ *
+ * Sets DCCM 0x80000dc4 to 1 when 0x80000dbe is non-zero, invokes
+ * (u16)pmu_cal_tracker_csr_commit(0x80000d9c) to compute the updated marker stored at
+ * DCCM 0x80000458, zeroes the 32-byte tracker block at 0x80000d9c, and clears
+ * the four trailing status halfwords at 0x80000dbe..0x80000dc4.
+ */
+u16 pmu_cal_marker_status_update(void)
+{
+ __volatile__ struct pmu_cal_tracker *trk = pmu_tracker();
+ u16 zero = 0U;
+ u16 marker;
+
+ if (trk->repeat_cnt != 0U)
+ trk->commit_flag = 1U;
+
+ marker = (u16)pmu_cal_tracker_csr_commit(trk);
+ pmu_sdata()->cal_marker_a = marker;
+ memset((void *)trk->slot0, 0, PMU_TRACKER_SLOTS_BYTES);
+ trk->commit_flag = zero;
+ trk->ext_flag = zero;
+ trk->slot0_mode = zero;
+ trk->repeat_cnt = zero;
+ return marker;
+}
+
+/**
+ * pmu_cal_pad_voltage_dac_set() - Evaluate Vref DAC polarity and override state
+ * @val: Input step/direction indicator (tested for non-zero byte value)
+ * @mode: Vref domain selector ((@mode & ~1) == 2 selects bit 7 of DCCM 0x62, else bit 6)
+ *
+ * Return: 1 or 0 from DCCM 0x800000e7 override bits [1:0] when active, or
+ * ((u8)@val != 0) masked by the selected polarity bit of DCCM 0x80000062.
+ */
+u32 pmu_cal_pad_voltage_dac_set(u32 val, u32 mode)
+{
+ u32 ctrl = dmem_read8(PMU_DMEM_VREF_OVR_CTRL_ADDR);
+ u32 dir;
+ u32 bit;
+
+ if (ctrl & 3U)
+ return ctrl & 1U;
+
+ dir = dmem_read8(PMU_DMEM_MR3_ADDR);
+ bit = ((mode & ~1U) == 2U) ? (dir >> 7) : ((dir >> 6) & 1U);
+ return ((u8)val != 0U) & bit;
+}
+
+/**
+ * pmu_cal_channel_rank_config_get() - Interrogate channel and rank configuration
+ * @rank: Logical rank/slice index (0..1 for channel 0, 2..3 for channel 1)
+ * @out_cfg: Output pointer receiving the 16-bit rank polarity bit (0 or 1)
+ * @out_mode_sel: Output pointer receiving the 1-hot rank bitmask (1 << bit)
+ * @out_num_slices: Output pointer receiving the active channel mode (1 or 2)
+ *
+ * Return: 0 if the requested rank is enabled in the channel mask (updating
+ * *@out_num_slices), or 1 if disabled.
+ */
+u32 pmu_cal_channel_rank_config_get(u32 rank, u16 *out_cfg, u8 *out_mode_sel,
+ u8 *out_num_slices)
+{
+ u32 idx = (u16)rank;
+ u32 is_ch1 = (idx >= 2U);
+ u32 base = is_ch1 ? PMU_DMEM_CH1_ACTIVE_FLAG :
+ PMU_DMEM_CH0_ANALOG_EN;
+ const __volatile__ u8 *cfg_base =
+ (const __volatile__ u8 *)dmem_ptr(base);
+ u32 bit = (cfg_base[13] ^ idx) & 1U;
+ u8 mask = (u8)(1U << bit);
+
+ *out_cfg = (u16)bit;
+ *out_mode_sel = mask;
+ if ((cfg_base[0] & mask) == 0U)
+ return 1U;
+
+ *out_num_slices = (u8)(is_ch1 + 1U);
+ return 0U;
+}
+
+/**
+ * pmu_cal_channel_rank_bounds_verify() - Dispatch fatal assertion for status 1..7
+ * @err_id: Error status code (1..7)
+ */
+void pmu_cal_channel_rank_bounds_verify(u32 err_id)
+{
+ u32 idx = (err_id & 0xffU) - 1U;
+
+ if (idx < 7U)
+ pmu_assert_or_halt(0U, PMU_ERR_RANK_BOUNDS_BASE + (idx << 16));
+}
+
+/**
+ * pmu_dbyte_dq_deskew_regs_save_and_ramp() - Extract and clamp gate eye margin descriptors
+ * @out_a: Output pointer for primary margin descriptor byte
+ * @out_b: Output pointer for secondary margin descriptor byte
+ * @out_c: Output pointer for tertiary margin descriptor byte
+ * @flag: Mode/rank flag selector
+ * @extra: Secondary offset selector
+ */
+void pmu_dbyte_dq_deskew_regs_save_and_ramp(u8 *out_a, u8 *out_b, u8 *out_c,
+ u32 flag, u32 extra)
+{
+ u32 f8 = (u8)flag;
+ u32 e8 = (u8)extra;
+ u32 offset = (f8 == 4U) ? 4U :
+ (((e8 == 0U) ? 2U : 0U) + (((f8 >> 1) == 1U) ? 1U : 0U));
+ const u8 *desc = (const u8 *)dmem_ptr(PMU_DMEM_GATE_DESC + offset);
+ u8 c = desc[10];
+
+ if (c == 0U)
+ return;
+
+ *out_a = desc[0];
+ *out_b = desc[5];
+ *out_c = c;
+
+ if (f8 != 4U && e8 == 0U) {
+ if ((dmem_read16(PMU_DMEM_DRAM_CFG_FLAGS) & 0x2000U) != 0U) {
+ u32 sum = (u32)(*out_b) + (u32)(*out_a);
+ u32 lim = 127U;
+
+ if (sum > lim)
+ sum = lim;
+ *out_b = (u8)sum;
+ *out_a = 0U;
+ }
+ }
+}
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 16/26] sunxi: dram: a733: Add ARC PMU deskew and BIST training firmware
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (14 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 15/26] sunxi: dram: a733: Add ARC PMU primitives, clock, telemetry, and CBT Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 17/26] sunxi: dram: a733: Add ARC PMU 1D/2D eye " Alastair D'Silva
` (11 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
Add the DBYTE per-lane LCDL deskew calibration, read-gate leveling, WCK
duty-cycle calibration, and BIST pattern routines for the Allwinner A733
DDR PHY ARC EM4 firmware.
Written from clean-room hardware register and JEDEC JESD209-5 protocol
specifications (see doc/board/allwinner/a733-pmu.rst).
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
arch/arm/mach-sunxi/pmu/pmu_deskew.c | 4895 ++++++++++++++++++++++++++
1 file changed, 4895 insertions(+)
create mode 100644 arch/arm/mach-sunxi/pmu/pmu_deskew.c
diff --git a/arch/arm/mach-sunxi/pmu/pmu_deskew.c b/arch/arm/mach-sunxi/pmu/pmu_deskew.c
new file mode 100644
index 00000000000..e624c30dd80
--- /dev/null
+++ b/arch/arm/mach-sunxi/pmu/pmu_deskew.c
@@ -0,0 +1,4895 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Copyright (C) 2026 Alastair D'Silva <alastair@d-silva.org>
+ *
+ * Allwinner A733 Synopsys DDR PHY PMU Slice Deskew & Lane Status Helpers
+ */
+
+#include "pmu_fw.h"
+
+static void pmu_cal_channel_rank_boundary_eval(u8 *buf, u32 channel, u32 rank, u32 cal_type);
+static void pmu_cal_dbyte_deskew_pin_results_apply(void);
+static void pmu_cal_dbyte_deskew_results_apply(void *slice_base, u32 param1, u32 opcode);
+static u32 pmu_cal_dbyte_dq_status_check(u32 channel_idx, u32 mode_rank, u32 flag);
+static void pmu_cal_dly_line_tap_step_adjust(u32 mode, u32 ptr_tag, u32 lane_mask, u32 check_tag);
+static u32 pmu_cal_lane_deskew_step_advance(u32 lane, u32 phase);
+static void pmu_cal_multi_rank_deskew_sweep(u32 channel, u32 mode);
+static u32 pmu_cal_slice_step_scan_eval(u32 step_val, u32 slice_mask);
+static u8 pmu_dbyte_deskew_sample_check(u32 channel, u32 mode_rank, u32 marker0, u32 marker1);
+static void pmu_dbyte_slice_cal_param_pulse(u32 rank, u32 channel, u32 slice_mask);
+static int pmu_odd_byte_win_span_check(u8 *out_min, u8 *out_max, const u8 *scan_buf);
+static void pmu_rank_slice_deskew_latch(u32 step_delay, u32 lane_mask);
+static void pmu_slice_deskew_pulse_train(u32 ch_idx, u32 rank, u32 mode);
+static void pmu_slice_deskew_state_latch(u32 step_delay, u32 lane_mask);
+static void pmu_slice_dly_tap_coeff_prog(u32 rank);
+static void pmu_slice_phy_reg_step_adjust(u32 reg_offset, u32 polarity_cond, u32 step_mode);
+
+/**
+ * pmu_slice_lane_status_save() - Latch active slice status into DCCM 0x80000b97
+ *
+ * Reads the active channel CSR offset from DCCM 0x8000041c, samples the 16-bit
+ * PHY slice status register at 0x90040018 + (offset << 1), and stores the low
+ * byte to DCCM 0x80000b97.
+ */
+void pmu_slice_lane_status_save(void)
+{
+ u32 off = pmu_sdata()->active_csr_offset;
+ u16 val = *(__volatile__ u16 *)(PHY_REG_SLICE_STAT + (off << 1));
+
+ *(__volatile__ u8 *)PMU_DMEM_LANE_STATUS = (u8)val;
+}
+
+/**
+ * pmu_cal_lane_error_status_check() - Evaluate per-lane deskew error mask bit
+ * @lane: Logical lane/rank index (0..3)
+ * @phase: Phase polarity selector
+ *
+ * Return: 1-bit error status from DCCM 0x80000101 and direction 0x80000b6b.
+ */
+noinline u32 pmu_cal_lane_error_status_check(u32 lane, u32 phase)
+{
+ u32 dir = dmem_read8(PMU_DMEM_DESKEW_DIR);
+ u32 mask = dmem_read8(PMU_DMEM_LANE_ERR_MASK);
+ u32 bit = lane + (((u8)phase > dir) ? 2U : 0U);
+
+ return (((mask >> bit) ^ phase) ^ 1U) & 1U;
+}
+
+/**
+ * pmu_cal_lane_deskew_step_advance() - Check if lane deskew step should advance
+ * @lane: Logical lane/rank index (0..3)
+ * @phase: Current step phase threshold
+ *
+ * Return: 1 if lane error status or DCCM 0x1c limit comparison requires step
+ * advance, 0 otherwise.
+ */
+static noinline u32 pmu_cal_lane_deskew_step_advance(u32 lane, u32 phase)
+{
+ u32 err = pmu_cal_lane_error_status_check(lane, 1U);
+
+ if (err != 0U)
+ return err;
+ err = pmu_cal_lane_error_status_check(lane, phase);
+ if (err != 0U)
+ return err;
+
+ return pmu_lane_mask_bit_get(*(const u8 *)PMU_DMEM_DESKEW_DIR, lane) <
+ (phase & 0xffU);
+}
+
+/**
+ * pmu_phy_dbyte_cfg_unpack() - Restore DBYTE delay registers from DCCM 0x80000ba0
+ *
+ * Unpacks 8 bytes per active DBYTE slice (count in DCCM 0x8000042c) from the
+ * backup array at DCCM 0x80000ba0 into PHY registers 0x90020100 + (slice << 13).
+ */
+noinline void pmu_phy_dbyte_cfg_unpack(void)
+{
+ u32 count = pmu_sdata()->dbyte_slice_cnt;
+ u32 s;
+ u32 i;
+
+ for (s = 0U; s < count; s++) {
+ __volatile__ u16 *reg =
+ (__volatile__ u16 *)(PHY_REG_DBYTE_DQ_SWIZZLE_BASE + (s << 13));
+ const __volatile__ u8 *src =
+ (const __volatile__ u8 *)(PMU_DMEM_DBYTE_BACKUP + (s << 3));
+
+ for (i = 0U; i < 8U; i++)
+ reg[i] = src[i];
+ }
+}
+
+/**
+ * pmu_cal_pad_impedance_lock() - Save DBYTE DXDMAP registers and set linear ramp
+ *
+ * Reads 8 halfword registers per active DBYTE slice (count in DCCM 0x8000042c)
+ * from PHY 0x90020100 + (slice << 13), packs their low bytes into DCCM
+ * 0x80000ba0, and initializes each register to its linear pin index 0..7.
+ */
+noinline void pmu_cal_pad_impedance_lock(void)
+{
+ u32 count = pmu_sdata()->dbyte_slice_cnt;
+ u32 s;
+ u32 i;
+
+ for (s = 0U; s < count; s++) {
+ __volatile__ u16 *reg =
+ (__volatile__ u16 *)(PHY_REG_DBYTE_DQ_SWIZZLE_BASE + (s << 13));
+ __volatile__ u8 *dst =
+ (__volatile__ u8 *)(PMU_DMEM_DBYTE_BACKUP + (s << 3));
+
+ for (i = 0U; i < 8U; i++) {
+ dst[i] = (u8)reg[i];
+ reg[i] = (u16)i;
+ }
+ }
+}
+
+/**
+ * pmu_cal_rx_dqs_delay_center() - Populate per-channel DQ pin map matrix at 0x80000a7e
+ *
+ * Samples 8 lane delay registers per active DBYTE slice from
+ * 0x90020100 + (ch << 13) + (lane << 1) and records matched lanes in entry[0]
+ * and mismatched (1 << tap, lane) pairs in entry[2..17].
+ */
+void pmu_cal_rx_dqs_delay_center(void)
+{
+ u32 max_ch = pmu_sdata()->dbyte_slice_cnt;
+ u32 ch;
+ u32 lane;
+
+ for (ch = 0U; ch < max_ch; ch++) {
+ __volatile__ u8 *entry =
+ (__volatile__ u8 *)(PMU_DMEM_DQ_PIN_MAP_BASE + (ch * 18U));
+ const __volatile__ u16 *reg =
+ (const __volatile__ u16 *)(PHY_REG_DBYTE_DQ_SWIZZLE_BASE + (ch << 13));
+
+ *(__volatile__ u16 *)entry = 0U;
+ for (lane = 0U; lane < 8U; lane++) {
+ u32 tap = (u8)reg[lane];
+
+ if (tap == lane) {
+ entry[0] |= (u8)(1U << lane);
+ } else {
+ u32 count = (u32)entry[1] + 1U;
+
+ entry[1] = (u8)count;
+ entry[count * 2U] = (u8)(1U << tap);
+ entry[count * 2U + 1U] = (u8)lane;
+ }
+ }
+ }
+}
+
+/**
+ * pmu_cal_deskew_quad_tap_set() - Write 4 deskew delay tap pairs and latch channel mask
+ * @ch: Channel/slice index (0x0f for broadcast)
+ * @tap0: First deskew tap value
+ * @tap1: Second deskew tap value
+ * @tap2: Third deskew tap value
+ * @tap3: Fourth deskew tap value
+ */
+noinline void pmu_cal_deskew_quad_tap_set(u32 ch, u16 tap0, u16 tap1,
+ u16 tap2, u16 tap3)
+{
+ __volatile__ u16 *reg =
+ (__volatile__ u16 *)(PHY_REG_DESKEW_QUAD_TAP | ((ch & 0xffU) << 9));
+
+ reg[0] = tap0;
+ reg[1] = tap1;
+ reg[2] = tap0;
+ reg[3] = tap1;
+ reg[4] = tap2;
+ reg[5] = tap3;
+ reg[6] = tap2;
+ reg[7] = tap3;
+
+ if ((ch & 0xffU) == 0x0fU)
+ *(__volatile__ u16 *)PHY_REG_AC_TX_EN_BCAST = 0x01ffU;
+ else
+ *(__volatile__ u16 *)PHY_REG_AC_TX_EN_BCAST |=
+ (u16)(1U << (ch & 0xffU));
+}
+
+/**
+ * pmu_phy_reg_buffer_stream() - Stream 6-byte [addr, val] tuples between buffer and PHY APB
+ * @buf: Pointer to array of 6-byte [u32 addr, u16 val] tuples
+ * @count: Number of 6-byte tuples to transfer
+ * @is_save: Non-zero to save PHY APB registers into @buf, 0 to restore @buf to PHY APB
+ */
+void pmu_phy_reg_buffer_stream(void *buf, u32 count, u32 is_save)
+{
+ u8 *ptr = (u8 *)buf;
+ u32 i;
+
+ if (is_save != 0U) {
+ for (i = 0U; i < count; i++) {
+ u32 addr = (u32)ptr[0] | ((u32)ptr[1] << 8) |
+ ((u32)ptr[2] << 16) | ((u32)ptr[3] << 24);
+ u16 val = *(__volatile__ const u16 *)(PHY_REG_PUB_BASE |
+ (addr << 1));
+
+ ptr[4] = (u8)val;
+ ptr[5] = (u8)(val >> 8);
+ ptr += 6U;
+ }
+ } else {
+ for (i = 0U; i < count; i++) {
+ u32 addr = (u32)ptr[0] | ((u32)ptr[1] << 8) |
+ ((u32)ptr[2] << 16) | ((u32)ptr[3] << 24);
+ u16 val = (u16)((u16)ptr[4] | ((u16)ptr[5] << 8));
+
+ *(__volatile__ u16 *)(PHY_REG_STREAM_ALT_BASE |
+ (addr << 1)) = val;
+ ptr += 6U;
+ }
+ }
+}
+
+/**
+ * pmu_sample_margin_center_eval() - Evaluate gate eye bounds and midpoint sample pairs
+ * @samples: Input 128-byte scan sample buffer
+ * @out_margins: Output 6-byte midpoint and sample-average array
+ * @out_mode: Output mode selector derived from DCCM 0x8000000a bits [7:6]
+ *
+ * Return: 0 on success, or non-zero error status from pmu_odd_byte_win_span_check().
+ */
+int pmu_sample_margin_center_eval(const u8 *samples, u8 *out_margins,
+ u8 *out_mode)
+{
+ u16 cfg = *(__volatile__ const u16 *)(PMU_DMEM_BASE +
+ PMU_DMEM_DRAM_CFG_FLAGS);
+ u32 mode_field = (cfg >> 6) & 3U;
+ u8 start, end, center;
+ u32 s, idx;
+ int ret;
+
+ *out_mode = (mode_field == 1U) ? 1U : ((mode_field == 3U) ? 3U : 2U);
+
+ ret = pmu_odd_byte_win_span_check(&end, &start, samples);
+ if (ret != 0)
+ return ret;
+
+ s = (u32)end + (u32)start;
+ center = (u8)(s >> 1);
+ out_margins[0] = center;
+ idx = s & ~1U;
+ out_margins[1] = (u8)(((u32)samples[idx] + (u32)samples[idx + 1U]) >> 1);
+
+ s = (u32)end + (u32)center;
+ out_margins[2] = (u8)(s >> 1);
+ idx = s & ~1U;
+ out_margins[3] = (u8)(((u32)samples[idx] + (u32)samples[idx + 1U]) >> 1);
+
+ s = (u32)start + (u32)center;
+ out_margins[4] = (u8)(s >> 1);
+ idx = s & ~1U;
+ out_margins[5] = (u8)(((u32)samples[idx] + (u32)samples[idx + 1U]) >> 1);
+
+ return 0;
+}
+
+/**
+ * pmu_odd_byte_win_span_check() - Find odd-byte non-zero window and verify >= 12 span
+ * @out_min: Output pointer receiving first active index (0..64)
+ * @out_max: Output pointer receiving last active index
+ * @scan_buf: Input 128-byte scan buffer (odd indices inspected)
+ *
+ * Return: 0 if (*out_max - *out_min) >= 12, or 6 on narrow window error.
+ */
+static int pmu_odd_byte_win_span_check(u8 *out_min, u8 *out_max, const u8 *scan_buf)
+{
+ u32 min_idx = 0U;
+ s32 max_idx = 63;
+ s32 span;
+
+ while (min_idx < 64U) {
+ if (scan_buf[min_idx * 2U + 1U] != 0U)
+ break;
+ min_idx++;
+ }
+ *out_min = (u8)min_idx;
+
+ while (max_idx >= 0) {
+ if (scan_buf[(u32)max_idx * 2U + 1U] != 0U)
+ break;
+ max_idx--;
+ }
+ *out_max = (u8)max_idx;
+
+ span = (s32)(*out_max) - (s32)(*out_min);
+ if (span >= 12)
+ return 0;
+
+ return 6;
+}
+
+/**
+ * pmu_slice_lcdl_delay_collect() - Collect PHY slice status halfwords or mapped lane error status
+ * @param: Register offset / control word (bit 24 selects lane error status mode)
+ * @out_buf: Output halfword buffer
+ * @count: Number of halfwords per slice when bit 24 of @param is clear
+ */
+void pmu_slice_lcdl_delay_collect(u32 param, void *out_buf, u32 count)
+{
+ const u8 *ch_range = (const u8 *)PMU_DMEM_CH_RANGE;
+ u32 start_slice;
+ u32 end_slice;
+ u16 *out;
+ u32 slice;
+
+ start_slice = ch_range[0];
+ end_slice = ch_range[1];
+ out = (u16 *)out_buf + count * start_slice;
+
+ for (slice = start_slice; slice <= end_slice; slice++) {
+ if ((param & (1U << 24)) == 0U) {
+ u32 i;
+
+ for (i = 0U; i < count; i++) {
+ u32 reg = PHY_REG_DBYTE_BASE |
+ ((((slice << 12) | param |
+ (i << 8)) << 1));
+
+ *out++ = *(__volatile__ const u16 *)reg;
+ }
+ } else {
+ u32 masked_param = (param & ~pmu_sdata()->active_csr_offset) &
+ 0x0fffffffeUL;
+ u32 status_code;
+ u32 is_lo;
+ u32 tbl_off;
+
+ masked_param += 0xfefffff0UL;
+ masked_param = (masked_param >> 4) | (masked_param << 28);
+ if (masked_param == 1U) {
+ u8 err = pmu_cal_lane_error_status_check((u8)param,
+ (u8)slice);
+
+ status_code = err ? 1U : 2U;
+ } else {
+ status_code = (masked_param > 3U) ? 4U : masked_param;
+ }
+
+ is_lo = (ch_range[3] < slice) ? 1U : 0U;
+ tbl_off = (is_lo << 1) + status_code * 4U +
+ (param & 1U);
+ *out++ = ch_range[6U + tbl_off];
+ }
+ }
+}
+
+/**
+ * pmu_slice_deskew_pulse_seq() - Average and program slice deskew delay taps across ranks
+ * @enable: Execution enable flag (returns immediately if != 1)
+ */
+void pmu_slice_deskew_pulse_seq(u32 enable)
+{
+ u32 rank;
+
+ if (enable != 1U)
+ return;
+
+ for (rank = 1U; rank < 3U; rank++) {
+ u8 cfg = (rank == 2U) ?
+ *(__volatile__ const u8 *)PMU_DMEM_CH1_ACTIVE_FLAG :
+ *(__volatile__ const u8 *)PMU_DMEM_CH0_ANALOG_EN;
+ const u8 *ch_range = (const u8 *)PMU_DMEM_CH_RANGE;
+ u8 start_slice, end_slice;
+ u16 delay_buf[8];
+ u32 lane;
+
+ if (cfg != 3U)
+ continue;
+
+ pmu_phy_mode_cfg_dispatch(rank);
+ pmu_phy_soft_reset_pulse();
+
+ start_slice = ch_range[0];
+ end_slice = ch_range[1];
+
+ for (lane = 0U; lane < 2U; lane++) {
+ u32 offset = ((0xe4U >> (lane * 2U)) & 3U) + 0x2aU;
+ u32 slice;
+
+ for (slice = start_slice; slice <= end_slice; slice++) {
+ u32 slice_offset = slice << 12;
+ u32 csr_offset = pmu_sdata()->active_csr_offset;
+ u32 reg = offset | slice_offset | csr_offset;
+ u16 val = *(__volatile__ const u16 *)
+ (PHY_REG_DBYTE_BASE + (reg << 1));
+
+ delay_buf[lane * 4U + slice] = val;
+
+ if (lane == 1U) {
+ u16 v0 = delay_buf[slice];
+ u32 sum = (u32)v0 + val;
+ u32 q = sum >> 7;
+ u32 rem = (sum >> 1) & 0x3fU;
+ u16 prog_val = (u16)(rem | (q << 6));
+
+ *(__volatile__ u16 *)
+ (PHY_REG_DBYTE_ANALOG +
+ ((slice_offset |
+ csr_offset) << 1)) =
+ prog_val;
+ *(__volatile__ u16 *)
+ (PHY_REG_DBYTE_BASE +
+ (reg << 1)) = prog_val;
+ }
+ }
+ }
+ }
+
+ pmu_phy_mode_cfg_dispatch(3U);
+}
+
+/**
+ * pmu_cal_delay_line_init() - Compute clamped CBT delay line initial offset
+ */
+void pmu_cal_delay_line_init(void)
+{
+ u32 sum = (u32)pmu_freq_scaled_timing_step_calc() +
+ (u32)pmu_cbt_delay_offset_lookup(0U, 0U) +
+ (u32)pmu_cal_cbt_desc_unpack(0U, 0U) + 1U;
+
+ sum = (u8)sum;
+ sum &= ~1U;
+ if (sum > 92U)
+ sum = 92U;
+
+ pmu_sdata()->cbt_dly_init = (u8)sum;
+}
+
+/**
+ * pmu_slice_phy_reg_step_adjust() - Adjust DBYTE slice delay register step across range
+ * @reg_offset: Relative register offset added to 0x2a
+ * @polarity_cond: Non-zero to add +0x20 when @step_mode == 0, 0 to subtract 0x20
+ * @step_mode: Non-zero to modulo-wrap by (dmem[0x08] << 6), 0 for +/- 0x20 step
+ */
+static void pmu_slice_phy_reg_step_adjust(u32 reg_offset, u32 polarity_cond,
+ u32 step_mode)
+{
+ const u8 *ch_range = (const u8 *)PMU_DMEM_CH_RANGE;
+ u32 slice, end_slice;
+ u32 base = reg_offset + 0x2aU;
+
+ slice = ch_range[0];
+ end_slice = ch_range[1];
+
+ for (; slice <= end_slice; slice++) {
+ u32 reg_addr = PHY_REG_DBYTE_BASE |
+ (((pmu_sdata()->active_csr_offset |
+ (slice << 12) | base)) << 1);
+ u16 val = *(__volatile__ const u16 *)reg_addr;
+ u32 adjusted_val;
+
+ if (step_mode != 0U) {
+ u32 denom = (u32)(*(__volatile__ const u8 *)
+ PMU_DMEM_PLL_RATIO) << 6;
+
+ adjusted_val = (denom != 0U) ? ((u32)val % denom) : 0U;
+ adjusted_val &= 0x07ffU;
+ } else {
+ if (polarity_cond != 0U)
+ adjusted_val = (u32)val + 0x20U;
+ else
+ adjusted_val = (val < 0x20U) ?
+ 0U : ((u32)val - 0x20U);
+ adjusted_val &= 0x07ffU;
+ }
+
+ *(__volatile__ u16 *)reg_addr = (u16)adjusted_val;
+ }
+
+ pmu_phy_soft_reset_pulse();
+}
+
+/**
+ * pmu_rate_tier_calc_inline() - Encode DDR clock frequency into high/low rate tier nibble pair
+ * @val: Scaled frequency metric value
+ *
+ * Return: Packed 8-bit tier code ((high << 4) | low).
+ */
+static __always_inline u32 pmu_rate_tier_calc_inline(u32 val)
+{
+ u32 low = (val <= 331U) ? 0U :
+ (val <= 531U) ? 1U :
+ (val <= 799U) ? 2U : 3U;
+ u32 high = (val <= 799U) ? (low + 2U) :
+ (val <= 1799U) ? 5U :
+ (val <= 2199U) ? 6U : 7U;
+
+ return (high << 4) | low;
+}
+
+/**
+ * pmu_pll_lock_freq_tier_cfg() - Check PLL lock and program frequency rate tiers
+ */
+void pmu_pll_lock_freq_tier_cfg(void)
+{
+ u32 ch_offset = pmu_sdata()->active_csr_offset << 1;
+ u16 lock_val_hi, lock_val_lo;
+ __volatile__ u16 *phy_ch;
+ u32 pll_ratio;
+ u32 freq;
+ u32 q;
+ u32 clock_ratio;
+ u32 rate_calc;
+ u32 tier_clock;
+ u32 tier_rate;
+ u32 slice_count;
+ u32 s;
+
+ pmu_cal_pll_lock_settle_spin(&lock_val_hi, &lock_val_lo);
+
+ phy_ch = (__volatile__ u16 *)(PHY_REG_MASTER_BASE | ch_offset);
+ phy_ch[0x14 >> 1] = lock_val_lo & 0x01ffU;
+ phy_ch[0x2e >> 1] = lock_val_hi & 0x01ffU;
+
+ pll_ratio = *(const __volatile__ u8 *)PMU_DMEM_PLL_RATIO;
+ if (pll_ratio == 0U)
+ pll_ratio = 2U;
+ freq = *(const __volatile__ u16 *)PMU_DMEM_DRAM_FREQ_MHZ;
+
+ q = freq / pll_ratio;
+ clock_ratio = (pll_ratio == 2U) ? (q << 1) : q;
+ rate_calc = (pll_ratio != 2U) ? (q << 2) : (q << 1);
+
+ tier_clock = pmu_rate_tier_calc_inline(clock_ratio);
+ tier_rate = pmu_rate_tier_calc_inline(rate_calc);
+
+ phy_ch[0x24 >> 1] = (u16)(((tier_rate >> 4) << 12) |
+ ((tier_rate & 0x0fU) << 8) | tier_clock);
+ phy_ch[0x30c >> 1] = (u16)tier_rate;
+ phy_ch[0x30e >> 1] = *(const __volatile__ u16 *)PHY_REG_DBYTE_PERIOD;
+
+ slice_count = pmu_sdata()->active_slice_cnt;
+ for (s = 0U; s < slice_count; s++)
+ *(__volatile__ u16 *)(PHY_REG_DBYTE_PLL_TIER +
+ (s * PHY_DBYTE_SLICE_STRIDE)) = 0x009cU;
+}
+
+/**
+ * pmu_dbyte_deskew_sample_check() - Check DBYTE deskew sample status across slices
+ * @channel: Target channel index (0 or 1)
+ * @mode_rank: Rank and mode selector
+ * @marker0: First 16-bit calibration marker
+ * @marker1: Second 16-bit calibration marker
+ *
+ * Return: 1 if any slice sample mask fails expected low/high checks, else 0.
+ */
+static u8 pmu_dbyte_deskew_sample_check(u32 channel, u32 mode_rank,
+ u32 marker0, u32 marker1)
+{
+ const __volatile__ u8 *slice_cnt_tbl =
+ (const __volatile__ u8 *)PMU_DMEM_SLICE_CNT_LIMIT;
+ u32 rank = ((mode_rank & 0x3U) == 0x2U) ? 1U : 0U;
+ const u8 *byte_map;
+ u8 num_slices;
+ u8 res = 0U;
+ u32 i;
+
+ byte_map = (const u8 *)(slice_cnt_tbl + 0x24U + (channel << 4) + (rank << 3));
+ num_slices = slice_cnt_tbl[(channel << 1) + rank];
+
+ pmu_cal_marker_win_step_prog(marker0, marker1, mode_rank, 0U);
+ pmu_delay_us(10000U, 0U);
+
+ for (i = 0U; i < num_slices; i++) {
+ u8 slice = byte_map[i];
+ u16 phy_val = *(__volatile__ const u16 *)
+ (PHY_REG_DBYTE_GATE_STAT + ((u32)slice << 13));
+ u8 mask = pmu_map_mask(phy_val, slice);
+
+ if ((mask & 0x7fU) != 0U)
+ res |= 1U;
+ }
+
+ pmu_cal_marker_win_step_prog(marker0, marker1, mode_rank, 0x7fU);
+
+ for (i = 0U; i < num_slices; i++) {
+ u8 slice = byte_map[i];
+ u16 phy_val = *(__volatile__ const u16 *)
+ (PHY_REG_DBYTE_GATE_STAT + ((u32)slice << 13));
+ u8 mask = pmu_map_mask(phy_val, slice) & 0x7fU;
+
+ if (mask != 0x7fU)
+ res |= 1U;
+ }
+
+ return res;
+}
+
+/**
+ * pmu_cal_dbyte_deskew_results_apply() - Apply calibrated DBYTE deskew results to CSRs
+ * @slice_base: Pointer to deskew results buffer
+ * @param1: Format selector shifted through 0xe4 lookup
+ * @opcode: Calibration stage identifier (0x11 for full quad-tap mode)
+ */
+static void pmu_cal_dbyte_deskew_results_apply(void *slice_base, u32 param1,
+ u32 opcode)
+{
+ u32 bit_val = pmu_bist_pattern_bit_get(param1, 0U);
+ u8 slice_limit;
+ u32 slice, lane;
+
+ if (opcode == 0x11U) {
+ u8 feat_ovr = *(const __volatile__ u8 *)PMU_DMEM_TRAIN_FEAT_MASK_ADDR;
+ u8 base_flag = ((const u8 *)slice_base)[512];
+ u32 flag_bit = (feat_ovr & 2U) >> 1;
+
+ if (flag_bit != 0U && base_flag == 3U) {
+ u32 ch_off = pmu_sdata()->active_csr_offset << 1;
+
+ *(__volatile__ u16 *)(PHY_REG_DBYTE_BCAST_MODE |
+ ch_off) = 4U;
+ }
+
+ slice_limit = dmem_read8(PMU_DMEM_SLICE_END);
+ for (slice = 0U; slice <= slice_limit; slice++) {
+ u16 *p_slice = (u16 *)((uintptr_t)slice_base +
+ slice * 20U);
+
+ for (lane = 0U; lane < 9U; lane++) {
+ u16 dly_fine = p_slice[lane + 16U];
+ u16 dly_coarse = p_slice[lane + 56U];
+ u32 csr_addr;
+
+ csr_addr = PHY_REG_DBYTE_BASE |
+ (((lane << 8) | (slice << 12) |
+ bit_val |
+ pmu_sdata()->active_csr_offset) << 1);
+ *(__volatile__ u16 *)(csr_addr + 0x24U) =
+ dly_fine;
+ *(__volatile__ u16 *)(csr_addr + 0x20U) =
+ dly_coarse;
+
+ if (flag_bit != 0U && base_flag == 3U) {
+ u16 *p_quad = (u16 *)
+ ((uintptr_t)slice_base +
+ slice * 80U + lane * 8U +
+ 192U);
+ __volatile__ u16 *quad_csr =
+ (__volatile__ u16 *)
+ (PHY_REG_DBYTE_EYE_DELAY |
+ (slice << 13));
+
+ quad_csr[0] = p_quad[0];
+ quad_csr[1] = p_quad[1];
+ quad_csr[2] = p_quad[2];
+ quad_csr[3] = p_quad[3];
+ }
+ }
+ }
+
+ pmu_cal_pad_impedance_lock();
+ return;
+ }
+
+ slice_limit = dmem_read8(PMU_DMEM_SLICE_END);
+
+ for (slice = 0U; slice <= slice_limit; slice++) {
+ u16 *p_slice = (u16 *)((uintptr_t)slice_base + slice * 20U);
+ u16 val_slice_dly;
+ u32 slice_csr;
+
+ for (lane = 0U; lane < 9U; lane++) {
+ u16 val_dly = p_slice[lane + 16U];
+ u32 csr_addr;
+
+ csr_addr = PHY_REG_DBYTE_BASE |
+ (((lane << 8) | (slice << 12) |
+ bit_val |
+ pmu_sdata()->active_csr_offset) << 1);
+ *(__volatile__ u16 *)(csr_addr + 0x26U) = val_dly;
+ }
+
+ val_slice_dly = p_slice[25U];
+ slice_csr = PHY_REG_DBYTE_BASE |
+ (((slice << 12) | bit_val |
+ pmu_sdata()->active_csr_offset) << 1);
+ *(__volatile__ u16 *)(slice_csr + 0x28U) = val_slice_dly;
+ }
+}
+
+static __always_inline u16
+pmu_phase_edge_stream_slot(u32 slot, u8 count, u32 max_val, u32 sweep_lim,
+ const u16 *p_reg, u32 dmem_val, u16 shadow_idx)
+{
+ u32 i, j, k;
+
+ for (i = 0U; i < count; i++) {
+ for (j = 0U; j < sweep_lim; j++) {
+ u32 slice_apb_offset = j << 12;
+
+ for (k = max_val; k > 0U; k--) {
+ u16 reg_off = p_reg[i];
+ u32 full_addr = (slice_apb_offset | dmem_val |
+ reg_off) << 1;
+ u16 v1 = *(const __volatile__ u16 *)
+ (uintptr_t)(PHY_REG_PUB_BASE | full_addr);
+ u16 v2 = 0U;
+ u32 type = 1U;
+
+ if (slot < 2U) {
+ if ((dmem_val & 1U) != 0U) {
+ v2 = v1;
+ v1 = 0U;
+ type = 2U;
+ }
+ } else {
+ v2 = *(const __volatile__ u16 *)
+ (uintptr_t)(PHY_REG_PUB_BASE |
+ (full_addr + 2U));
+ type = 3U;
+ }
+
+ if (shadow_idx != 0U ||
+ *(const __volatile__ u16 *)PMU_DMEM_PLL_GATE_CH0_ADDR != 0U ||
+ *(const __volatile__ u16 *)PMU_DMEM_PLL_GATE_CH1_ADDR != 0U)
+ pmu_phy_pll_cfg_quad_write_stream((u16)(shadow_idx << 2),
+ v1, v2,
+ full_addr >> 1,
+ type);
+ shadow_idx++;
+ }
+ }
+ }
+
+ return shadow_idx;
+}
+
+/**
+ * pmu_cal_phase_detector_edge_align() - Align phase detector edges across channels
+ */
+void pmu_cal_phase_detector_edge_align(void)
+{
+ u32 tbl_base = PMU_DMEM_CAL_SEQ_MAP;
+ const void *phase_tbl_a = (const void *)(tbl_base + 0xb0U);
+ const void *phase_tbl_b = (const void *)(tbl_base + 0x16aU);
+ const void *phase_tbl_c = (const void *)(tbl_base + 0x1baU);
+ const void *phase_tbl_d = (const void *)(tbl_base + 0x1beU);
+ const void *phase_tbl_e = (const void *)(tbl_base + 0x3baU);
+ u8 channel_detect_mask[4];
+ const void *tbl_ptrs[4];
+ u16 param_block[8];
+ u16 eye_params[8];
+ u16 shadow_idx = (u16)(pmu_sdata()->shadow_trace_ptr >> 1);
+ u16 shadow_base;
+ u32 final_shadow;
+ u32 mode_idx, slot, i, ch, rank;
+
+ *(__volatile__ u16 *)PMU_DMEM_PLL_SEQ_EN_ADDR = shadow_idx;
+
+ for (mode_idx = 0U; mode_idx < 3U; mode_idx++) {
+ u32 sweep_lim;
+ u32 mode_param;
+
+ if (mode_idx == 0U) {
+ channel_detect_mask[0] = 91U;
+ channel_detect_mask[1] = 35U;
+ channel_detect_mask[2] = 2U;
+ channel_detect_mask[3] = 0U;
+ mode_param = (u8)pmu_sdata()->active_ch_cnt;
+ tbl_ptrs[0] = phase_tbl_a;
+ tbl_ptrs[1] = phase_tbl_b;
+ tbl_ptrs[2] = phase_tbl_c;
+ tbl_ptrs[3] = NULL;
+ sweep_lim = pmu_sdata()->active_slice_cnt;
+ } else if (mode_idx == 1U) {
+ channel_detect_mask[0] = 171U;
+ channel_detect_mask[1] = 20U;
+ channel_detect_mask[2] = 0U;
+ channel_detect_mask[3] = 0U;
+ mode_param = 1U;
+ tbl_ptrs[0] = phase_tbl_d;
+ tbl_ptrs[1] = phase_tbl_e - 0x28U;
+ tbl_ptrs[2] = NULL;
+ tbl_ptrs[3] = NULL;
+ sweep_lim = (u8)(pmu_sdata()->active_ch_cnt >> 16);
+ } else {
+ channel_detect_mask[0] = 63U;
+ channel_detect_mask[1] = 0U;
+ channel_detect_mask[2] = 0U;
+ channel_detect_mask[3] = 0U;
+ mode_param = 1U;
+ tbl_ptrs[0] = phase_tbl_e;
+ tbl_ptrs[1] = NULL;
+ tbl_ptrs[2] = NULL;
+ tbl_ptrs[3] = NULL;
+ sweep_lim = 1U;
+ }
+
+ for (slot = 0U; slot < 4U; slot++) {
+ u32 repeat_cnt = ((slot | 2U) == 3U) ? mode_param : 1U;
+ u32 max_val = (repeat_cnt > 1U) ? repeat_cnt : 1U;
+ u32 dmem_val = *(const __volatile__ u32 *)
+ (tbl_base + mode_idx * 4U + 0x18U);
+
+ shadow_idx = pmu_phase_edge_stream_slot(slot,
+ channel_detect_mask[slot],
+ max_val, sweep_lim,
+ (const u16 *)tbl_ptrs[slot],
+ dmem_val,
+ shadow_idx);
+ }
+ }
+
+ *(__volatile__ u16 *)PHY_REG_SHADOW_CNT =
+ (u16)(shadow_idx - *(const __volatile__ u16 *)PMU_DMEM_PLL_SEQ_EN_ADDR);
+ shadow_base = (u16)(pmu_sdata()->shadow_trace_ptr >> 1);
+ *(__volatile__ u16 *)PMU_DMEM_SHADOW_MID_IDX_ADDR = shadow_base;
+ *(__volatile__ u16 *)PMU_DMEM_SHADOW_PHASE_CNT_ADDR =
+ (u16)(shadow_base - *(const __volatile__ u16 *)PMU_DMEM_SHADOW_INIT_IDX_ADDR);
+
+ for (i = 0U; i < 8U; i++)
+ eye_params[i] = 0U;
+ eye_params[3] = 0x1b00U;
+
+ for (ch = 0U; ch < 2U; ch++) {
+ u8 active = (ch != 0U) ?
+ *(const __volatile__ u8 *)PMU_DMEM_CH1_ACTIVE_FLAG :
+ *(const __volatile__ u8 *)PMU_DMEM_CH0_ANALOG_EN;
+ u16 pll_cfg;
+ u32 ch_shift;
+ u8 limit_entry;
+
+ if (active == 0U)
+ continue;
+
+ pll_cfg = (ch != 0U) ?
+ *(const __volatile__ u16 *)PMU_DMEM_PLL_GATE_CH1_ADDR :
+ *(const __volatile__ u16 *)PMU_DMEM_PLL_GATE_CH0_ADDR;
+ ch_shift = ch << 1;
+ limit_entry = pmu_sdata()->metric_tbl_entries;
+
+ for (rank = 0U; rank < limit_entry; rank++) {
+ u8 param_b = pmu_sdata()->cal_cmd_tbl[rank];
+ u8 param_c =
+ *(const __volatile__ u8 *)
+ (PMU_DMEM_DESKEW_STAT_TBL + ch_shift + rank);
+ u32 eval_mask = 0U;
+
+ if (param_b == 12U)
+ param_c |= 0x80U;
+
+ if (((*(const __volatile__ u8 *)PMU_DMEM_DESKEW_LIM >>
+ ch_shift) & 1U) != 0U)
+ eval_mask = (param_b != 12U) ? 1U : 0U;
+
+ pmu_cal_eye_param_block_init(param_block, param_b,
+ param_c, eval_mask);
+ pmu_pll_quad_param_tuple_stream(param_block, &shadow_idx,
+ &pll_cfg);
+
+ if (rank < (u32)(limit_entry - 1U))
+ pmu_pll_quad_param_tuple_stream(eye_params,
+ &shadow_idx,
+ &pll_cfg);
+ }
+ }
+
+ final_shadow = pmu_sdata()->shadow_trace_ptr >> 1;
+ *(__volatile__ u16 *)PMU_DMEM_SHADOW_FINAL_CNT_ADDR =
+ (u16)(final_shadow - *(const __volatile__ u16 *)PMU_DMEM_SHADOW_MID_IDX_ADDR);
+}
+
+/**
+ * pmu_dbyte_lane_error_mask_calc() - Trigger lane error mask evaluation pulse sequence
+ * @mode: Non-zero for two-step 0x25/0x26 pulse sequence, 0 for single 0x19 pulse
+ * @slice_or_rank: Target slice or rank bitmask
+ */
+noinline void pmu_dbyte_lane_error_mask_calc(u32 mode, u32 slice_or_rank)
+{
+ u32 zero = 0U;
+
+ pmu_cal_strobe_pulse_init();
+ if (mode != 0U) {
+ pmu_cal_pad_timing_delay_set(zero, 5U, 1U);
+ pmu_cal_sequence_pulse_send(zero, 0x25U, zero, 0x86U, zero,
+ slice_or_rank, zero);
+ pmu_cal_sequence_pulse_send(zero, 0x26U, zero, 0x86U, zero,
+ zero, zero);
+ pmu_cal_pad_timing_delay_set(zero, 0x10U, 1U);
+ pmu_cbt_3phase_pulse_seq();
+ } else {
+ pmu_cal_sequence_pulse_send(0x80U, 0x19U, 4U, 0x86U, zero,
+ slice_or_rank & 3U, zero);
+ }
+ pmu_cal_slice_step_diff_latch(zero, 1U);
+ if (mode == 0U)
+ pmu_delay_us(0x7530U, zero);
+}
+
+/**
+ * pmu_rank_slice_deskew_latch() - Latch rank slice deskew step delay
+ * @step_delay: Calibrated step delay value
+ * @lane_mask: Target lane/slice bitmask
+ */
+static void pmu_rank_slice_deskew_latch(u32 step_delay, u32 lane_mask)
+{
+ u32 shift = pmu_freq_ratio_mult(14U);
+
+ pmu_cal_strobe_pulse_init();
+ pmu_cal_sequence_pulse_send(0U, 6U, 2U, step_delay, 30U,
+ lane_mask, 0U);
+ if (shift < 5U)
+ shift = 5U;
+ pmu_cal_pad_timing_delay_set(0U, (u8)shift, 0U);
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+}
+
+/**
+ * pmu_slice_deskew_state_latch() - Latch slice deskew state via opcode 24 pulse
+ * @step_delay: Calibrated step delay value
+ * @lane_mask: Target lane/slice bitmask
+ */
+static void pmu_slice_deskew_state_latch(u32 step_delay, u32 lane_mask)
+{
+ u32 zero = 0U;
+
+ pmu_cal_strobe_pulse_init();
+ pmu_cal_sequence_pulse_send(0U, 6U, 0x40U, step_delay, 24U,
+ lane_mask, zero);
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+}
+
+/**
+ * pmu_channel_timing_deskew_reset() - Reset channel timing deskew parameters
+ * @dly_ac0: AC delay parameter 0 (tap 33)
+ * @dly_ac1: AC delay parameter 1 (tap 34)
+ * @dly_ac2: AC delay parameter 2 (tap 31)
+ * @dly_ac3: AC delay parameter 3 (tap 32)
+ * @target_slice: Target slice bitmask
+ */
+void pmu_channel_timing_deskew_reset(u32 dly_ac0, u32 dly_ac1, u32 dly_ac2,
+ u32 dly_ac3, u32 target_slice)
+{
+ u32 zero = 0U;
+ u32 val0;
+
+ pmu_cal_strobe_pulse_init();
+ val0 = *(__volatile__ const u8 *)PMU_DMEM_CH_MASK & 0x3fU;
+
+ pmu_cal_sequence_pulse_send(0U, 6U, 0x22U, val0, 20U,
+ target_slice, zero);
+ pmu_cal_sequence_pulse_send(0U, 6U, 0x22U, dly_ac0, 33U,
+ target_slice, zero);
+ pmu_cal_sequence_pulse_send(0U, 6U, 0x22U, dly_ac1, 34U,
+ target_slice, zero);
+ pmu_cal_sequence_pulse_send(0U, 6U, 0x22U, dly_ac2, 31U,
+ target_slice, zero);
+ pmu_cal_sequence_pulse_send(0U, 6U, 0x22U, dly_ac3, 32U,
+ target_slice, zero);
+
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+}
+
+/**
+ * pmu_slice_deskew_pulse_train() - Execute slice deskew pulse train for rank
+ * @ch_idx: Target channel index (0 or 1)
+ * @rank: Target rank index
+ * @mode: Non-zero for full multi-pulse sequence, 0 for short sequence
+ */
+static void pmu_slice_deskew_pulse_train(u32 ch_idx, u32 rank, u32 mode)
+{
+ u32 zero = 0U;
+ const u8 *p = (const u8 *)(PMU_DMEM_CAL_STRUCT_ADDR +
+ (ch_idx * 2U + rank) * PMU_SLICE_DESC_SIZE);
+ u32 mask1 = 1U << rank;
+ u8 slice_pair_mask = (u8)((mask1 << 2) | mask1);
+ u32 deskew_cmd;
+
+ deskew_cmd = ((mode == 0U) ? (p[0x10] | 5U) : (p[0x10] & 0xb0U)) |
+ (1U << 6);
+
+ pmu_cal_strobe_pulse_init();
+
+ if (mode != 0U) {
+ pmu_cal_sequence_pulse_send(zero, 6U, 0x22U, deskew_cmd | 4U,
+ 16U, slice_pair_mask, zero);
+
+ if ((p[1] & 8U) != 0U)
+ pmu_cal_sequence_pulse_send(zero, 6U, 0x22U, p[1], 1U,
+ slice_pair_mask, zero);
+ else if ((p[20] & 3U) == 2U)
+ pmu_cal_strobe_pulse_init();
+
+ if ((p[20] & 3U) != 2U)
+ pmu_cal_sequence_pulse_send(zero, 6U, 0x22U, p[20],
+ 20U, slice_pair_mask, zero);
+ }
+
+ pmu_cal_sequence_pulse_send(0x80U, 6U, 0x22U, deskew_cmd, 16U,
+ slice_pair_mask, zero);
+ pmu_cal_slice_step_diff_latch(zero, 1U);
+ pmu_delay_us(0x3d090U, zero);
+}
+
+/**
+ * pmu_dbyte_slice_cal_param_pulse() - Send DBYTE slice calibration parameter pulse
+ * @rank: Target rank index
+ * @channel: Target channel index (0 or 1)
+ * @slice_mask: Non-zero to set bits 3 and 6 in parameter byte
+ */
+static void pmu_dbyte_slice_cal_param_pulse(u32 rank, u32 channel, u32 slice_mask)
+{
+ u32 zero = 0U;
+ u8 val;
+ u32 mask1;
+
+ pmu_cal_strobe_pulse_init();
+
+ val = *(const u8 *)(PMU_DMEM_CAL_STRUCT_ADDR +
+ (rank * 2U + channel) * PMU_SLICE_DESC_SIZE + 18U);
+ if (slice_mask != 0U)
+ val |= 0x48U;
+
+ mask1 = 1U << channel;
+ pmu_cal_sequence_pulse_send(zero, 6U, 0x22U, val, 18U,
+ (u8)((mask1 << 2) | mask1), zero);
+ pmu_cal_sequence_pulse_send(0x80U, 7U, 2U, zero, zero, zero, zero);
+ pmu_cal_slice_step_diff_latch(zero, 1U);
+}
+
+static __always_inline void
+pmu_fine_tune_pulse_rank(const __volatile__ u8 *dmem, u32 ch, u32 rank,
+ u16 reg_val, u8 num_entries)
+{
+ u32 ch_shift = ch * 2U;
+ u8 pulse_stat = 0U;
+ int entry_idx;
+
+ pmu_cal_strobe_pulse_init();
+ pmu_cal_marker_pair_set((u16)(reg_val << 2));
+
+ for (entry_idx = 0; entry_idx < (int)num_entries; entry_idx++) {
+ u32 lane;
+
+ for (lane = 0U; lane < 2U; lane++) {
+ u8 cal_cmd = dmem[0x46cU + entry_idx];
+ u8 delay_val = dmem[0x474U + (rank ? 0x14U : 0U) +
+ (ch_shift + entry_idx * 4) + lane];
+ u32 pulse_mode = (lane != 0U &&
+ entry_idx != (int)num_entries - 1) ?
+ 0x22U : 0U;
+ u32 lane_bitmask = 1U << lane;
+
+ if (pulse_stat != 0U && cal_cmd == 0x0cU) {
+ if (((dmem[0x1c] >> ch_shift) &
+ lane_bitmask) == 0U)
+ continue;
+ delay_val |= 0x80U;
+ }
+
+ pmu_cal_sequence_pulse_send(0U, 6U, pulse_mode,
+ delay_val, cal_cmd,
+ (lane_bitmask << 2) |
+ lane_bitmask, 0U);
+
+ if (lane == 1U && pulse_stat == 0U && cal_cmd == 0x0cU) {
+ if (dmem[0x1c] != 0U) {
+ entry_idx--;
+ pulse_stat = 1U;
+ } else {
+ pulse_stat = 0U;
+ }
+ }
+
+ if (cal_cmd == 0x0cU && dmem[0x1c] == 0U)
+ pmu_cal_pad_timing_delay_set(0U, 4U, 0U);
+ }
+ }
+
+ pmu_cal_sequence_pulse_send(0U, 7U, 2U, 0U, 0U, 0U, 0x28U);
+}
+
+/**
+ * pmu_cal_dbyte_deskew_fine_tune() - Execute DBYTE deskew fine-tune pass
+ * @mode: Non-zero for CSR writeback mode, 0 for pulse sequence mode
+ */
+void pmu_cal_dbyte_deskew_fine_tune(u32 mode)
+{
+ const __volatile__ u8 *dmem = (const __volatile__ u8 *)PMU_DMEM_BASE;
+ u32 counter = 0U;
+ u32 ch;
+
+ for (ch = 0U; ch < 2U; ch++) {
+ u8 active = (ch == 0U) ? dmem[0x25] : dmem[0x40];
+ u16 reg_pair[2];
+ u8 num_entries;
+ u32 rank;
+
+ if (active == 0U)
+ continue;
+
+ pmu_phy_mode_cfg_dispatch(ch + 1U);
+
+ reg_pair[0] = *(const __volatile__ u16 *)
+ (dmem + 0xe8U + ch * 2U);
+ reg_pair[1] = *(const __volatile__ u16 *)
+ (dmem + 0xecU + ch * 2U);
+ num_entries = pmu_sdata()->metric_tbl_entries;
+
+ if (mode != 0U) {
+ u32 entry;
+
+ for (rank = 0U; rank < 2U; rank++) {
+ for (entry = 0U; entry < num_entries; entry++) {
+ u8 deskew_stat_val = dmem[0xb6eU + ch * 2U +
+ rank + entry * 4U];
+ u32 csr_addr;
+
+ csr_addr = PHY_REG_PUB_BASE |
+ ((((reg_pair[0] << 2) |
+ 0x41000U) +
+ (counter & 0xffU)) << 1);
+ *(__volatile__ u16 *)csr_addr = deskew_stat_val;
+ counter++;
+ }
+ }
+
+ if ((counter & 1U) != 0U) {
+ u32 csr_addr = PHY_REG_PUB_BASE |
+ ((((reg_pair[0] << 2) |
+ 0x41000U) +
+ (counter & 0xffU)) << 1);
+
+ *(__volatile__ u16 *)csr_addr = 0U;
+ counter++;
+ }
+ } else {
+ for (rank = 0U; rank < 2U; rank++)
+ pmu_fine_tune_pulse_rank(dmem, ch, rank,
+ reg_pair[rank],
+ num_entries);
+ }
+ }
+
+ *(__volatile__ u16 *)PHY_REG_CAL_STAT_CTRL0 = 0U;
+ *(__volatile__ u16 *)PHY_REG_CAL_STAT_CTRL1 = 0U;
+}
+
+/**
+ * pmu_cal_gate_eye_master_trigger() - Trigger AC lane profile and gate eye error check
+ */
+void pmu_cal_gate_eye_master_trigger(void)
+{
+ u8 a7c;
+ u32 mode = 0U;
+
+ pmu_phy_mode_cfg_dispatch(3U);
+ a7c = *(__volatile__ const u8 *)PMU_DMEM_AC_CAL_ACTIVE;
+
+ if (a7c != 0U) {
+ u8 d1 = *(__volatile__ const u8 *)PMU_DMEM_CLK_GATE_FLAG;
+
+ if ((d1 & (1U << 3)) == 0U) {
+ pmu_ac_lane_profile_setup();
+ mode = 1U;
+ }
+ }
+
+ pmu_dbyte_lane_error_mask_calc(mode, 0x0fU);
+}
+
+struct dly_line_entry {
+ u16 word_val;
+ u8 val8[2];
+};
+
+/**
+ * pmu_slice_dly_tap_coeff_prog() - Accumulate and program per-slice delay line tap coefficients
+ * @rank: Active 9-bit lane bitmask
+ */
+static void pmu_slice_dly_tap_coeff_prog(u32 rank)
+{
+ struct dly_line_entry buf[6][2];
+ u32 dram_type = pmu_phy_query_operating_mode();
+ u32 count = ((dmem_read8(PMU_DMEM_CH1_ACTIVE_FLAG) |
+ dmem_read8(PMU_DMEM_CH0_ANALOG_EN)) == 3U) ?
+ 2U : 1U;
+ u32 pstate = dmem_read8(PMU_DMEM_SLICE_START);
+ u32 pstate_end = dmem_read8(PMU_DMEM_SLICE_END);
+
+ for (; pstate <= pstate_end; pstate++) {
+ u32 lane;
+
+ if ((dmem_read8(PMU_DMEM_ACTIVE_SLICE_MASK) &
+ (1U << pstate)) == 0U)
+ continue;
+
+ for (lane = 0U; lane < 9U; lane++) {
+ u32 csr_base;
+ u32 k;
+
+ if ((rank & (1U << lane)) == 0U)
+ continue;
+
+ csr_base = pmu_sdata()->active_csr_offset |
+ (pstate << 12) | (lane << 8);
+
+ for (k = 0U; k < count; k++) {
+ u32 idx = pstate * 20U + lane * 2U + k;
+ __volatile__ const u16 *cbt_coeff_ptr =
+ (__volatile__ const u16 *)
+ (PMU_DMEM_CBT_COEFF_A_ADDR + idx * 2U);
+ __volatile__ const u8 *cbt_flags_ptr =
+ (__volatile__ const u8 *)
+ (PMU_DMEM_CBT_LANE_FLAGS_ADDR + idx);
+ u32 d;
+
+ buf[0][k].word_val = cbt_coeff_ptr[0];
+ buf[0][k].val8[0] = cbt_flags_ptr[0];
+ buf[0][k].val8[1] = cbt_flags_ptr[0xa0];
+
+ buf[1][k].word_val = cbt_coeff_ptr[0x50];
+ buf[1][k].val8[0] = cbt_flags_ptr[0x50];
+ buf[1][k].val8[1] = cbt_flags_ptr[0xf0];
+
+ for (d = 0U; d < 2U; d++) {
+ u32 c_addr = (PHY_REG_DBYTE_BASE |
+ (d ? 0x0020UL : 0x0024UL)) |
+ ((csr_base | k) << 1);
+ u16 raw_c = *(__volatile__ const u16 *)c_addr;
+ u32 s_idx;
+
+ buf[2U + d][k].word_val = pmu_dly_line_repack(raw_c);
+ for (s_idx = 0U; s_idx < 2U; s_idx++) {
+ u32 off = pmu_dly_reg_offset_select(d,
+ s_idx,
+ k);
+ u32 r_addr = PHY_REG_DBYTE_BASE |
+ ((csr_base | off) << 1);
+
+ buf[2U + d][k].val8[s_idx] =
+ (u8)(*(__volatile__ const u16 *)r_addr);
+ }
+ }
+ }
+
+ pmu_cal_delay_accum_dispatch(dram_type, count, buf);
+
+ for (k = 0U; k < count; k++) {
+ u32 d;
+ u32 num_s;
+ u32 s_idx;
+
+ for (d = 0U; d < 2U; d++) {
+ u32 c_addr = (PHY_REG_DBYTE_BASE |
+ (d ? 0x0020UL : 0x0024UL)) |
+ ((csr_base | k) << 1);
+
+ *(__volatile__ u16 *)c_addr =
+ buf[4U + d][k].word_val;
+ }
+
+ if (dmem_read8(PMU_DMEM_CAL_OVERRIDE) != 0U)
+ continue;
+
+ num_s = (pmu_phy_query_operating_mode() == 3U) ?
+ 2U : 1U;
+ for (s_idx = 0U; s_idx < num_s; s_idx++) {
+ for (d = 0U; d < 2U; d++) {
+ u32 off = pmu_dly_reg_offset_select(d,
+ s_idx,
+ k);
+ u32 w_addr = PHY_REG_DBYTE_BASE |
+ ((csr_base | off) << 1);
+
+ *(__volatile__ u16 *)w_addr =
+ (u16)buf[4U + d][k].val8[s_idx];
+ }
+ }
+ }
+ }
+ }
+}
+
+static __always_inline void
+deskew_eval_coarse_fine_slice(u32 slice, s32 freq_pt, s32 ph, u8 desc_flag8,
+ u16 lane_tap_steps_b[2][4],
+ u16 lane_metric_buf_b[2][32],
+ u16 lane_metric_buf_a[2][32],
+ s32 *max_coarse, s32 *max_fine)
+{
+ u32 bit;
+
+ if ((desc_flag8 & 1U) != 0U) {
+ s32 v_fp = (s32)(lane_tap_steps_b[freq_pt][slice] >> 6);
+ s32 v_ph = (s32)(lane_tap_steps_b[ph][slice] >> 6);
+ s32 diff = v_fp - v_ph;
+
+ if (diff > *max_coarse)
+ *max_coarse = diff;
+ } else {
+ for (bit = 0U; bit < 8U; bit++) {
+ s32 v_fp = (s32)(lane_metric_buf_b[freq_pt][slice * 8U + bit] >> 6);
+ s32 v_ph = (s32)(lane_metric_buf_b[ph][slice * 8U + bit] >> 6);
+ s32 diff = v_fp - v_ph;
+
+ if (diff > *max_coarse)
+ *max_coarse = diff;
+ }
+ }
+
+ for (bit = 0U; bit < 8U; bit++) {
+ s32 val_fp = (s32)lane_metric_buf_a[freq_pt][slice * 8U + bit];
+ s32 val_ph = (s32)lane_metric_buf_a[ph][slice * 8U + bit];
+ s32 diff = val_fp - val_ph;
+
+ if (diff > *max_fine)
+ *max_fine = diff;
+ }
+}
+
+static __always_inline void
+deskew_eval_cross_slice(u32 slice, s32 phase_b, s32 phase_a, u8 desc_flag8,
+ u8 cal_ctrl_lo, u16 lane_tap_steps_b[2][4],
+ u16 lane_tap_steps_a[2][4],
+ u16 lane_metric_buf_b[2][32],
+ u16 lane_metric_buf_a[2][32],
+ s32 *max_diff, s32 *max_cross)
+{
+ u32 bit;
+
+ if ((desc_flag8 & 1U) != 0U) {
+ s32 ref_dly = (s32)lane_tap_steps_b[phase_b][slice];
+
+ for (bit = 0U; bit < 8U; bit++) {
+ s32 samp_dly = (s32)lane_metric_buf_a[phase_a][slice * 8U + bit];
+ s32 diff = ref_dly - samp_dly;
+
+ if (diff > *max_diff)
+ *max_diff = diff;
+ }
+ if ((cal_ctrl_lo & 0x10U) != 0U) {
+ s32 aux_dly = (s32)lane_tap_steps_a[phase_a][slice];
+ s32 diff = ref_dly - aux_dly;
+
+ if (diff > *max_diff)
+ *max_diff = diff;
+ }
+ } else {
+ for (bit = 0U; bit < 8U; bit++) {
+ s32 tap_a = (s32)lane_metric_buf_a[phase_a][slice * 8U + bit];
+ s32 tap_b = (s32)lane_metric_buf_b[phase_b][slice * 8U + bit];
+ s32 diff = tap_b - tap_a;
+
+ if (diff > *max_diff)
+ *max_diff = diff;
+ }
+ }
+
+ if (phase_a != phase_b) {
+ for (bit = 0U; bit < 8U; bit++) {
+ s32 cross_a = (s32)lane_metric_buf_a[phase_a][slice * 8U + bit];
+ s32 cross_b = (s32)lane_metric_buf_a[phase_b][slice * 8U + bit];
+ s32 diff = cross_b - cross_a;
+
+ if (diff > *max_cross)
+ *max_cross = diff;
+ }
+ if ((cal_ctrl_lo & 0x10U) != 0U) {
+ s32 aux_b = (s32)lane_tap_steps_a[phase_b][slice];
+ s32 aux_a = (s32)lane_tap_steps_a[phase_a][slice];
+ s32 diff = aux_b - aux_a;
+
+ if (diff > *max_cross)
+ *max_cross = diff;
+ }
+ }
+}
+
+/**
+ * pmu_cal_lane_deskew_sweep_exec() - Execute per-lane deskew sweep across active slices
+ * @channel: Target channel index (0 or 1)
+ */
+void pmu_cal_lane_deskew_sweep_exec(u32 channel)
+{
+ u16 lane_metric_buf_a[2][32];
+ u16 lane_metric_buf_b[2][32];
+ u16 lane_tap_steps_b[2][4];
+ u16 lane_tap_steps_a[2][4];
+ const u8 *desc = (const u8 *)dmem_ptr(PMU_DMEM_PSTATE_FLAGS);
+ const u8 *desc_p;
+ u8 desc_mode;
+ u8 desc_flag8;
+ u8 cal_ctrl_lo;
+ u32 step_flag_f;
+ u32 csr_offset;
+ u32 phase;
+ u32 rank_off;
+ __volatile__ u8 *out_step_a;
+ __volatile__ u8 *out_step_d;
+ s32 freq_pt;
+ s32 ph;
+ u32 step_flag_e;
+ __volatile__ u8 *out_step_c;
+ __volatile__ u8 *out_step_b;
+ s32 phase_b;
+ s32 phase_a;
+
+ memset(lane_metric_buf_b, 0, sizeof(lane_metric_buf_b));
+ memset(lane_metric_buf_a, 0, sizeof(lane_metric_buf_a));
+ memset(lane_tap_steps_b, 0, sizeof(lane_tap_steps_b));
+ memset(lane_tap_steps_a, 0, sizeof(lane_tap_steps_a));
+
+ desc_p = desc + 130U;
+ desc_mode = desc_p[0];
+ desc_flag8 = desc_p[8];
+ cal_ctrl_lo = desc_p[16];
+ step_flag_f = desc_mode & 0x10U;
+
+ csr_offset = pmu_sdata()->active_csr_offset;
+
+ for (phase = 0U; phase < 2U; phase++) {
+ u8 start_slice = dmem_read8(PMU_DMEM_SLICE_START);
+ u8 end_slice = dmem_read8(PMU_DMEM_SLICE_END);
+ u32 slice;
+
+ if ((desc_flag8 & 1U) != 0U) {
+ pmu_slice_lcdl_delay_collect((phase + 0x20U) | csr_offset,
+ lane_tap_steps_b[phase], 1U);
+ } else {
+ for (slice = start_slice; slice <= end_slice; slice++) {
+ u32 slice_code = (slice << 12) |
+ ((phase + 0x24U) | csr_offset);
+ u32 bit;
+
+ for (bit = 0U; bit < 8U; bit++) {
+ u32 reg = PHY_REG_DBYTE_BASE |
+ ((slice_code |
+ (bit << 8)) << 1);
+
+ lane_metric_buf_b[phase][slice * 8U + bit] =
+ *(__volatile__ const u16 *)reg;
+ }
+ }
+ }
+
+ for (slice = start_slice; slice <= end_slice; slice++) {
+ u32 slice_code = (slice << 12) |
+ ((phase + 0x26U) | csr_offset);
+ u32 bit;
+
+ for (bit = 0U; bit < 8U; bit++) {
+ u32 reg = PHY_REG_DBYTE_BASE |
+ ((slice_code | (bit << 8)) << 1);
+
+ lane_metric_buf_a[phase][slice * 8U + bit] =
+ *(__volatile__ const u16 *)reg;
+ }
+ }
+
+ if ((cal_ctrl_lo & 0x10U) != 0U) {
+ pmu_slice_lcdl_delay_collect((phase + 0x28U) | csr_offset,
+ lane_tap_steps_a[phase], 1U);
+ }
+ }
+
+ rank_off = (channel != 0U) ? 27U : 0U;
+ out_step_a = (__volatile__ u8 *)(PMU_DMEM_CH0_DESKEW_COARSE + rank_off);
+ out_step_d = (__volatile__ u8 *)(PMU_DMEM_CH0_DESKEW_FINE + rank_off);
+
+ for (freq_pt = 1; freq_pt >= 0; freq_pt--) {
+ for (ph = 1; ph >= 0; ph--) {
+ s16 diff_coarse = 0;
+ s16 diff_fine = 0;
+
+ if (ph == freq_pt)
+ continue;
+
+ if (step_flag_f != 0U) {
+ u8 start_slice = dmem_read8(PMU_DMEM_SLICE_START);
+ u8 end_slice = dmem_read8(PMU_DMEM_SLICE_END);
+ s32 max_coarse = 0;
+ s32 max_fine = 0;
+ u32 slice;
+
+ for (slice = start_slice; slice <= end_slice;
+ slice++) {
+ deskew_eval_coarse_fine_slice(slice, freq_pt, ph,
+ desc_flag8,
+ lane_tap_steps_b,
+ lane_metric_buf_b,
+ lane_metric_buf_a,
+ &max_coarse,
+ &max_fine);
+ }
+
+ diff_coarse = (s16)pmu_math_freq_scale_div(max_coarse << 6);
+ diff_fine = (s16)pmu_math_freq_scale_div(max_fine);
+ }
+
+ *out_step_a++ = (u8)diff_coarse;
+ *out_step_d++ = (u8)diff_fine;
+ }
+ }
+
+ step_flag_e = (step_flag_f != 0U) ? 1U : 0U;
+ out_step_c = (__volatile__ u8 *)(PMU_DMEM_CH0_DESKEW_CROSS + rank_off);
+ out_step_b = (__volatile__ u8 *)(PMU_DMEM_ACTIVE_SLICE_CNT + rank_off);
+
+ for (phase_b = 1; phase_b >= 0; phase_b--) {
+ for (phase_a = 1; phase_a >= 0; phase_a--) {
+ s16 res_diff = 0;
+ s16 res_cross = 0;
+
+ if (!(phase_a != phase_b && step_flag_e == 0U)) {
+ u8 start_slice = dmem_read8(PMU_DMEM_SLICE_START);
+ u8 end_slice = dmem_read8(PMU_DMEM_SLICE_END);
+ s32 max_diff = 0;
+ s32 max_cross = 0;
+ u32 slice;
+
+ for (slice = start_slice; slice <= end_slice;
+ slice++) {
+ deskew_eval_cross_slice(slice, phase_b, phase_a,
+ desc_flag8, cal_ctrl_lo,
+ lane_tap_steps_b,
+ lane_tap_steps_a,
+ lane_metric_buf_b,
+ lane_metric_buf_a,
+ &max_diff, &max_cross);
+ }
+
+ res_diff = (s16)pmu_math_freq_scale_div(max_diff);
+ res_cross = (s16)pmu_math_freq_scale_div(max_cross);
+ }
+
+ *out_step_b++ = (u8)res_diff;
+ *out_step_c++ = (u8)res_cross;
+ }
+ }
+}
+
+/**
+ * pmu_cal_bist_pattern_setup() - Configure BIST lane masks and pattern generators
+ * @pattern_idx: BIST pattern index (0..8)
+ * @slice_or_lane: Target lane index or 0x0f for broadcast
+ * @enable_deskew_reset: Non-zero to reset channel timing deskew
+ * @validation_mode: Validation mode selector
+ */
+void pmu_cal_bist_pattern_setup(u32 pattern_idx, u32 slice_or_lane,
+ u32 enable_deskew_reset, u32 validation_mode)
+{
+ u16 pat = 0x5a3cU;
+ u32 valid_mode = 1U;
+ s8 offset_val = 0;
+ u32 is_mid;
+ u16 bcast_pat_ctrl;
+ u16 bctrl;
+
+ switch (pattern_idx) {
+ case 0U:
+ case 8U:
+ break;
+ case 1U:
+ pat = 0xff00U;
+ valid_mode = 0U;
+ break;
+ case 2U:
+ case 4U:
+ pat = 0xa536U;
+ valid_mode = 0U;
+ break;
+ case 3U:
+ pat = 0xaaaaU;
+ valid_mode = 0U;
+ break;
+ case 5U:
+ pat = 0xb2b2U;
+ valid_mode = 0U;
+ break;
+ case 6U:
+ pat = 0x8241U;
+ valid_mode = 0U;
+ break;
+ case 7U:
+ pat = *(__volatile__ const u16 *)PMU_DMEM_TEST_PAT;
+ valid_mode = 0U;
+ break;
+ default:
+ pmu_assert_or_halt(0U, PMU_ERR_BIST_PATTERN_IDX_INVALID);
+ pat = 0U;
+ valid_mode = 0U;
+ break;
+ }
+
+ if (enable_deskew_reset != 0U) {
+ u8 e6 = *(__volatile__ const u8 *)PMU_DMEM_DESKEW_OFFSET;
+
+ offset_val = (e6 != 0U) ? (s8)e6 : (s8)-86;
+ }
+
+ is_mid = (pattern_idx == 1U || (pattern_idx - 3U < 5U)) ? 1U : 0U;
+ if (is_mid != 0U)
+ pmu_channel_timing_deskew_reset((u8)pat,
+ (pat >> 8) & 0xffU,
+ (u32)offset_val,
+ (u32)offset_val,
+ slice_or_lane);
+
+ pmu_phy_pll_cfg_quad_write(0xffffU);
+ *(__volatile__ u16 *)PHY_REG_AC_BCAST_PAT_DATA = 0U;
+ pmu_cal_deskew_quad_tap_set(0x0fU, pat, pat, pat, pat);
+
+ if (enable_deskew_reset != 0U) {
+ if (pattern_idx == 0U) {
+ *(__volatile__ u16 *)PHY_REG_AC_BCAST = 22U;
+ } else {
+ u32 i;
+
+ pmu_assert_or_halt(((validation_mode != 0U) ? 1U : 0U) |
+ valid_mode,
+ PMU_ERR_BIST_VALIDATION_MODE);
+ *(__volatile__ u16 *)PHY_REG_AC_BCAST = 16U;
+ for (i = 0U; i < 8U; i++) {
+ u16 p = ((offset_val & (1 << i)) != 0) ?
+ (u16)~pat : pat;
+
+ pmu_cal_deskew_quad_tap_set(i, p, p, p, p);
+ }
+ pmu_cal_deskew_quad_tap_set(8U, pat, pat, pat, pat);
+ }
+ } else {
+ *(__volatile__ u16 *)PHY_REG_AC_BCAST = 16U;
+ }
+
+ bcast_pat_ctrl = (is_mid != 0U) ? 0U : 256U;
+ *(__volatile__ u16 *)PHY_REG_AC_BCAST_PAT_CTRL = bcast_pat_ctrl;
+ *(__volatile__ u16 *)PHY_REG_AC_BCAST_PAT_CFG = bcast_pat_ctrl;
+
+ bctrl = ((*(__volatile__ const u8 *)PMU_DMEM_PLL_RATIO == 4U) ?
+ 0x84U : 4U) | ((pattern_idx != 0U) ? 5U : 8U);
+ *(__volatile__ u16 *)PHY_REG_BIST_CTRL_MODE = bctrl;
+ if (pattern_idx == 0U)
+ *(__volatile__ u16 *)PHY_REG_AC_BCAST_PAT_DATA = 0U;
+}
+
+/**
+ * pmu_sweep_save_or_restore_lanes() - Save or restore per-lane delay registers across slices
+ * @local_buf: 40-halfword per-slice/lane backup buffer
+ * @slice_start: First slice index
+ * @slice_end: Last slice index
+ * @active_mask: Active slice bitmask
+ * @active_csr_offset: Shifted channel CSR offset
+ * @cmd_b_off: Shifted secondary register offset from sweep context
+ * @loop_bound: Number of lanes to save or restore per slice
+ * @save: true to read from PHY into @local_buf, false to write @local_buf back to PHY
+ */
+static void pmu_sweep_save_or_restore_lanes(u16 *local_buf, u8 slice_start,
+ u8 slice_end, u8 active_mask,
+ u32 active_csr_offset,
+ u32 cmd_b_off, u32 loop_bound,
+ bool save)
+{
+ u32 slice;
+
+ for (slice = slice_start; slice <= slice_end; slice++) {
+ u32 base;
+ u16 *p_buf;
+ u32 lane;
+
+ if ((active_mask & (1U << slice)) == 0U)
+ continue;
+
+ base = PHY_REG_DBYTE_BASE | (slice << 13) | active_csr_offset |
+ cmd_b_off;
+ p_buf = &local_buf[slice * 10U];
+
+ for (lane = 0U; lane < loop_bound; lane++) {
+ u32 addr = base | (lane << 9);
+
+ if (save)
+ p_buf[lane] = *(__volatile__ const u16 *)addr;
+ else
+ *(__volatile__ u16 *)addr = p_buf[lane];
+ }
+ }
+}
+
+/**
+ * pmu_sweep_update_lane_state() - Update per-lane eye edge state machine for one sample
+ * @entry: Pointer to 848-byte lane sweep record
+ * @eff_step: Effective step index within the record
+ * @cur_val: Current sweep step value
+ * @phy_sample: Sampled 16-bit PHY comparator status
+ */
+static void pmu_sweep_update_lane_state(__volatile__ u8 *entry, u32 eff_step,
+ u8 cur_val, u16 phy_sample)
+{
+ __volatile__ u8 *state_ptr = entry + 636U + eff_step;
+ __volatile__ u8 *rec = entry + eff_step * 3U;
+ u8 state = *state_ptr;
+
+ if (state == 1U) {
+ if (phy_sample != 0U) {
+ rec[1] = cur_val;
+ *state_ptr = 2U;
+ }
+ } else if (phy_sample == 0U) {
+ if ((state | 2U) == 2U) {
+ rec[state] = cur_val;
+ *state_ptr = state + 1U;
+ }
+ } else if (state == 3U) {
+ if ((u8)(cur_val - rec[2]) > (u8)(rec[1] - rec[0])) {
+ rec[0] = rec[2];
+ rec[1] = cur_val;
+ }
+ *state_ptr = 2U;
+ }
+}
+
+/**
+ * pmu_cal_slice_step_eval_sweep() - Execute slice delay step evaluation sweep
+ * @results_buf: Output buffer for per-slice/lane eye edge records
+ * @sweep_mode: Sweep mode selector
+ * @cal_mode: Calibration mode (0..3)
+ * @channel_rank: Active channel/rank selector
+ * @unused: Unused parameter
+ * @timing_val: Timing latch parameter
+ * @overflow_flag: Non-zero to enable coarse step wrapping
+ * @slice_delay_lut: 4-entry per-slice initial delay table
+ * @bound_flag: Non-zero for 9 lanes, 0 for 8 lanes
+ * @remap_flag: Non-zero to remap lane indices via 0x80000ba0
+ * @sweep_ctx: Context structure holding step parameters and valid masks at +0x1c
+ */
+void pmu_cal_slice_step_eval_sweep(void *results_buf, u32 sweep_mode,
+ u32 cal_mode, u32 channel_rank, u32 unused,
+ u32 timing_val, u32 overflow_flag,
+ const u16 *slice_delay_lut, u32 bound_flag,
+ u32 remap_flag, const void *sweep_ctx)
+{
+ u16 local_buf[40];
+ u16 zero_delays[4] = { 0 };
+ u32 loop_bound = (bound_flag != 0U) ? 9U : 8U;
+ u32 profile_mode = pmu_phy_query_operating_mode();
+ u32 flag_fp = 0U;
+ u32 active_csr_offset;
+ __volatile__ const u8 *ch_range = (__volatile__ const u8 *)PMU_DMEM_CH_RANGE;
+ u8 slice_start, slice_end, active_mask;
+ const u32 *ctx32;
+ u32 sec_ctrl_off, tert_ctrl_off, cmd_a_off, cmd_b_off;
+ u32 lane_start = 0U;
+ u32 step_flag_d, lane_csr_off, slice;
+ u16 f6a_val = 1U;
+ u32 max_sweep_steps, cal_mode_even, sweep_step;
+
+ (void)unused;
+ memset(local_buf, 0, 80U);
+
+ if (((u32)(sweep_mode - 3U) < 5U || sweep_mode == 1U) &&
+ pmu_stage_slice_flag_check() != 0U) {
+ u8 s_start = ch_range[0];
+ u8 s_end = ch_range[1];
+
+ flag_fp = 1U;
+ if (s_start <= s_end)
+ memset((u8 *)sweep_ctx + 0x1cU + s_start * 2U, 0,
+ (s_end - s_start + 1U) * 2U);
+ }
+
+ active_csr_offset = pmu_sdata()->active_csr_offset << 1;
+ slice_start = ch_range[0];
+ slice_end = ch_range[1];
+ active_mask = ch_range[0x30];
+
+ ctx32 = (const u32 *)sweep_ctx;
+ sec_ctrl_off = ctx32[1];
+ tert_ctrl_off = ctx32[2];
+ cmd_a_off = ctx32[3] << 1;
+ cmd_b_off = ctx32[4] << 1;
+
+ if (cal_mode == 3U) {
+ loop_bound = 10U;
+ lane_start = 9U;
+ }
+ step_flag_d = (cal_mode == 1U || flag_fp != 0U) ? 1U : 0U;
+
+ if (step_flag_d != 0U)
+ pmu_sweep_save_or_restore_lanes(local_buf, slice_start,
+ slice_end, active_mask,
+ active_csr_offset, cmd_b_off,
+ loop_bound, true);
+
+ lane_csr_off = (cal_mode == 3U) ? 0U : 0x1e00U;
+ for (slice = slice_start; slice <= slice_end; slice++) {
+ if ((active_mask & (1U << slice)) != 0U) {
+ u16 val = slice_delay_lut[slice];
+ u32 s_base = PHY_REG_DBYTE_BASE | (slice << 13) |
+ lane_csr_off | active_csr_offset;
+
+ *(__volatile__ u16 *)(s_base | cmd_a_off) = val;
+ if (step_flag_d != 0U)
+ *(__volatile__ u16 *)(s_base | cmd_b_off) = val;
+ }
+ }
+
+ pmu_phy_sec_cal_strobe_pulse();
+
+ if (channel_rank < 4U && (profile_mode & ~1U) == 2U) {
+ *(__volatile__ u16 *)(PHY_REG_DBYTE_BCAST_MODE |
+ active_csr_offset) =
+ (channel_rank >= 2U) ? 1U : 0U;
+ if (profile_mode == 3U) {
+ *(__volatile__ u16 *)PHY_REG_CBT_STAT0 = 8U;
+ *(__volatile__ u16 *)PHY_REG_CBT_STAT2 = 0U;
+ *(__volatile__ u16 *)PHY_REG_CBT_STAT1 =
+ (channel_rank > 1U) ? 8U : 0U;
+ f6a_val = 0x4001U;
+ }
+ }
+
+ for (slice = slice_start; slice <= slice_end; slice++) {
+ u16 *slice_dly_ptr = (u16 *)((u8 *)sweep_ctx + 0x1cU + slice * 2U);
+
+ *(__volatile__ u16 *)(PHY_REG_DBYTE_UPPER_GATE | (slice << 13)) =
+ (u16)((*slice_dly_ptr << 10) | f6a_val);
+ zero_delays[slice] = slice_delay_lut[slice];
+ }
+
+ pmu_phy_deskew_dly0_set(200U);
+ pmu_phy_deskew_dly1_set(*(const u16 *)((const u8 *)sweep_ctx + 0x14U));
+
+ max_sweep_steps = ((cal_mode | 1U) == 3U) ? 212U : 128U;
+ cal_mode_even = cal_mode & ~1U;
+
+ for (sweep_step = 0U; sweep_step < max_sweep_steps; sweep_step++) {
+ u32 eff_step = (sec_ctrl_off != 0U) ? sweep_step : 0U;
+ u8 cur_val = (sec_ctrl_off != 0U) ? (u8)*ctx32 : (u8)sweep_step;
+
+ pmu_cal_slice_timing_latch((u16)timing_val, 1U);
+
+ for (slice = slice_start; slice <= slice_end; slice++) {
+ u32 slice_base, s_base, lane;
+ u16 dly;
+ __volatile__ u8 *res_slice;
+
+ if ((active_mask & (1U << slice)) == 0U)
+ continue;
+
+ slice_base = PHY_REG_DBYTE_BASE | (slice << 13);
+ dly = zero_delays[slice];
+ if (cal_mode < 2U) {
+ dly += (u16)(tert_ctrl_off +
+ (((tert_ctrl_off + (dly & 0x7fU)) >
+ 0x7fU) ? 0x80U : 0U));
+ if (overflow_flag != 0U &&
+ (dly & 0x40U) != 0U) {
+ u16 *slice_dly_ptr = (u16 *)((u8 *)sweep_ctx +
+ 0x1cU + slice * 2U);
+ u16 sample_rdy_val = *(__volatile__ const u16 *)
+ (slice_base | 0x016aUL);
+ u16 inv = (u16)~(*slice_dly_ptr);
+
+ dly += 64U;
+ *slice_dly_ptr = inv;
+ *(__volatile__ u16 *)
+ (slice_base | 0x1f6aUL) =
+ (sample_rdy_val & 0xc3ffU) |
+ ((inv << 10) & 0x3c00U);
+ }
+ } else if (cal_mode_even == 2U) {
+ dly += (u16)tert_ctrl_off;
+ }
+ zero_delays[slice] = dly;
+
+ s_base = slice_base | 0x1e00U | active_csr_offset;
+ *(__volatile__ u16 *)(s_base | cmd_a_off) = dly;
+ if (step_flag_d != 0U)
+ *(__volatile__ u16 *)(s_base | cmd_b_off) = dly;
+
+ res_slice = (__volatile__ u8 *)results_buf +
+ slice * 8480U;
+ for (lane = lane_start; lane < loop_bound; lane++) {
+ u32 lane_idx = (remap_flag != 0U && lane < 8U) ?
+ ch_range[0x38U + slice * 8U + lane] :
+ lane;
+ u32 csr_field = (cal_mode == 3U) ?
+ 0x1000U : (lane << 9);
+ u16 phy_sample = *(__volatile__ const u16 *)
+ (slice_base | 0x016eUL | csr_field);
+
+ pmu_sweep_update_lane_state(res_slice +
+ lane_idx * 848U,
+ eff_step, cur_val,
+ phy_sample);
+ }
+ }
+
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_TRIG = 1023U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_TRIG = 0U;
+ }
+
+ if (step_flag_d != 0U)
+ pmu_sweep_save_or_restore_lanes(local_buf, slice_start,
+ slice_end, active_mask,
+ active_csr_offset, cmd_b_off,
+ loop_bound, false);
+
+ if ((profile_mode & ~1U) == 2U) {
+ *(__volatile__ u16 *)(PHY_REG_DBYTE_BCAST_MODE |
+ active_csr_offset) =
+ (profile_mode == 2U) ? 2U : 4U;
+ }
+}
+
+/**
+ * calc_step_val() - Compute reciprocal delay step scaling factor from combined tap count
+ * @combined: Measured tap span across consecutive edges
+ *
+ * Return: Reciprocal step value in Q17 format, or 0xffff if out of bounds.
+ */
+static inline u16 calc_step_val(u16 combined)
+{
+ if (*(__volatile__ const u16 *)PMU_DMEM_DRAM_FREQ_MHZ <= 199U ||
+ combined < 2U)
+ return 0xffffU;
+ return (u16)(((u32)(*(__volatile__ const u8 *)
+ PMU_DMEM_PLL_RATIO) << 17) / combined);
+}
+
+/**
+ * pmu_cal_slice_step_diff_commit() - Measure and commit per-slice step delay differences
+ * @slice_idx: Sub-slice index (0 or 1)
+ * @out_buf: Output 4-halfword slice delay buffer
+ * @cal_mode: Non-zero to write calibrated delays to PHY CSRs
+ */
+void pmu_cal_slice_step_diff_commit(u32 slice_idx, u16 *out_buf, u32 cal_mode)
+{
+ u16 temp_buf[4];
+ u32 slice_sub_idx = (1U << slice_idx) & 3U;
+ u8 slice_pair_mask = (u8)((1U << (slice_idx + 2U)) | (1U << slice_idx));
+ u8 start_slice = dmem_read8(PMU_DMEM_SLICE_START);
+ u8 end_slice = dmem_read8(PMU_DMEM_SLICE_END);
+ u32 pass;
+ u32 slice;
+
+ temp_buf[0] = 0U;
+ temp_buf[1] = 0U;
+ temp_buf[2] = 0U;
+ temp_buf[3] = 0U;
+
+ *(__volatile__ u16 *)PHY_REG_STROBE_CAL_EN = 1U;
+ pmu_cal_strobe_pulse_init();
+ pmu_cal_sequence_pulse_send(0U, 0x0bU, 0U, 0U, 0U, slice_sub_idx, 0U);
+
+ for (pass = 0U; pass < 2U; pass++) {
+ u16 *buf = (pass != 0U) ? temp_buf : out_buf;
+ u32 p_code = (pass != 0U) ? 0x28U : 0x25U;
+ u32 s_code = (pass != 0U) ? 0x83U : 0x81U;
+ u32 w_code = (pass != 0U) ? 0x26U : 0x23U;
+
+ pmu_cal_pad_timing_delay_set(0U, 0xffU, 0U);
+ pmu_cal_sequence_pulse_send(0U, 6U, 0x22U, 0x40U, p_code,
+ slice_pair_mask, 0U);
+ pmu_cal_sequence_pulse_send(0x80U, 0x19U, 4U, s_code, 0U,
+ slice_sub_idx, 0U);
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+ pmu_hw_timer_delay(0x800U);
+ pmu_delay_us(0x9c40U, 8U);
+ pmu_vref_signed_direction_pulse_dispatch(slice_pair_mask,
+ w_code);
+
+ for (slice = start_slice; slice <= end_slice; slice++) {
+ if (pmu_cal_lane_deskew_step_advance(slice_idx,
+ slice) != 0U) {
+ buf[slice] = (u16)pmu_dbyte_coarse_step_mask_sweep(slice, 4U,
+ slice_idx);
+ }
+ }
+
+ pmu_vref_signed_direction_pulse_dispatch(slice_pair_mask,
+ w_code + 1U);
+ for (slice = start_slice; slice <= end_slice; slice++) {
+ if (pmu_cal_lane_deskew_step_advance(slice_idx,
+ slice) != 0U) {
+ u16 val = (u16)pmu_dbyte_coarse_step_mask_sweep(slice, 4U,
+ slice_idx);
+ u16 combined = (u16)((val << 8) | buf[slice]);
+ u16 final_val = calc_step_val(combined);
+
+ buf[slice] = final_val;
+ }
+ }
+ pmu_cal_strobe_pulse_init();
+ }
+
+ for (slice = start_slice; slice <= end_slice; slice++) {
+ if (pmu_cal_lane_deskew_step_advance(slice_idx,
+ slice) == 0U) {
+ u32 neighbor = ((slice & 1U) != 0U) ?
+ (slice - 1U) : (slice + 1U);
+
+ out_buf[slice] = out_buf[neighbor];
+ temp_buf[slice] = temp_buf[neighbor];
+ }
+ }
+
+ if (cal_mode != 0U && end_slice >= start_slice) {
+ u32 csr_offset = pmu_sdata()->active_csr_offset;
+ u32 reg_off1 = (slice_idx == 0U) ? 0x1000cU : 0x1000dU;
+ u32 reg_off2 = (slice_idx == 0U) ? 0x10015U : 0x10016U;
+
+ for (slice = start_slice; slice <= end_slice; slice++) {
+ u32 slice_base = (slice << 12) | csr_offset;
+
+ *(__volatile__ u16 *)(PHY_REG_PUB_BASE |
+ ((slice_base | reg_off1) << 1)) =
+ out_buf[slice];
+ *(__volatile__ u16 *)(PHY_REG_PUB_BASE |
+ ((slice_base | reg_off2) << 1)) =
+ temp_buf[slice];
+ }
+ }
+
+ pmu_cal_strobe_pulse_init();
+ pmu_cal_sequence_pulse_send(0U, 0x0cU, 4U, 0U, 0U, slice_sub_idx, 0U);
+ pmu_cal_sequence_pulse_send(0U, 7U, 6U, 0U, 0U, 0U, 0xffU);
+ pmu_cal_sequence_pulse_send(0U, 0x10U, 4U, 0U, 0U, slice_sub_idx, 0U);
+ pmu_cal_sequence_pulse_send(0x80U, 7U, 6U, 0U, 0U, 0U, 0xffU);
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+ pmu_phy_sec_cal_strobe_pulse();
+}
+
+/**
+ * pmu_cal_slice_step_scan_eval() - Evaluate multiphase Vref and slice step pulse scan
+ * @step_val: Target delay step value
+ * @slice_mask: Active DBYTE slice bitmask
+ *
+ * Return: 0xff on completion.
+ */
+static u32 pmu_cal_slice_step_scan_eval(u32 step_val, u32 slice_mask)
+{
+ u32 vref_code = *(__volatile__ const u16 *)PHY_REG_VREF_STAT & 0x7fU;
+ u32 vref_scaled;
+ u32 vref_shifted;
+ s16 diff_bound;
+ u8 bound_low;
+ u8 vref_scaled_dec;
+ u8 bound_high;
+ u32 target_step = 10U;
+ int i;
+ u32 step_target;
+ s32 target_diff;
+ u32 post_param = 0x1280U;
+
+ if (vref_code == 0U)
+ vref_code = 1U;
+
+ vref_scaled = vref_code * 3U;
+ vref_shifted = vref_code << 2;
+ diff_bound = (s16)(vref_shifted + (u16)(vref_scaled ^ 0xffffU));
+ if (diff_bound < 0) {
+ u32 mult = 1U;
+ s16 not_scaled = (s16)~vref_scaled;
+
+ do {
+ mult++;
+ diff_bound = (s16)(vref_shifted * mult + not_scaled);
+ } while (diff_bound < 0);
+ }
+ bound_low = (u8)diff_bound;
+
+ pmu_cal_strobe_pulse_init();
+ pmu_cbt_cal_active_set();
+
+ vref_scaled_dec = (u8)(vref_code * 10U - 1U);
+ pmu_cal_sequence_pulse_send(0U, 7U, 0U, 0U, 0U, 0U, vref_scaled_dec);
+
+ bound_high = (u8)(vref_scaled + 1U);
+ pmu_multiphase_pulse_seq_repeat(7U, 0U, 0U, bound_high);
+
+ for (i = 0x1d; i <= 0x20; i++) {
+ pmu_multiphase_pulse_seq_repeat((u32)i, 0x12c8U,
+ (i == 0x20) ? 0U : slice_mask,
+ target_step);
+ }
+
+ pmu_multiphase_pulse_seq_repeat(7U, 0U, 0U, bound_low);
+ pmu_cal_sequence_pulse_send(0U, 7U, 0U, 0U, 0U, 0U, vref_scaled_dec);
+ pmu_cal_sequence_pulse_send(0U, 7U, 4U, 0U, 0U, 0U, (u8)(target_step - 1U));
+
+ for (i = 0; i < 16; i++) {
+ pmu_cal_pulse_burst_send((u32)((i & 1) ^ 1), vref_code,
+ slice_mask);
+ }
+
+ diff_bound = 1;
+ pmu_cal_sequence_pulse_send(0U, 7U, 0x5cU, 0U, 0U, 0U, 0U);
+ pmu_cal_sequence_pulse_send(0x40004U, 0x2eU, step_val, 0x1000U,
+ 0U, 0U, 0U);
+ pmu_cal_sequence_pulse_send(0x40004U, 0x2fU, step_val, 0x1000U,
+ 0U, 0U, 0U);
+ pmu_cal_sequence_pulse_send((1UL << 15), 7U, 4U, 0U, 0U, 0U, 1U);
+
+ step_target = (step_val * 2U) + 100U;
+ target_diff = (s32)vref_shifted - (s32)step_target;
+ while (target_diff < 0) {
+ diff_bound++;
+ target_diff = (s16)((vref_shifted * (u16)diff_bound) -
+ step_target);
+ }
+
+ pmu_cal_sequence_pulse_send(0U, 7U, 0U, 0U, 0U, 0U, vref_scaled_dec);
+ pmu_multiphase_pulse_seq_repeat(7U, 0U, 0U,
+ (u8)(target_diff + bound_high));
+
+ for (i = 0x1d; i <= 0x20; i++) {
+ pmu_multiphase_pulse_seq_repeat((u32)i, post_param,
+ (i == 0x20) ? 0U : slice_mask,
+ target_step);
+ }
+
+ pmu_multiphase_pulse_seq_repeat(7U, 0U, 0U, bound_low);
+ pmu_cal_sequence_pulse_send(0U, 7U, 0U, 0U, 0U, 0U, vref_scaled_dec);
+ pmu_cbt_cal_active_clear();
+
+ return 0xffU;
+}
+
+/**
+ * pmu_cal_slice_deskew_result_commit() - Sweep LCDL delays and commit slice deskew results
+ * @channel: Target memory channel index (0 or 1)
+ * @buf: Scratch/output halfword buffer for LCDL delay readback
+ * @flags: Non-zero to include DBI lane 8 and query calibration status
+ */
+void pmu_cal_slice_deskew_result_commit(u32 channel, void *buf, u32 flags)
+{
+ u32 csr_offset = pmu_sdata()->active_csr_offset;
+ u32 i;
+ u32 num_dly_regs = 8U;
+ u8 ch_mask = (u8)(5U << channel);
+ u8 start_slice;
+ u8 end_slice;
+ u32 slice;
+ u32 dly_idx;
+ u32 commit_flag = 0U;
+ u8 deskew_offset = 0U;
+ u32 cal_stat;
+ u32 iter;
+ u32 target_reg;
+
+ for (i = 0U; i < 9U; i++)
+ *(__volatile__ u16 *)(PHY_REG_DBYTE_BCAST_LANE +
+ i * 0x200UL) = (u16)(1U << i);
+
+ pmu_phy_pll_cfg_quad_write(0U);
+ pmu_cal_deskew_quad_tap_set(0xffU, 0x5555U, 0x5555U,
+ 0x5555U, 0x5555U);
+
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_RX_EN = 514U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_VREF = 0U;
+ *(__volatile__ u16 *)PHY_REG_AC_BCAST_GATE = 0U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_UPPER_SAMPLE_B = 1U;
+ *(__volatile__ u16 *)PHY_REG_AC_BCAST_PAT_CTRL = 0U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY2 = 0U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY3 = 0U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQS_DLY0 = 1U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQS_DLY1 = 1U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY0 = 0U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY1 = 2U;
+ *(__volatile__ u16 *)PHY_REG_BIST_CTRL_MODE = 5U;
+ *(__volatile__ u16 *)PHY_REG_AC_BCAST_PAT_DATA = 511U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY1 = 0U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_UPPER_COARSE = 1024U;
+
+ *(__volatile__ u16 *)(PHY_REG_DBYTE_BCAST_PULSE | (csr_offset << 1)) = 1U;
+ *(__volatile__ u16 *)(PHY_REG_DBYTE_BCAST_TX_CTRL | (csr_offset << 1)) = 2U;
+ pmu_phy_dbyte_cal_trig_pulse(0x0fU, 0x0fU);
+
+ if (flags != 0U) {
+ *(__volatile__ u16 *)PHY_REG_AC_BCAST_PAT_CTRL = 0U;
+ *(__volatile__ u16 *)PHY_REG_AC_BCAST =
+ *(__volatile__ const u16 *)PHY_REG_BIST_PLL_CFG | (1U << 4);
+ num_dly_regs = 9U;
+ }
+
+ *(__volatile__ u16 *)(PHY_REG_DBYTE_UPPER_DLY2 | (csr_offset << 1)) = 256U;
+ *(__volatile__ u16 *)(PHY_REG_DBYTE_UPPER_DLY3 | (csr_offset << 1)) = 256U;
+ *(__volatile__ u16 *)(PHY_REG_DBYTE_UPPER_DLY0 | (csr_offset << 1)) = 256U;
+ *(__volatile__ u16 *)(PHY_REG_DBYTE_UPPER_DLY1 | (csr_offset << 1)) = 256U;
+
+ pmu_phy_cal_trig_pulse();
+ pmu_phy_sec_cal_strobe_pulse();
+
+ pmu_channel_timing_deskew_reset(0x55U, 0x55U, 0U, 0U, ch_mask);
+ pmu_cbt_strobe_pulse_train(ch_mask, 0U);
+ pmu_cal_slice_step_diff_latch(0x60U, 1U);
+
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY2 = 511U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY3 = 511U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY0 =
+ *(__volatile__ const u16 *)PHY_REG_DBYTE_DQ_DLY0 | 0x30U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY1 = 1U;
+
+ start_slice = dmem_read8(PMU_DMEM_SLICE_START);
+ end_slice = dmem_read8(PMU_DMEM_SLICE_END);
+
+ for (slice = start_slice; slice <= end_slice; slice++) {
+ for (dly_idx = 0U; dly_idx < num_dly_regs; dly_idx++) {
+ __volatile__ u16 *reg = (__volatile__ u16 *)
+ (PHY_REG_DBYTE_COARSE | (slice << 13) |
+ (dly_idx << 9));
+ u16 val = *reg;
+ u32 adjusted = val + 0x20U;
+
+ if ((adjusted & 0x03c0U) != 0U)
+ adjusted = (val - 0x20U) + 1024U;
+
+ *reg = (u16)adjusted;
+ }
+ }
+
+ cal_stat = pmu_cal_pad_voltage_dac_set(flags, 0U);
+
+ if (cal_stat != 0U &&
+ *(__volatile__ const u8 *)PMU_DMEM_DESKEW_OFFSET != 0U) {
+ pmu_phy_dbyte_cfg_unpack();
+ deskew_offset = *(__volatile__ const u8 *)PMU_DMEM_DESKEW_OFFSET;
+ commit_flag = 1U;
+ }
+
+ pmu_channel_timing_deskew_reset(0x53U, 0xacU, deskew_offset,
+ deskew_offset, ch_mask);
+ pmu_cbt_strobe_pulse_train(ch_mask, 1U);
+ pmu_phy_pll_cfg_quad_write(0xffffU);
+ pmu_cal_deskew_quad_tap_set(0x0fU, 0xac53U, 0xac53U,
+ 0xac53U, 0xac53U);
+
+ if (cal_stat != 0U && deskew_offset != 0U) {
+ u32 bit;
+
+ for (bit = 0U; bit < 8U; bit++) {
+ u16 pattern = ((deskew_offset & (1U << bit)) != 0U) ?
+ 0x53acU : 0xac53U;
+
+ pmu_cal_deskew_quad_tap_set(bit, pattern, pattern,
+ pattern, pattern);
+ }
+ pmu_cal_deskew_quad_tap_set(8U, 0xac53U, 0xac53U,
+ 0xac53U, 0xac53U);
+ }
+
+ *(__volatile__ u16 *)PHY_REG_BIST_CTRL_MODE = 0U;
+ *(__volatile__ u16 *)PHY_REG_AC_BCAST_PAT_DATA = 0U;
+
+ for (iter = 0U; iter < 32U; iter++) {
+ pmu_cal_slice_step_diff_latch(8U, 1U);
+
+ for (slice = start_slice; slice <= end_slice; slice++) {
+ for (dly_idx = 0U; dly_idx < num_dly_regs; dly_idx++) {
+ u16 stat = *(__volatile__ const u16 *)
+ (PHY_REG_DBYTE_LCDL_STAT |
+ (slice << 13) | (dly_idx << 9));
+ u32 target_lane = dly_idx;
+ __volatile__ u16 *dly_reg;
+
+ if (commit_flag != 0U && dly_idx != 8U)
+ target_lane = *(__volatile__ const u8 *)
+ (PMU_DMEM_DBYTE_BACKUP +
+ slice * 8U + dly_idx);
+
+ dly_reg = (__volatile__ u16 *)
+ (PHY_REG_DBYTE_COARSE |
+ (slice << 13) | (target_lane << 9));
+ if (stat != 0U)
+ *dly_reg = (u16)(*dly_reg + 1024U);
+ }
+ }
+
+ pmu_phy_dbyte_cal_trig_pulse(0x0fU, 0x0fU);
+ }
+
+ if (commit_flag != 0U)
+ pmu_cal_pad_impedance_lock();
+
+ pmu_phy_lcdl_delay_read((u16 *)buf, 1U, 0U);
+
+ target_reg = pmu_bist_pattern_bit_get(channel, channel) + 0x24U;
+ pmu_slice_delay_step_program(target_reg, (const s16 *)buf, 1U, 0U, 0U, 0U);
+
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY0 = 0U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY1 = 0U;
+ pmu_phy_sec_cal_strobe_pulse();
+}
+
+/**
+ * fold_res() - Fold signed phase alignment residual into magnitude-plus-sign encoding
+ * @v: Signed phase residual value
+ *
+ * Return: Folded 16-bit residual value.
+ */
+static inline s16 fold_res(s16 v)
+{
+ return (v < 1) ? (s16)(-v) : (s16)(v | 8);
+}
+
+/**
+ * pmu_cal_wck_ck_align_pulse_send() - Perform 4-quadrant WCK-to-CK phase alignment sweep
+ * @ch: Target memory channel index (0 or 1)
+ * @rank: Target DRAM rank index (0 or 1)
+ */
+void pmu_cal_wck_ck_align_pulse_send(u32 ch, u32 rank)
+{
+ s16 buf1[4][15];
+ s16 buf2[4][15];
+ u16 buf3[4][15];
+ u16 buf4[4][15];
+ u16 step_cnt_b[4] = { 0 };
+ u16 step_cnt_a[4] = { 0 };
+ u16 saved_dly0[4] = { 0 };
+ u16 sample_stat[4];
+ s16 scan_result[4];
+ u32 csr_offset;
+ u32 vref_off;
+ u32 master_cfg_reg;
+ u16 saved_master_cfg;
+ u8 slice_start;
+ u8 dq_swap_mask;
+ u8 ch_mask;
+ u32 ch_rank_idx;
+ u32 test_bit;
+ u8 slice_idx_first;
+ u8 slice_idx_other;
+ u32 dq_swizzle_mask;
+ u32 step_flag_b = 0U;
+ u32 slice_stat_val;
+ u8 slice_end;
+ u32 s;
+ u32 iter;
+ s16 val_other;
+ s16 val_swiz;
+ u8 val_comb;
+
+ memset(buf1, 0, sizeof(buf1));
+ memset(buf2, 0, sizeof(buf2));
+ memset(buf3, 0, sizeof(buf3));
+ memset(buf4, 0, sizeof(buf4));
+
+ csr_offset = pmu_sdata()->active_csr_offset;
+ vref_off = csr_offset << 1;
+ master_cfg_reg = PHY_REG_MASTER_CFG | vref_off;
+
+ *(__volatile__ u16 *)(PHY_REG_VREF_TRIM_CFG | vref_off) = 1U;
+ *(__volatile__ u16 *)(PHY_REG_VREF_CTRL | vref_off) = 1U;
+
+ saved_master_cfg = *(__volatile__ const u16 *)master_cfg_reg;
+ *(__volatile__ u16 *)master_cfg_reg = saved_master_cfg | (1U << 8);
+
+ slice_start = dmem_read8(PMU_DMEM_SLICE_START);
+ dq_swap_mask = *(__volatile__ const u8 *)PMU_DMEM_LANE_ERR_MASK;
+
+ ch_mask = (u8)(1U << ch);
+ pmu_vref_signed_direction_pulse_dispatch(ch_mask, 5U);
+
+ ch_rank_idx = (ch << 1) + rank;
+ test_bit = dq_swap_mask & (1U << ch_rank_idx);
+ slice_idx_first = slice_start + (test_bit != 0U);
+ slice_idx_other = slice_start + (test_bit == 0U);
+
+ dq_swizzle_mask = pmu_dbyte_coarse_step_mask_sweep(slice_idx_first, 4U, ch);
+
+ if ((u8)dq_swizzle_mask == 1U) {
+ pmu_vref_signed_direction_pulse_dispatch(ch_mask, 6U);
+ if ((u8)pmu_dbyte_coarse_step_mask_sweep(dq_swizzle_mask,
+ 4U, ch) == 7U)
+ step_flag_b = 1U;
+ }
+
+ slice_stat_val = pmu_phy_ac_lane_timing_delay_set((ch_mask << 2) | ch_mask);
+ slice_end = dmem_read8(PMU_DMEM_SLICE_END);
+
+ for (s = slice_start; s <= slice_end; s++)
+ sample_stat[s] = (u16)step_flag_b;
+
+ for (iter = 0U; iter <= 14U; iter++) {
+ s8 tap = (iter < 8U) ? (s8)(7U - iter) : (s8)(iter + 1U);
+ s32 tap_val;
+ u32 target_step;
+
+ pmu_rank_slice_deskew_latch((u8)(((u8)tap << 4) |
+ ((u8)tap & 0x0fU)),
+ slice_stat_val);
+ pmu_phy_deskew_dly0_set(0xc8U);
+ pmu_phy_deskew_dly1_set((u16)slice_stat_val);
+ pmu_cal_slice_timing_latch(0U, 1U);
+ pmu_rank_slice_deskew_latch(0U, slice_stat_val);
+ pmu_vref_signed_direction_pulse_dispatch(slice_stat_val, 0x1aU);
+
+ tap_val = (iter < 8U) ? (s8)(-tap) : (s8)(tap & 7);
+
+ target_step = pmu_dbyte_coarse_step_mask_sweep(dq_swizzle_mask, 4U, ch);
+ if (pmu_lane_mask_bit_get(rank, ch) != 0U)
+ target_step |= (u8)pmu_dbyte_coarse_step_mask_sweep(slice_idx_other,
+ 4U, ch);
+
+ for (s = slice_start; s <= slice_end; s++) {
+ u32 val = target_step >> ((s == dq_swizzle_mask) ? 2U : 4U);
+ u16 bit0 = (u16)(val & 1U);
+ u16 bit1 = (u16)((val >> 1) & 1U);
+
+ buf4[s][iter] = bit1;
+ buf3[s][iter] = bit0;
+
+ if (bit0 != saved_dly0[s]) {
+ saved_dly0[s] = bit0;
+ buf1[s][step_cnt_b[s]++] = (s16)tap_val;
+ }
+
+ if (bit1 != sample_stat[s]) {
+ sample_stat[s] = bit1;
+ buf2[s][step_cnt_a[s]++] = (s16)tap_val;
+ }
+ }
+ }
+
+ for (s = slice_start; s <= slice_end; s++) {
+ u16 b0 = buf3[s][14];
+ u16 b1 = buf4[s][14];
+ s16 sum1;
+ s16 sum2;
+
+ if (step_cnt_b[s] == 0U)
+ buf1[s][0] = (b0 == 0U) ? 7 : -7;
+ else if (b0 == 0U && step_cnt_b[s] != 1U)
+ buf1[s][step_cnt_b[s]++] = 7;
+
+ if (step_cnt_a[s] == 0U)
+ buf2[s][0] = (b1 == 0U) ? 7 : -7;
+ else if (step_cnt_a[s] >= 2U &&
+ ((step_flag_b != 0U) == (b1 != 0U)))
+ buf2[s][step_cnt_a[s]++] = 7;
+
+ sum1 = (step_cnt_b[s] >= 2U) ?
+ (s16)(buf1[s][0] + buf1[s][step_cnt_b[s] - 1U]) / 2 :
+ buf1[s][0];
+ sum2 = (step_cnt_a[s] >= 2U) ?
+ (s16)(buf2[s][0] + buf2[s][step_cnt_a[s] - 1U]) / 2 :
+ buf2[s][0];
+ scan_result[s] = (s16)(sum1 + sum2) / 2;
+ }
+
+ val_other = scan_result[slice_idx_other];
+ val_swiz = scan_result[dq_swizzle_mask];
+
+ val_comb = (u8)(((fold_res(val_other) & 0x0f) << 4) |
+ (fold_res(val_swiz) & 0x0f));
+
+ pmu_rank_slice_deskew_latch(val_comb, slice_stat_val);
+
+ *(__volatile__ u8 *)(PMU_DMEM_ARCHIVED_CAL_STAT_ADDR + ch_rank_idx) = val_comb;
+ *(__volatile__ u8 *)(PMU_DMEM_WCK_PHASE + ch * 5U + rank * 10U) = val_comb;
+
+ *(__volatile__ u16 *)master_cfg_reg = saved_master_cfg;
+}
+
+static __always_inline void
+dly_line_tap_adjust_lane(u32 mode, u32 ptr_tag, u32 tag_valid,
+ u32 is_initial_mode, u32 csr_offset, u32 rank,
+ u32 slice, u8 step_val, u32 lane)
+{
+ u32 freq_pt = (slice << 12) | (lane << 8) | csr_offset | 0x10000U;
+ u32 tbl_offset = rank * 5280U + slice * 1320U + lane * 132U;
+ __volatile__ const u8 *tbl_ptr =
+ (__volatile__ const u8 *)(PMU_DMEM_LANE_PARAM_TBL_ADDR + tbl_offset);
+ u16 lane_param_val = *(const __volatile__ u16 *)tbl_ptr;
+ u8 cal_tap_status = *(__volatile__ const u8 *)
+ (PMU_DMEM_CAL_TAP_STAT + tbl_offset);
+ s8 lane_param_delta = (s8)tbl_ptr[2];
+ u8 cal_tap_base = tbl_ptr[3];
+ u32 tap_step_target = (u32)(lane_param_val + lane_param_delta);
+ u32 base_addr = (freq_pt | rank) << 1;
+ u32 lane_idx_lo = slice * 20U + lane * 2U + rank;
+ u32 coeff_idx = lane_idx_lo << 1;
+ u8 is_hi_freq = pmu_sdata()->high_freq_flag;
+
+ if (mode != 0U) {
+ u8 fine_dly_b[4] = { 0 };
+ u8 fine_dly_a[4] = { 0 };
+
+ if (ptr_tag == 0U && is_hi_freq != 0U) {
+ u16 cbt_coeff_val = *(__volatile__ const u16 *)
+ (PMU_DMEM_CBT_COEFF_A_ADDR + coeff_idx);
+
+ pmu_unpack_byte_pair(cbt_coeff_val, fine_dly_b);
+ } else {
+ u32 phy_addr = PHY_REG_AC_FINE_TIMING | base_addr;
+ u16 raw_phy = *(__volatile__ const u16 *)phy_addr;
+
+ pmu_cal_param_unpack(raw_phy, fine_dly_b);
+ }
+
+ pmu_unpack_byte_pair((u16)tap_step_target, fine_dly_a);
+ if (tag_valid == 0U) {
+ fine_dly_b[0] = fine_dly_a[0];
+ fine_dly_b[1] = fine_dly_a[1];
+ }
+
+ if (is_hi_freq != 0U) {
+ pmu_adjust_step_limit(fine_dly_b, ptr_tag);
+ pmu_adjust_step_limit(fine_dly_a, ptr_tag);
+ }
+
+ if ((dmem_read8(PMU_DMEM_CAL_MISC_FLAGS) & 0x20U) != 0U) {
+ int diff;
+
+ pmu_adjust_step_limit(fine_dly_b, step_val);
+ pmu_adjust_step_limit(fine_dly_a, step_val);
+
+ diff = (int)fine_dly_b[0] - (int)fine_dly_a[0];
+ if (diff > 1 || diff < -1) {
+ u32 phy_addr = PHY_REG_AC_FINE_TIMING | base_addr;
+ u32 step_adj = (step_val == 0U) ? 1U : 0U;
+ u16 raw_phy = *(__volatile__ const u16 *)phy_addr;
+
+ pmu_cal_param_unpack(raw_phy, fine_dly_b);
+ pmu_unpack_byte_pair((u16)tap_step_target,
+ fine_dly_a);
+ pmu_adjust_step_limit(fine_dly_b, step_adj);
+ pmu_adjust_step_limit(fine_dly_a, step_adj);
+ }
+ }
+
+ pmu_eye_margin_step_align(fine_dly_b, fine_dly_a,
+ (is_hi_freq != 0U) ? 2U : 1U);
+
+ if (ptr_tag == 0U) {
+ *(__volatile__ u16 *)(PMU_DMEM_CBT_COEFF_A_ADDR + coeff_idx) =
+ (u16)pmu_unpack_6(fine_dly_b);
+ *(__volatile__ u16 *)(PMU_DMEM_CBT_COEFF_B_ADDR + coeff_idx) =
+ (u16)tap_step_target;
+ *(__volatile__ u8 *)(PMU_DMEM_CBT_BASE_TBL_ADDR + lane_idx_lo) =
+ cal_tap_base;
+ *(__volatile__ u8 *)(PMU_DMEM_CBT_TRACK_STAT_ADDR + lane_idx_lo) =
+ cal_tap_status;
+ }
+
+ if (!(ptr_tag == 0U && is_hi_freq != 0U)) {
+ *(__volatile__ u16 *)(PHY_REG_AC_FINE_TIMING | base_addr) =
+ (u16)pmu_unpack_7(fine_dly_b);
+ }
+
+ *(__volatile__ u16 *)(PHY_REG_AC_COARSE_TIMING | base_addr) =
+ (u16)pmu_unpack_7(fine_dly_a);
+ } else {
+ u8 fine_dly_c[4] = { 0 };
+
+ if (ptr_tag == 0U) {
+ *(__volatile__ u8 *)(PMU_DMEM_CBT_LANE_FLAGS_ADDR + lane_idx_lo) =
+ cal_tap_base;
+ *(__volatile__ u16 *)(PMU_DMEM_CBT_COEFF_A_ADDR + coeff_idx) =
+ (u16)tap_step_target;
+ *(__volatile__ u8 *)(PMU_DMEM_CBT_INIT_STAT_ADDR + lane_idx_lo) =
+ cal_tap_status;
+ }
+
+ pmu_unpack_byte_pair((u16)tap_step_target, fine_dly_c);
+
+ if (is_hi_freq != 0U)
+ pmu_adjust_step_limit(fine_dly_c, ptr_tag);
+
+ if ((dmem_read8(PMU_DMEM_CAL_MISC_FLAGS) & 0x20U) != 0U)
+ pmu_adjust_step_limit(fine_dly_c, step_val);
+
+ *(__volatile__ u16 *)(PHY_REG_AC_FINE_TIMING | base_addr) =
+ (u16)pmu_unpack_7(fine_dly_c);
+ }
+
+ if (dmem_read8(PMU_DMEM_CAL_OVERRIDE_FLAG) == 0U) {
+ u32 num_t = (pmu_phy_query_operating_mode() == 3U) ? 2U : 1U;
+ u32 t;
+
+ for (t = 0U; t < num_t; t++) {
+ u8 val = (t != 0U) ? cal_tap_status : cal_tap_base;
+ u32 off = pmu_dly_reg_offset_select(mode, t, rank);
+
+ *(__volatile__ u16 *)
+ (PHY_REG_PUB_BASE | ((freq_pt | off) << 1)) = val;
+ if ((is_initial_mode | tag_valid) == 0U) {
+ off = pmu_dly_reg_offset_select(0U, t, rank);
+ *(__volatile__ u16 *)
+ (PHY_REG_PUB_BASE | ((freq_pt | off) << 1)) = val;
+ }
+ }
+ }
+}
+
+/**
+ * pmu_cal_dly_line_tap_step_adjust() - Adjust per-lane delay line taps across active slices
+ * @mode: Operating mode selector (0 = initial, non-zero = tracking)
+ * @ptr_tag: Pointer/stage tag selector
+ * @lane_mask: Active 9-bit lane mask
+ * @check_tag: Non-zero to validate @ptr_tag via pmu_stage_slice_flag_check
+ */
+static void pmu_cal_dly_line_tap_step_adjust(u32 mode, u32 ptr_tag, u32 lane_mask,
+ u32 check_tag)
+{
+ u32 tag_valid = (check_tag != 0U) ?
+ (pmu_stage_slice_flag_check() == 0U) :
+ 1U;
+ u32 is_initial_mode = (mode == 0U) ? 1U : 0U;
+ u32 csr_offset = pmu_sdata()->active_csr_offset;
+ u32 rank;
+
+ for (rank = 0U; rank < 2U; rank++) {
+ u8 rank_en = dmem_read8(PMU_DMEM_CH0_RANK_EN) |
+ dmem_read8(PMU_DMEM_CH1_RANK_EN);
+ u8 slice_start;
+ u8 slice_end;
+ u32 slice;
+
+ if ((rank_en & (1U << rank)) == 0U)
+ continue;
+
+ slice_start = dmem_read8(PMU_DMEM_SLICE_START);
+ slice_end = dmem_read8(PMU_DMEM_SLICE_END);
+ for (slice = slice_start; slice <= slice_end; slice++) {
+ u8 step_val;
+ u32 lane;
+
+ if ((dmem_read8(PMU_DMEM_ACTIVE_SLICE_MASK_CAL) &
+ (1U << slice)) == 0U)
+ continue;
+
+ step_val = pmu_sdata()->slice_timing_desc[rank * 4U + slice];
+
+ for (lane = 0U; lane < 9U; lane++) {
+ if ((lane_mask & (1U << lane)) == 0U)
+ continue;
+
+ dly_line_tap_adjust_lane(mode, ptr_tag,
+ tag_valid,
+ is_initial_mode,
+ csr_offset, rank,
+ slice, step_val, lane);
+ }
+ }
+ }
+}
+
+/**
+ * pmu_cal_dbyte_deskew_pin_results_apply() - Program per-pin deskew results to DBYTE CSRs
+ */
+static void pmu_cal_dbyte_deskew_pin_results_apply(void)
+{
+ uintptr_t lane_param_base = PMU_DMEM_LANE_PARAM_TBL_ADDR;
+ uintptr_t lane_step_base = PMU_DMEM_LANE_STEP_TBL_ADDR;
+ u32 csr_offset = pmu_sdata()->active_csr_offset;
+ u32 rank;
+
+ for (rank = 0U; rank < 2U; rank++) {
+ u8 mask = dmem_read8(PMU_DMEM_CH0_RANK_EN) |
+ dmem_read8(PMU_DMEM_CH1_RANK_EN);
+ u32 start_slice;
+ u32 end_slice;
+ uintptr_t slice_param_ptr;
+ uintptr_t slice_step_ptr;
+ u32 slice;
+
+ if ((mask & (1U << rank)) == 0U) {
+ lane_param_base += 5280U;
+ lane_step_base += 5280U;
+ continue;
+ }
+
+ start_slice = dmem_read8(PMU_DMEM_SLICE_START);
+ end_slice = dmem_read8(PMU_DMEM_SLICE_END);
+
+ slice_param_ptr = lane_param_base + start_slice * 1320U;
+ slice_step_ptr = lane_step_base + start_slice * 1320U;
+
+ for (slice = start_slice; slice <= end_slice; slice++) {
+ if ((dmem_read8(PMU_DMEM_ACTIVE_SLICE_MASK_CAL) &
+ (1U << slice)) != 0U) {
+ u32 slice_base = csr_offset | (slice << 12) |
+ rank;
+ uintptr_t ptr_a = slice_param_ptr;
+ uintptr_t ptr_b = slice_step_ptr;
+ u32 pin;
+
+ for (pin = 0U; pin < 10U; pin++) {
+ u16 val_a = *(__volatile__ const u16 *)ptr_a;
+ s8 val_b = *(__volatile__ const s8 *)ptr_b;
+ u16 sum = (u16)(val_a + (s32)val_b);
+
+ ptr_a += 132U;
+ ptr_b += 132U;
+
+ if (pin == 9U) {
+ uintptr_t reg =
+ (PHY_REG_DBYTE_BASE |
+ 0x0050UL) |
+ (slice_base << 1);
+
+ *(__volatile__ u16 *)reg = sum;
+ } else {
+ uintptr_t reg =
+ (PHY_REG_DBYTE_BASE |
+ 0x004cUL) |
+ ((slice_base |
+ (pin << 8)) << 1);
+
+ *(__volatile__ u16 *)reg = sum;
+ }
+ }
+
+ if (dmem_read8(PMU_DMEM_CAL_OVERRIDE) == 0U) {
+ uintptr_t tbl = PMU_DMEM_LANE_PARAM_TBL_ADDR +
+ rank * 5280U +
+ slice * 1320U;
+ u8 cal_step = *(__volatile__ const u8 *)
+ (tbl + 3U);
+ u32 is_hi = (slice >
+ dmem_read8(PMU_DMEM_DESKEW_DIR)) ?
+ 1U : 0U;
+ u32 swap;
+ u32 rank_idx;
+
+ pmu_phy_mode_cfg_dispatch(is_hi ?
+ 2U : 1U);
+ swap = pmu_cal_lane_error_status_check(rank,
+ slice);
+ rank_idx = rank + (is_hi ? 2U : 0U);
+
+ if (swap != 0U) {
+ dmem_write8(PMU_DMEM_DESKEW_SWAP + rank_idx,
+ cal_step);
+ pmu_cal_pulse_seq_coordinator(is_hi,
+ rank,
+ cal_step,
+ 1U);
+ dmem_write8(is_hi ?
+ (rank ? 0x51U : 0x50U) :
+ (rank ? 0x36U : 0x35U),
+ cal_step);
+ } else {
+ dmem_write8(PMU_DMEM_DESKEW_NORM + rank_idx,
+ cal_step);
+ pmu_cal_pulse_seq_coordinator(is_hi,
+ rank,
+ cal_step,
+ 2U);
+ dmem_write8(is_hi ?
+ (rank ? 0xdfU : 0xdaU) :
+ (rank ? 0xd5U : 0xd0U),
+ cal_step);
+ }
+ pmu_phy_mode_cfg_dispatch(3U);
+ }
+ }
+
+ slice_param_ptr += 1320U;
+ slice_step_ptr += 1320U;
+ }
+
+ lane_param_base += 5280U;
+ lane_step_base += 5280U;
+ }
+}
+
+/**
+ * pmu_cal_bist_cmd_strobe_dispatch() - Configure BIST command context and dispatch pulse sequence
+ * @ctx_addr: Address of struct pmu_cal_bist_cmd_ctx
+ * @pattern_mode: BIST pattern mode index
+ * @cmd_mode: Command mode (0..3)
+ * @target_mask: Target channel/rank mask
+ * @slice_flags_addr: Address of 4-element u16 slice flags array
+ * @status_idx: Status query index
+ * @setup_flags: Pattern setup flags
+ */
+__attribute__((optimize("no-jump-tables")))
+noinline void pmu_cal_bist_cmd_strobe_dispatch(uintptr_t ctx_addr,
+ u32 pattern_mode, u32 cmd_mode,
+ u32 target_mask,
+ uintptr_t slice_flags_addr,
+ u32 status_idx, u32 setup_flags)
+{
+ struct pmu_cal_bist_cmd_ctx *ctx =
+ (struct pmu_cal_bist_cmd_ctx *)ctx_addr;
+ u32 target_mask_reg = target_mask;
+ const u16 *slice_flags = (const u16 *)slice_flags_addr;
+ const __volatile__ u8 *ch_range;
+ u32 status;
+ u32 csr_offset;
+ u32 step_param;
+ u32 strobe_cmd;
+ u8 slice_start;
+ u8 slice_end;
+ u8 slice_mask;
+ u16 stride;
+ bool flag;
+ u32 k;
+ u32 slice;
+
+ status = pmu_cal_pad_voltage_dac_set(status_idx, cmd_mode);
+
+ if (pmu_sdata()->freq_mode <= 1U)
+ csr_offset = pmu_sdata()->active_csr_offset | 0x14U;
+ else
+ csr_offset = 0x56U;
+
+ ctx->ctrl0 = 0U;
+ ctx->ctrl1 = 0U;
+ ctx->ctrl2 = 1U;
+ ctx->csr_offset = csr_offset;
+
+ flag = (target_mask & 0x5U) != 0U;
+
+ if (cmd_mode == 3U) {
+ ctx->strobe_cmd_a = flag ? 0x28U : 0x29U;
+ } else if (cmd_mode == 1U) {
+ ctx->strobe_cmd_b = flag ? 16U : 17U;
+ ctx->strobe_cmd_a = flag ? 18U : 19U;
+ } else if (cmd_mode == 2U) {
+ ctx->strobe_cmd_a = flag ? 0x26U : 0x27U;
+ } else if (cmd_mode == 0U) {
+ ctx->strobe_cmd_b = flag ? 18U : 19U;
+ ctx->strobe_cmd_a = flag ? 16U : 17U;
+ } else {
+ pmu_assert_or_halt(0U, PMU_ERR_BIST_CMD_MODE_INVALID);
+ return;
+ }
+
+ phy_write16(PHY_REG_RESET_TRIGGER, 0x40U);
+ phy_write16(PHY_REG_DBYTE_BCAST_BASE | (csr_offset << 1), 1U);
+ phy_write16(PHY_REG_DBYTE_BCAST_RX_EN, 0U);
+
+ pmu_cal_bist_pattern_setup(pattern_mode, target_mask_reg, status,
+ setup_flags);
+ pmu_cal_marker_pair_set(800U);
+
+ step_param = (dmem_read8(PMU_DMEM_PLL_RATIO) == 2U) ? 4U : 2U;
+
+ if (pattern_mode < 9U && ((1U << pattern_mode) & 0x105U)) {
+ pmu_cal_sequence_pulse_send((1UL << 15), 7U, 0x18U, 0U, 0U, 0U, 0U);
+ pmu_cbt_cal_active_set();
+ pmu_cal_sequence_pulse_send(0U, 7U, 0U, 0U, 0U,
+ target_mask_reg, 0U);
+ pmu_cal_sequence_pulse_send(0x100000UL, 5U, 0U, 0x80U, 0U,
+ target_mask_reg, 0U);
+ pmu_cal_sequence_pulse_send((1UL << 23), 0x29U, step_param, 0U,
+ 0U, target_mask_reg, 7U);
+ pmu_cal_sequence_pulse_send(0U, 7U, pmu_sdata()->cbt_dly_init,
+ 0U, 0U, target_mask_reg, 0U);
+ strobe_cmd = 0x2aU;
+ } else {
+ pmu_cal_sequence_pulse_send((1UL << 15), 7U, 0x28U, 0U, 0U, 0U, 0U);
+ pmu_cbt_cal_active_set();
+ pmu_cal_sequence_pulse_send(0U, 7U, 8U, 0U, 0U,
+ target_mask_reg, 0U);
+ strobe_cmd = 0x2bU;
+ }
+
+ pmu_cal_sequence_pulse_send(0U, 7U, 0U, 0U, 0U, target_mask_reg, 0U);
+ pmu_cal_sequence_pulse_send(0x200000UL, 5U, 0U, 256U, 0U,
+ target_mask_reg, 0U);
+
+ pmu_cal_sequence_pulse_send(0x01040000UL, strobe_cmd, step_param, 0U, 0U,
+ target_mask_reg, 7U);
+ pmu_cal_sequence_pulse_send(0U, 7U, 4U, 0U, 0U, target_mask_reg, 10U);
+ pmu_cbt_cal_active_clear();
+ for (k = 0U; k < 2U; k++)
+ pmu_cal_sequence_pulse_send(0U, 7U, 0U, 0U, 0U, 0U, 0U);
+
+ stride = pmu_cal_stride_get();
+
+ ch_range = (const __volatile__ u8 *)PMU_DMEM_CH_RANGE;
+ phy_write16(PHY_REG_DBYTE_BCAST_DQ_DLY2, 1U);
+ phy_write16(PHY_REG_DBYTE_BCAST_DQ_DLY3, 1U);
+ phy_write16(PHY_REG_DBYTE_BCAST_DQ_DLY0,
+ (dmem_read8(PMU_DMEM_PLL_RATIO) == 4U) ? 0x34U : 0x30U);
+ ctx->stride_minus_1 = stride - 1U;
+ phy_write16(PHY_REG_DBYTE_BCAST_DQ_DLY1, 1U);
+
+ slice_start = ch_range[0];
+ slice_end = ch_range[1];
+ slice_mask = ch_range[0x30];
+
+ for (slice = slice_start; slice <= slice_end; slice++) {
+ u16 val = 0U;
+
+ if (!(slice_mask & (1U << slice)))
+ continue;
+
+ if (cmd_mode <= 1U)
+ val = (((slice_flags[slice] >> 7) & 1U) == cmd_mode) ?
+ 10U : 5U;
+
+ ctx->slice_delays[slice] = val;
+ }
+}
+
+/**
+ * pmu_dbyte_dq_commit_rank_steps() - Save eye tables and commit multi-rank DQ steps
+ * @channel_idx: Target channel index (0 or 1)
+ * @rank_idx: Target rank index (0 or 1)
+ * @cfg_mode_bit0: Bit 0 of the channel configuration mode byte
+ * @arg0_x5: Replicated rank mask ((mode_rank << 2) | mode_rank)
+ */
+static void pmu_dbyte_dq_commit_rank_steps(u32 channel_idx, u32 rank_idx,
+ u8 cfg_mode_bit0, u32 arg0_x5)
+{
+ u32 struct_base = PMU_DMEM_BASE | PMU_DMEM_CAL_STRUCT_BASE;
+ void *cal_struct = (void *)(uintptr_t)struct_base;
+ u32 not_cfg_mode_bit0 = !cfg_mode_bit0;
+ u32 table_addr;
+ u32 a3_mask;
+
+ pmu_cal_eye_table_save(cal_struct, 0x10U);
+ pmu_cal_lane_mask_or_all(struct_base, 0x10U, 0x41U);
+ pmu_cal_lane_mask_and_all(struct_base, 0x10U, 0xf3U);
+ pmu_cal_eye_table_save(cal_struct, 0x0bU);
+ pmu_cal_lane_mask_and_all(struct_base, 0x0bU, 0xf0U);
+ pmu_cal_eye_table_save(cal_struct, 0x12U);
+ pmu_cal_lane_mask_or_all(struct_base, 0x12U, 0x08U);
+ pmu_cal_lane_mask_and_all(struct_base, 0x12U, 0xf8U);
+ pmu_cal_marker_pair_set(1200U);
+ pmu_cbt_cal_pulse_seq(0U);
+ dmem_write16(PMU_DMEM_DESKEW_TARGET_DELAY, pmu_cal_stride_get() - 1U);
+ if (dmem_read8(PMU_DMEM_FREQ_MODE) != 0U) {
+ pmu_cal_pad_timing_step_settle();
+ pmu_delay_us(5000U, 12U);
+ pmu_clk_gate_handoff();
+ }
+ pmu_cal_lane_mask_or_all(struct_base, 0x0dU, 0x40U);
+
+ table_addr = struct_base +
+ (channel_idx * 2U + rank_idx) * PMU_SLICE_DESC_SIZE;
+ if (rank_idx == not_cfg_mode_bit0) {
+ u8 rank_mask;
+
+ pmu_cal_eye_table_save(cal_struct, 0x11U);
+ pmu_cal_lane_mask_or_all(struct_base, 0x11U, 0x08U);
+ pmu_cal_lane_mask_and_all(struct_base, 0x11U, 0xefU);
+ pmu_cal_lane_mask_or_all(struct_base, 0x11U, 0x20U);
+
+ rank_mask = (u8)((1U << (rank_idx + 2U)) | (1U << rank_idx));
+ pmu_cal_multi_rank_step_commit(rank_mask, table_addr, 0U,
+ 0xfffdffffUL, (u32)-1, 0U, 0U);
+ pmu_delay_us(2000U, 0U);
+ pmu_cal_eye_table_restore((u8 *)struct_base, 0x11U);
+ a3_mask = 0xfffbdff7UL;
+ } else {
+ a3_mask = 0xfff9d7f7UL;
+ }
+
+ pmu_cal_multi_rank_step_commit((u8)arg0_x5, table_addr, 0U,
+ a3_mask, (u32)-1, 0U, 0U);
+ pmu_cal_lane_mask_and_all(struct_base, 0x10U, 0xf3U);
+ pmu_cal_lane_mask_or_all(struct_base, 0x10U, 0x61U);
+ pmu_cal_lane_mask_and_all(struct_base, 0x10U, 0x7fU);
+}
+
+/**
+ * pmu_dbyte_dq_settle_rx_bias() - Settle DBYTE receiver bias and pad timing across slices
+ * @slice_list: Array of active slice indices for the rank
+ * @num_slices: Number of active slices in @slice_list
+ */
+static void pmu_dbyte_dq_settle_rx_bias(const u8 *slice_list, u8 num_slices)
+{
+ u32 iter, i;
+
+ if (dmem_read8(PMU_DMEM_FREQ_MODE) != 0U) {
+ for (i = 0U; i < num_slices; i++) {
+ u8 slice = slice_list[i];
+ u32 mask = pmu_cal_eye_margin_sample_latch(0x80U, slice);
+
+ phy_write16((PHY_REG_DBYTE_BASE | 0x0118UL) |
+ ((u32)slice << 13), (u16)mask);
+ }
+ pmu_delay_us(5000U, 12U);
+ pmu_phy_profile_param_program();
+ return;
+ }
+
+ for (iter = 0U; iter < 20U; iter++) {
+ for (i = 0U; i < num_slices; i++) {
+ u32 reg = (PHY_REG_DBYTE_BASE | 0x0118UL) |
+ ((u32)slice_list[i] << 13);
+
+ phy_write16(reg, 0x800U);
+ phy_write16(reg, 0U);
+ }
+ }
+
+ for (iter = 0U; iter < 40U; iter++) {
+ for (i = 0U; i < num_slices; i++) {
+ u8 slice = slice_list[i];
+ u32 reg = (PHY_REG_DBYTE_BASE | 0x0118UL) |
+ ((u32)slice << 13);
+ u32 mask = pmu_cal_eye_margin_sample_latch(0x80U, slice);
+
+ phy_write16(reg, (u16)(mask | 0x800U));
+ phy_write16(reg, (u16)(mask | 0x1000U));
+ }
+ }
+
+ for (i = 0U; i < num_slices; i++) {
+ u32 slice_off = (u32)slice_list[i] << 13;
+
+ phy_write16((PHY_REG_DBYTE_BASE | 0x0112UL) | slice_off, 511U);
+ phy_write16((PHY_REG_DBYTE_BASE | 0x0116UL) | slice_off, 511U);
+ phy_write16((PHY_REG_DBYTE_BASE | 0x0118UL) | slice_off,
+ (u16)pmu_cal_eye_margin_sample_latch(0x80U,
+ slice_list[i]));
+ }
+
+ pmu_clk_gate_handoff();
+ pmu_phy_profile_param_program();
+ pmu_cal_pad_timing_step_settle();
+ pmu_delay_us(5000U, 12U);
+ pmu_clk_gate_handoff();
+ pmu_delay_us(0x3d090UL, 0U);
+ pmu_cal_pad_timing_step_settle();
+}
+
+/**
+ * pmu_cal_dbyte_dq_status_check() - Configure DBYTE DQ pins and check training status
+ * @channel_idx: Target channel index (0 or 1)
+ * @mode_rank: Encoded mode and rank index
+ * @flag: Non-zero for full struct shadow and multi-rank step commit
+ *
+ * Return: @mode_rank unchanged.
+ */
+static noinline u32 pmu_cal_dbyte_dq_status_check(u32 channel_idx, u32 mode_rank,
+ u32 flag)
+{
+ u32 rank_idx = ((mode_rank & 3U) == 2U) ? 1U : 0U;
+ u32 csr_offset = dmem_read32(PMU_DMEM_ACTIVE_CSR_OFFSET);
+ u32 csr_offset_shl1 = csr_offset << 1;
+ u8 cfg_mode = dmem_read8(channel_idx ? 0x4dU : 0x32U);
+ u8 cfg_mode_bit0 = cfg_mode & 1U;
+ u16 master_cfg_val;
+ u8 *slice_list;
+ u8 num_slices;
+ u32 i, arg0_x5, p;
+ u8 slice_mr_ptr;
+
+ phy_write16(PHY_REG_VREF_CTRL | csr_offset_shl1, 1U);
+ phy_write16(PHY_REG_VREF_TRIM_CFG, 1U);
+ master_cfg_val = phy_read16(PHY_REG_MASTER_CFG | csr_offset_shl1);
+ phy_write16(PHY_REG_MASTER_CFG | csr_offset_shl1, master_cfg_val | 0x100U);
+
+ slice_list = (u8 *)(uintptr_t)((PMU_DMEM_BASE | PMU_DMEM_RANK_SLICE_MAP) +
+ channel_idx * 16U + rank_idx * 8U);
+ num_slices = dmem_read8(PMU_DMEM_RANK_SLICE_COUNTS +
+ channel_idx * 2U + rank_idx);
+
+ for (i = 0U; i < num_slices; i++) {
+ u32 slice_off = (u32)slice_list[i] << 13;
+
+ if (dmem_read8(PMU_DMEM_FREQ_MODE) != 0U) {
+ phy_write16((PHY_REG_DBYTE_BASE | 0x0112UL) | slice_off, 511U);
+ phy_write16((PHY_REG_DBYTE_BASE | 0x0116UL) | slice_off, 511U);
+ phy_write16((PHY_REG_DBYTE_BASE | 0x0118UL) | slice_off, 0U);
+ } else {
+ phy_write16((PHY_REG_DBYTE_BASE | 0x0112UL) | slice_off, 0x19ffU);
+ phy_write16((PHY_REG_DBYTE_BASE | 0x0116UL) | slice_off, 0x19ffU);
+ phy_write16((PHY_REG_DBYTE_BASE | 0x0118UL) | slice_off, 0x1000U);
+ }
+ }
+
+ arg0_x5 = (mode_rank << 2) | mode_rank;
+
+ if (flag != 0U) {
+ pmu_dbyte_dq_commit_rank_steps(channel_idx, rank_idx,
+ cfg_mode_bit0, arg0_x5);
+ } else {
+ u32 struct_base = PMU_DMEM_BASE | PMU_DMEM_CAL_STRUCT_BASE;
+
+ pmu_cal_lane_mask_and_all(struct_base, 0x10U, 0xf3U);
+ pmu_cal_lane_mask_or_all(struct_base, 0x10U, 0x61U);
+ pmu_cal_lane_mask_or_all(struct_base, 0x10U, 0x80U);
+ }
+
+ phy_write16(PHY_REG_PLL_CLK_CTRL, 1U);
+ if (cfg_mode_bit0 == rank_idx) {
+ u32 table_addr = (PMU_DMEM_BASE | PMU_DMEM_CAL_STRUCT_BASE) +
+ (channel_idx * 2U + rank_idx) *
+ PMU_SLICE_DESC_SIZE;
+
+ pmu_cbt_3phase_pulse_dispatch((u8)arg0_x5, (const u8 *)table_addr, 16U, 1U);
+ } else {
+ u32 dq_offset_sub = ((arg0_x5 & 3U) == 2U) ?
+ PMU_SLICE_DESC_SIZE : 0U;
+ u8 *tbl = (u8 *)(uintptr_t)((PMU_DMEM_BASE | PMU_DMEM_CAL_STRUCT_BASE) +
+ channel_idx * (2U * PMU_SLICE_DESC_SIZE) +
+ dq_offset_sub);
+ u8 mr16_cfg = tbl[0x10];
+ u8 dq_pin_mask = (u8)((1U << (cfg_mode_bit0 + 2U)) | (1U << cfg_mode_bit0));
+
+ pmu_cal_strobe_pulse_init();
+ pmu_ac_lane_profile_setup();
+ pmu_cal_margin_dir_get();
+ pmu_dbyte_cal_step_latch(0x10U, mr16_cfg, (u8)arg0_x5);
+ pmu_cal_pad_timing_delay_set(0U, 4U, 1U);
+ pmu_dbyte_cal_step_latch(0x10U, (mr16_cfg & 138U) | 69U, dq_pin_mask);
+ pmu_cal_pad_timing_delay_set(0U, 0x20U, 1U);
+ pmu_cbt_cal_pulse_seq(0U);
+ pmu_cbt_3phase_pulse_seq();
+ pmu_cal_slice_step_diff_latch(0U, 0U);
+ pmu_delay_us(0U, 0x32U);
+ }
+
+ pmu_dbyte_dq_settle_rx_bias(slice_list, num_slices);
+
+ slice_mr_ptr = dmem_read8(PMU_DMEM_CAL_STRUCT_BASE +
+ (channel_idx * 2U + rank_idx) *
+ PMU_SLICE_DESC_SIZE + 0xcU);
+ p = (slice_mr_ptr | 0x80U) & 0xffU;
+
+ for (i = 0U; i < num_slices; i++) {
+ u8 slice = slice_list[i];
+ u32 slice_off = (u32)slice << 13;
+ u32 mask = pmu_cal_eye_margin_sample_latch(p, slice);
+
+ phy_write16((PHY_REG_DBYTE_BASE | 0x0118UL) | slice_off,
+ (u16)(mask & 0xffU));
+ }
+ pmu_delay_us(5000U, 12U);
+
+ for (i = 0U; i < num_slices; i++) {
+ u8 slice = slice_list[i];
+ u32 slice_off = (u32)slice << 13;
+ u32 mask = pmu_cal_eye_margin_sample_latch(p, slice);
+
+ phy_write16((PHY_REG_DBYTE_BASE | 0x0118UL) | slice_off,
+ (u16)((mask & 0xffU) | 0x100U));
+ }
+ pmu_delay_us(5000U, 12U);
+
+ for (i = 0U; i < num_slices; i++) {
+ u32 slice_off = (u32)slice_list[i] << 13;
+ u32 mask = pmu_cal_eye_margin_sample_latch(0x80U, slice_list[i]) | 0x100U;
+
+ phy_write16((PHY_REG_DBYTE_BASE | 0x0112UL) | slice_off, (u16)mask);
+ phy_write16((PHY_REG_DBYTE_BASE | 0x0116UL) | slice_off, (u16)mask);
+ }
+
+ pmu_delay_us(0x3d090UL, 0U);
+ pmu_cal_eye_table_restore((u8 *)(PMU_DMEM_BASE | PMU_DMEM_CAL_STRUCT_BASE), 0x10U);
+ pmu_clk_gate_handoff();
+
+ return mode_rank;
+}
+
+/**
+ * pmu_cal_dbyte_pattern_loop_exec() - Compute slice delay bounds and unpack pattern taps
+ * @reg_offset: Channel/register offset index
+ * @pattern_mode: Pattern mode (0..3)
+ * @dest_buf: Output 4-element u16 array of decoded 15-bit pattern addresses
+ * @scratch_buf: Output 4-element u16 array of raw slice delay bounds
+ * @bound_start: Step adjust flag for high-frequency mode
+ * @bound_end: Non-zero to average min/max across active lanes
+ * @flags: Base delay offset selector (0 = 0x40, non-zero = 0x80)
+ */
+noinline void pmu_cal_dbyte_pattern_loop_exec(u32 reg_offset, u16 pattern_mode,
+ void *dest_buf, void *scratch_buf,
+ u32 bound_start, u32 bound_end,
+ u32 flags)
+{
+ u16 *out_dest = (u16 *)dest_buf;
+ u16 *out_scratch = (u16 *)scratch_buf;
+ u8 slice_start = dmem_read8(PMU_DMEM_SLICE_START);
+ u8 slice_end = dmem_read8(PMU_DMEM_SLICE_END);
+ u8 active_mask = dmem_read8(PMU_DMEM_ACTIVE_SLICE_MASK_CAL);
+ u16 freq;
+ u32 slice;
+
+ if (pattern_mode == 2U || pattern_mode == 3U) {
+ u16 local_buf[4] = { 0 };
+ u32 csr_offset;
+
+ pmu_cal_slice_step_diff_commit(reg_offset, local_buf, 1U);
+ csr_offset = pmu_sdata()->active_csr_offset;
+ for (slice = slice_start; slice <= slice_end; slice++) {
+ u32 addr = PHY_REG_DBYTE_BASE |
+ (((slice << 12) | csr_offset | (reg_offset + 0x2aU)) << 1);
+ u16 reg = phy_read16(addr);
+ u16 sum = (reg & 0x3fU) + local_buf[slice];
+ u16 val0 = sum + 0x20U;
+ u16 res = 0U;
+
+ if ((val0 & ~1U) >= 0x6aU)
+ res = sum - 0x4aU;
+ out_scratch[slice] = res;
+ out_dest[slice] = res;
+ }
+ return;
+ }
+
+ if (pattern_mode >= 2U) {
+ pmu_assert_or_halt(0U, PMU_ERR_BIST_PATTERN_MODE_RANGE);
+ return;
+ }
+
+ freq = dmem_read16(PMU_DMEM_DRAM_FREQ_OFF);
+ if (freq <= 1599U) {
+ u16 init_val = (pmu_sdata()->freq_mode < 2U) ? 0x41U : 0xc1U;
+
+ for (slice = slice_start; slice <= slice_end; slice++) {
+ if (active_mask & (1U << slice))
+ out_scratch[slice] = init_val;
+ }
+ } else {
+ u32 base_reg = (pattern_mode != 0U) ?
+ (PHY_REG_DBYTE_BASE | 0x0024UL) :
+ (PHY_REG_DBYTE_BASE | 0x0020UL);
+ u8 base_fp = (flags == 0U) ? 0x40U : 0x80U;
+ u32 num_lanes = (pmu_dmem_training_flags_eval() != 0U) ? 9U : 8U;
+ u32 csr_offset = pmu_sdata()->active_csr_offset;
+
+ for (slice = slice_start; slice <= slice_end; slice++) {
+ u32 dmem_addr = 0x12eU;
+ u8 sub_val;
+
+ if (!(active_mask & (1U << slice)))
+ continue;
+
+ if (bound_end != 0U) {
+ u16 min_val = 0xffffU;
+ u16 max_val = 0U;
+ u32 slice_base = (slice << 12) | csr_offset |
+ reg_offset;
+ u32 lane;
+
+ for (lane = 0U; lane < num_lanes; lane++) {
+ u32 reg_addr = base_reg |
+ ((slice_base | (lane << 8)) << 1);
+ u16 reg_val = phy_read16(reg_addr);
+ u16 val = (reg_val & 0x7fU) +
+ ((reg_val & ~0x7fU) >> 1);
+
+ if (val < min_val)
+ min_val = val;
+ if (val > max_val)
+ max_val = val;
+ }
+ out_scratch[slice] = (min_val + max_val) >> 1;
+ } else {
+ out_scratch[slice] =
+ pmu_phy_delay_bound_margin_log(slice);
+ }
+
+ if (reg_offset == 0U) {
+ if (slice <= 3U)
+ dmem_addr = 0x127U + slice;
+ } else if (reg_offset == 1U) {
+ if (slice <= 2U)
+ dmem_addr = 0x12bU + slice;
+ }
+
+ sub_val = base_fp + dmem_read8(dmem_addr);
+ if (dmem_read8(PMU_DMEM_CAL_MISC_FLAGS) & 0x20U) {
+ u8 pattern_bytes[4];
+ u32 dev;
+ u8 flag;
+
+ pmu_unpack_byte_pair(out_scratch[slice],
+ pattern_bytes);
+ dev = pmu_midpoint_deviation_calc(pattern_bytes[1]);
+ flag = ((dev < 0x21U) ^
+ (pattern_bytes[0] & 1U)) ? 1U : 0U;
+ pmu_sdata()->slice_timing_desc[slice +
+ (reg_offset << 2)] =
+ flag;
+ }
+
+ if (out_scratch[slice] < sub_val)
+ out_scratch[slice] = 0U;
+ else
+ out_scratch[slice] -= sub_val;
+ }
+ }
+
+ for (slice = slice_start; slice <= slice_end; slice++) {
+ u8 pattern_bytes[4];
+
+ if (!(active_mask & (1U << slice)))
+ continue;
+
+ pmu_unpack_byte_pair(out_scratch[slice], pattern_bytes);
+
+ if (pmu_sdata()->high_freq_flag != 0U)
+ pmu_adjust_step_limit(pattern_bytes, bound_start);
+
+ if (dmem_read8(PMU_DMEM_CAL_MISC_FLAGS) & 0x20U)
+ pmu_adjust_step_limit(pattern_bytes,
+ pmu_sdata()->slice_timing_desc[slice +
+ (reg_offset << 2)]);
+
+ out_dest[slice] = (u16)pmu_unpack_7(pattern_bytes);
+ }
+}
+
+/**
+ * pmu_slice_reg_pulse_write() - Write a pulse control value across a list of DBYTE slices
+ * @ptr: Array of target DBYTE slice indices
+ * @count: Number of slice indices in @ptr
+ * @val: 16-bit value to write to register offset 0x0118 of each slice
+ */
+static inline void pmu_slice_reg_pulse_write(const u8 *ptr, u32 count, u16 val)
+{
+ while (count--)
+ phy_write16((PHY_REG_DBYTE_BASE | 0x0118UL) |
+ ((u32)(*ptr++) << 13), val);
+}
+
+/**
+ * pmu_cal_multi_rank_deskew_sweep() - Execute multi-rank deskew pulse train and step commit
+ * @channel: Target channel index (0 or 1)
+ * @mode: Encoded rank/mode selector
+ */
+static noinline void pmu_cal_multi_rank_deskew_sweep(u32 channel, u32 mode)
+{
+ u32 struct_base =
+ PMU_DMEM_BASE | PMU_DMEM_CAL_STRUCT_BASE;
+ const u8 *ptr =
+ (const u8 *)(PMU_DMEM_BASE | PMU_DMEM_RANK_SLICE_COUNTS);
+ u32 count;
+ u8 cal_cfg;
+ u32 mode_flag = ((mode & 3U) == 2U) ? 1U : 0U;
+ u32 idx;
+ u32 csr_offset;
+ u32 reg;
+
+ cal_cfg = dmem_read8(channel ? PMU_DMEM_CH1_RANK_CFG :
+ PMU_DMEM_CH0_RANK_CFG);
+ if (mode_flag != (cal_cfg & 1U))
+ pmu_slice_deskew_pulse_train(channel, (u32)(cal_cfg & 1U), 1U);
+
+ if (pmu_sdata()->freq_mode != 0U) {
+ pmu_ac_lane_profile_setup();
+ pmu_delay_us(50000U, 3U);
+ }
+
+ pmu_cal_eye_table_restore((u8 *)struct_base, 0x0bU);
+ pmu_cal_eye_table_restore((u8 *)struct_base, 0x11U);
+ pmu_cal_eye_table_restore((u8 *)struct_base, 0x12U);
+
+ idx = mode_flag + (channel * 2U);
+ count = ptr[idx];
+ ptr += 0x24 + (idx << 3);
+
+ pmu_cal_pad_timing_step_settle();
+
+ pmu_slice_reg_pulse_write(ptr, count, 256U);
+
+ pmu_delay_us(5000U, 12U);
+ pmu_clk_gate_handoff();
+
+ csr_offset = pmu_sdata()->active_csr_offset;
+ phy_write16(PHY_REG_VREF_CTRL | (csr_offset << 1), 0U);
+ pmu_delay_us(5000U, 0U);
+
+ pmu_cal_eye_table_save((void *)(uintptr_t)(struct_base), 0x10U);
+ pmu_cal_lane_mask_and_all(struct_base, 0x10U, 0xcfU);
+ pmu_cal_lane_mask_and_all(struct_base, 0x10U, 0xffU);
+
+ pmu_cal_multi_rank_step_commit(mode,
+ struct_base + (idx * PMU_SLICE_DESC_SIZE),
+ 0U, 0xfffeffffUL, (u32)-1, 1U, 0U);
+
+ pmu_delay_us(14000U, 10U);
+
+ pmu_slice_reg_pulse_write(ptr, count, 0x1000U);
+
+ pmu_cal_eye_table_restore((u8 *)struct_base, 0x10U);
+ reg = PHY_REG_MASTER_CFG | (pmu_sdata()->active_csr_offset << 1);
+ phy_write16(reg, phy_read16(reg) & 0xfeffU);
+ pmu_hw_timer_delay(0x14U);
+}
+
+/**
+ * pmu_cal_channel_rank_boundary_eval() - Evaluate channel/rank boundary delays and unpack CSRs
+ * @buf: 584-byte work buffer receiving slice bounds and diagnostic CSR snapshots
+ * @channel: Target channel index (0 or 1)
+ * @rank: Target rank index
+ * @cal_type: Calibration stage type (0x11 or 0x12)
+ */
+static noinline void pmu_cal_channel_rank_boundary_eval(u8 *buf, u32 channel, u32 rank,
+ u32 cal_type)
+{
+ u8 *ctrl = buf + 512;
+ u32 lut_val;
+ u8 target_step;
+ u8 rank_start;
+ u8 rank_end;
+ u32 active_csr_offset;
+ u32 i;
+
+ lut_val = pmu_bist_pattern_bit_get(channel, channel);
+ memset(buf, 0, 526U);
+
+ target_step = (rank != 0U) ? 9U : 8U;
+ rank_start = dmem_read8(PMU_DMEM_SLICE_START);
+ rank_end = dmem_read8(PMU_DMEM_SLICE_END);
+ active_csr_offset = pmu_sdata()->active_csr_offset;
+
+ if (cal_type == 0x11U) {
+ u8 feat_mask = dmem_read8(PMU_DMEM_TRAIN_FEATURE_MASK);
+
+ ctrl[12] = 0U;
+ *(u16 *)&ctrl[8] = 0U;
+ *(u16 *)&ctrl[6] = 1U;
+ ctrl[10] = target_step;
+ *(u16 *)&ctrl[2] = 0x62U;
+ *(u16 *)&ctrl[4] = 0x64U;
+ pmu_phy_dbyte_cfg_unpack();
+
+ ctrl[13] = 1U;
+ pmu_cal_dbyte_pattern_loop_exec(channel, 1U, buf, buf + 16,
+ 0U, 1U, 0U);
+ pmu_cal_dbyte_pattern_loop_exec(channel, 0U, buf + 8,
+ buf + 24, 0U, 1U, 0U);
+
+ for (i = rank_start; i <= rank_end; i++) {
+ u16 *p = (u16 *)buf + i;
+ u32 accum_lo = 0U;
+ u32 accum_hi = 0U;
+ u8 tl;
+ u8 tu;
+
+ pmu_cal_param_unpack(p[0], (u8 *)&accum_lo);
+ pmu_cal_param_unpack(p[4], (u8 *)&accum_hi);
+ tl = (u8)accum_lo;
+ tu = (u8)accum_hi;
+ if (tl > tu) {
+ pmu_adjust_step_limit((u8 *)&accum_lo, !(tu & 1U));
+ p[0] = (u16)pmu_unpack_7((u8 *)&accum_lo);
+ } else if (tl < tu) {
+ pmu_adjust_step_limit((u8 *)&accum_hi, !(tl & 1U));
+ p[4] = (u16)pmu_unpack_7((u8 *)&accum_hi);
+ }
+ }
+
+ buf[523] = (dmem_read8(PMU_DMEM_BIST_PATTERN_LO) ||
+ dmem_read8(PMU_DMEM_DESKEW_OFFSET_OVERRIDE)) ?
+ 7U : 4U;
+
+ if (feat_mask & 2U) {
+ u8 ret = (u8)pmu_phy_query_operating_mode();
+
+ ctrl[0] = ret;
+ if (ret == 3U)
+ phy_write16(PHY_REG_DBYTE_BCAST_MODE |
+ (active_csr_offset << 1), 0U);
+ }
+
+ for (i = rank_start; i <= rank_end; i++) {
+ u32 j;
+
+ for (j = 0U; j < 9U; j++) {
+ u32 offset = active_csr_offset | (i << 12) |
+ (j << 8);
+ u16 val1 = phy_read16(PHY_REG_DBYTE_BASE |
+ (((lut_val + 0x12U) | offset) << 1));
+ u16 val2 = phy_read16(PHY_REG_DBYTE_BASE |
+ (((lut_val + 0x10U) | offset) << 1));
+
+ *(u16 *)(buf + 0x20 + (i * 20U) + (j * 2U)) = val1;
+ *(u16 *)(buf + 112 + (i * 20U) + (j * 2U)) = val2;
+
+ if ((feat_mask & 2U) && ctrl[0] == 3U) {
+ u32 csr_base = (PHY_REG_DBYTE_BASE | 0x009cUL) |
+ (offset << 1);
+ u16 reg0 = phy_read16(csr_base);
+ u16 reg1 = phy_read16(csr_base + 2U);
+ u16 reg2 = phy_read16(csr_base + 4U);
+ u16 reg3 = phy_read16(csr_base + 6U);
+ u16 *diag = (u16 *)(buf + 192 + (i * 80U) + (j * 8U));
+
+ diag[0] = reg0;
+ diag[1] = reg1;
+ diag[2] = reg2;
+ diag[3] = reg3;
+
+ phy_write16(csr_base, (reg0 + reg2) >> 1);
+ phy_write16(csr_base + 2U, (reg1 + reg3) >> 1);
+ }
+ }
+ }
+ } else {
+ u8 nib;
+
+ ctrl[11] = 2U;
+ *(u16 *)&ctrl[6] = 2U;
+ *(u16 *)&ctrl[2] = 0x5eU;
+ *(u16 *)&ctrl[4] = 0x60U;
+ *(u16 *)&ctrl[8] = 0xffU;
+
+ nib = (u8)pmu_mr22_bit4_check();
+ ctrl[10] = nib ? 10U : target_step;
+ ctrl[12] = nib;
+
+ pmu_cal_dbyte_pattern_loop_exec(channel, 2U, buf, buf + 16,
+ 0U, 1U, 0U);
+
+ for (i = rank_start; i <= rank_end; i++) {
+ u32 base_off = active_csr_offset | (i << 12);
+ u32 j;
+ u16 csr_val;
+
+ for (j = 0U; j < 9U; j++) {
+ u32 offset = base_off | (lut_val + 0x26U) |
+ (j << 8);
+
+ csr_val = phy_read16(PHY_REG_DBYTE_BASE |
+ (offset << 1));
+
+ *(u16 *)(buf + 0x20 + (i * 20U) + (j * 2U)) = csr_val;
+ }
+
+ csr_val = phy_read16(PHY_REG_DBYTE_BASE |
+ ((base_off | (lut_val + 0x28U)) << 1));
+ *(u16 *)(buf + 0x20 + (i * 20U) + (9U * 2U)) = csr_val;
+ }
+ }
+}
+
+/**
+ * pmu_cal_rank_param_commit() - Coordinate matrix scan, window check, and tap commit
+ * @freq_pt: Frequency point / channel mask
+ * @mode_cfg: Mode configuration (0, 4, or 7)
+ * @step_lim: Lane mask / step limit
+ * @type: Calibration type (0, 1, or 2)
+ */
+void pmu_cal_rank_param_commit(u32 freq_pt, u32 mode_cfg, u32 step_lim, u32 type)
+{
+ u8 ch0_ctrl_lo;
+ u8 ch0_ctrl_hi;
+ u8 cal_ctrl_hi;
+ u8 cal_mode_ext;
+ u32 step_flag_a;
+ u32 step_flag_b;
+ u32 step_bound_val;
+ u32 step_flag_e;
+ u32 step_flag_c;
+ u32 lane_cnt;
+ u32 i;
+
+ ch0_ctrl_lo = dmem_read8(PMU_DMEM_CH0_CTRL_LO);
+ ch0_ctrl_hi = dmem_read8(PMU_DMEM_CH0_CTRL_HI);
+ if (((ch0_ctrl_lo & 0x1fU) | (ch0_ctrl_hi & 0x1fU)) == 0U) {
+ dmem_write8(PMU_DMEM_CH0_CTRL_LO, 1U);
+ dmem_write8(PMU_DMEM_CH0_CTRL_HI, 1U);
+ }
+
+ cal_ctrl_hi = dmem_read8(PMU_DMEM_CAL_CTRL_HI);
+ cal_mode_ext = dmem_read8(PMU_DMEM_CAL_MODE_EXT);
+ if (((cal_ctrl_hi & 0x1fU) | (cal_mode_ext & 0x1fU)) == 0U) {
+ dmem_write8(PMU_DMEM_CAL_CTRL_HI, 1U);
+ dmem_write8(PMU_DMEM_CAL_MODE_EXT, 1U);
+ }
+
+ step_flag_a = (type != 1U) ? 1U : 0U;
+ step_flag_b = (type <= 1U) ? 1U : 0U;
+ step_bound_val = (type == 2U) ? 1U : 0U;
+ step_flag_e = (mode_cfg != 0U) ? 1U : 0U;
+ step_flag_c = (dmem_read8(PMU_DMEM_CAL_OVERRIDE_FLAG) == 0U) ?
+ dmem_read8(PMU_DMEM_DQ_SWAP_BASE) : 0U;
+ lane_cnt = (dmem_read8(PMU_DMEM_HIGH_FREQ_FLAG) != 0U) ? 2U : 1U;
+
+ for (i = 0U; i < lane_cnt; i++) {
+ u32 ret;
+
+ pmu_cal_margin_matrix_scan_eval(freq_pt, mode_cfg, step_lim, type,
+ step_flag_c, (u8)i);
+
+ if (dmem_read8(PMU_DMEM_CAL_OVERRIDE_FLAG) == 0U) {
+ ret = pmu_cal_rank_margin_window_check(step_bound_val, 0U,
+ step_lim);
+ if (ret == 0U)
+ ret = pmu_cal_rank_margin_window_check(step_bound_val,
+ 1U,
+ step_lim);
+ pmu_cal_channel_rank_bounds_verify(ret);
+ }
+
+ if (mode_cfg != 0U)
+ pmu_rank_slice_margin_eval();
+
+ if (step_flag_b != 0U) {
+ pmu_cal_dly_line_tap_step_adjust(step_flag_a, (u8)i,
+ step_lim, step_flag_e);
+ if (pmu_sdata()->high_freq_flag != 0U && i == 1U &&
+ step_flag_a != 0U)
+ pmu_slice_dly_tap_coeff_prog(step_lim);
+ } else {
+ pmu_cal_dbyte_deskew_pin_results_apply();
+ }
+ }
+}
+
+/**
+ * pmu_deskew_sweep_save_delays() - Save current per-slice/lane delay registers
+ * @dly_buf: Output 4x10 halfword delay backup matrix
+ * @slice_start: First slice index
+ * @slice_end: Last slice index
+ * @active_slice_mask: Active slice bitmask
+ * @ch_offset: Channel CSR offset
+ * @profile_base_addr: Profile register base offset
+ * @ch: Channel index (0 or 1)
+ * @active_lane_mask: Non-zero if lane 8 (DBI/DM) is active
+ * @is_lpddr5_type: Non-zero for LPDDR5 lane-9-only mode
+ */
+static void pmu_deskew_sweep_save_delays(u16 dly_buf[4][10], u8 slice_start,
+ u8 slice_end, u8 active_slice_mask,
+ u32 ch_offset, u32 profile_base_addr,
+ u32 ch, u32 active_lane_mask,
+ u32 is_lpddr5_type)
+{
+ u32 slice;
+
+ for (slice = slice_start; slice <= slice_end; slice++) {
+ u32 slice_offset;
+ uintptr_t base_addr;
+ u32 lane;
+
+ if (!(active_slice_mask & (1U << slice)))
+ continue;
+
+ slice_offset = (slice << 12) | ch_offset | profile_base_addr | ch;
+ base_addr = PHY_REG_PUB_BASE | (slice_offset << 1);
+
+ for (lane = 0U; lane < 10U; lane++) {
+ if (lane == 8U && !active_lane_mask)
+ continue;
+
+ if (is_lpddr5_type != 0U) {
+ if (lane == 9U)
+ dly_buf[slice][9] = phy_read16(base_addr);
+ } else if (lane != 9U) {
+ uintptr_t reg_addr = PHY_REG_PUB_BASE |
+ ((slice_offset | (lane << 8)) << 1);
+
+ dly_buf[slice][lane] = phy_read16(reg_addr);
+ }
+ }
+ }
+}
+
+/**
+ * pmu_deskew_write_type0_margins() - Clamp and write type-0 DBYTE lane margin registers
+ * @lane_reg_off: Shifted slice/channel/lane register offset
+ * @lo_margin_val: Computed left/low margin value
+ * @right_margin_val: Computed right/high margin value
+ */
+static void pmu_deskew_write_type0_margins(u32 lane_reg_off, s32 lo_margin_val,
+ s32 right_margin_val)
+{
+ uintptr_t lane_addr_a = (PHY_REG_DBYTE_BASE | 0x00d0UL) | lane_reg_off;
+ uintptr_t lane_addr_b = (PHY_REG_DBYTE_BASE | 0x00d4UL) | lane_reg_off;
+ u8 bound_lim_a = (u8)phy_read16(lane_addr_a);
+ u8 bound_lim_b = (u8)phy_read16(lane_addr_b);
+ u16 fine_reg_val = phy_read16((PHY_REG_DBYTE_BASE | 0x0024UL) | lane_reg_off);
+ u16 coarse_reg_val = phy_read16((PHY_REG_DBYTE_BASE | 0x0020UL) | lane_reg_off);
+ u8 cal_margin_offset = dmem_read8(PMU_DMEM_SEARCH_WINDOW_BIAS_HI);
+
+ if ((u16)lo_margin_val > bound_lim_a)
+ lo_margin_val = bound_lim_a;
+ if ((u16)right_margin_val > bound_lim_b)
+ right_margin_val = bound_lim_b;
+
+ if (cal_margin_offset != 0U) {
+ u16 lane_mask_hi = fine_reg_val & 0x7fU;
+ u16 lane_mask_lo = coarse_reg_val & 0x7fU;
+ u16 min_bmsk = (lane_mask_hi >= lane_mask_lo) ? lane_mask_lo : lane_mask_hi;
+ u16 max_bmsk = (lane_mask_lo >= lane_mask_hi) ? lane_mask_lo : lane_mask_hi;
+ u32 sum_hi = (u16)right_margin_val + lane_mask_hi;
+ u32 sum_lo = (u16)right_margin_val + lane_mask_lo;
+
+ if (!(lane_mask_hi > (u16)lo_margin_val && lane_mask_lo > (u16)lo_margin_val))
+ lo_margin_val = (s32)min_bmsk - (s32)cal_margin_offset;
+ if (sum_hi > 0x7eU || sum_lo >= 0x7fU)
+ right_margin_val = (s32)(max_bmsk ^ 0x7fU) -
+ (s32)cal_margin_offset;
+ }
+
+ phy_write16(lane_addr_a, (u16)lo_margin_val);
+ phy_write16((PHY_REG_DBYTE_BASE | 0x00d8UL) | lane_reg_off, (u16)lo_margin_val);
+ phy_write16(lane_addr_b, (u16)right_margin_val);
+ phy_write16((PHY_REG_DBYTE_BASE | 0x00dcUL) | lane_reg_off, (u16)right_margin_val);
+}
+
+/**
+ * pmu_deskew_decode_lane_edges() - Decode left and right eye edges from sweep sample buffer
+ * @sample_offset: Byte offset of the lane's 848-byte sample record
+ * @profile_stride: Maximum sweep tap stride (0x80 or 0xd4)
+ * @left_out: Output pointer for left edge tap
+ * @right_out: Output pointer for right edge tap
+ */
+static void pmu_deskew_decode_lane_edges(u32 sample_offset, u32 profile_stride,
+ u32 *left_out, u32 *right_out)
+{
+ u8 blink_flag = *(__volatile__ const u8 *)(uintptr_t)
+ ((PMU_DMEM_BASE | 0x6594UL) + sample_offset);
+ const __volatile__ u8 *s_ptr;
+ u8 sample_low;
+
+ if (blink_flag == 0U) {
+ *left_out = 0U;
+ *right_out = 0U;
+ return;
+ }
+
+ s_ptr = (const __volatile__ u8 *)(uintptr_t)
+ ((PMU_DMEM_BASE | PMU_DMEM_SWEEP_SAMPLE_BUF) + sample_offset);
+ sample_low = s_ptr[0];
+
+ if (blink_flag == 1U) {
+ *right_out = profile_stride;
+ *left_out = sample_low;
+ } else if (blink_flag == 3U) {
+ u8 sample_mid = s_ptr[1];
+ u8 sample_high = s_ptr[2];
+ u32 diff_sample = (u32)sample_mid - (u32)sample_low;
+ u32 diff_limit = profile_stride - (u32)sample_high;
+ bool use_hi = diff_limit > diff_sample;
+
+ *right_out = use_hi ? profile_stride : sample_mid;
+ *left_out = use_hi ? sample_high : sample_low;
+ } else {
+ *right_out = s_ptr[1];
+ *left_out = sample_low;
+ }
+}
+
+/**
+ * pmu_cal_multi_lane_deskew_sweep() - Execute multi-lane deskew sweep and program DBYTE margins
+ * @type: Sweep type (0..3)
+ * @ch_mask: Channel bitmask
+ * @lane_mask: Active lane bitmask
+ */
+void pmu_cal_multi_lane_deskew_sweep(u32 type, u32 ch_mask, u32 lane_mask)
+{
+ u16 dly_buf[4][10];
+ u16 lane_metric_max[4] = { 0 };
+ u16 lane_metric_min[4] = { 0 };
+ u32 desc_val[9];
+ u32 is_lpddr5_type, profile_stride, profile_base_addr;
+ u32 freq_lt_3200, active_lane_mask, active_mask, ch;
+
+ memset(desc_val, 0, sizeof(desc_val));
+
+ is_lpddr5_type = (type == 3U) ? 1U : 0U;
+ profile_stride = (type & 2U) ? 0xd4U : 0x80U;
+ profile_base_addr = 0x10010UL + ((type & 2U) ? 0x16U : 0U) +
+ ((type & 1U) ? 2U : 0U);
+
+ freq_lt_3200 = (dmem_read16(PMU_DMEM_DRAM_FREQ_OFF) < 3200U) ? 1U : 0U;
+ active_lane_mask = (lane_mask >> 8) & 0x7fU;
+ active_mask = (u32)dmem_read8(PMU_DMEM_CH1_RANK_EN) |
+ (u32)dmem_read8(PMU_DMEM_CH0_RANK_EN);
+
+ for (ch = 0U; ch < 2U; ch++) {
+ u32 ch_active_mask = (1U << (ch + 2U)) | (1U << ch);
+ u8 slice_start, slice_end, active_slice_mask;
+ u32 ch_offset, slice;
+
+ if (!(ch_active_mask & active_mask))
+ continue;
+
+ dmem_write8(PMU_DMEM_CAL_BYTE, (u8)ch);
+ pmu_phy_mode_cfg_dispatch(3U);
+
+ slice_start = dmem_read8(PMU_DMEM_SLICE_START);
+ slice_end = dmem_read8(PMU_DMEM_SLICE_END);
+ active_slice_mask = dmem_read8(PMU_DMEM_ACTIVE_SLICE_MASK_CAL);
+ ch_offset = dmem_read32(PMU_DMEM_ACTIVE_CSR_OFFSET);
+
+ pmu_deskew_sweep_save_delays(dly_buf, slice_start, slice_end,
+ active_slice_mask, ch_offset,
+ profile_base_addr, ch,
+ active_lane_mask, is_lpddr5_type);
+
+ pmu_cal_dbyte_pattern_loop_exec(ch, (u16)type, lane_metric_max,
+ lane_metric_min, 0U, 1U, 0U);
+
+ if (is_lpddr5_type != 0U)
+ pmu_dbyte_rx_en_slice_pulse_eval(1U, ch_active_mask);
+
+ pmu_cal_bist_cmd_strobe_dispatch((u32)(uintptr_t)desc_val, 8U,
+ type, ch_active_mask,
+ (u32)(uintptr_t)lane_metric_max,
+ active_lane_mask, 0U);
+
+ memset((void *)(PMU_DMEM_BASE | PMU_DMEM_SWEEP_SAMPLE_BUF),
+ 0, 0x8480U);
+
+ pmu_cal_slice_step_eval_sweep((void *)(PMU_DMEM_BASE | PMU_DMEM_SWEEP_SAMPLE_BUF),
+ 8U, type, 4U, 0U, (u8)ch_mask,
+ freq_lt_3200, lane_metric_max,
+ active_lane_mask, 0U, desc_val);
+
+ pmu_dmem_reg_stream_unpack(desc_val);
+
+ if (is_lpddr5_type != 0U)
+ pmu_dbyte_rx_en_slice_pulse_eval(0U, ch_active_mask);
+
+ for (slice = slice_start; slice <= slice_end; slice++) {
+ u32 slice_off, reg_base_ch, reg_base_var, slice_offset_val;
+ uintptr_t lane_reg_a, lane_reg_b, lane_reg_c;
+ u32 lane;
+
+ if (!(active_slice_mask & (1U << slice)))
+ continue;
+
+ slice_off = slice << 12;
+ reg_base_ch = slice_off | ch_offset | ch;
+ reg_base_var = slice_off | ch_offset | profile_base_addr | ch;
+ slice_offset_val = slice_off | ch_offset | (ch << 1);
+
+ lane_reg_a = (PHY_REG_DBYTE_BASE | 0x0044UL) | (reg_base_ch << 1);
+ lane_reg_b = (PHY_REG_DBYTE_BASE | 0x0032UL) | (slice_offset_val << 1);
+ lane_reg_c = PHY_REG_PUB_BASE | (reg_base_var << 1);
+
+ for (lane = 0U; lane < 10U; lane++) {
+ u32 right_edge, left_edge;
+ u16 cal_val_raw, cur_val_raw, metric_val_0, metric_val_1;
+ s32 term_0, delta_term, right_margin_val, lo_margin_val;
+ u32 lane_reg_off;
+
+ if (lane == 8U && !active_lane_mask)
+ continue;
+ if (is_lpddr5_type ? lane != 9U : lane == 9U)
+ continue;
+
+ pmu_deskew_decode_lane_edges((slice * 10U + lane) * 848U,
+ profile_stride,
+ &left_edge, &right_edge);
+
+ cal_val_raw = lane_metric_max[slice];
+ cur_val_raw = dly_buf[slice][lane];
+
+ if (type >= 2U) {
+ metric_val_0 = cur_val_raw;
+ metric_val_1 = cal_val_raw;
+ } else {
+ metric_val_0 = (u16)((cur_val_raw & 0x7fU) +
+ ((cur_val_raw >> 1) & ~0x3fU));
+ metric_val_1 = (u16)((cal_val_raw & 0x7fU) +
+ ((cal_val_raw >> 1) & ~0x3fU));
+ }
+
+ right_edge += metric_val_1;
+ term_0 = ((s32)metric_val_0 - (s32)metric_val_1) -
+ (s32)left_edge;
+ delta_term = (s32)right_edge - (s32)metric_val_0;
+
+ right_margin_val = (type < 2U) ? delta_term : term_0;
+ lo_margin_val = (type < 2U) ? term_0 : delta_term;
+ lane_reg_off = (reg_base_ch | (lane << 8)) << 1;
+
+ if (type == 0U) {
+ pmu_deskew_write_type0_margins(lane_reg_off,
+ lo_margin_val,
+ right_margin_val);
+ } else if (type == 2U) {
+ phy_write16((PHY_REG_DBYTE_BASE | 0x0038UL) | lane_reg_off,
+ (u16)lo_margin_val);
+ phy_write16((PHY_REG_DBYTE_BASE | 0x003cUL) | lane_reg_off,
+ (u16)right_margin_val);
+ } else if (type == 1U) {
+ phy_write16((PHY_REG_DBYTE_BASE | 0x00d0UL) | lane_reg_off,
+ (u16)lo_margin_val);
+ phy_write16((PHY_REG_DBYTE_BASE | 0x00d8UL) | lane_reg_off,
+ (u16)lo_margin_val);
+ phy_write16((PHY_REG_DBYTE_BASE | 0x00d4UL) | lane_reg_off,
+ (u16)right_margin_val);
+ phy_write16((PHY_REG_DBYTE_BASE | 0x00dcUL) | lane_reg_off,
+ (u16)right_margin_val);
+ } else {
+ phy_write16(lane_reg_b, (u16)lo_margin_val);
+ phy_write16(lane_reg_a, (u16)right_margin_val);
+ }
+
+ if (lane == 9U)
+ phy_write16(lane_reg_c, cur_val_raw);
+ else
+ phy_write16(PHY_REG_PUB_BASE |
+ (((reg_base_var | (lane << 8))) << 1),
+ cur_val_raw);
+ }
+ }
+ }
+}
+
+/**
+ * pmu_cal_bist_search_win_setup() - Setup BIST search window or program slice delay steps
+ * @stage: 0 for search window and Vref DAC setup, non-zero for slice delay step program
+ */
+void pmu_cal_bist_search_win_setup(u32 stage)
+{
+ u32 csr_offset;
+ u8 start_slice;
+ u8 end_slice;
+ u16 pin_delays[36];
+ u32 freq_pt;
+
+ if (stage == 0U) {
+ dmem_write16(PMU_DMEM_TRAIN_MODE, 0x0010U);
+ pmu_cal_search_win_init(PMU_DMEM_BASE | PMU_DMEM_SEARCH_WIN_FINE);
+ pmu_cal_vref_dac_step_adjust(8U);
+ pmu_sdata()->cal_active_flag = 1U;
+ pmu_sdata()->cal_active_flag = 0U;
+ pmu_post_cmd_conditional_dispatch(4U);
+ dmem_write16(PMU_DMEM_TRAIN_MODE, 0x8010U);
+ return;
+ }
+
+ csr_offset = pmu_sdata()->active_csr_offset;
+ start_slice = dmem_read8(PMU_DMEM_SLICE_START);
+ end_slice = dmem_read8(PMU_DMEM_SLICE_END);
+
+ for (freq_pt = 0U; freq_pt < 2U; freq_pt++) {
+ u32 cal_mode = freq_pt;
+ u16 window_params_buf[4];
+ u32 pin_idx;
+ u32 slice;
+
+ pmu_phy_cal_trig_pulse();
+ pmu_cal_slice_step_diff_commit(cal_mode, window_params_buf, 1U);
+ pmu_phy_soft_reset_pulse();
+
+ pin_idx = (u32)start_slice * 9U;
+
+ for (slice = start_slice; slice <= end_slice; slice++) {
+ u16 step_diff = window_params_buf[slice];
+ u32 slice_mask = csr_offset | (slice << 12);
+ u32 reg_addr = PHY_REG_DBYTE_BASE |
+ ((slice_mask | (cal_mode + 0x2aU)) << 1);
+ u32 low6;
+ u32 end_pin = pin_idx + 9U;
+
+ low6 = phy_read16(reg_addr) & 0x3fU;
+ step_diff = step_diff + low6 + 0x20U;
+
+ for (; pin_idx < end_pin; pin_idx++)
+ pin_delays[pin_idx] = step_diff;
+
+ phy_write16(reg_addr - 4U, step_diff);
+ }
+
+ pmu_slice_delay_step_program(csr_offset | (cal_mode + 0x26U),
+ (const s16 *)(pin_delays), 1U, 0U, 0U, 0U);
+ }
+}
+
+/**
+ * pmu_cal_dbyte_rx_fifo_reset_poll() - Poll DBYTE Rx FIFO reset and sample multi-channel phase
+ * @channel: Target channel index (0 or 1)
+ * @mode_sel: Mode/rank selector
+ * @enable_scan: Non-zero to enable gate delay sweep
+ * @unused_buf: Output phase sample matrix buffer
+ *
+ * Return: Signed 16-bit FIFO delay offset difference.
+ */
+s16 pmu_cal_dbyte_rx_fifo_reset_poll(u32 channel, u32 mode_sel,
+ u32 enable_scan, void *unused_buf)
+{
+ u8 buf[256];
+ s16 ret_val = 0;
+ u32 is_mode2;
+ u32 csr_offset;
+ u8 step;
+ u8 *out_matrix;
+ u32 base_reg;
+ u16 delay;
+ u32 ch;
+ u32 addr;
+
+ pmu_cal_dbyte_dq_status_check(channel, mode_sel, enable_scan);
+ dmem_write8(PMU_DMEM_AC_STEP_FLAG, 0U);
+
+ if (enable_scan != 0U && (dmem_read8(PMU_DMEM_CAL_MODE) & (1U << 1))) {
+ u32 mode_flag = (mode_sel != 1U) ? 1U : 0U;
+ u32 mode_val = mode_flag + 8U;
+ u32 slice_offset;
+ u32 win_params[2];
+ u32 win_val;
+ u32 freq_pt;
+ u32 reg_tx_dqs;
+ u16 stride;
+ u16 sample_val;
+ u16 stride_x4;
+ u32 reg_gate;
+ u32 i;
+ u32 scan_idx = 0U;
+ s16 fifo_diff_signed;
+ u8 diff_tap_val;
+ u32 tap_pair_off;
+
+ memset(buf, 0, 256U);
+
+ slice_offset = pmu_sdata()->active_csr_offset;
+ win_params[0] = 0U;
+ win_params[1] = 0U;
+
+ pmu_cal_strobe_pulse_init();
+
+ pmu_cal_sequence_pulse_send(0x41UL << 19, 7U, 8U, 0U, 0U, 0U, 0U);
+ pmu_cal_pad_timing_delay_set(0U, 4U, 1U);
+ win_val = pmu_cal_window_params_dispatch((const u8 *)win_params,
+ (u16)mode_sel, 3U);
+ pmu_cal_pad_timing_delay_set(0U, 4U, 1U);
+ pmu_cal_sequence_pulse_send(1UL << 18, 7U, 4U, 0U, 0U, 0U, 0U);
+ pmu_cal_sequence_pulse_send((1UL << 19) | 0x80UL, 7U, 8U, 0U, 0U,
+ 0U, 0U);
+
+ freq_pt = (channel << 12) | slice_offset;
+ reg_tx_dqs = (PHY_REG_AC_BASE | 0x0004UL) | (freq_pt << 1);
+ stride = pmu_cal_stride_get();
+ phy_write16(reg_tx_dqs, 0x40U);
+
+ sample_val = (u16)(win_val - 8U);
+ pmu_phy_cal_trig_pulse();
+
+ stride_x4 = (u16)(stride << 2);
+ pmu_assert_or_halt(pmu_dbyte_deskew_sample_check(channel, mode_sel,
+ (u16)(sample_val),
+ (u16)(stride_x4)) == 0U,
+ PMU_ERR_DESKEW_SAMPLE_CHECK);
+
+ reg_gate = (PHY_REG_AC_BASE | 0x0002UL) |
+ (((mode_val << 8) | freq_pt) << 1);
+ step = dmem_read8(PMU_DMEM_CAL_STRIDE);
+
+ for (i = 0U; i < 256U; i += step) {
+ phy_write16(reg_gate, (u16)i);
+ pmu_phy_cal_trig_pulse();
+ buf[i] = (u8)pmu_dbyte_deskew_sample_check(channel, mode_sel,
+ (u16)(sample_val),
+ (u16)(stride_x4));
+ }
+
+ phy_write16(reg_gate, 0U);
+ phy_write16(reg_tx_dqs, 0U);
+ pmu_phy_cal_trig_pulse();
+
+ while (scan_idx < 256U && !buf[scan_idx])
+ scan_idx += step;
+
+ fifo_diff_signed = (s16)(scan_idx - 64U);
+ pmu_assert_or_halt(fifo_diff_signed < 256,
+ PMU_ERR_RX_FIFO_SCAN_OVERFLOW);
+
+ diff_tap_val = dmem_read8(PMU_DMEM_DIFF_TAP_PAIR);
+ if ((u32)(diff_tap_val - 1U) >= 4U) {
+ diff_tap_val = 4U;
+ dmem_write8(PMU_DMEM_DIFF_TAP_PAIR, 4U);
+ }
+
+ tap_pair_off = (u32)diff_tap_val << 5;
+ ret_val = (s16)(fifo_diff_signed - (s16)tap_pair_off);
+ }
+
+ is_mode2 = ((mode_sel & 3U) == 2U) ? 1U : 0U;
+ pmu_cbt_timing_pulse_coordinator(mode_sel, enable_scan);
+
+ csr_offset = pmu_sdata()->active_csr_offset;
+ step = dmem_read8(PMU_DMEM_CAL_STRIDE);
+ out_matrix = (u8 *)unused_buf;
+
+ base_reg = (PHY_REG_AC_BASE | 0x0002UL) |
+ (((channel << 12) | csr_offset) << 1);
+ for (delay = 0U; delay < 128U; delay += step) {
+ u32 out_mask = 0U;
+ u8 *p;
+
+ for (ch = 0U, addr = base_reg; ch < 7U; ch++, addr += 0x200U)
+ phy_write16(addr, delay);
+ pmu_phy_cal_trig_pulse();
+
+ pmu_dbyte_deskew_phase_sample(is_mode2, &out_mask, 3U);
+
+ {
+ u32 mask = out_mask;
+
+ p = &out_matrix[(uintptr_t)channel * 1792U + delay];
+ for (ch = 0U; ch < 7U; ch++, p += 256U, mask >>= 1) {
+ p[0] |= (u8)(mask & 1U);
+ p[128] |= (u8)((mask >> 8) & 1U);
+ }
+ }
+ }
+
+ addr = PHY_AC_PROFILE_LANE_BASE | (csr_offset << 1);
+ for (ch = 0U; ch < 7U; ch++, addr += 0x200U)
+ phy_write16(addr, 0U);
+ pmu_phy_cal_trig_pulse();
+
+ dmem_write8(PMU_DMEM_AC_STEP_FLAG, 1U);
+ pmu_cal_multi_rank_deskew_sweep(channel, mode_sel);
+
+ return ret_val;
+}
+
+/**
+ * pmu_rx_en_select_and_commit_best() - Select best RxEn delay step per slice and commit to PHY
+ * @min_diff_table: 13x4 table of minimum per-lane eye widths per step and slice
+ * @rx_en_delays_b: 13x4 table of summed per-lane eye widths per step and slice
+ * @lut_val: Channel LUT offset
+ * @csr_offset: Channel CSR offset
+ * @off0: First DBYTE RxEn register offset
+ * @off1: Second DBYTE RxEn register offset
+ * @rx_en_val_lo: Upper byte of initial off0 register
+ * @rx_en_val_hi: Upper byte of initial off1 register
+ */
+static void pmu_rx_en_select_and_commit_best(const u16 min_diff_table[13][4],
+ const u16 rx_en_delays_b[13][4],
+ u32 lut_val, u32 csr_offset,
+ u16 off0, u16 off1,
+ u8 rx_en_val_lo, u8 rx_en_val_hi)
+{
+ u8 slice_start = dmem_read8(PMU_DMEM_SLICE_START);
+ u8 slice_end = dmem_read8(PMU_DMEM_SLICE_END);
+ u32 s;
+
+ for (s = slice_start; s <= slice_end; s++) {
+ s8 best_diff = -1;
+ u8 best_step = 0xffU;
+ s16 best_sum = -1;
+ u32 step, addr0, addr1;
+ u16 saved_iter_cnt, val_off0, val_off1;
+ u8 comp_step;
+
+ for (step = 0U; step < 13U; step++) {
+ s8 diff = (s8)(u8)min_diff_table[step][s];
+
+ if (diff > best_diff ||
+ (diff == best_diff &&
+ (s16)rx_en_delays_b[step][s] > best_sum)) {
+ best_step = (u8)step;
+ best_diff = diff;
+ best_sum = (s16)rx_en_delays_b[step][s];
+ }
+ }
+
+ saved_iter_cnt = dmem_read16(PMU_DMEM_ITER_COUNT);
+ dmem_write16(PMU_DMEM_CBT_DELAY_ARR_OFF + (s * 8U) +
+ (saved_iter_cnt * 2U), (u16)(s8)best_step);
+
+ comp_step = (u8)(12 - (s8)best_step);
+ val_off0 = ((u16)rx_en_val_lo << 8) |
+ ((u16)comp_step << 4) | (u8)best_step;
+ addr0 = PHY_REG_DBYTE_BASE |
+ (((lut_val + off0) | (s << 12) | csr_offset) << 1);
+ phy_write16(addr0, val_off0);
+
+ val_off1 = ((u16)rx_en_val_hi << 8) |
+ ((u16)(u8)best_step << 4) | comp_step;
+ addr1 = PHY_REG_DBYTE_BASE |
+ (((lut_val + off1) | (s << 12) | csr_offset) << 1);
+ phy_write16(addr1, val_off1);
+ }
+}
+
+/**
+ * pmu_cal_rank_rx_en_delay_commit() - Sweep 13 RxEn delay steps and commit best step
+ * @channel: Target channel index (0 or 1)
+ * @rank_flags: Frequency point flags
+ * @cal_stage_mode: Calibration stage mode
+ * @stage: Stage identifier (0x11 or 0x12)
+ */
+void pmu_cal_rank_rx_en_delay_commit(u32 channel, u16 rank_flags,
+ u32 cal_stage_mode, u32 stage)
+{
+ u8 deskew_work_buf[584];
+ u8 scratch_buf[64];
+ u16 min_diff_table[13][4];
+ u16 rx_en_delays_a[13][4];
+ u16 rx_en_delays_b[13][4];
+ u16 slice_matrix_a[4][10], slice_matrix_b[4][10], slice_matrix_c[4][10];
+ u16 slice_matrix_d[4][10], slice_matrix_e[4][10], slice_matrix_f[4][10];
+ u32 lut_val = pmu_bist_pattern_bit_get(channel, channel);
+ u32 csr_offset, chan_mask, is_lt_3200 = 1U, idx, s;
+ u16 off0, off1, reg0, reg1;
+ u8 rx_en_val_lo, rx_en_val_hi, slice_start, slice_end;
+
+ memset(rx_en_delays_b, 0, sizeof(rx_en_delays_b));
+ memset(rx_en_delays_a, 0, sizeof(rx_en_delays_a));
+ memset(min_diff_table, 0, sizeof(min_diff_table));
+
+ pmu_cal_channel_rank_boundary_eval(deskew_work_buf, (u8)channel,
+ cal_stage_mode, stage);
+
+ csr_offset = pmu_sdata()->active_csr_offset;
+ phy_write16(PHY_REG_DBYTE_BCAST_BASE | (csr_offset << 1), 1U);
+
+ off0 = *(u16 *)&deskew_work_buf[514];
+ off1 = *(u16 *)&deskew_work_buf[516];
+
+ reg0 = phy_read16(PHY_REG_DBYTE_BASE | ((csr_offset | off0) << 1));
+ rx_en_val_lo = (u8)(reg0 >> 8);
+
+ reg1 = phy_read16(PHY_REG_DBYTE_BASE | ((csr_offset | off1) << 1));
+ rx_en_val_hi = (u8)(reg1 >> 8);
+
+ chan_mask = (1U << channel) | ((1U << channel) << 2);
+ if (stage == 0x11U)
+ is_lt_3200 = pmu_check_freq_lt_3200();
+
+ for (idx = 0U; idx <= 12U; idx++) {
+ u8 comp_idx = (u8)(12U - idx);
+ u16 lo_margin_val = ((u16)rx_en_val_lo << 8) | ((u16)comp_idx << 4) | (u8)idx;
+ u16 hi_margin_val = ((u16)rx_en_val_hi << 8) | ((u16)idx << 4) | (u8)comp_idx;
+ void *sweep_buf = (void *)(PMU_DMEM_BASE | PMU_DMEM_SWEEP_SAMPLE_BUF);
+ u32 i;
+ u16 ctrl_8;
+ u8 num_samples;
+
+ memset(slice_matrix_f, 0, sizeof(slice_matrix_f));
+ memset(slice_matrix_e, 0, sizeof(slice_matrix_e));
+ memset(slice_matrix_d, 0, sizeof(slice_matrix_d));
+
+ slice_start = dmem_read8(PMU_DMEM_SLICE_START);
+ slice_end = dmem_read8(PMU_DMEM_SLICE_END);
+
+ for (s = slice_start; s <= slice_end; s++) {
+ u32 base = (s << 12) | csr_offset;
+
+ phy_write16(PHY_REG_DBYTE_BASE | (((lut_val + off0) | base) << 1),
+ lo_margin_val);
+ phy_write16(PHY_REG_DBYTE_BASE | (((lut_val + off1) | base) << 1),
+ hi_margin_val);
+ }
+
+ pmu_phy_cal_trig_pulse();
+
+ for (i = 0U; i < 40U; i++)
+ dmem_write8(PMU_DMEM_SWEEP_BLINK_OFF + i * 848U, 0U);
+
+ pmu_cal_bist_cmd_strobe_dispatch((u32)(uintptr_t)scratch_buf,
+ deskew_work_buf[523],
+ deskew_work_buf[518],
+ (u8)chan_mask,
+ (u32)(uintptr_t)deskew_work_buf,
+ cal_stage_mode,
+ deskew_work_buf[525]);
+
+ pmu_cal_slice_step_eval_sweep(sweep_buf, deskew_work_buf[523],
+ deskew_work_buf[518], 4U,
+ (u8)chan_mask, (u8)rank_flags,
+ is_lt_3200,
+ (const u16 *)deskew_work_buf,
+ cal_stage_mode,
+ deskew_work_buf[525],
+ scratch_buf);
+
+ if (deskew_work_buf[524] != 0U) {
+ pmu_dbyte_rx_en_slice_pulse_eval(1U, (u8)chan_mask);
+ pmu_cal_slice_step_eval_sweep(sweep_buf,
+ deskew_work_buf[523], 3U, 4U,
+ (u8)chan_mask, (u8)rank_flags,
+ 1U, (const u16 *)deskew_work_buf,
+ 0U, 0U, scratch_buf);
+ pmu_dbyte_rx_en_slice_pulse_eval(0U, (u8)chan_mask);
+ }
+
+ pmu_dmem_reg_stream_unpack(scratch_buf);
+ pmu_eye_centroid_avg_calc((const u16 *)(deskew_work_buf + 16),
+ (u16 *)slice_matrix_f,
+ (u16 *)slice_matrix_e,
+ (u16 *)slice_matrix_d);
+
+ ctrl_8 = *(u16 *)&deskew_work_buf[520];
+ if (ctrl_8 != 255U) {
+ for (i = 0U; i < 40U; i++)
+ dmem_write8(PMU_DMEM_SWEEP_BLINK_OFF +
+ i * 848U, 0U);
+
+ pmu_cal_bist_cmd_strobe_dispatch((u32)(uintptr_t)scratch_buf,
+ deskew_work_buf[523],
+ (u8)ctrl_8,
+ (u8)chan_mask,
+ (u32)(uintptr_t)(deskew_work_buf + 8),
+ cal_stage_mode,
+ deskew_work_buf[525]);
+
+ pmu_cal_slice_step_eval_sweep(sweep_buf,
+ deskew_work_buf[523],
+ (u8)ctrl_8, 4U,
+ (u8)chan_mask,
+ (u8)rank_flags,
+ is_lt_3200,
+ (const u16 *)(deskew_work_buf + 8),
+ cal_stage_mode,
+ deskew_work_buf[525],
+ scratch_buf);
+
+ pmu_dmem_reg_stream_unpack(scratch_buf);
+ pmu_eye_centroid_avg_calc((const u16 *)(deskew_work_buf + 24),
+ (u16 *)slice_matrix_c,
+ (u16 *)slice_matrix_b,
+ (u16 *)slice_matrix_a);
+ ctrl_8 = *(u16 *)&deskew_work_buf[520];
+ }
+
+ num_samples = deskew_work_buf[522];
+ for (s = slice_start; s <= slice_end; s++) {
+ s32 min_val = 127;
+
+ if (num_samples != 0U) {
+ s16 sum = 0;
+ u32 k;
+
+ for (k = 0U; k < num_samples; k++) {
+ s8 diff = (s8)(slice_matrix_d[s][k] -
+ slice_matrix_e[s][k]);
+
+ if (diff < min_val)
+ min_val = diff;
+ sum += diff;
+ }
+ rx_en_delays_b[idx][s] = sum;
+
+ if (ctrl_8 == 0U) {
+ s32 min_val2 = 127;
+ s16 sum2 = 0;
+
+ for (k = 0U; k < num_samples; k++) {
+ s8 diff = (s8)(slice_matrix_a[s][k] -
+ slice_matrix_b[s][k]);
+
+ min_val2 = (diff < min_val2) ? diff : min_val2;
+ sum2 += diff;
+ }
+ rx_en_delays_a[idx][s] = sum2;
+ if (min_val2 < min_val) {
+ min_val = min_val2;
+ rx_en_delays_b[idx][s] = sum2;
+ }
+ }
+ }
+ min_diff_table[idx][s] = (u16)(u8)min_val;
+ }
+ }
+
+ pmu_rx_en_select_and_commit_best(min_diff_table, rx_en_delays_b,
+ lut_val, csr_offset, off0, off1,
+ rx_en_val_lo, rx_en_val_hi);
+
+ pmu_cal_dbyte_deskew_results_apply(deskew_work_buf, (u8)channel,
+ stage);
+}
+
+/**
+ * pmu_cal_lane_window_sweep_coordinator() - Coordinate 4-tap lane window sweep and commit best taps
+ * @ch_mask: Channel mask
+ * @wide_range: Non-zero for 9 pins (511 mask), 0 for 8 pins (255 mask)
+ */
+void pmu_cal_lane_window_sweep_coordinator(u32 ch_mask, u32 wide_range)
+{
+ u16 margin_table[4][2][2][2];
+ u16 best_taps[2][2][2];
+ u32 max_step_range = (wide_range != 0U) ? 511U : 255U;
+ u32 pin_count = (wide_range != 0U) ? 9U : 8U;
+ u8 ch1_rank_mask;
+ u8 ch0_rank_mask;
+ u8 active_mask;
+ u32 cfg_idx;
+ u32 ch;
+
+ memset(margin_table, 0, sizeof(margin_table));
+ memset(best_taps, 0, sizeof(best_taps));
+
+ ch1_rank_mask = dmem_read8(PMU_DMEM_CH1_RANK_EN);
+ ch0_rank_mask = dmem_read8(PMU_DMEM_CH0_RANK_EN);
+ active_mask = ch0_rank_mask;
+
+ for (cfg_idx = 0U; cfg_idx < 4U; cfg_idx++) {
+ u8 cfg_val = (u8)(cfg_idx | (cfg_idx << 4));
+ u32 vref;
+
+ for (ch = 0U; ch < 2U; ch++) {
+ u8 mask = (u8)((1U << (ch + 2U)) | (1U << ch));
+
+ if (active_mask & mask)
+ pmu_slice_deskew_state_latch(cfg_val, mask);
+ }
+
+ vref = (dmem_read8(PMU_DMEM_CLK_GATE_FLAG) & 4U) ?
+ 0x3fU : dmem_read8(PMU_DMEM_DQ_SWAP_BASE);
+ pmu_cal_margin_matrix_scan_eval(ch_mask, 0U, max_step_range, 2U,
+ vref, 0U);
+
+ for (ch = 0U; ch < 2U; ch++) {
+ u8 mask = (u8)((1U << (ch + 2U)) | (1U << ch));
+ u8 start_slice;
+ u8 end_slice;
+ uintptr_t slice_buf;
+ u32 slice;
+ u32 rank;
+
+ if (!(active_mask & mask))
+ continue;
+
+ start_slice = dmem_read8(PMU_DMEM_RANK0_SLICE_START);
+ end_slice = dmem_read8(PMU_DMEM_RANK1_SLICE_END);
+ slice_buf = (PMU_DMEM_BASE | PMU_DMEM_EYE_SAMPLE_BUF) +
+ (ch * 5280U) + (start_slice * 1320U);
+
+ for (slice = start_slice; slice <= end_slice; slice++) {
+ uintptr_t pin_ptr = slice_buf;
+ u32 pin;
+
+ for (pin = 0U; pin < pin_count; pin++) {
+ pmu_window_centroid_calc((u8 *)pin_ptr);
+ pin_ptr += 132U;
+ }
+ slice_buf += 1320U;
+ }
+
+ for (rank = 0U; rank < 2U; rank++) {
+ u8 rank_active = (rank != 0U) ? ch1_rank_mask : ch0_rank_mask;
+
+ if (rank_active != 0U) {
+ u8 r_start = (rank != 0U) ?
+ dmem_read8(PMU_DMEM_RANK1_SLICE_START) :
+ dmem_read8(PMU_DMEM_RANK0_SLICE_START);
+ u8 r_end = (rank != 0U) ?
+ dmem_read8(PMU_DMEM_RANK1_SLICE_END) :
+ dmem_read8(PMU_DMEM_RANK_BOUNDARY);
+
+ margin_table[cfg_idx][ch][rank][0] =
+ pmu_rank_margin_channel_accum(ch, r_start, r_end,
+ pin_count, 1U);
+ margin_table[cfg_idx][ch][rank][1] =
+ pmu_rank_margin_channel_accum(ch, r_start, r_end,
+ pin_count, 2U);
+ }
+ }
+ }
+ }
+
+ for (ch = 0U; ch < 2U; ch++) {
+ u8 mask = (u8)((1U << (ch + 2U)) | (1U << ch));
+ u32 rank;
+
+ if (!(active_mask & mask))
+ continue;
+
+ for (rank = 0U; rank < 2U; rank++) {
+ u8 rank_active = (rank != 0U) ? ch1_rank_mask : ch0_rank_mask;
+ u32 dir;
+
+ if (!rank_active)
+ continue;
+
+ for (dir = 0U; dir < 2U; dir++) {
+ u16 max_width = 0U;
+ u16 best_idx = 0U;
+ u32 iter;
+
+ for (iter = 0U; iter < 4U; iter++) {
+ u16 width = margin_table[iter][ch][rank][dir];
+
+ if (width > max_width) {
+ max_width = width;
+ best_idx = (u16)iter;
+ }
+ }
+
+ best_taps[ch][rank][dir] = best_idx;
+
+ if (max_width == 0U)
+ pmu_assert_or_halt(0U, PMU_ERR_WCK_EYE_ZERO_WIDTH);
+ }
+ }
+ }
+
+ for (ch = 0U; ch < 2U; ch++) {
+ u8 mask = (u8)((1U << (ch + 2U)) | (1U << ch));
+ u32 rank;
+
+ if (!(active_mask & mask))
+ continue;
+
+ pmu_cal_pulse_seq_coordinator(0U, ch, 0x50U, 3U);
+
+ for (rank = 0U; rank < 2U; rank++) {
+ u8 rank_active = (rank != 0U) ? ch1_rank_mask : ch0_rank_mask;
+ u8 combined;
+
+ if (!rank_active)
+ continue;
+
+ pmu_phy_analog_cfg_mode_set(rank);
+
+ combined = (u8)((best_taps[ch][rank][1] << 4) |
+ best_taps[ch][rank][0]);
+ pmu_slice_deskew_state_latch(combined, mask);
+
+ dmem_write8(PMU_DMEM_BEST_TAP_ARR + (rank * 2U) + ch,
+ combined);
+
+ if (ch == 0U) {
+ if (rank == 0U)
+ dmem_write8(PMU_DMEM_WCK_TAP_C0R0, combined);
+ else
+ dmem_write8(PMU_DMEM_WCK_TAP_C0R1, combined);
+ } else {
+ if (rank == 0U)
+ dmem_write8(PMU_DMEM_WCK_TAP_C1R0, combined);
+ else
+ dmem_write8(PMU_DMEM_WCK_TAP_C1R1, combined);
+ }
+ }
+ }
+
+ phy_write16(PHY_REG_CH0_ANALOG_PWR, dmem_read8(PMU_DMEM_CH0_RANK_EN));
+ phy_write16(PHY_REG_CH1_ANALOG_PWR, dmem_read8(PMU_DMEM_CH1_RANK_EN));
+}
+
+/**
+ * pmu_rank_deskew_converge_slices() - Run 11-iteration coarse/fine deskew convergence across slices
+ * @csr_offset: Channel CSR offset
+ * @lut_mode_bits: 2-bit channel LUT mode offset
+ * @pll_ratio: Active WCK:CK frequency ratio from DMEM (4 = 4:1, 2 = 2:1)
+ */
+static void pmu_rank_deskew_converge_slices(u32 csr_offset, u32 lut_mode_bits,
+ u8 pll_ratio)
+{
+ u16 slice_done[4] = { 0 };
+ u16 slice_val[4] = { 0 };
+ u16 prev_val[4] = { 0 };
+ u32 converged_flag = 0U;
+ int iter;
+
+ for (iter = 11; iter > 0; iter--) {
+ u8 start_slice = dmem_read8(PMU_DMEM_SLICE_START);
+ u8 end_slice = dmem_read8(PMU_DMEM_SLICE_END);
+ u32 slice;
+
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+
+ for (slice = start_slice; slice <= end_slice; slice++) {
+ u32 slice_csr_off, slice_reg_base, reg_addr;
+ u16 stat, phy_status_reg, prev_stat;
+ u32 fine_delay, coarse_delay, new_val, target_reg;
+
+ if (slice_done[slice] != 0U)
+ continue;
+
+ slice_csr_off = slice << 13;
+ stat = phy_read16((PHY_REG_DBYTE_BASE | 0x016eUL) | slice_csr_off);
+ slice_val[slice] = stat;
+
+ slice_reg_base = (csr_offset | (slice << 12) |
+ (lut_mode_bits + 0x2aU)) << 1;
+ reg_addr = PHY_REG_DBYTE_BASE | slice_reg_base;
+ phy_status_reg = phy_read16(reg_addr);
+ prev_stat = prev_val[slice];
+
+ if (prev_stat != 0U && stat == 0U) {
+ slice_done[slice] = 1U;
+ continue;
+ }
+
+ if (prev_stat == 0U && iter == 11 && stat == 0U)
+ converged_flag = 1U;
+
+ fine_delay = phy_status_reg & 0x3fU;
+ coarse_delay = phy_status_reg >> 6;
+
+ coarse_delay += 2U;
+ new_val = (coarse_delay << 6) | fine_delay;
+ target_reg = (PHY_REG_DBYTE_BASE | 0x1e00UL) |
+ slice_reg_base;
+
+ if (coarse_delay >= 0x15U && converged_flag != 0U &&
+ (coarse_delay % pll_ratio) != 0U) {
+ u16 new_stat;
+
+ phy_write16(target_reg, (u16)fine_delay);
+ pmu_phy_dbyte_cal_trig_pulse(0x0fU, 0x0fU);
+ pmu_phy_soft_reset_pulse();
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+
+ new_stat = phy_read16((PHY_REG_DBYTE_BASE | 0x016eUL) |
+ slice_csr_off);
+ slice_val[slice] = new_stat;
+
+ if (new_stat != 0U) {
+ phy_write16(target_reg, (u16)new_val);
+ slice_done[slice] = 1U;
+ continue;
+ }
+ new_val = fine_delay;
+ }
+ if (coarse_delay >= 0x15U)
+ pmu_assert_or_halt(0U, PMU_ERR_COARSE_DELAY_OVERFLOW);
+
+ phy_write16(target_reg, (u16)new_val);
+ prev_val[slice] = slice_val[slice];
+ }
+
+ pmu_phy_dbyte_cal_trig_pulse(0x0fU, 0x0fU);
+ pmu_phy_soft_reset_pulse();
+ }
+}
+
+/**
+ * pmu_rank_deskew_run_convergence_stage() - Setup PHY BIST state and run deskew convergence
+ * @channel: Target channel index (0 or 1)
+ * @rank: Target rank index (0 or 1)
+ * @csr_offset: Channel CSR offset
+ * @lut_mode_bits: 2-bit channel LUT mode offset
+ * @cbt_dly_idx: Initial CBT delay index
+ * @pll_ratio: Active WCK:CK frequency ratio from DMEM (4 = 4:1, 2 = 2:1)
+ * @ch_mask_b: Active channel bitmask byte
+ */
+static void pmu_rank_deskew_run_convergence_stage(u32 channel, u32 rank,
+ u32 csr_offset,
+ u32 lut_mode_bits,
+ u32 cbt_dly_idx, u8 pll_ratio,
+ u8 ch_mask_b)
+{
+ u32 csr_offset_sh1 = csr_offset << 1;
+ u32 cbt_dly_offset, slice_mask;
+ u16 saved_dram_cfg, target_cfg_val, saved_vref_stat;
+
+ (void)channel;
+ (void)rank;
+
+ pmu_slice_phy_reg_step_adjust(lut_mode_bits, 1U, 0U);
+
+ cbt_dly_idx += (cbt_dly_idx & 1U);
+ pmu_ac_lane_profile_setup();
+
+ if (dmem_read8(PMU_DMEM_FREQ_MODE) == 0U) {
+ pmu_cal_strobe_pulse_init();
+ pmu_cal_sequence_pulse_send(0U, 7U, 4U, 0U, 0U, 0U, 0U);
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+ }
+
+ pmu_cal_strobe_pulse_init();
+ cbt_dly_offset = cbt_dly_idx + 0x1eU;
+
+ pmu_cal_margin_dir_get();
+ pmu_cbt_3phase_pulse_seq();
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+
+ saved_dram_cfg = phy_read16(PHY_REG_DRAM_TYPE_CFG);
+ target_cfg_val = (pll_ratio == 2U) ? 513U : 1026U;
+
+ phy_write16(PHY_REG_DRAM_TYPE_CFG, (pll_ratio == 2U) ? 3U : 2U);
+ phy_write16((PHY_REG_MASTER_BASE | 0x007aUL) | csr_offset_sh1,
+ target_cfg_val);
+ phy_write16((PHY_REG_MASTER_BASE | 0x007cUL) | csr_offset_sh1,
+ target_cfg_val);
+
+ saved_vref_stat = phy_read16(PHY_REG_VREF_STAT);
+ phy_write16(PHY_REG_VREF_STAT, saved_vref_stat | (1U << 7));
+ pmu_slice_phy_reg_step_adjust(lut_mode_bits, 1U, 1U);
+
+ slice_mask = (pmu_dmem_training_flags_eval() == 0U) ? 0xffU : 511U;
+ phy_write16(PHY_REG_DBYTE_BCAST_DQS_DLY0, 0xffffU);
+ phy_write16(PHY_REG_DBYTE_BCAST_DQS_DLY1, 0xffffU);
+ pmu_cal_dbyte_sample_ready_poll_all(slice_mask);
+
+ phy_write16(PHY_REG_BIST_CTRL_MODE, 1U);
+ phy_write16((PHY_REG_PUB_BASE | 0x3ff7cUL), 0U);
+
+ pmu_phy_dbyte_cal_trig_pulse(0x0fU, 0x0fU);
+ pmu_phy_soft_reset_pulse();
+
+ pmu_cal_slice_step_scan_eval(cbt_dly_offset & 0xffU, ch_mask_b);
+
+ pmu_rank_deskew_converge_slices(csr_offset, lut_mode_bits, pll_ratio);
+
+ phy_write16(PHY_REG_DRAM_TYPE_CFG, saved_dram_cfg);
+ phy_write16(PHY_REG_VREF_STAT, saved_vref_stat);
+ pmu_phy_profile_param_program();
+ pmu_slice_phy_reg_step_adjust(lut_mode_bits, 0U, 0U);
+}
+
+struct phy_reg_stream_entry {
+ u32 addr;
+ u16 val;
+} __packed;
+
+static __always_inline void pmu_stream_set(struct phy_reg_stream_entry *e,
+ u32 addr, u16 val)
+{
+ e->addr = addr;
+ e->val = val;
+}
+
+/**
+ * pmu_cal_rank_deskew_matrix_commit() - Execute rank deskew matrix calibration and iterative
+ * convergence
+ * @channel: Target channel index (0 or 1)
+ * @rank: Target rank index (0 or 1)
+ * @buf: Work buffer for LCDL delay readback
+ */
+void pmu_cal_rank_deskew_matrix_commit(u32 channel, u32 rank, void *buf)
+{
+ u32 csr_offset = pmu_sdata()->active_csr_offset;
+ u32 csr_offset_sh1 = csr_offset << 1;
+ u32 ch_mask = (1U << (channel + 2U)) | (1U << channel);
+ u8 ch_mask_b = (u8)ch_mask;
+ u32 cbt_idx = pmu_cal_rank_offset_get(rank, channel);
+ u32 cbt_val = pmu_cbt_delay_offset_lookup(rank, channel);
+ u32 cbt_lut_offset = (cbt_val > 186U) ? 0xfeU : (cbt_val * 2U + 0x22U);
+ u32 lut_val = (0xe4U >> ((channel << 1) & 6U)) & 3U;
+ struct phy_reg_stream_entry stream_save[15];
+ struct phy_reg_stream_entry stream_play[19];
+
+ pmu_stream_set(&stream_save[0], 0x10097UL, 0U);
+ pmu_stream_set(&stream_save[1], 0x10093UL, 0U);
+ pmu_stream_set(&stream_save[2], 0x10005UL | csr_offset, 0U);
+ pmu_stream_set(&stream_save[3], 0x1008bUL, 0U);
+ pmu_stream_set(&stream_save[4], 0x10095UL, 0U);
+ pmu_stream_set(&stream_save[5], 0x20074UL, 0U);
+ pmu_stream_set(&stream_save[6], 0x20057UL, 0U);
+ pmu_stream_set(&stream_save[7], 0x2002aUL, 0U);
+ pmu_stream_set(&stream_save[8], 0x2003fUL | csr_offset, 0U);
+ pmu_stream_set(&stream_save[9], 0x20041UL | csr_offset, 0U);
+ pmu_stream_set(&stream_save[10], 0x20037UL | csr_offset, 0U);
+ pmu_stream_set(&stream_save[11], 0x2003fUL | csr_offset, 0U);
+ pmu_stream_set(&stream_save[12], 0x2003dUL | csr_offset, 0U);
+ pmu_stream_set(&stream_save[13], 0x2003eUL | csr_offset, 0U);
+ pmu_stream_set(&stream_save[14], 0x70011UL, 0U);
+
+ pmu_phy_reg_buffer_stream((u8 *)stream_save, 15U, 1U);
+
+ phy_write16(PHY_REG_BIST_PLL_CFG, 0U);
+ phy_write16((PHY_REG_DBYTE_BCAST_BASE | 0x012eUL), 0U);
+ phy_write16(PHY_REG_DBYTE_BCAST_DQ_DLY0, 8U);
+ phy_write16(PHY_REG_DBYTE_BCAST_DQ_DLY1, 0x40U);
+ phy_write16((PHY_REG_PUB_SEC_BASE | 0x0aeUL), 0xaaU);
+
+ pmu_cal_strobe_pulse_init();
+
+ pmu_cal_sequence_pulse_send(0x41UL << 19, 7U, 4U, 0U, 0U, ch_mask_b, 0U);
+ pmu_cal_sequence_pulse_send(0U, 7U, 8U, 0U, 0U, 0U, 0U);
+ pmu_cal_sequence_pulse_send(0x80UL | (1UL << 19), 7U, 4U, 0U, 0U, ch_mask_b, 0U);
+
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+ pmu_delay_us(5000U, 5U);
+ pmu_dbyte_cal_strobe_pulse_seq(1U, 0xffU, 0U);
+
+ u8 pll_ratio = dmem_read8(PMU_DMEM_PLL_RATIO);
+
+ if (pll_ratio == 0U)
+ pll_ratio = 2U;
+
+ u32 is_wck_4to1 = (pll_ratio == 4U) ? 1U : 0U;
+ u32 step_ctrl_val = ((u32)pll_ratio << 6) + 6U;
+ u32 saved_step_param = (is_wck_4to1 << 2) + 4U;
+
+ if (step_ctrl_val > 255U)
+ step_ctrl_val = 255U;
+
+ dmem_write8(PMU_DMEM_2D_STEP_PARAM, (u8)saved_step_param);
+ dmem_write16(PMU_DMEM_2D_STEP_CTRL, (u16)step_ctrl_val);
+
+ u32 cbt_step_adj = cbt_lut_offset + (cbt_lut_offset & 1U);
+ u8 cbt_step_adj_b = (u8)cbt_step_adj;
+ u32 cbt_dly_adj = (cbt_idx + 0x1eU) + (cbt_idx & 1U);
+ u8 cbt_dly_adj_b = (u8)cbt_dly_adj;
+
+ phy_write16(PHY_REG_VREF_CTRL | csr_offset_sh1, 1U);
+ phy_write16(PHY_REG_VREF_TRIM_CFG, 1U);
+
+ u16 mr12_val = phy_read16((PHY_REG_MASTER_BASE | 0x006eUL) | csr_offset_sh1);
+ u16 mr16_val = mr12_val & 0x3fU;
+
+ if (pll_ratio == 2U)
+ mr16_val |= (1U << 11);
+ else
+ mr16_val |= (1U << 12);
+ phy_write16((PHY_REG_MASTER_BASE | 0x007eUL) | csr_offset_sh1, mr16_val);
+
+ u32 lut_mode_bits = lut_val & 3U;
+
+ phy_write16((PHY_REG_DBYTE_BCAST_BASE | 0x0126UL), 512U);
+ phy_write16(PHY_REG_DBYTE_BCAST_GATE | csr_offset_sh1, 1U);
+ pmu_rank_mask_multiparam_dispatch(channel, rank, 1U);
+
+ pmu_stream_set(&stream_play[0], 0x0001ffb5UL, 0U);
+ pmu_stream_set(&stream_play[1], 0x0001f0b5UL, 1U);
+ pmu_stream_set(&stream_play[2], 0x0001f4b5UL, 1U);
+ pmu_stream_set(&stream_play[3], 0x0001f0b3UL, 0U);
+ pmu_stream_set(&stream_play[4], 0x0001f0b4UL, 0U);
+ pmu_stream_set(&stream_play[5], 0x0001f0b9UL, 1U);
+ pmu_stream_set(&stream_play[6], 0x0001f0baUL, 1U);
+ pmu_stream_set(&stream_play[7], 0x0001f0b1UL, 2U);
+ pmu_stream_set(&stream_play[8], 0x0007f060UL, 1U);
+ pmu_stream_set(&stream_play[9], 0x0007f065UL, 511U);
+ pmu_stream_set(&stream_play[10], 0x0007ff28UL, 0xffffU);
+ pmu_stream_set(&stream_play[11], 0x0007ff29UL, 0xffffU);
+ pmu_stream_set(&stream_play[12], 0x0007ff2aUL, 0xffffU);
+ pmu_stream_set(&stream_play[13], 0x0007ff2bUL, 0xffffU);
+ pmu_stream_set(&stream_play[14], 0x0001ffbeUL, 0U);
+ pmu_stream_set(&stream_play[15], 0x0001f0beUL, 1U);
+ pmu_stream_set(&stream_play[16], 0x0001f4beUL, 16U);
+ pmu_stream_set(&stream_play[17], 0x0001ffaaUL, (u16)(saved_step_param << 10));
+ pmu_stream_set(&stream_play[18], 0x0001f0a7UL, 768U);
+
+ pmu_cal_seq_table_timed_replay((const u8 *)stream_play, sizeof(stream_play));
+
+ pmu_phy_dbyte_cal_trig_pulse(0x0fU, 0x0fU);
+ pmu_cal_dbyte_gate_mask_clear();
+ pmu_dbyte_slice_cal_param_pulse(rank, channel, 1U);
+
+ pmu_cal_strobe_pulse_init();
+ pmu_cal_sequence_pulse_send(0U, 7U, 4U, 0U, 0U, 0U, 0U);
+ pmu_cbt_cal_active_set();
+
+ u32 cbt_channel_mask = (ch_mask << 5) & 64U;
+
+ pmu_cal_sequence_pulse_send(cbt_channel_mask | (1UL << 25), 7U, 2U, 0U,
+ 0U, 0U, 0U);
+ pmu_cal_sequence_pulse_send(cbt_channel_mask, 7U, cbt_step_adj_b, 0U,
+ 0U, 0U, 0U);
+ pmu_cal_sequence_pulse_send(4UL | (1UL << 18), 1U, cbt_dly_adj_b,
+ 1U << 12, 0U, 0U, 0U);
+ pmu_cal_sequence_pulse_send(cbt_channel_mask | (1UL << 15), 7U, 4U, 0U,
+ 0U, 0U, 1U);
+
+ pmu_cbt_cal_active_clear();
+ pmu_cal_sequence_pulse_send(cbt_channel_mask, 7U, 0x10U, 0U, 0U, 0U, 0U);
+ pmu_cal_sequence_pulse_send(cbt_channel_mask | (1U << 7), 7U, 4U, 0U,
+ 0U, 0U, 0U);
+
+ phy_write16(PHY_REG_DBYTE_BCAST_RX_EN, 0x6300U);
+ pmu_cal_slice_step_diff_latch(8U, 1U);
+ phy_write16(PHY_REG_DBYTE_BCAST_RX_EN, 768U);
+
+ u16 saved_step_ctrl = dmem_read16(PMU_DMEM_2D_STEP_CTRL);
+
+ if (dmem_read8(PMU_DMEM_CLK_GATE_FLAG) & (1U << 2)) {
+ pmu_cal_slice_step_diff_latch(saved_step_ctrl, 1U);
+ } else {
+ int i;
+
+ for (i = 0; i < 10; i++)
+ pmu_cal_slice_step_diff_latch(saved_step_ctrl, 1U);
+ }
+
+ pmu_dbyte_slice_cal_param_pulse(rank, channel, 0U);
+ pmu_phy_lcdl_delay_read((u16 *)buf, 0U, 2U);
+
+ u32 reg_target_base = csr_offset | (lut_mode_bits + 0x2aU);
+
+ pmu_slice_delay_step_program(reg_target_base, (const s16 *)((const u16 *)buf), 0U, 0U,
+ 0U, 1U);
+
+ phy_write16(PHY_REG_DBYTE_BCAST_RX_EN, 0U);
+
+ if ((dmem_read8(PMU_DMEM_CAL_STEP) & 1U) == 0U)
+ pmu_rank_deskew_run_convergence_stage(channel, rank,
+ csr_offset,
+ lut_mode_bits, cbt_idx,
+ pll_ratio, ch_mask_b);
+
+ pmu_dbyte_cal_strobe_pulse_seq(0U, 0U, 0U);
+
+ phy_write16(PHY_REG_DBYTE_BCAST_DQ_DLY0, 0U);
+ phy_write16(PHY_REG_DBYTE_BCAST_DQ_DLY1, 0U);
+ pmu_cal_strobe_pulse_init();
+ pmu_cal_pad_timing_delay_set(0U, 5U, 1U);
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+ pmu_rank_mask_multiparam_dispatch(channel, rank, 0U);
+
+ pmu_cal_strobe_pulse_init();
+ pmu_cal_sequence_pulse_send(0U, 7U, 4U, 0U, 0U, 0U, 0U);
+ pmu_cal_sequence_pulse_send(1UL << 16, 3U, cbt_step_adj_b, 1U << 12,
+ 0U, ch_mask_b, 0U);
+ pmu_cal_sequence_pulse_send(0x80U, 7U, 4U, 0U, 0U, 0U, 0U);
+
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+ pmu_phy_soft_reset_pulse();
+ pmu_phy_sec_cal_strobe_pulse();
+
+ pmu_phy_reg_buffer_stream((u8 *)stream_save, 15U, 0U);
+}
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 17/26] sunxi: dram: a733: Add ARC PMU 1D/2D eye training firmware
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (15 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 16/26] sunxi: dram: a733: Add ARC PMU deskew and BIST training firmware Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 18/26] sunxi: dram: a733: Add ARC PMU training state machine Alastair D'Silva
` (10 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
Add the 1D and 2D read/write DQ eye margin sweep, Vref DAC search
window, and center-of-eye optimization routines for the Allwinner A733
DDR PHY ARC EM4 firmware.
Written from clean-room hardware register and JEDEC JESD209-5 protocol
specifications (see doc/board/allwinner/a733-pmu.rst).
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
arch/arm/mach-sunxi/pmu/pmu_eye.c | 3224 +++++++++++++++++++++++++++++
1 file changed, 3224 insertions(+)
create mode 100644 arch/arm/mach-sunxi/pmu/pmu_eye.c
diff --git a/arch/arm/mach-sunxi/pmu/pmu_eye.c b/arch/arm/mach-sunxi/pmu/pmu_eye.c
new file mode 100644
index 00000000000..1c1ebb35363
--- /dev/null
+++ b/arch/arm/mach-sunxi/pmu/pmu_eye.c
@@ -0,0 +1,3224 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Copyright (C) 2026 Alastair D'Silva <alastair@d-silva.org>
+ *
+ * Allwinner A733 Synopsys DDR PHY PMU 2D Eye Margin & Search Window Helpers
+ */
+
+#include "pmu_fw.h"
+
+static int pmu_ac_step_coord_adjust(u8 *struct_ptr);
+static int pmu_ac_step_pair_align_verify(u8 *struct_a, u8 *struct_b);
+static void pmu_cal_dbyte_delay_coarse_step(void);
+static void pmu_cal_dbyte_delay_restore_all_ch(void);
+static u32 pmu_cal_margin_sample_pair_extract(u8 *out_pair, const u8 *margins, const u8 *samples,
+ const u8 *scan_lut);
+static u32 pmu_eye_boundary_sample_search(u8 *samples, u32 target_y);
+static void pmu_eye_margin_bidir_scan(u8 *buf, u32 flag_a, u32 flag_b, u32 flag_c);
+static u32 pmu_eye_margin_step_balancer(u8 *buf0, u8 *buf1, u8 *buf2, u8 *buf3);
+static u32 pmu_eye_sample_metric_eval(u8 x, u8 y, const u8 *eye_scan_lut, const u8 *samples);
+static void pmu_eye_sample_table_transform(u8 *dest, const u8 *src);
+static u32 pmu_window_margin_diff_calc(const u8 *src0, const u8 *src1, u8 *out_dest, s32 offset,
+ s32 delta);
+
+/**
+ * pmu_cal_win_bounds_clear() - Reset 2D search window bounds at DCCM 0x8000043c
+ *
+ * Initializes the two 32-bit search window bound words at DCCM 0x8000043c and
+ * 0x80000440 to the default range 0x00080008 (low=8, high=8).
+ */
+void pmu_cal_win_bounds_clear(void)
+{
+ u32 val = 0x00080008UL;
+ u32 *tbl0 = (u32 *)(uintptr_t)pmu_sdata()->master_dly_tbl[0];
+ u32 *tbl1 = (u32 *)(uintptr_t)pmu_sdata()->master_dly_tbl[1];
+
+ *(__volatile__ u32 *)tbl0 = val;
+ *(__volatile__ u32 *)tbl1 = val;
+}
+
+/**
+ * pmu_cal_search_win_init() - Seed 0xff sentinel bounds in search window buffer
+ * @base: DCCM base address of the search window descriptor buffer
+ *
+ * Writes 0xff sentinel bytes into offsets [0..1], [4..5], [27..28], and
+ * [31..32] of the search window buffer at @base.
+ */
+void pmu_cal_search_win_init(u32 base)
+{
+ __volatile__ u8 *win_buf = (__volatile__ u8 *)base;
+ u8 ff = 0xffU;
+
+ win_buf[0] = ff;
+ win_buf[4] = ff;
+ win_buf[27] = ff;
+ win_buf[31] = ff;
+ win_buf[1] = ff;
+ win_buf[5] = ff;
+ win_buf[28] = ff;
+ win_buf[32] = ff;
+}
+
+/**
+ * pmu_cal_eye_table_restore() - Restore byte field across 4 slices from 0x80000bc0
+ * @dest: Destination 4-slice descriptor table (strides 108 and 54 bytes)
+ * @offset: Element offset within each slice descriptor
+ */
+noinline void pmu_cal_eye_table_restore(u8 *dest, u32 offset)
+{
+ u8 *d = dest + offset;
+ const __volatile__ u16 *s =
+ (const __volatile__ u16 *)PMU_DMEM_EYE_BACKUP + offset;
+ u32 i;
+
+ for (i = 0U; i < PMU_SLICE_DESC_TOTAL_BYTES; i += PMU_SLICE_DESC_SIZE)
+ d[i] = (u8)s[i];
+}
+
+/**
+ * pmu_cal_eye_table_save() - Save byte field across 4 slices into DCCM 0x80000bc0
+ * @src: Source 4-slice descriptor table (strides 108 and 54 bytes)
+ * @offset: Element offset within each slice descriptor
+ */
+noinline void pmu_cal_eye_table_save(const u8 *src, u32 offset)
+{
+ const u8 *s = src + offset;
+ __volatile__ u16 *d =
+ (__volatile__ u16 *)PMU_DMEM_EYE_BACKUP + offset;
+ u32 i;
+
+ for (i = 0U; i < PMU_SLICE_DESC_TOTAL_BYTES; i += PMU_SLICE_DESC_SIZE)
+ d[i] = s[i];
+}
+
+/**
+ * pmu_cal_margin_dir_get() - Configure default pad timing delay (0, 5, 1)
+ */
+void pmu_cal_margin_dir_get(void)
+{
+ pmu_cal_pad_timing_delay_set(0U, 5U, 1U);
+}
+
+struct pmu_pin_map_pair {
+ u8 match;
+ u8 bit;
+};
+
+struct pmu_pin_map_entry {
+ u8 base_mask;
+ u8 count;
+ struct pmu_pin_map_pair pairs[8];
+};
+
+/**
+ * pmu_cal_eye_margin_sample_latch() - Translate logical bitmask to PHY pin mask
+ * @mask: Logical lane bitmask byte
+ * @lane: Slice/lane index into the 18-byte pin map table at DCCM 0x80000a7e
+ *
+ * Return: Translated 8-bit PHY pin mask combining base_mask and matched pairs.
+ */
+u32 pmu_cal_eye_margin_sample_latch(u32 mask, u32 lane)
+{
+ const struct pmu_pin_map_entry *table =
+ (const struct pmu_pin_map_entry *)PMU_DMEM_DQ_PIN_MAP_BASE;
+ const struct pmu_pin_map_entry *entry = &table[lane];
+ u8 res = entry->base_mask & (u8)mask;
+ u32 i;
+
+ for (i = 0U; i < entry->count; i++) {
+ if (mask & (1U << entry->pairs[i].bit))
+ res |= entry->pairs[i].match;
+ }
+
+ return res;
+}
+
+/**
+ * pmu_cal_dbyte_gate_mask_clear() - Clear DBYTE gate mask outside active range
+ */
+noinline void pmu_cal_dbyte_gate_mask_clear(void)
+{
+ const __volatile__ u8 *range =
+ (const __volatile__ u8 *)PMU_DMEM_CH_RANGE;
+ u32 min_ch;
+ u32 max_ch;
+ u32 ch_limit = pmu_sdata()->dbyte_slice_cnt;
+ u32 ch;
+
+ min_ch = range[0];
+ max_ch = range[1];
+ for (ch = 0U; ch < ch_limit; ch++) {
+ if (ch < min_ch || ch > max_ch)
+ *(__volatile__ u16 *)(PHY_REG_DBYTE_UPPER_GATE |
+ (ch << 13)) = 0U;
+ }
+}
+
+/**
+ * pmu_cal_dbyte_delay_coarse_step() - Advance DBYTE coarse delay across channels
+ */
+static noinline void pmu_cal_dbyte_delay_coarse_step(void)
+{
+ const __volatile__ u8 *range =
+ (const __volatile__ u8 *)PMU_DMEM_CH_RANGE;
+ u32 min_ch;
+ u32 max_ch;
+ u32 ch;
+
+ min_ch = range[0];
+ max_ch = range[1];
+ for (ch = min_ch; ch <= max_ch; ch++) {
+ __volatile__ u16 *reg =
+ (__volatile__ u16 *)(PHY_REG_DBYTE_COARSE |
+ (ch << 13));
+ u16 cur = *reg;
+ u16 val = (u16)(cur + 0x20U);
+
+ if ((val & 0x03c0U) != 0U)
+ val = (u16)((cur - 0x20U) | 0x0400U);
+ *reg = val;
+ }
+}
+
+/**
+ * pmu_cal_eye_param_block_init() - Initialize 8-halfword 2D eye parameter block
+ * @param_block: Output 8-halfword parameter block
+ * @p1: First phase/delay parameter
+ * @p2: Second phase/delay parameter
+ * @p3: Control/rank selector parameter
+ */
+void pmu_cal_eye_param_block_init(u16 *param_block, u32 p1, u32 p2, u32 p3)
+{
+ u16 ctrl = (u16)((p3 << 3) & 0x0010U);
+ u16 p3_hi = (u16)(p3 << 14);
+
+ param_block[1] = 0U;
+ param_block[2] = 0U;
+ param_block[5] = 0U;
+ param_block[6] = 0U;
+ param_block[0] = (u16)((p1 << 7) | p3_hi | 88U);
+ param_block[3] = ctrl;
+ param_block[4] = (u16)(((p2 >> 1) & 0x001fU) | ((p2 << 7) & 0x3f80U) |
+ p3_hi | 8U);
+ param_block[7] = ctrl;
+}
+
+/**
+ * pmu_cal_marker_verify_and_buffer_clear() - Clear eye buffer and verify markers
+ * @flags: Stage bitmask controlling bit 11 and bit 7 clear/verify passes
+ */
+void pmu_cal_marker_verify_and_buffer_clear(u32 flags)
+{
+ __volatile__ u32 *buf = (__volatile__ u32 *)PMU_DMEM_EYE_CLEAR_BUF;
+ u32 zero = 0U;
+
+ if ((flags & (1UL << 11)) != 0UL) {
+ ((__volatile__ u16 *)buf)[10] |= 1U;
+ if ((pmu_sdata()->cal_marker_a & 0x07U) != 0U) {
+ buf[0] = zero;
+ buf[1] = zero;
+ buf[2] = zero;
+ buf[3] = zero;
+ }
+ pmu_cal_marker_verify();
+ } else {
+ if ((pmu_sdata()->cal_marker_a & 0x07U) == 0U)
+ pmu_cal_marker_verify();
+ }
+ if ((flags & (1UL << 7)) != 0UL) {
+ if ((pmu_sdata()->cal_marker_a & 0x07U) != 0U) {
+ buf[0] = zero;
+ buf[1] = zero;
+ buf[2] = zero;
+ buf[3] = zero;
+ }
+ pmu_cal_marker_verify();
+ }
+}
+
+/**
+ * pmu_cal_dbyte_delay_restore_all_ch() - Restore DBYTE WCK delays across channels
+ */
+static void pmu_cal_dbyte_delay_restore_all_ch(void)
+{
+ u32 slice = pmu_sdata()->active_slice_idx;
+ const __volatile__ u16 *save =
+ (const __volatile__ u16 *)PMU_DMEM_DBYTE_DLY_SAVE;
+ const u8 *range = (const u8 *)PMU_DMEM_CH_ACTIVE_RANGE;
+ u32 ch;
+
+ *(__volatile__ u16 *)(PHY_REG_CAL_STAT_CTRL2 + (slice << 1)) =
+ dmem_read16(PMU_DMEM_CAL_SAVE_CFG_ADDR);
+
+ if (dmem_read8(PMU_DMEM_CH0_ANALOG_EN) != 0U) {
+ for (ch = range[0]; ch <= range[1]; ch++)
+ *(__volatile__ u16 *)(PHY_REG_DBYTE_WCK_DLY |
+ (ch << 13)) = save[ch];
+ }
+ if (dmem_read8(PMU_DMEM_CH1_ACTIVE_FLAG) != 0U) {
+ for (ch = range[2]; ch <= range[3]; ch++)
+ *(__volatile__ u16 *)(PHY_REG_DBYTE_WCK_DLY |
+ (ch << 13)) = save[ch];
+ }
+}
+
+/**
+ * pmu_cal_dbyte_delay_backup_all_ch() - Backup DBYTE WCK delays across channels
+ */
+void pmu_cal_dbyte_delay_backup_all_ch(void)
+{
+ u32 slice = pmu_sdata()->active_slice_idx;
+ __volatile__ u16 *stat_reg =
+ (__volatile__ u16 *)(PHY_REG_CAL_STAT_CTRL2 + (slice << 1));
+ u16 orig = *stat_reg;
+ __volatile__ u16 *save =
+ (__volatile__ u16 *)PMU_DMEM_DBYTE_DLY_SAVE;
+ const u8 *range = (const u8 *)PMU_DMEM_CH_ACTIVE_RANGE;
+ u32 reg_base = PHY_REG_DBYTE_WCK_DLY;
+ u32 ch;
+
+ dmem_write16(PMU_DMEM_CAL_SAVE_CFG_ADDR, orig);
+ *stat_reg = (u16)(orig & 0xff9fU);
+
+ if (dmem_read8(PMU_DMEM_CH0_ANALOG_EN) != 0U) {
+ for (ch = range[0]; ch <= range[1]; ch++)
+ save[ch] = *(__volatile__ const u16 *)
+ (reg_base | (ch << 13));
+ }
+ if (dmem_read8(PMU_DMEM_CH1_ACTIVE_FLAG) != 0U) {
+ for (ch = range[2]; ch <= range[3]; ch++)
+ save[ch] = *(__volatile__ const u16 *)
+ (reg_base | (ch << 13));
+ }
+}
+
+/**
+ * pmu_cal_vref_trim_cfg_latch() - Save AC slice calibration status registers
+ */
+void pmu_cal_vref_trim_cfg_latch(void)
+{
+ __volatile__ u8 *base = (__volatile__ u8 *)PMU_DMEM_CAL_TABLE;
+ u32 s;
+
+ for (s = 0U; s < 2U; s++) {
+ const __volatile__ u16 *ac =
+ (const __volatile__ u16 *)(PHY_REG_AC_SLICE_CFG |
+ (s << 13));
+ __volatile__ u16 *bh = (__volatile__ u16 *)(base + (s << 1));
+ __volatile__ u8 *bb = base + s;
+
+ bh[4] = ac[20];
+ bh[10] = ac[18];
+ bh[12] = ac[19];
+ bb[0x0e] = (u8)ac[5];
+ bb[0x10] = (u8)ac[6];
+ bb[0x0c] = (u8)ac[0];
+ }
+}
+
+/**
+ * pmu_cal_dbyte_sample_ready_poll_all() - Configure DBYTE slice and lane masks
+ * @flags: Active lane bitmask (0xff for broadcast gate, or bits 0..8 per lane)
+ */
+void pmu_cal_dbyte_sample_ready_poll_all(u32 flags)
+{
+ const __volatile__ u8 *range =
+ (const __volatile__ u8 *)PMU_DMEM_CH_RANGE;
+ u32 min_ch;
+ u32 max_ch;
+ u32 ch;
+
+ min_ch = range[0];
+ max_ch = range[1];
+ *(__volatile__ u16 *)PHY_REG_DBYTE_UPPER_SAMPLE_B = 0U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_UPPER_SAMPLE_A = 1U;
+
+ for (ch = min_ch; ch <= max_ch; ch++) {
+ u32 base = PHY_REG_DBYTE_SAMPLE_RDY | (ch << 13);
+
+ if ((u8)flags == 0xffU) {
+ *(__volatile__ u16 *)(base | 0x1e00UL) = 1U;
+ if ((flags & 0x0100U) == 0U)
+ *(__volatile__ u16 *)PHY_REG_DBYTE_UPPER_SAMPLE_A =
+ 0U;
+ } else {
+ u32 bitmask = flags & 0x01ffU;
+
+ for (; bitmask != 0U; bitmask >>= 1, base += 0x0200UL) {
+ if ((bitmask & 1U) != 0U)
+ *(__volatile__ u16 *)base = 1U;
+ }
+ }
+ }
+}
+
+/**
+ * pmu_cal_vref_search_center_update() - Program calibrated RX VREF across DBYTE slices
+ */
+void pmu_cal_vref_search_center_update(void)
+{
+ u32 count = pmu_sdata()->active_slice_cnt;
+ u32 csr_ch_off = pmu_sdata()->active_csr_offset;
+ u8 ch1_en_mask = dmem_read8(PMU_DMEM_CH1_ACTIVE_FLAG);
+ u8 ch0_en_mask = dmem_read8(PMU_DMEM_CH0_ANALOG_EN);
+ u32 half_count = count >> 1;
+ const __volatile__ u16 *p =
+ (const __volatile__ u16 *)PMU_DMEM_CAL_TABLE;
+ u32 i;
+
+ for (i = 0U; i <= count; i++) {
+ u16 val;
+
+ if (ch1_en_mask != 0U && i >= half_count) {
+ val = p[1];
+ if (ch1_en_mask == 3U && p[3] > val)
+ val = p[3];
+ } else {
+ val = p[0];
+ if (ch0_en_mask == 3U && p[2] > val)
+ val = p[2];
+ }
+
+ *(__volatile__ u16 *)(PHY_REG_DBYTE_BASE |
+ (((csr_ch_off | (i << 12)) << 1))) = val;
+ }
+}
+
+/**
+ * pmu_cal_pad_timing_step_settle() - Latch deskew pad timing and settle 50 ms
+ */
+void pmu_cal_pad_timing_step_settle(void)
+{
+ pmu_phy_deskew_dly0_set(300U);
+ pmu_phy_deskew_dly1_set(pmu_sdata()->deskew_target_dly);
+ *(__volatile__ u16 *)PHY_REG_PLL_CLK_CTRL = 1U;
+ pmu_cal_slice_timing_latch(0U, 0U);
+ pmu_delay_us(50000UL, 15U);
+}
+
+/**
+ * pmu_cal_dbyte_bcast_pulse_width_clear() - Apply descriptor pulse width config
+ * @cfg: Pointer to slice calibration descriptor
+ */
+void pmu_cal_dbyte_bcast_pulse_width_clear(const u8 *cfg)
+{
+ u32 offset = pmu_sdata()->active_slice_idx << 1;
+ u8 cfg_word = cfg[0x14];
+ u8 bist_flag = cfg[0x01];
+ u16 val0 = (u16)((cfg_word & 0x0cU) | ((bist_flag >> 3) & 1U));
+ u16 is_zero = ((cfg_word & 3U) == 0U) ? 1U : 0U;
+
+ *(__volatile__ u16 *)(PHY_REG_PUB_ACK | offset) = val0;
+ *(__volatile__ u16 *)(PHY_REG_DBYTE_BCAST_WIDTH | offset) =
+ (u16)(is_zero << 1);
+ *(__volatile__ u16 *)(PHY_REG_DBYTE_BCAST_GATE | offset) = is_zero;
+}
+
+/**
+ * pmu_dbyte_deskew_phase_sample() - Sample and pack 4-step DBYTE gate status masks
+ * @ch_base: Channel group base index
+ * @out_mask: Output pointer receiving accumulated 16-bit lane mask
+ * @sel_mask: 2-bit step enable selector mask
+ */
+void pmu_dbyte_deskew_phase_sample(u32 ch_base, u32 *out_mask, u32 sel_mask)
+{
+ const u8 *b = (const u8 *)PMU_DMEM_SLICE_CNT_LIMIT;
+ u32 base_ch = b[70];
+ const u8 *slice_ptr = b + 36 + (base_ch << 4) + (ch_base << 3);
+ u32 slice_limit = b[(base_ch << 1) + ch_base];
+ u32 step;
+
+ *out_mask = 0U;
+ for (step = 0U; step < 4U; step++) {
+ const u16 *lut = (const u16 *)PMU_DMEM_DELAY_TAP_LUT +
+ step;
+ u32 s;
+
+ if ((sel_mask & (1U << (step & 1U))) == 0U)
+ continue;
+
+ pmu_phy_deskew_dly0_set(lut[0]);
+ pmu_phy_deskew_dly1_set(lut[6]);
+ pmu_cal_slice_timing_latch(0U, 1U);
+
+ for (s = 0U; s < slice_limit; s++) {
+ u8 slice_off = slice_ptr[s];
+ u16 raw_val = *(__volatile__ const u16 *)
+ (PHY_REG_DBYTE_GATE_STAT |
+ ((u32)slice_off << 13));
+ u32 mapped;
+
+ mapped = pmu_map_mask(raw_val, slice_off) & 0x7fU;
+
+ if (step < 2U)
+ mapped ^= 0x7fU;
+ *out_mask |= (mapped << ((step & 1U) << 3));
+ }
+ }
+}
+
+/**
+ * pmu_dbyte_cal_step_latch() - Format and stream 16-halfword fine/coarse descriptor
+ * @fine_val: Fine delay value (low 7 bits)
+ * @coarse_val: Coarse delay value (low 7 bits + upper step bits)
+ * @ch_flags: Channel/slice configuration flags
+ */
+void pmu_dbyte_cal_step_latch(u32 fine_val, u32 coarse_val, u32 ch_flags)
+{
+ u32 buf32[8];
+ u32 ctrl_hi = ((ch_flags << 3) & 0x10U) << 16;
+ u32 slice_cfg_field = (ch_flags & 0x03U) << 14;
+ u32 a0 = fine_val & 0x7fU;
+ u32 a1 = coarse_val & 0x7fU;
+ u32 step_val_mid = ((coarse_val >> 1) & 0xffc0U) | 8U;
+
+ buf32[0] = slice_cfg_field | 0x2c58U;
+ buf32[1] = ctrl_hi;
+ buf32[2] = a0 | (a0 << 7);
+ buf32[3] = 0U;
+ buf32[4] = (u16)((step_val_mid << 7) | slice_cfg_field | step_val_mid);
+ buf32[5] = ctrl_hi;
+ buf32[6] = a1 | (a1 << 7);
+ buf32[7] = 0U;
+
+ pmu_cal_tracker_reset_and_apb_zero((u16 *)buf32, 16U);
+}
+
+/**
+ * pmu_ac_step_coord_adjust() - Adjust AC timing step coordinate by signed offset
+ * @struct_ptr: Pointer to 132-byte eye/AC scan descriptor structure
+ *
+ * Return: 0 on valid step window, or 7 on out-of-bounds error.
+ */
+static int pmu_ac_step_coord_adjust(u8 *struct_ptr)
+{
+ s8 shift_cfg = *(__volatile__ const s8 *)PMU_DMEM_SIGNED_STEP;
+ s32 step_shift = shift_cfg >> 1;
+ s8 cur_step;
+ u8 target_val;
+ s32 edge = 0;
+ s32 new_step;
+ s8 step_idx;
+ u8 lower, upper;
+
+ if (step_shift == 0)
+ return 0;
+
+ cur_step = (s8)struct_ptr[130];
+ target_val = struct_ptr[131];
+
+ if (step_shift > 0) {
+ if (cur_step >= 0) {
+ s32 k;
+
+ for (k = cur_step; k >= 0; k--) {
+ lower = struct_ptr[k * 2];
+ upper = struct_ptr[k * 2 + 1];
+ if (upper < target_val || target_val < lower) {
+ edge = k;
+ break;
+ }
+ }
+ }
+ } else {
+ if (cur_step <= 0x3f) {
+ s32 k;
+
+ for (k = cur_step; k < 0x40; k++) {
+ lower = struct_ptr[k * 2];
+ upper = struct_ptr[k * 2 + 1];
+ if (upper < target_val || target_val < lower) {
+ edge = k;
+ break;
+ }
+ if (k == 0x3f)
+ edge = 0x3f;
+ }
+ }
+ }
+
+ new_step = edge + step_shift;
+ struct_ptr[130] = (u8)new_step;
+
+ step_idx = (s8)new_step;
+ if (step_idx < 0 || step_idx >= 0x40)
+ goto err;
+
+ lower = struct_ptr[step_idx * 2];
+ upper = struct_ptr[step_idx * 2 + 1];
+ if (upper < target_val || target_val < lower)
+ goto err;
+
+ return 0;
+
+err:
+ return 7;
+}
+
+/**
+ * pmu_ac_step_pair_align_verify() - Align pair of AC timing step descriptors and verify
+ * @struct_a: First 132-byte eye/AC scan descriptor structure
+ * @struct_b: Second 132-byte eye/AC scan descriptor structure
+ *
+ * Return: 0 on valid alignment, or non-zero error code (7 on window violation).
+ */
+static int pmu_ac_step_pair_align_verify(u8 *struct_a, u8 *struct_b)
+{
+ s8 shift_cfg = *(__volatile__ const s8 *)PMU_DMEM_SIGNED_STEP;
+ s32 freq_pt = shift_cfg >> 1;
+ int ret;
+ u16 base_b, base_a;
+ s8 step_b, step_a;
+ u16 total_b, total_a;
+ s8 final_a, final_b;
+ u8 target_a, target_b;
+
+ if (freq_pt == 0)
+ return 0;
+
+ ret = pmu_ac_step_coord_adjust(struct_a);
+ if (ret != 0)
+ return ret;
+
+ ret = pmu_ac_step_coord_adjust(struct_b);
+ if (ret != 0)
+ return ret;
+
+ base_b = *(__volatile__ const u16 *)(struct_b + 128);
+ step_b = (s8)struct_b[130];
+ base_a = *(__volatile__ const u16 *)(struct_a + 128);
+ step_a = (s8)struct_a[130];
+
+ total_b = (u16)(step_b + base_b);
+ total_a = (u16)(step_a + base_a);
+
+ if (total_b >= total_a) {
+ if (freq_pt < 1)
+ struct_b[130] = (u8)(total_a - base_b);
+ else
+ struct_a[130] = (u8)(total_b - base_a);
+ } else {
+ if (freq_pt < 1)
+ struct_a[130] = (u8)(total_b - base_a);
+ else
+ struct_b[130] = (u8)(total_a - base_b);
+ }
+
+ final_a = (s8)struct_a[130];
+ target_a = struct_a[131];
+ if (target_a < struct_a[final_a * 2] ||
+ struct_a[final_a * 2 + 1] < target_a)
+ return 7;
+
+ final_b = (s8)struct_b[130];
+ target_b = struct_b[131];
+ if (target_b < struct_b[final_b * 2] ||
+ struct_b[final_b * 2 + 1] < target_b)
+ return 7;
+
+ return 0;
+}
+
+/**
+ * pmu_phy_delay_bound_margin_log() - Measure DBYTE period and compute gate stage margin
+ * @slice: Target DBYTE slice index
+ *
+ * Return: Computed 16-bit period-normalized gate delay difference.
+ */
+u16 pmu_phy_delay_bound_margin_log(u32 slice)
+{
+ u32 base = PHY_REG_DBYTE_BASE | (slice << 13);
+ __volatile__ u16 *sreg;
+ const __volatile__ u16 *cbase;
+ u16 val;
+ u32 delay_stage;
+ u32 stage_offset = 0U;
+ u32 period_val;
+ u32 quot, bound = 160U, diff;
+
+ sreg = (__volatile__ u16 *)base;
+ val = sreg[0x158 >> 1];
+ sreg[0x158 >> 1] = val | 1U;
+ sreg[0x158 >> 1] = val & ~1U;
+
+ cbase = (const __volatile__ u16 *)(base | (pmu_sdata()->active_csr_offset << 1));
+ delay_stage = cbase[0x15a >> 1];
+
+ if (delay_stage < 5U)
+ stage_offset = cbase[delay_stage + 0xd0U];
+
+ period_val = ((const __volatile__ u16 *)base)[0x1aa >> 1];
+ quot = period_val ?
+ ((((period_val * delay_stage) - stage_offset) << 6) / period_val) :
+ 0U;
+
+ if (pmu_sdata()->freq_mode >= 2U)
+ bound += 128U;
+ diff = bound - quot;
+
+ return (u16)diff;
+}
+
+/**
+ * pmu_phy_profile_param_program() - Program AC slice profile parameters and settle
+ */
+void pmu_phy_profile_param_program(void)
+{
+ __volatile__ u16 *phy_base = (__volatile__ u16 *)PHY_REG_MASTER_BASE;
+ u16 saved_gate_mask = phy_base[0x00f4 >> 1];
+ u16 saved_gate_ctrl = phy_base[0x00fc >> 1];
+
+ phy_base[0x00fc >> 1] = saved_gate_ctrl & 0xff33U;
+ phy_base[0x00f4 >> 1] = saved_gate_mask | 0x00ccU;
+
+ if (*(__volatile__ const u8 *)PMU_DMEM_AC_PROFILE_EN_ADDR != 0U) {
+ u32 csr_offset;
+ u32 slice;
+
+ csr_offset = *(__volatile__ const u32 *)
+ PMU_DMEM_ACTIVE_CH_OFFSET;
+
+ for (slice = 0U; slice < 2U; slice++) {
+ u32 slice_csr_off = ((slice << 12) | csr_offset) << 1;
+ __volatile__ u16 *slice_base = (__volatile__ u16 *)
+ (slice_csr_off | PHY_REG_AC_BASE);
+ const __volatile__ u8 *ac_dly_ptr =
+ (const __volatile__ u8 *)
+ (PMU_DMEM_AC_PROFILE_DLY_ADDR + slice * 2U);
+ u16 val0 = *(const __volatile__ u16 *)ac_dly_ptr;
+ const __volatile__ u8 *ac_cfg_ptr =
+ (const __volatile__ u8 *)
+ (PMU_DMEM_AC_PROFILE_CFG_ADDR + slice);
+
+ slice_base[0x0084 >> 1] = val0;
+ slice_base[0x0086 >> 1] = val0;
+ slice_base[0x0080 >> 1] =
+ *(const __volatile__ u16 *)(ac_dly_ptr + 0x0cU);
+ slice_base[0x0082 >> 1] =
+ *(const __volatile__ u16 *)(ac_dly_ptr + 0x10U);
+
+ slice_base[0x005c >> 1] = ac_cfg_ptr[0];
+ slice_base[0x0066 >> 1] = ac_cfg_ptr[2];
+ slice_base[0x0068 >> 1] = ac_cfg_ptr[4];
+ }
+ }
+
+ phy_base[0x0084 >> 1] = 0U;
+ phy_base[0x0088 >> 1] = 0U;
+ pmu_delay_us(11000U, 0U);
+
+ phy_base[0x00f4 >> 1] = saved_gate_mask;
+ phy_base[0x00fc >> 1] = saved_gate_ctrl;
+}
+
+/**
+ * pmu_dbyte_cal_strobe_pulse_seq() - Reset or trigger DBYTE broadcast strobe sequence
+ * @mode: 0 to de-assert/reset strobe registers, non-zero to trigger strobe
+ * @val: Broadcast strobe delay halfword value
+ * @step: Broadcast strobe step mode (0 or 1; > 1 triggers assertion)
+ */
+void pmu_dbyte_cal_strobe_pulse_seq(u32 mode, u32 val, u32 step)
+{
+ __volatile__ u16 *apb =
+ (__volatile__ u16 *)PHY_REG_DBYTE_BCAST_STROBE;
+ u32 csr_off = pmu_sdata()->active_csr_offset << 1;
+ __volatile__ u16 *bcast_width =
+ (__volatile__ u16 *)(PHY_REG_DBYTE_BCAST_WIDTH + csr_off);
+ __volatile__ u16 *bcast_gate =
+ (__volatile__ u16 *)(PHY_REG_DBYTE_BCAST_GATE + csr_off);
+
+ if ((u8)mode == 0U) {
+ *bcast_width = 0U;
+ *bcast_gate = 0U;
+ pmu_hw_timer_delay(20U);
+ phy_write16(PHY_REG_DBYTE_BCAST_VREF, 1U);
+ apb[74] = 0U;
+ apb[76] = 0U;
+ apb[84] = 0U;
+ apb[73] = 0U;
+ } else {
+ apb[74] = (u16)val;
+ apb[73] = (u16)val;
+ apb[75] = 0U;
+ if ((u8)step == 0U) {
+ phy_write16(PHY_REG_DBYTE_BCAST_VREF, 0U);
+ *bcast_gate = 1U;
+ } else if ((u8)step == 1U) {
+ phy_write16(PHY_REG_DBYTE_BCAST_VREF, 0U);
+ apb[85] = 10U;
+ } else {
+ pmu_assert_or_halt(0U, PMU_ERR_VREF_STEP_DIR_INVALID);
+ }
+ apb[84] = (u16)step;
+ *bcast_width = 2U;
+ }
+}
+
+/**
+ * pmu_rank_shadow_status_sample() - Sample shadow calibration status across ranks
+ */
+void pmu_rank_shadow_status_sample(void)
+{
+ u32 total_idx = 0U;
+ u8 *table_base = (u8 *)PMU_DMEM_EYE_SWEEP_STAT;
+ u32 rank;
+
+ for (rank = 0U; rank < 2U; rank++) {
+ u8 active = (rank == 0U) ?
+ dmem_read8(PMU_DMEM_CH0_ANALOG_EN) :
+ dmem_read8(PMU_DMEM_CH1_ACTIVE_FLAG);
+
+ if (active != 0U) {
+ u16 base_val = dmem_read16(PMU_DMEM_PLL_GATE_CH0_ADDR +
+ rank * 2U);
+ u32 base_reg = ((u32)base_val << 2) | 0x00041000UL;
+ u32 freq_pt;
+
+ for (freq_pt = 0U; freq_pt < 2U; freq_pt++) {
+ u8 count = dmem_read8(PMU_DMEM_METRIC_TABLE_ENTRIES);
+ u8 *entry = table_base + freq_pt;
+ u32 entry_idx;
+
+ for (entry_idx = 0U; entry_idx < count;
+ entry_idx++) {
+ u16 val = *(__volatile__ const u16 *)
+ (PHY_REG_PUB_BASE |
+ ((base_reg +
+ (u8)total_idx) << 1));
+
+ *entry = (u8)val;
+ entry += 4U;
+ total_idx++;
+ }
+ }
+ }
+ table_base += 2U;
+ }
+}
+
+/**
+ * pmu_phy_lcdl_delay_read() - Read and unpack DBYTE coarse/fine lane delays
+ * @out_arr: Output halfword array
+ * @flag: Non-zero to unpack and sum 10-bit fine and 6-bit coarse subfields
+ * @mode: Lane range/stride selector (1 = lane 8 stride 9, 2 = lane 0, else 0..8)
+ */
+void pmu_phy_lcdl_delay_read(u16 *out_arr, u32 flag, u32 mode)
+{
+ const u8 *ch_range = (const u8 *)PMU_DMEM_CH_RANGE;
+ u8 start_ch, end_ch;
+ u32 start_lane = 0U, end_lane = 8U, stride = 1U;
+ u16 *out;
+ u32 ch, lane;
+
+ start_ch = ch_range[0];
+ end_ch = ch_range[1];
+
+ if (mode == 2U) {
+ end_lane = 0U;
+ } else if (mode == 1U) {
+ stride = 9U;
+ start_lane = 8U;
+ }
+
+ out = out_arr + ((mode == 2U) ? start_ch :
+ (start_ch * 9U + ((mode == 1U) ? 8U : 0U)));
+
+ for (ch = start_ch; ch <= end_ch; ch++) {
+ for (lane = start_lane; lane <= end_lane; lane++) {
+ u32 reg = PHY_REG_DBYTE_COARSE |
+ (ch << 13) | (lane << 9);
+ u16 val = *(__volatile__ const u16 *)reg;
+
+ if (flag != 0U)
+ val = (val & 0x03ffU) + ((val >> 4) & 0x0fc0U);
+ *out = val;
+ out += stride;
+ }
+ }
+}
+
+/**
+ * pmu_eye_margin_step_align() - Align coarse/fine eye margin pair with clamping
+ * @margin_cur: First 2-byte [coarse, fine] margin pair
+ * @margin_target: Second 2-byte [coarse, fine] margin pair
+ * @step_size: Coarse alignment step size
+ */
+void pmu_eye_margin_step_align(u8 *margin_cur, u8 *margin_target,
+ u32 step_size)
+{
+ u8 v0 = margin_cur[0];
+ u8 v1 = margin_target[0];
+ u8 step = (u8)step_size;
+
+ if (v0 > v1) {
+ if (v0 < step) {
+ margin_cur[0] = v0 - 1U;
+ margin_cur[1] = (u8)(margin_cur[1] + 64U);
+ if ((s8)margin_cur[1] < 0)
+ margin_cur[1] = 0x7fU;
+ } else {
+ margin_cur[0] = v0 - step;
+ if (step == 1U) {
+ margin_cur[1] = (u8)(margin_cur[1] + 64U);
+ if ((s8)margin_cur[1] < 0)
+ margin_cur[1] = 0x7fU;
+ } else if (step == 2U || (s8)margin_cur[1] < 0) {
+ margin_cur[1] = 0x7fU;
+ }
+ }
+ } else if (v0 < v1) {
+ if (v1 < step) {
+ margin_target[0] = v1 - 1U;
+ margin_target[1] = (u8)(margin_target[1] + 64U);
+ if ((s8)margin_target[1] < 0)
+ margin_target[1] = 0x7fU;
+ } else {
+ margin_target[0] = v1 - step;
+ if (step == 1U) {
+ margin_target[1] = (u8)(margin_target[1] + 64U);
+ if ((s8)margin_target[1] < 0)
+ margin_target[1] = 0x7fU;
+ } else if (step == 2U || (s8)margin_target[1] < 0) {
+ margin_target[1] = 0x7fU;
+ }
+ }
+ }
+
+ pmu_assert_or_halt(margin_cur[0] == margin_target[0],
+ PMU_ERR_EYE_STEP_ALIGN_MISMATCH);
+}
+
+/**
+ * pmu_cbt_3phase_pulse_seq() - Execute 3-step fixed-unit pad timing delay sequence
+ */
+void pmu_cbt_3phase_pulse_seq(void)
+{
+ pmu_cal_pad_timing_delay_set(0U, 0x10U, 1U);
+ pmu_cal_pad_timing_delay_set(0x10U, 0U, 1U);
+ pmu_cal_pad_timing_delay_set(0U, 0x0aU, 1U);
+}
+
+/**
+ * pmu_eye_margin_bidir_scan() - Scan 2D eye margins and update minimum X/Y bounds
+ * @buf: Pointer to 132-byte eye scan buffer (center at +130, threshold at +131)
+ * @flag_a: Rank/slice flag A (+2 offset when == 1)
+ * @flag_b: Channel/group flag B (+27 offset when != 0)
+ * @flag_c: Mode flag C (+4 offset when == 1)
+ */
+static void pmu_eye_margin_bidir_scan(u8 *buf, u32 flag_a, u32 flag_b, u32 flag_c)
+{
+ u8 center = buf[130];
+ u8 th = buf[131];
+ s32 r0_left = (s32)center;
+ s32 r0_right = (s32)center;
+ u32 dest = (flag_b != 0U) ? 0x52U : 0x37U;
+ s32 margin_x_left, margin_x_right, min_x;
+ s32 margin_y_lo, margin_y_hi, min_y;
+ u8 *d_ptr;
+
+ for (; r0_left >= 0; r0_left--) {
+ if (buf[r0_left * 2] >= th || th >= buf[r0_left * 2 + 1])
+ break;
+ }
+
+ if (center <= 0x3fU) {
+ for (; r0_right < 64; r0_right++) {
+ if (buf[r0_right * 2] >= th ||
+ th >= buf[r0_right * 2 + 1])
+ break;
+ }
+ }
+
+ if (flag_c == 1U)
+ dest += 4U;
+ if (flag_a == 1U)
+ dest += 2U;
+ dest |= PMU_DMEM_BASE;
+ d_ptr = (u8 *)dest;
+
+ margin_x_left = (s32)center - r0_left;
+ margin_x_right = r0_right - (s32)center;
+ min_x = (margin_x_left < margin_x_right) ?
+ margin_x_left : margin_x_right;
+
+ margin_y_lo = (s32)th - (s32)buf[center * 2];
+ margin_y_hi = (s32)buf[center * 2 + 1] - (s32)th;
+ min_y = (margin_y_lo < margin_y_hi) ? margin_y_lo : margin_y_hi;
+
+ if ((u8)min_x < d_ptr[0])
+ d_ptr[0] = (u8)min_x;
+ if ((u8)min_y < d_ptr[1])
+ d_ptr[1] = (u8)min_y;
+}
+
+/**
+ * pmu_rank_slice_margin_eval() - Evaluate AC timing step margins across active ranks/slices
+ */
+void pmu_rank_slice_margin_eval(void)
+{
+ u32 mode = pmu_phy_query_operating_mode();
+ u32 rank;
+
+ for (rank = 0U; rank < 2U; rank++) {
+ u8 ch0_ranks = *(__volatile__ const u8 *)PMU_DMEM_CH0_ANALOG_EN;
+ u8 ch1_ranks = *(__volatile__ const u8 *)
+ PMU_DMEM_CH1_ACTIVE_FLAG;
+ const u8 *ch_range = (const u8 *)PMU_DMEM_CH_RANGE;
+ u32 slice, end_slice;
+
+ if (((ch0_ranks | ch1_ranks) & (1U << rank)) == 0U)
+ continue;
+
+ slice = ch_range[0];
+ end_slice = ch_range[1];
+
+ dmem_write8(PMU_DMEM_STAGE_RANK_IDX_ADDR, (u8)rank);
+
+ for (; slice <= end_slice; slice++) {
+ u8 mask = ch_range[0x30];
+ u8 *ac_desc_ptr;
+ u8 *eye_sample_ptr;
+ u32 lane;
+
+ if ((mask & (1U << slice)) == 0U)
+ continue;
+
+ dmem_write8(PMU_DMEM_STAGE_SLICE_IDX_ADDR, (u8)slice);
+
+ ac_desc_ptr = (u8 *)(PMU_DMEM_AC_TIMING_DESC +
+ rank * 5280U + slice * 1320U);
+ eye_sample_ptr = (u8 *)(PMU_DMEM_AC_TIMING_TBL +
+ rank * 5280U + slice * 1320U);
+
+ for (lane = 0U; lane < 10U; lane++) {
+ if ((mode & (1U << lane)) != 0U) {
+ int ret;
+
+ dmem_write8(PMU_DMEM_STAGE_LANE_IDX_ADDR,
+ (u8)lane);
+ if (mode == 3U)
+ ret = pmu_ac_step_pair_align_verify(eye_sample_ptr,
+ ac_desc_ptr);
+ else
+ ret = pmu_ac_step_coord_adjust(eye_sample_ptr);
+
+ pmu_assert_or_halt(ret == 0,
+ PMU_ERR_AC_STEP_COORD_ADJUST);
+ }
+ ac_desc_ptr += 132U;
+ eye_sample_ptr += 132U;
+ }
+ }
+ }
+}
+
+struct pmu_eye_sample_entry {
+ s16 min_x;
+ s16 max_x;
+ s16 min_y;
+ s16 max_y;
+};
+
+/**
+ * pmu_eye_sample_table_transform() - Transform 64-pair eye sample array into 4-bound table
+ * @dest: Output 512-byte (64 x 8-byte) eye sample bound array
+ * @src: Input 128-byte (64 x 2-byte) raw eye sample pair array
+ */
+static void pmu_eye_sample_table_transform(u8 *dest, const u8 *src)
+{
+ struct pmu_eye_sample_entry *d = (struct pmu_eye_sample_entry *)dest;
+ u32 i;
+ u8 s0, s1, tail_lo, tail_hi;
+
+ for (i = 0U; i < 64U; i++) {
+ d[i].min_x = 0x7f;
+ d[i].max_x = 0;
+ d[i].min_y = 0x7f;
+ d[i].max_y = 0;
+ }
+
+ for (i = 1U; i < 63U; i++) {
+ const u8 *s_curr = src + (i << 1);
+ const u8 *s_prev = s_curr - 2;
+ const u8 *s_next = s_curr + 2;
+ u8 c0 = s_curr[0];
+ u8 c1 = s_curr[1];
+
+ if (c1 >= c0) {
+ s16 min_x = (s16)(s_next[0] + 1);
+ s16 max_y = (s16)(s_prev[1] - 1);
+
+ d[i].max_x = (s16)(c0 - 1);
+ if (min_x > (s16)(s_prev[0] + 1))
+ min_x = (s16)(s_prev[0] + 1);
+ if (min_x > (s16)c0)
+ min_x = (s16)c0;
+ d[i].min_x = min_x - 1;
+
+ d[i].min_y = (s16)(c1 + 1);
+ if (max_y < (s16)(s_next[1] - 1))
+ max_y = (s16)(s_next[1] - 1);
+ if (max_y < (s16)c1)
+ max_y = (s16)c1;
+ d[i].max_y = max_y + 1;
+ } else {
+ if (s_next[1] >= s_next[0]) {
+ d[i].min_x = s_next[0];
+ d[i].max_x = s_next[1];
+ }
+ if (s_prev[1] >= s_prev[0]) {
+ d[i].min_x = s_prev[0];
+ d[i].max_x = s_prev[1];
+ }
+ }
+ }
+
+ s0 = src[0];
+ s1 = src[1];
+ if (s1 >= s0) {
+ s16 min_x = (s16)(src[2] + 1);
+ s16 max_y = (s16)(src[3] - 1);
+
+ d[0].max_x = (s16)(s0 - 1);
+ if (min_x > (s16)s0)
+ min_x = (s16)s0;
+ d[0].min_x = min_x - 1;
+ d[0].min_y = (s16)(s1 + 1);
+ if (max_y < (s16)s1)
+ max_y = (s16)s1;
+ d[0].max_y = max_y + 1;
+ } else if (src[3] >= src[2]) {
+ d[0].min_x = src[2];
+ d[0].max_x = src[3];
+ }
+
+ tail_lo = src[126];
+ tail_hi = src[127];
+ if (tail_hi >= tail_lo) {
+ s16 min_x = (s16)(src[124] + 1);
+ s16 max_y = (s16)(src[125] - 1);
+
+ d[63].max_x = (s16)(tail_lo - 1);
+ if (min_x > (s16)tail_lo)
+ min_x = (s16)tail_lo;
+ d[63].min_x = min_x - 1;
+ d[63].min_y = (s16)(tail_hi + 1);
+ if (max_y < (s16)tail_hi)
+ max_y = (s16)tail_hi;
+ d[63].max_y = max_y + 1;
+ } else if (src[125] >= src[124]) {
+ d[63].min_x = src[124];
+ d[63].max_x = src[125];
+ }
+
+ for (i = 1U; i < 63U; i++) {
+ if (d[i].min_x == -1) {
+ d[i].min_x = 0x7f;
+ d[i].max_x = 0;
+ }
+ }
+}
+
+/**
+ * pmu_eye_centroid_avg_calc() - Compute average, right, and left eye bounds per lane
+ * @base_delays: Per-slice base delay halfword array
+ * @out_left: Output midpoint/average delay array (40 halfwords)
+ * @out_right: Output right bound delay array (40 halfwords)
+ * @out_width: Output left bound delay array (40 halfwords)
+ */
+void pmu_eye_centroid_avg_calc(const u16 *base_delays, u16 *out_left,
+ u16 *out_right, u16 *out_width)
+{
+ u8 analog_cfg_byte = *(__volatile__ const u8 *)PMU_DMEM_ANALOG_CFG;
+ s32 min_margin = ((analog_cfg_byte & 1U) != 0U) ? -44 : -128;
+ const u8 *ch_range = (const u8 *)PMU_DMEM_CH_RANGE;
+ u8 active_slices;
+ u32 slice, end_slice;
+ s32 target_margin = (s32)0xd4 & min_margin;
+ const __volatile__ u8 *lane_base =
+ (const __volatile__ u8 *)(PMU_DMEM_BASE |
+ PMU_DMEM_SWEEP_SAMPLE_BUF);
+
+ slice = ch_range[0];
+ end_slice = ch_range[1];
+ active_slices = ch_range[0x30];
+
+ for (; slice <= end_slice; slice++) {
+ u32 lane_idx;
+ u32 lane_sample_offset;
+ u16 base_val;
+ u32 lane;
+
+ if ((active_slices & (1U << slice)) == 0U)
+ continue;
+
+ lane_idx = slice * 10U;
+ lane_sample_offset = 848U * lane_idx;
+ base_val = base_delays[slice];
+
+ for (lane = 0U; lane < 10U;
+ lane++, lane_idx++, lane_sample_offset += 848U) {
+ const __volatile__ u8 *p =
+ lane_base + lane_sample_offset;
+ u8 type = p[0x27c];
+ u32 right_bound = 0U;
+ s32 left_bound = 0;
+ s32 avg;
+
+ if (type != 0U) {
+ right_bound = p[0];
+ if (type == 1U) {
+ left_bound = min_margin;
+ } else if (type == 3U) {
+ u8 val_lo = p[1];
+ u8 val_hi = p[2];
+ s32 span = (s32)val_lo -
+ (s32)right_bound;
+ s32 rem_span = target_margin -
+ (s32)val_hi;
+
+ if (span < rem_span) {
+ right_bound = val_hi;
+ left_bound = min_margin;
+ }
+ }
+ }
+
+ avg = ((s16)right_bound + (s8)left_bound) / 2;
+ out_left[lane_idx] = base_val + (u16)avg;
+ out_right[lane_idx] = base_val + (u16)right_bound;
+ out_width[lane_idx] = base_val + (u16)(s8)left_bound;
+ }
+ }
+}
+
+/**
+ * pmu_dbyte_coarse_step_mask_sweep() - Sweep 8 lanes at coarse step and return mapped mask
+ * @ch_idx: Target DBYTE channel/slice index
+ * @step_val: Coarse delay step offset
+ * @flag: Mode 2 sub-selector flag
+ *
+ * Return: Remapped 8-bit lane status mask via pmu_map_mask().
+ */
+u32 pmu_dbyte_coarse_step_mask_sweep(u32 ch_idx, u32 step_val, u32 flag)
+{
+ u32 mode = pmu_phy_query_operating_mode();
+ u32 reg_idx = (mode == 1U || (mode == 2U && flag == 1U)) ?
+ 0x74U : 0x73U;
+ u32 ch_shift = (ch_idx & 0xffffU) << 12;
+ __volatile__ u16 *freq_pt = (__volatile__ u16 *)
+ ((ch_shift << 1) | PHY_REG_DBYTE_COARSE_STEP);
+ __volatile__ const u16 *csr_base = (__volatile__ const u16 *)
+ (PHY_REG_PUB_BASE | (((reg_idx | (1U << 16) | ch_shift)) << 1));
+ u8 pll_ratio = *(__volatile__ const u8 *)PMU_DMEM_PLL_RATIO;
+ u32 sub_val = ((pll_ratio == 2U) ? 4U : 0U) + 8U;
+ u16 table[8];
+ u32 i;
+ u32 lane_mask = 0U;
+ __volatile__ const u16 *csr;
+
+ *freq_pt = 768U;
+
+ csr = csr_base - 3;
+ for (i = 0U; i < 8U; i++) {
+ table[i] = (u16)((*csr >> 10) - sub_val);
+ csr += 0x100U;
+ }
+
+ csr = csr_base;
+ for (i = 0U; i < 8U; i++) {
+ *freq_pt = (u16)((step_val + table[i]) | 192U);
+ pmu_hw_timer_delay(8U);
+ if ((*csr & 1U) != 0U)
+ lane_mask |= (1U << i);
+ csr += 0x100U;
+ }
+
+ *freq_pt = 0U;
+ return pmu_map_mask((u16)lane_mask, ch_idx);
+}
+
+/**
+ * pmu_phy_cal_strobe_latch_setup() - Restore DBYTE delays and latch sweep delay registers
+ */
+void pmu_phy_cal_strobe_latch_setup(void)
+{
+ u16 delay_a = (0xffU << 8) + 1U;
+ u16 delay_b = (0xffU << 8) + 1U;
+ u32 freq_mode;
+
+ phy_write16(PHY_REG_CH0_ANALOG_PWR,
+ dmem_read8(PMU_DMEM_CH0_ANALOG_EN));
+ phy_write16(PHY_REG_CH1_ANALOG_PWR,
+ dmem_read8(PMU_DMEM_CH1_ACTIVE_FLAG));
+
+ pmu_cal_dbyte_delay_restore_all_ch();
+ pmu_slice_lane_status_save();
+
+ if (dmem_read16(PMU_DMEM_DRAM_FREQ_MHZ) >= 3201U) {
+ const __volatile__ u16 *phase_dly_reg =
+ (const __volatile__ u16 *)PHY_REG_CLK_PHASE_DELAY;
+
+ delay_a = (u16)((phase_dly_reg[0] << 8) + 1U);
+ delay_b = (u16)((phase_dly_reg[1] << 8) + 1U);
+ }
+
+ freq_mode = dmem_read8(PMU_DMEM_CBT_LANE_LIM);
+ if (freq_mode >= 2U) {
+ __volatile__ u16 *dst = (__volatile__ u16 *)
+ (PHY_REG_CAL_LATCH_STREAM |
+ (pmu_sdata()->active_csr_offset << 1));
+
+ if (freq_mode >= 8U) {
+ dst[0x10 >> 1] = delay_a;
+ dst[0x26 >> 1] = delay_b;
+ } else {
+ dst[0x06 >> 1] = delay_a;
+ dst[0x12 >> 1] = delay_b;
+ }
+ }
+}
+
+/**
+ * pmu_window_centroid_calc() - Compute 2D eye window centroid and weighted midpoint
+ * @table_buf: Pointer to 132-byte eye scan window buffer (outputs at +130, +131)
+ *
+ * Return: 0 on valid eye window, or 3 when total window width is 0.
+ */
+u32 pmu_window_centroid_calc(void *table_buf)
+{
+ u8 *out = (u8 *)table_buf;
+ u16 analog_cfg_word = *(__volatile__ const u16 *)(PMU_DMEM_BASE +
+ PMU_DMEM_DRAM_CFG_FLAGS);
+ u32 weighting_mode = analog_cfg_word & 0x1800U;
+ u32 weighted_sum = 0U;
+ u32 sq_weighted_sum = 0U;
+ u32 total_width = 0U;
+ u32 sq_weight_sum = 0U;
+ u32 log_weight_sum = 0U;
+ u32 log_weighted_sum = 0U;
+ u32 total_weighted = 0U;
+ u32 i;
+ u32 centroid_val, centroid_mid, weighted_avg, res_centroid;
+
+ for (i = 0U; i < 64U; i++) {
+ u8 v0 = out[i * 2U];
+ u8 v1 = out[i * 2U + 1U];
+ s32 diff = (s32)v1 - (s32)v0;
+ u32 width, sum, prod;
+
+ if (diff < 0)
+ continue;
+
+ width = (u32)diff + 1U;
+ sum = (u32)v0 + (u32)v1;
+ prod = width * sum;
+ total_width += width;
+ total_weighted += prod;
+
+ if (weighting_mode == 0U) {
+ weighted_sum += width * i;
+ } else if (weighting_mode == 0x0800U) {
+ u32 sq = width * width;
+
+ sq_weight_sum += sq;
+ sq_weighted_sum += sq * i;
+ } else {
+ u32 weight = 0U;
+ u32 w = width >> 1;
+
+ while (w > 0U) {
+ weight++;
+ w >>= 1;
+ }
+ log_weight_sum += weight;
+ log_weighted_sum += weight * i;
+ }
+ }
+
+ if (total_width == 0U)
+ return 3U;
+
+ centroid_val = (total_weighted / total_width) + 1U;
+ centroid_mid = centroid_val >> 1;
+
+ if (weighting_mode != 0U) {
+ if (weighting_mode == 0x0800U) {
+ log_weighted_sum = sq_weighted_sum;
+ log_weight_sum = sq_weight_sum;
+ }
+ total_width = log_weight_sum;
+ weighted_sum = log_weighted_sum;
+ }
+
+ if (total_width == 0U)
+ return 3U;
+
+ weighted_avg = ((weighted_sum << 1) / total_width) + 1U;
+ res_centroid = weighted_avg >> 1;
+
+ out[131] = (u8)centroid_mid;
+ out[130] = (u8)res_centroid;
+ return 0U;
+}
+
+/**
+ * pmu_slice_delay_step_program() - Program per-lane DBYTE delay step deltas across slices
+ * @reg_base_off: Base CSR register offset
+ * @step_deltas: Signed 16-bit step delta table
+ * @format_mode: Non-zero for signed div/mod 64, 0 for packed 6-bit coarse / 10-bit fine
+ * @shift_sel: Non-zero for 7-bit coarse shift, 0 for 6-bit coarse shift
+ * @stride_flag: Non-zero for 10-lane stride, 0 for 9-lane stride
+ * @single_lane_flag: Non-zero to program a single lane per slice
+ */
+void pmu_slice_delay_step_program(u32 reg_base_off, const s16 *step_deltas,
+ u32 format_mode, u32 shift_sel,
+ u32 stride_flag, u32 single_lane_flag)
+{
+ const u8 *ch_range = (const u8 *)PMU_DMEM_CH_RANGE;
+ u32 start_rank;
+ u32 end_rank;
+ u32 lane_count;
+ u32 lane_offset;
+ u32 shift_amount;
+ const s16 *deltas_ptr;
+ u32 reg_csr;
+ u32 rank;
+
+ start_rank = ch_range[0];
+ end_rank = ch_range[1];
+
+ if (single_lane_flag != 0U) {
+ lane_count = 1U;
+ lane_offset = start_rank;
+ } else {
+ lane_count = (stride_flag != 0U) ? 10U : 9U;
+ lane_offset = start_rank * lane_count;
+ }
+
+ shift_amount = (shift_sel != 0U) ? 7U : 6U;
+ deltas_ptr = step_deltas + lane_offset;
+ reg_csr = PHY_REG_DBYTE_BASE |
+ ((pmu_sdata()->active_csr_offset | reg_base_off) << 1);
+
+ for (rank = start_rank; rank <= end_rank; rank++) {
+ u32 reg_addr = reg_csr | (rank << 13);
+ u32 i;
+
+ for (i = 0U; i < lane_count; i++, reg_addr += 0x200U) {
+ s32 delta = *deltas_ptr++;
+ s32 coarse, fine;
+
+ if (format_mode != 0U) {
+ coarse = delta / 64;
+ fine = delta % 64;
+ } else {
+ coarse = (delta & (s32)0xfc00) >> 10;
+ fine = delta & (s32)0x03ff;
+ }
+
+ *(__volatile__ u16 *)reg_addr =
+ (u16)fine | (u16)((u16)coarse << shift_amount);
+ }
+ }
+}
+
+/**
+ * pmu_vref_signed_direction_pulse_dispatch() - Dispatch signed Vref direction pulse sequence
+ * @slice_target: Target slice / rank selector
+ * @vref_step: Vref step parameter
+ */
+void pmu_vref_signed_direction_pulse_dispatch(u32 slice_target, u32 vref_step)
+{
+ u32 zero = 0U;
+ u32 cmd;
+
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_VREF = zero;
+ pmu_phy_sec_cal_strobe_pulse();
+ pmu_cal_strobe_pulse_init();
+
+ cmd = zero;
+ cmd |= (1U << 21);
+ pmu_cal_sequence_pulse_send(cmd, 5U, 1U, 256U, slice_target,
+ zero, zero);
+ cmd = 0x80U;
+ cmd |= (1U << 18);
+ pmu_cal_sequence_pulse_send(cmd, 0x0fU, 0x20U, zero,
+ slice_target, zero, vref_step);
+
+ pmu_cal_slice_step_diff_latch(zero, 1U);
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_VREF = 1U;
+}
+
+/**
+ * pmu_dbyte_rx_en_slice_pulse_eval() - Evaluate DBYTE RxEn slice pulse sequence
+ * @mode: Non-zero to enable broadcast window valid before pulse, 0 to clear after
+ * @slice_mask: Active slice bitmask
+ */
+void pmu_dbyte_rx_en_slice_pulse_eval(u32 mode, u32 slice_mask)
+{
+ u32 zero = 0U;
+
+ pmu_cal_strobe_pulse_init();
+
+ if (mode != 0U)
+ *(__volatile__ u16 *)(PHY_REG_DBYTE_BCAST_PULSE |
+ (pmu_sdata()->active_csr_offset << 1)) = 1U;
+ pmu_cal_sequence_pulse_send(zero, 6U, 0x22U, mode ? 4U : 0U, 0x2eU,
+ slice_mask, zero);
+ if (mode == 0U)
+ *(__volatile__ u16 *)(PHY_REG_DBYTE_BCAST_PULSE |
+ (pmu_sdata()->active_csr_offset << 1)) = zero;
+
+ pmu_cal_slice_step_diff_latch(zero, 1U);
+}
+
+/**
+ * pmu_rank_margin_channel_accum() - Accumulate rank/slice margin window and sum widths
+ * @rank: Active rank index (0 or 1)
+ * @start_slice: Starting DBYTE slice index
+ * @end_slice: Ending DBYTE slice index
+ * @flags: Frequency point count (defaults to 1 when 0)
+ * @swap_mode: DQ swap filter mode (0, 1, or 2)
+ *
+ * Return: Accumulated 16-bit window width across 64 delay coordinates.
+ */
+u16 pmu_rank_margin_channel_accum(u32 rank, u32 start_slice, u32 end_slice,
+ u32 flags, u32 swap_mode)
+{
+ u8 temp_buf[128];
+ u32 rank_offset = rank * 0x14a0U;
+ u32 iter_lim = (flags != 0U) ? flags : 1U;
+ u32 slice;
+ u32 total_width = 0U;
+ int i;
+
+ for (i = 0; i < 64; i++) {
+ temp_buf[i * 2] = 0x00U;
+ temp_buf[i * 2 + 1] = 0x7fU;
+ }
+
+ for (slice = start_slice; slice <= end_slice; slice++) {
+ const u8 *sub;
+ u32 freq_pt;
+
+ if (swap_mode == 2U) {
+ u8 b = (u8)pmu_cal_lane_error_status_check(0U, slice);
+
+ if (rank != 0U || b != 0U)
+ continue;
+ } else if (swap_mode == 1U) {
+ u8 b = (u8)pmu_cal_lane_error_status_check(0U, slice);
+
+ if (rank == 0U || b != 0U)
+ continue;
+ } else if (flags == 0U) {
+ continue;
+ }
+
+ sub = (const u8 *)(PMU_DMEM_AC_TIMING_TBL + rank_offset +
+ slice * 1320U);
+ for (freq_pt = iter_lim; freq_pt > 0U; freq_pt--) {
+ s8 offset = (s8)sub[130] - 31;
+
+ pmu_window_margin_diff_calc(temp_buf, sub, temp_buf,
+ offset, 0);
+ sub += 132U;
+ }
+ }
+
+ for (i = 0; i < 64; i++) {
+ u8 low = temp_buf[i * 2];
+ u8 high = temp_buf[i * 2 + 1];
+
+ if (high >= low)
+ total_width += (high - low + 1U);
+ }
+
+ return (u16)total_width;
+}
+
+/**
+ * pmu_window_margin_diff_calc() - Intersect two 64-entry window margin arrays with offset/delta
+ * @src0: First 128-byte (64 x [low, high]) window margin array
+ * @src1: Second 128-byte (64 x [low, high]) window margin array
+ * @out_dest: Output 128-byte window margin array (may alias @src0 or @src1)
+ * @offset: Signed index shift applied to @src1
+ * @delta: Signed bound shift subtracted from @src1 bounds
+ *
+ * Return: Always 0.
+ */
+static noinline u32 pmu_window_margin_diff_calc(const u8 *src0, const u8 *src1,
+ u8 *out_dest, s32 offset, s32 delta)
+{
+ u8 temp_buf[128];
+ u8 *dest;
+ s32 i;
+
+ dest = (out_dest == src0 || out_dest == src1) ? temp_buf : out_dest;
+
+ for (i = 0; i < 64; i++) {
+ s32 j = offset + i;
+ u8 *d = dest + i * 2;
+ s32 h;
+ u8 max_high;
+ s32 l;
+ u8 min_low;
+
+ if ((u32)j > 63U || src0[i * 2 + 1] == 0U ||
+ src1[j * 2 + 1] == 0U) {
+ *(u16 *)d = 0x00ffU;
+ continue;
+ }
+
+ h = (s32)src1[j * 2 + 1] - delta;
+ if (h > 127)
+ h = 127;
+ max_high = (src0[i * 2 + 1] < (u8)h) ?
+ src0[i * 2 + 1] : (u8)h;
+
+ l = (s32)src1[j * 2] - delta;
+ if (l < 0)
+ l = 0;
+ min_low = (src0[i * 2] > (u8)l) ? src0[i * 2] : (u8)l;
+
+ if (min_low < max_high) {
+ d[0] = min_low;
+ d[1] = max_high;
+ } else {
+ *(u16 *)d = 0x00ffU;
+ }
+ }
+
+ if (dest != out_dest)
+ memcpy(out_dest, temp_buf, 128U);
+
+ return 0U;
+}
+
+/**
+ * min_u() - Return the minimum of two unsigned 32-bit integers
+ * @a: First unsigned value
+ * @b: Second unsigned value
+ *
+ * Return: Smaller of @a and @b.
+ */
+static __always_inline u32 min_u(u32 a, u32 b)
+{
+ return (a < b) ? a : b;
+}
+
+/**
+ * pmu_eye_sample_metric_eval() - Evaluate 2D eye diagram sample quality score
+ * @x: Candidate X delay coordinate (0..63)
+ * @y: Candidate Y Vref coordinate (0..127)
+ * @eye_scan_lut: 512-byte transformed scan boundary table
+ * @samples: 256-byte eye sample margin table
+ *
+ * Return: Packed 32-bit quality score ((score << 8) | margin_sum), or 0 if invalid.
+ */
+static noinline u32 pmu_eye_sample_metric_eval(u8 x, u8 y, const u8 *eye_scan_lut,
+ const u8 *samples)
+{
+ const u8 *pair = samples + ((u32)x << 1);
+ u8 low = pair[0];
+ u8 high = pair[1];
+ u32 coord;
+ u8 min_err = 0xffU;
+ u32 min_val = 0xffffffffUL;
+ const s16 *p;
+ u32 i;
+ u32 dist;
+ u16 packed_coord;
+ u16 w32;
+ u32 x1;
+ u32 x2;
+ u32 y1;
+ u32 score;
+ u32 min_error_val;
+
+ if (low >= high || low > y || high <= y)
+ return 0U;
+
+ coord = ((u32)y << 8) | x;
+ p = (const s16 *)eye_scan_lut;
+
+ for (i = 0U; i < 64U; i++) {
+ u32 k;
+
+ for (k = 0U; k < 4U; k += 2U) {
+ s16 v0 = p[k];
+ s16 v1 = p[k + 1U];
+
+ if (v1 >= v0) {
+ u8 out_byte;
+ u32 ret;
+
+ ret = pmu_cal_eye_margin_quadratic_cost_calc(coord, i,
+ (u32)(s32)(s8)v0,
+ (u32)(s32)(s8)v1,
+ &out_byte);
+
+ min_val = min_u(min_val, ret);
+ min_err = (u8)min_u(min_err, out_byte);
+ }
+ }
+ p += 4;
+ }
+
+ dist = min_u((u32)(y - low), (u32)(high - y));
+ packed_coord = pmu_sdata()->num_ranks;
+ w32 = pmu_sdata()->fw_version_id;
+
+ x1 = (u32)(x + 1U);
+ x2 = (u32)(64U - x);
+ y1 = (u32)(128U - y);
+
+ min_val = min_u(min_val, x1 * x1 * packed_coord);
+ min_val = min_u(min_val, x2 * x2 * packed_coord);
+ min_val = min_u(min_val, y1 * y1 * w32);
+
+ if ((*(__volatile__ const u8 *)PMU_DMEM_SIGNED_STEP & 1U) == 0U) {
+ u32 y2 = (u32)(y + 1U);
+
+ min_val = min_u(min_val, y2 * y2 * w32);
+ }
+
+ score = min_u(min_val, 0xffffffUL);
+ min_error_val = min_u(min_u(min_err, x), 63U - x);
+
+ return (score << 8) | ((dist + min_error_val) & 0xffU);
+}
+
+/**
+ * pmu_eye_boundary_sample_search() - Search optimal X sample at fixed Y Vref slice
+ * @samples: 256-byte eye sample buffer (updated at [130..131] on success)
+ * @target_y: Target Y Vref coordinate
+ *
+ * Return: 0 on completion, or non-zero error code from center evaluation.
+ */
+static u32 pmu_eye_boundary_sample_search(u8 *samples, u32 target_y)
+{
+ u8 margins[6];
+ u8 mode;
+ u8 scan_lut[512];
+ u32 ret = (u32)pmu_sample_margin_center_eval(samples, margins, &mode);
+ u32 best_score = 0U;
+ u32 best_sample;
+ const u8 *p = samples;
+ u32 sample_x;
+ s32 step;
+
+ if (ret != 0U)
+ return ret;
+
+ pmu_eye_sample_table_transform(scan_lut, samples);
+ best_sample = margins[0];
+ best_score = pmu_eye_sample_metric_eval(margins[0], (u8)target_y,
+ scan_lut, samples);
+
+ for (sample_x = 0U; sample_x <= 0x3fU; sample_x += 8U, p += 16U) {
+ if (p[0] < target_y && target_y < p[1]) {
+ u32 score;
+
+ score = pmu_eye_sample_metric_eval((u8)sample_x,
+ (u8)target_y,
+ scan_lut,
+ samples);
+
+ if (score > best_score) {
+ best_score = score;
+ best_sample = sample_x;
+ }
+ }
+ }
+
+ for (step = 4; step >= 2; step >>= 1) {
+ s32 d;
+
+ for (d = step; ; d = -step) {
+ s32 cand = (s32)best_sample + d;
+
+ if ((cand & ~0x3f) == 0) {
+ u32 score;
+
+ score = pmu_eye_sample_metric_eval((u8)cand,
+ (u8)target_y,
+ scan_lut,
+ samples);
+
+ if (score > best_score) {
+ best_score = score;
+ best_sample = (u32)cand;
+ }
+ }
+ if (d < 0)
+ break;
+ }
+ }
+
+ if (best_score != 0U) {
+ samples[131] = (u8)target_y;
+ samples[130] = (u8)best_sample;
+ }
+
+ return 0U;
+}
+
+/**
+ * pmu_cal_margin_sample_pair_extract() - Optimize 2D (delay, Vref) eye center coordinates
+ * @out_pair: Output 2-byte array receiving [0] optimal X and [1] optimal Y
+ * @margins: Initial 2-byte seed coordinates [0] X and [1] Y
+ * @samples: 256-byte eye sample table
+ * @scan_lut: 512-byte transformed scan boundary table
+ *
+ * Return: Best 2D eye quality score achieved.
+ */
+static u32 pmu_cal_margin_sample_pair_extract(u8 *out_pair, const u8 *margins,
+ const u8 *samples, const u8 *scan_lut)
+{
+ u16 pmu_flags = *(__volatile__ const u16 *)(PMU_DMEM_BASE +
+ PMU_DMEM_DRAM_CFG_FLAGS);
+ u32 max_iters = (pmu_flags >> 1) & 0x1eU;
+ u32 early_exit = (max_iters == 2U) ? 1U : 0U;
+ u32 m[2];
+ u32 cand[2];
+ u32 best_score;
+ u32 step_reset;
+ u32 step = 4U;
+ u32 countdown;
+ u32 iter;
+ u32 axis;
+
+ if (max_iters <= 2U)
+ max_iters = 16U;
+
+ out_pair[0] = 0x20U;
+ out_pair[1] = 0x40U;
+
+ m[0] = margins[0];
+ m[1] = margins[1];
+ cand[0] = m[0];
+ cand[1] = m[1];
+ best_score = pmu_eye_sample_metric_eval((u8)m[0], (u8)m[1],
+ scan_lut, samples);
+
+ if (best_score < 2U)
+ return best_score;
+
+ step_reset = max_iters >> 2;
+ countdown = step_reset;
+
+ for (iter = 0U; iter < max_iters; iter++) {
+ u32 baseline_score = best_score;
+
+ for (axis = 0U; axis < 2U; axis++) {
+ u32 other = m[1U - axis];
+ u32 other_eq = (other == cand[1U - axis]) ? 1U : 0U;
+ u32 mask = (axis != 0U) ? 0x7fU : 0x3fU;
+ u32 sub = (m[axis] - step) & mask;
+ u32 add = (m[axis] + step) & mask;
+ u32 s0 = (axis != 0U) ? other : sub;
+ u32 s1 = (axis != 0U) ? sub : other;
+ u32 a0 = (axis != 0U) ? other : add;
+ u32 a1 = (axis != 0U) ? add : other;
+ u32 s_sub = (sub == cand[axis] && other_eq != 0U) ? 0U :
+ pmu_eye_sample_metric_eval((u8)s0, (u8)s1,
+ scan_lut,
+ samples);
+ u32 s_add = (add == cand[axis] && other_eq != 0U) ? 0U :
+ pmu_eye_sample_metric_eval((u8)a0, (u8)a1,
+ scan_lut,
+ samples);
+ u32 best_dir = (s_add > s_sub) ? add : sub;
+ u32 best_dir_score = (s_add > s_sub) ? s_add : s_sub;
+
+ if (best_dir_score >= best_score) {
+ cand[axis] = m[axis];
+ m[axis] = best_dir;
+ best_score = best_dir_score;
+ }
+ }
+
+ if (best_score != baseline_score)
+ countdown = step_reset;
+ else if (countdown != 0U)
+ countdown--;
+
+ if ((u8)countdown == 0U ||
+ (m[0] == out_pair[0] && m[1] == out_pair[1])) {
+ step >>= 1;
+ if (step != 0U) {
+ countdown = step_reset;
+ } else {
+ step = 1U;
+ if (early_exit != 0U)
+ return best_score;
+ }
+ }
+
+ out_pair[1] = (u8)m[1];
+ out_pair[0] = (u8)m[0];
+ }
+
+ return best_score;
+}
+
+/**
+ * call_multi_rank_step() - Execute multi-rank timing step for the given rank and channel
+ * @ch_mask: Active channel bitmask
+ * @channel: Target channel index (0 or 1)
+ * @rank: Target rank index (0 or 1)
+ */
+static __always_inline void call_multi_rank_step(u32 ch_mask, u32 channel,
+ u32 rank)
+{
+ u32 off = (PMU_DMEM_BASE | PMU_DMEM_CAL_STRUCT_BASE) +
+ (u32)(u8)rank * (2U * PMU_SLICE_DESC_SIZE) +
+ channel * PMU_SLICE_DESC_SIZE;
+
+ pmu_cal_multi_rank_timing_step(ch_mask, off, 0U,
+ 0xfffbfbffUL, 0xffffbfffUL, 0U);
+}
+
+/**
+ * pmu_cal_timing_margin_envelope_scan() - Scan DBYTE timing margin envelope across slices
+ * @channel: Target memory channel index (0 or 1)
+ * @rank: Target rank index (0 or 1)
+ * @buf: Output halfword buffer receiving calibrated slice delays
+ */
+void pmu_cal_timing_margin_envelope_scan(u32 channel, u32 rank, void *buf)
+{
+ u32 csr_offset = dmem_read32(PMU_DMEM_ACTIVE_CH_OFFSET);
+ u32 vref_off = csr_offset << 1;
+ u32 iter_count = 0U;
+ u32 struct_base = PMU_DMEM_BASE | PMU_DMEM_CAL_STRUCT_BASE;
+ u32 ch_bit_mask = 0x15U << channel;
+ u32 ch_lane_mask = 5U << channel;
+ u32 hist_w = 0xffffffffUL;
+ u8 *history = (u8 *)&hist_w;
+ u32 commit_flag;
+ u32 start_slice;
+ u32 end_slice;
+ u32 slice;
+ u16 *out_buf;
+ u32 target_reg;
+
+ *(__volatile__ u16 *)PHY_REG_AC_BCAST_GATE = 0U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_VREF = 0U;
+
+ *(__volatile__ u16 *)(PHY_REG_VREF_TRIM_CFG | vref_off) = 1U;
+ *(__volatile__ u16 *)(PHY_REG_VREF_CTRL | vref_off) = 1U;
+
+ pmu_phy_pll_cfg_quad_write(0U);
+ pmu_cal_deskew_quad_tap_set(0xffU, 0U, 0U, 0U, 0U);
+
+ *(__volatile__ u16 *)PHY_REG_SEC_CAL_CTRL = 3U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_UPPER_SAMPLE_B = 0U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_SAMPLE_RDY = 1U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY2 = 0U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY3 = 0U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY0 = 0x30U;
+ *(__volatile__ u16 *)PHY_REG_BIST_CTRL_MODE = 1U;
+ *(__volatile__ u16 *)PHY_REG_AC_BCAST_PAT_DATA = 511U;
+ *(__volatile__ u16 *)PHY_REG_BIST_PAT_CTRL0 = 0U;
+ *(__volatile__ u16 *)PHY_REG_BIST_PAT_CTRL1 = 0U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY1 = 386U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_UPPER_COARSE = 2048U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_UPPER_BCAST = 0U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_LANE = 1U;
+
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_RX_EN =
+ (dmem_read8(PMU_DMEM_CBT_LANE_LIM) == 0U) ? 520U : 512U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_CBT = 2U;
+
+ pmu_cal_dbyte_gate_mask_clear();
+ pmu_phy_dbyte_cal_trig_pulse(0x0fU, 0x0fU);
+ pmu_phy_soft_reset_pulse();
+
+ pmu_cal_eye_table_save((const u8 *)(uintptr_t)struct_base, 0x0aU);
+ pmu_cal_lane_mask_and_all(struct_base, 0x0aU, 0x00U);
+ pmu_cal_lane_mask_or_all(struct_base, 0x0aU, 0x59U);
+ pmu_cal_eye_table_save((const u8 *)(uintptr_t)struct_base, 0x2eU);
+ pmu_cal_lane_mask_and_all(struct_base, 0x2eU, 0x00U);
+ pmu_cal_eye_table_save((const u8 *)(uintptr_t)struct_base, 0x12U);
+ pmu_cal_lane_mask_or_all(struct_base, 0x12U, 0x10U);
+
+ call_multi_rank_step(ch_bit_mask, channel, rank);
+
+ pmu_cal_eye_table_restore((u8 *)(uintptr_t)struct_base, 0x0aU);
+ pmu_cal_eye_table_restore((u8 *)(uintptr_t)struct_base, 0x12U);
+ pmu_cal_eye_table_restore((u8 *)(uintptr_t)struct_base, 0x2eU);
+
+ pmu_channel_timing_deskew_reset(0U, 0U, 0U, 0U, ch_lane_mask);
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQS_DLY0 = 1U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQS_DLY1 = 1U;
+ pmu_cbt_deskew_pulse_train(ch_lane_mask, 0U);
+
+ pmu_cal_slice_step_diff_latch(0x60U, 1U);
+ pmu_cal_dbyte_delay_coarse_step();
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+
+ pmu_phy_dbyte_cal_trig_pulse(0x0fU, 0x0fU);
+ pmu_cbt_deskew_pulse_train(ch_lane_mask, 1U);
+
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY2 = 511U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY3 = 511U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY1 = 0x81U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY0 = 0x30U;
+
+ start_slice = dmem_read8(PMU_DMEM_SLICE_START);
+ end_slice = dmem_read8(PMU_DMEM_SLICE_END);
+
+ do {
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+
+ commit_flag = 0U;
+ slice = start_slice;
+ do {
+ u32 slice_off = slice << 13;
+ u16 stat = *(__volatile__ const u16 *)
+ (PHY_REG_DBYTE_LCDL_STAT + slice_off);
+ u8 hist = (u8)((history[slice] << 1) |
+ ((stat != 0U) ? 1U : 0U));
+
+ history[slice] = hist;
+ if ((hist & 7U) != 0U) {
+ __volatile__ u16 *dly_reg = (__volatile__ u16 *)
+ (PHY_REG_DBYTE_COARSE + slice_off);
+
+ *dly_reg = (u16)(*dly_reg + 1024U);
+ commit_flag = 1U;
+ }
+ } while (++slice <= end_slice);
+
+ pmu_phy_dbyte_cal_trig_pulse(0x0fU, 0x0fU);
+
+ if (++iter_count >= 40U)
+ break;
+ } while (commit_flag != 0U);
+
+ pmu_assert_or_halt(commit_flag == 0U, PMU_ERR_EYE_SWEEP_CONVERGENCE);
+
+ slice = start_slice;
+ do {
+ __volatile__ u16 *dly_reg = (__volatile__ u16 *)
+ (PHY_REG_DBYTE_COARSE + (slice << 13));
+
+ *dly_reg = (u16)(*dly_reg + 2048U);
+ } while (++slice <= end_slice);
+
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY2 = 0U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY3 = 0U;
+ pmu_cal_dbyte_delay_coarse_step();
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY0 = 0U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_SAMPLE_RDY = 0U;
+
+ slice = start_slice;
+ do {
+ *(__volatile__ u16 *)(PHY_REG_DBYTE_SAMPLE_RDY + (slice << 13)) = 1U;
+ } while (++slice <= end_slice);
+
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY1 = 0x81U;
+ pmu_cal_slice_step_diff_latch(0x40U, 1U);
+
+ pmu_phy_lcdl_delay_read((u16 *)buf, 1U, 2U);
+
+ out_buf = (u16 *)buf;
+ slice = start_slice;
+ do {
+ out_buf[slice] -= 128U;
+ } while (++slice <= end_slice);
+
+ target_reg = csr_offset | (pmu_bist_pattern_bit_get(channel, channel) + 0x20U);
+ pmu_slice_delay_step_program(target_reg, (const s16 *)buf,
+ 1U, 0U, 0U, 1U);
+
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_RX_EN = 0U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_CBT = 0x20U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_CBT = 0U;
+ *(__volatile__ u16 *)PHY_REG_SEC_CAL_CTRL = 0U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY1 = 0U;
+ *(__volatile__ u16 *)PHY_REG_AC_BCAST_GATE = 1U;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_VREF = 1U;
+
+ pmu_phy_sec_cal_strobe_pulse();
+ *(__volatile__ u16 *)(PHY_REG_VREF_CTRL | vref_off) = 0U;
+
+ call_multi_rank_step(ch_bit_mask, channel, rank);
+}
+
+/**
+ * pmu_eye_margin_step_balancer() - Balance 2D eye margin steps across ranks/points
+ * @buf0: Rank 0 / primary eye sample buffer
+ * @buf1: Rank 1 / secondary eye sample buffer
+ * @buf2: Alt point Rank 0 eye sample buffer
+ * @buf3: Alt point Rank 1 eye sample buffer
+ *
+ * Return: 0 on success, or non-zero calibration error code.
+ */
+static noinline u32 pmu_eye_margin_step_balancer(u8 *buf0, u8 *buf1, u8 *buf2,
+ u8 *buf3)
+{
+ u8 *bufs[4] = { buf0, buf1, buf2, buf3 };
+ u32 count1 = (buf1 != buf0) ? 2U : 1U;
+ u32 count2 = (buf2 != buf0) ? 2U : 1U;
+ u8 *primary_buf;
+ u8 *sec_buf;
+ u8 temp_buf[132];
+ u32 rank_idx;
+ u32 pair_idx;
+
+ for (rank_idx = 0U; rank_idx < count1; rank_idx++) {
+ u32 i;
+
+ pmu_sdata()->stage_rank_idx = (u8)rank_idx;
+ for (i = 0U; i < count2; i++) {
+ u8 *b = bufs[i * 2U + rank_idx];
+ u32 ret = pmu_window_centroid_calc(b);
+
+ if (ret != 0U)
+ return ret;
+ }
+ }
+
+ primary_buf = buf0;
+ sec_buf = buf1;
+
+ for (pair_idx = 1U; pair_idx <= count2; pair_idx++) {
+ s8 off = (s8)sec_buf[130] - (s8)primary_buf[130];
+ u8 d0b;
+ u32 ret;
+
+ pmu_sdata()->stage_rank_idx = 3U;
+ pmu_window_margin_diff_calc(primary_buf, sec_buf, temp_buf,
+ (s32)off, 0);
+
+ d0b = dmem_read8(PMU_DMEM_CAL_SELECT_FLAGS);
+ if ((d0b & 4U) != 0U)
+ ret = pmu_window_centroid_calc(temp_buf);
+ else
+ ret = pmu_eye_margin_window_search(temp_buf, 0U);
+ if (ret != 0U)
+ return ret;
+
+ primary_buf[131] = temp_buf[131];
+ sec_buf[131] = temp_buf[131];
+
+ if (pair_idx < count2) {
+ primary_buf = buf2;
+ sec_buf = buf3;
+ }
+ }
+
+ if (buf2 != buf0) {
+ u32 buf_idx;
+
+ for (buf_idx = 0U; buf_idx < count1; buf_idx++) {
+ u8 *rx = bufs[buf_idx];
+ u8 *ry = bufs[buf_idx + 2U];
+ s32 delta = (s32)ry[131] - (s32)rx[131];
+ s32 diff = (s32)(s16)*(const __volatile__ u16 *)(rx + 128) -
+ (s32)(s16)*(const __volatile__ u16 *)(ry + 128);
+ u32 ret;
+
+ pmu_window_margin_diff_calc(rx, ry, temp_buf, diff,
+ delta);
+ ret = pmu_eye_boundary_sample_search(temp_buf,
+ (u32)rx[131]);
+ if (ret != 0U)
+ return ret;
+ rx[130] = temp_buf[130];
+ ry[130] = (u8)((s32)temp_buf[130] + diff);
+ }
+ } else {
+ u32 ret0 = pmu_eye_boundary_sample_search(buf2,
+ (u32)buf2[131]);
+ u32 ret1;
+
+ if (ret0 != 0U)
+ return ret0;
+ ret1 = pmu_eye_boundary_sample_search(buf1, (u32)buf1[131]);
+ if (ret1 != 0U)
+ return ret1;
+ }
+
+ return 0U;
+}
+
+static __always_inline void
+pmu_cal_margin_matrix_prog_step(u32 cur_dly, u8 cfg_bit13,
+ u32 target_timing_reg, u32 source_timing_reg,
+ u8 accumulated_bias, u8 baseline_buf[4][10])
+{
+ u8 s_start = dmem_read8(PMU_DMEM_SLICE_START);
+ u8 s_end = dmem_read8(PMU_DMEM_SLICE_END);
+ u8 a_mask = dmem_read8(PMU_DMEM_ACTIVE_SLICE_MASK_CAL);
+ u32 csr_off = dmem_read32(PMU_DMEM_ACTIVE_CSR_OFFSET);
+ u32 s_idx;
+
+ for (s_idx = s_start; s_idx <= s_end; s_idx++) {
+ if (!(a_mask & (1U << s_idx)))
+ continue;
+
+ if (cfg_bit13 != 0U) {
+ u32 lane;
+
+ for (lane = 0U; lane < 10U; lane++) {
+ u16 dly_val = (u16)(cur_dly + baseline_buf[s_idx][lane]);
+ u32 lane_csr = (s_idx << 12) | csr_off | (lane << 8);
+ u32 pll_stat_val = PHY_REG_DBYTE_BASE |
+ ((lane_csr | target_timing_reg) << 1);
+
+ phy_write16(pll_stat_val, dly_val);
+ if (accumulated_bias != 0U) {
+ u32 clk_gate_val = PHY_REG_DBYTE_BASE |
+ ((lane_csr | source_timing_reg) << 1);
+
+ phy_write16(clk_gate_val, dly_val);
+ }
+ }
+ } else {
+ u32 pll_stat_val = (PHY_REG_DBYTE_BASE | 0x1e00UL) |
+ (((s_idx << 12) | csr_off | target_timing_reg) << 1);
+
+ phy_write16(pll_stat_val, (u16)cur_dly);
+ if (accumulated_bias != 0U) {
+ u32 clk_gate_val = (PHY_REG_DBYTE_BASE | 0x1e00UL) |
+ (((s_idx << 12) | csr_off | source_timing_reg) << 1);
+
+ phy_write16(clk_gate_val, (u16)cur_dly);
+ }
+ }
+ }
+ pmu_delay_us(300000U, 0U);
+}
+
+static __always_inline void
+pmu_cal_margin_matrix_eval_lane(u32 k, u32 ch, u32 s_idx, u32 lane,
+ u16 prim_reg_val, u16 sec_reg_val, u32 lane_div,
+ u8 scan_multiplier, u32 flag, u32 last_cur_dly,
+ u8 step_size, u8 cfg_bit13, u8 base_adj)
+{
+ u32 entry_idx = (s_idx * 10U) + lane;
+ __volatile__ u8 *state_bytes =
+ (__volatile__ u8 *)((PMU_DMEM_BASE | PMU_DMEM_SWEEP_SAMPLE_BUF) +
+ 0x27cUL + (entry_idx * 848U));
+ s32 win_start = 0;
+ s32 in_pass = 0;
+ s32 best_start = 0;
+ s32 best_end = 0;
+ s32 best_width = 0;
+ s32 last_step = -1;
+ s32 st_step;
+ uintptr_t entry_base;
+ __volatile__ u8 *rec_base;
+ u32 shift_val;
+ u32 idx;
+
+ for (st_step = 0; st_step < (s32)prim_reg_val; st_step++) {
+ u8 st = state_bytes[st_step];
+
+ if (in_pass != 0) {
+ if (st == 0U) {
+ s32 cand_width = last_step - win_start;
+
+ if (cand_width > (best_width << 1)) {
+ best_start = win_start;
+ best_end = last_step;
+ best_width = cand_width;
+ }
+ in_pass = 0;
+ }
+ } else if (st != 0U) {
+ in_pass = 1;
+ win_start = st_step;
+ }
+ last_step = st_step;
+ }
+
+ if (in_pass != 0) {
+ s32 cand_width = (s32)sec_reg_val - win_start;
+
+ if (cand_width > (best_width << 1)) {
+ best_start = win_start;
+ best_end = (s32)sec_reg_val;
+ best_width = cand_width;
+ }
+ }
+
+ if (best_width > (s32)lane_div)
+ best_start = best_end - (s32)lane_div;
+
+ entry_base = (PMU_DMEM_BASE | PMU_DMEM_EYE_SAMPLE_BUF) +
+ (k * 10560U) + (ch * 5280U) + (s_idx * 1320U) + (lane * 132U);
+ *(__volatile__ u16 *)(entry_base + 128U) = (u16)best_start;
+
+ rec_base = (__volatile__ u8 *)((PMU_DMEM_BASE | PMU_DMEM_SWEEP_SAMPLE_BUF) +
+ (entry_idx * 848U));
+ shift_val = scan_multiplier - 1U;
+
+ for (idx = 0U; idx < 64U; idx++) {
+ u32 step_sampled = (u8)(best_start + (idx >> shift_val));
+ u8 st = state_bytes[step_sampled];
+ u8 left_val;
+ u8 right_val;
+
+ if (flag != 0U) {
+ if (st == 0U) {
+ left_val = 127U;
+ right_val = 0U;
+ } else if (st == 1U) {
+ left_val = rec_base[step_sampled * 3U + 0U];
+ right_val = (u8)last_cur_dly;
+ } else if (st == 2U) {
+ left_val = rec_base[step_sampled * 3U + 0U];
+ right_val = rec_base[step_sampled * 3U + 1U] - step_size;
+ } else {
+ u8 cand_left = rec_base[step_sampled * 3U + 0U];
+ u8 cand_right = rec_base[step_sampled * 3U + 1U];
+ u8 cand_start = rec_base[step_sampled * 3U + 2U];
+ u32 cand_w = (u32)(cand_right - (cand_left + step_size));
+ u32 cur_w = (u32)((u8)last_cur_dly - cand_start);
+ bool pick_cur = cur_w > cand_w;
+
+ left_val = pick_cur ? cand_start : cand_left;
+ right_val = pick_cur ? (u8)last_cur_dly :
+ (u8)(cand_right - step_size);
+ }
+ } else {
+ left_val = ((st | 2U) == 3U) ? 0U : 127U;
+ right_val = ((st | 2U) == 3U) ? 127U : 0U;
+ }
+
+ if ((s8)right_val - (s8)left_val < 5) {
+ left_val = 127U;
+ right_val = 0U;
+ }
+
+ if (cfg_bit13 != 0U && (u8)right_val >= left_val) {
+ left_val += base_adj;
+ right_val += base_adj;
+ }
+
+ *(__volatile__ u8 *)(entry_base + (idx * 2U) + 0U) = left_val;
+ *(__volatile__ u8 *)(entry_base + (idx * 2U) + 1U) = right_val;
+ }
+}
+
+static __always_inline void
+pmu_cal_margin_matrix_center_lane(u32 k, u32 ch, u32 s_idx, u32 lane,
+ u16 metric_max, u8 scan_multiplier)
+{
+ uintptr_t entry_base = (PMU_DMEM_BASE | PMU_DMEM_EYE_SAMPLE_BUF) +
+ (k * 10560U) + (ch * 5280U) + (s_idx * 1320U) + (lane * 132U);
+ s32 win_start = 0;
+ s32 win_end = 0;
+ s32 best_w_start = 0;
+ s32 best_w_end = 0;
+ s32 best_w = -1;
+ s32 in_win = 0;
+ u32 st_step;
+ s32 sum_steps;
+ s32 center;
+ u16 best_start;
+ u16 new_baseline;
+
+ for (st_step = 0U; st_step < 64U; st_step++) {
+ u8 left = *(__volatile__ u8 *)(entry_base + (st_step * 2U) + 0U);
+ u8 right = *(__volatile__ u8 *)(entry_base + (st_step * 2U) + 1U);
+
+ if (right >= left) {
+ if (in_win == 0) {
+ in_win = 1;
+ win_start = (s32)st_step;
+ }
+ win_end = (s32)st_step;
+ } else if (in_win != 0) {
+ s32 cur_w = win_end - win_start;
+
+ in_win = 0;
+ if (cur_w > best_w) {
+ best_w = cur_w;
+ best_w_start = win_start;
+ best_w_end = win_end;
+ }
+ }
+ }
+
+ if (in_win != 0) {
+ s32 cur_w = win_end - win_start;
+
+ if (cur_w > best_w) {
+ best_w = cur_w;
+ best_w_start = win_start;
+ best_w_end = win_end;
+ }
+ }
+
+ sum_steps = best_w_start + best_w_end;
+ if (dmem_read8(PMU_DMEM_TRAIN_MODE_HI) & 4U) {
+ center = sum_steps;
+ } else {
+ center = sum_steps / 2;
+ if (dmem_read8(PMU_DMEM_CAL_OVERRIDE_FLAG) != 0U) {
+ u32 width = (u32)(best_w_end - best_w_start);
+
+ pmu_assert_or_halt((width > 5U) ? 1U : 0U,
+ PMU_ERR_EYE_WINDOW_TOO_NARROW);
+ }
+ }
+
+ best_start = *(__volatile__ u16 *)(entry_base + 128U);
+ new_baseline = metric_max + (best_start * scan_multiplier);
+ *(__volatile__ u16 *)(entry_base + 128U) = new_baseline;
+ *(__volatile__ s8 *)(entry_base + 130U) = (s8)center;
+}
+
+/**
+ * pmu_margin_matrix_capture_baseline() - Capture per-slice/lane baseline delay registers
+ * @baseline_buf: Output 4x10 byte baseline array
+ * @cfg_bit13: Configuration flag 13 value
+ * @target_timing_reg: Target DBYTE timing register offset
+ * @scan_direction: Initial scan direction / start tap
+ * @vref_boundary_clamp: Maximum Vref/delay boundary clamp
+ */
+static void pmu_margin_matrix_capture_baseline(u8 baseline_buf[4][10],
+ u32 cfg_bit13,
+ u32 target_timing_reg,
+ u8 scan_direction,
+ u8 vref_boundary_clamp)
+{
+ u8 s_start = dmem_read8(PMU_DMEM_SLICE_START);
+ u8 s_end = dmem_read8(PMU_DMEM_SLICE_END);
+ u8 a_mask = dmem_read8(PMU_DMEM_ACTIVE_SLICE_MASK_CAL);
+ u32 active_csr_offset = dmem_read32(PMU_DMEM_ACTIVE_CSR_OFFSET);
+ u8 limit_val = dmem_read8((cfg_bit13 <= 2U) ? 0x11bU : 0x11aU);
+ u8 blink_val = (u8)(scan_direction + (127U - vref_boundary_clamp));
+ u32 s_idx;
+
+ for (s_idx = s_start; s_idx <= s_end; s_idx++) {
+ u32 lane;
+
+ if (!(a_mask & (1U << s_idx)))
+ continue;
+ for (lane = 0U; lane < 10U; lane++) {
+ u32 lane_csr = (s_idx << 12) | active_csr_offset |
+ target_timing_reg | (lane << 8);
+ u8 val_b = (u8)phy_read16(PHY_REG_DBYTE_BASE |
+ (lane_csr << 1));
+ bool clamp_hi = (u32)vref_boundary_clamp + val_b >= 128U;
+ u8 adj = ((s8)val_b < (s8)limit_val) ?
+ 0U : (u8)(val_b - limit_val);
+
+ baseline_buf[s_idx][lane] = clamp_hi ? blink_val : adj;
+ }
+ }
+}
+
+/**
+ * pmu_margin_matrix_select_pass_regs() - Resolve source/target timing registers for a scan pass
+ * @k: Pass index (0 or 1)
+ * @mode: Calibration sweep mode (0..3)
+ * @profile_mode: PHY operating profile mode (0..3)
+ * @keep_pass_regs: Non-zero to retain precomputed primary/secondary registers
+ * @timing_reg_pri: Precomputed primary timing register
+ * @timing_reg_sec: Precomputed secondary timing register
+ * @slice_reg_stride: Precomputed slice register stride
+ * @target_out: Output pointer for target timing register
+ * @source_out: Output pointer for source timing register
+ * @stride_out: Output pointer for active slice stride
+ */
+static void pmu_margin_matrix_select_pass_regs(u32 k, u32 mode, u32 profile_mode,
+ u32 keep_pass_regs,
+ u32 timing_reg_pri,
+ u32 timing_reg_sec,
+ u32 slice_reg_stride,
+ u32 *target_out,
+ u32 *source_out,
+ u32 *stride_out)
+{
+ u32 target = timing_reg_sec;
+ u32 source = timing_reg_pri;
+ u32 stride = slice_reg_stride;
+
+ if (keep_pass_regs & 1U) {
+ /* Retain precomputed target, source, and stride */
+ } else if (mode == 0U && profile_mode == 3U) {
+ target = (k == 0U) ? 0x4eU : 0x50U;
+ source = (k == 0U) ? 0x4fU : 0x51U;
+ stride = slice_reg_stride << (k != 0U);
+ } else if (mode == 1U && profile_mode == 3U) {
+ target = (k == 0U) ? 0x4fU : 0x51U;
+ source = (k == 0U) ? 0x4eU : 0x50U;
+ stride = ((k != 0U) ? 2U : 0U) + 1U;
+ } else if (mode == 2U) {
+ target = 0x4eU;
+ source = 0x4fU;
+ stride = 4U;
+ } else {
+ pmu_assert_or_halt(0U, PMU_ERR_EYE_SLICE_REG_SOURCE);
+ source = 0x4fU;
+ target = 0x4eU;
+ }
+
+ *target_out = target;
+ *source_out = source;
+ *stride_out = stride;
+}
+
+/**
+ * pmu_cal_margin_matrix_scan_eval() - Execute 2D Vref/delay margin matrix scan
+ * @channel: Target channel index (0 or 1)
+ * @rank: Target rank index
+ * @slice: Slice/boundary mode flags
+ * @mode: Calibration sweep mode (0..3)
+ * @flag: Vref/delay step flag
+ * @step: Sweep pass index
+ */
+void pmu_cal_margin_matrix_scan_eval(u32 channel, u32 rank, u32 slice,
+ u32 mode, u32 flag, u32 step)
+{
+ u16 scan_metric_min[4] = { 0 };
+ u16 scan_metric_max[4] = { 0 };
+ u8 scan_profile_table[48] = { 0 };
+ u8 baseline_buf[4][10] = { { 0 } };
+ u32 bound_flag = (slice >> 8) & 1U;
+ u32 status_query = pmu_cal_pad_voltage_dac_set(bound_flag, mode);
+ u32 profile_mode = pmu_phy_query_operating_mode();
+ u8 misc_flags = dmem_read8(PMU_DMEM_CAL_MISC_FLAGS);
+ u32 channel_scan_mask = 1U;
+ u32 active_scan_flag = (misc_flags & 0x10U) ? 0U : 1U;
+ u32 is_extended_sweep = 0U;
+ u8 ch1_ranks, ch0_ranks;
+ u32 scan_multiplier, wide_win_flag, margin_lim_flag = 0U;
+ u32 is_mode_1, is_mode_0, is_pm_0, is_pm_1, is_pm_2;
+ u32 mode_zero_cond, channel_stride_adj, lo_margin_val, is_pm0_mode1;
+ u32 is_cond_or_m0, is_mode0_pm1, step_val, is_eval_active;
+ u32 timing_reg_alt_b, alt_reg_sel, hi_margin_val, pass_reg_a, keep_pass_regs;
+ u32 lane_step_val, timing_reg_alt_a, dmem_mask, ch;
+
+ if (rank != 0U) {
+ is_extended_sweep = (misc_flags >> 3) & 1U;
+ active_scan_flag = 0U;
+ }
+
+ if (profile_mode == 3U)
+ channel_scan_mask = (mode != 2U) ? 2U : 1U;
+
+ ch1_ranks = dmem_read8(PMU_DMEM_CH1_RANK_EN);
+ ch0_ranks = dmem_read8(PMU_DMEM_CH0_RANK_EN);
+ scan_multiplier = (is_extended_sweep != 0U) ? 2U : 1U;
+
+ wide_win_flag = (rank == 1U) || ((u32)(rank - 3U) < 5U);
+ if ((bound_flag != 0U && wide_win_flag) ||
+ (dmem_read8(PMU_DMEM_TRAIN_STATUS) & 1U)) {
+ pmu_phy_dbyte_cfg_unpack();
+ margin_lim_flag = 1U;
+ }
+
+ is_mode_1 = (mode == 1U);
+ is_mode_0 = (mode == 0U);
+ is_pm_0 = (profile_mode == 0U);
+ is_pm_1 = (profile_mode == 1U);
+ is_pm_2 = (profile_mode == 2U);
+
+ mode_zero_cond = (flag == 0U) || (is_pm_0 && is_mode_0);
+ channel_stride_adj = (flag == 0U) ? 4U : 0U;
+ lo_margin_val = mode_zero_cond ? 0x4eU : 0x4fU;
+ is_pm0_mode1 = (is_pm_0 && is_mode_1);
+ is_cond_or_m0 = is_pm0_mode1 || mode_zero_cond;
+ is_mode0_pm1 = (is_mode_0 && is_pm_1);
+ step_val = is_mode0_pm1 ? lo_margin_val : 0x50U;
+ is_eval_active = is_mode0_pm1 || is_cond_or_m0;
+ timing_reg_alt_b = is_eval_active ? step_val : 0x51U;
+
+ alt_reg_sel = (is_mode_1 && is_pm_1) || is_eval_active;
+ hi_margin_val = alt_reg_sel ? 0x4fU : 0x4eU;
+ pass_reg_a = alt_reg_sel || (is_mode_0 && is_pm_2);
+ keep_pass_regs = pass_reg_a || (is_pm_2 && is_mode_1);
+
+ lane_step_val = is_cond_or_m0 ? hi_margin_val : 0x51U;
+ timing_reg_alt_a = is_eval_active ? lane_step_val : 0x50U;
+
+ if (!is_eval_active)
+ channel_stride_adj = 3U;
+ else if (!is_cond_or_m0)
+ channel_stride_adj = 2U;
+ else if (!mode_zero_cond)
+ channel_stride_adj = 1U;
+
+ dmem_mask = ch1_ranks | ch0_ranks;
+
+ for (ch = 0U; ch < 2U; ch++) {
+ u32 ch_mask = (1U << (ch + 2U)) | (1U << ch);
+ u8 vref_boundary_clamp = 127U;
+ u32 margin_diff_accum = 0U;
+ u32 prim_reg_val, window_eval_mode, sec_reg_val;
+ u32 timing_reg_pri, timing_reg_sec, slice_reg_stride, k;
+
+ if (ch != 0U && (ch_mask & dmem_mask) == 0U)
+ continue;
+
+ dmem_write8(PMU_DMEM_CAL_BYTE, (u8)ch);
+ pmu_phy_mode_cfg_dispatch(3U);
+ pmu_cal_dbyte_pattern_loop_exec(ch, (u16)mode, scan_metric_min,
+ scan_metric_max, step,
+ (u8)active_scan_flag,
+ is_extended_sweep);
+
+ if (mode < 2U) {
+ prim_reg_val = 128U;
+ window_eval_mode =
+ (dmem_read16(PMU_DMEM_DRAM_FREQ_OFF) < 3200U) ?
+ 1U : 0U;
+ } else if (mode == 2U) {
+ prim_reg_val = 212U;
+ window_eval_mode = 1U;
+ margin_diff_accum =
+ (dmem_read8(PMU_DMEM_TRAIN_PARAM_CTRL) >> 4) & 1U;
+ } else {
+ prim_reg_val = 212U;
+ window_eval_mode = 0U;
+ }
+
+ pmu_cal_bist_cmd_strobe_dispatch((u32)(uintptr_t)scan_profile_table,
+ rank, mode, ch_mask,
+ (u32)(uintptr_t)scan_metric_min,
+ bound_flag, margin_lim_flag);
+
+ *(u32 *)(scan_profile_table + 0) = 0U;
+ *(u32 *)(scan_profile_table + 4) = 1U;
+ *(u32 *)(scan_profile_table + 8) = scan_multiplier;
+
+ sec_reg_val = prim_reg_val - 1U;
+
+ timing_reg_pri = (ch == 0U) ? 0x4eU : 0x50U;
+ timing_reg_sec = (ch == 0U) ? 0x4fU : 0x51U;
+ slice_reg_stride = (ch_mask << (ch != 0U)) + 1U;
+
+ if (alt_reg_sel & 1U) {
+ timing_reg_sec = timing_reg_alt_b;
+ timing_reg_pri = timing_reg_alt_a;
+ slice_reg_stride = channel_stride_adj;
+ } else if (pass_reg_a & 1U) {
+ timing_reg_sec = timing_reg_pri;
+ timing_reg_pri = (ch == 0U) ? 0x4fU : 0x51U;
+ slice_reg_stride = rank << (ch != 0U);
+ }
+
+ for (k = 0U; k < channel_scan_mask; k++) {
+ u32 target_timing_reg, source_timing_reg, slice_active_reg;
+ u8 dly_step_flag, scan_direction = 0U;
+ u32 accumulated_bias = 0U, cfg_bit13, step_size, last_cur_dly, cur_dly;
+ u8 slice_start, slice_end, active_mask;
+ u32 lane_div, s_idx;
+
+ pmu_margin_matrix_select_pass_regs(k, mode, profile_mode,
+ keep_pass_regs,
+ timing_reg_pri,
+ timing_reg_sec,
+ slice_reg_stride,
+ &target_timing_reg,
+ &source_timing_reg,
+ &slice_active_reg);
+
+ if (flag != 0U) {
+ dly_step_flag = (u8)flag;
+ pmu_dbyte_dq_deskew_regs_save_and_ramp(&scan_direction,
+ &vref_boundary_clamp,
+ &dly_step_flag,
+ slice_active_reg,
+ wide_win_flag);
+ } else {
+ vref_boundary_clamp = 0U;
+ dly_step_flag = 1U;
+ }
+
+ if (wide_win_flag)
+ accumulated_bias =
+ (pmu_stage_slice_flag_check() != 0U) ? 1U : 0U;
+
+ cfg_bit13 = pmu_dram_cfg_bit13_check(slice_active_reg,
+ wide_win_flag);
+
+ if (cfg_bit13 != 0U && ch == 0U)
+ pmu_margin_matrix_capture_baseline(baseline_buf,
+ cfg_bit13,
+ target_timing_reg,
+ scan_direction,
+ vref_boundary_clamp);
+
+ memset((void *)(PMU_DMEM_BASE | PMU_DMEM_SWEEP_SAMPLE_BUF), 0, 0x8480U);
+ step_size = dly_step_flag;
+ last_cur_dly = scan_direction;
+
+ for (cur_dly = scan_direction;
+ cur_dly <= vref_boundary_clamp;
+ cur_dly += step_size) {
+ void *sweep_buf = (void *)(PMU_DMEM_BASE |
+ PMU_DMEM_SWEEP_SAMPLE_BUF);
+ u32 active_csr_offset;
+
+ last_cur_dly = cur_dly;
+
+ if (flag != 0U) {
+ if (mode == 2U)
+ pmu_cal_pulse_seq_coordinator(0U, ch, (u8)cur_dly,
+ 3U);
+ else
+ pmu_cal_margin_matrix_prog_step(cur_dly, cfg_bit13,
+ target_timing_reg,
+ source_timing_reg,
+ accumulated_bias,
+ baseline_buf);
+ }
+
+ active_csr_offset = dmem_read32(PMU_DMEM_ACTIVE_CSR_OFFSET);
+ phy_write16(PHY_REG_DBYTE_BCAST_PULSE | (active_csr_offset << 1),
+ 0U);
+
+ *(u32 *)scan_profile_table = cur_dly;
+
+ pmu_cal_slice_step_eval_sweep(sweep_buf,
+ rank, mode, cfg_bit13, 0U,
+ (u8)channel,
+ window_eval_mode, scan_metric_min,
+ bound_flag,
+ margin_lim_flag, scan_profile_table);
+
+ if (margin_diff_accum != 0U) {
+ pmu_dbyte_rx_en_slice_pulse_eval(1U, ch_mask);
+ pmu_cal_bist_pattern_setup(rank, ch_mask,
+ 0U, margin_lim_flag);
+ pmu_cal_slice_step_eval_sweep(sweep_buf,
+ rank, 3U, cfg_bit13, 0U,
+ (u8)channel,
+ window_eval_mode,
+ scan_metric_min, 0U,
+ margin_lim_flag,
+ scan_profile_table);
+ pmu_cal_bist_pattern_setup(rank, ch_mask, status_query,
+ margin_lim_flag);
+ pmu_dbyte_rx_en_slice_pulse_eval(0U, ch_mask);
+ }
+ }
+
+ slice_start = dmem_read8(PMU_DMEM_SLICE_START);
+ slice_end = dmem_read8(PMU_DMEM_SLICE_END);
+ active_mask = dmem_read8(PMU_DMEM_ACTIVE_SLICE_MASK_CAL);
+ lane_div = 64U / scan_multiplier;
+
+ for (s_idx = slice_start; s_idx <= slice_end; s_idx++) {
+ u32 lane;
+
+ if (!(active_mask & (1U << s_idx)))
+ continue;
+
+ for (lane = 0U; lane < 10U; lane++)
+ pmu_cal_margin_matrix_eval_lane(k, ch, s_idx, lane,
+ prim_reg_val, sec_reg_val,
+ lane_div, scan_multiplier,
+ flag, last_cur_dly,
+ step_size, cfg_bit13,
+ baseline_buf[s_idx][lane]);
+ }
+
+ for (s_idx = slice_start; s_idx <= slice_end; s_idx++) {
+ u32 lane;
+
+ if (!(active_mask & (1U << s_idx)))
+ continue;
+
+ for (lane = 0U; lane < 10U; lane++) {
+ if (bound_flag == 0U && lane == 8U)
+ continue;
+ if (margin_diff_accum == 0U && lane == 9U)
+ continue;
+
+ pmu_cal_margin_matrix_center_lane(k, ch, s_idx, lane,
+ scan_metric_max[s_idx],
+ scan_multiplier);
+ }
+ }
+ }
+
+ pmu_dmem_reg_stream_unpack(scan_profile_table);
+ }
+
+ if (margin_lim_flag != 0U)
+ pmu_cal_pad_impedance_lock();
+}
+
+/**
+ * pmu_cal_lane_window_margin_eval() - Evaluate lane window margins and update timing tables
+ * @mode: Evaluation mode (0 for full rank/slice clamp and dispatch)
+ */
+void pmu_cal_lane_window_margin_eval(u32 mode)
+{
+ u8 *base = (u8 *)(PMU_DMEM_BASE | PMU_DMEM_CAL_STRUCT_BASE);
+ u32 ch;
+ u32 rank;
+ u32 csr_offset;
+ u8 count;
+ u8 misc_flag;
+
+ pmu_cal_timing_table_matrix_init(base);
+
+ pmu_cal_eye_table_save((void *)(uintptr_t)((u32)(uintptr_t)base), 0x0bU);
+ pmu_cal_eye_table_save((void *)(uintptr_t)((u32)(uintptr_t)base), 0x11U);
+ pmu_cal_lane_mask_and_all((u32)(uintptr_t)base, 0x12U, 0xefU);
+ pmu_cal_eye_table_save((void *)(uintptr_t)((u32)(uintptr_t)base), 0x12U);
+ pmu_cal_eye_table_save((void *)(uintptr_t)((u32)(uintptr_t)base), 0x29U);
+
+ for (ch = 0U; ch < 2U; ch++) {
+ for (rank = 0U; rank < 2U; rank++)
+ pmu_cal_training_flags_gate_update(base +
+ ch * (2U * PMU_SLICE_DESC_SIZE) +
+ rank * PMU_SLICE_DESC_SIZE);
+ }
+
+ dmem_write8(PMU_DMEM_LANE_ACTIVE_FLAGS, 0U);
+ dmem_write8(PMU_DMEM_CAL_BYTE, 0U);
+
+ for (ch = 0U; ch < 2U; ch++) {
+ for (rank = 0U; rank < 2U; rank++)
+ base[ch * (2U * PMU_SLICE_DESC_SIZE) +
+ rank * PMU_SLICE_DESC_SIZE + 3U] |= 0x40U;
+ }
+ pmu_phy_reg_offset_diff_adjust(1U);
+
+ pmu_slice_lane_status_save();
+
+ csr_offset = dmem_read32(PMU_DMEM_ACTIVE_CSR_OFFSET);
+ phy_write16(PHY_REG_VREF_CTRL | (csr_offset << 1), 0U);
+ phy_write16(PHY_REG_VREF_TRIM_CFG, 1U);
+
+ {
+ const u8 *d0 = (const u8 *)PMU_DMEM_BASE;
+
+ misc_flag = d0[0];
+ if (d0[0x18] == 0U) {
+ u32 dly1 = (misc_flag & 2U) ? 0x1e8480UL : 0xbebc200UL;
+ u32 dly2 = (misc_flag & 2U) ? 0x1e8480UL : 0x77359400UL;
+
+ pmu_delay_us(dly1, 0U);
+ if (d0[0xd] & 4U)
+ pmu_mailbox_post_cmd_dispatch(0x40U);
+ phy_write16(PHY_REG_PHY_STATUS, 3U);
+ pmu_delay_us(dly2, 0U);
+ }
+ if (d0[0xd] & 1U)
+ phy_write16(PHY_REG_PHY_STATUS, 1U);
+ }
+
+ pmu_cal_strobe_pulse_init();
+ count = (u8)pmu_cal_delay_line_accum_dispatch(0xbb8U, 0U);
+ if (count != 0U) {
+ do {
+ pmu_cal_sequence_pulse_send(0U, 7U, 0U, 0U, 0U, 3U, 0U);
+ } while (--count != 0U);
+ }
+ pmu_cal_sequence_pulse_send(0x80U, 7U, 0U, 0U, 0U, 3U, 0U);
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+
+ if (dmem_read8(PMU_DMEM_AC_CAL_ACTIVE) == 0U || (misc_flag & 8U) != 0U) {
+ pmu_delay_us(0x1e8480UL, 0U);
+ } else {
+ pmu_ac_lane_profile_setup();
+ pmu_cal_strobe_pulse_init();
+
+ pmu_cal_pad_timing_delay_set(0U, 8U, 1U);
+ if (dmem_read8(PMU_DMEM_AC_SIGNED_STEP) >= 2U)
+ pmu_cal_pad_timing_delay_set(0U, 0U, 1U);
+ pmu_cal_pad_timing_delay_set(0U, 10U, 1U);
+ pmu_cal_pad_timing_delay_set(8U, 0U, 1U);
+ pmu_cal_pad_timing_delay_set(0U, 0x96U, 1U);
+ pmu_cbt_3phase_pulse_seq();
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+ pmu_phy_profile_param_program();
+ }
+
+ if (mode == 0U) {
+ u8 num_entries;
+ u32 entry;
+
+ pmu_cal_gate_eye_master_trigger();
+ pmu_multirank_slice_cal_dispatch(base, 1U, 0xeffefffdUL,
+ (u32)-1, 0xffU, 0U);
+ pmu_cal_eye_table_restore(base, 0x0bU);
+ pmu_cal_eye_table_restore(base, 0x11U);
+ pmu_cal_eye_table_restore(base, 0x12U);
+ pmu_cal_eye_table_restore(base, 0x29U);
+
+ num_entries = dmem_read8(PMU_DMEM_METRIC_TABLE_ENTRIES);
+ for (entry = 0U; entry < num_entries; entry++) {
+ u8 offset = dmem_read8(PMU_DMEM_CAL_CMD_TABLE + entry);
+ u8 *entry_base = base + offset;
+
+ for (rank = 0U; rank < 2U; rank++) {
+ for (ch = 0U; ch < 2U; ch++) {
+ u8 val = entry_base[ch * (2U * PMU_SLICE_DESC_SIZE) +
+ rank * PMU_SLICE_DESC_SIZE];
+
+ dmem_write8(PMU_DMEM_DESKEW_STAT_TBL +
+ ch * 2U + entry * 4U + rank,
+ val);
+ }
+ }
+ }
+ }
+
+ pmu_cal_delay_line_init();
+
+ {
+ u8 *dmem = (u8 *)PMU_DMEM_BASE;
+
+ dmem[0x33] = dmem[0x6e];
+ dmem[0x34] = dmem[0x6f];
+ dmem[0x4e] = dmem[0x70];
+ dmem[0x4f] = dmem[0x71];
+ dmem[0x35] = dmem[0x76];
+ dmem[0x36] = dmem[0x77];
+ dmem[0x50] = dmem[0x78];
+ dmem[0xd0] = dmem[0x7a];
+ dmem[0x51] = dmem[0x79];
+ dmem[0xd5] = dmem[0x7b];
+ dmem[0xda] = dmem[0x7c];
+ dmem[0xd1] = dmem[0x9a];
+ dmem[0xdf] = dmem[0x7d];
+ dmem[0xd6] = dmem[0x9b];
+ dmem[0xdb] = dmem[0x9c];
+ dmem[0xe0] = dmem[0x9d];
+ dmem[0xd2] = dmem[0xae];
+ dmem[0xd7] = dmem[0xaf];
+ dmem[0xdc] = dmem[0xb0];
+ dmem[0xe1] = dmem[0xb1];
+ }
+}
+
+/**
+ * pmu_eye_margin_window_search() - Search 2D eye margin window for optimal center
+ * @samples: 256-byte eye sample buffer (updated at [130..131] on success)
+ * @reserved_arg: Reserved parameter
+ *
+ * Return: 0 on success, 5 if no valid metric found, or error code.
+ */
+noinline u32 pmu_eye_margin_window_search(u8 *samples, u32 reserved_arg)
+{
+ u8 *s_reg = samples;
+ const u8 *p_margin;
+ u32 best_metric = 0U;
+ u8 scan_lut[512];
+ u8 out_margins[6];
+ u8 out_mode;
+ u32 ret;
+ u32 count;
+
+ (void)reserved_arg;
+ pmu_eye_sample_table_transform(scan_lut, s_reg);
+ ret = pmu_sample_margin_center_eval(s_reg, out_margins, &out_mode);
+ if (ret != 0U)
+ return ret;
+
+ if ((u32)(out_mode - 2U) <= 1U) {
+ p_margin = out_margins + 4;
+ count = out_mode;
+ } else {
+ p_margin = out_margins;
+ count = 1U;
+ }
+
+ do {
+ u8 pair[2];
+ u32 metric;
+
+ metric = pmu_cal_margin_sample_pair_extract(pair,
+ p_margin,
+ s_reg,
+ scan_lut);
+
+ if (metric != 0U && metric >= best_metric) {
+ best_metric = metric;
+ s_reg[131] = pair[1];
+ s_reg[130] = pair[0];
+ }
+ p_margin -= 2;
+ } while (--count);
+
+ return (best_metric == 0U) ? 5U : 0U;
+}
+
+/**
+ * pmu_rank_margin_eval_zero_tap() - Evaluate per-channel slice/pin margins when tap_ptr is zero
+ * @start_slice: Starting DBYTE slice index
+ * @end_slice: Ending DBYTE slice index
+ * @rank_limit: Active rank limit (1 or 2)
+ * @mode: PHY operating profile mode (0..3)
+ * @channel: Target channel index (0 or 1)
+ * @slice_mask: Active pin/lane bitmask
+ *
+ * Return: 0 on success, or non-zero calibration error code.
+ */
+static u32 pmu_rank_margin_eval_zero_tap(u32 start_slice, u32 end_slice,
+ u32 rank_limit, u32 mode, u32 channel,
+ u32 slice_mask)
+{
+ u32 margin_window_ret = 0U;
+ u32 slice;
+ u8 c0, c1;
+
+ c0 = dmem_read8(PMU_DMEM_CH0_CTRL_LO);
+ dmem_write16(PMU_DMEM_METRIC_COEFF_X, (u16)c0 * c0);
+ c1 = dmem_read8(PMU_DMEM_CH0_CTRL_HI);
+ dmem_write16(PMU_DMEM_METRIC_COEFF_Y, (u16)c1 * c1);
+
+ for (slice = start_slice; slice <= end_slice; slice++) {
+ u32 pin;
+
+ dmem_write8(PMU_DMEM_STAGE_SLICE_IDX_ADDR, (u8)slice);
+
+ for (pin = 0U; pin < 10U; pin++) {
+ u32 off;
+ u8 *rank0, *rank1, *rank2, *rank3;
+
+ if (!(slice_mask & (1U << pin)))
+ continue;
+
+ dmem_write8(PMU_DMEM_STAGE_LANE_IDX_ADDR, (u8)pin);
+
+ off = slice * 1320U + pin * 132U;
+ rank0 = (u8 *)(uintptr_t)((PMU_DMEM_BASE |
+ PMU_DMEM_EYE_SAMPLE_BUF) + off);
+ rank1 = rank0 + 5280U;
+ rank2 = rank0 + (2U * 5280U);
+ rank3 = rank0 + (3U * 5280U);
+
+ if (rank_limit == 1U && mode < 3U) {
+ u32 ret = pmu_cal_slice_mask_eval(rank0);
+
+ if (ret != 0U)
+ return (u8)ret;
+ margin_window_ret = 0U;
+ if (dmem_read8(PMU_DMEM_CAL_ACTIVE_FLAG) != 0U)
+ pmu_eye_margin_bidir_scan(rank0, 0U, channel, 0U);
+ } else if (rank_limit == 1U && mode == 3U) {
+ u32 ret = pmu_eye_margin_step_balancer(rank0, rank0,
+ rank2, rank2);
+
+ if (ret != 0U)
+ return (u8)ret;
+ if (dmem_read8(PMU_DMEM_CAL_ACTIVE_FLAG) != 0U) {
+ pmu_eye_margin_bidir_scan(rank0, 0U, channel,
+ margin_window_ret);
+ pmu_eye_margin_bidir_scan(rank2, 0U, channel,
+ margin_window_ret);
+ }
+ } else if (rank_limit == 1U) {
+ pmu_assert_or_halt(0U, PMU_ERR_EYE_RANK_LIMIT_1);
+ } else if (mode < 2U) {
+ u32 ret = pmu_eye_margin_step_balancer(rank0, rank1,
+ rank0, rank1);
+
+ if (ret != 0U)
+ return (u8)ret;
+ if (dmem_read8(PMU_DMEM_CAL_ACTIVE_FLAG) != 0U) {
+ pmu_eye_margin_bidir_scan(rank0, 0U, channel, 0U);
+ pmu_eye_margin_bidir_scan(rank1, 0U, channel, 1U);
+ }
+ } else if (mode == 2U) {
+ u32 ret = pmu_cal_slice_mask_eval(rank0);
+
+ if (ret != 0U)
+ return (u8)ret;
+ if (dmem_read8(PMU_DMEM_CAL_ACTIVE_FLAG) != 0U)
+ pmu_eye_margin_bidir_scan(rank0, 0U, channel, 0U);
+
+ ret = pmu_cal_slice_mask_eval(rank1);
+ if (ret != 0U)
+ return (u8)ret;
+ margin_window_ret = 1U;
+ if (dmem_read8(PMU_DMEM_CAL_ACTIVE_FLAG) != 0U)
+ pmu_eye_margin_bidir_scan(rank1, 0U, channel, 1U);
+ } else if (mode == 3U) {
+ u32 ret = pmu_eye_margin_step_balancer(rank0, rank1,
+ rank2, rank3);
+
+ if (ret != 0U)
+ return (u8)ret;
+ if (dmem_read8(PMU_DMEM_CAL_ACTIVE_FLAG) != 0U) {
+ pmu_eye_margin_bidir_scan(rank0, 0U, channel, 0U);
+ pmu_eye_margin_bidir_scan(rank1, 0U, channel, 1U);
+ pmu_eye_margin_bidir_scan(rank2, 0U, channel, 0U);
+ pmu_eye_margin_bidir_scan(rank3, 0U, channel, 1U);
+ }
+ } else {
+ pmu_assert_or_halt(0U, PMU_ERR_EYE_RANK_MODE_INVALID);
+ }
+ }
+ }
+ return 0U;
+}
+
+/**
+ * pmu_cal_rank_margin_window_check() - Validate rank/slice margin windows and balance centers
+ * @tap_ptr: Packed tap pointer or 0 for default per-channel mode
+ * @channel: Target channel index (0 or 1)
+ * @slice_mask: Active pin/lane bitmask
+ *
+ * Return: 0 on success, or non-zero calibration error code.
+ */
+u32 pmu_cal_rank_margin_window_check(u32 tap_ptr, u32 channel, u32 slice_mask)
+{
+ u32 start_slice = 0U;
+ u32 end_slice = 0U;
+ u32 rank_limit = 0U;
+ u32 mode = pmu_phy_query_operating_mode();
+ u32 ptr_addr;
+ u8 c0;
+ u8 c1;
+ u32 table_ptr_base;
+ u32 rank;
+
+ if (channel == 1U) {
+ u8 ch1_rank_cfg;
+
+ end_slice = dmem_read8(PMU_DMEM_RANK1_SLICE_END);
+ start_slice = dmem_read8(PMU_DMEM_RANK1_SLICE_START);
+ ch1_rank_cfg = dmem_read8(PMU_DMEM_CH1_RANK_EN);
+ if (ch1_rank_cfg == 3U)
+ rank_limit = 2U;
+ else if (ch1_rank_cfg == 1U)
+ rank_limit = 1U;
+ else
+ rank_limit = 0U;
+ } else if (channel == 0U) {
+ u8 ch0_rank_cfg;
+
+ end_slice = dmem_read8(PMU_DMEM_RANK_BOUNDARY);
+ start_slice = dmem_read8(PMU_DMEM_RANK0_SLICE_START);
+ ch0_rank_cfg = dmem_read8(PMU_DMEM_CH0_RANK_EN);
+ rank_limit = (ch0_rank_cfg == 3U) ? 2U : 1U;
+ }
+
+ if (tap_ptr == 0U)
+ return pmu_rank_margin_eval_zero_tap(start_slice, end_slice,
+ rank_limit, mode, channel,
+ slice_mask);
+
+ ptr_addr = (tap_ptr >> 15) | (tap_ptr << 17);
+ c0 = *(__volatile__ u8 *)(uintptr_t)ptr_addr;
+ dmem_write16(PMU_DMEM_METRIC_COEFF_X, (u16)c0 * c0);
+ c1 = *(__volatile__ u8 *)(uintptr_t)(ptr_addr + 1U);
+ dmem_write16(PMU_DMEM_METRIC_COEFF_Y, (u16)c1 * c1);
+
+ if (rank_limit == 0U)
+ return 0U;
+
+ table_ptr_base = (PMU_DMEM_BASE | PMU_DMEM_EYE_SAMPLE_BUF) +
+ (1320U * start_slice);
+
+ for (rank = 0U; rank < rank_limit; rank++, table_ptr_base += 5280U) {
+ u8 eye_window_buf_a[132];
+ u8 eye_window_buf_b[132];
+ int i;
+ u32 slice_ptr;
+ u32 slice;
+ u32 ret;
+
+ dmem_write8(PMU_DMEM_STAGE_RANK_IDX_ADDR, (u8)rank);
+
+ for (i = 0; i < 64; i++) {
+ ((u16 *)eye_window_buf_a)[i] = 0x7f00U;
+ ((u16 *)eye_window_buf_b)[i] = 0x7f00U;
+ }
+ *(u32 *)&eye_window_buf_a[128] = 0x3f1f0000UL;
+ *(u32 *)&eye_window_buf_b[128] = 0x001f0000UL;
+
+ slice_ptr = table_ptr_base;
+ for (slice = start_slice; slice <= end_slice;
+ slice++, slice_ptr += 1320U) {
+ u8 *pin_ptr = (u8 *)(uintptr_t)slice_ptr;
+ u32 pin;
+
+ dmem_write8(PMU_DMEM_STAGE_SLICE_IDX_ADDR, (u8)slice);
+
+ for (pin = 0U; pin < 10U; pin++, pin_ptr += 132U) {
+ u32 swap;
+ u8 *target_buf;
+ s32 delta;
+
+ if (!(slice_mask & (1U << pin)))
+ continue;
+
+ ret = pmu_window_centroid_calc(pin_ptr);
+ dmem_write8(PMU_DMEM_STAGE_LANE_IDX_ADDR, (u8)pin);
+ if (ret != 0U)
+ return (u8)ret;
+
+ swap = pmu_cal_lane_error_status_check(rank, slice);
+ target_buf = (swap != 0U) ? eye_window_buf_a :
+ eye_window_buf_b;
+
+ delta = (s8)pin_ptr[130] - (s8)target_buf[130];
+ ret = pmu_window_margin_diff_calc(target_buf,
+ pin_ptr,
+ target_buf,
+ delta, 0);
+ if (ret != 0U)
+ return (u8)ret;
+ }
+ }
+
+ ret = pmu_cal_slice_mask_eval(eye_window_buf_a);
+ if (ret != 0U)
+ return (u8)ret;
+
+ ret = pmu_cal_slice_mask_eval(eye_window_buf_b);
+ if (ret != 0U)
+ return (u8)ret;
+
+ if (start_slice <= end_slice) {
+ slice_ptr = table_ptr_base;
+ for (slice = start_slice; slice <= end_slice;
+ slice++, slice_ptr += 1320U) {
+ u8 *pin_ptr = (u8 *)(uintptr_t)slice_ptr;
+ u32 pin;
+
+ dmem_write8(PMU_DMEM_STAGE_SLICE_IDX_ADDR, (u8)slice);
+
+ for (pin = 0U; pin < 10U;
+ pin++, pin_ptr += 132U) {
+ u32 swap;
+ u8 *target_buf;
+ u8 target_y;
+
+ if (!(slice_mask & (1U << pin)))
+ continue;
+
+ dmem_write8(PMU_DMEM_STAGE_LANE_IDX_ADDR,
+ (u8)pin);
+
+ swap = pmu_cal_lane_error_status_check(rank, slice);
+ target_buf = (swap != 0U) ?
+ eye_window_buf_a :
+ eye_window_buf_b;
+ target_y = target_buf[131];
+
+ ret = pmu_eye_boundary_sample_search(pin_ptr,
+ target_y);
+ if (ret != 0U)
+ return (u8)ret;
+
+ if (dmem_read8(PMU_DMEM_CAL_ACTIVE_FLAG) != 0U)
+ pmu_eye_margin_bidir_scan(pin_ptr,
+ tap_ptr,
+ channel,
+ rank);
+ }
+ }
+ }
+ }
+
+ return 0U;
+}
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 18/26] sunxi: dram: a733: Add ARC PMU training state machine
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (16 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 17/26] sunxi: dram: a733: Add ARC PMU 1D/2D eye " Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 19/26] sunxi: dram: a733: Add Allwinner A733 LPDDR5 DRAM driver Alastair D'Silva
` (9 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
Add the stage dispatcher, ZQ pad calibration, and multi-rank parameter
commit routines for the Allwinner A733 DDR PHY ARC EM4 firmware.
Written from clean-room hardware register and JEDEC JESD209-5 protocol
specifications (see doc/board/allwinner/a733-pmu.rst).
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
arch/arm/mach-sunxi/pmu/pmu_train.c | 2748 +++++++++++++++++++++++++++
1 file changed, 2748 insertions(+)
create mode 100644 arch/arm/mach-sunxi/pmu/pmu_train.c
diff --git a/arch/arm/mach-sunxi/pmu/pmu_train.c b/arch/arm/mach-sunxi/pmu/pmu_train.c
new file mode 100644
index 00000000000..c0b5e1d7dd1
--- /dev/null
+++ b/arch/arm/mach-sunxi/pmu/pmu_train.c
@@ -0,0 +1,2748 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Copyright (C) 2026 Alastair D'Silva <alastair@d-silva.org>
+ *
+ * Allwinner A733 Synopsys DDR PHY PMU Top-Level Calibration Dispatch Helpers
+ */
+
+#include "pmu_fw.h"
+
+static void pmu_cal_ca_train_step(void);
+static void pmu_cal_ca_vref_step(u32 arg0);
+static u32 pmu_cal_dbyte_impedance_fine_tune(void);
+static void pmu_cal_multi_branch_dispatch(u16 *out_desc, u32 flags, u32 opcode, u32 dispatch_param,
+ u32 val_arg0, u32 val_arg1, u32 flag_mode);
+static void pmu_cal_multi_param_step_commit(u32 param_table, u32 step_val, u32 eval_mask,
+ u32 filter_mask, u32 slice_flags);
+static void pmu_cal_phase_delay_tracker_update(void);
+static int pmu_cal_scaled_step_exec(int count, u32 flags);
+static void pmu_cal_rx_en_pre_init(void);
+static void pmu_cal_post_deskew_finalize(void);
+static void pmu_cal_zq_pad_init(void);
+static void pmu_cal_state_restore_offset_prog(void);
+static void pmu_cal_timing_margin_eval(void);
+static u32 pmu_cal_zq_pad_calibration(u32 is_primary);
+static void pmu_multiparam_cal_dispatch(void);
+static void pmu_phy_deskew_reset_strobe(void);
+static void pmu_train_dispatcher(void);
+
+/**
+ * pmu_cal_slice_mask_eval() - Dispatch CA Vref or deskew engine by DCCM 0x0b bit 2
+ * @table_buf: Pointer to slice descriptor buffer for margin evaluation
+ *
+ * Return: Result returned by pmu_window_centroid_calc(@table_buf, 0) when bit 2
+ * of DCCM 0x8000000b is set, or pmu_eye_margin_window_search(@table_buf, 0)
+ * when clear.
+ */
+u32 pmu_cal_slice_mask_eval(void *table_buf)
+{
+ if (*(__volatile__ u8 *)(PMU_DMEM_BASE + PMU_DMEM_CAL_SELECT_FLAGS) & 0x04U)
+ return pmu_window_centroid_calc(table_buf);
+ return pmu_eye_margin_window_search(table_buf, 0U);
+}
+
+/**
+ * pmu_cal_training_flags_gate_update() - Update gate/mode status bits in descriptor
+ * @descriptor: Pointer to calibration control descriptor byte array
+ */
+void pmu_cal_training_flags_gate_update(u8 *descriptor)
+{
+ u32 cfg_bit13;
+
+ descriptor[11] &= 0x87U;
+ descriptor[1] &= (u8)~0x08U;
+ descriptor[19] &= (u8)~0x0cU;
+
+ cfg_bit13 = descriptor[13];
+ if (pmu_dmem_training_flags_eval() != 0U) {
+ u32 d3 = descriptor[3];
+
+ cfg_bit13 |= 0x20U;
+ d3 |= 0x80U;
+ descriptor[3] = (u8)d3;
+ }
+
+ descriptor[20] = 2U;
+ descriptor[13] = (u8)(cfg_bit13 & ~(1U << 4));
+ descriptor[16] = 0x40U;
+ descriptor[17] |= 0x38U;
+ descriptor[18] &= 0x03U;
+ descriptor[28] &= (u8)~(1U << 5);
+ descriptor += 20;
+ descriptor[21] &= 0x1fU;
+}
+
+/**
+ * pmu_cal_tracker_reset_and_apb_zero() - Stream halfword results to PHY CSR 0x90082000
+ * @buf: Pointer to 16-bit halfword array to stream
+ * @len: Number of 16-bit halfwords to stream (minimum 1)
+ */
+void pmu_cal_tracker_reset_and_apb_zero(const void *buf, u32 len)
+{
+ const u16 *src = (const u16 *)buf;
+ u16 marker = pmu_sdata()->cal_marker_b;
+ u32 reg_addr = PHY_REG_CSR_RESULT_STREAM | ((u32)marker << 1);
+ u32 i = 0U;
+ u16 updated;
+
+ do {
+ *(__volatile__ u16 *)reg_addr = *src++;
+ reg_addr += 2U;
+ i++;
+ } while (i < len);
+
+ updated = (u16)(len + marker);
+ pmu_sdata()->cal_marker_a = updated;
+ pmu_sdata()->cal_marker_b = updated;
+}
+
+/**
+ * pmu_cal_scaled_step_exec() - Execute scaled calibration step count wrapper
+ * @count: Signed calibration step count (returns @count immediately if < 1)
+ * @flags: Control flags (bit 9 or DCCM 0x80000469 selects mode 0 vs mode 1)
+ *
+ * Return: Updated 16-bit calibration marker from pmu_cal_tracker_step_advance(), or
+ * @count when @count < 1.
+ */
+static int pmu_cal_scaled_step_exec(int count, u32 flags)
+{
+ u16 saved_marker;
+ u16 res;
+ int quotient;
+ u32 mode;
+
+ u16 cbt_state = pmu_sdata()->cbt_state;
+
+ if (count < 1)
+ return count;
+
+ quotient = cbt_state ? (count / (int)cbt_state) : count;
+ saved_marker = pmu_sdata()->cal_marker_a;
+ mode = ((flags & 0x0200UL) != 0UL || pmu_sdata()->step_mode_ovr != 0U) ?
+ 0U : 1U;
+
+ res = pmu_cal_tracker_step_advance(saved_marker, (u16)quotient, mode);
+ pmu_sdata()->cal_marker_a = res;
+ return (int)res;
+}
+
+/**
+ * pmu_cal_window_params_dispatch() - Pack 4-byte window parameters into 22-halfword tracker
+ * @params_buf: Pointer to 4-byte window parameter array
+ * @base_val: Base halfword value placed at index 5 and/or 13
+ * @mode_flags: Bitmask (bit 0 sets buf[5], bit 1 sets buf[13])
+ *
+ * Return: Result from pmu_cal_tracker_csr_commit(buf).
+ */
+u32 pmu_cal_window_params_dispatch(const u8 *params_buf, u16 base_val, u32 mode_flags)
+{
+ u16 buf[22];
+ u16 *b = memset(buf, 0, sizeof(buf));
+
+ b[3] = params_buf[0];
+ b[4] = params_buf[1];
+ b[11] = params_buf[2];
+ b[12] = params_buf[3];
+
+ if ((mode_flags & 1U) != 0U)
+ b[5] = base_val;
+ if ((mode_flags & 2U) != 0U)
+ b[13] = base_val;
+
+ return pmu_cal_tracker_csr_commit(b);
+}
+
+/**
+ * pmu_dly_reg_offset_select() - Select delay register offset by rank and mode
+ * @rank: Target rank index (0 or non-zero)
+ * @unused: Fallback direction selector when operating mode >= 3
+ * @direction: Sweep direction selector when operating mode == 2
+ *
+ * Return: Selected register index in {0x4e, 0x4f, 0x50, 0x51}.
+ */
+u32 pmu_dly_reg_offset_select(u32 rank, u32 unused, u32 direction)
+{
+ u32 mode = pmu_phy_query_operating_mode();
+ u32 base = (rank == 0U) ? 0x4fU : 0x4eU;
+ u32 dir_sel = (mode == 2U) ? direction : unused;
+
+ if (mode == 1U || (mode >= 2U && dir_sel != 0U))
+ base += 2U;
+
+ return base;
+}
+
+/**
+ * pmu_cal_state_restore_offset_prog() - Configure calibration strobe and Vref latch CSRs
+ */
+static void pmu_cal_state_restore_offset_prog(void)
+{
+ __volatile__ u16 *b5e = (__volatile__ u16 *)PHY_REG_PUB_SEC_BASE;
+ __volatile__ u16 *restore_trig_reg;
+ u32 cal_mode_val;
+ u16 vref_val;
+
+ cal_mode_val = pmu_cal_mode_flag_get();
+ pmu_delay_us(0U, 0x40U);
+ b5e[0x620 >> 1] = 1U;
+ b5e[0x1f0 >> 1] = 1U;
+
+ vref_val = *(__volatile__ const u16 *)PHY_REG_GATE_STAT;
+
+ if (cal_mode_val != 0U) {
+ b5e[0x186 >> 1] = 1U;
+ } else {
+ __volatile__ u16 *clk_gate_reg;
+ u16 cal_code;
+
+ vref_val &= 496U;
+ b5e[0x092 >> 1] = vref_val | (1U << 10);
+ pmu_delay_us(1000U, 0U);
+ pmu_delay_us(1000000U, 0U);
+ b5e[0x1c0 >> 1] = 1U;
+ vref_val |= (1U << 9);
+ clk_gate_reg = (__volatile__ u16 *)PHY_REG_MASTER_BASE;
+ cal_code = clk_gate_reg[0x19e >> 1];
+ b5e[0x186 >> 1] = 1U;
+ b5e[0x1c0 >> 1] = 0U;
+ b5e[0x092 >> 1] = vref_val;
+ clk_gate_reg[(0x09c >> 1) + pmu_sdata()->active_csr_offset] =
+ cal_code & 0x1fU;
+ }
+
+ restore_trig_reg = (__volatile__ u16 *)PHY_REG_CAL_RESTORE_TRIG;
+ if (pmu_sdata()->freq_mode != 0U)
+ restore_trig_reg += 5;
+ *restore_trig_reg = 1U;
+}
+
+/**
+ * pmu_freq_scaled_timing_step_calc() - Compute frequency-scaled timing parameter step
+ *
+ * Return: Clamped 8-bit timing step count (minimum 4).
+ */
+u32 pmu_freq_scaled_timing_step_calc(void)
+{
+ u32 mr19_mode = pmu_is_lpddr4();
+ u32 mr22_nibble = pmu_mr22_bit4_check();
+ u32 ps = (mr19_mode != 0U) ? 19000U :
+ ((mr22_nibble != 0U) ? 16000U : 12000U);
+ u32 steps;
+
+ if (*(__volatile__ const u8 *)PMU_DMEM_DESKEW_LIM == 0U)
+ ps += 2000U;
+
+ steps = pmu_freq_delay_step_calc(ps, 0U);
+ if (steps < 4U)
+ steps = 4U;
+
+ return (u8)steps;
+}
+
+/**
+ * pmu_cal_dbyte_impedance_fine_tune() - Build primary/secondary rank slice maps
+ *
+ * Return: Always 1.
+ */
+static u32 pmu_cal_dbyte_impedance_fine_tune(void)
+{
+ const __volatile__ u8 *low =
+ (const __volatile__ u8 *)PMU_DMEM_DESKEW_LIM;
+ __volatile__ u8 *slice_imp_tbl = (__volatile__ u8 *)PMU_DMEM_SLICE_IMP_CFG;
+ u32 rank_count = (pmu_phy_cal_stat_read() >> 6) & 0x3U;
+ u32 num_channels = pmu_sdata()->active_ch_cnt;
+ u32 rank, ch;
+
+ for (rank = 0U; rank < rank_count; rank++) {
+ for (ch = 0U; ch < num_channels; ch++) {
+ u8 rank_mask = (ch != 0U) ? low[36] : low[9];
+ const __volatile__ u8 *lanes;
+ u32 base_off;
+ u32 lane_start;
+ u32 lane_end;
+ __volatile__ u8 *pri;
+ __volatile__ u8 *sec;
+ u32 bit_query;
+ u32 swap;
+ u32 cur;
+
+ if ((rank_mask & (1U << rank)) == 0U)
+ continue;
+
+ lanes = slice_imp_tbl + ((ch != 0U) ? 0x48U : 0x46U);
+ lane_start = lanes[0];
+ lane_end = lanes[1];
+ base_off = (ch << 4) + (rank << 3);
+
+ pri = slice_imp_tbl + 0x20U + base_off;
+ sec = slice_imp_tbl + base_off;
+ bit_query = pmu_lane_mask_bit_get(ch, rank);
+ swap = ((low[0xe5] >> (rank + (ch << 1))) & 1U);
+
+ for (cur = lane_start; cur <= lane_end; cur += 2U) {
+ if (bit_query != 0U) {
+ *pri++ = (u8)cur;
+ *pri++ = (u8)(cur + 1U);
+ } else {
+ *pri++ = (u8)(cur + swap);
+ *sec++ = (u8)(cur + (swap ^ 1U));
+ }
+ }
+ }
+ }
+
+ return 1U;
+}
+
+/**
+ * pmu_cal_marker_win_step_prog() - Program calibration marker pair and window step diff
+ * @marker0: First 16-bit calibration marker value
+ * @marker1: Second 16-bit calibration marker value
+ * @win_param: Window base halfword parameter
+ * @win_mode: Window mode byte
+ */
+void pmu_cal_marker_win_step_prog(u32 marker0, u32 marker1, u32 win_param,
+ u32 win_mode)
+{
+ u32 buf_word = (win_mode & 0xffU) * 0x0101U;
+ u32 mode = (win_mode == 0U) ? 3U : 1U;
+
+ pmu_cal_marker_pair_set((u16)(marker0));
+ pmu_cal_window_params_dispatch((const u8 *)&buf_word, (u16)(win_param), (u16)(mode));
+ pmu_cal_marker_pair_set((u16)(marker1));
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+}
+
+/**
+ * pmu_pack_phase_desc() - Pack a 2-bit phase selector and 5-bit value into a 7-bit descriptor field
+ * @p: 2-bit phase selector placed in bits [5:4]
+ * @v: 5-bit value with bit 4 placed in bit 6 and bits [3:0] placed in bits [3:0]
+ *
+ * Return: Packed 16-bit descriptor field.
+ */
+static __always_inline u16 pmu_pack_phase_desc(u32 p, u32 v)
+{
+ return (u16)(((v << 2) & 0x40U) | ((p << 3) & 0x30U) | (v & 0x0fU));
+}
+
+/**
+ * pmu_cal_encode_opcode_pair() - Encode primary d3/d4 words for a calibration opcode
+ * @opcode: Calibration opcode (0..47)
+ * @dispatch_param: Opcode-specific parameter word
+ * @val_arg0: Primary value argument
+ * @flag_mode: Mode selector (0 vs non-zero)
+ * @d3_out: Output pointer for descriptor word 3
+ * @d4_out: Output pointer for descriptor word 4
+ */
+static void pmu_cal_encode_opcode_pair(u32 opcode, u32 dispatch_param,
+ u32 val_arg0, u32 flag_mode,
+ u16 *d3_out, u16 *d4_out)
+{
+ u16 xbfu_val = (u16)((dispatch_param >> 8) & 0x7fU);
+ u16 d3 = 0U, d4 = 0U;
+
+ switch (opcode) {
+ case 0:
+ if (flag_mode != 0U) {
+ d4 = (u16)(dispatch_param & 0x7fU);
+ d3 = (u16)(((dispatch_param >> 4) & 0x78U) | 3U);
+ } else {
+ d4 = (u16)(val_arg0 | ((dispatch_param >> 7) & 0x70U));
+ d3 = (u16)(((dispatch_param >> 11) & 0x78U) | 7U);
+ }
+ break;
+ case 1:
+ d4 = pmu_pack_phase_desc(dispatch_param, val_arg0);
+ d3 = (u16)(((dispatch_param << 3) & 0x08U) |
+ ((dispatch_param >> 1) & 0x1cU) | 1U);
+ break;
+ case 2:
+ d4 = pmu_pack_phase_desc(dispatch_param, val_arg0);
+ d3 = (u16)(((dispatch_param << 3) & 0x08U) |
+ ((dispatch_param >> 1) & 0x18U) | 5U);
+ break;
+ case 3:
+ d4 = pmu_pack_phase_desc(dispatch_param, val_arg0);
+ d3 = (u16)(((dispatch_param << 3) & 0x08U) |
+ ((dispatch_param >> 1) & 0x18U) | 6U);
+ break;
+ case 4:
+ d4 = pmu_pack_phase_desc(dispatch_param, val_arg0);
+ d3 = (u16)(((dispatch_param >> 1) & 0x18U) | 4U);
+ break;
+ case 5:
+ d4 = (u16)(val_arg0 & 0x7fU);
+ d3 = (u16)(((dispatch_param >> 3) & 0x70U) | 0x0cU);
+ break;
+ case 6:
+ if (flag_mode == 0U) {
+ d4 = (u16)(val_arg0 & 0x7fU);
+ d3 = 0x58U;
+ } else {
+ d4 = (u16)(dispatch_param & 0x7fU);
+ d3 = (u16)(((dispatch_param >> 1) & 0x40U) | 8U);
+ }
+ break;
+ case 7:
+ case 39:
+ break;
+ case 8:
+ case 27:
+ d3 = 0x78U;
+ d4 = (u16)(val_arg0 & 0x0fU);
+ break;
+ case 10:
+ case 17:
+ d3 = 0x38U;
+ d4 = (u16)(val_arg0 & 0x07U);
+ break;
+ case 11:
+ d3 = 0x68U;
+ d4 = (u16)((dispatch_param << 5) & 0x60U);
+ break;
+ case 12:
+ d3 = 0x28U;
+ break;
+ case 14:
+ if (flag_mode == 0U) {
+ d4 = xbfu_val;
+ d3 = 0x58U;
+ } else {
+ d4 = (u16)(dispatch_param & 0x7fU);
+ d3 = (u16)(((dispatch_param >> 1) & 0x40U) | 8U);
+ }
+ break;
+ case 15:
+ d4 = (u16)(val_arg0 & 0x7fU);
+ d3 = 0x18U;
+ break;
+ case 16:
+ d3 = 0x38U;
+ d4 = (u16)((val_arg0 & 0x07U) | 0x40U);
+ break;
+ case 25:
+ d4 = (u16)(dispatch_param & 0x7fU);
+ d3 = (u16)(((dispatch_param >> 1) & 0x40U) | 0x30U);
+ break;
+ case 26:
+ d3 = 0x78U;
+ d4 = (u16)((val_arg0 & 0x0fU) | 0x40U);
+ break;
+ case 28:
+ d4 = pmu_pack_phase_desc(dispatch_param, val_arg0);
+ d3 = (u16)(((dispatch_param >> 1) & 0x1cU) |
+ ((dispatch_param << 3) & 0x08U) | 2U);
+ break;
+ case 29:
+ d3 = 0x58U;
+ d4 = 0x58U;
+ break;
+ case 30:
+ d3 = xbfu_val;
+ d4 = xbfu_val;
+ break;
+ case 31:
+ d3 = (u16)(((dispatch_param >> 1) & 0x40U) | 8U);
+ d4 = d3;
+ break;
+ case 32:
+ case 38:
+ d3 = (u16)(dispatch_param & 0x7fU);
+ d4 = d3;
+ break;
+ case 33:
+ d3 = 0x18U;
+ break;
+ case 34:
+ d4 = xbfu_val;
+ break;
+ case 35:
+ d3 = (u16)(((dispatch_param >> 3) & 0x70U) | 0x0cU);
+ d4 = d3;
+ break;
+ case 36:
+ d3 = (u16)(val_arg0 & 0x7fU);
+ d4 = d3;
+ break;
+ case 37:
+ d3 = (u16)(((dispatch_param >> 1) & 0x40U) | 0x30U);
+ d4 = d3;
+ break;
+ case 40:
+ d3 = (flag_mode == 0U) ? 0x40U : 0U;
+ break;
+ case 41:
+ d3 = 0x60U;
+ break;
+ case 42:
+ d3 = 0x20U;
+ break;
+ case 43:
+ d3 = 0x50U;
+ break;
+ case 44:
+ d3 = 0x38U;
+ d4 = 0x38U;
+ break;
+ case 45:
+ d3 = (u16)((val_arg0 & 7U) | 0x40U);
+ d4 = d3;
+ break;
+ case 46:
+ d3 = (u16)(((dispatch_param << 3) & 0x08U) |
+ ((dispatch_param >> 1) & 0x1cU) | 1U);
+ d4 = d3;
+ break;
+ case 47:
+ d3 = pmu_pack_phase_desc(dispatch_param, val_arg0);
+ d4 = d3;
+ break;
+ default:
+ break;
+ }
+
+ *d3_out = d3;
+ *d4_out = d4;
+}
+
+/**
+ * pmu_cal_multi_branch_dispatch() - Format 8-halfword calibration command descriptor
+ * @out_desc: Output 8-halfword descriptor buffer
+ * @flags: Command flag bitmask
+ * @opcode: Calibration opcode (0..47)
+ * @dispatch_param: Opcode-specific parameter word
+ * @val_arg0: Primary value argument
+ * @val_arg1: Secondary value / slice mask argument
+ * @flag_mode: Mode selector (0 vs non-zero)
+ */
+static noinline void pmu_cal_multi_branch_dispatch(u16 *out_desc, u32 flags,
+ u32 opcode, u32 dispatch_param,
+ u32 val_arg0, u32 val_arg1,
+ u32 flag_mode)
+{
+ u16 fc = 0U;
+
+ out_desc[0] = 0U;
+ out_desc[1] = 0U;
+ out_desc[2] = 0U;
+ out_desc[6] = 0U;
+ out_desc[7] = 0U;
+
+ if (opcode <= 0x2fU)
+ pmu_cal_encode_opcode_pair(opcode, dispatch_param, val_arg0,
+ flag_mode, &out_desc[3],
+ &out_desc[4]);
+
+ if ((flags & (1U << 16)) != 0U ||
+ (opcode == 0x28U && flag_mode != 0U))
+ out_desc[5] = 0U;
+ else
+ out_desc[5] = (u16)(val_arg1 & 3U);
+
+ if ((flags & (1U << 20)) != 0U)
+ out_desc[7] = 13U;
+ else if ((flags & (1U << 21)) != 0U)
+ out_desc[7] = 0xd0U;
+ else if ((flags & (1U << 22)) != 0U)
+ out_desc[7] = 0xd00U;
+ else if ((flags & (1U << 23)) != 0U)
+ out_desc[7] = 8U;
+ else if ((flags & (1U << 24)) != 0U)
+ out_desc[7] = 0x80U;
+ else if ((flags & (1U << 25)) != 0U)
+ out_desc[7] = 1024U;
+ else if ((flags & (1U << 26)) != 0U)
+ out_desc[7] = 256U;
+ else if ((flags & (1U << 27)) != 0U)
+ out_desc[7] = 512U;
+
+ if ((flags & (1U << 19)) != 0U)
+ out_desc[7] |= 0x1000U;
+
+ if (opcode == 3U || opcode == 4U || opcode == 28U || opcode == 41U) {
+ fc = 4U;
+ } else if (opcode == 1U || opcode == 2U ||
+ opcode == 33U || opcode == 34U) {
+ fc = ((flags & (1U << 3)) != 0U) ? 0U : 2U;
+ if ((flags & 0x0cU) == 0x0cU)
+ fc |= 1U;
+ }
+
+ if ((flags & (1U << 18)) != 0U)
+ fc |= 3U;
+
+ if ((flags & (1U << 15)) != 0U)
+ out_desc[2] = 1U;
+
+ if ((flags & (1U << 12)) != 0U)
+ fc |= 8U;
+
+ out_desc[6] = fc;
+
+ if ((flags & (1U << 6)) != 0U || (val_arg1 & (1U << 1)) != 0U)
+ out_desc[0] = 1U;
+}
+
+/**
+ * pmu_scatter_slice_quad() - Scatter 4 per-slice profile bytes into a 4x54-byte slice matrix
+ * @buf: Destination 216-byte (4 x 54-byte) slice timing matrix buffer
+ * @off: Column offset within each 54-byte slice row (0..53)
+ * @src: Source 4-byte per-slice array in DCCM
+ */
+static __always_inline void pmu_scatter_slice_quad(u8 *buf, u32 off,
+ const __volatile__ u8 *src)
+{
+ buf[off] = src[0];
+ buf[off + PMU_SLICE_DESC_SIZE] = src[1];
+ buf[off + (2U * PMU_SLICE_DESC_SIZE)] = src[2];
+ buf[off + (3U * PMU_SLICE_DESC_SIZE)] = src[3];
+}
+
+/**
+ * pmu_cal_timing_table_matrix_init() - Unpack 29 slice profile quads into 4-slice matrix
+ * @buf: Destination 216-byte (4 x 54-byte) slice timing matrix buffer
+ */
+void pmu_cal_timing_table_matrix_init(u8 *buf)
+{
+ const __volatile__ u8 *src =
+ (const __volatile__ u8 *)PMU_DMEM_SLICE_PROFILE;
+
+ memset(buf, 0, PMU_SLICE_DESC_TOTAL_BYTES);
+
+ pmu_scatter_slice_quad(buf, 1U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 2U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 3U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 10U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 11U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 12U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 13U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 14U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 15U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 16U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 17U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 18U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 19U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 20U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 21U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 22U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 24U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 25U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 26U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 27U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 28U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 30U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 31U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 32U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 33U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 34U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 37U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 40U, src); src += 4;
+ pmu_scatter_slice_quad(buf, 41U, src);
+
+ buf[46] = 0U;
+ buf[100] = 0U;
+ buf[154] = 0U;
+ buf[208] = 0U;
+}
+
+/**
+ * pmu_cal_zq_pad_init() - Initialize Stage 8 pad configuration and log telemetry
+ */
+static void pmu_cal_zq_pad_init(void)
+{
+ u32 stat = pmu_phy_cal_stat_read();
+ u32 channels = (stat >> 4) & 3U;
+ u32 rank_metric = channels * (stat & 0x0fU);
+ u8 rank1_en;
+ u32 pad_cfg;
+ u32 val_8;
+ u16 pub_ctl;
+ u8 ctl_bit;
+ u8 pll_ratio;
+
+ pmu_sdata()->active_ch_cnt = channels;
+ pmu_sdata()->active_slice_cnt = rank_metric;
+
+ rank1_en = dmem_read8(PMU_DMEM_CH1_ACTIVE_FLAG);
+
+ if (rank1_en != 0U) {
+ pad_cfg = ((rank_metric << 2) & 248U) +
+ dmem_read8(PMU_DMEM_CH1_DQ_WIDTH);
+ } else {
+ pad_cfg = dmem_read8(PMU_DMEM_CH0_DQ_WIDTH);
+ if (channels == 2U) {
+ rank_metric >>= 1;
+ pmu_sdata()->active_slice_cnt = rank_metric;
+ }
+ }
+
+ val_8 = (u32)((u8)pad_cfg + 7U) >> 3;
+ pmu_sdata()->dbyte_slice_cnt = (u8)val_8;
+
+ pub_ctl = *(__volatile__ const u16 *)PHY_REG_PUB_CFG;
+ ctl_bit = 1U & ~((u8)(pub_ctl >> 2));
+ pmu_cal_win_bounds_clear();
+ pmu_sdata()->timer_shift = ctl_bit;
+
+ phy_write16(PHY_REG_PAD_CAL_CTRL, 1U);
+ phy_write16(PHY_REG_SEC_PUB_CTRL1, 1U);
+ phy_write16(PHY_REG_SEC_PUB_CTRL0, 0U);
+
+ pll_ratio = dmem_read8(PMU_DMEM_PLL_RATIO);
+ pmu_cbt_config_set(pll_ratio);
+
+ if (dmem_read16(PMU_DMEM_SEQ_CTRL) != 1U &&
+ (dmem_read8(PMU_DMEM_PSTATE_FLAGS) & 0xf0U) == 0x20U)
+ pmu_rank_shadow_status_sample();
+
+ if (dmem_read8(PMU_DMEM_AC_PROFILE_EN_ADDR & 0xffffU) != 0U)
+ pmu_cal_vref_trim_cfg_latch();
+}
+
+struct pmu_dly_line_entry {
+ u16 word_val;
+ u16 pad;
+};
+
+/**
+ * pmu_cal_delay_accum_dispatch() - Unpack, align, and pack 4-group LCDL coarse/fine pairs
+ * @dram_type: Active DRAM type identifier
+ * @count: Number of active slice entries (clamped to 4)
+ * @slice_base: Pointer to delay line entry matrix
+ */
+void pmu_cal_delay_accum_dispatch(u32 dram_type, u32 count, void *slice_base)
+{
+ u8 cand[4][4][2];
+ u8 thresh1[4];
+ u8 thresh2[4];
+ u32 max_grp1 = 0U;
+ u32 max_grp2 = 0U;
+ struct pmu_dly_line_entry (*buf)[2] =
+ (struct pmu_dly_line_entry (*)[2])slice_base;
+ u32 use_max;
+ u32 i, g, k;
+
+ if (count == 0U)
+ return;
+ if (count > 4U)
+ count = 4U;
+
+ for (i = 0U; i < count; i++) {
+ u8 d[4];
+ u32 m1, m2;
+
+ for (g = 0U; g < 4U; g++) {
+ u16 v = buf[g][i].word_val;
+
+ cand[g][i][0] = (u8)(v >> 6);
+ cand[g][i][1] = (u8)(v & 0x3fU);
+ pmu_adjust_step_limit(cand[g][i], g >> 1);
+ }
+
+ pmu_eye_margin_step_align(cand[0][i], cand[1][i], 2U);
+ pmu_eye_margin_step_align(cand[2][i], cand[3][i], 2U);
+
+ for (g = 0U; g < 4U; g++) {
+ u8 c = cand[g][i][1];
+
+ d[g] = (c < 65U) ? (64U - c) : (c - 64U);
+ }
+
+ m1 = (d[0] > d[1]) ? d[0] : d[1];
+ m2 = (d[2] > d[3]) ? d[2] : d[3];
+ thresh1[i] = (u8)m1;
+ thresh2[i] = (u8)m2;
+ if (m1 > max_grp1)
+ max_grp1 = m1;
+ if (m2 > max_grp2)
+ max_grp2 = m2;
+ }
+
+ use_max = (dram_type < 4U && dram_type != 2U &&
+ *(__volatile__ const u8 *)PMU_DMEM_CAL_OVERRIDE == 0U);
+
+ for (k = 0U; k < count; k++) {
+ u32 t1 = use_max ? max_grp1 : thresh1[k];
+ u32 t2 = use_max ? max_grp2 : thresh2[k];
+
+ pmu_cal_lcdl_coarse_fine_pair_pack(k, t1, t2,
+ cand[2][k], cand[3][k],
+ cand[0][k], cand[1][k],
+ slice_base);
+ }
+}
+
+/**
+ * pmu_cal_pulse_clear_slot1() - Zero the second 8-halfword command slot in trk_words
+ * @trk_words: Base pointer to the 38-halfword command buffer at 0x80000d9c
+ */
+static void pmu_cal_pulse_clear_slot1(u16 *trk_words)
+{
+ __volatile__ u32 *slot1 = (__volatile__ u32 *)trk_words + 4;
+
+ slot1[0] = 0U;
+ slot1[1] = 0U;
+ slot1[2] = 0U;
+ slot1[3] = 0U;
+}
+
+/**
+ * pmu_cal_pulse_handle_type6() - Format and commit a fast-path type-6 pulse
+ * @marker_bytes: Tracker state pointer at pmu_gp + 88
+ * @phase_id: Target phase count
+ * @param_code: Calibration parameter code
+ * @tap_idx: Delay tap / register index
+ * @slice_mask: Target slice or rank bitmask
+ *
+ * Return: Residual step difference after committing the type-6 tracker descriptor.
+ */
+static int pmu_cal_pulse_handle_type6(__volatile__ u8 *marker_bytes, u32 phase_id,
+ u32 param_code, u32 tap_idx,
+ u32 slice_mask)
+{
+ __volatile__ u16 *alt_trk = (__volatile__ u16 *)PMU_DMEM_ALT_TRACKER;
+ u16 a5_low = (u16)(slice_mask & 3U);
+ u16 a5_shift = a5_low >> 1;
+ u16 marker_a = *(__volatile__ u16 *)marker_bytes;
+ u32 cbt_state;
+
+ alt_trk[0] = a5_shift;
+ alt_trk[3] = 0x58U;
+ alt_trk[4] = (u16)tap_idx;
+ alt_trk[5] = a5_low;
+ alt_trk[8] = a5_shift;
+ alt_trk[11] = (u16)(((param_code >> 1) & 64U) | 8U);
+ alt_trk[12] = (u16)(param_code & 0x7fU);
+ alt_trk[13] = a5_low;
+ *(__volatile__ u16 *)marker_bytes = marker_a + 8U;
+
+ pmu_cal_tracker_csr_commit(alt_trk);
+ cbt_state = marker_bytes[2];
+ if ((marker_a & 7U) == 0U)
+ cbt_state <<= 1;
+
+ return (int)phase_id - (int)cbt_state;
+}
+
+/**
+ * pmu_cal_pulse_handle_extra() - Format and advance a multi-step extra pulse
+ * @marker_bytes: Tracker state pointer at pmu_gp + 88
+ * @trk_words: Base pointer to the 38-halfword command buffer at 0x80000d9c
+ * @cmd_flags: In/out pointer to command control flags
+ * @pulse_type: Pulse opcode / sequence type
+ * @phase_id: Phase count after marker_cfg adjustment
+ * @param_code: Calibration parameter code
+ * @tap_idx: Delay tap / register index
+ * @slice_mask: Target slice or rank bitmask
+ * @extra: Multi-step pulse count
+ */
+static void pmu_cal_pulse_handle_extra(__volatile__ u8 *marker_bytes, u16 *trk_words,
+ u32 *cmd_flags, u32 pulse_type,
+ u32 phase_id, u32 param_code,
+ u32 tap_idx, u32 slice_mask, u32 extra)
+{
+ u16 marker_a = *(__volatile__ u16 *)marker_bytes;
+ u32 cbt_state;
+ int sub1_diff;
+
+ if ((marker_a & 7U) == 4U) {
+ pmu_cal_pulse_clear_slot1(trk_words);
+ *(__volatile__ u16 *)marker_bytes = marker_a + 4U;
+ pmu_cal_marker_status_update();
+ }
+
+ if (marker_bytes[3] == 0U)
+ marker_bytes[3] = marker_bytes[4];
+
+ pmu_cal_multi_branch_dispatch(trk_words, *cmd_flags, pulse_type,
+ param_code, tap_idx, slice_mask, 0U);
+
+ if (pmu_check_stage(pulse_type) == 1U)
+ pmu_cal_multi_branch_dispatch(trk_words + 8, *cmd_flags, pulse_type,
+ param_code, tap_idx, slice_mask,
+ 1U);
+ else
+ pmu_cal_pulse_clear_slot1(trk_words);
+
+ trk_words[17] = (u16)extra;
+ cbt_state = marker_bytes[2];
+ marker_a = *(__volatile__ u16 *)marker_bytes;
+ *(__volatile__ u16 *)marker_bytes = marker_a + 8U;
+ sub1_diff = (int)phase_id - (int)(cbt_state << 1);
+
+ if (sub1_diff < 1) {
+ trk_words[18] = 3U;
+ } else {
+ int div_val;
+ u16 cur_m;
+
+ trk_words[18] = 2U;
+ pmu_cal_marker_status_update();
+ cbt_state = marker_bytes[2];
+ div_val = cbt_state ?
+ ((sub1_diff + (int)cbt_state - 1) / (int)cbt_state) :
+ 0;
+ cur_m = *(__volatile__ u16 *)marker_bytes;
+ *(__volatile__ u16 *)marker_bytes =
+ pmu_cal_tracker_step_advance(cur_m, (u16)div_val, 0U);
+ *cmd_flags |= (1U << 11);
+ }
+}
+
+/**
+ * pmu_cal_sequence_pulse_send() - Master calibration pulse sequence dispatcher
+ * @cmd_flags: Command control flags
+ * @pulse_type: Pulse opcode / sequence type
+ * @phase_id: Phase count or step target
+ * @param_code: Calibration parameter code
+ * @tap_idx: Delay tap / register index
+ * @slice_mask: Target slice or rank bitmask
+ * @extra: Non-zero for multi-step pulse dispatch
+ *
+ * Return: Calibration marker verification status.
+ */
+noinline u32 pmu_cal_sequence_pulse_send(u32 cmd_flags, u32 pulse_type,
+ u32 phase_id, u32 param_code,
+ u32 tap_idx, u32 slice_mask,
+ u32 extra)
+{
+ __volatile__ u8 *marker_bytes = (__volatile__ u8 *)&pmu_sdata()->cal_marker_a;
+ u16 *trk_words = (u16 *)pmu_tracker();
+ int step_diff;
+
+ while (1) {
+ u8 marker_cfg = marker_bytes[0x13];
+ u16 marker_a;
+ u32 chk;
+
+ if (pulse_type == 6U && marker_cfg != 0U) {
+ step_diff = pmu_cal_pulse_handle_type6(marker_bytes, phase_id,
+ param_code,
+ tap_idx,
+ slice_mask);
+ goto apply_out_mask;
+ }
+
+ if (marker_bytes[0x10] == 1U) {
+ u32 stage = pmu_check_stage(pulse_type);
+ u8 prefix = (stage == 1U) ? 0xf0U : 0x30U;
+
+ marker_bytes[8] = prefix + (marker_bytes[8] >> 4);
+ }
+
+ if ((cmd_flags & (1U << 8)) != 0U)
+ trk_words[16] |= (1U << 1);
+ if ((cmd_flags & (1U << 13)) != 0U)
+ trk_words[16] |= (1U << 0);
+
+ marker_a = *(__volatile__ u16 *)marker_bytes;
+
+ if (extra != 0U) {
+ if (marker_cfg == 0U)
+ phase_id++;
+ pmu_cal_pulse_handle_extra(marker_bytes, trk_words, &cmd_flags,
+ pulse_type, phase_id,
+ param_code, tap_idx,
+ slice_mask, extra);
+ goto out_set3;
+ }
+
+ if ((marker_a & 7U) == 0U) {
+ if (marker_bytes[3] == 0U)
+ marker_bytes[3] = marker_bytes[4];
+
+ pmu_cal_multi_branch_dispatch(trk_words, cmd_flags,
+ pulse_type, param_code,
+ tap_idx, slice_mask, 0U);
+ if (marker_cfg == 0U)
+ phase_id++;
+
+ *(__volatile__ u16 *)marker_bytes =
+ *(__volatile__ u16 *)marker_bytes + 4U;
+ if ((cmd_flags & (1U << 9)) != 0U)
+ trk_words[18] |= (1U << 1);
+
+ step_diff = (int)phase_id - (int)marker_bytes[2];
+ if (pmu_check_stage(pulse_type) == 1U) {
+ pmu_cal_multi_branch_dispatch(trk_words + 8,
+ cmd_flags,
+ pulse_type,
+ param_code,
+ tap_idx,
+ slice_mask, 1U);
+ *(__volatile__ u16 *)marker_bytes =
+ *(__volatile__ u16 *)marker_bytes + 4U;
+ pmu_cal_marker_verify();
+ step_diff -= (int)marker_bytes[2];
+ }
+ goto apply_out_mode3;
+ }
+
+ if ((marker_a & 7U) != 4U)
+ break;
+
+ if ((cmd_flags & (1U << 9)) != 0U || marker_bytes[4] != marker_bytes[3] ||
+ trk_words[18] != 0U) {
+ pmu_cal_pulse_clear_slot1(trk_words);
+ *(__volatile__ u16 *)marker_bytes = marker_a + 4U;
+ pmu_cal_marker_status_update();
+ continue;
+ }
+
+ chk = pmu_check_stage(pulse_type);
+ pmu_cal_multi_branch_dispatch(trk_words + 8, cmd_flags, pulse_type,
+ param_code, tap_idx, slice_mask,
+ 0U);
+ if (marker_cfg == 0U)
+ phase_id++;
+ *(__volatile__ u16 *)marker_bytes = *(__volatile__ u16 *)marker_bytes + 4U;
+
+ if (chk == 0U) {
+ step_diff = (int)phase_id - (int)marker_bytes[2];
+ if (step_diff >= 1) {
+ pmu_cal_marker_status_update();
+ pmu_cal_scaled_step_exec(step_diff, cmd_flags);
+ }
+ if ((cmd_flags & (1U << 11)) != 0U)
+ trk_words[18] |= (1U << 0);
+ break;
+ }
+
+ pmu_cal_marker_status_update();
+ pmu_cal_multi_branch_dispatch(trk_words, cmd_flags, pulse_type,
+ param_code, tap_idx, slice_mask,
+ 1U);
+ *(__volatile__ u16 *)marker_bytes = *(__volatile__ u16 *)marker_bytes + 4U;
+ step_diff = (int)phase_id - ((int)marker_bytes[2] << 1);
+apply_out_mask:
+ pmu_cal_scaled_step_exec(step_diff, cmd_flags);
+ break;
+apply_out_mode3:
+ pmu_cal_scaled_step_exec(step_diff, cmd_flags);
+out_set3:
+ marker_bytes[3] = marker_bytes[4];
+ break;
+ }
+
+ pmu_cal_marker_verify_and_buffer_clear(cmd_flags);
+ return 0U;
+}
+
+/**
+ * pmu_cal_pulse_seq_coordinator() - Coordinate rank calibration pulse sequence
+ * @ch_idx: Target channel index (0 or 1)
+ * @rank: Target rank index
+ * @param1: Frequency ratio parameter and tap 14/15 value
+ * @flags: Bitmask controlling tap 14 (bit 0) and tap 15 (bit 1) pulses
+ */
+void pmu_cal_pulse_seq_coordinator(u32 ch_idx, u32 rank, u32 param1, u32 flags)
+{
+ u32 ratio = pmu_freq_ratio_mult(0xfaU);
+ u32 offset;
+ u8 seq_ctrl_lo;
+ u32 mask1;
+ u8 mask;
+ u32 b;
+
+ pmu_cal_strobe_pulse_init();
+
+ offset = (ch_idx * 2U + rank) * PMU_SLICE_DESC_SIZE;
+ seq_ctrl_lo = *(__volatile__ const u8 *)
+ (PMU_DMEM_CAL_STRUCT_ADDR + offset + 0x10U);
+
+ mask1 = 1U << rank;
+ mask = (u8)((mask1 << 2) | mask1);
+
+ pmu_cal_sequence_pulse_send(0U, 6U, 0x22U, seq_ctrl_lo | (1U << 6),
+ 16U, mask, 0U);
+
+ ratio >>= 1;
+ for (b = 0U; b < 2U; b++) {
+ if (((flags >> b) & 1U) != 0U) {
+ u32 r = (b == 0U) ? (ratio + 8U) : ratio;
+
+ pmu_cal_sequence_pulse_send(0U, 7U, 2U, 0U, 0U, 0U,
+ (u8)r);
+ pmu_cal_sequence_pulse_send(0U, 6U, 0x22U, param1,
+ 14U + b, mask, 0U);
+ }
+ }
+
+ pmu_cal_sequence_pulse_send(0U, 7U, 2U, 0U, 0U, 0U, (u8)ratio);
+ pmu_cal_sequence_pulse_send(0U, 7U, 2U, 0U, 0U, 0U, 0x28U);
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+}
+
+/**
+ * pmu_phy_ac_lane_timing_delay_set() - Program AC lane timing delay sequence
+ * @lane: Target AC lane selector
+ *
+ * Return: Updated step stride minus 1.
+ */
+u16 pmu_phy_ac_lane_timing_delay_set(u32 lane)
+{
+ u32 zero = 0U;
+ u32 i;
+ u32 stride;
+
+ pmu_cal_marker_pair_set((u16)(800U));
+
+ pmu_cal_sequence_pulse_send(0x09U << 19, 5U, 0x0cU, 512U,
+ zero, lane, zero);
+ for (i = 0U; i < 3U; i++) {
+ pmu_cal_sequence_pulse_send(0U, 6U, 2U, 3U >> i, 0x1aU,
+ lane, zero);
+ if (i < 2U)
+ pmu_clk_ratio_pulse_trigger();
+ }
+
+ pmu_cal_pad_timing_delay_set(0U, 0x22U, 0U);
+
+ pmu_cal_sequence_pulse_send(0U, 5U, 4U, 896U, zero, lane, zero);
+ pmu_cal_sequence_pulse_send(0x80U | (1U << 19), 7U, 4U, 0U,
+ zero, zero, zero);
+
+ stride = pmu_cal_stride_get();
+ return (u16)(stride - 1U);
+}
+
+/**
+ * pmu_cal_pulse_burst_send() - Send repeated calibration pulse burst
+ * @burst_type: Non-zero for bit 27 command flag, 0 for bit 26 command flag
+ * @count: Number of burst pulses to send
+ * @slice: Slice selector (bit 1 mapped to bit 6)
+ */
+void pmu_cal_pulse_burst_send(u32 burst_type, u32 count, u32 slice)
+{
+ u32 zero = 0U;
+ u32 base;
+ u32 cmd;
+ u32 i;
+
+ if (count == 0U)
+ return;
+
+ base = (slice << 5) & 64U;
+ cmd = base | ((burst_type != 0U) ? (1U << 27) : (1U << 26));
+
+ for (i = 0U; i < count; i++)
+ pmu_cal_sequence_pulse_send(cmd, 7U, 0U, 0U,
+ zero, zero, zero);
+}
+
+/**
+ * pmu_multiphase_pulse_seq_repeat() - Repeat multiphase pulse sequence
+ * @phase_cmd: Pulse opcode
+ * @param_val: Parameter code
+ * @slice: Slice mask
+ * @count: Repeat count
+ */
+void pmu_multiphase_pulse_seq_repeat(u32 phase_cmd, u32 param_val, u32 slice,
+ u32 count)
+{
+ u32 zero = 0U;
+ __volatile__ u32 saved_param = param_val;
+ __volatile__ u32 saved_slice = slice;
+
+ while (count-- != 0U)
+ pmu_cal_sequence_pulse_send(0U, phase_cmd, 0U, saved_param,
+ zero, saved_slice, zero);
+}
+
+/**
+ * pmu_phy_deskew_reset_strobe() - Issue PHY deskew reset strobe across active channels
+ */
+static void pmu_phy_deskew_reset_strobe(void)
+{
+ u32 zero = 0U;
+ u8 ch1_en_mask;
+ u8 ch0_en_mask;
+ u32 mask;
+
+ ch1_en_mask = *(__volatile__ const u8 *)PMU_DMEM_CH1_ACTIVE_FLAG;
+ ch0_en_mask = *(__volatile__ const u8 *)PMU_DMEM_CH0_ANALOG_EN;
+ pmu_cal_strobe_pulse_init();
+ mask = (ch1_en_mask | ch0_en_mask) & 3U;
+ pmu_cal_sequence_pulse_send(0x80U, 0x0bU, 0x20U, 0U,
+ zero, mask, zero);
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+ pmu_hw_timer_delay(0x14U);
+}
+
+/**
+ * pmu_cal_multi_rank_timing_step() - Step multi-rank timing parameters across 21-entry table
+ * @rank_mode: Active rank mode mask
+ * @steps_table: Pointer to per-step timing table
+ * @eval_error_mask: Non-zero to evaluate lane error mask at completion
+ * @skip_mask_low: Skip bitmask for indices 0..31
+ * @skip_mask_high: Skip bitmask for indices 32..63
+ * @check_settle_flags: Non-zero to check settle flags on step 0
+ */
+noinline void pmu_cal_multi_rank_timing_step(u32 rank_mode, u32 steps_table,
+ u32 eval_error_mask,
+ u32 skip_mask_low,
+ u32 skip_mask_high,
+ u32 check_settle_flags)
+{
+ u32 zero = 0U;
+ const u8 *tbl = (const u8 *)(uintptr_t)steps_table;
+ u32 rank_bit = 1U << (1U & ~rank_mode);
+ u32 i;
+
+ pmu_cal_strobe_pulse_init();
+
+ for (i = 0U; i < 21U; i++) {
+ u8 cal_idx = pmu_sdata()->cal_seq_map[i];
+ u32 bit = 1U << (cal_idx & 31U);
+ u8 step_val;
+ u16 mult = 0U;
+
+ if ((cal_idx < 32U ? (bit & skip_mask_low) :
+ (bit & skip_mask_high)) != 0U)
+ continue;
+
+ step_val = tbl[cal_idx];
+
+ if (i == 4U)
+ pmu_cal_sequence_pulse_send(zero, 0x10U, 4U, zero,
+ zero, rank_mode, zero);
+
+ pmu_cal_sequence_pulse_send(zero, 6U, zero, step_val,
+ cal_idx, rank_mode, zero);
+ pmu_cal_pad_timing_delay_set(zero, 0x22U, zero);
+
+ if (i == 0U) {
+ if (check_settle_flags != 0U &&
+ ((tbl[1] & 0x08U) != 0U ||
+ (tbl[0x14] & 0x0fU) != 0U)) {
+ pmu_cal_slice_step_diff_latch(zero, 1U);
+ pmu_cal_strobe_pulse_init();
+ pmu_cal_dbyte_bcast_pulse_width_clear(tbl);
+ continue;
+ }
+ mult = 200U;
+ } else {
+ if (i == 15U &&
+ ((dmem_read8(PMU_DMEM_CH0_ANALOG_EN) |
+ dmem_read8(PMU_DMEM_CH1_ACTIVE_FLAG)) &
+ rank_bit) != 0U) {
+ pmu_cal_sequence_pulse_send(zero, 6U, zero,
+ step_val | 0x80U,
+ cal_idx,
+ rank_mode,
+ zero);
+ pmu_cal_pad_timing_delay_set(zero, 0x22U, zero);
+ }
+ if ((i | 1U) == 0x11U)
+ mult = 250U;
+ }
+
+ if (mult != 0U) {
+ u8 pll_ratio = dmem_read8(PMU_DMEM_PLL_RATIO);
+ u16 freq;
+ u32 result;
+
+ if (pll_ratio == 0U)
+ pll_ratio = 2U;
+ freq = dmem_read16(PMU_DMEM_DRAM_FREQ_MHZ);
+ result = ((u32)freq * mult) /
+ ((u32)pll_ratio * 2000U) + 1U;
+ pmu_cal_pad_timing_delay_set(zero, (u16)result, zero);
+ }
+ }
+
+ pmu_cal_pad_timing_delay_set(zero, 0x41U, zero);
+ pmu_cal_slice_step_diff_latch(zero, 1U);
+
+ if (eval_error_mask != 0U)
+ pmu_dbyte_lane_error_mask_calc(zero, rank_mode);
+}
+
+static __always_inline void
+pmu_cal_phase_delay_tracker_slice_update_inline(u32 rank_idx,
+ u32 active_csr_offset,
+ u32 csr_step_off,
+ u32 slice_param_off)
+{
+ u32 rank_shift = (rank_idx & 0xffU) << 12;
+ u32 slice_reg_base = (active_csr_offset | rank_shift) << 1;
+ u32 rank_csr_off = (rank_idx & 0xffU) << 13;
+ __volatile__ u16 *s_base =
+ (__volatile__ u16 *)(PHY_REG_DBYTE_BASE + slice_reg_base);
+ __volatile__ u16 *r_base =
+ (__volatile__ u16 *)(PHY_REG_DBYTE_BASE + rank_csr_off);
+ u16 snap[5];
+ u16 eye_strobe_val;
+ u16 period_meas_val;
+ u32 delay_threshold;
+ u32 phase_idx = 5U;
+ u16 dram_freq;
+ u32 p;
+ __volatile__ u16 *target_csr_p;
+ u16 saved_ctrl;
+ u16 current_csr_val;
+ u32 delay_cnt;
+
+ for (p = 0U; p < 5U; p++)
+ snap[p] = s_base[0xd0U + p];
+
+ eye_strobe_val = r_base[0xacU];
+ r_base[0xacU] = eye_strobe_val | 1U;
+ r_base[0xacU] = eye_strobe_val & ~1U;
+
+ period_meas_val = r_base[0xd5U];
+
+ delay_threshold = ((u32)period_meas_val * 14U) / 100U;
+ dram_freq = *(__volatile__ const u16 *)PMU_DMEM_DRAM_FREQ_MHZ;
+
+ for (p = 0U; p < 5U; p++) {
+ u16 curr = snap[p];
+
+ if (dram_freq < 3200U) {
+ if ((u32)period_meas_val < ((u32)curr * 2U)) {
+ phase_idx = p;
+ break;
+ }
+ } else {
+ u32 idx = (p == 0U) ? 1U : p;
+ s32 diff = (s32)snap[idx] - (s32)snap[idx - 1U] -
+ (s32)period_meas_val;
+
+ if (((u32)curr * 2U) >= (u32)period_meas_val &&
+ diff <= (s32)delay_threshold &&
+ diff >= -(s32)delay_threshold) {
+ phase_idx = p;
+ break;
+ }
+ }
+ }
+ pmu_assert_or_halt(phase_idx != 5U,
+ (dram_freq < 3200U) ? PMU_ERR_ZQ_CAL_LOW_FREQ :
+ PMU_ERR_ZQ_CAL_HIGH_FREQ);
+
+ target_csr_p = (__volatile__ u16 *)
+ (PHY_REG_DBYTE_BASE + ((rank_shift | slice_param_off) << 1));
+ s_base[0xadU] = (u8)phase_idx;
+
+ saved_ctrl = *target_csr_p;
+ *target_csr_p = saved_ctrl | (u16)csr_step_off;
+
+ current_csr_val = s_base[0x0fU];
+ s_base[0x0fU] = (current_csr_val & ~3U) | 1U;
+
+ delay_cnt = ((s_base[0x14U] << 1) & 510U) * 160U;
+ pmu_hw_timer_delay((delay_cnt << 2) + 0x28U);
+
+ *target_csr_p = saved_ctrl & ~(u16)csr_step_off;
+ s_base[0x0fU] = current_csr_val & ~1U;
+
+ eye_strobe_val = r_base[0xacU];
+ r_base[0xacU] = eye_strobe_val | 1U;
+ r_base[0xacU] = eye_strobe_val & ~1U;
+
+ s_base[0xafU] = r_base[0xd6U];
+}
+
+/**
+ * pmu_cal_phase_delay_tracker_update() - Backup RFIFO CSRs and update WCK phase delay tracker
+ */
+static void pmu_cal_phase_delay_tracker_update(void)
+{
+ u8 flag = *(__volatile__ const u8 *)PMU_DMEM_CLK_GATE_FLAG;
+ u16 wck_ctrl;
+ u8 freq_mode;
+ u32 csr_offset;
+ u32 wck_mode_param;
+ u32 wck_reg_param;
+ u32 slice_off;
+ u32 bcast_wck_addr;
+ u32 bcast_gate_addr;
+ u32 dbyte_gate_addr;
+ u16 dbyte_gate_val;
+ u32 slice_csr_b;
+ u32 slice_csr_a;
+ u16 master_dly_val;
+ u16 eye_ctrl_val;
+ u16 hi_margin_val;
+ __volatile__ u8 *slice_bounds;
+ u8 rank0_start;
+ u8 rank0_end;
+ u8 rank1_start;
+ u8 rank1_end;
+ u32 pass;
+ u32 rank_idx;
+ u16 dly_cnt_val;
+ u32 delay_count;
+
+ if ((flag & 0x20U) != 0U) {
+ __volatile__ const u16 *src =
+ (__volatile__ const u16 *)PHY_REG_RFIFO_CFG;
+ __volatile__ u16 *dst =
+ (__volatile__ u16 *)PMU_DMEM_RFIFO_BAK;
+ u32 slice_offset = pmu_sdata()->active_csr_offset;
+ __volatile__ const u16 *src_s = (__volatile__ const u16 *)
+ (PHY_REG_ZQ_STAT + (slice_offset << 1));
+
+ *dst++ = *src++;
+ *dst++ = *src++;
+ *dst++ = *src++;
+ *dst++ = src[1];
+ dst[0] = src_s[0];
+ dst[1] = src_s[1];
+ dst[2] = src_s[2];
+ }
+
+ phy_write16(PHY_REG_MASTER_CAL_CTRL, 1U);
+ phy_write16(PHY_REG_AC_PAT_CTRL, 0U);
+ phy_write16(PHY_REG_BIST_PLL_CFG, 0U);
+
+ wck_ctrl = ((flag & 0x80U) != 0U) ? 0U : (flag & 1U);
+ *(__volatile__ u16 *)PHY_REG_RFIFO_SUPPRESS = wck_ctrl;
+ *(__volatile__ u16 *)PHY_REG_CH0_RANK_CFG =
+ *(__volatile__ const u8 *)PMU_DMEM_CH0_ANALOG_EN;
+ *(__volatile__ u16 *)PHY_REG_CH1_RANK_CFG =
+ *(__volatile__ const u8 *)PMU_DMEM_CH1_ACTIVE_FLAG;
+
+ if (*(__volatile__ const u8 *)PMU_DMEM_RX_CLK_EN == 0U) {
+ u32 active_slices;
+
+ pmu_cal_dbyte_delay_backup_all_ch();
+ active_slices = pmu_sdata()->active_slice_cnt;
+ if (active_slices != 0U) {
+ u32 i;
+
+ for (i = 0U; i < active_slices; i++) {
+ __volatile__ u16 *reg = (__volatile__ u16 *)
+ (PHY_REG_DBYTE_WCK_DLY +
+ i * PHY_DBYTE_SLICE_STRIDE);
+
+ *reg &= 0x0cU;
+ }
+ }
+ }
+
+ pmu_phy_clk_phase_toggle_pulse();
+ pmu_pll_clk_gate_cfg();
+
+ if (*(__volatile__ const u16 *)PMU_DMEM_DRAM_FREQ_MHZ < 1600U)
+ goto out;
+
+ if (pmu_is_lpddr5() != 0U)
+ goto out;
+
+ freq_mode = pmu_sdata()->freq_mode;
+ csr_offset = pmu_sdata()->active_csr_offset;
+
+ if (freq_mode < 2U) {
+ wck_mode_param = 2U;
+ wck_reg_param = csr_offset | 0x14U;
+ } else if (freq_mode < 4U) {
+ wck_mode_param = 2U;
+ wck_reg_param = 0x56U;
+ } else {
+ wck_mode_param = 1U;
+ wck_reg_param = csr_offset | 87U;
+ }
+
+ slice_off = csr_offset << 1;
+ bcast_wck_addr = PHY_REG_DBYTE_BCAST_WCK;
+ *(__volatile__ u16 *)bcast_wck_addr = 0x1000U;
+
+ bcast_gate_addr = PHY_REG_DBYTE_BCAST_XCVR_GATE | slice_off;
+ dbyte_gate_addr = PHY_REG_DBYTE_XCVR_GATE | slice_off;
+ dbyte_gate_val = *(__volatile__ const u16 *)dbyte_gate_addr;
+
+ slice_csr_b = PHY_REG_DBYTE_BASE | (wck_reg_param << 1);
+ *(__volatile__ u16 *)bcast_gate_addr = (dbyte_gate_val & ~7U) | 6U;
+
+ slice_csr_a = PHY_REG_DBYTE_BCAST_BASE | (wck_reg_param << 1);
+ master_dly_val = *(__volatile__ const u16 *)slice_csr_b;
+ master_dly_val &= (u16)~wck_mode_param;
+ *(__volatile__ u16 *)slice_csr_a = master_dly_val;
+
+ eye_ctrl_val = *(__volatile__ const u16 *)PHY_REG_DBYTE_EYE_CTRL;
+ *(__volatile__ u16 *)PHY_REG_DBYTE_BCAST_EYE = eye_ctrl_val & ~1U;
+
+ hi_margin_val = *(__volatile__ const u16 *)dbyte_gate_addr;
+ *(__volatile__ u16 *)bcast_gate_addr = hi_margin_val | 5U;
+ pmu_hw_timer_delay(0x3000U);
+
+ *(__volatile__ u16 *)bcast_gate_addr = (hi_margin_val & ~5U) | 4U;
+
+ slice_bounds = (__volatile__ u8 *)PMU_DMEM_CH_RANGE;
+ rank0_start = slice_bounds[2];
+ rank0_end = slice_bounds[3];
+ rank1_start = slice_bounds[4];
+ rank1_end = slice_bounds[5];
+
+ slice_bounds[0] = rank0_start;
+ slice_bounds[1] = rank0_end;
+
+ for (pass = 0U; pass < 2U; pass++) {
+ u32 r_start = (pass == 0U) ? 0U : (u32)rank1_start;
+ u32 r_end = (pass == 0U) ? (u32)rank0_end : (u32)rank1_end;
+
+ if (pass == 0U && rank0_end == 255U)
+ continue;
+ for (rank_idx = r_start; rank_idx <= r_end; rank_idx++)
+ pmu_cal_phase_delay_tracker_slice_update_inline(rank_idx,
+ csr_offset,
+ wck_mode_param,
+ wck_reg_param);
+ }
+
+ master_dly_val = *(__volatile__ const u16 *)slice_csr_b;
+ *(__volatile__ u16 *)slice_csr_a = master_dly_val | (u16)wck_mode_param;
+
+ dbyte_gate_val = *(__volatile__ const u16 *)dbyte_gate_addr;
+ *(__volatile__ u16 *)bcast_gate_addr = (dbyte_gate_val & ~3U) | 1U;
+
+ dly_cnt_val = *(__volatile__ const u16 *)(PHY_REG_DBYTE_DELAY_CNT |
+ slice_off);
+ delay_count = ((dly_cnt_val << 1) & 510U) * 160U;
+ pmu_hw_timer_delay((delay_count << 2) + 0x28U);
+
+ dbyte_gate_val &= ~1U;
+ master_dly_val &= (u16)~wck_mode_param;
+ *(__volatile__ u16 *)slice_csr_a = master_dly_val;
+ *(__volatile__ u16 *)bcast_gate_addr = dbyte_gate_val;
+
+out:
+ pmu_pll_lock_freq_tier_cfg();
+}
+
+/**
+ * pmu_rank_mask_multiparam_dispatch() - Update shadow fields and dispatch multi-rank timing step
+ * @channel: Target channel index (0 or 1)
+ * @rank: Target rank index (0 or 1)
+ * @shadow_update: Non-zero to save and mask shadow fields, 0 to restore
+ */
+void pmu_rank_mask_multiparam_dispatch(u32 channel, u32 rank, u32 shadow_update)
+{
+ u8 *base = (u8 *)PMU_DMEM_CAL_STRUCT_ADDR;
+
+ if (shadow_update != 0U) {
+ pmu_cal_eye_table_save(base, 0x12U);
+ pmu_cal_lane_mask_or_all((u32)(uintptr_t)base, 0x12U, 0x80U);
+ } else {
+ pmu_cal_eye_table_restore(base, 0x12U);
+ }
+
+ pmu_cal_multi_rank_timing_step((u8)(5U << channel),
+ (u32)(uintptr_t)base +
+ rank * (2U * PMU_SLICE_DESC_SIZE) +
+ channel * PMU_SLICE_DESC_SIZE,
+ 0U, 0xfffbffffUL, 0xffffffffUL, 0U);
+ pmu_delay_us(20000U, 0U);
+}
+
+/**
+ * pmu_train_run_dev_init() - Execute device initialization and LPCA pre-init stage
+ * @dmem0: Base pointer to PMU DMEM byte array
+ * @dmem_words: Base pointer to PMU DMEM halfword array
+ * @seq_ctrl: Active training sequence control bitmask
+ * @clk_gate_flag: Clock gating configuration byte from DMEM
+ */
+static void pmu_train_run_dev_init(__volatile__ u8 *dmem0,
+ __volatile__ u16 *dmem_words,
+ u16 seq_ctrl, u8 clk_gate_flag)
+{
+ u32 clk_sync_flag = (clk_gate_flag >> 7) & 1U;
+ u32 clamp;
+ u32 clamp_limit;
+
+ pmu_phy_pll_lock_init();
+ if (!(dmem0[PMU_DMEM_CAL_DISABLE] | clk_sync_flag))
+ phy_write16(PHY_REG_PHY_STATUS, 2U);
+
+ pmu_cal_zq_pad_init();
+ pmu_phy_pll_clock_divider_align();
+ pmu_cal_phase_delay_tracker_update();
+ pmu_cal_strobe_pulse_init();
+ pmu_cal_pad_timing_delay_set(0U, 4U, 0U);
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+ pmu_phy_dbyte_broadcast_enable();
+ pmu_cal_lane_window_margin_eval(clk_sync_flag);
+ dmem_words[PMU_DMEM_TRAIN_MODE >> 1] = 1U;
+ pmu_cal_rx_dqs_delay_center();
+ if (clk_sync_flag == 0U)
+ pmu_cal_dbyte_impedance_fine_tune();
+
+ if (seq_ctrl & PMU_SEQ_LPCA_INIT) {
+ pmu_cal_rx_en_pre_init();
+ dmem_words[PMU_DMEM_TRAIN_MODE >> 1] = 0x1000U;
+ dmem_words[PMU_DMEM_TRAIN_MODE >> 1] = 0x9000U;
+ pmu_post_cmd_conditional_dispatch(0x0dU);
+ }
+ if (clk_sync_flag == 0U)
+ pmu_multiparam_cal_dispatch();
+
+ pmu_cal_strobe_pulse_init();
+ clamp = pmu_cal_delay_line_accum_dispatch(0x3a98U, 0U);
+ clamp_limit = ((u8)clamp > 8U) ? (u8)clamp : 8U;
+ pmu_cal_sequence_pulse_send(0U, 7U, clamp_limit, 0U, 0U, 0U, 0U);
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+ pmu_post_cmd_conditional_dispatch(0U);
+ pmu_sdata()->cal_active_flag = 0U;
+ dmem_words[PMU_DMEM_TRAIN_MODE >> 1] = 0x8001U;
+}
+
+/**
+ * pmu_train_run_lane_win_sweep() - Execute lane window sweep and LCDL trace dump
+ * @dmem0: Base pointer to PMU DMEM byte array
+ * @dmem_words: Base pointer to PMU DMEM halfword array
+ */
+static void pmu_train_run_lane_win_sweep(__volatile__ u8 *dmem0,
+ __volatile__ u16 *dmem_words)
+{
+ u8 clk_gate;
+ u8 lane_sweep_ctrl;
+ u32 active_cnt = 0U;
+ u32 wide_range;
+
+ clk_gate = dmem0[PMU_DMEM_CLK_GATE_FLAG & 0xffffU];
+ lane_sweep_ctrl = dmem0[PMU_DMEM_LANE_SWEEP_CTRL];
+ if (lane_sweep_ctrl >> 6)
+ active_cnt = (1U << (clk_gate & 3U)) - 1U;
+
+ wide_range = (lane_sweep_ctrl >> 6) ? 1U : 0U;
+ dmem_words[PMU_DMEM_TRAIN_MODE >> 1] = 1024U;
+ pmu_cal_lane_window_sweep_coordinator(active_cnt, wide_range);
+ dmem_words[PMU_DMEM_TRAIN_MODE >> 1] = 0x8400U;
+ pmu_post_cmd_conditional_dispatch(0x0eU);
+}
+
+/**
+ * pmu_train_run_bist_and_dqs_deskew() - Run BIST search window and DQS deskew sweeps
+ * @dmem0: Base pointer to PMU DMEM byte array
+ * @seq_ctrl: Active training sequence control bitmask
+ */
+static void pmu_train_run_bist_and_dqs_deskew(__volatile__ u8 *dmem0,
+ u16 seq_ctrl)
+{
+ if (seq_ctrl & PMU_SEQ_BIST_SEARCH_WIN) {
+ u32 train_step_lim =
+ (pmu_dmem_training_flags_eval() == 0U) ? 0xffU : 511U;
+ u32 train_lane_mask =
+ (dmem0[PMU_DMEM_CLK_GATE_FLAG & 0xffffU] & 4U) ? 0U : 0x0fU;
+ u16 saved_cal_cfg = pmu_sdata()->cal_save_cfg;
+
+ if (saved_cal_cfg & (1U << 5)) {
+ pmu_cal_multi_lane_deskew_sweep(2U, train_lane_mask,
+ train_step_lim);
+ if (dmem0[PMU_DMEM_TRAIN_PARAM_CTRL] & (1U << 4))
+ pmu_cal_multi_lane_deskew_sweep(3U,
+ train_lane_mask,
+ train_step_lim);
+ }
+ }
+
+ if (seq_ctrl & PMU_SEQ_STAGE_DQS) {
+ u32 train_loop_lim;
+ u32 train_step_mask;
+ u16 saved_cal_cfg;
+
+ pmu_cal_zq_pad_calibration(0U);
+ train_loop_lim =
+ (pmu_dmem_training_flags_eval() == 0U) ? 0xffU : 511U;
+ train_step_mask =
+ (dmem0[PMU_DMEM_CLK_GATE_FLAG & 0xffffU] & 4U) ? 0U : 0x0fU;
+ saved_cal_cfg = pmu_sdata()->cal_save_cfg;
+ if (saved_cal_cfg & (1U << 6)) {
+ pmu_cal_multi_lane_deskew_sweep(1U, train_step_mask,
+ train_loop_lim);
+ pmu_cal_multi_lane_deskew_sweep(0U, train_step_mask,
+ train_loop_lim);
+ }
+ }
+}
+
+/**
+ * pmu_sdata_init() - Initialize PMU .sdata scalars and stage sequence map in DCCM
+ */
+static void pmu_sdata_init(void)
+{
+ dmem_write8(PMU_DMEM_METRIC_TABLE_ENTRIES, 5U);
+ dmem_write8(PMU_DMEM_2D_STEP_PARAM, 4U);
+ dmem_write8(PMU_DMEM_CBT_STATE, 4U);
+ dmem_write8(PMU_DMEM_CBT_CONFIG, 4U);
+
+ dmem_write32(PMU_DMEM_CAL_CMD_TABLE, 0x180f0e0cUL);
+ dmem_write8(PMU_DMEM_CAL_CMD_TABLE + 4U, 0x1eU);
+
+ dmem_write32(PMU_DMEM_CAL_SEQ_MAP + 0x00U, 0x02011210UL);
+ dmem_write32(PMU_DMEM_CAL_SEQ_MAP + 0x04U, 0x1c0a1403UL);
+ dmem_write32(PMU_DMEM_CAL_SEQ_MAP + 0x08U, 0x0b112e29UL);
+ dmem_write32(PMU_DMEM_CAL_SEQ_MAP + 0x0cU, 0x0c19160dUL);
+ dmem_write32(PMU_DMEM_CAL_SEQ_MAP + 0x10U, 0x1e180f0eUL);
+ dmem_write8(PMU_DMEM_CAL_SEQ_MAP + 0x14U, 0x13U);
+}
+
+/**
+ * pmu_train_dispatcher() - Dispatch active training stages per sequence_ctrl
+ */
+static void pmu_train_dispatcher(void)
+{
+ __volatile__ u8 *dmem0 = (__volatile__ u8 *)PMU_DMEM_BASE;
+ __volatile__ u16 *dmem_words;
+ __volatile__ u16 *pub_ctl = (__volatile__ u16 *)PHY_REG_PUB_CTL;
+ u32 scratch_csr;
+ u8 pstate_cfg;
+ u32 ch_offset = 0U;
+ u16 seq_ctrl;
+ u32 ch_bcast_csr;
+ u16 master_phy_status;
+ u8 clk_gate_flag;
+ u32 pll_ratio;
+ u32 denom;
+ u32 window;
+ u8 train_feature_mask;
+ u32 flag_profile3 = 0U;
+
+ __asm__ __volatile__("flag %0" : : "r"(0x6000));
+ dmem_words = (__volatile__ u16 *)dmem0;
+ pmu_sdata_init();
+
+ phy_write16(PHY_REG_MASTER_INIT_CTRL, 1U);
+
+ if ((dmem0[PMU_DMEM_CLK_GATE_FLAG & 0xffffU] & 1U) != 0U) {
+ phy_write16(PHY_REG_PLL_STATUS, 0U);
+ phy_write16(PHY_REG_DBYTE_BCAST_PWR_GATE, 0U);
+ phy_write16(PHY_REG_MASTER_PWR_GATE, 0U);
+ phy_write16(PHY_REG_MASTER_COARSE_CFG, 0U);
+ phy_write16(PHY_REG_MASTER_PWR_GATE, 3U);
+ }
+
+ pmu_phy_clk_gate_sync_pulse();
+ pmu_cbt_step_stat_set();
+
+ pmu_sdata()->fw_version_id = PMU_FW_VERSION_ID;
+ pmu_cmd_code_map_log(*pub_ctl, 0U);
+
+ scratch_csr = PHY_REG_CSR_SCRATCH_BASE +
+ ((pmu_sdata()->freq_mode != 0U) ? 10U : 0U);
+ phy_write16(scratch_csr, PHY_CSR_SCRATCH_MAGIC_LO);
+ phy_write16(scratch_csr + 2U, PHY_CSR_SCRATCH_MAGIC_HI);
+ if (phy_read16(scratch_csr) != PHY_CSR_SCRATCH_MAGIC_LO ||
+ phy_read16(scratch_csr + 2U) != PHY_CSR_SCRATCH_MAGIC_HI)
+ pmu_assert_or_halt(0U, PMU_ERR_CSR_BUS_ACCESS_FAULT);
+
+ pstate_cfg = dmem_read8(PMU_DMEM_PSTATE_FLAGS);
+ if (!(pstate_cfg & 0x80U))
+ ch_offset = ((u32)pstate_cfg << 20) & 0x100000UL;
+ pmu_sdata()->active_csr_offset = ch_offset;
+
+ seq_ctrl = dmem_words[PMU_DMEM_SEQUENCE_CTRL >> 1];
+ ch_bcast_csr = PHY_REG_DBYTE_BCAST_MODE | (ch_offset << 1);
+ master_phy_status = phy_read16((PHY_REG_MASTER_BASE | 0x0012UL) |
+ (ch_offset << 1));
+ clk_gate_flag = dmem0[PMU_DMEM_CLK_GATE_FLAG & 0xffffU];
+
+ pll_ratio = dmem_read8(PMU_DMEM_PLL_RATIO);
+ if (pll_ratio == 0U)
+ pll_ratio = 2U;
+ denom = pll_ratio * 220U;
+ window = (dmem_words[PMU_DMEM_DRAM_FREQ_OFF >> 1] + denom - 1U) / denom;
+
+ dmem0[PMU_DMEM_AC_STEP_FLAG] = 1U;
+ dmem0[PMU_DMEM_SEARCH_WINDOW_SCALE] = 4U;
+ dmem0[PMU_DMEM_SEARCH_WINDOW_STEPS] = (u8)window;
+
+ pmu_cal_search_win_init((u32)(uintptr_t)(dmem0 + PMU_DMEM_SEARCH_WIN_COARSE));
+ pmu_cal_search_win_init((u32)(uintptr_t)(dmem0 + PMU_DMEM_SEARCH_WIN_FINE));
+
+ if (seq_ctrl & PMU_SEQ_DEV_INIT)
+ pmu_train_run_dev_init(dmem0, dmem_words, seq_ctrl,
+ clk_gate_flag);
+
+ pmu_cal_pad_impedance_lock();
+
+ if (seq_ctrl & PMU_SEQ_STAGE_VREF)
+ pmu_cal_ca_vref_step(1U);
+ if (seq_ctrl & PMU_SEQ_STAGE_DCD)
+ pmu_cal_ca_train_step();
+
+ train_feature_mask = (u8)dmem_words[PMU_DMEM_TRAIN_FEATURE_MASK >> 1];
+ if (pmu_phy_query_operating_mode() == 3U && (seq_ctrl & PMU_SEQ_ZQ_CAL) &&
+ !(train_feature_mask & 2U)) {
+ phy_write16(ch_bcast_csr, 0U);
+ flag_profile3 = 1U;
+ }
+
+ if (seq_ctrl & PMU_SEQ_STAGE_DQS)
+ pmu_cal_zq_pad_calibration(1U);
+
+ if (seq_ctrl & PMU_SEQ_SEARCH_WIN) {
+ pmu_cal_search_win_init((u32)(uintptr_t)(dmem0 + PMU_DMEM_SEARCH_WIN_FINE));
+ pmu_sdata()->cal_active_flag = 1U;
+ dmem_words[PMU_DMEM_TRAIN_MODE >> 1] = 0x0040U;
+ pmu_cal_vref_dac_step_adjust(0x13U);
+ dmem_words[PMU_DMEM_TRAIN_MODE >> 1] = 0x8040U;
+ pmu_post_cmd_conditional_dispatch(6U);
+ pmu_sdata()->cal_active_flag = 0U;
+ }
+
+ train_feature_mask = (u8)dmem_words[PMU_DMEM_TRAIN_FEATURE_MASK >> 1];
+ if (!(train_feature_mask & 1U) && (seq_ctrl & PMU_SEQ_SEARCH_WIN)) {
+ if (!(train_feature_mask & 8U) && (seq_ctrl & PMU_SEQ_STAGE_VREF))
+ pmu_cal_ca_vref_step(1U);
+ if ((seq_ctrl & PMU_SEQ_STAGE_DCD) && !(train_feature_mask & 0x10U))
+ pmu_cal_ca_train_step();
+ if ((seq_ctrl & PMU_SEQ_STAGE_DQS) && !(train_feature_mask & 0x20U))
+ pmu_cal_zq_pad_calibration(1U);
+ }
+
+ if (seq_ctrl & PMU_SEQ_ZQ_CAL) {
+ if (dmem_read8(PMU_DMEM_MR20_ADDR) & 3U) {
+ dmem_words[PMU_DMEM_TRAIN_MODE >> 1] = 0x0080U;
+ pmu_cal_vref_dac_step_adjust(0x11U);
+ dmem_words[PMU_DMEM_TRAIN_MODE >> 1] = 0x8080U;
+ pmu_post_cmd_conditional_dispatch(10U);
+ }
+
+ if (!(seq_ctrl & PMU_SEQ_DESKEW_SWEEP))
+ pmu_sdata()->iter_count = 0U;
+ }
+
+ train_feature_mask = (u8)dmem_words[PMU_DMEM_TRAIN_FEATURE_MASK >> 1];
+ if (!(train_feature_mask & 2U) && (seq_ctrl & PMU_SEQ_ZQ_CAL)) {
+ if (flag_profile3)
+ phy_write16(ch_bcast_csr, 4U);
+ if (seq_ctrl & PMU_SEQ_STAGE_DQS)
+ pmu_cal_zq_pad_calibration(1U);
+ }
+
+ if (seq_ctrl & PMU_SEQ_LANE_WIN_SWEEP)
+ pmu_train_run_lane_win_sweep(dmem0, dmem_words);
+
+ if (seq_ctrl != PMU_SEQ_DEV_INIT)
+ pmu_cal_bist_search_win_setup(!(seq_ctrl & PMU_SEQ_BIST_SEARCH_WIN));
+
+ if (seq_ctrl & PMU_SEQ_DESKEW_SWEEP) {
+ dmem_words[PMU_DMEM_TRAIN_MODE >> 1] = 256U;
+ pmu_cal_vref_dac_step_adjust(0x12U);
+ dmem_words[PMU_DMEM_TRAIN_MODE >> 1] = 0x8100U;
+ pmu_post_cmd_conditional_dispatch(0x0bU);
+ pmu_sdata()->iter_count = 0U;
+
+ if (!(dmem_words[PMU_DMEM_TRAIN_FEATURE_MASK >> 1] & 4U) &&
+ seq_ctrl != PMU_SEQ_DEV_INIT)
+ pmu_cal_bist_search_win_setup(!(seq_ctrl & PMU_SEQ_BIST_SEARCH_WIN));
+ }
+
+ if (seq_ctrl == PMU_SEQ_DEV_INIT &&
+ dmem0[PMU_DMEM_CAL_DISABLE] == 0U)
+ pmu_cal_vref_dac_step_adjust(0x0fU);
+
+ pmu_train_run_bist_and_dqs_deskew(dmem0, seq_ctrl);
+
+ if (seq_ctrl & PMU_SEQ_MARGIN_STEP)
+ pmu_cal_timing_margin_eval();
+
+ pmu_cal_post_deskew_finalize();
+
+ if (seq_ctrl != PMU_SEQ_DEV_INIT) {
+ u8 pstate_val = dmem_read8(PMU_DMEM_PSTATE_FLAGS);
+
+ if (pstate_val & 0x80U)
+ pmu_cal_phase_detector_edge_align();
+ else if ((pstate_val & 0x30U) == 0x20U)
+ pmu_cal_dbyte_deskew_fine_tune(0U);
+ else if ((pstate_val & 0x30U) == 0x10U)
+ pmu_cal_dbyte_deskew_fine_tune(1U);
+ }
+
+ pmu_phy_dbyte_cfg_unpack();
+ phy_write16((PHY_REG_PUB_SEC_BASE | 0x0012UL) | (ch_offset << 1),
+ master_phy_status);
+ pmu_mailbox_post_cmd_dispatch(PMU_STATUS_SUCCESS);
+ __asm__ __volatile__("mov_s r0, 0\n\tsr r0, [0x21]\n\tbrk_s\n1: b_s 1b\n");
+ __builtin_unreachable();
+}
+
+/**
+ * pmu_cal_multi_rank_step_commit() - Commit multi-rank calibration steps with AC profile setup
+ * @rank_mode: Active rank selector bitmask
+ * @steps_table: Base address of per-step calibration table
+ * @eval_error_mask: Non-zero to evaluate DBYTE lane error mask at completion
+ * @skip_mask_low: Skip bitmask for step indices 0..31
+ * @skip_mask_high: Skip bitmask for step indices 32..63
+ * @pulse_coarse_pre: Non-zero to send coarse pre-pulse before each step
+ * @check_settle_flags: Non-zero to check settle flags on step 0
+ */
+noinline void pmu_cal_multi_rank_step_commit(u32 rank_mode, u32 steps_table,
+ u32 eval_error_mask,
+ u32 skip_mask_low,
+ u32 skip_mask_high,
+ u32 pulse_coarse_pre,
+ u32 check_settle_flags)
+{
+ __volatile__ const u8 *dmem = (__volatile__ const u8 *)PMU_DMEM_BASE;
+ u32 rank_sel = rank_mode;
+ u32 rank_bit = 1U << (1U & ~rank_sel);
+ u32 i;
+
+ pmu_ac_lane_profile_setup();
+ pmu_cal_strobe_pulse_init();
+ pmu_cal_pad_timing_delay_set(0U, 5U, 1U);
+
+ for (i = 0U; i < 21U; i++) {
+ u8 cal_idx = pmu_sdata()->cal_seq_map[i];
+ u32 bit = 1U << (cal_idx & 31U);
+ u32 skip = (cal_idx < 32U) ? skip_mask_low : skip_mask_high;
+ u8 step_val;
+
+ if ((bit & skip) != 0U)
+ continue;
+
+ step_val = ((const u8 *)(uintptr_t)steps_table)[cal_idx];
+
+ if (pulse_coarse_pre != 0U)
+ pmu_cal_pad_timing_delay_set(0U, 0x1eU, 1U);
+
+ if (i == 4U) {
+ pmu_cal_margin_dir_get();
+ pmu_cal_sequence_pulse_send(0U, 0x2cU, 0U, 0U, 0U,
+ rank_sel, 0U);
+ pmu_cal_sequence_pulse_send(0U, 0x2dU, 0U, 0U, 0U,
+ rank_sel, 0U);
+ pmu_cal_margin_dir_get();
+ }
+
+ pmu_dbyte_cal_step_latch(cal_idx, step_val, rank_sel);
+
+ if (i == 0U) {
+ if (check_settle_flags != 0U) {
+ const u8 *st = (const u8 *)(uintptr_t)steps_table;
+ u8 init_done_flag = st[1];
+ u8 post_cal_flag = st[0x14];
+
+ if ((init_done_flag & 0x08U) != 0U ||
+ (post_cal_flag & 0x0fU) != 0U) {
+ pmu_cbt_3phase_pulse_seq();
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+ pmu_cal_strobe_pulse_init();
+ pmu_cal_pad_timing_delay_set(0U, 5U, 1U);
+ pmu_cal_dbyte_bcast_pulse_width_clear(st);
+ }
+ }
+ pmu_cal_pad_timing_delay_set(0U, 0x18U, 1U);
+ } else if (i == 15U &&
+ (dmem[PMU_DMEM_CH_RANGE & 0xffffU] & rank_bit) != 0U) {
+ pmu_cal_margin_dir_get();
+ pmu_dbyte_cal_step_latch(cal_idx, step_val | 0x80U,
+ rank_sel);
+ }
+
+ pmu_cal_margin_dir_get();
+ }
+
+ pmu_cbt_3phase_pulse_seq();
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+
+ if (eval_error_mask != 0U)
+ pmu_dbyte_lane_error_mask_calc(1U, rank_sel);
+
+ pmu_phy_profile_param_program();
+}
+
+static void pmu_cal_multi_rank_step_dispatch(u32 rank_mode, u32 steps_table,
+ u32 eval_error_mask,
+ u32 skip_mask_low,
+ u32 skip_mask_high,
+ u32 check_settle_flags)
+{
+ if (dmem_read8(PMU_DMEM_AC_CAL_ACTIVE) != 0U &&
+ (dmem_read8(PMU_DMEM_CLK_GATE_FLAG) & (1U << 3)) == 0U)
+ pmu_cal_multi_rank_step_commit(rank_mode, steps_table,
+ eval_error_mask, skip_mask_low,
+ skip_mask_high, 0U,
+ check_settle_flags);
+ else
+ pmu_cal_multi_rank_timing_step(rank_mode, steps_table,
+ eval_error_mask, skip_mask_low,
+ skip_mask_high,
+ check_settle_flags);
+}
+
+/**
+ * pmu_multirank_slice_cal_dispatch() - Dispatch calibration steps across enabled channels and ranks
+ * @table_ptr: Base pointer to 4-slice calibration table
+ * @eval_error_mask: Non-zero to evaluate lane error masks
+ * @skip_mask_low: Skip bitmask for step indices 0..31
+ * @skip_mask_high: Skip bitmask for step indices 32..63
+ * @ch_enable_mask: Active rank bitmask
+ * @check_settle_flags: Non-zero to check settle flags on step 0
+ */
+void pmu_multirank_slice_cal_dispatch(void *table_ptr, u32 eval_error_mask,
+ u32 skip_mask_low, s32 skip_mask_high,
+ u32 ch_enable_mask,
+ u32 check_settle_flags)
+{
+ u32 rank;
+ u32 freq_pt;
+
+ for (rank = 0U; rank < 2U; rank++) {
+ u32 bit = 1U << rank;
+ u32 rank_mask = 5U << rank;
+
+ for (freq_pt = 0U; freq_pt < 2U; freq_pt++) {
+ u32 mask = (freq_pt == 0U) ?
+ dmem_read8(PMU_DMEM_CH0_ANALOG_EN) :
+ dmem_read8(PMU_DMEM_CH1_ACTIVE_FLAG);
+
+ if (((ch_enable_mask & bit) & mask) != 0U) {
+ u32 addr = (u32)(uintptr_t)table_ptr +
+ freq_pt * (2U * PMU_SLICE_DESC_SIZE) +
+ rank * PMU_SLICE_DESC_SIZE;
+
+ pmu_phy_mode_cfg_dispatch(1U << freq_pt);
+ pmu_cal_multi_rank_step_dispatch((u8)rank_mask,
+ addr,
+ eval_error_mask,
+ skip_mask_low,
+ (u32)skip_mask_high,
+ check_settle_flags);
+ }
+ }
+ }
+
+ pmu_phy_mode_cfg_dispatch(3U);
+}
+
+/**
+ * pmu_cal_multi_param_step_commit() - Commit multi-parameter step for active channel and rank
+ * @param_table: Base address of per-step calibration table
+ * @step_val: Non-zero to evaluate lane error mask
+ * @eval_mask: Skip bitmask for step indices 0..31
+ * @filter_mask: Skip bitmask for step indices 32..63
+ * @slice_flags: Non-zero to check settle flags on step 0
+ */
+static void pmu_cal_multi_param_step_commit(u32 param_table, u32 step_val,
+ u32 eval_mask, u32 filter_mask,
+ u32 slice_flags)
+{
+ pmu_phy_mode_cfg_dispatch(3U);
+ pmu_cal_multi_rank_step_dispatch(0x0fU, param_table, step_val,
+ eval_mask, filter_mask, slice_flags);
+}
+
+/**
+ * pmu_cal_post_deskew_finalize() - Finalize Stage 39 calibration results and lane deskew
+ */
+static void pmu_cal_post_deskew_finalize(void)
+{
+ u8 *p_buf;
+ u8 cal_buf[PMU_SLICE_DESC_TOTAL_BYTES];
+ const u8 *cs;
+ u8 num_entries;
+ const u8 *cmd_table;
+ const u8 *dly;
+ u32 i;
+ u32 k;
+ u8 dram_type;
+ __volatile__ u16 *dq_dly;
+
+ p_buf = cal_buf;
+ pmu_cal_timing_table_matrix_init(p_buf);
+
+ cs = (const u8 *)PMU_DMEM_CAL_STRUCT_ADDR;
+ if (((cs[3] | cs[0x16]) & 0x40U) != ((p_buf[3] | p_buf[0x16]) & 0x40U))
+ pmu_phy_reg_offset_diff_adjust(0U);
+
+ pmu_cal_lane_mask_and_all((u32)(uintptr_t)p_buf, 0x13U, 0xf3U);
+ pmu_cal_lane_mask_or_all((u32)(uintptr_t)p_buf, 0x10U, 0x45U);
+
+ num_entries = dmem_read8(PMU_DMEM_METRIC_TABLE_ENTRIES);
+ cmd_table = (const u8 *)pmu_sdata()->cal_cmd_tbl;
+ dly = (const u8 *)PMU_DMEM_DESKEW_STAT_TBL;
+
+ for (i = 0U; i < num_entries; i++) {
+ u8 idx = cmd_table[i];
+
+ p_buf[idx] = *dly++;
+ p_buf[idx + PMU_SLICE_DESC_SIZE] = *dly++;
+ p_buf[idx + (2U * PMU_SLICE_DESC_SIZE)] = *dly++;
+ p_buf[idx + (3U * PMU_SLICE_DESC_SIZE)] = *dly++;
+ }
+
+ pmu_multirank_slice_cal_dispatch(p_buf, 1U, 0x800U, 0, 0xffU, 0U);
+
+ pmu_cal_lane_mask_and_all((u32)(uintptr_t)p_buf, 0x10U, 0xbcU);
+ if (!(p_buf[0x1c] & 0x20U))
+ pmu_cal_lane_mask_or_all((u32)(uintptr_t)p_buf, 0x1cU, 2U);
+
+ pmu_multirank_slice_cal_dispatch(p_buf, 1U, 0xfffeffffUL, -1,
+ 0xffU, 0U);
+
+ pmu_phy_deskew_reset_strobe();
+ pmu_cal_strobe_pulse_init();
+
+ if (!(p_buf[0x1cU] & 0x20U))
+ pmu_cal_sequence_pulse_send(0U, 6U, 0x22U,
+ p_buf[0x1cU] | 2U,
+ 28U, 15U, 0U);
+
+ for (k = 0U; k < 2U; k++)
+ pmu_cal_sequence_pulse_send(0U, k ? 7U : 0x28U,
+ k ? 0x1eU : 2U, 0U, 0U,
+ k ? 0U : 3U, 0U);
+ pmu_cal_slice_step_diff_latch(0U, 1U);
+
+ if (dmem_read8(PMU_DMEM_TRAIN_PARAM_CTRL) & 0x10U)
+ phy_write16(PHY_REG_DBYTE_BCAST_PULSE |
+ (pmu_sdata()->active_csr_offset << 1), 1U);
+
+ pmu_phy_pll_lock_init();
+ phy_write16((PHY_REG_MASTER_BASE | 0x00dcUL), 0U);
+
+ dram_type = dmem_read8(PMU_DMEM_CAL_DISABLE);
+ if (dram_type == 0U) {
+ u8 ch_idx;
+
+ for (ch_idx = 0U; ch_idx < 2U; ch_idx++) {
+ bool active = (ch_idx != 0U) ?
+ (dmem_read8(PMU_DMEM_CH1_ACTIVE_FLAG) != 0U) :
+ (dmem_read8(PMU_DMEM_CH0_ANALOG_EN) != 0U);
+
+ if (active) {
+ pmu_phy_mode_cfg_dispatch(ch_idx + 1U);
+ pmu_cal_lane_deskew_sweep_exec(ch_idx);
+ }
+ }
+ }
+
+ pmu_phy_mode_cfg_dispatch(3U);
+ dmem_write8(PMU_DMEM_CAL_BYTE, 0U);
+
+ if (dram_type == 0U)
+ pmu_phy_cal_strobe_latch_setup();
+
+ dq_dly = (__volatile__ u16 *)PHY_REG_DBYTE_BCAST_DQ_DLY2;
+ dq_dly[0] = 0U;
+ dq_dly[1] = 0U;
+
+ if (dmem_read8(PMU_DMEM_CAL_DISABLE) == 0U &&
+ (dmem_read16(PMU_DMEM_SEQ_CTRL) & 0x0400U) != 0U)
+ pmu_cal_vref_search_center_update();
+
+ pmu_cal_state_restore_offset_prog();
+}
+
+/**
+ * pmu_cal_vref_dac_step_adjust() - Dispatch per-stage Vref DAC and delay step adjustments
+ * @stage_step: Stage step selector (0..19)
+ */
+void pmu_cal_vref_dac_step_adjust(u32 stage_step)
+{
+ u32 vref_dac_mode = ((stage_step | 2U) != 6U) ? 1U : 0U;
+ u32 freq_pt = 0x0fU;
+ u8 type_flags_lo;
+ u8 ch_mask;
+ u32 channel;
+
+ if (dmem_read8(PMU_DMEM_CLK_GATE_FLAG) & (1U << 2)) {
+ u32 mode = pmu_phy_query_operating_mode();
+
+ if ((stage_step & ~1U) == 4U && mode == 3U)
+ freq_pt = 5U;
+ else
+ freq_pt = 0U;
+ }
+
+ type_flags_lo = dmem_read8(PMU_DMEM_DRAM_TYPE_FLAGS) & 7U;
+ if (type_flags_lo != 0U && stage_step < 0x13U &&
+ (0x601f0UL & (1UL << stage_step)))
+ freq_pt = (1U << type_flags_lo) - 1U;
+
+ ch_mask = dmem_read8(PMU_DMEM_CH0_RANK_EN) |
+ dmem_read8(PMU_DMEM_CH1_RANK_EN);
+
+ for (channel = 0U; channel < 2U; channel++) {
+ u32 rank;
+
+ if (!(ch_mask & (1U << channel)))
+ continue;
+
+ dmem_write8(PMU_DMEM_CAL_BYTE, (u8)channel);
+
+ for (rank = 0U; rank < 2U; rank++) {
+ u32 rank_vref_status = 0U;
+ u8 vref_eval_buf[80];
+
+ if (stage_step < 0x13U &&
+ (0x701f8UL & (1UL << stage_step)))
+ rank_vref_status =
+ pmu_dmem_training_flags_eval();
+
+ if (!pmu_channel_rank_avail_check(rank, channel))
+ continue;
+
+ if ((u32)(stage_step - 4U) < 5U) {
+ if (channel != 0U || rank != 0U)
+ continue;
+ }
+
+ pmu_phy_cal_trig_pulse();
+ memset(vref_eval_buf, 0, sizeof(vref_eval_buf));
+
+ switch (stage_step) {
+ case 0U:
+ pmu_cal_timing_margin_envelope_scan(channel,
+ rank,
+ vref_eval_buf);
+ pmu_cal_search_win_max_calc(channel, rank, 1U);
+ break;
+ case 1U:
+ pmu_cal_rank_deskew_matrix_commit(channel,
+ rank,
+ vref_eval_buf);
+ break;
+ case 4U:
+ case 5U:
+ case 6U:
+ case 7U: {
+ u32 commit_param =
+ (rank_vref_status == 0U) ? 0xffU : 511U;
+ u32 mode_cfg = 0U;
+
+ if ((stage_step & ~1U) == 6U) {
+ if (dmem_read8(PMU_DMEM_BIST_PATTERN_LO) != 0U ||
+ dmem_read8(PMU_DMEM_BIST_PATTERN_HI) != 0U ||
+ dmem_read8(PMU_DMEM_DESKEW_OFFSET_OVERRIDE) != 0U)
+ mode_cfg = 7U;
+ else
+ mode_cfg = 4U;
+ }
+ pmu_cal_rank_param_commit(freq_pt, mode_cfg,
+ commit_param,
+ vref_dac_mode);
+ break;
+ }
+ case 8U: {
+ u8 mr3_flags = dmem_read8(PMU_DMEM_VREF_FLAG_SIGNED);
+ u8 mr13_cfg = dmem_read8(PMU_DMEM_VREF_DAC_CTRL);
+ u8 mr22_cfg = dmem_read8(PMU_DMEM_TRAIN_PARAM_CTRL);
+ u32 cond = ((mr3_flags >> 6) & 1U) |
+ ((mr3_flags >> 7) & 1U) |
+ ((mr22_cfg >> 6) & 1U) |
+ (!(mr13_cfg & 0x20U) ? 1U : 0U);
+ u32 step_limit = (cond == 0U) ? 0xffU : 511U;
+ u32 commit_param =
+ ((mr22_cfg & 0x10U) << 5) | step_limit;
+
+ pmu_cal_rank_param_commit(freq_pt, 0U, commit_param,
+ 2U);
+ break;
+ }
+ case 15U: {
+ u16 vref_steps_buf[4];
+
+ pmu_cal_slice_step_diff_commit(channel,
+ vref_steps_buf,
+ 1U);
+ break;
+ }
+ case 16U:
+ pmu_cal_slice_deskew_result_commit(channel,
+ vref_eval_buf,
+ rank_vref_status);
+ pmu_cal_search_win_max_calc(channel, rank, 0U);
+ break;
+ case 17U:
+ pmu_cal_rank_rx_en_delay_commit(channel,
+ (u16)freq_pt,
+ rank_vref_status,
+ 0x11U);
+ break;
+ case 18U:
+ pmu_cal_rank_rx_en_delay_commit(channel,
+ (u16)freq_pt,
+ rank_vref_status,
+ 0x12U);
+ break;
+ case 19U:
+ pmu_cal_wck_ck_align_pulse_send(channel, rank);
+ break;
+ default:
+ break;
+ }
+ }
+
+ if ((u32)(stage_step - 0x11U) < 2U)
+ dmem_write16(PMU_DMEM_ITER_COUNT,
+ dmem_read16(PMU_DMEM_ITER_COUNT) + 1U);
+
+ pmu_phy_mode_cfg_dispatch(3U);
+ }
+
+ if (dmem_read8(PMU_DMEM_CLK_GATE_FLAG) & (1U << 6))
+ pmu_slice_deskew_pulse_seq(stage_step);
+
+ if ((u32)(stage_step - 0x11U) < 2U) {
+ if (dmem_read8(PMU_DMEM_CLK_GATE_FLAG) & (1U << 6))
+ pmu_slice_metric_scan_and_program();
+ }
+
+ dmem_write8(PMU_DMEM_TRAIN_MODE_HI, 0U);
+}
+
+/**
+ * pmu_multiparam_cal_dispatch() - Dispatch multi-parameter calibration structure updates
+ */
+static void pmu_multiparam_cal_dispatch(void)
+{
+ uintptr_t cal_addr =
+ (PMU_DMEM_BASE | PMU_DMEM_CAL_STRUCT_BASE);
+ const u8 *param =
+ (const u8 *)(PMU_DMEM_BASE | PMU_DMEM_SLICE_PROFILE_TABLE);
+ u8 *cal = (u8 *)cal_addr;
+
+ pmu_cal_lane_mask_or_all(cal_addr, 1U, param[0] & 8U);
+ pmu_cal_lane_mask_and_all(cal_addr, 0x14U, 0U);
+
+ pmu_cal_lane_mask_or_all(cal_addr, 0x14U, param[52] & 0x0fU);
+
+ if ((dmem_read8(PMU_DMEM_CAL_MODE) & 1U) &&
+ (*(const __volatile__ u8 *)(PMU_DMEM_BASE | PMU_DMEM_LANE_ACTIVE_FLAGS) != 0U)) {
+ const u8 *src =
+ (const u8 *)(PMU_DMEM_BASE | PMU_DMEM_DELAY_CAL_RESULTS);
+ u8 s0;
+ u8 s1;
+ u8 s2;
+ u8 s3;
+
+ s0 = src[0];
+ s1 = src[1];
+ s2 = src[2];
+ s3 = src[3];
+ cal[0x78] = s2;
+ cal[0x0c] = s0;
+ cal[0x42] = s1;
+ cal[0xae] = s3;
+ }
+
+ pmu_cal_lane_mask_and_all(cal_addr, 0x10U, 0xfbU);
+ pmu_cal_lane_mask_or_all(cal_addr, 0x10U, 0x41U);
+
+ pmu_multirank_slice_cal_dispatch((void *)cal_addr, 0U, 0U, 0,
+ 0xffU, 0U);
+
+ pmu_cal_lane_mask_or_all(cal_addr, 0x10U, 4U);
+ pmu_cal_multi_param_step_commit(cal_addr, 0U, 0xfffeffffUL,
+ 0xffffffffUL, 1U);
+}
+
+/**
+ * pmu_main_wrapper() - Dispatch top-level PMU training sequencer
+ *
+ * Return: Always 0.
+ */
+int pmu_main_wrapper(void)
+{
+ pmu_train_dispatcher();
+ return 0;
+}
+
+/**
+ * pmu_rx_en_build_swizzle_map() - Populate AC pin swizzle lookup map
+ * @swizzle_map: Output 14-byte pin-to-lane swizzle array
+ * @ch1_rank: Non-zero if Channel 1 is active
+ */
+static void pmu_rx_en_build_swizzle_map(u8 *swizzle_map, u8 ch1_rank)
+{
+ u32 ca;
+
+ for (ca = 0; ca < 7; ca++) {
+ u16 pin = phy_read16(PHY_AC_SWIZZLE_S0_BASE + (ca << 1));
+
+ if (pin < 7)
+ swizzle_map[pin] = (u8)ca;
+ }
+ if (ch1_rank != 0) {
+ for (ca = 0; ca < 7; ca++) {
+ u16 pin = phy_read16(PHY_AC_SWIZZLE_S1_BASE +
+ (ca << 1));
+
+ if (pin < 7)
+ swizzle_map[pin + 7] = (u8)ca;
+ }
+ }
+}
+
+/**
+ * pmu_rx_en_poll_rx_fifo_ranks() - Execute dual-scan multi-rank RX FIFO polling
+ * @ch0_rank: Active rank mask for Channel 0
+ * @ch1_rank: Active rank mask for Channel 1
+ * @initial_step: Output 4-element signed initial delay step array
+ * @scan_matrix: Output 3584-byte eye scan matrix
+ *
+ * Return: Updated active slice count (1 or 2).
+ */
+static u32 pmu_rx_en_poll_rx_fifo_ranks(u8 ch0_rank, u8 ch1_rank,
+ s8 *initial_step, u8 *scan_matrix)
+{
+ u32 num_slices = (ch1_rank != 0) ? 2 : 1;
+ u32 idx;
+
+ for (idx = 0; idx < 4; idx++) {
+ u16 out_cfg;
+ u8 out_mode_sel;
+ u8 out_num_slices;
+ u32 csr_offset;
+ u32 reg_base;
+ u8 channel;
+ u8 rank;
+ s8 ret;
+
+ if (pmu_cal_channel_rank_config_get(idx, &out_cfg,
+ &out_mode_sel,
+ &out_num_slices) != 0)
+ continue;
+
+ num_slices = (u32)out_num_slices;
+ dmem_write8(PMU_DMEM_AC_CAL_ACTIVE, 1);
+ pmu_phy_mode_cfg_dispatch(num_slices);
+
+ /* Re-assert both analog receiver power rails */
+ phy_write16(PHY_REG_CH0_ANALOG_PWR, (u16)ch0_rank);
+ phy_write16(PHY_REG_CH1_ANALOG_PWR, (u16)ch1_rank);
+
+ channel = out_num_slices - 1U;
+ rank = dmem_read8(PMU_DMEM_ACTIVE_RANK);
+ csr_offset = dmem_read32(PMU_DMEM_ACTIVE_CH_OFFSET);
+ reg_base = (((u32)channel << 12) | csr_offset) << 1;
+
+ phy_write16(PHY_AC_GATE_CTRL_RESET | reg_base, 0);
+ phy_write16(PHY_AC_BASE_OFFSET_CTRL | reg_base, 0);
+
+ /* Primary Scan: enable_scan = 1 */
+ ret = pmu_cal_dbyte_rx_fifo_reset_poll(channel,
+ (u32)out_mode_sel, 1,
+ scan_matrix);
+ initial_step[((u32)rank << 1) + (u32)out_cfg] = ret;
+
+ /* Baseline Scan: enable_scan = 0, first arg is out_mode_sel */
+ (void)pmu_cal_dbyte_rx_fifo_reset_poll(channel,
+ (u32)out_mode_sel, 0,
+ scan_matrix);
+ }
+
+ return num_slices;
+}
+
+/**
+ * pmu_rx_en_search_and_commit_slice() - Find 256-tap eye centers and commit AC slice delays
+ * @s: Slice index (0 or 1)
+ * @step: Normalized step stride (1, 2, 4, or 8)
+ * @swizzle_map: 14-byte pin-to-lane swizzle lookup table
+ * @initial_step: 4-element signed initial delay step array
+ * @scan_matrix: 3584-byte eye scan matrix
+ */
+static void pmu_rx_en_search_and_commit_slice(u32 s, u8 step,
+ const u8 *swizzle_map,
+ const s8 *initial_step,
+ const u8 *scan_matrix)
+{
+ u16 center_delay[7];
+ s32 base_delay = (dmem_read8(PMU_DMEM_CAL_STEP_CONFIG) &
+ BIT(3)) ? 0 : 0x40;
+ s32 lower_bound = base_delay - 128;
+ s32 upper_bound = base_delay + 128;
+ u32 csr_offset;
+ u32 s_base;
+ s32 dly0, dly1;
+ u32 ca;
+
+ for (ca = 0; ca < 7; ca++) {
+ u32 best_start = 0;
+ u32 best_len = 0;
+ u32 run_start = 0;
+ u32 cur_len = 0;
+ u32 in_run = 0;
+ u32 delay;
+
+ for (delay = 0; delay < 256; delay += step) {
+ u32 mat_idx = s * 1792 + ca * 256 + delay;
+
+ if (scan_matrix[mat_idx] == 0) {
+ if (!in_run) {
+ run_start = delay;
+ cur_len = 0;
+ in_run = 1;
+ }
+ cur_len += step;
+ } else if (in_run) {
+ if (cur_len > best_len && delay >= 129) {
+ best_start = run_start;
+ best_len = cur_len + step;
+ }
+ in_run = 0;
+ cur_len = 0;
+ }
+ }
+
+ if (in_run &&
+ (cur_len > best_len || (best_len == 0 && cur_len != 0))) {
+ best_start = run_start;
+ best_len = cur_len + step;
+ }
+
+ if (best_len == 0) {
+ /* Nominal midpoint fallback (0x40) */
+ center_delay[ca] = 0x40;
+ } else {
+ s32 center = (s32)best_start +
+ (s32)((best_len & ~1U) / 2) +
+ lower_bound;
+
+ if (center < 0)
+ center = 0;
+ if (center > upper_bound)
+ center = upper_bound;
+ center_delay[ca] = (u16)center;
+ }
+ }
+
+ /* Commit Calibrated Delays to Physical AC Slice */
+ csr_offset = dmem_read32(PMU_DMEM_ACTIVE_CH_OFFSET);
+ for (ca = 0; ca < 7; ca++) {
+ u8 mapped_lane = swizzle_map[s * 7 + ca];
+ u32 addr = PHY_AC_PIN_DELAY_BASE |
+ ((csr_offset | (s << 12) |
+ ((u32)mapped_lane << 8)) << 1);
+
+ phy_write16(addr, center_delay[ca]);
+ }
+
+ csr_offset = dmem_read32(PMU_DMEM_ACTIVE_CH_OFFSET);
+ s_base = (csr_offset | (s << 12)) << 1;
+ dly0 = base_delay + (s32)initial_step[s * 2 + 0];
+ dly1 = base_delay + (s32)initial_step[s * 2 + 1];
+ phy_write16(PHY_AC_MASTER_DESKEW0 | s_base,
+ (dly0 < 0) ? 0 : (u16)dly0);
+ phy_write16(PHY_AC_MASTER_DESKEW1 | s_base,
+ (dly1 < 0) ? 0 : (u16)dly1);
+ phy_write16(PHY_AC_BASE_OFFSET_CTRL | s_base, (u16)base_delay);
+}
+
+/**
+ * pmu_cal_rx_en_pre_init() - Stage 1 Command Bus Training / CA Pre-Initialization
+ *
+ * Orchestrates initial Command-Address (CA) timing calibration across active
+ * DRAM channels, ranks, and Address/Command (AC) slices:
+ *
+ * 1. Checks active channel rank flags (`0x80000025` and `0x80000040`), verifies
+ * the pre-condition clock gating flag (`0x80000001`), powers both Channel 0
+ * and Channel 1 analog receiver rails (`0x900400e4` and `0x900400e6`),
+ * enables AC macro pad drivers (`0x900e00ca = 0x01ff`), asserts Master CKE
+ * (`0x9004000e |= 0x0100`), and enables DBYTE broadcast calibration strobe
+ * gating.
+ * 2. Backs up the 6-byte AC PHY descriptor block from DCCM `0x8000099c`,
+ * enforcing the Cold-Boot Zero-Token APB Bus Guard (`0x0001008bUL` fallback
+ * when `token == 0`), and streams it to PHY registers.
+ * 3. Normalizes the 4-bit calibration step configuration (`0x8000041a`) into a
+ * one-hot power-of-2 stride (`1`, `2`, `4`, or `8`) at `0x8000047c` and
+ * zero-initializes the 3,584-byte raw eye scan matrix.
+ * 4. Remaps calibration structure masks and reads hardware AC pin swizzle
+ * registers (`0x90060120` and conditionally `0x90062120`, with strict `< 7`
+ * pin bounds on both slices) into `swizzle_map[14]`.
+ * 5. Iterates across ranks `0..3` using a 16-bit halfword `out_cfg` output
+ * pointer, re-asserts analog receiver power rails after
+ * `pmu_phy_mode_cfg_dispatch`, resets AC slice timing registers, and
+ * executes dual-phase 3-argument RX FIFO reset polling to populate
+ * `initial_step[4]` and `scan_matrix[3584]`.
+ * 6. Clears AC profile lane registers (`0x9007e002` with `ca << 8` stride) and
+ * master register (`0x9007e004`), then pulses the primary calibration strobe.
+ * 7. Executes a 256-tap contiguous zero-run eye window search (`delay >= 129`
+ * threshold, nominal midpoint fallback `0x40`) and
+ * programs per-lane AC delays and master deskew registers.
+ * 8. Pulses primary and secondary calibration strobes, restores the 6-byte AC
+ * PHY descriptor backup, disables DBYTE broadcast strobe gating, and
+ * re-asserts Master CKE (`0x9004000e |= 0x0100`).
+ */
+#define PMU_RX_SCAN_MATRIX_BYTES 3584U
+
+static void pmu_cal_rx_en_pre_init(void)
+{
+ u8 *scan_matrix = (u8 *)dmem_ptr(PMU_DMEM_RX_SCAN_MATRIX);
+ u8 swizzle_map[14] = { 0 };
+ s8 initial_step[4] = { 0 };
+ u8 descriptor_backup[6];
+ u8 ch0_rank;
+ u8 ch1_rank;
+ u8 raw_step;
+ u32 num_slices;
+ u32 ch_offset;
+ u8 step;
+ u32 ca;
+ u32 s;
+
+ ch0_rank = dmem_read8(PMU_DMEM_CH0_ANALOG_EN);
+ ch1_rank = dmem_read8(PMU_DMEM_CH1_ACTIVE_FLAG);
+ if (ch0_rank == 0 && ch1_rank == 0)
+ return;
+
+ /* Pre-condition clock gate check */
+ if ((dmem_read8(PMU_DMEM_CLK_GATE_FLAG) & BIT(3)) != 0) {
+ pmu_assert_or_halt(0, PMU_ERR_LPCA_CLK_GATE_PRECOND);
+ return;
+ }
+
+ /* Retain analog receiver power rails continuously */
+ phy_write16(PHY_REG_CH0_ANALOG_PWR, (u16)ch0_rank);
+ phy_write16(PHY_REG_CH1_ANALOG_PWR, (u16)ch1_rank);
+
+ /* Activate AC macro pad drivers & assert Master CKE */
+ phy_write16(PHY_REG_AC_TX_EN_BCAST, 0x01ff);
+ phy_write16(PHY_REG_MASTER_CFG,
+ phy_read16(PHY_REG_MASTER_CFG) | 0x0100);
+
+ /* Enable DBYTE broadcast strobe gating */
+ pmu_dbyte_cal_strobe_pulse_seq(1, 2047, 0);
+
+ /* Save 6-byte AC PHY register 0x0001008b descriptor */
+ memset(descriptor_backup, 0, sizeof(descriptor_backup));
+ descriptor_backup[0] = 0x8bU;
+ descriptor_backup[2] = 0x01U;
+ pmu_phy_reg_buffer_stream(descriptor_backup, 1, 1);
+
+ /* One-hot power-of-2 step stride normalization */
+ raw_step = dmem_read8(PMU_DMEM_STEP_CFG_RAW) >> 4;
+ if (raw_step & 8U)
+ step = 8U;
+ else if (raw_step & 4U)
+ step = 4U;
+ else if (raw_step & 2U)
+ step = 2U;
+ else
+ step = 1U;
+ dmem_write8(PMU_DMEM_NORM_STEP_STRIDE, step);
+
+ memset(scan_matrix, 0, PMU_RX_SCAN_MATRIX_BYTES);
+ pmu_cal_struct_mask_remap();
+
+ /* Phase 2-3: Build swizzle map and run dual-scan RX FIFO polling */
+ pmu_rx_en_build_swizzle_map(swizzle_map, ch1_rank);
+ num_slices = pmu_rx_en_poll_rx_fifo_ranks(ch0_rank, ch1_rank,
+ initial_step, scan_matrix);
+
+ /* Phase 4: Clear AC Profile Registers with ca << 8 stride */
+ ch_offset = dmem_read32(PMU_DMEM_ACTIVE_CH_OFFSET);
+ for (ca = 0; ca < 7; ca++) {
+ u32 addr = PHY_AC_PROFILE_LANE_BASE |
+ ((ch_offset | (ca << 8)) << 1);
+
+ phy_write16(addr, 0);
+ }
+ phy_write16(PHY_AC_PROFILE_MASTER_CTRL | (ch_offset << 1), 0);
+ pmu_phy_cal_trig_pulse();
+
+ /* Phase 5: 256-Tap Eye Margin Search with delay >= 129 Bias */
+ for (s = 0; s < num_slices; s++)
+ pmu_rx_en_search_and_commit_slice(s, step, swizzle_map,
+ initial_step, scan_matrix);
+
+ /* Phase 6: Epilogue, Strobes, Descriptor Restore & Master CKE Re-Assertion */
+ pmu_phy_cal_trig_pulse();
+ pmu_phy_sec_cal_strobe_pulse();
+ pmu_phy_reg_buffer_stream(descriptor_backup, 1, 0);
+ pmu_dbyte_cal_strobe_pulse_seq(0, 0, 0);
+
+ /* Re-assert Master CKE */
+ phy_write16(PHY_REG_MASTER_CFG,
+ phy_read16(PHY_REG_MASTER_CFG) | 0x0100);
+}
+
+/**
+ * pmu_cal_ca_vref_step() - Execute Stage 2 Command/Address Vref adjustment step
+ * @arg0: Pass selector (0 for initial pass, non-zero for secondary pass)
+ */
+static void pmu_cal_ca_vref_step(u32 arg0)
+{
+ u32 step;
+ u32 post_cmd;
+
+ step = (arg0 == 0) ? 2 : 1;
+ *(__volatile__ u16 *)PMU_DMEM_TRAIN_MODE_ADDR = 2;
+ pmu_cal_vref_dac_step_adjust(step);
+ *(__volatile__ u16 *)PMU_DMEM_TRAIN_MODE_ADDR = 0x8002;
+
+ post_cmd = (arg0 != 0) ? 1 : 0xfe;
+ pmu_post_cmd_conditional_dispatch(post_cmd);
+}
+
+/**
+ * pmu_cal_ca_train_step() - Execute Stage 3 Command/Address training step
+ *
+ * Selects CA Vref DAC adjustment mode and post-command dispatch index based
+ * on protocol bits [1:0] at DCCM 0x8000008e, bracketed by Stage 3 mode
+ * assertions (0x0004 -> 0x8004) at DCCM 0x80000014.
+ */
+static void pmu_cal_ca_train_step(void)
+{
+ u8 flag = dmem_read8(PMU_DMEM_MR20_ADDR);
+ u32 post_cmd;
+
+ *(__volatile__ u16 *)PMU_DMEM_TRAIN_MODE_ADDR = 4;
+
+ if ((flag & 3) != 0) {
+ pmu_cal_vref_dac_step_adjust(0);
+ post_cmd = 2;
+ } else {
+ pmu_cal_vref_dac_step_adjust(0x10);
+ post_cmd = 5;
+ }
+
+ pmu_post_cmd_conditional_dispatch(post_cmd);
+ *(__volatile__ u16 *)PMU_DMEM_TRAIN_MODE_ADDR = 0x8004;
+}
+
+/**
+ * pmu_cal_zq_pad_calibration() - Execute Stage 4 Master PUB ZQ pad calibration
+ * @is_primary: Non-zero for primary pass, 0 for secondary post-deskew pass
+ *
+ * Return: Stage 4 completion mode word (0x8008).
+ */
+static u32 pmu_cal_zq_pad_calibration(u32 is_primary)
+{
+ dmem_write16(PMU_DMEM_TRAIN_MODE, 8U);
+
+ if (dmem_read16(PMU_DMEM_DRAM_FREQ_OFF) > 0xc7fU && !is_primary) {
+ pmu_sdata()->high_freq_flag = 1;
+ if ((dmem_read8(PMU_DMEM_CAL_MISC_FLAGS) & 0x20U) == 0U)
+ pmu_sdata()->high_freq_flag = 0;
+ }
+
+ pmu_sdata()->cal_active_flag = !is_primary;
+
+ if (is_primary) {
+ if (!pmu_stage_slice_flag_check())
+ pmu_cal_vref_dac_step_adjust(7);
+ pmu_cal_vref_dac_step_adjust(6);
+ pmu_post_cmd_conditional_dispatch(0xfd);
+ } else {
+ pmu_cal_vref_dac_step_adjust(5);
+ pmu_cal_vref_dac_step_adjust(4);
+ pmu_post_cmd_conditional_dispatch(3);
+ }
+
+ pmu_sdata()->cal_active_flag = 0;
+ dmem_write16(PMU_DMEM_TRAIN_MODE, 0x8008U);
+ return 0x8008;
+}
+
+/**
+ * pmu_cal_timing_margin_eval() - Execute Stage 10 timing margin evaluation
+ *
+ * Configures PHY mode 3, updates PMU_DMEM_TRAIN_MODE (0x0200 -> 0x8200),
+ * clears PMU_DMEM_MARGIN_STAT, and dispatches post-command 9.
+ */
+static void pmu_cal_timing_margin_eval(void)
+{
+ pmu_phy_mode_cfg_dispatch(3);
+ *(__volatile__ u16 *)PMU_DMEM_TRAIN_MODE_ADDR = 0x0200;
+ *(__volatile__ u16 *)PMU_DMEM_TRAIN_MODE_ADDR = 0x8200;
+ *(__volatile__ u8 *)PMU_DMEM_MARGIN_STAT = 0;
+ pmu_post_cmd_conditional_dispatch(9);
+}
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 19/26] sunxi: dram: a733: Add Allwinner A733 LPDDR5 DRAM driver
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (17 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 18/26] sunxi: dram: a733: Add ARC PMU training state machine Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 20/26] sunxi: sun60i: initial support for A733 SoC Alastair D'Silva
` (8 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
Add the SPL LPDDR5 dual-channel uMCTL2 and Synopsys DesignWare DDR PHY
driver for the Allwinner A733 (sun60iw2p1) SoC, together with U-Boot
proper's board_add_ram_info() speed/generation reporting.
Features:
- Invokes build_pmu_fw.py via Kbuild during SPL build (supporting
ARC_CROSS_COMPILE, PMU_FW, and BINMAN_ALLOW_MISSING) and places the
one-time initialization tables and PMU ICCM image in .init.rodata.
- Scales PLL_DDR and PMU message-block data rate from CONFIG_DRAM_CLK
(defaulting to 1800 MHz WCK / 3600 MT/s LPDDR5).
- Attempts dual-rank training first with automatic fallback to
single-rank training, and probes exact capacity (including 3/4-density
6 Gb / 12 Gb dies) via JEDEC JESD209-5 Mode Register 8 (MR8).
- Guards the A733 reference / Orange Pi 4 Pro CA/DQ lane swizzle map
behind CONFIG_DRAM_SUN60I_A733_PIN_SWIZZLE.
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
arch/arm/include/asm/arch-sunxi/dram.h | 2 +
arch/arm/mach-sunxi/Kconfig | 21 +-
arch/arm/mach-sunxi/Makefile | 37 +
arch/arm/mach-sunxi/dram_sun60i_a733.c | 1303 +++++++++++++++++
arch/arm/mach-sunxi/dram_sun60i_a733_tables.h | 941 ++++++++++++
5 files changed, 2303 insertions(+), 1 deletion(-)
create mode 100644 arch/arm/mach-sunxi/dram_sun60i_a733.c
create mode 100644 arch/arm/mach-sunxi/dram_sun60i_a733_tables.h
diff --git a/arch/arm/include/asm/arch-sunxi/dram.h b/arch/arm/include/asm/arch-sunxi/dram.h
index 59e2e980bfa..58587a713eb 100644
--- a/arch/arm/include/asm/arch-sunxi/dram.h
+++ b/arch/arm/include/asm/arch-sunxi/dram.h
@@ -37,6 +37,8 @@
#include <asm/arch/dram_suniv.h>
#elif defined(CONFIG_MACH_SUN55I_A523)
#include <asm/arch/dram_sun55i_a523.h>
+#elif defined(CONFIG_DRAM_SUN60I_A733)
+/* A733 DRAM registers are private to dram_sun60i_a733.c */
#else
#include <asm/arch/dram_sun4i.h>
#endif
diff --git a/arch/arm/mach-sunxi/Kconfig b/arch/arm/mach-sunxi/Kconfig
index b7ee14c6678..f3f70b277b1 100644
--- a/arch/arm/mach-sunxi/Kconfig
+++ b/arch/arm/mach-sunxi/Kconfig
@@ -62,6 +62,21 @@ config DRAM_SUN55I_A523
help
Select this DRAM controller driver for A523/T527 SoCs.
+config DRAM_SUN60I_A733
+ bool
+ help
+ Select this LPDDR5 DRAM controller and Synopsys DDR PHY driver for
+ Allwinner A733/T736 SoCs.
+
+config DRAM_SUN60I_A733_PIN_SWIZZLE
+ bool "Allwinner A733 LPDDR5 CA/DQ pin swizzle"
+ depends on DRAM_SUN60I_A733
+ help
+ Program the CA0..CA6 and DQ0..DQ7 lane permutation used on boards
+ such as the Orange Pi 4 Pro. Disable this option on boards that use
+ a non-swizzled or custom PCB trace mapping to the LPDDR5 memory
+ package.
+
if DRAM_SUN50I_H616 || DRAM_SUN50I_A133 || DRAM_SUN55I_A523
config DRAM_SUNXI_DX_ODT
hex "DRAM DX ODT parameter"
@@ -741,10 +756,12 @@ config DRAM_CLK
default 744 if MACH_SUN50I_H6
default 720 if MACH_SUN50I_H616 || MACH_SUN50I_A133
default 1200 if MACH_SUN55I_A523
+ default 1800 if DRAM_SUN60I_A733
help
Set the dram clock speed, valid range 240 - 480 (prior to sun9i),
must be a multiple of 24. For the sun9i (A80), the tested values
- (for DDR3-1600) are 312 to 792.
+ (for DDR3-1600) are 312 to 792. For sun60i (A733 LPDDR5), the WCK
+ clock defaults to 1800 MHz (3600 MT/s).
if MACH_SUN5I || MACH_SUN7I
config DRAM_MBUS_CLK
@@ -760,6 +777,7 @@ config DRAM_ZQ
depends on !MACH_SUN50I_H616
depends on !MACH_SUN50I_A133
depends on !MACH_SUN55I_A523
+ depends on !DRAM_SUN60I_A733
default 123 if MACH_SUN4I || MACH_SUN5I || MACH_SUN6I || \
MACH_SUN8I_A23 || MACH_SUN8I_A33 || MACH_SUN8I_A83T
default 127 if MACH_SUN7I
@@ -774,6 +792,7 @@ config DRAM_ODT_EN
bool "sunxi dram odt enable"
depends on !MACH_SUN50I_H616
depends on !MACH_SUN55I_A523
+ depends on !DRAM_SUN60I_A733
default y if MACH_SUN8I_A23
default y if MACH_SUNXI_H3_H5
default y if MACH_SUN8I_R40
diff --git a/arch/arm/mach-sunxi/Makefile b/arch/arm/mach-sunxi/Makefile
index ff475d8fff5..8bee03524bf 100644
--- a/arch/arm/mach-sunxi/Makefile
+++ b/arch/arm/mach-sunxi/Makefile
@@ -26,6 +26,7 @@ endif
obj-$(CONFIG_MACH_SUN9I) += clock_sun9i.o gtbus_sun9i.o
obj-$(CONFIG_SUN50I_GEN_H6) += clock_sun50i_h6.o
obj-$(CONFIG_SUNXI_GEN_NCAT2) += clock_sun50i_h6.o
+obj-$(CONFIG_DRAM_SUN60I_A733) += dram_sun60i_a733.o
ifndef CONFIG_ARM64
obj-y += timer.o
endif
@@ -50,4 +51,40 @@ obj-$(CONFIG_DRAM_SUN50I_A133) += dram_sun50i_a133.o
obj-$(CONFIG_DRAM_SUN50I_A133) += dram_timings/
obj-$(CONFIG_MACH_SUN55I_A523) += dram_sun55i_a523.o dram_dw_helpers.o
obj-$(CONFIG_DRAM_SUN55I_A523) += dram_timings/
+
+ifdef CONFIG_DRAM_SUN60I_A733
+ifdef CONFIG_XPL_BUILD
+PMU_FW_DIR := $(srctree)/arch/arm/mach-sunxi/pmu
+PMU_FW_SRCS := \
+ $(PMU_FW_DIR)/pmu_primitives.c \
+ $(PMU_FW_DIR)/pmu_clock.c \
+ $(PMU_FW_DIR)/pmu_telemetry.c \
+ $(PMU_FW_DIR)/pmu_cbt.c \
+ $(PMU_FW_DIR)/pmu_deskew.c \
+ $(PMU_FW_DIR)/pmu_eye.c \
+ $(PMU_FW_DIR)/pmu_train.c
+PMU_FW_HDRS := \
+ $(PMU_FW_DIR)/sunxi_pmu_abi.h \
+ $(PMU_FW_DIR)/dwc_ddrphy_pmu_regs.h \
+ $(PMU_FW_DIR)/pmu_fw.h \
+ $(PMU_FW_DIR)/pmu.ld
+
+quiet_cmd_pmu_fw = PMUFW $@
+ cmd_pmu_fw = $(PYTHON3) -B $(PMU_FW_DIR)/build_pmu_fw.py \
+ $(if $(ARC_CROSS_COMPILE),--cross-compile $(ARC_CROSS_COMPILE)) \
+ $(if $(PMU_FW),--pmu-fw $(PMU_FW)) \
+ $(if $(filter 1 y,$(BINMAN_ALLOW_MISSING) $(ALLOW_MISSING)),--allow-missing) \
+ --src-dir $(PMU_FW_DIR) \
+ --ld-script $(PMU_FW_DIR)/pmu.ld \
+ --out $@ $(PMU_FW_SRCS)
+
+$(obj)/dram_sun60i_a733_pmu_fw.h: $(PMU_FW_DIR)/build_pmu_fw.py $(PMU_FW_SRCS) $(PMU_FW_HDRS) $(PMU_FW) FORCE
+ $(call if_changed,pmu_fw)
+
+targets += dram_sun60i_a733_pmu_fw.h
+clean-files += dram_sun60i_a733_pmu_fw.h
+CFLAGS_dram_sun60i_a733.o += -I$(obj)
+$(obj)/dram_sun60i_a733.o: $(obj)/dram_sun60i_a733_pmu_fw.h
+endif
+endif
endif
diff --git a/arch/arm/mach-sunxi/dram_sun60i_a733.c b/arch/arm/mach-sunxi/dram_sun60i_a733.c
new file mode 100644
index 00000000000..e228c7fe1d8
--- /dev/null
+++ b/arch/arm/mach-sunxi/dram_sun60i_a733.c
@@ -0,0 +1,1303 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Allwinner sun60i A733 / T736 DRAM Controller and Synopsys DDR PHY driver
+ *
+ * (C) Copyright 2026 Alastair D'Silva <alastair@d-silva.org>
+ */
+
+#include <hang.h>
+#include <init.h>
+#include <log.h>
+#include <stdio.h>
+#include <asm/arch/dram.h>
+#include <asm/barriers.h>
+#include <asm/io.h>
+#include <linux/bitops.h>
+#include <linux/build_bug.h>
+#include <linux/delay.h>
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/sizes.h>
+#include <linux/types.h>
+
+#include "pmu/sunxi_pmu_abi.h"
+
+#define CCU_PLL_DDR_CTRL_REG 0x02002020UL
+#define CCU_DRAM0_CLK_REG 0x02002c00UL
+#define UMCTL2_CH0_BASE 0x0a110000UL
+#define UMCTL2_MSTR_OFFSET 0x000UL
+#define UMCTL2_MSTR_LPDDR5 BIT(19)
+
+#ifdef CONFIG_XPL_BUILD
+#include "dram_sun60i_a733_pmu_fw.h"
+
+#ifdef SUN60I_A733_PMU_FW_MISSING
+unsigned long sunxi_dram_init(void)
+{
+ (void)sun60i_a733_pmu_imem;
+ pr_err("DRAM: ARC PMU firmware missing\n");
+ return 0;
+}
+#else
+#include "dram_sun60i_a733_tables.h"
+
+/* ========================================================================= */
+/* 1. Register Sequencer & Aperture Boundaries */
+/* ========================================================================= */
+
+#define SUNXI_MMIO_MIN_ADDR 0x02000000UL
+#define SUNXI_MMIO_MAX_ADDR 0x40000000UL
+
+#define CCU_PLL_DDR_LOCK_BIT BIT(28)
+#define CCU_PLL_DDR_OUTPUT_GATE BIT(27)
+#define CCU_MBUS_CLK_REG 0x02002580UL
+#define CCU_MBUS_BGR_REG 0x02002584UL
+#define CCU_DRAM_MUX_CLK_REG 0x02002588UL
+#define CCU_DRAM0_BGR_REG 0x02002c0cUL
+
+#define PRCM_MEM_ISOLATION_REG 0x07010250UL
+
+#define UMCTL2_CH0_APERTURE_BASE 0x0a100000UL
+#define UMCTL2_CH0_APERTURE_END 0x0a500000UL
+#define UMCTL2_CH1_BASE 0x0a510000UL
+#define UMCTL2_CH_STRIDE 0x00400000UL
+#define UMCTL2_STAT_OFFSET 0x014UL
+#define UMCTL2_MRCTRL0_OFFSET 0x080UL
+#define UMCTL2_MRCTRL1_OFFSET 0x084UL
+#define UMCTL2_PWRCTL_OFFSET 0x180UL
+#define UMCTL2_SWCTL_OFFSET 0xc80UL
+#define UMCTL2_SWSTAT_OFFSET 0xc84UL
+#define UMCTL2_DERATEINT_OFFSET 0xc9cUL
+
+#define UMCTL2_MSTR_RANKS_SHIFT 24
+#define UMCTL2_MSTR_RANKS_MASK 0x0fU
+
+#define DRAMC_SYS_CFG_REG 0x0a020010UL
+#define DRAMC_CH_EN_REG 0x0a020014UL
+#define DRAMC_MR_DATA_CH0_REG 0x0a02002cUL
+#define DRAMC_MR_DATA_CH1_REG 0x0a020040UL
+#define DRAMC_CH0_PORT_CFG_REG 0x0a02021cUL
+#define DRAMC_CH0_PORT_EN_REG 0x0a020220UL
+#define DRAMC_CH1_PORT_CFG_REG 0x0a020224UL
+#define DRAMC_CH1_PORT_EN_REG 0x0a020228UL
+
+/* ========================================================================= */
+/* 2. Synopsys PHY & Controller Apertures */
+/* ========================================================================= */
+
+#define SUNXI_UMCTL2_CH0_STAT (UMCTL2_CH0_BASE + UMCTL2_STAT_OFFSET)
+#define SUNXI_UMCTL2_CH1_STAT (UMCTL2_CH1_BASE + UMCTL2_STAT_OFFSET)
+
+#define SUNXI_PRCM_CH0_DMC_CLK_REG 0x07010310UL
+#define SUNXI_PRCM_CH0_DMC_FEAT_REG 0x07010314UL
+#define SUNXI_PRCM_CH1_DMC_CLK_REG 0x07410310UL
+#define SUNXI_PRCM_CH1_DMC_FEAT_REG 0x07410314UL
+
+#define SUNXI_DRAM_PHY_APB_BASE 0x0a900000UL
+#define SUNXI_DRAM_PHY_APB_END 0x0ab00000UL
+#define PHY_SLICE_STRIDE 0x2000UL
+#define PHY_LANE_STRIDE 0x0200UL
+
+#define PHY_DBYTE0_BASE 0x0a920000UL
+#define PHY_DBYTE_CLK_GATE_OFF 0x001eUL
+#define PHY_DBYTE0_DQ_SWIZZLE_BASE 0x0a920100UL
+#define PHY_DBYTE_RX_EN_OFF 0x0112UL
+#define PHY_DBYTE_TX_EN_OFF 0x0114UL
+#define PHY_DBYTE_COMP_EN_OFF 0x0116UL
+#define PHY_DBYTE_RX_BIAS_OFF 0x0118UL
+#define PHY_DBYTE_RFIFO_OFF 0x0146UL
+#define PHY_DBYTE_LCDL_OFF 0x0162UL
+#define PHY_DBYTE_STAT_OFF 0x016aUL
+
+#define PHY_PIE_PTR_LO_BASE 0x0a940248UL
+#define PHY_PIE_PTR_HI_BASE 0x0a940316UL
+#define PHY_AC0_SWIZZLE_BASE 0x0a960120UL
+#define PHY_PIE_ZERO_SRAM_BASE 0x0a982000UL
+#define PHY_PIE_INIT_SRAM_BASE 0x0a982010UL
+#define PHY_PIE_SEQ_SRAM_BASE 0x0a982710UL
+#define PHY_PIE_LANE_DELAY_BASE 0x0a9e0048UL
+#define PHY_PIE_LANE_DELAY_INIT 0x5a3cU
+#define PHY_ACSM_SRAM_BASE 0x0aa2005cUL
+#define PHY_PUB_POST_PMU_CFG_REG 0x0aa80100UL
+
+#ifdef DEBUG
+#define A733_DRAM_CH0_BASE 0x40000000UL
+#define A733_DRAM_CH1_BASE 0xc0000000UL
+#define A733_DRAM_TEST_WORDS 4096
+
+static const uintptr_t sparse_probe_addrs[] __section(".init.rodata") = {
+ 0x40000000UL, /* Ch0 Base */
+ 0x40800000UL, /* +8 MiB */
+ 0x44000000UL, /* +64 MiB, FIT image load aperture */
+ 0x4a000000UL, /* +160 MiB, U-Boot text base aperture */
+ 0x50000000UL, /* +256 MiB */
+ 0xc0000000UL, /* Ch1 Base */
+};
+#endif
+
+/* ========================================================================= */
+/* SECTION 1: Hardware Register Sequencer & State Polling */
+/* ========================================================================= */
+
+/**
+ * mctl_poll_reg32() - Poll a 32-bit MMIO register until masked bits match
+ * @addr: Physical address of status register to poll
+ * @mask: Bitmask to apply to the register value
+ * @expected: Expected masked value
+ * @timeout_us: Maximum poll iterations (1 us per iteration)
+ * @desc: Diagnostic description logged on timeout
+ * @trig_addr: Trigger register address logged on timeout
+ *
+ * Return: 0 on success, or -ETIMEDOUT on timeout.
+ */
+static int mctl_poll_reg32(uintptr_t addr, u32 mask, u32 expected,
+ int timeout_us, const char *desc,
+ uintptr_t trig_addr)
+{
+ for (; timeout_us > 0; timeout_us--) {
+ if ((readl((void __iomem *)addr) & mask) == expected)
+ return 0;
+ udelay(1);
+ }
+
+ pr_err("DRAM: %s timeout at 0x%08lx\n", desc,
+ (unsigned long)trig_addr);
+ return -ETIMEDOUT;
+}
+
+/**
+ * mctl_sync_hardware_state() - Synchronize hardware state after a register write
+ * @addr: Physical MMIO address that was just written
+ * @val: 32-bit value that was written to @addr
+ *
+ * Polls hardware status registers or inserts required settling delays when
+ * specific control registers (PLL_DDR, uMCTL2 SWCTL/MRCTRL0/DFIMISC, or PRCM
+ * memory isolation) are written during table replay.
+ *
+ * Return: 0 on success, or negative errno on poll failure.
+ */
+static int mctl_sync_hardware_state(uintptr_t addr, u32 val)
+{
+ /* CCU PLL_DDR lock before output gate enable */
+ if (addr == CCU_PLL_DDR_CTRL_REG &&
+ (val & 0xe0000000U) == 0xe0000000U &&
+ !(val & CCU_PLL_DDR_OUTPUT_GATE)) {
+ udelay(20);
+ return mctl_poll_reg32(CCU_PLL_DDR_CTRL_REG, CCU_PLL_DDR_LOCK_BIT,
+ CCU_PLL_DDR_LOCK_BIT, 100000,
+ "PLL_DDR lock", addr);
+ }
+
+ /* uMCTL2 SWCTL -> SWSTAT quasi-dynamic register update ack */
+ if ((addr == UMCTL2_CH0_BASE + UMCTL2_SWCTL_OFFSET ||
+ addr == UMCTL2_CH1_BASE + UMCTL2_SWCTL_OFFSET) && val == 1U) {
+ return mctl_poll_reg32(addr - UMCTL2_SWCTL_OFFSET + UMCTL2_SWSTAT_OFFSET,
+ 0x01U, 0x01U, 10000, "SWCTL handshake", addr);
+ }
+
+ /* uMCTL2 PWRCTL self-refresh exit to Normal Operating Mode */
+ if ((addr == UMCTL2_CH0_BASE + UMCTL2_PWRCTL_OFFSET ||
+ addr == UMCTL2_CH1_BASE + UMCTL2_PWRCTL_OFFSET) && val == 0U) {
+ uintptr_t stat_addr = addr - UMCTL2_PWRCTL_OFFSET +
+ UMCTL2_STAT_OFFSET;
+ int ret;
+
+ ret = mctl_poll_reg32(stat_addr, 0x03U << 4, 0U, 100000,
+ "Self-refresh exit", addr);
+ if (ret)
+ return ret;
+ return mctl_poll_reg32(stat_addr, 0x03U, 0x01U, 100000,
+ "Normal mode entry", addr);
+ }
+
+ /* uMCTL2 MRCTRL0 Mode Register command completion */
+ if ((addr == UMCTL2_CH0_BASE + UMCTL2_MRCTRL0_OFFSET ||
+ addr == UMCTL2_CH1_BASE + UMCTL2_MRCTRL0_OFFSET) &&
+ (val & BIT(31))) {
+ return mctl_poll_reg32(addr, BIT(31), 0U, 10000,
+ "MR command trigger", addr);
+ }
+
+ /* PRCM memory isolation release settle delay */
+ if (addr == PRCM_MEM_ISOLATION_REG) {
+ mdelay(2);
+ return 0;
+ }
+
+ return 0;
+}
+
+/**
+ * mctl_write_single_reg() - Write a single MMIO register and synchronize state
+ * @addr: Physical MMIO address
+ * @val: Value to write (16-bit for DDR PHY APB, 32-bit otherwise)
+ */
+static int mctl_write_single_reg(uintptr_t addr, u32 val)
+{
+ if (addr < SUNXI_MMIO_MIN_ADDR || addr >= SUNXI_MMIO_MAX_ADDR)
+ return -EINVAL;
+
+ if (addr >= SUNXI_DRAM_PHY_APB_BASE && addr < SUNXI_DRAM_PHY_APB_END)
+ writew((u16)val, (void __iomem *)addr);
+ else
+ writel(val, (void __iomem *)addr);
+
+ return mctl_sync_hardware_state(addr, val);
+}
+
+/**
+ * mctl_write_regs() - Replay an array of width- and broadcast-tagged MMIO writes
+ * @regs: Pointer to array of struct sunxi_dram_reg descriptors
+ * @count: Number of descriptors in @regs
+ *
+ * Writes each 16- or 32-bit register entry (expanding PHY_BCAST_DBYTE,
+ * PHY_BCAST_AC, and PHY_BCAST_9LANE broadcast tags across the corresponding
+ * PHY slices and lanes) and invokes mctl_sync_hardware_state() after each write.
+ *
+ * Return: 0 on success, or negative errno on sync timeout.
+ */
+static int mctl_write_regs(const struct sunxi_dram_reg *regs, u32 count)
+{
+ u32 i;
+ int ret;
+
+ if (!regs || !count)
+ return 0;
+
+ for (i = 0; i < count; i++) {
+ u32 raw_addr = regs[i].addr;
+ uintptr_t base = raw_addr & ~PHY_BCAST_MASK;
+ u32 val = regs[i].val;
+ unsigned int s, l;
+
+ if (raw_addr & PHY_BCAST_9LANE) {
+ for (s = 0; s < 4; s++) {
+ for (l = 0; l < 9; l++) {
+ ret = mctl_write_single_reg(base + s * PHY_SLICE_STRIDE +
+ l * PHY_LANE_STRIDE, val);
+ if (ret)
+ return ret;
+ }
+ }
+ } else if (raw_addr & PHY_BCAST_DBYTE) {
+ for (s = 0; s < 4; s++) {
+ ret = mctl_write_single_reg(base + s * PHY_SLICE_STRIDE, val);
+ if (ret)
+ return ret;
+ }
+ } else if (raw_addr & PHY_BCAST_AC) {
+ for (s = 0; s < 2; s++) {
+ ret = mctl_write_single_reg(base + s * PHY_SLICE_STRIDE, val);
+ if (ret)
+ return ret;
+ }
+ } else {
+ ret = mctl_write_single_reg(base, val);
+ if (ret)
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+/**
+ * mctl_write_regs_offset() - Replay an array of MMIO register writes with channel offset
+ * @regs: Pointer to array of struct sunxi_dram_reg descriptors
+ * @count: Number of descriptors in @regs
+ * @ch_offset: Address offset added to controller registers (0x0a100000..0x0a4fffff)
+ * @rank_mask: Active rank bitmask (0x03 for dual-rank, 0x01 for single-rank)
+ *
+ * Writes each register entry, adding @ch_offset to uMCTL2/port addresses while
+ * leaving global PHY/CCU addresses unchanged, and invokes
+ * mctl_sync_hardware_state() after each write.
+ *
+ * Return: 0 on success, or negative errno on sync timeout.
+ */
+static int mctl_write_regs_offset(const struct sunxi_dram_reg *regs, u32 count,
+ uintptr_t ch_offset, u8 rank_mask)
+{
+ u32 i;
+ int ret;
+
+ for (i = 0; i < count; i++) {
+ uintptr_t addr = regs[i].addr;
+ u32 val = regs[i].val;
+
+ if (addr == UMCTL2_CH0_BASE + UMCTL2_MSTR_OFFSET)
+ val = (val & ~(UMCTL2_MSTR_RANKS_MASK << UMCTL2_MSTR_RANKS_SHIFT)) |
+ ((u32)rank_mask << UMCTL2_MSTR_RANKS_SHIFT);
+ else if (addr == UMCTL2_CH0_BASE + UMCTL2_DERATEINT_OFFSET)
+ val = ((u32)rank_mask << 8) | rank_mask;
+
+ if (addr >= UMCTL2_CH0_APERTURE_BASE &&
+ addr < UMCTL2_CH0_APERTURE_END)
+ addr += ch_offset;
+
+ if (addr >= SUNXI_DRAM_PHY_APB_BASE &&
+ addr < SUNXI_DRAM_PHY_APB_END)
+ writew((u16)val, (void __iomem *)addr);
+ else
+ writel(val, (void __iomem *)addr);
+
+ ret = mctl_sync_hardware_state(addr, val);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+/**
+ * mctl_write_mr_all() - Issue a JEDEC Mode Register Write across both channels and ranks
+ * @mr_type: MRCTRL0 command type (0 for MRW, 1 for VRCG/FSP control)
+ * @mr_addr: Mode Register number (e.g. 14, 15, 16)
+ * @mr_val: 8-bit Mode Register payload
+ *
+ * Writes @mr_val to @mr_addr on active ranks of both Channel 0 and
+ * Channel 1 via uMCTL2 MRCTRL0/MRCTRL1 and polls for completion.
+ *
+ * Return: 0 on success, or negative errno on sync timeout.
+ */
+static int mctl_write_mr_all(u32 mr_type, u8 mr_addr, u8 mr_val)
+{
+ unsigned int ch, rank;
+ int ret;
+
+ for (ch = 0; ch < 2; ch++) {
+ uintptr_t base = (ch == 0) ? UMCTL2_CH0_BASE : UMCTL2_CH1_BASE;
+ uintptr_t mrctrl0 = base + UMCTL2_MRCTRL0_OFFSET;
+ uintptr_t mrctrl1 = base + UMCTL2_MRCTRL1_OFFSET;
+ u32 ch_mstr = readl((void __iomem *)(base + UMCTL2_MSTR_OFFSET));
+ u32 rank_mask = (ch_mstr >> UMCTL2_MSTR_RANKS_SHIFT) &
+ UMCTL2_MSTR_RANKS_MASK;
+
+ for (rank = 0; rank < 2; rank++) {
+ u32 ctrl0 = (BIT(rank) << 4) | (mr_type & 1U);
+ u32 ctrl1 = ((u32)mr_addr << 8) | mr_val;
+
+ if (!(rank_mask & BIT(rank)))
+ continue;
+
+ writel(ctrl0, (void __iomem *)mrctrl0);
+ writel(ctrl1, (void __iomem *)mrctrl1);
+ writel(ctrl0 | BIT(31), (void __iomem *)mrctrl0);
+ ret = mctl_sync_hardware_state(mrctrl0, ctrl0 | BIT(31));
+ if (ret)
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+/* ========================================================================= */
+/* SECTION 2: Synopsys DDR PHY & PMU Initializer */
+/* ========================================================================= */
+
+/**
+ * mctl_pmu_load_sram() - Load a 16-bit image into a 64 KiB PMU SRAM aperture
+ * @sram_base: Physical base address of the PMU IMEM or DMEM aperture
+ * @data: Pointer to array of 16-bit halfwords to load
+ * @count: Number of halfwords in @data
+ *
+ * Switches the PHY APB mux to host access and writes all 32,768 halfwords
+ * (64 KiB) of the target ICCM or DCCM SRAM to initialize hardware parity bits
+ * and zero any trailing words.
+ */
+static void mctl_pmu_load_sram(uintptr_t sram_base, const u16 *data,
+ size_t count)
+{
+ size_t i;
+
+ writew(0x0000,
+ (void __iomem *)(SUNXI_DRAM_PHY_CTRL_BASE + PMU_REG_APB_MUX));
+
+ for (i = 0; i < count; i++)
+ writew(data[i], (void __iomem *)(sram_base + i * 2));
+ for (; i < 0x8000; i++)
+ writew(0x0000, (void __iomem *)(sram_base + i * 2));
+}
+
+/**
+ * mctl_pmu_log_error() - Log PMU training failure diagnostics from DMEM
+ * @msg: Error summary prefix string
+ * @status: 32-bit PMU status word read from the mailbox registers
+ *
+ * Stalls the ARC core, switches the PHY APB mux back to host ARM access, and
+ * reads the active training mode and assertion code from DMEM.
+ */
+static void mctl_pmu_log_error(const char *msg, u32 status)
+{
+ uintptr_t ctrl = SUNXI_DRAM_PHY_CTRL_BASE;
+ uintptr_t dmem = SUNXI_DRAM_PHY_DMEM_BASE;
+ size_t assert_off = offsetof(struct pmu_message_block, assert_code);
+ size_t mode_off = offsetof(struct pmu_message_block, train_mode);
+ u32 assert_code;
+ u16 mode;
+
+ writew(0x0001, (void __iomem *)(ctrl + PMU_REG_RESET));
+ writew(0x0000, (void __iomem *)(ctrl + PMU_REG_APB_MUX));
+
+ assert_code = (u32)readw((void __iomem *)(dmem + assert_off)) |
+ ((u32)readw((void __iomem *)(dmem + assert_off + 2)) << 16);
+ mode = readw((void __iomem *)(dmem + mode_off));
+
+ pr_err("DRAM: %s, status = 0x%08x (mode=0x%04x, assert=0x%08x)\n",
+ msg, status, mode, assert_code);
+}
+
+/**
+ * mctl_pmu_run_training() - Hand over APB, trigger ARC EM4 PMU, and await completion
+ * @log_err: Log training failures at error level (false for speculative probes)
+ *
+ * Description: Switches the PHY APB mux to the PMU microsequencer, releases
+ * the ARC core from reset, and services the 4-phase Synopsys mailbox handshake
+ * across any intermediate streaming messages until terminal completion or
+ * fatal error.
+ *
+ * Return: 0 on successful training completion, or negative errno on failure.
+ */
+static int mctl_pmu_run_training(bool log_err)
+{
+ uintptr_t ctrl = SUNXI_DRAM_PHY_CTRL_BASE;
+ u32 status = 0;
+ int timeout;
+
+ /* Hand over APB to PMU Microsequencer and release ARC reset */
+ writew(0x0001, (void __iomem *)(ctrl + PMU_REG_APB_MUX));
+ writew(0x0009, (void __iomem *)(ctrl + PMU_REG_RESET));
+ writew(0x0001, (void __iomem *)(ctrl + PMU_REG_RESET));
+ writew(0x0001, (void __iomem *)(ctrl + PMU_REG_RESET));
+ writew(0x0000, (void __iomem *)(ctrl + PMU_REG_RESET));
+ writew(0x0001, (void __iomem *)(ctrl + PMU_REG_HANDOFF_1));
+ writew(0x0001, (void __iomem *)(ctrl + PMU_REG_MAILBOX_INT));
+
+ /* Wait for training to begin (bit 0 of PMU_REG_POLL_STAT == 1) */
+ for (timeout = 100000; timeout > 0; timeout--) {
+ if (readw((void __iomem *)(ctrl + PMU_REG_POLL_STAT)) & 0x01)
+ break;
+ udelay(1);
+ }
+ if (!timeout) {
+ if (log_err)
+ mctl_pmu_log_error("PMU training start timeout", 0);
+ else
+ goto err_stall;
+ return -ETIMEDOUT;
+ }
+
+ /* Service 4-phase PMU mailbox messages until terminal completion */
+ timeout = 5000000;
+ while (timeout > 0) {
+ if (!(readw((void __iomem *)(ctrl + PMU_REG_POLL_STAT)) & 0x01)) {
+ status = (u32)readw((void __iomem *)(ctrl + PMU_REG_STATUS_LO)) |
+ ((u32)readw((void __iomem *)(ctrl + PMU_REG_STATUS_HI)) << 16);
+
+ /* Acknowledge message (DctWriteProt = 0) and await release */
+ writew(0x0000, (void __iomem *)(ctrl + PMU_REG_HANDOFF_1));
+ while (!(readw((void __iomem *)(ctrl + PMU_REG_POLL_STAT)) & 0x01)) {
+ if (--timeout <= 0) {
+ if (log_err)
+ mctl_pmu_log_error("PMU mailbox ack timeout",
+ status);
+ else
+ goto err_stall;
+ return -ETIMEDOUT;
+ }
+ udelay(1);
+ }
+ writew(0x0001, (void __iomem *)(ctrl + PMU_REG_HANDOFF_1));
+
+ if (status == PMU_STATUS_SUCCESS) {
+ writew(0x0001, (void __iomem *)(ctrl + PMU_REG_RESET));
+ return 0;
+ }
+ if (status == PMU_STATUS_FATAL_ERROR) {
+ if (log_err)
+ mctl_pmu_log_error("PMU training failed",
+ status);
+ else
+ goto err_stall;
+ return -EIO;
+ }
+ debug("DRAM: PMU mailbox msg 0x%08x\n", status);
+ udelay(1);
+ timeout--;
+ continue;
+ }
+ udelay(1);
+ timeout--;
+ }
+
+ if (log_err)
+ mctl_pmu_log_error("PMU training completion timeout", 0);
+err_stall:
+ if (!log_err) {
+ writew(0x0001, (void __iomem *)(ctrl + PMU_REG_RESET));
+ writew(0x0000, (void __iomem *)(ctrl + PMU_REG_APB_MUX));
+ }
+ return (status == PMU_STATUS_FATAL_ERROR) ? -EIO : -ETIMEDOUT;
+}
+
+/**
+ * mctl_phy_dump_slices() - Dump trained PHY slice delay registers for debug
+ *
+ * Temporarily switches the PHY APB mux to host access, logs key trained delay
+ * registers across all four data slices, and restores the mux state.
+ */
+static void mctl_phy_dump_slices(void)
+{
+#ifdef DEBUG
+ uintptr_t ctrl = SUNXI_DRAM_PHY_CTRL_BASE;
+ unsigned int i;
+
+ writew(0x0000, (void __iomem *)(ctrl + PMU_REG_APB_MUX));
+ udelay(10);
+ for (i = 0; i < 4; i++) {
+ uintptr_t base = PHY_DBYTE0_BASE + (i * PHY_SLICE_STRIDE);
+
+ debug("Slice %u: CLK1E=0x%04x RxEn=0x%04x TxEn=0x%04x CompEn=0x%04x RxBias=0x%04x LCDL162=0x%04x STAT16A=0x%04x RFIFO=0x%04x\n",
+ i,
+ readw((void __iomem *)(base + PHY_DBYTE_CLK_GATE_OFF)),
+ readw((void __iomem *)(base + PHY_DBYTE_RX_EN_OFF)),
+ readw((void __iomem *)(base + PHY_DBYTE_TX_EN_OFF)),
+ readw((void __iomem *)(base + PHY_DBYTE_COMP_EN_OFF)),
+ readw((void __iomem *)(base + PHY_DBYTE_RX_BIAS_OFF)),
+ readw((void __iomem *)(base + PHY_DBYTE_LCDL_OFF)),
+ readw((void __iomem *)(base + PHY_DBYTE_STAT_OFF)),
+ readw((void __iomem *)(base + PHY_DBYTE_RFIFO_OFF)));
+ }
+ writew(0x0001, (void __iomem *)(ctrl + PMU_REG_APB_MUX));
+ udelay(10);
+#endif
+}
+
+/**
+ * mctl_umctl2_wait_normal_mode() - Wait for a uMCTL2 channel to enter Normal Mode
+ * @stat_reg: Physical address of the channel's UMCTL2 STAT register
+ * @ch: Channel index (0 or 1) for error reporting
+ *
+ * Return: 0 when operating_mode[1:0] == 0x01, or -ETIMEDOUT on timeout.
+ */
+static int mctl_umctl2_wait_normal_mode(uintptr_t stat_reg, unsigned int ch)
+{
+ int timeout;
+
+ for (timeout = 100000; timeout > 0; timeout--) {
+ if ((readl((void __iomem *)stat_reg) & 0x03U) == 0x01U)
+ return 0;
+ udelay(1);
+ }
+
+ pr_err("DRAM: Channel %u uMCTL2 failed to enter Normal Mode\n", ch);
+ return -ETIMEDOUT;
+}
+
+/**
+ * mctl_phy_load_pin_swizzle() - Program AC and DBYTE pin swizzle maps into the PHY
+ *
+ * Programs the board-specific CA0..CA6 pin permutation into AC slices 0..1
+ * and the DQ0..DQ7 pin permutation into DBYTE slices 0..3 prior to PMU
+ * training execution.
+ */
+static void mctl_phy_load_pin_swizzle(void)
+{
+ uintptr_t ctrl = SUNXI_DRAM_PHY_CTRL_BASE;
+ unsigned int ac, slice, pin;
+
+ if (!IS_ENABLED(CONFIG_DRAM_SUN60I_A733_PIN_SWIZZLE))
+ return;
+
+ writew(0x0001, (void __iomem *)(ctrl + PMU_REG_APB_MUX));
+ writew(0x0000, (void __iomem *)(ctrl + PMU_REG_APB_MUX));
+
+ for (ac = 0; ac < 2; ac++) {
+ uintptr_t ac_base = PHY_AC0_SWIZZLE_BASE +
+ ac * PHY_SLICE_STRIDE;
+
+ for (pin = 0; pin < 7; pin++)
+ writew(dram_ac_pin_swizzle[ac][pin],
+ (void __iomem *)(ac_base + pin * 2));
+ }
+
+ for (pin = 0; pin < 8; pin++) {
+ for (slice = 0; slice < 4; slice++) {
+ uintptr_t dq_addr = PHY_DBYTE0_DQ_SWIZZLE_BASE +
+ slice * PHY_SLICE_STRIDE + pin * 2;
+
+ writew(dram_dq_pin_swizzle[slice][pin],
+ (void __iomem *)dq_addr);
+ }
+ }
+
+ writew(0x0001, (void __iomem *)(ctrl + PMU_REG_APB_MUX));
+}
+
+/**
+ * mctl_load_pie_table() - Stream sparse non-zero entries into a PIE SRAM window
+ * @base: Physical base address of the target PIE SRAM aperture
+ * @total_words: Total number of 16-bit halfwords in the aperture
+ * @words: Ordered array of sparse non-zero (offset, value) entries
+ * @num_words: Number of entries in @words
+ *
+ * Writes @total_words consecutive 16-bit halfwords starting at @base,
+ * inserting non-zero values from @words at their matching byte offsets and
+ * zero-filling all other locations.
+ */
+static void mctl_load_pie_table(uintptr_t base, u32 total_words,
+ const struct sunxi_pie_word *words,
+ size_t num_words)
+{
+ size_t w = 0;
+ u32 i;
+
+ for (i = 0; i < total_words; i++) {
+ u16 byte_off = (u16)(i * 2);
+ u16 val = 0x0000;
+
+ if (w < num_words && words[w].off == byte_off) {
+ val = words[w].val;
+ w++;
+ }
+
+ writew(val, (void __iomem *)(base + byte_off));
+ }
+}
+
+/**
+ * mctl_apply_pie_config() - Program PIE sequencer, ACSM SRAM, and PHY CSRs
+ *
+ * Initializes the PHY Init Engine (PIE) instruction tables, streams the ACSM
+ * command sequence SRAM image, programs the FSP pointer tables and lane
+ * calibration registers, and triggers the post-training PIE state machine.
+ */
+static int mctl_apply_pie_config(void)
+{
+ unsigned int i, lane, reg;
+ int ret;
+
+ ret = mctl_write_regs(dram_pie_head_regs,
+ ARRAY_SIZE(dram_pie_head_regs));
+ if (ret)
+ return ret;
+ mctl_load_pie_table(PHY_PIE_INIT_SRAM_BASE, 176, dram_pie_init,
+ ARRAY_SIZE(dram_pie_init));
+ ret = mctl_write_regs(dram_pie_phy_setup_regs,
+ ARRAY_SIZE(dram_pie_phy_setup_regs));
+ if (ret)
+ return ret;
+ mctl_load_pie_table(PHY_PIE_ZERO_SRAM_BASE, 8, NULL, 0);
+ mctl_load_pie_table(PHY_PIE_SEQ_SRAM_BASE, 2256, dram_pie_seq,
+ ARRAY_SIZE(dram_pie_seq));
+
+ for (i = 0; i < ARRAY_SIZE(dram_acsm_sram); i++)
+ writew(dram_acsm_sram[i],
+ (void __iomem *)(PHY_ACSM_SRAM_BASE + i * 2));
+
+ ret = mctl_write_regs(dram_acsm_ctrl_regs,
+ ARRAY_SIZE(dram_acsm_ctrl_regs));
+ if (ret)
+ return ret;
+
+ for (i = 0; i < ARRAY_SIZE(dram_pie_ptrs) / 2; i++) {
+ writew(dram_pie_ptrs[i * 2],
+ (void __iomem *)(PHY_PIE_PTR_LO_BASE + i * 2));
+ writew(dram_pie_ptrs[i * 2 + 1],
+ (void __iomem *)(PHY_PIE_PTR_HI_BASE + i * 2));
+ }
+
+ ret = mctl_write_regs(dram_pie_patch_regs,
+ ARRAY_SIZE(dram_pie_patch_regs));
+ if (ret)
+ return ret;
+ mctl_load_pie_table(PHY_PIE_INIT_SRAM_BASE, 176, dram_pie_init,
+ ARRAY_SIZE(dram_pie_init));
+ ret = mctl_write_regs(dram_pie_pre_lane_regs,
+ ARRAY_SIZE(dram_pie_pre_lane_regs));
+ if (ret)
+ return ret;
+
+ for (lane = 0; lane < 9; lane++) {
+ uintptr_t lane_base = PHY_PIE_LANE_DELAY_BASE +
+ lane * PHY_LANE_STRIDE;
+
+ for (reg = 0; reg < 8; reg++)
+ writew(PHY_PIE_LANE_DELAY_INIT,
+ (void __iomem *)(lane_base + reg * 2));
+ }
+
+ return mctl_write_regs(dram_pie_tail_regs,
+ ARRAY_SIZE(dram_pie_tail_regs));
+}
+
+/**
+ * mctl_apply_post_pmu_config() - Apply post-PMU uMCTL2 timing, QoS, and MR handoff
+ * @rank_mask: Active rank bitmask (0x03 for dual-rank, 0x01 for single-rank)
+ *
+ * Programs the post-training uMCTL2 timing updates across Channel 0 and
+ * Channel 1, configures MBUS/RAXI QoS ports, and issues the final LPDDR5 Mode
+ * Register handoff sequence (MR16, MR14, MR15) across all ranks.
+ *
+ * Return: 0 on success, or negative errno on sync timeout.
+ */
+static int mctl_apply_post_pmu_config(u8 rank_mask)
+{
+ uintptr_t ctrl = SUNXI_DRAM_PHY_CTRL_BASE;
+ int ret;
+
+ ret = mctl_write_regs_offset(dram_post_pmu_ch_regs,
+ ARRAY_SIZE(dram_post_pmu_ch_regs),
+ 0x00000000UL, rank_mask);
+ if (ret)
+ return ret;
+
+ writew(0x0000, (void __iomem *)(ctrl + PMU_REG_APB_MUX));
+ writew(0x0003, (void __iomem *)PHY_PUB_POST_PMU_CFG_REG);
+ writew(0x0000, (void __iomem *)PHY_PUB_POST_PMU_CFG_REG);
+ writew(0x0001, (void __iomem *)(ctrl + PMU_REG_APB_MUX));
+
+ ret = mctl_write_regs_offset(dram_post_pmu_ch_regs,
+ ARRAY_SIZE(dram_post_pmu_ch_regs),
+ UMCTL2_CH_STRIDE, rank_mask);
+ if (ret)
+ return ret;
+
+ ret = mctl_write_regs(dram_post_pmu_tail_regs,
+ ARRAY_SIZE(dram_post_pmu_tail_regs));
+ if (ret)
+ return ret;
+
+ writew(0x0000, (void __iomem *)(ctrl + PMU_REG_APB_MUX));
+ ret = mctl_write_mr_all(0U, 16, 0x05);
+ if (ret)
+ return ret;
+ ret = mctl_write_mr_all(1U, 14, 0x00);
+ if (ret)
+ return ret;
+ ret = mctl_write_mr_all(1U, 15, 0x00);
+ if (ret)
+ return ret;
+ ret = mctl_write_mr_all(0U, 16, 0x04);
+ if (ret)
+ return ret;
+ writew(0x0001, (void __iomem *)(ctrl + PMU_REG_APB_MUX));
+
+ return 0;
+}
+
+/**
+ * mctl_ccu_init() - Configure CCU PLL_DDR, MBUS, DRAMC ports, and PRCM isolation
+ *
+ * Resets and ungates the DRAM0 and MBUS bus interfaces, locks PLL_DDR at
+ * 1800 MHz (3600 MT/s LPDDR5 WCK), enables dual-channel DRAMC AXI ports, and
+ * releases PRCM DRAM power isolation.
+ *
+ * Return: 0 on success, or negative errno on sync timeout.
+ */
+static int mctl_ccu_init(void)
+{
+ u32 pll_n = clamp((u32)CONFIG_DRAM_CLK / 24U, 1U, 256U) - 1U;
+ u32 n_bits = pll_n << 8;
+ int ret;
+
+ /* Reset MBUS and DRAM0 bus gates before reconfiguring PLL_DDR */
+ writel(0x08000000, (void __iomem *)CCU_DRAM_MUX_CLK_REG);
+ writel(0x00000000, (void __iomem *)CCU_MBUS_BGR_REG);
+ writel(0x00000000, (void __iomem *)CCU_DRAM0_BGR_REG);
+ writel(0x00010000, (void __iomem *)CCU_DRAM0_BGR_REG);
+ writel(0x00010001, (void __iomem *)CCU_DRAM0_BGR_REG);
+
+ writel(0x00000100, (void __iomem *)DRAMC_SYS_CFG_REG);
+ writel(0x00002600, (void __iomem *)DRAMC_SYS_CFG_REG);
+
+ /* Program PLL_DDR (24 MHz * (N + 1)), poll lock, then ungate */
+ writel(0x10000000, (void __iomem *)CCU_PLL_DDR_CTRL_REG);
+ writel(0x08000000, (void __iomem *)CCU_DRAM0_CLK_REG);
+ writel(0x11000000U | n_bits, (void __iomem *)CCU_PLL_DDR_CTRL_REG);
+ writel(0xd1000000U | n_bits, (void __iomem *)CCU_PLL_DDR_CTRL_REG);
+ ret = mctl_write_single_reg(CCU_PLL_DDR_CTRL_REG, 0xf1000000U | n_bits);
+ if (ret)
+ return ret;
+ writel(0xf9000000U | n_bits, (void __iomem *)CCU_PLL_DDR_CTRL_REG);
+
+ /* Enable DRAM0 and MBUS clocks */
+ writel(0x08000003, (void __iomem *)CCU_DRAM0_CLK_REG);
+ writel(0x89000000, (void __iomem *)CCU_DRAM_MUX_CLK_REG);
+ writel(0x00010000, (void __iomem *)CCU_MBUS_BGR_REG);
+ writel(0x00010001, (void __iomem *)CCU_MBUS_BGR_REG);
+ writel(0xcb000000, (void __iomem *)CCU_MBUS_CLK_REG);
+ writel(0x88000003, (void __iomem *)CCU_DRAM0_CLK_REG);
+
+ /* Enable dual-channel DRAMC and AXI host ports */
+ writel(0x00002728, (void __iomem *)DRAMC_SYS_CFG_REG);
+ writel(0x00000003, (void __iomem *)DRAMC_CH_EN_REG);
+ writel(0x00000103, (void __iomem *)DRAMC_CH0_PORT_CFG_REG);
+ writel(0x000013ff, (void __iomem *)DRAMC_CH0_PORT_EN_REG);
+ writel(0x00000103, (void __iomem *)DRAMC_CH1_PORT_CFG_REG);
+ writel(0x000013ff, (void __iomem *)DRAMC_CH1_PORT_EN_REG);
+ writel(0x00000107, (void __iomem *)DRAMC_CH0_PORT_CFG_REG);
+ writel(0x00000107, (void __iomem *)DRAMC_CH1_PORT_CFG_REG);
+
+ /* Release PRCM DRAM power isolation */
+ return mctl_write_single_reg(PRCM_MEM_ISOLATION_REG, 0x00000001);
+}
+
+/**
+ * mctl_pre_pmu_init() - Program CCU PLL_DDR, dual-channel uMCTL2, and PHY init
+ * @rank_mask: Active rank bitmask (0x03 for dual-rank, 0x01 for single-rank)
+ *
+ * Configures the CCU DDR PLL and MBUS clocks, replicates the uMCTL2 controller
+ * timing and QoS configuration across Channel 0 (0x0a100000) and Channel 1
+ * (0x0a500000), and programs the pre-PMU Synopsys PHY registers.
+ *
+ * Return: 0 on success, or negative errno on sync timeout.
+ */
+static int mctl_pre_pmu_init(u8 rank_mask)
+{
+ int ret;
+
+ ret = mctl_ccu_init();
+ if (ret)
+ return ret;
+ ret = mctl_write_regs_offset(dram_umctl2_ch_regs,
+ ARRAY_SIZE(dram_umctl2_ch_regs),
+ 0x00000000UL, rank_mask);
+ if (ret)
+ return ret;
+ ret = mctl_write_regs_offset(dram_umctl2_ch_regs,
+ ARRAY_SIZE(dram_umctl2_ch_regs),
+ UMCTL2_CH_STRIDE, rank_mask);
+ if (ret)
+ return ret;
+ return mctl_write_regs(dram_phy_init_regs, ARRAY_SIZE(dram_phy_init_regs));
+}
+
+/**
+ * mctl_phy_init() - Initialize the Synopsys DDR PHY and run ARC EM4 PMU training
+ * @rank_mask: Active rank bitmask (0x03 for dual-rank, 0x01 for single-rank)
+ *
+ * Description: Programs pre-training controller and PHY registers, loads the
+ * ARC EM4 PMU firmware into IMEM and training parameter block into DMEM, runs
+ * the PMU microsequencer training handshake, applies post-training PIE and
+ * uMCTL2 configuration, and waits for both channels to enter Normal Mode.
+ *
+ * Return: 0 on successful training and normal mode entry, or negative errno.
+ */
+static int mctl_phy_init(u8 rank_mask, bool log_err)
+{
+ struct pmu_message_block msg = dram_pmu_msg_block;
+ int ret;
+
+ BUILD_BUG_ON(sizeof(msg) != 0x106);
+ BUILD_BUG_ON(ARRAY_SIZE(sun60i_a733_pmu_imem) > 0x8000);
+ BUILD_BUG_ON(CONFIG_DRAM_CLK % 24 != 0);
+
+ msg.dram_rate_mts = clamp((u32)CONFIG_DRAM_CLK / 24U, 1U, 256U) * 48U;
+ msg.ch0_rank_en = rank_mask;
+ msg.ch1_rank_en = rank_mask;
+ if (!IS_ENABLED(CONFIG_DRAM_SUN60I_A733_PIN_SWIZZLE))
+ msg.dq_swap_en = 0;
+
+ ret = mctl_pre_pmu_init(rank_mask);
+ if (ret)
+ return ret;
+
+ mctl_pmu_load_sram(SUNXI_DRAM_PHY_IMEM_BASE, sun60i_a733_pmu_imem,
+ ARRAY_SIZE(sun60i_a733_pmu_imem));
+
+ ret = mctl_write_regs(dram_phy_pre_dmem_regs,
+ ARRAY_SIZE(dram_phy_pre_dmem_regs));
+ if (ret)
+ return ret;
+ mctl_pmu_load_sram(SUNXI_DRAM_PHY_DMEM_BASE,
+ (const u16 *)&msg,
+ sizeof(msg) / sizeof(u16));
+
+ mctl_phy_load_pin_swizzle();
+
+ ret = mctl_pmu_run_training(log_err);
+ if (ret)
+ return ret;
+
+ writew(0x0000,
+ (void __iomem *)(SUNXI_DRAM_PHY_CTRL_BASE + PMU_REG_APB_MUX));
+ ret = mctl_apply_pie_config();
+ if (ret)
+ return ret;
+ ret = mctl_apply_post_pmu_config(rank_mask);
+ if (ret)
+ return ret;
+ mctl_phy_dump_slices();
+
+ ret = mctl_umctl2_wait_normal_mode(SUNXI_UMCTL2_CH0_STAT, 0);
+ if (ret)
+ return ret;
+
+ return mctl_umctl2_wait_normal_mode(SUNXI_UMCTL2_CH1_STAT, 1);
+}
+
+/* ========================================================================= */
+/* SECTION 3: Memory Diagnostics & Dynamic Capacity Probing */
+/* ========================================================================= */
+
+/**
+ * mctl_read_mr() - Read a JEDEC Mode Register from an active LPDDR5 rank
+ * @ch: Controller channel index (0 or 1)
+ * @rank: Rank index within @ch (0 or 1)
+ * @mr_addr: Mode Register number (e.g. 5 for Manufacturer ID, 8 for Density/IO)
+ * @val: Pointer to store 8-bit Mode Register value
+ *
+ * Issues a Mode Register Read (MRR) command via the channel's uMCTL2 MRCTRL0
+ * and MRCTRL1 registers, waits for the MR transaction bit (bit 31) to clear,
+ * and extracts the 8-bit MR response from the DRAMC MR data register.
+ *
+ * Return: 0 on success, or -ETIMEDOUT on poll timeout.
+ */
+static int mctl_read_mr(unsigned int ch, unsigned int rank, u8 mr_addr, u8 *val)
+{
+ uintptr_t base = (ch == 0) ? UMCTL2_CH0_BASE : UMCTL2_CH1_BASE;
+ uintptr_t mr_data_reg = (ch == 0) ? DRAMC_MR_DATA_CH0_REG :
+ DRAMC_MR_DATA_CH1_REG;
+ u32 rank_sel = BIT(rank) << 4;
+ u32 ctrl0 = rank_sel | 0x01U;
+ int ret;
+
+ writel(ctrl0, (void __iomem *)(base + UMCTL2_MRCTRL0_OFFSET));
+ writel((u32)mr_addr << 8,
+ (void __iomem *)(base + UMCTL2_MRCTRL1_OFFSET));
+ writel(ctrl0 | BIT(31),
+ (void __iomem *)(base + UMCTL2_MRCTRL0_OFFSET));
+ ret = mctl_poll_reg32(base + UMCTL2_MRCTRL0_OFFSET, BIT(31), 0U, 10000,
+ "MR read", base + UMCTL2_MRCTRL0_OFFSET);
+ if (ret)
+ return ret;
+
+ udelay(10);
+
+ *val = (u8)((readl((void __iomem *)mr_data_reg) >> 8) & 0xffU);
+ return 0;
+}
+
+/**
+ * mctl_mr8_rank_size_mb() - Decode JEDEC LPDDR5 MR8 into rank capacity in MiB
+ * @mr8: Raw 8-bit JEDEC Mode Register 8 (JESD209-5) value
+ *
+ * Decodes the die density code (bits [5:2]) and I/O width (bits [7:6]) from
+ * MR8. Handles both power-of-two dies and non-power-of-two 3/4-density dies
+ * (such as 6 Gb, 12 Gb, and 24 Gb) as well as x8 byte-mode ranks where two
+ * dies populate a 16-bit channel.
+ *
+ * Return: Rank capacity in MiB, or 0 if @mr8 is invalid.
+ */
+static unsigned long mctl_mr8_rank_size_mb(u8 mr8)
+{
+ u8 type = mr8 & 0x03U;
+ u8 density_code = (mr8 >> 2) & 0x0fU;
+ u8 io_width = (mr8 >> 6) & 0x03U;
+ unsigned long die_mb;
+
+ if (mr8 == 0x00U || mr8 == 0xffU || type > 2U || io_width > 1U ||
+ density_code > 8U)
+ return 0;
+
+ die_mb = ((density_code & 1U) ? 384UL : 256UL) <<
+ (density_code >> 1);
+
+ /* In x8 byte mode (io_width == 1), two x8 dies form one 16-bit rank */
+ if (io_width == 1U)
+ die_mb <<= 1;
+
+ return die_mb;
+}
+
+/**
+ * sun60i_a733_get_dram_size() - Dynamically detect total physical DRAM capacity
+ *
+ * Description: Queries the active channel mask and uMCTL2 MSTR register to
+ * determine the active LPDDR5 ranks, then reads JEDEC Mode Register 8 (MR8)
+ * from each populated channel and rank to compute exact capacity (including
+ * non-power-of-two 3/4-density dies such as 6 Gb and 12 Gb). Falls back to
+ * U-Boot's standard get_ram_size() address-aliasing probe if MR8 probing
+ * yields 0.
+ *
+ * Return: Total detected DRAM capacity in bytes.
+ */
+static unsigned long sun60i_a733_get_dram_size(void)
+{
+ u32 ch0_mstr = readl((void __iomem *)(UMCTL2_CH0_BASE + UMCTL2_MSTR_OFFSET));
+ u32 ch_mask = readl((void __iomem *)DRAMC_CH_EN_REG) & 0x03U;
+
+ if (ch0_mstr & UMCTL2_MSTR_LPDDR5) {
+ unsigned long total_mb = 0;
+ unsigned int ch, rank;
+
+ if (!ch_mask)
+ ch_mask = 0x01U;
+
+ for (ch = 0; ch < 2; ch++) {
+ uintptr_t ch_base = (ch == 0) ? UMCTL2_CH0_BASE : UMCTL2_CH1_BASE;
+ u32 mstr = readl((void __iomem *)(ch_base + UMCTL2_MSTR_OFFSET));
+ u32 rank_mask = (mstr >> UMCTL2_MSTR_RANKS_SHIFT) &
+ UMCTL2_MSTR_RANKS_MASK;
+
+ if (!(ch_mask & BIT(ch)))
+ continue;
+
+ for (rank = 0; rank < 2; rank++) {
+ u8 mr8;
+
+ if (!(rank_mask & BIT(rank)))
+ continue;
+
+ if (mctl_read_mr(ch, rank, 8, &mr8) < 0)
+ continue;
+
+ total_mb += mctl_mr8_rank_size_mb(mr8);
+ }
+ }
+
+#ifdef DEBUG
+ {
+ u8 mr5 = 0, mr8_r0 = 0;
+
+ mctl_read_mr(0, 0, 5, &mr5);
+ mctl_read_mr(0, 0, 8, &mr8_r0);
+ debug("DRAM: MR5 mfg_id=0x%02x, MR8 rank0=0x%02x -> %lu MiB\n",
+ mr5, mr8_r0, total_mb);
+ }
+#endif
+
+ if (total_mb > 0)
+ return total_mb << 20;
+ }
+
+ return get_ram_size((long *)CFG_SYS_SDRAM_BASE, SZ_4G);
+}
+
+#ifdef DEBUG
+/**
+ * a733_test_channel_aperture() - Verify a 16 KiB test window on one DRAM channel
+ * @base: Physical base address of the channel test aperture
+ * @seed: 32-bit pattern seed added to the word index
+ * @label: Channel label ("Ch0" or "Ch1") for error reporting
+ *
+ * Return: Number of word readback mismatches detected.
+ */
+static unsigned int a733_test_channel_aperture(uintptr_t base, u32 seed,
+ const char *label)
+{
+ unsigned int errors = 0;
+ unsigned int i;
+
+ for (i = 0; i < A733_DRAM_TEST_WORDS; i++) {
+ u32 exp = seed + i;
+ u32 act = readl((void __iomem *)(base + (i * 4)));
+
+ if (act != exp) {
+ errors++;
+ if (errors <= 5)
+ printf(" %s mismatch @ 0x%08lx: exp 0x%08x, act 0x%08x\n",
+ label, (unsigned long)(base + (i * 4)),
+ exp, act);
+ }
+ }
+
+ return errors;
+}
+
+/**
+ * a733_dump_and_test_dram() - Run post-training dual-channel DRAM verification
+ * @dram_size: Total detected physical DRAM capacity in bytes
+ * @phase: Diagnostic phase label for debug/error logging
+ *
+ * Executes a sparse address probe across key physical apertures followed by
+ * a 16 KiB dual-channel pattern write and readback verification across
+ * Channel 0 (0x40000000) and Channel 1 (0xc0000000, if populated).
+ */
+static void a733_dump_and_test_dram(unsigned long dram_size, const char *phase)
+{
+ unsigned int probe_errors = 0;
+ unsigned int ch0_errors = 0, ch1_errors = 0;
+ unsigned int i;
+
+ debug("DRAM test [%s]: Starting memory diagnostics...\n",
+ phase ? phase : "unspecified");
+
+ /* 1. Single-Word Sparse Probe Test across target addresses within bounds */
+ for (i = 0; i < ARRAY_SIZE(sparse_probe_addrs); i++) {
+ uintptr_t addr = sparse_probe_addrs[i];
+ u32 pat = 0xa5a50000U | ((u32)(addr >> 16) & 0xffffU);
+ u32 val;
+
+ if (addr >= CFG_SYS_SDRAM_BASE + dram_size)
+ continue;
+
+ writel(pat, (void __iomem *)addr);
+ dsb();
+ isb();
+
+ val = readl((void __iomem *)addr);
+ if (val != pat) {
+ probe_errors++;
+ printf(" Probe [0x%08lx]: FAIL (exp 0x%08x, act 0x%08x)\n",
+ (unsigned long)addr, pat, val);
+ } else {
+ debug(" Probe [0x%08lx]: PASS (0x%08x)\n",
+ (unsigned long)addr, val);
+ }
+ }
+
+ /* 2. Dual-Aperture Pattern Test (16 KiB Ch0 + 16 KiB Ch1 if in bounds) */
+ for (i = 0; i < A733_DRAM_TEST_WORDS; i++) {
+ writel(0x01234567U + i,
+ (void __iomem *)(A733_DRAM_CH0_BASE + (i * 4)));
+ if (CFG_SYS_SDRAM_BASE + dram_size > A733_DRAM_CH1_BASE + (i * 4)) {
+ writel(0xfedcba98U + i,
+ (void __iomem *)(A733_DRAM_CH1_BASE + (i * 4)));
+ }
+ }
+
+ dsb();
+ isb();
+
+ ch0_errors = a733_test_channel_aperture(A733_DRAM_CH0_BASE,
+ 0x01234567U, "Ch0");
+ if (CFG_SYS_SDRAM_BASE + dram_size > A733_DRAM_CH1_BASE) {
+ ch1_errors = a733_test_channel_aperture(A733_DRAM_CH1_BASE,
+ 0xfedcba98U, "Ch1");
+ }
+
+ if (probe_errors || ch0_errors || ch1_errors) {
+ printf("DRAM test [%s]: Probe errors=%u, Ch0 16KB errors=%u, Ch1 16KB errors=%u -> FAIL\n",
+ phase ? phase : "unspecified",
+ probe_errors, ch0_errors, ch1_errors);
+ } else {
+ debug("DRAM test [%s]: Probe errors=0, Ch0 16KB errors=0, Ch1 16KB errors=0 -> PASS\n",
+ phase ? phase : "unspecified");
+ }
+}
+#else
+static inline void a733_dump_and_test_dram(unsigned long dram_size __maybe_unused,
+ const char *phase __maybe_unused)
+{
+}
+#endif
+
+/* ========================================================================= */
+/* SECTION 4: DMC Clocks & Master DRAM Initialization */
+/* ========================================================================= */
+
+/**
+ * sun60i_a733_dmc_clock_setup() - Configure PRCM DMC channel clocks and resets
+ *
+ * Configures dual-channel mode, sets the PRCM DMC feature registers for
+ * LPDDR5 operation, and sequences reset deassertion and clock gating for
+ * DRAM Channels 0 and 1.
+ */
+static void sun60i_a733_dmc_clock_setup(void)
+{
+ void __iomem *ch0_clk = (void __iomem *)SUNXI_PRCM_CH0_DMC_CLK_REG;
+ void __iomem *ch1_clk = (void __iomem *)SUNXI_PRCM_CH1_DMC_CLK_REG;
+ void __iomem *ch0_feat = (void __iomem *)SUNXI_PRCM_CH0_DMC_FEAT_REG;
+ void __iomem *ch1_feat = (void __iomem *)SUNXI_PRCM_CH1_DMC_FEAT_REG;
+
+ /* Step 1: Configure dual-channel mode */
+ clrsetbits_le32(ch0_clk, 0x3f3fU, 0x2728U);
+ writel(readl(ch0_clk), ch1_clk);
+
+ /* Step 2: Configure DRAM memory type */
+ clrsetbits_le32(ch0_feat, 0x03U, 0x03U);
+ writel(readl(ch0_feat), ch1_feat);
+
+ /* Step 3: Pulse channel resets and enable clock */
+ clrsetbits_le32(ch0_clk, 0x13U, 0x14U);
+ writel(readl(ch0_clk), ch1_clk);
+ udelay(1);
+
+ clrsetbits_le32(ch0_clk, 0x01U, 0x02U);
+ writel(readl(ch0_clk), ch1_clk);
+ udelay(1);
+
+ setbits_le32(ch0_clk, 0x03U);
+ writel(readl(ch0_clk), ch1_clk);
+ udelay(1);
+}
+
+/**
+ * sunxi_dram_init - Top-level DRAM subsystem initialization and bringup
+ *
+ * Return: Total operational DRAM capacity in bytes (hangs on training failure).
+ */
+unsigned long sunxi_dram_init(void)
+{
+ unsigned long dram_size;
+ int ret;
+
+ /* Allow LPDDR5 VDD1/VDD2/VDDQ rails configured by SPL PMIC init to settle */
+ mdelay(10);
+
+ /* PRCM DMC Channel Clock Setup */
+ sun60i_a733_dmc_clock_setup();
+
+ /* Try dual-rank training first, falling back to single-rank */
+ ret = mctl_phy_init(0x03, false);
+ if (ret) {
+ debug("DRAM: Dual-rank training failed (%d), retrying single-rank\n",
+ ret);
+ sun60i_a733_dmc_clock_setup();
+ ret = mctl_phy_init(0x01, true);
+ }
+ if (ret) {
+ pr_err("DRAM: Synopsys PHY training failed (%d)\n", ret);
+ hang();
+ }
+
+ dram_size = sun60i_a733_get_dram_size();
+
+ /* Optional Debug Diagnostics */
+ a733_dump_and_test_dram(dram_size, "Normal DRAM Read/Write Test");
+
+ return dram_size;
+}
+#endif /* !SUN60I_A733_PMU_FW_MISSING */
+#else /* !CONFIG_XPL_BUILD */
+/**
+ * sun60i_a733_dram_type_name() - Read active DRAM generation name from uMCTL2
+ *
+ * Description: Inspects the active DRAM type bit in the Channel 0 uMCTL2 MSTR
+ * register and returns a human-readable string.
+ *
+ * Return: Human-readable DRAM generation string.
+ */
+static const char *sun60i_a733_dram_type_name(void)
+{
+ u32 mstr = readl((void __iomem *)(UMCTL2_CH0_BASE + UMCTL2_MSTR_OFFSET));
+
+ if (mstr & UMCTL2_MSTR_LPDDR5)
+ return "LPDDR5";
+
+ return "Unknown";
+}
+
+/**
+ * board_add_ram_info() - Report trained DRAM type and effective data rate
+ * @use_default: Unused default-configuration flag
+ */
+void board_add_ram_info(int use_default __maybe_unused)
+{
+ u32 pll = readl((void __iomem *)CCU_PLL_DDR_CTRL_REG);
+ u32 n = ((pll >> 8) & 0xffU) + 1U;
+ u32 m_out = ((pll >> 20) & 0x07U) + 1U;
+ u32 p_in = ((pll >> 1) & 0x01U) + 1U;
+ u32 pll_freq_mhz = (24U * n) / (p_in * m_out);
+ u32 dram_clk_reg = readl((void __iomem *)CCU_DRAM0_CLK_REG);
+ u32 m_dram = (dram_clk_reg & 0x1fU) + 1U;
+ u32 ck_mhz = pll_freq_mhz / m_dram;
+ u32 rate_mts = ck_mhz * 8U;
+
+ printf(" (%s, %u MT/s)",
+ sun60i_a733_dram_type_name(),
+ rate_mts);
+}
+#endif /* CONFIG_XPL_BUILD */
diff --git a/arch/arm/mach-sunxi/dram_sun60i_a733_tables.h b/arch/arm/mach-sunxi/dram_sun60i_a733_tables.h
new file mode 100644
index 00000000000..8dfb33f63f3
--- /dev/null
+++ b/arch/arm/mach-sunxi/dram_sun60i_a733_tables.h
@@ -0,0 +1,941 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Allwinner A733 LPDDR5 uMCTL2, Synopsys PHY, and PIE initialization tables
+ *
+ * (C) Copyright 2026 Alastair D'Silva <alastair@d-silva.org>
+ */
+
+#ifndef _DRAM_SUN60I_A733_TABLES_H
+#define _DRAM_SUN60I_A733_TABLES_H
+
+#include <linux/compiler_attributes.h>
+
+/**
+ * struct sunxi_dram_reg - DRAM initialization MMIO register write entry
+ * @addr: Target physical MMIO address
+ * @val: Value to write (16-bit for DDR PHY APB 0x0a900000..0x0aafffff, 32-bit otherwise)
+ */
+struct sunxi_dram_reg {
+ u32 addr;
+ u32 val;
+};
+
+static const struct sunxi_dram_reg dram_umctl2_ch_regs[] __section(".init.rodata") = {
+ { 0x0a110380, 0x80013414 },
+ { 0x0a100c00, 0x000000ff },
+ { 0x0a200c00, 0x000000ff },
+ { 0x0a300c00, 0x000000ff },
+ { 0x0a400c00, 0x000000ff },
+ { 0x0a110b84, 0x00000001 },
+ { 0x0a110184, 0x00000002 },
+ { 0x0a110000, 0x03081008 },
+ { 0x0a110c9c, 0x00000303 },
+ { 0x0a110010, 0x00000001 },
+ { 0x0a11018c, 0x00000000 },
+ { 0x0a110180, 0x00000800 },
+ { 0x0a110508, 0xc0000000 },
+ { 0x0a110280, 0x80000000 },
+ { 0x0a110288, 0x00000001 },
+ { 0x0a110300, 0x00c000c0 },
+ { 0x0a110308, 0x00000001 },
+ { 0x0a110500, 0x00000000 },
+ { 0x0a110518, 0x25000000 },
+ { 0x0a110208, 0x00000001 },
+ { 0x0a110510, 0x00010015 },
+ { 0x0a130018, 0x02020200 },
+ { 0x0a130014, 0x1f020202 },
+ { 0x0a130010, 0x00000101 },
+ { 0x0a13000c, 0x003f0909 },
+ { 0x0a13002c, 0x00000808 },
+ { 0x0a130028, 0x08080808 },
+ { 0x0a130024, 0x08080808 },
+ { 0x0a130020, 0x08080808 },
+ { 0x0a13001c, 0x1f1f1f08 },
+ { 0x0a130004, 0x00003f17 },
+ { 0x0a110220, 0x0a090e01 },
+ { 0x0a110224, 0x00000000 },
+ { 0x0a100000, 0x1f090f13 },
+ { 0x0a100004, 0x0004041d },
+ { 0x0a100008, 0x070d1011 },
+ { 0x0a10000c, 0x000a1b24 },
+ { 0x0a100010, 0x0904030a },
+ { 0x0a100014, 0x02030706 },
+ { 0x0a100018, 0x00000006 },
+ { 0x0a10001c, 0x00000002 },
+ { 0x0a100024, 0x0002040d },
+ { 0x0a100030, 0x00030000 },
+ { 0x0a100034, 0x0a100002 },
+ { 0x0a100038, 0x001200af },
+ { 0x0a10005c, 0x00180006 },
+ { 0x0a100060, 0x000e120c },
+ { 0x0a100064, 0x00001f04 },
+ { 0x0a100078, 0x00556262 },
+ { 0x0a100080, 0x00000000 },
+ { 0x0a100d00, 0x00000001 },
+ { 0x0a110d00, 0xc00100db },
+ { 0x0a110510, 0x00010014 },
+ { 0x0a100d04, 0x0000090b },
+ { 0x0a100d08, 0x00001221 },
+ { 0x0a100d0c, 0x00400005 },
+ { 0x0a1005a0, 0x00020202 },
+ { 0x0a1005a4, 0x00000201 },
+ { 0x0a1005a8, 0x001c0003 },
+ { 0x0a1005ac, 0x005800a0 },
+ { 0x0a1005b0, 0x00000097 },
+ { 0x0a1005b8, 0x00000147 },
+ { 0x0a100580, 0x032f0217 },
+ { 0x0a100584, 0x00070303 },
+ { 0x0a100588, 0x00182f17 },
+ { 0x0a100590, 0x10080411 },
+ { 0x0a100594, 0x040c000f },
+ { 0x0a100600, 0x00000036 },
+ { 0x0a100604, 0x00ab00ab },
+ { 0x0a100608, 0x04290000 },
+ { 0x0a100650, 0x00000075 },
+ { 0x0a100800, 0x000e0a96 },
+ { 0x0a100804, 0x01700100 },
+ { 0x0a100808, 0x0000000e },
+ { 0x0a100a80, 0x00001f74 },
+ { 0x0a100b00, 0x07c0c3e2 },
+ { 0x0a100b04, 0x1024100a },
+ { 0x0a100b08, 0x00000033 },
+ { 0x0a100b80, 0x00170000 },
+ { 0x0a110b84, 0x00000000 },
+ { 0x0a110c80, 0x00000000 },
+ { 0x0a110510, 0x00010004 },
+ { 0x0a110c80, 0x00000001 },
+ { 0x0a110c80, 0x00000000 },
+ { 0x0a110510, 0x00010014 },
+ { 0x0a110c80, 0x00000001 },
+};
+
+static const struct sunxi_dram_reg dram_phy_init_regs[] __section(".init.rodata") = {
+ { 0x0a020010, 0x0000273c },
+ { 0x0a020010, 0x0000273e },
+ { 0x0a020010, 0x0000273f },
+ { 0x0a940094, 0x0402 },
+ { 0x0a940096, 0x123e },
+ { 0x0a940110, 0x0101 },
+ { 0x0a9400c0, 0x0002 },
+ { 0x0a94014a, 0x0001 },
+ { 0x0aaa006c, 0x0000 },
+ { 0x0a94017a, 0x00ff },
+ { 0x0a940600, 0x0808 },
+ { 0x0a940606, 0x0009 },
+ { 0x0a940604, 0x0026 },
+ { 0x0a940650, 0x0000 },
+ { 0x0a940602, 0x0003 },
+ { 0x0a940616, 0x0000 },
+ { 0x0a96015c, 0x0080 },
+ { 0x0a96015a, 0x0080 },
+ { 0x0a960158, 0x0080 },
+ { 0x0a96215c, 0x0080 },
+ { 0x0a96215a, 0x0080 },
+ { 0x0a962158, 0x0080 },
+ { 0x0aa8010c, 0x0000 },
+ { 0x0a920146, 0x0b3f },
+ { 0x0a922146, 0x0b33 },
+ { 0x0a924146, 0x0b3f },
+ { 0x0a926146, 0x0b33 },
+ { 0x0aa801e2, 0x6000 },
+ { 0x0aa801e4, 0x00a9 },
+ { 0x0aa801e6, 0x8000 },
+ { 0x0aa801e8, 0x0005 },
+ { 0x0aa801ea, 0x4000 },
+ { 0x0aa801ec, 0x0002 },
+ { 0x0aa801ee, 0xf000 },
+ { 0x0aa801f2, 0x6000 },
+ { 0x0aa801f4, 0x00a9 },
+ { 0x0aa801f6, 0x8000 },
+ { 0x0aa801f8, 0x0000 },
+ { 0x0aa801fa, 0x4000 },
+ { 0x0aa801fc, 0x0000 },
+ { 0x0aa801fe, 0xf000 },
+ { 0x0aa211fe, 0x000f },
+ { 0x0a920126, 0x0000 },
+ { 0x0a922126, 0x0000 },
+ { 0x0a924126, 0x0000 },
+ { 0x0a926126, 0x0000 },
+ { 0x0a9400a2, 0x0003 },
+};
+
+/*
+ * Address Tag Encoding for mctl_write_regs():
+ * Bit 31 (PHY_BCAST_DBYTE): Replicate entry across DBYTE slices 0..3 (+ s * 0x2000)
+ * Bit 30 (PHY_BCAST_AC): Replicate entry across AC slices 0..1 (+ ac * 0x2000)
+ * Bit 29 (PHY_BCAST_9LANE): Replicate entry across DBYTE 0..3 and lanes 0..8 (+ l * 0x200)
+ */
+#define PHY_BCAST_DBYTE BIT(31)
+#define PHY_BCAST_AC BIT(30)
+#define PHY_BCAST_9LANE BIT(29)
+#define PHY_BCAST_MASK (PHY_BCAST_DBYTE | PHY_BCAST_AC | PHY_BCAST_9LANE)
+
+static const struct sunxi_dram_reg dram_phy_pre_dmem_regs[] __section(".init.rodata") = {
+ { 0x0aa21004, 0x0001 },
+ { 0x0a940098, 0x2a8b },
+ { 0x0a94009a, 0x0000 },
+ { 0x0a940004, 0x0002 },
+ { 0x0a940000, 0x0002 },
+ { 0x0aa21002, 0x002c },
+ { 0x0aa21012, 0x6800 },
+ { PHY_BCAST_DBYTE | 0x0a9200bc, 0x0366 },
+ { PHY_BCAST_DBYTE | 0x0a9200c0, 0x0366 },
+ { PHY_BCAST_DBYTE | 0x0a9200c4, 0x0366 },
+ { PHY_BCAST_DBYTE | 0x0a9200c8, 0x0366 },
+ { PHY_BCAST_DBYTE | 0x0a9200be, 0x0366 },
+ { PHY_BCAST_DBYTE | 0x0a9200c2, 0x0366 },
+ { PHY_BCAST_DBYTE | 0x0a9200c6, 0x0366 },
+ { PHY_BCAST_DBYTE | 0x0a9200ca, 0x0366 },
+ { PHY_BCAST_DBYTE | 0x0a92000a, 0x0000 },
+ { PHY_BCAST_DBYTE | 0x0a920016, 0x0000 },
+ { 0x0a94000e, 0x0000 },
+ { 0x0a940026, 0x002c },
+ { PHY_BCAST_AC | 0x0a960070, 0x0003 },
+ { PHY_BCAST_AC | 0x0a960072, 0x0003 },
+ { PHY_BCAST_AC | 0x0a960074, 0x0003 },
+ { PHY_BCAST_DBYTE | 0x0a920070, 0x0003 },
+ { PHY_BCAST_DBYTE | 0x0a920074, 0x0003 },
+ { PHY_BCAST_DBYTE | 0x0a920076, 0x0003 },
+ { PHY_BCAST_DBYTE | 0x0a920012, 0x0000 },
+ { PHY_BCAST_DBYTE | 0x0a92006e, 0x0000 },
+ { PHY_BCAST_DBYTE | 0x0a920008, 0x0000 },
+ { PHY_BCAST_DBYTE | 0x0a920006, 0x0000 },
+ { 0x0a940008, 0x01c2 },
+ { PHY_BCAST_AC | 0x0a9600a0, 0x0035 },
+ { PHY_BCAST_AC | 0x0a9600a2, 0x0035 },
+ { PHY_BCAST_AC | 0x0a9600a4, 0x0035 },
+ { PHY_BCAST_AC | 0x0a9600a6, 0x0035 },
+ { PHY_BCAST_9LANE | 0x0a92009c, 0x0035 },
+ { PHY_BCAST_9LANE | 0x0a92009e, 0x0035 },
+ { PHY_BCAST_9LANE | 0x0a9200a0, 0x0035 },
+ { PHY_BCAST_9LANE | 0x0a9200a2, 0x0035 },
+ { PHY_BCAST_AC | 0x0a960060, 0x0000 },
+ { PHY_BCAST_AC | 0x0a960062, 0x0000 },
+ { PHY_BCAST_AC | 0x0a96006a, 0x0000 },
+ { PHY_BCAST_DBYTE | 0x0a920060, 0x0000 },
+ { PHY_BCAST_DBYTE | 0x0a92006a, 0x0000 },
+ { PHY_BCAST_DBYTE | 0x0a92006c, 0x0000 },
+ { PHY_BCAST_AC | 0x0a960078, 0x0005 },
+ { PHY_BCAST_DBYTE | 0x0a920078, 0x0005 },
+ { PHY_BCAST_DBYTE | 0x0a92007a, 0x0005 },
+ { PHY_BCAST_DBYTE | 0x0a92007c, 0x0005 },
+ { 0x0a940006, 0x0001 },
+ { 0x0a940002, 0x1122 },
+ { 0x0a940012, 0x0000 },
+ { 0x0a940010, 0x0000 },
+ { 0x0a9401b2, 0x0001 },
+ { 0x0a940028, 0x1301 },
+ { 0x0aa21014, 0x1300 },
+ { PHY_BCAST_DBYTE | 0x0a920080, 0x0f0f },
+ { PHY_BCAST_DBYTE | 0x0a920084, 0x0f0f },
+ { 0x0aa2101a, 0x0000 },
+ { PHY_BCAST_DBYTE | 0x0a920086, 0x0000 },
+ { PHY_BCAST_DBYTE | 0x0a920088, 0x0f0f },
+ { PHY_BCAST_DBYTE | 0x0a92008a, 0x0f0f },
+ { PHY_BCAST_AC | 0x0a960080, 0x0f0f },
+ { PHY_BCAST_AC | 0x0a960082, 0x0f0f },
+ { PHY_BCAST_AC | 0x0a960084, 0x0e0e },
+ { PHY_BCAST_AC | 0x0a960086, 0x0e0e },
+ { PHY_BCAST_AC | 0x0a960660, 0x0033 },
+ { 0x0a940662, 0x0033 },
+ { PHY_BCAST_DBYTE | 0x0a920090, 0x0e00 },
+ { PHY_BCAST_DBYTE | 0x0a920094, 0x0e00 },
+ { PHY_BCAST_DBYTE | 0x0a920096, 0x0e00 },
+ { PHY_BCAST_DBYTE | 0x0a920098, 0x0800 },
+ { PHY_BCAST_DBYTE | 0x0a92009a, 0x0800 },
+ { PHY_BCAST_AC | 0x0a960090, 0x0c00 },
+ { PHY_BCAST_AC | 0x0a960092, 0x0c00 },
+ { PHY_BCAST_AC | 0x0a960094, 0x0e00 },
+ { PHY_BCAST_AC | 0x0a960096, 0x0e00 },
+ { PHY_BCAST_AC | 0x0a960066, 0x0000 },
+ { PHY_BCAST_AC | 0x0a960068, 0x0000 },
+ { PHY_BCAST_AC | 0x0a96005c, 0x0000 },
+ { PHY_BCAST_DBYTE | 0x0a920066, 0x0000 },
+ { PHY_BCAST_DBYTE | 0x0a92005c, 0x0000 },
+ { PHY_BCAST_DBYTE | 0x0a92005e, 0x0000 },
+ { 0x0a94002a, 0x0000 },
+ { 0x0aa2100c, 0x0000 },
+ { PHY_BCAST_DBYTE | 0x0a9201d0, 0x0000 },
+ { PHY_BCAST_DBYTE | 0x0a9201d2, 0x0035 },
+ { PHY_BCAST_DBYTE | 0x0a920002, 0x0000 },
+ { 0x0a940024, 0xa1a1 },
+ { 0x0a94002e, 0x0040 },
+ { 0x0a940014, 0x0040 },
+ { 0x0a94030c, 0x00a1 },
+ { 0x0a94030e, 0x0040 },
+ { 0x0a940020, 0x0050 },
+ { 0x0a940022, 0x000f },
+ { 0x0aa21016, 0x000f },
+ { 0x0aa21018, 0x000f },
+ { PHY_BCAST_DBYTE | 0x0a92014a, 0x0001 },
+ { PHY_BCAST_DBYTE | 0x0a920028, 0x3232 },
+ { PHY_BCAST_DBYTE | 0x0a92001e, 0x0986 },
+ { 0x0a94006a, 0x0c08 },
+ { 0x0a94006c, 0x0c08 },
+ { 0x0a94006e, 0x0414 },
+ { 0x0a940070, 0x1918 },
+ { 0x0a940072, 0x0c10 },
+ { 0x0a940074, 0x0c10 },
+ { 0x0a940076, 0x041c },
+ { 0x0a940078, 0x2920 },
+ { 0x0a94007a, 0x0c04 },
+ { 0x0a94007c, 0x0c04 },
+ { 0x0a94007e, 0x0410 },
+ { 0x0a940080, 0x1114 },
+ { 0x0a940058, 0x082f },
+ { 0x0a94005a, 0x082f },
+ { 0x0a940060, 0x082f },
+ { 0x0a94005c, 0x0817 },
+ { 0x0a94005e, 0x0817 },
+ { 0x0a940018, 0x0000 },
+ { 0x0a940036, 0x0000 },
+ { PHY_BCAST_DBYTE | 0x0a92000e, 0x0000 },
+ { 0x0aa80100, 0x0003 },
+ { 0x0a940094, 0x0402 },
+ { 0x0a940096, 0x123e },
+ { 0x0a940098, 0x2a8b },
+ { 0x0a110c80, 0x00000000 },
+ { 0x0a110510, 0x00010014 },
+ { 0x0a110c80, 0x00000001 },
+ { 0x0a510c80, 0x00000000 },
+ { 0x0a510510, 0x00010014 },
+ { 0x0a510c80, 0x00000001 },
+ { 0x02002c00, 0x88000003 },
+ { 0x0aaa0000, 0x0000 },
+};
+
+static const struct pmu_message_block dram_pmu_msg_block __section(".init.rodata") = {
+ .pstate_flags = 0x30,
+ .dram_rate_mts = 3600,
+ .pll_ratio = 4,
+ .dfi_freq_ratio = 4,
+ .sequence_ctrl = PMU_SEQ_FULL_TRAIN,
+ .post_threshold = 0xff,
+ .cal_ctrl = 0x0002,
+ .ch0_dq_width = 16,
+ .ch0_rank_en = 0x03,
+ .ch1_dq_width = 16,
+ .ch1_rank_en = 0x03,
+ .mr1 = { 0x80, 0x80, 0x80, 0x80 },
+ .mr2 = { 0x88, 0x88, 0x88, 0x88 },
+ .mr3 = { 0x06, 0x06, 0x06, 0x06 },
+ .mr10 = { 0x54, 0x54, 0x54, 0x54 },
+ .mr11 = { 0x13, 0x13, 0x13, 0x13 },
+ .mr12 = { 0x44, 0x44, 0x44, 0x44 },
+ .mr14 = { 0x2e, 0x2e, 0x2e, 0x2e },
+ .mr15 = { 0x2e, 0x2e, 0x2e, 0x2e },
+ .mr17 = { 0x06, 0x2e, 0x06, 0x2e },
+ .mr18 = { 0x1c, 0x1c, 0x1c, 0x1c },
+ .mr19 = { 0x10, 0x10, 0x10, 0x10 },
+ .mr20 = { 0x02, 0x02, 0x02, 0x02 },
+ .mr28 = { 0x04, 0x04, 0x04, 0x04 },
+ .mr41 = { 0x60, 0x60, 0x60, 0x60 },
+ .pll_gate_ch = { 0x002e, 0x0050 },
+ .dq_swap_en = 0x01,
+ .lane_err_mask = 0x0f,
+ .ca_vref_rank1 = 0x33,
+ .ca_vref_rank0 = 0x2f,
+};
+
+/*
+ * CA0..CA6 and DQ0..DQ7 lane permutations used on the Orange Pi 4 Pro PCB routing.
+ */
+static const u8 dram_ac_pin_swizzle[2][7] __section(".init.rodata") = {
+ { 3, 6, 4, 0, 2, 5, 1 },
+ { 3, 0, 6, 4, 2, 5, 1 },
+};
+
+static const u8 dram_dq_pin_swizzle[4][8] __section(".init.rodata") = {
+ { 4, 3, 5, 7, 0, 1, 6, 2 },
+ { 3, 2, 1, 0, 5, 7, 6, 4 },
+ { 1, 4, 5, 2, 7, 6, 0, 3 },
+ { 6, 7, 3, 0, 4, 5, 1, 2 },
+};
+
+/**
+ * struct sunxi_pie_word - Sparse non-zero 16-bit PIE SRAM entry
+ * @off: Byte offset from the target PIE SRAM base address
+ * @val: Non-zero 16-bit value to write at @base + @off
+ */
+struct sunxi_pie_word {
+ u16 off;
+ u16 val;
+};
+
+static const struct sunxi_dram_reg dram_pie_head_regs[] __section(".init.rodata") = {
+ { 0x0aaa0000, 0x0000 },
+ { 0x0aa211e0, 0x0039 },
+ { 0x0aa211e2, 0x00a8 },
+ { 0x0aa211e4, 0x0000 },
+ { 0x0aa211e6, 0x0058 },
+ { 0x0aa211e8, 0x000b },
+ { 0x0aa211ea, 0x0000 },
+ { 0x0aa211ec, 0x00b0 },
+ { 0x0aa211ee, 0x0000 },
+};
+
+static const struct sunxi_pie_word dram_pie_init[] __section(".init.rodata") = {
+ { 0x0000, 0xc9d8 }, { 0x0008, 0xc808 }, { 0x0016, 0x2b00 }, { 0x0020, 0xc958 },
+ { 0x0028, 0xce08 }, { 0x0036, 0x2b00 }, { 0x0040, 0xc0d8 }, { 0x0048, 0xc048 },
+ { 0x0056, 0x2b00 }, { 0x0060, 0xc158 }, { 0x0068, 0xc448 }, { 0x0076, 0x2b00 },
+ { 0x0080, 0xc1d8 }, { 0x0088, 0xc308 }, { 0x0096, 0x2b00 }, { 0x00a0, 0xc558 },
+ { 0x00a8, 0xea08 }, { 0x00b6, 0x2b00 }, { 0x00c0, 0xc5d8 }, { 0x00c8, 0xc108 },
+ { 0x00d6, 0x2b00 }, { 0x00e0, 0x48d8 }, { 0x00e8, 0x4308 }, { 0x00f0, 0x88d8 },
+ { 0x00f8, 0x9708 }, { 0x0106, 0x2b00 }, { 0x0110, 0xca58 }, { 0x0118, 0xc108 },
+ { 0x0126, 0x2b00 }, { 0x0130, 0xcb58 }, { 0x0138, 0xc008 }, { 0x0146, 0x2b00 },
+ { 0x0150, 0xd4d8 }, { 0x0158, 0xf008 },
+};
+
+static const struct sunxi_dram_reg dram_pie_phy_setup_regs[] __section(".init.rodata") = {
+ { 0x0a94002a, 0x0000 },
+ { PHY_BCAST_AC | 0x0a960094, 0x0000 },
+ { PHY_BCAST_AC | 0x0a960096, 0x0000 },
+ { 0x0a940098, 0x080f },
+ { 0x0a94009a, 0x0000 },
+ { 0x0a940082, 0x0001 },
+ { 0x0aa21004, 0x0001 },
+ { 0x0a94008a, 0x0002 },
+ { PHY_BCAST_DBYTE | 0x0a9200ae, 0x0000 },
+ { PHY_BCAST_DBYTE | 0x0a92001e, 0x0988 },
+ { 0x0aa21006, 0x0004 },
+ { 0x0aa21008, 0x0082 },
+ { 0x0aa2100a, 0x0001 },
+ { 0x0aa21206, 0x0000 },
+ { 0x0aa21216, 0x0002 },
+ { 0x0aa21022, 0x0000 },
+ { 0x0aa21024, 0xff00 },
+ { 0x0a9400e4, 0x0003 },
+ { 0x0aa2101c, 0x0003 },
+ { 0x0a9400e6, 0x0003 },
+ { 0x0aa2101e, 0x0003 },
+ { 0x0a940094, 0x0e3e },
+ { 0x0a940096, 0x0e3e },
+ { 0x0a940098, 0x280b },
+ { 0x0aaa0000, 0x0000 },
+};
+
+static const struct sunxi_pie_word dram_pie_seq[] __section(".init.rodata") = {
+ { 0x0000, 0xc028 }, { 0x0006, 0x0010 }, { 0x0016, 0x0400 }, { 0x0030, 0xc858 },
+ { 0x0036, 0x0010 }, { 0x0038, 0xe088 }, { 0x003e, 0x0010 }, { 0x0040, 0xe038 },
+ { 0x0046, 0x0010 }, { 0x0048, 0xc858 }, { 0x004e, 0x0010 }, { 0x0050, 0xc088 },
+ { 0x0056, 0x0010 }, { 0x0060, 0xc028 }, { 0x0066, 0x0010 }, { 0x0076, 0x0400 },
+ { 0x0090, 0xc858 }, { 0x0096, 0x0010 }, { 0x0098, 0xe208 }, { 0x009e, 0x0010 },
+ { 0x00a0, 0xe038 }, { 0x00a6, 0x0010 }, { 0x00a8, 0xc858 }, { 0x00ae, 0x0010 },
+ { 0x00b0, 0xc208 }, { 0x00b6, 0x0010 }, { 0x00c0, 0xce58 }, { 0x00c6, 0x0010 },
+ { 0x00c8, 0xc208 }, { 0x00ce, 0x0010 }, { 0x00e0, 0xc370 }, { 0x00e6, 0x0010 },
+ { 0x00f0, 0xc040 }, { 0x00f6, 0x0010 }, { 0x00fe, 0x0010 }, { 0x0100, 0xd2d8 },
+ { 0x0106, 0x0010 }, { 0x0108, 0xe008 }, { 0x010e, 0x0010 }, { 0x0116, 0x5b00 },
+ { 0x0120, 0xc0f0 }, { 0x0126, 0x0010 }, { 0x0130, 0xcfd8 }, { 0x0136, 0x0010 },
+ { 0x0138, 0xc008 }, { 0x013e, 0x0010 }, { 0x0146, 0x2b00 }, { 0x0150, 0xd058 },
+ { 0x0156, 0x0010 }, { 0x0158, 0xc008 }, { 0x015e, 0x0010 }, { 0x0166, 0x2b00 },
+ { 0x0170, 0xd0d8 }, { 0x0176, 0x0010 }, { 0x0178, 0xc088 }, { 0x017e, 0x0010 },
+ { 0x0186, 0x2b00 }, { 0x0190, 0xd158 }, { 0x0196, 0x0010 }, { 0x0198, 0xc008 },
+ { 0x019e, 0x0010 }, { 0x01a6, 0x5b00 }, { 0x01b0, 0x402c }, { 0x01b2, 0x0d00 },
+ { 0x01b4, 0x0001 }, { 0x01b6, 0x0400 }, { 0x01b8, 0x4050 }, { 0x01ba, 0x0800 },
+ { 0x01c6, 0x0400 }, { 0x01d6, 0x0400 }, { 0x01d8, 0x4050 }, { 0x01da, 0x0803 },
+ { 0x01e6, 0x7f00 }, { 0x01ee, 0x0400 }, { 0x01f0, 0x407c }, { 0x01f6, 0x0400 },
+ { 0x0206, 0x0400 }, { 0x020c, 0x0001 }, { 0x0216, 0x0400 }, { 0x0230, 0x802c },
+ { 0x0232, 0x0d00 }, { 0x0234, 0x0001 }, { 0x0236, 0x0410 }, { 0x0238, 0x8050 },
+ { 0x023a, 0x0800 }, { 0x023e, 0x0010 }, { 0x0246, 0x0400 }, { 0x0256, 0x0400 },
+ { 0x0258, 0x8050 }, { 0x025a, 0x0803 }, { 0x025e, 0x0010 }, { 0x0266, 0x7f00 },
+ { 0x026e, 0x0400 }, { 0x0270, 0x807c }, { 0x0276, 0x0410 }, { 0x0286, 0x0400 },
+ { 0x028c, 0x0001 }, { 0x0296, 0x0400 }, { 0x02b0, 0x402c }, { 0x02b2, 0x0d00 },
+ { 0x02b4, 0x0001 }, { 0x02b8, 0x4050 }, { 0x02ba, 0x0800 }, { 0x02d8, 0x4050 },
+ { 0x02da, 0x0803 }, { 0x02f8, 0x4050 }, { 0x02fa, 0x0800 }, { 0x0306, 0x6b00 },
+ { 0x030e, 0x0400 }, { 0x0318, 0x407c }, { 0x0334, 0x0001 }, { 0x0346, 0x4b00 },
+ { 0x034e, 0x1800 }, { 0x0358, 0x802c }, { 0x035a, 0x0d00 }, { 0x035c, 0x0001 },
+ { 0x035e, 0x0010 }, { 0x0360, 0x8050 }, { 0x0362, 0x0800 }, { 0x0366, 0x0010 },
+ { 0x0380, 0x8050 }, { 0x0382, 0x0803 }, { 0x0386, 0x0010 }, { 0x03a0, 0x8050 },
+ { 0x03a2, 0x0800 }, { 0x03a6, 0x0010 }, { 0x03b6, 0x6b00 }, { 0x03be, 0x0400 },
+ { 0x03c0, 0x807c }, { 0x03c6, 0x0010 }, { 0x03dc, 0x0001 }, { 0x03e6, 0x7b00 },
+ { 0x03ee, 0x2000 }, { 0x03f0, 0x402c }, { 0x03f2, 0x0d00 }, { 0x03f4, 0x0001 },
+ { 0x03f8, 0x5198 }, { 0x03fa, 0x0803 }, { 0x0406, 0x3b00 }, { 0x0410, 0x5218 },
+ { 0x0412, 0x0803 }, { 0x0426, 0x6b00 }, { 0x042e, 0x0400 }, { 0x0430, 0x407c },
+ { 0x044c, 0x0001 }, { 0x0456, 0x1b00 }, { 0x0468, 0x802c }, { 0x046a, 0x0d00 },
+ { 0x046c, 0x0001 }, { 0x046e, 0x0010 }, { 0x0470, 0x9198 }, { 0x0472, 0x0803 },
+ { 0x0476, 0x0010 }, { 0x0486, 0x2b00 }, { 0x0498, 0x9218 }, { 0x049a, 0x0803 },
+ { 0x049e, 0x0010 }, { 0x04a6, 0x6b00 }, { 0x04ae, 0x0400 }, { 0x04b8, 0x807c },
+ { 0x04be, 0x0010 }, { 0x04d4, 0x0001 }, { 0x04e6, 0x0b00 }, { 0x04ee, 0x0400 },
+ { 0x0500, 0x401c }, { 0x0502, 0x00d0 }, { 0x0504, 0x0001 }, { 0x0508, 0x4060 },
+ { 0x050a, 0x0084 }, { 0x0516, 0x0b00 }, { 0x051e, 0x0800 }, { 0x0520, 0x407c },
+ { 0x0538, 0x4000 }, { 0x053c, 0x0001 }, { 0x0548, 0x402c }, { 0x054a, 0x0d00 },
+ { 0x054c, 0x0001 }, { 0x0550, 0x4020 }, { 0x0552, 0x0803 }, { 0x0566, 0x2b00 },
+ { 0x056e, 0x0800 }, { 0x0578, 0x407c }, { 0x0594, 0x0001 }, { 0x05a0, 0x801c },
+ { 0x05a2, 0x00d0 }, { 0x05a4, 0x0001 }, { 0x05a6, 0x0010 }, { 0x05a8, 0x8060 },
+ { 0x05aa, 0x0084 }, { 0x05ae, 0x0010 }, { 0x05b6, 0x0b00 }, { 0x05be, 0x0800 },
+ { 0x05c0, 0x807c }, { 0x05c6, 0x0010 }, { 0x05d8, 0x8000 }, { 0x05dc, 0x0001 },
+ { 0x05de, 0x0010 }, { 0x05e8, 0x802c }, { 0x05ea, 0x0d00 }, { 0x05ec, 0x0001 },
+ { 0x05ee, 0x0010 }, { 0x05f0, 0x8020 }, { 0x05f2, 0x0803 }, { 0x05f6, 0x0010 },
+ { 0x0606, 0x2b00 }, { 0x060e, 0x0800 }, { 0x0618, 0x807c }, { 0x061e, 0x0010 },
+ { 0x0634, 0x0001 }, { 0x0640, 0x402c }, { 0x0642, 0x0d00 }, { 0x0644, 0x0001 },
+ { 0x0648, 0x4050 }, { 0x064a, 0x0803 }, { 0x0656, 0x1b00 }, { 0x0670, 0x4050 },
+ { 0x0672, 0x0803 }, { 0x0676, 0x5a00 }, { 0x0696, 0x0100 }, { 0x06a0, 0x4050 },
+ { 0x06a2, 0x0803 }, { 0x06b6, 0x6b00 }, { 0x06be, 0x0400 }, { 0x06c0, 0x407c },
+ { 0x06d6, 0x3b00 }, { 0x06ec, 0x0001 }, { 0x06f0, 0x5758 }, { 0x06f8, 0x4208 },
+ { 0x0706, 0x4b00 }, { 0x0718, 0x401c }, { 0x071a, 0x00d0 }, { 0x071c, 0x0001 },
+ { 0x0720, 0x4060 }, { 0x0722, 0x0084 }, { 0x0736, 0x7b00 }, { 0x0740, 0x407c },
+ { 0x0758, 0x4000 }, { 0x075c, 0x0001 }, { 0x0768, 0x402c }, { 0x076a, 0x0d00 },
+ { 0x076c, 0x0001 }, { 0x0770, 0x4020 }, { 0x0772, 0x0803 }, { 0x0786, 0x2b00 },
+ { 0x078e, 0x0800 }, { 0x0798, 0x407c }, { 0x07b0, 0x4000 }, { 0x07b4, 0x0001 },
+ { 0x07c0, 0x5758 }, { 0x07c8, 0x4008 }, { 0x07e0, 0x802c }, { 0x07e2, 0x0d00 },
+ { 0x07e4, 0x0001 }, { 0x07e6, 0x0010 }, { 0x07e8, 0x8050 }, { 0x07ea, 0x0803 },
+ { 0x07ee, 0x0010 }, { 0x07f6, 0x1b00 }, { 0x0810, 0x8050 }, { 0x0812, 0x0803 },
+ { 0x0816, 0x5a10 }, { 0x0836, 0x0100 }, { 0x0840, 0x8050 }, { 0x0842, 0x0803 },
+ { 0x0846, 0x0010 }, { 0x0856, 0x6b00 }, { 0x085e, 0x0400 }, { 0x0860, 0x807c },
+ { 0x0866, 0x0010 }, { 0x0876, 0x3b00 }, { 0x088c, 0x0001 }, { 0x0890, 0x9758 },
+ { 0x0896, 0x0010 }, { 0x0898, 0x8208 }, { 0x089e, 0x0010 }, { 0x08a6, 0x4b00 },
+ { 0x08b8, 0x801c }, { 0x08ba, 0x00d0 }, { 0x08bc, 0x0001 }, { 0x08be, 0x0010 },
+ { 0x08c0, 0x8060 }, { 0x08c2, 0x0084 }, { 0x08c6, 0x0010 }, { 0x08d6, 0x7b00 },
+ { 0x08e0, 0x807c }, { 0x08e6, 0x0010 }, { 0x08f8, 0x8000 }, { 0x08fc, 0x0001 },
+ { 0x08fe, 0x0010 }, { 0x0908, 0x802c }, { 0x090a, 0x0d00 }, { 0x090c, 0x0001 },
+ { 0x090e, 0x0010 }, { 0x0910, 0x8020 }, { 0x0912, 0x0803 }, { 0x0916, 0x0010 },
+ { 0x0926, 0x2b00 }, { 0x092e, 0x0800 }, { 0x0938, 0x807c }, { 0x093e, 0x0010 },
+ { 0x0950, 0x8000 }, { 0x0954, 0x0001 }, { 0x0956, 0x0010 }, { 0x0960, 0x9758 },
+ { 0x0966, 0x0010 }, { 0x0968, 0x8008 }, { 0x096e, 0x0010 }, { 0x0980, 0xc068 },
+ { 0x0986, 0x0010 }, { 0x0990, 0xd2d8 }, { 0x0996, 0x0010 }, { 0x0998, 0xe008 },
+ { 0x099e, 0x0010 }, { 0x09a6, 0x5b00 }, { 0x09b0, 0xc0f0 }, { 0x09b6, 0x0010 },
+ { 0x09c0, 0xcfd8 }, { 0x09c6, 0x0010 }, { 0x09c8, 0xc008 }, { 0x09ce, 0x0010 },
+ { 0x09d6, 0x2b00 }, { 0x09e0, 0xd058 }, { 0x09e6, 0x0010 }, { 0x09e8, 0xc008 },
+ { 0x09ee, 0x0010 }, { 0x09f6, 0x2b00 }, { 0x0a00, 0xd0d8 }, { 0x0a06, 0x0010 },
+ { 0x0a08, 0xc088 }, { 0x0a0e, 0x0010 }, { 0x0a16, 0x2b00 }, { 0x0a20, 0xd158 },
+ { 0x0a26, 0x0010 }, { 0x0a28, 0xc008 }, { 0x0a2e, 0x0010 }, { 0x0a36, 0x5b00 },
+ { 0x0a40, 0x402c }, { 0x0a42, 0x0d00 }, { 0x0a44, 0x0001 }, { 0x0a46, 0x0400 },
+ { 0x0a48, 0x4050 }, { 0x0a4a, 0x0800 }, { 0x0a56, 0x0400 }, { 0x0a58, 0x4050 },
+ { 0x0a5a, 0x0803 }, { 0x0a66, 0x6f00 }, { 0x0a6e, 0x0400 }, { 0x0a70, 0x407c },
+ { 0x0a76, 0x0400 }, { 0x0a86, 0x0400 }, { 0x0a8c, 0x0001 }, { 0x0a96, 0x0400 },
+ { 0x0ab0, 0x802c }, { 0x0ab2, 0x0d00 }, { 0x0ab4, 0x0001 }, { 0x0ab6, 0x0410 },
+ { 0x0ab8, 0x8050 }, { 0x0aba, 0x0800 }, { 0x0abe, 0x0010 }, { 0x0ac6, 0x0400 },
+ { 0x0ac8, 0x8050 }, { 0x0aca, 0x0803 }, { 0x0ace, 0x0010 }, { 0x0ad6, 0x6f00 },
+ { 0x0ade, 0x0400 }, { 0x0ae0, 0x807c }, { 0x0ae6, 0x0410 }, { 0x0af6, 0x0400 },
+ { 0x0afc, 0x0001 }, { 0x0b06, 0x0400 }, { 0x0b20, 0x402c }, { 0x0b22, 0x0d00 },
+ { 0x0b24, 0x0001 }, { 0x0b28, 0x4050 }, { 0x0b2a, 0x0800 }, { 0x0b38, 0x4050 },
+ { 0x0b3a, 0x0803 }, { 0x0b48, 0x4050 }, { 0x0b4a, 0x0800 }, { 0x0b56, 0x4b00 },
+ { 0x0b5e, 0x0400 }, { 0x0b68, 0x407c }, { 0x0b84, 0x0001 }, { 0x0b96, 0x4b00 },
+ { 0x0b9e, 0x1800 }, { 0x0ba8, 0x802c }, { 0x0baa, 0x0d00 }, { 0x0bac, 0x0001 },
+ { 0x0bae, 0x0010 }, { 0x0bb0, 0x8050 }, { 0x0bb2, 0x0800 }, { 0x0bb6, 0x0010 },
+ { 0x0bc0, 0x8050 }, { 0x0bc2, 0x0803 }, { 0x0bc6, 0x0010 }, { 0x0bd0, 0x8050 },
+ { 0x0bd2, 0x0800 }, { 0x0bd6, 0x0010 }, { 0x0be6, 0x4b00 }, { 0x0bee, 0x0400 },
+ { 0x0bf0, 0x807c }, { 0x0bf6, 0x0010 }, { 0x0c0c, 0x0001 }, { 0x0c16, 0x5b00 },
+ { 0x0c1e, 0x2000 }, { 0x0c20, 0x402c }, { 0x0c22, 0x0d00 }, { 0x0c24, 0x0001 },
+ { 0x0c28, 0x5198 }, { 0x0c2a, 0x0803 }, { 0x0c36, 0x1b00 }, { 0x0c40, 0x5218 },
+ { 0x0c42, 0x0803 }, { 0x0c56, 0x4b00 }, { 0x0c5e, 0x0400 }, { 0x0c60, 0x407c },
+ { 0x0c7c, 0x0001 }, { 0x0c86, 0x1b00 }, { 0x0c98, 0x802c }, { 0x0c9a, 0x0d00 },
+ { 0x0c9c, 0x0001 }, { 0x0c9e, 0x0010 }, { 0x0ca0, 0x9198 }, { 0x0ca2, 0x0803 },
+ { 0x0ca6, 0x0010 }, { 0x0cc8, 0x9218 }, { 0x0cca, 0x0803 }, { 0x0cce, 0x0010 },
+ { 0x0cd6, 0x4b00 }, { 0x0cde, 0x0400 }, { 0x0ce8, 0x807c }, { 0x0cee, 0x0010 },
+ { 0x0d04, 0x0001 }, { 0x0d16, 0x0b00 }, { 0x0d1e, 0x0400 }, { 0x0d30, 0x401c },
+ { 0x0d32, 0x00d0 }, { 0x0d34, 0x0001 }, { 0x0d38, 0x4060 }, { 0x0d3a, 0x0084 },
+ { 0x0d46, 0x0b00 }, { 0x0d4e, 0x0800 }, { 0x0d50, 0x407c }, { 0x0d68, 0x4000 },
+ { 0x0d6c, 0x0001 }, { 0x0d78, 0x402c }, { 0x0d7a, 0x0d00 }, { 0x0d7c, 0x0001 },
+ { 0x0d80, 0x4020 }, { 0x0d82, 0x0803 }, { 0x0d96, 0x2b00 }, { 0x0d9e, 0x0800 },
+ { 0x0da8, 0x407c }, { 0x0dc4, 0x0001 }, { 0x0dd0, 0x801c }, { 0x0dd2, 0x00d0 },
+ { 0x0dd4, 0x0001 }, { 0x0dd6, 0x0010 }, { 0x0dd8, 0x8060 }, { 0x0dda, 0x0084 },
+ { 0x0dde, 0x0010 }, { 0x0de6, 0x0b00 }, { 0x0dee, 0x0800 }, { 0x0df0, 0x807c },
+ { 0x0df6, 0x0010 }, { 0x0e08, 0x8000 }, { 0x0e0c, 0x0001 }, { 0x0e0e, 0x0010 },
+ { 0x0e18, 0x802c }, { 0x0e1a, 0x0d00 }, { 0x0e1c, 0x0001 }, { 0x0e1e, 0x0010 },
+ { 0x0e20, 0x8020 }, { 0x0e22, 0x0803 }, { 0x0e26, 0x0010 }, { 0x0e36, 0x2b00 },
+ { 0x0e3e, 0x0800 }, { 0x0e48, 0x807c }, { 0x0e4e, 0x0010 }, { 0x0e64, 0x0001 },
+ { 0x0e70, 0x402c }, { 0x0e72, 0x0d00 }, { 0x0e74, 0x0001 }, { 0x0e78, 0x4050 },
+ { 0x0e7a, 0x0803 }, { 0x0ea0, 0x4050 }, { 0x0ea2, 0x0803 }, { 0x0ea6, 0x5a00 },
+ { 0x0eb6, 0x0100 }, { 0x0ec0, 0x4050 }, { 0x0ec2, 0x0803 }, { 0x0ed6, 0x4b00 },
+ { 0x0ede, 0x0400 }, { 0x0ee0, 0x407c }, { 0x0ef6, 0x3b00 }, { 0x0f0c, 0x0001 },
+ { 0x0f10, 0x5758 }, { 0x0f18, 0x4208 }, { 0x0f26, 0x4b00 }, { 0x0f38, 0x401c },
+ { 0x0f3a, 0x00d0 }, { 0x0f3c, 0x0001 }, { 0x0f40, 0x4060 }, { 0x0f42, 0x0084 },
+ { 0x0f56, 0x7b00 }, { 0x0f60, 0x407c }, { 0x0f78, 0x4000 }, { 0x0f7c, 0x0001 },
+ { 0x0f88, 0x402c }, { 0x0f8a, 0x0d00 }, { 0x0f8c, 0x0001 }, { 0x0f90, 0x4020 },
+ { 0x0f92, 0x0803 }, { 0x0fa6, 0x2b00 }, { 0x0fae, 0x0800 }, { 0x0fb8, 0x407c },
+ { 0x0fd0, 0x4000 }, { 0x0fd4, 0x0001 }, { 0x0fe0, 0x5758 }, { 0x0fe8, 0x4008 },
+ { 0x1000, 0x802c }, { 0x1002, 0x0d00 }, { 0x1004, 0x0001 }, { 0x1006, 0x0010 },
+ { 0x1008, 0x8050 }, { 0x100a, 0x0803 }, { 0x100e, 0x0010 }, { 0x1030, 0x8050 },
+ { 0x1032, 0x0803 }, { 0x1036, 0x5a10 }, { 0x1046, 0x0100 }, { 0x1050, 0x8050 },
+ { 0x1052, 0x0803 }, { 0x1056, 0x0010 }, { 0x1066, 0x4b00 }, { 0x106e, 0x0400 },
+ { 0x1070, 0x807c }, { 0x1076, 0x0010 }, { 0x1086, 0x3b00 }, { 0x109c, 0x0001 },
+ { 0x10a0, 0x9758 }, { 0x10a6, 0x0010 }, { 0x10a8, 0x8208 }, { 0x10ae, 0x0010 },
+ { 0x10b6, 0x4b00 }, { 0x10c8, 0x801c }, { 0x10ca, 0x00d0 }, { 0x10cc, 0x0001 },
+ { 0x10ce, 0x0010 }, { 0x10d0, 0x8060 }, { 0x10d2, 0x0084 }, { 0x10d6, 0x0010 },
+ { 0x10e6, 0x7b00 }, { 0x10f0, 0x807c }, { 0x10f6, 0x0010 }, { 0x1108, 0x8000 },
+ { 0x110c, 0x0001 }, { 0x110e, 0x0010 }, { 0x1118, 0x802c }, { 0x111a, 0x0d00 },
+ { 0x111c, 0x0001 }, { 0x111e, 0x0010 }, { 0x1120, 0x8020 }, { 0x1122, 0x0803 },
+ { 0x1126, 0x0010 }, { 0x1136, 0x2b00 }, { 0x113e, 0x0800 }, { 0x1148, 0x807c },
+ { 0x114e, 0x0010 }, { 0x1160, 0x8000 }, { 0x1164, 0x0001 }, { 0x1166, 0x0010 },
+ { 0x1170, 0x9758 }, { 0x1176, 0x0010 }, { 0x1178, 0x8008 }, { 0x117e, 0x0010 },
+ { 0x1190, 0xc068 }, { 0x1196, 0x0010 },
+};
+
+static const u16 dram_acsm_sram[] __section(".init.rodata") = {
+ 0x0010, 0x0400, 0x080e, 0x000b, 0x0480, 0x0809, 0x0009, 0x0308,
+ 0x0c09, 0x0028, 0x8160, 0x087c, 0x0098, 0x0820, 0x0c7e, 0x0002,
+ 0x0001, 0x0078, 0x000a, 0x0370, 0x0839, 0x0018, 0x8160, 0x083c,
+ 0x0009, 0x0150, 0x0839, 0x0070, 0x0392, 0x0039, 0x0078, 0x039a,
+ 0x0039, 0x0e09, 0x0100, 0x0839, 0x0002, 0x0001, 0x0038, 0x0000,
+ 0x0100, 0x0839, 0x0000, 0x0398, 0x0839, 0x0e19, 0x0100, 0x0839,
+ 0x0002, 0x0001, 0x0038, 0x0000, 0x0100, 0x0839, 0x0000, 0x0390,
+ 0x0839, 0x0078, 0x039a, 0x0039, 0x0e29, 0x0100, 0x0839, 0x0002,
+ 0x0001, 0x0038, 0x0000, 0x0100, 0x0839, 0x0070, 0x0392, 0x0039,
+ 0x0000, 0x00f0, 0x0909, 0x0018, 0x8160, 0x08bc, 0x0008, 0x013a,
+ 0x00b9, 0x024b, 0x0100, 0x08b9, 0x0002, 0x0001, 0x00b8, 0x0000,
+ 0x0100, 0x08b9, 0x0020, 0x0004, 0x00b8, 0x0020, 0x0004, 0x00b8,
+ 0x0005, 0x0000, 0x00b8, 0x0479, 0xc7f8, 0x08bc, 0x0000, 0x0000,
+ 0x00b8, 0x0000, 0x00a8, 0x08b9, 0x0009, 0xc9c0, 0x0cbc, 0x0000,
+ 0x0000, 0x00b8, 0x025b, 0x0100, 0x0cb9, 0x0020, 0x0004, 0x00b8,
+ 0x0020, 0x0004, 0x00b8, 0x0020, 0x0004, 0x00b8, 0x0020, 0x0004,
+ 0x00b8, 0x0010, 0x0400, 0x089e, 0x0008, 0x0370, 0x0839, 0x0000,
+ 0x00f0, 0x0909, 0x033b, 0x0100, 0x08b9, 0x0000, 0x0100, 0x08b9,
+ 0x023b, 0x0100, 0x0c99, 0x0002, 0x0150, 0x0839, 0x1800, 0x8660,
+ 0x0839, 0x0000, 0x02a0, 0x0809, 0x0008, 0x1880, 0x0839, 0x0000,
+ 0x82b0, 0x0838, 0x0018, 0x00e0, 0x0809, 0x0000, 0x00e0, 0x0869,
+ 0x000f, 0x07c0, 0x0839, 0x000a, 0x0510, 0x0839, 0x000a, 0x0700,
+ 0x0849, 0x000a, 0x0700, 0x0859, 0x000a, 0x0708, 0x0859, 0x0002,
+ 0x0708, 0x0849, 0x0ff8, 0x8410, 0x0839, 0x7ff8, 0x8498, 0x0838,
+ 0x0010, 0x8160, 0x083c, 0x0000, 0x07c8, 0x0809, 0x0000, 0x0001,
+ 0x0008, 0x0000, 0x045a, 0x0089, 0x0000, 0x8018, 0x0808, 0x0000,
+ 0x8020, 0x0808, 0x0479, 0xc7f8, 0x080c, 0x0000, 0x0000, 0x0008,
+ 0x0000, 0x8018, 0x0808, 0x0000, 0x8020, 0x0808, 0x0009, 0xc9c0,
+ 0x0c0c, 0x0000, 0x00a8, 0x08c9, 0x0029, 0x8160, 0x087c, 0x00b8,
+ 0x0820, 0x0c7e, 0x0001, 0x0000, 0x0078, 0x0000, 0x0001, 0x0078,
+ 0x00d8, 0x0820, 0x0c7e, 0x0001, 0x0000, 0x0078, 0x0000, 0x0001,
+ 0x0078, 0x00f8, 0x0820, 0x0c7e, 0x0001, 0x0000, 0x0078, 0x0000,
+ 0x0001, 0x0078, 0x0000, 0x0700, 0x0849, 0x0000, 0x0618, 0x0809,
+ 0x3ff8, 0x84b8, 0x0808, 0x0000, 0x0000, 0x0008, 0x0008, 0xc870,
+ 0x0c0c, 0x0018, 0x0710, 0x0849, 0x0000, 0x0004, 0x0048, 0x0000,
+ 0x0004, 0x0048, 0x0010, 0x0710, 0x0849, 0x0000, 0x0004, 0x0048,
+ 0x0000, 0x0710, 0x0849, 0x0000, 0x0004, 0x0048, 0x0000, 0x0004,
+ 0x0018, 0x0000, 0x0004, 0x0018, 0x0000, 0x0004, 0x0018, 0x0000,
+ 0x8410, 0x0809, 0x0000, 0x8498, 0x0808, 0x0005, 0x0510, 0x0809,
+ 0x0198, 0x1000, 0x0809, 0x0009, 0x03c0, 0x0809, 0x0010, 0x0510,
+ 0x0809, 0x0000, 0x03c0, 0x0809, 0x0018, 0x0004, 0x0008, 0x0002,
+ 0x0510, 0x0809, 0x0002, 0x1000, 0x0809, 0x0068, 0x801a, 0x0008,
+ 0x0068, 0x8022, 0x0008, 0x0479, 0xc7f8, 0x080c, 0x0000, 0x0000,
+ 0x0008, 0x0068, 0x801a, 0x0008, 0x0068, 0x8022, 0x0008, 0x0009,
+ 0xc9c0, 0x0c0c, 0x0000, 0xc870, 0x0c0c, 0x0000, 0x02a0, 0x0809,
+ 0x0010, 0x8160, 0x089c, 0x0002, 0x07c8, 0x0809, 0x0000, 0x0001,
+ 0x0008, 0x0040, 0x1932, 0x0129, 0x000a, 0xc140, 0x080c, 0x0098,
+ 0x193a, 0x0129, 0x0090, 0x1932, 0x0329, 0x0090, 0x193a, 0x0329,
+ 0x0088, 0x1882, 0x0009, 0x0000, 0xc140, 0x080c, 0x0010, 0x0400,
+ 0x080e, 0x000b, 0x0370, 0x08f9, 0x023b, 0x0100, 0x0cf9, 0x0000,
+ 0x02a0, 0x0809, 0x7800, 0x8660, 0x0809, 0x000b, 0x0370, 0x0809,
+ 0x000d, 0x07c0, 0x0809, 0x0000, 0x82b0, 0x0818, 0x0000, 0x82b0,
+ 0x0868, 0x0479, 0xc7f8, 0x091c, 0x0000, 0x0000, 0x0118, 0x0038,
+ 0x80a2, 0x0118, 0x0009, 0xc9c0, 0x0d1c, 0x0000, 0x0000, 0x0118,
+ 0x0008, 0x0508, 0x0809, 0x0005, 0x07c0, 0x0809, 0x0008, 0x82b0,
+ 0x0868, 0x0008, 0x82b0, 0x0818, 0x0008, 0x82b0, 0x0888, 0x0010,
+ 0x0510, 0x0809, 0x0000, 0x00a8, 0x08c9, 0x0008, 0x1900, 0x0c09,
+ 0x0002, 0x0000, 0x0008, 0x0000, 0x0508, 0x0809, 0x0002, 0x07c8,
+ 0x0829, 0x000a, 0x03c8, 0x0c29, 0x0199, 0x03d0, 0x0c19, 0x0003,
+ 0x0000, 0x0018, 0x0038, 0x0004, 0x0018, 0x0199, 0x03d0, 0x0c09,
+ 0x0000, 0x0000, 0x0008, 0x0009, 0x0150, 0x0809, 0x0009, 0xc800,
+ 0x0c0c, 0x0018, 0x8160, 0x08cc, 0x0008, 0x013a, 0x00c9, 0x024b,
+ 0x0100, 0x08c9, 0x0002, 0x0001, 0x00c8, 0x0000, 0x0100, 0x08c9,
+ 0x0020, 0x0004, 0x00c8, 0x0020, 0x0004, 0x00c8, 0x0005, 0x0000,
+ 0x00c8, 0x0479, 0xc7f8, 0x08cc, 0x0000, 0x0000, 0x00c8, 0x0030,
+ 0x00aa, 0x00c9, 0x0009, 0xc9c0, 0x0ccc, 0x0000, 0x0000, 0x00c8,
+ 0x0479, 0xc7f8, 0x08bc, 0x0000, 0x0000, 0x00b8, 0x0030, 0x00aa,
+ 0x00b9, 0x0009, 0xc9c0, 0x0cbc, 0x0000, 0x0000, 0x00b8, 0x025b,
+ 0x0100, 0x0cc9, 0x0020, 0x0004, 0x00c8, 0x0020, 0x0004, 0x00c8,
+ 0x0020, 0x0004, 0x00a8, 0x0004, 0x0000, 0x00c8, 0x0009, 0xc800,
+ 0x0c6c, 0x026f, 0x0100, 0x0c19, 0x0008, 0x0004, 0x0018, 0x027f,
+ 0x0100, 0x0c19, 0x0479, 0xc7f8, 0x080c, 0x0000, 0x0000, 0x0008,
+ 0x0058, 0x008a, 0x0009, 0x0009, 0xc9c0, 0x0c0c, 0x0000, 0x0000,
+ 0x0008, 0x0018, 0x8160, 0x086c, 0x1008, 0x853b, 0x0868, 0x0c00,
+ 0x8588, 0x0868, 0x0008, 0x85a8, 0x0868, 0x0008, 0x85c8, 0x0868,
+ 0x0000, 0x8050, 0x0868, 0x0060, 0x0138, 0x0869, 0x0018, 0x8510,
+ 0x0868, 0x0018, 0x02c8, 0x0869, 0x0010, 0x8520, 0x0868, 0x1ff8,
+ 0x85d8, 0x0c68, 0x0000, 0xfdf0, 0x0868, 0x0008, 0x85f0, 0x0868,
+ 0x0008, 0xa5f0, 0x0868, 0x0298, 0x0100, 0x0869, 0x0002, 0x0001,
+ 0x0068, 0x0000, 0x0100, 0x0869, 0x0000, 0x85f0, 0x0868, 0x0000,
+ 0xa5f0, 0x0868, 0x033b, 0x0100, 0x08c9, 0x0000, 0x0100, 0x08c9,
+ 0x1ff8, 0x85d8, 0x0c68, 0x0008, 0x8df0, 0x0868, 0x0008, 0xadf0,
+ 0x0868, 0x02a8, 0x0100, 0x0869, 0x0002, 0x0001, 0x0068, 0x0000,
+ 0x0100, 0x0869, 0x0000, 0x8df0, 0x0868, 0x0000, 0xadf0, 0x0868,
+ 0x0010, 0x02c8, 0x0869, 0x0028, 0x0004, 0x0068, 0x000a, 0x0500,
+ 0x0c69, 0x0000, 0x8520, 0x0868, 0x0008, 0x8530, 0x0808, 0x02ba,
+ 0x0100, 0x0869, 0x0002, 0x0001, 0x0068, 0x0002, 0x0100, 0x0869,
+ 0x0000, 0x8530, 0x0808, 0x0000, 0x8510, 0x0868, 0x0000, 0x02c8,
+ 0x0869, 0x0008, 0x8050, 0x0868, 0x0030, 0x0004, 0x0068, 0x0008,
+ 0x8520, 0x0868, 0x0000, 0x8328, 0x086b, 0x02ca, 0x0100, 0x0869,
+ 0x0002, 0x0001, 0x0068, 0x0004, 0x0100, 0x0869, 0x0ff8, 0x8328,
+ 0x086b, 0x0009, 0x0500, 0x0c29, 0x0000, 0x8520, 0x0868, 0x0008,
+ 0x03c8, 0x0c29, 0x0000, 0x0150, 0x0809, 0x0002, 0x0370, 0x0809,
+ 0x0000, 0x0400, 0x080e, 0x0010, 0x0004, 0x0118, 0x0020, 0x82b0,
+ 0x0808, 0x0479, 0xc7f8, 0x080c, 0x0000, 0x0000, 0x0008, 0x0060,
+ 0x008a, 0x0009, 0x0009, 0xc9c0, 0x0c0c, 0x0000, 0x0000, 0x0008,
+ 0x0008, 0x02a0, 0x0809, 0x0048, 0xc822, 0x000c, 0x0048, 0xcc32,
+ 0x000c, 0x0009, 0xc800, 0x0c0c, 0x0000, 0x8168, 0x080c, 0x0479,
+ 0xc7f8, 0x091c, 0x0000, 0x0000, 0x0118, 0x9190, 0x80a1, 0x0918,
+ 0x0009, 0xc9c0, 0x0d1c, 0x0000, 0x0000, 0x0118, 0x0018, 0x00f0,
+ 0x0909, 0x0008, 0x0300, 0x0819, 0x0010, 0x8160, 0x080c, 0x0008,
+ 0x07c8, 0x0801, 0x0008, 0x0000, 0x0008, 0x0008, 0x1880, 0x0809,
+ 0x0010, 0x8160, 0x083c, 0x1800, 0x8660, 0x0839, 0x4000, 0x8668,
+ 0x0839, 0x0000, 0x00f0, 0x0909, 0x000f, 0x07c0, 0x0809, 0x0000,
+ 0x02a0, 0x0809, 0x0008, 0x0618, 0x0809, 0x0000, 0x84b8, 0x0808,
+ 0x0ff8, 0x8410, 0x0809, 0x7ff8, 0x8498, 0x0808, 0x0000, 0x07c8,
+ 0x0809, 0x0008, 0x8168, 0x080c, 0x0000, 0x0001, 0x0008, 0x0000,
+ 0x0004, 0x0008, 0x0000, 0x0004, 0x0008, 0x0018, 0x0300, 0x0809,
+ 0x0000, 0x0480, 0x0809, 0x0008, 0x0510, 0x0809, 0x7800, 0x8660,
+ 0x0809, 0x0018, 0x00f0, 0x0909, 0x0008, 0x07c8, 0x0801, 0x0010,
+ 0x0400, 0x080e, 0x0008, 0x0150, 0x0809, 0x0009, 0x0370, 0x0809,
+ 0x0018, 0x8160, 0x080c, 0x0008, 0xc800, 0x0c0c, 0x0008, 0xc888,
+ 0x0c04, 0x0020, 0x8302, 0x000b, 0x0018, 0x8582, 0x0008, 0x9000,
+ 0x8539, 0x0808, 0x0000, 0x8588, 0x0808, 0x0040, 0x85c8, 0x0808,
+ 0x0000, 0xfdf0, 0x0808, 0x0800, 0xc5f0, 0x0808, 0x1ff8, 0xfdd8,
+ 0x0c08, 0x0018, 0xfda8, 0x0808, 0x0040, 0x8520, 0x08d8, 0x0080,
+ 0x8520, 0x08e8, 0x030a, 0x0100, 0x08d9, 0x032a, 0x0100, 0x08e9,
+ 0x0000, 0x0001, 0x0008, 0x0000, 0x0100, 0x0809, 0x1ff8, 0xfdd8,
+ 0x0c08, 0x0008, 0xfda8, 0x0808, 0x0140, 0x8520, 0x08d8, 0x0180,
+ 0x8520, 0x08e8, 0x030a, 0x0100, 0x08d9, 0x032a, 0x0100, 0x08e9,
+ 0x0000, 0x0001, 0x0008, 0x0000, 0x0100, 0x0809, 0x0002, 0x8520,
+ 0x0808, 0x0009, 0x0500, 0x0c09, 0x0010, 0x82b0, 0x0808, 0x0000,
+ 0x8520, 0x0808, 0x0008, 0x03c8, 0x0c09, 0x0000, 0x0150, 0x0809,
+ 0x0002, 0x0370, 0x0809, 0x0000, 0x0400, 0x080e, 0x0008, 0xc848,
+ 0x0c0c, 0x0000, 0xc890, 0x0804, 0x0020, 0x8302, 0x000b, 0x0018,
+ 0x8582, 0x0008, 0x0000, 0x82b0, 0x0808, 0x1000, 0x8539, 0x0808,
+ 0x0000, 0x8588, 0x0808, 0x0040, 0x85c8, 0x0808, 0x07f8, 0xfdf0,
+ 0x0808, 0x0800, 0xc5f0, 0x0808, 0x0080, 0x8088, 0x080b, 0x1ff8,
+ 0xfdd8, 0x0c08, 0x0018, 0xfda8, 0x0808, 0x0200, 0x8520, 0x08d8,
+ 0x0400, 0x8520, 0x08e8, 0x02da, 0x0100, 0x08d9, 0x02ea, 0x0100,
+ 0x08e9, 0x0002, 0x0001, 0x0008, 0x0000, 0x0100, 0x0809, 0x1ff8,
+ 0xfdd8, 0x0c08, 0x0008, 0xfda8, 0x0808, 0x0a00, 0x8520, 0x08d8,
+ 0x0c00, 0x8520, 0x08e8, 0x02da, 0x0100, 0x08d9, 0x02ea, 0x0100,
+ 0x08e9, 0x0000, 0x0001, 0x0008, 0x0000, 0x0100, 0x0809, 0x0009,
+ 0x0500, 0x0c09, 0x0010, 0x82b0, 0x0808, 0x0000, 0x8520, 0x0808,
+ 0x0008, 0x03c8, 0x0c09, 0x0000, 0x0150, 0x0809, 0x0002, 0x0370,
+ 0x0809, 0x0000, 0x0400, 0x080e, 0x0008, 0xc848, 0x0c0c, 0x0000,
+ 0xc890, 0x0804, 0x1008, 0x853b, 0x0808, 0x0c00, 0x8588, 0x0808,
+ 0x0008, 0x85a8, 0x0808, 0x0008, 0x85c8, 0x0808, 0x0060, 0x0138,
+ 0x0809, 0x0018, 0x00c8, 0x0809, 0x0000, 0x00a4, 0x0809, 0x0000,
+ 0x8510, 0x0808, 0x02fa, 0x0100, 0x08d9, 0x031a, 0x0100, 0x08e9,
+ 0x0002, 0x0001, 0x0008, 0x0002, 0x0100, 0x0809, 0x0202, 0x8510,
+ 0x0c08, 0x0050, 0x00a2, 0x0009, 0x0000, 0x0000, 0x0008, 0x0000,
+ 0x0000, 0x0008, 0x0000, 0x0000, 0x0008, 0x0000, 0x00c8, 0x0809,
+ 0x000b, 0x0508, 0x0809, 0x0008, 0x1900, 0x0c09, 0x0002, 0x0000,
+ 0x0008, 0x0000, 0x0508, 0x0809, 0x0009, 0x0500, 0x0c09, 0x0010,
+ 0x82b0, 0x0808, 0x0000, 0x8520, 0x0808, 0x0008, 0x03c8, 0x0c09,
+ 0x0000, 0x0150, 0x0809, 0x0002, 0x0370, 0x0809, 0x0000, 0x0400,
+ 0x080e, 0x0008, 0xc848, 0x0c0c, 0x0000, 0xc890, 0x0804, 0x0020,
+ 0x8302, 0x000b, 0x0018, 0x8582, 0x0008, 0x1000, 0x8539, 0x0808,
+ 0x0000, 0x8588, 0x0808, 0x0040, 0x85c8, 0x0808, 0x0008, 0xfdf0,
+ 0x0808, 0x0080, 0x8088, 0x080b, 0x0000, 0x03b8, 0x0869, 0x1ff8,
+ 0xfdd8, 0x0c08, 0x0018, 0xfda8, 0x0808, 0x0040, 0x8520, 0x08d8,
+ 0x0080, 0x8520, 0x08e8, 0x02da, 0x0100, 0x08d9, 0x02ea, 0x0100,
+ 0x08e9, 0x0002, 0x0001, 0x0008, 0x0000, 0x0100, 0x0809, 0x1ff8,
+ 0xfdd8, 0x0c08, 0x0008, 0xfda8, 0x0808, 0x0140, 0x8520, 0x08d8,
+ 0x0180, 0x8520, 0x08e8, 0x02da, 0x0100, 0x08d9, 0x02ea, 0x0100,
+ 0x08e9, 0x0000, 0x0001, 0x0008, 0x0000, 0x0100, 0x0809, 0x0009,
+ 0x0500, 0x0c09, 0x0010, 0x82b0, 0x0808, 0x0000, 0x8520, 0x0808,
+ 0x0008, 0x03c8, 0x0c09, 0x0000, 0x0150, 0x0809, 0x0002, 0x0370,
+ 0x0809, 0x0000, 0x0400, 0x080e, 0x0008, 0xc848, 0x0c0c, 0x0000,
+ 0xc890, 0x0804,
+};
+
+static const struct sunxi_dram_reg dram_acsm_ctrl_regs[] __section(".init.rodata") = {
+ { 0x0aaa01ce, 0x0600 },
+ { 0x0aa20038, 0x0000 },
+ { 0x0aa20040, 0x0077 },
+ { 0x0aa20048, 0x0082 },
+ { 0x0aa2004a, 0x002b },
+ { 0x0aa2004c, 0x002b },
+ { 0x0aa20056, 0x010e },
+ { 0x0aa20e10, 0x0125 },
+ { 0x0aa20e00, 0x014c },
+ { 0x0aa20e02, 0x016e },
+ { 0x0aa20e04, 0x018d },
+ { 0x0aa20e06, 0x012b },
+ { 0x0aa20e08, 0x0000 },
+ { 0x0aa20e0a, 0x0000 },
+ { 0x0aa20e0c, 0x0000 },
+ { 0x0aa20e0e, 0x0000 },
+ { 0x0a940314, 0x0000 },
+};
+
+static const u16 dram_pie_ptrs[] __section(".init.rodata") = {
+ 0x0001, 0x0016, 0x0039, 0x004e, 0x0001, 0x0016, 0x0039, 0x004e,
+ 0x0017, 0x0027, 0x004f, 0x005f, 0x0017, 0x0027, 0x004f, 0x005f,
+ 0x0028, 0x0038, 0x0060, 0x0070, 0x0028, 0x0038, 0x0060, 0x0070,
+ 0x0080, 0x0080, 0x0080, 0x0080, 0x0080, 0x0080, 0x0080, 0x0080,
+ 0x0071, 0x0074, 0x0077, 0x007a, 0x0071, 0x0074, 0x0077, 0x007a,
+ 0x0075, 0x0076, 0x007b, 0x007c, 0x0075, 0x0076, 0x007b, 0x007c,
+ 0x007d, 0x007e, 0x007d, 0x007e, 0x007d, 0x007e, 0x007d, 0x007e,
+ 0x007f, 0x007f, 0x007f, 0x007f, 0x007f, 0x007f, 0x007f, 0x007f,
+ 0x0080, 0x007f, 0x0080, 0x007f, 0x0080, 0x007f, 0x0080, 0x007f,
+ 0x0081, 0x0093, 0x0081, 0x0093, 0x010a, 0x011b, 0x010a, 0x011b,
+ 0x0094, 0x009b, 0x0094, 0x009b, 0x011c, 0x0122, 0x011c, 0x0122,
+ 0x009c, 0x00af, 0x009c, 0x00af, 0x0123, 0x0132, 0x0123, 0x0132,
+ 0x00b0, 0x00c0, 0x00b0, 0x00c0, 0x0133, 0x0143, 0x0133, 0x0143,
+ 0x00c1, 0x00ca, 0x00c1, 0x00ca, 0x0144, 0x014d, 0x0144, 0x014d,
+ 0x00cb, 0x00d4, 0x00cb, 0x00d4, 0x014e, 0x0157, 0x014e, 0x0157,
+ 0x00d5, 0x00df, 0x00d5, 0x00df, 0x0158, 0x0161, 0x0158, 0x0161,
+ 0x00e0, 0x00ee, 0x00e0, 0x00ee, 0x0162, 0x0170, 0x0162, 0x0170,
+ 0x00ef, 0x00f9, 0x00ef, 0x00f9, 0x0171, 0x017a, 0x0171, 0x017a,
+ 0x00fa, 0x0108, 0x00fa, 0x0108, 0x017b, 0x0189, 0x017b, 0x0189,
+ 0x0109, 0x0109, 0x0109, 0x0109, 0x018a, 0x018a, 0x018a, 0x018a,
+};
+
+static const struct sunxi_dram_reg dram_pie_patch_regs[] __section(".init.rodata") = {
+ { 0x0a982c26, 0x0b00 },
+ { 0x0a982c2e, 0x0000 },
+ { 0x0a982c76, 0x0b00 },
+ { 0x0a982c7e, 0x0000 },
+ { 0x0a982cc6, 0x0b00 },
+ { 0x0a982cce, 0x0000 },
+ { 0x0a982d16, 0x0b00 },
+ { 0x0a982d1e, 0x0000 },
+ { 0x0a982e46, 0x0b00 },
+ { 0x0a982e4e, 0x0000 },
+ { 0x0a982e96, 0x0b00 },
+ { 0x0a982e9e, 0x0000 },
+ { 0x0a982fe6, 0x0b00 },
+ { 0x0a982fee, 0x0000 },
+ { 0x0a983036, 0x0b00 },
+ { 0x0a98303e, 0x0000 },
+ { 0x0a983456, 0x5b00 },
+ { 0x0a98345e, 0x0000 },
+ { 0x0a9834a6, 0x7b00 },
+ { 0x0a9834ae, 0x0400 },
+ { 0x0a9834f6, 0x5b00 },
+ { 0x0a9834fe, 0x0000 },
+ { 0x0a983546, 0x7b00 },
+ { 0x0a98354e, 0x0400 },
+ { 0x0a983666, 0x5b00 },
+ { 0x0a98366e, 0x0000 },
+ { 0x0a9836b6, 0x7b00 },
+ { 0x0a9836be, 0x0400 },
+ { 0x0a9837f6, 0x5b00 },
+ { 0x0a9837fe, 0x0000 },
+ { 0x0a983846, 0x7b00 },
+ { 0x0a98384e, 0x0400 },
+};
+
+static const struct sunxi_dram_reg dram_pie_pre_lane_regs[] __section(".init.rodata") = {
+ { 0x0aa80002, 0x5061 },
+ { 0x0aa80004, 0x7098 },
+ { 0x0aa80006, 0x0000 },
+ { 0x0aa80000, 0xbedc },
+ { 0x0aa8000c, 0xb0db },
+ { 0x0aa8000e, 0x0000 },
+ { 0x0aa20e18, 0x0000 },
+ { 0x0aa20e1a, 0x07ff },
+ { 0x0aa20e1c, 0x0668 },
+ { 0x0aa20e1e, 0x7910 },
+ { 0x0aa20e20, 0x355e },
+ { 0x0aa20e22, 0xcbb5 },
+ { 0x0aa20e26, 0xffff },
+ { 0x0aa20e28, 0x0710 },
+ { 0x0aa20e2a, 0x7800 },
+ { 0x0aa21820, 0xffff },
+ { 0x0aa21824, 0xff10 },
+ { 0x0aa20e24, 0x2ee8 },
+ { 0x0aa21822, 0xffff },
+ { 0x0aa20e2c, 0xffdf },
+ { 0x0aa20e2e, 0x6f10 },
+ { 0x0aa20e30, 0x6f10 },
+ { 0x0aa20e32, 0x2868 },
+ { 0x0aa20e34, 0x2868 },
+ { 0x0aa20e36, 0xfeef },
+ { 0x0aa21208, 0xe400 },
+ { 0x0aa2130c, 0xe400 },
+ { 0x0aa21240, 0x0000 },
+ { 0x0aa21242, 0x0000 },
+ { 0x0aa21244, 0x0000 },
+ { 0x0aa21246, 0x0000 },
+ { 0x0aa21248, 0xff00 },
+ { 0x0aa2124a, 0xff00 },
+ { 0x0aa2124c, 0xff00 },
+ { 0x0aa2124e, 0xff00 },
+ { 0x0aa21260, 0x4000 },
+ { 0x0aa21262, 0x2000 },
+ { 0x0aa21264, 0xff00 },
+ { 0x0aa21266, 0xff00 },
+ { 0x0aa21214, 0x0000 },
+ { 0x0aa21218, 0x0000 },
+ { 0x0aa2121a, 0x0001 },
+ { 0x0a9401e0, 0x27f7 },
+ { 0x0a9401e2, 0x2108 },
+ { 0x0a9401e4, 0x2108 },
+ { PHY_BCAST_DBYTE | 0x0a920172, 0x0001 },
+ { PHY_BCAST_DBYTE | 0x0a920162, 0x0180 },
+ { PHY_BCAST_DBYTE | 0x0a920174, 0x0001 },
+ { PHY_BCAST_DBYTE | 0x0a920144, 0x0000 },
+ { PHY_BCAST_DBYTE | 0x0a92016a, 0x0001 },
+};
+
+static const struct sunxi_dram_reg dram_pie_tail_regs[] __section(".init.rodata") = {
+ { 0x0a9e00ca, 0x01ff },
+ { 0x0a9400fc, 0x0033 },
+ { 0x0a9401de, 0xffff },
+ { 0x0aa21220, 0x0002 },
+ { 0x0a940620, 0x0001 },
+ { 0x0a940622, 0x0001 },
+ { 0x0aa80100, 0x0002 },
+ { 0x0aaa0006, 0x0000 },
+ { 0x0aaa0000, 0x0001 },
+ { 0x0aaa0000, 0x0000 },
+ { 0x0aa80100, 0x0003 },
+ { 0x0aa80100, 0x0000 },
+ { 0x0aaa0000, 0x0001 },
+};
+
+static const struct sunxi_dram_reg dram_post_pmu_ch_regs[] __section(".init.rodata") = {
+ { 0x0a110b84, 0x00000001 },
+ { 0x0a110c80, 0x00000000 },
+ { 0x0a100d04, 0x0000090b },
+ { 0x0a100d08, 0x00001221 },
+ { 0x0a100008, 0x070d1011 },
+ { 0x0a100060, 0x000e120c },
+ { 0x0a100024, 0x0002040d },
+ { 0x0a100584, 0x00070303 },
+ { 0x0aaa0000, 0x0000 },
+ { 0x0aaa0000, 0x0001 },
+ { 0x0a100588, 0x00152f17 },
+ { 0x0a110c80, 0x00000001 },
+ { 0x0a110b84, 0x00000000 },
+};
+
+static const struct sunxi_dram_reg dram_post_pmu_tail_regs[] __section(".init.rodata") = {
+ { 0x0a110c80, 0x00000000 },
+ { 0x0a510c80, 0x00000000 },
+ { 0x0a110510, 0x00011015 },
+ { 0x0a510510, 0x00011015 },
+ { 0x0a110510, 0x00011035 },
+ { 0x0a510510, 0x00011035 },
+ { 0x0a110c80, 0x00000001 },
+ { 0x0a510c80, 0x00000001 },
+ { 0x07010250, 0x00000001 },
+ { 0x0a110c80, 0x00000000 },
+ { 0x0a510c80, 0x00000000 },
+ { 0x0a110510, 0x00011015 },
+ { 0x0a510510, 0x00011015 },
+ { 0x0a110180, 0x00000000 },
+ { 0x0a510180, 0x00000000 },
+ { 0x0a110510, 0x00011014 },
+ { 0x0a510510, 0x00011014 },
+ { 0x0a110c80, 0x00000001 },
+ { 0x0a510c80, 0x00000001 },
+ { 0x0a110c80, 0x00000000 },
+ { 0x0a510c80, 0x00000000 },
+ { 0x0a110208, 0x00000000 },
+ { 0x0a510208, 0x00000000 },
+ { 0x0a110510, 0x00015014 },
+ { 0x0a510510, 0x00015014 },
+ { 0x0a110c80, 0x00000001 },
+ { 0x0a510c80, 0x00000001 },
+ { 0x02024818, 0x00008002 },
+ { 0x0202481c, 0xffffffff },
+ { 0x0a030018, 0x00008002 },
+ { 0x0a040018, 0x00008002 },
+ { 0x0a030068, 0xffffffff },
+ { 0x0a040068, 0xffffffff },
+ { 0x02024838, 0x00000002 },
+ { 0x0202448c, 0x00000002 },
+ { 0x0202468c, 0x00000002 },
+ { 0x02023a04, 0xffffffff },
+ { 0x02023c04, 0xffffffff },
+ { 0x02023e04, 0xffffffff },
+ { 0x02024004, 0xffffffff },
+ { 0x02024204, 0xffffffff },
+ { 0x02024490, 0x10102010 },
+ { 0x02024494, 0x08081008 },
+ { 0x02024690, 0x10102010 },
+ { 0x02024694, 0x08081008 },
+};
+
+#endif /* _DRAM_SUN60I_A733_TABLES_H */
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 20/26] sunxi: sun60i: initial support for A733 SoC
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (18 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 19/26] sunxi: dram: a733: Add Allwinner A733 LPDDR5 DRAM driver Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 21/26] doc: sunxi: Document Allwinner A733 LPDDR5 PMU architecture Alastair D'Silva
` (7 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
From: Yixun Lan <dlan@gentoo.org>
Add basic support for Allwinner A733 SoC, including SPL settings:
- Introduce MACH_SUN60I_A733 Kconfig option and sun60i.h wrapper
- Program CPU_DA_DDR_CTRL_REG, AA64 mode, and RVBAR in boot0.h /
rmr_switch.S for the AArch32-to-AArch64 RMR warm reset
- Configure 16 GiB max DRAM size, SPL text/stack/size layout, and TF-A
BL31 load address at 0x58000 above active SPL
- Support A733 watchdog soft-reset key, R_TWI SPL PMIC pins, MMC2 pinmux,
UART0_PORT_B0 option, and FEL SH_SRAM SPL header lookup
Link: https://lore.kernel.org/r/20260113-01-a733-soc-support-v3-9-f19dcd098f60@gentoo.org
Signed-off-by: Yixun Lan <dlan@gentoo.org>
[Alastair: Wire DRAM_SUN60I_A733 and SPL_DISPOSABLE_INIT_RODATA, add RMR
AXI2HIF/AA64 sequence, watchdog reset key, R_I2C/MMC2/FEL SPL helpers]
Co-developed-by: Alastair D'Silva <alastair@d-silva.org>
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
arch/arm/include/asm/arch-sunxi/boot0.h | 20 +++++++++++++-
.../include/asm/arch-sunxi/cpu_sunxi_ncat2.h | 1 +
arch/arm/include/asm/arch-sunxi/spl.h | 3 +++
arch/arm/include/asm/arch-sunxi/watchdog.h | 1 +
arch/arm/mach-sunxi/Kconfig | 26 +++++++++++++++++++
arch/arm/mach-sunxi/board.c | 6 +++++
arch/arm/mach-sunxi/cpu_info.c | 2 ++
arch/arm/mach-sunxi/rmr_switch.S | 22 +++++++++++++---
board/sunxi/board.c | 21 ++++++++++++---
common/spl/Kconfig | 4 +++
include/configs/sun60i.h | 11 ++++++++
11 files changed, 110 insertions(+), 7 deletions(-)
create mode 100644 include/configs/sun60i.h
diff --git a/arch/arm/include/asm/arch-sunxi/boot0.h b/arch/arm/include/asm/arch-sunxi/boot0.h
index d79aea97a40..7010030a42f 100644
--- a/arch/arm/include/asm/arch-sunxi/boot0.h
+++ b/arch/arm/include/asm/arch-sunxi/boot0.h
@@ -32,7 +32,7 @@
.word 0xe580e00c // str lr, [r0, #12]
.word 0xee1cef10 // mrc 15, 0, lr, cr12, cr0, {0}
.word 0xe580e010 // str lr, [r0, #16]
-#ifdef CONFIG_MACH_SUN55I_A523
+#if defined(CONFIG_MACH_SUN55I_A523) || defined(CONFIG_MACH_SUN60I_A733)
.word 0xee1cefbc // mrc 15, 0, lr, cr12, cr12, {5}
.word 0xe31e0001 // tst lr, #1
.word 0x0a000003 // beq cc <start32+0x48>
@@ -41,6 +41,18 @@
.word 0xee1ceffc // mrc 15, 0, lr, cr12, cr12, {7}
.word 0xe580e01c // str lr, [r0, #28]
#endif
+#if defined(CONFIG_MACH_SUN60I_A733)
+ .word 0xe59f1040 // ldr r1, [pc, #64] ; CPU_DA_DDR_CTRL_REG
+ .word 0xe3a02001 // mov r2, #1
+ .word 0xe5812000 // str r2, [r1]
+ .word 0xe59f1038 // ldr r1, [pc, #56] ; CLU0_CPU0_CTRL_REG
+ .word 0xe3a02001 // mov r2, #1
+ .word 0xe5812000 // str r2, [r1]
+ .word 0xe59f0030 // ldr r0, [pc, #48] ; CONFIG_*TEXT_BASE
+ .word 0xe5810004 // str r0, [r1, #4] ; CLU0_RVBARADDR0_L
+ .word 0xe3a02000 // mov r2, #0
+ .word 0xe5812008 // str r2, [r1, #8] ; CLU0_RVBARADDR0_H
+#else
.word 0xe59f1034 // ldr r1, [pc, #52] ; RVBAR_ADDRESS
.word 0xe59f0034 // ldr r0, [pc, #52] ; SUNXI_SRAMC_BASE
.word 0xe5900024 // ldr r0, [r0, #36] ; SRAM_VER_REG
@@ -48,6 +60,7 @@
.word 0x159f102c // ldrne r1, [pc, #44] ; RVBAR_ALTERNATIVE
.word 0xe59f002c // ldr r0, [pc, #44] ; CONFIG_*TEXT_BASE
.word 0xe5810000 // str r0, [r1]
+#endif
.word 0xf57ff04f // dsb sy
.word 0xf57ff06f // isb sy
.word 0xee1c0f50 // mrc 15, 0, r0, cr12, cr0, {2} ; RMR
@@ -57,9 +70,14 @@
.word 0xe320f003 // wfi
.word 0xeafffffd // b @wfi
+#if defined(CONFIG_MACH_SUN60I_A733)
+ .word SUNXI_CPU_DA_DDR_CTRL_REG
+ .word SUNXI_CLU0_CPU0_CTRL_REG
+#else
.word CONFIG_SUNXI_RVBAR_ADDRESS // writable RVBAR mapping addr
.word SUNXI_SRAMC_BASE
.word CONFIG_SUNXI_RVBAR_ALTERNATIVE // address for die variant
+#endif
#ifdef CONFIG_XPL_BUILD
.word CONFIG_SPL_TEXT_BASE
#else
diff --git a/arch/arm/include/asm/arch-sunxi/cpu_sunxi_ncat2.h b/arch/arm/include/asm/arch-sunxi/cpu_sunxi_ncat2.h
index 260fed24741..b59a120d780 100644
--- a/arch/arm/include/asm/arch-sunxi/cpu_sunxi_ncat2.h
+++ b/arch/arm/include/asm/arch-sunxi/cpu_sunxi_ncat2.h
@@ -63,6 +63,7 @@
#define SUNXI_RTC_BASE 0x07090000
#if defined(CONFIG_MACH_SUN60I_A733)
+#define SUNXI_FEL_EL3_VBAR 0x00080000
#define SUNXI_CPUCFG_BASE 0x08000000
#define SUNXI_CPU_DA_DDR_CTRL_REG (SUNXI_CPUCFG_BASE + 0x0200)
#define SUNXI_CLU0_CPU0_CTRL_REG (SUNXI_CPUCFG_BASE + 0x1000)
diff --git a/arch/arm/include/asm/arch-sunxi/spl.h b/arch/arm/include/asm/arch-sunxi/spl.h
index 14944a20eac..4473c605461 100644
--- a/arch/arm/include/asm/arch-sunxi/spl.h
+++ b/arch/arm/include/asm/arch-sunxi/spl.h
@@ -11,6 +11,9 @@
#define SPL_ADDR CONFIG_SUNXI_SRAM_ADDRESS
+/* FEL swap buffer backup address in SH_SRAM (A733) */
+#define SUNXI_FEL_SPL_ADDR 0x000e0000
+
/* The low 8-bits of the 'boot_media' field in the SPL header */
#define SUNXI_BOOTED_FROM_MMC0 0
#define SUNXI_BOOTED_FROM_NAND 1
diff --git a/arch/arm/include/asm/arch-sunxi/watchdog.h b/arch/arm/include/asm/arch-sunxi/watchdog.h
index 14a6e89ccfa..950811523f3 100644
--- a/arch/arm/include/asm/arch-sunxi/watchdog.h
+++ b/arch/arm/include/asm/arch-sunxi/watchdog.h
@@ -31,6 +31,7 @@ struct sunxi_wdog {
#else
#define WDT_CFG_RESET (0x1)
+#define WDT_CFG_KEY (0x16aa << 16)
#define WDT_MODE_EN (0x1)
struct sunxi_wdog {
diff --git a/arch/arm/mach-sunxi/Kconfig b/arch/arm/mach-sunxi/Kconfig
index f3f70b277b1..df686d63186 100644
--- a/arch/arm/mach-sunxi/Kconfig
+++ b/arch/arm/mach-sunxi/Kconfig
@@ -243,6 +243,7 @@ config SUNXI_SRAM_ADDRESS
hex
default 0x10000 if MACH_SUN9I || MACH_SUN50I || MACH_SUN50I_H5
default 0x44000 if MACH_SUN55I_A523
+ default 0x47000 if MACH_SUN60I_A733
default 0x20000 if SUN50I_GEN_H6 || SUNXI_GEN_NCAT2
default 0x0
help
@@ -285,6 +286,7 @@ config SUNXI_BL31_BASE
default 0x40000000 if MACH_SUN50I_H616 || MACH_SUN50I_A133
default 0x00104000 if SUN50I_GEN_H6
default 0x00054000 if MACH_SUN55I_A523
+ default 0x00058000 if MACH_SUN60I_A733
default 0x0
help
Address where BL31 (TF-A) is loaded, or zero if BL31 is not used.
@@ -367,6 +369,7 @@ config MACH_SUNXI_H3_H5
config SUNXI_DRAM_MAX_SIZE
hex
default 0x100000000 if MACH_SUN50I_H616 || MACH_SUN50I_A133 || MACH_SUN55I_A523
+ default 0x400000000 if MACH_SUN60I_A733
default 0xC0000000 if MACH_SUN50I || MACH_SUN50I_H5 || MACH_SUN50I_H6
default 0x80000000
@@ -581,6 +584,21 @@ config MACH_SUN55I_A523
select SPL_LOAD_FIT if SPL
imply OF_UPSTREAM
+config MACH_SUN60I_A733
+ bool "sun60i (Allwinner A733)"
+ select ARM64
+ select DRAM_SUN60I_A733
+ select FIT
+ select SPL_DISPOSABLE_INIT_RODATA if SPL
+ select SPL_LOAD_FIT if SPL
+ select SUNXI_A733_PINCTRL
+ select SUNXI_GEN_NCAT2
+ select SUPPORT_SPL
+ imply SPL_SYS_I2C_LEGACY
+ help
+ Support for Allwinner A733 and T736 SoCs with octa-core ARMv8 cores,
+ NCAT2 interconnect, and LPDDR5 DRAM controller.
+
endchoice
# The sun8i SoCs share a lot, this helps to avoid a lot of "if A23 || A33"
@@ -902,6 +920,7 @@ config SYS_CONFIG_NAME
default "sun50i" if MACH_SUN50I_H616
default "sun50i" if MACH_SUN50I_A133
default "sun55i" if MACH_SUN55I_A523
+ default "sun60i" if MACH_SUN60I_A733
config SYS_BOARD
default "sunxi"
@@ -931,6 +950,13 @@ config UART0_PORT_F
at the same time, the system can be only booted in the FEL mode.
Only enable this if you really know what you are doing.
+config UART0_PORT_B0
+ bool "UART0 on PB0/PB1 pins"
+ depends on MACH_SUN60I_A733
+ help
+ Select this option on Allwinner A733 boards that route the UART0
+ serial console to PB0/PB1 instead of the default PB9/PB10 pins.
+
config OLD_SUNXI_KERNEL_COMPAT
bool "Enable workarounds for booting old kernels"
help
diff --git a/arch/arm/mach-sunxi/board.c b/arch/arm/mach-sunxi/board.c
index 38936afae24..c6dd3f317c3 100644
--- a/arch/arm/mach-sunxi/board.c
+++ b/arch/arm/mach-sunxi/board.c
@@ -523,6 +523,12 @@ void reset_cpu(void)
writel(WDT_CTRL_KEY | WDT_CTRL_RESTART, wdog + WDT_SRST_REG);
while (1)
;
+#elif defined(CONFIG_MACH_SUN60I_A733)
+ static const void *wdog = (void *)(ulong)SUNXI_WDT_BASE;
+
+ writel(WDT_CFG_KEY | WDT_CTRL_RESTART, wdog + WDT_SRST_REG);
+ while (1)
+ ;
#elif defined(CONFIG_SUNXI_GEN_SUN6I) || defined(CONFIG_SUN50I_GEN_H6) || defined(CONFIG_SUNXI_GEN_NCAT2)
#if defined(CONFIG_MACH_SUN50I_H6)
/* WDOG is broken for some H6 rev. use the R_WDOG instead */
diff --git a/arch/arm/mach-sunxi/cpu_info.c b/arch/arm/mach-sunxi/cpu_info.c
index c3a51d9956e..f0b33bb9097 100644
--- a/arch/arm/mach-sunxi/cpu_info.c
+++ b/arch/arm/mach-sunxi/cpu_info.c
@@ -108,6 +108,8 @@ int print_cpuinfo(void)
puts("CPU: Allwinner A133 (SUN50I)\n");
#elif defined CONFIG_MACH_SUN55I_A523
puts("CPU: Allwinner A523 (SUN55I)\n");
+#elif defined CONFIG_MACH_SUN60I_A733
+ puts("CPU: Allwinner A733 (SUN60I)\n");
#else
#warning Please update cpu_info.c with correct CPU information
puts("CPU: SUNXI Family\n");
diff --git a/arch/arm/mach-sunxi/rmr_switch.S b/arch/arm/mach-sunxi/rmr_switch.S
index a6d75c32ed9..2d3ca275037 100644
--- a/arch/arm/mach-sunxi/rmr_switch.S
+++ b/arch/arm/mach-sunxi/rmr_switch.S
@@ -55,7 +55,7 @@ start32:
str lr, [r0, #12]
mrc p15, 0, lr, cr12, cr0, 0 // VBAR
str lr, [r0, #16]
-//#ifdef CONFIG_MACH_SUN55I_A523
+#if defined(CONFIG_MACH_SUN55I_A523) || defined(CONFIG_MACH_SUN60I_A733)
mrc p15, 0, lr, cr12, cr12, 5 // ICC_SRE
tst lr, #1
beq 1f
@@ -64,8 +64,23 @@ start32:
mrc p15, 0, lr, c12, c12, 7 // ICC_IGRPEN1
str lr, [r0, #28]
1:
-//#endif
-
+#endif
+#if defined(CONFIG_MACH_SUN60I_A733)
+ ldr r1, =SUNXI_CPU_DA_DDR_CTRL_REG
+ mov r2, #1
+ str r2, [r1] // CPU_DA_DDR_CTRL_REG = 1 (AXI2HIF)
+ ldr r1, =SUNXI_CLU0_CPU0_CTRL_REG
+ mov r2, #1
+ str r2, [r1] // CLU0_CPU0_CTRL_REG = 1 (AA64 mode)
+#ifdef CONFIG_XPL_BUILD
+ ldr r0, =CONFIG_SPL_TEXT_BASE
+#else
+ ldr r0, =CONFIG_TEXT_BASE
+#endif
+ str r0, [r1, #4] // CLU0_RVBARADDR0_L
+ mov r2, #0
+ str r2, [r1, #8] // CLU0_RVBARADDR0_H = 0
+#else
ldr r1, =CONFIG_SUNXI_RVBAR_ADDRESS
ldr r0, =SUNXI_SRAMC_BASE
ldr r0, [r0, #36] // SRAM_VER_REG
@@ -77,6 +92,7 @@ start32:
ldr r0, =CONFIG_TEXT_BASE
#endif
str r0, [r1] // store start address in RVBAR
+#endif
dsb sy
isb sy
mrc 15, 0, r0, cr12, cr0, 2 @ read RMR register
diff --git a/board/sunxi/board.c b/board/sunxi/board.c
index 299faa890b9..5a5229c7068 100644
--- a/board/sunxi/board.c
+++ b/board/sunxi/board.c
@@ -117,10 +117,14 @@ void i2c_init_board(void)
clock_twi_onoff(5, 1);
sunxi_gpio_set_cfgpin(SUNXI_GPL(0), SUN50I_H616_GPL_R_TWI);
sunxi_gpio_set_cfgpin(SUNXI_GPL(1), SUN50I_H616_GPL_R_TWI);
-#elif CONFIG_MACH_SUN55I_A523
+#elif defined(CONFIG_MACH_SUN55I_A523) || defined(CONFIG_MACH_SUN60I_A733)
clock_twi_onoff(5, 1);
sunxi_gpio_set_cfgpin(SUNXI_GPL(0), SUN50I_GPL_R_TWI);
sunxi_gpio_set_cfgpin(SUNXI_GPL(1), SUN50I_GPL_R_TWI);
+#ifdef CONFIG_MACH_SUN60I_A733
+ sunxi_gpio_set_pull(SUNXI_GPL(0), SUNXI_GPIO_PULL_UP);
+ sunxi_gpio_set_pull(SUNXI_GPL(1), SUNXI_GPIO_PULL_UP);
+#endif
#else
clock_twi_onoff(5, 1);
sunxi_gpio_set_cfgpin(SUNXI_GPL(0), SUN8I_H3_GPL_R_TWI);
@@ -243,7 +247,18 @@ int board_init(void)
static struct boot_file_head * get_spl_header(uint8_t req_version)
{
struct boot_file_head *spl = (void *)(ulong)SPL_ADDR;
- uint8_t spl_header_version = spl->spl_signature[3];
+ u8 spl_header_version;
+
+ /*
+ * In FEL mode on A733, sunxi-fel uses an SRAM swap buffer to preserve
+ * the BROM stack. When returning from SPL to FEL, BROM state is
+ * restored to SPL_ADDR (0x47000), leaving the SPL header in SH_SRAM.
+ */
+ if (IS_ENABLED(CONFIG_MACH_SUN60I_A733) &&
+ memcmp(spl->spl_signature, SPL_SIGNATURE, 3) != 0)
+ spl = (void *)(ulong)SUNXI_FEL_SPL_ADDR;
+
+ spl_header_version = spl->spl_signature[3];
/* Is there really the SPL header (still) there? */
if (memcmp(spl->spl_signature, SPL_SIGNATURE, 3) != 0)
@@ -448,7 +463,7 @@ static void mmc_pinmux_setup(int sdc)
sunxi_gpio_set_drv(pin, 2);
}
#elif defined(CONFIG_MACH_SUN50I_H616) || defined(CONFIG_MACH_SUN50I_A133) || \
- defined(CONFIG_MACH_SUN55I_A523)
+ defined(CONFIG_MACH_SUN55I_A523) || defined(CONFIG_MACH_SUN60I_A733)
/* SDC2: PC0-PC1, PC5-PC6, PC8-PC11, PC13-PC16 */
for (pin = SUNXI_GPC(0); pin <= SUNXI_GPC(16); pin++) {
if (pin > SUNXI_GPC(1) && pin < SUNXI_GPC(5))
diff --git a/common/spl/Kconfig b/common/spl/Kconfig
index 170578ef895..f45115d0be9 100644
--- a/common/spl/Kconfig
+++ b/common/spl/Kconfig
@@ -73,6 +73,7 @@ config SPL_MAX_SIZE
default 0xec00 if OMAP34XX
default 0x10000 if ARCH_MX6 && !MX6_OCRAM_256KB
default 0xbfa0 if MACH_SUN50I_H616 || MACH_SUN50I_A133 || MACH_SUN55I_A523
+ default 0x28fa0 if MACH_SUN60I_A733
default 0x7000 if RCAR_GEN3
default 0x5fa0 if SUNXI_SRAM_ADDRESS = 0x0
default 0x7fa0 if ARCH_SUNXI
@@ -98,6 +99,7 @@ config SPL_DISPOSABLE_INIT_RODATA
config SPL_ACTIVE_MAX_SIZE
hex
depends on SPL_DISPOSABLE_INIT_RODATA
+ default 0x10fa0 if MACH_SUN60I_A733
default 0x0
help
When non-zero, maximum size in bytes of the active SPL image
@@ -302,6 +304,7 @@ config SPL_TEXT_BASE
default 0x40301350 if OMAP54XX
default 0x10060 if MACH_SUN50I || MACH_SUN50I_H5 || MACH_SUN9I
default 0x44060 if MACH_SUN55I_A523
+ default 0x47060 if MACH_SUN60I_A733
default 0x20060 if SUN50I_GEN_H6 || SUNXI_GEN_NCAT2
default 0x00060 if ARCH_SUNXI
default 0xfffc0000 if ARCH_ZYNQMP
@@ -457,6 +460,7 @@ config SPL_STACK
default 0x52a00 if MACH_SUN50I_H616
default 0x40000 if MACH_SUN8I_R528 || MACH_SUN50I_A133
default 0x44000 if MACH_SUN55I_A523
+ default 0x98000 if MACH_SUN60I_A733
default 0x54000 if MACH_SUN50I || MACH_SUN50I_H5
default 0x18000 if MACH_SUN9I
default 0x8000 if ARCH_SUNXI
diff --git a/include/configs/sun60i.h b/include/configs/sun60i.h
new file mode 100644
index 00000000000..8e9488be979
--- /dev/null
+++ b/include/configs/sun60i.h
@@ -0,0 +1,11 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Placeholder wrapper to allow addressing Allwinner devices with Cortex-A76/A55
+ * cores separately. Please do not add anything in here.
+ */
+#ifndef __CONFIG_H
+#define __CONFIG_H
+
+#include <configs/sunxi-common.h>
+
+#endif /* __CONFIG_H */
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 21/26] doc: sunxi: Document Allwinner A733 LPDDR5 PMU architecture
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (19 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 20/26] sunxi: sun60i: initial support for A733 SoC Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 22/26] phy: sunxi: Add Allwinner A733 Cadence Combo PHY driver Alastair D'Silva
` (6 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
Add documentation for the Allwinner A733 LPDDR5 PHY Management Unit
(PMU), covering its clean-room two-team specification and C
reimplementation provenance, hardware architecture, 4-phase Synopsys
mailbox handshake, failure diagnostics, ARCv2 toolchain / PMU_FW /
BINMAN_ALLOW_MISSING Kbuild integration, and SRAM A2 .init.rodata
layout, and update the general sunxi board guide with A523/A733 TF-A
platform targets.
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
doc/board/allwinner/a733-pmu.rst | 299 +++++++++++++++++++++++++++++++
doc/board/allwinner/index.rst | 1 +
doc/board/allwinner/sunxi.rst | 14 +-
3 files changed, 309 insertions(+), 5 deletions(-)
create mode 100644 doc/board/allwinner/a733-pmu.rst
diff --git a/doc/board/allwinner/a733-pmu.rst b/doc/board/allwinner/a733-pmu.rst
new file mode 100644
index 00000000000..1509c2b9a26
--- /dev/null
+++ b/doc/board/allwinner/a733-pmu.rst
@@ -0,0 +1,299 @@
+.. SPDX-License-Identifier: GPL-2.0+
+.. Copyright (C) 2026 Alastair D'Silva <alastair@d-silva.org>
+
+Allwinner A733 LPDDR5 PHY Management Unit (PMU)
+===============================================
+
+Introduction
+------------
+The Allwinner A733 (sun60iw2) SoC introduces a redesigned memory subsystem
+differing substantially from earlier sunxi SoCs (such as the H6, H616, and A523).
+While earlier generations performed DRAM delay line calibration directly on the
+main ARM core, the A733 pairs a Synopsys DesignWare uMCTL2 controller with a
+Synopsys DDR PHY containing an integrated 32-bit ARC EM4 microcontroller,
+termed the PHY Management Unit (PMU).
+
+U-Boot provides a clean-room freestanding C training firmware implementation
+(``arch/arm/mach-sunxi/pmu/``) that is compiled from source during the SPL
+build (or optionally supplied via ``PMU_FW``) and loaded into the ARC EM4 PMU
+at boot.
+
+Clean-Room Specification & Reimplementation Provenance
+------------------------------------------------------
+The ARC EM4 LPDDR5 training firmware in ``arch/arm/mach-sunxi/pmu/`` is an
+original clean-room C implementation authored under ``GPL-2.0+`` using a strict
+two-team clean-room process:
+
+1. **Specification Phase**: Hardware behavior, PHY register apertures, the DCCM
+ message block ABI (``struct pmu_message_block``), the 4-phase Synopsys
+ mailbox handshake, and the stage-by-stage LPDDR5 training state machine were
+ documented as hardware/protocol specifications without transferring vendor
+ source or machine code.
+2. **Independent Implementation Phase**: The 7 freestanding C translation units
+ and headers in ``arch/arm/mach-sunxi/pmu/`` were written independently from
+ those functional specifications and the JEDEC JESD209-5 LPDDR5 standard, and
+ validated against physical Allwinner A733 hardware.
+
+Background & Theory of Operation
+--------------------------------
+
+Why High-Speed LPDDR5 Requires Training
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+On the A733 at 3600 MT/s (1800 MHz WCK), each individual data bit (DQ) is valid
+for only ~278 picoseconds (the unit interval or "data eye").
+
+Because copper PCB traces have slightly different physical lengths, and silicon
+transistors inside the SoC and DRAM vary with manufacturing process, operating
+voltage, and temperature (PVT), signals arrive at slightly different times.
+If the memory controller sampled data with fixed delays, electrical noise and
+clock skew would corrupt reads and writes immediately.
+
+DRAM training electronically calibrates:
+
+* **Timing Deskew**: Adjusting per-bit programmable Local Clock Delay Lines
+ (LCDLs) so all bits (DQ0..DQ7, DM) and byte strobes (DQS/DQSN) arrive aligned
+ within the sampling window.
+* **Vref Voltage Centering**: Adjusting internal DAC reference voltages to place
+ the logic 0/1 decision threshold at the vertical center of the eye.
+* **Command/Address (CA) Training**: Synchronizing command lines with the memory
+ clock via Command Bus Training (CBT) and Low-Power CA (LPCA) training.
+* **ZQ Impedance Calibration**: Matching output driver (Ron) and on-die termination
+ (ODT) impedance against an external 240-ohm precision calibration resistor to
+ suppress transmission line reflections.
+
+Role of the Integrated ARC PMU Core
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Performing microsecond-precision analog delay line sweeps from the host ARM CPU
+would require millions of slow cross-domain APB bus round-trips. Instead, the
+PHY integrates a dedicated 32-bit ARC EM4 RISC core (the PMU) with direct
+access to the PHY CSR bus, allowing it to sweep timing and voltage envelopes,
+evaluate BIST patterns, and commit calibrated delays autonomously.
+
+Hardware Architecture
+---------------------
+The memory subsystem consists of:
+
+* **Synopsys DesignWare uMCTL2**: Dual-channel memory controller managing queuing,
+ scheduling, refresh, low-power states, and AXI port arbitration.
+* **Synopsys DesignWare DDR PHY**: High-speed physical layer interface supporting
+ LPDDR5/LPDDR5X and LPDDR4X across two independent 16-bit channels.
+* **ARC EM4 PMU Core**: Embedded 32-bit RISC core with Harvard architecture:
+
+ - **IMEM (Instruction Memory / ICCM)**: 64 KiB tightly coupled SRAM.
+ - **DMEM (Data Memory / DCCM)**: 64 KiB tightly coupled SRAM.
+ - **APB Master & Slave Interfaces**: Multiplexed PHY CSR bus shared between
+ the host ARM CPU (when ``PMU_REG_APB_MUX == 0``) and the ARC EM4 core
+ (when ``PMU_REG_APB_MUX == 1``).
+
+PHY Silicon Architecture & Slices
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Inside the Synopsys DesignWare DDR PHY on the Allwinner A733:
+
+* **DBYTE Slices (Data Byte Slices 0..3)**:
+ Each DBYTE slice manages one 8-bit byte lane (DQ0..DQ7, DMI, and differential
+ strobe WCK/RDQS). Across the two 16-bit channels there are 4 DBYTE slices:
+
+ - **Slice 0**: Channel 0 Byte Lane 0 (DQ0..DQ7)
+ - **Slice 1**: Channel 0 Byte Lane 1 (DQ8..DQ15)
+ - **Slice 2**: Channel 1 Byte Lane 0 (DQ0..DQ7)
+ - **Slice 3**: Channel 1 Byte Lane 1 (DQ8..DQ15)
+
+* **AC Slices (Address/Command Slices 0..1)**:
+ Manage differential clocks (CK_t/CK_c), chip selects, and Command/Address
+ (CA0..CA6) lines for each channel.
+* **PUB (PHY Utility Block)**:
+ Central control block housing master PLL configuration, clock gating, PMU
+ mailboxes, analog bias generators, and the host APB mux.
+* **LCDL (Local Clock Delay Line)**:
+ Tapped analog delay lines at each pin driver and receiver compensating for
+ PCB trace skew and PVT variation.
+
+Memory Map & Addressing
+-----------------------
+
+PMU Microsequencer Register Offsets (PUB)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+All PUB register accesses over the host APB interface are 16-bit wide
+(``readw`` / ``writew``) relative to ``SUNXI_DRAM_PHY_CTRL_BASE``
+(``0x0aaa0000``):
+
+=========================== ================== ========================================
+Register Name Offset Description
+=========================== ================== ========================================
+``PMU_REG_APB_MUX`` ``0x0000`` 0 = Host ARM owns APB; 1 = PMU ARC owns
+``PMU_REG_POLL_STAT`` ``0x0008`` ``UctShadowRegs`` bit 0: 0 = msg pending
+``PMU_REG_HANDOFF_0`` ``0x0060`` Handshake status 0
+``PMU_REG_HANDOFF_1`` ``0x0062`` ``DctWriteProt`` (0 = ack, 1 = release)
+``PMU_REG_STATUS_LO`` ``0x0064`` ``UctWriteOnlyShadow`` (status [15:0])
+``PMU_REG_MAILBOX_INT`` ``0x0066`` ``UctDatWriteOnlyShadow`` / init trigger
+``PMU_REG_STATUS_HI`` ``0x0068`` Status [31:16]
+``PMU_REG_RESET`` ``0x0132`` ``MicroReset`` (bit 0: stall, bit 3: rst)
+=========================== ================== ========================================
+
+Related System Controllers:
+
+* **CCU Base (0x02002000)**:
+
+ - ``PLL_DDR_CTRL_REG`` (``0x02002020``): DRAM subsystem PLL control, programmed
+ from ``CONFIG_DRAM_CLK`` in ``dram_sun60i_a733.c``.
+ - ``MBUS_CLK_REG`` (``0x02002580``): MBUS clock control.
+ - ``DRAM0_CLK_REG`` (``0x02002c00``): DRAM0 module clock control.
+ - ``DRAM0_BGR_REG`` (``0x02002c0c``): DRAM0 bus gating and reset.
+
+* **STBY_PRCM Base (0x07010000)**:
+
+ - ``VCC_DRAM_ISO_REG`` (``0x07010250``): DRAM power domain isolation release.
+ - ``CH0_DMC_CLK_REG`` (``0x07010310``): DMC clock and reset pulse control.
+
+* **CPUCFG Base (0x08000000)**:
+
+ - ``SUNXI_CPU_DA_DDR_CTRL_REG`` (``0x08000200``): CPU Direct Access DDR path
+ mux, set to ``1`` (AXI2HIF) early in ``boot0.h`` and ``rmr_switch.S`` prior
+ to the AArch32-to-AArch64 warm reset.
+
+Address Mapping from Host ARM Core
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The host ARM CPU accesses the DDR PHY and PMU SRAMs over the internal APB
+interconnect:
+
+======================= ================== ========================================
+Region ARM Physical Base ARC EM4 Local Address
+======================= ================== ========================================
+PMU IMEM (ICCM, 64 KiB) ``0x0a9a0000`` ``0x00000000`` (vector table and code)
+PMU DMEM (DCCM, 64 KiB) ``0x0a9b0000`` ``0x80000000`` (message block & state)
+PHY Control Registers ``0x0aaa0000`` Internal CSR bus via APB master
+uMCTL2 Channel 0 ``0x0a110000`` N/A (Host ARM only)
+uMCTL2 Channel 1 ``0x0a510000`` N/A (Host ARM only)
+======================= ================== ========================================
+
+Training Sequence & Firmware State Machine
+------------------------------------------
+
+Training Sequence Flags (SequenceCtrl)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The host specifies the requested calibration stages via ``sequence_ctrl``
+(offset ``0x10`` in ``struct pmu_message_block``):
+
+=========================== ====== ====================================================
+Flag Name Bit Description
+=========================== ====== ====================================================
+``PMU_SEQ_DEV_INIT`` Bit 0 Device init, PLL lock, CBT entry, ZQ calibration
+``PMU_SEQ_STAGE_VREF`` Bit 1 Receiver Vref voltage centering sweep
+``PMU_SEQ_STAGE_DCD`` Bit 2 Duty cycle distortion (DCD) correction
+``PMU_SEQ_STAGE_DQS`` Bit 3 Read/Write DQS strobe timing centering
+``PMU_SEQ_BIST_SEARCH_WIN`` Bit 4 BIST test pattern search window setup
+``PMU_SEQ_SEARCH_WIN`` Bit 6 High-resolution 2D eye margin search
+``PMU_SEQ_ZQ_CAL`` Bit 7 Standalone ZQ pad impedance calibration
+``PMU_SEQ_DESKEW_SWEEP`` Bit 8 Per-bit DQ deskew delay line tuning
+``PMU_SEQ_MARGIN_STEP`` Bit 9 Margin envelope boundary evaluation
+``PMU_SEQ_LANE_WIN_SWEEP`` Bit 10 Lane window sweep & cross-lane matrix scan
+``PMU_SEQ_LPCA_INIT`` Bit 12 Low-Power Command/Address (LPCA) training
+=========================== ====== ====================================================
+
+SPL invokes ``PMU_SEQ_FULL_TRAIN`` (``0x125f``: bits 0, 1, 2, 3, 4, 6, 9, 12)
+during cold boot. ``sunxi_dram_init()`` first attempts dual-rank training
+(``rank_mask = 0x03``); if unpopulated Rank 1 causes training to fail on a
+single-rank SKU, SPL pulses the DMC reset and automatically retries single-rank
+training (``rank_mask = 0x01``).
+
+Execution & 4-Phase Mailbox Handshake
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+1. **Firmware & Message Block Loading**:
+ ``mctl_phy_init()`` invokes ``mctl_pmu_load_sram()`` to set
+ ``PMU_REG_APB_MUX = 0`` (host owns APB) and write all 32,768 16-bit
+ halfwords (64 KiB) of IMEM (``0x0a9a0000``), applies pre-DMEM PHY
+ configuration (``dram_phy_pre_dmem_regs``), loads DMEM (``0x0a9b0000``)
+ with ``struct pmu_message_block`` (deriving ``dram_rate_mts`` from
+ ``CONFIG_DRAM_CLK``), and programs the CA/DQ pin swizzle via
+ ``mctl_phy_load_pin_swizzle()`` (when ``CONFIG_DRAM_SUN60I_A733_PIN_SWIZZLE=y``)
+ while initializing SRAM parity bits across the full 64 KiB apertures.
+2. **ARC Reset Release**:
+ ``mctl_pmu_run_training()`` switches ``PMU_REG_APB_MUX = 1`` (PMU owns APB),
+ sequences ``PMU_REG_RESET`` through ``0x0009 -> 0x0001 -> 0x0001 -> 0x0000``
+ (asserting ``ResetToMicro`` + ``StallToMicro``, clearing ``ResetToMicro``,
+ then releasing ``StallToMicro``), sets ``PMU_REG_HANDOFF_1 = 1`` and
+ ``PMU_REG_MAILBOX_INT = 1``, and waits for ``PMU_REG_POLL_STAT`` bit 0 to
+ rise to ``1``.
+3. **4-Phase Synopsys Mailbox Loop**:
+ Whenever the ARC core posts a streaming or terminal mailbox message, it
+ writes the status word and clears ``UctWriteProt`` so ``PMU_REG_POLL_STAT``
+ bit 0 drops to ``0``:
+
+ - Host reads ``status`` from ``PMU_REG_STATUS_LO`` and ``PMU_REG_STATUS_HI``.
+ - Host writes ``0x0000`` to ``PMU_REG_HANDOFF_1`` (``DctWriteProt = 0``) to
+ acknowledge receipt.
+ - Host polls ``PMU_REG_POLL_STAT`` until the ARC core sets ``UctWriteProt = 1``
+ (bit 0 returns to ``1``).
+ - Host writes ``0x0001`` to ``PMU_REG_HANDOFF_1`` (``DctWriteProt = 1``) to
+ re-arm the mailbox.
+ - If ``status == PMU_STATUS_SUCCESS`` (``0x00000007``),
+ ``mctl_pmu_run_training()`` stalls the ARC core (``PMU_REG_RESET = 1``)
+ and ``mctl_phy_init()`` reclaims APB ownership (``PMU_REG_APB_MUX = 0``)
+ immediately upon return.
+ - If ``status == PMU_STATUS_FATAL_ERROR`` (``0x000000ff``), host stalls the
+ ARC core (``PMU_REG_RESET = 1``), reclaims APB ownership
+ (``PMU_REG_APB_MUX = 0``), logs diagnostics via ``mctl_pmu_log_error()``
+ (when ``log_err`` is true, i.e., not a speculative dual-rank probe), and
+ aborts the current rank attempt.
+ - Intermediate messages (such as ``0x08`` streaming traces when
+ ``post_threshold != 0xff``) are logged via ``debug()`` and the loop
+ continues until terminal status.
+
+Diagnostics & Telemetry
+-----------------------
+* **Failure Diagnostics (``mctl_pmu_log_error``)**:
+ On a fatal assertion or timeout, SPL stalls the ARC core
+ (``PMU_REG_RESET = 1``), switches the APB mux back to host ARM access
+ (``PMU_REG_APB_MUX = 0``), and reads both ``train_mode`` (DMEM offset
+ ``0x14``) and ``assert_code`` (DMEM offset ``0x0c``) before printing an
+ error diagnostic.
+* **Trained Slice Inspection (``mctl_phy_dump_slices``)**:
+ After successful training, SPL logs calibrated delay registers across DBYTE
+ slices 0..3 via ``debug()`` while host APB access is active.
+
+Build System, Toolchain & SRAM Layout
+-------------------------------------
+The PMU firmware source resides in ``arch/arm/mach-sunxi/pmu/``:
+
+* ``pmu_primitives.c``: Reset vector, MMIO helpers, delays, and runtime primitives.
+* ``pmu_clock.c``: PHY PLL control, bounded clock-gate handoff, and ratio math.
+* ``pmu_telemetry.c``: 4-phase host mailbox IPC and diagnostic trace dispatch.
+* ``pmu_cbt.c``: Command Bus Training (CBT) and LPCA calibration routines.
+* ``pmu_deskew.c``: DBYTE & AC slice LCDL delay sweeps and BIST engines.
+* ``pmu_eye.c``: 1D/2D eye margin search and Vref/delay centroid calculation.
+* ``pmu_train.c``: Top-level firmware entry and ``sequence_ctrl`` state machine.
+* ``pmu_fw.h``, ``dwc_ddrphy_pmu_regs.h``, ``sunxi_pmu_abi.h``, ``pmu.ld``:
+ Internal prototypes, PHY register definitions, host/ARC ABI, and linker script.
+* ``build_pmu_fw.py``: Standalone builder and C header generator.
+
+Kbuild Variables & Toolchain Resolution
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+During SPL builds with ``CONFIG_DRAM_SUN60I_A733=y``, ``arch/arm/mach-sunxi/Makefile``
+generates ``spl/arch/arm/mach-sunxi/dram_sun60i_a733_pmu_fw.h`` in the build
+tree via ``$(call if_changed,pmu_fw)``:
+
+* **In-tree source build (default)**: ``build_pmu_fw.py`` locates an ARCv2 GCC
+ toolchain via ``ARC_CROSS_COMPILE``, ``~/.buildman`` (or
+ ``/home/uboot/.buildman``) ``[toolchain]`` / ``[toolchain-prefix]`` entries,
+ standard ``buildman-toolchains`` / ``/opt/gcc-*-nolibc/arc-linux/bin/``
+ paths, or ``arc-elf32-`` / ``arc-linux-`` prefixes on ``PATH``.
+* **Pre-built firmware override (``PMU_FW``)**: Passing
+ ``make PMU_FW=/path/to/pmu_fw.bin`` wraps an existing raw ARC PMU binary
+ directly without invoking an ARC cross-compiler.
+* **Missing-blob CI fallback (``BINMAN_ALLOW_MISSING=1``)**: When building on a
+ host without an ARCv2 compiler and ``BINMAN_ALLOW_MISSING=1`` (or
+ ``ALLOW_MISSING=1``) is set, ``build_pmu_fw.py`` emits a zero-filled stub
+ marked with ``SUN60I_A733_PMU_FW_MISSING`` so ``arm64`` compile-test CI
+ completes cleanly without errors; booting such a stub image prints a clear
+ runtime error in SPL.
+
+Disposable ``.init.rodata`` & SRAM A2 Layout
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+On the A733, SPL runs in SRAM A2 starting at ``0x00047000`` while Trusted
+Firmware-A ``BL31`` loads at ``CONFIG_SUNXI_BL31_BASE`` (``0x00058000``).
+``MACH_SUN60I_A733`` enables ``CONFIG_SPL_DISPOSABLE_INIT_RODATA``, placing the
+~52 KiB ``sun60i_a733_pmu_imem`` array and one-time PHY register tables into
+``.init.rodata`` *after* ``_image_binary_end``. ``CONFIG_SPL_ACTIVE_MAX_SIZE``
+(``0x10fa0``) enforces at link time that persistent SPL code, rodata, and data
+remain below ``0x00058000``, allowing ``BL31`` to safely reclaim the
+``.init.rodata`` tail once ``sunxi_dram_init()`` has finished training DRAM.
diff --git a/doc/board/allwinner/index.rst b/doc/board/allwinner/index.rst
index 7352ccd5c0a..6e89552f3d3 100644
--- a/doc/board/allwinner/index.rst
+++ b/doc/board/allwinner/index.rst
@@ -7,3 +7,4 @@ Allwinner (sunxi) boards
:maxdepth: 2
sunxi
+ a733-pmu
diff --git a/doc/board/allwinner/sunxi.rst b/doc/board/allwinner/sunxi.rst
index 5c2406d3959..b6389490b26 100644
--- a/doc/board/allwinner/sunxi.rst
+++ b/doc/board/allwinner/sunxi.rst
@@ -35,9 +35,9 @@ For more details, and alternative boot locations or installations, see below.
Building Arm Trusted Firmware (TF-A)
------------------------------------
-Boards using a 64-bit Soc (A64, H5, H6, H616, R329) require the BL31 stage of
-the `Arm Trusted Firmware-A`_ firmware. This provides the reference
-implementation of secure software for Armv8-A, offering PSCI and SMCCC
+Boards using a 64-bit SoC (A64, H5, H6, H616, R329, A523, A733) require the
+BL31 stage of the `Arm Trusted Firmware-A`_ firmware. This provides the
+reference implementation of secure software for Armv8-A, offering PSCI and SMCCC
services. Allwinner support is fully mainlined. To build bl31.bin::
$ git clone https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git
@@ -46,7 +46,8 @@ services. Allwinner support is fully mainlined. To build bl31.bin::
$ export BL31=$(pwd)/build/sun50i_a64/debug/bl31.bin
The target platform (``PLAT=``) for A64 and H5 SoCs is sun50i_a64, for the H6
-sun50i_h6, for the H616 sun50i_h616, and for the R329 sun50i_r329. Use::
+sun50i_h6, for the H616 sun50i_h616, for the R329 sun50i_r329, for the A523
+sun55i_a523, and for the A733 sun60i_a733 (pending upstream TF-A support). Use::
$ find plat/allwinner -name platform.mk
@@ -97,7 +98,10 @@ build command line::
$ make BL31=/src/tf-a.git/build/sun50i_h616/debug/bl31.bin
-The same applies to the (optional) SCP firmware.
+The same applies to the (optional) SCP firmware. Boards based on the Allwinner
+A733 additionally compile the LPDDR5 PHY Management Unit (PMU) firmware during
+the SPL build using an ARCv2 GCC cross-compiler (``ARC_CROSS_COMPILE``) or an
+external ``PMU_FW`` binary; see :doc:`a733-pmu` for details.
The file containing everything you need is called ``u-boot-sunxi-with-spl.bin``,
you will find it in the root folder of your U-Boot (build) tree. Except for
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 22/26] phy: sunxi: Add Allwinner A733 Cadence Combo PHY driver
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (20 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 21/26] doc: sunxi: Document Allwinner A733 LPDDR5 PMU architecture Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 23/26] pci: sunxi: Add Allwinner A733 DesignWare PCIe host controller driver Alastair D'Silva
` (5 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
Add a UCLASS_PHY driver for the Allwinner A733 (sun60i) SerDes subsystem
and Cadence Combo PHY 1 in PCIe mode, handling clock/reset enablement,
analog tuning, and PLL lock polling.
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
arch/arm/include/asm/arch-sunxi/clock.h | 1 +
arch/arm/mach-sunxi/clock_sun50i_h6.c | 68 +++++
board/sunxi/board.c | 4 +
drivers/clk/sunxi/clk_a733.c | 6 +
drivers/phy/allwinner/Kconfig | 8 +
drivers/phy/allwinner/Makefile | 1 +
drivers/phy/allwinner/phy-sun60i-combophy.c | 293 ++++++++++++++++++++
7 files changed, 381 insertions(+)
create mode 100644 drivers/phy/allwinner/phy-sun60i-combophy.c
diff --git a/arch/arm/include/asm/arch-sunxi/clock.h b/arch/arm/include/asm/arch-sunxi/clock.h
index fcc8966cb0b..444870997d6 100644
--- a/arch/arm/include/asm/arch-sunxi/clock.h
+++ b/arch/arm/include/asm/arch-sunxi/clock.h
@@ -35,6 +35,7 @@ void clock_set_de_mod_clock(u32 *clk_cfg, unsigned int hz);
void clock_init_safe(void);
void clock_init_sec(void);
void clock_init_uart(void);
+void sunxi_a733_pcie_soc_init(void);
#endif
#endif /* _SUNXI_CLOCK_H */
diff --git a/arch/arm/mach-sunxi/clock_sun50i_h6.c b/arch/arm/mach-sunxi/clock_sun50i_h6.c
index 517fdcb56ea..21ba3491919 100644
--- a/arch/arm/mach-sunxi/clock_sun50i_h6.c
+++ b/arch/arm/mach-sunxi/clock_sun50i_h6.c
@@ -331,3 +331,71 @@ unsigned int clock_get_pll6(void)
return 24000000U * n / m / div1 / div2;
}
+
+#ifndef CONFIG_XPL_BUILD
+#ifdef CONFIG_MACH_SUN60I_A733
+#define A733_RTC_XO_CTRL1_OFF 0x016c
+#define A733_RTC_XO_CLK_OUT_EN (BIT(9) | BIT(8) | BIT(5) | BIT(4))
+#define A733_CCU_MBUS_CLK_REG 0x0574
+#define A733_CCU_MBUS_MAT_CLK_GATE_REG 0x0580
+#define A733_CCU_MBUS_MAT_SERDES_EN (BIT(31) | BIT(30) | BIT(25) | BIT(24))
+#define A733_CCU_MBUS_SERDES_GATE_REG 0x0584
+#define A733_CCU_MBUS_SERDES_CFG_REG 0x0588
+#define A733_CCU_MBUS_CLK_CFG0_REG 0x058c
+#define A733_CCU_MBUS_CLK_CFG1_REG 0x05b4
+#define A733_CCU_PCIE_AXI_SLV_REG 0x1384
+#define A733_CCU_SERDES_PHY_CFG_REG 0x13c0
+#define A733_CCU_CM_PCIE0_MODE_REG 0x1b28
+
+#define A733_SYS_PLL_CTRL_REG 0x02001540UL
+#define A733_SYS_PLL_BGR_REG 0x0200154cUL
+#define A733_NSI_BASE 0x02020000UL
+#define A733_NSI_PORT_STRIDE 0x200UL
+#define A733_NSI_NUM_PORTS 16
+#define A733_PCIE_ITS_AXI_REG 0x08868020UL
+
+/*
+ * Configure SoC-level RTC DCXO feed, CCU muxes/dividers, MBUS/SYS_PLL gates,
+ * PCIe ITS AXI bridge, and NSI QoS not yet modeled by U-Boot's gate-only
+ * sunxi CCU/RTC/NSI drivers.
+ */
+void sunxi_a733_pcie_soc_init(void)
+{
+ void *const rtc = (void *)(ulong)SUNXI_RTC_BASE;
+ void *const ccm = (void *)(ulong)SUNXI_CCM_BASE;
+ int p;
+
+ /* Enable RTC DCXO clock feed to Combo PHY without clobbering siblings */
+ setbits_le32(rtc + A733_RTC_XO_CTRL1_OFF, A733_RTC_XO_CLK_OUT_EN);
+
+ /* Configure PCIe/SerDes CCU muxes/dividers and MBUS/NSI gating */
+ writel(0x82000000, ccm + A733_CCU_PCIE_AXI_SLV_REG); /* 400 MHz */
+ writel(0x81000005, ccm + A733_CCU_SERDES_PHY_CFG_REG); /* 100 MHz */
+ writel(0x00020001, ccm + A733_CCU_CM_PCIE0_MODE_REG);
+ writel(0x00010002, ccm + A733_CCU_MBUS_CLK_REG);
+ setbits_le32(ccm + A733_CCU_MBUS_MAT_CLK_GATE_REG,
+ A733_CCU_MBUS_MAT_SERDES_EN);
+ writel(0x00010001, ccm + A733_CCU_MBUS_SERDES_GATE_REG);
+ writel(0x81000000, ccm + A733_CCU_MBUS_SERDES_CFG_REG);
+ setbits_le32(ccm + A733_CCU_MBUS_CLK_CFG0_REG, 0x00010007);
+ setbits_le32(ccm + A733_CCU_MBUS_CLK_CFG1_REG, 0x00010007);
+
+ writel(0xc0000000, (void *)(ulong)A733_SYS_PLL_CTRL_REG);
+ writel(0x00010001, (void *)(ulong)A733_SYS_PLL_BGR_REG);
+
+ /* Configure NSI interconnect QoS & bandwidth across ports 0..15 */
+ for (p = 0; p < A733_NSI_NUM_PORTS; p++) {
+ void *port = (void *)(ulong)(A733_NSI_BASE +
+ A733_NSI_PORT_STRIDE * p);
+
+ writel(0x00000001, port + 0x0c);
+ writel(0x00000001, port + 0x18);
+ writel(0x000003e8, port + 0x28);
+ writel(0x000000ff, port + 0x2c);
+ writel(0x00000001, port + 0xcc);
+ }
+
+ writel(0x00ff00ff, (void *)(ulong)A733_PCIE_ITS_AXI_REG);
+}
+#endif
+#endif
diff --git a/board/sunxi/board.c b/board/sunxi/board.c
index 5a5229c7068..6b4140ace7d 100644
--- a/board/sunxi/board.c
+++ b/board/sunxi/board.c
@@ -235,6 +235,10 @@ int board_init(void)
eth_init_board();
+ if (IS_ENABLED(CONFIG_MACH_SUN60I_A733) &&
+ IS_ENABLED(CONFIG_PCIE_DW_SUNXI))
+ sunxi_a733_pcie_soc_init();
+
return 0;
}
diff --git a/drivers/clk/sunxi/clk_a733.c b/drivers/clk/sunxi/clk_a733.c
index 5c942fd5d58..4b0aaa7ab2b 100644
--- a/drivers/clk/sunxi/clk_a733.c
+++ b/drivers/clk/sunxi/clk_a733.c
@@ -20,6 +20,7 @@ static const struct ccu_clk_gate a733_gates[] = {
[CLK_APB_UART] = GATE_DUMMY,
[CLK_MBUS_GMAC0] = GATE(0x05e4, BIT(11)),
+ [CLK_MBUS_SERDES] = GATE(0x0584, BIT(0)),
[CLK_BUS_CE] = GATE(0x0ac4, BIT(0)),
[CLK_MMC0] = GATE(0x0d00, BIT(31)),
@@ -71,6 +72,9 @@ static const struct ccu_clk_gate a733_gates[] = {
[CLK_BUS_OHCI1] = GATE(0x130c, BIT(0)),
[CLK_BUS_EHCI1] = GATE(0x130c, BIT(4)),
+ [CLK_SERDES_PHY] = GATE(0x13c0, BIT(31)),
+ [CLK_AHB_SERDES] = GATE(0x13c4, BIT(0)),
+
[CLK_GMAC0_PHY] = GATE(0x1410, BIT(31)),
[CLK_BUS_GMAC0] = GATE(0x141c, BIT(0)),
};
@@ -120,6 +124,8 @@ static const struct ccu_reset a733_resets[] = {
[RST_BUS_OHCI1] = RESET(0x130c, BIT(16)),
[RST_BUS_EHCI1] = RESET(0x130c, BIT(20)),
+ [RST_BUS_SERDES] = RESET(0x13c4, BIT(16)),
+
[RST_BUS_GMAC0] = RESET(0x141c, BIT(16)),
[RST_BUS_GMAC0_AXI] = RESET(0x141c, BIT(17)),
};
diff --git a/drivers/phy/allwinner/Kconfig b/drivers/phy/allwinner/Kconfig
index bb0bd8d5f81..a408634e35b 100644
--- a/drivers/phy/allwinner/Kconfig
+++ b/drivers/phy/allwinner/Kconfig
@@ -33,3 +33,11 @@ config PHY_SUN50I_USB3
help
Enable this to support the USB3 transceiver that is part of
Allwinner sun50i SoCs.
+
+config PHY_SUN60I_COMBOPHY
+ bool "Allwinner sun60i A733 Combo PHY driver"
+ depends on MACH_SUN60I_A733
+ select PHY
+ help
+ Enable this to support the Cadence Combo PHY (PCIe / USB3)
+ on Allwinner sun60i (A733) SoCs.
diff --git a/drivers/phy/allwinner/Makefile b/drivers/phy/allwinner/Makefile
index f2b60ce1a64..84c653db53a 100644
--- a/drivers/phy/allwinner/Makefile
+++ b/drivers/phy/allwinner/Makefile
@@ -5,3 +5,4 @@
obj-$(CONFIG_PHY_SUN4I_USB) += phy-sun4i-usb.o
obj-$(CONFIG_PHY_SUN50I_USB3) += phy-sun50i-usb3.o
+obj-$(CONFIG_PHY_SUN60I_COMBOPHY) += phy-sun60i-combophy.o
diff --git a/drivers/phy/allwinner/phy-sun60i-combophy.c b/drivers/phy/allwinner/phy-sun60i-combophy.c
new file mode 100644
index 00000000000..0e3128ab0ed
--- /dev/null
+++ b/drivers/phy/allwinner/phy-sun60i-combophy.c
@@ -0,0 +1,293 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Allwinner A733 (sun60i) Cadence Combo PHY driver
+ *
+ * Copyright (c) 2026 Armbian Project
+ */
+
+#include <clk.h>
+#include <dm.h>
+#include <dm/device_compat.h>
+#include <dt-bindings/phy/phy.h>
+#include <generic-phy.h>
+#include <linux/bitops.h>
+#include <linux/iopoll.h>
+#include <reset.h>
+#include <asm/io.h>
+
+/* SerDes subsystem register offsets (relative to serdes base) */
+#define SERDES_PCIE_BGR_REG 0x0004
+#define SERDES_PCIE_BGR_DEASSERT (BIT(18) | BIT(17) | BIT(16) | \
+ BIT(1) | BIT(0))
+#define SERDES_CTRL0_REG 0x0010
+#define SERDES_CTRL0_PCIE_EN BIT(0)
+#define SERDES_PCIE_APP_SUB_CTRL_REG 0x0020
+#define SERDES_PCIE_APP_CLK_EN BIT(0)
+#define SERDES_CTRL1_REG 0x00f0
+#define SERDES_CTRL1_PCIE_MODE (BIT(29) | BIT(10))
+#define SERDES_PCIE_ITS_TAGT_ADDR_REG 0x0220
+#define SERDES_PCIE_ITS_TAGT_DEFAULT 0x00000345
+#define SERDES_AXI2TO1_TH_REG 0x0300
+#define SERDES_AXI2TO1_TH_DEFAULT 0x00ff00ff
+#define SERDES_COMB1_PIPE_REG 0x6c44
+#define SERDES_COMB1_PIPE_PCIE_SEL BIT(0)
+
+/* Combo PHY1 Top register offsets (serdes base + 0x2000) */
+#define COMBOPHY_TOP_OFFSET 0x2000
+#define COMBOPHY_LINK_RST_REG 0x0000
+#define COMBOPHY_LINK_RST_N BIT(0)
+#define COMBOPHY_CLK_CTRL_REG 0x0004
+#define COMBOPHY_CLK_REF_EN (BIT(24) | BIT(20) | BIT(0))
+#define COMBOPHY_CLK_PIPE_OUT_EN BIT(28)
+#define COMBOPHY_LANE0_RST_REG 0x0100
+#define COMBOPHY_LANE0_PCIE_RST_N BIT(0)
+#define COMBOPHY_PLL_STATUS_REG 0x0900
+#define COMBOPHY_PLL_LOCK_BIT BIT(0)
+
+/* Combo PHY1 Analog register offsets */
+#define COMBOPHY_ANALOG_TX_CTRL0 0x0098
+#define COMBOPHY_ANALOG_TX_CTRL1 0x00a0
+#define COMBOPHY_ANALOG_TX_CTRL2 0x00a8
+#define COMBOPHY_ANALOG_TX_CTRL3 0x00d8
+#define COMBOPHY_ANALOG_TX_IDLE_MODE 0x0002
+#define COMBOPHY_ANALOG_TX_POST_PLL 0x0270
+#define COMBOPHY_ANALOG_TX_EN_BIT BIT(4)
+#define COMBOPHY_ANALOG_EXT_OFFSET 0x18000
+
+struct sun60i_combophy_priv {
+ void __iomem *serdes_base;
+ void __iomem *top_base;
+ void __iomem *phy_base;
+ struct clk_bulk clks;
+ struct reset_ctl_bulk rsts;
+};
+
+struct phy_reg_val {
+ u32 off;
+ u16 val;
+};
+
+/*
+ * Vendor-characterized Cadence Combo PHY analog/PLL/CDR register tuning table
+ * for PCIe Gen3 (8.0 GT/s) operation on Allwinner A733.
+ */
+static const struct phy_reg_val cphy_analog_regs[] = {
+ { 0x44, 0x1a },
+ { 0x54, 0x34 }, { 0x58, 0xda },
+ { 0x64, 0x34 }, { 0x68, 0xda },
+ { 0xc8, 0x82 }, { 0xca, 0x82 },
+ { 0xe8, 0x1a },
+ { 0x208, 0x20 }, { 0x20a, 0x7 },
+ { 0x218, 0x20 }, { 0x21a, 0x7 },
+ { 0x228, 0x30c }, { 0x22a, 0x7 },
+ { 0x248, 0x7 }, { 0x24a, 0x3 }, { 0x24c, 0xf }, { 0x250, 0x132 },
+ { 0x8180, 0x208 }, { 0x8184, 0x9c },
+ { 0x10088, 0x1a }, { 0x1008a, 0x82 },
+ { 0x10098, 0x1a }, { 0x1009a, 0x82 },
+ { 0x8246, 0xa28 },
+ { 0x128, 0x4 }, { 0x148, 0x4 }, { 0x1a8, 0x4 },
+ { 0x348, 0x509 }, { 0x368, 0x509 }, { 0x388, 0x509 },
+ { 0x34a, 0xf00 }, { 0x36a, 0xf00 }, { 0x38a, 0xf00 },
+ { 0x34c, 0xf08 }, { 0x36c, 0xf08 }, { 0x38c, 0xf08 },
+ { 0x120, 0x180 }, { 0x140, 0x133 }, { 0x1a0, 0x133 },
+ { 0x122, 0x9d8a }, { 0x142, 0xb13b }, { 0x1a2, 0xb13b },
+ { 0x124, 0x2 }, { 0x144, 0x2 }, { 0x1a4, 0x2 },
+ { 0x126, 0x102 }, { 0x146, 0xce }, { 0x1a6, 0xce },
+ { 0x340, 0x22 }, { 0x360, 0x22 }, { 0x380, 0x22 },
+ { 0x130, 0x1 }, { 0x150, 0x1 }, { 0x1b0, 0x1 },
+ { 0x132, 0x45f }, { 0x152, 0x2f0 }, { 0x1b2, 0x399 },
+ { 0x134, 0x6b }, { 0x154, 0x68 }, { 0x1b4, 0x68 },
+ { 0x136, 0x4 }, { 0x156, 0x4 }, { 0x1b6, 0x4 },
+ { 0x108, 0x104 }, { 0x188, 0x104 },
+ { 0x10a, 0x5 }, { 0x18a, 0x5 },
+ { 0x10c, 0x337 }, { 0x18c, 0x337 },
+ { 0x110, 0x3dbe }, { 0x190, 0x3dbe },
+ { 0x104, 0x3 }, { 0x184, 0x3 },
+ { 0x138, 0x14 }, { 0x1b8, 0x14 },
+ { 0x13c, 0x192 }, { 0x1bc, 0x192 },
+ { 0x13e, 0x6 }, { 0x1be, 0x6 },
+ { 0x103c0, 0x0 },
+ { 0x102e2, 0x19 }, { 0x102e4, 0x19 },
+ { 0x103fe, 0x1 },
+};
+
+static int sun60i_combophy_xlate(struct phy *phy,
+ struct ofnode_phandle_args *args)
+{
+ if (args->args_count != 1)
+ return -EINVAL;
+
+ if (args->args[0] != PHY_TYPE_PCIE) {
+ dev_err(phy->dev, "Unsupported PHY type %u\n", args->args[0]);
+ return -EINVAL;
+ }
+
+ phy->id = args->args[0];
+
+ return 0;
+}
+
+static int sun60i_combophy_init(struct phy *phy)
+{
+ struct sun60i_combophy_priv *priv = dev_get_priv(phy->dev);
+ int ret;
+
+ ret = clk_enable_bulk(&priv->clks);
+ if (ret)
+ return ret;
+
+ ret = reset_deassert_bulk(&priv->rsts);
+ if (ret) {
+ clk_disable_bulk(&priv->clks);
+ return ret;
+ }
+
+ /* Configure SerDes subsystem for PCIe */
+ writel(SERDES_PCIE_BGR_DEASSERT, priv->serdes_base + SERDES_PCIE_BGR_REG);
+ writel(SERDES_CTRL0_PCIE_EN, priv->serdes_base + SERDES_CTRL0_REG);
+ writel(SERDES_PCIE_APP_CLK_EN,
+ priv->serdes_base + SERDES_PCIE_APP_SUB_CTRL_REG);
+ writel(SERDES_CTRL1_PCIE_MODE, priv->serdes_base + SERDES_CTRL1_REG);
+ writel(SERDES_COMB1_PIPE_PCIE_SEL,
+ priv->serdes_base + SERDES_COMB1_PIPE_REG);
+ writel(SERDES_PCIE_ITS_TAGT_DEFAULT,
+ priv->serdes_base + SERDES_PCIE_ITS_TAGT_ADDR_REG);
+ writel(SERDES_AXI2TO1_TH_DEFAULT,
+ priv->serdes_base + SERDES_AXI2TO1_TH_REG);
+
+ return 0;
+}
+
+static int sun60i_combophy_exit(struct phy *phy)
+{
+ struct sun60i_combophy_priv *priv = dev_get_priv(phy->dev);
+
+ reset_assert_bulk(&priv->rsts);
+ clk_disable_bulk(&priv->clks);
+
+ return 0;
+}
+
+static int sun60i_combophy_power_on(struct phy *phy)
+{
+ struct sun60i_combophy_priv *priv = dev_get_priv(phy->dev);
+ u32 val;
+ int i, ret;
+
+ writel(COMBOPHY_CLK_REF_EN, priv->top_base + COMBOPHY_CLK_CTRL_REG);
+
+ for (i = 0; i < ARRAY_SIZE(cphy_analog_regs); i++)
+ writew(cphy_analog_regs[i].val,
+ priv->phy_base + cphy_analog_regs[i].off);
+
+ writew(COMBOPHY_ANALOG_TX_IDLE_MODE,
+ priv->phy_base + COMBOPHY_ANALOG_TX_CTRL0);
+ writew(COMBOPHY_ANALOG_TX_IDLE_MODE,
+ priv->phy_base + COMBOPHY_ANALOG_TX_CTRL2);
+ writew(COMBOPHY_ANALOG_TX_IDLE_MODE,
+ priv->phy_base + COMBOPHY_ANALOG_TX_CTRL3);
+
+ /* Deassert link & lane 0 reset with Lane 0 Mode = PCIe */
+ writel(COMBOPHY_LINK_RST_N, priv->top_base + COMBOPHY_LINK_RST_REG);
+ writel(COMBOPHY_LANE0_PCIE_RST_N,
+ priv->top_base + COMBOPHY_LANE0_RST_REG);
+
+ ret = readl_poll_timeout(priv->top_base + COMBOPHY_PLL_STATUS_REG, val,
+ (val & COMBOPHY_PLL_LOCK_BIT), 50000);
+ if (ret) {
+ dev_err(phy->dev, "Combo PHY PLL lock timeout (val=0x%08x)\n",
+ val);
+ return ret;
+ }
+
+ /* Post PLL lock setup */
+ writew(COMBOPHY_ANALOG_TX_POST_PLL,
+ priv->phy_base + COMBOPHY_ANALOG_TX_CTRL1);
+ writew(readw(priv->phy_base + COMBOPHY_ANALOG_TX_CTRL0) |
+ COMBOPHY_ANALOG_TX_EN_BIT,
+ priv->phy_base + COMBOPHY_ANALOG_TX_CTRL0);
+ writew(0x0001, priv->phy_base + COMBOPHY_ANALOG_EXT_OFFSET);
+ writel(COMBOPHY_CLK_REF_EN | COMBOPHY_CLK_PIPE_OUT_EN,
+ priv->top_base + COMBOPHY_CLK_CTRL_REG);
+
+ return 0;
+}
+
+static int sun60i_combophy_power_off(struct phy *phy)
+{
+ struct sun60i_combophy_priv *priv = dev_get_priv(phy->dev);
+
+ writel(0x0, priv->top_base + COMBOPHY_LANE0_RST_REG);
+ writel(0x0, priv->top_base + COMBOPHY_LINK_RST_REG);
+
+ return 0;
+}
+
+static int sun60i_combophy_of_to_plat(struct udevice *dev)
+{
+ struct sun60i_combophy_priv *priv = dev_get_priv(dev);
+
+ priv->serdes_base = dev_read_addr_name_ptr(dev, "serdes");
+ if (!priv->serdes_base)
+ return -EINVAL;
+
+ priv->top_base = priv->serdes_base + COMBOPHY_TOP_OFFSET;
+
+ priv->phy_base = dev_read_addr_name_ptr(dev, "phy");
+ if (!priv->phy_base)
+ return -EINVAL;
+
+ return 0;
+}
+
+static int sun60i_combophy_probe(struct udevice *dev)
+{
+ struct sun60i_combophy_priv *priv = dev_get_priv(dev);
+ int ret;
+
+ ret = clk_get_bulk(dev, &priv->clks);
+ if (ret)
+ return ret;
+
+ ret = reset_get_bulk(dev, &priv->rsts);
+ if (ret) {
+ clk_release_bulk(&priv->clks);
+ return ret;
+ }
+
+ return 0;
+}
+
+static int sun60i_combophy_remove(struct udevice *dev)
+{
+ struct sun60i_combophy_priv *priv = dev_get_priv(dev);
+
+ reset_release_bulk(&priv->rsts);
+ clk_release_bulk(&priv->clks);
+
+ return 0;
+}
+
+static const struct phy_ops sun60i_combophy_ops = {
+ .of_xlate = sun60i_combophy_xlate,
+ .init = sun60i_combophy_init,
+ .exit = sun60i_combophy_exit,
+ .power_on = sun60i_combophy_power_on,
+ .power_off = sun60i_combophy_power_off,
+};
+
+static const struct udevice_id sun60i_combophy_ids[] = {
+ { .compatible = "allwinner,sun60i-a733-combophy" },
+ { }
+};
+
+U_BOOT_DRIVER(sun60i_combophy) = {
+ .name = "sun60i_combophy",
+ .id = UCLASS_PHY,
+ .of_match = sun60i_combophy_ids,
+ .ops = &sun60i_combophy_ops,
+ .of_to_plat = sun60i_combophy_of_to_plat,
+ .probe = sun60i_combophy_probe,
+ .remove = sun60i_combophy_remove,
+ .priv_auto = sizeof(struct sun60i_combophy_priv),
+};
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 23/26] pci: sunxi: Add Allwinner A733 DesignWare PCIe host controller driver
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (21 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 22/26] phy: sunxi: Add Allwinner A733 Cadence Combo PHY driver Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 24/26] sunxi: Add NVMe to distro boot targets Alastair D'Silva
` (4 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
Add a DesignWare PCIe Root Complex driver (CONFIG_PCIE_DW_SUNXI) for
the Allwinner A733 (sun60i) SoC, reusing pcie_dw_common for config
space and outbound iATU programming while handling Allwinner LTSSM
bringup, Combo PHY integration, PERST# sequencing, and 32-bit inbound
DMA ATU setup.
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
drivers/clk/sunxi/clk_a733.c | 4 +
drivers/pci/Kconfig | 11 ++
drivers/pci/Makefile | 1 +
drivers/pci/pcie_dw_sunxi.c | 355 +++++++++++++++++++++++++++++++++++
4 files changed, 371 insertions(+)
create mode 100644 drivers/pci/pcie_dw_sunxi.c
diff --git a/drivers/clk/sunxi/clk_a733.c b/drivers/clk/sunxi/clk_a733.c
index 4b0aaa7ab2b..ffc3509bd5c 100644
--- a/drivers/clk/sunxi/clk_a733.c
+++ b/drivers/clk/sunxi/clk_a733.c
@@ -72,6 +72,8 @@ static const struct ccu_clk_gate a733_gates[] = {
[CLK_BUS_OHCI1] = GATE(0x130c, BIT(0)),
[CLK_BUS_EHCI1] = GATE(0x130c, BIT(4)),
+ [CLK_PCIE_AUX] = GATE(0x1380, BIT(31)),
+ [CLK_PCIE_AXI_SLV] = GATE(0x1384, BIT(31)),
[CLK_SERDES_PHY] = GATE(0x13c0, BIT(31)),
[CLK_AHB_SERDES] = GATE(0x13c4, BIT(0)),
@@ -124,6 +126,8 @@ static const struct ccu_reset a733_resets[] = {
[RST_BUS_OHCI1] = RESET(0x130c, BIT(16)),
[RST_BUS_EHCI1] = RESET(0x130c, BIT(20)),
+ [RST_BUS_PCIE] = RESET(0x138c, BIT(16)),
+ [RST_BUS_PCIE_PWRUP] = RESET(0x138c, BIT(17)),
[RST_BUS_SERDES] = RESET(0x13c4, BIT(16)),
[RST_BUS_GMAC0] = RESET(0x141c, BIT(16)),
diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
index 86fafc5a3c7..ce5843c6b8e 100644
--- a/drivers/pci/Kconfig
+++ b/drivers/pci/Kconfig
@@ -411,6 +411,17 @@ config PCIE_ROCKCHIP
Say Y here if you want to enable PCIe controller support on
Rockchip SoCs.
+config PCIE_DW_SUNXI
+ bool "Allwinner A733 DesignWare PCIe controller"
+ depends on ARCH_SUNXI
+ select PCIE_DW_COMMON
+ select DM_GPIO
+ select PHY
+ help
+ Enable support for the Synopsys DesignWare PCIe Root Complex
+ controller found on Allwinner sun60i (A733) SoCs, including
+ Gen3 link training and outbound/inbound iATU configuration.
+
config PCIE_DW_ROCKCHIP
bool "Rockchip DesignWare based PCIe controller"
depends on ARCH_ROCKCHIP
diff --git a/drivers/pci/Makefile b/drivers/pci/Makefile
index f839ea1d396..0657aec3ecd 100644
--- a/drivers/pci/Makefile
+++ b/drivers/pci/Makefile
@@ -42,6 +42,7 @@ obj-$(CONFIG_PCI_XILINX) += pcie_xilinx.o
obj-$(CONFIG_PCI_PHYTIUM) += pcie_phytium.o
obj-$(CONFIG_PCIE_DW_COMMON) += pcie_dw_common.o
obj-$(CONFIG_PCI_KEYSTONE) += pcie_dw_ti.o
+obj-$(CONFIG_PCIE_DW_SUNXI) += pcie_dw_sunxi.o
obj-$(CONFIG_PCIE_MEDIATEK) += pcie_mediatek.o
obj-$(CONFIG_PCIE_MEDIATEK_GEN3) += pcie_mediatek_gen3.o
obj-$(CONFIG_PCIE_ROCKCHIP) += pcie_rockchip.o
diff --git a/drivers/pci/pcie_dw_sunxi.c b/drivers/pci/pcie_dw_sunxi.c
new file mode 100644
index 00000000000..c4f27cda15f
--- /dev/null
+++ b/drivers/pci/pcie_dw_sunxi.c
@@ -0,0 +1,355 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Allwinner A733 DesignWare based PCIe host controller driver
+ *
+ * Copyright (c) 2026 Armbian Project
+ */
+
+#include <clk.h>
+#include <dm.h>
+#include <dm/device_compat.h>
+#include <generic-phy.h>
+#include <pci.h>
+#include <power/regulator.h>
+#include <reset.h>
+#include <asm/global_data.h>
+#include <asm/io.h>
+#include <asm-generic/gpio.h>
+#include <linux/delay.h>
+#include <linux/iopoll.h>
+
+#include "pcie_dw_common.h"
+
+DECLARE_GLOBAL_DATA_PTR;
+
+struct sunxi_pcie {
+ /* Must be first member of the struct */
+ struct pcie_dw dw;
+ void __iomem *app_base;
+ struct udevice *vpcie3v3;
+ struct phy phy;
+ struct clk_bulk clks;
+ struct reset_ctl_bulk rsts;
+ struct gpio_desc rst_gpio;
+ u32 gen;
+ u32 num_lanes;
+};
+
+/* Allwinner PCIe application register offsets */
+#define PCIE_APP_CLK_CTRL 0x0800
+#define PCIE_APP_CLK_CTRL_DEFAULT 0x00a023f0
+#define PCIE_LTSSM_CTRL 0x0c00
+#define PCIE_DEVICE_TYPE_RC BIT(6)
+#define PCIE_LINK_TRAINING BIT(0)
+#define PCIE_LINK_STAT 0x0e0c
+#define SMLH_LINK_UP BIT(0)
+#define RDLH_LINK_UP BIT(1)
+
+/* Synopsys DWC port logic / coherent control offsets */
+#define PCIE_PORT_DEBUG0 0x0728
+#define PCIE_PORT_DEBUG0_LTSSM_MASK GENMASK(5, 0)
+#define PCIE_PORT_DEBUG0_LTSSM_L0 0x11
+#define PCIE_AXI_MISC_CTRL 0x0718
+#define PCIE_AXI_CTO_RESP_EN BIT(19)
+#define PCIE_COHERENCY_CTRL1 0x08e0
+#define PCIE_COHERENCY_CTRL2 0x08e8
+
+/* Unrolled Inbound iATU offset within atu_base */
+#define PCIE_ATU_INB_UNR_OFFSET 0x100
+
+static int sunxi_pcie_prog_inbound_atu(struct sunxi_pcie *priv, int index,
+ int type, u64 cpu_addr, u64 pci_addr,
+ u64 size)
+{
+ void __iomem *inb = priv->dw.atu_base + PCIE_ATU_INB_UNR_OFFSET +
+ PCIE_GET_ATU_OUTB_UNR_REG_OFFSET(index);
+ u32 val;
+ int ret;
+
+ dw_pcie_dbi_write_enable(&priv->dw, true);
+
+ writel(lower_32_bits(pci_addr), inb + PCIE_ATU_UNR_LOWER_BASE);
+ writel(upper_32_bits(pci_addr), inb + PCIE_ATU_UNR_UPPER_BASE);
+ writel(lower_32_bits(pci_addr + size - 1), inb + PCIE_ATU_UNR_LIMIT);
+ writel(upper_32_bits(pci_addr + size - 1), inb + PCIE_ATU_UNR_UPPER_LIMIT);
+ writel(lower_32_bits(cpu_addr), inb + PCIE_ATU_UNR_LOWER_TARGET);
+ writel(upper_32_bits(cpu_addr), inb + PCIE_ATU_UNR_UPPER_TARGET);
+ writel(type, inb + PCIE_ATU_UNR_REGION_CTRL1);
+ writel(PCIE_ATU_ENABLE, inb + PCIE_ATU_UNR_REGION_CTRL2);
+
+ ret = readl_poll_timeout(inb + PCIE_ATU_UNR_REGION_CTRL2, val,
+ val & PCIE_ATU_ENABLE,
+ LINK_WAIT_MAX_IATU_RETRIES * LINK_WAIT_IATU);
+ dw_pcie_dbi_write_enable(&priv->dw, false);
+ if (ret) {
+ dev_err(priv->dw.dev, "inbound iATU is not being enabled\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+static int sunxi_pcie_link_up(struct udevice *dev, struct sunxi_pcie *priv)
+{
+ u32 val;
+ int ret;
+
+ /* Set LTSSM Enable with RC Mode */
+ writel(PCIE_DEVICE_TYPE_RC | PCIE_LINK_TRAINING,
+ priv->app_base + PCIE_LTSSM_CTRL);
+
+ /* Wait for SMLH link up and L0 active */
+ ret = readl_poll_timeout(priv->app_base + PCIE_LINK_STAT, val,
+ ((val & SMLH_LINK_UP) &&
+ ((val & RDLH_LINK_UP) ||
+ ((readl(priv->dw.dbi_base + PCIE_PORT_DEBUG0) &
+ PCIE_PORT_DEBUG0_LTSSM_MASK) ==
+ PCIE_PORT_DEBUG0_LTSSM_L0))),
+ 300000);
+ if (ret) {
+ dev_err(dev, "Link training timeout (LINK_STAT=0x%08x, DBG0=0x%08x)\n",
+ readl(priv->app_base + PCIE_LINK_STAT),
+ readl(priv->dw.dbi_base + PCIE_PORT_DEBUG0));
+ return -ETIMEDOUT;
+ }
+
+ return 0;
+}
+
+static int sunxi_pcie_of_to_plat(struct udevice *dev)
+{
+ struct sunxi_pcie *priv = dev_get_priv(dev);
+
+ priv->dw.dbi_base = dev_read_addr_name_ptr(dev, "dbi");
+ if (!priv->dw.dbi_base)
+ return -EINVAL;
+
+ priv->app_base = dev_read_addr_name_ptr(dev, "app");
+ if (!priv->app_base)
+ return -EINVAL;
+
+ priv->dw.cfg_base =
+ dev_read_addr_size_name_ptr(dev, "config", &priv->dw.cfg_size);
+ if (!priv->dw.cfg_base)
+ return -EINVAL;
+
+ priv->gen = dev_read_u32_default(dev, "max-link-speed",
+ LINK_SPEED_GEN_3);
+ priv->num_lanes = dev_read_u32_default(dev, "num-lanes", 1);
+
+ return 0;
+}
+
+static int sunxi_pcie_probe(struct udevice *dev)
+{
+ struct sunxi_pcie *priv = dev_get_priv(dev);
+ u64 dma_size;
+ u32 val;
+ int ret;
+
+ priv->dw.first_busno = dev_seq(dev);
+ priv->dw.dev = dev;
+
+ ret = gpio_request_by_name(dev, "reset-gpios", 0, &priv->rst_gpio,
+ GPIOD_IS_OUT | GPIOD_IS_OUT_ACTIVE);
+ if (ret && ret != -ENOENT)
+ return ret;
+
+ ret = device_get_supply_regulator(dev, "vpcie3v3-supply",
+ &priv->vpcie3v3);
+ if (ret && ret != -ENOENT && ret != -ENOSYS)
+ goto err_gpio;
+
+ ret = clk_get_bulk(dev, &priv->clks);
+ if (ret)
+ goto err_gpio;
+
+ ret = reset_get_bulk(dev, &priv->rsts);
+ if (ret)
+ goto err_clk_rel;
+
+ ret = generic_phy_get_by_index(dev, 0, &priv->phy);
+ if (ret)
+ goto err_rst_rel;
+
+ if (priv->vpcie3v3) {
+ ret = regulator_set_enable_if_allowed(priv->vpcie3v3, true);
+ if (ret && ret != -ENOSYS) {
+ dev_err(dev, "Failed to enable vpcie3v3: %d\n", ret);
+ goto err_rst_rel;
+ }
+ mdelay(50);
+ }
+
+ /* Assert PERST# during PHY and controller initialization */
+ if (dm_gpio_is_valid(&priv->rst_gpio))
+ dm_gpio_set_value(&priv->rst_gpio, 1);
+
+ ret = clk_enable_bulk(&priv->clks);
+ if (ret)
+ goto err_reg;
+
+ ret = reset_deassert_bulk(&priv->rsts);
+ if (ret)
+ goto err_clk;
+
+ ret = generic_phy_init(&priv->phy);
+ if (ret)
+ goto err_rst;
+
+ ret = generic_phy_power_on(&priv->phy);
+ if (ret)
+ goto err_phy_exit;
+
+ /* Enable core, AXI, DBI, and AUX clock gates with PM defaults */
+ writel(PCIE_APP_CLK_CTRL_DEFAULT, priv->app_base + PCIE_APP_CLK_CTRL);
+
+ /* Hold PERST# asserted for 20 ms, then deassert and wait 100 ms */
+ if (dm_gpio_is_valid(&priv->rst_gpio)) {
+ mdelay(20);
+ dm_gpio_set_value(&priv->rst_gpio, 0);
+ mdelay(100);
+ }
+
+ /* Setup DWC controller host registers */
+ pcie_dw_setup_host(&priv->dw);
+
+ /* Clear PORT_LOGIC_SPEED_CHANGE set by pcie_dw_setup_host before link up */
+ clrbits_le32(priv->dw.dbi_base + PCIE_LINK_WIDTH_SPEED_CONTROL,
+ PORT_LOGIC_SPEED_CHANGE);
+
+ /*
+ * Clear RC BAR0/1 so U-Boot's PCI BAR scan does not allocate outbound
+ * memory space for unused root complex BARs, program max link
+ * capability/target speed and lane count, and enable CTO/coherency
+ * defaults for Allwinner A733.
+ */
+ dw_pcie_dbi_write_enable(&priv->dw, true);
+ writel(0x00000000, priv->dw.dbi_base + PCI_BASE_ADDRESS_0);
+ writel(0x00000000, priv->dw.dbi_base + PCI_BASE_ADDRESS_1);
+ clrsetbits_le32(priv->dw.dbi_base + PCIE_LINK_CAPABILITY,
+ TARGET_LINK_SPEED_MASK, priv->gen);
+ clrsetbits_le32(priv->dw.dbi_base + PCIE_LINK_CTL_2,
+ TARGET_LINK_SPEED_MASK, priv->gen);
+ dw_pcie_link_set_max_link_width(&priv->dw, priv->num_lanes);
+ setbits_le32(priv->dw.dbi_base + PCIE_AXI_MISC_CTRL,
+ PCIE_AXI_CTO_RESP_EN);
+ writel(0x00000000, priv->dw.dbi_base + PCIE_COHERENCY_CTRL1);
+ writel(0x00000000, priv->dw.dbi_base + PCIE_COHERENCY_CTRL2);
+ dw_pcie_dbi_write_enable(&priv->dw, false);
+
+ /* Start LTSSM link training and wait for SMLH & RDLH link up */
+ ret = sunxi_pcie_link_up(dev, priv);
+ if (ret)
+ goto err_phy_power;
+
+ /* Trigger speed change after Gen1 link is established */
+ if (priv->gen > LINK_SPEED_GEN_1) {
+ dw_pcie_dbi_write_enable(&priv->dw, true);
+ setbits_le32(priv->dw.dbi_base + PCIE_LINK_WIDTH_SPEED_CONTROL,
+ PORT_LOGIC_SPEED_CHANGE);
+ ret = readl_poll_timeout(priv->dw.dbi_base +
+ PCIE_LINK_WIDTH_SPEED_CONTROL,
+ val, !(val & PORT_LOGIC_SPEED_CHANGE),
+ 200000);
+ if (ret)
+ dev_warn(dev, "Gen%d speed change timeout\n",
+ priv->gen);
+ dw_pcie_dbi_write_enable(&priv->dw, false);
+ }
+
+ /*
+ * Program 32-bit inbound DMA ATU window matching the DT dma-ranges
+ * property (0x40000000..0xffffffff) for system DRAM.
+ */
+ if (gd->ram_base < 0x100000000ULL) {
+ dma_size = min_t(u64, gd->ram_size,
+ 0x100000000ULL - gd->ram_base);
+ if (dma_size) {
+ ret = sunxi_pcie_prog_inbound_atu(priv,
+ PCIE_ATU_REGION_INDEX0,
+ PCIE_ATU_TYPE_MEM,
+ gd->ram_base,
+ gd->ram_base,
+ dma_size);
+ if (ret)
+ goto err_phy_power;
+ }
+ }
+
+ /* Program outbound MEM ATU */
+ ret = pcie_dw_prog_outbound_atu_unroll(&priv->dw,
+ PCIE_ATU_REGION_INDEX0,
+ PCIE_ATU_TYPE_MEM,
+ priv->dw.mem.phys_start,
+ priv->dw.mem.bus_start,
+ priv->dw.mem.size);
+ if (ret)
+ goto err_phy_power;
+
+ dev_info(dev, "Link up (Gen%d-x%d, Bus%d)\n",
+ pcie_dw_get_link_speed(&priv->dw),
+ pcie_dw_get_link_width(&priv->dw),
+ priv->dw.first_busno);
+
+ return 0;
+
+err_phy_power:
+ generic_phy_power_off(&priv->phy);
+err_phy_exit:
+ generic_phy_exit(&priv->phy);
+err_rst:
+ reset_assert_bulk(&priv->rsts);
+err_clk:
+ clk_disable_bulk(&priv->clks);
+err_reg:
+ if (priv->vpcie3v3)
+ regulator_set_enable_if_allowed(priv->vpcie3v3, false);
+err_rst_rel:
+ reset_release_bulk(&priv->rsts);
+err_clk_rel:
+ clk_release_bulk(&priv->clks);
+err_gpio:
+ if (dm_gpio_is_valid(&priv->rst_gpio))
+ dm_gpio_free(dev, &priv->rst_gpio);
+ return ret;
+}
+
+static int sunxi_pcie_remove(struct udevice *dev)
+{
+ struct sunxi_pcie *priv = dev_get_priv(dev);
+
+ generic_phy_power_off(&priv->phy);
+ generic_phy_exit(&priv->phy);
+ reset_assert_bulk(&priv->rsts);
+ clk_disable_bulk(&priv->clks);
+ if (priv->vpcie3v3)
+ regulator_set_enable_if_allowed(priv->vpcie3v3, false);
+ if (dm_gpio_is_valid(&priv->rst_gpio))
+ dm_gpio_free(dev, &priv->rst_gpio);
+ reset_release_bulk(&priv->rsts);
+ clk_release_bulk(&priv->clks);
+
+ return 0;
+}
+
+static const struct dm_pci_ops sunxi_pcie_ops = {
+ .read_config = pcie_dw_read_config,
+ .write_config = pcie_dw_write_config,
+};
+
+static const struct udevice_id sunxi_pcie_ids[] = {
+ { .compatible = "allwinner,sun60i-a733-pcie" },
+ { }
+};
+
+U_BOOT_DRIVER(sunxi_pcie) = {
+ .name = "sunxi_pcie",
+ .id = UCLASS_PCI,
+ .of_match = sunxi_pcie_ids,
+ .ops = &sunxi_pcie_ops,
+ .of_to_plat = sunxi_pcie_of_to_plat,
+ .probe = sunxi_pcie_probe,
+ .remove = sunxi_pcie_remove,
+ .priv_auto = sizeof(struct sunxi_pcie),
+};
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 24/26] sunxi: Add NVMe to distro boot targets
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (22 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 23/26] pci: sunxi: Add Allwinner A733 DesignWare PCIe host controller driver Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 25/26] arm64: dts: allwinner: add A733 SoC description Alastair D'Silva
` (3 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
Add BOOT_TARGET_DEVICES_NVME to BOOT_TARGET_DEVICES in sunxi-common.h
guarded by #ifdef CONFIG_NVME so sunxi boards with PCIe/NVMe enabled
automatically probe and boot from NVMe devices.
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
include/configs/sunxi-common.h | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/include/configs/sunxi-common.h b/include/configs/sunxi-common.h
index 49d5617f028..3d3413fab02 100644
--- a/include/configs/sunxi-common.h
+++ b/include/configs/sunxi-common.h
@@ -184,6 +184,12 @@
#define BOOT_TARGET_DEVICES_MMC(func)
#endif
+#ifdef CONFIG_NVME
+#define BOOT_TARGET_DEVICES_NVME(func) func(NVME, nvme, 0)
+#else
+#define BOOT_TARGET_DEVICES_NVME(func)
+#endif
+
#ifdef CONFIG_AHCI
#define BOOT_TARGET_DEVICES_SCSI(func) func(SCSI, scsi, 0)
#else
@@ -221,6 +227,7 @@
#define BOOT_TARGET_DEVICES(func) \
func(FEL, fel, na) \
BOOT_TARGET_DEVICES_MMC(func) \
+ BOOT_TARGET_DEVICES_NVME(func) \
BOOT_TARGET_DEVICES_SCSI(func) \
BOOT_TARGET_DEVICES_USB(func) \
BOOT_TARGET_DEVICES_PXE(func) \
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 25/26] arm64: dts: allwinner: add A733 SoC description
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (23 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 24/26] sunxi: Add NVMe to distro boot targets Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 2:32 ` [PATCH 26/26] sunxi: Add Xunlong Orange Pi 4 Pro board support Alastair D'Silva
` (2 subsequent siblings)
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
From: Enzo Adriano <enzo.adriano.code@gmail.com>
Add initial device tree description for the Allwinner A733 SoC,
covering the octa-core CPU topology (6x Cortex-A55 + 2x Cortex-A76),
PSCI, ARMv8 architected timer, GICv3 + ITS, CCU/R-CCU, PIO/R-PIO,
UART0-6, R-UART0-1, I2C0-12, R-I2C0-2, MMC0-2, NMI controller, and RTC,
based on Enzo Adriano's v3 Linux kernel submission.
Extend the local U-Boot import with the watchdog, SPI0, SID efuse, USB2
PHY, and EHCI0/1 + OHCI0/1 controllers, along with optional PB0/PB1
UART0 and PC0 MMC2 hardware-reset pinctrl groups, until the SoC
devicetree lands in upstream Linux / dts/upstream.
Link: https://lore.kernel.org/r/7c0ee4e609db5979d2a6598f46f40896b4f47a57.1794227241.git.enzo.adriano.code@gmail.com
Signed-off-by: Enzo Adriano <enzo.adriano.code@gmail.com>
[Alastair: import into arch/arm/dts/, add wdt, spi0, sid, usbphy,
ehci0/1, ohci0/1, spi0_pc_pins, uart0_pb0_pins, mmc2_rst_pin, and
MAINTAINERS entry]
Co-developed-by: Alastair D'Silva <alastair@d-silva.org>
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
MAINTAINERS | 13 +
arch/arm/dts/sun60i-a733.dtsi | 816 ++++++++++++++++++++++++++++++++++
2 files changed, 829 insertions(+)
create mode 100644 arch/arm/dts/sun60i-a733.dtsi
diff --git a/MAINTAINERS b/MAINTAINERS
index 9fc6614e519..5b640785e87 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -707,6 +707,19 @@ F: drivers/pinctrl/sunxi/
F: drivers/video/sunxi/
F: tools/sunxi*
+ARM SUNXI A733 SOC
+M: Alastair D'Silva <alastair@d-silva.org>
+S: Maintained
+F: arch/arm/dts/sun60i-a733*
+F: arch/arm/mach-sunxi/dram_sun60i_a733*
+F: arch/arm/mach-sunxi/pmu/
+F: doc/board/allwinner/a733-pmu.rst
+F: drivers/clk/sunxi/clk_a733*
+F: drivers/pci/pcie_dw_sunxi.c
+F: drivers/phy/allwinner/phy-sun60i-combophy.c
+F: include/configs/sun60i.h
+F: include/dt-bindings/*/allwinner,sun60i-a733*
+
ARM TEGRA
M: Thierry Reding <treding@nvidia.com>
M: Svyatoslav Ryhel <clamor95@gmail.com>
diff --git a/arch/arm/dts/sun60i-a733.dtsi b/arch/arm/dts/sun60i-a733.dtsi
new file mode 100644
index 00000000000..4b19789266d
--- /dev/null
+++ b/arch/arm/dts/sun60i-a733.dtsi
@@ -0,0 +1,816 @@
+// SPDX-License-Identifier: (GPL-2.0-only OR MIT)
+/*
+ * Copyright (C) 2026 Enzo Adriano <enzo.adriano.code@gmail.com>
+ * Copyright (C) 2026 Alastair D'Silva <alastair@d-silva.org>
+ */
+
+#include <dt-bindings/clock/sun6i-rtc.h>
+#include <dt-bindings/clock/allwinner,sun60i-a733-ccu.h>
+#include <dt-bindings/clock/allwinner,sun60i-a733-r-ccu.h>
+#include <dt-bindings/interrupt-controller/arm-gic.h>
+#include <dt-bindings/phy/phy.h>
+#include <dt-bindings/reset/allwinner,sun60i-a733-ccu.h>
+#include <dt-bindings/reset/allwinner,sun60i-a733-r-ccu.h>
+
+/ {
+ #address-cells = <2>;
+ #size-cells = <2>;
+ interrupt-parent = <&gic>;
+
+ cpus {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ cpu0: cpu@0 {
+ compatible = "arm,cortex-a55";
+ device_type = "cpu";
+ reg = <0x0>;
+ enable-method = "psci";
+ };
+
+ cpu1: cpu@100 {
+ compatible = "arm,cortex-a55";
+ device_type = "cpu";
+ reg = <0x100>;
+ enable-method = "psci";
+ };
+
+ cpu2: cpu@200 {
+ compatible = "arm,cortex-a55";
+ device_type = "cpu";
+ reg = <0x200>;
+ enable-method = "psci";
+ };
+
+ cpu3: cpu@300 {
+ compatible = "arm,cortex-a55";
+ device_type = "cpu";
+ reg = <0x300>;
+ enable-method = "psci";
+ };
+
+ cpu4: cpu@400 {
+ compatible = "arm,cortex-a55";
+ device_type = "cpu";
+ reg = <0x400>;
+ enable-method = "psci";
+ };
+
+ cpu5: cpu@500 {
+ compatible = "arm,cortex-a55";
+ device_type = "cpu";
+ reg = <0x500>;
+ enable-method = "psci";
+ };
+
+ cpu6: cpu@600 {
+ compatible = "arm,cortex-a76";
+ device_type = "cpu";
+ reg = <0x600>;
+ enable-method = "psci";
+ };
+
+ cpu7: cpu@700 {
+ compatible = "arm,cortex-a76";
+ device_type = "cpu";
+ reg = <0x700>;
+ enable-method = "psci";
+ };
+ };
+
+ osc: osc-clk {
+ compatible = "fixed-clock";
+ #clock-cells = <0>;
+ };
+
+ psci {
+ compatible = "arm,psci-0.2";
+ method = "smc";
+ };
+
+ timer {
+ compatible = "arm,armv8-timer";
+ interrupts = <GIC_PPI 13 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_PPI 14 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_PPI 11 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_PPI 10 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_PPI 12 IRQ_TYPE_LEVEL_HIGH>;
+ arm,no-tick-in-suspend;
+ };
+
+ soc {
+ compatible = "simple-bus";
+ #address-cells = <1>;
+ #size-cells = <1>;
+ ranges = <0x0 0x0 0x0 0x40000000>;
+ dma-ranges = <0x40000000 0x0 0x40000000 0xc0000000>;
+
+ pio: pinctrl@2000000 {
+ compatible = "allwinner,sun60i-a733-pinctrl";
+ reg = <0x02000000 0x800>;
+ interrupts = <GIC_SPI 69 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 71 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 73 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 75 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 77 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 79 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 81 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 83 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 85 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 87 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_APB1>, <&osc>, <&rtc CLK_OSC32K>;
+ clock-names = "apb", "hosc", "losc";
+ gpio-controller;
+ #gpio-cells = <3>;
+ interrupt-controller;
+ #interrupt-cells = <3>;
+
+ mmc0_pins: mmc0-pins {
+ pins = "PF0", "PF1", "PF2", "PF3", "PF4", "PF5";
+ function = "mmc0";
+ allwinner,pinmux = <2>;
+ drive-strength = <30>;
+ bias-pull-up;
+ };
+
+ mmc1_pins: mmc1-pins {
+ pins = "PG0", "PG1", "PG2", "PG3", "PG4", "PG5";
+ function = "mmc1";
+ allwinner,pinmux = <2>;
+ drive-strength = <30>;
+ bias-pull-up;
+ };
+
+ pcie_pins: pcie-pins {
+ pins = "PD20";
+ function = "pcie";
+ allwinner,pinmux = <6>;
+ };
+
+ mmc2_pins: mmc2-pins {
+ pins = "PC5", "PC6", "PC8", "PC9", "PC10", "PC11",
+ "PC13", "PC14", "PC15", "PC16";
+ function = "mmc2";
+ allwinner,pinmux = <3>;
+ drive-strength = <30>;
+ bias-pull-up;
+ };
+
+ /omit-if-no-ref/
+ mmc2_rst_pin: mmc2-rst-pin {
+ pins = "PC0";
+ function = "mmc2";
+ allwinner,pinmux = <3>;
+ drive-strength = <30>;
+ bias-pull-up;
+ };
+
+ /omit-if-no-ref/
+ spi0_pc_pins: spi0-pc-pins {
+ pins = "PC2", "PC3", "PC4", "PC12";
+ function = "spi0";
+ allwinner,pinmux = <5>;
+ };
+
+ /omit-if-no-ref/
+ uart0_pb0_pins: uart0-pb0-pins {
+ pins = "PB0", "PB1";
+ function = "uart0";
+ allwinner,pinmux = <3>;
+ };
+
+ /omit-if-no-ref/
+ uart0_pb_pins: uart0-pb-pins {
+ pins = "PB9", "PB10";
+ function = "uart0";
+ allwinner,pinmux = <2>;
+ };
+ };
+
+ ccu: clock-controller@2002000 {
+ compatible = "allwinner,sun60i-a733-ccu";
+ reg = <0x02002000 0x2000>;
+ clocks = <&osc>, <&rtc CLK_OSC32K>, <&rtc CLK_IOSC>,
+ <&rtc CLK_OSC32K_FANOUT>;
+ clock-names = "hosc", "losc", "iosc", "losc-fanout";
+ #clock-cells = <1>;
+ #reset-cells = <1>;
+ };
+
+ wdt: watchdog@2050000 {
+ compatible = "allwinner,sun60i-a733-wdt",
+ "allwinner,sun55i-a523-wdt";
+ reg = <0x02050000 0x20>;
+ interrupts = <GIC_SPI 106 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&osc>, <&rtc CLK_OSC32K>;
+ clock-names = "hosc", "losc";
+ };
+
+ uart0: serial@2500000 {
+ compatible = "allwinner,sun60i-a733-uart",
+ "snps,dw-apb-uart";
+ reg = <0x02500000 0x400>;
+ interrupts = <GIC_SPI 2 IRQ_TYPE_LEVEL_HIGH>;
+ reg-shift = <2>;
+ reg-io-width = <4>;
+ clocks = <&ccu CLK_BUS_UART0>;
+ resets = <&ccu RST_BUS_UART0>;
+ status = "disabled";
+ };
+
+ uart1: serial@2501000 {
+ compatible = "allwinner,sun60i-a733-uart",
+ "snps,dw-apb-uart";
+ reg = <0x02501000 0x400>;
+ interrupts = <GIC_SPI 3 IRQ_TYPE_LEVEL_HIGH>;
+ reg-shift = <2>;
+ reg-io-width = <4>;
+ clocks = <&ccu CLK_BUS_UART1>;
+ resets = <&ccu RST_BUS_UART1>;
+ status = "disabled";
+ };
+
+ uart2: serial@2502000 {
+ compatible = "allwinner,sun60i-a733-uart",
+ "snps,dw-apb-uart";
+ reg = <0x02502000 0x400>;
+ interrupts = <GIC_SPI 4 IRQ_TYPE_LEVEL_HIGH>;
+ reg-shift = <2>;
+ reg-io-width = <4>;
+ clocks = <&ccu CLK_BUS_UART2>;
+ resets = <&ccu RST_BUS_UART2>;
+ status = "disabled";
+ };
+
+ uart3: serial@2503000 {
+ compatible = "allwinner,sun60i-a733-uart",
+ "snps,dw-apb-uart";
+ reg = <0x02503000 0x400>;
+ interrupts = <GIC_SPI 5 IRQ_TYPE_LEVEL_HIGH>;
+ reg-shift = <2>;
+ reg-io-width = <4>;
+ clocks = <&ccu CLK_BUS_UART3>;
+ resets = <&ccu RST_BUS_UART3>;
+ status = "disabled";
+ };
+
+ uart4: serial@2504000 {
+ compatible = "allwinner,sun60i-a733-uart",
+ "snps,dw-apb-uart";
+ reg = <0x02504000 0x400>;
+ interrupts = <GIC_SPI 6 IRQ_TYPE_LEVEL_HIGH>;
+ reg-shift = <2>;
+ reg-io-width = <4>;
+ clocks = <&ccu CLK_BUS_UART4>;
+ resets = <&ccu RST_BUS_UART4>;
+ status = "disabled";
+ };
+
+ uart5: serial@2505000 {
+ compatible = "allwinner,sun60i-a733-uart",
+ "snps,dw-apb-uart";
+ reg = <0x02505000 0x400>;
+ interrupts = <GIC_SPI 7 IRQ_TYPE_LEVEL_HIGH>;
+ reg-shift = <2>;
+ reg-io-width = <4>;
+ clocks = <&ccu CLK_BUS_UART5>;
+ resets = <&ccu RST_BUS_UART5>;
+ status = "disabled";
+ };
+
+ uart6: serial@2506000 {
+ compatible = "allwinner,sun60i-a733-uart",
+ "snps,dw-apb-uart";
+ reg = <0x02506000 0x400>;
+ interrupts = <GIC_SPI 8 IRQ_TYPE_LEVEL_HIGH>;
+ reg-shift = <2>;
+ reg-io-width = <4>;
+ clocks = <&ccu CLK_BUS_UART6>;
+ resets = <&ccu RST_BUS_UART6>;
+ status = "disabled";
+ };
+
+ i2c0: i2c@2510000 {
+ compatible = "allwinner,sun60i-a733-i2c",
+ "allwinner,sun8i-v536-i2c",
+ "allwinner,sun6i-a31-i2c";
+ reg = <0x02510000 0x400>;
+ interrupts = <GIC_SPI 11 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_BUS_I2C0>;
+ resets = <&ccu RST_BUS_I2C0>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "disabled";
+ };
+
+ i2c1: i2c@2511000 {
+ compatible = "allwinner,sun60i-a733-i2c",
+ "allwinner,sun8i-v536-i2c",
+ "allwinner,sun6i-a31-i2c";
+ reg = <0x02511000 0x400>;
+ interrupts = <GIC_SPI 12 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_BUS_I2C1>;
+ resets = <&ccu RST_BUS_I2C1>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "disabled";
+ };
+
+ i2c2: i2c@2512000 {
+ compatible = "allwinner,sun60i-a733-i2c",
+ "allwinner,sun8i-v536-i2c",
+ "allwinner,sun6i-a31-i2c";
+ reg = <0x02512000 0x400>;
+ interrupts = <GIC_SPI 13 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_BUS_I2C2>;
+ resets = <&ccu RST_BUS_I2C2>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "disabled";
+ };
+
+ i2c3: i2c@2513000 {
+ compatible = "allwinner,sun60i-a733-i2c",
+ "allwinner,sun8i-v536-i2c",
+ "allwinner,sun6i-a31-i2c";
+ reg = <0x02513000 0x400>;
+ interrupts = <GIC_SPI 14 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_BUS_I2C3>;
+ resets = <&ccu RST_BUS_I2C3>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "disabled";
+ };
+
+ i2c4: i2c@2514000 {
+ compatible = "allwinner,sun60i-a733-i2c",
+ "allwinner,sun8i-v536-i2c",
+ "allwinner,sun6i-a31-i2c";
+ reg = <0x02514000 0x400>;
+ interrupts = <GIC_SPI 15 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_BUS_I2C4>;
+ resets = <&ccu RST_BUS_I2C4>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "disabled";
+ };
+
+ i2c5: i2c@2515000 {
+ compatible = "allwinner,sun60i-a733-i2c",
+ "allwinner,sun8i-v536-i2c",
+ "allwinner,sun6i-a31-i2c";
+ reg = <0x02515000 0x400>;
+ interrupts = <GIC_SPI 16 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_BUS_I2C5>;
+ resets = <&ccu RST_BUS_I2C5>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "disabled";
+ };
+
+ i2c6: i2c@2516000 {
+ compatible = "allwinner,sun60i-a733-i2c",
+ "allwinner,sun8i-v536-i2c",
+ "allwinner,sun6i-a31-i2c";
+ reg = <0x02516000 0x400>;
+ interrupts = <GIC_SPI 17 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_BUS_I2C6>;
+ resets = <&ccu RST_BUS_I2C6>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "disabled";
+ };
+
+ i2c7: i2c@2517000 {
+ compatible = "allwinner,sun60i-a733-i2c",
+ "allwinner,sun8i-v536-i2c",
+ "allwinner,sun6i-a31-i2c";
+ reg = <0x02517000 0x400>;
+ interrupts = <GIC_SPI 18 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_BUS_I2C7>;
+ resets = <&ccu RST_BUS_I2C7>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "disabled";
+ };
+
+ i2c8: i2c@2518000 {
+ compatible = "allwinner,sun60i-a733-i2c",
+ "allwinner,sun8i-v536-i2c",
+ "allwinner,sun6i-a31-i2c";
+ reg = <0x02518000 0x400>;
+ interrupts = <GIC_SPI 19 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_BUS_I2C8>;
+ resets = <&ccu RST_BUS_I2C8>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "disabled";
+ };
+
+ i2c9: i2c@2519000 {
+ compatible = "allwinner,sun60i-a733-i2c",
+ "allwinner,sun8i-v536-i2c",
+ "allwinner,sun6i-a31-i2c";
+ reg = <0x02519000 0x400>;
+ interrupts = <GIC_SPI 20 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_BUS_I2C9>;
+ resets = <&ccu RST_BUS_I2C9>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "disabled";
+ };
+
+ i2c10: i2c@251a000 {
+ compatible = "allwinner,sun60i-a733-i2c",
+ "allwinner,sun8i-v536-i2c",
+ "allwinner,sun6i-a31-i2c";
+ reg = <0x0251a000 0x400>;
+ interrupts = <GIC_SPI 21 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_BUS_I2C10>;
+ resets = <&ccu RST_BUS_I2C10>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "disabled";
+ };
+
+ i2c11: i2c@251b000 {
+ compatible = "allwinner,sun60i-a733-i2c",
+ "allwinner,sun8i-v536-i2c",
+ "allwinner,sun6i-a31-i2c";
+ reg = <0x0251b000 0x400>;
+ interrupts = <GIC_SPI 22 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_BUS_I2C11>;
+ resets = <&ccu RST_BUS_I2C11>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "disabled";
+ };
+
+ i2c12: i2c@251c000 {
+ compatible = "allwinner,sun60i-a733-i2c",
+ "allwinner,sun8i-v536-i2c",
+ "allwinner,sun6i-a31-i2c";
+ reg = <0x0251c000 0x400>;
+ interrupts = <GIC_SPI 23 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_BUS_I2C12>;
+ resets = <&ccu RST_BUS_I2C12>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "disabled";
+ };
+
+ spi0: spi@2540000 {
+ compatible = "allwinner,sun60i-a733-spi",
+ "allwinner,sun55i-a523-spi";
+ reg = <0x02540000 0x1000>;
+ interrupts = <GIC_SPI 24 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_BUS_SPI0>, <&ccu CLK_SPI0>;
+ clock-names = "ahb", "mod";
+ resets = <&ccu RST_BUS_SPI0>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "disabled";
+ };
+
+ sid: efuse@3006000 {
+ compatible = "allwinner,sun60i-a733-sid",
+ "allwinner,sun50i-a64-sid";
+ reg = <0x03006000 0x1000>;
+ #address-cells = <1>;
+ #size-cells = <1>;
+ };
+
+ gic: interrupt-controller@3400000 {
+ compatible = "arm,gic-v3";
+ reg = <0x03400000 0x10000>, <0x03460000 0x100000>;
+ interrupts = <GIC_PPI 9 IRQ_TYPE_LEVEL_HIGH>;
+ dma-noncoherent;
+ interrupt-controller;
+ #interrupt-cells = <3>;
+ #address-cells = <1>;
+ #size-cells = <1>;
+ ranges;
+
+ its: msi-controller@3440000 {
+ compatible = "arm,gic-v3-its";
+ reg = <0x03440000 0x20000>;
+ msi-controller;
+ #msi-cells = <1>;
+ dma-noncoherent;
+ status = "disabled";
+ };
+ };
+
+ mmc0: mmc@4020000 {
+ compatible = "allwinner,sun60i-a733-mmc",
+ "allwinner,sun20i-d1-mmc";
+ reg = <0x04020000 0x1000>;
+ interrupts = <GIC_SPI 161 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_BUS_MMC0>, <&ccu CLK_MMC0>;
+ clock-names = "ahb", "mmc";
+ resets = <&ccu RST_BUS_MMC0>;
+ reset-names = "ahb";
+ pinctrl-names = "default";
+ pinctrl-0 = <&mmc0_pins>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "disabled";
+ };
+
+ mmc1: mmc@4021000 {
+ compatible = "allwinner,sun60i-a733-mmc",
+ "allwinner,sun20i-d1-mmc";
+ reg = <0x04021000 0x1000>;
+ interrupts = <GIC_SPI 162 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_BUS_MMC1>, <&ccu CLK_MMC1>;
+ clock-names = "ahb", "mmc";
+ resets = <&ccu RST_BUS_MMC1>;
+ reset-names = "ahb";
+ pinctrl-names = "default";
+ pinctrl-0 = <&mmc1_pins>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "disabled";
+ };
+
+ mmc2: mmc@4022000 {
+ compatible = "allwinner,sun60i-a733-mmc",
+ "allwinner,sun20i-d1-mmc";
+ reg = <0x04022000 0x1000>;
+ interrupts = <GIC_SPI 163 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_BUS_MMC2>, <&ccu CLK_MMC2>;
+ clock-names = "ahb", "mmc";
+ resets = <&ccu RST_BUS_MMC2>;
+ reset-names = "ahb";
+ pinctrl-names = "default";
+ pinctrl-0 = <&mmc2_pins>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "disabled";
+ };
+
+ usbphy: phy@4100400 {
+ compatible = "allwinner,sun60i-a733-usb-phy",
+ "allwinner,sun20i-d1-usb-phy";
+ reg = <0x04100400 0x100>,
+ <0x04101800 0x100>,
+ <0x04200800 0x100>;
+ reg-names = "phy_ctrl",
+ "pmu0",
+ "pmu1";
+ clocks = <&osc>,
+ <&osc>;
+ clock-names = "usb0_phy",
+ "usb1_phy";
+ resets = <&ccu RST_USB_PHY0>,
+ <&ccu RST_USB_PHY1>;
+ reset-names = "usb0_reset",
+ "usb1_reset";
+ status = "disabled";
+ #phy-cells = <1>;
+ };
+
+ ehci0: usb@4101000 {
+ compatible = "allwinner,sun60i-a733-ehci",
+ "allwinner,sun55i-a523-ehci",
+ "generic-ehci";
+ reg = <0x04101000 0x100>;
+ interrupts = <GIC_SPI 157 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_BUS_OHCI0>,
+ <&ccu CLK_BUS_EHCI0>,
+ <&ccu CLK_USB_OHCI0>;
+ resets = <&ccu RST_BUS_OHCI0>,
+ <&ccu RST_BUS_EHCI0>;
+ phys = <&usbphy 0>;
+ phy-names = "usb";
+ status = "disabled";
+ };
+
+ ohci0: usb@4101400 {
+ compatible = "allwinner,sun60i-a733-ohci",
+ "allwinner,sun55i-a523-ohci",
+ "generic-ohci";
+ reg = <0x04101400 0x100>;
+ interrupts = <GIC_SPI 158 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_BUS_OHCI0>,
+ <&ccu CLK_USB_OHCI0>;
+ resets = <&ccu RST_BUS_OHCI0>;
+ phys = <&usbphy 0>;
+ phy-names = "usb";
+ status = "disabled";
+ };
+
+ ehci1: usb@4200000 {
+ compatible = "allwinner,sun60i-a733-ehci",
+ "allwinner,sun55i-a523-ehci",
+ "generic-ehci";
+ reg = <0x04200000 0x100>;
+ interrupts = <GIC_SPI 159 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_BUS_OHCI1>,
+ <&ccu CLK_BUS_EHCI1>,
+ <&ccu CLK_USB_OHCI1>;
+ resets = <&ccu RST_BUS_OHCI1>,
+ <&ccu RST_BUS_EHCI1>;
+ phys = <&usbphy 1>;
+ phy-names = "usb";
+ status = "disabled";
+ };
+
+ ohci1: usb@4200400 {
+ compatible = "allwinner,sun60i-a733-ohci",
+ "allwinner,sun55i-a523-ohci",
+ "generic-ohci";
+ reg = <0x04200400 0x100>;
+ interrupts = <GIC_SPI 160 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&ccu CLK_BUS_OHCI1>,
+ <&ccu CLK_USB_OHCI1>;
+ resets = <&ccu RST_BUS_OHCI1>;
+ phys = <&usbphy 1>;
+ phy-names = "usb";
+ status = "disabled";
+ };
+
+ pcie_rc: pcie@6000000 {
+ compatible = "allwinner,sun60i-a733-pcie";
+ reg = <0x06000000 0x10000>,
+ <0x06400000 0x1000>,
+ <0x20000000 0x1000000>;
+ reg-names = "dbi", "app", "config";
+ clocks = <&ccu CLK_PCIE_AUX>,
+ <&ccu CLK_PCIE_AXI_SLV>;
+ clock-names = "aux", "axi_slv";
+ resets = <&ccu RST_BUS_PCIE_PWRUP>,
+ <&ccu RST_BUS_PCIE>;
+ reset-names = "pwrup", "pcie";
+ phys = <&combophy1 PHY_TYPE_PCIE>;
+ phy-names = "pcie-phy";
+ pinctrl-names = "default";
+ pinctrl-0 = <&pcie_pins>;
+ interrupts = <GIC_SPI 152 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 153 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 136 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 137 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 138 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 139 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 140 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 141 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 142 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 143 IRQ_TYPE_LEVEL_HIGH>;
+ interrupt-names = "sii", "msi",
+ "edma-w0", "edma-w1", "edma-w2", "edma-w3",
+ "edma-r0", "edma-r1", "edma-r2", "edma-r3";
+ #address-cells = <3>;
+ #size-cells = <2>;
+ #interrupt-cells = <1>;
+ device_type = "pci";
+ bus-range = <0x0 0xff>;
+ ranges = <0x81000000 0x0 0x21000000 0x21000000 0x0 0x01000000>,
+ <0x82000000 0x0 0x22000000 0x22000000 0x0 0x0e000000>;
+ dma-ranges = <0x02000000 0x0 0x40000000 0x40000000 0x0 0xc0000000>;
+ interrupt-map-mask = <0 0 0 7>;
+ interrupt-map = <0 0 0 1 &pcie_intc 0>,
+ <0 0 0 2 &pcie_intc 1>,
+ <0 0 0 3 &pcie_intc 2>,
+ <0 0 0 4 &pcie_intc 3>;
+ msi-map = <0x0 &its 0x0 0x10000>;
+ num-lanes = <1>;
+ max-link-speed = <3>;
+ status = "disabled";
+
+ pcie_intc: legacy-interrupt-controller {
+ interrupt-controller;
+ #address-cells = <0>;
+ #interrupt-cells = <1>;
+ };
+ };
+
+ combophy1: phy@6c00000 {
+ compatible = "allwinner,sun60i-a733-combophy";
+ reg = <0x06c00000 0x10000>,
+ <0x06ca0000 0x20000>;
+ reg-names = "serdes", "phy";
+ clocks = <&ccu CLK_SERDES_PHY>,
+ <&ccu CLK_AHB_SERDES>,
+ <&ccu CLK_MBUS_SERDES>;
+ clock-names = "phy", "ahb", "mbus";
+ resets = <&ccu RST_BUS_SERDES>;
+ #phy-cells = <1>;
+ status = "disabled";
+ };
+
+ r_ccu: clock-controller@7010000 {
+ compatible = "allwinner,sun60i-a733-r-ccu";
+ reg = <0x07010000 0x250>;
+ clocks = <&osc>, <&rtc CLK_OSC32K>, <&rtc CLK_IOSC>,
+ <&ccu CLK_SYS_24M>, <&ccu CLK_PLL_PERIPH0_200M>,
+ <&ccu CLK_PLL_PERIPH0_300M>,
+ <&ccu CLK_PLL_PERIPH1_300M>;
+ clock-names = "hosc", "losc", "iosc", "sys-24m",
+ "pll-periph0-200m", "pll-periph0-300m",
+ "pll-periph1-300m";
+ #clock-cells = <1>;
+ #reset-cells = <1>;
+ };
+
+ nmi_intc: interrupt-controller@7010320 {
+ compatible = "allwinner,sun60i-a733-nmi",
+ "allwinner,sun55i-a523-nmi";
+ reg = <0x07010320 0xc>;
+ interrupts = <GIC_SPI 188 IRQ_TYPE_LEVEL_HIGH>;
+ interrupt-controller;
+ #interrupt-cells = <2>;
+ };
+
+ r_pio: pinctrl@7025000 {
+ compatible = "allwinner,sun60i-a733-r-pinctrl";
+ reg = <0x07025000 0x800>;
+ interrupts = <GIC_SPI 198 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 200 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&r_ccu CLK_R_APB0>, <&osc>, <&rtc CLK_OSC32K>;
+ clock-names = "apb", "hosc", "losc";
+ gpio-controller;
+ #gpio-cells = <3>;
+ interrupt-controller;
+ #interrupt-cells = <3>;
+
+ r_i2c0_pl_pins: r-i2c0-pl-pins {
+ pins = "PL0", "PL1";
+ function = "r_i2c0";
+ allwinner,pinmux = <2>;
+ };
+ };
+
+ r_uart0: serial@7080000 {
+ compatible = "allwinner,sun60i-a733-uart",
+ "snps,dw-apb-uart";
+ reg = <0x07080000 0x400>;
+ interrupts = <GIC_SPI 201 IRQ_TYPE_LEVEL_HIGH>;
+ reg-shift = <2>;
+ reg-io-width = <4>;
+ clocks = <&r_ccu CLK_BUS_R_UART0>;
+ resets = <&r_ccu RST_BUS_R_UART0>;
+ status = "disabled";
+ };
+
+ r_uart1: serial@7081000 {
+ compatible = "allwinner,sun60i-a733-uart",
+ "snps,dw-apb-uart";
+ reg = <0x07081000 0x400>;
+ interrupts = <GIC_SPI 202 IRQ_TYPE_LEVEL_HIGH>;
+ reg-shift = <2>;
+ reg-io-width = <4>;
+ clocks = <&r_ccu CLK_BUS_R_UART1>;
+ resets = <&r_ccu RST_BUS_R_UART1>;
+ status = "disabled";
+ };
+
+ r_i2c0: i2c@7083000 {
+ compatible = "allwinner,sun60i-a733-i2c",
+ "allwinner,sun8i-v536-i2c",
+ "allwinner,sun6i-a31-i2c";
+ reg = <0x07083000 0x400>;
+ interrupts = <GIC_SPI 203 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&r_ccu CLK_BUS_R_I2C0>;
+ resets = <&r_ccu RST_BUS_R_I2C0>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "disabled";
+ };
+
+ r_i2c1: i2c@7084000 {
+ compatible = "allwinner,sun60i-a733-i2c",
+ "allwinner,sun8i-v536-i2c",
+ "allwinner,sun6i-a31-i2c";
+ reg = <0x07084000 0x400>;
+ interrupts = <GIC_SPI 204 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&r_ccu CLK_BUS_R_I2C1>;
+ resets = <&r_ccu RST_BUS_R_I2C1>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "disabled";
+ };
+
+ r_i2c2: i2c@7085000 {
+ compatible = "allwinner,sun60i-a733-i2c",
+ "allwinner,sun8i-v536-i2c",
+ "allwinner,sun6i-a31-i2c";
+ reg = <0x07085000 0x400>;
+ interrupts = <GIC_SPI 205 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&r_ccu CLK_BUS_R_I2C2>;
+ resets = <&r_ccu RST_BUS_R_I2C2>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "disabled";
+ };
+
+ rtc: rtc@7090000 {
+ compatible = "allwinner,sun60i-a733-rtc",
+ "allwinner,sun50i-r329-rtc";
+ reg = <0x07090000 0x400>;
+ interrupts = <GIC_SPI 196 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&r_ccu CLK_BUS_R_RTC>, <&osc>, <&r_ccu CLK_R_AHB>;
+ clock-names = "bus", "hosc", "ahb";
+ #clock-cells = <1>;
+ };
+ };
+};
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 26/26] sunxi: Add Xunlong Orange Pi 4 Pro board support
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (24 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 25/26] arm64: dts: allwinner: add A733 SoC description Alastair D'Silva
@ 2026-11-13 2:32 ` Alastair D'Silva
2026-11-13 7:28 ` [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
2026-11-20 0:31 ` Andre Przywara
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 2:32 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
Add initial U-Boot board support and defconfig for the Xunlong Orange Pi 4
Pro single-board computer based on the Allwinner A733 SoC:
- Allwinner A733 octa-core SoC (6x Cortex-A55 + 2x Cortex-A76)
- 4 GiB / 6 GiB / 8 GiB / 16 GiB LPDDR5 at 3600 MT/s
- X-Powers AXP318W PMIC on R-I2C0 (PL0/PL1)
- MicroSD slot on MMC0 and optional eMMC module on MMC2 (HS200)
- USB 2.0 host port on EHCI1/OHCI1
- Debug UART0 console on PB0/PB1 (115200 8N1)
- M.2 Key-M PCIe Gen3 x1 NVMe slot via Combo PHY 1 and 3.3V regulator
Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
---
arch/arm/dts/Makefile | 2 +
arch/arm/dts/sun60i-a733-orangepi-4-pro.dts | 237 ++++++++++++++++++++
board/sunxi/MAINTAINERS | 6 +
configs/orangepi_4_pro_defconfig | 36 +++
4 files changed, 281 insertions(+)
create mode 100644 arch/arm/dts/sun60i-a733-orangepi-4-pro.dts
create mode 100644 configs/orangepi_4_pro_defconfig
diff --git a/arch/arm/dts/Makefile b/arch/arm/dts/Makefile
index cf4674042a2..a89021b6b79 100644
--- a/arch/arm/dts/Makefile
+++ b/arch/arm/dts/Makefile
@@ -657,6 +657,8 @@ dtb-$(CONFIG_MACH_SUN50I) += \
sun50i-a64-pinetab.dtb \
sun50i-a64-sopine-baseboard.dtb \
sun50i-a64-teres-i.dtb
+dtb-$(CONFIG_MACH_SUN60I_A733) += \
+ sun60i-a733-orangepi-4-pro.dtb
dtb-$(CONFIG_MACH_SUN9I) += \
sun9i-a80-cx-a99.dtb
diff --git a/arch/arm/dts/sun60i-a733-orangepi-4-pro.dts b/arch/arm/dts/sun60i-a733-orangepi-4-pro.dts
new file mode 100644
index 00000000000..819d3b1c4c3
--- /dev/null
+++ b/arch/arm/dts/sun60i-a733-orangepi-4-pro.dts
@@ -0,0 +1,237 @@
+// SPDX-License-Identifier: (GPL-2.0-only OR MIT)
+/*
+ * Copyright (C) 2026 Alastair D'Silva <alastair@d-silva.org>
+ */
+
+/dts-v1/;
+
+#include "sun60i-a733.dtsi"
+
+#include <dt-bindings/gpio/gpio.h>
+
+/ {
+ model = "Xunlong Orange Pi 4 Pro";
+ compatible = "xunlong,orangepi-4-pro", "allwinner,sun60i-a733";
+
+ aliases {
+ serial0 = &uart0;
+ };
+
+ chosen {
+ stdout-path = "serial0:115200n8";
+ };
+
+ reg_vcc5v: regulator-vcc5v {
+ compatible = "regulator-fixed";
+ regulator-name = "vcc-5v";
+ regulator-min-microvolt = <5000000>;
+ regulator-max-microvolt = <5000000>;
+ regulator-always-on;
+ };
+
+ reg_usb1_vbus: regulator-usb1-vbus {
+ compatible = "regulator-fixed";
+ regulator-name = "usb1-vbus";
+ regulator-min-microvolt = <5000000>;
+ regulator-max-microvolt = <5000000>;
+ vin-supply = <®_vcc5v>;
+ gpios = <&pio 1 8 GPIO_ACTIVE_HIGH>; /* PB8 */
+ enable-active-high;
+ };
+
+ reg_vcc3v3_pcie: regulator-vcc3v3-pcie {
+ compatible = "regulator-fixed";
+ regulator-name = "vcc3v3-pcie";
+ regulator-min-microvolt = <3300000>;
+ regulator-max-microvolt = <3300000>;
+ gpios = <&r_pio 0 3 GPIO_ACTIVE_HIGH>; /* PL3 */
+ enable-active-high;
+ };
+};
+
+&ehci1 {
+ status = "okay";
+};
+
+&combophy1 {
+ status = "okay";
+};
+
+&mmc0 {
+ vmmc-supply = <®_swout2>;
+ cd-gpios = <&pio 5 6 (GPIO_ACTIVE_HIGH | GPIO_PULL_UP)>; /* PF6 */
+ bus-width = <4>;
+ status = "okay";
+};
+
+&mmc2 {
+ pinctrl-names = "default";
+ pinctrl-0 = <&mmc2_pins>, <&mmc2_rst_pin>;
+ vmmc-supply = <®_dcdc1>;
+ vqmmc-supply = <®_bldo1>;
+ bus-width = <8>;
+ non-removable;
+ cap-mmc-hw-reset;
+ mmc-ddr-1_8v;
+ mmc-hs200-1_8v;
+ status = "okay";
+};
+
+&ohci1 {
+ status = "okay";
+};
+
+&osc {
+ clock-frequency = <24000000>;
+};
+
+&pcie_rc {
+ vpcie3v3-supply = <®_vcc3v3_pcie>;
+ reset-gpios = <&pio 3 22 GPIO_ACTIVE_LOW>; /* PD22 */
+ status = "okay";
+};
+
+&pio {
+ vcc-pb-supply = <®_dcdc1>;
+ vcc-pc-supply = <®_bldo1>;
+ vcc-pf-supply = <®_dcdc1>;
+ vcc-pg-supply = <®_bldo1>;
+ vcc-pj-supply = <®_bldo2>;
+ vcc-pk-supply = <®_bldo1>;
+};
+
+&r_i2c0 {
+ pinctrl-names = "default";
+ pinctrl-0 = <&r_i2c0_pl_pins>;
+ clock-frequency = <400000>;
+ status = "okay";
+
+ axp318w: pmic@36 {
+ compatible = "x-powers,axp318w";
+ reg = <0x36>;
+
+ regulators {
+ reg_dcdc1: dcdc1 {
+ regulator-name = "vcc-io";
+ regulator-min-microvolt = <3300000>;
+ regulator-max-microvolt = <3300000>;
+ regulator-always-on;
+ };
+
+ reg_dcdc2: dcdc2 {
+ regulator-name = "vdd-cpul";
+ regulator-min-microvolt = <500000>;
+ regulator-max-microvolt = <1400000>;
+ regulator-always-on;
+ };
+
+ reg_dcdc3: dcdc3 {
+ regulator-name = "vdd-cpub";
+ regulator-min-microvolt = <500000>;
+ regulator-max-microvolt = <1400000>;
+ regulator-always-on;
+ };
+
+ reg_dcdc4: dcdc4 {
+ regulator-name = "vdd-gpu";
+ regulator-min-microvolt = <500000>;
+ regulator-max-microvolt = <1100000>;
+ regulator-always-on;
+ };
+
+ reg_dcdc5: dcdc5 {
+ regulator-name = "vdd-sys";
+ regulator-min-microvolt = <900000>;
+ regulator-max-microvolt = <1100000>;
+ regulator-always-on;
+ };
+
+ reg_dcdc6: dcdc6 {
+ regulator-name = "vddq-lpddr5";
+ regulator-min-microvolt = <500000>;
+ regulator-max-microvolt = <600000>;
+ regulator-always-on;
+ };
+
+ reg_dcdc7: dcdc7 {
+ regulator-name = "vdd2-lpddr5";
+ regulator-min-microvolt = <1050000>;
+ regulator-max-microvolt = <1120000>;
+ regulator-always-on;
+ };
+
+ reg_bldo1: bldo1 {
+ regulator-name = "vcc-1v8-io";
+ regulator-min-microvolt = <1800000>;
+ regulator-max-microvolt = <1800000>;
+ regulator-always-on;
+ };
+
+ reg_bldo2: bldo2 {
+ regulator-name = "vdd1-lpddr5";
+ regulator-min-microvolt = <1800000>;
+ regulator-max-microvolt = <1800000>;
+ regulator-always-on;
+ };
+
+ reg_bldo4: bldo4 {
+ regulator-name = "vcc-1v8-audio";
+ regulator-min-microvolt = <1800000>;
+ regulator-max-microvolt = <1800000>;
+ regulator-always-on;
+ };
+
+ reg_dldo1: dldo1 {
+ regulator-name = "vcc-sd";
+ regulator-min-microvolt = <3300000>;
+ regulator-max-microvolt = <3300000>;
+ regulator-always-on;
+ };
+
+ reg_eldo1: eldo1 {
+ regulator-name = "vdd2h-lpddr5";
+ regulator-min-microvolt = <1050000>;
+ regulator-max-microvolt = <1100000>;
+ regulator-always-on;
+ };
+
+ reg_swout1: swout1 {
+ regulator-name = "vcc-ephy";
+ regulator-always-on;
+ };
+
+ reg_swout2: swout2 {
+ regulator-name = "vcc-gmac-io";
+ regulator-always-on;
+ };
+ };
+ };
+};
+
+&r_pio {
+ vcc-pm-supply = <®_bldo1>;
+};
+
+/* Optional onboard SPI NOR flash footprint; unpopulated on standard SKUs. */
+&spi0 {
+ pinctrl-names = "default";
+ pinctrl-0 = <&spi0_pc_pins>;
+ status = "disabled";
+
+ flash@0 {
+ compatible = "jedec,spi-nor";
+ reg = <0>;
+ spi-max-frequency = <50000000>;
+ };
+};
+
+&uart0 {
+ pinctrl-names = "default";
+ pinctrl-0 = <&uart0_pb0_pins>;
+ status = "okay";
+};
+
+&usbphy {
+ usb1_vbus-supply = <®_usb1_vbus>;
+ status = "okay";
+};
diff --git a/board/sunxi/MAINTAINERS b/board/sunxi/MAINTAINERS
index 344f52ea325..a42c89c1304 100644
--- a/board/sunxi/MAINTAINERS
+++ b/board/sunxi/MAINTAINERS
@@ -443,6 +443,12 @@ M: Andre Heider <a.heider@gmail.com>
S: Maintained
F: configs/orangepi_3_defconfig
+ORANGEPI 4 PRO BOARD
+M: Alastair D'Silva <alastair@d-silva.org>
+S: Maintained
+F: arch/arm/dts/sun60i-a733-orangepi-4-pro.dts
+F: configs/orangepi_4_pro_defconfig
+
ORANGEPI 4A BOARD
M: Chen-Yu Tsai <wens@kernel.org>
S: Maintained
diff --git a/configs/orangepi_4_pro_defconfig b/configs/orangepi_4_pro_defconfig
new file mode 100644
index 00000000000..5f620a533a0
--- /dev/null
+++ b/configs/orangepi_4_pro_defconfig
@@ -0,0 +1,36 @@
+CONFIG_ARM=y
+CONFIG_ARCH_SUNXI=y
+CONFIG_DEFAULT_DEVICE_TREE="sun60i-a733-orangepi-4-pro"
+CONFIG_SPL=y
+CONFIG_DRAM_SUN60I_A733_PIN_SWIZZLE=y
+CONFIG_MACH_SUN60I_A733=y
+CONFIG_UART0_PORT_B0=y
+CONFIG_MMC_SUNXI_SLOT_EXTRA=2
+CONFIG_R_I2C_ENABLE=y
+CONFIG_PCI=y
+CONFIG_SPL_I2C=y
+CONFIG_CMD_PCI=y
+CONFIG_CMD_PMIC=y
+CONFIG_CMD_REGULATOR=y
+CONFIG_SYS_I2C_MVTWSI=y
+CONFIG_SYS_I2C_SLAVE=0x7f
+CONFIG_SYS_I2C_SPEED=400000
+CONFIG_SUPPORT_EMMC_BOOT=y
+CONFIG_NVME_PCI=y
+CONFIG_PCIE_DW_SUNXI=y
+CONFIG_PHY_SUN60I_COMBOPHY=y
+CONFIG_REGULATOR_AXP=y
+CONFIG_AXP_DCDC1_VOLT=3300
+CONFIG_AXP_DCDC2_VOLT=800
+CONFIG_AXP_DCDC3_VOLT=1050
+CONFIG_AXP_DCDC4_VOLT=800
+CONFIG_AXP_DCDC5_VOLT=1000
+CONFIG_AXP_DCDC6_VOLT=560
+CONFIG_AXP_DCDC7_VOLT=1080
+CONFIG_AXP_BLDO1_VOLT=1800
+CONFIG_AXP_BLDO2_VOLT=1800
+CONFIG_AXP_BLDO4_VOLT=1800
+CONFIG_AXP_DLDO1_VOLT=3300
+CONFIG_AXP_ELDO1_VOLT=1075
+CONFIG_USB_EHCI_HCD=y
+CONFIG_USB_OHCI_HCD=y
--
2.53.0
^ permalink raw reply related [flat|nested] 31+ messages in thread
* Re: [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (25 preceding siblings ...)
2026-11-13 2:32 ` [PATCH 26/26] sunxi: Add Xunlong Orange Pi 4 Pro board support Alastair D'Silva
@ 2026-11-13 7:28 ` Alastair D'Silva
2026-11-20 0:31 ` Andre Przywara
27 siblings, 0 replies; 31+ messages in thread
From: Alastair D'Silva @ 2026-11-13 7:28 UTC (permalink / raw)
To: u-boot
Cc: Tom Rini, Andre Przywara, Hans de Goede, Ilias Apalodimas,
Jaehoon Chung, Lukasz Majewski, Neil Armstrong, Peng Fan,
Tanmay Kathpalia, Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu,
Alastair D'Silva
Hi all,
A quick clarification for transparency on this series: LLM assistance
(Google Antigravity / Gemini) was used during development and debugging
of these patches.
I will ensure the appropriate 'Assisted-by: LLM' (or project-preferred
equivalent) tags are included in the commit trailers when rolling v2.
Thanks,
Alastair
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 13/26] sunxi: fel: Add Allwinner A733 ERET FEL return support
2026-11-13 2:32 ` [PATCH 13/26] sunxi: fel: Add Allwinner A733 ERET FEL return support Alastair D'Silva
@ 2026-11-19 23:53 ` Andre Przywara
0 siblings, 0 replies; 31+ messages in thread
From: Andre Przywara @ 2026-11-19 23:53 UTC (permalink / raw)
To: Alastair D'Silva
Cc: u-boot, Tom Rini, Hans de Goede, Ilias Apalodimas, Jaehoon Chung,
Lukasz Majewski, Neil Armstrong, Peng Fan, Tanmay Kathpalia,
Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu
Hi,
On Fri, 13 Nov 2026 13:32:23 +1100
Alastair D'Silva <alastair@d-silva.org> wrote:
> On the Allwinner A733, boot0.h latches CLU0_CPU0_CTRL_REG to AArch64
> and sunxi-fel uses SRAM swap buffers across 0x47000.
Sorry, but what? Claude, is that you? What's that supposed to mean, how
are those two statements related and why does that matter?
> Rather than arming
> R_CPUCFG hotplug magic and warm-resetting through BROM, drop directly
Why?
> to Secure AArch32 SVC via ERET after draining UART0 TX, and install an
I am afraid this doesn't really work: even though the core resets in
AArch32 secure SVC, this isn't the same as ERETing into that
mode: the former is PL3, so the highest implemented exception level, the
latter is not. This is admittedly a weird corner case of AArch32
firmware operation on Armv8 hardware, I guess trying to somewhat
retro-fit the clear ARMv8 exception levels into the slightly murky
situation on ARMv7.
The difference is that some operations are reserved for running in the
highest implemented exception level: writing CNTFRQ_EL0, and writing to
RVBAR are prominent examples. The former is bad already, but the latter
prevents switching to AArch64, which is crucial for our boot flow.
So again, why is this needed, and what is the problem with the existing
way of doing things?
> EL3 SMC vector in SH_SRAM (0x00080000) so sunxi-fel can re-enter AArch64
> EL3 for BL31.
>
> Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
> ---
> arch/arm/cpu/armv8/fel_utils.S | 113 ++++++++++++++++++++++++++++++++-
> 1 file changed, 112 insertions(+), 1 deletion(-)
>
> diff --git a/arch/arm/cpu/armv8/fel_utils.S b/arch/arm/cpu/armv8/fel_utils.S
> index ccddfaaf04c..fc0e0b93c37 100644
> --- a/arch/arm/cpu/armv8/fel_utils.S
> +++ b/arch/arm/cpu/armv8/fel_utils.S
> @@ -11,6 +11,30 @@
> #include <asm/system.h>
> #include <linux/linkage.h>
> #include <asm/arch/cpu.h>
> +#include <asm/arch/serial.h>
> +
> +#ifdef CONFIG_MACH_SUN60I_A733
> +#define A733_FEL_EL3_VBAR SUNXI_FEL_EL3_VBAR
> +#define A733_FEL_EL3_A32_SYNC_OFF 0x600
> +#define UART_LSR_OFFSET 0x14
> +#define UART_LSR_TEMT_BIT 6
> +#define SCR_EL3_IRQ_EN (1 << 1)
> +#define SCR_EL3_FIQ_EN (1 << 2)
> +#define SCR_EL3_FEL_A32_CLEAR (SCR_EL3_RW_AARCH64 | \
> + SCR_EL3_SMD_DIS | \
> + SCR_EL3_EA_EN | \
> + SCR_EL3_FIQ_EN | \
> + SCR_EL3_IRQ_EN | \
> + SCR_EL3_NS_EN)
> +#define ICC_SRE_EL3_ENABLE_ALL 0xf
> +#define ICC_SRE_EL1_ENABLE_ALL 0x7
> +#define ICC_IGRPEN1_EL3_ENABLE_ALL 0x3
> +#define SPSR_EL3_A32_SVC_MASKED (SPSR_EL_ASYN_MASK | \
> + SPSR_EL_IRQ_MASK | \
> + SPSR_EL_FIQ_MASK | \
> + SPSR_EL_M_AARCH32 | \
> + SPSR_EL_M_SVC)
> +#endif
>
> /*
> * We don't overwrite save_boot_params() here, to save the FEL state upon
> @@ -40,6 +64,89 @@ ENTRY(return_to_fel)
> adr x1, fel_stash_addr // to find the fel_stash address in AA32
> str w2, [x1]
>
> +#if defined(CONFIG_MACH_SUN60I_A733)
> +#if CONFIG_IS_ENABLED(SERIAL)
> + /* Drain UART0 TX FIFO before dropping to AArch32 */
Why? And why is this not done in C?
> + ldr x2, =SUNXI_UART0_BASE
> + mov w1, #0x10000
> +2: ldr w0, [x2, #UART_LSR_OFFSET]
> + tbnz w0, #UART_LSR_TEMT_BIT, 3f
> + subs w1, w1, #1
> + b.ne 2b
Yeah, no, this seems like another case of an LLM going wild in writing
assembly code, and adding many hard-to-parse lines (without comments),
to solve an issue that isn't even properly justified.
Cheers,
Andre
> +3:
> +#endif
> +
> + /*
> + * On A733, boot0.h latches CLU0_CPU0_CTRL_REG to AArch64 and FEL uses
> + * SRAM swap buffers across 0x47000. Rather than warm-resetting through
> + * the BROM hotplug path, drop directly to Secure AArch32 SVC via ERET
> + * and install an EL3 SMC vector in SH_SRAM outside the FEL swap buffer
> + * so sunxi-fel can re-enter AArch64 EL3 for BL31.
> + */
> + adr x4, el3_a32_sync
> + adr x5, el3_a32_sync_end
> + ldr x6, =(A733_FEL_EL3_VBAR + A733_FEL_EL3_A32_SYNC_OFF)
> + mov x8, x6
> +4: ldr w7, [x4], #4
> + str w7, [x6], #4
> + cmp x4, x5
> + b.lo 4b
> +
> +5: dc cvau, x8
> + add x8, x8, #64
> + cmp x8, x6
> + b.lo 5b
> + dsb ish
> + ic iallu
> + dsb ish
> + isb
> +
> + ldr x0, =A733_FEL_EL3_VBAR
> + msr vbar_el3, x0
> +
> + mrs x1, ICC_SRE_EL3
> + orr x1, x1, #ICC_SRE_EL3_ENABLE_ALL
> + msr ICC_SRE_EL3, x1
> + isb
> +
> + mrs x1, ICC_SRE_EL1
> + orr x1, x1, #ICC_SRE_EL1_ENABLE_ALL
> + msr ICC_SRE_EL1, x1
> + mov x1, #ICC_IGRPEN1_EL3_ENABLE_ALL
> + msr ICC_IGRPEN1_EL3, x1
> +
> + mrs x0, scr_el3
> + mov x1, #SCR_EL3_FEL_A32_CLEAR
> + bic x0, x0, x1
> + msr scr_el3, x0
> +
> + mov x0, #SPSR_EL3_A32_SVC_MASKED
> + msr spsr_el3, x0
> +
> + adr x0, back_in_32
> + msr elr_el3, x0
> +
> + dsb sy
> + isb
> + eret
> +
> + .align 2
> +el3_a32_sync:
> + mrs x9, scr_el3
> + mov x10, #SCR_EL3_RW_AARCH64
> + orr x9, x9, x10
> + msr scr_el3, x9
> + isb
> + cbnz w0, 6f
> + movz w0, #(CONFIG_SUNXI_BL31_BASE & 0xffff)
> + movk w0, #(CONFIG_SUNXI_BL31_BASE >> 16), lsl #16
> +6: uxtw x0, w0
> + ic iallu
> + dsb sy
> + isb
> + br x0
> +el3_a32_sync_end:
> +#else
> ldr w0, =0xfa50392f // CPU hotplug magic
> #if defined(CONFIG_MACH_SUN50I_H616) || defined(CONFIG_MACH_SUN50I_A133) || \
> defined(CONFIG_MACH_SUN55I_A523)
> @@ -63,6 +170,7 @@ ENTRY(return_to_fel)
> isb sy
> 1: wfi
> b 1b
> +#endif
>
> fel_stash_addr: // must immediately precede back_in_32:
> .word 0x00000000 // receives fel_stash addr, by AA64 code above
> @@ -71,6 +179,9 @@ fel_stash_addr: // must immediately precede back_in_32:
> back_in_32:
> .word 0xe51f000c // ldr r0, [pc, #-12] ; load fel_stash address
> .word 0xe5901008 // ldr r1, [r0, #8]
> +#ifdef CONFIG_MACH_SUN60I_A733
> + .word 0xe38110c0 // orr r1, r1, #0xc0 ; mask IRQ/FIQ in thunk
> +#endif
> .word 0xe129f001 // msr CPSR_fc, r1
> .word 0xf57ff06f // isb
> .word 0xe590d000 // ldr sp, [r0]
> @@ -82,7 +193,7 @@ back_in_32:
> .word 0xe590100c // ldr r1, [r0, #12]
> .word 0xee011f10 // mcr 15, 0, r1, cr1, cr0, {0} ; SCTLR
> .word 0xf57ff06f // isb
> -#ifdef CONFIG_MACH_SUN55I_A523
> +#if defined(CONFIG_MACH_SUN55I_A523) || defined(CONFIG_MACH_SUN60I_A733)
> .word 0xe5901018 // ldr r1, [r0, #24]
> .word 0xee041f16 // mcr 15, 0, r1, cr4, cr6, {0}; ICC_PMR
> .word 0xe590101c // ldr r1, [r0, #28]
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 01/26] power: regulator: axp: Fix AXP318W ELDO voltage mask
2026-11-13 2:32 ` [PATCH 01/26] power: regulator: axp: Fix AXP318W ELDO voltage mask Alastair D'Silva
@ 2026-11-20 0:06 ` Andre Przywara
0 siblings, 0 replies; 31+ messages in thread
From: Andre Przywara @ 2026-11-20 0:06 UTC (permalink / raw)
To: Alastair D'Silva
Cc: u-boot, Tom Rini, Hans de Goede, Ilias Apalodimas, Jaehoon Chung,
Lukasz Majewski, Neil Armstrong, Peng Fan, Tanmay Kathpalia,
Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu
On Fri, 13 Nov 2026 13:32:11 +1100
Alastair D'Silva <alastair@d-silva.org> wrote:
> On AXP318W, ELDO1-ELDO6 span 500 mV to 1500 mV in 25 mV steps (40 steps,
> selectors 0x00..0x28), requiring a 6-bit selector mask (0x3f) rather
> than the 5-bit mask (0x1f, max 0x1f -> 1275 mV) currently listed in
> axp318_regulators[].]
That is correct, but can you say what this fixes?
>
> Fixes: 71f07932756f ("power: regulator: add AXP318W support")
> Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
Reviewed-by: Andre Przywara <andre.przywara@arm.com>
Cheers,
Andre
> ---
> drivers/power/regulator/axp_regulator.c | 12 ++++++------
> 1 file changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/power/regulator/axp_regulator.c b/drivers/power/regulator/axp_regulator.c
> index 16d3a8f7f90..e425f3bf8b2 100644
> --- a/drivers/power/regulator/axp_regulator.c
> +++ b/drivers/power/regulator/axp_regulator.c
> @@ -227,12 +227,12 @@ static const struct axp_regulator_plat axp318_regulators[] = {
> { "dldo4", 0x22, BIT(3), 0x37, 0x1f, 500, 3400, 100, NA },
> { "dldo5", 0x22, BIT(4), 0x38, 0x1f, 500, 3400, 100, NA },
> { "dldo6", 0x22, BIT(5), 0x39, 0x1f, 500, 3400, 100, NA },
> - { "eldo1", 0x22, BIT(6), 0x3a, 0x1f, 500, 1500, 25, NA },
> - { "eldo2", 0x22, BIT(7), 0x3b, 0x1f, 500, 1500, 25, NA },
> - { "eldo3", 0x23, BIT(0), 0x3c, 0x1f, 500, 1500, 25, NA },
> - { "eldo4", 0x23, BIT(1), 0x3d, 0x1f, 500, 1500, 25, NA },
> - { "eldo5", 0x23, BIT(2), 0x3e, 0x1f, 500, 1500, 25, NA },
> - { "eldo6", 0x23, BIT(3), 0x3f, 0x1f, 500, 1500, 25, NA },
> + { "eldo1", 0x22, BIT(6), 0x3a, 0x3f, 500, 1500, 25, NA },
> + { "eldo2", 0x22, BIT(7), 0x3b, 0x3f, 500, 1500, 25, NA },
> + { "eldo3", 0x23, BIT(0), 0x3c, 0x3f, 500, 1500, 25, NA },
> + { "eldo4", 0x23, BIT(1), 0x3d, 0x3f, 500, 1500, 25, NA },
> + { "eldo5", 0x23, BIT(2), 0x3e, 0x3f, 500, 1500, 25, NA },
> + { "eldo6", 0x23, BIT(3), 0x3f, 0x3f, 500, 1500, 25, NA },
> { "swout1", 0x11, BIT(3), NA, NA, NA, NA, NA, NA },
> { "swout2", 0x11, BIT(4), NA, NA, NA, NA, NA, NA },
> { }
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
` (26 preceding siblings ...)
2026-11-13 7:28 ` [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
@ 2026-11-20 0:31 ` Andre Przywara
27 siblings, 0 replies; 31+ messages in thread
From: Andre Przywara @ 2026-11-20 0:31 UTC (permalink / raw)
To: Alastair D'Silva
Cc: u-boot, Tom Rini, Hans de Goede, Ilias Apalodimas, Jaehoon Chung,
Lukasz Majewski, Neil Armstrong, Peng Fan, Tanmay Kathpalia,
Yixun Lan, Yixun Lan, Enzo Adriano, Junhui Liu, Chen-Yu Tsai,
Jernej Škrabec
On Fri, 13 Nov 2026 13:32:10 +1100
Alastair D'Silva <alastair@d-silva.org> wrote:
Hi Alastair,
thanks for sending this, and for keeping existing patches and their
authors!
> This series adds initial support for the Allwinner A733 (sun60iw2 / sun60i)
> SoC and the Xunlong Orange Pi 4 Pro board to U-Boot.
Please note that the DT bits need to go through the kernel repo
first, so you better send your OrangePi DT to the kernel lists ASAP, to
get this in the wake of the proposed A733 .dtsi and Cubie A7Z patches.
Though it's not clear at this point whether they make it into v7.4.
> The A733 is an octa-core ARMv8 SoC (dual-cluster Cortex-A76 + Cortex-A55)
> with LPDDR5 memory trained by an on-chip Synopsys ARC PMU, an AXP318W PMIC,
> eMMC/SD (MMC0/1/2), USB 2.0 EHCI/OHCI, and a DesignWare PCIe Gen3 x1 host
> controller paired with a Cadence Combo PHY.
>
> Overview of the series:
xx/26 doesn't look very manageable, so we would need to split this up,
probably along the lines of your groups below:
> - Patches 1-3: Fix AXP318W ELDO voltage mask and sysreset shutdown register,
> and extend SPL PMIC support for DCDC6/7 and ALDO/BLDO/DLDO/ELDO rails
> required to bring up LPDDR5 and board I/O.
As the AXP318W support is already in (though sadly unused), those
fixes/changes should be a separate series, as there wouldn't be any new
dependencies.
> - Patches 4-11: Add A733 CCU/R-CCU dt-bindings, GPIO, pinctrl, clock/reset,
> MMC (including transfer-size-scaled data timeout), and UART console support.
> - Patches 12-13: Add disposable .init.rodata support for AArch64 SPL (allowing
> large DRAM init tables in SRAM A2 without exceeding the 0x58000 BL31 load
> address) and ERET-based FEL return support for A733.
As mentioned above, we import DT files and binding headers from the
Linux kernel (through the DT rebasing repo), so the clock bits have to
wait for that to hit mainline. Since Chen-Yu has prepared this already
for linux-next, this should come with a release tag beginning of
January, right in time to cherry-pick this for the next U-Boot merge
window. The clock and peripheral patches are fairly straight-forward,
so I am tempted to merge them ASAP, to allow at least a "U-Boot proper
only" build. This works quite nicely for me here locally. So those
patches should be a separate series as well.
> - Patches 14-21: Add freestanding ARC PMU LPDDR5 training firmware (built from
> C via ARC_CROSS_COMPILE, or supplied via PMU_FW= / BINMAN_ALLOW_MISSING=1),
> the A733 LPDDR5 DRAM driver, initial sun60i SoC support, and documentation
> in doc/board/allwinner/a733-pmu.rst.
Yeah, the DRAM setup code is of course ... interesting, and surely
requires some proper review. Also we need to figure out the boot flow,
alongside the TF-A implementation, to not let them clash. It looks
like build-wise this is somewhat independent from the rest, so could
be a separate series.
> - Patches 22-24: Add the A733 Cadence Combo PHY driver, DesignWare PCIe
> controller driver, and NVMe distro boot target support. Note that a small
> SoC helper (sunxi_a733_pcie_soc_init) handles PCIe/SerDes CCU muxes,
> RTC DCXO feed, SYS_PLL, and NSI QoS until U-Boot's gate-only clk_sunxi
> driver gains rate/parent support for A733.
NVMe support is a welcomed surprise, but again should be independent of
the rest, both build-wise and functionality-wise. I wonder if that
already works for the A523, maybe you can have a brief look into that.
Then this again could be upstreamed separately, as a distinct series.
And I guess this is also gated on upstream Linux DT bindings.
> - Patches 25-26: Add the sun60i-a733.dtsi SoC device tree and Xunlong
> Orange Pi 4 Pro board support (defconfig, DTS, and MAINTAINERS).
As mentioned above, this needs to go through the kernel first, then it
would be imported either automatically or courtesy of an explicit
cherry pick.
Thanks,
Andre
> Tested on live Xunlong Orange Pi 4 Pro (6 GiB LPDDR5 @ 3600 MT/s) via FEL
> boot with TF-A BL31 v2.15.0 + pending Allwinner A733 TF-A patches:
> - https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/55200
> (feat(allwinner): add A733 support)
> - https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/55204
> (feat(allwinner): a733: ungate security controllers)
> - https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/55205
> (fix(allwinner): restore core security on power-on)
>
> Verified functionality:
> - SPL DRAM training (6144 MiB LPDDR5, 3600 MT/s)
> - SD card (mmc0: SD32G 29.7 GiB, 4-bit)
> - PCIe Gen3 + NVMe SSD (UHPVN0153EQ0V2 238.4 GB, partition enumeration)
>
> Alastair D'Silva (17):
> power: regulator: axp: Fix AXP318W ELDO voltage mask
> power: pmic: axp: Fix AXP318W sysreset shutdown register
> power: axp: Extend AXP318W SPL support with DCDC6/7 and LDOs
> mmc: sunxi: Scale data transfer timeout with transfer size
> arm64: spl: Support disposable .init.rodata section after
> _image_binary_end
> sunxi: fel: Add Allwinner A733 ERET FEL return support
> sunxi: dram: a733: Add ARC PMU register, ABI, and build definitions
> sunxi: dram: a733: Add ARC PMU primitives, clock, telemetry, and CBT
> sunxi: dram: a733: Add ARC PMU deskew and BIST training firmware
> sunxi: dram: a733: Add ARC PMU 1D/2D eye training firmware
> sunxi: dram: a733: Add ARC PMU training state machine
> sunxi: dram: a733: Add Allwinner A733 LPDDR5 DRAM driver
> doc: sunxi: Document Allwinner A733 LPDDR5 PMU architecture
> phy: sunxi: Add Allwinner A733 Cadence Combo PHY driver
> pci: sunxi: Add Allwinner A733 DesignWare PCIe host controller driver
> sunxi: Add NVMe to distro boot targets
> sunxi: Add Xunlong Orange Pi 4 Pro board support
>
> Enzo Adriano (1):
> arm64: dts: allwinner: add A733 SoC description
>
> Junhui Liu (1):
> dt-bindings: clk: sun60i-a733-ccu: Add Allwinner A733 support
>
> Yixun Lan (7):
> gpio: a733: add initial support
> pinctrl: sunxi: a733: add initial support
> clock: reset: a733: add initial support
> clock: a733: implement support for mmc/uart
> sunxi: mmc: enable support for A733
> sunxi: a733: uart: setup serial console
> sunxi: sun60i: initial support for A733 SoC
>
> MAINTAINERS | 13 +
> arch/arm/config.mk | 4 +
> arch/arm/cpu/armv8/fel_utils.S | 113 +-
> arch/arm/cpu/armv8/u-boot-spl.lds | 20 +
> arch/arm/cpu/armv8/u-boot.lds | 1 +
> arch/arm/dts/Makefile | 2 +
> arch/arm/dts/sun60i-a733-orangepi-4-pro.dts | 237 +
> arch/arm/dts/sun60i-a733.dtsi | 816 +++
> arch/arm/include/asm/arch-sunxi/boot0.h | 20 +-
> arch/arm/include/asm/arch-sunxi/clock.h | 1 +
> .../include/asm/arch-sunxi/clock_sun50i_h6.h | 19 +
> .../include/asm/arch-sunxi/cpu_sunxi_ncat2.h | 43 +-
> arch/arm/include/asm/arch-sunxi/dram.h | 2 +
> arch/arm/include/asm/arch-sunxi/serial.h | 15 +-
> arch/arm/include/asm/arch-sunxi/spl.h | 3 +
> arch/arm/include/asm/arch-sunxi/watchdog.h | 1 +
> arch/arm/mach-sunxi/Kconfig | 47 +-
> arch/arm/mach-sunxi/Makefile | 37 +
> arch/arm/mach-sunxi/board.c | 16 +
> arch/arm/mach-sunxi/clock_sun50i_h6.c | 149 +-
> arch/arm/mach-sunxi/cpu_info.c | 2 +
> arch/arm/mach-sunxi/dram_sun60i_a733.c | 1303 +++++
> arch/arm/mach-sunxi/dram_sun60i_a733_tables.h | 941 ++++
> arch/arm/mach-sunxi/pmu/build_pmu_fw.py | 257 +
> arch/arm/mach-sunxi/pmu/dwc_ddrphy_pmu_regs.h | 463 ++
> arch/arm/mach-sunxi/pmu/pmu.ld | 38 +
> arch/arm/mach-sunxi/pmu/pmu_cbt.c | 417 ++
> arch/arm/mach-sunxi/pmu/pmu_clock.c | 938 ++++
> arch/arm/mach-sunxi/pmu/pmu_deskew.c | 4895 +++++++++++++++++
> arch/arm/mach-sunxi/pmu/pmu_eye.c | 3224 +++++++++++
> arch/arm/mach-sunxi/pmu/pmu_fw.h | 553 ++
> arch/arm/mach-sunxi/pmu/pmu_primitives.c | 531 ++
> arch/arm/mach-sunxi/pmu/pmu_telemetry.c | 619 +++
> arch/arm/mach-sunxi/pmu/pmu_train.c | 2748 +++++++++
> arch/arm/mach-sunxi/pmu/sunxi_pmu_abi.h | 251 +
> arch/arm/mach-sunxi/rmr_switch.S | 22 +-
> board/sunxi/MAINTAINERS | 6 +
> board/sunxi/board.c | 48 +-
> common/spl/Kconfig | 22 +
> configs/orangepi_4_pro_defconfig | 36 +
> doc/board/allwinner/a733-pmu.rst | 299 +
> doc/board/allwinner/index.rst | 1 +
> doc/board/allwinner/sunxi.rst | 14 +-
> drivers/clk/sunxi/Kconfig | 14 +
> drivers/clk/sunxi/Makefile | 2 +
> drivers/clk/sunxi/clk_a733.c | 142 +
> drivers/clk/sunxi/clk_a733_r.c | 45 +
> drivers/clk/sunxi/clk_sunxi.c | 10 +
> drivers/gpio/Kconfig | 7 +
> drivers/gpio/sunxi_gpio.c | 49 +-
> drivers/mmc/sunxi_mmc.c | 48 +-
> drivers/pci/Kconfig | 11 +
> drivers/pci/Makefile | 1 +
> drivers/pci/pcie_dw_sunxi.c | 355 ++
> drivers/phy/allwinner/Kconfig | 8 +
> drivers/phy/allwinner/Makefile | 1 +
> drivers/phy/allwinner/phy-sun60i-combophy.c | 293 +
> drivers/pinctrl/sunxi/Kconfig | 18 +
> drivers/pinctrl/sunxi/pinctrl-sunxi.c | 98 +-
> drivers/power/Kconfig | 110 +-
> drivers/power/axp_spl.c | 119 +-
> drivers/power/pmic/axp.c | 4 +-
> drivers/power/regulator/axp_regulator.c | 12 +-
> include/axp_pmic.h | 5 +
> include/configs/sun60i.h | 11 +
> include/configs/sunxi-common.h | 7 +
> .../clock/allwinner,sun60i-a733-ccu.h | 291 +
> .../clock/allwinner,sun60i-a733-r-ccu.h | 38 +
> .../reset/allwinner,sun60i-a733-ccu.h | 129 +
> .../reset/allwinner,sun60i-a733-r-ccu.h | 23 +
> include/sunxi_gpio.h | 10 +-
> 71 files changed, 20948 insertions(+), 100 deletions(-)
> create mode 100644 arch/arm/dts/sun60i-a733-orangepi-4-pro.dts
> create mode 100644 arch/arm/dts/sun60i-a733.dtsi
> create mode 100644 arch/arm/mach-sunxi/dram_sun60i_a733.c
> create mode 100644 arch/arm/mach-sunxi/dram_sun60i_a733_tables.h
> create mode 100755 arch/arm/mach-sunxi/pmu/build_pmu_fw.py
> create mode 100644 arch/arm/mach-sunxi/pmu/dwc_ddrphy_pmu_regs.h
> create mode 100644 arch/arm/mach-sunxi/pmu/pmu.ld
> create mode 100644 arch/arm/mach-sunxi/pmu/pmu_cbt.c
> create mode 100644 arch/arm/mach-sunxi/pmu/pmu_clock.c
> create mode 100644 arch/arm/mach-sunxi/pmu/pmu_deskew.c
> create mode 100644 arch/arm/mach-sunxi/pmu/pmu_eye.c
> create mode 100644 arch/arm/mach-sunxi/pmu/pmu_fw.h
> create mode 100644 arch/arm/mach-sunxi/pmu/pmu_primitives.c
> create mode 100644 arch/arm/mach-sunxi/pmu/pmu_telemetry.c
> create mode 100644 arch/arm/mach-sunxi/pmu/pmu_train.c
> create mode 100644 arch/arm/mach-sunxi/pmu/sunxi_pmu_abi.h
> create mode 100644 configs/orangepi_4_pro_defconfig
> create mode 100644 doc/board/allwinner/a733-pmu.rst
> create mode 100644 drivers/clk/sunxi/clk_a733.c
> create mode 100644 drivers/clk/sunxi/clk_a733_r.c
> create mode 100644 drivers/pci/pcie_dw_sunxi.c
> create mode 100644 drivers/phy/allwinner/phy-sun60i-combophy.c
> create mode 100644 include/configs/sun60i.h
> create mode 100644 include/dt-bindings/clock/allwinner,sun60i-a733-ccu.h
> create mode 100644 include/dt-bindings/clock/allwinner,sun60i-a733-r-ccu.h
> create mode 100644 include/dt-bindings/reset/allwinner,sun60i-a733-ccu.h
> create mode 100644 include/dt-bindings/reset/allwinner,sun60i-a733-r-ccu.h
>
^ permalink raw reply [flat|nested] 31+ messages in thread
end of thread, other threads:[~2026-11-20 0:31 UTC | newest]
Thread overview: 31+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-13 2:32 [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
2026-11-13 2:32 ` [PATCH 01/26] power: regulator: axp: Fix AXP318W ELDO voltage mask Alastair D'Silva
2026-11-20 0:06 ` Andre Przywara
2026-11-13 2:32 ` [PATCH 02/26] power: pmic: axp: Fix AXP318W sysreset shutdown register Alastair D'Silva
2026-11-13 2:32 ` [PATCH 03/26] power: axp: Extend AXP318W SPL support with DCDC6/7 and LDOs Alastair D'Silva
2026-11-13 2:32 ` [PATCH 04/26] dt-bindings: clk: sun60i-a733-ccu: Add Allwinner A733 support Alastair D'Silva
2026-11-13 2:32 ` [PATCH 05/26] mmc: sunxi: Scale data transfer timeout with transfer size Alastair D'Silva
2026-11-13 2:32 ` [PATCH 06/26] gpio: a733: add initial support Alastair D'Silva
2026-11-13 2:32 ` [PATCH 07/26] pinctrl: sunxi: " Alastair D'Silva
2026-11-13 2:32 ` [PATCH 08/26] clock: reset: " Alastair D'Silva
2026-11-13 2:32 ` [PATCH 09/26] clock: a733: implement support for mmc/uart Alastair D'Silva
2026-11-13 2:32 ` [PATCH 10/26] sunxi: mmc: enable support for A733 Alastair D'Silva
2026-11-13 2:32 ` [PATCH 11/26] sunxi: a733: uart: setup serial console Alastair D'Silva
2026-11-13 2:32 ` [PATCH 12/26] arm64: spl: Support disposable .init.rodata section after _image_binary_end Alastair D'Silva
2026-11-13 2:32 ` [PATCH 13/26] sunxi: fel: Add Allwinner A733 ERET FEL return support Alastair D'Silva
2026-11-19 23:53 ` Andre Przywara
2026-11-13 2:32 ` [PATCH 14/26] sunxi: dram: a733: Add ARC PMU register, ABI, and build definitions Alastair D'Silva
2026-11-13 2:32 ` [PATCH 15/26] sunxi: dram: a733: Add ARC PMU primitives, clock, telemetry, and CBT Alastair D'Silva
2026-11-13 2:32 ` [PATCH 16/26] sunxi: dram: a733: Add ARC PMU deskew and BIST training firmware Alastair D'Silva
2026-11-13 2:32 ` [PATCH 17/26] sunxi: dram: a733: Add ARC PMU 1D/2D eye " Alastair D'Silva
2026-11-13 2:32 ` [PATCH 18/26] sunxi: dram: a733: Add ARC PMU training state machine Alastair D'Silva
2026-11-13 2:32 ` [PATCH 19/26] sunxi: dram: a733: Add Allwinner A733 LPDDR5 DRAM driver Alastair D'Silva
2026-11-13 2:32 ` [PATCH 20/26] sunxi: sun60i: initial support for A733 SoC Alastair D'Silva
2026-11-13 2:32 ` [PATCH 21/26] doc: sunxi: Document Allwinner A733 LPDDR5 PMU architecture Alastair D'Silva
2026-11-13 2:32 ` [PATCH 22/26] phy: sunxi: Add Allwinner A733 Cadence Combo PHY driver Alastair D'Silva
2026-11-13 2:32 ` [PATCH 23/26] pci: sunxi: Add Allwinner A733 DesignWare PCIe host controller driver Alastair D'Silva
2026-11-13 2:32 ` [PATCH 24/26] sunxi: Add NVMe to distro boot targets Alastair D'Silva
2026-11-13 2:32 ` [PATCH 25/26] arm64: dts: allwinner: add A733 SoC description Alastair D'Silva
2026-11-13 2:32 ` [PATCH 26/26] sunxi: Add Xunlong Orange Pi 4 Pro board support Alastair D'Silva
2026-11-13 7:28 ` [PATCH 00/26] sunxi: Add Allwinner A733 (sun60i) and Orange Pi 4 Pro support Alastair D'Silva
2026-11-20 0:31 ` Andre Przywara
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.