LinuxPPC-Dev Archive on lore.kernel.org
 help / color / mirror / Atom feed
* [Help] Microwatt (Zynqwatt) — Kernel halts after Radix MMU init on booting Linux on Zynq version of Microwatt
@ 2025-11-12 14:45 Mohammad Amin Nili
  2025-11-13  0:29 ` Oliver O'Halloran
  0 siblings, 1 reply; 7+ messages in thread
From: Mohammad Amin Nili @ 2025-11-12 14:45 UTC (permalink / raw)
  To: linuxppc-dev

Dear PPC64 Kernel Developers,

Hi all,

I'm trying to redo the Microwatt project on my own ZCU104 evaluation board
(Zynq US+). The main goal is to expose PS-side peripherals (specifically
the UART0 and LPDDR4 memory) to the PL-side Microwatt via the AXI4 bus and
MMIO. For simplicity, I'll call this whole project 'Zynqwatt'.

I successfully managed to boot and execute both `Hello World` and
`MicroPython` on my Zynqwatt. Here's what I've done so far:

1. I removed almost every peripheral except the `XICS` from the `SoC`
   module.
2. I routed all addresses to a `WB2AXI` bridge so that the Zynq's PS-side
   memory/MMIO space is completely accessible by the PL-side `Microwatt`.
3. I booted up my PS (Arm Cortex-A53) and let the Arm's `FSBL` initialize
   almost all of the PS peripherals (e.g. UART, LPDDR4, etc.).
4. I copied my program from the SD card (`Hello World` or `MicroPython`)
   into a fixed location of the PS DRAM using the Arm.
5. I released the `Microwatt` reset signal.
6. I parked the Arm processor.
7. The system worked successfully.

However, booting Linux seems to be a whole different story! Here's what
I've done so far to boot Linux on my Zynqwatt platform:

1. I’ve written a simple **bootloader** that runs on the Zynq’s Arm core.
   Its job is to read the kernel image — `dtbImage.microwatt.elf` — from
   the SD card and relocate it to the proper PS-side memory according to
   the ELF header information.

2. The bootloader **releases the Microwatt reset signal**, and Microwatt
   starts executing a super simple “hello world” bootloader. This minimal
   program just prints a `Welcome` message and jumps to the kernel start
   address (usually at `0x01700000`). Here is the log of this stage:

```
Zynq MP First Stage Boot Loader 
Release 2025.1   Nov  9 2025  -  19:22:42
PMU-FW is not running, certain applications may not be supported.
Downloading bootloader to the DRAM...
Successfully downloaded bootloader to the DRAM at 0x20000000!
Downloading Linux ELF file to the DRAM...
Starting ELF read from SD card...
Initializing SDPS driver...
SDPS driver and card initialized successfully.
Reading 7340032 bytes from sector offset 0 to address 0x30000000
ELF file read from SD card successfully.
Successfully downloaded ELF file to the DRAM at 0x30000000!
Extracting Linux ELF file to the DRAM...
Successfully extracted ELF file to the DRAM!
Configuring Microwatt for booting...
Successfully configured Microwatt!
Booting up Microwatt from bootloader at 0x20000000...
--------------------------------------------------


   .oOOo.     
 ."      ". 
 ;  .mw.  ;   Microwatt, it works.
  . '  ' .    
   \ || /    
    ;..;      
    ;..;      
    `ww'      


Function <my_printf> is located at 0x00001354.
Executing: *(0x01700000) --> 0x480000d0.
Press any key to continue...

```

3. The **kernel wrapper** tries to read the DTB and decompress the gzipped
   kernel into `0x00000000`:

```
zImage starting: loaded at 0x0000000001700000 (sp: 0x0000000001b1beb0)
Allocating 0x16c5144 bytes for kernel...
Decompressing (0x0000000000000000 <- 0x0000000001714000:0x0000000001b19fa2)...
Done! Decompressed 0x16c5144 bytes

Linux/PowerPC load: earlycon earlyprintk debug loglevel=9
Finalizing device tree... flat tree at 0x1b1cc80
```

4. The kernel successfully decompresses and hands control over to
   `arch/powerpc/kernel/head_64.S`. This is exactly where I’m stuck. Here’s
   the relevant log output:

```
Linux/PowerPC load: earlycon earlyprintk debug loglevel=9
Finalizing device tree... flat tree at 0x1b1cc80
[    0.000000] printk: legacy bootconsole [udbg0] enabled
[    0.000000] ioremap() called early from of_setup_earlycon+0xd0/0x2d4. Use early_ioremap() instead
[    0.000000] earlycon: earlycon_map: Couldn't map 0x00000000ff000000
[    0.000000] earlycon: cdns0 at MMIO 0x00000000ff000000 (options '')
[    0.000000] printk: legacy bootconsole [cdns0] enabled
[    0.000000] dt-cpu-ftrs: setup for ISA 3100
[    0.000000] dt-cpu-ftrs: final cpu/mmu features = 0x00040083800bb181 0x20005040
[    0.000000] radix-mmu: Page sizes from device-tree:
[    0.000000] radix-mmu: Page size shift = 12 AP=0x0
[    0.000000] radix-mmu: Page size shift = 16 AP=0x5
[    0.000000] radix-mmu: Page size shift = 21 AP=0x1
[    0.000000] radix-mmu: Page size shift = 30 AP=0x2
[    0.000000] radix-mmu: Mapped 0x0000000000000000-0x0000000001800000 with 2.00 MiB pages (exec)
[    0.000000] radix-mmu: Mapped 0x0000000001800000-0x0000000010000000 with 2.00 MiB pages
[    0.000000] radix-mmu: Initializing Radix MMU
```

NOTE: As you know, the `earlycon` driver depends on calling `ioremap`
in `drivers/tty/serial/earlycon.c:L47` to get the virtual address of its
`mapbase/membase`, which is later used by the main console driver (e.g.
`cdns,uart-r1p12`). During the very early boot phase, `ioremap` returns
`0x00000000` because it is called before `early_init_mmu` and
`early_ioremap_setup` in the `early_setup` of
`arch/powerpc/kernel/setup_64.S`.

For debugging purposes, I hardcoded it to return the **physical address**
`0xFF000000` — which corresponds to the Zynq PS-side UART0.

5. From what I can tell, **the system crashes or loses console output
immediately after the MMU is activated** — specifically right after the
call to `RFI_TO_KERNEL` in `arch/powerpc/kernel/head_64.S:L1019`.

6. To test my assumption, I tried **disabling the MMU** by inserting the
following lines around `head_64.S:L1019`:

```
	LOAD_REG_ADDR(r3, start_here_common)
	ld	r4,PACAKMSR(r13)
//*****************************************************
	li	r5, -49
	and	r4, r4, r5
//*****************************************************
	mtspr	SPRN_SRR0,r3
	mtspr	SPRN_SRR1,r4
	RFI_TO_KERNEL
	b	.	/* prevent speculative execution */
```

The result was:

```
[    0.000000] printk: legacy bootconsole [udbg0] enabled
[    0.000000] ioremap() called early from earlycon_map+0x20/0x6c. Use early_ioremap() instead
[    0.000000] earlycon: earlycon_map: Couldn't map 0x00000000ff000000
[    0.000000] earlycon: Temporarily mapped it to 0x00000000ff000000
[    0.000000] earlycon: cdns0 at MMIO 0x00000000ff000000 (options '')
[    0.000000] printk: legacy bootconsole [cdns0] enabled
[    0.000000] dt-cpu-ftrs: setup for ISA 3100
[    0.000000] dt-cpu-ftrs: final cpu/mmu features = 0x00040083800bb181 0x20005040
[    0.000000] radix-mmu: Page sizes from device-tree:
[    0.000000] radix-mmu: Page size shift = 12 AP=0x0
[    0.000000] radix-mmu: Page size shift = 16 AP=0x5
[    0.000000] radix-mmu: Page size shift = 21 AP=0x1
[    0.000000] radix-mmu: Page size shift = 30 AP=0x2
[    0.000000] radix-mmu: Mapped 0x0000000000000000-0x0000000001800000 with 2.00 MiB pages (exec)
[    0.000000] radix-mmu: Mapped 0x0000000001800000-0x0000000010000000 with 2.00 MiB pages
[    0.000000] radix-mmu: Initializing Radix MMU
[    0.000000] Linux version 6.18.0-rc3-00007-gfd57572253bc-dirty (manili@manili) (powerpc64le-linux-gcc.br_real (Buildroot 2021.11-18033-g83947c7bb6) 14.3.0, GNU ld (GNU Binutils) 2.43.1) #124 Sun Nov  9 23:46:11 EST 2025
[    0.000000] Hardware name: microwatt Microwatt 0x630000 microwatt
[    0.000000] -----------------------------------------------------
[    0.000000] phys_mem_size     = 0x10000000
[    0.000000] dcache_bsize      = 0x40
[    0.000000] icache_bsize      = 0x40
[    0.000000] cpu_features      = 0x00040083800bb181
[    0.000000]   possible        = 0x003ffbebcb5fb185
[    0.000000]   always          = 0x0000000380008181
[    0.000000] cpu_user_features = 0xcc002102 0x8c940000
[    0.000000] mmu_features      = 0x20005040
[    0.000000] firmware_features = 0x0000000000000000
[    0.000000] vmalloc start     = 0xc008000000000000
[    0.000000] IO start          = 0xc00a000000000000
[    0.000000] vmemmap start     = 0xc00c000000000000
[    0.000000] -----------------------------------------------------
[    0.000000] barrier-nospec: using ORI speculation barrier
[    0.000000] barrier-nospec: patched 100 locations
[    0.000000] Top of RAM: 0x10000000, Total RAM: 0x10000000
[    0.000000] Memory hole size: 0MB
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x000000000fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x000000000fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000000fffffff]
```

So unlike the previous logs, it looks like the kernel continues the boot
process after `radix-mmu: Initializing Radix MMU`, but this time halts at
`Initmem setup...` because the MMU is off (I guess).

7. Here’s my current DTS configuration. I’ve kept it as **simple and close
to Microwatt’s default** as possible:

```
/dts-v1/;
#include <dt-bindings/gpio/gpio.h>

/ {
	#size-cells = <0x02>;
	#address-cells = <0x02>;
	model = "microwatt";
	compatible = "microwatt-soc";

	aliases {
		serial0 = &UART0;
	};

	reserved-memory {
		#size-cells = <0x02>;
		#address-cells = <0x02>;
		ranges;
	};

	memory@0 {
		device_type = "memory";
		reg = <0x00000000 0x00000000 0x00000000 0x10000000>;
	};

	clocks {
		sys_clk: litex_sys_clk {
			#clock-cells = <0>;
			compatible = "fixed-clock";
			clock-frequency = <100000000>;
		};
	};

	cpus {
		#size-cells = <0x00>;
		#address-cells = <0x01>;

		ibm,powerpc-cpu-features {
			display-name = "Microwatt";
			isa = <3100>;
			device_type = "cpu-features";
			compatible = "ibm,powerpc-cpu-features";

			mmu-radix {
				isa = <3000>;
				usable-privilege = <6>;
				os-support = <0>;
			};

			little-endian {
				isa = <0>;
				usable-privilege = <7>;
				os-support = <0>;
				hwcap-bit-nr = <1>;
			};

			cache-inhibited-large-page {
				isa = <0>;
				usable-privilege = <6>;
				os-support = <0>;
			};

			fixed-point-v3 {
				isa = <3000>;
				usable-privilege = <7>;
			};

			no-execute {
				isa = <0x00>;
				usable-privilege = <2>;
				os-support = <0>;
			};

			floating-point {
				hfscr-bit-nr = <0>;
				hwcap-bit-nr = <27>;
				isa = <0>;
				usable-privilege = <7>;
				hv-support = <1>;
				os-support = <0>;
			};

			prefixed-instructions {
				hfscr-bit-nr = <13>;
				fscr-bit-nr = <13>;
				isa = <3010>;
				usable-privilege = <7>;
				os-support = <1>;
				hv-support = <1>;
			};

			tar {
				hfscr-bit-nr = <8>;
				fscr-bit-nr = <8>;
				isa = <2070>;
				usable-privilege = <7>;
				os-support = <1>;
				hv-support = <1>;
				hwcap-bit-nr = <58>;
			};

			control-register {
				isa = <0>;
				usable-privilege = <7>;
			};

			system-call-vectored {
				isa = <3000>;
				usable-privilege = <7>;
				os-support = <1>;
				fscr-bit-nr = <12>;
				hwcap-bit-nr = <52>;
			};
		};

		PowerPC,Microwatt@0 {
			i-cache-sets = <2>;
			ibm,dec-bits = <64>;
			reservation-granule-size = <64>;
			clock-frequency = <100000000>;
			timebase-frequency = <100000000>;
			i-tlb-sets = <1>;
			ibm,ppc-interrupt-server#s = <0>;
			i-cache-block-size = <64>;
			d-cache-block-size = <64>;
			d-cache-sets = <2>;
			i-tlb-size = <64>;
			cpu-version = <0x990000>;
			status = "okay";
			i-cache-size = <0x1000>;
			ibm,processor-radix-AP-encodings = <0x0c 0xa0000010 0x20000015 0x4000001e>;
			tlb-size = <0>;
			tlb-sets = <0>;
			device_type = "cpu";
			d-tlb-size = <128>;
			d-tlb-sets = <2>;
			reg = <0>;
			general-purpose;
			64-bit;
			d-cache-size = <0x1000>;
			ibm,chip-id = <0>;
			ibm,mmu-lpid-bits = <12>;
			ibm,mmu-pid-bits = <20>;
		};
	};

	soc {
		compatible = "simple-bus";
		#address-cells = <0x02>;
		#size-cells = <0x02>;
		ranges;

		interrupt-controller@c0004000 {
			compatible = "openpower,xics-presentation", "ibm,ppc-xicp";
			ibm,interrupt-server-ranges = <0x0 0x1>;
			reg = <0x0 0xc0004000 0x0 0x10>;
		};

		ICS: interrupt-controller@c0005000 {
			compatible = "openpower,xics-sources";
			interrupt-controller;
			interrupt-ranges = <0x10 0x10>;
			reg = <0x0 0xc0005000 0x0 0x100>;
			#address-cells = <0>;
			#size-cells = <0>;
			#interrupt-cells = <2>;
		};

		UART0: serial@ff000000 {
			device_type = "serial";
			compatible = "cdns,uart-r1p12";
			reg = <0x0 0xff000000 0x0 0x1000>;
			clock-frequency = <100000000>;
			current-speed = <115200>;
		};
	};

	chosen {
		bootargs = "earlycon earlyprintk debug loglevel=9";
		ibm,architecture-vec-5 = [19 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00
					  00 00 00 00 00 00 00 00 40 00 40];
		stdout-path = &UART0;
	};
};
```

So, why is the MMU preventing the kernel from booting properly — or at
least stopping the UART driver from accessing its mapped address at
`0xFF000000`? Could this be an issue specific to the **Cadence/Xilinx UART
driver**? Or am I missing something fundamental about how the MMU should
be configured in this setup?

Let me know if you need any other sources or data to share. I’ve tried
everything I can think of but still cannot figure out what’s wrong with my
kernel porting process. Any help would be much appreciated.

Bests,  
Manili

^ permalink raw reply	[flat|nested] 7+ messages in thread
end of thread, other threads:[~2025-11-22 11:33 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-11-12 14:45 [Help] Microwatt (Zynqwatt) — Kernel halts after Radix MMU init on booting Linux on Zynq version of Microwatt Mohammad Amin Nili
2025-11-13  0:29 ` Oliver O'Halloran
2025-11-13 22:32   ` Mohammad Amin Nili
2025-11-14  2:42     ` Oliver O'Halloran
2025-11-21 14:59       ` Mohammad Amin Nili
2025-11-22  2:48         ` Oliver O'Halloran
2025-11-22 11:32           ` Mohammad Amin Nili

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox