[Help] Microwatt (Zynqwatt) — Kernel halts after Radix MMU init on booting Linux on Zynq version of Microwatt

[Help] Microwatt (Zynqwatt) — Kernel halts after Radix MMU init on booting Linux on Zynq version of Microwatt

[Mohammad Amin Nili]

Dear PPC64 Kernel Developers,

Hi all,

I'm trying to redo the Microwatt project on my own ZCU104 evaluation board
(Zynq US+). The main goal is to expose PS-side peripherals (specifically
the UART0 and LPDDR4 memory) to the PL-side Microwatt via the AXI4 bus and
MMIO. For simplicity, I'll call this whole project 'Zynqwatt'.

I successfully managed to boot and execute both `Hello World` and
`MicroPython` on my Zynqwatt. Here's what I've done so far:

1. I removed almost every peripheral except the `XICS` from the `SoC`
   module.
2. I routed all addresses to a `WB2AXI` bridge so that the Zynq's PS-side
   memory/MMIO space is completely accessible by the PL-side `Microwatt`.
3. I booted up my PS (Arm Cortex-A53) and let the Arm's `FSBL` initialize
   almost all of the PS peripherals (e.g. UART, LPDDR4, etc.).
4. I copied my program from the SD card (`Hello World` or `MicroPython`)
   into a fixed location of the PS DRAM using the Arm.
5. I released the `Microwatt` reset signal.
6. I parked the Arm processor.
7. The system worked successfully.

However, booting Linux seems to be a whole different story! Here's what
I've done so far to boot Linux on my Zynqwatt platform:

1. I’ve written a simple **bootloader** that runs on the Zynq’s Arm core.
   Its job is to read the kernel image — `dtbImage.microwatt.elf` — from
   the SD card and relocate it to the proper PS-side memory according to
   the ELF header information.

2. The bootloader **releases the Microwatt reset signal**, and Microwatt
   starts executing a super simple “hello world” bootloader. This minimal
   program just prints a `Welcome` message and jumps to the kernel start
   address (usually at `0x01700000`). Here is the log of this stage:

```
Zynq MP First Stage Boot Loader 
Release 2025.1   Nov  9 2025  -  19:22:42
PMU-FW is not running, certain applications may not be supported.
Downloading bootloader to the DRAM...
Successfully downloaded bootloader to the DRAM at 0x20000000!
Downloading Linux ELF file to the DRAM...
Starting ELF read from SD card...
Initializing SDPS driver...
SDPS driver and card initialized successfully.
Reading 7340032 bytes from sector offset 0 to address 0x30000000
ELF file read from SD card successfully.
Successfully downloaded ELF file to the DRAM at 0x30000000!
Extracting Linux ELF file to the DRAM...
Successfully extracted ELF file to the DRAM!
Configuring Microwatt for booting...
Successfully configured Microwatt!
Booting up Microwatt from bootloader at 0x20000000...
--------------------------------------------------


   .oOOo.     
 ."      ". 
 ;  .mw.  ;   Microwatt, it works.
  . '  ' .    
   \ || /    
    ;..;      
    ;..;      
    `ww'      


Function <my_printf> is located at 0x00001354.
Executing: *(0x01700000) --> 0x480000d0.
Press any key to continue...

```

3. The **kernel wrapper** tries to read the DTB and decompress the gzipped
   kernel into `0x00000000`:

```
zImage starting: loaded at 0x0000000001700000 (sp: 0x0000000001b1beb0)
Allocating 0x16c5144 bytes for kernel...
Decompressing (0x0000000000000000 <- 0x0000000001714000:0x0000000001b19fa2)...
Done! Decompressed 0x16c5144 bytes

Linux/PowerPC load: earlycon earlyprintk debug loglevel=9
Finalizing device tree... flat tree at 0x1b1cc80
```

4. The kernel successfully decompresses and hands control over to
   `arch/powerpc/kernel/head_64.S`. This is exactly where I’m stuck. Here’s
   the relevant log output:

```
Linux/PowerPC load: earlycon earlyprintk debug loglevel=9
Finalizing device tree... flat tree at 0x1b1cc80
[    0.000000] printk: legacy bootconsole [udbg0] enabled
[    0.000000] ioremap() called early from of_setup_earlycon+0xd0/0x2d4. Use early_ioremap() instead
[    0.000000] earlycon: earlycon_map: Couldn't map 0x00000000ff000000
[    0.000000] earlycon: cdns0 at MMIO 0x00000000ff000000 (options '')
[    0.000000] printk: legacy bootconsole [cdns0] enabled
[    0.000000] dt-cpu-ftrs: setup for ISA 3100
[    0.000000] dt-cpu-ftrs: final cpu/mmu features = 0x00040083800bb181 0x20005040
[    0.000000] radix-mmu: Page sizes from device-tree:
[    0.000000] radix-mmu: Page size shift = 12 AP=0x0
[    0.000000] radix-mmu: Page size shift = 16 AP=0x5
[    0.000000] radix-mmu: Page size shift = 21 AP=0x1
[    0.000000] radix-mmu: Page size shift = 30 AP=0x2
[    0.000000] radix-mmu: Mapped 0x0000000000000000-0x0000000001800000 with 2.00 MiB pages (exec)
[    0.000000] radix-mmu: Mapped 0x0000000001800000-0x0000000010000000 with 2.00 MiB pages
[    0.000000] radix-mmu: Initializing Radix MMU
```

NOTE: As you know, the `earlycon` driver depends on calling `ioremap`
in `drivers/tty/serial/earlycon.c:L47` to get the virtual address of its
`mapbase/membase`, which is later used by the main console driver (e.g.
`cdns,uart-r1p12`). During the very early boot phase, `ioremap` returns
`0x00000000` because it is called before `early_init_mmu` and
`early_ioremap_setup` in the `early_setup` of
`arch/powerpc/kernel/setup_64.S`.

For debugging purposes, I hardcoded it to return the **physical address**
`0xFF000000` — which corresponds to the Zynq PS-side UART0.

5. From what I can tell, **the system crashes or loses console output
immediately after the MMU is activated** — specifically right after the
call to `RFI_TO_KERNEL` in `arch/powerpc/kernel/head_64.S:L1019`.

6. To test my assumption, I tried **disabling the MMU** by inserting the
following lines around `head_64.S:L1019`:

```
	LOAD_REG_ADDR(r3, start_here_common)
	ld	r4,PACAKMSR(r13)
//*****************************************************
	li	r5, -49
	and	r4, r4, r5
//*****************************************************
	mtspr	SPRN_SRR0,r3
	mtspr	SPRN_SRR1,r4
	RFI_TO_KERNEL
	b	.	/* prevent speculative execution */
```

The result was:

```
[    0.000000] printk: legacy bootconsole [udbg0] enabled
[    0.000000] ioremap() called early from earlycon_map+0x20/0x6c. Use early_ioremap() instead
[    0.000000] earlycon: earlycon_map: Couldn't map 0x00000000ff000000
[    0.000000] earlycon: Temporarily mapped it to 0x00000000ff000000
[    0.000000] earlycon: cdns0 at MMIO 0x00000000ff000000 (options '')
[    0.000000] printk: legacy bootconsole [cdns0] enabled
[    0.000000] dt-cpu-ftrs: setup for ISA 3100
[    0.000000] dt-cpu-ftrs: final cpu/mmu features = 0x00040083800bb181 0x20005040
[    0.000000] radix-mmu: Page sizes from device-tree:
[    0.000000] radix-mmu: Page size shift = 12 AP=0x0
[    0.000000] radix-mmu: Page size shift = 16 AP=0x5
[    0.000000] radix-mmu: Page size shift = 21 AP=0x1
[    0.000000] radix-mmu: Page size shift = 30 AP=0x2
[    0.000000] radix-mmu: Mapped 0x0000000000000000-0x0000000001800000 with 2.00 MiB pages (exec)
[    0.000000] radix-mmu: Mapped 0x0000000001800000-0x0000000010000000 with 2.00 MiB pages
[    0.000000] radix-mmu: Initializing Radix MMU
[    0.000000] Linux version 6.18.0-rc3-00007-gfd57572253bc-dirty (manili@manili) (powerpc64le-linux-gcc.br_real (Buildroot 2021.11-18033-g83947c7bb6) 14.3.0, GNU ld (GNU Binutils) 2.43.1) #124 Sun Nov  9 23:46:11 EST 2025
[    0.000000] Hardware name: microwatt Microwatt 0x630000 microwatt
[    0.000000] -----------------------------------------------------
[    0.000000] phys_mem_size     = 0x10000000
[    0.000000] dcache_bsize      = 0x40
[    0.000000] icache_bsize      = 0x40
[    0.000000] cpu_features      = 0x00040083800bb181
[    0.000000]   possible        = 0x003ffbebcb5fb185
[    0.000000]   always          = 0x0000000380008181
[    0.000000] cpu_user_features = 0xcc002102 0x8c940000
[    0.000000] mmu_features      = 0x20005040
[    0.000000] firmware_features = 0x0000000000000000
[    0.000000] vmalloc start     = 0xc008000000000000
[    0.000000] IO start          = 0xc00a000000000000
[    0.000000] vmemmap start     = 0xc00c000000000000
[    0.000000] -----------------------------------------------------
[    0.000000] barrier-nospec: using ORI speculation barrier
[    0.000000] barrier-nospec: patched 100 locations
[    0.000000] Top of RAM: 0x10000000, Total RAM: 0x10000000
[    0.000000] Memory hole size: 0MB
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x000000000fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x000000000fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000000fffffff]
```

So unlike the previous logs, it looks like the kernel continues the boot
process after `radix-mmu: Initializing Radix MMU`, but this time halts at
`Initmem setup...` because the MMU is off (I guess).

7. Here’s my current DTS configuration. I’ve kept it as **simple and close
to Microwatt’s default** as possible:

```
/dts-v1/;
#include <dt-bindings/gpio/gpio.h>

/ {
	#size-cells = <0x02>;
	#address-cells = <0x02>;
	model = "microwatt";
	compatible = "microwatt-soc";

	aliases {
		serial0 = &UART0;
	};

	reserved-memory {
		#size-cells = <0x02>;
		#address-cells = <0x02>;
		ranges;
	};

	memory@0 {
		device_type = "memory";
		reg = <0x00000000 0x00000000 0x00000000 0x10000000>;
	};

	clocks {
		sys_clk: litex_sys_clk {
			#clock-cells = <0>;
			compatible = "fixed-clock";
			clock-frequency = <100000000>;
		};
	};

	cpus {
		#size-cells = <0x00>;
		#address-cells = <0x01>;

		ibm,powerpc-cpu-features {
			display-name = "Microwatt";
			isa = <3100>;
			device_type = "cpu-features";
			compatible = "ibm,powerpc-cpu-features";

			mmu-radix {
				isa = <3000>;
				usable-privilege = <6>;
				os-support = <0>;
			};

			little-endian {
				isa = <0>;
				usable-privilege = <7>;
				os-support = <0>;
				hwcap-bit-nr = <1>;
			};

			cache-inhibited-large-page {
				isa = <0>;
				usable-privilege = <6>;
				os-support = <0>;
			};

			fixed-point-v3 {
				isa = <3000>;
				usable-privilege = <7>;
			};

			no-execute {
				isa = <0x00>;
				usable-privilege = <2>;
				os-support = <0>;
			};

			floating-point {
				hfscr-bit-nr = <0>;
				hwcap-bit-nr = <27>;
				isa = <0>;
				usable-privilege = <7>;
				hv-support = <1>;
				os-support = <0>;
			};

			prefixed-instructions {
				hfscr-bit-nr = <13>;
				fscr-bit-nr = <13>;
				isa = <3010>;
				usable-privilege = <7>;
				os-support = <1>;
				hv-support = <1>;
			};

			tar {
				hfscr-bit-nr = <8>;
				fscr-bit-nr = <8>;
				isa = <2070>;
				usable-privilege = <7>;
				os-support = <1>;
				hv-support = <1>;
				hwcap-bit-nr = <58>;
			};

			control-register {
				isa = <0>;
				usable-privilege = <7>;
			};

			system-call-vectored {
				isa = <3000>;
				usable-privilege = <7>;
				os-support = <1>;
				fscr-bit-nr = <12>;
				hwcap-bit-nr = <52>;
			};
		};

		PowerPC,Microwatt@0 {
			i-cache-sets = <2>;
			ibm,dec-bits = <64>;
			reservation-granule-size = <64>;
			clock-frequency = <100000000>;
			timebase-frequency = <100000000>;
			i-tlb-sets = <1>;
			ibm,ppc-interrupt-server#s = <0>;
			i-cache-block-size = <64>;
			d-cache-block-size = <64>;
			d-cache-sets = <2>;
			i-tlb-size = <64>;
			cpu-version = <0x990000>;
			status = "okay";
			i-cache-size = <0x1000>;
			ibm,processor-radix-AP-encodings = <0x0c 0xa0000010 0x20000015 0x4000001e>;
			tlb-size = <0>;
			tlb-sets = <0>;
			device_type = "cpu";
			d-tlb-size = <128>;
			d-tlb-sets = <2>;
			reg = <0>;
			general-purpose;
			64-bit;
			d-cache-size = <0x1000>;
			ibm,chip-id = <0>;
			ibm,mmu-lpid-bits = <12>;
			ibm,mmu-pid-bits = <20>;
		};
	};

	soc {
		compatible = "simple-bus";
		#address-cells = <0x02>;
		#size-cells = <0x02>;
		ranges;

		interrupt-controller@c0004000 {
			compatible = "openpower,xics-presentation", "ibm,ppc-xicp";
			ibm,interrupt-server-ranges = <0x0 0x1>;
			reg = <0x0 0xc0004000 0x0 0x10>;
		};

		ICS: interrupt-controller@c0005000 {
			compatible = "openpower,xics-sources";
			interrupt-controller;
			interrupt-ranges = <0x10 0x10>;
			reg = <0x0 0xc0005000 0x0 0x100>;
			#address-cells = <0>;
			#size-cells = <0>;
			#interrupt-cells = <2>;
		};

		UART0: serial@ff000000 {
			device_type = "serial";
			compatible = "cdns,uart-r1p12";
			reg = <0x0 0xff000000 0x0 0x1000>;
			clock-frequency = <100000000>;
			current-speed = <115200>;
		};
	};

	chosen {
		bootargs = "earlycon earlyprintk debug loglevel=9";
		ibm,architecture-vec-5 = [19 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00
					  00 00 00 00 00 00 00 00 40 00 40];
		stdout-path = &UART0;
	};
};
```

So, why is the MMU preventing the kernel from booting properly — or at
least stopping the UART driver from accessing its mapped address at
`0xFF000000`? Could this be an issue specific to the **Cadence/Xilinx UART
driver**? Or am I missing something fundamental about how the MMU should
be configured in this setup?

Let me know if you need any other sources or data to share. I’ve tried
everything I can think of but still cannot figure out what’s wrong with my
kernel porting process. Any help would be much appreciated.

Bests,  
Manili

Re: [Help] Microwatt (Zynqwatt) — Kernel halts after Radix MMU init on booting Linux on Zynq version of Microwatt

[Oliver O'Halloran]

On Thu, Nov 13, 2025 at 1:45 AM Mohammad Amin Nili
<manili.devteam@gmail.com> wrote:
>
> [    0.000000] printk: legacy bootconsole [udbg0] enabled
> [    0.000000] ioremap() called early from of_setup_earlycon+0xd0/0x2d4. Use early_ioremap() instead
> [    0.000000] earlycon: earlycon_map: Couldn't map 0x00000000ff000000

pretty big red flag right there


> NOTE: As you know

bold of you to assume i know things

> the `earlycon` driver depends on calling `ioremap`
> in `drivers/tty/serial/earlycon.c:L47` to get the virtual address of its
> `mapbase/membase`, which is later used by the main console driver (e.g.
> `cdns,uart-r1p12`). During the very early boot phase, `ioremap` returns
> `0x00000000` because it is called before `early_init_mmu` and
> `early_ioremap_setup` in the `early_setup` of
> `arch/powerpc/kernel/setup_64.S`.
>
> For debugging purposes, I hardcoded it to return the **physical address**
> `0xFF000000` — which corresponds to the Zynq PS-side UART0.
>
> 5. From what I can tell, **the system crashes or loses console output
> immediately after the MMU is activated** — specifically right after the
> call to `RFI_TO_KERNEL` in `arch/powerpc/kernel/head_64.S:L1019`.

Right, you turn on the MMU and the next time printk() is called the
console driver tries to write to 0xFF00_0000. That's not a valid
virtual address so it explodes. To make an address usable in both real
mode (i.e. pre-mmu) and virtual mode you need to have the page tables
setup so that virtual address maps to the same physical address.
Setting up that mapping is what early_ioremap() does. That's why
there's a warning telling you to use it.

Based on the in-tree DTS files earlycon doesn't seem to be used on any
powerpc systems. My guess would be that most ppc platform use udbg
(very old, powerpc specific thing) rather than earlycon for this kind
of super-early debug output. Considering you're getting console output
via udbg I'd say just removing earlycon from your kernel command line
will probably fix your issue.

Re: [Help] Microwatt (Zynqwatt) — Kernel halts after Radix MMU init on booting Linux on Zynq version of Microwatt

[Mohammad Amin Nili]

[-- Attachment #1: Type: text/plain, Size: 1721 bytes --]

Hi Oliver,
Thanks a lot for your time and the answer.

> Right, you turn on the MMU and the next time printk() is called the
> console driver tries to write to 0xFF00_0000. That's not a valid
> virtual address so it explodes. To make an address usable in both real
> mode (i.e. pre-mmu) and virtual mode you need to have the page tables
> setup so that virtual address maps to the same physical address.
> Setting up that mapping is what early_ioremap() does. That's why
> there's a warning telling you to use it.
> 
> Based on the in-tree DTS files earlycon doesn't seem to be used on any
> powerpc systems. My guess would be that most ppc platform use udbg
> (very old, powerpc specific thing) rather than earlycon for this kind
> of super-early debug output. Considering you're getting console output
> via udbg I'd say just removing earlycon from your kernel command line
> will probably fix your issue.

Well, I disabled all the early logs (now bootargs = “” in dts) and modified .config
so that no earlycon.c gets compiled at all. The followings are the only compiled
files within the `/derivers/tty/serial`:

serial_base_bus.o, serial_core.o, serial_ctrl.o, serial_port.o, xilinx_uartps.o

Now, I get no outputs during booting procedure which sounds normal, I guess.
But still no luck getting to the rootfs or anywhere which actually initializes the
`xilinx_uartps` driver and print something.

Also please check shenki’s blogpost’s logs, if you have enough time:

https://shenki.github.io/boot-linux-on-microwatt <https://shenki.github.io/boot-linux-on-microwatt>

Based on the blog post, I think there should be no problem to get the early
booting logs.

Bests,
Manili

[-- Attachment #2: Type: text/html, Size: 3514 bytes --]

Re: [Help] Microwatt (Zynqwatt) — Kernel halts after Radix MMU init on booting Linux on Zynq version of Microwatt

[Oliver O'Halloran]

On Fri, Nov 14, 2025 at 9:32 AM Mohammad Amin Nili
<manili.devteam@gmail.com> wrote:
>
> Well, I disabled all the early logs (now bootargs = “” in dts) and modified .config
> so that no earlycon.c gets compiled at all. The followings are the only compiled
> files within the `/derivers/tty/serial`:
>
> serial_base_bus.o, serial_core.o, serial_ctrl.o, serial_port.o, xilinx_uartps.o
>
> Now, I get no outputs during booting procedure which sounds normal, I guess.
> But still no luck getting to the rootfs or anywhere which actually initializes the
> `xilinx_uartps` driver and print something.

It might be reaching the rootfs. Initialising a console doesn't
necessarily mean anything is going to use it. Try adding console=ttyS0
to your command line.

> Also please check shenki’s blogpost’s logs, if you have enough time:
>
> https://shenki.github.io/boot-linux-on-microwatt
>
> Based on the blog post, I think there should be no problem to get the early
> booting logs.

And yet you're having problems...

The series that blog post is based on added a udbg driver for the
potato UART they were using for microwatt development. The udbg driver
itself provides early console output and gets turned into hvc0 by the
hvc_udbg driver later on during boot. In your case you don't have
either of those things so you don't get any output. If you wanted to
do something similar you would need to implement a udbg driver for
your xilinx UART. That shouldn't be too hard to do, but fixing the
earlycon driver so it uses early_ioremap() might also work.

Re: [Help] Microwatt (Zynqwatt) — Kernel halts after Radix MMU init on booting Linux on Zynq version of Microwatt

[Mohammad Amin Nili]

[-- Attachment #1: Type: text/plain, Size: 4255 bytes --]

Hi Oliver!

> If you wanted to do something similar you would need to
> implement a udbg driver for your xilinx UART.

Actually this was the key point which finally led me to the
right direction. Thank you SOOO much for your help.
I implemented a udbg driver for Xilinx’s UART, worked
through a number of bugs, and managed to get bash and
userspace to start. During the work I hit two things in the
source that look odd to me; I’d appreciate any pointers or
explanations:

1. There appear to be two ways udbg can be initiated:
- https://elixir.bootlin.com/linux/v6.18-rc5/source/arch/powerpc/kernel/setup_64.c#L378 <https://elixir.bootlin.com/linux/v6.18-rc5/source/arch/powerpc/kernel/setup_64.c#L378>
- https://elixir.bootlin.com/linux/v6.18-rc5/source/init/main.c#L932 <https://elixir.bootlin.com/linux/v6.18-rc5/source/init/main.c#L932>
  -> https://elixir.bootlin.com/linux/v6.18-rc5/source/arch/powerpc/kernel/setup-common.c#L944 <https://elixir.bootlin.com/linux/v6.18-rc5/source/arch/powerpc/kernel/setup-common.c#L944>
  -> https://elixir.bootlin.com/linux/v6.18-rc5/source/arch/powerpc/kernel/setup-common.c#L947 <https://elixir.bootlin.com/linux/v6.18-rc5/source/arch/powerpc/kernel/setup-common.c#L947>

The problem with the *first* (early) path is that a udbg driver
often calls `early_ioremap()` **before** any MMU/memory
setup has completed. The MMU setup for 64-bit happens here:
- https://elixir.bootlin.com/linux/v6.18-rc5/source/arch/powerpc/kernel/setup_64.c#L418 <https://elixir.bootlin.com/linux/v6.18-rc5/source/arch/powerpc/kernel/setup_64.c#L418>

The 32-bit setup (`setup_32.c`) calls `udbg_early_init` *after*
`early_ioremap_init`, so it doesn’t hit this ordering problem:
- https://elixir.bootlin.com/linux/v6.18-rc5/source/arch/powerpc/kernel/setup_32.c#L87 <https://elixir.bootlin.com/linux/v6.18-rc5/source/arch/powerpc/kernel/setup_32.c#L87>

In my case, attempting to initialize the Xilinx udbg driver via
the early path can (and did) cause kernel panics because
`early_ioremap()` runs before the memory/MMU is ready.
Practically, the only reliable place for my driver to initialize
is later in `init/main.c`. Is this the intended/expected behavior
for PPC64? Or am I missing something?

2. I had to inject a busy-wait loop between lines 125–126 in
`kernel/rcu/tiny.c` to prevent a crash when/after switching to
userspace:
- https://elixir.bootlin.com/linux/v6.18-rc5/source/kernel/rcu/tiny.c#L125 <https://elixir.bootlin.com/linux/v6.18-rc5/source/kernel/rcu/tiny.c#L125>

Here is the loop I added:
for (volatile uint32_t i = 0; i < 10; i++);

If I omit that trivial busy-wait, the kernel crashes while/after
switching to userspace with an error LIKE:

[   42.397074] kernel tried to execute exec-protected page (c00c000000000000) - exploit attempt? (uid: 0)
[   42.408148] BUG: Unable to handle kernel instruction fetch
[   42.414964] Faulting instruction address: 0xc00c000000000000
Vector: 400 (Instruction Access) at [c00000000207fae0]
    pc: c00c000000000000
    lr: c00000000008c798: rcu_process_callbacks+0xf8/0x100
    sp: c00000000207fd80
   msr: 900000001000b033
  current = 0xc000000002056300
  paca    = 0xc0000000016e8000	 irqmask: 0x03	 irq_happened: 0x09
    pid   = 10, comm = ksoftirqd/0
Linux version 6.18.0-rc5-00111-g6fa9041b7177-dirty (manili@manili) (powerpc64le-linux-gcc.br_real (Buildroot 2021.11-18033-g83947c7bb6) 14.3.0, GNU ld (GNU Binutils) 2.43.1) #3 Thu Nov 20 09:33:11 EST 2025
enter ? for help
[link register   ] c00000000008c798 rcu_process_callbacks+0xf8/0x100
[c00000000207fd80] c00000000008c748 rcu_process_callbacks+0xa8/0x100 (unreliable)
[c00000000207fe00] c00000000003f320 handle_softirqs+0x1ec/0x23c
[c00000000207ff00] c00000000003f3a8 run_ksoftirqd+0x38/0x58
[c00000000207ff20] c00000000005f9c4 smpboot_thread_fn+0x1a0/0x1a8
[c00000000207ff80] c00000000005b190 kthread+0x1c0/0x1cc
[c00000000207ffe0] c00000000000b160 start_kernel_thread+0x14/0x18
mon>

The exact addresses in the error vary, but the crash
template is the same. My suspicion is that this is a
core/thread synchronization issue. Do you have any
ideas on this issue and why a simple while loop is able
to solve it?

Bests,
Manili

[-- Attachment #2: Type: text/html, Size: 6003 bytes --]

Re: [Help] Microwatt (Zynqwatt) — Kernel halts after Radix MMU init on booting Linux on Zynq version of Microwatt

[Oliver O'Halloran]

On Sat, Nov 22, 2025 at 1:59 AM Mohammad Amin Nili
<manili.devteam@gmail.com> wrote:
>
> The problem with the *first* (early) path is that a udbg driver
> often calls `early_ioremap()` **before** any MMU/memory
> setup has completed. The MMU setup for 64-bit happens here:
> - https://elixir.bootlin.com/linux/v6.18-rc5/source/arch/powerpc/kernel/setup_64.c#L418
>
> The 32-bit setup (`setup_32.c`) calls `udbg_early_init` *after*
> `early_ioremap_init`, so it doesn’t hit this ordering problem:
> - https://elixir.bootlin.com/linux/v6.18-rc5/source/arch/powerpc/kernel/setup_32.c#L87
>
> In my case, attempting to initialize the Xilinx udbg driver via
> the early path can (and did) cause kernel panics because
> `early_ioremap()` runs before the memory/MMU is ready.
> Practically, the only reliable place for my driver to initialize
> is later in `init/main.c`. Is this the intended/expected behavior
> for PPC64? Or am I missing something?

Nah it's just broken for your case. Most ppc64 development happens on
IBM hardware and the udbg drivers for those platforms (pseries for
VMs, powernv for bare metal) don't require the MMU to be setup.
Enabling debug output pre-MMU is handy since it lets you debug MMU
setup issues, but obviously it's not going to work if your udbg driver
needs to map stuff. You could avoid depending on the MMU by using the
potato uart driver's trick of just disabling data relocation when
writing to the UART's registers. That's pretty slow and a bit janky
though. A better fix might be to split udbg init into an early
(pre-mmu) and late variants with the udbg drivers that need the mmu
are initialised later.

> 2. I had to inject a busy-wait loop between lines 125–126 in
> `kernel/rcu/tiny.c` to prevent a crash when/after switching to
> userspace:
> - https://elixir.bootlin.com/linux/v6.18-rc5/source/kernel/rcu/tiny.c#L125
>
> Here is the loop I added:
> for (volatile uint32_t i = 0; i < 10; i++);
>
> If I omit that trivial busy-wait, the kernel crashes while/after
> switching to userspace with an error LIKE:
>
> [   42.397074] kernel tried to execute exec-protected page (c00c000000000000) - exploit attempt? (uid: 0)
> [   42.408148] BUG: Unable to handle kernel instruction fetch
> [   42.414964] Faulting instruction address: 0xc00c000000000000
> Vector: 400 (Instruction Access) at [c00000000207fae0]
>     pc: c00c000000000000
>     lr: c00000000008c798: rcu_process_callbacks+0xf8/0x100
>     sp: c00000000207fd80
>    msr: 900000001000b033
>   current = 0xc000000002056300
>   paca    = 0xc0000000016e8000 irqmask: 0x03 irq_happened: 0x09
>     pid   = 10, comm = ksoftirqd/0
> Linux version 6.18.0-rc5-00111-g6fa9041b7177-dirty (manili@manili) (powerpc64le-linux-gcc.br_real (Buildroot 2021.11-18033-g83947c7bb6) 14.3.0, GNU ld (GNU Binutils) 2.43.1) #3 Thu Nov 20 09:33:11 EST 2025
> enter ? for help
> [link register   ] c00000000008c798 rcu_process_callbacks+0xf8/0x100
> [c00000000207fd80] c00000000008c748 rcu_process_callbacks+0xa8/0x100 (unreliable)
> [c00000000207fe00] c00000000003f320 handle_softirqs+0x1ec/0x23c
> [c00000000207ff00] c00000000003f3a8 run_ksoftirqd+0x38/0x58
> [c00000000207ff20] c00000000005f9c4 smpboot_thread_fn+0x1a0/0x1a8
> [c00000000207ff80] c00000000005b190 kthread+0x1c0/0x1cc
> [c00000000207ffe0] c00000000000b160 start_kernel_thread+0x14/0x18
> mon>
>
> The exact addresses in the error vary, but the crash
> template is the same. My suspicion is that this is a
> core/thread synchronization issue. Do you have any
> ideas on this issue and why a simple while loop is able
> to solve it?

That's very odd. rcu_reclaim_tiny() is probably being folded into
rcu_process_callbacks() by the compiler and the crash occurs when
branching to the callback function from the rcu_head
(https://elixir.bootlin.com/linux/v6.18-rc5/source/kernel/rcu/tiny.c#L95).
That said, the "callback" address it branched to (0xc00c000000000000)
is actually the base of the vmemmap (i.e. the struct page array) so I
doubt that's actually the callback address stored in the rcu_head. You
can use xmon to dump the registers and examine memory to confirm this.
It's hard to say why this is happening, but it's pretty likely to
either be the compiler optimizing away code you'd prefer to keep or a
bug in the core itself.

I'd compare the disasm of rcu_process_callbacks() with and without
your wait loop added and see how the emitted code changes. If adding
the loop changes nothing then it might be a logic bug in microwatt
itself or some other timing induced problem.

>
> Bests,
> Manili

Re: [Help] Microwatt (Zynqwatt) — Kernel halts after Radix MMU init on booting Linux on Zynq version of Microwatt

[Mohammad Amin Nili]

[-- Attachment #1: Type: text/plain, Size: 9041 bytes --]

Hello Oliver,

> That's very odd. rcu_reclaim_tiny() is probably being folded into
> rcu_process_callbacks() by the compiler and the crash occurs when
> branching to the callback function from the rcu_head
> (https://elixir.bootlin.com/linux/v6.18-rc5/source/kernel/rcu/tiny.c#L95 <https://elixir.bootlin.com/linux/v6.18-rc5/source/kernel/rcu/tiny.c#L95>).
> That said, the "callback" address it branched to (0xc00c000000000000)
> is actually the base of the vmemmap (i.e. the struct page array) so I
> doubt that's actually the callback address stored in the rcu_head. You
> can use xmon to dump the registers and examine memory to confirm this.
> It's hard to say why this is happening, but it's pretty likely to
> either be the compiler optimizing away code you'd prefer to keep or a
> bug in the core itself.
> 
> I'd compare the disasm of rcu_process_callbacks() with and without
> your wait loop added and see how the emitted code changes. If adding
> the loop changes nothing then it might be a logic bug in microwatt
> itself or some other timing induced problem.

I just replaced the wait-loop with the following (and it also prevents the
kernel panic):

> 	volatile uint32_t dumb = 0xFEDC0000;
>	dumb++;

- Here is the disasm of the `rcu_process_callbacks` with the injected code:

c00000000008c5e8 <rcu_process_callbacks>:
c00000000008c5e8:       32 00 4c 3c     addis   r2,r12,50
c00000000008c5ec:       18 ca 42 38     addi    r2,r2,-13800
c00000000008c5f0:       a6 02 08 7c     mflr    r0
c00000000008c5f4:       a1 c3 23 48     bl      c0000000002c8994 <_savegpr0_27>
c00000000008c5f8:       a1 ff 21 f8     stdu    r1,-96(r1)
c00000000008c5fc:       fa 00 6d 88     lbz     r3,250(r13)
c00000000008c600:       01 00 69 60     ori     r9,r3,1
c00000000008c604:       fa 00 2d 99     stb     r9,250(r13)
c00000000008c608:       2b 01 42 3d     addis   r10,r2,299
c00000000008c60c:       2b 01 22 3d     addis   r9,r2,299
c00000000008c610:       d0 3c 4a e9     ld      r10,15568(r10)
c00000000008c614:       c8 3c 29 39     addi    r9,r9,15560
c00000000008c618:       00 48 2a 7c     cmpd    r10,r9
c00000000008c61c:       14 00 82 40     bne     c00000000008c630 <rcu_process_callbacks+0x48>
c00000000008c620:       6d d6 f8 4b     bl      c000000000019c8c <arch_local_irq_restore+0x8>
c00000000008c624:       00 00 00 60     nop
c00000000008c628:       60 00 21 38     addi    r1,r1,96
c00000000008c62c:       b8 c3 23 48     b       c0000000002c89e4 <_restgpr0_27>
c00000000008c630:       00 00 0a e9     ld      r8,0(r10)
c00000000008c634:       2b 01 e2 3c     addis   r7,r2,299
c00000000008c638:       c8 3c e7 eb     ld      r31,15560(r7)
c00000000008c63c:       c8 3c 07 f9     std     r8,15560(r7)
c00000000008c640:       00 00 00 39     li      r8,0
c00000000008c644:       00 00 0a f9     std     r8,0(r10)
c00000000008c648:       2b 01 42 3d     addis   r10,r2,299
c00000000008c64c:       d8 3c 0a e9     ld      r8,15576(r10)
c00000000008c650:       2b 01 42 3d     addis   r10,r2,299
c00000000008c654:       d0 3c 4a e9     ld      r10,15568(r10)
c00000000008c658:       00 50 28 7c     cmpd    r8,r10
c00000000008c65c:       08 00 82 40     bne     c00000000008c664 <rcu_process_callbacks+0x7c>
c00000000008c660:       10 00 29 f9     std     r9,16(r9)
c00000000008c664:       08 00 29 f9     std     r9,8(r9)
c00000000008c668:       dc fe 80 3f     lis     r28,-292 <<<<====== "THIS IS INJECTED CODE"
c00000000008c66c:       21 d6 f8 4b     bl      c000000000019c8c <arch_local_irq_restore+0x8>
c00000000008c670:       00 00 00 60     nop
c00000000008c674:       00 00 60 3b     li      r27,0
c00000000008c678:       00 00 3f 2c     cmpdi   r31,0
c00000000008c67c:       ac ff 82 41     beq     c00000000008c628 <rcu_process_callbacks+0x40>
c00000000008c680:       00 00 df eb     ld      r30,0(r31)
c00000000008c684:       00 00 3e 2c     cmpdi   r30,0
c00000000008c688:       08 00 82 41     beq     c00000000008c690 <rcu_process_callbacks+0xa8>
c00000000008c68c:       2c f2 00 7c     dcbtct  0,r30
c00000000008c690:       2c 00 81 93     stw     r28,44(r1)
c00000000008c694:       08 00 bf eb     ld      r29,8(r31)
c00000000008c698:       00 00 3d 2c     cmpdi   r29,0
c00000000008c69c:       2c 00 21 81     lwz     r9,44(r1)
c00000000008c6a0:       01 00 29 39     addi    r9,r9,1
c00000000008c6a4:       2c 00 21 91     stw     r9,44(r1)
c00000000008c6a8:       10 00 82 40     bne     c00000000008c6b8 <rcu_process_callbacks+0xd0>
c00000000008c6ac:       78 fb e3 7f     mr      r3,r31
c00000000008c6b0:       15 f8 03 48     bl      c0000000000cbec4 <kmem_dump_obj+0x8>
c00000000008c6b4:       00 00 00 60     nop
c00000000008c6b8:       08 00 7f fb     std     r27,8(r31)
c00000000008c6bc:       78 fb e3 7f     mr      r3,r31
c00000000008c6c0:       a6 03 a9 7f     mtctr   r29
c00000000008c6c4:       78 eb ac 7f     mr      r12,r29
c00000000008c6c8:       18 00 41 f8     std     r2,24(r1)
c00000000008c6cc:       78 f3 df 7f     mr      r31,r30
c00000000008c6d0:       21 04 80 4e     bctrl
c00000000008c6d4:       18 00 41 e8     ld      r2,24(r1)
c00000000008c6d8:       a0 ff ff 4b     b       c00000000008c678 <rcu_process_callbacks+0x90>

- And here is the disasm without the injected code:

c00000000008c5e8 <rcu_process_callbacks>:
c00000000008c5e8:       32 00 4c 3c     addis   r2,r12,50
c00000000008c5ec:       18 ca 42 38     addi    r2,r2,-13800
c00000000008c5f0:       a6 02 08 7c     mflr    r0
c00000000008c5f4:       95 c3 23 48     bl      c0000000002c8988 <_savegpr0_28>
c00000000008c5f8:       c1 ff 21 f8     stdu    r1,-64(r1)
c00000000008c5fc:       fa 00 6d 88     lbz     r3,250(r13)
c00000000008c600:       01 00 69 60     ori     r9,r3,1
c00000000008c604:       fa 00 2d 99     stb     r9,250(r13)
c00000000008c608:       2b 01 42 3d     addis   r10,r2,299
c00000000008c60c:       2b 01 22 3d     addis   r9,r2,299
c00000000008c610:       d0 3c 4a e9     ld      r10,15568(r10)
c00000000008c614:       c8 3c 29 39     addi    r9,r9,15560
c00000000008c618:       00 48 2a 7c     cmpd    r10,r9
c00000000008c61c:       14 00 82 40     bne     c00000000008c630 <rcu_process_callbacks+0x48>
c00000000008c620:       6d d6 f8 4b     bl      c000000000019c8c <arch_local_irq_restore+0x8>
c00000000008c624:       00 00 00 60     nop
c00000000008c628:       40 00 21 38     addi    r1,r1,64
c00000000008c62c:       ac c3 23 48     b       c0000000002c89d8 <_restgpr0_28>
c00000000008c630:       00 00 0a e9     ld      r8,0(r10)
c00000000008c634:       2b 01 e2 3c     addis   r7,r2,299
c00000000008c638:       c8 3c e7 eb     ld      r31,15560(r7)
c00000000008c63c:       c8 3c 07 f9     std     r8,15560(r7)
c00000000008c640:       00 00 00 39     li      r8,0
c00000000008c644:       00 00 0a f9     std     r8,0(r10)
c00000000008c648:       2b 01 42 3d     addis   r10,r2,299
c00000000008c64c:       d8 3c 0a e9     ld      r8,15576(r10)
c00000000008c650:       2b 01 42 3d     addis   r10,r2,299
c00000000008c654:       d0 3c 4a e9     ld      r10,15568(r10)
c00000000008c658:       00 50 28 7c     cmpd    r8,r10
c00000000008c65c:       08 00 82 40     bne     c00000000008c664 <rcu_process_callbacks+0x7c>
c00000000008c660:       10 00 29 f9     std     r9,16(r9)
c00000000008c664:       08 00 29 f9     std     r9,8(r9)
c00000000008c668:       00 00 80 3b     li      r28,0
c00000000008c66c:       21 d6 f8 4b     bl      c000000000019c8c <arch_local_irq_restore+0x8>
c00000000008c670:       00 00 00 60     nop
c00000000008c674:       00 00 3f 2c     cmpdi   r31,0
c00000000008c678:       b0 ff 82 41     beq     c00000000008c628 <rcu_process_callbacks+0x40>
c00000000008c67c:       00 00 df eb     ld      r30,0(r31)
c00000000008c680:       00 00 3e 2c     cmpdi   r30,0
c00000000008c684:       08 00 82 41     beq     c00000000008c68c <rcu_process_callbacks+0xa4>
c00000000008c688:       2c f2 00 7c     dcbtct  0,r30
c00000000008c68c:       08 00 bf eb     ld      r29,8(r31)
c00000000008c690:       00 00 3d 2c     cmpdi   r29,0
c00000000008c694:       10 00 82 40     bne     c00000000008c6a4 <rcu_process_callbacks+0xbc>
c00000000008c698:       78 fb e3 7f     mr      r3,r31
c00000000008c69c:       19 f8 03 48     bl      c0000000000cbeb4 <kmem_dump_obj+0x8>
c00000000008c6a0:       00 00 00 60     nop
c00000000008c6a4:       08 00 9f fb     std     r28,8(r31)
c00000000008c6a8:       78 fb e3 7f     mr      r3,r31
c00000000008c6ac:       a6 03 a9 7f     mtctr   r29
c00000000008c6b0:       78 eb ac 7f     mr      r12,r29
c00000000008c6b4:       18 00 41 f8     std     r2,24(r1)
c00000000008c6b8:       78 f3 df 7f     mr      r31,r30
c00000000008c6bc:       21 04 80 4e     bctrl
c00000000008c6c0:       18 00 41 e8     ld      r2,24(r1)
c00000000008c6c4:       b0 ff ff 4b     b       c00000000008c674 <rcu_process_callbacks+0x8c>


Sounds like some sorta compiler optimizations, right?

Bests,
Manili

[-- Attachment #2: Type: text/html, Size: 27833 bytes --]