[REGRESSION] mainline/master: Apalis iMX6 no longer boots

[REGRESSION] mainline/master: Apalis iMX6 no longer boots

[Leonardo Costa]

Hello,

We are seeing a regression on Apalis iMX6 where the kernel doesn't boot in the
newest v7.2-rc1 (it was working before, in v7.1). The device tree being used is the imx6q-apalis-eval.dtb. The kernel
configuration used is the one shown below:

    https://gist.github.com/lcosta37/53efdb2fb6e6e0fc05437c7e53b47737

The kernel logs stop almost immediately as the board starts to boot, and I 
don't notice any difference in the logs that points to the cause.

Is this known? We are seeing this behavior on all Apalis iMX6 modules, though
we don't see it on Colibri iMX6, so it is not SoC-specific.

Logs from v7.2-rc1 (not working, printing stops after the last line pasted
here):

    [    0.000000] Booting Linux on physical CPU 0x0
    [    0.000000] Linux version 7.2.0-rc1-0.0.0-devel (oe-user@oe-host) (arm-tdx-linux-gnueabi-gcc (GCC) 16.1.0, GNU ld (GNU Binutils) 2.46.1) #1 SMP PREEMPT Sun Jun 28 19:01:31 UTC 2026
    [    0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
    [    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
    [    0.000000] OF: fdt: Machine model: Toradex Apalis iMX6Q/D Module on Apalis Evaluation Board
    [    0.000000] Memory policy: Data cache writealloc
    [    0.000000] cma: Reserved 256 MiB at 0x40000000
    [    0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT
    [    0.000000] Zone ranges:
    [    0.000000]   Normal   [mem 0x0000000010000000-0x000000003fffffff]
    [    0.000000]   HighMem  [mem 0x0000000040000000-0x000000004fffffff]
    [    0.000000] Movable zone start for each node
    [    0.000000] Early memory node ranges
    [    0.000000]   node   0: [mem 0x0000000010000000-0x000000004fffffff]
    [    0.000000] Initmem setup node 0 [mem 0x0000000010000000-0x000000004fffffff]
    [    0.000000] percpu: Embedded 15 pages/cpu s28684 r8192 d24564 u61440
    [    0.000000] Kernel command line: root=PARTUUID=adb2cea1-02 ro rootwait console=tty1 console=ttymxc0,115200
    [    0.000000] printk: log buffer data + meta data: 131072 + 409600 = 540672 bytes
    [    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
    [    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
    [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 262144
    [    0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
    [    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
    [    0.000000] rcu: Preemptible hierarchical RCU implementation.
    [    0.000000] rcu:     RCU event tracing is enabled.
    [    0.000000]  Trampoline variant of Tasks RCU enabled.
    [    0.000000]  Tracing variant of Tasks RCU enabled.
    [    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
    [    0.000000] RCU Tasks: Setting shift to 2 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=4.
    [    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
    [    0.000000] L2C-310 errata 752271 769419 enabled
    [    0.000000] L2C-310 enabling early BRESP for Cortex-A9
    [    0.000000] L2C-310 full line of zeros enabled for Cortex-A9
    [    0.000000] L2C-310 ID prefetch enabled, offset 16 lines


Logs from v7.1 (working) (full logs here: https://paste.debian.net/hidden/0f65ae5f)

    [    0.000000] Booting Linux on physical CPU 0x0
    [    0.000000] Linux version 7.1.0-0.0.0-devel (oe-user@oe-host) (arm-tdx-linux-gnueabi-gcc (GCC) 16.1.0, GNU ld (GNU Binutils) 2.46.1) #1 SMP PREEMPT Wed Jun 24 01:36:41 UTC 2026
    [    0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
    [    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
    [    0.000000] OF: fdt: Machine model: Toradex Apalis iMX6Q/D Module on Apalis Evaluation Board
    [    0.000000] Memory policy: Data cache writealloc
    [    0.000000] cma: Reserved 256 MiB at 0x40000000
    [    0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT
    [    0.000000] Zone ranges:
    [    0.000000]   Normal   [mem 0x0000000010000000-0x000000003fffffff]
    [    0.000000]   HighMem  [mem 0x0000000040000000-0x000000004fffffff]
    [    0.000000] Movable zone start for each node
    [    0.000000] Early memory node ranges
    [    0.000000]   node   0: [mem 0x0000000010000000-0x000000004fffffff]
    [    0.000000] Initmem setup node 0 [mem 0x0000000010000000-0x000000004fffffff]
    [    0.000000] percpu: Embedded 15 pages/cpu s28684 r8192 d24564 u61440
    [    0.000000] pcpu-alloc: s28684 r8192 d24564 u61440 alloc=15*4096
    [    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
    [    0.000000] Kernel command line: root=PARTUUID=4ce4ba92-02 ro rootwait console=tty1 console=ttymxc0,115200
    [    0.000000] printk: log buffer data + meta data: 131072 + 409600 = 540672 bytes
    [    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
    [    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
    [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 262144
    [    0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
    [    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
    [    0.000000] rcu: Preemptible hierarchical RCU implementation.
    [    0.000000] rcu:     RCU event tracing is enabled.
    [    0.000000]  Trampoline variant of Tasks RCU enabled.
    [    0.000000]  Tracing variant of Tasks RCU enabled.
    [    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
    [    0.000000] RCU Tasks: Setting shift to 2 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=4.
    [    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
    [    0.000000] L2C-310 errata 752271 769419 enabled
    [    0.000000] L2C-310 enabling early BRESP for Cortex-A9
    [    0.000000] L2C-310 full line of zeros enabled for Cortex-A9
    [    0.000000] L2C-310 ID prefetch enabled, offset 16 lines
    [    0.000000] L2C-310 dynamic clock gating enabled, standby mode enabled
    [    0.000000] L2C-310 cache controller enabled, 16 ways, 1024 kB
    [    0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x76470001
    [    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
    [    0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
    [    0.000000] Switching to timer-based delay loop, resolution 333ns
    [    0.000001] sched_clock: 32 bits at 3000kHz, resolution 333ns, wraps every 715827882841ns
    [    0.000018] clocksource: mxc_timer1: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 637086815595 ns
    [    0.001910] Console: colour dummy device 80x30
    [    0.001926] printk: legacy console [tty1] enabled
    [    0.002519] Calibrating delay loop (skipped), value calculated using timer frequency.. 6.00 BogoMIPS (lpj=30000)
    [    0.002561] CPU: Testing write buffer coherency: ok
    [    0.002627] CPU0: Spectre v2: using BPIALL workaround
    [    0.002650] pid_max: default: 32768 minimum: 301
    [    0.002989] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
    [    0.003038] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
    [    0.003430] VFS: Finished mounting rootfs on nullfs
    [    0.004538] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
    [    0.006552] Setting up static identity map for 0x10100000 - 0x10100060
    [    0.006835] rcu: Hierarchical SRCU implementation.
    [    0.006864] rcu:     Max phase no-delay instances is 1000.
    [    0.007320] Timer migration: 1 hierarchy levels; 8 children per group; 1 crossnode level
    [    0.008854] smp: Bringing up secondary CPUs ...
    [    0.010035] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
    [    0.010218] CPU1: Spectre v2: using BPIALL workaround
    [    0.011412] CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
    [    0.011581] CPU2: Spectre v2: using BPIALL workaround
    [    0.012747] CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
    [    0.012917] CPU3: Spectre v2: using BPIALL workaround
    [    0.013109] smp: Brought up 1 node, 4 CPUs
    ...

Re: [REGRESSION] mainline/master: Apalis iMX6 no longer boots

[Frank Li]

On Mon, Jun 29, 2026 at 11:34:32AM -0300, Leonardo Costa wrote:
> Hello,
>
> We are seeing a regression on Apalis iMX6 where the kernel doesn't boot in the
> newest v7.2-rc1 (it was working before, in v7.1). The device tree being used is the imx6q-apalis-eval.dtb. The kernel
> configuration used is the one shown below:
>
>     https://gist.github.com/lcosta37/53efdb2fb6e6e0fc05437c7e53b47737
>
> The kernel logs stop almost immediately as the board starts to boot, and I
> don't notice any difference in the logs that points to the cause.
>
> Is this known? We are seeing this behavior on all Apalis iMX6 modules, though
> we don't see it on Colibri iMX6, so it is not SoC-specific.

Can you help bisect to locate which commit cause this problem?

Frank

>
> Logs from v7.2-rc1 (not working, printing stops after the last line pasted
> here):
>
>     [    0.000000] Booting Linux on physical CPU 0x0
>     [    0.000000] Linux version 7.2.0-rc1-0.0.0-devel (oe-user@oe-host) (arm-tdx-linux-gnueabi-gcc (GCC) 16.1.0, GNU ld (GNU Binutils) 2.46.1) #1 SMP PREEMPT Sun Jun 28 19:01:31 UTC 2026
>     [    0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
>     [    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
>     [    0.000000] OF: fdt: Machine model: Toradex Apalis iMX6Q/D Module on Apalis Evaluation Board
>     [    0.000000] Memory policy: Data cache writealloc
>     [    0.000000] cma: Reserved 256 MiB at 0x40000000
>     [    0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT
>     [    0.000000] Zone ranges:
>     [    0.000000]   Normal   [mem 0x0000000010000000-0x000000003fffffff]
>     [    0.000000]   HighMem  [mem 0x0000000040000000-0x000000004fffffff]
>     [    0.000000] Movable zone start for each node
>     [    0.000000] Early memory node ranges
>     [    0.000000]   node   0: [mem 0x0000000010000000-0x000000004fffffff]
>     [    0.000000] Initmem setup node 0 [mem 0x0000000010000000-0x000000004fffffff]
>     [    0.000000] percpu: Embedded 15 pages/cpu s28684 r8192 d24564 u61440
>     [    0.000000] Kernel command line: root=PARTUUID=adb2cea1-02 ro rootwait console=tty1 console=ttymxc0,115200
>     [    0.000000] printk: log buffer data + meta data: 131072 + 409600 = 540672 bytes
>     [    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
>     [    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
>     [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 262144
>     [    0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
>     [    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
>     [    0.000000] rcu: Preemptible hierarchical RCU implementation.
>     [    0.000000] rcu:     RCU event tracing is enabled.
>     [    0.000000]  Trampoline variant of Tasks RCU enabled.
>     [    0.000000]  Tracing variant of Tasks RCU enabled.
>     [    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
>     [    0.000000] RCU Tasks: Setting shift to 2 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=4.
>     [    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
>     [    0.000000] L2C-310 errata 752271 769419 enabled
>     [    0.000000] L2C-310 enabling early BRESP for Cortex-A9
>     [    0.000000] L2C-310 full line of zeros enabled for Cortex-A9
>     [    0.000000] L2C-310 ID prefetch enabled, offset 16 lines
>
>
> Logs from v7.1 (working) (full logs here: https://paste.debian.net/hidden/0f65ae5f)
>
>     [    0.000000] Booting Linux on physical CPU 0x0
>     [    0.000000] Linux version 7.1.0-0.0.0-devel (oe-user@oe-host) (arm-tdx-linux-gnueabi-gcc (GCC) 16.1.0, GNU ld (GNU Binutils) 2.46.1) #1 SMP PREEMPT Wed Jun 24 01:36:41 UTC 2026
>     [    0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
>     [    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
>     [    0.000000] OF: fdt: Machine model: Toradex Apalis iMX6Q/D Module on Apalis Evaluation Board
>     [    0.000000] Memory policy: Data cache writealloc
>     [    0.000000] cma: Reserved 256 MiB at 0x40000000
>     [    0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT
>     [    0.000000] Zone ranges:
>     [    0.000000]   Normal   [mem 0x0000000010000000-0x000000003fffffff]
>     [    0.000000]   HighMem  [mem 0x0000000040000000-0x000000004fffffff]
>     [    0.000000] Movable zone start for each node
>     [    0.000000] Early memory node ranges
>     [    0.000000]   node   0: [mem 0x0000000010000000-0x000000004fffffff]
>     [    0.000000] Initmem setup node 0 [mem 0x0000000010000000-0x000000004fffffff]
>     [    0.000000] percpu: Embedded 15 pages/cpu s28684 r8192 d24564 u61440
>     [    0.000000] pcpu-alloc: s28684 r8192 d24564 u61440 alloc=15*4096
>     [    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
>     [    0.000000] Kernel command line: root=PARTUUID=4ce4ba92-02 ro rootwait console=tty1 console=ttymxc0,115200
>     [    0.000000] printk: log buffer data + meta data: 131072 + 409600 = 540672 bytes
>     [    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
>     [    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
>     [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 262144
>     [    0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
>     [    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
>     [    0.000000] rcu: Preemptible hierarchical RCU implementation.
>     [    0.000000] rcu:     RCU event tracing is enabled.
>     [    0.000000]  Trampoline variant of Tasks RCU enabled.
>     [    0.000000]  Tracing variant of Tasks RCU enabled.
>     [    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
>     [    0.000000] RCU Tasks: Setting shift to 2 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=4.
>     [    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
>     [    0.000000] L2C-310 errata 752271 769419 enabled
>     [    0.000000] L2C-310 enabling early BRESP for Cortex-A9
>     [    0.000000] L2C-310 full line of zeros enabled for Cortex-A9
>     [    0.000000] L2C-310 ID prefetch enabled, offset 16 lines
>     [    0.000000] L2C-310 dynamic clock gating enabled, standby mode enabled
>     [    0.000000] L2C-310 cache controller enabled, 16 ways, 1024 kB
>     [    0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x76470001
>     [    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
>     [    0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
>     [    0.000000] Switching to timer-based delay loop, resolution 333ns
>     [    0.000001] sched_clock: 32 bits at 3000kHz, resolution 333ns, wraps every 715827882841ns
>     [    0.000018] clocksource: mxc_timer1: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 637086815595 ns
>     [    0.001910] Console: colour dummy device 80x30
>     [    0.001926] printk: legacy console [tty1] enabled
>     [    0.002519] Calibrating delay loop (skipped), value calculated using timer frequency.. 6.00 BogoMIPS (lpj=30000)
>     [    0.002561] CPU: Testing write buffer coherency: ok
>     [    0.002627] CPU0: Spectre v2: using BPIALL workaround
>     [    0.002650] pid_max: default: 32768 minimum: 301
>     [    0.002989] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
>     [    0.003038] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
>     [    0.003430] VFS: Finished mounting rootfs on nullfs
>     [    0.004538] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
>     [    0.006552] Setting up static identity map for 0x10100000 - 0x10100060
>     [    0.006835] rcu: Hierarchical SRCU implementation.
>     [    0.006864] rcu:     Max phase no-delay instances is 1000.
>     [    0.007320] Timer migration: 1 hierarchy levels; 8 children per group; 1 crossnode level
>     [    0.008854] smp: Bringing up secondary CPUs ...
>     [    0.010035] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
>     [    0.010218] CPU1: Spectre v2: using BPIALL workaround
>     [    0.011412] CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
>     [    0.011581] CPU2: Spectre v2: using BPIALL workaround
>     [    0.012747] CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
>     [    0.012917] CPU3: Spectre v2: using BPIALL workaround
>     [    0.013109] smp: Brought up 1 node, 4 CPUs
>     ...
>

RE: [REGRESSION] mainline/master: Apalis iMX6 no longer boots

[Hongxing Zhu (OSS)]

> -----Original Message-----
> From: Frank Li <Frank.li@oss.nxp.com>
> Sent: Tuesday, June 30, 2026 4:19 AM
> To: Leonardo Costa <leoreis.costa@gmail.com>
> Cc: robh@kernel.org; krzk+dt@kernel.org; conor+dt@kernel.org; Frank Li
> <frank.li@nxp.com>; s.hauer@pengutronix.de; kernel@pengutronix.de;
> festevam@gmail.com; leonardo.costa@toradex.com;
> devicetree@vger.kernel.org; imx@lists.linux.dev; linux-arm-
> kernel@lists.infradead.org; linux-kernel@vger.kernel.org;
> regressions@lists.linux.dev
> Subject: Re: [REGRESSION] mainline/master: Apalis iMX6 no longer boots
> 
> On Mon, Jun 29, 2026 at 11:34:32AM -0300, Leonardo Costa wrote:
> > Hello,
> >
> > We are seeing a regression on Apalis iMX6 where the kernel doesn't
> > boot in the newest v7.2-rc1 (it was working before, in v7.1). The
> > device tree being used is the imx6q-apalis-eval.dtb. The kernel configuration
> used is the one shown below:
> >
> >
> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist
> > .github.com%2Flcosta37%2F53efdb2fb6e6e0fc05437c7e53b47737&data=
> 05%7C02
> > %7Chongxing.zhu%40nxp.com%7Ca4fdd946d8064da8b63d08ded61bab4
> 2%7C686ea1d
> >
> 3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C639183611522951306%7CU
> nknown%7CTWF
> >
> pbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW
> 4zMiIsI
> >
> kFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=yVygffYEhpLZ
> f5tpwu4
> > yVZn5sL%2BdQySGZwryUh3i0zU%3D&reserved=0
> >
> > The kernel logs stop almost immediately as the board starts to boot,
> > and I don't notice any difference in the logs that points to the cause.
> >
> > Is this known? We are seeing this behavior on all Apalis iMX6 modules,
> > though we don't see it on Colibri iMX6, so it is not SoC-specific.
Hi Leonardo:
Can you try this fixes commit?
https://lore.kernel.org/imx/20260707062250.GA20259@francesco-nb/T/#m06ea82923d53e28a234d0052596559eaa2c64987

Best Regards
Richard Zhu
> 
> Can you help bisect to locate which commit cause this problem?
> 
> Frank
> 
> >
> > Logs from v7.2-rc1 (not working, printing stops after the last line
> > pasted
> > here):
> >
> >     [    0.000000] Booting Linux on physical CPU 0x0
> >     [    0.000000] Linux version 7.2.0-rc1-0.0.0-devel (oe-user@oe-host)
> (arm-tdx-linux-gnueabi-gcc (GCC) 16.1.0, GNU ld (GNU Binutils) 2.46.1) #1
> SMP PREEMPT Sun Jun 28 19:01:31 UTC 2026
> >     [    0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7),
> cr=10c5387d
> >     [    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing
> instruction cache
> >     [    0.000000] OF: fdt: Machine model: Toradex Apalis iMX6Q/D Module
> on Apalis Evaluation Board
> >     [    0.000000] Memory policy: Data cache writealloc
> >     [    0.000000] cma: Reserved 256 MiB at 0x40000000
> >     [    0.000000] OF: reserved mem: Reserved memory: No reserved-memory
> node in the DT
> >     [    0.000000] Zone ranges:
> >     [    0.000000]   Normal   [mem 0x0000000010000000-
> 0x000000003fffffff]
> >     [    0.000000]   HighMem  [mem 0x0000000040000000-
> 0x000000004fffffff]
> >     [    0.000000] Movable zone start for each node
> >     [    0.000000] Early memory node ranges
> >     [    0.000000]   node   0: [mem 0x0000000010000000-
> 0x000000004fffffff]
> >     [    0.000000] Initmem setup node 0 [mem 0x0000000010000000-
> 0x000000004fffffff]
> >     [    0.000000] percpu: Embedded 15 pages/cpu s28684 r8192 d24564
> u61440
> >     [    0.000000] Kernel command line: root=PARTUUID=adb2cea1-02 ro
> rootwait console=tty1 console=ttymxc0,115200
> >     [    0.000000] printk: log buffer data + meta data: 131072 + 409600 =
> 540672 bytes
> >     [    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288
> bytes, linear)
> >     [    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144
> bytes, linear)
> >     [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 262144
> >     [    0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
> >     [    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=4,
> Nodes=1
> >     [    0.000000] rcu: Preemptible hierarchical RCU implementation.
> >     [    0.000000] rcu:     RCU event tracing is enabled.
> >     [    0.000000]  Trampoline variant of Tasks RCU enabled.
> >     [    0.000000]  Tracing variant of Tasks RCU enabled.
> >     [    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10
> jiffies.
> >     [    0.000000] RCU Tasks: Setting shift to 2 and lim to 1
> rcu_task_cb_adjust=1 rcu_task_cpu_ids=4.
> >     [    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
> >     [    0.000000] L2C-310 errata 752271 769419 enabled
> >     [    0.000000] L2C-310 enabling early BRESP for Cortex-A9
> >     [    0.000000] L2C-310 full line of zeros enabled for Cortex-A9
> >     [    0.000000] L2C-310 ID prefetch enabled, offset 16 lines
> >
> >
> > Logs from v7.1 (working) (full logs here:
> >
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpast
> >
> e.debian.net%2Fhidden%2F0f65ae5f&data=05%7C02%7Chongxing.zhu%40
> nxp.com
> > %7Ca4fdd946d8064da8b63d08ded61bab42%7C686ea1d3bc2b4c6fa92cd
> 99c5c301635
> > %7C0%7C0%7C639183611522973764%7CUnknown%7CTWFpbGZsb3d8e
> yJFbXB0eU1hcGki
> >
> OnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjo
> yfQ
> > %3D%3D%7C0%7C%7C%7C&sdata=s3LVv01DMUor2Kwv25bOMyk8Q5oFs
> qBhUVx6Odozwp4%
> > 3D&reserved=0)
> >
> >     [    0.000000] Booting Linux on physical CPU 0x0
> >     [    0.000000] Linux version 7.1.0-0.0.0-devel (oe-user@oe-host) (arm-
> tdx-linux-gnueabi-gcc (GCC) 16.1.0, GNU ld (GNU Binutils) 2.46.1) #1 SMP
> PREEMPT Wed Jun 24 01:36:41 UTC 2026
> >     [    0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7),
> cr=10c5387d
> >     [    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing
> instruction cache
> >     [    0.000000] OF: fdt: Machine model: Toradex Apalis iMX6Q/D Module
> on Apalis Evaluation Board
> >     [    0.000000] Memory policy: Data cache writealloc
> >     [    0.000000] cma: Reserved 256 MiB at 0x40000000
> >     [    0.000000] OF: reserved mem: Reserved memory: No reserved-memory
> node in the DT
> >     [    0.000000] Zone ranges:
> >     [    0.000000]   Normal   [mem 0x0000000010000000-
> 0x000000003fffffff]
> >     [    0.000000]   HighMem  [mem 0x0000000040000000-
> 0x000000004fffffff]
> >     [    0.000000] Movable zone start for each node
> >     [    0.000000] Early memory node ranges
> >     [    0.000000]   node   0: [mem 0x0000000010000000-
> 0x000000004fffffff]
> >     [    0.000000] Initmem setup node 0 [mem 0x0000000010000000-
> 0x000000004fffffff]
> >     [    0.000000] percpu: Embedded 15 pages/cpu s28684 r8192 d24564
> u61440
> >     [    0.000000] pcpu-alloc: s28684 r8192 d24564 u61440 alloc=15*4096
> >     [    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
> >     [    0.000000] Kernel command line: root=PARTUUID=4ce4ba92-02 ro
> rootwait console=tty1 console=ttymxc0,115200
> >     [    0.000000] printk: log buffer data + meta data: 131072 + 409600 =
> 540672 bytes
> >     [    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288
> bytes, linear)
> >     [    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144
> bytes, linear)
> >     [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 262144
> >     [    0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
> >     [    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=4,
> Nodes=1
> >     [    0.000000] rcu: Preemptible hierarchical RCU implementation.
> >     [    0.000000] rcu:     RCU event tracing is enabled.
> >     [    0.000000]  Trampoline variant of Tasks RCU enabled.
> >     [    0.000000]  Tracing variant of Tasks RCU enabled.
> >     [    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10
> jiffies.
> >     [    0.000000] RCU Tasks: Setting shift to 2 and lim to 1
> rcu_task_cb_adjust=1 rcu_task_cpu_ids=4.
> >     [    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
> >     [    0.000000] L2C-310 errata 752271 769419 enabled
> >     [    0.000000] L2C-310 enabling early BRESP for Cortex-A9
> >     [    0.000000] L2C-310 full line of zeros enabled for Cortex-A9
> >     [    0.000000] L2C-310 ID prefetch enabled, offset 16 lines
> >     [    0.000000] L2C-310 dynamic clock gating enabled, standby mode
> enabled
> >     [    0.000000] L2C-310 cache controller enabled, 16 ways, 1024 kB
> >     [    0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x76470001
> >     [    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
> >     [    0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
> max_idle_ns: 19112604462750000 ns
> >     [    0.000000] Switching to timer-based delay loop, resolution 333ns
> >     [    0.000001] sched_clock: 32 bits at 3000kHz, resolution 333ns, wraps
> every 715827882841ns
> >     [    0.000018] clocksource: mxc_timer1: mask: 0xffffffff max_cycles:
> 0xffffffff, max_idle_ns: 637086815595 ns
> >     [    0.001910] Console: colour dummy device 80x30
> >     [    0.001926] printk: legacy console [tty1] enabled
> >     [    0.002519] Calibrating delay loop (skipped), value calculated using timer
> frequency.. 6.00 BogoMIPS (lpj=30000)
> >     [    0.002561] CPU: Testing write buffer coherency: ok
> >     [    0.002627] CPU0: Spectre v2: using BPIALL workaround
> >     [    0.002650] pid_max: default: 32768 minimum: 301
> >     [    0.002989] Mount-cache hash table entries: 2048 (order: 1, 8192
> bytes, linear)
> >     [    0.003038] Mountpoint-cache hash table entries: 2048 (order: 1, 8192
> bytes, linear)
> >     [    0.003430] VFS: Finished mounting rootfs on nullfs
> >     [    0.004538] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> >     [    0.006552] Setting up static identity map for 0x10100000 -
> 0x10100060
> >     [    0.006835] rcu: Hierarchical SRCU implementation.
> >     [    0.006864] rcu:     Max phase no-delay instances is 1000.
> >     [    0.007320] Timer migration: 1 hierarchy levels; 8 children per group; 1
> crossnode level
> >     [    0.008854] smp: Bringing up secondary CPUs ...
> >     [    0.010035] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> >     [    0.010218] CPU1: Spectre v2: using BPIALL workaround
> >     [    0.011412] CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
> >     [    0.011581] CPU2: Spectre v2: using BPIALL workaround
> >     [    0.012747] CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
> >     [    0.012917] CPU3: Spectre v2: using BPIALL workaround
> >     [    0.013109] smp: Brought up 1 node, 4 CPUs
> >     ...
> >

Re: [REGRESSION] mainline/master: Apalis iMX6 no longer boots

[Bjorn Helgaas]

[+cc Sherry, author of 610fa91d9863, Richard, Mani, linux-pci]

On Mon, Jun 29, 2026 at 11:34:32AM -0300, Leonardo Costa wrote:
> Hello,
> 
> We are seeing a regression on Apalis iMX6 where the kernel doesn't boot in the
> newest v7.2-rc1 (it was working before, in v7.1). The device tree being used is the imx6q-apalis-eval.dtb. The kernel
> configuration used is the one shown below:
> 
>     https://gist.github.com/lcosta37/53efdb2fb6e6e0fc05437c7e53b47737
> 
> The kernel logs stop almost immediately as the board starts to boot, and I 
> don't notice any difference in the logs that points to the cause.
> 
> Is this known? We are seeing this behavior on all Apalis iMX6 modules, though
> we don't see it on Colibri iMX6, so it is not SoC-specific.

This regression is listed on
https://linux-regtracking.leemhuis.info/regzbot/mainline/ as being
caused by
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=master&id=610fa91d9863
("PCI: imx6: Assert PERST# before enabling regulators") but I can't
find any bisect or other report that specifically points to
610fa91d9863.

Can somebody confirm that reverting 610fa91d9863 fixes this
regression, or at least that 610fa91d9863^ works and 610fa91d9863
fails?

Bjorn

RE: [REGRESSION] mainline/master: Apalis iMX6 no longer boots

[Hongxing Zhu (OSS)]

> -----Original Message-----
> From: Bjorn Helgaas <helgaas@kernel.org>
> Sent: Thursday, July 16, 2026 7:48 AM
> To: Leonardo Costa <leoreis.costa@gmail.com>
> Cc: robh@kernel.org; krzk+dt@kernel.org; conor+dt@kernel.org; Frank Li
> <frank.li@nxp.com>; s.hauer@pengutronix.de; kernel@pengutronix.de;
> festevam@gmail.com; leonardo.costa@toradex.com;
> devicetree@vger.kernel.org; imx@lists.linux.dev; linux-arm-
> kernel@lists.infradead.org; linux-kernel@vger.kernel.org;
> regressions@lists.linux.dev; Sherry Sun <sherry.sun@nxp.com>; Manivannan
> Sadhasivam <mani@kernel.org>; Hongxing Zhu <hongxing.zhu@nxp.com>; linux-
> pci@vger.kernel.org
> Subject: Re: [REGRESSION] mainline/master: Apalis iMX6 no longer boots
>
> [+cc Sherry, author of 610fa91d9863, Richard, Mani, linux-pci]
>
> On Mon, Jun 29, 2026 at 11:34:32AM -0300, Leonardo Costa wrote:
> > Hello,
> >
> > We are seeing a regression on Apalis iMX6 where the kernel doesn't
> > boot in the newest v7.2-rc1 (it was working before, in v7.1). The
> > device tree being used is the imx6q-apalis-eval.dtb. The kernel configuration
> used is the one shown below:
> >
> >
> > https://gist/
> > .github.com%2Flcosta37%2F53efdb2fb6e6e0fc05437c7e53b47737&data=05%7
> C02
> > %7Chongxing.zhu%40nxp.com%7C3d58575d22384d61d15608dee2cb7471%7C
> 686ea1d
> >
> 3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C639197560620871110%7CUnknow
> n%7CTWF
> >
> pbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMi
> IsI
> >
> kFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=KtTOaEVJI3aq6Bt
> V7Js
> > Y7r%2F9B5LY%2F%2BVj3xEMk592ghk%3D&reserved=0
> >
> > The kernel logs stop almost immediately as the board starts to boot,
> > and I don't notice any difference in the logs that points to the cause.
> >
> > Is this known? We are seeing this behavior on all Apalis iMX6 modules,
> > though we don't see it on Colibri iMX6, so it is not SoC-specific.
>
> This regression is listed on
> https://linux-/
> regtracking.leemhuis.info%2Fregzbot%2Fmainline%2F&data=05%7C02%7Chongx
> ing.zhu%40nxp.com%7C3d58575d22384d61d15608dee2cb7471%7C686ea1d3bc
> 2b4c6fa92cd99c5c301635%7C0%7C0%7C639197560620886501%7CUnknown%
> 7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXa
> W4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=K%2FBs
> xg56w4Jv4Phf%2B7i4CePTQA1M45xNH%2BxcrjuA4r0%3D&reserved=0 as being
> caused by
> https://git.kernel/.
> org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Ftorvalds%2Flinux.git%2Fcommit
> %2F%3Fh%3Dmaster%26id%3D610fa91d9863&data=05%7C02%7Chongxing.zhu
> %40nxp.com%7C3d58575d22384d61d15608dee2cb7471%7C686ea1d3bc2b4c6f
> a92cd99c5c301635%7C0%7C0%7C639197560620895764%7CUnknown%7CTWF
> pbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMi
> IsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ZT0p2CaXc8EJ
> YM2mlEAJegvmDqObaa%2FFiZtbCvM3PHw%3D&reserved=0
> ("PCI: imx6: Assert PERST# before enabling regulators") but I can't find any bisect
> or other report that specifically points to 610fa91d9863.
>
> Can somebody confirm that reverting 610fa91d9863 fixes this regression, or at
> least that 610fa91d9863^ works and 610fa91d9863 fails?
Sorry for this severe regression. I have posted a fix that has been tested on
my i.MX6Q SabreSD board:

https://lore.kernel.org/imx/20260708035928.580236-2-hongxing.zhu@oss.nxp.com/

Hi Leonardo,

Would you be able to test this patch on your affected boards? Your testing
would help expedite the merge process and ensure the fix resolves the boot
issue you're experiencing.

Best Regards
Richard Zhu
>
> Bjorn

Re: [REGRESSION] mainline/master: Apalis iMX6 no longer boots

[Bjorn Helgaas]

On Thu, Jul 16, 2026 at 02:15:42AM +0000, Hongxing Zhu (OSS) wrote:
> > On Mon, Jun 29, 2026 at 11:34:32AM -0300, Leonardo Costa wrote:
> > > We are seeing a regression on Apalis iMX6 where the kernel
> > > doesn't boot in the newest v7.2-rc1 (it was working before, in
> > > v7.1). The device tree being used is the imx6q-apalis-eval.dtb.
> > > The kernel configuration
> > > used is the one shown below:
> ...

> > Can somebody confirm that reverting 610fa91d9863 fixes this
> > regression, or at least that 610fa91d9863^ works and 610fa91d9863
> > fails?
>
> Sorry for this severe regression. I have posted a fix that has been
> tested on my i.MX6Q SabreSD board:
> 
> https://lore.kernel.org/imx/20260708035928.580236-2-hongxing.zhu@oss.nxp.com/

My question was whether somebody had actually bisected to 610fa91d9863
("PCI: imx6: Assert PERST# before enabling regulators").  I don't see
anything specific on the mailing list, so I don't know how
610fa91d9863 got blamed for the regression.

https://linux-regtracking.leemhuis.info/regzbot/mainline/ lists this
as a regression caused by 610fa91d9863, which means we need (1) the
details to prove that 610fa91d9863 actually caused it and (2) a tested
revert or fix.

Re: [REGRESSION] mainline/master: Apalis iMX6 no longer boots

[Leonardo Costa]

On Thu, Jul 16, 2026 at 02:15:42AM +0000, Hongxing Zhu (OSS) wrote:
> Hi Leonardo,
> 
> Would you be able to test this patch on your affected boards? Your testing
> would help expedite the merge process and ensure the fix resolves the boot
> issue you're experiencing.
> 
> Best Regards
> Richard Zhu

Hi Richard,

Sorry for taking long to respond, I was caught up with something. I
tested the patch and it solved the issue. I sent a Tested-by tag there.
Thanks for looking into this!

Best Regards
Leonardo

Re: [REGRESSION] mainline/master: Apalis iMX6 no longer boots

[Thorsten Leemhuis]

On 7/16/26 18:40, Bjorn Helgaas wrote:
> On Thu, Jul 16, 2026 at 02:15:42AM +0000, Hongxing Zhu (OSS) wrote:
>>> On Mon, Jun 29, 2026 at 11:34:32AM -0300, Leonardo Costa wrote:
>>>> We are seeing a regression on Apalis iMX6 where the kernel
>>>> doesn't boot in the newest v7.2-rc1 (it was working before, in
>>>> v7.1). The device tree being used is the imx6q-apalis-eval.dtb.
>>>> The kernel configuration
>>>> used is the one shown below:
>> ...
> 
>>> Can somebody confirm that reverting 610fa91d9863 fixes this
>>> regression, or at least that 610fa91d9863^ works and 610fa91d9863
>>> fails?
>>
>> Sorry for this severe regression. I have posted a fix that has been
>> tested on my i.MX6Q SabreSD board:
>>
>> https://lore.kernel.org/imx/20260708035928.580236-2-hongxing.zhu@oss.nxp.com/
> 
> My question was whether somebody had actually bisected to 610fa91d9863
> ("PCI: imx6: Assert PERST# before enabling regulators").  I don't see
> anything specific on the mailing list, so I don't know how
> 610fa91d9863 got blamed for the regression.

That was me and might have been a stupid idea. I normally wouldn't have
added this, but a earlier message from Hongxing Zhu sounded a pretty
convincingly like something along the lines of  "this is the problem I
already have a fix for there" -- and then I took the commit-id from there.

Sorry.

Ciao, Thorsten

Re: [REGRESSION] mainline/master: Apalis iMX6 no longer boots

[Bjorn Helgaas]

On Thu, Jul 16, 2026 at 07:16:31PM +0200, Thorsten Leemhuis wrote:
> On 7/16/26 18:40, Bjorn Helgaas wrote:
> > On Thu, Jul 16, 2026 at 02:15:42AM +0000, Hongxing Zhu (OSS) wrote:
> >>> On Mon, Jun 29, 2026 at 11:34:32AM -0300, Leonardo Costa wrote:
> >>>> We are seeing a regression on Apalis iMX6 where the kernel
> >>>> doesn't boot in the newest v7.2-rc1 (it was working before, in
> >>>> v7.1). The device tree being used is the imx6q-apalis-eval.dtb.
> >>>> The kernel configuration
> >>>> used is the one shown below:
> >> ...
> > 
> >>> Can somebody confirm that reverting 610fa91d9863 fixes this
> >>> regression, or at least that 610fa91d9863^ works and 610fa91d9863
> >>> fails?
> >>
> >> Sorry for this severe regression. I have posted a fix that has been
> >> tested on my i.MX6Q SabreSD board:
> >>
> >> https://lore.kernel.org/imx/20260708035928.580236-2-hongxing.zhu@oss.nxp.com/
> > 
> > My question was whether somebody had actually bisected to 610fa91d9863
> > ("PCI: imx6: Assert PERST# before enabling regulators").  I don't see
> > anything specific on the mailing list, so I don't know how
> > 610fa91d9863 got blamed for the regression.
> 
> That was me and might have been a stupid idea. I normally wouldn't have
> added this, but a earlier message from Hongxing Zhu sounded a pretty
> convincingly like something along the lines of  "this is the problem I
> already have a fix for there" -- and then I took the commit-id from there.

Ah, ok, that explains it, thanks!  Hopefully we can get some clear
connection in the patch that eventually gets merged.