qemu-system-aarch64 hangs in a cortex-a72 test

qemu-system-aarch64 hangs in a cortex-a72 test

[Liviu Ionescu]

Hi!

I have a small semihosted test that I use to test the xPack QEMU releases.

The test worked just fine up to 8.2.9, but hangs with 9.0.0 or later, including 10.0.0.

The test is invoked with:

```
$ qemu-system-aarch64 --machine virt --cpu cortex-a72 --kernel hello-world-cortex-a72.elf --nographic -smp 1 -d unimp,guest_errors --semihosting-config enable=on,target=native,arg=hello-world,arg=A7
^C
$
```

The test binary is available from:

- https://github.com/xpack-dev-tools/qemu-arm-xpack/tree/xpack/build-assets/test-assets


I tested on macOS with 9.0.0, 9.0.4, 9.2.4 compiled from sources, and with 10.0.0 from the official Manjaro distribution. The 9.2.4 tests were also performed on Linux, with the same result.

The binary was compiled from sources; I don't have the sources at hand but I can reconstruct them. I do not have a physical board to validate that the test code is functional, I always ran it on QEMU. The test might very well have a problem that was not detected by pre-9.x QEMU, but I'm not familiar enough with A72 to diagnose it. 

Any ideas what changed from 8.x to 9.0 to cause this?

Thank you,

Liviu

Re: qemu-system-aarch64 hangs in a cortex-a72 test

[Peter Maydell]

On Sun, 5 Oct 2025 at 17:55, Liviu Ionescu <ilg@livius.net> wrote:
>
> Hi!
>
> I have a small semihosted test that I use to test the xPack QEMU releases.
>
> The test worked just fine up to 8.2.9, but hangs with 9.0.0 or later, including 10.0.0.

> Any ideas what changed from 8.x to 9.0 to cause this?

Notably 9.0 is the first release which included
commit 59754f85ed, which enforces the architectural
requirement that when the MMU is disabled unaligned
loads and stores must fault. We didn't use to check
that, so bare metal code that was compiled with a
compiler that assumes unaligned accesses are permitted
(or which simply has a bug where it does an unaligned
access not permitted by the C spec) will now take an
exception where it didn't before. (You should be able
to see that in the -d debug logs if it's happening,
or via gdbstub if you put a breakpoint on the exception
entry point.)

If it doesn't look like that's what it's falling over on,
you could try a git bisect of QEMU to pinpoint the
commit where it stopped working.

thanks
-- PMM

Re: qemu-system-aarch64 hangs in a cortex-a72 test

[Liviu Ionescu]

> On 6 Oct 2025, at 12:55, Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> Notably 9.0 is the first release which included
> commit 59754f85ed, which enforces the architectural
> requirement that when the MMU is disabled unaligned
> loads and stores must fault.

Ok, this is a good hint, a loop in an exception handler is a very likely cause of the program hanging.

> ... (You should be able
> to see that in the -d debug logs if it's happening,
> or via gdbstub if you put a breakpoint on the exception
> entry point.)

I'll try to run a debug session and see if this is the case, and fix it.

> If it doesn't look like that's what it's falling over on,
> you could try a git bisect of QEMU to pinpoint the
> commit where it stopped working.

Right. Hopefully this will not be necessary, it must be some small stupid mistake in my code, not detected before.


Thank you,

Liviu

Re: qemu-system-aarch64 hangs in a cortex-a72 test

[Peter Maydell]

On Mon, 6 Oct 2025 at 11:05, Liviu Ionescu <ilg@livius.net> wrote:
>
>
>
> > On 6 Oct 2025, at 12:55, Peter Maydell <peter.maydell@linaro.org> wrote:
> >
> > Notably 9.0 is the first release which included
> > commit 59754f85ed, which enforces the architectural
> > requirement that when the MMU is disabled unaligned
> > loads and stores must fault.
>
> Ok, this is a good hint, a loop in an exception handler is a very likely cause of the program hanging.
>
> > ... (You should be able
> > to see that in the -d debug logs if it's happening,
> > or via gdbstub if you put a breakpoint on the exception
> > entry point.)
>
> I'll try to run a debug session and see if this is the case, and fix it.

Note that depending on your compiler's config it might assume
unaligned accesses are OK by default (because they are when
the MMU is on for v7 and later CPUs). If that's the case, it's
possible the fix might be as simple as "recompile with the
-mno-unaligned-access option in CFLAGS".

-- PMM

Re: qemu-system-aarch64 hangs in a cortex-a72 test

[Liviu Ionescu]

I fixed the test to run on QEMU 9.2.4.

In case others encounter the same issue, here are some details.

The problem stemmed from incomplete initialization in the test — specifically, the MMU was disabled.

My experience with ARMv8-A is limited, but it appears that unaligned accesses are handled by the MMU. If the MMU is not enabled, the core can only perform aligned accesses; otherwise, it throws exceptions.

On the other hand, the default aarch64-none-elf-gcc compiler optimises memory copy operations using ldp/stp (load/store pair) instructions, such as:

```
 ldp  x0,x1,[x3]
 stp  x0,x1,[x4]
```

With the MMU enabled, these instructions work correctly regardless of the alignment of x3 and x4. However, if the MMU is disabled, both registers must be 16-byte aligned.

It is the user's responsibility to ensure proper alignment. Unfortunately, some structures in newlib do not meet this requirement.

In my case, the program crashed inside a printf() implementation used for trace functions:

```
namespace micro_os_plus::trace
{
 int
 printf (const char* format, ...)
 {
   std::va_list arguments;
   va_start (arguments, format);

   int ret = vprintf (format, arguments);

   va_end (arguments);
   return ret;
 }
}
```

The crash occurred when the `arguments` structure was copied to the stack for the `vprintf()` call, as its members were not 16-byte aligned. 

Since `std::va_list` is defined by the standard library, there's no way to adjust its layout.

The compiler provides an option to enforce alignment: `-fstrict-align`.

For AArch64, this option prevents the use of ldp/stp (16-byte) instructions and instead generates ldr/str (8-byte) instructions — twice as many, but alignment-safe.

I tested this and it worked: my code no longer crashed. However, it still failed later in the newlib implementation of `snprintf()`, called by `vprintf()`. The reason was the same — newlib uses ldp/stp and is not compiled with `-fstrict-align`.

I couldn't find a way to run the existing newlib with the MMU disabled. (The `-mno-unaligned-access` option is not available for aarch64-none-elf-gcc.)

The solution was to enable the MMU (bit 0 in sctlr_el1) and ensure that alignment fault checking is disabled (bit 1 in sctlr_el1).

In summary, for embedded AArch64 applications, I identified two viable approaches:

   • Compile all code with -fstrict-align, which effectively precludes the use of newlib.
   • Enable the MMU in transparent mode to allow unaligned accesses.

As always — and now more than ever — the rule stands: tests should only be run in emulated mode once they've been confirmed to work correctly on physical hardware.

Regards,

Liviu


> On 6 Oct 2025, at 13:45, Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> ... Note that depending on your compiler's config it might assume
> unaligned accesses are OK by default (because they are when
> the MMU is on for v7 and later CPUs). If that's the case, it's
> possible the fix might be as simple as "recompile with the
> -mno-unaligned-access option in CFLAGS".
> 
> -- PMM