EDK2 ArmVirtQemu behaviour with multiple UARTs

EDK2 ArmVirtQemu behaviour with multiple UARTs

[Peter Maydell]

Hi; I've been looking again at a very long standing missing feature in
the QEMU virt board, which is that we only have one UART. One of the
things that has stalled this in the past has been the odd behaviour of
EDK2 if the DTB that QEMU passes it describes two UARTs.

I'm going to describe the behaviour I see in more detail below, but to
put the summary up front:
 * EDK2 puts some debug output on one UART and some on the other
   (the exact arrangement depends on ordering of the dtb nodes)
 * EDK2 doesn't look at either stdout-path or the serial* aliases,
   so its choices about how to use the UARTs differ from those
   made by the guest kernel it is booting (and it also seems to be
   iterating through the dtb in the opposite order to the kernel)

The current proposal for adding a second UART is that it only happens
if you explicitly add one on the command line (with a second "-serial
something" option), so whatever we do won't break existing user
setups. So we have scope for saying "if you want to use a second UART,
you're going to want a newer EDK2 which handles it better". Exactly
what "better" means here is up for grabs, but honouring stdout-path
and the serial aliases would be the ideal I think. It would also be
possible to select a particular ordering for the DTB nodes to produce
"least-worst" behaviour from an existing EDK2 binary, but I'm not
sure if that's worth doing.

What do the EDK2 folks think about what the correct behaviour
should be for a 2-UART setup?

Anyway, on to the details about the setup and what I see from EDK2:

This is all with a debug ArmVirtQemu build, running at EL2 (i.e.
entirely non-secure), with some patches I've been working on to add
the extra UART to the board and the DTB. The DTB has the two UARTs:

        pl011@9000000 {
                clock-names = "uartclk\0apb_pclk";
                clocks = <0x8000 0x8000>;
                interrupts = <0x00 0x01 0x04>;
                reg = <0x00 0x9000000 0x00 0x1000>;
                compatible = "arm,pl011\0arm,primecell";
        };

        pl011@9040000 {
                clock-names = "uartclk\0apb_pclk";
                clocks = <0x8000 0x8000>;
                interrupts = <0x00 0x08 0x04>;
                reg = <0x00 0x9040000 0x00 0x1000>;
                compatible = "arm,pl011\0arm,primecell";
        };

and aliases:

        aliases {
                serial0 = "/pl011@9000000";
                serial1 = "/pl011@9040000";
        };

and in the /chosen node:
                stdout-path = "/pl011@9000000";

The ACPI table fragments generated by QEMU have entries for both
UARTs, as COM0 and COM1.

Given all this, EDK2 outputs:

uart0:
 * some UEFI output including debug output, starting:
UEFI firmware (version  built at 15:19:20 on Sep 19 2023)
add-symbol-file
/home/petmay01/linaro/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll
0x2000
add-symbol-file
/home/petmay01/linaro/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Pei/PeiMain/DEBUG/PeiCore.dll
0xD240
Register PPI Notify: DCD0BE23-9586-40F4-B643-06522CED4EDE
Install PPI: 8C8CE578-8A3D-4F1C-9935-896185C32DD3
Install PPI: 5473C07A-3DCB-4DCA-BD6F-1E9689E7349A
The 0th FV start address is 0x00000001000, size is 0x001FF000, handle is 0x1000

 * the guest Linux kernel output (on what Linux says is ttyAMA0)

uart1:
 * a lot of UEFI output including debug output, starting:
DxeMain: MemoryBaseAddress=0x48000000 MemoryLength=0x38000000
add-symbol-file
/home/petmay01/linaro/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
0x47860000
HOBLIST address in DXE = 0x7FA35018
Memory Allocation 0x00000004 0x47FFF000 - 0x47FFFFFF
Memory Allocation 0x00000004 0x47FFE000 - 0x47FFEFFF

 * the GNU GRUB OS select screen and other GRUB output

The full output dumps can be seen at:
https://people.linaro.org/~peter.maydell/uart0.txt
https://people.linaro.org/~peter.maydell/uart1.txt

With only 1 UART, all the above appears on the single UART:
https://people.linaro.org/~peter.maydell/uart-single.txt

If I change QEMU to reverse the order of the nodes in the DTB (so the
pl011@9040000 nodes is listed first in the dtc output, and
pl011@900000 is listed second), then EDK2's output changes: the debug
output previously on uart0 is now on uart1, and vice-versa. The GRUB
output also switches to uart0. The Linux kernel output remains on
uart0 (this makes sense, because Linux is looking at the ACPI tables,
which are generated independently from the dtb). Output for this
setup is here:

https://people.linaro.org/~peter.maydell/uart0-rev.txt
https://people.linaro.org/~peter.maydell/uart1-rev.txt

A direct boot of Linux doesn't care about the dtb node ordering -- it
honours the aliases node and the chosen stdout-path string.
(Without the 'aliases' node, only the "pl011@9000000 first" dtb
order works, because it assigns ttyAMA0 and ttyAMA1 in the same
order as dtc prints them in the dtb disassembly.)

I would be happier if I understood why putting the nodes in reverse
order works, given that the code in EDK2 seems to be iterating through
the dtb forwards. I know there is at least one place in QEMU where the
node ordering gets reversed in the process of writing out the dtb, so
maybe there are more depending on how exactly the dtb is read. That
would I suppose explain why some EDK2 debug output goes to one UART
and some to the other, if the dtb read process differs during different
phases of EDK2 boot.

If you want to play around with this, I have some WIP patches at
https://git.linaro.org/people/pmaydell/qemu-arm.git uart-edk-investigation
(content wise they should be fine, but I haven't cleaned them up into
a coherent set of distinct patches yet, so they're a bit messy.)
A run of QEMU with both UARTs which sends all output to files looks like:

./build/arm-clang/qemu-system-aarch64 -display none -vga none \
  -machine virt,acpi=on,virtualization=on,mte=on,gic-version=max,iommu=smmuv3 \
  -smp 2 -m 1024 -cpu max,pauth-impdef=on \
  -bios ~/linaro/edk2/QEMU_EFI_DEBUG.fd \
  -drive file=/home/petmay01/avocado/data/cache/by_location/0154b7cd3a4f5e135299060c8cabbeec10b70b6d/alpine-standard-3.17.2-aarch64.iso,format=raw
\
  -device virtio-rng-pci,rng=rng0 \
  -object rng-random,id=rng0,filename=/dev/urandom \
  -chardev file,id=chr0,path=/tmp/uart0-rev.txt \
  -chardev file,id=chr1,path=/tmp/uart1-rev.txt \
  -serial chardev:chr0 -serial chardev:chr1

(adjust -serial options to taste)

thanks
-- PMM

Re: EDK2 ArmVirtQemu behaviour with multiple UARTs

[Ard Biesheuvel]

On Thu, 21 Sept 2023 at 10:50, Peter Maydell <peter.maydell@linaro.org> wrote:
>
> Hi; I've been looking again at a very long standing missing feature in
> the QEMU virt board, which is that we only have one UART. One of the
> things that has stalled this in the past has been the odd behaviour of
> EDK2 if the DTB that QEMU passes it describes two UARTs.
>
> I'm going to describe the behaviour I see in more detail below, but to
> put the summary up front:
>  * EDK2 puts some debug output on one UART and some on the other
>    (the exact arrangement depends on ordering of the dtb nodes)
>  * EDK2 doesn't look at either stdout-path or the serial* aliases,
>    so its choices about how to use the UARTs differ from those
>    made by the guest kernel it is booting (and it also seems to be
>    iterating through the dtb in the opposite order to the kernel)
>
> The current proposal for adding a second UART is that it only happens
> if you explicitly add one on the command line (with a second "-serial
> something" option), so whatever we do won't break existing user
> setups. So we have scope for saying "if you want to use a second UART,
> you're going to want a newer EDK2 which handles it better". Exactly
> what "better" means here is up for grabs, but honouring stdout-path
> and the serial aliases would be the ideal I think. It would also be
> possible to select a particular ordering for the DTB nodes to produce
> "least-worst" behaviour from an existing EDK2 binary, but I'm not
> sure if that's worth doing.
>
> What do the EDK2 folks think about what the correct behaviour
> should be for a 2-UART setup?
>

Hi Peter,

Thanks for the elaborate analysis.

EDK2's DEBUG output is extremely noisy, so being able to redirect this
output to a different UART would be very useful.

The stdout-path is the intended console, and so we should honour that.
This also means that we should parse aliases. But the console is
actually configurable [persistenly] via the UEFI menu, and so it would
be nice if we could take advantage of this flexibility. This means in
principle that the UARTs should be represented via different device
paths (which would include the base address so they are
distinguishable) with perhaps a magical alias which is the default and
is tied to whatever stdout-path points to. This way, all the logic we
introduce is spec compliant and reusable on physical platforms with
multiple UARTs.

The DEBUG output is a different matter. On physical hardware, this is
typically configured at build time, as the info is needed extremely
early and on a physical platform, the debug port generally doesn't
change. Currently, we just grab the first UART that we encounter in
the DT, but the logic used by the DEBUG code and the ordinary console
driver are mostly separate.

What we might do is use stdout-path as well, unless a certain DT alias
exist perhaps? We should probably align here with other projects,
although this a distinction of the same nature may not exist there.

-- 
Ard.

Re: EDK2 ArmVirtQemu behaviour with multiple UARTs

[Gerd Hoffmann]

On Thu, Sep 21, 2023 at 11:50:20AM +0100, Peter Maydell wrote:
> Hi; I've been looking again at a very long standing missing feature in
> the QEMU virt board, which is that we only have one UART. One of the
> things that has stalled this in the past has been the odd behaviour of
> EDK2 if the DTB that QEMU passes it describes two UARTs.

Note that edk2 recently got support for virtio-serial, so you can use
that for the console and leave the uart for debug logging.  The prebuild
edk2 binaries in qemu have been updated days ago and these already
support for virtio-serial..

take care,
  Gerd

Re: EDK2 ArmVirtQemu behaviour with multiple UARTs

[Peter Maydell]

On Thu, 21 Sept 2023 at 16:26, Gerd Hoffmann <kraxel@redhat.com> wrote:
>
> On Thu, Sep 21, 2023 at 11:50:20AM +0100, Peter Maydell wrote:
> > Hi; I've been looking again at a very long standing missing feature in
> > the QEMU virt board, which is that we only have one UART. One of the
> > things that has stalled this in the past has been the odd behaviour of
> > EDK2 if the DTB that QEMU passes it describes two UARTs.
>
> Note that edk2 recently got support for virtio-serial, so you can use
> that for the console and leave the uart for debug logging.  The prebuild
> edk2 binaries in qemu have been updated days ago and these already
> support for virtio-serial..

As long as EDK2 does something sensible when the DTB says "two
UARTs here and here" and it also finds a virtio-serial PCI
device, I don't mind what exactly it does. The problem here is
more that EDK2 currently does strange things when told that
the hardware is present, rather than that anybody specifically wants
EDK2 to use multiple serial outputs.

Though given there's no way to say in the DTB "use a PCI card
for your console" I think the virtio-serial approach is likely
to be awkward for users in practice.

-- PMM

Re: EDK2 ArmVirtQemu behaviour with multiple UARTs

[Gerd Hoffmann]

On Thu, Sep 21, 2023 at 04:34:27PM +0100, Peter Maydell wrote:
> As long as EDK2 does something sensible when the DTB says "two
> UARTs here and here" and it also finds a virtio-serial PCI
> device, I don't mind what exactly it does. The problem here is
> more that EDK2 currently does strange things when told that
> the hardware is present, rather than that anybody specifically wants
> EDK2 to use multiple serial outputs.
> 
> Though given there's no way to say in the DTB "use a PCI card
> for your console" I think the virtio-serial approach is likely
> to be awkward for users in practice.

edk2 adds a virtio console to the edk2 console multiplexer if
present (for both pci and mmio virtio transports), and systemd
spawns also spawns a getty on /dev/hvc0 if present.  So this
works mostly automatic.  Only if you also want the linux boot
messages show up there too you need to add 'console=hvc0' to
your kernel command line.

take care,
  Gerd