QEMU Development Questions

QEMU Development Questions

[Randy Yates]

[-- Attachment #1: Type: text/plain, Size: 4562 bytes --]

1      Introduction



I came to the QEMU project with the goal of writing a new machine. We are reverse
engineering this machine and do not have the SoC reference manual.  We do have
the flash code, the bare metal startup code.  Thus we're having to guess and infer a
lot about the machine from the code.  Since this is an ARM A9, my initial thought
was to base the new machine on the vexpress-a9.


I've been asking lots of questions in #qemu@irc.oftc.net and some of the folks
there (mainly pm215 and f4bug) suggested I put these questions and my comments
into writing and send out to the mailing list to help new people become familiar
with the project.


I confess, I am ignorant.  Some of the questions may not be QEMU questions at all
but ARM architecture, hardware, or other questions.  Please bear with me and help
me out.


It is believed we are using an ARM Cortex A9 MPCORE machine with 4 cores and
with  TrustZone  (and  possibly  hypervisor)  functionality.   The  machine  eventually
loads and runs linux, but we believe there are at least three levels of bootloaders
prior to linux:



    1.  bootrom

    2.  first-level bootloader.

    3.  second-level bootloader via an embedded elf file.



It appears that only two of the four cores are used and it appears to be an asymmetric
multiprocessing design,  in which the two cores are running different OSes,  one a
bare metal and the other linux.


The goal is to define the minimum amount of machine (cpu, sram, dram, itcm, dtcm,
gic, flash, ?)  be able to reverse engineer code up to the second-level bootloader.
Then hopefully we can leverage QEMU's capabilities, e.g., exception reporting in
the debug output, to determine missing functionality.


The QEMU project was git cloned on 4/16/2020.  The work is being done under
Fedora 31.



                                                  1
\f



2      Questions



2.1      Top-Level  Architecture



There is a MachineClass, MachineState, and MachineInstance.  What are purposes
of these logical divisions?



2.2      Miscellaneous


    1.  Apparently the CPU has properties. What are the specific properties and their
        defaults?  How do you change a property's default value?

    2.  Ditto previous question for other machine components.

    3.  How do you model and implement startup from power on reset?


          (a) Bootrom code is modeled by specifying -bios !file? on the command line.
              What format should the code in !file? be?

         (b)  Bootrom code performs required initializations, e.g., populating the in-
              terrupt vector table at the appropriate spot, GIC? other registers?  Copy-
              ing code to DRAM?

          (c) Once bootrom code is finished, what happens?  Should it simply branch
              to the first-level bootloader (which is supposedly loaded into DRAM)?
              Is the reset vector used somehow?


    4.  I'm using the -pflash command-line option to map a copy of the flash code.
        Do you simply specify the address of an existing memory (DRAM) of where
        to load the file?  Is there a way to specify an entry point address (of where to
        begin execution).  How does this interact with the -bios option?

    5.  There may be custom hardware involved, e.g., GPIOs, I2Cs, etc.  How would
        these be defined in the machine?

    6.  Documentation for routing the monitor to another place (e.g., a telnet termi-
        nal) could be better.  pm215 gave me the following complex set of command-
        line options:


        -chardev  socket,id=monitor,host=127.0.0.1,port=4444,server,nowait,telnet  -mon  chardev=monitor,mode=readline


    7.  Ditto above comment for the -d option.

    8.  How are multiple cores traced in the debug output?

    9.  The vexpress code had clocks,  voltages,  etc.  defined.  Are these necessary
        for a bare-metal, base functioning machine?  Exactly which components are
        absolutely required?  E.g., is the busdev required?



                                                  2
\f



  10.   Terminology questions:

          (a) Throughout  there  is  mention  of  highmem/lowmem.   What  does  this
              mean?

         (b)  Throughout  there  are  "PL"  devices,  e.g.,  PL111.   What  does  "PL"
              mean?



3      Conclusion



That's all for now - thanks for all help/suggestions/pointers/etc.



                                                  3
Randy Yates, DSP/Embedded Firmware Developer
Digital Signal Labs
http://www.digitalsignallabs.com

[-- Attachment #2: Type: text/html, Size: 6804 bytes --]

Re: QEMU Development Questions

[Peter Maydell]

On Fri, 17 Apr 2020 at 09:07, Randy Yates <yates@digitalsignallabs.com> wrote:

> 2.1  Top-Level Architecture
>
> There is a MachineClass, MachineState, and MachineInstance. What are purposes of these logical divisions?

class vs state is QEMU's usual object model (QOM) structs:
there is one instance of a class struct which contains
things like method pointers; each instance of the object
as a FooState struct which has the varies-per-instance
data. The doc comments in include/qom/object.h are probably
a good place to start for a summary of the object model.

'git grep MachineInstance' has no matches, so I don't know
what you have in mind there.

> 2.2  Miscellaneous
>
> Apparently the CPU has properties. What are the specific properties and their defaults? How do you change a property's default value?

Mostly undocumented; nominally introspectable. In practice,
just look at what other source code is doing. For some
devices the .h file includes a comment describing this
kind of thing, but often not.

> How do you model and implement startup from power on reset?

Not sure what you have in mind here. It just works: the
CPU starts running from the reset address.

> Bootrom code is modeled by specifying -bios <file> on the
> command line. What format should the code in <file> be?

This depends on what your board model does with the file.
If your board code doesn't do anything with the filename
in the 'bios_name' global, then the command line option
will be ignored entirely. If your board code passes the
filename to a "load this ELF file" function, then the file
has to be an ELF file. If your board code passes it to
a "load this raw data file", then the file has to be a raw
data file. Raw-data is the most common approach.

> Bootrom code performs required initializations, e.g.,
> populating the interrupt vector table at the appropriate
> spot, GIC? other registers? Copying code to DRAM?

Your guest code can do what it likes. What you need
to do is make sure you've loaded an image which provides
the initial reset vector table (usually at address 0
for Arm). QEMU will start executing from the reset vector
in the same way real h/w does, and your guest code
takes it from there.

> Once bootrom code is finished, what happens? Should it simply
> branch to the first-level bootloader (which is supposedly
> loaded into DRAM)? Is the reset vector used somehow?

You seem to be making a distinction between different
levels of boot code (bootrom, first level bootloader)
which QEMU simply does not make. QEMU provides two
possible models for loading Arm guest code:
 (1) boot this thing like a linux kernel with the
     CPU set up as the linux kernel's boot protocol
     specifies
 (2) guest code gets control of the emulated machine
     exactly as it comes out of reset

So you need to provide some guest code that does the
equivalent of what you're calling "bootrom". If you
want that to pass control to a "first level bootloader"
then you need to write the bootrom guest code to do that.

> I'm using the -pflash command-line option to map a copy of the flash code. Do you simply specify the address of an existing memory (DRAM) of where to load the file? Is there a way to specify an entry point address (of where to begin execution). How does this interact with the -bios option?

See above: you don't get to control where execution starts,
because for Arm CPUs (unless your board uses hivecs)
that's always 0. Your board code wants to load the initial
guest code into the right place, which should be where
the flash device/rom/whatever is, which ought to be address 0.

Board code gets to decide whether it pays any attention to
-bios or -pflash at all. You can ignore one and just implement
the other if you like, or you can support both, which is
what vexpress does.

> There may be custom hardware involved, e.g., GPIOs, I2Cs, etc. How would these be defined in the machine?

Your best reference for this is to look at existing
device and board model sources.

> Ditto above comment for the -d option.
>
> How are multiple cores traced in the debug output?

In the 'exec' tracing, "Trace 0:" means trace for core 0,
"Trace 1:" core 1, etc. Some tracing is independent of
cores (like in_asm). Some tracing doesn't specify (like
'cpu') but it's obvious from its position relative to
the 'exec' trace lines.

> The vexpress code had clocks, voltages, etc. defined. Are these necessary for a bare-metal, base functioning machine? Exactly which components are absolutely required? E.g., is the busdev required?

You want a minimal machine which provides RAM,
the CPU, the interrupt controller, and maybe flash
(though you could just stick the initial boot
code into ram to start with if you don't happen
to know what the flash device is on your board).
vexpress provides a lot of stuff that's specific
to vexpress. Don't look just at vexpress : look
also at other board models. You probably want things
that most or every board model does; you probably don't
want things that only the vexpress code is doing.

> Terminology questions:
>
> Throughout there is mention of highmem/lowmem. What does this mean?

This is vexpress specific -- this board happens to
have memory in two different regions of its address
space; they are for convenience named 'highmem' and
'lowmem'.

> Throughout there are "PL" devices, e.g., PL111. What does "PL" mean?

These are Arm devices commonly used on Arm's own devboards
but very rarely found elsewhere (eg a PL011 is a UART,
a PL111 is a graphics controller). Almost certainly your
system's devices will be completely different and you won't
be able to reuse any of the PLxxx device models.

thanks
-- PMM

Re: QEMU Development Questions

[BALATON Zoltan]

On Thu, 16 Apr 2020, Randy Yates wrote:
>    1.  Apparently the CPU has properties. What are the specific properties and their
>        defaults?  How do you change a property's default value?

Maybe look for DEFINE_PROP_ to find out, For example:

qemu/target/arm/cpu.c:2757:
static Property arm_cpu_properties[] = {
     DEFINE_PROP_BOOL("start-powered-off", ARMCPU, start_powered_off, false),
     DEFINE_PROP_UINT32("psci-conduit", ARMCPU, psci_conduit, 0),
     DEFINE_PROP_UINT32("midr", ARMCPU, midr, 0),
     DEFINE_PROP_UINT64("mp-affinity", ARMCPU,
                         mp_affinity, ARM64_AFFINITY_INVALID),
     DEFINE_PROP_INT32("node-id", ARMCPU, node_id, CPU_UNSET_NUMA_NODE_ID),
     DEFINE_PROP_INT32("core-count", ARMCPU, core_count, -1),
     DEFINE_PROP_END_OF_LIST()
};

then qemu/hw/arm/bcm2836.c:138:
         /* start powered off if not enabled */
         object_property_set_bool(OBJECT(&s->cpu[n].core), n >= s->enabled_cpus,
                                  "start-powered-off", &err);


There's probably also a way to query and set these from monitor, info 
qtree command lists properties of instantiated devices I think. There's 
also some device_add option to get info on other devices but I don't know 
how that works. Maybe help device_add in monitor tells you.

>    2.  Ditto previous question for other machine components.

Same as above, but look for property definition in device model under 
qemu/hw.

>    5.  There may be custom hardware involved, e.g., GPIOs, I2Cs, etc.  How would
>        these be defined in the machine?

You need to find the appropriate device model or write it if does not 
exist then set it up in the board code to appear at the right addresses 
(or attach to right bus e.g. for i2c devices). The info mtree monitor 
command is useful to see what's mapped where and you can compare it to 
memory map of real hardware. For writing device models look at exising 
similar examples. Also enabling #define DEBUG_UNASSIGNED in qemu/memory.c 
is useful to get logs about guest attempting to access missing devices 
(although logged address can be relative to a memory region so may not be 
actual memory address in some cases). There's also a placeholder 
unimplemented device model that you can map over known unimplemented 
areas before you actually implement them, see for example 
hw/arm/xlnx-versal.c for how it's used.

>    6.  Documentation for routing the monitor to another place (e.g., a telnet termi-
>        nal) could be better.  pm215 gave me the following complex set of command-
>        line options:
>
>
>        -chardev  socket,id=monitor,host=127.0.0.1,port=4444,server,nowait,telnet  -mon  chardev=monitor,mode=readline

This works, for telnet I'm afraid that's the simplest way. There's 
-monitor stdio or -serial stdio that can be useful.

>    7.  Ditto above comment for the -d option.

I think there's -D logfile to specify log file, these are somewhat 
documented in --help.

Regards,
BALATON Zoltan

Re: QEMU Development Questions

[Randy Yates]

[-- Attachment #1: Type: text/plain, Size: 309 bytes --]

Thanks for the answers I've received so far - much appreciated.


Is there a standard way to define your own custom coprocessor CP15 registers?


----Randy



                                                  1
Randy Yates, DSP/Embedded Firmware Developer
Digital Signal Labs
http://www.digitalsignallabs.com

[-- Attachment #2: Type: text/html, Size: 1196 bytes --]

Re: QEMU Development Questions

[Peter Maydell]

On Fri, 17 Apr 2020 at 16:50, Randy Yates <yates@digitalsignallabs.com> wrote:
>
> Thanks for the answers I've received so far - much appreciated.
>
> Is there a standard way to define your own custom coprocessor CP15 registers?

It can be done but you very likely don't want to do it...

thanks
-- PMM