[Qemu-devel] Simulating a composite machine

[Qemu-devel] Simulating a composite machine

[Martin Schroeder]

Is it possible to instantiate multiple CPUs of different architectures
and simuate them with different images at the same time? Some examples
include ARM socs with m3/m4 coprocessor core but also boards with
multiple processors where it is desirable to connect the chips over
for example virtual SPI or UART and then simulate the composite system
as a single machine where each of the cores runs a separate firmware.

Is something like this easy to implement given current processor
objects or does this require substantial changes to how qemu works?
One area I do not fully understand is native code generator and
whether it would be able to cope with two cores of *different*
architectures at the same time.

Re: [Qemu-devel] Simulating a composite machine

[Peter Maydell]

On 19 August 2018 at 13:54, Martin Schroeder via Qemu-devel
<qemu-devel@nongnu.org> wrote:
> Is it possible to instantiate multiple CPUs of different architectures
> and simuate them with different images at the same time? Some examples
> include ARM socs with m3/m4 coprocessor core but also boards with
> multiple processors where it is desirable to connect the chips over
> for example virtual SPI or UART and then simulate the composite system
> as a single machine where each of the cores runs a separate firmware.

Not currently, no. There's some out of tree stuff various people
have done involving connecting up separate QEMU processes.

> Is something like this easy to implement given current processor
> objects or does this require substantial changes to how qemu works?
> One area I do not fully understand is native code generator and
> whether it would be able to cope with two cores of *different*
> architectures at the same time.

At the moment some bits of our core code assume all the CPUs
in the system are basically identical (shared code cache, etc).
I'm planning to do some work to fix the simpler parts of this,
so you can have two different CPUs of the same architecture
in a system (eg a Cortex-M4 with an FPU plus one without an FPU,
or an M3 and an A-class core). Multiple completely different
architectures (eg Microblaze + ARM, or ARM + PPC) is rather
harder, as at the moment we build entirely separate
qemu-system-* binaries for each architecture and there are
some compile-time assumptions made. I'd like to see us work
towards making that possible, but there's potentially quite
a bit of effort required.

thanks
-- PMM

Re: [Qemu-devel] Simulating a composite machine

[Martin Schroeder]

Sent it off as reply instead of reply all.. my bad.

I was also surprised that it was mid into an instruction. The code
works fine on STM32 but if FPU is not enabled on the STM32 then I
believe the code will fault in much the same way. I have had this
problem before and it was also hard faulting inside libc snprintf. It
has definitely something to do with FPU and snprintf faults when fpu
is not enabled on F4. When FPU is enabled everything works. Why I
don't know.

It seems the HardFault is preceded by two UsageFaults:
Taking exception 17 [v7M NOCP UsageFault]
...taking pending nonsecure exception 3
Taking exception 17 [v7M NOCP UsageFault]
qemu: fatal: Lockup: can't escalate 3 to HardFault (current priority -1)

Ok so FPU support is not there. What would it take for me to add it? I
would very much like to add it if I get some guidance as to what needs
to be done and where I can find information on how to do it. I see
there is some code in fpu/softfloat.c but I guess it is not enough?
On Mon, Aug 20, 2018 at 8:12 PM Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On 20 August 2018 at 19:03, Martin Schroeder
> <mkschreder.uk@googlemail.com> wrote:
> > Peter, perhaps a slightly off-topic question,
>
> Could you keep traffic on the mailing lists, please? That
> way everybody can read the answer and reply if they have a view.
>
> (I append the rest because I'd typed it all out before I noticed
> that this was a private email.)
>
> > but how do I enable FPU
> > support for armv7m M4 emulation? Is it supported? FPU instructions
> > currently make the armv7m core go into hard fault on master.
>
> It's not supported, because FPU support requires changes to
> the M profile exception handling (stacking FPU regs on exception
> entry, handling the lazy-stacking feature, etc). This is on my
> todo list, so there is a reasonable chance it'll get done either
> for QEMU 3.1 or more likely for 3.2, if nobody else gets to it first.
>
> > If I disable FPU in firmware build and use -mfloat-abi=soft then I get
> > a hard fault even sooner with following warning: "warning: TCG
> > temporary leaks before 0805875a"
>
> The warning is harmless -- I think the only remaining
> encodings that can cause it to be produced are all ones
> where the instruction in question generates an exception
> (it's caused because we generate a bit of code before
> realising the insn is going to fault, and then the TCG
> temp is not cleaned up by the per-insn translation code.
> The temp is cleaned up at the end of the TB, so it's not
> a permanent leak -- the warning is there because per-insn
> leaks let the guest run us out of temps by having a TB with
> a lot of the same insn in it.)
>
> > (gdb) disas 0x0805875a
> > Dump of assembler code for function snprintf:
> >    0x08058750 <+0>:     push    {r2, r3}
> >    0x08058754 <+4>:     push    {r4, r5, r6, lr}
> >    0x08058758 <+8>:     ldr     r3, [pc, #136]  ; 0x80587e8 <snprintf+152>
> >    0x0805875c <+12>:    subs    r4, r1, #0
> >    0x08058760 <+16>:    ldr     r5, [r3]
> >    0x08058764 <+20>:    movlt   r3, #139        ; 0x8b
> >    0x08058768 <+24>:    sub     sp, sp, #112    ; 0x70
> >    0x0805876c <+28>:    strlt   r3, [r5]
> >    0x08058770 <+32>:    mvnlt   r0, #0
> >    0x08058774 <+36>:    blt     0x80587d8 <snprintf+136>
> >    0x08058778 <+40>:    add     r12, sp, #132   ; 0x84
> >    0x0805877c <+44>:    mov     r3, #520        ; 0x208
> >    0x08058780 <+48>:    subne   lr, r4, #1
> >    0x08058784 <+52>:    moveq   lr, r4
> >    0x08058788 <+56>:    ldr     r2, [sp, #128]  ; 0x80
>
> Not clear from that why you're getting a hard fault, but it
> seems a bit suspicious that the quoted PC is midway through an
> insn -- suggests that maybe the code being executed is not
> what your disassembly says it is?
>
> You might find that the -d debug logs are helpful in indicating
> what's happened.
>
> thanks
> -- PMM

Re: [Qemu-devel] Simulating a composite machine

[Martin Schroeder]

Is it that I need to add emulation for CP10 and CP11?
On Mon, Aug 20, 2018 at 8:29 PM Martin Schroeder
<mkschreder.uk@googlemail.com> wrote:
>
> Sent it off as reply instead of reply all.. my bad.
>
> I was also surprised that it was mid into an instruction. The code
> works fine on STM32 but if FPU is not enabled on the STM32 then I
> believe the code will fault in much the same way. I have had this
> problem before and it was also hard faulting inside libc snprintf. It
> has definitely something to do with FPU and snprintf faults when fpu
> is not enabled on F4. When FPU is enabled everything works. Why I
> don't know.
>
> It seems the HardFault is preceded by two UsageFaults:
> Taking exception 17 [v7M NOCP UsageFault]
> ...taking pending nonsecure exception 3
> Taking exception 17 [v7M NOCP UsageFault]
> qemu: fatal: Lockup: can't escalate 3 to HardFault (current priority -1)
>
> Ok so FPU support is not there. What would it take for me to add it? I
> would very much like to add it if I get some guidance as to what needs
> to be done and where I can find information on how to do it. I see
> there is some code in fpu/softfloat.c but I guess it is not enough?
> On Mon, Aug 20, 2018 at 8:12 PM Peter Maydell <peter.maydell@linaro.org> wrote:
> >
> > On 20 August 2018 at 19:03, Martin Schroeder
> > <mkschreder.uk@googlemail.com> wrote:
> > > Peter, perhaps a slightly off-topic question,
> >
> > Could you keep traffic on the mailing lists, please? That
> > way everybody can read the answer and reply if they have a view.
> >
> > (I append the rest because I'd typed it all out before I noticed
> > that this was a private email.)
> >
> > > but how do I enable FPU
> > > support for armv7m M4 emulation? Is it supported? FPU instructions
> > > currently make the armv7m core go into hard fault on master.
> >
> > It's not supported, because FPU support requires changes to
> > the M profile exception handling (stacking FPU regs on exception
> > entry, handling the lazy-stacking feature, etc). This is on my
> > todo list, so there is a reasonable chance it'll get done either
> > for QEMU 3.1 or more likely for 3.2, if nobody else gets to it first.
> >
> > > If I disable FPU in firmware build and use -mfloat-abi=soft then I get
> > > a hard fault even sooner with following warning: "warning: TCG
> > > temporary leaks before 0805875a"
> >
> > The warning is harmless -- I think the only remaining
> > encodings that can cause it to be produced are all ones
> > where the instruction in question generates an exception
> > (it's caused because we generate a bit of code before
> > realising the insn is going to fault, and then the TCG
> > temp is not cleaned up by the per-insn translation code.
> > The temp is cleaned up at the end of the TB, so it's not
> > a permanent leak -- the warning is there because per-insn
> > leaks let the guest run us out of temps by having a TB with
> > a lot of the same insn in it.)
> >
> > > (gdb) disas 0x0805875a
> > > Dump of assembler code for function snprintf:
> > >    0x08058750 <+0>:     push    {r2, r3}
> > >    0x08058754 <+4>:     push    {r4, r5, r6, lr}
> > >    0x08058758 <+8>:     ldr     r3, [pc, #136]  ; 0x80587e8 <snprintf+152>
> > >    0x0805875c <+12>:    subs    r4, r1, #0
> > >    0x08058760 <+16>:    ldr     r5, [r3]
> > >    0x08058764 <+20>:    movlt   r3, #139        ; 0x8b
> > >    0x08058768 <+24>:    sub     sp, sp, #112    ; 0x70
> > >    0x0805876c <+28>:    strlt   r3, [r5]
> > >    0x08058770 <+32>:    mvnlt   r0, #0
> > >    0x08058774 <+36>:    blt     0x80587d8 <snprintf+136>
> > >    0x08058778 <+40>:    add     r12, sp, #132   ; 0x84
> > >    0x0805877c <+44>:    mov     r3, #520        ; 0x208
> > >    0x08058780 <+48>:    subne   lr, r4, #1
> > >    0x08058784 <+52>:    moveq   lr, r4
> > >    0x08058788 <+56>:    ldr     r2, [sp, #128]  ; 0x80
> >
> > Not clear from that why you're getting a hard fault, but it
> > seems a bit suspicious that the quoted PC is midway through an
> > insn -- suggests that maybe the code being executed is not
> > what your disassembly says it is?
> >
> > You might find that the -d debug logs are helpful in indicating
> > what's happened.
> >
> > thanks
> > -- PMM

Re: [Qemu-devel] Simulating a composite machine

[Peter Maydell]

On 20 August 2018 at 19:29, Martin Schroeder
<mkschreder.uk@googlemail.com> wrote:
> Ok so FPU support is not there. What would it take for me to add it? I
> would very much like to add it if I get some guidance as to what needs
> to be done and where I can find information on how to do it. I see
> there is some code in fpu/softfloat.c but I guess it is not enough?

As I say, it's mostly the exception handling parts (and also some
things like M profile having memory mapped registers for the FPU
ID registers). The instructions themselves are the same as the
A-profile FPU.

My summary breakdown of this task looks like:
 * check which instructions we need to enable and which parts
   of the current FP implementation are A/R specific and need
   "not if M" conditionals
   (this is going to involve some comparison-of-specifications
   between the M-profile architecture spec manual and the A-profile
   one. The answer is probably going to wind up being "the
   A-profile-style FPU status and control register accesses
   need to be disabled", but there might be more. You can assume
   that the actual behaviour of each instruction is the same, though.)
 * implement the status and ID registers that M profile wants
   (mostly wiring up register accessors in hw/intc/armv7m_nvic.c
   to existing CPU state fields, I think)
 * exception model changes for FP (new exception frame layout, etc)
   -- this is all code in target/arm/helper.c which now needs to
   handle the possibility of an FPU. The pseudocode in the ARMv7M
   and v8M manuals will help in identifying where changes need
   to be made.
 * make sure co-processor enable/disable is wired up right
   (the coprocessor access is handled via a memory-mapped register)
 * implement lazy saving of FP registers (complicated; when the
   CPU status registers indicate that lazy-state-preservation is
   in effect, we need to arrange that attempting to execute an
   FP insn causes us to first stop and save the FPU state to the
   stack, and then do the instruction. The best way to do this is
   going to be to add a TB flag for "lazy state preservation required",
   and if that flag is set then when we encounter an FP insn we
   generate code for "raise a lazy-stacking exception". We can
   then deal with the exception by "do the lazy-stacking, then
   resume execution".
 * turn this on for Cortex-M4, Cortex-M33 and test

thanks
-- PMM