[Qemu-devel] Multi heterogenous CPU archs for SoC sim?

[Qemu-devel] Multi heterogenous CPU archs for SoC sim?

[Greg McGary]

I would like to create a QEMU model of an SoC that has several CPU cores having different architectures.  I'm guessing this can be done.  Has anyone has thought much about this, and/or have advice?  FYI, I am reasonably experienced with QEMU--I ported it to a new proprietary generic-RISC architecture last year and have it running in linux-user mode and in system mode running a newly-ported linux kernel.

G

Re: [Qemu-devel] Multi heterogenous CPU archs for SoC sim?

[Peter Maydell]

On 25 May 2011 19:44, Greg McGary <greg.mcgary@gmail.com> wrote:
> I would like to create a QEMU model of an SoC that has several
> CPU cores having different architectures.  I'm guessing this
> can be done.

It's not supported currently as far as I'm aware. There was
at least one paper at the QEMU Forum earlier this year describing
an approach to multi-CPU environments (embedding QEMU into a
SystemC world) that basically saved and restored all QEMU's
global variables every time it switched cores...

It would be good if it was supported in QEMU proper, but I
suspect you may be in for some large-scale restructuring work.

-- PMM

Re: [Qemu-devel] Multi heterogenous CPU archs for SoC sim?

[Blue Swirl]

On Wed, May 25, 2011 at 10:20 PM, Peter Maydell
<peter.maydell@linaro.org> wrote:
> On 25 May 2011 19:44, Greg McGary <greg.mcgary@gmail.com> wrote:
>> I would like to create a QEMU model of an SoC that has several
>> CPU cores having different architectures.  I'm guessing this
>> can be done.
>
> It's not supported currently as far as I'm aware. There was
> at least one paper at the QEMU Forum earlier this year describing
> an approach to multi-CPU environments (embedding QEMU into a
> SystemC world) that basically saved and restored all QEMU's
> global variables every time it switched cores...
>
> It would be good if it was supported in QEMU proper, but I
> suspect you may be in for some large-scale restructuring work.

One of the long standing goals for QEMU has been to be able to use a
single executable to emulate multiple architectures. I think for
example the lines like
#define cpu_init cpu_sparc_init
#define cpu_exec cpu_sparc_exec
etc. stand for this purpose, so there has been some consideration for this.

Re: [Qemu-devel] Multi heterogenous CPU archs for SoC sim?

[Lluís]

Blue Swirl writes:

> On Wed, May 25, 2011 at 10:20 PM, Peter Maydell
> <peter.maydell@linaro.org> wrote:
>> On 25 May 2011 19:44, Greg McGary <greg.mcgary@gmail.com> wrote:
>>> I would like to create a QEMU model of an SoC that has several
>>> CPU cores having different architectures.  I'm guessing this
>>> can be done.
>> 
>> It's not supported currently as far as I'm aware. There was
>> at least one paper at the QEMU Forum earlier this year describing
>> an approach to multi-CPU environments (embedding QEMU into a
>> SystemC world) that basically saved and restored all QEMU's
>> global variables every time it switched cores...
>> 
>> It would be good if it was supported in QEMU proper, but I
>> suspect you may be in for some large-scale restructuring work.

> One of the long standing goals for QEMU has been to be able to use a
> single executable to emulate multiple architectures. I think for
> example the lines like
> #define cpu_init cpu_sparc_init
> #define cpu_exec cpu_sparc_exec
> etc. stand for this purpose, so there has been some consideration for this.

Nicely handling per-arch functions would be one of the benefits of using
C++ in QEMU (I know, it's sufficient but not necessary). What were the
conclusions regarding such a change?


Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: [Qemu-devel] Multi heterogenous CPU archs for SoC sim?

[Blue Swirl]

On Thu, May 26, 2011 at 11:07 PM, Lluís <xscript@gmx.net> wrote:
> Blue Swirl writes:
>
>> On Wed, May 25, 2011 at 10:20 PM, Peter Maydell
>> <peter.maydell@linaro.org> wrote:
>>> On 25 May 2011 19:44, Greg McGary <greg.mcgary@gmail.com> wrote:
>>>> I would like to create a QEMU model of an SoC that has several
>>>> CPU cores having different architectures.  I'm guessing this
>>>> can be done.
>>>
>>> It's not supported currently as far as I'm aware. There was
>>> at least one paper at the QEMU Forum earlier this year describing
>>> an approach to multi-CPU environments (embedding QEMU into a
>>> SystemC world) that basically saved and restored all QEMU's
>>> global variables every time it switched cores...
>>>
>>> It would be good if it was supported in QEMU proper, but I
>>> suspect you may be in for some large-scale restructuring work.
>
>> One of the long standing goals for QEMU has been to be able to use a
>> single executable to emulate multiple architectures. I think for
>> example the lines like
>> #define cpu_init cpu_sparc_init
>> #define cpu_exec cpu_sparc_exec
>> etc. stand for this purpose, so there has been some consideration for this.
>
> Nicely handling per-arch functions would be one of the benefits of using
> C++ in QEMU (I know, it's sufficient but not necessary). What were the
> conclusions regarding such a change?

I don't think the discussions gave enough motivation for the change.
There's resistance to qdevification already and that is far from a
real object model.

Re: [Qemu-devel] Multi heterogenous CPU archs for SoC sim?

[Lluís]

Blue Swirl writes:

> On Thu, May 26, 2011 at 11:07 PM, Lluís <xscript@gmx.net> wrote:
>> Nicely handling per-arch functions would be one of the benefits of using
>> C++ in QEMU (I know, it's sufficient but not necessary). What were the
>> conclusions regarding such a change?

> I don't think the discussions gave enough motivation for the change.
> There's resistance to qdevification already and that is far from a
> real object model.

Well, C++ templates would help clean the current define and macro-based
code generation labyrinth without switching the whole QEMU codebase into
an OO design, but I suppose this was also part of the duscussion.

Thanks,
   Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: [Qemu-devel] Multi heterogenous CPU archs for SoC sim?

[Andreas Färber]

Am 26.05.2011 um 21:11 schrieb Blue Swirl:

> On Wed, May 25, 2011 at 10:20 PM, Peter Maydell
> <peter.maydell@linaro.org> wrote:
>> On 25 May 2011 19:44, Greg McGary <greg.mcgary@gmail.com> wrote:
>>> I would like to create a QEMU model of an SoC that has several
>>> CPU cores having different architectures.  I'm guessing this
>>> can be done.
>>
>> It's not supported currently as far as I'm aware. There was
>> at least one paper at the QEMU Forum earlier this year describing
>> an approach to multi-CPU environments (embedding QEMU into a
>> SystemC world) that basically saved and restored all QEMU's
>> global variables every time it switched cores...
>>
>> It would be good if it was supported in QEMU proper, but I
>> suspect you may be in for some large-scale restructuring work.
>
> One of the long standing goals for QEMU has been to be able to use a
> single executable to emulate multiple architectures. I think for
> example the lines like
> #define cpu_init cpu_sparc_init
> #define cpu_exec cpu_sparc_exec
> etc. stand for this purpose, so there has been some consideration  
> for this.

Gary never followed up on this, it seems.

I have now come across such a heterogeneous SoC myself: Renesas  
announced the R-Car H1 this week, a SoC with one SH4A core and four  
ARM Cortex-A9 cores. That would make them all 32-bit, and I am hoping  
to get confirmation that this is consistently Little Endian.

The only realistic way to get started with such setups I see is to  
create a new target-xxx for the specific mix, define TARGET_LONG_BITS  
etc. appropriately in a new cpu.h, compile the needed target-xyz/*.c  
to unique xxx-softmmu/xyz-*.o and dispatch from a cpu_init() to the  
two cpu_*_init().

I'm guessing we may need to distinguish the TBs at runtime? Reserving  
log2(#architectures) bits in the TBFLAGS might do, but feels ugly.
Probably a lot of other issues I'm not seeing yet.

Comments?

Andreas

Re: [Qemu-devel] Multi heterogenous CPU archs for SoC sim?

[Peter Maydell]

On 20 October 2011 23:51, Andreas Färber <andreas.faerber@web.de> wrote:
> I have now come across such a heterogeneous SoC myself: Renesas announced
> the R-Car H1 this week, a SoC with one SH4A core and four ARM Cortex-A9
> cores.

Does it expose the SH4 to apps/OSes, or is it mostly used for
power management or similar ignorable duties? (For several
of the ARM boards we currently just ignore the fact that the real
h/w has a Cortex-M3 doing power management type stuff.)

> That would make them all 32-bit, and I am hoping to get confirmation
> that this is consistently Little Endian.

I think the endianness is a red herring for heterogenous systems
anyway -- what QEMU defines as the target endianness is really something
more like the "system bus endianness", as far as I can tell.
An extra core with a different idea of endianness shouldn't be
any harder to handle than cores which can switch endianness
at runtime. You just either insert swizzling or not.

> The only realistic way to get started with such setups I see is to create a
> new target-xxx for the specific mix, define TARGET_LONG_BITS etc.
> appropriately in a new cpu.h, compile the needed target-xyz/*.c to unique
> xxx-softmmu/xyz-*.o and dispatch from a cpu_init() to the two cpu_*_init().

Yuck. Longer term if we want to support this kind of heterogeneity
we should be removing all the compile-time assumptions and generally
making the target-specifics suitably contained rather than leaking
into the rest of the code.

> I'm guessing we may need to distinguish the TBs at runtime? Reserving
> log2(#architectures) bits in the TBFLAGS might do, but feels ugly.
> Probably a lot of other issues I'm not seeing yet.

We may want the tb cache to be per-core anyway (and one thread per core),
which would avoid the problem of trying to wedge everything into one set
of tb_flags.

(Has anybody had a look at http://sourceforge.net/p/coremu/home/Home/ ?)

-- PMM

Re: [Qemu-devel] Multi heterogenous CPU archs for SoC sim?

[陳韋任]

> We may want the tb cache to be per-core anyway (and one thread per core),
> which would avoid the problem of trying to wedge everything into one set
> of tb_flags.
> 
> (Has anybody had a look at http://sourceforge.net/p/coremu/home/Home/ ?)

  COREMU treats QEMU as an entity and lauches multiple QEMUs at the same
time. QEMUs communicates to each other by using a underlying thin layer
provided by COREMU. I think this approach is much clean than trying to
parallelize QEMU itself.

Regards,
chenwj

-- 
Wei-Ren Chen (陳韋任)
Computer Systems Lab, Institute of Information Science,
Academia Sinica, Taiwan (R.O.C.)
Tel:886-2-2788-3799 #1667

Re: [Qemu-devel] Multi heterogenous CPU archs for SoC sim?

[Andreas]

Am 21.10.2011 09:26, schrieb 陳韋任:
>   COREMU treats QEMU as an entity and lauches multiple QEMUs at the same
> time. QEMUs communicates to each other by using a underlying thin layer
> provided by COREMU.

> I think this approach is much clean than trying to
> parallelize QEMU itself.

In this case I disagree. Given shared global memory and peripherals in
the SoC case, any IPC or shared-memory setup is destined to create
performance or management overhead.

When there's independent nodes connected via CAN/LIN/FlexRay, then I
agree that multiple processes communicating via UNIX sockets make a lot
of sense.

My use case here is testing and debugging, so I think we could live with
the blocks being executed in an interleaved fashion until someone has
the ultimate parallelization solution for upstream.

Regards,
Andreas

Re: [Qemu-devel] Multi heterogenous CPU archs for SoC sim?

[陳韋任]

> My use case here is testing and debugging, so I think we could live with
> the blocks being executed in an interleaved fashion until someone has
> the ultimate parallelization solution for upstream.

  So you prefer parallelizing QEMU itself in your case?

-- 
Wei-Ren Chen (陳韋任)
Computer Systems Lab, Institute of Information Science,
Academia Sinica, Taiwan (R.O.C.)
Tel:886-2-2788-3799 #1667

Re: [Qemu-devel] Multi heterogenous CPU archs for SoC sim?

[Lluís Vilanova]

Peter Maydell writes:

>> The only realistic way to get started with such setups I see is to create a
>> new target-xxx for the specific mix, define TARGET_LONG_BITS etc.
>> appropriately in a new cpu.h, compile the needed target-xyz/*.c to unique
>> xxx-softmmu/xyz-*.o and dispatch from a cpu_init() to the two cpu_*_init().

> Yuck. Longer term if we want to support this kind of heterogeneity
> we should be removing all the compile-time assumptions and generally
> making the target-specifics suitably contained rather than leaking
> into the rest of the code.

Yup, this sounds like pointer tables. That's one of the main reasons I advocated
C++. Another reason is improving code maintainability through templates (without
impacting performance).


>> I'm guessing we may need to distinguish the TBs at runtime? Reserving
>> log2(#architectures) bits in the TBFLAGS might do, but feels ugly.
>> Probably a lot of other issues I'm not seeing yet.

> We may want the tb cache to be per-core anyway (and one thread per core),
> which would avoid the problem of trying to wedge everything into one set
> of tb_flags.

Well, AFAIU if targets have different ISAs, the same physical tb will never be
used by differing targets (ignoring the case of different targets but same ISA).



Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: [Qemu-devel] Multi heterogenous CPU archs for SoC sim?

[Andreas Färber]

Am 21.10.2011 08:58, schrieb Peter Maydell:
> On 20 October 2011 23:51, Andreas Färber <andreas.faerber@web.de> wrote:
>> Renesas announced
>> the R-Car H1 this week, a SoC with one SH4A core and four ARM Cortex-A9
>> cores.
> 
> Does it expose the SH4 to apps/OSes, or is it mostly used for
> power management or similar ignorable duties?

The predecessors were all SuperH based only, and the ARM cores don't
seem to have VFPv3 so the SH4A would feature a 128-bit FPU.
As for what automative customers may do with it once available, I have
no clue. My focus is on investigating where QEMU has architectural
shortcomings or undocumented assumptions blocking embedded development
and addressing these.

> (For several
> of the ARM boards we currently just ignore the fact that the real
> h/w has a Cortex-M3 doing power management type stuff.)

Mind to share which boards? I'm only aware of the NXP LPC43xx asymmetric
SoC (Cortex-M4 + Cortex-M0), which still is in development stage. The
datasheet doesn't really enlighten me how such a combo is supposed to
work in shared memory: Do all ARM cores share a reset vector (or what
you call it on arm) so that one has to branch based on CPUID to do
different tasks on different cores?

Andreas

Re: [Qemu-devel] Multi heterogenous CPU archs for SoC sim?

[Peter Maydell]

On 21 October 2011 23:05, Andreas Färber <andreas.faerber@web.de> wrote:
> Am 21.10.2011 08:58, schrieb Peter Maydell:
>> (For several
>> of the ARM boards we currently just ignore the fact that the real
>> h/w has a Cortex-M3 doing power management type stuff.)
>
> Mind to share which boards? I'm only aware of the NXP LPC43xx asymmetric
> SoC (Cortex-M4 + Cortex-M0), which still is in development stage.

I had in mind the Versatile Express -- the main core is an A9x4
but there is also an M3 on the board. (There are some other
interestng ARM based heterogeneous architectures coming
up too, like the nVidia Tegra3 with 4xA9+1xA9, and the ARM
big.LITTLE systems with 4xA7+4xA15.)

> The
> datasheet doesn't really enlighten me how such a combo is supposed to
> work in shared memory: Do all ARM cores share a reset vector (or what
> you call it on arm) so that one has to branch based on CPUID to do
> different tasks on different cores?

In general a multicore bootup will use the core ID to figure out
what it is at reset. I don't know what a multicluster system
would do, but probably the same thing. The other approach I've
seen is that you can give the two CPUs different memory maps
so they share most things but have different memory layouts
where the vector table/startup code is. Or you could have
the power controller hold the secondary core in reset until
the primary core has booted, and have your boot code work
differently for first and second execution, I guess.

(The reset architecture on M profile and AR profiles is different,
but I don't think the differences are important for these purposes.)

-- PMM

Re: [Qemu-devel] Multi heterogenous CPU archs for SoC sim?

[Andreas Färber]

Am 25.05.2011 um 20:44 schrieb Greg McGary:

> I would like to create a QEMU model of an SoC that has several CPU  
> cores having different architectures.  I'm guessing this can be done.

In theory, many things are possible. In practice, it's a matter of  
manhours you're willing to invest. Without further details on your SoC  
it's hard to provide a useful answer.

An SoC with mixed ARM Cortex-M4 and Cortex-M0, for instance, may be  
much easier to accomplish than mixing architectures with differing  
endianness and especially bitness. That will likely involve compiling  
for ceil(bitness) - since I don't see how linking hw/*.o compiled for  
diffent bit-widths would work - and making sure any lower-bitness  
architectures continue to work, while probably dropping in overall  
performance...

Andreas