[Qemu-devel] Building QEMU with multiple CPU targets.

[Qemu-devel] Building QEMU with multiple CPU targets.

[Peter Crosthwaite]

Hi All,

Im currently investigating the possibility of building QEMU with
multiple CPU architectures active concurrently. That is, I have a
binary with both an target-arm and target-microblaze and wish to run
them as a heterogeneous multiprocessor platform.

Given the recent QOM development in making CPUs just another object,
shouldn't be possible with a bit of Makefile and configure rework to
build qemu-system-arm+microblaze and then create machine models
instantiating both CPU types?

Are the major complications here from either a Make or QOM perspective?

Has anyone done/tried this before with other architectures and have a
tree out there somewhere?

Regards,
Peter

Re: [Qemu-devel] Building QEMU with multiple CPU targets.

[Peter Maydell]

On 8 October 2012 07:39, Peter Crosthwaite
<peter.crosthwaite@petalogix.com> wrote:
> Im currently investigating the possibility of building QEMU with
> multiple CPU architectures active concurrently. That is, I have a
> binary with both an target-arm and target-microblaze and wish to run
> them as a heterogeneous multiprocessor platform.
>
> Given the recent QOM development in making CPUs just another object,
> shouldn't be possible with a bit of Makefile and configure rework to
> build qemu-system-arm+microblaze and then create machine models
> instantiating both CPU types?
>
> Are the major complications here from either a Make or QOM perspective?

I certainly think this would be a nice feature to have, but I suspect
the makefile/QOM bits are probably the easy parts :-)

At the moment things like the translated code cache are basically
globals and would need to be moved to be per-CPU. Also there are
still various settings that are compile time which would need to
become runtime (though we just got rid of the 'size of physical
address type' one at least).

-- PMM

Re: [Qemu-devel] Building QEMU with multiple CPU targets.

[Evgeny Voevodin]

On 10/08/2012 02:54 PM, Peter Maydell wrote:
> On 8 October 2012 07:39, Peter Crosthwaite
> <peter.crosthwaite@petalogix.com> wrote:
>> Im currently investigating the possibility of building QEMU with
>> multiple CPU architectures active concurrently. That is, I have a
>> binary with both an target-arm and target-microblaze and wish to run
>> them as a heterogeneous multiprocessor platform.
>>
>> Given the recent QOM development in making CPUs just another object,
>> shouldn't be possible with a bit of Makefile and configure rework to
>> build qemu-system-arm+microblaze and then create machine models
>> instantiating both CPU types?
>>
>> Are the major complications here from either a Make or QOM perspective?
>
> I certainly think this would be a nice feature to have, but I suspect
> the makefile/QOM bits are probably the easy parts :-)
>
> At the moment things like the translated code cache are basically
> globals and would need to be moved to be per-CPU. Also there are
> still various settings that are compile time which would need to
> become runtime (though we just got rid of the 'size of physical
> address type' one at least).

Did anybody start this work?
I'm interested in localiation of tcg per cpu.

>
> -- PMM
>
>

-- 
Kind regards,
Evgeny Voevodin,
Leading Software Engineer,
ASWG, Moscow R&D center, Samsung Electronics
e-mail: e.voevodin@samsung.com

Re: [Qemu-devel] Building QEMU with multiple CPU targets.

[Anthony Liguori]

Peter Maydell <peter.maydell@linaro.org> writes:

> On 8 October 2012 07:39, Peter Crosthwaite
> <peter.crosthwaite@petalogix.com> wrote:
>> Im currently investigating the possibility of building QEMU with
>> multiple CPU architectures active concurrently. That is, I have a
>> binary with both an target-arm and target-microblaze and wish to run
>> them as a heterogeneous multiprocessor platform.
>>
>> Given the recent QOM development in making CPUs just another object,
>> shouldn't be possible with a bit of Makefile and configure rework to
>> build qemu-system-arm+microblaze and then create machine models
>> instantiating both CPU types?
>>
>> Are the major complications here from either a Make or QOM perspective?
>
> I certainly think this would be a nice feature to have, but I suspect
> the makefile/QOM bits are probably the easy parts :-)
>
> At the moment things like the translated code cache are basically
> globals and would need to be moved to be per-CPU. Also there are
> still various settings that are compile time which would need to
> become runtime (though we just got rid of the 'size of physical
> address type' one at least).
>
> -- PMM

It may be possible to cheat and compile the TCG + CPU code multiple
times as dynamic libraries.  You can then load the libraries with
dlopen() with local symbol resolution.

Then it's mostly just a build file exercise.

Regards,

Anthony Liguori

Re: [Qemu-devel] Building QEMU with multiple CPU targets.

[Peter Maydell]

On 8 October 2012 21:23, Anthony Liguori <anthony@codemonkey.ws> wrote:
> It may be possible to cheat and compile the TCG + CPU code multiple
> times as dynamic libraries.  You can then load the libraries with
> dlopen() with local symbol resolution.

...and when the common code wants to (say) call cpu_exit() on a
particular core, how does it know which DLL's version to call?
That said, I have a feeling I've heard about people trying this
kind of approach before, so it's probably possible to get something
that works on at least one host OS. I think doing it "properly"
would be less of a hack, though...

-- PMM

Re: [Qemu-devel] Building QEMU with multiple CPU targets.

[Anthony Liguori]

Peter Maydell <peter.maydell@linaro.org> writes:

> On 8 October 2012 21:23, Anthony Liguori <anthony@codemonkey.ws> wrote:
>> It may be possible to cheat and compile the TCG + CPU code multiple
>> times as dynamic libraries.  You can then load the libraries with
>> dlopen() with local symbol resolution.
>
> ...and when the common code wants to (say) call cpu_exit() on a
> particular core, how does it know which DLL's version to call?
> That said, I have a feeling I've heard about people trying this
> kind of approach before, so it's probably possible to get something
> that works on at least one host OS. I think doing it "properly"
> would be less of a hack, though...

I think you basically need to treat a "foreign" CPU separately from a
local CPU.  You would load foreign CPUs as libraries and interact with
it separately.

I think the initial focus should be on hacking something to work.  Then
we can figure out what's the best path to merging.

Regards,

Anthony Liguori

>
> -- PMM

Re: [Qemu-devel] Building QEMU with multiple CPU targets.

[Edgar E. Iglesias]

On Mon, Oct 08, 2012 at 03:53:37PM -0500, Anthony Liguori wrote:
> Peter Maydell <peter.maydell@linaro.org> writes:
> 
> > On 8 October 2012 21:23, Anthony Liguori <anthony@codemonkey.ws> wrote:
> >> It may be possible to cheat and compile the TCG + CPU code multiple
> >> times as dynamic libraries.  You can then load the libraries with
> >> dlopen() with local symbol resolution.
> >
> > ...and when the common code wants to (say) call cpu_exit() on a
> > particular core, how does it know which DLL's version to call?
> > That said, I have a feeling I've heard about people trying this
> > kind of approach before, so it's probably possible to get something
> > that works on at least one host OS. I think doing it "properly"
> > would be less of a hack, though...
> 
> I think you basically need to treat a "foreign" CPU separately from a
> local CPU.  You would load foreign CPUs as libraries and interact with
> it separately.
> 
> I think the initial focus should be on hacking something to work.  Then
> we can figure out what's the best path to merging.


Hi,

I hacked on a similar shared lib approach a couple of years ago and got
it kind of working. It wasn't exactly the thing beeing discussed here
because in my case the various qemu instances were driven by an external
simulator but anyway. This was before the io-thread was mandatory and
glib was not so integrated. Later when I tried to update, I ran into
lots of thread issues and glib stepping on it's self. I didn't look
to much into it but it seem to get very messy.

I think that Anothonys and Maydells ideas can in some sense be combined.
If people start working on the "long term" approach of fixing things
within QEMU, other people will show up with ideas/patches that "make things"
work. But IMO, we need to start working on the inner issues asap.

Andreas work with cleaning up the CPU interfaces is great. Others
have helped with making the devices much more target independent etc.
All of those steps help. If we could build all the devices into a single
qemu and have just the CPU's in separate libraries, that would be a first
step (maybe that's almost possible now).  Then get rid of the endianness and
pagesize and other target cpu issues.

Heterogeneous cpus is becoming a killer feature in SOC emulation. Or, I'm
probably wrong. It _already_ is a killer feature. You can't model many modern
socs without it, unless you take shortcuts and model remote cpus and their
interfaces as pure hw device models or maybe if you do some kind of
remote/IPC connections.

Cheers

Re: [Qemu-devel] Building QEMU with multiple CPU targets.

[Peter Crosthwaite]

Hi All,

Thanks for the responses. Ill look into further in the near future,
probably starting the the Linker based approaches.

Regards,
Peter

Re: [Qemu-devel] Building QEMU with multiple CPU targets.

[Andreas Färber]

Hi Peter,

Am 08.10.2012 08:39, schrieb Peter Crosthwaite:
> Im currently investigating the possibility of building QEMU with
> multiple CPU architectures active concurrently. That is, I have a
> binary with both an target-arm and target-microblaze and wish to run
> them as a heterogeneous multiprocessor platform.
> 
> Given the recent QOM development in making CPUs just another object,
> shouldn't be possible with a bit of Makefile and configure rework to
> build qemu-system-arm+microblaze and then create machine models
> instantiating both CPU types?
> 
> Are the major complications here from either a Make or QOM perspective?

Neither. The major complication is cpu.h and the per-target functions it
defines/declares.

My CPUState refactoring was a step into that direction, to get a solid
QOM base class rather than macro contents inserted in the middle of the
CPUArchState struct.
But the CPU-as-a-device refactorings and x86 CPU hotplug (APIC among
others) collided with my part 4 series. It also turned out that
refactorings to the TLB code will be necessary to do this cleanly. It
also turned out that my old qom-cpu-4 branch no longer applies, so
instead of rebasing that large branch only to find out I need to rebase
again, I have been cherry-picking or redoing only individual patches so far.

> Has anyone done/tried this before with other architectures and have a
> tree out there somewhere?

See http://wiki.qemu.org/Features/QOM/CPU and feel free to document your
example there.

I had a tree experimentally combining sh4 and arm but that was pre-QOM.
What I tried was creating a new target-rcar with a frontend cpu.h that
delegates to per-CPU code. This was pretty invasive. The proper route to
get there is to move fields into the QOM CPUState struct, so that the
core CPU handling can be shared among softmmus, split off the TLB from
CPUState (as suggested by Blue; requires touching each tcg/*/),
replacing most per-target (inline) functions with hooks on CPUState or
CPU-specific ones (e.g., cpu_mb_init+cpu_arm_init rather than cpu_init).

It's certainly possible to achieve this and I'm working on it, we just
won't be done by tomorrow. ;)

Regards,
Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] Building QEMU with multiple CPU targets.

[Stefan Weil]

Am 08.10.2012 15:17, schrieb Andreas Färber:
> Hi Peter,
>
> Am 08.10.2012 08:39, schrieb Peter Crosthwaite:
>> Im currently investigating the possibility of building QEMU with
>> multiple CPU architectures active concurrently. That is, I have a
>> binary with both an target-arm and target-microblaze and wish to run
>> them as a heterogeneous multiprocessor platform.

If this works, it will also be possible to create a single
"qemu" executable for all system emulations.

>>
>> Given the recent QOM development in making CPUs just another object,
>> shouldn't be possible with a bit of Makefile and configure rework to
>> build qemu-system-arm+microblaze and then create machine models
>> instantiating both CPU types?
>>
>> Are the major complications here from either a Make or QOM perspective?
> Neither. The major complication is cpu.h and the per-target functions it
> defines/declares.
>
> My CPUState refactoring was a step into that direction, to get a solid
> QOM base class rather than macro contents inserted in the middle of the
> CPUArchState struct.

Adding an entry for the guest endianess to the CPU state
is also needed when the emulated system includes
CPUs with different endianess. Today we have conditional
compilation based on TARGET_WORDS_BIGENDIAN.

Regards

Stefan