Re: [Qemu-devel] Unified device model

[Fabrice Bellard <fabrice@bellard.org>]

Hi,

My feeling is that QEMU already has what is needed to make a "plugin 
API". In particular, each device driver is clearly separated from the 
rest of the emulator and adding a new "machine" just consists in adding 
a new C file instanciating the different devices and processors. The 
fact that the devices or machines cannot come from a DLL or cannot be 
compiled separately is a cosmetic issue and could be easily added. I see 
little advantage in it except to make binary device models which I don't 
want to promote.

The real issues I see to improve the QEMU device model for the short 
term are:

- Rename some structures or functions to have a more consistent naming 
and rename vl.h to something like qemu/qemu.h and eventually split it to 
have a few subsystems such as isa, pci, usb... The question whether we 
keep a QEMU prefix is mostly political.

- Implement reproductible execution support and see if API changes are 
needed in the devices.

- Add explicit contexts in the (rare) cases where there are not already 
explicit. The API shall allow any number of CPUs and buses. A device 
tree should be added to give a global name for each device and cpu that 
can be referenced in the monitor and in the traces.

- Improve the tracing API. It should be possible to enable traces for 
each instance of a device. The various I/Os and events should be 
displayed in a uniform way.

- Make it possible to integrate different CPU types and all devices in 
the same executables without loosing performance. There are some API 
issues to solve regarding endianness.

- Document the API various functions by adding javadoc comments.

Another interesting point would be to see how to interface with hardware 
device models (for example written in VHDL and compiled with GHDL).

Looking at the API of other projects such as MAME which are more mature 
than QEMU or Bochs in the sense that they have far more devices and 
processors would be interesting too.

Fabrice.

Stanislav Shwartsman wrote:
> Hello All,
> 
> I'll try to answer to everybody in single mail and explain my vision of
> unified plugin API and why I see them essential.
> 
> Yes, I am Bochs developer, but I am more CPU expert and only learning the
> devices subsystem and dynamic translation subjects so Bochs is a right place
> for me ;) I am pretty understand all the Bochs code including devices and
> also have some vision of highly portable and extendable simulator.
> 
> Now I'll try to explain how I see the plugin architecture and why I think it
> essential.
> 
> - I am not talking about binary level plugins support and maintenance or
> sources level. 
> - I am not talking about open source or closed source, commercial and
> freeware devices for emulator, but I think the plugin architecture should
> support both.
> - I not think the plugin API should be C++, actually Bochs plugin API is
> pure-C based. The device itself is written in C++ but it has something like
> C++ -> C proxy implemented as a part of plugin API. I think here we are
> agree.
> - It is doesn't matter if the plugin is shared or static library, the code
> size of Bochs or QEMU is so small and I don't think it is important feature
> to "share" devices in memory.
> 
> The keyword is expendability. Now if you would like to write a new device
> for Bochs or QEMU you cannot open a separate project on SF.net and start
> development. Even if you need to add some exotic experimental device which
> is not exists in real life you should interfere with simulator code.
> Moreover, if you need to add this device to Bochs, QEMU and Xen you
> basically have to do the work three times !
> 
> I hope I am not disclose any secret if I say that HiPEAC beginning a new
> project of open source full-system simulator, something that could be
> alternative to SimpleScalar and also have full-system emulation capabilities
> (platform + power + performance in single box). The simulator would reuse
> the devices from some existent open source emulator, Bochs or QEMU. But as
> any academia model it should be highly extendable and super-modular. The
> ability to add a new device to the system without touching the main
> simulator kernel is a MUST. 
> 
> I believe it is right direction too. Somebody could establish a new project
> of developing new device and when contribute the device when it will be
> done. In my opinion, it is even not important if the device written in C/C++
> or any other language, if it is distributed with sources or not, under
> commercial license or not. If I could download device binary, plug it in to
> the simulator and get new capabilities, which were not available before -
> why not ?
> 
> I could provide more and more examples, more and more advantages of this
> arch and almost no disadvantages. There are a lot of successful examples of
> products with extendable user plugins where functionality of the product as
> a very small piece near the variety of plugins written by users.
> 
> Now the advantages of making every device separate library (shared or
> static). When every device is separate library you try to build your
> emulated system by yourself choosing the CPU core emulator + chipset plugin
> + devices plugins you want. If some set of plugins distributed as single
> library - they are not separatable one from another. This is only a
> disadvantage which could simply avoided, so again, why not ?
> 
> I am already not talking about devices maintenance. The best example is
> Brendan BIOS. Brendan started a new project of writing new ACPI-compatible
> BIOS for open source emulators. His BIOS require ACPI compatibility changes
> in the device models and CPU emulator and Bochs already started to support
> his requirements. Currently, when his BIOS will be done, QEMU will remain
> behind it and not be able to run with it. Anyway, to support the
> requirements the work should be done 2-3 times, from the beginning for each
> simulator.
> 
> The Bochs will start from some PCI device and try to separate it from the
> simulator code, compile it stand-alone and make it pluginable in run-time.
> Why PCI device ? Just because it has more functionality than PC-speaker card
> and require more interference with core simulator. Actually might be several
> device models for core plugins (chipset, PCI bus, AGP bus), ISA/PCI cards
> and COM devices, everything is open. During the separation work I hope to
> figure out requirements from devices and simulator core for new plugin
> architecture. In Bochs, a lot of changes required to make it happen, and we
> believe we'll make it possible until next Bochs 2.3 release.
> 
> Thanks,
> Stanislav
> 
> -----Original Message-----
> From: qemu-devel-bounces+stl=fidonet.org.il@nongnu.org
> [mailto:qemu-devel-bounces+stl=fidonet.org.il@nongnu.org] On Behalf Of Jim
> C. Brown
> Sent: Sunday, April 09, 2006 6:08 PM
> To: qemu-devel@nongnu.org
> Subject: Re: [Qemu-devel] Unified device model
> 
> On Sun, Apr 09, 2006 at 04:21:42PM +0100, Paul Brook wrote:
> 
>>>>I'm not a fan of binary plugins (for the same reasons I'm don't like
>>>>binary kernel modules), and don't think there's any real need to them.
>>>
>>>A binary plugin API and a source plugin API (one that requires each
> 
> driver
> 
>>>device to be recompiled for each of the platforms (Xen, qemu, bochs,
> 
> etc.)
> 
>>>would probably be equally hard to design and maintain.
>>
>>You've missed my point. The only reason I can see for wanting binary
> 
> plugins 
> 
>>is so that people can distribute proprietary closed-source device
> 
> emulation.
> 
> I agree that proprietary or closed-source device emulation is a bad thing
> and
> should not be supported.
> 
> 
>>A stable source API is a prerequisite for any sort of binary plugins.
>>
> 
> 
> In that case, perhaps a stable source API would be best.
> 
> Like I said before, the type of API/sharing (source vs binary API, and
> static
> vs shared libraries) is a separate issue from the one we are discussing
> (should
> we have or support a unified plugin API?).
> 
> 
>>>>I can't see
>>>>any good reasons why open source devices would need to be broken out
> 
> into
> 
>>>>a separate shared library.
>>>
>>>I think the case was already made for this.
>>>
>>>Xen's hardware emulation, while based on qemu's, is already ahead in
>>>several aspects. A separate library would make it more convenient for
> 
> these
> 
>>>changes to be shared back with qemu. Or with E/OS.
>>
>>I don't buy that. We either share the same drivers (in which case keeping
> 
> the 
> 
>>two in sync is trivial) or we don't. All of the systems under
> 
> consideration 
> 
>>are [L]GPL licences. We can easily copy the source, so I don't think being
> 
> 
>>able to copy bits of binary goo gains us anything.
> 
> 
> A) Makes it simpler for end users to move devices over (they don't have to
> know
> how to cut-and-paste C code)
> 
> B) Bochs is not related to qemu at all code-wise, so the cut-and-paste
> senario
> doesn't work here. If we want to share drivers with Bochs we'd need at least
> a
> source API. (The starter of this thread is a Bochs developer I believe...
> but correct me if I'm wrong here. :) The alternative is to rewrite Bochs
> drivers for qemu from scratch (possbly using the Bochs code as a guide) but
> that is even harder than the qemu-xen case.
> 
> C) If they are in a special library (say maintained by a 3rd party group
> that
> consists of developers from all the major projects) then maintainance is
> greatly
> simplified over time.
> 
> 
>>I don't think executable size is a valid argument either. Device emulation
> 
> 
>>code generally isn't that big so the overhead of breaking it up into
> 
> multiple 
> 
>>shared libraries would outweigh the benefits of not loading the bits
> 
> you're 
> 
>>not using.
> 
> 
> Perhaps you are right about that. The size of having even 4 or 5 copies of
> complete PC hardware emulation code isn't so large as to be a problem except
> on systems that are either embedded or ancient (in which case you probably
> have
> no business running 4 different PC emulators anyways).
> 
> Personally, it just seems inelegant to have such code duplication.
> 
> 
>>Paul
>>
> 
>