Re: [Qemu-devel] Unified device model

Re: [Qemu-devel] Unified device model

[Jim C. Brown]

On Sat, Apr 08, 2006 at 09:57:10PM +0200, Stanislav Shwartsman wrote:
> Hello All,
> 
>  
> 
> It is not a secret that all open source emulators (QEMU, Bochs, Xen) use the
> same emulated devices and mostly copy-paste their emulation one from
> another.

While from my understanding Xen uses qemu's hardware emulation for it's VT
support, this is not really true otherwise.

The devices emulated by qemu and bochs are quite similar, but the code looks
completely different (appears to be a ground-up rewrite).

> 
> I don't know who originally wrote the device models but now Bochs and QEMU
> maintain two similar implementations of the same devices.
> 
> If one of the teams fixes the implementation or add functionality, another
> team mostly copy-paste the changes to their model.

I don't know how well Bochs and qemu keep in touch with each other. I've never
seen a Bochs developer announce themselves here, though.

> 
> Xen project forked from QEMU and want to stay in touch with Bochs and QEMU
> device models and contribute the changes to make the model better.

Not true. Xen is completely independent. Unless you are refering to the
hardware emulation - which I believe is qemu's stuff.

> 
> I am wondering about making unified device models architecture for open
> source simulators.
> 
> The device models will be used in QEMU, Bochs, Xen and other open source
> simulators which would use the device models.
> 

I would support this idea, if it was possible.

> I know about two professional teams working in simulation which would like
> to use these device models in their simulator and 
> 
> could enrich the device library with new devices device interfaces, for
> example with AGP and 3D graphics.
> 
> Bochs is already in middle of definition of new true pluginable devices
> architecture. 
> 

This is welcome news.

> In near future Bochs devices will fully separatable from Bochs binary and
> when could be developed separately from Bochs.
> 
> I call to QEMU developers join to this project and come with their
> requirements to plugin architecture.
> 
> I don't know if QEMU supports device plugins now but I would like to see
> QEMU a part of this idea, 
> 

I would as well.

> I would like to get single device shared library which could be loaded to
> Bochs and QEMU and work perfectly for both.
> 
> This will eliminate the need to maintain two separate implementations of the
> same devices, 
> 
> these implementations very fast will converge to single one, C or C++ based,
> Bochs or QEMU based, doesn't matter.
> 
> I am listening for your opinions !
> 
>  

The primary reason given for not making a plugin API for qemu hardware emulation
is that qemu isn't stable enough - the code changes too often to support a stable
API.

Still, it might be easier to add support for plugins based on an external API,
rather than trying to keep a qemu plugin API consistent.

> 
> Thanks,
> 
> Stanislav
> 

> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel


-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.

Re: [Qemu-devel] Unified device model

[Johannes Schindelin]

Hi,

On Sat, 8 Apr 2006, Jim C. Brown wrote:

> On Sat, Apr 08, 2006 at 09:57:10PM +0200, Stanislav Shwartsman wrote:
> > 
> > It is not a secret that all open source emulators (QEMU, Bochs, Xen) use the
> > same emulated devices and mostly copy-paste their emulation one from
> > another.
> 
> While from my understanding Xen uses qemu's hardware emulation for it's VT
> support, this is not really true otherwise.
> 
> The devices emulated by qemu and bochs are quite similar, but the code looks
> completely different (appears to be a ground-up rewrite).

IIRC bochs does it in C++. Which makes it rather impossible to share code 
:-(

Ciao,
Dscho

Re: [Qemu-devel] Unified device model

[Leonardo E. Reiter]

Well, not completely impossible, but it would require some really ugly 
"glue" code.  And, the glue would have to happen outside of QEMU (i.e. 
like in the BOCHS code), to keep C++ out of QEMU.

To have a truly portable API, it should definitely have C language 
"bindings".  I'm sure this could be added to the BOCHS implementation 
somehow if this is important.

- Leo Reiter

Johannes Schindelin wrote:
> IIRC bochs does it in C++. Which makes it rather impossible to share code 
> :-(
> 
> Ciao,
> Dscho

-- 
Leonardo E. Reiter
Vice President of Product Development, CTO

Win4Lin, Inc.
Virtual Computing from Desktop to Data Center
Main: +1 512 339 7979
Fax: +1 512 532 6501
http://www.win4lin.com

Re: [Qemu-devel] Unified device model

[Jim C. Brown]

On Sat, Apr 08, 2006 at 09:17:52PM +0200, Johannes Schindelin wrote:
> IIRC bochs does it in C++. Which makes it rather impossible to share code 
> :-(
> 
> Ciao,
> Dscho
> 

This is why I guessed qemu was a rewrite-from-scratch - C++ code is forbidden
in qemu.

Fortunately this does not prevent a properly designed binary plugin API.

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.

[Qemu-devel] Unified device model

[Stanislav Shwartsman]

[-- Attachment #1: Type: text/plain, Size: 1823 bytes --]

Hello All,

 

It is not a secret that all open source emulators (QEMU, Bochs, Xen) use the
same emulated devices and mostly copy-paste their emulation one from
another.

I don't know who originally wrote the device models but now Bochs and QEMU
maintain two similar implementations of the same devices.

If one of the teams fixes the implementation or add functionality, another
team mostly copy-paste the changes to their model.

Xen project forked from QEMU and want to stay in touch with Bochs and QEMU
device models and contribute the changes to make the model better.

I am wondering about making unified device models architecture for open
source simulators.

The device models will be used in QEMU, Bochs, Xen and other open source
simulators which would use the device models.

I know about two professional teams working in simulation which would like
to use these device models in their simulator and 

could enrich the device library with new devices device interfaces, for
example with AGP and 3D graphics.

Bochs is already in middle of definition of new true pluginable devices
architecture. 

In near future Bochs devices will fully separatable from Bochs binary and
when could be developed separately from Bochs.

I call to QEMU developers join to this project and come with their
requirements to plugin architecture.

I don't know if QEMU supports device plugins now but I would like to see
QEMU a part of this idea, 

I would like to get single device shared library which could be loaded to
Bochs and QEMU and work perfectly for both.

This will eliminate the need to maintain two separate implementations of the
same devices, 

these implementations very fast will converge to single one, C or C++ based,
Bochs or QEMU based, doesn't matter.

I am listening for your opinions !

 

Thanks,

Stanislav


[-- Attachment #2: Type: text/html, Size: 5766 bytes --]

RE: [Qemu-devel] Unified device model

[Stanislav Shwartsman]

Hi again,
 
>>It is not a secret that all open source emulators (QEMU, Bochs, Xen) use
>>the same emulated devices and mostly copy-paste their emulation one from
>>another.

>While from my understanding Xen uses qemu's hardware emulation for it's VT
>support, this is not really true otherwise.

>The devices emulated by qemu and bochs are quite similar, but the code 
>looks completely different (appears to be a ground-up rewrite).

I am talking about the device models only. Inside CPU emulation QEMU, Bochs
and Xen are completely different, but they use the same devices for full
system emulation and they most likely want to move forward together. The
devices system of QEMU and Bochs become outdate, it is required to add ACPI
compliance and emulate new modern ACPI compatible devices, currently
emulated i440FX already not so represent the real life and most of the
modern OSes already have no support for it.

 
>> I don't know who originally wrote the device models but now Bochs and
>> QEMU maintain two similar implementations of the same devices.
>> If one of the teams fixes the implementation or add functionality,
>> another team mostly copy-paste the changes to their model.

>I don't know how well Bochs and qemu keep in touch with each other. I've 
>never seen a Bochs developer announce themselves here, though.

So may we should ;)
At least I see the code changes from Bochs migrating to QEMU and vise versa
when I am looking into the code.

>>Xen project forked from QEMU and want to stay in touch with Bochs and QEMU
>>device models and contribute the changes to make the model better.

>Not true. Xen is completely independent. Unless you are refering to the
>hardware emulation - which I believe is qemu's stuff.

I know that it is completely independent project. But their device models
could be called separate project Xen-HWEMU and it is fork from QEMU.

>>I am wondering about making unified device models architecture for open
>>source simulators.
>>The device models will be used in QEMU, Bochs, Xen and other open source
>>simulators which would use the device models.

>I would support this idea, if it was possible.

Why not ?
You always could consider to add simple modules C++ to QEMU or build C++
device -> C device interface bridge ...
I don't know all the possibilities, but I am sure there are more.

>>I know about two professional teams working in simulation which would like
>>to use these device models in their simulator and 
>>could enrich the device library with new devices device interfaces, for
>>example with AGP and 3D graphics.
>>Bochs is already in middle of definition of new true pluginable devices
>>architecture. 

>This is welcome news.

Currently we are thinking about making user-plugin PCI devices and when
later user-plugin for any device. The C++ hierarchy for now might be:
bx_devmodel_c - bx_pcidev_c - bx_user_pcidev_c

The plans are to take one of the Bochs PCI devices and convert it into
user-level plugin which should not be compiled with Bochs and even not known
at compile time, but loaded at runtime if selected in runtime config file.
When any other device models could be added, even with closed-sources and
commercial licenses.

>The primary reason given for not making a plugin API for qemu hardware
>emulationis that qemu isn't stable enough - the code changes too often to
>support a stable API.

>Still, it might be easier to add support for plugins based on an external
>API, rather than trying to keep a qemu plugin API consistent.

Ok, I didn't knew that QEMU is so unstable. But at least there are several
trivial requirements from plugin architecture which you could mention here
and I am still not thought about it. I know, basically I am writing the
definition in my idle time and Volker supporting me. But it is not enough
...

Thanks,
Stanislav

RE: [Qemu-devel] Unified device model

[Stanislav Shwartsman]

Currently Bochs plugins APCI already begins with such C-language bindings.
But inside the device is C++ based and derives from Bochs device model. It
has the same log writer and debugging interface as any other Bochs module
but nothing more.

Could you look into the Bochs devices code and plugin code, I think you
could say how much it is compatible fro QEMU and if there is something that
we should consider to change ...

Stanislav

-----Original Message-----
From: qemu-devel-bounces+stl=fidonet.org.il@nongnu.org
[mailto:qemu-devel-bounces+stl=fidonet.org.il@nongnu.org] On Behalf Of
Leonardo E. Reiter
Sent: Saturday, April 08, 2006 9:27 PM
To: qemu-devel@nongnu.org
Subject: Re: [Qemu-devel] Unified device model

Well, not completely impossible, but it would require some really ugly 
"glue" code.  And, the glue would have to happen outside of QEMU (i.e. 
like in the BOCHS code), to keep C++ out of QEMU.

To have a truly portable API, it should definitely have C language 
"bindings".  I'm sure this could be added to the BOCHS implementation 
somehow if this is important.

- Leo Reiter

Johannes Schindelin wrote:
> IIRC bochs does it in C++. Which makes it rather impossible to share code 
> :-(
> 
> Ciao,
> Dscho

-- 
Leonardo E. Reiter
Vice President of Product Development, CTO

Win4Lin, Inc.
Virtual Computing from Desktop to Data Center
Main: +1 512 339 7979
Fax: +1 512 532 6501
http://www.win4lin.com


_______________________________________________
Qemu-devel mailing list
Qemu-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/qemu-devel

Re: [Qemu-devel] Unified device model

[Paul Brook]

> >>I am wondering about making unified device models architecture for open
> >>source simulators.
> >>The device models will be used in QEMU, Bochs, Xen and other open source
> >>simulators which would use the device models.
> >
> >I would support this idea, if it was possible.
>
> Why not ?
> You always could consider to add simple modules C++ to QEMU or build C++
> device -> C device interface bridge ...

I think to be acceptable to qemu (and probably also for Xen) the devices would 
have to be written in C. C++ is more pain that it's worth in this context.
Of course there's no reason why we couldn't use the subset of C that's also 
valid C++. You could also write C++ wrappers round the interface for bochs to 
use.

I'm not a fan of binary plugins (for the same reasons I'm don't like binary 
kernel modules), and don't think there's any real need to them. I can't see 
any good reasons why open source devices would need to be broken out into a 
separate shared library.

If you do want to accommodate proprietary binary plugins then C++ is a really 
bad idea. The C++/libstdc++ ABI simply isn't stable enough to make this a 
realistic option.

Paul

Re: [Qemu-devel] Unified device model

[Jim C. Brown]

On Sun, Apr 09, 2006 at 11:38:28AM +0100, Paul Brook wrote:
> I think to be acceptable to qemu (and probably also for Xen) the devices would 
> have to be written in C. C++ is more pain that it's worth in this context.
> Of course there's no reason why we couldn't use the subset of C that's also 
> valid C++. You could also write C++ wrappers round the interface for bochs to 
> use.
> 

Same here.

> I'm not a fan of binary plugins (for the same reasons I'm don't like binary 
> kernel modules), and don't think there's any real need to them.

A binary plugin API and a source plugin API (one that requires each driver
device to be recompiled for each of the platforms (Xen, qemu, bochs, etc.)
would probably be equally hard to design and maintain.

With a binary plugin API you at least win out.

> I can't see 
> any good reasons why open source devices would need to be broken out into a 
> separate shared library.
> 

I think the case was already made for this.

Xen's hardware emulation, while based on qemu's, is already ahead in several
aspects. A separate library would make it more convenient for these changes
to be shared back with qemu. Or with E/OS.

This is actually a completely separate issue from a unified device driver API
(as qemu could support the API, but only in source code form, or could require
that drivers be linked in statically, etc) and should be recognized as such.

> If you do want to accommodate proprietary binary plugins then C++ is a really 
> bad idea. The C++/libstdc++ ABI simply isn't stable enough to make this a 
> realistic option.

Considering that the ABI does not guarantee compatibility between versions, I
am inclined to agree.

No reason the drivers themselves can't be done in C++, but the API itself
should be pure C.

> 
> Paul
> 
> 
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel
> 

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.

Re: [Qemu-devel] Unified device model

[Jim C. Brown]

On Sun, Apr 09, 2006 at 08:26:11AM +0200, Stanislav Shwartsman wrote:
> I am talking about the device models only. Inside CPU emulation QEMU, Bochs
> and Xen are completely different, but they use the same devices for full
> system emulation and they most likely want to move forward together.

I haven't kept up with Bochs lately, so I have no idea if what qemu emulates
and what Bochs emulates are still the same or if the devices supported have
divulged.

I do know that the code is completely different (they aren't even in the same
language!).

This part of the discussion is getting offtopic tho - how devices are done
right now is not important.

> The
> devices system of QEMU and Bochs become outdate, it is required to add ACPI
> compliance and emulate new modern ACPI compatible devices, currently
> emulated i440FX already not so represent the real life and most of the
> modern OSes already have no support for it.
> 

Agreed.

> So may we should ;)

Agreed.

> At least I see the code changes from Bochs migrating to QEMU and vise versa
> when I am looking into the code.
> 

You do?

I wasn't aware that bochs took anything from qemu, and the only piece of code
that I know that is shared between the two is for the bios.

> Why not ?
> You always could consider to add simple modules C++ to QEMU or build C++
> device -> C device interface bridge ...
> I don't know all the possibilities, but I am sure there are more.

C++ code in qemu is forbidden (by the author and maintainer).

So a plugin API based on C sounds like the best option.

Bochs is unique as it is the only active open source PC emulator/virtualizer that
is done in C++. So "Bochs has C++ support" is not a good reason for the API
to be based on C++.

> 
> >>I know about two professional teams working in simulation which would like
> >>to use these device models in their simulator and 
> >>could enrich the device library with new devices device interfaces, for
> >>example with AGP and 3D graphics.
> >>Bochs is already in middle of definition of new true pluginable devices
> >>architecture. 
> 
> >This is welcome news.
> 
> Currently we are thinking about making user-plugin PCI devices and when
> later user-plugin for any device. The C++ hierarchy for now might be:
> bx_devmodel_c - bx_pcidev_c - bx_user_pcidev_c
> 
> The plans are to take one of the Bochs PCI devices and convert it into
> user-level plugin which should not be compiled with Bochs and even not known
> at compile time, but loaded at runtime if selected in runtime config file.

Why only PCI devices?

Why not USB, ISA (keyboard, monitor, pc speaker), devices that attach to the
serial port (like the current wacom patch), etc?

Also note that it is possible to use C to set up such a hierarchy - perhaps
not as natural as in C++, but still doable.

> When any other device models could be added, even with closed-sources and
> commercial licenses.

I don't support this.

Under what circumstances would commercial qemu/xen/bochs/eos driver devices
be a good thing?

I can understand (tho I still dislike) having commercial drivers in say
the kernel - if its the only way to get support for a particular device, and
that device would make it easier for others to use a different OS (e.g.
nvidia under linux so ppl can have good 3d support) then I find it acceptable
and tolerable.

But why would we want to do this for qemu et. al.?

> 
> >The primary reason given for not making a plugin API for qemu hardware
> >emulationis that qemu isn't stable enough - the code changes too often to
> >support a stable API.
> 
> >Still, it might be easier to add support for plugins based on an external
> >API, rather than trying to keep a qemu plugin API consistent.
> 
> Ok, I didn't knew that QEMU is so unstable. But at least there are several
> trivial requirements from plugin architecture which you could mention here
> and I am still not thought about it. I know, basically I am writing the
> definition in my idle time and Volker supporting me. But it is not enough
> ...

If I think of anything else, I'll let you know. I do want to see this happen.

> 
> Thanks,
> Stanislav
> 
> 
> 
> 
> 
> 
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel
> 

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.

Re: [Qemu-devel] Unified device model

[Paul Brook]

> > I'm not a fan of binary plugins (for the same reasons I'm don't like
> > binary kernel modules), and don't think there's any real need to them.
>
> A binary plugin API and a source plugin API (one that requires each driver
> device to be recompiled for each of the platforms (Xen, qemu, bochs, etc.)
> would probably be equally hard to design and maintain.

You've missed my point. The only reason I can see for wanting binary plugins 
is so that people can distribute proprietary closed-source device emulation.

A stable source API is a prerequisite for any sort of binary plugins.

> > I can't see
> > any good reasons why open source devices would need to be broken out into
> > a separate shared library.
>
> I think the case was already made for this.
>
> Xen's hardware emulation, while based on qemu's, is already ahead in
> several aspects. A separate library would make it more convenient for these
> changes to be shared back with qemu. Or with E/OS.

I don't buy that. We either share the same drivers (in which case keeping the 
two in sync is trivial) or we don't. All of the systems under consideration 
are [L]GPL licences. We can easily copy the source, so I don't think being 
able to copy bits of binary goo gains us anything.

I don't think executable size is a valid argument either. Device emulation 
code generally isn't that big so the overhead of breaking it up into multiple 
shared libraries would outweigh the benefits of not loading the bits you're 
not using.

Paul

Re: [Qemu-devel] Unified device model

[Sam Barnett-Cormack]

Paul Brook wrote:

> I don't buy that. We either share the same drivers (in which case keeping the 
> two in sync is trivial) or we don't. All of the systems under consideration 
> are [L]GPL licences. We can easily copy the source, so I don't think being 
> able to copy bits of binary goo gains us anything.

In my experience of writing multiple systems that do the same thing, 
some shared library code (be it static or dynamic (shared object) in 
form) makes life much simpler. It can be maintained as a seperate source 
module, libtool-ized, and made into a nice bit of magic that just needs 
to be modified once, without manually copying-and-pasting bits back and 
forth. If it's done statically, it can still be distributed with each 
source package to ensure compatibility.

Sam

Re: [Qemu-devel] Unified device model

[Jim C. Brown]

On Sun, Apr 09, 2006 at 04:21:42PM +0100, Paul Brook wrote:
> > > I'm not a fan of binary plugins (for the same reasons I'm don't like
> > > binary kernel modules), and don't think there's any real need to them.
> >
> > A binary plugin API and a source plugin API (one that requires each driver
> > device to be recompiled for each of the platforms (Xen, qemu, bochs, etc.)
> > would probably be equally hard to design and maintain.
> 
> You've missed my point. The only reason I can see for wanting binary plugins 
> is so that people can distribute proprietary closed-source device emulation.

I agree that proprietary or closed-source device emulation is a bad thing and
should not be supported.

> 
> A stable source API is a prerequisite for any sort of binary plugins.
> 

In that case, perhaps a stable source API would be best.

Like I said before, the type of API/sharing (source vs binary API, and static
vs shared libraries) is a separate issue from the one we are discussing (should
we have or support a unified plugin API?).

> > > I can't see
> > > any good reasons why open source devices would need to be broken out into
> > > a separate shared library.
> >
> > I think the case was already made for this.
> >
> > Xen's hardware emulation, while based on qemu's, is already ahead in
> > several aspects. A separate library would make it more convenient for these
> > changes to be shared back with qemu. Or with E/OS.
> 
> I don't buy that. We either share the same drivers (in which case keeping the 
> two in sync is trivial) or we don't. All of the systems under consideration 
> are [L]GPL licences. We can easily copy the source, so I don't think being 
> able to copy bits of binary goo gains us anything.

A) Makes it simpler for end users to move devices over (they don't have to know
how to cut-and-paste C code)

B) Bochs is not related to qemu at all code-wise, so the cut-and-paste senario
doesn't work here. If we want to share drivers with Bochs we'd need at least a
source API. (The starter of this thread is a Bochs developer I believe...
but correct me if I'm wrong here. :) The alternative is to rewrite Bochs
drivers for qemu from scratch (possbly using the Bochs code as a guide) but
that is even harder than the qemu-xen case.

C) If they are in a special library (say maintained by a 3rd party group that
consists of developers from all the major projects) then maintainance is greatly
simplified over time.

> 
> I don't think executable size is a valid argument either. Device emulation 
> code generally isn't that big so the overhead of breaking it up into multiple 
> shared libraries would outweigh the benefits of not loading the bits you're 
> not using.

Perhaps you are right about that. The size of having even 4 or 5 copies of
complete PC hardware emulation code isn't so large as to be a problem except
on systems that are either embedded or ancient (in which case you probably have
no business running 4 different PC emulators anyways).

Personally, it just seems inelegant to have such code duplication.

> 
> Paul
> 

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.

RE: [Qemu-devel] Unified device model

[Stanislav Shwartsman]

Hello All,

I'll try to answer to everybody in single mail and explain my vision of
unified plugin API and why I see them essential.

Yes, I am Bochs developer, but I am more CPU expert and only learning the
devices subsystem and dynamic translation subjects so Bochs is a right place
for me ;) I am pretty understand all the Bochs code including devices and
also have some vision of highly portable and extendable simulator.

Now I'll try to explain how I see the plugin architecture and why I think it
essential.

- I am not talking about binary level plugins support and maintenance or
sources level. 
- I am not talking about open source or closed source, commercial and
freeware devices for emulator, but I think the plugin architecture should
support both.
- I not think the plugin API should be C++, actually Bochs plugin API is
pure-C based. The device itself is written in C++ but it has something like
C++ -> C proxy implemented as a part of plugin API. I think here we are
agree.
- It is doesn't matter if the plugin is shared or static library, the code
size of Bochs or QEMU is so small and I don't think it is important feature
to "share" devices in memory.

The keyword is expendability. Now if you would like to write a new device
for Bochs or QEMU you cannot open a separate project on SF.net and start
development. Even if you need to add some exotic experimental device which
is not exists in real life you should interfere with simulator code.
Moreover, if you need to add this device to Bochs, QEMU and Xen you
basically have to do the work three times !

I hope I am not disclose any secret if I say that HiPEAC beginning a new
project of open source full-system simulator, something that could be
alternative to SimpleScalar and also have full-system emulation capabilities
(platform + power + performance in single box). The simulator would reuse
the devices from some existent open source emulator, Bochs or QEMU. But as
any academia model it should be highly extendable and super-modular. The
ability to add a new device to the system without touching the main
simulator kernel is a MUST. 

I believe it is right direction too. Somebody could establish a new project
of developing new device and when contribute the device when it will be
done. In my opinion, it is even not important if the device written in C/C++
or any other language, if it is distributed with sources or not, under
commercial license or not. If I could download device binary, plug it in to
the simulator and get new capabilities, which were not available before -
why not ?

I could provide more and more examples, more and more advantages of this
arch and almost no disadvantages. There are a lot of successful examples of
products with extendable user plugins where functionality of the product as
a very small piece near the variety of plugins written by users.

Now the advantages of making every device separate library (shared or
static). When every device is separate library you try to build your
emulated system by yourself choosing the CPU core emulator + chipset plugin
+ devices plugins you want. If some set of plugins distributed as single
library - they are not separatable one from another. This is only a
disadvantage which could simply avoided, so again, why not ?

I am already not talking about devices maintenance. The best example is
Brendan BIOS. Brendan started a new project of writing new ACPI-compatible
BIOS for open source emulators. His BIOS require ACPI compatibility changes
in the device models and CPU emulator and Bochs already started to support
his requirements. Currently, when his BIOS will be done, QEMU will remain
behind it and not be able to run with it. Anyway, to support the
requirements the work should be done 2-3 times, from the beginning for each
simulator.

The Bochs will start from some PCI device and try to separate it from the
simulator code, compile it stand-alone and make it pluginable in run-time.
Why PCI device ? Just because it has more functionality than PC-speaker card
and require more interference with core simulator. Actually might be several
device models for core plugins (chipset, PCI bus, AGP bus), ISA/PCI cards
and COM devices, everything is open. During the separation work I hope to
figure out requirements from devices and simulator core for new plugin
architecture. In Bochs, a lot of changes required to make it happen, and we
believe we'll make it possible until next Bochs 2.3 release.

Thanks,
Stanislav

-----Original Message-----
From: qemu-devel-bounces+stl=fidonet.org.il@nongnu.org
[mailto:qemu-devel-bounces+stl=fidonet.org.il@nongnu.org] On Behalf Of Jim
C. Brown
Sent: Sunday, April 09, 2006 6:08 PM
To: qemu-devel@nongnu.org
Subject: Re: [Qemu-devel] Unified device model

On Sun, Apr 09, 2006 at 04:21:42PM +0100, Paul Brook wrote:
> > > I'm not a fan of binary plugins (for the same reasons I'm don't like
> > > binary kernel modules), and don't think there's any real need to them.
> >
> > A binary plugin API and a source plugin API (one that requires each
driver
> > device to be recompiled for each of the platforms (Xen, qemu, bochs,
etc.)
> > would probably be equally hard to design and maintain.
> 
> You've missed my point. The only reason I can see for wanting binary
plugins 
> is so that people can distribute proprietary closed-source device
emulation.

I agree that proprietary or closed-source device emulation is a bad thing
and
should not be supported.

> 
> A stable source API is a prerequisite for any sort of binary plugins.
> 

In that case, perhaps a stable source API would be best.

Like I said before, the type of API/sharing (source vs binary API, and
static
vs shared libraries) is a separate issue from the one we are discussing
(should
we have or support a unified plugin API?).

> > > I can't see
> > > any good reasons why open source devices would need to be broken out
into
> > > a separate shared library.
> >
> > I think the case was already made for this.
> >
> > Xen's hardware emulation, while based on qemu's, is already ahead in
> > several aspects. A separate library would make it more convenient for
these
> > changes to be shared back with qemu. Or with E/OS.
> 
> I don't buy that. We either share the same drivers (in which case keeping
the 
> two in sync is trivial) or we don't. All of the systems under
consideration 
> are [L]GPL licences. We can easily copy the source, so I don't think being

> able to copy bits of binary goo gains us anything.

A) Makes it simpler for end users to move devices over (they don't have to
know
how to cut-and-paste C code)

B) Bochs is not related to qemu at all code-wise, so the cut-and-paste
senario
doesn't work here. If we want to share drivers with Bochs we'd need at least
a
source API. (The starter of this thread is a Bochs developer I believe...
but correct me if I'm wrong here. :) The alternative is to rewrite Bochs
drivers for qemu from scratch (possbly using the Bochs code as a guide) but
that is even harder than the qemu-xen case.

C) If they are in a special library (say maintained by a 3rd party group
that
consists of developers from all the major projects) then maintainance is
greatly
simplified over time.

> 
> I don't think executable size is a valid argument either. Device emulation

> code generally isn't that big so the overhead of breaking it up into
multiple 
> shared libraries would outweigh the benefits of not loading the bits
you're 
> not using.

Perhaps you are right about that. The size of having even 4 or 5 copies of
complete PC hardware emulation code isn't so large as to be a problem except
on systems that are either embedded or ancient (in which case you probably
have
no business running 4 different PC emulators anyways).

Personally, it just seems inelegant to have such code duplication.

> 
> Paul
> 

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.


_______________________________________________
Qemu-devel mailing list
Qemu-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/qemu-devel

Re: [Qemu-devel] Unified device model

[Fabrice Bellard]

Hi,

My feeling is that QEMU already has what is needed to make a "plugin 
API". In particular, each device driver is clearly separated from the 
rest of the emulator and adding a new "machine" just consists in adding 
a new C file instanciating the different devices and processors. The 
fact that the devices or machines cannot come from a DLL or cannot be 
compiled separately is a cosmetic issue and could be easily added. I see 
little advantage in it except to make binary device models which I don't 
want to promote.

The real issues I see to improve the QEMU device model for the short 
term are:

- Rename some structures or functions to have a more consistent naming 
and rename vl.h to something like qemu/qemu.h and eventually split it to 
have a few subsystems such as isa, pci, usb... The question whether we 
keep a QEMU prefix is mostly political.

- Implement reproductible execution support and see if API changes are 
needed in the devices.

- Add explicit contexts in the (rare) cases where there are not already 
explicit. The API shall allow any number of CPUs and buses. A device 
tree should be added to give a global name for each device and cpu that 
can be referenced in the monitor and in the traces.

- Improve the tracing API. It should be possible to enable traces for 
each instance of a device. The various I/Os and events should be 
displayed in a uniform way.

- Make it possible to integrate different CPU types and all devices in 
the same executables without loosing performance. There are some API 
issues to solve regarding endianness.

- Document the API various functions by adding javadoc comments.

Another interesting point would be to see how to interface with hardware 
device models (for example written in VHDL and compiled with GHDL).

Looking at the API of other projects such as MAME which are more mature 
than QEMU or Bochs in the sense that they have far more devices and 
processors would be interesting too.

Fabrice.

Stanislav Shwartsman wrote:
> Hello All,
> 
> I'll try to answer to everybody in single mail and explain my vision of
> unified plugin API and why I see them essential.
> 
> Yes, I am Bochs developer, but I am more CPU expert and only learning the
> devices subsystem and dynamic translation subjects so Bochs is a right place
> for me ;) I am pretty understand all the Bochs code including devices and
> also have some vision of highly portable and extendable simulator.
> 
> Now I'll try to explain how I see the plugin architecture and why I think it
> essential.
> 
> - I am not talking about binary level plugins support and maintenance or
> sources level. 
> - I am not talking about open source or closed source, commercial and
> freeware devices for emulator, but I think the plugin architecture should
> support both.
> - I not think the plugin API should be C++, actually Bochs plugin API is
> pure-C based. The device itself is written in C++ but it has something like
> C++ -> C proxy implemented as a part of plugin API. I think here we are
> agree.
> - It is doesn't matter if the plugin is shared or static library, the code
> size of Bochs or QEMU is so small and I don't think it is important feature
> to "share" devices in memory.
> 
> The keyword is expendability. Now if you would like to write a new device
> for Bochs or QEMU you cannot open a separate project on SF.net and start
> development. Even if you need to add some exotic experimental device which
> is not exists in real life you should interfere with simulator code.
> Moreover, if you need to add this device to Bochs, QEMU and Xen you
> basically have to do the work three times !
> 
> I hope I am not disclose any secret if I say that HiPEAC beginning a new
> project of open source full-system simulator, something that could be
> alternative to SimpleScalar and also have full-system emulation capabilities
> (platform + power + performance in single box). The simulator would reuse
> the devices from some existent open source emulator, Bochs or QEMU. But as
> any academia model it should be highly extendable and super-modular. The
> ability to add a new device to the system without touching the main
> simulator kernel is a MUST. 
> 
> I believe it is right direction too. Somebody could establish a new project
> of developing new device and when contribute the device when it will be
> done. In my opinion, it is even not important if the device written in C/C++
> or any other language, if it is distributed with sources or not, under
> commercial license or not. If I could download device binary, plug it in to
> the simulator and get new capabilities, which were not available before -
> why not ?
> 
> I could provide more and more examples, more and more advantages of this
> arch and almost no disadvantages. There are a lot of successful examples of
> products with extendable user plugins where functionality of the product as
> a very small piece near the variety of plugins written by users.
> 
> Now the advantages of making every device separate library (shared or
> static). When every device is separate library you try to build your
> emulated system by yourself choosing the CPU core emulator + chipset plugin
> + devices plugins you want. If some set of plugins distributed as single
> library - they are not separatable one from another. This is only a
> disadvantage which could simply avoided, so again, why not ?
> 
> I am already not talking about devices maintenance. The best example is
> Brendan BIOS. Brendan started a new project of writing new ACPI-compatible
> BIOS for open source emulators. His BIOS require ACPI compatibility changes
> in the device models and CPU emulator and Bochs already started to support
> his requirements. Currently, when his BIOS will be done, QEMU will remain
> behind it and not be able to run with it. Anyway, to support the
> requirements the work should be done 2-3 times, from the beginning for each
> simulator.
> 
> The Bochs will start from some PCI device and try to separate it from the
> simulator code, compile it stand-alone and make it pluginable in run-time.
> Why PCI device ? Just because it has more functionality than PC-speaker card
> and require more interference with core simulator. Actually might be several
> device models for core plugins (chipset, PCI bus, AGP bus), ISA/PCI cards
> and COM devices, everything is open. During the separation work I hope to
> figure out requirements from devices and simulator core for new plugin
> architecture. In Bochs, a lot of changes required to make it happen, and we
> believe we'll make it possible until next Bochs 2.3 release.
> 
> Thanks,
> Stanislav
> 
> -----Original Message-----
> From: qemu-devel-bounces+stl=fidonet.org.il@nongnu.org
> [mailto:qemu-devel-bounces+stl=fidonet.org.il@nongnu.org] On Behalf Of Jim
> C. Brown
> Sent: Sunday, April 09, 2006 6:08 PM
> To: qemu-devel@nongnu.org
> Subject: Re: [Qemu-devel] Unified device model
> 
> On Sun, Apr 09, 2006 at 04:21:42PM +0100, Paul Brook wrote:
> 
>>>>I'm not a fan of binary plugins (for the same reasons I'm don't like
>>>>binary kernel modules), and don't think there's any real need to them.
>>>
>>>A binary plugin API and a source plugin API (one that requires each
> 
> driver
> 
>>>device to be recompiled for each of the platforms (Xen, qemu, bochs,
> 
> etc.)
> 
>>>would probably be equally hard to design and maintain.
>>
>>You've missed my point. The only reason I can see for wanting binary
> 
> plugins 
> 
>>is so that people can distribute proprietary closed-source device
> 
> emulation.
> 
> I agree that proprietary or closed-source device emulation is a bad thing
> and
> should not be supported.
> 
> 
>>A stable source API is a prerequisite for any sort of binary plugins.
>>
> 
> 
> In that case, perhaps a stable source API would be best.
> 
> Like I said before, the type of API/sharing (source vs binary API, and
> static
> vs shared libraries) is a separate issue from the one we are discussing
> (should
> we have or support a unified plugin API?).
> 
> 
>>>>I can't see
>>>>any good reasons why open source devices would need to be broken out
> 
> into
> 
>>>>a separate shared library.
>>>
>>>I think the case was already made for this.
>>>
>>>Xen's hardware emulation, while based on qemu's, is already ahead in
>>>several aspects. A separate library would make it more convenient for
> 
> these
> 
>>>changes to be shared back with qemu. Or with E/OS.
>>
>>I don't buy that. We either share the same drivers (in which case keeping
> 
> the 
> 
>>two in sync is trivial) or we don't. All of the systems under
> 
> consideration 
> 
>>are [L]GPL licences. We can easily copy the source, so I don't think being
> 
> 
>>able to copy bits of binary goo gains us anything.
> 
> 
> A) Makes it simpler for end users to move devices over (they don't have to
> know
> how to cut-and-paste C code)
> 
> B) Bochs is not related to qemu at all code-wise, so the cut-and-paste
> senario
> doesn't work here. If we want to share drivers with Bochs we'd need at least
> a
> source API. (The starter of this thread is a Bochs developer I believe...
> but correct me if I'm wrong here. :) The alternative is to rewrite Bochs
> drivers for qemu from scratch (possbly using the Bochs code as a guide) but
> that is even harder than the qemu-xen case.
> 
> C) If they are in a special library (say maintained by a 3rd party group
> that
> consists of developers from all the major projects) then maintainance is
> greatly
> simplified over time.
> 
> 
>>I don't think executable size is a valid argument either. Device emulation
> 
> 
>>code generally isn't that big so the overhead of breaking it up into
> 
> multiple 
> 
>>shared libraries would outweigh the benefits of not loading the bits
> 
> you're 
> 
>>not using.
> 
> 
> Perhaps you are right about that. The size of having even 4 or 5 copies of
> complete PC hardware emulation code isn't so large as to be a problem except
> on systems that are either embedded or ancient (in which case you probably
> have
> no business running 4 different PC emulators anyways).
> 
> Personally, it just seems inelegant to have such code duplication.
> 
> 
>>Paul
>>
> 
> 

[Qemu-devel] Unified device model

[pete sullivan]

[-- Attachment #1: Type: text/plain, Size: 2681 bytes --]

Hi

I have been following the discussion about some more dynamic way to
configure and define a simulation set-up
for Qemu. If You want to attract others than hard core developers, I think
that an unified device model is
something to aim for. With a generic device model I think you open up a sea
of possibilities for Qemu.

There are for sure conflicting interests among the users (and potential
users) of Qemu (not to talk about the
developers), ranging from those who are satisfied with a monolith acting as
some well defined machine, to
those that want to runt different configurations on every launch. Take
beyond that,there might as well be a
need for some users to test configurations that is unimaginable be the
developers, as well as running a network
of interconnected machines in a deterministic way. But I hope that there is
a common denominator that
all can agree on, some minimal changes that are required to satisfy a larger
community of both users
and developers. I acknowledge that fact that Fabrice calls the shots, but I
only want to emphasize a few points
that I think are required to be solved for Qemu to really lift off.

What is the smallest set of requirements on a device model that Qemu
developers can embrace? Few
questions comes to my mind
* Is it a framework that requires the models to be open source, or is a
truly dynamic one with DLL files with
  well defined interfaces something to strive for? I strongly believe that a
dynamic model is a prerequisite to
  enable sharing of devices, open source or not.
* Scripting languages, what about those? Can we agree upon some language to
use for configurations
  definitions, like Haskel, M, Tcl, XML or elisp? A scripting language makes
it fairly easy for an inexperienced
  user to create and alters configurations, those configurations can also
describe well defined state for a machine.
  Think, in terms of not booting windowz every time you need it for some
obscure task.
* What about interface definition languages (IDL)? By using an IDL it is
possible to auto generate proxies
  for different requirements, like wrappers for devices developed for Bochs,
Mame and so on, as well as remove
  the not so important important part on how to interconnect components (the
technical aspects).
* All this comes with a performance penalty, is that acceptable? As soon as
you add any kind of dynamic
  device model, you more or less have to remove all global variables and
most of the global functions.


Fellow developers, what do you say? And most importantly, what do You,
Fabrice say, is the above something
that Qemu can evolve into?

BR
Pete

[-- Attachment #2: Type: text/html, Size: 2979 bytes --]

Re: [Qemu-devel] Unified device model

[Einar Larsson]

[-- Attachment #1: Type: text/plain, Size: 1805 bytes --]

I think that a device model with support for devices in shared libraries
combined with dynamic definition of the simulated HW would be great to be
able to support a large number of HW variants in QEmu.

I have developed several full system simulators for large embedded systems,
and we have always ended up with a systems with a very small component
runtime combined with a script based description of the system to simulate.
All parts of the system except the component loading and instantiation was
implemented as dynamically loaded components. I have worked on
implementations that use a pure C based component model as well as systems
that supports both C and C++ bindings for interface implementation.

I have not been able to find any information about the proposed Bochs
component model. Stanislav: where can I find any information about the Bochs
plugin architecture that you talk about?

Among the open source component frameworks i guess that XPCOM used in the
Mozilla products is one of the most interesting. It currently support C++,
Python and Java bindings but it can easily support C as well.

/Einar


I strongly support the idea of being able to use shared objects to be
> able to have a more dynamic device model; I can work on moving some of
> the drivers over to whatever new model that you figure out.
>
> It'd be nice too to have a dynamic board definition.  For instance,
> being able to describe in a configuration file a board with some odd
> configuration (eg. 10 serial port, no IDE controller) without
> recompiling would be helpful.  I'm not sure what kind of file format's
> appropriate.
>
> The number of supported devices on QEMU is going to explode... Having a
> good scalable architecture is going to be very useful.
>
> - Alex
>
>

[-- Attachment #2: Type: text/html, Size: 1989 bytes --]