[Qemu-devel] Modular qemu?

[Qemu-devel] Modular qemu?

[Joop Boonen]

Hello All,

I'm wondering if a modular qemu would be an option (loadable modules)?
Currently a lot different patched versions of qemu are around.
qemu as provided from the qemu project
qemu for openmoko wiki.openmoko.org
qemu for coreboot.org
and probably others.

If it would be possible t load modules like the (linux)kernel it would be
possible to use one qemu version for all the projects.
I don't know if this is possible?

Regards,

Joop.

Re: [Qemu-devel] Modular qemu?

[Carl-Daniel Hailfinger]

On 05.12.2008 11:53, Joop Boonen wrote:
> I'm wondering if a modular qemu would be an option (loadable modules)?
> Currently a lot different patched versions of qemu are around.
> qemu as provided from the qemu project
> qemu for openmoko wiki.openmoko.org
> qemu for coreboot.org
>   

The coreboot project tries to get their patches merged. Hopefully that
will reduce the amounts of coreboot/qemu patches to zero.

> and probably others.
>   

Regards,
Carl-Daniel

-- 
http://www.hailfinger.org/

Re: [Qemu-devel] Modular qemu?

[Anthony Liguori]

Joop Boonen wrote:
> Hello All,
>
> I'm wondering if a modular qemu would be an option (loadable modules)?
> Currently a lot different patched versions of qemu are around.
> qemu as provided from the qemu project
> qemu for openmoko wiki.openmoko.org
> qemu for coreboot.org
> and probably others.
>
> If it would be possible t load modules like the (linux)kernel it would be
> possible to use one qemu version for all the projects.
> I don't know if this is possible?
>   

This has been discussed before and I believe the overall consensus was 
that plugins are not desirable.

With respect to the various forks of QEMU, I believe the real problem is 
that historically, people have had a tough time getting changes into 
QEMU.  This is not just a matter of getting patches accepted, but also 
getting the appropriate guidance about how to refactor things to take 
into account all of the various architecture combinations that QEMU 
supports and some of the longer term efforts.

I hope this situation is improving.  If people have feedback in how 
things could be improved, I think everyone is eager to here it.  Plugins 
are not the solution though.

Regards,

Anthony Liguori

> Regards,
>
> Joop.
>
>
>
>   

Re: [Qemu-devel] Modular qemu?

[Laurent Desnogues]

On Fri, Dec 5, 2008 at 3:32 PM, Anthony Liguori <anthony@codemonkey.ws> wrote:
> With respect to the various forks of QEMU, I believe the real problem is
> that historically, people have had a tough time getting changes into QEMU.
>  This is not just a matter of getting patches accepted, but also getting the
> appropriate guidance about how to refactor things to take into account all
> of the various architecture combinations that QEMU supports and some of the
> longer term efforts.
>
> I hope this situation is improving.  If people have feedback in how things
> could be improved, I think everyone is eager to here it.  Plugins are not
> the solution though.

Sorry to say, but many forks are linux-user ones and qemu has no
maintainer for that.  This might become even worse in the future
with, for instance, Nokia using qemu linux-user for the SDK of their
upcoming OMAP3 based tablet.


Laurent

Re: [Qemu-devel] Modular qemu?

[Glauber Costa]

On Fri, Dec 5, 2008 at 12:52 PM, Laurent Desnogues
<laurent.desnogues@gmail.com> wrote:
> On Fri, Dec 5, 2008 at 3:32 PM, Anthony Liguori <anthony@codemonkey.ws> wrote:
>> With respect to the various forks of QEMU, I believe the real problem is
>> that historically, people have had a tough time getting changes into QEMU.
>>  This is not just a matter of getting patches accepted, but also getting the
>> appropriate guidance about how to refactor things to take into account all
>> of the various architecture combinations that QEMU supports and some of the
>> longer term efforts.
>>
>> I hope this situation is improving.  If people have feedback in how things
>> could be improved, I think everyone is eager to here it.  Plugins are not
>> the solution though.
>
> Sorry to say, but many forks are linux-user ones and qemu has no
> maintainer for that.  This might become even worse in the future
> with, for instance, Nokia using qemu linux-user for the SDK of their
> upcoming OMAP3 based tablet.

We have no maintainer here, and there's a lot of people maintaining the forks
all over. What's stoping those people from becoming maintainers of the
linux-user architectures
in upstream qemu?

I believe this is part of what anthony is saying. The situation about
how do we welcome
new people improved by a large leap.

Re: [Qemu-devel] Modular qemu?

[Anthony Liguori]

Laurent Desnogues wrote:
> On Fri, Dec 5, 2008 at 3:32 PM, Anthony Liguori <anthony@codemonkey.ws> wrote:
>   
>> With respect to the various forks of QEMU, I believe the real problem is
>> that historically, people have had a tough time getting changes into QEMU.
>>  This is not just a matter of getting patches accepted, but also getting the
>> appropriate guidance about how to refactor things to take into account all
>> of the various architecture combinations that QEMU supports and some of the
>> longer term efforts.
>>
>> I hope this situation is improving.  If people have feedback in how things
>> could be improved, I think everyone is eager to here it.  Plugins are not
>> the solution though.
>>     
>
> Sorry to say, but many forks are linux-user ones and qemu has no
> maintainer for that.  This might become even worse in the future
> with, for instance, Nokia using qemu linux-user for the SDK of their
> upcoming OMAP3 based tablet.
>   

Yeah, I understand that the linux-user stuff is currently unmaintained.  
Just appointing someone is really the right approach either.  With the 
kernel.org git mirror, it would be relatively easy for someone to 
maintain a linux-user tree.

If someone were to do that, they could post pull requests.  I think it's 
then natural to promote the maintainer of such a tree to an official 
maintainer.

It just requires someone willing to put in the time.  git is not the 
only option, just maintaining a patch queue for linux-user would be a 
good start too.

Regards,

Anthony Liguori

> Laurent
>
>
>   

Re: [Qemu-devel] Modular qemu?

[Anthony Liguori]

Glauber Costa wrote:
> We have no maintainer here, and there's a lot of people maintaining the forks
> all over. What's stoping those people from becoming maintainers of the
> linux-user architectures
> in upstream qemu?
>   

git can be really helpful here.  As I said, anyone who wants to start 
maintaining a git tree for a particular subsystem they're interested in, 
I'm more than happy to do git pulls and then push to svn via git-svn.  
Of course, all commits should be posted as patches and they should be 
well separated and bisectable, etc.  All discussions should happen on 
qemu-devel too.

I know part of the problem for maintainership is chicken-and-the-egg.  
If you don't have commit access, it's hard to prove that you would be a 
good maintainer.  git really helps this though because anyone can setup 
a git tree.

Regards,

Anthony Liguori

> I believe this is part of what anthony is saying. The situation about
> how do we welcome
> new people improved by a large leap.
>   

Re: [Qemu-devel] Modular qemu?

[Avi Kivity]

Anthony Liguori wrote:
> Plugins are not the solution though.

What about non-plugin dlopen()?  Right now building qemu (with all 
options enabled) requires a large amount of libraries, hence a lot of 
dependencies.  For example, a server setup that will only be used with 
-vnc needs to have SDL installed.  This will only get worse with opengl 
support.

I'm thinking of something similar to linux kernel modules: no backward 
compatible ABI, simply load-on-demand functionality that can be packaged 
separately to reduce dependencies.  With kvm integrated, we could even 
make the cpu emulator an optional loadable module.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] Modular qemu?

[Thiemo Seufer]

Avi Kivity wrote:
> Anthony Liguori wrote:
>> Plugins are not the solution though.
>
> What about non-plugin dlopen()?  Right now building qemu (with all  
> options enabled) requires a large amount of libraries, hence a lot of  
> dependencies.  For example, a server setup that will only be used with  
> -vnc needs to have SDL installed.  This will only get worse with opengl  
> support.
>
> I'm thinking of something similar to linux kernel modules: no backward  
> compatible ABI, simply load-on-demand functionality that can be packaged  
> separately to reduce dependencies.  With kvm integrated, we could even  
> make the cpu emulator an optional loadable module.

What problem would this solve?


Thiemo

[Qemu-devel] Re: Modular qemu?

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1606 bytes --]

Avi Kivity wrote:
> Anthony Liguori wrote:
>> Plugins are not the solution though.
> 
> What about non-plugin dlopen()?  Right now building qemu (with all
> options enabled) requires a large amount of libraries, hence a lot of
> dependencies.  For example, a server setup that will only be used with
> -vnc needs to have SDL installed.  This will only get worse with opengl
> support.
> 
> I'm thinking of something similar to linux kernel modules: no backward
> compatible ABI, simply load-on-demand functionality that can be packaged
> separately to reduce dependencies.  With kvm integrated, we could even
> make the cpu emulator an optional loadable module.

At work, we are currently using this model to separate a very special
machine emulation from the latest and greatest qemu (or kvm-userspace)
core. We only patch the core with support to load "machine libraries"
that provide additional QEMUMachine definitions. The idea is definitely
_not_ to distribute a binary-only machine library with the product one
day, but to ease the build process. We also face the unavoidable API
breakages from time to time, and the whole things is not really clean
due to callbacks into the core all over the place, but otherwise it does
its job.

As it is not clean and we are still busy with more important stuff, I
haven't posted any RFC yet. The biggest issue, besides finding a
consensus if such pluging concepts are desired at all, is working out
clean (but not necessarily stable) APIs for calling from the plugins
into the core. Right now we simply export everything.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Qemu-devel] Modular qemu?

[Anthony Liguori]

Avi Kivity wrote:
> Anthony Liguori wrote:
>> Plugins are not the solution though.
>
> What about non-plugin dlopen()?  Right now building qemu (with all 
> options enabled) requires a large amount of libraries, hence a lot of 
> dependencies.  For example, a server setup that will only be used with 
> -vnc needs to have SDL installed.  This will only get worse with 
> opengl support.

Practically speaking, how helpful is this?  You still need to have the 
libraries present at build time and it's arguable about how much text 
savings you get because there's some cruft added from loading the 
libraries themselves.

If we had cleaner, self-registering interfaces, doing the dlopen() is 
real easy.  I definitely think we should move to that model.  The 
dlopen() part is somewhat orthogonal.

Regards,

Anthony Liguori

> I'm thinking of something similar to linux kernel modules: no backward 
> compatible ABI, simply load-on-demand functionality that can be 
> packaged separately to reduce dependencies.  With kvm integrated, we 
> could even make the cpu emulator an optional loadable module.

Re: [Qemu-devel] Modular qemu?

[Avi Kivity]

Thiemo Seufer wrote:
>> What about non-plugin dlopen()?  Right now building qemu (with all  
>> options enabled) requires a large amount of libraries, hence a lot of  
>> dependencies.  For example, a server setup that will only be used with  
>> -vnc needs to have SDL installed.  This will only get worse with opengl  
>> support.
>>
>> I'm thinking of something similar to linux kernel modules: no backward  
>> compatible ABI, simply load-on-demand functionality that can be packaged  
>> separately to reduce dependencies.  With kvm integrated, we could even  
>> make the cpu emulator an optional loadable module.
>>     
>
> What problem would this solve?
>
>   

You build qemu once, but with separate sub-packages:

  qemu
  qemu-sdl
  qemu-vnc
  qemu-tcg
  qemu-kvm

On a desktop deployment, you install qemu, qemu-kvm, qemu-tcg, and 
qemu-sdl (which pulls a bunch of dependencies).  On a server deployment, 
you install qemu, qemu-vnc, and qemu-kvm (which avoids dependencies and 
the tcg code which is not useful on a server).  This allows a 
distribution to only maintain and support one qemu binary for different 
deployment scenarios.

We might even make the various targets loadable modules, so you have a 
single binary which supports a bunch of targets.  This is more difficult 
due to the large numbers of #defines, but is doable.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] Modular qemu?

[Avi Kivity]

Anthony Liguori wrote:
>>> Plugins are not the solution though.
>>
>> What about non-plugin dlopen()?  Right now building qemu (with all 
>> options enabled) requires a large amount of libraries, hence a lot of 
>> dependencies.  For example, a server setup that will only be used 
>> with -vnc needs to have SDL installed.  This will only get worse with 
>> opengl support.
>
> Practically speaking, how helpful is this?  You still need to have the 
> libraries present at build time

Build time is not an issue; distros usually build with all possible 
options enabled.

> and it's arguable about how much text savings you get because there's 
> some cruft added from loading the libraries themselves.

The goal is not to save text (though that's an added benefit), but to 
drop an infinite chain of dependencies.  Right now, to have both desktop 
and server deployments, there are two options:

- build two different binaries with different build-time options.  
Distro maintainers will hug and kiss you whenever you mention this option.
- install SDL, X11 client libraries, and their dependencies on the 
server.  In the future, this gets worse, with opengl libraries, DRI 
drivers, and maybe even gtk2 and qt for a sane UI.  Sysadmins will 
propose on the spot if you suggest this to them.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] Modular qemu?

[Anthony Liguori]

Avi Kivity wrote:
>
> You build qemu once, but with separate sub-packages:
>
>  qemu
>  qemu-sdl
>  qemu-vnc
>  qemu-tcg
>  qemu-kvm

Fair enough, but we're getting the cart before the horse.  We need to 
have clean internal separations between these components before a shared 
library interface really matters.  Note, this is really just dividing 
QEMU into shared libraries, it's not really a plugin mechanism.

I'm less interested in the shared library bits (although I understand 
it's usefulness from a distro perspective) and more interested in being 
able to build out a lot of these things.

Regards,

Anthony Liguori

Re: [Qemu-devel] Modular qemu?

[Avi Kivity]

Anthony Liguori wrote:
> Avi Kivity wrote:
>>
>> You build qemu once, but with separate sub-packages:
>>
>>  qemu
>>  qemu-sdl
>>  qemu-vnc
>>  qemu-tcg
>>  qemu-kvm
>
> Fair enough, but we're getting the cart before the horse.  We need to 
> have clean internal separations between these components before a 
> shared library interface really matters.  Note, this is really just 
> dividing QEMU into shared libraries, it's not really a plugin mechanism.

Right, I even introduced it as such.  I don't think the separation will 
be difficult (not does it need to be perfect, given that we're not 
committing to an ABI).

>
> I'm less interested in the shared library bits (although I understand 
> it's usefulness from a distro perspective) and more interested in 
> being able to build out a lot of these things.

Parse failure.  What does building out mean? ./configure --disable-blah?

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] Modular qemu?

[Anthony Liguori]

Avi Kivity wrote:
> Anthony Liguori wrote:
>
>>
>> I'm less interested in the shared library bits (although I understand 
>> it's usefulness from a distro perspective) and more interested in 
>> being able to build out a lot of these things.
>
> Parse failure.  What does building out mean? ./configure --disable-blah?

In the least, yes.  I'd like a kernel style .config though so that each 
hardware device can be conditionally built or not built.  Adding a ='m' 
option is pretty obvious addition to this.

Regards,

Anthony Liguori

Re: [Qemu-devel] Modular qemu?

[Edgar E. Iglesias]

On Fri, Dec 05, 2008 at 08:32:45AM -0600, Anthony Liguori wrote:
> Joop Boonen wrote:
>> Hello All,
>>
>> I'm wondering if a modular qemu would be an option (loadable modules)?
>> Currently a lot different patched versions of qemu are around.
>> qemu as provided from the qemu project
>> qemu for openmoko wiki.openmoko.org
>> qemu for coreboot.org
>> and probably others.
>>
>> If it would be possible t load modules like the (linux)kernel it would be
>> possible to use one qemu version for all the projects.
>> I don't know if this is possible?
>>   
>
> This has been discussed before and I believe the overall consensus was that 
> plugins are not desirable.
>
> With respect to the various forks of QEMU, I believe the real problem is 
> that historically, people have had a tough time getting changes into QEMU.  
> This is not just a matter of getting patches accepted, but also getting the 
> appropriate guidance about how to refactor things to take into account all 
> of the various architecture combinations that QEMU supports and some of the 
> longer term efforts.
>
> I hope this situation is improving.  If people have feedback in how things 

In general I think the situation has improved alot. Personally I think
the work that in particular Anthony (others too) is doing with reviewing
and applying patches is just fantastic.

> could be improved, I think everyone is eager to here it.  Plugins are not 
> the solution though.

Personnaly I would welcome some kind of plugin interface for some parts of
QEMU but only if someone comes up wiht a an interface that does not hurt
performace. I'd definitely have a use to plugin callbacks for every memory
accesse in system-mode and I've noticed a reocurring request for hooks to
trap syscalls while user-mode emulating.

Best regards
Edgar

Re: [Qemu-devel] Modular qemu?

[Lionel Landwerlin]

[-- Attachment #1: Type: text/plain, Size: 1078 bytes --]

Le vendredi 05 décembre 2008 à 23:10 +0100, Edgar E. Iglesias a écrit :
> On Fri, Dec 05, 2008 at 08:32:45AM -0600, Anthony Liguori wrote:
> > Joop Boonen wrote:
> >> Hello All,
> >>
> 
> Personnaly I would welcome some kind of plugin interface for some parts of
> QEMU but only if someone comes up wiht a an interface that does not hurt
> performace. I'd definitely have a use to plugin callbacks for every memory
> accesse in system-mode and I've noticed a reocurring request for hooks to
> trap syscalls while user-mode emulating.
> 
> Best regards
> Edgar
> 
> 

Hello everyone,

Edgar,

that's exactly what I implemented the past week. I needed to simulate a
proprietary driver from a syscall point of view.

I added hooks for syscalls with basic priorities.

Here a few patch (it's a work in progress).

I'm also interrested in splitting the big sys.call.c (~6k lines) using
theses hooks.

Regards,

-- 
Lione Landwerlin                                         

O p e n W i d e                    14, rue Gaillon 75002 Paris

[-- Attachment #2: 0001-Added-syscall-hooks.patch --]
[-- Type: application/mbox, Size: 6858 bytes --]

[-- Attachment #3: 0002-Rename-syscall_hook-functions.patch --]
[-- Type: application/mbox, Size: 5565 bytes --]

[-- Attachment #4: 0003-Integration-of-syscall-hooks.patch --]
[-- Type: application/mbox, Size: 3881 bytes --]

[-- Attachment #5: 0004-hooks-integration.patch --]
[-- Type: application/mbox, Size: 3221 bytes --]

[-- Attachment #6: 0005-Splitted-syscalls-static-arrays.patch --]
[-- Type: application/mbox, Size: 1869 bytes --]

Re: [Qemu-devel] Modular qemu?

[Stefano Stabellini]

Anthony Liguori wrote:

> This has been discussed before and I believe the overall consensus was
> that plugins are not desirable.
> 
> With respect to the various forks of QEMU, I believe the real problem is
> that historically, people have had a tough time getting changes into
> QEMU.  This is not just a matter of getting patches accepted, but also
> getting the appropriate guidance about how to refactor things to take
> into account all of the various architecture combinations that QEMU
> supports and some of the longer term efforts.
> 
> I hope this situation is improving.  If people have feedback in how
> things could be improved, I think everyone is eager to here it.  Plugins
> are not the solution though.
> 


I think that getting patches accepted (or at least reviewed) is far more
important than plugins or shared libraries.
>From my point of view surely the situation has improved a lot recently
on this aspect.

Re: [Qemu-devel] Modular qemu?

[Salvatore Lionetti]

>Anthony Liguori wrote:
>
>> This has been discussed before and I believe the overall consensus was
>> that plugins are not desirable.
>> 
>> With respect to the various forks of QEMU, I believe the real problem is
>> that historically, people have had a tough time getting changes into
>> QEMU.  This is not just a matter of getting patches accepted, but also
>> getting the appropriate guidance about how to refactor things to take
>> into account all of the various architecture combinations that QEMU
>> supports and some of the longer term efforts.
>> 
>> I hope this situation is improving.  If people have feedback in how
>> things could be improved, I think everyone is eager to here it.  Plugins
>> are not the solution though.
>> 
>
>
>I think that getting patches accepted (or at least reviewed) is far more
>important than plugins or shared libraries.
>>From my point of view surely the situation has improved a lot recently
>on this aspect.

Re: [Qemu-devel] Modular qemu?

[Jun Koi]

On Sat, Dec 6, 2008 at 4:12 AM, Avi Kivity <avi@redhat.com> wrote:
> Anthony Liguori wrote:
>>>>
>>>> Plugins are not the solution though.
>>>
>>> What about non-plugin dlopen()?  Right now building qemu (with all
>>> options enabled) requires a large amount of libraries, hence a lot of
>>> dependencies.  For example, a server setup that will only be used with -vnc
>>> needs to have SDL installed.  This will only get worse with opengl support.
>>
>> Practically speaking, how helpful is this?  You still need to have the
>> libraries present at build time
>
> Build time is not an issue; distros usually build with all possible options
> enabled.
>
>> and it's arguable about how much text savings you get because there's some
>> cruft added from loading the libraries themselves.
>
> The goal is not to save text (though that's an added benefit), but to drop
> an infinite chain of dependencies.  Right now, to have both desktop and
> server deployments, there are two options:
>
> - build two different binaries with different build-time options.  Distro
> maintainers will hug and kiss you whenever you mention this option.
> - install SDL, X11 client libraries, and their dependencies on the server.
>  In the future, this gets worse, with opengl libraries, DRI drivers, and
> maybe even gtk2 and qt for a sane UI.

Interesting! Are you aware of any project currently doing on DRI
drivers?? Anh OpenGL drivers?

I suppose that you want to say "DRI device model" and "OpenGL device
model" for QEMU?

Thanks,
Jun

Re: [Qemu-devel] Modular qemu?

[Stefano Stabellini]

Jun Koi wrote:

> Interesting! Are you aware of any project currently doing on DRI
> drivers?? Anh OpenGL drivers?
> 

I posted a patch to implement OpenGL rendering few months ago but was
not applied.
If I have the time I'll try to clean it up and repost it in the future.

Re: [Qemu-devel] Modular qemu?

[Avi Kivity]

Jun Koi wrote:
>> - install SDL, X11 client libraries, and their dependencies on the server.
>>  In the future, this gets worse, with opengl libraries, DRI drivers, and
>> maybe even gtk2 and qt for a sane UI.
>>     
>
> Interesting! Are you aware of any project currently doing on DRI
> drivers?? Anh OpenGL drivers?
>
> I suppose that you want to say "DRI device model" and "OpenGL device
> model" for QEMU?
>   

No, I'm talking about using opengl to accelerate rendering of the normal 
2D emulated cards.  The Xen people have posted patches to do this.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Modular qemu?

[Ian Jackson]

Anthony Liguori writes ("Re: [Qemu-devel] Modular qemu?"):
> With respect to the various forks of QEMU, I believe the real problem is 
> that historically, people have had a tough time getting changes [in].
> 
> I hope this situation is improving.  If people have feedback in how 
> things could be improved, I think everyone is eager to here it.  Plugins 
> are not the solution though.

Yes, I agree that it has improved an awful lot.  (Sorry that I'm a
week behind on reading this list.)

It's really good to see such rapid movement even though it means that
downstreams like me have a bigger job each time we want to merge up.

Ian.