[RFC] QEMU as Xcode project on macOS

[RFC] QEMU as Xcode project on macOS

[Christian Schoenebeck]

I've recently been thinking about how feasible a stripped down Xcode project 
for QEMU would be, i.e. you just get the QEMU sources, click on 
qemu.xcodeproj, Cmd + B, done. No extra installation, no configure, nothing.

I've done this before for gtk(mm), which you might know, depends on approx. 24 
individual libraries (glib, jpeg, png, pixman, atk, gdk, cairo, pixman, 
graphene, sigcpp, ... gtk, gtkmm) that you would usually all need to download 
and

	./configure && make & make install

each individually on macOS. Or right, you could alternatively "just install" 
them from Homebrew, MacPorts, Fink. But no matter which solution you choose, 
it easily ends up in a mess (conflicts, misbehaviours) on macOS to install 
libs and apps globally. And I think that's the problem why there are currently 
relatively little contribution for QEMU coming from devs on macOS. Because you 
don't want to install things globally on a macOS system, it's simply not 
working well there as it does with Linux distros.

And the other thing is: I've tested the waters with Apple and filed a QEMU 
related macOS bug with them. The response was like expected; they basically 
said 'if there's no Xcode project, then we don't investigate it'.

The question is, and I don't have the big picture of QEMU yet to judge that, 
how much is auto generated for QEMU i.e. with custom scripts that would 
probably destroy this plan? There are these trace calls that are auto 
generated, is there more like the TCG part for instance?

What I could imagine: a hand crafted Xcode project as a starting point, and if  
that works out, switching to auto generating that Xcode project from the Meson 
inftrastructure to avoid multiplication of maintenance effort.

Best regards,
Christian Schoenebeck

Re: [RFC] QEMU as Xcode project on macOS

[Peter Maydell]

On Wed, 9 Sep 2020 at 13:56, Christian Schoenebeck
<qemu_oss@crudebyte.com> wrote:
> I've recently been thinking about how feasible a stripped down Xcode project
> for QEMU would be, i.e. you just get the QEMU sources, click on
> qemu.xcodeproj, Cmd + B, done. No extra installation, no configure, nothing.

How would this work? Can you have an XCode project that just
says "to build this program we will run configure and make" ?
We definitely don't want to have two parallel mechanisms for
specifying how to build QEMU, because they'll just get out of sync.

> each individually on macOS. Or right, you could alternatively "just install"
> them from Homebrew, MacPorts, Fink. But no matter which solution you choose,
> it easily ends up in a mess (conflicts, misbehaviours) on macOS to install
> libs and apps globally. And I think that's the problem why there are currently
> relatively little contribution for QEMU coming from devs on macOS. Because you
> don't want to install things globally on a macOS system, it's simply not
> working well there as it does with Linux distros.

My experience with homebrew has been pretty good overall.
If there's a better way to handle OSX hosts that doesn't
require us to carry around all our dependencies (which is
impractical) that would be interesting to investigate.

> The question is, and I don't have the big picture of QEMU yet to judge that,
> how much is auto generated for QEMU i.e. with custom scripts that would
> probably destroy this plan? There are these trace calls that are auto
> generated, is there more like the TCG part for instance?

Lots of stuff is autogenerated, yes.

thanks
-- PMM

Re: [RFC] QEMU as Xcode project on macOS

[Liviu Ionescu]

> On 9 Sep 2020, at 16:30, Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> My experience with homebrew has been pretty good overall.

Homebrew is fine for use cases that don't mind it is a rolling distribution. :-(

For complex builds it is safer to lock the environment to certain known to work versions, and homebrew cannot do this. You can disable the auto-update feature, and keep it locked to a set of versions, but usually only within a macOS version. After you update the system, usually you have to update homebrew.

For my production xPack builds, I simply could not use homebrew, I had to recreate a build environment in a separate folder, with the desired versions of the tools.


Regards,

Liviu

Re: [RFC] QEMU as Xcode project on macOS

[Christian Schoenebeck]

On Mittwoch, 9. September 2020 15:30:09 CEST Peter Maydell wrote:
> On Wed, 9 Sep 2020 at 13:56, Christian Schoenebeck
> 
> <qemu_oss@crudebyte.com> wrote:
> > I've recently been thinking about how feasible a stripped down Xcode
> > project for QEMU would be, i.e. you just get the QEMU sources, click on
> > qemu.xcodeproj, Cmd + B, done. No extra installation, no configure,
> > nothing.
> How would this work? Can you have an XCode project that just
> says "to build this program we will run configure and make" ?

You can run shell scripts. There are two main mechanisms in Xcode for that, 
one is "Build phases" where you can define shell code being executed for an 
individual "target" (e.g. a lib/app being built), which shell to be used (e.g. 
sh, zsh, bash, python), the actual shell code to be executed, and optionally a 
list of output files (e.g. .c files) said to be generated by the shell code 
which then in turn are compiled by Xcode according to their file type:

https://www.mokacoding.com/blog/better-build-phase-scripts/

Limitation: the precise list of generated output files needs to be specified 
to the Xcode project if they need to be compiled. So you cannot let the shell 
script dynamically tell Xcode "here are the source files I generated: [...] 
compile them please".

For that reason there is a 2nd mechanism in Xcode called "Build Rules", which 
basically allow you defining custom source file types, so e.g. Xcode would run 
a shell script for every source file that ends with "*.foo", the shell script 
would then generate for each one of them e.g. a .c file, which then would in 
turn be compiled by Xcode with clang.

In this example somebody auto generates .c files from .css files (sorry, dusty 
screenshots, but it's still the same thing):

https://www.cocoanetics.com/2012/02/xcode-build-rules/

> We definitely don't want to have two parallel mechanisms for
> specifying how to build QEMU, because they'll just get out of sync.

Sure, the Xcode project file should be auto generated for that reason, 
otherwise it's DOA.

> > each individually on macOS. Or right, you could alternatively "just
> > install" them from Homebrew, MacPorts, Fink. But no matter which solution
> > you choose, it easily ends up in a mess (conflicts, misbehaviours) on
> > macOS to install libs and apps globally. And I think that's the problem
> > why there are currently relatively little contribution for QEMU coming
> > from devs on macOS. Because you don't want to install things globally on
> > a macOS system, it's simply not working well there as it does with Linux
> > distros.
> 
> My experience with homebrew has been pretty good overall.
> If there's a better way to handle OSX hosts that doesn't
> require us to carry around all our dependencies (which is
> impractical) that would be interesting to investigate.

It probably depends on how many and which FOSS projects you compiled on Mac.
I no longer install from Homebrew and co, nor do I "pip install" things for 
the same reasons. And AFAICS other devs on Mac made similar experiences.

Plus I am used to FOSS projects having more issues on Mac then on Linux. If 
it's an Xcode generated app that crashes, then Xcode automatically opens the 
precise crashed lib's source file in the editor, along with the stacktrace, 
you fix the bug, click on "Run" and that's it. Without Xcode that would be 
like 45 mins just for recompiling the relevant dependencies.

On Mittwoch, 9. September 2020 15:40:05 CEST Daniel P. Berrangé wrote:
> Ideally any xcode setup would just invoke whatever our standard
> build tools are IMHO, so it has zero possibility of introducing
> new build problems.

Then you would not win anything on Mac. The problematic on macOS is that Apple 
froze many standard FOSS tools that switched to GPL3. So by default you just 
have e.g. GNU make 3.81 (2006), Bash 3.2.57 (2007), ...  unless you start to 
manually install them (e.g. from Homebrew & Co). And being forced to use Meson 
on Mac with all its Python dependencies does not make it easier.

IMO the Xcode project file would ideally be auto generated from the existing 
build infrastructure (e.g. Meson) and the scripts that are still required to 
be run by Xcode should be able to work with the old versions of build tools 
installed on a standard macOS system.

On Mittwoch, 9. September 2020 15:41:02 CEST Thomas Huth wrote:
> On 09/09/2020 14.56, Christian Schoenebeck wrote:
> > I've recently been thinking about how feasible a stripped down Xcode
> > project for QEMU would be, i.e. you just get the QEMU sources, click on
> > qemu.xcodeproj, Cmd + B, done. No extra installation, no configure,
> > nothing.
> Meson seems to have some exporter for Xcode according to
> https://mesonbuild.com/IDE-integration.html ... maybe you can harness
> that feature somehow?

Good to know, thanks! I'll give it a try.

> > The question is, and I don't have the big picture of QEMU yet to judge
> > that, how much is auto generated for QEMU i.e. with custom scripts that
> > would probably destroy this plan? There are these trace calls that are
> > auto generated, is there more like the TCG part for instance?
> 
> Yes, I think we generate code in a couple of places, e.g. the code in
> target/s390x/ uses a "gen-features" helper to generate some code. So
> implementing a separate Xcode project that does not use the main build
> files does not sound very appealing.

It's probably not that relevant how many files are generated, but more how 
many scripts approximately are generating these. If you just need to install 
like two dozens of shell hooks into the Xcode project, then Ok, that's 
manageable, if it's rather like hundreds or even thousands of script hooks, 
that would make the whole thing probably unrealistic, unless Meson's allegedly 
built-in Xcode feature already does a very great job. ;-)

On Mittwoch, 9. September 2020 16:40:10 CEST Programmingkid wrote:
> I think the solution to this problem is to switch over the CMake
> (https://en.wikipedia.org/wiki/CMake). It is a build system that lets you
> specify how you want to build software. There are many targets available
> including 'make' and an Xcode project file.

Well, the project just switched to Meson. It's probably not likely that there 
will be another build system switch any time soon I guess. But who knows, 
maybe Meson already does a good job on Xcode, or if not, maybe Meson could be 
fixed. We'll see.

Best regards,
Christian Schoenebeck

Re: [RFC] QEMU as Xcode project on macOS

[Liviu Ionescu]

> On 9 Sep 2020, at 20:32, Christian Schoenebeck <qemu_oss@crudebyte.com> wrote:
> 
> And being forced to use Meson 
> on Mac with all its Python dependencies does not make it easier.

FYI, I'm working on a cross-platform standalone meson package, a .tar.gz that you can install in any location and includes the python run-time.

I already have the build scripts functional for Windows (which is generally the most difficult platform); the build scripts for Linux/macOS will follow soon.

https://github.com/xpack-dev-tools/meson-build-xpack/blob/xpack-develop/scripts/common-apps-functions-source.sh


Regards,

Liviu

Re: [RFC] QEMU as Xcode project on macOS

[Daniel P. Berrangé]

> On Mittwoch, 9. September 2020 15:40:05 CEST Daniel P. Berrangé wrote:
> > Ideally any xcode setup would just invoke whatever our standard
> > build tools are IMHO, so it has zero possibility of introducing
> > new build problems.
> 
> Then you would not win anything on Mac. The problematic on macOS is that Apple 
> froze many standard FOSS tools that switched to GPL3. So by default you just 
> have e.g. GNU make 3.81 (2006), Bash 3.2.57 (2007), ...  unless you start to 
> manually install them (e.g. from Homebrew & Co). And being forced to use Meson 
> on Mac with all its Python dependencies does not make it easier.

It is going to need to use homebrew (or an equivalent) to bring in various
3rd party libraries QEMU depends on at build time, not least glib. So avoiding
homebrew for the build system doesn't appear like an obvious win to me.

I don't think QEMU wants to restrict itself to merely what is available in
the basic macOS install. We expect users to have to pull in stuff from
homebrew or an equivalent to get functionality that Apple has no intention
of ever shipping in their base OS install.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC] QEMU as Xcode project on macOS

[Christian Schoenebeck]

On Mittwoch, 9. September 2020 20:13:55 CEST Daniel P. Berrangé wrote:
> > On Mittwoch, 9. September 2020 15:40:05 CEST Daniel P. Berrangé wrote:
> > > Ideally any xcode setup would just invoke whatever our standard
> > > build tools are IMHO, so it has zero possibility of introducing
> > > new build problems.
> > 
> > Then you would not win anything on Mac. The problematic on macOS is that
> > Apple froze many standard FOSS tools that switched to GPL3. So by default
> > you just have e.g. GNU make 3.81 (2006), Bash 3.2.57 (2007), ...  unless
> > you start to manually install them (e.g. from Homebrew & Co). And being
> > forced to use Meson on Mac with all its Python dependencies does not make
> > it easier.
> 
> It is going to need to use homebrew (or an equivalent) to bring in various
> 3rd party libraries QEMU depends on at build time, not least glib.

External libraries are not the problem. AFAICS that's just libffi, glib and 
pixman. These could be added as submodules.

The more relevant ones would be the build tools that those QEMU scripts are 
running to auto generate source files.

> So avoiding homebrew for the build system doesn't appear like an obvious win
> to me.

I would see several benefits:

1. High potential of attracting much more devs on macOS if it's just click and 
   run.

2. Increased quality on Mac due to more devs.

3. Being able to delegate QEMU related macOS bugs to Apple.

4. *Much* more efficient work-flow when developing with the Xcode IDE
   (-> both increased development speed & quality).

On Mittwoch, 9. September 2020 19:45:04 CEST Liviu Ionescu wrote:
> FYI, I'm working on a cross-platform standalone meson package, a .tar.gz
> that you can install in any location and includes the python run-time.
> 
> I already have the build scripts functional for Windows (which is generally
> the most difficult platform); the build scripts for Linux/macOS will follow
> soon.
> 
> https://github.com/xpack-dev-tools/meson-build-xpack/blob/xpack-develop/scri
> pts/common-apps-functions-source.sh

Interesting! That would defuse the homebrew problematic. However you would 
still be opted out from (4.) all those Xcode IDE features that make 
development such efficient.

Best regards,
Christian Schoenebeck

Re: [RFC] QEMU as Xcode project on macOS

[Liviu Ionescu]

> On 9 Sep 2020, at 21:56, Christian Schoenebeck <qemu_oss@crudebyte.com> wrote:
> 
> ... However you would 
> still be opted out from (4.) all those Xcode IDE features that make 
> development such efficient.

Could you elaborate?

I thought that a standalone meson can be used with any environment, it just reduces the dependency on Python and possibly other system libraries, otherwise it provides the same functionality. You unpack the archive anywhere, place that location in the path and use it as usual.

Why would it be opted out for Xcode?


Thank you,

Liviu

Re: [RFC] QEMU as Xcode project on macOS

[Christian Schoenebeck]

On Mittwoch, 9. September 2020 21:03:32 CEST Liviu Ionescu wrote:
> > On 9 Sep 2020, at 21:56, Christian Schoenebeck <qemu_oss@crudebyte.com>
> > wrote:
> > 
> > ... However you would
> > still be opted out from (4.) all those Xcode IDE features that make
> > development such efficient.
> 
> Could you elaborate?
> 
> I thought that a standalone meson can be used with any environment, it just
> reduces the dependency on Python and possibly other system libraries,
> otherwise it provides the same functionality. You unpack the archive
> anywhere, place that location in the path and use it as usual.
> 
> Why would it be opted out for Xcode?

You would still not have a .xcodeproj file, which is a separate thing, as 
that's the only binding point for really making use of Xcode in an efficient 
way. Otherwise it does not have any knowledge about which source files belong 
to the project, what are the compiler flags, what preprocessor statements have 
to be applied and in which order. So you could still open source files in 
Xcode of course, but it would just be an ordinary plain text editor.

So this is about having or not having features like: auto completion, 
refactoring, one-click auto fixes of trivial issues, click to jump to 
definition/declaration (which might be in an external lib), built-in debugger, 
graphical static analysis, graphical sanitizers and much more ...

Best regards,
Christian Schoenebeck

Re: [RFC] QEMU as Xcode project on macOS

[Peter Maydell]

On Wed, 9 Sep 2020 at 19:56, Christian Schoenebeck
<qemu_oss@crudebyte.com> wrote:
> Interesting! That would defuse the homebrew problematic.

You still need a Python 3.5 or better for the other scripts
QEMU has as part of its build process, so you will still
need a python other than the Apple /usr/bin/python to build
even if you have a standalone meson with its own python.

thanks
-- PMM

Re: [RFC] QEMU as Xcode project on macOS

[Liviu Ionescu]

> On 9 Sep 2020, at 22:16, Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> ... You still need a Python 3.5 or better for the other scripts
> QEMU has as part of its build process, so you will still
> need a python other than the Apple /usr/bin/python to build
> even if you have a standalone meson with its own python.

Ah, sure, if the build script uses Python for other purposes you'll need one, but the embedded Python will allow the standalone meson to run properly regardless of what Python you install.

One less dependency to worry about is usually a good thing.


Regards,

Liviu

Re: [RFC] QEMU as Xcode project on macOS

[Paolo Bonzini]

On 09/09/20 22:13, Liviu Ionescu wrote:
>> ... You still need a Python 3.5 or better for the other scripts 
>> QEMU has as part of its build process, so you will still need a
>> python other than the Apple /usr/bin/python to build even if you
>> have a standalone meson with its own python.
>
> Ah, sure, if the build script uses Python for other purposes you'll
> need one, but the embedded Python will allow the standalone meson to
> run properly regardless of what Python you install.

Note that if you use the version of Meson that is bundled with QEMU, you
can use

  --python=/path/to/python3 --meson=internal

and the build will automatically use that same Python executable to run
the scripts.

Paolo

Re: [RFC] QEMU as Xcode project on macOS

[Liviu Ionescu]

> On 10 Sep 2020, at 12:20, Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
>  --python=/path/to/python3 --meson=internal
> 
> and the build will automatically use that same Python executable to run
> the scripts.

Ok, good to know.

My standalone meson is intended for integration into Eclipse and Visual Studio Code, and should also run on CI servers, so it has a different audience.

Regards,

Liviu

Re: [RFC] QEMU as Xcode project on macOS

[Daniel P. Berrangé]

On Wed, Sep 09, 2020 at 08:56:48PM +0200, Christian Schoenebeck wrote:
> On Mittwoch, 9. September 2020 20:13:55 CEST Daniel P. Berrangé wrote:
> > > On Mittwoch, 9. September 2020 15:40:05 CEST Daniel P. Berrangé wrote:
> > > > Ideally any xcode setup would just invoke whatever our standard
> > > > build tools are IMHO, so it has zero possibility of introducing
> > > > new build problems.
> > > 
> > > Then you would not win anything on Mac. The problematic on macOS is that
> > > Apple froze many standard FOSS tools that switched to GPL3. So by default
> > > you just have e.g. GNU make 3.81 (2006), Bash 3.2.57 (2007), ...  unless
> > > you start to manually install them (e.g. from Homebrew & Co). And being
> > > forced to use Meson on Mac with all its Python dependencies does not make
> > > it easier.
> > 
> > It is going to need to use homebrew (or an equivalent) to bring in various
> > 3rd party libraries QEMU depends on at build time, not least glib.
> 
> External libraries are not the problem. AFAICS that's just libffi, glib and 
> pixman. These could be added as submodules.
> 
> The more relevant ones would be the build tools that those QEMU scripts are 
> running to auto generate source files.

I don't think we want to be adding more 3rd party deps as submodules, quite
the opposite, we want to remove more submodules we currently have.

Bundling every important dep we use as a submodule and providing build rules,
means we're effectively re-inventing Homebrew / MacPorts and this is not a
sane use of our time.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC] QEMU as Xcode project on macOS

[Christian Schoenebeck]

On Donnerstag, 10. September 2020 09:37:10 CEST Daniel P. Berrangé wrote:
> I don't think we want to be adding more 3rd party deps as submodules, quite
> the opposite, we want to remove more submodules we currently have.
> 
> Bundling every important dep we use as a submodule and providing build
> rules, means we're effectively re-inventing Homebrew / MacPorts and this is
> not a sane use of our time.

Well, that's actually the whole point of this thread: saving developers' time. 
And I think the submodule solution is the most sane one.

Frankly if you compile FOSS software on Mac that asks you to "just" install 
dependencies with Homebrew and co, it feels like you have 2 jobs: a software 
developer, and a distribution maintainer. The difference to the submodule 
though: a much larger amount of developers have to do that maintainer job 
(manually resolving conflicts & crashes, rollbacks, etc.).

Best regards,
Christian Schoenebeck

Re: [RFC] QEMU as Xcode project on macOS

[Daniel P. Berrangé]

On Thu, Sep 10, 2020 at 11:32:35AM +0200, Christian Schoenebeck wrote:
> On Donnerstag, 10. September 2020 09:37:10 CEST Daniel P. Berrangé wrote:
> > I don't think we want to be adding more 3rd party deps as submodules, quite
> > the opposite, we want to remove more submodules we currently have.
> > 
> > Bundling every important dep we use as a submodule and providing build
> > rules, means we're effectively re-inventing Homebrew / MacPorts and this is
> > not a sane use of our time.
> 
> Well, that's actually the whole point of this thread: saving developers' time. 
> And I think the submodule solution is the most sane one.
>
> Frankly if you compile FOSS software on Mac that asks you to "just" install 
> dependencies with Homebrew and co, it feels like you have 2 jobs: a software 
> developer, and a distribution maintainer. The difference to the submodule 
> though: a much larger amount of developers have to do that maintainer job 
> (manually resolving conflicts & crashes, rollbacks, etc.).


I don't think it saves time. If you look at the bigger picture across
multiple project it costs time. Every project that depends on glib or
gtk or gnutls or etc  reinvents the wheel figuring out a suitable
recipe for bundling & building these dependencies. Worse is that they
will all do it slightly differently, or use a variety of versions, and
so users get a worse experiance too with different features available
and different things broken. Projects like HomeBrew exist precisely to
save developer time because these build steps can be figured out once,
and every project can now just rely on the well maintained HomeBrew
packages.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC] QEMU as Xcode project on macOS

[Christian Schoenebeck]

On Donnerstag, 10. September 2020 11:39:10 CEST Daniel P. Berrangé wrote:
> > Well, that's actually the whole point of this thread: saving developers'
> > time. And I think the submodule solution is the most sane one.
> > 
> > Frankly if you compile FOSS software on Mac that asks you to "just"
> > install
> > dependencies with Homebrew and co, it feels like you have 2 jobs: a
> > software developer, and a distribution maintainer. The difference to the
> > submodule though: a much larger amount of developers have to do that
> > maintainer job (manually resolving conflicts & crashes, rollbacks, etc.).
> 
> I don't think it saves time. If you look at the bigger picture across
> multiple project it costs time. Every project that depends on glib or
> gtk or gnutls or etc  reinvents the wheel figuring out a suitable
> recipe for bundling & building these dependencies. Worse is that they
> will all do it slightly differently, or use a variety of versions, and
> so users get a worse experiance too with different features available
> and different things broken. Projects like HomeBrew exist precisely to
> save developer time because these build steps can be figured out once,
> and every project can now just rely on the well maintained HomeBrew
> packages.

That only works for consumers at most.

For developers it is actually the complete opposite on Mac: you start to 
install things from somewhere, then you need to install something from 
somewhere else, manually build & install stuff, and you end up in conflicts 
and misbehaviours all over the place.

The way to go for devs on Mac is: 3rd party libs should not be installed into 
global space, rather be built & linked either as dynamic frameworks (including 
assets) or as static libs. Then apps always run with the precise version and 
flags of libs they were tested with and never conflict with another app's 
version/config of libs.

Best regards,
Christian Schoenebeck

Re: [RFC] QEMU as Xcode project on macOS

[Liviu Ionescu]

> On 10 Sep 2020, at 13:14, Christian Schoenebeck <qemu_oss@crudebyte.com> wrote:
> 
> The way to go for devs on Mac is: 3rd party libs should not be installed into 
> global space, rather be built & linked either as dynamic frameworks (including 
> assets) or as static libs. Then apps always run with the precise version and 
> flags of libs they were tested with and never conflict with another app's 
> version/config of libs.

Fully agree.

Just that I use the same concept for all my builds (Linux & Windows), not only the Mac builds.

Liviu

Re: [RFC] QEMU as Xcode project on macOS

[Peter Maydell]

On Thu, 10 Sep 2020 at 11:14, Christian Schoenebeck
<qemu_oss@crudebyte.com> wrote:
> For developers it is actually the complete opposite on Mac: you start to
> install things from somewhere, then you need to install something from
> somewhere else, manually build & install stuff, and you end up in conflicts
> and misbehaviours all over the place.

This has not been my experience -- homebrew has everything, and
it doesn't have problems like this.

> The way to go for devs on Mac is: 3rd party libs should not be installed into
> global space, rather be built & linked either as dynamic frameworks (including
> assets) or as static libs. Then apps always run with the precise version and
> flags of libs they were tested with and never conflict with another app's
> version/config of libs.

Does Apple (or anybody else) provide a framework for doing this
so that developers of individual applications can just say "my
app needs libs X, Y, Z" and they don't have to mess around
finding, figuring out how to compile, and shipping the sources of
libs X, Y, Z? If there's a better mechanism than Homebrew for this
that's great, but at the moment what you seem to be saying is
"you should do a lot more work to manually set something up where
you ship the sources to all your dependencies and then build them
all". There's no way we're ever going to do that, it is just
way too much work for very little gain.

thanks
-- PMM

Re: [RFC] QEMU as Xcode project on macOS

[Daniel P. Berrangé]

On Thu, Sep 10, 2020 at 11:35:46AM +0100, Peter Maydell wrote:
> On Thu, 10 Sep 2020 at 11:14, Christian Schoenebeck
> <qemu_oss@crudebyte.com> wrote:
> > For developers it is actually the complete opposite on Mac: you start to
> > install things from somewhere, then you need to install something from
> > somewhere else, manually build & install stuff, and you end up in conflicts
> > and misbehaviours all over the place.
> 
> This has not been my experience -- homebrew has everything, and
> it doesn't have problems like this.
> 
> > The way to go for devs on Mac is: 3rd party libs should not be installed into
> > global space, rather be built & linked either as dynamic frameworks (including
> > assets) or as static libs. Then apps always run with the precise version and
> > flags of libs they were tested with and never conflict with another app's
> > version/config of libs.
> 
> Does Apple (or anybody else) provide a framework for doing this
> so that developers of individual applications can just say "my
> app needs libs X, Y, Z" and they don't have to mess around
> finding, figuring out how to compile, and shipping the sources of
> libs X, Y, Z? If there's a better mechanism than Homebrew for this
> that's great, but at the moment what you seem to be saying is
> "you should do a lot more work to manually set something up where
> you ship the sources to all your dependencies and then build them
> all". There's no way we're ever going to do that, it is just
> way too much work for very little gain.

AFAICT both MacPorts and HomeBrew can be installed into custom locations,
at least if you do a "from source" install of them, rather than using the
pre-built packages.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC] QEMU as Xcode project on macOS

[Liviu Ionescu]

> On 10 Sep 2020, at 13:45, Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> AFAICT both MacPorts and HomeBrew can be installed into custom locations,
> at least if you do a "from source" install of them, rather than using the
> pre-built packages.

Yes, I use this for HomeBrew, but generally you cannot select the individual versions for libraries and tools, the Ruby code used to build packages is in a Git, you can checkout a moment in time, but you get the selection of versions at that moment.

If you try using individual Ruby files for different packages, you generally run into dependency issues.


Regards,

Liviu

Re: [RFC] QEMU as Xcode project on macOS

[Christian Schoenebeck]

On Donnerstag, 10. September 2020 12:35:46 CEST Peter Maydell wrote:
> Does Apple (or anybody else) provide a framework for doing this
> so that developers of individual applications can just say "my
> app needs libs X, Y, Z" and they don't have to mess around
> finding, figuring out how to compile, and shipping the sources of
> libs X, Y, Z? If there's a better mechanism than Homebrew for this
> that's great, but at the moment what you seem to be saying is
> "you should do a lot more work to manually set something up where
> you ship the sources to all your dependencies and then build them
> all". There's no way we're ever going to do that, it is just
> way too much work for very little gain.

On Mac I have 'framework'ed & Xcoded the FOSS libs I need. And when I need to 
compile a new FOSS lib/app that requires libs x,y,z then I just drag those 
already existing Xcode projects as referenced subprojects into that new Xcode 
project:

https://developer.apple.com/library/archive/documentation/ToolsLanguages/
Conceptual/Xcode_Overview/WorkingonRelatedProjects.html

For instance if I have to compile some new app that needs gtk(mm), all I need 
to do is pulling in gtkmm.xcodeproj (e.g. per drag & drop) which in turn 
already references all its own dependencies (subsubsub..projects) like glib, 
cairo, etc. So I just click 'Run' and that's it.

And if the app triggers a crash in some of the subprojects, Xcode stops at the 
precise lib's source location, you fix it, click 'Run' and you're done in 
seconds. With the regular 'make' approach you would spend hours for that.

So that's a highly modular end-to-end-from-source approach. The precise 
location of the individual xcode projects does not matter BTW as Xcode 
references subprojects by UDID. You can move them around as you want and it 
still works. No need to install them at global space, predefined structures, 
or even hard coded locations, and no duplication of sources either.

I know, all this sounds very unorthodox what I am doing at first glance. But 
if you think about it, all it would take is if individual FOSS projects would 
start maintaining an .xcodeproj file, which in turn could then be referenced 
by UUID from other FOSS project's .xcodeproj file as dependency and you would 
get a low cost end-to-end-by-source solution, already on upstream level, i.e. 
without the need of having dedicated projects managing "distributions".

On Donnerstag, 10. September 2020 12:41:45 CEST Paolo Bonzini wrote:
> You're very welcome to do this, but what you're doing is effectively a
> *distribution of QEMU* that targets the "macOS build from source" scenario,
> for people that don't want to use Homebrew. It is *not* going to be used by
> QEMU developers, because QEMU developers have conflicting requirements:

Okay, no problem. I will see where to go from here. I am not even sure yet 
whether all those source generating scripts of QEMU would be a show stopper 
for what I have planned. I mean (counting): ... yes, still just 2 hands here.

If it turns out to work fine, then maybe I just push a branch of QEMU for Mac 
developers somewhere for a while, as I think it would lower the entry level 
for new developers on Mac tremendously. Apparently there is still more changes 
with Meson ahead anyway (getting rid of the Makefiles or whatever else is 
planned). So we'll see ...

Best regards,
Christian Schoenebeck

Re: [RFC] QEMU as Xcode project on macOS

[Paolo Bonzini]

On 10/09/20 16:40, Christian Schoenebeck wrote:
> If it turns out to work fine, then maybe I just push a branch of QEMU for Mac 
> developers somewhere for a while, as I think it would lower the entry level 
> for new developers on Mac tremendously. Apparently there is still more changes 
> with Meson ahead anyway (getting rid of the Makefiles or whatever else is 
> planned). So we'll see ...

If there are no submodule dependencies (capstone/libfdt/slirp),
effectively you can already behave as if the Makefiles did not exist and
build QEMU with ninja and "meson test".  So you could build your scripts
from there.

Paolo

Re: [RFC] QEMU as Xcode project on macOS

[Christian Schoenebeck]

On Freitag, 11. September 2020 19:19:05 CEST Paolo Bonzini wrote:
> On 10/09/20 16:40, Christian Schoenebeck wrote:
> > If it turns out to work fine, then maybe I just push a branch of QEMU for
> > Mac developers somewhere for a while, as I think it would lower the entry
> > level for new developers on Mac tremendously. Apparently there is still
> > more changes with Meson ahead anyway (getting rid of the Makefiles or
> > whatever else is planned). So we'll see ...
> 
> If there are no submodule dependencies (capstone/libfdt/slirp),
> effectively you can already behave as if the Makefiles did not exist and
> build QEMU with ninja and "meson test".  So you could build your scripts
> from there.
> 
> Paolo

Good to know, thanks!

Best regards,
Christian Schoenebeck

Re: [RFC] QEMU as Xcode project on macOS

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 2568 bytes --]

Il gio 10 set 2020, 11:32 Christian Schoenebeck <qemu_oss@crudebyte.com> ha
scritto:

> On Donnerstag, 10. September 2020 09:37:10 CEST Daniel P. Berrangé wrote:
> > I don't think we want to be adding more 3rd party deps as submodules,
> quite
> > the opposite, we want to remove more submodules we currently have.
> >
> > Bundling every important dep we use as a submodule and providing build
> > rules, means we're effectively re-inventing Homebrew / MacPorts and this
> is
> > not a sane use of our time.
>
> Well, that's actually the whole point of this thread: saving developers'
> time.
> And I think the submodule solution is the most sane one.
>

You're very welcome to do this, but what you're doing is effectively a
*distribution of QEMU* that targets the "macOS build from source" scenario,
for people that don't want to use Homebrew. It is *not* going to be used by
QEMU developers, because QEMU developers have conflicting requirements:

* Not getting in the business of maintaining a distribution of all their
dependencies (glib, pixman, etc.)

* Not wanting to maintain multiple build systems

both of which are non-negotiable.

What you might do is use the configure script and the ninja backend to
build all generated sources (produced by either configure, meson or make);
then copy those generated file over to a new build directory, invoke meson
again with the xcode backend, and distribute the result so that it is ready
to build from source using xcode. The resulting distribution is not
appropriate to develop QEMU, but it would be okay to install it and even
for some simple debugging.

The above approach still doesn't solve the problem of building glib and
friends as part of the same xcode project. Perhaps this can be fixed by
patching the xcodeproj that Meson generated.

The scripts needed to do so should be relatively stable and can certainly
be included in the upstream QEMU sources. You can't expect that other
people will test them unless you also include them somehow in our CI, but
just having the scripts would be a start.

Thanks,

Paolo



> Frankly if you compile FOSS software on Mac that asks you to "just"
> install
> dependencies with Homebrew and co, it feels like you have 2 jobs: a
> software
> developer, and a distribution maintainer. The difference to the submodule
> though: a much larger amount of developers have to do that maintainer job
> (manually resolving conflicts & crashes, rollbacks, etc.).
>
> Best regards,
> Christian Schoenebeck
>
>
>

[-- Attachment #2: Type: text/html, Size: 3589 bytes --]

Re: [RFC] QEMU as Xcode project on macOS

[Daniel P. Berrangé]

On Thu, Sep 10, 2020 at 12:41:45PM +0200, Paolo Bonzini wrote:
> Il gio 10 set 2020, 11:32 Christian Schoenebeck <qemu_oss@crudebyte.com> ha
> scritto:
> 
> > On Donnerstag, 10. September 2020 09:37:10 CEST Daniel P. Berrangé wrote:
> > > I don't think we want to be adding more 3rd party deps as submodules,
> > quite
> > > the opposite, we want to remove more submodules we currently have.
> > >
> > > Bundling every important dep we use as a submodule and providing build
> > > rules, means we're effectively re-inventing Homebrew / MacPorts and this
> > is
> > > not a sane use of our time.
> >
> > Well, that's actually the whole point of this thread: saving developers'
> > time.
> > And I think the submodule solution is the most sane one.
> >
> 
> You're very welcome to do this, but what you're doing is effectively a
> *distribution of QEMU* that targets the "macOS build from source" scenario,
> for people that don't want to use Homebrew. It is *not* going to be used by
> QEMU developers, because QEMU developers have conflicting requirements:
> 
> * Not getting in the business of maintaining a distribution of all their
> dependencies (glib, pixman, etc.)
> 
> * Not wanting to maintain multiple build systems
> 
> both of which are non-negotiable.
> 
> What you might do is use the configure script and the ninja backend to
> build all generated sources (produced by either configure, meson or make);
> then copy those generated file over to a new build directory, invoke meson
> again with the xcode backend, and distribute the result so that it is ready
> to build from source using xcode. The resulting distribution is not
> appropriate to develop QEMU, but it would be okay to install it and even
> for some simple debugging.
> 
> The above approach still doesn't solve the problem of building glib and
> friends as part of the same xcode project. Perhaps this can be fixed by
> patching the xcodeproj that Meson generated.
> 
> The scripts needed to do so should be relatively stable and can certainly
> be included in the upstream QEMU sources. You can't expect that other
> people will test them unless you also include them somehow in our CI, but
> just having the scripts would be a start.

I'm not convinced we want to take what amounts to distribution packaging
into the main QEMU repo, as QEMU has generally stayed out of this area
leaving it to be done as downstream users/projects desire.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC] QEMU as Xcode project on macOS

[Daniel P. Berrangé]

On Wed, Sep 09, 2020 at 02:56:46PM +0200, Christian Schoenebeck wrote:
> I've recently been thinking about how feasible a stripped down Xcode project 
> for QEMU would be, i.e. you just get the QEMU sources, click on 
> qemu.xcodeproj, Cmd + B, done. No extra installation, no configure, nothing.
> 
> I've done this before for gtk(mm), which you might know, depends on approx. 24 
> individual libraries (glib, jpeg, png, pixman, atk, gdk, cairo, pixman, 
> graphene, sigcpp, ... gtk, gtkmm) that you would usually all need to download 
> and
> 
> 	./configure && make & make install
> 
> each individually on macOS. Or right, you could alternatively "just install" 
> them from Homebrew, MacPorts, Fink. But no matter which solution you choose, 
> it easily ends up in a mess (conflicts, misbehaviours) on macOS to install 
> libs and apps globally. And I think that's the problem why there are currently 
> relatively little contribution for QEMU coming from devs on macOS. Because you 
> don't want to install things globally on a macOS system, it's simply not 
> working well there as it does with Linux distros.
> 
> And the other thing is: I've tested the waters with Apple and filed a QEMU 
> related macOS bug with them. The response was like expected; they basically 
> said 'if there's no Xcode project, then we don't investigate it'.
> 
> The question is, and I don't have the big picture of QEMU yet to judge that, 
> how much is auto generated for QEMU i.e. with custom scripts that would 
> probably destroy this plan? There are these trace calls that are auto 
> generated, is there more like the TCG part for instance?
> 
> What I could imagine: a hand crafted Xcode project as a starting point, and if  
> that works out, switching to auto generating that Xcode project from the Meson 
> inftrastructure to avoid multiplication of maintenance effort.

The current way we generate a makefile from ninja output is not our
long term desired approach. Eventually the intent is that we should
be able to use  meson + ninja exclusively.

The ninja -> make convertor we currently rely on introduces maint
problems of its own. So I don't think we want to introduce a
ninja -> Xcode converter, as that is still effectively giving us
1 + 1/2 different build systems, so is a new maint burden.

Ideally any xcode setup would just invoke whatever our standard
build tools are IMHO, so it has zero possibility of introducing
new build problems.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC] QEMU as Xcode project on macOS

[Thomas Huth]

On 09/09/2020 14.56, Christian Schoenebeck wrote:
> I've recently been thinking about how feasible a stripped down Xcode project 
> for QEMU would be, i.e. you just get the QEMU sources, click on 
> qemu.xcodeproj, Cmd + B, done. No extra installation, no configure, nothing.

Meson seems to have some exporter for Xcode according to
https://mesonbuild.com/IDE-integration.html ... maybe you can harness
that feature somehow?

> The question is, and I don't have the big picture of QEMU yet to judge that, 
> how much is auto generated for QEMU i.e. with custom scripts that would 
> probably destroy this plan? There are these trace calls that are auto 
> generated, is there more like the TCG part for instance?

Yes, I think we generate code in a couple of places, e.g. the code in
target/s390x/ uses a "gen-features" helper to generate some code. So
implementing a separate Xcode project that does not use the main build
files does not sound very appealing.

 Thomas

Re: [RFC] QEMU as Xcode project on macOS

[Daniel P. Berrangé]

On Wed, Sep 09, 2020 at 03:41:02PM +0200, Thomas Huth wrote:
> On 09/09/2020 14.56, Christian Schoenebeck wrote:
> > I've recently been thinking about how feasible a stripped down Xcode project 
> > for QEMU would be, i.e. you just get the QEMU sources, click on 
> > qemu.xcodeproj, Cmd + B, done. No extra installation, no configure, nothing.
> 
> Meson seems to have some exporter for Xcode according to
> https://mesonbuild.com/IDE-integration.html ... maybe you can harness
> that feature somehow?

I feel like using the Meson XCode generator still ends up giving us
1 + 1/2 distinct build systems, as I very much doubt that there is
a guarantee of behavioural parity between its XCode output and Ninja
output.

> 
> > The question is, and I don't have the big picture of QEMU yet to judge that, 
> > how much is auto generated for QEMU i.e. with custom scripts that would 
> > probably destroy this plan? There are these trace calls that are auto 
> > generated, is there more like the TCG part for instance?
> 
> Yes, I think we generate code in a couple of places, e.g. the code in
> target/s390x/ uses a "gen-features" helper to generate some code. So
> implementing a separate Xcode project that does not use the main build
> files does not sound very appealing.
> 
>  Thomas
> 
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC] QEMU as Xcode project on macOS

[Paolo Bonzini]

On 09/09/20 15:41, Thomas Huth wrote:
>> The question is, and I don't have the big picture of QEMU yet to judge that, 
>> how much is auto generated for QEMU i.e. with custom scripts that would 
>> probably destroy this plan? There are these trace calls that are auto 
>> generated, is there more like the TCG part for instance?
> Yes, I think we generate code in a couple of places, e.g. the code in
> target/s390x/ uses a "gen-features" helper to generate some code. So
> implementing a separate Xcode project that does not use the main build
> files does not sound very appealing.

More than a couple places---all of QAPI and tracing code is autogenerated.

Perhaps Meson's xcodeproj backend can be improved to support everything
that QEMU requires, I'm not sure if there are any limitations in Xcode
that prevent that.

Paolo

Re: [RFC] QEMU as Xcode project on macOS

[Programmingkid]

> On Sep 9, 2020, at 8:56 AM, Christian Schoenebeck <qemu_oss@crudebyte.com> wrote:
> 
> I've recently been thinking about how feasible a stripped down Xcode project 
> for QEMU would be, i.e. you just get the QEMU sources, click on 
> qemu.xcodeproj, Cmd + B, done. No extra installation, no configure, nothing.
> 
> I've done this before for gtk(mm), which you might know, depends on approx. 24 
> individual libraries (glib, jpeg, png, pixman, atk, gdk, cairo, pixman, 
> graphene, sigcpp, ... gtk, gtkmm) that you would usually all need to download 
> and
> 
> 	./configure && make & make install
> 
> each individually on macOS. Or right, you could alternatively "just install" 
> them from Homebrew, MacPorts, Fink. But no matter which solution you choose, 
> it easily ends up in a mess (conflicts, misbehaviours) on macOS to install 
> libs and apps globally. And I think that's the problem why there are currently 
> relatively little contribution for QEMU coming from devs on macOS. Because you 
> don't want to install things globally on a macOS system, it's simply not 
> working well there as it does with Linux distros.
> 
> And the other thing is: I've tested the waters with Apple and filed a QEMU 
> related macOS bug with them. The response was like expected; they basically 
> said 'if there's no Xcode project, then we don't investigate it'.
> 
> The question is, and I don't have the big picture of QEMU yet to judge that, 
> how much is auto generated for QEMU i.e. with custom scripts that would 
> probably destroy this plan? There are these trace calls that are auto 
> generated, is there more like the TCG part for instance?
> 
> What I could imagine: a hand crafted Xcode project as a starting point, and if  
> that works out, switching to auto generating that Xcode project from the Meson 
> inftrastructure to avoid multiplication of maintenance effort.
> 
> Best regards,
> Christian Schoenebeck
> 

I think the solution to this problem is to switch over the CMake (https://en.wikipedia.org/wiki/CMake). It is a build system that lets you specify how you want to build software. There are many targets available including 'make' and an Xcode project file.