docker: how to use it when developing on QEMU?

docker: how to use it when developing on QEMU?

[Philippe Mathieu-Daudé]

Recently more developers are enthusiast to use Docker/Podman,
and have been confused by the different configurations currently in the
QEMU repository.

There are at least 3 kind of categories I use:

1/ Image used to build QEMU

These images should be restricted/updated to our "supported targets".
They are useful to (cross)build QEMU on variety of
host/target/distributions/distrib_versions.

Example:

- cross build Cris binary using the Fedora 30 toolchain on a Ubuntu
18.04 x86_64.
  host:Ubuntu18.04/x86_64 docker_image:fedora-cris-cross

- cross build MinGW64 binary using Debian 9 MXE toolchain on a Ubuntu
aarch64:
  host:Ubuntu18.04/x86_64 docker_image:debian-win64-cross

An image can not be meant to use on a daily basis, but to avoid
regression previous to release (I'd run them only on release candidate).

Example: building QEMU for the Gentoo PlayStation2 port:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg574468.html

2/ Image used to build test program used by QEMU

These images provide enough to build binaries you can then use to test
QEMU. If you want to build more of these binaries, there is probably a
better way. Here we are only interested in testing.

Example:

- Test PowerPC Linux-user binaries with qemu-powerpc-linux-user
  docker_image:debian-powerpc-user-cross

- Build EDK2 payload for Virt/AArch64
  It currently doesn't build with Fedora 30 and I'v to use a Fedora 29
image.

Another case I had is when I tried to build a kernel for the Mipssim
machine (supported by QEMU). The Linux kernel code has been removed, so
I had to checkout an old kernel which is not buildable with my recent
host GCC. Using a docker based on a very old distribution worked. Anyway
Thomas Huth found it is easier to use buildroot for pre-3.6 kernels.

Similarly, I am testing QEMU port from Stefan Weil, and he shared a
working binary supporting the MIPS AR7 target. To be able to use this
QEMU I use Debian Lenny and set
DEB_URL=https://snapshot.debian.org/archive/debian/20091004T111800Z.
Yes, this will instanciate a Debian from 10 years ago.

3/ Bisecting

Another of my docker uses is when bisecting before QEMU v3. I use image
using snapshot slighly older than the QEMU release, so my bisect script
can run without worrying about the library API incompatibilities or
newer GCC warnings.

So not all image have the same use. While they might not be useful to
build the latest QEMU, there are still useful for day-to-day development.

Anyhow I agree we should document that better. Maybe the wiki is a good
starting point.

Regards,

Phil.

Re: docker: how to use it when developing on QEMU?

[John Snow]

On 9/24/19 11:25 AM, Philippe Mathieu-Daudé wrote:
> Recently more developers are enthusiast to use Docker/Podman,
> and have been confused by the different configurations currently in the
> QEMU repository.
> 

I think it's good evidence we need to improve the abstractions for this
testing module. It's not easy to know what to do without getting direct
feedback from the maintainers directly, which won't scale.

> There are at least 3 kind of categories I use:
> 
> 1/ Image used to build QEMU
> 
> These images should be restricted/updated to our "supported targets".
> They are useful to (cross)build QEMU on variety of
> host/target/distributions/distrib_versions.
> 
> Example:
> 
> - cross build Cris binary using the Fedora 30 toolchain on a Ubuntu
> 18.04 x86_64.
>   host:Ubuntu18.04/x86_64 docker_image:fedora-cris-cross
> 

Is this supposed to be a command invocation? Can we see full command
lines instead?

> - cross build MinGW64 binary using Debian 9 MXE toolchain on a Ubuntu
> aarch64:
>   host:Ubuntu18.04/x86_64 docker_image:debian-win64-cross
> 
> An image can not be meant to use on a daily basis, but to avoid
> regression previous to release (I'd run them only on release candidate).
> 

Do you mean to say "These images are not meant to be used on a daily
basis, but instead on occasion to prevent regressions during release
candidate testing." ?

They can build QEMU, but I assume they can't run any tests. Is there a
special "build-only" target that you can invoke from the Makefile to get
these to run?

> Example: building QEMU for the Gentoo PlayStation2 port:
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg574468.html
> 
> 2/ Image used to build test program used by QEMU
> 

I think this is the category most people are wanting to get their hands
on, usually.

> These images provide enough to build binaries you can then use to test
> QEMU. If you want to build more of these binaries, there is probably a
> better way. Here we are only interested in testing.
> 
> Example:
> 
> - Test PowerPC Linux-user binaries with qemu-powerpc-linux-user
>   docker_image:debian-powerpc-user-cross
> 
> - Build EDK2 payload for Virt/AArch64
>   It currently doesn't build with Fedora 30 and I'v to use a Fedora 29
> image.
> 
> Another case I had is when I tried to build a kernel for the Mipssim
> machine (supported by QEMU). The Linux kernel code has been removed, so
> I had to checkout an old kernel which is not buildable with my recent
> host GCC. Using a docker based on a very old distribution worked. Anyway
> Thomas Huth found it is easier to use buildroot for pre-3.6 kernels.
> 
> Similarly, I am testing QEMU port from Stefan Weil, and he shared a
> working binary supporting the MIPS AR7 target. To be able to use this
> QEMU I use Debian Lenny and set
> DEB_URL=https://snapshot.debian.org/archive/debian/20091004T111800Z.
> Yes, this will instanciate a Debian from 10 years ago.
> 
> 3/ Bisecting
> 
> Another of my docker uses is when bisecting before QEMU v3. I use image
> using snapshot slighly older than the QEMU release, so my bisect script
> can run without worrying about the library API incompatibilities or
> newer GCC warnings.
> 

Which images, tests, or invocations are useful for this?

> So not all image have the same use. While they might not be useful to
> build the latest QEMU, there are still useful for day-to-day development.
> 

I would say that this has grown beyond the abstractions presented by the
Docker makefile, which makes it hard to figure out how to use any of the
tools in this directory.

> Anyhow I agree we should document that better. Maybe the wiki is a good
> starting point.
> 

That might be a good place to organize your thoughts, but the source of
truth must be the Makefile help invocation.

> Regards,
> 
> Phil.
> 


We should split the images into a few categories in the makefile
directly, in a programmatic way;

1. Partial Images

These images are not meant for building or testing anything directly.
They are dependencies; they should be better distinguished in the
Makefile help.

2. Unrunnable Images

These images would otherwise not be considered partial, but are
conditionally unrunnable depending on the architecture of the host system.

3a. Images that can build QEMU

Runnable images; but only useful for building QEMU and/or its tools. A
lot of the cross-compiler images might fit into this category.

3b. Images that can test QEMU

Runnable images; but can run iotests, make check, and so on. We expect
all of the tests defined, in general, to work for this image.


Does that sound about right?

--js

Re: docker: how to use it when developing on QEMU?

[Alex Bennée]

John Snow <jsnow@redhat.com> writes:

> On 9/24/19 11:25 AM, Philippe Mathieu-Daudé wrote:
>> Recently more developers are enthusiast to use Docker/Podman,
>> and have been confused by the different configurations currently in the
>> QEMU repository.
>>
>
> I think it's good evidence we need to improve the abstractions for this
> testing module. It's not easy to know what to do without getting direct
> feedback from the maintainers directly, which won't scale.

It has certainly grown more warty as we've gone on. For one thing we
don't cleanly handle different host architectures.

>
>> There are at least 3 kind of categories I use:
>>
>> 1/ Image used to build QEMU
>>
>> These images should be restricted/updated to our "supported targets".
>> They are useful to (cross)build QEMU on variety of
>> host/target/distributions/distrib_versions.
>>
>> Example:
>>
>> - cross build Cris binary using the Fedora 30 toolchain on a Ubuntu
>> 18.04 x86_64.
>>   host:Ubuntu18.04/x86_64 docker_image:fedora-cris-cross
>>
>
> Is this supposed to be a command invocation? Can we see full command
> lines instead?

 make run-tcg-tests-cris-softmmu V=1

and you'll see the docker invocation underneath.

>
>> - cross build MinGW64 binary using Debian 9 MXE toolchain on a Ubuntu
>> aarch64:
>>   host:Ubuntu18.04/x86_64 docker_image:debian-win64-cross
>>
>> An image can not be meant to use on a daily basis, but to avoid
>> regression previous to release (I'd run them only on release candidate).
>>
>
> Do you mean to say "These images are not meant to be used on a daily
> basis, but instead on occasion to prevent regressions during release
> candidate testing." ?
>
> They can build QEMU, but I assume they can't run any tests. Is there a
> special "build-only" target that you can invoke from the Makefile to get
> these to run?

 make docker-test-build

Should run all QEMU builds on all images that support it. Works in testing/next.

>
>> Example: building QEMU for the Gentoo PlayStation2 port:
>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg574468.html
>>
>> 2/ Image used to build test program used by QEMU
>>
>
> I think this is the category most people are wanting to get their hands
> on, usually.
>
>> These images provide enough to build binaries you can then use to test
>> QEMU. If you want to build more of these binaries, there is probably a
>> better way. Here we are only interested in testing.
>>
>> Example:
>>
>> - Test PowerPC Linux-user binaries with qemu-powerpc-linux-user
>>   docker_image:debian-powerpc-user-cross
>>
>> - Build EDK2 payload for Virt/AArch64
>>   It currently doesn't build with Fedora 30 and I'v to use a Fedora 29
>> image.
>>
>> Another case I had is when I tried to build a kernel for the Mipssim
>> machine (supported by QEMU). The Linux kernel code has been removed, so
>> I had to checkout an old kernel which is not buildable with my recent
>> host GCC. Using a docker based on a very old distribution worked. Anyway
>> Thomas Huth found it is easier to use buildroot for pre-3.6 kernels.
>>
>> Similarly, I am testing QEMU port from Stefan Weil, and he shared a
>> working binary supporting the MIPS AR7 target. To be able to use this
>> QEMU I use Debian Lenny and set
>> DEB_URL=https://snapshot.debian.org/archive/debian/20091004T111800Z.
>> Yes, this will instanciate a Debian from 10 years ago.

So generally once built you can re-use an image with cross compilers to
build any random source tree. Usually you invoke docker directly with
something like:

  docker run --rm -it -u $(id -u) -v $(pwd):$(pwd) -w $(pwd) qemu:debian-ubuntu-bionic-arm64 /bin/bash

Which basically drops you into a shell with your build directory mapped
into the docker image. However this usage is out of scope for the QEMU
build machinery although a useful way to use the images.


>>
>> 3/ Bisecting
>>
>> Another of my docker uses is when bisecting before QEMU v3. I use image
>> using snapshot slighly older than the QEMU release, so my bisect script
>> can run without worrying about the library API incompatibilities or
>> newer GCC warnings.
>>
>
> Which images, tests, or invocations are useful for this?
>
>> So not all image have the same use. While they might not be useful to
>> build the latest QEMU, there are still useful for day-to-day development.
>>
>
> I would say that this has grown beyond the abstractions presented by the
> Docker makefile, which makes it hard to figure out how to use any of the
> tools in this directory.
>
>> Anyhow I agree we should document that better. Maybe the wiki is a good
>> starting point.
>>
>
> That might be a good place to organize your thoughts, but the source of
> truth must be the Makefile help invocation.

Agreed.

>
>> Regards,
>>
>> Phil.
>>
>
>
> We should split the images into a few categories in the makefile
> directly, in a programmatic way;
>
> 1. Partial Images
>
> These images are not meant for building or testing anything directly.
> They are dependencies; they should be better distinguished in the
> Makefile help.
>
> 2. Unrunnable Images
>
> These images would otherwise not be considered partial, but are
> conditionally unrunnable depending on the architecture of the host system.
>
> 3a. Images that can build QEMU
>
> Runnable images; but only useful for building QEMU and/or its tools. A
> lot of the cross-compiler images might fit into this category.
>
> 3b. Images that can test QEMU
>
> Runnable images; but can run iotests, make check, and so on. We expect
> all of the tests defined, in general, to work for this image.

3c. Images that can build tests that run with QEMU

These are minimal cross compilers with just the foreign glibc and can
build our minimal tcg test cases.

>
>
> Does that sound about right?
>
> --js


--
Alex Bennée

Re: docker: how to use it when developing on QEMU?

[Markus Armbruster]

Alex, thank you for these hints.  Let me add one more:
docs/devel/testing.rst section "Docker based tests".

Does it contain your hints already?  If not, it may need an update.