[Qemu-devel] Docker Tests

[Qemu-devel] Docker Tests

[John Snow]

Hi, I'm having a regular trainwreck here w/ the Docker test suite and I
have a few questions;

1. Which tests should I expect to work by default? for instance,
make-debug doesn't but I think that might be normal. Is that the only one?

2. Should all tests work for all targets?

3. Which images can I use to run tests? e.g. make docker-test-quick@XXXX
.. the help output shows me ALL images, including your partial ones. I
think we only want to see non-partial images for help output, or make
clear which ones are for tests and which ones are just images.

4. docker8 is listed as a partial image, but doesn't appear to be
consumed by anything. Can it be removed?

- Ditto for debian-sid.
- Ditto for debian-ports.

(should debian-sid and debian-ports actually be promoted to non-partial
images? or, how do I configure the other debian targets to use these as
a base instead?)

Re: [Qemu-devel] Docker Tests

[Alex Bennée]

John Snow <jsnow@redhat.com> writes:

> Hi, I'm having a regular trainwreck here w/ the Docker test suite and I
> have a few questions;
>
> 1. Which tests should I expect to work by default? for instance,
> make-debug doesn't but I think that might be normal. Is that the only
> one?

test-build is the only one that just builds. As a lot of the docker
images are cross compilers you generally can't run tests with them which
limits the use of tests that run make check.

> 2. Should all tests work for all targets?

No for reasons of cross compilers. However we should be at the point
that:

  make docker-test-build

should run and complete all the builds it can (with non-x86 systems
skipping a whole chunk). There is some re-factoring work to be done to
make the whole system a bit more friendly to multiple host architectures.

> 3. Which images can I use to run tests? e.g. make docker-test-quick@XXXX
> .. the help output shows me ALL images, including your partial ones.

It does? It filters out the partial ones for me. Broadly the mainline OS
ones are all capable of building and running tests
(centos/debian-amd64/fedora/ubuntu/travis).

> I
> think we only want to see non-partial images for help output, or make
> clear which ones are for tests and which ones are just images.
>
> 4. docker8 is listed as a partial image, but doesn't appear to be
> consumed by anything. Can it be removed?

Yes.

>
> - Ditto for debian-sid.

Probably - I almost removed it last time but Phillipe convinced me to
keep it in. I think now buster is out and has packaged gcc's for all the
old arches (alpha etc) we can probably get rid of it.

> - Ditto for debian-ports.

This was useful for compilers for old arches but I think it can be got
rid of now. It's basically sid with a different repo URL.

> (should debian-sid and debian-ports actually be promoted to non-partial
> images? or, how do I configure the other debian targets to use these as
> a base instead?)

For true any Debian you want you can use Debian bootstrap - although
it's main use case is for setting up binfmt_misc foreign arch images.

--
Alex Bennée

Re: [Qemu-devel] Docker Tests

[John Snow]

On 9/19/19 4:55 AM, Alex Bennée wrote:
> 
> John Snow <jsnow@redhat.com> writes:
> 
>> Hi, I'm having a regular trainwreck here w/ the Docker test suite and I
>> have a few questions;
>>
>> 1. Which tests should I expect to work by default? for instance,
>> make-debug doesn't but I think that might be normal. Is that the only
>> one?
> 
> test-build is the only one that just builds. As a lot of the docker
> images are cross compilers you generally can't run tests with them which
> limits the use of tests that run make check.
> 
>> 2. Should all tests work for all targets?
> 
> No for reasons of cross compilers. However we should be at the point
> that:
> 
>   make docker-test-build
> 
> should run and complete all the builds it can (with non-x86 systems
> skipping a whole chunk). There is some re-factoring work to be done to
> make the whole system a bit more friendly to multiple host architectures.
> 
>> 3. Which images can I use to run tests? e.g. make docker-test-quick@XXXX
>> .. the help output shows me ALL images, including your partial ones.
> 
> It does? It filters out the partial ones for me. Broadly the mainline OS
> ones are all capable of building and running tests
> (centos/debian-amd64/fedora/ubuntu/travis).
> 

Available container images:
    centos7 debian-alpha-cross debian-amd64 debian-amd64-cross
debian-arm64-cross debian-armel-cross debian-armhf-cross
debian-bootstrap debian-hppa-cross debian-m68k-cross debian-mips-cross
debian-mips64-cross debian-mips64el-cross debian-mipsel-cross
debian-ports debian-powerpc-cross debian-powerpc-user-cross
debian-ppc64-cross debian-ppc64el-cross debian-riscv64-cross
debian-s390x-cross debian-sh4-cross debian-sid debian-sparc64-cross
debian-tricore-cross debian-win32-cross debian-win64-cross
debian-xtensa-cross debian10 debian8 debian9 debian9-mxe fedora
fedora-cris-cross fedora-i386-cross travis ubuntu ubuntu1804

At the least, debian9-mxe and debian8, debian9, and debian10 are not
images you can use for testing. They're partial images and you can't use
the Makefile to test against them.

The makefile rules seem to apply to:

$(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES)
$(DOCKER_DEPRECATED_IMAGES)), \

but the printed images in help are:

        @echo 'Available container images:'
        @echo '    $(DOCKER_IMAGES)'
ifneq ($(DOCKER_USER_IMAGES),)


I think you want to filter out the types of images you can't use for
tests here and print those; and then if it's important to document
PARTIAL_IMAGES too, list those in a separate segment, because they can
be used for invocations to build the partial images.

>> I
>> think we only want to see non-partial images for help output, or make
>> clear which ones are for tests and which ones are just images.
>>
>> 4. docker8 is listed as a partial image, but doesn't appear to be
>> consumed by anything. Can it be removed?
> 
> Yes.
> 
>>
>> - Ditto for debian-sid.
> 
> Probably - I almost removed it last time but Phillipe convinced me to
> keep it in. I think now buster is out and has packaged gcc's for all the
> old arches (alpha etc) we can probably get rid of it.
> 
>> - Ditto for debian-ports.
> 
> This was useful for compilers for old arches but I think it can be got
> rid of now. It's basically sid with a different repo URL.
> 

I don't mean to campaign directly for their removal -- It's just that I
don't see anything that consumes them and it seems they cannot be used
for any tests that I am aware of because of being classified as "partial".

Maybe it's an error that nothing uses them, instead, but it's confusing
as-is.

(What brought this on? I'm replacing python ==> python3 and wanted to
test that this worked, but I have no way to test that this works with
these images, as there's no canonical consumption for them to test.)

>> (should debian-sid and debian-ports actually be promoted to non-partial
>> images? or, how do I configure the other debian targets to use these as
>> a base instead?)
> 
> For true any Debian you want you can use Debian bootstrap - although
> it's main use case is for setting up binfmt_misc foreign arch images.
> 

Not sure I follow; because my only real goal is testing changes to the
dockerfiles and I'm not really actually interested in testing any
particular version or configuration of debian at all.

So what I'm asking is: if we have partial debian images that choose
certain configurations, are they statically configured by the top-level
targets, or is there a way to make on-the-fly configurations for
swapping out 8's base with 9, or 9 with 10, etc?

If not, I don't see how or where -ports or -sid are used presently which
makes them hard to test.

--js