[Qemu-devel] [PATCH 0/2] docker: allow running "docker-travis" with arbitrary parallelism

[Qemu-devel] [PATCH 0/2] docker: allow running "docker-travis" with arbitrary parallelism

[Paolo Bonzini]

docker-travis is limited to the -j3 flag specified for running on the
SaaS Travis CI infrastructure.

When running the test matrix locally with docker, however, we can run
much more jobs at the same time on a suitably beefy server.  Let people
do that with the usual J=nn option already supported by the other
docker-based tests.

Paolo

Paolo Bonzini (2):
  docker: allow customizing Travis global_env variables
  travis: move make -j flag out of script

 .travis.yml            | 3 ++-
 tests/docker/travis.py | 6 +++---
 2 files changed, 5 insertions(+), 4 deletions(-)

-- 
2.13.0

[Qemu-devel] [PATCH 1/2] docker: allow customizing Travis global_env variables

[Paolo Bonzini]

This is useful so that we can do builds at higher than -j3 when running
travis.py locally.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 tests/docker/travis.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/docker/travis.py b/tests/docker/travis.py
index 8dcc964da4..703a7fde85 100755
--- a/tests/docker/travis.py
+++ b/tests/docker/travis.py
@@ -21,13 +21,12 @@ def load_yaml(fname):
 def conf_iter(conf):
     def env_to_list(env):
         return env if isinstance(env, list) else [env]
-    global_env = conf["env"]["global"]
     for entry in conf["matrix"]["include"]:
-        yield {"env": global_env + env_to_list(entry["env"]),
+        yield {"env": env_to_list(entry["env"]),
                "compiler": entry["compiler"]}
     for entry in itertools.product(conf["compiler"],
                                    conf["env"]["matrix"]):
-        yield {"env": global_env + env_to_list(entry[1]),
+        yield {"env": env_to_list(entry[1]),
                "compiler": entry[0]}
 
 def main():
@@ -35,6 +34,7 @@ def main():
         sys.stderr.write("Usage: %s <travis-file>\n" % sys.argv[0])
         return 1
     conf = load_yaml(sys.argv[1])
+    print "\n".join((": ${%s}" % var for var in conf["env"]["global"]))
     for config in conf_iter(conf):
         print "("
         print "\n".join(config["env"])
-- 
2.13.0

[Qemu-devel] [PATCH 2/2] travis: move make -j flag out of script

[Paolo Bonzini]

Because global environment variables can be overridden when .travis.yml
is processed by the docker-travis target, the effect of this patch is
that docker-travis now obeys the "J=n" option.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 .travis.yml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index 27a2d9cfb3..6809e25eca 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -46,6 +46,7 @@ notifications:
 env:
   global:
     - TEST_CMD="make check"
+    - MAKEFLAGS="-j3"
   matrix:
     - CONFIG=""
     - CONFIG="--enable-debug --enable-debug-tcg --enable-trace-backends=log"
@@ -64,7 +65,7 @@ before_install:
 before_script:
   - ./configure ${CONFIG}
 script:
-  - make -j3 && ${TEST_CMD}
+  - make ${MAKEFLAGS} && ${TEST_CMD}
 matrix:
   include:
     # Test with CLang for compile portability
-- 
2.13.0

Re: [Qemu-devel] [PATCH 0/2] docker: allow running "docker-travis" with arbitrary parallelism

[Alex Bennée]

Paolo Bonzini <pbonzini@redhat.com> writes:

> docker-travis is limited to the -j3 flag specified for running on the
> SaaS Travis CI infrastructure.
>
> When running the test matrix locally with docker, however, we can run
> much more jobs at the same time on a suitably beefy server.  Let people
> do that with the usual J=nn option already supported by the other
> docker-based tests.

Eh, I was debugging stuff on our beefy server with J=20 and then I
realised I was just running:

  make docker-test-quick@travis J=20

So I guess some people do use the pseudo Travis expansion then?

>
> Paolo
>
> Paolo Bonzini (2):
>   docker: allow customizing Travis global_env variables
>   travis: move make -j flag out of script
>
>  .travis.yml            | 3 ++-
>  tests/docker/travis.py | 6 +++---
>  2 files changed, 5 insertions(+), 4 deletions(-)


--
Alex Bennée

Re: [Qemu-devel] [PATCH 0/2] docker: allow running "docker-travis" with arbitrary parallelism

[Paolo Bonzini]

On 14/07/2017 12:05, Alex Bennée wrote:
> 
> Paolo Bonzini <pbonzini@redhat.com> writes:
> 
>> docker-travis is limited to the -j3 flag specified for running on the
>> SaaS Travis CI infrastructure.
>>
>> When running the test matrix locally with docker, however, we can run
>> much more jobs at the same time on a suitably beefy server.  Let people
>> do that with the usual J=nn option already supported by the other
>> docker-based tests.
> 
> Eh, I was debugging stuff on our beefy server with J=20 and then I
> realised I was just running:
> 
>   make docker-test-quick@travis J=20
> 
> So I guess some people do use the pseudo Travis expansion then?

How would that end up running tests/docker/travis?

Paolo

>>
>> Paolo
>>
>> Paolo Bonzini (2):
>>   docker: allow customizing Travis global_env variables
>>   travis: move make -j flag out of script
>>
>>  .travis.yml            | 3 ++-
>>  tests/docker/travis.py | 6 +++---
>>  2 files changed, 5 insertions(+), 4 deletions(-)
> 
> 
> --
> Alex Bennée
> 

Re: [Qemu-devel] [PATCH 0/2] docker: allow running "docker-travis" with arbitrary parallelism

[Alex Bennée]

Paolo Bonzini <pbonzini@redhat.com> writes:

> On 14/07/2017 12:05, Alex Bennée wrote:
>>
>> Paolo Bonzini <pbonzini@redhat.com> writes:
>>
>>> docker-travis is limited to the -j3 flag specified for running on the
>>> SaaS Travis CI infrastructure.
>>>
>>> When running the test matrix locally with docker, however, we can run
>>> much more jobs at the same time on a suitably beefy server.  Let people
>>> do that with the usual J=nn option already supported by the other
>>> docker-based tests.
>>
>> Eh, I was debugging stuff on our beefy server with J=20 and then I
>> realised I was just running:
>>
>>   make docker-test-quick@travis J=20
>>
>> So I guess some people do use the pseudo Travis expansion then?
>
> How would that end up running tests/docker/travis?

It runs the normal build and make check but in the Travis build
environment. Essentially the .travis.yml is just a bunch of alternative
configure invocations, there is nothing to stop you manually setting up
a build with TARGET_LIST and EXTRA_CONFIGURE_OPTS which is what I do if
there is a particular Travis test which is failing.

>
> Paolo
>
>>>
>>> Paolo
>>>
>>> Paolo Bonzini (2):
>>>   docker: allow customizing Travis global_env variables
>>>   travis: move make -j flag out of script
>>>
>>>  .travis.yml            | 3 ++-
>>>  tests/docker/travis.py | 6 +++---
>>>  2 files changed, 5 insertions(+), 4 deletions(-)
>>
>>
>> --
>> Alex Bennée
>>


--
Alex Bennée

Re: [Qemu-devel] [PATCH 0/2] docker: allow running "docker-travis" with arbitrary parallelism

[Paolo Bonzini]

On 14/07/2017 15:56, Alex Bennée wrote:
> 
> Paolo Bonzini <pbonzini@redhat.com> writes:
> 
>> On 14/07/2017 12:05, Alex Bennée wrote:
>>>
>>> Paolo Bonzini <pbonzini@redhat.com> writes:
>>>
>>>> docker-travis is limited to the -j3 flag specified for running on the
>>>> SaaS Travis CI infrastructure.
>>>>
>>>> When running the test matrix locally with docker, however, we can run
>>>> much more jobs at the same time on a suitably beefy server.  Let people
>>>> do that with the usual J=nn option already supported by the other
>>>> docker-based tests.
>>>
>>> Eh, I was debugging stuff on our beefy server with J=20 and then I
>>> realised I was just running:
>>>
>>>   make docker-test-quick@travis J=20
>>>
>>> So I guess some people do use the pseudo Travis expansion then?
>>
>> How would that end up running tests/docker/travis?
> 
> It runs the normal build and make check but in the Travis build
> environment. Essentially the .travis.yml is just a bunch of alternative
> configure invocations, there is nothing to stop you manually setting up
> a build with TARGET_LIST and EXTRA_CONFIGURE_OPTS which is what I do if
> there is a particular Travis test which is failing.

This is different, I want to run all Travis builds to ensure I don't
break one before going on vacation. :)

Paolo