[Qemu-devel] [PATCH 0/4] A number of Travis CI tweaks

[Qemu-devel] [PATCH 0/4] A number of Travis CI tweaks

[Alex Bennée]

Hi,

While I was in-between kernel builds last week I attempted to improve
the Travis build a little. Alexander Graf pointed out we were missing
a number of the linux-user targets. To avoid exploding the matrix too
much I've grouped builds together where they hopefully benefit from
sharing some objtect files. The biggest win however was using a
tarball to pre-seed the sub-module checkouts. I'm not sure if this is
because hammering our git server slows down or just because it's a lot
of data but it was adding up to around half the execution time of the
build.

Finally I removed "make check" from every build. It still gets run
once in the matrix but this means the current instability will bite
less often.

I failed in the task of adding mingw builds because it's hard on the
Travis Ubuntu hosts to install the headers/libs for a cross compile.
If someone can come up with a nice solid script that hand pulls in
these dependancies then we can add a call to that for the mingw
builds. Any volenteers?

If there are no objections/negative reviews I'll push these at the
Trivial tree later this week. Or is it time we create a specific
testing tree for these particular type of patches?

Alex Bennée (4):
  .travis.yml: add more linux-user to the build matrix
  .travis.yml: make the make slightly more parallel
  .travis.yml: pre-seed sub-modules for speed
  .travis.yml: remove "make check" from main matrix

 .travis.yml | 56 ++++++++++++++++++++++++++++++++++++++------------------
 1 file changed, 38 insertions(+), 18 deletions(-)

-- 
1.9.1

[Qemu-devel] [PATCH 1/4] .travis.yml: add more linux-user to the build matrix

[Alex Bennée]

At the same time I've grouped the $ARCH-linux-user and $ARCH-softmmu
builds together (hoping FS cache helps) and grouped all $ARCH-softmmu
only builds into one target. This reduces the build matrix slightly
which will hopefully help with build times.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/.travis.yml b/.travis.yml
index 89c30ae..72cfc9f 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -20,23 +20,23 @@ env:
     - GUI_PKGS="libgtk-3-dev libvte-2.90-dev libsdl1.2-dev libpng12-dev libpixman-1-dev"
     - EXTRA_PKGS=""
   matrix:
+    # Group major targets together with their linux-user counterparts
     - TARGETS=alpha-softmmu,alpha-linux-user
-    - TARGETS=arm-softmmu,arm-linux-user
-    - TARGETS=aarch64-softmmu,aarch64-linux-user
-    - TARGETS=cris-softmmu
-    - TARGETS=i386-softmmu,x86_64-softmmu
-    - TARGETS=lm32-softmmu
-    - TARGETS=m68k-softmmu
-    - TARGETS=microblaze-softmmu,microblazeel-softmmu
+    - TARGETS=arm-softmmu,arm-linux-user,armeb-linux-user,aarch64-softmmu,aarch64-linux-user
+    - TARGETS=cris-softmmu,cris-linux-user
+    - TARGETS=i386-softmmu,i386-linux-user,x86_64-softmmu,x86_64-linux-user
+    - TARGETS=m68k-softmmu,m68k-linux-user
+    - TARGETS=microblaze-softmmu,microblazeel-softmmu,microblaze-linux-user,microblazeel-linux-user
     - TARGETS=mips-softmmu,mips64-softmmu,mips64el-softmmu,mipsel-softmmu
-    - TARGETS=moxie-softmmu
-    - TARGETS=or32-softmmu,
-    - TARGETS=ppc-softmmu,ppc64-softmmu,ppcemb-softmmu
-    - TARGETS=s390x-softmmu
-    - TARGETS=sh4-softmmu,sh4eb-softmmu
-    - TARGETS=sparc-softmmu,sparc64-softmmu
-    - TARGETS=unicore32-softmmu
-    - TARGETS=xtensa-softmmu,xtensaeb-softmmu
+    - TARGETS=mips-linux-user,mips64-linux-user,mips64el-linux-user,mipsel-linux-user,mipsn32-linux-user,mipsn32el-linux-user
+    - TARGETS=or32-softmmu,or32-linux-user
+    - TARGETS=ppc-softmmu,ppc64-softmmu,ppcemb-softmmu,ppc-linux-user,ppc64-linux-user,ppc64abi32-linux-user,ppc64le-linux-user
+    - TARGETS=s390x-softmmu,s390x-linux-user
+    - TARGETS=sh4-softmmu,sh4eb-softmmu,sh4-linux-user sh4eb-linux-user
+    - TARGETS=sparc-softmmu,sparc64-softmmu,sparc-linux-user,sparc32plus-linux-user,sparc64-linux-user
+    - TARGETS=unicore32-softmmu,unicore32-linux-user
+    # Group remaining softmmu only targets into one build
+    - TARGETS=lm32-softmmu,moxie-softmmu,tricore-softmmu,xtensa-softmmu,xtensaeb-softmmu
 before_install:
   - git submodule update --init --recursive
   - sudo apt-get update -qq
-- 
1.9.1

[Qemu-devel] [PATCH 2/4] .travis.yml: make the make slightly more parallel

[Alex Bennée]

The Travis VMs have 1.5 cores so we might as well make some use of the
paralellism.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/.travis.yml b/.travis.yml
index 72cfc9f..f113339 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -41,7 +41,10 @@ before_install:
   - git submodule update --init --recursive
   - sudo apt-get update -qq
   - sudo apt-get install -qq ${CORE_PKGS} ${NET_PKGS} ${GUI_PKGS} ${EXTRA_PKGS}
-script: "./configure --target-list=${TARGETS} ${EXTRA_CONFIG} && make && ${TEST_CMD}"
+before_script:
+  - ./configure --target-list=${TARGETS} --enable-debug-tcg ${EXTRA_CONFIG}
+script:
+  - make -j2 && {TEST_CMD}
 matrix:
   # We manually include a number of additional build for non-standard bits
   include:
-- 
1.9.1

[Qemu-devel] [PATCH 3/4] .travis.yml: pre-seed sub-modules for speed

[Alex Bennée]

A significant portion of the build time is spent initialising all the
sub-modules we use in the source tree. Often this is almost as long as
the build itself. By pre-seeding the .git/modules tree this will
hopefully improve things.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/.travis.yml b/.travis.yml
index f113339..8df02a4 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -37,7 +37,12 @@ env:
     - TARGETS=unicore32-softmmu,unicore32-linux-user
     # Group remaining softmmu only targets into one build
     - TARGETS=lm32-softmmu,moxie-softmmu,tricore-softmmu,xtensa-softmmu,xtensaeb-softmmu
+git:
+  # we want to do this ourselves
+  submodules: false
 before_install:
+  - wget http://people.linaro.org/~alex.bennee/qemu-submodule-git-seed.tar.xz
+  - tar -xvf qemu-submodule-git-seed.tar.xz
   - git submodule update --init --recursive
   - sudo apt-get update -qq
   - sudo apt-get install -qq ${CORE_PKGS} ${NET_PKGS} ${GUI_PKGS} ${EXTRA_PKGS}
-- 
1.9.1

[Qemu-devel] [PATCH 4/4] .travis.yml: remove "make check" from main matrix

[Alex Bennée]

There are problems with unreliability in "make check" which still need
to be tracked down. As the tests are broadly the same for all targets if
added one explicit target to the matrix to run it. However this does
build all softmmu targets to ensure they at least "run"

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/.travis.yml b/.travis.yml
index 8df02a4..337360c 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -12,7 +12,7 @@ notifications:
     on_failure: always
 env:
   global:
-    - TEST_CMD="make check"
+    - TEST_CMD=""
     - EXTRA_CONFIG=""
     # Development packages, EXTRA_PKGS saved for additional builds
     - CORE_PKGS="libusb-1.0-0-dev libiscsi-dev librados-dev libncurses5-dev"
@@ -49,10 +49,23 @@ before_install:
 before_script:
   - ./configure --target-list=${TARGETS} --enable-debug-tcg ${EXTRA_CONFIG}
 script:
-  - make -j2 && {TEST_CMD}
+  - make -j2 && ${TEST_CMD}
 matrix:
   # We manually include a number of additional build for non-standard bits
   include:
+    # Make check target (we only do this once)
+    - env:
+        - TARGETS=alpha-softmmu,arm-softmmu,aarch64-softmmu,cris-softmmu,
+                  i386-softmmu,x86_64-softmmu,m68k-softmmu,microblaze-softmmu,
+                  microblazeel-softmmu,mips-softmmu,mips64-softmmu,
+                  mips64el-softmmu,mipsel-softmmu,or32-softmmu,ppc-softmmu,
+                  ppc64-softmmu,ppcemb-softmmu,s390x-softmmu,sh4-softmmu,
+                  sh4eb-softmmu,sparc-softmmu,sparc64-softmmu,
+                  unicore32-softmmu,unicore32-linux-user,
+                  lm32-softmmu,moxie-softmmu,tricore-softmmu,xtensa-softmmu,
+                  xtensaeb-softmmu
+          TEST_CMD="make check"
+      compiler: gcc
     # Debug related options
     - env: TARGETS=i386-softmmu,x86_64-softmmu
            EXTRA_CONFIG="--enable-debug"
@@ -81,7 +94,6 @@ matrix:
       compiler: gcc
     - env: TARGETS=i386-softmmu,x86_64-softmmu
            EXTRA_CONFIG="--enable-trace-backends=ftrace"
-           TEST_CMD=""
       compiler: gcc
     - env: TARGETS=i386-softmmu,x86_64-softmmu
           EXTRA_PKGS="liblttng-ust-dev liburcu-dev"
-- 
1.9.1

Re: [Qemu-devel] [PATCH 1/4] .travis.yml: add more linux-user to the build matrix

[Alex Bennée]

Alex Bennée writes:

> At the same time I've grouped the $ARCH-linux-user and $ARCH-softmmu
> builds together (hoping FS cache helps) and grouped all $ARCH-softmmu
> only builds into one target. This reduces the build matrix slightly
> which will hopefully help with build times.

Pinging Alex who I forgot to CC when I sent this out.

>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>
> diff --git a/.travis.yml b/.travis.yml
> index 89c30ae..72cfc9f 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -20,23 +20,23 @@ env:
>      - GUI_PKGS="libgtk-3-dev libvte-2.90-dev libsdl1.2-dev libpng12-dev libpixman-1-dev"
>      - EXTRA_PKGS=""
>    matrix:
> +    # Group major targets together with their linux-user counterparts
>      - TARGETS=alpha-softmmu,alpha-linux-user
> -    - TARGETS=arm-softmmu,arm-linux-user
> -    - TARGETS=aarch64-softmmu,aarch64-linux-user
> -    - TARGETS=cris-softmmu
> -    - TARGETS=i386-softmmu,x86_64-softmmu
> -    - TARGETS=lm32-softmmu
> -    - TARGETS=m68k-softmmu
> -    - TARGETS=microblaze-softmmu,microblazeel-softmmu
> +    - TARGETS=arm-softmmu,arm-linux-user,armeb-linux-user,aarch64-softmmu,aarch64-linux-user
> +    - TARGETS=cris-softmmu,cris-linux-user
> +    - TARGETS=i386-softmmu,i386-linux-user,x86_64-softmmu,x86_64-linux-user
> +    - TARGETS=m68k-softmmu,m68k-linux-user
> +    - TARGETS=microblaze-softmmu,microblazeel-softmmu,microblaze-linux-user,microblazeel-linux-user
>      - TARGETS=mips-softmmu,mips64-softmmu,mips64el-softmmu,mipsel-softmmu
> -    - TARGETS=moxie-softmmu
> -    - TARGETS=or32-softmmu,
> -    - TARGETS=ppc-softmmu,ppc64-softmmu,ppcemb-softmmu
> -    - TARGETS=s390x-softmmu
> -    - TARGETS=sh4-softmmu,sh4eb-softmmu
> -    - TARGETS=sparc-softmmu,sparc64-softmmu
> -    - TARGETS=unicore32-softmmu
> -    - TARGETS=xtensa-softmmu,xtensaeb-softmmu
> +    - TARGETS=mips-linux-user,mips64-linux-user,mips64el-linux-user,mipsel-linux-user,mipsn32-linux-user,mipsn32el-linux-user
> +    - TARGETS=or32-softmmu,or32-linux-user
> +    - TARGETS=ppc-softmmu,ppc64-softmmu,ppcemb-softmmu,ppc-linux-user,ppc64-linux-user,ppc64abi32-linux-user,ppc64le-linux-user
> +    - TARGETS=s390x-softmmu,s390x-linux-user
> +    - TARGETS=sh4-softmmu,sh4eb-softmmu,sh4-linux-user sh4eb-linux-user
> +    - TARGETS=sparc-softmmu,sparc64-softmmu,sparc-linux-user,sparc32plus-linux-user,sparc64-linux-user
> +    - TARGETS=unicore32-softmmu,unicore32-linux-user
> +    # Group remaining softmmu only targets into one build
> +    - TARGETS=lm32-softmmu,moxie-softmmu,tricore-softmmu,xtensa-softmmu,xtensaeb-softmmu
>  before_install:
>    - git submodule update --init --recursive
>    - sudo apt-get update -qq


-- 
Alex Bennée

Re: [Qemu-devel] [PATCH 3/4] .travis.yml: pre-seed sub-modules for speed

[Alexander Graf]

On 15.09.14 18:48, Alex Bennée wrote:
> A significant portion of the build time is spent initialising all the
> sub-modules we use in the source tree. Often this is almost as long as
> the build itself. By pre-seeding the .git/modules tree this will
> hopefully improve things.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> 
> diff --git a/.travis.yml b/.travis.yml
> index f113339..8df02a4 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -37,7 +37,12 @@ env:
>      - TARGETS=unicore32-softmmu,unicore32-linux-user
>      # Group remaining softmmu only targets into one build
>      - TARGETS=lm32-softmmu,moxie-softmmu,tricore-softmmu,xtensa-softmmu,xtensaeb-softmmu
> +git:
> +  # we want to do this ourselves
> +  submodules: false
>  before_install:
> +  - wget http://people.linaro.org/~alex.bennee/qemu-submodule-git-seed.tar.xz
> +  - tar -xvf qemu-submodule-git-seed.tar.xz

wget -O - | tar?

>    - git submodule update --init --recursive

Doesn't this overwrite the code you just downloaded?


Alex

>    - sudo apt-get update -qq
>    - sudo apt-get install -qq ${CORE_PKGS} ${NET_PKGS} ${GUI_PKGS} ${EXTRA_PKGS}
> 

Re: [Qemu-devel] [PATCH 0/4] A number of Travis CI tweaks

[Alexander Graf]

On 15.09.14 18:48, Alex Bennée wrote:
> Hi,
> 
> While I was in-between kernel builds last week I attempted to improve
> the Travis build a little. Alexander Graf pointed out we were missing
> a number of the linux-user targets. To avoid exploding the matrix too
> much I've grouped builds together where they hopefully benefit from
> sharing some objtect files. The biggest win however was using a
> tarball to pre-seed the sub-module checkouts. I'm not sure if this is
> because hammering our git server slows down or just because it's a lot
> of data but it was adding up to around half the execution time of the
> build.
> 
> Finally I removed "make check" from every build. It still gets run
> once in the matrix but this means the current instability will bite
> less often.
> 
> I failed in the task of adding mingw builds because it's hard on the
> Travis Ubuntu hosts to install the headers/libs for a cross compile.
> If someone can come up with a nice solid script that hand pulls in
> these dependancies then we can add a call to that for the mingw
> builds. Any volenteers?
> 
> If there are no objections/negative reviews I'll push these at the
> Trivial tree later this week. Or is it time we create a specific
> testing tree for these particular type of patches?

I like it :). Except for the small nit on 3/4.

Reviewed-by: Alexander Graf <agraf@suse.de>


Alex

Re: [Qemu-devel] [PATCH 3/4] .travis.yml: pre-seed sub-modules for speed

[Alex Bennée]

Alexander Graf writes:

> On 15.09.14 18:48, Alex Bennée wrote:
>> A significant portion of the build time is spent initialising all the
>> sub-modules we use in the source tree. Often this is almost as long as
>> the build itself. By pre-seeding the .git/modules tree this will
>> hopefully improve things.
>> 
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> 
>> diff --git a/.travis.yml b/.travis.yml
>> index f113339..8df02a4 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -37,7 +37,12 @@ env:
>>      - TARGETS=unicore32-softmmu,unicore32-linux-user
>>      # Group remaining softmmu only targets into one build
>>      - TARGETS=lm32-softmmu,moxie-softmmu,tricore-softmmu,xtensa-softmmu,xtensaeb-softmmu
>> +git:
>> +  # we want to do this ourselves
>> +  submodules: false
>>  before_install:
>> +  - wget http://people.linaro.org/~alex.bennee/qemu-submodule-git-seed.tar.xz
>> +  - tar -xvf qemu-submodule-git-seed.tar.xz
>
> wget -O - | tar?

Sure I can do this.

>>    - git submodule update --init --recursive
>
> Doesn't this overwrite the code you just downloaded?

It didn't in my testing. However I need to do the "--init" to ensure all
the modules are correctly checked out. When manually deleting one of the
packs in my testing git did the right thing and fetched the missing pack
data. Once the submodules move forward I'll double check the incremental
update works as well.

>
>
> Alex
>
>>    - sudo apt-get update -qq
>>    - sudo apt-get install -qq ${CORE_PKGS} ${NET_PKGS} ${GUI_PKGS} ${EXTRA_PKGS}
>> 

-- 
Alex Bennée

Re: [Qemu-devel] [PATCH 3/4] .travis.yml: pre-seed sub-modules for speed

[Alexander Graf]

On 23.09.14 16:50, Alex Bennée wrote:
> 
> Alexander Graf writes:
> 
>> On 15.09.14 18:48, Alex Bennée wrote:
>>> A significant portion of the build time is spent initialising all the
>>> sub-modules we use in the source tree. Often this is almost as long as
>>> the build itself. By pre-seeding the .git/modules tree this will
>>> hopefully improve things.
>>>
>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>>
>>> diff --git a/.travis.yml b/.travis.yml
>>> index f113339..8df02a4 100644
>>> --- a/.travis.yml
>>> +++ b/.travis.yml
>>> @@ -37,7 +37,12 @@ env:
>>>      - TARGETS=unicore32-softmmu,unicore32-linux-user
>>>      # Group remaining softmmu only targets into one build
>>>      - TARGETS=lm32-softmmu,moxie-softmmu,tricore-softmmu,xtensa-softmmu,xtensaeb-softmmu
>>> +git:
>>> +  # we want to do this ourselves
>>> +  submodules: false
>>>  before_install:
>>> +  - wget http://people.linaro.org/~alex.bennee/qemu-submodule-git-seed.tar.xz
>>> +  - tar -xvf qemu-submodule-git-seed.tar.xz
>>
>> wget -O - | tar?
> 
> Sure I can do this.
> 
>>>    - git submodule update --init --recursive
>>
>> Doesn't this overwrite the code you just downloaded?
> 
> It didn't in my testing. However I need to do the "--init" to ensure all
> the modules are correctly checked out. When manually deleting one of the
> packs in my testing git did the right thing and fetched the missing pack
> data. Once the submodules move forward I'll double check the incremental
> update works as well.

Cool, git is pretty smart then :). Nice.


Alex