* [Qemu-devel] [PATCH 1/4] .travis.yml: add more linux-user to the build matrix
2014-09-15 16:48 [Qemu-devel] [PATCH 0/4] A number of Travis CI tweaks Alex Bennée
@ 2014-09-15 16:48 ` Alex Bennée
2014-09-17 18:36 ` Alex Bennée
2014-09-15 16:48 ` [Qemu-devel] [PATCH 2/4] .travis.yml: make the make slightly more parallel Alex Bennée
` (3 subsequent siblings)
4 siblings, 1 reply; 10+ messages in thread
From: Alex Bennée @ 2014-09-15 16:48 UTC (permalink / raw)
To: qemu-devel; +Cc: peter.maydell, Alex Bennée
At the same time I've grouped the $ARCH-linux-user and $ARCH-softmmu
builds together (hoping FS cache helps) and grouped all $ARCH-softmmu
only builds into one target. This reduces the build matrix slightly
which will hopefully help with build times.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
diff --git a/.travis.yml b/.travis.yml
index 89c30ae..72cfc9f 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -20,23 +20,23 @@ env:
- GUI_PKGS="libgtk-3-dev libvte-2.90-dev libsdl1.2-dev libpng12-dev libpixman-1-dev"
- EXTRA_PKGS=""
matrix:
+ # Group major targets together with their linux-user counterparts
- TARGETS=alpha-softmmu,alpha-linux-user
- - TARGETS=arm-softmmu,arm-linux-user
- - TARGETS=aarch64-softmmu,aarch64-linux-user
- - TARGETS=cris-softmmu
- - TARGETS=i386-softmmu,x86_64-softmmu
- - TARGETS=lm32-softmmu
- - TARGETS=m68k-softmmu
- - TARGETS=microblaze-softmmu,microblazeel-softmmu
+ - TARGETS=arm-softmmu,arm-linux-user,armeb-linux-user,aarch64-softmmu,aarch64-linux-user
+ - TARGETS=cris-softmmu,cris-linux-user
+ - TARGETS=i386-softmmu,i386-linux-user,x86_64-softmmu,x86_64-linux-user
+ - TARGETS=m68k-softmmu,m68k-linux-user
+ - TARGETS=microblaze-softmmu,microblazeel-softmmu,microblaze-linux-user,microblazeel-linux-user
- TARGETS=mips-softmmu,mips64-softmmu,mips64el-softmmu,mipsel-softmmu
- - TARGETS=moxie-softmmu
- - TARGETS=or32-softmmu,
- - TARGETS=ppc-softmmu,ppc64-softmmu,ppcemb-softmmu
- - TARGETS=s390x-softmmu
- - TARGETS=sh4-softmmu,sh4eb-softmmu
- - TARGETS=sparc-softmmu,sparc64-softmmu
- - TARGETS=unicore32-softmmu
- - TARGETS=xtensa-softmmu,xtensaeb-softmmu
+ - TARGETS=mips-linux-user,mips64-linux-user,mips64el-linux-user,mipsel-linux-user,mipsn32-linux-user,mipsn32el-linux-user
+ - TARGETS=or32-softmmu,or32-linux-user
+ - TARGETS=ppc-softmmu,ppc64-softmmu,ppcemb-softmmu,ppc-linux-user,ppc64-linux-user,ppc64abi32-linux-user,ppc64le-linux-user
+ - TARGETS=s390x-softmmu,s390x-linux-user
+ - TARGETS=sh4-softmmu,sh4eb-softmmu,sh4-linux-user sh4eb-linux-user
+ - TARGETS=sparc-softmmu,sparc64-softmmu,sparc-linux-user,sparc32plus-linux-user,sparc64-linux-user
+ - TARGETS=unicore32-softmmu,unicore32-linux-user
+ # Group remaining softmmu only targets into one build
+ - TARGETS=lm32-softmmu,moxie-softmmu,tricore-softmmu,xtensa-softmmu,xtensaeb-softmmu
before_install:
- git submodule update --init --recursive
- sudo apt-get update -qq
--
1.9.1
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [Qemu-devel] [PATCH 1/4] .travis.yml: add more linux-user to the build matrix
2014-09-15 16:48 ` [Qemu-devel] [PATCH 1/4] .travis.yml: add more linux-user to the build matrix Alex Bennée
@ 2014-09-17 18:36 ` Alex Bennée
0 siblings, 0 replies; 10+ messages in thread
From: Alex Bennée @ 2014-09-17 18:36 UTC (permalink / raw)
To: Alex Bennée; +Cc: peter.maydell, qemu-devel, Alexander Graf
Alex Bennée writes:
> At the same time I've grouped the $ARCH-linux-user and $ARCH-softmmu
> builds together (hoping FS cache helps) and grouped all $ARCH-softmmu
> only builds into one target. This reduces the build matrix slightly
> which will hopefully help with build times.
Pinging Alex who I forgot to CC when I sent this out.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>
> diff --git a/.travis.yml b/.travis.yml
> index 89c30ae..72cfc9f 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -20,23 +20,23 @@ env:
> - GUI_PKGS="libgtk-3-dev libvte-2.90-dev libsdl1.2-dev libpng12-dev libpixman-1-dev"
> - EXTRA_PKGS=""
> matrix:
> + # Group major targets together with their linux-user counterparts
> - TARGETS=alpha-softmmu,alpha-linux-user
> - - TARGETS=arm-softmmu,arm-linux-user
> - - TARGETS=aarch64-softmmu,aarch64-linux-user
> - - TARGETS=cris-softmmu
> - - TARGETS=i386-softmmu,x86_64-softmmu
> - - TARGETS=lm32-softmmu
> - - TARGETS=m68k-softmmu
> - - TARGETS=microblaze-softmmu,microblazeel-softmmu
> + - TARGETS=arm-softmmu,arm-linux-user,armeb-linux-user,aarch64-softmmu,aarch64-linux-user
> + - TARGETS=cris-softmmu,cris-linux-user
> + - TARGETS=i386-softmmu,i386-linux-user,x86_64-softmmu,x86_64-linux-user
> + - TARGETS=m68k-softmmu,m68k-linux-user
> + - TARGETS=microblaze-softmmu,microblazeel-softmmu,microblaze-linux-user,microblazeel-linux-user
> - TARGETS=mips-softmmu,mips64-softmmu,mips64el-softmmu,mipsel-softmmu
> - - TARGETS=moxie-softmmu
> - - TARGETS=or32-softmmu,
> - - TARGETS=ppc-softmmu,ppc64-softmmu,ppcemb-softmmu
> - - TARGETS=s390x-softmmu
> - - TARGETS=sh4-softmmu,sh4eb-softmmu
> - - TARGETS=sparc-softmmu,sparc64-softmmu
> - - TARGETS=unicore32-softmmu
> - - TARGETS=xtensa-softmmu,xtensaeb-softmmu
> + - TARGETS=mips-linux-user,mips64-linux-user,mips64el-linux-user,mipsel-linux-user,mipsn32-linux-user,mipsn32el-linux-user
> + - TARGETS=or32-softmmu,or32-linux-user
> + - TARGETS=ppc-softmmu,ppc64-softmmu,ppcemb-softmmu,ppc-linux-user,ppc64-linux-user,ppc64abi32-linux-user,ppc64le-linux-user
> + - TARGETS=s390x-softmmu,s390x-linux-user
> + - TARGETS=sh4-softmmu,sh4eb-softmmu,sh4-linux-user sh4eb-linux-user
> + - TARGETS=sparc-softmmu,sparc64-softmmu,sparc-linux-user,sparc32plus-linux-user,sparc64-linux-user
> + - TARGETS=unicore32-softmmu,unicore32-linux-user
> + # Group remaining softmmu only targets into one build
> + - TARGETS=lm32-softmmu,moxie-softmmu,tricore-softmmu,xtensa-softmmu,xtensaeb-softmmu
> before_install:
> - git submodule update --init --recursive
> - sudo apt-get update -qq
--
Alex Bennée
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Qemu-devel] [PATCH 2/4] .travis.yml: make the make slightly more parallel
2014-09-15 16:48 [Qemu-devel] [PATCH 0/4] A number of Travis CI tweaks Alex Bennée
2014-09-15 16:48 ` [Qemu-devel] [PATCH 1/4] .travis.yml: add more linux-user to the build matrix Alex Bennée
@ 2014-09-15 16:48 ` Alex Bennée
2014-09-15 16:48 ` [Qemu-devel] [PATCH 3/4] .travis.yml: pre-seed sub-modules for speed Alex Bennée
` (2 subsequent siblings)
4 siblings, 0 replies; 10+ messages in thread
From: Alex Bennée @ 2014-09-15 16:48 UTC (permalink / raw)
To: qemu-devel; +Cc: peter.maydell, Alex Bennée
The Travis VMs have 1.5 cores so we might as well make some use of the
paralellism.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
diff --git a/.travis.yml b/.travis.yml
index 72cfc9f..f113339 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -41,7 +41,10 @@ before_install:
- git submodule update --init --recursive
- sudo apt-get update -qq
- sudo apt-get install -qq ${CORE_PKGS} ${NET_PKGS} ${GUI_PKGS} ${EXTRA_PKGS}
-script: "./configure --target-list=${TARGETS} ${EXTRA_CONFIG} && make && ${TEST_CMD}"
+before_script:
+ - ./configure --target-list=${TARGETS} --enable-debug-tcg ${EXTRA_CONFIG}
+script:
+ - make -j2 && {TEST_CMD}
matrix:
# We manually include a number of additional build for non-standard bits
include:
--
1.9.1
^ permalink raw reply related [flat|nested] 10+ messages in thread* [Qemu-devel] [PATCH 3/4] .travis.yml: pre-seed sub-modules for speed
2014-09-15 16:48 [Qemu-devel] [PATCH 0/4] A number of Travis CI tweaks Alex Bennée
2014-09-15 16:48 ` [Qemu-devel] [PATCH 1/4] .travis.yml: add more linux-user to the build matrix Alex Bennée
2014-09-15 16:48 ` [Qemu-devel] [PATCH 2/4] .travis.yml: make the make slightly more parallel Alex Bennée
@ 2014-09-15 16:48 ` Alex Bennée
2014-09-23 10:37 ` Alexander Graf
2014-09-15 16:48 ` [Qemu-devel] [PATCH 4/4] .travis.yml: remove "make check" from main matrix Alex Bennée
2014-09-23 10:38 ` [Qemu-devel] [PATCH 0/4] A number of Travis CI tweaks Alexander Graf
4 siblings, 1 reply; 10+ messages in thread
From: Alex Bennée @ 2014-09-15 16:48 UTC (permalink / raw)
To: qemu-devel; +Cc: peter.maydell, Alex Bennée
A significant portion of the build time is spent initialising all the
sub-modules we use in the source tree. Often this is almost as long as
the build itself. By pre-seeding the .git/modules tree this will
hopefully improve things.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
diff --git a/.travis.yml b/.travis.yml
index f113339..8df02a4 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -37,7 +37,12 @@ env:
- TARGETS=unicore32-softmmu,unicore32-linux-user
# Group remaining softmmu only targets into one build
- TARGETS=lm32-softmmu,moxie-softmmu,tricore-softmmu,xtensa-softmmu,xtensaeb-softmmu
+git:
+ # we want to do this ourselves
+ submodules: false
before_install:
+ - wget http://people.linaro.org/~alex.bennee/qemu-submodule-git-seed.tar.xz
+ - tar -xvf qemu-submodule-git-seed.tar.xz
- git submodule update --init --recursive
- sudo apt-get update -qq
- sudo apt-get install -qq ${CORE_PKGS} ${NET_PKGS} ${GUI_PKGS} ${EXTRA_PKGS}
--
1.9.1
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [Qemu-devel] [PATCH 3/4] .travis.yml: pre-seed sub-modules for speed
2014-09-15 16:48 ` [Qemu-devel] [PATCH 3/4] .travis.yml: pre-seed sub-modules for speed Alex Bennée
@ 2014-09-23 10:37 ` Alexander Graf
2014-09-23 14:50 ` Alex Bennée
0 siblings, 1 reply; 10+ messages in thread
From: Alexander Graf @ 2014-09-23 10:37 UTC (permalink / raw)
To: Alex Bennée, qemu-devel; +Cc: peter.maydell
On 15.09.14 18:48, Alex Bennée wrote:
> A significant portion of the build time is spent initialising all the
> sub-modules we use in the source tree. Often this is almost as long as
> the build itself. By pre-seeding the .git/modules tree this will
> hopefully improve things.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>
> diff --git a/.travis.yml b/.travis.yml
> index f113339..8df02a4 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -37,7 +37,12 @@ env:
> - TARGETS=unicore32-softmmu,unicore32-linux-user
> # Group remaining softmmu only targets into one build
> - TARGETS=lm32-softmmu,moxie-softmmu,tricore-softmmu,xtensa-softmmu,xtensaeb-softmmu
> +git:
> + # we want to do this ourselves
> + submodules: false
> before_install:
> + - wget http://people.linaro.org/~alex.bennee/qemu-submodule-git-seed.tar.xz
> + - tar -xvf qemu-submodule-git-seed.tar.xz
wget -O - | tar?
> - git submodule update --init --recursive
Doesn't this overwrite the code you just downloaded?
Alex
> - sudo apt-get update -qq
> - sudo apt-get install -qq ${CORE_PKGS} ${NET_PKGS} ${GUI_PKGS} ${EXTRA_PKGS}
>
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [Qemu-devel] [PATCH 3/4] .travis.yml: pre-seed sub-modules for speed
2014-09-23 10:37 ` Alexander Graf
@ 2014-09-23 14:50 ` Alex Bennée
2014-09-23 15:37 ` Alexander Graf
0 siblings, 1 reply; 10+ messages in thread
From: Alex Bennée @ 2014-09-23 14:50 UTC (permalink / raw)
To: Alexander Graf; +Cc: peter.maydell, qemu-devel
Alexander Graf writes:
> On 15.09.14 18:48, Alex Bennée wrote:
>> A significant portion of the build time is spent initialising all the
>> sub-modules we use in the source tree. Often this is almost as long as
>> the build itself. By pre-seeding the .git/modules tree this will
>> hopefully improve things.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>
>> diff --git a/.travis.yml b/.travis.yml
>> index f113339..8df02a4 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -37,7 +37,12 @@ env:
>> - TARGETS=unicore32-softmmu,unicore32-linux-user
>> # Group remaining softmmu only targets into one build
>> - TARGETS=lm32-softmmu,moxie-softmmu,tricore-softmmu,xtensa-softmmu,xtensaeb-softmmu
>> +git:
>> + # we want to do this ourselves
>> + submodules: false
>> before_install:
>> + - wget http://people.linaro.org/~alex.bennee/qemu-submodule-git-seed.tar.xz
>> + - tar -xvf qemu-submodule-git-seed.tar.xz
>
> wget -O - | tar?
Sure I can do this.
>> - git submodule update --init --recursive
>
> Doesn't this overwrite the code you just downloaded?
It didn't in my testing. However I need to do the "--init" to ensure all
the modules are correctly checked out. When manually deleting one of the
packs in my testing git did the right thing and fetched the missing pack
data. Once the submodules move forward I'll double check the incremental
update works as well.
>
>
> Alex
>
>> - sudo apt-get update -qq
>> - sudo apt-get install -qq ${CORE_PKGS} ${NET_PKGS} ${GUI_PKGS} ${EXTRA_PKGS}
>>
--
Alex Bennée
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [Qemu-devel] [PATCH 3/4] .travis.yml: pre-seed sub-modules for speed
2014-09-23 14:50 ` Alex Bennée
@ 2014-09-23 15:37 ` Alexander Graf
0 siblings, 0 replies; 10+ messages in thread
From: Alexander Graf @ 2014-09-23 15:37 UTC (permalink / raw)
To: Alex Bennée; +Cc: peter.maydell, qemu-devel
On 23.09.14 16:50, Alex Bennée wrote:
>
> Alexander Graf writes:
>
>> On 15.09.14 18:48, Alex Bennée wrote:
>>> A significant portion of the build time is spent initialising all the
>>> sub-modules we use in the source tree. Often this is almost as long as
>>> the build itself. By pre-seeding the .git/modules tree this will
>>> hopefully improve things.
>>>
>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>>
>>> diff --git a/.travis.yml b/.travis.yml
>>> index f113339..8df02a4 100644
>>> --- a/.travis.yml
>>> +++ b/.travis.yml
>>> @@ -37,7 +37,12 @@ env:
>>> - TARGETS=unicore32-softmmu,unicore32-linux-user
>>> # Group remaining softmmu only targets into one build
>>> - TARGETS=lm32-softmmu,moxie-softmmu,tricore-softmmu,xtensa-softmmu,xtensaeb-softmmu
>>> +git:
>>> + # we want to do this ourselves
>>> + submodules: false
>>> before_install:
>>> + - wget http://people.linaro.org/~alex.bennee/qemu-submodule-git-seed.tar.xz
>>> + - tar -xvf qemu-submodule-git-seed.tar.xz
>>
>> wget -O - | tar?
>
> Sure I can do this.
>
>>> - git submodule update --init --recursive
>>
>> Doesn't this overwrite the code you just downloaded?
>
> It didn't in my testing. However I need to do the "--init" to ensure all
> the modules are correctly checked out. When manually deleting one of the
> packs in my testing git did the right thing and fetched the missing pack
> data. Once the submodules move forward I'll double check the incremental
> update works as well.
Cool, git is pretty smart then :). Nice.
Alex
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Qemu-devel] [PATCH 4/4] .travis.yml: remove "make check" from main matrix
2014-09-15 16:48 [Qemu-devel] [PATCH 0/4] A number of Travis CI tweaks Alex Bennée
` (2 preceding siblings ...)
2014-09-15 16:48 ` [Qemu-devel] [PATCH 3/4] .travis.yml: pre-seed sub-modules for speed Alex Bennée
@ 2014-09-15 16:48 ` Alex Bennée
2014-09-23 10:38 ` [Qemu-devel] [PATCH 0/4] A number of Travis CI tweaks Alexander Graf
4 siblings, 0 replies; 10+ messages in thread
From: Alex Bennée @ 2014-09-15 16:48 UTC (permalink / raw)
To: qemu-devel; +Cc: peter.maydell, Alex Bennée
There are problems with unreliability in "make check" which still need
to be tracked down. As the tests are broadly the same for all targets if
added one explicit target to the matrix to run it. However this does
build all softmmu targets to ensure they at least "run"
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
diff --git a/.travis.yml b/.travis.yml
index 8df02a4..337360c 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -12,7 +12,7 @@ notifications:
on_failure: always
env:
global:
- - TEST_CMD="make check"
+ - TEST_CMD=""
- EXTRA_CONFIG=""
# Development packages, EXTRA_PKGS saved for additional builds
- CORE_PKGS="libusb-1.0-0-dev libiscsi-dev librados-dev libncurses5-dev"
@@ -49,10 +49,23 @@ before_install:
before_script:
- ./configure --target-list=${TARGETS} --enable-debug-tcg ${EXTRA_CONFIG}
script:
- - make -j2 && {TEST_CMD}
+ - make -j2 && ${TEST_CMD}
matrix:
# We manually include a number of additional build for non-standard bits
include:
+ # Make check target (we only do this once)
+ - env:
+ - TARGETS=alpha-softmmu,arm-softmmu,aarch64-softmmu,cris-softmmu,
+ i386-softmmu,x86_64-softmmu,m68k-softmmu,microblaze-softmmu,
+ microblazeel-softmmu,mips-softmmu,mips64-softmmu,
+ mips64el-softmmu,mipsel-softmmu,or32-softmmu,ppc-softmmu,
+ ppc64-softmmu,ppcemb-softmmu,s390x-softmmu,sh4-softmmu,
+ sh4eb-softmmu,sparc-softmmu,sparc64-softmmu,
+ unicore32-softmmu,unicore32-linux-user,
+ lm32-softmmu,moxie-softmmu,tricore-softmmu,xtensa-softmmu,
+ xtensaeb-softmmu
+ TEST_CMD="make check"
+ compiler: gcc
# Debug related options
- env: TARGETS=i386-softmmu,x86_64-softmmu
EXTRA_CONFIG="--enable-debug"
@@ -81,7 +94,6 @@ matrix:
compiler: gcc
- env: TARGETS=i386-softmmu,x86_64-softmmu
EXTRA_CONFIG="--enable-trace-backends=ftrace"
- TEST_CMD=""
compiler: gcc
- env: TARGETS=i386-softmmu,x86_64-softmmu
EXTRA_PKGS="liblttng-ust-dev liburcu-dev"
--
1.9.1
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [Qemu-devel] [PATCH 0/4] A number of Travis CI tweaks
2014-09-15 16:48 [Qemu-devel] [PATCH 0/4] A number of Travis CI tweaks Alex Bennée
` (3 preceding siblings ...)
2014-09-15 16:48 ` [Qemu-devel] [PATCH 4/4] .travis.yml: remove "make check" from main matrix Alex Bennée
@ 2014-09-23 10:38 ` Alexander Graf
4 siblings, 0 replies; 10+ messages in thread
From: Alexander Graf @ 2014-09-23 10:38 UTC (permalink / raw)
To: Alex Bennée, qemu-devel; +Cc: peter.maydell
On 15.09.14 18:48, Alex Bennée wrote:
> Hi,
>
> While I was in-between kernel builds last week I attempted to improve
> the Travis build a little. Alexander Graf pointed out we were missing
> a number of the linux-user targets. To avoid exploding the matrix too
> much I've grouped builds together where they hopefully benefit from
> sharing some objtect files. The biggest win however was using a
> tarball to pre-seed the sub-module checkouts. I'm not sure if this is
> because hammering our git server slows down or just because it's a lot
> of data but it was adding up to around half the execution time of the
> build.
>
> Finally I removed "make check" from every build. It still gets run
> once in the matrix but this means the current instability will bite
> less often.
>
> I failed in the task of adding mingw builds because it's hard on the
> Travis Ubuntu hosts to install the headers/libs for a cross compile.
> If someone can come up with a nice solid script that hand pulls in
> these dependancies then we can add a call to that for the mingw
> builds. Any volenteers?
>
> If there are no objections/negative reviews I'll push these at the
> Trivial tree later this week. Or is it time we create a specific
> testing tree for these particular type of patches?
I like it :). Except for the small nit on 3/4.
Reviewed-by: Alexander Graf <agraf@suse.de>
Alex
^ permalink raw reply [flat|nested] 10+ messages in thread