[Qemu-devel] QEMU travis builds: can we move the preseed tarball to qemu.org ?

[Qemu-devel] QEMU travis builds: can we move the preseed tarball to qemu.org ?

[Peter Maydell]

Hi; a recent travis build failure made me notice that our .travis.yml
config references a preseed tarball from here:
http://people.linaro.org/~alex.bennee/qemu-submodule-git-seed.tar.xz

I think this is a bit less than ideal -- we should really be hosting
this on qemu.org. Can we arrange to move it?

thanks
-- PMM

Re: [Qemu-devel] QEMU travis builds: can we move the preseed tarball to qemu.org ?

[Daniel P. Berrange]

On Tue, Apr 25, 2017 at 03:51:17PM +0100, Peter Maydell wrote:
> Hi; a recent travis build failure made me notice that our .travis.yml
> config references a preseed tarball from here:
> http://people.linaro.org/~alex.bennee/qemu-submodule-git-seed.tar.xz
> 
> I think this is a bit less than ideal -- we should really be hosting
> this on qemu.org. Can we arrange to move it?

I'm curious how much speed difference there is in seeding the git submodules
in this way vs letting git pull down from git.qemu.org directly ?


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] QEMU travis builds: can we move the preseed tarball to qemu.org ?

[Alex Bennée]

Peter Maydell <peter.maydell@linaro.org> writes:

> Hi; a recent travis build failure made me notice that our .travis.yml
> config references a preseed tarball from here:
> http://people.linaro.org/~alex.bennee/qemu-submodule-git-seed.tar.xz
>
> I think this is a bit less than ideal -- we should really be hosting
> this on qemu.org. Can we arrange to move it?

Sure - it was more out of convince than anything else. It is basically
the .git/modules/ subtree from a full checkout that has been tarred up.
Given how long ago it was generated it is probably worth updating the
seed anyway to speed up the submodule initialisation step.

Should we script it up on the qemu.org server or just upload a
hand-created seed?

--
Alex Bennée

Re: [Qemu-devel] QEMU travis builds: can we move the preseed tarball to qemu.org ?

[Alex Bennée]

Daniel P. Berrange <berrange@redhat.com> writes:

> On Tue, Apr 25, 2017 at 03:51:17PM +0100, Peter Maydell wrote:
>> Hi; a recent travis build failure made me notice that our .travis.yml
>> config references a preseed tarball from here:
>> http://people.linaro.org/~alex.bennee/qemu-submodule-git-seed.tar.xz
>>
>> I think this is a bit less than ideal -- we should really be hosting
>> this on qemu.org. Can we arrange to move it?
>
> I'm curious how much speed difference there is in seeding the git submodules
> in this way vs letting git pull down from git.qemu.org directly ?

It was quite high from what I recall, exacerbated by the fact we have
quite so many submodules. Unfortunately it doesn't seem that easy to go
back in the history of the tests to find out so I'll have to re-run the
test:

  https://travis-ci.org/stsquad/qemu/builds/225648653

So roughly 62s without a seed vs 15s with. I presume the seed is also
cached by Travis's web-caching.

--
Alex Bennée

Re: [Qemu-devel] QEMU travis builds: can we move the preseed tarball to qemu.org ?

[Daniel P. Berrange]

On Tue, Apr 25, 2017 at 04:24:18PM +0100, Alex Bennée wrote:
> 
> Daniel P. Berrange <berrange@redhat.com> writes:
> 
> > On Tue, Apr 25, 2017 at 03:51:17PM +0100, Peter Maydell wrote:
> >> Hi; a recent travis build failure made me notice that our .travis.yml
> >> config references a preseed tarball from here:
> >> http://people.linaro.org/~alex.bennee/qemu-submodule-git-seed.tar.xz
> >>
> >> I think this is a bit less than ideal -- we should really be hosting
> >> this on qemu.org. Can we arrange to move it?
> >
> > I'm curious how much speed difference there is in seeding the git submodules
> > in this way vs letting git pull down from git.qemu.org directly ?
> 
> It was quite high from what I recall, exacerbated by the fact we have
> quite so many submodules. Unfortunately it doesn't seem that easy to go
> back in the history of the tests to find out so I'll have to re-run the
> test:
> 
>   https://travis-ci.org/stsquad/qemu/builds/225648653
> 
> So roughly 62s without a seed vs 15s with. I presume the seed is also
> cached by Travis's web-caching.

Wow, yes, that is quite a difference !

I wonder if Travis' arbitrary sub-dir caching feature would help us to
the same extent, while avoiding need to manually maintain the preseed.

  https://docs.travis-ci.com/user/caching/

It just caches entire content of a given subdir between runs. First time
it would be slow, but presumably fast thereafter, and any time the submodule
gets new updates, those would get cached too

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] QEMU travis builds: can we move the preseed tarball to qemu.org ?

[Alex Bennée]

Daniel P. Berrange <berrange@redhat.com> writes:

> On Tue, Apr 25, 2017 at 04:24:18PM +0100, Alex Bennée wrote:
>>
>> Daniel P. Berrange <berrange@redhat.com> writes:
>>
>> > On Tue, Apr 25, 2017 at 03:51:17PM +0100, Peter Maydell wrote:
>> >> Hi; a recent travis build failure made me notice that our .travis.yml
>> >> config references a preseed tarball from here:
>> >> http://people.linaro.org/~alex.bennee/qemu-submodule-git-seed.tar.xz
>> >>
>> >> I think this is a bit less than ideal -- we should really be hosting
>> >> this on qemu.org. Can we arrange to move it?
>> >
>> > I'm curious how much speed difference there is in seeding the git submodules
>> > in this way vs letting git pull down from git.qemu.org directly ?
>>
>> It was quite high from what I recall, exacerbated by the fact we have
>> quite so many submodules. Unfortunately it doesn't seem that easy to go
>> back in the history of the tests to find out so I'll have to re-run the
>> test:
>>
>>   https://travis-ci.org/stsquad/qemu/builds/225648653
>>
>> So roughly 62s without a seed vs 15s with. I presume the seed is also
>> cached by Travis's web-caching.
>
> Wow, yes, that is quite a difference !
>
> I wonder if Travis' arbitrary sub-dir caching feature would help us to
> the same extent, while avoiding need to manually maintain the preseed.
>
>   https://docs.travis-ci.com/user/caching/
>
> It just caches entire content of a given subdir between runs. First time
> it would be slow, but presumably fast thereafter, and any time the submodule
> gets new updates, those would get cached too

Maybe but I think the cwd of the build (and therefor the git tree) is
user and project name dependent, e.g.:

  PWD=/home/travis/build/stsquad/testcases

But we could certainly try.

>
> Regards,
> Daniel


--
Alex Bennée

Re: [Qemu-devel] QEMU travis builds: can we move the preseed tarball to qemu.org ?

[Jeff Cody]

On Apr 25, 2017 11:55 AM, "Alex Bennée" <alex.bennee@linaro.org> wrote:


Daniel P. Berrange <berrange@redhat.com> writes:

> On Tue, Apr 25, 2017 at 04:24:18PM +0100, Alex Bennée wrote:
>>
>> Daniel P. Berrange <berrange@redhat.com> writes:
>>
>> > On Tue, Apr 25, 2017 at 03:51:17PM +0100, Peter Maydell wrote:
>> >> Hi; a recent travis build failure made me notice that our .travis.yml
>> >> config references a preseed tarball from here:
>> >> http://people.linaro.org/~alex.bennee/qemu-submodule-git-seed.tar.xz
>> >>
>> >> I think this is a bit less than ideal -- we should really be hosting
>> >> this on qemu.org. Can we arrange to move it?
>> >
>> > I'm curious how much speed difference there is in seeding the git
submodules
>> > in this way vs letting git pull down from git.qemu.org directly ?
>>
>> It was quite high from what I recall, exacerbated by the fact we have
>> quite so many submodules. Unfortunately it doesn't seem that easy to go
>> back in the history of the tests to find out so I'll have to re-run the
>> test:
>>
>>   https://travis-ci.org/stsquad/qemu/builds/225648653
>>
>> So roughly 62s without a seed vs 15s with. I presume the seed is also
>> cached by Travis's web-caching.
>
> Wow, yes, that is quite a difference !
>
> I wonder if Travis' arbitrary sub-dir caching feature would help us to
> the same extent, while avoiding need to manually maintain the preseed.
>
>   https://docs.travis-ci.com/user/caching/
>
> It just caches entire content of a given subdir between runs. First time
> it would be slow, but presumably fast thereafter, and any time the
submodule
> gets new updates, those would get cached too

Maybe but I think the cwd of the build (and therefor the git tree) is
user and project name dependent, e.g.:

  PWD=/home/travis/build/stsquad/testcases

But we could certainly try.

>
> Regards,
> Daniel


--
Alex Bennée


I do currently have an async script that runs for qemu.git, that is
triggered by a post-receive hook.   Currently, it just does a push to
GitHub, so that the mirror is updated in real-time.  It would be pretty
trivial to have it also compose a tarball as well.

Jeff