retrying failed gitlab CI external jobs (travis)

retrying failed gitlab CI external jobs (travis)

[Peter Maydell]

Hi; we now have travis's CI hooked into gitlab, which is nice. However,
unlike the gitlab native CI jobs, there's no UI for saying "retry this"
when the "travis CI" part of the overall gitlab pipeline fails.
This is awkward because travis seems to be prone to intermittent failures.
Is there any way we can make the jobs retryable?

thanks
-- PMM

Re: retrying failed gitlab CI external jobs (travis)

[Peter Maydell]

On Sat, 10 Jul 2021 at 14:34, Peter Maydell <peter.maydell@linaro.org> wrote:
>
> Hi; we now have travis's CI hooked into gitlab, which is nice. However,
> unlike the gitlab native CI jobs, there's no UI for saying "retry this"
> when the "travis CI" part of the overall gitlab pipeline fails.
> This is awkward because travis seems to be prone to intermittent failures.
> Is there any way we can make the jobs retryable?

Also on the subject of the external travis job, what determines
when it runs? I would expect it to be run always, but if you look
at https://gitlab.com/qemu-project/qemu/-/pipelines
you can see that it didn't get run for the pipeline for
staging commit fc32b91a. It's not just "doesn't run for staging"
because it did run in the pipeline for staging ebd1f710.

-- PMM

Re: retrying failed gitlab CI external jobs (travis)

[Daniel P. Berrangé]

On Sat, Jul 10, 2021 at 05:29:24PM +0100, Peter Maydell wrote:
> On Sat, 10 Jul 2021 at 14:34, Peter Maydell <peter.maydell@linaro.org> wrote:
> >
> > Hi; we now have travis's CI hooked into gitlab, which is nice. However,
> > unlike the gitlab native CI jobs, there's no UI for saying "retry this"
> > when the "travis CI" part of the overall gitlab pipeline fails.
> > This is awkward because travis seems to be prone to intermittent failures.
> > Is there any way we can make the jobs retryable?
> 
> Also on the subject of the external travis job, what determines
> when it runs? I would expect it to be run always, but if you look
> at https://gitlab.com/qemu-project/qemu/-/pipelines
> you can see that it didn't get run for the pipeline for
> staging commit fc32b91a. It's not just "doesn't run for staging"
> because it did run in the pipeline for staging ebd1f710.

The way the Travis integration works is largely driven from Travis
itself.

So for retrying a failed pipeline, I think it is neccessary to hop
over to the travis-ci.com site.

This one had a failed Travis job:

  https://gitlab.com/qemu-project/qemu/-/pipelines/334623495

If you follow the link from the travis job there over to

  https://app.travis-ci.com/gitlab/qemu-project/qemu/builds/232314773

then I'd really hope they show a retry button.

Authentication is likely the key. Hopefully retry isn't tied to the
specific person who configured the Travis setup, and will instead
be shown to anyone who does SSO auth with GitLab and has rights
over to the GitLab project. I've no way to confirm this myself
though.

WRT missing job for commit fc32b91a, I see there is a Travis stage
reported here:

  https://gitlab.com/qemu-project/qemu/-/pipelines/334907106/builds

So I presume there was some delay in running the Travis jobs and
thus they only got reported after you sent the mail. 

The extra stage in the pipeline is not triggered/tracked by GitLab
itself. Rather it relies on Travis to see the changes, runs its job
and pushes information back to GitLab. This is completely asynchronous
to the rest of the normal GitLab pipeline, so unfortunately if Travis
hasn't even started the job yet, we see nothing :=(

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: retrying failed gitlab CI external jobs (travis)

[Peter Maydell]

On Mon, 12 Jul 2021 at 10:22, Daniel P. Berrangé <berrange@redhat.com> wrote:
> WRT missing job for commit fc32b91a, I see there is a Travis stage
> reported here:
>
>   https://gitlab.com/qemu-project/qemu/-/pipelines/334907106/builds
>
> So I presume there was some delay in running the Travis jobs and
> thus they only got reported after you sent the mail.
>
> The extra stage in the pipeline is not triggered/tracked by GitLab
> itself. Rather it relies on Travis to see the changes, runs its job
> and pushes information back to GitLab. This is completely asynchronous
> to the rest of the normal GitLab pipeline, so unfortunately if Travis
> hasn't even started the job yet, we see nothing :=(

Hmm. If nothing shows up in the gitlab pipeline, it doesn't really
work as a gating CI step, because I won't see if it fails...

thanks
-- PMM

Re: retrying failed gitlab CI external jobs (travis)

[Daniel P. Berrangé]

On Mon, Jul 12, 2021 at 12:03:02PM +0100, Peter Maydell wrote:
> On Mon, 12 Jul 2021 at 10:22, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > WRT missing job for commit fc32b91a, I see there is a Travis stage
> > reported here:
> >
> >   https://gitlab.com/qemu-project/qemu/-/pipelines/334907106/builds
> >
> > So I presume there was some delay in running the Travis jobs and
> > thus they only got reported after you sent the mail.
> >
> > The extra stage in the pipeline is not triggered/tracked by GitLab
> > itself. Rather it relies on Travis to see the changes, runs its job
> > and pushes information back to GitLab. This is completely asynchronous
> > to the rest of the normal GitLab pipeline, so unfortunately if Travis
> > hasn't even started the job yet, we see nothing :=(
> 
> Hmm. If nothing shows up in the gitlab pipeline, it doesn't really
> work as a gating CI step, because I won't see if it fails...

Agreed, the Travis integration isn't ideal in this respect. We probably
have to consider to be merely a nice sanity check, rather than fully
gating CI.

The way I've integrated Cirrus CI into GitLab is more useful, because
that uses the GitLab job to directly trigger the Cirrus job, and so
we don't suffer from the asynchronous feedback loop going missing
entirely as with Travis.

It would be nice to integrate Traviss in the same way as Cirrus, but
I'm still not sure it is possible with Travis' REST API.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|