patchew - gitlab-ci notifications during the Xen 4.16 cycle

patchew - gitlab-ci notifications during the Xen 4.16 cycle

[Stefano Stabellini]

Hi all,

During the last 6 months we have been working on improving the Xen
Project gitlab-ci and patchew infrastructure.

You can see the results from gitlab-ci tests on the staging branch here:

https://gitlab.com/xen-project/xen/-/pipelines
https://gitlab.com/xen-project/xen/-/pipelines/269678673

In addition, and more interestingly, now we have patchew integration.
Patchew picks up patch series sent to xen-devel by any contributor and
commits them to branches under this repository:

https://gitlab.com/xen-project/patchew/xen

Then, gitlab-ci tests start automatically. Patchew waits for the results
and send a notification email. You can see patchew pipelines here:

https://gitlab.com/xen-project/patchew/xen/-/pipelines

All this typically happens before patches are even reviewed. Today the
patchew notification emails are only sent to us in the Gitlab-CI working
group. But the good news is that we think the results got to the point
where they are reliable enough that it would be good to share them with
the community.

We plan to have patchew send email notifications to xen-devel starting
from the beginning of the 4.16 development cycle.

Stay tuned.

Cheers,

Stefano

Re: patchew - gitlab-ci notifications during the Xen 4.16 cycle

[Roger Pau Monné]

On Fri, Mar 12, 2021 at 12:55:38PM -0800, Stefano Stabellini wrote:
> Hi all,
> 
> During the last 6 months we have been working on improving the Xen
> Project gitlab-ci and patchew infrastructure.
> 
> You can see the results from gitlab-ci tests on the staging branch here:
> 
> https://gitlab.com/xen-project/xen/-/pipelines
> https://gitlab.com/xen-project/xen/-/pipelines/269678673
> 
> In addition, and more interestingly, now we have patchew integration.
> Patchew picks up patch series sent to xen-devel by any contributor and
> commits them to branches under this repository:
> 
> https://gitlab.com/xen-project/patchew/xen
> 
> Then, gitlab-ci tests start automatically. Patchew waits for the results
> and send a notification email. You can see patchew pipelines here:
> 
> https://gitlab.com/xen-project/patchew/xen/-/pipelines
> 
> All this typically happens before patches are even reviewed. Today the
> patchew notification emails are only sent to us in the Gitlab-CI working
> group. But the good news is that we think the results got to the point
> where they are reliable enough that it would be good to share them with
> the community.

I'm not opposed to that, but we seem to have some random git clone
failures during builds (at least on the x86 side), that are more
frequent that expected. At least recently I had trouble getting a full
gitlab CI pipeline run that didn't hit one of those.

I will try to figure out what's going on by adding some more debug to
git to see if I can get more verbose information from git on exactly
what's failing.

Thanks, Roger.

Re: patchew - gitlab-ci notifications during the Xen 4.16 cycle

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 1910 bytes --]

On Sat, 13 Mar 2021, Roger Pau Monné wrote:
> On Fri, Mar 12, 2021 at 12:55:38PM -0800, Stefano Stabellini wrote:
> > Hi all,
> > 
> > During the last 6 months we have been working on improving the Xen
> > Project gitlab-ci and patchew infrastructure.
> > 
> > You can see the results from gitlab-ci tests on the staging branch here:
> > 
> > https://gitlab.com/xen-project/xen/-/pipelines
> > https://gitlab.com/xen-project/xen/-/pipelines/269678673
> > 
> > In addition, and more interestingly, now we have patchew integration.
> > Patchew picks up patch series sent to xen-devel by any contributor and
> > commits them to branches under this repository:
> > 
> > https://gitlab.com/xen-project/patchew/xen
> > 
> > Then, gitlab-ci tests start automatically. Patchew waits for the results
> > and send a notification email. You can see patchew pipelines here:
> > 
> > https://gitlab.com/xen-project/patchew/xen/-/pipelines
> > 
> > All this typically happens before patches are even reviewed. Today the
> > patchew notification emails are only sent to us in the Gitlab-CI working
> > group. But the good news is that we think the results got to the point
> > where they are reliable enough that it would be good to share them with
> > the community.
> 
> I'm not opposed to that, but we seem to have some random git clone
> failures during builds (at least on the x86 side), that are more
> frequent that expected. At least recently I had trouble getting a full
> gitlab CI pipeline run that didn't hit one of those.
> 
> I will try to figure out what's going on by adding some more debug to
> git to see if I can get more verbose information from git on exactly
> what's failing.

Yeah we brifly talked about those during the last gitlab-ci meeting.
The current theory is that they might be an infrastructure issue and
we ask Doug to have a look too.

Your idea to add more debug info to git is excellent.

Re: patchew - gitlab-ci notifications during the Xen 4.16 cycle

[Roger Pau Monné]

On Mon, Mar 15, 2021 at 01:05:08PM -0700, Stefano Stabellini wrote:
> On Sat, 13 Mar 2021, Roger Pau Monné wrote:
> > On Fri, Mar 12, 2021 at 12:55:38PM -0800, Stefano Stabellini wrote:
> > > Hi all,
> > > 
> > > During the last 6 months we have been working on improving the Xen
> > > Project gitlab-ci and patchew infrastructure.
> > > 
> > > You can see the results from gitlab-ci tests on the staging branch here:
> > > 
> > > https://gitlab.com/xen-project/xen/-/pipelines
> > > https://gitlab.com/xen-project/xen/-/pipelines/269678673
> > > 
> > > In addition, and more interestingly, now we have patchew integration.
> > > Patchew picks up patch series sent to xen-devel by any contributor and
> > > commits them to branches under this repository:
> > > 
> > > https://gitlab.com/xen-project/patchew/xen
> > > 
> > > Then, gitlab-ci tests start automatically. Patchew waits for the results
> > > and send a notification email. You can see patchew pipelines here:
> > > 
> > > https://gitlab.com/xen-project/patchew/xen/-/pipelines
> > > 
> > > All this typically happens before patches are even reviewed. Today the
> > > patchew notification emails are only sent to us in the Gitlab-CI working
> > > group. But the good news is that we think the results got to the point
> > > where they are reliable enough that it would be good to share them with
> > > the community.
> > 
> > I'm not opposed to that, but we seem to have some random git clone
> > failures during builds (at least on the x86 side), that are more
> > frequent that expected. At least recently I had trouble getting a full
> > gitlab CI pipeline run that didn't hit one of those.
> > 
> > I will try to figure out what's going on by adding some more debug to
> > git to see if I can get more verbose information from git on exactly
> > what's failing.
> 
> Yeah we brifly talked about those during the last gitlab-ci meeting.
> The current theory is that they might be an infrastructure issue and
> we ask Doug to have a look too.
> 
> Your idea to add more debug info to git is excellent.

I've just realized that all the failures seem to happen on the -bobcat
runner, which is maintained by the Xen Project itself. The -dingo
runner seems to be fine. AFAIK they both run on Rackspace
infrastructure, so maybe there's some tweaking to be done?

Maybe the machines used by the XenProject runner have some kind of
network limitation?

Thanks, Roger.