Should we maybe move Cirrus-CI jobs away from Gitlab again?

Should we maybe move Cirrus-CI jobs away from Gitlab again?

[Thomas Huth]

  Hi all,

now that Gitlab is giving us pressure on the amount of free CI minutes, I 
wonder whether we should maybe move the Cirrus-CI jobs out of the gitlab-CI 
dashboard again? We could add the jobs to our .cirrus-ci.yml file instead, 
like we did it in former times...

Big advantage would be of course that the time for those jobs would not 
count in the Gitlab-CI minutes anymore. Disadvantage is of course that they 
do not show up in the gitlab-CI dashboard anymore, so there is no more 
e-mail notification about failed jobs, and you have to push to github, too, 
and finally check the results manually on cirrus-ci.com ...

Opinions?

  Thomas

Re: Should we maybe move Cirrus-CI jobs away from Gitlab again?

[Stefan Hajnoczi]

On Tue, 27 Sept 2022 at 05:02, Thomas Huth <thuth@redhat.com> wrote:
> now that Gitlab is giving us pressure on the amount of free CI minutes, I
> wonder whether we should maybe move the Cirrus-CI jobs out of the gitlab-CI
> dashboard again? We could add the jobs to our .cirrus-ci.yml file instead,
> like we did it in former times...
>
> Big advantage would be of course that the time for those jobs would not
> count in the Gitlab-CI minutes anymore. Disadvantage is of course that they
> do not show up in the gitlab-CI dashboard anymore, so there is no more
> e-mail notification about failed jobs, and you have to push to github, too,
> and finally check the results manually on cirrus-ci.com ...

My understanding is that .gitlab-ci.d/cirrus.yml uses a GitLab CI job
to run the cirrus-run container image that forwards jobs to Cirrus-CI.
So GitLab CI resources are consumed waiting for Cirrus-CI to finish.

This shouldn't affect gitlab.com/qemu-project where there are private
runners that do not consume GitLab CI minutes.

Individual developers are affected though because they most likely
rely on the GitLab shared runner minutes quota.

Does GitLab CI support some kind of async job so a container doesn't
have to monitor Cirrus-CI for the duration of the tests? I guess it
would require a job like the cirrus-run job, except a webhook signals
completion and therefore the GitLab CI container doesn't need to wait
around.

Stefan

Re: Should we maybe move Cirrus-CI jobs away from Gitlab again?

[Daniel P. Berrangé]

On Tue, Sep 27, 2022 at 11:44:45AM -0400, Stefan Hajnoczi wrote:
> On Tue, 27 Sept 2022 at 05:02, Thomas Huth <thuth@redhat.com> wrote:
> > now that Gitlab is giving us pressure on the amount of free CI minutes, I
> > wonder whether we should maybe move the Cirrus-CI jobs out of the gitlab-CI
> > dashboard again? We could add the jobs to our .cirrus-ci.yml file instead,
> > like we did it in former times...
> >
> > Big advantage would be of course that the time for those jobs would not
> > count in the Gitlab-CI minutes anymore. Disadvantage is of course that they
> > do not show up in the gitlab-CI dashboard anymore, so there is no more
> > e-mail notification about failed jobs, and you have to push to github, too,
> > and finally check the results manually on cirrus-ci.com ...
> 
> My understanding is that .gitlab-ci.d/cirrus.yml uses a GitLab CI job
> to run the cirrus-run container image that forwards jobs to Cirrus-CI.
> So GitLab CI resources are consumed waiting for Cirrus-CI to finish.
> 
> This shouldn't affect gitlab.com/qemu-project where there are private
> runners that do not consume GitLab CI minutes.
> 
> Individual developers are affected though because they most likely
> rely on the GitLab shared runner minutes quota.

NB, none of the jobs should ever be run automatically anymore in
QEMU CI pipelines. It always requires the maintainer to set the
env var when pushing to git, to explicitly create a pipeline.
You can then selectively start each individual job as desired.

While the Cirrus CI minutes burn is undesirable, it is not
inherantly worse than the CI minutes burn from all the other
build jobs. Contributors unforunately just need to be aware
of this and be more selective in running jobs.

If QEMU does eventually join the OSS program, then I believe
forks of QEMU will get an elevate CI allowance.

> 
> Does GitLab CI support some kind of async job so a container doesn't
> have to monitor Cirrus-CI for the duration of the tests? I guess it
> would require a job like the cirrus-run job, except a webhook signals
> completion and therefore the GitLab CI container doesn't need to wait
> around.

No, this consumption of GitLab CI minutes is an inherant limitation
of using the cirrus-run hack.  The real solution would be native
GItLab support for Cirrus CI. This would have Cirrus CI listening
for git changes, triggers the pipeline itself, and then reports
them back as a fake GitLab CI pipeline job. This is what Travis
is able todo these days, and the recommended way for external CI
systems to integrate.  No ETA on when Cirrus may support this.

https://github.com/cirruslabs/cirrus-ci-docs/issues/10


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Should we maybe move Cirrus-CI jobs away from Gitlab again?

[Stefan Hajnoczi]

On Tue, 27 Sept 2022 at 11:54, Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Tue, Sep 27, 2022 at 11:44:45AM -0400, Stefan Hajnoczi wrote:
> > On Tue, 27 Sept 2022 at 05:02, Thomas Huth <thuth@redhat.com> wrote:
> > > now that Gitlab is giving us pressure on the amount of free CI minutes, I
> > > wonder whether we should maybe move the Cirrus-CI jobs out of the gitlab-CI
> > > dashboard again? We could add the jobs to our .cirrus-ci.yml file instead,
> > > like we did it in former times...
> > >
> > > Big advantage would be of course that the time for those jobs would not
> > > count in the Gitlab-CI minutes anymore. Disadvantage is of course that they
> > > do not show up in the gitlab-CI dashboard anymore, so there is no more
> > > e-mail notification about failed jobs, and you have to push to github, too,
> > > and finally check the results manually on cirrus-ci.com ...
> >
> > My understanding is that .gitlab-ci.d/cirrus.yml uses a GitLab CI job
> > to run the cirrus-run container image that forwards jobs to Cirrus-CI.
> > So GitLab CI resources are consumed waiting for Cirrus-CI to finish.
> >
> > This shouldn't affect gitlab.com/qemu-project where there are private
> > runners that do not consume GitLab CI minutes.
> >
> > Individual developers are affected though because they most likely
> > rely on the GitLab shared runner minutes quota.
>
> NB, none of the jobs should ever be run automatically anymore in
> QEMU CI pipelines. It always requires the maintainer to set the
> env var when pushing to git, to explicitly create a pipeline.
> You can then selectively start each individual job as desired.

Cirrus CI is not automatically started when pushing to a personal
GitLab repo? If starting it requires manual action anyway then I think
nothing needs to be changed here.

Stefan

Re: Should we maybe move Cirrus-CI jobs away from Gitlab again?

[Daniel P. Berrangé]

On Tue, Sep 27, 2022 at 01:36:20PM -0400, Stefan Hajnoczi wrote:
> On Tue, 27 Sept 2022 at 11:54, Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Tue, Sep 27, 2022 at 11:44:45AM -0400, Stefan Hajnoczi wrote:
> > > On Tue, 27 Sept 2022 at 05:02, Thomas Huth <thuth@redhat.com> wrote:
> > > > now that Gitlab is giving us pressure on the amount of free CI minutes, I
> > > > wonder whether we should maybe move the Cirrus-CI jobs out of the gitlab-CI
> > > > dashboard again? We could add the jobs to our .cirrus-ci.yml file instead,
> > > > like we did it in former times...
> > > >
> > > > Big advantage would be of course that the time for those jobs would not
> > > > count in the Gitlab-CI minutes anymore. Disadvantage is of course that they
> > > > do not show up in the gitlab-CI dashboard anymore, so there is no more
> > > > e-mail notification about failed jobs, and you have to push to github, too,
> > > > and finally check the results manually on cirrus-ci.com ...
> > >
> > > My understanding is that .gitlab-ci.d/cirrus.yml uses a GitLab CI job
> > > to run the cirrus-run container image that forwards jobs to Cirrus-CI.
> > > So GitLab CI resources are consumed waiting for Cirrus-CI to finish.
> > >
> > > This shouldn't affect gitlab.com/qemu-project where there are private
> > > runners that do not consume GitLab CI minutes.
> > >
> > > Individual developers are affected though because they most likely
> > > rely on the GitLab shared runner minutes quota.
> >
> > NB, none of the jobs should ever be run automatically anymore in
> > QEMU CI pipelines. It always requires the maintainer to set the
> > env var when pushing to git, to explicitly create a pipeline.
> > You can then selectively start each individual job as desired.
> 
> Cirrus CI is not automatically started when pushing to a personal
> GitLab repo? If starting it requires manual action anyway then I think
> nothing needs to be changed here.

No pipeline at all is created unless you do

  git push -o ci.variable=QEMU_CI=1 <your-fork-remote>

that creates the pipeliune but doesn't run any jobs - they're manual
start. Or QEMU_CI=2 creates & starts the jobs (like the old way we
had CI until a few months ago, which burns CI quota hugely).


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Should we maybe move Cirrus-CI jobs away from Gitlab again?

[Thomas Huth]

On 27/09/2022 19.57, Daniel P. Berrangé wrote:
> On Tue, Sep 27, 2022 at 01:36:20PM -0400, Stefan Hajnoczi wrote:
>> On Tue, 27 Sept 2022 at 11:54, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>>
>>> On Tue, Sep 27, 2022 at 11:44:45AM -0400, Stefan Hajnoczi wrote:
>>>> On Tue, 27 Sept 2022 at 05:02, Thomas Huth <thuth@redhat.com> wrote:
>>>>> now that Gitlab is giving us pressure on the amount of free CI minutes, I
>>>>> wonder whether we should maybe move the Cirrus-CI jobs out of the gitlab-CI
>>>>> dashboard again? We could add the jobs to our .cirrus-ci.yml file instead,
>>>>> like we did it in former times...
>>>>>
>>>>> Big advantage would be of course that the time for those jobs would not
>>>>> count in the Gitlab-CI minutes anymore. Disadvantage is of course that they
>>>>> do not show up in the gitlab-CI dashboard anymore, so there is no more
>>>>> e-mail notification about failed jobs, and you have to push to github, too,
>>>>> and finally check the results manually on cirrus-ci.com ...
>>>>
>>>> My understanding is that .gitlab-ci.d/cirrus.yml uses a GitLab CI job
>>>> to run the cirrus-run container image that forwards jobs to Cirrus-CI.
>>>> So GitLab CI resources are consumed waiting for Cirrus-CI to finish.
>>>>
>>>> This shouldn't affect gitlab.com/qemu-project where there are private
>>>> runners that do not consume GitLab CI minutes.
>>>>
>>>> Individual developers are affected though because they most likely
>>>> rely on the GitLab shared runner minutes quota.
>>>
>>> NB, none of the jobs should ever be run automatically anymore in
>>> QEMU CI pipelines. It always requires the maintainer to set the
>>> env var when pushing to git, to explicitly create a pipeline.
>>> You can then selectively start each individual job as desired.
>>
>> Cirrus CI is not automatically started when pushing to a personal
>> GitLab repo? If starting it requires manual action anyway then I think
>> nothing needs to be changed here.
> 
> No pipeline at all is created unless you do
> 
>    git push -o ci.variable=QEMU_CI=1 <your-fork-remote>
> 
> that creates the pipeliune but doesn't run any jobs - they're manual
> start.

Yes, sure, the jobs are not started automatically. But I *do* want to run 
the jobs before sending pull requests - but since the gitlab-CI minutes are 
now very limited, I'd like to avoid burning these minutes via gitlab and 
start those jobs directly on cirrus-ci.com again. For that the jobs would 
need to be moved to our .cirrus-ci.yml file again.

Well, maybe we could also have both, jobs via cirrus-run for those who want 
to see them in their gitlab-CI dashboard, and via .cirrus-ci.yml for those 
who want to avoid burning CI minutes on Gitlab. It's a little bit of 
double-maintenance, but maybe acceptable?

  Thomas

Re: Should we maybe move Cirrus-CI jobs away from Gitlab again?

[Stefan Hajnoczi]

On Tue, 27 Sept 2022 at 14:40, Thomas Huth <thuth@redhat.com> wrote:
>
> On 27/09/2022 19.57, Daniel P. Berrangé wrote:
> > On Tue, Sep 27, 2022 at 01:36:20PM -0400, Stefan Hajnoczi wrote:
> >> On Tue, 27 Sept 2022 at 11:54, Daniel P. Berrangé <berrange@redhat.com> wrote:
> >>>
> >>> On Tue, Sep 27, 2022 at 11:44:45AM -0400, Stefan Hajnoczi wrote:
> >>>> On Tue, 27 Sept 2022 at 05:02, Thomas Huth <thuth@redhat.com> wrote:
> >>>>> now that Gitlab is giving us pressure on the amount of free CI minutes, I
> >>>>> wonder whether we should maybe move the Cirrus-CI jobs out of the gitlab-CI
> >>>>> dashboard again? We could add the jobs to our .cirrus-ci.yml file instead,
> >>>>> like we did it in former times...
> >>>>>
> >>>>> Big advantage would be of course that the time for those jobs would not
> >>>>> count in the Gitlab-CI minutes anymore. Disadvantage is of course that they
> >>>>> do not show up in the gitlab-CI dashboard anymore, so there is no more
> >>>>> e-mail notification about failed jobs, and you have to push to github, too,
> >>>>> and finally check the results manually on cirrus-ci.com ...
> >>>>
> >>>> My understanding is that .gitlab-ci.d/cirrus.yml uses a GitLab CI job
> >>>> to run the cirrus-run container image that forwards jobs to Cirrus-CI.
> >>>> So GitLab CI resources are consumed waiting for Cirrus-CI to finish.
> >>>>
> >>>> This shouldn't affect gitlab.com/qemu-project where there are private
> >>>> runners that do not consume GitLab CI minutes.
> >>>>
> >>>> Individual developers are affected though because they most likely
> >>>> rely on the GitLab shared runner minutes quota.
> >>>
> >>> NB, none of the jobs should ever be run automatically anymore in
> >>> QEMU CI pipelines. It always requires the maintainer to set the
> >>> env var when pushing to git, to explicitly create a pipeline.
> >>> You can then selectively start each individual job as desired.
> >>
> >> Cirrus CI is not automatically started when pushing to a personal
> >> GitLab repo? If starting it requires manual action anyway then I think
> >> nothing needs to be changed here.
> >
> > No pipeline at all is created unless you do
> >
> >    git push -o ci.variable=QEMU_CI=1 <your-fork-remote>
> >
> > that creates the pipeliune but doesn't run any jobs - they're manual
> > start.
>
> Yes, sure, the jobs are not started automatically. But I *do* want to run
> the jobs before sending pull requests - but since the gitlab-CI minutes are
> now very limited, I'd like to avoid burning these minutes via gitlab and
> start those jobs directly on cirrus-ci.com again. For that the jobs would
> need to be moved to our .cirrus-ci.yml file again.
>
> Well, maybe we could also have both, jobs via cirrus-run for those who want
> to see them in their gitlab-CI dashboard, and via .cirrus-ci.yml for those
> who want to avoid burning CI minutes on Gitlab. It's a little bit of
> double-maintenance, but maybe acceptable?

I just noticed that qemu.git/master doesn't run Cirrus-CI. I guess it
hasn't been set up in our GitLab project.

Since it's not enabled for qemu.git/master nothing will change from my
perspective. Feel free to change it as you wish.

Stefan

Re: Should we maybe move Cirrus-CI jobs away from Gitlab again?

[Thomas Huth]

On 27/09/2022 20.47, Stefan Hajnoczi wrote:
> On Tue, 27 Sept 2022 at 14:40, Thomas Huth <thuth@redhat.com> wrote:
>>
>> On 27/09/2022 19.57, Daniel P. Berrangé wrote:
>>> On Tue, Sep 27, 2022 at 01:36:20PM -0400, Stefan Hajnoczi wrote:
>>>> On Tue, 27 Sept 2022 at 11:54, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>>>>
>>>>> On Tue, Sep 27, 2022 at 11:44:45AM -0400, Stefan Hajnoczi wrote:
>>>>>> On Tue, 27 Sept 2022 at 05:02, Thomas Huth <thuth@redhat.com> wrote:
>>>>>>> now that Gitlab is giving us pressure on the amount of free CI minutes, I
>>>>>>> wonder whether we should maybe move the Cirrus-CI jobs out of the gitlab-CI
>>>>>>> dashboard again? We could add the jobs to our .cirrus-ci.yml file instead,
>>>>>>> like we did it in former times...
>>>>>>>
>>>>>>> Big advantage would be of course that the time for those jobs would not
>>>>>>> count in the Gitlab-CI minutes anymore. Disadvantage is of course that they
>>>>>>> do not show up in the gitlab-CI dashboard anymore, so there is no more
>>>>>>> e-mail notification about failed jobs, and you have to push to github, too,
>>>>>>> and finally check the results manually on cirrus-ci.com ...
>>>>>>
>>>>>> My understanding is that .gitlab-ci.d/cirrus.yml uses a GitLab CI job
>>>>>> to run the cirrus-run container image that forwards jobs to Cirrus-CI.
>>>>>> So GitLab CI resources are consumed waiting for Cirrus-CI to finish.
>>>>>>
>>>>>> This shouldn't affect gitlab.com/qemu-project where there are private
>>>>>> runners that do not consume GitLab CI minutes.
>>>>>>
>>>>>> Individual developers are affected though because they most likely
>>>>>> rely on the GitLab shared runner minutes quota.
>>>>>
>>>>> NB, none of the jobs should ever be run automatically anymore in
>>>>> QEMU CI pipelines. It always requires the maintainer to set the
>>>>> env var when pushing to git, to explicitly create a pipeline.
>>>>> You can then selectively start each individual job as desired.
>>>>
>>>> Cirrus CI is not automatically started when pushing to a personal
>>>> GitLab repo? If starting it requires manual action anyway then I think
>>>> nothing needs to be changed here.
>>>
>>> No pipeline at all is created unless you do
>>>
>>>     git push -o ci.variable=QEMU_CI=1 <your-fork-remote>
>>>
>>> that creates the pipeliune but doesn't run any jobs - they're manual
>>> start.
>>
>> Yes, sure, the jobs are not started automatically. But I *do* want to run
>> the jobs before sending pull requests - but since the gitlab-CI minutes are
>> now very limited, I'd like to avoid burning these minutes via gitlab and
>> start those jobs directly on cirrus-ci.com again. For that the jobs would
>> need to be moved to our .cirrus-ci.yml file again.
>>
>> Well, maybe we could also have both, jobs via cirrus-run for those who want
>> to see them in their gitlab-CI dashboard, and via .cirrus-ci.yml for those
>> who want to avoid burning CI minutes on Gitlab. It's a little bit of
>> double-maintenance, but maybe acceptable?
> 
> I just noticed that qemu.git/master doesn't run Cirrus-CI. I guess it
> hasn't been set up in our GitLab project.
> 
> Since it's not enabled for qemu.git/master nothing will change from my
> perspective. Feel free to change it as you wish.

It's only run for the "staging" branch, I think. The idea was that things 
get tested before merge on the "staging" branch, then there is no need 
anymore to rerun everything when it gets pushed into the "master" branch.

  Thomas

Re: Should we maybe move Cirrus-CI jobs away from Gitlab again?

[Stefan Hajnoczi]

On Tue, 27 Sept 2022 at 15:04, Thomas Huth <thuth@redhat.com> wrote:
>
> On 27/09/2022 20.47, Stefan Hajnoczi wrote:
> > On Tue, 27 Sept 2022 at 14:40, Thomas Huth <thuth@redhat.com> wrote:
> >>
> >> On 27/09/2022 19.57, Daniel P. Berrangé wrote:
> >>> On Tue, Sep 27, 2022 at 01:36:20PM -0400, Stefan Hajnoczi wrote:
> >>>> On Tue, 27 Sept 2022 at 11:54, Daniel P. Berrangé <berrange@redhat.com> wrote:
> >>>>>
> >>>>> On Tue, Sep 27, 2022 at 11:44:45AM -0400, Stefan Hajnoczi wrote:
> >>>>>> On Tue, 27 Sept 2022 at 05:02, Thomas Huth <thuth@redhat.com> wrote:
> >>>>>>> now that Gitlab is giving us pressure on the amount of free CI minutes, I
> >>>>>>> wonder whether we should maybe move the Cirrus-CI jobs out of the gitlab-CI
> >>>>>>> dashboard again? We could add the jobs to our .cirrus-ci.yml file instead,
> >>>>>>> like we did it in former times...
> >>>>>>>
> >>>>>>> Big advantage would be of course that the time for those jobs would not
> >>>>>>> count in the Gitlab-CI minutes anymore. Disadvantage is of course that they
> >>>>>>> do not show up in the gitlab-CI dashboard anymore, so there is no more
> >>>>>>> e-mail notification about failed jobs, and you have to push to github, too,
> >>>>>>> and finally check the results manually on cirrus-ci.com ...
> >>>>>>
> >>>>>> My understanding is that .gitlab-ci.d/cirrus.yml uses a GitLab CI job
> >>>>>> to run the cirrus-run container image that forwards jobs to Cirrus-CI.
> >>>>>> So GitLab CI resources are consumed waiting for Cirrus-CI to finish.
> >>>>>>
> >>>>>> This shouldn't affect gitlab.com/qemu-project where there are private
> >>>>>> runners that do not consume GitLab CI minutes.
> >>>>>>
> >>>>>> Individual developers are affected though because they most likely
> >>>>>> rely on the GitLab shared runner minutes quota.
> >>>>>
> >>>>> NB, none of the jobs should ever be run automatically anymore in
> >>>>> QEMU CI pipelines. It always requires the maintainer to set the
> >>>>> env var when pushing to git, to explicitly create a pipeline.
> >>>>> You can then selectively start each individual job as desired.
> >>>>
> >>>> Cirrus CI is not automatically started when pushing to a personal
> >>>> GitLab repo? If starting it requires manual action anyway then I think
> >>>> nothing needs to be changed here.
> >>>
> >>> No pipeline at all is created unless you do
> >>>
> >>>     git push -o ci.variable=QEMU_CI=1 <your-fork-remote>
> >>>
> >>> that creates the pipeliune but doesn't run any jobs - they're manual
> >>> start.
> >>
> >> Yes, sure, the jobs are not started automatically. But I *do* want to run
> >> the jobs before sending pull requests - but since the gitlab-CI minutes are
> >> now very limited, I'd like to avoid burning these minutes via gitlab and
> >> start those jobs directly on cirrus-ci.com again. For that the jobs would
> >> need to be moved to our .cirrus-ci.yml file again.
> >>
> >> Well, maybe we could also have both, jobs via cirrus-run for those who want
> >> to see them in their gitlab-CI dashboard, and via .cirrus-ci.yml for those
> >> who want to avoid burning CI minutes on Gitlab. It's a little bit of
> >> double-maintenance, but maybe acceptable?
> >
> > I just noticed that qemu.git/master doesn't run Cirrus-CI. I guess it
> > hasn't been set up in our GitLab project.
> >
> > Since it's not enabled for qemu.git/master nothing will change from my
> > perspective. Feel free to change it as you wish.
>
> It's only run for the "staging" branch, I think. The idea was that things
> get tested before merge on the "staging" branch, then there is no need
> anymore to rerun everything when it gets pushed into the "master" branch.

I don't see a cirrus job:
https://gitlab.com/qemu-project/qemu/-/pipelines/652051335

Stefan

Re: Should we maybe move Cirrus-CI jobs away from Gitlab again?

[Thomas Huth]

On 27/09/2022 21.10, Stefan Hajnoczi wrote:
> On Tue, 27 Sept 2022 at 15:04, Thomas Huth <thuth@redhat.com> wrote:
>>
>> On 27/09/2022 20.47, Stefan Hajnoczi wrote:
>>> On Tue, 27 Sept 2022 at 14:40, Thomas Huth <thuth@redhat.com> wrote:
>>>>
>>>> On 27/09/2022 19.57, Daniel P. Berrangé wrote:
>>>>> On Tue, Sep 27, 2022 at 01:36:20PM -0400, Stefan Hajnoczi wrote:
>>>>>> On Tue, 27 Sept 2022 at 11:54, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>>>>>>
>>>>>>> On Tue, Sep 27, 2022 at 11:44:45AM -0400, Stefan Hajnoczi wrote:
>>>>>>>> On Tue, 27 Sept 2022 at 05:02, Thomas Huth <thuth@redhat.com> wrote:
>>>>>>>>> now that Gitlab is giving us pressure on the amount of free CI minutes, I
>>>>>>>>> wonder whether we should maybe move the Cirrus-CI jobs out of the gitlab-CI
>>>>>>>>> dashboard again? We could add the jobs to our .cirrus-ci.yml file instead,
>>>>>>>>> like we did it in former times...
>>>>>>>>>
>>>>>>>>> Big advantage would be of course that the time for those jobs would not
>>>>>>>>> count in the Gitlab-CI minutes anymore. Disadvantage is of course that they
>>>>>>>>> do not show up in the gitlab-CI dashboard anymore, so there is no more
>>>>>>>>> e-mail notification about failed jobs, and you have to push to github, too,
>>>>>>>>> and finally check the results manually on cirrus-ci.com ...
>>>>>>>>
>>>>>>>> My understanding is that .gitlab-ci.d/cirrus.yml uses a GitLab CI job
>>>>>>>> to run the cirrus-run container image that forwards jobs to Cirrus-CI.
>>>>>>>> So GitLab CI resources are consumed waiting for Cirrus-CI to finish.
>>>>>>>>
>>>>>>>> This shouldn't affect gitlab.com/qemu-project where there are private
>>>>>>>> runners that do not consume GitLab CI minutes.
>>>>>>>>
>>>>>>>> Individual developers are affected though because they most likely
>>>>>>>> rely on the GitLab shared runner minutes quota.
>>>>>>>
>>>>>>> NB, none of the jobs should ever be run automatically anymore in
>>>>>>> QEMU CI pipelines. It always requires the maintainer to set the
>>>>>>> env var when pushing to git, to explicitly create a pipeline.
>>>>>>> You can then selectively start each individual job as desired.
>>>>>>
>>>>>> Cirrus CI is not automatically started when pushing to a personal
>>>>>> GitLab repo? If starting it requires manual action anyway then I think
>>>>>> nothing needs to be changed here.
>>>>>
>>>>> No pipeline at all is created unless you do
>>>>>
>>>>>      git push -o ci.variable=QEMU_CI=1 <your-fork-remote>
>>>>>
>>>>> that creates the pipeliune but doesn't run any jobs - they're manual
>>>>> start.
>>>>
>>>> Yes, sure, the jobs are not started automatically. But I *do* want to run
>>>> the jobs before sending pull requests - but since the gitlab-CI minutes are
>>>> now very limited, I'd like to avoid burning these minutes via gitlab and
>>>> start those jobs directly on cirrus-ci.com again. For that the jobs would
>>>> need to be moved to our .cirrus-ci.yml file again.
>>>>
>>>> Well, maybe we could also have both, jobs via cirrus-run for those who want
>>>> to see them in their gitlab-CI dashboard, and via .cirrus-ci.yml for those
>>>> who want to avoid burning CI minutes on Gitlab. It's a little bit of
>>>> double-maintenance, but maybe acceptable?
>>>
>>> I just noticed that qemu.git/master doesn't run Cirrus-CI. I guess it
>>> hasn't been set up in our GitLab project.
>>>
>>> Since it's not enabled for qemu.git/master nothing will change from my
>>> perspective. Feel free to change it as you wish.
>>
>> It's only run for the "staging" branch, I think. The idea was that things
>> get tested before merge on the "staging" branch, then there is no need
>> anymore to rerun everything when it gets pushed into the "master" branch.
> 
> I don't see a cirrus job:
> https://gitlab.com/qemu-project/qemu/-/pipelines/652051335

That's likely because it had been disabled on github when we switched to 
cirrus-run via gitlab. I still have it enabled for my forked repo on github, 
so you can see runs here:

  https://cirrus-ci.com/github/huth/qemu/

  Thomas

Re: Should we maybe move Cirrus-CI jobs away from Gitlab again?

[Daniel P. Berrangé]

On Tue, Sep 27, 2022 at 03:10:54PM -0400, Stefan Hajnoczi wrote:
> On Tue, 27 Sept 2022 at 15:04, Thomas Huth <thuth@redhat.com> wrote:
> >
> > On 27/09/2022 20.47, Stefan Hajnoczi wrote:
> > > On Tue, 27 Sept 2022 at 14:40, Thomas Huth <thuth@redhat.com> wrote:
> > >>
> > >> On 27/09/2022 19.57, Daniel P. Berrangé wrote:
> > >>> On Tue, Sep 27, 2022 at 01:36:20PM -0400, Stefan Hajnoczi wrote:
> > >>>> On Tue, 27 Sept 2022 at 11:54, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > >>>>>
> > >>>>> On Tue, Sep 27, 2022 at 11:44:45AM -0400, Stefan Hajnoczi wrote:
> > >>>>>> On Tue, 27 Sept 2022 at 05:02, Thomas Huth <thuth@redhat.com> wrote:
> > >>>>>>> now that Gitlab is giving us pressure on the amount of free CI minutes, I
> > >>>>>>> wonder whether we should maybe move the Cirrus-CI jobs out of the gitlab-CI
> > >>>>>>> dashboard again? We could add the jobs to our .cirrus-ci.yml file instead,
> > >>>>>>> like we did it in former times...
> > >>>>>>>
> > >>>>>>> Big advantage would be of course that the time for those jobs would not
> > >>>>>>> count in the Gitlab-CI minutes anymore. Disadvantage is of course that they
> > >>>>>>> do not show up in the gitlab-CI dashboard anymore, so there is no more
> > >>>>>>> e-mail notification about failed jobs, and you have to push to github, too,
> > >>>>>>> and finally check the results manually on cirrus-ci.com ...
> > >>>>>>
> > >>>>>> My understanding is that .gitlab-ci.d/cirrus.yml uses a GitLab CI job
> > >>>>>> to run the cirrus-run container image that forwards jobs to Cirrus-CI.
> > >>>>>> So GitLab CI resources are consumed waiting for Cirrus-CI to finish.
> > >>>>>>
> > >>>>>> This shouldn't affect gitlab.com/qemu-project where there are private
> > >>>>>> runners that do not consume GitLab CI minutes.
> > >>>>>>
> > >>>>>> Individual developers are affected though because they most likely
> > >>>>>> rely on the GitLab shared runner minutes quota.
> > >>>>>
> > >>>>> NB, none of the jobs should ever be run automatically anymore in
> > >>>>> QEMU CI pipelines. It always requires the maintainer to set the
> > >>>>> env var when pushing to git, to explicitly create a pipeline.
> > >>>>> You can then selectively start each individual job as desired.
> > >>>>
> > >>>> Cirrus CI is not automatically started when pushing to a personal
> > >>>> GitLab repo? If starting it requires manual action anyway then I think
> > >>>> nothing needs to be changed here.
> > >>>
> > >>> No pipeline at all is created unless you do
> > >>>
> > >>>     git push -o ci.variable=QEMU_CI=1 <your-fork-remote>
> > >>>
> > >>> that creates the pipeliune but doesn't run any jobs - they're manual
> > >>> start.
> > >>
> > >> Yes, sure, the jobs are not started automatically. But I *do* want to run
> > >> the jobs before sending pull requests - but since the gitlab-CI minutes are
> > >> now very limited, I'd like to avoid burning these minutes via gitlab and
> > >> start those jobs directly on cirrus-ci.com again. For that the jobs would
> > >> need to be moved to our .cirrus-ci.yml file again.
> > >>
> > >> Well, maybe we could also have both, jobs via cirrus-run for those who want
> > >> to see them in their gitlab-CI dashboard, and via .cirrus-ci.yml for those
> > >> who want to avoid burning CI minutes on Gitlab. It's a little bit of
> > >> double-maintenance, but maybe acceptable?
> > >
> > > I just noticed that qemu.git/master doesn't run Cirrus-CI. I guess it
> > > hasn't been set up in our GitLab project.
> > >
> > > Since it's not enabled for qemu.git/master nothing will change from my
> > > perspective. Feel free to change it as you wish.
> >
> > It's only run for the "staging" branch, I think. The idea was that things
> > get tested before merge on the "staging" branch, then there is no need
> > anymore to rerun everything when it gets pushed into the "master" branch.
> 
> I don't see a cirrus job:
> https://gitlab.com/qemu-project/qemu/-/pipelines/652051335

There are in fact five cirrus CI jobs there - x64-freebsd-12-build,
x64-freebsd-13-build, x64-macos-11-base-build, x86-netbsd and
x86-openbsd, though we skip the latter two because Cirrus limits
to 2 parallel jobs and running more 3 total would mean we timeout
too often.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Should we maybe move Cirrus-CI jobs away from Gitlab again?

[Daniel P. Berrangé]

On Tue, Sep 27, 2022 at 08:40:44PM +0200, Thomas Huth wrote:
> On 27/09/2022 19.57, Daniel P. Berrangé wrote:
> > On Tue, Sep 27, 2022 at 01:36:20PM -0400, Stefan Hajnoczi wrote:
> > > On Tue, 27 Sept 2022 at 11:54, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > > > 
> > > > On Tue, Sep 27, 2022 at 11:44:45AM -0400, Stefan Hajnoczi wrote:
> > > > > On Tue, 27 Sept 2022 at 05:02, Thomas Huth <thuth@redhat.com> wrote:
> > > > > > now that Gitlab is giving us pressure on the amount of free CI minutes, I
> > > > > > wonder whether we should maybe move the Cirrus-CI jobs out of the gitlab-CI
> > > > > > dashboard again? We could add the jobs to our .cirrus-ci.yml file instead,
> > > > > > like we did it in former times...
> > > > > > 
> > > > > > Big advantage would be of course that the time for those jobs would not
> > > > > > count in the Gitlab-CI minutes anymore. Disadvantage is of course that they
> > > > > > do not show up in the gitlab-CI dashboard anymore, so there is no more
> > > > > > e-mail notification about failed jobs, and you have to push to github, too,
> > > > > > and finally check the results manually on cirrus-ci.com ...
> > > > > 
> > > > > My understanding is that .gitlab-ci.d/cirrus.yml uses a GitLab CI job
> > > > > to run the cirrus-run container image that forwards jobs to Cirrus-CI.
> > > > > So GitLab CI resources are consumed waiting for Cirrus-CI to finish.
> > > > > 
> > > > > This shouldn't affect gitlab.com/qemu-project where there are private
> > > > > runners that do not consume GitLab CI minutes.
> > > > > 
> > > > > Individual developers are affected though because they most likely
> > > > > rely on the GitLab shared runner minutes quota.
> > > > 
> > > > NB, none of the jobs should ever be run automatically anymore in
> > > > QEMU CI pipelines. It always requires the maintainer to set the
> > > > env var when pushing to git, to explicitly create a pipeline.
> > > > You can then selectively start each individual job as desired.
> > > 
> > > Cirrus CI is not automatically started when pushing to a personal
> > > GitLab repo? If starting it requires manual action anyway then I think
> > > nothing needs to be changed here.
> > 
> > No pipeline at all is created unless you do
> > 
> >    git push -o ci.variable=QEMU_CI=1 <your-fork-remote>
> > 
> > that creates the pipeliune but doesn't run any jobs - they're manual
> > start.
> 
> Yes, sure, the jobs are not started automatically. But I *do* want to run
> the jobs before sending pull requests - but since the gitlab-CI minutes are
> now very limited, I'd like to avoid burning these minutes via gitlab and
> start those jobs directly on cirrus-ci.com again. For that the jobs would
> need to be moved to our .cirrus-ci.yml file again.

We do need a better story for maintainers sending pull requests to have
ability to run CI. We have 50+ jobs in the bujild stage of which the
cirrus jobs are just 3 - removing the cirrus jobs won't make a difference
to how quickly we run out of minutes if people try to run all of them.

We need to define a much tighter minimalist set of recommended jobs to
run.

I believe that if QEMU joins the OSS program, then the forks of QEMU
also benefit from a reduced cost factor for jobs they run, effectively
giving you much higher CI quota

> Well, maybe we could also have both, jobs via cirrus-run for those who want
> to see them in their gitlab-CI dashboard, and via .cirrus-ci.yml for those
> who want to avoid burning CI minutes on Gitlab. It's a little bit of
> double-maintenance, but maybe acceptable?

Key info about the jobs is in .gitlab-ci.d/cirrus/freebsd-12.vars which
could be referenced from the cirrus-ci.yml to reduce duplication

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Should we maybe move Cirrus-CI jobs away from Gitlab again?

[Alex Bennée]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Tue, Sep 27, 2022 at 08:40:44PM +0200, Thomas Huth wrote:
>> On 27/09/2022 19.57, Daniel P. Berrangé wrote:
>> > On Tue, Sep 27, 2022 at 01:36:20PM -0400, Stefan Hajnoczi wrote:
>> > > On Tue, 27 Sept 2022 at 11:54, Daniel P. Berrangé <berrange@redhat.com> wrote:
>> > > > 
>> > > > On Tue, Sep 27, 2022 at 11:44:45AM -0400, Stefan Hajnoczi wrote:
>> > > > > On Tue, 27 Sept 2022 at 05:02, Thomas Huth <thuth@redhat.com> wrote:
>> > > > > > now that Gitlab is giving us pressure on the amount of free CI minutes, I
>> > > > > > wonder whether we should maybe move the Cirrus-CI jobs out of the gitlab-CI
>> > > > > > dashboard again? We could add the jobs to our .cirrus-ci.yml file instead,
>> > > > > > like we did it in former times...
>> > > > > > 
>> > > > > > Big advantage would be of course that the time for those jobs would not
>> > > > > > count in the Gitlab-CI minutes anymore. Disadvantage is of course that they
>> > > > > > do not show up in the gitlab-CI dashboard anymore, so there is no more
>> > > > > > e-mail notification about failed jobs, and you have to push to github, too,
>> > > > > > and finally check the results manually on cirrus-ci.com ...
>> > > > > 
>> > > > > My understanding is that .gitlab-ci.d/cirrus.yml uses a GitLab CI job
>> > > > > to run the cirrus-run container image that forwards jobs to Cirrus-CI.
>> > > > > So GitLab CI resources are consumed waiting for Cirrus-CI to finish.
>> > > > > 
>> > > > > This shouldn't affect gitlab.com/qemu-project where there are private
>> > > > > runners that do not consume GitLab CI minutes.
>> > > > > 
>> > > > > Individual developers are affected though because they most likely
>> > > > > rely on the GitLab shared runner minutes quota.
>> > > > 
>> > > > NB, none of the jobs should ever be run automatically anymore in
>> > > > QEMU CI pipelines. It always requires the maintainer to set the
>> > > > env var when pushing to git, to explicitly create a pipeline.
>> > > > You can then selectively start each individual job as desired.
>> > > 
>> > > Cirrus CI is not automatically started when pushing to a personal
>> > > GitLab repo? If starting it requires manual action anyway then I think
>> > > nothing needs to be changed here.
>> > 
>> > No pipeline at all is created unless you do
>> > 
>> >    git push -o ci.variable=QEMU_CI=1 <your-fork-remote>
>> > 
>> > that creates the pipeliune but doesn't run any jobs - they're manual
>> > start.
>> 
>> Yes, sure, the jobs are not started automatically. But I *do* want to run
>> the jobs before sending pull requests - but since the gitlab-CI minutes are
>> now very limited, I'd like to avoid burning these minutes via gitlab and
>> start those jobs directly on cirrus-ci.com again. For that the jobs would
>> need to be moved to our .cirrus-ci.yml file again.
>
> We do need a better story for maintainers sending pull requests to have
> ability to run CI. We have 50+ jobs in the bujild stage of which the
> cirrus jobs are just 3 - removing the cirrus jobs won't make a difference
> to how quickly we run out of minutes if people try to run all of them.
>
> We need to define a much tighter minimalist set of recommended jobs to
> run.
>
> I believe that if QEMU joins the OSS program, then the forks of QEMU
> also benefit from a reduced cost factor for jobs they run, effectively
> giving you much higher CI quota

We shall find out soon enough. The code came through today so I've
applied it to the project which shows we are now on the "Ultimate" tier.

>
>> Well, maybe we could also have both, jobs via cirrus-run for those who want
>> to see them in their gitlab-CI dashboard, and via .cirrus-ci.yml for those
>> who want to avoid burning CI minutes on Gitlab. It's a little bit of
>> double-maintenance, but maybe acceptable?
>
> Key info about the jobs is in .gitlab-ci.d/cirrus/freebsd-12.vars which
> could be referenced from the cirrus-ci.yml to reduce duplication
>
> With regards,
> Daniel


-- 
Alex Bennée

Re: Should we maybe move Cirrus-CI jobs away from Gitlab again?

[Stefan Hajnoczi]

On Wed, 28 Sept 2022 at 08:28, Alex Bennée <alex.bennee@linaro.org> wrote:
> We shall find out soon enough. The code came through today so I've
> applied it to the project which shows we are now on the "Ultimate" tier.

Thank you, Alex!

Stefan