Should we apply for GitLab's open source program?

Should we apply for GitLab's open source program?

[Alex Bennée]

Hi,

Given our growing reliance on GitLab and the recent announcement about
free tier minutes:

  https://about.gitlab.com/pricing/faq-consumption-cicd/

is it time we officially apply for GitLab's Open Source Program:

  https://about.gitlab.com/solutions/open-source/program/

?

From what I can see the QEMU project will easily qualify - the only
minor inconvenience is needing to reapply every year. So far it seems as
a public project their are no usage limits anyway. We are currently
listed as 0 of 50,000 minutes:

  https://gitlab.com/groups/qemu-project/-/usage_quotas#pipelines-quota-tab

So we are in no pressing hurry to do this for the minutes but I suspect
there may be other things that are made easier by having access to all
the toys GitLab provides. Daniel has already posted to the forum
requesting details about how this might affect our workflow so maybe we
should just wait for feedback until pressing on?

  https://forum.gitlab.com/t/ci-cd-minutes-for-free-tier/40241/33

-- 
Alex Bennée

Re: Should we apply for GitLab's open source program?

[Daniel P. Berrangé]

On Fri, Sep 04, 2020 at 04:35:34PM +0100, Alex Bennée wrote:
> 
> Hi,
> 
> Given our growing reliance on GitLab and the recent announcement about
> free tier minutes:
> 
>   https://about.gitlab.com/pricing/faq-consumption-cicd/
> 
> is it time we officially apply for GitLab's Open Source Program:
> 
>   https://about.gitlab.com/solutions/open-source/program/
> 
> ?
> 
> From what I can see the QEMU project will easily qualify - the only
> minor inconvenience is needing to reapply every year. So far it seems as
> a public project their are no usage limits anyway. We are currently
> listed as 0 of 50,000 minutes:
> 
>   https://gitlab.com/groups/qemu-project/-/usage_quotas#pipelines-quota-tab

I suspect that ultimately we will end up wanting / needing to do
this. As you say, there's no significant downside, and it will
unlock features.

Note there is an issue open about making the application process
simpler:

https://gitlab.com/groups/gitlab-com/marketing/community-relations/opensource-program/-/epics/18

which could be a reason to not rush into applying immediately
if we don't have an obvious pressing need.

As you say there's no enforcement of CI minutes at all right now
anyway as reported at:

  https://gitlab.com/gitlab-org/gitlab/-/issues/243722


> So we are in no pressing hurry to do this for the minutes but I suspect
> there may be other things that are made easier by having access to all
> the toys GitLab provides. Daniel has already posted to the forum
> requesting details about how this might affect our workflow so maybe we
> should just wait for feedback until pressing on?
> 
>   https://forum.gitlab.com/t/ci-cd-minutes-for-free-tier/40241/33

Yep, my key observation is that I don't think the open source program
fully solves the CI minutes problems for projects using a fork'ing
workflow, as the increased minutes only applies for jobs run in the
main repo. Each individual contributor is still limited in their
fork.  Similarly their suggestion that people bring their own
custom runners doesn't really help projects with a forking workflow
as custom runners aren't accessible to forks.

It would be crazy if they enforced limited CI time on public repos
right now, because if any individual contributor runs out of CI
minutes they will be unable to contribute to a project that mandates
succesful CI pipeline, even if the project is in the OSS program.

Hopefully they will come up with a more practical answer to the
forking-workflow problems.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Should we apply for GitLab's open source program?

[Daniel P. Berrangé]

On Fri, Sep 04, 2020 at 05:08:36PM +0100, Daniel P. Berrangé wrote:
> On Fri, Sep 04, 2020 at 04:35:34PM +0100, Alex Bennée wrote:
> > 
> > Hi,
> > 
> > Given our growing reliance on GitLab and the recent announcement about
> > free tier minutes:
> > 
> >   https://about.gitlab.com/pricing/faq-consumption-cicd/
> > 
> > is it time we officially apply for GitLab's Open Source Program:
> > 
> >   https://about.gitlab.com/solutions/open-source/program/
> > 
> > ?
> > 
> > From what I can see the QEMU project will easily qualify - the only
> > minor inconvenience is needing to reapply every year. So far it seems as
> > a public project their are no usage limits anyway. We are currently
> > listed as 0 of 50,000 minutes:
> > 
> >   https://gitlab.com/groups/qemu-project/-/usage_quotas#pipelines-quota-tab
> 
> I suspect that ultimately we will end up wanting / needing to do
> this. As you say, there's no significant downside, and it will
> unlock features.
> 
> Note there is an issue open about making the application process
> simpler:
> 
> https://gitlab.com/groups/gitlab-com/marketing/community-relations/opensource-program/-/epics/18
> 
> which could be a reason to not rush into applying immediately
> if we don't have an obvious pressing need.
> 
> As you say there's no enforcement of CI minutes at all right now
> anyway as reported at:
> 
>   https://gitlab.com/gitlab-org/gitlab/-/issues/243722

FYI, I've very roughly summed up the CI minutes consumed on

 https://gitlab.com/qemu-project/qemu/-/pipelines/192481227/builds

and it comes to around 1400 minutes.

So the default 400 minute limit proposed is clearly useless.

The Open Source Program  offers 50,000 minutes. With our current
set of jobs that allows for 35 CI pipelines per month, barely
one per day.

IOW, I fear QEMU will easily hit the GitLab CI minute quota even
with the 50k minutes per month limit.

Libvirt only uses about 450 CI minutes per pipeline, but we
merge code on a more frequent basis since we don't use the
pull request model, so we'll use up a 50k minutes allowance
very quickly too.

It looks like joining the Open Source program is a must have
as the proposde 400 CI minute quota is unusably small.

Even once we do that though, it looks inescapable that projects
of our scale will need to bring more of our own CI hardware to
the table, or failing that, continue to leverage a wide variety
3rd party CI systems (travis, cirrus, etc). 

It is still unclear how we'll cope with contributors doing
CI in their own forks, but GitLab continue to offer positive
feedback that they want to make projects in our situation
succesful and thus want to find some kind of solution to the
CI quota problem with the forking workflow.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Should we apply for GitLab's open source program?

[Paolo Bonzini]

On 21/09/20 16:03, Daniel P. Berrangé wrote:
> It is still unclear how we'll cope with contributors doing
> CI in their own forks, but GitLab continue to offer positive
> feedback that they want to make projects in our situation
> succesful and thus want to find some kind of solution to the
> CI quota problem with the forking workflow.

The forking workflow is basically their value proposition, so my
impression is that they actually have three tiers:

1) small DIY projects that can live with the 400 minutes

2) growing projects that they want to keep an eye on and to enthuse with
their more proprietary offering (the open source program)

3) established projects that give them brand recognition, for which they
are okay with infinite minutes (within reasonable bounds).

Hopefully QEMU falls under the third.

Paolo

Re: Should we apply for GitLab's open source program?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1377 bytes --]

On Fri, Sep 04, 2020 at 04:35:34PM +0100, Alex Bennée wrote:
> Given our growing reliance on GitLab and the recent announcement about
> free tier minutes:
> 
>   https://about.gitlab.com/pricing/faq-consumption-cicd/
> 
> is it time we officially apply for GitLab's Open Source Program:
> 
>   https://about.gitlab.com/solutions/open-source/program/
> 
> ?
> 
> From what I can see the QEMU project will easily qualify - the only
> minor inconvenience is needing to reapply every year. So far it seems as
> a public project their are no usage limits anyway. We are currently
> listed as 0 of 50,000 minutes:
> 
>   https://gitlab.com/groups/qemu-project/-/usage_quotas#pipelines-quota-tab
> 
> So we are in no pressing hurry to do this for the minutes but I suspect
> there may be other things that are made easier by having access to all
> the toys GitLab provides. Daniel has already posted to the forum
> requesting details about how this might affect our workflow so maybe we
> should just wait for feedback until pressing on?
> 
>   https://forum.gitlab.com/t/ci-cd-minutes-for-free-tier/40241/33

Yes, please apply. If there is anything I can help with, please let me
know.

QEMU has Rackspace hosting credit so we can run x86 CI runners. I can
help set that up but need input regarding the number of instances, RAM,
OSes, etc.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Should we apply for GitLab's open source program?

[Bradley M. Kuhn]

> On Fri, Sep 04, 2020 at 04:35:34PM +0100, Alex Bennée wrote:
>> Given our growing reliance on GitLab and the recent announcement about
>> free tier minutes:
>> 
>>   https://about.gitlab.com/pricing/faq-consumption-cicd/
>> 
>> is it time we officially apply for GitLab's Open Source Program:
>> 
>>   https://about.gitlab.com/solutions/open-source/program/

Sorry for my late response on this thread.  Since GitLab is requiring that
organizations be non-profits as part of this program, Conservancy will need
to coordinate your application with you as we're the parent non-profit for
this purpose.

Given my late reply, you may already be coordinating with my colleague Brett
about this, but I'll check in with him tomorrow to verify.

One thing to note is that my understanding is that most of what you're
getting access to through this program is proprietary software features that
GitLab offers as add-ons.  I really encourage you as a project not enable
such features, as ultimately you'll probably start to rely on them, and then
you'll be effectively relying on proprietary software infrastructure to
develop your project.

I'm also curious: is there something you need now from the GitLab software
that self-hosting might improve?
-- 
Bradley M. Kuhn - he/him
Policy Fellow & Hacker-in-Residence at Software Freedom Conservancy
========================================================================
Become a Conservancy Supporter today: https://sfconservancy.org/supporter

Re: Should we apply for GitLab's open source program?

[Paolo Bonzini]

On 17/09/20 01:39, Bradley M. Kuhn wrote:
> One thing to note is that my understanding is that most of what you're
> getting access to through this program is proprietary software features that
> GitLab offers as add-ons.

Basically all we need is the increased access to the CI environment (not
just 400 minutes), and none of the add-on features.  Self-hosting would
of course help but we'd have to pay for the hardware resources to run
the CI, and have someone that can keep the hardware running.

Paolo

Re: Should we apply for GitLab's open source program?

[Alex Bennée]

Paolo Bonzini <bonzini@gnu.org> writes:

> On 17/09/20 01:39, Bradley M. Kuhn wrote:
>> One thing to note is that my understanding is that most of what you're
>> getting access to through this program is proprietary software features that
>> GitLab offers as add-ons.
>
> Basically all we need is the increased access to the CI environment (not
> just 400 minutes), and none of the add-on features.  Self-hosting would
> of course help but we'd have to pay for the hardware resources to run
> the CI, and have someone that can keep the hardware running.

It seems for the time being that public CI is still unlimited. The idea
of making our position as an FLOSS project "official" was to preempt any
changes to that might come down the track.

The question of using proprietary features hadn't come up beyond a
hand-waving of "ohh there is a long list". We are however thinking about
consolidating some of our more disparate infrastructure onto gitlab so
it's mostly in one place - for example the bug tracker currently hosted
on launchpad. Personally I'd think it's unlikely we want to move things
like the mailing lists which are currently on nongnu (via Savannah).

Ultimately as developers having to manage infrastructure is a bit of a
time-sink and currently it's hard for volunteer admins to be as
responsive as cloud-scale hosting companies who's income from non-free
software hosting pays for all our server time. If there was a free
software only instance of GitLab which offered the same level of service
I would personally be interested but I don't know how much of the
projects income could be diverted to supporting that versus the travel
bursaries and other such things we usually spend our money on.

In this regard FLOSS projects are both leaches on paid for services as
well as being useful public facing PR for a SaaS platforms abilities.

-- 
Alex Bennée

Re: Should we apply for GitLab's open source program?

[Daniel P. Berrangé]

On Thu, Sep 17, 2020 at 09:32:57AM +0100, Alex Bennée wrote:
> 
> Paolo Bonzini <bonzini@gnu.org> writes:
> 
> > On 17/09/20 01:39, Bradley M. Kuhn wrote:
> >> One thing to note is that my understanding is that most of what you're
> >> getting access to through this program is proprietary software features that
> >> GitLab offers as add-ons.
> >
> > Basically all we need is the increased access to the CI environment (not
> > just 400 minutes), and none of the add-on features.  Self-hosting would
> > of course help but we'd have to pay for the hardware resources to run
> > the CI, and have someone that can keep the hardware running.
> 
> It seems for the time being that public CI is still unlimited. The idea
> of making our position as an FLOSS project "official" was to preempt any
> changes to that might come down the track.
> 
> The question of using proprietary features hadn't come up beyond a
> hand-waving of "ohh there is a long list". We are however thinking about
> consolidating some of our more disparate infrastructure onto gitlab so
> it's mostly in one place - for example the bug tracker currently hosted
> on launchpad. Personally I'd think it's unlikely we want to move things
> like the mailing lists which are currently on nongnu (via Savannah).
> 
> Ultimately as developers having to manage infrastructure is a bit of a
> time-sink and currently it's hard for volunteer admins to be as
> responsive as cloud-scale hosting companies who's income from non-free
> software hosting pays for all our server time.

All the evidence we have is that developers generally do not do a good
job at maintaining infrastructure in their spare time. This is largely
unavoidable since a developer is always going to treat sysadmin tasks as
a part time thing to spend as little time as possible on, prioritizing
their coding work. So we end up either with unreliable services that
continuously break (look how many times has Patchew died from ENOSPC
this year), or the service happens to work reliably but is unmaintained
becoming increasingly out of date and thus vulnerable, or the service
simply ends up lagging behind state of the art offered by alternatives.

>                                                 If there was a free
> software only instance of GitLab which offered the same level of service
> I would personally be interested but I don't know how much of the
> projects income could be diverted to supporting that versus the travel
> bursaries and other such things we usually spend our money on.

Clearly the ideal situation would be 100% free software infrastructure
we can use at zero cost, while still being state of the art in terms
of services available to support our workflow. This doesn't exist, so
we have to figure out the most effective tradeoff to make that supports
QEMU's needs in an effective manner.

GitLab's open core model means we're at least partially using open
source infra, even if some features are closed. The basic GitLab CI
features are actually open source AFAIK, so we're not relying on the
closed source infra part of GitLab in that area. IOW, joining the
open source program there is simply about getting an increased grant
of free hardware resources to use.

This is certainly preferrable to us making more use of Travis or
Cirrus CI, or GitHub Actions, all of which are 100% closed CI
systems AFAIK.

Some projects have deployed their own GitLab instances (GNOME,
FreeDesktop), but that is not without significant challenges in
terms of deploying and maintaining the software as well as
providing sufficient hardware to go along. eg See the surprising
effect this self-hosting had on FreeDesktop costs:

  https://lists.freedesktop.org/archives/wayland-devel/2020-February/041232.html

It has been a long tedious road merely to bring up a small number
of CI runners on hardware sponsored by Red Hat, let alone get
enough hardware to replace everything that we and our contributors
currentl get for free via various public services. 

There's ultimately a big gap that exists in terms of a publically
host Git Forge that offers state of the art features, based on
100% open source infra. I'm not seeing anyone being able to solve
that in the forseeable future, and it doesn't seem like a sane use
of QEMU's limited contributor time to divert resources away from
development into infrastructure. We need to make a pragmatic
tradeoff, certainly favouring open source where possible, but if
we need to use other services too that's acceptable.

Our switch to increasingly use GitLab for CI is certainly an
improvement over our historical use of Travis, Shippable and
Cirrus, and better for our contributors than our reliance on
a handful of machines that only the QEMU Git maintainer can
access, or the patchew system that breaks multiple times a
year.

> In this regard FLOSS projects are both leaches on paid for services as
> well as being useful public facing PR for a SaaS platforms abilities.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Should we apply for GitLab's open source program?

[Daniel P. Berrangé]

On Fri, Sep 04, 2020 at 04:35:34PM +0100, Alex Bennée wrote:
> 
> Hi,
> 
> Given our growing reliance on GitLab and the recent announcement about
> free tier minutes:
> 
>   https://about.gitlab.com/pricing/faq-consumption-cicd/
> 
> is it time we officially apply for GitLab's Open Source Program:
> 
>   https://about.gitlab.com/solutions/open-source/program/
> 
> ?
> 
> From what I can see the QEMU project will easily qualify - the only
> minor inconvenience is needing to reapply every year. So far it seems as
> a public project their are no usage limits anyway. We are currently
> listed as 0 of 50,000 minutes:
> 
>   https://gitlab.com/groups/qemu-project/-/usage_quotas#pipelines-quota-tab
> 
> So we are in no pressing hurry to do this for the minutes but I suspect
> there may be other things that are made easier by having access to all
> the toys GitLab provides. Daniel has already posted to the forum
> requesting details about how this might affect our workflow so maybe we
> should just wait for feedback until pressing on?
> 
>   https://forum.gitlab.com/t/ci-cd-minutes-for-free-tier/40241/33

Fast forward 18 months... their timeframe for enforcing quotas has
slipped out as they better understood the need to finese plans to
minimise impact on OSS projects & their contributors. It is starting
to crystalize a little more now though.


One key thing that we've missed in previous discussion is that there is
a distinction between "wall clock CI minutes" and "billed CI minutes",
with the latter being calculated from the former using a "cost factor".

IIUC right now

   - public projects created before July 2021 have a cost factor of 0.
     That's QEMU & libvirt covered.

   - public projects created after July 2021 have a cost factor of 0.008

   - private projects have cost factor of 1

   - users and groups created in the past got a billed CI minutes
     allowance of 2,000, while new users/groups get 400. Can't recall
     the date of that change in allowance.

     For reasons I don't understand though, QEMU appears to have
     an allowance of 50,000 not 2,000. AFAIK that should only be
     possible on the Gold tier, which requires payment or membership
     of the OSS program.

NB this is all for Linux shared runners, macOS runners have a cost
factor much higher than 1. I'm ignoring that since we don't use them.


When this means is that if you have a user/group quota of '400' billed CI
minutes, and a cost factor of 0.008 for your project, you get 50,000
minutes of wall clock CI time.  The "low" 400 minute quota sounds a bit
less insane once you understand the cost factor impact.


For projects which are currently on a cost factor of 0 it is currently
impractical to determine your current CI usage from the Git Lab UI. The
"Usage Quotas" page shows "billed CI minutes" and so will always be
zero.

They implemented new UI to expose wallclock CI mintes ("Shared runners
duration" under the 'Analytics' page):

   https://gitlab.com/gitlab-org/gitlab/-/issues/340504

however that turned out to be inaccessible because the Analytics
pages at the group level are a premium feature. In response to me
pointing that out, they've got a new issue to track making this
available to all:

  https://gitlab.com/gitlab-org/gitlab/-/issues/353062

Once that happens we'll be able to get a clearer understanding of
our CI shared runner usage over time, and thus figure out the likely
impact on us before quotas become more strictly applied.



As mentioned above, QEMU currently get a cost fact of zero applied
to our project(s) as they have existed a long time. GitLab's proposed
next step in applying stricter quotas will be to change all public
projects to a cost factor of 0.008. With our (strangely high) 50,000
minute quota, that'll give 625,000 minutes shared runner time.
This should be more than enough for our pipelines per month.


Their longer term proposed plan is that all public projects will
get a cost factor of 1. That would reduce our wall clock time on
shared runners to 50,000 minutes which gives us on the order of
30 pipelines a month. Not enough for our needs during freeze times.


By that stage they expect any OSS projects which need more, to
have applied for membership of their Open Source Program. That
will retain a cost factor of 0.008 and give a quota of 50,000
billed CI minutes. ie 625,000 wall  clock minutes. That would
easily cover QEMU pipeline usage.


To minimise harm to contributors, public forks will also retain
the 0.008 cost factor. ie QEMU developers will still get either
50,000 or 250,000 CI minutes of wallclock time on shared runners
depending on how old their account is. That's quite alot but
we'll still want to be much more prudent in what we run to
minimize consumption for our contributors.  I'm fuzzy on whether
the "public forks" special case applies to any public fork, or
only forks of OSS Program projects, or only forks of projects
with a detected OSS license. None the less, QEMU contributors
will be OK if we join thue OSS program.

If you want to follow along, the following comment thread on their
public issue tracker gives the best up2date record of their proposals:

  https://gitlab.com/gitlab-org/gitlab/-/issues/243722#note_843079845

you'll notice I've been raising QEMU/libvirt's concerns in this
thread and issue more broadly. They appear quite genuine in wanting
to find a balance that doesn't impose too much pain on OSS projects
and their contributors, while still protecting their CI resource from
abuse.

In summary

 - QEMU will need to join the GitLab Open Source Program in the not
   too distant future. We can live with first change to a cost factor
   of 0.008, but we'll definitely not cope with the change after that
   to a cost factor of 1. There's no firm date for the latter, and we
   should get some warning ahead of time if we keep following the
   issue trackers and announcements.


 - We will need to change our pipelines rules to not run in user
   forks at all by default. We'll need a way for users to explicitly
   request a pipeline at time of their choosing instead. While they
   will have quite a lot of CI wallclock minutes, we will still need
   to make it easier to control how many jobs get launched, to
   preserve as many billed CI minutes as practical

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|