OSUOSL in trouble

OSUOSL in trouble

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 1020 bytes --]

From LWN

> Lance Albertson writes that the Oregon State University Open Source Lab,
the home of many prominent free-software projects over the years, has run
into financial trouble:
>
> I am writing to inform you about a critical and time-sensitive situation
facing the Open Source Lab. Over the past several years, we have been
operating at a deficit due to a decline in corporate donations. While OSU's
College of Engineering (CoE) has generously filled this gap, recent changes
in university funding have led to a significant reduction in CoE's budget.
As a result, our current funding model is no longer sustainable and CoE
needs to find ways to cut programs.
> Earlier this week, I was informed that unless we secure $250,000 in
committed funds, the OSL will be forced to shut down later this year.

I propose migrating the two VMs we have to Digital Ocean and scrapping the
DO Kubernetes cluster. However that means we need to move CI to Amazon in
the next 1-2 months, before the Microsoft credits expire.

Paolo

[-- Attachment #2: Type: text/html, Size: 1264 bytes --]

Re: OSUOSL in trouble

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1306 bytes --]

On Thu, May 1, 2025, 01:11 Paolo Bonzini <pbonzini@redhat.com> wrote:

> From LWN
>
> > Lance Albertson writes that the Oregon State University Open Source Lab,
> the home of many prominent free-software projects over the years, has run
> into financial trouble:
> >
> > I am writing to inform you about a critical and time-sensitive situation
> facing the Open Source Lab. Over the past several years, we have been
> operating at a deficit due to a decline in corporate donations. While OSU's
> College of Engineering (CoE) has generously filled this gap, recent changes
> in university funding have led to a significant reduction in CoE's budget.
> As a result, our current funding model is no longer sustainable and CoE
> needs to find ways to cut programs.
> > Earlier this week, I was informed that unless we secure $250,000 in
> committed funds, the OSL will be forced to shut down later this year.
>
> I propose migrating the two VMs we have to Digital Ocean and scrapping the
> DO Kubernetes cluster. However that means we need to move CI to Amazon in
> the next 1-2 months, before the Microsoft credits expire.
>

I agree. I will also apply for the next round of Amazon funding.

Paolo, please let me know how I can help with migrating VMs or setting up a
CI Kubernetes cluster on Amazon.

Stefan

[-- Attachment #2: Type: text/html, Size: 1937 bytes --]

Re: OSUOSL in trouble

[Daniel P. Berrangé]

On Thu, May 01, 2025 at 11:43:28AM -0400, Stefan Hajnoczi wrote:
> On Thu, May 1, 2025, 01:11 Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> > From LWN
> >
> > > Lance Albertson writes that the Oregon State University Open Source Lab,
> > the home of many prominent free-software projects over the years, has run
> > into financial trouble:
> > >
> > > I am writing to inform you about a critical and time-sensitive situation
> > facing the Open Source Lab. Over the past several years, we have been
> > operating at a deficit due to a decline in corporate donations. While OSU's
> > College of Engineering (CoE) has generously filled this gap, recent changes
> > in university funding have led to a significant reduction in CoE's budget.
> > As a result, our current funding model is no longer sustainable and CoE
> > needs to find ways to cut programs.
> > > Earlier this week, I was informed that unless we secure $250,000 in
> > committed funds, the OSL will be forced to shut down later this year.
> >
> > I propose migrating the two VMs we have to Digital Ocean and scrapping the
> > DO Kubernetes cluster. However that means we need to move CI to Amazon in
> > the next 1-2 months, before the Microsoft credits expire.
> >
> 
> I agree. I will also apply for the next round of Amazon funding.
> 
> Paolo, please let me know how I can help with migrating VMs or setting up a
> CI Kubernetes cluster on Amazon.

Please ensure the any new CI machines for gitlab have persistent cache
enabled. Our functional test jobs are still downloading images on every
run due to lack of configured cache, which makes them unreliable when
sites have transient outages, as well as making jobs slower.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: OSUOSL in trouble

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 807 bytes --]

Il gio 1 mag 2025, 17:51 Daniel P. Berrangé <berrange@redhat.com> ha
scritto:

> > Paolo, please let me know how I can help with migrating VMs or setting
> up a
> > CI Kubernetes cluster on Amazon.
>

Since in the past Camilla did it but I am not sure if she has bandwidth,
let's run it with Red Hat first. All I did was give her admin privileges on
Azure.

In the meanwhile I will start migrating the VMs.

Please ensure the any new CI machines for gitlab have persistent cache
> enabled. Our functional test jobs are still downloading images on every
> run due to lack of configured cache, which makes them unreliable when
> sites have transient outages, as well as making jobs slower.
>

Thanks for the suggestion—just to be clear do the Azure runners have that?

Paolo

>
>

[-- Attachment #2: Type: text/html, Size: 1759 bytes --]

Re: OSUOSL in trouble

[Daniel P. Berrangé]

On Thu, May 01, 2025 at 06:20:50PM +0200, Paolo Bonzini wrote:
> Il gio 1 mag 2025, 17:51 Daniel P. Berrangé <berrange@redhat.com> ha
> scritto:
> 
> > > Paolo, please let me know how I can help with migrating VMs or setting
> > up a
> > > CI Kubernetes cluster on Amazon.
> >
> 
> Since in the past Camilla did it but I am not sure if she has bandwidth,
> let's run it with Red Hat first. All I did was give her admin privileges on
> Azure.
> 
> In the meanwhile I will start migrating the VMs.
> 
> Please ensure the any new CI machines for gitlab have persistent cache
> > enabled. Our functional test jobs are still downloading images on every
> > run due to lack of configured cache, which makes them unreliable when
> > sites have transient outages, as well as making jobs slower.
> >
> 
> Thanks for the suggestion—just to be clear do the Azure runners have that?

I'm unclear - what I can say is that whatever runners are currently
being used always appear to start with an empty cache, as in the log:

https://gitlab.com/qemu-project/qemu/-/jobs/9853814057

We see countless messages

  Downloading http://ftpmirror.your.org/pub/misc/ftp.software.ibm.com/rs6000/firmware/7020-40p/P12H0456.IMG to /builds/qemu-project/qemu/functional-cache/download/d957f79c73f760d1455d2286fcd901ed6d06167320eb73511b478a939be25b3f...

and in both the restore/save of cache gitlab reports:

   "No URL provided, cache will not be uploaded to shared cache server.
    Cache will be stored only locally. "

which makes me think, if there is a cache, then it is local to the
build env and getting thrown away at the end of the job

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: OSUOSL in trouble

[Stefan Hajnoczi]

On Thu, May 1, 2025 at 12:31 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Thu, May 01, 2025 at 06:20:50PM +0200, Paolo Bonzini wrote:
> > Il gio 1 mag 2025, 17:51 Daniel P. Berrangé <berrange@redhat.com> ha
> > scritto:
> >
> > > > Paolo, please let me know how I can help with migrating VMs or setting
> > > up a
> > > > CI Kubernetes cluster on Amazon.
> > >
> >
> > Since in the past Camilla did it but I am not sure if she has bandwidth,
> > let's run it with Red Hat first. All I did was give her admin privileges on
> > Azure.
> >
> > In the meanwhile I will start migrating the VMs.
> >
> > Please ensure the any new CI machines for gitlab have persistent cache
> > > enabled. Our functional test jobs are still downloading images on every
> > > run due to lack of configured cache, which makes them unreliable when
> > > sites have transient outages, as well as making jobs slower.
> > >
> >
> > Thanks for the suggestion—just to be clear do the Azure runners have that?
>
> I'm unclear - what I can say is that whatever runners are currently
> being used always appear to start with an empty cache, as in the log:
>
> https://gitlab.com/qemu-project/qemu/-/jobs/9853814057
>
> We see countless messages
>
>   Downloading http://ftpmirror.your.org/pub/misc/ftp.software.ibm.com/rs6000/firmware/7020-40p/P12H0456.IMG to /builds/qemu-project/qemu/functional-cache/download/d957f79c73f760d1455d2286fcd901ed6d06167320eb73511b478a939be25b3f...
>
> and in both the restore/save of cache gitlab reports:
>
>    "No URL provided, cache will not be uploaded to shared cache server.
>     Cache will be stored only locally. "
>
> which makes me think, if there is a cache, then it is local to the
> build env and getting thrown away at the end of the job

AFAIK gitlab-runner supports two types of caching: object storage (S3)
or a file system mount.

When the new cluster is created in AWS we will have the ability to try
S3 object storage. I think that is easier to manage than ReadWriteMany
file system volumes in Kubernetes, since that would require NFS or
another solution.

Stefan