Using GitLab for OE/Yocto layers

Using GitLab for OE/Yocto layers

[Paul Barker]

Hey folks,

We had some initial discussion at OEDEM about the patch submission process and how repositories are hosted (see https://docs.google.com/document/d/1Qm1mJ6xLozWW9NgBnokue7TmhwgFP8b2HR8TUlaQwNs/edit#heading=h.o4qtddy8bh9i).

Patch submission via mailing list still has some key features (such as ability to compose a patch response offline) and I don't expect that the core OE/Yocto repositories will be changing their workflow in the short term. However, lots of tools and non-core layers already use a pull request model for patch submission, typically via GitHub or GitLab.

I'd like to propose that we set up a common place for OE/Yocto repositories which want to use this pull request model and start to collaborate on some of the tooling. As discussed in OEDEM, GitLab has a few major advantages over GitHub, BitBucket, etc here as they've got a track record of working well with open source projects and we can run the software on our own infrastructure if desired. We can use something like https://gitlab.com/openembedded at first to trial things and then maybe move to https://gitlab.openembedded.org (or gitlab.yoctoproject.org) if things look good.

I see the following advantages to this approach:

* Increased bus factor for repository administration - several layers currently have only one admin or live in someone's personal GitHub account, we risk losing access to these if the person in question disappears.

* Integrated issue tracking - we can standardise on a few issue labels and use milestones in GitLab to correspond to Yocto release branches. As the repositories would be in the same group we would be able to make use of group-level issue lists and boards to show issues across several layers.

* Integrated merge request tracking - Similar to issues, we would be able to see a combined group-level list of open merge requests. This is a good view to use to help out other layer maintainers as you can easily see new merge requests across all repositories and pick a few to review.

* Separation of patches to different repositories/branches - Where one mailing list takes patches for many repositories it's often confusing which repository or branch is the target for a given patch.

* Easier contribution from behind corporate firewalls or buggy email servers - contribution is via ssh/https.

* Subgroups - GitLab supports more than one layer of grouping so we can organise repositories even within the top level "openembedded" group.

* Ability to enable/disable merge requests per-repository - With GitHub you can't disable pull requests. If your repository takes patches by another method you have to manually watch out for pull requests and comment on each one to tell the contributor how to submit patches. With GitLab this isn't necessary as you can just turn off merge requests for that repository.

* Ability to standardise on automated testing - we can integrate patchtest and yocto-check-layer with GitLab CI and run these scripts on each merge request. This requires a bit of work in both test scripts to allow configuration as not all tests will be appropriate for all layers. However it's much easier than integrating these scripts with several incompatible repository hosts and CI systems.

* Ability to standardise on security - If we want to we can enforce two-factor authentication (2FA) for logins, make use of the merge request approvals feature in GitLab, etc. We don't need to do that initially but it may be useful in the future.

At the risk of bikeshedding I'd like to get some feedback on these ideas at this stage. Have I missed any advantages/disadvantages? Is anyone interested in helping to guinea pig this with a couple of layers/repositories to see how it works in practice?

-- 
Paul Barker

Re: [yocto] Using GitLab for OE/Yocto layers

[Adrian Bunk]

On Wed, Nov 06, 2019 at 04:01:03PM +0000, Paul Barker wrote:
>...
> At the risk of bikeshedding I'd like to get some feedback on these ideas at this stage. Have I missed any advantages/disadvantages?
>...

Three comments from me:


1. Patch review

Merge requests work well when there is one maintainer who reviews 
everything. For not regressing on the current level of review before
something hits master, merging a merge request into master-next
should then result in patch review emails sent to a list.
Or a setup where creation of a merge request automatically generates
review emails.

This is similar to all patches for stable branches now being sent for 
review to the mailing list a few days before they get merged into the 
stable branch, which has caught problematic patches due to more people 
reviewing them.


2. Maintaining an own GitLab instance

This was mentioned as an option. Expect upgrades to new GitLab releases 
once per month, which is work and as with all software never without 
regression risk.

Not a dealbreaker, but has to be resourced.


3. Long-term suistainability

Whatever the past track record of GitLab is, chances are the company 
behind it will sooner or later be bought by another company - and then 
anything can happen.

The code behind SourceForge was also at some point made available under
an open source licence, and forks being used in instances like Debian
Alioth ended up being unmaintainable dead ends long-term.

Berkeley DB would be an example where the company behind the software 
was bought by another company, and now there are plenty of CVEs that
are unfixable due to changed licencing.

Is there anyone capable and willing to continue open source maintainance 
of the GitLab open source sources if the company behind it would stop 
the open source releases tomorrow?

With projects like GNOME using GitLab the answer might be "yes",
but this should be evaluated before moving infrastructure to GitLab.


> Paul Barker

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Using GitLab for OE/Yocto layers

[Adrian Bunk]

On Wed, Nov 06, 2019 at 04:01:03PM +0000, Paul Barker wrote:
>...
> At the risk of bikeshedding I'd like to get some feedback on these ideas at this stage. Have I missed any advantages/disadvantages?
>...

Three comments from me:


1. Patch review

Merge requests work well when there is one maintainer who reviews 
everything. For not regressing on the current level of review before
something hits master, merging a merge request into master-next
should then result in patch review emails sent to a list.
Or a setup where creation of a merge request automatically generates
review emails.

This is similar to all patches for stable branches now being sent for 
review to the mailing list a few days before they get merged into the 
stable branch, which has caught problematic patches due to more people 
reviewing them.


2. Maintaining an own GitLab instance

This was mentioned as an option. Expect upgrades to new GitLab releases 
once per month, which is work and as with all software never without 
regression risk.

Not a dealbreaker, but has to be resourced.


3. Long-term suistainability

Whatever the past track record of GitLab is, chances are the company 
behind it will sooner or later be bought by another company - and then 
anything can happen.

The code behind SourceForge was also at some point made available under
an open source licence, and forks being used in instances like Debian
Alioth ended up being unmaintainable dead ends long-term.

Berkeley DB would be an example where the company behind the software 
was bought by another company, and now there are plenty of CVEs that
are unfixable due to changed licencing.

Is there anyone capable and willing to continue open source maintainance 
of the GitLab open source sources if the company behind it would stop 
the open source releases tomorrow?

With projects like GNOME using GitLab the answer might be "yes",
but this should be evaluated before moving infrastructure to GitLab.


> Paul Barker

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Using GitLab for OE/Yocto layers

[Ross Burton]

On 06/11/2019 21:41, Adrian Bunk wrote:
> 1. Patch review
> 
> Merge requests work well when there is one maintainer who reviews
> everything. For not regressing on the current level of review before
> something hits master, merging a merge request into master-next
> should then result in patch review emails sent to a list.
> Or a setup where creation of a merge request automatically generates
> review emails.
> 
> This is similar to all patches for stable branches now being sent for
> review to the mailing list a few days before they get merged into the
> stable branch, which has caught problematic patches due to more people
> reviewing them.

I've had discussions about this in the past too.  With merge requests it 
seems reasonable to automatically tag in relevant maintainers when the 
request is created to request review.  For example if oe-core was on 
GitLab then a MR touching GCC would automatically assign to Khem.

I agree that drive-by review is an important part for a core layer such 
as oe-core where a single maintainer can't know all of the details, so 
the ability to post at least new merge requests - and ideally all 
traffic - to a list for this is essential.

Ross

Re: [yocto] Using GitLab for OE/Yocto layers

[Paul Barker]

Following up on the email below, I've created a "community" subgroup to the OpenEmbedded group on GitLab: https://gitlab.com/openembedded/community. This should help to avoid any confusion with the repositories at the top level of https://gitlab.com/openembedded which mirror those at git.openembedded.org.

I'm going to use this for one or two layers I'm currently developing. If anyone else would like to setup a new layer here or transfer/mirror a repository from elsewhere then let me know and I'll get you set up.

On Wed, 6 Nov 2019, at 16:01, Paul Barker wrote:
> Hey folks,
> 
> We had some initial discussion at OEDEM about the patch submission 
> process and how repositories are hosted (see 
> https://docs.google.com/document/d/1Qm1mJ6xLozWW9NgBnokue7TmhwgFP8b2HR8TUlaQwNs/edit#heading=h.o4qtddy8bh9i).
> 
> Patch submission via mailing list still has some key features (such as 
> ability to compose a patch response offline) and I don't expect that 
> the core OE/Yocto repositories will be changing their workflow in the 
> short term. However, lots of tools and non-core layers already use a 
> pull request model for patch submission, typically via GitHub or GitLab.
> 
> I'd like to propose that we set up a common place for OE/Yocto 
> repositories which want to use this pull request model and start to 
> collaborate on some of the tooling. As discussed in OEDEM, GitLab has a 
> few major advantages over GitHub, BitBucket, etc here as they've got a 
> track record of working well with open source projects and we can run 
> the software on our own infrastructure if desired. We can use something 
> like https://gitlab.com/openembedded at first to trial things and then 
> maybe move to https://gitlab.openembedded.org (or 
> gitlab.yoctoproject.org) if things look good.
> 
> I see the following advantages to this approach:
> 
> * Increased bus factor for repository administration - several layers 
> currently have only one admin or live in someone's personal GitHub 
> account, we risk losing access to these if the person in question 
> disappears.
> 
> * Integrated issue tracking - we can standardise on a few issue labels 
> and use milestones in GitLab to correspond to Yocto release branches. 
> As the repositories would be in the same group we would be able to make 
> use of group-level issue lists and boards to show issues across several 
> layers.
> 
> * Integrated merge request tracking - Similar to issues, we would be 
> able to see a combined group-level list of open merge requests. This is 
> a good view to use to help out other layer maintainers as you can 
> easily see new merge requests across all repositories and pick a few to 
> review.
> 
> * Separation of patches to different repositories/branches - Where one 
> mailing list takes patches for many repositories it's often confusing 
> which repository or branch is the target for a given patch.
> 
> * Easier contribution from behind corporate firewalls or buggy email 
> servers - contribution is via ssh/https.
> 
> * Subgroups - GitLab supports more than one layer of grouping so we can 
> organise repositories even within the top level "openembedded" group.
> 
> * Ability to enable/disable merge requests per-repository - With GitHub 
> you can't disable pull requests. If your repository takes patches by 
> another method you have to manually watch out for pull requests and 
> comment on each one to tell the contributor how to submit patches. With 
> GitLab this isn't necessary as you can just turn off merge requests for 
> that repository.
> 
> * Ability to standardise on automated testing - we can integrate 
> patchtest and yocto-check-layer with GitLab CI and run these scripts on 
> each merge request. This requires a bit of work in both test scripts to 
> allow configuration as not all tests will be appropriate for all 
> layers. However it's much easier than integrating these scripts with 
> several incompatible repository hosts and CI systems.
> 
> * Ability to standardise on security - If we want to we can enforce 
> two-factor authentication (2FA) for logins, make use of the merge 
> request approvals feature in GitLab, etc. We don't need to do that 
> initially but it may be useful in the future.
> 
> At the risk of bikeshedding I'd like to get some feedback on these 
> ideas at this stage. Have I missed any advantages/disadvantages? Is 
> anyone interested in helping to guinea pig this with a couple of 
> layers/repositories to see how it works in practice?
> 
> -- 
> Paul Barker
> -- 
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

-- 
Paul Barker
Managing Director & Principal Engineer
Beta Five Ltd