[igt-dev] RFC: Migration to Gitlab

[igt-dev] RFC: Migration to Gitlab

[Daniel Vetter]

Hi all,

I think it's time to brainstorm a bit about the gitlab migration. Basic reasons:

- fd.o admins want to deprecate shell accounts and hand-rolled
infrastructure, because it's a pain to keep secure&updated.

- gitlab will allow us to add committers on our own, greatly
simplifying that process (and offloading that task from fd.o admins).

There's also some more benefits we might want to reap, like better CI
integration for basic build testing - no more "oops didn't build
drm-misc defconfigs" or "sry, forgot make check in maintainer-tools".
But that's all fully optional.

For the full in-depth writeup of everything, see

https://www.fooishbar.org/blog/gitlab-fdo-introduction/

I think now is also a good time, with mesa, xorg, wayland/weston and
others moved, to start thinking about how we'll move drm. There's a
few things to figure out though:

- We probably want to split out maintainer-tools. That would address
the concern that there's 50+ committers to an auto-updating shell
script ...

- We need to figure out how to handle the ACL trickery around drm-tip in gitlab.

- Probably good to stage the migration, with maintainer-tools, igt
leading. That will also make fd.o admins happy, who want to rework
their cloud infrastructure a bit before migrating the big kernel repos
over.

- Figuring out the actual migration - we've been adding a pile of
committers since fd.o LDAP was converted to gitlab once back in
spring. We need to at least figure out how to move the new
accounts/committers.

- Similar, maintainer-tools needs to move. We probably want to move
all the dim maintained kernel repos in one go, to avoid headaches with
double-accounts needed for committers.

- CI, linux-next and everyone else should be fine, since the
cgit/non-ssh paths will keep working (they'll be read-only mirrors).
Need to double-check that with everyone.

- Some organization structure would be good.

https://cgit.freedesktop.org/drm

libdrm won't be part of the gitlab drm group because that's already
moved under mesa (and you can't symlink/mulit-home anymore on gitlab):

https://gitlab.freedesktop.org/mesa/drm

But there's also drm_hwcomposer, which we might want to migrate into
drm too - gitlab requires a containing group, and
drm_hwcomposer/drm_hwcomposer is a bit silly.

Note: Access rights can be done at any level in the hierarchy, the
organization is orthogonal to commit rights.

- Anything else I've forgotten.

A lot of this still needs to be figured out first. As a first step I'm
looking for volunteers who want to join the fun, besides comments and
thoughts on the overall topic of course.

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [Intel-gfx] RFC: Migration to Gitlab

[Sean Paul]

On Wed, Aug 22, 2018 at 01:44:56PM +0200, Daniel Vetter wrote:
> Hi all,
> 
> I think it's time to brainstorm a bit about the gitlab migration. Basic reasons:
> 
> - fd.o admins want to deprecate shell accounts and hand-rolled
> infrastructure, because it's a pain to keep secure&updated.
> 
> - gitlab will allow us to add committers on our own, greatly
> simplifying that process (and offloading that task from fd.o admins).
> 
> There's also some more benefits we might want to reap, like better CI
> integration for basic build testing - no more "oops didn't build
> drm-misc defconfigs" or "sry, forgot make check in maintainer-tools".
> But that's all fully optional.
> 
> For the full in-depth writeup of everything, see
> 
> https://www.fooishbar.org/blog/gitlab-fdo-introduction/
> 
> I think now is also a good time, with mesa, xorg, wayland/weston and
> others moved, to start thinking about how we'll move drm. There's a
> few things to figure out though:
> 
> - We probably want to split out maintainer-tools. That would address
> the concern that there's 50+ committers to an auto-updating shell
> script ...
> 

/me laughs nervously

> - We need to figure out how to handle the ACL trickery around drm-tip in gitlab.
> 
> - Probably good to stage the migration, with maintainer-tools, igt
> leading. That will also make fd.o admins happy, who want to rework
> their cloud infrastructure a bit before migrating the big kernel repos
> over.
> 
> - Figuring out the actual migration - we've been adding a pile of
> committers since fd.o LDAP was converted to gitlab once back in
> spring. We need to at least figure out how to move the new
> accounts/committers.
> 
> - Similar, maintainer-tools needs to move. We probably want to move
> all the dim maintained kernel repos in one go, to avoid headaches with
> double-accounts needed for committers.
> 
> - CI, linux-next and everyone else should be fine, since the
> cgit/non-ssh paths will keep working (they'll be read-only mirrors).
> Need to double-check that with everyone.

They can also pull the trees from git://gitlab.fd.o/blah as normal, just need to
give them new pointers once we're stable.

> 
> - Some organization structure would be good.
> 
> https://cgit.freedesktop.org/drm
> 
> libdrm won't be part of the gitlab drm group because that's already
> moved under mesa (and you can't symlink/mulit-home anymore on gitlab):
> 
> https://gitlab.freedesktop.org/mesa/drm
> 
> But there's also drm_hwcomposer, which we might want to migrate into
> drm too - gitlab requires a containing group, and
> drm_hwcomposer/drm_hwcomposer is a bit silly.

This seems fine to me. Our expansion plans likely aren't big enough to warrant a
separate group.

> 
> Note: Access rights can be done at any level in the hierarchy, the
> organization is orthogonal to commit rights.
> 
> - Anything else I've forgotten.
> 
> A lot of this still needs to be figured out first. As a first step I'm
> looking for volunteers who want to join the fun, besides comments and
> thoughts on the overall topic of course.

I'm pretty keen on getting this done, so I'll volunteer some cycles if there's
something that needs doing.

Sean

> 
> Cheers, Daniel
> -- 
> Daniel Vetter
> Software Engineer, Intel Corporation
> +41 (0) 79 365 57 48 - http://blog.ffwll.ch
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/intel-gfx

-- 
Sean Paul, Software Engineer, Google / Chromium OS
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [Intel-gfx] [igt-dev] RFC: Migration to Gitlab

[Jani Nikula]

On Wed, 22 Aug 2018, Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
> Hi all,
>
> I think it's time to brainstorm a bit about the gitlab migration. Basic reasons:
>
> - fd.o admins want to deprecate shell accounts and hand-rolled
> infrastructure, because it's a pain to keep secure&updated.
>
> - gitlab will allow us to add committers on our own, greatly
> simplifying that process (and offloading that task from fd.o admins).
>
> There's also some more benefits we might want to reap, like better CI
> integration for basic build testing - no more "oops didn't build
> drm-misc defconfigs" or "sry, forgot make check in maintainer-tools".
> But that's all fully optional.
>
> For the full in-depth writeup of everything, see
>
> https://www.fooishbar.org/blog/gitlab-fdo-introduction/
>
> I think now is also a good time, with mesa, xorg, wayland/weston and
> others moved, to start thinking about how we'll move drm. There's a
> few things to figure out though:
>
> - We probably want to split out maintainer-tools. That would address
> the concern that there's 50+ committers to an auto-updating shell
> script ...
>
> - We need to figure out how to handle the ACL trickery around drm-tip in gitlab.
>
> - Probably good to stage the migration, with maintainer-tools, igt
> leading. That will also make fd.o admins happy, who want to rework
> their cloud infrastructure a bit before migrating the big kernel repos
> over.
>
> - Figuring out the actual migration - we've been adding a pile of
> committers since fd.o LDAP was converted to gitlab once back in
> spring. We need to at least figure out how to move the new
> accounts/committers.
>
> - Similar, maintainer-tools needs to move. We probably want to move
> all the dim maintained kernel repos in one go, to avoid headaches with
> double-accounts needed for committers.
>
> - CI, linux-next and everyone else should be fine, since the
> cgit/non-ssh paths will keep working (they'll be read-only mirrors).
> Need to double-check that with everyone.
>
> - Some organization structure would be good.
>
> https://cgit.freedesktop.org/drm
>
> libdrm won't be part of the gitlab drm group because that's already
> moved under mesa (and you can't symlink/mulit-home anymore on gitlab):
>
> https://gitlab.freedesktop.org/mesa/drm
>
> But there's also drm_hwcomposer, which we might want to migrate into
> drm too - gitlab requires a containing group, and
> drm_hwcomposer/drm_hwcomposer is a bit silly.
>
> Note: Access rights can be done at any level in the hierarchy, the
> organization is orthogonal to commit rights.
>
> - Anything else I've forgotten.
>
> A lot of this still needs to be figured out first. As a first step I'm
> looking for volunteers who want to join the fun, besides comments and
> thoughts on the overall topic of course.

Just a couple of concerns from drm/i915 perspective for starters:

- Patchwork integration. I think we'll want to keep patchwork for at
  least intel-gfx etc. for the time being. IIUC the one thing we need is
  some server side hook to update patchwork on git push.

- Sticking to fdo bugzilla and disabling gitlab issues for at least
  drm-intel for the time being. Doing that migration in the same go is a
  bit much I think. Reassignment across bugzilla and gitlab will be an
  issue.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Graphics Center
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [igt-dev] RFC: Migration to Gitlab

[Adam Jackson]

On Wed, 2018-08-22 at 16:13 +0300, Jani Nikula wrote:

> - Sticking to fdo bugzilla and disabling gitlab issues for at least
>   drm-intel for the time being. Doing that migration in the same go is a
>   bit much I think. Reassignment across bugzilla and gitlab will be an
>   issue.

Can you elaborate a bit on the issues here? The actual move-the-bugs
process has been pretty painless for the parts of xorg we've done so
far.

- ajax
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] RFC: Migration to Gitlab

[Rodrigo Vivi]

On Wed, Aug 22, 2018 at 10:19:19AM -0400, Adam Jackson wrote:
> On Wed, 2018-08-22 at 16:13 +0300, Jani Nikula wrote:
> 
> > - Sticking to fdo bugzilla and disabling gitlab issues for at least
> >   drm-intel for the time being. Doing that migration in the same go is a
> >   bit much I think. Reassignment across bugzilla and gitlab will be an
> >   issue.
> 
> Can you elaborate a bit on the issues here? The actual move-the-bugs
> process has been pretty painless for the parts of xorg we've done so
> far.

I guess there is nothing against moving the bugs there. The concern is only on
doing everything at once.

I'm in favor of moving gits for now and after we are confident that
everything is there and working we move the bugs.

One question about the bugzilla:

Will all the referrences on all commit messages get outdated after
bugzilla is dead?
Or bugzilla will stay up for referrence but closed for interaction?
or all old closed stuff are always moved and bugzilla.fd.o as well and
bugzilla.fd.o will be mirroring gitlab?

Thanks,
Rodrigo.

> 
> - ajax
> _______________________________________________
> dim-tools mailing list
> dim-tools@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dim-tools
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] RFC: Migration to Gitlab

[Daniel Stone]

Hi Rodrigo,

On Wed, 22 Aug 2018 at 17:06, Rodrigo Vivi <rodrigo.vivi@intel.com> wrote:
> On Wed, Aug 22, 2018 at 10:19:19AM -0400, Adam Jackson wrote:
> > On Wed, 2018-08-22 at 16:13 +0300, Jani Nikula wrote:
> > > - Sticking to fdo bugzilla and disabling gitlab issues for at least
> > >   drm-intel for the time being. Doing that migration in the same go is a
> > >   bit much I think. Reassignment across bugzilla and gitlab will be an
> > >   issue.
> >
> > Can you elaborate a bit on the issues here? The actual move-the-bugs
> > process has been pretty painless for the parts of xorg we've done so
> > far.
>
> I guess there is nothing against moving the bugs there. The concern is only on
> doing everything at once.
>
> I'm in favor of moving gits for now and after we are confident that
> everything is there and working we move the bugs.

As Daniel alluded to, the only issue I really have is moving _all_ the
kernel repos at once. At the end of the year we'll have easy automatic
scaling thanks to the independent services being separated. As it is,
all the GitLab services (apart from CI runners) run on a single
machine, so we have limited options if it becomes overwhelmed with
load.

Do you have a particular concern about the repos? e.g. what would you
check for to make sure things are 'there and working'?

> One question about the bugzilla:
>
> Will all the referrences on all commit messages get outdated after
> bugzilla is dead?
> Or bugzilla will stay up for referrence but closed for interaction?
> or all old closed stuff are always moved and bugzilla.fd.o as well and
> bugzilla.fd.o will be mirroring gitlab?

When bugs are migrated from Bugzilla to GitLab, only open bugs are
migrated. Closed ones are left in place, as is; open ones have a
comment at the end saying that the bug has moved to GitLab, a URL
linking to the new GitLab issue, and telling them to please chase it
up there.

Even when we move everyone completely off Bugzilla, we will keep it as
a read-only mirror forever. Even with Phabricator, which very few
people ever used, has had all its bugs and code review captured and
archived, so we can continue to preserve all the old content and
links, without having to run the actual service.

Cheers,
Daniel
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] RFC: Migration to Gitlab

[Rodrigo Vivi]

On Wed, Aug 22, 2018 at 05:37:22PM +0100, Daniel Stone wrote:
> Hi Rodrigo,
> 
> On Wed, 22 Aug 2018 at 17:06, Rodrigo Vivi <rodrigo.vivi@intel.com> wrote:
> > On Wed, Aug 22, 2018 at 10:19:19AM -0400, Adam Jackson wrote:
> > > On Wed, 2018-08-22 at 16:13 +0300, Jani Nikula wrote:
> > > > - Sticking to fdo bugzilla and disabling gitlab issues for at least
> > > >   drm-intel for the time being. Doing that migration in the same go is a
> > > >   bit much I think. Reassignment across bugzilla and gitlab will be an
> > > >   issue.
> > >
> > > Can you elaborate a bit on the issues here? The actual move-the-bugs
> > > process has been pretty painless for the parts of xorg we've done so
> > > far.
> >
> > I guess there is nothing against moving the bugs there. The concern is only on
> > doing everything at once.
> >
> > I'm in favor of moving gits for now and after we are confident that
> > everything is there and working we move the bugs.
> 
> As Daniel alluded to, the only issue I really have is moving _all_ the
> kernel repos at once. At the end of the year we'll have easy automatic
> scaling thanks to the independent services being separated. As it is,
> all the GitLab services (apart from CI runners) run on a single
> machine, so we have limited options if it becomes overwhelmed with
> load.
> 
> Do you have a particular concern about the repos?

no concerns from my side about the repos itself. From my committer
perspective on libdrm, mesa the migration was really smooth.

> e.g. what would you
> check for to make sure things are 'there and working'?

more in terms of other committers getting used to it, dim working
for most of committers, links in documentations and wikis updated...

but no concerns with the infra itself.

> 
> > One question about the bugzilla:
> >
> > Will all the referrences on all commit messages get outdated after
> > bugzilla is dead?
> > Or bugzilla will stay up for referrence but closed for interaction?
> > or all old closed stuff are always moved and bugzilla.fd.o as well and
> > bugzilla.fd.o will be mirroring gitlab?
> 
> When bugs are migrated from Bugzilla to GitLab, only open bugs are
> migrated. Closed ones are left in place, as is; open ones have a
> comment at the end saying that the bug has moved to GitLab, a URL
> linking to the new GitLab issue, and telling them to please chase it
> up there.
> 
> Even when we move everyone completely off Bugzilla, we will keep it as
> a read-only mirror forever. Even with Phabricator, which very few
> people ever used, has had all its bugs and code review captured and
> archived, so we can continue to preserve all the old content and
> links, without having to run the actual service.

Great!
Thanks for all clarification,
Rodrigo.

> 
> Cheers,
> Daniel
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] RFC: Migration to Gitlab

[Jani Nikula]

On Wed, 22 Aug 2018, Rodrigo Vivi <rodrigo.vivi@intel.com> wrote:
> On Wed, Aug 22, 2018 at 10:19:19AM -0400, Adam Jackson wrote:
>> On Wed, 2018-08-22 at 16:13 +0300, Jani Nikula wrote:
>> 
>> > - Sticking to fdo bugzilla and disabling gitlab issues for at least
>> >   drm-intel for the time being. Doing that migration in the same go is a
>> >   bit much I think. Reassignment across bugzilla and gitlab will be an
>> >   issue.
>> 
>> Can you elaborate a bit on the issues here? The actual move-the-bugs
>> process has been pretty painless for the parts of xorg we've done so
>> far.
>
> I guess there is nothing against moving the bugs there. The concern is only on
> doing everything at once.

No, it's not just that.

We have some automation using the bugzilla APIs directly, and
someone(tm) needs to figure out how this should work with gitlab. Maybe
we have a better chance of doing things with gitlab's APIs, maybe we can
reduce our dependence on external logic altogether.

We have special "i915 platform" and "i915 features" fields in
bugzilla. We use a mailing list default assignee. Some of us use the
"whiteboard" and "keywords" fields. Etc.

I don't think figuring all this out is rocket science, but someone needs
to actually do it, and get our workflows straightened out *before* we
flip the switch. I'm just trying to figure out if that is a blocker to
migrating the repos.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Graphics Center
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [igt-dev] RFC: Migration to Gitlab

[Daniel Vetter]

On Fri, Aug 24, 2018 at 8:52 AM, Jani Nikula
<jani.nikula@linux.intel.com> wrote:
> On Wed, 22 Aug 2018, Rodrigo Vivi <rodrigo.vivi@intel.com> wrote:
>> On Wed, Aug 22, 2018 at 10:19:19AM -0400, Adam Jackson wrote:
>>> On Wed, 2018-08-22 at 16:13 +0300, Jani Nikula wrote:
>>>
>>> > - Sticking to fdo bugzilla and disabling gitlab issues for at least
>>> >   drm-intel for the time being. Doing that migration in the same go is a
>>> >   bit much I think. Reassignment across bugzilla and gitlab will be an
>>> >   issue.
>>>
>>> Can you elaborate a bit on the issues here? The actual move-the-bugs
>>> process has been pretty painless for the parts of xorg we've done so
>>> far.
>>
>> I guess there is nothing against moving the bugs there. The concern is only on
>> doing everything at once.
>
> No, it's not just that.
>
> We have some automation using the bugzilla APIs directly, and
> someone(tm) needs to figure out how this should work with gitlab. Maybe
> we have a better chance of doing things with gitlab's APIs, maybe we can
> reduce our dependence on external logic altogether.
>
> We have special "i915 platform" and "i915 features" fields in
> bugzilla. We use a mailing list default assignee. Some of us use the
> "whiteboard" and "keywords" fields. Etc.
>
> I don't think figuring all this out is rocket science, but someone needs
> to actually do it, and get our workflows straightened out *before* we
> flip the switch. I'm just trying to figure out if that is a blocker to
> migrating the repos.

I think the mesa approach of flat-out disabling gitlab issues and
merge request is solid. That way there's no confusions with
contributions and bug reports stranded in the wrong place. This should
allow us to handle all the different feature sets gitlab provides
independently and as we see fit: Repo hosting, documentation/artifact
hosting, CI integration, patch submissions, issue tracking. We can
choose&pick what we think is good, and ignore everything else.

And of course if 1-2 years down the road things change, we can
reevaluate. I think for a rough timeline if things go well we'll have
igt and maintainer-tools migrated by end of year (just the repos,
nothing else), and a solid plan for migrating all the drm* repos next
year. Everything else is too far away to have solid plans for, or even
just a timeline.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] RFC: Migration to Gitlab

[Daniel Vetter]

On Wed, Aug 22, 2018 at 3:13 PM, Jani Nikula
<jani.nikula@linux.intel.com> wrote:
> On Wed, 22 Aug 2018, Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
>> Hi all,
>>
>> I think it's time to brainstorm a bit about the gitlab migration. Basic reasons:
>>
>> - fd.o admins want to deprecate shell accounts and hand-rolled
>> infrastructure, because it's a pain to keep secure&updated.
>>
>> - gitlab will allow us to add committers on our own, greatly
>> simplifying that process (and offloading that task from fd.o admins).
>>
>> There's also some more benefits we might want to reap, like better CI
>> integration for basic build testing - no more "oops didn't build
>> drm-misc defconfigs" or "sry, forgot make check in maintainer-tools".
>> But that's all fully optional.
>>
>> For the full in-depth writeup of everything, see
>>
>> https://www.fooishbar.org/blog/gitlab-fdo-introduction/
>>
>> I think now is also a good time, with mesa, xorg, wayland/weston and
>> others moved, to start thinking about how we'll move drm. There's a
>> few things to figure out though:
>>
>> - We probably want to split out maintainer-tools. That would address
>> the concern that there's 50+ committers to an auto-updating shell
>> script ...
>>
>> - We need to figure out how to handle the ACL trickery around drm-tip in gitlab.
>>
>> - Probably good to stage the migration, with maintainer-tools, igt
>> leading. That will also make fd.o admins happy, who want to rework
>> their cloud infrastructure a bit before migrating the big kernel repos
>> over.
>>
>> - Figuring out the actual migration - we've been adding a pile of
>> committers since fd.o LDAP was converted to gitlab once back in
>> spring. We need to at least figure out how to move the new
>> accounts/committers.
>>
>> - Similar, maintainer-tools needs to move. We probably want to move
>> all the dim maintained kernel repos in one go, to avoid headaches with
>> double-accounts needed for committers.
>>
>> - CI, linux-next and everyone else should be fine, since the
>> cgit/non-ssh paths will keep working (they'll be read-only mirrors).
>> Need to double-check that with everyone.
>>
>> - Some organization structure would be good.
>>
>> https://cgit.freedesktop.org/drm
>>
>> libdrm won't be part of the gitlab drm group because that's already
>> moved under mesa (and you can't symlink/mulit-home anymore on gitlab):
>>
>> https://gitlab.freedesktop.org/mesa/drm
>>
>> But there's also drm_hwcomposer, which we might want to migrate into
>> drm too - gitlab requires a containing group, and
>> drm_hwcomposer/drm_hwcomposer is a bit silly.
>>
>> Note: Access rights can be done at any level in the hierarchy, the
>> organization is orthogonal to commit rights.
>>
>> - Anything else I've forgotten.
>>
>> A lot of this still needs to be figured out first. As a first step I'm
>> looking for volunteers who want to join the fun, besides comments and
>> thoughts on the overall topic of course.
>
> Just a couple of concerns from drm/i915 perspective for starters:
>
> - Patchwork integration. I think we'll want to keep patchwork for at
>   least intel-gfx etc. for the time being. IIUC the one thing we need is
>   some server side hook to update patchwork on git push.
>
> - Sticking to fdo bugzilla and disabling gitlab issues for at least
>   drm-intel for the time being. Doing that migration in the same go is a
>   bit much I think. Reassignment across bugzilla and gitlab will be an
>   issue.

Good points, forgot about both. Patchwork reading the mailing list
should keep working as-is, but the update hook needs looking into.

Disabling gitlab issues is a non-brainer, same with merge requests.
Mesa is already doing that. For bugs I think it's best to entirely
leave them out for now, and maybe reconsider when/if mesa has moved.
Before that I don't think gitlab issues make any sense at all.

For merge requests I think best approach is to enable them very
selectively at first for testing out, and then making a per-subproject
decision whether they make sense. E.g. I think for maintainer-tools
integrating make check and the doc building into gitlab CI would be
sweet, and worth looking into gitlab merge requests just to automate
that. Again best left out of scope for the initial migration.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [igt-dev] RFC: Migration to Gitlab

[Daniel Stone]

Hi,

On Wed, 22 Aug 2018 at 15:44, Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
> On Wed, Aug 22, 2018 at 3:13 PM, Jani Nikula <jani.nikula@linux.intel.com> wrote:
> > Just a couple of concerns from drm/i915 perspective for starters:
> >
> > - Patchwork integration. I think we'll want to keep patchwork for at
> >   least intel-gfx etc. for the time being. IIUC the one thing we need is
> >   some server side hook to update patchwork on git push.
> >
> > - Sticking to fdo bugzilla and disabling gitlab issues for at least
> >   drm-intel for the time being. Doing that migration in the same go is a
> >   bit much I think. Reassignment across bugzilla and gitlab will be an
> >   issue.
>
> Good points, forgot about both. Patchwork reading the mailing list
> should keep working as-is, but the update hook needs looking into.

All the hooks are retained. gitlab.fd.o pushes to git.fd.o, and
git.fd.o still executes all the old hooks. This includes Patchwork,
readthedocs, AppVeyor, and whatever else.

> For merge requests I think best approach is to enable them very
> selectively at first for testing out, and then making a per-subproject
> decision whether they make sense. E.g. I think for maintainer-tools
> integrating make check and the doc building into gitlab CI would be
> sweet, and worth looking into gitlab merge requests just to automate
> that. Again best left out of scope for the initial migration.

You don't need MRs to do that. You can build a .gitlab-ci.yml file
which will run make check or build the docs or whatever, and have that
run on pushes. Anyone can then fork the repo, push their changes to
that fork, and see the CI results from there. It's like Travis: the CI
configuration is a (mutable) part of the codebase, not a separate
'thing' hanging off a specific repo. So if you write the CI pipeline
first, you can have people running CI on push completely independently
of switching the workflow to use MRs.

Cheers,
Daniel
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] RFC: Migration to Gitlab

[Emil Velikov]

Hi Dan,

On 22 August 2018 at 12:44, Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
> Hi all,
>
> I think it's time to brainstorm a bit about the gitlab migration. Basic reasons:
>
> - fd.o admins want to deprecate shell accounts and hand-rolled
> infrastructure, because it's a pain to keep secure&updated.
>
> - gitlab will allow us to add committers on our own, greatly
> simplifying that process (and offloading that task from fd.o admins).
>
Random thought - I really wish the admins spoke early and louder about issues.
From infra to manpower and adhoc tool maintenance.

> There's also some more benefits we might want to reap, like better CI
> integration for basic build testing - no more "oops didn't build
> drm-misc defconfigs" or "sry, forgot make check in maintainer-tools".
> But that's all fully optional.
>
> For the full in-depth writeup of everything, see
>
> https://www.fooishbar.org/blog/gitlab-fdo-introduction/
>
> I think now is also a good time, with mesa, xorg, wayland/weston and
> others moved, to start thinking about how we'll move drm. There's a
> few things to figure out though:
>
> - We probably want to split out maintainer-tools. That would address
> the concern that there's 50+ committers to an auto-updating shell
> script ...
>
> - We need to figure out how to handle the ACL trickery around drm-tip in gitlab.
>
> - Probably good to stage the migration, with maintainer-tools, igt
> leading. That will also make fd.o admins happy, who want to rework
> their cloud infrastructure a bit before migrating the big kernel repos
> over.
>
> - Figuring out the actual migration - we've been adding a pile of
> committers since fd.o LDAP was converted to gitlab once back in
> spring. We need to at least figure out how to move the new
> accounts/committers.
>
As a observer, allow me to put some ideas. You've mostly covered them
all, my emphasis is to seriously stick with _one_ thing at a time.
Attempting to do multiple things in parallel will end up with
sub-optimal results.

 - (at random point) cleanup the committers list - people who have not
contributed in the last 1 year?
 - setup drm group, copy/migrate accounts - one could even reuse the
existing credentials
 - move git repos to gitlab, the push URL change, cgit mirror
preserves the normal fetch ones as well as PW hooks
 - work out how new accounts are handled - still in bugzilla, otherwise

At this stage only workflow change is a) once-off account setup and b)
pushURL update
As a follow-up one can setup anything fancy.
 - migrate PW/other hooks
 - migrate bugs - if applicable
 - add new hooks - DRM docs, other
 - etc


> - Similar, maintainer-tools needs to move. We probably want to move
> all the dim maintained kernel repos in one go, to avoid headaches with
> double-accounts needed for committers.
>
One should be able to create a separate repo for these. And then either:
 - one by one add the required features into the gitlab MR machinery,
 - or, wire the execution in pre/post merge stage.

IIRC there are some upstream requests about the former.

> - CI, linux-next and everyone else should be fine, since the
> cgit/non-ssh paths will keep working (they'll be read-only mirrors).
> Need to double-check that with everyone.
>
> - Some organization structure would be good.
>
> https://cgit.freedesktop.org/drm
>
> libdrm won't be part of the gitlab drm group because that's already
> moved under mesa (and you can't symlink/mulit-home anymore on gitlab):
>
> https://gitlab.freedesktop.org/mesa/drm
>
> But there's also drm_hwcomposer, which we might want to migrate into
> drm too - gitlab requires a containing group, and
> drm_hwcomposer/drm_hwcomposer is a bit silly.
>
It did strike me a lot when drm_hwcomposer/drm_hwcomposer was
introduced. Fortunately moving repos in gitlab is reasonably pain-free


HTH
Emil
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] RFC: Migration to Gitlab

[Daniel Stone]

 Hi,

On Wed, 22 Aug 2018 at 16:02, Emil Velikov <emil.l.velikov@gmail.com> wrote:
> On 22 August 2018 at 12:44, Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
> > I think it's time to brainstorm a bit about the gitlab migration. Basic reasons:
> >
> > - fd.o admins want to deprecate shell accounts and hand-rolled
> > infrastructure, because it's a pain to keep secure&updated.
> >
> > - gitlab will allow us to add committers on our own, greatly
> > simplifying that process (and offloading that task from fd.o admins).
>
> Random thought - I really wish the admins spoke early and louder about issues.
> From infra to manpower and adhoc tool maintenance.

I thought I mostly had it covered, but fair enough. What knowledge are
you missing and how should it best be delivered?

One first-order issue is that as documented at
https://www.freedesktop.org/wiki/AccountRequests/ creating accounts
requires manual admin intervention. You can also search the
'freedesktop.org' product on Bugzilla to see the amount of time we
spend shuffling around GPG & SSH keys, which is about the worst
possible use of my time. Many other people have had access to drive
the ancient shell-script frontend to LDAP before, but for some reason
they mostly aren't very enthusiastic about doing it all the time.

In the mesa-dev@ thread about Mesa's migration, which is linked from
my blog post, I went into quite a lot of detail about why Jenkins was
not suitable to roll out across fd.o globally. That knowledge was
gained from trial & error, which was a lot of time burnt. The end
result is that we don't have any CI, except if people hang
Travis/AppVeyor off GitHub mirrors.

You've personally seen what's involved in setting up Git repository
hooks so we can build docs. We can't give access to let people work on
those, without giving them direct access to the literal Git repository
itself on disk. The hooks mostly involve bespoke sets of rsync jobs
and hand-managed SSH credentials and external services to build docs
and so on and so forth. None of this is accessible to a newcomer who
wants to make a non-code contribution: you have to find someone with
access to the repo to go bash some shell scripts directly and hope you
didn't screw it up. None of this is auditable, so if the repo
mysteriously gets wiped, then hopefully there are some backups
somewhere. But there definitely aren't any logs. Luckily we prevent
most people from having access to most repositories via a mandatory
'shell' which only allows people to push Git; this was written by hand
by us 12 years ago.

What else? Our fork of Patchwork was until recently based on an
ancient fork of Django, in a bespoke unreproducible production
environment. Bugzilla is patched for spam and templates (making
upgrades complex), yet we still have a surprising amount of spam pass
through. There's no way to delete spam, but you have to manually move
every bug to the 'spam' group, then go through and delete the user
which involves copying & pasting the email and a few clicks per user.
Mailman is patched to support Recaptcha, bringing more upgrade fun.
ikiwiki breaks all the time because it's designed to have the
public-access web server on the same host as the writeable Git
repositories.

Our servers are several years old and approaching life expiry, and we
have no more spare disk. You can see in #freedesktop IRC the constant
network connectivity issues people have to PSU almost every day. Our
attempts to root-cause and solve those have got nowhere.

I could go on, but the 'moved elsewhere' list in
https://gitlab.freedesktop.org/freedesktop/freedesktop/issues/2
indicates that we do have problems to solve, and that changing
peoples' SSH keys for them isn't the best way for us to be spending
the extremely limited time that we do have.

> > For the full in-depth writeup of everything, see
> >
> > https://www.fooishbar.org/blog/gitlab-fdo-introduction/

If you haven't seen this, or the post it was linked from, they would
be a good start:
https://lists.freedesktop.org/archives/freedesktop/2018-July/000370.html

There's also the long thread on mesa-dev a long time back which covers
a lot of this ground already.

> > - Figuring out the actual migration - we've been adding a pile of
> > committers since fd.o LDAP was converted to gitlab once back in
> > spring. We need to at least figure out how to move the new
> > accounts/committers.
>
> As a observer, allow me to put some ideas. You've mostly covered them
> all, my emphasis is to seriously stick with _one_ thing at a time.
> Attempting to do multiple things in parallel will end up with
> sub-optimal results.
>
>  - (at random point) cleanup the committers list - people who have not
> contributed in the last 1 year?

libdrm was previously covered under the Mesa ACL. Here are the other
committer lists, which you can see via 'getent group' on annarchy or
another machine:

amdkfd:x:922:fxkuehl,agd5f,deathsimple,danvet,jazhou,jbridgman,hwentland,tstdenis,gitlab-mirror,rui
drm-meson:x:936:narmstrong
drm:x:940:airlied,danvet
drm-intel:x:920:daniels,ickle,danvet,jwrdegoede,pzanoni,mlankhorst,ideak,vivijim,vsyrjala,jani,miku,mattrope,tursulin,dolphin,llandwerlin,lyudess,dpandiya,mthierry,mdnavare
drm-misc:x:932:daniels,daenzer,anholt,agd5f,ickle,deathsimple,kraxel,danvet,jwrdegoede,mlankhorst,tagr,mperes,vivijim,dvdhrm,vsyrjala,pinchartl,jani,evelikov,lynxeye,dolphin,sumits,architt,seanpaul,llandwerlin,lyudess,padovan,narmstrong,hwentland,bbrezillion,shawnguo,ahajda,lukas,vinceab,benjamin.gaignard,patrik,pcornu,notro,linusw,mripard,hjc,anushas,mmind,hyunk,tomba,jsarha,wens,andr2000,alexg1,dliviu,ayankh,dlech,mdnavare,shashank,ph5

>  - setup drm group, copy/migrate accounts - one could even reuse the
> existing credentials

Yes, dealing with accounts is covered here:
https://gitlab.freedesktop.org/freedesktop/freedesktop/wikis/home#how-do-i-createaccess-my-user-account

That page is linked from the blog post as well as the freedesktop@ post.

>  - move git repos to gitlab, the push URL change, cgit mirror
> preserves the normal fetch ones as well as PW hooks

Yes, this is what happens. The old repos give you a long message
explaining how to set up your GitLab account and change your Git
remote if you try to push to them.

>  - work out how new accounts are handled - still in bugzilla, otherwise
>
> At this stage only workflow change is a) once-off account setup and b)
> pushURL update
> As a follow-up one can setup anything fancy.
>  - migrate PW/other hooks
>  - migrate bugs - if applicable
>  - add new hooks - DRM docs, other
>  - etc
>
>
> > - Similar, maintainer-tools needs to move. We probably want to move
> > all the dim maintained kernel repos in one go, to avoid headaches with
> > double-accounts needed for committers.
>
> One should be able to create a separate repo for these. And then either:
>  - one by one add the required features into the gitlab MR machinery,
>  - or, wire the execution in pre/post merge stage.
>
> IIRC there are some upstream requests about the former.

He's just talking about migrating repository hosting (where people
push), not opening up for MRs (how people review code); see the reply
to Jani upthread for more detail.

> > - CI, linux-next and everyone else should be fine, since the
> > cgit/non-ssh paths will keep working (they'll be read-only mirrors).
> > Need to double-check that with everyone.
> >
> > - Some organization structure would be good.
> >
> > https://cgit.freedesktop.org/drm
> >
> > libdrm won't be part of the gitlab drm group because that's already
> > moved under mesa (and you can't symlink/mulit-home anymore on gitlab):
> >
> > https://gitlab.freedesktop.org/mesa/drm
> >
> > But there's also drm_hwcomposer, which we might want to migrate into
> > drm too - gitlab requires a containing group, and
> > drm_hwcomposer/drm_hwcomposer is a bit silly.
> >
> It did strike me a lot when drm_hwcomposer/drm_hwcomposer was
> introduced. Fortunately moving repos in gitlab is reasonably pain-free

It is, but if they don't share an ACL and the only benefit is
cosmetic, then ... meh? Weston doesn't live in drm/ either and that's
OK.

Cheers,
Daniel
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev