Shutting down of the git-daemon (git protocol, https and ssh still remain)

Shutting down of the git-daemon (git protocol, https and ssh still remain)

[Claudius Heine]

Hi everyone,

in the past we used cgit, which provided cloning the u-boot repo via 
http, https, ssh and the git protocol.

When we migrated to gitlab, which doesn't support the git protocol, we 
implemented some additional service files to continue to support it for 
some of our git repositories.

We just noticed that cloning via the git protocol is currently broken, 
likely some changes in gitlab happened and we would need to recreate the 
symlinks on the server to point to the right repositories again.

Since that is another thing to maintain and keep up, we would like to 
just shut the git protocol down. AFAIK most projects moved to use https 
for cloning anyway, since there were not many reports of broken cloning.

What are your thoughts? Is it worth the effort to keep the git protocol up?

regards,
Claudius


-- 
DENX Software Engineering GmbH,        Managing Director: Erika Unter
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: Shutting down of the git-daemon (git protocol, https and ssh still remain)

[Quentin Schulz]

Hi Claudius,

On 6/12/24 12:20 PM, Claudius Heine wrote:
> Hi everyone,
> 
> in the past we used cgit, which provided cloning the u-boot repo via 
> http, https, ssh and the git protocol.
> 
> When we migrated to gitlab, which doesn't support the git protocol, we 
> implemented some additional service files to continue to support it for 
> some of our git repositories.
> 
> We just noticed that cloning via the git protocol is currently broken, 
> likely some changes in gitlab happened and we would need to recreate the 
> symlinks on the server to point to the right repositories again.
> 
> Since that is another thing to maintain and keep up, we would like to 
> just shut the git protocol down. AFAIK most projects moved to use https 
> for cloning anyway, since there were not many reports of broken cloning.
> 
> What are your thoughts? Is it worth the effort to keep the git protocol up?
> 

On the side of OpenEmbedded-Core, HTTPS is being used since Langdale 
(4.1) and the first dot release of Kirkstone (4.0.1) so only outdated 
layers would be impacted (provided they aren't already, but since 
there's a download cache system in Yocto, it's possible they're hitting 
their own local cache instead of fetching it from your server(s)).

It's never nice to see things going/breaking compatibility because it 
may break old builds which is always a nice trick up one's sleeve to 
figure out when things started to go wrong in one's image. Also, while I 
checked OE-Core, other third party/community maintained layer could be 
impacted. I assume this is the case of meta-amarula-engicam[1] and 
meta-allwinner-hx[2] for example.

I would suggest sending a mail to the different build systems mailing 
lists, I assume https://lists.openembedded.org/g/openembedded-devel and 
https://lists.yoctoproject.org/g/yocto for the OE-Core/YP side? Either 
to ask for their opinion or notify them (and specifically third party 
layers) it's going away or both.

[1] 
https://github.com/amarula/meta-amarula-engicam/blob/master/recipes-bsp/u-boot/u-boot-common_2018.05.inc
[2] 
https://gitlab.com/dimtass/meta-allwinner-hx/blob/master/recipes-bsp/u-boot/u-boot-common.inc

Cheers,
Quentin

Re: Shutting down of the git-daemon (git protocol, https and ssh still remain)

[Claudius Heine]

Hi Quentin,

On 2024-06-12 12:37 pm, Quentin Schulz wrote:
> Hi Claudius,
> 
> On 6/12/24 12:20 PM, Claudius Heine wrote:
>> Hi everyone,
>>
>> in the past we used cgit, which provided cloning the u-boot repo via 
>> http, https, ssh and the git protocol.
>>
>> When we migrated to gitlab, which doesn't support the git protocol, we 
>> implemented some additional service files to continue to support it 
>> for some of our git repositories.
>>
>> We just noticed that cloning via the git protocol is currently broken, 
>> likely some changes in gitlab happened and we would need to recreate 
>> the symlinks on the server to point to the right repositories again.
>>
>> Since that is another thing to maintain and keep up, we would like to 
>> just shut the git protocol down. AFAIK most projects moved to use 
>> https for cloning anyway, since there were not many reports of broken 
>> cloning.
>>
>> What are your thoughts? Is it worth the effort to keep the git 
>> protocol up?
>>
> 
> On the side of OpenEmbedded-Core, HTTPS is being used since Langdale 
> (4.1) and the first dot release of Kirkstone (4.0.1) so only outdated 
> layers would be impacted (provided they aren't already, but since 
> there's a download cache system in Yocto, it's possible they're hitting 
> their own local cache instead of fetching it from your server(s)).
> 
> It's never nice to see things going/breaking compatibility because it 
> may break old builds which is always a nice trick up one's sleeve to 
> figure out when things started to go wrong in one's image.

Well, this happens all the time, but maintaining backwards compatibility 
here should be done in a best-effort way.

With old build, they likely need old compilers and systems, which will 
probably break anyway if they use https or ssh, because obsolete crypto. 
If people want to build old stuff, they probably have to fiddle around 
everywhere anyway, or have a download cache.

The internet, or any network, isn't nice to old systems...

> Also, while I 
> checked OE-Core, other third party/community maintained layer could be 
> impacted. I assume this is the case of meta-amarula-engicam[1] and 
> meta-allwinner-hx[2] for example.
> 
> I would suggest sending a mail to the different build systems mailing 
> lists, I assume https://lists.openembedded.org/g/openembedded-devel and 
> https://lists.yoctoproject.org/g/yocto for the OE-Core/YP side? Either 
> to ask for their opinion or notify them (and specifically third party 
> layers) it's going away or both.

Not sure if I want to go through them all. Looking into all these 3rd 
party, downstream repos and informing everyone seems to be even more 
effort than maintaining the git protocol.

And if someone uses their repo, they will notice it quickly and can 
easily fix it.

Anyway, I fixed it yesterday, and it doesn't make much sense to stop it 
now. So I will leave it in place until it breaks again, then I just 
remove it.

thanks everyone for their input,
Claudius

> 
> [1] 
> https://github.com/amarula/meta-amarula-engicam/blob/master/recipes-bsp/u-boot/u-boot-common_2018.05.inc
> [2] 
> https://gitlab.com/dimtass/meta-allwinner-hx/blob/master/recipes-bsp/u-boot/u-boot-common.inc
> 
> Cheers,
> Quentin

-- 
DENX Software Engineering GmbH,        Managing Director: Erika Unter
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: Shutting down of the git-daemon (git protocol, https and ssh still remain)

[Marek Vasut]

On 6/13/24 9:49 AM, Claudius Heine wrote:

Hi,

[...]

>> Also, while I checked OE-Core, other third party/community maintained 
>> layer could be impacted. I assume this is the case of 
>> meta-amarula-engicam[1] and meta-allwinner-hx[2] for example.
>>
>> I would suggest sending a mail to the different build systems mailing 
>> lists, I assume https://lists.openembedded.org/g/openembedded-devel 
>> and https://lists.yoctoproject.org/g/yocto for the OE-Core/YP side? 
>> Either to ask for their opinion or notify them (and specifically third 
>> party layers) it's going away or both.
> 
> Not sure if I want to go through them all. Looking into all these 3rd 
> party, downstream repos and informing everyone seems to be even more 
> effort than maintaining the git protocol.
> 
> And if someone uses their repo, they will notice it quickly and can 
> easily fix it.

I very much agree, that ^ is probably the best way to go.

If some legacy repository breaks, fix the repository, hopefully there 
aren't too many of those to fix. Github deprecated git protocol a few 
years back and users managed too, except the fallout here is much 
smaller as it affects only one recipe (or a few).

I suspect the following patch to oe-core (or local.conf equivalent) 
might also entirely mitigate the issue by providing https fallback 
automatically, without the need to modify the legacy layer/recipe ?

If so, maybe this could be added to OE LTS releases at least ?

"
diff --git a/meta/classes-global/mirrors.bbclass 
b/meta/classes-global/mirrors.bbclass
index 862648eec5..50a14c5881 100644
--- a/meta/classes-global/mirrors.bbclass
+++ b/meta/classes-global/mirrors.bbclass
@@ -77,6 +77,7 @@ https?://downloads.yoctoproject.org/mirror/sources/ 
https://mirrors.kernel.org/y

  MIRRORS += "\
  git://salsa.debian.org/.*     git://salsa.debian.org/PATH;protocol=https \
+git://git.denx.de/.*          git://git.denx.de/PATH;protocol=https \
  git://git.gnome.org/.* 
git://gitlab.gnome.org/GNOME/PATH;protocol=https \
  git://git.infradead.org/.*    git://git.infraroot.at/PATH;protocol=https \
  git://.*/.*                   git://HOST/PATH;protocol=https \
"

> Anyway, I fixed it yesterday, and it doesn't make much sense to stop it 
> now. So I will leave it in place until it breaks again, then I just 
> remove it.

Do you know how long it was actually broken ? (i.e. how long nobody 
noticed and complained about it?)

Re: Shutting down of the git-daemon (git protocol, https and ssh still remain)

[Claudius Heine]

Hi Marek,

On 2024-06-13 3:12 pm, Marek Vasut wrote:
> On 6/13/24 9:49 AM, Claudius Heine wrote:
> 
> Hi,
> 
> [...]
> 
>>> Also, while I checked OE-Core, other third party/community maintained 
>>> layer could be impacted. I assume this is the case of 
>>> meta-amarula-engicam[1] and meta-allwinner-hx[2] for example.
>>>
>>> I would suggest sending a mail to the different build systems mailing 
>>> lists, I assume https://lists.openembedded.org/g/openembedded-devel 
>>> and https://lists.yoctoproject.org/g/yocto for the OE-Core/YP side? 
>>> Either to ask for their opinion or notify them (and specifically 
>>> third party layers) it's going away or both.
>>
>> Not sure if I want to go through them all. Looking into all these 3rd 
>> party, downstream repos and informing everyone seems to be even more 
>> effort than maintaining the git protocol.
>>
>> And if someone uses their repo, they will notice it quickly and can 
>> easily fix it.
> 
> I very much agree, that ^ is probably the best way to go.
> 
> If some legacy repository breaks, fix the repository, hopefully there 
> aren't too many of those to fix. Github deprecated git protocol a few 
> years back and users managed too, except the fallout here is much 
> smaller as it affects only one recipe (or a few).
> 
> I suspect the following patch to oe-core (or local.conf equivalent) 
> might also entirely mitigate the issue by providing https fallback 
> automatically, without the need to modify the legacy layer/recipe ?
> 
> If so, maybe this could be added to OE LTS releases at least ?
> 
> "
> diff --git a/meta/classes-global/mirrors.bbclass 
> b/meta/classes-global/mirrors.bbclass
> index 862648eec5..50a14c5881 100644
> --- a/meta/classes-global/mirrors.bbclass
> +++ b/meta/classes-global/mirrors.bbclass
> @@ -77,6 +77,7 @@ https?://downloads.yoctoproject.org/mirror/sources/ 
> https://mirrors.kernel.org/y
> 
>   MIRRORS += "\
>   git://salsa.debian.org/.*     
> git://salsa.debian.org/PATH;protocol=https \
> +git://git.denx.de/.*          git://git.denx.de/PATH;protocol=https \

Well the path will change a bit as well:

git://git.denx.de/u-boot.git -> https://source.denx.de/u-boot/u-boot.git
git://git.denx.de/u-boot-efi.git -> 
https://source.denx.de/u-boot/custodians/u-boot-efi.git
git://git.denx.de/xenomai.git -> https://source.denx.de/xenomai/xenomai.git
git://git.denx.de/mxssb.git -> https://source.denx.de/denx/mxssb.git

If we want to make it perfect, it will be more difficult than that.

>   git://git.gnome.org/.* git://gitlab.gnome.org/GNOME/PATH;protocol=https \
>   git://git.infradead.org/.*    
> git://git.infraroot.at/PATH;protocol=https \
>   git://.*/.*                   git://HOST/PATH;protocol=https \
> "
> 
>> Anyway, I fixed it yesterday, and it doesn't make much sense to stop 
>> it now. So I will leave it in place until it breaks again, then I just 
>> remove it.
> 
> Do you know how long it was actually broken ? (i.e. how long nobody 
> noticed and complained about it?)

So since 3rd of June, so about 5-8 days or so until someone noticed.

I mean I could make it fail 10% of the time, and issue a deprecation 
notice, and then raise that percentage each month a bit higher... Maybe 
some kind of scaling, the fewer people use it in a certain time, the 
oftener it fails... :D

regards,
Claudius

-- 
DENX Software Engineering GmbHt,        Managing Director: Erika Unter
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: Shutting down of the git-daemon (git protocol, https and ssh still remain)

[Marek Vasut]

On 6/13/24 4:13 PM, Claudius Heine wrote:

Hi,

>> I suspect the following patch to oe-core (or local.conf equivalent) 
>> might also entirely mitigate the issue by providing https fallback 
>> automatically, without the need to modify the legacy layer/recipe ?
>>
>> If so, maybe this could be added to OE LTS releases at least ?
>>
>> "
>> diff --git a/meta/classes-global/mirrors.bbclass 
>> b/meta/classes-global/mirrors.bbclass
>> index 862648eec5..50a14c5881 100644
>> --- a/meta/classes-global/mirrors.bbclass
>> +++ b/meta/classes-global/mirrors.bbclass
>> @@ -77,6 +77,7 @@ https?://downloads.yoctoproject.org/mirror/sources/ 
>> https://mirrors.kernel.org/y
>>
>>   MIRRORS += "\
>>   git://salsa.debian.org/.* git://salsa.debian.org/PATH;protocol=https \
>> +git://git.denx.de/.*          git://git.denx.de/PATH;protocol=https \
> 
> Well the path will change a bit as well:
> 
> git://git.denx.de/u-boot.git -> https://source.denx.de/u-boot/u-boot.git
> git://git.denx.de/u-boot-efi.git -> 
> https://source.denx.de/u-boot/custodians/u-boot-efi.git

Better avoid custodian trees in OE builds unless this is a test build.

> git://git.denx.de/xenomai.git -> https://source.denx.de/xenomai/xenomai.git
> git://git.denx.de/mxssb.git -> https://source.denx.de/denx/mxssb.git

mxssb is I think long superseded by U-Boot mkimage integration, so this 
can also be dropped.

The remaining two, u-boot and xenomai, yeah, those would likely need to 
be separate.

> If we want to make it perfect, it will be more difficult than that.
> 
>>   git://git.gnome.org/.* 
>> git://gitlab.gnome.org/GNOME/PATH;protocol=https \
>>   git://git.infradead.org/.* git://git.infraroot.at/PATH;protocol=https \
>>   git://.*/.*                   git://HOST/PATH;protocol=https \
>> "
>>
>>> Anyway, I fixed it yesterday, and it doesn't make much sense to stop 
>>> it now. So I will leave it in place until it breaks again, then I 
>>> just remove it.
>>
>> Do you know how long it was actually broken ? (i.e. how long nobody 
>> noticed and complained about it?)
> 
> So since 3rd of June, so about 5-8 days or so until someone noticed.

Hum, since it works now again, I think we can wrap this discussion up.

However, maybe this can serve as a wake up call for people to switch to 
https cloning ?

Re: Shutting down of the git-daemon (git protocol, https and ssh still remain)

[Rasmus Villemoes]

On 12/06/2024 12.20, Claudius Heine wrote:
> Hi everyone,
> 
> in the past we used cgit, which provided cloning the u-boot repo via
> http, https, ssh and the git protocol.
> 
> When we migrated to gitlab, which doesn't support the git protocol, we
> implemented some additional service files to continue to support it for
> some of our git repositories.
> 
> We just noticed that cloning via the git protocol is currently broken,
> likely some changes in gitlab happened and we would need to recreate the
> symlinks on the server to point to the right repositories again.
> 
> Since that is another thing to maintain and keep up, we would like to
> just shut the git protocol down. AFAIK most projects moved to use https
> for cloning anyway, since there were not many reports of broken cloning.
> 
> What are your thoughts? Is it worth the effort to keep the git protocol up?

So I don't think you should necessarily keep the git:// working if it
means extra maintenance. I just checked a couple of my local copies of
u-boot.git (don't ask, it's many different customer projects) and it
seems to be more or less random if I added upstream with git:// or
https:// . I can certainly change those, and thanks for the heads up.

I'm just wondering if it's possible to have the server send some message
to the client when the git protocol is used, for some deprecation period
(6 months?), so a developer doing "git remote update" will get notified.
He would then also have a chance to propagate that to whatever local CI
he's responsible for, instead of only noticing when things do start
breaking.

Rasmus

Re: Shutting down of the git-daemon (git protocol, https and ssh still remain)

[Claudius Heine]

Hi Rasmus,

On 2024-06-12 1:18 pm, Rasmus Villemoes wrote:
> On 12/06/2024 12.20, Claudius Heine wrote:
>> Hi everyone,
>>
>> in the past we used cgit, which provided cloning the u-boot repo via
>> http, https, ssh and the git protocol.
>>
>> When we migrated to gitlab, which doesn't support the git protocol, we
>> implemented some additional service files to continue to support it for
>> some of our git repositories.
>>
>> We just noticed that cloning via the git protocol is currently broken,
>> likely some changes in gitlab happened and we would need to recreate the
>> symlinks on the server to point to the right repositories again.
>>
>> Since that is another thing to maintain and keep up, we would like to
>> just shut the git protocol down. AFAIK most projects moved to use https
>> for cloning anyway, since there were not many reports of broken cloning.
>>
>> What are your thoughts? Is it worth the effort to keep the git protocol up?
> 
> So I don't think you should necessarily keep the git:// working if it
> means extra maintenance. I just checked a couple of my local copies of
> u-boot.git (don't ask, it's many different customer projects) and it
> seems to be more or less random if I added upstream with git:// or
> https:// . I can certainly change those, and thanks for the heads up.
> 
> I'm just wondering if it's possible to have the server send some message
> to the client when the git protocol is used, for some deprecation period
> (6 months?), so a developer doing "git remote update" will get notified.
> He would then also have a chance to propagate that to whatever local CI
> he's responsible for, instead of only noticing when things do start
> breaking.

We are using `git-daemon` and there is a `--accept-hook` which can print 
a single line to the requestor, but only if the request is denied.
So there doesn't seem to be a way to implement a deprecation period.

regards,
Claudius

-- 
DENX Software Engineering GmbH,        Managing Director: Erika Unter
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: Shutting down of the git-daemon (git protocol, https and ssh still remain)

[Rasmus Villemoes]

On 13/06/2024 09.32, Claudius Heine wrote:

>> I'm just wondering if it's possible to have the server send some message
>> to the client when the git protocol is used, for some deprecation period
>> (6 months?), so a developer doing "git remote update" will get notified.
>> He would then also have a chance to propagate that to whatever local CI
>> he's responsible for, instead of only noticing when things do start
>> breaking.
> 
> We are using `git-daemon` and there is a `--accept-hook` which can print
> a single line to the requestor, but only if the request is denied.
> So there doesn't seem to be a way to implement a deprecation period.

That's a pity. But perhaps, when it breaks again, you could still make
use of this to keep a stub up and running for a while: point
--accept-hook at a two-liner script that just prints a notice that the
git protocol has been disabled, what to use instead, and exit 1. That
will still be nicer than just "connection refused".

You could also point the user at some mailing list thread (this, or a
new post when you do pull the plug) with a bit more info than what can
reasonably fit in that --accept-hook message.

Rasmus