[Buildroot] [Bug 14396] New: GIthub schema is broken

[Buildroot] [Bug 14396] New: GIthub schema is broken

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14396

            Bug ID: 14396
           Summary: GIthub schema is broken
           Product: buildroot
           Version: 2021.08.2
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: Other
          Assignee: unassigned@buildroot.uclibc.org
          Reporter: okitain@gmail.com
                CC: buildroot@uclibc.org
  Target Milestone: ---

Github seems to have adopted a new schema for releases: instead of:

https://github.com/<user>/<project>/archive/<version>/<project>-<version>.tar.gz

it's now:

https://github.com/<user>/<project>/archive/refs/tags/<version>.tar.gz

which is then redirected to a location containing <project>-<version>.tar.gz.
This effectively breaks at least jsmn and gtest, but probably all github
packages.

Steps to reproduce:

1. Download a fresh Buildroot, e.g. the master branch.
2. make menuconfig and select gtest
3. In menuconfig: disable BR2_BACKUP_SITE
3. make source

This results in GTest download failing with error 404:

wget --passive-ftp -nd -t 3 -O
'/home/user/projects/buildroot/output/build/.gtest-1.11.0.tar.gz.FcavV0/output'
'https://github.com/google/googletest/archive/release-1.11.0/gtest-1.11.0.tar.gz' 
--2021-11-26 18:12:27-- 
https://github.com/google/googletest/archive/release-1.11.0/gtest-1.11.0.tar.gz
Location:
https://codeload.github.com/google/googletest/tar.gz/release-1.11.0/gtest-1.11.0
[following]
--2021-11-26 18:12:27-- 
https://codeload.github.com/google/googletest/tar.gz/release-1.11.0/gtest-1.11.0
Connecting to proxy.elvees.com (proxy.elvees.com)|192.168.1.15|:3128...
connected.
Proxy request sent, awaiting response... 404 Not Found
2021-11-26 18:12:28 ERROR 404: Not Found.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14396] GIthub schema is broken

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14396

--- Comment #1 from Jens Maus <mail@jens-maus.de> ---
I am seeing the same behaviour here, thus can reproduce it.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14396] GIthub schema is broken

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14396

Yann E. MORIN <yann.morin.1998@free.fr> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |yann.morin.1998@free.fr
           Assignee|unassigned@buildroot.uclibc |yann.morin.1998@free.fr
                   |.org                        |
             Status|NEW                         |ASSIGNED

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14396] GIthub schema is broken

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14396

--- Comment #2 from Olga <okitain@gmail.com> ---
To save maintainers some trouble, I've been brainstorming how to fix this
issue.

The main problem that I see in this is that the value that should be assigned
to <PACKAGE>_SOURCE and the actual name of the .tar.gz differ. The bluntest
solution is to define <PACKAGE>_SOURCE for each package as something like
release-$(PACKAGE_VERSION).tar.gz, but that means that all mirrors of those
packages will fail and have to be rebuilt.

Instead, maybe $(call github,author,repo,version) should define a fake URI
protocol called github, which allows us to bodge the download process via a new
script located in support/download/github, saving the original <PACKAGE>_SOURCE
definitions while allowing download from unusual URLs that github provides.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14396] GIthub schema is broken

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14396

--- Comment #3 from Yann E. MORIN <yann.morin.1998@free.fr> ---
Olga, All,

Thanks for the report.

Indeed, this is very broken: for tags, github.com returns 404, while for
hashes, it returns 500. It sometimes also returns 502 or 504 in either
case.

> it's now:
> https://github.com/<user>/<project>/archive/refs/tags/<version>.tar.gz

This unfortunately does not work either: it returns 500:

    wget wget
https://github.com/google/googletest/refs/tags/release-1.11.0.tar.gz
    [...]
    HTTP request sent, awaiting response... 500 Internal Server Error

Another try a few minutes later yielded:

    HTTP request sent, awaiting response... 502 Bad Gateway

Besides, the URLs makes it look that it would anyway only work for tag, not
hashes.

Of course, this is not going to be easy to fix, and there does not seem like
there is an obvious fix anyway...

My opinion is that we should stop using the forges' archive mechanisms, which
have proved to be fluky in time, and use git for what it is and use git-clone
to retrieve git-hosted code (or svn et al.). This will however raise two
issues.
First, we have a lot (like a *lot*) of packages using the github macro; fixing
them is quite easy with a bit of sed, and manual review/tweaks in corner cases.
Second, this will trigger much bigger downloads, which can be a bit
overwhelming
for large repositories. That latter issue is smoothened by the use of the local
git cache, so this is only really a problem on the first download.

But right now, github.com seems to be having issues; it is very slow to return
any result, and even the website is very un-responsive... I'll try to
investigate
further tomorrow (GMT+1)...

Regards,
Yann E. MORIN.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14396] GIthub schema is broken

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14396

--- Comment #4 from Olga <okitain@gmail.com> ---
(In reply to Yann E. MORIN from comment #3)
> This unfortunately does not work either: it returns 500:
>
>    wget wget https://github.com/google/googletest/refs/tags/release-1.11.0.tar.gz
>    [...]
>    HTTP request sent, awaiting response... 500 Internal Server Error
>
>Another try a few minutes later yielded:
>
>    HTTP request sent, awaiting response... 502 Bad Gateway

That was because you missed the "/archive/" part of the URL between <project>
and /refs/tags/. With that part, it currently works.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14396] GIthub schema is broken

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14396

--- Comment #5 from Yann E. MORIN <yann.morin.1998@free.fr> ---
(In reply to Olga from comment #4)
> That was because you missed the "/archive/" part of the URL between <project> and /refs/tags/.

Indeed. And although this does work for tags, it does not for hashes.

It seems the old schema works for hashes, though. But given a FOO_VERSION,
there is no systematic solution to automatically tell hashes apart from tags,
so we don't have a solution to expand the github macro one way or the other.

Regards,
Yann E. MORIN.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14396] GIthub schema is broken

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14396

--- Comment #6 from Jens Maus <mail@jens-maus.de> ---
(In reply to Yann E. MORIN from comment #3)

> Second, this will trigger much bigger downloads, which can be a bit overwhelming
> for large repositories. That latter issue is smoothened by the use of the local
> git cache, so this is only really a problem on the first download.

Please keep in mind that there are a lot of CI driven build environments out
there (like in my case) where no git cache environment is available, thus
packages are always freshly downloaded. Therefore the direct archive download -
rather than doing a full fledged git clone on each run - resulted in
significantly smaller (i.e faster) downloads and thus faster CI build-times.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14396] GIthub schema is broken

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14396

--- Comment #7 from Yann E. MORIN <yann.morin.1998@free.fr> ---
FTR: this is tracked in Github with:
https://github.com/github/feedback/discussions/8149

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14396] GIthub schema is broken

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14396

--- Comment #8 from Yann E. MORIN <yann.morin.1998@free.fr> ---
(In reply to Jens Maus from comment #6)
> there are a lot of CI driven build environments out there (like in my case)
> where no git cache environment is available, thus packages are always freshly
> downloaded

Yes, sure, this was the reason for the github helper macro in the first place.

But if Github changes their download scheme (that's not the first time), there
is no way we can have our releases that are not broken at some point. For
example, 2021.02.7 (latest LTS release) is now broken, 2021.08.2 (latest
maintenance release) is now broken.

And if we have no way to detect whether we need the hash scheme, or the tag
scheme, what is the alternate option?

(In reply to Olga from comment #2)
> The bluntest solution is to define <PACKAGE>_SOURCE for each package as
> something like release-$(PACKAGE_VERSION).tar.gz,

For some packages, the tag is exactly the version string, for others a
leading 'v' is added, others have a custom, non-standard prefix (like gtest,
which has 'release-').

But such a mass fixup is not very nice... It needs carefull sed replacements
and a lot of review time to catch corner cases... We should try and find
another solution...

> but that means that all mirrors of those packages will fail and have to
> be rebuilt.

sources.buildroot.org is automatically rebuilt, so that is not an issue for
us. Others whi maintain their own mirror, will have to, well, just maintain
it by opushing the new archives as needed. ;-)

Regards,
Yann E. MORIN.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14396] GIthub schema is broken

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14396

Yann E. MORIN <yann.morin.1998@free.fr> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |WONTFIX
             Status|ASSIGNED                    |RESOLVED

--- Comment #9 from Yann E. MORIN <yann.morin.1998@free.fr> ---
Olga, All,

Github has reverted the change now:

https://github.com/github/feedback/discussions/8149#discussioncomment-1712006

As they said they will avoid such a regression in the future, we can
(hopefully!) consider the issue fixed, I believe.

Thanks for the report! :-)

Regards,
Yann E. MORIN.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot