[Buildroot] Github download helper possibly not working

[Buildroot] Github download helper possibly not working

[Woody Douglass via buildroot]

Hello all,

I've noticed that packages that use the `github` download helper are 
falling over to buildroot mirrors. I've tried with packages `yaml-cpp`, 
`zlog`, `swupdate`, and `pcm-tools` -- all are redirected before 
eventually getting a 403 from codeload.github.com and falling back to 
sources.buildroot.net. Is anyone else seeing this behavior? I'm trying 
to find a workaround now, but I'd appreciate any help!

Thank you very much,

Woodrow Douglass


_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Github download helper possibly not working

[Woody Douglass via buildroot]

To follow up,

It seems if I uninstall wget, and install busybox, and use busybox's 
version of wget, downloads are successful. I'm not sure what, in the 
confluence of wget and github.com, causes this issue, but I have a 
workaround for now. I'm not sure if/how the buildroot project should 
respond to this issue, but that's the information I have.

Thanks again,
Woodrow Douglass

On 8/28/23 16:02, Woodrow Douglass wrote:
> Hello all,
>
> I've noticed that packages that use the `github` download helper are 
> falling over to buildroot mirrors. I've tried with packages 
> `yaml-cpp`, `zlog`, `swupdate`, and `pcm-tools` -- all are redirected 
> before eventually getting a 403 from codeload.github.com and falling 
> back to sources.buildroot.net. Is anyone else seeing this behavior? 
> I'm trying to find a workaround now, but I'd appreciate any help!
>
> Thank you very much,
>
> Woodrow Douglass
>
>

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Github download helper possibly not working

[Thomas Petazzoni via buildroot]

On Mon, 28 Aug 2023 20:02:34 +0000
Woody Douglass via buildroot <buildroot@buildroot.org> wrote:

> I've noticed that packages that use the `github` download helper are 
> falling over to buildroot mirrors. I've tried with packages `yaml-cpp`, 
> `zlog`, `swupdate`, and `pcm-tools` -- all are redirected before 
> eventually getting a 403 from codeload.github.com and falling back to 
> sources.buildroot.net. Is anyone else seeing this behavior? I'm trying 
> to find a workaround now, but I'd appreciate any help!

It works fine here:

$ make yaml-cpp-source
>>> yaml-cpp 0.7.0 Downloading
wget --passive-ftp -nd -t 3 -O '/home/thomas/projets/buildroot/output/build/.yaml-cpp-0.7.0.tar.gz.TezLTj/output' 'https://github.com/jbeder/yaml-cpp/archive/yaml-cpp-0.7.0/yaml-cpp-0.7.0.tar.gz' 
--2023-08-28 22:43:34--  https://github.com/jbeder/yaml-cpp/archive/yaml-cpp-0.7.0/yaml-cpp-0.7.0.tar.gz
Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/jbeder/yaml-cpp/tar.gz/refs/tags/yaml-cpp-0.7.0 [following]
--2023-08-28 22:43:34--  https://codeload.github.com/jbeder/yaml-cpp/tar.gz/refs/tags/yaml-cpp-0.7.0
Resolving codeload.github.com (codeload.github.com)... 140.82.121.10
Connecting to codeload.github.com (codeload.github.com)|140.82.121.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1033237 (1009K) [application/x-gzip]
Saving to: ‘/home/thomas/projets/buildroot/output/build/.yaml-cpp-0.7.0.tar.gz.TezLTj/output’

/home/thomas/projets/buildroot/ 100%[=====================================================>]   1009K   923KB/s    in 1,1s    

2023-08-28 22:43:36 (923 KB/s) - ‘/home/thomas/projets/buildroot/output/build/.yaml-cpp-0.7.0.tar.gz.TezLTj/output’ saved [1033237/1033237]

yaml-cpp-0.7.0.tar.gz: OK (sha256: 43e6a9fcb146ad871515f0d0873947e5d497a1c9c60c58cb102a97b47208b7c3)

$ make zlog-source
>>> zlog 1.2.16 Downloading
wget --passive-ftp -nd -t 3 -O '/home/thomas/projets/buildroot/output/build/.zlog-1.2.16.tar.gz.jktoPe/output' 'https://github.com/HardySimpson/zlog/archive/1.2.16/zlog-1.2.16.tar.gz' 
--2023-08-28 22:43:51--  https://github.com/HardySimpson/zlog/archive/1.2.16/zlog-1.2.16.tar.gz
Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/HardySimpson/zlog/tar.gz/refs/tags/1.2.16 [following]
--2023-08-28 22:43:51--  https://codeload.github.com/HardySimpson/zlog/tar.gz/refs/tags/1.2.16
Resolving codeload.github.com (codeload.github.com)... 140.82.121.10
Connecting to codeload.github.com (codeload.github.com)|140.82.121.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: ‘/home/thomas/projets/buildroot/output/build/.zlog-1.2.16.tar.gz.jktoPe/output’

/home/thomas/projets/buildroot/     [ <=>                                                  ] 122,20K  --.-KB/s    in 0,1s    

2023-08-28 22:43:52 (1,08 MB/s) - ‘/home/thomas/projets/buildroot/output/build/.zlog-1.2.16.tar.gz.jktoPe/output’ saved [125131]

zlog-1.2.16.tar.gz: OK (sha256: 742401902f2134eb272c49631fe5c38d7aeb9a2ad56fa3ec3d15219b371ba655)

$ make swupdate-source
>>> swupdate 2022.12 Downloading
wget --passive-ftp -nd -t 3 -O '/home/thomas/projets/buildroot/output/build/.swupdate-2022.12.tar.gz.ofwOCw/output' 'https://github.com/sbabic/swupdate/archive/2022.12/swupdate-2022.12.tar.gz' 
--2023-08-28 22:44:17--  https://github.com/sbabic/swupdate/archive/2022.12/swupdate-2022.12.tar.gz
Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/sbabic/swupdate/tar.gz/refs/tags/2022.12 [following]
--2023-08-28 22:44:17--  https://codeload.github.com/sbabic/swupdate/tar.gz/refs/tags/2022.12
Resolving codeload.github.com (codeload.github.com)... 140.82.121.10
Connecting to codeload.github.com (codeload.github.com)|140.82.121.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: ‘/home/thomas/projets/buildroot/output/build/.swupdate-2022.12.tar.gz.ofwOCw/output’

/home/thomas/projets/buildroot/     [              <=>                                     ]   6,63M  2,42MB/s    in 2,7s    

2023-08-28 22:44:20 (2,42 MB/s) - ‘/home/thomas/projets/buildroot/output/build/.swupdate-2022.12.tar.gz.ofwOCw/output’ saved [6957587]

swupdate-2022.12.tar.gz: OK (sha256: e6335e87812a98a87f1c55df03c9f4e4ef042789570002c5db120b09f64b0d86)

However, for pcm-tools, we do have a problem (though not the one you
mentioned):

$ make pcm-tools-source
>>> pcm-tools 202110 Downloading
wget --passive-ftp -nd -t 3 -O '/home/thomas/projets/buildroot/output/build/.pcm-tools-202110.tar.gz.39EDUL/output' 'https://github.com/opcm/pcm/archive/202110/pcm-tools-202110.tar.gz' 
--2023-08-28 22:44:37--  https://github.com/opcm/pcm/archive/202110/pcm-tools-202110.tar.gz
Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://github.com/intel/pcm/archive/202110/pcm-tools-202110.tar.gz [following]
--2023-08-28 22:44:38--  https://github.com/intel/pcm/archive/202110/pcm-tools-202110.tar.gz
Reusing existing connection to github.com:443.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/intel/pcm/tar.gz/refs/tags/202110 [following]
--2023-08-28 22:44:38--  https://codeload.github.com/intel/pcm/tar.gz/refs/tags/202110
Resolving codeload.github.com (codeload.github.com)... 140.82.121.9
Connecting to codeload.github.com (codeload.github.com)|140.82.121.9|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: ‘/home/thomas/projets/buildroot/output/build/.pcm-tools-202110.tar.gz.39EDUL/output’

/home/thomas/projets/buildroot/     [   <=>                                                ]   1,15M  2,18MB/s    in 0,5s    

2023-08-28 22:44:39 (2,18 MB/s) - ‘/home/thomas/projets/buildroot/output/build/.pcm-tools-202110.tar.gz.39EDUL/output’ saved [1205829]

ERROR: pcm-tools-202110.tar.gz has wrong sha256 hash:
ERROR: expected: aa48ab1473720aeb7837b67bfc612100f484748720a8b8034daff00419709057
ERROR: got     : 90a5931cea24f1b0da76e22c712e55375df157e87f26edaa70b9660405852725
ERROR: Incomplete download, or man-in-the-middle (MITM) attack

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Github download helper possibly not working

[Yann E. MORIN]

Thomas, Woody, All,

On 2023-08-28 22:45 +0200, Thomas Petazzoni via buildroot spake thusly:
> On Mon, 28 Aug 2023 20:02:34 +0000
> Woody Douglass via buildroot <buildroot@buildroot.org> wrote:
> > I've noticed that packages that use the `github` download helper are 
> > falling over to buildroot mirrors. I've tried with packages `yaml-cpp`, 
> > `zlog`, `swupdate`, and `pcm-tools` -- all are redirected before 
> > eventually getting a 403 from codeload.github.com and falling back to 
> > sources.buildroot.net. Is anyone else seeing this behavior? I'm trying 
> > to find a workaround now, but I'd appreciate any help!
> It works fine here:

Works fine here too.

[--SNIP--]
> However, for pcm-tools, we do have a problem (though not the one you
> mentioned):
> $ make pcm-tools-source
[--SNIP--]
> ERROR: pcm-tools-202110.tar.gz has wrong sha256 hash:
> ERROR: expected: aa48ab1473720aeb7837b67bfc612100f484748720a8b8034daff00419709057
> ERROR: got     : 90a5931cea24f1b0da76e22c712e55375df157e87f26edaa70b9660405852725
> ERROR: Incomplete download, or man-in-the-middle (MITM) attack

I also have this issue, and indeed the cntent changed; here's the diff:

    diff -durN pcm-202110.old/version.h pcm-202110.new/version.h
    --- pcm-202110.old/version.h    2021-10-25 16:07:54.000000000 +0200
    +++ pcm-202110.new/version.h    2021-10-25 16:07:54.000000000 +0200
    @@ -1 +1 @@
    -#define PCM_VERSION " (2021-10-25 16:07:54 +0200 ID=93fc9193)"
    +#define PCM_VERSION " (2021-10-25 16:07:54 +0200 ID=93fc919)"

That's all: a delta in the length of the short hash.

So, what does version.h looks like in git (at the tag):

    $ cat version.h
    #define PCM_VERSION " ($Format:%ci ID=%h$)"

OK, does that ring a bell? Yes, that's the same thing that we solved for
subversion in c92be85e3a29 (support/download: make the svn backend more
reproducible):

    $ man 5 gitattributes
       Creating an archive
         export-subst
           If the attribute export-subst is set for a file then Git
           will expand several placeholders when adding this file to
           an archive. The expansion depends on the availability of
           a commit ID, i.e., if git-archive(1) has been given a tree
           instead of a commit or a tag then no replacement will be
           done. The placeholders are the same as those for the option
           --pretty=format: of git-log(1), except that they need to be
           wrapped like this: $Format:PLACEHOLDERS$ in the file. E.g.
           the string $Format:%H$ will be replaced by the commit hash.

This is something that is then done when the archive is generated, i.e.
on the github side. So, Github again changed the way they generate their
archives, except this is a very sneaky change.

So, for pcm-tools, the only solution we have is to drop use of the
github helper and switch over to a git download...

Long term, I am still of the opinion that we should no longer, ever,
rely on the remote to generate the archive, and we should always do
that localy, even at the cost of download bandwidth, because we too
often have similar issues, and this one is indeed very, very sneaky, and
there is nothing that prevents a remote to change their archive
generation on a whim (Github never guaranteed stability for those
autogenerated archives to begin with...)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Github download helper possibly not working

[Kristopher Adler]

Hi Woodrow,

I experienced the same issue yesterday. There's a community discussion 
on GitHub about this that explains what went wrong:

https://github.com/orgs/community/discussions/65227#discussioncomment-6848413

Requests using auto TLS protocol selection (wget's default behavior) 
were inadvertently blocked by GitHub servers, because some changes the 
GitHub team made to block a scraper were overly broad.

I was able to workaround the issue temporarily by adding 
"secure-protocol=TLSv1_2" to a ~/.wgetrc file. Fortunately, the GitHub 
team resolved the issue, so the workaround is no longer necessary.

Best wishes,
Kris Adler

On 8/28/23 15:45, Woody Douglass via buildroot wrote:
> To follow up,
> 
> It seems if I uninstall wget, and install busybox, and use busybox's
> version of wget, downloads are successful. I'm not sure what, in the
> confluence of wget and github.com, causes this issue, but I have a
> workaround for now. I'm not sure if/how the buildroot project should
> respond to this issue, but that's the information I have.
> 
> Thanks again,
> Woodrow Douglass
> 
> On 8/28/23 16:02, Woodrow Douglass wrote:
>> Hello all,
>>
>> I've noticed that packages that use the `github` download helper are
>> falling over to buildroot mirrors. I've tried with packages
>> `yaml-cpp`, `zlog`, `swupdate`, and `pcm-tools` -- all are redirected
>> before eventually getting a 403 from codeload.github.com and falling
>> back to sources.buildroot.net. Is anyone else seeing this behavior?
>> I'm trying to find a workaround now, but I'd appreciate any help!
>>
>> Thank you very much,
>>
>> Woodrow Douglass
>>
>>
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot