* [Buildroot] Github download helper possibly not working @ 2023-08-28 20:02 Woody Douglass via buildroot 2023-08-28 20:45 ` Woody Douglass via buildroot 2023-08-28 20:45 ` Thomas Petazzoni via buildroot 0 siblings, 2 replies; 5+ messages in thread From: Woody Douglass via buildroot @ 2023-08-28 20:02 UTC (permalink / raw) To: buildroot@buildroot.org Hello all, I've noticed that packages that use the `github` download helper are falling over to buildroot mirrors. I've tried with packages `yaml-cpp`, `zlog`, `swupdate`, and `pcm-tools` -- all are redirected before eventually getting a 403 from codeload.github.com and falling back to sources.buildroot.net. Is anyone else seeing this behavior? I'm trying to find a workaround now, but I'd appreciate any help! Thank you very much, Woodrow Douglass _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Buildroot] Github download helper possibly not working 2023-08-28 20:02 [Buildroot] Github download helper possibly not working Woody Douglass via buildroot @ 2023-08-28 20:45 ` Woody Douglass via buildroot 2023-08-29 16:46 ` Kristopher Adler 2023-08-28 20:45 ` Thomas Petazzoni via buildroot 1 sibling, 1 reply; 5+ messages in thread From: Woody Douglass via buildroot @ 2023-08-28 20:45 UTC (permalink / raw) To: buildroot@buildroot.org To follow up, It seems if I uninstall wget, and install busybox, and use busybox's version of wget, downloads are successful. I'm not sure what, in the confluence of wget and github.com, causes this issue, but I have a workaround for now. I'm not sure if/how the buildroot project should respond to this issue, but that's the information I have. Thanks again, Woodrow Douglass On 8/28/23 16:02, Woodrow Douglass wrote: > Hello all, > > I've noticed that packages that use the `github` download helper are > falling over to buildroot mirrors. I've tried with packages > `yaml-cpp`, `zlog`, `swupdate`, and `pcm-tools` -- all are redirected > before eventually getting a 403 from codeload.github.com and falling > back to sources.buildroot.net. Is anyone else seeing this behavior? > I'm trying to find a workaround now, but I'd appreciate any help! > > Thank you very much, > > Woodrow Douglass > > _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Buildroot] Github download helper possibly not working 2023-08-28 20:45 ` Woody Douglass via buildroot @ 2023-08-29 16:46 ` Kristopher Adler 0 siblings, 0 replies; 5+ messages in thread From: Kristopher Adler @ 2023-08-29 16:46 UTC (permalink / raw) To: buildroot@buildroot.org; +Cc: Woody Douglass Hi Woodrow, I experienced the same issue yesterday. There's a community discussion on GitHub about this that explains what went wrong: https://github.com/orgs/community/discussions/65227#discussioncomment-6848413 Requests using auto TLS protocol selection (wget's default behavior) were inadvertently blocked by GitHub servers, because some changes the GitHub team made to block a scraper were overly broad. I was able to workaround the issue temporarily by adding "secure-protocol=TLSv1_2" to a ~/.wgetrc file. Fortunately, the GitHub team resolved the issue, so the workaround is no longer necessary. Best wishes, Kris Adler On 8/28/23 15:45, Woody Douglass via buildroot wrote: > To follow up, > > It seems if I uninstall wget, and install busybox, and use busybox's > version of wget, downloads are successful. I'm not sure what, in the > confluence of wget and github.com, causes this issue, but I have a > workaround for now. I'm not sure if/how the buildroot project should > respond to this issue, but that's the information I have. > > Thanks again, > Woodrow Douglass > > On 8/28/23 16:02, Woodrow Douglass wrote: >> Hello all, >> >> I've noticed that packages that use the `github` download helper are >> falling over to buildroot mirrors. I've tried with packages >> `yaml-cpp`, `zlog`, `swupdate`, and `pcm-tools` -- all are redirected >> before eventually getting a 403 from codeload.github.com and falling >> back to sources.buildroot.net. Is anyone else seeing this behavior? >> I'm trying to find a workaround now, but I'd appreciate any help! >> >> Thank you very much, >> >> Woodrow Douglass >> >> > > _______________________________________________ > buildroot mailing list > buildroot@buildroot.org > https://lists.buildroot.org/mailman/listinfo/buildroot _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Buildroot] Github download helper possibly not working 2023-08-28 20:02 [Buildroot] Github download helper possibly not working Woody Douglass via buildroot 2023-08-28 20:45 ` Woody Douglass via buildroot @ 2023-08-28 20:45 ` Thomas Petazzoni via buildroot 2023-08-29 10:30 ` Yann E. MORIN 1 sibling, 1 reply; 5+ messages in thread From: Thomas Petazzoni via buildroot @ 2023-08-28 20:45 UTC (permalink / raw) To: Woody Douglass via buildroot; +Cc: Woody Douglass On Mon, 28 Aug 2023 20:02:34 +0000 Woody Douglass via buildroot <buildroot@buildroot.org> wrote: > I've noticed that packages that use the `github` download helper are > falling over to buildroot mirrors. I've tried with packages `yaml-cpp`, > `zlog`, `swupdate`, and `pcm-tools` -- all are redirected before > eventually getting a 403 from codeload.github.com and falling back to > sources.buildroot.net. Is anyone else seeing this behavior? I'm trying > to find a workaround now, but I'd appreciate any help! It works fine here: $ make yaml-cpp-source >>> yaml-cpp 0.7.0 Downloading wget --passive-ftp -nd -t 3 -O '/home/thomas/projets/buildroot/output/build/.yaml-cpp-0.7.0.tar.gz.TezLTj/output' 'https://github.com/jbeder/yaml-cpp/archive/yaml-cpp-0.7.0/yaml-cpp-0.7.0.tar.gz' --2023-08-28 22:43:34-- https://github.com/jbeder/yaml-cpp/archive/yaml-cpp-0.7.0/yaml-cpp-0.7.0.tar.gz Resolving github.com (github.com)... 140.82.121.4 Connecting to github.com (github.com)|140.82.121.4|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://codeload.github.com/jbeder/yaml-cpp/tar.gz/refs/tags/yaml-cpp-0.7.0 [following] --2023-08-28 22:43:34-- https://codeload.github.com/jbeder/yaml-cpp/tar.gz/refs/tags/yaml-cpp-0.7.0 Resolving codeload.github.com (codeload.github.com)... 140.82.121.10 Connecting to codeload.github.com (codeload.github.com)|140.82.121.10|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1033237 (1009K) [application/x-gzip] Saving to: ‘/home/thomas/projets/buildroot/output/build/.yaml-cpp-0.7.0.tar.gz.TezLTj/output’ /home/thomas/projets/buildroot/ 100%[=====================================================>] 1009K 923KB/s in 1,1s 2023-08-28 22:43:36 (923 KB/s) - ‘/home/thomas/projets/buildroot/output/build/.yaml-cpp-0.7.0.tar.gz.TezLTj/output’ saved [1033237/1033237] yaml-cpp-0.7.0.tar.gz: OK (sha256: 43e6a9fcb146ad871515f0d0873947e5d497a1c9c60c58cb102a97b47208b7c3) $ make zlog-source >>> zlog 1.2.16 Downloading wget --passive-ftp -nd -t 3 -O '/home/thomas/projets/buildroot/output/build/.zlog-1.2.16.tar.gz.jktoPe/output' 'https://github.com/HardySimpson/zlog/archive/1.2.16/zlog-1.2.16.tar.gz' --2023-08-28 22:43:51-- https://github.com/HardySimpson/zlog/archive/1.2.16/zlog-1.2.16.tar.gz Resolving github.com (github.com)... 140.82.121.4 Connecting to github.com (github.com)|140.82.121.4|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://codeload.github.com/HardySimpson/zlog/tar.gz/refs/tags/1.2.16 [following] --2023-08-28 22:43:51-- https://codeload.github.com/HardySimpson/zlog/tar.gz/refs/tags/1.2.16 Resolving codeload.github.com (codeload.github.com)... 140.82.121.10 Connecting to codeload.github.com (codeload.github.com)|140.82.121.10|:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [application/x-gzip] Saving to: ‘/home/thomas/projets/buildroot/output/build/.zlog-1.2.16.tar.gz.jktoPe/output’ /home/thomas/projets/buildroot/ [ <=> ] 122,20K --.-KB/s in 0,1s 2023-08-28 22:43:52 (1,08 MB/s) - ‘/home/thomas/projets/buildroot/output/build/.zlog-1.2.16.tar.gz.jktoPe/output’ saved [125131] zlog-1.2.16.tar.gz: OK (sha256: 742401902f2134eb272c49631fe5c38d7aeb9a2ad56fa3ec3d15219b371ba655) $ make swupdate-source >>> swupdate 2022.12 Downloading wget --passive-ftp -nd -t 3 -O '/home/thomas/projets/buildroot/output/build/.swupdate-2022.12.tar.gz.ofwOCw/output' 'https://github.com/sbabic/swupdate/archive/2022.12/swupdate-2022.12.tar.gz' --2023-08-28 22:44:17-- https://github.com/sbabic/swupdate/archive/2022.12/swupdate-2022.12.tar.gz Resolving github.com (github.com)... 140.82.121.4 Connecting to github.com (github.com)|140.82.121.4|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://codeload.github.com/sbabic/swupdate/tar.gz/refs/tags/2022.12 [following] --2023-08-28 22:44:17-- https://codeload.github.com/sbabic/swupdate/tar.gz/refs/tags/2022.12 Resolving codeload.github.com (codeload.github.com)... 140.82.121.10 Connecting to codeload.github.com (codeload.github.com)|140.82.121.10|:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [application/x-gzip] Saving to: ‘/home/thomas/projets/buildroot/output/build/.swupdate-2022.12.tar.gz.ofwOCw/output’ /home/thomas/projets/buildroot/ [ <=> ] 6,63M 2,42MB/s in 2,7s 2023-08-28 22:44:20 (2,42 MB/s) - ‘/home/thomas/projets/buildroot/output/build/.swupdate-2022.12.tar.gz.ofwOCw/output’ saved [6957587] swupdate-2022.12.tar.gz: OK (sha256: e6335e87812a98a87f1c55df03c9f4e4ef042789570002c5db120b09f64b0d86) However, for pcm-tools, we do have a problem (though not the one you mentioned): $ make pcm-tools-source >>> pcm-tools 202110 Downloading wget --passive-ftp -nd -t 3 -O '/home/thomas/projets/buildroot/output/build/.pcm-tools-202110.tar.gz.39EDUL/output' 'https://github.com/opcm/pcm/archive/202110/pcm-tools-202110.tar.gz' --2023-08-28 22:44:37-- https://github.com/opcm/pcm/archive/202110/pcm-tools-202110.tar.gz Resolving github.com (github.com)... 140.82.121.4 Connecting to github.com (github.com)|140.82.121.4|:443... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: https://github.com/intel/pcm/archive/202110/pcm-tools-202110.tar.gz [following] --2023-08-28 22:44:38-- https://github.com/intel/pcm/archive/202110/pcm-tools-202110.tar.gz Reusing existing connection to github.com:443. HTTP request sent, awaiting response... 302 Found Location: https://codeload.github.com/intel/pcm/tar.gz/refs/tags/202110 [following] --2023-08-28 22:44:38-- https://codeload.github.com/intel/pcm/tar.gz/refs/tags/202110 Resolving codeload.github.com (codeload.github.com)... 140.82.121.9 Connecting to codeload.github.com (codeload.github.com)|140.82.121.9|:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [application/x-gzip] Saving to: ‘/home/thomas/projets/buildroot/output/build/.pcm-tools-202110.tar.gz.39EDUL/output’ /home/thomas/projets/buildroot/ [ <=> ] 1,15M 2,18MB/s in 0,5s 2023-08-28 22:44:39 (2,18 MB/s) - ‘/home/thomas/projets/buildroot/output/build/.pcm-tools-202110.tar.gz.39EDUL/output’ saved [1205829] ERROR: pcm-tools-202110.tar.gz has wrong sha256 hash: ERROR: expected: aa48ab1473720aeb7837b67bfc612100f484748720a8b8034daff00419709057 ERROR: got : 90a5931cea24f1b0da76e22c712e55375df157e87f26edaa70b9660405852725 ERROR: Incomplete download, or man-in-the-middle (MITM) attack Thomas -- Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Buildroot] Github download helper possibly not working 2023-08-28 20:45 ` Thomas Petazzoni via buildroot @ 2023-08-29 10:30 ` Yann E. MORIN 0 siblings, 0 replies; 5+ messages in thread From: Yann E. MORIN @ 2023-08-29 10:30 UTC (permalink / raw) To: Thomas Petazzoni; +Cc: Woody Douglass, Woody Douglass via buildroot Thomas, Woody, All, On 2023-08-28 22:45 +0200, Thomas Petazzoni via buildroot spake thusly: > On Mon, 28 Aug 2023 20:02:34 +0000 > Woody Douglass via buildroot <buildroot@buildroot.org> wrote: > > I've noticed that packages that use the `github` download helper are > > falling over to buildroot mirrors. I've tried with packages `yaml-cpp`, > > `zlog`, `swupdate`, and `pcm-tools` -- all are redirected before > > eventually getting a 403 from codeload.github.com and falling back to > > sources.buildroot.net. Is anyone else seeing this behavior? I'm trying > > to find a workaround now, but I'd appreciate any help! > It works fine here: Works fine here too. [--SNIP--] > However, for pcm-tools, we do have a problem (though not the one you > mentioned): > $ make pcm-tools-source [--SNIP--] > ERROR: pcm-tools-202110.tar.gz has wrong sha256 hash: > ERROR: expected: aa48ab1473720aeb7837b67bfc612100f484748720a8b8034daff00419709057 > ERROR: got : 90a5931cea24f1b0da76e22c712e55375df157e87f26edaa70b9660405852725 > ERROR: Incomplete download, or man-in-the-middle (MITM) attack I also have this issue, and indeed the cntent changed; here's the diff: diff -durN pcm-202110.old/version.h pcm-202110.new/version.h --- pcm-202110.old/version.h 2021-10-25 16:07:54.000000000 +0200 +++ pcm-202110.new/version.h 2021-10-25 16:07:54.000000000 +0200 @@ -1 +1 @@ -#define PCM_VERSION " (2021-10-25 16:07:54 +0200 ID=93fc9193)" +#define PCM_VERSION " (2021-10-25 16:07:54 +0200 ID=93fc919)" That's all: a delta in the length of the short hash. So, what does version.h looks like in git (at the tag): $ cat version.h #define PCM_VERSION " ($Format:%ci ID=%h$)" OK, does that ring a bell? Yes, that's the same thing that we solved for subversion in c92be85e3a29 (support/download: make the svn backend more reproducible): $ man 5 gitattributes Creating an archive export-subst If the attribute export-subst is set for a file then Git will expand several placeholders when adding this file to an archive. The expansion depends on the availability of a commit ID, i.e., if git-archive(1) has been given a tree instead of a commit or a tag then no replacement will be done. The placeholders are the same as those for the option --pretty=format: of git-log(1), except that they need to be wrapped like this: $Format:PLACEHOLDERS$ in the file. E.g. the string $Format:%H$ will be replaced by the commit hash. This is something that is then done when the archive is generated, i.e. on the github side. So, Github again changed the way they generate their archives, except this is a very sneaky change. So, for pcm-tools, the only solution we have is to drop use of the github helper and switch over to a git download... Long term, I am still of the opinion that we should no longer, ever, rely on the remote to generate the archive, and we should always do that localy, even at the cost of download bandwidth, because we too often have similar issues, and this one is indeed very, very sneaky, and there is nothing that prevents a remote to change their archive generation on a whim (Github never guaranteed stability for those autogenerated archives to begin with...) Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2023-08-29 16:47 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2023-08-28 20:02 [Buildroot] Github download helper possibly not working Woody Douglass via buildroot 2023-08-28 20:45 ` Woody Douglass via buildroot 2023-08-29 16:46 ` Kristopher Adler 2023-08-28 20:45 ` Thomas Petazzoni via buildroot 2023-08-29 10:30 ` Yann E. MORIN
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox