[Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[James Hilliard]

Some packages like the upcoming tailscale package need to set GOPROXY
to get the correct dependencies.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
 package/pkg-golang.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/package/pkg-golang.mk b/package/pkg-golang.mk
index 8e27602d41..118908a159 100644
--- a/package/pkg-golang.mk
+++ b/package/pkg-golang.mk
@@ -85,10 +85,12 @@ define $(2)_GEN_GOMOD
 endef
 $(2)_POST_PATCH_HOOKS += $(2)_GEN_GOMOD
 
+$(2)_DL_GOPROXY ?= direct
+
 $(2)_DOWNLOAD_POST_PROCESS = go
 $(2)_DL_ENV += \
 	$$(HOST_GO_COMMON_ENV) \
-	GOPROXY=direct \
+	GOPROXY=$$($(2)_DL_GOPROXY) \
 	$$($(2)_GO_ENV)
 
 # Because we append vendored info, we can't rely on the values being empty
-- 
2.34.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v4 2/2] package/tailscale: new package

[James Hilliard]

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
Changes v3 -> v4:
  - restore GOPROXY override
Changes v2 -> v3:
  - set required CONFIG_TUN kernel config option
Changes v1 -> v2:
  - don't set GOPROXY
---
 package/Config.in                |  1 +
 package/tailscale/Config.in      |  8 ++++++++
 package/tailscale/tailscale.hash |  3 +++
 package/tailscale/tailscale.mk   | 23 +++++++++++++++++++++++
 4 files changed, 35 insertions(+)
 create mode 100644 package/tailscale/Config.in
 create mode 100644 package/tailscale/tailscale.hash
 create mode 100644 package/tailscale/tailscale.mk

diff --git a/package/Config.in b/package/Config.in
index 2ac351cce5..a12010b32d 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -2622,6 +2622,7 @@ endif
 	source "package/stunnel/Config.in"
 	source "package/suricata/Config.in"
 	source "package/system-config-printer/Config.in"
+	source "package/tailscale/Config.in"
 	source "package/tcpdump/Config.in"
 	source "package/tcping/Config.in"
 	source "package/tcpreplay/Config.in"
diff --git a/package/tailscale/Config.in b/package/tailscale/Config.in
new file mode 100644
index 0000000000..c81a7134ee
--- /dev/null
+++ b/package/tailscale/Config.in
@@ -0,0 +1,8 @@
+config BR2_PACKAGE_TAILSCALE
+	bool "tailscale"
+	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
+	select BR2_PACKAGE_HOST_GO
+	help
+	  The easiest, most secure way to use WireGuard and 2FA.
+
+	  https://github.com/tailscale/tailscale
diff --git a/package/tailscale/tailscale.hash b/package/tailscale/tailscale.hash
new file mode 100644
index 0000000000..882ff9d5d0
--- /dev/null
+++ b/package/tailscale/tailscale.hash
@@ -0,0 +1,3 @@
+# Locally computed
+sha256  a961aa07a3b0927cd647d6ae69fc212afbbdb0e4c47b8aeb181cb64360b7f143  tailscale-1.70.0-go2.tar.gz
+sha256  d1ee1c7947d4b2c1963ea214d5324f1d4c89f2f1d0f0224889b4dfb868dad725  LICENSE
diff --git a/package/tailscale/tailscale.mk b/package/tailscale/tailscale.mk
new file mode 100644
index 0000000000..882ab9b584
--- /dev/null
+++ b/package/tailscale/tailscale.mk
@@ -0,0 +1,23 @@
+################################################################################
+#
+# tailscale
+#
+################################################################################
+
+TAILSCALE_VERSION = 1.70.0
+TAILSCALE_SITE = $(call github,tailscale,tailscale,v$(TAILSCALE_VERSION))
+TAILSCALE_LICENSE = BSD-3-Clause
+TAILSCALE_LICENSE_FILES = LICENSE
+TAILSCALE_GOMOD = tailscale.com
+TAILSCALE_BUILD_TARGETS = cmd/tailscale cmd/tailscaled
+TAILSCALE_INSTALL_BINS = tailscale tailscaled
+TAILSCALE_DL_GOPROXY = https://proxy.golang.org
+TAILSCALE_LDFLAGS = \
+	-X tailscale.com/version.longStamp=$(TAILSCALE_VERSION) \
+	-X tailscale.com/version.shortStamp=$(TAILSCALE_VERSION)
+
+define TAILSCALE_LINUX_CONFIG_FIXUPS
+	$(call KCONFIG_ENABLE_OPT,CONFIG_TUN)
+endef
+
+$(eval $(golang-package))
-- 
2.34.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 2/2] package/tailscale: new package

[Christian Stewart via buildroot]

James,


On Mon, Aug 5, 2024 at 9:31 AM James Hilliard <james.hilliard1@gmail.com> wrote:
>
> Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
> ---
> Changes v3 -> v4:
>   - restore GOPROXY override
> Changes v2 -> v3:
>   - set required CONFIG_TUN kernel config option
> Changes v1 -> v2:
>   - don't set GOPROXY
> ---

> +TAILSCALE_DL_GOPROXY = https://proxy.golang.org

The issue with GOPROXY=direct is fixed as of tailscale main branch at
commit 227509547f0e719ef6ce2ec30d956fde518c0367

I requested that they spin a new release here:

https://github.com/tailscale/tailscale/issues/12859#issuecomment-2270177811

In the meantime we could merge this package pinned to the commit hash
I gave above.

The GOPROXY topic - I mentioned it here:

https://patchwork.ozlabs.org/project/buildroot/patch/20240805163133.4126564-1-james.hilliard1@gmail.com/#3357415

I think we should add HOST_GO_GOPROXY to allow the user to override
this, and definitely not override it per-package.

But this may not be necessary now that tailscale fixed their deps.

Best regards,
Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 2/2] package/tailscale: new package

[Christian Stewart via buildroot]

Hi all,

On Mon, Aug 5, 2024 at 6:14 PM Christian Stewart <christian@aperture.us> wrote:
> The issue with GOPROXY=direct is fixed as of tailscale main branch at
> commit 227509547f0e719ef6ce2ec30d956fde518c0367
>
> I requested that they spin a new release here:
>
> https://github.com/tailscale/tailscale/issues/12859#issuecomment-2270177811

>
> The GOPROXY topic - I mentioned it here:
>
> https://patchwork.ozlabs.org/project/buildroot/patch/20240805163133.4126564-1-james.hilliard1@gmail.com/#3357415
>
> I think we should add HOST_GO_GOPROXY to allow the user to override
> this, and definitely not override it per-package.
>
> But this may not be necessary now that tailscale fixed their deps.

Tailscale says they will release a new point release this week.

We can merge the tailscale package once that happens w/o changing anything.

That said I will try to submit a patch soon that switches the default
to use the usual Google GOPROXY, and adds a Config.in option to
override this.

Thanks & best regards,
Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 2/2] package/tailscale: new package

[Christian Stewart via buildroot]

Hi James, all,

On Mon, Aug 12, 2024 at 1:17 PM Christian Stewart <christian@aperture.us> wrote:
>
> Hi all,
>
> On Mon, Aug 5, 2024 at 6:14 PM Christian Stewart <christian@aperture.us> wrote:
> > The issue with GOPROXY=direct is fixed as of tailscale main branch at
> > commit 227509547f0e719ef6ce2ec30d956fde518c0367
> >
> > I requested that they spin a new release here:
> >
> > https://github.com/tailscale/tailscale/issues/12859#issuecomment-2270177811

> Tailscale says they will release a new point release this week.
>
> We can merge the tailscale package once that happens w/o changing anything.

Tailscale has released the new version which should work fine with
GOPROXY direct.

https://github.com/tailscale/tailscale/releases/tag/v1.72.0

James, please re-submit the package with the newer version.

Best regards,
Christian Stewart

> That said I will try to submit a patch soon that switches the default
> to use the usual Google GOPROXY, and adds a Config.in option to
> override this.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Christian Stewart via buildroot]

[-- Attachment #1.1: Type: text/plain, Size: 892 bytes --]

James,

On Mon, Aug 5, 2024, 9:31 AM James Hilliard <james.hilliard1@gmail.com>
wrote:

> Some packages like the upcoming tailscale package need to set GOPROXY
> to get the correct dependencies..
>

I still highly disagree with setting GOPROXY per package.

I have been working with the tailscale devs to fix the dependency issues.
We fixed most of it.

Additionally I'm waiting for the anchore/quill developers to get back to me
about removing some Git LFS files from their module that also break GOPROXY.

The decision to use the proxy or not (run by Google) should be up to the
user. If I want to download everything from Git, that should be fully
possible. (And usually is with GOPROXY direct)

I'll follow up with those GitHub issues today and see if I can get the ball
rolling on another tailscale release with fixed dependencies.

Thanks,
Christian Stewart

[-- Attachment #1.2: Type: text/html, Size: 1459 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Thomas Petazzoni via buildroot]

Hello Christian,

On Mon, 5 Aug 2024 10:34:44 -0700
Christian Stewart via buildroot <buildroot@buildroot.org> wrote:

> I still highly disagree with setting GOPROXY per package.
> 
> I have been working with the tailscale devs to fix the dependency issues.
> We fixed most of it.
> 
> Additionally I'm waiting for the anchore/quill developers to get back to me
> about removing some Git LFS files from their module that also break GOPROXY.
> 
> The decision to use the proxy or not (run by Google) should be up to the
> user. If I want to download everything from Git, that should be fully
> possible. (And usually is with GOPROXY direct)
> 
> I'll follow up with those GitHub issues today and see if I can get the ball
> rolling on another tailscale release with fixed dependencies.

Thanks a lot for your feedback. When seeing the new iteration from
James, my initial reaction was "I need the feedback from Christian on
this", and you had already given your feedback.

That being said: should we still have a way to patch/fix those kind of
issues on our side, until upstream fixes it? My understanding is that
this could be fixed on our side if we were able to apply a patch before
the Go vendoring happens. Is that correct?

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[James Hilliard]

On Mon, Aug 5, 2024 at 2:31 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello Christian,
>
> On Mon, 5 Aug 2024 10:34:44 -0700
> Christian Stewart via buildroot <buildroot@buildroot.org> wrote:
>
> > I still highly disagree with setting GOPROXY per package.
> >
> > I have been working with the tailscale devs to fix the dependency issues.
> > We fixed most of it.
> >
> > Additionally I'm waiting for the anchore/quill developers to get back to me
> > about removing some Git LFS files from their module that also break GOPROXY.
> >
> > The decision to use the proxy or not (run by Google) should be up to the
> > user. If I want to download everything from Git, that should be fully
> > possible. (And usually is with GOPROXY direct)

So this allows packages to choose the GOPROXY settings, previously
that was not possible as it was hard-coded as direct.

I don't really see much wrong with simply setting compatible GOPROXY
settings as required for a specific package. It's not really much different
from setting the download locations in general for packages IMO.

> >
> > I'll follow up with those GitHub issues today and see if I can get the ball
> > rolling on another tailscale release with fixed dependencies.
>
> Thanks a lot for your feedback. When seeing the new iteration from
> James, my initial reaction was "I need the feedback from Christian on
> this", and you had already given your feedback.
>
> That being said: should we still have a way to patch/fix those kind of
> issues on our side, until upstream fixes it? My understanding is that
> this could be fixed on our side if we were able to apply a patch before
> the Go vendoring happens. Is that correct?

Well I think since upstreams often expect the google goproxy then it's
easiest to just have us set that when required rather than try and patch
prior to vendoring.

>
> Thomas
> --
> Thomas Petazzoni, co-owner and CEO, Bootlin
> Embedded Linux and Kernel engineering and training
> https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Christian Stewart via buildroot]

Hi James, Thomas,

On Mon, Aug 5, 2024 at 3:01 PM James Hilliard <james.hilliard1@gmail.com> wrote:
> On Mon, Aug 5, 2024 at 2:31 PM Thomas Petazzoni
> <thomas.petazzoni@bootlin.com> wrote:
> > On Mon, 5 Aug 2024 10:34:44 -0700
> > Christian Stewart via buildroot <buildroot@buildroot.org> wrote:
> >
> > > I still highly disagree with setting GOPROXY per package.
> > >
> > > I have been working with the tailscale devs to fix the dependency issues.
> > > We fixed most of it.
> > >
> > > Additionally I'm waiting for the anchore/quill developers to get back to me
> > > about removing some Git LFS files from their module that also break GOPROXY.
> > >
> > > The decision to use the proxy or not (run by Google) should be up to the
> > > user. If I want to download everything from Git, that should be fully
> > > possible. (And usually is with GOPROXY direct)
>
> So this allows packages to choose the GOPROXY settings, previously
> that was not possible as it was hard-coded as direct.
>
> I don't really see much wrong with simply setting compatible GOPROXY
> settings as required for a specific package. It's not really much different
> from setting the download locations in general for packages IMO.
>
> > >
> > > I'll follow up with those GitHub issues today and see if I can get the ball
> > > rolling on another tailscale release with fixed dependencies.
> >
> > Thanks a lot for your feedback. When seeing the new iteration from
> > James, my initial reaction was "I need the feedback from Christian on
> > this", and you had already given your feedback.
> >
> > That being said: should we still have a way to patch/fix those kind of
> > issues on our side, until upstream fixes it? My understanding is that
> > this could be fixed on our side if we were able to apply a patch before
> > the Go vendoring happens. Is that correct?
>
> Well I think since upstreams often expect the google goproxy then it's
> easiest to just have us set that when required rather than try and patch
> prior to vendoring.

My suggestion is to either set GOPROXY=direct for all packages, or for
none of them.

We could default to setting GOPROXY to empty, which uses the default
Google proxy.

The best approach is likely to add a Config.in option:

Something like: HOST_GO_GOPROXY

Then we can default that to: "https://proxy.golang.org,direct"

Which is the default value in the Go source code.

I think this would be the best approach if we do want to enable
GOPROXY. This way, users that don't want to use the proxy can override
HOST_GO_GOPROXY to "direct"

We could keep it as "direct" once this issue is closed:
https://github.com/tailscale/tailscale/issues/12859

Best regards,
Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Thomas Petazzoni via buildroot]

On Mon, 5 Aug 2024 18:07:37 -0700
Christian Stewart <christian@aperture.us> wrote:

> My suggestion is to either set GOPROXY=direct for all packages, or for
> none of them.
> 
> We could default to setting GOPROXY to empty, which uses the default
> Google proxy.
> 
> The best approach is likely to add a Config.in option:
> 
> Something like: HOST_GO_GOPROXY
> 
> Then we can default that to: "https://proxy.golang.org,direct"
> 
> Which is the default value in the Go source code.
> 
> I think this would be the best approach if we do want to enable
> GOPROXY. This way, users that don't want to use the proxy can override
> HOST_GO_GOPROXY to "direct"

But that is independent of the current problem we're trying to solve
with tailscale.

> We could keep it as "direct" once this issue is closed:
> https://github.com/tailscale/tailscale/issues/12859

Still, I think it's a problem if we can't fix problems on our side and
that we have to wait for upstream to make a new release. For all other
problems, we can apply a patch in Buildroot that gets us going until
upstream fixes the problem and makes a release. I would like to have
the same ability to fix Go vendoring issues.

Could you explain to a Go ignorant like me how is this proxy thing
helping in solving this issue? Should a proxy just "speed up" the
download? Why does it exhibit a different behavior between a
GOPROXY=direct build and a GOPROXY=somethingelse ?

Thanks!

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[James Hilliard]

On Tue, Aug 6, 2024 at 8:50 AM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> On Mon, 5 Aug 2024 18:07:37 -0700
> Christian Stewart <christian@aperture.us> wrote:
>
> > My suggestion is to either set GOPROXY=direct for all packages, or for
> > none of them.
> >
> > We could default to setting GOPROXY to empty, which uses the default
> > Google proxy.

Why weren't we just using the default settings in the first place? I mean
if upstreams basically only test with default settings here then using
something different seems problematic in general.

> >
> > The best approach is likely to add a Config.in option:
> >
> > Something like: HOST_GO_GOPROXY
> >
> > Then we can default that to: "https://proxy.golang.org,direct"
> >
> > Which is the default value in the Go source code.
> >
> > I think this would be the best approach if we do want to enable
> > GOPROXY. This way, users that don't want to use the proxy can override
> > HOST_GO_GOPROXY to "direct"
>
> But that is independent of the current problem we're trying to solve
> with tailscale.
>
> > We could keep it as "direct" once this issue is closed:
> > https://github.com/tailscale/tailscale/issues/12859
>
> Still, I think it's a problem if we can't fix problems on our side and
> that we have to wait for upstream to make a new release. For all other
> problems, we can apply a patch in Buildroot that gets us going until
> upstream fixes the problem and makes a release. I would like to have
> the same ability to fix Go vendoring issues.

To me it sounds like just switching to the default GOPROXY settings
may be the best solution since that's typically what most projects use.

There's reports that GOPROXY=direct is kinda broken:
https://drewdevault.com/2021/08/06/goproxy-breaks-go.html

>
> Could you explain to a Go ignorant like me how is this proxy thing
> helping in solving this issue? Should a proxy just "speed up" the
> download? Why does it exhibit a different behavior between a
> GOPROXY=direct build and a GOPROXY=somethingelse ?

Presumably the proxy acts as a cache which prevents breakage if
the dependency direct source changes or something like that.

>
> Thanks!
>
> Thomas
> --
> Thomas Petazzoni, co-owner and CEO, Bootlin
> Embedded Linux and Kernel engineering and training
> https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Christian Stewart via buildroot]

[-- Attachment #1.1: Type: text/plain, Size: 3382 bytes --]

Thomas, James,

On Tue, Aug 6, 2024, 8:49 AM James Hilliard <james.hilliard1@gmail.com>
wrote:

> On Tue, Aug 6, 2024 at 8:50 AM Thomas Petazzoni
> <thomas.petazzoni@bootlin.com> wrote:
> >
> > On Mon, 5 Aug 2024 18:07:37 -0700
> > Christian Stewart <christian@aperture.us> wrote:
> >
> > > My suggestion is to either set GOPROXY=direct for all packages, or for
> > > none of them.
> > >
> > > We could default to setting GOPROXY to empty, which uses the default
> > > Google proxy.
>
> Why weren't we just using the default settings in the first place? I mean
> if upstreams basically only test with default settings here then using
> something different seems problematic in general.
>

> There's reports that GOPROXY=direct is kinda broken:
> https://drewdevault.com/2021/08/06/goproxy-breaks-go.html
>
> >
> > Could you explain to a Go ignorant like me how is this proxy thing
> > helping in solving this issue? Should a proxy just "speed up" the
> > download? Why does it exhibit a different behavior between a
> > GOPROXY=direct build and a GOPROXY=somethingelse ?
>
> Presumably the proxy acts as a cache which prevents breakage if
> the dependency direct source changes or something like that.
>

The reasoning for using GOPROXY=direct is as follows:

When we enable the proxy, the Go tool will download the sources for the
dependencies from Google.

This is OK in general, but in the context of buildroot we typically
download sources from their upstream releases or Git repositories.

GOPROXY is not broken.

That said, a lot of dependencies cause problems when they change what tag a
git release points to retroactively (rare but happens), or delete a older
commit from their repository (which happens extremely rarely).

The proxy smooths this over by always serving the same thing for a git tag
or commit hash.

The problem with this is, now we have something different being returned
for a dependency than what it says in go.mod.

The go.mod might say we are fetching foo at version 1.0.0, but in fact, if
you go to the repository for foo, and go to version 1.0.0, it could be a
completely different source tree from the one we get in the proxied
download.

This opens up the opportunity for bad actors to hide code in the proxied
version and then retroactively change the git tag to hide that bad code.

For this reason the default was set to always use Git to fetch the
dependencies for Buildroot as we would prefer to detect the mismatch and
deal with it up front, possibly falling back to fetching the .tar.gz from
our own mirror, rather than depend on this behavior from the Google proxy
to always be there and save us.

As you can see we have never had an issue with this setting until now, when
we have an actual broken dependency somewhere, and its when we are adding
the package, not retroactively, that the issue is found and solved.

This is my pitch for keeping GOPROXY=direct. If we want to use the Google
proxy with all the cavets I mentioned above, we could add the option
HOST_GO_GOPROXY and default it to the Google proxy. But personally I will
still always override this value to direct in my projects.

For now we can just include the latest commit hash of tailscale until they
make a new release and nothing else needs to be changed.

Thanks,
Christian Stewart

[-- Attachment #1.2: Type: text/html, Size: 4854 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[James Hilliard]

On Tue, Aug 6, 2024 at 10:14 AM Christian Stewart <christian@aperture.us> wrote:
>
> Thomas, James,
>
> On Tue, Aug 6, 2024, 8:49 AM James Hilliard <james.hilliard1@gmail.com> wrote:
>>
>> On Tue, Aug 6, 2024 at 8:50 AM Thomas Petazzoni
>> <thomas.petazzoni@bootlin.com> wrote:
>> >
>> > On Mon, 5 Aug 2024 18:07:37 -0700
>> > Christian Stewart <christian@aperture.us> wrote:
>> >
>> > > My suggestion is to either set GOPROXY=direct for all packages, or for
>> > > none of them.
>> > >
>> > > We could default to setting GOPROXY to empty, which uses the default
>> > > Google proxy.
>>
>> Why weren't we just using the default settings in the first place? I mean
>> if upstreams basically only test with default settings here then using
>> something different seems problematic in general.
>>
>>
>> There's reports that GOPROXY=direct is kinda broken:
>> https://drewdevault.com/2021/08/06/goproxy-breaks-go.html
>>
>> >
>> > Could you explain to a Go ignorant like me how is this proxy thing
>> > helping in solving this issue? Should a proxy just "speed up" the
>> > download? Why does it exhibit a different behavior between a
>> > GOPROXY=direct build and a GOPROXY=somethingelse ?
>>
>> Presumably the proxy acts as a cache which prevents breakage if
>> the dependency direct source changes or something like that.
>
>
> The reasoning for using GOPROXY=direct is as follows:
>
> When we enable the proxy, the Go tool will download the sources for the dependencies from Google.
>
> This is OK in general, but in the context of buildroot we typically download sources from their upstream releases or Git repositories.

For cargo which I think is the most similar to how we handle dependencies
for golang we vendor dependencies from https://crates.io/ downloads AFAIU.

So I don't think it's normal for us to download vendored sources from their
upstream locations directly.

>
> GOPROXY is not broken.
>
> That said, a lot of dependencies cause problems when they change what tag a git release points to retroactively (rare but happens), or delete a older commit from their repository (which happens extremely rarely).
>
> The proxy smooths this over by always serving the same thing for a git tag or commit hash.

So upstreams seem to expect this caching behavior.

>
> The problem with this is, now we have something different being returned for a dependency than what it says in go.mod.
>
> The go.mod might say we are fetching foo at version 1.0.0, but in fact, if you go to the repository for foo, and go to version 1.0.0, it could be a completely different source tree from the one we get in the proxied download.
>
> This opens up the opportunity for bad actors to hide code in the proxied version and then retroactively change the git tag to hide that bad code.

I mean, the dependencies are all locked to the upstream hashes either way.

>
> For this reason the default was set to always use Git to fetch the dependencies for Buildroot as we would prefer to detect the mismatch and deal with it up front, possibly falling back to fetching the .tar.gz from our own mirror, rather than depend on this behavior from the Google proxy to always be there and save us.

I don't really see anything wrong with using google if that's what all
upstreams seem to expect.

>
> As you can see we have never had an issue with this setting until now, when we have an actual broken dependency somewhere, and its when we are adding the package, not retroactively, that the issue is found and solved.

We also don't seem to have a lot of golang packages.

>
> This is my pitch for keeping GOPROXY=direct. If we want to use the Google proxy with all the cavets I mentioned above, we could add the option HOST_GO_GOPROXY and default it to the Google proxy. But personally I will still always override this value to direct in my projects.

I'm not seeing much difference when the upstream project is using hash
based dependency lock files however, like as long as the hashes match
we know we have the sources specified by the upstream. If the upstream
itself can't be trusted then that's a whole different issue IMO and the
GOPROXY settings won't make much of a difference there.

>
> For now we can just include the latest commit hash of tailscale until they make a new release and nothing else needs to be changed.

IMO switching to the default GOPROXY settings is probably best.

I'm somewhat wary of using a commit hash for something like tailscale
which interfaces with an external service.

>
> Thanks,
> Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Thomas Petazzoni via buildroot]

Hello,

On Tue, 6 Aug 2024 09:13:50 -0700
Christian Stewart <christian@aperture.us> wrote:


> The reasoning for using GOPROXY=direct is as follows:
> 
> When we enable the proxy, the Go tool will download the sources for the
> dependencies from Google.
> 
> This is OK in general, but in the context of buildroot we typically
> download sources from their upstream releases or Git repositories.
> 
> GOPROXY is not broken.
> 
> That said, a lot of dependencies cause problems when they change what tag a
> git release points to retroactively (rare but happens), or delete a older
> commit from their repository (which happens extremely rarely).

Crap, those projects do that? They change a tag, or remove an older
commit from their repo? Yikes for reproducibility :-/

> The proxy smooths this over by always serving the same thing for a git tag
> or commit hash.
> 
> The problem with this is, now we have something different being returned
> for a dependency than what it says in go.mod.
> 
> The go.mod might say we are fetching foo at version 1.0.0, but in fact, if
> you go to the repository for foo, and go to version 1.0.0, it could be a
> completely different source tree from the one we get in the proxied
> download.

And that obviously sucks, and using a proxy doesn't seem like a great
workaround.

> This opens up the opportunity for bad actors to hide code in the proxied
> version and then retroactively change the git tag to hide that bad code.
> 
> For this reason the default was set to always use Git to fetch the
> dependencies for Buildroot as we would prefer to detect the mismatch and
> deal with it up front, possibly falling back to fetching the .tar.gz from
> our own mirror, rather than depend on this behavior from the Google proxy
> to always be there and save us.
> 
> As you can see we have never had an issue with this setting until now, when
> we have an actual broken dependency somewhere, and its when we are adding
> the package, not retroactively, that the issue is found and solved.

I'm not sure to follow you on this. Why would this only happen when we
add the package? If we use GOPROXY=direct, and 3 months after we've
added the package some upstream dependency decides to change its tag or
remove some commits, then it will start failing (of course,
sources.buildroot.net will have a backup, but sources.buildroot.net is
not meant to really be used in "normal" circumstances).

> This is my pitch for keeping GOPROXY=direct. If we want to use the Google
> proxy with all the cavets I mentioned above, we could add the option
> HOST_GO_GOPROXY and default it to the Google proxy. But personally I will
> still always override this value to direct in my projects.
> 
> For now we can just include the latest commit hash of tailscale until they
> make a new release and nothing else needs to be changed.

Don't understand this last part. What is your proposal to fix the
tailscale situation?

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Christian Stewart via buildroot]

[-- Attachment #1.1: Type: text/plain, Size: 1136 bytes --]

Thomas,

On Tue, Aug 6, 2024, 9:57 AM Thomas Petazzoni <thomas.petazzoni@bootlin.com>
wrote:

> > For now we can just include the latest commit hash of tailscale until
> they
> > make a new release and nothing else needs to be changed.
>
> Don't understand this last part. What is your proposal to fix the
> tailscale situation?
>

They fixed the issue upstream and now the main branch of tailscale works
fine with GOPROXY=direct. See:

https://github.com/tailscale/tailscale/issues/12859#issuecomment-2239157563

It is fine to use the main branch of tailscale here even if it's an
external service as tailscale keeps compatibility between releases - I even
have many machines running quite out of date tailscale clients which still
connect and run perfectly. The same can be said for forwards compatibility.

All that I ask with regards to GOPROXY is that:

- we do not override it per package,
- if we do decide to change the default to use Google, please make it a
configurable option (like BR2_MIRROR) so that I (and other users) can
override it back to be direct.

Best regards,
Christian Stewart

[-- Attachment #1.2: Type: text/html, Size: 1819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[James Hilliard]

On Tue, Aug 6, 2024 at 12:53 PM Christian Stewart <christian@aperture.us> wrote:
>
> Thomas,
>
> On Tue, Aug 6, 2024, 9:57 AM Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote:
>>
>> > For now we can just include the latest commit hash of tailscale until they
>> > make a new release and nothing else needs to be changed.
>>
>> Don't understand this last part. What is your proposal to fix the
>> tailscale situation?
>
>
> They fixed the issue upstream and now the main branch of tailscale works fine with GOPROXY=direct. See:

I think trying to get packages to make GOPROXY=direct work is getting
rather annoying, while trying to add another package(netbird) I once again
ran into this exact same issue.

>
> https://github.com/tailscale/tailscale/issues/12859#issuecomment-2239157563
>
> It is fine to use the main branch of tailscale here even if it's an external service as tailscale keeps compatibility between releases - I even have many machines running quite out of date tailscale clients which still connect and run perfectly. The same can be said for forwards compatibility.
>
> All that I ask with regards to GOPROXY is that:
>
> - we do not override it per package,

I'm a bit confused why we would not want to override it per package if
the breakage is per package.

> - if we do decide to change the default to use Google, please make it a configurable option (like BR2_MIRROR) so that I (and other users) can override it back to be direct.

If a package is known to not build with GOPROXY=direct then why would
one want to override the default GOPROXY rather than a package specific
GOPROXY?

>
> Best regards,
> Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Christian Stewart via buildroot]

[-- Attachment #1.1: Type: text/plain, Size: 2612 bytes --]

James,

On Wed, Oct 23, 2024, 8:21 AM James Hilliard <james.hilliard1@gmail.com>
wrote:

> On Tue, Aug 6, 2024 at 12:53 PM Christian Stewart <christian@aperture.us>
> wrote:
> >
> > Thomas,
> >
> > On Tue, Aug 6, 2024, 9:57 AM Thomas Petazzoni <
> thomas.petazzoni@bootlin.com> wrote:
> >>
> >> > For now we can just include the latest commit hash of tailscale until
> they
> >> > make a new release and nothing else needs to be changed.
> >>
> >> Don't understand this last part. What is your proposal to fix the
> >> tailscale situation?
> >
> >
> > They fixed the issue upstream and now the main branch of tailscale works
> fine with GOPROXY=direct. See:
>
> I think trying to get packages to make GOPROXY=direct work is getting
> rather annoying, while trying to add another package(netbird) I once again
> ran into this exact same issue.
>
> >
> >
> https://github.com/tailscale/tailscale/issues/12859#issuecomment-2239157563
> >
> > It is fine to use the main branch of tailscale here even if it's an
> external service as tailscale keeps compatibility between releases - I even
> have many machines running quite out of date tailscale clients which still
> connect and run perfectly. The same can be said for forwards compatibility.
> >
> > All that I ask with regards to GOPROXY is that:
> >
> > - we do not override it per package,
>
> I'm a bit confused why we would not want to override it per package if
> the breakage is per package.
>
> > - if we do decide to change the default to use Google, please make it a
> configurable option (like BR2_MIRROR) so that I (and other users) can
> override it back to be direct.
>
> If a package is known to not build with GOPROXY=direct then why would
> one want to override the default GOPROXY rather than a package specific
> GOPROXY?
>


It is a proxy, not a download URL. The intent of the entire Go module
system is to be based on source control under the hood. We should be able
to check out the git repositories of the underlying source without having
to use the Google proxy.

If we make a habit of ignoring these hash mismatches it will become very
easy for people to stuff code into a release on GitHub and then change the
tag to something else later, leaving behind the old record in GOPROXY and
tricking maintainers into reviewing the wrong code on the modified GitHub
tag.

There must be some reason why your package is having this issue. With
anchore quill package we found that it was git lfs with test files causing
the problem.

Best regards,
Christian Stewart

>

[-- Attachment #1.2: Type: text/html, Size: 3713 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[James Hilliard]

On Wed, Oct 23, 2024 at 10:35 AM Christian Stewart
<christian@aperture.us> wrote:
>
> James,
>
> On Wed, Oct 23, 2024, 8:21 AM James Hilliard <james.hilliard1@gmail.com> wrote:
>>
>> On Tue, Aug 6, 2024 at 12:53 PM Christian Stewart <christian@aperture.us> wrote:
>> >
>> > Thomas,
>> >
>> > On Tue, Aug 6, 2024, 9:57 AM Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote:
>> >>
>> >> > For now we can just include the latest commit hash of tailscale until they
>> >> > make a new release and nothing else needs to be changed.
>> >>
>> >> Don't understand this last part. What is your proposal to fix the
>> >> tailscale situation?
>> >
>> >
>> > They fixed the issue upstream and now the main branch of tailscale works fine with GOPROXY=direct. See:
>>
>> I think trying to get packages to make GOPROXY=direct work is getting
>> rather annoying, while trying to add another package(netbird) I once again
>> ran into this exact same issue.
>>
>> >
>> > https://github.com/tailscale/tailscale/issues/12859#issuecomment-2239157563
>> >
>> > It is fine to use the main branch of tailscale here even if it's an external service as tailscale keeps compatibility between releases - I even have many machines running quite out of date tailscale clients which still connect and run perfectly. The same can be said for forwards compatibility.
>> >
>> > All that I ask with regards to GOPROXY is that:
>> >
>> > - we do not override it per package,
>>
>> I'm a bit confused why we would not want to override it per package if
>> the breakage is per package.
>>
>> > - if we do decide to change the default to use Google, please make it a configurable option (like BR2_MIRROR) so that I (and other users) can override it back to be direct.
>>
>> If a package is known to not build with GOPROXY=direct then why would
>> one want to override the default GOPROXY rather than a package specific
>> GOPROXY?
>
>
>
> It is a proxy, not a download URL. The intent of the entire Go module system is to be based on source control under the hood. We should be able to check out the git repositories of the underlying source without having to use the Google proxy.

From my understanding it's a caching proxy essentially designed to
ensure that the build can be reproduced if the git repositories are ever
broken in some way down the line.

>
> If we make a habit of ignoring these hash mismatches it will become very easy for people to stuff code into a release on GitHub and then change the tag to something else later, leaving behind the old record in GOPROXY and tricking maintainers into reviewing the wrong code on the modified GitHub tag.

I mean, if someone wants to reproduce the build and the git repo doesn't
function when used directly for whatever reason then reviewing from the
cache would appear to be the only available option to get the correct
source code.

>
> There must be some reason why your package is having this issue. With anchore quill package we found that it was git lfs with test files causing the problem.
>
> Best regards,
> Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Thomas Petazzoni via buildroot]

Hello,

On Wed, 23 Oct 2024 09:35:16 -0700
Christian Stewart via buildroot <buildroot@buildroot.org> wrote:

> It is a proxy, not a download URL. The intent of the entire Go module
> system is to be based on source control under the hood. We should be able
> to check out the git repositories of the underlying source without having
> to use the Google proxy.
> 
> If we make a habit of ignoring these hash mismatches it will become very
> easy for people to stuff code into a release on GitHub and then change the
> tag to something else later, leaving behind the old record in GOPROXY and
> tricking maintainers into reviewing the wrong code on the modified GitHub
> tag.
> 
> There must be some reason why your package is having this issue. With
> anchore quill package we found that it was git lfs with test files causing
> the problem.

I continue to agree with Christian. If using caching proxies is needed
to avoid downloading the "real" upstream because the real upstream has
changed/is broken, things are severely wrong.

Instead, I would recommend to patch netbird to resolve this issue. I
know this requires patches to be applied *before* the vendoring step,
but that it something that anyway would be useful to have. So I would
suggest to find a way to do that. Perhaps some special patch extension,
like package/<foo>/*.vendoring-patch or some other better name, that
would be applied before the vendoring step takes place?

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[James Hilliard]

On Wed, Oct 23, 2024 at 1:25 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello,
>
> On Wed, 23 Oct 2024 09:35:16 -0700
> Christian Stewart via buildroot <buildroot@buildroot.org> wrote:
>
> > It is a proxy, not a download URL. The intent of the entire Go module
> > system is to be based on source control under the hood. We should be able
> > to check out the git repositories of the underlying source without having
> > to use the Google proxy.
> >
> > If we make a habit of ignoring these hash mismatches it will become very
> > easy for people to stuff code into a release on GitHub and then change the
> > tag to something else later, leaving behind the old record in GOPROXY and
> > tricking maintainers into reviewing the wrong code on the modified GitHub
> > tag.
> >
> > There must be some reason why your package is having this issue. With
> > anchore quill package we found that it was git lfs with test files causing
> > the problem.
>
> I continue to agree with Christian. If using caching proxies is needed
> to avoid downloading the "real" upstream because the real upstream has
> changed/is broken, things are severely wrong.

This is only for dependencies of the project, the issue is they can often
point to unstable dependency repositories AFAIU and upstreams just
tend to rely on this caching proxy.

>
> Instead, I would recommend to patch netbird to resolve this issue. I
> know this requires patches to be applied *before* the vendoring step,
> but that it something that anyway would be useful to have. So I would
> suggest to find a way to do that. Perhaps some special patch extension,
> like package/<foo>/*.vendoring-patch or some other better name, that
> would be applied before the vendoring step takes place?

I'm not actually trying to modify the dependencies specified at all, if
the only download location for a specified dependency I need is on
https://proxy.golang.org then I'm not sure how I would patch the
package to use that instead.

Also then what do we do when a package dependency inevitably
breaks again due to the download location becoming broken, do
we just keep patching the package over and over again every time
one of its dependencies breaks?

>
> Thomas
> --
> Thomas Petazzoni, co-owner and CEO, Bootlin
> Embedded Linux and Kernel engineering and training
> https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Christian Stewart via buildroot]

[-- Attachment #1.1: Type: text/plain, Size: 3297 bytes --]

James,

On Wed, Oct 23, 2024, 12:49 PM James Hilliard <james.hilliard1@gmail.com>
wrote:

> On Wed, Oct 23, 2024 at 1:25 PM Thomas Petazzoni
> <thomas.petazzoni@bootlin.com> wrote:
> >
> > Hello,
> >
> > On Wed, 23 Oct 2024 09:35:16 -0700
> > Christian Stewart via buildroot <buildroot@buildroot.org> wrote:
> >
> > > It is a proxy, not a download URL. The intent of the entire Go module
> > > system is to be based on source control under the hood. We should be
> able
> > > to check out the git repositories of the underlying source without
> having
> > > to use the Google proxy.
> > >
> > > If we make a habit of ignoring these hash mismatches it will become
> very
> > > easy for people to stuff code into a release on GitHub and then change
> the
> > > tag to something else later, leaving behind the old record in GOPROXY
> and
> > > tricking maintainers into reviewing the wrong code on the modified
> GitHub
> > > tag.
> > >
> > > There must be some reason why your package is having this issue. With
> > > anchore quill package we found that it was git lfs with test files
> causing
> > > the problem.
> >
> > I continue to agree with Christian. If using caching proxies is needed
> > to avoid downloading the "real" upstream because the real upstream has
> > changed/is broken, things are severely wrong.
>
> This is only for dependencies of the project, the issue is they can often
> point to unstable dependency repositories AFAIU and upstreams just
> tend to rely on this caching proxy.
>

That really doesn't make any sense because Go modules versions and go.sum
point to not only a specific release tag, but a specific hash of that tag,
as well as (if not using a release) a commit hash and date.

That system is designed to ensure integrity.

The repository being "unstable dependencies" doesn't give any explanation
for why the checksums are failing.


>
> > Instead, I would recommend to patch netbird to resolve this issue. I
> > know this requires patches to be applied *before* the vendoring step,
> > but that it something that anyway would be useful to have. So I would
> > suggest to find a way to do that. Perhaps some special patch extension,
> > like package/<foo>/*.vendoring-patch or some other better name, that
> > would be applied before the vendoring step takes place?
>
> I'm not actually trying to modify the dependencies specified at all, if
> the only download location for a specified dependency I need is on
> https://proxy.golang.org then I'm not sure how I would patch the
> package to use that instead.
>

You should report the issue to the upstream repository like I did with
tailscale, they will update the dependency to a newer version that doesn't
have the checksums mismatch

Or feel free to post the issue here and we can try to figure out why the
checksums are failing.


Also then what do we do when a package dependency inevitably
> breaks again due to the download location becoming broken, do
> we just keep patching the package over and over again every time
> one of its dependencies breaks?
>

Why is the location breaking at all?...

For this purpose we mirror to the buildroot mirror with all the
dependencies vendored.

Best regards,
Christian

[-- Attachment #1.2: Type: text/html, Size: 4876 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Thomas Petazzoni via buildroot]

On Wed, 23 Oct 2024 13:36:50 -0700
Christian Stewart <christian@aperture.us> wrote:

> > I'm not actually trying to modify the dependencies specified at all, if
> > the only download location for a specified dependency I need is on
> > https://proxy.golang.org then I'm not sure how I would patch the
> > package to use that instead.
> >  
> 
> You should report the issue to the upstream repository like I did with
> tailscale, they will update the dependency to a newer version that doesn't
> have the checksums mismatch

I agree... but we should have a way in Buildroot to fix the issue even
if upstream is not responsive or cooperative. So the answer to a
problem cannot only be "let's wait until upstream fixes it".

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[James Hilliard]

On Wed, Oct 23, 2024 at 2:37 PM Christian Stewart <christian@aperture.us> wrote:
>
> James,
>
> On Wed, Oct 23, 2024, 12:49 PM James Hilliard <james.hilliard1@gmail.com> wrote:
>>
>> On Wed, Oct 23, 2024 at 1:25 PM Thomas Petazzoni
>> <thomas.petazzoni@bootlin.com> wrote:
>> >
>> > Hello,
>> >
>> > On Wed, 23 Oct 2024 09:35:16 -0700
>> > Christian Stewart via buildroot <buildroot@buildroot.org> wrote:
>> >
>> > > It is a proxy, not a download URL. The intent of the entire Go module
>> > > system is to be based on source control under the hood. We should be able
>> > > to check out the git repositories of the underlying source without having
>> > > to use the Google proxy.
>> > >
>> > > If we make a habit of ignoring these hash mismatches it will become very
>> > > easy for people to stuff code into a release on GitHub and then change the
>> > > tag to something else later, leaving behind the old record in GOPROXY and
>> > > tricking maintainers into reviewing the wrong code on the modified GitHub
>> > > tag.
>> > >
>> > > There must be some reason why your package is having this issue. With
>> > > anchore quill package we found that it was git lfs with test files causing
>> > > the problem.
>> >
>> > I continue to agree with Christian. If using caching proxies is needed
>> > to avoid downloading the "real" upstream because the real upstream has
>> > changed/is broken, things are severely wrong.
>>
>> This is only for dependencies of the project, the issue is they can often
>> point to unstable dependency repositories AFAIU and upstreams just
>> tend to rely on this caching proxy.
>
>
> That really doesn't make any sense because Go modules versions and go.sum point to not only a specific release tag, but a specific hash of that tag, as well as (if not using a release) a commit hash and date.
>
> That system is designed to ensure integrity.

Which makes how we get the dependencies largely unimportant as long
as they pass the integrity checks AFAIU.

It seems another advantage to using https://proxy.golang.org is that it
significantly speeds up downloads.

>
> The repository being "unstable dependencies" doesn't give any explanation for why the checksums are failing.
>
>
>> >
>> > Instead, I would recommend to patch netbird to resolve this issue. I
>> > know this requires patches to be applied *before* the vendoring step,
>> > but that it something that anyway would be useful to have. So I would
>> > suggest to find a way to do that. Perhaps some special patch extension,
>> > like package/<foo>/*.vendoring-patch or some other better name, that
>> > would be applied before the vendoring step takes place?
>>
>> I'm not actually trying to modify the dependencies specified at all, if
>> the only download location for a specified dependency I need is on
>> https://proxy.golang.org then I'm not sure how I would patch the
>> package to use that instead.
>
>
> You should report the issue to the upstream repository like I did with tailscale, they will update the dependency to a newer version that doesn't have the checksums mismatch

Which is tricky in this case since I'm not using the latest release due to
it requiring golang 1.23.0 while we only have 1.22.8.

>
> Or feel free to post the issue here and we can try to figure out why the checksums are failing.

This is the error:
go: gvisor.dev/gvisor@v0.0.0-20230927004350-cbd86285d259: invalid
version: unknown revision cbd86285d259

>
>
>> Also then what do we do when a package dependency inevitably
>> breaks again due to the download location becoming broken, do
>> we just keep patching the package over and over again every time
>> one of its dependencies breaks?
>
>
> Why is the location breaking at all?...
>
> For this purpose we mirror to the buildroot mirror with all the dependencies vendored.

Yes...however we try not to rely on that as a failing download could
AFAIU trigger autobuilder failures.

>
> Best regards,
> Christian
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Christian Stewart via buildroot]

Hi James,

On Wed, Oct 23, 2024 at 2:51 PM James Hilliard
<james.hilliard1@gmail.com> wrote:
> Which makes how we get the dependencies largely unimportant as long
> as they pass the integrity checks AFAIU.
>
> It seems another advantage to using https://proxy.golang.org is that it
> significantly speeds up downloads.

Right, this was why I was proposing allowing setting GOPROXY globally
with a configuration option.

If people want to use it (should be the default, even) then
BR2_PACKAGE_GO_GOPROXY is set to https://proxy.golang.org.

> Which is tricky in this case since I'm not using the latest release due to
> it requiring golang 1.23.0 while we only have 1.22.8.

Let's update Go to 1.23.x?

> > Or feel free to post the issue here and we can try to figure out why the checksums are failing.
>
> This is the error:
> go: gvisor.dev/gvisor@v0.0.0-20230927004350-cbd86285d259: invalid
> version: unknown revision cbd86285d259

That indeed looks like that commit was deleted from the gvisor repository.

Issue: https://github.com/google/gvisor/issues/11085

Reproduce:

git clone -b v0.29.1 https://github.com/netbirdio/netbird
cd netbird
export GOPROXY=direct
go mod tidy

I get the same error as you.

> >> Also then what do we do when a package dependency inevitably
> >> breaks again due to the download location becoming broken, do
> >> we just keep patching the package over and over again every time
> >> one of its dependencies breaks?
> >
> >
> > Why is the location breaking at all?...
> >
> > For this purpose we mirror to the buildroot mirror with all the dependencies vendored.
>
> Yes...however we try not to rely on that as a failing download could
> AFAIU trigger autobuilder failures.

This should not happen assuming the commit that was referenced in the
version string was either included in one of the branches of the
repository or in the history of one of the git tags.

I guess the commit they referenced in netbird was not on any of the branches

It's odd because I do find the commit on GitHub:

https://github.com/google/gvisor/commit/cbd86285d2592a1d1d3915b75a14d54d3b241bb1

Best regards,
Christian
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Christian Stewart via buildroot]

Hi James,

On Fri, Oct 25, 2024 at 1:11 PM Christian Stewart <christian@aperture.us> wrote:
> On Wed, Oct 23, 2024 at 2:51 PM James Hilliard
> > Which is tricky in this case since I'm not using the latest release due to
> > it requiring golang 1.23.0 while we only have 1.22.8.
>
> Let's update Go to 1.23.x?

I sent a patch for this:

https://patchwork.ozlabs.org/project/buildroot/patch/20241026045421.44823-1-christian@aperture.us/

> > > Or feel free to post the issue here and we can try to figure out why the checksums are failing.
> >
> > This is the error:
> > go: gvisor.dev/gvisor@v0.0.0-20230927004350-cbd86285d259: invalid
> > version: unknown revision cbd86285d259
>
> That indeed looks like that commit was deleted from the gvisor repository.
>
> Issue: https://github.com/google/gvisor/issues/11085

According to the discussion in that issue, the commit was deleted from
gvisor because it contained large binary files. They were trimming
down the repo size.

I recommend instead re-submitting your latest netbird patch after the
Go 1.23.x patch is merged.

Best regards,
Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Christian Stewart via buildroot]

Hi James,

On Fri, Oct 25, 2024 at 9:58 PM Christian Stewart <christian@aperture.us> wrote:
>
> Hi James,
>
> On Fri, Oct 25, 2024 at 1:11 PM Christian Stewart <christian@aperture.us> wrote:
> > On Wed, Oct 23, 2024 at 2:51 PM James Hilliard
> > > Which is tricky in this case since I'm not using the latest release due to
> > > it requiring golang 1.23.0 while we only have 1.22.8.
> >
> > Let's update Go to 1.23.x?
>
> I sent a patch for this:
>
> https://patchwork.ozlabs.org/project/buildroot/patch/20241026045421.44823-1-christian@aperture.us/
>
> > > > Or feel free to post the issue here and we can try to figure out why the checksums are failing.
> > >
> > > This is the error:
> > > go: gvisor.dev/gvisor@v0.0.0-20230927004350-cbd86285d259: invalid
> > > version: unknown revision cbd86285d259
> >
> > That indeed looks like that commit was deleted from the gvisor repository.
> >
> > Issue: https://github.com/google/gvisor/issues/11085
>
> According to the discussion in that issue, the commit was deleted from
> gvisor because it contained large binary files. They were trimming
> down the repo size.

https://github.com/netbirdio/netbird/pull/2789

There is now a PR against netbird to update gvisor and fix this.

They referred to my original issue I filed against gvisor, so this is
good evidence that reporting this type of issue to upstream repos
actually does lead to a prompt fix.

Thanks & best regards,
Chrisitan Stewart
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Christian Stewart via buildroot]

[-- Attachment #1.1: Type: text/plain, Size: 4254 bytes --]

Hi all,

I wanted to follow up on this old topic of GOPROXY.

Below I mention a potential security issue where someone uploads a
malicious package version to the proxy and then changes the git tag to
remove the malware. In this attack mode the maintainers might check the git
tag, see no malware, and add a package with that tag.

The problem is that Go fetches from GOPROXY and silently ignores the
mismatch.

This was the reason why I have always lobbied to keep the default as
GOPROXY=direct in buildroot, as this avoids the case where the git repo
mismatches the proxy without visibility at the git level.

This attack has been used in the wild now:

https://socket.dev/blog/malicious-package-exploits-go-module-proxy-caching-for-persistence

See below for my reasoning

On Tue, Aug 6, 2024, 9:13 AM Christian Stewart <christian@aperture.us>
wrote:

> Thomas, James,
>
> On Tue, Aug 6, 2024, 8:49 AM James Hilliard <james.hilliard1@gmail.com>
> wrote:
>
>> On Tue, Aug 6, 2024 at 8:50 AM Thomas Petazzoni
>> <thomas.petazzoni@bootlin.com> wrote:
>> >
>> > On Mon, 5 Aug 2024 18:07:37 -0700
>> > Christian Stewart <christian@aperture.us> wrote:
>> >
>> > > My suggestion is to either set GOPROXY=direct for all packages, or for
>> > > none of them.
>> > >
>> > > We could default to setting GOPROXY to empty, which uses the default
>> > > Google proxy.
>>
>> Why weren't we just using the default settings in the first place? I mean
>> if upstreams basically only test with default settings here then using
>> something different seems problematic in general.
>>
>
>> There's reports that GOPROXY=direct is kinda broken:
>> https://drewdevault.com/2021/08/06/goproxy-breaks-go.html
>>
>> >
>> > Could you explain to a Go ignorant like me how is this proxy thing
>> > helping in solving this issue? Should a proxy just "speed up" the
>> > download? Why does it exhibit a different behavior between a
>> > GOPROXY=direct build and a GOPROXY=somethingelse ?
>>
>> Presumably the proxy acts as a cache which prevents breakage if
>> the dependency direct source changes or something like that.
>>
>
> The reasoning for using GOPROXY=direct is as follows:
>
> When we enable the proxy, the Go tool will download the sources for the
> dependencies from Google.
>
> This is OK in general, but in the context of buildroot we typically
> download sources from their upstream releases or Git repositories.
>
> GOPROXY is not broken.
>
> That said, a lot of dependencies cause problems when they change what tag
> a git release points to retroactively (rare but happens), or delete a older
> commit from their repository (which happens extremely rarely).
>
> The proxy smooths this over by always serving the same thing for a git tag
> or commit hash.
>
> The problem with this is, now we have something different being returned
> for a dependency than what it says in go.mod.
>
> The go.mod might say we are fetching foo at version 1.0.0, but in fact, if
> you go to the repository for foo, and go to version 1.0.0, it could be a
> completely different source tree from the one we get in the proxied
> download.
>
> This opens up the opportunity for bad actors to hide code in the proxied
> version and then retroactively change the git tag to hide that bad code.
>
> For this reason the default was set to always use Git to fetch the
> dependencies for Buildroot as we would prefer to detect the mismatch and
> deal with it up front, possibly falling back to fetching the .tar.gz from
> our own mirror, rather than depend on this behavior from the Google proxy
> to always be there and save us.
>
> As you can see we have never had an issue with this setting until now,
> when we have an actual broken dependency somewhere, and its when we are
> adding the package, not retroactively, that the issue is found and solved.
>
> This is my pitch for keeping GOPROXY=direct. If we want to use the Google
> proxy with all the cavets I mentioned above, we could add the option
> HOST_GO_GOPROXY and default it to the Google proxy. But personally I will
> still always override this value to direct in my projects.
>


Best regards,
Christian Stewart

>

[-- Attachment #1.2: Type: text/html, Size: 6568 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[James Hilliard]

On Wed, Feb 5, 2025 at 2:51 PM Christian Stewart <christian@aperture.us> wrote:
>
> Hi all,
>
> I wanted to follow up on this old topic of GOPROXY.
>
> Below I mention a potential security issue where someone uploads a malicious package version to the proxy and then changes the git tag to remove the malware. In this attack mode the maintainers might check the git tag, see no malware, and add a package with that tag.
>
> The problem is that Go fetches from GOPROXY and silently ignores the mismatch.

As it says in the article this is how it's designed to work:

"Go modules are intentionally immutable once published and fetched by
the proxy, ensuring that every user pulling a tagged version (e.g.,
v1.2.3) receives the exact same bits every time, and preventing silent
changes or overwrites after publication. This immutability underpins
Go’s reproducible builds, helping guarantee that build outputs always
match the associated source code."

>
> This was the reason why I have always lobbied to keep the default as GOPROXY=direct in buildroot, as this avoids the case where the git repo mismatches the proxy without visibility at the git level.

As it says in the article:

"While immutability can be abused by malicious actors — allowing
harmful code to persist once cached — it remains a net security
benefit. Since the Go module system is functioning as intended, and
this is not a security vulnerability, there is no patch or switch to
turn off immutability; instead, developers must recognize that once a
malicious version is published, it will remain malicious in the
cache."

So it's effectively expected that go packages use the cache as
GOPROXY=direct is an unusual non-default configuration that simply
breaks builds for many non-malicious reasons(dead urls for example).
Distributions like buildroot are not typically auditing package
dependencies in a way that would catch these sort of attacks either
way, auditing dependencies would potentially be something that
upstream package maintainers would want to do but we mostly care that
we are building the package with the correct dependencies specified by
the upstream, using the GOPROXY cache is therefore correct for our use
case. If we have a go package in buildroot that we know won't build
with GOPROXY=direct then I don't see how we can possibly default to
using GOPROXY=direct as that would obviously result in failed builds.

>
> This attack has been used in the wild now:
>
> https://socket.dev/blog/malicious-package-exploits-go-module-proxy-caching-for-persistence
>
> See below for my reasoning
>
> On Tue, Aug 6, 2024, 9:13 AM Christian Stewart <christian@aperture.us> wrote:
>>
>> Thomas, James,
>>
>> On Tue, Aug 6, 2024, 8:49 AM James Hilliard <james.hilliard1@gmail.com> wrote:
>>>
>>> On Tue, Aug 6, 2024 at 8:50 AM Thomas Petazzoni
>>> <thomas.petazzoni@bootlin.com> wrote:
>>> >
>>> > On Mon, 5 Aug 2024 18:07:37 -0700
>>> > Christian Stewart <christian@aperture.us> wrote:
>>> >
>>> > > My suggestion is to either set GOPROXY=direct for all packages, or for
>>> > > none of them.
>>> > >
>>> > > We could default to setting GOPROXY to empty, which uses the default
>>> > > Google proxy.
>>>
>>> Why weren't we just using the default settings in the first place? I mean
>>> if upstreams basically only test with default settings here then using
>>> something different seems problematic in general.
>>>
>>>
>>> There's reports that GOPROXY=direct is kinda broken:
>>> https://drewdevault.com/2021/08/06/goproxy-breaks-go.html
>>>
>>> >
>>> > Could you explain to a Go ignorant like me how is this proxy thing
>>> > helping in solving this issue? Should a proxy just "speed up" the
>>> > download? Why does it exhibit a different behavior between a
>>> > GOPROXY=direct build and a GOPROXY=somethingelse ?
>>>
>>> Presumably the proxy acts as a cache which prevents breakage if
>>> the dependency direct source changes or something like that.
>>
>>
>> The reasoning for using GOPROXY=direct is as follows:
>>
>> When we enable the proxy, the Go tool will download the sources for the dependencies from Google.
>>
>> This is OK in general, but in the context of buildroot we typically download sources from their upstream releases or Git repositories.
>>
>> GOPROXY is not broken.
>>
>> That said, a lot of dependencies cause problems when they change what tag a git release points to retroactively (rare but happens), or delete a older commit from their repository (which happens extremely rarely).
>>
>> The proxy smooths this over by always serving the same thing for a git tag or commit hash.
>>
>> The problem with this is, now we have something different being returned for a dependency than what it says in go.mod.
>>
>> The go.mod might say we are fetching foo at version 1.0.0, but in fact, if you go to the repository for foo, and go to version 1.0.0, it could be a completely different source tree from the one we get in the proxied download.
>>
>> This opens up the opportunity for bad actors to hide code in the proxied version and then retroactively change the git tag to hide that bad code.
>>
>> For this reason the default was set to always use Git to fetch the dependencies for Buildroot as we would prefer to detect the mismatch and deal with it up front, possibly falling back to fetching the .tar.gz from our own mirror, rather than depend on this behavior from the Google proxy to always be there and save us.
>>
>> As you can see we have never had an issue with this setting until now, when we have an actual broken dependency somewhere, and its when we are adding the package, not retroactively, that the issue is found and solved.
>>
>> This is my pitch for keeping GOPROXY=direct. If we want to use the Google proxy with all the cavets I mentioned above, we could add the option HOST_GO_GOPROXY and default it to the Google proxy. But personally I will still always override this value to direct in my projects.
>
>
>
> Best regards,
> Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Arnout Vandecappelle via buildroot]

On 05/02/2025 15:02, James Hilliard wrote:
> On Wed, Feb 5, 2025 at 2:51 PM Christian Stewart <christian@aperture.us> wrote:
>>
>> Hi all,
>>
>> I wanted to follow up on this old topic of GOPROXY.
>>
>> Below I mention a potential security issue where someone uploads a malicious package version to the proxy and then changes the git tag to remove the malware. In this attack mode the maintainers might check the git tag, see no malware, and add a package with that tag.
>>
>> The problem is that Go fetches from GOPROXY and silently ignores the mismatch.
> 
> As it says in the article this is how it's designed to work:
> 
> "Go modules are intentionally immutable once published and fetched by
> the proxy, ensuring that every user pulling a tagged version (e.g.,
> v1.2.3) receives the exact same bits every time, and preventing silent
> changes or overwrites after publication. This immutability underpins
> Go’s reproducible builds, helping guarantee that build outputs always
> match the associated source code."

  ... and thanks to our GOPROXY=direct, it works perfectly: when you try to 
build the package, you get a hash mismatch error instead of silently using a 
(possibly malicious) conflicting version from the proxy. Thus, you can 
investigate the issue and fix it properly.

  The fix should normally be to replace the version number in go.mod with a sha1 
that does exist upstream (as close as possible to the original release, probably 
you also want to compare it with the version that you got through GOPROXY). And 
update go.sum accordingly of course.

  Currently Buildroot doesn't allow that, we would need to do some pre-vendoring 
manipulation of the source. There is a patch for that [1] but it needs more work.

  Regards,
  Arnout

[1] 
https://patchwork.ozlabs.org/project/buildroot/patch/20250203175316.1658490-1-guillaume.chaye@zeetim.com/



>> This was the reason why I have always lobbied to keep the default as GOPROXY=direct in buildroot, as this avoids the case where the git repo mismatches the proxy without visibility at the git level.
> 
> As it says in the article:
> 
> "While immutability can be abused by malicious actors — allowing
> harmful code to persist once cached — it remains a net security
> benefit. Since the Go module system is functioning as intended, and
> this is not a security vulnerability, there is no patch or switch to
> turn off immutability; instead, developers must recognize that once a
> malicious version is published, it will remain malicious in the
> cache."
> 
> So it's effectively expected that go packages use the cache as
> GOPROXY=direct is an unusual non-default configuration that simply
> breaks builds for many non-malicious reasons(dead urls for example).
> Distributions like buildroot are not typically auditing package
> dependencies in a way that would catch these sort of attacks either
> way, auditing dependencies would potentially be something that
> upstream package maintainers would want to do but we mostly care that
> we are building the package with the correct dependencies specified by
> the upstream, using the GOPROXY cache is therefore correct for our use
> case. If we have a go package in buildroot that we know won't build
> with GOPROXY=direct then I don't see how we can possibly default to
> using GOPROXY=direct as that would obviously result in failed builds.
> 
>>
>> This attack has been used in the wild now:
>>
>> https://socket.dev/blog/malicious-package-exploits-go-module-proxy-caching-for-persistence
>>
>> See below for my reasoning
>>
>> On Tue, Aug 6, 2024, 9:13 AM Christian Stewart <christian@aperture.us> wrote:
>>>
>>> Thomas, James,
>>>
>>> On Tue, Aug 6, 2024, 8:49 AM James Hilliard <james.hilliard1@gmail.com> wrote:
>>>>
>>>> On Tue, Aug 6, 2024 at 8:50 AM Thomas Petazzoni
>>>> <thomas.petazzoni@bootlin.com> wrote:
>>>>>
>>>>> On Mon, 5 Aug 2024 18:07:37 -0700
>>>>> Christian Stewart <christian@aperture.us> wrote:
>>>>>
>>>>>> My suggestion is to either set GOPROXY=direct for all packages, or for
>>>>>> none of them.
>>>>>>
>>>>>> We could default to setting GOPROXY to empty, which uses the default
>>>>>> Google proxy.
>>>>
>>>> Why weren't we just using the default settings in the first place? I mean
>>>> if upstreams basically only test with default settings here then using
>>>> something different seems problematic in general.
>>>>
>>>>
>>>> There's reports that GOPROXY=direct is kinda broken:
>>>> https://drewdevault.com/2021/08/06/goproxy-breaks-go.html
>>>>
>>>>>
>>>>> Could you explain to a Go ignorant like me how is this proxy thing
>>>>> helping in solving this issue? Should a proxy just "speed up" the
>>>>> download? Why does it exhibit a different behavior between a
>>>>> GOPROXY=direct build and a GOPROXY=somethingelse ?
>>>>
>>>> Presumably the proxy acts as a cache which prevents breakage if
>>>> the dependency direct source changes or something like that.
>>>
>>>
>>> The reasoning for using GOPROXY=direct is as follows:
>>>
>>> When we enable the proxy, the Go tool will download the sources for the dependencies from Google.
>>>
>>> This is OK in general, but in the context of buildroot we typically download sources from their upstream releases or Git repositories.
>>>
>>> GOPROXY is not broken.
>>>
>>> That said, a lot of dependencies cause problems when they change what tag a git release points to retroactively (rare but happens), or delete a older commit from their repository (which happens extremely rarely).
>>>
>>> The proxy smooths this over by always serving the same thing for a git tag or commit hash.
>>>
>>> The problem with this is, now we have something different being returned for a dependency than what it says in go.mod.
>>>
>>> The go.mod might say we are fetching foo at version 1.0.0, but in fact, if you go to the repository for foo, and go to version 1.0.0, it could be a completely different source tree from the one we get in the proxied download.
>>>
>>> This opens up the opportunity for bad actors to hide code in the proxied version and then retroactively change the git tag to hide that bad code.
>>>
>>> For this reason the default was set to always use Git to fetch the dependencies for Buildroot as we would prefer to detect the mismatch and deal with it up front, possibly falling back to fetching the .tar.gz from our own mirror, rather than depend on this behavior from the Google proxy to always be there and save us.
>>>
>>> As you can see we have never had an issue with this setting until now, when we have an actual broken dependency somewhere, and its when we are adding the package, not retroactively, that the issue is found and solved.
>>>
>>> This is my pitch for keeping GOPROXY=direct. If we want to use the Google proxy with all the cavets I mentioned above, we could add the option HOST_GO_GOPROXY and default it to the Google proxy. But personally I will still always override this value to direct in my projects.
>>
>>
>>
>> Best regards,
>> Christian Stewart
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[James Hilliard]

On Wed, Feb 5, 2025 at 5:45 PM Arnout Vandecappelle <arnout@mind.be> wrote:
>
>
>
> On 05/02/2025 15:02, James Hilliard wrote:
> > On Wed, Feb 5, 2025 at 2:51 PM Christian Stewart <christian@aperture.us> wrote:
> >>
> >> Hi all,
> >>
> >> I wanted to follow up on this old topic of GOPROXY.
> >>
> >> Below I mention a potential security issue where someone uploads a malicious package version to the proxy and then changes the git tag to remove the malware. In this attack mode the maintainers might check the git tag, see no malware, and add a package with that tag.
> >>
> >> The problem is that Go fetches from GOPROXY and silently ignores the mismatch.
> >
> > As it says in the article this is how it's designed to work:
> >
> > "Go modules are intentionally immutable once published and fetched by
> > the proxy, ensuring that every user pulling a tagged version (e.g.,
> > v1.2.3) receives the exact same bits every time, and preventing silent
> > changes or overwrites after publication. This immutability underpins
> > Go’s reproducible builds, helping guarantee that build outputs always
> > match the associated source code."
>
>   ... and thanks to our GOPROXY=direct, it works perfectly: when you try to
> build the package, you get a hash mismatch error instead of silently using a
> (possibly malicious) conflicting version from the proxy. Thus, you can
> investigate the issue and fix it properly.

I'm confused, we want to build packages with their upstream specified
dependencies, as long as the hash matches then the dependencies are exactly
what was specified upstream, I really don't see why we want to care about the
non-cached download locations breaking/changing if the go ecosystem at large
does not care about that. I really see no reasonable justification for
intentionally
breaking golang's immutability guarantees by not using the cache.

Using dependencies that don't match the upstream exactly will make issue
reporting potentially problematic due to using different dependencies and
could introduce security vulnerabilities if we accidentally replace a
non-backdoored dependency with one that is backdoored.

Keep in mind anyone checking upstream packages for vulnerabilities would
likely be using the cached dependency versions in most cases, for example
if the upstream uses some sort of runtime behavior vulnerability scanner
they would almost certainly be using it against the package dependencies
exactly as they are specified in the upstream project and not those available
as a direct download from the original locations.

>
>   The fix should normally be to replace the version number in go.mod with a sha1
> that does exist upstream (as close as possible to the original release, probably
> you also want to compare it with the version that you got through GOPROXY). And
> update go.sum accordingly of course.

This is kinda a maintainability nightmare, dependencies will break randomly for
various reasons with GOPROXY=direct due to unreliable dependency download
locations/urls.

For example netbird is still broken with GOPROXY=direct due to url breakage
issues which have not had fixes merged(surprise surprise maintainers don't
care or test with GOPROXY=direct seemingly ever as it's not the default):
https://github.com/netbirdio/netbird/pull/3240

This will be especially problematic for stable releases as go packages often
have a ton of dependencies, any one of which can break a package build
when not using the cache thus requiring patches that are unlikely to be accepted
upstream due to package versions in stable release often not being maintained
upstream anymore.

>
>   Currently Buildroot doesn't allow that, we would need to do some pre-vendoring
> manipulation of the source. There is a patch for that [1] but it needs more work.
>
>   Regards,
>   Arnout
>
> [1]
> https://patchwork.ozlabs.org/project/buildroot/patch/20250203175316.1658490-1-guillaume.chaye@zeetim.com/
>
>
>
> >> This was the reason why I have always lobbied to keep the default as GOPROXY=direct in buildroot, as this avoids the case where the git repo mismatches the proxy without visibility at the git level.
> >
> > As it says in the article:
> >
> > "While immutability can be abused by malicious actors — allowing
> > harmful code to persist once cached — it remains a net security
> > benefit. Since the Go module system is functioning as intended, and
> > this is not a security vulnerability, there is no patch or switch to
> > turn off immutability; instead, developers must recognize that once a
> > malicious version is published, it will remain malicious in the
> > cache."
> >
> > So it's effectively expected that go packages use the cache as
> > GOPROXY=direct is an unusual non-default configuration that simply
> > breaks builds for many non-malicious reasons(dead urls for example).
> > Distributions like buildroot are not typically auditing package
> > dependencies in a way that would catch these sort of attacks either
> > way, auditing dependencies would potentially be something that
> > upstream package maintainers would want to do but we mostly care that
> > we are building the package with the correct dependencies specified by
> > the upstream, using the GOPROXY cache is therefore correct for our use
> > case. If we have a go package in buildroot that we know won't build
> > with GOPROXY=direct then I don't see how we can possibly default to
> > using GOPROXY=direct as that would obviously result in failed builds.
> >
> >>
> >> This attack has been used in the wild now:
> >>
> >> https://socket.dev/blog/malicious-package-exploits-go-module-proxy-caching-for-persistence
> >>
> >> See below for my reasoning
> >>
> >> On Tue, Aug 6, 2024, 9:13 AM Christian Stewart <christian@aperture.us> wrote:
> >>>
> >>> Thomas, James,
> >>>
> >>> On Tue, Aug 6, 2024, 8:49 AM James Hilliard <james.hilliard1@gmail.com> wrote:
> >>>>
> >>>> On Tue, Aug 6, 2024 at 8:50 AM Thomas Petazzoni
> >>>> <thomas.petazzoni@bootlin.com> wrote:
> >>>>>
> >>>>> On Mon, 5 Aug 2024 18:07:37 -0700
> >>>>> Christian Stewart <christian@aperture.us> wrote:
> >>>>>
> >>>>>> My suggestion is to either set GOPROXY=direct for all packages, or for
> >>>>>> none of them.
> >>>>>>
> >>>>>> We could default to setting GOPROXY to empty, which uses the default
> >>>>>> Google proxy.
> >>>>
> >>>> Why weren't we just using the default settings in the first place? I mean
> >>>> if upstreams basically only test with default settings here then using
> >>>> something different seems problematic in general.
> >>>>
> >>>>
> >>>> There's reports that GOPROXY=direct is kinda broken:
> >>>> https://drewdevault.com/2021/08/06/goproxy-breaks-go.html
> >>>>
> >>>>>
> >>>>> Could you explain to a Go ignorant like me how is this proxy thing
> >>>>> helping in solving this issue? Should a proxy just "speed up" the
> >>>>> download? Why does it exhibit a different behavior between a
> >>>>> GOPROXY=direct build and a GOPROXY=somethingelse ?
> >>>>
> >>>> Presumably the proxy acts as a cache which prevents breakage if
> >>>> the dependency direct source changes or something like that.
> >>>
> >>>
> >>> The reasoning for using GOPROXY=direct is as follows:
> >>>
> >>> When we enable the proxy, the Go tool will download the sources for the dependencies from Google.
> >>>
> >>> This is OK in general, but in the context of buildroot we typically download sources from their upstream releases or Git repositories.
> >>>
> >>> GOPROXY is not broken.
> >>>
> >>> That said, a lot of dependencies cause problems when they change what tag a git release points to retroactively (rare but happens), or delete a older commit from their repository (which happens extremely rarely).
> >>>
> >>> The proxy smooths this over by always serving the same thing for a git tag or commit hash.
> >>>
> >>> The problem with this is, now we have something different being returned for a dependency than what it says in go.mod.
> >>>
> >>> The go.mod might say we are fetching foo at version 1.0.0, but in fact, if you go to the repository for foo, and go to version 1.0.0, it could be a completely different source tree from the one we get in the proxied download.
> >>>
> >>> This opens up the opportunity for bad actors to hide code in the proxied version and then retroactively change the git tag to hide that bad code.
> >>>
> >>> For this reason the default was set to always use Git to fetch the dependencies for Buildroot as we would prefer to detect the mismatch and deal with it up front, possibly falling back to fetching the .tar.gz from our own mirror, rather than depend on this behavior from the Google proxy to always be there and save us.
> >>>
> >>> As you can see we have never had an issue with this setting until now, when we have an actual broken dependency somewhere, and its when we are adding the package, not retroactively, that the issue is found and solved.
> >>>
> >>> This is my pitch for keeping GOPROXY=direct. If we want to use the Google proxy with all the cavets I mentioned above, we could add the option HOST_GO_GOPROXY and default it to the Google proxy. But personally I will still always override this value to direct in my projects.
> >>
> >>
> >>
> >> Best regards,
> >> Christian Stewart
> > _______________________________________________
> > buildroot mailing list
> > buildroot@buildroot.org
> > https://lists.buildroot.org/mailman/listinfo/buildroot
>
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Christian Stewart via buildroot]

[-- Attachment #1.1: Type: text/plain, Size: 2629 bytes --]

James,

On Wed, Feb 5, 2025, 10:02 AM James Hilliard <james.hilliard1@gmail.com>
wrote:

I agree with your baseline assertion that the security of these packages
comes from the go.sum hashes.


This is kinda a maintainability nightmare, dependencies will break randomly
> for
> various reasons with GOPROXY=direct due to unreliable dependency download
> locations/urls.
>

Do you have any examples of cases where this has happened?

If it does happen, we can mirror the package .tar.gz in the buildroot
mirror. Have we ever needed to do this thus far?


For example netbird is still broken with GOPROXY=direct due to url breakage
> issues which have not had fixes merged(surprise surprise maintainers don't
> care or test with GOPROXY=direct seemingly ever as it's not the default):
> https://github.com/netbirdio/netbird/pull/3240


I don't agree that your 1 week old PR not being merged yet is a suitable
indicator that maintainers don't care about their dependencies matching
what is in Git.

This will be especially problematic for stable releases as go packages often
> have a ton of dependencies, any one of which can break a package build
> when not using the cache thus requiring patches that are unlikely to be
> accepted
> upstream due to package versions in stable release often not being
> maintained
> upstream anymore.
>

The remedy for a dependency not being available anymore via Git is to
mirror the entire package .tar.gz in the buildroot mirror, not to submit
patches.

The submitting patches is part of the initial process of adding or updating
a Go package in buildroot. Consider it due diligence to make sure that any
mismatches between the proxy and the upstream Git source are addressed.

I agree with your logic that:

1) The developers of go projects like Docker or Netbird have pinned
dependencies in go.sum which we can reasonably trust.
2) buildroot maintainers don't typically exhaustively audit all
dependencies.

What I disagree with is;

1) It is impossible to avoid adding a Google hosted service as a dependency
for building Go packages in buildroot.
2) It is not feasible to, at the moment a package is merged into buildroot,
expect new packages or package versions to build cleanly without the proxy
mentioned in 1).

Should a upstream Git source go missing, we can mirror that package .tar.gz
in the buildroot mirror. We can easily reproduce that .tar.gz using the
Google proxy as needed, by setting GOPROXY to the Google proxy and running
"make pkg-extract"

Hopefully this clarifies a bit.

Best regards,
Christian

[-- Attachment #1.2: Type: text/html, Size: 4238 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[James Hilliard]

On Wed, Feb 5, 2025 at 10:01 PM Christian Stewart <christian@aperture.us> wrote:
>
> James,
>
> On Wed, Feb 5, 2025, 10:02 AM James Hilliard <james.hilliard1@gmail.com> wrote:
>
> I agree with your baseline assertion that the security of these packages comes from the go.sum hashes.
>
>
>> This is kinda a maintainability nightmare, dependencies will break randomly for
>> various reasons with GOPROXY=direct due to unreliable dependency download
>> locations/urls.
>
>
> Do you have any examples of cases where this has happened?

Look at my netbird PR, literally a case of 2 urls going invalid at some point.

> If it does happen, we can mirror the package .tar.gz in the buildroot mirror. Have we ever needed to do this thus far?

We don't really have a lot of golang packages yet, but based on the breakage
I see in packages like netbird it seems like an inevitable issue at any
meaningful scale.

I think we don't want the buildroot mirror to be the primary download location
either.

>> For example netbird is still broken with GOPROXY=direct due to url breakage
>> issues which have not had fixes merged(surprise surprise maintainers don't
>> care or test with GOPROXY=direct seemingly ever as it's not the default):
>> https://github.com/netbirdio/netbird/pull/3240
>
>
> I don't agree that your 1 week old PR not being merged yet is a suitable indicator that maintainers don't care about their dependencies matching what is in Git.

Well that and the fact that GOPROXY=direct was broken in the first place
is a pretty strong indicator upstream projects don't care about it much.

>> This will be especially problematic for stable releases as go packages often
>> have a ton of dependencies, any one of which can break a package build
>> when not using the cache thus requiring patches that are unlikely to be accepted
>> upstream due to package versions in stable release often not being maintained
>> upstream anymore.
>
>
> The remedy for a dependency not being available anymore via Git is to mirror the entire package .tar.gz in the buildroot mirror, not to submit patches.

Or we could choose the much simpler and more reliable option of simply
not defaulting to GOPROXY=direct.

> The submitting patches is part of the initial process of adding or updating a Go package in buildroot. Consider it due diligence to make sure that any mismatches between the proxy and the upstream Git source are addressed.

What's the purpose of this? What actual real world benefits have we gained
from fixing these broken url/download issues upstream as opposed to simply
not using GOPROXY=direct like the rest of the golang ecosystem?

> I agree with your logic that:
>
> 1) The developers of go projects like Docker or Netbird have pinned dependencies in go.sum which we can reasonably trust.
> 2) buildroot maintainers don't typically exhaustively audit all dependencies.
>
> What I disagree with is;
>
> 1) It is impossible to avoid adding a Google hosted service as a dependency for building Go packages in buildroot.

What I'm saying is that it's impossible to avoid the goproxy cache without
relying on buildroot itself mirroring packages with their dependencies in a
generalized manner long term.

Also those with out of tree golang packages in a BR2_EXTERNAL one can
not rely on buildroot's mirror which is a further issue when it comes to
requiring GOPROXY=direct.

> 2) It is not feasible to, at the moment a package is merged into buildroot, expect new packages or package versions to build cleanly without the proxy mentioned in 1).

I mean, I suppose buildroot can in theory introduce many arbitrary and
entirely unnecessary merge requirements for golang packages, what I'm
saying is that there's no justification for defaulting to or requiring
GOPROXY=direct support for merging golang packages when the golang
ecosystem doesn't require GOPROXY=direct support normally.

> Should a upstream Git source go missing, we can mirror that package .tar.gz in the buildroot mirror. We can easily reproduce that .tar.gz using the Google proxy as needed, by setting GOPROXY to the Google proxy and running "make pkg-extract"

Yes it is possible to introduce arbitrary roadblocks and then work around
them when one gets bitten down the line, but I see no good reason for
actually introducing these arbitrary roadblocks in the first place.

There are virtually zero benefits to using GOPROXY=direct outside of
some auditing use cases(which aren't really buildroot applicable) from
what I can tell. I'm sure it will also make downloads slower as well.

There's only downsides to using GOPROXY=direct in buildroot from
what I can tell, I can't even think of a single real world benefit.

>
> Hopefully this clarifies a bit.
>
> Best regards,
> Christian
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Christian Stewart via buildroot]

Hi James,

On Wed, Feb 5, 2025 at 12:38 PM James Hilliard
<james.hilliard1@gmail.com> wrote:
> > Do you have any examples of cases where this has happened?
>
> Look at my netbird PR, literally a case of 2 urls going invalid at some point.
>
> > If it does happen, we can mirror the package .tar.gz in the buildroot mirror. Have we ever needed to do this thus far?
>
> We don't really have a lot of golang packages yet, but based on the breakage
> I see in packages like netbird it seems like an inevitable issue at any
> meaningful scale.
>
> >> For example netbird is still broken with GOPROXY=direct due to url breakage
> >> issues which have not had fixes merged(surprise surprise maintainers don't
> >> care or test with GOPROXY=direct seemingly ever as it's not the default):
> >> https://github.com/netbirdio/netbird/pull/3240
> >
> > I don't agree that your 1 week old PR not being merged yet is a suitable indicator that maintainers don't care about their dependencies matching what is in Git.
>
> Well that and the fact that GOPROXY=direct was broken in the first place
> is a pretty strong indicator upstream projects don't care about it much.

They have replied to your PR requesting an alternative strategy to
fixing the issue:

https://github.com/netbirdio/netbird/pull/3240#issuecomment-2637956537

Here are the new PRs implementing what they requested:

- https://github.com/netbirdio/netbird/pull/3287
- https://github.com/netbirdio/netbird/pull/3288

> What's the purpose of this? What actual real world benefits have we gained
> from fixing these broken url/download issues upstream as opposed to simply
> not using GOPROXY=direct like the rest of the golang ecosystem?

You may note that the default of GOPROXY is:

netbird % go env GOPROXY
https://proxy.golang.org,direct

Note that "direct" is the second in the list, the fallback if the
proxy does not work.

If the default were the proxy without the "direct" part it would be
true that "the rest of the golang ecosystem" does not use
GOPROXY=direct.

> > I agree with your logic that:
> >
> > 1) The developers of go projects like Docker or Netbird have pinned dependencies in go.sum which we can reasonably trust.
> > 2) buildroot maintainers don't typically exhaustively audit all dependencies.
> >
> > What I disagree with is;
> >
> > 1) It is impossible to avoid adding a Google hosted service as a dependency for building Go packages in buildroot.
>
> What I'm saying is that it's impossible to avoid the goproxy cache without
> relying on buildroot itself mirroring packages with their dependencies in a
> generalized manner long term.

Do you have any examples of these breakages so far? We use a number of
Go packages in Buildroot extensively already today, like docker-engine
and docker-cli.

Given the tradeoffs we have discussed between:

- "always require the Google proxy" and
- "fix mismatches at package add/update time, mirror if dependencies
are unavailable long-term"

There is insufficient evidence that the mirroring part is such a
maintenance burden that we must go with option 1 here.

> There's only downsides to using GOPROXY=direct in buildroot from
> what I can tell, I can't even think of a single real world benefit.

Upsides:

- No dependency on a third-party proxy service
- Ability to build with GOPROXY=direct for those that require it
(otherwise Buildroot breaks for them on default)
- If go.mod says "foo/bar at version v1.0.0" then we can be sure the
Git tag v1.0.0 matches the go.sum.

Downsides:

- Slower fetch (git pull instead of proxy.golang.org)
- We need to mirror packages if their dependencies go away (most are
on GitHub, though)

When requesting that upstream projects fix the build with
GOPROXY=direct it helps to mention why in the PR description.

Best regards,
Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Arnout Vandecappelle via buildroot]

  Hi Christian,

  Thank you for summarizing the upsides and downsides of GOPROXY=direct!

  I have a few additions though.

On 05/02/2025 22:22, Christian Stewart wrote:
[snip]
> Upsides:
> 
> - No dependency on a third-party proxy service
> - Ability to build with GOPROXY=direct for those that require it
> (otherwise Buildroot breaks for them on default)

  Do we actually have examples of packages that break with the default proxy?

> - If go.mod says "foo/bar at version v1.0.0" then we can be sure the
> Git tag v1.0.0 matches the go.sum.

  Additional upside:

- No risk of failures due to GOPROXY setting in user's environment.

  And final upside (but I don't think it's really relevant for Buildroot):

- We detect issues with upstream go.mod.


> Downsides:
> 
> - Slower fetch (git pull instead of proxy.golang.org)
> - We need to mirror packages if their dependencies go away (most are
> on GitHub, though)

  It's not actually that simple though. It should rather be:

- If any upstream disappears, they will need to fall back on sources.buildroot.net.
- If an upstream is already gone when a package is submitted, we need to do 
pre-vendoring patching to fix it, and we don't currently have infrastructure for 
pre-vendoring patching.
- There is a (small?) chance that it's impossible to find back an upstream that 
corresponds to (is identical to) the original version that disappeared.


  I'm still on the fence on what the best approach is... What I do think based 
on the discussion is that we should use the same GOPROXY for everything - there 
really should be no reason to do it differently for different packages.

  Right now, I'm actually leaning towards removing the GOPROXY=direct - but then 
we should still set it explicitly. We should probably even allow the proxy to be 
configured. We should definitely not rely on whatever the user set in their 
environment.

  The problem with doing it the right way (i.e. keeping GOPROXY=direct) is that 
it adds a lot of work for us:
- when adding a package that has a broken go.mod, we have to fix it;
- we currently can't even fix it until the fix is merged upstream because we 
don't have the pre-vendoring patchng infrastructure.

  A lot will depend on how often and why packages break with the default GOPROXY.


  Regards,
  Arnout


> When requesting that upstream projects fix the build with
> GOPROXY=direct it helps to mention why in the PR description.


_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[James Hilliard]

On Wed, Feb 5, 2025 at 11:54 PM Arnout Vandecappelle <arnout@mind.be> wrote:
>
>   Hi Christian,
>
>   Thank you for summarizing the upsides and downsides of GOPROXY=direct!
>
>   I have a few additions though.
>
> On 05/02/2025 22:22, Christian Stewart wrote:
> [snip]
> > Upsides:
> >
> > - No dependency on a third-party proxy service
> > - Ability to build with GOPROXY=direct for those that require it
> > (otherwise Buildroot breaks for them on default)
>
>   Do we actually have examples of packages that break with the default proxy?

AFAIU the default proxy will not break packages due to the fallback.

>
> > - If go.mod says "foo/bar at version v1.0.0" then we can be sure the
> > Git tag v1.0.0 matches the go.sum.
>
>   Additional upside:
>
> - No risk of failures due to GOPROXY setting in user's environment.
>
>   And final upside (but I don't think it's really relevant for Buildroot):
>
> - We detect issues with upstream go.mod.
>
>
> > Downsides:
> >
> > - Slower fetch (git pull instead of proxy.golang.org)
> > - We need to mirror packages if their dependencies go away (most are
> > on GitHub, though)
>
>   It's not actually that simple though. It should rather be:
>
> - If any upstream disappears, they will need to fall back on sources.buildroot.net.
> - If an upstream is already gone when a package is submitted, we need to do
> pre-vendoring patching to fix it, and we don't currently have infrastructure for
> pre-vendoring patching.
> - There is a (small?) chance that it's impossible to find back an upstream that
> corresponds to (is identical to) the original version that disappeared.

Technically you can just use the caching proxy but I really see no reason
to not just use that in the first place since that's what it was designed for.

>
>
>   I'm still on the fence on what the best approach is... What I do think based
> on the discussion is that we should use the same GOPROXY for everything - there
> really should be no reason to do it differently for different packages.

We should use the golang default of GOPROXY=https://proxy.golang.org,direct
here IMO.

>
>   Right now, I'm actually leaning towards removing the GOPROXY=direct - but then
> we should still set it explicitly. We should probably even allow the proxy to be
> configured. We should definitely not rely on whatever the user set in their
> environment.

Yeah, I was explicitly setting the default GOPROXY in this patch:
https://patchwork.ozlabs.org/project/buildroot/patch/20241028161507.186161-1-james.hilliard1@gmail.com/

>
>   The problem with doing it the right way (i.e. keeping GOPROXY=direct) is that
> it adds a lot of work for us:

I disagree that requiring GOPROXY=direct is a more correct approach,
mostly since it's not the default the golang ecosystem uses.

> - when adding a package that has a broken go.mod, we have to fix it;
> - we currently can't even fix it until the fix is merged upstream because we
> don't have the pre-vendoring patchng infrastructure.

Yeah, it's super messy to workaround the GOPROXY=direct brokenness.

>   A lot will depend on how often and why packages break with the default GOPROXY.

They won't ever break AFAIU since there's still a fallback, so I don't think
that's even an issue to worry about.

>
>
>   Regards,
>   Arnout
>
>
> > When requesting that upstream projects fix the build with
> > GOPROXY=direct it helps to mention why in the PR description.
>
>
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Christian Stewart via buildroot]

Hi Arnout,

On Wed, Feb 5, 2025 at 1:53 PM Arnout Vandecappelle <arnout@mind.be> wrote:
> On 05/02/2025 22:22, Christian Stewart wrote:
> [snip]
> > Upsides:
> >
> > - No dependency on a third-party proxy service
> > - Ability to build with GOPROXY=direct for those that require it
> > (otherwise Buildroot breaks for them on default)
>
>   Do we actually have examples of packages that break with the default proxy?

I should have been more clear here. I meant, as a user, if I set
GOPROXY=direct and run a Buildroot build, the build would fail 100% of
the time if we merge packages that have mismatches between Git and
GOPROXY.

>   It's not actually that simple though. It should rather be:
>
> - If any upstream disappears, they will need to fall back on sources.buildroot.net.
> - If an upstream is already gone when a package is submitted, we need to do
> pre-vendoring patching to fix it, and we don't currently have infrastructure for
> pre-vendoring patching.

Or fix upstream.

> - There is a (small?) chance that it's impossible to find back an upstream that
> corresponds to (is identical to) the original version that disappeared.

We can get the .tar.gz for the package via setting GOPROXY to the
default and running "make pkg-extract" then mirror that.

>   I'm still on the fence on what the best approach is... What I do think based
> on the discussion is that we should use the same GOPROXY for everything - there
> really should be no reason to do it differently for different packages.

Agreed.

I do agree that having GOPROXY enable the Google proxy on default is a
good thing.

However I strongly believe that we should enforce GOPROXY=direct at
least works when merging the package initially or updating to a new
version.

>   Right now, I'm actually leaning towards removing the GOPROXY=direct - but then
> we should still set it explicitly. We should probably even allow the proxy to be
> configured. We should definitely not rely on whatever the user set in their
> environment.
>
>   The problem with doing it the right way (i.e. keeping GOPROXY=direct) is that
> it adds a lot of work for us:
> - when adding a package that has a broken go.mod, we have to fix it;
> - we currently can't even fix it until the fix is merged upstream because we
> don't have the pre-vendoring patchng infrastructure.

That's true, at the moment we have to wait for upstream to fix it.

>   A lot will depend on how often and why packages break with the default GOPROXY.

We have so far seen that with "netbird" (they are merging the PRs to
fix it currently) and "tailscale" (they merged a PR to fix it quite
quickly).

At most there should be a maybe 1 week delay for upstream to respond.

netbird took particularly long since we have not let them know why we
need this to work.

Thanks,
Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[James Hilliard]

On Wed, Feb 5, 2025 at 11:23 PM Christian Stewart <christian@aperture.us> wrote:
>
> Hi James,
>
> On Wed, Feb 5, 2025 at 12:38 PM James Hilliard
> <james.hilliard1@gmail.com> wrote:
> > > Do you have any examples of cases where this has happened?
> >
> > Look at my netbird PR, literally a case of 2 urls going invalid at some point.
> >
> > > If it does happen, we can mirror the package .tar.gz in the buildroot mirror. Have we ever needed to do this thus far?
> >
> > We don't really have a lot of golang packages yet, but based on the breakage
> > I see in packages like netbird it seems like an inevitable issue at any
> > meaningful scale.
> >
> > >> For example netbird is still broken with GOPROXY=direct due to url breakage
> > >> issues which have not had fixes merged(surprise surprise maintainers don't
> > >> care or test with GOPROXY=direct seemingly ever as it's not the default):
> > >> https://github.com/netbirdio/netbird/pull/3240
> > >
> > > I don't agree that your 1 week old PR not being merged yet is a suitable indicator that maintainers don't care about their dependencies matching what is in Git.
> >
> > Well that and the fact that GOPROXY=direct was broken in the first place
> > is a pretty strong indicator upstream projects don't care about it much.
>
> They have replied to your PR requesting an alternative strategy to
> fixing the issue:
>
> https://github.com/netbirdio/netbird/pull/3240#issuecomment-2637956537
>
> Here are the new PRs implementing what they requested:
>
> - https://github.com/netbirdio/netbird/pull/3287
> - https://github.com/netbirdio/netbird/pull/3288
>
> > What's the purpose of this? What actual real world benefits have we gained
> > from fixing these broken url/download issues upstream as opposed to simply
> > not using GOPROXY=direct like the rest of the golang ecosystem?
>
> You may note that the default of GOPROXY is:
>
> netbird % go env GOPROXY
> https://proxy.golang.org,direct
>
> Note that "direct" is the second in the list, the fallback if the
> proxy does not work.

I'm aware of the default fallback, AFAIU it is typically only used if
https://proxy.golang.org is broken/down. IMO we should use the
same default for GOPROXY because there is no good reason
not to do so.

> If the default were the proxy without the "direct" part it would be
> true that "the rest of the golang ecosystem" does not use
> GOPROXY=direct.

When I'm saying the ecosystem does not use GOPROXY=direct,
I'm referring specifically to disabling the caching proxy specifically.

>
> > > I agree with your logic that:
> > >
> > > 1) The developers of go projects like Docker or Netbird have pinned dependencies in go.sum which we can reasonably trust.
> > > 2) buildroot maintainers don't typically exhaustively audit all dependencies.
> > >
> > > What I disagree with is;
> > >
> > > 1) It is impossible to avoid adding a Google hosted service as a dependency for building Go packages in buildroot.
> >
> > What I'm saying is that it's impossible to avoid the goproxy cache without
> > relying on buildroot itself mirroring packages with their dependencies in a
> > generalized manner long term.
>
> Do you have any examples of these breakages so far? We use a number of
> Go packages in Buildroot extensively already today, like docker-engine
> and docker-cli.

I haven't seen any specifically, but based on the build failures
I've seen when attempting to use GOPROXY=direct it's clearly
an issue we would eventually start running into.

>
> Given the tradeoffs we have discussed between:
>
> - "always require the Google proxy" and
> - "fix mismatches at package add/update time, mirror if dependencies
> are unavailable long-term"
>
> There is insufficient evidence that the mirroring part is such a
> maintenance burden that we must go with option 1 here.
>
> > There's only downsides to using GOPROXY=direct in buildroot from
> > what I can tell, I can't even think of a single real world benefit.
>
> Upsides:
>
> - No dependency on a third-party proxy service

As you mentioned above, removing the GOPROXY=direct override
would not make the proxy server a hard dependency, but would
simply made direct downloads a fallback, so this isn't an upside
in the real world.

One can still fix GOPROXY=direct in upstream projects either way,
but I see no reason for it to be a hard requirement in buildroot.

Either way using the proxy cache drastically reduces the number of
services that must be available for a package to download successfully.

What upside is there to avoiding this dependency when the alternative
is way way worse?

> - Ability to build with GOPROXY=direct for those that require it
> (otherwise Buildroot breaks for them on default)

Who actually requires it? I think requiring GOPROXY=direct is an
extremely niche use case that's not really even a practical requirement.

> - If go.mod says "foo/bar at version v1.0.0" then we can be sure the
> Git tag v1.0.0 matches the go.sum.

If the hash matches we know we have the right dependency, that's
all that we really care about here other than being able to download
the dependencies reliably.

> Downsides:
>
> - Slower fetch (git pull instead of proxy.golang.org)
> - We need to mirror packages if their dependencies go away (most are
> on GitHub, though)
>
> When requesting that upstream projects fix the build with
> GOPROXY=direct it helps to mention why in the PR description.

I guess I should mention something like we need to fix this because
buildroot uses bad GOPROXY defaults since we like doing lots of
extra work for no practical reason?

>
> Best regards,
> Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Christian Stewart via buildroot]

Hi James,

On Wed, Feb 5, 2025 at 1:55 PM James Hilliard <james.hilliard1@gmail.com> wrote:
> > > Well that and the fact that GOPROXY=direct was broken in the first place
> > > is a pretty strong indicator upstream projects don't care about it much.
> >
> > They have replied to your PR requesting an alternative strategy to
> > fixing the issue:
> >
> > https://github.com/netbirdio/netbird/pull/3240#issuecomment-2637956537
> >
> > Here are the new PRs implementing what they requested:
> >
> > - https://github.com/netbirdio/netbird/pull/3287
> > - https://github.com/netbirdio/netbird/pull/3288

One of these is already merged, and the other is being looked at right now.

It is not as hard as you make it seem to fix upstream.

> > You may note that the default of GOPROXY is:
> >
> > netbird % go env GOPROXY
> > https://proxy.golang.org,direct
> >
> > Note that "direct" is the second in the list, the fallback if the
> > proxy does not work.
>
> I'm aware of the default fallback, AFAIU it is typically only used if
> https://proxy.golang.org is broken/down. IMO we should use the
> same default for GOPROXY because there is no good reason
> not to do so.

I have listed the good reasons earlier in this thread.

See later in this mail...

> > If the default were the proxy without the "direct" part it would be
> > true that "the rest of the golang ecosystem" does not use
> > GOPROXY=direct.
>
> When I'm saying the ecosystem does not use GOPROXY=direct,
> I'm referring specifically to disabling the caching proxy specifically.

Please provide evidence for this claim.

> > Do you have any examples of these breakages so far? We use a number of
> > Go packages in Buildroot extensively already today, like docker-engine
> > and docker-cli.
>
> I haven't seen any specifically, but based on the build failures
> I've seen when attempting to use GOPROXY=direct it's clearly
> an issue we would eventually start running into.

It hasn't happened at all yet after a package was merged, then.

> > Given the tradeoffs we have discussed between:
> >
> > - "always require the Google proxy" and
> > - "fix mismatches at package add/update time, mirror if dependencies
> > are unavailable long-term"
> >
> > There is insufficient evidence that the mirroring part is such a
> > maintenance burden that we must go with option 1 here.
> >
> > > There's only downsides to using GOPROXY=direct in buildroot from
> > > what I can tell, I can't even think of a single real world benefit.
> >
> > Upsides:
> >
> > - No dependency on a third-party proxy service
>
> As you mentioned above, removing the GOPROXY=direct override
> would not make the proxy server a hard dependency, but would
> simply made direct downloads a fallback, so this isn't an upside
> in the real world.

I do not understand what you mean by this sentence.

> One can still fix GOPROXY=direct in upstream projects either way,
> but I see no reason for it to be a hard requirement in buildroot.
>
> Either way using the proxy cache drastically reduces the number of
> services that must be available for a package to download successfully.
>
> What upside is there to avoiding this dependency when the alternative
> is way way worse?

I understand your thinking on it but fixing things at merge time so
that we can build without the Google proxy is not impossible nor as
difficult as you make it seem.

> > - Ability to build with GOPROXY=direct for those that require it
> > (otherwise Buildroot breaks for them on default)
>
> Who actually requires it? I think requiring GOPROXY=direct is an
> extremely niche use case that's not really even a practical requirement.

If I set "GOPROXY=direct" and run a Buildroot build I would expect it to pass.

Knowingly merging packages that have mismatches between Git and
GOPROXY seems like a mistake.

> > - If go.mod says "foo/bar at version v1.0.0" then we can be sure the
> > Git tag v1.0.0 matches the go.sum.
>
> If the hash matches we know we have the right dependency, that's
> all that we really care about here other than being able to download
> the dependencies reliably.

Yes and no. That may be all you care about as someone wanting to ship packages.

> > Downsides:
> >
> > - Slower fetch (git pull instead of proxy.golang.org)
> > - We need to mirror packages if their dependencies go away (most are
> > on GitHub, though)
> >
> > When requesting that upstream projects fix the build with
> > GOPROXY=direct it helps to mention why in the PR description.
>
> I guess I should mention something like we need to fix this because
> buildroot uses bad GOPROXY defaults since we like doing lots of
> extra work for no practical reason?

You keep saying "no practical reason" but we clearly have upsides and
downsides here.

Thanks,
Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[James Hilliard]

On Thu, Feb 6, 2025 at 12:18 AM Christian Stewart <christian@aperture.us> wrote:
>
> Hi James,
>
> On Wed, Feb 5, 2025 at 1:55 PM James Hilliard <james.hilliard1@gmail.com> wrote:
> > > > Well that and the fact that GOPROXY=direct was broken in the first place
> > > > is a pretty strong indicator upstream projects don't care about it much.
> > >
> > > They have replied to your PR requesting an alternative strategy to
> > > fixing the issue:
> > >
> > > https://github.com/netbirdio/netbird/pull/3240#issuecomment-2637956537
> > >
> > > Here are the new PRs implementing what they requested:
> > >
> > > - https://github.com/netbirdio/netbird/pull/3287
> > > - https://github.com/netbirdio/netbird/pull/3288
>
> One of these is already merged, and the other is being looked at right now.
>
> It is not as hard as you make it seem to fix upstream.

Sure, maybe for someone that regularly develops golang projects
it should be easy, but I don't really write any go code, so debugging
confusing GOPROXY=direct related breakage is not always that easy.

> > > You may note that the default of GOPROXY is:
> > >
> > > netbird % go env GOPROXY
> > > https://proxy.golang.org,direct
> > >
> > > Note that "direct" is the second in the list, the fallback if the
> > > proxy does not work.
> >
> > I'm aware of the default fallback, AFAIU it is typically only used if
> > https://proxy.golang.org is broken/down. IMO we should use the
> > same default for GOPROXY because there is no good reason
> > not to do so.
>
> I have listed the good reasons earlier in this thread.
>
> See later in this mail...
>
> > > If the default were the proxy without the "direct" part it would be
> > > true that "the rest of the golang ecosystem" does not use
> > > GOPROXY=direct.
> >
> > When I'm saying the ecosystem does not use GOPROXY=direct,
> > I'm referring specifically to disabling the caching proxy specifically.
>
> Please provide evidence for this claim.

I mean, it's not the default, and I simply don't see much evidence of use
when searching github.

> > > Do you have any examples of these breakages so far? We use a number of
> > > Go packages in Buildroot extensively already today, like docker-engine
> > > and docker-cli.
> >
> > I haven't seen any specifically, but based on the build failures
> > I've seen when attempting to use GOPROXY=direct it's clearly
> > an issue we would eventually start running into.
>
> It hasn't happened at all yet after a package was merged, then.

Sure, I'm just saying it almost certainly will happen eventually.

> > > Given the tradeoffs we have discussed between:
> > >
> > > - "always require the Google proxy" and
> > > - "fix mismatches at package add/update time, mirror if dependencies
> > > are unavailable long-term"
> > >
> > > There is insufficient evidence that the mirroring part is such a
> > > maintenance burden that we must go with option 1 here.
> > >
> > > > There's only downsides to using GOPROXY=direct in buildroot from
> > > > what I can tell, I can't even think of a single real world benefit.
> > >
> > > Upsides:
> > >
> > > - No dependency on a third-party proxy service
> >
> > As you mentioned above, removing the GOPROXY=direct override
> > would not make the proxy server a hard dependency, but would
> > simply made direct downloads a fallback, so this isn't an upside
> > in the real world.
>
> I do not understand what you mean by this sentence.

I mean you mention that avoiding a third-party proxy service as a
dependency is an upside, but it's not clear what sort of advantage
that would provide us in practice for buildroot.

>
> > One can still fix GOPROXY=direct in upstream projects either way,
> > but I see no reason for it to be a hard requirement in buildroot.
> >
> > Either way using the proxy cache drastically reduces the number of
> > services that must be available for a package to download successfully.
> >
> > What upside is there to avoiding this dependency when the alternative
> > is way way worse?
>
> I understand your thinking on it but fixing things at merge time so
> that we can build without the Google proxy is not impossible nor as
> difficult as you make it seem.

I'm not saying it's impossible but it's annoying and doesn't prevent
dependencies from breaking in the future. So IMO there's very little
reason to require golang packages to build with GOPROXY=direct
in the first place.

> > > - Ability to build with GOPROXY=direct for those that require it
> > > (otherwise Buildroot breaks for them on default)
> >
> > Who actually requires it? I think requiring GOPROXY=direct is an
> > extremely niche use case that's not really even a practical requirement.
>
> If I set "GOPROXY=direct" and run a Buildroot build I would expect it to pass.

Why? The rest of the golang ecosystem certainly doesn't seem to care
much about GOPROXY=direct. What makes buildroot special here?

> Knowingly merging packages that have mismatches between Git and
> GOPROXY seems like a mistake.

I mean, we only care about the download matching the checksums, so
I really see no reason we would care about this. If upstream projects
want to do this, I'm sure one could make a github action or something
fairly easily to audit dependencies automatically there.

> > > - If go.mod says "foo/bar at version v1.0.0" then we can be sure the
> > > Git tag v1.0.0 matches the go.sum.
> >
> > If the hash matches we know we have the right dependency, that's
> > all that we really care about here other than being able to download
> > the dependencies reliably.
>
> Yes and no. That may be all you care about as someone wanting to ship packages.

Buildroot is a downstream integration of many upstream projects,
we don't develop the upstream golang package code here, we integrate
existing projects largely as-is without modifying their code. Using the
GOPROXY defaults simplifies this integration.

> > > Downsides:
> > >
> > > - Slower fetch (git pull instead of proxy.golang.org)
> > > - We need to mirror packages if their dependencies go away (most are
> > > on GitHub, though)
> > >
> > > When requesting that upstream projects fix the build with
> > > GOPROXY=direct it helps to mention why in the PR description.
> >
> > I guess I should mention something like we need to fix this because
> > buildroot uses bad GOPROXY defaults since we like doing lots of
> > extra work for no practical reason?
>
> You keep saying "no practical reason" but we clearly have upsides and
> downsides here.

The upsides you mention aren't practical upsides for buildroot however,
they may be practical for auditing project dependencies, but we don't
do that here as part of buildroot project in practice, hence the
"no practical reason" as no upsides for buildroot exist IMO.

>
> Thanks,
> Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Christian Stewart via buildroot]

Hi James,

On Wed, Feb 5, 2025 at 2:56 PM James Hilliard <james.hilliard1@gmail.com> wrote:
> > On Wed, Feb 5, 2025 at 1:55 PM James Hilliard <james.hilliard1@gmail.com> wrote:
> > > > > Well that and the fact that GOPROXY=direct was broken in the first place
> > > > > is a pretty strong indicator upstream projects don't care about it much.
> > > >
> > > > They have replied to your PR requesting an alternative strategy to
> > > > fixing the issue:
> > > >
> > > > https://github.com/netbirdio/netbird/pull/3240#issuecomment-2637956537
> > > >
> > > > Here are the new PRs implementing what they requested:
> > > >
> > > > - https://github.com/netbirdio/netbird/pull/3287
> > > > - https://github.com/netbirdio/netbird/pull/3288
> >
> > One of these is already merged, and the other is being looked at right now.
> >
> > It is not as hard as you make it seem to fix upstream.
>
> Sure, maybe for someone that regularly develops golang projects
> it should be easy, but I don't really write any go code, so debugging
> confusing GOPROXY=direct related breakage is not always that easy.

Ok. We will trust that upstream packages (Docker, netbird, etc) have
the correct hash in go.sum, and use that as the source-of-trust here.

We should set the default BR2_GOPROXY to match $(go env GOPROXY), as
Arnout mentioned.

I submitted a v4 of your patch, with some minor modifications:

https://patchwork.ozlabs.org/project/buildroot/patch/20250205232837.70303-1-christian@aperture.us/

I would request that we still at least test with BR2_GOPROXY="direct"
when merging packages.

We might want to make BR2_GOPROXY="direct" be the default if
BR2_PRIMARY_SITE_ONLY is set.

Best regards,
Christian Stewart

> The upsides you mention aren't practical upsides for buildroot however,
> they may be practical for auditing project dependencies, but we don't
> do that here as part of buildroot project in practice, hence the
> "no practical reason" as no upsides for buildroot exist IMO.

OK.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[James Hilliard]

On Thu, Feb 6, 2025 at 1:31 AM Christian Stewart <christian@aperture.us> wrote:
>
> Hi James,
>
> On Wed, Feb 5, 2025 at 2:56 PM James Hilliard <james.hilliard1@gmail.com> wrote:
> > > On Wed, Feb 5, 2025 at 1:55 PM James Hilliard <james.hilliard1@gmail.com> wrote:
> > > > > > Well that and the fact that GOPROXY=direct was broken in the first place
> > > > > > is a pretty strong indicator upstream projects don't care about it much.
> > > > >
> > > > > They have replied to your PR requesting an alternative strategy to
> > > > > fixing the issue:
> > > > >
> > > > > https://github.com/netbirdio/netbird/pull/3240#issuecomment-2637956537
> > > > >
> > > > > Here are the new PRs implementing what they requested:
> > > > >
> > > > > - https://github.com/netbirdio/netbird/pull/3287
> > > > > - https://github.com/netbirdio/netbird/pull/3288
> > >
> > > One of these is already merged, and the other is being looked at right now.
> > >
> > > It is not as hard as you make it seem to fix upstream.
> >
> > Sure, maybe for someone that regularly develops golang projects
> > it should be easy, but I don't really write any go code, so debugging
> > confusing GOPROXY=direct related breakage is not always that easy.
>
> Ok. We will trust that upstream packages (Docker, netbird, etc) have
> the correct hash in go.sum, and use that as the source-of-trust here.
>
> We should set the default BR2_GOPROXY to match $(go env GOPROXY), as
> Arnout mentioned.
>
> I submitted a v4 of your patch, with some minor modifications:
>
> https://patchwork.ozlabs.org/project/buildroot/patch/20250205232837.70303-1-christian@aperture.us/
>
> I would request that we still at least test with BR2_GOPROXY="direct"
> when merging packages.
>
> We might want to make BR2_GOPROXY="direct" be the default if
> BR2_PRIMARY_SITE_ONLY is set.

I think it isn't needed, as AFAIU BR2_PRIMARY_SITE_ONLY forces
a single server to be used for all downloads, and anyways as
GOPROXY="direct" wouldn't download from the primary site so
I don't think that would be correct.

If anything a proxy would be closer to what BR2_PRIMARY_SITE_ONLY
is intended for AFAIU which is to redirect all downloads to say a single
package mirror server.

>
> Best regards,
> Christian Stewart
>
> > The upsides you mention aren't practical upsides for buildroot however,
> > they may be practical for auditing project dependencies, but we don't
> > do that here as part of buildroot project in practice, hence the
> > "no practical reason" as no upsides for buildroot exist IMO.
>
> OK.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] Fwd: [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Christian Stewart via buildroot]

Hi James, Arnout,

I would like to add that there is an open issue where the godocs
website is displaying the cached proxy version and not the GitHub
version, which is considered another potential malware vector.

Open issue with activity in the last couple days:

https://github.com/golang/go/issues/66653#issuecomment-2636358401

On Wed, Feb 5, 2025 at 6:02 AM James Hilliard <james.hilliard1@gmail.com> wrote:
> On Wed, Feb 5, 2025 at 2:51 PM Christian Stewart <christian@aperture.us> wrote:
> > Below I mention a potential security issue where someone uploads a malicious package version to the proxy and then changes the git tag to remove the malware. In this attack mode the maintainers might check the git tag, see no malware, and add a package with that tag.
> >
> > The problem is that Go fetches from GOPROXY and silently ignores the mismatch.
>
> As it says in the article this is how it's designed to work:
>
> "Go modules are intentionally immutable once published and fetched by
> the proxy [snip]

Correct, but this can lead to a security issue when the code in the
proxy is malicious.

> > This was the reason why I have always lobbied to keep the default as GOPROXY=direct in buildroot, as this avoids the case where the git repo mismatches the proxy without visibility at the git level.
>
> As it says in the article:
>
> "While immutability can be abused by malicious actors — allowing
> harmful code to persist once cached — it remains a net security
> benefit. Since the Go module system is functioning as intended, and
> this is not a security vulnerability, there is no patch or switch to
> turn off immutability; instead, developers must recognize that once a
> malicious version is published, it will remain malicious in the
> cache."

Yes. But it is not acceptable for us to ship malicious code.

Code in the proxy mismatching the code downloaded with GOPROXY=direct
is a good indicator that something suspicious is going on and worth
further investigation.

> So it's effectively expected that go packages use the cache as
> GOPROXY=direct is an unusual non-default configuration that simply
> breaks builds for many non-malicious reasons(dead urls for example).

The proxy is an optimization, not a source of truth.

> Distributions like buildroot are not typically auditing package
> dependencies in a way that would catch these sort of attacks either
> way, auditing dependencies would potentially be something that
> upstream package maintainers would want to do

If the code mismatches the GitHub tag there is a good sign something is off.

> If we have a go package in buildroot that we know won't build
> with GOPROXY=direct then I don't see how we can possibly default to
> using GOPROXY=direct as that would obviously result in failed builds.

If it doesn't build with GOPROXY=direct we do not ship that version.

Instead we can file a GitHub issue with the upstream asking why there
is a mismatch.

We did this with tailscale and a few other packages successfully already.

Overall it should be possible to fetch directly from Git and bypass GOPROXY.

As Arnout mentioned, if this becomes a major issue in the future, we
could add a patching mechanism.

But I don't see this as being the case. We very rarely get hash
mismatches and when we have it was fixed by upstream.

So far the only time this has actually been an issue was when one
project had Git LFS test files in the repo.

One additional alternative is to ship a commit-hash version
temporarily until a new Git tag is created.

Best regards,
Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Fwd: [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[James Hilliard]

On Wed, Feb 5, 2025 at 7:56 PM Christian Stewart via buildroot
<buildroot@buildroot.org> wrote:
>
> Hi James, Arnout,
>
> I would like to add that there is an open issue where the godocs
> website is displaying the cached proxy version and not the GitHub
> version, which is considered another potential malware vector.

The immutability of the dependencies is a security feature, forcing us
to use direct dependencies for no good reason if anything is going
to introduce more attack vectors, especially if it forces us to stop
using upstream provided dependency checksums which are an
actual security feature.

In buildroot we use direct downloads but also have a mirror, however
the download locations by themselves for either aren't what is providing
reproducibility/security, the hashes are what ensure we get the intended
dependency.

> Open issue with activity in the last couple days:
>
> https://github.com/golang/go/issues/66653#issuecomment-2636358401
>
> On Wed, Feb 5, 2025 at 6:02 AM James Hilliard <james.hilliard1@gmail.com> wrote:
> > On Wed, Feb 5, 2025 at 2:51 PM Christian Stewart <christian@aperture.us> wrote:
> > > Below I mention a potential security issue where someone uploads a malicious package version to the proxy and then changes the git tag to remove the malware. In this attack mode the maintainers might check the git tag, see no malware, and add a package with that tag.
> > >
> > > The problem is that Go fetches from GOPROXY and silently ignores the mismatch.
> >
> > As it says in the article this is how it's designed to work:
> >
> > "Go modules are intentionally immutable once published and fetched by
> > the proxy [snip]
>
> Correct, but this can lead to a security issue when the code in the
> proxy is malicious.

It doesn't matter if the proxy is entirely malicious as long as the hashes are
not malicious and are being validated properly.

If you want to actually use GOPROXY=direct to try and catch attacks you
can do those checks outside of buildroot just fine, but I see no reason that
buildroot itself should be doing this as it's just going to cause a bunch of
pointless maintainability issues/breakage down the line as download
locations will inevitably break. Buildroot AFAIU has always relied on hashes
for dependency security, golang is the same.

> > > This was the reason why I have always lobbied to keep the default as GOPROXY=direct in buildroot, as this avoids the case where the git repo mismatches the proxy without visibility at the git level.
> >
> > As it says in the article:
> >
> > "While immutability can be abused by malicious actors — allowing
> > harmful code to persist once cached — it remains a net security
> > benefit. Since the Go module system is functioning as intended, and
> > this is not a security vulnerability, there is no patch or switch to
> > turn off immutability; instead, developers must recognize that once a
> > malicious version is published, it will remain malicious in the
> > cache."
>
> Yes. But it is not acceptable for us to ship malicious code.

Sure but GOPROXY=direct isn't going to help with that IMO, the hashes
in the sum file are what matter for security.

> Code in the proxy mismatching the code downloaded with GOPROXY=direct
> is a good indicator that something suspicious is going on and worth
> further investigation.

Not really IMO, I mean I've hit this same issue multiple times and it's always
something along the lines of the download location being removed/changed
for non-malicious reasons.

>
> > So it's effectively expected that go packages use the cache as
> > GOPROXY=direct is an unusual non-default configuration that simply
> > breaks builds for many non-malicious reasons(dead urls for example).
>
> The proxy is an optimization, not a source of truth.

The article effectively states the opposite.

>
> > Distributions like buildroot are not typically auditing package
> > dependencies in a way that would catch these sort of attacks either
> > way, auditing dependencies would potentially be something that
> > upstream package maintainers would want to do
>
> If the code mismatches the GitHub tag there is a good sign something is off.

So far in the cases I've run into it simply means the download is broken,
so I'd hardly say it's a good indication of a security issue, sure it can be
used to obfuscate a backdoored package, but that's an issue for upstream
maintainers to deal with IMO. Mucking with upstream package dependencies
is IMO more likely to introduce security issues.

> > If we have a go package in buildroot that we know won't build
> > with GOPROXY=direct then I don't see how we can possibly default to
> > using GOPROXY=direct as that would obviously result in failed builds.
>
> If it doesn't build with GOPROXY=direct we do not ship that version.

What about cases where it initially builds with GOPROXY=direct but
at some point stops building with GOPROXY=direct due to url breakage
or similar reasons? Expecting GOPROXY=direct to work doesn't seem
feasible since it can break at any point in the future and isn't something
upstream projects will be inclined to fix due to the lack of upstreams
caring about it.

> Instead we can file a GitHub issue with the upstream asking why there
> is a mismatch.
>
> We did this with tailscale and a few other packages successfully already.

This has done nothing to improve security IMO however.

> Overall it should be possible to fetch directly from Git and bypass GOPROXY.

Maybe in an ideal world, but I'm convinced it's not feasible in practice
due to the potential for downloads to break at any point in the future.

>
> As Arnout mentioned, if this becomes a major issue in the future, we
> could add a patching mechanism.
>
> But I don't see this as being the case. We very rarely get hash
> mismatches and when we have it was fixed by upstream.

The majority of go based projects I've tried adding to buildroot have
been incompatible with GOPROXY=direct initially it seems, so I'd
hardly say it's a rare issue at all.

> So far the only time this has actually been an issue was when one
> project had Git LFS test files in the repo.

Netbird is still broken with GOPROXY=direct, with multiple reasons
for breakage.

> One additional alternative is to ship a commit-hash version
> temporarily until a new Git tag is created.

Or we could just stop deliberately breaking builds with GOPROXY=direct
as a default.

>
> Best regards,
> Christian Stewart
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Fwd: [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Fiona Klute via buildroot]

Hi everyone!

Am 05.02.25 um 18:56 schrieb Christian Stewart via buildroot:
> Hi James, Arnout,
>
> I would like to add that there is an open issue where the godocs
> website is displaying the cached proxy version and not the GitHub
> version, which is considered another potential malware vector.
>
> Open issue with activity in the last couple days:
>
> https://github.com/golang/go/issues/66653#issuecomment-2636358401
>
> On Wed, Feb 5, 2025 at 6:02 AM James Hilliard <james.hilliard1@gmail.com> wrote:
>> On Wed, Feb 5, 2025 at 2:51 PM Christian Stewart <christian@aperture.us> wrote:
>>> Below I mention a potential security issue where someone uploads a malicious package version to the proxy and then changes the git tag to remove the malware. In this attack mode the maintainers might check the git tag, see no malware, and add a package with that tag.
>>>
>>> The problem is that Go fetches from GOPROXY and silently ignores the mismatch.
>>
>> As it says in the article this is how it's designed to work:
>>
>> "Go modules are intentionally immutable once published and fetched by
>> the proxy [snip]
>
> Correct, but this can lead to a security issue when the code in the
> proxy is malicious.
>
>>> This was the reason why I have always lobbied to keep the default as GOPROXY=direct in buildroot, as this avoids the case where the git repo mismatches the proxy without visibility at the git level.
>>
>> As it says in the article:
>>
>> "While immutability can be abused by malicious actors — allowing
>> harmful code to persist once cached — it remains a net security
>> benefit. Since the Go module system is functioning as intended, and
>> this is not a security vulnerability, there is no patch or switch to
>> turn off immutability; instead, developers must recognize that once a
>> malicious version is published, it will remain malicious in the
>> cache."
>
> Yes. But it is not acceptable for us to ship malicious code.
>
> Code in the proxy mismatching the code downloaded with GOPROXY=direct
> is a good indicator that something suspicious is going on and worth
> further investigation.
This is the critical part in my opinion: failing safe. As annoying as
seemingly randomly failing builds are, they're not going to do damage
beyond time spent to investigate and fix whatever the cause is, whether
it's a mistake or (rarely) malicious action. Once we have a trusted
value in package.hash the download source doesn't matter much, but I
agree that requiring downloads to match when introducing/updating a
package is a security feature.

Best regards,
Fiona

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Fwd: [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[James Hilliard]

On Thu, Feb 6, 2025 at 9:31 PM Fiona Klute via buildroot
<buildroot@buildroot.org> wrote:
>
> Hi everyone!
>
> Am 05.02.25 um 18:56 schrieb Christian Stewart via buildroot:
> > Hi James, Arnout,
> >
> > I would like to add that there is an open issue where the godocs
> > website is displaying the cached proxy version and not the GitHub
> > version, which is considered another potential malware vector.
> >
> > Open issue with activity in the last couple days:
> >
> > https://github.com/golang/go/issues/66653#issuecomment-2636358401
> >
> > On Wed, Feb 5, 2025 at 6:02 AM James Hilliard <james.hilliard1@gmail.com> wrote:
> >> On Wed, Feb 5, 2025 at 2:51 PM Christian Stewart <christian@aperture.us> wrote:
> >>> Below I mention a potential security issue where someone uploads a malicious package version to the proxy and then changes the git tag to remove the malware. In this attack mode the maintainers might check the git tag, see no malware, and add a package with that tag.
> >>>
> >>> The problem is that Go fetches from GOPROXY and silently ignores the mismatch.
> >>
> >> As it says in the article this is how it's designed to work:
> >>
> >> "Go modules are intentionally immutable once published and fetched by
> >> the proxy [snip]
> >
> > Correct, but this can lead to a security issue when the code in the
> > proxy is malicious.
> >
> >>> This was the reason why I have always lobbied to keep the default as GOPROXY=direct in buildroot, as this avoids the case where the git repo mismatches the proxy without visibility at the git level.
> >>
> >> As it says in the article:
> >>
> >> "While immutability can be abused by malicious actors — allowing
> >> harmful code to persist once cached — it remains a net security
> >> benefit. Since the Go module system is functioning as intended, and
> >> this is not a security vulnerability, there is no patch or switch to
> >> turn off immutability; instead, developers must recognize that once a
> >> malicious version is published, it will remain malicious in the
> >> cache."
> >
> > Yes. But it is not acceptable for us to ship malicious code.
> >
> > Code in the proxy mismatching the code downloaded with GOPROXY=direct
> > is a good indicator that something suspicious is going on and worth
> > further investigation.
> This is the critical part in my opinion: failing safe. As annoying as
> seemingly randomly failing builds are, they're not going to do damage
> beyond time spent to investigate and fix whatever the cause is, whether
> it's a mistake or (rarely) malicious action. Once we have a trusted
> value in package.hash the download source doesn't matter much, but I
> agree that requiring downloads to match when introducing/updating a
> package is a security feature.

Just to clarify there are 2 different hash checks going on here essentially,
there's the hashes in the go.sum file and the package hashes in buildroot.

When we vendor the dependencies they are being validated against the
go.sum file that's part of the package source, as long as this validates we
know the correct dependencies as specified by the upstream package
were downloaded correctly, the issue is that GOPROXY=direct disables
the caching proxy which is designed to ensure that one can download
the dependencies matching the hashes in the go.sum file in the event
the original download source broke/changed for whatever reason.

A direct download failing does not necessarily indicate a mistake or
malicious action, it may simply be that the dependency download
location no longer exists. As we have the caching proxy we should
simply be using that by default as that's the expected way for
golang to download dependencies as it prevents download failures
that can happen when a direct download location disappears or
changes. Using the proxy instead of direct downloads does not
meaningfully reduce security due to go.sum being validated either
way.

>
> Best regards,
> Fiona
>
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Fwd: [PATCH v4 1/2] package/pkg-golang.mk: allow packages to override download GOPROXY

[Christian Stewart via buildroot]

[-- Attachment #1.1: Type: text/plain, Size: 1647 bytes --]

James,

On Fri, Feb 7, 2025, 9:35 AM James Hilliard <james.hilliard1@gmail.com>
wrote:

> Just to clarify there are 2 different hash checks going on here
> essentially,
> there's the hashes in the go.sum file and the package hashes in buildroot.
>
> When we vendor the dependencies they are being validated against the
> go.sum file that's part of the package source, as long as this validates we
> know the correct dependencies as specified by the upstream package
> were downloaded correctly, the issue is that GOPROXY=direct disables
> the caching proxy which is designed to ensure that one can download
> the dependencies matching the hashes in the go.sum file in the event
> the original download source broke/changed for whatever reason.
>
> A direct download failing does not necessarily indicate a mistake or
> malicious action, it may simply be that the dependency download
> location no longer exists. As we have the caching proxy we should
> simply be using that by default as that's the expected way for
> golang to download dependencies as it prevents download failures
> that can happen when a direct download location disappears or
> changes. Using the proxy instead of direct downloads does not
> meaningfully reduce security due to go.sum being validated either
> way.
>

This is true but as you have identified many upstream developers are also
not checking dependencies carefully either.

If we look for build failures with GOPROXY=direct and notify upstream we
can help catch dependencies that may have malicious mismatches between the
git tag and the GOPROXY version.

Thanks,
Christian

[-- Attachment #1.2: Type: text/html, Size: 2139 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot