[Buildroot] [PATCH 1/2] package/go: set GOCACHE for download post-process

[Buildroot] [PATCH 1/2] package/go: set GOCACHE for download post-process

[Peter Korsgaard]

The go mod vendor call in support/download/go-post-process accesses the go
cache, so pass GOCACHE= in the environment to ensure our cache directory is
used.

The go cache defaults to ~/.cache/go-build if not set, so this fixes builds
where that location (or GOCACHE if set in the environment) is not writable:

rm -rf ~/.cache/go-build
chmod -w ~/.cache
make docker-compose-source
..
failed to initialize build cache at /home/peko/.cache/go-build: mkdir /home/peko/.cache/go-build: permission denied
make[1]: *** [package/pkg-generic.mk:189: /home/peko/source/buildroot/output/build/docker-compose-2.14.0/.stamp_downloaded] Error 1

We use two different cache directories for target and host builds, but the
download/vendoring should be independent of the architecture, so use the
target variant even for host-only packages for simplicity.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
---
 package/go/go.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/go/go.mk b/package/go/go.mk
index d3ed07f03b..c4f72d403c 100644
--- a/package/go/go.mk
+++ b/package/go/go.mk
@@ -25,6 +25,7 @@ HOST_GO_COMMON_ENV = \
 	GOFLAGS=-mod=vendor \
 	GOROOT="$(HOST_GO_ROOT)" \
 	GOPATH="$(HOST_GO_GOPATH)" \
+	GOCACHE="$(HOST_GO_TARGET_CACHE)" \
 	GOPROXY=off \
 	PATH=$(BR_PATH) \
 	GOBIN= \
@@ -75,7 +76,6 @@ HOST_GO_TARGET_ENV = \
 	$(HOST_GO_COMMON_ENV) \
 	GOOS="linux" \
 	GOARCH=$(GO_GOARCH) \
-	GOCACHE="$(HOST_GO_TARGET_CACHE)" \
 	CC="$(TARGET_CC)" \
 	CXX="$(TARGET_CXX)" \
 	CGO_CFLAGS="$(TARGET_CFLAGS)" \
-- 
2.30.2

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 2/2] package/go: explicitly set GOMODCACHE

[Peter Korsgaard]

go mod vendor caches downloaded modules to the Go module cache, which
defaults to $GOPATH/pkg/mod - But can be overridden with the GOMODCACHE
environment variable:

https://go.dev/ref/mod#module-cache

So explicitly set GOMODCACHE= for reproducibility.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
---
 package/go/go.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/go/go.mk b/package/go/go.mk
index c4f72d403c..c38ae0b99c 100644
--- a/package/go/go.mk
+++ b/package/go/go.mk
@@ -26,6 +26,7 @@ HOST_GO_COMMON_ENV = \
 	GOROOT="$(HOST_GO_ROOT)" \
 	GOPATH="$(HOST_GO_GOPATH)" \
 	GOCACHE="$(HOST_GO_TARGET_CACHE)" \
+	GOMODCACHE="$(HOST_GO_GOPATH)/pkg/mod" \
 	GOPROXY=off \
 	PATH=$(BR_PATH) \
 	GOBIN= \
-- 
2.30.2

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/2] package/go: explicitly set GOMODCACHE

[Yann E. MORIN]

Peter, All,

On 2023-01-16 11:50 +0100, Peter Korsgaard spake thusly:
> go mod vendor caches downloaded modules to the Go module cache, which
> defaults to $GOPATH/pkg/mod - But can be overridden with the GOMODCACHE
> environment variable:
> 
> https://go.dev/ref/mod#module-cache
> 
> So explicitly set GOMODCACHE= for reproducibility.
> 
> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>

Applied to master, thanks, but see below...

> ---
>  package/go/go.mk | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/package/go/go.mk b/package/go/go.mk
> index c4f72d403c..c38ae0b99c 100644
> --- a/package/go/go.mk
> +++ b/package/go/go.mk
> @@ -26,6 +26,7 @@ HOST_GO_COMMON_ENV = \
>  	GOROOT="$(HOST_GO_ROOT)" \
>  	GOPATH="$(HOST_GO_GOPATH)" \
>  	GOCACHE="$(HOST_GO_TARGET_CACHE)" \
> +	GOMODCACHE="$(HOST_GO_GOPATH)/pkg/mod" \

HOST_GO_GOPATH is $(HOST_DIR)/share/go-path, however, with PPD, HOST_DIR
is per-package, so there will not besharing of the cache between two
packages, unless there is a build dependency between them (unlikely in
the go ecosystem).

If we wanted to really share the go cache(s), we need to hit somewhere
in $(BR2_DL_DIR), like we do for cargo, see 8450b7691870 (package/pkg-cargo:
move CARGO_HOME into DL_DIR).

Still, this patch guarantees that the current situation is reproducible,
and does not depend on the user's environment.

Applied to master, thanks.

Regards,
Yann E. MORIN.

>  	GOPROXY=off \
>  	PATH=$(BR_PATH) \
>  	GOBIN= \
> -- 
> 2.30.2
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/2] package/go: explicitly set GOMODCACHE

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Peter, All,
 > On 2023-01-16 11:50 +0100, Peter Korsgaard spake thusly:
 >> go mod vendor caches downloaded modules to the Go module cache, which
 >> defaults to $GOPATH/pkg/mod - But can be overridden with the GOMODCACHE
 >> environment variable:
 >> 
 >> https://go.dev/ref/mod#module-cache
 >> 
 >> So explicitly set GOMODCACHE= for reproducibility.
 >> 
 >> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>

 > Applied to master, thanks, but see below...

 >> ---
 >> package/go/go.mk | 1 +
 >> 1 file changed, 1 insertion(+)
 >> 
 >> diff --git a/package/go/go.mk b/package/go/go.mk
 >> index c4f72d403c..c38ae0b99c 100644
 >> --- a/package/go/go.mk
 >> +++ b/package/go/go.mk
 >> @@ -26,6 +26,7 @@ HOST_GO_COMMON_ENV = \
 >> GOROOT="$(HOST_GO_ROOT)" \
 >> GOPATH="$(HOST_GO_GOPATH)" \
 >> GOCACHE="$(HOST_GO_TARGET_CACHE)" \
 >> +	GOMODCACHE="$(HOST_GO_GOPATH)/pkg/mod" \

 > HOST_GO_GOPATH is $(HOST_DIR)/share/go-path, however, with PPD, HOST_DIR
 > is per-package, so there will not besharing of the cache between two
 > packages, unless there is a build dependency between them (unlikely in
 > the go ecosystem).

 > If we wanted to really share the go cache(s), we need to hit somewhere
 > in $(BR2_DL_DIR), like we do for cargo, see 8450b7691870 (package/pkg-cargo:
 > move CARGO_HOME into DL_DIR).

 > Still, this patch guarantees that the current situation is reproducible,
 > and does not depend on the user's environment.

Yes, I know. As far as I can see this was already the case (on systems
where GOMODCACHE isn't set), and I wanted a "safe" fix we could backport
- But indeed, longer term we should probably put the go module cache in
DL_DIR like we recently did for the cargo cache.

 > Applied to master, thanks.

Thanks!

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/2] package/go: explicitly set GOMODCACHE

[Christian Stewart via buildroot]

[-- Attachment #1.1: Type: text/plain, Size: 1167 bytes --]

Hi Peter,

This reminds me of when GO111MODULE was leaking from the host environment
as well. I wonder if there's any merit to wiping the host environment
variables if br2 reproducible is enabled?



On Mon, Jan 16, 2023, 2:52 AM Peter Korsgaard <peter@korsgaard.com> wrote:

> go mod vendor caches downloaded modules to the Go module cache, which
> defaults to $GOPATH/pkg/mod - But can be overridden with the GOMODCACHE
> environment variable:
>
> https://go.dev/ref/mod#module-cache
>
> So explicitly set GOMODCACHE= for reproducibility.
>
> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
> ---
>  package/go/go.mk | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/package/go/go.mk b/package/go/go.mk
> index c4f72d403c..c38ae0b99c 100644
> --- a/package/go/go.mk
> +++ b/package/go/go.mk
> @@ -26,6 +26,7 @@ HOST_GO_COMMON_ENV = \
>         GOROOT="$(HOST_GO_ROOT)" \
>         GOPATH="$(HOST_GO_GOPATH)" \
>         GOCACHE="$(HOST_GO_TARGET_CACHE)" \
> +       GOMODCACHE="$(HOST_GO_GOPATH)/pkg/mod" \
>         GOPROXY=off \
>         PATH=$(BR_PATH) \
>         GOBIN= \
> --
> 2.30.2
>


Reviewed-by: Christian Stewart <christian@paral.in>

>
>

[-- Attachment #1.2: Type: text/html, Size: 2486 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/2] package/go: explicitly set GOMODCACHE

[Peter Korsgaard]

>>>>> "Christian" == Christian Stewart <christian@paral.in> writes:

 > Hi Peter,
 > This reminds me of when GO111MODULE was leaking from the host environment
 > as well. I wonder if there's any merit to wiping the host environment
 > variables if br2 reproducible is enabled?

I definately think it is, ideally we want Buildroot to work the same way
on all hosts - But why only for reproducible builds?

Notice that we do override GO111MODULE= in HOST_GO_COMMON_ENV.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/2] package/go: explicitly set GOMODCACHE

[Christian Stewart via buildroot]

Hi Peter,

On Mon, Jan 16, 2023 at 12:55 PM Peter Korsgaard <peter@korsgaard.com> wrote:
>
> >>>>> "Christian" == Christian Stewart <christian@paral.in> writes:
>
>  > Hi Peter,
>  > This reminds me of when GO111MODULE was leaking from the host environment
>  > as well. I wonder if there's any merit to wiping the host environment
>  > variables if br2 reproducible is enabled?
>
> I definately think it is, ideally we want Buildroot to work the same way
> on all hosts - But why only for reproducible builds?
>
> Notice that we do override GO111MODULE= in HOST_GO_COMMON_ENV.

Apologies my wording was a bit ambiguous - we should always drop any
Go related environment variables (you can get a full list with "go
env") from the host environment.

I was additionally saying it might make sense to drop the entire host
environment for all of buildroot, but maybe only if REPRODUCIBLE is
enabled.

Best,
Christian
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/2] package/go: explicitly set GOMODCACHE

[Peter Korsgaard]

>>>>> "Peter" == Peter Korsgaard <peter@korsgaard.com> writes:

 > go mod vendor caches downloaded modules to the Go module cache, which
 > defaults to $GOPATH/pkg/mod - But can be overridden with the GOMODCACHE
 > environment variable:

 > https://go.dev/ref/mod#module-cache

 > So explicitly set GOMODCACHE= for reproducibility.

 > Signed-off-by: Peter Korsgaard <peter@korsgaard.com>

Committed to 2022.11.x and 2022.02.x, thanks.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] package/go: set GOCACHE for download post-process

[Yann E. MORIN]

Peter, All,

On 2023-01-16 11:50 +0100, Peter Korsgaard spake thusly:
> The go mod vendor call in support/download/go-post-process accesses the go
> cache, so pass GOCACHE= in the environment to ensure our cache directory is
> used.
> 
> The go cache defaults to ~/.cache/go-build if not set, so this fixes builds
> where that location (or GOCACHE if set in the environment) is not writable:
> 
> rm -rf ~/.cache/go-build
> chmod -w ~/.cache
> make docker-compose-source
> ..
> failed to initialize build cache at /home/peko/.cache/go-build: mkdir /home/peko/.cache/go-build: permission denied
> make[1]: *** [package/pkg-generic.mk:189: /home/peko/source/buildroot/output/build/docker-compose-2.14.0/.stamp_downloaded] Error 1
> 
> We use two different cache directories for target and host builds, but the
> download/vendoring should be independent of the architecture, so use the
> target variant even for host-only packages for simplicity.
> 
> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>

Applied to master, thanks.

Regards,
Yann E. MORIN.

> ---
>  package/go/go.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/package/go/go.mk b/package/go/go.mk
> index d3ed07f03b..c4f72d403c 100644
> --- a/package/go/go.mk
> +++ b/package/go/go.mk
> @@ -25,6 +25,7 @@ HOST_GO_COMMON_ENV = \
>  	GOFLAGS=-mod=vendor \
>  	GOROOT="$(HOST_GO_ROOT)" \
>  	GOPATH="$(HOST_GO_GOPATH)" \
> +	GOCACHE="$(HOST_GO_TARGET_CACHE)" \
>  	GOPROXY=off \
>  	PATH=$(BR_PATH) \
>  	GOBIN= \
> @@ -75,7 +76,6 @@ HOST_GO_TARGET_ENV = \
>  	$(HOST_GO_COMMON_ENV) \
>  	GOOS="linux" \
>  	GOARCH=$(GO_GOARCH) \
> -	GOCACHE="$(HOST_GO_TARGET_CACHE)" \
>  	CC="$(TARGET_CC)" \
>  	CXX="$(TARGET_CXX)" \
>  	CGO_CFLAGS="$(TARGET_CFLAGS)" \
> -- 
> 2.30.2
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] package/go: set GOCACHE for download post-process

[Christian Stewart via buildroot]

[-- Attachment #1.1: Type: text/plain, Size: 1903 bytes --]

Hi Peter,

Thought this was set already!


On Mon, Jan 16, 2023, 2:52 AM Peter Korsgaard <peter@korsgaard.com> wrote:

> The go mod vendor call in support/download/go-post-process accesses the go
> cache, so pass GOCACHE= in the environment to ensure our cache directory is
> used.
>
> The go cache defaults to ~/.cache/go-build if not set, so this fixes builds
> where that location (or GOCACHE if set in the environment) is not writable:
>
> rm -rf ~/.cache/go-build
> chmod -w ~/.cache
> make docker-compose-source
> ..
> failed to initialize build cache at /home/peko/.cache/go-build: mkdir
> /home/peko/.cache/go-build: permission denied
> make[1]: *** [package/pkg-generic.mk:189:
> /home/peko/source/buildroot/output/build/docker-compose-2.14.0/.stamp_downloaded]
> Error 1
>
> We use two different cache directories for target and host builds, but the
> download/vendoring should be independent of the architecture, so use the
> target variant even for host-only packages for simplicity.
>
> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
> ---
>  package/go/go.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/package/go/go.mk b/package/go/go.mk
> index d3ed07f03b..c4f72d403c 100644
> --- a/package/go/go.mk
> +++ b/package/go/go.mk
> @@ -25,6 +25,7 @@ HOST_GO_COMMON_ENV = \
>         GOFLAGS=-mod=vendor \
>         GOROOT="$(HOST_GO_ROOT)" \
>         GOPATH="$(HOST_GO_GOPATH)" \
> +       GOCACHE="$(HOST_GO_TARGET_CACHE)" \
>         GOPROXY=off \
>         PATH=$(BR_PATH) \
>         GOBIN= \
> @@ -75,7 +76,6 @@ HOST_GO_TARGET_ENV = \
>         $(HOST_GO_COMMON_ENV) \
>         GOOS="linux" \
>         GOARCH=$(GO_GOARCH) \
> -       GOCACHE="$(HOST_GO_TARGET_CACHE)" \
>         CC="$(TARGET_CC)" \
>         CXX="$(TARGET_CXX)" \
>         CGO_CFLAGS="$(TARGET_CFLAGS)" \
> --
> 2.30.2
>


Reviewed-by: Christian Stewart <christian@paral.in>

>
>

[-- Attachment #1.2: Type: text/html, Size: 3300 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] package/go: set GOCACHE for download post-process

[Peter Korsgaard]

>>>>> "Christian" == Christian Stewart <christian@paral.in> writes:

 > Hi Peter,
 > Thought this was set already!

It is for the build step - but not for the download step, so we can end
up in problems with the go mod vendor call in the download post-process.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] package/go: set GOCACHE for download post-process

[Peter Korsgaard]

>>>>> "Peter" == Peter Korsgaard <peter@korsgaard.com> writes:

 > The go mod vendor call in support/download/go-post-process accesses the go
 > cache, so pass GOCACHE= in the environment to ensure our cache directory is
 > used.

 > The go cache defaults to ~/.cache/go-build if not set, so this fixes builds
 > where that location (or GOCACHE if set in the environment) is not writable:

 > rm -rf ~/.cache/go-build
 > chmod -w ~/.cache
 > make docker-compose-source
 > ..
 > failed to initialize build cache at /home/peko/.cache/go-build: mkdir /home/peko/.cache/go-build: permission denied
 > make[1]: *** [package/pkg-generic.mk:189:
 > /home/peko/source/buildroot/output/build/docker-compose-2.14.0/.stamp_downloaded]
 > Error 1

 > We use two different cache directories for target and host builds, but the
 > download/vendoring should be independent of the architecture, so use the
 > target variant even for host-only packages for simplicity.

 > Signed-off-by: Peter Korsgaard <peter@korsgaard.com>

Committed to 2022.11.x and 2022.02.x, thanks.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot