* Re: [Buildroot] [PATCH v8 1/1] package/go: use host compiler when go-bootstrap unsupported
2024-08-19 20:33 [Buildroot] [PATCH v8 1/1] package/go: use host compiler when go-bootstrap unsupported Christian Stewart via buildroot
@ 2024-08-19 21:17 ` Yann E. MORIN
2024-08-19 21:34 ` Christian Stewart via buildroot
2024-09-15 9:15 ` Arnout Vandecappelle via buildroot
1 sibling, 1 reply; 6+ messages in thread
From: Yann E. MORIN @ 2024-08-19 21:17 UTC (permalink / raw)
To: Christian Stewart; +Cc: Anisse Astier, Thomas Petazzoni, buildroot
Christian, All,
On 2024-08-19 13:33 -0700, Christian Stewart via buildroot spake thusly:
> All Go compiler versions > 1.4.x (old) are written in Go, and require a existing
> compiled Go version to use to build from source.
>
> https://golang.org/doc/install/source#bootstrapFromSource
>
> The process for "bootstrapping" the Go compiler in Buildroot is:
>
> 1. Compile a C/C++ cross-compiler (gcc) as the host toolchain.
> 2. Build go-bootstrap-stage1 (which is Go 1.4.x and written in C)
> 3. Build go-bootstrap-stage2 (which is Go 1.19.x and written in Go)
> 4. Build go-bootstrap-stage3 (which is Go 1.21.x and written in Go)
> 5. Build go-src (written in Go) using go-bootstrap-stage3.
>
> go-bootstrap-stage1 does not work on 64-bit arm. The Go 1.4.x bootstrap compiler
> is compatible with x86, x86_64, and arm (32 bit) only.
>
> This patch adds a fallback to require a host Go compiler to build host-go-src
> when BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS is not set.
>
> Recent changes added go-src with the intent of adding go-bin later. This
> commit changes go-src to depend on the host Go compiler to bootstrap
> go-src on architectures that cannot build go-bootstrap-stage1 (such as arm64).
>
> Signed-off-by: Christian Stewart <christian@aperture.us>
Thanks for this new iteration. But I'd rather first have the support for
host-go-bin, rather than rely on the system go.
Indeed, usng host-go-bin would not require the user to install go on
their system; host-go-bin can be used on oder distributions where a go
compiler is not available, or where the go compiler is too old to build
go-src.
So, I am not convinced about this "use the system go" solution.
Regards,
Yann E. MORIN.
> ---
>
> changes prior to inclusion in this series:
>
> - thanks Thomas for the review & suggestions
> - added NEEDS_HOST_GO boolean
> - added dependency checks to support/dependencies/dependencies.sh
> - removed unnecessary changes to go-bootstrap package
> - add dependency on toolchain if Cgo is enabled
> - updates for go1.20
> - updates for go-bootstrap-stage{1,2}
>
> changes from v1 -> v2:
>
> - remove whitespace fix in bootstrap stage2
>
> changes from v2 -> v3:
>
> - rebase
>
> changes from v7 -> v8:
>
> - rebase on go-src changes
> - Discussion related to adding go-src and go-bin:
> - https://patchwork.ozlabs.org/project/buildroot/patch/20230711220849.1702358-4-christian@aperture.us/
>
> Signed-off-by: Christian Stewart <christian@aperture.us>
> ---
> Config.in | 4 ++++
> package/go/Config.in.host | 3 +--
> package/go/go-src/go-src.mk | 12 ++++++++----
> support/dependencies/dependencies.sh | 4 ++++
> 4 files changed, 17 insertions(+), 6 deletions(-)
>
> diff --git a/Config.in b/Config.in
> index df43db7eff..bd7227a049 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -68,6 +68,10 @@ config BR2_HOST_GCC_AT_LEAST_11
> # When adding new entries above, be sure to update
> # the HOSTCC_MAX_VERSION variable in the Makefile.
>
> +# Hidden boolean selected if bootstrapping Go w/ GCC is not supported.
> +config BR2_NEEDS_HOST_GO
> + bool
> +
> # Hidden boolean selected by packages in need of Java in order to build
> # (example: kodi)
> config BR2_NEEDS_HOST_JAVA
> diff --git a/package/go/Config.in.host b/package/go/Config.in.host
> index 11ce6df369..e006dfc830 100644
> --- a/package/go/Config.in.host
> +++ b/package/go/Config.in.host
> @@ -2,7 +2,6 @@
> config BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
> bool
> default y
> - depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS
> # See https://go.dev/doc/install/source#environment
> # See src/go/build/syslist.go for the list of supported architectures
> depends on (BR2_arm && BR2_TOOLCHAIN_SUPPORTS_PIE) || BR2_aarch64 \
> @@ -34,7 +33,6 @@ config BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS
> config BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS
> bool
> default y
> - depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS
>
> # Go packages should select BR2_PACKAGE_HOST_GO
> config BR2_PACKAGE_HOST_GO
> @@ -53,6 +51,7 @@ choice
>
> config BR2_PACKAGE_HOST_GO_SRC
> bool "host go (source)"
> + select BR2_NEEDS_HOST_GO if
> !BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS
> help
> This package will build the go compiler for the host.
>
> diff --git a/package/go/go-src/go-src.mk b/package/go/go-src/go-src.mk
> index 0d1a9b3187..7e78f6e4d2 100644
> --- a/package/go/go-src/go-src.mk
> +++ b/package/go/go-src/go-src.mk
> @@ -12,9 +12,7 @@ GO_SRC_LICENSE_FILES = LICENSE
> GO_SRC_CPE_ID_VENDOR = golang
>
> HOST_GO_SRC_PROVIDES = host-go
> -HOST_GO_SRC_DEPENDENCIES = \
> - host-go-bootstrap-stage3 \
> - $(HOST_GO_DEPENDENCIES_CGO)
> +HOST_GO_SRC_DEPENDENCIES = $(HOST_GO_DEPENDENCIES_CGO)
>
> ifeq ($(BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS),y)
>
> @@ -34,7 +32,6 @@ endif
> HOST_GO_SRC_MAKE_ENV = \
> GO111MODULE=off \
> GOCACHE=$(HOST_GO_HOST_CACHE) \
> - GOROOT_BOOTSTRAP=$(HOST_GO_BOOTSTRAP_STAGE3_ROOT) \
> GOROOT_FINAL=$(HOST_GO_ROOT) \
> GOROOT="$(@D)" \
> GOBIN="$(@D)/bin" \
> @@ -44,6 +41,13 @@ HOST_GO_SRC_MAKE_ENV = \
> CGO_ENABLED=$(HOST_GO_CGO_ENABLED) \
> $(HOST_GO_SRC_CROSS_ENV)
>
> +# Use the Go compiler bootstrapped by Buildroot if available.
> +# Otherwise, use the host Go compiler.
> +ifeq ($(BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS),y)
> +HOST_GO_DEPENDENCIES += host-go-bootstrap-stage3
> +HOST_GO_MAKE_ENV += GOROOT_BOOTSTRAP=$(HOST_GO_BOOTSTRAP_STAGE3_ROOT)
> +endif
> +
> define HOST_GO_SRC_BUILD_CMDS
> cd $(@D)/src && \
> $(HOST_GO_SRC_MAKE_ENV) ./make.bash $(if $(VERBOSE),-v)
> diff --git a/support/dependencies/dependencies.sh
> b/support/dependencies/dependencies.sh
> index 6d5fc36037..7b1c51b4f1 100755
> --- a/support/dependencies/dependencies.sh
> +++ b/support/dependencies/dependencies.sh
> @@ -217,6 +217,10 @@ if grep -q ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG ; then
> fi
> fi
>
> +if grep -q ^BR2_NEEDS_HOST_GO=y $BR2_CONFIG ; then
> + check_prog_host "go"
> +fi
> +
> if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then
> check_prog_host "java"
> JAVA_GCJ=$(java -version 2>&1 | grep gcj)
> --
> 2.45.2
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [Buildroot] [PATCH v8 1/1] package/go: use host compiler when go-bootstrap unsupported
2024-08-19 20:33 [Buildroot] [PATCH v8 1/1] package/go: use host compiler when go-bootstrap unsupported Christian Stewart via buildroot
2024-08-19 21:17 ` Yann E. MORIN
@ 2024-09-15 9:15 ` Arnout Vandecappelle via buildroot
2024-09-15 14:17 ` Christian Stewart via buildroot
1 sibling, 1 reply; 6+ messages in thread
From: Arnout Vandecappelle via buildroot @ 2024-09-15 9:15 UTC (permalink / raw)
To: Christian Stewart, buildroot
Cc: Anisse Astier, Thomas Petazzoni, Yann E . MORIN
Hi Christian,
On 19/08/2024 22:33, Christian Stewart via buildroot wrote:
> All Go compiler versions > 1.4.x (old) are written in Go, and require a existing
> compiled Go version to use to build from source.
>
> https://golang.org/doc/install/source#bootstrapFromSource
>
> The process for "bootstrapping" the Go compiler in Buildroot is:
>
> 1. Compile a C/C++ cross-compiler (gcc) as the host toolchain.
> 2. Build go-bootstrap-stage1 (which is Go 1.4.x and written in C)
> 3. Build go-bootstrap-stage2 (which is Go 1.19.x and written in Go)
> 4. Build go-bootstrap-stage3 (which is Go 1.21.x and written in Go)
> 5. Build go-src (written in Go) using go-bootstrap-stage3.
>
> go-bootstrap-stage1 does not work on 64-bit arm. The Go 1.4.x bootstrap compiler
> is compatible with x86, x86_64, and arm (32 bit) only.
We discussed this use case at the Buildroot Developer meeting. Clearly we do
want to support this use case in Buildroot. However, we think it should be more
generic. In particular, we want to be able to build go-src without going through
all three of the bootstrap stages even if the host architecture is supported.
The use case there is if there's a security or other patch that you want to
apply to go-src.
The solution is quite simple: make this approach an option, rather than
automatically selecting depending on if go-bootstrap-stage1 is supported or not.
Something like: when you select go-src, a new choice pops up to choose between
full bootstrap or bootstrap from a downloaded binary.
> This patch adds a fallback to require a host Go compiler to build host-go-src
> when BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS is not set.
We also discussed this one. Relying on the host Go compiler is tricky because
then you also have to check its version (which this patch doesn't do AFAICS).
And it's always tricky to rely on host tools when it comes to reproducibility.
So we think it's better to instead download the Go compiler. In terms of build
time, it shouldn't make much of a difference whether it's downloaded or the host
Go is used. And with the download it always works (at least on supported
platforms), even if there is no Go installed on the host.
> Recent changes added go-src with the intent of adding go-bin later. This
go-bin should hopefully be merged today.
Ideally, it should be possible to reuse go-bin directly to replace
go-bootstrap3, by adding some conditions to go-bin.mk (e.g. if it's used as
bootstrap3, it has to be installed in a different location). But if that is too
complex, it's also OK to create a completely separate go-boostrap3-bin.
> commit changes go-src to depend on the host Go compiler to bootstrap
> go-src on architectures that cannot build go-bootstrap-stage1 (such as arm64).
>
> Signed-off-by: Christian Stewart <christian@aperture.us>
>
> ---
>
> changes prior to inclusion in this series:
>
> - thanks Thomas for the review & suggestions
> - added NEEDS_HOST_GO boolean
> - added dependency checks to support/dependencies/dependencies.sh
> - removed unnecessary changes to go-bootstrap package
> - add dependency on toolchain if Cgo is enabled
> - updates for go1.20
> - updates for go-bootstrap-stage{1,2}
>
> changes from v1 -> v2:
>
> - remove whitespace fix in bootstrap stage2
>
> changes from v2 -> v3:
>
> - rebase
>
> changes from v7 -> v8:
>
> - rebase on go-src changes
> - Discussion related to adding go-src and go-bin:
> - https://patchwork.ozlabs.org/project/buildroot/patch/20230711220849.1702358-4-christian@aperture.us/
>
> Signed-off-by: Christian Stewart <christian@aperture.us>
> ---
> Config.in | 4 ++++
> package/go/Config.in.host | 3 +--
> package/go/go-src/go-src.mk | 12 ++++++++----
> support/dependencies/dependencies.sh | 4 ++++
> 4 files changed, 17 insertions(+), 6 deletions(-)
>
> diff --git a/Config.in b/Config.in
> index df43db7eff..bd7227a049 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -68,6 +68,10 @@ config BR2_HOST_GCC_AT_LEAST_11
> # When adding new entries above, be sure to update
> # the HOSTCC_MAX_VERSION variable in the Makefile.
>
> +# Hidden boolean selected if bootstrapping Go w/ GCC is not supported.
> +config BR2_NEEDS_HOST_GO
> + bool
> +
> # Hidden boolean selected by packages in need of Java in order to build
> # (example: kodi)
> config BR2_NEEDS_HOST_JAVA
> diff --git a/package/go/Config.in.host b/package/go/Config.in.host
> index 11ce6df369..e006dfc830 100644
> --- a/package/go/Config.in.host
> +++ b/package/go/Config.in.host
> @@ -2,7 +2,6 @@
> config BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
> bool
> default y
> - depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS
So this would become something like
depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS \
|| BR2_PACKAGE_HOST_GO_BIN_HOST_ARCH_SUPPORTS
> # See https://go.dev/doc/install/source#environment
> # See src/go/build/syslist.go for the list of supported architectures
> depends on (BR2_arm && BR2_TOOLCHAIN_SUPPORTS_PIE) || BR2_aarch64 \
> @@ -34,7 +33,6 @@ config BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS
> config BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS
> bool
> default y
> - depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS
>
> # Go packages should select BR2_PACKAGE_HOST_GO
> config BR2_PACKAGE_HOST_GO
> @@ -53,6 +51,7 @@ choice
>
> config BR2_PACKAGE_HOST_GO_SRC
> bool "host go (source)"
> + select BR2_NEEDS_HOST_GO if
> !BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS
BTW, this line was wrapped...
Regards,
Arnout
> help
> This package will build the go compiler for the host.
>
> diff --git a/package/go/go-src/go-src.mk b/package/go/go-src/go-src.mk
> index 0d1a9b3187..7e78f6e4d2 100644
> --- a/package/go/go-src/go-src.mk
> +++ b/package/go/go-src/go-src.mk
> @@ -12,9 +12,7 @@ GO_SRC_LICENSE_FILES = LICENSE
> GO_SRC_CPE_ID_VENDOR = golang
>
> HOST_GO_SRC_PROVIDES = host-go
> -HOST_GO_SRC_DEPENDENCIES = \
> - host-go-bootstrap-stage3 \
> - $(HOST_GO_DEPENDENCIES_CGO)
> +HOST_GO_SRC_DEPENDENCIES = $(HOST_GO_DEPENDENCIES_CGO)
>
> ifeq ($(BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS),y)
>
> @@ -34,7 +32,6 @@ endif
> HOST_GO_SRC_MAKE_ENV = \
> GO111MODULE=off \
> GOCACHE=$(HOST_GO_HOST_CACHE) \
> - GOROOT_BOOTSTRAP=$(HOST_GO_BOOTSTRAP_STAGE3_ROOT) \
> GOROOT_FINAL=$(HOST_GO_ROOT) \
> GOROOT="$(@D)" \
> GOBIN="$(@D)/bin" \
> @@ -44,6 +41,13 @@ HOST_GO_SRC_MAKE_ENV = \
> CGO_ENABLED=$(HOST_GO_CGO_ENABLED) \
> $(HOST_GO_SRC_CROSS_ENV)
>
> +# Use the Go compiler bootstrapped by Buildroot if available.
> +# Otherwise, use the host Go compiler.
> +ifeq ($(BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE3_ARCH_SUPPORTS),y)
> +HOST_GO_DEPENDENCIES += host-go-bootstrap-stage3
> +HOST_GO_MAKE_ENV += GOROOT_BOOTSTRAP=$(HOST_GO_BOOTSTRAP_STAGE3_ROOT)
> +endif
> +
> define HOST_GO_SRC_BUILD_CMDS
> cd $(@D)/src && \
> $(HOST_GO_SRC_MAKE_ENV) ./make.bash $(if $(VERBOSE),-v)
> diff --git a/support/dependencies/dependencies.sh
> b/support/dependencies/dependencies.sh
> index 6d5fc36037..7b1c51b4f1 100755
> --- a/support/dependencies/dependencies.sh
> +++ b/support/dependencies/dependencies.sh
> @@ -217,6 +217,10 @@ if grep -q ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG ; then
> fi
> fi
>
> +if grep -q ^BR2_NEEDS_HOST_GO=y $BR2_CONFIG ; then
> + check_prog_host "go"
> +fi
> +
> if grep -q ^BR2_NEEDS_HOST_JAVA=y $BR2_CONFIG ; then
> check_prog_host "java"
> JAVA_GCJ=$(java -version 2>&1 | grep gcj)
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 6+ messages in thread