[Buildroot] [PATCH v1 1/1] package/go: use host compiler when go-bootstrap unsupported

[Buildroot] [PATCH v1 1/1] package/go: use host compiler when go-bootstrap unsupported

[Christian Stewart]

All Go compiler versions > 1.4.x (old) are written in Go, and require a existing
compiled Go version to use to build from source.

https://golang.org/doc/install/source#bootstrapFromSource

The process for "bootstrapping" the Go compiler in Buildroot is:

1. Compile a C/C++ cross-compiler (gcc) as the host toolchain.
2. Build go-bootstrap (which is Go 1.4.x and written in C)
3. Build go 1.16.x (written in Go) using go-bootstrap.

The problem is that step 2 - build go-bootstrap - does not work on 64-bit arm.
The Go compiler from 1.4.x is compatible with x86, x86_64, and arm (32 bit).

This means that arm64 host machines will skip building Go and all go-based
packages like Docker.

This patch instead uses the host Go compiler to bootstrap host-go when
BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS is not set. This is similar to how
the host GCC is used to bootstrap the Buildroot toolchain.

Signed-off-by: Christian Stewart <christian@paral.in>
---
 package/go-bootstrap/go-bootstrap.mk | 6 ++++++
 package/go/Config.in.host            | 2 --
 package/go/go.mk                     | 4 ++++
 3 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/package/go-bootstrap/go-bootstrap.mk b/package/go-bootstrap/go-bootstrap.mk
index 6710e31561..8324a67795 100644
--- a/package/go-bootstrap/go-bootstrap.mk
+++ b/package/go-bootstrap/go-bootstrap.mk
@@ -17,7 +17,13 @@ GO_BOOTSTRAP_LICENSE_FILES = LICENSE
 # host-go-bootstrap.
 HOST_GO_BOOTSTRAP_DEPENDENCIES = toolchain
 
+# If we do not support this architecture with go-bootstrap, depend on the host
+# Go compiler to bootstrap the host-go compiler instead.
+ifeq ($(BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS),y)
 HOST_GO_BOOTSTRAP_ROOT = $(HOST_DIR)/lib/go-$(GO_BOOTSTRAP_VERSION)
+else
+HOST_GO_BOOTSTRAP_ROOT = /usr/lib/go
+endif
 
 # The go build system is not compatable with ccache, so use HOSTCC_NOCCACHE
 # here.  See https://github.com/golang/go/issues/11685.
diff --git a/package/go/Config.in.host b/package/go/Config.in.host
index e82ab6e81a..c07605dd02 100644
--- a/package/go/Config.in.host
+++ b/package/go/Config.in.host
@@ -2,7 +2,6 @@
 config BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
 	bool
 	default y
-	depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS
 	depends on (BR2_arm && BR2_TOOLCHAIN_SUPPORTS_PIE) || BR2_aarch64 \
 		|| BR2_i386 || BR2_x86_64 || BR2_powerpc64le \
 		|| BR2_mips64 || BR2_mips64el || BR2_s390x
@@ -22,4 +21,3 @@ config BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS
 config BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS
 	bool
 	default y
-	depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS
diff --git a/package/go/go.mk b/package/go/go.mk
index 4252691343..5f501d2d8c 100644
--- a/package/go/go.mk
+++ b/package/go/go.mk
@@ -12,7 +12,11 @@ GO_LICENSE = BSD-3-Clause
 GO_LICENSE_FILES = LICENSE
 GO_CPE_ID_VENDOR = golang
 
+# Depend on the host Go compiler if go-bootstrap is not available.
+ifeq ($(BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS),y)
 HOST_GO_DEPENDENCIES = host-go-bootstrap
+endif
+
 HOST_GO_GOPATH = $(HOST_DIR)/usr/share/go-path
 HOST_GO_HOST_CACHE = $(HOST_DIR)/usr/share/host-go-cache
 HOST_GO_ROOT = $(HOST_DIR)/lib/go
-- 
2.32.0

[Buildroot] [PATCH v1 1/1] package/go: use host compiler when go-bootstrap unsupported

[Baruch Siach]

Hi Christian,

On Tue, Jun 22 2021, Christian Stewart wrote:

> All Go compiler versions > 1.4.x (old) are written in Go, and require a existing
> compiled Go version to use to build from source.
>
> https://golang.org/doc/install/source#bootstrapFromSource
>
> The process for "bootstrapping" the Go compiler in Buildroot is:
>
> 1. Compile a C/C++ cross-compiler (gcc) as the host toolchain.
> 2. Build go-bootstrap (which is Go 1.4.x and written in C)
> 3. Build go 1.16.x (written in Go) using go-bootstrap.
>
> The problem is that step 2 - build go-bootstrap - does not work on 64-bit arm.
> The Go compiler from 1.4.x is compatible with x86, x86_64, and arm (32 bit).
>
> This means that arm64 host machines will skip building Go and all go-based
> packages like Docker.
>
> This patch instead uses the host Go compiler to bootstrap host-go when
> BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS is not set. This is similar to how
> the host GCC is used to bootstrap the Buildroot toolchain.
>
> Signed-off-by: Christian Stewart <christian@paral.in>
> ---
>  package/go-bootstrap/go-bootstrap.mk | 6 ++++++
>  package/go/Config.in.host            | 2 --
>  package/go/go.mk                     | 4 ++++
>  3 files changed, 10 insertions(+), 2 deletions(-)
>
> diff --git a/package/go-bootstrap/go-bootstrap.mk b/package/go-bootstrap/go-bootstrap.mk
> index 6710e31561..8324a67795 100644
> --- a/package/go-bootstrap/go-bootstrap.mk
> +++ b/package/go-bootstrap/go-bootstrap.mk
> @@ -17,7 +17,13 @@ GO_BOOTSTRAP_LICENSE_FILES = LICENSE
>  # host-go-bootstrap.
>  HOST_GO_BOOTSTRAP_DEPENDENCIES = toolchain
>  
> +# If we do not support this architecture with go-bootstrap, depend on the host
> +# Go compiler to bootstrap the host-go compiler instead.
> +ifeq ($(BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS),y)
>  HOST_GO_BOOTSTRAP_ROOT = $(HOST_DIR)/lib/go-$(GO_BOOTSTRAP_VERSION)
> +else
> +HOST_GO_BOOTSTRAP_ROOT = /usr/lib/go

I think we need a check for host go existence and maybe also version under
support/dependencies/.

baruch

> +endif
>  
>  # The go build system is not compatable with ccache, so use HOSTCC_NOCCACHE
>  # here.  See https://github.com/golang/go/issues/11685.
> diff --git a/package/go/Config.in.host b/package/go/Config.in.host
> index e82ab6e81a..c07605dd02 100644
> --- a/package/go/Config.in.host
> +++ b/package/go/Config.in.host
> @@ -2,7 +2,6 @@
>  config BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
>  	bool
>  	default y
> -	depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS
>  	depends on (BR2_arm && BR2_TOOLCHAIN_SUPPORTS_PIE) || BR2_aarch64 \
>  		|| BR2_i386 || BR2_x86_64 || BR2_powerpc64le \
>  		|| BR2_mips64 || BR2_mips64el || BR2_s390x
> @@ -22,4 +21,3 @@ config BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS
>  config BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS
>  	bool
>  	default y
> -	depends on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS
> diff --git a/package/go/go.mk b/package/go/go.mk
> index 4252691343..5f501d2d8c 100644
> --- a/package/go/go.mk
> +++ b/package/go/go.mk
> @@ -12,7 +12,11 @@ GO_LICENSE = BSD-3-Clause
>  GO_LICENSE_FILES = LICENSE
>  GO_CPE_ID_VENDOR = golang
>  
> +# Depend on the host Go compiler if go-bootstrap is not available.
> +ifeq ($(BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS),y)
>  HOST_GO_DEPENDENCIES = host-go-bootstrap
> +endif
> +
>  HOST_GO_GOPATH = $(HOST_DIR)/usr/share/go-path
>  HOST_GO_HOST_CACHE = $(HOST_DIR)/usr/share/host-go-cache
>  HOST_GO_ROOT = $(HOST_DIR)/lib/go


-- 
                                                     ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.52.368.4656, http://www.tkos.co.il -

[Buildroot] [PATCH v1 1/1] package/go: use host compiler when go-bootstrap unsupported

[Christian Stewart]

Hi Baruch,

On Mon, Jun 21, 2021 at 8:10 PM Baruch Siach <baruch@tkos.co.il> wrote:
> > +HOST_GO_BOOTSTRAP_ROOT = /usr/lib/go
>
> I think we need a check for host go existence and maybe also version under
> support/dependencies/.
>
> baruch

I had a look at those scripts and am not familiar enough to know the
correct way to only check for host-go existence on the system only if
all of:

 - BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS=y
 - BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS is not set
 - BR2_PACKAGE_HOST_GO=y

If any of these conditions are not met, it's not necessary to have Go
installed on the host system.

Best,
Christian

Re: [Buildroot] [PATCH v1 1/1] package/go: use host compiler when go-bootstrap unsupported

[Thomas Petazzoni]

Hello Christian,

On Mon, 21 Jun 2021 19:43:37 -0700
Christian Stewart <christian@paral.in> wrote:

> All Go compiler versions > 1.4.x (old) are written in Go, and require a existing
> compiled Go version to use to build from source.
> 
> https://golang.org/doc/install/source#bootstrapFromSource
> 
> The process for "bootstrapping" the Go compiler in Buildroot is:
> 
> 1. Compile a C/C++ cross-compiler (gcc) as the host toolchain.
> 2. Build go-bootstrap (which is Go 1.4.x and written in C)
> 3. Build go 1.16.x (written in Go) using go-bootstrap.
> 
> The problem is that step 2 - build go-bootstrap - does not work on 64-bit arm.
> The Go compiler from 1.4.x is compatible with x86, x86_64, and arm (32 bit).

So perhaps my brain is not smart enough here, but how is the Go
compiler then build on 64-bit ARM ?

> Signed-off-by: Christian Stewart <christian@paral.in>
> ---
>  package/go-bootstrap/go-bootstrap.mk | 6 ++++++
>  package/go/Config.in.host            | 2 --
>  package/go/go.mk                     | 4 ++++
>  3 files changed, 10 insertions(+), 2 deletions(-)

Implementation-wise, I agree with the feedback from Baruch, we need
some checks in support/dependencies/.

Normally, the way we do things is that packages express their
dependency on a particular system tool by selecting a hidden boolean.
For example packages that need java on the host will select
BR2_NEEDS_HOST_JAVA, and in support/dependencies/dependencies.sh, if
BR2_NEEDS_HOST_JAVA=y, we check if java is installed.

The other model that we have is for optional host tools, where we can
use the system-provided one or fallback to building our own if not.
That's for example what support/dependencies/check-host-lzip.{mk,sh} is
doing.

check-host-lzip.mk calls suitable-host-package, which itself calls the
check-host-lzip.sh script to verify if lzip is installed. If there is
no suitable lzip available on the host, then check-host-lzip.mk defines
BR2_LZIP_HOST_DEPENDENCY to host-lzip, so that packages who need the
lzip utility can depend on $(BR2_LZIP_HOST_DEPENDENCY). It will be
empty if the system has a lzip program, or it will be set to host-lzip
if Buildroot needs to build its own.

Another feature for the lzip situation is that we can force Buildroot
to build the host packages even if the tool is found on the machine,
using BR2_FORCE_HOST_BUILD=y.

The thing is that your host-go-bootstrap situation does not exactly fit
these situations.

I think the easiest would be to do like we do for Java, but only select
BR2_NEEDS_HOST_GO if we're on a host architecture that doesn't support
building host-go-bootstrap.

If we go this route, then it should look like this:

 * Top-level Config.in defines BR2_NEEDS_HOST_GO as an hidden boolean

 * support/dependencies/dependencies.sh should test if a functional Go
   compiler is available on the machine if BR2_NEEDS_HOST_GO is
   defined, and if not bail out with a hard error

 * BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS and
   BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS should loose their "depends
   on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS" and instead have
   something like:

	select BR2_NEEDS_HOST_GO if !BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS

 * go.mk should be modified to not use host-go-bootstrap if
   !BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS. I don't think
   host-go-bootstrap should be modified like you did however, the
   changes should be contained into package/go/go.mk.

The one thing that bothers me however is the hardcoding of /usr/lib/go
in your proposal as the location of the Go compiler. I guess we could
have some logic in the top-level Makefile that looks for the path of
the host Go compiler perhaps ?

Does this make sense ?

Best regards,

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v1 1/1] package/go: use host compiler when go-bootstrap unsupported

[Christian Stewart]

Hi Thomas, all,

On Wed, Jul 21, 2021 at 1:49 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
> Christian Stewart <christian@paral.in> wrote:
> > The problem is that step 2 - build go-bootstrap - does not work on 64-bit arm.
> > The Go compiler from 1.4.x is compatible with x86, x86_64, and arm (32 bit).
>
> So perhaps my brain is not smart enough here, but how is the Go
> compiler then build on 64-bit ARM ?

It has to be bootstrapped using an existing arm64 go compiler binary,
or cross compiled.

Presumably a bootstrap process which produces first an armv7 Go
compiler, and then uses it to build a arm64 Go compiler, is also
possible.

With gccgo, it should be possible to bootstrap using the Gcc version
of the Go compiler (which is included with the Buildroot toolchain
today, but not enabled.)

> The one thing that bothers me however is the hardcoding of /usr/lib/go
> in your proposal as the location of the Go compiler. I guess we could
> have some logic in the top-level Makefile that looks for the path of
> the host Go compiler perhaps ?

Yes, it could be done like this:

PATH_TO_GOROOT=$(go env GOROOT)

> Does this make sense ?

Yes, this should work fine.

Eventually the gccgo based bootstrap approach could also be added as
another alternative.

Best regards,
Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v1 1/1] package/go: use host compiler when go-bootstrap unsupported

[Christian Stewart via buildroot]

Hi Thomas,

I have implemented your suggestions below & sent as a V2.

On Wed, Jul 21, 2021 at 1:49 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
> Normally, the way we do things is that packages express their
> dependency on a particular system tool by selecting a hidden boolean.
> For example packages that need java on the host will select
> BR2_NEEDS_HOST_JAVA, and in support/dependencies/dependencies.sh, if
> BR2_NEEDS_HOST_JAVA=y, we check if java is installed.
>
> The other model that we have is for optional host tools, where we can
> use the system-provided one or fallback to building our own if not.
> That's for example what support/dependencies/check-host-lzip.{mk,sh} is
> doing.
>
> check-host-lzip.mk calls suitable-host-package, which itself calls the
> check-host-lzip.sh script to verify if lzip is installed. If there is
> no suitable lzip available on the host, then check-host-lzip.mk defines
> BR2_LZIP_HOST_DEPENDENCY to host-lzip, so that packages who need the wo
> lzip utility can depend on $(BR2_LZIP_HOST_DEPENDENCY). It will be
> empty if the system has a lzip program, or it will be set to host-lzip
> if Buildroot needs to build its own.
>
> Another feature for the lzip situation is that we can force Buildroot
> to build the host packages even if the tool is found on the machine,
> using BR2_FORCE_HOST_BUILD=y.
>
> The thing is that your host-go-bootstrap situation does not exactly fit
> these situations.
>
> I think the easiest would be to do like we do for Java, but only select
> BR2_NEEDS_HOST_GO if we're on a host architecture that doesn't support
> building host-go-bootstrap.
>
> If we go this route, then it should look like this:
>
>  * Top-level Config.in defines BR2_NEEDS_HOST_GO as an hidden boolean
>
>  * support/dependencies/dependencies.sh should test if a functional Go
>    compiler is available on the machine if BR2_NEEDS_HOST_GO is
>    defined, and if not bail out with a hard error
>
>  * BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS and
>    BR2_PACKAGE_HOST_GO_HOST_ARCH_SUPPORTS should loose their "depends
>    on BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS" and instead have
>    something like:
>
>         select BR2_NEEDS_HOST_GO if !BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS
>
>  * go.mk should be modified to not use host-go-bootstrap if
>    !BR2_PACKAGE_HOST_GO_BOOTSTRAP_ARCH_SUPPORTS. I don't think
>    host-go-bootstrap should be modified like you did however, the
>    changes should be contained into package/go/go.mk.

Thanks & best,
Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot