[PATCH] go: Allow bootstrapping using host go compiler

[PATCH] go: Allow bootstrapping using host go compiler

[Daniel Thompson]

Currently go-native bootstraps without any dependency on the host go
compiler by building go-1.4 (the last version written in C) and using
that to build a more recent version of go. This does not work on host
architectures, such as arm64, that are not supported by go-1.4. To
support these host architectures we must rely on an existing host go
compiler and use that to build go-native instead.

Rather than add arm64 specific code that forces the use of the host go
compiler, we use an architecture neutral approach based on adopting the
host go compiler if it is both recent enough and the output of `go env
GOROOT` passes basic sanity testing.

Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
---
 meta/conf/bitbake.conf                 |  4 +++
 meta/recipes-devtools/go/go-native.inc | 38 ++++++++++++++++++++++++--
 2 files changed, 39 insertions(+), 3 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 7f8b043cc4..61a3069579 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -514,6 +514,10 @@ HOSTTOOLS_NONFATAL += "bzr"
 # Used by ssh fetcher
 HOSTTOOLS_NONFATAL += "scp"
 
+# Used to bootstrap go more quickly (and to support host architectures not
+# supported by go-1.4)
+HOSTTOOLS_NONFATAL += "go"
+
 CCACHE ??= ""
 
 TOOLCHAIN_OPTIONS = " --sysroot=${STAGING_DIR_TARGET}"
diff --git a/meta/recipes-devtools/go/go-native.inc b/meta/recipes-devtools/go/go-native.inc
index 207708745e..adf1666e29 100644
--- a/meta/recipes-devtools/go/go-native.inc
+++ b/meta/recipes-devtools/go/go-native.inc
@@ -10,14 +10,46 @@ CC = "${@d.getVar('BUILD_CC').strip()}"
 
 GOMAKEARGS ?= "--no-banner"
 
+check_host_go () {
+	local GOROOT="`go env GOROOT`"
+	if [ ! -x "$GOROOT/bin/go" ]
+	then
+		return 1
+	fi
+
+	local GOVERSION=`$GOROOT/bin/go version | tr ' ' '\n' | grep ^go[0-9] | cut -b3-`
+	local GOMAJOR=`echo $GOVERSION | cut -d. -f 1`
+	local GOMINOR=`echo $GOVERSION | cut -d. -f 2`
+	if [ "`echo $GOMAJOR.$GOMINOR | tr -d 0-9`" != "." ] || \
+	   [ "$GOMAJOR" -lt 1 ] || \
+	   [ "$GOMAJOR" -eq 1 -a "$GOMINOR" -lt 4 ]
+	then
+		return 1
+	fi
+
+	return 0
+}
+
 do_configure() {
-	cd ${WORKDIR}/go1.4/go/src
-	CGO_ENABLED=0 GOROOT=${WORKDIR}/go1.4/go ./make.bash
+	if check_host_go
+	then
+		bbplain "Bootstrap using host go compiler: `go env GOROOT`"
+	else
+		cd ${WORKDIR}/go1.4/go/src
+		CGO_ENABLED=0 GOROOT=${WORKDIR}/go1.4/go ./make.bash
+	fi
 }
+do_configure[cleandirs] =+ "${GOTMPDIR}"
 
 do_compile() {
 	export GOROOT_FINAL="${libdir_native}/go"
-	export GOROOT_BOOTSTRAP="${WORKDIR}/go1.4/go"
+
+	if check_host_go
+	then
+		export GOROOT_BOOTSTRAP="`go env GOROOT`"
+	else
+		export GOROOT_BOOTSTRAP="${WORKDIR}/go1.4/go"
+	fi
 
 	cd src
 	./make.bash ${GOMAKEARGS}
-- 
2.20.1

Re: [PATCH] go: Allow bootstrapping using host go compiler

[Richard Purdie]

On Fri, 2019-04-12 at 13:30 +0100, Daniel Thompson wrote:
> Currently go-native bootstraps without any dependency on the host go
> compiler by building go-1.4 (the last version written in C) and using
> that to build a more recent version of go. This does not work on host
> architectures, such as arm64, that are not supported by go-1.4. To
> support these host architectures we must rely on an existing host go
> compiler and use that to build go-native instead.
> 
> Rather than add arm64 specific code that forces the use of the host
> go
> compiler, we use an architecture neutral approach based on adopting
> the
> host go compiler if it is both recent enough and the output of `go
> env
> GOROOT` passes basic sanity testing.
> 
> Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
> ---
>  meta/conf/bitbake.conf                 |  4 +++
>  meta/recipes-devtools/go/go-native.inc | 38
> ++++++++++++++++++++++++--
>  2 files changed, 39 insertions(+), 3 deletions(-)

Whilst this is quite a creative approach it isn't deterministic and I
can see this introducing bugs which are hard to track down :(

I think we need to make this a specific configuration option.

Cheers,

Richard

Re: [PATCH] go: Allow bootstrapping using host go compiler

[Daniel Thompson]

On Fri, Apr 12, 2019 at 01:39:52PM +0100, Richard Purdie wrote:
> On Fri, 2019-04-12 at 13:30 +0100, Daniel Thompson wrote:
> > Currently go-native bootstraps without any dependency on the host go
> > compiler by building go-1.4 (the last version written in C) and using
> > that to build a more recent version of go. This does not work on host
> > architectures, such as arm64, that are not supported by go-1.4. To
> > support these host architectures we must rely on an existing host go
> > compiler and use that to build go-native instead.
> > 
> > Rather than add arm64 specific code that forces the use of the host
> > go
> > compiler, we use an architecture neutral approach based on adopting
> > the
> > host go compiler if it is both recent enough and the output of `go
> > env
> > GOROOT` passes basic sanity testing.
> > 
> > Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org>
> > ---
> >  meta/conf/bitbake.conf                 |  4 +++
> >  meta/recipes-devtools/go/go-native.inc | 38
> > ++++++++++++++++++++++++--
> >  2 files changed, 39 insertions(+), 3 deletions(-)
> 
> Whilst this is quite a creative approach it isn't deterministic and I
> can see this introducing bugs which are hard to track down :(

I guess I can't disagree too strongly... the version check and bbplain
message about the go toolchain we adopted were intended to mitigate
this by at least making the selected go toolchain conspicious (and thus
a git blame would probably draw flames towards me).

We could make things fully deterministic by downloading go compiler
binaries (rather than source) if the host architecture is not supported
by go-1.4. Is this feasible or should I continue trying to adopt the
host go compiler for these architectures?


> I think we need to make this a specific configuration option.

I guess we could wrap this up in a config option although I hope
it is ok to have per-architecture defaults so the config option is
always sane out-of-the-box.


Daniel.

Re: [PATCH] go: Allow bootstrapping using host go compiler

[richard.purdie]

On Fri, 2019-04-12 at 14:16 +0100, Daniel Thompson wrote:
> On Fri, Apr 12, 2019 at 01:39:52PM +0100, Richard Purdie wrote:
> > On Fri, 2019-04-12 at 13:30 +0100, Daniel Thompson wrote:
> > Whilst this is quite a creative approach it isn't deterministic and
> > I can see this introducing bugs which are hard to track down :(
> 
> I guess I can't disagree too strongly... the version check and
> bbplain message about the go toolchain we adopted were intended to
> mitigate this by at least making the selected go toolchain
> conspicious (and thus a git blame would probably draw flames towards
> me).

Once something makes it into sstate, the message is lost and you don't
know how it was created. The problems invariably end up on the mailing
lists/bugzilla with others being on the "front line" for this. Rightly
or wrongly its just the way it works out.

> We could make things fully deterministic by downloading go compiler
> binaries (rather than source) if the host architecture is not
> supported by go-1.4. Is this feasible or should I continue trying to
> adopt the host go compiler for these architectures?

Is the go compiler a standalone binary that can cope with this? I think
I prefer that to requiring people install host dependencies.


> > I think we need to make this a specific configuration option.
> 
> I guess we could wrap this up in a config option although I hope
> it is ok to have per-architecture defaults so the config option is
> always sane out-of-the-box.

Yes, per-arch is fine and we definitely like "out the box"! Not sure
any of us are classed as sane anymore but I digress! :)

Cheers,

Richard

Re: [PATCH] go: Allow bootstrapping using host go compiler

[Daniel Thompson]

On Fri, Apr 12, 2019 at 02:34:14PM +0100, richard.purdie@linuxfoundation.org wrote:
> On Fri, 2019-04-12 at 14:16 +0100, Daniel Thompson wrote:
> > On Fri, Apr 12, 2019 at 01:39:52PM +0100, Richard Purdie wrote:
> > > On Fri, 2019-04-12 at 13:30 +0100, Daniel Thompson wrote:
> > > Whilst this is quite a creative approach it isn't deterministic and
> > > I can see this introducing bugs which are hard to track down :(
> > 
> > I guess I can't disagree too strongly... the version check and
> > bbplain message about the go toolchain we adopted were intended to
> > mitigate this by at least making the selected go toolchain
> > conspicious (and thus a git blame would probably draw flames towards
> > me).
> 
> Once something makes it into sstate, the message is lost and you don't
> know how it was created. The problems invariably end up on the mailing
> lists/bugzilla with others being on the "front line" for this. Rightly
> or wrongly its just the way it works out.
> 
> > We could make things fully deterministic by downloading go compiler
> > binaries (rather than source) if the host architecture is not
> > supported by go-1.4. Is this feasible or should I continue trying to
> > adopt the host go compiler for these architectures?
> 
> Is the go compiler a standalone binary that can cope with this? I think
> I prefer that to requiring people install host dependencies.

For arm64 there are binaries since go1.8. They should be pretty light
w.r.t. dependencies (close to C library only IIRC) and the archived
releases go back as far as the end of 2013 (go1.2) so links should
be durable.


Daniel.

Re: [PATCH] go: Allow bootstrapping using host go compiler

[Randy MacLeod]

On 4/12/19 9:16 AM, Daniel Thompson wrote:
> We could make things fully deterministic by downloading go compiler
> binaries (rather than source) if the host architecture is not supported
> by go-1.4. Is this feasible or should I continue trying to adopt the
> host go compiler for these architectures?

You may already know but in case you hadn't consider the use case,
we also have a requirement to do off-line builds
so this download would have to fit into the fetcher system
that we rely upon to avoid build-time rather than fetch-time
downloads.

Also adding a downloaded binary could be a (minor?) complication
for IP analysis for some organizations. If a user is paranoid, is
there a well-tested, reproducible procedure for generating the
binary from source on an x86-64 host? If so should a recipe
for that be added to oe-core?

In the long run, don't we want to avoid any new binaries or host tools? 
Is anyone else in the go community (Debian maybe?) interested in
bootstrapping go on arm64 hosts or do they all just use the binary?

After all that I suppose that adding the go-1.4 binary download
could be a reasonable short-term approach that we should live with
until a source-based solution is available.



Finally, as an aside, some time ago I noticed:
     https://bootstrappable.org/
and:
 
https://www.freelists.org/post/bootstrappable/A-major-milestone-in-bootstrapping
and thought that would be a good thing for oe-core that would
allow dropping even the host gcc requirement albeit at
likely significant build time cost.

-- 
# Randy MacLeod
# Wind River Linux

Re: [PATCH] go: Allow bootstrapping using host go compiler

[Adrian Bunk]

On Fri, Apr 12, 2019 at 02:26:20PM -0400, Randy MacLeod wrote:
>...
> In the long run, don't we want to avoid any new binaries or host tools? Is
> anyone else in the go community (Debian maybe?) interested in
> bootstrapping go on arm64 hosts or do they all just use the binary?
>...

Binary distributions like Debian have to bootstrap compilers only
once per architecture.

After the initial bootstrap there is always the previous version package 
in the distribution that can be used to build the next version.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH] go: Allow bootstrapping using host go compiler

[Adrian Bunk]

On Fri, Apr 12, 2019 at 01:30:46PM +0100, Daniel Thompson wrote:
> Currently go-native bootstraps without any dependency on the host go
> compiler by building go-1.4 (the last version written in C) and using
> that to build a more recent version of go. This does not work on host
> architectures, such as arm64, that are not supported by go-1.4. To
> support these host architectures we must rely on an existing host go
> compiler and use that to build go-native instead.
>...

Why is this the only option for you?

The clean way to bootstrap go would be by building gccgo and using that 
as starting point for the bootstrap.

Haven't tried how trivial or difficult it is to make that work,
but it sounds more long-term maintainable than having different
problems depending on what architecture and host distribution
is used - it doesn't sound like a good idea to have to test
and maintain all combinations, e.g. whether Go 1.6 on arm64
in Ubuntu 16.04 manages to bootstrap the latest Go.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH] go: Allow bootstrapping using host go compiler

[Daniel Thompson]

On Fri, Apr 12, 2019 at 10:45:07PM +0300, Adrian Bunk wrote:
> On Fri, Apr 12, 2019 at 01:30:46PM +0100, Daniel Thompson wrote:
> > Currently go-native bootstraps without any dependency on the host go
> > compiler by building go-1.4 (the last version written in C) and using
> > that to build a more recent version of go. This does not work on host
> > architectures, such as arm64, that are not supported by go-1.4. To
> > support these host architectures we must rely on an existing host go
> > compiler and use that to build go-native instead.
> >...
> 
> Why is this the only option for you?

Good point. Not the most accurate use of "must"... more "these
changes".

 
> The clean way to bootstrap go would be by building gccgo and using that 
> as starting point for the bootstrap.
> 
> Haven't tried how trivial or difficult it is to make that work,
> but it sounds more long-term maintainable than having different
> problems depending on what architecture and host distribution
> is used - it doesn't sound like a good idea to have to test
> and maintain all combinations, e.g. whether Go 1.6 on arm64
> in Ubuntu 16.04 manages to bootstrap the latest Go.

Source wise it should work (depending on your level of belief in the go
compatibility promise) since the go compiler is written in go-1.4 but I
agree that code generation bugs in old versions of go tools could make
things less predictable.

I'm suddenly a bit swamped so it could me a long time to look at but
using gccgo to bootstrap is not structurally much more complex than
bootstrapping with go-1.4 and certainly could be more flexible. Sounds
like "the right thing to do" so definitely worth a shot.


Thanks


Daniel.


> -- 
> 
>        "Is there not promise of rain?" Ling Tan asked suddenly out
>         of the darkness. There had been need of rain for many days.
>        "Only a promise," Lao Er said.
>                                        Pearl S. Buck - Dragon Seed
>