[Buildroot] pkg-golang: host package not supported

[Buildroot] pkg-golang: host package not supported

[Mirza Krak]

Hi!

I have recently started doing some work with Buildroot and
specifically I am building packages that are based on golang.

Build packages for "target" works just fine. But I noticed that the
pkg-golang.mk [1] does not support "host" target yet [2] which is
required for one my packages.

Buildroot is new to me but looking at pkg-golang.mk it does _not_ seem
that a lot of needs to change to support "host" packages as well.

So two questions:
- Any specific reason "host" target is not supported?
- Any hints on to add host-golang-package support?

Thanks.

[1]. https://github.com/buildroot/buildroot/blob/master/package/pkg-golang.mk
[2]. https://github.com/buildroot/buildroot/blob/master/package/pkg-golang.mk#L47

-- 
Mirza Krak | Embedded Solutions Architect | https://mender.io

 Northern.tech AS | @northerntechHQ

[Buildroot] pkg-golang: host package not supported

[Thomas Petazzoni]

Hello Mirza,

On Fri, 10 Aug 2018 13:34:06 +0200, Mirza Krak wrote:

> I have recently started doing some work with Buildroot and
> specifically I am building packages that are based on golang.
> 
> Build packages for "target" works just fine. But I noticed that the
> pkg-golang.mk [1] does not support "host" target yet [2] which is
> required for one my packages.
> 
> Buildroot is new to me but looking at pkg-golang.mk it does _not_ seem
> that a lot of needs to change to support "host" packages as well.
> 
> So two questions:
> - Any specific reason "host" target is not supported?

Because it wasn't needed until now.

> - Any hints on to add host-golang-package support?

The first question is whether the existing host-go package installs a
Go compiler that can do native compilation in addition to
cross-compilation.

If that is the case, then pkg-golang.mk can be extended a bit like
pkg-cmake.mk, with a host-golang-package macro, which expands to:

$(call inner-golang-package,$(pkgname),$(call UPPERCASE,$(pkgname)),$(call UPPERCASE,$(pkgname)),host)

and then, in the inner-golang-package macro, do some ifeq
($(5),target) ... else ... endif to define the configure/build/install
commands differently depending on whether the package is a target
package or a host package. Again, see pkg-cmake.mk for an example.

Hope this helps!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] pkg-golang: host package not supported

[Cam Hutchison]

On 10 August 2018 at 22:00, Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
> Hello Mirza,
>
> On Fri, 10 Aug 2018 13:34:06 +0200, Mirza Krak wrote:
>
>> - Any hints on to add host-golang-package support?
>
> The first question is whether the existing host-go package installs a
> Go compiler that can do native compilation in addition to
> cross-compilation.

Yes, the existing host-go package will do native compilation. It is
just a matter of calling the compiler without the extra environment
variables set in HOST_GO_TARGET_ENV, specifically GOARCH, GOTOOLDIR,
CC and CXX. I'm not sure about GOROOT - the docs talk about that var
when un-archiving a binary installation of the toolchain anywhere but
/usr/local. I suspect a built-from-source toolchain will have the
correct GOROOT compiled in.

I have successfully compiled kubernetes under buildroot. It has a
pre-build generate-files step that builds a generator program for the
host and runs it. All I needed to do was put $(BR_PATH) at the head of
the path and ran the kubernetes generated_files step.

[I also posted the patch https://patchwork.ozlabs.org/patch/949972/ to
selectively set CGO_ENABLED, which I also needed to build kubernetes].

[Buildroot] pkg-golang: host package not supported

[Mirza Krak]

On Sun, Aug 12, 2018 at 12:23 AM, Cam Hutchison <camh@xdna.net> wrote:
> On 10 August 2018 at 22:00, Thomas Petazzoni
> <thomas.petazzoni@bootlin.com> wrote:

Thanks Cam and Thomas for your helpful responses. Will be taking a
closer look at this soon.

-- 
Mirza Krak | Embedded Solutions Architect | https://mender.io

 Northern.tech AS | @northerntechHQ

[Buildroot] pkg-golang: host package not supported

[Mirza Krak]

On Sun, Aug 12, 2018 at 12:23 AM, Cam Hutchison <camh@xdna.net> wrote:
> On 10 August 2018 at 22:00, Thomas Petazzoni
> <thomas.petazzoni@bootlin.com> wrote:
>> Hello Mirza,
>>
>> On Fri, 10 Aug 2018 13:34:06 +0200, Mirza Krak wrote:
>>
>>> - Any hints on to add host-golang-package support?
>>
>> The first question is whether the existing host-go package installs a
>> Go compiler that can do native compilation in addition to
>> cross-compilation.
>
> Yes, the existing host-go package will do native compilation. It is
> just a matter of calling the compiler without the extra environment
> variables set in HOST_GO_TARGET_ENV, specifically GOARCH, GOTOOLDIR,
> CC and CXX. I'm not sure about GOROOT - the docs talk about that var
> when un-archiving a binary installation of the toolchain anywhere but
> /usr/local. I suspect a built-from-source toolchain will have the
> correct GOROOT compiled in.

Indeed it was mostly stripping away the target stuff and it worked out
just fine.

I did have one problem with $(2)_SRC_SUBDIR, it turns out to be empty
when building host target. Workaround was to explicitly set this in
the .mk file with the HOST_ prefix. Any hints on why? Need some ifeq
($(4),target) ... else ... endif  magic? :)

The changes I made to the pkg-golang.mk file can be seen here [1].

I have some more work to do and some testing but will eventually send
some patch-sets. It is the Mender package that I am working on and
additional tooling around that, but you could have figured that out as
well :).

[1].https://github.com/mirzak/buildroot/commit/3ae6960692fe225a2a1233d489746929666a1ecd

-- 
Mirza Krak | Embedded Solutions Architect | https://mender.io

 Northern.tech AS | @northerntechHQ

[Buildroot] pkg-golang: host package not supported

[Thomas Petazzoni]

Hello,

On Wed, 15 Aug 2018 00:05:47 +0200, Mirza Krak wrote:

> I did have one problem with $(2)_SRC_SUBDIR, it turns out to be empty
> when building host target. Workaround was to explicitly set this in
> the .mk file with the HOST_ prefix. Any hints on why? Need some ifeq
> ($(4),target) ... else ... endif  magic? :)

Could you try this change:

diff --git a/package/pkg-golang.mk b/package/pkg-golang.mk
index bf178622b5..6eacd14180 100644
--- a/package/pkg-golang.mk
+++ b/package/pkg-golang.mk
@@ -77,9 +77,9 @@ $(2)_INSTALL_BINS ?= $(1)
 # of GOPATH. It usually resolves around domain/vendor/software. By default, we
 # derive domain/vendor/software from the upstream URL of the project, but we
 # allow $(2)_SRC_SUBDIR to be overridden if needed.
-$(2)_SRC_DOMAIN = $$(call domain,$($(2)_SITE))
-$(2)_SRC_VENDOR = $$(word 1,$$(subst /, ,$$(call notdomain,$($(2)_SITE))))
-$(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$($(2)_SITE))))
+$(2)_SRC_DOMAIN = $$(call domain,$$($(2)_SITE))
+$(2)_SRC_VENDOR = $$(word 1,$$(subst /, ,$$(call notdomain,$$($(2)_SITE))))
+$(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$$($(2)_SITE))))
 
 $(2)_SRC_SUBDIR ?= $$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE)
 $(2)_SRC_PATH = $$(@D)/$$($(2)_WORKSPACE)/src/$$($(2)_SRC_SUBDIR)

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] pkg-golang: host package not supported

[Mirza Krak]

On Wed, Aug 15, 2018 at 12:14 AM, Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
> Hello,
>
> On Wed, 15 Aug 2018 00:05:47 +0200, Mirza Krak wrote:
>
>> I did have one problem with $(2)_SRC_SUBDIR, it turns out to be empty
>> when building host target. Workaround was to explicitly set this in
>> the .mk file with the HOST_ prefix. Any hints on why? Need some ifeq
>> ($(4),target) ... else ... endif  magic? :)
>
> Could you try this change:
>
> diff --git a/package/pkg-golang.mk b/package/pkg-golang.mk
> index bf178622b5..6eacd14180 100644
> --- a/package/pkg-golang.mk
> +++ b/package/pkg-golang.mk
> @@ -77,9 +77,9 @@ $(2)_INSTALL_BINS ?= $(1)
>  # of GOPATH. It usually resolves around domain/vendor/software. By default, we
>  # derive domain/vendor/software from the upstream URL of the project, but we
>  # allow $(2)_SRC_SUBDIR to be overridden if needed.
> -$(2)_SRC_DOMAIN = $$(call domain,$($(2)_SITE))
> -$(2)_SRC_VENDOR = $$(word 1,$$(subst /, ,$$(call notdomain,$($(2)_SITE))))
> -$(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$($(2)_SITE))))
> +$(2)_SRC_DOMAIN = $$(call domain,$$($(2)_SITE))
> +$(2)_SRC_VENDOR = $$(word 1,$$(subst /, ,$$(call notdomain,$$($(2)_SITE))))
> +$(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$$($(2)_SITE))))

Yeah, this works!

-- 
Mirza Krak | Embedded Solutions Architect | https://mender.io

 Northern.tech AS | @northerntechHQ

[Buildroot] pkg-golang: host package not supported

[Arnout Vandecappelle]

On 15-08-18 00:05, Mirza Krak wrote:
> I did have one problem with $(2)_SRC_SUBDIR, it turns out to be empty
> when building host target. Workaround was to explicitly set this in
> the .mk file with the HOST_ prefix. Any hints on why? Need some ifeq
> ($(4),target) ... else ... endif  magic? :)

 I guess you expect _SRC_SUBDIR to be inherited from the target package, so you
need something like this in the golang infra:

ifndef $(2)_SRC_SUBDIR
 ifdef $(3)_SRC_SUBDIR
  $(2)_SRC_SUBDIR = $$($(3)_SRC_SUBDIR)
 else
  $(2)_SRC_SUBDIR ?=
$$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE) endif
endif

 And similar for SRC_DOMAIN, SRC_VENDOR, SRC_SOFTWARE...

 Regards,
 Arnout
-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] pkg-golang: host package not supported

[Mirza Krak]

On Wed, Aug 15, 2018 at 12:37 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
>
>
> On 15-08-18 00:05, Mirza Krak wrote:
>> I did have one problem with $(2)_SRC_SUBDIR, it turns out to be empty
>> when building host target. Workaround was to explicitly set this in
>> the .mk file with the HOST_ prefix. Any hints on why? Need some ifeq
>> ($(4),target) ... else ... endif  magic? :)
>
>  I guess you expect _SRC_SUBDIR to be inherited from the target package, so you
> need something like this in the golang infra:

For this specific package I am not defining a "target" package, host
only. Do not know if that affects the outcome.

>
> ifndef $(2)_SRC_SUBDIR
>  ifdef $(3)_SRC_SUBDIR
>   $(2)_SRC_SUBDIR = $$($(3)_SRC_SUBDIR)
>  else
>   $(2)_SRC_SUBDIR ?=
> $$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE) endif
> endif
>
>  And similar for SRC_DOMAIN, SRC_VENDOR, SRC_SOFTWARE...

To fresh on Buildroot to comment on above :).

-- 
Mirza Krak | Embedded Solutions Architect | https://mender.io

 Northern.tech AS | @northerntechHQ

[Buildroot] pkg-golang: host package not supported

[Arnout Vandecappelle]

On 15-08-18 00:43, Mirza Krak wrote:
> On Wed, Aug 15, 2018 at 12:37 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
>>
>>
>> On 15-08-18 00:05, Mirza Krak wrote:
>>> I did have one problem with $(2)_SRC_SUBDIR, it turns out to be empty
>>> when building host target. Workaround was to explicitly set this in
>>> the .mk file with the HOST_ prefix. Any hints on why? Need some ifeq
>>> ($(4),target) ... else ... endif  magic? :)
>>
>>  I guess you expect _SRC_SUBDIR to be inherited from the target package, so you
>> need something like this in the golang infra:
> 
> For this specific package I am not defining a "target" package, host
> only. Do not know if that affects the outcome.

 As I said, I'm guessing since you didn't provide details...

 You're defining a host-only package, but you still set FOO_SITE instead of
HOST_FOO_SITE, I guess. HOST_FOO_SITE is set automatically by the generic infra,
but only *after* the empty definition has already been used to define
HOST_FOO_SRC_SUBDIR:

$(2)_SRC_DOMAIN = $$(call domain,$($(2)_SITE))
$(2)_SRC_VENDOR = $$(word 1,$$(subst /, ,$$(call notdomain,$($(2)_SITE))))
$(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$($(2)_SITE))))

$(2)_SRC_SUBDIR ?= $$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE)

expands to

HOST_FOO_SRC_DOMAIN = $(call domain,)
HOST_FOO_SRC_VENDOR = $(word 1,$(subst /, ,$(call notdomain,)
HOST_FOO_SRC_SOFTWARE = $(word 2,$(subst /, ,$(call notdomain,)

HOST_FOO_SRC_SUBDIR ?=
$(HOST_FOO_SRC_DOMAIN)/$(HOST_FOO_SRC_VENDOR)/$(HOST_FOO_SRC_SOFTWARE)

and later on you get

$(call inner-generic-package,$(1),$(2),$(3),$(4))

which contains

ifndef $(2)_SITE
 ifdef $(3)_SITE
  $(2)_SITE = $$($(3)_SITE)
 endif
endif

which expands to

HOST_FOO_SITE = $(FOO_SITE)


 Actually, looking at it now, the real problem is that _SRC_DOMAIN definitions
have a single dollar only. Try this:

-$(2)_SRC_DOMAIN = $$(call domain,$($(2)_SITE))
-$(2)_SRC_VENDOR = $$(word 1,$$(subst /, ,$$(call notdomain,$($(2)_SITE))))
-$(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$($(2)_SITE))))
+$(2)_SRC_DOMAIN = $$(call domain,$$($(2)_SITE))
+$(2)_SRC_VENDOR = $$(word 1,$$(subst /, ,$$(call notdomain,$$($(2)_SITE))))
+$(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$$($(2)_SITE))))


 Regards,
 Arnout


>>
>> ifndef $(2)_SRC_SUBDIR
>>  ifdef $(3)_SRC_SUBDIR
>>   $(2)_SRC_SUBDIR = $$($(3)_SRC_SUBDIR)
>>  else
>>   $(2)_SRC_SUBDIR ?=
>> $$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE) endif
>> endif
>>
>>  And similar for SRC_DOMAIN, SRC_VENDOR, SRC_SOFTWARE...
> 
> To fresh on Buildroot to comment on above :).
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] pkg-golang: host package not supported

[Mirza Krak]

On Wed, 15 Aug 2018, 01:13 Arnout Vandecappelle, <arnout@mind.be> wrote:

>
>
> On 15-08-18 00:43, Mirza Krak wrote:
> > On Wed, Aug 15, 2018 at 12:37 AM, Arnout Vandecappelle <arnout@mind.be>
> wrote:
> >>
> >>
> >> On 15-08-18 00:05, Mirza Krak wrote:
> >>> I did have one problem with $(2)_SRC_SUBDIR, it turns out to be empty
> >>> when building host target. Workaround was to explicitly set this in
> >>> the .mk file with the HOST_ prefix. Any hints on why? Need some ifeq
> >>> ($(4),target) ... else ... endif  magic? :)
> >>
> >>  I guess you expect _SRC_SUBDIR to be inherited from the target
> package, so you
> >> need something like this in the golang infra:
> >
> > For this specific package I am not defining a "target" package, host
> > only. Do not know if that affects the outcome.
>
>  As I said, I'm guessing since you didn't provide details...
>
>  You're defining a host-only package, but you still set FOO_SITE instead of
> HOST_FOO_SITE, I guess. HOST_FOO_SITE is set automatically by the generic
> infra,
> but only *after* the empty definition has already been used to define
> HOST_FOO_SRC_SUBDIR:
>
> $(2)_SRC_DOMAIN = $$(call domain,$($(2)_SITE))
> $(2)_SRC_VENDOR = $$(word 1,$$(subst /, ,$$(call notdomain,$($(2)_SITE))))
> $(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call
> notdomain,$($(2)_SITE))))
>
> $(2)_SRC_SUBDIR ?=
> $$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE)
>
> expands to
>
> HOST_FOO_SRC_DOMAIN = $(call domain,)
> HOST_FOO_SRC_VENDOR = $(word 1,$(subst /, ,$(call notdomain,)
> HOST_FOO_SRC_SOFTWARE = $(word 2,$(subst /, ,$(call notdomain,)
>
> HOST_FOO_SRC_SUBDIR ?=
> $(HOST_FOO_SRC_DOMAIN)/$(HOST_FOO_SRC_VENDOR)/$(HOST_FOO_SRC_SOFTWARE)
>
> and later on you get
>
> $(call inner-generic-package,$(1),$(2),$(3),$(4))
>
> which contains
>
> ifndef $(2)_SITE
>  ifdef $(3)_SITE
>   $(2)_SITE = $$($(3)_SITE)
>  endif
> endif
>
> which expands to
>
> HOST_FOO_SITE = $(FOO_SITE)
>
>
>  Actually, looking at it now, the real problem is that _SRC_DOMAIN
> definitions
> have a single dollar only. Try this:
>
> -$(2)_SRC_DOMAIN = $$(call domain,$($(2)_SITE))
> -$(2)_SRC_VENDOR = $$(word 1,$$(subst /, ,$$(call notdomain,$($(2)_SITE))))
> -$(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call
> notdomain,$($(2)_SITE))))
> +$(2)_SRC_DOMAIN = $$(call domain,$$($(2)_SITE))
> +$(2)_SRC_VENDOR = $$(word 1,$$(subst /, ,$$(call
> notdomain,$$($(2)_SITE))))
> +$(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call
> notdomain,$$($(2)_SITE))))
>

Yeah, Thomas suggested this as well and that got rid of the problems I was
having. Is this the correct fix?

/ Mirza
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20180815/fd9006e9/attachment.html>

[Buildroot] pkg-golang: host package not supported

[Arnout Vandecappelle]

On 15-08-18 01:22, Mirza Krak wrote:
> On Wed, 15 Aug 2018, 01:13 Arnout Vandecappelle, <arnout@mind.be> wrote:
[snip]
>>  Actually, looking at it now, the real problem is that _SRC_DOMAIN
>> definitions
>> have a single dollar only. Try this:
>>
>> -$(2)_SRC_DOMAIN = $$(call domain,$($(2)_SITE))
>> -$(2)_SRC_VENDOR = $$(word 1,$$(subst /, ,$$(call notdomain,$($(2)_SITE))))
>> -$(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call
>> notdomain,$($(2)_SITE))))
>> +$(2)_SRC_DOMAIN = $$(call domain,$$($(2)_SITE))
>> +$(2)_SRC_VENDOR = $$(word 1,$$(subst /, ,$$(call
>> notdomain,$$($(2)_SITE))))
>> +$(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call
>> notdomain,$$($(2)_SITE))))
>>
> 
> Yeah, Thomas suggested this as well and that got rid of the problems I was
> having. Is this the correct fix?

 Yes.

 Regards,
 Arnout
-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF