[Buildroot] [PATCH 1/1] package/go: bump to version 1.12

[Buildroot] [PATCH 1/1] package/go: bump to version 1.12

[Christian Stewart]

Go 1.12 was released on 2/25/2019: https://blog.golang.org/go1.12

Additional notes on how Go modules will evolve in 2019 are here:
https://blog.golang.org/modules2019

In Go 1.12, module support remains the same as in Go 1.11. GOPATH is scheduled
for deprecation in 1.13, however. The Buildroot Go package infrastructure should
therefore aim to migrate to a new, currently undefined, Go module friendly
compilation approach before Go 1.13 is released.

Signed-off-by: Christian Stewart <christian@paral.in>
---
 package/go/go.hash | 2 +-
 package/go/go.mk   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/package/go/go.hash b/package/go/go.hash
index 29891399c3..e78cfe410c 100644
--- a/package/go/go.hash
+++ b/package/go/go.hash
@@ -1,3 +1,3 @@
 # From https://golang.org/dl/
-sha256	bc1ef02bb1668835db1390a2e478dcbccb5dd16911691af9d75184bbe5aa943e  go1.11.5.src.tar.gz
+sha256	09c43d3336743866f2985f566db0520b36f4992aea2b4b2fd9f52f17049e88f2  go1.12.src.tar.gz
 sha256	2d36597f7117c38b006835ae7f537487207d8ec407aa9d9980794b2030cbc067  LICENSE
diff --git a/package/go/go.mk b/package/go/go.mk
index 7755a3f785..8709198b63 100644
--- a/package/go/go.mk
+++ b/package/go/go.mk
@@ -4,7 +4,7 @@
 #
 ################################################################################
 
-GO_VERSION = 1.11.5
+GO_VERSION = 1.12
 GO_SITE = https://storage.googleapis.com/golang
 GO_SOURCE = go$(GO_VERSION).src.tar.gz
 
-- 
2.19.2

[Buildroot] [PATCH 1/1] package/go: bump to version 1.12

[Thomas Petazzoni]

Hello Christian,

Thanks for your patch.

On Mon, 25 Feb 2019 19:45:54 -0800
Christian Stewart <christian@paral.in> wrote:

> Go 1.12 was released on 2/25/2019: https://blog.golang.org/go1.12
> 
> Additional notes on how Go modules will evolve in 2019 are here:
> https://blog.golang.org/modules2019
> 
> In Go 1.12, module support remains the same as in Go 1.11. GOPATH is scheduled
> for deprecation in 1.13, however. The Buildroot Go package infrastructure should
> therefore aim to migrate to a new, currently undefined, Go module friendly
> compilation approach before Go 1.13 is released.

I am puzzled by your statement "currently undefined". Is it undefined
in the sense "it is not defined yet how Buildroot will handle this" or
"it is not defined in Go yet how the module compilation will work in
1.13" ?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/1] package/go: bump to version 1.12

[Christian Stewart]

Hi Thomas,

On Mon, Feb 25, 2019 at 11:24 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
> I am puzzled by your statement "currently undefined". Is it undefined
> in the sense "it is not defined yet how Buildroot will handle this" or
> "it is not defined in Go yet how the module compilation will work in
> 1.13" ?

The Buildroot approach to handling it is currently not defined. Go
Modules are built around fetching from the network and verifying
downloads within the Go tool itself, while Buildroot likely will avoid
any network calls outside of the official Buildroot download
mechanism.So, something else will have to be designed to fix this.

Best,
Christian

[Buildroot] [PATCH 1/1] package/go: bump to version 1.12

[Angelo Compagnucci]

Hi Christian,

On Tue, Feb 26, 2019 at 8:46 AM Christian Stewart <christian@paral.in> wrote:
>
> Hi Thomas,
>
> On Mon, Feb 25, 2019 at 11:24 PM Thomas Petazzoni
> <thomas.petazzoni@bootlin.com> wrote:
> > I am puzzled by your statement "currently undefined". Is it undefined
> > in the sense "it is not defined yet how Buildroot will handle this" or
> > "it is not defined in Go yet how the module compilation will work in
> > 1.13" ?
>
> The Buildroot approach to handling it is currently not defined. Go
> Modules are built around fetching from the network and verifying
> downloads within the Go tool itself, while Buildroot likely will avoid
> any network calls outside of the official Buildroot download
> mechanism.So, something else will have to be designed to fix this.

It's less of a problem with module support for go now, because the
dependencies are proxyed and verified before being delivered to the go
download mechanism, so the reproducibility is somewhat guaranteed.
The problem remains for breaking the buildroot internal download
mechanism and for not a so probably clear licensing of downloaded
modules (I have to investigate this).

I proposed the approach of tainting a buildroot configuration in such
cases but the series was not accepted...

>
> Best,
> Christian

[Buildroot] [PATCH 1/1] package/go: bump to version 1.12

[Christian Stewart]

Hi all,

Angelo Compagnucci <angelo@amarulasolutions.com> writes:
>> On Mon, Feb 25, 2019 at 11:24 PM Thomas Petazzoni
>> <thomas.petazzoni@bootlin.com> wrote:
>> > I am puzzled by your statement "currently undefined". Is it undefined
>> > in the sense "it is not defined yet how Buildroot will handle this" or
>> > "it is not defined in Go yet how the module compilation will work in
>> > 1.13" ?
>
> It's less of a problem with module support for go now, because the
> dependencies are proxyed and verified before being delivered to the go
> download mechanism, so the reproducibility is somewhat guaranteed.
> The problem remains for breaking the buildroot internal download
> mechanism and for not a so probably clear licensing of downloaded
> modules (I have to investigate this).

I'm beginning to work on this now, would it be acceptable if Buildroot
adopted the following approach:

 - Remove GOPATH
 - Configure Go to download module files to buildroot/dl/go-modules
 - Set GOPROXY to allow network during download step only.
 - Require go.mod and go.sum hashes for all Go packages.
 - During extract step, run "go mod vendor" to extract code to build/
 - Use vendored building mode "-mod=vendor"

This should allow Go to download and verify modules correctly during the
download step, to the Buildroot dl/ tree, as well as disallowing network
fetching during the build phase, and extracting versioned module code to
the build directory.

Best,
Christian

[Buildroot] [PATCH 1/1] package/go: bump to version 1.12

[Yann E. MORIN]

Christian, All,

On 2019-03-10 01:23 -0800, Christian Stewart spake thusly:
> Angelo Compagnucci <angelo@amarulasolutions.com> writes:
> >> On Mon, Feb 25, 2019 at 11:24 PM Thomas Petazzoni
> >> <thomas.petazzoni@bootlin.com> wrote:
> >> > I am puzzled by your statement "currently undefined". Is it undefined
> >> > in the sense "it is not defined yet how Buildroot will handle this" or
> >> > "it is not defined in Go yet how the module compilation will work in
> >> > 1.13" ?
> >
> > It's less of a problem with module support for go now, because the
> > dependencies are proxyed and verified before being delivered to the go
> > download mechanism, so the reproducibility is somewhat guaranteed.
> > The problem remains for breaking the buildroot internal download
> > mechanism and for not a so probably clear licensing of downloaded
> > modules (I have to investigate this).
> 
> I'm beginning to work on this now, would it be acceptable if Buildroot
> adopted the following approach:
> 
>  - Remove GOPATH
>  - Configure Go to download module files to buildroot/dl/go-modules

How would you get the modules list and their URIs, when we are just
downloading? The package sources are not yet extracted.

>  - Set GOPROXY to allow network during download step only.
>  - Require go.mod and go.sum hashes for all Go packages.

Not sure what go.mod and go.sum are, but if you're suggesting that we
have hashes in foo.hash for all modules in package foo, that would be
nice, yes!

>  - During extract step, run "go mod vendor" to extract code to build/

... getting the modules from $(BR2_DL_DIR) instead of downloading them
that is?

>  - Use vendored building mode "-mod=vendor"
> 
> This should allow Go to download and verify modules correctly during the
> download step, to the Buildroot dl/ tree, as well as disallowing network
> fetching during the build phase, and extracting versioned module code to
> the build directory.

If that were possible, I would like that, yes. That'd be good!

However, I'd suggest that this extra module vendoring dance only occur
if the package has set a specific variable, like we do for git
submodules:

    FOO_GO_MODULES = YES

(I almost wrote GOO_MODULES ;-) )

That way, if a package does not explicitly request them, it would fail
at build time since they would not be available and would not be already
downloaded.

Of course, if we expect the majority of go packages to have modules,
that variable would be moot...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/1] package/go: bump to version 1.12

[Christian Stewart]

Hi Yann,

"Yann E. MORIN" <yann.morin.1998@free.fr> writes:
>> I'm beginning to work on this now, would it be acceptable if Buildroot
>> adopted the following approach:

Here's a brief explanation of what Go modules are from the perspective
of a Buildroot developer.

Reference: https://github.com/golang/go/wiki/Modules

Go code uses a package-based imports system. Imports are relative to a
root directory, previously called GOPATH.

import "github.com/docker/docker/pkg/myutils"

This would resolve to $GOPATH/src/github.com/docker/docker/pkg/myutils.

Note, although Github is in the import, it is not necessary to fetch the
code from Github. This is just a naming convention.

Buildroot packages currently use an additional feature in the Go tool
which allows packages to avoid using GOPATH by "vendoring" dependencies
- copying the code directly into the Git repository.

vendor/github.com/docker/docker/pkg/myutils

Today all of our Buildroot Go modules use "vendor" to avoid downloading
dependencies. But, this requires that the Go projects we add to
Buildroot include all of their dependencies in their repositories, which
is a mess. It also does not allow us the opportunity to validate
dependency changes when upgrading packages in Buildroot.

The Go module system aims to completely replace the GOPATH mechanism by
allowing the Go tool to reason about package versioning, code fetching,
proxying, and checksumming.

A project can contain any number of go.mod files. A go.mod file is akin
to Node's package.json file. It specifies all direct and indirect
dependencies of all Go packages in the subtree below the file. The Go
tool can manage this file automatically if desired, and specifies a
required format. Go.mod additionally requires dependency versions to be
explicitly specified. There is no semantic versioning or asterisk-based
version specifiers.

module mymodule

require (
github.com/aws/aws-sdk-go v1.17.12 // indirect
github.com/blang/semver v3.5.2-0.20180723201105-3c1074078d32+incompatible
)

The Go tool creates a go.sum file next to the go.mod file. The go.sum
file is akin to Node's package-shrinkwrap.json.

github.com/aws/aws-sdk-go v1.15.31/go.mod
h1:mFuSZ37Z9YOHbQEwBWztmVzqXrEkub65tZoCYDt7FT0=
github.com/aws/aws-sdk-go v1.17.12
h1:jMFwRUaM0LcfdenfvbDLePNoWSoCdOHqF4RCvSB4xNQ=
github.com/aws/aws-sdk-go v1.17.12/go.mod
h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=
github.com/blang/semver
v3.5.2-0.20180723201105-3c1074078d32+incompatible
h1:8fBbhRkI5/0ocLFbrhPgnGUm0ogc+Gko1cRodPWDKX4=
github.com/blang/semver
v3.5.2-0.20180723201105-3c1074078d32+incompatible/go.mod
h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk=

The file contains import paths, versions, and file hashes for the
download as well as the contained go.mod file.

With replace directives, it's possible to link together local copies of
modules:

replace (
k8s.io/gengo => ./staging/src/k8s.io/gengo
k8s.io/kube-openapi => ./staging/src/k8s.io/kube-openapi
)

When GO111MODULE=on, all of the Go tool commands become "module-aware."
GOPATH is no longer required. Running "go build" for example will fetch
dependencies from the Internet, checksum them, and extract to a staging
directory (defaulting currently to GOPATH/pkg/gomod or so). Imports are
automatically resolved to the appropriately versioned staging directory.

There are various ways to control the download / extract behavior:

  The GOPROXY environment variable allows further control over the
  download source. If GOPROXY is unset, is the empty string, or is the
  string "direct", downloads use the default direct connection to
  version control systems. Setting GOPROXY to "off" disallows
  downloading modules from any source. Otherwise, GOPROXY is expected to
  be the URL of a module proxy, in which case the go command will fetch
  all modules from that proxy. No matter the source of the modules,
  downloaded modules must match existing entries in go.sum...

  Even when downloading directly from version control systems, the go
  command synthesizes explicit info, mod, and zip files and stores them
  in its local cache, $GOPATH/pkg/mod/cache/download, the same as if it
  had downloaded them directly from a proxy. The cache layout is the
  same as the proxy URL space, so serving $GOPATH/pkg/mod/cache/download
  at (or copying it to) https://example.com/proxy would let other users
  access those cached module versions with
  GOPROXY=https://example.com/proxy.

GOPROXY additionally supports file:// URLs.

---

With the above in mind, here's how I believe we could integrate this
with Buildroot:

>>  - Remove GOPATH

GOPATH is deprecated anyway so this will be necessary. I'd strongly
recommend that Buildroot make a hard requirement that all Go-based
packages use the go.mod and go.sum construction, and configure Go to
ignore the existing vendor/ trees. This would allow us to pin versioning
and tightly control download checksumming through the package.hash ->
go.sum merkle tree.

If a package does not have a go.mod or go.sum file, a patch would be
added to Buildroot to include the files with the correct pinned
dependencies.

>>  - Configure Go to download module files to buildroot/dl/go-modules

We can set the paths such that the $GOPATH/pkg/mod/cache/download
directory resolves to buildroot/dl/go-modules. According to the above
documentation, a Buildroot user could then serve the dl/go-modules
directory directly with a HTTP server and set GOPROXY such that all
module downloads come from that server. This could be configurable
eventually via the Buildroot KConfig architecture.

Initially, the desired behavior would be implemented by setting GOPROXY
to "direct" (or perhaps the Buildroot download mirrors?) during the
downloading phase.

> How would you get the modules list and their URIs, when we are just
> downloading? The package sources are not yet extracted.

This is a good question. There are a couple of options. One conservative
approach is to download and extract, and use the "extract" step to do
the second stage of module downloading after extracting the package
sources.

A more radical approach, which would perhaps be cleaner, would be to
disable the Buildroot download mechanism completely for Go packages,
include the root go.mod and go.sum in Buildroot next to the .mk and
Config files, and allow Go to manage fetching all of the dependencies
including the root package.

>>  - Set GOPROXY to allow network during download step only.

This would be, during the download phase, execute "go mod download"

During the extract phase, we would create the build/package/ dir under
the output directory as usual in Buildroot, then execute "go mod vendor"
to extract the code sources into build/package/vendor

During the build phase, we would then disable the Go moduling system and
compile from the extracted "vendor" tree. This would help improve build
stability and determinism by only engaging the Module code during
download / extract.

>>  - Require go.mod and go.sum hashes for all Go packages.

> Not sure what go.mod and go.sum are, but if you're suggesting that we
> have hashes in foo.hash for all modules in package foo, that would be
> nice, yes!

They are explained above - these files specify the required dependency
versions, and hash the downloaded code artifacts to ensure integrity.

>>  - During extract step, run "go mod vendor" to extract code to build/
> ... getting the modules from $(BR2_DL_DIR) instead of downloading them
> that is?

Exactly.

>>  - Use vendored building mode "-mod=vendor"
>
> If that were possible, I would like that, yes. That'd be good!
>
> However, I'd suggest that this extra module vendoring dance only occur
> if the package has set a specific variable, like we do for git
> submodules:
>
>     FOO_GO_MODULES = YES
>
> (I almost wrote GOO_MODULES ;-) )
>
> That way, if a package does not explicitly request them, it would fail
> at build time since they would not be available and would not be already
> downloaded.

The only situation where we would not want to use Go modules is if the
package has no external dependencies, or if we want to use the
old-fashioned vendor/ tree approach. I can prototype this in a way where
the old behavior will be kept without a FOO_GO_MODULES flag, if this is
something that we want.

> Of course, if we expect the majority of go packages to have modules,
> that variable would be moot...

Best regards,
Christian Stewart

[Buildroot] [PATCH 1/1] package/go: bump to version 1.12

[Arnout Vandecappelle]

 Hi Christian,

 Thanks for this excellent write-up! It certainly clarifies things.

On 11/03/2019 01:08, Christian Stewart wrote:
[snip]
> A project can contain any number of go.mod files. A go.mod file is akin
> to Node's package.json file. It specifies all direct and indirect
> dependencies of all Go packages in the subtree below the file. The Go
> tool can manage this file automatically if desired, and specifies a
> required format. Go.mod additionally requires dependency versions to be
> explicitly specified. There is no semantic versioning or asterisk-based
> version specifiers.
> 
> module mymodule
> 
> require (
> github.com/aws/aws-sdk-go v1.17.12 // indirect
> github.com/blang/semver v3.5.2-0.20180723201105-3c1074078d32+incompatible
> )

 So, since the version is explicit, it's actually completely equivalent to the
vendoring: it's still impossible to treat the modules as packages in the
Buildroot sense, since Buildroot doesn't allow the same package to have
different versions. It's essentially just making our life more difficult :-(.

 That said, at least it's a consistent way for Go packages to work.

 Another thing: the module specification is just a path component, not a full
URL. Is the https:// implicit? Does it always pull from git? Or does it have
some kind of autoconversion similar to our github helper?

[snip]
> With the above in mind, here's how I believe we could integrate this
> with Buildroot:
> 
>>>  - Remove GOPATH
> 
> GOPATH is deprecated anyway so this will be necessary. I'd strongly
> recommend that Buildroot make a hard requirement that all Go-based
> packages use the go.mod and go.sum construction, and configure Go to
> ignore the existing vendor/ trees.

 I've taken a look at the upstream master branch of the go packages we currently
have, and none of them seems to have a go.mod yet... So I think this is a little
bit premature.

> This would allow us to pin versioning
> and tightly control download checksumming through the package.hash ->
> go.sum merkle tree.

 I may miss something, but if it's in vendor/ in the upstream git repo (either
as a submodule or as a copy), it's also tightly controlled and checksummed,
right? So I don't think this is really bringing any value. For us, I mean - I
imagine it *does* bring value for people developing Go packages.


> If a package does not have a go.mod or go.sum file, a patch would be
> added to Buildroot to include the files with the correct pinned
> dependencies.
> 
>>>  - Configure Go to download module files to buildroot/dl/go-modules

 If all modules go together in a single location, there is a risk that there
will be two modules with the same name and version that are in reality different
modules. Or are the full paths repeated there? I.e., does the zip file end up
under a github.com/foo/bar directory?

> 
> We can set the paths such that the $GOPATH/pkg/mod/cache/download
> directory resolves to buildroot/dl/go-modules. According to the above
> documentation, a Buildroot user could then serve the dl/go-modules
> directory directly with a HTTP server and set GOPROXY such that all
> module downloads come from that server. This could be configurable
> eventually via the Buildroot KConfig architecture.

 No need for it to be configurable, it could be the BR2_PRIMARY_SITE.

 However, IIUC, the GOPROXY mechanism doesn't have any fallback. So I guess the
full solution would be to add something like go download infra that first sets
GOPROXY to file://$(BR2_DL_DIR)/go-modules, then to
$(BR2_PRIMARY_SITE)/go-modules, then direct, and finally
$(BR2_SECONDARY_SITE)/go-modules.

 Except that I have another idea, see below.


> Initially, the desired behavior would be implemented by setting GOPROXY
> to "direct" (or perhaps the Buildroot download mirrors?) during the
> downloading phase.
> 
>> How would you get the modules list and their URIs, when we are just
>> downloading? The package sources are not yet extracted.
> 
> This is a good question. There are a couple of options. One conservative
> approach is to download and extract, and use the "extract" step to do
> the second stage of module downloading after extracting the package
> sources.
> 
> A more radical approach, which would perhaps be cleaner, would be to
> disable the Buildroot download mechanism completely for Go packages,
> include the root go.mod and go.sum in Buildroot next to the .mk and
> Config files, and allow Go to manage fetching all of the dependencies
> including the root package.

 That, combined with the go download infra I mentioned above, would indeed be a
feasible approach.

 Ideally, the go.mod would include the package itself too, so we don't need to
download in two steps (i.e. first the package itself, then the modules it uses).
But that might be a bit more difficult.

 But again, see below for an alternative (cliffhanger...)


>>>  - Set GOPROXY to allow network during download step only.
> 
> This would be, during the download phase, execute "go mod download"
> 
> During the extract phase, we would create the build/package/ dir under
> the output directory as usual in Buildroot, then execute "go mod vendor"
> to extract the code sources into build/package/vendor

 Alternatively, we could add a post-extract hook that executes "go mod vendor"
with GOPROXY=file://$(BR2_DL_DIR)/go-modules. But maybe that's what you meant.


> During the build phase, we would then disable the Go moduling system and

 disable == GOPROXY=off, right?

> compile from the extracted "vendor" tree. This would help improve build
> stability and determinism by only engaging the Module code during
> download / extract.

 So this gives me the idea of the alternative...

 If we anyway end up with everything extracted in the vendor tree, we can just
as well do that during the *download* step. Similar like we do for VCS
downloads: we get the *entire* source tree, including go modules, and tar that up.

 We'd still have a go SITE_METHOD and corresponding download helper. This one
would first use another download method to get the base tarball, then extract
it, run 'go mod vendor' with GOPROXY=direct, and create a new tarball.

 This way, the PRIMARY_SITE, SECONDARY_SITE, 'make source', PRIMARY_ONLY, 'make
source-check' would all still work. So it would be (I think) the least invasive
way to introduce this.

[snip]
>> However, I'd suggest that this extra module vendoring dance only occur
>> if the package has set a specific variable, like we do for git
>> submodules:
>>
>>     FOO_GO_MODULES = YES

 With my proposal, this would be FOO_SITE_METHOD = go

 Oh BTW we'd probably also need to support something like FOO_SITE_METHOD =
go+git. dl-wrapper will (should) strip off the first +, the go download helper
would have to handle the second +. But that can be done in a follow-up patch :-)

>>
>> (I almost wrote GOO_MODULES ;-) )
>>
>> That way, if a package does not explicitly request them, it would fail
>> at build time since they would not be available and would not be already
>> downloaded.
> 
> The only situation where we would not want to use Go modules is if the
> package has no external dependencies, or if we want to use the
> old-fashioned vendor/ tree approach. I can prototype this in a way where
> the old behavior will be kept without a FOO_GO_MODULES flag, if this is
> something that we want.

 I think it's worth prototyping something that is not complete yet, and send it
to the list with a lot of comments in the commit message about what is still
missing. That's probably a better basis for discussion that English text.

 Regards,
 Arnout

> 
>> Of course, if we expect the majority of go packages to have modules,
>> that variable would be moot...
> 
> Best regards,
> Christian Stewart
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
> 

[Buildroot] [PATCH 1/1] package/go: bump to version 1.12

[Christian Stewart]

Hi Arnout,

Arnout Vandecappelle <arnout@mind.be> writes:
>  So, since the version is explicit, it's actually completely equivalent to the
> vendoring: it's still impossible to treat the modules as packages in the
> Buildroot sense, since Buildroot doesn't allow the same package to have
> different versions. It's essentially just making our life more difficult :-(.

I don't think it makes sense to treat them as packages in this sense anyway.

>  That said, at least it's a consistent way for Go packages to work.

This also seems to all be in line with the Go developers' intent and
philosophy around dependency management: they prioritize compatibility
and robust operation over deduplicating dependency instances.

> Another thing: the module specification is just a path component, not a full
> URL. Is the https:// implicit? Does it always pull from git? Or does it have
> some kind of autoconversion similar to our github helper?

The Go code fetching mechanisms are complex, but heavily used, and I
would assume they always use https shallow Git clones of the desired
versions. The go.sum checksumming system ensures everything is
consistent as well.

>> GOPATH is deprecated anyway so this will be necessary. I'd strongly
>> recommend that Buildroot make a hard requirement that all Go-based
>> packages use the go.mod and go.sum construction, and configure Go to
>> ignore the existing vendor/ trees.
>
>  I've taken a look at the upstream master branch of the go packages we currently
> have, and none of them seems to have a go.mod yet... So I think this is a little
> bit premature.

Removing GOPATH is not as extreme as it sounds. This is just a
re-organization of the build code to simplify it quite a bit.

Today, the build code tries to synthesize a gopath (pkg-golang.mk):

$(2)_WORKSPACE ?= _gopath

$(2)_SRC_DOMAIN = $$(call domain,$$($(2)_SITE))
$(2)_SRC_VENDOR = $$(word 1,$$(subst /, ,$$(call notdomain,$$($(2)_SITE))))
$(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$$($(2)_SITE))))
$(2)_SRC_SUBDIR ?= $$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE)
$(2)_SRC_PATH = $$(@D)/$$($(2)_WORKSPACE)/src/$$($(2)_SRC_SUBDIR)

mkdir -p $$(dir $$($(2)_SRC_PATH))
ln -sf $$(@D) $$($(2)_SRC_PATH)

GOPATH="$$(@D)/$$($(2)_WORKSPACE)"

This creates the following directory structure:

$(@D)/pkg_gopath/src/github.com/myorg/mypackage/CODE_SYMLINKED_HERE

Then the Go tool is configured to use pkg_gopath as the GOPATH.

With the Go modules system, even if a Go project is not configured to
use Go modules at all, it no longer is necessary to build a GOPATH.

With the absence of a go.mod and go.sum file, and the presence of a
vendor/ dir, the Go tool will now correctly recognize the vendor/ and
build outside GOPATH. This defaults GO111MODULE=on when Go is invoked
outside GOPATH. In Go 1.13, GOPATH will be removed completely (or at
least, deprecated).

In buildroot, we can simplify the code significantly with this new
construction. We can simply extract code to $(@D), enable GO111MODULE,
and the build will function correctly.

>> This would allow us to pin versioning
>> and tightly control download checksumming through the package.hash ->
>> go.sum merkle tree.
>
>  I may miss something, but if it's in vendor/ in the upstream git repo (either
> as a submodule or as a copy), it's also tightly controlled and checksummed,
> right? 

The build would be executed with go build -mod=vendor, which tells the
Go tool to ignore go.mod and use vendor/ only. Then, if vendor/ is
already present in the upstream Git repo, we can skip the "go mod
download" and "go mod vendor" steps.

The result is that the Go tool uses the following directory layout
regardless of if Go modules download/extract was used during the
download phase:

 - $(@D)/vendor: contains all deps extracted
 - $(@D)/go.mod: contains at minimum "module github.com/myorg/mymodule"

> So I don't think this is really bringing any value. For us, I mean - I
> imagine it *does* bring value for people developing Go packages.

If we ignore vendor/ in the upstream Git repository and introduce our
own go.mod, it would allow us to tightly control changes to dependencies
(including security updates) within the Buildroot project, even if
upstream maintainers do not update their vendor trees. This is
especially important when we are using older revisions of projects that
do not maintain release branches with vendored code.

In short, this would allow us to centralize the responsibility of Go
dependency management in the Go module system, store all dependencies in
the GOPROXY mechanism / Buildroot download directory, and force upgrades
of dependencies for older revisions of projects without introducing
large code vendoring patches into the Buildroot tree.

>> If a package does not have a go.mod or go.sum file, a patch would be
>> added to Buildroot to include the files with the correct pinned
>> dependencies.

This seems like a viable solution -

 1. Copy the Buildroot package directory go.mod, go.sum in if exists.
 2. If no go.mod exists in the project, write one with the minimum
 "module github.com/myorg/mymodule" line.
 3. Execute "go mod download" and "go mod vendor" as discussed.

In point #1, if vendor/ already exists, and we indicate we want to
override vendor/, then we can delete the upstream Git vendor/ tree in
this step.

>>>>  - Configure Go to download module files to buildroot/dl/go-modules
>
>  If all modules go together in a single location, there is a risk that there
> will be two modules with the same name and version that are in reality different
> modules. Or are the full paths repeated there? I.e., does the zip file end up
> under a github.com/foo/bar directory?

No. The Go tool uses an intricate path scheme to avoid collisions, and
does not assume integrity of the on-disk cache (as far as I know).

>> We can set the paths such that the $GOPATH/pkg/mod/cache/download
>> directory resolves to buildroot/dl/go-modules. According to the above
>> documentation, a Buildroot user could then serve the dl/go-modules
>> directory directly with a HTTP server and set GOPROXY such that all
>> module downloads come from that server. This could be configurable
>> eventually via the Buildroot KConfig architecture.
>
>  No need for it to be configurable, it could be the BR2_PRIMARY_SITE.

This seems like the best approach.

Here is some reference from go help goproxy:

  A Go module proxy is any web server that can respond to GET requests for
  URLs of a specified form. The requests have no query parameters, so even
  a site serving from a fixed file system (including a file:/// URL)
  can be a module proxy.

  The GET requests sent to a Go module proxy are:

  GET $GOPROXY/<module>/@v/list returns a list of all known versions of the
  given module, one per line.

  GET $GOPROXY/<module>/@v/<version>.info returns JSON-formatted metadata
  about that version of the given module.

  GET $GOPROXY/<module>/@v/<version>.mod returns the go.mod file
  for that version of the given module.

  GET $GOPROXY/<module>/@v/<version>.zip returns the zip archive
  for that version of the given module.

  To avoid problems when serving from case-sensitive file systems,
  the <module> and <version> elements are case-encoded, replacing every
  uppercase letter with an exclamation mark followed by the corresponding
  lower-case letter: github.com/Azure encodes as github.com/!azure.

So perhaps we would want to add some suffix to PRIMARY_SITE.

Something further down the pipe, but worth looking at for those
interested, is the upcoming Go code notary project: 

https://go.googlesource.com/proposal/+/master/design/25530-notary.md

>  However, IIUC, the GOPROXY mechanism doesn't have any fallback. So I guess the
> full solution would be to add something like go download infra that first sets
> GOPROXY to file://$(BR2_DL_DIR)/go-modules, then to
> $(BR2_PRIMARY_SITE)/go-modules, then direct, and finally
> $(BR2_SECONDARY_SITE)/go-modules.

https://github.com/golang/go/issues/26334

Looks like indeed, it doesn't have a fallback yet. However, this is a
bit awkward: what if our GOPROXY doesn't have just a single dependency?

The download tree looks like:

??? github.com
??? ??? davecgh
??? ??? ??? go-spew
??? ???     ??? @v
??? ???         ??? list
??? ???         ??? list.lock
??? ???         ??? v1.1.1.info
??? ???         ??? v1.1.1.mod
??? golang.org
    ??? x
        ??? crypto
        ??? ??? @v
        ???     ??? list
        ???     ??? list.lock
        ???     ??? v0.0.0-20180904163835-0709b304e793.info
        ???     ??? v0.0.0-20180904163835-0709b304e793.lock
        ???     ??? v0.0.0-20180904163835-0709b304e793.mod
        ???     ??? v0.0.0-20180904163835-0709b304e793.zip
        ???     ??? v0.0.0-20180904163835-0709b304e793.ziphash

I tried downloading just one dependency, then copying the
$GOPATH/pkg/mod/cache/download/* files to GOPROXY=file://path/to/copy.

Then, invoking Go build:

go: finding golang.org/x/crypto v0.0.0-20180904163835-0709b304e793
Fetching file://goproxy/golang.org/x/crypto/@v/v0.0.0-20180904163835-0709b304e793.info
Fetching file://goproxy/golang.org/x/crypto/@v/v0.0.0-20180904163835-0709b304e793.mod

Okay, this worked as expected. Now, adding a new dependency:

Fetching file://goproxy/github.com/urfave/cli/@v/list
Fetching file://goproxy/github.com/urfave/@v/list
Fetching file://goproxy/github.com/@v/list
build tmod: cannot load github.com/urfave/cli: cannot find module providing package github.com/urfave/cli

Hmm. Looks like it doesn't have a fallback yet.

Athens is another project addressing this area: https://github.com/gomods/athens

>  Except that I have another idea, see below.

>> A more radical approach, which would perhaps be cleaner, would be to
>> disable the Buildroot download mechanism completely for Go packages,
>> include the root go.mod and go.sum in Buildroot next to the .mk and
>> Config files, and allow Go to manage fetching all of the dependencies
>> including the root package.
>
>  That, combined with the go download infra I mentioned above, would indeed be a
> feasible approach.
>
>  Ideally, the go.mod would include the package itself too, so we don't need to
> download in two steps (i.e. first the package itself, then the modules it uses).
> But that might be a bit more difficult.

This is completely doable.

This approach actually opens up a few other interesting possibilties. We
can add a command to set the environment variables up for the Go tool,
and then execute the tool to automatically maintain the go.mod file for
a particular package.

  With no package arguments, 'go get' applies to the main module, and to
  the Go package in the current directory, if any. In particular, 'go
  get -u' and 'go get -u=patch' update all the dependencies of the main
  module. 

  If invoked with -mod=readonly, the go command is disallowed from the
  implicit automatic updating of go.mod described above. Instead, it
  fails when any changes to go.mod are needed. This setting is most
  useful to check that go.mod does not need updates, such as in a
  continuous integration and testing system. The "go get" command
  remains permitted to update go.mod even with -mod=readonly, and the
  "go mod" commands do not take the -mod flag (or any other build
  flags).

Of course, this is speculative and not necessary for an initial implementation.

>  Alternatively, we could add a post-extract hook that executes "go mod vendor"
> with GOPROXY=file://$(BR2_DL_DIR)/go-modules. But maybe that's what you meant.

This is correct.

>> During the build phase, we would then disable the Go moduling system and
>
>  disable == GOPROXY=off, right?

GOPROXY=off in every situation other than when we want to download code.

Additionally, during build, we would use "-mod=vendor."

  To build using the main module's top-level vendor directory to satisfy
  dependencies (disabling use of the usual network sources and local
  caches), use 'go build -mod=vendor'. Note that only the main module's
  top-level vendor directory is used; vendor directories in other
  locations are still ignored.

>  If we anyway end up with everything extracted in the vendor tree, we can just
> as well do that during the *download* step. Similar like we do for VCS
> downloads: we get the *entire* source tree, including go modules, and tar that up.
>
>  We'd still have a go SITE_METHOD and corresponding download helper. This one
> would first use another download method to get the base tarball, then extract
> it, run 'go mod vendor' with GOPROXY=direct, and create a new tarball.
>
>  This way, the PRIMARY_SITE, SECONDARY_SITE, 'make source', PRIMARY_ONLY, 'make
> source-check' would all still work. So it would be (I think) the least invasive
> way to introduce this.

This seems like the right approach. This all feels similar to how
Buildroot uses the Git tool to clone a repository, check out a revision,
and bundle up a tarball today.

>  With my proposal, this would be FOO_SITE_METHOD = go
>
>  Oh BTW we'd probably also need to support something like FOO_SITE_METHOD =
> go+git. dl-wrapper will (should) strip off the first +, the go download helper
> would have to handle the second +. But that can be done in a follow-up patch :-)

This is way too fancy for me to implement :)

>  I think it's worth prototyping something that is not complete yet, and send it
> to the list with a lot of comments in the commit message about what is still
> missing. That's probably a better basis for discussion that English text.

Something to note: you can set GOMOD variable to the path to a go.mod file.

Okay, I'll have a look at:

 - use vendor/ with GO111MODULE=on, remove gopath
 - use -mod=vendor during build phase
 - set GOCACHE properly (what is it set to today??)
 - set GOPATH such that the Go tool downloads to dl/go-modules/pkg/mod

And (pick one):

 - post-extract hook with optional go.mod in repository, "go mod vendor"
 - additional site method which "go mod vendor" and re-compress to tarball

Best regards,
Christian Stewart

[Buildroot] [PATCH 1/1] package/go: bump to version 1.12

[Arnout Vandecappelle]

On 12/03/2019 04:35, Christian Stewart wrote:
> Hi Arnout,
> 
> Arnout Vandecappelle <arnout@mind.be> writes:
>>  So, since the version is explicit, it's actually completely equivalent to the
>> vendoring: it's still impossible to treat the modules as packages in the
>> Buildroot sense, since Buildroot doesn't allow the same package to have
>> different versions. It's essentially just making our life more difficult :-(.
> 
> I don't think it makes sense to treat them as packages in this sense anyway.

 Actually, below, you do seem to suggest something that goes in that direction...

[snip]
> Removing GOPATH is not as extreme as it sounds. This is just a
> re-organization of the build code to simplify it quite a bit.
> 
> Today, the build code tries to synthesize a gopath (pkg-golang.mk):
> 
> $(2)_WORKSPACE ?= _gopath
> 
> $(2)_SRC_DOMAIN = $$(call domain,$$($(2)_SITE))
> $(2)_SRC_VENDOR = $$(word 1,$$(subst /, ,$$(call notdomain,$$($(2)_SITE))))
> $(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$$($(2)_SITE))))
> $(2)_SRC_SUBDIR ?= $$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE)
> $(2)_SRC_PATH = $$(@D)/$$($(2)_WORKSPACE)/src/$$($(2)_SRC_SUBDIR)
> 
> mkdir -p $$(dir $$($(2)_SRC_PATH))
> ln -sf $$(@D) $$($(2)_SRC_PATH)
> 
> GOPATH="$$(@D)/$$($(2)_WORKSPACE)"
> 
> This creates the following directory structure:
> 
> $(@D)/pkg_gopath/src/github.com/myorg/mypackage/CODE_SYMLINKED_HERE
> 
> Then the Go tool is configured to use pkg_gopath as the GOPATH.
> 
> With the Go modules system, even if a Go project is not configured to
> use Go modules at all, it no longer is necessary to build a GOPATH.
> 
> With the absence of a go.mod and go.sum file, and the presence of a
> vendor/ dir, the Go tool will now correctly recognize the vendor/ and
> build outside GOPATH. This defaults GO111MODULE=on when Go is invoked
> outside GOPATH. In Go 1.13, GOPATH will be removed completely (or at
> least, deprecated).
> 
> In buildroot, we can simplify the code significantly with this new
> construction. We can simply extract code to $(@D), enable GO111MODULE,
> and the build will function correctly.

 Hell yeah! As a first step, that sounds like a great move. So we'd have:

1. Bump package/go
2. Set GO111MODULE=on and remove GOPATH from pkg-golang
3. Remove now-unused PKG_SRC_{DOMAIN,VENDOR,...} from packages
4. Remove this stuff from documentation

 Step 2 should probably also set GOPROXY=off in all steps to avoid accidental
downloads. Maybe we should even export GOPROXY=off and enable it only in the
specific cases where it needs to be enabled? That would be a fifth patch.


>>> This would allow us to pin versioning
>>> and tightly control download checksumming through the package.hash ->
>>> go.sum merkle tree.
>>
>>  I may miss something, but if it's in vendor/ in the upstream git repo (either
>> as a submodule or as a copy), it's also tightly controlled and checksummed,
>> right? 
> 
> The build would be executed with go build -mod=vendor, which tells the
> Go tool to ignore go.mod and use vendor/ only. Then, if vendor/ is
> already present in the upstream Git repo, we can skip the "go mod
> download" and "go mod vendor" steps.

 So that would be the current sitatuation then.


> The result is that the Go tool uses the following directory layout
> regardless of if Go modules download/extract was used during the
> download phase:
> 
>  - $(@D)/vendor: contains all deps extracted
>  - $(@D)/go.mod: contains at minimum "module github.com/myorg/mymodule"

 Is this go.mod then autogenerated by 'go build -mod=vendor' or do we need to
generate it in a post-patch hook?


>> So I don't think this is really bringing any value. For us, I mean - I
>> imagine it *does* bring value for people developing Go packages.
> 
> If we ignore vendor/ in the upstream Git repository and introduce our
> own go.mod, it would allow us to tightly control changes to dependencies
> (including security updates) within the Buildroot project, even if
> upstream maintainers do not update their vendor trees. This is
> especially important when we are using older revisions of projects that
> do not maintain release branches with vendored code.
>
> In short, this would allow us to centralize the responsibility of Go
> dependency management in the Go module system, store all dependencies in
> the GOPROXY mechanism / Buildroot download directory, and force upgrades
> of dependencies for older revisions of projects without introducing
> large code vendoring patches into the Buildroot tree.

 It sounds to me like a bit too big of an endeavor for the small Buildroot
project to take on... Upstream Go just doesn't have a decent security story for
not-well-maintained packages, and I don't think that there's much we can do
about it.


>>> If a package does not have a go.mod or go.sum file, a patch would be
>>> added to Buildroot to include the files with the correct pinned
>>> dependencies.
> 
> This seems like a viable solution -
> 
>  1. Copy the Buildroot package directory go.mod, go.sum in if exists.
>  2. If no go.mod exists in the project, write one with the minimum
>  "module github.com/myorg/mymodule" line.
>  3. Execute "go mod download" and "go mod vendor" as discussed.
> 
> In point #1, if vendor/ already exists, and we indicate we want to
> override vendor/, then we can delete the upstream Git vendor/ tree in
> this step.

 Yes, sounds good to me. Though I would first like to see if there actually is
any project that has both a go.mod and an existing vendor/ tree.

 What does "go mod vendor" do if a vendor/ tree already exists? Maybe we just
have to keep that behaviour, i.e. not delete the existing vendor tree and let
the go tool take care of things. It should in principle do the right thing.


>>>>>  - Configure Go to download module files to buildroot/dl/go-modules
>>
>>  If all modules go together in a single location, there is a risk that there
>> will be two modules with the same name and version that are in reality different
>> modules. Or are the full paths repeated there? I.e., does the zip file end up
>> under a github.com/foo/bar directory?
> 
> No. The Go tool uses an intricate path scheme to avoid collisions, and
> does not assume integrity of the on-disk cache (as far as I know).
> 
>>> We can set the paths such that the $GOPATH/pkg/mod/cache/download
>>> directory resolves to buildroot/dl/go-modules. According to the above
>>> documentation, a Buildroot user could then serve the dl/go-modules
>>> directory directly with a HTTP server and set GOPROXY such that all
>>> module downloads come from that server. This could be configurable
>>> eventually via the Buildroot KConfig architecture.
>>
>>  No need for it to be configurable, it could be the BR2_PRIMARY_SITE.
> 
> This seems like the best approach.
> 
> Here is some reference from go help goproxy:
> 
>   A Go module proxy is any web server that can respond to GET requests for
>   URLs of a specified form. The requests have no query parameters, so even
>   a site serving from a fixed file system (including a file:/// URL)
>   can be a module proxy.
> 
>   The GET requests sent to a Go module proxy are:
> 
>   GET $GOPROXY/<module>/@v/list returns a list of all known versions of the
>   given module, one per line.
> 
>   GET $GOPROXY/<module>/@v/<version>.info returns JSON-formatted metadata
>   about that version of the given module.
> 
>   GET $GOPROXY/<module>/@v/<version>.mod returns the go.mod file
>   for that version of the given module.
> 
>   GET $GOPROXY/<module>/@v/<version>.zip returns the zip archive
>   for that version of the given module.
> 
>   To avoid problems when serving from case-sensitive file systems,
>   the <module> and <version> elements are case-encoded, replacing every
>   uppercase letter with an exclamation mark followed by the corresponding
>   lower-case letter: github.com/Azure encodes as github.com/!azure.
> 
> So perhaps we would want to add some suffix to PRIMARY_SITE.

 That's what you proposed: go-modules.

 But I'm not sure now: does the go mod download generate these auxiliary files
automatically in pkg/mod/cache/download? I.e. can we just serve $(BR2_DL_DIR) as
PRIMARY, set GOPROXY to $(BR2_PRIMARY_SITE)/go-modules, and Bob's your uncle?

> Something further down the pipe, but worth looking at for those
> interested, is the upcoming Go code notary project: 
> 
> https://go.googlesource.com/proposal/+/master/design/25530-notary.md
> 
>>  However, IIUC, the GOPROXY mechanism doesn't have any fallback. So I guess the
>> full solution would be to add something like go download infra that first sets
>> GOPROXY to file://$(BR2_DL_DIR)/go-modules, then to
>> $(BR2_PRIMARY_SITE)/go-modules, then direct, and finally
>> $(BR2_SECONDARY_SITE)/go-modules.
> 
> https://github.com/golang/go/issues/26334
> 
> Looks like indeed, it doesn't have a fallback yet. However, this is a
> bit awkward: what if our GOPROXY doesn't have just a single dependency?
> 
> The download tree looks like:
> 
> ??? github.com
> ??? ??? davecgh
> ??? ??? ??? go-spew
> ??? ???     ??? @v
> ??? ???         ??? list
> ??? ???         ??? list.lock
> ??? ???         ??? v1.1.1.info
> ??? ???         ??? v1.1.1.mod
> ??? golang.org
>     ??? x
>         ??? crypto
>         ??? ??? @v
>         ???     ??? list
>         ???     ??? list.lock
>         ???     ??? v0.0.0-20180904163835-0709b304e793.info
>         ???     ??? v0.0.0-20180904163835-0709b304e793.lock
>         ???     ??? v0.0.0-20180904163835-0709b304e793.mod
>         ???     ??? v0.0.0-20180904163835-0709b304e793.zip
>         ???     ??? v0.0.0-20180904163835-0709b304e793.ziphash
> 
> I tried downloading just one dependency, then copying the
> $GOPATH/pkg/mod/cache/download/* files to GOPROXY=file://path/to/copy.
> 
> Then, invoking Go build:
> 
> go: finding golang.org/x/crypto v0.0.0-20180904163835-0709b304e793
> Fetching file://goproxy/golang.org/x/crypto/@v/v0.0.0-20180904163835-0709b304e793.info
> Fetching file://goproxy/golang.org/x/crypto/@v/v0.0.0-20180904163835-0709b304e793.mod
> 
> Okay, this worked as expected. Now, adding a new dependency:
> 
> Fetching file://goproxy/github.com/urfave/cli/@v/list
> Fetching file://goproxy/github.com/urfave/@v/list
> Fetching file://goproxy/github.com/@v/list
> build tmod: cannot load github.com/urfave/cli: cannot find module providing package github.com/urfave/cli
> 
> Hmm. Looks like it doesn't have a fallback yet.

 Heh, I didn't think of that: the tool tries to download *all* modules...

 What would help is if there was some kind of keep-going option. Then we can do
all the fallback on the Buildroot side, like I mentioned: first try with PRIMARY
(and download some dependencies and put it in the cache), then try with direct,
then try with SECONDARY. The last one should succeed fully, reusing things from
the cache if they were already downloaded before, and fetching them from direct
or secondary if not.

> 
> Athens is another project addressing this area: https://github.com/gomods/athens

[snip]
>>  If we anyway end up with everything extracted in the vendor tree, we can just
>> as well do that during the *download* step. Similar like we do for VCS
>> downloads: we get the *entire* source tree, including go modules, and tar that up.
>>
>>  We'd still have a go SITE_METHOD and corresponding download helper. This one
>> would first use another download method to get the base tarball, then extract
>> it, run 'go mod vendor' with GOPROXY=direct, and create a new tarball.
>>
>>  This way, the PRIMARY_SITE, SECONDARY_SITE, 'make source', PRIMARY_ONLY, 'make
>> source-check' would all still work. So it would be (I think) the least invasive
>> way to introduce this.
> 
> This seems like the right approach. This all feels similar to how
> Buildroot uses the Git tool to clone a repository, check out a revision,
> and bundle up a tarball today.

 And I realize now: it would also handle legal-info, which would not be handled
correctly with everything that was proposed up to now (because the go modules
are not part of $(PKG)_ALL_DOWNLOADS, they don't get copied to legal-info output).

> 
>>  With my proposal, this would be FOO_SITE_METHOD = go
>>
>>  Oh BTW we'd probably also need to support something like FOO_SITE_METHOD =
>> go+git. dl-wrapper will (should) strip off the first +, the go download helper
>> would have to handle the second +. But that can be done in a follow-up patch :-)
> 
> This is way too fancy for me to implement :)

 This would in fact not be needed if we would have the go.mod in the Buildroot
tree and use go mod to get the source itself too.

> 
>>  I think it's worth prototyping something that is not complete yet, and send it
>> to the list with a lot of comments in the commit message about what is still
>> missing. That's probably a better basis for discussion that English text.
> 
> Something to note: you can set GOMOD variable to the path to a go.mod file.
> 
> Okay, I'll have a look at:
> 
>  - use vendor/ with GO111MODULE=on, remove gopath

 That one is not controversial for sure.

>  - use -mod=vendor during build phase

 Ditto. It should essentially change nothing for our current packages, right?

>  - set GOCACHE properly (what is it set to today??)

 You already sent a patch for that :-)

>  - set GOPATH such that the Go tool downloads to dl/go-modules/pkg/mod

 That one seems useless - with the steps above, there should never be any
download done by go mod.


> And (pick one):
> 
>  - post-extract hook with optional go.mod in repository, "go mod vendor"
>  - additional site method which "go mod vendor" and re-compress to tarball

 So these are the more controversial ones. I think you should pick the one you
like most, make a quick prototype that works for one specific package, note what
is missing in the commit log, and send it to the list for discussion.

 Thanks for this work!

 Regards,
 Arnout