[Buildroot] [PATCH v3 0/7] New infrastructure for golang packages

[Buildroot] [PATCH v3 0/7] New infrastructure for golang packages

[Angelo Compagnucci]

This series add a new pkg-golang infrastructure and converts some of the
packages based on go.

Changelog:

v2 -> v3:
* Fixed a bug with GOBIN variable
* Fixed mender package

v1 -> v2:
* Completed the conversion of all golang packages to the new infrastructure
adding docker-engine
* Added a new package. mender

Angelo Compagnucci (7):
  package/pkg-golang: new package infrastructure
  docs/manual: adding documentation for the golang infrastructure
  package/flannel: converting to golang infrastructure
  package/runc: converting to golang infrastructure
  package/docker-containerd: converting to golang infrastructure
  package/docker-engine: converting to golang infrastructure
  package/mender: new package

 docs/manual/adding-packages-golang.txt         | 116 +++++++++++++++++++++++
 docs/manual/adding-packages.txt                |   2 +
 package/Config.in                              |   1 +
 package/Makefile.in                            |   1 +
 package/docker-containerd/docker-containerd.mk |  37 +++-----
 package/docker-engine/docker-engine.mk         |  53 ++++-------
 package/flannel/flannel.mk                     |  32 +------
 package/mender/Config.in                       |  14 +++
 package/mender/mender-device-identity          |  52 +++++++++++
 package/mender/mender-inventory-hostinfo       |  21 +++++
 package/mender/mender-inventory-network        |  47 ++++++++++
 package/mender/mender.conf                     |  14 +++
 package/mender/mender.hash                     |   2 +
 package/mender/mender.mk                       |  39 ++++++++
 package/mender/mender.service                  |  15 +++
 package/mender/server.crt                      |  22 +++++
 package/mender/tenant.conf                     |   0
 package/pkg-golang.mk                          | 123 +++++++++++++++++++++++++
 package/runc/runc.mk                           |  37 +-------
 19 files changed, 510 insertions(+), 118 deletions(-)
 create mode 100644 docs/manual/adding-packages-golang.txt
 create mode 100644 package/mender/Config.in
 create mode 100644 package/mender/mender-device-identity
 create mode 100644 package/mender/mender-inventory-hostinfo
 create mode 100644 package/mender/mender-inventory-network
 create mode 100644 package/mender/mender.conf
 create mode 100644 package/mender/mender.hash
 create mode 100644 package/mender/mender.mk
 create mode 100644 package/mender/mender.service
 create mode 100644 package/mender/server.crt
 create mode 100644 package/mender/tenant.conf
 create mode 100644 package/pkg-golang.mk

-- 
2.7.4

[Buildroot] [PATCH v3 1/7] package/pkg-golang: new package infrastructure

[Angelo Compagnucci]

This patch adds a new infrastructure for golang based packages.

Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com>
---
Changes:
v2 -> v3:
* If GOBIN variable is already present in environment it
  will used by the go command for compiling and fails
  with the error 
  "cannot install cross-compiled binaries when GOBIN is set"
v1 -> v2:
* Renamed GOBIN to GO_BIN for clarity

 package/Makefile.in   |   1 +
 package/pkg-golang.mk | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 124 insertions(+)
 create mode 100644 package/pkg-golang.mk

diff --git a/package/Makefile.in b/package/Makefile.in
index e387ce6..379af5d 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -441,3 +441,4 @@ include package/pkg-kconfig.mk
 include package/pkg-rebar.mk
 include package/pkg-kernel-module.mk
 include package/pkg-waf.mk
+include package/pkg-golang.mk
diff --git a/package/pkg-golang.mk b/package/pkg-golang.mk
new file mode 100644
index 0000000..370db07
--- /dev/null
+++ b/package/pkg-golang.mk
@@ -0,0 +1,123 @@
+################################################################################
+# Golang package infrastructure
+#
+# This file implements an infrastructure that eases development of
+# package .mk files for Go packages. It should be used for all
+# packages that are written in go.
+#
+# See the Buildroot documentation for details on the usage of this
+# infrastructure
+#
+#
+# In terms of implementation, this golang infrastructure requires
+# the .mk file to only specify metadata information about the
+# package: name, version, download URL, etc.
+#
+# We still allow the package .mk file to override what the different
+# steps are doing, if needed. For example, if <PKG>_BUILD_CMDS is
+# already defined, it is used as the list of commands to perform to
+# build the package, instead of the default golang behavior. The
+# package can also define some post operation hooks.
+#
+################################################################################
+
+unexport GOBIN
+
+GO_BIN = $(HOST_DIR)/bin/go
+
+################################################################################
+# inner-golang-package -- defines how the configuration, compilation and
+# installation of a Go package should be done, implements a few hooks to
+# tune the build process for Go specificities and calls the generic package
+# infrastructure to generate the necessary make targets
+#
+#  argument 1 is the lowercase package name
+#  argument 2 is the uppercase package name, including a HOST_ prefix
+#             for host packages
+#  argument 3 is the uppercase package name, without the HOST_ prefix
+#             for host packages
+#  argument 4 is the type (target or host)
+################################################################################
+
+define inner-golang-package
+
+$(2)_GOPATH ?= _gopath
+$(2)_GO_ENV ?= $$(HOST_GO_TARGET_ENV) \
+		GOPATH="$$(@D)/$$($(2)_GOPATH)" \
+		CGO_ENABLED=$$(HOST_GO_CGO_ENABLED)
+
+ifeq ($(BR2_STATIC_LIBS),y)
+	$(2)_GO_LDFLAGS += -extldflags '-static'
+endif
+
+ifdef $(2)_GO_LDFLAGS
+	$(2)_BUILD_OPTS += -ldflags "$$($(2)_GO_LDFLAGS)"
+endif
+
+ifdef $(2)_GO_TAGS
+	$(2)_BUILD_OPTS += -tags "$$($(2)_GO_TAGS)"
+endif
+
+# Target packages need the Go compiler on the host.
+$(2)_DEPENDENCIES += host-go
+
+#
+# The go build/install command installs the binaries inside
+# gopath/bin/linux_GOARCH/ when cross compilation is enabled.
+# We set this variable here to be used by packages if needed.
+#
+$(2)_BINDIR = $$($(2)_GOPATH)/bin/linux_$$(GO_GOARCH)
+
+#
+# Source files in Go should be uncompressed in a precise folder in the
+# hierarchy of GOPATH. It usually resolves around domain/vendor/software.
+#
+$(2)_GO_SRC_PATH ?= $$(call domain,$($(2)_SITE))/$$(firstword $$(subst /, ,$$(call notdomain,$($(2)_SITE))))
+$(2)_SRC_PATH  = $$(@D)/$$($(2)_GOPATH)/src/$$($(2)_GO_SRC_PATH)/$(1)
+
+#
+# Configure step. Only define it if not already defined by the package
+# .mk file.
+#
+ifndef $(2)_CONFIGURE_CMDS
+define $(2)_CONFIGURE_CMDS
+	mkdir -p $$(@D)/$$($(2)_GOPATH)/bin
+	mkdir -p $$(@D)/$$($(2)_GOPATH)/src/$$($(2)_GO_SRC_PATH)
+	ln -sf $$(@D) $$($(2)_SRC_PATH)
+endef
+endif
+
+#
+# Build step. Only define it if not already defined by the package .mk file.
+# We use the install command instead of build command here because the
+# install command also moves the package binaries in gopath/bin/linux_GOARCH/.
+# Using the install command also leverages the go build infrastructure
+# for building and installing multiple binaries.
+#
+ifndef $(2)_BUILD_CMDS
+define $(2)_BUILD_CMDS
+	cd $$($(2)_SRC_PATH) && $$($(2)_GO_ENV) $$(GO_BIN) install -v $$($(2)_BUILD_OPTS)
+endef
+endif
+
+#
+# Target installation step. Only define it if not already defined by the
+# package .mk file.
+#
+ifndef $(2)_INSTALL_TARGET_CMDS
+define $(2)_INSTALL_TARGET_CMDS
+	$(INSTALL) -D -m 0755 $$(@D)/$$($(2)_BINDIR)/$(1) $(TARGET_DIR)/usr/bin/
+endef
+endif
+
+# Call the generic package infrastructure to generate the necessary make
+# targets
+$(call inner-generic-package,$(1),$(2),$(3),$(4))
+
+endef # inner-golang-package
+
+################################################################################
+# golang-package -- the target generator macro for Go packages
+################################################################################
+
+golang-package = $(call inner-golang-package,$(pkgname),$(call UPPERCASE,$(pkgname)),$(call UPPERCASE,$(pkgname)),target)
-- 
2.7.4

[Buildroot] [PATCH v3 2/7] docs/manual: adding documentation for the golang infrastructure

[Angelo Compagnucci]

This patch adds the documentation for the golang infrastructure.

Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com>
---
 docs/manual/adding-packages-golang.txt | 116 +++++++++++++++++++++++++++++++++
 docs/manual/adding-packages.txt        |   2 +
 2 files changed, 118 insertions(+)
 create mode 100644 docs/manual/adding-packages-golang.txt

diff --git a/docs/manual/adding-packages-golang.txt b/docs/manual/adding-packages-golang.txt
new file mode 100644
index 0000000..5d6de3a
--- /dev/null
+++ b/docs/manual/adding-packages-golang.txt
@@ -0,0 +1,116 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+=== Infrastructure for Go packages
+
+This infrastructure applies to Go packages that use the standard
+build system and use bundled dependencies.
+
+[[golang-package-tutorial]]
+
+==== +golang-package+ tutorial
+
+First, let's see how to write a +.mk+ file for a go package,
+with an example :
+
+------------------------
+01: ################################################################################
+02: #
+03: # go-foo
+04: #
+05: ################################################################################
+06:
+07: GO_FOO_VERSION = 1.0
+08: GO_FOO_SOURCE = go-foo-$(GO_FOO_VERSION).tar.xz
+09: GO_FOO_SITE = http://www.foosoftware.org/download
+10: GO_FOO_LICENSE = BSD-3-Clause
+11: GO_FOO_LICENSE_FILES = LICENSE
+12:
+13: $(eval $(golang-package))
+------------------------
+
+On line 7, we declare the version of the package.
+
+On line 8 and 9, we declare the name of the tarball (xz-ed tarball
+recommended) and the location of the tarball on the Web. Buildroot
+will automatically download the tarball from this location.
+
+On line 10 and 11, we give licensing details about the package (its
+license on line 10, and the file containing the license text on line
+11).
+
+Finally, on line 13, we invoke the +golang-package+ macro that
+generates all the Makefile rules that actually allow the package to be
+built.
+
+[[golang-package-reference]]
+
+==== +golang-package+ reference
+
+As a policy packages can freely choose their name (existing example in 
+Buildroot is +flannel+).
+
+In their +Config.in+ file, they should depend on 
++BR2_PACKAGE_HOST_GO_ARCH_SUPPORTS+ and
++BR2_PACKAGE_HOST_GO_CGO_LINKING_SUPPORTS+ cause host-go will compile when Buildroot will add the
+dependency automatically.
+
+The main macro of the Go package infrastructure is
++golang-package+. It is similar to the +generic-package+ macro.
+
+Just like the generic infrastructure, the Go infrastructure works
+by defining a number of variables before calling the +golang-package+.
+
+All the package metadata information variables that exist in the
+xref:generic-package-reference[generic package infrastructure] also
+exist in the Go infrastructure: +GO_FOO_VERSION+,
++GO_FOO_SOURCE+, +GO_FOO_PATCH+, +GO_FOO_SITE+,
++GO_FOO_SUBDIR+, +GO_FOO_DEPENDENCIES+, +GO_FOO_LICENSE+,
++GO_FOO_LICENSE_FILES+, +GO_FOO_INSTALL_STAGING+, etc.
+
+Note that:
+
+ * It is not necessary to add +go+ or +host-go+ in the
+   +GO_FOO_DEPENDENCIES+ variable of a package, since these basic
+   dependencies are automatically added as needed by the Go
+   package infrastructure.
+
+A few additional variables, specific to the Go infrastructure, can
+optionally be defined, depending on the package's needs. Many of them
+are only useful in very specific cases, typical packages will
+therefore only use a few of them, or none.
+
+* If your package need a custom GOPATH to be compiled in, you can use the
+  +GO_FOO_GOPATH+.
+
+* +GO_FOO_GO_SRC_PATH+ is the path where your source will be compiled
+  relatively to the GOPATH.
+  The golang package infrastructure tries to guess the correct 
+  sub folder to compile in but if guessing is not correct for you or your
+  package behaves differently, you can use this variable to
+  redefine the path.
+
+* +GOO_FOO_GO_LDFLAGS+ and +GO_FOO_GO_TAGS+ can be used to pass
+  respectively the LDFLAGS or the TAGS to the build command.
+  +GOO_FOO_GO_LDFLAGS+ and +GO_FOO_GO_TAGS+ will be combined into
+  the proper command line options to be passed to the compiler.
+
+* If you need to customize the install location for you binaries, you
+  can use +GOO_FOO_BINDIR+ to know where the binaries are.
+
+* If you need to customize the building tags you can use GOO_FOO_GO_TAGS
+
+With the Go infrastructure, all the steps required to build and
+install the packages are already defined, and they generally work well
+for most Go-based packages. However, when required, it is still
+possible to customize what is done in any particular step:
+
+* By adding a post-operation hook (after extract, patch, configure,
+  build or install). See xref:hooks[] for details.
+
+* By overriding one of the steps. For example, even if the Go
+  infrastructure is used, if the package +.mk+ file defines its own
+  +GO_FOO_BUILD_CMDS+ variable, it will be used instead of the
+  default Go one. However, using this method should be restricted
+  to very specific cases. Do not use it in the general case.
+
diff --git a/docs/manual/adding-packages.txt b/docs/manual/adding-packages.txt
index e8d40da..c642146 100644
--- a/docs/manual/adding-packages.txt
+++ b/docs/manual/adding-packages.txt
@@ -38,6 +38,8 @@ include::adding-packages-meson.txt[]
 
 include::adding-packages-cargo.txt[]
 
+include::adding-packages-golang.txt[]
+
 include::adding-packages-kernel-module.txt[]
 
 include::adding-packages-asciidoc.txt[]
-- 
2.7.4

[Buildroot] [PATCH v3 3/7] package/flannel: converting to golang infrastructure

[Angelo Compagnucci]

This patch converts the flannel package to the new golang
infrastructure.

Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com>
---
 package/flannel/flannel.mk | 32 ++++----------------------------
 1 file changed, 4 insertions(+), 28 deletions(-)

diff --git a/package/flannel/flannel.mk b/package/flannel/flannel.mk
index bbb2c72..78f9a09 100644
--- a/package/flannel/flannel.mk
+++ b/package/flannel/flannel.mk
@@ -11,36 +11,12 @@ FLANNEL_SOURCE = $(FLANNEL_VERSION).tar.gz
 FLANNEL_LICENSE = Apache-2.0
 FLANNEL_LICENSE_FILES = LICENSE
 
-FLANNEL_DEPENDENCIES = host-go
-
-FLANNEL_MAKE_ENV = \
-	$(HOST_GO_TARGET_ENV) \
-	GOBIN="$(@D)/bin" \
-	GOPATH="$(@D)/gopath" \
-	CGO_ENABLED=1
-
-FLANNEL_GLDFLAGS = \
-	-X github.com/coreos/flannel/version.Version=$(FLANNEL_VERSION)
-
-ifeq ($(BR2_STATIC_LIBS),y)
-FLANNEL_GLDFLAGS += -extldflags '-static'
-endif
-
-define FLANNEL_CONFIGURE_CMDS
-	# Put sources at prescribed GOPATH location.
-	mkdir -p $(@D)/gopath/src/github.com/coreos
-	ln -s $(@D) $(@D)/gopath/src/github.com/coreos/flannel
-endef
-
-define FLANNEL_BUILD_CMDS
-	cd $(@D) && $(FLANNEL_MAKE_ENV) $(HOST_DIR)/bin/go \
-		build -v -o $(@D)/bin/flanneld -ldflags "$(FLANNEL_GLDFLAGS)" .
-endef
+FLANNEL_GO_LDFLAGS = -X github.com/coreos/flannel/version.Version=$(FLANNEL_VERSION)
 
+# Install flannel to its well known location.
 define FLANNEL_INSTALL_TARGET_CMDS
-	# Install flannel to its well known location.
-	$(INSTALL) -D -m 0755 $(@D)/bin/flanneld $(TARGET_DIR)/opt/bin/flanneld
+	$(INSTALL) -D -m 0755 $(@D)/$(FLANNEL_BINDIR)/flannel $(TARGET_DIR)/opt/bin/flanneld
 	$(INSTALL) -D -m 0755 $(@D)/dist/mk-docker-opts.sh $(TARGET_DIR)/opt/bin/mk-docker-opts.sh
 endef
 
-$(eval $(generic-package))
+$(eval $(golang-package))
-- 
2.7.4

[Buildroot] [PATCH v3 4/7] package/runc: converting to golang infrastructure

[Angelo Compagnucci]

This patch converts the runc package to the new golang
infrastructure.

Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com>
---
 package/runc/runc.mk | 37 +++++--------------------------------
 1 file changed, 5 insertions(+), 32 deletions(-)

diff --git a/package/runc/runc.mk b/package/runc/runc.mk
index fb3fec2..f497b6d 100644
--- a/package/runc/runc.mk
+++ b/package/runc/runc.mk
@@ -9,42 +9,15 @@ RUNC_SITE = $(call github,opencontainers,runc,$(RUNC_VERSION))
 RUNC_LICENSE = Apache-2.0
 RUNC_LICENSE_FILES = LICENSE
 
-RUNC_DEPENDENCIES = host-go
+RUNC_GOPATH = Godeps/_workspace
 
-RUNC_GOPATH = "$(@D)/Godeps/_workspace"
-RUNC_MAKE_ENV = $(HOST_GO_TARGET_ENV) \
-	CGO_ENABLED=1 \
-	GOBIN="$(@D)/bin" \
-	GOPATH="$(RUNC_GOPATH)" \
-	PATH=$(BR_PATH)
+RUNC_GO_LDFLAGS = -X main.gitCommit=$(RUNC_VERSION)
 
-RUNC_GLDFLAGS = \
-	-X main.gitCommit=$(RUNC_VERSION)
-
-ifeq ($(BR2_STATIC_LIBS),y)
-RUNC_GLDFLAGS += -extldflags '-static'
-endif
-
-RUNC_GOTAGS = cgo static_build
+RUNC_GO_TAGS = cgo static_build
 
 ifeq ($(BR2_PACKAGE_LIBSECCOMP),y)
-RUNC_GOTAGS += seccomp
+RUNC_GO_TAGS += seccomp
 RUNC_DEPENDENCIES += libseccomp host-pkgconf
 endif
 
-define RUNC_CONFIGURE_CMDS
-	mkdir -p $(RUNC_GOPATH)/src/github.com/opencontainers
-	ln -s $(@D) $(RUNC_GOPATH)/src/github.com/opencontainers/runc
-endef
-
-define RUNC_BUILD_CMDS
-	cd $(@D) && $(RUNC_MAKE_ENV) $(HOST_DIR)/bin/go \
-		build -v -o $(@D)/bin/runc \
-		-tags "$(RUNC_GOTAGS)" -ldflags "$(RUNC_GLDFLAGS)" .
-endef
-
-define RUNC_INSTALL_TARGET_CMDS
-	$(INSTALL) -D -m 0755 $(@D)/bin/runc $(TARGET_DIR)/usr/bin/runc
-endef
-
-$(eval $(generic-package))
+$(eval $(golang-package))
-- 
2.7.4

[Buildroot] [PATCH v3 5/7] package/docker-containerd: converting to golang infrastructure

[Angelo Compagnucci]

This patch converts the docker-containerd package to the new golang
infrastructure.

Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com>
---
 package/docker-containerd/docker-containerd.mk | 37 +++++++++++---------------
 1 file changed, 15 insertions(+), 22 deletions(-)

diff --git a/package/docker-containerd/docker-containerd.mk b/package/docker-containerd/docker-containerd.mk
index ffbadb0..5d21172 100644
--- a/package/docker-containerd/docker-containerd.mk
+++ b/package/docker-containerd/docker-containerd.mk
@@ -9,39 +9,32 @@ DOCKER_CONTAINERD_SITE = $(call github,docker,containerd,$(DOCKER_CONTAINERD_VER
 DOCKER_CONTAINERD_LICENSE = Apache-2.0
 DOCKER_CONTAINERD_LICENSE_FILES = LICENSE.code
 
-DOCKER_CONTAINERD_DEPENDENCIES = host-go
+DOCKER_CONTAINERD_GOPATH = vendor
 
-DOCKER_CONTAINERD_GOPATH = "$(@D)/vendor"
-DOCKER_CONTAINERD_MAKE_ENV = $(HOST_GO_TARGET_ENV) \
-	CGO_ENABLED=1 \
-	GOBIN="$(@D)/bin" \
-	GOPATH="$(DOCKER_CONTAINERD_GOPATH)"
-
-DOCKER_CONTAINERD_GLDFLAGS = \
+DOCKER_CONTAINERD_GO_LDFLAGS = \
 	-X github.com/docker/containerd.GitCommit=$(DOCKER_CONTAINERD_VERSION)
 
-ifeq ($(BR2_STATIC_LIBS),y)
-DOCKER_CONTAINERD_GLDFLAGS += -extldflags '-static'
-endif
-
-define DOCKER_CONTAINERD_CONFIGURE_CMDS
-	mkdir -p $(DOCKER_CONTAINERD_GOPATH)/src/github.com/docker
-	ln -s $(@D) $(DOCKER_CONTAINERD_GOPATH)/src/github.com/docker/containerd
-	mkdir -p $(DOCKER_CONTAINERD_GOPATH)/src/github.com/opencontainers
-	ln -s $(RUNC_SRCDIR) $(DOCKER_CONTAINERD_GOPATH)/src/github.com/opencontainers/runc
+define DOCKER_CONTAINERD_LINK_DIR
+	ln -sf $(@D) $(@D)/$(DOCKER_CONTAINERD_GOPATH)/src/github.com/docker/containerd
+	mkdir -p $(@D)/$(DOCKER_CONTAINERD_GOPATH)/src/github.com/opencontainers
+	ln -s $(RUNC_SRCDIR) $(@D)/$(DOCKER_CONTAINERD_GOPATH)/src/github.com/opencontainers/runc
 endef
 
+DOCKER_CONTAINERD_POST_CONFIGURE_HOOKS += DOCKER_CONTAINERD_LINK_DIR
+
 define DOCKER_CONTAINERD_BUILD_CMDS
 	$(foreach d,ctr containerd containerd-shim,\
-		cd $(@D); $(DOCKER_CONTAINERD_MAKE_ENV) $(HOST_DIR)/bin/go build \
-			-v -o $(@D)/bin/$(d) -ldflags "$(DOCKER_CONTAINERD_GLDFLAGS)" ./$(d)$(sep))
+		cd $(@D)/$(DOCKER_CONTAINERD_GOPATH)/src/github.com/docker/containerd; \
+		$(DOCKER_CONTAINERD_GO_ENV) \
+		$(GO_BIN) install -v \
+		$(DOCKER_CONTAINERD_BUILD_OPTS) ./$(d)$(sep))
 endef
 
 define DOCKER_CONTAINERD_INSTALL_TARGET_CMDS
 	ln -fs runc $(TARGET_DIR)/usr/bin/docker-runc
-	$(INSTALL) -D -m 0755 $(@D)/bin/containerd $(TARGET_DIR)/usr/bin/docker-containerd
-	$(INSTALL) -D -m 0755 $(@D)/bin/containerd-shim $(TARGET_DIR)/usr/bin/containerd-shim
+	$(INSTALL) -D -m 0755 $(@D)/$(DOCKER_CONTAINERD_BINDIR)/containerd $(TARGET_DIR)/usr/bin/docker-containerd
+	$(INSTALL) -D -m 0755 $(@D)/$(DOCKER_CONTAINERD_BINDIR)/containerd-shim $(TARGET_DIR)/usr/bin/containerd-shim
 	ln -fs containerd-shim $(TARGET_DIR)/usr/bin/docker-containerd-shim
 endef
 
-$(eval $(generic-package))
+$(eval $(golang-package))
-- 
2.7.4

[Buildroot] [PATCH v3 6/7] package/docker-engine: converting to golang infrastructure

[Angelo Compagnucci]

This patch converts the docker-engine package to the new golang
infrastructure.

Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com>
---
 package/docker-engine/docker-engine.mk | 53 +++++++++++-----------------------
 1 file changed, 17 insertions(+), 36 deletions(-)

diff --git a/package/docker-engine/docker-engine.mk b/package/docker-engine/docker-engine.mk
index 8928f07..33b825e 100644
--- a/package/docker-engine/docker-engine.mk
+++ b/package/docker-engine/docker-engine.mk
@@ -13,70 +13,53 @@ DOCKER_ENGINE_LICENSE_FILES = LICENSE
 
 DOCKER_ENGINE_DEPENDENCIES = host-go host-pkgconf
 
-DOCKER_ENGINE_GOPATH = "$(@D)/gopath"
-DOCKER_ENGINE_MAKE_ENV = $(HOST_GO_TARGET_ENV) \
-	CGO_ENABLED=1 \
-	CGO_NO_EMULATION=1 \
-	GOBIN="$(@D)/bin" \
-	GOPATH="$(DOCKER_ENGINE_GOPATH)" \
-	PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
-	$(TARGET_MAKE_ENV)
-
-DOCKER_ENGINE_GLDFLAGS = \
+DOCKER_ENGINE_GO_LDFLAGS = \
 	-X main.GitCommit=$(DOCKER_ENGINE_VERSION) \
 	-X main.Version=$(DOCKER_ENGINE_VERSION)
 
-ifeq ($(BR2_STATIC_LIBS),y)
-DOCKER_ENGINE_GLDFLAGS += -extldflags '-static'
-else
-ifeq ($(BR2_PACKAGE_DOCKER_ENGINE_STATIC_CLIENT),y)
-DOCKER_ENGINE_GLDFLAGS_DOCKER += -extldflags '-static'
-endif
-endif
-
-DOCKER_ENGINE_BUILD_TAGS = cgo exclude_graphdriver_zfs autogen
+DOCKER_ENGINE_GO_TAGS = cgo exclude_graphdriver_zfs autogen
 DOCKER_ENGINE_BUILD_TARGETS = docker
 
 ifeq ($(BR2_PACKAGE_LIBSECCOMP),y)
-DOCKER_ENGINE_BUILD_TAGS += seccomp
+DOCKER_ENGINE_GO_TAGS += seccomp
 DOCKER_ENGINE_DEPENDENCIES += libseccomp
 endif
 
 ifeq ($(BR2_INIT_SYSTEMD),y)
-DOCKER_ENGINE_BUILD_TAGS += journald
+DOCKER_ENGINE_GO_TAGS += journald
 DOCKER_ENGINE_DEPENDENCIES += systemd
 endif
 
 ifeq ($(BR2_PACKAGE_DOCKER_ENGINE_DAEMON),y)
-DOCKER_ENGINE_BUILD_TAGS += daemon
+DOCKER_ENGINE_GO_TAGS += daemon
 DOCKER_ENGINE_BUILD_TARGETS += dockerd
 endif
 
 ifeq ($(BR2_PACKAGE_DOCKER_ENGINE_EXPERIMENTAL),y)
-DOCKER_ENGINE_BUILD_TAGS += experimental
+DOCKER_ENGINE_GO_TAGS += experimental
 endif
 
 ifeq ($(BR2_PACKAGE_DOCKER_ENGINE_DRIVER_BTRFS),y)
 DOCKER_ENGINE_DEPENDENCIES += btrfs-progs
 else
-DOCKER_ENGINE_BUILD_TAGS += exclude_graphdriver_btrfs
+DOCKER_ENGINE_GO_TAGS += exclude_graphdriver_btrfs
 endif
 
 ifeq ($(BR2_PACKAGE_DOCKER_ENGINE_DRIVER_DEVICEMAPPER),y)
 DOCKER_ENGINE_DEPENDENCIES += lvm2
 else
-DOCKER_ENGINE_BUILD_TAGS += exclude_graphdriver_devicemapper
+DOCKER_ENGINE_GO_TAGS += exclude_graphdriver_devicemapper
 endif
 
 ifeq ($(BR2_PACKAGE_DOCKER_ENGINE_DRIVER_VFS),y)
 DOCKER_ENGINE_DEPENDENCIES += gvfs
 else
-DOCKER_ENGINE_BUILD_TAGS += exclude_graphdriver_vfs
+DOCKER_ENGINE_GO_TAGS += exclude_graphdriver_vfs
 endif
 
 define DOCKER_ENGINE_CONFIGURE_CMDS
-	mkdir -p $(DOCKER_ENGINE_GOPATH)/src/github.com/docker
-	ln -fs $(@D) $(DOCKER_ENGINE_GOPATH)/src/github.com/docker/docker
+	mkdir -p $(@D)/$(DOCKER_ENGINE_GOPATH)/src/github.com/docker
+	ln -s $(@D) $(@D)/$(DOCKER_ENGINE_GOPATH)/src/github.com/docker/docker
 	cd $(@D) && \
 		GITCOMMIT="$$(echo $(DOCKER_ENGINE_COMMIT) | head -c7)" \
 		BUILDTIME="$$(date)" \
@@ -105,20 +88,18 @@ endif
 
 define DOCKER_ENGINE_BUILD_CMDS
 	$(foreach target,$(DOCKER_ENGINE_BUILD_TARGETS), \
-		cd $(@D)/gopath/src/github.com/docker/docker; \
-		$(DOCKER_ENGINE_MAKE_ENV) \
-		$(HOST_DIR)/bin/go build -v \
-			-o $(@D)/bin/$(target) \
-			-tags "$(DOCKER_ENGINE_BUILD_TAGS)" \
-			-ldflags "$(DOCKER_ENGINE_GLDFLAGS) $(DOCKER_ENGINE_GLDFLAGS_$(call UPPERCASE,$(target)))" \
+		cd $(DOCKER_ENGINE_SRC_PATH); \
+		$(DOCKER_ENGINE_GO_ENV) \
+		$(GO_BIN) install -v \
+			$(DOCKER_ENGINE_BUILD_OPTS) \
 			github.com/docker/docker/cmd/$(target)
 	)
 endef
 
 define DOCKER_ENGINE_INSTALL_TARGET_CMDS
 	$(foreach target,$(DOCKER_ENGINE_BUILD_TARGETS), \
-		$(INSTALL) -D -m 0755 $(@D)/bin/$(target) $(TARGET_DIR)/usr/bin/$(target)
+		$(INSTALL) -D -m 0755 $(@D)/$(DOCKER_ENGINE_BINDIR)/$(target) $(TARGET_DIR)/usr/bin/$(target)
 	)
 endef
 
-$(eval $(generic-package))
+$(eval $(golang-package))
-- 
2.7.4

[Buildroot] [PATCH v3 7/7] package/mender: new package

[Angelo Compagnucci]

This patch add mender, an open source over-the-air (OTA) software
updater for embedded Linux devices.

Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com>
---
Changelog:
v2->v3:
* Moving the installing step to an hook to permit
  the execution of the default target install step
* Fixing a duplicate install entry
* adding the installation of the systemd service file

 package/Config.in                        |  1 +
 package/mender/Config.in                 | 14 +++++++++
 package/mender/mender-device-identity    | 52 ++++++++++++++++++++++++++++++++
 package/mender/mender-inventory-hostinfo | 21 +++++++++++++
 package/mender/mender-inventory-network  | 47 +++++++++++++++++++++++++++++
 package/mender/mender.conf               | 14 +++++++++
 package/mender/mender.hash               |  2 ++
 package/mender/mender.mk                 | 39 ++++++++++++++++++++++++
 package/mender/mender.service            | 15 +++++++++
 package/mender/server.crt                | 22 ++++++++++++++
 package/mender/tenant.conf               |  0
 11 files changed, 227 insertions(+)
 create mode 100644 package/mender/Config.in
 create mode 100644 package/mender/mender-device-identity
 create mode 100644 package/mender/mender-inventory-hostinfo
 create mode 100644 package/mender/mender-inventory-network
 create mode 100644 package/mender/mender.conf
 create mode 100644 package/mender/mender.hash
 create mode 100644 package/mender/mender.mk
 create mode 100644 package/mender/mender.service
 create mode 100644 package/mender/server.crt
 create mode 100644 package/mender/tenant.conf

diff --git a/package/Config.in b/package/Config.in
index 6abbb43..d5e6db8 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -1968,6 +1968,7 @@ menu "System tools"
 	source "package/kvmtool/Config.in"
 	source "package/libostree/Config.in"
 	source "package/lxc/Config.in"
+	source "package/mender/Config.in"
 	source "package/monit/Config.in"
 	source "package/ncdu/Config.in"
 	source "package/numactl/Config.in"
diff --git a/package/mender/Config.in b/package/mender/Config.in
new file mode 100644
index 0000000..201128a
--- /dev/null
+++ b/package/mender/Config.in
@@ -0,0 +1,14 @@
+config BR2_PACKAGE_MENDER
+	bool "mender"
+	depends on BR2_PACKAGE_HOST_GO_ARCH_SUPPORTS
+	depends on BR2_PACKAGE_HOST_GO_CGO_LINKING_SUPPORTS
+	depends on BR2_TOOLCHAIN_HAS_THREADS
+	depends on BR2_INIT_SYSTEMD
+	
+	help
+	  Mender is an open source over-the-air (OTA) software updater for
+	  embedded Linux devices. Mender comprises a client running at the
+	  embedded device, as well as a server that manages deployments
+	  across many devices.
+
+	  https://github.com/mendersoftware/mender
diff --git a/package/mender/mender-device-identity b/package/mender/mender-device-identity
new file mode 100644
index 0000000..d87f843
--- /dev/null
+++ b/package/mender/mender-device-identity
@@ -0,0 +1,52 @@
+#!/bin/sh
+
+# Example script called by Mender agent to collect device identity data. The
+# script needs to be located at
+# $(datadir)/mender/identity/mender-device-identity path for the agent to find
+# it. The script shall exit with non-0 status on errors. In this case the agent
+# will discard any output the script may have produced.
+#
+# The script shall output identity data in <key>=<value> format, one
+# entry per line. Example
+#
+# $ ./mender-device-identity
+# mac=de:ad:ca:fe:00:01
+# cpuid=1112233
+#
+# The example script collects the MAC address of a network interface with the
+# type ARPHRD_ETHER and it will pick the interface with the lowest ifindex
+# number if there are multiple interfaces with that type. The identity data is
+# output in the following format:
+#
+# mac=00:01:02:03:04:05
+#
+
+set -ue
+
+SCN=/sys/class/net
+min=65535
+arphrd_ether=1
+ifdev=
+
+# find iface with lowest ifindex, skip non ARPHRD_ETHER types (lo, sit ...)
+for dev in $SCN/*; do
+    iftype=$(cat $dev/type)
+    if [ $iftype -ne $arphrd_ether ]; then
+        continue
+    fi
+
+    idx=$(cat $dev/ifindex)
+    if [ $idx -lt $min ]; then
+        min=$idx
+        ifdev=$dev
+    fi
+done
+
+if [ -z "$ifdev" ]; then
+    echo "no suitable interfaces found" >&2
+    exit 1
+else
+    echo "using interface $ifdev" >&2
+    # grab MAC address
+    echo "mac=$(cat $ifdev/address)"
+fi
diff --git a/package/mender/mender-inventory-hostinfo b/package/mender/mender-inventory-hostinfo
new file mode 100644
index 0000000..cf508fd
--- /dev/null
+++ b/package/mender/mender-inventory-hostinfo
@@ -0,0 +1,21 @@
+#!/bin/sh
+#
+# The example script collects information about current host
+#
+
+set -ue
+
+LC_ALL=C
+export LC_ALL
+
+grep 'model name' /proc/cpuinfo | uniq | awk -F': ' '
+     // { printf("cpu_model=%s\n", $2);}
+'
+echo "kernel=$(cat /proc/version)"
+
+cat /proc/meminfo | awk '
+/MemTotal/ {printf("mem_total_kB=%d\n", $2)}
+'
+
+echo "hostname=$(cat /etc/hostname)"
+
diff --git a/package/mender/mender-inventory-network b/package/mender/mender-inventory-network
new file mode 100644
index 0000000..b017c4e
--- /dev/null
+++ b/package/mender/mender-inventory-network
@@ -0,0 +1,47 @@
+#!/bin/sh
+#
+# Example script called by Mender agent to collect inventory data for a
+# particular devce. The script needs to be located in $(datadir)/mender and its
+# name shall start with `mender-inventory-` prefix. The script shall exit with
+# non-0 status on errors. In this case the agent will discard any output the
+# script may have produced.
+#
+# The script shall output inventory data in <key>=<value> format, one entry per
+# line. Entries appearing multiple times will be joined in a list under the same
+# key.
+#
+# $ ./mender-inventory-network
+# mac_br-fbfdad18c33c=02:42:7e:74:96:85
+# network_interfaces=br-fbfdad18c33c
+# ipv4_br-fbfdad18c33c=172.21.0.1/16
+# mac_enp0s25=de:ad:be:ef:bb:05
+# network_interfaces=enp0s25
+# ipv4_enp0s25=123.22.0.197/16
+# ipv4_enp0s25=10.20.20.105/16
+# ipv6_enp0s25=fe80::2aad:beff:feef:bb05/64
+#
+#
+# The example script collects the list of network interfaces, as well as
+# ethernet and IP addresses of each of the interfaces.
+#
+
+set -ue
+
+SCN=/sys/class/net
+min=65535
+ifdev=
+
+# find iface with lowest ifindex, except loopback
+for devpath in $SCN/*; do
+    dev=$(basename $devpath)
+    if [ $dev = "lo" ]; then
+        continue
+    fi
+    echo "mac_$dev=$(cat $devpath/address)"
+    echo "network_interfaces=$dev"
+
+    ip addr show dev $dev | awk -v dev=$dev '
+       /inet / { printf("ipv4_%s=%s\n", dev, $2) }
+       /inet6 / {printf("ipv6_%s=%s\n", dev, $2) }
+    '
+done
diff --git a/package/mender/mender.conf b/package/mender/mender.conf
new file mode 100644
index 0000000..a5c7c54
--- /dev/null
+++ b/package/mender/mender.conf
@@ -0,0 +1,14 @@
+{
+  "ClientProtocol": "http",
+  "HttpsClient": {
+    "Certificate": "",
+    "Key": ""
+  },
+  "RootfsPartA": "@MENDER_ROOTFS_PART_A@",
+  "RootfsPartB": "@MENDER_ROOTFS_PART_B@",
+  "UpdatePollIntervalSeconds": @MENDER_UPDATE_POLL_INTERVAL_SECONDS@,
+  "InventoryPollIntervalSeconds": @MENDER_INVENTORY_POLL_INTERVAL_SECONDS@,
+  "RetryPollIntervalSeconds": @MENDER_RETRY_POLL_INTERVAL_SECONDS@,
+  "ServerURL": "@MENDER_SERVER_URL@",
+  "ServerCertificate": "@MENDER_CERT_LOCATION@"
+}
diff --git a/package/mender/mender.hash b/package/mender/mender.hash
new file mode 100644
index 0000000..574b675
--- /dev/null
+++ b/package/mender/mender.hash
@@ -0,0 +1,2 @@
+# Locally computed:
+sha256 a2fd103185431946c3455fef0c08909d13413a09638463266e5b6889df8163c5 mender-1.2.1.tar.gz
diff --git a/package/mender/mender.mk b/package/mender/mender.mk
new file mode 100644
index 0000000..ef9c1ae
--- /dev/null
+++ b/package/mender/mender.mk
@@ -0,0 +1,39 @@
+################################################################################
+#
+# mender
+#
+################################################################################
+
+MENDER_VERSION = 1.2.1
+MENDER_SOURCE = mender-$(MENDER_VERSION).tar.gz
+MENDER_SITE = $(call github,mendersoftware,mender,$(MENDER_VERSION))
+
+define MENDER_INSTALL_SUPPORT_FILES
+	$(INSTALL) -dm 0755 $(TARGET_DIR)/etc/mender/
+	$(INSTALL) -dm 0755 $(TARGET_DIR)/var/share/mender/identity/
+	$(INSTALL) -dm 0755 $(TARGET_DIR)/var/share/mender/inventory/
+	$(INSTALL) -D -m 0644 package/mender/mender.conf \
+		$(TARGET_DIR)/etc/mender/mender.conf
+	$(INSTALL) -D -m 0644 package/mender/tenant.conf \
+		$(TARGET_DIR)/etc/mender/tenant.conf
+	$(INSTALL) -D -m 0644 package/mender/server.crt \
+		$(TARGET_DIR)/etc/mender/server.crt
+	$(INSTALL) -D -m 0755 package/mender/mender-device-identity \
+		$(TARGET_DIR)/var/share/mender/identity/mender-device-identity
+	$(INSTALL) -D -m 0755 package/mender/mender-inventory-network \
+		$(TARGET_DIR)/var/share/mender/inventory/mender-inventory-network
+	$(INSTALL) -D -m 0755 package/mender/mender-inventory-hostinfo \
+		$(TARGET_DIR)/var/share/mender/inventory/mender-inventory-hostinfo
+endef
+
+define MENDER_INSTALL_INIT_SYSTEMD
+	$(INSTALL) -D -m 0644 package/mender/mender.service \
+		$(TARGET_DIR)/usr/lib/systemd/system/mender.service
+	mkdir -p $(TARGET_DIR)/etc/systemd/system/multi-user.target.wants
+	ln -fs ../../../../usr/lib/systemd/system/mender.service \
+		$(TARGET_DIR)/etc/systemd/system/multi-user.target.wants/mender.service
+endef
+
+MENDER_POST_INSTALL_TARGET_HOOKS = MENDER_INSTALL_SUPPORT_FILES
+
+$(eval $(golang-package))
diff --git a/package/mender/mender.service b/package/mender/mender.service
new file mode 100644
index 0000000..ec77fbc
--- /dev/null
+++ b/package/mender/mender.service
@@ -0,0 +1,15 @@
+[Unit]
+Description=Mender OTA update service
+After=systemd-resolved.service
+
+[Service]
+Type=idle
+User=root
+Group=root
+ExecStartPre=/bin/mkdir -p -m 0700 /data/mender
+ExecStartPre=/bin/ln -sf /etc/mender/tenant.conf /var/lib/mender/authtentoken
+ExecStart=/usr/bin/mender -daemon
+Restart=on-abort
+
+[Install]
+WantedBy=multi-user.target
diff --git a/package/mender/server.crt b/package/mender/server.crt
new file mode 100644
index 0000000..79a57e1
--- /dev/null
+++ b/package/mender/server.crt
@@ -0,0 +1,22 @@
+-----BEGIN CERTIFICATE-----
+MIIBfTCCASOgAwIBAgIJAJOS76a0qWuZMAoGCCqGSM49BAMCMBsxGTAXBgNVBAMM
+EGRvY2tlci5tZW5kZXIuaW8wHhcNMTYxMjE0MTk1MjQ2WhcNMjYxMjEyMTk1MjQ2
+WjAbMRkwFwYDVQQDDBBkb2NrZXIubWVuZGVyLmlvMFkwEwYHKoZIzj0CAQYIKoZI
+zj0DAQcDQgAE7AVYis6MWGPGQYU1/tlLEnskRifDIhvkRb8Y4nQPekRkLkiBYYT3
+iJ46wHrnejbHaLstU9GRdKWOmOuU6HGdO6NQME4wHQYDVR0OBBYEFGOIU4q++Vz8
+9HuT1jg9V+wFeJcyMB8GA1UdIwQYMBaAFGOIU4q++Vz89HuT1jg9V+wFeJcyMAwG
+A1UdEwQFMAMBAf8wCgYIKoZIzj0EAwIDSAAwRQIhAPLnEeWPNeN7eDCEYRitBfyO
+X1yf2kzOm4ohBE5GY9gzAiBCq7HOSkzQDkelmQCCCpGXf/UwYNgQJjSoeGfk0j1a
+TQ==
+-----END CERTIFICATE-----
+-----BEGIN CERTIFICATE-----
+MIIBhDCCASmgAwIBAgIJALQrf4QDot4IMAoGCCqGSM49BAMCMB4xHDAaBgNVBAMM
+E3MzLmRvY2tlci5tZW5kZXIuaW8wHhcNMTYxMjE0MTk1MjQ2WhcNMjYxMjEyMTk1
+MjQ2WjAeMRwwGgYDVQQDDBNzMy5kb2NrZXIubWVuZGVyLmlvMFkwEwYHKoZIzj0C
+AQYIKoZIzj0DAQcDQgAEEc/Y3T+l3DvINePkpvVZORMIdHVs29jgsl48ia7z/NRX
+HlKtKxVGJyFN5Y7sBZeLgBYH3F4Bo3KfmxI7ad0tI6NQME4wHQYDVR0OBBYEFIUm
+cip00QZYpe4ULflbGNJan+Y9MB8GA1UdIwQYMBaAFIUmcip00QZYpe4ULflbGNJa
+n+Y9MAwGA1UdEwQFMAMBAf8wCgYIKoZIzj0EAwIDSQAwRgIhANHij9VZBDHOUPaC
+pFiagnWnYL2HBR72W1xTKQbrLLTXAiEAvpwA4HzSnGmLd3010+jqQuMRHArN5WaX
+h0fy7niBbIQ=
+-----END CERTIFICATE-----
diff --git a/package/mender/tenant.conf b/package/mender/tenant.conf
new file mode 100644
index 0000000..e69de29
-- 
2.7.4

[Buildroot] [PATCH v3 5/7] package/docker-containerd: converting to golang infrastructure

[Thomas Petazzoni]

Hello Angelo,

I'm now looking at your patch series, hoping to make good progress with
it as part of the hackathon. I have one question below.

On Wed,  7 Mar 2018 23:19:37 +0100, Angelo Compagnucci wrote:

> -define DOCKER_CONTAINERD_CONFIGURE_CMDS
> -	mkdir -p $(DOCKER_CONTAINERD_GOPATH)/src/github.com/docker
> -	ln -s $(@D) $(DOCKER_CONTAINERD_GOPATH)/src/github.com/docker/containerd
> -	mkdir -p $(DOCKER_CONTAINERD_GOPATH)/src/github.com/opencontainers
> -	ln -s $(RUNC_SRCDIR) $(DOCKER_CONTAINERD_GOPATH)/src/github.com/opencontainers/runc
> +define DOCKER_CONTAINERD_LINK_DIR
> +	ln -sf $(@D) $(@D)/$(DOCKER_CONTAINERD_GOPATH)/src/github.com/docker/containerd
> +	mkdir -p $(@D)/$(DOCKER_CONTAINERD_GOPATH)/src/github.com/opencontainers
> +	ln -s $(RUNC_SRCDIR) $(@D)/$(DOCKER_CONTAINERD_GOPATH)/src/github.com/opencontainers/runc

I am a bit confused by this symlink to the runc package source
directory. Indeed, the docker-containerd package does not depend on
runc from a build dependency point of view. It does select runc at the
Config.in level, but that doesn't guarantee that runc will be
built/extracted before docker-containerd. And therefore, this symlink
might point to a non-existing directory.

Am I missing something ? How does it work ?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com