[Buildroot] [PATCH v9 0/8] Add support for the Rust programming

[Buildroot] [PATCH v9 0/8] Add support for the Rust programming

[Eric Le Bihan]

This series adds support for the Rust programming language by adding the
following packages:

 - rustc: a virtual package for the Rust compiler.
 - rust-bin: provides a pre-built version of rustc.
 - cargo-bin: provides a pre-built version of Rust package manager.
 - rust: builds rustc from source.
 - cargo: builds Rust package manager from source.

Only the host variants are provided.

The rustc virtual package is inspired by the mysql one.

v8 -> v9:

 - replace giant patch by post-extract hook
 - drop rust-cargo virtual package
 - explicitly enable SSL support in host-libcurl

v7 -> v8:

 - replace cargo-bootstrap by cargo-bin
 - drop rust-bootstrap and use rust-bin instead
 - rust-bin and rust versions are the same
 - cargo-bin and cargo versions are the same
 - bump cargo to 0.23.0
 - bump rust to 1.22.1
 - add rust-cargo virtual package

v6 -> v7:

 - add cargo 0.20.0
 - bump rust to 1.19.0
 - bump rust-bootstrap to 1.18.0
 - bump cargo-bootstrap to 0.19.0
 - bump rust-bin to 1.19.0
 - fix mips64 target name in rustc

v5 -> v6:

 - bump rust to 1.18.0
 - bump rust-bootstrap to 1.17.0
 - bump rust-bin to 1.18.0

v4 -> v5:

 - add rustc virtual package
 - add rust-bin, provider for rustc
 - rework rust to be a provider for rustc
 - rework some commit messages

v3 -> v4:

 - bump rust to 1.16.0
 - bump rust-bootstrap to 1.15.1
 - add cargo-bootstrap
 - use built-in target specifications
 - drop external Python script to generate target specifications
 - enable support for PowerPC
 - expose host variant in menuconfig

v2 -> v3:

 - bump rust to 1.10.0
 - rework and bump rust-bootstrap to 1.9.0
 - host-rust requires at least GCC 4.7 because of LLVM (suggested by R. Naour)
 - rust requires a glibc-based cross-compiler
 - rust requires GCC 5.x or above for Aarch64

v1 -> v2:

 - bump rust to version 1.9.0.
 - drop patch for bzip2 support in host-python.
 - add package for jemalloc.
 - add dependency on host being a x86 machine.
 - add dependency on toolchain.
 - use dedicated package to provide bootstrapping binary: rust-bootstrap.
 - let ./configure find out host/build on its own.
 - remove entry from configuration menu.

Eric Le Bihan (8):
  rustc: new virtual package
  rust-bin: new package
  cargo-bin: new package
  rust: new package
  libssh2: add host variant
  libhttpparser: add host variant
  libcurl: add host variant
  cargo: new package

 DEVELOPERS                             |  4 ++
 package/Config.in.host                 |  2 +
 package/cargo-bin/cargo-bin.hash       |  4 ++
 package/cargo-bin/cargo-bin.mk         | 14 +++++
 package/cargo/Config.in.host           |  8 +++
 package/cargo/cargo.hash               |  4 ++
 package/cargo/cargo.mk                 | 96 ++++++++++++++++++++++++++++++++++
 package/cargo/config.in                |  2 +
 package/libcurl/libcurl.mk             | 14 +++++
 package/libhttpparser/libhttpparser.mk |  9 ++++
 package/libssh2/libssh2.mk             |  6 +++
 package/rust-bin/rust-bin.hash         | 30 +++++++++++
 package/rust-bin/rust-bin.mk           | 63 ++++++++++++++++++++++
 package/rust/rust.hash                 |  2 +
 package/rust/rust.mk                   | 82 +++++++++++++++++++++++++++++
 package/rustc/Config.in.host           | 62 ++++++++++++++++++++++
 package/rustc/rustc.mk                 | 31 +++++++++++
 17 files changed, 433 insertions(+)
 create mode 100644 package/cargo-bin/cargo-bin.hash
 create mode 100644 package/cargo-bin/cargo-bin.mk
 create mode 100644 package/cargo/Config.in.host
 create mode 100644 package/cargo/cargo.hash
 create mode 100644 package/cargo/cargo.mk
 create mode 100644 package/cargo/config.in
 create mode 100644 package/rust-bin/rust-bin.hash
 create mode 100644 package/rust-bin/rust-bin.mk
 create mode 100644 package/rust/rust.hash
 create mode 100644 package/rust/rust.mk
 create mode 100644 package/rustc/Config.in.host
 create mode 100644 package/rustc/rustc.mk

--
2.14.3

[Buildroot] [PATCH v9 1/8] rustc: new virtual package

[Eric Le Bihan]

The compiler for the Rust programming language is called rustc.

There is only one reference implementation for it, based on LLVM, from
the Rust project [1]. It can generate code for various architectures so
it can be labeled as a cross-compiler. But, as for GCC, building it
from source takes time.

So it would be sensible to have at least one package which provides it
as a pre-built version, fetched from the upstream project. Later another
package can be added, to build it from source code.

In addition to the compiler, the standard library for the host and/or
the target should also be fetched/built.

So, add a virtual package named rustc to enable support for multiple
providers.

Currently, only the host variant will be available to allow the user to
cross-compile Rust programs for the target.

[1] http://rust-lang.org

Signed-off-by: Eric Le Bihan <eric.le.bihan.dev@free.fr>
---
 package/Config.in.host       |  1 +
 package/rustc/Config.in.host | 17 +++++++++++++++++
 package/rustc/rustc.mk       | 31 +++++++++++++++++++++++++++++++
 3 files changed, 49 insertions(+)
 create mode 100644 package/rustc/Config.in.host
 create mode 100644 package/rustc/rustc.mk

diff --git a/package/Config.in.host b/package/Config.in.host
index dd6415bba5..2f20dd0e48 100644
--- a/package/Config.in.host
+++ b/package/Config.in.host
@@ -46,6 +46,7 @@ menu "Host utilities"
 	source "package/qemu/Config.in.host"
 	source "package/raspberrypi-usbboot/Config.in.host"
 	source "package/rauc/Config.in.host"
+	source "package/rustc/Config.in.host"
 	source "package/s6-rc/Config.in.host"
 	source "package/sam-ba/Config.in.host"
 	source "package/squashfs/Config.in.host"
diff --git a/package/rustc/Config.in.host b/package/rustc/Config.in.host
new file mode 100644
index 0000000000..16c95fa0b4
--- /dev/null
+++ b/package/rustc/Config.in.host
@@ -0,0 +1,17 @@
+config BR2_PACKAGE_HOST_RUSTC_ARCH_SUPPORTS
+	bool
+	default y
+	depends on BR2_HOSTARCH = "x86_64" || BR2_HOSTARCH = "x86"
+	depends on BR2_i386 || BR2_x86_64 \
+		|| BR2_arm  || BR2_aarch64 \
+		|| BR2_powerpc  || BR2_powerpc64 \
+		|| BR2_mips || BR2_mipsel || BR2_mips64 || BR2_mips64el
+	depends on !BR2_ARM_CPU_ARMV4 && !BR2_ARM_CPU_ARMV5
+	depends on !BR2_MIPS_NABI32
+	depends on BR2_TOOLCHAIN_USES_GLIBC
+
+config BR2_PACKAGE_HAS_HOST_RUSTC
+	bool
+
+config BR2_PACKAGE_PROVIDES_HOST_RUSTC
+	string
diff --git a/package/rustc/rustc.mk b/package/rustc/rustc.mk
new file mode 100644
index 0000000000..4163a44bc1
--- /dev/null
+++ b/package/rustc/rustc.mk
@@ -0,0 +1,31 @@
+################################################################################
+#
+# rustc
+#
+################################################################################
+
+RUST_ARCH := $(call qstrip,$(BR2_ARCH))
+
+ifeq ($(BR2_ARM_CPU_ARMV7A),y)
+RUST_ARCH := armv7
+endif
+
+ifeq ($(BR2_ARM_EABI),y)
+RUST_ABI := eabi
+else ifeq ($(BR2_ARM_EABIHF),y)
+RUST_ABI := eabihf
+else ifeq ($(BR2_mips64)$(BR2_mips64el),y)
+RUST_ABI := abi64
+endif
+
+RUST_TARGET_NAME := $(RUST_ARCH)-unknown-linux-gnu$(RUST_ABI)
+
+ifeq ($(HOSTARCH),x86)
+RUST_HOST_ARCH = i686
+else
+RUST_HOST_ARCH = $(HOSTARCH)
+endif
+
+RUST_HOST_NAME = $(RUST_HOST_ARCH)-unknown-linux-gnu
+
+$(eval $(host-virtual-package))
-- 
2.14.3

[Buildroot] [PATCH v9 2/8] rust-bin: new package

[Eric Le Bihan]

This package provides a pre-built version of rustc, the compiler for the
Rust programming language, fetched from the upstream project.

A pre-built version of the standard library for the host as well as one
for the chosen target are also fetched and installed.

Only the host variant is provided to allow the user to cross-compile
Rust programs and run them on the target.

This package could also be used to provide a bootstrap compiler when building
Rust from source. So, in order to add it as a build dependency, the compiler and
standard libraries are only installed in $(HOST_DIR) if the package is
explicitly selected.

The menuconfig entry for rustc is also updated to expose this provider.

Signed-off-by: Eric Le Bihan <eric.le.bihan.dev@free.fr>
---
 DEVELOPERS                     |  1 +
 package/rust-bin/rust-bin.hash | 30 ++++++++++++++++++++
 package/rust-bin/rust-bin.mk   | 63 ++++++++++++++++++++++++++++++++++++++++++
 package/rustc/Config.in.host   | 29 +++++++++++++++++++
 4 files changed, 123 insertions(+)
 create mode 100644 package/rust-bin/rust-bin.hash
 create mode 100644 package/rust-bin/rust-bin.mk

diff --git a/DEVELOPERS b/DEVELOPERS
index 9f4f282203..bf4766886d 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -543,6 +543,7 @@ F:	package/execline/
 F:	package/hicolor-icon-theme/
 F:	package/jemalloc/
 F:	package/ninja/
+F:	package/rust-bin/
 F:	package/s6/
 F:	package/s6-dns/
 F:	package/s6-linux-init/
diff --git a/package/rust-bin/rust-bin.hash b/package/rust-bin/rust-bin.hash
new file mode 100644
index 0000000000..211f9fbbe0
--- /dev/null
+++ b/package/rust-bin/rust-bin.hash
@@ -0,0 +1,30 @@
+# From https://static.rust-lang.org/dist/rustc-1.22.1-i686-unknown-linux-gnu.tar.xz.sha256
+sha256 d87e15a8f8b3d53270866076824559926d01145733144c4a5905affff3819b47  rustc-1.22.1-i686-unknown-linux-gnu.tar.xz
+# From https://static.rust-lang.org/dist/rustc-1.22.1-x86_64-unknown-linux-gnu.tar.xz.sha256
+sha256 9730e46b2301417a5c5e1410f8ab8b8b6979f448b7fffc5e63221bd1bdaa15a3  rustc-1.22.1-x86_64-unknown-linux-gnu.tar.xz
+# From https://static.rust-lang.org/dist/rust-std-1.22.1-aarch64-unknown-linux-gnu.tar.xz.sha256
+sha256 b9c004288c5516dae88e325d24f26c10702dc617268e35b51b4bc83f4f2fc30d  rust-std-1.22.1-aarch64-unknown-linux-gnu.tar.xz
+# From https://static.rust-lang.org/dist/rust-std-1.22.1-arm-unknown-linux-gnueabi.tar.xz.sha256
+sha256 d48e918af118cacd2074c1c357cea46f2db73d42510c54fdfdcdf0ff60ba58ec  rust-std-1.22.1-arm-unknown-linux-gnueabi.tar.xz
+# From https://static.rust-lang.org/dist/rust-std-1.22.1-arm-unknown-linux-gnueabihf.tar.xz.sha256
+sha256 b64b4047dcdfa4aa617f7606a92320c95cdd44cadadac72d3e7d162addc62c92  rust-std-1.22.1-arm-unknown-linux-gnueabihf.tar.xz
+# From https://static.rust-lang.org/dist/rust-std-1.22.1-armv7-unknown-linux-gnueabihf.tar.xz.sha256
+sha256 e85b649aa18e1903e3c9c899ded55652713017a1e3a35e6fd58a7f40738805be  rust-std-1.22.1-armv7-unknown-linux-gnueabihf.tar.xz
+# From https://static.rust-lang.org/dist/rust-std-1.22.1-i686-unknown-linux-gnu.tar.xz.sha256
+sha256 112e17646bc5c513e81396b2c8042a44071a268b0d77810e378bdc9122abedc1  rust-std-1.22.1-i686-unknown-linux-gnu.tar.xz
+# From https://static.rust-lang.org/dist/rust-std-1.22.1-mips-unknown-linux-gnu.tar.xz.sha256
+sha256 80f0bc95273805e1ffe424b004b71a23d6d36acac2999d7361ce3cd3b17f39e9  rust-std-1.22.1-mips-unknown-linux-gnu.tar.xz
+# From https://static.rust-lang.org/dist/rust-std-1.22.1-mips64-unknown-linux-gnuabi64.tar.xz.sha256
+sha256 d50e4b52133b58235ff6f2c9de18d2ca81647fc2ef50aeaf9c1df5d726b3b2a3  rust-std-1.22.1-mips64-unknown-linux-gnuabi64.tar.xz
+# From https://static.rust-lang.org/dist/rust-std-1.22.1-mips64el-unknown-linux-gnuabi64.tar.xz.sha256
+sha256 6f61b30cd54c24fcdffddb67eb4073b2199dbea487c7be2c9ab6dd4bab46dabf  rust-std-1.22.1-mips64el-unknown-linux-gnuabi64.tar.xz
+# From https://static.rust-lang.org/dist/rust-std-1.22.1-mipsel-unknown-linux-gnu.tar.xz.sha256
+sha256 01278fa6596d8193bff76ddaacb78c339d610c5dbe035a38f7d0d9ebcf2bcce7  rust-std-1.22.1-mipsel-unknown-linux-gnu.tar.xz
+# From https://static.rust-lang.org/dist/rust-std-1.22.1-powerpc-unknown-linux-gnu.tar.xz.sha256
+sha256 39b26eaa3006aea11539d63bffdeada7078d79a4a06916cc8e6f6d66434e0a91  rust-std-1.22.1-powerpc-unknown-linux-gnu.tar.xz
+# From https://static.rust-lang.org/dist/rust-std-1.22.1-powerpc64-unknown-linux-gnu.tar.xz.sha256
+sha256 dcaf8f636ad4290bc761566d5fac77f4d6757046fe0ee539417fc6543246877b  rust-std-1.22.1-powerpc64-unknown-linux-gnu.tar.xz
+# From https://static.rust-lang.org/dist/rust-std-1.22.1-powerpc64le-unknown-linux-gnu.tar.xz.sha256
+sha256 9f5f047c8817b9c9bb45465fdccf55be9a551ebae5859100fdfa2c429f9d0574  rust-std-1.22.1-powerpc64le-unknown-linux-gnu.tar.xz
+# From https://static.rust-lang.org/dist/rust-std-1.22.1-x86_64-unknown-linux-gnu.tar.xz.sha256
+sha256 edd3313eb6523b2c08c84fc9cd7a80e4fb14c32168639d1833fd1b46e088a9a1  rust-std-1.22.1-x86_64-unknown-linux-gnu.tar.xz
diff --git a/package/rust-bin/rust-bin.mk b/package/rust-bin/rust-bin.mk
new file mode 100644
index 0000000000..29f94b7f2d
--- /dev/null
+++ b/package/rust-bin/rust-bin.mk
@@ -0,0 +1,63 @@
+################################################################################
+#
+# rust-bin
+#
+################################################################################
+
+RUST_BIN_VERSION = 1.22.1
+RUST_BIN_SITE = https://static.rust-lang.org/dist
+RUST_BIN_LICENSE = Apache-2.0 or MIT
+RUST_BIN_LICENSE_FILES = LICENSE-APACHE LICENSE-MIT
+
+HOST_RUST_BIN_PROVIDES = host-rustc
+
+HOST_RUST_BIN_SOURCE = rustc-$(RUST_BIN_VERSION)-$(RUST_HOST_NAME).tar.xz
+HOST_RUST_BIN_LIBSTD_SOURCES = \
+	rust-std-$(RUST_BIN_VERSION)-$(RUST_HOST_NAME).tar.xz \
+	rust-std-$(RUST_BIN_VERSION)-$(RUST_TARGET_NAME).tar.xz
+
+HOST_RUST_BIN_EXTRA_DOWNLOADS = $(HOST_RUST_BIN_LIBSTD_SOURCES)
+
+HOST_RUST_BIN_LIBSTD_HOST_PREFIX = rust-std-$(RUST_BIN_VERSION)-$(RUST_HOST_NAME)/rust-std-$(RUST_HOST_NAME)
+
+define HOST_RUST_BIN_LIBSTD_EXTRACT
+	mkdir -p $(@D)/std
+	for file in $(addprefix $(DL_DIR)/,$(HOST_RUST_BIN_LIBSTD_SOURCES)); do \
+		$(TAR) -C $(@D)/std -xJf $${file}; \
+	done
+	(\
+		cd $(@D)/rustc/lib/rustlib; \
+		ln -sf ../../../std/$(HOST_RUST_BIN_LIBSTD_HOST_PREFIX)/lib/rustlib/$(RUST_HOST_NAME) \
+	)
+endef
+
+HOST_RUST_BIN_POST_EXTRACT_HOOKS += HOST_RUST_BIN_LIBSTD_EXTRACT
+
+HOST_RUST_BIN_INSTALL_OPTS = \
+	--prefix=$(HOST_DIR) \
+	--disable-ldconfig
+
+ifeq ($(BR2_PACKAGE_HOST_RUST_BIN),y)
+define HOST_RUST_BIN_INSTALL_RUSTC
+	(cd $(@D); \
+		./install.sh $(HOST_RUST_BIN_INSTALL_OPTS))
+endef
+
+define HOST_RUST_BIN_INSTALL_LIBSTD_HOST
+	(cd $(@D)/std/rust-std-$(RUST_BIN_VERSION)-$(RUST_HOST_NAME); \
+		./install.sh $(HOST_RUST_BIN_INSTALL_OPTS))
+endef
+
+define HOST_RUST_BIN_INSTALL_LIBSTD_TARGET
+	(cd $(@D)/std/rust-std-$(RUST_BIN_VERSION)-$(RUST_TARGET_NAME); \
+		./install.sh $(HOST_RUST_BIN_INSTALL_OPTS))
+endef
+endif
+
+define HOST_RUST_BIN_INSTALL_CMDS
+	$(HOST_RUST_BIN_INSTALL_RUSTC)
+	$(HOST_RUST_BIN_INSTALL_LIBSTD_HOST)
+	$(HOST_RUST_BIN_INSTALL_LIBSTD_TARGET)
+endef
+
+$(eval $(host-generic-package))
diff --git a/package/rustc/Config.in.host b/package/rustc/Config.in.host
index 16c95fa0b4..e42187d477 100644
--- a/package/rustc/Config.in.host
+++ b/package/rustc/Config.in.host
@@ -10,8 +10,37 @@ config BR2_PACKAGE_HOST_RUSTC_ARCH_SUPPORTS
 	depends on !BR2_MIPS_NABI32
 	depends on BR2_TOOLCHAIN_USES_GLIBC
 
+config BR2_PACKAGE_HOST_RUSTC
+	bool "host rustc"
+	depends on BR2_PACKAGE_HOST_RUSTC_ARCH_SUPPORTS
+	help
+	  Compiler for the Rust language
+
+	  http://www.rust-lang.org
+
+if BR2_PACKAGE_HOST_RUSTC
+
+choice
+	prompt "Rust compiler variant"
+	default BR2_PACKAGE_HOST_RUST_BIN
+	help
+	  Select a Rust compiler
+
+config BR2_PACKAGE_HOST_RUST_BIN
+	bool "host rust (pre-built)"
+	select BR2_PACKAGE_HAS_HOST_RUSTC
+	help
+	  This package will install pre-built versions of the compiler
+	  for the host and the Rust standard library for the target.
+
+endchoice
+
 config BR2_PACKAGE_HAS_HOST_RUSTC
 	bool
 
 config BR2_PACKAGE_PROVIDES_HOST_RUSTC
 	string
+	default "host-rust-bin" if BR2_PACKAGE_HOST_RUST_BIN
+
+endif
+
-- 
2.14.3

[Buildroot] [PATCH v9 3/8] cargo-bin: new package

[Eric Le Bihan]

This new package fetches a binary version of Cargo, suitable to
bootstrap the host variants of the Rust compiler and Cargo, the package
manager.

Signed-off-by: Eric Le Bihan <eric.le.bihan.dev@free.fr>
---
 DEVELOPERS                       |  1 +
 package/cargo-bin/cargo-bin.hash |  4 ++++
 package/cargo-bin/cargo-bin.mk   | 14 ++++++++++++++
 3 files changed, 19 insertions(+)
 create mode 100644 package/cargo-bin/cargo-bin.hash
 create mode 100644 package/cargo-bin/cargo-bin.mk

diff --git a/DEVELOPERS b/DEVELOPERS
index bf4766886d..0374e45eda 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -537,6 +537,7 @@ F:	package/xxhash/
 
 N:	Eric Le Bihan <eric.le.bihan.dev@free.fr>
 F:	package/adwaita-icon-theme/
+F:	package/cargo-bin/
 F:	package/darkhttpd/
 F:	package/eudev/
 F:	package/execline/
diff --git a/package/cargo-bin/cargo-bin.hash b/package/cargo-bin/cargo-bin.hash
new file mode 100644
index 0000000000..70fb101be7
--- /dev/null
+++ b/package/cargo-bin/cargo-bin.hash
@@ -0,0 +1,4 @@
+# From https://static.rust-lang.org/dist/cargo-0.23.0-i686-unknown-linux-gnu.tar.gz.sha256
+sha256 7ebe231e5da2a06370f17050285ee694cf09ac2010d87dab334ae7eb7fb2d975  cargo-0.23.0-i686-unknown-linux-gnu.tar.gz
+# From https://static.rust-lang.org/dist/cargo-0.23.0-x86_64-unknown-linux-gnu.tar.xz.sha256
+sha256 4f33d9e511cfcbb370f470d3813e4877da10761d5f75460e538503fa07fa64f8  cargo-0.23.0-x86_64-unknown-linux-gnu.tar.xz
diff --git a/package/cargo-bin/cargo-bin.mk b/package/cargo-bin/cargo-bin.mk
new file mode 100644
index 0000000000..e0f6381070
--- /dev/null
+++ b/package/cargo-bin/cargo-bin.mk
@@ -0,0 +1,14 @@
+################################################################################
+#
+# cargo-bin
+#
+################################################################################
+
+CARGO_BIN_VERSION = 0.23.0
+CARGO_BIN_SITE = https://static.rust-lang.org/dist
+CARGO_BIN_SOURCE = cargo-$(CARGO_BIN_VERSION)-$(RUST_HOST_NAME).tar.xz
+CARGO_BIN_LICENSE = Apache-2.0 or MIT
+CARGO_BIN_LICENSE_FILES = LICENSE-APACHE LICENSE-MIT
+CARGO_BIN_STRIP_COMPONENTS = 1
+
+$(eval $(host-generic-package))
-- 
2.14.3

[Buildroot] [PATCH v9 4/8] rust: new package

[Eric Le Bihan]

This new package provides rustc, the compiler for the Rust programming
language, built from source.

Currently, only the host variant is built.

The Rust compiler uses LLVM as its backend: a copy of LLVM source code
is provided and CMake is used to build it. It is possible to use a
pre-built external copy. When LLVM/clang will be available in Buildroot,
it would be possible to benefit from this feature and thus decrease
build time.

LLVM is configured to generate code for x86, ARM, PowerPC and MIPS
architectures.

The Rust compiler uses Cargo as its build system and is written in Rust.
Therefore this package depends on cargo-bin and rust-bin.

The internal build process is as follows:

 1. rustc-stage0, provided by rust-bin, is used to build rustc-stage1.
 2. rust-stage1 builds the final Rust compiler (rust-stage2)
    and the standard library for the host architecture.
 3. the standard library for the target architecture is built.

The target architecture to support is given by the GNU/LLVM target
triple. Rust supports some predefined targets [1]. As the build system
expects the triple to be in the form of <arch>-unknown-<system> and
Buildroot toolchain wrapper uses <arch>-buildroot-<system>, the package
Makefile uses $(RUST_TARGET_NAME) defined in the rustc package and uses
it instead of $(GNU_TARGET_NAME).

When compiling Rust code with this compiler, the generated program only
depends on the target C library, as it is statically linked to the Rust
standard library and any other code from Rust packages (a.k.a.
"crates").

If the jemalloc package is selected, support for this memory allocator
will be enabled in the target standard library.

The menuconfig entry for rustc is also updated to expose this provider.

[1] https://forge.rust-lang.org/platform-support.html

Signed-off-by: Eric Le Bihan <eric.le.bihan.dev@free.fr>
---
 DEVELOPERS                   |  1 +
 package/rust/rust.hash       |  2 ++
 package/rust/rust.mk         | 82 ++++++++++++++++++++++++++++++++++++++++++++
 package/rustc/Config.in.host | 16 +++++++++
 4 files changed, 101 insertions(+)
 create mode 100644 package/rust/rust.hash
 create mode 100644 package/rust/rust.mk

diff --git a/DEVELOPERS b/DEVELOPERS
index 0374e45eda..1c16ec9598 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -545,6 +545,7 @@ F:	package/hicolor-icon-theme/
 F:	package/jemalloc/
 F:	package/ninja/
 F:	package/rust-bin/
+F:	package/rust/
 F:	package/s6/
 F:	package/s6-dns/
 F:	package/s6-linux-init/
diff --git a/package/rust/rust.hash b/package/rust/rust.hash
new file mode 100644
index 0000000000..51aa9e3361
--- /dev/null
+++ b/package/rust/rust.hash
@@ -0,0 +1,2 @@
+# From https://static.rust-lang.org/dist/rustc-1.22.1-src.tar.xz.sha256
+sha256 80ee9ecc1e03ee63ea13c2612b61fc04fce9240476836f70c553ebaebd58fed6  rustc-1.22.1-src.tar.xz
diff --git a/package/rust/rust.mk b/package/rust/rust.mk
new file mode 100644
index 0000000000..8dbec99bc5
--- /dev/null
+++ b/package/rust/rust.mk
@@ -0,0 +1,82 @@
+################################################################################
+#
+# rust
+#
+################################################################################
+
+RUST_VERSION = 1.22.1
+RUST_SOURCE = rustc-$(RUST_VERSION)-src.tar.xz
+RUST_SITE = https://static.rust-lang.org/dist
+RUST_LICENSE = Apache-2.0 or MIT
+RUST_LICENSE_FILES = LICENSE-APACHE LICENSE-MIT
+
+HOST_RUST_PROVIDES = host-rustc
+
+HOST_RUST_DEPENDENCIES = \
+	toolchain \
+	host-rust-bin \
+	host-cargo-bin \
+	host-python \
+	host-cmake
+
+ifeq ($(BR2_PACKAGE_JEMALLOC),y)
+HOST_RUST_DEPENDENCIES += jemalloc
+HOST_RUST_JEMALLOC_ENABLED = true
+HOST_RUST_JEMALLOC_CONF = 'jemalloc = "$(STAGING_DIR)/usr/lib/libjemalloc_pic.a"'
+else
+HOST_RUST_JEMALLOC_ENABLED = false
+endif
+
+HOST_RUST_VERBOSITY = $(if $(VERBOSE),2,0)
+
+# Some vendored crates contain Cargo.toml.orig files. The associated
+# .cargo-checksum.json file will contain a checksum for Cargo.toml.orig but
+# support/scripts/apply-patches.sh will delete them. This will cause the build
+# to fail, as Cargo will not be able to find the file and verify the checksum.
+# So, remove all Cargo.toml.orig entries from the affected .cargo-checksum.json
+# files
+define HOST_RUST_EXCLUDE_ORIG_FILES
+	for file in $$(find $(@D) -name '*.orig'); do \
+		crate=$$(dirname $${file}); \
+		fn=$${crate}/.cargo-checksum.json; \
+		sed -i -e 's/"Cargo.toml.orig":"[a-z0-9]\+",//g' $${fn}; \
+	done
+endef
+
+HOST_RUST_POST_EXTRACT_HOOKS += HOST_RUST_EXCLUDE_ORIG_FILES
+
+define HOST_RUST_CONFIGURE_CMDS
+	( \
+		echo '[build]'; \
+		echo 'target = ["$(RUST_TARGET_NAME)"]'; \
+		echo 'cargo = "$(HOST_CARGO_BIN_DIR)/cargo/bin/cargo"'; \
+		echo 'rustc = "$(HOST_RUST_BIN_DIR)/rustc/bin/rustc"'; \
+		echo 'python = "$(HOST_DIR)/bin/python2"'; \
+		echo 'submodules = false'; \
+		echo 'vendor = true'; \
+		echo 'compiler-docs = false'; \
+		echo 'docs = false'; \
+		echo 'verbose = $(HOST_RUST_VERBOSITY)'; \
+		echo '[install]'; \
+		echo 'prefix = "$(HOST_DIR)"'; \
+		echo '[rust]'; \
+		echo 'use-jemalloc = $(HOST_RUST_JEMALLOC_ENABLED)'; \
+		echo '[target.$(RUST_TARGET_NAME)]'; \
+		echo 'cc = "$(TARGET_CROSS)gcc"'; \
+		echo $(HOST_RUST_JEMALLOC_CONF); \
+	) > $(@D)/config.toml
+endef
+
+define HOST_RUST_BUILD_CMDS
+	(cd $(@D); $(HOST_MAKE_ENV) $(HOST_DIR)/bin/python2 x.py \
+		build $(HOST_RUST_BUILD_OPTS))
+endef
+
+define HOST_RUST_INSTALL_CMDS
+	(cd $(@D); $(HOST_MAKE_ENV) $(HOST_DIR)/bin/python2 x.py \
+		dist $(HOST_RUST_BUILD_OPTS))
+	(cd $(@D); $(HOST_MAKE_ENV) $(HOST_DIR)/bin/python2 x.py \
+		install $(HOST_RUST_BUILD_OPTS))
+endef
+
+$(eval $(host-generic-package))
diff --git a/package/rustc/Config.in.host b/package/rustc/Config.in.host
index e42187d477..3a03140bcd 100644
--- a/package/rustc/Config.in.host
+++ b/package/rustc/Config.in.host
@@ -26,6 +26,21 @@ choice
 	help
 	  Select a Rust compiler
 
+config BR2_PACKAGE_HOST_RUST
+	bool "host rust"
+	depends on BR2_HOST_GCC_AT_LEAST_4_7 # required by LLVM
+	# triggers ICE on trunc_int_for_mode, at explow.c:56
+	depends on BR2_TOOLCHAIN_GCC_AT_LEAST_5 || !BR2_aarch64
+	select BR2_PACKAGE_HAS_HOST_RUSTC
+	help
+	  This package will build the compiler for the host as well as
+	  two flavors of the standard library: one for the host, another
+	  for the target. Both are installed in the host directory.
+
+comment "host-rust needs a toolchain w/ gcc >= 5"
+	depends on BR2_aarch64
+	depends on !BR2_TOOLCHAIN_GCC_AT_LEAST_5
+
 config BR2_PACKAGE_HOST_RUST_BIN
 	bool "host rust (pre-built)"
 	select BR2_PACKAGE_HAS_HOST_RUSTC
@@ -40,6 +55,7 @@ config BR2_PACKAGE_HAS_HOST_RUSTC
 
 config BR2_PACKAGE_PROVIDES_HOST_RUSTC
 	string
+	default "host-rust" if BR2_PACKAGE_HOST_RUST
 	default "host-rust-bin" if BR2_PACKAGE_HOST_RUST_BIN
 
 endif
-- 
2.14.3

[Buildroot] [PATCH v9 5/8] libssh2: add host variant

[Eric Le Bihan]

Allow build of host variant of libssh2, which depends on host-openssl.

Signed-off-by: Eric Le Bihan <eric.le.bihan.dev@free.fr>
---
 package/libssh2/libssh2.mk | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/package/libssh2/libssh2.mk b/package/libssh2/libssh2.mk
index 18e772c722..c5b0bd855e 100644
--- a/package/libssh2/libssh2.mk
+++ b/package/libssh2/libssh2.mk
@@ -40,4 +40,10 @@ else
 LIBSSH2_CONF_OPTS += --without-libz
 endif
 
+HOST_LIBSSH2_DEPENDENCIES += host-openssl
+HOST_LIBSSH2_CONF_OPTS += --with-openssl \
+	--with-libssl-prefix=$(HOST_DIR)/usr \
+	--without-libgcrypt
+
 $(eval $(autotools-package))
+$(eval $(host-autotools-package))
-- 
2.14.3

[Buildroot] [PATCH v9 6/8] libhttpparser: add host variant

[Eric Le Bihan]

Allow build of host variant of libhttpparser.

Signed-off-by: Eric Le Bihan <eric.le.bihan.dev@free.fr>
---
 package/libhttpparser/libhttpparser.mk | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/package/libhttpparser/libhttpparser.mk b/package/libhttpparser/libhttpparser.mk
index 8ff0413322..1df26017e7 100644
--- a/package/libhttpparser/libhttpparser.mk
+++ b/package/libhttpparser/libhttpparser.mk
@@ -23,4 +23,13 @@ define LIBHTTPPARSER_INSTALL_TARGET_CMDS
 	$(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(@D) PREFIX=$(TARGET_DIR)/usr install
 endef
 
+define HOST_LIBHTTPPARSER_BUILD_CMDS
+	$(MAKE) $(HOST_CONFIGURE_OPTS) -C $(@D) library package
+endef
+
+define HOST_LIBHTTPPARSER_INSTALL_CMDS
+	$(MAKE) $(HOST_CONFIGURE_OPTS) -C $(@D) PREFIX=$(HOST_DIR)/usr install
+endef
+
 $(eval $(generic-package))
+$(eval $(host-generic-package))
-- 
2.14.3

[Buildroot] [PATCH v9 7/8] libcurl: add host variant

[Eric Le Bihan]

Signed-off-by: Eric Le Bihan <eric.le.bihan.dev@free.fr>
---
 package/libcurl/libcurl.mk | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/package/libcurl/libcurl.mk b/package/libcurl/libcurl.mk
index 0aab58a7de..ea5c99cf70 100644
--- a/package/libcurl/libcurl.mk
+++ b/package/libcurl/libcurl.mk
@@ -88,4 +88,18 @@ endef
 LIBCURL_POST_INSTALL_TARGET_HOOKS += LIBCURL_TARGET_CLEANUP
 endif
 
+HOST_LIBCURL_DEPENDENCIES = host-openssl
+HOST_LIBCURL_CONF_OPTS = \
+	--disable-manual \
+	--disable-ntlm-wb \
+	--disable-curldebug \
+	--with-ssl \
+	--without-gnutls \
+	--without-mbedtls \
+	--without-polarssl \
+	--without-nss
+
+HOST_LIBCURL_POST_PATCH_HOOKS += LIBCURL_FIX_DOT_PC
+
 $(eval $(autotools-package))
+$(eval $(host-autotools-package))
-- 
2.14.3

[Buildroot] [PATCH v9 8/8] cargo: new package

[Eric Le Bihan]

This new package provides Cargo, the Rust official package manager.
Cargo is written in Rust and uses Cargo as its build system. It also
depends on other Rust packages.

Normally, a previously installed version of Cargo would be used to:

 1. Fetch the dependencies.
 2. Build the new version of Cargo, using the available Rust compiler.

But the fetching step prevents offline builds. So instead two features
of Cargo are levelled: vendoring [1] and local registry.

First, a tarball of the build dependencies generated using `cargo
vendor` is fetched along with Cargo source code.

Then, the build process is as follows:

 1. The tarball of the build dependencies is uncompressed in a local
    registry.
 2. A snapshot of Cargo, provided by cargo-bin, builds the final
    version of Cargo.
 3. A configuration file telling Cargo how to cross-compile programs for
    the target is generated and installed.

Currently, only the host variant is provided.

[1] https://github.com/alexcrichton/cargo-vendor

Signed-off-by: Eric Le Bihan <eric.le.bihan.dev@free.fr>
---
 DEVELOPERS                   |  1 +
 package/Config.in.host       |  1 +
 package/cargo/Config.in.host |  8 ++++
 package/cargo/cargo.hash     |  4 ++
 package/cargo/cargo.mk       | 96 ++++++++++++++++++++++++++++++++++++++++++++
 package/cargo/config.in      |  2 +
 6 files changed, 112 insertions(+)
 create mode 100644 package/cargo/Config.in.host
 create mode 100644 package/cargo/cargo.hash
 create mode 100644 package/cargo/cargo.mk
 create mode 100644 package/cargo/config.in

diff --git a/DEVELOPERS b/DEVELOPERS
index 1c16ec9598..0902fd2fd4 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -538,6 +538,7 @@ F:	package/xxhash/
 N:	Eric Le Bihan <eric.le.bihan.dev@free.fr>
 F:	package/adwaita-icon-theme/
 F:	package/cargo-bin/
+F:	package/cargo/
 F:	package/darkhttpd/
 F:	package/eudev/
 F:	package/execline/
diff --git a/package/Config.in.host b/package/Config.in.host
index 2f20dd0e48..199a8e9856 100644
--- a/package/Config.in.host
+++ b/package/Config.in.host
@@ -2,6 +2,7 @@ menu "Host utilities"
 
 	source "package/aespipe/Config.in.host"
 	source "package/android-tools/Config.in.host"
+	source "package/cargo/Config.in.host"
 	source "package/cbootimage/Config.in.host"
 	source "package/checkpolicy/Config.in.host"
 	source "package/cmake/Config.in.host"
diff --git a/package/cargo/Config.in.host b/package/cargo/Config.in.host
new file mode 100644
index 0000000000..0f1ca305c6
--- /dev/null
+++ b/package/cargo/Config.in.host
@@ -0,0 +1,8 @@
+config BR2_PACKAGE_HOST_CARGO
+	bool "host cargo"
+	depends on BR2_PACKAGE_HAS_HOST_RUSTC
+	help
+	  Cargo is the package manager for the Rust programming
+	  language.
+
+	  https://crates.io/
diff --git a/package/cargo/cargo.hash b/package/cargo/cargo.hash
new file mode 100644
index 0000000000..2b2ae43f6b
--- /dev/null
+++ b/package/cargo/cargo.hash
@@ -0,0 +1,4 @@
+# Locally generated
+sha256 f4bbe2a8719dbb8da20842235093f7f70f034d01633189e83f75897d68cd274f  cargo-0.23.0.tar.gz
+sha512 9060ec6e67b54f7fad7da8dd8450dd051d62b3f8ed4606196fc238a98beba1c3b43087c787f35d012d9b641c8572e70f50b95b0e01fdd75ed82932b6e6efbbf0  cargo-0.23.0-vendor.tar.xz
+sha256 dc7240d60a869fa24a68c8734fb7c810c27cca0a6dad52df6279865e4e8e7fae  rust-installer-4f994850808a572e2cc8d43f968893c8e942e9bf.tar.gz
diff --git a/package/cargo/cargo.mk b/package/cargo/cargo.mk
new file mode 100644
index 0000000000..3794f450c1
--- /dev/null
+++ b/package/cargo/cargo.mk
@@ -0,0 +1,96 @@
+################################################################################
+#
+# cargo
+#
+################################################################################
+
+CARGO_VERSION = 0.23.0
+CARGO_SITE = $(call github,rust-lang,cargo,$(CARGO_VERSION))
+CARGO_LICENSE = Apache-2.0 or MIT
+CARGO_LICENSE_FILES = LICENSE-APACHE LICENSE-MIT
+
+CARGO_DEPS_SHA512 = 9060ec6e67b54f7fad7da8dd8450dd051d62b3f8ed4606196fc238a98beba1c3b43087c787f35d012d9b641c8572e70f50b95b0e01fdd75ed82932b6e6efbbf0
+CARGO_DEPS_SITE = http://pkgs.fedoraproject.org/repo/pkgs/cargo/$(CARGO_DEPS_SOURCE)/sha512/$(CARGO_DEPS_SHA512)
+CARGO_DEPS_SOURCE = cargo-$(CARGO_VERSION)-vendor.tar.xz
+
+CARGO_INSTALLER_VERSION = 4f994850808a572e2cc8d43f968893c8e942e9bf
+CARGO_INSTALLER_SITE = $(call github,rust-lang,rust-installer,$(CARGO_INSTALLER_VERSION))
+CARGO_INSTALLER_SOURCE = rust-installer-$(CARGO_INSTALLER_VERSION).tar.gz
+
+HOST_CARGO_EXTRA_DOWNLOADS = \
+	$(CARGO_DEPS_SITE)/$(CARGO_DEPS_SOURCE) \
+	$(CARGO_INSTALLER_SITE)/$(CARGO_INSTALLER_SOURCE)
+
+HOST_CARGO_DEPENDENCIES = \
+	host-cmake \
+	host-pkgconf \
+	host-openssl \
+	host-libhttpparser \
+	host-libssh2 \
+	host-libcurl \
+	host-rustc \
+	host-cargo-bin
+
+HOST_CARGO_SNAP_BIN = $(HOST_CARGO_BIN_DIR)/cargo/bin/cargo
+HOST_CARGO_HOME = $(HOST_DIR)/share/cargo
+
+define HOST_CARGO_EXTRACT_DEPS
+	@mkdir -p $(@D)/vendor
+	$(call suitable-extractor,$(CARGO_DEPS_SOURCE)) \
+		$(DL_DIR)/$(CARGO_DEPS_SOURCE) | \
+		$(TAR) --strip-components=1 -C $(@D)/vendor $(TAR_OPTIONS) -
+endef
+
+HOST_CARGO_POST_EXTRACT_HOOKS += HOST_CARGO_EXTRACT_DEPS
+
+define HOST_CARGO_EXTRACT_INSTALLER
+	@mkdir -p $(@D)/src/rust-installer
+	$(call suitable-extractor,$(CARGO_INSTALLER_SOURCE)) \
+		$(DL_DIR)/$(CARGO_INSTALLER_SOURCE) | \
+		$(TAR) --strip-components=1 -C $(@D)/src/rust-installer $(TAR_OPTIONS) -
+endef
+
+HOST_CARGO_POST_EXTRACT_HOOKS += HOST_CARGO_EXTRACT_INSTALLER
+
+define HOST_CARGO_SETUP_DEPS
+	mkdir -p $(@D)/.cargo
+	( \
+		echo "[source.crates-io]"; \
+		echo "registry = 'https://github.com/rust-lang/crates.io-index'"; \
+		echo "replace-with = 'vendored-sources'"; \
+		echo "[source.vendored-sources]"; \
+		echo "directory = '$(@D)/vendor'"; \
+	) > $(@D)/.cargo/config
+endef
+
+HOST_CARGO_PRE_CONFIGURE_HOOKS += HOST_CARGO_SETUP_DEPS
+
+HOST_CARGO_SNAP_OPTS = --release
+HOST_CARGO_SNAP_OPTS += $(if $(VERBOSE),--verbose)
+
+HOST_CARGO_ENV = \
+	RUSTFLAGS="-Clink-arg=-Wl,-rpath,$(HOST_DIR)/lib" \
+	CARGO_HOME=$(HOST_DIR)/share/cargo
+
+define HOST_CARGO_BUILD_CMDS
+	(cd $(@D); $(HOST_MAKE_ENV) $(HOST_CARGO_ENV) $(HOST_CARGO_SNAP_BIN) \
+		build $(HOST_CARGO_SNAP_OPTS))
+endef
+
+define HOST_CARGO_INSTALL_CMDS
+	$(INSTALL) -d -m 0755 $(HOST_DIR)/bin
+	$(INSTALL) -m 0755 $(@D)/target/release/cargo $(HOST_DIR)/bin/cargo
+endef
+
+define HOST_CARGO_INSTALL_CONF_FILE
+	$(INSTALL) -D package/cargo/config.in \
+		$(HOST_DIR)/share/cargo/config
+	$(SED) 's/@RUST_TARGET_NAME@/$(RUST_TARGET_NAME)/' \
+		$(HOST_DIR)/share/cargo/config
+	$(SED) 's/@CROSS_PREFIX@/$(notdir $(TARGET_CROSS))/' \
+		$(HOST_DIR)/share/cargo/config
+endef
+
+HOST_CARGO_POST_INSTALL_HOOKS += HOST_CARGO_INSTALL_CONF_FILE
+
+$(eval $(host-generic-package))
diff --git a/package/cargo/config.in b/package/cargo/config.in
new file mode 100644
index 0000000000..cc048c71c4
--- /dev/null
+++ b/package/cargo/config.in
@@ -0,0 +1,2 @@
+[target. at RUST_TARGET_NAME@]
+linker = "@CROSS_PREFIX at gcc"
-- 
2.14.3

[Buildroot] [PATCH v9 0/8] Add support for the Rust programming

[Thomas Petazzoni]

Hello,

On Thu, 28 Dec 2017 16:51:38 +0100, Eric Le Bihan wrote:
> This series adds support for the Rust programming language by adding the
> following packages:
> 
>  - rustc: a virtual package for the Rust compiler.
>  - rust-bin: provides a pre-built version of rustc.
>  - cargo-bin: provides a pre-built version of Rust package manager.
>  - rust: builds rustc from source.
>  - cargo: builds Rust package manager from source.

One question: with your series, one can therefore chose whether he
wants to build its own Rust compiler and standard library ("rust"
package) or use the pre-built Rust compiler and standard library
("rustc-bin" package) ?

In other words, does this means that there are two options:

 - User chooses rustc-bin, every Rust package is built using the
   pre-built compiler, and the pre-built standard library is used.

 - User chooses rust. In this case, rust is built using rustc-bin, but
   every other Rust package will be built with the Rust compiler we
   have built from source, and will be using the Rust standard library
   we have built from source.

Is this correct ?

(I'm just trying to understand the overall series.)

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v9 1/8] rustc: new virtual package

[Thomas Petazzoni]

Hello,

On Thu, 28 Dec 2017 16:51:39 +0100, Eric Le Bihan wrote:

> diff --git a/package/rustc/Config.in.host b/package/rustc/Config.in.host
> new file mode 100644
> index 0000000000..16c95fa0b4
> --- /dev/null
> +++ b/package/rustc/Config.in.host
> @@ -0,0 +1,17 @@
> +config BR2_PACKAGE_HOST_RUSTC_ARCH_SUPPORTS
> +	bool
> +	default y
> +	depends on BR2_HOSTARCH = "x86_64" || BR2_HOSTARCH = "x86"
> +	depends on BR2_i386 || BR2_x86_64 \
> +		|| BR2_arm  || BR2_aarch64 \
> +		|| BR2_powerpc  || BR2_powerpc64 \
> +		|| BR2_mips || BR2_mipsel || BR2_mips64 || BR2_mips64el

This would be more readable as:

	default y if BR2_aarch64
	default y if BR2_arm && !BR2_ARM_CPU_ARMV4 && !BR2_ARM_CPU_ARMV5
	default y if BR2_i386
	default y if ...
	default y if BR2_mips || BR2_mipsel
	default y if (BR2_mips64 || BR2_mips64el) && BR2_MIPS_NABI64
	default y if BR2_x86_64

> +	depends on !BR2_ARM_CPU_ARMV4 && !BR2_ARM_CPU_ARMV5
> +	depends on !BR2_MIPS_NABI32
> +	depends on BR2_TOOLCHAIN_USES_GLIBC

Since it's weird for a host package to have a target architecture
dependency and target C library dependency, it would be good to have a
comment that explains this. I assume it's because of the Rust standard
library, so something like:

	# The pre-built Rust standard library is only available for the
	# following architectures/ABIs, and is built against glibc.

> diff --git a/package/rustc/rustc.mk b/package/rustc/rustc.mk
> new file mode 100644
> index 0000000000..4163a44bc1
> --- /dev/null
> +++ b/package/rustc/rustc.mk
> @@ -0,0 +1,31 @@
> +################################################################################
> +#
> +# rustc
> +#
> +################################################################################
> +
> +RUST_ARCH := $(call qstrip,$(BR2_ARCH))
> +
> +ifeq ($(BR2_ARM_CPU_ARMV7A),y)
> +RUST_ARCH := armv7
> +endif
> +
> +ifeq ($(BR2_ARM_EABI),y)
> +RUST_ABI := eabi
> +else ifeq ($(BR2_ARM_EABIHF),y)
> +RUST_ABI := eabihf
> +else ifeq ($(BR2_mips64)$(BR2_mips64el),y)
> +RUST_ABI := abi64
> +endif

Perhaps this could be defined in the Config.in file as well ?

config BR2_PACKAGE_HOST_RUSTC_ARCH
	string
	default "armv7" 	if BR2_ARM_CPU_ARMV7A
	default BR2_ARCH 	if !BR2_ARM_CPU_ARMV7A

config BR2_PACKAGE_HOST_RUSTC_ABI
	string
	default "eabi" 		if BR2_ARM_EABI
	default "eabihf" 	if BR2_ARM_EABIHF
	default "abi64" 	if BR2_MIPS_NABI64

> +RUST_TARGET_NAME := $(RUST_ARCH)-unknown-linux-gnu$(RUST_ABI)

Why are you using := in this file instead of = ?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v9 0/8] Add support for the Rust programming

[Eric Le Bihan]

Hi!

On 17-12-28 17:21:29, Thomas Petazzoni wrote:
> Hello,
>
> On Thu, 28 Dec 2017 16:51:38 +0100, Eric Le Bihan wrote:
> > This series adds support for the Rust programming language by adding the
> > following packages:
> >
> >  - rustc: a virtual package for the Rust compiler.
> >  - rust-bin: provides a pre-built version of rustc.
> >  - cargo-bin: provides a pre-built version of Rust package manager.
> >  - rust: builds rustc from source.
> >  - cargo: builds Rust package manager from source.
>
> One question: with your series, one can therefore chose whether he
> wants to build its own Rust compiler and standard library ("rust"
> package) or use the pre-built Rust compiler and standard library
> ("rustc-bin" package) ?
>
> In other words, does this means that there are two options:
>
>  - User chooses rustc-bin, every Rust package is built using the
>    pre-built compiler, and the pre-built standard library is used.
>
>  - User chooses rust. In this case, rust is built using rustc-bin, but
>    every other Rust package will be built with the Rust compiler we
>    have built from source, and will be using the Rust standard library
>    we have built from source.
>
> Is this correct ?
>
> (I'm just trying to understand the overall series.)

Yes, this is correct. The introduction of a virtual package for the Rust
compiler was suggested in the mail thread about version v4 of this
series [1], as building the compiler from source takes time and using the
available pre-built version would speed things up (as done for GCC).

[1] http://lists.busybox.net/pipermail/buildroot/2017-April/189451.html

Regards,

--
ELB

[Buildroot] [PATCH v9 2/8] rust-bin: new package

[Thomas Petazzoni]

Hello,

On Thu, 28 Dec 2017 16:51:40 +0100, Eric Le Bihan wrote:
> This package provides a pre-built version of rustc, the compiler for the
> Rust programming language, fetched from the upstream project.
> 
> A pre-built version of the standard library for the host as well as one
> for the chosen target are also fetched and installed.

I was wondering if a separate package could be used to install the
standard library, in order to avoid the extra downloads and custom
extract step.

One possibility would be to have a rust-std-bin package, which rust-bin
depends on. The target version of rust-std-bin would install the target
libraries, while the host version would install the host libraries. The
weird thing is that the target libraries are installed in $(HOST_DIR),
while one would expect them to be in $(STAGING_DIR). Is this something
we can tell rustc about perhaps ?

Even if we keep the single package approach, having the target
libraries in $(STAGING_DIR) would be more logical. But that's a soft
requirement, so if it's too tedious to achieve with rustc, we can live
with them being in HOST_DIR.

> diff --git a/package/rust-bin/rust-bin.mk b/package/rust-bin/rust-bin.mk
> new file mode 100644
> index 0000000000..29f94b7f2d
> --- /dev/null
> +++ b/package/rust-bin/rust-bin.mk
> @@ -0,0 +1,63 @@
> +################################################################################
> +#
> +# rust-bin
> +#
> +################################################################################
> +
> +RUST_BIN_VERSION = 1.22.1
> +RUST_BIN_SITE = https://static.rust-lang.org/dist
> +RUST_BIN_LICENSE = Apache-2.0 or MIT
> +RUST_BIN_LICENSE_FILES = LICENSE-APACHE LICENSE-MIT
> +
> +HOST_RUST_BIN_PROVIDES = host-rustc
> +
> +HOST_RUST_BIN_SOURCE = rustc-$(RUST_BIN_VERSION)-$(RUST_HOST_NAME).tar.xz
> +HOST_RUST_BIN_LIBSTD_SOURCES = \
> +	rust-std-$(RUST_BIN_VERSION)-$(RUST_HOST_NAME).tar.xz \
> +	rust-std-$(RUST_BIN_VERSION)-$(RUST_TARGET_NAME).tar.xz
> +
> +HOST_RUST_BIN_EXTRA_DOWNLOADS = $(HOST_RUST_BIN_LIBSTD_SOURCES)

Assuming we keep the single package approach, then why not use
HOST_RUST_BIN_EXTRA_DOWNLOADS directly ? HOST_RUST_BIN_LIBSTD_SOURCES
looks a bit useless.

> +HOST_RUST_BIN_LIBSTD_HOST_PREFIX = rust-std-$(RUST_BIN_VERSION)-$(RUST_HOST_NAME)/rust-std-$(RUST_HOST_NAME)
> +
> +define HOST_RUST_BIN_LIBSTD_EXTRACT
> +	mkdir -p $(@D)/std
> +	for file in $(addprefix $(DL_DIR)/,$(HOST_RUST_BIN_LIBSTD_SOURCES)); do \
> +		$(TAR) -C $(@D)/std -xJf $${file}; \

This would benefit from using $(foreach ...) and should use
suitable-extractor. Something like (untested):

	$(foreach f,$(HOST_RUST_BIN_EXTRA_DOWNLOADS), \
		$(call suitable-extractor,$(f)) $(DL_DIR)/$(f) | \
			$(TAR) -C $(@D)/std $(TAR_OPTIONS) -
	)

> +	done
> +	(\
> +		cd $(@D)/rustc/lib/rustlib; \
> +		ln -sf ../../../std/$(HOST_RUST_BIN_LIBSTD_HOST_PREFIX)/lib/rustlib/$(RUST_HOST_NAME) \
> +	)

There's no need for this command to run in a sub-shell, so:

	cd $(@D)/rustc/lib/rustlib; \
		ln -sf ../../../std/$(HOST_RUST_BIN_LIBSTD_HOST_PREFIX)/lib/rustlib/$(RUST_HOST_NAME)

should be sufficient.

> +endef
> +
> +HOST_RUST_BIN_POST_EXTRACT_HOOKS += HOST_RUST_BIN_LIBSTD_EXTRACT
> +
> +HOST_RUST_BIN_INSTALL_OPTS = \
> +	--prefix=$(HOST_DIR) \
> +	--disable-ldconfig
> +
> +ifeq ($(BR2_PACKAGE_HOST_RUST_BIN),y)

Why do you have this conditional ? If the package isn't built, then
those definitions aren't used anyway.

The rest looks good to me. Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v9 3/8] cargo-bin: new package

[Thomas Petazzoni]

Hello,

On Thu, 28 Dec 2017 16:51:41 +0100, Eric Le Bihan wrote:
> This new package fetches a binary version of Cargo, suitable to
> bootstrap the host variants of the Rust compiler and Cargo, the package
> manager.
> 
> Signed-off-by: Eric Le Bihan <eric.le.bihan.dev@free.fr>

Looks good, just one comment below.

> diff --git a/package/cargo-bin/cargo-bin.mk b/package/cargo-bin/cargo-bin.mk
> new file mode 100644
> index 0000000000..e0f6381070
> --- /dev/null
> +++ b/package/cargo-bin/cargo-bin.mk
> @@ -0,0 +1,14 @@
> +################################################################################
> +#
> +# cargo-bin
> +#
> +################################################################################
> +
> +CARGO_BIN_VERSION = 0.23.0
> +CARGO_BIN_SITE = https://static.rust-lang.org/dist
> +CARGO_BIN_SOURCE = cargo-$(CARGO_BIN_VERSION)-$(RUST_HOST_NAME).tar.xz
> +CARGO_BIN_LICENSE = Apache-2.0 or MIT
> +CARGO_BIN_LICENSE_FILES = LICENSE-APACHE LICENSE-MIT
> +CARGO_BIN_STRIP_COMPONENTS = 1

This is the default, so why do you need this ?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v9 4/8] rust: new package

[Thomas Petazzoni]

Hello,

On Thu, 28 Dec 2017 16:51:42 +0100, Eric Le Bihan wrote:
> This new package provides rustc, the compiler for the Rust programming
> language, built from source.
> 
> Currently, only the host variant is built.
> 
> The Rust compiler uses LLVM as its backend: a copy of LLVM source code
> is provided and CMake is used to build it. It is possible to use a
> pre-built external copy. When LLVM/clang will be available in Buildroot,
> it would be possible to benefit from this feature and thus decrease
> build time.
> 
> LLVM is configured to generate code for x86, ARM, PowerPC and MIPS
> architectures.
> 
> The Rust compiler uses Cargo as its build system and is written in Rust.
> Therefore this package depends on cargo-bin and rust-bin.
> 
> The internal build process is as follows:
> 
>  1. rustc-stage0, provided by rust-bin, is used to build rustc-stage1.
>  2. rust-stage1 builds the final Rust compiler (rust-stage2)
>     and the standard library for the host architecture.

You're sometimes using rustc-stageX, sometimes just rust-stageX. Is
this intentional ?

> diff --git a/package/rust/rust.mk b/package/rust/rust.mk
> new file mode 100644
> index 0000000000..8dbec99bc5
> --- /dev/null
> +++ b/package/rust/rust.mk
> @@ -0,0 +1,82 @@
> +################################################################################
> +#
> +# rust
> +#
> +################################################################################
> +
> +RUST_VERSION = 1.22.1
> +RUST_SOURCE = rustc-$(RUST_VERSION)-src.tar.xz
> +RUST_SITE = https://static.rust-lang.org/dist
> +RUST_LICENSE = Apache-2.0 or MIT
> +RUST_LICENSE_FILES = LICENSE-APACHE LICENSE-MIT
> +
> +HOST_RUST_PROVIDES = host-rustc
> +
> +HOST_RUST_DEPENDENCIES = \
> +	toolchain \

It is somewhat unusual to have a host package depend on the target
toolchain. I'm not sure we have any other host package in the tree
doing this. I do understand why you're doing this, I'm just a little
bit concerned that this special situation may be forgotten in the
future. But there isn't much other solution, so it's probably good as
it is.

> +	host-rust-bin \
> +	host-cargo-bin \
> +	host-python \

So you absolutely need python2 ? The build system is not python3
compliant ?

> +	host-cmake

Could you use $(BR2_CMAKE_HOST_DEPENDENCY) instead ? This way, it would
only build host-cmake if CMake is not already present on the system.
This would save a bit of build time.

> +ifeq ($(BR2_PACKAGE_JEMALLOC),y)
> +HOST_RUST_DEPENDENCIES += jemalloc
> +HOST_RUST_JEMALLOC_ENABLED = true
> +HOST_RUST_JEMALLOC_CONF = 'jemalloc = "$(STAGING_DIR)/usr/lib/libjemalloc_pic.a"'
> +else
> +HOST_RUST_JEMALLOC_ENABLED = false
> +endif
> +
> +HOST_RUST_VERBOSITY = $(if $(VERBOSE),2,0)
> +
> +# Some vendored crates contain Cargo.toml.orig files. The associated

vendored -> vendor ?

> +# .cargo-checksum.json file will contain a checksum for Cargo.toml.orig but
> +# support/scripts/apply-patches.sh will delete them. This will cause the build
> +# to fail, as Cargo will not be able to find the file and verify the checksum.
> +# So, remove all Cargo.toml.orig entries from the affected .cargo-checksum.json
> +# files
> +define HOST_RUST_EXCLUDE_ORIG_FILES
> +	for file in $$(find $(@D) -name '*.orig'); do \
> +		crate=$$(dirname $${file}); \
> +		fn=$${crate}/.cargo-checksum.json; \
> +		sed -i -e 's/"Cargo.toml.orig":"[a-z0-9]\+",//g' $${fn}; \
> +	done
> +endef
> +
> +HOST_RUST_POST_EXTRACT_HOOKS += HOST_RUST_EXCLUDE_ORIG_FILES
> +
> +define HOST_RUST_CONFIGURE_CMDS
> +	( \
> +		echo '[build]'; \
> +		echo 'target = ["$(RUST_TARGET_NAME)"]'; \
> +		echo 'cargo = "$(HOST_CARGO_BIN_DIR)/cargo/bin/cargo"'; \
> +		echo 'rustc = "$(HOST_RUST_BIN_DIR)/rustc/bin/rustc"'; \
> +		echo 'python = "$(HOST_DIR)/bin/python2"'; \
> +		echo 'submodules = false'; \
> +		echo 'vendor = true'; \
> +		echo 'compiler-docs = false'; \
> +		echo 'docs = false'; \
> +		echo 'verbose = $(HOST_RUST_VERBOSITY)'; \
> +		echo '[install]'; \
> +		echo 'prefix = "$(HOST_DIR)"'; \
> +		echo '[rust]'; \
> +		echo 'use-jemalloc = $(HOST_RUST_JEMALLOC_ENABLED)'; \
> +		echo '[target.$(RUST_TARGET_NAME)]'; \
> +		echo 'cc = "$(TARGET_CROSS)gcc"'; \
> +		echo $(HOST_RUST_JEMALLOC_CONF); \
> +	) > $(@D)/config.toml
> +endef
> +
> +define HOST_RUST_BUILD_CMDS
> +	(cd $(@D); $(HOST_MAKE_ENV) $(HOST_DIR)/bin/python2 x.py \
> +		build $(HOST_RUST_BUILD_OPTS))
> +endef
> +
> +define HOST_RUST_INSTALL_CMDS
> +	(cd $(@D); $(HOST_MAKE_ENV) $(HOST_DIR)/bin/python2 x.py \
> +		dist $(HOST_RUST_BUILD_OPTS))
> +	(cd $(@D); $(HOST_MAKE_ENV) $(HOST_DIR)/bin/python2 x.py \
> +		install $(HOST_RUST_BUILD_OPTS))
> +endef
> +
> +$(eval $(host-generic-package))
> diff --git a/package/rustc/Config.in.host b/package/rustc/Config.in.host
> index e42187d477..3a03140bcd 100644
> --- a/package/rustc/Config.in.host
> +++ b/package/rustc/Config.in.host
> @@ -26,6 +26,21 @@ choice
>  	help
>  	  Select a Rust compiler
>  
> +config BR2_PACKAGE_HOST_RUST
> +	bool "host rust"
> +	depends on BR2_HOST_GCC_AT_LEAST_4_7 # required by LLVM
> +	# triggers ICE on trunc_int_for_mode, at explow.c:56
> +	depends on BR2_TOOLCHAIN_GCC_AT_LEAST_5 || !BR2_aarch64
> +	select BR2_PACKAGE_HAS_HOST_RUSTC
> +	help
> +	  This package will build the compiler for the host as well as
> +	  two flavors of the standard library: one for the host, another
> +	  for the target. Both are installed in the host directory.
> +
> +comment "host-rust needs a toolchain w/ gcc >= 5"
> +	depends on BR2_aarch64
> +	depends on !BR2_TOOLCHAIN_GCC_AT_LEAST_5

What about the gcc 4.7 dependency? It would also need a comment.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v9 5/8] libssh2: add host variant

[Thomas Petazzoni]

Hello,

On Thu, 28 Dec 2017 16:51:43 +0100, Eric Le Bihan wrote:
> Allow build of host variant of libssh2, which depends on host-openssl.
> 
> Signed-off-by: Eric Le Bihan <eric.le.bihan.dev@free.fr>

Reviewed-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

Long-term, it would be nice to have Config.in options for host
packages, so that we don't forcefully build openssl support in
host-libssh2, but since you're the only user for now, that's fine.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v9 6/8] libhttpparser: add host variant

[Thomas Petazzoni]

Hello,

On Thu, 28 Dec 2017 16:51:44 +0100, Eric Le Bihan wrote:
> Allow build of host variant of libhttpparser.
> 
> Signed-off-by: Eric Le Bihan <eric.le.bihan.dev@free.fr>

Reviewed-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v9 7/8] libcurl: add host variant

[Thomas Petazzoni]

Hello,

On Thu, 28 Dec 2017 16:51:45 +0100, Eric Le Bihan wrote:
> Signed-off-by: Eric Le Bihan <eric.le.bihan.dev@free.fr>

Reviewed-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

Same as host-libssh2, it's a bit annoying to force the dependency on
host-openssl, but we don't have much choice today, and you are anyway
the only user of host-libcurl at this point.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v9 8/8] cargo: new package

[Thomas Petazzoni]

Hello,

On Thu, 28 Dec 2017 16:51:46 +0100, Eric Le Bihan wrote:
> This new package provides Cargo, the Rust official package manager.
> Cargo is written in Rust and uses Cargo as its build system. It also
> depends on other Rust packages.
> 
> Normally, a previously installed version of Cargo would be used to:
> 
>  1. Fetch the dependencies.
>  2. Build the new version of Cargo, using the available Rust compiler.
> 
> But the fetching step prevents offline builds. So instead two features
> of Cargo are levelled: vendoring [1] and local registry.

Did you want to say "leveraged" instead of "levelled" ?

> diff --git a/package/cargo/Config.in.host b/package/cargo/Config.in.host
> new file mode 100644
> index 0000000000..0f1ca305c6
> --- /dev/null
> +++ b/package/cargo/Config.in.host
> @@ -0,0 +1,8 @@
> +config BR2_PACKAGE_HOST_CARGO
> +	bool "host cargo"
> +	depends on BR2_PACKAGE_HAS_HOST_RUSTC
> +	help
> +	  Cargo is the package manager for the Rust programming
> +	  language.
> +
> +	  https://crates.io/

How would "host cargo" be used in the context of Buildroot ? Would it
be used by packages that are written in Rust as part of their build
process ? What would such packages look like ?

> diff --git a/package/cargo/cargo.hash b/package/cargo/cargo.hash
> new file mode 100644
> index 0000000000..2b2ae43f6b
> --- /dev/null
> +++ b/package/cargo/cargo.hash
> @@ -0,0 +1,4 @@
> +# Locally generated
> +sha256 f4bbe2a8719dbb8da20842235093f7f70f034d01633189e83f75897d68cd274f  cargo-0.23.0.tar.gz
> +sha512 9060ec6e67b54f7fad7da8dd8450dd051d62b3f8ed4606196fc238a98beba1c3b43087c787f35d012d9b641c8572e70f50b95b0e01fdd75ed82932b6e6efbbf0  cargo-0.23.0-vendor.tar.xz
> +sha256 dc7240d60a869fa24a68c8734fb7c810c27cca0a6dad52df6279865e4e8e7fae  rust-installer-4f994850808a572e2cc8d43f968893c8e942e9bf.tar.gz

Why are you using sha256 sometimes, and sha512 sometimes? Seems like
for the cargo-0.23.0-vendor tarball, the hash is not exactly locally
generated, but really comes from upstream.

> +HOST_CARGO_DEPENDENCIES = \
> +	host-cmake \

Do you need host-cmake absolutely here, or can you use
$(BR2_CMAKE_HOST_DEPENDENCY) instead ?

> +	host-pkgconf \
> +	host-openssl \
> +	host-libhttpparser \
> +	host-libssh2 \
> +	host-libcurl \
> +	host-rustc \
> +	host-cargo-bin
> +
> +HOST_CARGO_SNAP_BIN = $(HOST_CARGO_BIN_DIR)/cargo/bin/cargo
> +HOST_CARGO_HOME = $(HOST_DIR)/share/cargo
> +
> +define HOST_CARGO_EXTRACT_DEPS
> +	@mkdir -p $(@D)/vendor
> +	$(call suitable-extractor,$(CARGO_DEPS_SOURCE)) \
> +		$(DL_DIR)/$(CARGO_DEPS_SOURCE) | \
> +		$(TAR) --strip-components=1 -C $(@D)/vendor $(TAR_OPTIONS) -
> +endef
> +
> +HOST_CARGO_POST_EXTRACT_HOOKS += HOST_CARGO_EXTRACT_DEPS
> +
> +define HOST_CARGO_EXTRACT_INSTALLER
> +	@mkdir -p $(@D)/src/rust-installer
> +	$(call suitable-extractor,$(CARGO_INSTALLER_SOURCE)) \
> +		$(DL_DIR)/$(CARGO_INSTALLER_SOURCE) | \
> +		$(TAR) --strip-components=1 -C $(@D)/src/rust-installer $(TAR_OPTIONS) -
> +endef
> +
> +HOST_CARGO_POST_EXTRACT_HOOKS += HOST_CARGO_EXTRACT_INSTALLER
> +
> +define HOST_CARGO_SETUP_DEPS
> +	mkdir -p $(@D)/.cargo
> +	( \
> +		echo "[source.crates-io]"; \
> +		echo "registry = 'https://github.com/rust-lang/crates.io-index'"; \
> +		echo "replace-with = 'vendored-sources'"; \
> +		echo "[source.vendored-sources]"; \
> +		echo "directory = '$(@D)/vendor'"; \
> +	) > $(@D)/.cargo/config
> +endef
> +
> +HOST_CARGO_PRE_CONFIGURE_HOOKS += HOST_CARGO_SETUP_DEPS
> +
> +HOST_CARGO_SNAP_OPTS = --release
> +HOST_CARGO_SNAP_OPTS += $(if $(VERBOSE),--verbose)

A single assignment would be prettier:

HOST_CARGO_SNAP_OPTS = \
	--release \
	$(if $(VERBOSE),--verbose)

> +HOST_CARGO_ENV = \
> +	RUSTFLAGS="-Clink-arg=-Wl,-rpath,$(HOST_DIR)/lib" \
> +	CARGO_HOME=$(HOST_DIR)/share/cargo

Why don't you use $(HOST_CARGO_HOME) here ?

> +define HOST_CARGO_BUILD_CMDS
> +	(cd $(@D); $(HOST_MAKE_ENV) $(HOST_CARGO_ENV) $(HOST_CARGO_SNAP_BIN) \
> +		build $(HOST_CARGO_SNAP_OPTS))
> +endef
> +
> +define HOST_CARGO_INSTALL_CMDS
> +	$(INSTALL) -d -m 0755 $(HOST_DIR)/bin

This is not needed, just use the -D option in the command command
installing the cargo binary.

> +	$(INSTALL) -m 0755 $(@D)/target/release/cargo $(HOST_DIR)/bin/cargo
> +endef
> +
> +define HOST_CARGO_INSTALL_CONF_FILE
> +	$(INSTALL) -D package/cargo/config.in \
> +		$(HOST_DIR)/share/cargo/config
> +	$(SED) 's/@RUST_TARGET_NAME@/$(RUST_TARGET_NAME)/' \
> +		$(HOST_DIR)/share/cargo/config
> +	$(SED) 's/@CROSS_PREFIX@/$(notdir $(TARGET_CROSS))/' \
> +		$(HOST_DIR)/share/cargo/config

This should be moved inside HOST_CARGO_INSTALL_CMDS, there is no need
for a separate post-install hook.

Also, I'm wondering of your choice between generating config files from
the .mk file, or having a template that gets tweaked using SED. In this
cargo.mk, you are producing a 5 lines file in HOST_CARGO_SETUP_DEPS
where only a single value varies (and would need to be tweaked with
sed). But on the other hand, you create a two lines share/cargo/config
file from a template. Perhaps we should settle on one or the other
solution ?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v9 0/8] Add support for the Rust programming

[Thomas Petazzoni]

Hello,

On Thu, 28 Dec 2017 18:52:56 +0100, Eric Le Bihan wrote:

> Yes, this is correct. The introduction of a virtual package for the Rust
> compiler was suggested in the mail thread about version v4 of this
> series [1], as building the compiler from source takes time and using the
> available pre-built version would speed things up (as done for GCC).

OK, thanks for the additional explanation, makes sense.

I reviewed the different patches in your series, and I think it's
really close to a state where it can be merged. If you can take care of
the few comments/questions in the near future, it would definitely
help, as I would still have the context in mind.

Two other things that would be nice to do is:

 - Add two test cases in support/testing/. One that builds a system with
   rust-bin, builds a hello world program in Rust, and run it on the
   target. And another that does the same, but by building the rust
   compiler from source. Perhaps additional test cases can be added for
   Cargo, but I'm not sure how it's supposed to be used.

 - Add a short section in the documentation that explains how the Rust
   support in Buildroot is working. This would help people trying to do
   stuff with Rust to understand how things work.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v9 1/8] rustc: new virtual package

[Eric Le Bihan]

Hi!

On 17-12-28 17:27:49, Thomas Petazzoni wrote:
> Hello,
>
> On Thu, 28 Dec 2017 16:51:39 +0100, Eric Le Bihan wrote:
>
> > diff --git a/package/rustc/Config.in.host b/package/rustc/Config.in.host
> > new file mode 100644
> > index 0000000000..16c95fa0b4
> > --- /dev/null
> > +++ b/package/rustc/Config.in.host
> > @@ -0,0 +1,17 @@
> > +config BR2_PACKAGE_HOST_RUSTC_ARCH_SUPPORTS
> > +	bool
> > +	default y
> > +	depends on BR2_HOSTARCH = "x86_64" || BR2_HOSTARCH = "x86"
> > +	depends on BR2_i386 || BR2_x86_64 \
> > +		|| BR2_arm  || BR2_aarch64 \
> > +		|| BR2_powerpc  || BR2_powerpc64 \
> > +		|| BR2_mips || BR2_mipsel || BR2_mips64 || BR2_mips64el
>
> This would be more readable as:
>
> 	default y if BR2_aarch64
> 	default y if BR2_arm && !BR2_ARM_CPU_ARMV4 && !BR2_ARM_CPU_ARMV5
> 	default y if BR2_i386
> 	default y if ...
> 	default y if BR2_mips || BR2_mipsel
> 	default y if (BR2_mips64 || BR2_mips64el) && BR2_MIPS_NABI64
> 	default y if BR2_x86_64
>

OK.

> > +	depends on !BR2_ARM_CPU_ARMV4 && !BR2_ARM_CPU_ARMV5
> > +	depends on !BR2_MIPS_NABI32
> > +	depends on BR2_TOOLCHAIN_USES_GLIBC
>
> Since it's weird for a host package to have a target architecture
> dependency and target C library dependency, it would be good to have a
> comment that explains this. I assume it's because of the Rust standard
> library, so something like:
>
> 	# The pre-built Rust standard library is only available for the
> 	# following architectures/ABIs, and is built against glibc.

OK.

> > diff --git a/package/rustc/rustc.mk b/package/rustc/rustc.mk
> > new file mode 100644
> > index 0000000000..4163a44bc1
> > --- /dev/null
> > +++ b/package/rustc/rustc.mk
> > @@ -0,0 +1,31 @@
> > +################################################################################
> > +#
> > +# rustc
> > +#
> > +################################################################################
> > +
> > +RUST_ARCH := $(call qstrip,$(BR2_ARCH))
> > +
> > +ifeq ($(BR2_ARM_CPU_ARMV7A),y)
> > +RUST_ARCH := armv7
> > +endif
> > +
> > +ifeq ($(BR2_ARM_EABI),y)
> > +RUST_ABI := eabi
> > +else ifeq ($(BR2_ARM_EABIHF),y)
> > +RUST_ABI := eabihf
> > +else ifeq ($(BR2_mips64)$(BR2_mips64el),y)
> > +RUST_ABI := abi64
> > +endif
>
> Perhaps this could be defined in the Config.in file as well ?
>
> config BR2_PACKAGE_HOST_RUSTC_ARCH
> 	string
> 	default "armv7" 	if BR2_ARM_CPU_ARMV7A
> 	default BR2_ARCH 	if !BR2_ARM_CPU_ARMV7A
>
> config BR2_PACKAGE_HOST_RUSTC_ABI
> 	string
> 	default "eabi" 		if BR2_ARM_EABI
> 	default "eabihf" 	if BR2_ARM_EABIHF
> 	default "abi64" 	if BR2_MIPS_NABI64

OK.

> > +RUST_TARGET_NAME := $(RUST_ARCH)-unknown-linux-gnu$(RUST_ABI)
>
> Why are you using := in this file instead of = ?

A long time ago, I ran into an error about $(RUST_TARGET_NAME) not being
defined, hence the simply expanded variable. I'll check this again.

Regards,

--
ELB

[Buildroot] [PATCH v9 2/8] rust-bin: new package

[Eric Le Bihan]

Hi!

On 17-12-28 21:37:02, Thomas Petazzoni wrote:
> Hello,
>
> On Thu, 28 Dec 2017 16:51:40 +0100, Eric Le Bihan wrote:
> > This package provides a pre-built version of rustc, the compiler for the
> > Rust programming language, fetched from the upstream project.
> >
> > A pre-built version of the standard library for the host as well as one
> > for the chosen target are also fetched and installed.
>
> I was wondering if a separate package could be used to install the
> standard library, in order to avoid the extra downloads and custom
> extract step.
>
> One possibility would be to have a rust-std-bin package, which rust-bin
> depends on. The target version of rust-std-bin would install the target
> libraries, while the host version would install the host libraries. The
> weird thing is that the target libraries are installed in $(HOST_DIR),
> while one would expect them to be in $(STAGING_DIR). Is this something
> we can tell rustc about perhaps ?

The idea of having a separate package for the standard library was
previously mentionned. The pros for it were the following:

 - smaller/cleaner packages
 - as Rust programs can be built without the standard library (e.g. for
   bare metal and microcontrollers), allowing the user to select the
   standard library separately looks sensible.

However, as a virtual package is used for the host Rust compiler (with
host-rust and host-rust-bin as its providers), does adding a separate
package for the binary standard library implies adding a virtual package
for the standard library too?

We would have:

 - host-rustc: virtual package for Rust compiler
 - host-rust-std: virtual package for Rust standard library
 - host-rustc-bin: provider for host-rustc (binary)
 - host-rust-std-bin: provider for host-rust-std (binary)
 - host-rust: provider for host-rustc and host-rust-std (built from source)

Would not that lead to too many compiler/library combinations?

Besides, compiling a program without standard library relies on a
compiler directive in the source code, so having a selectable standard
library package does not look that necessary from a user point of view.

So, to me, there are two options:

a) the current one:

  - host-rustc: virtual package for the compiler
  - host-rust-bin: provides host-rustc and standard library (binary)
  - host-rust: provides host-rustc and standard library (built from source)

b) a potentially cleaner version:

  - host-rustc: virtual package for the compiler
  - host-rustc-bin: provides host-rustc (binary), formerly known as
    host-rust-bin (renamed for clarity)
  - host-rust-std-bin: provides standard library (binary), a new dependency
    for host-rustc-bin, not selectable at user level.
  - host-rust: provides host-rustc and standard library (built from source)

> Even if we keep the single package approach, having the target
> libraries in $(STAGING_DIR) would be more logical. But that's a soft
> requirement, so if it's too tedious to achieve with rustc, we can live
> with them being in HOST_DIR.

The standard library tarball provides *.rlib (static Rust libraries) and
*.so (dynamic Rust libraries) files. The path to these files can be
passed to rustc using the --sysroot command line option, which can be
passed to Cargo (Rust package manager) by using the environment variable
RUSTFLAGS or the "rustflags" parameter in $(HOST_DIR)/share/cargo/config.

So it should be possible to relocate files from
$(HOST_DIR)/lib/rustlib/$(RUST_TARGET_NAME) to
$(STAGING_DIR)/lib/rustlib/$(RUST_TARGET_NAME):

- host-rust(-std)-bin: straightforward (change command line option to
  installation script).
- host-rust: install everything to $(HOST_DIR), then move target libraries to
  $(STAGING_DIR)

By default, all Rust programs are statically linked with the standard
library. It is possible to link dynamically [1] and in that case the
*.so files should indeed be copied to $(TARGET_DIR). So maybe in the
future support for $(BR2_SHARED_LIBS) should be added to handle this.

IMHO, the standard library should be managed more like libgcc.so than
glibc/uclibc/musl. (Things would have been simpler if there has been an
easy way to build the cross-compiled standard library separately from
the compiler).

> > diff --git a/package/rust-bin/rust-bin.mk b/package/rust-bin/rust-bin.mk
> > new file mode 100644
> > index 0000000000..29f94b7f2d
> > --- /dev/null
> > +++ b/package/rust-bin/rust-bin.mk
> > @@ -0,0 +1,63 @@
> > +################################################################################
> > +#
> > +# rust-bin
> > +#
> > +################################################################################
> > +
> > +RUST_BIN_VERSION = 1.22.1
> > +RUST_BIN_SITE = https://static.rust-lang.org/dist
> > +RUST_BIN_LICENSE = Apache-2.0 or MIT
> > +RUST_BIN_LICENSE_FILES = LICENSE-APACHE LICENSE-MIT
> > +
> > +HOST_RUST_BIN_PROVIDES = host-rustc
> > +
> > +HOST_RUST_BIN_SOURCE = rustc-$(RUST_BIN_VERSION)-$(RUST_HOST_NAME).tar.xz
> > +HOST_RUST_BIN_LIBSTD_SOURCES = \
> > +	rust-std-$(RUST_BIN_VERSION)-$(RUST_HOST_NAME).tar.xz \
> > +	rust-std-$(RUST_BIN_VERSION)-$(RUST_TARGET_NAME).tar.xz
> > +
> > +HOST_RUST_BIN_EXTRA_DOWNLOADS = $(HOST_RUST_BIN_LIBSTD_SOURCES)
>
> Assuming we keep the single package approach, then why not use
> HOST_RUST_BIN_EXTRA_DOWNLOADS directly ? HOST_RUST_BIN_LIBSTD_SOURCES
> looks a bit useless.

OK.

> > +HOST_RUST_BIN_LIBSTD_HOST_PREFIX = rust-std-$(RUST_BIN_VERSION)-$(RUST_HOST_NAME)/rust-std-$(RUST_HOST_NAME)
> > +
> > +define HOST_RUST_BIN_LIBSTD_EXTRACT
> > +	mkdir -p $(@D)/std
> > +	for file in $(addprefix $(DL_DIR)/,$(HOST_RUST_BIN_LIBSTD_SOURCES)); do \
> > +		$(TAR) -C $(@D)/std -xJf $${file}; \
>
> This would benefit from using $(foreach ...) and should use
> suitable-extractor. Something like (untested):
>
> 	$(foreach f,$(HOST_RUST_BIN_EXTRA_DOWNLOADS), \
> 		$(call suitable-extractor,$(f)) $(DL_DIR)/$(f) | \
> 			$(TAR) -C $(@D)/std $(TAR_OPTIONS) -
> 	)

OK.

> > +	done
> > +	(\
> > +		cd $(@D)/rustc/lib/rustlib; \
> > +		ln -sf ../../../std/$(HOST_RUST_BIN_LIBSTD_HOST_PREFIX)/lib/rustlib/$(RUST_HOST_NAME) \
> > +	)
>
> There's no need for this command to run in a sub-shell, so:
>
> 	cd $(@D)/rustc/lib/rustlib; \
> 		ln -sf ../../../std/$(HOST_RUST_BIN_LIBSTD_HOST_PREFIX)/lib/rustlib/$(RUST_HOST_NAME)
>
> should be sufficient.

OK.

> > +endef
> > +
> > +HOST_RUST_BIN_POST_EXTRACT_HOOKS += HOST_RUST_BIN_LIBSTD_EXTRACT
> > +
> > +HOST_RUST_BIN_INSTALL_OPTS = \
> > +	--prefix=$(HOST_DIR) \
> > +	--disable-ldconfig
> > +
> > +ifeq ($(BR2_PACKAGE_HOST_RUST_BIN),y)
>
> Why do you have this conditional ? If the package isn't built, then
> those definitions aren't used anyway.
>
> The rest looks good to me. Thanks!

The idea is to define the installation commands only if the package is
selected at user level. This is the trick to use this package as a
bootstrap compiler to build Rust from source: in that case, this package
is added as a dependency and not selected at user level, so it is
downloaded and extracted but not installed in $(HOST_DIR). This prevents
polluting $(HOST_DIR) by installing rust-bin then rust on top of it.

[1] http://www.elebihan.com/posts/generating-dynamically-linked-programs-with-cargo.html

Regards,

--
ELB

[Buildroot] [PATCH v9 3/8] cargo-bin: new package

[Eric Le Bihan]

On 17-12-28 21:38:33, Thomas Petazzoni wrote:
> Hello,
>
> On Thu, 28 Dec 2017 16:51:41 +0100, Eric Le Bihan wrote:
> > This new package fetches a binary version of Cargo, suitable to
> > bootstrap the host variants of the Rust compiler and Cargo, the package
> > manager.
> >
> > Signed-off-by: Eric Le Bihan <eric.le.bihan.dev@free.fr>
>
> Looks good, just one comment below.
>
> > diff --git a/package/cargo-bin/cargo-bin.mk b/package/cargo-bin/cargo-bin.mk
> > new file mode 100644
> > index 0000000000..e0f6381070
> > --- /dev/null
> > +++ b/package/cargo-bin/cargo-bin.mk
> > @@ -0,0 +1,14 @@
> > +################################################################################
> > +#
> > +# cargo-bin
> > +#
> > +################################################################################
> > +
> > +CARGO_BIN_VERSION = 0.23.0
> > +CARGO_BIN_SITE = https://static.rust-lang.org/dist
> > +CARGO_BIN_SOURCE = cargo-$(CARGO_BIN_VERSION)-$(RUST_HOST_NAME).tar.xz
> > +CARGO_BIN_LICENSE = Apache-2.0 or MIT
> > +CARGO_BIN_LICENSE_FILES = LICENSE-APACHE LICENSE-MIT
> > +CARGO_BIN_STRIP_COMPONENTS = 1
>
> This is the default, so why do you need this ?

Leftover from previous patch version. I'll remove it.

Regards,

--
ELB

[Buildroot] [PATCH v9 4/8] rust: new package

[Eric Le Bihan]

Hi!

On 17-12-28 22:08:53, Thomas Petazzoni wrote:
> Hello,
>
> On Thu, 28 Dec 2017 16:51:42 +0100, Eric Le Bihan wrote:
> > This new package provides rustc, the compiler for the Rust programming
> > language, built from source.
> >
> > Currently, only the host variant is built.
> >
> > The Rust compiler uses LLVM as its backend: a copy of LLVM source code
> > is provided and CMake is used to build it. It is possible to use a
> > pre-built external copy. When LLVM/clang will be available in Buildroot,
> > it would be possible to benefit from this feature and thus decrease
> > build time.
> >
> > LLVM is configured to generate code for x86, ARM, PowerPC and MIPS
> > architectures.
> >
> > The Rust compiler uses Cargo as its build system and is written in Rust.
> > Therefore this package depends on cargo-bin and rust-bin.
> >
> > The internal build process is as follows:
> >
> >  1. rustc-stage0, provided by rust-bin, is used to build rustc-stage1.
> >  2. rust-stage1 builds the final Rust compiler (rust-stage2)
> >     and the standard library for the host architecture.
>
> You're sometimes using rustc-stageX, sometimes just rust-stageX. Is
> this intentional ?

No, just a typo. I'll rephrase using "stageX compiler".

> > diff --git a/package/rust/rust.mk b/package/rust/rust.mk
> > new file mode 100644
> > index 0000000000..8dbec99bc5
> > --- /dev/null
> > +++ b/package/rust/rust.mk
> > @@ -0,0 +1,82 @@
> > +################################################################################
> > +#
> > +# rust
> > +#
> > +################################################################################
> > +
> > +RUST_VERSION = 1.22.1
> > +RUST_SOURCE = rustc-$(RUST_VERSION)-src.tar.xz
> > +RUST_SITE = https://static.rust-lang.org/dist
> > +RUST_LICENSE = Apache-2.0 or MIT
> > +RUST_LICENSE_FILES = LICENSE-APACHE LICENSE-MIT
> > +
> > +HOST_RUST_PROVIDES = host-rustc
> > +
> > +HOST_RUST_DEPENDENCIES = \
> > +	toolchain \
>
> It is somewhat unusual to have a host package depend on the target
> toolchain. I'm not sure we have any other host package in the tree
> doing this. I do understand why you're doing this, I'm just a little
> bit concerned that this special situation may be forgotten in the
> future. But there isn't much other solution, so it's probably good as
> it is.
>
> > +	host-rust-bin \
> > +	host-cargo-bin \
> > +	host-python \
>
> So you absolutely need python2 ? The build system is not python3
> compliant ?

The README explicitly states "`python` 2.7 (but not 3.x)", though it may
not need much work to port to Python3. Should I try to patch this?

> > +	host-cmake
>
> Could you use $(BR2_CMAKE_HOST_DEPENDENCY) instead ? This way, it would
> only build host-cmake if CMake is not already present on the system.
> This would save a bit of build time.

OK.

> > +ifeq ($(BR2_PACKAGE_JEMALLOC),y)
> > +HOST_RUST_DEPENDENCIES += jemalloc
> > +HOST_RUST_JEMALLOC_ENABLED = true
> > +HOST_RUST_JEMALLOC_CONF = 'jemalloc = "$(STAGING_DIR)/usr/lib/libjemalloc_pic.a"'
> > +else
> > +HOST_RUST_JEMALLOC_ENABLED = false
> > +endif
> > +
> > +HOST_RUST_VERBOSITY = $(if $(VERBOSE),2,0)
> > +
> > +# Some vendored crates contain Cargo.toml.orig files. The associated
>
> vendored -> vendor ?

Of course!

> > +# .cargo-checksum.json file will contain a checksum for Cargo.toml.orig but
> > +# support/scripts/apply-patches.sh will delete them. This will cause the build
> > +# to fail, as Cargo will not be able to find the file and verify the checksum.
> > +# So, remove all Cargo.toml.orig entries from the affected .cargo-checksum.json
> > +# files
> > +define HOST_RUST_EXCLUDE_ORIG_FILES
> > +	for file in $$(find $(@D) -name '*.orig'); do \
> > +		crate=$$(dirname $${file}); \
> > +		fn=$${crate}/.cargo-checksum.json; \
> > +		sed -i -e 's/"Cargo.toml.orig":"[a-z0-9]\+",//g' $${fn}; \
> > +	done
> > +endef
> > +
> > +HOST_RUST_POST_EXTRACT_HOOKS += HOST_RUST_EXCLUDE_ORIG_FILES
> > +
> > +define HOST_RUST_CONFIGURE_CMDS
> > +	( \
> > +		echo '[build]'; \
> > +		echo 'target = ["$(RUST_TARGET_NAME)"]'; \
> > +		echo 'cargo = "$(HOST_CARGO_BIN_DIR)/cargo/bin/cargo"'; \
> > +		echo 'rustc = "$(HOST_RUST_BIN_DIR)/rustc/bin/rustc"'; \
> > +		echo 'python = "$(HOST_DIR)/bin/python2"'; \
> > +		echo 'submodules = false'; \
> > +		echo 'vendor = true'; \
> > +		echo 'compiler-docs = false'; \
> > +		echo 'docs = false'; \
> > +		echo 'verbose = $(HOST_RUST_VERBOSITY)'; \
> > +		echo '[install]'; \
> > +		echo 'prefix = "$(HOST_DIR)"'; \
> > +		echo '[rust]'; \
> > +		echo 'use-jemalloc = $(HOST_RUST_JEMALLOC_ENABLED)'; \
> > +		echo '[target.$(RUST_TARGET_NAME)]'; \
> > +		echo 'cc = "$(TARGET_CROSS)gcc"'; \
> > +		echo $(HOST_RUST_JEMALLOC_CONF); \
> > +	) > $(@D)/config.toml
> > +endef
> > +
> > +define HOST_RUST_BUILD_CMDS
> > +	(cd $(@D); $(HOST_MAKE_ENV) $(HOST_DIR)/bin/python2 x.py \
> > +		build $(HOST_RUST_BUILD_OPTS))
> > +endef
> > +
> > +define HOST_RUST_INSTALL_CMDS
> > +	(cd $(@D); $(HOST_MAKE_ENV) $(HOST_DIR)/bin/python2 x.py \
> > +		dist $(HOST_RUST_BUILD_OPTS))
> > +	(cd $(@D); $(HOST_MAKE_ENV) $(HOST_DIR)/bin/python2 x.py \
> > +		install $(HOST_RUST_BUILD_OPTS))
> > +endef
> > +
> > +$(eval $(host-generic-package))
> > diff --git a/package/rustc/Config.in.host b/package/rustc/Config.in.host
> > index e42187d477..3a03140bcd 100644
> > --- a/package/rustc/Config.in.host
> > +++ b/package/rustc/Config.in.host
> > @@ -26,6 +26,21 @@ choice
> >  	help
> >  	  Select a Rust compiler
> >
> > +config BR2_PACKAGE_HOST_RUST
> > +	bool "host rust"
> > +	depends on BR2_HOST_GCC_AT_LEAST_4_7 # required by LLVM
> > +	# triggers ICE on trunc_int_for_mode, at explow.c:56
> > +	depends on BR2_TOOLCHAIN_GCC_AT_LEAST_5 || !BR2_aarch64
> > +	select BR2_PACKAGE_HAS_HOST_RUSTC
> > +	help
> > +	  This package will build the compiler for the host as well as
> > +	  two flavors of the standard library: one for the host, another
> > +	  for the target. Both are installed in the host directory.
> > +
> > +comment "host-rust needs a toolchain w/ gcc >= 5"
> > +	depends on BR2_aarch64
> > +	depends on !BR2_TOOLCHAIN_GCC_AT_LEAST_5
>
> What about the gcc 4.7 dependency? It would also need a comment.

OK.

Regards,

--
ELB

[Buildroot] [PATCH v9 8/8] cargo: new package

[Eric Le Bihan]

Hi!

On 17-12-28 22:32:38, Thomas Petazzoni wrote:
> Hello,
>
> On Thu, 28 Dec 2017 16:51:46 +0100, Eric Le Bihan wrote:
> > This new package provides Cargo, the Rust official package manager.
> > Cargo is written in Rust and uses Cargo as its build system. It also
> > depends on other Rust packages.
> >
> > Normally, a previously installed version of Cargo would be used to:
> >
> >  1. Fetch the dependencies.
> >  2. Build the new version of Cargo, using the available Rust compiler.
> >
> > But the fetching step prevents offline builds. So instead two features
> > of Cargo are levelled: vendoring [1] and local registry.
>
> Did you want to say "leveraged" instead of "levelled" ?

That's the word I was looking for!

> > diff --git a/package/cargo/Config.in.host b/package/cargo/Config.in.host
> > new file mode 100644
> > index 0000000000..0f1ca305c6
> > --- /dev/null
> > +++ b/package/cargo/Config.in.host
> > @@ -0,0 +1,8 @@
> > +config BR2_PACKAGE_HOST_CARGO
> > +	bool "host cargo"
> > +	depends on BR2_PACKAGE_HAS_HOST_RUSTC
> > +	help
> > +	  Cargo is the package manager for the Rust programming
> > +	  language.
> > +
> > +	  https://crates.io/
>
> How would "host cargo" be used in the context of Buildroot ? Would it
> be used by packages that are written in Rust as part of their build
> process ? What would such packages look like ?

I will add a new file named docs/manual/adding-packages-rust.txt to
document Cargo usage, with an example for adding a package for a Rust
program.

> > diff --git a/package/cargo/cargo.hash b/package/cargo/cargo.hash
> > new file mode 100644
> > index 0000000000..2b2ae43f6b
> > --- /dev/null
> > +++ b/package/cargo/cargo.hash
> > @@ -0,0 +1,4 @@
> > +# Locally generated
> > +sha256 f4bbe2a8719dbb8da20842235093f7f70f034d01633189e83f75897d68cd274f  cargo-0.23.0.tar.gz
> > +sha512 9060ec6e67b54f7fad7da8dd8450dd051d62b3f8ed4606196fc238a98beba1c3b43087c787f35d012d9b641c8572e70f50b95b0e01fdd75ed82932b6e6efbbf0  cargo-0.23.0-vendor.tar.xz
> > +sha256 dc7240d60a869fa24a68c8734fb7c810c27cca0a6dad52df6279865e4e8e7fae  rust-installer-4f994850808a572e2cc8d43f968893c8e942e9bf.tar.gz
>
> Why are you using sha256 sometimes, and sha512 sometimes? Seems like
> for the cargo-0.23.0-vendor tarball, the hash is not exactly locally
> generated, but really comes from upstream.

Yes, the SHA512 comes from upstream, so I'll add its source URL and keep
SHA256 for the locally generated ones.

> > +HOST_CARGO_DEPENDENCIES = \
> > +	host-cmake \
>
> Do you need host-cmake absolutely here, or can you use
> $(BR2_CMAKE_HOST_DEPENDENCY) instead ?

$(BR2_CMAKE_HOST_DEPENDENCY) should do the trick.

> > +	host-pkgconf \
> > +	host-openssl \
> > +	host-libhttpparser \
> > +	host-libssh2 \
> > +	host-libcurl \
> > +	host-rustc \
> > +	host-cargo-bin
> > +
> > +HOST_CARGO_SNAP_BIN = $(HOST_CARGO_BIN_DIR)/cargo/bin/cargo
> > +HOST_CARGO_HOME = $(HOST_DIR)/share/cargo
> > +
> > +define HOST_CARGO_EXTRACT_DEPS
> > +	@mkdir -p $(@D)/vendor
> > +	$(call suitable-extractor,$(CARGO_DEPS_SOURCE)) \
> > +		$(DL_DIR)/$(CARGO_DEPS_SOURCE) | \
> > +		$(TAR) --strip-components=1 -C $(@D)/vendor $(TAR_OPTIONS) -
> > +endef
> > +
> > +HOST_CARGO_POST_EXTRACT_HOOKS += HOST_CARGO_EXTRACT_DEPS
> > +
> > +define HOST_CARGO_EXTRACT_INSTALLER
> > +	@mkdir -p $(@D)/src/rust-installer
> > +	$(call suitable-extractor,$(CARGO_INSTALLER_SOURCE)) \
> > +		$(DL_DIR)/$(CARGO_INSTALLER_SOURCE) | \
> > +		$(TAR) --strip-components=1 -C $(@D)/src/rust-installer $(TAR_OPTIONS) -
> > +endef
> > +
> > +HOST_CARGO_POST_EXTRACT_HOOKS += HOST_CARGO_EXTRACT_INSTALLER
> > +
> > +define HOST_CARGO_SETUP_DEPS
> > +	mkdir -p $(@D)/.cargo
> > +	( \
> > +		echo "[source.crates-io]"; \
> > +		echo "registry = 'https://github.com/rust-lang/crates.io-index'"; \
> > +		echo "replace-with = 'vendored-sources'"; \
> > +		echo "[source.vendored-sources]"; \
> > +		echo "directory = '$(@D)/vendor'"; \
> > +	) > $(@D)/.cargo/config
> > +endef
> > +
> > +HOST_CARGO_PRE_CONFIGURE_HOOKS += HOST_CARGO_SETUP_DEPS
> > +
> > +HOST_CARGO_SNAP_OPTS = --release
> > +HOST_CARGO_SNAP_OPTS += $(if $(VERBOSE),--verbose)
>
> A single assignment would be prettier:
>
> HOST_CARGO_SNAP_OPTS = \
> 	--release \
> 	$(if $(VERBOSE),--verbose)

OK.

> > +HOST_CARGO_ENV = \
> > +	RUSTFLAGS="-Clink-arg=-Wl,-rpath,$(HOST_DIR)/lib" \
> > +	CARGO_HOME=$(HOST_DIR)/share/cargo
>
> Why don't you use $(HOST_CARGO_HOME) here ?

Leftover for previous patch version.

> > +define HOST_CARGO_BUILD_CMDS
> > +	(cd $(@D); $(HOST_MAKE_ENV) $(HOST_CARGO_ENV) $(HOST_CARGO_SNAP_BIN) \
> > +		build $(HOST_CARGO_SNAP_OPTS))
> > +endef
> > +
> > +define HOST_CARGO_INSTALL_CMDS
> > +	$(INSTALL) -d -m 0755 $(HOST_DIR)/bin
>
> This is not needed, just use the -D option in the command command
> installing the cargo binary.

Of course!

> > +	$(INSTALL) -m 0755 $(@D)/target/release/cargo $(HOST_DIR)/bin/cargo
> > +endef
> > +
> > +define HOST_CARGO_INSTALL_CONF_FILE
> > +	$(INSTALL) -D package/cargo/config.in \
> > +		$(HOST_DIR)/share/cargo/config
> > +	$(SED) 's/@RUST_TARGET_NAME@/$(RUST_TARGET_NAME)/' \
> > +		$(HOST_DIR)/share/cargo/config
> > +	$(SED) 's/@CROSS_PREFIX@/$(notdir $(TARGET_CROSS))/' \
> > +		$(HOST_DIR)/share/cargo/config
>
> This should be moved inside HOST_CARGO_INSTALL_CMDS, there is no need
> for a separate post-install hook.
>
> Also, I'm wondering of your choice between generating config files from
> the .mk file, or having a template that gets tweaked using SED. In this
> cargo.mk, you are producing a 5 lines file in HOST_CARGO_SETUP_DEPS
> where only a single value varies (and would need to be tweaked with
> sed). But on the other hand, you create a two lines share/cargo/config
> file from a template. Perhaps we should settle on one or the other
> solution ?

For the Meson cross-compilation configuration file, I used a
template/sed combination as some values were used multiple times. So
generating it from the .mk file would have decreased its readability.
Besides it is a file to be installed in $(HOST_DIR), so it is
"important".

As $(HOST_DIR)/share/cargo/config belongs to the same category, I used
the same method, though the file is (currently?) less complex.

On the other side $(@D)/.cargo/config file is just some kind of build
artefact, with a limited lifetime. Hence the generation from the .mk
file.

Is there a general rule?

Regards,

--
ELB

[Buildroot] [PATCH v9 0/8] Add support for the Rust programming

[Eric Le Bihan]

Hi!

On 17-12-28 22:36:29, Thomas Petazzoni wrote:
> I reviewed the different patches in your series, and I think it's
> really close to a state where it can be merged. If you can take care of
> the few comments/questions in the near future, it would definitely
> help, as I would still have the context in mind.
>
> Two other things that would be nice to do is:
>
>  - Add two test cases in support/testing/. One that builds a system with
>    rust-bin, builds a hello world program in Rust, and run it on the
>    target. And another that does the same, but by building the rust
>    compiler from source. Perhaps additional test cases can be added for
>    Cargo, but I'm not sure how it's supposed to be used.
>
>  - Add a short section in the documentation that explains how the Rust
>    support in Buildroot is working. This would help people trying to do
>    stuff with Rust to understand how things work.

I'll add these. Thanks for the review!

Regards,

--
ELB

[Buildroot] [PATCH v9 0/8] Add support for the Rust programming

[Eric Le Bihan]

Hi!

On 2017-12-28 22:36, Thomas Petazzoni wrote:
>
> Two other things that would be nice to do is:
>
>  - Add two test cases in support/testing/. One that builds a system with
>    rust-bin, builds a hello world program in Rust, and run it on the
>    target. And another that does the same, but by building the rust
>    compiler from source. Perhaps additional test cases can be added for
>    Cargo, but I'm not sure how it's supposed to be used.

Is there a glibc-based toolchain suitable for arm vexpress available on
http://autobuild.buildroot.org/toolchains/tarballs/?

I see br-aarch64-glibc-2017.05-1078-g95b1dae.tar.bz2, but there is no
support for Aarch64 in testing and no artefacts are available.

Regards,

--
ELB

[Buildroot] [PATCH v9 0/8] Add support for the Rust programming

[Thomas Petazzoni]

Hello Eric,

On Thu, 18 Jan 2018 08:48:50 +0100, Eric Le Bihan wrote:

> On 2017-12-28 22:36, Thomas Petazzoni wrote:
> >
> > Two other things that would be nice to do is:
> >
> >  - Add two test cases in support/testing/. One that builds a system with
> >    rust-bin, builds a hello world program in Rust, and run it on the
> >    target. And another that does the same, but by building the rust
> >    compiler from source. Perhaps additional test cases can be added for
> >    Cargo, but I'm not sure how it's supposed to be used.  
> 
> Is there a glibc-based toolchain suitable for arm vexpress available on
> http://autobuild.buildroot.org/toolchains/tarballs/?

Use the Linaro ARM toolchain ?

> I see br-aarch64-glibc-2017.05-1078-g95b1dae.tar.bz2, but there is no
> support for Aarch64 in testing

I'm not sure what you mean by "no support for AArch64 in testing". The
testing infrastructure can test any arbitrary configuration, so there
is nothing like "support for architecture <foo> in testing". Perhaps
you are confused by BASIC_TOOLCHAIN_CONFIG being an ARM toolchain
configuration. But this BASIC_TOOLCHAIN_CONFIG is not mandatory, it is
just one toolchain configuration that you can use if your test doesn't
really care about the architecture/toolchain.

We have tests like support/testing/tests/fs/test_iso9660.py that don't
use BASIC_TOOLCHAIN_CONFIG.

> and no artefacts are available.

I'm not sure what "artefacts" you are looking for here.

If you need an ARM or AArcH64 glibc toolchain, just use the Linaro
external toolchains I'd say.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v9 0/8] Add support for the Rust programming

[Eric Le Bihan]

Hi!

On 2018-01-18 09:03, Thomas Petazzoni wrote:
> Hello Eric,
>
> On Thu, 18 Jan 2018 08:48:50 +0100, Eric Le Bihan wrote:
>
> > I see br-aarch64-glibc-2017.05-1078-g95b1dae.tar.bz2, but there is no
> > support for Aarch64 in testing
>
> I'm not sure what you mean by "no support for AArch64 in testing". The
> testing infrastructure can test any arbitrary configuration, so there
> is nothing like "support for architecture <foo> in testing". Perhaps
> you are confused by BASIC_TOOLCHAIN_CONFIG being an ARM toolchain
> configuration. But this BASIC_TOOLCHAIN_CONFIG is not mandatory, it is
> just one toolchain configuration that you can use if your test doesn't
> really care about the architecture/toolchain.
>
> We have tests like support/testing/tests/fs/test_iso9660.py that don't
> use BASIC_TOOLCHAIN_CONFIG.
>

Looking at the existing tests, I thought it was a requirement to use
toolchains hosted at http://autobuild.buildroot.org/.

> > and no artefacts are available.
>
> I'm not sure what "artefacts" you are looking for here.

I was referring to ARTIFACTS_URL, where ARM versatile/vexpress
kernels are available. As I thought I should have used
br-aarch64-glibc-2017.05-1078-g95b1dae.tar.bz2 to get a glibc-based
toolchain, adding Aarch64 support to the emulator seemed necessary.

I'll go with the Linaro toolchain, using ARM vexpress as target.

Thanks for the clarification.

Regards,

--
ELB

[Buildroot] [PATCH v9 0/8] Add support for the Rust programming

[Thomas Petazzoni]

Hello,

On Thu, 18 Jan 2018 23:41:58 +0100, Eric Le Bihan wrote:

> > We have tests like support/testing/tests/fs/test_iso9660.py that don't
> > use BASIC_TOOLCHAIN_CONFIG.
> 
> Looking at the existing tests, I thought it was a requirement to use
> toolchains hosted at http://autobuild.buildroot.org/.

Not at all. We do have some tests that don't use those toolchains. Look
at support/testing/tests/boot/test_atf.py, the tests use Linaro
toolchains.

> > I'm not sure what "artefacts" you are looking for here.  
> 
> I was referring to ARTIFACTS_URL, where ARM versatile/vexpress
> kernels are available. As I thought I should have used
> br-aarch64-glibc-2017.05-1078-g95b1dae.tar.bz2 to get a glibc-based
> toolchain, adding Aarch64 support to the emulator seemed necessary.
> 
> I'll go with the Linaro toolchain, using ARM vexpress as target.

If you want to do runtime test on an AArch64 platform without building
the kernel as part of your test configuration, then yes, we would have
to add an artifact on http://autobuild.buildroot.net/artefacts/. I can
definitely do that if needed.

I'm not super happy with this artefacts solution, but I wanted to avoid
putting all those binary files under version control inside the
Buildroot repository. If people have better proposals, I'm definitely
interested.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v9 0/8] Add support for the Rust programming

[Arnout Vandecappelle]

On 19-01-18 09:03, Thomas Petazzoni wrote:
> Hello,
> 
> On Thu, 18 Jan 2018 23:41:58 +0100, Eric Le Bihan wrote:
> 
>>> We have tests like support/testing/tests/fs/test_iso9660.py that don't
>>> use BASIC_TOOLCHAIN_CONFIG.
>>
>> Looking at the existing tests, I thought it was a requirement to use
>> toolchains hosted at http://autobuild.buildroot.org/.
> 
> Not at all. We do have some tests that don't use those toolchains. Look
> at support/testing/tests/boot/test_atf.py, the tests use Linaro
> toolchains.
> 
>>> I'm not sure what "artefacts" you are looking for here.  
>>
>> I was referring to ARTIFACTS_URL, where ARM versatile/vexpress
>> kernels are available. As I thought I should have used
>> br-aarch64-glibc-2017.05-1078-g95b1dae.tar.bz2 to get a glibc-based
>> toolchain, adding Aarch64 support to the emulator seemed necessary.
>>
>> I'll go with the Linaro toolchain, using ARM vexpress as target.
> 
> If you want to do runtime test on an AArch64 platform without building
> the kernel as part of your test configuration, then yes, we would have
> to add an artifact on http://autobuild.buildroot.net/artefacts/. I can
> definitely do that if needed.
> 
> I'm not super happy with this artefacts solution, but I wanted to avoid
> putting all those binary files under version control inside the
> Buildroot repository. If people have better proposals, I'm definitely
> interested.

 We could have the artefacts produced by gitlab-CI, and use the gitlab pipeline
feature to make sure the most recently built one is used. Well, that's the
theory, I haven't looked at the details :-)

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF