[Buildroot] [PATCH 1/1] rust: make sure the cxx compiler is also set for the target

[Buildroot] [PATCH 1/1] rust: make sure the cxx compiler is also set for the target

[charles.hardin at storagecraft.com]

From: Charles Hardin <charles.hardin@storagecraft.com>

Noticed that building the LLVM build -DCMAKE_CXX_COMPILER=c++
during the rust build which the C compiler was set to the
expected buildroot compiler. So, update the toml file to
also set the cxx variable.

Signed-off-by: Charles Hardin <charles.hardin@storagecraft.com>
---
 package/rust/rust.mk | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/package/rust/rust.mk b/package/rust/rust.mk
index ff2634b996..9ea4c79af5 100644
--- a/package/rust/rust.mk
+++ b/package/rust/rust.mk
@@ -62,7 +62,10 @@ define HOST_RUST_CONFIGURE_CMDS
 		echo '[rust]'; \
 		echo 'use-jemalloc = $(HOST_RUST_JEMALLOC_ENABLED)'; \
 		echo '[target.$(RUSTC_TARGET_NAME)]'; \
-		echo 'cc = "$(TARGET_CROSS)gcc"'; \
+		if [ "x$(RUSTC_HOST_NAME)" != "x$(RUSTC_TARGET_NAME)" ]; then \
+			echo 'cc = "$(TARGET_CROSS)gcc"'; \
+			echo 'cxx = "$(TARGET_CROSS)g++"'; \
+		fi; \
 		echo $(HOST_RUST_JEMALLOC_CONF); \
 	) > $(@D)/config.toml
 endef
-- 
2.14.3 (Apple Git-98)

[Buildroot] [PATCH 1/1] rust: make sure the cxx compiler is also set for the target

[Thomas Petazzoni]

Hello Charles,

Eric: could you review the patch below ?

On Wed, 28 Feb 2018 09:38:52 -0800, charles.hardin at storagecraft.com
wrote:
> From: Charles Hardin <charles.hardin@storagecraft.com>
> 
> Noticed that building the LLVM build -DCMAKE_CXX_COMPILER=c++
> during the rust build which the C compiler was set to the
> expected buildroot compiler.

I am sorry, but I don't understand this sentence. Could you explain in
more details what is going on ?

> diff --git a/package/rust/rust.mk b/package/rust/rust.mk
> index ff2634b996..9ea4c79af5 100644
> --- a/package/rust/rust.mk
> +++ b/package/rust/rust.mk
> @@ -62,7 +62,10 @@ define HOST_RUST_CONFIGURE_CMDS
>  		echo '[rust]'; \
>  		echo 'use-jemalloc = $(HOST_RUST_JEMALLOC_ENABLED)'; \
>  		echo '[target.$(RUSTC_TARGET_NAME)]'; \
> -		echo 'cc = "$(TARGET_CROSS)gcc"'; \
> +		if [ "x$(RUSTC_HOST_NAME)" != "x$(RUSTC_TARGET_NAME)" ]; then \
> +			echo 'cc = "$(TARGET_CROSS)gcc"'; \
> +			echo 'cxx = "$(TARGET_CROSS)g++"'; \
> +		fi; \

This is doing two things:

 1. Defining cxx, like the commit title says

 2. Only defining cc and cxx when RUSTC_HOST_NAME and
    RUSTC_TARGET_NAME, which is not explained anywhere.

Could you clarify what this patch does, and why it is needed, with a
better commit log ?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/1] rust: make sure the cxx compiler is also set for the target

[Charles Hardin]

We have a problem where we build on a HOST with x86_64 and RUST
doesn?t realize the difference between the host tool chain and the buildroot
toolchain.

In the old days - we had to patch rust to make a ?x86_64-buildroot-linux-gnu?
target that would be recognized, but the latest patches removed that around
rust 1.9 or so.

This was literally just looking at the logs and seeing the ?cc? was the buildroot
cc and the the cxx was ?c++?, so it is just what we hacked together to get
things to work.

But - definitely need to set ?cxx? in some appropriate fashion for rust to get
compiled correctly.

> On Apr 1, 2018, at 3:48 PM, Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote:
> 
> Hello Charles,
> 
> Eric: could you review the patch below ?
> 
> On Wed, 28 Feb 2018 09:38:52 -0800, charles.hardin at storagecraft.com
> wrote:
>> From: Charles Hardin <charles.hardin@storagecraft.com>
>> 
>> Noticed that building the LLVM build -DCMAKE_CXX_COMPILER=c++
>> during the rust build which the C compiler was set to the
>> expected buildroot compiler.
> 
> I am sorry, but I don't understand this sentence. Could you explain in
> more details what is going on ?
> 
>> diff --git a/package/rust/rust.mk b/package/rust/rust.mk
>> index ff2634b996..9ea4c79af5 100644
>> --- a/package/rust/rust.mk
>> +++ b/package/rust/rust.mk
>> @@ -62,7 +62,10 @@ define HOST_RUST_CONFIGURE_CMDS
>> 		echo '[rust]'; \
>> 		echo 'use-jemalloc = $(HOST_RUST_JEMALLOC_ENABLED)'; \
>> 		echo '[target.$(RUSTC_TARGET_NAME)]'; \
>> -		echo 'cc = "$(TARGET_CROSS)gcc"'; \
>> +		if [ "x$(RUSTC_HOST_NAME)" != "x$(RUSTC_TARGET_NAME)" ]; then \
>> +			echo 'cc = "$(TARGET_CROSS)gcc"'; \
>> +			echo 'cxx = "$(TARGET_CROSS)g++"'; \
>> +		fi; \
> 
> This is doing two things:
> 
> 1. Defining cxx, like the commit title says
> 
> 2. Only defining cc and cxx when RUSTC_HOST_NAME and
>    RUSTC_TARGET_NAME, which is not explained anywhere.
> 
> Could you clarify what this patch does, and why it is needed, with a
> better commit log ?
> 
> Thanks!
> 
> Thomas
> -- 
> Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
> Embedded Linux and Kernel engineering
> https://bootlin.com
> 

[Buildroot] [PATCH 1/1] rust: make sure the cxx compiler is also set for the target

[Thomas Petazzoni]

Hello,

Thanks for the quick feedback!

On Sun, 1 Apr 2018 22:52:42 +0000, Charles Hardin wrote:
> We have a problem where we build on a HOST with x86_64 and RUST
> doesn?t realize the difference between the host tool chain and the buildroot
> toolchain.
> 
> In the old days - we had to patch rust to make a ?x86_64-buildroot-linux-gnu?
> target that would be recognized, but the latest patches removed that around
> rust 1.9 or so.
> 
> This was literally just looking at the logs and seeing the ?cc? was the buildroot
> cc and the the cxx was ?c++?, so it is just what we hacked together to get
> things to work.
> 
> But - definitely need to set ?cxx? in some appropriate fashion for rust to get
> compiled correctly.

Setting cxx looks definitely OK in your patch. It's the
RUSTC_HOST_NAME != RUSTC_TARGET_NAME that looks wrong.

Could you provide some specific examples (with defconfigs to reproduce)
for the two problems:

 - Lack of cxx variable

 - Case of RUSTC_HOST_NAME == RUSTC_TARGET_NAME

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/1] rust: make sure the cxx compiler is also set for the target

[Charles Hardin]

Yeah - for ?upstream? drop the HOST - TARGET thing that is a problem for
us - I think?

The def config was basically this? but, it doesn?t show an error - i just read
the logs of the build.

BR2_x86_64=y
BR2_x86_core_avx2=y
BR2_TOOLCHAIN_BUILDROOT_CXX=y
BR2_TOOLCHAIN_BUILDROOT_GLIBC=y
BR2_TOOLCHAIN_HEADERS_AT_LEAST_4_9=y
BR2_GCC_VERSION_7_X=y
BR2_GCC_ENABLE_LTO=y
BR2_GCC_ENABLE_LIBMUDFLAP=y
BR2_ENABLE_LOCALE_PURGE=y
BR2_LINUX_KERNEL=n
BR2_KERNEL_HEADERS_4_9=y
BR2_INIT_NONE=y
BR2_PACKAGE_HOST_GO=y
BR2_PACKAGE_HOST_RUST=y
BR2_PACKAGE_HOST_RUSTC=y
BR2_PACKAGE_HOST_CARGO=y

To basically just build a toolchain.

On Apr 1, 2018, at 3:57 PM, Thomas Petazzoni <thomas.petazzoni at bootlin.com<mailto:thomas.petazzoni@bootlin.com>> wrote:

Hello,

Thanks for the quick feedback!

On Sun, 1 Apr 2018 22:52:42 +0000, Charles Hardin wrote:
We have a problem where we build on a HOST with x86_64 and RUST
doesn?t realize the difference between the host tool chain and the buildroot
toolchain.

In the old days - we had to patch rust to make a ?x86_64-buildroot-linux-gnu?
target that would be recognized, but the latest patches removed that around
rust 1.9 or so.

This was literally just looking at the logs and seeing the ?cc? was the buildroot
cc and the the cxx was ?c++?, so it is just what we hacked together to get
things to work.

But - definitely need to set ?cxx? in some appropriate fashion for rust to get
compiled correctly.

Setting cxx looks definitely OK in your patch. It's the
RUSTC_HOST_NAME != RUSTC_TARGET_NAME that looks wrong.

Could you provide some specific examples (with defconfigs to reproduce)
for the two problems:

- Lack of cxx variable

- Case of RUSTC_HOST_NAME == RUSTC_TARGET_NAME

Thanks!

Thomas
--
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20180401/807488c4/attachment.html>

[Buildroot] [PATCH 1/1] rust: make sure the cxx compiler is also set for the target

[Charles Hardin]

Oh, I think I might remember the problem on TARGET - HOST, notice the core_avx2
is our target and our buildslaves don?t have avx2 and rust has some stage where it
built a local tool and it didn?t run on the buildslave because it used the cross compiler
again because it doesn?t distinguish the buildroot targets from generic targets.

The cxx thing is all that most people will care about getting in upstream.

On Apr 1, 2018, at 4:00 PM, Charles Hardin <charles.hardin at storagecraft.com<mailto:charles.hardin@storagecraft.com>> wrote:

Yeah - for ?upstream? drop the HOST - TARGET thing that is a problem for
us - I think?

The def config was basically this? but, it doesn?t show an error - i just read
the logs of the build.

BR2_x86_64=y
BR2_x86_core_avx2=y
BR2_TOOLCHAIN_BUILDROOT_CXX=y
BR2_TOOLCHAIN_BUILDROOT_GLIBC=y
BR2_TOOLCHAIN_HEADERS_AT_LEAST_4_9=y
BR2_GCC_VERSION_7_X=y
BR2_GCC_ENABLE_LTO=y
BR2_GCC_ENABLE_LIBMUDFLAP=y
BR2_ENABLE_LOCALE_PURGE=y
BR2_LINUX_KERNEL=n
BR2_KERNEL_HEADERS_4_9=y
BR2_INIT_NONE=y
BR2_PACKAGE_HOST_GO=y
BR2_PACKAGE_HOST_RUST=y
BR2_PACKAGE_HOST_RUSTC=y
BR2_PACKAGE_HOST_CARGO=y

To basically just build a toolchain.

On Apr 1, 2018, at 3:57 PM, Thomas Petazzoni <thomas.petazzoni at bootlin.com<mailto:thomas.petazzoni@bootlin.com>> wrote:

Hello,

Thanks for the quick feedback!

On Sun, 1 Apr 2018 22:52:42 +0000, Charles Hardin wrote:
We have a problem where we build on a HOST with x86_64 and RUST
doesn?t realize the difference between the host tool chain and the buildroot
toolchain.

In the old days - we had to patch rust to make a ?x86_64-buildroot-linux-gnu?
target that would be recognized, but the latest patches removed that around
rust 1.9 or so.

This was literally just looking at the logs and seeing the ?cc? was the buildroot
cc and the the cxx was ?c++?, so it is just what we hacked together to get
things to work.

But - definitely need to set ?cxx? in some appropriate fashion for rust to get
compiled correctly.

Setting cxx looks definitely OK in your patch. It's the
RUSTC_HOST_NAME != RUSTC_TARGET_NAME that looks wrong.

Could you provide some specific examples (with defconfigs to reproduce)
for the two problems:

- Lack of cxx variable

- Case of RUSTC_HOST_NAME == RUSTC_TARGET_NAME

Thanks!

Thomas
--
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com<https://bootlin.com/>



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20180401/f1e22795/attachment.html>

[Buildroot] [PATCH 1/1] rust: make sure the cxx compiler is also set for the target

[Thomas Petazzoni]

Hello,

[Side note: it is customary on open-source mailing list to avoid
 top-posting, and instead reply below the text you are replying to.
 Thanks!]

On Sun, 1 Apr 2018 23:05:11 +0000, Charles Hardin wrote:

> Oh, I think I might remember the problem on TARGET - HOST, notice the core_avx2
> is our target and our buildslaves don?t have avx2 and rust has some stage where it
> built a local tool and it didn?t run on the buildslave because it used the cross compiler

Well, that's a severe bug. If Rust uses the cross-compiler to build a
program that it then runs on the host, this is not going to work on a
number of situations. Your defconfig uses glibc for the target, and
your host most likely uses glibc as well, so there are some chances
that a tool built with the cross-compiler will also run on the host.
But if the CPU optimization options are different (like you had with
AVX2) or the C library is different than glibc, then the tool built
with the cross-compiler will definitely not run on the host.

However, we do have a x86-64 uClibc configuration in our autobuilders,
which is built on a x86-64 machine, and I don't think we have seen any
issue. So, I'm a bit puzzled.

Which specific internal tool was causing the problem ?

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/1] rust: make sure the cxx compiler is also set for the target

[Charles Hardin]

so? hopefully the following makes sense...

On Apr 1, 2018, at 11:38 PM, Thomas Petazzoni <thomas.petazzoni at bootlin.com<mailto:thomas.petazzoni@bootlin.com>> wrote:

Hello,

[Side note: it is customary on open-source mailing list to avoid
top-posting, and instead reply below the text you are replying to.
Thanks!]

On Sun, 1 Apr 2018 23:05:11 +0000, Charles Hardin wrote:

Oh, I think I might remember the problem on TARGET - HOST, notice the core_avx2
is our target and our buildslaves don?t have avx2 and rust has some stage where it
built a local tool and it didn?t run on the buildslave because it used the cross compiler

Well, that's a severe bug. If Rust uses the cross-compiler to build a
program that it then runs on the host, this is not going to work on a
number of situations. Your defconfig uses glibc for the target, and
your host most likely uses glibc as well, so there are some chances
that a tool built with the cross-compiler will also run on the host.
But if the CPU optimization options are different (like you had with
AVX2) or the C library is different than glibc, then the tool built
with the cross-compiler will definitely not run on the host.

However, we do have a x86-64 uClibc configuration in our autobuilders,
which is built on a x86-64 machine, and I don't think we have seen any
issue. So, I'm a bit puzzled.


When you fix the CXX linking you will see this in the stage0 - notice the target is unknown - not buildroot, and
so what happens is the tools use the ?same? compiler as the target for the host side, which means that I ended
up getting ?target? compiled with avx2 on a host that doesn?t support it

Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage0 test artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)

? snip snip ?

Building LLVM for x86_64-unknown-linux-gnu
running: "cmake" "/home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/build/host-rust-1.23.0/src/llvm" "-DLLVM_ENABLE_ASSERTIONS=OFF" "-DLLVM_TARGETS_TO_BUILD=X86;ARM;AArch64;Mips;PowerPC;SystemZ;JSBackend;MSP430;Sparc;NVPTX;Hexagon" "-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=" "-DLLVM_INCLUDE_EXAMPLES=OFF" "-DLLVM_INCLUDE_TESTS=OFF" "-DLLVM_INCLUDE_DOCS=OFF" "-DLLVM_ENABLE_ZLIB=OFF" "-DWITH_POLLY=OFF" "-DLLVM_ENABLE_TERMINFO=OFF" "-DLLVM_ENABLE_LIBEDIT=OFF" "-DLLVM_PARALLEL_COMPILE_JOBS=2" "-DLLVM_TARGET_ARCH=x86_64" "-DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu" "-DLLVM_LINK_LLVM_DYLIB=ON" "-DCMAKE_C_COMPILER=/home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-gcc" "-DCMAKE_CXX_COMPILER=/home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-g++" "-DCMAKE_C_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_CXX_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_AR=/home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-ar" "-DCMAKE_INSTALL_PREFIX=/home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/build/host-rust-1.23.0/build/x86_64-unknown-linux-gnu/llvm" "-DCMAKE_BUILD_TYPE=Release"
-- The C compiler identification is GNU 7.3.0
-- The CXX compiler identification is GNU 7.3.0
-- The ASM compiler identification is GNU
-- Found assembler: /home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-gcc
-- Check for working C compiler: /home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-gcc
-- Check for working C compiler: /home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-g++
-- Check for working CXX compiler: /home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Warning at CMakeLists.txt:140 (message):
  Job pooling is only available with Ninja generators.

leads to building this ?host? tool

[  2%] Built target LLVMTableGen

which cann?t run on that host because the compiler it used was the ?buildroot? compiler

[ 19%] Linking CXX static library ../../libLLVMDebugInfoPDB.a
[ 19%] Built target LLVMDebugInfoPDB
Makefile:151: recipe for target 'all' failed
make[3]: *** [all] Error 2
thread 'main' panicked at '
command did not execute successfully, got: exit code: 2

build script failed, must exit now', src/vendor/cmake/src/lib.rs:631<http://lib.rs:631>:4
note: Run with `RUST_BACKTRACE=1` for a backtrace.
finished in 267.033
failed to run: /home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/build/host-rust-1.23.0/build/bootstrap/debug/bootstrap build
Build completed unsuccessfully in 0:06:22
package/pkg-generic.mk:247: recipe for target '/home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/build/host-rust-1.23.0/.stamp_built' failed
make[2]: *** [/home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/build/host-rust-1.23.0/.stamp_built] Error 1
Makefile:79: recipe for target '_all' failed
make[1]: *** [_all] Error 2
make[1]: Leaving directory '/home/vagrant/oneblox-buildroot/repos/buildroot'
Makefile:93: recipe for target 'br-x86_64-core_avx2-glibc-tmp.tar.bz2' failed
make: *** [br-x86_64-core_avx2-glibc-tmp.tar.bz2] Error 2


Which specific internal tool was causing the problem ?

Thanks,

Thomas
--
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20180402/8acbdcd9/attachment.html>

[Buildroot] [PATCH 1/1] rust: make sure the cxx compiler is also set for the target

[Eric Le Bihan]

Hi!

On 2018-04-02 19:54, Charles Hardin wrote:
> so? hopefully the following makes sense...
>
> When you fix the CXX linking you will see this in the stage0 - notice the target is unknown - not buildroot, and
> so what happens is the tools use the ?same? compiler as the target for the host side, which means that I ended
> up getting ?target? compiled with avx2 on a host that doesn?t support it
>
> Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
> Building stage0 test artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
>
> ? snip snip ?
>
> Building LLVM for x86_64-unknown-linux-gnu
> running: "cmake" "/home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/build/host-rust-1.23.0/src/llvm" "-DLLVM_ENABLE_ASSERTIONS=OFF" "-DLLVM_TARGETS_TO_BUILD=X86;ARM;AArch64;Mips;PowerPC;SystemZ;JSBackend;MSP430;Sparc;NVPTX;Hexagon" "-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=" "-DLLVM_INCLUDE_EXAMPLES=OFF" "-DLLVM_INCLUDE_TESTS=OFF" "-DLLVM_INCLUDE_DOCS=OFF" "-DLLVM_ENABLE_ZLIB=OFF" "-DWITH_POLLY=OFF" "-DLLVM_ENABLE_TERMINFO=OFF" "-DLLVM_ENABLE_LIBEDIT=OFF" "-DLLVM_PARALLEL_COMPILE_JOBS=2" "-DLLVM_TARGET_ARCH=x86_64" "-DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu" "-DLLVM_LINK_LLVM_DYLIB=ON" "-DCMAKE_C_COMPILER=/home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-gcc" "-DCMAKE_CXX_COMPILER=/home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-g++" "-DCMAKE_C_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_CXX_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_AR=/home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-ar" "-DCMAKE_INSTALL_PREFIX=/home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/build/host-rust-1.23.0/build/x86_64-unknown-linux-gnu/llvm" "-DCMAKE_BUILD_TYPE=Release"
> -- The C compiler identification is GNU 7.3.0
> -- The CXX compiler identification is GNU 7.3.0
> -- The ASM compiler identification is GNU
> -- Found assembler: /home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-gcc
> -- Check for working C compiler: /home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-gcc
> -- Check for working C compiler: /home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-gcc -- works
> -- Detecting C compiler ABI info
> -- Detecting C compiler ABI info - done
> -- Detecting C compile features
> -- Detecting C compile features - done
> -- Check for working CXX compiler: /home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-g++
> -- Check for working CXX compiler: /home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-g++ -- works
> -- Detecting CXX compiler ABI info
> -- Detecting CXX compiler ABI info - done
> -- Detecting CXX compile features
> -- Detecting CXX compile features - done
> CMake Warning at CMakeLists.txt:140 (message):
>   Job pooling is only available with Ninja generators.
>
> leads to building this ?host? tool
>
> [  2%] Built target LLVMTableGen
>
> which cann?t run on that host because the compiler it used was the ?buildroot? compiler
>
> [ 19%] Linking CXX static library ../../libLLVMDebugInfoPDB.a
> [ 19%] Built target LLVMDebugInfoPDB
> Makefile:151: recipe for target 'all' failed
> make[3]: *** [all] Error 2
> thread 'main' panicked at '
> command did not execute successfully, got: exit code: 2
>
> build script failed, must exit now', src/vendor/cmake/src/lib.rs:631<http://lib.rs:631>:4
> note: Run with `RUST_BACKTRACE=1` for a backtrace.
> finished in 267.033
> failed to run: /home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/build/host-rust-1.23.0/build/bootstrap/debug/bootstrap build
> Build completed unsuccessfully in 0:06:22
> package/pkg-generic.mk:247: recipe for target '/home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/build/host-rust-1.23.0/.stamp_built' failed
> make[2]: *** [/home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/build/host-rust-1.23.0/.stamp_built] Error 1
> Makefile:79: recipe for target '_all' failed
> make[1]: *** [_all] Error 2
> make[1]: Leaving directory '/home/vagrant/oneblox-buildroot/repos/buildroot'
> Makefile:93: recipe for target 'br-x86_64-core_avx2-glibc-tmp.tar.bz2' failed
> make: *** [br-x86_64-core_avx2-glibc-tmp.tar.bz2] Error 2

Building Rust for x86_64 on a x86_64 is indeed tricky!

As seen in the build section of src/bootstrap/config.toml.example, Rust
manages "build", "host" and "target" parameters, which all default to
"x86_64-unknown-linux-gnu".

When building, the compiler is built for the host and the standard
library for the host and the target.

So when the target is the same as the host, only the compiler and the
standard library for the host should be built.

As the compiler is built on LLVM, coded in C++, a C++ compiler for the
host is indeed required. It happens that in the build of LLVM, a host
tool named FileCheck is built and run for test.

IIUC, the C compiler is only used for building the standard library
(this is what I've seen when cross-compiling for ARM).

So when the target is the same as the host, it might be sensible to
avoid defining c and cxx in the target configuration file, as suggested
by your patch.

I'll poke upstream for clarifications on this special case, though.

The first versions of the patch series did try to work with new targets
using triples with "buildroot" instead of "unknown", but defining new
targets instead of using the available ones turned out to be
difficult/cumbersome.

But maybe it would have avoided the issue raised here where the target
is the same as the host.

Out of curiosity, does using the pre-built toolchain may be of use in
your situation?

Regards,

--
ELB

[Buildroot] [PATCH 1/1] rust: make sure the cxx compiler is also set for the target

[Charles Hardin]

Eric,

Good to talk to you again?

> On Apr 3, 2018, at 3:01 PM, Eric Le Bihan <eric.le.bihan.dev@free.fr> wrote:
> 
> Hi!
> 
> On 2018-04-02 19:54, Charles Hardin wrote:
>> so? hopefully the following makes sense...
>> 
>> When you fix the CXX linking you will see this in the stage0 - notice the target is unknown - not buildroot, and
>> so what happens is the tools use the ?same? compiler as the target for the host side, which means that I ended
>> up getting ?target? compiled with avx2 on a host that doesn?t support it
>> 
>> Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
>> Building stage0 test artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
>> 
>> ? snip snip ?
>> 
>> Building LLVM for x86_64-unknown-linux-gnu
>> running: "cmake" "/home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/build/host-rust-1.23.0/src/llvm" "-DLLVM_ENABLE_ASSERTIONS=OFF" "-DLLVM_TARGETS_TO_BUILD=X86;ARM;AArch64;Mips;PowerPC;SystemZ;JSBackend;MSP430;Sparc;NVPTX;Hexagon" "-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=" "-DLLVM_INCLUDE_EXAMPLES=OFF" "-DLLVM_INCLUDE_TESTS=OFF" "-DLLVM_INCLUDE_DOCS=OFF" "-DLLVM_ENABLE_ZLIB=OFF" "-DWITH_POLLY=OFF" "-DLLVM_ENABLE_TERMINFO=OFF" "-DLLVM_ENABLE_LIBEDIT=OFF" "-DLLVM_PARALLEL_COMPILE_JOBS=2" "-DLLVM_TARGET_ARCH=x86_64" "-DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu" "-DLLVM_LINK_LLVM_DYLIB=ON" "-DCMAKE_C_COMPILER=/home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-gcc" "-DCMAKE_CXX_COMPILER=/home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-g++" "-DCMAKE_C_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_CXX_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_AR=/home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-ar" "-DCMAKE_INSTALL_PREFIX=/home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/build/host-rust-1.23.0/build/x86_64-unknown-linux-gnu/llvm" "-DCMAKE_BUILD_TYPE=Release"
>> -- The C compiler identification is GNU 7.3.0
>> -- The CXX compiler identification is GNU 7.3.0
>> -- The ASM compiler identification is GNU
>> -- Found assembler: /home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-gcc
>> -- Check for working C compiler: /home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-gcc
>> -- Check for working C compiler: /home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-gcc -- works
>> -- Detecting C compiler ABI info
>> -- Detecting C compiler ABI info - done
>> -- Detecting C compile features
>> -- Detecting C compile features - done
>> -- Check for working CXX compiler: /home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-g++
>> -- Check for working CXX compiler: /home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/host/bin/x86_64-buildroot-linux-gnu-g++ -- works
>> -- Detecting CXX compiler ABI info
>> -- Detecting CXX compiler ABI info - done
>> -- Detecting CXX compile features
>> -- Detecting CXX compile features - done
>> CMake Warning at CMakeLists.txt:140 (message):
>>  Job pooling is only available with Ninja generators.
>> 
>> leads to building this ?host? tool
>> 
>> [  2%] Built target LLVMTableGen
>> 
>> which cann?t run on that host because the compiler it used was the ?buildroot? compiler
>> 
>> [ 19%] Linking CXX static library ../../libLLVMDebugInfoPDB.a
>> [ 19%] Built target LLVMDebugInfoPDB
>> Makefile:151: recipe for target 'all' failed
>> make[3]: *** [all] Error 2
>> thread 'main' panicked at '
>> command did not execute successfully, got: exit code: 2
>> 
>> build script failed, must exit now', src/vendor/cmake/src/lib.rs:631<http://lib.rs:631>:4
>> note: Run with `RUST_BACKTRACE=1` for a backtrace.
>> finished in 267.033
>> failed to run: /home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/build/host-rust-1.23.0/build/bootstrap/debug/bootstrap build
>> Build completed unsuccessfully in 0:06:22
>> package/pkg-generic.mk:247: recipe for target '/home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/build/host-rust-1.23.0/.stamp_built' failed
>> make[2]: *** [/home/vagrant/oneblox-buildroot/toolchains/br-x86_64-core_avx2-glibc_toolchains.build/build/host-rust-1.23.0/.stamp_built] Error 1
>> Makefile:79: recipe for target '_all' failed
>> make[1]: *** [_all] Error 2
>> make[1]: Leaving directory '/home/vagrant/oneblox-buildroot/repos/buildroot'
>> Makefile:93: recipe for target 'br-x86_64-core_avx2-glibc-tmp.tar.bz2' failed
>> make: *** [br-x86_64-core_avx2-glibc-tmp.tar.bz2] Error 2
> 
> Building Rust for x86_64 on a x86_64 is indeed tricky!
> 
> As seen in the build section of src/bootstrap/config.toml.example, Rust
> manages "build", "host" and "target" parameters, which all default to
> "x86_64-unknown-linux-gnu".
> 
> When building, the compiler is built for the host and the standard
> library for the host and the target.
> 
> So when the target is the same as the host, only the compiler and the
> standard library for the host should be built.
> 
> As the compiler is built on LLVM, coded in C++, a C++ compiler for the
> host is indeed required. It happens that in the build of LLVM, a host
> tool named FileCheck is built and run for test.
> 
> IIUC, the C compiler is only used for building the standard library
> (this is what I've seen when cross-compiling for ARM).
> 
> So when the target is the same as the host, it might be sensible to
> avoid defining c and cxx in the target configuration file, as suggested
> by your patch.
> 
> I'll poke upstream for clarifications on this special case, though.
> 
> The first versions of the patch series did try to work with new targets
> using triples with "buildroot" instead of "unknown", but defining new
> targets instead of using the available ones turned out to be
> difficult/cumbersome.
> 
> But maybe it would have avoided the issue raised here where the target
> is the same as the host.
> 
> Out of curiosity, does using the pre-built toolchain may be of use in
> your situation?
> 

Yes and no, on some targets we use the pre-built and on some (ie. mips64) we
have to build for now at least.

So, this isn?t blocking us - I have it working ?good enough? but this was just sending
it upstream incase anyone else hit it. I am not inclined to know the fix or prefer one
way over another.

> Regards,
> 
> --
> ELB
> 

[Buildroot] [PATCH 1/1] rust: make sure the cxx compiler is also set for the target

[Thomas Petazzoni]

Hello Eric,

Sorry for the very slow feedback.

On Wed, 4 Apr 2018 00:01:32 +0200, Eric Le Bihan wrote:

> Building Rust for x86_64 on a x86_64 is indeed tricky!
> 
> As seen in the build section of src/bootstrap/config.toml.example, Rust
> manages "build", "host" and "target" parameters, which all default to
> "x86_64-unknown-linux-gnu".
> 
> When building, the compiler is built for the host and the standard
> library for the host and the target.
> 
> So when the target is the same as the host, only the compiler and the
> standard library for the host should be built.

When you are talking about "target" and "host", are you talking just
about the CPU architecture, or about the full target and host
definition, which includes the C library ?

If you build on x86-64, targeting a x86-64 platform, you have host
arch == target arch, but you still can't use the cross-compiler to
build programs and expect them to run on the host:

 - The C library might be different: uClibc or musl for the target,
   glibc available on the host

 - The kernel headers version may be different: the C library for the
   target may be built to use very recent/modern kernel headers and
   therefore meant to run on a system that uses a recent kernel, while
   the host machine may be running an older kernel.

So basically, using the cross-compiler to build something that will be
executed on the host machine is always wrong.

So the initial patch from Charles, that does this:

+		if [ "x$(RUSTC_HOST_NAME)" != "x$(RUSTC_TARGET_NAME)" ]; then \
+			echo 'cc = "$(TARGET_CROSS)gcc"'; \
+			echo 'cxx = "$(TARGET_CROSS)g++"'; \
+		fi; \

is clearly not good, because:

RUSTC_HOST_NAME = $(RUSTC_HOST_ARCH)-unknown-linux-gnu
RUSTC_TARGET_NAME = $(RUSTC_ARCH)-unknown-linux-gnu$(RUSTC_ABI)

So when you target x86-64 and build on x86-64, RUSTC_HOST_NAME ==
RUSTC_TARGET_NAME, and therefore it will continue to use the target
compiler to build programs meant to run on the host, which is very
wrong.

> As the compiler is built on LLVM, coded in C++, a C++ compiler for the
> host is indeed required. It happens that in the build of LLVM, a host
> tool named FileCheck is built and run for test.
> 
> IIUC, the C compiler is only used for building the standard library
> (this is what I've seen when cross-compiling for ARM).
> 
> So when the target is the same as the host, it might be sensible to
> avoid defining c and cxx in the target configuration file, as suggested
> by your patch.
> 
> I'll poke upstream for clarifications on this special case, though.

Did you get some feedback ?

I'll mark the patch as Changes Requested. We need at least the
following things to be changed:

 - Split into two patches the addition of "cxx" vs. the target != host condition

 - A detailed and documented commit log

Charles, ?ric, could you work on a new version of this patch ?

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com