[Buildroot] [RFC] external toolchain scanner

[Buildroot] [RFC] external toolchain scanner

[Hollis Blanchard]

The attached python script inspects an external toolchain and spits out 
BR config settings, like so (with a Linaro toolchain):

    aurora:buildroot$ ./support/scripts/scan-ext-toolchain /foo/gcc-linaro-aarch64-linux-gnu-4.9-2014.09_linux
    BR2_TOOLCHAIN_EXTERNAL=y
    BR2_TOOLCHAIN_EXTERNAL_GCC_4_9=y
    BR2_TOOLCHAIN_EXTERNAL_HEADERS_3_7=y
    BR2_TOOLCHAIN_EXTERNAL_CXX=y
    BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
    BR2_TOOLCHAIN_EXTERNAL_GF=y
    BR2_TOOLCHAIN_EXTERNAL_CUSTOM_PREFIX="aarch64-linux-gnu"
    BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
    BR2_TOOLCHAIN_EXTERNAL_PATH="/foo/gcc-linaro-aarch64-linux-gnu-4.9-2014.09_linux"

It also works with multi-arch toolchains (this one from Mentor Graphics):

    aurora:buildroot$ ./support/scripts/scan-ext-toolchain /foo/codesourcery/codebench/
    Toolchain supports multiple targets. Please choose one of the following: ['aarch64-linux-gnu', 'arm-none-eabi', 'arm-none-linux-gnueabi']
    aurora:buildroot$ ./support/scripts/scan-ext-toolchain -t arm-none-linux-gnueabi /foo/codesourcery/codebench/
    BR2_TOOLCHAIN_EXTERNAL=y
    BR2_TOOLCHAIN_EXTERNAL_HEADERS_3_16=y
    BR2_TOOLCHAIN_EXTERNAL_GCC_4_9=y
    BR2_TOOLCHAIN_EXTERNAL_CXX=y
    BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
    BR2_TOOLCHAIN_EXTERNAL_CUSTOM_PREFIX=arm-none-linux-gnueabi
    BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
    BR2_TOOLCHAIN_EXTERNAL_PATH="/foo/codesourcery/codebench/"

It complains about bare-metal toolchains:

    aurora:buildroot$ ./support/scripts/scan-ext-toolchain -t arm-none-eabi /foo/codesourcery/codebench/
    Is this a Linux toolchain? Couldn't find the sysroot in:
             /foo/codesourcery/codebench/arm-none-eabi/libc
             /foo/codesourcery/codebench/arm-none-eabi/sysroot


Current limitations that I know of:

1. I haven't run through a full build with it yet, but it looks like 
it's doing the right thing.

2. It detects MUSL and UCLIBC toolchains, but it looks like further work 
is needed to detect SSP, RPC, etc, for those toolchain types.

3. There is no guarantee that BR2_arch matches 
BR2_TOOLCHAIN_EXTERNAL_CUSTOM_PREFIX.

4. Users would run it something like this:

    ./support/scripts/scan-ext-toolchain > toolchain.defconfig
    cat board.defconfig toolchain.defconfig > defconfig
    make ... BR2_DEFCONFIG=defconfig

Creating and managing board.defconfig without toolchain configuration 
data is left as an (awkward?) exercise for the user.


Comments?

-- 
Hollis Blanchard <hollis_blanchard@mentor.com>
Mentor Graphics Emulation Division

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20161122/9ddf9a99/attachment.html>
-------------- next part --------------
#!/usr/bin/env python

# Copyright 2016 Mentor Graphics Corporation
# All Rights Reserved
#
# THIS WORK CONTAINS TRADE SECRET AND PROPRIETARY
# INFORMATION WHICH ARE THE PROPERTY OF MENTOR
# GRAPHICS CORPORATION OR ITS LICENSORS AND IS
# SUBJECT TO LICENSE TERMS.

import sys
import os
import subprocess
import re
from optparse import OptionParser

class ExtToolchainMetadata:
	def __init__(self, base_path):
		self.cfg = {}
		self.tools = {}
		self.base_path = base_path
		self.sysroot_path = None

	def get_buildroot_cfg(self):
		result = 'BR2_TOOLCHAIN_EXTERNAL=y\n'
		for k, v in self.cfg.items():
			result += 'BR2_TOOLCHAIN_EXTERNAL_%s=%s\n' % (k, v)
		return result

def probe_prefix(path, target, metadata):
	libexec_path = os.path.join(path, 'libexec', 'gcc') 

	if not os.path.isdir(libexec_path):
		raise RuntimeError("Couldn't examine directory %s" % libexec_path)

	targets = os.listdir(libexec_path)
	if len(targets) == 0:
		raise RuntimeError("Couldn't find any targets in %s" % libexec_path)

	if not target:
		if len(targets) > 1:
			raise RuntimeError('Toolchain supports multiple targets. '
					'Please choose one of the following: %s' % (targets))
		target = targets[0]
	else:
		if target not in targets:
			raise RuntimeError('Toolchain does not support target %s.' % target)
		target = target

	# XXX use ARCH instead?
	# cpu, vendor_os = target.split('-', 1)
	# metadata.cfg['CUSTOM_PREFIX'] = '"$(ARCH)-%s"' % vendor_os
	metadata.cfg['CUSTOM_PREFIX'] = '"%s"' % target

def probe_tools(path, metadata):
	class Tool:
		def __init__(self, executable, cfgname=None):
			self.executable = executable
			self.cfgname = cfgname

	tools = (
		Tool('gcc'),
		Tool('readelf'),
		Tool('g++', 'CXX'),
		Tool('gfortran', 'GF'),
	)

	prefix = metadata.cfg['CUSTOM_PREFIX'].strip('"')

	for tool in tools:
		full_name = '%s-%s' % (prefix, tool.executable)
		full_path = os.path.join(path, 'bin', full_name)

		if os.path.exists(full_path):
			metadata.tools[tool.executable] = full_path
			if tool.cfgname:
				metadata.cfg[tool.cfgname] = 'y'

def probe_gcc_version(metadata):
	argv = [
		metadata.tools['gcc'],
		'--version',
	]

	proc = subprocess.Popen(argv, stdout=subprocess.PIPE)
	output = proc.communicate()[0]
	line1 = output.splitlines()[0]

	m = re.match('^[^)]+\) ([^ ]+)', line1)
	if not m:
		raise RuntimeError("%s\n\tdidn't report a recognizable version:\n%s" %
				(metadata.tools['gcc'], line1))

	version = m.group(1) # E.g. 4.9.2
	major, minor = [ int(i) for i in version.split('.', 2)[:2] ]
	metadata.cfg['GCC_%d_%d' % (major, minor)] = 'y'

def probe_gcc_sysroot(metadata):
	# Sysroot directories could have a couple names:
	subdirs = ('libc', 'sysroot')

	# Construct a list of full paths so that in case of failure we can tell the
	# user exactly where we searched.
	base = metadata.base_path
	prefix = metadata.cfg['CUSTOM_PREFIX'].strip('"')
	sysroot_paths = [ os.path.join(base, prefix, d) for d in subdirs ]

	sysroot_path = None
	for path in sysroot_paths:
		if os.path.exists(path):
			sysroot_path = path
			break

	if not sysroot_path:
		msg = "Is this a Linux toolchain? Couldn't find the sysroot in:\n\t%s"
		raise RuntimeError(msg % '\n\t'.join(sysroot_paths))

	metadata.sysroot_path = sysroot_path

def probe_gcc_headers(metadata):
	version_path = os.path.join(metadata.sysroot_path,
		'usr',
		'include',
		'linux',
		'version.h'
	)
	version_re = '#define LINUX_VERSION_CODE ([0-9]+)'

	with open(version_path) as version_file:
		linux_version_code = version_file.readline()

	m = re.match(version_re, linux_version_code)
	if not m:
		msg = "Didn't recognize LINUX_VERSION_CODE in %s"
		raise RuntimeError(msg % version_path)

	version = int(m.group(1))
	major = (version >> 16) & 0xff
	minor = (version >> 8) & 0xff
	metadata.cfg['HEADERS_%d_%d' % (major, minor)] = 'y'

def probe_libc(metadata):
	libc_re = '  -m(glibc|musl|eglibc)\s+\[enabled\]'
	argv = [
		metadata.tools['gcc'],
		'-Q',
		'--help=target',
	]

	proc = subprocess.Popen(argv, stdout=subprocess.PIPE)
	output = proc.communicate()[0].splitlines()
	for line in output:
		m = re.match(libc_re, line)
		if m:
			libc = m.group(1)
			metadata.cfg['CUSTOM_%s' % libc.upper()] = 'y'
			break

def probe_rpc(metadata):
	rpc_path = os.path.join(metadata.sysroot_path,
		'usr',
		'include',
		'rpc',
		'rpc.h')
	if os.path.exists(rpc_path):
		metadata.cfg['HAS_RPC'] = 'y'

def probe(path, options, metadata):
	metadata.cfg['CUSTOM'] = 'y'
	metadata.cfg['PATH'] = '"%s"' % path

	probe_prefix(path, options.target, metadata)
	probe_tools(path, metadata)
	probe_gcc_version(metadata)
	probe_gcc_sysroot(metadata)
	probe_gcc_headers(metadata)
	probe_libc(metadata)
	probe_rpc(metadata)

def output_cfg(metadata):
	for k, v in metadata.items():
		sys.stdout.write('BR2_TOOLCHAIN_%s=%s\n' % (k, v))

def main():
	parser = OptionParser()
	parser.add_option('-t', '--target', dest='target')

	(options, arguments) = parser.parse_args()

	if len(arguments) < 1:
		print "Missing path to toolchain base directory"
		sys.exit(1)
	toolchain_dir = arguments[0]
	if not os.path.isdir(toolchain_dir):
		print "Not a directory: %s" % toolchain_dir
		sys.exit(2)

	metadata = ExtToolchainMetadata(toolchain_dir)
	probe(toolchain_dir, options, metadata)
	print metadata.get_buildroot_cfg().strip()

if __name__ == '__main__':
	try:
		main()
	except RuntimeError, e:
		print e

[Buildroot] [RFC] external toolchain scanner

[Baruch Siach]

Hi Hollis,

On Tue, Nov 22, 2016 at 05:32:51PM -0800, Hollis Blanchard wrote:
> It also works with multi-arch toolchains (this one from Mentor Graphics):
> 
>    aurora:buildroot$ ./support/scripts/scan-ext-toolchain /foo/codesourcery/codebench/
>    Toolchain supports multiple targets. Please choose one of the following: ['aarch64-linux-gnu', 'arm-none-eabi', 'arm-none-linux-gnueabi']
>    aurora:buildroot$ ./support/scripts/scan-ext-toolchain -t arm-none-linux-gnueabi /foo/codesourcery/codebench/
>    BR2_TOOLCHAIN_EXTERNAL=y
>    BR2_TOOLCHAIN_EXTERNAL_HEADERS_3_16=y
>    BR2_TOOLCHAIN_EXTERNAL_GCC_4_9=y
>    BR2_TOOLCHAIN_EXTERNAL_CXX=y
>    BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
>    BR2_TOOLCHAIN_EXTERNAL_CUSTOM_PREFIX=arm-none-linux-gnueabi
>    BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
>    BR2_TOOLCHAIN_EXTERNAL_PATH="/foo/codesourcery/codebench/"

Just curious, is this a publicly available toolchain? The latest Sourcery 
CodeBench ARM toolchain that Buildroot knows about is 2014.05, which is based 
on gcc 4.8.3.

[...]

> #!/usr/bin/env python
> 
> # Copyright 2016 Mentor Graphics Corporation
> # All Rights Reserved
> #
> # THIS WORK CONTAINS TRADE SECRET AND PROPRIETARY
> # INFORMATION WHICH ARE THE PROPERTY OF MENTOR
> # GRAPHICS CORPORATION OR ITS LICENSORS AND IS
> # SUBJECT TO LICENSE TERMS.

Are you sure about that?

baruch

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.52.368.4656, http://www.tkos.co.il -

[Buildroot] [RFC] external toolchain scanner

[Hollis Blanchard]

Newer copy attached. I've been using this version for a couple weeks.

This can be run with defconfig as an extra option, like so:

    ./scan-ext-toolchain <toolchain> [<files>]

That will prepend the content of <files> to the toolchain metadata, so 
that it can be used like so:

    ./scan-ext-toolchain -o output/combined_defconfig <toolchain> my-board-defconfig
    make BR2_DEFCONFIG=output/combined_defconfig defconfig


I still haven't tried to probe SSP, RPC, etc, for non-glibc toolchains.

Hollis Blanchard <hollis_blanchard@mentor.com>
Mentor Graphics Emulation Division

On 11/22/2016 05:32 PM, Hollis Blanchard wrote:
> The attached python script inspects an external toolchain and spits 
> out BR config settings, like so (with a Linaro toolchain):
>
>     aurora:buildroot$ ./support/scripts/scan-ext-toolchain /foo/gcc-linaro-aarch64-linux-gnu-4.9-2014.09_linux
>     BR2_TOOLCHAIN_EXTERNAL=y
>     BR2_TOOLCHAIN_EXTERNAL_GCC_4_9=y
>     BR2_TOOLCHAIN_EXTERNAL_HEADERS_3_7=y
>     BR2_TOOLCHAIN_EXTERNAL_CXX=y
>     BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
>     BR2_TOOLCHAIN_EXTERNAL_GF=y
>     BR2_TOOLCHAIN_EXTERNAL_CUSTOM_PREFIX="aarch64-linux-gnu"
>     BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
>     BR2_TOOLCHAIN_EXTERNAL_PATH="/foo/gcc-linaro-aarch64-linux-gnu-4.9-2014.09_linux"
>
> It also works with multi-arch toolchains (this one from Mentor Graphics):
>
>     aurora:buildroot$ ./support/scripts/scan-ext-toolchain /foo/codesourcery/codebench/
>     Toolchain supports multiple targets. Please choose one of the following: ['aarch64-linux-gnu', 'arm-none-eabi', 'arm-none-linux-gnueabi']
>     aurora:buildroot$ ./support/scripts/scan-ext-toolchain -t arm-none-linux-gnueabi /foo/codesourcery/codebench/
>     BR2_TOOLCHAIN_EXTERNAL=y
>     BR2_TOOLCHAIN_EXTERNAL_HEADERS_3_16=y
>     BR2_TOOLCHAIN_EXTERNAL_GCC_4_9=y
>     BR2_TOOLCHAIN_EXTERNAL_CXX=y
>     BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
>     BR2_TOOLCHAIN_EXTERNAL_CUSTOM_PREFIX=arm-none-linux-gnueabi
>     BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
>     BR2_TOOLCHAIN_EXTERNAL_PATH="/foo/codesourcery/codebench/"
>
> It complains about bare-metal toolchains:
>
>     aurora:buildroot$ ./support/scripts/scan-ext-toolchain -t arm-none-eabi /foo/codesourcery/codebench/
>     Is this a Linux toolchain? Couldn't find the sysroot in:
>              /foo/codesourcery/codebench/arm-none-eabi/libc
>              /foo/codesourcery/codebench/arm-none-eabi/sysroot
>
>
> Current limitations that I know of:
>
> 1. I haven't run through a full build with it yet, but it looks like 
> it's doing the right thing.
>
> 2. It detects MUSL and UCLIBC toolchains, but it looks like further 
> work is needed to detect SSP, RPC, etc, for those toolchain types.
>
> 3. There is no guarantee that BR2_arch matches 
> BR2_TOOLCHAIN_EXTERNAL_CUSTOM_PREFIX.
>
> 4. Users would run it something like this:
>
>     ./support/scripts/scan-ext-toolchain > toolchain.defconfig
>     cat board.defconfig toolchain.defconfig > defconfig
>     make ... BR2_DEFCONFIG=defconfig
>
> Creating and managing board.defconfig without toolchain configuration 
> data is left as an (awkward?) exercise for the user.
>
>
> Comments?
>
> -- 
> Hollis Blanchard<hollis_blanchard@mentor.com>
> Mentor Graphics Emulation Division

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20161207/2a49114a/attachment.html>
-------------- next part --------------
#!/usr/bin/env python

# Copyright 2016 Mentor Graphics Corporation
# All Rights Reserved
#
# THIS WORK CONTAINS TRADE SECRET AND PROPRIETARY
# INFORMATION WHICH ARE THE PROPERTY OF MENTOR
# GRAPHICS CORPORATION OR ITS LICENSORS AND IS
# SUBJECT TO LICENSE TERMS.

import sys
import os
import subprocess
import re
from optparse import OptionParser

class ExtToolchainMetadata:
	def __init__(self, base_path):
		self.cfg = {}
		self.tools = {}
		self.base_path = base_path
		self.sysroot_path = None

	def get_buildroot_cfg(self):
		result = 'BR2_TOOLCHAIN_EXTERNAL=y\n'
		for k, v in self.cfg.items():
			result += 'BR2_TOOLCHAIN_EXTERNAL_%s=%s\n' % (k, v)
		return result

def probe_prefix(path, target, metadata):
	libexec_path = os.path.join(path, 'libexec', 'gcc') 

	if not os.path.isdir(libexec_path):
		raise RuntimeError("Couldn't examine directory %s" % libexec_path)

	targets = os.listdir(libexec_path)
	if len(targets) == 0:
		raise RuntimeError("Couldn't find any targets in %s" % libexec_path)

	if not target:
		if len(targets) > 1:
			raise RuntimeError('Toolchain supports multiple targets. '
					'Please choose one of the following: %s' % (targets))
		target = targets[0]
	else:
		if target not in targets:
			raise RuntimeError('Toolchain does not support target %s.' % target)
		target = target

	# XXX use ARCH instead?
	# cpu, vendor_os = target.split('-', 1)
	# metadata.cfg['CUSTOM_PREFIX'] = '"$(ARCH)-%s"' % vendor_os
	metadata.cfg['CUSTOM_PREFIX'] = '"%s"' % target

def probe_tools(path, metadata):
	class Tool:
		def __init__(self, executable, cfgname=None):
			self.executable = executable
			self.cfgname = cfgname

	tools = (
		Tool('gcc'),
		Tool('readelf'),
		Tool('g++', 'CXX'),
		Tool('gfortran', 'GF'),
	)

	prefix = metadata.cfg['CUSTOM_PREFIX'].strip('"')

	for tool in tools:
		full_name = '%s-%s' % (prefix, tool.executable)
		full_path = os.path.join(path, 'bin', full_name)

		if os.path.exists(full_path):
			metadata.tools[tool.executable] = full_path
			if tool.cfgname:
				metadata.cfg[tool.cfgname] = 'y'

def probe_gcc_version(metadata):
	argv = [
		metadata.tools['gcc'],
		'--version',
	]

	proc = subprocess.Popen(argv, stdout=subprocess.PIPE)
	output = proc.communicate()[0]
	line1 = output.splitlines()[0]

	m = re.match('^[^)]+\) ([^ ]+)', line1)
	if not m:
		raise RuntimeError("%s\n\tdidn't report a recognizable version:\n%s" %
				(metadata.tools['gcc'], line1))

	version = m.group(1) # E.g. 4.9.2
	major, minor = [ int(i) for i in version.split('.', 2)[:2] ]
	metadata.cfg['GCC_%d_%d' % (major, minor)] = 'y'

def probe_gcc_sysroot(metadata):
	# Sysroot directories could have a couple names:
	subdirs = ('libc', 'sysroot')

	# Construct a list of full paths so that in case of failure we can tell the
	# user exactly where we searched.
	base = metadata.base_path
	prefix = metadata.cfg['CUSTOM_PREFIX'].strip('"')
	sysroot_paths = [ os.path.join(base, prefix, d) for d in subdirs ]

	sysroot_path = None
	for path in sysroot_paths:
		if os.path.exists(path):
			sysroot_path = path
			break

	if not sysroot_path:
		msg = "Is this a Linux toolchain? Couldn't find the sysroot in:\n\t%s"
		raise RuntimeError(msg % '\n\t'.join(sysroot_paths))

	metadata.sysroot_path = sysroot_path

def probe_gcc_headers(metadata):
	version_path = os.path.join(metadata.sysroot_path,
		'usr',
		'include',
		'linux',
		'version.h'
	)
	version_re = '#define LINUX_VERSION_CODE ([0-9]+)'

	with open(version_path) as version_file:
		linux_version_code = version_file.readline()

	m = re.match(version_re, linux_version_code)
	if not m:
		msg = "Didn't recognize LINUX_VERSION_CODE in %s"
		raise RuntimeError(msg % version_path)

	version = int(m.group(1))
	major = (version >> 16) & 0xff
	minor = (version >> 8) & 0xff
	metadata.cfg['HEADERS_%d_%d' % (major, minor)] = 'y'

def probe_libc(metadata):
	libc_re = '  -m(glibc|musl|eglibc)\s+\[enabled\]'
	argv = [
		metadata.tools['gcc'],
		'-Q',
		'--help=target',
	]

	proc = subprocess.Popen(argv, stdout=subprocess.PIPE)
	output = proc.communicate()[0].splitlines()
	for line in output:
		m = re.match(libc_re, line)
		if m:
			libc = m.group(1)
			metadata.cfg['CUSTOM_%s' % libc.upper()] = 'y'
			break

def probe_rpc(metadata):
	rpc_path = os.path.join(metadata.sysroot_path,
		'usr',
		'include',
		'rpc',
		'rpc.h')
	if os.path.exists(rpc_path):
		metadata.cfg['HAS_RPC'] = 'y'

def probe(path, options, metadata):
	metadata.cfg['CUSTOM'] = 'y'
	metadata.cfg['PATH'] = '"%s"' % path

	probe_prefix(path, options.target, metadata)
	probe_tools(path, metadata)
	probe_gcc_version(metadata)
	probe_gcc_sysroot(metadata)
	probe_gcc_headers(metadata)
	probe_libc(metadata)
	probe_rpc(metadata)

def cat_files(out_file, files):
	for path in files:
		with open(path) as f:
			for line in f.readlines():
				out_file.write(line)

def main():
	parser = OptionParser()
	parser.add_option('-t', '--target', dest='target')
	parser.add_option('-o', '--output', default='/dev/stdout')

	(options, arguments) = parser.parse_args()

	if len(arguments) < 1:
		print "Missing path to toolchain base directory"
		sys.exit(1)

	toolchain_dir = arguments[0]
	if not os.path.isdir(toolchain_dir):
		print "Not a directory: %s" % toolchain_dir
		sys.exit(2)

	metadata = ExtToolchainMetadata(toolchain_dir)
	probe(toolchain_dir, options, metadata)

	with open(options.output, 'w') as out_file:
		if len(arguments) >= 1:
			cat_files(out_file, arguments[1:])

		out_file.write(metadata.get_buildroot_cfg())

if __name__ == '__main__':
	try:
		main()
	except RuntimeError, e:
		print e

[Buildroot] [RFC] external toolchain scanner

[Yann E. MORIN]

Hollis, All,

On 2016-12-07 10:14 -0800, Hollis Blanchard spake thusly:
> Newer copy attached. I've been using this version for a couple weeks.
> 
> This can be run with defconfig as an extra option, like so:
> 
>  ./scan-ext-toolchain <toolchain> [<files>]
> 
> That will prepend the content of <files> to the toolchain metadata, so that it can be used like so:
> 
>  ./scan-ext-toolchain -o output/combined_defconfig <toolchain> my-board-defconfig
>  make BR2_DEFCONFIG=output/combined_defconfig defconfig
> 
> I still haven't tried to probe SSP, RPC, etc, for non-glibc toolchains.

Thanks for working on this! :-)

Here are a few comments, plus a big question...

> #!/usr/bin/env python
> 
> # Copyright 2016 Mentor Graphics Corporation
> # All Rights Reserved
> #
> # THIS WORK CONTAINS TRADE SECRET AND PROPRIETARY
> # INFORMATION WHICH ARE THE PROPERTY OF MENTOR
> # GRAPHICS CORPORATION OR ITS LICENSORS AND IS
> # SUBJECT TO LICENSE TERMS.

So, it has already been pointed out, but this boilerplate is not
acceptable. It really means what it says: one needs a license to this
script before using it in any way. Said license ios not specified, so it
is as good as no license.

One may argue that the license of the script defaults to the license of
Buidlroot, in which it is going to be distributed. However, before we
_accept_ the script in Buildroot, we need to be sure the license is
compatible with that of Buildroot. The boiulerplate is definitely not
saying it is.

It is totally acceptable that Mentor and you be identified as the
authors and/or copyright holders, but the license must be explicit. We
can only carry that script in Buildroot if the license is acceptable.
Buildroot is currently GPLv2, so GPLv2 would be the obvious choice of a
licnese. But we could very well accept any license that is compatible
with GPLv2 (BSD and such, for example...).

Please fix this before we can consider including it in Buildroot.

> import sys
> import os
> import subprocess
> import re
> from optparse import OptionParser
> 
> class ExtToolchainMetadata:
> 	def __init__(self, base_path):
> 		self.cfg = {}
> 		self.tools = {}
> 		self.base_path = base_path
> 		self.sysroot_path = None
> 
> 	def get_buildroot_cfg(self):
> 		result = 'BR2_TOOLCHAIN_EXTERNAL=y\n'
> 		for k, v in self.cfg.items():
> 			result += 'BR2_TOOLCHAIN_EXTERNAL_%s=%s\n' % (k, v)
> 		return result
> 
> def probe_prefix(path, target, metadata):
> 	libexec_path = os.path.join(path, 'libexec', 'gcc') 
> 
> 	if not os.path.isdir(libexec_path):
> 		raise RuntimeError("Couldn't examine directory %s" % libexec_path)
> 
> 	targets = os.listdir(libexec_path)
> 	if len(targets) == 0:
> 		raise RuntimeError("Couldn't find any targets in %s" % libexec_path)
> 
> 	if not target:
> 		if len(targets) > 1:
> 			raise RuntimeError('Toolchain supports multiple targets. '
> 					'Please choose one of the following: %s' % (targets))
> 		target = targets[0]
> 	else:
> 		if target not in targets:
> 			raise RuntimeError('Toolchain does not support target %s.' % target)
> 		target = target
> 
> 	# XXX use ARCH instead?
> 	# cpu, vendor_os = target.split('-', 1)
> 	# metadata.cfg['CUSTOM_PREFIX'] = '"$(ARCH)-%s"' % vendor_os
> 	metadata.cfg['CUSTOM_PREFIX'] = '"%s"' % target
> 
> def probe_tools(path, metadata):
> 	class Tool:
> 		def __init__(self, executable, cfgname=None):
> 			self.executable = executable
> 			self.cfgname = cfgname
> 
> 	tools = (
> 		Tool('gcc'),
> 		Tool('readelf'),
> 		Tool('g++', 'CXX'),
> 		Tool('gfortran', 'GF'),
> 	)
> 
> 	prefix = metadata.cfg['CUSTOM_PREFIX'].strip('"')
> 
> 	for tool in tools:
> 		full_name = '%s-%s' % (prefix, tool.executable)
> 		full_path = os.path.join(path, 'bin', full_name)
> 
> 		if os.path.exists(full_path):
> 			metadata.tools[tool.executable] = full_path
> 			if tool.cfgname:
> 				metadata.cfg[tool.cfgname] = 'y'
> 
> def probe_gcc_version(metadata):
> 	argv = [
> 		metadata.tools['gcc'],
> 		'--version',
> 	]
> 
> 	proc = subprocess.Popen(argv, stdout=subprocess.PIPE)
> 	output = proc.communicate()[0]
> 	line1 = output.splitlines()[0]
> 
> 	m = re.match('^[^)]+\) ([^ ]+)', line1)
> 	if not m:
> 		raise RuntimeError("%s\n\tdidn't report a recognizable version:\n%s" %
> 				(metadata.tools['gcc'], line1))
> 
> 	version = m.group(1) # E.g. 4.9.2
> 	major, minor = [ int(i) for i in version.split('.', 2)[:2] ]
> 	metadata.cfg['GCC_%d_%d' % (major, minor)] = 'y'
> 
> def probe_gcc_sysroot(metadata):
> 	# Sysroot directories could have a couple names:
> 	subdirs = ('libc', 'sysroot')
> 
> 	# Construct a list of full paths so that in case of failure we can tell the
> 	# user exactly where we searched.
> 	base = metadata.base_path
> 	prefix = metadata.cfg['CUSTOM_PREFIX'].strip('"')
> 	sysroot_paths = [ os.path.join(base, prefix, d) for d in subdirs ]
> 
> 	sysroot_path = None
> 	for path in sysroot_paths:
> 		if os.path.exists(path):
> 			sysroot_path = path
> 			break
> 
> 	if not sysroot_path:
> 		msg = "Is this a Linux toolchain? Couldn't find the sysroot in:\n\t%s"
> 		raise RuntimeError(msg % '\n\t'.join(sysroot_paths))
> 
> 	metadata.sysroot_path = sysroot_path

Well, finding the correct syseroot is not trivial: you need to know the
CFLAGS that are to be used, because they directly influence what sysroot
gcc will be searching for.

See in toolchain/toolchain-external/pkg-toolchain-external.mk how we
find the sysroot: It is a bit convoluted, but it basically goes as such:

  - we ask gcc where libc.a is, line 338:
        gcc $(CFLAGS) -print-file-name=libc.a

  - we strip the trailing relative path to get the sysroot path:

        echo $(libc.a) |sed -r -e 's:(usr/)?lib(32|64)?([^/]*)?/([^/]*/)?libc\.a::'

The worst part here is getting a hold on the CFLAGS. Without them, you
can't find the sysroot.

That's because the sysroot may change with various flags, like -march of
-mfpu or whatever. You can check by asking gcc what multilib and/or
multiarch it supports, and the flags that will redirect to another
sysroot:

    # host/opt/ext-toolchain/bin/mips-img-linux-gnu-gcc -print-multi-lib
    mips-r1-hard/lib;@mips32 at mabi=32 at EB
    mipsel-r1-hard/lib;@mips32 at mabi=32 at EL
    mips-r1-soft/lib;@mips32 at mabi=32 at EB@msoft-float
    mipsel-r1-soft/lib;@mips32 at mabi=32 at EL@msoft-float
    mips-r1-mips16-hard/lib;@mips32 at mips16@mabi=32 at EB
    mipsel-r1-mips16-hard/lib;@mips32 at mips16@mabi=32 at EL
    mips-r1-mips16-soft/lib;@mips32 at mips16@mabi=32 at EB@msoft-float
    mipsel-r1-mips16-soft/lib;@mips32 at mips16@mabi=32 at EL@msoft-float
    mips-r2-hard/lib;@mips32r2 at mabi=32 at EB
    mipsel-r2-hard/lib;@mips32r2 at mabi=32 at EL
    mips-r2-soft/lib;@mips32r2 at mabi=32 at EB@msoft-float
    mips-r2-hard-nan2008/lib;@mips32r2 at mabi=32 at EB@mnan=2008
    mipsel-r2-soft/lib;@mips32r2 at mabi=32 at EL@msoft-float
    mipsel-r2-hard-nan2008/lib;@mips32r2 at mabi=32 at EL@mnan=2008
    mips-r2-mips16-hard/lib;@mips32r2 at mips16@mabi=32 at EB
    mipsel-r2-mips16-hard/lib;@mips32r2 at mips16@mabi=32 at EL
    mips-r2-mips16-soft/lib;@mips32r2 at mips16@mabi=32 at EB@msoft-float
    mips-r2-mips16-hard-nan2008/lib;@mips32r2 at mips16@mabi=32 at EB@mnan=2008
    mipsel-r2-mips16-soft/lib;@mips32r2 at mips16@mabi=32 at EL@msoft-float
    mipsel-r2-mips16-hard-nan2008/lib;@mips32r2 at mips16@mabi=32 at EL@mnan=2008
    micromips-r2-soft/lib;@mips32r2 at mmicromips@mabi=32 at EB@msoft-float
    micromips-r2-hard-nan2008/lib;@mips32r2 at mmicromips@mabi=32 at EB@mnan=2008
    micromipsel-r2-soft/lib;@mips32r2 at mmicromips@mabi=32 at EL@msoft-float
    micromipsel-r2-hard-nan2008/lib;@mips32r2 at mmicromips@mabi=32 at EL@mnan=2008
    mips-r6-hard/lib;@mips32r6 at mabi=32 at EB@mnan=2008
    mips-r6-soft/lib;@mips32r6 at mabi=32 at EB@msoft-float at mnan=2008
    mipsel-r6-hard/lib;@mips32r6 at mabi=32 at EL@mnan=2008
    mipsel-r6-soft/lib;@mips32r6 at mabi=32 at EL@msoft-float at mnan=2008
    mipsel-r6-singleshort/lib;@mips32r6 at mabi=32 at EL@msingle-float at mnan=2008 at fshort-double
    micromips-r6-hard/lib;@mips32r6 at mmicromips@mabi=32 at EB@mnan=2008
    micromips-r6-soft/lib;@mips32r6 at mmicromips@mabi=32 at EB@msoft-float at mnan=2008
    micromipsel-r6-hard/lib;@mips32r6 at mmicromips@mabi=32 at EL@mnan=2008
    micromipsel-r6-soft/lib;@mips32r6 at mmicromips@mabi=32 at EL@msoft-float at mnan=2008
    mips-r1-hard/lib32;@mips64 at mabi=n32 at EB
    mipsel-r1-hard/lib32;@mips64 at mabi=n32 at EL
    mips-r1-soft/lib32;@mips64 at mabi=n32 at EB@msoft-float
    mipsel-r1-soft/lib32;@mips64 at mabi=n32 at EL@msoft-float
    mips-r1-hard/lib64;@mips64 at mabi=64 at EB
    mipsel-r1-hard/lib64;@mips64 at mabi=64 at EL
    mips-r1-soft/lib64;@mips64 at mabi=64 at EB@msoft-float
    mipsel-r1-soft/lib64;@mips64 at mabi=64 at EL@msoft-float
    mips-r2-hard/lib32;@mips64r2 at mabi=n32 at EB
    mipsel-r2-hard/lib32;@mips64r2 at mabi=n32 at EL
    mips-r2-soft/lib32;@mips64r2 at mabi=n32 at EB@msoft-float
    mips-r2-hard-nan2008/lib32;@mips64r2 at mabi=n32 at EB@mnan=2008
    mipsel-r2-soft/lib32;@mips64r2 at mabi=n32 at EL@msoft-float
    mipsel-r2-hard-nan2008/lib32;@mips64r2 at mabi=n32 at EL@mnan=2008
    mips-r2-hard/lib64;@mips64r2 at mabi=64 at EB
    mipsel-r2-hard/lib64;@mips64r2 at mabi=64 at EL
    mips-r2-soft/lib64;@mips64r2 at mabi=64 at EB@msoft-float
    mips-r2-hard-nan2008/lib64;@mips64r2 at mabi=64 at EB@mnan=2008
    mipsel-r2-soft/lib64;@mips64r2 at mabi=64 at EL@msoft-float
    mipsel-r2-hard-nan2008/lib64;@mips64r2 at mabi=64 at EL@mnan=2008
    mips-r6-hard/lib32;@mips64r6 at mabi=n32 at EB@mnan=2008
    mips-r6-soft/lib32;@mips64r6 at mabi=n32 at EB@msoft-float at mnan=2008
    mipsel-r6-hard/lib32;@mips64r6 at mabi=n32 at EL@mnan=2008
    mipsel-r6-soft/lib32;@mips64r6 at mabi=n32 at EL@msoft-float at mnan=2008
    mips-r6-hard/lib64;@mips64r6 at mabi=64 at EB@mnan=2008
    mips-r6-soft/lib64;@mips64r6 at mabi=64 at EB@msoft-float at mnan=2008
    mipsel-r6-hard/lib64;@mips64r6 at mabi=64 at EL@mnan=2008
    mipsel-r6-soft/lib64;@mips64r6 at mabi=64 at EL@msoft-float at mnan=2008
    mips-r2-hard-uclibc/lib;@muclibc at mips32r2@mabi=32 at EB
    mipsel-r2-hard-uclibc/lib;@muclibc at mips32r2@mabi=32 at EL
    mips-r2-hard-nan2008-uclibc/lib;@muclibc at mips32r2@mabi=32 at EB@mnan=2008
    mipsel-r2-hard-nan2008-uclibc/lib;@muclibc at mips32r2@mabi=32 at EL@mnan=2008

Yes, that's 65 different ways to use a specific sysroot. The last three
lines mean:

    when these flags flags are in use,              then use this sysroot
    -muclibc -mips32r2 -mabi=32 -EL                 mipsel-r2-hard-uclibc/lib
    -muclibc -mips32r2 -mabi=32 -EB -mnan=2008      mips-r2-hard-nan2008-uclibc/lib
    -muclibc -mips32r2 -mabi=32 -EL -mnan=2008      mipsel-r2-hard-nan2008-uclibc/lib

The last two are pretty obvious: big vs. little endian. But when you
look at the antepenultimate line, you'll see that it differs only by not
specifying the way to treat NaNs.

So, without the actual CFLAGS, you can't find the sysroot...

And you can't know the actual CFLAGS untill you actually have a
Buildroot .config file.

But you need the sysroot...

> def probe_gcc_headers(metadata):
> 	version_path = os.path.join(metadata.sysroot_path,
> 		'usr',
> 		'include',
> 		'linux',
> 		'version.h'
> 	)
> 	version_re = '#define LINUX_VERSION_CODE ([0-9]+)'
> 
> 	with open(version_path) as version_file:
> 		linux_version_code = version_file.readline()
> 
> 	m = re.match(version_re, linux_version_code)
> 	if not m:
> 		msg = "Didn't recognize LINUX_VERSION_CODE in %s"
> 		raise RuntimeError(msg % version_path)
> 
> 	version = int(m.group(1))
> 	major = (version >> 16) & 0xff
> 	minor = (version >> 8) & 0xff
> 	metadata.cfg['HEADERS_%d_%d' % (major, minor)] = 'y'
> 
> def probe_libc(metadata):
> 	libc_re = '  -m(glibc|musl|eglibc)\s+\[enabled\]'
> 	argv = [
> 		metadata.tools['gcc'],
> 		'-Q',
> 		'--help=target',
> 	]
> 
> 	proc = subprocess.Popen(argv, stdout=subprocess.PIPE)
> 	output = proc.communicate()[0].splitlines()
> 	for line in output:
> 		m = re.match(libc_re, line)
> 		if m:
> 			libc = m.group(1)
> 			metadata.cfg['CUSTOM_%s' % libc.upper()] = 'y'
> 			break
> 
> def probe_rpc(metadata):
> 	rpc_path = os.path.join(metadata.sysroot_path,
> 		'usr',
> 		'include',
> 		'rpc',
> 		'rpc.h')
> 	if os.path.exists(rpc_path):
> 		metadata.cfg['HAS_RPC'] = 'y'

... to find whther RPC is supoprted or not.

An alternative to testing RPC support would be to do a test-build, to
see if gcc can find rpc.h, but again you'd need the correct CFLAGS to do
such a build.

So, without CFLAGS, you can't know whether RPC is available or not.

We could see at using a shirtcut: if the default gcc can find a rpc.h,
we could infer that any variant would have it as well.

But that;s not necessarily the case. If the toolchain supports various C
libraries (like this one: it supports bith glibc and uClibc), we can't
do that supposition: if glibc is the default, then rpc.h will be present
[0], but uClibc can very well have been configured without RPC support
in that toolchain. So we can't even test the defaults either... :-/

[0] except for a few releases dating back a few years, where glibc did
not have rpc.h for two or three releases in a row.

Well, I can't see an easy path... :-/

OK, enough for tonight. Maybe more this WE...

Regards,
Yann E. MORIN.

> def probe(path, options, metadata):
> 	metadata.cfg['CUSTOM'] = 'y'
> 	metadata.cfg['PATH'] = '"%s"' % path
> 
> 	probe_prefix(path, options.target, metadata)
> 	probe_tools(path, metadata)
> 	probe_gcc_version(metadata)
> 	probe_gcc_sysroot(metadata)
> 	probe_gcc_headers(metadata)
> 	probe_libc(metadata)
> 	probe_rpc(metadata)
> 
> def cat_files(out_file, files):
> 	for path in files:
> 		with open(path) as f:
> 			for line in f.readlines():
> 				out_file.write(line)
> 
> def main():
> 	parser = OptionParser()
> 	parser.add_option('-t', '--target', dest='target')
> 	parser.add_option('-o', '--output', default='/dev/stdout')
> 
> 	(options, arguments) = parser.parse_args()
> 
> 	if len(arguments) < 1:
> 		print "Missing path to toolchain base directory"
> 		sys.exit(1)
> 
> 	toolchain_dir = arguments[0]
> 	if not os.path.isdir(toolchain_dir):
> 		print "Not a directory: %s" % toolchain_dir
> 		sys.exit(2)
> 
> 	metadata = ExtToolchainMetadata(toolchain_dir)
> 	probe(toolchain_dir, options, metadata)
> 
> 	with open(options.output, 'w') as out_file:
> 		if len(arguments) >= 1:
> 			cat_files(out_file, arguments[1:])
> 
> 		out_file.write(metadata.get_buildroot_cfg())
> 
> if __name__ == '__main__':
> 	try:
> 		main()
> 	except RuntimeError, e:
> 		print e


-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [RFC] external toolchain scanner

[Hollis Blanchard]

On 12/08/2016 01:02 PM, Yann E. MORIN wrote:
> Well, finding the correct syseroot is not trivial: you need to know the
> CFLAGS that are to be used, because they directly influence what sysroot
> gcc will be searching for.
>
> ...
>      # host/opt/ext-toolchain/bin/mips-img-linux-gnu-gcc -print-multi-lib
>      mipsel-r2-hard-uclibc/lib;@muclibc at mips32r2@mabi=32 at EL
>      mips-r2-hard-nan2008-uclibc/lib;@muclibc at mips32r2@mabi=32 at EB@mnan=2008
>      mipsel-r2-hard-nan2008-uclibc/lib;@muclibc at mips32r2@mabi=32 at EL@mnan=2008
>
> Yes, that's 65 different ways to use a specific sysroot. The last three
> lines mean:
>
>      when these flags flags are in use,              then use this sysroot
>      -muclibc -mips32r2 -mabi=32 -EL                 mipsel-r2-hard-uclibc/lib
>      -muclibc -mips32r2 -mabi=32 -EB -mnan=2008      mips-r2-hard-nan2008-uclibc/lib
>      -muclibc -mips32r2 -mabi=32 -EL -mnan=2008      mipsel-r2-hard-nan2008-uclibc/lib
>
> The last two are pretty obvious: big vs. little endian. But when you
> look at the antepenultimate line, you'll see that it differs only by not
> specifying the way to treat NaNs.
>
> So, without the actual CFLAGS, you can't find the sysroot...
>
> And you can't know the actual CFLAGS untill you actually have a
> Buildroot .config file.
>
> But you need the sysroot...
>

A few thoughts:

1. I can't escape the observation that what I have is actually useful 
for my limited case of a 1:1 target:sysroot relationship. Given that it 
works with Linaro's free downloadable toolchains, which I believe are 
popular, I suspect it could help many people.

2. What options exactly determine the CFLAGS? If not a lot of processing 
is involved, those options could also be fed into the scanner for it to 
calculate CFLAGS independently, i.e.

  * .config -> scanner -> combined.config
  * combined.config -> make -> output/*


3. Currently, the only user input for this script is the target 
architecture tuple, from which it tries to figure out the sysroot.
3.1. Perhaps that should be flipped: the user could instead be required 
to input the sysroot, from which the target architecture could be 
determined?
3.2. Perhaps the user could be required to input both the architecture 
/and/ sysroot. That would be an optional prompt if multiple sysroots are 
preset. Just like with the multiarch prompting that's done today, it 
could look like this:

aurora:buildroot$ ./support/scripts/scan-ext-toolchain \
> host/opt/ext-toolchain/bin/mips-img-linux-gnu-gcc
Toolchain supports multiple targets. Please choose one of the following:
   aarch64-linux-gnu
   arm-none-eabi
   arm-none-linux-gnueabi

aurora:buildroot$ ./support/scripts/scan-ext-toolchain \
> -t arm-none-linux-gnueabi \
> host/opt/ext-toolchain/bin/mips-img-linux-gnu-gcc
Multiple sysroots available for target arm-none-linux-gnueabi. Please choose one of the following:
   mipsel-r2-hard-uclibc
   mips-r2-hard-nan2008-uclibc
   mipsel-r2-hard-nan2008-uclibc

aurora:buildroot$ ./support/scripts/scan-ext-toolchain \
> -t arm-none-linux-gnueabi \
> -s mipsel-r2-hard-uclibc \
> host/opt/ext-toolchain/bin/mips-img-linux-gnu-gcc
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_HEADERS_3_16=y
...

(I have no idea is this is feasible, just trying to figure out the right 
approach before I go digging.)

Hollis Blanchard <hollis_blanchard@mentor.com>
Mentor Graphics Emulation Division

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20161219/955618bc/attachment.html>

[Buildroot] [RFC] external toolchain scanner

[Hollis Blanchard]

On 12/19/2016 12:07 PM, Hollis Blanchard wrote:
> On 12/08/2016 01:02 PM, Yann E. MORIN wrote:
>> Well, finding the correct syseroot is not trivial: you need to know the
>> CFLAGS that are to be used, because they directly influence what sysroot
>> gcc will be searching for.
>>
>> ...
>>      # host/opt/ext-toolchain/bin/mips-img-linux-gnu-gcc -print-multi-lib
>>      mipsel-r2-hard-uclibc/lib;@muclibc at mips32r2@mabi=32 at EL
>>      mips-r2-hard-nan2008-uclibc/lib;@muclibc at mips32r2@mabi=32 at EB@mnan=2008
>>      mipsel-r2-hard-nan2008-uclibc/lib;@muclibc at mips32r2@mabi=32 at EL@mnan=2008
>>
>> Yes, that's 65 different ways to use a specific sysroot. The last three
>> lines mean:
>>
>>      when these flags flags are in use,              then use this sysroot
>>      -muclibc -mips32r2 -mabi=32 -EL                 mipsel-r2-hard-uclibc/lib
>>      -muclibc -mips32r2 -mabi=32 -EB -mnan=2008      mips-r2-hard-nan2008-uclibc/lib
>>      -muclibc -mips32r2 -mabi=32 -EL -mnan=2008      mipsel-r2-hard-nan2008-uclibc/lib
>>
>> The last two are pretty obvious: big vs. little endian. But when you
>> look at the antepenultimate line, you'll see that it differs only by not
>> specifying the way to treat NaNs.
>>
>> So, without the actual CFLAGS, you can't find the sysroot...
>>
>> And you can't know the actual CFLAGS untill you actually have a
>> Buildroot .config file.
>>
>> But you need the sysroot...
>>
>
> A few thoughts:
>
> 1. I can't escape the observation that what I have is actually useful 
> for my limited case of a 1:1 target:sysroot relationship. Given that 
> it works with Linaro's free downloadable toolchains, which I believe 
> are popular, I suspect it could help many people.
>
> 2. What options exactly determine the CFLAGS? If not a lot of 
> processing is involved, those options could also be fed into the 
> scanner for it to calculate CFLAGS independently, i.e.
>
>   * .config -> scanner -> combined.config
>   * combined.config -> make -> output/*
>
>
> 3. Currently, the only user input for this script is the target 
> architecture tuple, from which it tries to figure out the sysroot.
> 3.1. Perhaps that should be flipped: the user could instead be 
> required to input the sysroot, from which the target architecture 
> could be determined?
> 3.2. Perhaps the user could be required to input both the architecture 
> /and/ sysroot. That would be an optional prompt if multiple sysroots 
> are preset. Just like with the multiarch prompting that's done today, 
> it could look like this:
> aurora:buildroot$ ./support/scripts/scan-ext-toolchain \
> > host/opt/ext-toolchain/bin/mips-img-linux-gnu-gcc
> Toolchain supports multiple targets. Please choose one of the following:
>    aarch64-linux-gnu
>    arm-none-eabi
>    arm-none-linux-gnueabi
>
> aurora:buildroot$ ./support/scripts/scan-ext-toolchain \
> > -t arm-none-linux-gnueabi \
> > host/opt/ext-toolchain/bin/mips-img-linux-gnu-gcc
> Multiple sysroots available for target arm-none-linux-gnueabi. Please choose one of the following:
>    mipsel-r2-hard-uclibc
>    mips-r2-hard-nan2008-uclibc
>    mipsel-r2-hard-nan2008-uclibc
>
> aurora:buildroot$ ./support/scripts/scan-ext-toolchain \
> > -t arm-none-linux-gnueabi \
> > -s mipsel-r2-hard-uclibc \
> > host/opt/ext-toolchain/bin/mips-img-linux-gnu-gcc
> BR2_TOOLCHAIN_EXTERNAL=y
> BR2_TOOLCHAIN_EXTERNAL_HEADERS_3_16=y
> ...
> (I have no idea is this is feasible, just trying to figure out the 
> right approach before I go digging.)
Ping.

Hollis Blanchard <hollis_blanchard@mentor.com>
Mentor Graphics Emulation Division

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20170109/f5eb165b/attachment.html>

[Buildroot] [RFC] external toolchain scanner

[Yann E. MORIN]

Hollis, All,

On 2017-01-09 15:07 -0800, Hollis Blanchard spake thusly:
> On 12/19/2016 12:07 PM, Hollis Blanchard wrote:
[--SNIP--]
>   A few thoughts:
> 
>   1. I can't escape the observation that what I have is actually useful for my limited case of a 1:1 target:sysroot relationship.
>   Given that it works with Linaro's free downloadable toolchains, which I believe are popular, I suspect it could help many people.
> 
>   2. What options exactly determine the CFLAGS? If not a lot of processing is involved, those options could also be fed into the
>   scanner for it to calculate CFLAGS independently, i.e.
>     * .config -> scanner -> combined.config
>     * combined.config -> make -> output/*

Well, the whole point of the scanner is to come up with an initial
.config pre-filled with the toolchain settings.

Going to the menuconfig then back to the shell to run the scanner, then
back to the menuconfig is not user-friendly... :-/

>   3. Currently, the only user input for this script is the target architecture tuple, from which it tries to figure out the sysroot.
>   3.1. Perhaps that should be flipped: the user could instead be required to input the sysroot, from which the target architecture
>   could be determined?

No, that is not a viable solution.

The underlying reason is that we do determine the sysroot from the
CFLAGS, and we would risk a mis-match between the sysroot as set by the
user and the settings as determined by Buildroot.

We do not want to risk such a mis-match.

Besides, most users are totally oblivious at what a sysroot is. All they
get is a toolchain.

And back to your first comment: yes, Linaro toolchains might be very
common and widely used by hobbyists that use Buildroot. But in an
enterprise situation, the toolchain more often than not comes from a SoC
or system vendor. So in such a setup, pre-built public toolchains are
seldom used.

>   3.2. Perhaps the user could be required to input both the architecture and sysroot. That would be an optional prompt if multiple
>   sysroots are preset. Just like with the multiarch prompting that's done today, it could look like this:

Or the scanner could just bail out, instructing the user to go to the
menuconfig and fill the values all by himself. Which defeats the very
purpose of the toolchain in the first place.

We don't want a scanner if all it is able to do is just fill the obvious
values. The scanner only becomes interesting if it can do the job in the
most complex cases (IMHO).

>  aurora:buildroot$ ./support/scripts/scan-ext-toolchain \
>  > host/opt/ext-toolchain/bin/mips-img-linux-gnu-gcc
>  > Toolchain supports multiple targets. Please choose one of the following:
>  >   aarch64-linux-gnu  arm-none-eabi  arm-none-linux-gnueabi
>  > aurora:buildroot$ ./support/scripts/scan-ext-toolchain \
>  > -t arm-none-linux-gnueabi \> host/opt/ext-toolchain/bin/mips-img-linux-gnu-gcc
>  > Multiple sysroots available for target arm-none-linux-gnueabi. Please choose one of the following:
>  >   mipsel-r2-hard-uclibc  mips-r2-hard-nan2008-uclibc
>  >   mipsel-r2-hard-nan2008-uclibc

Again, in most cases, the user has no way to know before-hand what
sysroot to use, even less so what a sysroot is.

And suffice that the user chooses the wrong sysroot (all to probable),
we would end up setting the wrong values in the generated .config, and
thus the checks we do would still kick in and refuse the configuartion.

>  > aurora:buildroot$ ./support/scripts/scan-ext-toolchain \
>  > -t arm-none-linux-gnueabi \> -s mipsel-r2-hard-uclibc \
>  > host/opt/ext-toolchain/bin/mips-img-linux-gnu-gcc
>  BR2_TOOLCHAIN_EXTERNAL=yBR2_TOOLCHAIN_EXTERNAL_HEADERS_3_16=y
>  ...
> 
>   (I have no idea is this is feasible, just trying to figure out the right approach before I go digging.)

If it were only for me, I would say that it is not posible to do a
scanner. As I already explained, there are un-breakable chicken-n-egg
problems...

However, what we could do is spit-out all the incorrect settings on
first run, rather than stop on the first problem. That way, the user
would have a single run to check the settings and fix them, rather than
the current behaviour of going back to the menuconfig for each and every
incorrect setting...

I would rather that we work toward this goal, which would be a
termendous improvement on what we currently have.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [RFC] external toolchain scanner

[Hollis Blanchard]

On 01/12/2017 09:32 AM, Yann E. MORIN wrote:
> Hollis, All,
>
> On 2017-01-09 15:07 -0800, Hollis Blanchard spake thusly:
>> On 12/19/2016 12:07 PM, Hollis Blanchard wrote:
>>    2. What options exactly determine the CFLAGS? If not a lot of processing is involved, those options could also be fed into the
>>    scanner for it to calculate CFLAGS independently, i.e.
>>      * .config -> scanner -> combined.config
>>      * combined.config -> make -> output/*
> Well, the whole point of the scanner is to come up with an initial
> .config pre-filled with the toolchain settings.
>
> Going to the menuconfig then back to the shell to run the scanner, then
> back to the menuconfig is not user-friendly... :-/

In my case, the user is already provided with a .config, which is why I 
mention it above. However, in the case where there is no .config nor 
board/*/defconfig, no initial .config is necessary.

In other words, your scenario becomes:

  * scanner -> .config
  * .config -> make menuconfig -> .config
  * .config -> make -> output/*

>>    3.2. Perhaps the user could be required to input both the architecture and sysroot. That would be an optional prompt if multiple
>>    sysroots are preset. Just like with the multiarch prompting that's done today, it could look like this:
> Or the scanner could just bail out, instructing the user to go to the
> menuconfig and fill the values all by himself. Which defeats the very
> purpose of the toolchain in the first place.
>
> We don't want a scanner if all it is able to do is just fill the obvious
> values. The scanner only becomes interesting if it can do the job in the
> most complex cases (IMHO).
On this point, I strongly believe the opposite. The whole reason I did 
this in the first place is because there are values that Buildroot 
/doesn't/ fill in which are easily-calculable, but not obvious to the 
end-user. Buildroot actually already knows the answers, but makes the 
user try to figure them out anyways, then complains if the user gets it 
wrong. That's... "not desirable," to put it politely. :-)

If there's some reason that Buildroot /can't/ calculate the answers, 
then by all means the scanner should abort too (or simply decline to 
emit the settings).
> However, what we could do is spit-out all the incorrect settings on
> first run, rather than stop on the first problem. That way, the user
> would have a single run to check the settings and fix them, rather than
> the current behaviour of going back to the menuconfig for each and every
> incorrect setting...
>
> I would rather that we work toward this goal, which would be a
> termendous improvement on what we currently have.
I agree that emitting 5 errors at once, rather than emitting 1 error a 
time for 5 iterations, is an improvement on a disagreeable situation. I 
bet you can guess what I'll say next, though: when you look at the 
bigger picture, we shouldn't be in this situation to begin with. :-) 
When we already know the answers, we should tell the user, not the other 
way around.

Hollis Blanchard <hollis_blanchard@mentor.com>
Mentor Graphics Emulation Division

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20170112/ed6b4316/attachment.html>

[Buildroot] [RFC] external toolchain scanner

[Arnout Vandecappelle]

On 12-01-17 23:12, Hollis Blanchard wrote:
> On 01/12/2017 09:32 AM, Yann E. MORIN wrote:
>> Hollis, All,
>>
>> On 2017-01-09 15:07 -0800, Hollis Blanchard spake thusly:
>>> On 12/19/2016 12:07 PM, Hollis Blanchard wrote:
>>>   2. What options exactly determine the CFLAGS? If not a lot of processing is involved, those options could also be fed into the
>>>   scanner for it to calculate CFLAGS independently, i.e.
>>>     * .config -> scanner -> combined.config
>>>     * combined.config -> make -> output/*
>> Well, the whole point of the scanner is to come up with an initial
>> .config pre-filled with the toolchain settings.
>>
>> Going to the menuconfig then back to the shell to run the scanner, then
>> back to the menuconfig is not user-friendly... :-/

 It's a *lot* more user-friendly than going to the menuconfig, getting a bunch
of choices you don't know how to answer, then going to the shell, typing make,
getting an error, trying to find out the needed value, going to menuconfig,
setting a value that is hopefully correct, going to the shell, typing make,
getting another error, ... you get the picture.

 Could we perhaps aim for the following scenario:

* make menuconfig
* make
 -> error "Incorrect toolchain settings detected. An updated config file has
been generated in $(CONFIG_DIR)/config.toolchain. Copy it to
$(CONFIG_DIR)/.config and run 'make menuconfig' to verify."
* Copy
* make menuconfig
* make

 Or perhaps even better, add an option
BR2_TOOLCHAIN_EXTERNAL_CUSTOM_AUTODISCOVER. This option would disable all the
other options except BR2_TOOLCHAIN_EXTRA_EXTERNAL_LIBS, and also disable all the
other menus. This option would run the autodiscovery and overwrite the .config
file (removing the CUSTOM_AUTODISCOVER option again). So it would force the
following scenario:

* make menuconfig
* make
* make menuconfig
* make

 But it's probably best to implement the first scenario and see where we go from
there.

> In my case, the user is already provided with a .config, which is why I mention
> it above. However, in the case where there is no .config nor board/*/defconfig,
> no initial .config is necessary.
> 
> In other words, your scenario becomes:
> 
>   * scanner -> .config
>   * .config -> make menuconfig -> .config
>   * .config -> make -> output/*
> 
>>>   3.2. Perhaps the user could be required to input both the architecture and sysroot. That would be an optional prompt if multiple
>>>   sysroots are preset. Just like with the multiarch prompting that's done today, it could look like this:
>> Or the scanner could just bail out, instructing the user to go to the
>> menuconfig and fill the values all by himself. Which defeats the very
>> purpose of the toolchain in the first place.
>>
>> We don't want a scanner if all it is able to do is just fill the obvious
>> values. The scanner only becomes interesting if it can do the job in the
>> most complex cases (IMHO).BR2_TOOLCHAIN_EXTRA_EXTERNAL_LIBS
> On this point, I strongly believe the opposite. 

 I agree. Anything that makes it easier to set the custom toolchain options is
worth it.


> The whole reason I did this in
> the first place is because there are values that Buildroot /doesn't/ fill in
> which are easily-calculable, but not obvious to the end-user. Buildroot actually
> already knows the answers, but makes the user try to figure them out anyways,
> then complains if the user gets it wrong. That's... "not desirable," to put it
> politely. :-)
> 
> If there's some reason that Buildroot /can't/ calculate the answers, then by all
> means the scanner should abort too (or simply decline to emit the settings).
>> However, what we could do is spit-out all the incorrect settings on
>> first run, rather than stop on the first problem. That way, the user
>> would have a single run to check the settings and fix them, rather than
>> the current behaviour of going back to the menuconfig for each and every
>> incorrect setting...
>>
>> I would rather that we work toward this goal, which would be a
>> termendous improvement on what we currently have.
> I agree that emitting 5 errors at once, rather than emitting 1 error a time for
> 5 iterations, is an improvement on a disagreeable situation. I bet you can guess
> what I'll say next, though: when you look at the bigger picture, we shouldn't be
> in this situation to begin with. :-) When we already know the answers, we should
> tell the user, not the other way around.

 But perhaps that would be an easier first step. Instead of just reporting one
single error, report all errors and the correct config options so the user can
edit the .config file and add them.


 Coming back to your original RFC: there were quite a few things I didn't like
about it.

- passing the arch tuple isn't nice, better use
 $(TOOLCHAIN_EXTERNAL_CC) $(TOOLCHAIN_EXTERNAL_CFLAGS)
 like is done in the existing checks

- I'd prefer if the scanner was integrated in the normal build process, i.e.
that it's not needed to call it explicitly

- but then, I'm not so enthusiastic about a python script that gets called in
the normal build process...

 Regards,
 Arnout

> 
> Hollis Blanchard <hollis_blanchard@mentor.com>
> Mentor Graphics Emulation Division
> 
> 
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [RFC] external toolchain scanner

[Thomas Petazzoni]

Hello,

On Fri, 13 Jan 2017 00:43:16 +0100, Arnout Vandecappelle wrote:

> * make menuconfig
> * make
>  -> error "Incorrect toolchain settings detected. An updated config file has  
> been generated in $(CONFIG_DIR)/config.toolchain. Copy it to
> $(CONFIG_DIR)/.config and run 'make menuconfig' to verify."
> * Copy
> * make menuconfig
> * make
> 
>  Or perhaps even better, add an option
> BR2_TOOLCHAIN_EXTERNAL_CUSTOM_AUTODISCOVER. This option would disable all the
> other options except BR2_TOOLCHAIN_EXTRA_EXTERNAL_LIBS, and also disable all the
> other menus. This option would run the autodiscovery and overwrite the .config
> file (removing the CUSTOM_AUTODISCOVER option again). So it would force the
> following scenario:
> 
> * make menuconfig
> * make
> * make menuconfig
> * make
> 
>  But it's probably best to implement the first scenario and see where we go from
> there.

To be honest, I find all of this very complicated, and even more
confusing than what we have today. Using custom external toolchain is
already an "advanced" feature, which users just getting started with
Buildroot are unlikely to use. To me, using custom external toolchain
is a bit of an advanced functionality, and people who need that will
surely be able to understand the information they need to fill in
"menuconfig" to make Buildroot "accept" the toolchain.

> >> I would rather that we work toward this goal, which would be a
> >> termendous improvement on what we currently have.  
> > I agree that emitting 5 errors at once, rather than emitting 1 error a time for
> > 5 iterations, is an improvement on a disagreeable situation. I bet you can guess
> > what I'll say next, though: when you look at the bigger picture, we shouldn't be
> > in this situation to begin with. :-) When we already know the answers, we should
> > tell the user, not the other way around.  
> 
>  But perhaps that would be an easier first step. Instead of just reporting one
> single error, report all errors and the correct config options so the user can
> edit the .config file and add them.

Yes. *This* would be useful.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [RFC] external toolchain scanner

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

Hi,

 >> But it's probably best to implement the first scenario and see where we go from
 >> there.

 > To be honest, I find all of this very complicated, and even more
 > confusing than what we have today. Using custom external toolchain is
 > already an "advanced" feature, which users just getting started with
 > Buildroot are unlikely to use. To me, using custom external toolchain
 > is a bit of an advanced functionality, and people who need that will
 > surely be able to understand the information they need to fill in
 > "menuconfig" to make Buildroot "accept" the toolchain.

Yes, I'm also not really sure what realistic use cases this scripts
covers.

 >> But perhaps that would be an easier first step. Instead of just reporting one
 >> single error, report all errors and the correct config options so the user can
 >> edit the .config file and add them.

 > Yes. *This* would be useful.

Agreed.

-- 
Bye, Peter Korsgaard

[Buildroot] [RFC] external toolchain scanner

[Hollis Blanchard]

On 01/16/2017 06:54 AM, Peter Korsgaard wrote:
>>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:
> Hi,
>
>   >> But it's probably best to implement the first scenario and see where we go from
>   >> there.
>
>   > To be honest, I find all of this very complicated, and even more
>   > confusing than what we have today. Using custom external toolchain is
>   > already an "advanced" feature, which users just getting started with
>   > Buildroot are unlikely to use. To me, using custom external toolchain
>   > is a bit of an advanced functionality, and people who need that will
>   > surely be able to understand the information they need to fill in
>   > "menuconfig" to make Buildroot "accept" the toolchain.
>
> Yes, I'm also not really sure what realistic use cases this scripts
> covers.
Sure, I can elaborate.

My team provides target software in binary and source form, which our 
users can rebuild. Our users are accustomed to using an external 
toolchain to rebuild the source, and so already have a toolchain 
installed that they've been using (in some cases for years). In other 
words, using an external toolchain is quite the opposite of "advanced": 
being able to use their known/trusted toolchain is step 1.

So that's perspective #1, for our users. Our team has a different 
perspective (#2): we have target software expertise, but not a lot of 
toolchain expertise. We would prefer to let producers of toolchains 
distribute and maintain them. Given Buildroot's limitations on providing 
host tools (which we've discussed in separate threads), we certainly 
don't want to have Buildroot create an internal toolchain. We can 
probably build and distribute a completely external toolchain (e.g. via 
crosstool-ng), but then we need to a) maintain it and b) convince our 
users that they should use ours instead of the one they've already 
obtained from toolchain people who know what they're doing. :-) That 
creates more work for us, so we don't want that either.

To summarize, our users want to rebuild our source using the toolchain 
of their choice. They don't want a toolchain from us, and we don't want 
to give them a toolchain. Buildroot currently makes this difficult. 
Therefore, I have written a script that will make it easier to use 
arbitrary external toolchains, and since it can help other people too 
I'm offering to contribute it. Right now we are distributing this script 
outside Buildroot, which works for us but doesn't help any other 
Buildroot users.

Hollis Blanchard <hollis_blanchard@mentor.com>
Mentor Graphics Emulation Division

[Buildroot] [RFC] external toolchain scanner

[Blanchard, Hollis]

On Nov 22, 2016 10:24 PM, Baruch Siach <baruch@tkos.co.il> wrote:
>
> Hi Hollis,
>
> On Tue, Nov 22, 2016 at 05:32:51PM -0800, Hollis Blanchard wrote:
> > It also works with multi-arch toolchains (this one from Mentor Graphics):
> >
> >    aurora:buildroot$ ./support/scripts/scan-ext-toolchain /foo/codesourcery/codebench/
> >    Toolchain supports multiple targets. Please choose one of the following: ['aarch64-linux-gnu', 'arm-none-eabi', 'arm-none-linux-gnueabi']
> >    aurora:buildroot$ ./support/scripts/scan-ext-toolchain -t arm-none-linux-gnueabi /foo/codesourcery/codebench/
> >    BR2_TOOLCHAIN_EXTERNAL=y
> >    BR2_TOOLCHAIN_EXTERNAL_HEADERS_3_16=y
> >    BR2_TOOLCHAIN_EXTERNAL_GCC_4_9=y
> >    BR2_TOOLCHAIN_EXTERNAL_CXX=y
> >    BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
> >    BR2_TOOLCHAIN_EXTERNAL_CUSTOM_PREFIX=arm-none-linux-gnueabi
> >    BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
> >    BR2_TOOLCHAIN_EXTERNAL_PATH="/foo/codesourcery/codebench/"
>
> Just curious, is this a publicly available toolchain? The latest Sourcery
> CodeBench ARM toolchain that Buildroot knows about is 2014.05, which is based
> on gcc 4.8.3.

Do you mean public, or free? I believe this toolchain is the former but not the latter.

> [...]
>
> > #!/usr/bin/env python
> >
> > # Copyright 2016 Mentor Graphics Corporation
> > # All Rights Reserved
> > #
> > # THIS WORK CONTAINS TRADE SECRET AND PROPRIETARY
> > # INFORMATION WHICH ARE THE PROPERTY OF MENTOR
> > # GRAPHICS CORPORATION OR ITS LICENSORS AND IS
> > # SUBJECT TO LICENSE TERMS.
>
> Are you sure about that?

Yes. The GPL is a license...

-Hollis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20161123/d3065e55/attachment.html>