[U-Boot] [PATCH 0/3] buildman updates

[U-Boot] [PATCH 0/3] buildman updates

[Trevor Woerner]

Hi,

This patchset was created to update buildman, and fix a couple small issues
that were discovered. kernel.org has a new set of toolchains (8.1.0), and I
ran into some issues trying to download them.

I'm not a python guru, so if there are better ways to code these fixes, please
let me know.

Best regards,
	Trevor


Trevor Woerner (3):
  buildman/toolchain.py: update versions
  buildman/toolchain.py: fix toolchain directory
  buildman/toolchain.py: handle inconsistent tarball names

 tools/buildman/toolchain.py | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

-- 
2.17.0.582.gccdcbd54c

[U-Boot] [PATCH 1/3] buildman/toolchain.py: update versions

[Trevor Woerner]

On kernel.org, a newer 8.1.0 version of the toolchains is available. Also,
update the kernel.org version list to include some of the older versions so
more architectures can be supported.

Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
---
 tools/buildman/toolchain.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/buildman/toolchain.py b/tools/buildman/toolchain.py
index 4b35f400e9..442a7f977a 100644
--- a/tools/buildman/toolchain.py
+++ b/tools/buildman/toolchain.py
@@ -430,7 +430,7 @@ class Toolchains:
         """
         arch = command.OutputOneLine('uname', '-m')
         base = 'https://www.kernel.org/pub/tools/crosstool/files/bin'
-        versions = ['7.3.0', '6.4.0', '4.9.4']
+        versions = ['8.1.0', '7.3.0', '6.4.0', '5.5.0', '4.9.4', '4.9.0', '4.8.5', '4.8.0', '4.7.3', '4.6.3', '4.6.2', '4.6.1', '4.5.1', '4.2.4']
         links = []
         for version in versions:
             url = '%s/%s/%s/' % (base, arch, version)
-- 
2.17.0.582.gccdcbd54c

[U-Boot] [PATCH 1/3] buildman/toolchain.py: update versions

[Simon Glass]

On Wed, 21 Nov 2018 at 01:32, Trevor Woerner <trevor@toganlabs.com> wrote:
>
> On kernel.org, a newer 8.1.0 version of the toolchains is available. Also,
> update the kernel.org version list to include some of the older versions so
> more architectures can be supported.
>
> Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
> ---
>  tools/buildman/toolchain.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Reviewed-by: Simon Glass <sjg@chromium.org>

[U-Boot] [PATCH 1/3] buildman/toolchain.py: update versions

[Daniel Schwierzeck]

Am 21.11.18 um 09:31 schrieb Trevor Woerner:
> On kernel.org, a newer 8.1.0 version of the toolchains is available. Also,
> update the kernel.org version list to include some of the older versions so
> more architectures can be supported.
> 
> Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
> ---
>  tools/buildman/toolchain.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tools/buildman/toolchain.py b/tools/buildman/toolchain.py
> index 4b35f400e9..442a7f977a 100644
> --- a/tools/buildman/toolchain.py
> +++ b/tools/buildman/toolchain.py
> @@ -430,7 +430,7 @@ class Toolchains:
>          """
>          arch = command.OutputOneLine('uname', '-m')
>          base = 'https://www.kernel.org/pub/tools/crosstool/files/bin'
> -        versions = ['7.3.0', '6.4.0', '4.9.4']
> +        versions = ['8.1.0', '7.3.0', '6.4.0', '5.5.0', '4.9.4', '4.9.0', '4.8.5', '4.8.0', '4.7.3', '4.6.3', '4.6.2', '4.6.1', '4.5.1', '4.2.4']

with 4c58d273e0f2cc21821c1de32494abd5de411d9b we removed the older and
unneeded versions. So it doesn't make sense to add them again.

Also did you a test run on Travis CI to verify that all archs still
compile where a 8.1.0 toolchain is available? Travis CI will
automatically use the latest avaiable toolchain for each arch. Possible
new warnings or errors need to be fixed before doing the switch to 8.1.0.

>          links = []
>          for version in versions:
>              url = '%s/%s/%s/' % (base, arch, version)
> 

-- 
- Daniel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20181129/76f94910/attachment.sig>

[U-Boot] [PATCH 1/3] buildman/toolchain.py: update versions

[Tom Rini]

On Thu, Nov 29, 2018 at 02:35:56AM +0100, Daniel Schwierzeck wrote:
> 
> 
> Am 21.11.18 um 09:31 schrieb Trevor Woerner:
> > On kernel.org, a newer 8.1.0 version of the toolchains is available. Also,
> > update the kernel.org version list to include some of the older versions so
> > more architectures can be supported.
> > 
> > Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
> > ---
> >  tools/buildman/toolchain.py | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/tools/buildman/toolchain.py b/tools/buildman/toolchain.py
> > index 4b35f400e9..442a7f977a 100644
> > --- a/tools/buildman/toolchain.py
> > +++ b/tools/buildman/toolchain.py
> > @@ -430,7 +430,7 @@ class Toolchains:
> >          """
> >          arch = command.OutputOneLine('uname', '-m')
> >          base = 'https://www.kernel.org/pub/tools/crosstool/files/bin'
> > -        versions = ['7.3.0', '6.4.0', '4.9.4']
> > +        versions = ['8.1.0', '7.3.0', '6.4.0', '5.5.0', '4.9.4', '4.9.0', '4.8.5', '4.8.0', '4.7.3', '4.6.3', '4.6.2', '4.6.1', '4.5.1', '4.2.4']
> 
> with 4c58d273e0f2cc21821c1de32494abd5de411d9b we removed the older and
> unneeded versions. So it doesn't make sense to add them again.
> 
> Also did you a test run on Travis CI to verify that all archs still
> compile where a 8.1.0 toolchain is available? Travis CI will
> automatically use the latest avaiable toolchain for each arch. Possible
> new warnings or errors need to be fixed before doing the switch to 8.1.0.

Agreed.  Are you using buildman outside of U-Boot where you have a
usecase for adding in old versions / other architectures? I do want to
move up to 8.1.0 but I'm sure we have warnings to fix before we can do
that.  I _think_ I even posted a link to what was failing a while ago
and fixed a few things, but there's some harder to solve problems left.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20181129/7c3d1b85/attachment.sig>

[U-Boot] [PATCH 1/3] buildman/toolchain.py: update versions

[Trevor Woerner]

On Thu, Nov 29, 2018 at 2:03 PM Tom Rini <trini@konsulko.com> wrote:

> On Thu, Nov 29, 2018 at 02:35:56AM +0100, Daniel Schwierzeck wrote:
> >
> >
> > Am 21.11.18 um 09:31 schrieb Trevor Woerner:
> > > On kernel.org, a newer 8.1.0 version of the toolchains is available.
> Also,
> > > update the kernel.org version list to include some of the older
> versions so
> > > more architectures can be supported.
> > >
> > > Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
> > > ---
> > >  tools/buildman/toolchain.py | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/tools/buildman/toolchain.py b/tools/buildman/toolchain.py
> > > index 4b35f400e9..442a7f977a 100644
> > > --- a/tools/buildman/toolchain.py
> > > +++ b/tools/buildman/toolchain.py
> > > @@ -430,7 +430,7 @@ class Toolchains:
> > >          """
> > >          arch = command.OutputOneLine('uname', '-m')
> > >          base = 'https://www.kernel.org/pub/tools/crosstool/files/bin'
> > > -        versions = ['7.3.0', '6.4.0', '4.9.4']
> > > +        versions = ['8.1.0', '7.3.0', '6.4.0', '5.5.0', '4.9.4',
> '4.9.0', '4.8.5', '4.8.0', '4.7.3', '4.6.3', '4.6.2', '4.6.1', '4.5.1',
> '4.2.4']
> >
> > with 4c58d273e0f2cc21821c1de32494abd5de411d9b we removed the older and
> > unneeded versions. So it doesn't make sense to add them again.
> >
> > Also did you a test run on Travis CI to verify that all archs still
> > compile where a 8.1.0 toolchain is available? Travis CI will
> > automatically use the latest avaiable toolchain for each arch. Possible
> > new warnings or errors need to be fixed before doing the switch to 8.1.0.
>
> Agreed.  Are you using buildman outside of U-Boot where you have a
> usecase for adding in old versions / other architectures?


Yes. I was using it as a general-purpose toolchain-downloading tool.


> I do want to
> move up to 8.1.0 but I'm sure we have warnings to fix before we can do
> that.  I _think_ I even posted a link to what was failing a while ago
> and fixed a few things, but there's some harder to solve problems left.
>

Okay, no problem.

In any case I'm partway through doing a test with and without my patches.
I'll post my results for the sake of completeness.

I'm also interested in exploring whether it might be possible to have
buildman download pre-built toolchains from bootlin. Maybe that would be of
some interest?

[U-Boot] [PATCH 1/3] buildman/toolchain.py: update versions

[Tom Rini]

On Fri, Nov 30, 2018 at 06:48:35AM -0500, Trevor Woerner wrote:
> On Thu, Nov 29, 2018 at 2:03 PM Tom Rini <trini@konsulko.com> wrote:
> 
> > On Thu, Nov 29, 2018 at 02:35:56AM +0100, Daniel Schwierzeck wrote:
> > >
> > >
> > > Am 21.11.18 um 09:31 schrieb Trevor Woerner:
> > > > On kernel.org, a newer 8.1.0 version of the toolchains is available.
> > Also,
> > > > update the kernel.org version list to include some of the older
> > versions so
> > > > more architectures can be supported.
> > > >
> > > > Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
> > > > ---
> > > >  tools/buildman/toolchain.py | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > > diff --git a/tools/buildman/toolchain.py b/tools/buildman/toolchain.py
> > > > index 4b35f400e9..442a7f977a 100644
> > > > --- a/tools/buildman/toolchain.py
> > > > +++ b/tools/buildman/toolchain.py
> > > > @@ -430,7 +430,7 @@ class Toolchains:
> > > >          """
> > > >          arch = command.OutputOneLine('uname', '-m')
> > > >          base = 'https://www.kernel.org/pub/tools/crosstool/files/bin'
> > > > -        versions = ['7.3.0', '6.4.0', '4.9.4']
> > > > +        versions = ['8.1.0', '7.3.0', '6.4.0', '5.5.0', '4.9.4',
> > '4.9.0', '4.8.5', '4.8.0', '4.7.3', '4.6.3', '4.6.2', '4.6.1', '4.5.1',
> > '4.2.4']
> > >
> > > with 4c58d273e0f2cc21821c1de32494abd5de411d9b we removed the older and
> > > unneeded versions. So it doesn't make sense to add them again.
> > >
> > > Also did you a test run on Travis CI to verify that all archs still
> > > compile where a 8.1.0 toolchain is available? Travis CI will
> > > automatically use the latest avaiable toolchain for each arch. Possible
> > > new warnings or errors need to be fixed before doing the switch to 8.1.0.
> >
> > Agreed.  Are you using buildman outside of U-Boot where you have a
> > usecase for adding in old versions / other architectures?
> 
> 
> Yes. I was using it as a general-purpose toolchain-downloading tool.
> 
> 
> > I do want to
> > move up to 8.1.0 but I'm sure we have warnings to fix before we can do
> > that.  I _think_ I even posted a link to what was failing a while ago
> > and fixed a few things, but there's some harder to solve problems left.
> >
> 
> Okay, no problem.
> 
> In any case I'm partway through doing a test with and without my patches.
> I'll post my results for the sake of completeness.
> 
> I'm also interested in exploring whether it might be possible to have
> buildman download pre-built toolchains from bootlin. Maybe that would be of
> some interest?

We had thought about that for a bit, for a while, but Arnd is keeping
the kernel.org ones more up to date than Bootlin was, when we had need
of easy access to a specific new version.  Things may have changed since
then since it was 1-2 years ago, at this point.  But yes, a patch to add
an option to download from elsewhere (perhaps by base URL?) would be
useful I think.  I can see cases of pointing buildman at an internal
mirror instead for example.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20181130/cec37fb7/attachment.sig>

[U-Boot] [PATCH 1/3] buildman/toolchain.py: update versions

[Trevor Woerner]

On Fri 2018-11-30 @ 06:48:35 AM, Trevor Woerner wrote:
> In any case I'm partway through doing a test with and without my patches.
> I'll post my results for the sake of completeness.

Using a relatively recent pull, I have the "master" test, and the "twoerner"
test which simply adds my patches on top of "master":

	$ git show-branch --sha1-name master contrib/twoerner/buildman
	! [master] Merge branch '2018-11-28-master-imports'
	 * [contrib/twoerner/buildman] buildman/toolchain.py: handle inconsistent tarball names
	--
	 * [11028a5d2a] buildman/toolchain.py: handle inconsistent tarball names
	 * [ab4e8c6d2c] buildman/toolchain.py: fix toolchain directory
	 * [e8b8be4d32] buildman/toolchain.py: update versions
	-- [e16c888fab] Merge branch '2018-11-28-master-imports'

Cleaning out ~/.buildman and ~/.buildman-toolchains between each test, here
are my results after fetching the toolchains:

	$ ./tools/buildman/buildman --fetch-arch all

	$ diff -u master/toolchains twoerner/toolchains 
	--- master/toolchains   2018-11-30 20:32:33.593062110 -0500
	+++ twoerner/toolchains 2018-11-30 20:33:20.025307799 -0500
	@@ -1,14 +1,17 @@
	-List of available toolchains (35):
	+List of available toolchains (41):
	 aarch64
	 alpha
	 am33_2.0
	 arc
	 arm
	+avr32
	 bfin
	 c6x
	 cris
	+crisv32
	 frv
	 h8300
	+hexagon
	 hppa
	 hppa64
	 i386
	@@ -18,14 +21,17 @@
	 microblaze
	 mips
	 mips64
	+nds32le
	 nios2
	 or1k
	+or32
	 powerpc
	 powerpc64
	 ppc64le
	 riscv32
	 riscv64
	 s390
	+s390x
	 sh2
	 sh4
	 sparc

including version information:

	$ diff -u master/toolchains-versions twoerner/toolchains-versions 
	--- master/toolchains-versions  2018-11-30 20:31:56.328864954 -0500
	+++ twoerner/toolchains-versions        2018-11-30 20:31:51.088837231 -0500
	@@ -1,36 +1,42 @@
	-List of available toolchains (35):
	-aarch64   : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/aarch64-linux/bin/aarch64-linux-gcc
	-alpha     : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/alpha-linux/bin/alpha-linux-gcc
	+List of available toolchains (41):
	+aarch64   : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/aarch64-linux/bin/aarch64-linux-gcc
	+alpha     : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/alpha-linux/bin/alpha-linux-gcc
	 am33_2.0  : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/am33_2.0-linux/bin/am33_2.0-linux-gcc
	-arc       : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/arc-elf/bin/arc-elf-gcc
	-arm       : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/arm-linux-gnueabi/bin/arm-linux-gnueabi-gcc
	+arc       : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/arc-linux/bin/arc-linux-gcc
	+arm       : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/arm-linux-gnueabi/bin/arm-linux-gnueabi-gcc
	+avr32     : /home/trevor/.buildman-toolchains/gcc-4.2.4-nolibc/avr32-linux/bin/avr32-linux-gcc
	 bfin      : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/bfin-uclinux/bin/bfin-uclinux-gcc
	-c6x       : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/c6x-elf/bin/c6x-elf-gcc
	+c6x       : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/c6x-elf/bin/c6x-elf-gcc
	 cris      : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/cris-linux/bin/cris-linux-gcc
	+crisv32   : /home/trevor/.buildman-toolchains/gcc-4.6.3-nolibc/crisv32-linux/bin/crisv32-linux-gcc
	 frv       : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/frv-linux/bin/frv-linux-gcc
	-h8300     : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/h8300-linux/bin/h8300-linux-gcc
	-hppa      : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/hppa-linux/bin/hppa-linux-gcc
	-hppa64    : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/hppa64-linux/bin/hppa64-linux-gcc
	-i386      : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/i386-linux/bin/i386-linux-gcc
	-ia64      : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/ia64-linux/bin/ia64-linux-gcc
	+h8300     : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/h8300-linux/bin/h8300-linux-gcc
	+hexagon   : /home/trevor/.buildman-toolchains/gcc-4.6.1-nolibc/hexagon-linux/bin/hexagon-linux-gcc
	+hppa      : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/hppa-linux/bin/hppa-linux-gcc
	+hppa64    : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/hppa64-linux/bin/hppa64-linux-gcc
	+i386      : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/i386-linux/bin/i386-linux-gcc
	+ia64      : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/ia64-linux/bin/ia64-linux-gcc
	 m32r      : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/m32r-linux/bin/m32r-linux-gcc
	-m68k      : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/m68k-linux/bin/m68k-linux-gcc
	-microblaze: /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/microblaze-linux/bin/microblaze-linux-gcc
	-mips      : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/mips-linux/bin/mips-linux-gcc
	-mips64    : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/mips64-linux/bin/mips64-linux-gcc
	-nios2     : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/nios2-linux/bin/nios2-linux-gcc
	+m68k      : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/m68k-linux/bin/m68k-linux-gcc
	+microblaze: /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/microblaze-linux/bin/microblaze-linux-gcc
	+mips      : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/mips-linux/bin/mips-linux-gcc
	+mips64    : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/mips64-linux/bin/mips64-linux-gcc
	+nds32le   : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/nds32le-elf/bin/nds32le-elf-gcc
	+nios2     : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/nios2-linux/bin/nios2-linux-gcc
	 or1k      : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/or1k-linux/bin/or1k-linux-gcc
	-powerpc   : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/powerpc-linux/bin/powerpc-linux-gcc
	-powerpc64 : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/powerpc64-linux/bin/powerpc64-linux-gcc
	+or32      : /home/trevor/.buildman-toolchains/gcc-4.5.1-nolibc/or32-linux/bin/or32-linux-gcc
	+powerpc   : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/powerpc-linux/bin/powerpc-linux-gcc
	+powerpc64 : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/powerpc64-linux/bin/powerpc64-linux-gcc
	 ppc64le   : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/ppc64le-linux/bin/ppc64le-linux-gcc
	-riscv32   : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/riscv32-linux/bin/riscv32-linux-gcc
	-riscv64   : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/riscv64-linux/bin/riscv64-linux-gcc
	-s390      : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/s390-linux/bin/s390-linux-gcc
	-sh2       : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/sh2-linux/bin/sh2-linux-gcc
	-sh4       : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/sh4-linux/bin/sh4-linux-gcc
	-sparc     : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/sparc-linux/bin/sparc-linux-gcc
	-sparc64   : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/sparc64-linux/bin/sparc64-linux-gcc
	+riscv32   : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/riscv32-linux/bin/riscv32-linux-gcc
	+riscv64   : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/riscv64-linux/bin/riscv64-linux-gcc
	+s390      : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/s390-linux/bin/s390-linux-gcc
	+s390x     : /home/trevor/.buildman-toolchains/gcc-4.9.0-nolibc/s390x-linux/bin/s390x-linux-gcc
	+sh2       : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/sh2-linux/bin/sh2-linux-gcc
	+sh4       : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/sh4-linux/bin/sh4-linux-gcc
	+sparc     : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/sparc-linux/bin/sparc-linux-gcc
	+sparc64   : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/sparc64-linux/bin/sparc64-linux-gcc
	 tilegx    : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/tilegx-linux/bin/tilegx-linux-gcc
	 tilepro   : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/tilepro-linux/bin/tilepro-linux-gcc
	-x86_64    : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/x86_64-linux/bin/x86_64-linux-gcc
	-xtensa    : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/xtensa-linux/bin/xtensa-linux-gcc
	+x86_64    : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/x86_64-linux/bin/x86_64-linux-gcc
	+xtensa    : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/xtensa-linux/bin/xtensa-linux-gcc

Getting buildman to build the head revision for everything that it can, the
command I used is:

	$ ./tools/buildman/buildman --output-dir ~/tmp/u-boot-buildman/master/output-dir 2>&1 | tee ~/tmp/u-boot-buildman/master/build.log
and
	$ ./tools/buildman/buildman --output-dir ~/tmp/u-boot-buildman/twoerner/output-dir 2>&1 | tee ~/tmp/u-boot-buildman/twoerner/build.log

Then, for results:

	$ ./tools/buildman/buildman --output-dir ~/tmp/u-boot-buildman/master/output-dir -s > ~/tmp/u-boot-buildman/master/build-summary
and
	$ ./tools/buildman/buildman --output-dir ~/tmp/u-boot-buildman/twoerner/output-dir -s > ~/tmp/u-boot-buildman/twoerner/build-summary

I editied these result files so that each build target was listed on its own
line (which makes the comparison easier). Comparing between these two builds
yields:

	$ diff -u master/build-summary.clean twoerner/build-summary.clean
	--- master/build-summary.clean  2018-11-30 20:10:33.590069536 -0500
	+++ twoerner/build-summary.clean        2018-11-30 20:09:16.325652908 -0500
	@@ -1,5 +1,6 @@
	 boards.cfg is up to date. Nothing to do.
	 Summary of current source for 1305 boards (20 threads, 1 job per thread)
	+       x86:  +    efi-x86_app
	    aarch64:  w+   khadas-vim
			   pine_h64
			   pinebook
	@@ -34,7 +35,13 @@
			   sopine_baseboard
			   orangepi_zero_plus2
			   mvebu_db-88f3720
	-       arc:  w+   hsdk
	+   powerpc:  w+   MPC8641HPCN_36BIT
	+                  MPC8610HPCD
	+                  MPC8641HPCN
	+                  xpedite517x
	+                  MCR3000
	+                  sbc8641d
	+       arc:  +    iot_devkit
	      nios2:  w+   10m50
	       mips:  w+   imgtec_xilfpga
			   bcm968380gerg_ram
	@@ -65,6 +72,7 @@
			   mk802_a10s
			   am43xx_hs_evm
			   ls1021aqds_nor_lpuart
	+                  h2200
			   mx7ulp_evk_plugin
			   Linksprite_pcDuino
			   display5
	@@ -101,6 +109,7 @@
			   q8_a23_tablet_800x480
			   A20-OLinuXino-Lime
			   imx6qdl_icore_mipi
	+                  zipitz2
			   ls1021atwr_nor_lpuart
			   k2e_hs_evm
			   ls1021atwr_sdcard_qspi
	@@ -139,6 +148,7 @@
			   Bananapro
			   Cubieboard4
			   ls1021aiot_sdcard
	+                  colibri_pxa270
			   ls1021aiot_qspi
			   Yones_Toptech_BS1078_V2
			   imx6qdl_icore_nand
	@@ -272,3 +282,5 @@
			   q8_a13_tablet
			   A13-OLinuXinoM
			   +   openrd_base openrd_client openrd_ultimate
	+     nds32:  +    adp-ag101p
	+                  adp-ae3xx

So we see 3 who new architectures are added: x86, powerpc, and nds32. And even
with the existing architectures, 3 new targets are built, but arc switches
from "hsdk" to "iot_devkit".

Is this meaningful?

[U-Boot] [PATCH 2/3] buildman/toolchain.py: fix toolchain directory

[Trevor Woerner]

The hexagon toolchain (4.6.1) from kernel.org, for example, was packaged in
a way that is different from most toolchains. The first entry when unpacking
most toolchain tarballs is:

	gcc-<version>-nolib/<targetarch>-<system>

e.g.:

	gcc-8.1.0-nolibc/aarch64-linux/

The first entry of the hexagon toolchain, however, is:

	gcc-4.6.1-nolibc/

This causes the buildman logic in toolchain.py::ScanPath() to not be able to
find the "*gcc" executable since it looks in gcc-4.6.1-nolib/{.|bin|usr/bin}
instead of gcc-4.6.1/hexagon-linux/{.|bin|usr/bin}. Therefore when buildman
tries to download a set of toolchains that includes hexagon, the script fails.

This update takes the second line of the tarball unpacking (which works for
all the toolchains I've tested from kernel.org) and parses it to take the
first two elements, separated by '/'. It makes this logic a bit more robust.

Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
---
 tools/buildman/toolchain.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tools/buildman/toolchain.py b/tools/buildman/toolchain.py
index 442a7f977a..326c609a41 100644
--- a/tools/buildman/toolchain.py
+++ b/tools/buildman/toolchain.py
@@ -502,7 +502,8 @@ class Toolchains:
             trailing /
         """
         stdout = command.Output('tar', 'xvfJ', fname, '-C', dest)
-        return stdout.splitlines()[0][:-1]
+        dirs = stdout.splitlines()[1].split('/')[:2]
+        return '/'.join(dirs)
 
     def TestSettingsHasPath(self, path):
         """Check if buildman will find this toolchain
-- 
2.17.0.582.gccdcbd54c

[U-Boot] [PATCH 2/3] buildman/toolchain.py: fix toolchain directory

[Simon Glass]

On Wed, 21 Nov 2018 at 01:32, Trevor Woerner <trevor@toganlabs.com> wrote:
>
> The hexagon toolchain (4.6.1) from kernel.org, for example, was packaged in
> a way that is different from most toolchains. The first entry when unpacking
> most toolchain tarballs is:
>
>         gcc-<version>-nolib/<targetarch>-<system>
>
> e.g.:
>
>         gcc-8.1.0-nolibc/aarch64-linux/
>
> The first entry of the hexagon toolchain, however, is:
>
>         gcc-4.6.1-nolibc/
>
> This causes the buildman logic in toolchain.py::ScanPath() to not be able to
> find the "*gcc" executable since it looks in gcc-4.6.1-nolib/{.|bin|usr/bin}
> instead of gcc-4.6.1/hexagon-linux/{.|bin|usr/bin}. Therefore when buildman
> tries to download a set of toolchains that includes hexagon, the script fails.
>
> This update takes the second line of the tarball unpacking (which works for
> all the toolchains I've tested from kernel.org) and parses it to take the
> first two elements, separated by '/'. It makes this logic a bit more robust.
>
> Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
> ---
>  tools/buildman/toolchain.py | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)

Reviewed-by: Simon Glass <sjg@chromium.org>

BTW I really appreciate your great commit msg.

- Simon

[U-Boot] [PATCH 3/3] buildman/toolchain.py: handle inconsistent tarball names

[Trevor Woerner]

Unfortunately, for some releases the kernel.org toolchain tarball names adhere
to the following pattern:

	<hostarch>-gcc-<ver>-nolib-<targetarch>-<type>.tar.xz

e.g.:
	x86_64-gcc-8.1.0-nolibc-aarch64-linux.tar.xz

while others use the following pattern:

	<hostarch>-gcc-<ver>-nolib_<targetarch>-<type>.tar.xz

e.g.:

	x86_64-gcc-7.3.0-nolibc_aarch64-linux.tar.xz

Notice that the first pattern has dashes throughout, while the second has
dashes throughout except just before the target architecture which has an
underscore.

The "dash throughout" versions from kernel.org are:

	8.1.0, 6.4.0, 5.5.0, 4.9.4, 4.8.5, 4.6.1

while the "dash and underscore" versions from kernel.org are:

	7.3.0, 4.9.0, 4.8.0, 4.7.3, 4.6.3, 4.6.2, 4.5.1, 4.2.4

This tweak allows the code to handle both versions. Note that this tweak also
causes the architecture parsing to get confused and find the following two
bogus architectures, "2.0" and "64", which are explicitly checked for, and
removed.

Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
---
 tools/buildman/toolchain.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/tools/buildman/toolchain.py b/tools/buildman/toolchain.py
index 326c609a41..4db70572e1 100644
--- a/tools/buildman/toolchain.py
+++ b/tools/buildman/toolchain.py
@@ -517,13 +517,14 @@ class Toolchains:
     def ListArchs(self):
         """List architectures with available toolchains to download"""
         host_arch, archives = self.LocateArchUrl('list')
-        re_arch = re.compile('[-a-z0-9.]*_([^-]*)-.*')
+        re_arch = re.compile('[-a-z0-9.]*[-_]([^-]*)-.*')
         arch_set = set()
         for archive in archives:
             # Remove the host architecture from the start
             arch = re_arch.match(archive[len(host_arch):])
             if arch:
-                arch_set.add(arch.group(1))
+                if arch.group(1) != "2.0" and arch.group(1) != "64":
+                    arch_set.add(arch.group(1))
         return sorted(arch_set)
 
     def FetchAndInstall(self, arch):
-- 
2.17.0.582.gccdcbd54c

[U-Boot] [PATCH 3/3] buildman/toolchain.py: handle inconsistent tarball names

[Simon Glass]

On Wed, 21 Nov 2018 at 01:32, Trevor Woerner <trevor@toganlabs.com> wrote:
>
> Unfortunately, for some releases the kernel.org toolchain tarball names adhere
> to the following pattern:
>
>         <hostarch>-gcc-<ver>-nolib-<targetarch>-<type>.tar.xz
>
> e.g.:
>         x86_64-gcc-8.1.0-nolibc-aarch64-linux.tar.xz
>
> while others use the following pattern:
>
>         <hostarch>-gcc-<ver>-nolib_<targetarch>-<type>.tar.xz
>
> e.g.:
>
>         x86_64-gcc-7.3.0-nolibc_aarch64-linux.tar.xz
>
> Notice that the first pattern has dashes throughout, while the second has
> dashes throughout except just before the target architecture which has an
> underscore.
>
> The "dash throughout" versions from kernel.org are:
>
>         8.1.0, 6.4.0, 5.5.0, 4.9.4, 4.8.5, 4.6.1
>
> while the "dash and underscore" versions from kernel.org are:
>
>         7.3.0, 4.9.0, 4.8.0, 4.7.3, 4.6.3, 4.6.2, 4.5.1, 4.2.4
>
> This tweak allows the code to handle both versions. Note that this tweak also
> causes the architecture parsing to get confused and find the following two
> bogus architectures, "2.0" and "64", which are explicitly checked for, and
> removed.
>
> Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
> ---
>  tools/buildman/toolchain.py | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)

Reviewed-by: Simon Glass <sjg@chromium.org>

But please use single quotes for strings.

[U-Boot] [PATCH 3/3] buildman/toolchain.py: handle inconsistent tarball names

[Trevor Woerner]

On Mon, Nov 26, 2018 at 8:06 PM Simon Glass <sjg@chromium.org> wrote:

> Reviewed-by: Simon Glass <sjg@chromium.org>
> But please use single quotes for strings.
>

I can send a v2 if you like?

[U-Boot] [PATCH 3/3] buildman/toolchain.py: handle inconsistent tarball names

[Simon Glass]

Hi Trevor,

On Tue, 27 Nov 2018 at 17:51, Trevor Woerner <twoerner@gmail.com> wrote:
>
> On Mon, Nov 26, 2018 at 8:06 PM Simon Glass <sjg@chromium.org> wrote:
>>
>> Reviewed-by: Simon Glass <sjg@chromium.org>
>> But please use single quotes for strings.
>
>
> I can send a v2 if you like?

If you like, but I can fix it up when applying.

- Simon

[U-Boot] [PATCH 3/3] buildman/toolchain.py: handle inconsistent tarball names

[Trevor Woerner]

On Wed, Nov 28, 2018 at 2:38 PM Simon Glass <sjg@chromium.org> wrote:

> Hi Trevor,
>
> On Tue, 27 Nov 2018 at 17:51, Trevor Woerner <twoerner@gmail.com> wrote:
> >
> > On Mon, Nov 26, 2018 at 8:06 PM Simon Glass <sjg@chromium.org> wrote:
> >>
> >> Reviewed-by: Simon Glass <sjg@chromium.org>
> >> But please use single quotes for strings.
> >
> >
> > I can send a v2 if you like?
>
> If you like, but I can fix it up when applying.
>
>
Yes, please go ahead and fix it up.

[U-Boot] [PATCH 3/3] buildman/toolchain.py: handle inconsistent tarball names

[sjg at google.com]

On Wed, Nov 28, 2018 at 2:38 PM Simon Glass <sjg@chromium.org> wrote:

> Hi Trevor,
>
> On Tue, 27 Nov 2018 at 17:51, Trevor Woerner <twoerner@gmail.com> wrote:
> >
> > On Mon, Nov 26, 2018 at 8:06 PM Simon Glass <sjg@chromium.org> wrote:
> >>
> >> Reviewed-by: Simon Glass <sjg@chromium.org>
> >> But please use single quotes for strings.
> >
> >
> > I can send a v2 if you like?
>
> If you like, but I can fix it up when applying.
>
>
Yes, please go ahead and fix it up.

Applied to u-boot-dm/master, thanks!