binutils woes

binutils woes

[Russell King]

Hi guys,

On ARM, we appear to have somewhat of a problem with binutils.  At
least the following binutils suffer from a problem whereby it is
possible to create programs which contain undefined symbols:

GNU assembler 2.13.90.0.18 20030206
GNU assembler 2.14 20030612
GNU assembler 2.14.90 20031229
GNU ld version 2.14.90.0.7 20031029 Debian GNU/Linux
Assembleur GNU 2.15.90.0.1 20040303

We discovered this back in April, and at the time we thought that
adding --no-undefined to the linker command line would ensure that
we caught this bug.

However, this appears not to be the case.  I recently committed
this change:

  http://linux.bkbits.net:8080/linux-2.5/cset@1.1769.3.12??nav=index.html|ChangeSet@-4d

and tested it - it appeared to build fine, and nothing untoward.
However, as you can see from this patch:

  http://www.arm.linux.org.uk/developer/patches/viewpatch.php?id=1950/1

It appears that we are successfully build a kernel without the correct
value of SIZEOF_MACHINE_DESC - in fact, this constant appears to be
undefined in arch/arm/kernel/head.S.  Upon checking the build logs
and object files here, I have been able to confirm that this is
exactly what has happened.

Here is an example of the build output around the affected files:

...
  AS      arch/arm/kernel/debug.o
  LD      arch/arm/kernel/built-in.o
  AS      arch/arm/kernel/head.o
  CC      arch/arm/kernel/init_task.o
...
  CC      arch/arm/lib/udivdi3.o
  AR      arch/arm/lib/lib.a
  GEN     .version
  CHK     include/linux/compile.h
  UPD     include/linux/compile.h
  CC      init/version.o
  LD      init/built-in.o
  LD      .tmp_vmlinux1
  KSYM    .tmp_kallsyms1.S
  AS      .tmp_kallsyms1.o
  LD      .tmp_vmlinux2
  KSYM    .tmp_kallsyms2.S
  AS      .tmp_kallsyms2.o
  LD      vmlinux
  OBJCOPY arch/arm/boot/Image
  Kernel: arch/arm/boot/Image is ready

As you can see - there is no indication of failure what so ever.
However, if we inspect the symbol information for vmlinux, we find:

[rmk@dyn-67 versatile]$ arm-linux-nm vmlinux | grep SIZEOF -2
c0008000 T _sinittext
c0060634 T si_swapinfo
         U SIZEOF_MACHINE_DESC
c013cb7c T sk_alloc
c014cc34 T sk_attach_filter

and further investigation of the disassembly indicates that the
assembler subsituted a value of zero for this constant.

Since ld --no-undefined does not catch this error, and this error
affects a range of binutils versions from 2.13 through to 2.15, I
don't think we can ignore the problem.

I think the only way we can ensure kernel correctness is to add a
subsequent stage to kbuild such that whenever we generate a final
program, we grep the 'nm' output for undefined symbols.

Comments?

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Re: binutils woes

[Tom Rini]

On Thu, Jul 01, 2004 at 05:52:31PM +0100, Russell King wrote:

> Hi guys,
> 
> On ARM, we appear to have somewhat of a problem with binutils.  At
> least the following binutils suffer from a problem whereby it is
> possible to create programs which contain undefined symbols:
[snip]
> I think the only way we can ensure kernel correctness is to add a
> subsequent stage to kbuild such that whenever we generate a final
> program, we grep the 'nm' output for undefined symbols.
> 
> Comments?

Is there a version of binutils that really does get things right?  If
so, can't you Just Say No to older versions and force people to upgrade
(with a simple testcase done upfront) ?

-- 
Tom Rini
http://gate.crashing.org/~trini/

Re: binutils woes

[Russell King]

On Thu, Jul 01, 2004 at 10:47:31AM -0700, Tom Rini wrote:
> On Thu, Jul 01, 2004 at 05:52:31PM +0100, Russell King wrote:
> 
> > Hi guys,
> > 
> > On ARM, we appear to have somewhat of a problem with binutils.  At
> > least the following binutils suffer from a problem whereby it is
> > possible to create programs which contain undefined symbols:
> [snip]
> > I think the only way we can ensure kernel correctness is to add a
> > subsequent stage to kbuild such that whenever we generate a final
> > program, we grep the 'nm' output for undefined symbols.
> > 
> > Comments?
> 
> Is there a version of binutils that really does get things right?  If
> so, can't you Just Say No to older versions and force people to upgrade
> (with a simple testcase done upfront) ?

I've just tested:

 GNU assembler 2.15.90 20040409
 Copyright 2002 Free Software Foundation, Inc.
 This program is free software; you may redistribute it under the terms of
 the GNU General Public License.  This program has absolutely no warranty.
 This assembler was configured for a target of `arm-linux'.

and it also is aflicted with the same problem.  I guess the answer
to your question is that currently there are no recent versions which
work correctly.

Looking at the timescale of 2.13 through to present day, we're talking
potentially of 2 years of binutils versions with this problem, and even
if it was fixed tomorrow, I don't think it's acceptable to say "use the
latest binutils version."

Also, I have just received another report that somewhere in the kernel
is an undefined reference to another symbol - I_BIT, but where we have
no idea at present...

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Re: binutils woes

[Martin Schlemmer]

[-- Attachment #1: Type: text/plain, Size: 1595 bytes --]

On Thu, 2004-07-01 at 20:07, Russell King wrote:
> On Thu, Jul 01, 2004 at 10:47:31AM -0700, Tom Rini wrote:
> > On Thu, Jul 01, 2004 at 05:52:31PM +0100, Russell King wrote:
> > 
> > > Hi guys,
> > > 
> > > On ARM, we appear to have somewhat of a problem with binutils.  At
> > > least the following binutils suffer from a problem whereby it is
> > > possible to create programs which contain undefined symbols:
> > [snip]
> > > I think the only way we can ensure kernel correctness is to add a
> > > subsequent stage to kbuild such that whenever we generate a final
> > > program, we grep the 'nm' output for undefined symbols.
> > > 
> > > Comments?
> > 
> > Is there a version of binutils that really does get things right?  If
> > so, can't you Just Say No to older versions and force people to upgrade
> > (with a simple testcase done upfront) ?
> 
> I've just tested:
> 
>  GNU assembler 2.15.90 20040409
>  Copyright 2002 Free Software Foundation, Inc.
>  This program is free software; you may redistribute it under the terms of
>  the GNU General Public License.  This program has absolutely no warranty.
>  This assembler was configured for a target of `arm-linux'.
> 

You might try (which should be on any kernel mirror):

GNU assembler 2.15.91.0.1 20040527
Copyright 2002 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License.  This program has absolutely no warranty.
This assembler was configured for a target of `i686-pc-linux-gnu'.


-- 
Martin Schlemmer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: binutils woes

[Russell King]

On Thu, Jul 01, 2004 at 09:44:10PM +0200, Martin Schlemmer wrote:
> You might try (which should be on any kernel mirror):
> 
> GNU assembler 2.15.91.0.1 20040527
> Copyright 2002 Free Software Foundation, Inc.
> This program is free software; you may redistribute it under the terms of
> the GNU General Public License.  This program has absolutely no warranty.
> This assembler was configured for a target of `i686-pc-linux-gnu'.

I don't think there's any point in trying - I just received a mail
from Nick Clifton who has found the problem in the ARM backend of
GAS and is apparantly in the process of committing it to CVS.

So, all ARM binutils versions from 2002 to date are affected by
the bug.

Whether ld --no-undefined successfully producing binaries with
undefined symbols is a problem or not remains unclear, however.

Even so, we still can not check whether we are actually using a late
enough version of binutils and force kernel build failure on that
basis.  About the best we can do is as I originally suggested which
is to add a postprocessing step after we link a binary object to
ensure there are no undefined symbols remaining.

The alternative is we just say "Sorry, we're not prepared to provide
any form of support for problems reported for ARM builds until the
next stable release of binutils."  Realistically, I don't think
that's an acceptable position to take.

Therefore, unless anyone has any objections, I shall be cooking up
a patch which adds an extra pass to any final object link for the
kernel build system.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Re: binutils woes

[Russell King]

On Thu, Jul 01, 2004 at 08:52:55PM +0100, Russell King wrote:
> Therefore, unless anyone has any objections, I shall be cooking up
> a patch which adds an extra pass to any final object link for the
> kernel build system.

Ok, I decided that touching the rules used for the boot loader(s) was
probably going to be too fraught, so here's one which just checks the
final link for the main kernel image.

Essentially, we run 'nm' against the object, and look for any line
which matches the pattern '^ *U '.  With this, a failing output looks
like:

  LD      init/built-in.o
  LD      .tmp_vmlinux1
ldchk: .tmp_vmlinux1: final image has undefined symbols:
  SIZEOF_MACHINE_DESC
make[1]: *** [.tmp_vmlinux1] Error 1
make: *** [vmlinux] Error 2

and successful output:

  LD      init/built-in.o
  LD      .tmp_vmlinux1
  KSYM    .tmp_kallsyms1.S
  AS      .tmp_kallsyms1.o
  LD      .tmp_vmlinux2
  KSYM    .tmp_kallsyms2.S
  AS      .tmp_kallsyms2.o
  LD      vmlinux


===== Makefile 1.500 vs edited =====
--- 1.500/Makefile	Tue Jun 29 15:44:49 2004
+++ edited/Makefile	Thu Jul  1 23:10:04 2004
@@ -533,6 +533,8 @@
 endef
 
 #	set -e makes the rule exit immediately on error
+#	Note: Ensure that there are no undefined symbols in the final
+#	linked image.  Not doing this can lead to silent link failures.
 
 define rule_vmlinux__
 	+set -e;							\
@@ -545,6 +547,12 @@
 	$(if $($(quiet)cmd_vmlinux__),					\
 	  echo '  $($(quiet)cmd_vmlinux__)' &&) 			\
 	$(cmd_vmlinux__);						\
+	if $(NM) $@ | grep -q '^ *U '; then				\
+		echo 'ldchk: $@: final image has undefined symbols:';	\
+		$(NM) $@ | sed 's/^ *U \(.*\)/  \1/p;d';		\
+		$(RM) -f $@;						\
+		exit 1;							\
+	fi;								\
 	echo 'cmd_$@ := $(cmd_vmlinux__)' > $(@D)/.$(@F).cmd
 endef
 


-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Re: binutils woes

[Andrew Morton]

Russell King <rmk+lkml@arm.linux.org.uk> wrote:
>
> Essentially, we run 'nm' against the object, and look for any line
> which matches the pattern '^ *U '.  With this, a failing output looks
> like:
> 
>   LD      init/built-in.o
>   LD      .tmp_vmlinux1
> ldchk: .tmp_vmlinux1: final image has undefined symbols:
>   SIZEOF_MACHINE_DESC

What is the user supposed to do when this happens?

Re: binutils woes

[Russell King]

On Thu, Jul 01, 2004 at 03:21:34PM -0700, Andrew Morton wrote:
> Russell King <rmk+lkml@arm.linux.org.uk> wrote:
> >
> > Essentially, we run 'nm' against the object, and look for any line
> > which matches the pattern '^ *U '.  With this, a failing output looks
> > like:
> > 
> >   LD      init/built-in.o
> >   LD      .tmp_vmlinux1
> > ldchk: .tmp_vmlinux1: final image has undefined symbols:
> >   SIZEOF_MACHINE_DESC
> 
> What is the user supposed to do when this happens?

Report it as an error to the most appropriate forum.  Then it becomes
a hunt for why the symbol is undefined.

Sorry, afaics it's not possible to do this any better.

To illustrate the severity of the problem better, the following
program:

        .globl  _start
_start:
        add     r0, r0, #UNDEFINED_SYMBOL
        mov     pc, lr

will successfully assemble and link on ARM, and is in fact runnable
despite having a symbol table of the following:

$ arm-linux-nm t
00008074 t $a
0001007c A __bss_end__
0001007c A _bss_end__
0001007c A __bss_start
0001007c A __bss_start__
0001007c D __data_start
0001007c A _edata
0001007c A _end
0001007c A __end__
00008074 T _start
         U UNDEFINED_SYMBOL


-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Re: binutils woes

[Michael Buesch]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoting Russell King <rmk+lkml@arm.linux.org.uk>:
> GNU ld version 2.14.90.0.7 20031029 Debian GNU/Linux

Is there a chance that this screws up binaries for x86, too?
I'm having various segfault problems at the moment. Maybe I should
update.

- --
Regards Michael Buesch  [ http://www.tuxsoft.de.vu ]


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFA5T9QFGK1OIvVOP4RAnraAJ0eODTOfojvwKfwISmbK4ZSRlD2NgCgj7o7
QV61gaI87O0K7xBbzcLf5Sc=
=bKrH
-----END PGP SIGNATURE-----

Re: binutils woes

[Russell King]

On Thu, Jul 01, 2004 at 05:52:31PM +0100, Russell King wrote:
> I think the only way we can ensure kernel correctness is to add a
> subsequent stage to kbuild such that whenever we generate a final
> program, we grep the 'nm' output for undefined symbols.
...

BTW, binutils 2.15 on ARM has other issues, which I've reported to
Nick Clifton/binutils-bug.  The assembler seems to produce extra
symbolic information in the form of symbols starting with '$', eg:

c0067860 T __fput
c0067860 t $a
c0067954 t $d
c0067958 T fget
c0067958 t $a

Not only do these symbols interfere with ld's error reporting, but
they also interfere with kallsyms - the backtrace information from
such a kernel is not very useful when it reports function names of
'$a':

drivers/built-in.o(.text+0x6042c): In function `$a':
: relocation truncated to fit: R_ARM_PC24 .exit.text

Obviously not useful to anyone.

This seems to only affect binutils 2.15, so I think we should hang
fire on any work-arounds until the binutils folk have commented.

-- 
Russell King
 ... who dearly wishes that the kernel build was still compatible
  with the good old stable binutils 2.11 versions.

Re: binutils woes

[Russell King]

On Thu, Jul 01, 2004 at 05:52:31PM +0100, Russell King wrote:
> On ARM, we appear to have somewhat of a problem with binutils.  At
> least the following binutils suffer from a problem whereby it is
> possible to create programs which contain undefined symbols:

Ok, here's the latest version of the patch.  It shouldn't complain
on Sparc64, and neither should it complain while linking the
.tmp_vmlinux* objects - the original regexp on the objdump --syms
output was catching the weak kallsyms symbols.

The only issue with this is that, when a problem is detected, the
reported symbols will also include the Sparc64 register symbols.

===== Makefile 1.500 vs edited =====
--- 1.500/Makefile	Tue Jun 29 15:44:49 2004
+++ edited/Makefile	Sun Jul 11 14:52:37 2004
@@ -534,6 +534,8 @@
 
 #	set -e makes the rule exit immediately on error
 
+#	Note: Ensure that there are no undefined symbols in the final
+#	linked image.  Not doing this can lead to silent link failures.
 define rule_vmlinux__
 	+set -e;							\
 	$(if $(filter .tmp_kallsyms%,$^),,				\
@@ -545,6 +547,12 @@
 	$(if $($(quiet)cmd_vmlinux__),					\
 	  echo '  $($(quiet)cmd_vmlinux__)' &&) 			\
 	$(cmd_vmlinux__);						\
+	if $(OBJDUMP) --syms $@ | egrep -q '^([^R]|R[^E]|RE[^G])[^w]*\*UND\*'; then	\
+		echo 'ldchk: $@: final image has undefined symbols:';	\
+		$(NM) $@ | sed 's/^ *U \(.*\)/  \1/p;d';		\
+		$(RM) -f $@;						\
+		exit 1;							\
+	fi;								\
 	echo 'cmd_$@ := $(cmd_vmlinux__)' > $(@D)/.$(@F).cmd
 endef
 


-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core