Question about "Inconsistent kallsyms data"

Question about "Inconsistent kallsyms data"

[Kukjin Kim]

Hi all,

When build with s5pc100_defconfig, happens following.

...
KSYM    .tmp_kallsyms2.S
AS      .tmp_kallsyms2.o
LD      vmlinux
SYSMAP  System.map
SYSMAP  .tmp_System.map
Inconsistent kallsyms data
This is a bug - please report about it
Try make KALLSYMS_EXTRA_PASS=1 as a workaround
make[1]: *** [vmlinux] Error 1
make: *** [sub-make] Error 2

Its HEAD is following (current latest mainline)
(commit 55f9c40ff632d03c527d6a6ceddcda0a224587a6)
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc

Please kindly let me know how it should be fixed...

Thanks.

Best regards,
Kgene.
--
Kukjin Kim <kgene.kim@samsung.com>, Senior Engineer,
SW Solution Development Team, Samsung Electronics Co., Ltd.

Question about "Inconsistent kallsyms data"

[Tomasz Figa]

Hi,

On Friday 29 of July 2011 at 10:48:46, Kukjin Kim wrote:
> Hi all,
> 
> When build with s5pc100_defconfig, happens following.
> 
> ...
> KSYM    .tmp_kallsyms2.S
> AS      .tmp_kallsyms2.o
> LD      vmlinux
> SYSMAP  System.map
> SYSMAP  .tmp_System.map
> Inconsistent kallsyms data
> This is a bug - please report about it
> Try make KALLSYMS_EXTRA_PASS=1 as a workaround
> make[1]: *** [vmlinux] Error 1
> make: *** [sub-make] Error 2
> 
> Its HEAD is following (current latest mainline)
> (commit 55f9c40ff632d03c527d6a6ceddcda0a224587a6)
> Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc

I can confirm that it also happens when building with s3c6400_defconfig and 
HEAD at commit 5fd00b031530cc476240f654c078c930f1dcd6ea (Merge branch 'for-
linus' of git://git.kernel.org/pub/scm/linux/kernel/git/shaggy/jfs-2.6).

The suggested workaround (make KALLSYMS_EXTRA_PASS=1) helps, but it is only a 
workaround and not a solution.

Any suggestions?

Best regards,
Tomasz Figa

Question about "Inconsistent kallsyms data"

[Russell King - ARM Linux]

On Fri, Jul 29, 2011 at 12:00:07PM +0200, Tomasz Figa wrote:
> Hi,
> 
> On Friday 29 of July 2011 at 10:48:46, Kukjin Kim wrote:
> > Hi all,
> > 
> > When build with s5pc100_defconfig, happens following.
> > 
> > ...
> > KSYM    .tmp_kallsyms2.S
> > AS      .tmp_kallsyms2.o
> > LD      vmlinux
> > SYSMAP  System.map
> > SYSMAP  .tmp_System.map
> > Inconsistent kallsyms data
> > This is a bug - please report about it
> > Try make KALLSYMS_EXTRA_PASS=1 as a workaround
> > make[1]: *** [vmlinux] Error 1
> > make: *** [sub-make] Error 2
> > 
> > Its HEAD is following (current latest mainline)
> > (commit 55f9c40ff632d03c527d6a6ceddcda0a224587a6)
> > Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc
> 
> I can confirm that it also happens when building with s3c6400_defconfig and 
> HEAD at commit 5fd00b031530cc476240f654c078c930f1dcd6ea (Merge branch 'for-
> linus' of git://git.kernel.org/pub/scm/linux/kernel/git/shaggy/jfs-2.6).
> 
> The suggested workaround (make KALLSYMS_EXTRA_PASS=1) helps, but it is only a 
> workaround and not a solution.
> 
> Any suggestions?

Someone want to try bisecting to find the commit?

Question about "Inconsistent kallsyms data"

[Tomasz Figa]

On Friday 29 of July 2011 at 11:02:41, Russell King - ARM Linux wrote:
> On Fri, Jul 29, 2011 at 12:00:07PM +0200, Tomasz Figa wrote:
> > Hi,
> > 
> > On Friday 29 of July 2011 at 10:48:46, Kukjin Kim wrote:
> > > Hi all,
> > > 
> > > When build with s5pc100_defconfig, happens following.
> > > 
> > > ...
> > > KSYM    .tmp_kallsyms2.S
> > > AS      .tmp_kallsyms2.o
> > > LD      vmlinux
> > > SYSMAP  System.map
> > > SYSMAP  .tmp_System.map
> > > Inconsistent kallsyms data
> > > This is a bug - please report about it
> > > Try make KALLSYMS_EXTRA_PASS=1 as a workaround
> > > make[1]: *** [vmlinux] Error 1
> > > make: *** [sub-make] Error 2
> > > 
> > > Its HEAD is following (current latest mainline)
> > > (commit 55f9c40ff632d03c527d6a6ceddcda0a224587a6)
> > > Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc
> > 
> > I can confirm that it also happens when building with s3c6400_defconfig
> > and HEAD at commit 5fd00b031530cc476240f654c078c930f1dcd6ea (Merge
> > branch 'for- linus' of
> > git://git.kernel.org/pub/scm/linux/kernel/git/shaggy/jfs-2.6).
> > 
> > The suggested workaround (make KALLSYMS_EXTRA_PASS=1) helps, but it is
> > only a workaround and not a solution.
> > 
> > Any suggestions?
> 
> Someone want to try bisecting to find the commit?

I will try to bisect the case with s3c6400 defconfig.

Question about "Inconsistent kallsyms data"

[Tomasz Figa]

On Friday 29 of July 2011 at 12:58:32, Tomasz Figa wrote:
> On Friday 29 of July 2011 at 11:02:41, Russell King - ARM Linux wrote:
> > On Fri, Jul 29, 2011 at 12:00:07PM +0200, Tomasz Figa wrote:
> > > Hi,
> > > 
> > > On Friday 29 of July 2011 at 10:48:46, Kukjin Kim wrote:
> > > > Hi all,
> > > > 
> > > > When build with s5pc100_defconfig, happens following.
> > > > 
> > > > ...
> > > > KSYM    .tmp_kallsyms2.S
> > > > AS      .tmp_kallsyms2.o
> > > > LD      vmlinux
> > > > SYSMAP  System.map
> > > > SYSMAP  .tmp_System.map
> > > > Inconsistent kallsyms data
> > > > This is a bug - please report about it
> > > > Try make KALLSYMS_EXTRA_PASS=1 as a workaround
> > > > make[1]: *** [vmlinux] Error 1
> > > > make: *** [sub-make] Error 2
> > > > 
> > > > Its HEAD is following (current latest mainline)
> > > > (commit 55f9c40ff632d03c527d6a6ceddcda0a224587a6)
> > > > Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc
> > > 
> > > I can confirm that it also happens when building with
> > > s3c6400_defconfig
> > > and HEAD at commit 5fd00b031530cc476240f654c078c930f1dcd6ea (Merge
> > > branch 'for- linus' of
> > > git://git.kernel.org/pub/scm/linux/kernel/git/shaggy/jfs-2.6).
> > > 
> > > The suggested workaround (make KALLSYMS_EXTRA_PASS=1) helps, but it
> > > is
> > > only a workaround and not a solution.
> > > 
> > > Any suggestions?
> > 
> > Someone want to try bisecting to find the commit?
> 
> I will try to bisect the case with s3c6400 defconfig.

I have no idea why, but I cannot reproduce the issue anymore, even after make 
distclean or starting with a clean tree. A build system bug?

I do not know much technical details about the kernel build system, but might 
it be a concurrency issue (I use make -j5 for building with 5 jobs)?

Question about "Inconsistent kallsyms data"

[Arnd Bergmann]

On Friday 29 July 2011 10:48:46 Kukjin Kim wrote:
> Hi all,
> 
> When build with s5pc100_defconfig, happens following.
> 
> ...
> KSYM    .tmp_kallsyms2.S
> AS      .tmp_kallsyms2.o
> LD      vmlinux
> SYSMAP  System.map
> SYSMAP  .tmp_System.map
> Inconsistent kallsyms data
> This is a bug - please report about it
> Try make KALLSYMS_EXTRA_PASS=1 as a workaround
> make[1]: *** [vmlinux] Error 1
> make: *** [sub-make] Error 2
> 
> Its HEAD is following (current latest mainline)
> (commit 55f9c40ff632d03c527d6a6ceddcda0a224587a6)
> Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc
> 
> Please kindly let me know how it should be fixed...

[Taking Michal and linux-kbuild on cc]

I've seen this before with some of my ARM randconfig builds and
spent some time debugging into it, without success.

Can you build with KALLSYMS_EXTRA_PASS=1 and send the diff between
.tmp_kallsyms2.S and .tmp_kallsyms3.S?

	Arnd

Question about "Inconsistent kallsyms data"

[Arnd Bergmann]

On Friday 29 July 2011 13:30:20 Tomasz Figa wrote:
> 
> I have no idea why, but I cannot reproduce the issue anymore, even after make 
> distclean or starting with a clean tree. A build system bug?
> 
> I do not know much technical details about the kernel build system, but might 
> it be a concurrency issue (I use make -j5 for building with 5 jobs)?

I could reproduce it on some systems, but it changes when you switch compiler
versions or other flags, or the exact kernel version, so it's impossible
to bisect if you have the same problem that I saw.

I got all the way to the point where I could add a nop instruction in one
file and it would go away because the code size grew across some magic
number.

	Arnd

Question about "Inconsistent kallsyms data"

[Paulo Marques]

Tomasz Figa wrote:
> On Friday 29 of July 2011 at 12:58:32, Tomasz Figa wrote:
>>[...]
>> I will try to bisect the case with s3c6400 defconfig.
> 
> I have no idea why, but I cannot reproduce the issue anymore, even after make 
> distclean or starting with a clean tree. A build system bug?
> 
> I do not know much technical details about the kernel build system, but might 
> it be a concurrency issue (I use make -j5 for building with 5 jobs)?

Usually this is caused by symbols moving slightly between kernel compiles.

This is an explanation of the process I wrote a while ago in a different
thread at LKML:

> Now as for CONFIG_KALLSYMS_EXTRA_PASS: to build the kallsyms table, the
> build process first links a kernel image with an empty kallsyms table
> and use that to fetch information for all the symbols.
> 
> It then uses that information to build the table with the right size,
> and links it again. If everything goes ok, this new version as all the
> symbols in the correct places and the final table can be built with the
> correct addresses.
> 
> The final linking should produce the same result as only the data on the
> kallsyms table changed, but not its size.
> 
> However, there have been bugs in the past with section alignments and
> symbol reordering for symbols with the same address, etc., etc. that
> make this final table not have the exact same size, and the build fails
> with an inconsistent kallsyms data message. At this point, the user can
> turn on the CONFIG_KALLSYMS_EXTRA_PASS and temporarily solve the problem
> while the developers find the correct fix. Without this option, in this
> situation the kernel would simply fail the compilation.
> 
> All this has been stable for a while and this option hasn't been needed
> recently (AFAIK), but if there is some bug in some new binutils or
> something, the option might be needed again.

To check inconsistencies, the Makefile compares ".tmp_System.map" and
"System.map" for differences.

On a normal build they should be identical, but if this fails with
"Inconsistent kallsyms data" you could try comparing the two files to
check which symbols are causing trouble.

-- 
Paulo Marques - www.grupopie.com

"All generalizations are false."

Question about "Inconsistent kallsyms data"

[Eric Miao]

On Wed, Aug 3, 2011 at 12:29 AM, Paulo Marques <pmarques@grupopie.com> wrote:
> Tomasz Figa wrote:
>> On Friday 29 of July 2011 at 12:58:32, Tomasz Figa wrote:
>>>[...]
>>> I will try to bisect the case with s3c6400 defconfig.
>>
>> I have no idea why, but I cannot reproduce the issue anymore, even after make
>> distclean or starting with a clean tree. A build system bug?
>>
>> I do not know much technical details about the kernel build system, but might
>> it be a concurrency issue (I use make -j5 for building with 5 jobs)?
>
> Usually this is caused by symbols moving slightly between kernel compiles.
>
> This is an explanation of the process I wrote a while ago in a different
> thread at LKML:
>
>> Now as for CONFIG_KALLSYMS_EXTRA_PASS: to build the kallsyms table, the
>> build process first links a kernel image with an empty kallsyms table
>> and use that to fetch information for all the symbols.
>>
>> It then uses that information to build the table with the right size,
>> and links it again. If everything goes ok, this new version as all the
>> symbols in the correct places and the final table can be built with the
>> correct addresses.
>>
>> The final linking should produce the same result as only the data on the
>> kallsyms table changed, but not its size.
>>
>> However, there have been bugs in the past with section alignments and
>> symbol reordering for symbols with the same address, etc., etc. that
>> make this final table not have the exact same size, and the build fails
>> with an inconsistent kallsyms data message. At this point, the user can
>> turn on the CONFIG_KALLSYMS_EXTRA_PASS and temporarily solve the problem
>> while the developers find the correct fix. Without this option, in this
>> situation the kernel would simply fail the compilation.
>>
>> All this has been stable for a while and this option hasn't been needed
>> recently (AFAIK), but if there is some bug in some new binutils or
>> something, the option might be needed again.
>
> To check inconsistencies, the Makefile compares ".tmp_System.map" and
> "System.map" for differences.
>
> On a normal build they should be identical, but if this fails with
> "Inconsistent kallsyms data" you could try comparing the two files to
> check which symbols are causing trouble.

I met this issue today as well. Didn't have a chance to compare the
files, as I did a 'rm -fr ../build' and reconfigured/re-built everything,
and 'make O=../build clean' didn't work for me.

I'm using a toolchain with version info as below:

$ arm-linux-gnueabi-gcc -v
Using built-in specs.
COLLECT_GCC=arm-linux-gnueabi-gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabi/4.5.2/lto-wrapper
Target: arm-linux-gnueabi
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro
4.5.2-8ubuntu3'
--with-bugurl=file:///usr/share/doc/gcc-4.5/README.Bugs
--enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr
--program-suffix=-4.5 --enable-shared --enable-multiarch
--with-multiarch-defaults=i386-linux-gnu --enable-linker-build-id
--with-system-zlib --libexecdir=/usr/lib --without-included-gettext
--enable-threads=posix
--with-gxx-include-dir=/usr/arm-linux-gnueabi/include/c++/4.5.2
--libdir=/usr/lib --enable-nls --enable-clocale=gnu
--enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin
--enable-gold --enable-ld=default --with-plugin-ld=ld.gold
--enable-objc-gc --disable-sjlj-exceptions --with-arch=armv7-a
--with-float=softfp --with-fpu=vfpv3-d16 --with-mode=thumb
--disable-werror --enable-checking=release
--program-prefix=arm-linux-gnueabi-
--includedir=/usr/arm-linux-gnueabi/include --build=i686-linux-gnu
--host=i686-linux-gnu --target=arm-linux-gnueabi
--with-headers=/usr/arm-linux-gnueabi/include
--with-libs=/usr/arm-linux-gnueabi/lib
Thread model: posix
gcc version 4.5.2 (Ubuntu/Linaro 4.5.2-8ubuntu3)


>
> --
> Paulo Marques - www.grupopie.com
>
> "All generalizations are false."
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>

Question about "Inconsistent kallsyms data"

[Arnd Bergmann]

On Saturday 08 October 2011 18:17:49 Eric Miao wrote:
> >
> > To check inconsistencies, the Makefile compares ".tmp_System.map" and
> > "System.map" for differences.
> >
> > On a normal build they should be identical, but if this fails with
> > "Inconsistent kallsyms data" you could try comparing the two files to
> > check which symbols are causing trouble.
> 
> I met this issue today as well. Didn't have a chance to compare the
> files, as I did a 'rm -fr ../build' and reconfigured/re-built everything,
> and 'make O=../build clean' didn't work for me.

I can easily reproduce it here, but it's extremely hard to nail
down what is actually going on.

For all I know, this depends on the specific configuration, source
version and gcc version. If you change any of these in any meaningful
way, the problem can go away or reappear. In one case, I've been
able to get to the point where inserting a single 'nop' instruction
in one (any) source file triggered the problem. This happens to be
the point when one of the linker sections grows beyond some power-of-two
boundary.

If anyone wants to reproduce it, I can recommend taking a fast machine
and the for-next+randconfig branch of git://git.linaro.org/people/arnd/arm-soc.git
There is a script called randconfig.sh in it. Remove the KALLSYMS_EXTRA_PASS=1
line from it and run it for some time. After a few hours you will get
multiple configurations for your compiler that show the problem.

I've had one case where even with KALLSYMS_EXTRA_PASS=1 I was still getting
inconsistent kallsyms data. Manually changing the Makefile to do a fourth
run fixed it.

	Arnd