Re: Heads up Linux 2.6.38-rc4 compile problems.

[ebiederm@xmission.com (Eric W. Biederman)]

[-- Attachment #1: Type: text/plain, Size: 10199 bytes --]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> On Tue, Feb 8, 2011 at 12:28 PM, Eric W. Biederman
> <ebiederm@xmission.com> wrote:
>>
>> A quick heads up.  2.6.38-rc4 looks like the worst kernel I've ever tried to
>> test.  It boots up properly and looks ok, but I can't get it to even
>> compile the programs I usually test with. 2.6.38-rc3 at least managed
>> that yesterday.
>
> gcc dying with ICE or SIGSEGV? Or what? Seriously lacking information
> here.
>
> Are you using btrfs or cifs (the two filesystems that had
> bigger-than-average changes)?
>
> Any chance to bisect it (even if just partially - a couple of
> bisection runs would already narrow it down quite a bit)?

What I have been able to do today on this one is, narrow it down a
little and provide a reasonable description of one of the problem.

In a fedora 12 world in a subdirectory on ext4 with the applications
pinned into a mount namespace.  I am seeing compiles of gcc-4.4.3 fail.  A
quick survey of the compile failures makes it look like these compiles
failures are deterministic, at least I see the same error messages in multiple
failed builds.

The machine is a dual socket quad core non hyperthread machine with 10G of RAM.
The cpus are: Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz

In the case that fails I am simultaneously compiling a 32bit gcc cross
compiler for x86_64 and i386, for the same source but with different
object directories.  The compiles failures happen exactly as before if I
rerun the build commands.  I'm still trying to figure out why the
individual files are failing.  I expect if I do that will give me a much
bigger clue.

What is interesting is that this fails exactly the same way in at least three invocations
of the command and I expect all of them.  With failures taking about 30 minutes.
I might try a bisect tomorrow.

The unfortunate thing is that this weird compile failure during a
parallel build doesn't tell me much yet about what the root cause is.

I have attached my build scripts in case that helps.  To understand.
At the moment I am still trying to break this down into something smaller
so I can see just what the heck is going on.  And I will keep poking at
it until I get somewhere.

The annoying thing is that looking at the other failures I was seeing
before the compile fell apart.  I think I am might be on top of half a
dozen other regressions as well.  I don't have a clue how I am going to
get through all of those regressions before before 2.6.38 is out. 

Eric

The failures look like:

$ make -C gcc-stage1-x86_64 all-gcc all-target-libgcc install-gcc install-target-libgcc DESTDIR=/bld/Across/Across-2.0.0/sysroot-x86_64
gcc -qlanglvl=ansi -c -DHAVE_CONFIG_H -g -O2 -I. -I/bld/Across/Across-2.0.0/gcc-4.4.3/libiberty/../include  -W -Wall
-Wwrite-strings -Wc++-compat -Wstrict-prototypes /bld/Across/Across-2.0.0/gcc-4.4.3/libiberty/fibheap.c -o fibheap.o
gcc: unrecognized option '-qlanglvl=ansi'
/bld/Across/Across-2.0.0/gcc-4.4.3/libiberty/fibheap.c: In function ‘fibheap_union’:
/bld/Across/Across-2.0.0/gcc-4.4.3/libiberty/fibheap.c:151: warning: implicit declaration of function ‘free’
/bld/Across/Across-2.0.0/gcc-4.4.3/libiberty/fibheap.c:151: warning: incompatible implicit declaration of built-in function ‘free’
/bld/Across/Across-2.0.0/gcc-4.4.3/libiberty/fibheap.c:156: warning: incompatible implicit declaration of built-in function ‘free’
/bld/Across/Across-2.0.0/gcc-4.4.3/libiberty/fibheap.c:172: warning: incompatible implicit declaration of built-in function ‘free’
/bld/Across/Across-2.0.0/gcc-4.4.3/libiberty/fibheap.c: In function ‘fibheap_extract_min’:
/bld/Across/Across-2.0.0/gcc-4.4.3/libiberty/fibheap.c:190: warning: incompatible implicit declaration of built-in function ‘free’
/bld/Across/Across-2.0.0/gcc-4.4.3/libiberty/fibheap.c: In function ‘fibheap_delete_node’:
/bld/Across/Across-2.0.0/gcc-4.4.3/libiberty/fibheap.c:258: error: ‘LONG_MIN’ undeclared (first use in this function)
/bld/Across/Across-2.0.0/gcc-4.4.3/libiberty/fibheap.c:258: error: (Each undeclared identifier is reported only once
/bld/Across/Across-2.0.0/gcc-4.4.3/libiberty/fibheap.c:258: error: for each function it appears in.)
/bld/Across/Across-2.0.0/gcc-4.4.3/libiberty/fibheap.c: In function ‘fibheap_delete’:
/bld/Across/Across-2.0.0/gcc-4.4.3/libiberty/fibheap.c:269: warning: incompatible implicit declaration of built-in function ‘free’
/bld/Across/Across-2.0.0/gcc-4.4.3/libiberty/fibheap.c: In function ‘fibheap_consolidate’:
/bld/Across/Across-2.0.0/gcc-4.4.3/libiberty/fibheap.c:360: warning: implicit declaration of function ‘memset’
/bld/Across/Across-2.0.0/gcc-4.4.3/libiberty/fibheap.c:360: warning: incompatible implicit declaration of built-in function ‘memset’


$ make -C gcc-stage1-i386 all-gcc all-target-libgcc install-gcc install-target-libgcc DESTDIR=/bld/Across/Across-2.0.0/sysroot-i386
make: Entering directory `/bld/Across/Across-2.0.0/gcc-stage1-i386'
make[1]: Entering directory `/bld/Across/Across-2.0.0/gcc-stage1-i386/libiberty'
make[2]: Entering directory `/bld/Across/Across-2.0.0/gcc-stage1-i386/libiberty/testsuite'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/bld/Across/Across-2.0.0/gcc-stage1-i386/libiberty/testsuite'
make[1]: Leaving directory `/bld/Across/Across-2.0.0/gcc-stage1-i386/libiberty'
make[1]: Entering directory `/bld/Across/Across-2.0.0/gcc-stage1-i386/intl'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/bld/Across/Across-2.0.0/gcc-stage1-i386/intl'
make[1]: Entering directory `/bld/Across/Across-2.0.0/gcc-stage1-i386/build-i686-pc-linux-gnu/libiberty'
make[2]: Entering directory `/bld/Across/Across-2.0.0/gcc-stage1-i386/build-i686-pc-linux-gnu/libiberty/testsuite'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/bld/Across/Across-2.0.0/gcc-stage1-i386/build-i686-pc-linux-gnu/libiberty/testsuite'
make[1]: Leaving directory `/bld/Across/Across-2.0.0/gcc-stage1-i386/build-i686-pc-linux-gnu/libiberty'
make[1]: Entering directory `/bld/Across/Across-2.0.0/gcc-stage1-i386/build-i686-pc-linux-gnu/fixincludes'
gcc -g -O2  -o fixincl fixincl.o fixtests.o fixfixes.o server.o procopen.o fixlib.o fixopts.o ../libiberty/libiberty.a
fixincl.o: In function `quoted_file_exists':
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixincl.c:624: undefined reference to `_sch_istable'
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixincl.c:624: undefined reference to `_sch_istable'
fixincl.o: In function `egrep_test':
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixincl.c:600: undefined reference to `xregexec'
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixincl.c:600: undefined reference to `xregexec'
fixincl.o: In function `test_for_changes':
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixincl.c:1211: undefined reference to `xregexec'
fixincl.o: In function `extract_quoted_files':
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixincl.c:711: undefined reference to `xregexec'
fixincl.o: In function `initialize':
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixincl.c:245: undefined reference to `_sch_istable'
fixincl.o: In function `main':
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixincl.c:139: undefined reference to `_sch_istable'
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixincl.c:157: undefined reference to `_sch_istable'
fixtests.o: In function `machine_name_test':
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixtests.c:79: undefined reference to `xregexec'
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixtests.c:104: undefined reference to `xregexec'
fixfixes.o: In function `gnu_type_fix':
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixfixes.c:681: undefined reference to `xregexec'
fixfixes.o: In function `emit_gnu_type':
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixfixes.c:144: undefined reference to `_sch_toupper'
fixfixes.o: In function `wrap_fix':
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixfixes.c:599: undefined reference to `xregexec'
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixfixes.c:617: undefined reference to `_sch_istable'
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixfixes.c:621: undefined reference to `_sch_toupper'
fixfixes.o: In function `machine_name_fix':
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixfixes.c:508: undefined reference to `xregexec'
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixfixes.c:539: undefined reference to `xregexec'
fixfixes.o: In function `format_fix':
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixfixes.c:282: undefined reference to `xregexec'
fixfixes.o: In function `format_write':
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixfixes.c:198: undefined reference to `_sch_istable'
fixfixes.o: In function `char_macro_use_fix':
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixfixes.c:332: undefined reference to `xregexec'
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixfixes.c:439: undefined reference to `_sch_istable'
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixfixes.c:448: undefined reference to `_sch_istable'
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixfixes.c:451: undefined reference to `_sch_istable'
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixfixes.c:453: undefined reference to `_sch_istable'
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixfixes.c:448: undefined reference to `_sch_istable'
server.o:/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/server.c:124: more undefined references to `_sch_istable' follow
fixlib.o: In function `compile_re':
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixlib.c:193: undefined reference to `xregcomp'
/bld/Across/Across-2.0.0/gcc-4.4.3/fixincludes/fixlib.c:198: undefined reference to `xregerror'
collect2: ld returned 1 exit status
make[1]: *** [full-stamp] Error 1
make[1]: Leaving directory `/bld/Across/Across-2.0.0/gcc-stage1-i386/build-i686-pc-linux-gnu/fixincludes'
make: *** [all-build-fixincludes] Error 2
make: Leaving directory `/bld/Across/Across-2.0.0/gcc-stage1-i386'




[-- Attachment #2: Across.spec --]
[-- Type: text/plain, Size: 2117 bytes --]

Name:           Across
Version:        2.0.0
Release:        %{?release:%{release}}%{!?release:eng}
Summary:        uClibc-based development toolchain

Group:          Development/Tools
License:        GPL
Source0:        uClibc-0.9.30.2.tar.bz2
Source1:        binutils-2.20.tar.bz2
Source2:        gcc-core-4.4.3.tar.bz2
Source3:        gmp-4.3.2.tar.bz2
Source4:        mpfr-2.4.2.tar.bz2
Source5:        kernel-headers-2.6.29.6.tar.bz2
Source10:       build.sh
Source11:       uclibc-i386.config
Source12:       uclibc-x86_64.config
BuildRoot:      %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)

BuildRequires:  quilt

# Disable automatic dependency processing: we don't want to override
# standard system libraries like libresolv.so.0
AutoReqProv:    no

# Disable automatic debuginfo package generation and other post-build
# processing, because it does not handle uClibc binaries
%define __spec_install_post %{nil}
%define debug_package %{nil}

%description
This package provides development tools based on the uClibc library, a
space-efficient replacement for glibc.

%prep
%setup -q -c -T -a 0 -a 1 -a 2 -a 3 -a 4 -a 5
ln -s %{_sourcedir} patches
#echo %{PATCHURL1} | tr ' ' '\n' >series
#quilt push -a
cp %{SOURCE10} %{SOURCE11} %{SOURCE12} .
chmod +x build.sh

%build
%{?a4_configure:exit 0}
( cd gmp-4.3.2; ./configure )
make -C gmp-4.3.2 all
( cd mpfr-2.4.2; ./configure --with-gmp-build=../gmp-4.3.2 )
make -C mpfr-2.4.2 all
./build.sh i386 $(pwd)/sysroot-i386 $(pwd)/uClibc-0.9.30.2 $(pwd)/binutils-2.20 $(pwd)/gcc-4.4.3 $(pwd)/gmp-4.3.2 $(pwd)/mpfr-2.4.2 $(pwd)/kernel-headers-2.6.29.6/include &
./build.sh x86_64 $(pwd)/sysroot-x86_64 $(pwd)/uClibc-0.9.30.2 $(pwd)/binutils-2.20 $(pwd)/gcc-4.4.3 $(pwd)/gmp-4.3.2 $(pwd)/mpfr-2.4.2 $(pwd)/kernel-headers-2.6.29.6/include &
wait

%install
rm -rf $RPM_BUILD_ROOT
mkdir -p $RPM_BUILD_ROOT/usr/share/Across
cp -alv sysroot-i386 $RPM_BUILD_ROOT/usr/share/Across/i386-linux-uclibc
cp -alv sysroot-x86_64 $RPM_BUILD_ROOT/usr/share/Across/x86_64-linux-uclibc

%clean
rm -rf $RPM_BUILD_ROOT

%files
%defattr(-,root,root,-)
/usr/share/Across/*

[-- Attachment #3: build.sh --]
[-- Type: text/plain, Size: 2854 bytes --]

#!/bin/bash
#
# Build a gcc + uClibc toolchain
#
# References:
# http://gcc.gnu.org/ml/gcc/2007-08/msg00327.html
# http://busybox.net/~vda/HOWTO/i486-linux-uclibc/HOWTO.txt
# http://wiki.osdev.org/GCC_Cross-Compiler

set -ex

arch=$1
sysroot=$2
uclibc=$3
binutils=$4
gcc=$5
gmp=$6
mpfr=$7
kernel_headers=$8

unset CPATH LIBRARY_PATH INSTALL
PATH=/usr/lib/ccache:/bin:/usr/bin
rm -rf ${sysroot}
mkdir ${sysroot}
ln -s . ${sysroot}/usr

# Link target-specific usr directory to usr, otherwise gcc has trouble finding ld
ln -s . ${sysroot}/usr/${arch}-linux-uclibc

# Install kernel headers
mkdir ${sysroot}/usr/include
cp -pLR ${kernel_headers}/{asm,linux,asm-generic} ${sysroot}/usr/include/

# Build binutils
mkdir binutils-${arch}
( cd binutils-${arch}; ${binutils}/configure --prefix=/usr --target=${arch}-linux-uclibc --with-gmp-include=${gmp} --with-gmp-lib=${gmp}/.libs --with-mpfr-include=${mpfr} --with-mpfr-lib=${mpfr}/.libs --with-sysroot=${sysroot} --program-prefix=${arch}-linux-uclibc- )
make -C binutils-${arch} all install DESTDIR=${sysroot}

# Build stage 1 gcc (required to build uClibc)
mkdir gcc-stage1-${arch}
( cd gcc-stage1-${arch}; ${gcc}/configure --prefix=/usr --target=${arch}-linux-uclibc --enable-languages=c --with-gmp-include=${gmp} --with-gmp-lib=${gmp}/.libs --with-mpfr-include=${mpfr} --with-mpfr-lib=${mpfr}/.libs --disable-shared --disable-nls --disable-multilib --disable-threads --disable-libmudflap --disable-libssp --disable-libgomp --disable-decimal-float --disable-tls --without-headers --with-build-time-tools=${sysroot}/usr/bin --with-gnu-ld )
make -C gcc-stage1-${arch} all-gcc all-target-libgcc install-gcc install-target-libgcc DESTDIR=${sysroot}

# Build uClibc
cp -a ${uclibc} uclibc-${arch}
cp uclibc-${arch}.config uclibc-${arch}/.config
make -C uclibc-${arch} oldconfig
make -C uclibc-${arch} extra/scripts/unifdef
PATH=${sysroot}/usr/bin:${PATH} make -C uclibc-${arch} CROSS=${arch}-linux-uclibc- KERNEL_HEADERS=${sysroot}/usr/include PREFIX=${sysroot} RUNTIME_PREFIX=/ DEVEL_PREFIX=/ RUNTIME_PREFIX_LIB_FROM_DEVEL_PREFIX_LIB= V=1 all install
PATH=${sysroot}/usr/bin:${PATH} make -C uclibc-${arch}/utils CROSS=${arch}-linux-uclibc- KERNEL_HEADERS=${sysroot}/usr/include PREFIX=${sysroot} RUNTIME_PREFIX=/ DEVEL_PREFIX=/ RUNTIME_PREFIX_LIB_FROM_DEVEL_PREFIX_LIB= V=1 hostutils utils_install

# Build final gcc targeting uClibc
mkdir gcc-final-${arch}
( cd gcc-final-${arch}; ${gcc}/configure --prefix=/usr --target=${arch}-linux-uclibc --enable-languages=c --with-sysroot=${sysroot} --with-gmp-include=${gmp} --with-gmp-lib=${gmp}/.libs --with-mpfr-include=${mpfr} --with-mpfr-lib=${mpfr}/.libs --disable-nls --disable-multilib --disable-threads --disable-libgomp --disable-decimal-float --with-build-time-tools=${sysroot}/usr/bin --with-gnu-ld )
make -C gcc-final-${arch} all install DESTDIR=${sysroot}