Re: Bus error on make allyesconfig, kernelbuild with HEAD

Re: Bus error on make allyesconfig, kernelbuild with HEAD

[Sam Ravnborg]

On Sat, Nov 06, 2010 at 11:49:10AM -0400, Ben Gamari wrote:
> On Sat, 06 Nov 2010 11:07:48 -0400, Ben Gamari <bgamari.foss@gmail.com> wrote:
> > On Fri, 30 Jul 2010 16:43:53 +0800, Américo Wang <xiyou.wangcong@gmail.com> wrote:
> > > This is useful. :) Looks like parse_dep_file() accesses out of
> > > the mmap'ed memory range...
> > > 
> > Did anything ever happen with this? I seem to be experiencing similar
> > issues while cross-compiling for ARM on x86-64. All tested kernels
> > (v2.6.35 to master) fail with,

Too k a quick look.
Does the following patch fix it?

if m == p then we will stay in the while look looking for a space.
I did not audit all of the code - there may be other issues..

	Sam

diff --git a/scripts/basic/fixdep.c b/scripts/basic/fixdep.c
index ea26b23..f472ada 100644
--- a/scripts/basic/fixdep.c
+++ b/scripts/basic/fixdep.c
@@ -318,7 +318,7 @@ static void parse_dep_file(void *map, size_t len)
 		while (m < end && (*m == ' ' || *m == '\\' || *m == '\n'))
 			m++;
 		p = m;
-		while (p < end && *p != ' ') p++;
+		while (p <= end && *p != ' ') p++;
 		if (p == end) {
 			do p--; while (!isalnum(*p));
 			p++;

Re: Bus error on make allyesconfig, kernelbuild with HEAD

[Ben Gamari]

On Sat, 6 Nov 2010 17:24:26 +0100, Sam Ravnborg <sam@ravnborg.org> wrote:
> Took a quick look.
> Does the following patch fix it?
> 
> if m == p then we will stay in the while look looking for a space.
> I did not audit all of the code - there may be other issues..
> 
Looks like something very strange is happening here. Debugging patch attached.

Re: Bus error on make allyesconfig, kernelbuild with HEAD

[Ben Gamari]

On Sat, 6 Nov 2010 17:24:26 +0100, Sam Ravnborg <sam@ravnborg.org> wrote:
> Took a quick look.
> Does the following patch fix it?
> 
> if m == p then we will stay in the while look looking for a space.
> I did not audit all of the code - there may be other issues..
> 
P.S. Sorry for the noise

Looks like something very odd is happening here. Seems like stat is
giving us the wrong size. As seen below, ls reports the file size to be
5247 whereas st.stat indicates a size substantially larger than
that. Output from make with attached patch applied:

$ ls -l scripts/basic/.fixdep.d
-rw-r--r-- 1 bgamari bgamari 5247 2010-11-07 11:35 scripts/basic/.fixdep.d
$ make V=1
In file included from /usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/usr/include/stdlib.h:903,
                 from scripts/basic/fixdep.c:112:
/usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/usr/include/bits/stdlib.h:65: warning: no previous prototype for ‘ptsname_r’
file=scripts/basic/.fixdep.d
st.size=2ad5f6cecb36
map=0x2ad5f6fe4000, len=2ad5f6cecb36
m=0x2ad5f6fe4000, end=0x55abedcd0b36
eh?=2ad5f6fe4000
map=0x2ad5f6fe4000, len=2ad5f6cecb36, end=0x55abedcd0b36
m=0x2ad5f6fe4009
A: p=0x2ad5f6fe400a, end=0x55abedcd0b36, str=scripts/basic/fixdep.c \
 /usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/usr/include/sys/types.h \
 /usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/usr/include/features.h \
 /usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/usr/include/sys/cdefs.h \
 /usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/usr/include/bits/wordsize.h \
 /usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/usr/include/gnu/stubs.h \
 /usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/usr/include/bits/types.h \
 /usr/lib/gcc/x86_64-linux-gnu/4.4.3/include/stddef.h \
 /usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/usr/include/bits/typesizes.h \
 /usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/usr/include/time.h \

[SNIP]

B: p=0x2ad5f6fe53e1, end=0x55abedcd0b36
C: p=0x2ad5f6fe53e1, s=/usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/usr/include/asm/sockios.h
m=0x2ad5f6fe53e2
m=0x2ad5f6fe53e3
m=0x2ad5f6fe53e4
A: p=0x2ad5f6fe53e5, end=0x55abedcd0b36, str=/usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/usr/include/bits/socket2.h \
 /usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/usr/include/bits/in.h

B: p=0x2ad5f6fe5432, end=0x55abedcd0b36
C: p=0x2ad5f6fe5432, s=/usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/usr/include/bits/socket2.h
m=0x2ad5f6fe5433
m=0x2ad5f6fe5434
m=0x2ad5f6fe5435
A: p=0x2ad5f6fe5436, end=0x55abedcd0b36, str=/usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/usr/include/bits/in.h

/bin/sh: line 1:  8456 Bus error               scripts/basic/fixdep scripts/basic/.fixdep.d scripts/basic/fixdep 'gcc -Wp,-MD,scripts/basic/.fixdep.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer     -o scripts/basic/fixdep scripts/basic/fixdep.c  ' > scripts/basic/.fixdep.tmp
make[2]: *** [scripts/basic/fixdep] Error 135
make[1]: *** [scripts_basic] Error 2
make: *** No rule to make target `include/config/auto.conf', needed by `include/config/kernel.release'.  Stop.

[PATCH] Debug fixdep

[Ben Gamari]

---
 scripts/basic/fixdep.c |   16 ++++++++++++++--
 1 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/scripts/basic/fixdep.c b/scripts/basic/fixdep.c
index ea26b23..f1b58d6 100644
--- a/scripts/basic/fixdep.c
+++ b/scripts/basic/fixdep.c
@@ -303,6 +303,9 @@ static void parse_dep_file(void *map, size_t len)
 	char *p;
 	char s[PATH_MAX];
 
+        fprintf(stderr, "map=%p, len=%zx\n", map, len);
+        fprintf(stderr, "m=%p, end=%p\n", m, end);
+        fprintf(stderr, "eh?=%zx\n", (size_t)end - (size_t)len);
 	p = strchr(m, ':');
 	if (!p) {
 		fprintf(stderr, "fixdep: parse error\n");
@@ -314,16 +317,23 @@ static void parse_dep_file(void *map, size_t len)
 
 	clear_config();
 
+        fprintf(stderr, "map=%p, len=%zx, end=%p\n", map, len, end);
 	while (m < end) {
-		while (m < end && (*m == ' ' || *m == '\\' || *m == '\n'))
+		while (m <= end && (*m == ' ' || *m == '\\' || *m == '\n')) {
+                        fprintf(stderr, "m=%p\n", m);
 			m++;
+                }
 		p = m;
-		while (p < end && *p != ' ') p++;
+                fprintf(stderr, "A: p=%p, end=%p, str=%s\n", p, end, p);
+		while (p <= end && *p != ' ') p++;
+                fprintf(stderr, "B: p=%p, end=%p\n", p, end);
 		if (p == end) {
 			do p--; while (!isalnum(*p));
+                        fprintf(stderr, "c%c ", *p);
 			p++;
 		}
 		memcpy(s, m, p-m); s[p-m] = 0;
+                fprintf(stderr, "C: p=%p, s=%s\n", p, s);
 		if (strrcmp(s, "include/generated/autoconf.h") &&
 		    strrcmp(s, "arch/um/include/uml-config.h") &&
 		    strrcmp(s, ".ver")) {
@@ -342,6 +352,7 @@ static void print_deps(void)
 	int fd;
 	void *map;
 
+        fprintf(stderr, "file=%s\n", depfile);
 	fd = open(depfile, O_RDONLY);
 	if (fd < 0) {
 		fprintf(stderr, "fixdep: ");
@@ -354,6 +365,7 @@ static void print_deps(void)
 		close(fd);
 		return;
 	}
+        fprintf(stderr, "st.size=%zx\n", st.st_size);
 	map = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
 	if ((long) map == -1) {
 		perror("fixdep: mmap");
-- 
1.7.0.4

Re: Bus error on make allyesconfig, kernelbuild with HEAD

[Sam Ravnborg]

On Sun, Nov 07, 2010 at 11:44:13AM -0500, Ben Gamari wrote:
> On Sat, 6 Nov 2010 17:24:26 +0100, Sam Ravnborg <sam@ravnborg.org> wrote:
> > Took a quick look.
> > Does the following patch fix it?
> > 
> > if m == p then we will stay in the while look looking for a space.
> > I did not audit all of the code - there may be other issues..
> > 
> P.S. Sorry for the noise
> 
> Looks like something very odd is happening here. Seems like stat is
> giving us the wrong size. As seen below, ls reports the file size to be
> 5247 whereas st.stat indicates a size substantially larger than
> that. Output from make with attached patch applied:
> 
> $ ls -l scripts/basic/.fixdep.d
> -rw-r--r-- 1 bgamari bgamari 5247 2010-11-07 11:35 scripts/basic/.fixdep.d
> $ make V=1
> In file included from /usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/usr/include/stdlib.h:903,
>                  from scripts/basic/fixdep.c:112:
> /usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/usr/include/bits/stdlib.h:65: warning: no previous prototype for ‘ptsname_r’
> file=scripts/basic/.fixdep.d
> st.size=2ad5f6cecb36

Interesting and valueable information!

Looks like fstat() failed for some odd reason
Could you try to check the return value of fstat().

If it retruns less than 0 then print out errno (preferable using perror()).

	Sam

Re: Bus error on make allyesconfig, kernelbuild with HEAD

[Ben Gamari]

On Sun, 7 Nov 2010 18:07:00 +0100, Sam Ravnborg <sam@ravnborg.org> wrote:
> Interesting and valueable information!
> 
> Looks like fstat() failed for some odd reason
> Could you try to check the return value of fstat().
> 
fixdep: fstat failed: Invalid argument

Interesting indeed. According to my manpage fstat doesn't even return
EINVAL.

- Ben

fstat suddenly return -EINVAL [Was: Bus error on make allyesconfig, kernelbuild with HEAD]

[Sam Ravnborg]

Hi Ben.

[Context to new readers: Ben experience that fixdep (part of kbuild)
 exists with an error during kernel compilations.
 This is now tracked down to fstat() returning EINVAL.
 Due to missing error checks in fixdep.c this has been a long journey
 to reach this far. ]

I looked briefly at the fstat implmnetation in the kernel and
this did not give me any hints.
I therefore changed to subject to attrack attention of more readers.

Can you supply us with a bit more information about your system:
- filesystem (as fstat seems to reach filesystem specific code)
- glibc version 
- box (architecture + 32 or 64 bit kernel)
- do you see other things misbehave or is this only a kernel build thing

If you have provided this info in the past I have missed it.

	Sam

[Kept the below for new readers]

On Sun, Nov 07, 2010 at 01:09:30PM -0500, Ben Gamari wrote:
> On Sun, 7 Nov 2010 18:07:00 +0100, Sam Ravnborg <sam@ravnborg.org> wrote:
> > Interesting and valueable information!
> > 
> > Looks like fstat() failed for some odd reason
> > Could you try to check the return value of fstat().
> > 
> fixdep: fstat failed: Invalid argument
> 
> Interesting indeed. According to my manpage fstat doesn't even return
> EINVAL.
> 
> - Ben
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 
> 

Re: fstat suddenly return -EINVAL [Was: Bus error on make allyesconfig, kernelbuild with HEAD]

[Ben Gamari]

On Sun, 7 Nov 2010 23:30:56 +0100, Sam Ravnborg <sam@ravnborg.org> wrote:
> I looked briefly at the fstat implmnetation in the kernel and
> this did not give me any hints.
> I therefore changed to subject to attrack attention of more readers.
> 
Thanks!

> Can you supply us with a bit more information about your system:
> - filesystem (as fstat seems to reach filesystem specific code)
> - glibc version 
> - box (architecture + 32 or 64 bit kernel)
> - do you see other things misbehave or is this only a kernel build thing
> 
> If you have provided this info in the past I have missed it.
> 
The kernel tree resides on an ext4 partition. The machine has two Xeon
E5520 processors and runs 64-bit Ubuntu 10.04 which uses eglibc 2.11.1.
Note that the kernel is being cross-compiled for ARM under an
OpenEmbedded toolchain (have tried both gcc 4.4.4 and 4.5). This is the
only strange behavior that I know of.

Cheers,

- Ben

Re: fstat suddenly return -EINVAL [Was: Bus error on make allyesconfig, kernelbuild with HEAD]

[Américo Wang]

On Sun, Nov 07, 2010 at 06:07:08PM -0500, Ben Gamari wrote:
>On Sun, 7 Nov 2010 23:30:56 +0100, Sam Ravnborg <sam@ravnborg.org> wrote:
>> I looked briefly at the fstat implmnetation in the kernel and
>> this did not give me any hints.
>> I therefore changed to subject to attrack attention of more readers.
>> 
>Thanks!
>
>> Can you supply us with a bit more information about your system:
>> - filesystem (as fstat seems to reach filesystem specific code)
>> - glibc version 
>> - box (architecture + 32 or 64 bit kernel)
>> - do you see other things misbehave or is this only a kernel build thing
>> 
>> If you have provided this info in the past I have missed it.
>> 
>The kernel tree resides on an ext4 partition. The machine has two Xeon
>E5520 processors and runs 64-bit Ubuntu 10.04 which uses eglibc 2.11.1.
>Note that the kernel is being cross-compiled for ARM under an
>OpenEmbedded toolchain (have tried both gcc 4.4.4 and 4.5). This is the
>only strange behavior that I know of.
>

But fixdep should be compiled by the host compiler...

So, does this error also occur when you do non-cross compiling on the
same partition?

Thanks for your testing!

Re: fstat suddenly return -EINVAL [Was: Bus error on make allyesconfig, kernelbuild with HEAD]

[Ben Gamari]

On Mon, 8 Nov 2010 18:20:02 +0800, Américo Wang <xiyou.wangcong@gmail.com> wrote:
> But fixdep should be compiled by the host compiler...
> 
Very good point.

> So, does this error also occur when you do non-cross compiling on the
> same partition?
> 
Hmm, it seems to be fine when not cross-compiling. Interesting.

> Thanks for your testing!

No worries!

- Ben

Re: fstat suddenly return -EINVAL [Was: Bus error on make allyesconfig, kernelbuild with HEAD]

[Sam Ravnborg]

On Mon, Nov 08, 2010 at 07:38:13AM -0500, Ben Gamari wrote:
> On Mon, 8 Nov 2010 18:20:02 +0800, Américo Wang <xiyou.wangcong@gmail.com> wrote:
> > But fixdep should be compiled by the host compiler...
> > 
> Very good point.
> 
> > So, does this error also occur when you do non-cross compiling on the
> > same partition?
> > 
> Hmm, it seems to be fine when not cross-compiling. Interesting.
> 
> > Thanks for your testing!
> 
> No worries!

Hi Ben - interesting information.

Do you see that this bug trigger for the same file always?
You could try to print out the filename if fstat fails.

If this happens while accessing the same file then try to
check if this file has any special permissions / security settings.
[I looked a bit on the kernel side of fstat() and it looked
like a security check could result in EINVAL].

Again if it is always the same file try if you can read it
using less/vi.

From your previous posting it looks like you have some special setting
that impact your choice of HOST gcc.
Do you define HOSTCC somewhere?


	Sam

Re: fstat suddenly return -EINVAL [Was: Bus error on make allyesconfig, kernelbuild with HEAD]

[Ben Gamari]

On Mon, 8 Nov 2010 19:22:09 +0100, Sam Ravnborg <sam@ravnborg.org> wrote:
> Hi Ben - interesting information.
> 
> Do you see that this bug trigger for the same file always?
> You could try to print out the filename if fstat fails.
>
fstat appears to be failing for every file (the file= line is printed
right before the fstat; there seems to be a corresponding error for
every file= so I'm inferring that it must always fail):

$ make
  HOSTCC  scripts/basic/fixdep
file=scripts/basic/.fixdep.d
fixdep: fstat failed: Invalid argument
  HOSTCC  scripts/basic/docproc
file=scripts/basic/.docproc.d
fixdep: fstat failed: Invalid argument
  HOSTCC  scripts/kconfig/conf.o
  HOSTCC  scripts/kconfig/kxgettext.o
file=scripts/kconfig/.conf.o.d
fixdep: fstat failed: Invalid argument
file=scripts/kconfig/.kxgettext.o.d
fixdep: fstat failed: Invalid argument
  HOSTCC  scripts/kconfig/zconf.tab.o
file=scripts/kconfig/.zconf.tab.o.d
fixdep: fstat failed: Invalid argument
  HOSTLD  scripts/kconfig/conf
scripts/kconfig/conf --silentoldconfig Kconfig
***
*** Configuration file ".config" not found!
***
*** Please run some configurator (e.g. "make oldconfig" or
*** "make menuconfig" or "make xconfig").
***
make[2]: *** [silentoldconfig] Error 1
make[1]: *** [silentoldconfig] Error 2
make: *** No rule to make target `include/config/auto.conf', needed by `include/config/kernel.release'.  Stop.

> If this happens while accessing the same file then try to
> check if this file has any special permissions / security settings.
> [I looked a bit on the kernel side of fstat() and it looked
> like a security check could result in EINVAL].
>
> Again if it is always the same file try if you can read it
> using less/vi.
> 
It actually seems as if the problem might be that none of these files
actually exist:

$ find scripts -iname .*.d
$

How are .d files supposed to be generated?


> From your previous posting it looks like you have some special setting
> that impact your choice of HOST gcc.
> Do you define HOSTCC somewhere?
> 
I don't think I do define HOSTCC anywhere. My cross compilation
environment is as follows,

source /usr/local/angstrom/arm/environment-setup
export ARCH=arm
export CROSS_COMPILE=arm-angstrom-linux-gnueabi-
export CC=${CROSS_COMPILE}gcc
export CXX=${CROSS_COMPILE}g++

OE_ENV_ROOT=/home/bgamari/OE/angstrom-dev
SYSROOT=$OE_ENV_ROOT/sysroots/armv7a-angstrom-linux-gnueabi
export LD_LIBRARY_PATH=$SYSROOT/lib:$SYSROOT/usr/lib
export CFLAGS=-I$OE_ROOT/lib/gcc/arm-angstrom-linux-gnueabi/4.3.1/include

Re: fstat suddenly return -EINVAL [Was: Bus error on make allyesconfig, kernelbuild with HEAD]

[Sam Ravnborg]

> >
> fstat appears to be failing for every file (the file= line is printed
> right before the fstat; there seems to be a corresponding error for
> every file= so I'm inferring that it must always fail):
> 
> $ make
>   HOSTCC  scripts/basic/fixdep
> file=scripts/basic/.fixdep.d
> fixdep: fstat failed: Invalid argument

But open succedd - because we see no output from that one.

> It actually seems as if the problem might be that none of these files
> actually exist:
> 
> $ find scripts -iname .*.d
> $
> 
> How are .d files supposed to be generated?

They are generated by kbuild.
See scripts/Kbuild.include:

# Execute the command and also postprocess generated .d dependencies file.
if_changed_dep = $(if $(strip $(any-prereq) $(arg-check) ),                  \
        @set -e;                                                             \
        $(echo-cmd) $(cmd_$(1));                                             \
        scripts/basic/fixdep $(depfile) $@ '$(make-cmd)' > $(dot-target).tmp;\
==>     rm -f $(depfile);                                                    \
        mv -f $(dot-target).tmp $(dot-target).cmd)


If you delete the line marked "==>" then you should have .d files are a build.

> > From your previous posting it looks like you have some special setting
> > that impact your choice of HOST gcc.
> > Do you define HOSTCC somewhere?
> > 
> I don't think I do define HOSTCC anywhere. My cross compilation
> environment is as follows,
> 
> source /usr/local/angstrom/arm/environment-setup
> export ARCH=arm
> export CROSS_COMPILE=arm-angstrom-linux-gnueabi-
So far it looks OK.

> export CC=${CROSS_COMPILE}gcc
> export CXX=${CROSS_COMPILE}g++
This part is wrong. The kernel will do this for you.
It should not cause any harm...

> 
> OE_ENV_ROOT=/home/bgamari/OE/angstrom-dev
> SYSROOT=$OE_ENV_ROOT/sysroots/armv7a-angstrom-linux-gnueabi
> export LD_LIBRARY_PATH=$SYSROOT/lib:$SYSROOT/usr/lib
> export CFLAGS=-I$OE_ROOT/lib/gcc/arm-angstrom-linux-gnueabi/4.3.1/include

These looks unrelated.

	Sam

Re: fstat suddenly return -EINVAL [Was: Bus error on make allyesconfig, kernelbuild with HEAD]

[Ben Gamari]

On Mon, 8 Nov 2010 20:05:01 +0100, Sam Ravnborg <sam@ravnborg.org> wrote:
> > 
> > $ make
> >   HOSTCC  scripts/basic/fixdep
> > file=scripts/basic/.fixdep.d
> > fixdep: fstat failed: Invalid argument
> 
> But open succedd - because we see no output from that one.
> 
Yes, this is true.

> > How are .d files supposed to be generated?
> 
> They are generated by kbuild.
> See scripts/Kbuild.include:
> 
> If you delete the line marked "==>" then you should have .d files are a build.
> 
True.

> > export CC=${CROSS_COMPILE}gcc
> > export CXX=${CROSS_COMPILE}g++
> This part is wrong. The kernel will do this for you.
> It should not cause any harm...
> 
Yeah, this is for other applications.

Any idea of what could be happening?

- Ben

Re: fstat suddenly return -EINVAL [Was: Bus error on make allyesconfig, kernelbuild with HEAD]

[Américo Wang]

On Mon, Nov 08, 2010 at 03:17:16PM -0500, Ben Gamari wrote:
>On Mon, 8 Nov 2010 20:05:01 +0100, Sam Ravnborg <sam@ravnborg.org> wrote:
>> > 
>> > $ make
>> >   HOSTCC  scripts/basic/fixdep
>> > file=scripts/basic/.fixdep.d
>> > fixdep: fstat failed: Invalid argument
>> 
>> But open succedd - because we see no output from that one.
>> 
>Yes, this is true.
>
>> > How are .d files supposed to be generated?
>> 
>> They are generated by kbuild.
>> See scripts/Kbuild.include:
>> 
>> If you delete the line marked "==>" then you should have .d files are a build.
>> 
>True.
>
>> > export CC=${CROSS_COMPILE}gcc
>> > export CXX=${CROSS_COMPILE}g++
>> This part is wrong. The kernel will do this for you.
>> It should not cause any harm...
>> 
>Yeah, this is for other applications.
>
>Any idea of what could be happening?
>

No idea why this only happens to those .*.d files.
A blind guess would be commit 365b1818. :-/