2.6.39-rc6-mmotm0506 and -next - __kfree_rcu breaks third-party kernel code

2.6.39-rc6-mmotm0506 and -next - __kfree_rcu breaks third-party kernel code

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 2441 bytes --]

So I have a third-party kernel module which wants to check what kernel version
it's building against, and it fails to build under -rc6-mmotm0506 (it works
just fine under -rc5-mmotm0429).

I finally isolated it down to a two-liner section of code, which seems to die
because somebody didn't include something that rcupdate.h wanted to see:

cat > /tmp/kernconf.c <<EOF
#include <linux/version.h>
#include <linux/utsname.h>
EOF
+ cat

# This builds fine
kern="2.6.39-rc5-mmotm0429"
+ kern=2.6.39-rc5-mmotm0429
cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/${kern}/build/include -include /lib/modules/${kern}/build/include/generated/autoconf.h -I/lib/modules/${kern}/build/arch/x86/include -c /tmp/kernconf.c
+ cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/2.6.39-rc5-mmotm0429/build/include -include /lib/modules/2.6.39-rc5-mmotm0429/build/include/generated/autoconf.h -I/lib/modules/2.6.39-rc5-mmotm0429/build/arch/x86/include -c /tmp/kernconf.c

# this one dies
kern="2.6.39-rc6-mmotm0506"
+ kern=2.6.39-rc6-mmotm0506
cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/${kern}/build/include -include /lib/modules/${kern}/build/include/generated/autoconf.h -I/lib/modules/${kern}/build/arch/x86/include -c /tmp/kernconf.c
+ cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/2.6.39-rc6-mmotm0506/build/include -include /lib/modules/2.6.39-rc6-mmotm0506/build/include/generated/autoconf.h -I/lib/modules/2.6.39-rc6-mmotm0506/build/arch/x86/include -c /tmp/kernconf.c
In file included from /lib/modules/2.6.39-rc6-mmotm0506/build/include/linux/sem.h:81:0,
                 from /lib/modules/2.6.39-rc6-mmotm0506/build/include/linux/sched.h:72,
                 from /lib/modules/2.6.39-rc6-mmotm0506/build/include/linux/utsname.h:35,
                 from /tmp/kernconf.c:2:
/lib/modules/2.6.39-rc6-mmotm0506/build/include/linux/rcupdate.h: In function '__kfree_rcu':
/lib/modules/2.6.39-rc6-mmotm0506/build/include/linux/rcupdate.h:822:2: error: size of unnamed array is negative

rm /tmp/kernconf.c
+ rm /tmp/kernconf.c

'git blame include/linux/rcudate.h' points at this commit:

commit 896d4eb94e0b7cd896ee92c380fadc1e9d867cfc
Author: Lai Jiangshan <laijs@cn.fujitsu.com>
Date:   Fri Mar 18 11:15:47 2011 +0800

    rcu: introduce kfree_rcu()



[-- Attachment #2: Type: application/pgp-signature, Size: 227 bytes --]

Re: 2.6.39-rc6-mmotm0506 and -next - __kfree_rcu breaks third-party kernel code

[Randy Dunlap]

On Mon, 09 May 2011 11:05:00 -0400 Valdis.Kletnieks@vt.edu wrote:

> So I have a third-party kernel module which wants to check what kernel version
> it's building against, and it fails to build under -rc6-mmotm0506 (it works
> just fine under -rc5-mmotm0429).
> 
> I finally isolated it down to a two-liner section of code, which seems to die
> because somebody didn't include something that rcupdate.h wanted to see:
> 
> cat > /tmp/kernconf.c <<EOF
> #include <linux/version.h>
> #include <linux/utsname.h>
> EOF
> + cat
> 
> # This builds fine
> kern="2.6.39-rc5-mmotm0429"
> + kern=2.6.39-rc5-mmotm0429
> cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/${kern}/build/include -include /lib/modules/${kern}/build/include/generated/autoconf.h -I/lib/modules/${kern}/build/arch/x86/include -c /tmp/kernconf.c
> + cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/2.6.39-rc5-mmotm0429/build/include -include /lib/modules/2.6.39-rc5-mmotm0429/build/include/generated/autoconf.h -I/lib/modules/2.6.39-rc5-mmotm0429/build/arch/x86/include -c /tmp/kernconf.c
> 
> # this one dies
> kern="2.6.39-rc6-mmotm0506"
> + kern=2.6.39-rc6-mmotm0506
> cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/${kern}/build/include -include /lib/modules/${kern}/build/include/generated/autoconf.h -I/lib/modules/${kern}/build/arch/x86/include -c /tmp/kernconf.c
> + cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/2.6.39-rc6-mmotm0506/build/include -include /lib/modules/2.6.39-rc6-mmotm0506/build/include/generated/autoconf.h -I/lib/modules/2.6.39-rc6-mmotm0506/build/arch/x86/include -c /tmp/kernconf.c
> In file included from /lib/modules/2.6.39-rc6-mmotm0506/build/include/linux/sem.h:81:0,
>                  from /lib/modules/2.6.39-rc6-mmotm0506/build/include/linux/sched.h:72,
>                  from /lib/modules/2.6.39-rc6-mmotm0506/build/include/linux/utsname.h:35,
>                  from /tmp/kernconf.c:2:
> /lib/modules/2.6.39-rc6-mmotm0506/build/include/linux/rcupdate.h: In function '__kfree_rcu':
> /lib/modules/2.6.39-rc6-mmotm0506/build/include/linux/rcupdate.h:822:2: error: size of unnamed array is negative
> 
> rm /tmp/kernconf.c
> + rm /tmp/kernconf.c
> 
> 'git blame include/linux/rcudate.h' points at this commit:
> 
> commit 896d4eb94e0b7cd896ee92c380fadc1e9d867cfc
> Author: Lai Jiangshan <laijs@cn.fujitsu.com>
> Date:   Fri Mar 18 11:15:47 2011 +0800
> 
>     rcu: introduce kfree_rcu()

I'd be happy to see a simple fix, but we don't generally support building
external modules without using the kernel kbuild infrastructure AFAIK --
and when I add a Makefile along with your kernconf.c file, it builds successfully
on mmotm-2011-0506-1639.

Makefile is:
# usage:
# cd /path/to/kernel/source/ && make SUBDIRS=/path/to/source/procfs_ex/ modules
# aka:
# make -C /path/to/kernel/source M=/path/to/module/source [O=/path/to/kernel/build] modules

obj-m := kernconf.o

clean-files := *.o *.ko *.mod.c
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

then:
> make -C /lnx/src/MM/mmotm-2011-0506-1639 M=$PWD O=/lnx/src/MM/mmotm-2011-0506-1639/xx64 modules 2>&1 | tee bld1.out
make: Entering directory `/lnx/src/MM/mmotm-2011-0506-1639'
  CC [M]  /home/rddunlap/src/kernmodule/kernconf.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /home/rddunlap/src/kernmodule/kernconf.mod.o
  LD [M]  /home/rddunlap/src/kernmodule/kernconf.ko
make: Leaving directory `/lnx/src/MM/mmotm-2011-0506-1639'


done.
---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

Re: 2.6.39-rc6-mmotm0506 and -next - __kfree_rcu breaks third-party kernel code

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 2930 bytes --]

On Mon, 09 May 2011 09:17:57 PDT, Randy Dunlap said:

> I'd be happy to see a simple fix, but we don't generally support building
> external modules without using the kernel kbuild infrastructure AFAIK --
> and when I add a Makefile along with your kernconf.c file, it builds successfully
> on mmotm-2011-0506-1639.

And in fact the module *does* use the Makefile stuff for the actual build - it
was just the installer's "am I looking at a sane 2.6 tree" sanity check that
bombed. So I go looking for what special sauce the Makefile adds to make it
work, and....

OK, *now* I'm confused.  The magic sauce appears to be "-Os". Adding that makes
the compile work.  And for no obvious reason - there's something in there that
causes indigestion, but only if the optimizer isn't invoked.

*WTF*?

 LANG=C ./kern.sh 
cat > /tmp/kernconf.c <<EOF
#include <linux/version.h>
#include <linux/utsname.h>
EOF
+ cat

kern="/lib/modules/2.6.39-rc6-mmotm0506/"
+ kern=/lib/modules/2.6.39-rc6-mmotm0506/
cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I${kern}/build/include -I$kern/arch/x86/include/generated -include ${kern}/build/include/generated/autoconf.h -I${kern}/build/arch/x86/include -I${kern}/include/generated -Os -c /tmp/kernconf.c
+ cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/2.6.39-rc6-mmotm0506//build/include -I/lib/modules/2.6.39-rc6-mmotm0506//arch/x86/include/generated -include /lib/modules/2.6.39-rc6-mmotm0506//build/include/generated/autoconf.h -I/lib/modules/2.6.39-rc6-mmotm0506//build/arch/x86/include -I/lib/modules/2.6.39-rc6-mmotm0506//include/generated -Os -c /tmp/kernconf.c
cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I${kern}/build/include -I$kern/arch/x86/include/generated -include ${kern}/build/include/generated/autoconf.h -I${kern}/build/arch/x86/include -I${kern}/include/generated -c /tmp/kernconf.c
+ cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/2.6.39-rc6-mmotm0506//build/include -I/lib/modules/2.6.39-rc6-mmotm0506//arch/x86/include/generated -include /lib/modules/2.6.39-rc6-mmotm0506//build/include/generated/autoconf.h -I/lib/modules/2.6.39-rc6-mmotm0506//build/arch/x86/include -I/lib/modules/2.6.39-rc6-mmotm0506//include/generated -c /tmp/kernconf.c
In file included from /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/sem.h:81:0,
                 from /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/sched.h:72,
                 from /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/utsname.h:35,
                 from /tmp/kernconf.c:2:
/lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/rcupdate.h: In function '__kfree_rcu':
/lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/rcupdate.h:822:2: error: size of unnamed array is negative

rm /tmp/kernconf.c
+ rm /tmp/kernconf.c





[-- Attachment #2: Type: application/pgp-signature, Size: 227 bytes --]

Re: 2.6.39-rc6-mmotm0506 and -next - __kfree_rcu breaks third-party kernel code

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 769 bytes --]

On Mon, 09 May 2011 12:56:32 EDT, Valdis.Kletnieks@vt.edu said:

> OK, *now* I'm confused.  The magic sauce appears to be "-Os". Adding that makes
> the compile work.  And for no obvious reason - there's something in there that
> causes indigestion, but only if the optimizer isn't invoked.

/lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/rcupdate.h: In function '__kfree_rcu':
/lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/rcupdate.h:822:2: error: size of unnamed array is negative

Apparently, it's in fact bad code - but the entire __kfree_rcu() function is
being reaped by the optimizer as dead code before the code generator
sees the size issue with the array.  Without optimization, the code hangs
around long enough for the problem to crop up.


[-- Attachment #2: Type: application/pgp-signature, Size: 227 bytes --]

Re: 2.6.39-rc6-mmotm0506 and -next - __kfree_rcu breaks third-party kernel code

[Randy Dunlap]

On 05/09/11 09:56, Valdis.Kletnieks@vt.edu wrote:
> On Mon, 09 May 2011 09:17:57 PDT, Randy Dunlap said:
> 
>> I'd be happy to see a simple fix, but we don't generally support building
>> external modules without using the kernel kbuild infrastructure AFAIK --
>> and when I add a Makefile along with your kernconf.c file, it builds successfully
>> on mmotm-2011-0506-1639.
> 
> And in fact the module *does* use the Makefile stuff for the actual build - it
> was just the installer's "am I looking at a sane 2.6 tree" sanity check that
> bombed. So I go looking for what special sauce the Makefile adds to make it
> work, and....
> 
> OK, *now* I'm confused.  The magic sauce appears to be "-Os". Adding that makes
> the compile work.  And for no obvious reason - there's something in there that
> causes indigestion, but only if the optimizer isn't invoked.

You can disable that by disabling:
CONFIG_CC_OPTIMIZE_FOR_SIZE=y

> *WTF*?

I dunno either.

>  LANG=C ./kern.sh 
> cat > /tmp/kernconf.c <<EOF
> #include <linux/version.h>
> #include <linux/utsname.h>
> EOF
> + cat
> 
> kern="/lib/modules/2.6.39-rc6-mmotm0506/"
> + kern=/lib/modules/2.6.39-rc6-mmotm0506/
> cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I${kern}/build/include -I$kern/arch/x86/include/generated -include ${kern}/build/include/generated/autoconf.h -I${kern}/build/arch/x86/include -I${kern}/include/generated -Os -c /tmp/kernconf.c
> + cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/2.6.39-rc6-mmotm0506//build/include -I/lib/modules/2.6.39-rc6-mmotm0506//arch/x86/include/generated -include /lib/modules/2.6.39-rc6-mmotm0506//build/include/generated/autoconf.h -I/lib/modules/2.6.39-rc6-mmotm0506//build/arch/x86/include -I/lib/modules/2.6.39-rc6-mmotm0506//include/generated -Os -c /tmp/kernconf.c
> cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I${kern}/build/include -I$kern/arch/x86/include/generated -include ${kern}/build/include/generated/autoconf.h -I${kern}/build/arch/x86/include -I${kern}/include/generated -c /tmp/kernconf.c
> + cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/2.6.39-rc6-mmotm0506//build/include -I/lib/modules/2.6.39-rc6-mmotm0506//arch/x86/include/generated -include /lib/modules/2.6.39-rc6-mmotm0506//build/include/generated/autoconf.h -I/lib/modules/2.6.39-rc6-mmotm0506//build/arch/x86/include -I/lib/modules/2.6.39-rc6-mmotm0506//include/generated -c /tmp/kernconf.c
> In file included from /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/sem.h:81:0,
>                  from /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/sched.h:72,
>                  from /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/utsname.h:35,
>                  from /tmp/kernconf.c:2:
> /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/rcupdate.h: In function '__kfree_rcu':
> /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/rcupdate.h:822:2: error: size of unnamed array is negative
> 
> rm /tmp/kernconf.c
> + rm /tmp/kernconf.c

rcupdate.h should #include <linux/kernel.h>, but I don't think that will fix this problem.

-- 
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

Re: 2.6.39-rc6-mmotm0506 and -next - __kfree_rcu breaks third-party kernel code

[Paul E. McKenney]

On Mon, May 09, 2011 at 10:14:30AM -0700, Randy Dunlap wrote:
> On 05/09/11 09:56, Valdis.Kletnieks@vt.edu wrote:
> > On Mon, 09 May 2011 09:17:57 PDT, Randy Dunlap said:
> > 
> >> I'd be happy to see a simple fix, but we don't generally support building
> >> external modules without using the kernel kbuild infrastructure AFAIK --
> >> and when I add a Makefile along with your kernconf.c file, it builds successfully
> >> on mmotm-2011-0506-1639.
> > 
> > And in fact the module *does* use the Makefile stuff for the actual build - it
> > was just the installer's "am I looking at a sane 2.6 tree" sanity check that
> > bombed. So I go looking for what special sauce the Makefile adds to make it
> > work, and....
> > 
> > OK, *now* I'm confused.  The magic sauce appears to be "-Os". Adding that makes
> > the compile work.  And for no obvious reason - there's something in there that
> > causes indigestion, but only if the optimizer isn't invoked.
> 
> You can disable that by disabling:
> CONFIG_CC_OPTIMIZE_FOR_SIZE=y
> 
> > *WTF*?
> 
> I dunno either.

It appears that some compilers at some optimization levels aren't willing
to trace build-time constants through local parameters...  Could you please
try out the following patch?

							Thanx, Paul

------------------------------------------------------------------------

rcu: avoid build error for third-party modules

The initial definition of __kfree_rcu() checked a static inline function
argument to see if it was a compile-time constant.  Apparently not all
compilers are willing to put up with this at all optimization levels.
Add a nasty comment and remove the warning, relying on the fact that
__kfree_rcu() is called only from kfree_rcu(), which always passes in
a compile-time constant.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 99f9aa7..58b13f1 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -814,13 +814,14 @@ static __always_inline bool __is_kfree_rcu_offset(unsigned long offset)
 	return offset < 4096;
 }
 
+/*
+ * Intended to be called only from the kfree_rcu() macro.
+ */
 static __always_inline
 void __kfree_rcu(struct rcu_head *head, unsigned long offset)
 {
 	typedef void (*rcu_callback)(struct rcu_head *);
 
-	BUILD_BUG_ON(!__builtin_constant_p(offset));
-
 	/* See the kfree_rcu() header comment. */
 	BUILD_BUG_ON(!__is_kfree_rcu_offset(offset));
 

Re: 2.6.39-rc6-mmotm0506 and -next - __kfree_rcu breaks third-party kernel code

[Lai Jiangshan]

On 05/10/2011 12:56 AM, Valdis.Kletnieks@vt.edu wrote:
> On Mon, 09 May 2011 09:17:57 PDT, Randy Dunlap said:
> 
>> I'd be happy to see a simple fix, but we don't generally support building
>> external modules without using the kernel kbuild infrastructure AFAIK --
>> and when I add a Makefile along with your kernconf.c file, it builds successfully
>> on mmotm-2011-0506-1639.
> 
> And in fact the module *does* use the Makefile stuff for the actual build - it
> was just the installer's "am I looking at a sane 2.6 tree" sanity check that
> bombed. So I go looking for what special sauce the Makefile adds to make it
> work, and....
> 
> OK, *now* I'm confused.  The magic sauce appears to be "-Os". Adding that makes
> the compile work.  And for no obvious reason - there's something in there that
> causes indigestion, but only if the optimizer isn't invoked.
> 
> *WTF*?
> 
>  LANG=C ./kern.sh 
> cat > /tmp/kernconf.c <<EOF
> #include <linux/version.h>
> #include <linux/utsname.h>
> EOF
> + cat
> 
> kern="/lib/modules/2.6.39-rc6-mmotm0506/"
> + kern=/lib/modules/2.6.39-rc6-mmotm0506/
> cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I${kern}/build/include -I$kern/arch/x86/include/generated -include ${kern}/build/include/generated/autoconf.h -I${kern}/build/arch/x86/include -I${kern}/include/generated -Os -c /tmp/kernconf.c
> + cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/2.6.39-rc6-mmotm0506//build/include -I/lib/modules/2.6.39-rc6-mmotm0506//arch/x86/include/generated -include /lib/modules/2.6.39-rc6-mmotm0506//build/include/generated/autoconf.h -I/lib/modules/2.6.39-rc6-mmotm0506//build/arch/x86/include -I/lib/modules/2.6.39-rc6-mmotm0506//include/generated -Os -c /tmp/kernconf.c
> cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I${kern}/build/include -I$kern/arch/x86/include/generated -include ${kern}/build/include/generated/autoconf.h -I${kern}/build/arch/x86/include -I${kern}/include/generated -c /tmp/kernconf.c
> + cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/2.6.39-rc6-mmotm0506//build/include -I/lib/modules/2.6.39-rc6-mmotm0506//arch/x86/include/generated -include /lib/modules/2.6.39-rc6-mmotm0506//build/include/generated/autoconf.h -I/lib/modules/2.6.39-rc6-mmotm0506//build/arch/x86/include -I/lib/modules/2.6.39-rc6-mmotm0506//include/generated -c /tmp/kernconf.c
> In file included from /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/sem.h:81:0,
>                  from /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/sched.h:72,
>                  from /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/utsname.h:35,
>                  from /tmp/kernconf.c:2:
> /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/rcupdate.h: In function '__kfree_rcu':
> /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/rcupdate.h:822:2: error: size of unnamed array is negative
> 
> rm /tmp/kernconf.c
> + rm /tmp/kernconf.c
> 
> 

We use __always_inline for __Kfree_rcu(), we hope compiler does not compile
it stand-alone, but it seems that __always_inline can't work without optimization,
so you can add "-O2"(kernel build option) or "-Os" option for you third-party kernel
modules' build.