[patch] 2.4, head: PAGE_SHIFT changes break glibc

[patch] 2.4, head: PAGE_SHIFT changes break glibc

[Maciej W. Rozycki]

Ralf,

 Recent changes made to <asm/page.h> break a build of glibc 2.2.5 for me.  
Compilation bails out due to PAGE_SHIFT being undeclared -- glibc pulls it
as it uses PAGE_SIZE in linuxthreads/internals.h.  The PAGE_SHIFT macro
depends on configuration now (I use an empty cofinguration for glibc
headers, hence the error) and thus it'd better be simply private to the
kernel.  Glibc will then use sysconf(_SC_PAGE_SIZE) which now better
reflects actual configuration of the system it's run on.

 Here's a patch that limits PAGE_SIZE to the kernel scope.  If there's any
other program that needs PAGE_SIZE, it should be converted to
sysconf(_SC_PAGE_SIZE) as well.

 OK to apply?

 Additionally, I think we should also implement the getpagesize syscall to
benefit statically linked programs (and make glibc use it like for other
platforms that use variable page sizes).  Finally, I'm not sure such a
noticeable change was a good move in these late days of 2.4...

  Maciej

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

linux-include-mips-2.4.22-20031115-mips-page_size.patch
diff -up --recursive --new-file linux-include-mips-2.4.22-20031115.macro/include/asm-mips/page.h linux-include-mips-2.4.22-20031115/include/asm-mips/page.h
--- linux-include-mips-2.4.22-20031115.macro/include/asm-mips/page.h	2003-10-17 02:57:33.000000000 +0000
+++ linux-include-mips-2.4.22-20031115/include/asm-mips/page.h	2003-11-21 09:29:52.000000000 +0000
@@ -12,6 +12,8 @@
 
 #include <linux/config.h>
 
+#ifdef __KERNEL__
+
 /*
  * PAGE_SHIFT determines the page size
  */
@@ -27,8 +29,6 @@
 #define PAGE_SIZE	(1L << PAGE_SHIFT)
 #define PAGE_MASK	(~(PAGE_SIZE-1))
 
-#ifdef __KERNEL__
-
 #ifndef __ASSEMBLY__
 
 #include <asm/cacheflush.h>
diff -up --recursive --new-file linux-include-mips-2.4.22-20031115.macro/include/asm-mips64/page.h linux-include-mips-2.4.22-20031115/include/asm-mips64/page.h
--- linux-include-mips-2.4.22-20031115.macro/include/asm-mips64/page.h	2003-10-17 02:57:34.000000000 +0000
+++ linux-include-mips-2.4.22-20031115/include/asm-mips64/page.h	2003-11-21 09:30:10.000000000 +0000
@@ -11,6 +11,8 @@
 
 #include <linux/config.h>
 
+#ifdef __KERNEL__
+
 /*
  * PAGE_SHIFT determines the page size
  */
@@ -26,8 +28,6 @@
 #define PAGE_SIZE	(1UL << PAGE_SHIFT)
 #define PAGE_MASK	(~(PAGE_SIZE-1))
 
-#ifdef __KERNEL__
-
 #ifndef __ASSEMBLY__
 
 #include <asm/cacheflush.h>

Re: [patch] 2.4, head: PAGE_SHIFT changes break glibc

[Ralf Baechle]

On Fri, Nov 21, 2003 at 06:33:37PM +0100, Maciej W. Rozycki wrote:

>  Recent changes made to <asm/page.h> break a build of glibc 2.2.5 for me.  
> Compilation bails out due to PAGE_SHIFT being undeclared -- glibc pulls it
> as it uses PAGE_SIZE in linuxthreads/internals.h.  The PAGE_SHIFT macro
> depends on configuration now (I use an empty cofinguration for glibc
> headers, hence the error) and thus it'd better be simply private to the
> kernel.  Glibc will then use sysconf(_SC_PAGE_SIZE) which now better
> reflects actual configuration of the system it's run on.

Interesting.  IA-64 does the same thing, for example.  Wonder why they
seem to be able to get away with that.  At the very least including the
kernel header file may pick up a wrong value for PAGE_SHIFT.

>  Here's a patch that limits PAGE_SIZE to the kernel scope.  If there's any
> other program that needs PAGE_SIZE, it should be converted to
> sysconf(_SC_PAGE_SIZE) as well.
> 
>  OK to apply?

Yes, please go ahead.

>  Additionally, I think we should also implement the getpagesize syscall to
> benefit statically linked programs (and make glibc use it like for other
> platforms that use variable page sizes).

The kernel is already passing AT_PAGESZ to ELF binaries.  Wouldn't that
be sufficient?  Currently it's passing the largest supported page size,
that is 64k.  However this constant is always passed even when a smaller
page size is configured.

>  Finally, I'm not sure such a
> noticeable change was a good move in these late days of 2.4...

TLB reloads have been shown to be a major performance problem; this is an
not yet completed attempt to improve the situation so people don't need
to go for crude hacks such as wired TLB entires and similar.

Other parts of improvments such as hugetlbfs are available in 2.6 only
anyway.  I'm also thinking of changing the pagetable structure back to
the aggressivly optmized thing we were using before Linux 2.0.14 - but
certainly not for 2.4 - too intrusive, as you say.  With most MIPS users
being embedded users I still expect 2.4 to live for quite a while -
certainly longer than I'd like to see ...

  Ralf

Re: [patch] 2.4, head: PAGE_SHIFT changes break glibc

[Maciej W. Rozycki]

On Fri, 21 Nov 2003, Ralf Baechle wrote:

> >  Recent changes made to <asm/page.h> break a build of glibc 2.2.5 for me.  
> > Compilation bails out due to PAGE_SHIFT being undeclared -- glibc pulls it
> > as it uses PAGE_SIZE in linuxthreads/internals.h.  The PAGE_SHIFT macro
> > depends on configuration now (I use an empty cofinguration for glibc
> > headers, hence the error) and thus it'd better be simply private to the
> > kernel.  Glibc will then use sysconf(_SC_PAGE_SIZE) which now better
> > reflects actual configuration of the system it's run on.
> 
> Interesting.  IA-64 does the same thing, for example.  Wonder why they
> seem to be able to get away with that.  At the very least including the
> kernel header file may pick up a wrong value for PAGE_SHIFT.

 I think people (this may include distributors as well) build glibc with
Linux headers fully configured or, worse yet, with whatever happens to be
grabbed by gcc as <linux/...> and <asm/...> (i.e. usually headers from the
previous installation).  So in this case PAGE_SHIFT gets defined somehow,
although not necessarily reflecting what will be true for the host system.

 The question is which one is the *right* full configuration.  I think all
are valid, so none is the single right, so I chose not to define kernel
configuration macros for the userland -- the userland ABI shouldn't depend
on a particular kernel configuration.  I hope this issue will disappear
automatically once the kernel header space gets divided into an internal
part and one forming the ABI.

 If you look at my spec file for glibc, you'll see it only does:

echo "#define AUTOCONF_INCLUDED" > linux/include/linux/autoconf.h

to get kernel headers working.

> >  Additionally, I think we should also implement the getpagesize syscall to
> > benefit statically linked programs (and make glibc use it like for other
> > platforms that use variable page sizes).
> 
> The kernel is already passing AT_PAGESZ to ELF binaries.  Wouldn't that
> be sufficient?  Currently it's passing the largest supported page size,

 Well, AFAICS in glibc it's ld.so that is responsible for interpreting the
auxiliary vector -- see _dl_aux_init() in elf/dl-support.c.  If the
dynamic linker isn't run (which is the usual case for static binaries,
although calling dlopen() from them might complicate things), the
dl_pagesize variable remains set to zero.  Please prove me wrong if I am
missing anything.

> that is 64k.  However this constant is always passed even when a smaller
> page size is configured.

 Are you sure?  I can see create_elf_tables() in fs/binfmt_elf.c sets 
AT_PAGESZ to ELF_EXEC_PAGESIZE, which, in turn, is set in 
include/asm-mips/elf.h to PAGE_SIZE.  Which is the currently used page 
size and probably the optimal solution.

> >  Finally, I'm not sure such a
> > noticeable change was a good move in these late days of 2.4...
> 
> TLB reloads have been shown to be a major performance problem; this is an
> not yet completed attempt to improve the situation so people don't need
> to go for crude hacks such as wired TLB entires and similar.

 OK, but the safe (and practiced in the mainline) procedure is to test
with the development version and do a backport if necessary.  Well, it
might not work that well this time, as apparently I'm the only one to come
upon such oddities ;-) and, as you probably know, I haven't switched to
2.6, yet.

> Other parts of improvments such as hugetlbfs are available in 2.6 only
> anyway.  I'm also thinking of changing the pagetable structure back to
> the aggressivly optmized thing we were using before Linux 2.0.14 - but
> certainly not for 2.4 - too intrusive, as you say.  With most MIPS users
> being embedded users I still expect 2.4 to live for quite a while -
> certainly longer than I'd like to see ...

 As I wrote, such changes may get backported if there's a need and a
volunteer to do the task.  I won't support 2.4 for a long time, yet -- I
just want to upgrade the toolchain before I upgrade the kernel.  After
then I may only backport some DECstation driver fixes.

 BTW, can you imagine how huge gcc 3.4 is? -- for the i386 it needs 1GB of 
disk space and plenty of hours to be built on a reasonable system.  For 
MIPS I suppose the disk space needed will be yet larger and building may 
require a few days...

  Maciej

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: [patch] 2.4, head: PAGE_SHIFT changes break glibc

[Maciej W. Rozycki]

On Fri, 21 Nov 2003, Maciej W. Rozycki wrote:

> > The kernel is already passing AT_PAGESZ to ELF binaries.  Wouldn't that
> > be sufficient?  Currently it's passing the largest supported page size,
> 
>  Well, AFAICS in glibc it's ld.so that is responsible for interpreting the
> auxiliary vector -- see _dl_aux_init() in elf/dl-support.c.  If the
> dynamic linker isn't run (which is the usual case for static binaries,
> although calling dlopen() from them might complicate things), the
> dl_pagesize variable remains set to zero.  Please prove me wrong if I am
> missing anything.
> 
> > that is 64k.  However this constant is always passed even when a smaller
> > page size is configured.
> 
>  Are you sure?  I can see create_elf_tables() in fs/binfmt_elf.c sets 
> AT_PAGESZ to ELF_EXEC_PAGESIZE, which, in turn, is set in 
> include/asm-mips/elf.h to PAGE_SIZE.  Which is the currently used page 
> size and probably the optimal solution.

 After rebuilding glibc (2.2.5) with the patch applied, the following
program:

#include <stdio.h>
#include <unistd.h>

int main(void)
{
	printf("%u\n", getpagesize());
	return 0;
}

prints "4096" if dynamically linked and "65536" if statically linked on my
system, as expected.

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: [patch] 2.4, head: PAGE_SHIFT changes break glibc

[Ralf Baechle]

On Tue, Nov 25, 2003 at 04:27:49PM +0100, Maciej W. Rozycki wrote:

>  After rebuilding glibc (2.2.5) with the patch applied, the following
> program:
> 
> #include <stdio.h>
> #include <unistd.h>
> 
> int main(void)
> {
> 	printf("%u\n", getpagesize());
> 	return 0;
> }
> 
> prints "4096" if dynamically linked and "65536" if statically linked on my
> system, as expected.

Guess we'll need a solution along the lines of
unix/sysv/linux/sparc/sparc32/getpagesize.c then ...

  Ralf

Re: [patch] 2.4, head: PAGE_SHIFT changes break glibc

[Maciej W. Rozycki]

On Wed, 26 Nov 2003, Ralf Baechle wrote:

> Guess we'll need a solution along the lines of
> unix/sysv/linux/sparc/sparc32/getpagesize.c then ...

 I suppose so, although being not that fond of insane numbers of syscalls
I wonder how sysdeps/unix/sysv/linux/ia64/getpagesize.c gets away with
static binaries...  Perhaps we should ask glibc hackers?

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: [patch] 2.4, head: PAGE_SHIFT changes break glibc

[Daniel Jacobowitz]

On Wed, Nov 26, 2003 at 01:09:16AM +0100, Maciej W. Rozycki wrote:
> On Wed, 26 Nov 2003, Ralf Baechle wrote:
> 
> > Guess we'll need a solution along the lines of
> > unix/sysv/linux/sparc/sparc32/getpagesize.c then ...
> 
>  I suppose so, although being not that fond of insane numbers of syscalls
> I wonder how sysdeps/unix/sysv/linux/ia64/getpagesize.c gets away with
> static binaries...  Perhaps we should ask glibc hackers?

Look at elf/dl-support.c:_dl_aux_init? dl_pagesize should end up
initialized for static binaries too.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: [patch] 2.4, head: PAGE_SHIFT changes break glibc

[Maciej W. Rozycki]

On Wed, 26 Nov 2003, Daniel Jacobowitz wrote:

> >  I suppose so, although being not that fond of insane numbers of syscalls
> > I wonder how sysdeps/unix/sysv/linux/ia64/getpagesize.c gets away with
> > static binaries...  Perhaps we should ask glibc hackers?
> 
> Look at elf/dl-support.c:_dl_aux_init? dl_pagesize should end up
> initialized for static binaries too.

 Thanks for the hint.  I can see _dl_aux_init() gets indeed called from
__libc_start_main() in my i386/Linux libc-start.o binary (in libc.a), but
it does not in my MIPSel/Linux one.  So there must be a bug somewhere in
2.2.5, perhaps fixed later.  I'll have a look at it.  Once fixed, I
guess we should choose the IA64 way.

 After a brief look at the sources I suspect sysdeps/mips/elf/ldsodefs.h
overrides sysdeps/unix/sysv/linux/ldsodefs.h -- if that's the case, the
bug still exists in the trunk.  I'll work on a fix later, probably
tonight.

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: [patch] 2.4, head: PAGE_SHIFT changes break glibc

[Daniel Jacobowitz]

On Thu, Nov 27, 2003 at 01:45:00PM +0100, Maciej W. Rozycki wrote:
> On Wed, 26 Nov 2003, Daniel Jacobowitz wrote:
> 
> > >  I suppose so, although being not that fond of insane numbers of syscalls
> > > I wonder how sysdeps/unix/sysv/linux/ia64/getpagesize.c gets away with
> > > static binaries...  Perhaps we should ask glibc hackers?
> > 
> > Look at elf/dl-support.c:_dl_aux_init? dl_pagesize should end up
> > initialized for static binaries too.
> 
>  Thanks for the hint.  I can see _dl_aux_init() gets indeed called from
> __libc_start_main() in my i386/Linux libc-start.o binary (in libc.a), but
> it does not in my MIPSel/Linux one.  So there must be a bug somewhere in
> 2.2.5, perhaps fixed later.  I'll have a look at it.  Once fixed, I
> guess we should choose the IA64 way.
> 
>  After a brief look at the sources I suspect sysdeps/mips/elf/ldsodefs.h
> overrides sysdeps/unix/sysv/linux/ldsodefs.h -- if that's the case, the
> bug still exists in the trunk.  I'll work on a fix later, probably
> tonight.

_dl_aux_init ought to be called from __libc_start_main in
sysdeps/generic/libc-start.c.  In current sources I can't see any way
for that to go wrong on MIPS (but my MIPS board is off right now so I
haven't tried it)...

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: [patch] 2.4, head: PAGE_SHIFT changes break glibc

[Maciej W. Rozycki]

On Thu, 27 Nov 2003, Daniel Jacobowitz wrote:

> >  After a brief look at the sources I suspect sysdeps/mips/elf/ldsodefs.h
> > overrides sysdeps/unix/sysv/linux/ldsodefs.h -- if that's the case, the
> > bug still exists in the trunk.  I'll work on a fix later, probably
> > tonight.
> 
> _dl_aux_init ought to be called from __libc_start_main in
> sysdeps/generic/libc-start.c.  In current sources I can't see any way
> for that to go wrong on MIPS (but my MIPS board is off right now so I
> haven't tried it)...

 I did a build of 2.2.5 and my suspicion got confirmed -- the exact 
command line to build libc-start.o is as follows:

mipsel-linux-gcc ../sysdeps/generic/libc-start.c -c -O2 -Wall -Winline
-Wstrict-prototypes -Wwrite-strings -pipe -fomit-frame-pointer -g0 -O99
-fomit-frame-pointer -D__USE_STRING_INLINES -I../include -I. -I..
-I../libio -I../sysdeps/mips/elf -I../linuxthreads/sysdeps/unix/sysv/linux
-I../linuxthreads/sysdeps/pthread -I../sysdeps/pthread
-I../linuxthreads/sysdeps/unix/sysv -I../linuxthreads/sysdeps/unix
-I../linuxthreads/sysdeps/mips -I../sysdeps/unix/sysv/linux/mips
-I../sysdeps/unix/sysv/linux -I../sysdeps/gnu -I../sysdeps/unix/common
-I../sysdeps/unix/mman -I../sysdeps/unix/inet -I../sysdeps/unix/sysv
-I../sysdeps/unix/mips -I../sysdeps/unix -I../sysdeps/posix
-I../sysdeps/mips/mipsel -I../sysdeps/mips/fpu -I../sysdeps/mips
-I../sysdeps/wordsize-32 -I../sysdeps/ieee754/flt-32
-I../sysdeps/ieee754/dbl-64 -I../sysdeps/ieee754 -I../sysdeps/generic/elf
-I../sysdeps/generic -nostdinc -isystem
/usr/lib/gcc-lib/mipsel-linux/2.95.4/include -isystem
/home/macro/src/redhat/BUILD/glibc-2.2.5/linux/include -D_LIBC_REENTRANT
-include ../include/libc-symbols.h -DPIC -DHAVE_INITFINI -o libc-start.o

i.e. it's sysdeps/mips/elf/ldsodefs.h that gets included as a result of
"#include <ldsodefs.h>", not sysdeps/unix/sysv/linux/ldsodefs.h.  Then
sysdeps/mips/elf/ldsodefs.h uses "#include <sysdeps/generic/ldsodefs.h>"  
in 2.2.5, and "#include_next <ldsodefs.h>" in the trunk, so I'll just 
rebuild 2.2.5 with this change and consider the trunk fixed.

  Maciej

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +