[RFC][PATCH] introduce TASK_SIZE_OF() for all arches

[RFC][PATCH] introduce TASK_SIZE_OF() for all arches

[Dave Hansen]

For the /proc/<pid>/pagemap code[1], we need to able to query how
much virtual address space a particular task has.  The trick is
that we do it through /proc and can't use TASK_SIZE since it
references "current" on some arches.  The process opening the
/proc file might be a 32-bit process opening a 64-bit process's
pagemap file.

x86_64 already has a TASK_SIZE_OF() macro:

#define TASK_SIZE_OF(child)     ((test_tsk_thread_flag(child, TIF_IA32)) ? IA32_PAGE_OFFSET : TASK_SIZE64)

I'd like to have that for other architectures.  So, add it
for all the architectures that actually use "current" in 
their TASK_SIZE.  For the others, just add a quick #define
in sched.h to use plain old TASK_SIZE.

1. http://www.linuxworld.com/news/2007/042407-kernel.html

Signed-off-by: Dave Hansen <haveblue@us.ibm.com>
---

 lxc-dave/include/asm-ia64/processor.h    |    3 ++-
 lxc-dave/include/asm-parisc/processor.h  |    3 ++-
 lxc-dave/include/asm-powerpc/processor.h |    4 +++-
 lxc-dave/include/asm-s390/processor.h    |    2 ++
 lxc-dave/include/linux/sched.h           |    4 ++++
 5 files changed, 13 insertions(+), 3 deletions(-)

diff -puN include/asm-ia64/processor.h~task_size_of include/asm-ia64/processor.h
--- lxc/include/asm-ia64/processor.h~task_size_of	2007-08-07 15:30:54.000000000 -0700
+++ lxc-dave/include/asm-ia64/processor.h	2007-08-07 15:30:54.000000000 -0700
@@ -31,7 +31,8 @@
  * each (assuming 8KB page size), for a total of 8TB of user virtual
  * address space.
  */
-#define TASK_SIZE		(current->thread.task_size)
+#define TASK_SIZE_OF(tsk)	((tsk)->thread.task_size)
+#define TASK_SIZE       	TASK_SIZE_OF(current)
 
 /*
  * This decides where the kernel will search for a free chunk of vm
diff -puN include/asm-parisc/processor.h~task_size_of include/asm-parisc/processor.h
--- lxc/include/asm-parisc/processor.h~task_size_of	2007-08-07 15:30:54.000000000 -0700
+++ lxc-dave/include/asm-parisc/processor.h	2007-08-07 15:30:54.000000000 -0700
@@ -32,7 +32,8 @@
 #endif
 #define current_text_addr() ({ void *pc; current_ia(pc); pc; })
 
-#define TASK_SIZE               (current->thread.task_size)
+#define TASK_SIZE_OF(tsk)       ((tsk)->thread.task_size)
+#define TASK_SIZE	         (current->thread.task_size)
 #define TASK_UNMAPPED_BASE      (current->thread.map_base)
 
 #define DEFAULT_TASK_SIZE32	(0xFFF00000UL)
diff -puN include/asm-powerpc/processor.h~task_size_of include/asm-powerpc/processor.h
--- lxc/include/asm-powerpc/processor.h~task_size_of	2007-08-07 15:30:54.000000000 -0700
+++ lxc-dave/include/asm-powerpc/processor.h	2007-08-07 15:30:54.000000000 -0700
@@ -107,7 +107,9 @@ extern struct task_struct *last_task_use
  */
 #define TASK_SIZE_USER32 (0x0000000100000000UL - (1*PAGE_SIZE))
 
-#define TASK_SIZE (test_thread_flag(TIF_32BIT) ? \
+#define TASK_SIZE	  (test_thread_flag(TIF_32BIT) ? \
+		TASK_SIZE_USER32 : TASK_SIZE_USER64)
+#define TASK_SIZE_OF(tsk) (test_tsk_thread_flag(tsk, TIF_32BIT) ? \
 		TASK_SIZE_USER32 : TASK_SIZE_USER64)
 
 /* This decides where the kernel will search for a free chunk of vm
diff -puN include/asm-s390/processor.h~task_size_of include/asm-s390/processor.h
--- lxc/include/asm-s390/processor.h~task_size_of	2007-08-07 15:30:54.000000000 -0700
+++ lxc-dave/include/asm-s390/processor.h	2007-08-07 15:30:54.000000000 -0700
@@ -75,6 +75,8 @@ extern struct task_struct *last_task_use
 
 # define TASK_SIZE		(test_thread_flag(TIF_31BIT) ? \
 					(0x80000000UL) : (0x40000000000UL))
+# define TASK_SIZE_OF(tsk)	(test_tsk_thread_flag(tsk, TIF_31BIT) ? \
+					(0x80000000UL) : (0x40000000000UL))
 # define TASK_UNMAPPED_BASE	(TASK_SIZE / 2)
 # define DEFAULT_TASK_SIZE	(0x40000000000UL)
 
diff -puN include/linux/sched.h~task_size_of include/linux/sched.h
--- lxc/include/linux/sched.h~task_size_of	2007-08-07 15:30:54.000000000 -0700
+++ lxc-dave/include/linux/sched.h	2007-08-07 15:30:54.000000000 -0700
@@ -1712,6 +1712,10 @@ static inline void inc_syscw(struct task
 }
 #endif
 
+#ifndef TASK_SIZE_OF
+#define TASK_SIZE_OF(tsk)	TASK_SIZE
+#endif
+
 #endif /* __KERNEL__ */
 
 #endif
_

Re: [RFC][PATCH] introduce TASK_SIZE_OF() for all arches

[Matthew Wilcox]

On Tue, Aug 21, 2007 at 12:38:35PM -0700, Dave Hansen wrote:
> I'd like to have that for other architectures.  So, add it
> for all the architectures that actually use "current" in 
> their TASK_SIZE.  For the others, just add a quick #define
> in sched.h to use plain old TASK_SIZE.

MIPS seems to do things a little differently ... Ralf, what should Dave
be doing here?

#define TASK_SIZE32     0x7fff8000UL
#define TASK_SIZE       0x10000000000UL

-- 
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: [RFC][PATCH] introduce TASK_SIZE_OF() for all arches

[Ralf Baechle]

On Tue, Aug 21, 2007 at 01:43:09PM -0600, Matthew Wilcox wrote:

> On Tue, Aug 21, 2007 at 12:38:35PM -0700, Dave Hansen wrote:
> > I'd like to have that for other architectures.  So, add it
> > for all the architectures that actually use "current" in 
> > their TASK_SIZE.  For the others, just add a quick #define
> > in sched.h to use plain old TASK_SIZE.
> 
> MIPS seems to do things a little differently ... Ralf, what should Dave
> be doing here?
> 
> #define TASK_SIZE32     0x7fff8000UL
> #define TASK_SIZE       0x10000000000UL

Indeed. Below patch should do the trick.

  Ralf

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

diff --git a/include/asm-mips/processor.h b/include/asm-mips/processor.h
index 83bc945..0e79e03 100644
--- a/include/asm-mips/processor.h
+++ b/include/asm-mips/processor.h
@@ -45,6 +45,8 @@ extern unsigned int vced_count, vcei_count;
  * space during mmap's.
  */
 #define TASK_UNMAPPED_BASE	(PAGE_ALIGN(TASK_SIZE / 3))
+#define TASK_SIZE_OF(tsk)						\
+	(test_thread_flag(TIF_32BIT_ADDR) ? TASK_SIZE32 : TASK_SIZE)
 #endif
 
 #ifdef CONFIG_64BIT
@@ -65,6 +67,8 @@ extern unsigned int vced_count, vcei_count;
 #define TASK_UNMAPPED_BASE						\
 	(test_thread_flag(TIF_32BIT_ADDR) ?				\
 		PAGE_ALIGN(TASK_SIZE32 / 3) : PAGE_ALIGN(TASK_SIZE / 3))
+#define TASK_SIZE_OF(tsk)						\
+	(test_thread_flag(TIF_32BIT_ADDR) ? TASK_SIZE32 : TASK_SIZE)
 #endif
 
 #define NUM_FPU_REGS	32

Re: [RFC][PATCH] introduce TASK_SIZE_OF() for all arches

[Dave Hansen]

On Wed, 2007-08-22 at 18:30 +0100, Ralf Baechle wrote:
> On Tue, Aug 21, 2007 at 01:43:09PM -0600, Matthew Wilcox wrote:
> 
> > On Tue, Aug 21, 2007 at 12:38:35PM -0700, Dave Hansen wrote:
> > > I'd like to have that for other architectures.  So, add it
> > > for all the architectures that actually use "current" in 
> > > their TASK_SIZE.  For the others, just add a quick #define
> > > in sched.h to use plain old TASK_SIZE.
> > 
> > MIPS seems to do things a little differently ... Ralf, what should Dave
> > be doing here?
> > 
> > #define TASK_SIZE32     0x7fff8000UL
> > #define TASK_SIZE       0x10000000000UL
> 
> Indeed. Below patch should do the trick.

Thanks for the patch.  I'll roll it into my larger one.

-- Dave

Re: [RFC][PATCH] introduce TASK_SIZE_OF() for all arches

[David Howells]

Dave Hansen <haveblue@us.ibm.com> wrote:

> For the /proc/<pid>/pagemap code[1], we need to able to query how
> much virtual address space a particular task has.

How does this relate to NOMMU conditions?

David

Re: [RFC][PATCH] introduce TASK_SIZE_OF() for all arches

[Dave Hansen]

On Wed, 2007-08-22 at 10:06 +0100, David Howells wrote:
> Dave Hansen <haveblue@us.ibm.com> wrote:
> 
> > For the /proc/<pid>/pagemap code[1], we need to able to query how
> > much virtual address space a particular task has.
> 
> How does this relate to NOMMU conditions?

It probably doesn't make any sense to use the /proc/<pid>/pagemap
feature on NOMMU systems.  There are no pagetables to dump, and no
shared pages, right?

-- Dave

Re: [RFC][PATCH] introduce TASK_SIZE_OF() for all arches

[Matt Mackall]

On Wed, Aug 22, 2007 at 03:55:50PM -0700, Dave Hansen wrote:
> On Wed, 2007-08-22 at 10:06 +0100, David Howells wrote:
> > Dave Hansen <haveblue@us.ibm.com> wrote:
> > 
> > > For the /proc/<pid>/pagemap code[1], we need to able to query how
> > > much virtual address space a particular task has.
> > 
> > How does this relate to NOMMU conditions?
> 
> It probably doesn't make any sense to use the /proc/<pid>/pagemap
> feature on NOMMU systems.  There are no pagetables to dump, and no
> shared pages, right?

Nor is there any present/not present to report. VSS == RSS on NOMMU,
right?

So we could provide pagemap, but it's not clear it'd be very
interesting. kpagemap might be interesting, simply because of the
flags though.

-- 
Mathematics is the supreme nostalgia of our time.

Re: [RFC][PATCH] introduce TASK_SIZE_OF() for all arches

[Dave Hansen]

On Wed, 2007-08-22 at 10:06 +0100, David Howells wrote:
> Dave Hansen <haveblue@us.ibm.com> wrote:
> > For the /proc/<pid>/pagemap code[1], we need to able to query how
> > much virtual address space a particular task has.
> 
> How does this relate to NOMMU conditions?

Were you asking about /proc/<pid>/pagemap, or about my TASK_SIZE_OF()
macro?  

/proc/<pid>/pagemap is completely implemented in task_mmu.c, so it
shouldn't show up on NOMMU systems.

As far as TASK_SIZE_OF(), it still makes sense on those systems because
they (at least m68k) still has TASK_SIZE.  Was there a particular
problem you're concerned about?  I'm not very intimately familiar with
the NOMMU code.

-- Dave

Re: [RFC][PATCH] introduce TASK_SIZE_OF() for all arches

[David Howells]

Dave Hansen <haveblue@us.ibm.com> wrote:

> Were you asking about /proc/<pid>/pagemap, or about my TASK_SIZE_OF()
> macro?  

TASK_SIZE_OF().

> As far as TASK_SIZE_OF(), it still makes sense on those systems because
> they (at least m68k) still has TASK_SIZE.

Fair enough.

> Was there a particular problem you're concerned about?  I'm not very
> intimately familiar with the NOMMU code.

Since you were talking about virtual memory, I was wondering as to its
applicability for NOMMU.

David

2.6.23-rc3-git3 make warnings

[Jarek Poplawski]

[-- Attachment #1: Type: text/plain, Size: 1576 bytes --]

Here are some more of, probably well-known, warnings with attached
testing-only .config.

Regards,
Jarek P.


sed: -e expression #1, char 154: Unknown option to 's'
drivers/acpi/ec.c: In function `acpi_ec_ecdt_probe':
drivers/acpi/ec.c:873: warning: passing arg 1 of `acpi_get_devices' discards qualifiers from pointer target type
drivers/pci/search.c: In function `pci_find_slot':
drivers/pci/search.c:99: warning: `pci_find_device' is deprecated (declared at include/linux/pci.h:480)
drivers/pci/search.c: At top level:
drivers/pci/search.c:437: warning: `pci_find_device' is deprecated (declared at drivers/pci/search.c:244)
drivers/pci/search.c:437: warning: `pci_find_device' is deprecated (declared at drivers/pci/search.c:244)
drivers/pci/search.c:438: warning: `pci_find_slot' is deprecated (declared at drivers/pci/search.c:96)
drivers/pci/search.c:438: warning: `pci_find_slot' is deprecated (declared at drivers/pci/search.c:96)
drivers/pci/msi.c:686: warning: weak declaration of `arch_msi_check_device' after first use results in unspecified behavior
drivers/pci/msi.c:698: warning: weak declaration of `arch_setup_msi_irqs' after first use results in unspecified behavior
drivers/pci/msi.c:718: warning: weak declaration of `arch_teardown_msi_irqs' after first use results in unspecified behavior
In file included from drivers/usb/host/ohci-hcd.c:859:
drivers/usb/host/ohci-pci.c: In function `ohci_pci_start':
drivers/usb/host/ohci-pci.c:202: warning: unused variable `pdev'
Root device is (8, 1)
Setup is 11036 bytes (padded to 11264 bytes).
System is 2428 kB

[-- Attachment #2: .config.bz2 --]
[-- Type: application/octet-stream, Size: 7184 bytes --]

Re: 2.6.23-rc3-git3 make warnings

[Randy Dunlap]

On Tue, 21 Aug 2007 15:20:38 +0200 Jarek Poplawski wrote:

> Here are some more of, probably well-known, warnings with attached
> testing-only .config.
> 
> Regards,
> Jarek P.
> 
> 
> sed: -e expression #1, char 154: Unknown option to 's'
> drivers/acpi/ec.c: In function `acpi_ec_ecdt_probe':
> drivers/acpi/ec.c:873: warning: passing arg 1 of `acpi_get_devices' discards qualifiers from pointer target type
> drivers/pci/search.c: In function `pci_find_slot':
> drivers/pci/search.c:99: warning: `pci_find_device' is deprecated (declared at include/linux/pci.h:480)
> drivers/pci/search.c: At top level:
> drivers/pci/search.c:437: warning: `pci_find_device' is deprecated (declared at drivers/pci/search.c:244)
> drivers/pci/search.c:437: warning: `pci_find_device' is deprecated (declared at drivers/pci/search.c:244)
> drivers/pci/search.c:438: warning: `pci_find_slot' is deprecated (declared at drivers/pci/search.c:96)
> drivers/pci/search.c:438: warning: `pci_find_slot' is deprecated (declared at drivers/pci/search.c:96)
> drivers/pci/msi.c:686: warning: weak declaration of `arch_msi_check_device' after first use results in unspecified behavior
> drivers/pci/msi.c:698: warning: weak declaration of `arch_setup_msi_irqs' after first use results in unspecified behavior
> drivers/pci/msi.c:718: warning: weak declaration of `arch_teardown_msi_irqs' after first use results in unspecified behavior
> In file included from drivers/usb/host/ohci-hcd.c:859:
> drivers/usb/host/ohci-pci.c: In function `ohci_pci_start':
> drivers/usb/host/ohci-pci.c:202: warning: unused variable `pdev'
> Root device is (8, 1)
> Setup is 11036 bytes (padded to 11264 bytes).
> System is 2428 kB

What gcc version?  I don't get the arch_ warnings in drivers/pci/msi.c.

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

RFC: drop support for gcc < 4.0

[Adrian Bunk]

On Tue, Aug 21, 2007 at 09:31:03AM -0700, Randy Dunlap wrote:
> On Tue, 21 Aug 2007 15:20:38 +0200 Jarek Poplawski wrote:
> 
> > Here are some more of, probably well-known, warnings with attached
> > testing-only .config.
> >...
> > drivers/pci/msi.c:686: warning: weak declaration of `arch_msi_check_device' after first use results in unspecified behavior
> > drivers/pci/msi.c:698: warning: weak declaration of `arch_setup_msi_irqs' after first use results in unspecified behavior
> > drivers/pci/msi.c:718: warning: weak declaration of `arch_teardown_msi_irqs' after first use results in unspecified behavior
> >...
> 
> What gcc version?  I don't get the arch_ warnings in drivers/pci/msi.c.

Obviously a gcc <= 3.4 [1], and therefore no unit-at-a-time.

You can reproduce it with a more recent gcc when adding 
-fno-unit-at-a-time to the CFLAGS.

And it's becoming a real maintainance problem that not only this problem 
but also other problems like some section mismatches [2] are only 
present without unit-at-a-time.

Currently we support 6 different stable gcc release series, and it might 
be the right time to consider dropping support for the older ones.

Are there any architectures still requiring a gcc < 4.0 ?

> ~Randy

cu
Adrian

[1] unit-at-a-time was added in gcc 3.4, but on gcc 3.4 we disable it on
    i386 due to stack usage problems
[2] example: static __init function with exactly one caller, and this
             caller is non-__init

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: RFC: drop support for gcc < 4.0

[Andi Kleen]

On Tue, Aug 21, 2007 at 07:35:50PM +0200, Adrian Bunk wrote:
> Obviously a gcc <= 3.4 [1], and therefore no unit-at-a-time.

Actually there are widely used 3.3 variants that support unit-at-a-time
(e.g. 3.3-hammer which was shipped by several distributions for some time)

There are still a lot of systems around which use gcc 3.3 (less so with
3.4). Unless there's a major bug that is hard to work around I would
prefer to keep it supported.

Bogus warnings should be relatively harmless.

> And it's becoming a real maintainance problem that not only this problem 
> but also other problems like some section mismatches [2] are only 
> present without unit-at-a-time.

The unit-at-a-time output order is not defined, so even if it works
with the current compiler a compiler change might still trigger
that problem. So it would be better to fix those anyways.

-Andi

Re: RFC: drop support for gcc < 4.0

[Adrian Bunk]

On Tue, Aug 21, 2007 at 09:19:59PM +0200, Andi Kleen wrote:
> On Tue, Aug 21, 2007 at 07:35:50PM +0200, Adrian Bunk wrote:
> > Obviously a gcc <= 3.4 [1], and therefore no unit-at-a-time.
> 
> Actually there are widely used 3.3 variants that support unit-at-a-time
> (e.g. 3.3-hammer which was shipped by several distributions for some time)
> 
> There are still a lot of systems around which use gcc 3.3 (less so with
> 3.4). Unless there's a major bug that is hard to work around I would
> prefer to keep it supported.
> 
> Bogus warnings should be relatively harmless.

How many kernel developers use such old gcc versions?

And how many people notice the valid modpost warnings that can indicate
a runtime Oops?

> > And it's becoming a real maintainance problem that not only this problem 
> > but also other problems like some section mismatches [2] are only 
> > present without unit-at-a-time.
> 
> The unit-at-a-time output order is not defined, so even if it works
> with the current compiler a compiler change might still trigger
> that problem. So it would be better to fix those anyways.

The example [2] from my email is guaranteed to not be a problem with 
unit-at-a-time (as long as unit-at-a-time implies 
inline-functions-called-once - and that's although theoretically 
possible quite unlikely to change in practice).

This example is for a bug that should be fixed, but my point is the
maintainability, IOW: issues with older compilers might not be 
discovered and fixed before they go into a stable kernel.

We currently support 6 different stable gcc release series plus heavily 
modified vendor branches like 3.3-hammer. We can discuss whether it is 
now already the right time, and where to make the cut, but medium-term 
we must reduce the number of supported compilers.

> -Andi

cu
Adrian

[2] example: static __init function with exactly one caller, and this
             caller is non-__init

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [RFC][PATCH] introduce TASK_SIZE_OF() for all arches

[Matthew Wilcox]

On Tue, Aug 21, 2007 at 09:54:33PM +0200, Adrian Bunk wrote:
> We currently support 6 different stable gcc release series plus heavily 
> modified vendor branches like 3.3-hammer. We can discuss whether it is 
> now already the right time, and where to make the cut, but medium-term 
> we must reduce the number of supported compilers.

I don't think that's feasible.  We're dealing with a situation where:

 - GCC aim to release a new compiler series every 6 months.
   Fortunately, they don't achieve this goal, but they do release every
   12 months or thereabouts [1].
 - Enterprise distros are supported for seven years
 - We still care about people being able to compile kernels on
   enterprise distros that are still supported by their vendor.

Yes, it causes us some pain to support all these different compilers,
but it's not *that* big a pain.

[1] Release dates, according to the GCC website

2007-05-17 4.2.0	(14 months)
2006-02-28 4.1.0	(10 months)
2005-04-20 4.0.0	(12 months)
2004-04-20 3.4.0	(11 months)
2003-05-20 3.3		(12 months, ignoring gcc 3.2 which was really 3.1.2)
2002-05-15 3.1		(11 months)
2001-06-18 3.0

-- 
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."