Re: [2.6 patch] i386: always use 4k stacks

Re: [2.6 patch] i386: always use 4k stacks

[Jeff V. Merkey]

Andrew,

Thanks. I concur.

Jeff

Andrew Morton wrote:

>Adrian Bunk <bunk@stusta.de> wrote:
>  
>
>>This patch was already sent on:
>>- 11 Dec 2005
>>- 5 Dec 2005
>>- 30 Nov 2005
>>- 23 Nov 2005
>>- 14 Nov 2005
>>    
>>
>
>Sigh.  I saw the volume of email last time and though "gee, glad I wasn't
>cc'ed on that lot".
>
>Supporting 8k stacks is a small amount of code and nobody has seen a need
>to make changes in there for quite a long time.  So there's little cost to
>keeping the existing code.
>
>And the existing code is useful:
>
>a) people can enable it to confirm that their weird crash was due to a
>   stack overflow.
>
>b) If I was going to put together a maximally-stable kernel for a
>   complex server machine, I'd select 8k stacks.  We're still just too
>   squeezy, and we've had too many relatively-recent overflows, and there
>   are still some really deep callpaths in there.
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>  
>

Re: [2.6 patch] i386: always use 4k stacks

[Jeff V. Merkey]

Adrian Bunk wrote:

>On Thu, Dec 15, 2005 at 02:00:13PM -0800, Andrew Morton wrote:
>  
>
>>Adrian Bunk <bunk@stusta.de> wrote:
>>    
>>
>>>This patch was already sent on:
>>>- 11 Dec 2005
>>>- 5 Dec 2005
>>>- 30 Nov 2005
>>>- 23 Nov 2005
>>>- 14 Nov 2005
>>>      
>>>
>>Sigh.  I saw the volume of email last time and though "gee, glad I wasn't
>>cc'ed on that lot".
>>    
>>
>
>If you substract the "this breaks my binary-only M$ Windows driver" 
>emails there's not much volume left.
>
>  
>
>>Supporting 8k stacks is a small amount of code and nobody has seen a need
>>to make changes in there for quite a long time.  So there's little cost to
>>keeping the existing code.
>>
>>And the existing code is useful:
>>
>>a) people can enable it to confirm that their weird crash was due to a
>>   stack overflow.
>>
>>b) If I was going to put together a maximally-stable kernel for a
>>   complex server machine, I'd select 8k stacks.  We're still just too
>>   squeezy, and we've had too many relatively-recent overflows, and there
>>   are still some really deep callpaths in there.
>>    
>>
>
>a1) People turn off 4k stacks and never report the problem / noone 
>    really debugs and fixes the reported problem.
>
>Me threatening people with enabling 4k stacks for everyone already 
>resulted in several fixes.
>
>An how many weird crashes with _different_ causes have you seen?
>It could be that there are only _very_ few problems that noone really 
>debugs brcause disabling 4k stacks fixes the issue.
>  
>

When you are on the phone with an irrate customer at 2:00 am in the 
morning, and just turning off your broken 4K stack fix
and getting the customer running matters. 4K stacks are a BAD idea. I 
have even found USER SPACE apps
that crash linux without the 8K option. Andrew has spoken. Suck it up 
and deal with it. It's not a problem limited to Windows
drivers.

Jeff

>cu
>Adrian
>
>  
>

[2.6 patch] i386: always use 4k stacks

[Adrian Bunk]

It seems most problems with 4k stacks are already resolved at least
in -mm.

I'd like to see this patch to always use 4k stacks in -mm now for 
finding any remaining problems before submitting this patch for Linus' 
tree.


Signed-off-by: Adrian Bunk <bunk@stusta.de>
Acked-by: Arjan van de Ven <arjan@infradead.org>

---

This patch was already sent on:
- 11 Dec 2005
- 5 Dec 2005
- 30 Nov 2005
- 23 Nov 2005
- 14 Nov 2005

 arch/i386/Kconfig.debug        |   10 ----------
 arch/i386/kernel/irq.c         |   10 ----------
 include/asm-i386/irq.h         |   11 +++--------
 include/asm-i386/module.h      |    8 +-------
 include/asm-i386/thread_info.h |    6 +-----
 5 files changed, 5 insertions(+), 40 deletions(-)

--- linux-2.6.14-mm2-full/arch/i386/Kconfig.debug.old	2005-11-14 01:30:54.000000000 +0100
+++ linux-2.6.14-mm2-full/arch/i386/Kconfig.debug	2005-11-14 01:31:06.000000000 +0100
@@ -52,16 +52,6 @@
 	  portion of the kernel code won't be covered by a 2MB TLB anymore.
 	  If in doubt, say "N".
 
-config 4KSTACKS
-	bool "Use 4Kb for kernel stacks instead of 8Kb"
-	depends on DEBUG_KERNEL
-	help
-	  If you say Y here the kernel will use a 4Kb stacksize for the
-	  kernel stack attached to each process/thread. This facilitates
-	  running more threads on a system and also reduces the pressure
-	  on the VM subsystem for higher order allocations. This option
-	  will also use IRQ stacks to compensate for the reduced stackspace.
-
 config X86_FIND_SMP_CONFIG
 	bool
 	depends on X86_LOCAL_APIC || X86_VOYAGER
--- linux-2.6.14-mm2-full/include/asm-i386/irq.h.old	2005-11-14 01:31:18.000000000 +0100
+++ linux-2.6.14-mm2-full/include/asm-i386/irq.h	2005-11-14 01:31:29.000000000 +0100
@@ -27,14 +27,9 @@
 # define ARCH_HAS_NMI_WATCHDOG		/* See include/linux/nmi.h */
 #endif
 
-#ifdef CONFIG_4KSTACKS
-  extern void irq_ctx_init(int cpu);
-  extern void irq_ctx_exit(int cpu);
-# define __ARCH_HAS_DO_SOFTIRQ
-#else
-# define irq_ctx_init(cpu) do { } while (0)
-# define irq_ctx_exit(cpu) do { } while (0)
-#endif
+extern void irq_ctx_init(int cpu);
+extern void irq_ctx_exit(int cpu);
+#define __ARCH_HAS_DO_SOFTIRQ
 
 #ifdef CONFIG_IRQBALANCE
 extern int irqbalance_disable(char *str);
--- linux-2.6.14-mm2-full/include/asm-i386/thread_info.h.old	2005-11-14 01:31:45.000000000 +0100
+++ linux-2.6.14-mm2-full/include/asm-i386/thread_info.h	2005-11-14 01:32:11.000000000 +0100
@@ -53,11 +53,7 @@
 #endif
 
 #define PREEMPT_ACTIVE		0x10000000
-#ifdef CONFIG_4KSTACKS
-#define THREAD_SIZE            (4096)
-#else
-#define THREAD_SIZE		(8192)
-#endif
+#define THREAD_SIZE		(4096)
 
 #define STACK_WARN             (THREAD_SIZE/8)
 /*
--- linux-2.6.14-mm2-full/include/asm-i386/module.h.old	2005-11-14 01:32:18.000000000 +0100
+++ linux-2.6.14-mm2-full/include/asm-i386/module.h	2005-11-14 01:32:42.000000000 +0100
@@ -64,12 +64,6 @@
 #define MODULE_REGPARM ""
 #endif
 
-#ifdef CONFIG_4KSTACKS
-#define MODULE_STACKSIZE "4KSTACKS "
-#else
-#define MODULE_STACKSIZE ""
-#endif
-
-#define MODULE_ARCH_VERMAGIC MODULE_PROC_FAMILY MODULE_REGPARM MODULE_STACKSIZE
+#define MODULE_ARCH_VERMAGIC MODULE_PROC_FAMILY MODULE_REGPARM
 
 #endif /* _ASM_I386_MODULE_H */

--- linux-2.6.15-rc5-mm2-full/arch/i386/kernel/irq.c.old	2005-12-11 15:10:27.000000000 +0100
+++ linux-2.6.15-rc5-mm2-full/arch/i386/kernel/irq.c	2005-12-11 15:11:29.000000000 +0100
@@ -33,7 +33,6 @@
 }
 #endif
 
-#ifdef CONFIG_4KSTACKS
 /*
  * per-CPU IRQ handling contexts (thread information and stack)
  */
@@ -44,7 +43,6 @@
 
 static union irq_ctx *hardirq_ctx[NR_CPUS];
 static union irq_ctx *softirq_ctx[NR_CPUS];
-#endif
 
 /*
  * do_IRQ handles all normal device IRQ's (the special
@@ -55,10 +53,8 @@
 {	
 	/* high bits used in ret_from_ code */
 	int irq = regs->orig_eax & 0xff;
-#ifdef CONFIG_4KSTACKS
 	union irq_ctx *curctx, *irqctx;
 	u32 *isp;
-#endif
 
 	irq_enter();
 #ifdef CONFIG_DEBUG_STACKOVERFLOW
@@ -76,8 +72,6 @@
 	}
 #endif
 
-#ifdef CONFIG_4KSTACKS
-
 	curctx = (union irq_ctx *) current_thread_info();
 	irqctx = hardirq_ctx[smp_processor_id()];
 
@@ -104,7 +98,6 @@
 			: "memory", "cc", "ecx"
 		);
 	} else
-#endif
 		__do_IRQ(irq, regs);
 
 	irq_exit();
@@ -114,8 +107,6 @@
 	return 1;
 }
 
-#ifdef CONFIG_4KSTACKS
-
 /*
  * These should really be __section__(".bss.page_aligned") as well, but
  * gcc's 3.0 and earlier don't handle that correctly.
@@ -200,7 +191,6 @@
 }
 
 EXPORT_SYMBOL(do_softirq);
-#endif
 
 /*
  * Interrupt statistics:

Re: [2.6 patch] i386: always use 4k stacks

[Jeff V. Merkey]

Lee Revell wrote:

>On Thu, 2005-12-15 at 14:07 -0700, Jeff V. Merkey wrote:
>  
>
>>When you are on the phone with an irrate customer at 2:00 am in the 
>>morning, and just turning off your broken 4K stack fix
>>and getting the customer running matters. 
>>    
>>
>
>Bugzilla link please.  Otherwise STFU.
>  
>

??????

Jeff

>Lee
>
>
>  
>

Re: [2.6 patch] i386: always use 4k stacks

[Andrew Morton]

Adrian Bunk <bunk@stusta.de> wrote:
>
> This patch was already sent on:
> - 11 Dec 2005
> - 5 Dec 2005
> - 30 Nov 2005
> - 23 Nov 2005
> - 14 Nov 2005

Sigh.  I saw the volume of email last time and though "gee, glad I wasn't
cc'ed on that lot".

Supporting 8k stacks is a small amount of code and nobody has seen a need
to make changes in there for quite a long time.  So there's little cost to
keeping the existing code.

And the existing code is useful:

a) people can enable it to confirm that their weird crash was due to a
   stack overflow.

b) If I was going to put together a maximally-stable kernel for a
   complex server machine, I'd select 8k stacks.  We're still just too
   squeezy, and we've had too many relatively-recent overflows, and there
   are still some really deep callpaths in there.

Re: [2.6 patch] i386: always use 4k stacks

[Jeff V. Merkey]

Lee Revell wrote:

>On Thu, 2005-12-15 at 14:46 -0700, Jeff V. Merkey wrote:
>  
>
>>Lee Revell wrote:
>>
>>    
>>
>>>On Thu, 2005-12-15 at 14:07 -0700, Jeff V. Merkey wrote:
>>> 
>>>
>>>      
>>>
>>>>When you are on the phone with an irrate customer at 2:00 am in the 
>>>>morning, and just turning off your broken 4K stack fix
>>>>and getting the customer running matters. 
>>>>   
>>>>
>>>>        
>>>>
>>>Bugzilla link please.  Otherwise STFU.
>>> 
>>>
>>>      
>>>
>>??????
>>
>>Jeff
>>    
>>
>
>You imply that your customer's problem was due to a kernel bug triggered
>by CONFIG_4KSTACKS.  I am asking you to provide a link to the bug report
>or get lost.
>
>Lee
>  
>

You hack on this code base (hack is the right word) -- I sell it, 
service and support it with customers in a dozen countries.  I don't report
company level issues in "bugzilla" or anywhere else public unless they 
apply to kernel code.  calls from several of our apps (which use
larger than 4K kernel space on a stack) from user space crash -- so do 
wireless drivers -- and kdb crashes as well with some bugs with 4K stacks
turned on when you are trying to debug something. 

Hope that addresses your concerns "joe job".

Jeff

>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>  
>

Re: [2.6 patch] i386: always use 4k stacks

[Adrian Bunk]

On Thu, Dec 15, 2005 at 02:00:13PM -0800, Andrew Morton wrote:
> Adrian Bunk <bunk@stusta.de> wrote:
> >
> > This patch was already sent on:
> > - 11 Dec 2005
> > - 5 Dec 2005
> > - 30 Nov 2005
> > - 23 Nov 2005
> > - 14 Nov 2005
> 
> Sigh.  I saw the volume of email last time and though "gee, glad I wasn't
> cc'ed on that lot".

If you substract the "this breaks my binary-only M$ Windows driver" 
emails there's not much volume left.

> Supporting 8k stacks is a small amount of code and nobody has seen a need
> to make changes in there for quite a long time.  So there's little cost to
> keeping the existing code.
> 
> And the existing code is useful:
> 
> a) people can enable it to confirm that their weird crash was due to a
>    stack overflow.
> 
> b) If I was going to put together a maximally-stable kernel for a
>    complex server machine, I'd select 8k stacks.  We're still just too
>    squeezy, and we've had too many relatively-recent overflows, and there
>    are still some really deep callpaths in there.

a1) People turn off 4k stacks and never report the problem / noone 
    really debugs and fixes the reported problem.

Me threatening people with enabling 4k stacks for everyone already 
resulted in several fixes.

An how many weird crashes with _different_ causes have you seen?
It could be that there are only _very_ few problems that noone really 
debugs brcause disabling 4k stacks fixes the issue.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [2.6 patch] i386: always use 4k stacks

[Jeff V. Merkey]

Fri, 16 Dec 2005 01:08:02 +0100 wrote:

>El Thu, 15 Dec 2005 15:04:38 -0700,
>"Jeff V. Merkey" <jmerkey@wolfmountaingroup.com> escribió:
>
>  
>
>>apply to kernel code.  calls from several of our apps (which use
>>larger than 4K kernel space on a stack) from user space crash -- so do 
>>wireless drivers -- and kdb crashes as well with some bugs with 4K stacks
>>turned on when you are trying to debug something. 
>>    
>>
>
>If you (or other people) don't report those bugs, nobody else except
>you will care about them, I'm afraid.
>
>"My customer says it crashes but I don't want to report it publically".
>What kind of excuse is that? O_o
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>  
>
You need to go back and read the whole thread.  These bugs were reported 
by me weeks ago.

Jeff

Re: [2.6 patch] i386: always use 4k stacks

[Lee Revell]

On Thu, 2005-12-15 at 14:07 -0700, Jeff V. Merkey wrote:
> When you are on the phone with an irrate customer at 2:00 am in the 
> morning, and just turning off your broken 4K stack fix
> and getting the customer running matters. 

Bugzilla link please.  Otherwise STFU.

Lee

Re: [2.6 patch] i386: always use 4k stacks

[Dave Jones]

On Thu, Dec 15, 2005 at 11:30:00PM +0100, Adrian Bunk wrote:

 > An how many weird crashes with _different_ causes have you seen?
 > It could be that there are only _very_ few problems that noone really 
 > debugs brcause disabling 4k stacks fixes the issue.

the block layer issue that Neil had patches for was the only one
that rings any bells for me[*] (and the only one in Fedora bugzilla
that anyone has actually hit -- and that's 2-3 people out of
a *lot* of users).

		Dave

[*] Plus a few XFS ones, but that's been a lost cause wrt stack usage
for a long time -- people were reporting overflows there before we
enabled 4K stacks.

Re: [2.6 patch] i386: always use 4k stacks

[Lee Revell]

On Thu, 2005-12-15 at 14:46 -0700, Jeff V. Merkey wrote:
> Lee Revell wrote:
> 
> >On Thu, 2005-12-15 at 14:07 -0700, Jeff V. Merkey wrote:
> >  
> >
> >>When you are on the phone with an irrate customer at 2:00 am in the 
> >>morning, and just turning off your broken 4K stack fix
> >>and getting the customer running matters. 
> >>    
> >>
> >
> >Bugzilla link please.  Otherwise STFU.
> >  
> >
> 
> ??????
> 
> Jeff

You imply that your customer's problem was due to a kernel bug triggered
by CONFIG_4KSTACKS.  I am asking you to provide a link to the bug report
or get lost.

Lee

Re: [2.6 patch] i386: always use 4k stacks

[Ismail Donmez]

Cuma 16 Aralık 2005 00:04 tarihinde şunları yazmıştınız:
> Lee Revell wrote:
> >On Thu, 2005-12-15 at 14:46 -0700, Jeff V. Merkey wrote:
> >>Lee Revell wrote:
> >>>On Thu, 2005-12-15 at 14:07 -0700, Jeff V. Merkey wrote:
> >>>>When you are on the phone with an irrate customer at 2:00 am in the
> >>>>morning, and just turning off your broken 4K stack fix
> >>>>and getting the customer running matters.
> >>>
> >>>Bugzilla link please.  Otherwise STFU.
> >>
> >>??????
> >>
> >>Jeff
> >
> >You imply that your customer's problem was due to a kernel bug triggered
> >by CONFIG_4KSTACKS.  I am asking you to provide a link to the bug report
> >or get lost.
> >
> >Lee
>
> You hack on this code base (hack is the right word) -- I sell it,
> service and support it with customers in a dozen countries.  I don't report
> company level issues in "bugzilla" or anywhere else public unless they
> apply to kernel code.  calls from several of our apps (which use
> larger than 4K kernel space on a stack) from user space crash -- so do
> wireless drivers -- and kdb crashes as well with some bugs with 4K stacks
> turned on when you are trying to debug something.
>
> Hope that addresses your concerns "joe job".

You are supposed to report those bugs in a manner it won't conflict with the 
privacy of your customer(s). Linux distros do this already.

/ismail

Re: [2.6 patch] i386: always use 4k stacks

[Fri, 16 Dec 2005 01:08:02 +0100]

El Thu, 15 Dec 2005 15:04:38 -0700,
"Jeff V. Merkey" <jmerkey@wolfmountaingroup.com> escribió:

> apply to kernel code.  calls from several of our apps (which use
> larger than 4K kernel space on a stack) from user space crash -- so do 
> wireless drivers -- and kdb crashes as well with some bugs with 4K stacks
> turned on when you are trying to debug something. 

If you (or other people) don't report those bugs, nobody else except
you will care about them, I'm afraid.

"My customer says it crashes but I don't want to report it publically".
What kind of excuse is that? O_o

Re: [2.6 patch] i386: always use 4k stacks

[Ray Lee]

(Man, I've been holding my tongue on this conversation for a while,
but it seems my better angels have deserted me.)

On 12/15/05, Lee Revell <rlrevell@joe-job.com> wrote:
> Bugzilla link please.

No, that's not how failure engineering is done. A guy designing a
bridge doesn't cut all the supports back to the bare minimum just to
save money because his design says that the remaining metal should be
strong enough. If you can't prove it, and it's a safety issue
(continuing my analogy in the physical world), then you engineer for
failure. Can you handle all occurrences? No, a hurricane Katrina comes
along every once in a while. Can you weather more than you did before?
Yes. In the meantime, their are fewer poor sods falling off the bridge
that have to open a bugzilla report.

The world of software is no different. If someone wants to remove the
8k stacks option, they'd better prove that they're making my servers
more reliable. I've seen zero arguments for why 8k stacks is unviable.
(I've also wondered why we can't just have IRQ stacks plus 8k thread
stacks -- seemingly the best of both worlds) Instead, what I've seen
is that we have coders who don't like the idea of any non-order-zero
allocations taking place, because big systems running poorly coded
Java apps with massive threading can hit problems with allocations
from time to time.

The answer for that is the same answer the kernel community usually
gives about poorly designed userspace applications: rewrite them.

I'm quite open to being proved wrong. If someone has a counter case
they can toss forth, please do so. Systems taking lots of interrupts?
Then how about 8k + IRQ stacks? With a counterexample I'll gladly
concede that I'm an ignorant slut[*] -- excuse me, Saturday Night Live
flashbacks -- an ignorant git, and shut up. ([*] is only half right,
I'm not all that ignorant).

If someone doesn't show a counter case, then may I suggest people
consider the possibility that this is not proper engineering. Prove
it, or provide a safety blanket. But don't yank the blanket without
proving the lack of problem.

Ray

Re: [2.6 patch] i386: always use 4k stacks

[Alan Cox]

On Iau, 2005-12-15 at 15:04 -0700, Jeff V. Merkey wrote:
> apply to kernel code.  calls from several of our apps (which use
> larger than 4K kernel space on a stack) 

Then you've got bugs anyway. In 8K stack mode that stack is shared with
the IRQ/BH/etc stack so you've only got 4K to play with. Its just more
random whether your box explodes.

Re: [2.6 patch] i386: always use 4k stacks

[Adrian Bunk]

On Thu, Dec 15, 2005 at 06:15:38PM -0500, Dave Jones wrote:
> On Thu, Dec 15, 2005 at 11:30:00PM +0100, Adrian Bunk wrote:
> 
>  > An how many weird crashes with _different_ causes have you seen?
>  > It could be that there are only _very_ few problems that noone really 
>  > debugs brcause disabling 4k stacks fixes the issue.
> 
> the block layer issue that Neil had patches for was the only one
> that rings any bells for me[*] (and the only one in Fedora bugzilla
> that anyone has actually hit -- and that's 2-3 people out of
> a *lot* of users).

Neil's patch is required, and since it's not in 2.6.15-rc we might still 
get bug reports with 4k stacks that are fixed by his patch.

Do we have any bug reports due to 4k stacks against -mm since Neil's 
patch was included?

People were able to convince me in the past to delay my patch to always 
use 4k stacks by pointing to unsolved problems (or I pointed them like 
in the reiser4 case) - and these were constructive delays since the code 
was fixed. So if someone wants to convince me that it's too early for my 
patch, simply send me some pointers to 4k stack issues still present in 
a recent -mm.  :-)

Hm, I just found two serious stack usage regressions in 2.6.15-rc (bug 
report will be in a separate email), but allocating arrays with more 
than 2000 elements on the stack is always wrong in the kernel 
independent of the stack size...

> 		Dave
> 
> [*] Plus a few XFS ones, but that's been a lost cause wrt stack usage
> for a long time -- people were reporting overflows there before we
> enabled 4K stacks.

I remember someone from the XFS maintainers (Nathan?) saying they 
believe having solved all XFS stack issues.

If there are any XFS issues left, do you have a pointer to them?

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [2.6 patch] i386: always use 4k stacks

[Dave Jones]

On Fri, Dec 16, 2005 at 01:47:40AM +0100, Adrian Bunk wrote:

 > > [*] Plus a few XFS ones, but that's been a lost cause wrt stack usage
 > > for a long time -- people were reporting overflows there before we
 > > enabled 4K stacks.
 > 
 > I remember someone from the XFS maintainers (Nathan?) saying they 
 > believe having solved all XFS stack issues.
 > 
 > If there are any XFS issues left, do you have a pointer to them?

The last one I saw may have been actually been more related
to the block layer problem. iirc that was a user NFS exporting
XFS on a raid1 array.

		Dave

Re: [2.6 patch] i386: always use 4k stacks

[Michael Buesch]

[-- Attachment #1: Type: text/plain, Size: 869 bytes --]

On Friday 16 December 2005 01:08, you wrote:
> El Thu, 15 Dec 2005 15:04:38 -0700,
> "Jeff V. Merkey" <jmerkey@wolfmountaingroup.com> escribió:
> 
> > apply to kernel code.  calls from several of our apps (which use
> > larger than 4K kernel space on a stack) from user space crash -- so do 
> > wireless drivers -- and kdb crashes as well with some bugs with 4K stacks
> > turned on when you are trying to debug something. 
> 
> If you (or other people) don't report those bugs, nobody else except
> you will care about them, I'm afraid.
> 
> "My customer says it crashes but I don't want to report it publically".
> What kind of excuse is that? O_o

Your customer runs an -mm kernel on his production systems?
Smash him.
This is about removing 8k support in the -mm kernel, to
find the remaining bugs (if there are any).

-- 
Greetings Michael.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [2.6 patch] i386: always use 4k stacks

[Nathan Scott]

On Fri, Dec 16, 2005 at 01:47:40AM +0100, Adrian Bunk wrote:
> ...
> > [*] Plus a few XFS ones, but that's been a lost cause wrt stack usage
> > for a long time -- people were reporting overflows there before we
> > enabled 4K stacks.
> 
> I remember someone from the XFS maintainers (Nathan?) saying they 
> believe having solved all XFS stack issues.

We don't know of any remaining issues...

> If there are any XFS issues left, do you have a pointer to them?

...so I was curious to see these too, since we've never had any
reported from Dave / anyone else @RH.

cheers.

-- 
Nathan

Re: [2.6 patch] i386: always use 4k stacks

[Dave Jones]

On Fri, Dec 16, 2005 at 12:15:19PM +1100, Nathan Scott wrote:
 > On Fri, Dec 16, 2005 at 01:47:40AM +0100, Adrian Bunk wrote:
 > > ...
 > > > [*] Plus a few XFS ones, but that's been a lost cause wrt stack usage
 > > > for a long time -- people were reporting overflows there before we
 > > > enabled 4K stacks.
 > > 
 > > I remember someone from the XFS maintainers (Nathan?) saying they 
 > > believe having solved all XFS stack issues.
 > 
 > We don't know of any remaining issues...
 > 
 > > If there are any XFS issues left, do you have a pointer to them?
 > 
 > ...so I was curious to see these too, since we've never had any
 > reported from Dave / anyone else @RH.

When they've come up in bugzilla, I've pointed them at the xfs
mailing lists. As these folks can reproduce the problems, it seemed
pointless to play middle-man.

		Dave

Re: [2.6 patch] i386: always use 4k stacks

[Zwane Mwaikambo]

On Thu, 15 Dec 2005, Adrian Bunk wrote:

> On Thu, Dec 15, 2005 at 02:00:13PM -0800, Andrew Morton wrote:
> 
> > Supporting 8k stacks is a small amount of code and nobody has seen a need
> > to make changes in there for quite a long time.  So there's little cost to
> > keeping the existing code.
> > 
> > And the existing code is useful:
> > 
> > a) people can enable it to confirm that their weird crash was due to a
> >    stack overflow.
> > 
> > b) If I was going to put together a maximally-stable kernel for a
> >    complex server machine, I'd select 8k stacks.  We're still just too
> >    squeezy, and we've had too many relatively-recent overflows, and there
> >    are still some really deep callpaths in there.
> 
> a1) People turn off 4k stacks and never report the problem / noone 
>     really debugs and fixes the reported problem.
> 
> Me threatening people with enabling 4k stacks for everyone already 
> resulted in several fixes.

How about this, we apply this patch and perhaps add some debug option to 
enable 8k by changing THREAD_SIZE. This way we have the seperate interrupt 
stacks and 8k stacks for when someone suspects a stack overflow.

Re: [2.6 patch] i386: always use 4k stacks

[Neil Brown]

On Thursday December 15, davej@redhat.com wrote:
> On Fri, Dec 16, 2005 at 01:47:40AM +0100, Adrian Bunk wrote:
> 
>  > > [*] Plus a few XFS ones, but that's been a lost cause wrt stack usage
>  > > for a long time -- people were reporting overflows there before we
>  > > enabled 4K stacks.
>  > 
>  > I remember someone from the XFS maintainers (Nathan?) saying they 
>  > believe having solved all XFS stack issues.
>  > 
>  > If there are any XFS issues left, do you have a pointer to them?
> 
> The last one I saw may have been actually been more related
> to the block layer problem. iirc that was a user NFS exporting
> XFS on a raid1 array.

Yeh, I've noticed that nfsd seems to figure often in these.  As nfsd
lives on the same (in-kernel) stack as the filesystem and device
drives, it will add a couple of hundred bytes to the call trace.

A typical nfsd call trace is
 nfsd -> svc_process -> nfsd_dispatch -> nfsd3_proc_write ->
   nfsd_write ->nfsd_vfs_write -> vfs_writev

(errr. nfsd_vfs_write is inline, large, and called twice, that ain't
good)

These add up to over 300 bytes on the stack.
Looking at each of these, I see that nfsd_write (which includes
 nfsd_vfs_write) contributes 0x8c to stack usage itself!!

It turns out this is because it puts a 'struct iattr' on the stack so
it can kill suid if needed.  The following patch saves about 50 bytes
off the stack in this call path.

I sometimes wish that gcc could be told to optimise for stack usage -
a lot of variables on the stack are dead at some call points, yet they
stay there using space anyway.  The only way to save this space seem
to be to move the code which uses those variable into a separate
function, but we really shouldn't *have* to do these optimisations by
hand!

NeilBrown

Signed-off-by: Neil Brown <neilb@suse.de>

### Diffstat output
 ./fs/nfsd/vfs.c |   20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)

diff ./fs/nfsd/vfs.c~current~ ./fs/nfsd/vfs.c
--- ./fs/nfsd/vfs.c~current~	2005-12-12 16:00:40.000000000 +1100
+++ ./fs/nfsd/vfs.c	2005-12-16 13:48:31.000000000 +1100
@@ -869,6 +869,16 @@ out:
 	return err;
 }
 
+static void kill_suid(struct dentry *dentry)
+{
+	struct iattr	ia;
+	ia.ia_valid = ATTR_KILL_SUID | ATTR_KILL_SGID;
+
+	down(&dentry->d_inode->i_sem);
+	notify_change(dentry, &ia);
+	up(&dentry->d_inode->i_sem);
+}
+
 static inline int
 nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp, struct file *file,
 				loff_t offset, struct kvec *vec, int vlen,
@@ -922,14 +932,8 @@ nfsd_vfs_write(struct svc_rqst *rqstp, s
 	}
 
 	/* clear setuid/setgid flag after write */
-	if (err >= 0 && (inode->i_mode & (S_ISUID | S_ISGID))) {
-		struct iattr	ia;
-		ia.ia_valid = ATTR_KILL_SUID | ATTR_KILL_SGID;
-
-		down(&inode->i_sem);
-		notify_change(dentry, &ia);
-		up(&inode->i_sem);
-	}
+	if (err >= 0 && (inode->i_mode & (S_ISUID | S_ISGID)))
+		kill_suid(dentry);
 
 	if (err >= 0 && stable) {
 		static ino_t	last_ino;

Re: [2.6 patch] i386: always use 4k stacks

[Dave Jones]

On Fri, Dec 16, 2005 at 01:56:58PM +1100, Neil Brown wrote:

 > It turns out this is because it puts a 'struct iattr' on the stack so
 > it can kill suid if needed.  The following patch saves about 50 bytes
 > off the stack in this call path.

See! it *was* worth Adrian bringing up the "kill 8kb stacks" patch again :-)

		Dave

Re: [2.6 patch] i386: always use 4k stacks

[Adrian Bunk]

On Fri, Dec 16, 2005 at 01:56:58PM +1100, Neil Brown wrote:
> On Thursday December 15, davej@redhat.com wrote:
> > On Fri, Dec 16, 2005 at 01:47:40AM +0100, Adrian Bunk wrote:
> > 
> >  > > [*] Plus a few XFS ones, but that's been a lost cause wrt stack usage
> >  > > for a long time -- people were reporting overflows there before we
> >  > > enabled 4K stacks.
> >  > 
> >  > I remember someone from the XFS maintainers (Nathan?) saying they 
> >  > believe having solved all XFS stack issues.
> >  > 
> >  > If there are any XFS issues left, do you have a pointer to them?
> > 
> > The last one I saw may have been actually been more related
> > to the block layer problem. iirc that was a user NFS exporting
> > XFS on a raid1 array.
> 
> Yeh, I've noticed that nfsd seems to figure often in these.  As nfsd
> lives on the same (in-kernel) stack as the filesystem and device
> drives, it will add a couple of hundred bytes to the call trace.
> 
> A typical nfsd call trace is
>  nfsd -> svc_process -> nfsd_dispatch -> nfsd3_proc_write ->
>    nfsd_write ->nfsd_vfs_write -> vfs_writev
> 
> (errr. nfsd_vfs_write is inline, large, and called twice, that ain't
> good)

The nfsd code uses inline in too many places.

gcc can figure out itself that static functions called only once should 
be inline (except currently on i386 due to no-unit-at-a-time, see 
below).

> These add up to over 300 bytes on the stack.
> Looking at each of these, I see that nfsd_write (which includes
>  nfsd_vfs_write) contributes 0x8c to stack usage itself!!
> 
> It turns out this is because it puts a 'struct iattr' on the stack so
> it can kill suid if needed.  The following patch saves about 50 bytes
> off the stack in this call path.
>...

This works currently on i386 (and only on i386) because we are using 
-fno-unit-at-a-time there.

In the medium-term, we want to get rid of no-unit-at-a-time because this 
makes the code both bigger and slower, and I'm therefore not a big fan 
of this kind of workarounds.

If this struct is really a problem (which I doubt considering it's 
size), I'd prefer it being kmalloc'ed.

> NeilBrown
>...

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [2.6 patch] i386: always use 4k stacks

[Diego Calleja]

El Thu, 15 Dec 2005 14:00:13 -0800,
Andrew Morton <akpm@osdl.org> escribió:


> Supporting 8k stacks is a small amount of code and nobody has seen a need
> to make changes in there for quite a long time.  So there's little cost to
> keeping the existing code.
> 
> And the existing code is useful:

Maybe this slighty different approach is better? 



Signed-off-by: Diego Calleja <diegocg@gmail.com>

Index: test/arch/i386/Kconfig.debug
===================================================================
--- test.orig/arch/i386/Kconfig.debug	2005-12-16 13:59:54.000000000 +0100
+++ test/arch/i386/Kconfig.debug	2005-12-16 14:03:27.000000000 +0100
@@ -42,15 +42,16 @@
 	  This results in a large slowdown, but helps to find certain types
 	  of memory corruptions.
 
-config 4KSTACKS
-	bool "Use 4Kb for kernel stacks instead of 8Kb"
+config 8KSTACKS
+	bool "Use 8Kb for kernel stacks instead of 4Kb"
 	depends on DEBUG_KERNEL
 	help
-	  If you say Y here the kernel will use a 4Kb stacksize for the
-	  kernel stack attached to each process/thread. This facilitates
-	  running more threads on a system and also reduces the pressure
+	  If you say Y here the kernel will use a 8Kb stacksize for the
+	  kernel stack attached to each process/thread. This makes harder
+	  to overflow the stack, and it's used to debug possible stack
+	  overflow problems. Notice that this increases the pressure
 	  on the VM subsystem for higher order allocations. This option
-	  will also use IRQ stacks to compensate for the reduced stackspace.
+	  will also disable IRQ stacks.
 
 config X86_FIND_SMP_CONFIG
 	bool
Index: test/arch/i386/kernel/irq.c
===================================================================
--- test.orig/arch/i386/kernel/irq.c	2005-12-16 13:59:54.000000000 +0100
+++ test/arch/i386/kernel/irq.c	2005-12-16 14:01:24.000000000 +0100
@@ -33,7 +33,7 @@
 }
 #endif
 
-#ifdef CONFIG_4KSTACKS
+#ifndef CONFIG_8KSTACKS
 /*
  * per-CPU IRQ handling contexts (thread information and stack)
  */
@@ -55,7 +55,7 @@
 {	
 	/* high bits used in ret_from_ code */
 	int irq = regs->orig_eax & 0xff;
-#ifdef CONFIG_4KSTACKS
+#ifndef CONFIG_8KSTACKS
 	union irq_ctx *curctx, *irqctx;
 	u32 *isp;
 #endif
@@ -76,7 +76,7 @@
 	}
 #endif
 
-#ifdef CONFIG_4KSTACKS
+#ifndef CONFIG_8KSTACKS
 
 	curctx = (union irq_ctx *) current_thread_info();
 	irqctx = hardirq_ctx[smp_processor_id()];
@@ -112,7 +112,7 @@
 	return 1;
 }
 
-#ifdef CONFIG_4KSTACKS
+#ifndef CONFIG_8KSTACKS
 
 /*
  * These should really be __section__(".bss.page_aligned") as well, but
Index: test/include/asm-i386/irq.h
===================================================================
--- test.orig/include/asm-i386/irq.h	2005-12-16 13:59:54.000000000 +0100
+++ test/include/asm-i386/irq.h	2005-12-16 14:04:05.000000000 +0100
@@ -27,7 +27,7 @@
 # define ARCH_HAS_NMI_WATCHDOG		/* See include/linux/nmi.h */
 #endif
 
-#ifdef CONFIG_4KSTACKS
+#ifndef CONFIG_8KSTACKS
   extern void irq_ctx_init(int cpu);
   extern void irq_ctx_exit(int cpu);
 # define __ARCH_HAS_DO_SOFTIRQ
Index: test/include/asm-i386/module.h
===================================================================
--- test.orig/include/asm-i386/module.h	2005-12-16 13:59:54.000000000 +0100
+++ test/include/asm-i386/module.h	2005-12-16 14:04:36.000000000 +0100
@@ -64,8 +64,8 @@
 #define MODULE_REGPARM ""
 #endif
 
-#ifdef CONFIG_4KSTACKS
-#define MODULE_STACKSIZE "4KSTACKS "
+#ifdef CONFIG_8KSTACKS
+#define MODULE_STACKSIZE "8KSTACKS "
 #else
 #define MODULE_STACKSIZE ""
 #endif
Index: test/include/asm-i386/thread_info.h
===================================================================
--- test.orig/include/asm-i386/thread_info.h	2005-12-16 13:59:54.000000000 +0100
+++ test/include/asm-i386/thread_info.h	2005-12-16 14:04:57.000000000 +0100
@@ -53,7 +53,7 @@
 #endif
 
 #define PREEMPT_ACTIVE		0x10000000
-#ifdef CONFIG_4KSTACKS
+#ifndef CONFIG_8KSTACKS
 #define THREAD_SIZE            (4096)
 #else
 #define THREAD_SIZE		(8192)

Re: [2.6 patch] i386: always use 4k stacks

[Adrian Bunk]

On Fri, Dec 16, 2005 at 02:10:02PM +0100, Diego Calleja wrote:
> El Thu, 15 Dec 2005 14:00:13 -0800,
> Andrew Morton <akpm@osdl.org> escribió:
> 
> 
> > Supporting 8k stacks is a small amount of code and nobody has seen a need
> > to make changes in there for quite a long time.  So there's little cost to
> > keeping the existing code.
> > 
> > And the existing code is useful:
> 
> Maybe this slighty different approach is better? 
>...

My count of bug reports for problems with 4k stacks after Neil's patch
went into -mm is still at 0.

Either there are no problems left or noone pays attention to them since 
disabling 4k stacks "fixed" the problem.

In both cases there's no reason against applying my patch.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [2.6 patch] i386: always use 4k stacks

[Adrian Bunk]

On Thu, Dec 15, 2005 at 05:49:45PM -0800, Zwane Mwaikambo wrote:
> On Thu, 15 Dec 2005, Adrian Bunk wrote:
> 
> > On Thu, Dec 15, 2005 at 02:00:13PM -0800, Andrew Morton wrote:
> > 
> > > Supporting 8k stacks is a small amount of code and nobody has seen a need
> > > to make changes in there for quite a long time.  So there's little cost to
> > > keeping the existing code.
> > > 
> > > And the existing code is useful:
> > > 
> > > a) people can enable it to confirm that their weird crash was due to a
> > >    stack overflow.
> > > 
> > > b) If I was going to put together a maximally-stable kernel for a
> > >    complex server machine, I'd select 8k stacks.  We're still just too
> > >    squeezy, and we've had too many relatively-recent overflows, and there
> > >    are still some really deep callpaths in there.
> > 
> > a1) People turn off 4k stacks and never report the problem / noone 
> >     really debugs and fixes the reported problem.
> > 
> > Me threatening people with enabling 4k stacks for everyone already 
> > resulted in several fixes.
> 
> How about this, we apply this patch and perhaps add some debug option to 
> enable 8k by changing THREAD_SIZE. This way we have the seperate interrupt 
> stacks and 8k stacks for when someone suspects a stack overflow.

You can always manually change THREAD_SIZE using a text editor.

My count of bug reports for problems with in-kernel code with 4k stacks 
after Neil's patch went into -mm is still at 0.

Either there are no problems left or noone pays attention to them since 
disabling 4k stacks "fixed" the problem. And not having an option makes 
it more likely that we get reports for and people interested in fixes 
for the latter.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [2.6 patch] i386: always use 4k stacks

[linux-os (Dick Johnson)]

On Thu, 15 Dec 2005, Lee Revell wrote:

> On Thu, 2005-12-15 at 14:46 -0700, Jeff V. Merkey wrote:
>> Lee Revell wrote:
>>
>>> On Thu, 2005-12-15 at 14:07 -0700, Jeff V. Merkey wrote:
>>>
>>>
>>>> When you are on the phone with an irrate customer at 2:00 am in the
>>>> morning, and just turning off your broken 4K stack fix
>>>> and getting the customer running matters.
>>>>
>>>>
>>>
>>> Bugzilla link please.  Otherwise STFU.
>>>
>>>
>>
>> ??????
>>
>> Jeff
>
> You imply that your customer's problem was due to a kernel bug triggered
> by CONFIG_4KSTACKS.  I am asking you to provide a link to the bug report
> or get lost.
>
> Lee

Throughout the past two years of 4k stack-wars, I never heard why
such a small stack was needed (not wanted, needed). It seems that
everybody "knows" that smaller is better and most everybody thinks
that one page in ix86 land is "optimum". However I don't think
anybody ever even tried to analyze what was better from a technical
perspective. Instead it's been analyzed as religious dogma, i.e.,
keep the stack small, it will prevent idiots from doing bad things.

I'm fairly sure that if you started from scratch and decided to
write a new operating system, your choice of a stack-size would
probably be something like 64k. I have no clue why somebody
decided to use a 4k stack and force their choice upon others.
And, yes, I am well aware that each system-call requires a
seperate stack upon entry and it even needs to keep that stack
while sleeping.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13.4 on an i686 machine (5589.56 BogoMips).
Warning : 98.36% of all statistics are fiction.
.

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: [2.6 patch] i386: always use 4k stacks

[Xavier Bestel]

On Fri, 2005-12-16 at 15:39, linux-os (Dick Johnson) wrote:

> Throughout the past two years of 4k stack-wars, I never heard why
> such a small stack was needed (not wanted, needed).

Because after some prolonged uptime, memory can be heavily fragmented.
In this case an order-0 allocation will always succeed (as long as some
memory is free), whereas an order-1 allocation may easily fail.

	Xav

Re: [2.6 patch] i386: always use 4k stacks

[Adrian Bunk]

On Fri, Dec 16, 2005 at 09:39:56AM -0500, linux-os (Dick Johnson) wrote:
> 
> Throughout the past two years of 4k stack-wars, I never heard why
> such a small stack was needed (not wanted, needed). It seems that
> everybody "knows" that smaller is better and most everybody thinks
> that one page in ix86 land is "optimum". However I don't think
> anybody ever even tried to analyze what was better from a technical
> perspective. Instead it's been analyzed as religious dogma, i.e.,
> keep the stack small, it will prevent idiots from doing bad things.
>...

It seems you missed the discussion of this issue last month.

Arjan had a good list of all technical advantages of 4k stacks:
  http://www.ussg.iu.edu/hypermail/linux/kernel/0511.2/0042.html

> Cheers,
> Dick Johnson

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [2.6 patch] i386: always use 4k stacks

[Diego Calleja]

El Fri, 16 Dec 2005 15:04:25 +0100,
Adrian Bunk <bunk@stusta.de> escribió:

> My count of bug reports for problems with 4k stacks after Neil's patch
> went into -mm is still at 0.
> 
> Either there are no problems left or noone pays attention to them since 
> disabling 4k stacks "fixed" the problem.
> 
> In both cases there's no reason against applying my patch.

I know, but there's too much resistance to the "pure" 4kb patch. The
8 KB patch does the same thing (enables 4kb stacks)  and at the same
time the 8kb groupies can't flamewar you for it, it covers akpm's
concerns, it puts some pressure on the ndiswrapper guys and leaves
time for the broadcom driver developers to finish, merge and push
to the distributions their driver. The 8kb config option can be
removed in the future when we're sure that it's 100% safe (neil
brown's patch isn''t a good sign). It makes every happy IMO ;)

Re: [2.6 patch] i386: always use 4k stacks

[Kyle Moffett]

On Dec 16, 2005, at 10:35, Diego Calleja wrote:
> I know, but there's too much resistance to the "pure" 4kb patch.

I have yet to see any resistance to the 4Kb patch this time around  
that was not "*whine* don't break my ndiswrapper plz".   There are  
extremely few remaining 4kb stack bugs, and 4k stacks have a sizeable  
list of technical advantages.


> The 8 KB patch does the same thing (enables 4kb stacks)

The point is to force it in -mm so most people can't just disable it  
because it fixes their problem.  We want 8k stacks to go away, and  
the only way to get out the last issues before sending to mainline is  
by forcing it in -mm.


> and at the same time the 8kb groupies can't flamewar you for it

This matter how?


> it covers akpm's concerns

I get the impression that they are already sufficiently addressed,  
although Andrew should feel free to correct me if I'm wrong :-D.


> it puts some pressure on the ndiswrapper guys

And removing 8kb all-together doesn't do this?


> and leaves time for the broadcom driver developers to finish, merge  
> and push to the distributions their driver

It's working partially now.  This is the time when we should really  
try to force ndiswrapper junkies over to the driver to get it tested  
and bugfixed for inclusion.


> The 8kb config option can be removed in the future when we're sure  
> that it's 100% safe (neil brown's patch isn''t a good sign). It  
> makes every happy IMO ;)

We're not trying to get it removed from mainline (yet), just from - 
mm, which has never been anywhere close to 100% safe _anyways_.  If  
it breaks lots of things horribly, it will get reverted and  
development will continue as normal, but that's probably not going to  
be the case.

Cheers,
Kyle Moffett

--
Debugging is twice as hard as writing the code in the first place.   
Therefore, if you write the code as cleverly as possible, you are, by  
definition, not smart enough to debug it.
   -- Brian Kernighan

Re: [2.6 patch] i386: always use 4k stacks

[Adrian Bunk]

On Fri, Dec 16, 2005 at 04:35:03PM +0100, Diego Calleja wrote:
> El Fri, 16 Dec 2005 15:04:25 +0100,
> Adrian Bunk <bunk@stusta.de> escribió:
> 
> > My count of bug reports for problems with 4k stacks after Neil's patch
> > went into -mm is still at 0.
> > 
> > Either there are no problems left or noone pays attention to them since 
> > disabling 4k stacks "fixed" the problem.
> > 
> > In both cases there's no reason against applying my patch.
> 
> I know, but there's too much resistance to the "pure" 4kb patch. The
> 8 KB patch does the same thing (enables 4kb stacks)  and at the same
> time the 8kb groupies can't flamewar you for it, it covers akpm's

I have no problems with people flaming me.

I had problems if people would actually find technical reasons where my 
patch breaks in-kernel code.  ;-)

> concerns, it puts some pressure on the ndiswrapper guys and leaves
> time for the broadcom driver developers to finish, merge and push
> to the distributions their driver. The 8kb config option can be
> removed in the future when we're sure that it's 100% safe (neil
> brown's patch isn''t a good sign). It makes every happy IMO ;)

Neil's patch fixes the last known poroblems.

My count of bug reports for problems with 4k stacks after Neil's patch
went into -mm is still at 0.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [2.6 patch] i386: always use 4k stacks

[Michael Buesch]

[-- Attachment #1: Type: text/plain, Size: 955 bytes --]

On Friday 16 December 2005 16:49, you wrote:
> > and leaves time for the broadcom driver developers to finish, merge  
> > and push to the distributions their driver
> 
> It's working partially now.  This is the time when we should really  
> try to force ndiswrapper junkies over to the driver to get it tested  
> and bugfixed for inclusion.

Partially means:
- Connections on CCK Rates (802.11b) are stable,
  if the signal quality is good (you are close to the AP).
- No encryption support, yet (I am working on it, but it's
  a bit difficult)

Now, I want to test bcm43xx on 4k stacks. But only have a
ppc32 machine with such a broadcom card. ppc32 has 8k stacks.
How am I supposed to test the driver for 4kstack conformance?
Given this, why aren't there people working on 4kstacks for
ppc32? Is it not needed there, or did simply nobody care to
do this now?
Thanks for your nonflaming suggestions. ;)

-- 
Greetings Michael.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [2.6 patch] i386: always use 4k stacks

[Arjan van de Ven]

On Fri, 2005-12-16 at 14:10 +0100, Diego Calleja wrote:
> El Thu, 15 Dec 2005 14:00:13 -0800,
> Andrew Morton <akpm@osdl.org> escribió:
> 
> 
> > Supporting 8k stacks is a small amount of code and nobody has seen a need
> > to make changes in there for quite a long time.  So there's little cost to
> > keeping the existing code.
> > 
> > And the existing code is useful:
> 
> Maybe this slighty different approach is better? 
> 
> 
> 
> Signed-off-by: Diego Calleja <diegocg@gmail.com>


I like this one; it makes the default 4K while leaving the 8K option for
those who really want it...

Acked-by: Arjan van de Ven <arjan@infradead.org>

Fwd: [2.6 patch] i386: always use 4k stacks

[Mike Snitzer]

---------- Forwarded message ----------
From: Mike Snitzer <snitzer@gmail.com>
Date: Dec 16, 2005 12:02 PM
Subject: Re: [2.6 patch] i386: always use 4k stacks
To: Arjan van de Ven <arjan@infradead.org>
Cc: Diego Calleja <diegocg@gmail.com>, Andrew Morton <akpm@osdl.org>,
bunk@stusta.de, linux-kernel@vger.kernel.org



On 12/16/05, Arjan van de Ven <arjan@infradead.org> wrote:
>  On Fri, 2005-12-16 at 14:10 +0100, Diego Calleja wrote:
> > El Thu, 15 Dec 2005 14:00:13 -0800,
> > Andrew Morton <akpm@osdl.org> escribió:
> >
> >
> > > Supporting 8k stacks is a small amount of code and nobody has seen a need
> > > to make changes in there for quite a long time.  So there's little cost to
> > > keeping the existing code.
> > >
> > > And the existing code is useful:
> >
> > Maybe this slighty different approach is better?
> >
> >
> >
> > Signed-off-by: Diego Calleja <diegocg@gmail.com>
>
>
> I like this one; it makes the default 4K while leaving the 8K option for
> those who really want it...
>
> Acked-by: Arjan van de Ven <arjan@infradead.org>

 Any chance that IRQ stacks could be made available with 8K?  Set
default to 4K but expose the _option_ for choice of 8K+IRQ stacks?  I
realize it takes away from the fanatical push toward imposed 4K purity
but Linux is really all about options.

Re: [2.6 patch] i386: always use 4k stacks

[Giridhar Pemmasani]

Kyle Moffett wrote:

> I have yet to see any resistance to the 4Kb patch this time around
> that was not "*whine* don't break my ndiswrapper plz".   There are

I haven't seen anyone demanding others not to have 4k stacks; only requests
to leave 4k/8k stack option as it is. If _you_ want to have 4k stacks, you
already have that option. You are only pushing what you want on others and
bad mouthing people that are requesting the option to have either 4k or 8k
stacks.

> It's working partially now.  This is the time when we should really

ndiswrapper is used not just for broadcom. There are plenty of other
chipsets that don't even have a project started to write open source
driver.

> try to force ndiswrapper junkies over to the driver to get it tested
                           ^^^^^^^
Shame on you. Your last mail was a promise to be "more reserved". Even
otherwise, such profanities against a group of people are unwarranted.

To kernel developers: I have earlier requested if it is possible to create
threads with different stack sizes (e.g., 4k/8k/16k etc.) at run-time, ala
FreeBSD. In that case, one could chose whatever it is that fits their
needs. Any comments on this idea?

To those that depend on ndiswrapper to have wireless in Linux: A few days
ago I started working on NDIS implementation in user space. However, it
will take considerable time before this is usable. Moreover, I only have
hope with USB drivers. PCI/mini-PCI/PCMCIA drivers need to run interrupt
service routines, which can't be run in user space, so they won't work in
user space.

Giri

Re: [2.6 patch] i386: always use 4k stacks

[Arjan van de Ven]

On Fri, 2005-12-16 at 13:18 -0500, Giridhar Pemmasani wrote:
> Kyle Moffett wrote:
> 
> > I have yet to see any resistance to the 4Kb patch this time around
> > that was not "*whine* don't break my ndiswrapper plz".   There are
> 
> I haven't seen anyone demanding others not to have 4k stacks; only requests
> to leave 4k/8k stack option as it is. 

in hindsight making this a config option was a mistake. Why? Because
we're not making every single patch we add to the kernel a config
option, nor should it be. Config options for drivers or expensive debug
options are fine, debug options for random patches... aren't really. To
be fair the config option was intended to be really temporary, like 1
kernel release, until it was sure there were no kinks. Oh well, there's
too many people moaning now about ndiswrapper that I fear we'll never
get rid of it.

And no I do not think a kernel with 9000 config options is still useful;
not every single trivial thing should be a config option.

Re: [2.6 patch] i386: always use 4k stacks

[Horst von Brand]

linux-os \(Dick Johnson\) <linux-os@analogic.com> wrote:

[...]

> Throughout the past two years of 4k stack-wars, I never heard why
> such a small stack was needed (not wanted, needed). It seems that
> everybody "knows" that smaller is better and most everybody thinks
> that one page in ix86 land is "optimum". However I don't think
> anybody ever even tried to analyze what was better from a technical
> perspective. Instead it's been analyzed as religious dogma, i.e.,
> keep the stack small, it will prevent idiots from doing bad things.

OK, so here goes again...

The kernel stack has to be contiguous in /physical/ memory. Keep the stack
/one/ page, that way you can always get a new stack when needed (== each
fork(2) or clone(2)). If the stack is 2 (or more) pages, you'll have to
find (or create) a multi-page free area, and (fragmentation being what it
is, and Linux routinely running for months at a time) you are in a whole
new world of pain.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: [2.6 patch] i386: always use 4k stacks

[Oliver Neukum]

Am Freitag, 16. Dezember 2005 19:42 schrieb Horst von Brand:
> linux-os \(Dick Johnson\) <linux-os@analogic.com> wrote:
> 
> [...]
> 
> > Throughout the past two years of 4k stack-wars, I never heard why
> > such a small stack was needed (not wanted, needed). It seems that
> > everybody "knows" that smaller is better and most everybody thinks
> > that one page in ix86 land is "optimum". However I don't think
> > anybody ever even tried to analyze what was better from a technical
> > perspective. Instead it's been analyzed as religious dogma, i.e.,
> > keep the stack small, it will prevent idiots from doing bad things.
> 
> OK, so here goes again...
> 
> The kernel stack has to be contiguous in /physical/ memory. Keep the stack
> /one/ page, that way you can always get a new stack when needed (== each
> fork(2) or clone(2)). If the stack is 2 (or more) pages, you'll have to
> find (or create) a multi-page free area, and (fragmentation being what it
> is, and Linux routinely running for months at a time) you are in a whole
> new world of pain.

How about ignoring physical pages and going to virtual, say, 16K pages?
After all, 4K is 15 years old. Disks and RAM have grown enormously.

	Regards
		Oliver

Re: [2.6 patch] i386: always use 4k stacks

[Brian Gerst]

Horst von Brand wrote:
> linux-os \(Dick Johnson\) <linux-os@analogic.com> wrote:
> 
> [...]
> 
> 
>>Throughout the past two years of 4k stack-wars, I never heard why
>>such a small stack was needed (not wanted, needed). It seems that
>>everybody "knows" that smaller is better and most everybody thinks
>>that one page in ix86 land is "optimum". However I don't think
>>anybody ever even tried to analyze what was better from a technical
>>perspective. Instead it's been analyzed as religious dogma, i.e.,
>>keep the stack small, it will prevent idiots from doing bad things.
> 
> 
> OK, so here goes again...
> 
> The kernel stack has to be contiguous in /physical/ memory. Keep the stack
> /one/ page, that way you can always get a new stack when needed (== each
> fork(2) or clone(2)). If the stack is 2 (or more) pages, you'll have to
> find (or create) a multi-page free area, and (fragmentation being what it
> is, and Linux routinely running for months at a time) you are in a whole
> new world of pain.

So what about arches where single-page stacks aren't viable (for example 
x86_64)?  Are we just screwed?

--
				Brian Gerst

Re: [2.6 patch] i386: always use 4k stacks

[Steven Rostedt]

On Fri, 2005-12-16 at 15:42 -0300, Horst von Brand wrote:
> linux-os \(Dick Johnson\) <linux-os@analogic.com> wrote:
> 
> [...]
> 
> > Throughout the past two years of 4k stack-wars, I never heard why
> > such a small stack was needed (not wanted, needed). It seems that
> > everybody "knows" that smaller is better and most everybody thinks
> > that one page in ix86 land is "optimum". However I don't think
> > anybody ever even tried to analyze what was better from a technical
> > perspective. Instead it's been analyzed as religious dogma, i.e.,
> > keep the stack small, it will prevent idiots from doing bad things.
> 
> OK, so here goes again...
> 
> The kernel stack has to be contiguous in /physical/ memory. Keep the stack
> /one/ page, that way you can always get a new stack when needed (== each
> fork(2) or clone(2)). If the stack is 2 (or more) pages, you'll have to
> find (or create) a multi-page free area, and (fragmentation being what it
> is, and Linux routinely running for months at a time) you are in a whole
> new world of pain.

So people should really be asking for a PAGE_SIZE = 8k option ;)

Sorry,

-- Steve

Re: [2.6 patch] i386: always use 4k stacks

[Arjan van de Ven]

> 
> So what about arches where single-page stacks aren't viable (for example 
> x86_64)?  Are we just screwed?


x86 is specially handicapped due to the fact that the stacks need to be
in the lowmem zone. Even if you have 8Gb ram, the lowmem zone is still
800Mb and a bit, and this gets to be under a high pressure, like
hyper-fragmentation. Same for bounce buffers etc etc.

note that the order thing is by far not the only advantage, pure memory
usage alone and cache locality also are wins. The memory usage halves
for kernel stacks after all (which means you can do more threads in
java, or use the memory for disk cache ;)

Re: [2.6 patch] i386: always use 4k stacks

[Horst von Brand]

Adrian Bunk <bunk@stusta.de> wrote:
> On Fri, Dec 16, 2005 at 01:56:58PM +1100, Neil Brown wrote:

[...]

> gcc can figure out itself that static functions called only once should 
> be inline (except currently on i386 due to no-unit-at-a-time, see 
> below).
> 
> > These add up to over 300 bytes on the stack.
> > Looking at each of these, I see that nfsd_write (which includes
> >  nfsd_vfs_write) contributes 0x8c to stack usage itself!!
> > 
> > It turns out this is because it puts a 'struct iattr' on the stack so
> > it can kill suid if needed.  The following patch saves about 50 bytes
> > off the stack in this call path.
> >...

And if you set up a compound literal for the task then? It is just used to
shove data into the called function.

My short test case (attached) has a smaller stack with the compound
literal (gcc-4.1, Fedora rawhide on i686), and IMHO it is clearer what is
going on here.

> This works currently on i386 (and only on i386) because we are using 
> -fno-unit-at-a-time there.
> 
> In the medium-term, we want to get rid of no-unit-at-a-time because this 
> makes the code both bigger and slower, and I'm therefore not a big fan 
> of this kind of workarounds.
> 
> If this struct is really a problem (which I doubt considering it's 
> size), I'd prefer it being kmalloc'ed.

Nodz.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: [2.6 patch] i386: always use 4k stacks

[Oliver Neukum]

Am Freitag, 16. Dezember 2005 20:02 schrieb Arjan van de Ven:
> 
> > 
> > So what about arches where single-page stacks aren't viable (for example 
> > x86_64)?  Are we just screwed?
> 
> 
> x86 is specially handicapped due to the fact that the stacks need to be
> in the lowmem zone. Even if you have 8Gb ram, the lowmem zone is still
> 800Mb and a bit, and this gets to be under a high pressure, like
> hyper-fragmentation. Same for bounce buffers etc etc.
> 
> note that the order thing is by far not the only advantage, pure memory
> usage alone and cache locality also are wins. The memory usage halves
> for kernel stacks after all (which means you can do more threads in
> java, or use the memory for disk cache ;)

1. Cache usage depends on actual stack usage. How much you allocate
doesn't matter
2. You are surely getting a cache effect by using interrupt stacks. Which
is larger?
3. When you use kmalloc instead of the stack you are reducing locality.

	Regards
		Oliver

Re: [2.6 patch] i386: always use 4k stacks

[linux-os (Dick Johnson)]

On Fri, 16 Dec 2005, Horst von Brand wrote:

> linux-os \(Dick Johnson\) <linux-os@analogic.com> wrote:
>
> [...]
>
>> Throughout the past two years of 4k stack-wars, I never heard why
>> such a small stack was needed (not wanted, needed). It seems that
>> everybody "knows" that smaller is better and most everybody thinks
>> that one page in ix86 land is "optimum". However I don't think
>> anybody ever even tried to analyze what was better from a technical
>> perspective. Instead it's been analyzed as religious dogma, i.e.,
>> keep the stack small, it will prevent idiots from doing bad things.
>
> OK, so here goes again...
>
> The kernel stack has to be contiguous in /physical/ memory. Keep the stack
> /one/ page, that way you can always get a new stack when needed (== each
> fork(2) or clone(2)). If the stack is 2 (or more) pages, you'll have to
> find (or create) a multi-page free area, and (fragmentation being what it
> is, and Linux routinely running for months at a time) you are in a whole
> new world of pain.

The interrupt stack needs to be non-paged. Are you sure the user-stacks
need to be 'physical', non-paged too? If so, that's probably the
problem. All addresses accessed by the CPUs in the kernel are virtual
which means one needs some mapping anyway.

So, why can't one map non-contiguous pages into the kernel-user-stack?
That entry-into-the-kernel stack is the one that's giving everybody
fits because it needs to remain allocated, even for sleeping tasks.

If it was virtual, built just like other data, it could be made up
from any available RAM and, in the case of a preemptive kernel,
even swapped!

In that case, the total amount of real RAM you actually need
is defined only by the number of concurrent tasks in a preemptive
kernel, plus the page(s) for the interrupt stack. That's far lower
than the 'N' pages times everybody who forked and slept, which is
what we seem to have now.

FYI, there is nothing wrong with a 2-level stack, i.e., the
system-call occurs on the interrupt stack, then the user-kernel
stack gets allocated from paged RAM.

Now I know this isn't VMS, but in VMS we didn't have anything
that really needed to be contiguous except for the interrupt
stack (which was arbitrarily 64k). And on VAXen the pages were
tiny 512 things so you really need to use paged RAM for just
about everything.

> --
> Dr. Horst H. von Brand                   User #22616 counter.li.org
> Departamento de Informatica                     Fono: +56 32 654431
> Universidad Tecnica Federico Santa Maria              +56 32 654239
> Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
>

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13.4 on an i686 machine (5589.56 BogoMips).
Warning : 98.36% of all statistics are fiction.
.

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: [2.6 patch] i386: always use 4k stacks

[Mike Snitzer]

On 12/16/05, Arjan van de Ven <arjan@infradead.org> wrote:
> On Fri, 2005-12-16 at 13:18 -0500, Giridhar Pemmasani wrote:
> > Kyle Moffett wrote:
> >
> > > I have yet to see any resistance to the 4Kb patch this time around
> > > that was not "*whine* don't break my ndiswrapper plz".   There are
> >
> > I haven't seen anyone demanding others not to have 4k stacks; only requests
> > to leave 4k/8k stack option as it is.
>
> in hindsight making this a config option was a mistake. Why? Because
> we're not making every single patch we add to the kernel a config
> option, nor should it be. Config options for drivers or expensive debug
> options are fine, debug options for random patches... aren't really. To
> be fair the config option was intended to be really temporary, like 1
> kernel release, until it was sure there were no kinks. Oh well, there's
> too many people moaning now about ndiswrapper that I fear we'll never
> get rid of it.
>
> And no I do not think a kernel with 9000 config options is still useful;
> not every single trivial thing should be a config option.

You're using overly generalized assertions to try to convince others
that the configurability of a particularly important (to some, albeit
not you) config option is unnecessary.  4K vs 8K is hardly a "trivial"
configuration option of the Linux kernel.  At this point in time it
has not been sufficiently demonstrated that 4K "just works".

Taking a step back, I'm all for -mm being a 4K only tree to force the
issue; but even once all in-tree code is deemed 4K clean people still
may want to be extremely cautious by enabling 8K stacks (possibly
_with_ IRQ stacks).  Its merely a question of can/will Linux (or some
vendor) provide this level of stack overflow safety as is; or does one
need to patch the kernel to get the desired safety?  IF upstream
kernel.org doesn't even provide the knobs to ensure safety at all
costs (and vendors like Redhat have people at the helm who are
advocating 4K stacks in the "Enterprise" Linux kernel configurations
of the world) how does one get a Linux kernel that provides a sizable
safety net that is _SUPPORTED_ for true enterprise-grade applications?

Simply put 4K vs 8K is not as trivial a decision as you'd have people believe.

Mike

Re: [2.6 patch] i386: always use 4k stacks

[Arjan van de Ven]

> You're using overly generalized assertions to try to convince others

I'm no longer interested in arguing removing this thing. Too much
whining by ndiswrapper addicts [1].

> that the configurability of a particularly important (to some, albeit
> not you) config option is unnecessary.  4K vs 8K is hardly a "trivial"
> configuration option of the Linux kernel. 

Compared to many of the other changes that went in since 2.6.0? 4K
stacks is a minor change. There's no config option for 4 level
pagetables for example, and that's a far more invasive change in many
ways. 

>  At this point in time it
> has not been sufficiently demonstrated that 4K "just works".

Excuse me?
Fedora released 3 distributions with it enabled, and Red Hat uses it in
an enterprise distribution. That's a whopping lot of users right there
with a very wide range of workloads.

>  kernel to get the desired safety?  IF upstream
> kernel.org doesn't even provide the knobs to ensure safety at all
> costs (and vendors like Redhat have people at the helm who are
> advocating 4K stacks in the "Enterprise" Linux kernel configurations
> of the world) how does one get a Linux kernel that provides a sizable
> safety net that is _SUPPORTED_ for true enterprise-grade applications?

eh I don't know if you paid attention, but Red Hat Enterprise Linux 4
only has 4Kb stacks kernels... so that covers your supported true
enterprise-grade application thing. 

> Simply put 4K vs 8K is not as trivial a decision as you'd have people believe.

that's too simply put in fact, especially if you look at it
historically. It's a bit of irony that part of the reason 4K stacks was
developed was that the 2.4 kernels ran out of stack space for customers
occasionally (just as example look at lkml this week, there was a report
about such an overflow there as well). Remember that 4K+4K has more
stack space than the 4K+2K as 2.4 kernels have. Sure 2.6 bumped this to
5.5k/2.5k roughly in the "8K" case, but fundamentally the change to
4k/4k isn't all that big even inside 2.6. 

You can go on and keep painting this as a cowboy development, but it
really isn't....



[1] Yes addicts; binary drivers are in many ways similar to heroin;
they're really hard to get rid of for example and highly addictive, they
also cause some people to act like junkies-in-withdrawl when their
binary driver breaks, or when someone suggests breaking it.

Re: Fwd: [2.6 patch] i386: always use 4k stacks

[Alan Cox]

On Gwe, 2005-12-16 at 12:17 -0500, Mike Snitzer wrote:
>  Any chance that IRQ stacks could be made available with 8K?  Set
> default to 4K but expose the _option_ for choice of 8K+IRQ stacks?  I
> realize it takes away from the fanatical push toward imposed 4K purity
> but Linux is really all about options.

That would just encourage people to be sloppy with their stack usage and
make the problem worse.

Alan

Re: [2.6 patch] i386: always use 4k stacks

[Alan Cox]

On Gwe, 2005-12-16 at 17:23 +0100, Michael Buesch wrote:
> Now, I want to test bcm43xx on 4k stacks. But only have a
> ppc32 machine with such a broadcom card. ppc32 has 8k stacks.
> How am I supposed to test the driver for 4kstack conformance?

Unless you've been writing fairly careless code putting a lot of objects
on stack a driver is going to work fine with 4K stacks.

> Given this, why aren't there people working on 4kstacks for
> ppc32? Is it not needed there, or did simply nobody care to
> do this now?

AFAIK nobody is working on 4K stack for PPC32. I've no idea myself if it
is needed or useful there. In terms of debugging if your code exceeds a
4K stack you'll find out quite rapidly from x86 users. One thing the
seperate IRQ stacks mean is that stack overflows generally show up as
overflows and consistently rather than as weird crashes when timing
co-incides between your heavy stack usage and IRQ heavy stack usage, at
which point the mess is rarely repeatable or debuggable

Re: [2.6 patch] i386: always use 4k stacks

[Arjan van de Ven]

On Fri, 2005-12-16 at 20:02 +0000, Alan Cox wrote:
> On Gwe, 2005-12-16 at 17:23 +0100, Michael Buesch wrote:
> > Now, I want to test bcm43xx on 4k stacks. But only have a
> > ppc32 machine with such a broadcom card. ppc32 has 8k stacks.
> > How am I supposed to test the driver for 4kstack conformance?
> 
> Unless you've been writing fairly careless code putting a lot of objects
> on stack a driver is going to work fine with 4K stacks.

there is also "make checkstack" that works on many architectures, and
lists offenders. If you're clean there it's very likely you're very
ok ;)

Re: [2.6 patch] i386: always use 4k stacks

[David Lang]

On Fri, 16 Dec 2005, linux-os (Dick Johnson) wrote:

> On Fri, 16 Dec 2005, Horst von Brand wrote:
>
>> linux-os \(Dick Johnson\) <linux-os@analogic.com> wrote:
>>
>> [...]
>>
>>> Throughout the past two years of 4k stack-wars, I never heard why
>>> such a small stack was needed (not wanted, needed). It seems that
>>> everybody "knows" that smaller is better and most everybody thinks
>>> that one page in ix86 land is "optimum". However I don't think
>>> anybody ever even tried to analyze what was better from a technical
>>> perspective. Instead it's been analyzed as religious dogma, i.e.,
>>> keep the stack small, it will prevent idiots from doing bad things.
>>
>> OK, so here goes again...
>>
>> The kernel stack has to be contiguous in /physical/ memory. Keep the
> stack
>> /one/ page, that way you can always get a new stack when needed (==
> each
>> fork(2) or clone(2)). If the stack is 2 (or more) pages, you'll have
> to
>> find (or create) a multi-page free area, and (fragmentation being what
> it
>> is, and Linux routinely running for months at a time) you are in a
> whole
>> new world of pain.
>
> The interrupt stack needs to be non-paged. Are you sure the user-stacks
> need to be 'physical', non-paged too? If so, that's probably the
> problem. All addresses accessed by the CPUs in the kernel are virtual
> which means one needs some mapping anyway.

actually, the kernel always uses real addresses, userspace uses virtual 
addresses.

This came up recently with the page tables, Linus said that he was 
absolutly opposed to adding the complication and overhead of changine the 
kernel to user virtual addresses instead of real addresses for it's data 
structures. it would add an extra level of redirection to just about every 
memory access (which also means an additional load on the cache to store 
the mapping info to resolve this redirection). The performance hit for 
this would be considerable.

David Lang

Re: [2.6 patch] i386: always use 4k stacks

[Mike Snitzer]

Arjan,

I'm aware that RedHat has been shipping FC and RHEL kernels with 4K
stacks for some time.  The recent 4K stack fixes pretty much establish
that RedHat's early adoption of 4K stacks was a "cowbody development"
no?  I don't think RedHat kept similar 4K stack fixes from upstream,
so a bit of luck maybe?  I do agree that at this point the 4K-only
proposal is NOT an overly rash decision given continued adequate
vetting in -mm.  But even then there may be untested workloads that
expose stack issues.  Which is perfectly fine if users have the option
to use _supported_ alternate configs.

I got the sense that you were trying to paint me as something of a
closet "ndiswrapper addict".  Amusing and all but my motivations for
requesting continued options on the stack size are rooted in concerns
that long call chains can and still do result when running kernel.org
and RHEL4 kernels under particular workloads.  An example workload
being cluster filesystems that in a single call chain historically
_could_ leverage iptables + RPC (tcp) + DM (LVM) + etc.

Given Neil Brown's fix for the block layer these stack-heavy workloads
that included DM in the call chain need to be revisited.  However, the
savings associated with those particular fixes still may not leave
sufficient breathing room.  The logic that all users must NOW provide
workloads which undermine 4K stack viability otherwise the 8K option
will be completely removed _seems_ quite irrational (even though we
are _supposedly_ just talking about doing so in -mm).

All of us appreciate the desire to have Linux be more efficient and 4K
stacks will get us that.  If it comes with the cost of instability
under more exotic workloads then the bad outweighs the perceived good
of imposed 4K stacks.  With RHEL4 it would seem we're past the point
of no-return for supported 8K stacks.  I'm merely advocating upstream
give users the 8K+IRQ stack _options_ and set the default to 4K.

Mike


On 12/16/05, Arjan van de Ven <arjan@infradead.org> wrote:
>
> > You're using overly generalized assertions to try to convince others
>
> I'm no longer interested in arguing removing this thing. Too much
> whining by ndiswrapper addicts [1].
>
> > that the configurability of a particularly important (to some, albeit
> > not you) config option is unnecessary.  4K vs 8K is hardly a "trivial"
> > configuration option of the Linux kernel.
>
> Compared to many of the other changes that went in since 2.6.0? 4K
> stacks is a minor change. There's no config option for 4 level
> pagetables for example, and that's a far more invasive change in many
> ways.
>
> >  At this point in time it
> > has not been sufficiently demonstrated that 4K "just works".
>
> Excuse me?
> Fedora released 3 distributions with it enabled, and Red Hat uses it in
> an enterprise distribution. That's a whopping lot of users right there
> with a very wide range of workloads.
>
> >  kernel to get the desired safety?  IF upstream
> > kernel.org doesn't even provide the knobs to ensure safety at all
> > costs (and vendors like Redhat have people at the helm who are
> > advocating 4K stacks in the "Enterprise" Linux kernel configurations
> > of the world) how does one get a Linux kernel that provides a sizable
> > safety net that is _SUPPORTED_ for true enterprise-grade applications?
>
> eh I don't know if you paid attention, but Red Hat Enterprise Linux 4
> only has 4Kb stacks kernels... so that covers your supported true
> enterprise-grade application thing.
>
> > Simply put 4K vs 8K is not as trivial a decision as you'd have people believe.
>
> that's too simply put in fact, especially if you look at it
> historically. It's a bit of irony that part of the reason 4K stacks was
> developed was that the 2.4 kernels ran out of stack space for customers
> occasionally (just as example look at lkml this week, there was a report
> about such an overflow there as well). Remember that 4K+4K has more
> stack space than the 4K+2K as 2.4 kernels have. Sure 2.6 bumped this to
> 5.5k/2.5k roughly in the "8K" case, but fundamentally the change to
> 4k/4k isn't all that big even inside 2.6.
>
> You can go on and keep painting this as a cowboy development, but it
> really isn't....
>
>
>
> [1] Yes addicts; binary drivers are in many ways similar to heroin;
> they're really hard to get rid of for example and highly addictive, they
> also cause some people to act like junkies-in-withdrawl when their
> binary driver breaks, or when someone suggests breaking it.
>
>

Re: [2.6 patch] i386: always use 4k stacks

[linux-os (Dick Johnson)]

On Fri, 16 Dec 2005, David Lang wrote:

> On Fri, 16 Dec 2005, linux-os (Dick Johnson) wrote:
>
>> On Fri, 16 Dec 2005, Horst von Brand wrote:
>>
>>> linux-os \(Dick Johnson\) <linux-os@analogic.com> wrote:
>>>
>>> [...]
>>>
>>>> Throughout the past two years of 4k stack-wars, I never heard why
>>>> such a small stack was needed (not wanted, needed). It seems that
>>>> everybody "knows" that smaller is better and most everybody thinks
>>>> that one page in ix86 land is "optimum". However I don't think
>>>> anybody ever even tried to analyze what was better from a technical
>>>> perspective. Instead it's been analyzed as religious dogma, i.e.,
>>>> keep the stack small, it will prevent idiots from doing bad things.
>>>
>>> OK, so here goes again...
>>>
>>> The kernel stack has to be contiguous in /physical/ memory. Keep the
>> stack
>>> /one/ page, that way you can always get a new stack when needed (==
>> each
>>> fork(2) or clone(2)). If the stack is 2 (or more) pages, you'll have
>> to
>>> find (or create) a multi-page free area, and (fragmentation being what
>> it
>>> is, and Linux routinely running for months at a time) you are in a
>> whole
>>> new world of pain.
>>
>> The interrupt stack needs to be non-paged. Are you sure the user-stacks
>> need to be 'physical', non-paged too? If so, that's probably the
>> problem. All addresses accessed by the CPUs in the kernel are virtual
>> which means one needs some mapping anyway.
>
> actually, the kernel always uses real addresses, userspace uses virtual
> addresses.
>

No. Hint: What is PAGE_OFFSET?
Everything the CPU executes/reads/writes is translated from physical (bus)
addresses to virtual addresses.

What you may have heard or read is that the address-space that the
kernel uses for its code and resident data has fixed translation tables
which means one doesn't have to scan a bunch of tables to locate the
bus address, given the virtual address.

physical_address = virtual_address  - PAGE_OFFSET;
virtual_address  = physical_address + PAGE_OFFSET;

This is totally an artifact of how the page-tables are set up.

> This came up recently with the page tables, Linus said that he was
> absolutly opposed to adding the complication and overhead of changine the
> kernel to user virtual addresses instead of real addresses for it's data
> structures. it would add an extra level of redirection to just about every
> memory access (which also means an additional load on the cache to store
> the mapping info to resolve this redirection). The performance hit for
> this would be considerable.
>
> David Lang
>

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13.4 on an i686 machine (5589.56 BogoMips).
Warning : 98.36% of all statistics are fiction.
.

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: [2.6 patch] i386: always use 4k stacks

[Arjan van de Ven]

On Fri, 2005-12-16 at 16:28 -0500, Mike Snitzer wrote:
> Arjan,
> 
> I'm aware that RedHat has been shipping FC and RHEL kernels with 4K
> stacks for some time.  The recent 4K stack fixes pretty much establish
> that RedHat's early adoption of 4K stacks was a "cowbody development"
> no? 

(disclaimer: I don't work for Red Hat)

actually no. The only overflows seen in reality are with XFS, and Red
Hat doesn't support XFS anyway. The recent fix (not plural) was more a
precaution to let XFS work more reliable.


>  I don't think RedHat kept similar 4K stack fixes from upstream,
> so a bit of luck maybe?  I do agree that at this point the 4K-only
> proposal is NOT an overly rash decision given continued adequate
> vetting in -mm.  But even then there may be untested workloads that
> expose stack issues.  Which is perfectly fine if users have the option
> to use _supported_ alternate configs.

the problem to some degree is that those people if they hit this, won't
report it most likely. This is a problem because the same overflow can
hit with 8K stacks, just less frequent, so it really needs to be fixed
regardless of what anyone thinks of 4k-vs-8k stacks.


> I got the sense that you were trying to paint me as something of a
> closet "ndiswrapper addict".  

Actually I don't even know if you use ndiswrapper, but some others are
behaving like that; you're one of the "others" who actually try to use
real arguments.

> Amusing and all but my motivations for
> requesting continued options on the stack size are rooted in concerns
> that long call chains can and still do result when running kernel.org
> and RHEL4 kernels under particular workloads.  An example workload
> being cluster filesystems that in a single call chain historically
> _could_ leverage iptables + RPC (tcp) + DM (LVM) + etc.

funny you mention this one: iptables/RPC(tcp) actually run in the OTHER
4K stack; this workload has actually LESS chance of an overflow than
before... due to having more space in irq context.


> Given Neil Brown's fix for the block layer these stack-heavy workloads

which was pure preemptive and not based on actually observed problems
btw. it's a good fix nevertheless.

> All of us appreciate the desire to have Linux be more efficient and 4K
> stacks will get us that.  If it comes with the cost of instability
> under more exotic workloads then the bad outweighs the perceived good
> of imposed 4K stacks.  With RHEL4 it would seem we're past the point
> of no-return for supported 8K stacks.  I'm merely advocating upstream
> give users the 8K+IRQ stack _options_ and set the default to 4K.

note that I no longer care about this option going away or not; it's not
worth the silly flames (present company excluded ;)

options are good, to a degree. The extreme would be a kernel with 40.000
different config options, one for each patch that goes in. That's of
course silly! The other extreme is the gnome idea that preferences are
bad period. To a degree, Havoc and co are right in that there shouldn't
be a "unbreak my app" option, just like there shouldn't be a "unbreak my
kernel" config option. There needs to be some sort of reason and
proportion in all this. 

Where to draw the line is tricky I suppose and to a large degree a
matter of taste of the individual developer; but to be realistic; things
like 4-level pagetables or objrmap have a similar or higher risk of
breakage when they got in, and those didn't get config options
(something I agree with fwiw). In fact each 2.6.X release so far has had
2 or 3 major changes more risky/invasive than 4K stacks.

All that people say about 4k/4k stacks vs unified 8k stacks is
smoke-and-daggers to be honest (yes there were some problems in the past
with XFS, XFS got mostly fixed and Neil's patch fixes that for real, but
the basic things got fixed long ago). They forget that 2.4 got along
fine for a long time with a 4k/2k stack, or maybe chose to forget. All
those overflows you mention should hit double on 2.4 (and to be honest,
2.4 does hit some overflows, mostly due to the effective 2k irq side).
The 4k/4k approach is an extension on top of the 2.4 situation. 2.6 with
a unified 8k stack has a bit extra space, sure. but not dramatic much
more either. 

Re: [2.6 patch] i386: always use 4k stacks

[Adrian Bunk]

On Fri, Dec 16, 2005 at 04:28:15PM -0500, Mike Snitzer wrote:
>...
> Given Neil Brown's fix for the block layer these stack-heavy workloads
> that included DM in the call chain need to be revisited.  However, the
> savings associated with those particular fixes still may not leave
> sufficient breathing room.  The logic that all users must NOW provide
> workloads which undermine 4K stack viability otherwise the 8K option
> will be completely removed _seems_ quite irrational (even though we
> are _supposedly_ just talking about doing so in -mm).
> 
> All of us appreciate the desire to have Linux be more efficient and 4K
> stacks will get us that.  If it comes with the cost of instability
> under more exotic workloads then the bad outweighs the perceived good
> of imposed 4K stacks.  With RHEL4 it would seem we're past the point
> of no-return for supported 8K stacks.  I'm merely advocating upstream
> give users the 8K+IRQ stack _options_ and set the default to 4K.

My count of bug reports for problems with in-kernel code with 4k stacks 
after Neil's patch went into -mm is still at 0. That's amazing 
considering how many people have claimed in this thread how unstable
4k stacks were...

Enabling 4k stacks unconditionally for all -mm users will give us a 
wider testing coverage and will tell us whether we have really fixed all 
bugs that become visible with 4k stacks or whether there are still bugs 
left.

-mm kernels contain many experimental features, and "completely removed" 
isn't really true because we can expect that people running the 
experimental -mm kernels to know how to un-apply a patch.

> Mike

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [2.6 patch] i386: always use 4k stacks

[Dave Jones]

On Fri, Dec 16, 2005 at 10:50:54PM +0100, Adrian Bunk wrote:
 > On Fri, Dec 16, 2005 at 04:28:15PM -0500, Mike Snitzer wrote:
 > >...
 > > Given Neil Brown's fix for the block layer these stack-heavy workloads
 > > that included DM in the call chain need to be revisited.  However, the
 > > savings associated with those particular fixes still may not leave
 > > sufficient breathing room.  The logic that all users must NOW provide
 > > workloads which undermine 4K stack viability otherwise the 8K option
 > > will be completely removed _seems_ quite irrational (even though we
 > > are _supposedly_ just talking about doing so in -mm).
 > > 
 > > All of us appreciate the desire to have Linux be more efficient and 4K
 > > stacks will get us that.  If it comes with the cost of instability
 > > under more exotic workloads then the bad outweighs the perceived good
 > > of imposed 4K stacks.  With RHEL4 it would seem we're past the point
 > > of no-return for supported 8K stacks.  I'm merely advocating upstream
 > > give users the 8K+IRQ stack _options_ and set the default to 4K.
 > 
 > My count of bug reports for problems with in-kernel code with 4k stacks 
 > after Neil's patch went into -mm is still at 0. That's amazing 
 > considering how many people have claimed in this thread how unstable
 > 4k stacks were...
 > 
 > Enabling 4k stacks unconditionally for all -mm users will give us a 
 > wider testing coverage and will tell us whether we have really fixed all 
 > bugs that become visible with 4k stacks or whether there are still bugs 
 > left.

As another anecdotal point, the number of oomkill/page alloc failure
related bugs that get filed against Fedora these days you can count
on one hand.  Before we switched over to 4K stacks, we were
getting regular reports from users having quite sane workloads on
capable machines getting jobs killed left and right.

Now the only cases we're seeing is usually loonies trying to
put silly amounts of RAM in 32bit systems, and the occasional
bug which turns out to be a slab leak or something similar.
(There's one oddball right now where someone sees his 5GB
 x86-64 run out of DMA zone, but that might 'go away' when
 we push out a kernel with the DMA32 zone).

		Dave

Re: [2.6 patch] i386: always use 4k stacks

[Horst von Brand]

Steven Rostedt <rostedt@goodmis.org> wrote:
> On Fri, 2005-12-16 at 15:42 -0300, Horst von Brand wrote:
> > linux-os \(Dick Johnson\) <linux-os@analogic.com> wrote:
> > 
> > [...]
> > 
> > > Throughout the past two years of 4k stack-wars, I never heard why
> > > such a small stack was needed (not wanted, needed). It seems that
> > > everybody "knows" that smaller is better and most everybody thinks
> > > that one page in ix86 land is "optimum". However I don't think
> > > anybody ever even tried to analyze what was better from a technical
> > > perspective. Instead it's been analyzed as religious dogma, i.e.,
> > > keep the stack small, it will prevent idiots from doing bad things.
> > 
> > OK, so here goes again...
> > 
> > The kernel stack has to be contiguous in /physical/ memory. Keep the stack
> > /one/ page, that way you can always get a new stack when needed (== each
> > fork(2) or clone(2)). If the stack is 2 (or more) pages, you'll have to
> > find (or create) a multi-page free area, and (fragmentation being what it
> > is, and Linux routinely running for months at a time) you are in a whole
> > new world of pain.

> So people should really be asking for a PAGE_SIZE = 8k option ;)

On i386 is is either 4KiB or 4MiB. Guess what I prefer...
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: [2.6 patch] i386: always use 4k stacks

[Andi Kleen]

Kyle Moffett <mrmacman_g4@mac.com> writes:

> On Dec 16, 2005, at 10:35, Diego Calleja wrote:
> > I know, but there's too much resistance to the "pure" 4kb patch.
> 
> I have yet to see any resistance to the 4Kb patch this time around
> that was not "*whine* don't break my ndiswrapper plz".  

My comment from last time about the missing safety net still applies 100%.

Kernel code is getting more complex all the time and running with
very tight stack is just risky.

> The point is to force it in -mm so most people can't just disable it
> because it fixes their problem.  We want 8k stacks to go away, and

Who is we? And why? 

About the only half way credible arguments I've seen for it were:  

- "it might reduce stalls in the VM with order 1". Didn't quite
convince me because there were no numbers presented and at least on
x86-64 I've never noticed or got reported significant stalls because
of this.

- "it allows more threads for 32bit which might run out of lowmem" - i
think everybody agrees that the 10k threads case is not really
something to encourage. And even when you want to add it then only a factor
two increase (which this patch brings) is not really too helpful.

The main argument thrown around seems to be "but it will break
binary only modules" - while I'm not fully unsympathetic I don't
think technical issues in the kernel should be guided by 
such political considerations.

I suspect you will be reposting it so often till the voices
of reasons get tired? 

-Andi

Re: [2.6 patch] i386: always use 4k stacks

[Kyle Moffett]

On Dec 17, 2005, at 12:44, Andi Kleen wrote:
> Kernel code is getting more complex all the time and running with  
> very tight stack is just risky.

IMPORTANT POINT:  The 4k-stacks code does *NOT* reduce overall  
available stack!!!  With the old code we have 8k of _total_ stack.   
With the new code we have 4k of interrupt stack and 4k of per-process  
stack.  This makes stack-overflows a _LOT_ more debuggable, because  
it's not a coincidence of high process-stack-usage and high interrupt- 
stack-usage.

>> The point is to force it in -mm so most people can't just disable  
>> it  because it fixes their problem.  We want 8k stacks to go away
>
> Who is we? And why?
>
> About the only half way credible arguments I've seen for it were:

I posted a list of links to the archives of various reasons a day or  
so ago, but for summary:

This helps for some NUMA systems because single pages can come out of  
a per-cpu pool instead of requiring global allocator locks.

> - "it might reduce stalls in the VM with order 1". Didn't quite  
> convince me because there were no numbers presented and at least on  
> x86-64 I've never noticed or got reported significant stalls  
> because of this.

One comment on x86-64 vs. x86:  There are restrictions on where in  
memory your process stacks can be located on a 32-bit platform.  They  
need to reside in lowmem, which means under certain circumstances  
your lowmem can get too fragmented to create new processes even  
though you still have a lot of available RAM.

> - "it allows more threads for 32bit which might run out of lowmem"  
> - i think everybody agrees that the 10k threads case is not really  
> something to encourage.

Who is this "everybody" of whom you speak?  :-D.  Personally I agree  
that we shouldn't _encourage_ 10k threads, but there are existing  
userspace programs which do that, and I think we should support them  
as much as possible.

> And even when you want to add it then only a factor two increase  
> (which this patch brings) is not really too helpful.

The fragmentation behavior and optimizations for order-1 vs. order-0  
_is_ significant.  You can _always_ allocate order-0 pages if you  
have any free memory in that zone, which is _not_ necessarily true  
for order-N pages. (even if N==1).  Also, I think some of the  
fragmentation avoidance attempts get significantly easier and produce  
much better results if all the kernel stacks are order-0.

Cheers,
Kyle Moffett

--
If you don't believe that a case based on [nothing] could potentially  
drag on in court for _years_, then you have no business playing with  
the legal system at all.
   -- Rob Landley

Re: [2.6 patch] i386: always use 4k stacks

[Paul Rolland]

Hello,

> One comment on x86-64 vs. x86:  There are restrictions on where in  
> memory your process stacks can be located on a 32-bit 
> platform.  They  
> need to reside in lowmem, which means under certain circumstances  
> your lowmem can get too fragmented to create new processes even  
> though you still have a lot of available RAM.

But where does these restrictions come from ? As far as I know, stack
is referenced to by SS:ESP registers, and nothing in the x86 architecture
prevents them from pointing outside of lowmem... Isn't this simply a
Linux design restriction ?

Regards,
Paul


Paul Rolland, rol(at)as2917.net
ex-AS2917 Network administrator and Peering Coordinator

--

Please no HTML, I'm not a browser - Pas d'HTML, je ne suis pas un navigateur
"Some people dream of success... while others wake up and work hard at it" 

"I worry about my child and the Internet all the time, even though she's 
too young to have logged on yet. Here's what I worry about. I worry that 
10 or 15 years from now, she will come to me and say 'Daddy, where were 
you when they took freedom of the press away from the Internet?'"
--Mike Godwin, Electronic Frontier Foundation 

Re: [2.6 patch] i386: always use 4k stacks

[Arjan van de Ven]

On Sat, 2005-12-17 at 21:23 +0100, Paul Rolland wrote:
> Hello,
> 
> > One comment on x86-64 vs. x86:  There are restrictions on where in  
> > memory your process stacks can be located on a 32-bit 
> > platform.  They  
> > need to reside in lowmem, which means under certain circumstances  
> > your lowmem can get too fragmented to create new processes even  
> > though you still have a lot of available RAM.
> 
> But where does these restrictions come from ? As far as I know, stack
> is referenced to by SS:ESP registers, and nothing in the x86 architecture
> prevents them from pointing outside of lowmem... Isn't this simply a
> Linux design restriction ?

lowmem is a linux design restriction; only lowmem is directly
addressable.

(also remember that you can have 36 bits of physical ram, but only 32
bit in a pointer, so even if lowmem wasn't 870Mb itd be limited to 4Gb)

Re: [2.6 patch] i386: always use 4k stacks

[Adrian Bunk]

On Sat, Dec 17, 2005 at 06:44:07PM +0100, Andi Kleen wrote:
> Kyle Moffett <mrmacman_g4@mac.com> writes:
> 
> > On Dec 16, 2005, at 10:35, Diego Calleja wrote:
> > > I know, but there's too much resistance to the "pure" 4kb patch.
> > 
> > I have yet to see any resistance to the 4Kb patch this time around
> > that was not "*whine* don't break my ndiswrapper plz".  
> 
> My comment from last time about the missing safety net still applies 100%.
> 
> Kernel code is getting more complex all the time and running with
> very tight stack is just risky.

My patch reduces it from roughly 6kB to 4kB.

I'm with you that we need a safety net, but I don't see a problem with 
this being between 3kB and 4kB. The goal should be to _never_ use more 
than 3kB stack having a 1kB safety net.

And in my experience, many stack problems don't come from code getting 
more complex but from people allocating 1kB structs or arrays of
> 2k chars on the stack. In these cases, the code has to be fixed and 
"make checkstack" makes it easy to find such cases.

And as a data point, my count of bug reports for problems with in-kernel 
code with 4k stacks after Neil's patch went into -mm is still at 0.

> > The point is to force it in -mm so most people can't just disable it
> > because it fixes their problem.  We want 8k stacks to go away, and
> 
> Who is we? And why? 
> 
> About the only half way credible arguments I've seen for it were:  
> 
> - "it might reduce stalls in the VM with order 1". Didn't quite
> convince me because there were no numbers presented and at least on
> x86-64 I've never noticed or got reported significant stalls because
> of this.
> 
> - "it allows more threads for 32bit which might run out of lowmem" - i
> think everybody agrees that the 10k threads case is not really
> something to encourage. And even when you want to add it then only a factor
> two increase (which this patch brings) is not really too helpful.
>...

Unfortunately, "is not really something to encourage" doesn'a make 
"happens in real-life applications" impossible...

Reducing the stack by one third brings a factor two reduction in the 
memory usage of threads - I wouldn't say this sounds too bad.

> -Andi

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [2.6 patch] i386: always use 4k stacks

[Andi Kleen]

> > - "it allows more threads for 32bit which might run out of lowmem" - i
> > think everybody agrees that the 10k threads case is not really
> > something to encourage. And even when you want to add it then only a factor
> > two increase (which this patch brings) is not really too helpful.
> >...
> 
> Unfortunately, "is not really something to encourage" doesn'a make 
> "happens in real-life applications" impossible...

real-life applications can either use user space threads or 64bit
machines. The days when Linux did otherwise unjustificable ha^w^wdesign 
changes just to work around the 900MB lowmem on weird loads on 
extremly big 32bit machines are pretty much over I think... 

-Andi

Re: [2.6 patch] i386: always use 4k stacks

[Parag Warudkar]

On Dec 17, 2005, at 3:52 PM, Adrian Bunk wrote:

> And in my experience, many stack problems don't come from code getting
> more complex but from people allocating 1kB structs or arrays of

And we catch this type of problems fairly easily in the patch review  
itself, even before accepting the code in mainline. Plus there is  
make checkstack to help find and fix any such issues, isn't it? So  
it's not like forcing the stack to 4Kb and making the offending code  
to crash is the best solution to force people to write code which  
plays nice with the stack.

I think on i386 most people do fine with the 8Kb stack - whoever  
benefits from 4Kb stack, can always choose the 4Kb stack config  
option and recompile.

Alternatively, default to 4Kb and let people choose 8Kb and recompile  
if that's what suits their workloads.

In any case having options doesn't hurt anything and we don't benefit  
in any way from taking away the 8Kb option.

My 2 cents.

Parag

Re: [2.6 patch] i386: always use 4k stacks

[Andi Kleen]

On Sun, Dec 18, 2005 at 12:03:39AM -0500, Parag Warudkar wrote:
> 
> On Dec 17, 2005, at 3:52 PM, Adrian Bunk wrote:
> 
> >And in my experience, many stack problems don't come from code getting
> >more complex but from people allocating 1kB structs or arrays of
> 
> And we catch this type of problems fairly easily in the patch review  
> itself, even before accepting the code in mainline. Plus there is  

You can catch the obvious ones, but the really hard ones
that only occur under high load in obscure exceptional
circumstances with large configurations and suitable nesting you won't. 
These would be only found at real world users.

-Andi

Re: [2.6 patch] i386: always use 4k stacks

[Parag Warudkar]

On Dec 18, 2005, at 12:43 AM, Andi Kleen wrote:

> You can catch the obvious ones, but the really hard ones
> that only occur under high load in obscure exceptional
> circumstances with large configurations and suitable nesting you  
> won't.
> These would be only found at real world users.

Yep, as it all depends on code complexity, some of these cases might  
not be "errors" at all - instead for that kind of functionality they  
might _require_ bigger stacks.

If you have 64 bit machines common place and memory a lot cheaper I  
don't see how it is beneficial to force smaller stack sizes without  
giving consideration to the code complexity, architecture and  
requirements.
(Solaris for example, seems to be going to have 16Kb kernel stacks on  
64 bit machines.)

So, please let's leave stack size as an option for users to choose  
and stop this 4Kb stack war. May be after a little rest I will start  
another one demanding 16Kb stacks :)

Parag

Re: [2.6 patch] i386: always use 4k stacks

[Adrian Bunk]

On Sun, Dec 18, 2005 at 12:03:39AM -0500, Parag Warudkar wrote:
> 
> On Dec 17, 2005, at 3:52 PM, Adrian Bunk wrote:
> 
> >And in my experience, many stack problems don't come from code getting
> >more complex but from people allocating 1kB structs or arrays of
> 
> And we catch this type of problems fairly easily in the patch review  
> itself, even before accepting the code in mainline. Plus there is  
> make checkstack to help find and fix any such issues, isn't it? So  
> it's not like forcing the stack to 4Kb and making the offending code  
> to crash is the best solution to force people to write code which  
> plays nice with the stack.

4kB stacks are already an option for some time. There were some problems 
in the beginning, but as far as we know we have have fixed all of them.

There are so many possible bugs people writing kernel code could 
introduce bugs that cause crashes. The solution is not to add 
workarounds for programming bugs at every possible place, but as the
code review and "make checkstack" before accepting code.

As a data point, my count of bug reports for problems with in-kernel 
code with 4k stacks after Neil's patch went into -mm is still at 0.

> I think on i386 most people do fine with the 8Kb stack - whoever  
> benefits from 4Kb stack, can always choose the 4Kb stack config  
> option and recompile.
> 
> Alternatively, default to 4Kb and let people choose 8Kb and recompile  
> if that's what suits their workloads.
>...

There is no workload where 8kB suits better.

> My 2 cents.
> 
> Parag

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [2.6 patch] i386: always use 4k stacks

[Adrian Bunk]

On Sun, Dec 18, 2005 at 06:43:23AM +0100, Andi Kleen wrote:
> On Sun, Dec 18, 2005 at 12:03:39AM -0500, Parag Warudkar wrote:
> > 
> > On Dec 17, 2005, at 3:52 PM, Adrian Bunk wrote:
> > 
> > >And in my experience, many stack problems don't come from code getting
> > >more complex but from people allocating 1kB structs or arrays of
> > 
> > And we catch this type of problems fairly easily in the patch review  
> > itself, even before accepting the code in mainline. Plus there is  
> 
> You can catch the obvious ones, but the really hard ones
> that only occur under high load in obscure exceptional
> circumstances with large configurations and suitable nesting you won't. 
> These would be only found at real world users.

You miss the fact that many of these problems can be detected by static 
analysis.

We know that we don't have any non-recursive paths with > 3 kB stack 
usage anymore since the beginning of this year, and the known recursive 
problems should be attacked by Neil's patch.

> -Andi

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [2.6 patch] i386: always use 4k stacks

[Parag Warudkar]

On Dec 18, 2005, at 7:09 AM, Adrian Bunk wrote:

> There is no workload where 8kB suits better.

People have pointed out that there is currently at least one  
incompatibility introduced by 4K stacks and there may be many others  
which are corner cases, that only occur under high load in obscure  
exceptional circumstances with large configurations and suitable  
nesting.

Moreover for 64 bit architectures there is no proven point that 4Kb  
stacks are solving a specific problem there (Like the lowmem  
fragmentation on i386 for e.g.). Nor can we predict for sure that in  
future no type of functionality will require more stack. So taking  
away 8Kb stack size on such arches solves no known problems and  
introduces artificial limitations on code complexity.

All I am asking is what is wrong with having options? You can even  
default to 4Kb and let people choose 8Kb when they absolutely benefit  
from it. Does having options introduce code bloat or what is it that  
is pressing so hard to remove the 8Kb "option"?

Parag

Re: [2.6 patch] i386: always use 4k stacks

[Adrian Bunk]

On Sun, Dec 18, 2005 at 01:05:52AM -0500, Parag Warudkar wrote:
> 
> On Dec 18, 2005, at 12:43 AM, Andi Kleen wrote:
> 
> >You can catch the obvious ones, but the really hard ones
> >that only occur under high load in obscure exceptional
> >circumstances with large configurations and suitable nesting you  
> >won't.
> >These would be only found at real world users.
> 
> Yep, as it all depends on code complexity, some of these cases might  
> not be "errors" at all - instead for that kind of functionality they  
> might _require_ bigger stacks.

Is this just FUD or can you give an example where this is a real 
problem that can't be solved by using kmalloc()?

> If you have 64 bit machines common place and memory a lot cheaper I  
> don't see how it is beneficial to force smaller stack sizes without  
> giving consideration to the code complexity, architecture and  
> requirements.
>...

Note that we are talking about reducing the stack size _by one third_.

Therefore, your point it would make code much more complex sounds 
strange.

> Parag

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [2.6 patch] i386: always use 4k stacks

[Adrian Bunk]

On Sun, Dec 18, 2005 at 10:49:31AM -0500, Parag Warudkar wrote:
> 
> On Dec 18, 2005, at 7:09 AM, Adrian Bunk wrote:
> 
> >There is no workload where 8kB suits better.
> 
> People have pointed out that there is currently at least one  
> incompatibility introduced by 4K stacks and there may be many others  

That's wrong.

My count of bug reports for problems with 4k stacks with in-kernel code 
after Neil's patch went into -mm is still at 0.

> which are corner cases, that only occur under high load in obscure  
> exceptional circumstances with large configurations and suitable  
> nesting.

And this is not that much of an issue since most of these cases can and 
have already been analyzed by static analysis to be below 3 kB stack 
usage.

> Moreover for 64 bit architectures there is no proven point that 4Kb  
> stacks are solving a specific problem there (Like the lowmem  
> fragmentation on i386 for e.g.). Nor can we predict for sure that in  
> future no type of functionality will require more stack. So taking  
> away 8Kb stack size on such arches solves no known problems and  
> introduces artificial limitations on code complexity.
>...

That's complete bullshit considering that we are talking about an
i386-only patch.

> Parag

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [2.6 patch] i386: always use 4k stacks

[Parag Warudkar]

On Dec 18, 2005, at 10:51 AM, Adrian Bunk wrote:

> Is this just FUD or can you give an example where this is a real
> problem that can't be solved by using kmalloc()?

Can you prove "Under all uses, circumstances and code requirements we  
will do fine with 4K stacks today and tomorrow" ? How will deeply  
nested function calls, longer call chains etc. be solved by kmalloc()?

> Therefore, your point it would make code much more complex sounds
> strange.

My point wasn't that reducing stack will make code much more complex.  
My point was some type of functionality might validly require complex  
code which requires more stack - there are capable and affordable  
machines to solve such problems and all we are doing with 4kB stacks  
is that preventing it.

Again - what is the pressing need to remove the "8Kb Stack _Option_"?  
What problem does it solve on 64 bit arches?

Parag

Re: [2.6 patch] i386: always use 4k stacks

[Parag Warudkar]

On Dec 18, 2005, at 10:57 AM, Adrian Bunk wrote:

> That's complete bullshit considering that we are talking about an
> i386-only patch.

Ok, ignore my rants, I wrongly assumed there is a drive to make all  
arches live with 4Kb! But if it's taking away the 8Kb option on i386  
please consider keeping the 8Kb option and making 4Kb default.

Parag

Re: [2.6 patch] i386: always use 4k stacks

[Neil Brown]

On Friday December 16, bunk@stusta.de wrote:
> 
> The nfsd code uses inline in too many places.

Does it?
Most of the uses are either
 - truly tiny bits of code
 - code that is used only once which, as you as, will not currently 
   be auto-inlined on i386, so we do it by hand.

An exception is some of the xdr code.
If I 
  #define inline
in nfs3xdr.c, the nfsd.o changes from 
   text    data     bss     dec     hex filename
  76132    3464    2408   82004   14054 ../mm-i386/fs/nfsd/nfsd.o
to
   text    data     bss     dec     hex filename
  72452    3464    2408   78324   131f4 ../mm-i386/fs/nfsd/nfsd.o
which is probably a win.

Is that what you were referring to?

> 
> If this struct is really a problem (which I doubt considering it's 
> size), I'd prefer it being kmalloc'ed.

It's hard to *know* if it is a problem, but I am conscious that nfsd
adds measurably to stack depth for filesystem paths, and probably
isn't measured nearly as often.
It's true that 50 bytes out of 4K isn't a lot, but wastage that can be
avoided, should be avoided.

NeilBrown

Re: [2.6 patch] i386: always use 4k stacks

[Adrian Bunk]

On Mon, Dec 19, 2005 at 11:45:24AM +1100, Neil Brown wrote:
> On Friday December 16, bunk@stusta.de wrote:
> > 
> > The nfsd code uses inline in too many places.
> 
> Does it?
> Most of the uses are either
>  - truly tiny bits of code

That's OK if they stay tiny and don't grow as time passes by.

>  - code that is used only once which, as you as, will not currently 
>    be auto-inlined on i386, so we do it by hand.

That's OK if it isn't forgotten to un-inline them when they get more 
callers.

Unfortunately, people often don't check whether an "inline" is still 
appropriate when the code evolves.

Unless this is an extreme hot path, it's therefore IMHO not a good idea 
to use "inline" in such cases.

Additionally, it's a medium-term goal for me to re-enable unit-at-a-time 
on i386 for recent gcc's.

> An exception is some of the xdr code.
> If I 
>   #define inline
> in nfs3xdr.c, the nfsd.o changes from 
>    text    data     bss     dec     hex filename
>   76132    3464    2408   82004   14054 ../mm-i386/fs/nfsd/nfsd.o
> to
>    text    data     bss     dec     hex filename
>   72452    3464    2408   78324   131f4 ../mm-i386/fs/nfsd/nfsd.o
> which is probably a win.
> 
> Is that what you were referring to?

I didn't had one specific example in mind, but yes this seems to be an 
example for inline's that might have been reasonable at one time in the 
past, but are no longer today.

> > If this struct is really a problem (which I doubt considering it's 
> > size), I'd prefer it being kmalloc'ed.
> 
> It's hard to *know* if it is a problem, but I am conscious that nfsd
> adds measurably to stack depth for filesystem paths, and probably
> isn't measured nearly as often.
> It's true that 50 bytes out of 4K isn't a lot, but wastage that can be
> avoided, should be avoided.

"make checkstack" tells that nfsd_vfs_write is below 100 bytes of stack 
usage. So even calling 30 such functions would not get you above
3 kB stack usage.

It's also interesting that according to Jörn Engel's static analysis of 
call paths in kernel 2.6.11 [1], the string "nfs" does occur in neither 
any of the functions involved in call paths with > 2 kB stack usage, nor 
in any recursive call paths.

It's OK to use some bytes from the stack, and you haven't yet convinced 
me that the code you are responsible for is using too much stack.  ;-)

> NeilBrown

cu
Adrian

[1] http://wh.fh-wedel.de/~joern/stackcheck.2.6.11

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [2.6 patch] i386: always use 4k stacks

[Helge Hafting]

Ray Lee wrote:

>(Man, I've been holding my tongue on this conversation for a while,
>but it seems my better angels have deserted me.)
>
>On 12/15/05, Lee Revell <rlrevell@joe-job.com> wrote:
>  
>
>>Bugzilla link please.
>>    
>>
>
>No, that's not how failure engineering is done. A guy designing a
>bridge doesn't cut all the supports back to the bare minimum just to
>save money because his design says that the remaining metal should be
>strong enough. If you can't prove it, and it's a safety issue
>(continuing my analogy in the physical world), then you engineer for
>failure. Can you handle all occurrences? No, a hurricane Katrina comes
>along every once in a while. Can you weather more than you did before?
>Yes. In the meantime, their are fewer poor sods falling off the bridge
>that have to open a bugzilla report.
>  
>
This is quite different - you know much better what stack loads
the kernel may get into.  A bridge gets all sorts of weather,
with the very extreme cases occuring now and then.  With the
kernel, you can look at the code and determine the maximum
possible stack depth that can ever occur. It won't get deeper
even in some very rare case.


>The world of software is no different. If someone wants to remove the
>8k stacks option, they'd better prove that they're making my servers
>more reliable. I've seen zero arguments for why 8k stacks is unviable.
>  
>
Well, would you like the kernel to kill your webserver (or whatever
important app) because it attempted to fork at a time where
no two consecutive pages could be found?   That happens
occationally in real life - with 8k stack.  Going to 12k stack, 16k stacks,
or (shudder) 64 stacks would make that much worse than it
is today.

>(I've also wondered why we can't just have IRQ stacks plus 8k thread
>stacks -- seemingly the best of both worlds) Instead, what I've seen
>is that we have coders who don't like the idea of any non-order-zero
>allocations taking place, because big systems running poorly coded
>Java apps with massive threading can hit problems with allocations
>from time to time.
>  
>
You don't need big threaded apps for this to happen.  All
you need is a handful of forking apps and memory
fragmentation.  Many common server apps (web, mail, fileserver,...)
tend to use forking.  Massively threaded apps like 4k stacks simply
because that saves 4k for each of the many threads.

>The answer for that is the same answer the kernel community usually
>gives about poorly designed userspace applications: rewrite them.
>
>I'm quite open to being proved wrong. If someone has a counter case
>they can toss forth, please do so. Systems taking lots of interrupts?
>Then how about 8k + IRQ stacks? With a counterexample I'll gladly
>concede that I'm an ignorant slut[*] -- excuse me, Saturday Night Live
>flashbacks -- an ignorant git, and shut up. ([*] is only half right,
>I'm not all that ignorant).
>
>If someone doesn't show a counter case, then may I suggest people
>consider the possibility that this is not proper engineering. Prove
>it, or provide a safety blanket. But don't yank the blanket without
>proving the lack of problem.
>  
>
Well, failing order-1 allocations is _the_ counterexample.  IT
never happens with 4k stacks unless you run totally out of
memory, and then nothing can save you.

Helge Hafting

Re: [2.6 patch] i386: always use 4k stacks

[Helge Hafting]

linux-os (Dick Johnson) wrote:

>On Thu, 15 Dec 2005, Lee Revell wrote:
>
>  
>
>>On Thu, 2005-12-15 at 14:46 -0700, Jeff V. Merkey wrote:
>>    
>>
>>>Lee Revell wrote:
>>>
>>>      
>>>
>>>>On Thu, 2005-12-15 at 14:07 -0700, Jeff V. Merkey wrote:
>>>>
>>>>
>>>>        
>>>>
>>>>>When you are on the phone with an irrate customer at 2:00 am in the
>>>>>morning, and just turning off your broken 4K stack fix
>>>>>and getting the customer running matters.
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>Bugzilla link please.  Otherwise STFU.
>>>>
>>>>
>>>>        
>>>>
>>>??????
>>>
>>>Jeff
>>>      
>>>
>>You imply that your customer's problem was due to a kernel bug triggered
>>by CONFIG_4KSTACKS.  I am asking you to provide a link to the bug report
>>or get lost.
>>
>>Lee
>>    
>>
>
>Throughout the past two years of 4k stack-wars, I never heard why
>such a small stack was needed (not wanted, needed). It seems that
>everybody "knows" that smaller is better and most everybody thinks
>that one page in ix86 land is "optimum". However I don't think
>anybody ever even tried to analyze what was better from a technical
>perspective. Instead it's been analyzed as religious dogma, i.e.,
>keep the stack small, it will prevent idiots from doing bad things.
>
>I'm fairly sure that if you started from scratch and decided to
>write a new operating system, your choice of a stack-size would
>probably be something like 64k. I have no clue why somebody
>decided to use a 4k stack and force their choice upon others.
>And, yes, I am well aware that each system-call requires a
>seperate stack upon entry and it even needs to keep that stack
>while sleeping.
>  
>
No.  If writing an os from scratch, then using 4k stacks would
be absolutely trivial.  The problems now happens only because
we're switching away from the 8k stacks people were used to having.
Design with 4k stacks from the start, and you'll see people writing
code with the assumption that they can't stick more than a handful
of ints/pointers on the stack.

If you design with 64k stacks then people _use_ that memory, and
soon you hear someone wanting 128k stacks to be "safe".  That
way you end up with windows.  Note that 64k is 16 pages, and they
have to be _consecutive_.  It don't take much fragmentation before
you can't get that many consecutive pages - you can easily have 3/4 of your
memory unused, ready to be taken, but still be unable to get 16 
consecutive pages.

To see this - create a small kernel module that tries to allocate 64k
of consecutive memory when loaded.  Try loading it after a few days
of normal use, and see how often your server fails to do it.

Helge Hafting

Re: [2.6 patch] i386: always use 4k stacks

[Helge Hafting]

Horst von Brand wrote:

>>So people should really be asking for a PAGE_SIZE = 8k option ;)
>>    
>>
>
>On i386 is is either 4KiB or 4MiB. Guess what I prefer...
>  
>
Well, you can always use 8k pages - by setting PAGE_SIZE to 8k
and always set up the real 4k pages in pairs.  Wheter we want to do this
is another issue - but it is simple enough and avoids fragmentation.

Helge Hafting

Re: [2.6 patch] i386: always use 4k stacks

[Helge Hafting]

Parag Warudkar wrote:

>
> On Dec 18, 2005, at 12:43 AM, Andi Kleen wrote:
>
>> You can catch the obvious ones, but the really hard ones
>> that only occur under high load in obscure exceptional
>> circumstances with large configurations and suitable nesting you  won't.
>> These would be only found at real world users.
>
>
> Yep, as it all depends on code complexity, some of these cases might  
> not be "errors" at all - instead for that kind of functionality they  
> might _require_ bigger stacks.
>
No complex problem ever requires a big stack.  It may require a large amount
of memory - which can be allocated explicitly outside the stack.

> If you have 64 bit machines common place and memory a lot cheaper I  
> don't see how it is beneficial to force smaller stack sizes without  
> giving consideration to the code complexity, architecture and  
> requirements.
> (Solaris for example, seems to be going to have 16Kb kernel stacks on  
> 64 bit machines.)
>
> So, please let's leave stack size as an option for users to choose  
> and stop this 4Kb stack war. May be after a little rest I will start  
> another one demanding 16Kb stacks :)

I suggest a little experiment for you.  Make a kernel module which do 
nothing
except try to allocate 16k of _contigous_ kernel memory, and
printk whether it succeeded or failed before exiting.  Have cron run that
every 5 minutes.  After a few weeks of running this low-impact test on
a busy loaded server, look at statistics about how often the 16k allocation
worked - and how often it failed. 

Whatever failure rate you get, expect the same failure rate for server
processes forking to handle new connections while running with 16k stacks.
Failing one out of a hundred times would probably not be tolerated
for a webserver, and I suspect the failure rate for this will be higher - if
the machine has a reasonable memory load and the usual fragmentation.

On the other hand, if you can surprise us about how this works very
well - then you have a strong argument!

Helge Hafting

Re: [2.6 patch] i386: always use 4k stacks

[Jörn Engel]

On Mon, 19 December 2005 02:34:29 +0100, Adrian Bunk wrote:
> On Mon, Dec 19, 2005 at 11:45:24AM +1100, Neil Brown wrote:
> > 
> > It's hard to *know* if it is a problem, but I am conscious that nfsd
> > adds measurably to stack depth for filesystem paths, and probably
> > isn't measured nearly as often.
> > It's true that 50 bytes out of 4K isn't a lot, but wastage that can be
> > avoided, should be avoided.
> 
> "make checkstack" tells that nfsd_vfs_write is below 100 bytes of stack 
> usage. So even calling 30 such functions would not get you above
> 3 kB stack usage.
> 
> It's also interesting that according to Jörn Engel's static analysis of 
> call paths in kernel 2.6.11 [1], the string "nfs" does occur in neither 
> any of the functions involved in call paths with > 2 kB stack usage, nor 
> in any recursive call paths.
> 
> It's OK to use some bytes from the stack, and you haven't yet convinced 
> me that the code you are responsible for is using too much stack.  ;-)

Well, my metrics show the worst non-recursive paths and recursions
only.  The case at hand is a relatively innocent path on its own, but
is stacked on top of one of the recursions.

Therefore, if my tool could make more sense of recursions and f.e. see
that raid over raid is unlikely, but nfsd over xfs over raid over
block is likely, nfsd would definitely show up.  Recursions are the
hard problem to worry about.

Don't blame Neil for my tool being stupid. :)

Jörn

-- 
Don't patch bad code, rewrite it.
-- Kernigham and Pike, according to Rusty

Re: [2.6 patch] i386: always use 4k stacks

[Parag Warudkar]

On Dec 19, 2005, at 6:09 AM, Helge Hafting wrote:

> I suggest a little experiment for you.  Make a kernel module which  
> do nothing
> except try to allocate 16k of _contigous_ kernel memory, and
> printk whether it succeeded or failed before exiting.  Have cron  
> run that
> every 5 minutes.  After a few weeks of running this low-impact test on
> a busy loaded server, look at statistics about how often the 16k  
> allocation
> worked - and how often it failed.

I am aware of the limitations of Linux MM and the problems associated  
with anything more than zero order allocations over a period of time.

My argument was it's not that a ton of i386 users are affected by  
having choice of stack sizes (I read LKML quite frequently and for  
long I don't remember seeing allocation failure errors - either  
people moved to 64  bits without LOWMEM and that helped or people  
just do fine with the current 8K stack on i386) and even if some are,  
let's leave the stack size as an option - it's not like it cause a  
lot of code bloat or other problems (I read your argument about VM  
developers bogged down by having to deal with 8K stacks but quite  
frankly I don't understand how.)

Whoever benefits can use the 4K stacks, others who feel it risky to  
have 4K stacks for whatever reason, can be happy too. We can even  
make the 4K default, but having supported option of 8K is important  
and almost all operating systems are having >4K stacks on i386  
machines, so there is some reason for having it.

But I rest my argument, I no longer use i386 and I am being told this  
patch only affects i386! ;)

Parag

Re: [2.6 patch] i386: always use 4k stacks

[Dumitru Ciobarcianu]

În data de Lu, 19-12-2005 la 11:22 -0500, Parag Warudkar a scris:
> Whoever benefits can use the 4K stacks, others who feel it risky to  
> have 4K stacks for whatever reason, can be happy too. We can even  
> make the 4K default, but having supported option of 8K is important  
> and almost all operating systems are having >4K stacks on i386  
> machines, so there is some reason for having it.

Sloppy coding ? As long you don't have the source you can't be sure.
Point to an open source (and not tainting by just reading it) code which
uses >4k+IRQstack stacks.

Millions of flyes eat shit.
It must be a reason for having it...

-- 
Cioby

Re: [2.6 patch] i386: always use 4k stacks

[Parag Warudkar]

On Dec 19, 2005, at 12:43 PM, Dumitru Ciobarcianu wrote:

> Sloppy coding ? As long you don't have the source you can't be sure.
> Point to an open source (and not tainting by just reading it) code  
> which
> uses >4k+IRQstack stacks.
>

First you gotta understand that I am not arguing to take away the 4K  
stacks - I am arguing about keeping both options and defaulting to 4K.

How do you determine how much stack space a piece of code is going to  
need without knowing what functionality it needs to build? There  
might be deeply nested, long call chains etc. which certain types of  
functionality might warrant. How do you prove "4K otta be enough  
stack for everyone doing everything", on what basis? (Reminds me of  
old DOS days and the famous statement relating to 640K)

> Millions of flyes eat shit.
> It must be a reason for having it...
>

Yeah, compare that same thing to FORCING 4K stacks - it sounds as  
illogical as the above statement.

No one is answering what are we gaining from removing the 8K stack  
"_option_" - few bytes of code size, reason to not fix the VM, for  
fun, for screwing over? Why not let people choose 8K if they need it?

Parag

Re: [2.6 patch] i386: always use 4k stacks

[Dumitru Ciobarcianu]

În data de Lu, 19-12-2005 la 14:10 -0500, Parag Warudkar a scris:
> On Dec 19, 2005, at 12:43 PM, Dumitru Ciobarcianu wrote:
> 
> > Sloppy coding ? As long you don't have the source you can't be sure.
> > Point to an open source (and not tainting by just reading it) code  
> > which
> > uses >4k+IRQstack stacks.
> >
> 
> First you gotta understand that I am not arguing to take away the 4K  
> stacks - I am arguing about keeping both options and defaulting to 4K.

Yes, and I agree with you on that, but you din't answered my question
regarding _which_ os you mentioned needing more stack space and why.

-- 
Cioby

Re: [2.6 patch] i386: always use 4k stacks

[Parag Warudkar]

On Dec 19, 2005, at 2:27 PM, Dumitru Ciobarcianu wrote:

> but you din't answered my question
> regarding _which_ os you mentioned needing more stack space and why.

The two other commercially successful OSes - Windows and Solaris have  
12Kb and 8Kb default kernel stack sizes. And both seem to do well  
(hold on :) with the large stack sizes - meaning there is no  
commercially observed problem created by the 8K stack size. Solaris  
even lets you change the kernel stack size at runtime.

Even if we keep aside the impending argument about both OS'es being  
crap and we shouldn't be imitating them, we could still derive one  
conclusion from them - it is possible to have larger stack on i386  
without problems (albeit with some drawbacks) which could be used  
under certain circumstances.

Parag

Re: [2.6 patch] i386: always use 4k stacks

[Dumitru Ciobarcianu]

În data de Lu, 19-12-2005 la 15:17 -0500, Parag Warudkar a scris:
> On Dec 19, 2005, at 2:27 PM, Dumitru Ciobarcianu wrote:
> 
> > but you din't answered my question
> > regarding _which_ os you mentioned needing more stack space and why.
> 
> The two other commercially successful OSes - Windows and Solaris have  
> 12Kb and 8Kb default kernel stack sizes. And both seem to do well  
> (hold on :) with the large stack sizes - meaning there is no  
> commercially observed problem created by the 8K stack size. Solaris  
> even lets you change the kernel stack size at runtime.

My point was that you don't know why those two OS have such a large
stack. Just because you can't look at the source without being
contaminated.


-- 
Cioby - "I'll just stop feeding the troll now"

Re: [2.6 patch] i386: always use 4k stacks

[Tue, 20 Dec 2005 02:24:16 +0100]

El Mon, 19 Dec 2005 22:34:57 +0200,
Dumitru Ciobarcianu <Dumitru.Ciobarcianu@iNES.RO> escribió:

> My point was that you don't know why those two OS have such a large
> stack. Just because you can't look at the source without being
> contaminated.

opensolaris is open source, you can look at their code.

But I don't think you'll find an answer there. My bet is: because
it'd be more difficult for their customers (even if opensolaris
is opensource it was born as propietary OS), because doing it
doesn't buys you performance and customers, there're not
lot of reasons for doing it, etc.

As I understand it, linux is "different". I'd say that the main
"philosophic" (not technical) reason for going 4K is: "because
we have the balls to write a 4k-stack-safe kernel". Quoting Linus:

"I hold open source people to higher standards. They are supposed to be
 the people who do programming because it's an art-form, not because it's
 their job."

Re: [2.6 patch] i386: always use 4k stacks

[Adrian Bunk]

On Mon, Dec 19, 2005 at 02:10:17PM -0500, Parag Warudkar wrote:
>...
> How do you determine how much stack space a piece of code is going to  
> need without knowing what functionality it needs to build? There  
> might be deeply nested, long call chains etc. which certain types of  
> functionality might warrant.

Static analysis of this problem is possible.

"make checkstack" is a good starting point.

And the automatic analysis of all possible call chains can and has 
already found problems.

> How do you prove "4K otta be enough  
> stack for everyone doing everything", on what basis? (Reminds me of  
> old DOS days and the famous statement relating to 640K)
>...

We are talking about reducing the stack size by one third which doesn't 
result in a fundamental difference.

There is no technical reason why 4 kB shouldn't be enough - I don't 
count sloppy coding as a reason for it since in such cases we better 
correct the code.

> Parag

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [2.6 patch] i386: always use 4k stacks

[Horst von Brand]

Parag Warudkar <kernel-stuff@comcast.net> wrote:
> On Dec 19, 2005, at 2:27 PM, Dumitru Ciobarcianu wrote:
> > but you din't answered my question
> > regarding _which_ os you mentioned needing more stack space and why.

> The two other commercially successful OSes - Windows and Solaris have
> 12Kb and 8Kb default kernel stack sizes. And both seem to do well
> (hold on :) with the large stack sizes - meaning there is no
> commercially observed problem created by the 8K stack size. Solaris
> even lets you change the kernel stack size at runtime.

> Even if we keep aside the impending argument about both OS'es being
> crap 

Right.

>      and we shouldn't be imitating them,

Nodz. That doesn't mean they don't have their strong points, which we
should consider carefully.

>                                          we could still derive one
> conclusion from them - it is possible to have larger stack on i386
> without problems (albeit with some drawbacks) which could be used
> under certain circumstances.

"With some drawbacks" is the point: It has been determined that the
drawbacks are heavy enough that the 8KiB stack option should go. Given
there is /no/ compelling argument /against/ 4KiB stacks, even very minor
drawbacks are important. So first make 4KiB the standard (popular
distributions work that way for /years/ now, with no measurable downsides),
then axe 8KiB stack as an option. Also note that 8KiB stacks really only
gives 4KiB to the process (plus (or minus!) a random ammount depending on
the interrupts being serviced ATM), and this has been so forever.

Oh, well, one of the larger drawbacks of 4KiB stacks is the inevitable
flamewar, each time with /less/ data (this round I've seen none) supporting
the need for larger stacks, into which all kinds of idiots* are suckered.

* This certainly includes myself
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: [2.6 patch] i386: always use 4k stacks

[Felix Oxley]

On 19 Dec 2005, at 19:10, Parag Warudkar wrote:

>
> No one is answering what are we gaining from removing the 8K stack  
> "_option_" - few bytes of code size, reason to not fix the VM, for  
> fun, for screwing over? Why not let people choose 8K if they need it?

The proposed patch is for mm only. What you are gaining is wider  
testing of 4K stacks.

I am just a lurker but, having read the entire thread, it seems to me  
that:
1) the patch should be applied to mm.
2) ndiswrapper should be modified to work with 4K stacks.

It seems unlikely to me that this patch will be pushed to Linus just  
because it has been in mm.
If that possibility comes up in 6-12 months then the flamewar can  
begin again.

regards,
Felix
  

Re: [2.6 patch] i386: always use 4k stacks

[David Lang]

On Tue, 20 Dec 2005, Horst von Brand wrote:

>
> "With some drawbacks" is the point: It has been determined that the
> drawbacks are heavy enough that the 8KiB stack option should go. Given
> there is /no/ compelling argument /against/ 4KiB stacks, even very minor
> drawbacks are important. So first make 4KiB the standard (popular
> distributions work that way for /years/ now, with no measurable downsides),

at least one of the 'popular distributions' that switched to 4k stacks 
years ago worked around the problems that it generated by simply lableing 
the portions that didn't work with 4k stacks as 'unsupported by this 
distro' (XFS has been explicitly stated to be in this catagory in these 
discussions)

how many other corner cases are there that these distros just choose not 
to support, but need to be supported and tested for the vanilla kernel?

also for those who are arguing that it's only dropping from 6k to 4k, you 
are forgetting that the patches to move the interrupts to a seperate stack 
have already gone into the kernel, so today it is really 8k+4k and the 
talk is to move it to 4k+4k.

I think it's a good idea to change the default (especially in -mm) to 4k 
stacks and to schedule a change of the default in mainline for a few 
versions out, but there needs to be a safety net other then telling people 
to downgrade to a prior kernel if they run into problems when the switch 
is made

David Lang

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

Re: [2.6 patch] i386: always use 4k stacks

[Arjan van de Ven]

> 
> how many other corner cases are there that these distros just choose not 
> to support, but need to be supported and tested for the vanilla kernel?

as someone who was at that distro in the time.. none other than XFS and
reiserfs4.

> also for those who are arguing that it's only dropping from 6k to 4k, you 
> are forgetting that the patches to move the interrupts to a seperate stack 
> have already gone into the kernel, so today it is really 8k+4k and the 
> talk is to move it to 4k+4k.

actually irq stacks aren't enabled with 8K stacks right now, so your
statement isn't correct.

Re: [2.6 patch] i386: always use 4k stacks

[Adrian Bunk]

On Tue, Dec 20, 2005 at 09:56:10AM -0800, David Lang wrote:
> On Tue, 20 Dec 2005, Horst von Brand wrote:
> 
> >
> >"With some drawbacks" is the point: It has been determined that the
> >drawbacks are heavy enough that the 8KiB stack option should go. Given
> >there is /no/ compelling argument /against/ 4KiB stacks, even very minor
> >drawbacks are important. So first make 4KiB the standard (popular
> >distributions work that way for /years/ now, with no measurable downsides),
> 
> at least one of the 'popular distributions' that switched to 4k stacks 
> years ago worked around the problems that it generated by simply lableing 
> the portions that didn't work with 4k stacks as 'unsupported by this 
> distro' (XFS has been explicitly stated to be in this catagory in these 
> discussions)

AFAIK, XFS is the only example.
And the XFS related problems have already been fixed.

> how many other corner cases are there that these distros just choose not 
> to support, but need to be supported and tested for the vanilla kernel?

My count of bug reports for problems with in-kernel code with 4k stacks 
after Neil's patch went into -mm is still at 0.

If 4k stacks were as unstable as you imply, why has noone been able to 
point to _one single_ problem with 4k stacks that is still present 
after Neil's patch went into -mm?

> also for those who are arguing that it's only dropping from 6k to 4k, you 
> are forgetting that the patches to move the interrupts to a seperate stack 
> have already gone into the kernel, so today it is really 8k+4k and the 
> talk is to move it to 4k+4k.
>...

That's complete bullshit.

Currently, seperate irq stacks are only used with 
CONFIG_4KSTACKS=y.

> David Lang

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [2.6 patch] i386: always use 4k stacks

[Sean]

On Tue, December 20, 2005 12:56 pm, David Lang said:
>
> at least one of the 'popular distributions' that switched to 4k stacks
> years ago worked around the problems that it generated by simply lableing
> the portions that didn't work with 4k stacks as 'unsupported by this
> distro' (XFS has been explicitly stated to be in this catagory in these
> discussions)

Times change, the XFS issues have been resolved.

> how many other corner cases are there that these distros just choose not
> to support, but need to be supported and tested for the vanilla kernel?

This is called FUD.  If you have examples of a problem go ahead and post
them.   *EVERY* change made to the kernel has the potential to cause a
problem.   But this one has been carefully scrutinized and it seems that
only people with general notions of FUD remain to object.   Nobody seems
to have any real objections any more.  YAY!

> also for those who are arguing that it's only dropping from 6k to 4k, you
> are forgetting that the patches to move the interrupts to a seperate stack
> have already gone into the kernel, so today it is really 8k+4k and the
> talk is to move it to 4k+4k.

So what?   The point is that if you compare the world from what it was a
few years back with just 8K, now there is just as much stack space,
although it happens to be split in half.

> I think it's a good idea to change the default (especially in -mm) to 4k
> stacks and to schedule a change of the default in mainline for a few
> versions out, but there needs to be a safety net other then telling people
> to downgrade to a prior kernel if they run into problems when the switch
> is made

Of course.   If there are problems discovered in -mm with 4K stack (highly
unlikely since its been in production use on several vendor kernals for a
few years) they will be dealt with.   But once a reasonable period of time
has passed with no issues, its time to move it into mainline.

Sean

Re: [2.6 patch] i386: always use 4k stacks

[Dave Jones]

On Tue, Dec 20, 2005 at 09:56:10AM -0800, David Lang wrote:

 > at least one of the 'popular distributions' that switched to 4k stacks 
 > years ago worked around the problems that it generated by simply lableing 
 > the portions that didn't work with 4k stacks as 'unsupported by this 
 > distro' (XFS has been explicitly stated to be in this catagory in these 
 > discussions)

Actually there are several reasons why certain parts are
unsupported in RHEL kernels.  The fact that XFS blew up with
4k stacks was purely coincidental to it being unsupported.

		Dave

Re: [2.6 patch] i386: always use 4k stacks

[David Lang]

On Tue, 20 Dec 2005, Arjan van de Ven wrote:

>> how many other corner cases are there that these distros just choose not
>> to support, but need to be supported and tested for the vanilla kernel?
>
> as someone who was at that distro in the time.. none other than XFS and
> reiserfs4.

good to hear, outsiders don't know these details. all we know is that some 
things aren't supported, but (without a lot of effort) don't know what 
things.

>> also for those who are arguing that it's only dropping from 6k to 4k, you
>> are forgetting that the patches to move the interrupts to a seperate stack
>> have already gone into the kernel, so today it is really 8k+4k and the
>> talk is to move it to 4k+4k.
>
> actually irq stacks aren't enabled with 8K stacks right now, so your
> statement isn't correct.

Ok, I stand corrected, I didn't look at the code, I was going on the 
memories of the discussions on l-k where the advocates were pushing to 
enable the interrupt stacks unconditionally, and I was remembering that a 
patch to do so had gone in.

David Lang

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

Re: [2.6 patch] i386: always use 4k stacks

[Sander]

Arjan van de Ven wrote (ao):
> > how many other corner cases are there that these distros just choose
> > not to support, but need to be supported and tested for the vanilla
> > kernel?
> 
> as someone who was at that distro in the time.. none other than XFS
> and reiserfs4.

FWIW, I have a few servers and my workstation running Reiser4 and
CONFIG_4KSTACKS=y for several months now, and haven't encountered
problems yet. One server also runs Reiser4 on top op lvm2, and another
Reiser4 on top of sw raid1.

I know -mm + Reiser4 + 4kstacks is bleeding edge in more than one way,
but I like that for my workstations and the servers are
test/non-critical.

All systems do have real-life load though. I'd be happy to provide data
from these systems. Just mail me the commands.

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: [2.6 patch] i386: always use 4k stacks

[Jeffrey Hundstad]

Sander wrote:

>Arjan van de Ven wrote (ao):
>  
>
>>>how many other corner cases are there that these distros just choose
>>>not to support, but need to be supported and tested for the vanilla
>>>kernel?
>>>      
>>>
>>as someone who was at that distro in the time.. none other than XFS
>>and reiserfs4.
>>    
>>
>
>FWIW, I have a few servers and my workstation running Reiser4 and
>CONFIG_4KSTACKS=y for several months now, and haven't encountered
>problems yet. One server also runs Reiser4 on top op lvm2, and another
>Reiser4 on top of sw raid1.
>
>I know -mm + Reiser4 + 4kstacks is bleeding edge in more than one way,
>but I like that for my workstations and the servers are
>test/non-critical.
>
>All systems do have real-life load though. I'd be happy to provide data
>from these systems. Just mail me the commands.
>
>  
>

I would like to add to this.  I've been using XFS+LVM+SCSI in a 14,000+ 
user University email server with 4k stacks since it became available.  
I didn't even have problems BEFORE the XFS stuff was stack-dieted.  I 
would also be happy to provide more data.

-- 
Jeffrey Hundstad