Re: x86: 4kstacks default

Re: x86: 4kstacks default

[Bodo Eggert]

Andrew Morton <akpm@linux-foundation.org> wrote:
> Linux Kernel Mailing List <linux-kernel@vger.kernel.org> wrote:

>> Gitweb:    
>>
http://git.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=d61ecf0b53131564949bc4196e70f676000a845a
>> Commit:     d61ecf0b53131564949bc4196e70f676000a845a
>> Author:     Ingo Molnar <mingo@elte.hu>
>> Committer:  Ingo Molnar <mingo@elte.hu>

>>     x86: 4kstacks default

> This patch will cause kernels to crash.
> 
> It has no changelog which explains or justifies the alteration.
> 
> afaict the patch was not posted to the mailing list and was not
> discussed or reviewed.

The patch (or a similar one) was discussed and rejected for the reason above.

Re: x86: 4kstacks default

[Bodo Eggert]

Daniel Hazelton <dhazelton@enter.net> wrote:
> On Sunday 20 April 2008 08:27:14 Andi Kleen wrote:
>> Adrian Bunk <bunk@kernel.org> writes:
>> > 6k is known to work, and there aren't many problems known with 4k.
>> >
>> > And from a QA point of view the only way of getting 4k thoroughly tested
>>
>> But you have to first ask why do you want 4k tested? Does it serve
>> any useful purpose in itself? I don't think so. Or you're saying
>> it's important to support 50k kernel threads on 32bit kernels?

> Andi, you're the only one I've seen seriously pounding the "50k threads"
> thing - I don't think anyone is really fooled by the straw-man, so I'd
> suggest you drop it.
> 
> The real issue is that you think (and are correct in thinking) that people are
> idiots. Yes, there will be breakages if the default is changed to 4k stacks -
> but if people are running new kernels on boxes that'll hit stack use problems
> (that *AREN'T* related to ndiswrapper) and haven't made sure that they've
> configured the kernel properly, then they deserve the outcome. It isn't the
> job of the Linux Kernel to protect the incompetent - nor is it the job of
> linux kernel developers to do such.

It's the job of the kernel developers to mark experimental and broken options,
and to put a warning:

"This will break stacking of drivers, especially if disk manager, xfs, RAID
and nfs are used. Yes, linux is broken by default, but only if you intend to
set up a reliable system, so this will be OK!"

into the help text, instead of expecting each admin to read lkml.

Re: x86: 4kstacks default

[Daniel Hazelton]

On Sunday 20 April 2008 16:23:45 Bodo Eggert wrote:
> Daniel Hazelton <dhazelton@enter.net> wrote:
> > On Sunday 20 April 2008 08:27:14 Andi Kleen wrote:
> >> Adrian Bunk <bunk@kernel.org> writes:
> >> > 6k is known to work, and there aren't many problems known with 4k.
> >> >
> >> > And from a QA point of view the only way of getting 4k thoroughly
> >> > tested
> >>
> >> But you have to first ask why do you want 4k tested? Does it serve
> >> any useful purpose in itself? I don't think so. Or you're saying
> >> it's important to support 50k kernel threads on 32bit kernels?
> >
> > Andi, you're the only one I've seen seriously pounding the "50k threads"
> > thing - I don't think anyone is really fooled by the straw-man, so I'd
> > suggest you drop it.
> >
> > The real issue is that you think (and are correct in thinking) that
> > people are idiots. Yes, there will be breakages if the default is changed
> > to 4k stacks - but if people are running new kernels on boxes that'll hit
> > stack use problems (that *AREN'T* related to ndiswrapper) and haven't
> > made sure that they've configured the kernel properly, then they deserve
> > the outcome. It isn't the job of the Linux Kernel to protect the
> > incompetent - nor is it the job of linux kernel developers to do such.
>
> It's the job of the kernel developers to mark experimental and broken
> options, and to put a warning:
>
> "This will break stacking of drivers, especially if disk manager, xfs, RAID
> and nfs are used. Yes, linux is broken by default, but only if you intend
> to set up a reliable system, so this will be OK!"
>
> into the help text, instead of expecting each admin to read lkml.

Note that I've yet to meet a competent admin that creates brand new 
configurations each time they build a new kernel for a machine. Usually they 
have a "default configuration" for each machine that gets updated each time a 
new kernel is built. Usually they don't change working options. And since 
changing things to 4K stacks default would cause a new option - the "8K 
stacks" option to show up in a "make oldconfig" run - the admin would see it 
and, hopefully, check the help text and see that it his system, with a deeply 
stacked driver system (nfs+xfs+raid, for example) and set the 8K stacks 
option to "Y".

As I said, it isn't the job of the kernel or kernel developers to protect the 
incompetent (or the lazy).

DRH

-- 
Dialup is like pissing through a pipette. Slow and excruciatingly painful.

Re: x86: 4kstacks default

[Bodo Eggert]

On Sun, 20 Apr 2008, Daniel Hazelton wrote:
> On Sunday 20 April 2008 16:23:45 Bodo Eggert wrote:
> > Daniel Hazelton <dhazelton@enter.net> wrote:
> > > On Sunday 20 April 2008 08:27:14 Andi Kleen wrote:
> > >> Adrian Bunk <bunk@kernel.org> writes:

> > >> > 6k is known to work, and there aren't many problems known with 4k.
> > >> >
> > >> > And from a QA point of view the only way of getting 4k thoroughly
> > >> > tested
> > >>
> > >> But you have to first ask why do you want 4k tested? Does it serve
> > >> any useful purpose in itself? I don't think so. Or you're saying
> > >> it's important to support 50k kernel threads on 32bit kernels?
> > >
> > > Andi, you're the only one I've seen seriously pounding the "50k threads"
> > > thing - I don't think anyone is really fooled by the straw-man, so I'd
> > > suggest you drop it.
> > >
> > > The real issue is that you think (and are correct in thinking) that
> > > people are idiots. Yes, there will be breakages if the default is changed
> > > to 4k stacks - but if people are running new kernels on boxes that'll hit
> > > stack use problems (that *AREN'T* related to ndiswrapper) and haven't
> > > made sure that they've configured the kernel properly, then they deserve
> > > the outcome. It isn't the job of the Linux Kernel to protect the
> > > incompetent - nor is it the job of linux kernel developers to do such.
> >
> > It's the job of the kernel developers to mark experimental and broken
> > options, and to put a warning:
> >
> > "This will break stacking of drivers, especially if disk manager, xfs, RAID
> > and nfs are used. Yes, linux is broken by default, but only if you intend
> > to set up a reliable system, so this will be OK!"
> >
> > into the help text, instead of expecting each admin to read lkml.
> 
> Note that I've yet to meet a competent admin that creates brand new 
> configurations each time they build a new kernel for a machine.

Once is enough, and if you build a costom kernel, you'll certainly not 
want to start from the distribution's allmodconfig.

> Usually they 
> have a "default configuration" for each machine that gets updated each time a 
> new kernel is built. Usually they don't change working options. And since 
> changing things to 4K stacks default would cause a new option - the "8K 
> stacks" option to show up in a "make oldconfig" run - the admin would see it 
> and, hopefully, check the help text and see that it his system, with a deeply 
> stacked driver system (nfs+xfs+raid, for example) and set the 8K stacks 
> option to "Y".

The help text does not yet say anything about crashing.

> As I said, it isn't the job of the kernel or kernel developers to protect the 
> incompetent (or the lazy).

It's only incompetent if it's reasonable to expect a crashing kernel to
result from chosing the default values.
-- 
bus error. passengers dumped.

Re: x86: 4kstacks default

[Daniel Hazelton]

On Monday 21 April 2008 16:05:38 you wrote:
> On Sun, 20 Apr 2008, Daniel Hazelton wrote:
> > On Sunday 20 April 2008 16:23:45 Bodo Eggert wrote:
> > > Daniel Hazelton <dhazelton@enter.net> wrote:
> > > > On Sunday 20 April 2008 08:27:14 Andi Kleen wrote:
> > > >> Adrian Bunk <bunk@kernel.org> writes:
> > > >> > 6k is known to work, and there aren't many problems known with 4k.
> > > >> >
> > > >> > And from a QA point of view the only way of getting 4k thoroughly
> > > >> > tested
> > > >>
> > > >> But you have to first ask why do you want 4k tested? Does it serve
> > > >> any useful purpose in itself? I don't think so. Or you're saying
> > > >> it's important to support 50k kernel threads on 32bit kernels?
> > > >
> > > > Andi, you're the only one I've seen seriously pounding the "50k
> > > > threads" thing - I don't think anyone is really fooled by the
> > > > straw-man, so I'd suggest you drop it.
> > > >
> > > > The real issue is that you think (and are correct in thinking) that
> > > > people are idiots. Yes, there will be breakages if the default is
> > > > changed to 4k stacks - but if people are running new kernels on boxes
> > > > that'll hit stack use problems (that *AREN'T* related to ndiswrapper)
> > > > and haven't made sure that they've configured the kernel properly,
> > > > then they deserve the outcome. It isn't the job of the Linux Kernel
> > > > to protect the incompetent - nor is it the job of linux kernel
> > > > developers to do such.
> > >
> > > It's the job of the kernel developers to mark experimental and broken
> > > options, and to put a warning:
> > >
> > > "This will break stacking of drivers, especially if disk manager, xfs,
> > > RAID and nfs are used. Yes, linux is broken by default, but only if you
> > > intend to set up a reliable system, so this will be OK!"
> > >
> > > into the help text, instead of expecting each admin to read lkml.
> >
> > Note that I've yet to meet a competent admin that creates brand new
> > configurations each time they build a new kernel for a machine.
>
> Once is enough, and if you build a costom kernel, you'll certainly not
> want to start from the distribution's allmodconfig.

No, you wouldn't. But, at least at the companies I've worked for, there was 
already a custom kernel running and it had a default configuration file that 
was updated and carried over to each new kernel, with a "make oldconfig" done 
to update it.

> > Usually they
> > have a "default configuration" for each machine that gets updated each
> > time a new kernel is built. Usually they don't change working options.
> > And since changing things to 4K stacks default would cause a new option -
> > the "8K stacks" option to show up in a "make oldconfig" run - the admin
> > would see it and, hopefully, check the help text and see that it his
> > system, with a deeply stacked driver system (nfs+xfs+raid, for example)
> > and set the 8K stacks option to "Y".
>
> The help text does not yet say anything about crashing.

It should be updated to note that there are configurations that will overrun 
the stack and cause crashes. (However, it shouldn't be the default - I'll 
agree to that)

> > As I said, it isn't the job of the kernel or kernel developers to protect
> > the incompetent (or the lazy).
>
> It's only incompetent if it's reasonable to expect a crashing kernel to
> result from chosing the default values.

Agreed. I've been arguing about it without being clear that my arguments for 
it (including making it the default) are from the perspective of a desktop 
user who had run the "stack depth check" on an 8K stacks kernel for a long 
time and found that the only times he ever had problems was during boot - 
with "sed" and "grep" being the culprits.

Booting a 4K stacks kernel wouldn't work if I hadn't also modified the 
initscripts here, and that wasn't easy - I've got a report of grep using 
enough stack that only about 3900 bytes were left on the 8K stack.  However, 
I think this does show that there are problems left in a number of places 
that make moving to a default of 4K stacks dangerous.

(I've recently checked this by undoing my changes and setting up an 8K kernel 
on this laptop - I don't know if the next version of the distro will ship 
with 4K stacks, but I'm pretty certain it won't)

In light of this I'm going to pull out of this discussion, because I can no 
longer support a move to 4K stacks default. This doesn't mean I don't still 
support having them around, or even a move to them as a default at a later 
date, but right now there are still many places where there are problems that 
would cause "mysterious" failures and corruption with 4K stacks.

DRH

-- 
Dialup is like pissing through a pipette. Slow and excruciatingly painful.

Re: x86: 4kstacks default

[Andrew Morton]

On Fri, 18 Apr 2008 17:37:36 GMT
Linux Kernel Mailing List <linux-kernel@vger.kernel.org> wrote:

> Gitweb:     http://git.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=d61ecf0b53131564949bc4196e70f676000a845a
> Commit:     d61ecf0b53131564949bc4196e70f676000a845a
> Parent:     f408b43ceedce49f26c01cd4a68dbbdbe2743e51
> Author:     Ingo Molnar <mingo@elte.hu>
> AuthorDate: Fri Apr 4 17:11:09 2008 +0200
> Committer:  Ingo Molnar <mingo@elte.hu>
> CommitDate: Thu Apr 17 17:41:34 2008 +0200
> 
>     x86: 4kstacks default
>     
>     Signed-off-by: Ingo Molnar <mingo@elte.hu>
> ---
>  arch/x86/Kconfig.debug |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/arch/x86/Kconfig.debug b/arch/x86/Kconfig.debug
> index f4413c0..610aaec 100644
> --- a/arch/x86/Kconfig.debug
> +++ b/arch/x86/Kconfig.debug
> @@ -106,8 +106,8 @@ config DEBUG_NX_TEST
>  
>  config 4KSTACKS
>  	bool "Use 4Kb for kernel stacks instead of 8Kb"
> -	depends on DEBUG_KERNEL
>  	depends on X86_32
> +	default y

This patch will cause kernels to crash.

It has no changelog which explains or justifies the alteration.

afaict the patch was not posted to the mailing list and was not
discussed or reviewed.

Re: x86: 4kstacks default

[Ingo Molnar]

* Andrew Morton <akpm@linux-foundation.org> wrote:

> >  config 4KSTACKS
> >  	bool "Use 4Kb for kernel stacks instead of 8Kb"
> > -	depends on DEBUG_KERNEL
> >  	depends on X86_32
> > +	default y
> 
> This patch will cause kernels to crash.

what mainline kernels crash and how will they crash? Fedora and other 
distros have had 4K stacks enabled for years:

  $ grep 4K /boot/config-2.6.24-9.fc9
  CONFIG_4KSTACKS=y

and we've conducted tens of thousands of bootup tests with all sorts of 
drivers and kernel options enabled and have yet to see a single crash 
due to 4K stacks. So basically the kernel default just follows the 
common distro default now. (distros and users can still disable it)

	Ingo

Re: x86: 4kstacks default

[Oliver Pinter]

Hi Ingo!

with the older kernel is typical: xfs+nfs+4k stack(+lvm)



On 4/19/08, Ingo Molnar <mingo@elte.hu> wrote:
>
> * Andrew Morton <akpm@linux-foundation.org> wrote:
>
> > >  config 4KSTACKS
> > >  	bool "Use 4Kb for kernel stacks instead of 8Kb"
> > > -	depends on DEBUG_KERNEL
> > >  	depends on X86_32
> > > +	default y
> >
> > This patch will cause kernels to crash.
>
> what mainline kernels crash and how will they crash? Fedora and other
> distros have had 4K stacks enabled for years:
>
>   $ grep 4K /boot/config-2.6.24-9.fc9
>   CONFIG_4KSTACKS=y
>
> and we've conducted tens of thousands of bootup tests with all sorts of
> drivers and kernel options enabled and have yet to see a single crash
> due to 4K stacks. So basically the kernel default just follows the
> common distro default now. (distros and users can still disable it)
>
> 	Ingo
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>


-- 
Thanks,
Oliver

Re: x86: 4kstacks default

[Adrian Bunk]

On Sat, Apr 19, 2008 at 04:35:31PM +0200, Oliver Pinter wrote:
>...
> with the older kernel is typical: xfs+nfs+4k stack(+lvm)

Does anyone still experience problems with 2.6.25?

We all know that there once were problems, but if there are any left 
they should be reported and fixed.

> Thanks,
> Oliver

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: x86: 4kstacks default

[Oliver Pinter]

I dont know, thet this problem presentiert in 2.6.25, but im older
kernels yes (2.6.22> or 2.6.23>).

On 4/19/08, Adrian Bunk <bunk@kernel.org> wrote:
> On Sat, Apr 19, 2008 at 04:35:31PM +0200, Oliver Pinter wrote:
> >...
> > with the older kernel is typical: xfs+nfs+4k stack(+lvm)
>
> Does anyone still experience problems with 2.6.25?
>
> We all know that there once were problems, but if there are any left
> they should be reported and fixed.
>
> > Thanks,
> > Oliver
>
> cu
> Adrian
>
> --
>
>        "Is there not promise of rain?" Ling Tan asked suddenly out
>         of the darkness. There had been need of rain for many days.
>        "Only a promise," Lao Er said.
>                                        Pearl S. Buck - Dragon Seed
>
>


-- 
Thanks,
Oliver

Re: x86: 4kstacks default

[Eric Sandeen]

Adrian Bunk wrote:
> On Sat, Apr 19, 2008 at 04:35:31PM +0200, Oliver Pinter wrote:
>> ...
>> with the older kernel is typical: xfs+nfs+4k stack(+lvm)
> 
> Does anyone still experience problems with 2.6.25?

There are always problems.  You can always come up with something that
will crash in 4k, IMHO.

Rather than foisting this upon everyone, I'd rather see work put into
making stack size a boot parameter or something, so that people can
choose what's appropriate for their workload (or their IO stack, if you
prefer).

-Eric

> We all know that there once were problems, but if there are any left 
> they should be reported and fixed.
> 
>> Thanks,
>> Oliver
> 
> cu
> Adrian
> 

Re: x86: 4kstacks default

[Adrian Bunk]

On Sat, Apr 19, 2008 at 08:56:09PM -0500, Eric Sandeen wrote:
> Adrian Bunk wrote:
> > On Sat, Apr 19, 2008 at 04:35:31PM +0200, Oliver Pinter wrote:
> >> ...
> >> with the older kernel is typical: xfs+nfs+4k stack(+lvm)
> > 
> > Does anyone still experience problems with 2.6.25?
> 
> There are always problems.  You can always come up with something that
> will crash in 4k, IMHO.

We are going from 6k to 4k.

Your "You can always come up with something that will crash in" point 
would be invariant to this change (although it might be harder to 
trigger in real life).

> Rather than foisting this upon everyone, I'd rather see work put into
> making stack size a boot parameter or something, so that people can
> choose what's appropriate for their workload (or their IO stack, if you
> prefer).

Why should users have to poke with such deeply internal things?
That doesn't sound right.

Excessive stack usage in the kernel is considered to be a bug.

We should identify and fix all remaining problems (if any).

> -Eric

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: x86: 4kstacks default

[Chris Wedgwood]

On Sun, Apr 20, 2008 at 10:42:28AM +0300, Adrian Bunk wrote:

> We are going from 6k to 4k.

6k?

> Why should users have to poke with such deeply internal things?
> That doesn't sound right.

they shouldn't, so a 4k default is a problem for them

> Excessive stack usage in the kernel is considered to be a bug.

define excessive

> We should identify and fix all remaining problems (if any).

let's see your patches then

Re: x86: 4kstacks default

[Andi Kleen]

Eric Sandeen <sandeen@sandeen.net> writes:

> Adrian Bunk wrote:
>> On Sat, Apr 19, 2008 at 04:35:31PM +0200, Oliver Pinter wrote:
>>> ...
>>> with the older kernel is typical: xfs+nfs+4k stack(+lvm)
>> 
>> Does anyone still experience problems with 2.6.25?
>
> There are always problems.  You can always come up with something that
> will crash in 4k, IMHO.

But what are a few crashes compared against the ability to run 50000
kernel threads on a 32bit machine? Something has to give in the aim
for useless checkbox numbers after all. 

-Andi

Re: x86: 4kstacks default

[Shawn Bohrer]

On Sat, Apr 19, 2008 at 04:23:29PM +0200, Ingo Molnar wrote:
> 
> * Andrew Morton <akpm@linux-foundation.org> wrote:
> 
> > >  config 4KSTACKS
> > >  	bool "Use 4Kb for kernel stacks instead of 8Kb"
> > > -	depends on DEBUG_KERNEL
> > >  	depends on X86_32
> > > +	default y
> > 
> > This patch will cause kernels to crash.
> 
> what mainline kernels crash and how will they crash? Fedora and other 
> distros have had 4K stacks enabled for years:

If by other distros you mean RHEL then yes.  However, openSUSE,
Ubuntu, and Mandriva all still have 8K stacks.  I know of no other
distributions that default to 4K.

--
Shawn

Re: x86: 4kstacks default

[Arjan van de Ven]

On Sat, 19 Apr 2008 09:59:48 -0500
Shawn Bohrer <shawn.bohrer@gmail.com> wrote:

> On Sat, Apr 19, 2008 at 04:23:29PM +0200, Ingo Molnar wrote:
> > 
> > * Andrew Morton <akpm@linux-foundation.org> wrote:
> > 
> > > >  config 4KSTACKS
> > > >  	bool "Use 4Kb for kernel stacks instead of 8Kb"
> > > > -	depends on DEBUG_KERNEL
> > > >  	depends on X86_32
> > > > +	default y
> > > 
> > > This patch will cause kernels to crash.
> > 
> > what mainline kernels crash and how will they crash? Fedora and
> > other distros have had 4K stacks enabled for years:
> 
> If by other distros you mean RHEL then yes.  However, openSUSE,
> Ubuntu, and Mandriva all still have 8K stacks.  I know of no other
> distributions that default to 4K.

centos, oracle and redflag tend to follow the RHEL/fedora settings.

To be honest, at this point we're at a situation where
* Several very popular distributions have this enabled for 5+ years,
  apparently without any real issues (otherwise the enterprise releases
  would have turned this off)
* The early "hot known issues" have been resolved afaik, things like
  block device stacking, and symlink recursion lookups are either no longer
  recursive, or a lot less recursive than they used to be.

There are clear benefits to 4K stacks (no need to reiterate the flamewar,
but worth mentioning)
* Less memory consumption in the lowmem zone (critical for enterprise use,
  also good for general performance)
* Kernel stacks at 8K are one of the most prominent order-1 allocations in the
  kernel; again with big-memory systems the fragmentation of the lowmem zone
  is a problem (and the distros that ship 4K stacks went there because of customer
  complaints)

On the flipside the arguments tend to be
1) certain stackings of components still runs the risk of overflowing
2) I want to run ndiswrapper
3) general, unspecified uneasyness.

For 1), we need to know which they are, and then solve them, because even on x86-64 with 8k stacks
they can be a problem (just because the stack frames are bigger, although not quite double, there).
I've not seen any recent reports, I'll try to extend the kerneloops.org client to collect the
"stack is getting low" warning to be able to see how much this really happens.

for 2), the real answer there is "ndiswrapper needs 12kb not 8kb"

for 3), this is hard to deal with but also generally unfounded... you can use this argument against any change in the kernel.

Re: x86: 4kstacks default

[Ingo Molnar]

* Arjan van de Ven <arjan@infradead.org> wrote:

> On Sat, 19 Apr 2008 09:59:48 -0500
> Shawn Bohrer <shawn.bohrer@gmail.com> wrote:
> 
> > On Sat, Apr 19, 2008 at 04:23:29PM +0200, Ingo Molnar wrote:
> > > 
> > > * Andrew Morton <akpm@linux-foundation.org> wrote:
> > > 
> > > > >  config 4KSTACKS
> > > > >  	bool "Use 4Kb for kernel stacks instead of 8Kb"
> > > > > -	depends on DEBUG_KERNEL
> > > > >  	depends on X86_32
> > > > > +	default y
> > > > 
> > > > This patch will cause kernels to crash.
> > > 
> > > what mainline kernels crash and how will they crash? Fedora and
> > > other distros have had 4K stacks enabled for years:
> > 
> > If by other distros you mean RHEL then yes.  However, openSUSE,
> > Ubuntu, and Mandriva all still have 8K stacks.  I know of no other
> > distributions that default to 4K.
> 
> centos, oracle and redflag tend to follow the RHEL/fedora settings.
> 
> To be honest, at this point we're at a situation where
> * Several very popular distributions have this enabled for 5+ years,
>   apparently without any real issues (otherwise the enterprise releases
>   would have turned this off)
> * The early "hot known issues" have been resolved afaik, things like
>   block device stacking, and symlink recursion lookups are either no longer
>   recursive, or a lot less recursive than they used to be.
> 
> There are clear benefits to 4K stacks (no need to reiterate the flamewar,
> but worth mentioning)
> * Less memory consumption in the lowmem zone (critical for enterprise use,
>   also good for general performance)
> * Kernel stacks at 8K are one of the most prominent order-1 allocations in the
>   kernel; again with big-memory systems the fragmentation of the lowmem zone
>   is a problem (and the distros that ship 4K stacks went there because of customer
>   complaints)
> 
> On the flipside the arguments tend to be
> 1) certain stackings of components still runs the risk of overflowing
> 2) I want to run ndiswrapper
> 3) general, unspecified uneasyness.
> 
> For 1), we need to know which they are, and then solve them, because 
> even on x86-64 with 8k stacks they can be a problem (just because the 
> stack frames are bigger, although not quite double, there). I've not 
> seen any recent reports, I'll try to extend the kerneloops.org client 
> to collect the "stack is getting low" warning to be able to see how 
> much this really happens.
> 
> for 2), the real answer there is "ndiswrapper needs 12kb not 8kb"
> 
> for 3), this is hard to deal with but also generally unfounded... you 
> can use this argument against any change in the kernel.

and lets observe it that 8K stacks are of course still offered, so if 
anyone disables 4K stacks in the .config, it will stay disabled.

	Ingo

Re: x86: 4kstacks default

[Stefan Richter]

Ingo Molnar wrote:
> and lets observe it that 8K stacks are of course still offered, so if 
> anyone disables 4K stacks in the .config, it will stay disabled.

While you change the default, maybe move it also from the "Kernel 
hacking" menu into the "General setup" menu?  An option with default=y 
is probably not an option that is targeted towards kernel hackers only.
-- 
Stefan Richter
-=====-==--- -=-- =--==
http://arcgraph.de/sr/

Re: x86: 4kstacks default

[Eric Sandeen]

Arjan van de Ven wrote:

> On the flipside the arguments tend to be
> 1) certain stackings of components still runs the risk of overflowing
> 2) I want to run ndiswrapper
> 3) general, unspecified uneasyness.
> 
> For 1), we need to know which they are, and then solve them, because even on x86-64 with 8k stacks
> they can be a problem (just because the stack frames are bigger, although not quite double, there).

Except, apparently, not, at least in my experience.

Ask the xfs guys if they see stack overflows on x86_64, or on x86.

I've personally never seen common stack problems with xfs on x86_64, but
it's very common on x86.  I don't have a great answer for why, but
that's my anecdotal evidence.

> I've not seen any recent reports, I'll try to extend the kerneloops.org client to collect the
> "stack is getting low" warning to be able to see how much this really happens.

That sounds like a very good thing to collect, and maybe if I re-send a
"clearly state stack overflows at oops time" patch you can easily keep tabs.

Thanks,

-Eric

Re: x86: 4kstacks default

[Arjan van de Ven]

On Sat, 19 Apr 2008 21:36:16 -0500
Eric Sandeen <sandeen@sandeen.net> wrote:


> > For 1), we need to know which they are, and then solve them,
> > because even on x86-64 with 8k stacks they can be a problem (just
> > because the stack frames are bigger, although not quite double,
> > there).
> 
> Except, apparently, not, at least in my experience.

if you actually go over on x86, it's not unlikely that you're getting close to the edge on 64 bit.

At minimum we really do want to fix these things...

> I've personally never seen common stack problems with xfs on x86_64,
> but it's very common on x86.  I don't have a great answer for why, but
> that's my anecdotal evidence.

One thing I've learned with the kerneloops.org work is that people don't read
their dmesg..... 
> 
> > I've not seen any recent reports, I'll try to extend the
> > kerneloops.org client to collect the "stack is getting low" warning
> > to be able to see how much this really happens.
> 
> That sounds like a very good thing to collect, and maybe if I re-send
> a "clearly state stack overflows at oops time" patch you can easily
> keep tabs.

... which makes me think we need to strengthen this part of the kernel.
(and then have kerneloops.org collect the issues)

If there's a clear pattern in the backtraces we will find it. 
And then we can fix it... which is absolutely the right thing,
I don't think anyone disagrees with that.

So yes if you can dig up your patch, yes please!


-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: x86: 4kstacks default

[David Chinner]

On Sat, Apr 19, 2008 at 09:36:16PM -0500, Eric Sandeen wrote:
> Arjan van de Ven wrote:
> 
> > On the flipside the arguments tend to be
> > 1) certain stackings of components still runs the risk of overflowing
> > 2) I want to run ndiswrapper
> > 3) general, unspecified uneasyness.
> > 
> > For 1), we need to know which they are, and then solve them, because even on x86-64 with 8k stacks
> > they can be a problem (just because the stack frames are bigger, although not quite double, there).
> 
> Except, apparently, not, at least in my experience.
> 
> Ask the xfs guys if they see stack overflows on x86_64, or on x86.

We see them regularly enough on x86 to know that the first question
to any strange crash is "are you using 4k stacks?". In comparison,
I have never heard of a single stack overflow on x86_64....

> I've personally never seen common stack problems with xfs on x86_64, but
> it's very common on x86.  I don't have a great answer for why, but
> that's my anecdotal evidence.

Why? Because XFS makes extensive use of 64 bit types and so stack
usage in the critical paths changes by a relatively small amount
between 32 bit and 64 bit machines.  IIRC, x86_64 only uses about
30% more stack than x86. So given that the stack doubles on x86_64
and we only increase usage (in XFS) from about 1500 bytes to 2000
bytes of stack usage, we have *lots* more stack space to spare on
x86_64 compared to 4k stacks on x86....

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: x86: 4kstacks default

[Adrian Bunk]

On Sat, Apr 19, 2008 at 09:59:48AM -0500, Shawn Bohrer wrote:
> On Sat, Apr 19, 2008 at 04:23:29PM +0200, Ingo Molnar wrote:
> > 
> > * Andrew Morton <akpm@linux-foundation.org> wrote:
> > 
> > > >  config 4KSTACKS
> > > >  	bool "Use 4Kb for kernel stacks instead of 8Kb"
> > > > -	depends on DEBUG_KERNEL
> > > >  	depends on X86_32
> > > > +	default y
> > > 
> > > This patch will cause kernels to crash.
> > 
> > what mainline kernels crash and how will they crash? Fedora and other 
> > distros have had 4K stacks enabled for years:
> 
> If by other distros you mean RHEL then yes.  However, openSUSE,
> Ubuntu, and Mandriva all still have 8K stacks.  I know of no other
> distributions that default to 4K.

MontaVista offers 4k stacks for arm (currently an external patch) and 
markets that as a feature to customers, so many of them might use it.

In-kernel the sh and m68knommu ports also offer 4k stacks (for both 
archs there's also a defconfig using it), and the mn10300 port contains 
an #ifdef but no config option.

The stack problems in the kernel tend to not be in arch code, and if 
we don't get i386 to always run with 4k stacks there's no chance that 
it will ever work reliably on other architectures.

> Shawn

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: x86: 4kstacks default

[Alan Cox]

> The stack problems in the kernel tend to not be in arch code, and if 
> we don't get i386 to always run with 4k stacks there's no chance that 
> it will ever work reliably on other architectures.

Not really the case - embedded tends not to use deep stacks of drivers.

Alan

Re: x86: 4kstacks default

[Adrian Bunk]

On Sun, Apr 20, 2008 at 09:06:23AM +0100, Alan Cox wrote:
> > The stack problems in the kernel tend to not be in arch code, and if 
> > we don't get i386 to always run with 4k stacks there's no chance that 
> > it will ever work reliably on other architectures.
> 
> Not really the case - embedded tends not to use deep stacks of drivers.

Something like nfsd-over-xfs-over-raid is (or was) the most common 
problem - and this or similar stackings might be used in NAS devices.

> Alan

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: x86: 4kstacks default

[Alan Cox]

On Sun, 20 Apr 2008 11:51:04 +0300
Adrian Bunk <bunk@kernel.org> wrote:

> On Sun, Apr 20, 2008 at 09:06:23AM +0100, Alan Cox wrote:
> > > The stack problems in the kernel tend to not be in arch code, and if 
> > > we don't get i386 to always run with 4k stacks there's no chance that 
> > > it will ever work reliably on other architectures.
> > 
> > Not really the case - embedded tends not to use deep stacks of drivers.
> 
> Something like nfsd-over-xfs-over-raid is (or was) the most common 
> problem - and this or similar stackings might be used in NAS devices.

Specific cases yes, but such NAS devices have big processors and are not
little emdedded CPUs. On an embedded box you know at build time what it
will be doing.

Re: x86: 4kstacks default

[Adrian Bunk]

On Sun, Apr 20, 2008 at 10:36:11AM +0100, Alan Cox wrote:
> On Sun, 20 Apr 2008 11:51:04 +0300
> Adrian Bunk <bunk@kernel.org> wrote:
> 
> > On Sun, Apr 20, 2008 at 09:06:23AM +0100, Alan Cox wrote:
> > > > The stack problems in the kernel tend to not be in arch code, and if 
> > > > we don't get i386 to always run with 4k stacks there's no chance that 
> > > > it will ever work reliably on other architectures.
> > > 
> > > Not really the case - embedded tends not to use deep stacks of drivers.
> > 
> > Something like nfsd-over-xfs-over-raid is (or was) the most common 
> > problem - and this or similar stackings might be used in NAS devices.
> 
> Specific cases yes, but such NAS devices have big processors and are not
> little emdedded CPUs. On an embedded box you know at build time what it
> will be doing.

The code in the kernel that gets the fewest coverage at all are our 
error paths, and some vendor might try 4k stacks, validate it works in 
all use cases - and then it will blow up in some error condition he 
didn't test.

6k is known to work, and there aren't many problems known with 4k.

And from a QA point of view the only way of getting 4k thoroughly tested 
by users, and well also tested in -rc kernels for catching regressions 
before they get into stable kernels, is if we get 4k stacks enabled 
unconditionally on i386.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: x86: 4kstacks default

[Alan Cox]

> The code in the kernel that gets the fewest coverage at all are our 
> error paths, and some vendor might try 4k stacks, validate it works in 
> all use cases - and then it will blow up in some error condition he 
> didn't test.

Which you won't fix by changing the x86 defaults. More of a problem in
embedded small devices is the 8K allocation failing in the first place -
plus 4K x 80 processes == lots.

> And from a QA point of view the only way of getting 4k thoroughly tested 
> by users, and well also tested in -rc kernels for catching regressions 
> before they get into stable kernels, is if we get 4k stacks enabled 
> unconditionally on i386.

At which point some distros will simply patch it back no doubt.

Re: x86: 4kstacks default

[Adrian Bunk]

On Sun, Apr 20, 2008 at 12:02:50PM +0100, Alan Cox wrote:
> > The code in the kernel that gets the fewest coverage at all are our 
> > error paths, and some vendor might try 4k stacks, validate it works in 
> > all use cases - and then it will blow up in some error condition he 
> > didn't test.
> 
> Which you won't fix by changing the x86 defaults.

Stuff like nfsd, xfs and raid is covered by the x86 defaults.

It's not a 100% coverage, but quite much.

> More of a problem in
> embedded small devices is the 8K allocation failing in the first place -
> plus 4K x 80 processes == lots.
> 
> > And from a QA point of view the only way of getting 4k thoroughly tested 
> > by users, and well also tested in -rc kernels for catching regressions 
> > before they get into stable kernels, is if we get 4k stacks enabled 
> > unconditionally on i386.
> 
> At which point some distros will simply patch it back no doubt.

Red Hat seems to get usable kernels with 4k for some years?

If we get whatever is still missing for 4k working once and then the 
coverage of all i386 -rc testers for noticing new issues immediately
there should be no stability reason for distros to patch it back in.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: x86: 4kstacks default

[Alan Cox]

On Sun, 20 Apr 2008 14:54:55 +0300
Adrian Bunk <bunk@kernel.org> wrote:

> Red Hat seems to get usable kernels with 4k for some years?

Yes and I think it is the right setting.

> If we get whatever is still missing for 4k working once and then the 
> coverage of all i386 -rc testers for noticing new issues immediately
> there should be no stability reason for distros to patch it back in.

You don't get to dictate to people however. 

Alan
--
  "If we become a great evil avaricious hegemony, I wanna cool uniform"
                        -- robk

Re: x86: 4kstacks default

[Adrian Bunk]

On Sun, Apr 20, 2008 at 12:37:31PM +0100, Alan Cox wrote:
> On Sun, 20 Apr 2008 14:54:55 +0300
> Adrian Bunk <bunk@kernel.org> wrote:
> 
> > Red Hat seems to get usable kernels with 4k for some years?
> 
> Yes and I think it is the right setting.
> 
> > If we get whatever is still missing for 4k working once and then the 
> > coverage of all i386 -rc testers for noticing new issues immediately
> > there should be no stability reason for distros to patch it back in.
> 
> You don't get to dictate to people however. 

Everyone is free to patch whatever stacksize he wants into his kernel.

But the more users will get 4k stacks the more testing we have, and the 
better both existing and new bugs get shaken out.

And if there were only 4k stacks in the vanilla kernel, and therefore 
all people on i386 testing -rc kernels would get it, that would give a 
better chance of finding stack regressions before they get into a 
stable kernel.

If a distribution or user then wants to increase it that's his choice 
(and easy to do), but nothing the upstream kernel has to offer.

> Alan

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: x86: 4kstacks default

[Eric Sandeen]

Adrian Bunk wrote:

> But the more users will get 4k stacks the more testing we have, and the 
> better both existing and new bugs get shaken out.
> 
> And if there were only 4k stacks in the vanilla kernel, and therefore 
> all people on i386 testing -rc kernels would get it, that would give a 
> better chance of finding stack regressions before they get into a 
> stable kernel.

Heck, maybe you should make it 2k by default in all -rc kernels; that
way when people run -final with the 4k it'll be 100% bulletproof, right?
 'cause all those piggy drivers that blow a 2k stack will finally have
to get fixed?  Or leave it at 2k and find a way to share pages for
stacks, think how much memory you could save and how many java threads
you could run!

4K just happens to be the page size; other than that it's really just
some random/magic number picked, and now dictated that if you (and
everyting around you) doesn't fit, you're broken.

That bugs me.

-Eric

(yes, I know there are advantages to only allocating a single page for a
new thread, but from an "all callchains after that must fit in that
space" perspective, it's just a randomly picked number)

Re: x86: 4kstacks default

[Adrian Bunk]

On Sun, Apr 20, 2008 at 09:05:40AM -0500, Eric Sandeen wrote:
> Adrian Bunk wrote:
> 
> > But the more users will get 4k stacks the more testing we have, and the 
> > better both existing and new bugs get shaken out.
> > 
> > And if there were only 4k stacks in the vanilla kernel, and therefore 
> > all people on i386 testing -rc kernels would get it, that would give a 
> > better chance of finding stack regressions before they get into a 
> > stable kernel.
> 
> Heck, maybe you should make it 2k by default in all -rc kernels; that
> way when people run -final with the 4k it'll be 100% bulletproof, right?
>  'cause all those piggy drivers that blow a 2k stack will finally have
> to get fixed?

I'm arguing for aiming at having all 32bit architectures with 4k page 
size using the same stack size. Not for having -rc kernels differ from 
release kernels.

>  Or leave it at 2k and find a way to share pages for
> stacks, think how much memory you could save and how many java threads
> you could run!

The only architecture that already defaults to 4k stacks is m68knommu, 
and I doubt they do it for many java threads...

>...
> -Eric
>...

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: x86: 4kstacks default

[Eric Sandeen]

Adrian Bunk wrote:
> On Sun, Apr 20, 2008 at 09:05:40AM -0500, Eric Sandeen wrote:
>> Adrian Bunk wrote:
>>
>>> But the more users will get 4k stacks the more testing we have, and the 
>>> better both existing and new bugs get shaken out.
>>>
>>> And if there were only 4k stacks in the vanilla kernel, and therefore 
>>> all people on i386 testing -rc kernels would get it, that would give a 
>>> better chance of finding stack regressions before they get into a 
>>> stable kernel.
>> Heck, maybe you should make it 2k by default in all -rc kernels; that
>> way when people run -final with the 4k it'll be 100% bulletproof, right?
>>  'cause all those piggy drivers that blow a 2k stack will finally have
>> to get fixed?
> 
> I'm arguing for aiming at having all 32bit architectures with 4k page 
> size using the same stack size. Not for having -rc kernels differ from 
> release kernels.

Oh, I know.  I'm just saying that 4k seems chosen out of convenience for
memory management, without any real correlation to what you might
actually need to run a thread.  They do happen to be roughly equivalent
for many cases, but not all.  Setting a default which is not safe for
several common use cases does not seem wise...

I guess what I'm saying is, I don't agree that any callchain which needs
more than 4k of stack indicates brokenness that must be fixed, as
various posts in this thread seem to suggest.

Sure, 1k char buffers on the stack and massive structs and unlimited
recursion we can agree on as things to fix, but complex/deep/stacked
callchains which don't fit in 4k are much more of a grey area.

-Eric

Re: x86: 4kstacks default

[Arjan van de Ven]

On Sun, 20 Apr 2008 09:05:40 -0500
Eric Sandeen <sandeen@sandeen.net> wrote:

> 
> 4K just happens to be the page size; other than that it's really just
> some random/magic number picked, and now dictated that if you (and
> everyting around you) doesn't fit, you're broken.

it wasn't randomly picked; it was based on 2.4 kernels
(where we had 8kb, but that was roughly 2.5Kb or so for the task struct,
which was on stack back then, then 4Kb for user context and 2Kb for IRQ context)

> 
> That bugs me.
> 

yes. Adrian is waay off in the weeds on this one. Nobody but him is suggesting to remove
8Kb stacks. I think everyone else agrees that having both options is valuable; and there
are better ways to find+fix stack bloat than removing this config option.


-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: x86: 4kstacks default

[Adrian Bunk]

On Sun, Apr 20, 2008 at 08:41:27AM -0700, Arjan van de Ven wrote:
>...
> yes. Adrian is waay off in the weeds on this one. Nobody but him is suggesting to remove
> 8Kb stacks. I think everyone else agrees that having both options is valuable; and there
> are better ways to find+fix stack bloat than removing this config option.

I'm not arguing for removing the option immediately, but long-term we 
shouldn't need it.

This comes from my experience of removing obsolete drivers for hardware 
for which also a more recent driver exists:
As long as there is some workaround (e.g. using an older driver or
8k stacks) the workaround will be used instead of the getting proper 
bug reports and fixes.

As far as I know all problems that are known with 4k stacks are some 
nested things with XFS in the trace.

If this class of issues would get fixed one day, why would it be 
valuable to also offer 8k stacks long-term? Especially weigthed
against the fact that with only 4k stacks we will have more people
running into stack problems in -rc kernels if any new ones pop up,
resulting in getting more such problems fixed during -rc.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: x86: 4kstacks default

[Alexander E. Patrakov]

Adrian Bunk wrote:
> On Sun, Apr 20, 2008 at 08:41:27AM -0700, Arjan van de Ven wrote:
>> ...
>> yes. Adrian is waay off in the weeds on this one. Nobody but him is suggesting to remove
>> 8Kb stacks. I think everyone else agrees that having both options is valuable; and there
>> are better ways to find+fix stack bloat than removing this config option.
> 
> I'm not arguing for removing the option immediately, but long-term we 
> shouldn't need it.
> 
> This comes from my experience of removing obsolete drivers for hardware 
> for which also a more recent driver exists:
> As long as there is some workaround (e.g. using an older driver or
> 8k stacks) the workaround will be used instead of the getting proper 
> bug reports and fixes.
> 
> As far as I know all problems that are known with 4k stacks are some 
> nested things with XFS in the trace.

This "as far as I know" is a problem itself. Is it possible to implement (e.g., 
using some form of memory protection in hardware, but I am not an expert here) 
an option with 8k stacks that, however, spams the log if the actual usage goes 
above 4k, and have this as a default for some time? If 4k stacks are the goal 
that is almost achieved, then this debugging option should have zero impact on 
performance.

-- 
Alexander E. Patrakov

Re: x86: 4kstacks default

[Helge Hafting]

Alexander E. Patrakov wrote:
> Adrian Bunk wrote:
>> On Sun, Apr 20, 2008 at 08:41:27AM -0700, Arjan van de Ven wrote:
>>> ...
>>> yes. Adrian is waay off in the weeds on this one. Nobody but him is 
>>> suggesting to remove
>>> 8Kb stacks. I think everyone else agrees that having both options is 
>>> valuable; and there
>>> are better ways to find+fix stack bloat than removing this config 
>>> option.
>>
>> I'm not arguing for removing the option immediately, but long-term we 
>> shouldn't need it.
>>
>> This comes from my experience of removing obsolete drivers for 
>> hardware for which also a more recent driver exists:
>> As long as there is some workaround (e.g. using an older driver or
>> 8k stacks) the workaround will be used instead of the getting proper 
>> bug reports and fixes.
>>
>> As far as I know all problems that are known with 4k stacks are some 
>> nested things with XFS in the trace.
>
> This "as far as I know" is a problem itself. Is it possible to 
> implement (e.g., using some form of memory protection in hardware, but 
> I am not an expert here) an option with 8k stacks that, however, spams 
> the log if the actual usage goes above 4k, and have this as a default 
> for some time? If 4k stacks are the goal that is almost achieved, then 
> this debugging option should have zero impact on performance.
Shouldn't be hard. Use the 8k stack, and have the system mark the second 
page as "not present"
If it ever gets used you get a page fault. The page fault handler then 
have to mark the page
present before returning, as well as queue up some spam (the call chain 
perhaps) for the log.

A less intrusive way is to use 8k stacks as-is, but put a signature in 
the second page.
When the process quits, examine the second stack page to see if the 
signature
got overwritten. This approach will only show that a problem exists, it 
won't
pinpoint exactly what does it.

Helge Hafting

Re: x86: 4kstacks default

[Denys Vlasenko]

On Sunday 20 April 2008 16:05, Eric Sandeen wrote:
> Adrian Bunk wrote:
> > But the more users will get 4k stacks the more testing we have, and the 
> > better both existing and new bugs get shaken out.
> > 
> > And if there were only 4k stacks in the vanilla kernel, and therefore 
> > all people on i386 testing -rc kernels would get it, that would give a 
> > better chance of finding stack regressions before they get into a 
> > stable kernel.
> 
> Heck, maybe you should make it 2k by default in all -rc kernels; that
> way when people run -final with the 4k it'll be 100% bulletproof, right?
>  'cause all those piggy drivers that blow a 2k stack will finally have
> to get fixed?  Or leave it at 2k and find a way to share pages for
> stacks, think how much memory you could save and how many java threads
> you could run!
> 
> 4K just happens to be the page size; other than that it's really just
> some random/magic number picked, and now dictated that if you (and
> everyting around you) doesn't fit, you're broken.

Some number has to be picked. Why fitting in 4k is "bad" and fitting
in 8k is "not bad"?

Look what happens when this number is too big: Windows is "generous",
and as a result Windows drivers routinely need 12k, sometimes 16k of stack.
We know it from ndiswrapper. We don't want to go that way, right?

Forget about 50k threads. 4k of waste per process is a waste nevertheless.
It's not at all unusual to have 250+ processes, and 250 processes with 8k
stack each waste 1M. Do you think extra 1M won't be useful to have?

It seems that 4k works for everybody sans xfs. Making it work took some effort,
but it is already done. Why not use it after all?

And since i386 is such a common architecture, other 32-bit arches will be
relieved from the burden of hunting down stack overflows which happen
only on those arches. (For example, different ABI or different gcc behavior
may make $OTHER_ARCH slightly more stack-greedy). God knows non-mainstream
arches have enough problems already.
--
vda

Re: x86: 4kstacks default

[Andi Kleen]

Denys Vlasenko <vda.linux@googlemail.com> writes:
>
> Forget about 50k threads. 4k of waste per process is a waste nevertheless.
> It's not at all unusual to have 250+ processes, and 250 processes with 8k
> stack each waste 1M. Do you think extra 1M won't be useful to have?

If the 1M gives you more reliability (and I think it does) I don't 
think it is "wasted". Would you trade occasional crashes for 1MB? 
I wouldn't.

Also a typical process uses much more memory than just 4K. If it's
not a thread it needs own page tables and from those alone you're
easily into 10+ pages even for a quite small process. But even threads
in practice have other overheads too if they actually do something.
The 4K won't save or break you.

[BTW if you're really interested in saving memory there are lots
of other subsystems where you could very likely save more. A common
example are the standard hash tables which are still too big]

The trends are also against it: kernel code is getting more and more
complex all the time with more and more complicated stacks of
different subsystems on top of each other. It wouldn't surprise me if
at some point 8KB isn't even enough anymore. Going into the
other direction is definitely the wrong way.

-Andi

Re: x86: 4kstacks default

[Eric Sandeen]

Denys Vlasenko wrote:
> On Sunday 20 April 2008 16:05, Eric Sandeen wrote:
>> Adrian Bunk wrote:
>>> But the more users will get 4k stacks the more testing we have, and the 
>>> better both existing and new bugs get shaken out.
>>>
>>> And if there were only 4k stacks in the vanilla kernel, and therefore 
>>> all people on i386 testing -rc kernels would get it, that would give a 
>>> better chance of finding stack regressions before they get into a 
>>> stable kernel.
>> Heck, maybe you should make it 2k by default in all -rc kernels; that
>> way when people run -final with the 4k it'll be 100% bulletproof, right?
>>  'cause all those piggy drivers that blow a 2k stack will finally have
>> to get fixed?  Or leave it at 2k and find a way to share pages for
>> stacks, think how much memory you could save and how many java threads
>> you could run!
>>
>> 4K just happens to be the page size; other than that it's really just
>> some random/magic number picked, and now dictated that if you (and
>> everyting around you) doesn't fit, you're broken.
> 
> Some number has to be picked. Why fitting in 4k is "bad" and fitting
> in 8k is "not bad"?


Because well-written code in several subsystems, used in combination in
common configurations, does not always fit, that is why.

Show me the "bug" in an nfs+xfs+md+scsi writeback stack oops and I'm
sure it'll get "fixed."  But if it's simply complex code that happens to
need >4k, I will continue to argue that the limited stack size selection
is the problem, not the code running in it.

Perhaps not surprisingly, ext4, which is significantly more complex than
ext3, has many more individual functions > 100 bytes than ext3 has.  As
others have said, there is no trend towards smaller, simpler, less
interesting, and less functional code which fits in a smaller and
smaller footprint in the general case.

If someone has a workload and configuration which happens to fit in 4k
then turn it on, test the heck out of it, and have fun.  I've not seen
what I consider to be a convincing argument for making it the default
for everyone.

-Eric

Re: x86: 4kstacks default

[Denys Vlasenko]

On Monday 21 April 2008 15:29, Eric Sandeen wrote:
> > Some number has to be picked. Why fitting in 4k is "bad" and fitting
> > in 8k is "not bad"?
> 
> 
> Because well-written code in several subsystems, used in combination in
> common configurations, does not always fit, that is why.
> 
> Show me the "bug" in an nfs+xfs+md+scsi writeback stack oops

Why nfs+xfs+md+ide works? Does scsi intrinsically require more stack
than ide?

Why xfs code is said to be 5 timed bigged than e.g. reiserfs?
Does it have to be that big? Does it really have to eat lots of stack?

> and I'm 
> sure it'll get "fixed."  But if it's simply complex code that happens to
> need >4k, I will continue to argue that the limited stack size selection
> is the problem, not the code running in it.

8k stack is limited too. Other Operating System, no doubt in the name
of better stability, has even larger stack (16k or more).

For what its worth, I do realize that there is a point of diminishing
returns and increased pain when one tries to reduce stack usage.
I feel that 4k for 32-bit x86 is not too painful. IMO, of course.

> If someone has a workload and configuration which happens to fit in 4k
> then turn it on, test the heck out of it, and have fun.  I've not seen
> what I consider to be a convincing argument for making it the default
> for everyone.

Conversely:

"If someone is strongly concerned about possibility of stack overflow,
then turn on 8k option, and enjoy the benefits of wide testing which
is provided by millions of people who run 4k stacks. If _that_ works
ok in practice, 8k _ought_ to be 100.00% safe versus stack overflow".

These threads about 4k stack seem to degenerate in ping-ponging
of these arguments again and again.
--
vda

Re: x86: 4kstacks default

[Denys Vlasenko]

On Monday 21 April 2008 21:51, Denys Vlasenko wrote:
> Why xfs code is said to be 5 timed bigged than e.g. reiserfs?

s/timed bigged/times bigger/

Re: x86: 4kstacks default

[David Chinner]

On Mon, Apr 21, 2008 at 09:51:02PM +0200, Denys Vlasenko wrote:
> On Monday 21 April 2008 15:29, Eric Sandeen wrote:
> > > Some number has to be picked. Why fitting in 4k is "bad" and fitting
> > > in 8k is "not bad"?
> > 
> > 
> > Because well-written code in several subsystems, used in combination in
> > common configurations, does not always fit, that is why.
> > 
> > Show me the "bug" in an nfs+xfs+md+scsi writeback stack oops
> 
> Why nfs+xfs+md+ide works?

Luck?

With 4k stacks, you really don't need NFS at all - you just have
enter memory reclaim at the wrong time (i.e. when something else
was already consuming 2/3rds of the 4k stack).

> Does scsi intrinsically require more stack than ide?

<shrug>

> Why xfs code is said to be 5 timed bigged than e.g. reiserfs?
> Does it have to be that big?

If we cut the bulkstat code out, the handle interface, the
preallocation, the journalled quota, the delayed allocation, all the
runtime validation, the shutdown code, the debug code, the tracing
code, etc, then we might get down to the same size reiser....

> Does it really have to eat lots of stack?

Writeback is done under ENOMEM pressure, and XFS can't provide the
guarantees mempools need to work. That leaves the stack as the only
place we can put the things we need. e.g. the args structures that
tell the allocator what to do and retain state between subsequent
low level allocation calls use ~250 bytes of stack just by
themselves....

We've already chopped off the low hanging fruit, added noinline to
every function definition to prevent compiler heuristics from
blowing out stack usage by 25% and reduced use of temporary
variables as much as possible. There's very little fat left to trim,
and still we can't reliably fit in 4k stacks.

Patches are welcome - I'd be over the moon if any of the known 4k
stack advocates sent a stack reduction patch for XFS, but it seems
that actually trying to fix the problems is much harder than
resending a one line patch every few months....

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: x86: 4kstacks default

[Denys Vlasenko]

On Tuesday 22 April 2008 03:28, David Chinner wrote:
> We've already chopped off the low hanging fruit, added noinline to
> every function definition to prevent compiler heuristics from
> blowing out stack usage by 25% and reduced use of temporary
> variables as much as possible. There's very little fat left to trim,
> and still we can't reliably fit in 4k stacks.

At yet, I got four screenfuls of

fs/xfs/XXXXX.c: warning: unused parameter 'foo'

when I added -Wunused_parameter to Makefile.
Clearly there is some room for improvement.

> Patches are welcome - I'd be over the moon if any of the known 4k
> stack advocates sent a stack reduction patch for XFS, but it seems
> that actually trying to fix the problems is much harder than
> resending a one line patch every few months....

Sent a few.
I would like to ask you to ACK/NAK every individual patch
in some reasonable period of time, say, 1-3 days. If you NAK a patch,
please let me know what is wrong with it.

I am not eager at all to experience a repeat of aic7xxx
patch saga, when I was not getting any meaningful reply
for months.

Best regards, Denys.
--
vda

Re: x86: 4kstacks default

[Adrian Bunk]

On Tue, Apr 22, 2008 at 02:48:09PM +0200, Denys Vlasenko wrote:
> On Tuesday 22 April 2008 03:28, David Chinner wrote:
>...
> > Patches are welcome - I'd be over the moon if any of the known 4k
> > stack advocates sent a stack reduction patch for XFS, but it seems
> > that actually trying to fix the problems is much harder than
> > resending a one line patch every few months....
> 
> Sent a few.
> I would like to ask you to ACK/NAK every individual patch
> in some reasonable period of time, say, 1-3 days. If you NAK a patch,
> please let me know what is wrong with it.
>...

I know the feeling of resending patches again and again without any 
reaction quite well, but that's not David's fault and not true for XFS 
patches, so when you try to put pressure on him you hit the wrong 
person.

> Best regards, Denys.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: x86: 4kstacks default

[Denys Vlasenko]

On Tuesday 22 April 2008 15:01, Adrian Bunk wrote:
> > Sent a few.
> > I would like to ask you to ACK/NAK every individual patch
> > in some reasonable period of time, say, 1-3 days. If you NAK a patch,
> > please let me know what is wrong with it.
> >...
> 
> I know the feeling of resending patches again and again without any 
> reaction quite well, but that's not David's fault and not true for XFS 
> patches, so when you try to put pressure on him you hit the wrong 
> person.

Yeah, sorry about that. I was not implying that XFS people were
not responsive.
--
vda

Re: x86: 4kstacks default

[Jörn Engel]

On Tue, 22 April 2008 11:28:19 +1000, David Chinner wrote:
> On Mon, Apr 21, 2008 at 09:51:02PM +0200, Denys Vlasenko wrote:
> 
> > Why xfs code is said to be 5 timed bigged than e.g. reiserfs?
> > Does it have to be that big?
> 
> If we cut the bulkstat code out, the handle interface, the
> preallocation, the journalled quota, the delayed allocation, all the
> runtime validation, the shutdown code, the debug code, the tracing
> code, etc, then we might get down to the same size reiser....

Just noticed this bit of FUD.  Last time I did some static analysis on
stack usage, reiserfs alone would blow away 3k, while xfs was somewhere
below.  Reiserfs was improved afaik, but I'd still expect it to be worse
than xfs until shown otherwise.

Maybe reiserfs simply isn't used that much in nfs+*fs+md+whatnot+scsi
setups?

Jörn

-- 
Courage is not the absence of fear, but rather the judgement that
something else is more important than fear.
-- Ambrose Redmoon

Re: x86: 4kstacks default

[Denys Vlasenko]

On Sunday 27 April 2008 21:27, Jörn Engel wrote:
> On Tue, 22 April 2008 11:28:19 +1000, David Chinner wrote:
> > On Mon, Apr 21, 2008 at 09:51:02PM +0200, Denys Vlasenko wrote:
> > 
> > > Why xfs code is said to be 5 times bigger than e.g. reiserfs?
> > > Does it have to be that big?
> > 
> > If we cut the bulkstat code out, the handle interface, the
> > preallocation, the journalled quota, the delayed allocation, all the
> > runtime validation, the shutdown code, the debug code, the tracing
> > code, etc, then we might get down to the same size reiser....
> 
> Just noticed this bit of FUD. Last time I did some static analysis on
> stack usage, reiserfs alone would blow away 3k, while xfs was somewhere
> below.

I'm sorry, but it's not what I said.
I didn't say reiserfs eats less stack. I don't know.
I said it is smaller.

reiserfs/*  821474 bytes
xfs/*      3019689 bytes
--
vda

Re: x86: 4kstacks default

[Eric Sandeen]

Denys Vlasenko wrote:
> On Sunday 27 April 2008 21:27, Jörn Engel wrote:
>> On Tue, 22 April 2008 11:28:19 +1000, David Chinner wrote:
>>> On Mon, Apr 21, 2008 at 09:51:02PM +0200, Denys Vlasenko wrote:
>>>
>>>> Why xfs code is said to be 5 times bigger than e.g. reiserfs?
>>>> Does it have to be that big?
>>> If we cut the bulkstat code out, the handle interface, the
>>> preallocation, the journalled quota, the delayed allocation, all the
>>> runtime validation, the shutdown code, the debug code, the tracing
>>> code, etc, then we might get down to the same size reiser....
>> Just noticed this bit of FUD. Last time I did some static analysis on
>> stack usage, reiserfs alone would blow away 3k, while xfs was somewhere
>> below.
> 
> I'm sorry, but it's not what I said.
> I didn't say reiserfs eats less stack. I don't know.
> I said it is smaller.
> 
> reiserfs/*  821474 bytes
> xfs/*      3019689 bytes

FWIW, the reason for that is in large part all the features Dave listed
above, and probably more.

And, while certainly not yet tiny, the recent trend actually is that xfs
is getting a bit smaller:

http://oss.sgi.com/~sandeen/xfs-linedata.png

(note, though - the Y axis does not start at 0)  :)

-Eric

Re: x86: 4kstacks default

[Denys Vlasenko]

On Monday 28 April 2008 01:08, Eric Sandeen wrote:
> >>>> Why xfs code is said to be 5 times bigger than e.g. reiserfs?
> >>>> Does it have to be that big?
> >>> If we cut the bulkstat code out, the handle interface, the
> >>> preallocation, the journalled quota, the delayed allocation, all the
> >>> runtime validation, the shutdown code, the debug code, the tracing
> >>> code, etc, then we might get down to the same size reiser....
> >> Just noticed this bit of FUD. Last time I did some static analysis on
> >> stack usage, reiserfs alone would blow away 3k, while xfs was somewhere
> >> below.
> > 
> > I'm sorry, but it's not what I said.
> > I didn't say reiserfs eats less stack. I don't know.
> > I said it is smaller.
> > 
> > reiserfs/*  821474 bytes
> > xfs/*      3019689 bytes
> 
> FWIW, the reason for that is in large part all the features Dave listed
> above, and probably more.
> 
> And, while certainly not yet tiny, the recent trend actually is that xfs
> is getting a bit smaller:
> 
> http://oss.sgi.com/~sandeen/xfs-linedata.png

~30% line count reduction? Impressive, especially in this age
of creeping bloat. Thanks.
--
vda

Re: x86: 4kstacks default

[Andi Kleen]

Adrian Bunk <bunk@kernel.org> writes:
>
> Red Hat seems to get usable kernels with 4k for some years?

One way they do that is by marking significant parts of the kernel
unsupported. I don't think that's an option for mainline.

-Andi

Re: x86: 4kstacks default

[Andi Kleen]

Adrian Bunk <bunk@kernel.org> writes:
>
> 6k is known to work, and there aren't many problems known with 4k.
>
> And from a QA point of view the only way of getting 4k thoroughly tested 

But you have to first ask why do you want 4k tested? Does it serve
any useful purpose in itself? I don't think so. Or you're saying
it's important to support 50k kernel threads on 32bit kernels?

-Andi

Re: x86: 4kstacks default

[Adrian Bunk]

On Sun, Apr 20, 2008 at 02:27:14PM +0200, Andi Kleen wrote:
> Adrian Bunk <bunk@kernel.org> writes:
> >
> > 6k is known to work, and there aren't many problems known with 4k.
> >
> > And from a QA point of view the only way of getting 4k thoroughly tested 
> 
> But you have to first ask why do you want 4k tested? Does it serve
> any useful purpose in itself? I don't think so. Or you're saying
> it's important to support 50k kernel threads on 32bit kernels?

Small embedded systems like the space savings.

> -Andi

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: x86: 4kstacks default

[Willy Tarreau]

On Sun, Apr 20, 2008 at 02:27:14PM +0200, Andi Kleen wrote:
> Adrian Bunk <bunk@kernel.org> writes:
> >
> > 6k is known to work, and there aren't many problems known with 4k.
> >
> > And from a QA point of view the only way of getting 4k thoroughly tested 
> 
> But you have to first ask why do you want 4k tested? Does it serve
> any useful purpose in itself? I don't think so. Or you're saying
> it's important to support 50k kernel threads on 32bit kernels?

Clearly if I have the choice between a kernel which can run 50k threads
and a kernel which does not crash under me during an I/O error, I choose
the later! I don't even imagine what purpose 50k kernel threads may serve.
I certainly can understand that reducing memory footprint is useful, but
if we want wider testing of 4k stacks, considering they may fail in error
path in complex I/O environment, it's not likely during -rc kernels that
we'll detect problems, and if we push them down the throat of users in a
stable release, of course they will thank us very much for crashing their
NFS servers in production during peak hours.

I have nothing against changing the default setting to 4k provided that
it is easy to get back to the save setting (ie changing a config option,
or better, a cmdline parameter). I just don't agree with the idea of
forcing users to swim in the sh*t, it only brings bad reputation to
Linux.

What would really help would be to have 8k stacks with the lower page
causing a fault and print a stack trace upon first access. That way,
the safe setting would still report us useful information without
putting users into trouble.

Willy

Re: x86: 4kstacks default

[Andi Kleen]

Willy Tarreau wrote:

> Clearly if I have the choice between a kernel which can run 50k threads
> and a kernel which does not crash under me during an I/O error, I choose
> the later! I don't even imagine what purpose 50k kernel threads may serve.

I don't know either but it was quoted to me earlier as the primary
reason 4k stacks were introduced.

> I have nothing against changing the default setting to 4k provided that
> it is easy to get back to the save setting

So you're saying that only advanced users who understand all their
CONFIG options should have the safe settings? And everyone else
the "only explodes once a week" mode?

For me that is exactly the wrong way around.

If someone is sure they know what they're doing they can set whatever
crazy settings they want (given there is a quick way to check
for the crazy settings in oops reports so that I can ignore those), but
the default should be always safe and optimized for reliability.

-Andi

Re: x86: 4kstacks default

[Adrian Bunk]

On Sun, Apr 20, 2008 at 03:06:23PM +0200, Andi Kleen wrote:
> Willy Tarreau wrote:
>...
> > I have nothing against changing the default setting to 4k provided that
> > it is easy to get back to the save setting
> 
> So you're saying that only advanced users who understand all their
> CONFIG options should have the safe settings? And everyone else
> the "only explodes once a week" mode?
> 
> For me that is exactly the wrong way around.
> 
> If someone is sure they know what they're doing they can set whatever
> crazy settings they want (given there is a quick way to check
> for the crazy settings in oops reports so that I can ignore those), but
> the default should be always safe and optimized for reliability.

That means we'll have nearly zero testing of the "crazy setting" and 
when someone tries it he'll have a high probability of running into some
problems.

Such a "crazy setting" shouldn't be offered to users at all.

We should either aim at 4k stacks unconditionally for all 32bit 
architectures with 4k page size or don't allow any architecture
to offer 4k stacks.

> -Andi

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: x86: 4kstacks default

[Willy Tarreau]

On Sun, Apr 20, 2008 at 04:30:07PM +0300, Adrian Bunk wrote:
> Such a "crazy setting" shouldn't be offered to users at all.
> 
> We should either aim at 4k stacks unconditionally for all 32bit 
> architectures with 4k page size or don't allow any architecture
> to offer 4k stacks.

I agree you make a valid point here. Then wouldn't it be easier to
simply remove 4k and agree it was a wet dream ?

Willy

Re: x86: 4kstacks default

[Adrian Bunk]

On Sun, Apr 20, 2008 at 03:34:41PM +0200, Willy Tarreau wrote:
> On Sun, Apr 20, 2008 at 04:30:07PM +0300, Adrian Bunk wrote:
> > Such a "crazy setting" shouldn't be offered to users at all.
> > 
> > We should either aim at 4k stacks unconditionally for all 32bit 
> > architectures with 4k page size or don't allow any architecture
> > to offer 4k stacks.
> 
> I agree you make a valid point here. Then wouldn't it be easier to
> simply remove 4k and agree it was a wet dream ?

If the sh maintainer and the m68knommu maintainer (and perhaps 
MontaVista) agree that it was a wet dream.

> Willy

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: x86: 4kstacks default

[Bill Davidsen]

Adrian Bunk wrote:
> On Sun, Apr 20, 2008 at 03:06:23PM +0200, Andi Kleen wrote:
>> Willy Tarreau wrote:
>> ...
>>> I have nothing against changing the default setting to 4k provided that
>>> it is easy to get back to the save setting
>> So you're saying that only advanced users who understand all their
>> CONFIG options should have the safe settings? And everyone else
>> the "only explodes once a week" mode?
>>
>> For me that is exactly the wrong way around.
>>
>> If someone is sure they know what they're doing they can set whatever
>> crazy settings they want (given there is a quick way to check
>> for the crazy settings in oops reports so that I can ignore those), but
>> the default should be always safe and optimized for reliability.
> 
> That means we'll have nearly zero testing of the "crazy setting" and 
> when someone tries it he'll have a high probability of running into some
> problems.
> 
> Such a "crazy setting" shouldn't be offered to users at all.
> 
> We should either aim at 4k stacks unconditionally for all 32bit 
> architectures with 4k page size or don't allow any architecture
> to offer 4k stacks.
> 
I have suggested before that the solution is to allocate memory in 
"stack size" units (obviously must be a multiple of the hardware page 
size). The reason allocation fails is more often fragmentation than 
actual lack of memory, or so it has been reported.

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: x86: 4kstacks default

[Adrian Bunk]

On Sun, Apr 20, 2008 at 02:47:17PM +0200, Willy Tarreau wrote:
>...
> I certainly can understand that reducing memory footprint is useful, but
> if we want wider testing of 4k stacks, considering they may fail in error
> path in complex I/O environment, it's not likely during -rc kernels that
> we'll detect problems, and if we push them down the throat of users in a
> stable release, of course they will thank us very much for crashing their
> NFS servers in production during peak hours.

I've seen many bugs in error paths in the kernel and fixed quite a 
few of them - and stack problems were not a significant part of them.

There are so many possible bugs (that also occur in practice) that 
singling out stack usage won't gain much.

> I have nothing against changing the default setting to 4k provided that
> it is easy to get back to the save setting (ie changing a config option,
> or better, a cmdline parameter). I just don't agree with the idea of
> forcing users to swim in the sh*t, it only brings bad reputation to
> Linux.
>...

What actually brings bad reputation is shipping a 4k option that is 
known to break under some circumstances.

And history has shown that as long as 8k stacks are available on i386 
some problems will not get fixed. 4k stacks are available as an option 
on i386 for more than 4 years, and at about as long we know that there 
are some setups (AFAIK all that might still be present seem to include 
XFS) that are known to not work reliably with 4k stacks.

If we go after stability and reputation, we have to make a decision 
whether we want to get 4k stacks on 32bit architectures with 4k page 
size unconditionally or not at all. That's the way that gets the maximal 
number of bugs shaken out [1] for all supported configurations before 
they would hit a stable kernel.

> Willy

cu
Adrian

[1] obviously not all, but that's true for all classes of bugs

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: x86: 4kstacks default

[Helge Hafting]

Adrian Bunk wrote:
> What actually brings bad reputation is shipping a 4k option that is 
> known to break under some circumstances.
>   
How about making 4k stacks incompatible with those circumstances then?
I.e. is you select 4k stacks, then you can't select XFS because we know
that _may_ fail. Similiar for ndiswrapper networking, and other
stuff where problems have been noticed.

Some people don't need any of these, and can then use
safe 4k stacks. Well, at least as safe as the 8k stacks are, there is no
mathematical proof for their safety in all cases either.

Helge Hafting

Re: x86: 4kstacks default

[David Chinner]

On Wed, Apr 23, 2008 at 11:13:55AM +0200, Helge Hafting wrote:
> Adrian Bunk wrote:
> >What actually brings bad reputation is shipping a 4k option that is 
> >known to break under some circumstances.
> >  
> How about making 4k stacks incompatible with those circumstances then?
> I.e. is you select 4k stacks, then you can't select XFS because we know
> that _may_ fail. Similiar for ndiswrapper networking, and other
> stuff where problems have been noticed.

Yeah, that means every distro that supports XFS (i.e. pretty much
all of them including Fedora) will be forced disable 4k stacks on
x86.  I'd be happy with this solution.

FWIW, this would make 4k stacks pretty much unused outside of custom
kernels.  At which point I'd suggest a default of 4k is wrong....

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: x86: 4kstacks default

[Eric Sandeen]

Helge Hafting wrote:
> Adrian Bunk wrote:
>> What actually brings bad reputation is shipping a 4k option that is 
>> known to break under some circumstances.
>>   
> How about making 4k stacks incompatible with those circumstances then?
> I.e. is you select 4k stacks, then you can't select XFS because we know
> that _may_ fail. Similiar for ndiswrapper networking, and other
> stuff where problems have been noticed.

Problem is, it's the storage configuration (at administration time, not
kernel build time) that matters, too.

I have XFS on Fedora with 4k stacks on SATA /dev/sdb1 on my x86 mythbox,
and it's perfectly fine.  But that's a nice, simple setup.  If I stacked
more things over/under it, I'd be more likely to have trouble.

-Eric

Re: x86: 4kstacks default

[Bill Davidsen]

Adrian Bunk wrote:
> On Sun, Apr 20, 2008 at 02:47:17PM +0200, Willy Tarreau wrote:
>> ...
>> I certainly can understand that reducing memory footprint is useful, but
>> if we want wider testing of 4k stacks, considering they may fail in error
>> path in complex I/O environment, it's not likely during -rc kernels that
>> we'll detect problems, and if we push them down the throat of users in a
>> stable release, of course they will thank us very much for crashing their
>> NFS servers in production during peak hours.
> 
> I've seen many bugs in error paths in the kernel and fixed quite a 
> few of them - and stack problems were not a significant part of them.
> 
> There are so many possible bugs (that also occur in practice) that 
> singling out stack usage won't gain much.
> 
>> I have nothing against changing the default setting to 4k provided that
>> it is easy to get back to the save setting (ie changing a config option,
>> or better, a cmdline parameter). I just don't agree with the idea of
>> forcing users to swim in the sh*t, it only brings bad reputation to
>> Linux.
>> ...
> 
> What actually brings bad reputation is shipping a 4k option that is 
> known to break under some circumstances.
> 
> And history has shown that as long as 8k stacks are available on i386 
> some problems will not get fixed. 4k stacks are available as an option 
> on i386 for more than 4 years, and at about as long we know that there 
> are some setups (AFAIK all that might still be present seem to include 
> XFS) that are known to not work reliably with 4k stacks.
> 
> If we go after stability and reputation, we have to make a decision 
> whether we want to get 4k stacks on 32bit architectures with 4k page 
> size unconditionally or not at all. That's the way that gets the maximal 
> number of bugs shaken out [1] for all supported configurations before 
> they would hit a stable kernel.
> 
A good argument for keeping the default 8k and letting people who know 
what they are doing, or think they do, test their system for 4k 
operation. Embedded systems typically have far better defined loads than 
servers or desktops, and are less likely to have different behavior 
change the stack requirements. That doesn't mean they do less, just that 
the load is usually better characterized.

Vendors shipping a 4k stack kernel are probably not going to be happy if 
someone nfs exports an xfs filesystem on lvm, running on md raid0 
composed of raid5 arrays, containing multipath, iSCSI, SATA and nbd 
devices. No, I didn't make that up, someone asked me what I thought 
their problem was with that setup.

The kernel is getting more complex, and I don't think that anyone but 
you is interested in making 4k stacks mandatory, or in eliminating them, 
either.

You frequently take the attitude that something you don't like (like all 
the old but WORKING network drivers) should be removed from the kernel, 
so that people will be forced to use the new whatever and find bugs so 
they can be fixed. Unfortunately in some cases the bugs are never fixed 
and Linux loses a capability it once had.

The arbitrary 4k limit requires a lot of work on dropping stack usage 
even more than has already been done, and is mostly an effort you want 
other people to make so you can be happy (I assume that if you were 
offering to do it all yourself you already would have), and most 
importantly it would waste a lot of developer effort on a low return 
goal, which could be used on useful new features or fixing corner case 
bugs. Or drinking beer...

Hell, it wastes your time arguing about it, and you do lots of useful 
things when you're not trying to force your minimalist philosophy on people.

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: x86: 4kstacks default

[Mark Lord]

Willy Tarreau wrote:
>
> What would really help would be to have 8k stacks with the lower page
> causing a fault and print a stack trace upon first access. That way,
> the safe setting would still report us useful information without
> putting users into trouble.
..

That's the best suggestion from this thread, by far!
Can you produce a patch for 2.6.26 for this?
Or perhaps someone else here, with the right code familiarity, could?

Some sort of CONFIG option would likely be wanted to
either enable/disable this feature, of course.

Cheers

Re: x86: 4kstacks default

[Willy Tarreau]

On Sun, Apr 20, 2008 at 09:27:32AM -0400, Mark Lord wrote:
> Willy Tarreau wrote:
> >
> >What would really help would be to have 8k stacks with the lower page
> >causing a fault and print a stack trace upon first access. That way,
> >the safe setting would still report us useful information without
> >putting users into trouble.
> ..
> 
> That's the best suggestion from this thread, by far!
> Can you produce a patch for 2.6.26 for this?

Unfortunately, I can't. I wouldn't know where to start from.

> Or perhaps someone else here, with the right code familiarity, could?

I hope so.

> Some sort of CONFIG option would likely be wanted to
> either enable/disable this feature, of course.

If we want to migrate to 4k sooner or later, this behaviour would not
need a config option, maybe just a /proc or /sys tunable to disable
the warning. Config would be either (4k + risk of crash) or (8k + warning).

The *real* issue is to decide whether we need/want 4k or not, because
I think we're still discussing the subject for no reason, as usual...

Willy

Re: x86: 4kstacks default

[Andi Kleen]

Willy Tarreau wrote:
> On Sun, Apr 20, 2008 at 09:27:32AM -0400, Mark Lord wrote:
>> Willy Tarreau wrote:
>>> What would really help would be to have 8k stacks with the lower page
>>> causing a fault and print a stack trace upon first access. That way,
>>> the safe setting would still report us useful information without
>>> putting users into trouble.
>> ..
>>
>> That's the best suggestion from this thread, by far!

Only if you believe that 4K stack pages are a worthy goal.
As far as I can figure out they are not. They might have been
a worthy goal on crappy 2.4 VMs, but these times are long gone.

The "saving memory on embedded" argument also does not 
quite convince me, it is unclear if that is really
a significant amount of memory on these systems and if that 
couldn't be addressed better (e.g. in running generally
less kernel threads).  I don't have numbers on this,
but then the people who made this argument didn't have any
either :) 

If anybody has concrete statistics on this
(including other kernel memory users in realistic situations)
please feel free to post them.


>> Can you produce a patch for 2.6.26 for this?
> 
> Unfortunately, I can't. I wouldn't know where to start from.

The problem with his suggestion is that the lower 4K of the stack page 
are accessed in normal operation too because it contains the thread_struct.
That could be changed, but it would be a relatively large change
because you would need to audit/change a lot of code who assumes 
thread_struct and stack are continuous

If that was changed implementing Willy's suggestion would not be that 
difficult using cpa()  at the cost of some general slowdown in 
increased TLB misses and much higher thread creation/tear down cost etc, 
Using the alternative vmalloc way has also other issues.

But still the fundamental problem is that it would likely only
hit the interesting cases in real production setups and I don't 
think the production users would be very happy to slow down
their kernels and handle strange backtraces just to act as guinea pigs 
for something dubious

-Andi

Re: x86: 4kstacks default

[Jörn Engel]

On Sun, 20 April 2008 16:19:29 +0200, Andi Kleen wrote:
> 
> Only if you believe that 4K stack pages are a worthy goal.
> As far as I can figure out they are not. They might have been
> a worthy goal on crappy 2.4 VMs, but these times are long gone.
> 
> The "saving memory on embedded" argument also does not 
> quite convince me, it is unclear if that is really
> a significant amount of memory on these systems and if that 
> couldn't be addressed better (e.g. in running generally
> less kernel threads).  I don't have numbers on this,
> but then the people who made this argument didn't have any
> either :) 

It is not uncommon for embedded systems to be designed around 16MiB.
Some may even have less, although I haven't encountered any of those
lately.

When dealing in those dimensions, savings of 100k are substantial.  In
some causes they may be the difference between 16MiB or 32MiB, which
translates to manufacturing costs.  In others it simply means that the
system can cache a bit more and run faster, or it can have a little more
functionality.

In most cases it simply allows userspace programmers to avoid looking
harder to save those 100k, as they are already saved in kernel space.
Therefore we made life hard for us in order to make life easier for
someone else, saving them time and money.

Whether that is worth it depends on your personal point of view.  Many
embedded people will claim "Hell yes!"  Of those that don't, most are
simply ignoring currently mainline kernels and will regret the
development later.  They care, thay just don't tend to care enough to
engage in these discussions or even know about them. :(

Jörn

-- 
Eighty percent of success is showing up.
-- Woody Allen

Re: x86: 4kstacks default

[Andi Kleen]

Jörn Engel wrote:
> On Sun, 20 April 2008 16:19:29 +0200, Andi Kleen wrote:
>> Only if you believe that 4K stack pages are a worthy goal.
>> As far as I can figure out they are not. They might have been
>> a worthy goal on crappy 2.4 VMs, but these times are long gone.
>>
>> The "saving memory on embedded" argument also does not 
>> quite convince me, it is unclear if that is really
>> a significant amount of memory on these systems and if that 
>> couldn't be addressed better (e.g. in running generally
>> less kernel threads).  I don't have numbers on this,
>> but then the people who made this argument didn't have any
>> either :) 
> 
> It is not uncommon for embedded systems to be designed around 16MiB.

But these are SoC systems. Do they really run x86?
(note we're talking about an x86 default option here)

Also I suspect in a true 16MB system you have to strip down
everything kernel side so much that you're pretty much outside
the "validated by testers" realm that Adrian cares about.

> When dealing in those dimensions, savings of 100k are substantial.  In
> some causes they may be the difference between 16MiB or 32MiB, which
> translates to manufacturing costs.  In others it simply means that the
> system can cache 

If you need the stack you don't have any less cache foot print.
If you don't need it you don't have any either.

-Andi

Re: x86: 4kstacks default

[Jörn Engel]

On Sun, 20 April 2008 19:19:26 +0200, Andi Kleen wrote:
> 
> But these are SoC systems. Do they really run x86?
> (note we're talking about an x86 default option here)
> 
> Also I suspect in a true 16MB system you have to strip down
> everything kernel side so much that you're pretty much outside
> the "validated by testers" realm that Adrian cares about.

Maybe.  I merely showed that embedded people (not me) have good reasons
to care about small stacks.  Whether they care enough to actually spend
work on it - doubtful.

> > When dealing in those dimensions, savings of 100k are substantial.  In
> > some causes they may be the difference between 16MiB or 32MiB, which
> > translates to manufacturing costs.  In others it simply means that the
> > system can cache 
> 
> If you need the stack you don't have any less cache foot print.
> If you don't need it you don't have any either.

This part I don't understand.

Jörn

-- 
You ain't got no problem, Jules. I'm on the motherfucker. Go back in
there, chill them niggers out and wait for the Wolf, who should be
coming directly.
-- Marsellus Wallace

Re: x86: 4kstacks default

[Andi Kleen]

Jörn Engel wrote:
> On Sun, 20 April 2008 19:19:26 +0200, Andi Kleen wrote:
>> But these are SoC systems. Do they really run x86?
>> (note we're talking about an x86 default option here)
>>
>> Also I suspect in a true 16MB system you have to strip down
>> everything kernel side so much that you're pretty much outside
>> the "validated by testers" realm that Adrian cares about.
> 
> Maybe.  I merely showed that embedded people (not me) have good reasons
> to care about small stacks. 

Sure but I don't think they're x86 embedded people. Right now there
are very little x86 SOCs if any (iirc there is only some obscure rise
core) and future SOCs will likely have more RAM.

Anyways I don't have a problem to give these people any special options
they need to do whatever they want.  I just object to changing the
default options on important architectures to force people in completely
different setups to do part of their testing.


 Whether they care enough to actually spend
> work on it - doubtful.
> 
>>> When dealing in those dimensions, savings of 100k are substantial.  In
>>> some causes they may be the difference between 16MiB or 32MiB, which
>>> translates to manufacturing costs.  In others it simply means that the
>>> system can cache 
>> If you need the stack you don't have any less cache foot print.
>> If you don't need it you don't have any either.
> 
> This part I don't understand.

I was just objecting to your claim that small stack implies smaller
cache foot print. Smaller stacks rarely give you smaller cache foot
print in my kernel coding experience:

First some stack is always safety and in practice unused. It won't
be in cache.

Then typically standard kernel stack pigs are just too large
buffers on the stack which are not fully used. These also
don't have much cache foot print.

Or if you have a complicated call stack the typical fix
is to move parts of it into another thread. But that doesn't
give you less cache footprint because the cache foot print
is just in someone else's stack. In fact you'll likely
have slightly more cache foot print from that due to the
context of the other thread.

In theory if you e.g. convert a recursive algorithm
to iterative you might save some cache foot print, but I don't
think that really happens in kernel code.

-Andi

Re: x86: 4kstacks default

[Arjan van de Ven]

On Sun, 20 Apr 2008 20:19:30 +0200
Andi Kleen <andi@firstfloor.org> wrote:

> In theory if you e.g. convert a recursive algorithm
> to iterative you might save some cache foot print, but I don't
> think that really happens in kernel code.
> 

this is what Al did for the symlink recursion thing, and Jens did for the block layer...
so yes this conversion does happen for real.

-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: x86: 4kstacks default

[Andi Kleen]

Arjan van de Ven wrote:

> this is what Al did for the symlink recursion thing, 

AFAIK most symlink lookups are still recursive.

-Andi

Re: x86: 4kstacks default

[Andrew Morton]

> On Sun, 20 Apr 2008 11:50:53 -0700 Arjan van de Ven <arjan@infradead.org> wrote:
> On Sun, 20 Apr 2008 20:19:30 +0200
> Andi Kleen <andi@firstfloor.org> wrote:
> 
> > In theory if you e.g. convert a recursive algorithm
> > to iterative you might save some cache foot print, but I don't
> > think that really happens in kernel code.
> > 
> 
> this is what Al did for the symlink recursion thing, and Jens did for the block layer...
> so yes this conversion does happen for real.

md got mostly-fixed too, via Neil's patch which sat in -mm for nearly two
years.

Re: x86: 4kstacks default

[Andi Kleen]

Andrew Morton wrote:
>> On Sun, 20 Apr 2008 11:50:53 -0700 Arjan van de Ven <arjan@infradead.org> wrote:
>> On Sun, 20 Apr 2008 20:19:30 +0200
>> Andi Kleen <andi@firstfloor.org> wrote:
>>
>>> In theory if you e.g. convert a recursive algorithm
>>> to iterative you might save some cache foot print, but I don't
>>> think that really happens in kernel code.
>>>
>> this is what Al did for the symlink recursion thing, and Jens did for the block layer...
>> so yes this conversion does happen for real.
> 
> md got mostly-fixed too, via Neil's patch which sat in -mm for nearly two
> years.

Congratulations, you found three examples in 8.4MLOC.

Ok ok I should have said it only happens very rarely (I still stand by
that :)

Anyways it is moot because it was a miscommunication between me and Joerg.

-Andi

Re: x86: 4kstacks default

[Ingo Molnar]

* Andrew Morton <akpm@linux-foundation.org> wrote:

> > On Sun, 20 Apr 2008 11:50:53 -0700 Arjan van de Ven <arjan@infradead.org> wrote:
> > On Sun, 20 Apr 2008 20:19:30 +0200
> > Andi Kleen <andi@firstfloor.org> wrote:
> > 
> > > In theory if you e.g. convert a recursive algorithm
> > > to iterative you might save some cache foot print, but I don't
> > > think that really happens in kernel code.
> > > 
> > 
> > this is what Al did for the symlink recursion thing, and Jens did 
> > for the block layer... so yes this conversion does happen for real.
> 
> md got mostly-fixed too, via Neil's patch which sat in -mm for nearly 
> two years.

had we done the de-obfuscate-4K-stacks Kconfig change earlier it might 
have gotten upstream faster.

	Ingo

Re: x86: 4kstacks default

[Jörn Engel]

On Sun, 20 April 2008 20:19:30 +0200, Andi Kleen wrote:
> > 
> >>> When dealing in those dimensions, savings of 100k are substantial.  In
> >>> some causes they may be the difference between 16MiB or 32MiB, which
> >>> translates to manufacturing costs.  In others it simply means that the
> >>> system can cache 
> >> If you need the stack you don't have any less cache foot print.
> >> If you don't need it you don't have any either.
> > 
> > This part I don't understand.
> 
> I was just objecting to your claim that small stack implies smaller
> cache foot print.

The cache I referred to is called DRAM, not L1.

Jörn

-- 
Don't worry about people stealing your ideas. If your ideas are any good,
you'll have to ram them down people's throats.
-- Howard Aiken quoted by Ken Iverson quoted by Jim Horning quoted by
   Raph Levien, 1979

Re: x86: 4kstacks default

[Jörn Engel]

On Sun, 20 April 2008 20:19:30 +0200, Andi Kleen wrote:
> 
> Sure but I don't think they're x86 embedded people. Right now there
> are very little x86 SOCs if any (iirc there is only some obscure rise
> core) and future SOCs will likely have more RAM.
> 
> Anyways I don't have a problem to give these people any special options
> they need to do whatever they want.  I just object to changing the
> default options on important architectures to force people in completely
> different setups to do part of their testing.

Ah, ok.  The question whether 4k stacks should become the default I
prefer not touching with an 80' pole.

Jörn

-- 
Why do musicians compose symphonies and poets write poems?
They do it because life wouldn't have any meaning for them if they didn't.
That's why I draw cartoons.  It's my life.
-- Charles Shultz

Re: x86: 4kstacks default

[Eric Sandeen]

Mark Lord wrote:
> Willy Tarreau wrote:
>> What would really help would be to have 8k stacks with the lower page
>> causing a fault and print a stack trace upon first access. That way,
>> the safe setting would still report us useful information without
>> putting users into trouble.
> ..
> 
> That's the best suggestion from this thread, by far!
> Can you produce a patch for 2.6.26 for this?
> Or perhaps someone else here, with the right code familiarity, could?
> 
> Some sort of CONFIG option would likely be wanted to
> either enable/disable this feature, of course.

Changing the default warning threshold is easy, it's just a #define.
Although setting it too low would spam syslogs on some setups.

When I was trying to cram stuff into 4k in the past, I had a patch which
added a sysctl to dynamically change the warning threshold, and
optionally BUG() when I hit it for crash analysis.  It was good for
debugging, at least.  If something along those lines is desired, I could
resurrect it.

-Eric

Re: x86: 4kstacks default

[Willy Tarreau]

On Sun, Apr 20, 2008 at 09:09:37AM -0500, Eric Sandeen wrote:
> Mark Lord wrote:
> > Willy Tarreau wrote:
> >> What would really help would be to have 8k stacks with the lower page
> >> causing a fault and print a stack trace upon first access. That way,
> >> the safe setting would still report us useful information without
> >> putting users into trouble.
> > ..
> > 
> > That's the best suggestion from this thread, by far!
> > Can you produce a patch for 2.6.26 for this?
> > Or perhaps someone else here, with the right code familiarity, could?
> > 
> > Some sort of CONFIG option would likely be wanted to
> > either enable/disable this feature, of course.
> 
> Changing the default warning threshold is easy, it's just a #define.

I thought it was checked only at a few places (eg: during irqs). If so,
maybe it can miss some call chains ?

> Although setting it too low would spam syslogs on some setups.

we should set it slightly below the 4k limit if we want users to switch
to 4k.

> When I was trying to cram stuff into 4k in the past, I had a patch which
> added a sysctl to dynamically change the warning threshold, and
> optionally BUG() when I hit it for crash analysis.  It was good for
> debugging, at least.  If something along those lines is desired, I could
> resurrect it.

While it's good for debugging, having users tweak the limit to eliminate
the warning is the opposite of what we're looking for. We just want to
have them report the warning without their service being disrupted.

Willy

Re: x86: 4kstacks default

[Eric Sandeen]

Willy Tarreau wrote:
> On Sun, Apr 20, 2008 at 09:09:37AM -0500, Eric Sandeen wrote:
>> Mark Lord wrote:
>>> Willy Tarreau wrote:
>>>> What would really help would be to have 8k stacks with the lower page
>>>> causing a fault and print a stack trace upon first access. That way,
>>>> the safe setting would still report us useful information without
>>>> putting users into trouble.
>>> ..
>>>
>>> That's the best suggestion from this thread, by far!
>>> Can you produce a patch for 2.6.26 for this?
>>> Or perhaps someone else here, with the right code familiarity, could?
>>>
>>> Some sort of CONFIG option would likely be wanted to
>>> either enable/disable this feature, of course.
>> Changing the default warning threshold is easy, it's just a #define.
> 
> I thought it was checked only at a few places (eg: during irqs). If so,
> maybe it can miss some call chains ?

Ah, ok I skimmed your first suggestion too quickly.  100% coverage
reports on the initial access to the 2nd 4k that way would be nice.
Well, it would be nice if we all really wanted 4k stacks some day... :)

-Eric

Re: x86: 4kstacks default

[Daniel Hazelton]

On Sunday 20 April 2008 08:27:14 Andi Kleen wrote:
> Adrian Bunk <bunk@kernel.org> writes:
> > 6k is known to work, and there aren't many problems known with 4k.
> >
> > And from a QA point of view the only way of getting 4k thoroughly tested
>
> But you have to first ask why do you want 4k tested? Does it serve
> any useful purpose in itself? I don't think so. Or you're saying
> it's important to support 50k kernel threads on 32bit kernels?
>
> -Andi

Andi, you're the only one I've seen seriously pounding the "50k threads" 
thing - I don't think anyone is really fooled by the straw-man, so I'd 
suggest you drop it.

The real issue is that you think (and are correct in thinking) that people are 
idiots. Yes, there will be breakages if the default is changed to 4k stacks - 
but if people are running new kernels on boxes that'll hit stack use problems 
(that *AREN'T* related to ndiswrapper) and haven't made sure that they've 
configured the kernel properly, then they deserve the outcome. It isn't the 
job of the Linux Kernel to protect the incompetent - nor is it the job of 
linux kernel developers to do such.

If people are doing a "zcat /proc/kconfig.gz > .config && make oldconfig" (or 
similar) the problem shouldn't even appear, really. They'll get whatever 
setting was in their old config for the stack size. And until the problems 
with deep-stack setups - like nfs+xfs+raid - get resolved I'd think that the 
option to configure the stack size would remain.

Since the second-most-common reason for stack overages is ndiswrapper... Well, 
with there being so much more hardware now supported directly by the linux 
kernel... I'm stunned every time someone tells me "I can't run Linux on my 
laptop, there is hardware that isn't supported without me having to get 
ndiswrapper". The last time someone said that to me I pointed to the fact 
that their hardware is supported by the latest kernel and even offered to 
build&install it for them.

DRH

-- 
Dialup is like pissing through a pipette. Slow and excruciatingly painful.

Re: x86: 4kstacks default

[Andi Kleen]

Daniel Hazelton wrote:

> Andi, you're the only one I've seen seriously pounding the "50k threads" 
> thing. I don't think anyone is really fooled by the straw-man, so I'd
> suggest you drop it.

Ok, perhaps we can settle this properly. Like historicans. We study the
original sources.

The primary resource is the original commit adding the 4k stack code.
You cannot find this in latest git because it predates 2.6.12, but it is
available in one of the historic trees imported from BitKeeper like
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git

Here's the log:
>>
commit 95f238eac82907c4ccbc301cd5788e67db0715ce
Author: Andrew Morton <akpm@osdl.org>
Date:   Sun Apr 11 23:18:43 2004 -0700

    [PATCH] ia32: 4Kb stacks (and irqstacks) patch

    From: Arjan van de Ven <arjanv@redhat.com>

    Below is a patch to enable 4Kb stacks for x86. The goal of this is to

    1) Reduce footprint per thread so that systems can run many more threads
       (for the java people)

    2) Reduce the pressure on the VM for order > 0 allocations. We see
real life
       workloads (granted with 2.4 but the fundamental fragmentation
issue isn't
       solved in 2.6 and isn't solvable in theory) where this can be a
problem.
       In addition order > 0 allocations can make the VM "stutter" and
give more
       latency due to having to do much much more work trying to defragment

...
<<

This gives us two reasons as you can see, one of them many threads
and another mostly only relevant to 2.4

Now I was also assuming that nobody took (1) really serious and
attacked (2) in earlier thread; in particular in

http://article.gmane.org/gmane.linux.kernel/665584

>>
Actually the real reason the 4K stacks were introduced IIRC was that
the VM is not very good at allocation of order > 0 pages and that only
using order 0 and not order 1 in normal operation prevented some stalls.

This rationale also goes back to 2.4 (especially some of the early 2.4
VMs were not very good) and the 2.6 VM is generally better and on
x86-64 I don't see much evidence that these stalls are a big problem
(but then x86-64 also has more lowmem).
<<

This was corrected by Ingo who was one of the primary authors of the patch:

http://thread.gmane.org/gmane.linux.kernel/665420:

>>
no, the primary motivation Arjan and me started working on 4K stacks and
implemented it was what Denys mentioned: i had a testcase that ran
50,000 threads before it ran out of memory - i wanted it to run 100,000
threads. The improved order-0 behavior was just icing on the cake.

	Ingo
<<

and then from Arjan:

http://thread.gmane.org/gmane.linux.kernel/665420

>>
> no, the primary motivation Arjan and me started working on 4K stacks
> and implemented it was what Denys mentioned: i had a testcase that

well that and the fact that RH had customers who had major issues at
fewer threads
with 8Kb versus fragmentation.
<<

So both the primary authors of the patch state that 50k threads
was the main reason. I didn't believe it at first either, but after
these forceful corrections I do now.

You're totally wrong when you call it a straw man.

-Andi

Re: x86: 4kstacks default

[Arjan van de Ven]

On Sun, 20 Apr 2008 19:26:10 +0200
Andi Kleen <andi@firstfloor.org> wrote:

> Daniel Hazelton wrote:
> 
> > Andi, you're the only one I've seen seriously pounding the "50k
> > threads" thing. I don't think anyone is really fooled by the
> > straw-man, so I'd suggest you drop it.
> 
> Ok, perhaps we can settle this properly. Like historicans. We study
> the original sources.
> 
> The primary resource is the original commit adding the 4k stack code.
> You cannot find this in latest git because it predates 2.6.12, but it
> is available in one of the historic trees imported from BitKeeper like
> git://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git
> 
> Here's the log:
> >>
> commit 95f238eac82907c4ccbc301cd5788e67db0715ce
> Author: Andrew Morton <akpm@osdl.org>
> Date:   Sun Apr 11 23:18:43 2004 -0700
> 
>     [PATCH] ia32: 4Kb stacks (and irqstacks) patch
> 
>     From: Arjan van de Ven <arjanv@redhat.com>
> 
>     Below is a patch to enable 4Kb stacks for x86. The goal of this
> is to
> 
>     1) Reduce footprint per thread so that systems can run many more
> threads (for the java people)
> 
>     2) Reduce the pressure on the VM for order > 0 allocations. We see
> real life
>        workloads (granted with 2.4 but the fundamental fragmentation
> issue isn't
>        solved in 2.6 and isn't solvable in theory) where this can be a
> problem.
>        In addition order > 0 allocations can make the VM "stutter" and
> give more
>        latency due to having to do much much more work trying to
> defragment
> 
> ...
> <<
> 
> This gives us two reasons as you can see, one of them many threads
> and another mostly only relevant to 2.4
> 
> Now I was also assuming that nobody took (1) really serious and

I'm sorry but I really hope nobody shares your assumption here.
These are real customer workloads; java based "many things going on" at a time
showed several thousands of threads fin the system (a dozen or two per request, multiplied
by the number of outstanding connections) for *real customers*.
That you don't take that serious, fair, you can take serious whatever you want.


> attacked (2) in earlier thread; in particular in

yes you did attack. But lets please use more friendly conversation here than words like
"attack". This is not a war, and we really shouldn't be hostile in this forum, neither
in words nor in intention.

> 
> http://article.gmane.org/gmane.linux.kernel/665584
> 
> >>
> Actually the real reason the 4K stacks were introduced IIRC was that
> the VM is not very good at allocation of order > 0 pages and that only
> using order 0 and not order 1 in normal operation prevented some
> stalls.
> 
> This rationale also goes back to 2.4 (especially some of the early 2.4
> VMs were not very good) and the 2.6 VM is generally better and on
> x86-64 I don't see much evidence that these stalls are a big problem
> (but then x86-64 also has more lowmem).
> <<

What you didn't atta^Waddress was the observation that fragmentation is fundamentally unsolvable.
Yes 2.4 sucked a lot more than 2.6 does. But even 2.6 will (and does) have fragmentation issues.
We don't have effective physical address based reclaim yet for higher order allocs.

> 
> http://thread.gmane.org/gmane.linux.kernel/665420:
> 
> >>
> no, the primary motivation Arjan and me started working on 4K stacks
> and implemented it was what Denys mentioned: i had a testcase that ran
> 50,000 threads before it ran out of memory - i wanted it to run
> 100,000 threads. The improved order-0 behavior was just icing on the
> cake.
> 
> 	Ingo
> <<
> 
> and then from Arjan:
> 
> http://thread.gmane.org/gmane.linux.kernel/665420
> 
> >>
> > no, the primary motivation Arjan and me started working on 4K stacks
> > and implemented it was what Denys mentioned: i had a testcase that
> 
> well that and the fact that RH had customers who had major issues at
> fewer threads
> with 8Kb versus fragmentation.
> <<
> 
> So both the primary authors of the patch state that 50k threads
> was the main reason. I didn't believe it at first either, but after
> these forceful corrections I do now.

I'm sorry but I fail to entirely understand where your "So" or the rest of your 
conclusion comes from in terms of "both the authors". Which part of "fewer threads" and
"8kb versus fragmentation" did you misunderstand to get to your conclusion?

-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: x86: 4kstacks default

[Andi Kleen]

> These are real customer workloads; java based "many things going on" at a time
> showed several thousands of threads fin the system (a dozen or two per request, multiplied
> by the number of outstanding connections) for *real customers*.

Several thousands or 50k? Several thousands sounds large, but not entirely unreasonable, 
but it is far from 50k.

> That you don't take that serious, fair, you can take serious whatever you want.

No I don't take 50k threads on 32bit serious. And I hope you do not
either.

Why I don't take it serious: on 32bit 50k threads will lead
to lowmem exhaustion if the threads are actually doing something 
(like keeping select pages around or similar and having some thread
local data). You'll easily be at 16-32K/thread and that is already 
far beyond the lowmem available on any 3:1 split 32bit kernel, likely 
even beyond 2:2. Even with 3:1 it could be tight.

So you can say about customer workloads what you want, but you'll
have a hard time convincing me they really run 50k threads 
doing something on 32bit.

Now if we take the real realistic overhead of a thread into 
account 4k or more less don't really matter all that much
and the decreased safety from the 4k stack starts to look
like a very bad bargain.

>> attacked (2) in earlier thread; in particular in
> 
> yes you did attack. 
> But lets please use more friendly conversation here than words like
> "attack". This is not a war, and we really shouldn't be hostile in this forum, neither
> in words nor in intention.

Ok what word would you prefer? 

There is no war involved right, just a technical argument. I previously 
always assumed that "attacking" was a standard term in discussions, but 
if you don't like I can switch to another one. 

Regarding war like terminology: I used to think that people who commonly 
talk about "nuking code" went a little too far, but at some point
I adapted to them I think. Perhaps it comes from that.


> What you didn't atta^Waddress 

Fine, I will call it address from now.

> was the observation that fragmentation is fundamentally unsolvable.

Where was that observation? 

> Yes 2.4 sucked a lot more than 2.6 does. But even 2.6 will (and does) have fragmentation issues.
> We don't have effective physical address based reclaim yet for higher order allocs.

I don't see any evidence that there are serious order 1 fragmentation 
issues on 2.6. If you have any please post it.

-Andi

Re: x86: 4kstacks default

[Daniel Hazelton]

On Sunday 20 April 2008 16:01:46 Andi Kleen wrote:
> > These are real customer workloads; java based "many things going on" at a
> > time showed several thousands of threads fin the system (a dozen or two
> > per request, multiplied by the number of outstanding connections) for
> > *real customers*.
>
> Several thousands or 50k? Several thousands sounds large, but not entirely
> unreasonable, but it is far from 50k.
>

At 12 threads per request it'd only take about 4200 outstanding requests. That 
is high, but I can see it happening. At 24 threads per request the number of 
outstanding requests it takes to reach that is cut in half, to about 2100. 
That number is more realistic. Since all outstanding requests aren't going to 
be at the extremes, let us assume that it's a mid-point between the two for 
the number of outstanding requests - say somewhere around 3150 outstanding 
requests.

While that is a rather high number, if a company - a decently sized one - is 
using a piece of Java code internally for some reason they could easily have 
that level of requests coming in from the users. For a website with a decent 
load that routes a common request to the machine running the code it'd be 
even easier to hit that limit. So yes, 50K threads *IS* actually pretty easy 
to reach and could be a common workload.

> > That you don't take that serious, fair, you can take serious whatever you
> > want.
>
> No I don't take 50k threads on 32bit serious. And I hope you do not
> either.

Just makes you sound foolish. Run the numbers yourself and you'll see that it 
is easy for a machine running highly threaded code to easily hit 50K threads.

<snip>
> > Yes 2.4 sucked a lot more than 2.6 does. But even 2.6 will (and does)
> > have fragmentation issues. We don't have effective physical address based
> > reclaim yet for higher order allocs.
>
> I don't see any evidence that there are serious order 1 fragmentation
> issues on 2.6. If you have any please post it.

Due to me screwing up the configuration of Apache (2) and MySQL I have seen a 
machine I own hit problems with memory fragmentation - and it's running a 2.6 
series kernel (a distro 2.6.17)

Because I was able to see that it was a problem I caused I didn't even *THINK* 
about posting information about it to LKML. I didn't keep the logs of that 
around - it happened more than three months ago and I clean the logs out 
every three months or so.

DRH

-- 
Dialup is like pissing through a pipette. Slow and excruciatingly painful.

Re: x86: 4kstacks default

[Andi Kleen]

Daniel Hazelton wrote:

> At 12 threads per request it'd only take about 4200 outstanding requests. That 
> is high, but I can see it happening.

If it happens it just won't work on 32bit.

> Just makes you sound foolish. Run the numbers yourself and you'll see that it 
> is easy for a machine running highly threaded code to easily hit 50K threads.

I ran the numbers and the numbers showed that you need > 1.5GB of lowmem
with a somewhat realistic scenario (32K per thread) at 50k threads. And
subtracting 4k from that 32k number won't make any significant
difference (still 1.3GB)

If you claim that works on a 32bit system with typically 300-600MB
lowmem available (which is also shared by other subsystem) I know who
sounds foolish.

-Andi

Re: x86: 4kstacks default

[Bernd Eckenfels]

In article <480BB85A.6020508@firstfloor.org> you wrote:
> If you claim that works on a 32bit system with typically 300-600MB
> lowmem available (which is also shared by other subsystem) I know who
> sounds foolish.

A question along this line. Why is the Userspace Thread bound to a
Kernel-Space Stack at all? I could imagine a solution like Stack Pools
assigned only of a Thread enters kernel space, or something like this?

Gruss
Bernd

Re: x86: 4kstacks default

[Avi Kivity]

Bernd Eckenfels wrote:
> A question along this line. Why is the Userspace Thread bound to a
> Kernel-Space Stack at all? I could imagine a solution like Stack Pools
> assigned only of a Thread enters kernel space, or something like this?
>
>   

The vast majority of threads are sleeping (with a stack footprint in the 
kernel).  If you have an N-way system, at most N threads can be in 
userspace at any given moment.

You could multiplex several userspace threads on one kernel thread (the 
M:N model), but it gets fairly complex.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: x86: 4kstacks default

[Daniel Hazelton]

On Sunday 20 April 2008 17:40:42 Andi Kleen wrote:
> Daniel Hazelton wrote:
> > At 12 threads per request it'd only take about 4200 outstanding requests.
> > That is high, but I can see it happening.
>
> If it happens it just won't work on 32bit.

No, it won't. Which is what I was pointing out. You're hitting a different 
wall there.

> > Just makes you sound foolish. Run the numbers yourself and you'll see
> > that it is easy for a machine running highly threaded code to easily hit
> > 50K threads.
>
> I ran the numbers and the numbers showed that you need > 1.5GB of lowmem
> with a somewhat realistic scenario (32K per thread) at 50k threads. And
> subtracting 4k from that 32k number won't make any significant
> difference (still 1.3GB)
>
> If you claim that works on a 32bit system with typically 300-600MB
> lowmem available (which is also shared by other subsystem) I know who
> sounds foolish.

Never said it worked on a 32bit system. I was pointing out that there can be 
workloads that do reach that 50K thread-count that you seem to be 
calling "stupid". 

As I pointed out later in the message, I *HAVE* run into lowmem starvation on 
a 32bit x86 system. You thoughtfully removed this, perhaps because you felt 
it damaged your argument. The machine in question is an old P3 box with less 
than 1G of memory in it. (Phys+Swap on that machine is only about 1.4G)

So yes, on a 32bit machine you run into problems at much, much less of a 
workload and a much lower thread-count than the magic 50K you are so fond of 
talking about. If I had been running 4K stacks on that machine I probably 
would have survived the mis-configuration without the reboot it took to make 
the machine functional again - I probably would still have reconfigured 
Apache and MySQL, though - the machine still would have gone largely 
unresponsive.

DRH

--
Dialup is like pissing through a pipette. Slow and excruciatingly painful.

Re: x86: 4kstacks default

[Andi Kleen]

> Never said it worked on a 32bit system. I was pointing out that there can be 
> workloads that do reach 

Ah your point was that people might do this on 64bit systems? 

They could indeed. It would not be very efficient but it should work
in theory at least with enough memory. Of course they don't need 4k
stacks for it. They can also try it on 32bit and it will work
to some extent too, just not scale very far.  And 4k stack more or less
won't make much difference for that because the stack is only
a small part of the lowmem needed for a blocked thread with
open sockets.

But this thread clearly was about 32bit systems only.

> that 50K thread-count that you seem to be 
> calling "stupid". 

Note I didn't come up with that number, it was quoted to me earlier
(but one of its authors has distanced itself from it now, so it 
seems to becoming more and more irrelevant indeed now) 

Stupid in this case just refers to the general observation that
it is quite inefficient to do one thread per request on servers
who are expected to process lots of long running connections.

Perhaps I could have put that better I will give you that. Please
assume I always meant "inefficient" when I wrote "stupid".

> talking about. If I had been running 4K stacks on that machine I probably 
> would have survived the mis-configuration without the reboot it took to make 

Now that is a very doubtful claim. You realize that a functional network
server thread needs a lot more lowmem than just the stack? 

-Andi

Re: x86: 4kstacks default

[Daniel Hazelton]

On Monday 21 April 2008 03:51:02 Andi Kleen wrote:
> > Never said it worked on a 32bit system. I was pointing out that there can
> > be workloads that do reach
>
> Ah your point was that people might do this on 64bit systems?

My point was that people might try to make such a system work on a 32bit 
system and fail. The  fact that the limit does exist and changing the stack 
size doesn't really help things is a key there.

My point is that you can get a few more threads out of a machine with 4K 
stacks, even on 32bit. Sure, the difference is basically negligible, but it 
does happen. That extra available space may be the difference between a 
poorly coded program triggering random crashes (and the OOM killer) and the 
system surviving it. 

While it's true that I feel that the job of the kernel isn't to protect the 
incompetent, it should protect the competent admins from the incompetent 
developers (and middle management).

> They could indeed. It would not be very efficient but it should work
> in theory at least with enough memory. Of course they don't need 4k
> stacks for it. They can also try it on 32bit and it will work
> to some extent too, just not scale very far.  And 4k stack more or less
> won't make much difference for that because the stack is only
> a small part of the lowmem needed for a blocked thread with
> open sockets.

True. But having that tiny bit of extra memory might be the difference between 
a crash and a somewhat memory starved but surviving system.

> But this thread clearly was about 32bit systems only.

I didn't say otherwise. I was pointing out that 50K threads isn't out of the 
question when looking at the workload provided (and ignoring all other memory 
concerns.

However, I had hoped I wouldn't have to spell out the stuff I've had to point 
out in this mail.

> > that 50K thread-count that you seem to be
> > calling "stupid".
>
> Note I didn't come up with that number, it was quoted to me earlier
> (but one of its authors has distanced itself from it now, so it
> seems to becoming more and more irrelevant indeed now)

Yes, I know you didn't come up with it. But in seeing the original commit-log 
for it, I'm thinking that the '50K' number was initially meant as either a 
small joke or a dream of a maximum.

> Stupid in this case just refers to the general observation that
> it is quite inefficient to do one thread per request on servers
> who are expected to process lots of long running connections.

Remember, you're talking about people that write the code in Java. It's going 
to spawn all kinds of threads anyway. I, personally, would write the code in 
a language giving me better control over the available resources. However, 
I'm not employed by any major company because I will almost always refuse to 
work on a project if it's being done in an inefficient manner.

> Perhaps I could have put that better I will give you that. Please
> assume I always meant "inefficient" when I wrote "stupid".

In that case I agree. It is very inefficient to do things that way.

> > talking about. If I had been running 4K stacks on that machine I probably
> > would have survived the mis-configuration without the reboot it took to
> > make
>
> Now that is a very doubtful claim. You realize that a functional network
> server thread needs a lot more lowmem than just the stack?

There was nothing else running on the machine and it was reporting lowmem free 
in the logs, just none "usable". Since the two biggest hogs on that box are 
Apache2 and MySQL - and since repairing the Apache2 config damage has halted 
further OOM's on that machine, I'm pretty much certain that it was Apache2 at 
fault, though since there were reports of free lowmem, I'm pretty certain it 
was a combination of fragmentation and Apache2.

DRH


-- 
Dialup is like pissing through a pipette. Slow and excruciatingly painful.

Re: x86: 4kstacks default

[Arjan van de Ven]

On Sun, 20 Apr 2008 22:01:46 +0200
Andi Kleen <andi@firstfloor.org> wrote:

> 
> 
> > These are real customer workloads; java based "many things going
> > on" at a time showed several thousands of threads fin the system (a
> > dozen or two per request, multiplied by the number of outstanding
> > connections) for *real customers*.
> 
> Several thousands or 50k? Several thousands sounds large, but not
> entirely unreasonable, but it is far from 50k.

it is you who keeps putting up the 50k argument.
What I'm talking about is in the 10k to 20k range; and that is actual workloads
by real customers.
> 
> > That you don't take that serious, fair, you can take serious
> > whatever you want.
> 
> No I don't take 50k threads on 32bit serious. And I hope you do not
> either.

[ removed a bunch of stuff about 50k again ]

> 
> > was the observation that fragmentation is fundamentally unsolvable.
> 
> Where was that observation? 

it was in the commit message from me you quoted, and was rather widely discussed at the time.
It's also basic math; the Linux VM gets to deal with both short and long lasting allocations;
no matter how hard you try to get some degree of fragmentation; especially due to the
15:1 acceleration you get due to the lowmem issue.

And before you say "you should use 64 bit on such machines"; I would love it if more people used 64 bit linux.
Sadly the adoption rate of that is not very good still.... by far ;(

> 
> > Yes 2.4 sucked a lot more than 2.6 does. But even 2.6 will (and
> > does) have fragmentation issues. We don't have effective physical
> > address based reclaim yet for higher order allocs.
> 
> I don't see any evidence that there are serious order 1 fragmentation 
> issues on 2.6. 

I assume you're not asking me to give you customer confidential data from a previous job in public ;)

>If you have any please post it.

just like you're posting the evidence that 4k stacks overflows?

Google scores:

1-order allocation failed		54000 pages
do_IRQ: stack overflow			4560 pages


-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: x86: 4kstacks default

[Arjan van de Ven]

On Sun, 20 Apr 2008 22:01:46 +0200
Andi Kleen <andi@firstfloor.org> wrote:

> 
> 
> > These are real customer workloads; java based "many things going
> > on" at a time showed several thousands of threads fin the system (a
> > dozen or two per request, multiplied by the number of outstanding
> > connections) for *real customers*.
> 
> Several thousands or 50k? Several thousands sounds large, but not
> entirely unreasonable, but it is far from 50k.

it is you who keeps putting up the 50k argument.
What I'm talking about is in the 10k to 20k range; and that is actual workloads
by real customers.
> 
> > That you don't take that serious, fair, you can take serious
> > whatever you want.
> 
> No I don't take 50k threads on 32bit serious. And I hope you do not
> either.

[ removed a bunch of stuff about 50k again ]

> 
> > was the observation that fragmentation is fundamentally unsolvable.
> 
> Where was that observation? 

it was in the commit message from me you quoted, and was rather widely discussed at the time.
It's also basic math; the Linux VM gets to deal with both short and long lasting allocations;
no matter how hard you try to get some degree of fragmentation; especially due to the
15:1 acceleration you get due to the lowmem issue.

And before you say "you should use 64 bit on such machines"; I would love it if more people used 64 bit linux.
Sadly the adoption rate of that is not very good still.... by far ;(

> 
> > Yes 2.4 sucked a lot more than 2.6 does. But even 2.6 will (and
> > does) have fragmentation issues. We don't have effective physical
> > address based reclaim yet for higher order allocs.
> 
> I don't see any evidence that there are serious order 1 fragmentation 
> issues on 2.6. 

I assume you're not asking me to give you customer confidential data from a previous job in public ;)

>If you have any please post it.

just like you're posting the evidence that 4k stacks overflows?

Google scores:

1-order allocation failed		54000 pages
do_IRQ: stack overflow			4560 pages


-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: x86: 4kstacks default

[Andi Kleen]

Arjan van de Ven <arjan@infradead.org> writes:
>
> it is you who keeps putting up the 50k argument.

See the links I posted and quote in an earlier message up the thread if you
don't remember what you wrote yourself.

I originally only hold up the fragmentation argument (or rather only
argued against it), until I was corrected by both Ingo and you in the
earlier thread and you both insisted that 50k threads were the real
reason'd'etre for 4k stacks.  

You're saying that was wrong and the fragmentation issue was really the 
real reason for 4k stacks? If both you and Ingo can agree on that
I would be happy to forget the 50k threads :)

> What I'm talking about is in the 10k to 20k range; and that is actual workloads
> by real customers.

On a 32bit kernel? 

My estimate is that you need around 32k for a functional blocked thread
in a network server (8k + 2*4k for poll with large fd table and wait queues + 
some pinned dentries and inodes + misc other stuff). With 20k you're 625MB into
your lowmem which leaves about 200MB left on a 3:1 system with 16GB 
(and ~128MB mem_map).  That might work for some time, but I expect it will fall
over at some point because there is just too much pinned lowmem
and not enough left for other stuff (like networking buffers etc.) 

10k sounds more doable. But again do 4k more or less make 
a big difference with the other thread overhead? I don't think so.

And trading reliability (and functionality -- you basically have to
cut off XFS)just for 4k/thread doesn't seem like good bargain to
me. Especially with kernel code getting more complicated all the time.

>> I don't see any evidence that there are serious order 1 fragmentation 
>> issues on 2.6. 
>
> I assume you're not asking me to give you customer confidential data from a previous job in public ;)

Well if it is that serious a problem surely it will have hit some public
bugzillas or mailing lists?  Arguing with something secret is also not 
very useful.

Also I find it always important to reevaluate assumptions when new 
facts come up. In this case we should reevaluate a decision that made
sense[1] in 2.4 with the new facts of 2.6 (e.g. new VM with much better
reclaim)

[1] refering to the fragmentation argument, not the 50k threads which
were always unrealistic.

-Andi

Re: x86: 4kstacks default

[Arjan van de Ven]

On Mon, 21 Apr 2008 01:16:22 +0200
Andi Kleen <andi@firstfloor.org> wrote:

> Arjan van de Ven <arjan@infradead.org> writes:
> >
> > it is you who keeps putting up the 50k argument.
> 
> See the links I posted and quote in an earlier message up the thread
> if you don't remember what you wrote yourself.
> 
> I originally only hold up the fragmentation argument (or rather only
> argued against it), until I was corrected by both Ingo and you in the
> earlier thread and you both insisted that 50k threads were the real
> reason'd'etre for 4k stacks.  
> 
> You're saying that was wrong and the fragmentation issue was really
> the real reason for 4k stacks? If both you and Ingo can agree on that
> I would be happy to forget the 50k threads :)

I already corrected you misquoting/misunderstanding me; should I do this again?

> 
> > What I'm talking about is in the 10k to 20k range; and that is
> > actual workloads by real customers.
> 
> On a 32bit kernel? 
> 
> My estimate is that you need around 32k for a functional blocked
> thread in a network server (8k + 2*4k for poll with large fd table
> and wait queues + some pinned dentries and inodes + misc other
> stuff). With 20k you're 625MB into your lowmem which leaves about
> 200MB left on a 3:1 system with 16GB (and ~128MB mem_map).  That
> might work for some time, but I expect it will fall over at some
> point because there is just too much pinned lowmem and not enough
> left for other stuff (like networking buffers etc.) 
> 
> 10k sounds more doable. But again do 4k more or less make 
> a big difference with the other thread overhead? I don't think so.

no but the other ones are order 0..

-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: x86: 4kstacks default

[Eric Sandeen]

Arjan van de Ven wrote:

> Google scores:
> 
> 1-order allocation failed		54000 pages
> do_IRQ: stack overflow			4560 pages

with quotes for exact matches:

"1-order allocation failed"	790 pages
"do_IRQ: stack overflow"	1,880 pages

http://www.google.com/search?q=%221-order+allocation+failed%22
http://www.google.com/search?q=%22do_IRQ%3A+stack+overflow%22

-Eric

Re: x86: 4kstacks default

[Andrew Morton]

> On Sun, 20 Apr 2008 11:48:45 -0700 Arjan van de Ven <arjan@infradead.org> wrote:
> We don't have effective physical address based reclaim yet for higher order allocs.

Lumpy reclaim is supposed to be exactly that.

Re: x86: 4kstacks default

[Andi Kleen]

Andrew Morton wrote:
>> On Sun, 20 Apr 2008 11:48:45 -0700 Arjan van de Ven <arjan@infradead.org> wrote:
>> We don't have effective physical address based reclaim yet for higher order allocs.
> 
> Lumpy reclaim is supposed to be exactly that.

Also if order 1 allocs were a significant problem on i386 we must have
had lots of reports of EAGAIN on fork/clone with !4k stack kernels. I'm
not aware of an significant number of such reports (there were a few
occasionally, but that is probably normal and unavoidable and can
be caused by other things too like simply running out of lowmem)

-Andi

Re: x86: 4kstacks default

[Romano Giannetti]

On Sun, 2008-04-20 at 11:44 -0400, Daniel Hazelton wrote:
> Since the second-most-common reason for stack overages is ndiswrapper... Well, 
> with there being so much more hardware now supported directly by the linux 
> kernel... 

How would I like you being right... Atheros AR5008, AR5414 PHY, "not yet
here". It's almost one year now since I bought this laptop, and till now
it's the cable or ndiswrapper. But yes, it's going better. For my first
wifi laptop I waited two and a half years, now it seems that in a bit
more than one there will be an open source driver...

I know all the trouble ndiswrapper signify. But I see also that people
around me with a laptop and linux use more ndiswrapper than a real
driver, so... be gentle with it. 

Thanks,
	Romano 	

-- 
Sorry for the disclaimer --- ¡I cannot stop it!



--
La presente comunicación tiene carácter confidencial y es para el exclusivo uso del destinatario indicado en la misma. Si Ud. no es el destinatario indicado, le informamos que cualquier forma de distribución, reproducción o uso de esta comunicación y/o de la información contenida en la misma están estrictamente prohibidos por la ley. Si Ud. ha recibido esta comunicación por error, por favor, notifíquelo inmediatamente al remitente contestando a este mensaje y proceda a continuación a destruirlo. Gracias por su colaboración.

This communication contains confidential information. It is for the exclusive use of the intended addressee. If you are not the intended addressee, please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited by law. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this message. Thank you for your cooperation. 

Re: x86: 4kstacks default

[Denys Vlasenko]

On Tuesday 22 April 2008 20:20, Romano Giannetti wrote:
> On Sun, 2008-04-20 at 11:44 -0400, Daniel Hazelton wrote:
> > Since the second-most-common reason for stack overages is ndiswrapper... Well, 
> > with there being so much more hardware now supported directly by the linux 
> > kernel... 
> 
> How would I like you being right... Atheros AR5008, AR5414 PHY, "not yet
> here". It's almost one year now since I bought this laptop, and till now
> it's the cable or ndiswrapper. But yes, it's going better. For my first
> wifi laptop I waited two and a half years, now it seems that in a bit
> more than one there will be an open source driver...
> 
> I know all the trouble ndiswrapper signify. But I see also that people
> around me with a laptop and linux use more ndiswrapper than a real
> driver, so... be gentle with it. 

Nobody knows how much potential development is not done because
"you can make your wifi work with ndiswrapper".
--
vda

Re: x86: 4kstacks default

[Daniel Hazelton]

On Wednesday 23 April 2008 01:03:11 Denys Vlasenko wrote:
> On Tuesday 22 April 2008 20:20, Romano Giannetti wrote:
> > On Sun, 2008-04-20 at 11:44 -0400, Daniel Hazelton wrote:
> > > Since the second-most-common reason for stack overages is
> > > ndiswrapper... Well, with there being so much more hardware now
> > > supported directly by the linux kernel...
> >
> > How would I like you being right... Atheros AR5008, AR5414 PHY, "not yet
> > here". It's almost one year now since I bought this laptop, and till now
> > it's the cable or ndiswrapper. But yes, it's going better. For my first
> > wifi laptop I waited two and a half years, now it seems that in a bit
> > more than one there will be an open source driver...
> >
> > I know all the trouble ndiswrapper signify. But I see also that people
> > around me with a laptop and linux use more ndiswrapper than a real
> > driver, so... be gentle with it.
>
> Nobody knows how much potential development is not done because
> "you can make your wifi work with ndiswrapper".

I've got to agree with that sentiment. Once a working solution is found, no 
matter how crappy, it seems that almost all development stops.

DRH

-- 
Dialup is like pissing through a pipette. Slow and excruciatingly painful.

Re: x86: 4kstacks default

[david]

On Wed, 23 Apr 2008, Daniel Hazelton wrote:

> On Wednesday 23 April 2008 01:03:11 Denys Vlasenko wrote:
>> On Tuesday 22 April 2008 20:20, Romano Giannetti wrote:
>>> On Sun, 2008-04-20 at 11:44 -0400, Daniel Hazelton wrote:
>>>> Since the second-most-common reason for stack overages is
>>>> ndiswrapper... Well, with there being so much more hardware now
>>>> supported directly by the linux kernel...
>>>
>>> How would I like you being right... Atheros AR5008, AR5414 PHY, "not yet
>>> here". It's almost one year now since I bought this laptop, and till now
>>> it's the cable or ndiswrapper. But yes, it's going better. For my first
>>> wifi laptop I waited two and a half years, now it seems that in a bit
>>> more than one there will be an open source driver...
>>>
>>> I know all the trouble ndiswrapper signify. But I see also that people
>>> around me with a laptop and linux use more ndiswrapper than a real
>>> driver, so... be gentle with it.
>>
>> Nobody knows how much potential development is not done because
>> "you can make your wifi work with ndiswrapper".
>
> I've got to agree with that sentiment. Once a working solution is found, no
> matter how crappy, it seems that almost all development stops.

and nobody knows how many people are running linux instead of windows 
becouse they were able to use ndiswrapper to get things running. most of 
those people contributed nothing to the kernel, but they all contributed 
to Linux, if nothing else as examples that Linux is a reasonable option 
(and some percentage of those users have contrinbuted to other opensource 
projects that they would probably never have bumped into if they were 
running windows instead)

I know we will never convince each other, but we do need to recognise that 
there is another valid point of view.

David Lang

Re: x86: 4kstacks default

[Daniel Hazelton]

On Wednesday 23 April 2008 01:25:27 david@lang.hm wrote:
> On Wed, 23 Apr 2008, Daniel Hazelton wrote:
> > On Wednesday 23 April 2008 01:03:11 Denys Vlasenko wrote:
> >> On Tuesday 22 April 2008 20:20, Romano Giannetti wrote:
> >>> On Sun, 2008-04-20 at 11:44 -0400, Daniel Hazelton wrote:
> >>>> Since the second-most-common reason for stack overages is
> >>>> ndiswrapper... Well, with there being so much more hardware now
> >>>> supported directly by the linux kernel...
> >>>
> >>> How would I like you being right... Atheros AR5008, AR5414 PHY, "not
> >>> yet here". It's almost one year now since I bought this laptop, and
> >>> till now it's the cable or ndiswrapper. But yes, it's going better. For
> >>> my first wifi laptop I waited two and a half years, now it seems that
> >>> in a bit more than one there will be an open source driver...
> >>>
> >>> I know all the trouble ndiswrapper signify. But I see also that people
> >>> around me with a laptop and linux use more ndiswrapper than a real
> >>> driver, so... be gentle with it.
> >>
> >> Nobody knows how much potential development is not done because
> >> "you can make your wifi work with ndiswrapper".
> >
> > I've got to agree with that sentiment. Once a working solution is found,
> > no matter how crappy, it seems that almost all development stops.
>
> and nobody knows how many people are running linux instead of windows
> becouse they were able to use ndiswrapper to get things running. most of
> those people contributed nothing to the kernel, but they all contributed
> to Linux, if nothing else as examples that Linux is a reasonable option
> (and some percentage of those users have contrinbuted to other opensource
> projects that they would probably never have bumped into if they were
> running windows instead)
>
> I know we will never convince each other, but we do need to recognise that
> there is another valid point of view.

And who knows how many more people would be running Linux if they didn't need 
ndiswrapper at all?

And how much better would it be if the drivers were native linux code and were 
fully supportable because of that?

There are many, many reasons why it'd be better if ndiswrapper didn't exist as 
a solution or if development on native solutions continued on at the level it 
would without ndiswrapper. 

DRH

-- 
Dialup is like pissing through a pipette. Slow and excruciatingly painful.

Re: x86: 4kstacks default

[Romano Giannetti]

[Trimmed, I hope I got the authors right...]

On Wed, 2008-04-23 at 01:41 -0400, Daniel Hazelton wrote:
> > On Wed, 23 Apr 2008, Daniel Hazelton wrote:
> > > On Wednesday 23 April 2008 01:03:11 Denys Vlasenko wrote:
> > >> On Tuesday 22 April 2008 20:20, Romano Giannetti wrote:

> > >>> I know all the trouble ndiswrapper signify. But I see also that people
> > >>> around me with a laptop and linux use more ndiswrapper than a real
> > >>> driver, so... be gentle with it.
> > >>
> > >> Nobody knows how much potential development is not done because
> > >> "you can make your wifi work with ndiswrapper".
> > >

> And how much better would it be if the drivers were native linux code and were 
> fully supportable because of that?

I understand your position, but let me give my example. I have this
laptop that is one year old. I'm helping in all what I can to the
development of ath5k --- IOW, offering testing, I am not an expert on
this. 

But the mere fact that ndiswrapper exists enabled me to use this laptop
on a daily basis, and so I could test new kernel (and if you look at the
logs you'll see I had at least helped to fix a nasty MMC bug, and to
make sound work in this laptop) and help in other areas, like
suspend/resume testing and bug chasing. 

There is not only wireless development. Without ndiswrapper, I wouldn't
have been in any position to help other areas. I would have had a
crippled laptop[1], a much higher Vista uptime (which now is 0), and a
far bitter Linux experience.  

And this is the point of view of someone that is using Linux since
0.99pl9, so I have quite a bit of experience. 99% of normal users would
simply say "don't work"[2].

		Romano

[1] yes, there's a madwifi version locked to a specific kernel that
works with my card. But I do not think that this would be so much
different. 

[2] a nice page with "_this_ laptop will fully work with linux" would be
nice. Linux on laptop or similar is too complex to be a real help when
you have to buy a laptop in 2 days. 


-- 
Sorry for the disclaimer --- ¡I cannot stop it!



--
La presente comunicación tiene carácter confidencial y es para el exclusivo uso del destinatario indicado en la misma. Si Ud. no es el destinatario indicado, le informamos que cualquier forma de distribución, reproducción o uso de esta comunicación y/o de la información contenida en la misma están estrictamente prohibidos por la ley. Si Ud. ha recibido esta comunicación por error, por favor, notifíquelo inmediatamente al remitente contestando a este mensaje y proceda a continuación a destruirlo. Gracias por su colaboración.

This communication contains confidential information. It is for the exclusive use of the intended addressee. If you are not the intended addressee, please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited by law. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this message. Thank you for your cooperation. 

Re: x86: 4kstacks default

[Stefan Richter]

Romano Giannetti wrote:
[stack size requirements -> ndiswrapper -> hardware support status -> ?]
> a nice page with "_this_ laptop will fully work with linux" would be
> nice. Linux on laptop or similar is too complex to be a real help when
> you have to buy a laptop in 2 days. 

If sites like tuxmobil.org, hardware4linux.info, and the hardware
compatibility databases of Linux distributors don't work for you, then
just ask the notebook vendors directly.
-- 
Stefan Richter
-=====-==--- -=-- =-===
http://arcgraph.de/sr/

Re: x86: 4kstacks default

[Romano Giannetti]

On Wed, 2008-04-23 at 13:24 +0200, Stefan Richter wrote:
> Romano Giannetti wrote:
> [stack size requirements -> ndiswrapper -> hardware support status -> ?]
> > a nice page with "_this_ laptop will fully work with linux" would be
> > nice. Linux on laptop or similar is too complex to be a real help when
> > you have to buy a laptop in 2 days. 
> 
> If sites like tuxmobil.org, hardware4linux.info, and the hardware
> compatibility databases of Linux distributors don't work for you, then
> just ask the notebook vendors directly.

Unfortunately, it is quite a complex thing to check. Mind you, I've
bought this laptop after looking all over there, but:

- tuxmobil & Co are very user-driven, and you have to swim among tenth
of "similar" computer;

- it's not so easy to know what exactly is bundled with a laptop[1];

- vendor say "works" (and often is listed as works in the aforementioned
sites too) independently if it works with an open source driver or not.
As an example, all the nvidia-based graphics are marked "works".

Romano 

[1] In my case, I selected this toshiba over for example a HP or an Acer
because it had "atheros wifi" (but guess that the PHY version is too new
to be supported...), "intel hda sound" (but guess that the specific
codec didn't work at all, and continues to have a lot of problems), 
"intel graphics" (and that at least was a good decision!). 


-- 
Sorry for the disclaimer --- ¡I cannot stop it!



--
La presente comunicación tiene carácter confidencial y es para el exclusivo uso del destinatario indicado en la misma. Si Ud. no es el destinatario indicado, le informamos que cualquier forma de distribución, reproducción o uso de esta comunicación y/o de la información contenida en la misma están estrictamente prohibidos por la ley. Si Ud. ha recibido esta comunicación por error, por favor, notifíquelo inmediatamente al remitente contestando a este mensaje y proceda a continuación a destruirlo. Gracias por su colaboración.

This communication contains confidential information. It is for the exclusive use of the intended addressee. If you are not the intended addressee, please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited by law. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this message. Thank you for your cooperation. 

Re: x86: 4kstacks default

[Lennart Sorensen]

On Wed, Apr 23, 2008 at 02:15:01PM +0200, Romano Giannetti wrote:
> - vendor say "works" (and often is listed as works in the aforementioned
> sites too) independently if it works with an open source driver or not.
> As an example, all the nvidia-based graphics are marked "works".

The nv driver does work for all the nvidia cards as far as I know.  Sure
you don't get 3D acceleration, but you do get working X.

But yes it is quite annoying when companies like highpoint (and others)
claim to support linux when all they have is binary blobs as part of their
"driver".

-- 
Len Sorensen

Re: x86: 4kstacks default

[Mark Lord]

Adrian Bunk wrote:
>
> The code in the kernel that gets the fewest coverage at all are our 
> error paths, and some vendor might try 4k stacks, validate it works in 
> all use cases - and then it will blow up in some error condition he 
> didn't test.
..

That's exactly the worry.

If anyone want's to take a crack at testing some of the more likely
fail paths there, just introduce a media error onto a SATA disk
that's buried at the bottom of a stacked RAID1 over RAID0 over LVM,
with XFS and nfsd on top.

Or something like that.
And then experiment with corrupting meta data rather than simply file data.
How-to introduce a media error?  hdparm --make-bad-sector nnnnnn /dev/sdX

This catches the most likely (IMHO) failure scenarios,
but still comes nowhere near 100% code coverage.  :(

Cheers

Re: x86: 4kstacks default

[Andrew Morton]

> On Sat, 19 Apr 2008 16:23:29 +0200 Ingo Molnar <mingo@elte.hu> wrote:
> 
> * Andrew Morton <akpm@linux-foundation.org> wrote:
> 
> > >  config 4KSTACKS
> > >  	bool "Use 4Kb for kernel stacks instead of 8Kb"
> > > -	depends on DEBUG_KERNEL
> > >  	depends on X86_32
> > > +	default y
> > 
> > This patch will cause kernels to crash.
> 
> what mainline kernels crash and how will they crash?

There has been a dribble of reports - I don't have the links handy, nor did
I search for them.

> Fedora and other 
> distros have had 4K stacks enabled for years:
> 
>   $ grep 4K /boot/config-2.6.24-9.fc9
>   CONFIG_4KSTACKS=y
> 
> and we've conducted tens of thousands of bootup tests with all sorts of 
> drivers and kernel options enabled and have yet to see a single crash 
> due to 4K stacks.

I doubt if you're testing things like nfsd-on-xfs-on-md-on-porky-scsi-driver.

Enable CONFIG_DEBUG_STACK_USAGE.  Monitor the results.  It's so scary that
I wonder if the feature is busted.

> So basically the kernel default just follows the 
> common distro default now. (distros and users can still disable it)

Apparently not.  I wouldn't enable it if I had a distro.

Anyway.  We should be having this sort of discussion _before_ a patch
gets merged, no?

Re: x86: 4kstacks default

[Parag Warudkar]

Andrew Morton <akpm <at> linux-foundation.org> writes:

> 
> > On Sat, 19 Apr 2008 16:23:29 +0200 Ingo Molnar <mingo <at> elte.hu> wrote:
> > 
> > * Andrew Morton <akpm <at> linux-foundation.org> wrote:
> > 
> > > >  config 4KSTACKS
> > > >  	bool "Use 4Kb for kernel stacks instead of 8Kb"
> > > > -	depends on DEBUG_KERNEL
> > > >  	depends on X86_32
> > > > +	default y
> > > 
> > > This patch will cause kernels to crash.
> > 
> > what mainline kernels crash and how will they crash?
> 
> There has been a dribble of reports - I don't have the links handy, nor did
> I search for them.
> 
> > Fedora and other 
> > distros have had 4K stacks enabled for years:
> > 
> >   $ grep 4K /boot/config-2.6.24-9.fc9
> >   CONFIG_4KSTACKS=y

Here is a report - Fedora 8 default kernel, Mac Mini file server, Not Tainted. 
Attempt to copy 100Gb+ of data from a hfsplus file system on a USB drive to a 
firewire drive with XFS filesystem - I got a nasty panic with a huge stack 
backtrace. I gave up and switched to Ubuntu. With a stock kernel.org kernel I 
was able to successfully copy the data over. I still have the machine and the 
restored drives and can try to reproduce it with Fedora 9 w/4K stacks if 
anyone thinks it is worthwhile (i.e. fixable).

Parag

Re: x86: 4kstacks default

[Eric Sandeen]

Ingo Molnar wrote:
> * Andrew Morton <akpm@linux-foundation.org> wrote:
> 
>>>  config 4KSTACKS
>>>  	bool "Use 4Kb for kernel stacks instead of 8Kb"
>>> -	depends on DEBUG_KERNEL
>>>  	depends on X86_32
>>> +	default y
>> This patch will cause kernels to crash.
> 
> what mainline kernels crash and how will they crash? Fedora and other 
> distros have had 4K stacks enabled for years:
> 
>   $ grep 4K /boot/config-2.6.24-9.fc9
>   CONFIG_4KSTACKS=y
> 
> and we've conducted tens of thousands of bootup tests with all sorts of 
> drivers and kernel options enabled and have yet to see a single crash 
> due to 4K stacks. 

Really, not one?

https://bugzilla.redhat.com/show_bug.cgi?id=247158
https://bugzilla.redhat.com/show_bug.cgi?id=227331
https://bugzilla.redhat.com/show_bug.cgi?id=240077

(hehe, ok, xfs is a common component there...)

and it's not always obvious that you've overflowed the stack.

CONFIG_DEBUG_STACKOVERFLOW isn't ery useful because the warning printk
it generates uses the remaining amount of stack, and tips the box.

> So basically the kernel default just follows the 
> common distro default now. (distros and users can still disable it)

If Fedora is the common distro, ok. :)

Fedora is a pretty narrow sample in terms of IO stacks at least.  I have
plenty of fondness for Fedora, but it's almost 100% ext3[1].  I spent a
fair amount of time getting xfs+lvm to survive 4k on F8; gcc caused
stack usage to grow in general from F7 to F8, and F9 seems to have
gotten tight again but I haven't gotten to the bottom of yet.

Heck my ext3-root-on-sda1 pre-beta F9 box, no nfs or lvm or xfs or
anything gets within 744 bytes of the end of the 4k stack simply by
*booting* (it was a modprobe process... maybe some module needs help)

How many other distros use 4K stacks on x86, really?

-Eric

[1] http://www.smolts.org/static/stats/stats.html shows 24588 ext3
filesystems, compared to 366 xfs, 248 reiserfs, 76 jfs ...

Re: x86: 4kstacks default

[Andi Kleen]

Eric Sandeen <sandeen@sandeen.net> writes:
>
> CONFIG_DEBUG_STACKOVERFLOW isn't ery useful because the warning printk
> it generates uses the remaining amount of stack, and tips the box.

That could be easily fixed by executing the printk on the interrupt
stack on i386.  Currently it is before the stack switch which is wrong
agreed. On x86-64 it should already execute on the interrupt stack. Or
perhaps it would be better to just move the stack switch on i386 into
entry.S too similar to 64bit.

That wouldn't help without interrupt stacks of course, but these
should be always on anyways even with 8k stacks.

Experimental patch appended to do this.

-Andi

---

i386: Execute stack overflow warning on interrupt stack

Previously it would run on the process stack, which risks overflow
an already low stack. Instead execute it on the interrupt stack.

Based on an observation by Eric Sandeen.

Signed-off-by: Andi Kleen <andi@firstfloor.org>

Index: linux/arch/x86/kernel/irq_32.c
===================================================================
--- linux.orig/arch/x86/kernel/irq_32.c
+++ linux/arch/x86/kernel/irq_32.c
@@ -61,6 +61,26 @@ static union irq_ctx *hardirq_ctx[NR_CPU
 static union irq_ctx *softirq_ctx[NR_CPUS] __read_mostly;
 #endif
 
+static void stack_overflow(void)
+{
+	printk("low stack detected by irq handler\n");
+	dump_stack();
+}
+
+static inline void call_on_stack2(void *func, unsigned long stack,
+			   unsigned long arg1, unsigned long arg2)
+{
+	unsigned long bx;
+	asm volatile(
+			"       xchgl  %%ebx,%%esp    \n"
+			"       call   *%%edi         \n"
+			"       movl   %%ebx,%%esp    \n"
+			: "=a" (arg1), "=d" (arg2), "=b" (bx)
+			:  "0" (arg1),   "1" (arg2),  "2" (stack),
+			   "D" (func)
+			: "memory", "cc");
+}
+
 /*
  * do_IRQ handles all normal device IRQ's (the special
  * SMP cross-CPU interrupts have their own specific
@@ -76,6 +96,7 @@ unsigned int do_IRQ(struct pt_regs *regs
 	union irq_ctx *curctx, *irqctx;
 	u32 *isp;
 #endif
+	int overflow = 0;
 
 	if (unlikely((unsigned)irq >= NR_IRQS)) {
 		printk(KERN_EMERG "%s: cannot handle IRQ %d\n",
@@ -92,11 +113,8 @@ unsigned int do_IRQ(struct pt_regs *regs
 
 		__asm__ __volatile__("andl %%esp,%0" :
 					"=r" (sp) : "0" (THREAD_SIZE - 1));
-		if (unlikely(sp < (sizeof(struct thread_info) + STACK_WARN))) {
-			printk("do_IRQ: stack overflow: %ld\n",
-				sp - sizeof(struct thread_info));
-			dump_stack();
-		}
+		if (unlikely(sp < (sizeof(struct thread_info) + STACK_WARN)))
+			overflow = 1;
 	}
 #endif
 
@@ -112,8 +130,6 @@ unsigned int do_IRQ(struct pt_regs *regs
 	 * current stack (which is the irq stack already after all)
 	 */
 	if (curctx != irqctx) {
-		int arg1, arg2, bx;
-
 		/* build the stack frame on the IRQ stack */
 		isp = (u32*) ((char*)irqctx + sizeof(*irqctx));
 		irqctx->tinfo.task = curctx->tinfo.task;
@@ -127,18 +143,20 @@ unsigned int do_IRQ(struct pt_regs *regs
 			(irqctx->tinfo.preempt_count & ~SOFTIRQ_MASK) |
 			(curctx->tinfo.preempt_count & SOFTIRQ_MASK);
 
-		asm volatile(
-			"       xchgl  %%ebx,%%esp    \n"
-			"       call   *%%edi         \n"
-			"       movl   %%ebx,%%esp    \n"
-			: "=a" (arg1), "=d" (arg2), "=b" (bx)
-			:  "0" (irq),   "1" (desc),  "2" (isp),
-			   "D" (desc->handle_irq)
-			: "memory", "cc"
-		);
+		/* Execute warning on interrupt stack */
+		if (unlikely(overflow))
+			call_on_stack2(stack_overflow, isp, 0, 0);
+
+		call_on_stack2(desc->handle_irq, isp, irq, desc);
+
 	} else
 #endif
+	{
+		/* AK: Slightly bogus here */
+		if (overflow)
+			stack_overflow();
 		desc->handle_irq(irq, desc);
+	}
 
 	irq_exit();
 	set_irq_regs(old_regs);

Re: x86: 4kstacks default

[Ingo Molnar]

* Eric Sandeen <sandeen@sandeen.net> wrote:

> > and we've conducted tens of thousands of bootup tests with all sorts 
> > of drivers and kernel options enabled and have yet to see a single 
> > crash due to 4K stacks.
> 
> Really, not one?
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=247158
> https://bugzilla.redhat.com/show_bug.cgi?id=227331
> https://bugzilla.redhat.com/show_bug.cgi?id=240077
> 
> (hehe, ok, xfs is a common component there...)
> 
> and it's not always obvious that you've overflowed the stack.
> 
> CONFIG_DEBUG_STACKOVERFLOW isn't ery useful because the warning printk 
> it generates uses the remaining amount of stack, and tips the box.

note that in -rt we have an ftrace plugin that measures _precise_ stack 
footprint, when it happens.

so it's possible to measure exact stack footprint and save a stack trace 
when that happens.

	Ingo

Re: x86: 4kstacks default

[Benjamin Herrenschmidt]

On Sat, 2008-04-19 at 16:23 +0200, Ingo Molnar wrote:
> * Andrew Morton <akpm@linux-foundation.org> wrote:
> 
> > >  config 4KSTACKS
> > >  	bool "Use 4Kb for kernel stacks instead of 8Kb"
> > > -	depends on DEBUG_KERNEL
> > >  	depends on X86_32
> > > +	default y
> > 
> > This patch will cause kernels to crash.
> 
> what mainline kernels crash and how will they crash? Fedora and other 
> distros have had 4K stacks enabled for years:
> 
>   $ grep 4K /boot/config-2.6.24-9.fc9
>   CONFIG_4KSTACKS=y
> 
> and we've conducted tens of thousands of bootup tests with all sorts of 
> drivers and kernel options enabled and have yet to see a single crash 
> due to 4K stacks. So basically the kernel default just follows the 
> common distro default now. (distros and users can still disable it)

Do we routinely test nasty scenarii such as a GFP_KERNEL allocation deep
in a call stack trying to swap something out to NFS ?

Ben.

Re: x86: 4kstacks default

[David Chinner]

On Wed, Apr 23, 2008 at 03:27:01PM +1000, Benjamin Herrenschmidt wrote:
> On Sat, 2008-04-19 at 16:23 +0200, Ingo Molnar wrote:
> > * Andrew Morton <akpm@linux-foundation.org> wrote:
> > 
> > > >  config 4KSTACKS
> > > >  	bool "Use 4Kb for kernel stacks instead of 8Kb"
> > > > -	depends on DEBUG_KERNEL
> > > >  	depends on X86_32
> > > > +	default y
> > > 
> > > This patch will cause kernels to crash.
> > 
> > what mainline kernels crash and how will they crash? Fedora and other 
> > distros have had 4K stacks enabled for years:
> > 
> >   $ grep 4K /boot/config-2.6.24-9.fc9
> >   CONFIG_4KSTACKS=y
> > 
> > and we've conducted tens of thousands of bootup tests with all sorts of 
> > drivers and kernel options enabled and have yet to see a single crash 
> > due to 4K stacks. So basically the kernel default just follows the 
> > common distro default now. (distros and users can still disable it)
> 
> Do we routinely test nasty scenarii such as a GFP_KERNEL allocation deep
> in a call stack trying to swap something out to NFS ?

I doubt it, because this is the place that a local XFS filesystem
typically blows a 4k stack (direct memory reclaim triggering
->writepage). Boot testing does nothing to exercise the potential
paths for stack overflows....

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: x86: 4kstacks default

[Arjan van de Ven]

On Thu, 24 Apr 2008 09:36:52 +1000
David Chinner <dgc@sgi.com> wrote:

> On Wed, Apr 23, 2008 at 03:27:01PM +1000, Benjamin Herrenschmidt
> wrote:
> > On Sat, 2008-04-19 at 16:23 +0200, Ingo Molnar wrote:
> > > * Andrew Morton <akpm@linux-foundation.org> wrote:
> > > 
> > > > >  config 4KSTACKS
> > > > >  	bool "Use 4Kb for kernel stacks instead of 8Kb"
> > > > > -	depends on DEBUG_KERNEL
> > > > >  	depends on X86_32
> > > > > +	default y
> > > > 
> > > > This patch will cause kernels to crash.
> > > 
> > > what mainline kernels crash and how will they crash? Fedora and
> > > other distros have had 4K stacks enabled for years:
> > > 
> > >   $ grep 4K /boot/config-2.6.24-9.fc9
> > >   CONFIG_4KSTACKS=y
> > > 
> > > and we've conducted tens of thousands of bootup tests with all
> > > sorts of drivers and kernel options enabled and have yet to see a
> > > single crash due to 4K stacks. So basically the kernel default
> > > just follows the common distro default now. (distros and users
> > > can still disable it)
> > 
> > Do we routinely test nasty scenarii such as a GFP_KERNEL allocation
> > deep in a call stack trying to swap something out to NFS ?
> 
> I doubt it, because this is the place that a local XFS filesystem
> typically blows a 4k stack (direct memory reclaim triggering
> ->writepage). Boot testing does nothing to exercise the potential
> paths for stack overflows....
> 

THe good news is that direct reclaim is.. rare.
And I also doubt XFS is unique here; imagine the whole stacking thing on x86-64 just the same ...

I wonder if the direct reclaim path should avoid direct reclaim if the stack has only X bytes left.
(where the value of X is... well we can figure that one out later)

The rarity of direct reclaim during normal use ought to make this not a performance problem per se,
and the benefits go further than just "XFS" or "4K stacks".

-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: x86: 4kstacks default

[Christoph Hellwig]

On Wed, Apr 23, 2008 at 05:45:16PM -0700, Arjan van de Ven wrote:
> THe good news is that direct reclaim is.. rare.
> And I also doubt XFS is unique here; imagine the whole stacking thing on x86-64 just the same ...

It's bad news actually.  Beause it means the stack overflow happens
totally random and hard to reproduce.   And no, XFS is not unique there,
any filesystem with a complex enough writeback path (aka extents +
delalloc + smart allocator) will have to use quite a lot here.  I'll be
my 2 cent that ext4 one finished up will run into this just as likely.

> I wonder if the direct reclaim path should avoid direct reclaim if the stack has only X bytes left.
> (where the value of X is... well we can figure that one out later)

Actually direct reclaim should be totally avoided for complex
filesystems.  It's horrible for the stack and for the filesystem
writeout policy and ondisk allocation strategies.

Re: x86: 4kstacks default

[Peter Zijlstra]

On Thu, 2008-04-24 at 05:52 -0400, Christoph Hellwig wrote:

> > I wonder if the direct reclaim path should avoid direct reclaim if the stack has only X bytes left.
> > (where the value of X is... well we can figure that one out later)
> 
> Actually direct reclaim should be totally avoided for complex
> filesystems.  It's horrible for the stack and for the filesystem
> writeout policy and ondisk allocation strategies.

That's basically any reclaim, even kswapd will ruin policy and block
allocation smarts.

Re: x86: 4kstacks default

[Chris Mason]

On Thursday 24 April 2008, Christoph Hellwig wrote:
> On Wed, Apr 23, 2008 at 05:45:16PM -0700, Arjan van de Ven wrote:
> > THe good news is that direct reclaim is.. rare.
> > And I also doubt XFS is unique here; imagine the whole stacking thing on
> > x86-64 just the same ...
>
> It's bad news actually.  Beause it means the stack overflow happens
> totally random and hard to reproduce.   And no, XFS is not unique there,
> any filesystem with a complex enough writeback path (aka extents +
> delalloc + smart allocator) will have to use quite a lot here.  I'll be
> my 2 cent that ext4 one finished up will run into this just as likely.
>
> > I wonder if the direct reclaim path should avoid direct reclaim if the
> > stack has only X bytes left. (where the value of X is... well we can
> > figure that one out later)
>
> Actually direct reclaim should be totally avoided for complex
> filesystems.  It's horrible for the stack and for the filesystem
> writeout policy and ondisk allocation strategies.

Just as a data point, XFS isn't alone.  I run through once or twice a month 
and try to get rid of any new btrfs stack pigs, but keeping under the 4k 
stack barrier is a constant challenge.  

My storage configuration is fairly simple, if we spin the wheel of stacked IO 
devices...it won't be pretty.

Does it make more sense to kill off some brain cells on finding ways to 
dynamically increase the stack as we run out?  Or even give the robust stack 
users like xfs/btrfs a way to say: I'm pretty sure this call path is going to 
hurt, please make my stack bigger now.

We have relatively few entry points between the rest of the kernel and the FS, 
there should be some ways to compromise here.

-chris

Re: x86: 4kstacks default

[Alexander van Heukelum]

On Thu, 24 Apr 2008 11:41:30 -0400, "Chris Mason"
<chris.mason@oracle.com> said:
> On Thursday 24 April 2008, Christoph Hellwig wrote:
> > On Wed, Apr 23, 2008 at 05:45:16PM -0700, Arjan van de Ven wrote:
> > > THe good news is that direct reclaim is.. rare. And I also doubt
> > > XFS is unique here; imagine the whole stacking thing on x86-64
> > > just the same ...
> >
> > It's bad news actually.  Beause it means the stack overflow happens
> > totally random and hard to reproduce.   And no, XFS is not unique
> > there, any filesystem with a complex enough writeback path (aka
> > extents + delalloc + smart allocator) will have to use quite a lot
> > here.  I'll be my 2 cent that ext4 one finished up will run into
> > this just as likely.
> >
> > > I wonder if the direct reclaim path should avoid direct reclaim if
> > > the stack has only X bytes left. (where the value of X is... well
> > > we can figure that one out later)
> >
> > Actually direct reclaim should be totally avoided for complex
> > filesystems.  It's horrible for the stack and for the filesystem
> > writeout policy and ondisk allocation strategies.
>
> Just as a data point, XFS isn't alone.  I run through once or twice a
> month and try to get rid of any new btrfs stack pigs, but keeping
> under the 4k  stack barrier is a constant challenge.
>
> My storage configuration is fairly simple, if we spin the wheel of
> stacked IO devices...it won't be pretty.
>
> Does it make more sense to kill off some brain cells on finding ways
> to dynamically increase the stack as we run out?  Or even give the
> robust stack users like xfs/btrfs a way to say: I'm pretty sure this
> call path is going to hurt, please make my stack bigger now.

Hi,

(Rookie warning goes here.) To me, growing the stack at more or less
random places in the kernel seems to be quite a complicated thing to do
and it will be quite a maintainance burden to find the right spots to
insert stack usage checks. So I'ld say: lose the dynamic aspect.

How about unconditionally switching stacks at some defined points within
the core code of the kernel, just before calling into any driver code,
for example? The 4k-option has separate irq stacks already, why not have
driver stacks too?

I think the most important consideration to keep the stack size small
was that non-order-0 allocations are unreliable under/after memory
pressure due to fragmentation and that this allocation has to be done
for each thread. It is therefore preferable not to do any higher-order
allocations at all, unless there is a fall-back mechanism if the
allocation fails. For higher-order stacks there isn't such a fallback...
Can the system get by (without deadlocks at least in practice) with a
limited number of preallocated but 'large' stacks (in addition to a
small per-thread stack)?

It was discussed that stack space is needed for any sleeping process.
Could it be arranged that this waiting happens on the smallish stack, at
least for the most common cases, while non-waiting activity can use the
big stacks?

Greetings,
    Alexander

> We have relatively few entry points between the rest of the kernel and
> the FS, there should be some ways to compromise here.
>
> -chris
> --
> To unsubscribe from this list: send the line "unsubscribe linux-
> kernel" in the body of a message to majordomo@vger.kernel.org More
> majordomo info at  http://vger.kernel.org/majordomo-info.html Please
> read the FAQ at  http://www.tux.org/lkml/
>
>
-- 
  Alexander van Heukelum
  heukelum@fastmail.fm

-- 
http://www.fastmail.fm - A fast, anti-spam email service.

Re: x86: 4kstacks default

[Benjamin Herrenschmidt]

On Thu, 2008-04-24 at 09:36 +1000, David Chinner wrote:
> > Do we routinely test nasty scenarii such as a GFP_KERNEL allocation
> deep
> > in a call stack trying to swap something out to NFS ?
> 
> I doubt it, because this is the place that a local XFS filesystem
> typically blows a 4k stack (direct memory reclaim triggering
> ->writepage). Boot testing does nothing to exercise the potential
> paths for stack overflows....

Yup, note even counting when the said NFS is on top of some fancy
network stack with a driver on top of USB .... I mean, we do have
potential for worst case scenario that I think -will- blow a 4k stack.

Ben.