* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Adrian Bunk @ 2008-08-26 20:21 UTC (permalink / raw)
To: Linus Torvalds
Cc: Rusty Russell, Alan D. Brunelle, Rafael J. Wysocki,
Linux Kernel Mailing List, Kernel Testers List, Andrew Morton,
Arjan van de Ven, Ingo Molnar,
linux-embedded-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <alpine.LFD.1.10.0808261134530.3363-nfNrOhbfy2R17+2ddN/4kux8cNe9sq/dYPYVAmT7z5s@public.gmane.org>
On Tue, Aug 26, 2008 at 11:40:10AM -0700, Linus Torvalds wrote:
>
>
> On Tue, 26 Aug 2008, Adrian Bunk wrote:
> >
> > A debugging option (for better traces) to disallow gcc some inlining
> > might make sense (and might even make sense for distributions to
> > enable in their kernels), but when you go to use cases that require
> > really small kernels the cost is too high.
>
> You ignore the fact that it's really not just about debugging.
I had in mind that we anyway have to support it for tiny kernels.
I simply don't see that we add kconfig options for 5kB of code for
tiny kernels but remove something like this that can cause size
increases > 1%.
> Inlining really isn't the great tool some people think it is. Especially
> not since gcc stack allocation is so horrid that it won't re-use stack
> slots etc (which I don't disagree with per se - it's _hard_ to re-use
> stack slots while still allowing code scheduling).
gcc's stack allocation has become better
(that's why we disable unit-at-a-time only for gcc 3.4 on i386).
> NOTE! I also would never claim that _our_ choices of "inline" are all that
> great, and we've often inlined too much or not inlined things that really
> could be inlined. But at least when a developer says "inline" (or forgets
> to say it), we have somebody to blame. When the compiler does insane
> things that doesn't suit us, we're just screwed.
Most LOCs of the kernel are not written by people like you or Al Viro or
David Miller, and the average kernel developer is unlikely to do it as
good as gcc.
For the average driver the choice is realistically between
"inline's randomly sprinkled across the driver" and
"no inline's, leave it to gcc".
And code evolves during the years from tiny with 1 caller to huge with
many callers.
BTW:
I just ran checkstack on a (roughly) allyesconfig kernel, and we have a
new driver that allocates "unsigned char recvbuf[1500];" on the stack...
> > But if you don't trust gcc's inlining you should revert
> > commit 3f9b5cc018566ad9562df0648395649aebdbc5e0 that increases gcc's
> > freedom regarding what to inline in 2.6.27
>
> Actually, that just allows gcc to _not_ inline. Which is probably ok.
>
> (Well, it would be ok if gcc did it well enough, it obviously has some
> problems at times).
With the "gcc inline's static functions" you complain about we have
4-5 years of experience.
Suddenly allowing 4 release series of gcc to ignore any inline's is a
completely new area for us. I'd generally agree with giving gcc more
freedom here, but I'd rather do it right by removing tons of wrong
inline's than doing one global change hoping that it will make things
better.
And whether the "optimized inlining" actually makes the kernel bigger or
smaller depends in my experience on the .config and the gcc version.
> Linus
cu
Adrian
[1] there are some rare exceptions
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Linus Torvalds @ 2008-08-26 20:41 UTC (permalink / raw)
To: Adrian Bunk
Cc: Rusty Russell, Alan D. Brunelle, Rafael J. Wysocki,
Linux Kernel Mailing List, Kernel Testers List, Andrew Morton,
Arjan van de Ven, Ingo Molnar, linux-embedded
In-Reply-To: <20080826202134.GA11734@cs181140183.pp.htv.fi>
On Tue, 26 Aug 2008, Adrian Bunk wrote:
>
> I had in mind that we anyway have to support it for tiny kernels.
I actually don't think that is true.
If we really were to decide to be stricter about it, and it makes a big
size difference, we can probably also add a tool to warn about functions
that really should be inline.
> > Inlining really isn't the great tool some people think it is. Especially
> > not since gcc stack allocation is so horrid that it won't re-use stack
> > slots etc (which I don't disagree with per se - it's _hard_ to re-use
> > stack slots while still allowing code scheduling).
>
> gcc's stack allocation has become better
> (that's why we disable unit-at-a-time only for gcc 3.4 on i386).
I agree that it has become better. But it still absolutely *sucks*.
For example, see the patch I just posted about e1000 stack usage. Even
though the variables were all in completely separate scopes, they all got
individual space on the stack over the whole lifetime of the function,
causing an explosion of stack-space. As such, gcc used 500 bytes too much
of stack, just because it didn't re-use the stackspace.
That was with gcc-4.3.0, and no, there were hardly any inlining issues
involevd, although it is true that inlining actually did make it slightly
worse in that case too (but since it was essentially a leaf function, that
had little real life impact, since there were no deep callchains below it
to care).
So the fact is, "better" simply is not "good enough". We still need to do
a lot of optimizations _manually_, because gcc cannot see that it can
re-use the stack-slots.
And sometimes those "optimizations" are actually performance
pessimizations, because in order to make gcc not use all the stack at the
same time, you simply have to break things out and force-disable inlining.
> Most LOCs of the kernel are not written by people like you or Al Viro or
> David Miller, and the average kernel developer is unlikely to do it as
> good as gcc.
Sure. But we do have tools. We do have checkstack.pl, it's just that it
hasn't been an issue in a long time, so I suspect many people didn't even
_realize_ we have it, and I certainly can attest to the fact that even
people who remember it - like me - don't actually tend to run it all that
often.
> For the average driver the choice is realistically between
> "inline's randomly sprinkled across the driver" and
> "no inline's, leave it to gcc".
And neither is likely to be a big problem.
> BTW:
> I just ran checkstack on a (roughly) allyesconfig kernel, and we have a
> new driver that allocates "unsigned char recvbuf[1500];" on the stack...
Yeah, it's _way_ too easy to do bad things.
> With the "gcc inline's static functions" you complain about we have
> 4-5 years of experience.
Sure. And most of it isn't all that great.
But I do agree that lettign gcc make more decisions is _dangerous_.
However, in this case, at least, the decisions it makes would at least
make for less inlining, and thus less stack space explosion.
Linus
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Adrian Bunk @ 2008-08-26 20:59 UTC (permalink / raw)
To: Linus Torvalds
Cc: Rusty Russell, Alan D. Brunelle, Rafael J. Wysocki,
Linux Kernel Mailing List, Kernel Testers List, Andrew Morton,
Arjan van de Ven, Ingo Molnar,
linux-embedded-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <alpine.LFD.1.10.0808261144510.3363-nfNrOhbfy2R17+2ddN/4kux8cNe9sq/dYPYVAmT7z5s@public.gmane.org>
On Tue, Aug 26, 2008 at 11:47:01AM -0700, Linus Torvalds wrote:
>
>
> On Tue, 26 Aug 2008, Adrian Bunk wrote:
> >
> > I added "-fno-inline-functions-called-once -fno-early-inlining" to
> > KBUILD_CFLAGS, and (with gcc 4.3) that increased the size of my kernel
> > image by 2%.
>
> Btw, did you check with just "-fno-inline-functions-called-once"?
>
> The -fearly-inlining decisions _should_ be mostly right. If gcc sees early
> that a function is so small (even without any constant propagation etc)
> that it can be inlined, it's probably right.
>
> The inline-functions-called-once thing is what causes even big functions
> to be inlined, and that's where you find the big downsides too (eg the
> stack usage).
-fno-inline-functions-called-once alone costs me nearly 1% in code size.
And I'd expect it to become more with "-fwhole-program --combine".
If you think we have too many stacksize problems I'd suggest to consider
removing the choice of 4k stacks on i386, sh and m68knommu instead of
using -fno-inline-functions-called-once:
Now that 32bit x86 is no longer used for extreme highend configurations
the only serious usecase for 4k stacks are AFAIK space savings on
embedded archs.
4k stacks have caused us much pain [1], and the cases where gcc inlined
too much were the easy ones.
I'm not saying that I'd like removing the choice of 4k stacks, but if we
want to reduce the number of stack related problems that's IMHO the
better alternative.
> Linus
cu
Adrian
[1] AFAIR some callpaths in the kernel are still too big
BTW: In case anyone wonders about why I suggest removing 4k stacks:
My position is that 4k stacks should either be enabled
unconditionally or no longer offered at all.
And if we remove 4k stacks from 32bit x86 it's no longer
realistically maintainable for other architectures.
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Linus Torvalds @ 2008-08-26 21:04 UTC (permalink / raw)
To: Adrian Bunk
Cc: Rusty Russell, Alan D. Brunelle, Rafael J. Wysocki,
Linux Kernel Mailing List, Kernel Testers List, Andrew Morton,
Arjan van de Ven, Ingo Molnar, linux-embedded
In-Reply-To: <20080826205916.GB11734@cs181140183.pp.htv.fi>
On Tue, 26 Aug 2008, Adrian Bunk wrote:
>
> If you think we have too many stacksize problems I'd suggest to consider
> removing the choice of 4k stacks on i386, sh and m68knommu instead of
> using -fno-inline-functions-called-once:
Don't be silly. That makes the problem _worse_.
We're much better off with a 1% code-size reduction than forcing big
stacks on people. The 4kB stack option is also a good way of saying "if it
works with this, then 8kB is certainly safe".
And embedded people (the ones that might care about 1% code size) are the
ones that would also want smaller stacks even more!
Linus
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Parag Warudkar @ 2008-08-26 22:54 UTC (permalink / raw)
To: Linus Torvalds
Cc: Adrian Bunk, Rusty Russell, Alan D. Brunelle, Rafael J. Wysocki,
Linux Kernel Mailing List, Kernel Testers List, Andrew Morton,
Arjan van de Ven, Ingo Molnar,
linux-embedded-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <alpine.LFD.1.10.0808261403360.3363-nfNrOhbfy2R17+2ddN/4kux8cNe9sq/dYPYVAmT7z5s@public.gmane.org>
On Tue, Aug 26, 2008 at 5:04 PM, Linus Torvalds
<torvalds-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org> wrote:
> And embedded people (the ones that might care about 1% code size) are the
> ones that would also want smaller stacks even more!
This is something I never understood - embedded devices are not going
to run more than a few processes and 4K*(Few Processes)
IMHO is not worth a saving now a days even in embedded world given
falling memory prices. Or do I misunderstand?
Parag
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: David VomLehn @ 2008-08-26 23:00 UTC (permalink / raw)
To: Parag Warudkar
Cc: Linus Torvalds, Adrian Bunk, Rusty Russell, Alan D. Brunelle,
Rafael J. Wysocki, Linux Kernel Mailing List, Kernel Testers List,
Andrew Morton, Arjan van de Ven, Ingo Molnar,
linux-embedded-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <f7848160808261554j2f4eaaa6i1ee8801ae75ca7bf-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
Parag Warudkar wrote:
> On Tue, Aug 26, 2008 at 5:04 PM, Linus Torvalds
> <torvalds-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org> wrote:
>
>> And embedded people (the ones that might care about 1% code size) are the
>> ones that would also want smaller stacks even more!
>
> This is something I never understood - embedded devices are not going
> to run more than a few processes and 4K*(Few Processes)
> IMHO is not worth a saving now a days even in embedded world given
> falling memory prices. Or do I misunderstand?
Embedded applications span a huge range of sizes, from the very small devices to
which you refer, to quite complex devices. The cable settop boxes we develop have
over a hundred interrupt sources, typically run 250-300 threads, and have 192+
MiB of memory. For all that, we are very cost sensitive and are under constant
pressure to come up with reliable ways to save memory.
> Parag
--
David VomLehn
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Adrian Bunk @ 2008-08-26 23:24 UTC (permalink / raw)
To: Linus Torvalds
Cc: Rusty Russell, Alan D. Brunelle, Rafael J. Wysocki,
Linux Kernel Mailing List, Kernel Testers List, Andrew Morton,
Arjan van de Ven, Ingo Molnar,
linux-embedded-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <alpine.LFD.1.10.0808261403360.3363-nfNrOhbfy2R17+2ddN/4kux8cNe9sq/dYPYVAmT7z5s@public.gmane.org>
On Tue, Aug 26, 2008 at 02:04:57PM -0700, Linus Torvalds wrote:
>
>
> On Tue, 26 Aug 2008, Adrian Bunk wrote:
> >
> > If you think we have too many stacksize problems I'd suggest to consider
> > removing the choice of 4k stacks on i386, sh and m68knommu instead of
> > using -fno-inline-functions-called-once:
>
> Don't be silly. That makes the problem _worse_.
>
> We're much better off with a 1% code-size reduction than forcing big
> stacks on people. The 4kB stack option is also a good way of saying "if it
> works with this, then 8kB is certainly safe".
>...
You implicitely assume both would solve the same problem.
While 4kB stacks are something we anyway never got 100% working, the
cases where gcc inlining functions causes a critical increase in stack
usage are usually not that hard to find, and once found the fix is
trivial.
We should anyway monitor stack usages better since we have frequent
programming errors in this area, and problems caused by gcc can this
way be detected en passant.
You have a good point that aiming at 4kB makes 8kB a very safe choice.
But I do not think the problem you'd solve with
-fno-inline-functions-called-once is big enough to warrant the size
increase it causes.
> Linus
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Adrian Bunk @ 2008-08-26 23:45 UTC (permalink / raw)
To: David VomLehn
Cc: Parag Warudkar, Linus Torvalds, Adrian Bunk, Rusty Russell,
Alan D. Brunelle, Rafael J. Wysocki, Linux Kernel Mailing List,
Kernel Testers List, Andrew Morton, Arjan van de Ven, Ingo Molnar,
linux-embedded
In-Reply-To: <48B48B11.8050000@cisco.com>
On Tue, Aug 26, 2008 at 04:00:33PM -0700, David VomLehn wrote:
> Parag Warudkar wrote:
>> On Tue, Aug 26, 2008 at 5:04 PM, Linus Torvalds
>> <torvalds@linux-foundation.org> wrote:
>>
>>> And embedded people (the ones that might care about 1% code size) are the
>>> ones that would also want smaller stacks even more!
>>
>> This is something I never understood - embedded devices are not going
>> to run more than a few processes and 4K*(Few Processes)
>> IMHO is not worth a saving now a days even in embedded world given
>> falling memory prices. Or do I misunderstand?
>
> Embedded applications span a huge range of sizes, from the very small
> devices to which you refer, to quite complex devices. The cable settop
> boxes we develop have over a hundred interrupt sources, typically run
> 250-300 threads, and have 192+ MiB of memory. For all that, we are very
> cost sensitive and are under constant pressure to come up with reliable
> ways to save memory.
As you say correctly the term "embedded" gets used for many different
devices.
And if you have 192+ MiB of memory you have so much that all these
kernel size discussions don't really matter.
> David VomLehn
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Linus Torvalds @ 2008-08-26 23:47 UTC (permalink / raw)
To: Parag Warudkar
Cc: Adrian Bunk, Rusty Russell, Alan D. Brunelle, Rafael J. Wysocki,
Linux Kernel Mailing List, Kernel Testers List, Andrew Morton,
Arjan van de Ven, Ingo Molnar, linux-embedded
In-Reply-To: <f7848160808261554j2f4eaaa6i1ee8801ae75ca7bf@mail.gmail.com>
On Tue, 26 Aug 2008, Parag Warudkar wrote:
>
> This is something I never understood - embedded devices are not going
> to run more than a few processes and 4K*(Few Processes)
> IMHO is not worth a saving now a days even in embedded world given
> falling memory prices. Or do I misunderstand?
Well, by that argument, 1% of kernel size doesn't matter either..
1% of a kernel for an embedded device is roughly 10-30kB or so depending
on how small you make the configuration.
If that matters, then so should the difference of 3-8 processes' kernel
stack usage when you have a 4k/8k stack choice.
And they _all_ will have at least 3-8 processes on them. Even the simplest
ones will tend to have many more.
Linus
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Linus Torvalds @ 2008-08-26 23:51 UTC (permalink / raw)
To: Adrian Bunk
Cc: Rusty Russell, Alan D. Brunelle, Rafael J. Wysocki,
Linux Kernel Mailing List, Kernel Testers List, Andrew Morton,
Arjan van de Ven, Ingo Molnar,
linux-embedded-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20080826232411.GC11734-re2QNgSbS3j4D6uPqz5PAwR5/fbUUdgG@public.gmane.org>
On Wed, 27 Aug 2008, Adrian Bunk wrote:
> >
> > We're much better off with a 1% code-size reduction than forcing big
> > stacks on people. The 4kB stack option is also a good way of saying "if it
> > works with this, then 8kB is certainly safe".
>
> You implicitely assume both would solve the same problem.
I'm just saying that your logic doesn't hold water.
If we can save kernel stack usage, then a 1% increase in kernel size is
more than worth it.
> While 4kB stacks are something we anyway never got 100% working
What? Don't be silly.
Linux _historically_ always used 4kB stacks.
No, they are likely not usable on x86-64, but dammit, they should be more
than usable on x86-32 still.
> But I do not think the problem you'd solve with
> -fno-inline-functions-called-once is big enough to warrant the size
> increase it causes.
You continually try to see the inlining as a single solution to one
problem (debuggability, stack, whatever).
The biggest problem with gcc inlining has always been that it has been
_unpredictable_. It causes problems in many different ways. It has caused
stability issues due to gcc versions doing random things. It causes the
stack expansion. It makes stack traces harder for debugging, etc.
If it was any one thing, I wouldn't care. But it's exactly the fact that
it causes all these problems in different areas.
Linus
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Adrian Bunk @ 2008-08-27 0:23 UTC (permalink / raw)
To: Linus Torvalds
Cc: Rusty Russell, Alan D. Brunelle, Rafael J. Wysocki,
Linux Kernel Mailing List, Kernel Testers List, Andrew Morton,
Arjan van de Ven, Ingo Molnar,
linux-embedded-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <alpine.LFD.1.10.0808261648140.3363-nfNrOhbfy2R17+2ddN/4kux8cNe9sq/dYPYVAmT7z5s@public.gmane.org>
On Tue, Aug 26, 2008 at 04:51:52PM -0700, Linus Torvalds wrote:
>
>
> On Wed, 27 Aug 2008, Adrian Bunk wrote:
> > >
> > > We're much better off with a 1% code-size reduction than forcing big
> > > stacks on people. The 4kB stack option is also a good way of saying "if it
> > > works with this, then 8kB is certainly safe".
> >
> > You implicitely assume both would solve the same problem.
>
> I'm just saying that your logic doesn't hold water.
>
> If we can save kernel stack usage, then a 1% increase in kernel size is
> more than worth it.
From some tests the size increase seems to become bigger for smaller
kernels, but I don't have any really good data.
An interesting question is why most of our architectures for embedded
devices only offer bigger stacks:
The only architectures offering a 4kB stacks option are:
- m68knommu
- sh
- 32bit x86
The following architectures that are used in embedded devices
always use 8kB stacks (or bigger) in your tree:
- arm
- avr32
- blackfin
- cris
- frv
- h8300
- m32r
- m68k
- mips
- mn10300 (has an #ifdef CONFIG_4KSTACKS but no kconfig option)
- powerpc
- xtensa
> > While 4kB stacks are something we anyway never got 100% working
>
> What? Don't be silly.
>
> Linux _historically_ always used 4kB stacks.
>
> No, they are likely not usable on x86-64, but dammit, they should be more
> than usable on x86-32 still.
When did we get callpaths like like nfs+xfs+md+scsi reliably
working with 4kB stacks on x86-32?
>...
> Linus
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Linus Torvalds @ 2008-08-27 0:28 UTC (permalink / raw)
To: Adrian Bunk
Cc: Rusty Russell, Alan D. Brunelle, Rafael J. Wysocki,
Linux Kernel Mailing List, Kernel Testers List, Andrew Morton,
Arjan van de Ven, Ingo Molnar, linux-embedded
In-Reply-To: <20080827002316.GE11734@cs181140183.pp.htv.fi>
On Wed, 27 Aug 2008, Adrian Bunk wrote:
>
> When did we get callpaths like like nfs+xfs+md+scsi reliably
> working with 4kB stacks on x86-32?
XFS may never have been usable, but the rest, sure.
And you seem to be making this whole argument an excuse to SUCK, adn an
excuse to let gcc crap even more on our stack space.
Why?
Why aren't you saying that we should be able to do better? Instead, you
seem to asking us to do even worse than we do now?
Linus
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Greg Ungerer @ 2008-08-27 0:53 UTC (permalink / raw)
To: Linus Torvalds
Cc: Parag Warudkar, Adrian Bunk, Rusty Russell, Alan D. Brunelle,
Rafael J. Wysocki, Linux Kernel Mailing List, Kernel Testers List,
Andrew Morton, Arjan van de Ven, Ingo Molnar,
linux-embedded-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <alpine.LFD.1.10.0808261644260.3363-nfNrOhbfy2R17+2ddN/4kux8cNe9sq/dYPYVAmT7z5s@public.gmane.org>
Linus Torvalds wrote:
> On Tue, 26 Aug 2008, Parag Warudkar wrote:
>> This is something I never understood - embedded devices are not going
>> to run more than a few processes and 4K*(Few Processes)
>> IMHO is not worth a saving now a days even in embedded world given
>> falling memory prices. Or do I misunderstand?
>
> Well, by that argument, 1% of kernel size doesn't matter either..
>
> 1% of a kernel for an embedded device is roughly 10-30kB or so depending
> on how small you make the configuration.
>
> If that matters, then so should the difference of 3-8 processes' kernel
> stack usage when you have a 4k/8k stack choice.
>
> And they _all_ will have at least 3-8 processes on them. Even the simplest
> ones will tend to have many more.
I have some simple devices (network access/routers) with 8MB of RAM,
at power up not really being configured to do anything running 25
processes. (Heck there is over 10 kernel processes running!). Configure
some interfaces and services and that will easily push past 40.
I'd be happy with a 160k saving :-)
The init memory being freed at the end of the kernel boot is 88k,
4k stacks could save more than that.
Regards
Greg
------------------------------------------------------------------------
Greg Ungerer -- Chief Software Dude EMAIL: gerg-XXXsiaCtIV5Wk0Htik3J/w@public.gmane.org
Secure Computing Corporation PHONE: +61 7 3435 2888
825 Stanley St, FAX: +61 7 3891 3630
Woolloongabba, QLD, 4102, Australia WEB: http://www.SnapGear.com
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Parag Warudkar @ 2008-08-27 0:58 UTC (permalink / raw)
To: Linus Torvalds
Cc: Adrian Bunk, Rusty Russell, Alan D. Brunelle, Rafael J. Wysocki,
Linux Kernel Mailing List, Kernel Testers List, Andrew Morton,
Arjan van de Ven, Ingo Molnar,
linux-embedded-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <alpine.LFD.1.10.0808261644260.3363-nfNrOhbfy2R17+2ddN/4kux8cNe9sq/dYPYVAmT7z5s@public.gmane.org>
On Tue, Aug 26, 2008 at 7:47 PM, Linus Torvalds
<torvalds-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org> wrote:
> If that matters, then so should the difference of 3-8 processes' kernel
> stack usage when you have a 4k/8k stack choice.
The savings part -financial ones- are not always realizable with the
way memory is priced/sized/fitted.
Savings in few Mb of Kernel stack are not necessarily going to allow
getting rid of a single memory chip of 64M or so.
Either that or embedded manufacturing/configurations are different
than the desktop world.
(If my device has 2 memory slots and my user space requires 100Mb
including kernel memory - I anyways have to put in 64Mx2 there to take
advantage of mass manufactured, general purpose memory - so no big
deal if I saved 1.2Mb in Kernel stack or not. And savings of 64Mb
Kernel memory are not feasible anyways to allow user space to work
with 64Mb.)
On the other hand reducing user space memory usage on those devices
(not counting savings from kernel stack size) is a way more attractive
option.
And although you said in your later reply that Linux x86 with 4K
stacks should be more than usable - my experiences running a untainted
desktop/file server with 4K stack have been always disastrous XFS or
not. It _might_ work for some well defined workloads but you would
not want to risk 4K stacks otherwise.
I understand the having 4K stack option as a non-default for very
specific workloads is a good idea but apart from that I think no one
else seems to bother with reducing stack sizes (by no one I mean other
OSes.)
Parag
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Parag Warudkar @ 2008-08-27 1:08 UTC (permalink / raw)
To: Greg Ungerer
Cc: Linus Torvalds, Adrian Bunk, Rusty Russell, Alan D. Brunelle,
Rafael J. Wysocki, Linux Kernel Mailing List, Kernel Testers List,
Andrew Morton, Arjan van de Ven, Ingo Molnar, linux-embedded
In-Reply-To: <48B4A577.3020303@snapgear.com>
On Tue, Aug 26, 2008 at 8:53 PM, Greg Ungerer <gerg@snapgear.com> wrote:
> I have some simple devices (network access/routers) with 8MB of RAM,
> at power up not really being configured to do anything running 25
> processes. (Heck there is over 10 kernel processes running!). Configure
> some interfaces and services and that will easily push past 40.
> I'd be happy with a 160k saving :-)
>
So you really need to run all 25 processes on that 8Mb box?
(For reference even the NGW100 development board comes with 16Mb RAM).
Even if you do need those all 25 processes on the 8Mb box, fixing the
memory usage of those user space hogs is lot better than trying to
save 160Kb in kernel stacks.
Last I looked, user space wasn't particularly frugal with memory usage.
Parag
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Greg Ungerer @ 2008-08-27 1:31 UTC (permalink / raw)
To: Parag Warudkar
Cc: Linus Torvalds, Adrian Bunk, Rusty Russell, Alan D. Brunelle,
Rafael J. Wysocki, Linux Kernel Mailing List, Kernel Testers List,
Andrew Morton, Arjan van de Ven, Ingo Molnar, linux-embedded
In-Reply-To: <f7848160808261808p3391c99ekfa1d8991bbf73c10@mail.gmail.com>
Parag Warudkar wrote:
> On Tue, Aug 26, 2008 at 8:53 PM, Greg Ungerer <gerg@snapgear.com> wrote:
>
>> I have some simple devices (network access/routers) with 8MB of RAM,
>> at power up not really being configured to do anything running 25
>> processes. (Heck there is over 10 kernel processes running!). Configure
>> some interfaces and services and that will easily push past 40.
>> I'd be happy with a 160k saving :-)
>>
>
> So you really need to run all 25 processes on that 8Mb box?
Yes, of course. Considerable effort has been put into running
a minimal set of processes (that still for fills the required function
set of this device).
> (For reference even the NGW100 development board comes with 16Mb RAM).
Lots of development boards are fitted with lots of RAM.
And the pressure will still be on in _real_ products to reduce
the RAM footprint as much as possible. There are exceptions but
generally less is cheaper. Simple economics really.
> Even if you do need those all 25 processes on the 8Mb box, fixing the
> memory usage of those user space hogs is lot better than trying to
> save 160Kb in kernel stacks.
Yep, been done too. You don't squeeze a lot into these smaller
devices without looking at everything in it.
> Last I looked, user space wasn't particularly frugal with memory usage.
Then you haven't looked in the right places :-)
There are plenty of choices for making things small in user space.
Simple stuff like using uClibc, busybox, etc.
In this specific example things like /bin/init is 10k, /bin/inetd
is 10k, /bin/crond is 11k, etc. (Ofcourse it is a shared uClibc setup,
uClibc is ~300k). And XIP can help out here too.
Regards
Greg
------------------------------------------------------------------------
Greg Ungerer -- Chief Software Dude EMAIL: gerg@snapgear.com
Secure Computing Corporation PHONE: +61 7 3435 2888
825 Stanley St, FAX: +61 7 3891 3630
Woolloongabba, QLD, 4102, Australia WEB: http://www.SnapGear.com
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Linus Torvalds @ 2008-08-27 1:49 UTC (permalink / raw)
To: Parag Warudkar
Cc: Adrian Bunk, Rusty Russell, Alan D. Brunelle, Rafael J. Wysocki,
Linux Kernel Mailing List, Kernel Testers List, Andrew Morton,
Arjan van de Ven, Ingo Molnar,
linux-embedded-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <f7848160808261758q7b84aab1m188c1ebb59304818-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
On Tue, 26 Aug 2008, Parag Warudkar wrote:
>
> And although you said in your later reply that Linux x86 with 4K
> stacks should be more than usable - my experiences running a untainted
> desktop/file server with 4K stack have been always disastrous XFS or
> not. It _might_ work for some well defined workloads but you would
> not want to risk 4K stacks otherwise.
Umm. How long?
4kB used to be the _only_ choice. And no, there weren't even irq stacks.
So that 4kB was not just the whole kernel call-chain, it was also all the
irq nesting above it.
And yes, we've gotten much worse over time, and no, I can't really suggest
going back to that in general. The code bloat has certainly been
accompanied by a stack bloat too.
But part of it is definitely gcc. Some versions of gcc used to be
absolutely _horrid_ when it came to stack usage, especially with some
flags, and especially with the crazy inlining that module-at-a-time
caused.
But I'd be really happy if some embedded people tried to take some of that
bloat back, and aim for 4kB stacks. Because it's definitely not
unrealistic. At least it _shouldn't_ be. And a lot of the cases of us
having structures on the stack is actually not worth it, and tends to be
about being lazy rather than anything else.
Linus
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Parag Warudkar @ 2008-08-27 2:16 UTC (permalink / raw)
To: Greg Ungerer
Cc: Linus Torvalds, Adrian Bunk, Rusty Russell, Alan D. Brunelle,
Rafael J. Wysocki, Linux Kernel Mailing List, Kernel Testers List,
Andrew Morton, Arjan van de Ven, Ingo Molnar,
linux-embedded-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <48B4AE68.4040205-XXXsiaCtIV5Wk0Htik3J/w@public.gmane.org>
On Tue, Aug 26, 2008 at 9:31 PM, Greg Ungerer <gerg-XXXsiaCtIV5Wk0Htik3J/w@public.gmane.org> wrote:
>
> And the pressure will still be on in _real_ products to reduce
> the RAM footprint as much as possible. There are exceptions but
> generally less is cheaper. Simple economics really.
Well, sure - but the industry as a whole seems to have gone the other
way - do more with more at the similar or lower price points!
By that definition of less is better we should try and make the kernel
memory pageable (or has someone already done that?) - Windows does it,
by default ;)
Parag
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Parag Warudkar @ 2008-08-27 2:36 UTC (permalink / raw)
To: Linus Torvalds
Cc: Adrian Bunk, Rusty Russell, Alan D. Brunelle, Rafael J. Wysocki,
Linux Kernel Mailing List, Kernel Testers List, Andrew Morton,
Arjan van de Ven, Ingo Molnar,
linux-embedded-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <alpine.LFD.1.10.0808261837530.3363-nfNrOhbfy2R17+2ddN/4kux8cNe9sq/dYPYVAmT7z5s@public.gmane.org>
On Tue, Aug 26, 2008 at 9:49 PM, Linus Torvalds
<torvalds-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org> wrote:
>
>
> On Tue, 26 Aug 2008, Parag Warudkar wrote:
>>
>> And although you said in your later reply that Linux x86 with 4K
>> stacks should be more than usable - my experiences running a untainted
>> desktop/file server with 4K stack have been always disastrous XFS or
>> not. It _might_ work for some well defined workloads but you would
>> not want to risk 4K stacks otherwise.
>
> Umm. How long?
>
IIRC the last I tried 4K stacks with x86 was on 2.6.21 - Fedora 7
kernel, around June 07 time frame.
The oops included a ugly and long call trace that I still remember.
> And a lot of the cases of us
> having structures on the stack is actually not worth it, and tends to be
> about being lazy rather than anything else.
What about deep call chains? The problem with the uptake of 4K stacks
seems to be that is not reliably provable that it will work under all
circumstances.
Parag
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Linus Torvalds @ 2008-08-27 2:52 UTC (permalink / raw)
To: Parag Warudkar
Cc: Adrian Bunk, Rusty Russell, Alan D. Brunelle, Rafael J. Wysocki,
Linux Kernel Mailing List, Kernel Testers List, Andrew Morton,
Arjan van de Ven, Ingo Molnar,
linux-embedded-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <f7848160808261936m18c69dc0r26f41850efae4b91-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
On Tue, 26 Aug 2008, Parag Warudkar wrote:
>
> What about deep call chains? The problem with the uptake of 4K stacks
> seems to be that is not reliably provable that it will work under all
> circumstances.
Umm. Neither is 8k stacks. Nobody "proved" anything.
But yes, some subsystems have insanely deep call chains. And yes, things
like the VFS recursion (for symlinks) makes that deeper yet for
filesystems, although only on the lookup path. And that is exactly the
kind of thing that can exacerbate the problem of the compiler artificially
making for a bigger stack footprint of a function (*).
For things like the VFS layer, right now we allow a nesting level of 8, I
think. If I remember correctly, it was 5 historically. Part of raising
that depth, though, was that we actually moved the recursive part into
fs/namei.c, and the nesting stack-depth was something pretty damn small
when the filesystem used "follow_link" properly and let the VFS do it for
it (ie the callchain to actually look up the link could be deep, but it
would not recurse back, and instead just return a pointer, so that the
actual _recursive_ part was just __do_follow_link() and is just a few
words on the stack).
So yes, we do have some deep callchains, but they tend to be pretty well
managed for _good_ code. The problems tend to be the areas with lots of
indirection layers, and yeah, XFS, MD and ACPI all have those kinds of
things.
In an embdedded world, many of those should be a non-issue, though.
Linus
(*) ie the function that _is_ on the deep chain doesn't actually need much
of a stack footprint at all itself, but it may call a helper function that
is _not_ in the deep chain, and if it gets inlined it may give its
excessive stack footprint to the deep chain - and this is _exactly_ the
problem that happened with inlining "load_module()".
^ permalink raw reply
* "kernel access of bad area" while accessing flash
From: Fundu @ 2008-08-27 3:13 UTC (permalink / raw)
To: linux embedded
I have a PPC440GX based board.
Do I need to reinitialize Peripheral Bank Configuration Registers (EBC0_B0CR-EBC0_B7CR) from the kernel ?
I'm quite sure that we don't, but just want to make sure. Because the problem
that i'm seeing strongly suggest that it's probably not accessible.
Here's the problem, i'm getting "kernel access of bad
area" when i erase any of the upper 64mb out of 128mb flash region, surprisingly i can easily erase (all of the flash) from u-boot. which makes me ask the earlier question.
any insights welcome.
thanks !
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Paul Mackerras @ 2008-08-27 6:01 UTC (permalink / raw)
To: Linus Torvalds
Cc: Parag Warudkar, Adrian Bunk, Rusty Russell, Alan D. Brunelle,
Rafael J. Wysocki, Linux Kernel Mailing List, Kernel Testers List,
Andrew Morton, Arjan van de Ven, Ingo Molnar,
linux-embedded-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <alpine.LFD.1.10.0808261837530.3363-nfNrOhbfy2R17+2ddN/4kux8cNe9sq/dYPYVAmT7z5s@public.gmane.org>
Linus Torvalds writes:
> 4kB used to be the _only_ choice. And no, there weren't even irq stacks.
> So that 4kB was not just the whole kernel call-chain, it was also all the
> irq nesting above it.
I think your memory is failing you. In 2.4 and earlier, the kernel
stack was 8kB minus the size of the task_struct, which sat at the
start of the 8kB. For instance, from include/asm-i386/processor.h for
2.4.29:
#define THREAD_SIZE (2*PAGE_SIZE)
#define alloc_task_struct() ((struct task_struct *) __get_free_pages(GFP_KERNEL,1))
#define free_task_struct(p) free_pages((unsigned long) (p), 1)
Paul.
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Alan Cox @ 2008-08-27 8:25 UTC (permalink / raw)
To: Adrian Bunk
Cc: Linus Torvalds, Rusty Russell, Alan D. Brunelle,
Rafael J. Wysocki, Linux Kernel Mailing List, Kernel Testers List,
Andrew Morton, Arjan van de Ven, Ingo Molnar,
linux-embedded-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20080826232411.GC11734-re2QNgSbS3j4D6uPqz5PAwR5/fbUUdgG@public.gmane.org>
> You have a good point that aiming at 4kB makes 8kB a very safe choice.
Not really no - we use separate IRQ stacks in 4K but not 8K mode on
x86-32. That means you've actually got no more space if you are unlucky
with the timing of events. The 8K mode is merely harder to debug.
If 4K stacks really are not safe then x86-32 really really needs to
switch to using IRQ stacks in 8K stack mode as well.
Alan
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Alan Cox @ 2008-08-27 8:32 UTC (permalink / raw)
To: Parag Warudkar
Cc: Linus Torvalds, Adrian Bunk, Rusty Russell, Alan D. Brunelle,
Rafael J. Wysocki, Linux Kernel Mailing List, Kernel Testers List,
Andrew Morton, Arjan van de Ven, Ingo Molnar, linux-embedded
In-Reply-To: <f7848160808261936m18c69dc0r26f41850efae4b91@mail.gmail.com>
> What about deep call chains? The problem with the uptake of 4K stacks
> seems to be that is not reliably provable that it will work under all
> circumstances.
On x86-32 with 8K stacks your IRQ paths share them so that is even harder
to prove (not that you can prove any of them) and the bugs are more
obscure and random.
^ permalink raw reply
* Re: [Bug #11342] Linux 2.6.27-rc3: kernel BUG at mm/vmalloc.c - bisected
From: Bernd Petrovitsch @ 2008-08-27 8:34 UTC (permalink / raw)
To: Parag Warudkar
Cc: Linus Torvalds, Adrian Bunk, Rusty Russell, Alan D. Brunelle,
Rafael J. Wysocki, Linux Kernel Mailing List, Kernel Testers List,
Andrew Morton, Arjan van de Ven, Ingo Molnar, linux-embedded
In-Reply-To: <f7848160808261554j2f4eaaa6i1ee8801ae75ca7bf@mail.gmail.com>
On Tue, 2008-08-26 at 18:54 -0400, Parag Warudkar wrote:
> On Tue, Aug 26, 2008 at 5:04 PM, Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
>
> > And embedded people (the ones that might care about 1% code size) are the
> > ones that would also want smaller stacks even more!
>
> This is something I never understood - embedded devices are not going
> to run more than a few processes and 4K*(Few Processes)
> IMHO is not worth a saving now a days even in embedded world given
> falling memory prices. Or do I misunderstand?
Falling prices are no reason to increase the amount of available RAM (or
other hardware).
Especially if you (intend to) build >1E5 devices - where every Euro
counts.
Bernd
--
Firmix Software GmbH http://www.firmix.at/
mobil: +43 664 4416156 fax: +43 1 7890849-55
Embedded Linux Development and Services
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox