Re: [PATCH -v7][RFC]: mutex: implement adaptive spinning

[Ingo Molnar <mingo@elte.hu>]

* Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Fri, 9 Jan 2009, H. Peter Anvin wrote:
> > 
> > I was thinking about experimenting with this, to see what level of 
> > upside it might add.  Ingo showed me numbers which indicate that a 
> > fairly significant fraction of the cases where removing inline helps 
> > is in .h files, which would require code movement to fix.  Hence to 
> > see if it can be automated.
> 
> We _definitely_ have too many inline functions in headers. They usually 
> start out small, and then they grow. And even after they've grown big, 
> it's usually not at all clear exactly where else they should go, so even 
> when you realize that "that shouldn't be inlined", moving them and 
> making them uninlined is not obvious.
> 
> And quite often, some of them go away - or at least shrink a lot - when 
> some config option or other isn't set. So sometimes it's an inline 
> because a certain class of people really want it inlined, simply because 
> for _them_ it makes sense, but when you enable debugging or something, 
> it absolutely explodes.

IMO it's all quite dynamic when it comes to inlining.

Beyond the .config variances (which alone is enough degrees of freedom to 
make this non-static, it at least is a complexity we can control in the 
kernel to a certain degree) it also depends on the platform, the CPU type, 
the compiler version - factors which we dont (and probably dont want to) 
control.

There's also the in-source variance of how many times an inline function 
is used within a .c file, and that factor is not easily tracked. If it's 
used once in a single .c file it should be inlined even if it's large. If 
it's used twice in a .c file it might be put out of line. Transition 
between those states is not obvious in all cases.

There's certainly clear-cut cases: the very small/constant ones that must 
be short and inlined in any environment, and the very large/complex ones 
that must not be inlined under any circumstance.

But there's a lot of shades of grey inbetween - and that's where the size 
wins come from. I'm not sure we can (or should) generally expect kernel 
coders to continuously maintain the 30,000+ inline attributes in the 
kernel that involve 100,000+ functions:

 - Nothing breaks if it's there, nothing breaks if it's not there.
   It's a completely opaque, transparent entity that never pushes itself 
   to the foreground of human attention.

 - It's so easy to add an inline function call site to a .c file without 
   noticing that it should not be inlined anymore.

 - It's so easy to _remove_ a usage site from a .c file without noticing
   that something should be inlined. I.e. local changes will have an 
   effect on the inline attribute _elsewhere_ - and this link is not 
   obvious and not tooled when editing the code.

 - The mapping from C statements to assembly can be non-obvious even to
   experienced developers. Data type details (signed/unsigned, width,
   etc.) can push an inline function over the (very hard to define)
   boundary.

I.e. IMO it's all very dynamic, it's opaque, it's not visualized and it's 
hard to track - so it's very fundamentally not for humans to maintain 
[except for the really clear-cut cases].

Add to that that in _theory_ the decision to inline or not is boringly 
mechanic and tools ought to be able to do a near-perfect job with it and 
just adopt to whatever environment the kernel is in at a given moment when 
it's built.

GCC limps along with its annoyingly mis-designed inlining heuristics, 
hopefully LLVC will become a real compiler that is aware of little details 
like instruction size and has a built-in assembler ...

So IMO all the basic psychological mechanics are missing from the picture 
that would result in really good, "self-maintained" inline attributes.

We can try to inject the requirement to have good inline attributes as an 
external rule, as a principle we want to see met - but we cannot expect it 
to be followed really in its current form, as it goes subtly against the 
human nature on various levels.

	Ingo