Re: x86/Voyager

[Ingo Molnar <mingo@elte.hu>]

* James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

> > > But no-one's yet made any argument for why it's a worthwhile thing 
> > > to be doing.
> > 
> > Because the sub-arch code is butt-ugly.
> > 
> > x86 subarchitectures are a hack that should never have gone upstream - 
> > and we are now reversing that braindamage, step by step. 
> > Subarchitectures are a compile-time "HAL", but a highly 
> > non-transparent one at that. They complicates the x86 architecture in 
> > a couple of key structures and very fundamentally so - and that 
> > results in continued complications in critical areas of the x86 code.
> 
> Right: it's a compile time HAL.  Your "cleanup" is to convert it to a 
> runtime one, which is a lot more complex because voyager still has to 
> influence the x86 path in those locations regardless: it's not an x86 PC 
> architecture and so is never going to be able to boot through an APIC/MP 
> table SMP boot sequence.
> 
> I'm not actually bothered by the complexity, though ... it would be cool 
> for me to have a kernel that boots on both voyager and a PC.  What 
> worries me is the cost (both in terms of execution time and maintenance 
> burden) this would impose on the standard PC path.

That cost is negligible. Voyager already uses smp_ops which solves most of 
the callbacks. The remaining non-slowpath (non-init, non-shutdown, etc.) 
bits are, roughly:

   flush_tlb_[all|current_task|mm|page]()
   smp_apic_timer_interrupt

Which can either reuse PARAVIRT with a Voyager-specific custom template 
for these methods, or, if that's easier, we can add x86_quirk handlers as 
well.

There is near zero overhead for normal PCs, ~5 functions will have:

    if (unlikely(x86_quirks.smp_apic_timer_interrupt)) {
            ... quirk-path ...
    }

type of constructs.

> The other thing I will point out is that if you think a runtime HAL 
> simplification could be applied to the current kernel, a compile time 
> HAL could equally well be done.

There are numerous advantages. Just a few of them, from the top of my 
head:

 - The point is to not split the build space on such a fundamental level - 
   testing is way too complex already.

 - Runtime quirks tend to be tested far more than build-time quirks: for 
   example because a test kernel can just include all the runtime quirks 
   all at once - while it cannot possibly include all the
   subarchitectures.

 - It is far more apparent from the source code what happens if quirks are 
   out and visible. A compile-time 'HAL' is far less transparent: it is 
   very easy to miss that a function is present in multiple copies and 
   behaves in different ways, dependent on which subarch we are building 
   for. Developers tend to concentrate on a single piece of code, so
   consolidating code flows is a development advantage.

 - Runtime quirks tend to be much more usable. A distro kernel can include
   all the quirks with near zero impact. Now in the specific case of
   x86/Voyager this is probably not a big factor: as you seem to be owning 
   the last two (one?) working Voyager box in existence that runs
   development kernels? But it is a factor in the general push to
   eliminate subarchitectures.

(For similarly good reasons most of the hw quirks in the normal Linux 
driver space are done via runtime constructs and not via build-time 
constructs.)

Anyway, none of this is really new. We eliminated the ES7000, the VISWS 
and the RDC321X subarchitectures already - Voyager is the holdout.

	Ingo