Re: [kvmarm] [RFC PATCH 0/3] KVM: ARM: Get rid of hardcoded VGIC addresses

[Benjamin Herrenschmidt <benh@kernel.crashing.org>]

On Fri, 2012-10-26 at 12:17 +0100, Peter Maydell wrote:
> > Well, that's the thing, I haven't managed to figure that out so far,
> it
> > looks very x86-specific to me. To begin with there's no such thing
> as a
> > "GSI" in our world.
> 
> This was roughly the feeling I had looking at these APIs. There
> might be some underlying generic concept but there is a definite
> tendency for the surface representation to use x86 specific
> terminology to the extent that you can't tell whether an API
> is x86 specific or merely apparently so...

Right. Which is why I'm sure I'm actually missing something there :-)
And I'm hoping Paolo and Jan will help shed some light.

It might help if somebody could explain a bit more what a GSI is in x86
land and how it relates to the various APICs, along with what exactly
they mean by "routing" , ie. what are the different elements that get
associated. Basically, if somebody could describe how the x86 stuff
works, that might help.

>From my view of things, we have various "sources" of interrupts. On my
list are emulated device LSIs, emulated device MSIs, both in qemu, then
vhost and finally pass-through, I suppose on some platforms IPIs come in
as well though. Those "sources" need, one way or another, to hit a
source controller which will then itself, in a platform specific way,
shoot the interrupt to a presentation controller.

The routing between source and presentation controllers is fairly
platform specific as far as I can tell even within a given CPU family.
Ie the way an OpenPIC (aka MPIC, used on macs) does it is different than
the way the XICS system does it on pseries, and is different from most
embedded stuff (which typically doesn't have that source/presentation
distinction but just cascaded dumber PICs). The amount of
configurability, the type of configuration information etc... that
governs such a layout is also very specific to the platform and the type
of interrupt controller system used on it.

Remains the "routing" between source of "events" and actual "inputs" to
a source controller.

This too doesn't seem totally obvious to generalize. For example an
embedded platform with a bunch of cascaded dumb interrupt controllers
doesn't have a concept of a flat number space in HW, an interrupt
"input" to be identified properly, needs to identify the controller and
the interrupt within that controller. However, within KVM/qemu, it's
pretty easy to assign to each controller a number and by collating the
two, get some kind of flat space, though it's not arbitrary and the
routing is thus fairly constrained if not totally fixed.

In the pseries case, the global number is split in two bit fields, the
BUID identifying the specific source controller and the source within
that controller. Here too it's fairly fixed though. So the ioctl we use
to create a source controller in the kernel takes the BUID as an
argument, and from there the kernel will "find" the right source
controller based solely on the interrupt number.

So basically on one side we have a global interrupt number that
identifies an "input", I assume that's what x86 calls a GSI ?

Remains how to associate the various sources of interrupts to that
'global number'... and that is fairly specific to each source type isn't
it ?

In our current powerpc code, the emulated devices toggle the qirq which
ends up shooting an ioctl to set/reset or "message" (for MSIs) the
corresponding global interrupt. The mapping is established entirely
within qemu, we just tell the kernel to trigger a given interrupt.

We haven't really sorted vhost out yet so I'm not sure how that will
work out but the idea would be to have an ioctl to associate an eventfd
or whatever vhost uses as interrupt "outputs" with a global interrupt
number.

For pass-through, currently our VFIO is dumb, interrupts get to qemu
which then shoots them back to the kernel using the standard qirq stuff
used by emulated devices. Here I suppose we would want something similar
to vhost to associate the VFIO irq fd with a "global number".

Is that what the existing ioctl's provide ? Their semantics aren't
totally obvious to me.

Note that for pass-through at least, and possibly for vhost, we'd like
to actually totally bypass the irqfd & eventfd stuff for performance
reasons. At least for VFIO, if we are going to get the max performance
out of it, we need to take all generic code out of the picture. IE. If
the interrupts are routed to the physical CPU where the guest is
running, we want to be able to catch and distribute the interrupts to
the guest entirely within guest context, ie, with KVM arch specific low
level code that runs in "real mode" (ie MMU off) without context
switching the MMU back to the host, which on POWER is fairly costly.

That means that at least the association between a guest global
interrupt number and a host global interrupt number for pass-through
will be something that goes entirely through arch specific code path. We
might still be able to use generic APIs to establish it if they are
suitable though.

Cheers,
Ben.