[Qemu-devel] passing interrupts from QEMU to KVM

[Qemu-devel] passing interrupts from QEMU to KVM

[Peter Maydell]

Hi; I'm a bit confused about handling of passing interrupts
to KVM from QEMU.

Looking at a few of the current platforms:

x86, no in-kernel irqchip:
 * we seem to use the usual non-KVM cpu_interrupt() and
   cpu_reset_interrupt() functions, which in a KVM enabled QEMU
   set flags in env->interrupt_request and do a qemu_cpu_kick().
   Then later on we might try to inject an interrupt based on
   env->interrupt_request as part of the code in kvm_arch_pre_run.

x86, in kernel irqchip:
 * I think that the hw/kvm/ devices call kvm_irqchip_set_irq(),
   which just does a kvm_vm_ioctl() to insert an interrupt.
   This seems straightforward enough.

ppc:
 * hw/ppc calls cpu_interrupt()/cpu_reset_interrupt(), but if
   KVM is enabled it *also* calls kvmppc_set_interrupt(), which
   does a kvm_vcpu_ioctl. It's not clear to me why we have to
   do both here.

Basically I'm not sure why there's all this variety here,
or why x86 does things differently for in-kernel irqchip
versus not -- I would have expected that the only difference
for an in-kernel irqchip is that there are more interrupt
lines. Kicking the CPU out of the kernel in particular seems
a very roundabout way of telling it about an interrupt,
but I assume there's a rationale for doing it that way...

The way I'm thinking about handling this for ARM is just
to have both the irqchip and no-irqchip approaches be
roughly the same: the device code just makes the relevant
KVM ioctl to inject interrupts (the semantics of the irq
number change for irqchip vs not irqchip but the general
behaviour is the same), and we never call cpu_interrupt()
if KVM is enabled. Is there any reason that wouldn't work?

-- PMM

Re: [Qemu-devel] passing interrupts from QEMU to KVM

[Avi Kivity]

On 07/19/2012 02:14 PM, Peter Maydell wrote:
> 
> Basically I'm not sure why there's all this variety here,
> or why x86 does things differently for in-kernel irqchip
> versus not -- I would have expected that the only difference
> for an in-kernel irqchip is that there are more interrupt
> lines. Kicking the CPU out of the kernel in particular seems
> a very roundabout way of telling it about an interrupt,
> but I assume there's a rationale for doing it that way...

Non-in-kernel irqchip is synchronous; everything must be executed in
vcpu context.  This is because the kernel does not queue any interrupts,
rather userspace requests an "interrupt window" (an instruction boundary
where the vcpu is ready for interrupt injection) and then qemu injects
that interrupt.

In-kernel irqchip added the logic for queueing interrupts, and is
completely asynchronous.  You can queue an interrupt from a different
thread, the kernel will inject it when the vcpu is ready.

> 
> The way I'm thinking about handling this for ARM is just
> to have both the irqchip and no-irqchip approaches be
> roughly the same: the device code just makes the relevant
> KVM ioctl to inject interrupts (the semantics of the irq
> number change for irqchip vs not irqchip but the general
> behaviour is the same), and we never call cpu_interrupt()
> if KVM is enabled. Is there any reason that wouldn't work?
> 

Let's make them even more similar, by removing !in_kernel_irqchip.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] passing interrupts from QEMU to KVM

[Peter Maydell]

On 19 July 2012 12:43, Avi Kivity <avi@redhat.com> wrote:
> On 07/19/2012 02:14 PM, Peter Maydell wrote:
>>
>> Basically I'm not sure why there's all this variety here,
>> or why x86 does things differently for in-kernel irqchip
>> versus not -- I would have expected that the only difference
>> for an in-kernel irqchip is that there are more interrupt
>> lines. Kicking the CPU out of the kernel in particular seems
>> a very roundabout way of telling it about an interrupt,
>> but I assume there's a rationale for doing it that way...
>
> Non-in-kernel irqchip is synchronous; everything must be executed in
> vcpu context.  This is because the kernel does not queue any interrupts,
> rather userspace requests an "interrupt window" (an instruction boundary
> where the vcpu is ready for interrupt injection) and then qemu injects
> that interrupt.
>
> In-kernel irqchip added the logic for queueing interrupts, and is
> completely asynchronous.  You can queue an interrupt from a different
> thread, the kernel will inject it when the vcpu is ready.

Ah, right. So in that sense I think ARM currently has two
different kinds of in-kernel-irqchip: the VGIC (has lots of
interrupt lines, memory mapped registers for control, etc),
and the not-VGIC (just two interrupt lines FIQ and IRQ).
In either case there's no requirement for synchronous operation.

>> The way I'm thinking about handling this for ARM is just
>> to have both the irqchip and no-irqchip approaches be
>> roughly the same: the device code just makes the relevant
>> KVM ioctl to inject interrupts (the semantics of the irq
>> number change for irqchip vs not irqchip but the general
>> behaviour is the same), and we never call cpu_interrupt()
>> if KVM is enabled. Is there any reason that wouldn't work?

> Let's make them even more similar, by removing !in_kernel_irqchip.

Mmm, I do rather want to just mandate use of the VGIC...
(somebody will probably come along later and try to get A9
guests working with KVM acceleration but I don't think it
will be me :-))

-- PMM

Re: [Qemu-devel] passing interrupts from QEMU to KVM

[Alexander Graf]

On 07/19/2012 02:00 PM, Peter Maydell wrote:
> On 19 July 2012 12:43, Avi Kivity <avi@redhat.com> wrote:
>> On 07/19/2012 02:14 PM, Peter Maydell wrote:
>>> Basically I'm not sure why there's all this variety here,
>>> or why x86 does things differently for in-kernel irqchip
>>> versus not -- I would have expected that the only difference
>>> for an in-kernel irqchip is that there are more interrupt
>>> lines. Kicking the CPU out of the kernel in particular seems
>>> a very roundabout way of telling it about an interrupt,
>>> but I assume there's a rationale for doing it that way...
>> Non-in-kernel irqchip is synchronous; everything must be executed in
>> vcpu context.  This is because the kernel does not queue any interrupts,
>> rather userspace requests an "interrupt window" (an instruction boundary
>> where the vcpu is ready for interrupt injection) and then qemu injects
>> that interrupt.
>>
>> In-kernel irqchip added the logic for queueing interrupts, and is
>> completely asynchronous.  You can queue an interrupt from a different
>> thread, the kernel will inject it when the vcpu is ready.
> Ah, right. So in that sense I think ARM currently has two
> different kinds of in-kernel-irqchip: the VGIC (has lots of
> interrupt lines, memory mapped registers for control, etc),
> and the not-VGIC (just two interrupt lines FIQ and IRQ).
> In either case there's no requirement for synchronous operation.

The latter is how PPC works today. We have an EXT IRQ line on the CPU 
which gets pulled up or lowered by the piece of code you were wondering 
about. It's different from x86's model, which wants to know 
synchronously when to inject an interrupt. We are fully asynchronous, 
but only provide a single line to control.

>
>>> The way I'm thinking about handling this for ARM is just
>>> to have both the irqchip and no-irqchip approaches be
>>> roughly the same: the device code just makes the relevant
>>> KVM ioctl to inject interrupts (the semantics of the irq
>>> number change for irqchip vs not irqchip but the general
>>> behaviour is the same), and we never call cpu_interrupt()
>>> if KVM is enabled. Is there any reason that wouldn't work?
>> Let's make them even more similar, by removing !in_kernel_irqchip.
> Mmm, I do rather want to just mandate use of the VGIC...
> (somebody will probably come along later and try to get A9
> guests working with KVM acceleration but I don't think it
> will be me :-))

Heh. I would really like to keep the !in_kernel_irqchip path (so only an 
EXT IRQ line exposed) available for PPC at least. It has helped 
tremendously in the past to be able to just throw a few debug printfs 
into QEMU and/or compare with TCG what's happening when things go wrong.


Alex

Re: [Qemu-devel] passing interrupts from QEMU to KVM

[Peter Maydell]

On 19 July 2012 15:13, Alexander Graf <agraf@suse.de> wrote:
> On 07/19/2012 02:00 PM, Peter Maydell wrote:
>> On 19 July 2012 12:43, Avi Kivity <avi@redhat.com> wrote:
>>> Let's make them even more similar, by removing !in_kernel_irqchip.
>>
>> Mmm, I do rather want to just mandate use of the VGIC...
>> (somebody will probably come along later and try to get A9
>> guests working with KVM acceleration but I don't think it
>> will be me :-))
>
> Heh. I would really like to keep the !in_kernel_irqchip path (so only an EXT
> IRQ line exposed) available for PPC at least. It has helped tremendously in
> the past to be able to just throw a few debug printfs into QEMU and/or
> compare with TCG what's happening when things go wrong.

I think the difficulty here is that QEMU's in_kernel_irqchip
test is being used for two things:
 * which APIC model etc should we use?
 * details of the synchronous vs asynchronous model (for instance
whether halt is handled by cpus.c depends on this: cpu_thread_is_idle
always returns false if kvm_irqchip_in_kernel())

because for x86 these two things happen for historical reasons
to be in sync. The non-x86 architectures probably need to separate
them out so that we are always using the 'asynchronous inject'
model but may (architecture-dependent) allow the user to pick
which irqchip model gets used.

-- PMM

Re: [Qemu-devel] passing interrupts from QEMU to KVM

[Alexander Graf]

On 07/19/2012 04:16 PM, Peter Maydell wrote:
> On 19 July 2012 15:13, Alexander Graf <agraf@suse.de> wrote:
>> On 07/19/2012 02:00 PM, Peter Maydell wrote:
>>> On 19 July 2012 12:43, Avi Kivity <avi@redhat.com> wrote:
>>>> Let's make them even more similar, by removing !in_kernel_irqchip.
>>> Mmm, I do rather want to just mandate use of the VGIC...
>>> (somebody will probably come along later and try to get A9
>>> guests working with KVM acceleration but I don't think it
>>> will be me :-))
>> Heh. I would really like to keep the !in_kernel_irqchip path (so only an EXT
>> IRQ line exposed) available for PPC at least. It has helped tremendously in
>> the past to be able to just throw a few debug printfs into QEMU and/or
>> compare with TCG what's happening when things go wrong.
> I think the difficulty here is that QEMU's in_kernel_irqchip
> test is being used for two things:
>   * which APIC model etc should we use?
>   * details of the synchronous vs asynchronous model (for instance
> whether halt is handled by cpus.c depends on this: cpu_thread_is_idle
> always returns false if kvm_irqchip_in_kernel())
>
> because for x86 these two things happen for historical reasons
> to be in sync. The non-x86 architectures probably need to separate
> them out so that we are always using the 'asynchronous inject'
> model but may (architecture-dependent) allow the user to pick
> which irqchip model gets used.

Or we just remove the !in_kernel path for x86. Then everyone else can be 
asynchronous even for user space interrupt controller emulation.


Alex